§1 · Autessa Evaluations
Agent testing
Autessa Evaluations tests an AI agent before launch by running test sets as real conversations, driven by simulated users, and grading every one.
The area
What is agent testing in Autessa Evaluations?
Agent testing in Autessa is offline evaluation built around conversations. You write test cases the way a customer would start a conversation, describe what should happen, and run them as real, headless conversations against the agent’s draft or any published version.
Add simulated users and each case becomes a multi-turn conversation with a persona that is impatient, confused, frustrated or skeptical. Every finished conversation is graded, and the run reports a pass rate with sandbox faults counted apart from agent failures.
Subtopics
Everything in agent testing
Test AI agents before people use them: test sets, expected outcomes, simulated users that talk like real customers, runs against any version, and graded results.
Test sets and test cases
Autessa Evaluations includes test sets: a named group of test cases for one agent, each with an initial message, an optional expected outcome and template variables.
Read more →Simulated users
Simulated users are reusable personas that a model plays to hold a real multi-turn conversation with your agent, so you test how it handles people, not just prompts.
Read more →Test runs
A test run executes every case as a real, headless conversation against the draft or a published version, once per chosen persona, then grades each one.
Read more →Run results and case viewer
Run results show the pass rate, outcome breakdowns, results by persona and every case transcript with the judge reasoning, with platform faults counted apart from agent failures.
Read more →Eval history
Eval history lists every test run of an agent on its own page, sortable by every column, so you can see whether each version did better than the last.
Read more →Other areas
Scoring and judges →
Every evaluated conversation is scored on four quality dimensions and, when you describe what should happen, judged against that expected outcome.
Production monitoring →
In production, every conversation is logged in full and can be scored automatically, labeled by people and compared version by version.
Response checks →
Response checks evaluate each reply before it is released, so a made-up price or a leaked card number never reaches the customer.
Grade any agent →
Transcript Grading applies the same judges to conversations from any agent, wherever it runs, with no Autessa agent required.
Developers →
Everything in Evaluations has an API and a CLI command, so grading and test runs fit into scripts and release pipelines.
100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours