Agent testing · Autessa Evaluations
Run results and case viewer
Run results show how an agent did on a test run: the pass rate, how many passed, failed and errored, results by persona, why each conversation stopped, and every case’s transcript with the judge’s reasoning. Problems on the platform side are counted separately as sandbox faults, so they are never mistaken for agent failures.
The problem
A single score hides the reason
A pass rate tells you something moved, not why. Teams need to see which cases failed, for which kind of customer, how the conversation ended and what the judge read in the transcript, so the fix lands on the right instruction or tool.
What you get
- Pass rate is passed divided by every conversation in the run, errors included.
- Outcome breakdowns show cases by expected-outcome rating (Met, Partially met, Not met) and by evaluation status.
- Sandbox faults, like the simulated user failing or a timeout, are counted apart from agent failures.
- Results by persona show each simulated user’s pass rate and turns, side by side.
- Stop reasons explain every ending: goal reached, stop condition, max turns, agent error, simulated user error, unsupported interaction or timeout.
- Average turns used shows how hard the agent made people work.
- Each case opens to its full transcript, tool calls, scores and the judge’s reasoning.
How it works
How does run results and case viewer work in Autessa Evaluations?
Watch it fill in
While a run is going, progress shows how many conversations have finished.
Read the headline
Pass rate, passed, failed, errors, sandbox faults and average turns used.
Compare personas
In a multi-persona run, the By persona table shows which customers the agent struggles with.
Open a case
The case viewer shows the expected outcome, the judge’s rating and reasoning, each dimension score, turns used, stop reason and the transcript.
Fix and rerun
Change the instruction or tool behind the failure, then rerun the same test set on the draft.
Every option
What a run reports
| Option | What it does |
|---|---|
| Pass rate | Passed ÷ every conversation in the run, errors included. |
| Passed / Failed / Errors | Cases that passed, ran but failed grading, or could not run or be graded. |
| Sandbox faults | The part of errors caused by the platform, such as the simulated user failing or a timeout. |
| Avg turns used | One turn is one user message and the agent’s reply. |
| By persona | Pass rate and turns for each simulated user in the run. |
| Stop reason | Goal reached, Stop condition, Max turns, Agent error, Simulated user error, Unsupported interaction, Timeout, or Single turn done. |
| Case status | Passed, Failed or Error, with the expected-outcome rating and evaluation status. |
| Judge’s reasoning | The justification behind the rating, citing the transcript. |
Permissions
Who can do what?
| Who | Can |
|---|---|
| Read | View the agent and its versions, test-run it, and read its history and results. |
| Write | Everything Read can, plus edit the draft and record human evaluations. |
| Publish | Publish and roll back versions, which is what customers get. |
| Admin | Share and delete the agent. The owner is always an Admin. |
Where the judge comes in
How the model is used here
Two model judges grade each case: the quality evaluation scores the four dimensions and gives a status, and, when the case has an expected outcome, a second judge rates it Met, Partially met or Not met and writes one or two sentences citing the transcript.
How grading works →Connected
How it connects to the rest of Evaluations
EVALUATIONS FEATURE
Quality dimensions
Every case carries groundedness, task completeness, argument faithfulness and efficiency.
EVALUATIONS FEATURE
How grading works
The pass rule, sandbox faults and stop reasons explained in one place.
EVALUATIONS FEATURE
Conversation logs
The case transcript shows each tool request and result, just like live history.
FAQ
Questions about run results and case viewer
How is the pass rate calculated?
Passed divided by every conversation in the run, with errors counted against it. A run with 10 cases and 3 personas has 30 conversations.
What is a sandbox fault?
An error on the platform side, like the simulated user failing or a timeout. It is shown separately so it isn’t mistaken for the agent failing.
Why did a conversation end early?
The stop reason says. Goal reached and stop condition are normal endings; max turns, agent error, simulated user error, unsupported interaction and timeout explain early ones.
Can I see what the agent actually said?
Yes. Every case opens to its full transcript, including each tool call, with the scores and the judge’s reasoning beside it.
100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours