Scoring and judges · Autessa Evaluations
How grading works
Every verdict in Autessa Evaluations follows the same published rules, whether it comes from a test run, a live conversation or Transcript Grading. A partly done task never passes, a judge that can’t decide is retried once and then marked as an error, and nothing ever defaults to a pass.
The problem
An evaluation is only useful if you can trust it
LLM judges can time out, return nonsense or be confused by long or unusual conversations. If those quietly count as passes, the score is fiction. Clear rules for the hard cases keep the number honest.
What you get
- With an expected outcome, a conversation passes only when the judge rates it Met.
- Without one, it passes when the evaluation is Successful or Successful (Not Efficient).
- Fabricated tool arguments make the evaluation unsuccessful.
- A missing or unreadable verdict is retried once, then the conversation is marked as an error, never a pass.
- For very long conversations the judge sees as much history as its model allows, and the final part decides task completeness. Each user message is shown once.
- Manager conversations are judged on the full history and every user request; MCP agents are evaluated too.
- Drafts that were rejected and retried by response checks are labeled and not graded, retrieved documents count as grounding, and interrupted voice replies are graded on what the caller heard.
How it works
How does how grading works work in Autessa Evaluations?
Collect the conversation
Every user message, reply, tool call and tool result, plus the agent’s instructions.
Score quality
The four dimensions and a status.
Judge the outcome
If there is an expected outcome, rate it Met, Partially met or Not met.
Apply the pass rule
Met passes; otherwise the status decides. Errors count against the pass rate.
Every option
The rules
| Option | What it does |
|---|---|
| Pass or fail | Expected outcome present: only Met passes. Absent: Successful or Successful (Not Efficient) passes. |
| What is scored | Groundedness, task completeness, argument faithfulness, efficiency. |
| When a judge can’t decide | Retried once; then marked as an error. Never a pass. |
| Long conversations | As much history as the model allows; the final part decides task completeness. |
| Different kinds of agents | Managers judged on full history; MCP agents evaluated; retried drafts labeled and not graded; voice graded on what was heard. |
| Pass rate | Passed ÷ all conversations, errors included; sandbox faults reported separately. |
Permissions
Who can do what?
| Who | Can |
|---|---|
| Everyone | The same rules apply to every agent, run and transcript; they can’t be loosened per agent. |
Where the judge comes in
How the model is used here
Two kinds of model judges are involved: the quality evaluation and the expected-outcome judge. Both are held to these rules.
How grading works →Connected
How it connects to the rest of Evaluations
FAQ
Questions about how grading works
What happens if the judge fails?
Its verdict is retried once. If it still fails, the conversation is marked as an error, which counts against the pass rate. It never defaults to a pass.
Are voice conversations graded on the full reply?
No, on what the caller actually heard. If a reply was interrupted, only the spoken part is graded.
Are retried drafts graded?
No. Drafts that a response check rejected and retried are labeled and not graded; the reply that was released is.
100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours