Scoring and judges · Autessa Evaluations
AutoEval
AutoEval is online evaluation for live agents. Turn it on and Autessa scores production conversations automatically, either every conversation or a random sample between 20% and 40%, so quality is measured on real traffic, not only on test sets.
The problem
Test sets only cover what you thought of
Real users bring requests nobody wrote a test case for. Without scoring live conversations, the first sign of a problem is a complaint. AutoEval keeps a steady quality signal on the traffic that matters most.
What you get
- Three modes: Off, On (evaluate all conversations), or Random sample.
- Random sample rates run from 20% to 40% of conversations.
- Scored with the same four dimensions and statuses used everywhere else.
- Evaluation notes unlock on an agent once AutoEval is on, so the judge can use your rubric.
- Results feed Agent History, version performance and the quality dashboards.
- A human evaluation always takes precedence over the automated one.
How it works
How does autoeval work in Autessa Evaluations?
Open Configure AutoEval
Choose Off, On or Random sample.
Set a sample rate
For Random sample, choose between 20% and 40%.
Save
The setting applies company-wide: new conversations across your agents are evaluated automatically from then on.
Read the results
Each conversation in Agent History shows its evaluation status and scores.
Every option
AutoEval modes
| Option | What it does |
|---|---|
| Off | No automatic evaluations are performed. |
| On (Evaluate All Conversations) | All conversations are automatically evaluated. |
| Random Sample | A random sample of conversations is automatically evaluated, between 20% and 40%. |
Permissions
Who can do what?
| Who | Can |
|---|---|
| People who manage billing for the company | Configure AutoEval in the web app. |
| Anyone who can read the agent | See automated evaluations on conversations they can see. |
Where the judge comes in
How the model is used here
A model judge scores each sampled conversation. It runs on your company default model, which can be your own connected model provider.
How grading works →Connected
How it connects to the rest of Evaluations
FAQ
Questions about autoeval
Does AutoEval score every conversation?
It can. Choose On to evaluate all conversations, or Random sample to evaluate between 20% and 40% of them.
Where do I configure AutoEval?
In the web app. It isn’t available through the CLI.
Can a person overrule AutoEval?
Yes. A human evaluation recorded on a conversation always takes precedence in history and statistics.
100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours