§2 · Autessa Evaluations
Scoring and judges
Every evaluated conversation is scored on four quality dimensions and, when you describe what should happen, judged against that expected outcome.
The area
What is scoring and judges in Autessa Evaluations?
Every evaluated conversation in Autessa, whether it came from a test run, live traffic or an uploaded transcript, is scored by the same LLM-as-a-judge evaluation: groundedness, task completeness, argument faithfulness and efficiency, each with a justification.
When you describe the expected outcome, a second judge rates the conversation Met, Partially met or Not met. The pass rules are published and strict: a partly done task never passes, and an unreadable verdict is an error, never a pass.
Subtopics
Everything in scoring and judges
How Autessa scores agent conversations: groundedness, task completeness, argument faithfulness and efficiency, an expected-outcome judge, AutoEval and rubrics.
Groundedness, completeness, faithfulness and efficiency
Autessa Evaluations scores every evaluated conversation on four dimensions: groundedness, task completeness, argument faithfulness and efficiency, each with a justification.
Read more →Expected-outcome judge
An expected outcome is plain-English success criteria; an LLM judge reads the full transcript and rates it Met, Partially met or Not met, and only Met passes.
Read more →AutoEval
AutoEval scores live conversations automatically, either all of them or a random sample of 20% to 40%, so quality is measured on real traffic.
Read more →Evaluation notes (rubric)
Evaluation notes are a per-agent grading rubric that tells the automated judge what a finished task looks like for that agent.
Read more →How grading works
Grading follows published rules: a partly done task never passes, a missing verdict is retried once then marked an error, and it never defaults to a pass.
Read more →Other areas
Agent testing →
Autessa Evaluations tests an AI agent before launch by running test sets as real conversations, driven by simulated users, and grading every one.
Production monitoring →
In production, every conversation is logged in full and can be scored automatically, labeled by people and compared version by version.
Response checks →
Response checks evaluate each reply before it is released, so a made-up price or a leaked card number never reaches the customer.
Grade any agent →
Transcript Grading applies the same judges to conversations from any agent, wherever it runs, with no Autessa agent required.
Developers →
Everything in Evaluations has an API and a CLI command, so grading and test runs fit into scripts and release pipelines.
100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours