§3 · Autessa Evaluations
Production monitoring
In production, every conversation is logged in full and can be scored automatically, labeled by people and compared version by version.
The area
What is production monitoring in Autessa Evaluations?
Online evaluation keeps measuring after launch. AutoEval scores live conversations, Agent History keeps the full log of each one with every tool call, and reviewers can label any conversation with a verdict that overrides the automated one.
Results split by version and charted by day, so a regression after a publish is visible right away, and multi-agent runs open to every delegation and sub-agent transcript.
Subtopics
Everything in production monitoring
Online evaluation for live AI agents: full conversation logs, human labeling, version-by-version performance, quality dashboards and multi-agent evaluation.
Conversation logs
Agent History logs every conversation in full, with each tool request and response, so any score can be traced to the exact step behind it.
Read more →Human evaluation and labeling
Human evaluation lets reviewers label any conversation with a status and notes, and a human verdict always overrides the automated one.
Read more →Version performance
Version performance splits evaluation results by published version and draft, so a regression after a publish is visible the same day.
Read more →Quality dashboards
Quality dashboards chart pass rates, scores over time, human versus automated verdicts and what your checks caught, for one agent or all of them.
Read more →Multi-agent evaluation
Multi-agent conversations are judged on the full history and every user request, with each delegation and nested sub-agent transcript open for review.
Read more →Export and sync
Conversation history and evaluations export as JSON or CSV, or sync automatically into a database table for your own analysis and labeling.
Read more →Other areas
Agent testing →
Autessa Evaluations tests an AI agent before launch by running test sets as real conversations, driven by simulated users, and grading every one.
Scoring and judges →
Every evaluated conversation is scored on four quality dimensions and, when you describe what should happen, judged against that expected outcome.
Response checks →
Response checks evaluate each reply before it is released, so a made-up price or a leaked card number never reaches the customer.
Grade any agent →
Transcript Grading applies the same judges to conversations from any agent, wherever it runs, with no Autessa agent required.
Developers →
Everything in Evaluations has an API and a CLI command, so grading and test runs fit into scripts and release pipelines.
100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours