Autessa Evaluations · Features
Every feature of the Autessa Evaluations platform
Autessa Evaluations is a complete AI agent evaluation platform you can use from day one: agent testing with test sets and simulated users, LLM-as-a-judge scoring, production monitoring, response checks before send, transcript grading for any agent, and a developer API and CLI.
At a glance
The map
§1
Agent testing →
Autessa Evaluations tests an AI agent before launch by running test sets as real conversations, driven by simulated users, and grading every one.
Test sets and test cases
Autessa Evaluations includes test sets: a named group of test cases for one agent, each with an initial message, an optional expected outcome and template variables.
Read more →Simulated users
Simulated users are reusable personas that a model plays to hold a real multi-turn conversation with your agent, so you test how it handles people, not just prompts.
Read more →Test runs
A test run executes every case as a real, headless conversation against the draft or a published version, once per chosen persona, then grades each one.
Read more →Run results and case viewer
Run results show the pass rate, outcome breakdowns, results by persona and every case transcript with the judge reasoning, with platform faults counted apart from agent failures.
Read more →Eval history
Eval history lists every test run of an agent on its own page, sortable by every column, so you can see whether each version did better than the last.
Read more →§2
Scoring and judges →
Every evaluated conversation is scored on four quality dimensions and, when you describe what should happen, judged against that expected outcome.
Groundedness, completeness, faithfulness and efficiency
Autessa Evaluations scores every evaluated conversation on four dimensions: groundedness, task completeness, argument faithfulness and efficiency, each with a justification.
Read more →Expected-outcome judge
An expected outcome is plain-English success criteria; an LLM judge reads the full transcript and rates it Met, Partially met or Not met, and only Met passes.
Read more →AutoEval
AutoEval scores live conversations automatically, either all of them or a random sample of 20% to 40%, so quality is measured on real traffic.
Read more →Evaluation notes (rubric)
Evaluation notes are a per-agent grading rubric that tells the automated judge what a finished task looks like for that agent.
Read more →How grading works
Grading follows published rules: a partly done task never passes, a missing verdict is retried once then marked an error, and it never defaults to a pass.
Read more →§3
Production monitoring →
In production, every conversation is logged in full and can be scored automatically, labeled by people and compared version by version.
Conversation logs
Agent History logs every conversation in full, with each tool request and response, so any score can be traced to the exact step behind it.
Read more →Human evaluation and labeling
Human evaluation lets reviewers label any conversation with a status and notes, and a human verdict always overrides the automated one.
Read more →Version performance
Version performance splits evaluation results by published version and draft, so a regression after a publish is visible the same day.
Read more →Quality dashboards
Quality dashboards chart pass rates, scores over time, human versus automated verdicts and what your checks caught, for one agent or all of them.
Read more →Multi-agent evaluation
Multi-agent conversations are judged on the full history and every user request, with each delegation and nested sub-agent transcript open for review.
Read more →Export and sync
Conversation history and evaluations export as JSON or CSV, or sync automatically into a database table for your own analysis and labeling.
Read more →§4
Response checks →
Response checks evaluate each reply before it is released, so a made-up price or a leaked card number never reaches the customer.
Fact grounding check
The fact grounding check pulls every fact out of a reply and blocks or flags any that did not come from the customer, a tool, retrieved knowledge or the instructions.
Read more →Sensitive data detectors and pattern rules
Sensitive data detectors and pattern rules find PII, credentials, leaked instructions and your own formats in a reply, and can remove just the matched text.
Read more →Script rules
Script rules are short AutessaScript checks for logic a pattern cannot express, like a competitor list kept in one place.
Read more →AI review
AI review is an LLM-as-a-judge check that runs before the reply is sent, judging tone, policy and completeness against criteria you write in plain language.
Read more →Actions and release modes
Each check has one action, the strongest action wins, and how replies are released (right away, sentence by sentence or held) follows from the checks you turn on.
Read more →Testing checks and reading results
The Test checks panel shows each check result and the exact text a customer would receive, and every checked reply in history carries its own results card.
Read more →§5
Grade any agent →
Transcript Grading applies the same judges to conversations from any agent, wherever it runs, with no Autessa agent required.
Transcript Grading
Transcript Grading scores conversations from any agent, built on Autessa or anywhere else, with the same judges that score Autessa agents.
Read more →Batch grading jobs
Batch grading jobs grade up to 500 transcripts in the background, with a signed completion webhook and your own ids for safe resubmission.
Read more →§6
Developers →
Everything in Evaluations has an API and a CLI command, so grading and test runs fit into scripts and release pipelines.
Grading API and CLI
The Grading API and `autessa evaluations` CLI grade one transcript in a single call or a batch in the background, from any language or pipeline.
Read more →Sandbox CLI
The `autessa sandbox` CLI creates test sets, simulated users and runs from code, so an agent test can gate a release.
Read more →Human evaluation and statistics API
The agents API records human evaluations and returns human, automated and human-first statistics with a per-version breakdown.
Read more →100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours