Know how your AI agent really does.

Autessa Evaluations is an AI agent evaluation platform. It tests agents before launch with test sets and simulated users, scores live conversations for groundedness, task completeness, argument faithfulness and efficiency, checks each reply before the customer sees it, and grades transcripts from any agent through an API.

Yes. Turn on AutoEval, add the starter personas and run your first test set on day one. Then tailor it in Autessa: write evaluation notes so the judge grades against your own definition of done, add personas that sound like your customers, and set the response checks and actions that fit your policies.

Each conversation is scored on four dimensions and given a status. Groundedness asks whether claims are backed by tools, documents or context; task completeness asks whether the job got done; argument faithfulness asks whether every tool argument came from the conversation; efficiency asks whether tools were used without waste. The status is Successful, Successful (Not Efficient), Successful (Not All Steps) or Unsuccessful, and each dimension carries a written justification.

Yes. The quality evaluation and the expected-outcome verdict both come from model judges, and AI review checks use a model you pick. Judges never default to a pass: a missing or unreadable verdict is retried once and then marked as an error. Rule-based checks, like the fact grounding check and sensitive data detectors, use no model at all.

Yes. Build a test set of cases, each with an opening message and an optional expected outcome, then run it against the draft or a published version. Add simulated users and every case becomes a real multi-turn conversation with a persona that pushes back, asks for a human or checks every number.

Yes. Transcript Grading accepts a conversation from any agent, with user and assistant messages, tool calls and tool results, plus the instructions the agent was given. You can grade one in the browser, call the Grading API, or send up to 500 per batch job with a signed webhook when it finishes.

Turn on response checks. The fact grounding check catches a number, date, link, ID or quote that did not come from the conversation, a tool or your instructions, and sensitive data detectors find SSNs, card numbers, credentials and leaked instructions. Each check can flag, remove the matched text, block the reply or ask the agent to try again.

Yes, and their verdict wins. Anyone with write access to an agent can record a human evaluation with a status and notes on any conversation, and a human evaluation always takes precedence over the automated one in history and statistics. Full logs of every conversation, including each tool request and response, make review and labeling quick.

Two ways. Run the same test set against the new draft and the current published version and compare pass rates in eval history, and watch the version performance breakdown in Agent History, which splits live results by version so a drop after a publish is visible right away.

Yes. Voice replies are graded on what the caller actually heard, response checks release each spoken sentence only after it passes, and manager (multi-agent) conversations are judged on the full history and every user request, with each sub-agent transcript open for review.

Yes. `autessa evaluations` grades transcripts and manages batch jobs, `autessa sandbox` manages test sets, simulated users and runs, and `autessa agents submit-evaluation` and `autessa agents statistics` record human verdicts and return success counts by version. The Grading API is available to scripts over an API key at /clients/evaluations.