Agent testing · Autessa Evaluations

Test runs

A test run executes every case of a test set as a real, headless conversation against your agent, then grades each one. You choose the version, the simulated users and whether to run every case or a random sample, so regression testing for AI agents is one click.

A core AI agent evaluation capability, done properly

Start free

Run "Refund requests"SAMPLE

Agent version

Published version… 4

Simulated users

Choose personas… Impatient customer, Detail-checker

Sample

☑ Run a random sample of 150 of 1,000 cases

Every case runs once per persona: 150 cases × 2 personas = 300 conversations.

Start run

The problem

You need to know before customers do

A prompt edit that fixes one case can quietly break three others. Without a run against the same cases, you find out from complaints. Test runs let you check the draft before you publish, and recheck a published version any time.

What you get

Run against the current draft or any published version by number.
Use the test set’s default simulated user, choose several personas, or run single turn with none.
With several personas, every case runs once per persona, so total conversations are cases times personas.
Run a random sample of a large test set, say 150 of 1,000 cases, for a quick read before a full run.
Runs happen in the background, so you can keep working and come back to the results.
Every finished conversation is graded with the automated quality evaluation and, when the case has one, the expected-outcome judge.

How it works

How does test runs work in Autessa Evaluations?

Choose Run test set
From the Sandbox, open a test set that has at least one case and choose Run test set.
Pick the agent version
Draft (the current working version) or a published version number.
Pick simulated users
Test set default, None for single turn, or choose personas from your list.
Optionally sample
Turn on sampling and say how many cases to pick at random.
Start the run
The run is queued, conversations run headlessly, and each one is graded as it finishes.

Every option

Run options

Option	What it does
Agent version	Draft (current working version), or Published version with a version number.
Simulated users	Test set default, None (single turn), or Choose personas.
Sample	Run only this many cases, picked uniformly at random. Leave off to run every case.
Run status	Pending, Running, Completed or Failed.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →

Start a run from the CLI, then poll it

autessa sandbox runs-start \
  --testSetId <testSetId> \
  --versionNumber 4 \
  --policyIds '["<policyId>","<policyId>"]' \
  --sampleSize 150

autessa sandbox runs-get --runId <runId>

Permissions

Who can do what?

Who	Can
Read	View the agent and its versions, test-run it, and read its history and results.
Write	Everything Read can, plus edit the draft and record human evaluations.
Publish	Publish and roll back versions, which is what customers get.
Admin	Share and delete the agent. The owner is always an Admin.

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Run results

Pass rate, outcome breakdowns and every transcript, the moment the run finishes.

EVALUATIONS FEATURE

Eval history

Every run lands in the agent’s eval history so versions can be compared.

AUTESSA PLATFORM

AI Agents

Runs test the real agent with its real tools, instructions and versions.

FAQ

Questions about test runs

Do test runs use the real agent?

Yes. Each case is a real conversation with the agent’s draft or published version, including its tools, so point tools that change data at test data first.

Can every agent be tested headlessly?

Most can. An agent that needs a browser or local tool, or asks the user for a private value, can’t be driven headlessly; those cases end with an error and the stop reason Unsupported interaction.

How do I test a published version against the draft?

Start two runs of the same test set, one on the draft and one on the published version number, then compare their pass rates in eval history.

What does sampling do?

It runs a random subset of the test set’s cases. Use it for a fast read on a large suite before a full run.

Also in Agent testing:Test sets and test cases Simulated users Run results and case viewer Eval history

Try test runs in Autessa Evaluations

A test run executes every case as a real, headless conversation against the draft or a published version, once per chosen persona, then grades each one.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours