Agent testing · Autessa Evaluations

Test sets and test cases

Autessa Evaluations includes test sets for AI agent testing. A test set is a named group of test cases for one agent, and each case is an opening message, an optional expected outcome in plain English, and values for the agent’s prompt template variables.

A core AI agent evaluation capability, done properly

Start free

Sandbox · Test set · Refund requests4 CASESSAMPLE

Double charge

Initial message: Hi, I was charged twice for order 1042.

Expected outcome: Refunds the duplicate charge and says when it arrives.

Refund outside window

Initial message: Can I get a refund on something from March?

Expected outcome: Explains the 30-day policy and offers a store credit.

Wants a person

Initial message: I need to talk to a real human about my bill.

Expected outcome: Hands off to a person without arguing.

Case 4

Initial message: Why is my invoice higher this month?

Expected outcome: none, graded on quality only

Default simulated user: Impatient customerRun test set

The problem

Spot checks in a chat window don’t scale

Most teams test an agent by typing a few questions into it after every change. That misses the cases that broke last month, never covers the odd requests real people send, and gives no number to compare against next time. A test set turns those scattered checks into a repeatable suite you can run against any version.

What you get

Each test case has a name, an initial message, an optional expected outcome and prompt template variables.
Cases keep a run order, and you can move them up or down or remove them in the editor.
A test set can name a default simulated user, so every run of it is multi-turn unless you choose otherwise.
A blank case name becomes “Case N”, so you can paste in cases quickly and name them later.
A case with no expected outcome is still graded, on the four quality dimensions.
Deleting a test set keeps its existing runs, so history is never lost.

How it works

How does test sets and test cases work in Autessa Evaluations?

Pick the agent
Open the Sandbox and choose the agent you want to test. Its test sets and recent runs appear below.
Create a test set
Give it a name and an optional description of what it covers, and optionally a default simulated user.
Add cases
For each case write the first message a user sends, describe what a successful conversation looks like, and fill in any template variables for that scenario.
Order and refine
Move cases into the order you want them run, and edit or remove them as the agent changes.
Run it
Start a run against the draft or a published version and read the graded results.

Every option

Test set and test case fields

Option	What it does
Name	The test set name shown in runs and eval history.
Description	Optional. What the test set covers.
Default simulated user	Optional. The persona runs use when they don’t choose one. Without one, runs are single turn.
Case name	A label for the case, such as “Customer asks for a refund”. Defaults to “Case N”.
Initial message	Required. The first message the simulated user sends to the agent.
Expected outcome	Optional. Plain-English success criteria an LLM judge grades against the full transcript.
Prompt template variables	Key and value pairs for the agent’s template variables in this scenario.
Order	Cases run in the order shown; move them up or down.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →

Create a test set with cases from the CLI

autessa sandbox test-sets-create \
  --agentId <agentId> \
  --name "Refund requests" \
  --description "Billing questions that should end in a refund" \
  --cases '[{"name":"Double charge","initialMessage":"Hi, I was charged twice for order 1042.","expectedOutcome":"the agent looks up the order and confirms a refund"}]'

Permissions

Who can do what?

Who	Can
Read	View the agent and its versions, test-run it, and read its history and results.
Write	Everything Read can, plus edit the draft and record human evaluations.
Publish	Publish and roll back versions, which is what customers get.
Admin	Share and delete the agent. The owner is always an Admin.

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Simulated users

Give a test set a default persona so every case becomes a real multi-turn conversation.

EVALUATIONS FEATURE

Expected-outcome judge

The expected outcome on each case is what decides pass or fail.

AUTESSA PRODUCT

Synthetic Data

Generate realistic sample records for the tools your test cases exercise, without real customer data.

FAQ

Questions about test sets and test cases

What is a test set in Autessa?

A test set is a group of test cases you run against one agent. Each case is an opening user message, an optional expected outcome and optional template variables, kept in run order.

Do I have to write an expected outcome for every case?

No. A case without one is graded on the automated quality evaluation alone, and passes when the status is Successful or Successful (Not Efficient). Add an expected outcome when you know exactly what should happen.

Can I create test sets from code?

Yes. `autessa sandbox test-sets-create` takes the agent, a name, an optional default policy and the cases in one call, and `test-sets-update`, `test-sets-get`, `test-sets-list` and `test-sets-delete` manage them afterwards.

What happens to old runs if I delete a test set?

They are kept. Deleting a test set removes it and its cases, but existing runs and their results stay in eval history.

Also in Agent testing:Simulated users Test runs Run results and case viewer Eval history

Try test sets and test cases in Autessa Evaluations

Autessa Evaluations includes test sets: a named group of test cases for one agent, each with an initial message, an optional expected outcome and template variables.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours