Agent testing · Autessa Evaluations

Simulated users

Simulated users make AI agent testing multi-turn. A simulated user is a reusable persona, with a background, a goal, behavior instructions and stop conditions, that a model plays to hold a real conversation with your agent, the way a customer would.

Goes beyond a typical eval tool

Start free

Simulated users · Starter personasSAMPLE

Impatient customer

Short on time, terse, pushes for a fast answer.

max turns · 6

Confused first-timer

Unsure what they need; vague, needs guiding.

max turns · 10

Frustrated, wants a human

Upset after a bad experience; tests escalation.

max turns · 8

Detail-checker

Precise and skeptical; checks every number.

max turns · 8

The problem

Real people don’t send one tidy message

A one-shot prompt tells you how an agent answers a perfect question. Real customers are impatient, vague, upset or suspicious of every number, and they follow up. Simulated users bring those people into testing, so you see whether the agent asks the right question, holds its ground, or hands off to a person when it should.

What you get

Each simulated user has a name, a persona, a goal, behavior instructions, stop conditions and a max number of turns.
Four starter personas are one click away: Impatient customer, Confused first-timer, Frustrated and wants a human, and Detail-checker.
The simulated user can end the conversation early once its goal is reached or a stop condition happens.
Max turns defaults to 6 and is capped at 10, so every conversation finishes.
Pick several personas in one run and every case runs once per persona, with results broken down by persona.
You can pick the model a simulated user runs on, or leave it on the default.
Personas are shared across your company’s test sets, so one well-written persona tests every agent.

How it works

How does simulated users work in Autessa Evaluations?

Start from a starter persona or your own
Add the four starter personas in one click, or create a new simulated user from scratch.
Describe who they are
Write the persona (background, tone, what they know) and the goal they are trying to accomplish.
Say how they behave
Behavior instructions shape each turn, such as pushing back before accepting an answer, or only answering the first question asked.
Set when they stop
Stop conditions and max turns decide when the conversation ends besides reaching the goal.
Use them in runs
Set one as a test set’s default, or choose several when you start a run.

Every option

Simulated user settings

Option	What it does
Name	Required. For example “Frustrated customer”.
Persona	Required. Who the simulated user is: background, personality, tone and knowledge.
Goal	What the simulated user is trying to accomplish.
Behavior instructions	How to behave turn to turn: short replies, digressions, typos, escalation.
Stop conditions	Conditions, besides reaching the goal, that end the conversation.
Max turns	Max user turns per conversation. Defaults to 6, capped at 10.
Model	Optional model for the simulated user; leave empty for the default.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →

Create a simulated user from the CLI

autessa sandbox policies-create \
  --name "Impatient customer" \
  --persona "A busy customer who is short on time and mildly annoyed. Types short sentences and skips pleasantries." \
  --goal "Get the request from your first message fully resolved." \
  --behaviorInstructions "Answer questions briefly. Push back on long explanations." \
  --stopConditions "Stop once the request is resolved, or after the agent has failed to make progress twice." \
  --maxTurns 6

Permissions

Who can do what?

Who	Can
Agent builders	Create, edit and delete simulated users, and choose them for runs of agents they can test.
Test set owners	Set a simulated user as a test set’s default. If it is deleted, that test set falls back to single turn.

Where the judge comes in

How the model is used here

A model plays each simulated user, writing every message after the first from its persona, goal and behavior instructions, and deciding when its goal is reached. If the simulated user itself fails, the case is counted as a sandbox fault rather than an agent failure.

How grading works →

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Test runs

Choose several personas at run time and every case runs once per persona.

EVALUATIONS FEATURE

Results by persona

See pass rate and average turns for each persona side by side.

AUTESSA PRODUCT

Receptionist

Rehearse how a phone receptionist handles impatient or confused callers before it takes real calls.

FAQ

Questions about simulated users

What is a simulated user?

A simulated user is a persona that a model plays to talk to your agent over several turns. You describe who they are, what they want, how they behave and when they stop.

How long can a simulated conversation be?

Up to 10 user turns. The default is 6, and the conversation ends sooner when the goal is reached or a stop condition happens.

What are the starter personas?

Four ready-made simulated users: an impatient customer who pushes for a fast answer, a confused first-timer who needs guiding, a frustrated customer who asks for a human, and a detail-checker who confirms every number and date.

Can I test the same cases with different kinds of customers?

Yes. Pick several personas when you start a run. Every case runs once per persona, and the results include a by-persona breakdown.

Also in Agent testing:Test sets and test cases Test runs Run results and case viewer Eval history

Try simulated users in Autessa Evaluations

Simulated users are reusable personas that a model plays to hold a real multi-turn conversation with your agent, so you test how it handles people, not just prompts.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours