AppsEvaluations

Developers · Autessa Evaluations

Sandbox CLI

The `autessa sandbox` CLI is automated testing for AI agents from the terminal. Create test sets with their cases, define simulated-user policies, start a run against the draft or a published version, and poll for graded results. Every case runs as a real, headless conversation and is graded with the platform evaluation and, when the case has one, the expected-outcome judge.

A core AI agent evaluation capability, done properly
Start free
Persona, test set, run
autessa sandbox policies-create \
  --name "Frustrated customer" \
  --persona "A customer who has already had a bad experience and is frustrated. Polite but firm." \
  --goal "Get the request from your first message fully resolved." \
  --stopConditions "Stop once the request is resolved or the agent hands you to a person." \
  --maxTurns 8

autessa sandbox test-sets-create \
  --agentId <agentId> \
  --name "Refunds" \
  --cases '[{"name":"Customer asks for a refund","initialMessage":"I was charged twice for order 1042.","expectedOutcome":"the agent looks up the order and confirms a refund"}]'

autessa sandbox runs-start --testSetId <testSetId> --policyIds '["<policyId>"]'
autessa sandbox runs-get --runId <runId>

The problem

A test you have to click through is a test that gets skipped

Agent changes ship fast: a new instruction, a new tool, a new model. The Sandbox CLI lets you keep test sets as data and run them from a script before you publish, so "did this break anything?" has an answer in a pass rate, not a hunch.

What you get

  • Autessa test sets hold test cases for one agent: an opening user message, prompt template variables and an optional expected outcome.
  • Simulated-user policies are reusable personas with a goal, behavior instructions, stop conditions and max turns (default 6, capped at 10).
  • runs-start tests the draft by default or a published version with --versionNumber.
  • Pass --policyIds to run every case once per persona, --omitPolicy for single turn, or --sampleSize for a random sample.
  • Runs are asynchronous: runs-start returns a PENDING run, then poll runs-get until COMPLETED or FAILED.
  • A run passRate is passed divided by all cases, so errors count against it; sandbox faults are counted separately.
  • Agents that need a browser or local tool, or ask the user for private arguments, cannot be tested headlessly; those cases end with UNSUPPORTED_INTERACTION.

How it works

How does sandbox cli work in Autessa Evaluations?

  1. Create a simulated user

    policies-create with a name and persona, plus goal, behavior, stop conditions and max turns.

  2. Create a test set

    test-sets-create for an agent, with its cases in the same call and an optional default policy.

  3. Start a run

    runs-start with the test set id, and optionally a version, personas or a sample size.

  4. Poll and read

    runs-get returns every case result: status, pass or fail, turns used, stop reason and the evaluation.

Every option

Sandbox commands

OptionWhat it does
test-sets-create / update / get / list / delete--agentId, --name, --description, --defaultPolicyId, --cases (name, initialMessage, promptTemplateVariables, expectedOutcome).
policies-create / update / list / delete--name, --persona (required), --goal, --behaviorInstructions, --stopConditions, --maxTurns, --model.
runs-start--testSetId (required), --versionNumber, --policyIds, --omitPolicy, --sampleSize.
runs-getOne run with every case result; the method to poll after runs-start.
runs-list / runs-case-get / runs-deleteList runs newest first, fetch one case result with its run summary, or delete a run.

Permissions

Who can do what?

WhoCan
CLI userManage test sets, policies and runs in the active company.
Agent viewer (READ)Test-run the agent.
Team membersSee the same test sets and runs in the Sandbox and the agent’s Eval history.

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Test sets

The same test sets you edit in the Sandbox web page.

EVALUATIONS FEATURE

Simulated users

Personas that push back, ask for a human or check every number.

EVALUATIONS FEATURE

Run results

Pass rate, stop reasons, results by persona and every transcript.

FAQ

Questions about sandbox cli

Do sandbox runs call my real tools?

Yes. Each case is a real, headless conversation against the agent, so point tools with side effects at test data first.

Can I test a published version, not just the draft?

Yes. Pass --versionNumber to runs-start; omit it to test the current draft.

How do I run a quick check on a large test set?

Pass --sampleSize to run only that many cases, picked uniformly at random.

Why did a case end with UNSUPPORTED_INTERACTION?

The agent needed a browser or local tool, or asked the user for a private argument, which a headless run cannot provide.

Try sandbox cli in Autessa Evaluations

The `autessa sandbox` CLI creates test sets, simulated users and runs from code, so an agent test can gate a release.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours