AppsEvaluations

Scoring and judges · Autessa Evaluations

Expected-outcome judge

An expected outcome is plain-English success criteria for a conversation, such as “the agent looks up the order and confirms a refund.” An LLM judge reads the full transcript and rates it Met, Partially met or Not met, with one or two sentences citing the transcript, and only Met passes.

A core AI agent evaluation capability, done properly
Start free
Expected outcome · LLM judgeSAMPLE
EXPECTED OUTCOME

“The agent refunds the duplicate $48 charge and says when it will arrive.”

METThe conversation achieved the expected outcome.PASS
PARTIALLY METPart of the expected outcome happened, but not all of it.FAIL
NOT METThe expected outcome didn’t happen.FAIL

Judge’s reasoning: The agent confirmed the duplicate $48 charge from the billing lookup, issued the refund and said it arrives in 3–5 business days.

The problem

Good quality is not the same as the right result

A conversation can be grounded and polite and still not do the one thing the business needed. Expected outcomes let you say what had to happen in your own words, without writing code or exact-match strings that break whenever the agent rephrases.

What you get

  • Write the outcome in plain English; no code or exact strings.
  • Three ratings: Met (clearly happened), Partially met (some but not all), Not met.
  • Only Met passes. Partially met fails, so half-finished work never inflates the score.
  • The judge writes a short justification that cites the transcript.
  • Works on test cases, in Transcript Grading, and over the Grading API.
  • When an expected outcome is present, it decides pass or fail; the quality evaluation is still shown for detail.

How it works

How does expected-outcome judge work in Autessa Evaluations?

  1. Describe success

    On a test case, or with a transcript you grade, write what a successful conversation looks like.

  2. The conversation runs or is sent

    A test run plays it out, or you send a finished transcript.

  3. The judge rates it

    Met, Partially met or Not met, with a justification.

  4. Pass or fail is decided

    Met passes. Anything else fails.

Every option

Ratings

OptionWhat it does
MetThe conversation achieved the expected outcome. Only Met passes.
Partially metPart of the expected outcome happened, but not all of it. Fails.
Not metThe expected outcome didn’t happen. Fails.
JustificationOne or two sentences citing the transcript.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →
Grade a transcript against an expected outcome
autessa evaluations grade \
  --agentInstructions "You are a billing support agent. Refund duplicate charges." \
  --expectedOutcome "The agent refunds the duplicate charge." \
  --turns '[{"role":"USER","content":"I was charged twice for order 1042."},{"role":"ASSISTANT","content":"I refunded the duplicate $48. It arrives in 3-5 days."}]'

Permissions

Who can do what?

WhoCan
Write access to the agentAdd and edit expected outcomes on test cases.
Anyone grading transcriptsSend an expected outcome with each transcript.

Where the judge comes in

How the model is used here

An LLM judge compares the full transcript to your expected outcome. If its verdict is missing or unreadable it is retried once, and if it still fails the conversation is marked as an error. It never defaults to a pass.

How grading works →

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Test sets

Every test case can carry its own expected outcome.

EVALUATIONS FEATURE

Transcript Grading

Use the same judge on conversations from any agent.

EVALUATIONS FEATURE

How grading works

The pass rule and what happens when a judge can’t decide.

FAQ

Questions about expected-outcome judge

What does Partially met mean?

Part of the expected outcome happened, but not all of it. It counts as a fail, so a half-finished task never passes.

Do I need code to write an expected outcome?

No. It is plain English, like “the agent refunds the duplicate $48 charge and says when it will arrive.”

What if there is no expected outcome?

Then the quality evaluation decides: Successful and Successful (Not Efficient) pass, anything else fails.

Try expected-outcome judge in Autessa Evaluations

An expected outcome is plain-English success criteria; an LLM judge reads the full transcript and rates it Met, Partially met or Not met, and only Met passes.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours