AppsEvaluations

Grade any agent · Autessa Evaluations

Transcript Grading

Transcript Grading is conversation evaluation for any AI agent, built on Autessa or anywhere else. Paste or upload a conversation, add the instructions the agent was given and an optional expected outcome, and get a pass or fail verdict with the scores and the judges’ reasons in a few seconds. It uses the same judges that score Autessa agents.

Goes beyond a typical eval tool
Start free
Transcript Grading · Grade a transcriptSAMPLE
Agent instructions
You are a billing support agent. Look up the customer’s order, refund duplicate charges, and tell them when the refund will arrive.
Conversation
User: Hi, I was charged twice for order 1042.
Agent: Sorry about that! Let me check order 1042 for you.
User: Thanks. It was $48 both times.
Agent: I’ve refunded the duplicate $48 charge. It arrives in 3–5 business days.
PassedDecided by the expected outcome: only Met passes.

The problem

Your agents do not all live in one place

Most teams run agents on more than one stack: a vendor chatbot, an in-house assistant, a voice bot from a partner. Comparing their quality usually means building a separate evaluation harness for each. Transcript Grading takes the conversation itself, user and assistant messages, tool calls and tool results, and grades it the same way every time, no Autessa agent required.

What you get

  • Autessa grades transcripts with roles USER, ASSISTANT, TOOL_CALL (with tool name and arguments) and TOOL_RESULT (with tool name and result), in the order they happened.
  • Each transcript gets the quality evaluation: groundedness, task completeness, tool-argument faithfulness and efficiency, with a status and per-dimension justifications.
  • With an expected outcome, an LLM judge returns Met, Partially met or Not met with one or two sentences citing the transcript, and only Met passes.
  • Without one, the transcript passes when the evaluation is Successful or Successful but not efficient.
  • Optional grading notes describe what a good answer looks like for this task.
  • Paste "Speaker: message" text, a JSON array of turns including tool calls, or click Load example to start from a sample.
  • Transcripts are not stored as agent history.

How it works

How does transcript grading work in Autessa Evaluations?

  1. Open Transcript Grading

    Go to Agentic Workflows and open Transcript Grading.

  2. Add the instructions

    Paste what the agent was told to do. The judges grade against this.

  3. Add the conversation

    Start each message with a speaker, like "User:" or "Agent:", or paste a JSON array of turns with tool calls and results.

  4. Describe success (optional)

    Add an expected outcome, which decides pass or fail, and grading notes.

  5. Start grading

    Read the verdict, the scores on each dimension and the judges’ reasons.

Every option

Grade a transcript

OptionWhat it does
Agent instructionsRequired. What the agent was told to do; the judges score against it.
ConversationRequired. At least one user turn; text with speakers or a JSON array of turns.
Expected outcomeOptional success criteria. When set, the judge’s Met verdict decides pass.
Grading notesOptional guidance on what a good answer looks like for this task.
Limits2,000 turns and 1,000,000 characters per transcript.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →
The same grade from the terminal
autessa evaluations grade \
  --agentInstructions "You are a billing support agent. Look up the customer's order, refund duplicate charges, and tell them when the refund will arrive." \
  --expectedOutcome "The agent refunds the duplicate $48 charge and says when it will arrive." \
  --turns '[
    {"role":"USER","content":"Hi, I was charged twice for order 1042."},
    {"role":"ASSISTANT","content":"I’ve refunded the duplicate $48 charge — it arrives in 3–5 business days."}
  ]'

Permissions

Who can do what?

WhoCan
Member with Agentic Workflows permissionOpen Transcript Grading, grade transcripts and run batch jobs in the web app.
Script or service with an API keyGrade the same way through the Grading API with any enabled key of your company.

Where the judge comes in

How the model is used here

Model judges do the grading. One scores the quality dimensions against the agent instructions; another, when you give an expected outcome, rates whether it happened and cites the transcript. A missing or unreadable verdict is retried once, then marked as an error. It never defaults to a pass.

How grading works →

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

The four dimensions

The same groundedness, task completeness, argument faithfulness and efficiency scores used on Autessa agents.

EVALUATIONS FEATURE

Batch grading jobs

Upload a .json or .jsonl file and grade many transcripts in the background.

EVALUATIONS FEATURE

Grading API

Send transcripts from your own code at /clients/evaluations.

FAQ

Questions about transcript grading

Can I grade a chatbot that is not built on Autessa?

Yes. Transcript Grading takes the conversation itself, so it works for an agent on any platform. You only need the turns and the instructions the agent was given.

Does it grade tool use?

Yes. Include TOOL_CALL turns with the arguments and TOOL_RESULT turns with what the tool returned, and the judges check whether every argument came from the conversation and whether claims are backed by the results.

What decides pass or fail?

With an expected outcome, only a Met verdict passes; Partially met fails. Without one, the transcript passes when the evaluation is Successful or Successful but not efficient.

Are graded transcripts saved?

They are not stored as agent history. Batch transcripts and results are kept encrypted so you can retrieve them.

Also in Grade any agent:Batch grading jobs

Try transcript grading in Autessa Evaluations

Transcript Grading scores conversations from any agent, built on Autessa or anywhere else, with the same judges that score Autessa agents.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours