AppsEvaluations

Production monitoring · Autessa Evaluations

Human evaluation and labeling

Human evaluation lets reviewers label any agent conversation with a status and notes. A human evaluation always takes precedence over the automated one, in history and in statistics, so your experts have the final word and your numbers reflect it.

A core AI agent evaluation capability, done properly
Start free
Evaluation Details · Human evaluationSAMPLE
AutomatedSUCCESSFUL
Human evaluationSUCCESSFUL (NOT ALL STEPS)

“Refund was right, but it never confirmed the customer’s email for the receipt.”

Evaluated by reviewer · takes precedence

The problem

Automated judges need a human check

An automated judge is fast, but domain experts catch things it can’t. Without a place to record their verdicts, that knowledge stays in a spreadsheet and never moves the numbers. Here a reviewer’s label is part of the record.

What you get

  • Statuses: Successful, Successful (Not Efficient), Successful (Not All Steps) and Unsuccessful.
  • Free-form notes on every evaluation.
  • The reviewer and time are recorded automatically from who is signed in.
  • Human verdicts override automated ones in every count.
  • Statistics report human, automated and a merged human-first view side by side.
  • The same panel exists in Multi-Agent History for manager runs.

How it works

How does human evaluation and labeling work in Autessa Evaluations?

  1. Open a conversation

    In Agent History, choose View Details.

  2. Show Evaluation

    Open the Evaluation Details panel beside the transcript.

  3. Add Human Evaluation

    Pick a status, write notes and save. Edit it later if needed.

Every option

Human evaluation fields

OptionWhat it does
StatusSuccessful, Successful (Not Efficient), Successful (Not All Steps), Unsuccessful.
NotesFree-form reviewer notes.
Evaluated by / atSet automatically from the signed-in reviewer.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →
Record a human evaluation from the CLI
autessa agents submit-evaluation \
  --agentId <agentId> \
  --conversationId <conversationId> \
  --humanEvaluation '{"status":"SUCCESSFUL","notes":"Correct summary, one tool call."}'

Permissions

Who can do what?

WhoCan
ReadView the agent and its versions, test-run it, and read its history and results.
WriteEverything Read can, plus edit the draft and record human evaluations.
PublishPublish and roll back versions, which is what customers get.
AdminShare and delete the agent. The owner is always an Admin.

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Human vs automated

Compare the outcomes reviewers recorded with the automated judge’s, side by side.

EVALUATIONS FEATURE

Conversation logs

Full transcripts and tool cards make labeling quick.

EVALUATIONS FEATURE

Statistics API

Pull human, automated and human-first counts by version.

FAQ

Questions about human evaluation and labeling

Who can record a human evaluation?

Anyone with Write access to the agent, as well as its Admins and owner.

What wins if a person and the judge disagree?

The person. A human evaluation always takes precedence over the automated one.

Can I label conversations in bulk from code?

Yes. Call `autessa agents submit-evaluation` (or the submit-evaluation API) once per conversation.

Try human evaluation and labeling in Autessa Evaluations

Human evaluation lets reviewers label any conversation with a status and notes, and a human verdict always overrides the automated one.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours