§6 · Autessa Evaluations

Developers

Everything in Evaluations has an API and a CLI command, so grading and test runs fit into scripts and release pipelines.

Start free

autessa CLI

autessa evaluations grade        # grade one transcript now
autessa evaluations jobs-create  # up to 500 in the background
autessa evaluations jobs-get     # poll progress and pass rate
autessa evaluations jobs-items   # page through results
autessa evaluations items-get    # one result, by itemId or externalId
autessa evaluations jobs-list    # your company's jobs

autessa sandbox test-sets-create
autessa sandbox policies-create
autessa sandbox runs-start
autessa sandbox runs-get

The area

What is developers in Autessa Evaluations?

Everything in Evaluations runs from code. The Grading API and the `autessa evaluations` CLI grade transcripts; the `autessa sandbox` CLI creates test sets, simulated users and runs; and the agents API records human evaluations and returns statistics by version.

Examples on these pages are taken from the CLI help and public docs.

In this area

Subtopics

Everything in developers

Run evaluations from code: the Grading API and `autessa evaluations` CLI, the `autessa sandbox` CLI for test sets and runs, and APIs for human evaluations and statistics.

Grading API and CLI

The Grading API and `autessa evaluations` CLI grade one transcript in a single call or a batch in the background, from any language or pipeline.

Sandbox CLI

The `autessa sandbox` CLI creates test sets, simulated users and runs from code, so an agent test can gate a release.

Human evaluation and statistics API

The agents API records human evaluations and returns human, automated and human-first statistics with a per-version breakdown.

Other areas

Agent testing →

Autessa Evaluations tests an AI agent before launch by running test sets as real conversations, driven by simulated users, and grading every one.

Scoring and judges →

Every evaluated conversation is scored on four quality dimensions and, when you describe what should happen, judged against that expected outcome.

Production monitoring →

In production, every conversation is logged in full and can be scored automatically, labeled by people and compared version by version.

Response checks →

Response checks evaluate each reply before it is released, so a made-up price or a leaked card number never reaches the customer.

Grade any agent →

Transcript Grading applies the same judges to conversations from any agent, wherever it runs, with no Autessa agent required.

Start with developers

Run evaluations from code: the Grading API and `autessa evaluations` CLI, the `autessa sandbox` CLI for test sets and runs, and APIs for human evaluations and statistics.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours