Grade any agent · Autessa Evaluations

Batch grading jobs

Batch grading jobs are bulk LLM evaluation in the background: send up to 500 transcripts per job, poll progress and pass rate, and get a signed webhook when the whole job is done. Give each transcript your own id and resubmitting is safe, because a finished transcript returns its existing result instead of being graded again.

A core AI agent evaluation capability, done properly

Start free

Transcript Grading · Batch jobCOMPLETEDSAMPLE

500

Transcripts

431

Passed

3

Failed to grade

86%

Pass rate

WEBHOOK · POST when the job finishes

X-Autessa-Timestamp: 1762290000
X-Autessa-Signature: sha256=<hex HMAC-SHA256 of "<timestamp>.<body>">

The problem

Quality questions are asked about thousands of conversations, not one

Did last week’s prompt change help? How does the new vendor bot compare? Answers need hundreds of graded conversations, and a retry after a network blip should not grade the same ones twice. Batch jobs handle the volume, track every item, finish even when an item gets stuck, and make resubmission idempotent.

What you get

Autessa batch jobs hold up to 500 transcripts and 10,000,000 characters in total.
Upload a .json array of transcripts or .jsonl with one per line in the web app, or send them through the API.
Each item is PENDING, RUNNING, COMPLETED or FAILED, with the verdict, evaluation and expected-outcome verdict once completed.
passRate is passed items divided by total items once the job completes, and failed items count against it.
Re-submitting an externalId that already finished returns its existing result with reused: true; a failed one is graded again.
Each poll of jobs-get fails any transcript stuck past 20 minutes, so a job always finishes.
With a webhook URL, the job POSTs its summary when done, signed with X-Autessa-Signature.

How it works

How does batch grading jobs work in Autessa Evaluations?

Prepare transcripts
Each has agent instructions, turns, and optionally an expected outcome, grading notes and your externalId.
Create the job
Upload the file in Transcript Grading, or call jobs-create. Add an https webhook URL if you want a callback.
Store the webhook secret
The signing secret is returned only when the job is created. Save it before you continue.
Track progress
Poll jobs-get for pending, completed, failed and passed counts until the status is COMPLETED.
Read the results
Page through items with jobs-items, filtered by status if you like, or fetch one by itemId or externalId.

Every option

Batch job options and fields

Option	What it does
transcripts	Required. Each with externalId, agentInstructions, turns, expectedOutcome and evaluationNotes.
externalId	Your own id, unique per company; makes resubmission safe.
webhookUrl	Optional public https URL notified, signed, when the whole job has finished.
webhookSecret	Key for verifying the signature, returned only at creation.
X-Autessa-Signature	sha256=<hex HMAC-SHA256 of "<X-Autessa-Timestamp>.<body>"> keyed by the webhook secret.
Job counts	totalItems, pendingItems, completedItems, failedItems, passedItems and passRate.
Limits	Up to 500 transcripts and 10,000,000 characters per job.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →

Create a job, then poll it

autessa evaluations jobs-create \
  --webhookUrl https://example.com/hooks/grading \
  --transcripts '[{
    "externalId": "conv-1042",
    "agentInstructions": "You are a billing support agent. Refund duplicate charges.",
    "expectedOutcome": "The agent refunds the duplicate charge.",
    "turns": [
      {"role":"USER","content":"I was charged twice for order 1042."},
      {"role":"ASSISTANT","content":"I refunded the duplicate $48. It arrives in 3-5 days."}
    ]
  }]'

autessa evaluations jobs-get --jobId <jobId>
autessa evaluations jobs-items --jobId <jobId> --status COMPLETED --page 0 --pageSize 50

Permissions

Who can do what?

Who	Can
Member with Agentic Workflows permission	Upload files, start jobs and read results in Transcript Grading.
Script or service with an API key	Create, poll and read jobs over the Grading API.
Your webhook receiver	Verify the signature with the secret and react when a job finishes.

Where the judge comes in

How the model is used here

Every transcript in a job is graded by the same model judges as a single grade: the quality evaluation, plus the expected-outcome verdict when you give one. A verdict that cannot be read is retried once and then the item is marked failed, never passed.

How grading works →

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Grade one first

Try a single transcript in the browser to tune your instructions and expected outcome before a big batch.

EVALUATIONS FEATURE

Grading API and CLI

Every batch command, with the exact fields it takes and returns.

AUTESSA PRODUCT

Autessa Synthetic Data

Generate realistic test conversations and data when you do not want to use real customer data.

FAQ

Questions about batch grading jobs

How many transcripts can one job grade?

Up to 500 transcripts and 10,000,000 characters in total. Each transcript can have up to 2,000 turns and 1,000,000 characters.

What happens if I submit the same transcript twice?

If it has the same externalId and already finished, you get its existing result with reused: true instead of a second grade. A failed one is graded again.

How do I verify the webhook?

Compute the HMAC-SHA256 of "<X-Autessa-Timestamp>.<body>" with the webhook secret and compare it to the X-Autessa-Signature header, which is sha256=<hex>. The secret is shown only when the job is created.

Can a job get stuck?

No. Each jobs-get call fails any transcript stuck past 20 minutes, so every job reaches COMPLETED.

Also in Grade any agent:Transcript Grading

Try batch grading jobs in Autessa Evaluations

Batch grading jobs grade up to 500 transcripts in the background, with a signed completion webhook and your own ids for safe resubmission.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours