Scoring and judges · Autessa Evaluations
Evaluation notes (rubric)
Evaluation notes are a grading rubric for one agent: notes that tell the automated judge what a finished task looks like for that agent. They make task completeness reflect your business, not a generic idea of “helpful.”
The problem
Only you know what “done” means
For a ticket-triage agent, done might mean classified and a reply drafted. For a booking agent, it means a confirmed time. A generic judge can’t know that. Evaluation notes tell it.
What you get
- A free-text rubric saved on the agent, in its settings.
- Used by the automated evaluation to judge whether a task was finished.
- Available on an agent once AutoEval is on.
- Also on managers (multi-agent), so orchestrated runs are graded against your rubric.
- Transcript Grading has the same idea: grading notes describe what a good answer looks like for that task.
- Saved with the agent version, like the rest of its settings.
How it works
How does evaluation notes (rubric) work in Autessa Evaluations?
Open the agent’s settings
Find the Evaluation notes section.
Describe a finished task
For example: “A good response classifies the ticket and drafts a reply.”
Save and publish
New evaluations use the notes.
Every option
Where rubrics live
| Option | What it does |
|---|---|
| Agent evaluation notes | Notes that help the automated evaluation judge whether a task was finished. |
| Manager evaluation notes | The same rubric for a multi-agent manager. |
| Grading notes (Transcript Grading) | Optional guidance per transcript: what a good answer looks like for this task. |
Permissions
Who can do what?
| Who | Can |
|---|---|
| Read | View the agent and its versions, test-run it, and read its history and results. |
| Write | Everything Read can, plus edit the draft and record human evaluations. |
| Publish | Publish and roll back versions, which is what customers get. |
| Admin | Share and delete the agent. The owner is always an Admin. |
Where the judge comes in
How the model is used here
The automated judge reads your notes alongside the agent’s instructions when it scores task completeness.
How grading works →Connected
How it connects to the rest of Evaluations
FAQ
Questions about evaluation notes (rubric)
What should I put in evaluation notes?
What a finished task looks like for this agent, in a sentence or two. For example, “A good conversation resolves the request end to end.”
Are evaluation notes the same as an expected outcome?
No. Evaluation notes are a standing rubric for every conversation with the agent; an expected outcome is the success criteria for one specific test case or transcript.
Can I set them from the API?
Yes. Agents and managers have an `evaluationNotes` field, the grading rubric automated evaluation uses.
100% customizable, code and all
Make Autessa Evaluations fully your own.
Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.
Start free and make it yours