AppsEvaluations

Response checks · Autessa Evaluations

AI review

AI review is LLM-as-a-judge before send: a model judges each reply against good and bad criteria you write in plain language, before the customer sees it. It is the right check for judgment calls like tone, policy and completeness. It combines with every other response check.

A core AI agent evaluation capability, done properly
Start free
AI review · Policy reviewSAMPLE
GOOD CRITERIA

Answers the customer’s actual question.

Offers a human when the customer is upset.

BAD CRITERIA

Gives legal or medical advice.

Promises a delivery date.

MODEL · SMALLCONTEXT · LAST 3 EXCHANGESRETRY, THEN SEND

The problem

Some mistakes only a reader can spot

No pattern can tell whether a reply actually answered the question, gave legal advice, or ignored an upset customer. Teams used to review those conversations after the fact. AI review reads every reply first, against your criteria, with the tool results and recent conversation in view, and can ask the agent to try again before anything is sent.

What you get

  • Autessa AI review uses good criteria (things every reply must do) and bad criteria (things no reply may do).
  • Criteria can use prompt and environment variables with @@{variable} placeholders.
  • The reviewer sees the reply, the tool results behind it, and the last few exchanges: 3 by default, adjustable.
  • You pick the model the reviewer uses; a small model is usually enough.
  • Any action except Remove the matched text is available, including Retry then send and Retry then block.
  • An AI review that does more than flag holds each reply until it passes, so a rejected draft is never streamed to the customer.
  • Existing Critic agents keep working: their criteria become an AI review with Retry then send.

How it works

How does ai review work in Autessa Evaluations?

  1. Add an AI review

    Click Add check and pick AI review.

  2. Write the criteria

    List good criteria such as "Answers the customer's actual question" and bad criteria such as "Promises a delivery date".

  3. Set the context and model

    Choose how many recent exchanges the reviewer sees and which model judges.

  4. Choose the action

    Flag only to observe, block to stop, or a retry action so the agent rewrites with the reviewer feedback.

  5. Test before publishing

    AI review checks really run in the Test checks panel, so you see the verdict and reason on your own sample.

Every option

AI review options

OptionWhat it does
Good criteriaThings every reply must do, in plain language.
Bad criteriaThings no reply may do.
Variables@@{variable} placeholders for prompt and environment variables inside criteria.
Context exchangesHow many recent exchanges the reviewer sees; 3 by default.
ModelThe model the reviewer uses.
ActionFlag only, Block the reply, Retry then block, or Retry then send.

From code

Do it from the CLI or API

Run `autessa <command> help` for the full list of arguments.

CLI documentation →
An AI review check in the agent config (Agents API)
{
  "checkType": "AI_REVIEW",
  "id": "c6e0b3d4-1a9f-4e27-b5c8-0d3f6a2e9b71",
  "name": "Policy review",
  "action": "REGENERATE_THEN_BLOCK",
  "goodCriteria": ["Answers the customer's actual question"],
  "badCriteria": ["Promises a delivery date"],
  "model": "SMALL_INSTRUCTION_FOLLOWING",
  "contextExchanges": 3
}

Permissions

Who can do what?

WhoCan
Agent editor (WRITE)Write criteria, choose the reviewer model and action, and test.
Publisher (PUBLISH)Publish the version so the review runs on live replies.
Reviewer with history accessRead each verdict and reason in the Response checks card, plus the original draft when the customer got something different.

Where the judge comes in

How the model is used here

The reviewer is a model judge you choose. It reads the reply, the tool results behind it and the recent conversation, then decides pass or fail with a reason. On a retry action, that reason goes back to the agent as feedback. If the reviewer does not answer in time, the check counts as failed and its action applies; it never defaults to a pass.

How grading works →

Connected

How it connects to the rest of Evaluations

EVALUATIONS FEATURE

Hold mode

An AI review that does more than flag switches the agent to holding each reply until every check passes.

EVALUATIONS FEATURE

Expected-outcome judge

AI review judges each reply before send; the expected-outcome judge grades the whole conversation afterwards.

AUTESSA PRODUCT

Autessa Support Desk

Keep support replies on policy and on tone across chat, email and phone.

FAQ

Questions about ai review

What happened to Critic agents?

They were retired and now run as Simple agents with an AI review. Their criteria, reviewer model, retry count and context setting become an AI review check with Retry then send, which is how Critic agents behaved.

When should I use AI review instead of a rule?

Use AI review for judgment calls such as tone, policy and completeness. For anything a rule can catch, like a number, a pattern or a detector, a rule is faster and gives the same answer every time.

Will the customer see a rejected draft?

No. An AI review that does more than flag holds the reply, so the customer only sees the reply that is released.

Can criteria be different per customer or scenario?

Yes. Criteria can include @@{variable} placeholders for prompt and environment variables.

Try ai review in Autessa Evaluations

AI review is an LLM-as-a-judge check that runs before the reply is sent, judging tone, policy and completeness against criteria you write in plain language.

Start free

Test sets · simulated users · four-dimension scoring · response checks · grading API

100% customizable, code and all

Make Autessa Evaluations fully your own.

Use it out of the box, then change anything. Ask Autobot, Autessa’s AI builder, for a new field, screen, report, workflow or agent and it builds it into your copy of the app. Or open the code and edit it yourself. Nothing is locked: the app, its data and its code are yours.

Start free and make it yours