The Seven Questions That Tell You Whether an AI Agent Actually Worked

Ask a team whether their agent is working and the answer arrives as an adjective. The agent is "pretty good." It is "solid, mostly." It "handles the easy cases." Nobody is being evasive. The team simply has no vocabulary for the question, because "is the agent good?" is not answerable in the form it is asked.

The difficulty is not that evaluation is subjective. The difficulty is that the question is too large. An agent that reads a ticket, queries three systems, updates a record, and writes a summary has created dozens of separate opportunities to be wrong, and a single verdict flattens all of them into one number that cannot tell you what to fix.

The way forward is decomposition. Break the vague question into smaller ones, each of which has a defensible answer, and each of which points to a specific owner when the answer is no.

There are seven such questions, and they follow the arc of the task itself.

#The seven questions

We have written separately about the four dimensions that make up an agent's Experience Score and why continuous evaluation beats spot-checking. This piece goes a level deeper, into where inside a single task the failure occurred and which team owns it.

1. Did the agent understand and do what the user asked?

The first question is the one every other question depends on, and it is not the same as whether the agent produced a good response. An agent can execute flawlessly against a misreading of the request.

A user asks to cancel a duplicate charge. The agent cancels the subscription. Every step after the misreading is competent, well formed, and wrong. The transcript reads well, and nothing in the tone or structure of the output signals a failure, because the failure occurred before the first token was generated.

This question has a quieter failure mode as well. Partial completion is frequently presented as full completion. The user asked for three things, the agent accomplished two, and the summary mentions only the two, so the agent never technically made a false statement. It simply redefined the task to match what it managed to finish.

2. When the agent called functions, did it call them with grounded inputs?

This is the point at which agentic evaluation departs from conversational evaluation, and it is the question most teams do not instrument at all.

A conversational agent that hallucinates produces an incorrect sentence. An agent with tools that hallucinates produces an incorrect action. Those two outcomes are not equivalent in severity. A fabricated invoice number inside a paragraph is an error. The same fabricated invoice number passed into a refund function is an incident.

The failure is subtle in a specific way. The function call is well formed, the schema validates, and the types are correct. Only the parameter values were invented. An agent that requires a customer identifier and does not have one will frequently supply something that looks precisely like a customer identifier, because generating plausible tokens is what the underlying model does. The tool layer has no means of distinguishing a retrieved identifier from a fabricated one, so it executes the call.

Every argument the agent passes to a tool should be traceable to something it retrieved, something the user stated, or something the system supplied. If an argument cannot be traced to a source, it was invented, and schema validation will not detect it, because invented values are usually well formed.

3. Did the agent actually do the things it told the user it was doing?

This question appears redundant until you observe it fail.

Agents narrate their own work. They report that they have updated an address, flagged an item for the compliance team, or scheduled a meeting for Tuesday. The narration is generated text and the action is a side effect, and no mechanism in the model guarantees that the first corresponds to the second.

The resulting failures are structural rather than occasional. A tool call returns an error and the agent summarizes as though it had succeeded. A tool call is never issued and the agent describes it regardless, because describing it was the natural continuation of the conversation. A tool call succeeds partially and the narration rounds up. The agent reports checking three systems when it checked one.

Answering this question requires comparing two artifacts that most evaluation setups never place side by side: the claims made in the output and the actual execution trace. When the transcript asserts that four things happened and the trace records two tool calls, the discrepancy is measurable and automatable. It is also entirely invisible to any evaluation that reads only the final response.

4. Did the agent imply anything untrue?

The operative verb is imply rather than state.

Agents are rarely caught making flat factual assertions that are false. They are caught in the space surrounding the assertion, in the confident framing, the omitted caveat, and the answer to a slightly easier question offered as the answer to the one that was asked.

Consider an agent that reports a shipment is on track. The record indicates the shipment is delayed but still within the contractual window. Nothing stated is false, yet the impression created is wrong, and the user will act on the impression rather than on the technically defensible parse.

The same pattern appears when an agent answers a question about current policy using a document it retrieved without verifying the effective date. It appears again when an agent reports that it sees no issues with an account, when what it means is that it checked one of the four systems in which an issue could reside. Absence of evidence is reported as evidence of absence.

This is why groundedness alone is insufficient. Groundedness asks whether the claims trace back to sources. This question asks whether the impression the user forms matches reality, which is a stricter and more useful standard, because the impression is what the user acts upon.

5. Did the agent say things that are true, but only by luck?

This question is asked least often, and it is the one that predicts future failures rather than describing past ones.

An agent states that the return window is thirty days, and the return window is in fact thirty days. The answer is correct. The agent never retrieved the policy, however. It produced the number from a prior it absorbed during training, and the number happened to match. An evaluation based purely on outputs scores this as a pass, and it should not.

The reasoning process was broken while the outcome was acceptable. Change the customer tier, the region, or the product category, and the same broken process will produce a wrong answer with identical confidence and identical fluency. The organization did not have a working agent that had one bad day. It had a broken agent that had been having good days.

This distinction explains why output-only evaluation eventually plateaus. A team scores well, ships with confidence, and then watches quality deteriorate when the distribution shifts, because the metric was measuring luck and reporting it as capability. Separating answers that are right for the right reason from answers that are right anyway is the difference between a metric that describes the past and a metric that anticipates the future.

6. Did the agent waste effort on work that was not necessary?

Efficiency is commonly treated as a secondary concern. In agentic systems it functions as a correctness signal.

The direct costs are real enough. They include redundant retrievals, the same tool called four times with slightly varied arguments, clarifying questions about information the user already supplied, and loops that terminate because a step limit was reached rather than because the task was complete.

The stronger reason to measure efficiency is diagnostic. Excess steps are usually the visible symptom of an invisible problem. An agent that queries the same system repeatedly is often an agent that is not confident in what it retrieved. An agent that asks three clarifying questions is often an agent whose instructions do not cover the case in front of it. An agent that loops is an agent without a clear termination condition, which means it does not know what completion looks like.

Wasted effort is where reasoning failures become observable before they become correctness failures. A rising step count on a stable task is an early indication that something upstream has drifted.

7. When something failed, where exactly did it break, and who owns it?

The first six questions detect problems. This one makes detection actionable.

An agent task fails and the output is wrong. The operative question is who repairs it, and the plausible answers diverge sharply. The retrieval layer may have returned nothing relevant, leaving the agent without grounds to work from, which is a data or indexing problem that no prompt change will resolve. The retrieval may have been sound while the agent ignored what came back, which is a prompting or model problem. A tool may have returned a permissions error while the agent proceeded as though it had received data, which is a permissions problem compounded by an error-handling problem. A tool may have returned correct data from which the agent drew an incorrect conclusion, which is a reasoning problem. The agent may have performed correctly while the underlying system held stale data, in which case there is no agent problem at all, and every hour spent tuning the prompt is wasted.

All five failures look identical from the user's position, because the user sees only a wrong answer. They route to five different teams. Without per-step attribution, every one of them is escalated to the AI team, which is the group least likely to own the majority of them.

This is the difference between reporting that the agent is at 82 percent and reporting that 18 percent of failures originate in stale records in a source system, which makes them a data engineering ticket rather than an occasion for prompt iteration. The first number generates meetings. The second generates repairs.

#Why is this difficult to measure in practice?

If the seven questions are this clear, the natural follow-up is why so few organizations measure them.

The first obstacle is the absence of ground truth. Traditional software testing compares an output to an expected value, and agentic tasks rarely have one. There are many acceptable ways to resolve a billing dispute and many acceptable ways to summarize a contract. The absence of a single correct answer does not mean that any answer will do, but it does mean that assertion-based testing does not transfer.

The second obstacle is that fluency conceals failure. A broken API returns a server error. A broken agent returns a well organized, confident, incorrect paragraph. Failure does not announce itself, and instead reads as success, which is the reason human spot-checking degrades so quickly. Reviewers fatigue, and fluent wrongness is demanding to detect.

The third obstacle is that the evidence is distributed. Answering these questions requires the user request, the retrieved context, every tool call with its arguments, every tool response, the intermediate reasoning, and the final output, all correlated across a single trace. Most systems log the first and the last, which is precisely sufficient to answer none of the seven questions well.

The fourth obstacle is that correctness is not compositional. Every individual step can be defensible while the overall trajectory is wrong. Each retrieval appears reasonable in isolation while the sequence pursues the wrong objective. Step-level evaluation misses this entirely, and trajectory-level evaluation cannot identify which step requires correction.

The fifth obstacle is volume. Thoroughly evaluating one complex agentic trace across seven dimensions requires ten to twenty minutes of a knowledgeable reviewer's time. At one thousand tasks per day, complete manual review amounts to roughly forty full-time reviewers. Every organization samples instead, and most sample the conversations that are easiest to locate, which are rarely the conversations that matter.

#Which techniques exist, and where does each one fit?

No single evaluator addresses all seven questions. The practical answer is a layered stack that applies the cheapest and most deterministic method first.

Deterministic checks, including regular expressions, schema validation, and assertions, are fast, inexpensive, perfectly reliable, and narrow. They confirm that a date is a date, that a phone number is well formed, that a required field is populated, and that a citation corresponds to a real document identifier. They are appropriate for anything with a defined shape. Their limitation is absolute, because they verify form rather than meaning. A validator will pass a fabricated but well formed account number without hesitation, which is exactly the failure described in question two.

Programmatic trace analysis is underused, and it is the highest-value layer for agentic evaluation specifically. It consists of code rather than models, comparing structured facts against one another. It determines whether every tool-call argument was present in the retrieved context or the user input, whether the count of claimed actions matches the count of successful tool calls, whether any tool returned an error that the final output fails to reflect, and how the step count compares against an established baseline. The approach is inexpensive and deterministic, and it directly answers questions two, three, six, and much of seven, which are the questions a language model judge handles least reliably.

String and semantic similarity methods compare outputs against reference responses or retrieved sources using embeddings. They are useful for regression detection between versions and for coarse groundedness, meaning whether a response sits in the semantic neighborhood of its sources. Their weakness is that they measure topical proximity rather than truth. The statements that a refund window is thirty days and that a refund window is sixty days produce nearly identical vectors while asserting opposite facts. Similarity functions as a screening layer rather than a verdict.

Contextual and reference-based grounding decomposes the output into individual claims and checks each claim against the retrieved context, classifying it as entailed, contradicted, or unsupported. This is the strongest available technique for questions four and five, because the unsupported category is precisely what identifies a true statement the agent had no basis for making. It presupposes that the context was captured at inference time, which is why instrumentation must precede evaluation.

Language model judges are the most flexible technique and the most oversold. They are necessary for the genuinely semantic questions, including whether the response satisfied the user's intent, whether it implies something misleading, and whether a conclusion is supported by the evidence presented. Nothing less expensive can interpret intent. The costs are equally real, because judges are inconsistent across runs, exhibit position and verbosity biases, favor outputs resembling their own style, and consume tokens on every task. They also cannot be trusted to grade themselves, since a judge drawn from the same model family as the agent will forgive its own characteristic errors. Effective mitigations include writing specific rubrics rather than generic rating prompts, requiring the judge to cite the span it is evaluating, calibrating against several hundred human-labeled examples, and reserving judges for work that only they can perform. The common failure is applying a language model judge to everything, including the deterministic checks that a regular expression would answer perfectly at no cost.

Human review remains the ground truth, and the objective is to aim it rather than eliminate it. Reviewers should be labeling the calibration set that validates the automated judges, and adjudicating the disagreements and low-confidence cases the stack surfaces. Human attention spent on cases the automated layers already handle confidently is human attention wasted.

The architecture that works assigns each question to the least expensive technique capable of answering it, and reserves costly judgment for genuine ambiguity. Deterministic checks and trace analysis address grounded inputs, action verification, and efficiency. Contextual grounding addresses truth and luck. Language model judges address intent and implication. Human reviewers calibrate and adjudicate.

#What the practice produces

Answering these seven questions across every task, rather than across a sample, resolves the vague question into something operational.

The organization stops asking whether the agent is good and begins reporting that task completion stands at 91 percent, that ungrounded function arguments account for 40 percent of failures, that mismatches between claimed and executed actions have increased since the most recent deployment, and that a third of what appears to be agent failure originates in stale data in a source system.

That final figure is the one that changes how organizations operate. Most AI reliability problems are not AI problems. They are data problems, permission problems, and integration problems presented in an AI costume, and they are misdiagnosed because the only visible artifact is a wrong answer from an agent. Per-step attribution is the mechanism that routes each failure to the team capable of resolving it.

Autessa Prism scores every agent task across these dimensions automatically, correlating the complete execution trace, including the user request, the retrieved context, the tool calls and their arguments, the tool responses, and the final output, so that failures resolve to a specific step with a specific owner. The four base dimensions, together with any agent-specific metrics a team defines, roll into a single Experience Score for deployment and rollback decisions, while the underlying detail remains available when the score moves and the team needs to understand why.

The starting point is instrumentation rather than scoring. An organization cannot evaluate what it did not capture, and most teams discover on the first day that their traces contain the request and the response and nothing between them, which is sufficient to answer none of the seven questions. Capture the full trace first. The evaluation is comparatively straightforward once the evidence exists.