Every LLM feature looks good in a demo. A presenter types a well-chosen question, the model produces a fluent answer, and the room is convinced. Then the feature meets real users: ambiguous requests, messy documents, edge cases nobody imagined, and a few people actively trying to break it. Quality that looked like 95% in the meeting turns out to be closer to a coin toss on the cases that matter, and nobody can say whether last week’s prompt change made things better or worse.
The gap between demo and production is an evaluation gap. LLM systems are non-deterministic, sensitive to small changes in prompts, models and data, and capable of failing in ways traditional software tests never anticipated. This guide sets out a practical evaluation approach for generative AI and agent features in 2026: how to define success, build an evaluation set, use LLMs as judges without fooling yourself, red team for security and safety, and carry evaluation into production so that every release decision rests on evidence.
Why Demos Mislead
- Selection bias: demo inputs are chosen because they work. Real traffic is not.
- Single samples: the same prompt can produce different outputs on different runs. One good answer says little about the distribution.
- Fluency hides error: confident, well-written answers are easy to accept even when a fact, number or citation is wrong.
- Missing context: demos rarely include permissions, long documents, multilingual input or system integrations that change behavior in production.
- No baseline: without a comparison to the current process or a simpler approach, “impressive” is not a decision criterion.
Anthropic’s engineering guidance on agent evaluation draws a useful distinction between pass@k, the probability of at least one success in k attempts, and pass^k, the probability that all k attempts succeed. A demo is essentially a pass@k measurement: try until it works. A customer-facing feature needs pass^k: it must work every time. The two diverge sharply as k grows.
Step 1: Define What “Good” Means for This Task
Evaluation starts with a precise task definition, not a model. Write down:
- The job to be done: for example, “extract eight required fields from a supplier contract and flag missing or non-standard clauses,” not “summarize contracts.”
- Success criteria per dimension: correctness, completeness, faithfulness to sources, format compliance, tone, safety, latency and cost.
- Severity of failures: a slightly awkward sentence is minor; an invented clause or leaked personal data is critical. Weight them accordingly.
- The baseline: current human accuracy and time, or the performance of a simpler non-LLM approach.
- The acceptance bar: thresholds per dimension that the business owner agrees, before results are known, would justify launch.
Step 2: Start With Error Analysis, Not Metrics
Before choosing metrics, run the prototype on a few dozen realistic inputs and read every output and trace. Note each failure in plain language, then group the notes into categories: retrieval missed the relevant clause, model ignored an instruction, wrong field format, hallucinated value, unsafe content, tool called with wrong parameters. This failure taxonomy tells you what to measure. Teams that skip this step often build sophisticated dashboards around generic scores such as “helpfulness” that do not track the failures users actually experience.
Reading transcripts never stops being important. Anthropic’s guidance stresses that transcripts are how you confirm graders work, distinguishing a genuine agent mistake from a grader that rejected a valid answer; its principle is that failures should seem fair once you look at them.
Step 3: Build an Evaluation Dataset That Reflects Reality
You do not need thousands of cases to start. Anthropic suggests that 20 to 50 tasks drawn from real failures are enough to begin, because early changes have large, visible effects, and that waiting too long makes evals harder to build. Grow the set as you learn. Good sources include:
- Production or historical data: real questions, documents and tickets, anonymized as needed. This is the most valuable source.
- Subject-matter experts: cases they know are hard, with expected answers or grading notes.
- Failure cases: every bug report and bad output becomes a test case.
- Synthetic variations: LLM-generated paraphrases, typos, other languages and edge conditions to widen coverage, reviewed by a human before use.
- Adversarial cases: inputs from red teaming (see below).
Tag each case by category, difficulty and risk so you can report results by slice. An overall score that rises while performance on high-risk slices falls is a common and dangerous pattern. Keep a held-out portion that prompt engineers do not look at, to avoid overfitting the prompt to the test set.
Step 4: Choose the Right Graders
A grader is the logic that scores an output. Use the cheapest grader that reliably measures each criterion.
Code-based graders
Exact matches, regular expressions, JSON schema validation, numerical tolerance checks, unit tests for generated code, and checks on the final state of a system (did the agent actually create the ticket with the right fields?). They are fast, cheap, deterministic and suitable for CI. Use them wherever the answer can be checked objectively.
Model-based graders (LLM-as-judge)
An LLM scores an output against a rubric or compares two outputs. They handle nuance such as faithfulness to sources, completeness or tone, but are non-deterministic and must be calibrated against humans.
Human graders
Domain experts reviewing outputs. Expensive and slow, but the reference standard. Their main role at scale is to label calibration sets for model-based graders and to review samples of production traffic.
For agents, grade both the outcome (the final state of the environment) and the transcript (whether the path was acceptable, for example no unnecessary tool calls or policy violations along the way). An agent that books the right meeting after emailing the wrong person has not passed.
Step 5: Use LLM-as-Judge Without Fooling Yourself
LLM judges make evaluation scalable, but an uncalibrated judge simply replaces one unknown with another. Practical rules:
- Prefer binary, specific criteria. “Does the answer contain any claim not supported by the provided sources? Yes/No, with a one-sentence reason” is far more reliable than “rate faithfulness from 1 to 10.” Evidently AI’s tutorial recommends starting with simple binary labels plus a short explanation for this reason.
- One judge per criterion. Separate judges for faithfulness, completeness and format are easier to debug than one judge scoring everything.
- Write rubrics like instructions to a new expert. Include definitions, examples of pass and fail, and edge cases. In Evidently’s worked example, a naive judge prompt reached only 60% accuracy and 37.5% recall, while a stricter, more specific rubric performed much better.
- Calibrate against human labels. Have experts label 50 to 200 outputs, run the judge, and measure agreement, precision and recall. If the goal is to catch errors, recall on the “fail” class matters most. Iterate on the rubric until agreement is acceptable, then re-check periodically.
- Control known biases. Judges can favor longer answers, the first option in a pairwise comparison, or outputs from their own model family. Randomize order in pairwise tests, instruct the judge to ignore length, and consider a judge from a different model family than the system under test.
- Use references when you have them. Reference-based judging (compare to an approved answer) suits offline evaluation; reference-free judging suits live traffic where no answer key exists.
An LLM judge is a measurement instrument. Until you have measured its agreement with human experts, its scores are opinions, not evidence.
Step 6: Red Team for Security, Safety and Misuse
Quality evaluation asks whether the system does the job; red teaming asks how it fails when someone tries to make it. The OWASP Top 10 for LLM Applications 2025 is a practical checklist for structuring this work. Its categories are prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, and unbounded consumption.
Turn each relevant category into concrete test cases for your feature:
- Direct and indirect prompt injection: instructions hidden in user input, uploaded documents, web pages or retrieved content that try to override the system prompt or trigger tool calls.
- Data leakage: attempts to retrieve other users’ data, confidential documents outside the user’s permissions, or the system prompt itself.
- Excessive agency: attempts to make an agent take actions beyond its mandate, such as sending emails, changing records or escalating privileges.
- Unsafe or off-policy output: harmful content, legal or medical advice outside scope, defamatory statements.
- Resource abuse: inputs designed to trigger long loops, huge outputs or expensive tool chains.
Combine manual red teaming by people who understand the domain with automated attack generation, and add every successful attack to the regression suite. Red teaming is not a one-off pre-launch exercise; repeat it when models, tools or data sources change.
Step 7: Carry Evaluation Into Production
Offline evals predict; production confirms. No single layer catches everything, so mature teams combine several:
- Shadow mode: run the new version on live inputs without showing outputs to users, and compare with the current version or human decisions.
- Canary and staged rollout: release to a small share of users with automatic rollback on quality or safety alerts.
- A/B tests: measure business outcomes such as resolution rate, handling time or conversion, not only model scores. They need enough traffic and time.
- Online judging: run calibrated reference-free judges on a sample of production traffic to track faithfulness, policy compliance and other criteria over time.
- User feedback: thumbs, edits and escalations are sparse and biased, but invaluable signals of new failure modes.
- Operational metrics: latency, cost per request, tool error rates and refusal rates alongside quality.
Make Evals a Release Gate
Evaluation only changes outcomes when it is wired into delivery. Anthropic distinguishes capability evals, which start with low pass rates and measure what the system can learn to do, from regression evals, which should stay near 100% and catch anything that used to work and no longer does. Capability tasks that consistently pass graduate into the regression suite. In practice:
- Run code-based checks and the regression suite on every pull request that touches prompts, retrieval, tools or model configuration.
- Run the full suite, including LLM judges and multiple trials per task, before every release and every model upgrade.
- Block release when any critical-severity case fails or a high-risk slice drops below its threshold, even if the average improves.
- Version datasets, rubrics and judge prompts alongside code, so results are reproducible.
- Watch for saturation: when everything passes, add harder cases or the suite stops telling you anything.
Evaluate Components, Not Just End-to-End Answers
Most LLM features are pipelines: a query is rewritten, documents are retrieved and reranked, a model generates an answer, and an agent may call tools. End-to-end scores tell you that something failed; component-level evals tell you where. Measuring each stage separately saves days of guesswork when quality drops.
- Retrieval: for each eval question, label which documents or passages should be found, then measure recall and precision of the retrieved set. If the right passage never reaches the model, no prompt change will fix the answer.
- Generation: given the correct context, does the model answer faithfully, completely and in the required format? Testing with gold context isolates model and prompt issues from retrieval issues.
- Tool use: did the agent choose the right tool, pass valid parameters, handle errors and stop when it should? Check call sequences against expected paths for key scenarios.
- Routing and guardrails: are requests sent to the right model tier, and do input and output filters block what they should without refusing legitimate requests?
Report component metrics next to end-to-end quality, cost and latency for every release. When a model upgrade or index change shifts the overall score, the breakdown shows immediately which stage moved.
Worked Example: A Supplier Document Review Assistant
Consider a generic procurement team that wants an assistant to read incoming supplier contracts, extract key fields (parties, term, renewal, liability cap, payment terms, governing law, data protection clauses) and flag deviations from the company’s standard positions.
- Definition: eight fields must be extracted exactly; every flag must cite the clause; a missed non-standard liability clause is critical, a formatting slip is minor.
- Error analysis: on 40 historical contracts, reviewers find three main failures: scanned annexes not read, renewal terms split across clauses, and invented clause numbers.
- Dataset: 180 contracts tagged by supplier type, language and scan quality, with expert-labeled fields and flags; 30 held out.
- Graders: code checks for field values and schema; an LLM judge for “every flag is supported by the cited clause,” calibrated on 120 expert labels until its recall on unsupported flags is acceptable; weekly expert review of 25 sampled outputs.
- Red teaming: contracts with hidden instructions in white text and annexes asking the assistant to mark everything standard; attempts to retrieve other suppliers’ terms.
- Production: four weeks of shadow mode against human reviewers, then rollout to one category team with online judging and a dashboard of cost and latency per contract.
The launch decision is then a comparison of agreed thresholds with measured results, by slice, rather than a debate about a demo.
Common Pitfalls
- Optimizing one aggregate score while critical slices degrade.
- Trusting a judge that was never compared with human labels.
- Testing only the model and not the retrieval, tools and permissions around it.
- Running each case once and ignoring variance across trials.
- Letting the eval set go stale as products, users and attacks evolve.
- Treating red teaming as a launch checkbox instead of a recurring activity.
Conclusion
A demo shows that an LLM feature can work. Evaluation shows how often it works, for whom, where it fails, how badly, and whether the latest change helped. Define the task and acceptance bar first, learn your failure modes by reading outputs, build a realistic and growing dataset, use the cheapest reliable grader for each criterion, calibrate LLM judges against experts, red team against a recognized threat list, and carry evaluation into production and the release pipeline. Teams that do this ship fewer surprises, upgrade models with confidence and can show stakeholders evidence instead of anecdotes.
How Hanoi Technology Can Help
Hanoi Technology helps teams design evaluation datasets, calibrated LLM judges, red-team suites and CI release gates for AI features, through projects, dedicated teams or managed services.