A practical LLM evaluation loop for AI features that need to ship

Prompt testing is not an evaluation strategy

An AI feature can look excellent in a demo and still fail the first week of real use. A user asks a question in a different way, a retrieved document is incomplete, a tool returns an unexpected shape, or a model update changes the tone of the answer. The team then opens the prompt, changes a sentence, and tests three examples by hand.

That process feels productive because the output changes immediately. It is not a reliable way to know whether the product improved.

A production AI feature needs an evaluation loop. The loop should make a change observable, repeatable, and reversible. You need a set of representative inputs, a definition of acceptable behavior, automated checks for obvious failures, and a way to learn from what users do after the answer appears.

The useful question is not “does this prompt work?” It is “does this version perform better on the jobs our users actually need done?”

Start with a small, owned dataset

Do not wait for a perfect benchmark. Start with the examples your product already creates.

Collect successful requests, failed requests, user corrections, support tickets, abandoned flows, and inputs that required a human handoff. Remove personal information and secrets. Then label each example with the behavior that matters. A support answer might need to cite the correct policy, refuse an unsupported request, and avoid inventing a deadline. A code-generation feature might need to preserve an API contract, produce valid syntax, and avoid changing a protected file.

Fifty carefully chosen examples can reveal more than a thousand random prompts. Divide them into a development set and a holdout set. Use the development set while improving the feature. Keep the holdout set hidden from day-to-day prompt editing so it can tell you whether the change generalizes.

Version the dataset. Add a short reason whenever an example changes. If a case was added because a user found a serious failure, keep that context. Your evaluation set is not disposable test data; it is the product’s growing memory of what “good” means.

Define behavior before choosing a grader

A single score such as “quality: 8.4” is not enough. Define the dimensions that make an output useful or dangerous.

For an AI assistant that answers product questions, useful dimensions might be:

• Factual accuracy
• Completeness
• Citation or evidence use
• Appropriate refusal
• Clear next step
• Tone and readability

For an agent that can call tools, add:

• Correct tool selection
• Valid arguments
• Authorization compliance
• Side-effect safety
• Recovery after a failed tool call

Write a short rubric for each dimension. A good rubric tells the grader what a passing answer must contain and what counts as a failure. It should also identify severe failures that cannot be averaged away. Leaking another tenant’s data is not merely a low score; it is a release blocker.

This is where many evaluation projects go wrong. The team asks a grader whether an answer is “good” and then trusts the number. A useful grader is closer to a reviewer with a checklist: it can explain which requirement was missed and assign a result that maps to a product decision.

Combine deterministic and judgment-based checks

Use the cheapest reliable check first.

Deterministic checks are ideal for structure and safety. Validate JSON against a schema. Confirm required fields exist. Reject unknown tool names. Check that a citation field is present when the workflow requires evidence. Enforce maximum length, latency, token, and tool-call budgets. Verify that an action requiring confirmation has not executed without confirmation.

These checks should run on every change because they are fast and predictable.

Reference-based checks compare the result with known facts or required entities. Exact string matching is often too brittle, especially when several answers can be correct. Instead, check whether required facts appear, forbidden claims are absent, and important names, dates, or identifiers are correct.

Judgment-based checks are useful for qualities such as helpfulness, completeness, and tone. They can be performed by a human reviewer or a separate model with a strict rubric. Keep the evaluator separate from the system being tested where possible. Record the rubric version and evaluator model alongside the result.

Never let a judgment score replace a safety check. A fluent answer can still contain a forbidden claim, expose private data, or trigger an unauthorized action.

Evaluate the whole workflow

The model response is only one part of the user experience.

Suppose a retrieval assistant gives a wrong answer. The root cause might be the model, but it might also be that the correct document was never retrieved, the tenant filter was missing, the context was truncated, or an old result was displayed in the interface. If you evaluate only the final paragraph, you cannot tell which subsystem needs attention.

Capture evaluation data for the workflow stages that matter:

1. The user input and normalized task
2. Retrieval queries, filters, and result identifiers
3. Context assembled for the model
4. Model output and structured fields
5. Tool calls and authorization decisions
6. Validation and retry results
7. The final response shown to the user

Keep sensitive content out of logs unless your data policy explicitly allows it. You can often retain redacted samples, hashes, identifiers, and structured outcomes instead of every raw prompt and document.

For asynchronous workflows, evaluate recovery too. A job that succeeds on the first attempt but duplicates an email after a retry is not reliable. Include timeouts, provider errors, cancellation, partial completion, and stale status in the test set.

Turn every production failure into a regression case

The most valuable evaluation case is often the one that just failed in production.

When a user corrects an answer, save a sanitized version of the input, the bad behavior, the expected behavior, and the condition that caused the failure. Add it to the development set immediately. Move it to the holdout set after the fix has been tested. That prevents the team from celebrating a local improvement while silently overfitting to the same examples.

Track failure categories, not only pass rates. You want to know whether prompt injection attempts are increasing, whether retrieval failures cluster around one document type, whether one customer segment sees more refusals, or whether a model change increases tool-call errors.

Slice results by language, workflow version, customer tier, input length, document type, and task difficulty. Aggregate numbers hide important regressions. A new version can improve the average while breaking the exact high-value workflow your best customers use.

Make the evaluation a release gate

An evaluation is useful when it changes what you do.

Set a small release policy. For example, a change may ship only when deterministic checks pass, no critical safety case regresses, the holdout score does not fall below its threshold, and the latency or cost budget remains acceptable. The exact thresholds should reflect the product, but the decision must be explicit before the result arrives.

Store each run with the prompt version, model identifier, retrieval configuration, tool definitions, dataset version, grader version, and timestamp. Without this metadata, a result cannot explain what changed.

A simple runner can make the contract visible:

type EvalCase = {
  id: string;
  input: string;
  requiredClaims?: string[];
  forbiddenClaims?: string[];
  expectedTools?: string[];
};

type EvalResult = {
  id: string;
  passed: boolean;
  failures: string[];
  latencyMs: number;
};

async function evaluateCase(testCase: EvalCase): Promise<EvalResult> {
  const started = Date.now();
  const output = await runFeature(testCase.input);
  const failures = checkOutput(output, testCase);

  return {
    id: testCase.id,
    passed: failures.length === 0,
    failures,
    latencyMs: Date.now() - started,
  };
}

The implementation will evolve, but the principle stays stable: every result should say what was tested, what failed, and whether the change is safe to release.

Use real user outcomes as the final signal

Offline evaluations are necessary, but they are not the finish line. Watch what users do after the response.

Do they accept it, edit it, retry, abandon the workflow, open a support request, or ask for a human? For generated code, does the patch pass tests and survive review? For a support answer, does it resolve the case without a follow-up correction?

These signals are not perfect labels. A user may accept a bad answer or reject a good one for reasons unrelated to quality. Still, they reveal cases your curated dataset missed. Sample them responsibly, remove sensitive data, and feed the important failures back into the evaluation set.

That creates a durable loop: collect representative cases, define acceptable behavior, run cheap checks, review meaningful judgments, ship only against explicit thresholds, observe real outcomes, and turn failures into new tests.

The result is a better way to build with AI. You can change the prompt, model, retrieval strategy, or tool workflow without relying on instinct alone. You can explain why a version shipped. Most importantly, you can catch a regression before a user has to become your evaluator.

OTF is built for teams that want to move from an AI-shaped prototype to a product with clear conventions and a real release path. Keep the evaluation loop close to the code, make the pass criteria visible, and let every production lesson improve the next build.

Sources

OpenAI Evals — framework and registry for evaluating LLMs and LLM systems: https://github.com/openai/evals