An LLM response can sound convincing and still fail the task. It might ignore a formatting rule, leave out an important detail, or make a claim that the supplied evidence does not support. An LLM analyze API is most useful when it turns those concerns into explicit, testable criteria. A polished paragraph and a successful application outcome are not necessarily the same thing.
This guide describes a practical evaluation workflow for a defined application. The aim is not to rank every model in the abstract, but to understand whether a particular configuration performs the job you need. The LLM analysis toolkit provides a sample report, and the prompt regression guide covers how to compare changes over time.
Define success through the user's task
Write a concrete objective. A support assistant might need to identify the relevant policy, answer the question, and avoid claiming that an action was completed when it was not. A document extractor might need to return the correct fields in a valid structure. Those applications require different evaluations, even when they use the same underlying model.
Describe unacceptable outcomes alongside successful ones. This helps expose tradeoffs that a general satisfaction score can hide. An answer that is concise but omits a required condition may be worse than a slightly longer answer. Decide those priorities before running the evaluation so the team is not adjusting the definition of success to favor whichever output it happens to prefer.
Build a dataset that represents real use
Collect examples from the intended workflow, subject to appropriate permissions and privacy controls. Include routine cases, incomplete inputs, and situations where the right behavior is to ask for more information or decline to make an unsupported claim. Avoid constructing a dataset entirely from easy examples that make the system look good without challenging its important boundaries.
OpenAI's evaluation best practices recommend task-specific evaluation, representative examples, human judgment, and continuing evaluation as systems change. Use that as a starting principle, then define the actual dataset for your application. A useful test collection reflects your users and consequences rather than borrowing an unrelated benchmark as proof of readiness.
Write criteria that reviewers can apply
Separate correctness, evidence support, instruction following, and presentation. Give each criterion a short definition and examples of passing and failing behavior. “Good answer” is too vague for a reliable rubric. “Every factual claim about the supplied policy is supported by the provided passage” is a much clearer instruction for a reviewer.
Avoid overlapping criteria that count the same problem several times. If an omitted detail makes the response incomplete, decide whether it also affects correctness or belongs only under completeness. Consistency matters more than finding the perfect label. Record the decision so future reviewers can apply the rubric without having to reconstruct the discussion that produced it.
Match the evaluator to the criterion
Use deterministic checks for conditions that can be tested directly. A JSON parser can check whether a response is parseable. A schema validator can check required fields and types. Those checks do not determine whether the values are true, so pair them with evidence-aware review where the task requires it.
Human reviewers can inspect nuanced failures and ambiguous cases. A model-based judge may help triage a larger collection, but it should also be evaluated against reviewed examples. Keep its instructions and configuration versioned. Do not treat a judge's numeric output as a calibrated probability merely because the number includes decimal places or appears in a structured response.
Keep evaluator errors visible
An evaluator can fail too. It may receive incomplete context, misunderstand the rubric, or produce invalid output. Record that as an evaluation error rather than marking the application response as correct or incorrect. Otherwise, problems in the measuring system will quietly contaminate the results you use to make product decisions.
Preserve the evidence behind each judgment
A useful evaluation record contains the input, application output, rubric version, evaluator configuration, and rationale. For evidence-based criteria, include the supporting or conflicting passage. This lets a reviewer understand the judgment without replaying the entire application or accepting an unexplained score.
Keep raw output and the reviewed interpretation separate. A person may later correct an evaluator's mistake, and that correction should not overwrite the original application response. For sensitive datasets, consider whether the evidence can be represented through approved excerpts or identifiers rather than copying entire documents into every report. Traceability and data minimization can be designed together.
Compare changes using paired cases
When testing a new prompt or model configuration, run it against the same examples as the baseline where practical. Inspect which cases improved, regressed, or stayed unchanged. The overall average is useful context, but it should not hide a failure in a particularly important category.
Record the complete configuration needed to interpret the comparison: model identifier, prompt version, relevant generation settings, tool behavior, and dataset version. If several components change at once, describe the result as a comparison of configurations rather than attributing the difference to one specific change. Clear records reduce the temptation to tell a stronger causal story than the experiment supports.
Decide how uncertainty affects release
Choose acceptance rules that reflect the application's risks. Some formatting errors might be handled by a safe validation failure. Unsupported statements in a sensitive workflow may require a stronger review gate. Keep the rule explicit, including what happens when the evaluator cannot reach a confident judgment.
Do not borrow an arbitrary percentage because it looks familiar on a dashboard. Explain why the selected threshold is appropriate for the task and sample. If the dataset is small, describe the observed count rather than implying broad statistical certainty. Release decisions should include the nature of failures, not merely whether the average crossed a visually attractive line.
Inspect failures for repairable patterns
Read the failed examples and group them by likely cause. The application may lack relevant context, the prompt may be ambiguous, a retrieval step may omit the needed passage, or the response contract may be unclear. Those causes suggest different fixes. Adding more instructions to the prompt is not automatically the right response to every failure.
Create a small repair plan and test the change on both the failed cases and a separate comparison set. A fix that only works on the examples used to design it may not generalize. Preserve the original failed case as a regression test where appropriate, so the same issue is less likely to disappear from view and return later.
Use a concrete review example
Consider an assistant answering a question about a supplied return policy. One criterion checks whether it gives the correct return window. Another checks whether it mentions the exception relevant to the user's situation. A third checks whether it invents a completed refund. These are distinct behaviors with different consequences. A reviewer should be able to point to the policy passage and the response sentence behind each judgment. This small example illustrates why a single rating for helpfulness can conceal a consequential mistake even when the answer is polite and easy to read.
Keep evaluation connected to operation
After deployment, collect approved feedback and investigate new failure patterns. Add representative examples to the evaluation set through a deliberate review process. Version the dataset rather than silently changing it, so historical comparisons remain understandable. Retain separate development and acceptance collections when possible.
A practical evaluation program is a loop: define the task, collect evidence, apply a rubric, inspect failures, and test a targeted change. Continue with AI analysis for extraction and classification workflows, or structured report design for the output contract. The purpose of evaluation is to support a better decision with inspectable evidence—not to turn every model response into a confident-looking score.
