LLM Evaluation: 8 Tests to Run Before and After Launch
LLM evaluation is the process of checking whether a large language model or AI application produces outputs that meet defined quality, safety, and business requirements. A response can sound fluent and still be inaccurate, incomplete, unsafe, or expensive to produce. Evaluation gives teams repeatable evidence about how a system behaves across realistic tasks.
This guide covers the main evaluation types, how to build a useful test set, eight dimensions to measure, and ways to keep checks relevant after release. The central principle is simple: evaluate the whole application in the context where people will use it.
Why LLM evaluation matters
Traditional software often returns the same output for the same input. Language models can produce varied answers, and small changes to prompts, models, tools, or source data may shift quality. A single successful demo therefore tells little about how a feature will behave across customer questions or edge cases.
Evaluation helps teams compare versions, find regressions, and decide whether a change is ready to ship. It also helps reveal trade-offs. A model may improve answer completeness while increasing latency or cost. A retrieval change may improve source coverage but introduce irrelevant context. Teams need measures tied to the actual use case, not a single generic score.
Three complementary ways to evaluate
Automated checks
Use deterministic checks for requirements that have clear answers: valid JSON, required fields, correct links, allowed labels, or whether a response includes a needed disclosure. These checks are fast and repeatable, but they cannot assess every nuance of usefulness.
Model-based grading
A separate model can assess qualities such as relevance, grounding, or adherence to a rubric. Model graders can scale human review, but they may share weaknesses with the system being evaluated. Calibrate them against examples that people have scored and inspect disagreements.
Human review
People remain important for judging clarity, tone, harm, and domain correctness, especially when the stakes are high. Use reviewers with relevant expertise, give them a consistent rubric, and sample difficult as well as ordinary cases.
The OpenAI evaluation guide describes defining a task, testing representative inputs, and analyzing results. It currently notes planned changes to the Evals platform, so teams should verify the current lifecycle and choose supported tooling before building dependencies. Google Cloud’s AI and ML performance guidance also recommends defining measures, selecting suitable data, and standardizing evaluation workflows.
Build a representative evaluation dataset
A useful dataset reflects the inputs and outcomes the application will encounter. Collect real examples where permitted, remove or protect personal information, and include a range of task difficulty. Keep expected answers, source references, acceptable variations, and risk labels separate from the model prompt.
Include at least these categories:
- Common requests that represent the normal workload.
- Ambiguous requests that should trigger a clarifying question.
- Missing information that should lead the system to abstain or escalate.
- Edge cases such as unusual formatting, typos, long inputs, and conflicting details.
- Policy and safety cases that test prohibited requests and sensitive data boundaries.
- Examples from multiple user groups, regions, products, or languages that matter to the service.
Split examples into a development set for iteration and a holdout set for a more independent check. Version both datasets. When policies, products, or user behavior change, add new cases rather than quietly replacing the old ones. This lets the team track whether a change improved the feature overall or merely fit the examples it had already seen.
Eight dimensions to test
1. Task correctness
Does the response solve the user’s request? Define the expected outcome before reviewing the output. For a ticket classifier, this might mean the correct category. For a support assistant, it may require answering the question and following the right process.
2. Factual accuracy and grounding
Check factual statements against trusted evidence. If the application retrieves documents, measure whether the cited sources support the answer and whether the system avoids claims beyond them. A plausible tone is not proof.
3. Instruction following
Test required formats, length limits, tone, and business rules. Include requests that try to override system guidance or ask the model to ignore required steps. Score the resulting behavior rather than only inspecting prompt wording.
4. Completeness and relevance
An answer can be correct but incomplete, or detailed but off topic. Use a rubric that defines the key points needed for a useful response. For long-form outputs, check whether it addresses the promised sections and avoids irrelevant padding.
5. Safety, privacy, and permissions
Test refusal behavior, confidential-data exposure, unsafe instructions, and access boundaries. When tools can change business records, confirm that the application checks authorization and validates the action independently of the model’s language.
6. Robustness
Vary the wording, spelling, order, and format of equivalent requests. Add noisy inputs, prompt injection attempts, conflicting instructions, and source documents with misleading text. The feature should fail clearly instead of behaving unpredictably.
7. Latency and reliability
Measure end-to-end response time, timeouts, retry behavior, and tool failures under realistic load. A correct answer that arrives after the user has moved on may not serve the workflow. Test degraded modes and user-facing messages when a dependency becomes unavailable.
8. Cost and operational fit
Track tokens or other usage measures, calls to external tools, and human review time. Compare these costs with the value of the task. More model steps do not automatically mean a better system; sometimes a smaller model, simpler prompt, or deterministic rule can handle part of the workflow.
Evaluate the full application, not just the model
Many failures occur outside the model itself. A retrieval system may provide stale content. A tool may return incomplete records. A UI may hide a caveat. A workflow may allow an output to trigger an action without sufficient review. Test the entire path from user input through retrieval and tool execution to the final result.
For RAG features, separately assess retrieval quality and answer quality. For agentic features, check tool selection, arguments, permissions, and final handoff. For structured output, validate the parsed result in the consuming system rather than assuming that readable text means valid data.
Compare versions without being misled by averages
Report results by category as well as overall. A high average can conceal a serious failure in a smaller group or an important edge case. Review examples where the new version regressed, and decide whether the change creates a risk that an average score cannot express.
Use the same test set when comparing versions, and record model, prompt, tools, retrieval settings, and source-data version. Change one major factor at a time where practical. If a change improves an evaluation score, confirm that it also helps actual users.
Build evaluation into delivery
- Write acceptance criteria before implementation.
- Run quick deterministic checks during development.
- Run a representative evaluation before a major release.
- Require human review for high-impact use cases or new failure modes.
- Release gradually and monitor corrections, escalations, latency, and cost.
- Turn meaningful production failures into new test cases.
A useful evaluation suite is maintained alongside the feature. Assign an owner for test data, rubric changes, and release decisions. Keep private or regulated data protected, and limit access to evaluation artifacts just as you would for production data.
When requirements are still changing, a small workflow prototype can help stakeholders agree on what a passing result looks like before the team commits to a full implementation.
Example: evaluate a support answer from end to end
Suppose an assistant answers questions about order status. A strong test case contains the user’s question, the expected account, the approved data source, required answer points, and a policy label. A passing response must use the right account, avoid exposing another customer’s data, give the current status, and explain when the record lacks a delivery date. This is more informative than asking a grader whether the answer “sounds good.”
Track separate scores for correct record selection, factual support, completeness, and appropriate escalation. If the assistant calls the wrong tool, investigate routing. If it retrieves the correct data but misstates the result, the issue may lie in instructions or response generation. Separating failure categories shortens the path to a useful fix.
Set release gates that match risk
Not every metric needs a universal threshold. A formatting issue may be acceptable for a draft-only feature but unacceptable when software consumes the output automatically. Define release rules based on impact: block a release for a critical privacy regression, require review for a notable fall in task success, and record low-severity style changes for later iteration.
Keep a holdout set that the prompt designer does not use during daily tuning. Otherwise, a team can improve its score on familiar examples while missing new failure patterns. Add hard cases from incidents and user corrections, but preserve a separate slice to compare releases consistently.
Monitor real use without over-collecting data
After launch, measure failure reports, user edits, fallback rates, latency, and cost. Sample outputs for human review with appropriate privacy controls. Avoid collecting more personal or confidential data than evaluation needs; where possible, use redaction, access-limited storage, and short retention periods.
Evaluation results are evidence, not a guarantee. Human review can disagree, labels can be incomplete, and a changing user population can make an old test set less representative. Schedule a review of the dataset and rubric when the product, model, or source data changes substantially.
For a new AI feature, a concise requirements specification can make pass criteria, exception handling, data access, and escalation rules explicit before implementation begins.
Frequently Asked Questions
What is an LLM evaluation?
It is a repeatable assessment of how a language model or AI application performs against defined criteria such as correctness, relevance, safety, and cost.
Can another LLM grade AI outputs?
Yes, for some criteria, but validate the grader against human judgments and inspect disagreements. Use deterministic checks where requirements have clear expected values.
How many test examples do I need?
There is no universal count. Start with enough examples to cover common tasks, important variants, and risk cases. Add examples when monitoring reveals new failures.
Should AI evaluation continue after launch?
Yes. Models, prompts, tools, source data, and user behavior can change. Monitor performance and update the evaluation set as the application evolves.
Conclusion
LLM evaluation turns subjective confidence into evidence about a feature’s strengths and limits. Define success around user needs, build representative examples, combine automated checks with human judgment, and assess the entire workflow. Keep the test suite current so that every important change can be reviewed against the same standards.
