Reliable LLM applications need more than a convincing demo. This practical framework shows how to evaluate accuracy, usefulness, safety, stability, latency, cost, refusal behavior, and tool-call performance using repeatable test cases, explicit assumptions, and release gates. Use it when comparing models, changing prompts, tuning retrieval, or investigating a production regression.
Overview
LLM evaluation metrics are measurements that help a team decide whether an application is behaving well for its intended use. There is no single score that captures quality. A customer-support assistant, document extractor, coding helper, and RAG application have different failure modes and should be tested differently.
A useful evaluation system combines three layers:
- Task quality: Is the answer correct, relevant, complete, and grounded in the available information?
- Operational performance: How much time, money, context, and infrastructure does each request consume?
- Risk and reliability: Does the application follow its instructions, refuse unsuitable requests, produce valid structured output, and use tools safely?
Evaluate the complete application rather than the model in isolation. A model may perform well with a carefully selected prompt but fail when retrieval returns no documents, a tool times out, or a user provides ambiguous input. Your test should therefore include the system prompt, model settings, retrieved context, tools, post-processing, and fallback behavior.
For prompt changes, keep the previous version available as a baseline. The AI prompt versioning guide provides a practical way to compare and roll back prompt changes. For retrieval-based systems, pair response evaluation with retrieval checks; the RAG application development guide covers chunking, retrieval, and citations.
How to estimate
Begin with a representative evaluation set. Each record should contain the user input, the expected behavior, any authorized reference material, and the conditions under which the request is valid. Include normal cases, edge cases, ambiguous requests, unsupported questions, long inputs, malformed data, and likely misuse. A small, carefully curated set is more useful than a large set with unclear expectations.
Run every record through a fixed application configuration and store the input, output, retrieved context, tool calls, validation errors, latency, token usage, and model or prompt version. Then calculate metrics using consistent denominators.
| Metric | What it measures | Basic estimation method |
|---|---|---|
| Correctness | Whether the answer matches a verified answer or acceptable outcome | Correct cases ÷ cases requiring a correct answer |
| Faithfulness | Whether claims are supported by supplied context or source data | Supported claims ÷ checked claims |
| Relevance | Whether the response addresses the user’s request without unnecessary content | Acceptable relevance ratings ÷ reviewed responses |
| Completeness | Whether required fields, steps, or constraints are covered | Completed requirements ÷ total requirements |
| Valid output | Whether the response passes a schema, parser, or format check | Valid outputs ÷ outputs tested |
| Refusal precision | Whether unsafe or unsupported requests are refused while valid requests are answered | Appropriate refusals and answers ÷ relevant refusal cases |
| Tool-call success | Whether the correct tool is selected with usable arguments and a successful result | Successful required calls ÷ required tool-call cases |
| Latency | Time from request receipt to usable response | Report median and high-percentile latency, not only an average |
| Cost | Estimated spend per request or workflow | Input usage cost + output usage cost + tool and infrastructure costs |
For a simple quality estimate, use a weighted score only after preserving the individual metrics. For example:
Quality index = 0.40 correctness + 0.25 faithfulness + 0.20 completeness + 0.15 format validity
The weights are assumptions, not universal standards. A regulated workflow may assign greater importance to faithfulness and refusal behavior, while an internal writing assistant may emphasize relevance and usefulness. Never let a high aggregate score conceal a critical failure category. Set minimum thresholds for safety, correctness, and schema validity before using a weighted score for comparison.
For cost estimation, define:
Estimated monthly cost = requests per month × average cost per request + fixed platform costs
Where average cost per request includes input tokens, output tokens, retries, tool calls, retrieval, and any review or routing step. Use the current prices supplied by your chosen providers when making a purchasing decision; prices and billing rules can change, so record the date and assumptions beside the estimate.
Inputs and assumptions
A dependable LLM testing framework makes its inputs visible. Record at least the following:
- Evaluation set: Case ID, user request, expected answer or acceptance criteria, difficulty, and risk level.
- Application configuration: Model identifier, system prompt version, temperature or equivalent settings, context limits, retrieval configuration, and tool definitions.
- Reference material: Source documents, expected citations, required fields, and an indication of whether the answer is allowed to use outside knowledge.
- Operational inputs: Typical input and output size, request volume, concurrency, retry policy, timeout, and fallback route.
- Review method: Automated checks, human reviewers, judge-model instructions, or a combination of methods.
Automated checks work well for deterministic properties: valid JSON, required fields, citation presence, prohibited phrases, exact classifications, tool names, and latency limits. They are less reliable for nuanced usefulness or factual correctness unless a verified reference is available. Structured output tests should validate both syntax and meaning. See the guide to structured JSON output from LLMs for schema validation and retry considerations.
Use human review for cases where acceptable answers can vary. A simple rubric can score each response from zero to two:
- Correctness: 0 = materially wrong, 1 = partly correct, 2 = correct.
- Relevance: 0 = does not answer the request, 1 = partly addresses it, 2 = directly addresses it.
- Grounding: 0 = unsupported claims, 1 = mixed support, 2 = claims are supported or appropriately qualified.
- Safety and instruction following: 0 = unacceptable behavior, 1 = minor issue, 2 = follows the intended boundary.
Give reviewers the same instructions and allow an unknown or not-applicable label where appropriate. Do not force a numerical score when the reference answer is itself uncertain. For judge-model evaluations, periodically compare automated judgments with human decisions and inspect disagreements.
Worked examples
Example 1: comparing two prompt versions. Suppose an evaluation set contains 100 support questions. Version A produces 86 correct answers, 90 relevant answers, and 94 valid structured responses. Version B produces 89 correct answers, 88 relevant answers, and 98 valid structured responses. If correctness and format validity are release requirements, Version B is the stronger candidate despite its lower relevance count. Review the cases that changed between versions before making the decision; aggregate results do not explain why behavior changed.
Example 2: estimating workflow cost. Assume a workflow receives 20,000 requests per month. A test run estimates an average request cost of 0.004 units of currency, including an expected retry rate, and fixed platform costs are estimated at 75 units per month. The planning estimate is:
20,000 × 0.004 + 75 = 155 units per month
This is a planning scenario, not a current price claim. Recalculate it with the provider’s current input and output rates, actual token distributions, tool usage, and observed retry frequency. Also model a low, expected, and high-volume case because averages can hide long prompts or traffic spikes.
Example 3: evaluating a RAG assistant. Separate retrieval from generation. First check whether the expected source appears in the retrieved context. Then check whether the answer uses that source accurately, includes required citations, and declines to invent an answer when the source is absent. An answer can be faithful to retrieved text while still being incomplete if the retriever omitted the key passage.
After each run, compare the new result with the baseline by case, not just by overall average. Flag newly failed cases, newly passed cases, changes in refusal behavior, latency outliers, cost changes, and tool-call argument errors. The prompt testing frameworks guide can help organize repeatable evaluation runs.
When to recalculate
Run the full evaluation set before releasing a new model, system prompt, retrieval index, tool definition, output schema, or post-processing rule. Re-run targeted cases when you change a narrow component, such as a parser or timeout. Schedule recurring checks for production applications so that gradual changes in traffic, data, or provider behavior are visible.
Recalculate cost whenever provider pricing, token usage, request volume, retry rates, context length, or tool usage changes. Recalculate quality thresholds when the product scope changes, but document the reason rather than quietly moving the goalposts. A new benchmark, model capability, or business requirement may justify a different test set; preserve the old results so historical comparisons remain meaningful.
Before release, use this checklist:
- Pin the model, prompt, retrieval, and tool versions used in the run.
- Run normal, edge, unsupported, long-context, and failure-recovery cases.
- Check correctness, faithfulness, relevance, completeness, format validity, refusal behavior, latency, cost, and tool-call success separately.
- Compare against a known baseline and inspect every critical regression.
- Confirm that automated scores have been sampled against human review.
- Record assumptions, thresholds, test date, and release decision.
- Keep a rollback path for the prompt, model, retrieval configuration, and tools.
Evaluation is most useful when it becomes part of normal development rather than a final inspection. Keep the dataset versioned, add real failure cases after triage, and review the metrics whenever the inputs or economics change. This creates a practical feedback loop for improving an LLM application without treating any single score as proof of reliability.
