Hallucinations are not only a model problem; they are usually a system-design problem. This checklist shows how to reduce unsupported answers in production LLM applications by grounding responses, setting clear refusal rules, validating outputs, testing failure cases, and adding review where errors carry meaningful risk.
Overview
An LLM generates plausible language, not guaranteed facts. Even a capable model can invent a source, misread retrieved context, apply an outdated instruction, or express uncertainty as confidence. Reliability therefore depends on the entire application: prompts, data pipelines, retrieval, tools, validation, user interface, monitoring, and human escalation.
A useful way to approach LLM hallucination mitigation is to create several independent controls. Ground the model in appropriate information. Tell it what to do when evidence is missing. Constrain the response format. Verify important claims before displaying or acting on them. Then measure the system against representative examples, including deliberately difficult ones.
No single prompt can guarantee factual accuracy. A system prompt may establish behavior, but it cannot repair incomplete source data or validate an external action. Likewise, retrieval-augmented generation (RAG) can provide relevant context without proving that the context is correct, current, or sufficient. Treat each layer as part of a reliability process rather than a complete solution.
For implementation details, see this RAG application development guide and the practical guide to building a reliable RAG application.
Checklist by scenario
When the answer should come from a known knowledge base
- Define which documents, records, or APIs are authoritative for the use case.
- Attach source metadata such as document title, version, date, section, or record identifier.
- Use retrieval filters for permissions, tenant, region, language, and freshness where applicable.
- Set a minimum relevance or evidence threshold. If retrieval produces no adequate evidence, the application should allow a refusal or clarification response.
- Instruct the model to distinguish between information supported by the supplied context and information it cannot establish.
- Display citations or evidence links when users need to verify the answer.
A grounded prompt should not merely say “use the context.” It should define the boundary. For example: “Answer only from the supplied sources. If the sources do not support an answer, say that the information is not available and identify what would be needed to answer.” This is guidance, not a substitute for retrieval and application-level checks.
When the model summarizes or transforms user-provided text
- Separate the user’s source text from instructions using clear delimiters or structured fields.
- Specify whether the task is extraction, compression, classification, rewriting, or interpretation.
- Tell the model not to add facts, examples, conclusions, or recommendations that are absent from the source unless the task explicitly permits them.
- Preserve important qualifiers such as “may,” “approximately,” “not confirmed,” and “according to.”
- Test long documents, contradictory passages, tables, footnotes, and text containing adversarial instructions.
For extraction tasks, prefer a schema with explicit null or “unknown” values instead of forcing the model to fill every field. A missing value is usually safer than a plausible invention.
When the model can call tools or trigger actions
- Give each tool a narrow purpose and describe its arguments precisely.
- Validate every argument in application code, including types, ranges, permissions, and required fields.
- Use allowlists for destinations, operations, and records whenever possible.
- Require confirmation before irreversible, external, financial, or high-impact actions.
- Return tool errors to the orchestration layer in a controlled format rather than exposing raw internal details.
- Log the user request, selected tool, validated arguments, result status, and final response for investigation.
Tool calling can reduce unsupported claims when the application obtains facts from a dependable system, but a tool result can still be incomplete or misinterpreted. Validate the result and tell the model how to handle an empty, ambiguous, or failed response. The LLM tool calling tutorial covers safe function design, argument validation, and failure handling.
When the answer affects a person or an important decision
- Define which outputs are advisory and which require human approval.
- Route low-confidence, conflicting, or policy-sensitive cases to a reviewer.
- Make the model’s limitations visible without overwhelming the user with generic warnings.
- Retain enough evidence for a reviewer to reconstruct why the answer was produced.
- Provide a correction path so users can report unsupported answers and identify the affected workflow.
Human review works best when it is specific. A reviewer should be asked to verify the cited evidence, key extracted fields, proposed action, or unresolved ambiguity—not simply approve a fluent paragraph.
What to double-check
Grounding quality
Check whether the retrieved or supplied context actually contains the answer. Relevant-looking text is not always sufficient evidence. Review chunk boundaries, duplicate passages, stale records, access filters, and query rewriting. If the system retrieves several conflicting versions, define how conflicts are surfaced rather than silently selecting one.
Refusal behavior
A refusal should be useful and proportionate. It can state that the available evidence is insufficient, identify the missing detail, and suggest a safe next step. Test refusals for both false positives, where the model declines answerable questions, and false negatives, where it answers despite inadequate evidence.
Structured output and validation
Use structured output when downstream code depends on the result. Validate the response against a schema, reject unexpected fields where appropriate, and handle malformed or incomplete output explicitly. A valid JSON document is not necessarily a truthful answer: schema validation checks shape, not factual correctness. Pair it with evidence checks, business rules, or a second verification step. See the guide to structured JSON output from LLMs for a deeper implementation checklist.
Evaluation coverage
Build a test set from real or carefully constructed cases. Include ordinary requests, ambiguous questions, missing evidence, conflicting sources, prompt injection attempts, long inputs, and tool failures. Track more than one result: factual support, citation accuracy, refusal quality, task completion, formatting validity, latency, and cost may all matter. A prompt change that improves one metric can weaken another.
Run the same evaluation after changing the model, system prompt, retrieval settings, chunking method, tool schema, or post-processing code. The prompt testing framework guide provides a practical foundation for maintaining these checks.
Common mistakes
- Relying on confidence language. “Be accurate” and “do not hallucinate” are useful intentions, but they do not define evidence, refusal conditions, or verification.
- Treating citations as proof. A cited source may be irrelevant, stale, or only loosely related to the claim. Check the relationship between each important statement and its evidence.
- Forcing an answer. Prompts that require every question to receive a direct response encourage guessing. Permit “unknown,” clarification, and refusal paths.
- Ignoring the data pipeline. Poor parsing, truncated documents, bad access filters, and stale indexes can look like model failures. Inspect the context sent to the model.
- Validating only syntax. Parseable JSON, a correct enum, or a completed template does not establish that the content is true or appropriate.
- Testing only successful examples. Reliability work must include missing, contradictory, malicious, and out-of-scope inputs.
- Changing several layers at once. If the prompt, model, retrieval settings, and post-processing change together, it becomes difficult to identify what improved or caused a regression.
- Logging sensitive content without a plan. Observability is important, but decide what is necessary to retain, who can access it, and how test data is separated from production data.
When to revisit
Use this checklist before launching a new AI feature, changing a system prompt, switching models, modifying retrieval, adding a tool, or changing the source data. Revisit it after an incident, a user-reported error, a meaningful shift in query patterns, or a change in the consequences of an incorrect answer.
At each review, record the current model and application configuration, evidence sources, refusal rules, tool permissions, evaluation set, and known limitations. Run a small regression suite before deployment and a broader review when the underlying workflow changes. Compare failures with the previous version rather than judging the new output in isolation.
A practical release gate is:
- Known-answer tests meet the required support and accuracy standard.
- Unsupported questions produce an acceptable refusal or clarification.
- Citations lead to relevant evidence when citations are shown.
- Structured responses pass schema and business-rule validation.
- Tool calls are authorized, validated, logged, and safely recoverable.
- Human review is defined for cases the automated controls cannot reliably resolve.
Reliability is a maintenance practice, not a one-time prompt engineering task. Keep the checklist close to the deployment process, update it when workflows or tools change, and use production feedback to improve the evidence, tests, and escalation paths that surround the model.
