LLM prompt security is an application security problem, not only a prompt engineering problem. This checklist helps developers protect system instructions, user inputs, retrieved content, sensitive data, and tool calls against prompt injection, data exposure, jailbreak attempts, and unsafe actions. Use it before release, whenever you connect a new data source or tool, and after meaningful changes to prompts, models, permissions, or workflows.
Overview
A language model does not reliably distinguish between trusted instructions and untrusted text unless the surrounding application enforces that distinction. A system prompt can establish priorities and constraints, but it should not be treated as a secret security boundary or as the only control protecting an application.
Secure LLM applications use layered defenses. The model receives carefully scoped context, the application validates inputs and outputs, retrieval systems label and constrain external content, and tool calls pass through explicit authorization and argument checks. Sensitive operations should be handled by ordinary application logic wherever possible rather than delegated entirely to a model.
Start by mapping the complete path of data through the application:
- What the user can submit.
- What the model can see from conversation history, files, retrieval, memory, or system instructions.
- Which tools, APIs, databases, or workflows the model can request.
- What the application returns to the user, logs, stores, or sends to another service.
This inventory makes it easier to identify where an attacker could influence instructions, extract information, trigger an action, or cause a harmful output. For prompt design background, compare this checklist with the system prompt examples for common LLM applications, then apply security controls outside the prompt as well.
Checklist by scenario
| Threat | What can happen | Practical controls |
|---|---|---|
| Direct prompt injection | A user asks the model to ignore its rules, reveal hidden instructions, or change its task. | Separate trusted instructions from user content, minimize sensitive prompt material, validate requested actions, and treat refusals or policy checks as application logic rather than prose alone. |
| Indirect prompt injection | Instructions hidden in a webpage, document, email, ticket, or retrieved passage influence the model. | Label external content as untrusted data, restrict which fields reach the model, remove unnecessary markup, require citations or evidence for factual tasks, and prevent retrieved text from authorizing tools. |
| Sensitive-data exposure | The model reveals secrets, personal information, internal prompts, private documents, or data from another user. | Apply least-privilege access before retrieval, redact or tokenize sensitive values, isolate tenants, avoid placing secrets in prompts, and test output handling and logs for accidental disclosure. |
| Jailbreak attempts | A user uses role-play, encoding, translation, repeated instructions, or emotional pressure to bypass restrictions. | Define the allowed task clearly, use consistent refusal behavior, add abuse-oriented test cases, rate-limit suspicious activity, and keep high-risk decisions behind explicit controls. |
| Unsafe tool request | The model requests an unauthorized transaction, destructive command, message, file change, or data export. | Use an allowlist of tools, validate every argument against a schema, enforce user and service permissions, require confirmation for consequential actions, and record a complete audit trail. |
Input-filtering checklist
- Classify inputs by trust level: user text, administrator instructions, retrieved content, tool results, and application-generated metadata should not be interchangeable.
- Limit input size, file types, nesting, and conversation history to what the task requires.
- Normalize content before inspection, while preserving enough information for the model to perform its intended task.
- Scan for secrets and personal data before storage, logging, retrieval, or external API transmission.
- Do not rely on keyword blocking alone. Attackers can rephrase, encode, translate, or distribute an instruction across multiple messages.
- Return structured output where practical, validate it against a schema, and reject or quarantine malformed results.
For applications that use retrieval-augmented generation, treat retrieved passages as evidence rather than commands. The RAG application development guide can help with retrieval and citation design, but security still depends on access control, content handling, and tool authorization outside the model.
Tool and API controls
- Give each tool the narrowest permissions and data scope it needs.
- Use separate credentials for separate actions; never expose a general-purpose administrative credential to a model-driven workflow.
- Validate identifiers, paths, recipients, quantities, filters, and query limits on the server side.
- Use dry-run or preview modes for actions that change data or communicate externally.
- Require a fresh user confirmation when an action is irreversible, costly, privacy-sensitive, or directed at a third party.
- Set timeouts, quotas, concurrency limits, and failure behavior so a model cannot create an uncontrolled loop.
See the LLM tool calling tutorial for a deeper treatment of safe functions, argument validation, and failure handling.
What to double-check before release
System prompt security
Write system instructions for clarity and task control, not as a place to store credentials or irreplaceable business secrets. State the model's role, permitted scope, output requirements, and escalation behavior. Explain that user and retrieved content may contain instructions that are not authoritative. Avoid including information that would cause a serious breach if repeated in an answer.
Then test whether the surrounding application enforces those instructions. A model may still disclose, misinterpret, or transform protected content. Prompt rules should complement authorization, filtering, and monitoring rather than replace them.
Data boundaries
Verify that a user can retrieve only records they are authorized to access. Check tenant identifiers, document permissions, cached responses, conversation memory, backups, and logs. Confirm that error messages do not expose raw prompts, credentials, internal URLs, stack traces, or retrieved documents.
Output handling
Do not send model output directly into HTML, SQL, shell commands, email, code execution, or configuration files without context-appropriate encoding and validation. If the application expects structured output JSON, parse it strictly, validate types and allowed values, and define a safe fallback when parsing fails.
Red-team test cases
Maintain a small repeatable suite of adversarial tests. Include requests to reveal the system prompt, imitate an administrator, ignore previous instructions, retrieve another user's record, follow instructions hidden in a document, call an unapproved tool, export secrets, and continue after a refusal. Test variations using different languages, encodings, long context, multiple turns, and misleading tool results.
Record the input, relevant context, model response, tool decision, authorization result, and final user-visible output. Use the results to improve both prompts and application controls. A broader LLM evaluation and testing framework can help turn these cases into regression tests.
Common mistakes
- Putting secrets in the system prompt: Prompts can appear in traces, debugging tools, evaluation datasets, or model responses. Keep credentials in a secret manager and outside model-visible text.
- Assuming hidden instructions cannot be extracted: Treat prompt confidentiality as desirable, not guaranteed. Design the application so disclosure does not grant access.
- Trusting retrieved content: A document can contain text that looks like an instruction. Label its role and prevent it from granting authority.
- Letting the model authorize itself: The model may propose an action, but the server should make the final permission decision.
- Filtering only the first message: Injection can enter through follow-up messages, attachments, tool results, memory, or third-party content.
- Logging everything: Detailed logs support debugging, but unrestricted logs can become a data-leak channel. Redact, restrict, retain, and review them deliberately.
- Changing prompts without regression tests: A small wording change can affect refusals, tool use, data handling, and output parsing. Use versioning and compare results before deployment; the prompt versioning guide provides a practical workflow.
- Confusing a refusal with security: A polite refusal does not prevent a separate endpoint, tool, cache, or log from exposing the same information.
When to revisit
Re-run this checklist before launch and whenever the workflow changes. The most important review points are:
- Adding a new model, provider, agent framework, retrieval source, file type, or memory feature.
- Changing the system prompt, tool definitions, output schema, context-window strategy, or fallback behavior.
- Granting access to a new database, SaaS API, filesystem, messaging channel, or automation job.
- Moving from read-only assistance to actions that edit, delete, purchase, publish, or contact people.
- Changing identity, tenant, permission, retention, logging, or data-residency arrangements.
- After an incident, suspicious usage pattern, unexpected tool call, data leak, or evaluation regression.
- During planned seasonal or operational reviews, especially before periods of higher traffic or staffing changes.
For each review, assign an owner and record the application version, prompt version, model configuration, connected tools, test results, open risks, and approval decision. Start with one normal task and one adversarial variant for every sensitive workflow. Confirm that the model cannot bypass server-side authorization, that untrusted content cannot authorize an action, and that failures default to a safe state.
Finally, update the checklist when the application changes rather than treating it as a one-time launch document. Secure LLM applications are maintained through small, repeatable controls: least privilege, strict validation, isolated data access, cautious tool use, red-team regression tests, and clear incident handling.
