LLM context window management is a design problem, not simply a matter of choosing a model with a larger limit. This guide shows how to budget tokens, chunk documents, compress context, preserve conversation memory, and assemble prompts so the information most relevant to the current task remains available. It also includes repeatable worksheets and worked examples you can use when building or tuning a retrieval-augmented generation (RAG) application.
Overview
An LLM context window contains everything the model can use for one request: system instructions, conversation history, retrieved passages, tool results, the latest user message, and space reserved for the response. The usable budget is therefore smaller than the provider's advertised maximum.
A practical context budget can be expressed as:
Available input budget = context limit − reserved output − safety margin
For example, if a model's documented context limit is C, your maximum expected response is O, and your safety margin is S, then the input budget is C − O − S. The safety margin accounts for tokenizer variation, tool-call arguments, formatting overhead, and changes to the prompt assembled by your application.
Context management has two related goals:
- Fit: prevent requests from exceeding the model or provider limit.
- Preserve signal: retain the facts, instructions, and recent decisions needed for a useful answer.
Sending more text is not automatically better. Irrelevant passages can compete with important evidence, repeated history can obscure the current question, and untrusted text can introduce prompt-injection risks. A reliable LLM application treats context as a ranked, typed input rather than an unstructured transcript.
Before implementing a strategy, measure tokens with the tokenizer or SDK associated with the model you plan to use. Character counts and word counts are useful rough estimates, but they are not a substitute for token counting because tokenization varies by language, formatting, code, and model family. For a related implementation workflow, see the LLM token counting and cost estimation guide.
How to estimate
Start with a context-budget worksheet for every request type in your application. Record the following values:
| Component | Symbol | What to record |
|---|---|---|
| Model context limit | C | The limit documented for the selected model and endpoint |
| System instructions | I | Stable rules, output requirements, and safety guidance |
| Conversation history | H | Messages retained for this request |
| Retrieved context | R | Documents, chunks, metadata, and citations |
| Tool results | T | Structured or textual results returned by tools |
| Current user input | U | The latest request and any attached data |
| Reserved output | O | Space needed for the model's response or tool arguments |
| Safety margin | S | Room for variation and future prompt changes |
The request fits when:
I + H + R + T + U + O + S ≤ C
Use measured token counts where possible. If a component varies, estimate its high or high-normal value rather than its average. A request that fits only under ideal conditions will eventually fail when a user submits a longer question, a tool returns more fields, or a retrieved passage contains unusually dense code.
When the budget is tight, trim in an order that reflects information value:
- Remove duplicate metadata, repeated instructions, and irrelevant tool fields.
- Filter retrieved passages by relevance, document permissions, and freshness.
- Replace old conversational turns with a validated summary.
- Compress long passages while preserving names, numbers, constraints, and exceptions.
- Reduce the number of chunks or lower the reserved output only if the application can tolerate the change.
Do not silently truncate from the beginning or end of a prompt without testing. The discarded portion may contain the user’s constraints, a definition, or a recent correction.
Inputs and assumptions
Choosing a chunking strategy
Chunking determines what a retriever can return and how much surrounding context each result carries. There is no universal chunk size; the right choice depends on document structure and task.
| Strategy | Strength | Trade-off | Good starting use |
|---|---|---|---|
| Fixed length | Simple and predictable | May split a thought, table, or code block | Uniform technical text or an initial baseline |
| Paragraph or section based | Preserves natural units | Chunk sizes can vary considerably | Guides, policies, and prose documents |
| Structure-aware | Preserves headings, lists, tables, and code boundaries | Requires format-specific parsing | Documentation, source code, and structured files |
| Hierarchical | Supports broad summaries and detailed retrieval | Needs more indexing and ranking logic | Large manuals and knowledge bases |
Overlap can protect against information being split across boundaries. However, overlap also consumes retrieval budget and can cause near-duplicate results. Begin with modest overlap, then test whether it improves answers rather than assuming that more overlap is safer. For documents with headings, include the parent heading in each child chunk; this often supplies useful meaning without repeating an entire section.
Conversation memory patterns
Conversation memory should match the product’s requirements, not merely preserve every message.
- Recent-turn window: Keep the latest messages. This is simple and works when the task depends mainly on immediate context.
- Rolling summary: Replace older turns with a compact summary containing decisions, unresolved questions, user preferences, and important facts. Treat the summary as derived data and update it with tests.
- Fact memory: Store selected, user-approved facts separately from the transcript. Retrieve only facts relevant to the current request.
- Conversation plus retrieval: Store prior messages or documents in a searchable system, then retrieve specific evidence instead of replaying the full history.
Use a clear distinction between instructions, remembered facts, retrieved evidence, and untrusted content. This helps the application decide what may guide behavior and what should only be quoted or analyzed. Review the LLM prompt security checklist when retrieved documents or tool output may contain instructions.
Prompt assembly template
A predictable order makes debugging easier. One useful template is:
[System instructions and output contract]
[Task-specific constraints]
[Relevant conversation summary]
[Selected user-approved memory]
[Retrieved evidence with source identifiers]
[Tool results, labeled by tool and timestamp]
[Current user request]
Answer using the evidence where applicable. If evidence is missing or conflicting, say so.
Keep source identifiers outside the prose when possible, and validate structured output before passing it to another component. The structured output guide covers schemas, validation, and recovery patterns.
Worked examples
Example 1: RAG question answering
Assume an application has a context limit of C. The team reserves O for the answer and S for safety. Its stable instructions use I, the user question uses U, and there is no conversation history or tool result. The retrieval budget is:
R = C − O − S − I − U
The retriever should not simply return as many chunks as will fit. It should rank candidates, remove duplicates, apply access controls, and stop when the measured token total approaches R. If the top results contain overlapping excerpts from the same section, replace them with one representative chunk or a section-level summary. If an answer requires a definition and an exception, reserve space for both rather than filling the budget with several similar definitions.
Example 2: A long support conversation
Suppose recent turns consume most of the available input budget after a user has exchanged many messages with an assistant. Keep the latest turns that establish the current task, then create a summary with four fields: goal, decisions, constraints, and open questions. Store exact values, identifiers, and quotations separately when precision matters. Before using the summary, compare it with a small set of known questions from the original conversation. If the assistant loses a constraint or changes a decision, revise the summarization prompt or memory schema.
Example 3: Tool results and structured data
A tool may return a full database record when the model needs only a status, owner, and last-update field. Define a response schema for the tool and project the result into that schema before adding it to the prompt. This reduces tokens and lowers ambiguity. For tool-calling design, validation, and failure handling, see the tool calling guide.
In each example, log the token count for every component, not only the final request. Component-level logging shows whether failures come from retrieval, memory growth, tool verbosity, or a prompt change.
When to recalculate
Recalculate the context budget whenever an underlying input changes. At minimum, review it when:
- you change the model, endpoint, tokenizer, SDK, or output format;
- the provider changes context limits, caching behavior, or request accounting;
- you add a new document type, language, attachment format, or tool;
- retrieval quality changes because chunk size, overlap, metadata, or ranking changed;
- users report that the assistant forgot a recent constraint or cited irrelevant evidence;
- summaries become longer, less accurate, or harder to validate;
- latency, token usage, or error rates move outside the application's accepted range.
Make context tests part of your prompt testing framework. Include short and long conversations, conflicting documents, boundary-split facts, empty retrieval results, verbose tool responses, multilingual text, and adversarial instructions inside retrieved content. Measure task accuracy, citation or evidence coverage, context overflow rate, latency, and token usage. The LLM evaluation metrics and testing framework provides a broader approach to testing AI applications.
For a practical next step, create one worksheet per request type, instrument token counts by component, and establish a small regression set before changing chunking or memory logic. Revisit the worksheet after model or provider changes, then compare quality and cost with the previous version. Context management is successful when the application fits reliably while preserving the specific information its users need—not when it fills the largest possible window.