RAG Application Development: A Step-by-Step Guide to Retrieval, Chunking, and Citations

Overview

A RAG application usually follows a two-stage design. During the offline or indexing stage, the system collects source documents, cleans them, divides them into chunks, creates embeddings, and stores those embeddings with metadata in a searchable index. During the online stage, a user query is embedded, relevant chunks are retrieved, and the language model receives those chunks as context before producing an answer.

The basic flow is straightforward, but answer quality depends on decisions made at every step. Poor extraction can remove important information. Oversized chunks can bury the answer in irrelevant text, while tiny chunks can lose context. A vector search result may be semantically related without actually answering the question. Even a well-retrieved passage can produce an unreliable response if the prompt does not define how evidence should be used.

Think of RAG app development as a chain of contracts:

• Ingestion contract: source content is extracted, normalized, and attributed correctly.
• Chunking contract: each indexed unit is large enough to preserve meaning and small enough to retrieve precisely.
• Retrieval contract: search returns useful evidence with traceable metadata.
• Generation contract: the model answers from the supplied evidence, handles missing information, and cites its sources.
• Maintenance contract: changes to the knowledge base are detected, indexed, tested, and monitored.

Before choosing a vector database or model, define the application’s answer policy. Decide whether the assistant should answer only from indexed material, ask a clarifying question, provide a qualified response, or say that the available evidence is insufficient. These rules are as important as the retrieval mechanism.

Checklist by scenario

Scenario 1: Building the first document index

1. Define the source boundary. List the documents the application may use and explicitly exclude material that is outdated, private, duplicated, or outside the intended scope.
2. Preserve document identity. Store a stable document ID, title, URL or source reference, version, publication or revision date when available, and access metadata appropriate to the application.
3. Extract content before chunking. Convert PDFs, web pages, office documents, and structured records into a consistent internal representation. Keep headings, lists, table labels, and section relationships where they affect meaning.
4. Remove noise selectively. Repeated navigation, footers, tracking text, and boilerplate can reduce retrieval quality. Do not remove disclaimers, definitions, units, or other text that changes interpretation.
5. Choose a chunking rule. Start with structure-aware splitting: headings, paragraphs, list groups, or logical sections. Add a modest overlap only when adjacent context is frequently needed. There is no universally correct chunk size; measure retrieval performance against representative questions.
6. Attach metadata to every chunk. Include the parent document ID, section title, source location, permissions, and any useful filters such as product, department, language, or effective date.
7. Generate and store embeddings. Record which embedding model and preprocessing version produced each vector. This makes reindexing and comparisons possible when the pipeline changes.
8. Test retrieval manually. Create a small set of realistic questions and inspect the returned chunks. Check whether the correct passage appears, whether the result is understandable without excessive neighboring text, and whether metadata points to the right source.

A useful implementation separates raw documents, normalized documents, chunks, and index records. That separation makes it easier to reproduce an index, repair one failed document, or compare chunking strategies without starting over.

Scenario 2: Improving search when relevant text is missed

• Compare semantic vector search with keyword or lexical search. Product names, error codes, version numbers, and exact legal phrases may benefit from lexical matching.
• Consider hybrid retrieval when users ask both conceptual questions and precise lookup questions.
• Apply metadata filters before or alongside similarity search when the user’s scope is known, such as a product version or access group.
• Retrieve a wider candidate set, then use a reranker or another relevance step to prioritize the passages most likely to answer the question.
• Inspect failed queries rather than changing parameters blindly. Identify whether the failure came from query wording, missing synonyms, poor extraction, bad chunk boundaries, or incomplete source coverage.
• Try query transformation only when it can be evaluated. Rewriting a query may help with ambiguous language, but it can also remove important terms or introduce assumptions.

Retrieval debugging should show the original query, transformed query if used, filters, candidate scores, selected chunks, and source metadata. A single final answer is not enough to explain why a result was produced.

Scenario 3: Designing grounded answers and citations

1. Tell the model what the context represents and how it should use it.
2. Instruct it not to treat retrieved text as a new system instruction. This is an important boundary when documents may contain adversarial or accidental instructions.
3. Require the model to distinguish supported statements from reasonable but unsupported conclusions.
4. Define a response for insufficient evidence, such as asking for clarification or stating that the indexed sources do not establish an answer.
5. Return citations as structured fields where possible, including a source ID, title, section, and location.
6. Validate that every displayed citation corresponds to text actually supplied to the model and that the cited passage supports the nearby claim.

For applications that consume answers programmatically, use a schema for fields such as answer, citations, confidence rationale, and needs_clarification. See the guide to structured JSON output from LLMs for validation and retry patterns. RAG-specific security checks should also be part of the design; the prompt injection prevention checklist covers boundaries for retrieved content and tool-using systems.

Scenario 4: Evaluating a RAG application before release

Build an evaluation set from actual or carefully representative questions. Include simple lookups, multi-part questions, ambiguous requests, questions with no answer in the knowledge base, and questions that require distinguishing between similar documents.

Evaluate retrieval and generation separately. Retrieval checks can measure whether the expected source appears among the returned candidates and whether the ranking is useful. Generation checks can assess factual support, completeness, citation accuracy, refusal or uncertainty behavior, and formatting. Also record latency, token usage, and failure rates for realistic workloads.

Keep the evaluation set versioned. When you change chunking, embeddings, search parameters, reranking, prompts, or models, compare the new pipeline with the previous one. A broader discussion of evaluation design is available in the prompt testing frameworks guide.

What to double-check

• Access control: Retrieval must not expose chunks a user is not permitted to see. Enforce permissions at indexing, filtering, or both, and test cross-user isolation.
• Freshness: A document update should trigger a predictable reindex or invalidation process. Avoid silently serving an old chunk after its source has changed.
• Duplicates: Repeated documents can cause the model to over-weight one statement and make citations appear more reliable than they are. Detect duplicates and near-duplicates where practical.
• Chunk provenance: Make sure every chunk can be traced back to a human-readable source location. A vector ID without usable provenance is not a useful citation.
• Context limits: Retrieving more text is not always better. Excess context can increase cost, latency, and distraction. The LLM context window guide offers a framework for planning context budgets.
• Prompt boundaries: Retrieved documents are data, not trusted instructions. Separate system rules, user requests, and retrieved content clearly in the prompt.
• Observability: Log enough information to reproduce a result while respecting privacy and retention requirements. Useful fields include pipeline version, query, filters, retrieved IDs, model version, latency, and error type.
• Unsupported certainty: A fluent answer is not proof of retrieval quality. Test whether each important claim is supported by the selected evidence.

Common mistakes

Indexing raw files without normalization. Layout artifacts, repeated headers, broken character encoding, and missing table context can make otherwise useful documents difficult to retrieve. Inspect extracted text before generating embeddings.

Treating chunk size as a fixed best practice. Chunking should reflect the structure and question patterns of the source material. Compare alternatives using a small evaluation set instead of selecting a size by convention.

Using vector similarity as the only relevance signal. Similar meaning does not guarantee that a passage contains the required fact. Hybrid search, filters, reranking, and explicit retrieval tests can improve precision.

Passing retrieved text into an unconstrained prompt. The model may follow instructions embedded in a document, confuse examples with facts, or blend sources without attribution. Label context clearly and define how conflicts are handled.

Generating citations after the answer is written. This encourages unsupported references. Design citations as part of the response contract and validate them against retrieved chunks.

Evaluating only successful examples. A reliable application must also handle missing evidence, conflicting documents, malformed files, permission failures, and ambiguous questions. Include these cases before release.

Ignoring knowledge-base operations. RAG is not finished when the first index is built. Without ownership, update detection, deletion handling, and monitoring, answer quality will drift as the source material changes.

When to revisit

Revisit this RAG checklist whenever the underlying inputs or workflow changes. At minimum, review it before a major planning or release cycle, after changing document sources, and whenever users report answers that are missing, stale, poorly cited, or overly confident.

Use this practical review sequence:

1. Sample recent questions and classify failures as ingestion, chunking, retrieval, generation, citation, permissions, or freshness problems.
2. Check whether source documents, schemas, access rules, embedding models, search settings, prompts, or model versions changed since the last evaluation.
3. Run the versioned evaluation set and compare retrieval quality, groundedness, citation accuracy, latency, and cost with the previous baseline.
4. Inspect a small number of traces manually, including at least one no-answer case and one permission-sensitive case.
5. Update the index, tests, prompts, or operational procedures only when the change addresses an observed failure or a clearly defined requirement.
6. Record the new pipeline version and the reason for the change so future regressions can be diagnosed.

A dependable retrieval-augmented generation system is maintained as a product, not treated as a one-time prompt experiment. Keep the source boundary clear, test retrieval independently from generation, preserve provenance, and make uncertainty visible. That discipline gives an LLM application a stronger foundation for useful answers as documents, tools, and workflows evolve.