Retrieval basics for LLM applications: chunking, passage identifiers and citing what was retrieved

What it is

Retrieval-augmented generation (RAG), as introduced by Lewis et al., combines a language model with a retriever over an external corpus so that answers can draw on documents rather than on the model's parameters alone; the paper motivates this with provenance and the ability to update knowledge. In application terms: documents are split into chunks, chunks are indexed (embeddings, BM25 or both), the top matches for a query are placed in the prompt, and the model answers with references to them.

Why it matters

Retrieval quality bounds answer quality: a chunk that lost its heading no longer says which product or which year it is about, and the retriever cannot find it. Answers without passage-level references cannot be checked by anyone, including the agent that produced them.

How to apply

• Chunk along the document's structure (headings, sections, table rows), not at a fixed character count that cuts sentences; keep chunks small enough that several fit into the prompt and large enough to be understood alone.
• Carry context with the chunk: title, section path, date, source URL. Anthropic's contextual-retrieval write-up describes prepending a chunk-specific explanatory context to each chunk before embedding and BM25 indexing, and reports on its test data sets a 49% reduction in the top-20-chunk retrieval failure rate, 67% when a reranking step is added.
• Give every chunk a stable identifier (document ID plus offset or section) and pass it into the prompt with the text; ask the model to cite identifiers, not titles from memory.
• Verify citations after generation: the cited identifier must be one of the retrieved chunks and any quoted text must occur in it. Provider citation features do this on the API side; the vendor citations documentation describes documents chunked into sentences and citations returned with cited_text and location indices pointing into the provided documents.
• Combine lexical and embedding retrieval when identifiers, codes or names matter; embeddings alone confuse near-identical part numbers.
• Evaluate retrieval separately from generation: for a set of questions, is the passage that contains the answer among the top results?

Pitfalls

Retrieving many chunks and hoping the model sorts them out; irrelevant passages crowd out the right one. Indexing stale copies without a re-indexing job. Treating retrieved text as instructions rather than data. Presenting an answer as sourced when the citation was generated from memory rather than from a retrieved passage.