RAG in One Picture

Indexing, offline: load documents, split them into chunks, embed each chunk, and store the vectors alongside the text and metadata. This runs when content changes, not when a user asks something.

Retrieval and generation, online: take the question, search the index, take the top k chunks, assemble a prompt that contains those chunks plus the question plus an instruction to answer only from the provided material, call the model, and return the answer with citations back to the chunks.

That is the whole architecture, and it has been the default for knowledge products since 2020 for good reasons: knowledge updates instantly when a document changes, answers can cite their sources, access control is enforceable at retrieval time, and nothing needs training.

The thing worth internalising is where quality actually comes from. If retrieval returns the wrong chunks, no model can save the answer. Most disappointing RAG systems are retrieval failures wearing a generation costume, and teams misdiagnose them because the symptom — a confidently wrong answer — looks like a model problem. When a RAG system underperforms, measure retrieval first, in isolation: did the chunk containing the answer appear in the top k? That number, not the answer quality, is what you improve.

The naive pipeline is a starting point, and the standard upgrades are all about getting better chunks in front of the model. Hybrid search adds keyword matching alongside vectors, because embeddings miss exact identifiers. Reranking retrieves generously then reorders with a cross-encoder that reads query and chunk together. Query rewriting turns "what about the second one?" into a standalone searchable question, which naive systems get badly wrong in multi-turn conversations. Contextual retrieval prepends a short document-level summary to each chunk before embedding, which substantially reduces the failure where a chunk is meaningless without its surroundings. Metadata filtering narrows by tenant, date, or document type before the semantic search runs.

The operational realities: the index is a system that needs freshness, re-embedding on model changes, and deletion that actually propagates; access control must be applied at retrieval, since anything retrieved is effectively disclosed; and retrieval adds latency before the first token, which shapes the UI.

RAG is the single most common architecture in AI products, so "design a system that answers questions over our documentation" is a near-guaranteed design prompt. The senior answer moves quickly past the basic diagram to chunking, hybrid search, reranking, evaluation, access control, and freshness — and names retrieval quality as the thing that determines success.

• Two pipelines: offline indexing (split, embed, store) and online retrieval-and-generation (search, assemble, answer, cite).
• Retrieval quality is the ceiling on answer quality; measure whether the right chunk appeared in the top k before touching the prompt or the model.
• RAG gives instant updates, citations, and access control — three things fine-tuning cannot give you at all.
• Hybrid search is close to mandatory: pure vector search misses identifiers, codes, and exact terms.
• Rerank a generous candidate set rather than trusting the first k from the vector store.
• Rewrite the query for standalone searchability in multi-turn conversations; raw follow-up questions retrieve badly.
• Contextual retrieval — prepending document-level context to each chunk before embedding — is a large, cheap accuracy win.
• Apply access control at retrieval time; a chunk in the prompt is disclosed regardless of what the answer says.
• Retrieval sits in the latency path before the first token, so budget it and consider showing a retrieval step in the UI.