CAG vs RAG: When to Just Load the Whole Document

RAG, short for retrieval augmented generation, is the approach almost every team reaches for when they need a language model to answer questions over a private document corpus. The pattern is familiar now. You chunk your documents into segments, embed those segments into a vector database, embed the incoming question, search for the nearest matching chunks, and feed those chunks plus the question into the model's prompt. It works. Engineers have been shipping it for two years. There are dozens of libraries that make the setup faster, and the pattern is documented exhaustively.

What is documented far less is where it quietly breaks. RAG makes a structural promise at the retrieval step: that the chunks you retrieve contain the information the model needs to answer the question correctly. Under controlled conditions, with clean documents and well-formed queries, that promise usually holds. Under real conditions, it breaks more often than most teams realize, and because the failure is not loud, it accumulates invisibly until someone is embarrassed.

The alternative is called CAG, or cache augmented generation. Instead of searching for the relevant pieces, you load the entire document into the model's context window. No vector database. No embedding pipeline. No retrieval step. The question of whether you found the right chunk disappears, because the entire document is present. If the answer exists anywhere in the source material, the model has it.

This alternative was not practical two years ago. It is practical now, and the reasons why are worth understanding before deciding which approach your next project should use.

The Promise RAG Makes and Quietly Breaks

The retrieval step in RAG is doing something structurally ambitious. It is claiming to identify, from a potentially enormous corpus of text, the specific passages that contain the information needed to answer a question correctly. Embedding similarity search is good at this in many cases. It is not reliably good at it in the cases that matter most.

Consider a software development team using a RAG system over their internal codebase documentation to help developers find implementation examples. A developer asks how to handle authentication token refresh in the project's API client. The relevant code example spans three files. The chunking step divided each file into 500-character segments. The authentication logic is split across multiple chunks, and the critical dependency that connects the token refresh function to the error handling wrapper is in a chunk that did not rank highly enough in the similarity search to be included in the retrieval set.

The model receives a partial picture. It produces an answer that is technically coherent, draws on the fragments it received, and is subtly wrong in a way that costs the developer two hours to debug. The failure does not look like a failure at the point of delivery. It looks like a confident, well-formatted response. The error surfaces later, at a desk, in a codebase, when someone has already acted on the answer.

This is the structural problem with RAG that no amount of chunk-size tuning fully solves. The retrieval step cannot see the answer it is trying to find. It ranks chunks by embedding similarity to the question, which is a proxy for relevance, not a direct measure of it. In domains where the answer depends on context distributed across a document rather than concentrated in a single passage, that proxy breaks down. Legal agreements, technical specifications, medical protocols, and complex financial instruments all have this property. The answer to almost any question about them depends on how multiple clauses, definitions, and exceptions interact, and those interactions are exactly what chunking and retrieval cannot guarantee to preserve.

Why Loading the Whole Document Is More Honest About What It Can Guarantee

CAG's core claim is simpler and more defensible: if the entire document is in context, and the model's recall across that context is reliable, then the completeness problem disappears. The model is not working from a retrieval selection. It is working from the full source.

Two years ago this was impractical for any real-world corpus. The state-of-the-art context window at that point was roughly 4,000 tokens, barely large enough for a short PDF. Today most flagship models handle between 100,000 and 200,000 tokens, and Google's Gemini supports up to 2 million tokens. Two million tokens is approximately 1.5 million words, which is larger than most novels. The structural barrier that made CAG impractical has been removed.

The accuracy question is also answered with real data. Gemini 1.5 Pro showed near-perfect recall on the needle-in-a-haystack benchmark across up to 1 million tokens: a test where a specific fact is hidden at a random position in a very long document and the model is asked to locate it. Near-perfect recall across that range means the model can find relevant information anywhere in a document loaded into its context, which is the baseline capability CAG needs to be useful.

The cost objection, which was also real two years ago, has largely been resolved by the most recent generation of long-context models. GPT-4o input costs approximately $2.50 per million tokens. Gemini 2.0 Flash input costs approximately $0.10 per million tokens, a reduction of roughly 96 percent. At that price point, loading a full developer documentation site into a single context costs around six hundredths of a cent. A team that balked at CAG because of API costs in 2023 is working with different math now.

The code simplicity argument is also real and should not be dismissed as a soft benefit. Building a RAG pipeline means managing an embedding model, a vector database, a chunking strategy, a retrieval configuration, and the prompt assembly logic that combines retrieved chunks with the query. Building a CAG implementation means passing a document and a question to a long-context API endpoint. That is not a minor difference in engineering overhead. For a team that is maintaining the system over time, the reduced surface area of CAG means fewer things that can drift, fail, or require reconfiguration when the underlying document corpus changes.

What a Professional Services Firm Discovers the Hard Way

Take a mid-size law firm that built a RAG system over its contract library eighteen months ago. The system was used by associates to quickly look up relevant precedents in prior agreements, standard clause language, and exception provisions across a corpus of several thousand documents. The build was clean, the retrieval was tuned, and the system reduced the time associates spent on routine clause research by a meaningful amount.

Six months in, a senior associate noticed that the system was consistently giving incomplete answers on questions involving cross-reference clauses, the kinds of provisions where one section of an agreement modifies the interpretation of another section several pages away. The retrieval step was finding the primary clause, but it was not reliably including the cross-reference modifier in the same retrieval set. Associates were getting answers that were accurate about the primary clause and silent about the modifier, which in a legal context is not partial accuracy. It is incorrect.

The fix required reconfiguring the chunking strategy to preserve clause relationships, re-embedding the entire corpus, and retuning the retrieval parameters. The work took several weeks and produced an improved but not fully resolved outcome, because the fundamental problem was not the chunk size. It was the retrieval step's inability to guarantee semantic completeness for documents where meaning depends on inter-clause relationships distributed across dozens of pages.

A CAG implementation over the same corpus, using a long-context model at current pricing, would cost approximately one to two dollars in API calls per day of active associate usage, based on the volume and length of the agreements being queried. It would eliminate the inter-clause completeness problem entirely, because every agreement queried would be present in full in the context window, cross-references and all. The firm would replace an engineering maintenance burden, a recurrent accuracy problem, and the reputational risk of an associate acting on an incomplete answer, with a simpler pipeline that costs less than the coffee budget to run.

For corpora too large to fit in a single context window, the approach still benefits from thinking like CAG before falling back to RAG. A corpus of 10,000 contracts is too large to load at once, but a single contract, or a small set of contracts relevant to a specific matter, is not. The right architecture in that case is a lightweight metadata search that identifies the relevant documents first, a step that does not require semantic retrieval and can use simple keyword or filename matching, followed by loading those specific documents in full. The result is a CAG approach applied to a filtered subset rather than a RAG approach applied to embedded chunks.

The central insight that drives CAG adoption at this stage of the field is not that RAG is wrong. It is that RAG is more complex than it appears and less reliable than its proponents usually acknowledge, particularly in the high-stakes domains where the accuracy requirement is strictest. Feeding the full document is more honest about what the system can actually guarantee. When the context window is large enough and the model's recall is reliable, that honesty is also the better-performing choice. The question is no longer whether long-context approaches are feasible. It is whether teams are willing to reexamine assumptions they made under constraints that no longer exist. ## The Adoption Curve and Who Gets There First

The teams that adopt CAG early are not primarily the ones with the most engineering sophistication. They are the ones with the most concrete experience of RAG's failure modes. A team that has never been embarrassed by a retrieval failure has no particular reason to question the architecture. A team that watched an associate act on a partial answer pulled from a badly chunked contract has a very specific reason to ask whether loading the full document might have been simpler and more reliable from the start.

The adoption path for most businesses does not require abandoning RAG entirely. It requires developing the discipline to ask, for each new AI document task, whether retrieval is actually necessary or whether the corpus is small enough to load in full. A company with twenty vendor contracts and a 200,000-token context window does not need a vector database for those twenty contracts. A company with 200,000 vendor contracts does. The threshold question is not which architecture is more sophisticated. It is which architecture is appropriate for the actual corpus size and the actual stakes of a retrieval failure.

For businesses where long-context models are now accessible, the practical starting point is not a technical migration. It is an audit. Walk through every RAG implementation currently in production and answer three questions: How large is the corpus? How expensive would it be to load the relevant documents in full? And how often do users report answers that feel incomplete or contextually disconnected? If the corpus is small, the cost is low, and the complaints are real, the argument for CAG is already there.

The economics make the audit worth doing now rather than later. At the pricing trajectory of the last two years, the cost to query a 200,000-token context has dropped by more than 90 percent. That trajectory continues. A company that runs the audit today and identifies three RAG implementations where full-document loading is now affordable will pay a different price for the infrastructure in six months than they would have paid when those implementations were originally built. Waiting for the technology to mature further is a reasonable choice. It is also worth knowing that the window to build with the current cost structure is not permanent.

The systems built on RAG in 2022 and 2023 were built under constraints that no longer exist. Revisiting them is not an admission that the original architecture was wrong under the conditions of the time. It is a practical response to conditions that have changed in a direction that changes the math. The teams that treat their existing AI infrastructure as permanently fixed will find the gap between their results and their competitors' results widening at a pace that is uncomfortable to explain.