RAG Is a Search Problem in a Trench Coat

1. The claim

If the passage containing the answer is not in the retrieved set, nothing downstream recovers it. Not a better reranker, not a larger model, not a more elaborate prompt. That is a hard ceiling, and it is set by decisions that happen before any embedding is computed.

The claim is therefore about attention, not about technology: the marginal engineering hour in most retrieval systems is better spent on chunking, query construction and evaluation than on the embedding model, and teams reliably spend it the other way because the model is the part with a leaderboard.

2. Where the losses actually happen

  document
     |  split            <- most loss enters here
     v
  chunk  (metadata kept? structure respected?)
     |  embed            <- the part everyone tunes
     v
  vector  ->  index      <- recall ceiling set here
     |
  query  (as typed? expanded? filtered?)
     |  retrieve k     <- k set once, never tuned
     v
  rank   (dense only? hybrid?)
     |
     v
  assemble context     <- order, truncation
     |
     v
  generation

2.1. Splitting. Fixed-size windows with a fixed overlap are the default in every starter template, and they cut through tables, headings, list items and the sentence that defines the term used in the next paragraph. A split that respects document structure (sections, table boundaries, list integrity) usually beats any model change, and it costs a day.

2.2. Metadata loss. The chunk arrives at the index having forgotten which document, which version, which section, which effective date and which tenant it came from. Every one of those is a filter that would have removed most of the false positives, and the loss is silent because the pipeline still returns plausible results.

2.3. The query is used raw. Users type fragments, misspellings and internal jargon. The corpus is written in formal prose. Dense retrieval is good at bridging that gap and lexical retrieval is good at exact identifiers, part numbers and error codes, which is precisely where dense retrieval fails and precisely what users paste in.

2.4. k was chosen once. Almost always 3, 5 or 10, on the first day, and it is never tuned against a measurement because there is no measurement.

3. The measurement that changes the conversation

Retrieval quality and answer quality are different numbers and must be measured separately. Almost every team measures the second and infers the first, which makes every regression ambiguous.

Once step 3 exists, the argument about embedding models resolves itself empirically in an afternoon, in either direction, which is the outcome this paper actually wants.

Figure 2. My ordering of expected marginal gain, not a measurement. It is a hypothesis this paper asks you to test with Section 3, and it is exactly the kind of ordinal claim that ought to carry a confidence value.

4. The order of work this argues for

1. Evaluation set with labelled answer locations. Nothing else is decidable without it.
2. Fix splitting so it respects document structure, and keep document, section, version and tenant on every chunk.
3. Add lexical retrieval alongside dense, and fuse the rankings. Identifiers and error codes stop disappearing.
4. Use the metadata as filters before ranking rather than as display fields after it.
5. Tune k and add reranking against the measurement from step 1.
6. Then, and only with a number to compare against, consider the embedding model.

5. The strongest objection

6. What this paper does not claim

Embedding models matter, and vector indexes are necessary. Saying otherwise about the second of those is the over-claim that retired 5.10, and I am not making it again. Figure 2 does not generalise; it is labelled a hypothesis because that is what it is.

Classical information retrieval does not solve this on its own either. The claim is only that the classical parts are where the unspent engineering hours are, and that a team without the measurement in Section 3 has no way to find out whether that is true of their corpus.