How RAG finds answers

A star stands for a question on a map of handbook passages; lines join it to its three nearest passages.

Imagine you work at Kettlewick, a small company that makes garden watering controllers. A colleague who joined three weeks ago types a simple question into the company’s chat assistant: “How much time off do new staff get?” The answer is sitting in the leave policy on the shared drive. The model behind the assistant has never seen that file, so on its own it can only guess, and a confident guess about someone’s holidays is worse than no answer at all.

Retrieval-augmented generation, or RAG, is the usual fix. Before the model answers, a search step finds the few passages most likely to hold the answer and places them in front of the model together with the question. The model answers from those passages and points to them. The interesting part is how a computer decides which passages are about a question when they don’t use the same words. The picture at the top is that search in miniature: the star is a question (it starts out on the one about time off), and the lines always join it to the three passages nearest in meaning. Drag the star and watch the lines jump.

By the end of this post you will know:

• why you can’t just paste every document into the model, and what chunks are for;
• how a piece of text becomes a position on a map of meaning, and how a search finds the chunks nearest to a question;
• how many chunks to fetch, what you pay for fetching more, and how teams measure whether a search works.

A model reads its input as one long stretch of text, and there is a limit to how much it can take in at once. That limit differs from model to model and keeps changing, so the figures in this post count in words and use made-up budgets, not the numbers of any real model.

Can’t we just paste the whole shared drive into the chat and let the model find the answer itself?

Let’s try it. Each square below stands for about 250 words from Kettlewick’s shared drive, in folder order: meeting notes, support tickets, wiki pages, the handbook and a few reports. The ringed square is the leave policy, which holds the answer. Kettlewick and every document in this post are fictional sample data, made up for teaching. Drag the slider to change how many words the model can read at once, and find the budget at which the ring turns green: squares that fit turn green, and the ones left out stay grey. Before you drag, guess how many words of other documents come before the leave policy.

This figure needs JavaScript. It shows that at a budget of 8,000 words only the first few meeting notes fit, and the leave policy, which holds the answer, is left out until the budget reaches about 51,000 words, where it is 0.3% of everything the model reads. Sending only the relevant chunks takes about 80 words.

The leave policy only fits after about 50,000 words of meeting notes, tickets and wiki pages, and then it is 0.3% of everything the model reads. Now press the second button, “Send only what is relevant”: that is what RAG sends instead, about 80 words, and every word of it is about leave.

Even when it all fits, more isn’t automatically better: researchers found that models use the start and the end of a long input more reliably than the middle.

At a budget of 60,000 words the whole leave policy fits. Roughly how much of what the model reads is the leave policy?

1. About 1 word in 40.
2. About 1 word in 400.
3. About 1 word in 4,000.

Show answer

The leave policy is 158 words, and 158 of 60,000 is about 0.3%: roughly 1 word in 400. That is the small ring in the budget figure, and it is why RAG sends the relevant chunks instead of the whole drive.

Not A: 1 in 40 would be 2.5%, a document of 1,500 words. There are 44 documents and the policy is one of the shortest, nowhere near that size.

Not C: 1 in 4,000 would make the policy only 15 words long. It has 158.

So instead of sending everything, RAG sends a little: only the parts that look relevant to this particular question. To do that, we first need parts.

A whole document is usually too big and too mixed to be useful as one piece: a question about sick days needs only one part of the leave policy. So before anything else, every document is split into smaller pieces called chunks.

The simplest way to chunk is to count: take the first 30 words, then the next 30, and so on. The cut falls wherever the count runs out, even in the middle of a sentence. Sizes here are in words, a stand-in for tokens, the words and word pieces that real systems count.

What if the answer ends up split between two chunks?

Let’s find out. Here is the leave policy, tinted in chunks. The underlined sentence, “Everyone receives 25 days of annual leave per calendar year.”, is the one that answers a question about annual leave, and the line above the text says whether it is whole in one chunk or cut. The dashed box, and the box below the text, show exactly what the search would hand to the model if it picked that chunk: nothing before it and nothing after it. Before you slide the chunk size, guess whether the sentence stays whole.

This figure needs JavaScript. It shows the leave policy cut into chunks of 30 words, which keeps the answer sentence whole in the first chunk. At 26 words that sentence is cut between two chunks, and cutting at paragraph breaks keeps it whole at every size.

It survives at some sizes and is cut at others, purely by where the boundary happens to fall. At 26 the first chunk ends with “annual” and the next one begins with “leave”, so a search that picks the first hands the model a sentence that stops halfway. Press “At paragraph breaks” and the sentence is whole at every size, which is why most teams cut that way. Many systems also let neighbouring chunks share a few words, called overlap: turn on “Share some words between chunks” at size 26 and raise it to 10, and the sentence is whole again. At other sizes it still is not, so overlap patches some cuts, not all.

Then smaller must be better! Tiny chunks would match a question really precisely, wouldn’t they?

It’s a reasonable guess, so push the size down to 6 and click the word “prorated”. On its own, that chunk says something is prorated from your start date, but not what: the words “annual leave” landed in an earlier chunk. Tiny chunks lose their context. Huge chunks have the opposite problem: they bury the one useful sentence among many others and eat into the budget. Most teams settle in between, and cut at natural breaks such as headings and paragraphs instead of in the middle of a sentence.

Your chunks are 26 words and the answer sentence is cut between two of them. Which change keeps it whole at every chunk size?

1. Cut at paragraph breaks, not at a word count.
2. Make the chunks bigger, so fewer cuts happen.
3. Share a few words between neighbouring chunks.

Show answer

A paragraph break never falls inside a sentence, so the sentence stays whole whatever the size: press “At paragraph breaks” in the figure and slide the size to see it. That is why most teams cut at natural breaks.

Not B: bigger chunks keep the sentence whole at some sizes and cut it at others (28 words cuts it, 30 keeps it), and they bury it in more text.

Not C: overlap patches some cuts and not others. At size 26 it helps from 6 shared words on, but at other sizes the sentence is still cut. Try it in the figure.

That is what the rest of this post does. Kettlewick’s five handbook documents, the leave policy, the expense policy, the onboarding guide, the security guideline and an FAQ for its Rill controller, are split at paragraph breaks into 23 chunks.

We have 23 chunks and a question. How does a computer decide which chunks are about the question? Matching words is one way, and we’ll come back to it. The idea that made RAG work well is different: turn each piece of text into a list of numbers that captures what it means, so that texts with similar meanings get similar numbers.

The program that does this is called an embedding model, and the list of numbers it produces is an embedding. You can think of an embedding as a position: texts about the same thing end up close together. Real embeddings have hundreds or thousands of numbers each, which nobody can draw, so the map below is a simplification. The 23 chunks were placed on a flat page by hand, roughly the way an embedding model groups them.

Here is that map of all 23 chunks. Point at a dot to read what it holds (with a keyboard, tab to the dots and use the arrow keys). Above the map is a card with a sentence that is not in the handbook. Guess where it belongs, by dragging the card onto the map or tapping a spot, and see where it really lands.

This figure needs JavaScript. It shows the map of the 23 handbook chunks and five made-up sentences, such as “Can I stay at home when I am unwell?”, each of which lands beside a chunk about the same thing without sharing a single word with it.

None of those sentences shares a word with the two chunks it landed beside: “unwell” sits next to “ill”, “cab fare” next to “taxis”. Position follows meaning, not spelling. The fifth one landed between two documents, because a question about who is in the building on a Sunday is about both office hours and visitors. And that is what makes the next figure possible: a question can be placed on the map too.

But who decides where each point goes?

The embedding model does. It is a neural network trained on large amounts of text so that passages used in similar ways end up with similar numbers; nobody writes rules like “holidays go near leave”. The grouping comes out of training, which is also why it isn’t perfect.

Here is the key move. The question gets an embedding too, from the same model, so it lands on the same map. Finding relevant chunks turns into a geometry problem: which points are closest to the question?

The star below is the question. Pick another question from the list, drag the star, or tap an empty spot, and change how many chunks to fetch. The lines join the question to its nearest chunks, the list ranks them by distance, and a ring marks the chunk that really holds the answer. Before you touch anything else, pick the E-214 question and guess whether three chunks are enough.

This figure needs JavaScript. It shows the question “How much time off do new staff get?” landing among the leave policy chunks; its three nearest chunks are about annual leave, carrying leave over and public holidays.

For “How much time off do new staff get?”, the nearest chunk is the one about annual leave, even though the question never says “annual” or “leave”: the search compares meanings, not spellings. The number of chunks to fetch is usually called top-k. For time off, one chunk is enough. For E-214, the answer is only the fourth nearest, so with k = 3 the model gets the right cluster of chunks and still misses the answer; raise k to 4 and it arrives.

Raising k has a price: the model has more text to read, and every extra chunk is one more that does not hold the answer. That trade is the whole story of top-k, and we will measure it properly at the end.

So why not fetch twenty and be safe?

Because every extra chunk costs something. It takes room in the prompt, and it adds text the model might follow instead of the answer; research on long inputs also found that models use the middle of them less reliably than the start or the end. A few well-chosen chunks beat a pile. Some teams do fetch twenty, but only to pick the best few from them, as we will see at the end.

What if my question lands somewhere empty, far away from everything?

Then the nearest chunks aren’t very near, and they may not contain the answer at all. A search always returns its top k, even when every one of them is a poor match. Drag the star into an empty corner: the lines turn dashed and the figure says that even the closest chunk is far away. Or pick the dog question: the nearest chunks, about visitors and office hours, sit inside the dashed circle, yet the figure says that no chunk in the handbook answers it. Near is not the same as answering. We’ll come back to what the model should do then.

For “What does E-214 mean?” the chunk with the answer is the fourth nearest. With k = 3, what does the model see?

1. The E-214 chunk, because the nearest chunks all come from the same FAQ.
2. Nothing, because the search fails when the answer is not in its top three.
3. Three FAQ chunks, and none of them is the one that explains E-214.

Show answer

With k = 3 the search returns the three nearest chunks: “The status light is red”, “Error codes” and “What is the Rill?”. All three come from the Rill FAQ and none explains E-214, which ranks fourth. In the question map, slide k to 4 and the answer joins them.

Not A: k counts the nearest chunks, not documents, so sitting in the same FAQ does not get a chunk in.

Not B: the search always returns its top k. It has no way to know that the answer was fourth.

On the map, closeness is measured with a ruler, but many systems measure the angle between two embeddings instead, which ignores how long they are; when every embedding has the same length, the two rankings agree exactly. Nothing later depends on it, so open the detour only if you want to see how the angle works.

Optional detour: measuring closeness by angle

Picture each embedding as an arrow from a shared starting point. Cosine similarity ignores how long the arrows are and asks only how closely they point the same way. In the figure below, arrow A is fixed. Drag the tip of arrow B, click the plot to send it somewhere, or use the sliders, and watch the angle and the score.

This figure needs JavaScript. It shows two arrows 39 degrees apart with a cosine similarity of 0.78.

When the arrows point the same way the score is 1, at a right angle it is 0, and pointing in opposite directions gives −1. Now change only the length of arrow B: the score doesn’t move, because only direction counts. The arithmetic is short. Multiply the matching numbers of the two arrows and add the results, then divide by the two lengths multiplied together. Real embeddings do the same sum over hundreds or thousands of numbers instead of two.

If you ever see “cosine” among a vector database’s settings, this is the score it means: the same sum, over many more numbers.

Meaning search sounds strictly better than matching words. It isn’t. The older approach, keyword search, scores a chunk by how many of the question’s words it contains. Real keyword engines are smarter about rare and common words, but the idea is the same. Keyword search fails in a different way from meaning search, and the two failures fit together neatly.

The figure runs one question three ways. Each row is a chunk, and the ring marks the one that really holds the answer. Before you click, guess which search will find the leave chunk. Then switch between keyword match, meaning match and both combined, try the second question, and watch whether the ringed chunk lands above the dashed line, where the chunks that go to the model end. Each row shows one score from 0 to 1. The scores are illustrative: the meaning scores come from the hand-placed map.

This figure needs JavaScript. It shows that keyword search misses the annual leave chunk for a question about time off, meaning search misses the chunk about code E-214, and combining both finds each answer.

For “How much time off do new staff get?”, keyword search misses the annual leave chunk entirely. The chunk says “annual leave”, the question says “time off”, and they share no word, so its score is 0 and no setting could ever retrieve it. “Your first week” comes first instead, because it happens to contain “new”, “staff” and “get”. Meaning search finds the leave chunks easily.

Now switch to “What does E-214 mean?” and the pattern reverses. Meaning search sees a question about a device problem and returns the chunks about warning lights and error codes. The exact code is a rare string that carries little weight in a meaning, so the chunk about E-214 itself comes fourth, just below the line.

Keyword search finds it at once, because “E-214” is right there in its text. Each search is strong exactly where the other is weak.

Combining the two, usually called hybrid search, catches both answers. Here the combined score is simply the average of the two, after putting the keyword counts on the same 0 to 1 scale. It isn’t perfect: for the time-off question, “Your first week” still comes first, carried by three lucky words.

Averaging is one way to combine. Another, reciprocal rank fusion, ignores the scores and gives each chunk points for its position in each list.

Which of these questions would keyword search handle best?

1. “What is the daily limit for eating out?”
2. “What does the E-1 error code mean?”
3. “Who looks after new starters here?”

Show answer

The one with a rare exact code. “E-1” sits in both the question and the “Error codes” chunk, so keyword search puts that chunk first. A question that rephrases the handbook shares no words with the chunk that answers it, which is why hybrid search adds meaning to the keywords.

Not A: the Meals chunk says “meals” and “per day”, the question says “eating out” and “daily limit”. Same idea, no shared word, so its keyword score is 0.

Not C: the chunks about a “buddy” answer it, but they share no word with “looks after”. Keyword search puts the Travel chunk first, on the strength of “after”.

We now have every piece. In the figure below each box is one step: the question, the three nearest chunks, the prompt that numbers them as sources, and the answer that cites them. The prompt and the answers are scripted examples, written by hand for this post, not output from a real model.

Can’t I just trust the model to say when it doesn’t know?

Pick a question, step through the pipeline (or press Play all steps), and look at the first line of the prompt: it is an instruction, drawn as a dashed box. At step 4, switch it off and see what changes.

This figure needs JavaScript. It shows the question, the three nearest leave policy chunks, a prompt that numbers them as sources, and a scripted answer that cites sources 1 and 3. For a question about bringing a dog to work, which the handbook does not answer, it shows a scripted answer that says the sources do not cover pets, and what an invented answer looks like when the instruction is switched off.

With a question the handbook answers, the instruction makes no difference. Each claim points to a source in its prompt that you can check: the 25 days and the 1 July example come from source 1, the public holidays from source 3. Source 2, about carrying days over, was fetched but not needed. Point at or focus a citation in the figure to light up its source.

Citations make an answer checkable, not automatically correct. A model can still misread a source, so a good interface shows the cited text right next to the answer.

With a question it does not answer, the instruction is the whole difference. The nearest chunks are about visitors and office hours, the best the search could do. With the instruction in the prompt, the model says the sources are silent; without it, nothing stops a fluent guess, and notice that the guess cites nothing. That guess is invented for this example, not output from a real model.

A search comes back with only weak matches. What decides whether the answer says “the sources do not cover this” or makes something up?

1. How small the distance numbers are for the chunks found.
2. The [1], [2] and [3] numbers that let the answer cite sources.
3. A prompt line saying to use only the sources and admit gaps.

Show answer

The prompt line does. In the pipeline figure, pick the dog question and switch the instruction off: the same weak sources now come with a fluent answer that cites nothing. Switch it on and the answer says the sources do not mention dogs. (The fluent answer is invented for the example; no real model was run.)

Not A: the distances are used by the search and never reach the model, so they cannot change what it says.

Not B: numbers make citing possible; they do not make an answer honest. A model can put a [1] on a claim that the source never made.

An answer ends with “[1]”. What does that citation let you check, and what doesn’t it promise?

Show answer

It tells you which source a claim came from, so you can read that source yourself: in the pipeline example, source 1 really does say 25 days. It doesn’t make the answer correct on its own. A model can still misread a source, which is why a good interface shows the cited text right next to the answer.

RAG moves much of the hard work from the model to the search. Wrong answers usually trace back to a few familiar problems.

• Stale documents. The search finds last year’s travel rules because nobody removed them. RAG is only as current as what you index, so re-index when files change and retire old versions.
• Bad chunking. A chunk can lose what it was about, or cut the answer in two. Cutting at headings and paragraphs, adding the document title to each chunk and a modest overlap all help.
• Retrieval misses. The answer exists but never reaches the top k, like the E-214 chunk when k is 3. The model can’t use what it never sees.

So how does a team know its search is finding the right chunks at all?

They measure it. A common habit is to collect real questions together with the chunks that answer them, then check how often those chunks show up in the top k. Here is that habit in miniature: twelve made-up test questions, each paired with the chunk that answers it. They are already run at k = 3. Slide k and see which ones the search finds, and what each extra chunk costs.

This figure needs JavaScript. Of twelve fictional test questions, at k = 3 keyword search finds the answer chunk for 8, meaning search for 10 and both combined for 11; at k = 5, meaning search and both combined find all 12.

At k = 3, both searches together find 11 of the 12. Raise k to 5 and meaning and both find all twelve, but the model now reads five chunks for every question, and only one of them holds the answer.

Keyword search stays at 8 however far you slide. A chunk that shares no word with the question scores zero, and no k can promote a zero.

The one question all three miss at k = 3 is about code E-101, which the handbook only mentions as “E-1”; meaning search finds it fifth. This is how a team picks k and decides whether the extra reading is worth it.

A team’s search finds the answer chunk in the top 3 for 11 of 12 test questions. What does that number not tell you?

1. Whether the final answers it gives are right.
2. Whether the search finds anything at all.
3. How many chunks were fetched for each question.

Show answer

The number says only that the right chunk made the top 3. Whether the model then reads that chunk correctly and gives a good answer is a separate measurement; the figure in this section measures retrieval alone.

Not B: 11 of 12 is a lot, so the search clearly finds answer chunks.

Not C: that is k, fixed by definition. The count was taken with three chunks per question.

With that measuring in place, a few techniques come up again and again:

• Hybrid search, as in the figure above, so exact names and codes aren’t lost.
• Reranking: fetch a generous top 20 or so, then let a slower, more careful model reorder them and keep the best few. That is the usual answer to lucky matches like “Your first week”.
• Metadata filters: tag chunks with facts such as document type, team or date, and filter before searching, so a question about current policy only searches current policies.

The name comes from Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (2020), which paired a neural retriever with a text generator. Searching with embeddings instead of keywords was shown to work well for question answering in “Dense Passage Retrieval for Open-Domain Question Answering” (Karpukhin and colleagues, 2020). The finding about information in the middle of long inputs is from “Lost in the Middle: How Language Models Use Long Contexts” (Liu and colleagues, 2023), and reciprocal rank fusion from Cormack, Clarke and Büttcher, “Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods” (SIGIR 2009).

This post leaves a lot out: how embedding models are trained, the approximate search indexes that keep lookups quick across millions of chunks, and how to judge the quality of answers rather than of retrieval alone. The map, the scores and the scripted answers are simplified for teaching.

RAG is a search step followed by a reading step. Split your documents into chunks that still make sense on their own, place them on a map of meaning, fetch the chunks nearest to the question (with keywords on the side, to catch exact names and codes), decide how many chunks you can afford to read, and ask the model to answer only from what was fetched, to cite it, and to say so when the sources fall short. Most of the quality comes from the search, so that is the first place to look when answers go wrong.

The same recipe also works with a local model: with Hati, the documents can stay on hardware you control and the prompt, retrieved chunks included, is processed by the model on your own machine, while any tools or integrations you connect may still send data elsewhere.