Why an AI model can't answer from your company's documents on its own, the eight steps RAG uses to fix that, and where each one quietly goes wrong.
Ask an AI model what your company’s travel policy says and it will guess, or admit it doesn’t know. It has never seen your travel policy. RAG is the usual way around that, and it’s the idea behind the AI answers I work on at BHyve. Here’s what the model is missing, the eight steps that fill the gap, and why each one can quietly go wrong.
A model only knows what it read in training
A large language model (LLM) is a program trained on a huge amount of text to predict which words come next. That training is where it picked up grammar, general facts and how to work through a question. It happens before you ever use the model, on text gathered up to a cutoff date.
Your company’s documents were not in that text. The contract signed last month, the HR policy in SharePoint, the thread in someone’s Outlook: the model has never seen any of them. Ask about them and it can only guess, and a model’s guess often sounds as sure of itself as a real answer.
Why not paste every document into the question
A model can read text you send along with your question. So why not send everything?
Because there’s far too much. A single company’s knowledge base can run to well over a hundred thousand chunks, the short passages that search works on. No model can usefully read that in one go. You also pay for every token you send (a token is a piece of a word, the unit models read and bill by), so sending a company’s whole library with every question would be slow and expensive even if it fit.
You want the model to read only the few pages that matter for this question. Finding those pages is the whole job.
RAG in one sentence
Retrieval-augmented generation means: find the few passages that answer the question (that’s the retrieval), then give them to the model with the question so it writes the answer from them (that’s the generation).
The model still does the writing. It just stops writing from memory.
It works like an open-book exam
Think of an open-book exam. You aren’t expected to memorise the textbook. You need two things: a good index, prepared before the exam, and the habit of using it well under time pressure. Look up the topic, turn to the right pages, read them, then write your answer from what’s on the page.
RAG has the same two halves. The first builds the index, once for each document. The second runs every time someone asks a question.
A plain chatbot is the closed-book version. It can only write what it remembers.
The eight steps
At BHyve, the textbook is whatever a company connects: its own knowledge base and tools like SharePoint, Gmail and Outlook. That’s PDFs, scans and email threads, not one tidy folder, and every step below has to cope with all of them.
1. Extract the text
Files aren’t text. A PDF mostly stores where to draw letters and lines on a page, not which lines make a heading or a table. A scanned page is just a picture. So the first step pulls clean text out of every file and tries to keep tables and headings intact.
Scans need OCR (optical character recognition: software that reads letters off an image). This step sets a ceiling on everything after it. If a table comes out as a jumble of numbers, nothing later can put it back together, and the model will answer from the jumble.
For us it decides more than most steps do. 91% of our corpus needs OCR, and the extractor we started with lost table structure and headings, which are exactly what answers need. That’s why we measured extraction engines on our own documents before choosing: how we compared them.
2. Chunk
A chunk is a short passage cut from a document, often a few paragraphs long. Search works on chunks rather than whole files because a 40-page policy might answer your question in one paragraph. Small chunks let search point at that paragraph, and they keep what the model has to read short.
There’s a balance. Cut too small and a chunk loses its context, so “it” no longer says what “it” is. Cut too big and the one useful sentence gets buried.
3. Embed
An embedding is a list of numbers that stands for the meaning of a piece of text. An embedding model turns every chunk into one. Texts that mean similar things get similar numbers, even when they share no words: “annual leave” and “vacation days” end up close together.
This is what lets search work by meaning instead of exact wording. The embedding model decides what counts as “similar”, so a model that does well on someone else’s benchmark can still do badly on your documents. That’s why I chose ours on our own test set, with a judge that didn’t know which model produced which result: choosing an embedding model.
4. Store
The chunks and their embeddings go into a vector store: a database built to find the embeddings closest to a given one, fast. This is the index from the exam.
Two things matter here. The first is that search has to stay fast as a company adds documents. Our first setup couldn’t use its vector index for the way the queries were written, so each question was compared against every chunk the user could see, and search slowed as customers grew. After we moved to LanceDB, query time stopped depending on how many documents a company has. That’s in why we left Postgres for LanceDB.
The second is knowing who may see what. The HR folder shouldn’t turn up in an intern’s answer, so ours filters by each user’s access on every search.
That’s the first half. It runs when a document is added or changed, not when someone asks.
5. Search
Now someone asks a question. The question gets embedded with the same model, and the store returns the chunks whose embeddings sit closest to it. This is vector search, and it’s good at meaning.
It’s weaker at exact names, product codes and IDs. An embedding captures what a passage means, and a name or a code doesn’t mean much, so two invoice numbers one digit apart can look almost the same. That’s why many systems also run keyword search, which matches the actual words, and combine the two.
We saw this when I scored each search on its own against a test set. On the overall total, keyword search looked like the weaker one. On name-heavy questions it rescued 5 of 14. How you merge the two lists matters as much as running both, which is the subject of why rank fusion flattened our scores.
6. Pick
Search returns a ranked list, usually longer than you want the model to read. Picking means keeping the few results worth reading and dropping the rest.
Pick too few and the right document gets cut. Pick too many and the model wades through noise, and you pay for every token of it.
7. Read
The model reads what was picked and finds the parts that answer the question. This is where the model’s own skill shows: following a table, noticing two documents disagree, or spotting that the answer simply isn’t there.
A bigger, pricier model isn’t automatically a better reader. In our tests a cheaper reader model made fewer mistakes than a pricier one. We only saw that because the read stage had a score of its own, instead of being folded into the final answer.
8. Answer
Finally the model writes the answer from what it read. A good answer makes it easy to check where it came from, and a good system says “I couldn’t find that” instead of guessing.
Every answer also has a cost, set mostly by how much text the model is sent to read. That’s one more reason to pick a few good passages instead of many. What an AI answer costs breaks it down.
A fluent answer can still be wrong
Each step can fail quietly while the final answer still reads well. The text came out garbled. The right chunk was never found. It was found but not picked. It was picked and misread.
That’s the trap with RAG. Our system once counted an answer as a success even when it was wrong. A check that only looks at the end can’t tell a right answer from a confident wrong one, and it can’t say which step failed.
The fix is to write down the right document for each test question and score every stage against it. Then the first stage that fails is the one to fix. Once we did that, most of the fixes were obvious. Smart mode, one of our answer modes, went from 19 to 24 correct answers on 47 hand-graded questions. The method is in how to test a RAG system, and the numbers behind this series are in the AI answers case study.
When RAG fits, and when it doesn’t
RAG is a good fit when:
- The answers live in documents, and those documents change.
- The documents are private, so no model was trained on them.
- People need to check where an answer came from.
- Different people are allowed to see different documents.
Look elsewhere when:
- The question is a calculation over data, like total sales by region last quarter. That’s a database query. Search returns passages, not sums.
- Everything fits comfortably in one prompt. If your whole knowledge base is a few pages, just send it with the question.
- You want to change how the model writes rather than what it knows. That’s closer to fine-tuning.
The whole flow on one card
- Extract: get clean text out of every file, tables and headings intact.
- Chunk: cut it into passages small enough to point at.
- Embed: turn each passage into numbers that capture its meaning.
- Store: keep them in a vector store that knows who can see what.
- Search: find the passages closest to the question, by meaning and by keyword.
- Pick: keep the few worth reading.
- Read: let the model find the answer in them.
- Answer: write it from those passages, or say it isn’t there.
Quick answers
- What is RAG in AI?
- RAG, short for retrieval-augmented generation, makes an AI model answer from your own documents. The system first searches the documents for passages that match the question, then gives those passages to the model so it writes the answer from them instead of from memory.
- Why doesn't an AI model know my company's documents?
- A language model only knows the text it was trained on, and your private files were never part of it. RAG works around this by finding the relevant passages at question time and handing them to the model.
- What is the difference between RAG and fine-tuning?
- RAG looks up your documents every time someone asks, so answers stay current and can point to a source. Fine-tuning trains the model further, which suits changing how it writes more than keeping up with documents that change.
- What is chunking in RAG?
- Chunking splits each document into short passages before it is indexed, so search can return the one passage that answers a question instead of a whole file.
- What is a vector database in RAG?
- A vector database stores each chunk of text with its embedding, a list of numbers that stands for its meaning, and quickly finds the chunks whose numbers are closest to the question's.