A RAG answer can't be better than the text extraction hands it. Why, how to run a fair bake-off on your own documents, and what combining engines did.
When an AI answer about a document is wrong, the model usually gets the blame. Often the mistake happened much earlier, when the document was turned into text.
At BHyve, 91% of our corpus needs OCR, and the extractor we ran in production lost table structure and headings. Some pages, packed with vector drawings, timed out altogether. This is why extraction caps answer quality, how to test extraction engines fairly on your own documents, and what that test led us to.
Extraction sets the ceiling
RAG (retrieval-augmented generation) means the AI first searches your documents for passages that match a question, then writes its answer from those passages. If that’s new, start with what RAG is.
The steps run in a line. A document is extracted into text, the text is cut into chunks, the chunks are indexed for search, and the best ones go to the model. Each step only sees what the step before it produced. So a mistake in extraction isn’t one error among many. It’s a ceiling. No chunking, search or prompt can bring back structure that was never extracted.
Two losses do the most harm:
- Flattened tables. Read line by line, a table becomes a run of numbers with no column names. The right cell can reach the model with nothing to say which row or column it belongs to. That’s how a RAG system ends up confidently wrong.
- Lost headings. A heading says what the text under it is about. Many chunkers split on headings, and a chunk that keeps its heading is easier to find. Without it, a paragraph about refunds looks like any other paragraph.
OCR (optical character recognition) reads text from an image of a page. A born-digital PDF, made directly by software, already has a text layer and doesn’t need it. A scanned PDF is a stack of images, so it does. Every OCR page is another chance for text and structure to come out wrong.
Why one accuracy score hides the problem
Extraction comparisons usually give one accuracy number per engine. That number averages over everything on the page, and on most documents plain paragraphs make up most of the text. Most engines read those well. Tables and headings are a small share of the text and a large share of what answers depend on. An engine can score well overall and still flatten every table.
So the principle for a useful test: score the parts your answers depend on, on pages like yours, under the conditions you’ll run.
Three engines, three different bets
Each project describes itself roughly like this:
- Docling, started at IBM Research, runs a layout model to find paragraphs, headings, tables and figures, then a table-structure model to rebuild rows and columns. OCR is built in.
- MinerU, from OpenDataLab, runs separate models for layout, OCR, formulas and tables, and aims at complex layouts such as scientific papers.
- MarkItDown, from Microsoft, is a lightweight converter that turns many file types into Markdown for language models. For PDFs it reads the text layer that’s already there, and OCR needs an add-on.
They’re built for different documents, so which one wins depends on yours. That’s what the test is for.
How to run a fair bake-off
A bake-off is one test bench that runs every engine on the same pages and scores them the same way. I built one for Docling, MinerU, MarkItDown and others. This is how to set one up.
- Use two sets of pages. A public set with ground truth, the known correct output for each page, gives scores you can repeat and compare. A sample of your own documents shows the pages you really get. Neither is enough alone.
- Count how many of your pages need OCR. It decides which engines are even in the running. For our corpus it was 91%.
- Run every engine on exactly the same pages, with the same timeout, and keep the outputs so you can read the failures, not just the scores.
- Score tables and headings on their own, next to the overall number.
- Measure what it takes to run an engine, not only how accurate it is. We scored six things:
| Measure | Why it matters |
|---|---|
| Speed | How long a page or a document takes |
| Cost | The compute each page needs |
| Accuracy | How close the output is to ground truth |
| Concurrency | Whether it holds up with many documents at once |
| Warm-up | Time to load models, paid every time a worker starts |
| Stalls | Pages that hang and hold up the whole document they sit in |
- Look at the slowest pages, not the average. A page that never finishes matters more than a slightly better mean. Set a timeout and count how often it fires.
- Run it on the hardware you’ll pay for. Speed results can flip between machines. Benchmark on the hardware you pay for is the story of one that did.
- Don’t assume one engine has to do everything. An engine that loses overall can still win on one kind of content.
Take each kind of content from the engine that wins on it
That last step became our design. Different engines win on different kinds of content, so instead of picking one, we combined them. Pipeline v1 is a multi-engine fusion: each kind of content, such as body text, headings or tables, comes from the engine that handles it best.
What it changed
On a 200-PDF public benchmark, pipeline v1 scored 0.83 overall against 0.73 for the old extractor, and its table score beat every single engine we tested. The two weak spots moved furthest:
| Measure | Old extractor | Pipeline v1 | Change |
|---|---|---|---|
| Table accuracy | 0.401 | 0.665 | +66% |
| Heading accuracy | 0.412 | 0.722 | +75% |
Pages that came back blank, across 4,039 test pages, fell from 27% to 0.1%.
Those are the parts an answer leans on. A number that keeps its column and a paragraph that keeps its heading reach the model with their context attached. The speed side of the test turned into a product choice too, covered in pricing document processing per page.
The lesson
Your extractor sets the ceiling for every answer built on it. Test it on your own pages, score the parts answers depend on, and measure on the machines you’ll run.
The numbers are in the extraction case study.
Quick answers
- What is the difference between Docling, MinerU and MarkItDown?
- Docling uses a layout model and a table-structure model to rebuild a document's structure. MinerU runs separate layout, OCR, formula and table models aimed at complex layouts. MarkItDown is a lightweight converter that reads a PDF's existing text layer into Markdown and needs an add-on for OCR.
- Why does PDF extraction quality matter for RAG?
- A RAG system can only answer from the text it retrieves. If extraction flattens a table or drops a heading, every later step works on broken text, and no model can put the structure back.
- How do you benchmark PDF extraction tools?
- Run every tool on the same pages, from a public set with ground truth and a sample of your own documents. Score tables and headings separately, and measure speed, cost, concurrency, warm-up and stalls on the hardware you will run in production.
- Do PDFs need OCR?
- Only pages without a text layer, such as scans and photos. A born-digital PDF, made directly by software, already contains text that can be read without OCR.