AI answers are only as good as the text we pull out of documents. I benchmarked the extraction engines, built a fusion pipeline that beats our old one on tables and headings, and turned it into three tiers each customer can pick by speed and cost.
One extractor for everything: weak on tables and headings, and pages with thousands of drawing objects timed out after more than 75 seconds.
Three tiers (Lite, Auto, Agentic) chosen per customer, with dense pages routed to a cheaper engine that finishes in 0.02–1.4 seconds.
The problem
91% of our corpus needs OCR, and the extractor in production lost table structure and headings, which is exactly what answers need. Some pages, packed with vector drawings, simply timed out.
The idea
Measure on public ground truth and on our own documents, combine the engines that win on each kind of content, and let customers choose the trade-off instead of picking one for everyone.
How I did it
- Built a bake-off harness for Docling, MinerU, MarkItDown and others, scoring six things: speed, cost, accuracy, concurrency, warm-up and stalls.
- Shipped pipeline v1, a multi-engine fusion. On a 200-PDF public benchmark it scored 0.83 overall against 0.73 for the old extractor, and its table score beat every engine tested.
- Sent pages packed with thousands of drawing objects to a lighter inspector. Production pages that timed out after 75 seconds now finish in 0.02–1.4 seconds, and one OCR pass per page replaced nine (9.9 s to 0.8 s).
- Measured on our own cluster, not a research machine, which changed the answer: an int8 model that was 40% faster on the research box ran 2.6× slower on our CPUs, so it never shipped.
- Priced the tiers per page: a full production re-index of about 136,000 pages costs $4.70 on Lite and $86.48 on the deepest tier with OCR.
The impact
Customers now choose speed or fidelity per document type. Born-digital PDFs take 2 seconds instead of 81, and the deepest tier keeps tables and charts intact for the documents that need it. The finding that surprised everyone: compute cost isn’t the limit, wall-clock time is.
Benchmark on the hardware you’ll actually pay for. The research box made the wrong option look like a free 40%.