The challenge
Your tool reads loan files for a mid-sized lender’s credit operations team. In the demo, a cloud parser read 20 clean bank-generated PDFs almost perfectly. Then the pilot started, and the real files came in.
- 20 clean PDFs, generated by the bank
- Statement photos forwarded on WhatsApp
- A 14-page statement, one table across pages
- Rent agreements with a seal over the tenant’s name
- Handwritten forms in Marathi
Now operations staff re-check every field, so your tool has added a step instead of removing one. And on one statement the parser quietly corrected a total that didn’t add up, which hid a sign of tampering.
The review is in 5 weeks and nobody has defined success. So the sponsor will judge the pilot against the demo’s 98%.
What is really going on
It’s tempting to think you just need a better parser. But if you listen to the operations staff, they’re describing three separate problems, and a better parser fixes only part of each.
If they say “It cannot read the scans or the phone photos”, it’s usually about how the file was captured more than about the parser. You can see how much capture quality matters in a benchmark from the Allen Institute for AI, even for the best system [1].
To check, sort a week of files by how they arrived. If the errors bunch up in phone photos and forwarded files, capture is your problem. Then stop quoting one accuracy figure, and make a separate promise for each document group, like phone photos of bank statements.
If they say “The numbers looked fine, but the total was wrong”, that one is new. Older OCR left a blank when it couldn’t read something. AI parsers fill in a believable value instead. One benchmark author saw models often fixing a printed total that didn’t match its line items [2].
- Line items, added up
- Total A
- Total printed on the statement
- Total B, which doesn’t match
- Total the parser returned
- Total A, so the mismatch is gone
Nothing looks blank or garbled, so reviewers only catch these by doing the maths. Recalculate totals from the line items on a plain OCR copy, because a model that corrected the total will pass a check on its own output. And let fields that feed credit or fraud decisions skip review only when a check like that confirms them.
And if they say “Typed English forms are fine. The handwritten and Marathi ones fail”, handwriting and script are each a problem of their own. A June 2026 preprint by one researcher found systems scored close to each other on clean Devanagari text, the script Marathi and Hindi use. On real scans they were up to 76 points apart out of 100 [3].
There’s no independent benchmark for Indian business documents yet, so your own test is the only evidence you’ll have. Split the error rate by script, and by typed or handwritten, because the typed English number hides the rest. Then treat each script, and handwriting too, as its own document group, and test it on real scans before you promise it.
How accurate is the parser?
Which documents, captured how, can pass without a person, and how would we know if they went wrong?
The mistake most teams make
The usual move is a parser contest on the customer’s sample folder, then one accuracy figure for everything. But that folder holds the files someone chose to show you. The contest ends up picking the best engine for documents that were never the problem.
Then the first phone photos come in and the single figure stops holding up. Staff stop trusting the output, and the savings disappear even on files the tool reads well. If you want a contest, run it on the worst real files.
The options you have
Most teams combine two or three of these, starting with the ones that make the problem smaller.
| Option | Best when | What it costs you | Watch out |
|---|---|---|---|
| Promise only the documents that already work | A few document types carry most of the volume. | The launch covers less than the demo implied, and the sponsor has to accept that and explain it. | It fails if the pain sits in the files you leave out, like phone photos from rural branches. |
| Get the data without reading the document | Your customers already use channels that share structured records, or can ask their applicants to. | Integration and consent flows take engineering time, and every extra step can lose applicants. | It misses files outside those channels, like small suppliers’ paper invoices. |
| Fix the file where it is captured | Your product or the customer’s field staff control the upload, as in a lending app. | Uploading takes extra steps, and the capture screen needs its own design work. | It does nothing for files forwarded by email or chat, with nobody there to retake them. |
| Send each document type to the engine that reads it best | Several different document types each carry real volume. | Every outside engine that sees customer documents needs infosec and risk approval, which can take months in a bank, so start before the test. | If the first step misreads a document’s type, the page goes to the wrong engine and the error carries through every later step [4]. |
| Make human review part of the product | Documents vary a lot and one wrong field is expensive, as in credit, insurance claims or KYC. | Reviewer time stays in the running cost, so the business case should promise time saved per file, not fewer staff. | It fails if you route on confidence scores you haven’t tested on your own files. |
PM tools to put in place
Start with a document mix grid. The rows are document types and the columns are how they arrive, from a system-generated PDF to a phone photo. Each box gets its share of volume and one decision. You either fetch the data elsewhere, pass it without review, send it to a reviewer, or ask for a retake.
| Document type | System PDF | Office scan | Phone photo or forwarded |
|---|---|---|---|
| Bank statements | Fetch elsewhere | Reviewer | Retake |
| Rent agreements | Pass | Reviewer | Retake |
| Typed application forms | Pass | Pass | Reviewer |
| Handwritten Marathi forms | – | Reviewer | Reviewer |
Then build a set of the worst real files. Ask the operations staff, not the sponsor, for the 50 files they struggle with most, and agree the correct value of every critical field. Keep them as whole documents, not single pages [6]. At the review, a document group passes only if it clears this set and the weekly audit.
And give reviewers a screen that shows where each value came from, and flags a total that doesn’t add up instead of fixing it.
- Account holder
- Passed on its own
- Monthly income
- Confirmed by a person
- Closing balance
- Possible tampering · doesn’t match the line items
Your first three steps
- Take the last two weeks of real pilot files, not the sample folder, and sort them into the grid. You’ll see where the volume and the errors sit before anyone starts arguing about engines.
- Run the current parser on the operations lead’s 50 worst files, and mark each error as blank or confidently wrong. The confident ones show which fields need an independent check before the review.
- Book 30 minutes with the sponsor and a credit risk lead, plus your model risk contact if you’re in a regulated firm. Agree which fields are critical, like monthly income, the error limit on each, and the evidence validation will need.
What to measure, and how
Check these every week, in this order.
Are operations staff relying on the output instead of re-checking it?
While staff re-check every field, better accuracy doesn’t save them anything. For each document group, sample how many minutes reviewers spend per file, and count how many auto-passed fields they edit anyway.
If the minutes stay flat, sit with two reviewers for an afternoon. Learn which fields they don’t trust and why, then change what the screen shows or send those documents back to full review.
Are wrong values reaching decisions without anyone noticing?
A confident wrong value does more damage than a blank, and nobody sees it unless they look. Each week, have someone re-type the critical fields from a random sample of auto-passed files, without seeing the tool’s values. Size the sample so you’d expect about 10 errors at your agreed limit. That’s about 5,000 fields at 0.2%.
If any document group goes over the limit, send it back to full review the same day, and fix the cause before it passes unreviewed again.
Real examples
Alan
Alan is a French health insurer. Its team built an AI pipeline to read claim documents like prescriptions and invoices, and it got to 70% automation.
Then the model started putting values on records that weren’t on the page.
It turned out the problem was in the examples it learned from. When operators corrected a record, they sometimes filled in what they knew from elsewhere, like another document in the same claim, or a procedure’s price they’d looked up online. Those corrected records became the examples the model was shown. As the team put it, “the examples taught it that those values should be there” [4].
The model was learning to read the page and to add what the business knows, both at once.
Alan is pulling them apart. The examples now show only what’s on the page, and the outside facts get added in a separate step that doesn’t involve the model. When the team wrote about it in March 2026, that fix was still in progress.
The lessonHave reviewers record what the page says separately from what’s true, and train the parser only on what the page says.
Sources
- olmOCR README with olmOCR-Bench results — Allen Institute for AI, olmOCR v0.4.0, Oct 2025, fetched 28 Sep 2026. https://raw.githubusercontent.com/allenai/olmocr/main/README.md
- Show HN: Benchmarking VLMs vs. Traditional OCR — OmniAI, Hacker News, Feb 2025. https://news.ycombinator.com/item?id=43118514
- Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study — Aditya Pratap Singh, arXiv preprint, Jun 2026. https://arxiv.org/abs/2606.29213
- Lessons from Running an LLM Document Processing Pipeline in Production — Othman Moumni Abdou, Alan Product and Technical Blog, Mar 2026. https://medium.com/alan/lessons-from-running-an-llm-document-processing-pipeline-in-production-33d87f99cdb1
- Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction — Roy et al. (AWS), arXiv, Aug 2026. https://arxiv.org/html/2608.01792
- MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing — Zhou et al., arXiv, May 2026. https://arxiv.org/html/2605.22100v1
- Automation at all costs: the fallacy of business value with straight-through processing — Jordan Hanley, Hyperscience, Nov 2024. https://www.hyperscience.ai/blog/automation-at-all-costs-the-fallacy-of-business-value-with-straight-through-processing/