Scan or text?

Try it

The question

Every document pipeline hits the same fork. A digital PDF already has its text, and extracting it is cheap and exact. A scanned one is only images, and needs OCR, which is slow and sometimes wrong. Many files are both: a typed contract with a scanned signature page. The fork has to be decided before anything else happens, so it needs to be fast.

pdf-inspector, a Rust library from Firecrawl, does this check and quotes about 10 to 50 ms. The idea I started from asked for under 5 ms. So: can a deliberately minimal classifier make that decision in under 5 ms, as accurately, and still do it inside a web page?

How it works

• Read the structure, not the pixels. One pass indexes every object in the file, including those packed inside compressed object streams, and walks the page tree.
• A tiny interpreter per page. Each page's drawing instructions run through about a hundred lines of Rust that track three things: characters of text drawn, whether that text is invisible (how OCR layers hide themselves), and how much of the page images cover. It follows the transformation matrix so it knows each image's real size, and it looks inside nested forms and inline images too.
• A verdict per page. Text, scanned with a text layer, image only, or blank. A document needs OCR if any page is image only, and the answer says which pages.
• Small. About 700 lines of Rust with three dependencies: inflate (miniz_oxide), and MD5 and AES for files encrypted without a password. The WebAssembly build is 110 KB, 56 KB compressed.

How I tested it

• 375 real contracts. The source PDFs of ContractNLI (Koreeda and Manning, 2021; CC BY 4.0): public NDAs from company sites and filings. The ground truth is a slow, full parse of every page with PyMuPDF, recording its text and how much image covers it. The files are 370 digital, 2 scans with an OCR layer, and 3 mixed, each an executed contract with a scanned signature page. 18 are encrypted with an empty password, to restrict printing or editing.
• 150 constructed files. Real digital contracts can't test the other side much, so I built known cases from 60 randomly chosen ones:
  • 60 turned into pure scans (half stored as JPEG, like most scanners);
  • 60 made mixed by scanning the last page;
  • 30 scanned and then run through Tesseract OCR, giving an image with an invisible text layer, which doesn't need OCR again.
  An independent labeller agreed with the construction on all 150.
• A fair race. My classifier and pdf-inspector 1.24.0 (its routing call, with default sampling of 8 pages) ran in the same Rust process on the same bytes, 9 times each, and the median counts.

Results

This laptop's speed drifted by about two times between runs, so compare numbers within a run. In the head-to-head, both tools ran under identical conditions, and this classifier was faster on 523 of the 525 files, typically 2.3 times. In two back-to-back runs of this classifier alone, the median was 0.9 ms native and 1.2 ms in WebAssembly, with the 95th percentile at 2.6 ms and 3.6 ms. That puts WebAssembly at about 1.3 times native, comfortably under 5 ms either way.

Where the two disagree

Every one of pdf-inspector's 34 misses sends a file to OCR that doesn't need it. It never missed a file that did.

• 32 are scans that already have a text layer: the 30 Tesseract files and the 2 real ones. pdf-inspector calls them scanned. That's a difference in definition more than a bug: if you distrust an existing OCR layer, running OCR again is the right call. If you trust it, it's wasted work.
• 2 are digital contracts it called scanned. Both have extractable text on every page.

What didn't work

• The first perfect score was luck. The first version had no decryption and returned "unknown" for the 18 encrypted files, and its default of "no OCR" happened to be right for all of them. Counting unknown as undecided instead gave the honest score: 507 right, 18 undecided. pdf-inspector decrypted them all from the start.
• Decryption took two tries. RC4 worked first time. AES failed on every file, because the bytes read from each stream included the line break before its end marker, so the data wasn't a whole number of 16-byte blocks. Decrypting whole blocks only fixed it.
• One file needed a text encoding I'd left out. Its content was ASCII85-encoded as well as compressed, so it took 25 more lines.

Limits

• The real set has very few files that need OCR (3), so the scanned side rests mostly on constructed files, which are cleaner than real scans: no skew, no noise, no tiled strips.
• It doesn't handle AES-256 encryption, the LZW and RunLength encodings, or files that need a password. Those come back "can't tell".
• It counts characters roughly. The thresholds (40 characters; images over 30% or 80% of the page) mirror the ground-truth labeller's, which suits this test but may need tuning elsewhere.

Credits

• pdf-inspector by Firecrawl (MIT), for the idea and the comparison.
• ContractNLI (Koreeda and Manning, 2021), CC BY 4.0, for the real PDFs. The three samples in the demo are made-up documents, generated for this page.
• PyMuPDF for the ground truth, Tesseract for the OCR'd scans, and miniz_oxide and the RustCrypto crates inside the classifier.