What is this file?

Try it

The question

Before a pipeline can read an upload, it has to route it: PDF to one parser, spreadsheet to another, old binary Word files to a third. Most systems route on the extension. The costly failures are the ones the extension hides: a password-protected .docx that no parser can open, a spreadsheet saved as .pdf, a download that stopped halfway.

Checking the first few bytes, the "magic number", is a solved problem, and it isn't enough. Every modern Office file starts the same way as any zip, and every old Office file, Outlook message and locked document starts the same way as each other. So: can a check with no model and no dependencies tell these apart, correctly, in microseconds? And does it beat Magika, Google's deep-learning detector, and the long-standing tools?

How it works

• The first 8 KB settle most formats: PDF (allowing a little junk before the header, as readers do), images, archives, Rich Text, and the text formats.
• Zip files: the table of contents, not the contents. A zip lists its entries at the end. word/document.xml means Word, xl/ Excel, ppt/ PowerPoint; a stored mimetype entry first means OpenDocument or EPUB. Nothing is decompressed.
• Old Office files: the directory. .doc, .xls, .ppt, .msg and password-protected modern Office files all use one container format, a small file system inside the file. The check walks its directory and reads the names of the top-level streams: WordDocument, Workbook, PowerPoint Document, Outlook's __substg1.0_ streams, or EncryptedPackage for a locked file.
• Three flags. Encrypted; the extension doesn't match; truncated (a PDF with no end marker, a zip with no table of contents, a container whose directory runs past the end of the file).
• Small. About 600 lines of Rust with no dependencies. The WebAssembly build is 39 KB, 18 KB compressed.

How I tested it

• The truth comes from parsers, not extensions. A file's true type is whatever a real parser can open it as: PyMuPDF for PDF, python-docx, openpyxl and python-pptx for modern Office, olefile for the old container, Pillow for images, and the standard gzip, JSON and XML readers. A file no parser opens isn't scored.
• Tuned on one set. 976 files from govdocs1 (thread 001, crawled from US government sites), plus 270 files I built to cover what it lacks: fresh Office files, 30 password-protected ones, OpenDocument, email, 20 renamed files and 15 cut in half.
• Tested on another. Then the rules were frozen, checked by a hash of the source, and run once on files nobody had looked at: govdocs1 thread 002 (990 files) and the test data of Apache POI (1,529 files), the Java library for Office formats. POI's collection is hard on purpose: real files from bug reports, odd variants, and fuzzer-generated wrecks.
• The same files for everyone. Magika 1.0.3, libmagic (the engine behind the Unix file command, 5.45) and file-type 22.1.1 each ran in their own batch mode. Each tool's answers were mapped to the same 30 labels by a fixed table.

Results

The second row is the honest headline. On ordinary files, trusting the extension is right 97.8% of the time, about as often as Magika. The average isn't where the difference is. It's in the cases a pipeline pays for.

Where it matters

• Password-protected Office files (47: 30 built, 17 in POI). This check found all 47 and libmagic 46. Magika found none: it called them Windows thumbnail caches (25), old Word files (16), old Excel files (2) or unknown (4). file-type calls them a generic container, and the extension calls them ordinary Word, Excel and PowerPoint files, which they aren't until someone types the password.
• Old Office and Outlook files. file-type sees only the shared container, so it misses 885 held-out .doc, .xls, .ppt and .msg files. That is most of its gap.
• Renamed files. All 29 flagged (20 built, 9 found in the held-out set), with no false alarms.
• Cut-off files. All 15 built ones flagged, and still named by what they were meant to be. In the held-out set it flagged 35: 31 are POI's fuzzer and crash tests, and the 4 others, checked by hand, are all genuinely broken.
• Junk before the header. Three old PDFs start with extra bytes, two of them wrapped in a classic Mac file header. file-type missed all three, Magika two and libmagic one.

Speed and size

These cross languages and runtimes, so read them as orders of magnitude, not a race; this laptop also drifts about two times between runs. The WebAssembly build gave the same answer as the native one on all 3,765 files. The check typically asks for 8 to 9 KB of a file, and at most 182 KB (overlapping reads counted twice).

What this doesn't prove

• Text formats. Whether a file is CSV or plain text, or HTML or plain text, has no parser to settle it. On the easy files I built it got 60 of 60 (libmagic 54, Magika 52), but on real ones I can only report where the tools agree. Magika is built for this, and for source code, which is outside these 30 labels.
• Today's uploads. govdocs1 is mostly from the 2000s, and POI leans towards odd files.
• Safety. Naming a file is not validating it. Files built to fool a detector weren't tested, and a file that opens in one Python library isn't proof that every pipeline can process it.
• Magika's range. It knows over 200 types. This compares only the 30 here.

What didn't work

• Three rules were wrong at first, all found on the tuning set before the freeze.
  • A locked .docx was flagged as renamed, but the name is right: Office opens it after asking for the password.
  • Cut-off spreadsheets came back as plain zips, because the file listing their parts is written last.
  • An old SGML document with a <title> tag was called HTML.
• The first comparison was broken, not the tools. The file list had Windows line endings, so libmagic couldn't open a single file and scored 12. Looking at its raw output, before believing the table, caught it.
• The truth needed a fix too. The labeller called any zip the Python libraries refused a plain zip, which is wrong for macro-enabled files, templates and binary workbooks. Those became unscored, and this was fixed before any held-out results were looked at.
• Three held-out misses stay in. Two fuzzer-made files it called PowerPoint and Excel, where the parser found no Office stream, and one .txt that is really a binary weather-data format, which it called text.

Credits

• govdocs1 from Digital Corpora (Garfinkel et al., 2009), public domain, and Apache POI's test data (Apache-2.0), for the files. None of them is served here. The four samples in the demo are made-up files, generated for this page.
• Magika by Google, libmagic by the file project, and file-type by Sindre Sorhus, for the comparison.
• PyMuPDF, olefile, python-docx, openpyxl, python-pptx and Pillow for the ground truth, msoffcrypto-tool for the locked files, and wasm-bindgen for the browser build.