Cover photograph by Daniel Andraski on Pexels.
← Insights & ArticlesWhich path it read the document by, whether it knows it might be wrong, what it cost, and where the document went.
Sep 13, 2026 · 14 min read
Put an invoice in front of a current model and ask for the vendor, the number, the dates and the totals, and you will get them. Reliably, in structured form, for a fraction of a cent. That capability is eighteen months old and it is not in dispute.
Which is why a document extraction demo is no longer an achievement, and why almost all of them look identical: upload, spinner, a panel of fields, the implicit claim that the hard part is done.
Here is the invoice that shows where the hard part actually is.
subtotal 100 319.27
tax 19 060.66
---------
sum 119 379.93
printed total 119 479.93Every field on that document was read correctly. The extraction is perfect. The document is wrong, by exactly one hundred, and a system that returns the printed total and stops has done its job and handed you a problem.
This is an account of the parts that come after reading: how the thing decides which way to read a document, how it knows when to doubt itself, what it costs, where the document goes, and how it is deployed. It has been public for about a month at a URL anybody can open.
The cost question gets answered with model prices, which is the least interesting part of it. The thing that makes this cheap is a routing decision made before any model is involved.
A PDF either has a text layer or it does not. A digitally generated invoice, which is most of them in business-to-business accounts payable, carries the characters as characters. Reading them is exact, free, instantaneous and happens on the machine the application is running on. Nothing leaves.
A scan carries pixels. There is no text to extract, and getting text out of it needs something that was not free.
So the application tries the text layer first, and only a document with no extractable text goes anywhere else.
text layer present -> read locally. exact, free, nothing leaves
no text layer -> parsing service, then the model reads thatThat sentence is the entire cost model. The expensive path is the exception rather than the default, which means the bill scales with how many of your documents are scans rather than with how many documents you have.
One implementation detail carries it, and it is the kind of thing that decides whether a system stays correct. The routing decision keys on an exception type raised when there is no text layer, never on matching an error message. A boundary held together by prose is a boundary that breaks the first time somebody rewords an error, silently, in a way no test notices.
When a document has no text, the obvious move in 2026 is to show it to a multimodal model. That option exists in this codebase and it deliberately refuses to run.
The reason is that the two approaches are not substitutes and pretending otherwise makes the system lie about itself.
LlamaParse takes a document and returns markdown. It is a parsing service: it recovers the text and the table structure, and what comes back is text that the same extraction prompt can read. The model that then reads it is doing exactly what it does with a digital PDF, on input that has been converted into the same shape.
A vision model is doing something else entirely. It looks at rasterised pages. It can read a handwritten annotation in the margin, a stamp across the total, a signature, a layout that carries meaning. Converting a scan to text throws all of that away before the model ever sees it.
So OCR is its own ingestion path rather than a reuse of the vision one, and the vision path still raises rather than quietly falling back to OCR.
Labelling one as the other would make the module lie, and the module exists precisely so the two paths never blend. A scan converted to text and a scan looked at are different readings of the same document, and a report that mixed them would be describing neither.
Three further decisions came with it.
The EU endpoint is the default in code, not an environment variable. The vendor's SDK defaults to its US endpoint. A deployment that simply forgot to set a variable would keep working perfectly while exporting every scanned invoice to another jurisdiction. Nothing raises, no test goes red, and the only symptom is a privacy claim that has quietly become false. Overriding it is possible and means editing the privacy notice in the same change.
Both paths share one prompt, byte for byte. Warning the model that its input came from a scan would change how it resolves ambiguity, and the two paths would stop being comparable. Comparability is the one thing the ingestion split exists to allow.
The package everyone recommends is deprecated. Maintenance on llama-cloud-services ended in May 2026 and most guides still show it. The dependency here is llama-cloud. That is a boring fact with a short shelf life, and it is exactly the sort of thing that separates something someone runs from something someone wrote about once.
Measured rather than assumed: a corpus invoice rasterised to 200 DPI as an image-only PDF, with zero extractable text, came back with all fifteen ground-truth values in 21 seconds, table structure intact. End to end through the full chain it scored 14 of 14 fields against ground truth with no validation findings, at roughly half a cent.
digital PDF free. read locally, nothing leaves the machine
scanned PDF free to 1,000 pages a day on the EU endpoint
model, per doc about half a cent on Sonnet 5
50-document run ~$0.10 on Haiku 4.5, ~$0.20 on Sonnet 5
server EUR 10.69 a month, shared with three other things
spend cap EUR 20, measured against recorded tokensThe spend cap is the part I would keep. It does not estimate. It sums token counts already recorded against every extraction and prices them from a dated table, so the number it enforces is the number that was actually spent.
And it fails closed. A model missing from that pricing table stops the demo rather than costing zero.
A cap that switches itself off at exactly the moment somebody changes the model is worse than no cap, because it reports success.
An hourly document limit sits beside it, because the two failure modes are different: the cap stops a slow drain, the limit stops a burst.
The same Hetzner box that runs my site, through the same self-hosted control plane, built from this repository's own Dockerfile, with a Postgres database the panel provisions. A separate application record, a different stack, one machine. The companion article covers that path in detail.
What being publicly reachable added is a posture rather than a feature.
One application, two modes. Everything that makes it safe to expose sits behind a single flag, inert unless it is on. A local run and the public deployment are the same code, because two codebases drift and the one that drifts is always the one nobody runs locally.
Being public has to be chosen. It refuses to start without an access token and refuses one shorter than sixteen characters. Open by accident and open on purpose must not look the same from outside, so one of them is made impossible.
There are no accounts, and it says so. One shared queue, everyone sees everything, disclosed in a notice that cannot be dismissed. The answer offered instead of privacy is a delete button that genuinely deletes: the file first, then the row, because the row is the only record that the file exists.
Automatic deletion is deliberately off. A demo that empties itself overnight demonstrates nothing, and the accumulated documents are part of what makes it look like a working system rather than a screenshot. The residual risk is stated rather than engineered away: whatever somebody uploads stays until somebody removes it.
This is the structural decision that separates an application from a demo more than any other, and it is invisible from the screen.
The database does not store invoices. It stores extraction events.
document -> extraction -> invoice -> line_item
|
+-> findingA row means: this provider read this document at this moment and got this. Reading the same document again adds an attempt beside the previous one rather than replacing it.
That shape costs storage and buys a question that cannot otherwise be asked: whether the same model, given the same document twice, returns the same answer. Consistency becomes a query over live traffic rather than a laboratory exercise.
An application storing only the latest invoice per document could answer none of those questions, and would have discarded the evidence before anybody thought to ask.
A failed extraction is stored too, never dropped, and it sorts to the top of the queue. An application that discarded failures would show a clean list while the work piled up out of sight, which is a demo of a working system rather than a working system.
Amounts are stored as an exact decimal type that refuses a float outright at bind time, and they cross the wire as strings, because a JSON number becomes an IEEE double in every browser that receives it.
This sounds like pedantry until you see what the ordinary path produces. The obvious database mapping converts through a Python float on SQLite and hands back 1483.1599999999999. In a project whose entire argument is that the numbers add up, that is not acceptable even in a test-only configuration.
There is a subtler one beside it. Postgres returns 43555.0400 where SQLite returns 43555.04, because scale is storage's business. Rendering trims to the currency's minor units, two for euro, none for yen, three for dinar, and keeps every significant digit beyond them.
A model returning three decimals for a euro amount should still look wrong, rather than being rounded into agreement. Rounding is the most polite way to hide an error.
The validator does not reject. It annotates, and the useful output is a list ordered by how much each document wants a human.
The weights ship with the code rather than living in a configuration nobody reads: an unparseable response outranks everything, one hard error outranks several warnings, and warnings alone do not stop the line. That last one is a judgement about people rather than about documents. Queueing soft findings trains reviewers to clear a queue without reading it, which is worse than not queueing them at all.
The suspicion score needs no ground truth, which is the property that makes it work in production. It comes from the document disagreeing with itself, so it applies equally to an invoice nobody has ever labelled, from a vendor the system has never seen.
Both of these made the product look worse and are the reason its numbers are worth reading.
The extractor used to hand the model a schema, so the API's own grammar guaranteed the response would parse. That is the recommended practice and it was removed on purpose.
A grammar-constrained model cannot return unparseable output or a wrong-typed field. Parse rate would have been 100% by construction. The deliberately planted unparseable fault in the test corpus would have become unreachable. Part of what the system reports as reliability would have been measuring the API's decoder rather than the extractor.
For something whose entire value is that its numbers describe the thing under test, numbers that flattering are not worth having. A low parse rate is a result, not a bug to be engineered away.
Removing it revealed something else immediately: the prompt had been carrying less than anyone thought. It stated the field conventions and never the object shape, because the grammar had silently been doing that half. Nothing would ever have revealed it while the grammar was there.
The harness exists to produce one. The comparison across model tiers has not been run, so no measured accuracy number exists, so neither this article nor the product states one.
The 95.5% in the screenshots is a validation pass rate: the share of documents whose own arithmetic agrees with itself. That is self-consistency, not correctness. An invoice can be internally consistent and still have been read wrong, and presenting that number as accuracy would be precisely the move this system is built to refuse.
None of the above prevented the worst defect in the project, and leaving it out would make this an advertisement.
For about a month, the site said the tool could read scanned documents, and every scanned upload failed. The routing described at the top of this article was written, tested and documented. The service never called it: it passed the text-layer option as a literal and never asked.
Nothing was red. The OCR tests passed, because they test the library. The service tests passed, because none of them uploaded a scan. There were 315 passing tests and a green build.
A test that exercises a capability is not a test that the product reaches it. The suite proved the routing worked. Nothing proved anything ever asked it to.
Fixing it then turned a harmless database default into a false statement: the column recording which path a document took defaulted to text layer, which had been true for as long as there was only one path, and became a lie on the one screen that tells a visitor whether their document left the machine. A column default is an assertion about the world, and it expires silently when the world changes around it.
Both were found by opening the deployed product and using it, not by the suite.
The corpus is synthetic. Documents are generated from a seed, which is what makes the harness reproducible and also means no figure here describes performance on real invoices from real vendors with real layout chaos.
The routing claim, that most documents in business-to-business accounts payable carry a text layer, is a general observation about that domain and not a measurement of anybody's particular mailbox. If yours is mostly scans, the cost model above inverts and the parsing service becomes the main line item.
And the cost figures were taken on a specific date at specific prices, on a corpus of ordinary one-page invoices. A dense multi-page document costs more in proportion to what it contains.
A document demo shows you that a model can read a document, which is no longer in question.
An application decides how to read it and takes the free path when it can, tells you when the document disagrees with itself, stores what happened rather than what it concluded, refuses to round a number into agreement, knows what it spent and stops when it has spent enough, and tells you which jurisdiction the document was processed in.
None of that is model work. All of it is the part that was skipped.
It is open at demo.alansalomon.ai, including the invoice at the top of this article, which is one of the two sample buttons. The engagement it grew out of is written up at Invoice extraction, with what it does not do on the same page.