Invoice extraction, built to be measured

Money never touches a float, including in storage.

In binary floating point, 0.1 plus 0.2 is not 0.3. This is a program whose entire job is checking whether numbers add up, so a float is refused at the boundary rather than quietly converted, and the database column is a decimal type that rejects one at bind time. The obvious shortcut here fails silently and months later, which is the worst combination available.

def process_bind_param(self, value: Any, dialect: Any) -> Any:
    if value is None:
        return None
    if isinstance(value, float):
        # Refused rather than converted. A float here means a caller
        # has already lost precision, and quietly accepting it would
        # store the loss and make it permanent.
        raise TypeError(
            "money must not be a float; pass a Decimal or a string of digits"
        )
    value = value if isinstance(value, Decimal) else Decimal(str(value))
    return str(value) if dialect.name == "sqlite" else value

Validation annotates. It never rejects.

Every rule returns a finding instead of raising, because in production the useful output is not pass or fail, it is a list sorted by how much each document needs a human. The weights are part of the code and ship with it: an unreadable document outranks everything, one error outranks several warnings, and warnings alone never stop the line. Queueing soft findings trains people to clear a queue without reading it, which is worse than not having a queue.

#: Nothing was read at all.
UNPARSED = 100

#: A document that contradicts itself. Three errors saturate the scale.
PER_ERROR = 30

#: Plausible but odd. Warnings alone cap below a single error.
PER_WARNING = 8

#: Highest score a parsed document can reach, so an unparsed one is
#: always first in the queue rather than tied with a very bad reading.
PARSED_CEILING = 99


def suspicion_score(findings: list[Finding], *, parsed: bool) -> int:
    """0-100, higher means look at this one first."""
    if not parsed:
        return UNPARSED

    errors = sum(1 for f in findings if f.severity is Severity.ERROR)
    warnings = sum(1 for f in findings if f.severity is Severity.WARNING)
    return min(PARSED_CEILING, errors * PER_ERROR + warnings * PER_WARNING)

An invoice carries its own checksum, so use it.

Line items should sum to the subtotal, subtotal plus tax should equal the total, and a due date should not precede its issue date. None of that needs ground truth, which is the point: it gives a quality signal on unlabelled production data, where a benchmark score cannot reach. It is cheap, deterministic and completely uninteresting to build, which is roughly why most projects skip it.

The database stores extraction events, not invoices.

A row means this provider read this document at this moment and got this. Reading again adds an attempt beside the last rather than overwriting it, so consistency becomes a query over real traffic instead of a laboratory exercise. A system storing only the latest value per document would have discarded the evidence before anyone thought to ask for it.

The application implements no extraction of its own.

Everything goes through the same provider boundary the measurement harness uses. The moment the application reads a document differently from the harness, the published numbers stop describing the product. Keeping the harness and the application in one codebase makes that structurally difficult rather than merely discouraged.

The model's output grammar was removed on purpose.

The API can constrain a model so its response is guaranteed to parse. That was in place, and it was hiding the measurement: a grammar-constrained model cannot return unparseable output, so the parse rate would have read 100% by construction and part of the quality score would have been measuring the API's decoder rather than the extractor. The shape moved into the prompt, an unparseable response now counts as a failure, and there is no repair step and no retry. A low parse rate is a result.

A value that cannot be located is reported as not found.

The review pane never draws an approximate box. A box in roughly the right place tells a reviewer the number was checked, and the entire purpose of the screen is that the reviewer checks it.

56% to 70%

One bug, found by looking at the screen rather than by running the tests.

On an invoice with no tax row, the only VAT string on the page is the vendor's own registration number in the letterhead. The heuristic accepted any candidate containing a digit, and so read an identifier as a tax charge of 148 million dinar. Requiring a money label's value to look like a number moved exactly extracted documents from 56% to 70%, and validation pass rate from 80% to 98%.

251

Passing tests that did not surface that bug, or the one beside it.

Both were found by opening the running application and reading the screen. Three more layout defects surfaced the same way, including a sticky table header that covered the first row of the queue so clicks on that row hit the header instead. This project verifies by measurement, and measurement does not cover layout.

2 of 2

Models in the intended comparison disagreed about their own parameters.

One rejected an effort setting outright with a 400. The other enabled extended thinking unless explicitly told not to, which would have billed reasoning tokens at output rates for a task that is reading, not reasoning. A hardcoded table of that goes stale at the next model release, so the provider asks the API what the target model accepts and configures itself once per run.