Your parser didn't miss the address. It made one up.

There is a category of bug that does not fail. It produces output of exactly the right shape, in the right volume, with the right column names, and every value in it is fiction.

A parser that misses data is a nuisance. You notice the empty column, you widen the pattern, you move on. A parser that manufactures data is a different animal, because nothing downstream can tell the difference. Neither can your test suite, for a reason I think is worth twenty minutes of anybody's time.

I spent a day building an extractor that reads a company's website and returns the email addresses and phone numbers it publishes. Twenty-nine tests covered the extraction layer. All of them passed. The first time I pointed it at seven real companies, roughly two thirds of the addresses it returned did not exist.

Not malformed. Not stale. Invented: assembled out of ordinary English sentences by a pattern that was doing exactly what I had written it to do.

Here is what that looked like, why my tests were structurally incapable of catching it, and the one cheap habit that catches the entire class.

1. The parser that writes fiction

Websites hide email addresses from scrapers. They write hello [at] acme [dot] com, or press (at) acme.com, or sales at acme dot com. If you want the addresses a human can plainly see, you have to undo that.

So I wrote a pattern for it. Local part, then an "at" in any of its forms (bracketed, the word, or the symbol), then a domain whose dots may also be spelled out.

Then I ran it against apify.com, stripe.com, basecamp.com, n8n.io, hetzner.com and zapier.com, and it produced these:

autom@ion.our           from "Full automation. Our matter is different."
integr@ionen.skalieren  from "Integrationen skalieren mit Stripe"
pl@form.list            from "Build your platform. List your apps here."
communic@ion.thank      from "...communication. Thank you..."
th@matter.we            from "...that matter. We..."

Read autom@ion.our slowly. The word is autom·at·ion. My pattern found the letters "at" inside an ordinary word, took "autom" as the local part, and then swallowed the start of the next sentence as a domain.

Every one of those is a well-formed email address. Correct syntax. Plausible TLD. It would pass any validator you can name. I know that for certain, because mine has one, and it passed.

The output had the right shape and the wrong contents, which is the only failure mode that survives to production.

2. Why twenty-nine passing tests could not see it

This is the part I find genuinely instructive, because the tests were not lazy. They covered HTML entities, Cloudflare's obfuscation cipher, mailto: links with query strings and multiple recipients, percent-encoding, display names, retina image filenames that parse as addresses, placeholder addresses in form fields.

Every one of them looked like this:

def test_at_and_dot_in_brackets_are_understood(self):
    assert emails("<p>hello [at] acme [dot] com</p>") == {"hello@acme.com"}

I hand the parser a string that is an obfuscated address, and assert it finds it. Twenty-nine variations on that.

A test that feeds a parser the pattern it is hunting proves only that the pattern matches. It says nothing about what else the pattern matches.

Nowhere in that suite was a paragraph of ordinary marketing copy. Why would there be? I was testing an email extractor, so I wrote tests full of emails. The inputs came from my own idea of the problem, and my idea of the problem was the thing that was wrong.

A pattern matcher is defined by two things: what it accepts, and what it refuses. I had tested one of them.

3. Fixing it, and the metric that lied about the fix

First fix: a bare "at" must have whitespace on both sides. autom·at·ion has none, so it stops matching. Ten of the invented strings became regression tests, quoted verbatim from the run.

Then I measured yield properly: sixteen real companies, how many yield an address. Before: 10 of 16. After a separate improvement: 15 of 16, 93%.

Except two of those fifteen were these:

available@zapier.com   from "The app is available at zapier.com"
footer@www.notion.com  from "...the footer at www.notion.com"
measures@shopify.com   from "See the measures at shopify.com"

Round two of the same bug. English writes "available at zapier.com" constantly. Requiring whitespace around the "at" does nothing, because the whitespace is there.

And notice what the metric did. Fabricated addresses are indistinguishable from real ones to the thing counting them, so:

Precision failures inflate the number you use to measure recall.

93% was worse than the 87% that replaced it. The two extra "hits" were fiction, and my headline number was rewarding me for producing it. If you are tuning an extractor against a coverage metric, you are, unless you are careful, optimising partly for confabulation.

The fix that finally held is a distinction about intent rather than syntax:

# Real obfuscation spells BOTH separators - "sales at acme dot com" - because the
# author is hiding the address from exactly this kind of parser.
# Prose spells only the "at".
_AT_WORDED  = r"\s+at\s+"
_DOT_WORDED = r"(?:\s+dot\s+|\s*[\[({]\s*dot\s*[\])}]\s*)"

# A bracketed [at] may stand beside a literal dot, because brackets are never prose.
_AT_BRACKETED = r"(?:\s*[\[({]\s*at\s*[\])}]\s*|\s+@\s+)"

A worded "at" now demands a worded "dot". Somebody writing sales at acme dot com is deliberately evading a parser. Somebody writing available at zapier.com is writing a sentence.

privacy@zapier.com replaced the fabrication. team@makenotion.com, Notion's actual published address, replaced footer@www.notion.com. Shopify went back to reporting nothing, which is true.

4. The habit that catches all of it: a negative corpus

Three rounds of one bug is not bad luck, it is a missing test category. The fix is embarrassingly cheap.

Keep a corpus of text that must produce nothing. Not edge cases of your format. Actual prose, in the languages your users' sites are written in, harvested from real pages. Then assert emptiness.

@pytest.mark.parametrize("prose", [
    "Full automation. Our matter is different.",
    "Integrationen skalieren mit Stripe",
    "The app is available at zapier.com today",
    "See the measures at shopify.com for details",
    "Everything you need at basecamp.com",
])
def test_ordinary_prose_produces_no_address(self, prose):
    assert emails(f"<p>{prose}</p>") == set()

Fifteen of these now, every one a real sentence from a real website. They cost nothing to run and they are the only tests in the suite that could have caught any of the three rounds.

The principle generalises past regexes, and it is worth stating in the form that survives the specific technology:

Half your extractor's job is refusing. Test the refusing half with material that was never meant to match, and get it from the real world rather than your imagination.

This applies with more force, not less, when the extractor is a language model. Ask one to pull the contact address out of a page that has none, and it will often oblige you. The mitigation is identical: a held-out set of inputs whose correct answer is nothing, and a metric that counts a confident empty answer as a win.

5. The opposite failure, on the same day

Over-reading is the loud version. Under-reading is quieter and cost me more.

My extractor did the standard, sensible thing before parsing: strip <script>, <style>, <noscript> and <svg>, because they are not visible text.

Except schema.org structured data lives inside a <script> tag. WordPress via Yoast, Shopify and Squarespace all emit it by default, and it contains the company's contact details as a declared machine-readable fact rather than a string you scraped out of a paragraph:

{"@context":"https://schema.org","@type":"Organization",
 "email":"hello@acme.co.uk","telephone":"+44 20 7946 0958"}

The most reliable source on the page, deleted one line before anybody looked at it. Reading it costs about thirty lines and it is now the highest-confidence tier in the system.

The related miss was about where rather than what. Six of sixteen companies appeared to publish no address. Four of them do:

My page ranker scored /privacy at 20 and /terms at 15, because on a small business site they are boilerplate and the contact page is what you want. On a large company they are frequently the only place a human address survives legal review.

The ranking was not wrong; it was incomplete. A privacy page is correctly low priority while better pages exist, and correctly the best hope once they are exhausted, and one ranking cannot express both. So a second pass runs only when the first finds nothing. 62% → 87%, and because it only fires on a miss, it cost nothing on the sites that already worked.

While checking this I also settled a question I had been carrying as an assumption: those addresses were all in the raw HTML, even on pages that advertise themselves as single-page apps. JavaScript rendering was not the gap, and I would have spent a week and ten times the compute on a headless browser to discover that.

6. Don't write the regex if somebody has measured the problem

Phone numbers tempted me into the same mistake in a new costume. A business page is full of digit strings that are not phone numbers.

Google's libphonenumber ships a matcher built for this, with two leniency settings. Instead of picking one by feel, I ran both against realistic page noise:

The permissive setting would have shipped VAT identifiers and order references as phone numbers. The strict setting was clean on all six and lost nothing real.

The cost of strictness is honest and belongs in the documentation: a number its country's numbering plan does not recognise is not returned. One test case taught me that the hard way: 07700 900123 is not returned, because Ofcom reserves that range for film and television. The library was right and my test data was wrong.

There is a second-order lesson in the region handling. 020 7946 0958 is a London landline or nothing at all; the parser needs a country. I take it from the site's ccTLD, then the document's lang, and where there is neither, as on a .com with no declared region, only international +44… format is accepted. Defaulting to US would silently mangle every non-American number, and a deliberate under-read is better than a confident mistranslation.

7. Two things that were each correct and jointly wrong

The last one is not about parsing at all, and it is the one I would have been most embarrassed to ship.

Every address gets a 0–100 "safe to send" score and a recommended action. A real run produced this row:

safe_to_send_score = 0    recommended_action = send    hello@apify.com

Both halves were defensible. The policy knows an info@/hello@ address on a company's own contact page is the address they ask strangers to use, so: send. The scorer, inherited from a sibling product, deducts 45 points for a shared inbox. That is correct when you are prospecting into a company for a named decision-maker, and wrong here.

Two components, each right about its own question, incoherent when placed in one row. And it lands in the column a user sorts by, so sorting the output would have buried the best contact on the page beneath the worst.

The fix was not to edit the scorer, since another product depends on its judgement, but to register a new scoring version that reconsiders that single penalty, and to move the shared classifier into one module so the policy and the score can never disagree about the same inbox again.

Which leads to the idea I would actually keep from all of this.

8. Provenance is evidence, not decoration

Every address in this system carries where it was found and how it was published: declared in schema.org, written as a mailto: link, or sitting in page text.

I originally carried that for auditing. It turns out to be information.

A validator handed an address from somebody's list knows nothing about its history: it may be a typo, a three-year-old scrape, or invented. An address found on the company's own contact page tells you two things that DNS cannot: the mailbox is intended to receive mail from strangers, and the company believed it worked recently enough to publish it.

So it raises confidence, graded by how deliberately it was published:

That last row is why the grading matters. Text-only evidence earns almost nothing, and the same address declared in structured data earns the most.

There is a nice symmetry here with the fabrication bug. Both are the same question asked twice: what does this system actually know, as opposed to what can it produce? A pattern that matches is not knowledge. A string that parses is not a fact. Where it came from is what separates them.

What it measures, honestly

Sixteen well-known companies, measured rather than estimated:

• 87.5%, fourteen of sixteen, publish at least one email address findable without logging in
• 12.5%, two of sixteen, publish none at all, and one of those has no contact form either
• 2.3 addresses on average per company that publishes any (32 across 14)
• five of sixteen publish a phone number at all
• four of the sixteen publish nothing on their contact page and everything on their legal pages

One caveat I would want if I were reading this: the sample is sixteen well-known software companies, which is close to the hardest case. Large companies route enquiries through forms; a plumber in Leeds puts the address in the header. Read 87.5% as a floor for that kind of list, not as a general rate.

I publish the 12.5% because a stated limit is information and an unstated one is a refund request. A company with no published address gets a row saying exactly that, with the URL of its contact form where there is one. That is more useful than an empty cell and considerably more honest than a guess.

That is the same discipline as the negative corpus, applied to the product rather than the test suite: the system should be unable to express a confident claim it cannot support.

The extractor is packaged as an Apify Actor, Website Email & Phone Finder, if you would rather run it than build it. The negative corpus is the part I think is worth stealing.