Four silent failure modes from building a real extraction pipeline, and what actually fixed them
Aug 16, 2026 · 8 min read
A pipeline that fails is a nuisance. You get an alert, you read a stack trace, you fix it. The bug has a shape.
A pipeline that succeeds and returns wrong data is a different animal. Exit code zero. Green monitors. Rows in the warehouse. Someone downstream makes a decision on it three weeks later, and by then nobody remembers which run produced the number.
I spent a day building an extraction pipeline that pulls public profile and contact data from YouTube channels. It has 221 tests. Every one of them passed while the pipeline was quietly returning nonsense: four separate times, in four different ways. None of the four were visible from reading the code. All of them were obvious within thirty seconds of reading an actual run.
This is what those four looked like, and what fixed them.
The first run against real channels returned a clean, plausible result: this creator publishes no contact details.
That was false for every single channel.
From a European IP address, YouTube 302-redirects to a cookie consent page and then serves an empty 200. Not a 403. Not a redirect loop. A successful response with no content in it.
response = await client.get(url, follow_redirects=True)
response.raise_for_status() # passes
data = extract(response.text) # finds nothing
return data # "no contact details published"Nothing here is wrong, exactly. raise_for_status() did its job. The parser found no data because there was no data. Every layer behaved correctly and the output was a confident lie.
The failure mode is that "empty" and "blocked" produced identical output. In this domain they are opposite facts: one is a statement about the creator, the other is a statement about us.
The fix is two cookies. The lesson is not two cookies:
if "consent.youtube.com" in str(response.url):
raise ChannelFetchError(ChannelError.CONSENT_WALL, "...")
if "ytInitialData" not in text:
raise ChannelFetchError(
ChannelError.UNPARSEABLE, f"200 OK but no channel data ({len(text)} bytes)"
)A 200 that contains none of the structure you expect is not a successful fetch. If your extractor can return an empty result for two different reasons, and you cannot tell which from the output, you have not written an extractor. You have written a coin flip that reports heads.
I now treat "success with an empty payload" as an error condition by default, and make the caller opt into treating it as data.
YouTube returns channel search results as a JSON structure with named fields. One of them is subscriberCountText. Another is videoCountText.
subscriberCountText contains the channel handle. videoCountText contains the subscriber count.
{
"channelRenderer": {
"title": { "simpleText": "MadFit" },
"subscriberCountText":{ "simpleText": "@MadFit" },
"videoCountText": { "simpleText": "11.6M subscribers" }
}
}Read that by field name and you ship a column labelled subscribers full of @handles. Types are valid. Nothing is null. Nothing throws. The column is simply wrong, and it is wrong in a way that survives every schema check you would normally write, because the shape is correct.
The naive fix is to read by position instead. That is worse: it breaks silently the day upstream fixes their bug.
What I actually did was read by content:
values = [text_of(r.get("subscriberCountText")), text_of(r.get("videoCountText"))]
handle = next((v for v in values if v.startswith("@")), None)
subscribers = next((v for v in values if "subscriber" in v.lower()), None)Ugly, and correct under both regimes. It survives upstream fixing the shift, and it survives them not fixing it.
The general point: an upstream schema is an observation, not a contract. You did not negotiate it, you cannot version it, and nobody will tell you when it changes. Validate against what the values are, not what someone named them.
This one nearly shipped, and it is the one I would least like to have explained to a customer.
The pipeline follows a creator's linked website to find a contact address. For a fitness creator, it returned:
Home Workouts By Fit Tiff → hello@gorillamats.comWell-formed address. Live domain. Working mail server. Real business. Passes every validation rule you can write about an email address.
It belongs to a sponsor. A creator's link list is mostly sponsors, affiliate codes and merch stores; their own site is often the minority of it. The pipeline followed the first plausible link and returned the wrong company's contact as the creator's.
This is not a data-quality problem. It is an entity resolution problem wearing a data-quality costume. Every field was valid. The record was about someone else.
The fix was to ask a narrower, answerable question. Not "does this site belong to this creator?", which needs information I do not have, but "is this domain built out of this creator's own distinctive words?"
matched = [t for t in name_tokens if t in brand]
if not matched:
return False # "The Official Home Video Channel" ≠ videochannel.com
residue = brand
for token in matched:
residue = residue.replace(token, "")
return len(residue) <= 2 # "fitnessgear" keeps "gear" → not hersIt refuses more than it accepts. It costs real coverage: three of fourteen channels in my test set have websites I now decline to crawl.
That trade is not close, and the reason is an asymmetry worth stating explicitly:
A missing row costs the user one lead. A confidently wrong row costs them a wasted action and costs you their trust in every other row.
Recall failures are visible and bounded. Precision failures are invisible and compound. When they trade off, price them differently.
The last one is not a correctness bug. It is the kind of bug that only shows up on an invoice.
Two operations in this pipeline look similar and cost wildly different amounts:
| Operation | Payload |
|---|---|
| Discover a channel via search | ~43 KB |
| Enrich one channel's profile page | ~2.14 MB |
Fifty times. I had written the obvious pipeline: fetch everything, then filter. That meant paying the expensive operation for every row I was about to discard. A user asking for creators between 10,000 and 500,000 subscribers was paying to fully enrich eight-million-subscriber channels that were never going to qualify.
Reordering is trivial. Noticing is the hard part, and I only noticed by reading a real run's billing breakdown rather than its output.
shortlist = [c for c in discovered if request.channel_filter.accepts(c)]
targets = self._targets(shortlist, request) # filter BEFORE enrich
results = await self._enrich(targets, request)It is now pinned by a test, because it is an economic invariant and not a style preference:
assert len(recorder.channel_fetches) == summary.shortlistedThis generalises well beyond scraping. Any pipeline with a cheap identifying step and an expensive enriching step has this shape. It is the same mistake as sending every candidate document to an LLM before filtering with metadata you already had. Put the selective operation first and make it cheap to be wrong.
Four unrelated bugs, one underlying discipline: the pipeline must be unable to express a confident claim it cannot support.
Provenance as a type constraint, not a convention. A fact in this system cannot be constructed without the URL it came from:
@dataclass(frozen=True)
class FoundEmail:
address: str
source: EmailSource
source_url: str # not optionalYou cannot forget it, because there is no constructor that omits it. An email with no provenance is a rumour, and rumours should not typecheck.
Refusal as a value, distinct from absence. "Not checked" and "checked and fine" are different facts and must stay different all the way into the output file. The moment they collapse into the same empty cell, the user has to guess. And users guess optimistically.
Versioned judgement rather than mutated judgement. The same signal can mean opposite things in two contexts. A shared info@ inbox is a weak lead when you are prospecting a company, and it is the intended contact when you are approaching a creator for a sponsorship. Same address, opposite verdict, both correct.
So I registered a new policy version instead of editing the old one:
_REGISTRY = {
v1.POLICY_VERSION: v1.decide,
v2.POLICY_VERSION: v2.decide,
creator_v1.POLICY_VERSION: creator_v1.decide,
}Every stored result carries the version that produced it. Old rows stay reproducible, new judgement ships freely, and a test pins the old versions to their documented answers so divergence is deliberate rather than accidental.
If you have ever tried to explain why a number changed retroactively, you already know why this matters more than it looks.
Everything above is ordinary data engineering. It becomes sharper when the consumer is not a person.
A human looking at hello@gorillamats.com under a fitness creator's name has a decent chance of frowning at it. An agent does not frown. It reads a well-formed field, treats it as fact, and acts: drafts the email, calls the next tool, writes the row.
Tool output is context, and an agent cannot distinguish a confident wrong answer from a right one unless the tool makes the distinction structurally. That means:
Retrieval quality gets discussed endlessly for RAG. The same argument applies to every tool an agent calls, and tools are usually held to a lower standard than retrievers despite being trusted more.
The pipeline finds creators by niche, filters them by subscriber band, and returns their profile with a contact address where one is publicly available. On mid-sized creators (5k–500k subscribers), measured rather than estimated:
I publish the first number even though the second one is the flattering one, because a stated limit is information and an unstated one is a refund request. The gate is reported as a field saying this creator has an address we cannot read, which turns a limitation into something the user can act on.
That is the same discipline as the four fixes above, applied to the product rather than the code: say what you know, say what you don't, and never let the second one look like the first.
The pipeline is packaged as an Apify Actor, YouTube Creator Finder, if you want to run it rather than build it. The engineering above is the part I think is worth stealing.