Your data pipeline passes its own tests. Here's how it lies anyway.

Green CI is not evidence that your data is correct. It is evidence that the questions you asked got the answers you expected. Those are different claims, and the gap between them is where the expensive bugs live.

On 2026-08-27 I wrote seven build gates for a site that publishes the test results of eight trading strategies: four killed, four unproven, none passing its threshold. Five of the seven had defects the same day. Not one of them reported anything. Every one of them printed the same line a working gate prints.

The five failed differently, which is the point. There is no single mistake to avoid here. What generalises is the order in which you ask two questions.

Five gates, five failure modes, all green

Gate	What it printed	What was actually happening
Delisting reconciliation	pass	Swap one symbol for another, count unchanged, still passes
Paragraph length	718 false alarms	The regex was eating <polyline> tags
Repetition detection	fake “repeats”	Matching a word, not the claim the word appears in
Meta-prose ratio	pass	Ruler swapped, threshold left behind, nothing could fail
Cross-language pairing	grep hreflang finds hits	The markup found does not count to Google

Source for all five is below. None of it touches trading logic; these are the checks, not the strategy.

The one that cost the most: 718 false alarms from one missing character

A gate that flags over-long paragraphs, so a reader does not hit a wall of text and skip the whole block, including the limitation statement buried in it.

// first version, wrong
const PARA = /<p[^>]*>(.*?)<\/p>/s;

First run: 718 coin pages each reported as having a 358-character paragraph. The text it quoted was a figure caption, not a paragraph.

<p[^>]*> matches <polyline class="ser" points="…">. After <p, the character class swallows olyline … up to the closing bracket. Then .*? runs to the next genuine </p>, and everything in between counts as one paragraph.

// fixed
const PARA = /<p(?![a-zA-Z])[^>]*>(.*?)<\/p>/s;

One negative lookahead. The interesting part is not the bug, it is the cost of the symptom: false alarms are more expensive than misses, and somebody has measured that. Google’s testing blog reports one team’s data: when a stable test turned flaky and the change that caused it could be identified, the underlying problem was a real production bug one time in six (Google Testing Blog, 2017, retrieved 2026-08-27).

Five times in six, the test was wrong. A gate that cries wolf gets disabled or ignored, and on the day it gets ignored, the thing it guarded stops being guarded.

Comparing counts instead of sets: 29 against 29, wrong contents

The site keeps a page listing every delisted perpetual futures contract. That page is the evidence for a claim: that these records exist nowhere else. A gate checks it against the underlying data.

The first version compared counts.

# first version, wrong
def delisted_rows(dist):
    want = count_delisted_in_data()      # files with a non-empty delisted field
    got = len(COIN_LINK.findall(page))   # symbols linked on the page
    return want == got

Replace one symbol with a different one. The count does not move. The gate passes. That is precisely the case worth catching: the page disagrees with the data, and the disagreement is not “one is missing”.

# fixed
def delisted_rows(dist):
    want = {d['symbol'] for d in data if d.get('delisted')}
    got = set(COIN_LINK.findall(page.read_text(encoding='utf-8')))
    return page, sorted(want - got), sorted(got - want)   # both directions

Both directions matter. want - got means a page nothing links to; got - want means a link to a page that does not exist. A check that returns one side leaves the other side permanently invisible.

The list it guards is here, with listing date, delisting date and weeks survived for each contract.

The embarrassing detail: thirty lines above it in the same file, another check was already comparing sets. I wrote the second one without rereading the first.

Matching a word when you meant a claim

A gate for repetition: the same point made more than twice on one page. The unit is the claim, not the string, so it uses keyword patterns.

# first version, wrong
CALIBERS = {
    'not summable / different basis': r'加總|基準(不|一)',
    'evaluation account is not real money': r'評測.{0,12}(不是|非真實|不計入)',
    ...
}

What it actually matched on the first run:

Matched text	What it really is
“divergence sum 36.22 pp”	the name of a quantity
“entry price sum across 15 bets is 5.11”	same
“§02 Capital and current value … evaluation account excluded”	a section heading, flattened
“§04 Evaluation account · not real money”	same

“Sum” on this site is the name of a quantity. It is not the claim “these cannot be summed”. I was matching the word and I wanted the assertion.

The fix had two halves: patterns restricted to negated forms, and tables and headings excluded the way every other measurement on the site already excludes them. A table cell is not a sentence.

Then a third thing, which matters as much as the fix. Every corrected pattern now carries the text it wrongly matched, and those strings became assertions:

for x in ('divergence sum is 36.22 pp, and fees ate over seventy percent of it',
          'entry price sum across 15 bets is 5.11, and the per-bet figures reconcile'):
    assert not any(re.search(p, x) for _, p in CALIBERS.items()), f'false positive: {x}'

Three corrections in a row, all in the direction of a smaller number. A ruler that misfires reports “repetition” that looks exactly like the real thing, so adjusting a measuring instrument is itself an action that needs evidence attached.

A threshold left over from a different ruler

This one is the worst of the five.

A script measures what share of a page’s sentences talk about its own layout rather than about data. The thresholds came from a reviewer, derived from their word list: 20% / 25% / 30% for three pages. Then I swapped in a list of my own, roughly one-eighth the size, and left the thresholds alone.

Their ruler  32% / 43% / 54%     limits 20 / 25 / 30  →  cut a third to a half
my ruler      4% /  7% / 23%     limits 20 / 25 / 30  →  all three walk through

The header of that same script, written by me, says this:

The limit is not the true share of meta-prose. It is the number this ruler produces. Swap the ruler and you must re-derive the limit.

I wrote that sentence, then swapped the ruler, and did not re-derive the limit.

The damage is not the hole. It is that the gate printed “pass” on every build. A gate with a hole gives itself away when you test it. A gate whose judging range contains nothing that could fail passes the test too, because the range is empty.

Markup that looks like the real thing: <a hreflang> is not an hreflang annotation

The fifth was pointed out by someone else, and it is the hardest of the five to find on your own.

Article listings on the site carry links like this:

<a href="/notes/prop-firm-bot-breaks/" hreflang="en" lang="en">…</a>

That attribute is for screen readers: a Chinese-locale speech engine should not read an English headline with Chinese phonetics. It is correct, and it is not a cross-language annotation.

Google reads three places and no others: <link rel="alternate" hreflang> inside a well-formed <head>, an HTTP Link header, or xhtml:link entries in a sitemap. It also requires reciprocity: if two pages do not both point at each other, the tags are ignored (Google Search Central, retrieved 2026-08-27).

So the state of the site was not “we forgot”. It was “we shipped something that looks like it and isn’t”, which is worse:

Anyone who greps for hreflang sees hits, and therefore stops looking.

The fix emits the real three links and adds a gate that checks the pairing is bidirectional. The gate’s comment says so out loud, because the next person will grep:

# This gate checks <link rel="alternate"> inside <head>.
# It does NOT check for the string "hreflang" anywhere in the document.
# A grep-based check here would be green forever.

What generalises: ask the questions in this order

Five defects, five different mechanisms:

#	Where it went wrong	Symptom	One line
1	Compared the wrong thing	green	equal counts are not equal contents
2	Match window too wide	718 false alarms	false alarms disable a gate faster than misses
3	Matched words, not claims	fake repeats	a misfiring ruler’s output looks like the real thing
4	Judging range was empty	green	testing it also passes
5	Shipped a lookalike	grep finds hits	it looks right, so nobody checks again

What they share is not “write more tests”:

Ask whether it can pass for the wrong reason before you ask whether it is right. The order does not commute.

There is a literature for this. Mutation testing deliberately breaks the program and checks that the tests go red. Just et al. validated the approach at FSE 2014 across 5 open-source applications, 321,000 lines of code and 357 real faults, finding a statistically significant correlation between mutant detection and real fault detection, independent of code coverage (ACM Digital Library, retrieved 2026-08-27). Coverage measures whether a line ran. Every line in all five gates above ran, and returned normally.

Metamorphic testing covers the other half: when you cannot decide whether a single output is correct, check a relation between inputs and outputs instead. Chen et al. survey it in ACM Computing Surveys 51(1) as one answer to the oracle problem (Victoria University repository, retrieved 2026-08-27). Fix #1 is exactly that shape: I cannot tell you whether 29 is the right number, but I can tell you the set difference must be empty.

Silent failure is not a software-only problem

The same shape has been measured a layer down. Dixit et al. analysed CPUs across hundreds of thousands of machines at Meta and found silent data corruption to be systemic across hardware generations: the errors are not captured by the CPU’s own error reporting, so they are untraceable at the hardware level, and they propagate up the stack until they surface as application problems (arXiv:2102.11245, retrieved 2026-08-27).

Translated into pipeline terms: your checks being green does not mean the data is right. It means the questions you asked returned the answers you expected.

Five things to do about it

1. Compare sets, not counts, and return both directions. One-sided checks make the other side permanently invisible.
2. Take any gate that has never gone red and confirm it can. Inject the fault into the artifact — the built HTML, the generated JSON — not the source. Breaking the source can fail the build on a syntax error first, and that reads like the gate working.
3. After tightening any check, reread the assertions that were already green. They will not turn red, so no signal comes to find you. Tighten one field from optional to required and six existing assertions can quietly start testing “field missing” instead of what they were written for.
4. Swap a ruler and you must re-derive the threshold, by a rule fixed before the new numbers are visible.
5. Make the gate print what it let through, not only what it stopped. A chart label rendering at 25px passed a minimum-size check legitimately; it was the report mode that surfaced it.

The fifth is the cheapest and the rarest. A gate you only read when it is red is blind to the class of error that stays inside the allowed range.

FAQ

Isn’t this just test coverage? No. Coverage measures whether a line executed. Every line in all five gates executed and returned normally. The defect is in the criterion, not in reachability.

Why publish your own bugs? Because a piece that only cites other people’s mistakes is a sermon. These five happened on 2026-08-27, in one project, with the full sequence recorded, and all five were caught before they did damage. Not because I got sharper — because a second party read the work with different assumptions.

Do these checks touch your trading strategy? No. Everything above is the checking code: set comparison, paragraph splitting, repeat detection, rendered-size measurement. Entry conditions, thresholds, holding periods and stop placement are never published; the boundary and the reasoning are on the methodology page, and a build gate enforces it — it stopped me four times that day.

Where does the same disease show up in the trading work? Nine variants of it, plus 22 instances caught in-house, are written up in Checks that pass for the wrong reason. Real money, closed trades and the daily equity series are on the performance page.