A test that has never failed is not evidence of anything. It is an untested claim about a claim. We learned that the uncomfortable way: our locale-parity gate sat green for two days beside a file that shipped inside every binary and that no player could ever select — green not because the codebase was clean, but because the check enumerated the very list it was validating. Two rules came out of that week. The first is about where a check gets its universe. The second matters more: a guard nobody has seen fail is indistinguishable from a guard that cannot fail.
The check that could not see the extra file
The project is Boardlore, our multi-variant chess and
Go app — free and ad-supported, with a single optional Pro purchase that
removes the ads. It ships sixteen locales; the lib/l10n directory holds
eighteen ARB files (the translation bundles Flutter compiles into a generated
localization barrel), and the two extra are not translations — which is where
this story starts. Hence a parity gate: it held a list of the sixteen locales it knew
about and checked each one against the barrel.
The hole is obvious in hindsight. The check could
detect a locale file that was short, and could never, in principle, detect
an extra one — its universe of things that exist was a hardcoded
list, so anything off the list was not merely unchecked, it was invisible. For two
days an unreferenced app_ar.arb sat on disk, compiled into every
binary we built, selectable by nobody, while a green tick reported the localization
layer consistent. The check was correct about everything it looked at. That was the
problem.
The file was not a translation
Here is what turned a tidy little bug into a lesson.
Everyone assumed the orphan was an Arabic translation somebody had left half-wired.
It was not. app_ar.arb is a machine-generated right-to-left
pseudo-locale fixture — a fake language whose job is to stress a layout with
mirrored text and inflated strings, the technique we wrote about in
the locale that
breaks your layout.
Which sets a trap for the obvious verification. If you
want to know whether a file is an Arabic translation, you scan it for Arabic
codepoints. Do that here and you get 100%, and you have learned nothing, because the
generator appends the Arabic word
اختبار —
“test” — to every string it produces. A perfect score on a proxy
with no bearing on the question. The one property that actually separates a fixture
from a translation is the generator’s own sentinel: it wraps every value it
emits in ⟦…⟧ brackets. That is what the
replacement test asserts, across all 1,985 keys in the template.
The old key-count check was deleted rather than kept alongside, which feels like vandalism until you say plainly what it measured: a file with the right number of keys and the wrong content passes it. A test that asserts a proxy confidently is worse than no test, because the green tick reads as an answer to the question somebody actually had.
Enumerate from ground truth, and write down your exceptions
The first rule is one sentence: derive the set you are
checking from the authority, never from the same list you are validating. The locale
set now comes from ls lib/l10n/*.arb — the disk, which cannot lie
about what is on it. Overlay coverage comes from the set of
OverlayEntry( call sites in the source, so a new overlay is in scope the
moment somebody writes one. Host coverage comes from the set of
supportedLocales: bindings. Each is a fact about the codebase rather
than a belief about it.
The second half of the rule is the half people skip. An intentional exemption must be declared data carrying a reason, never an absence. The pseudo-locale genuinely should not be player-selectable, so it genuinely needs to be exempt — but “exempt” used to mean “happens not to appear in a list”, which is indistinguishable from an oversight. The exemption is now a typed record of a category and a reason, with exactly one legal category, so inventing a new kind of exception fails the test until somebody writes down its rationale. Two files previously treated as unrelated coincidences are now declared the same kind of thing.
Break it on purpose, then put it back
The first rule fixes one class of bug. The second is the one we would keep if we could keep only one, and it changes what a test task is expected to hand over. The primary deliverable of a new guard is not a green test. It is evidence that the guard can go red. A green new test proves nothing about whether it is wired to reality — which is exactly how the old gate passed for two days beside a file it was nominally responsible for. So we broke the new parity guard four independent ways and the new overlay guard once, read the real failure message each time, restored, and re-verified green.
The messages are the artefact, because a guard that
fires unhelpfully is a guard the next person deletes. Removing the fixture’s
declared exemption produced this: “ARB file(s) ship compiled into every
binary but NO PLAYER CAN SELECT THEM, and no exemption declares why: [ar]”.
It names the file, states the consequence, and implies the fix. Moving a real locale
file off disk was the more interesting break: two legs fired at once — the barrel advertising a locale with no backing file, and the
registry offering players a locale with nothing behind it — which proved the
missing-file direction without touching the locale registry at all. Handed a scratch
file with a second OverlayEntry(, the overlay guard said the game clock
keeps running behind it and no existing pause test can detect that, because all of
them exercise routes. That is a defect stated in gameplay terms rather than assertion
terms, the standard we set for the automated review lane in
an AI QA gate for games.
One detail carries more weight than it looks like it
should: every restoration was verified by sha256 equality, not by eye.
Breaking your own codebase to watch a test fail is only safe if putting it back is
mechanical, and “I put it back” is not evidence — it is a memory of
an intention. A hash is a fact, and it is the difference between a technique you can
use on a shared working tree and one that eventually leaves a stray edit behind, the
failure mode we know too well from
what happens when two
agents edit one file.
One shape, six disguises
What made this worth writing down was the count. In a single work wave the same failure shape produced six defects in six costumes, and they were filed as six unrelated bugs before anyone recognised one shape. The ARB gate above. A second pseudo-locale exempt only by accident of code generation. A text-reflow test blind to clipping. A route-based pause observer blind to any overlay that is not a route — a visible thing on screen with the clock ticking behind it. A widget test that had only ever pumped one window: 800 by 600, at a device pixel ratio of 1.0.
And the sixth, which is the one that keeps us honest. A
census of game variants matched ids with the pattern [a-z_]+, so it
silently dropped every hyphenated id and produced a confident, wrong count. Nothing
was broken. Nothing looked broken. No red test, no crash, no visible artefact —
just a number a reasonable person would plan against. Our own architecture document
carried a stale ADR count for ten days — 68 when the truth was 95 —
surfacing only when the row total, the highest ID, and the on-disk file count were
reconciled against each other.
A guard also has to age well
Sometimes a guard finds a real bug whose fix belongs in a different change. Both tempting options are bad: leave the suite red and it gets ignored; add an exemption and the guard blesses the bug. So the divergence is a declared row whose own assertion is that this host still binds what it claims — fix the bug and the row goes red, forcing its own deletion.
And every exclusion is a place a check can
quietly become theatre. Our scanner has to ignore a mention of
OverlayEntry inside a doc comment — correct, and also a hole in
the enumeration. So the exclusion is itself asserted: it must contribute exactly zero
sites, and that self-check fails loudly if the file ever stops mentioning the term.
An exclusion nobody re-checks is indistinguishable from one that no longer
applies.
The green tick is not the deliverable
Neither rule is clever, and that is the point. Ask where a check gets its universe of things that exist; if the answer is “a list somebody maintains”, it cannot see what that list omits. Then ask the harder question: has anyone watched this thing fail? Not reasoned that it would. Watched it. Until somebody breaks the condition on purpose and reads the message that comes out, a suite of green ticks is a set of assertions about assertions.
The cost is one extra cycle per guard — break, read, restore, hash, re-run — and the return is that a test stops being something we believe and becomes something we have observed. Make the artifact prove the property about itself rather than trusting someone to remember it, which deterministic by design argues from the other end. A guard nobody has seen fail is indistinguishable from a guard that cannot fail. The only way to tell them apart is to go and break it.
Notes
This is a first-hand account of two rules we adopted in our own codebase, not a general testing methodology. Every figure here is a count from our own repository on 2026-07-28 — sixteen shipping locales, eighteen ARB files on disk, 1,985 template keys, one legal exemption category, a stale 68 against a true 95 ADRs, four ways we broke one guard and one the other — not benchmarks or industry numbers, and they drift the moment we add a locale or a string. There are no performance figures or defect rates here, because we have none we would stand behind; inventing one to look rigorous would undercut the point.
One clarification worth stating plainly:
Boardlore does not ship Arabic. The app_ar.arb file above is a
machine-generated pseudo-locale fixture for right-to-left testing, not a translation,
and no player can select it — which is exactly why it needed a declared reason
rather than a quiet exception.
Keep reading: all posts on the WizusLabs Engineering blog.