A failing check tells you something. A check that comes back
empty tells you almost nothing, and the reason is arithmetic rather than diligence: there is
one observable — 0 — and at least five states that produce it. Four
are broken instruments; the fifth is the answer you wanted, and no careful reading separates
them, because there is nothing in the output to read. The remedy is not more care. It
is a second measurement in the same invocation whose expected value is known and cannot be
zero.
The gate that reads an empty result as “production build”
Our staging builds carry a compile-time debug switch that must
never reach a store, so every artifact is checked before promotion: dump the strings in the
compiled binary, count a per-project marker, read the count. One or more means QA build; zero
means clean. Why that marker exists, and why it is a deliberate side effect rather than a
literal in the source, is
a debug switch that ships
in release. On NeuralSpark’s 1.9.8+231 staging cut, the check returned
empty for all six of its checks — on a correctly-built release artifact whose real
marker count was one.
Nothing was wrong with the binary or with strings: the
output had been captured into a shell variable instead of piped, so the grep that followed
searched something that was not there. The instrument never ran, and read naively that empty
result says production build — the one direction a pre-promotion gate must never
fail in. The fix took two lines: grep a string you know is present, the bundle identifier, in
the same invocation as the marker, and report both counts. The other half was to establish the
build mode first, because in a debug build the Dart code is not compiled into the iOS binary
the check inspects, so the marker grep returns zero for an entirely innocent reason — and
zero was the documented legend for “production build.” That gap sat in our
workspace-wide procedure too, covering three further shipping apps. An unestablished mode now
makes the result inconclusive and blocking rather than passing.
Zero is overloaded
Once you have seen it twice you start seeing the class. A count of
0, an empty string, a zero-byte file, an exit status of 1 — the values
verification tooling leans on hardest, and the least informative values a program can produce,
because an absence has no fingerprint.
An invalid pattern. A sweep in one of our
unreleased projects reported zero hits for four test pins that had just landed. The edits were
perfect; the pattern was not. An unescaped ( in an extended regular
expression opened a capture group, so the sweep searched for the literal
expectrunner.anchorX, 190);, which exists in no file anywhere — and read as
it stood, that clean zero looks exactly like a silently reverted edit. What exposed it was the
shape of the result, not its value: the correctly escaped patterns in the same sweep
returned correct counts.
A path that is not there. On a staging cut of Embergrid — a game of ours that has not shipped yet — a verification pass grepped an asset manifest that does not exist on the Flutter version that app builds against. It was declared void rather than letting four “No such file” lines and four absent counts read as clean.
A capture that never captured. The
1.9.8+231 cut above, and a more embarrassing one further down this page.
An exit status eaten by a pipe. A pipeline reports
only its last command’s status, so grep -cE "abortRun(" api.dart | wc -l is
grep dying on a malformed pattern with status 2, followed by wc printing
0 and returning success. Measured in both shells here: status 2 direct, status 0
piped, the true value still in pipestatus at index 0 or recoverable with
pipefail. Not a bug — the specified behaviour of pipelines, in the service
of a false number.
The limit case is a query that cannot match at all: we once tested
whether a background worker was alive with pgrep -f <agent-id>, and agent
identifiers appear in no process command line, so it returns exit 1 whether the worker is alive
or dead.
We had the mechanism wrong, and the truth is worse
Our own commit message, recording the fix for the worst of these, explained the cause like this: POSIX performs the command substitution before the redirection. It is a satisfying sentence — a rule, with a standard behind it. We read the standard while writing this post, and it is wrong on two counts.
The Open Group’s Shell and Utilities volume lists redirections as step 3 of processing a simple command, and variable-assignment expansion — where a command substitution on the right of an assignment happens — as step 4. The stated default order is the opposite of what we claimed. The same section then permits those two steps to be reversed when no command name results, and a following subsection adds that a command with no command name performs its redirections in a subshell whose relationship to a command substitution’s subshell is unspecified. A line consisting only of an assignment has no command name, so the construct sits exactly in the region the standard declines to pin down.
What we observed, on one machine: bash 3.2.57 as macOS ships it,
/bin/sh and bash --posix all produce a zero-byte capture file and let
the stderr escape to the shell’s own stderr, while zsh 5.9 — including under
emulate sh — and dash capture correctly. Both behaviours are conformant.
That is worse than a universal bug, which fails for everybody on the first attempt and
therefore gets found: this one works while you are developing the check and stops
working when the check is automated. The same line captures at a zsh prompt and
silently does not inside a #!/bin/bash script, a /bin/sh, or a
Makefile recipe. Write the redirection inside the substitution —
out=$( { cmd; } 2>err ) — and every shell we tested agrees.
Our own defensive habits manufacture the zero
Here we had to correct ourselves a second time, and it improves the
argument. We had been saying that an unescaped ( “returns a silent
zero.” It does not. Bare, it is loud: BSD grep prints parentheses not balanced
and exits 2, and the ugrep build our harness substitutes at its top level prints a caret
diagram at the offending character and also exits 2. Neither prints a number at all. Grep is
not the villain here.
The zero is manufactured downstream, by three idioms that are each
individually reasonable. 2>/dev/null, which is how you keep a
multi-line sweep readable — discard the diagnostic and the sweep prints an empty line
where a count belongs, indistinguishable in a wall of output from zero.
${n:-0}, the defensive default, and the sharpest of the three
because it prints a literal 0 that no tool ever produced: the count did not come
back low, it did not come back. And | wc -l, which buys a printed
0 and a successful exit status in one character. What indicts us rather
than the tools is that the assignment form does preserve the status — we measured
it — and it is simply never consulted, because the script asked for a count and got
something shaped like one. grep -cF for literal anchors sidesteps the whole
class.
The fix can also be undone by tidiness elsewhere. A bare substring
search for ios once returned nine hits across Embergrid’s Spanish and
Portuguese store listings, every one inside an ordinary word — anuncios,
anúncios, contrarrelógios — where the
word-boundary form returned zero genuine hits alongside two known-present control tokens. But
\b is locale-sensitive: under LC_ALL=C the accented false positives
return, so such a guard has to pin LC_ALL=C.UTF-8 rather than inherit it.
A bite-proof interrogates the predicate. A control interrogates the measurement
Two days ago we published an argument that sounds adjacent to this one and is in fact orthogonal. In a guard nobody has seen fail we argued that a check which has only ever been observed to pass is indistinguishable from a check that cannot fail, and that the remedy is to break the condition on purpose, watch the guard go red, and put it back. That is a claim about the predicate — is this logic wired to the ground truth it believes it is. This post asks a different question: did the measurement happen at all. One interrogates what the check decides. The other interrogates whether the check ran.
Neither substitutes for the other, and we can prove it in the least
flattering form available: our own bite-proof harness fell to this exact failure
class. A harness written specifically to demonstrate that a new guard bites captured
its stderr as out=$(…) 2>file. Every capture file was zero bytes. Four of
its assertions then failed spuriously and one passed spuriously — the assertion
that stderr had been silent, satisfied by a file nothing had ever written to. The same run
produced three distinct broken instruments, not one.
The harness’s predicate was correct. Breaking the condition on purpose would have demonstrated nothing, because the condition was never being read. The only thing that catches it is a control on the capture mechanism itself, which is what the fixed harness now ends with: an assertion that each capture file is non-empty before any verdict is drawn from it. Stated generally: whenever a check’s pass condition is an emptiness or a zero, verify that the instrument produced output at all. And the 4-to-1 split is itself the argument for a control over vigilance: noisy false failures get investigated within the hour, while a false pass ships precisely because it asks for nothing.
None of them was caught by a test going red
We reconstructed seven of these incidents from our primary records to write this post, and the through-line is uncomfortable enough to state flatly. Three were caught by the agent that made the mistake, reporting itself: the harness commit carries its confession as its own section heading (its harness produced a false pass and it caught it); the four-pin sweep is written up under a false negative in my own instrument, recorded because it is the interesting part; and the third, published separately, is an engineer confirming that a stranded locale file really was Arabic by scanning every value for Arabic characters — which the generator that produced the file appends to every string. The wrong basis, inside the finding that coined the rule against it.
Two more surfaced because somebody was reading state for an unrelated reason. None of the seven was caught by a test going red. That is not a boast about our review culture and not an indictment of our suites — it is structural. These defects live in the measuring apparatus, and a test suite sits downstream of it, able to report only what the instrument says. A green suite is a statement about what was measured, conditional on a measurement having happened, and nothing asserts that condition. Earlier today we published the same silence from the other direction, in two homes for one rule, where two consecutive fixes landed on a stale copy of a rule and both shipped green.
Two controls, because one of them is not enough
The remedy states in a sentence: grep a string you know is present
in the same invocation as the claim, and report both counts. If the control reads 0, every other
number in that pass is void, and the right response is to repair the pipeline and re-run rather
than interpret anything. We built that check and broke it on purpose four ways — a typo in
the artifact’s filename, strings pointed at a directory, a dump too large for
an external command’s argument list, and stderr discarded so that none of them printed a
diagnostic. Every time the control read 0 and the check exited on instrument broken,
where without it the identical run reads marker count 0 and returns
“production-clean.”
An honest correction on that third one: we could not reproduce the
original incident as our own record described it. Capturing strings output into a
variable and piping echo $S into grep worked at every scale we tried, up to a
4.4 MB dump of 109,002 lines, because echo is a shell builtin and the
argument-list limit never applies. It breaks when the dump goes to an external command
— /bin/echo against a limit of 1,048,576 bytes dies with argument list
too long, and with stderr discarded that is a clean 0. That is a
mechanism we can demonstrate, not a diagnosis of that cut, where what is attested is narrower
and sufficient: the output was captured rather than piped, and every check came back empty.
Now the part that surprised us, and the reason the heading has a plural in it. A control that shares only the pipeline with the real check is not enough. Our working two-control version — a positive control that must be at least 1, a negative control on a token that must exist nowhere, then the marker — was handed a genuine QA artifact and a marker with one underscore missing. Control: 1. Negative control: 0. Marker: 0. Verdict: marker absent and instrument verified — production-clean, exit 0, on a binary that carried the marker. The pipeline was demonstrably alive; the pattern was wrong, and a bundle-identifier control has no opinion about the pattern.
The missing control is the pattern itself: run the same marker
against a reference artifact you know contains it, and if that reads 0 the pattern is wrong
however healthy the pipeline. Four counts in one pass, all through one helper —
cnt() { strings -a "$1" 2>/dev/null | /usr/bin/grep -cF "$2"; }:
[ctl]— the bundle identifier in the target binary. Must be 1 or more, or the pipeline is dead.[neg]— a token that must exist nowhere. Must be 0, or the matcher is over-matching.[pat]— the marker in a reference artifact known to carry it. Must be 1 or more, or the pattern is wrong.[real]— the marker in the target binary. The actual claim, and the only one of the four allowed to be 0.
Direct pipes rather than a captured dump, -cF so no
metacharacter can misfire, and an explicit path to grep because on this machine the bare name
resolves to two different programs depending on how deeply nested the shell is. Any of the
first three counts off its expected value exits 3 and refuses to render a verdict at
all: a broken instrument must not be permitted to produce an answer, because the answer it
produces is always the flattering one. And yes, that helper discards stderr — the first
idiom this post indicts. It is the one place the redirect is safe, because here the detecting is
done by the control counts, not by the diagnostic.
One further admission, because it is the difference between a demonstration and a coincidence. Our first attempt at the mistyped-marker case used a truncated marker, one underscore short, and it reported correctly — we briefly mislabelled that run as a false pass. A truncated marker is a prefix of the real one, so a fixed-string grep still matches. A wrong pattern only produces a false clean when it is not a substring of the true one. Which gives the design rule: a control’s diagnostic power is exactly proportional to how much of the real check’s machinery it shares. Share the pipeline and you catch a dead pipeline; share the pattern and you catch a wrong pattern; share neither and you have added decoration.
The zero worth trusting is the one with a witness
The whole discipline costs three extra counts beside the one you wanted, and a reference artifact somebody has to keep around — a rounding error against the failure it prevents. The reason to make it a rule rather than a good habit is the thing we keep relearning: every zero in this post was produced by a competent engineer running a check they had written deliberately. Nobody here was careless. A guard whose failure mode is indistinguishable from a pass does not need more care. It needs a witness.
So name your instrument, and prove it was awake. A zero is not an answer — it is a reading, and a reading with no witness is a sentence about your tooling that you have chosen to read as a sentence about your code.
Notes
This is a first-hand account drawn from four of our own codebases, from the records as they stood on 2026-07-30. It carries no performance metrics, no revenue figures and no defect rates, which is deliberate: a post about false numbers has no business carrying an invented one. The incident figures come from our primary records; the shell behaviour comes from experiments run for this post and pasted from the terminal.
The tested limits, because they bound the claims above. Every shell
result was observed on a single machine under zsh 5.9 and bash 3.2.57 as macOS ships it; that
was the only bash available, so whether bash 5.x resolves the assignment-and-redirection case
the same way is untested and the claim should not be generalised. The original
captured-variable incident was not reproduced as recorded — the argument-list mechanism we
demonstrate produces the same observable, and is not the diagnosis of that cut. The locale
caveat used a constructed word, because no common Spanish or Portuguese word places
ios immediately after an accented letter, so the LC_ALL=C false
positive is real but its real-world frequency is unmeasured. And the control experiments ran on
synthetic fixtures rather than real store metadata or real shipped binaries, on purpose: other
work was live in those repositories at the time.
Two of the incidents come from a project of ours that has not been released and has no public name yet, so it appears here without one. Embergrid has not shipped either — its store metadata exists because we author listings before release, not because you can download it. NeuralSpark is our brain-training app, and nothing here is a claim about what it does for anyone who uses it: this is a piece about verification tooling, and it makes no cognitive, health or performance claim of any kind. Our games are free and ad-supported, with a single optional Pro purchase that removes the ads.
Sources
- The Open Group Base Specifications Issue 8 / POSIX.1-2024 — Shell Command Language, §2.9.1 “Simple Commands”. Supports exactly three things in this post: that redirections are step 3 of processing a simple command while variable-assignment expansion is step 4; that the order of those two steps may be reversed when no command name results; and that a simple command with no command name performs its redirections in a subshell environment whose relationship to a command substitution’s subshell is unspecified. It is why we describe the bash-versus-zsh divergence as unspecified behaviour rather than a bug in either shell. It says nothing about what any particular shell actually does — that part is measurement.
- Tool versions, observed on the host these experiments ran on, because a
claim in this family that does not name its instrument is not a claim: zsh 5.9; bash
3.2.57(1)-release as shipped by macOS; BSD grep 2.6.0-FreeBSD at
/usr/bin/grep. That last one earns a sentence of its own: at the top level of our agent harness the bare namegrepresolves to a shell function that re-executes as ugrep 7.5.0 in basic-regex compatibility mode, while any child shell — a script, a Makefile recipe, abash -c— gets BSD grep instead. Two different greps depending on how deeply nested you are, which is why we wrote every experiment against an explicit/usr/bin/grep. If you take one operational habit from this post: name your instrument, then check which one you actually got.
Keep reading: all posts on the WizusLabs Engineering blog.