Our machine-wide build coordinator has a flag that says how long to wait for a free build slot. It was declared bare: a name, a type, a default, and no help string at all. Its sibling, the coordinator that serializes our test runs, carried the decisive sentence. Same lease design, same admission behaviour, documentation in exactly one of the two. That asymmetry produced two separate defects, and neither of them was a bug in the code — the code did precisely what it had always done. What was missing was a sentence, and in the absence of a sentence a meaning got invented and then copied into three files that run our releases. Help text is not documentation sitting beside an interface. It is the interface.
Two intervals wearing one name
Every coordinated build contains two
durations that have almost nothing to do with each other. There is how
long you wait to be admitted — the queue, the contention,
the part that depends entirely on what else the machine is doing. And
there is how long the admitted command then runs — the
archive, the signing, the upload, the part that depends on the work.
Our flag bounded the first and had nothing whatsoever to say about the
second. From the call site those two readings are indistinguishable:
--timeout-seconds 1800 looks exactly like “give this
thirty minutes,” and thirty minutes of what is not a
question the syntax is able to answer.
In the code the distinction is unambiguous
and only a minute deep. The value is consumed by the admission path
alone: when the wait is exhausted the tool raises, and it raises
before the child process is launched. The loop that supervises
the admitted command has no total-runtime bound in it at all. Every
os.killpg() in the tool is gated on an external signal
arriving — a cancellation, an interrupt — never on a clock
this flag controls. A reader with the source open gets there quickly. A
reader with only --help got nothing, because there was no
help text to get.
The meaning nobody wrote, and three files copied
Here is what filled the gap. Our deploy
procedure and the release automation in both of our mobile projects all
asserted the same thing: that expiry “calls
os.killpg() on the build tree.” Nobody invented that
carelessly. It is the most natural thing a timeout can mean, and it is
what a timeout does mean in most tools a release engineer has
ever touched. Combine that reading with a default wait of 1,800 seconds
and it produces a belief that is specific, actionable and wrong: that a
lane which builds an archive and then uploads it is running under a
silent thirty-minute ceiling on the work itself.
A belief like that changes engineering decisions. You split a lane that did not need splitting. You raise a number that was never binding. You write a retry around a failure mode that cannot occur, and you reason about a long upload as though something were counting against it. None of that surfaces as a defect, because the tool keeps behaving correctly the whole time. The code and the paperwork disagreed, and only one of them was being read.
The second defect gives the game away
A lead needed to establish what this flag
did, went looking for authoritative evidence, and cited the
sibling tool's --help output for it. That is not a
careless move. It is a reader doing the right thing with the materials
available: the two tools share a design, the sibling's help is specific
and clear, and the sibling's help was the only help in existence. The
citation was procedurally impeccable and substantively about a different
program. When documentation exists in one of two places, a
reader will reach for the one that exists.
That belongs to a family we have written up before — a check that fires exactly on time and then accepts a plausible answer, which is how a running gate is not a lock went unnoticed for several turns. The pattern is not people being lazy with their evidence. It is evidence being available only in the wrong shape, and the wrong shape winning because nothing else was on the shelf.
“One convention” is a claim about the help text too
The build coordinator's own source explains that it follows its sibling's convention so that one convention covers both machine-wide coordinators. That sentence was true about behaviour and false about the interface, and we had been reading it as covering both. Two tools whose flags take the same values with the same semantics and whose help text differs do not have one convention. They have one convention plus one undocumented implementation of it, and the undocumented one is where the next misreading comes from. Behavioural parity is the easy half; it is also the half that tests can see.
We care about this particular knob more than a studio our size probably should, because we hold builds to one build at a time machine-wide on purpose. That cap is deliberate and we do not apologise for it, but it means waiting is the normal case rather than the exception, and the wait flag is therefore something nearly every release path touches. A knob everyone touches is the last place to leave a meaning unstated.
The most valuable sentence is usually the negation
The repaired help does something that reads
oddly at first: it leads with what the flag does not do. It
states plainly that this does not limit the admitted command's runtime.
Then it names the exact expiry contract, because “it times
out” is not a contract — the command is never executed, a
..._TIMEOUT_NEVER_RAN marker is emitted, and the exit code
is 75. Then it tells you how to size the value: for plausible
contention, not for the work. Acquisition is a quarter-second
spin race with no first-in-first-out fairness, and we have measured
waits of roughly fifteen minutes under multi-project load, so the number
you want is a function of how many projects are competing rather than of
how long an archive takes.
The part worth keeping is why a purely positive description would not have been enough. “How long to wait for a lease” is accurate, and it would have prevented nothing, because the wrong reading is also a wait — a build that gets killed at thirty minutes has, from the caller's seat, simply stopped waiting. Only the explicit negation closes the door the wrong reading walks through. Which generalises uncomfortably well: the first line of a flag's help should be whichever misreading it most needs to rule out, not the tidiest statement of what it is for.
Thirty-six lines, no behaviour change
The fix shipped on 2026-10-09 and it is
almost embarrassingly small: thirty-six lines, all of it help text and a
rename. The flag is now --admission-timeout-seconds, with
--timeout-seconds retained as a documented legacy alias
— which is exactly what the sibling had looked like the whole
time. The rename carries more weight than it appears to, because a name
is help text you cannot skip reading:
--admission-timeout-seconds 1800 is much harder to misread
as a ceiling on the work than --timeout-seconds 1800 is.
Keeping the alias mattered too. Breaking every existing call site in
order to repair a documentation defect would trade a known confusion for
a known outage, and that is not a trade we would take.
Verification was proportionate and dull, which is the correct shape for a change like this. The help renders the sentence. The legacy alias still acquires and releases a lease and returns zero. The status subcommand still parses. Nothing behaves differently, and nothing was supposed to.
What a missing sentence actually costs
We have shipped a guard nobody had seen fail and found out later that it could not, in principle, see the file it was guarding. This is the interface version of the same failure. An undocumented flag is not inert; it does not sit unused waiting for someone to look it up. It gets used immediately, with a meaning supplied by whoever reached for it first, and that meaning hardens into the files that run your releases while the code underneath stays blamelessly correct. The gap was never in the tool. The gap was in what the tool could tell you about itself.
No triumphant ending is available. We do not know how many other bare flags we are carrying, and we know this one would have gone unnoticed indefinitely, because both tools kept working perfectly throughout. What we changed is narrow and checkable: when two tools claim a shared convention, help-text parity is part of the contract rather than a nicety, and a flag's help opens on the reading it needs to rule out. More of these small operational post-mortems are on the WizusLabs Engineering blog, and the builds queued behind this lease are the ones that ship things like our Sudoku. The line worth carrying out of it: the repair was help text; the damage was real engineering decisions, taken on a sentence nobody had written.
Notes
This is a first-hand account of our own
tooling, read out of our own repository and the change that shipped the
fix on 2026-10-09 rather than recalled. Every figure in it was read:
the 1,800-second default wait, the exit code
75, the quarter-second acquisition spin, the roughly
fifteen-minute waits measured under multi-project load,
the thirty-six lines in the fix, the
two defects traced to the gap, and the
three files carrying the false expiry claim. The
os.killpg() sentence is quoted as the content of a claim we
had written down, not as a description of what the tool does.
There are no throughput, time-saved, defect-rate or
developer-productivity figures anywhere in this post, because we have
none we would stand behind: we did not measure what the wrong
belief cost us in hours, and manufacturing a number to make the argument
land would be the exact failure the piece is about. Two scope limits
worth stating plainly. The fifteen-minute wait figure comes from one
host under contention we did not control — it is evidence for how
to size a default, not a benchmark of anything. And this is one pair of
tools in one workspace: it is the reason we now treat help-text parity
as part of a shared convention, not proof that doing so prevents the
next version of this.
Keep reading: all posts on the WizusLabs Engineering blog.