External validators were stuck at 60% participation and nobody could say why. This is the story of a week spent finding out — and of a bug where the chain was right and every screen showing it was wrong.
Measure first
The temptation with a number like 60% is to guess. We'd been burned by guessing before, so this time we built the instrument first: a metrics endpoint on every block producer that records, per validator, when each vote arrives versus when the producer seals it into a block's reward bitmap. Arrived in time, arrived late, or never arrived — with timings.
The first thing the data killed was our favorite theory. We thought externals were losing because they all polled a single overloaded upstream node. So we A/B tested it: one external spread across all four producers with a smarter fetch strategy, one left alone as a control. Both ended up at the same 60%. The upstream wasn't the bottleneck.
What the data actually showed: an external's vote can never win the race as things stood. Proposals are only served to pull-mode validators once a block is finalized, so their vote for a block always arrives after the network has finished with it. The window to still make it into the reward bitmap was structurally one block slot wide. Over HTTP, across an ocean, they lost almost every time.
Two fixes, both boring on purpose
First: the producer now seals the reward bitmap one round behind the tip, reading it fresh at seal time so late-folded votes count. No timers, no waiting, zero cost to block cadence — the same blocks come out at the same speed, they just carry a fairer bitmap. On the test rig this took a latency-injected external from 66% inclusion to 100%.
Second: a vote that arrives at one producer over HTTP now gets re-broadcast to the other producers over our transport layer. One hop, deduplicated, signature-checked at every ingest like any other vote. Without it, a community validator that only knows about one upstream landed in that producer's blocks and nobody else's.
Both shipped the same way everything ships now: purpose-built devnet rig proving the failure and the fix, then a rolling deploy, one node at a time, chain never pausing.
Then the dashboard called our best validator dead
Days later the explorer showed half the externals dormant — including boxes we could watch voting successfully in real time. Meanwhile a validator whose VPS had been switched off for days showed 60% participation. A dead machine, out-signing live ones.
The reward bitmap is packed against the deterministic committee — the exact list every node derives from staking state. The chain got that right everywhere it mattered: block production, finality, reward payouts. But four reader paths — the participation endpoint, the dormancy judge, peer admission, and a boot-time bookkeeping step — decoded that same bitmap against a locally filtered copy of the committee that dropped inactive members.
A filtered list is the same list until the filter actually removes someone. The moment the first dead validator got dropped, every bit after its position slid onto the wrong address. Live validators' votes were credited to dead ones. The dormancy judge then read its own corrupted output and demoted the live validators for "not signing." The screens all agreed with each other, because they all shared the same wrong decode.
The part worth saying twice: the money was never wrong. Payouts decode against the deterministic list, same as packing. Every reward across the whole incident went to the validator that earned it. Only the humans were misinformed.
All four readers now use the deterministic committee, and there's a rig that bonds a validator, kills it, lets it get dropped, and asserts every live vote still lands on its own index. The explorer now shows live externals at 77–82% and the dead boxes at zero, which has the notable property of being true.
The lesson that keeps recurring
One list, two representations, five consumers. Every consumer must derive it the same way or the bit positions mean different things to different readers. We have a rule about this for consensus state and it's why the chain never forked through any of this. The bug lived in the code we'd mentally filed as "just display" — and "just display" turned out to include the judge that decides who stays in the committee.
Nine days, one genesis, sixteen validators bonded, externals earning what they actually sign. The instrument that started this week stays on permanently — the next mystery number won't get to hide as long.