Yesterday's post described how we fixed a class of consensus bugs by embedding the previous block's precommit signatures directly in each block header. Every node now reads the same bytes for reward math, no more divergence from local memory.
That post was true and the fix works. This post is about what we found the next day when we looked at a specific validator's balance.
The setup
Six days ago we ran the one-liner installer on a fresh Hetzner VPS in Hillsboro. Standard flow — curl -fsSL testnet.asentum.com/install/validator | bash, wait for auto-bond, watch it appear in the validator set. It sat there for the next six days, bonded 51,000 ASE, marked active on the API.
Today I checked its balance. 5.5 ASE. That's the leftover from the initial faucet drip minus the bond transaction fee. In six days it had earned nothing.
That's wrong. If it's signing blocks, it should be earning rewards. If it's not signing, it shouldn't be in the active set.
The mismatch
Every block on Asentum carries a lastCommitBitmap, one bit per validator in the committee, set for validators whose precommit landed on the previous block. It also carries the actual LastCommit cert — the signatures those precommits produced. The bitmap is a summary. The cert is the truth.
Reward distribution reads the cert. For each signature, it looks up the signer's address, looks up their stake, adds their share to a payouts map. Then it applies the payouts.
I pulled a recent block and inspected both. The bitmap had seven bits set. The cert had three signatures. All three baked validators, no externals.
Hillsboro's bit was in the bitmap. Its signature was not in the cert. That's why it had earned nothing.
Why
Externals on Asentum run in "pull-mode." They can't accept inbound consensus traffic (most home operators are behind NAT), so instead they poll a small number of baked relay nodes for the current proposal and votes. When they see quorum forming, they sign locally and POST their precommit back to the relay.
This is slightly slower than direct gossip. In practice, by the time an external's precommit arrives at the proposer, the proposer has already reached quorum from the baked signatures alone and committed the block.
The engine has a buffer for these late precommits. When one arrives, it adds the validator's address to a lateCommittedPrecommits set, which is used to set the corresponding bit in the next block's bitmap. That's how the address shows up as "signed" from the bitmap's point of view.
But we were only saving the address, not the signature. The next block's bitmap said seven signers. The next block's cert still had three signatures.
Reward math read the cert. Externals got zero.
The fix
Add a second list on the consensus engine: lastCommittedLateVotes. When a late precommit arrives, push the full signed vote onto this list. When the producer builds the next block's cert, union the primary committed-votes with the late-votes and use both.
This list is intentionally ephemeral. It's not persisted to disk, it's not read on restart, and it's cleared when the next height starts. The reason is subtle: Phase B, our validator liveness tracker, uses the bitmap to update "when did this validator last sign." If late signers ended up in the persisted cert on disk, they'd survive a restart, get counted as "recently active," and inflate the quorum threshold denominator. On a cluster where one baked is momentarily slow, that inflation is enough to wedge consensus.
We hit exactly that failure mode when we first shipped the fix without the ephemeral separation. Chain wedged for ten minutes at h=70,640. Rolled back, redesigned, redeployed. Second version held.
Verified
Hillsboro's balance climbed from 5.5 to 8.5 ASE in the minute after redeploy. Every external whose bit lands in the bitmap is now getting paid. The three baked are still getting the majority of each block's emission because their stake is 500x larger, but externals now accrue at the correct proportional rate.
What this fixed for us
Six days of one validator earning nothing wasn't the story. The story is that we've been telling users "install the operator, bond some stake, earn rewards" for weeks, and for most of them the reward path was silently broken. Anyone who set up an external validator saw their bonded balance stay flat and probably assumed the network wasn't producing enough emission, or that the docs were wrong, or that they'd misconfigured something. It looked like a UX problem. It was a chain bug.
The class of consensus bug (divergent reward math from local engine state) is closed. The specific bug (late signers not credited despite bitmap inclusion) is closed. Externals whose bits land in the bitmap now show up in payouts.
We check the balance of every validator in our test set weekly now. It's a cheap check to keep in the loop.