The last post ended with a reset. This one covers what happened after: the same chain, nine days later, still on its first genesis. 120,000 blocks and counting. That has never happened here before, and the reason it happened is one bug we finally caught in the act.
The freeze that only hit restarted nodes
The morning after the reset, our primary node restarted on its usual snapshot timer and just... stopped. No errors. No crash. Consensus messages still flowing. It applied nothing and voted on nothing, frozen at the height it woke up on, while the rest of the fleet moved on without it.
We'd seen this shape before and blamed a dozen different things for it. This time we could reproduce it: restart any node mid-epoch, watch it freeze. Restart it after an epoch boundary, watch it work. That's a real clue.
The cause was clean once we saw it. With the dynamic committee, membership changes at epoch boundaries, and the code that installs the new committee into the engine only ran when a block apply crossed the boundary. A node booting mid-epoch never applies the boundary block — it starts past it. So it came up holding the previous epoch's committee, and every vote from the current committee failed an eligibility check that, at the time, didn't even log.
Two changes. At boot, compute the committee for the epoch you're actually joining, not the one your disk remembers. And log the eligibility rejection, because a node silently dropping every vote it receives should never be silent.
What that one fix bought
Restarts stopped being dangerous. The snapshot timer fires every four hours and stops the node to tar its state — that's six restarts a day on the primary alone, and every one of them used to be a dice roll. Since the fix, dozens of timer cycles, node crashes, and manual restarts across the fleet, and the chain hasn't dropped a beat.
That's the boring-sounding property that makes everything else possible. The chain crossed 50,000 blocks with external validators signing — the goal we'd been chasing across four testnets. Then it just kept going.
The committee grew on its own
While the chain ran, people kept bonding. Committee went from 4 nodes to 17 without us touching anything: bond 500 ASE, submit an epoch entry, you're producing blocks the next epoch. Several of those are boxes we've never seen, installed with one curl command by people we've never spoken to.
Some of those boxes are also dead. People bond a VPS, get bored, turn it off. It turns out a committee with dead members in it stresses code paths that a committee of always-on nodes never touches. The next post is about what that surfaced — it's the best bug of the whole run.