It's 1am. The testnet has been down since yesterday. We've tried four different recovery strategies. The mempool is full of 60,000 invalid transactions. The block builder is producing empty blocks. Three validators are stuck 200,000 blocks behind the primary. And we just discovered that the bond function we've been calling doesn't match the API that's actually deployed on-chain.
This is fine.
The setup
Earlier that day, we tried to restart the testnet validators after syncing their code. Each attempt uncovered a new problem:
- Attempt 1: permissions error on the data directory. Fixed with
chmod 777. - Attempt 2: genesis hash mismatch. The peers computed a different genesis than the primary because their dependencies differed.
- Attempt 3: mempool spam. The faucet endpoint on each peer created signed transactions, peers forwarded them to each other, duplicates multiplied. 63,000 transactions, zero getting included in blocks.
- Attempt 4: bond transactions failing silently.
Cannot convert undefined to a BigInt. The staking contract API had changed in local development but the on-chain contract was still the old version.
The fix
Stop everything. Wipe all data on all four nodes. Start the primary. Temporarily disable the faucet rate limit. Fund three validators one at a time. Call bond(pubKey, amount) — the old API, not the new one. Wait for confirmation.
Four validators. Block 624. Mempool empty. Chain advancing.
What we shipped alongside
The mempool got a proper fix:
- Deduplication: transactions are keyed by their BLAKE3 hash. The same tx submitted twice is silently rejected.
- Size cap: 10,000 maximum. Beyond that, new submissions are rejected.
- Stale eviction: after every block, transactions with nonces below the sender's current on-chain nonce are removed.
The empty-block bug is gone. Invalid transactions can't accumulate and block valid ones from being included.
The lesson
Testnet exists for exactly this. We found a mempool bug that would have been catastrophic on mainnet — a spam attack could have halted block production indefinitely. We found a deploy process gap that caused validator divergence. We found a contract API mismatch that made bond transactions fail silently.
All of this is now documented, fixed, and deployed. The genesis reset procedure is written up. The mempool has hygiene. The deploy script is on the roadmap.
It's 2am. The chain is producing blocks. Time for bed.