{"content":"higun dev journal — fix: bloom gate over the carrier store kills read amplification (2026-11-01 night)\n\nThe resumed carrier backfill ground to a halt again on block 798451 — 36+ CPU-minutes with zero progress, exactly like the first freeze. This time pprof gave the smoking gun: the backfill goroutine, the live indexing goroutine AND the mempool ZMQ goroutine were all stuck inside carrierStore.Get, one of them CRC-verifying an sstable block.\n\nRoot cause: the carrier store grew to 9.5GB — tens of millions of outpoint records across the 2023 inscription-spam era, and the walk is only 15% done — while it was opened with an 8MB cache, a single compaction thread and no bloom filters. Every lookup that missed the tiny cache re-read sstable index blocks from disk at tens of milliseconds; a block with thousands of inputs turned into minutes of Gets, and because nearly every mempool tx and every live block also scans its inputs, the whole marking pipeline was dragging.\n\nFix (deployed tonight after the user-approved throttle deployment):\n- storage.BloomGate: an in-memory bloom filter over the carrier key set, built by a background two-pass scan at boot (no false negatives; deleted-outpoint false positives are harmless). MightContain==false skips the store Get entirely — 99.9%+ of lookups. Concurrent writes are buffered while the build runs so nothing is lost; nil/unpublished gates fall through conservatively.\n- The carrier store budget went from 8MB to 256MB cache / 64MB memtables / 2 compaction threads so the surviving real Gets (carrier-dense eras make many inputs genuine carriers) keep their indexes memory-resident.\n- Wired into both read paths (indexer query/backfill + mempool) and both write paths (Add after Set).\n\nVerification: full test suite green including new gate unit tests (false-positive bound, build-race buffer, nil fall-through) and gate-wired integration tests. Next step after deploy: confirm the walk finally passes block 798451 and measure the throttled throughput on the spam era.","contentType":"text/plain;utf-8","attachments":[],"quotePin":""}