State history as segments (design concept)

Status

Accepted — direction set. We are building this. The deciding advantage is detachability: a sealed history segment is a self-contained file, so it can be cut to reclaim disk and re-attached (re-downloaded) later — exactly like a block segment, and impossible with an LSM. The compaction / write-amplification savings (below) are a bonus on top. See EVM & state for the current design and Block store for the segment machinery this reuses.

The question

Sealed blocks already live in an immutable, segmented, append-only store (Block store): blocks past the reorg horizon are packed into fixed-size, compressed segments (s2 by default) and never rewritten. State history — the per-block pre-value changesets — is just as immutable and just as append-only. So why does it live in a log-structured merge tree (Pebble), which is built for data that mutates?

This proposal: move the changesets into a segment store, the same way block artifacts already work, and keep only the small mutable index in Pebble.

The headline payoff is operational, not just internal: once history is a set of immutable files, you can treat it as a detachable storage tier — drop the oldest segments when the disk is tight, and pull them back from a backup, object store, or peer when you need deep history again. The rest of this page works through the design; the detachable tier section is the reason we are doing it.

What we measured first

This isn't theoretical. A from-genesis archive build spends most of its tail fighting exactly this mismatch:

• Changesets are keyed block-first and written in strictly ascending order, so every Pebble memtable flush produces a non-overlapping L0 file. Pebble's L0 compaction is driven by sublevel (read-amplification) count, which for non-overlapping files stays at 1 — so the normal trigger never fires and files pile up to the 500-file backstop before one giant compaction runs, with tens of GB of compaction debt in between (see the history-store L0 backlog).
• Even with that tuned (--state-db-l0-compact), the store still pays write amplification (rewriting changesets down the LSM levels) and compaction CPU that contends with the single-threaded executor — visible as execution-rate dips whenever compaction fires.

We are paying an LSM's running costs to store data that is written once and never updated. A segment store pays none of them.

Today's layout

The state store is two Pebble instances (EVM & state):

Instance	Holds	Access pattern
flat/	latest accounts, storage, code, head/floor	read-hot point lookups (and misses)
hist/	changesets + history index	write-mostly

The hist/ instance is doing two different jobs, and only one of them is a poor fit for an LSM:

• Changesets — prefixAccountChanges ('A' + block + addr → pre-account) and prefixStorageChanges ('S' + block + addr + slot → pre-value). Immutable, written once, ascending by block, the bulk of the bytes. Read only on unwind and historical RPC — never on the forward execution path, because pre-values are captured during execution (Diff.HasPreValues).
• History index — prefixAccountHistory ('h' + addr + shardMax → block list) and prefixStorageHistory ('H' + addr + slot + shardMax → block list). Random-access by key, read-modify-written every block (appendHistoryMany). Address-keyed, so its flushes overlap and it compacts normally. Small.

The insight is the split: changesets are block artifacts; the index is not.

Proposed design

flowchart LR
    EXEC[Executor] -->|pre-values, ascending| CS[Changeset segment store]
    EXEC -->|flat writes + head| FLAT[(flat/ — Pebble)]
    EXEC -->|per-key block lists| IDX[(index/ — Pebble)]
    CS -->|tail within reorg depth| TAIL[pending Pebble tail]
    CS -->|sealed beyond 256 conf| SEG[sealed compressed segments on disk]
    IDX -. derivable from .-> CS

What moves

The changesets move into a changeset segment store, keyed by block number, exactly like the block store:

• One blob per block. Block N's changeset serializes all of its (addr → pre-account) and (addr, slot → pre-value) entries into a single record. Block-keyed, written once, ascending — identical shape to a block record.
• Mutable tail, sealed body. Recent blocks (within sealConfirmations, 256) stay in a pending Pebble tail so a reorg can rewrite them; once confirmed they seal into immutable compressed segments and are never rewritten. This is the block store's exact lifecycle.
• Reuses store/segment — either directly (a second segment.Store instance with a changeset payload instead of a block payload) or via a small generalization of it. The segment/group/seal/manifest machinery is unchanged.

What stays in Pebble

• flat/ — unchanged. Read-hot point lookups want an LSM with bloom filters; that is the right tool and we keep it.
• The history index — moves to its own small index/ Pebble instance (or stays a prefix range). It is random-access, read-modify-write, and overlapping, so it compacts cleanly. Crucially, it is small (block-number lists, not full pre-values).

A key property: the index is derived, not primary

The history index is fully reconstructable from the changesets — scanning changesets gives, per key, the exact list of blocks it changed. So it is a secondary index, not a third source of truth. That collapses what looks like a three-store consistency problem back to two:

• Primary, must stay consistent: changesets + flat head.
• Derived, may lag and be repaired: the index.

If a crash leaves the index behind (or ahead of) the changesets, it can be rebuilt for the affected block range on open, instead of needing its own durability ordering. (This is also a natural home for a future "rebuild index" maintenance command.)

Crash consistency & ordering

Today ApplyBatch commits history before flat so flat (carrying the head) is never durable ahead of the history that explains it; the WAL-less path makes this explicit in checkpointLocked (flush hist → write head marker → flush flat). The proposal keeps the same invariant, with the changeset store in the history role:

1. Append block N's changeset to the changeset store (pending tail).
2. (index updated; may be repaired later, so not on the critical ordering path.)
3. Make the changeset tail durable.
4. Then make flat + head durable.

A crash between 3 and 4 leaves changesets above the flat head — the same already-safe "orphans above head" shape we handle today; re-execution rewrites them. The WAL-less checkpoint gains one step (flush the changeset tail before the flat flush), but the ordering rule is unchanged.

Reorg / unwind

Unwind is where "immutable segments" could scare us — but it doesn't, for the same reason it doesn't scare the block store:

• Reorgs only reach back within the mutable tail (sealConfirmations = 256). Seal the changeset tail at the same depth and unwind only ever touches the pending Pebble tail, never a sealed segment. A deeper reorg is ErrDeepReorg, exactly as for blocks.
• Unwind reads changesets for [to+1, head] descending by block — a contiguous range scan, which is the segment store's strength (forEachAccountChange/forEachStorageChange already iterate by block range).

Pruning

--state-history N (retain only the last N blocks of history) becomes dropping whole sealed segments below the floor — a file delete — instead of a Pebble DeleteRange that leaves tombstones to be compacted away later. This is strictly better, and it is the same operation the block pruner already performs.

Detachable, portable history

This is the reason we are building it. Because each sealed segment is a self-contained, immutable file, history stops being a monolithic database and becomes a set of movable artifacts — the same property the block store already enjoys.

Detach to reclaim disk. Move (or delete) the segment files for an old block range out of the state directory. The disk is freed immediately; no compaction, no rewrite. The manifest records the range as detached rather than present.

Re-attach to restore. Drop the segment files back into the directory (from a backup, an object store, or a peer). The store verifies each against its recorded content hash, re-indexes it, and the range is live again. No rebuild, no re-execution.

Determinism makes them shareable. A block's changeset is a pure function of chain execution, so every honest node produces byte-identical changeset segments for a given range and codec. That makes segments content-addressable — verifiable by hash and safely fetchable from anywhere (a CDN, S3, or eventually P2P alongside block segments). "Download them again" is a first-class operation, not a rebuild.

Detach vs. prune — two different operations

	Prune (--state-history N)	Detach (new)
Intent	permanently discard old history	temporarily offload to reclaim disk
Reversible	not by itself — the data is gone; a backup brings it back when the window grows (fetched back)	yes — re-attach the file
Floor	raises the readable floor permanently	floor unchanged; range is a gap
Read in range	ErrPruned (never coming back)	ErrDetached (re-attachable)

The distinction matters at the RPC boundary: a point-in-time read into a pruned range is a permanent "not available"; a read into a detached range should say "history offloaded for blocks [x, y] — re-attach the segment", a recoverable condition. Both are unaffected by latest-state reads, which the flat instance always serves in full.

What detaching does and doesn't touch

Detaching old segments never affects unwind (it only reaches within the 256-block reorg tail) or latest-state queries (served by flat/). It only gates historical point-in-time reads inside the detached range. A node can offload all history below block X and still serve latest state fully plus point-in-time reads for blocks ≥ X. The small history index stays resident, so re-attaching a segment needs no index rebuild.

Read paths

Path	Hot?	Today	Proposed
Forward execution	hot	no changeset read (pre-values captured)	unchanged — still no changeset read
Unwind (reorg)	cold	Pebble range scan	segment range scan (tail, uncompressed)
Point-in-time RPC	cold	index point-get → changeset point-get	index point-get → segment read (group decompress)

The only path that gets slower is the cold historical RPC read: locating the first-change-after block via the index, then reading that block's changeset from a segment costs a group decompression (≈groupSize blocks, 128 by default) rather than a Pebble point lookup — memoized per read view, so a reader that touches several blocks of one group decompresses it once. Sealed groups are compressed with s2 at its better tier by default (--state-changeset-compression s2, --state-changeset-compression-level 6; zstd is available for a smaller, slower store), the same codec and tier the block store seals at. Every segment header names its codec, so the codec may change between runs: segments sealed earlier stay readable, and --repack-segments converts them in place for a uniform store. For occasional eth_call-at-height this is acceptable; if it matters, compress changesets at a finer granularity than blocks.

What it buys

• A detachable, portable history tier — cut old segments to reclaim disk, re-attach (or re-download) them on demand, with deterministic content-addressable files. This is the deciding advantage; see above.
• No compaction and no write amplification for the bulk of state history. Changesets are written once and sealed. This removes the dominant background CPU/IO that currently contends with the single-threaded executor — directly recovering the execution-rate dips we measured.
• Smaller on disk. Seal-time grouped compression over large segments beats per-sstable compression (the block store sees ≈14 % from grouping alone).
• Trivial, instant pruning (drop segment files).
• Architectural symmetry — block artifacts and state history use the same immutable-segment model, one mental model and one set of tools.

What it does not buy

It does not raise the single-threaded execution ceiling (~400 Mgas/s on one performance core). The forward path already doesn't read changesets, so this is a write-side / background-cost win: a leaner, steadier, smaller archive — not a faster interpreter. The path to multiplying throughput is parallel transaction execution, which is orthogonal to this.

Costs & risks

• More moving parts. A dedicated changeset store plus the flat and index Pebble instances. Justified only because changesets are a large, cleanly separable, write-once mass — but it is real surface area.
• Checkpoint coordination. The WAL-less coordinated checkpoint must now flush the changeset tail in the right order relative to flat. Delicate, though not more so than today's hist-before-flat rule.
• Cold RPC read regression. Group decompression on historical reads (see above).
• Index rebuild logic. We gain a "repair the derived index" path; it must be correct and tested (it is also a latent maintenance feature, so not pure cost).
• Schema bump + migration. schemaVersion is currently 2 (flat/hist split). This is an incompatible layout change → bump to 3, and since state is derived, the migration is "rebuild from genesis" (no in-place migration), consistent with the v1→v2 policy.

Open questions

1. Reuse vs. fork segment.Store. Decided: a slim parallel store. The block store's Store is typed to blocks (header/body, by-hash tx index); changesets need none of that. Rather than risk a refactor of a tested store, Phase 1 stands up a leaner segmented store for changeset blobs and accepts some duplicated seal/manifest logic. We revisit consolidation only if the overlap proves large.
2. Index home. Separate index/ Pebble instance, or keep it as a prefix in a shared instance? Separate makes "rebuild the index" a clean wipe.
3. Changeset blob granularity. Compress per-block, per-group, or per-segment? Trades historical-read latency against ratio.
4. Does the index even need to be persistent? If rebuild-from-changesets is fast enough, the index could be treated as a rebuildable cache (smaller durability surface) — or even built lazily for ranges that get queried.
5. Sealing cadence vs. the executor. Sealing runs off the hot path in the block store; confirm the changeset seal/re-pack never lands on the execution thread.

Decision

We are building it. The deciding factor is the detachable, portable history tier; the write-side savings (no compaction, no write-amp, smaller on disk, less executor contention) and trivial pruning come along for free. The accepted caveat is that this does not raise the single-thread execution ceiling — it is a footprint/operability win, pursued alongside (not instead of) parallel execution.

Implementation plan

Phased so each step is independently shippable and measurable, and so the risky cutover comes only after the win is proven.

1. Changeset segment store (write path). Implemented — --state-changeset-segments. A slim parallel segmented store (store/changeset) holds one content-hashed, group-compressed blob per block; the reorg-prone tail lives in a bounded pending Pebble and seals past the 256-block confirmation horizon. The state store dual-writes each block's changeset into it alongside the existing hist/ instance; reads still come from Pebble. Off by default.

   Measuring with the dual-write on

   Run a catch-up with --state-changeset-segments and compare on-disk size: du -sh <state-db>/changesets (segments) vs the changeset share of <state-db>/hist. The dual-write deliberately writes to both, so it shows the segment store's size and write cost without yet removing anything from Pebble; the compaction/CPU drop is realized in Phase 2b, when the changeset prefixes leave hist/.

   1. Read cutover. Split into a safe validation step and the irreversible perf step:
   2. 2a — read sealed from segments. Implemented. Point-in-time reads of sealed (immutable) blocks resolve their pre-value from the segment store (history index in Pebble → block → changeset blob → key); the unsealed tail still reads from the hist snapshot, which keeps reorg-prone changesets snapshot-isolated. Genesis is seeded as an empty changeset so the store is dense from block 0; Reset (rebuild) wipes it. Validated by the oracle differential test with a tiny seal geometry. Dual-write stays, so Pebble remains authoritative and the change is reversible.
   3. 2b — drop changesets from Pebble + schema v3. Implemented. Changesets are no longer written to hist/ (now index-only); ApplyBatch builds one blob per block and appends it to the segment store, in the commit order index → changesets (durable) → flat+head that preserves history-before-flat across the two stores. All historical reads and unwind now resolve changesets from the segment store; the Reader captures the index and changeset snapshots together under the store lock, so an index entry it sees always has its changeset present. schemaVersion is bumped to 3 (a v2 store fails identity validation → rebuild from genesis). The changeset store is always on (under <state-db>/changesets); the old --state-changeset-segments flag is a deprecated no-op. On open, the changeset tail is reconciled to the durable flat head.

   Crash recovery in --state-db-no-wal mode — resolved in Phase 3

   WAL-less durability is coordinated: the checkpoint fsyncs the changeset tail before recording the head, and (Phase 3) sealing is capped at the durable flat head by a seal ceiling the checkpoint raises — so a sealed (fsync'd) segment never gets ahead of the head an unclean stop resumes from. Reopen rolls the changeset tail back to the head and continues; no rebuild.

   1. Checkpoint & crash-consistency. Implemented. The WAL-less checkpoint fsyncs the changeset tail (SyncWAL) before the head marker, then raises a seal ceiling to the now-durable head; the changeset store seals only segments at or below the ceiling, so sealing — which fsyncs files at once — never outruns the crash-resumable head. An unclean stop therefore reopens by rewinding the changeset tail to the durable head, no rebuild. Index repair is not needed: the checkpoint flushes the history index before the marker, so the durable index is always complete up to the durable head; orphan index entries above it resolve to the flat value (their changeset, if rewound, is read as "no applicable change").
   2. Prune → drop segments. Implemented. PruneHistoryBelow raises the floor (gating reads to ErrPruned), then changeset.Store.Prune deletes every sealed segment whose blocks are all below the floor — a file delete, no compaction. A segment straddling the floor is kept whole (its blocks at or above the floor are still readable). The manifest is rewritten before the files are removed, so a crash mid-prune leaves only harmless orphan files, which the next open cleans up (orphan .cseg/.cidx not in the manifest are dropped on reconcile). Irreversible — unlike detach (Phase 5). With --backup, Prune first asks the archive uploader's prune guard and stops at the lowest segment the archive does not hold yet, and PruneHistoryBelow records that floor (PrunableBelow) rather than the one asked for, so the floor never runs ahead of the files.
   3. Detach / attach. The headline feature: commands (and/or RPC admin calls) to detach a sealed block range (record it, free the files) and attach segment files back (verify content hash, re-index). Add ErrDetached and the corresponding RPC error distinct from ErrPruned.
   4. Portability tooling. Content-hash manifest entries, export/import helpers, and (later) fetching detached segments from an object store or peer — reusing the determinism that makes segments content-addressable. Upload side implemented: the segment archive backs every sealed changeset and receipts segment up to S3 or a directory alongside the block segments; fetching them back is the attach half of Phase 5.

Phase 1 is the spike; we decide on phases 2+ from its numbers before the schema bump locks anything in.