EVM & state (design)

This page is the design document for UMNode's execution subsystem: an embedded EVM plus a state store with history. All five milestones exist in code: the execution engine (evm package), the state store (store/state), the execution worker (--execute), the calls surface (eth_call, the state getters, eth_chainId), and the trace methods (debug_traceBlockByNumber/ByHash, debug_traceTransaction with the callTracer) — see JSON-RPC. What remains is the follow-up list at the end of Milestones. Where a decision has been made it is marked decided; open points are collected at the end.

Goal

Serve the state-dependent RPC that the store alone cannot answer:

• eth_call — arbitrary read-only calls, e.g. token balances of any standard (ERC-20 / 721 / 1155 balanceOf, or any bespoke contract), plus the cheap point-read getters (eth_getBalance, eth_getCode, eth_getStorageAt, eth_getTransactionCount).
• debug_traceBlockByNumber / debug_traceBlockByHash / debug_traceTransaction — with the callTracer only. Other tracers (including the default struct logger) are rejected with a clean error.

And do it compactly — the same discipline as the segmented block store: store only what cannot be derived, compress what is stored, and bound what the operator does not need.

UMNode is a universal node: mainnet Ethereum is the test target, but the design must hold for any EVM chain, with OP-stack families as a planned extension (see Universality).

The shape of the problem

Traces are not stored by anyone — not even archive nodes. The trick reth uses (and we copy) is to split chain data by mutability and re-execute on demand:

Data	Mutability	reth	UMNode
Headers, bodies, receipts	immutable	static files	segmented block store (exists); receipts optionally derived into the receipts stream
Latest state	mutated every block	flat KV tables	planned: store/state
State history	append-only deltas	changesets + history index	planned: store/state
Traces	derivable	re-executed on demand	re-executed on demand

A debug_traceBlock call is then: reconstruct the state as of the parent block, re-run the block through the EVM with callTracer hooks attached, return the frames. The EVM work is cheap; what makes or breaks it is fast historical state reads — covered below.

The catch: state can only be built forward, and — decided — it is always built from genesis: correctness comes from executing the whole chain locally, never from importing someone else's state. Since the block store fills backward from a trust anchor, state-dependent RPC stays unavailable (an explicit state not ready error, not an approximation) until backfill has reached block 0 and execution has caught up — see Genesis anchor & state readiness.

Design at a glance

flowchart TB
    subgraph existing[Exists today]
        RT[Realtime worker]
        BF[Backfill worker]
        ST[(Segmented<br/>block store)]
        EX[Execution worker<br/>EVM, one block at a time]
        SS[(State store<br/>flat state + changesets<br/>+ history index)]
        TR[Trace/call handlers<br/>re-execute on demand]
    end

    RPC[JSON-RPC server]

    RT -->|Put blocks| ST
    BF -->|Put blocks| ST
    ST -->|next block after state head| EX
    EX -->|updates + changesets| SS
    SS -->|historical reads| TR
    ST -->|block to re-execute| TR
    TR --> RPC
    SS -->|latest state reads| RPC

The execution worker trails the realtime/backfill fronts: whenever the block after the current state head is present in the store, it executes it and advances. RPC reads never block imports — they read at the last committed state head through a database snapshot.

State store

Schema v3: changesets moved to a segment store

As of schema v3, per-block changesets no longer live in the hist/ Pebble instance — they are stored in an immutable, detachable segment store under data/state/changesets/ (one content-hashed blob per block). hist/ is now index-only. The A/S changeset tables and the block-first / pruning / read-path notes below describe the pre-v3 layout and are kept for context; see State history as segments for the current design, why changesets fit a segment store far better than an LSM, and the detach/re-attach tier it enables.

Two Pebble instances under data/state/, split by access pattern (separate from the pending area, the hash index, and the changeset segments), each carrying the identity record locking networkID, genesisHash and schema version — same pattern as the block store manifest:

• flat/ — the latest state and metadata: read-hot, dominated by point lookups (and misses). Pebble's default block compression: Snappy framing, encoded and decoded by S2 (the root go.mod replaces golang/snappy with the klauspost drop-in, see Block store).
• hist/ — the history index (per-key ascending block lists): write-mostly, point-read on historical lookups. The same S2-encoded Snappy by default; --state-db-zstd switches it to zstd. (Pre-v3 it also held the changesets, now in the segment store.)

A logical block commits as one batch per instance with a fixed ordering (history before flat on Apply, flat before history on Unwind), chosen so that a crash between the two batches only ever leaves orphaned future-history above the head — provably harmless and rewritten by deterministic re-execution. Readers snapshot flat before history for the same reason.

Tables

Instance	Prefix	Key	Value	Role
flat	a	address	nonce, balance, code hash (compact, zero-stripped)	flat accounts — latest
flat	s	address ++ slot	value, zero-stripped	flat storage — latest
flat	c	code hash	bytecode	code, deduplicated
hist	A	block ++ address	account pre-value	account changeset
hist	S	block ++ address ++ slot	slot pre-value	storage changeset
hist	h	address [++ slot] ++ shard	delta-compressed ascending block list	history index
flat	meta	—	state head (num+hash), state floor, schema	bookkeeping

Design notes:

• Pre-values, not post-values. The changeset for block N records what each touched key held before N. This single convention buys two things: point-in-time reads (below) and reorg unwind (apply changesets in reverse).
• Block-first changeset keys make pruning a contiguous range-delete: dropping history below a floor is one cheap Pebble operation per table.
• Code is deduplicated by hash — proxy/clone-heavy chains collapse to a handful of bytecodes.
• No trie. No hashed-state tables, no Merkle nodes. This is the largest single saving versus a conventional archive node; what replaces the state root as the correctness check is described in Correctness without a trie.
• Schema changes mean a rebuild, not a migration. State is derived data: on an incompatible layout change (like the v2 split) the store refuses to open the old directory with a clear message — delete it and let --execute rebuild from genesis.

Read-path performance

Four layers keep the lookup-heavy workloads (catch-up execution and traces) off the disk:

• bloom filters on every level of both instances (10 bits/key — Pebble ships with none by default): a point lookup of a key that does not exist — the dominant operation when the EVM probes never-created accounts — becomes an in-memory filter check instead of index-block seeks across the LSM. Filters enter sstables at write/compaction time, so a pre-existing store gains them gradually; a from-genesis rebuild has them everywhere immediately;
• the execution worker's cross-block read cache with negative entries (described under Execution engine) — the first-line account/slot/code reader, sized by --state-cache-accounts / --state-cache-slots / --state-cache-code (sized for the host; the static defaults are ~300k / 600k / 64 MB). It sits in front of the block cache, so on a read-heavy range (e.g. the 2016 DoS-spam blocks) a larger cache removes the repeated point-misses before they ever reach Pebble;
• a shared Pebble block cache (--state-db-cache, sized for the host; the static default is 512 MB — Pebble's own default is 8 MB) behind it, keeping index/filter/hot blocks resident for the lookups the first-line cache misses;
• per-debug_trace* request-scoped caches, plus a reusable history iterator per Reader — historical point-in-time reads were dominated by per-key iterator construction (~9× on the historical-read benchmark);
• batched commits during catch-up: the worker executes up to 256 blocks (--state-execute-batch) in memory (reads layer the pending diffs over the persisted snapshot) and commits them as one ApplyBatch — a larger batch amortizes the per-commit cost and coalesces more per-key history appends, at more memory held before each commit. Every block keeps its own changeset rows, but the per-key history read-modify-writes that hot contracts pay in every block coalesce into one per key per batch. The writes go to a plain Pebble batch — not an indexed one, whose per-key index maintenance and apply-time sort made storage-heavy commits (large SSTORE runs and big-contract SELFDESTRUCT wipes) take tens of seconds and stall execution. The batch flushes immediately whenever the next block is not yet available, so a node following the tip commits per block and RPC freshness is unchanged; a crash re-executes at most one batch.
• pre-values captured during execution: a changeset records, for every changed account and slot, its value before the block — which the EVM already read while executing (every SSTORE first loads the committed slot; every touched account is read from the parent view). Those reads are retained write-once in the StateDB and handed to the store on the Diff (PreAccounts/PreStorage), so ApplyBatch no longer re-reads them with a random snapshot point lookup per key — the cost that dominated storage-heavy commits on cold state. Capture happens at the reader boundary, so it stays correct across mid-block destruct/resurrect and account overwrites. Two cases still read the store: a SELFDESTRUCT wipe, whose entire pre-existing storage must be enumerated (execution only touches some slots), and any diff built without capture (genesis, tooling), which falls back to a snapshot plus an in-memory overlay of the batch's writes.
• cached live history shards: each changed account and slot appends the block number to a "live" (still-filling) history shard, which ApplyBatch must first read to extend. Once pre-values are captured, this per-key read is the dominant one left in the commit path, and it repeats every block a hot key changes. Because execution is the sole writer, the live shard of each key is cached in memory (--state-db-hist-cache, default 200000 keys) and read back from the store only on a miss; it is a read-through cache (eviction or a clear only costs a re-read), and any path that rewrites live shards outside the append (unwind, prune, reset, a failed commit) clears it so it never serves a shard newer than the durable store.

Write-path performance

Building state from genesis is write-heavy: every block commits flat updates plus full account/storage changesets plus history-index appends, and in archive mode none of it is ever pruned. Pebble's stock write path is sized for far lighter loads, so both instances raise its defaults:

• concurrent compaction (--state-db-compaction, default = half the cores, min 2). Pebble's own default is a single compaction goroutine, which cannot drain the compaction debt this load generates: L0 fills, Pebble stalls writes, and the stalls lengthen as the store grows — the catch-up rate decays with state size and spends most of its time blocked. More compactors keep pace. (The setting is a ceiling; Pebble only spawns compactions when there is debt, so a caught-up node is unaffected.)
• a larger memtable (256 MB vs Pebble's 4 MB), paired with the 256-block catch-up batch so a batch's writes fit in one memtable, so flushes are fewer and larger, a higher L0 stop-writes threshold (24 vs 12) so a transient L0 spike absorbs the next batch instead of stalling it, and a larger LBaseMaxBytes to flatten the LSM (fewer levels → less write amplification) as the archive grows.
• a much lower L0 compaction trigger (4 files vs Pebble's 500) so the history instance's ascending, non-overlapping changeset flushes are compacted early instead of piling into a 500-file backlog — see the history-store L0 backlog below.

These apply on open, so an existing store benefits immediately on restart — no rebuild needed; compaction simply works the backlog down faster.

Write-stall headroom

A commit blocks the execution worker only when Pebble stalls writes: either all memtables are full and waiting on a flush, or L0 has reached its stop-writes file count and is waiting on compaction. On a fast disk this is rare, but on a host where background flushes and compaction can't keep up with the write stream, commits that normally take milliseconds balloon into multi-second — even multi-minute — stalls, and the worker sits idle most of the time (visible as a high blocked% in the Execution commit stats log line, paired with a climbing histDebtMB/histL0).

These flags size how much burst the store absorbs in memory and L0 before it stalls, trading RAM and read amplification for fewer stalls. Each is 0-means-default, where the default for the memtable size and stop count is sized for the host:

• --state-db-memtable (MB) — size of one memtable before it flushes.
• --state-db-memtable-stop — queued memtables allowed before writes stall. Raising this is the most direct lever against flush-bound stalls; the cost is up to memtable-stop × memtable MB of RAM per instance (flat and history).
• --state-db-l0-stop — L0 file count that stalls writes pending compaction.
• --state-db-l0-compact — L0 file count that triggers compaction (see below).
• --state-db-lbase (MB) — base-level size; larger flattens the LSM.

On a memory-rich host whose disk briefly falls behind (large RAM, datacenter NVMe), raising the two stop thresholds — e.g. --state-db-memtable-stop 8 --state-db-l0-stop 96 — lets execution keep running through a backlog and lets compaction catch up afterwards, instead of stalling the worker the moment the disk lags. It converts hard stalls into graceful degradation; it does not raise the disk's sustained write throughput, so a permanently saturated disk still needs faster storage (or less write volume — see the WAL and pre-value work).

The history-store L0 backlog

The stall thresholds above are about tolerating a backlog; --state-db-l0-compact is about not creating one. Pebble decides to compact L0 primarily from its read amplification — the number of overlapping L0 sublevels — and only secondarily from raw file count, whose backstop defaults to 500. That default is a poor fit for the history instance: changesets are keyed by ascending block number, so every memtable flush produces an sstable whose keys sit above all previous ones. The L0 files never overlap, sublevels stay at 1, and the read-amplification trigger never fires — so files accumulate all the way to the 500-file backstop before a single, huge compaction runs. Meanwhile estimated compaction debt climbs into the tens of GB, and because Pebble rebuilds its L0 sublevel structure on every flush at a cost that grows superlinearly with the L0 file count (under the write mutex), commit latency creeps up the longer the backlog stands — exactly the decay visible as a climbing histDebtMB/histL0 with a near-zero histComps in the Execution commit stats line.

Lowering the trigger to a handful of files (default 4) keeps L0 short: compactions fire early and steadily, and because the data is non-overlapping and ascending they are largely trivial file moves down the LSM rather than rewrites — cheap to run and they hold debt near zero. The flat instance is address-hashed, so its flushes overlap and the sublevel trigger already compacts it well before 4 files; the low file trigger is therefore effectively history-only. Raise it (e.g. --state-db-l0-compact 16) only to batch compactions more coarsely on a host where many tiny compactions cost more than the debt they avoid.

Disabling the WAL

Every write otherwise lands on disk twice: once in Pebble's write-ahead log (so an applied batch survives a crash before its memtable flushes), then again in an sstable when the memtable flushes. For a build that is fully replayable from the block store, the WAL is redundant durability — a crash can simply re-execute the lost blocks. --state-db-no-wal turns it off on both instances, roughly halving write volume and removing the WAL's own fsync/flush traffic; this is the single biggest write-IO reduction for a catch-up/backfill run.

With the WAL off, an applied batch is durable only once its memtable flushes, and the two instances (flat, hist) flush independently. Left to Pebble that could make flat (which carries the head) durable ahead of the history that explains it — a hole re-execution cannot fill. So durability is instead forced at coordinated checkpoints: once either instance has buffered roughly a memtable's worth of writes (--state-db-checkpoint, default ¾ of the memtable), the store flushes hist, records the head in a wal-checkpoint marker, then flushes flat — an order in which flat is never durable ahead of hist. A clean shutdown checkpoints the tail; a crash loses only the blocks since the last checkpoint, which the executor re-runs from the block store on restart.

The checkpoint threshold must stay below the memtable size so a checkpoint flushes before Pebble would auto-flush a full memtable on its own; raising --state-db-memtable makes checkpoints (and flushes) larger and rarer. As a safety net, on open the persisted flat head is checked against the marker: a head above the marker means a single oversized batch auto-flushed ahead of a checkpoint, so the store is refused rather than served with a possible hole — rebuild from genesis in that case.

--state-db-no-wal is for replayable catch-up/backfill only and is rejected together with the realtime worker (--realtime), whose reorg unwinds are not part of the replay-from-blocks contract.

Execution itself is single-threaded by consensus definition (block N+1 needs block N's state), but two side workers put more cores to use:

• a block prefetcher runs ahead of execution, reading and assembling upcoming blocks from the store and recovering every transaction's sender in parallel (ecrecover is 50–100µs per transaction — 10–20ms of every busy block's budget when done inline). The executor's signer lookups then hit the per-transaction sender cache. Parent-hash checks against the in-memory head make stale prefetches (reorgs) harmless: the feed is dropped and rebuilt. When the feed momentarily empties between blocks the executor waits briefly for the next handover rather than dropping to its idle poll, so during catch-up it runs at the prefetch rate instead of quantizing to one buffer per poll interval; the idle poll is only reached on a genuine stall (the next block is not in the store yet — import behind, or the tip).
• batch commits run on a committer goroutine: the commit of batch B overlaps the execution of batch B+1 (at most one in flight — the natural backpressure). Reads layer the committing and accumulating diffs over the persisted snapshot, which is read-idempotent, so commit/snapshot timing cannot double-apply; unwinds and rebuilds wait for the in-flight commit before touching the head.

Sizing for the host

Every memory consumer above has a knob, and no single static value suits both a laptop and a 256 GB server: the stock defaults starve a large host, and the values a large host wants push a small one into swap — where every commit that touches a paged-out block turns into disk I/O, and the catch-up rate decays the longer the process runs as its caches fill. So by default (--autotune, on) the node sizes those knobs for the machine it starts on (package tune):

• What it looks at: physical RAM (capped by the container's cgroup memory limit when one applies), the core count, and whether the build links cgo — with cgo, Pebble allocates its block cache and memtables with C malloc, outside the Go heap and invisible to GOMEMLIMIT; without it (the Docker image), they live on the heap and must fit inside the limit.
• The budget: about 60% of RAM, leaving at least 1.5 GB to the OS, the block store's mmapped indexes and whatever else runs. It is split as:
  • Pebble block cache (--state-db-cache): a quarter of the budget, 128 MB – 32 GB.
  • Memtables (--state-db-memtable, --state-db-memtable-stop): the size follows the host tier (64 / 128 / 256 MB for under 8 GB, under 16 GB, and above), the stop count comes from a 15% share of the budget for the two instances' queues (2–8), so a bigger host rides out a longer flush backlog before the executor stalls. The catch-up batch (--state-execute-batch) is paired with the memtable — one block per memtable MB — so a batch's writes fit one memtable and a WAL-less checkpoint always flushes first.
  • Go heap limit (GOMEMLIMIT, set through debug.SetMemoryLimit unless the environment already sets it): the remainder. Within it the cross-block read cache takes ~30% (one account per two slots, at ~200 / ~240 bytes an entry), contract code 2% (32–512 MB) and the live history-shard cache 4% (200k–2M keys). The read cache is a long-lived, pointer-heavy structure the collector marks in full every cycle, and GOGC=100 lets the heap double before a cycle, so the live set is kept near half the limit for the limit never to drive collection.
  • Compaction concurrency (--state-db-compaction): half the cores, at least 2 — the same rule the store applied before.
• What it never touches: a knob the operator set. A non-zero flag, or a GOMEMLIMIT in the environment, is kept exactly and charged against the budget, and autotune fills only the gaps around it — pin --state-cache-accounts and the slots follow at twice that; pin GOMEMLIMIT and the read caches are sized from it. When the pinned values alone overshoot the host, startup logs a warning naming them, so a run that is tight says why.

The chosen values are logged once at startup (Execution sized for this host, with the detected RAM, cores and budget). --autotune=false restores the static defaults (512 MB cache, 256 MB memtables × 2, 300k / 600k entries, 256-block batches) and leaves the Go heap limit alone. On a platform without a RAM probe (anything but Linux, macOS and Windows) the static defaults apply with a warning.

Laptops sleep

Autotune keeps the resident set inside RAM, but it cannot stop the host from suspending the process. A sleeping MacBook pages the node out; on wake the first commits take tens of seconds while the pages come back, then recover on their own — a restart only re-executes the blocks since the last checkpoint. Run a long backfill under caffeinate -i to keep the machine awake.

Reading state at any block

"Value of key K as of block N" (N = post-state of block N):

1. History index: find the first block B > N where K changed (shard seek). Shards are uvarint-delta block lists rather than reth's roaring bitmaps: history only ever appends monotonically increasing blocks and only answers first-entry-greater-than queries, so a delta list is both smaller and simpler.
2. Found → the answer is the pre-value in B's changeset — one point read.
3. Not found → K has not changed since; read the flat table.

Two point reads worst case, no tree traversal, independent of chain length. This is exactly reth's account/storage-history mechanism.

Pruning: --state-history

Decided: archive by default. History retention stays an operator knob, mirroring the bounded-backfill philosophy:

• --state-history=all (default) — full archive: every block from genesis is traceable, forever.
• --state-history=N — opt-in rolling window: changesets and index shards below head − N are range-deleted in the background. eth_call at recent blocks and the tip always works; traces below the floor return a clean "state history not available below block X" error.

Growing the window. A node restarted with a larger --state-history (or all) wants history it pruned: the sealed changeset segments the window now covers come back from the segment archive — the node's own backup, or any archive of the chain — top down from the floor. Each segment is adopted into the changeset store, its blocks are filed into the history index, and the floor is lowered to it, so reads become available segment by segment and progress survives a restart. The extension stops at the first segment the archive lacks, saying so; without an archive the node says once what cannot come back. --receipts-history grows the same way (no index to rebuild). Re-execution is never an option here: the state the pruned blocks started from is gone.

Trace availability is bounded by both stores: re-executing block N also needs its body from the block store. The block pruner (--block-history N) keeps the two in step by being the master retention: --state-history is capped to N, so the state floor never lies below the block floor, and the pruner never drops a block the executor has not run yet, so execution never finds its next block gone. With --backup, neither floor passes a sealed segment the archive does not hold yet: the segment stays — and the floor with it — until it is backed up. With --extra-archive-blocks the history floor also never passes the block before the lowest extra range — the state its first block executes on: a read at a block needs every change above it, so state reads and traces inside the ranges need the history continuous from there to the head — the executor's prune stops at it, and the history extender fetches what was pruned below the old floor back from the archive.

Receipts

Receipts are the one piece of chain data that is both fetched over the wire and reproduced by execution, so the node gets to choose. --receipts (default derive; fetch and none are explicit choices, and auto — the former default — follows --execute) decides:

• fetch — receipts arrive with each block and are stored inline in the block record, as before.
• derive — the fetcher requests headers and bodies only (from any peer, eth/68 included), and the executor hands each block's receipts to the state store, which appends them to a receipts stream: a second instance of the slim segmented changeset store under <state-db>/receipts, one blob per block in the block store's own ReceiptForStorage RLP, with the same pending tail, sealed content-hashed segments and prune-by-file-delete. The pre-Byzantium range stays fetched: its receipts carry a per-transaction state root the trie-free executor cannot recompute (the caveat above).
• none — receipts are neither fetched nor stored. With execution on, a point lookup re-executes the block on request (as for a pruned block); eth_getLogs has nothing to scan.

Both derive and none import blocks without receipts; derive without --execute is the same import with nothing to fill the stream, and says so at start. The block store remembers every such block (a per-block marker while the block is unsealed, folded into a per-segment bitmap at seal — see Receipts backfill), so a later run under --receipts fetch can backfill them: the receipts filler fetches the missing receipts from peers and attaches them in place, rewriting sealed segments, until the store matches a fetch-mode import. Starting light is never irreversible.

Ordering. A block's receipts are appended after its changeset and before the flat head advances, so every block at or below the durable head has its receipts (or had them pruned); an unclean stop can only leave the stream ahead of the head, which the open-time reconcile rolls back. The WAL-less checkpoint fsyncs the stream before the head marker and raises its seal ceiling with the changesets'. A reorg unwind rewinds the stream's tail with the state; a rebuild resets it.

Retention. --receipts-history N prunes the stream on the same cadence as the history prune: the floor is recorded first, then every sealed segment entirely below it is deleted. Reads below the floor fail with receipts pruned; the RPC layer re-derives a pruned block's receipts for a point lookup (re-executing it on its parent's state) and refuses a range scan — see Receipts and logs. With --block-history N the window is capped to N: receipts for a pruned block could not be rendered anyway. The segments covering an extra block range (--extra-archive-blocks) stay below the floor and serve the range's blocks; the rest below the floor reads as pruned.

Upgrade. A store built before the stream existed — or with receipts fetched — gains the stream on the next start: it begins at the next segment boundary above the executed head (a partial first segment could never seal), and earlier blocks keep serving their inline receipts, or re-derive when the block record has none.

Execution engine

Decided: a custom vm.StateDB implementation (rather than embedding geth's core/state.StateDB over a stubbed trie). geth's EVM interpreter (core/vm) is consumed as-is — it is already a dependency — but the state interface it executes against is ours:

• An in-memory journaled overlay per block: snapshot/revert, gas refunds, access lists (EIP-2929/2930), transient storage (EIP-1153), selfdestruct semantics (EIP-6780). The journal is the fiddly part — its semantics are ported from geth and differential-tested against a reference node (execute the same blocks, byte-compare receipts) plus the ethereum/tests state-test corpus.
• A StateReader abstraction underneath with two implementations: latest (flat tables) and historical (history index + changesets). The EVM, the journal and the tracers do not know which one they are running on.

This choice keeps every storage access on our flat KV path (no hidden trie code paths), and it is what makes the chain-rules seam below possible.

Around the bare EVM, the block-execution loop owns the consensus-adjacent steps, all reusing geth helpers where they are exported:

• pre-merge ethash block & uncle rewards (era schedule) and the DAO irregular state change;
• post-Shanghai withdrawal credits;
• post-merge system calls: EIP-4788 beacon root, EIP-2935 block-hash history, Prague request operations;
• BLOCKHASH served from segment headers via a chain-context shim.

Rules through Osaka/Fusaka (and the BPO blob schedules) are executed; a block under Amsterdam (Glamsterdam's execution fork — block access lists per EIP-7928 and state gas per EIP-8037, live on Sepolia since 2026-10-06) is refused with an explicit error rather than executed, because the executor neither produces access lists nor verifies the new gas accounting yet. The fork is still in the bundled fork schedule, so the Sepolia handshake stays accepted; execution parks at the first Amsterdam block until support lands.

Read caching. The execution worker keeps a cross-block LRU cache over the flat state — accounts, storage slots and code — including negative entries ("account does not exist", "slot is zero"). Profiling the 2016 DoS range (blocks ~2,283,397+, where one transaction probes thousands of nonexistent accounts via 20-gas EXTCODESIZE) showed ~3/4 of execution time inside Pebble point-miss lookups; with the cache, repeated probes are memory hits. Storage entries are anchored to their account entry by an incarnation number, so a selfdestruct orphans every cached slot of the account in O(1). The cache is folded forward with each applied diff and purged on reorg unwind and rebuild. Synthetic DoS benchmarks: repeated-address attack blocks ~3–4.5× faster; unique-address blocks ~2.7× once warm across blocks.

Sender recovery (ecrecover per tx) costs as much as execution itself on busy blocks. It is embarrassingly parallel, so the worker pre-recovers senders for upcoming blocks in a pipeline; traces do the same per block. If profiling ever shows it still dominating, persisting recovered senders into segments is the known escape hatch (reth stores them; ~20 bytes/tx) — not planned initially.

Throughput expectation

Catch-up execution on modern NVMe should land in the same band as reth's execution stage (order of 100+ Mgas/s): early mainnet blocks fly at thousands per second, recent 15–30 Mgas blocks at a handful per second. A genesis-to-head mainnet build is days, not weeks; following the tip (~1 block / 12 s) is negligible load. Mgas/s, executed height and state-DB size join the metrics registry.

Alongside each Executed blocks report the worker logs an Execution commit stats line for diagnosing where a slow window's time went: commitAvgMs / commitMaxMs (time the committer spent in ApplyBatch), blocked (the fraction of the window execution sat waiting on an in-flight commit — high means the commit/write path, not the EVM, is the limiter), pfWait (the fraction it sat waiting for the prefetcher's next block — high means the block feed, i.e. store reads plus sender recovery, is the limiter), and the history instance's LSM health (histL0, histSub, histDebtMB, histComps, histMem, plus the window's flush counts histFlushΔ / flatFlushΔ). Compaction-bound stalls show climbing histL0/histDebtMB and a memtable count pinned at the stop-writes threshold; commit-compute-bound stalls show a large commitMaxMs while those stay flat; a WAL-less checkpoint stall shows a single huge commitMaxMs with blocked high and the memtable count dropping to 1 in the same window (see below).

Where execution time goes

A profiling pass over an unverified --execute-from catch-up on BNB mainnet (blocks 4.50M–4.75M, Apple M2, 8 cores) established the shape of the catch-up hot path; the method (a 60 s --pprof-addr CPU profile, per-window Mgas/s rather than blocks/s, and --state-debug-no-history as the control) is reusable for any range.

• Judge throughput in Mgas/s, not blocks/s. Across that range gas per block rose 2.5× while blocks/s fell 3× and Mgas/s stayed within ~15 %; a falling block rate is usually heavier blocks, not a slower node.
• History writes are cheap on light blocks and expensive on heavy ones. Over 4.50M–4.75M, --state-debug-no-history ran ~6 % faster — the size of run-to-run noise — with on-disk history 256 MB vs 2 MB. Over the full 4.0M–5.0M it was 19 % faster (1273 s vs 1516 s), and the gap is concentrated: equal or better Mgas/s with history up to ~4.9M, then −24 % on the heaviest band (4.90M–5.00M, ~7 Mgas blocks) where a 256-block commit takes 0.4–1.4 s with history vs ~0.1 s without. blocked stays 0 % there — the commit is still asynchronous — so the slowdown is indirect: a commit in flight keeps inflight populated and doubles the overlay chain every cache miss walks, and the committer's history work (per-key shard read-modify-writes, batch maps, changeset blobs) adds allocation and GC pressure the executor shares. The decisive measurement is a CPU profile in that band with and without history; the overlay collapse below is the first fix either way.
• The one direct write-path stall is the WAL-less checkpoint. With --state-db-no-wal, every checkpoint (after ¾ of a memtable of applied bytes per instance, ≈ every 175k blocks at 4.5M) flushes everything the instances have accumulated since the previous one synchronously inside ApplyBatch: a window with blocked ≈ 45–70 % and a 16–36 s commitMaxMs with history, 3–8 s without (the flat flush alone). Pebble defers background flushes until the queued memtables exceed half a memtable, and the memtable size ramps up from 256 KB, so between checkpoints nothing is flushed at all (histFlushΔ stays 0 until the checkpoint window). Over 4.0M–5.0M the stalls sum to 173 s with history vs 44 s without — half of the whole history cost. Making the checkpoint flush asynchronous (or draining the history instance continuously) is the fix.
• The executor goroutine was only ~60 % busy, and half of that was a store read. Parlia needs the parent header's timestamp; reading it back from the segment store decompressed a whole 64-block group per block, and because the prefetcher runs exactly one group ahead, the two fought over the reader's one-group cache. The executor now serves the parent from the header it just executed.
• Per-block sender-recovery fan-out cost more than ecrecover. Spawning GOMAXPROCS goroutines per block (~100 µs of work each, 1–2k blocks/s) put ~30 % of all CPU into scheduler wake-ups and parks. Recovery now fans out once per run of up to 16 stored blocks, which cut that to ~7 % and let the cgo ecrecover work dominate instead (~1.2 cores at ~16k signatures/s). A cgo-free build (pure-Go secp256k1) was 2.5× slower end to end — keep cgo.
• Together the two fixes took the range from 302–321 s to 235–256 s (≈ 20 %, with run-to-run noise of about ±6 %), with Mgas/s up from ~730 to ~930. GOGC=400 changed nothing further (GC was ~5 % of CPU after the fixes).
• What is left is the single execution goroutine. With the fixes in, pfWait is 13–20 % on the light early blocks and ~0 % on the heavier ones, and blocked is 0 % outside the checkpoint window — so on heavy blocks the executor goroutine is busy for the whole window while the process as a whole averages ~350–400 % of the 800 % available, the disk 10–20 MB/s and the heap ~1.5 GB of 24. Of the executor's own time about half is the EVM proper, a fifth is state reads walking the per-block overlay chain (one OverlayReader per pending and in-flight block, up to 2 × --state-execute-batch layers deep on a cache miss), the rest hashing, RLP and diff bookkeeping. The headroom is therefore structural, not tunable: collapse the pending overlays into one merged map, and spend the idle cores on a state prefetcher that runs upcoming blocks against a throwaway view to warm the execution cache (geth's statePrefetcher pattern) — or on genuinely parallel execution.

Open: commit pressure and the hardware ceiling

The goal for the next round is to drive catch-up execution to the whole machine — every core, the disk's full bandwidth, the RAM — and to make state execution independent of history: the executor's pace must not depend on how expensive the history write path is, on any range. Kernel bypasses (io_uring / direct I/O / memory-mapped stores) are on the table if the measurements point there. Concretely, in the order the numbers above suggest:

1. Decouple the checkpoint from the commit path. The WAL-less checkpoint flushes synchronously inside ApplyBatch (16–36 s with history); make it asynchronous while keeping the hist-before-flat ordering, and drain the history instance continuously so a checkpoint never has a backlog to flush.
2. Take the history write off the executor's critical resources. Even an asynchronous commit slows execution on heavy blocks (−24 % Mgas/s with blocked at 0 %): collapse the overlay chain so commit latency no longer deepens it, move the history batch (shard read-modify-writes, changeset encoding) to its own worker with its own memory budget, and measure with a CPU profile in the 4.9M–5.0M band with and without --state-debug-no-history until the two curves coincide.
3. Fill the idle cores. State prefetching for upcoming blocks, then parallel execution where dependencies allow.

Correctness without a trie

Without Merkle state, the state root in headers cannot be checked. The replacement signal: after executing block N, derive the receipts root and gas used and compare them to the header — the same header fields the fetcher already validates against peers today. Any divergence in gas, logs, status or bloom trips it immediately.

• Pre-Byzantium caveat: receipts before block 4,370,000 embed per-tx state roots, which we cannot recompute. For those blocks the comparison runs modulo the PostState field (the canonical values are already in the store).
• This check is strong but not cryptographically exhaustive — a wrong write that never influences gas, logs or any later execution would slip through. Accepted trade-off for a compact node; periodic spot-checks against a reference node's eth_getProof can be added as a cheap audit later.

Bandwidth dividend — shipped as --receipts derive: once execution is trusted, receipts no longer need to be fetched from peers at all — they are a by-product of execution. Receipts are the bulkiest wire data the node downloads, so sync gets materially cheaper; see Receipts.

Tracing

Decided: import geth's native callTracer (eth/tracers + eth/tracers/native; the JS tracer directory and its goja dependency stay out). UMNode gets geth's exact output schema — calls nesting, onlyTopCall, withLog — so existing tooling works unmodified.

The integration seam is core/tracing.Hooks: the EVM is handed a hooks struct and does not care who built it. Replacing geth's callTracer with a leaner built-in implementation later is a contained, drop-in change — the RPC surface and the EVM loop do not move. (Planned follow-up, not first milestone.)

Request handling:

• debug_traceBlockByNumber / ByHash: build the historical reader at the parent block, execute the block's transactions in order — each with a fresh tracer, all sharing one overlay so intra-block dependencies resolve — return per-tx {txHash, result} frames.
• debug_traceTransaction: hash-index lookup → (block, index); execute the prefix untraced, the target traced.
• Distinct blocks are independent given their pre-state, so block-range tracing parallelizes across workers naturally.
• Only callTracer is accepted; anything else (including the implicit struct-logger default) returns "tracer not supported".
• Guard rails: --rpc-gas-cap for eth_call, --rpc-trace-timeout per trace request.

Concept: prebuilt call traces

Storing every block's callTracer output instead of re-executing on demand — and what that would do to history and disk — is worked through in Prebuilt call traces (concept). Short version: it is a latency and scan-throughput win, not a footprint win.

Reorgs & unwind

The realtime worker already detects tip reorgs and rewinds the store tail. The state store joins that transaction: unwind = apply changesets in reverse from the state head down to the common ancestor, deleting the unwound changesets and index bits. Because deep reorgs (beyond the tail) already halt the node, unwind depth is bounded by SealConfirmations. The execution worker then simply re-follows the new canonical tail.

Genesis anchor & state readiness

Decided: state comes from local execution, never from a foreign database. A snapshot-import path (seeding flat state from an external reth/geth dump) was considered and rejected: it would rest state correctness on a foreign database instead of on local execution of the canonical chain. UMNode either has a state that UMNode execution derived from block 0, or it has no state — there is no middle ground.

The one qualification is the state snapshot in the segment archive: a node's own derived state, published to its own archive with a content hash and a state digest, and restored by a node that chooses to trust that publisher — which is what --syncmode snap, the default, does with the archive's newest snapshot when the archive publishes one (the public archive carries none; a node that wants to start from a snapshot names an archive it trusts), building from genesis otherwise. It is not an import of a foreign database — it is a backup of what local execution produced, with the digest making it comparable across publishers.

What that requires:

• The genesis allocation joins the network config (geth-format genesis.json; for mainnet the alloc ships embedded in the geth dependency).
• The store must be contiguous from genesis: backfill has to reach block 0 before execution can start walking forward. On mainnet that means re-executing ~22M blocks — days of compute — accepted as the price of a trustless state.
• A chain-checker job (planned) verifies stored linkage genesis → head so execution never walks an inconsistent store; and if the configured backfill range changes between runs, backfill must be (re)triggered to close the new gap, in case a range change is not picked up automatically (planned follow-up).

Readiness semantics. Until all of that holds — backfill not configured to reach genesis (a bounded --backfill-to > 0), still filling, or execution still catching up — the node does not serve approximations: eth_call, the state getters and every debug_trace* method return an explicit JSON-RPC error, "state not ready" (exact code chosen at implementation; distinct from the "state history not available below block X" pruning error). debug_status reports the executed head and what readiness is waiting on. The state floor (genesis, possibly raised by pruning) remains the lower bound of traceability.

Debugging a block range

The one exception to always from genesis is deliberate and marked as such: the --execute-from N / --execute-to M flags bound the executor for a debugging session (re-running a block that diverged, profiling a range, inspecting the state the executor leaves behind). They default to off, and a run without them continues from whatever head the state store has.

• --execute-to M stops the executor after block M: pending work is committed and the state parks there, serving RPC, while the import workers keep storing later blocks. Without the flag the next start resumes from M.
• --execute-from N with an existing state cuts it: the state is unwound to the post-state of N-1 — through the changesets, and, unlike a reorg unwind, through the sealed changeset and receipts segments, which are deleted above the cut (the segment holding N-1 is unsealed back into the tail and re-seals as the refill passes it) — and blocks N upward are re-executed. The block store has to hold the blocks down to N-1 (the anchor's hash comes from there); a state floor above N-1 (--state-history) makes the cut impossible. N=1 is a full rebuild.
• --execute-from N with no state seeds the genesis allocation as the post-state of block N-1, anchored to that block's stored hash (waiting for it to land when the import is behind), with the state floor at N-1. That state is wrong by construction: every balance, nonce, slot and code below N is the genesis one, so no real block would pass verification on it. The store is therefore marked unverified (shown in debug_status under execution.unverified, and warned about at every start), and the executor runs it in unverified mode, built for performance research: the work per block stays what it would be on the real state, and only what validity needs is patched:
  • a sender whose nonce does not match its transaction gets the transaction's nonce (so CREATE addresses match the chain's);
  • a sender short of gas × fee cap + value (+ blob fees) is topped up by exactly the shortfall, before the transaction, so it executes as a funded one would;
  • on BNB Smart Chain, the validator's system transactions run the same way: the nonce is patched, the system address is topped up to the amount a reward transfer or deposit carries, and a system call a stale system contract refuses is recorded as failed rather than halting;
  • the block gas pool is not bounded by the header's gas limit (on a wrong state a call that reverted early on the chain can run to completion, so the real limit would stop the block short of its last transactions); the gas used is still counted exactly;
  • the header checks (gas used, receipts root, logs bloom, swept fees) and the Prague request-queue calls are not enforced. The executor's progress report counts the patched senders and the blocks that would have failed verification.

The resulting state must never be served as the chain's; only a rebuild from genesis (debug_rebuildState, or --execute-from 1) clears the mark. RPC on such a state runs re-derivation and traces in the same mode, so they reproduce what the executor did.

Both take effect on every start they are passed; the node warns at startup while either is set. A reorg unwind, by contrast, keeps refusing to cross the sealed floor, and now does so before touching the flat state, so a refused deep unwind leaves flat and changesets in step.

Universality (beyond mainnet)

Decisions here are driven by the universal-node goal:

• Chain config mapping. The YAML fork schedule maps onto a params.ChainConfig for the EVM rules. NetworkConfig gains a ChainID field (consensus identity for signers/eth_call) — on mainnet it equals NetworkID, on many chains it does not.
• Chain-rules seam. The execution loop is structured as a small ChainRules interface — tx-type decoding, intrinsic-cost/fee logic, receipt derivation, block finalization — with the Ethereum implementation first. This is the load-bearing decision for OP-stack support.
• BNB Smart Chain (done). The first non-Ethereum family. Parlia changes the block procedure rather than the EVM: fees accrue to the system address and are swept by the validator's system transactions, system contracts are re-coded at hard forks, PREVRANDAO stays the difficulty, and BSC has extra precompiles. The executor's block runner (evm.blockRun) applies these at the points bsc does, driven by an evm.Chain that pairs the go-ethereum config with the bnb.Config; the rules themselves live in the bnb package. The seam proved narrow enough that the Ethereum path is untouched. The three BNB Beacon Chain bridge precompiles, which early history depends on, are a pure-Go port of bsc's validators checked differentially against bsc — see BNB Smart Chain.
• OP-stack (planned, scoped, not in the first milestones): OP chains need deposit transactions (type 0x7E, which upstream geth types cannot even RLP-decode — this touches the sync layer's body validation, not just execution), the L1-fee cost function reading the L1-block-info predeploy, and extended receipt fields. All deterministic from stored data, all behind the ChainRules seam; the likely source of truth is vendoring the relevant op-geth types.

New RPC surface

Method	Needs	Notes
eth_call	state ≥ floor	block tag resolved against the executed head
eth_getBalance / eth_getCode / eth_getStorageAt / eth_getTransactionCount	state ≥ floor	point reads, no EVM
debug_traceBlockByNumber / debug_traceBlockByHash	state history ≥ floor	callTracer only
debug_traceTransaction	state history ≥ floor + hash index	callTracer only
eth_estimateGas	state ≥ floor	later: binary search over eth_call

Every method in the table returns the state not ready error until the node is genesis-contiguous and execution has caught up to the requested block.

Flags: --execute (on by default; =false skips state building), --state-history=all|N (default all), --rpc-gas-cap, --rpc-call-timeout, --rpc-trace-timeout.

Sizing expectations (mainnet, rough)

To be validated during the spike — these are order-of-magnitude planning numbers, not promises:

Component	Ballpark
Flat state (accounts + storage)	~50 GB
Contract code (deduplicated)	single-digit GB
Changesets + history index	~50–100 GB per million recent blocks
Full genesis archive (all)	low-terabyte class

The default (archive) mainnet deployment is low-terabyte class on top of the block store; the opt-in --state-history=N window (e.g. 1M blocks) brings it down to the low hundreds of GB. Small EVM chains are trivially full-archive.

Milestones

1. Spike — ✅ done: the evm package holds the custom vm.StateDB (journaled overlay: snapshot/revert, refunds, access lists, transient storage, EIP-161/6780 semantics) over the StateReader abstraction, plus ExecuteBlock with the consensus-adjacent steps and the in-engine receipts-root/bloom/gas verification. Differential-tested against geth's reference processor (core.GenerateChain executes a merged Cancun chain; our re-execution must produce byte-identical receipts).
2. store/state — ✅ done: a dedicated Pebble instance holding the flat tables, block-first pre-value changesets (incl. full storage wipes on selfdestruct), the sharded history index, snapshot-isolated latest and point-in-time readers (evm.StateReader implementations), reorg unwind, history pruning with floor enforcement (ErrNotReady / ErrFuture / ErrPruned), and identity locking (schema, network id, genesis). Acceptance test: randomized chains — wipes, resurrections, slot clears, no-op noise — verified key-by-key at every height against an in-memory oracle, across unwind/re-apply cycles, pruning and reopen.
3. Execution worker — ✅ done: --execute runs chainsync.Executor, which initializes the genesis state (genesis resolved from the network config: built-ins for mainnet/sepolia/hoodi by hash, or a GenesisFile for any other EVM chain — Ephemery among the bundled ones — always validated against GenesisHash), executes each stored block as it becomes available, and applies the diff. The realtime worker's rewind hook unwinds state to the reorg's common ancestor; a parent-hash check makes the seam race-free. --state-history=N drives a rolling prune. debug_status gained an execution section (enabled/ready/head/floor/lag); Prometheus gained executed-blocks/gas counters and the executed-head gauge. debug_rebuildState queues a full (from-genesis) or partial (unwind-to-block) state rebuild and revives an executor parked on a divergence halt.
4. RPC: calls — ✅ done: eth_call (block-context execution on the point-in-time reader, --rpc-gas-cap, --rpc-call-timeout, revert reasons as error code 3 with the raw data attached), the point-read getters (eth_getBalance, eth_getTransactionCount, eth_getCode, eth_getStorageAt) and eth_chainId, all gated on the readiness errors. See JSON-RPC → State methods.
5. RPC: traces — ✅ done: debug_traceBlockByNumber / ByHash and debug_traceTransaction with geth's native callTracer (exact output schema, onlyTopCall/withLog/timeout config); blocks re-execute on the parent's point-in-time reader with a fresh tracer per transaction (untraced prefix for single-transaction traces), system calls applied untraced, per-transaction --rpc-trace-timeout, every other tracer — including the implicit struct logger — rejected explicitly.
6. Follow-ups — built-in callTracer, OP-stack ChainRules; store jobs: chain checker (linkage verification genesis → head), backfill re-trigger when the configured range changes. Done: local receipt generation (--receipts derive, see Receipts); the block-pruner background job (--block-history, the master retention that caps the trace floor rather than merely respecting it — see Pruning).

Open questions

• Package naming: evm for the engine + store/state for persistence is the working assumption.

Previously open, now decided: state is genesis-anchored only (no snapshot import), and --state-history defaults to all (archive) with pruning as the opt-in.