nc-gcode-interpreter 0.2.3__tar.gz → 0.2.5__tar.gz

{nc_gcode_interpreter-0.2.3 → nc_gcode_interpreter-0.2.5}/CHANGELOG.md

@@ -4,6 +4,40 @@ Notable changes to **nc-gcode-interpreter**. The format loosely follows
 [Keep a Changelog](https://keepachangelog.com/); versions are git tags,
 released to PyPI.
 
+## [v0.2.5] - 2026-07-08
+
+### Changed
+
+- Interpreter throughput on the DataFrame/batch path: the per-output-row `Row`
+  allocations are now recycled instead of freed. Profiling the 1.1 GB → DataFrame
+  conversion showed ~50% of CPU in the system allocator, dominated by allocating
+  and freeing each of the 22M rows' cell buffers. After a batch is built its rows
+  are cleared (capacity retained) into a pool and handed back to the interpreter
+  to refill, bounding live row allocations to ~2× the batch size instead of the
+  whole-file row count. Output is byte-identical; the streaming (`nc_to_rows`)
+  and in-memory collect paths are unchanged. ~10% off end-to-end on the large
+  real-world program (#61).
+
+## [v0.2.4] - 2026-07-08
+
+### Changed
+
+- Interpreter throughput: the hot, closed-vocabulary lookup maps (`axes`,
+  `translation`, `output_keys`, and the table builder's per-cell/forward-fill
+  maps) now use the non-cryptographic FxHash hasher, and `State::update_axis`
+  overwrites in place instead of re-allocating the key on repeat writes. ~17%
+  off the collect (dataframe) path on the large real-world benchmark, with no
+  behavior change. `symbol_table` deliberately stays on the default SipHash
+  hasher: its keys are user-controlled variable names, so it keeps its
+  hash-flooding resistance (#60).
+
+### Added
+
+- Experimental execution-cursor interpreter (opt-in via `NC_VM=1`): the
+  recursive control-flow walk is reified into an explicit frame stack, enabling
+  in-memory checkpoint/resume. Off by default; the standard path is unchanged.
+  Groundwork for resumable/streaming interpretation (#47, #59).
+
 ## [v0.2.3] - 2026-07-07
 
 ### Added

{nc_gcode_interpreter-0.2.3 → nc_gcode_interpreter-0.2.5}/Cargo.lock

@@ -506,7 +506,7 @@ checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
 
 [[package]]
 name = "nc-gcode-interpreter"
-version = "0.2.3"
+version = "0.2.5"
 dependencies = [
  "arrow-array",
  "arrow-data",
@@ -517,6 +517,7 @@ dependencies = [
  "pest_derive",
  "pyo3",
  "rayon",
+ "rustc-hash",
  "thiserror",
 ]
 
@@ -726,6 +727,12 @@ dependencies = [
  "crossbeam-utils",
 ]
 
+[[package]]
+name = "rustc-hash"
+version = "2.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
+
 [[package]]
 name = "rustversion"
 version = "1.0.22"

{nc_gcode_interpreter-0.2.3 → nc_gcode_interpreter-0.2.5}/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "nc-gcode-interpreter"
-version = "0.2.3"
+version = "0.2.5"
 edition = "2021"
 readme = "README.md"
 
@@ -19,6 +19,7 @@ pest_derive = "2.8.6"
 thiserror = "2.0.18"
 clap = { version = "4.6", features = ["derive"] }
 rayon = "1.10"
+rustc-hash = "2.1"
 
 # NOTE: keep every dependency version inline (`crate = { version = "..." }`),
 # NOT as a `[dependencies.crate]` block with a standalone `version = "..."`

{nc_gcode_interpreter-0.2.3 → nc_gcode_interpreter-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nc-gcode-interpreter
-Version: 0.2.3
+Version: 0.2.5
 Classifier: Programming Language :: Rust
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Programming Language :: Python :: 3.12

nc_gcode_interpreter-0.2.5/docs/resumable-interpretation-design.md

@@ -0,0 +1,233 @@
+# Resumable interpretation — checkpoint/resume design (#47, targeting 0.3.0)
+
+Status: **design + proof-of-concept**. This document scopes the resumable-`NcSession`
+feature requested in issue #47 and the interpreter change it requires. A working
+Phase-1 proof of concept — the execution-cursor VM (`src/vm.rs`, opt-in via
+`NC_VM=1`) plus in-memory checkpoint/resume — now exists alongside this design;
+the public API and Phases 2–3 remain future work.
+
+## 1. Goal and consumer requirements
+
+Expose a resumable interpretation handle:
+
+- `checkpoint() -> token` — capture the complete execution position roughly
+  every *N* output rows.
+- `resume(token, patched_text)` — restart interpretation from a checkpoint,
+  against an edited program.
+
+Consumer: ribweaver `/sim` phase-E *incremental re-interpret* for multi-million-row
+loop expansions. When the user edits source line *L*, it resumes from the last
+checkpoint whose block cursor precedes *L* instead of re-running the whole program.
+Downstream is evidence-gated: debounced-full re-interpret is fine below ~10⁵ rows,
+so the resume path only has to pay off for the *giant* case — which is exactly the
+hard one, because a million-row expansion is typically **one** top-level `WHILE`/`LOOP`.
+Top-level-only checkpoints would yield a single checkpoint for the whole expansion
+and be useless; checkpoints must be capturable **mid-loop**.
+
+Cancellation already exists (drop the iterator → `StreamClosed`); resume composes
+with it as the latest-wins primitive.
+
+## 2. Why this is hard — the crux
+
+The issue's premise ("no call stack exists, so block-granularity checkpoints are
+tractable") is half right. There is no *subprogram* call stack (#48), but the
+**control structures form an implicit stack**, and the instruction pointer lives
+on the **Rust call stack**, not in any serializable structure.
+
+Precise findings (`src/interpret_rules.rs`):
+
+- The instruction pointer is `index: usize`, a **stack-local** in `run_blocks`
+  (`:2071-2103`) — one per active scope. There is no IP counter in `State`.
+- Every control construct drives its body by **recursively calling**
+  `interpret_blocks`, holding borrowed `Pair<Rule>` iterators and a `loop_count`
+  as stack locals: `WHILE` (`:1386`), `LOOP` (`:1413`), `FOR` (`:1434`),
+  `REPEAT/UNTIL` (`:1493`), `IF/ELSE` (`:1305`), `CASE` (`:1623`).
+- `Pair<Rule>` is `pest::iterators::Pair<'i, Rule>` — a **borrowed cursor into the
+  parse tree**, tied to the lifetime of the input `&str`. Execution positions are
+  expressed as *paths within the pest tree*: inherently non-serializable.
+- `loop_count` (WHILE/LOOP/REPEAT) is **purely stack-local** — not reflected in
+  `State`. `FOR`'s counter *is* in `symbol_table`, but its `end_value` bound and
+  `variable_name` are stack-local. Loop *conditions* re-evaluate from `State` each
+  iteration, so they are re-derivable — but the position within the tree is not.
+- `IF` branch choice is implicit in *which recursive call is currently live*.
+- `BlockFlow::Jump` bubbles up through scopes via `resolve_jump` (`:184`); an
+  unresolved jump propagates to the enclosing `run_blocks` (`:2098`) — that is how
+  a jump leaves a loop or IF body. `EndProgram` (M2/M17/M30) terminates.
+
+**Paused three loops deep inside an IF**, the resume position that is *not* in
+`State` is: each nested `index`, each `jumps_taken`, each `loop_count`, each loop's
+cached `end_value`/condition/blocks pest pairs, the live IF branch, and every
+`block_pairs` slice cursor — all borrowed views into the pest tree.
+
+So mid-loop resume requires **reifying the control stack** into an explicit,
+serializable structure: replacing recursion with an execution cursor.
+
+## 3. Convergence — this is already on the roadmap
+
+`docs/sinumerik-execution-model.md` §"Scaling limits and the road to streaming",
+point 4, already prescribes this exact change for gigabyte-scale programs:
+
+> *Execution cursor* instead of recursion, yielding `(line_no, row)` — … makes
+> seek-by-checkpoint (snapshot of `pc` + `State`) cheap.
+
+So #47 is **not a throwaway rewrite**. The execution cursor it needs is the same
+one already planned for streaming/scaling. Reifying the control stack:
+
+1. unblocks checkpoint/resume (#47),
+2. unblocks O(1)-start lazy streaming for >10⁷-line programs, and
+3. removes the deep-recursion stack-depth risk on pathological nesting.
+
+One change, three payoffs — that is the argument for spending a 0.3.0 on it.
+
+## 4. What a checkpoint must capture (full inventory)
+
+Assembled from a line-anchored audit of the three state-bearing subsystems.
+
+### 4.1 `State` (`src/state.rs:52-86`) — already `#[derive(Clone)]`, cheap
+`axes`, `symbol_table`, `string_table`, `translation`, `jump_scopes` (label sets,
+mirrors scope depth but not the index within each), `seen_jump_targets`,
+`warned_addresses`; config (`axis_identifiers`, `iteration_limit`,
+`axis_index_map`, `allow_undefined_variables`, `output_keys`); `input`/`line_offsets`
+shared via `Arc` so clone is O(live symbols). ✅ Already snapshot-ready in memory.
+
+### 4.2 Control-flow position (`src/interpret_rules.rs`) — THE reified stack
+Not in `State` today; must become an explicit `Vec<Frame>`:
+- per scope: `index` (IP), `jumps_taken`;
+- per loop frame: kind (WHILE/LOOP/FOR/REPEAT), `loop_count`, and cached bounds
+  (`FOR` `end_value` + `variable_name`);
+- the scope/branch identity (which blocks list, which IF branch) as a **stable
+  address**, not a borrowed pest `Pair`.
+
+### 4.3 Flattener (`src/flatten.rs:130-150`) — NOT currently `Clone`
+- Reconstructable from `new(tolerance, axis_identifiers)`: `tolerance`,
+  `geometric_axes`.
+- Must snapshot: `positions`, `motion` (modal), `plane` (modal), `spline_buffer`
+  (raw buffered `Row`s of an in-progress multi-block spline — no compressed form,
+  capture verbatim), `spline_start`, `spline_degree`, `warned_motions` (warn-once).
+- Blockers: `SplineItem` needs a `Clone` derive (mechanical; wraps `Row: Clone`);
+  interned `&'static str` keys re-intern via `intern_column` on *disk* restore only.
+
+### 4.4 Output pipeline (`src/output.rs`)
+- `OutputRows::current` — the **in-flight row** for the block being interpreted
+  (not yet flushed to the sink); `warned_g91` latch. A checkpoint between blocks
+  must capture `current` together with the sink, and reproduce the
+  flush-on-next-`start_row` ordering so no row is dropped or double-emitted.
+- Batch path forward-fill carry (`BatchBuilder`): `columns` (growing canonical
+  set, monotonic — resuming with an empty set changes later-batch schema),
+  `fill` (last-seen value per forward-filled column — the core carry).
+- `BatchStreamSink`: `buffer` (rows since last emitted batch), `events`,
+  `name_ids`, `output_row_count` (drives `row_idx` attribution and batch cadence).
+- Plumbing that a checkpoint *cannot* recreate: the channel handles (`Stream`
+  sender, batch `sender`, `events_sender`). Resume attaches fresh channels.
+- Process-global (outside any checkpoint): the `intern_column` `OnceLock` pool —
+  serialize key *content*, rehydrate through `intern_column`.
+
+## 5. Design: the reified execution VM
+
+### 5.1 Addressing a position without a borrowed pest `Pair`
+The single change that makes everything else fall out: express an execution
+position as a **stable, owned address** instead of a `pest::Pair` cursor.
+
+Two candidate representations:
+
+- **(a) Path into the retained tree.** Keep the parse tree alive (already `Arc`-able
+  via the input) and address a block by a `Vec<u32>` path of child indices from the
+  root. Serializable (just integers); resolved to a `Pair` by walking `into_inner`.
+  Smallest change; still tied to pest at runtime.
+- **(b) Pre-flattened owned IR (the execution cursor).** One whole-file pass lowers
+  the block/control tree into an owned, indexable program: a `Vec` of block
+  instructions with explicit scope boundaries and pre-resolved jump tables (a
+  whole-program `scan_jump_targets`). Execution is a `pc` over the IR with an
+  explicit frame stack; pest is gone at runtime. This is the roadmap's "execution
+  cursor" — larger, but unlocks lazy parse (Phase 3) and full serialization.
+
+**Recommendation: target (b).** It is the roadmap direction; doing (a) then (b) is
+double work. (a) is a viable *interim* only if we need #47 shipped before the IR
+lands.
+
+### 5.2 The frame stack and the token
+```
+Frame = { scope: ScopeId, ip: u32, jumps_taken: u32, loop: Option<LoopState> }
+LoopState = While { count } | Loop { count } | Repeat { count }
+          | For { count, var: SymbolId, end: f64 }
+Checkpoint = {
+    program_fingerprint,          // structural hash of the program prefix ≤ cursor
+    stack: Vec<Frame>,            // the reified control stack (innermost last)
+    state: State,                 // cloned
+    flattener: FlattenerSnapshot, // §4.3 evolving fields
+    output_carry: OutputCarry,    // §4.4 forward-fill + counts + in-flight row
+    cursor_line_no: u32,          // for the consumer's "checkpoint precedes L" gate
+}
+```
+Loop conditions are **not** stored — they re-evaluate from `state` on resume,
+exactly as the interpreter does each iteration today. That is what keeps the token
+small and correct.
+
+### 5.3 `checkpoint()` / `resume()` semantics
+- `checkpoint()` is valid only at a **between-rows boundary** (`output.start_row`,
+  `interpret_rules.rs:2005`) where the in-flight `current` row is well-defined.
+  Emit checkpoints every *N* output rows.
+- `resume(token, patched_text)`:
+  1. Re-lower `patched_text` to the IR.
+  2. Verify `program_fingerprint` against the patched program's prefix up to
+     `cursor_line_no`. If the edit changed block structure *before* the cursor, the
+     stored `ip`/`ScopeId`s are stale → **fail loudly** (`ParsingError`, a new
+     `kind = "checkpoint_invalidated"`), never resume into a shifted tree. This
+     matches the consumer's own gate ("last checkpoint whose block cursor precedes
+     L") and the project's loud-failure-over-silent-wrongness rule.
+  3. Restore `state` + frame stack + flattener + output carry; attach fresh output
+     channels; continue the VM loop from `stack`.
+
+### 5.4 Resume + edited text — the validity contract
+A checkpoint is resumable against a patched program **iff the patch lies at or
+after the checkpoint's block cursor**. Enforced by the prefix fingerprint. The
+consumer already honours this; the API makes it a checked invariant rather than a
+convention.
+
+## 6. Phasing
+
+- **Phase 1 — reify the stack (delivers #47).** Replace the recursive tree-walk in
+  `run_blocks`/`interpret_*` with the explicit VM + frame stack over IR (b), keeping
+  eager whole-file parse. Add `NcSession` with in-memory, **same-process**
+  `checkpoint()`/`resume()` (token holds owned `State`/frames/snapshots; no disk
+  serialization yet). This is everything the consumer needs.
+- **Phase 2 — serializable token.** `Serialize`/`Deserialize` for the snapshots;
+  re-intern `&'static str` keys on load; address symbols/scopes by stable ids.
+  Only if cross-process/persisted checkpoints are wanted.
+- **Phase 3 — lazy parse / O(1) start.** The streaming payoff: lazy per-line parse
+  feeding the same VM, demoting pest to the rare structural lines (the two-pass
+  design already prototyped in `src/line_driver.rs`). Independent follow-on.
+
+Phases 2 and 3 are optional and can slip past 0.3.0; Phase 1 is the release.
+
+## 7. Correctness strategy
+
+The VM must be a **behaviour-preserving** refactor of the tree-walker before
+`checkpoint` is even exposed. Guardrails:
+
+- **Differential parity**: run the whole existing golden-file suite + the mill-sim
+  corpus through both the old recursive path and the new VM, asserting byte-identical
+  output — the same discipline `test_stage1.py` already uses for the line-driver.
+- **Checkpoint/resume equivalence**: for a program run straight through vs.
+  checkpointed-and-resumed at every row boundary, assert identical `(line_no, row)`
+  streams and identical final `State` — including the hard cases: mid-`WHILE`,
+  mid-`FOR` (counter in `State`, bound in the frame), mid-spline (buffered
+  `spline_buffer`), a jump out of a loop body, and forward-fill continuity across a
+  resumed batch boundary.
+- **Invalidation**: edits before the cursor must fail loudly, never silently resume.
+
+## 8. Rough scope
+
+Phase 1 is real interpreter surgery — the IR lowering, the VM loop, the frame
+stack, `FlattenerSnapshot`/`OutputCarry` capture-restore, and the parity harness —
+but it is bounded and well-understood after this audit, and it is load-bearing for
+the streaming roadmap regardless of #47. Estimate: the dominant 0.3.0 work item.
+Phases 2–3 are additive and independently schedulable.
+
+## 9. Open questions for review
+- (b) IR now vs (a) tree-path interim — how urgently does the consumer need #47
+  relative to the streaming work? If both are wanted this quarter, go straight to (b).
+- Token type: opaque handle only (Phase 1) vs. a documented serializable schema —
+  affects how much of §5.2 is public API from day one.
+- Checkpoint cadence policy: fixed *N* rows vs. adaptive (e.g. denser near loop
+  headers so an edit resumes closer to *L*).

{nc_gcode_interpreter-0.2.3 → nc_gcode_interpreter-0.2.5}/src/interpret_rules.rs

@@ -7,6 +7,11 @@ use std::collections::HashMap;
 
 type Output = crate::output::OutputRows;
 
+// Experimental execution-cursor VM (#47), a child module so it can reuse this
+// module's private leaf fns via `super::`. Gated behind `NC_VM=1`.
+#[path = "vm.rs"]
+pub(crate) mod vm;
+
 /// Control-flow signal returned by block interpretation: either fall through
 /// to the next block, or a pending GOTO that must be resolved against the
 /// block list of the current scope or, failing that, an enclosing scope
@@ -1850,8 +1855,11 @@ fn interpret_statement(element: Pair<Rule>, output: &mut Output, state: &mut Sta
                 match state.resolve_output_key(&key) {
                     Some((ColKind::Axis, skey)) => {
                         // State keeps local coordinates; the output row gets the machine
-                        // coordinate under the translation active at this point in the program.
-                        let machine_value = state.get_axis_machine(skey).unwrap_or(local_value);
+                        // coordinate under the translation active here. `update_axis` just
+                        // stored this axis' local == `local_value`, so machine =
+                        // local_value + translation — one lookup instead of
+                        // `get_axis_machine`'s two (it would re-read the axis we just set).
+                        let machine_value = local_value + state.get_translation(skey);
                         last.insert(skey, Value::Float(machine_value));
                     }
                     Some((ColKind::Block, skey)) => {

{nc_gcode_interpreter-0.2.3 → nc_gcode_interpreter-0.2.5}/src/interpreter.rs

@@ -270,7 +270,14 @@ fn interpret_file(input: &str, state: &mut State, output: &mut OutputRows) -> Re
         message: "No inner blocks found".to_string(),
     })?;
 
-    match interpret_blocks(blocks, output, state)? {
+    // Experimental: the explicit-stack VM (#47) runs the same structured path
+    // when NC_VM=1, for differential parity against the recursive walker.
+    let flow = if crate::interpret_rules::vm::vm_enabled() {
+        crate::interpret_rules::vm::run(blocks, output, state)?
+    } else {
+        interpret_blocks(blocks, output, state)?
+    };
+    match flow {
         BlockFlow::Continue | BlockFlow::EndProgram => Ok(()),
         // A jump that no scope could resolve: the destination does not exist
         // in the programmed search direction (alarm 14080 on a real control).

{nc_gcode_interpreter-0.2.3 → nc_gcode_interpreter-0.2.5}/src/output.rs

@@ -14,6 +14,11 @@ use crate::types::Value;
 use std::collections::{HashMap, HashSet};
 use std::sync::{Mutex, OnceLock};
 
+/// FxHash map for the hot table-build lookups (`index_of`, per cell per row;
+/// `fill`, per column per batch) — trusted `&'static str` keys, no need for
+/// SipHash's DoS resistance. See [`crate::state::FxMap`].
+type FxMap<K, V> = HashMap<K, V, rustc_hash::FxBuildHasher>;
+
 /// Intern a column name to a process-stable `&'static str`.
 ///
 /// The set of distinct output-column names is a small closed vocabulary
@@ -77,6 +82,13 @@ impl CellMap {
         None
     }
 
+    /// Empty the cells, retaining the backing allocation so a recycled row can
+    /// be refilled without a fresh heap allocation (see [`Row::reset_for_reuse`]).
+    #[inline]
+    pub fn clear(&mut self) {
+        self.entries.clear();
+    }
+
     #[inline]
     pub fn get_mut(&mut self, key: &str) -> Option<&mut Value> {
         self.entries.iter_mut().find(|(k, _)| *k == key).map(|(_, v)| v)
@@ -126,6 +138,20 @@ pub struct Row {
     pub variable_changes: Vec<(String, f64)>,
 }
 
+impl Row {
+    /// Empty this row for reuse as the next `current`, retaining both the cells
+    /// and variable-change allocations. On the batch path a finished row's
+    /// buffers are consumed into the columnar batch and then recycled here,
+    /// which collapses the per-row heap alloc/free (~22M on the 1.1 GB file,
+    /// the dominant allocator cost) into a small pool cycled batch-to-batch.
+    #[inline]
+    pub fn reset_for_reuse(&mut self) {
+        self.line_no = 0;
+        self.cells.clear();
+        self.variable_changes.clear();
+    }
+}
+
 /// Sparse side-table of variable-change events for the batch path, mirroring
 /// what the streaming `nc_to_rows` yields per row as `variable_changes`. Stored
 /// columnar (struct-of-arrays) so it converts to an Arrow table cheaply:
@@ -200,6 +226,10 @@ pub struct BatchStreamSink {
     sender: std::sync::mpsc::SyncSender<Table>,
     builder: BatchBuilder,
     buffer: Vec<Row>,
+    /// Emptied rows kept for reuse: after a batch is built the buffered rows'
+    /// allocations are cleared (not freed) into this pool, and `accept` hands
+    /// them back to the interpreter to refill instead of allocating fresh.
+    recycle: Vec<Row>,
     batch_size: usize,
     /// Whether to accumulate variable-change events (opt-in `include_variables`).
     record_variables: bool,
@@ -221,7 +251,7 @@ impl BatchStreamSink {
     /// rows have accumulated. Variable-only rows carry no output cells and are
     /// dropped from the batch table, but their variable changes are still
     /// recorded into the sparse event log when `record_variables` is on.
-    fn accept(&mut self, row: Row) -> Result<(), ParsingError> {
+    fn accept(&mut self, mut row: Row) -> Result<Option<Row>, ParsingError> {
         if self.record_variables {
             for (name, value) in &row.variable_changes {
                 let id = match self.name_ids.get(name) {
@@ -241,15 +271,20 @@ impl BatchStreamSink {
                 self.events.value.push(*value);
             }
         }
+        // A variable-only row carries no output cell: it is not buffered, so hand
+        // its (cleared) allocation straight back for reuse.
         if row.cells.is_empty() {
-            return Ok(());
+            row.reset_for_reuse();
+            return Ok(Some(row));
         }
         self.output_row_count += 1;
         self.buffer.push(row);
         if self.buffer.len() >= self.batch_size {
             self.emit()?;
         }
-        Ok(())
+        // Hand back a pooled row (refilled after the last emit) so the caller
+        // refills a retained allocation instead of allocating a fresh one.
+        Ok(self.recycle.pop())
     }
 
     /// Send the accumulated variable-change events once, at finish. A no-op when
@@ -267,8 +302,20 @@ impl BatchStreamSink {
         if self.buffer.is_empty() {
             return Ok(());
         }
-        let rows = std::mem::take(&mut self.buffer);
+        // Take the buffered rows out to build the batch, then recycle their
+        // allocations: `build_batch` only borrows the rows (it copies each cell
+        // value into the columnar builders), so once it returns the row buffers
+        // are free to be cleared and pooled rather than dropped.
+        let mut rows = std::mem::take(&mut self.buffer);
         let table = self.builder.build_batch(&rows);
+        self.recycle.reserve(rows.len());
+        for mut row in rows.drain(..) {
+            row.reset_for_reuse();
+            self.recycle.push(row);
+        }
+        // `rows` is now empty but keeps its `Vec<Row>` capacity - reinstate it as
+        // the buffer so the outer vector is not reallocated each batch either.
+        self.buffer = rows;
         self.sender.send(table).map_err(|_| ParsingError::StreamClosed)
     }
 }
@@ -357,6 +404,7 @@ impl OutputRows {
                 sender,
                 builder: BatchBuilder::new(disable_forward_fill).with_line_numbers(emit_line_no),
                 buffer: Vec::new(),
+                recycle: Vec::new(),
                 batch_size,
                 record_variables,
                 events_sender,
@@ -378,32 +426,86 @@ impl OutputRows {
         self.flattener = Some(flattener);
     }
 
-    /// Route a finished row to the sink, passing it through the flattener
-    /// first when one is installed. Shared by `flush`.
-    fn deliver(&mut self, row: Row) -> Result<(), ParsingError> {
+    /// Number of committed rows in a `Collect` sink (0 for streaming sinks).
+    /// Lets the experimental resumable VM (#47) pause at a row boundary.
+    #[allow(dead_code)]
+    pub(crate) fn collected_len(&self) -> usize {
+        match &self.sink {
+            RowSink::Collect(rows) => rows.len(),
+            _ => 0,
+        }
+    }
+
+    /// Deep-clone a checkpointable `Collect`-sink output: the committed rows,
+    /// the in-flight row, and the warn latches. `None` when the sink is a
+    /// streaming channel or a flattener is installed (their state isn't
+    /// snapshottable this simply). For the experimental resumable VM (#47);
+    /// the whole-file forward-fill happens later in `Table::from_rows`, so a
+    /// Collect snapshot needs no columnar carry.
+    #[allow(dead_code)]
+    pub(crate) fn snapshot_collect(&self) -> Option<OutputRows> {
+        if self.flattener.is_some() {
+            return None;
+        }
+        match &self.sink {
+            RowSink::Collect(rows) => Some(OutputRows {
+                current: self.current.clone(),
+                sink: RowSink::Collect(rows.clone()),
+                record_variables: self.record_variables,
+                flattener: None,
+                warned_g91: self.warned_g91,
+            }),
+            _ => None,
+        }
+    }
+
+    /// Return a recycled row to the sink's pool. Used when a single `deliver`
+    /// yields more recycled rows than the one the caller can reuse as `current`
+    /// (the flattener fans one block out to many rows): the surplus goes back to
+    /// the pool instead of being freed, so recycling stays effective on
+    /// flattened paths too. A no-op for the non-recycling sinks.
+    fn reclaim(&mut self, row: Row) {
+        if let RowSink::Batch(sink) = &mut self.sink {
+            sink.recycle.push(row);
+        }
+    }
+
+    /// Route a row to the sink, passing it through the flattener first when one
+    /// is installed, and returning an emptied row the sink no longer needs
+    /// (batch path) so the caller can reuse its allocation as the next
+    /// `current`. `None` when nothing is available to recycle.
+    fn deliver(&mut self, row: Row) -> Result<Option<Row>, ParsingError> {
         if let Some(mut flattener) = self.flattener.take() {
             let mut flattened = Vec::new();
             flattener.push(row, &mut flattened);
             self.flattener = Some(flattener);
+            // Keep one recycled row for the caller to reuse; hand any others the
+            // sink returns this call straight back to the pool so they are not
+            // freed (a flattened block delivers many rows in one `deliver`).
+            let mut recycled = None;
             for row in flattened {
-                self.deliver_to_sink(row)?;
+                if let Some(r) = self.deliver_to_sink(row)? {
+                    if let Some(prev) = recycled.replace(r) {
+                        self.reclaim(prev);
+                    }
+                }
             }
-            return Ok(());
+            return Ok(recycled);
         }
         self.deliver_to_sink(row)
     }
 
     /// Route a finished row to the sink: collected, streamed row-at-a-time, or
     /// fed to the worker-side batch producer.
-    fn deliver_to_sink(&mut self, row: Row) -> Result<(), ParsingError> {
+    fn deliver_to_sink(&mut self, row: Row) -> Result<Option<Row>, ParsingError> {
         match &mut self.sink {
             RowSink::Collect(rows) => {
                 rows.push(row);
-                Ok(())
+                Ok(None)
             }
             // The receiver hung up: the consumer stopped iterating. Abort
             // interpretation instead of running the rest of the program.
-            RowSink::Stream(sender) => sender.send(row).map_err(|_| ParsingError::StreamClosed),
+            RowSink::Stream(sender) => sender.send(row).map(|_| None).map_err(|_| ParsingError::StreamClosed),
             RowSink::Batch(sink) => sink.accept(row),
         }
     }
@@ -426,7 +528,13 @@ impl OutputRows {
                 self.warned_g91 = true;
             }
         }
-        self.deliver(row)
+        // `mem::take` above left `current` as an empty (unallocated) default; if
+        // the sink handed back a pooled row, adopt its retained allocation for
+        // the next block instead of allocating fresh on the first cell insert.
+        if let Some(recycled) = self.deliver(row)? {
+            self.current = recycled;
+        }
+        Ok(())
     }
 
     /// Begin the row for the block at `line_no`, flushing the previous row.
@@ -710,7 +818,7 @@ pub struct BatchBuilder {
     /// table.
     columns: Vec<&'static str>,
     /// Last carried non-null value per forward-filled column.
-    fill: HashMap<&'static str, Carry>,
+    fill: FxMap<&'static str, Carry>,
 }
 
 impl BatchBuilder {
@@ -719,7 +827,7 @@ impl BatchBuilder {
             disable_forward_fill,
             emit_line_no: false,
             columns: Vec::new(),
-            fill: HashMap::new(),
+            fill: FxMap::default(),
         }
     }
 
@@ -756,7 +864,8 @@ impl BatchBuilder {
         // One typed builder per column, in canonical order, each pre-filled
         // with `height` nulls. A name->position index lets each cell find its
         // builder in O(1).
-        let mut index_of: HashMap<&'static str, usize> = HashMap::with_capacity(self.columns.len());
+        let mut index_of: FxMap<&'static str, usize> =
+            FxMap::with_capacity_and_hasher(self.columns.len(), Default::default());
         let mut builders: Vec<ColumnBuilder> = Vec::with_capacity(self.columns.len());
         for (position, &name) in self.columns.iter().enumerate() {
             index_of.insert(name, position);
@@ -765,7 +874,9 @@ impl BatchBuilder {
 
         // Single pass: dispatch each present cell to its column builder. Every
         // cell key is in `present` (hence in `index_of`), so the lookup always
-        // hits; the `if let` is defensive only.
+        // hits; the `if let` is defensive only. FxHash beat a linear scan of
+        // `self.columns` here (~22 keys need content compare — interning does
+        // NOT guarantee pointer identity across all column sources).
         for (row_index, cell) in cells.iter().enumerate() {
             for (&key, value) in cell.iter() {
                 if let Some(&position) = index_of.get(key) {