flingit 0.0.63 → 0.0.65

package/node_modules/flingflow/TESTING.md

@@ -0,0 +1,506 @@
+# Testing
+
+flingflow uses [Vitest](https://vitest.dev/) as its test runner and the [`determined`](https://www.npmjs.com/package/determined) package for deterministic simulation testing.
+
+## Running Tests
+
+```bash
+# Run all tests
+npm test
+
+# Run in watch mode
+npm run test:watch
+
+# Run a specific test file
+npx vitest run test/engine.test.ts
+
+# Run tests matching a pattern
+npx vitest run -t "retries"
+```
+
+## Test Suite Summary
+
+| File | Tests | Category | Description |
+|---|---|---|---|
+| `test/store-memory.test.ts` | 17 | Unit | Conformance tests on `MemoryEventStore` |
+| `test/store-sqlite.test.ts` | 17 | Unit | Conformance tests on `SqliteEventStore` |
+| `test/context.test.ts` | 8 | Unit | Scratchpad loading, buffered writes, overrides |
+| `test/engine.test.ts` | 22 | Unit | Engine state machine: happy path, retries, failures, workflow ID uniqueness |
+| `test/recovery.test.ts` | 10 | Integration | Timeout detection, stuck workflows, recovery + engine |
+| `test/simulation.test.ts` | 8 | Simulation | Concurrent workers, failure injection, multi-seed (both stores) |
+| **Total** | **82** | | |
+
+There is also a CLI stress test (`src/stress.ts`) that runs outside of Vitest. See [Stress Test (CLI)](#stress-test-cli) below.
+
+## Test Architecture
+
+### Three Tiers
+
+1. **Unit tests** — Single-component, `MemoryEventStore`, `ManualClock`, deterministic IDs. No concurrency.
+2. **Integration tests** — Multiple components wired together (engine + store + recovery). Still single-threaded, but tests cross-component interactions.
+3. **Simulation tests** — Multiple concurrent tasks via `determined`'s `SimulationImpl`. Tests interleaving, race conditions, and crash recovery.
+
+### Shared Infrastructure
+
+All unit and integration tests use the same setup pattern:
+
+```typescript
+let store: EventStore;
+let registry: WorkflowRegistry;
+let clock: ManualClock;
+
+beforeEach(async () => {
+  store = new MemoryEventStore();
+  (await store.initialize())._unsafeUnwrap();
+  registry = new WorkflowRegistry();
+  clock = new ManualClock(1000);     // deterministic starting time
+  engine = new Engine({
+    store,
+    registry,
+    clock,
+    generateId: () => `run-${++idCounter}`,  // deterministic run IDs
+  });
+});
+```
+
+This gives every test:
+- A fresh, empty store
+- Deterministic timestamps via `ManualClock`
+- Deterministic run IDs via a counter
+
+---
+
+## Store Conformance Tests
+
+**Files:** `test/store-conformance.ts`, `test/store-memory.test.ts`, `test/store-sqlite.test.ts`
+
+A parameterized test suite that runs against any `EventStore` implementation. Each implementation file instantiates the suite with a factory function:
+
+```typescript
+// test/store-memory.test.ts
+storeConformanceTests("MemoryEventStore", async () => {
+  const store = new MemoryEventStore();
+  (await store.initialize())._unsafeUnwrap();
+  return store;
+});
+```
+
+### Tests (17 per implementation, 34 total)
+
+| Test | What It Verifies |
+|---|---|
+| `append + getLastEvent round-trip` | Basic write-then-read returns the same event |
+| `getLastEvent returns null for unknown run` | Querying a non-existent run returns `null` |
+| `getLastEvent returns highest seq` | With multiple events, returns the one with the highest sequence number |
+| `returns UniqueConstraintError on duplicate (run_id, seq)` | Inserting the same `(run_id, seq)` twice returns `UniqueConstraintError` |
+| `appendBatch inserts multiple events atomically` | Multiple events inserted in one call are all readable |
+| `appendBatch is atomic — no partial inserts on duplicate` | If any event in a batch conflicts, the entire batch is rolled back. Events that were valid are **not** persisted. |
+| `getEvent returns specific event` | Retrieve a single event by `(run_id, seq)` |
+| `getEvent returns null for missing seq` | Missing events return `null` |
+| `getCompletedEvents returns step_completed events in seq DESC order` | Only `step_completed` events are returned, newest first |
+| `getAllEvents returns events in seq ASC order` | Full event log in chronological order |
+| `getActiveWorkflows excludes terminal workflows` | Workflows ending in `workflow_completed` or `workflow_failed` are not returned |
+| `JSON round-trips payloads correctly` | Nested objects, arrays, nulls, booleans, and floats survive serialization. Retrieved payload is a deep copy, not a reference. |
+| `isolates events between different runs` | Events from run A are not visible when querying run B |
+| `appendIfNoActiveWorkflow succeeds when no active run` | First creation of a workflow ID succeeds |
+| `appendIfNoActiveWorkflow rejects when active run exists` | Second creation of the same workflow ID while the first is active returns `WorkflowAlreadyActiveError` |
+| `appendIfNoActiveWorkflow allows after terminal event` | After a workflow completes or fails, the same workflow ID can be reused for a new run |
+| `rejects appending event with mismatched workflow_id for existing run_id` | Adding an event with a different `workflow_id` than existing events for the same `run_id` returns `WorkflowIdMismatchError` |
+
+### Why Both Implementations
+
+The `MemoryEventStore` serializes/deserializes payloads via JSON on every write/read, matching the behavior of SQLite's text column storage. This ensures:
+
+- Serialization bugs are caught in fast, in-memory tests
+- Both stores produce identical behavior (same test suite, same assertions)
+- Tests don't accidentally rely on object identity or non-serializable values (functions, `undefined` as object values, etc.)
+
+---
+
+## Context Tests
+
+**File:** `test/context.test.ts`
+
+Tests for `buildContext()`, which constructs the `Ctx` object for step execution by loading the scratchpad from persisted `step_completed` events.
+
+### Tests (8)
+
+| Test | What It Verifies |
+|---|---|
+| `loads initial writes into scratchpad` | `ctx.get()` returns values from the `workflow_created` event's `writes`, and `ctx.workflowId`, `ctx.runId`, and `ctx.step` are set from the arguments passed to `buildContext` |
+| `reads scratchpad from step_completed events` | Values written by previous steps via `ctx.set()` are available via `ctx.get()` in subsequent steps |
+| `later writes override earlier ones` | When multiple steps write the same key, the most recent value wins (newest-first scan, first match) |
+| `returns undefined for missing keys` | `ctx.get("nonexistent")` returns `undefined` |
+| `buffers writes without persisting` | `ctx.set()` calls are buffered in memory. The store remains empty. `getPendingWrites()` returns the buffered state. |
+| `pending writes are readable via get` | A value set with `ctx.set()` in the current step is immediately readable via `ctx.get()` without a store round-trip |
+| `pending writes override scratchpad values` | If a key exists in the persisted scratchpad and is also set via `ctx.set()`, the pending write takes priority |
+| `empty scratchpad when no step_completed events exist` | On the first step of a workflow, all `ctx.get()` calls return `undefined` |
+
+---
+
+## Engine Tests
+
+**File:** `test/engine.test.ts`
+
+Tests for the `Engine` class using `MemoryEventStore` and `ManualClock`.
+
+### Test Fixtures (`test/fixtures.ts`)
+
+Reusable workflow definitions:
+
+| Fixture | Behavior |
+|---|---|
+| `singleStepWorkflow` | Completes in one step. Returns `{ done: true, result: { value: "processed-{input}" } }`. |
+| `multiStepWorkflow` | Three steps (`start` → `process` → `finish`) that use the scratchpad to pass data. Counts input items, doubles the count. |
+| `failingWorkflow` | Always throws `Error("transient failure")`. Used to test max-attempts exhaustion. |
+| `failOnceWorkflow` | Throws on first invocation per run ID, succeeds on retry. Uses an in-memory counter keyed by `runId`. Call `resetFailOnceCounter()` in `beforeEach`. |
+| `nonRetryableWorkflow` | Throws `NonRetryableError("invalid input")`. Tests non-retryable failure path. |
+
+### Tests (22)
+
+#### `createWorkflow` (2)
+
+| Test | What It Verifies |
+|---|---|
+| `creates a workflow_created event` | Inserts a single event at `seq=0` with correct `event_type`, `payload`, and `created_at` |
+| `rejects unknown workflow names` | Returns `UnknownWorkflowError` if the name isn't in the registry |
+
+#### `claimStep` (4)
+
+| Test | What It Verifies |
+|---|---|
+| `claims the first step after workflow_created` | Returns a `ClaimResult` with `step: "start"`, `attempt: 1`, and correct `workflowName` |
+| `returns null for non-existent workflow` | No crash, just `null` |
+| `returns null when step is already in progress` | After one successful claim, a second claim returns `null` (the `step_started` event is the last event, and `resolveTransition` returns `action: "none"`) |
+| `transitions to workflow_failed on non-retryable step_failed` | After executing a non-retryable workflow, calling `claimStep` emits `workflow_failed` and returns `null` |
+
+#### `executeStep` (2)
+
+| Test | What It Verifies |
+|---|---|
+| `single-step happy path` | Creates event trace: `workflow_created → step_started → step_completed → workflow_completed`. Result payload is correct. |
+| `returns false when nothing to execute` | On an already-completed workflow, returns `false` |
+
+#### `runToCompletion` (1)
+
+| Test | What It Verifies |
+|---|---|
+| `multi-step workflow with scratchpad` | Drives a 3-step workflow to completion. Verifies the full 8-event trace and that scratchpad values are passed correctly between steps (input items counted → doubled → returned as result). |
+
+#### `retries` (3)
+
+| Test | What It Verifies |
+|---|---|
+| `retries on retryable failure` | `failOnceWorkflow` produces: `step_started(attempt=1) → step_failed → step_started(attempt=2) → step_completed → workflow_completed` |
+| `max attempts exceeded → workflow_failed` | `failingWorkflow` (maxAttempts=5) produces 5 start/fail cycles, then `workflow_failed` with reason containing `"max attempts (5) exceeded"` |
+| `non-retryable failure → immediate workflow_failed` | `nonRetryableWorkflow` produces: `step_started → step_failed(retryable=false) → workflow_failed`. Only 1 attempt. |
+
+#### `event log validity` (5)
+
+Structural invariant checks that run against completed workflows:
+
+| Test | What It Verifies |
+|---|---|
+| `all events have ascending seq numbers` | `events[i].seq === i` for all events |
+| `all events have the same run_id` | No cross-run contamination |
+| `all events have the same workflow_id` | No cross-workflow contamination |
+| `created_at values come from the clock` | Timestamps match the `ManualClock` values at the time of creation |
+| `event trace follows valid state machine transitions` | Every consecutive pair of events is a valid transition per the state machine table. Uses a lookup table of `{ from_event_type: [allowed_to_event_types] }`. |
+
+#### `workflow ID uniqueness` (5)
+
+| Test | What It Verifies |
+|---|---|
+| `rejects duplicate active workflow ID` | Creating a second workflow with the same `workflowId` while the first is active returns `WorkflowAlreadyActiveError` |
+| `allows reuse of workflow ID after completion` | After a workflow completes, the same `workflowId` can be used for a new run |
+| `allows reuse of workflow ID after failure` | After a workflow fails, the same `workflowId` can be reused |
+| `different workflow IDs coexist` | Two workflows with different `workflowId`s can be active simultaneously |
+| `successive runs of same workflow ID get different run IDs` | Each `createWorkflow` call generates a unique `runId` |
+
+---
+
+## Recovery Tests
+
+**File:** `test/recovery.test.ts`
+
+Tests for `runRecovery()` using manually constructed event histories and `ManualClock`.
+
+### Tests (10)
+
+| Test | What It Verifies |
+|---|---|
+| `detects timed-out step_started` | A `step_started` event that's been sitting for >= `stepTimeoutMS` gets a `step_failed(reason: "timeout", retryable: true)` appended. Also verifies that just-under-timeout workflows are **not** recovered. |
+| `detects stuck workflow at workflow_created` | A workflow with only `workflow_created` (no `step_started` ever inserted) is flagged as stuck. |
+| `detects stuck workflow at step_completed with next_step` | A workflow whose last event is `step_completed` with a `next_step` but no subsequent `step_started` is flagged as stuck. |
+| `detects stuck workflow at retryable step_failed` | A workflow whose last event is a retryable `step_failed` (under max attempts) but with no retry `step_started` is flagged as stuck. |
+| `ignores terminal workflows` | Workflows in `workflow_completed` or `workflow_failed` state are not included in the active workflow scan (verified at the `getActiveWorkflows` level). |
+| `ignores non-timed-out in-progress steps` | A `step_started` that hasn't exceeded the timeout is left alone. |
+| `recovery + engine integration` | End-to-end: a worker "dies" (leaves a `step_started` that times out), recovery marks it as `step_failed`, then the engine retries and completes the workflow. |
+| `detects step_completed without next_step as needing workflow_completed` | A workflow whose last event is `step_completed` without `next_step` (meaning it should transition to `workflow_completed`) is flagged as `stuck_terminal`. |
+| `detects non-retryable step_failed as needing workflow_failed` | A workflow whose last event is a non-retryable `step_failed` (should transition to `workflow_failed`) is flagged as `stuck_terminal`. |
+| `detects max-attempts-exhausted step_failed as needing workflow_failed` | A workflow whose last event is a `step_failed` at max attempts (should transition to `workflow_failed`) is flagged as `stuck_terminal`. |
+
+---
+
+## Simulation Tests
+
+**File:** `test/simulation.test.ts`
+
+Tests using `determined`'s `SimulationImpl` for deterministic simulation testing. These tests run multiple concurrent "worker" tasks that compete to execute workflow steps, with `determined` controlling the interleaving at every checkpoint and failpoint in the engine and stores.
+
+The entire test suite is parameterized and runs against both `MemoryEventStore` and `SqliteEventStore` (in-memory), giving 8 tests total (4 per store).
+
+### How It Works
+
+1. A store is created with an `EventStoreCheck` callback that validates the event log on every mutation.
+2. An `Engine` is wired to the store.
+3. A `SimulationImpl` is instantiated with a logger, entropy source, and failure probability.
+4. Multiple `TaskSpec`s are defined — each runs a loop calling `engine.executeStep(runId, task)` where `task` is the `SimulationTask` provided by `determined`.
+5. `sim.runTasks(specs)` runs all tasks concurrently with controlled interleaving.
+6. After completion, the event log is validated for structural correctness.
+
+### Validation Function
+
+`validateEventTrace(events)` checks three invariants on a completed workflow's event log:
+
+1. **Ascending sequence numbers**: `events[i].seq === i` for all `i`
+2. **Valid state machine transitions**: every consecutive pair `(events[i], events[i+1])` is a legal transition per the state machine table
+3. **Terminal state**: the last event is `workflow_completed` or `workflow_failed`
+
+### Event-Log Check Callback
+
+Both store implementations are constructed with an `EventStoreCheck` callback (`makeCheck()`) that runs after every `append` and `appendBatch`. On every single store mutation, it verifies:
+
+1. Contiguous sequence numbers (`seq = 0, 1, 2, ...`)
+2. First event is `workflow_created`
+3. Every consecutive pair is a valid state machine transition
+4. At most one terminal event (`workflow_completed` or `workflow_failed`)
+
+This catches corruption the moment it occurs, rather than only at the end of the test.
+
+### Tests (4 per store, 8 total)
+
+#### `concurrent workers: exactly one claims each step`
+
+- 3 workers race to execute steps on a single `counter` workflow (loops 4 times via scratchpad).
+- No failure injection (`failureProbability: 0`).
+- Verifies the event trace is valid and the workflow completes with `{ finalCount: 3 }`.
+- The core concurrency guarantee: despite interleaving, exactly one worker wins each step claim.
+
+#### `concurrent workers across multiple seeds`
+
+- Runs 100 iterations with different entropy seeds.
+- 2 workers per iteration, racing on a single-step `simple` workflow.
+- Each iteration gets a fresh store, registry, engine.
+- Validates the event trace for every seed.
+- Ensures correctness is not dependent on a lucky interleaving schedule.
+
+#### `failure injection with recovery`
+
+- 2 workers + 1 recovery task, all concurrent.
+- `failureProbability: 0.3` — 30% chance that each `sim.failpoint()` call throws, simulating a crash.
+- Workers catch failpoint exceptions and keep retrying.
+- The recovery task periodically advances the `ManualClock` past the step timeout and runs `runRecovery()` to unstick timed-out workflows.
+- Validates the event log's structural integrity (ascending seq, valid transitions) regardless of the simulation outcome.
+- This test exercises the most adversarial conditions: concurrent access + arbitrary crash timing + recovery racing normal workers.
+
+#### `appendBatch atomicity under concurrency`
+
+- 3 workers race on a single-step `simple` workflow.
+- The final step uses `appendBatch` to write both `step_completed` and `workflow_completed` atomically.
+- Validates the event trace and asserts exactly 1 terminal event exists.
+- Ensures `appendBatch` doesn't produce partial writes even under concurrent access.
+
+---
+
+## Stress Test (CLI)
+
+**File:** `src/stress.ts`
+
+A configurable CLI stress test that runs a non-trivial multi-step workflow under simulation with concurrent workers, failure injection, and recovery. Unlike the Vitest simulation tests (which are fast, focused, and run in CI), the stress test is designed for manual exploration — crank up the parameters and shake out concurrency bugs.
+
+### Running
+
+```bash
+# Defaults: count=10, memory store, 4 workers, 50 seeds, 10% failure, 5 max attempts
+npm run stress
+
+# All options
+npm run stress -- --count 20 --store sqlite --workers 8 --seeds 100 --failure 0.2 --max-attempts 10
+```
+
+| Flag | Default | Description |
+|---|---|---|
+| `--count N` | `10` | Number of items to process (controls workflow length) |
+| `--store TYPE` | `memory` | Store backend: `memory` or `sqlite` (both use in-memory databases) |
+| `--workers N` | `4` | Number of concurrent worker tasks in the simulation |
+| `--seeds N` | `50` | Number of simulation seeds to run (each seed = one full workflow) |
+| `--failure P` | `0.1` | Failpoint failure probability (0-1) |
+| `--max-attempts N` | `5` | Max step attempts before workflow fails |
+
+Exit code is 0 if all seeds pass, 1 if any fail.
+
+### SimKV: In-Memory Key-Value Store
+
+The stress test uses a `SimKV` class — a simple `Map<string, unknown>` with failpoints before and after every `get` and `set` operation. The workflow function accesses the simulation task via `ctx.sim`, which `buildContext` sets from the engine's `sim` parameter.
+
+```typescript
+class SimKV {
+  async get(key: string, sim: SimulationTask): Promise<unknown>;
+  async set(key: string, value: unknown, sim: SimulationTask): Promise<void>;
+}
+```
+
+Each method has two failpoints (`kv.get:before`/`kv.get:after`, `kv.set:before`/`kv.set:after`). Since these failpoints run inside the workflow function's try/catch in `executeStep`, a thrown failpoint is recorded as a retryable `step_failed` and the step is retried. The kv operations are idempotent (same input always produces the same value), so retries are safe.
+
+### Workflow: Sum of Squares
+
+Given `count = N`, the workflow computes `1² + 2² + ... + N²` across `2N + 1` steps:
+
+| Steps | Phase | What each step does |
+|---|---|---|
+| `start`, `set-2` .. `set-N` | Set | Store `kv[i] = i²` in the external SimKV |
+| `sum-init` | Init | Set scratchpad `sum = 0` |
+| `add-1` .. `add-N` | Add | Read `kv[i]`, add to scratchpad `sum` |
+
+The final step returns `{ sum }`. The expected result is `N(N+1)(2N+1)/6`.
+
+This workflow exercises both the scratchpad (passing `sum` between steps) and an external service with its own failpoints (SimKV). Each step is small and deterministic, making the expected result easy to verify.
+
+### Simulation Structure
+
+Each seed creates a fresh store, kv, engine, and `ManualClock`, then runs a `SimulationImpl` with:
+
+- **N worker tasks** — each loops calling `engine.executeStep(runId, task)`, catching failpoint errors and retrying. Workers break out of their loop when the workflow reaches a terminal state.
+- **1 recovery task** — loops calling `clock.advance(stepTimeoutMS + 1000)` followed by `runRecovery()`. This advances time past the step timeout so that stuck `step_started` events (from workers that "crashed" at a failpoint) are detected, marked as `step_failed(reason: "timeout")`, and retried.
+
+All tasks run concurrently via `determined`'s cooperative scheduler, which interleaves them at every checkpoint and failpoint.
+
+### Iteration Budget
+
+With store-level failpoints, there are approximately 16 failpoints per step execution attempt (store reads/writes in `claimStep`, `buildContext`, `commitStep`, plus kv failpoints in the workflow function, plus engine-level failpoints). The probability that a single attempt succeeds end-to-end is `(1 - failureProbability)^16`.
+
+The iteration budget per worker scales automatically:
+
+```
+attemptsPerStep = ceil(1 / (1 - failureProbability)^16)
+iterationsPerWorker = max(1000, totalSteps * attemptsPerStep * 5)
+```
+
+This ensures the workflow completes even at higher failure rates. Examples:
+
+| `--failure` | Attempts/step | Iterations/worker (count=10) |
+|---|---|---|
+| 0.05 | 3 | 1000 (floor) |
+| 0.1 | 6 | 1000 (floor) |
+| 0.2 | 36 | 3,780 |
+| 0.3 | 170 | 17,850 |
+
+### Consistency Check
+
+Every store is constructed with an `EventStoreCheck` callback (same as the simulation tests) that validates the event log on every mutation — contiguous seq numbers, valid transitions, at most one terminal event. This catches corruption at the point it occurs.
+
+### Verification
+
+After each seed's simulation completes, the test:
+
+1. Reads the full event log and validates the trace structure (contiguous seq, valid transitions)
+2. Asserts the last event is `workflow_completed`
+3. Compares the result payload's `sum` against `N(N+1)(2N+1)/6`
+
+A seed passes only if all three checks succeed.
+
+### Seeded Entropy
+
+Each seed uses a deterministic LCG-based entropy source seeded by the seed number. This makes failures reproducible — if seed 42 fails, running with `--seeds 43` will reproduce it.
+
+---
+
+## Failpoint Coverage
+
+Failpoints are placed at every I/O boundary across the engine, stores, and external services. They serve two purposes: (1) yield points for `determined`'s cooperative scheduler to interleave concurrent tasks, and (2) failure injection points where `determined` can throw to simulate process crashes.
+
+### Store Failpoints
+
+Every `EventStore` method (except `initialize`) has failpoints before and after the actual operation. Both `MemoryEventStore` and `SqliteEventStore` share the same failpoint labels:
+
+| Method | Failpoints |
+|---|---|
+| `append` | `store.append:before`, `store.append:after` |
+| `appendBatch` | `store.appendBatch:before`, `store.appendBatch:after` |
+| `appendIfNoActiveWorkflow` | `store.appendIfNoActiveWorkflow:before`, `store.appendIfNoActiveWorkflow:after` |
+| `getLastEvent` | `store.getLastEvent:before`, `store.getLastEvent:after` |
+| `getEvent` | `store.getEvent:before`, `store.getEvent:after` |
+| `getCompletedEvents` | `store.getCompletedEvents:before`, `store.getCompletedEvents:after` |
+| `getAllEvents` | `store.getAllEvents:before`, `store.getAllEvents:after` |
+| `getActiveWorkflows` | `store.getActiveWorkflows:before`, `store.getActiveWorkflows:after` |
+
+Store failpoints propagate out of the engine (they are not caught as step failures). A store failpoint throwing simulates a process crash mid-I/O — recovery handles the aftermath.
+
+### Engine Failpoints and Checkpoints
+
+| Location | Type | Label |
+|---|---|---|
+| `createWorkflow` | checkpoint | `createWorkflow:before-append`, `createWorkflow:after-append` |
+| `claimStep` | checkpoint | `claimStep:after-getLastEvent`, `claimStep:after-append` |
+| `executeStep` | failpoint | `executeStep:before-fn`, `executeStep:after-fn` |
+| `commitStep` | failpoint | `commitStep:before-append` |
+
+Checkpoints yield control but don't throw. Failpoints can throw. The `executeStep` method only catches errors from the workflow function body — failpoints around the function call and commit propagate as crashes.
+
+### Recovery Checkpoints
+
+| Location | Type | Label |
+|---|---|---|
+| `runRecovery` | checkpoint | `recovery:before-getActiveWorkflows`, `recovery:per-workflow` |
+
+### External Service Failpoints (Stress Test)
+
+The stress test's `SimKV` key-value store adds failpoints inside the workflow function:
+
+| Method | Failpoints |
+|---|---|
+| `kv.get` | `kv.get:before`, `kv.get:after` |
+| `kv.set` | `kv.set:before`, `kv.set:after` |
+
+Since these execute inside `def.fn(ctx)`, thrown failpoints are caught by the engine and recorded as retryable `step_failed` events.
+
+---
+
+## Invariants Verified Across All Tests
+
+These invariants hold across unit, integration, simulation, and stress tests:
+
+1. **Sequential seq numbers**: Every workflow's events have `seq = 0, 1, 2, ...` with no gaps or duplicates.
+2. **Valid state machine**: Every consecutive event pair is a legal transition per the state machine in SPEC.md.
+3. **Exactly-once step execution**: At most one worker succeeds at claiming any given step (enforced by `UniqueConstraintError` on `(run_id, seq)`).
+4. **Atomic batch writes**: `appendBatch` either writes all events or none. No partial inserts.
+5. **Scratchpad consistency**: Later writes to the same key override earlier writes. Failed steps discard their buffered writes.
+6. **Terminal state correctness**: Completed workflows end with exactly one `workflow_completed` or `workflow_failed` event.
+7. **Clock-driven timestamps**: All `created_at` values come from the injected `Clock`, not from system time or SQLite defaults.
+8. **JSON serialization round-trip**: Both store implementations serialize payloads through JSON, ensuring no non-serializable data slips through.
+9. **Continuous consistency** (simulation and stress tests): The `EventStoreCheck` callback validates the event log after every single store mutation, not just at the end of the test. Corruption is caught at the point of introduction.
+10. **Workflow ID uniqueness**: Only one active run per workflow ID at a time, enforced atomically by `appendIfNoActiveWorkflow`.
+
+---
+
+## Adding a New EventStore Implementation
+
+To add a new store backend:
+
+1. Implement the `EventStore` interface.
+2. Create a test file that calls the conformance suite:
+
+```typescript
+// test/store-mybackend.test.ts
+import { storeConformanceTests } from "./store-conformance.js";
+import { MyBackendEventStore } from "../src/store-mybackend.js";
+
+storeConformanceTests("MyBackendEventStore", async () => {
+  const store = new MyBackendEventStore(/* ... */);
+  (await store.initialize())._unsafeUnwrap();
+  return store;
+});
+```
+
+The conformance tests will verify your implementation handles all edge cases correctly.

package/node_modules/flingflow/dist/backoff.d.ts

@@ -0,0 +1,9 @@
+import type { WorkflowConfig } from "./types.js";
+/**
+ * Calculate the backoff delay for a given attempt number.
+ *
+ * Attempt 1 has no delay (first try).
+ * Attempt 2 has `initialBackoffMS` delay.
+ * Attempt N has `min(initialBackoffMS * backoffMultiplier^(N-2), maxBackoffMS)`.
+ */
+export declare function calculateBackoffMS(config: WorkflowConfig, attempt: number): number;

package/node_modules/flingflow/dist/backoff.js

@@ -0,0 +1,14 @@
+/**
+ * Calculate the backoff delay for a given attempt number.
+ *
+ * Attempt 1 has no delay (first try).
+ * Attempt 2 has `initialBackoffMS` delay.
+ * Attempt N has `min(initialBackoffMS * backoffMultiplier^(N-2), maxBackoffMS)`.
+ */
+export function calculateBackoffMS(config, attempt) {
+    if (attempt <= 1)
+        return 0;
+    const raw = config.initialBackoffMS * Math.pow(config.backoffMultiplier, attempt - 2);
+    return Math.min(raw, config.maxBackoffMS);
+}
+//# sourceMappingURL=backoff.js.map

package/node_modules/flingflow/dist/backoff.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"backoff.js","sourceRoot":"","sources":["../src/backoff.ts"],"names":[],"mappings":"AAEA;;;;;;GAMG;AACH,MAAM,UAAU,kBAAkB,CAAC,MAAsB,EAAE,OAAe;IACxE,IAAI,OAAO,IAAI,CAAC;QAAE,OAAO,CAAC,CAAC;IAC3B,MAAM,GAAG,GAAG,MAAM,CAAC,gBAAgB,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,CAAC,iBAAiB,EAAE,OAAO,GAAG,CAAC,CAAC,CAAC;IACtF,OAAO,IAAI,CAAC,GAAG,CAAC,GAAG,EAAE,MAAM,CAAC,YAAY,CAAC,CAAC;AAC5C,CAAC"}

package/node_modules/flingflow/dist/clock.d.ts

@@ -0,0 +1,13 @@
+export interface Clock {
+    now(): number;
+}
+export declare class SystemClock implements Clock {
+    now(): number;
+}
+export declare class ManualClock implements Clock {
+    private _now;
+    constructor(initial: number);
+    now(): number;
+    advance(ms: number): void;
+    set(ms: number): void;
+}

package/node_modules/flingflow/dist/clock.js

@@ -0,0 +1,21 @@
+export class SystemClock {
+    now() {
+        return Date.now();
+    }
+}
+export class ManualClock {
+    _now;
+    constructor(initial) {
+        this._now = initial;
+    }
+    now() {
+        return this._now;
+    }
+    advance(ms) {
+        this._now += ms;
+    }
+    set(ms) {
+        this._now = ms;
+    }
+}
+//# sourceMappingURL=clock.js.map

package/node_modules/flingflow/dist/clock.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"clock.js","sourceRoot":"","sources":["../src/clock.ts"],"names":[],"mappings":"AAIA,MAAM,OAAO,WAAW;IACtB,GAAG;QACD,OAAO,IAAI,CAAC,GAAG,EAAE,CAAC;IACpB,CAAC;CACF;AAED,MAAM,OAAO,WAAW;IACd,IAAI,CAAS;IAErB,YAAY,OAAe;QACzB,IAAI,CAAC,IAAI,GAAG,OAAO,CAAC;IACtB,CAAC;IAED,GAAG;QACD,OAAO,IAAI,CAAC,IAAI,CAAC;IACnB,CAAC;IAED,OAAO,CAAC,EAAU;QAChB,IAAI,CAAC,IAAI,IAAI,EAAE,CAAC;IAClB,CAAC;IAED,GAAG,CAAC,EAAU;QACZ,IAAI,CAAC,IAAI,GAAG,EAAE,CAAC;IACjB,CAAC;CACF"}

package/node_modules/flingflow/dist/context.d.ts

@@ -0,0 +1,8 @@
+import { type Result } from "neverthrow";
+import type { SimulationTask } from "determined";
+import type { EventStore } from "./store.js";
+import type { Ctx } from "./types.js";
+export declare function buildContext(store: EventStore, runId: string, workflowId: string, step: string, initialWrites: Record<string, unknown>, sim: SimulationTask): Promise<Result<{
+    ctx: Ctx;
+    getPendingWrites: () => Record<string, unknown>;
+}, Error>>;

package/node_modules/flingflow/dist/context.js

@@ -0,0 +1,45 @@
+import { ok, err } from "neverthrow";
+export async function buildContext(store, runId, workflowId, step, initialWrites, sim) {
+    // Load scratchpad from step_completed events (newest first, first match wins)
+    const completedResult = await store.getCompletedEvents(runId, sim);
+    if (completedResult.isErr())
+        return err(completedResult.error);
+    const completedEvents = completedResult.value;
+    const scratchpad = new Map();
+    for (const event of completedEvents) {
+        const payload = event.payload;
+        for (const [key, value] of Object.entries(payload.writes)) {
+            if (!scratchpad.has(key)) {
+                scratchpad.set(key, value);
+            }
+        }
+    }
+    // Apply initial writes (lowest priority — step writes take precedence)
+    for (const [key, value] of Object.entries(initialWrites)) {
+        if (!scratchpad.has(key)) {
+            scratchpad.set(key, value);
+        }
+    }
+    const pendingWrites = {};
+    const ctx = {
+        workflowId,
+        runId,
+        step,
+        sim,
+        get(key) {
+            // Pending writes take priority
+            if (key in pendingWrites) {
+                return Promise.resolve(pendingWrites[key]);
+            }
+            return Promise.resolve(scratchpad.get(key));
+        },
+        set(key, value) {
+            pendingWrites[key] = value;
+        },
+    };
+    return ok({
+        ctx,
+        getPendingWrites: () => ({ ...pendingWrites }),
+    });
+}
+//# sourceMappingURL=context.js.map

package/node_modules/flingflow/dist/context.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"context.js","sourceRoot":"","sources":["../src/context.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,EAAE,EAAE,GAAG,EAAe,MAAM,YAAY,CAAC;AAKlD,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,KAAiB,EACjB,KAAa,EACb,UAAkB,EAClB,IAAY,EACZ,aAAsC,EACtC,GAAmB;IAEnB,8EAA8E;IAC9E,MAAM,eAAe,GAAG,MAAM,KAAK,CAAC,kBAAkB,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC;IACnE,IAAI,eAAe,CAAC,KAAK,EAAE;QAAE,OAAO,GAAG,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;IAC/D,MAAM,eAAe,GAAG,eAAe,CAAC,KAAK,CAAC;IAE9C,MAAM,UAAU,GAAG,IAAI,GAAG,EAAmB,CAAC;IAE9C,KAAK,MAAM,KAAK,IAAI,eAAe,EAAE,CAAC;QACpC,MAAM,OAAO,GAAG,KAAK,CAAC,OAA+B,CAAC;QACtD,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC;YAC1D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;gBACzB,UAAU,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;YAC7B,CAAC;QACH,CAAC;IACH,CAAC;IAED,uEAAuE;IACvE,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,aAAa,CAAC,EAAE,CAAC;QACzD,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;YACzB,UAAU,CAAC,GAAG,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,MAAM,aAAa,GAA4B,EAAE,CAAC;IAElD,MAAM,GAAG,GAAQ;QACf,UAAU;QACV,KAAK;QACL,IAAI;QACJ,GAAG;QACH,GAAG,CAAC,GAAW;YACb,+BAA+B;YAC/B,IAAI,GAAG,IAAI,aAAa,EAAE,CAAC;gBACzB,OAAO,OAAO,CAAC,OAAO,CAAC,aAAa,CAAC,GAAG,CAAC,CAAC,CAAC;YAC7C,CAAC;YACD,OAAO,OAAO,CAAC,OAAO,CAAC,UAAU,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC;QAC9C,CAAC;QACD,GAAG,CAAC,GAAW,EAAE,KAAc;YAC7B,aAAa,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;QAC7B,CAAC;KACF,CAAC;IAEF,OAAO,EAAE,CAAC;QACR,GAAG;QACH,gBAAgB,EAAE,GAAG,EAAE,CAAC,CAAC,EAAE,GAAG,aAAa,EAAE,CAAC;KAC/C,CAAC,CAAC;AACL,CAAC"}

package/node_modules/flingflow/dist/engine.d.ts

@@ -0,0 +1,29 @@
+import { type Result } from "neverthrow";
+import { type SimulationTask } from "determined";
+import { type WorkflowAlreadyActiveError, type EventStore } from "./store.js";
+import { type Clock } from "./clock.js";
+import type { WorkflowRegistry } from "./registry.js";
+import type { ClaimResult, StepOutcome } from "./types.js";
+export declare class UnknownWorkflowError extends Error {
+    constructor(name: string);
+}
+export type CreateWorkflowError = WorkflowAlreadyActiveError | UnknownWorkflowError | Error;
+export interface EngineConfig {
+    store: EventStore;
+    registry: WorkflowRegistry;
+    clock?: Clock;
+    generateId?: () => string;
+}
+export declare class Engine {
+    private store;
+    private registry;
+    private clock;
+    private generateId;
+    constructor(config: EngineConfig);
+    initialize(): Promise<Result<void, Error>>;
+    createWorkflow(workflowId: string, name: string, writes: Record<string, unknown>, sim: SimulationTask): Promise<Result<string, CreateWorkflowError>>;
+    claimStep(runId: string, sim: SimulationTask): Promise<Result<ClaimResult | null, Error>>;
+    executeStep(runId: string, sim: SimulationTask): Promise<Result<StepOutcome, Error>>;
+    private commitStep;
+    runToCompletion(runId: string, sim: SimulationTask): Promise<Result<void, Error>>;
+}