@smthrs/engine-store 0.0.0-stage → 1.0.0-rc.4

package/docs/concepts/attempts-and-replay.md

@@ -0,0 +1,149 @@
+---
+title: "Attempts and replay"
+description: "What the engine persists for each step attempt, how a restart replays a settled row instead of re-executing it, and which admission checks refuse an attempt outright."
+sidebar:
+  order: 2
+---
+
+An attempt is one execution of one step under one key. The engine writes a row
+before the body runs and settles that row afterwards, so the record of what
+happened outlives the process that made it. Replay is the consequence: a
+restarted engine reads the row and reproduces its outcome instead of running
+the body a second time.
+
+## The step key is the address
+
+Every dispatch is keyed. The key is a digest of the declaration and the
+material the step consumes, and the engine stores it hashed: cache addresses
+are the injected `Sha256` transformation of the step key, never the raw
+`key1_...` value.
+
+Two consequences follow. A declaration change produces a new key, so a step
+whose code moved is never served the old answer. And two runs that declare the
+same step under the same material land on the same address, which is what makes
+a result shareable at all. Whether it is actually shared is a separate
+question, answered in [Cache admission](./cache-admission.md).
+
+## What a replay does with each row state
+
+| Persisted row | What a replay does                                                                                                              |
+| ------------- | ------------------------------------------------------------------------------------------------------------------------------- |
+| `succeeded`   | Returns the recorded result, and calls `StepBoundary.replayOutputs` first so the workspace holds the outputs the step produced. |
+| `failed`      | Rethrows the persisted domain failure. It never re-admits the attempt.                                                          |
+| `running`     | Recovers under the run fence and shared admission permit, honoring any recorded effect crossing.                                |
+| `suspended`   | Continues the same attempt rather than burning a new one against the retry budget.                                              |
+
+An admitted attempt remains authoritative when a matching shared cache row
+exists; another run's cached success cannot replace its result or failure.
+Failure values were schema-encoded before persistence, so a `_tag` survives
+the JSON round trip and a `RetryPolicy`'s non-retryable matching still applies
+on replay.
+
+## Which columns are redacted
+
+Attempt and run rows share a database with the journal, so failure text meets
+the journal's redaction rules (`Redaction.defaultRules`) on write. Only text
+that replay never classifies on is rewritten:
+
+| Column                        | Redacted                                                                                                 |
+| ----------------------------- | -------------------------------------------------------------------------------------------------------- |
+| `flows_attempts.error_json`   | String `message` and `stack` members of each persisted reason. `_tag` and every other member stay exact. |
+| `flows_runs.state_json`       | The `message` and `stack` text of an unencodable-settlement projection.                                  |
+| `flows_attempts.outcome_json` | Never. It is the executable copy replay returns, byte for byte.                                          |
+
+A redacted failure still replays as the same typed failure, because
+`RetryPolicy` matches on `_tag`. A value that must never reach `outcome_json`
+belongs in a `Redacted` field of the action's own success schema.
+
+## Admission is exclusive per key
+
+Each engine or plan scheduler shares admission permits across its dispatches.
+A permit covers one run and step key through admission, execution, and
+settlement. Concurrent same-key dispatches wait, then replay the settled
+attempt instead of executing another body. Interruption releases the permit.
+
+Holding the permit and the run fence permits recovery of a stranded `running`
+attempt. A recorded successful effect crossing supplies the result directly;
+an unresolved irreversible crossing requires an idempotency key before retry.
+The durable owner and admission checks still refuse a superseded fence or a
+conflicting attempt row.
+
+## Attempt counters survive pruning
+
+Two questions come up on a resume, and both are answered from the surviving
+attempt rows rather than from a counter held in memory:
+
+- Which attempt number is this? The engine resumes from the persisted
+  sequence rather than restarting at 1.
+- When did retrying start? `actionRetryOrigin` degrades to the earliest
+  surviving attempt row when a retention pass pruned attempt 1.
+- What did the world look like before retrying started? `actionSnapshot`
+  returns the earliest `SnapshotBoundary` handle a surviving compensable
+  attempt row persisted, so a restarted retry restores that pre-image. The
+  handle is written into the running row before the body runs, and a replayed
+  attempt never touches the boundary.
+
+`DurableEngineState.attemptSurvivors` answers both in one range read, returning
+`{ earliestAttempt, earliestStartedAtMs, latest }`. It is optional on the
+service: storage that cannot range-scan `flows_attempts` omits it, and the
+engine falls back to per-attempt point reads against `AttemptStore`.
+
+## The cache converges with the journal
+
+A crash can land between `attempts.finish` and `cache.put`. The restarted
+executor re-records the sealed completion, with fresh cache provenance, rather
+than leaving the cache permanently behind the journal. If the row that was
+already recorded disagrees with what this run produced, that divergence is not
+resolved silently: it goes to the `Inconsistency` receiver, which is strict by
+default and fails the dispatch.
+
+## Corrupt evidence on a succeeded row is quarantined, not evicted
+
+A shared cache row whose bytes no longer hash to their recorded digest is
+evictable: the next dispatch re-executes and re-captures cleanly, and the
+failure is reported as `CacheCorruptionDetected` so a failing disk stays
+visible.
+
+A succeeded attempt row is different. It records that this run's side effects
+already ran, so evicting and re-executing would break exactly-once for an
+irreversible action. Under the strict verdict the corrupt evidence is
+quarantined instead: the driver parks the first detection in the `quarantine`
+waiting state after `AttemptEvidenceQuarantined`. The engine's automatic polls,
+discarded-execution followers, internal wakes, and fresh `execute` calls leave
+that park intact. A waiting caller
+continues under its `suspendedRetryPolicy`; exhausting that budget reports
+`SuspendedResumeGaveUp` without clearing quarantine. Only an explicit
+`Flow.resume(executionId)` or engine `resume` authorizes recovery, returning the
+durable outcome without re-materializing the poisoned evidence or re-executing
+the action.
+
+## Every lifecycle write takes the durable channel
+
+Run decisions, attempt started and finished, hard boundary violations, snapshot
+identity, cache provenance, deferred completions, clock schedules, interruption
+records, and the `Inconsistency` cache-conflict record all go through the
+journal's `emitDurable` channel. A saturated lossy queue can never drop one.
+Attempt lifecycle writes additionally pass the owner, so a reclaimed owner
+fails with `fence_lost` and self-interrupts instead of appending.
+
+## Typed history and execution authority
+
+The shared [state and event authority contract](https://journal.smithers.sh/concepts/state-event-authority/)
+defines additive versioned event families. Current attempt writers retain their
+existing bytes. The engine's internal attempt lifecycle and projection adapters
+do not change that writer contract or expose new public entrypoints.
+
+Full history and retained snapshot plus suffix rebuild the disclosed attempt
+projection. Redacted history does not prove recovery of arbitrary private
+results or executable engine state. Keep the authoritative attempt and run
+stores; a writer cutover requires complete lineage and encoded results in the
+same state transaction, with an explicit migration for retained history.
+
+## Related
+
+- [Ownership and fencing](./ownership-and-fencing.md): who is allowed to write
+  an attempt row at all.
+- [Step boundaries](./step-boundaries.md): what `replayOutputs` reproduces, and
+  where it gets the bytes.
+- [Troubleshooting](../troubleshooting.md): each of these failures with its
+  cause and its fix.

package/docs/concepts/cache-admission.md

@@ -0,0 +1,265 @@
+---
+title: "Cache admission"
+description: "The conditions a step result must meet before another run may reuse it, how a hit is verified before it is served, and the ordering that makes a shared tier safe."
+sidebar:
+  order: 6
+---
+
+Serving a recorded result instead of executing a step is the most valuable and
+the most dangerous thing this package does. Cache admission is the set of
+conditions a result must meet before it becomes reusable, and cache
+verification is the check that runs before a recorded result is actually
+served.
+
+## What gets admitted
+
+The engine admits a cache record only when all of the following hold:
+
+- The action is `sealed`.
+- The boundary is `hard`.
+- All declared reads are exact inputs, not unresolved globs.
+- No deviation occurred.
+- The evidence explicitly carries `wholeTreeWritesVerified: true`.
+- The evidence explicitly carries `hermeticReadsVerified: true`.
+- The completed attempt carries `readSetVerified: true`: its measured inputs
+  matched the declaration when the body executed.
+
+Fresh results and recovered durable completions use the same internal
+publication decision. Refusal keeps a completed action's run-local outcome;
+it does not require the body to execute again. Known-corrupt evidence cannot
+be published merely because the inconsistency policy tolerates that outcome.
+
+Older evidence, and boundaries that observe declared paths only, are
+conservatively refused. Under the production composition the proof comes from
+[`WorkspaceSandbox`](./workspace-transactions.md): the body ran in an isolated
+workspace, so a write outside the declared set is a map comparison rather than
+an inference, and the engine sets the flag itself. When the whole-tree
+diff shows a deviation the declared-read scan would have missed, it records that
+deviation and withholds the entry instead.
+
+One more distinction decides reach rather than admission. Only a content-key
+record has an address another run can reproduce; an ordinal-key record stays
+run-local.
+
+## A hit is verified before it is served
+
+A matching key is not sufficient. Before serving, the store calls
+`StepBoundary.prepare` and compares the descriptor's declared `readSet` against
+the `readSnapshot` the host measured:
+
+- Every declared read still matching permits output replay verification.
+- A declared path that is missing, or whose digest differs, refuses the hit,
+  journals a `cache-provenance` record with `action: "stale_read_set"`, and
+  falls through to a real execution.
+- Reads the host reports but the declaration never claimed are ignored: the
+  declaration is what the key was computed over.
+
+This is Skyframe's dirty-check invariant. The key alone detects a changed
+declaration, never a stale one.
+
+A verified hit then calls `replayOutputs` before returning the stored result.
+Unlike completion publication, candidate admission does not require the old
+`readSetVerified` flag: current inputs are freshly measured here. Missing
+whole-tree or hermetic-read evidence still refuses the candidate. An unavailable
+measurement does not prove staleness and does not justify evicting the row.
+
+When that refuses with `MissingArtifact`, the normal first answer for a row
+recorded on a machine whose artifacts this one has never seen, the dispatch
+hydrates from the shared tier and retries the replay exactly once before
+falling through to a real execution. A second failure means the tier cannot
+serve it either, and executing is strictly better than looping.
+
+## Current output authority and legacy flags
+
+Before measuring a candidate or replaying a durable outcome, the engine checks
+the entire recorded output manifest against the current boundary descriptor.
+Each write must be covered by the current write set, each deletion must be an
+explicit removal, and every exact output and removal must be present. Tree
+pruning requires the same declared tree root; a glob or a narrower subtree
+cannot authorize pruning a broader recorded tree. A tree root must remain a
+directory; a manifest cannot replay a file at that path. Duplicate and noncanonical
+recorded paths are refused. The existing abstract `{ paths: [...] }` boundary
+format requires an identical current write set and no removals. An unknown
+output format is refused as `unsupported-output-evidence`. Malformed production
+evidence cannot use the abstract path-list fallback.
+
+A descriptor mismatch is `output-boundary-mismatch`, recorded in the existing
+`cache-provenance` family with `action: "replay_failed"`. It permits no replay
+call, hydration, pruning, or age-based eviction of that candidate. This is a
+preflight guarantee for policy refusal, not atomic rollback of filesystem work
+if an admitted replay subsequently encounters host failure or corruption.
+A succeeded attempt retains its original outcome and metadata, returns that
+outcome without executing its body again, and withholds cache publication when
+its output evidence is refused.
+
+Stored `boundaryQuarantined: true` always forbids shared reuse and publication.
+Quarantine together with boundary evidence or `readSetVerified: true` is
+`contradictory-evidence`; verified reads with no boundary evidence have the same
+refusal. Neither a historical verification flag nor a tolerant inconsistency
+policy overrides quarantine. These legacy rows remain unchanged and their
+run-local durable outcomes remain authoritative. A candidate without the old
+read-verification flag can still be admitted by fresh current-read measurement.
+Proof flags retain their existing true-or-omitted schema. Stored false values
+or strings fail metadata decoding as `invalid-meta` for shared reuse, while
+the run-local durable outcome remains authoritative.
+
+## Empty output sets
+
+An action with no declared writes or removals can publish and converge a cache
+entry. For replay and convergence, its evidence must explicitly encode the empty
+set as `{ outputs: [] }` (with no tree roots) or the abstract `{ paths: [] }`
+format. All ordinary boundary, whole-tree, hermetic-read and execution-time read
+proofs still apply.
+An empty set does not establish those proofs by itself. A durable completion
+without a finish timestamp uses the current clock when it converges the entry.
+
+Empty outputs do not bypass replay verification. If the boundary reports
+`BoundaryCorruption` in admitted evidence, the succeeded attempt's evidence is
+quarantined, its boundary and read-verification proof are cleared, and strict
+policy parks the run until an explicit resume; polling and repeated execution
+cannot clear quarantine. That explicit resume returns the durable
+outcome without rechecking that evidence, repeating the body, or publishing it.
+
+An opaque `{}` is unsupported evidence, not an encoded empty set. An empty
+manifest that omits a declared exact output, removal, or tree root is a boundary
+mismatch. These preflight refusals preserve the succeeded outcome and original
+metadata, withhold convergence, and never call the boundary, so they do not
+trigger corruption quarantine. Undecodable metadata likewise supplies no
+reusable proof; it does not manufacture a corruption verdict.
+
+## Age decisions, policy removal, and forked history
+
+The first age verdict for a cache provenance is durable: age equal to `ttlMs`
+is admitted, and age one millisecond greater is expired. Replaying that verdict
+uses its original answer when the clock moves in either direction. An exact
+duplicate of the opposite measured verdict proves a clock change; an arbitrary
+journal conflict never does.
+
+Removing `ttlMs` after any recorded age verdict for this cache address within
+the run is refused with `JournalError.code: "idempotency_conflict"`. This holds
+even if the head has since been evicted or replaced. Restore the original
+policy or use a new action identity. Before a verdict exists, the policy can
+still change. Scope continues to narrow the cache address without changing the
+durable attempt identity.
+
+The no-TTL guard indexes verdicts per executor and reads only journal entries
+after its last validated sequence on subsequent dispatches. It checks new
+entries before answering absence. Resume, a different journal service, or a
+changed rewind generation rebuilds the index from retained history once.
+EngineStore shares it across a run's action instances; PlanScheduler shares it
+across its node dispatches.
+
+A fork may reuse a copied parent's age verdict only after a paged history
+lookup validates the complete producer identity, event type, payload including
+TTL and recorded provenance, sequence, timestamp, and lineage metadata against
+the retained ancestor record. Each ancestry edge also requires the recorded
+fork marker, its producer, exact cutoff, and cutoff lineage. Forks of forks
+must validate every intervening copied prefix. Missing ancestry, contradictory
+fields, and unrelated foreign lineage retain the typed `idempotency_conflict`
+with its cause. A journal read failure, including a compacted cursor, preserves
+the journal's original typed error. An identity refusal also records
+`action: "replay_failed", reason: "incompatible-age-history"` in the existing
+cache provenance family. No new event family, persisted schema, key, or producer
+identity is introduced; transparent reuse of unvalidated foreign history is
+unsupported.
+
+## Publication order across two tiers
+
+A shared tier turns one machine's result into every machine's result, and the
+order the two writes happen in is a correctness property, not a preference.
+
+1. `ArtifactSync.publish(digests)` runs first, immediately before the
+   transaction that records the cache entry and never inside it. It probes the
+   shared tier with `findMissing`, uploads what is missing, and re-probes to
+   confirm. A publication that cannot make the artifacts durable fails with
+   `ArtifactPublicationFailed`, and the shared entry is withheld.
+2. The local cache row and the journal record explaining it commit in one
+   `DurableWriter` transaction.
+3. `CacheSync.publishEntry(entry)` runs after that transaction, publishing the
+   already-durable local entry to the shared step-result tier.
+
+Step 1 before step 2 is Bazel's REAPI ordering constraint: an action result is
+uploaded after every blob it refers to, because a result accessed before its
+blobs are present cannot be validated. Step 3 after step 2 is why `CacheSync`
+is a separate seam from the `CacheStore` tag at all: a `CacheStore` whose `put`
+also wrote a shared HTTP tier would put a network round trip inside a write
+transaction, blocking every other writer for its duration and rolling the local
+row back whenever a shared cache was unreachable.
+
+The ordering makes a shared tier consistent, not trustworthy. A hit replays the
+entry's evidence into the workspace, writing its outputs and deleting its
+`digest: null` paths, and every check that runs is against digests the same
+entry recorded. Only producers you trust as much as your own machines may write
+to the shared tier. See
+[Trust every writer of the shared tier](../guides/share-a-cache-across-machines.md#trust-every-writer-of-the-shared-tier).
+
+## Neither publication step can fail a run
+
+Both run after `attempts.finish`, so the result is already durably recorded on
+this host. Failing a completed run because an optional accelerator is
+unreachable trades a real result for an unavailable one.
+
+A refusal withholds the shared copy, never the local row, and journals a
+`cache-provenance` record with `action: "unpublished"` carrying the stage
+(`artifacts` or `entry`) and the reason. A missing shared entry is therefore
+explainable from the journal rather than inferred from its absence.
+
+Local cache lookup and publication failures are also accelerator failures. A
+typed store refusal is journaled and a succeeded durable attempt remains
+usable. A failed cache transaction leaves neither its row nor its provenance
+record; reopening can publish from the terminal attempt without repeating the
+body. A crash after the local cache row commits may leave the optional remote
+entry absent: a subsequent local hit does not retry remote publication.
+
+## Download policy on the read side
+
+`ArtifactSync.hydrate` establishes that this host can resolve every referenced
+artifact. How eagerly it materializes them is `DownloadPolicy`, declared on the
+shared tier as `RemoteArtifacts.Options.downloadPolicy` and read from the store
+`ArtifactSync.make` was handed, so one deployment setting reaches both seams. An
+explicit `downloadPolicy` on `make` or `layer` overrides it.
+
+| Policy          | Behavior on hydrate                                                                                                                                                                             |
+| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `all` (default) | Downloads every referenced artifact into this host's store while admitting the replay. Every later read is local, and a shared tier that goes down afterwards costs nothing.                    |
+| `toplevel`      | Downloads nothing. One batched `findMissing` establishes that the shared tier can serve what is missing, and `CombinedArtifacts.get` fetches and writes back the blobs a reader actually reads. |
+| `minimal`       | The same probe and the same zero downloads; `CombinedArtifacts.get` then serves without writing back, so this host never accumulates other machines' artifacts.                                 |
+
+The two lazy policies are sound only when the store the replay reads through
+can reach the shared tier, which means `CombinedArtifacts` with the same remote
+tier. Under a purely local `ArtifactStore` an admitted lazy replay would later
+read an artifact this host never fetched. A tier that refuses the probe is
+indistinguishable from one that holds nothing, so the replay is refused either
+way and the step executes.
+
+## When two runs disagree
+
+Two runs that record different results under the same content key mean the
+declaration does not fully describe what the step depends on. That is a
+hermeticity violation, and it goes to the `Inconsistency` receiver:
+
+- `Inconsistency.layerStrict(owner)` journals the conflict and fails the
+  dispatch. This is the default for engine wiring, because a non-hermetic
+  sealed hard-boundary action is a defect rather than a condition to paper
+  over.
+- `Inconsistency.layerTolerant(owner)` journals and continues, preserving the
+  first-recorded row. That is Skyframe's tolerant production configuration.
+- `Inconsistency.layerNoop()` journals nothing and tolerates everything.
+
+The record goes through the journal's durable channel, so a `tolerate` verdict
+that silently dropped its only record cannot wire the detector to nothing.
+
+An explicitly nondeterministic declaration instead keeps the first local writer
+and records `conflict_first_writer`; it never calls the deterministic
+inconsistency receiver. The optional remote publisher also retains its existing
+first-writer policy. Neither policy turns a local deterministic conflict into
+an availability failure.
+
+## Related
+
+- [Step boundaries](./step-boundaries.md): where `wholeTreeWritesVerified`
+  comes from.
+- [Share a cache across machines](../guides/share-a-cache-across-machines.md):
+  the composition that turns this on.
+- [Content addressing](/docs/concepts/content-addressing/) on smithers.sh: how
+  a step key is built in the first place.

package/docs/concepts/call-facts.md

@@ -0,0 +1,86 @@
+# Native call facts and their outbox
+
+`flows.harness.call-fact.v1` is the versioned durable producer for agent cell
+calls. It is separate from `AgentSession`'s best-effort trace consumer. The
+immutable native journal is the outbox; there is no second delivery queue or
+consumer-side writer that reconstructs a missing call from a flow name.
+
+## Identity and authorization
+
+The harness constructs the existing dispatch identity before invoking a call:
+logical run, frame, cell digest, invocation ordinal, declaration digest, and
+ordered active layers. Its stable `cell-call-v1:<sha256>` digest is unchanged.
+The public fact calls the logical run coordinate `identity.runId`; it is not
+an authentication session. Existing action keys and legacy trace producer IDs
+are unchanged. The annotations are additive metadata, never key material.
+
+`FlowEngineLike.call` authorizes before constructing the annotated native
+action. Only then can its owner commit the `invoked` fact. Permission refusal
+or a parked approval does not claim a native invocation. Legacy trace telemetry
+may describe that attempted call, and the separate approval stream describes
+the durable request.
+
+## Commit and delivery
+
+Fresh attempt admission, its existing lifecycle records, and the invocation
+fact share one fenced native transaction. Failure rolls admission back and
+prevents the handler, sandbox, and result publication. A verified cache replay
+commits invocation before materializing outputs. An older completed native
+attempt can gain an invocation observation without claiming the handler ran
+again.
+
+The harness records the exact success/failure branch it will return to the
+cell through its existing `record("cell-call")` action. Its additive annotation
+carries the same dispatch identity. `ActionPersistence` commits the `settled`
+fact with that controller attempt's succeeded row, using the actual encoded
+outcome and validating its result codec. The host operation's success alone
+cannot replace a timeout or failure delivered by the harness.
+
+A failed settlement commit returns no completed result and publishes no fact.
+On reopen, an already durable succeeded effect crossing can finish that same
+controller attempt and fact without rerunning its body. The fact proves a
+committed result ready for delivery; a process can still crash before consuming
+it and replay it later. This does not promise exactly-once external side effects
+or change the existing timeout/reissue policy.
+
+Fact identity is `(native run, "call-fact-v1:<callId>:<phase>", 0)` under the
+native journal generation. Exact retries receive the existing commit receipt.
+A changed payload under that identity is a conflict. No handler result is
+published through a best-effort fallback after a durable write failure.
+
+## Outbox and projections
+
+`EngineJournalProjection` copies committed native entries into the control
+journal with its existing source identities. It resumes after a lost receipt
+or process reopen; the destination's unique producer key suppresses already
+committed copies. Source and destination are separate transactions, so the
+bridge can lag and cannot turn an uncommitted source attempt into public state.
+
+Gateway's browser-safe shared call normalizer validates the envelope version,
+native coordinates, logical run, and producer identity. Its pure fold prefers
+native facts over identified telemetry, keeping the first display position and
+stable public node IDs. CLI progress corrects an earlier telemetry outcome
+without double-counting. Transcript subscriptions use the existing snapshot
+reset when a later native fact changes an already emitted row; fresh snapshots
+and resumed subscriptions then agree.
+
+All public inputs, values, and messages remain subject to journal redaction.
+An individual field larger than 65,536 UTF-8 bytes becomes a byte count and
+canonical SHA-256 commitment. Protected native attempt rows retain the full
+encoded result. A public fact cannot reconstruct a credential or a truncated
+private value, and no redactor is bypassed.
+
+## Migration and coverage
+
+The writer is additive: no historical rewrite, action-key migration, or trace
+identity change is required. A legacy native result acquires a fact only when
+the actual annotated record boundary reads it successfully. An invalid encoded
+outcome cannot become a deliverable fact.
+
+Legacy trace rows without a call ID retain FIFO matching among unidentified
+starts. They cannot safely be merged with new native identities by flow name,
+input, or guessed ordinal. Native facts do not fill missing historical model
+deltas, printed output, or complete trace prefixes. Low-level custom engine
+ports without these annotations remain legacy producers. Native journal
+generation gaps remain visible in raw bridge history; the call fold does not
+claim complete run-history coverage or authorize work from public projections.