phronomy 0.26.0 → 0.28.0

data/docs/persistence-backends.md

@@ -1,179 +1,165 @@
-# Persistence backend contract
-
-`Phronomy::Persistence` is the single durable-state backend abstraction used by
-stateful Agents and durable Workflows. This document is the normative contract
-for authors of custom Persistence backends.
-
-The Backend SPI is **Beta**. It may evolve in a minor pre-1.0 release, but a
-backend should not depend on Phronomy private APIs or Runtime internals.
-
-## Architecture boundary
-
-A backend implements durable storage only:
-
-```text
-Application
-    ↓
-Agent / Workflow
-    ↓
-Runtime / EventLoop / ExecutionCoordinator
-    ↓
-Phronomy::Persistence synchronous Backend SPI
-    ↓
-Database / durable storage
-```
-
-Persistence does not own live Agent identity, top-level Runtime admission, or
-live execution state. In particular, a backend must not persist or reconstruct
-the following as part of this SPI:
-
-- Runtime Agent ownership-registry entries;
-- EventLoop Agent top-level admission entries;
-- EventLoop Agent execution-directory entries;
-- `AgentInvocation`;
-- `FSMSession`;
-- `TaskResult` or callbacks;
-- EventLoop queue contents;
-- Runtime Workflow admission entries;
-- in-flight provider operations.
-
-Persistence operations are synchronous. Framework-owned blocking Persistence I/O
-is submitted to the Runtime OffloadPool by Phronomy; a backend must not post
-EventLoop events or introduce `load_async` / `save_async` variants into this
-contract.
-
-## Required root surface
-
-A Persistence backend exposes eight durable repositories:
-
-```text
-contents
-agents
-journals
-executions
-workflow_states
-handoff_states
-teams
-team_executions
-```
-
-and two root operations:
+# Persistence and neutral Storage SPI 2
+
+[ADR-058](decisions/058-neutral-storage-primitives.md) defines the current
+extension contract. This is an intentional breaking replacement of the old
+fixed-repository Backend SPI. Application construction now uses
+`PersistenceComposition.build(backend:)` or `.in_memory`; pass its `agent`, `team`,
+or `workflow` component to that domain. `Persistence.new(backend:)` provides only
+the neutral coordinator. The old eight-repository facade and its queries were
+removed; see [r8 unit 3](architecture/r8-unit3.md).
+
+## Ownership and composition
+
+Storage owns `Resource`, `Backend`, `View`, `Records`, `Streams`, `Blobs`, immutable
+`Entry` values, guards, conditions and neutral exceptions. It has no Agent/Team
+record types, active-execution policy, content digest algorithm or watermark API.
+
+Agent, Team, Workflow and ContentStore declare their own resource schemas.
+`PersistenceComposition::StorageSchema` gathers those declarations and
+`PersistenceComposition` binds domain record adapters to one coordinator.
+Team and Workflow schemas live in their existing `persistence/` implementation
+directories. An isolated Agent uses `PersistenceComposition.agent` and does not
+construct Team or Workflow stores.
+`ContentStore::StoredContents` owns SHA-256 identity and digest verification.
+The Agent-owned `Watermark` composes guarded revision and stream-head conditions.
+
+InMemory receives `resources:` and supplies one Monitor/snapshot transaction
+across the catalog. SQL reference composition supplies the same catalog and a
+separate physical table/column mapping to the neutral driver in examples
+`shared/storage`. Table names, columns, indexes, DurableRecord envelopes, payloads,
+format versions and content identities remain unchanged.
+
+The [S3 closure review](architecture/refactoring-closure.md) records why these
+names and placements remain and separates applied evidence from the candidate.
+
+## Backend and View
 
 ```ruby
-persistence.transaction { |tx| ... }
-persistence.assert_agent_watermark!(
-  agent_id:,
-  agent_revision:,
-  journal_position:
-)
+backend = Phronomy::Storage::Backends::InMemory.new(resources: resources)
+backend.view.records(resource)
+backend.view.streams(resource)
+backend.view.blobs(resource)
+backend.transaction { |view| ... }
+view.check!(guards: guards, conditions: conditions)
 ```
 
-The object yielded by `transaction` is a transaction-scoped Persistence view. It
-must respond to all eight repository accessors and
-`assert_agent_watermark!`. It may be the Persistence instance itself, but SQL
-backends may instead yield an object bound to a checked-out connection or
-transaction session.
-
-## Required capabilities
-
-Every backend must advertise:
-
-```ruby
-{
-  atomic_all: true,
-  atomic_admission: true,
-  optimistic_revision: true
-}
-```
-
-`Phronomy::Persistence::REQUIRED_CAPABILITIES` is the executable definition of
-this requirement.
-
-### `atomic_all`
-
-All durable repositories must be able to participate in one atomic transaction
-domain. A transaction may change `contents`, `agents`, `journals`, `executions`,
-`workflow_states`, `handoff_states`, `teams`, and `team_executions`, and then either commit all changes or roll them all back.
-
-This requirement deliberately does not claim exactly-once semantics after an
-indeterminate database/network failure. If the underlying database cannot tell
-the caller whether a commit happened, the backend should surface the storage
-failure rather than pretending the outcome is known.
-
-### `atomic_admission`
-
-This capability is a **durable Agent execution integrity defense**. It is not the
-primary same-process Agent ownership/admission mechanism and it is not Workflow
-distributed locking.
-
-For one Agent, `executions.create_active` must atomically guarantee both:
-
-```text
-execution_id is unique
-AND
-no active/suspended execution already exists for agent_id
-```
-
-A conflict with an existing active/suspended execution raises
-`Phronomy::AgentBusyError`.
-
-Within one process, Runtime/EventLoop admission is acquired before the initial
-Persistence operation and is the primary competing-execution exclusion
-mechanism. `atomic_admission` remains required as the durable second line of
-defense against stale paths, durable conflicts, and unsupported cross-process
-races. It must not be removed merely because Runtime admission exists.
-
-Workflow admission remains Runtime/process-local. Cross-process Agent or
-Workflow ownership/lease/fencing is a separate distributed-coordination concern
-and is not part of this Backend SPI.
-
-### `optimistic_revision`
-
-The backend must implement compare-and-swap semantics used by Agent roots,
-Agent executions, Journals, Workflow snapshots, and the durable Agent watermark.
-Stale writers must receive `Phronomy::Persistence::ConflictError`; they must not
-silently overwrite newer durable state.
-
-## Error contract
-
-Backends should translate backend-specific constraint errors into the following
-portable Phronomy errors when the meaning matches.
-
-### `Phronomy::Persistence::NotFoundError`
-
-A requested durable record does not exist.
-
-### `Phronomy::Persistence::ConflictError`
-
-A persistence precondition failed, including revision, Journal position,
-identity, duplicate-ID, or compare-and-swap conflicts.
-
-### `Phronomy::AgentBusyError`
-
-A durable nonterminal Agent execution already exists and another durable
-execution record cannot be established. Phronomy also uses the same public error
-for a competing process-local top-level request rejected by Runtime/EventLoop
-before the backend is called.
-
-### `Phronomy::Persistence::SerializationError`
-
-The backend cannot encode a value into its supported durable representation.
-This is intended primarily for durable backends whose Workflow state domain is
-narrower than the InMemory backend's Ruby-object domain.
-
-### `Phronomy::Persistence::UnsupportedBackendError`
-
-The backend does not provide a required structural or capability contract.
-
-Database availability, connection loss, and other transport/storage failures
-must not be misreported as ordinary optimistic conflicts merely to fit this
-error taxonomy.
+The backend declares `spi_version: 2` and true values for `atomic_resources`,
+`record_cas`, `stream_cas`, `conditional_unique`, `guarded_checks` and
+`nested_savepoints`. Persistence validates backend capabilities; composition
+validates the required domain resource declarations before binding adapters.
+Unsupported backends raise `Persistence::UnsupportedBackendError`. There is no
+eight-slot compatibility adapter or duplicated facade capability API.
+
+A root handle routes to the current transaction on the same backend/thread.
+A bound view and all its handles expire on commit or rollback and reject use from
+another thread. Explicit nested transactions use savepoints on the same SQL
+connection, or nested InMemory snapshots. Inner success depends on outer commit;
+an inner exception is re-raised after rollback and can be caught by the outer
+scope. `ActiveRecord::Rollback` also propagates.
+
+A failure during physical work marks the scope failed. Catching it inside that
+same scope does not permit further operations or successful commit. Establish an
+explicit inner transaction before a recoverable operation and catch outside it.
+Input validation occurs before physical work. A successful optional read returning
+nil is not a failed physical operation; domain required-load errors may be raised
+after that read. Domain decode/returned-metadata failures stay inside the atomic
+boundary so a corrupt response after a write causes rollback.
+
+Transaction blocks must finish normally. `return`, `break` and `throw` escaping
+them cause rollback and `TransactionError`. These exits are not commit controls.
+Commit/rollback transport failures remain database failures; this contract does
+not promise exactly-once external effects or infer commit certainty.
+
+## Resource declarations
+
+A `Resource` is an immutable value with `id`, `kind`, `attributes`,
+`immutable_attributes`, `indexes`, `unique` and optional `guard`. Attributes use
+`:string`, `:integer`, `:boolean` and their explicit `:nullable_*` forms. Keys and
+text attributes use valid UTF-8 without NUL; keys are nonempty. Binary data belongs
+to Blobs. No Proc, SQL expression or payload predicate is accepted.
+
+Named equality indexes specify exact fields. Records may declare conditional
+unique constraints with a symbol name, fields and equality `where` values. Null
+unique-key fields are distinct, matching the default SQL unique-index semantics.
+Streams have no indexed attributes; Blobs have attributes but no indexes/guards.
+A record guard uses its key or an immutable non-null string attribute. A stream
+guard uses its stream ID. Its anchor must be a registered Records resource.
+Required anchors must exist; missing parents raise `NotFoundError` in all drivers.
+
+## Records
+
+| Operation | Contract |
+|---|---|
+| `insert(key:, revision:, attributes:, record:)` | Insert only when absent; return an independent immutable `Entry::Record`. Primary duplicates and named unique failures differ. |
+| `read(key)` / `fetch(key)` | Optional nil / required `NotFoundError`. |
+| `replace(key:, expected_revision:, next_revision:, attributes:, record:, expected_attributes: {})` | Check existence, revision, expected attributes, immutable fields and uniqueness atomically; next revision must equal expected + 1. Replace all attributes. |
+| `delete(key:, expected_revision: Records::UNCHECKED)` | Unchecked deletion is idempotent. A supplied revision requires existence and equality. Return nil. |
+| `scan(index:, equals:, after: nil, limit: nil)` | Exact named-index fields, UTF-8 byte order, exclusive key cursor, positive limit or nil for all. |
+| `delete_matching(index:, equals:)` | Delete equality matches atomically; a parent-guarded resource must constrain the guard attribute. Return nil. |
+
+An Entry carries key, revision, attributes and an opaque `DurableRecord`.
+The driver does not reconstruct metadata from payload. Domain wrappers verify
+that returned identity, revision and attributes agree with decoded domain data.
+Workflow/Handoff choose initial revision 1 and route nil expected revision to
+insert; nil is never an unchecked update. Team terminal-to-active rejection is a
+domain `expected_attributes: {active: true}` precondition when saving active state.
+
+## Streams and Blobs
+
+Streams expose `append(stream:, expected_head:, entries:)`,
+`read(stream:, after: 0, limit: nil)`, `head(stream:)`, and `delete(stream:)`.
+Append takes `Entry::Append(id:, record:)`, validates the entire batch before any
+write, enforces unique entry IDs within a stream, assigns contiguous positions
+and updates the head atomically. Empty append still checks the expected head.
+Reads return immutable `Entry::Stream(position:, id:, record:)` values in position
+order. Delete removes head and entries together. The domain Journal wrapper
+preserves its existing `limit: 0` empty-result behavior without a raw zero-limit
+operation.
+
+Blobs expose `put_if_absent(key:, bytes:, attributes:)`, `fetch(key)` and
+`exist?(key)`. Same bytes retain the first attributes; different bytes for an
+existing key raise `BlobConflictError`. Entry bytes are independent immutable
+binary strings. Blob keys are arbitrary storage keys; ContentStore adds the
+`sha256:<digest>` contract and maps integrity failures to its own `IntegrityError`.
+
+## Guards, conditions and errors
+
+`GuardRef(resource:, key:)` names a stable existing parent record. `View#check!`
+acquires guards in resource/key byte order before evaluating the closed condition
+set: `RevisionIs`, `StreamHeadIs`, `NoRows`. A condition must include the guard
+required by its resource and scope. PostgreSQL locks parents before child heads
+or records; SQLite relies on the transaction/CAS/unique constraints and must not
+claim a SELECT alone reserves a writer. Transactions spanning multiple owners
+must establish a consistent owner lock order; database deadlocks remain database
+errors, never optimistic conflicts.
+
+`UniqueConstraintError < ConflictError` carries the resource ID and constraint
+name. Agent/Team translate only their exact `one_active_owner` constraint to
+`AgentBusyError`. Their idle checks translate their own `NoRows` condition failure.
+`ConditionFailedError` carries the failed condition. Duplicate identities,
+stale revisions and ordinary constraint conflicts remain `ConflictError`.
+`NotFoundError`, `SerializationError`, `UnsupportedBackendError` retain their
+meanings; `TransactionError` identifies invalid scope use. The removed
+`ActiveExecutionConflictError` and `Storage::Repositories` have no aliases.
+
+The following sections describe the **domain-owned record protocols**, used by
+domain implementations and backend conformance tests. They are not a shared
+application facade. Unqualified
+ConflictError, NotFoundError, SerializationError and UnsupportedBackendError in
+these domain sections refer to `Phronomy::Persistence` errors. Repository exits
+translate the four raw Storage categories, preserving `cause`, message and trace.
+Backend implementation code continues to raise `Phronomy::Storage` errors as
+specified above. Unknown/transport errors and Storage::TransactionError are not
+translated. See [ADR-061](decisions/061-persistence-failure-contracts.md) and the
+[application migration](migrations/persistence-failure-contracts.md).
 
 ## Contents repository
 
-The content repository should normally inherit from
-`Phronomy::ContentStore::Base`, which supplies text/JSON helpers and the canonical
-content-ID calculation.
+`ContentStore::StoredContents < ContentStore::Base` supplies text/JSON helpers
+and canonical content-ID calculation over neutral Blobs. The private
+Persistence::ContentRepository adapts its primitive methods to the upper failure
+contract when accessed through `persistence.contents`. Direct StoredContents
+callers retain the raw Storage error surface.
 
 Required primitive surface:
 
@@ -188,7 +174,7 @@ Required semantics:
 - content is immutable and content-addressed;
 - writing identical bytes is idempotent and returns the same content ID;
 - `fetch` returns a binary `String` isolated from caller mutation;
-- a missing content ID raises `Persistence::NotFoundError`;
+- a missing content ID through `persistence.contents` raises `Persistence::NotFoundError`;
 - one content ID must never resolve to different bytes; a digest-integrity
   violation raises `ContentStore::IntegrityError`.
 
@@ -374,10 +360,9 @@ honestly rather than converting them into `ConflictError`.
 
 ### Workflow value serialization
 
-`WorkflowContext#to_h` may contain ordinary Ruby application values. The
-InMemory backend can preserve a broader set of Ruby values than a JSON database.
-A durable backend is not required to serialize arbitrary Ruby objects such as
-`Proc`, IO objects, sockets, or runtime callbacks.
+The Workflow domain codec accepts canonical JSON-compatible snapshot values in
+all drivers, including InMemory. Proc, IO, sockets and runtime callbacks are not
+durable snapshot values.
 
 A JSON/JSONB backend should document its supported value domain. A recommended
 domain is:
@@ -391,8 +376,9 @@ Array of supported values
 Hash with String/Symbol keys and supported values
 ```
 
-If a value cannot be represented, raise `Persistence::SerializationError` rather
-than silently converting it into a lossy form. JSON backends may return String
+If a value cannot be represented, the backend/codec raises
+`Storage::SerializationError`; the domain repository exposes
+`Persistence::SerializationError`. Do not silently convert it into a lossy form. JSON backends may return String
 keys after decoding; `WorkflowRunner` deliberately accepts String and Symbol keys
 and normalizes them when comparing durable snapshots.
 
@@ -401,8 +387,8 @@ particular database backend accept arbitrary Workflow values.
 
 ## Durable Agent watermark
 
-`assert_agent_watermark!` is a public **Backend SPI** operation. It is not an
-ordinary application API.
+`assert_agent_watermark!` is a domain Persistence operation. Agent-owned
+`Watermark` composes a parent guard, root revision and Journal head conditions.
 
 Phronomy uses it at durable barriers because a hydrated live Agent owns the
 current logical state and Phronomy deliberately does not reload mutable Agent
@@ -421,16 +407,17 @@ differs, raise `ConflictError`. On success return `true`.
 The operation must not return a replacement AgentRoot or Journal. A mismatch is a
 conflict, not a request to reload/merge mutable state.
 
-When used inside `Persistence#transaction`, a SQL backend should perform the
+When used inside `Agent::Store#transaction`, a SQL backend should perform the
 watermark check in the same database transaction as the subsequent durable
 write.
 
 ## Transaction contract
 
-A transaction block may combine operations across all repositories:
+A domain transaction combines its own record operations. Multiple domains join
+one coordinator scope explicitly; each receives only its own record protocol:
 
 ```ruby
-persistence.transaction do |tx|
+stores.agent.transaction do |tx|
   tx.assert_agent_watermark!(...)
   content_ref = tx.contents.put_text("...")
   tx.journals.append(...)
@@ -482,9 +469,9 @@ This is a targeted migration rule for the removed generic identity field. It doe
 not establish a general unknown-field or long-term codec/schema-versioning
 policy.
 
-The canonical Hash representation is the Phronomy/domain boundary. A backend is
-free to map that representation to normalized SQL columns, JSON, or another
-storage format internally.
+Domain codecs own canonical Hash representation. The raw driver stores the
+DurableRecord envelope and separately supplied metadata without interpreting
+domain payload fields.
 
 ## Conformance tests
 
@@ -508,6 +495,11 @@ an Agent repository
 a Journal repository
 an Execution repository
 a workflow state repository
+a Handoff state repository
+a Team repository
+a Team execution repository
+neutral storage primitives
+storage transaction boundaries
 a Persistence backend
 ```
 
@@ -518,7 +510,7 @@ require "phronomy"
 require "phronomy/testing/persistence_contract"
 
 RSpec.describe MyPersistenceBackend do
-  let(:persistence) { described_class.new(...) }
+  let(:stores) { Phronomy::PersistenceComposition.build(backend: described_class.new(...)) }
 
   it_behaves_like "a persistence content store"
   it_behaves_like "an Agent repository"
@@ -532,7 +524,7 @@ RSpec.describe MyPersistenceBackend do
 end
 ```
 
-`Persistence::InMemory` is run through the same shipped contract source in
+`PersistenceComposition.in_memory` is run through the same shipped contract source in
 Phronomy CI. The files under `spec/support/shared_examples/` are compatibility
 require wrappers only; the authoritative shared-example implementations live
 under `lib/phronomy/testing/persistence_contract/` so the core suite and external
@@ -560,34 +552,17 @@ may use combinations of:
 Backend-specific database exceptions should be translated to the Phronomy error
 contract where their meaning is known.
 
-## V2 coordination Backend SPI (clean break)
+## Coordination domain repositories
 
-Both `Persistence.new` and `build_transaction_view` require all eight raw
-repositories. Five-repository fallback is removed. Existing record formats remain
-`0.1`; three additional record types are `phronomy.handoff_state`,
-`phronomy.team_root`, and `phronomy.team_execution`. Backend index metadata is
-explicit and MUST NOT be reconstructed by parsing DurableRecord payloads.
+Handoff starts at revision 1 with expected revision nil; later saves advance one
+revision. Team roots and executions begin at revision 0. Team admission retains
+`AgentBusyError`; stale CAS and duplicate identities use `ConflictError`.
+Missing Team/execution loads raise `NotFoundError`; absent Handoff state returns
+nil. The unchanged record types are `phronomy.handoff_state`, `phronomy.team_root`
+and `phronomy.team_execution`, all version `0.1`.
 
-| Raw repository | Required operations and explicit metadata |
-|---|---|
-| `handoff_states` | `load(main_agent_id)`; `save(main_agent_id, expected_revision:, next_revision:, active_agent_id:, record:)`; `delete(main_agent_id, expected_revision:)` |
-| `teams` | `create(team_id:, team_revision:, record:)`; `load(team_id)`; `save(team_id, expected_revision:, next_revision:, record:)`; `delete(team_id)` |
-| `team_executions` | `create_active(team_execution_id:, team_id:, execution_revision:, record:)`; `load(id)`; `save(id, expected_revision:, next_revision:, team_id:, active:, record:)`; `list_active(team_id)`; `list(team_id, after: nil, limit: 100)`; `delete(id)`; `delete_for_team(team_id)`; `assert_idle!(team_id)` |
-| `executions` extension | `list(agent_id, after: nil, limit: 100)` for retained active and terminal records |
-
-Handoff starts at revision 1 with expected revision nil; later saves advance
-exactly one revision. Team roots and executions begin at revision 0. Team
-admission has the existing `AgentBusyError` contract for an already active owner;
-stale CAS/duplicate identity uses `ConflictError`. Missing Team/execution loads
-raise `NotFoundError`; absent Handoff state returns nil. Unavailable reads and
-codec errors must propagate separately.
-
-Lists sort IDs lexically, use an exclusive ID cursor, return at most the positive
-integer limit, include only the requested owner, and return immutable copies.
-Backends define retention; enumeration does not implement input correlation or
-request deduplication. Terminal-to-active rewrites are invalid.
-
-For F1, atomic commit does not imply a known response. Phronomy reads back the
-same intended ID/fact before advancing. A backend must provide authoritative
-reads/CAS; failure of readback is returned without new semantic work. No callback
-ACK/index is part of the SPI. Existing cross-process exclusion limitations remain.
+Backend authors should also read the [SPI 2 migration guide](migrations/neutral-storage-spi.md).
+The Stable/Beta product API snapshot and the explicit Storage SPI 2 signature
+snapshot are separate gates. Live PostgreSQL locking and failure tests must run
+against the candidate core and examples revisions; earlier SPI results do not
+satisfy this gate.