phronomy 0.18.0 → 0.20.0

data/docs/persistence-backends.md

@@ -0,0 +1,504 @@
+# Persistence backend contract
+
+`Phronomy::Persistence` is the single durable-state backend abstraction used by
+stateful Agents and durable Workflows. This document is the normative contract
+for authors of custom Persistence backends.
+
+The Backend SPI is **Beta**. It may evolve in a minor pre-1.0 release, but a
+backend should not depend on Phronomy private APIs or Runtime internals.
+
+## Architecture boundary
+
+A backend implements durable storage only:
+
+```text
+Application
+    ↓
+Agent / Workflow
+    ↓
+Runtime / EventLoop / ExecutionCoordinator
+    ↓
+Phronomy::Persistence synchronous Backend SPI
+    ↓
+Database / durable storage
+```
+
+Persistence does not own live execution state. In particular, a backend must not
+persist or reconstruct the following as part of this SPI:
+
+- `AgentExecutionActivation`;
+- `AgentInvocation`;
+- `FSMSession`;
+- `Task` or callbacks;
+- EventLoop queue contents;
+- Runtime Workflow admission entries;
+- in-flight provider operations.
+
+Persistence operations are synchronous. Framework-owned blocking Persistence I/O
+is submitted to the Runtime OffloadPool by Phronomy; a backend must not post
+EventLoop events or introduce `load_async` / `save_async` variants into this
+contract.
+
+## Required root surface
+
+A Persistence backend exposes five durable repositories:
+
+```text
+contents
+agents
+journals
+executions
+workflow_states
+```
+
+and two root operations:
+
+```ruby
+persistence.transaction { |tx| ... }
+persistence.assert_agent_watermark!(
+  agent_id:,
+  agent_revision:,
+  journal_position:
+)
+```
+
+The object yielded by `transaction` is a transaction-scoped Persistence view. It
+must respond to all five repository accessors and
+`assert_agent_watermark!`. It may be the Persistence instance itself, but SQL
+backends may instead yield an object bound to a checked-out connection or
+transaction session.
+
+## Required capabilities
+
+Every backend must advertise:
+
+```ruby
+{
+  atomic_all: true,
+  atomic_admission: true,
+  optimistic_revision: true
+}
+```
+
+`Phronomy::Persistence::REQUIRED_CAPABILITIES` is the executable definition of
+this requirement.
+
+### `atomic_all`
+
+All durable repositories must be able to participate in one atomic transaction
+domain. A transaction may change `contents`, `agents`, `journals`, `executions`,
+and `workflow_states` and then either commit all changes or roll them all back.
+
+This requirement deliberately does not claim exactly-once semantics after an
+indeterminate database/network failure. If the underlying database cannot tell
+the caller whether a commit happened, the backend should surface the storage
+failure rather than pretending the outcome is known.
+
+### `atomic_admission`
+
+This capability refers to **Agent execution admission**, not Workflow distributed
+locking.
+
+For one Agent, `executions.create_active` must atomically guarantee both:
+
+```text
+execution_id is unique
+AND
+no active/suspended execution already exists for agent_id
+```
+
+A conflict with an existing active/suspended execution raises
+`Phronomy::AgentBusyError`.
+
+Workflow admission remains Runtime/process-local. Cross-process Workflow
+lease/fencing is an application/distributed-coordination concern and is not part
+of this Backend SPI.
+
+### `optimistic_revision`
+
+The backend must implement compare-and-swap semantics used by Agent roots,
+Agent executions, Journals, Workflow snapshots, and the durable Agent watermark.
+Stale writers must receive `Phronomy::Persistence::ConflictError`; they must not
+silently overwrite newer durable state.
+
+## Error contract
+
+Backends should translate backend-specific constraint errors into the following
+portable Phronomy errors when the meaning matches.
+
+### `Phronomy::Persistence::NotFoundError`
+
+A requested durable record does not exist.
+
+### `Phronomy::Persistence::ConflictError`
+
+A persistence precondition failed, including revision, Journal position,
+identity, duplicate-ID, or compare-and-swap conflicts.
+
+### `Phronomy::AgentBusyError`
+
+An Agent already has an active or suspended execution and another execution
+cannot be admitted.
+
+### `Phronomy::Persistence::SerializationError`
+
+The backend cannot encode a value into its supported durable representation.
+This is intended primarily for durable backends whose Workflow state domain is
+narrower than the InMemory backend's Ruby-object domain.
+
+### `Phronomy::Persistence::UnsupportedBackendError`
+
+The backend does not provide a required structural or capability contract.
+
+Database availability, connection loss, and other transport/storage failures
+must not be misreported as ordinary optimistic conflicts merely to fit this
+error taxonomy.
+
+## Contents repository
+
+The content repository should normally inherit from
+`Phronomy::ContentStore::Base`, which supplies text/JSON helpers and the canonical
+content-ID calculation.
+
+Required primitive surface:
+
+```ruby
+def put(bytes, canonicalization_version:)
+def fetch(content_id)
+def exist?(content_id)
+```
+
+Required semantics:
+
+- content is immutable and content-addressed;
+- writing identical bytes is idempotent and returns the same content ID;
+- `fetch` returns a binary `String` isolated from caller mutation;
+- a missing content ID raises `Persistence::NotFoundError`;
+- one content ID must never resolve to different bytes; a digest-integrity
+  violation raises `ContentStore::IntegrityError`.
+
+Do not redefine the `sha256:<digest>` identity scheme in a backend. Content
+references are durable data used by other Phronomy records.
+
+## Agents repository
+
+Required surface:
+
+```ruby
+def create(root)
+def load(agent_id)
+def save(agent_id, expected_revision:, root:)
+def delete(agent_id)
+```
+
+`create`:
+
+- rejects an empty Agent ID;
+- rejects a duplicate Agent ID with `ConflictError`;
+- returns the stored `AgentRoot`.
+
+`load`:
+
+- returns `Phronomy::Agent::AgentRoot`, not a raw database Hash;
+- raises `NotFoundError` when missing.
+
+`save` atomically checks:
+
+```text
+stored.agent_revision == expected_revision
+root.agent_id == requested agent_id
+root.agent_revision == expected_revision + 1
+```
+
+Any failed precondition raises `ConflictError`.
+
+`delete` is idempotent.
+
+## Journals repository
+
+Required surface:
+
+```ruby
+def append(agent_id, expected_position:, records:)
+def read(agent_id, after: nil, limit: nil)
+def head(agent_id)
+def delete(agent_id)
+```
+
+`append` atomically checks:
+
+```text
+current Journal position == expected_position
+every record.agent_id == agent_id
+record_id is not already present in that Agent Journal
+record_id is not duplicated inside the incoming batch
+```
+
+Successful append assigns monotonically increasing sequences beginning at
+`expected_position + 1` and returns the sequence-bearing `JournalRecord` values.
+
+`read` returns records in ascending sequence order. `after: N` means records with
+`sequence > N`; `limit:` caps the returned count. Caller mutation of a returned
+collection must not mutate durable state.
+
+`head` returns the current Journal position, or `0` for an empty Journal.
+
+## Executions repository
+
+Required surface:
+
+```ruby
+def create_active(execution)
+def load(execution_id)
+def save(execution_id, expected_revision:, execution:)
+def list_active(agent_id)
+def delete(execution_id)
+def delete_for_agent(agent_id)
+def assert_idle!(agent_id)
+```
+
+`create_active` performs atomic Agent admission. A duplicate `execution_id`
+raises `ConflictError`; an already busy Agent raises `AgentBusyError`.
+
+`load` returns `Phronomy::Agent::AgentExecution`, not a raw database Hash, and
+raises `NotFoundError` when missing.
+
+`save` atomically checks:
+
+```text
+stored.execution_revision == expected_revision
+execution.execution_id == requested execution_id
+execution.execution_revision == expected_revision + 1
+```
+
+A failed precondition raises `ConflictError`.
+
+`list_active(agent_id)` returns the Agent's active/suspended executions.
+
+`assert_idle!` is used inside transactions before Agent context/Knowledge changes
+and destructive operations. It must raise `AgentBusyError` if an active/suspended
+execution exists. A SQL implementation must make this check part of a consistency
+boundary that cannot race with Agent execution admission; a best-effort SELECT
+outside the transaction is not sufficient.
+
+## Workflow states repository
+
+Required surface:
+
+```ruby
+def load(thread_id)
+def save(thread_id, expected_revision:, snapshot:)
+def delete(thread_id, expected_revision:)
+```
+
+`load` returns `nil` when no row exists. Otherwise it returns a Hash containing a
+snapshot and revision. String or Symbol Hash keys are accepted by Phronomy:
+
+```ruby
+{
+  snapshot: {
+    fields: { ... },
+    phase: "awaiting_approval"
+  },
+  revision: 3
+}
+```
+
+`save` is compare-and-swap:
+
+- missing row + `expected_revision: nil` creates revision `1`;
+- existing revision `N` + `expected_revision: N` creates revision `N + 1`;
+- any mismatch raises `ConflictError`.
+
+`delete` succeeds only at the supplied current revision; a mismatch raises
+`ConflictError`.
+
+Caller mutation of a loaded snapshot must not mutate durable storage.
+
+### Workflow value serialization
+
+`WorkflowContext#to_h` may contain ordinary Ruby application values. The
+InMemory backend can preserve a broader set of Ruby values than a JSON database.
+A durable backend is not required to serialize arbitrary Ruby objects such as
+`Proc`, IO objects, sockets, or runtime callbacks.
+
+A JSON/JSONB backend should document its supported value domain. A recommended
+domain is:
+
+```text
+nil
+String
+Integer / Float representable by the chosen JSON format
+true / false
+Array of supported values
+Hash with String/Symbol keys and supported values
+```
+
+If a value cannot be represented, raise `Persistence::SerializationError` rather
+than silently converting it into a lossy form. JSON backends may return String
+keys after decoding; `WorkflowRunner` deliberately accepts String and Symbol keys
+and normalizes them when comparing durable snapshots.
+
+Do not add generic Ruby object serialization to Phronomy core merely to make a
+particular database backend accept arbitrary Workflow values.
+
+## Durable Agent watermark
+
+`assert_agent_watermark!` is a public **Backend SPI** operation. It is not an
+ordinary application API.
+
+Phronomy uses it at durable barriers because a hydrated live Agent owns the
+current logical state and Phronomy deliberately does not reload mutable Agent
+state before every LLM/Tool cycle.
+
+The backend must verify, in one storage consistency view:
+
+```text
+stored AgentRoot.agent_revision == agent_revision
+current Journal position         == journal_position
+```
+
+If the Agent is missing, raise `NotFoundError`. If either watermark component
+differs, raise `ConflictError`. On success return `true`.
+
+The operation must not return a replacement AgentRoot or Journal. A mismatch is a
+conflict, not a request to reload/merge mutable state.
+
+When used inside `Persistence#transaction`, a SQL backend should perform the
+watermark check in the same database transaction as the subsequent durable
+write.
+
+## Transaction contract
+
+A transaction block may combine operations across all repositories:
+
+```ruby
+persistence.transaction do |tx|
+  tx.assert_agent_watermark!(...)
+  content_ref = tx.contents.put_text("...")
+  tx.journals.append(...)
+  tx.executions.save(...)
+  tx.agents.save(...)
+end
+```
+
+If an exception escapes the block, mutations performed through the transaction
+view must be rolled back as one unit.
+
+A backend must not satisfy the SPI by committing Agent, Journal, Execution, or
+Workflow changes in independent transactions and relying on later compensation.
+
+## Durable domain codecs
+
+Repositories return Phronomy domain objects. A database backend should not copy
+constructor knowledge for those objects into adapter code.
+
+The supported canonical Hash codecs are:
+
+```ruby
+Phronomy::Agent::AgentRoot#to_h
+Phronomy::Agent::AgentRoot.from_h(hash)
+
+Phronomy::Agent::JournalRecord#to_h
+Phronomy::Agent::JournalRecord.from_h(hash)
+
+Phronomy::Agent::LLMCallRecord#to_h
+Phronomy::Agent::LLMCallRecord.from_h(hash)
+
+Phronomy::Agent::AgentExecution#to_h
+Phronomy::Agent::AgentExecution.from_h(hash)
+```
+
+`AgentExecution.from_h` recursively restores nested `working_records` as
+`JournalRecord` objects and nested `llm_calls` as `LLMCallRecord` objects.
+String and Symbol top-level keys are accepted by these new execution/call codecs,
+which permits adapters to use parsed JSON without reimplementing constructors.
+
+The canonical Hash representation is the Phronomy/domain boundary. A backend is
+free to map that representation to normalized SQL columns, JSON, or another
+storage format internally.
+
+## Conformance tests
+
+Phronomy ships its backend-independent RSpec shared examples as explicit test
+support in the released gem. Backend projects opt in with:
+
+```ruby
+require "phronomy/testing/persistence_contract"
+```
+
+RSpec remains a **development/test dependency of the backend project**, not a
+Phronomy runtime dependency. Ordinary `require "phronomy"` does not load RSpec,
+and Phronomy's production Zeitwerk eager-load explicitly excludes the contract
+support paths.
+
+The entry point registers these shared examples:
+
+```text
+a persistence content store
+an Agent repository
+a Journal repository
+an Execution repository
+a workflow state repository
+a Persistence backend
+```
+
+A backend's RSpec suite can apply the complete contract as follows:
+
+```ruby
+require "phronomy"
+require "phronomy/testing/persistence_contract"
+
+RSpec.describe MyPersistenceBackend do
+  let(:persistence) { described_class.new(...) }
+
+  it_behaves_like "a persistence content store"
+  it_behaves_like "an Agent repository"
+  it_behaves_like "a Journal repository"
+  it_behaves_like "an Execution repository"
+  it_behaves_like "a workflow state repository"
+  it_behaves_like "a Persistence backend"
+end
+```
+
+`Persistence::InMemory` is run through the same shipped contract source in
+Phronomy CI. The files under `spec/support/shared_examples/` are compatibility
+require wrappers only; the authoritative shared-example implementations live
+under `lib/phronomy/testing/persistence_contract/` so the core suite and external
+backends cannot drift through copied definitions.
+
+The generic suite verifies repository behavior, CAS semantics, admission,
+mutation isolation, and whole-backend transaction behavior. Database-specific
+concurrency/locking mechanisms remain backend integration-test concerns; the SPI
+specifies outcomes rather than a particular SQL locking strategy.
+
+## SQL implementation guidance
+
+The SPI specifies outcomes, not a locking mechanism. Typical SQL implementations
+may use combinations of:
+
+- unique constraints;
+- conditional `UPDATE ... WHERE revision = ?`;
+- row locks;
+- serializable/repeatable-read isolation where appropriate;
+- partial unique indexes for active Agent execution admission;
+- transaction-scoped checks for Agent revision + Journal head.
+
+Backend-specific database exceptions should be translated to the Phronomy error
+contract where their meaning is known.
+
+## Explicit non-goals
+
+This Backend SPI does not provide:
+
+- durable reconstruction of a lost Agent Activation;
+- serialization of Runtime objects;
+- cross-process Workflow execution exclusion;
+- exactly-once external Tool side effects;
+- automatic conflict reload/merge;
+- a generic serializer registry for arbitrary Workflow field classes;
+- an async Persistence API.
+
+For the architectural reasons behind these boundaries, see
+[ADR-014: Unified Persistence for Durable State](decisions/014-unified-persistence-durable-state.md)
+and [Runtime and concurrency](runtime-and-concurrency.md).

data/docs/runtime-and-concurrency.md

@@ -8,6 +8,8 @@ handle, not an execution backend. Synchronous work that must stay off EventLoop
 
 For the design rationale, see Architecture Decision Record (ADR)
 [ADR-010: EventLoop / FSMSession First Concurrency](decisions/010-cooperative-first-concurrency.md).
+Durable-state ownership is defined by
+[ADR-014: Unified Persistence and Durable-State Ownership](decisions/014-unified-persistence-durable-state.md).
 
 ## Runtime model
 
@@ -19,6 +21,7 @@ Runtime
 │     ├─ Workflow
 │     ├─ ToolInvocation
 │     └─ MultiAgent fan-out
+├─ process-local Agent ActivationRegistry
 ├─ OffloadPool (bounded operating-system Threads)
 │  ├─ blocking input/output (I/O)
 │  ├─ central-processing-unit (CPU)-bound synchronous work
@@ -32,6 +35,67 @@ Task = completion handle
 The framework does not allocate one operating-system Thread per logical Agent/Workflow/Tool
 lifecycle. Logical waits remain explicit states plus later EventLoop events.
 
+## Live state and durable state
+
+A live Agent or Workflow owns its current logical state. `Persistence` is the
+last committed durable representation and recovery source; it is not reloaded at
+every semantic boundary.
+
+For Agents, the live owner consists of the Agent instance plus its current
+`AgentRoot`, hydrated Journal view, and `AgentExecutionActivation`. Mutable
+Agent/Execution/Journal state is not automatically reloaded before every LLM or
+Tool step. Durable writes use optimistic revision/position guardrails; an
+external writer that advances the durable base causes `Persistence::ConflictError`
+rather than automatic reload or merge.
+
+For Workflows, the current `WorkflowContext` and FSMSession own the active
+logical state. A durable Workflow hydrates once at invocation/resume and saves
+at the halted/terminal boundary.
+
+Content-addressed `Persistence#contents` values are immutable. Fetching a known
+content reference is value materialization rather than mutable state refresh.
+
+## Workflow identities and durable admission
+
+Workflow execution keeps three identities separate:
+
+```text
+session_id
+  application session/correlation identity, for example a Rails session
+
+thread_id
+  durable Workflow identity and Persistence#workflow_states key
+
+fsm_session_id
+  one Runtime FSMSession execution identity; generated again for each
+  invoke/resume operation
+```
+
+The existing application `session_id` is tracing/caller metadata and is not used
+for durable Workflow ownership. EventLoop registers active FSMs by
+`fsm_session_id`; durable Workflow admission is a separate owner map:
+
+```text
+thread_id -> owner_fsm_session_id
+```
+
+The owner is acquired before `workflow_states.load(thread_id)` and remains held
+until the halted/terminal `workflow_states.save(...)` completes. Only the current
+owner may release the admission. `fsm_session_id` is Runtime-only metadata and is
+not stored in Workflow fields or durable snapshots.
+
+The admission map belongs to one Runtime and is process-local. It prevents two
+executions with the same durable `thread_id` from being admitted concurrently
+inside that Runtime, but it is not shared across Ruby processes, containers, or
+service replicas. Separate processes may therefore execute the same `thread_id`
+concurrently unless the application adds distributed coordination.
+
+`workflow_states` optimistic revisions detect stale terminal commits across those
+processes. They do not prevent duplicate execution from starting and cannot undo
+external side effects that both executions already performed before one save
+loses the revision race. CAS is stale/double-commit detection, not a distributed
+execution lock or duplicate-side-effect prevention mechanism.
+
 ## Tool execution modes
 
 Phronomy exposes two execution modes for capabilities:
@@ -72,6 +136,28 @@ parent FSMSession
 This distinction prevents worker-slot starvation when many logical lifecycles are
 waiting at the same time.
 
+## Persistence I/O boundary
+
+`Persistence` repositories expose synchronous operations. Framework lifecycle
+code must not perform potentially blocking durable reads/writes on EventLoop.
+Agent preparation/commit and Workflow hydrate/save operations are submitted to
+`OffloadPool`; completion continues through completion callbacks or explicit
+EventLoop events.
+
+A durable barrier may pause one logical lifecycle without blocking EventLoop.
+The next Agent provider call does not start until the corresponding Manifest and
+logical execution snapshot commit succeeds. A persistence failure or optimistic
+conflict fails that step rather than continuing with stale state.
+
+Approval wait is not a hydration boundary. The same live Agent instance,
+Activation, and AgentInvocation remain the owner and are resumed after approval.
+Approval itself remains an Agent-instance operation. An application that only has
+an `execution_id` first resolves the current process's owner with
+`Phronomy::Agent::Base.live_for_execution(execution_id)` or the expected concrete
+Agent class, then calls `agent.approve(...)` or `agent.approve_async(...)`.
+`live_for_execution` consults the Runtime-local ActivationRegistry and does not
+load a replacement Agent from Persistence.
+
 ## Sync versus async application APIs
 
 | Calling context | Recommended approach |
@@ -82,11 +168,12 @@ waiting at the same time.
 | EventLoop callback | Never block waiting for a Task that requires EventLoop progress |
 | Top-level streaming | `agent.stream(...)` |
 | Non-blocking streaming | `agent.stream_async(...)` |
-| Approval from EventLoop callback | `approve_async` |
+| Approval from EventLoop callback | Resolve with `live_for_execution`, call `agent.approve_async(...)`, and return immediately |
 
 Blocking synchronous APIs reject EventLoop re-entry with
 `Phronomy::EventLoopReentrancyError` when waiting would stall the same EventLoop
-needed for progress.
+needed for progress. `live_for_execution` itself only performs a Runtime-local
+registry lookup and does not wait for Task progress.
 
 ## Task
 
@@ -247,6 +334,10 @@ Use these to distinguish worker saturation from EventLoop backlog/latency.
 Runtime-owned EventLoop, then closes pools and timers according to the Runtime
 shutdown contract.
 
+Workflow durable admission participates in EventLoop idleness: a Workflow whose
+FSMSession has ended but whose durable save is still in flight remains owned until
+that save completes and owner-aware admission is released.
+
 `Phronomy.reset_runtime!` exists primarily for test isolation and performs a real
 Runtime shutdown before resetting configuration.
 

data/lib/phronomy/agent/agent_execution.rb

@@ -84,6 +84,11 @@ module Phronomy
         self.class.new(**values)
       end
 
+      # Returns the canonical durable representation of this execution.
+      # Nested JournalRecord and LLMCallRecord values are recursively encoded.
+      #
+      # @return [Hash{String => Object}]
+      # @api public
       def to_h
         ATTRIBUTES.to_h do |name|
           value = public_send(name)
@@ -92,6 +97,30 @@ module Phronomy
           [name.to_s, value]
         end
       end
+
+      # Restores an execution from its canonical durable representation.
+      # String and Symbol top-level keys are accepted. Nested working Journal
+      # records and LLM Call records are restored through their public codecs so
+      # storage backends do not need to know their constructor details.
+      #
+      # @param hash [Hash]
+      # @return [AgentExecution]
+      # @api public
+      def self.from_h(hash)
+        attributes = ATTRIBUTES.to_h do |name|
+          key = hash.key?(name.to_s) ? name.to_s : name
+          [name, hash.fetch(key)]
+        end
+
+        attributes[:working_records] = attributes.fetch(:working_records).map do |record|
+          record.is_a?(JournalRecord) ? record : JournalRecord.from_h(record)
+        end
+        attributes[:llm_calls] = attributes.fetch(:llm_calls).map do |call|
+          call.is_a?(LLMCallRecord) ? call : LLMCallRecord.from_h(call)
+        end
+
+        new(**attributes)
+      end
     end
   end
 end