phronomy 0.19.0 → 0.20.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 34dcf04a611b47c058b1620f34b1663d926421d6db1d98603d282b6b2e49ff0a
-  data.tar.gz: 60bcf027dbca418974fdf4cd9bbb292a4a8a5e934800db08cee6845b313b2337
+  metadata.gz: 8a9f9f2ab4acdb4e17edeb4e0b7f0b33660b7102e5b345c2bd8d411e6b0760de
+  data.tar.gz: 331265c52dfadbe4be8f85b9a48b35de384042664538f4fa74db0d34829e3b62
 SHA512:
-  metadata.gz: 63a976c4476cf47fe8c6e0d6bc2b3727eb439401f4502106091fd0e6d072ac36513604b1bcfc3ee4a961a29772ed218a21ceb0f75aaab47a7332775ca208f434
-  data.tar.gz: ff95b192d20e915b5bdc65b23244c68ff27ccc0d6c884e84335b2cc805aa086ac302349c33079207a1377c72eb2cce9fded3d0b1c13da4fd2482d69c166cbe40
+  metadata.gz: f3c7e0b15ede3129a2ac426ed86ebde36e353c664b5c64fc559878996ab6331b86f88c289e2f47179f0d1c58e91ec23c0c3d23060f066168c1a441e349dd534f
+  data.tar.gz: 527e858a300d3e8fb305782cdab29d6195b3836686df5b4d41a7abf8a40b584de8a8f0dcdab8ccd5b7b5dcfbcbf0c0dad45a7be2f092eae69c2c643ff69e9734

data/CONTRIBUTING.md

@@ -52,6 +52,36 @@ When adding, removing, or renaming a public method or class:
    bundle exec ruby scripts/api_snapshot.rb --write
    ```
 
+### `@api` classification vs Ruby visibility
+
+Phronomy's YARD `@api` annotation describes the compatibility boundary; it is
+not a synonym for Ruby's `public` / `protected` / `private` keywords.
+
+- `@api public` means consumers or extension implementers may rely on the
+  documented contract. Ruby visibility still follows the intended calling
+  model: ordinary methods may be public, subclass extension helpers may be
+  protected, and `initialize` remains Ruby-private while construction is
+  exposed through `.new`.
+- `@api private` means the method is internal and carries no public compatibility
+  promise. It may still be Ruby-public when Phronomy components need to call it
+  through an explicit receiver.
+- Ruby visibility is therefore not inferred from the `@api` annotation in
+  either direction.
+
+Run the annotation coverage guard when changing documented methods:
+
+```bash
+ruby scripts/check_api_annotations.rb
+```
+
+Ruby-public compatibility for the primary Stable/Beta product surface is
+protected by `scripts/api_snapshot.rb` and
+`spec/phronomy/api_compatibility_spec.rb`. Extension contracts whose calling
+model is protected/private should be protected by focused specs for that
+contract rather than by a repository-wide visibility inference rule.
+
+Do not change Ruby visibility merely to make it match an `@api` annotation.
+
 ---
 
 ## Architecture Decision Records

data/docs/decisions/014-unified-persistence-durable-state.md

@@ -69,6 +69,11 @@ persistence.workflow_states.save(
 A stale expected revision raises `Phronomy::Persistence::ConflictError`. Upper
 layers do not automatically reload and merge after such a conflict.
 
+The normative contract for custom durable backends, including repository
+semantics, transaction requirements, capabilities, durable codecs, and the Agent
+watermark precondition, is documented in
+[`docs/persistence-backends.md`](../persistence-backends.md).
+
 ### Activation is transient Runtime state
 
 `AgentExecutionActivation` is not part of the Persistence contract. Live

data/docs/features.md

@@ -19,7 +19,7 @@ for production deployments.
 |---|---|
 | **Workflow** — Stateful, branching workflows with `wait_state` and explicit events | Stable |
 | **Agent** — Stateful ReAct-style agents with stable `agent_id`, persistence-backed execution state, canonical history, guardrails, and conversation context | Stable |
-| **Unified Persistence** — One durable backend abstraction for Agent state and Workflow `workflow_states`; live Agent/Workflow state remains owned by the active instance/session between durable commits | Beta |
+| **Unified Persistence** — One durable backend abstraction for Agent state and Workflow `workflow_states`; live Agent/Workflow state remains owned by the active instance/session between durable commits; custom backends implement the documented Backend SPI and repository/transaction semantics | Beta |
 | **Before-Large-Language-Model (LLM) Input Hook** — Three-tier per-call LLM input customization via `before_llm_input` and `LLMInputPatch` | Stable |
 | **Context Management** — Journal + Context Policy + per-LLM-call Manifest with token-budget-aware selection and protocol-safe Tool Call / Tool message dependencies | Stable |
 | **Filters** — Input/output transformation and blocking via `Filter::Base` | Beta |
@@ -78,6 +78,21 @@ Source declarations marked `@api private`, including most EventLoop/FSMSession a
 OffloadPool internals, are implementation details and may change without the same
 compatibility guarantees.
 
+The YARD `@api` classification is independent from Ruby language visibility in
+both directions. `@api public` marks a compatibility contract, but the Ruby
+visibility still follows the intended calling model: ordinary APIs may be
+public, subclass extension helpers may be protected, and constructors use
+Ruby-private `initialize` behind `.new`. `@api private` means "internal/no
+compatibility promise" and does not require a Ruby `private` declaration; some
+internal methods remain Ruby-public because Phronomy components call them
+through explicit receivers.
+
+Persistence Backend SPI methods are a deliberate exception to the ordinary
+application-facing interpretation of `@api public`: they are public extension
+contracts for backend implementers, but application business logic should usually
+interact with Agents/Workflows instead of calling low-level repository operations.
+See [Persistence backend contract](persistence-backends.md).
+
 `Phronomy::StateStore` is no longer a public backend abstraction. Workflow
 durability is provided through `Phronomy::Persistence#workflow_states`; see the
 0.19 migration guide when upgrading code that used `state_store:`.
@@ -88,6 +103,11 @@ durability is provided through `Phronomy::Persistence#workflow_states`; see the
 |---|---|
 | **`Phronomy::Diagnostics`** — Snapshot of EventLoop lag/queue state and OffloadPool activity | Experimental |
 | **`Phronomy::Testing::FakeClock`** — Test-only deterministic clock helper | Beta |
+| **`Phronomy::Testing::PersistenceContract`** — Explicitly loaded RSpec conformance suite for custom Persistence backends | Beta |
+
+`Phronomy::Testing::PersistenceContract` is available only after explicit
+`require "phronomy/testing/persistence_contract"`. Ordinary
+`require "phronomy"` and production eager-load do not load RSpec.
 
 For runtime ownership and the distinction between public lifecycle APIs and
 private execution machinery, see [Runtime and concurrency](runtime-and-concurrency.md).

data/docs/migrations/0.19.md

@@ -79,12 +79,18 @@ persistence.workflow_states.save(
 A stale revision must raise `Phronomy::Persistence::ConflictError` rather than
 silently overwriting newer durable state.
 
-Custom durable backends must also implement the internal
-`Persistence#assert_agent_watermark!` precondition used before next-LLM durable
-barriers. It verifies the caller's Agent revision and Journal position without
-returning replacement mutable state; mismatches raise `ConflictError`. SQL
-backends should implement this check inside the same database transaction as the
-subsequent durable write.
+Custom durable backends must also implement the public Backend SPI operation
+`Persistence#assert_agent_watermark!` used before next-LLM durable barriers. It
+verifies the caller's Agent revision and Journal position without returning
+replacement mutable state; mismatches raise `ConflictError`. SQL backends should
+implement this check inside the same database transaction as the subsequent
+durable write.
+
+The complete custom-backend contract is documented in
+[`docs/persistence-backends.md`](../persistence-backends.md). In particular,
+custom backends must provide the required capabilities, transaction atomicity,
+repository compare-and-swap semantics, and durable domain-object reconstruction
+described there.
 
 ## Agent ownership and approval resume
 

data/docs/persistence-backends.md

@@ -0,0 +1,504 @@
+# Persistence backend contract
+
+`Phronomy::Persistence` is the single durable-state backend abstraction used by
+stateful Agents and durable Workflows. This document is the normative contract
+for authors of custom Persistence backends.
+
+The Backend SPI is **Beta**. It may evolve in a minor pre-1.0 release, but a
+backend should not depend on Phronomy private APIs or Runtime internals.
+
+## Architecture boundary
+
+A backend implements durable storage only:
+
+```text
+Application
+    ↓
+Agent / Workflow
+    ↓
+Runtime / EventLoop / ExecutionCoordinator
+    ↓
+Phronomy::Persistence synchronous Backend SPI
+    ↓
+Database / durable storage
+```
+
+Persistence does not own live execution state. In particular, a backend must not
+persist or reconstruct the following as part of this SPI:
+
+- `AgentExecutionActivation`;
+- `AgentInvocation`;
+- `FSMSession`;
+- `Task` or callbacks;
+- EventLoop queue contents;
+- Runtime Workflow admission entries;
+- in-flight provider operations.
+
+Persistence operations are synchronous. Framework-owned blocking Persistence I/O
+is submitted to the Runtime OffloadPool by Phronomy; a backend must not post
+EventLoop events or introduce `load_async` / `save_async` variants into this
+contract.
+
+## Required root surface
+
+A Persistence backend exposes five durable repositories:
+
+```text
+contents
+agents
+journals
+executions
+workflow_states
+```
+
+and two root operations:
+
+```ruby
+persistence.transaction { |tx| ... }
+persistence.assert_agent_watermark!(
+  agent_id:,
+  agent_revision:,
+  journal_position:
+)
+```
+
+The object yielded by `transaction` is a transaction-scoped Persistence view. It
+must respond to all five repository accessors and
+`assert_agent_watermark!`. It may be the Persistence instance itself, but SQL
+backends may instead yield an object bound to a checked-out connection or
+transaction session.
+
+## Required capabilities
+
+Every backend must advertise:
+
+```ruby
+{
+  atomic_all: true,
+  atomic_admission: true,
+  optimistic_revision: true
+}
+```
+
+`Phronomy::Persistence::REQUIRED_CAPABILITIES` is the executable definition of
+this requirement.
+
+### `atomic_all`
+
+All durable repositories must be able to participate in one atomic transaction
+domain. A transaction may change `contents`, `agents`, `journals`, `executions`,
+and `workflow_states` and then either commit all changes or roll them all back.
+
+This requirement deliberately does not claim exactly-once semantics after an
+indeterminate database/network failure. If the underlying database cannot tell
+the caller whether a commit happened, the backend should surface the storage
+failure rather than pretending the outcome is known.
+
+### `atomic_admission`
+
+This capability refers to **Agent execution admission**, not Workflow distributed
+locking.
+
+For one Agent, `executions.create_active` must atomically guarantee both:
+
+```text
+execution_id is unique
+AND
+no active/suspended execution already exists for agent_id
+```
+
+A conflict with an existing active/suspended execution raises
+`Phronomy::AgentBusyError`.
+
+Workflow admission remains Runtime/process-local. Cross-process Workflow
+lease/fencing is an application/distributed-coordination concern and is not part
+of this Backend SPI.
+
+### `optimistic_revision`
+
+The backend must implement compare-and-swap semantics used by Agent roots,
+Agent executions, Journals, Workflow snapshots, and the durable Agent watermark.
+Stale writers must receive `Phronomy::Persistence::ConflictError`; they must not
+silently overwrite newer durable state.
+
+## Error contract
+
+Backends should translate backend-specific constraint errors into the following
+portable Phronomy errors when the meaning matches.
+
+### `Phronomy::Persistence::NotFoundError`
+
+A requested durable record does not exist.
+
+### `Phronomy::Persistence::ConflictError`
+
+A persistence precondition failed, including revision, Journal position,
+identity, duplicate-ID, or compare-and-swap conflicts.
+
+### `Phronomy::AgentBusyError`
+
+An Agent already has an active or suspended execution and another execution
+cannot be admitted.
+
+### `Phronomy::Persistence::SerializationError`
+
+The backend cannot encode a value into its supported durable representation.
+This is intended primarily for durable backends whose Workflow state domain is
+narrower than the InMemory backend's Ruby-object domain.
+
+### `Phronomy::Persistence::UnsupportedBackendError`
+
+The backend does not provide a required structural or capability contract.
+
+Database availability, connection loss, and other transport/storage failures
+must not be misreported as ordinary optimistic conflicts merely to fit this
+error taxonomy.
+
+## Contents repository
+
+The content repository should normally inherit from
+`Phronomy::ContentStore::Base`, which supplies text/JSON helpers and the canonical
+content-ID calculation.
+
+Required primitive surface:
+
+```ruby
+def put(bytes, canonicalization_version:)
+def fetch(content_id)
+def exist?(content_id)
+```
+
+Required semantics:
+
+- content is immutable and content-addressed;
+- writing identical bytes is idempotent and returns the same content ID;
+- `fetch` returns a binary `String` isolated from caller mutation;
+- a missing content ID raises `Persistence::NotFoundError`;
+- one content ID must never resolve to different bytes; a digest-integrity
+  violation raises `ContentStore::IntegrityError`.
+
+Do not redefine the `sha256:<digest>` identity scheme in a backend. Content
+references are durable data used by other Phronomy records.
+
+## Agents repository
+
+Required surface:
+
+```ruby
+def create(root)
+def load(agent_id)
+def save(agent_id, expected_revision:, root:)
+def delete(agent_id)
+```
+
+`create`:
+
+- rejects an empty Agent ID;
+- rejects a duplicate Agent ID with `ConflictError`;
+- returns the stored `AgentRoot`.
+
+`load`:
+
+- returns `Phronomy::Agent::AgentRoot`, not a raw database Hash;
+- raises `NotFoundError` when missing.
+
+`save` atomically checks:
+
+```text
+stored.agent_revision == expected_revision
+root.agent_id == requested agent_id
+root.agent_revision == expected_revision + 1
+```
+
+Any failed precondition raises `ConflictError`.
+
+`delete` is idempotent.
+
+## Journals repository
+
+Required surface:
+
+```ruby
+def append(agent_id, expected_position:, records:)
+def read(agent_id, after: nil, limit: nil)
+def head(agent_id)
+def delete(agent_id)
+```
+
+`append` atomically checks:
+
+```text
+current Journal position == expected_position
+every record.agent_id == agent_id
+record_id is not already present in that Agent Journal
+record_id is not duplicated inside the incoming batch
+```
+
+Successful append assigns monotonically increasing sequences beginning at
+`expected_position + 1` and returns the sequence-bearing `JournalRecord` values.
+
+`read` returns records in ascending sequence order. `after: N` means records with
+`sequence > N`; `limit:` caps the returned count. Caller mutation of a returned
+collection must not mutate durable state.
+
+`head` returns the current Journal position, or `0` for an empty Journal.
+
+## Executions repository
+
+Required surface:
+
+```ruby
+def create_active(execution)
+def load(execution_id)
+def save(execution_id, expected_revision:, execution:)
+def list_active(agent_id)
+def delete(execution_id)
+def delete_for_agent(agent_id)
+def assert_idle!(agent_id)
+```
+
+`create_active` performs atomic Agent admission. A duplicate `execution_id`
+raises `ConflictError`; an already busy Agent raises `AgentBusyError`.
+
+`load` returns `Phronomy::Agent::AgentExecution`, not a raw database Hash, and
+raises `NotFoundError` when missing.
+
+`save` atomically checks:
+
+```text
+stored.execution_revision == expected_revision
+execution.execution_id == requested execution_id
+execution.execution_revision == expected_revision + 1
+```
+
+A failed precondition raises `ConflictError`.
+
+`list_active(agent_id)` returns the Agent's active/suspended executions.
+
+`assert_idle!` is used inside transactions before Agent context/Knowledge changes
+and destructive operations. It must raise `AgentBusyError` if an active/suspended
+execution exists. A SQL implementation must make this check part of a consistency
+boundary that cannot race with Agent execution admission; a best-effort SELECT
+outside the transaction is not sufficient.
+
+## Workflow states repository
+
+Required surface:
+
+```ruby
+def load(thread_id)
+def save(thread_id, expected_revision:, snapshot:)
+def delete(thread_id, expected_revision:)
+```
+
+`load` returns `nil` when no row exists. Otherwise it returns a Hash containing a
+snapshot and revision. String or Symbol Hash keys are accepted by Phronomy:
+
+```ruby
+{
+  snapshot: {
+    fields: { ... },
+    phase: "awaiting_approval"
+  },
+  revision: 3
+}
+```
+
+`save` is compare-and-swap:
+
+- missing row + `expected_revision: nil` creates revision `1`;
+- existing revision `N` + `expected_revision: N` creates revision `N + 1`;
+- any mismatch raises `ConflictError`.
+
+`delete` succeeds only at the supplied current revision; a mismatch raises
+`ConflictError`.
+
+Caller mutation of a loaded snapshot must not mutate durable storage.
+
+### Workflow value serialization
+
+`WorkflowContext#to_h` may contain ordinary Ruby application values. The
+InMemory backend can preserve a broader set of Ruby values than a JSON database.
+A durable backend is not required to serialize arbitrary Ruby objects such as
+`Proc`, IO objects, sockets, or runtime callbacks.
+
+A JSON/JSONB backend should document its supported value domain. A recommended
+domain is:
+
+```text
+nil
+String
+Integer / Float representable by the chosen JSON format
+true / false
+Array of supported values
+Hash with String/Symbol keys and supported values
+```
+
+If a value cannot be represented, raise `Persistence::SerializationError` rather
+than silently converting it into a lossy form. JSON backends may return String
+keys after decoding; `WorkflowRunner` deliberately accepts String and Symbol keys
+and normalizes them when comparing durable snapshots.
+
+Do not add generic Ruby object serialization to Phronomy core merely to make a
+particular database backend accept arbitrary Workflow values.
+
+## Durable Agent watermark
+
+`assert_agent_watermark!` is a public **Backend SPI** operation. It is not an
+ordinary application API.
+
+Phronomy uses it at durable barriers because a hydrated live Agent owns the
+current logical state and Phronomy deliberately does not reload mutable Agent
+state before every LLM/Tool cycle.
+
+The backend must verify, in one storage consistency view:
+
+```text
+stored AgentRoot.agent_revision == agent_revision
+current Journal position         == journal_position
+```
+
+If the Agent is missing, raise `NotFoundError`. If either watermark component
+differs, raise `ConflictError`. On success return `true`.
+
+The operation must not return a replacement AgentRoot or Journal. A mismatch is a
+conflict, not a request to reload/merge mutable state.
+
+When used inside `Persistence#transaction`, a SQL backend should perform the
+watermark check in the same database transaction as the subsequent durable
+write.
+
+## Transaction contract
+
+A transaction block may combine operations across all repositories:
+
+```ruby
+persistence.transaction do |tx|
+  tx.assert_agent_watermark!(...)
+  content_ref = tx.contents.put_text("...")
+  tx.journals.append(...)
+  tx.executions.save(...)
+  tx.agents.save(...)
+end
+```
+
+If an exception escapes the block, mutations performed through the transaction
+view must be rolled back as one unit.
+
+A backend must not satisfy the SPI by committing Agent, Journal, Execution, or
+Workflow changes in independent transactions and relying on later compensation.
+
+## Durable domain codecs
+
+Repositories return Phronomy domain objects. A database backend should not copy
+constructor knowledge for those objects into adapter code.
+
+The supported canonical Hash codecs are:
+
+```ruby
+Phronomy::Agent::AgentRoot#to_h
+Phronomy::Agent::AgentRoot.from_h(hash)
+
+Phronomy::Agent::JournalRecord#to_h
+Phronomy::Agent::JournalRecord.from_h(hash)
+
+Phronomy::Agent::LLMCallRecord#to_h
+Phronomy::Agent::LLMCallRecord.from_h(hash)
+
+Phronomy::Agent::AgentExecution#to_h
+Phronomy::Agent::AgentExecution.from_h(hash)
+```
+
+`AgentExecution.from_h` recursively restores nested `working_records` as
+`JournalRecord` objects and nested `llm_calls` as `LLMCallRecord` objects.
+String and Symbol top-level keys are accepted by these new execution/call codecs,
+which permits adapters to use parsed JSON without reimplementing constructors.
+
+The canonical Hash representation is the Phronomy/domain boundary. A backend is
+free to map that representation to normalized SQL columns, JSON, or another
+storage format internally.
+
+## Conformance tests
+
+Phronomy ships its backend-independent RSpec shared examples as explicit test
+support in the released gem. Backend projects opt in with:
+
+```ruby
+require "phronomy/testing/persistence_contract"
+```
+
+RSpec remains a **development/test dependency of the backend project**, not a
+Phronomy runtime dependency. Ordinary `require "phronomy"` does not load RSpec,
+and Phronomy's production Zeitwerk eager-load explicitly excludes the contract
+support paths.
+
+The entry point registers these shared examples:
+
+```text
+a persistence content store
+an Agent repository
+a Journal repository
+an Execution repository
+a workflow state repository
+a Persistence backend
+```
+
+A backend's RSpec suite can apply the complete contract as follows:
+
+```ruby
+require "phronomy"
+require "phronomy/testing/persistence_contract"
+
+RSpec.describe MyPersistenceBackend do
+  let(:persistence) { described_class.new(...) }
+
+  it_behaves_like "a persistence content store"
+  it_behaves_like "an Agent repository"
+  it_behaves_like "a Journal repository"
+  it_behaves_like "an Execution repository"
+  it_behaves_like "a workflow state repository"
+  it_behaves_like "a Persistence backend"
+end
+```
+
+`Persistence::InMemory` is run through the same shipped contract source in
+Phronomy CI. The files under `spec/support/shared_examples/` are compatibility
+require wrappers only; the authoritative shared-example implementations live
+under `lib/phronomy/testing/persistence_contract/` so the core suite and external
+backends cannot drift through copied definitions.
+
+The generic suite verifies repository behavior, CAS semantics, admission,
+mutation isolation, and whole-backend transaction behavior. Database-specific
+concurrency/locking mechanisms remain backend integration-test concerns; the SPI
+specifies outcomes rather than a particular SQL locking strategy.
+
+## SQL implementation guidance
+
+The SPI specifies outcomes, not a locking mechanism. Typical SQL implementations
+may use combinations of:
+
+- unique constraints;
+- conditional `UPDATE ... WHERE revision = ?`;
+- row locks;
+- serializable/repeatable-read isolation where appropriate;
+- partial unique indexes for active Agent execution admission;
+- transaction-scoped checks for Agent revision + Journal head.
+
+Backend-specific database exceptions should be translated to the Phronomy error
+contract where their meaning is known.
+
+## Explicit non-goals
+
+This Backend SPI does not provide:
+
+- durable reconstruction of a lost Agent Activation;
+- serialization of Runtime objects;
+- cross-process Workflow execution exclusion;
+- exactly-once external Tool side effects;
+- automatic conflict reload/merge;
+- a generic serializer registry for arbitrary Workflow field classes;
+- an async Persistence API.
+
+For the architectural reasons behind these boundaries, see
+[ADR-014: Unified Persistence for Durable State](decisions/014-unified-persistence-durable-state.md)
+and [Runtime and concurrency](runtime-and-concurrency.md).

data/lib/phronomy/agent/agent_execution.rb

@@ -84,6 +84,11 @@ module Phronomy
         self.class.new(**values)
       end
 
+      # Returns the canonical durable representation of this execution.
+      # Nested JournalRecord and LLMCallRecord values are recursively encoded.
+      #
+      # @return [Hash{String => Object}]
+      # @api public
       def to_h
         ATTRIBUTES.to_h do |name|
           value = public_send(name)
@@ -92,6 +97,30 @@ module Phronomy
           [name.to_s, value]
         end
       end
+
+      # Restores an execution from its canonical durable representation.
+      # String and Symbol top-level keys are accepted. Nested working Journal
+      # records and LLM Call records are restored through their public codecs so
+      # storage backends do not need to know their constructor details.
+      #
+      # @param hash [Hash]
+      # @return [AgentExecution]
+      # @api public
+      def self.from_h(hash)
+        attributes = ATTRIBUTES.to_h do |name|
+          key = hash.key?(name.to_s) ? name.to_s : name
+          [name, hash.fetch(key)]
+        end
+
+        attributes[:working_records] = attributes.fetch(:working_records).map do |record|
+          record.is_a?(JournalRecord) ? record : JournalRecord.from_h(record)
+        end
+        attributes[:llm_calls] = attributes.fetch(:llm_calls).map do |call|
+          call.is_a?(LLMCallRecord) ? call : LLMCallRecord.from_h(call)
+        end
+
+        new(**attributes)
+      end
     end
   end
 end