phronomy 0.22.0 → 0.24.0

data/docs/decisions/022-agent-execution-parent-identity-and-runtime-routing-boundary.md

@@ -0,0 +1,193 @@
+# ADR 022: Agent Execution Parent Identity and Runtime Routing Boundary
+
+**Status**: Accepted
+**Date**: 2026-08-23
+**Related**:
+- [ADR-014](014-unified-persistence-durable-state.md)
+- [ADR-021](021-generic-agent-invocation-identity-removal.md)
+- [ADR-023](023-fsm-session-incarnation-identity-and-routing.md)
+- [ADR-024](024-event-loop-single-writer-agent-runtime.md)
+
+---
+
+## Context
+
+Phronomy has a canonical durable Agent execution identity, `execution_id`, but
+Tool and approval paths still expose or carry the older
+`agent_invocation_id`/`parent_agent_invocation_id` terminology.
+
+Those older values mix two different concerns:
+
+1. the logical parent Agent execution; and
+2. the concrete Runtime FSMSession target used to deliver an event.
+
+The two lifetimes differ. An Agent execution can survive a suspension while its
+concrete FSMSession incarnation is rebuilt. A Runtime delivery target therefore
+must not become the durable or application-facing parent identity.
+
+`AgentInvocation` is the live FSM context of an `AgentExecution`; it is not a
+separate logical/domain entity. `ToolInvocation` has its own Tool lifecycle
+identity and provider Tool Call identity, but its logical parent is the owning
+Agent execution.
+
+## Decision
+
+The canonical logical parent of Agent-owned Tool and approval work is
+`execution_id`.
+
+The current identity relationship is:
+
+```text
+AgentExecution
+  execution_id
+
+AgentInvocation
+  execution_id          # parent reference, not an independent identity
+
+ToolInvocation
+  execution_id          # logical parent
+  tool_invocation_id    # Tool lifecycle identity
+  tool_call_id          # Provider-originated Tool Call identity
+
+ToolApprovalRequest
+  approval request id
+  execution_id          # logical parent
+
+ApprovalEvaluationRequest
+  execution_id          # logical parent
+  tool_invocation_id
+  tool_call_id
+```
+
+Application-facing approval surfaces change as a clean break:
+
+```text
+ToolApprovalRequest#agent_invocation_id
+  -> ToolApprovalRequest#execution_id
+
+ToolApprovalRequest#to_h[:agent_invocation_id]
+  -> ToolApprovalRequest#to_h[:execution_id]
+
+ApprovalEvaluationRequest#agent_invocation_id
+  -> ApprovalEvaluationRequest#execution_id
+```
+
+No deprecated alias is retained. Agent invocation configuration also rejects the
+legacy `agent_invocation_id` key instead of allowing application code to control
+a Runtime routing identifier.
+
+## Runtime routing is separate
+
+CG-03a does not redesign Runtime routing.
+
+The implementation may temporarily retain private
+`parent_agent_invocation_id`-named storage as the existing parent FSMSession
+routing carrier until the Runtime foundation is reconciled. That temporary
+carrier:
+
+- is not the logical/domain parent;
+- is not application-facing identity;
+- is not a durable field;
+- must not be renamed to `execution_id` and then used as an EventLoop target.
+
+CG-03b, integrated with ACS-10/ACS-11, will give each concrete FSMSession its own
+fresh ID and replace domain-object routing reuse with session-local Runtime
+bindings. Rebuilt sessions receive rebuilt bindings. Stale completion must not be
+applied to a new session merely because the logical execution is the same.
+
+No long-lived `parent_fsm_session_id` field is introduced on `ToolInvocation`.
+
+## Durable approval compatibility
+
+Current approval suspension stores `ToolApprovalRequest#to_h` both as referenced
+ContentStore audit content and in `AgentExecution#approval_request`.
+
+New canonical approval request representations use `execution_id`.
+
+Existing durable `AgentExecution` hashes may contain an embedded approval request
+with `agent_invocation_id`. During `AgentExecution.from_h`, that legacy key is
+discarded and the current logical parent is derived from the enclosing
+`AgentExecution#execution_id`. The old `agent_invocation_id` value is not renamed
+or reinterpreted as an execution ID.
+
+Existing content-addressed approval audit bodies are not rewritten. Their bytes
+are historical execution evidence and may retain the legacy field. New writes use
+the current representation.
+
+This is a targeted compatibility rule. It does not establish a general durable
+schema/versioning policy, which remains separate persistence work.
+
+## Staging
+
+```text
+CG-03a
+  execution_id logical parent
+  public approval clean break
+  targeted embedded durable-read compatibility
+  status: reconciled by the CG-03a change
+
+CG-03b
+  fresh FSMSession identity
+  session-local Runtime routing binding
+  stale-session completion rejection
+  duplicate AgentInvocation/ToolInvocation session fields cleanup
+  status: identity/routing slice implemented by ADR-023; result/live-state
+          authority completed by ADR-024 / ACS-11
+```
+
+CG-03's Agent/Tool Runtime-foundation criteria are reconciled by the joint
+ADR-023 / ADR-024 implementation: concrete-session routing is separated from
+domain identity, and worker results return to EventLoop for current-state and
+semantic-ID validation before live apply. Workflow admission ownership remains
+separate ACS-13 work.
+
+## CG-03b routing foundation implementation
+
+The Runtime now binds Agent/Tool/Multi-Agent asynchronous completion through a
+session-local `FSMSession::EventSink`. Rebuilt sessions receive fresh sinks and
+IDs; old sinks are never rebound to a new `execution_id` incarnation.
+
+`AgentInvocation` therefore has no independent `id`/`session_id`.
+`ToolInvocation#id` remains the semantic Tool lifecycle identity and is no
+longer reused as an FSMSession ID. Workflow uses a private FSMSession-owned
+identity reservation only because its existing pre-load admission still uses the
+future FSMSession ID; opaque Workflow admission ownership is explicitly ACS-13.
+
+## Explicit non-goals
+
+This decision does not in CG-03a:
+
+- stop using `AgentInvocation#id` as an Agent FSMSession ID;
+- stop using `ToolInvocation#id` as a Tool FSMSession ID;
+- remove the temporary `parent_agent_invocation_id` Runtime routing carrier;
+- remove `AgentInvocation#session_id` or `ToolInvocation#session_id`;
+- add a long-lived `parent_fsm_session_id`;
+- remove `AgentExecutionActivation`;
+- implement EventLoop single-writer ownership;
+- implement recovery, rehydration, fencing, or cross-process ownership;
+- define general Persistence schema/version evolution.
+
+## Consequences
+
+### Positive
+
+- Agent, Tool, and approval logical-parent vocabulary uses an existing
+  purpose-specific domain identity.
+- Application policy/notification code no longer receives a Runtime-oriented
+  Agent invocation identity.
+- Provider `tool_call_id`, Phronomy `tool_invocation_id`, approval request ID, and
+  Agent `execution_id` remain distinct.
+- Old embedded suspended-execution data remains readable without inventing a
+  false identity mapping.
+- The Runtime foundation can later change FSMSession incarnation/routing
+  independently of durable and application-facing parent identity.
+
+### Trade-offs
+
+- This is a pre-1.0 breaking application API change.
+- Applications that persisted `ToolApprovalRequest#to_h` must read the new
+  `execution_id` key for newly produced requests.
+- Historical content-addressed approval audit bodies are intentionally not
+  rewritten and may still contain `agent_invocation_id`.
+- The Agent/Tool portion of CG-03 is complete only when ADR-023 routing and
+  ADR-024 EventLoop result authority are both present.

data/docs/decisions/023-fsm-session-incarnation-identity-and-routing.md

@@ -0,0 +1,139 @@
+# ADR-023: FSMSession Incarnation Identity and Runtime Routing Foundation
+
+**Status**: Accepted
+**Date**: 2026-08-23
+**Partially superseded by**: [ADR-026](026-workflow-runtime-admission-and-durable-terminal-barrier.md) for the transitional Workflow FSMSession-identity reservation/admission bridge
+**Related**:
+- [ADR-010](010-cooperative-first-concurrency.md)
+- [ADR-014](014-unified-persistence-durable-state.md)
+- [ADR-020](020-canonical-workflow-instance-identity.md)
+- [ADR-022](022-agent-execution-parent-identity-and-runtime-routing-boundary.md)
+- [ADR-024](024-event-loop-single-writer-agent-runtime.md)
+
+---
+
+## Context
+
+Phronomy's EventLoop routes events to concrete `FSMSession` instances. Before
+this decision, Agent, Tool, and Multi-Agent context objects generated IDs that
+were injected into `FSMSession`, and callbacks later reused those object IDs as
+EventLoop routing targets. Workflow likewise pre-generated a Runtime ID because
+its current same-process admission is acquired before durable hydration.
+
+This mixes domain/context identity with the identity of one concrete Runtime FSM
+incarnation and makes suspend/resume stale-result safety difficult to reason
+about.
+
+## Decision
+
+### 1. Concrete FSMSession identity is allocated by FSMSession infrastructure
+
+A normal concrete `FSMSession` generates and owns a fresh `FSMSession#id` when
+constructed. Agent, Tool, and Multi-Agent domain/context IDs are not injected as
+that identity.
+
+Across Runtime object/event boundaries the value is named `fsm_session_id`.
+Terminal management payloads use `fsm_session_id` rather than generic
+`session_id`.
+
+Workflow is a narrow transitional case. Its current admission must be acquired
+before durable state load, while the concrete FSMSession is constructed only
+after hydration. Until ACS-13 separates Workflow admission ownership from
+FSMSession routing identity, Workflow obtains a single-use identity reservation
+from `FSMSession.reserve_identity`; the concrete FSMSession later claims exactly
+that Runtime-owned reservation. Arbitrary raw `id:` injection is removed.
+
+This reservation is Runtime-only and is not a Workflow/domain identity.
+
+### 2. Async routing uses session-local event sinks
+
+A concrete FSMSession has a Runtime-only event sink bound exactly once to its
+`fsm_session_id`. Async work captures the sink belonging to the session that
+started it. A rebuilt session receives a different sink and ID.
+
+An old sink is never rebound or retargeted. If its session has terminated,
+`EventLoop#post_to_session` rejects the old target rather than translating it to
+a new session of the same logical execution.
+
+This foundation is applied to Agent LLM completion/stream chunks, Tool
+authorization/execution, Tool-to-parent notifications, callback-failure
+notification, and Multi-Agent fan-out completion/timeout/cancellation.
+
+### 3. Agent and Tool live/domain objects do not duplicate routing identity
+
+`AgentInvocation` is a live FSM context belonging to `execution_id`; it has no
+independent `id` and no duplicate Runtime `session_id`.
+
+`ToolInvocation#id` remains the semantic `tool_invocation_id`. It is not an
+FSMSession ID. ToolInvocation stores neither a duplicate session ID nor a
+long-lived parent FSMSession ID. Parent routing is supplied as a session-local
+sink when a Tool FSMSession is constructed.
+
+No `parent_fsm_session_id`, generic generation token, or replacement generic
+correlation identity is introduced.
+
+### 4. Provider completion is routed before live result application
+
+Provider completion callbacks post an immutable `LLMOperationResult` through the
+session-local sink. The result carries the Provider Call's semantic `llm_call_id`.
+The AgentInvocation EventLoop handler applies it only when the concrete session
+still owns the event and the `llm_call_id` still matches the current Provider
+Call.
+
+ADR-024 completes this result-authority rule by making EventLoop the single
+writer of Phronomy-managed live Agent execution state and by removing the former
+Activation shared-mutable state model.
+
+### 5. Runtime incarnation identity is not durable state
+
+`fsm_session_id`, identity reservations, event sinks, callbacks, Tasks, and
+other process-local Runtime values are not persisted as logical recovery state.
+Recovery creates fresh Runtime objects from confirmed durable semantic state.
+
+## Relationship to ACS-11 and ACS-13
+
+This decision is the ACS-10 identity/routing foundation and the implementation
+half of CG-03b. ADR-024/ACS-11 builds directly on it: Offload work now returns
+operation-specific results and EventLoop validates/applies those results against
+current Runtime state and purpose-specific semantic identity.
+
+Together, ADR-023 and ADR-024 close the Agent/Tool result-routing and live-state
+authority portion of CG-03 without introducing another generic identity.
+
+ACS-13 separately owns Workflow's opaque admission owner handle and the durable
+terminal-save barrier. This ADR does not pull those Workflow lifecycle changes
+forward. The current `owner_fsm_session_id` admission representation remains an
+explicit transitional mismatch until ACS-13.
+
+## Explicit non-goals
+
+This decision does not implement:
+
+- ACS-13 Workflow opaque admission owner or durable-save-before-terminal barrier;
+- restart-safe HITL/Workflow rehydration;
+- same-process Agent admission redesign;
+- cross-process ownership, leases, or fencing;
+- general Persistence schema/version evolution.
+
+Those items remain governed by their own later change sets. EventLoop single-
+writer Agent ownership itself is defined by ADR-024 rather than duplicated here.
+
+## Consequences
+
+### Positive
+
+- Agent/Tool/Multi-Agent domain IDs are no longer EventLoop routing IDs.
+- Rebuilt sessions receive fresh session-local routing sinks.
+- Old-session Provider completion is not applied to a newer session merely
+  because the logical `execution_id` is unchanged.
+- Provider results are additionally protected by current semantic `llm_call_id`.
+- Workflow preserves pre-load admission ordering without retaining arbitrary
+  caller-supplied FSMSession IDs.
+- ACS-13 admission-owner redesign remains cleanly separated.
+
+### Trade-offs
+
+- Workflow temporarily uses a private FSMSession identity reservation because
+  its current admission owner is still the future FSMSession ID.
+- Runtime result application now requires explicit state/semantic-ID validation
+  rather than relying on a shared mutable continuation container.

data/docs/decisions/024-event-loop-single-writer-agent-runtime.md

@@ -0,0 +1,188 @@
+# ADR-024: EventLoop Single-Writer Agent Runtime State
+
+**Status**: Accepted
+**Date**: 2026-08-24
+**Partially supersedes**: [ADR-014](014-unified-persistence-durable-state.md) for live Agent Runtime ownership and `AgentExecutionActivation`
+**Complements**: [ADR-010](010-cooperative-first-concurrency.md), [ADR-012](012-canonical-execution-log-and-context-policy.md), [ADR-018](018-durability-guarantees-and-failure-model.md), [ADR-022](022-agent-execution-parent-identity-and-runtime-routing-boundary.md), [ADR-023](023-fsm-session-incarnation-identity-and-routing.md)
+
+---
+
+## Context
+
+ADR-014 correctly separated durable Persistence from process-local Agent continuation state, but its implementation introduced `AgentExecutionActivation` and `ActivationRegistry` as a shared mutable Runtime container. The Activation was protected by a Mutex and was mutated from EventLoop, OffloadPool workers, and asynchronous completion callbacks.
+
+That model prevents one component from being the unambiguous live-state authority. In particular, Agent durable preparation and terminal commit paths could successfully persist a transition and then directly mutate live `AgentExecution`, `AgentRoot`, Journal projection, Provider-call state, or Runtime projection from an OffloadPool worker.
+
+CG-03b / ADR-023 established the prerequisite routing foundation: every concrete Agent/Tool/Multi-Agent FSMSession owns a fresh Runtime incarnation identity and asynchronous completion is routed through that session's local EventSink. ACS-11 closes the remaining state-ownership gap.
+
+## Decision
+
+### EventLoop is the single writer of Phronomy-managed live Agent execution state
+
+All Phronomy-managed live mutation associated with an active Agent execution is applied on the Runtime EventLoop thread.
+
+The Runtime no longer owns an `ActivationRegistry`, and `AgentExecutionActivation` is removed. EventLoop owns a process-local execution directory:
+
+```text
+execution_id
+  -> immutable AgentExecutionState value
+       agent
+       coordinator
+       current AgentExecution
+       current RuntimeProjection
+       base Manifest
+       current AgentInvocation
+       current owning fsm_session_id
+```
+
+The directory is the mutable authority. Individual entries are immutable values and are replaced atomically on EventLoop rather than mutated in place.
+
+External live-owner lookup does not expose this state. `Runtime#__agent_execution_owner` returns only a read-only process-local owner view sufficient to resolve the current Agent/coordinator and execution status.
+
+### AgentInvocation owns FSM-local mutable invocation facts
+
+Mutable facts that are meaningful only while one Agent FSMSession progresses belong to `AgentInvocation` and are mutated only from EventLoop-driven FSM handling. These include:
+
+- the active Provider Call provenance;
+- uncommitted Provider outcomes;
+- uncommitted Tool/runtime events;
+- application callback failure state;
+- Tool batch and approval-resume state.
+
+This is not a replacement Activation. `AgentInvocation` is the FSM context of one logical Agent execution and is not shared as a worker-side mutable authority.
+
+### OffloadPool receives operation-specific snapshots and returns operation-specific results
+
+Synchronous Persistence I/O and other long synchronous work remain off EventLoop. Each operation captures the value data it needs before submission. Hash, Array, and String command data is recursively copied/frozen at the Tool authorization worker boundary.
+
+A worker command may also carry an explicitly classified Application-owned behavior handle, such as an approval-policy callable. Such a callable is executable behavior, not Phronomy live-state authority. Phronomy does not place live Agent, Tool, ToolInvocation, FSMSession, Runtime, EventLoop, or other Phronomy-managed live domain objects into the callable's command/request data.
+
+Application-defined opaque objects embedded in Application-owned context/metadata are not given a complete general value-type protocol by ACS-11. They remain Application-owned and must be safe for the Application's chosen worker usage. General serialization/value-type enforcement for those opaque objects is deferred hardening.
+
+The Agent pipeline uses distinct operation shapes for at least:
+
+```text
+InitialPreparationCommand -> InitialPreparationResult
+FollowupPreparationCommand -> FollowupPreparationResult
+ResumeCommitCommand -> ResumeCommitResult
+TerminalCommitCommand -> TerminalOutcome
+```
+
+A worker may perform blocking Persistence I/O and operation-local calculation. It must not call live-state mutation hooks such as EventLoop execution replacement, Agent root replacement, Journal live-view append, or AgentInvocation runtime-fact acknowledgement.
+
+After worker completion, a lightweight callback posts the result to EventLoop. EventLoop validates authority and only then applies the committed result to live state.
+
+### Durable commit and live apply are distinct phases
+
+Persistence is the last committed durable representation and recovery source. Successful Persistence operations return the resulting immutable durable values to EventLoop; they do not make Persistence the live read authority.
+
+The normal path remains:
+
+```text
+EventLoop-owned live snapshot
+        -> OffloadPool durable operation
+        -> optimistic durable commit
+        -> operation result
+        -> EventLoop authority validation
+        -> EventLoop live apply
+```
+
+Mutable Agent root, execution, and Journal state are not reloaded from Persistence merely to obtain freshness. Existing revision and Agent watermark checks remain the conflict boundary.
+
+### Provider result authority uses FSMSession state and `llm_call_id`
+
+A Provider Call receives its semantic `llm_call_id` on EventLoop before transport begins. Provider completion and streaming chunks carry that ID back through the owning FSMSession EventSink.
+
+A result is applicable only when the Runtime still recognizes the owning FSMSession incarnation and the AgentInvocation still has the same active `llm_call_id` in the required FSM state.
+
+A result for an old Provider Call is consumed as stale and does not advance the FSM. A callback targeting an old FSMSession incarnation is rejected by the session-local routing boundary established by ADR-023.
+
+Phronomy does not introduce a generic generation token, generic invocation ID, or Offload operation identity as a second semantic authority.
+
+### Tool worker results use the same direction of ownership
+
+Tool authorization and execution continue to settle through explicit FSMSession events. Authorization worker input is captured on EventLoop before submission. Its value data contains Agent identity metadata and Tool description/operation data, not live Agent or Tool objects. Tool authorization behavior (`approval_facts`, `requires_approval`, and Agent approval policy when callable) is captured as explicitly classified Application-owned behavior handles.
+
+`ApprovalEvaluationRequest` is therefore a value-only policy input. It exposes `agent_id`, `agent_definition_id`, `agent_definition_version`, execution identity, Tool name/schema, arguments, facts, context, origin, metadata, and default decision; it does not expose live `agent` or `tool` references.
+
+Actual Tool execution is a separate behavior boundary: executing the configured Tool is the purpose of that operation. The authorization worker does not need the Tool instance and must not use one as live authorization input.
+
+Authorization/execution outcomes are immutable result carriers and are applied to `ToolInvocation` only by EventLoop-driven FSM handling. They carry `tool_invocation_id`; EventLoop-driven Tool FSM handling consumes an outcome as stale when that semantic ID does not match the current ToolInvocation. `tool_invocation_id` remains the semantic Tool-operation identity. FSMSession identity remains Runtime routing identity. The two are not conflated.
+
+### Approval suspension retains the same live owner without Activation
+
+Approval suspension retains the same process-local Agent and AgentInvocation. The suspended execution remains present in EventLoop's execution directory, but no active FSMSession owns it while suspended.
+
+`Agent::Base.live_for_execution(execution_id)` and `agent.approve_async(...)` resolve the process-local execution owner through EventLoop's read-only owner view. They do not load a replacement Agent/Execution from Persistence.
+
+A resume performs its durable approval transition off EventLoop and applies the result on EventLoop before constructing a fresh resume FSMSession incarnation.
+
+If no live owner exists, durable rehydration remains a separate capability and `ExecutionRehydrationRequiredError` is raised.
+
+### Application callbacks do not become a worker-side authority
+
+Application stream/event callbacks are invoked from EventLoop-owned AgentInvocation event handling. Callback failure is recorded independently of canonical runtime event capture, converted into an explicit FSM failure event, and cannot mutate durable or live execution state from an Offload worker.
+
+Approval notification callbacks may execute off EventLoop because they are application work. Their execution does not advance Phronomy-managed live state.
+
+## Required invariants
+
+The implementation must preserve all of the following:
+
+1. Journal / Manifest / ContentStore remain the canonical execution/context record authorities defined by ADR-012.
+2. Persistence remains the durable recovery authority, not the normal live refresh source.
+3. FSMSession incarnation identity and EventSink routing remain as defined by ADR-023.
+4. `execution_id` remains the logical Agent execution parent identity defined by ADR-022.
+5. Provider result authority is checked with current FSMSession/FSM state plus `llm_call_id`.
+6. Tool result authority remains tied to the current Tool FSMSession and `tool_invocation_id`.
+7. OffloadPool never waits synchronously for logical EventLoop progress.
+8. Worker completion callbacks do not fall back to direct live mutation when EventLoop is unavailable.
+9. Tool authorization command/request value data contains no Phronomy-managed live domain object; explicitly classified Application-owned behavior handles remain permitted.
+
+## Consequences
+
+### Positive
+
+- Agent live-state ownership is explicit and mechanically enforceable.
+- Mutex-protected shared Activation state disappears.
+- Worker-side durable I/O can scale independently without becoming a second live-state writer.
+- Late Provider results cannot overwrite the provenance of a newer Provider Call.
+- Approval lookup remains process-local without exposing mutable execution internals.
+- The design provides the state/result foundation required by later recovery, cancellation, and durable-barrier work.
+
+### Trade-offs
+
+- Durable commit and EventLoop live apply are separate phases, so code must explicitly model and validate result application.
+- Internal Agent execution coordination uses more typed command/result values than the Activation model.
+- Process loss still loses in-flight Runtime continuation; this decision does not implement rehydration.
+
+## Explicitly deferred work
+
+This decision does **not** implement:
+
+- ACS-12: same-process Agent admission/exclusion policy;
+- ACS-13: opaque Workflow admission ownership and Workflow terminal-save ordering;
+- ACS-14: cross-process leases/fencing;
+- ACS-15: durable Agent/FSMSession rehydration;
+- ACS-16: cancellation/HITL Task semantic completion;
+- ACS-17: semantic retry and causal durable barriers.
+
+Those changes build on this ownership/result model and must not be folded into ACS-11 implicitly.
+
+## Rejected alternatives
+
+### Rename Activation and keep the same shared mutable object
+
+Rejected. A renamed mutex-protected container shared by EventLoop and workers preserves the ownership defect.
+
+### Let workers update Agent live state after a successful commit
+
+Rejected. Successful durability does not grant a worker live mutation authority. The result must return to EventLoop for apply.
+
+### Reload Agent state from Persistence before every apply
+
+Rejected. This would make Persistence the live source of truth and reintroduce implicit refresh/merge semantics rejected by ADR-014.
+
+### Add a generic generation/correlation token
+
+Rejected. Result authority is expressed using the current FSM state plus existing purpose-specific semantic identifiers. A new generic token would recreate identity ambiguity already removed by ADR-021 through ADR-023.