phronomy 0.22.0 → 0.24.0

data/docs/architecture/knowledge-and-rag.md

@@ -0,0 +1,130 @@
+> **CURRENT explanatory architecture**
+>
+> This document describes the reconciled current Phronomy system. Normative
+> architecture decisions remain in the [ADR index](../decisions/README.md);
+> source/runtime behavior remains implementation reality and does not silently
+> amend an ADR.
+
+# Knowledge and RAG
+
+## 1. Boundary
+
+Phronomy separates information acquisition from Agent Context management.
+
+```text
+Application / Tool / retrieval pipeline
+        |
+        v
+plain logical Knowledge content
+        |
+        +-- persist: Agent Journal (`kind: :knowledge`)
+        |
+        +-- request-scoped: before_llm_input candidate
+        |
+        v
+ContextPolicyInput.knowledge
+        |
+        v
+ContextPolicy
+        |
+        v
+Manifest
+```
+
+Phronomy does not define a core hierarchy of `StaticKnowledge`,
+`EntityKnowledge`, `RAGKnowledge`, or `KnowledgeSource` objects. How information
+was obtained is not a Context-selection type.
+
+Persistent Knowledge authority is defined by
+[ADR-013](../decisions/013-journal-backed-knowledge-as-context-candidates.md).
+
+## 2. Persistent Knowledge
+
+Persistent Knowledge is registered on a live Agent:
+
+```ruby
+agent = ResearchAgent.new(
+  knowledge: [
+    "Account type: enterprise",
+    "Data residency: Japan"
+  ]
+)
+
+agent.add_knowledge(
+  "Customer locale: ja-JP",
+  metadata: {"origin" => "customer_profile"}
+)
+```
+
+Content is stored through the durable content/Journal path and remains available
+after Agent reload.
+
+`clear_knowledge!` logically invalidates prior Knowledge for future Context
+selection without deleting historical Journal records. `clear_transcript!` does
+not clear Knowledge. `reset_context!` resets both relevant eligibility domains
+without rewriting old Journal facts.
+
+## 3. Request-scoped Knowledge
+
+Information needed for only one LLM Call should not be persisted merely to make
+it available to Context Policy.
+
+`before_llm_input` may return an `LLMInputPatch` with a Knowledge
+`segment_candidate`. The candidate enters the same typed Policy path and is not
+automatically journaled.
+
+See [before_llm_input](before-llm-input.md).
+
+## 4. Retrieval paths
+
+RAG can enter Agent reasoning through more than one ordinary application path.
+
+### Tool-result retrieval
+
+A Tool may query a VectorStore/external search system and return results to the
+Agent. The Tool result participates in the normal Tool protocol and can influence
+later Context through normal Journal/Context rules.
+
+### Pre-Manifest retrieval
+
+Application code may retrieve/rank content before Context finalization and supply
+selected logical Knowledge through `before_llm_input` or an Application
+`ContextPolicy`.
+
+The core does not require one vector-retrieval lifecycle to be the universal RAG
+path.
+
+## 5. Vector/embedding responsibility
+
+Vector stores, loaders, splitters, embeddings, ranking, and external retrieval
+are acquisition/integration capabilities. They are not themselves durable Agent
+Knowledge state.
+
+An application may:
+
+1. retrieve content;
+2. rank/filter it;
+3. convert it to plain logical Knowledge;
+4. persist it with `add_knowledge`, or keep it request-scoped.
+
+## 6. Selection and budget
+
+Knowledge is optional by default. Context Policy decides whether it enters one
+LLM Call and may omit it under budget pressure.
+
+Information that is structurally required for the call must be represented
+through the corresponding required Context mechanism; registration as Knowledge
+does not make content mandatory merely by existence.
+
+## 7. Trust/security boundary
+
+Phronomy does not apply a universal semantic security Filter when Knowledge is
+added or retrieved. Source validation, retrieval policy, ingestion sanitization,
+and domain-specific trust decisions are Application/Tool responsibilities.
+
+At one-call Context selection time, Application `ContextPolicy` may inspect
+Knowledge provenance/metadata and decide whether to select, derive/sanitize, or
+fail the preparation. Framework validation still enforces structural authority
+and final Manifest invariants.
+
+See [Security Boundaries](security-boundaries.md).

data/docs/architecture/multi-agent-handoff.md

@@ -0,0 +1,147 @@
+> **CURRENT explanatory architecture**
+>
+> This document describes the reconciled current Phronomy system. Normative
+> architecture decisions remain in the [ADR index](../decisions/README.md);
+> source/runtime behavior remains implementation reality and does not silently
+> amend an ADR.
+
+# Semantic Multi-Agent Handoff
+
+## 1. Meaning
+
+Handoff transfers active responsibility from one live Source Agent to one live
+Target Agent together with sufficient policy-bounded Context for the Target to
+continue.
+
+It is a control-plane operation, not an ordinary Tool-result protocol.
+
+Handoff is distinct from Agent-as-Tool delegation. Delegation performs work and
+returns control to the caller. Handoff changes the active Agent for the current
+coordination lifetime.
+
+Normative Handoff intent is
+[ADR-030](../decisions/030-agent-handoff-domain-and-durable-responsibility.md).
+
+## 2. Public API
+
+```ruby
+handoff = Phronomy::Agent::Handoff.new(
+  source_agent: triage,
+  target_agent: billing,
+  description: "Transfer billing responsibility",
+  policy: policy
+)
+
+runner = Phronomy::Agent::HandoffRunner.new(
+  main_agent: triage,
+  handoffs: [handoff]
+)
+
+result = runner.invoke("My invoice is wrong")
+```
+
+The current public Runner facade is synchronous `#invoke`. Handoff architecture
+does not add async/stream APIs merely for symmetry.
+
+## 3. Private transport
+
+Outgoing Handoffs are represented to the Source LLM through generated Tool
+schemas. The generated Tool name is private transport encoding, not Handoff
+identity or public semantic contract.
+
+Phronomy intercepts the Provider Tool Call into a typed private
+`HandoffRequest`. It does not execute an ordinary `ToolInvocation` or emit a
+sentinel Tool result.
+
+## 4. Handoff Policy
+
+`HandoffPolicy` controls what material from the effective finalized Source
+Manifest may cross the Agent boundary.
+
+```text
+current_request
+history
+knowledge
+tool_exchanges
+```
+
+Each category is required, forbidden, or selectable. Handoff selection is bounded
+to the finalized Source Manifest.
+
+## 5. Context Policy relationship
+
+```text
+Source ContextPolicy
+  -> Source LLMInputManifest
+
+HandoffPolicy
+  -> what may cross Source -> Target
+
+immutable HandoffContext
+  -> request-scoped Target material
+
+Target ContextPolicy
+  -> what enters one Target LLM Call
+```
+
+Transfer does not automatically append content to Target Journal or persistent
+Knowledge.
+
+## 6. Conversation/Tool dependencies
+
+An assistant Tool Call and its corresponding Tool-role result message(s) remain
+an indivisible conversation group across the Handoff boundary.
+
+Framework-owned semantic/category/content-format metadata is carried only through
+the trusted typed Handoff boundary.
+
+## 7. Responsibility and provenance
+
+Transferred `responsibility` is the dynamic instruction for what the Target must
+continue. `description` is the static edge/capability description. No mandatory
+generic `handoff_reason` identity field is added.
+
+Handoff Context preserves origin Agent and, where available, Journal record,
+Agent execution, LLM call, and Tool call provenance. Multi-hop transfers extend
+the transfer path while retaining original provenance.
+
+## 8. Target state ownership
+
+Handoff does not give Source authority to mutate Target canonical state. Target
+state changes only through normal Target-owned execution/mutation paths.
+
+A Target Agent execution has its own `execution_id`; Source execution identity is
+provenance/audit context, not Target execution identity.
+
+## 9. Persisted responsibility and recovery
+
+The original main Agent ID anchors `HandoffState`; the active Agent is retained
+across compatible Runtime restarts. The graph must use one Persistence instance.
+Source `handed_off`, its journal/root transition, immutable HandoffContext and the
+reserved Target execution ID commit atomically. The Target is admitted under
+that ID only after an authoritative absence read and the usual Agent admission.
+An active exact Target uses Agent Recovery; a terminal Target result is reused.
+Target terminal settlement and routing stabilization share a transaction.
+
+A new Runtime supplies the graph/current definitions again. Tool transport names
+are deterministic from stable Source/Target Agent IDs and are resolved from that
+graph, without a global class registry. Persisted Context is not reprojected under
+current policy. Missing graph edges/definitions or a different Persistence domain
+fail before semantic continuation.
+
+`HandoffRunner#result(source_execution_id)` follows retained transfer links.
+`cancel(execution_id)` records a request against that exact turn; pending absent
+Targets are stopped without admission, active Targets use their existing Agent
+cancellation token, terminal outcomes remain immutable. No cancellation or
+observer loss rewinds active responsibility to the main Agent. Cancellation is
+not compensation for external effects; unresolved X0 still needs Agent Recovery.
+
+F1 commit response loss is resolved by exact readback. Read/decode failure is not
+absence. F4 recovery requires retained storage and compatible current wiring.
+`on_event` is Runtime-only; terminal results can be read without redelivery.
+
+Current graph objects and observers are not durably rehydrated; Application
+supplies compatible wiring. Source execution identity links each completed
+transfer to its exact reserved Target execution, independently of later turns.
+`Persistence#handoff_result(source_execution_id)` reads that turn without loading
+Agent owners, graph definitions, or listeners.

data/docs/architecture/persistence.md

@@ -0,0 +1,186 @@
+> **CURRENT explanatory architecture**
+>
+> This document describes the reconciled current Phronomy system. Normative
+> architecture decisions remain in the [ADR index](../decisions/README.md);
+> source/runtime behavior remains implementation reality and does not silently
+> amend an ADR.
+
+# Persistence, Durability, and Recovery
+
+## 1. Durable boundary
+
+`Phronomy::Persistence` is the single durable-state backend abstraction for
+stateful Agents and durable Workflows.
+
+| Repository | Durable authority |
+|---|---|
+| `contents` | Immutable content and manifests |
+| `agents` | AgentRoot |
+| `journals` | Append-only Agent journal |
+| `executions` | AgentExecution and owned child coordination |
+| `workflow_states` | Workflow snapshots |
+| `handoff_states` | Active responsibility and exact Target reservation |
+| `teams` | TeamRoot |
+| `team_executions` | Team tasks, assignments and outcomes |
+
+Persistence stores defined durable logical state. It is not serialization of the
+currently running Runtime object graph.
+
+Detailed custom-backend method/codec contracts belong in
+[Persistence backends](../persistence-backends.md).
+
+Normative durability vocabulary is
+[ADR-018](../decisions/018-durability-guarantees-and-failure-model.md).
+
+## 2. Durable versus Runtime-only state
+
+Durable examples include AgentRoot, Journal records, AgentExecution records,
+content/Manifest references, and durable Workflow snapshots.
+
+Runtime-only examples include FSMSession objects/IDs, AgentInvocation objects,
+Task instances/callbacks, EventLoop entries, Runtime admission entries, and
+in-flight Provider/Tool operation objects.
+
+Runtime/process loss removes Runtime-only objects but does not imply confirmed
+durable state was lost.
+
+## 3. Agent live ownership
+
+After create/load, one live Agent instance owns current mutable logical Agent
+state for that `agent_id` within one Runtime/process.
+
+During execution, EventLoop is the single writer of Phronomy-managed live Agent
+execution state.
+
+```text
+EventLoop authoritative snapshot
+  -> OffloadPool command
+  -> Persistence transaction
+  -> immutable operation-specific result
+  -> EventLoop authority validation
+  -> live apply
+```
+
+See
+[ADR-024](../decisions/024-event-loop-single-writer-agent-runtime.md).
+
+## 4. Durable transitions and conflicts
+
+Defined semantic durable transitions are atomic according to the Persistence
+transaction contract and conforming backend.
+
+Revision/watermark/CAS checks reject stale durable transitions with
+`Persistence::ConflictError` rather than silently merging/reloading competing
+state.
+
+Conflict detection is not competing-execution exclusion and cannot undo an
+external side effect already performed.
+
+## 5. Agent and Workflow admission
+
+Within one Runtime/process, Agent admission prevents prohibited competing
+top-level execution for one logical Agent owner.
+
+The canonical logical/durable Workflow identity is `workflow_instance_id`, which
+is separate from one Runtime `fsm_session_id`.
+
+Workflow same-process admission is acquired before durable hydration and retained
+through the authoritative terminal/halted save barrier. The caller-facing Task
+settles after that authoritative durable barrier.
+
+See
+[ADR-025](../decisions/025-process-local-agent-ownership-and-runtime-admission.md),
+[ADR-020](../decisions/020-canonical-workflow-instance-identity.md), and
+[ADR-026](../decisions/026-workflow-runtime-admission-and-durable-terminal-barrier.md).
+
+## 6. Cross-process guarantee
+
+Cross-process competing-execution exclusion is conditional.
+
+A multi-process deployment must provide stable routing/partitioning by logical
+identity or another coordination mechanism that establishes exclusive authority.
+Persistence CAS/revision checks alone detect stale commits; they do not prevent
+duplicate semantic execution from starting.
+
+Phronomy does not claim arbitrary external exactly-once side effects.
+
+## 7. Durable codec
+
+Durable backend exchange uses immutable
+`Phronomy::Persistence::DurableRecord` values with:
+
+```text
+record_type
+format_version
+payload
+```
+
+`format_version` is a durable representation version, not Agent definition
+version, Workflow identity, or Runtime incarnation identity.
+
+Payloads follow the canonical durable value/codec contract. A backend may use its
+own physical storage representation but must return an equivalent logical
+DurableRecord.
+
+## 8. Recovery model
+
+Recovery reconstructs logical state from confirmed durable evidence; it does not
+restore the old Runtime object graph.
+
+The shared Recovery vocabulary classifies unfinished work as:
+
+```text
+resumable
+reconcilable
+resolution_required
+```
+
+Where outcome is uncertain, Phronomy does not infer "not performed" merely from
+connection/process failure and does not blindly re-dispatch an external
+operation.
+
+Application resolution, when required, records one of:
+
+```text
+succeeded
+failed
+not_performed
+```
+
+before dependent continuation may proceed.
+
+Agent Recovery is integrated into supported `Agent.load` lifecycle semantics.
+
+## 9. Manifest authority during Recovery
+
+A finalized historical `LLMInputManifest` is reused for the Provider input it
+represents. Recovery does not rerun historical ContextPolicy code to reconstruct
+that finalized input.
+
+## 10. External-effect boundary
+
+External Provider/Tool/Application effects are outside the ordinary Phronomy
+Persistence transaction boundary (`X0` in ADR-018).
+
+Therefore:
+
+- durable-transition atomicity is not external-effect atomicity;
+- semantic IDs alone do not prevent duplicate external effects;
+- retry eligibility depends on outcome certainty plus operation-specific
+  idempotency/reconciliation contracts; and
+- arbitrary exactly-once external side-effect execution is not an unconditional
+  Phronomy guarantee.
+
+## 11. Reference backend details are not architecture
+
+InMemory implementation mechanisms do not become requirements for external
+backends unless the Backend SPI states the corresponding semantic property.
+
+## Durable semantic coordination extension
+
+ADR-031 extends the single transaction domain to `handoff_states`, `teams` and
+`team_executions`, alongside the existing five repositories. Current root and
+transaction views require all eight. Explicit identity/revision/admission metadata
+remains separate from opaque DurableRecord payloads. See the normative
+[backend contract](../persistence-backends.md) and
+[recovery clarifications](../design/durable-semantic-coordination/RECOVERY_CONTRACT_CLARIFICATIONS.md).

data/docs/architecture/removed/agent-context.md

@@ -0,0 +1,72 @@
+> **CURRENT negative architecture guidance**
+>
+> This document records removed Agent Context/Memory designs that must not be
+> mistaken for current contracts. Current architecture starts at
+> [Agent Context](../agent-context.md) and
+> [Context Management](../context-management.md).
+
+# Removed Agent Context and Memory Architecture
+
+## Mutable Provider chat as canonical state
+
+RubyLLM/runtime message collections are projections/materializations, not
+canonical Agent history. Canonical facts live in the append-only Journal and
+each Provider input is fixed by an `LLMInputManifest`.
+
+## Generic Memory subsystem
+
+Phronomy does not use `Memory::ConversationManager`, `Memory::WindowMemory`,
+`SummaryMemory`, or another mutable Memory object as Agent Context authority.
+
+Long-context selection/derivation belongs to Context Policy.
+
+## LlmContextWindow / ContextVersionCache
+
+The removed window assembler/cache model is not a current extension boundary.
+Context selection uses typed `ContextPolicyInput` and `ContextPlan`, with final
+canonicalization/budget validation by `ContextAssembler`.
+
+## Automatic history rewriting / Memory Compression
+
+Phronomy does not delete or rewrite canonical Journal history to fit a model
+window.
+
+Application Policy may omit optional source items for one call and create
+derived/compacted current-call content. Canonical source facts remain unchanged.
+
+There is no separate current Memory Compression subsystem and no automatic
+Default-Policy summarization LLM call.
+
+## Static / Entity / RAG Knowledge hierarchy
+
+Persistent Knowledge is Journal-backed logical Context, not a hierarchy of
+source object classes. Retrieval/vector/entity extraction are acquisition
+strategies owned by Application/Tool integrations.
+
+## Removed Context selection DSLs
+
+Not current public contracts:
+
+```text
+ContextRequest
+ContextPolicyDescriptor
+ContextPolicyRegistry
+DerivedContentSpec
+Selection::Unit
+UnitBuilder / Selector / TokenBudgetPacker
+ContextPlan#derived_contents
+ContextPlan#ordering_hints
+```
+
+## Generic Agent session/thread identity
+
+Agent Context is not keyed by a generic Application conversation/thread/session
+identity. Phronomy uses purpose-specific logical/runtime operation identities.
+
+## Cross-Agent shared history as Handoff
+
+Semantic Handoff does not merge Journal histories or automatically copy all
+Source history/Knowledge into Target durable state. It transfers policy-bounded
+immutable Context and Target ContextPolicy still decides one-call input.
+
+See [Multi-Agent Handoff](../multi-agent-handoff.md).

data/docs/architecture/security-boundaries.md

@@ -0,0 +1,173 @@
+> **CURRENT explanatory architecture**
+>
+> This document describes the reconciled current Phronomy system. Normative
+> architecture decisions remain in the [ADR index](../decisions/README.md);
+> source/runtime behavior remains implementation reality and does not silently
+> amend an ADR.
+
+# Security Boundaries
+
+## 1. Responsibility model
+
+Phronomy deliberately keeps content-policy interception, Context semantic trust,
+Tool authorization, and execution isolation as separate mechanisms.
+
+```text
+raw invocation input
+    -> input_filter
+
+Tool result
+    -> tool_result_filter
+
+possible LLM Context material
+    -> ContextPolicyInput
+    -> Application ContextPolicy
+         semantic selection / omission / derivation /
+         source-aware trust/security policy
+    -> ContextPlan
+    -> Framework structural validation
+    -> LLMInputManifest
+
+final Agent output
+    -> output_filter
+```
+
+No mechanism above is an all-purpose security layer.
+
+The current Filter contract is
+[ADR-019](../decisions/019-filter-contract-and-security-boundaries.md).
+
+## 2. Explicit Filter call sites
+
+`Phronomy::Filter::Base` may transform a value or block processing through
+`FilterBlockError`.
+
+Framework Filter call sites are explicit:
+
+```text
+input
+output
+Tool result
+```
+
+A Filter registered at one call site is not silently applied to every value that
+may later reach an LLM.
+
+In particular, Phronomy does **not** add a fourth `context_filter` /
+`add_context_filter` Framework call site.
+
+## 3. Context semantic trust belongs to Application Policy
+
+LLM Context is domain-dependent: content that is suspicious in one application
+may be exactly the content another Agent is supposed to analyze.
+
+The Framework therefore owns structural/integrity validation, while
+Application-defined `ContextPolicy` owns semantic trust decisions for one LLM
+Call.
+
+An Application Policy may:
+
+- inspect typed category/provenance/metadata;
+- omit optional material;
+- fail preparation when required material is unacceptable;
+- derive sanitized/redacted current-call material;
+- retrieve/rerank content; or
+- call a `Filter::Base` instance internally as ordinary Ruby policy logic.
+
+There is no separate Framework invocation guarantee when a Policy chooses to
+compose a Filter internally.
+
+## 4. Non-destructive transformation
+
+Context trust policy must not rewrite canonical Journal history.
+
+For optional content, transformation follows the current Context model:
+
+```text
+canonical/source item remains intact
+        |
+Policy omits it for this call
+        +
+Policy creates derived current-call item
+        |
+ContextPlan -> Manifest
+```
+
+Required material is protected by Framework invariants. A Policy cannot silently
+drop a required source item; if the required input cannot be accepted, Context
+preparation fails.
+
+## 5. Framework validation is not semantic filtering
+
+After Policy selection, Phronomy validates:
+
+- typed category/item identity;
+- required material;
+- Framework-owned metadata/provenance authority;
+- conversation and Tool-call/result dependency structure;
+- effective Tool wiring;
+- canonical representation; and
+- final token budget.
+
+These checks establish integrity, not truth, safety, trustworthiness, or domain
+appropriateness of arbitrary content.
+
+## 6. Knowledge/RAG boundary
+
+Phronomy does not add a mandatory semantic security hook to `add_knowledge`.
+Retrieval-source validation and ingestion policy remain Application/Tool
+responsibilities.
+
+Per-call use of Knowledge is decided by Context Policy.
+
+See [Knowledge and RAG](knowledge-and-rag.md).
+
+## 7. Tool approval is authorization, not sanitization
+
+Tool approval controls whether a side-effecting/capability operation may execute.
+It does not sanitize arbitrary Tool arguments/results, establish prompt-injection
+safety, or create an OS sandbox.
+
+Authorization and content policy are separate concerns.
+
+## 8. PromptInjectionFilter
+
+`Phronomy::Filter::PromptInjectionFilter` is an optional bounded heuristic for
+common patterns.
+
+It does not guarantee detection of all prompt injection, automatically inspect
+all Context sources, or turn LLM execution into a security sandbox.
+
+Applications remain responsible for domain-specific controls.
+
+## 9. Execution placement is not isolation
+
+EventLoop, OffloadPool, cooperative cancellation, and Tool `execution_mode`
+describe execution/lifecycle mechanics.
+
+They do not provide:
+
+- OS process isolation;
+- container isolation;
+- filesystem or network sandboxing;
+- privilege separation; or
+- containment of arbitrary malicious Application/Tool code.
+
+Those properties are Application/deployment responsibilities unless a future
+explicit isolation subsystem is accepted.
+
+## 10. No new security-policy SPI
+
+This architecture does not add:
+
+```text
+context_filter
+SecurityContext
+TrustLevel
+security metadata authority API
+sandbox SPI
+```
+
+Arbitrary item `metadata` remains Application data except for
+Framework-reserved control keys. Applications must not treat arbitrary metadata
+as a Framework-certified trust assertion.