phronomy 0.16.0 → 0.18.0

data/docs/decisions/013-journal-backed-knowledge-as-context-candidates.md

@@ -0,0 +1,122 @@
+# ADR-013: Journal-backed Knowledge as Context Candidates
+
+## Status
+
+Accepted.
+
+## Context
+
+The Manifest-first Agent refactor established the Journal as the canonical
+append-only record of Agent state and `LLMInputManifest` as the authority for
+one provider call. The older Knowledge design remained outside that model:
+`StaticKnowledge` and other `KnowledgeSource` objects were fetched separately,
+then concatenated into the mandatory system prompt. That created several
+problems:
+
+- static/entity/RAG distinctions described acquisition strategy rather than
+  Context semantics;
+- class-level and instance-level Knowledge followed different storage paths;
+- persistent Knowledge did not share Agent persistence/reload semantics;
+- Knowledge bypassed Context Policy and was always mandatory once configured;
+- `static?`, `source`, fetch/caching APIs and EntityKnowledge behavior existed
+  primarily to support the obsolete source abstraction.
+
+Applications still need two different lifetimes:
+
+1. information that becomes durable Agent Knowledge at creation time or later;
+2. request-scoped information used for only one LLM call.
+
+## Decision
+
+Phronomy has one Knowledge Context category.
+
+### Persistent Knowledge
+
+Persistent Knowledge is stored as ordinary append-only Journal records with
+`kind: :knowledge`, `channel: :context`, `role: :user` and
+`context_candidate: true`. Content lives in ContentStore and the Journal holds
+its content reference and application metadata.
+
+Agent instances accept Knowledge at creation and after creation:
+
+```ruby
+agent = MyAgent.new(knowledge: ["Policy: ..."])
+agent.add_knowledge("Customer locale: ja-JP")
+```
+
+`Agent.load` requires no separate Knowledge source reconstruction because the
+records are already persisted with the Agent.
+
+### Selection
+
+Knowledge is not part of the public conversation transcript.
+`JournalProjection` exposes active Knowledge together with active transcript
+records for Context selection. `ContextCandidateResolver` and Context Policy
+therefore handle persistent Knowledge through the same selection pipeline as
+other optional Context.
+
+Knowledge is optional by default. The fact that content is Knowledge does not
+make it mandatory.
+
+Selected Knowledge is materialized before ordinary conversation-history
+segments so that persistent background Context is not interleaved into the
+middle of dialogue chronology.
+
+### Reset semantics
+
+`clear_knowledge!` appends a `knowledge_cleared` marker. Earlier Knowledge
+records remain in the Journal but are excluded from later Context projections.
+No Knowledge-generation counter is required.
+
+`clear_transcript!` affects conversation history only. `reset_context!` resets
+both transcript eligibility and Knowledge eligibility while retaining raw
+Journal records.
+
+### Per-call Context
+
+`before_llm_input` continues to accept `LLMInputPatch#segment_candidates`.
+Those candidates are not persisted. They enter the same Context Policy request
+as Journal-backed candidates and may be omitted when optional and over budget.
+
+### Acquisition responsibility
+
+Phronomy core does not model `StaticKnowledge`, `EntityKnowledge`,
+`RAGKnowledge` or `KnowledgeSource` subclasses. File loading, retrieval, entity
+extraction and other acquisition strategies belong to applications or Tools.
+Once an application chooses to retain the resulting information, it registers
+plain logical Knowledge with the Agent.
+
+## Consequences
+
+### Positive
+
+- one persistent representation and one Context-selection path;
+- creation-time and post-creation Knowledge behave identically;
+- durable Knowledge naturally survives Agent reload;
+- token-budget selection can omit optional Knowledge without rewriting state;
+- RAG/entity extraction can evolve independently of Agent Context persistence;
+- no class-level static cache or `static?` distinction is needed.
+
+### Tradeoffs
+
+- applications that previously declared class-level static Knowledge must pass
+  common Knowledge when constructing each Agent instance;
+- source-specific refresh behavior is no longer a framework abstraction;
+- provenance that matters to an application must be stored explicitly in
+  metadata rather than through a dedicated `source:` API.
+
+## Removed contracts
+
+This decision removes the active contracts for:
+
+- `Phronomy::KnowledgeSource`;
+- `Agent::Context::Knowledge::Base`;
+- `StaticKnowledge`;
+- `EntityKnowledge`;
+- `static_knowledge`, `static_knowledge_sources`, `static_knowledge_chunks`,
+  `static_knowledge_refresh!`;
+- `add_knowledge_source`, `instance_knowledge_chunks`;
+- `clear_memory!` / `memory_generation` as obsolete Agent Context concepts.
+
+ADR-005 is superseded by this decision. ADR-012 remains the authority for the
+Journal/Manifest separation and Context Policy model.

data/docs/features.md

@@ -0,0 +1,87 @@
+# Features and API stability
+
+API means Application Programming Interface in this document.
+
+Phronomy is pre-1.0. Minor releases may include breaking changes even to APIs
+labelled Stable; patch releases should remain non-breaking. Consult
+[`CHANGELOG.md`](../CHANGELOG.md) when upgrading.
+
+- **Stable** — API is considered complete and suitable for production use.
+- **Beta** — functionality is complete and tested, but signatures or behaviour may evolve in a minor release.
+- **Experimental** — functionality may change without notice and is not recommended as a long-term compatibility boundary.
+
+The `main` branch contains unreleased development work. Pin a released gem version
+for production deployments.
+
+## Core building blocks
+
+| Feature | Stability |
+|---|---|
+| **Workflow** — Stateful, branching workflows with `wait_state` and explicit events | Stable |
+| **Agent** — Stateful ReAct-style agents with stable `agent_id`, persistence-backed execution state, canonical history, guardrails, and conversation context | Stable |
+| **Before-Large-Language-Model (LLM) Input Hook** — Three-tier per-call LLM input customization via `before_llm_input` and `LLMInputPatch` | Stable |
+| **Context Management** — Journal + Context Policy + per-LLM-call Manifest with token-budget-aware selection and protocol-safe Tool Call / Tool message dependencies | Stable |
+| **Filters** — Input/output transformation and blocking via `Filter::Base` | Beta |
+| **`PromptInjectionFilter`** — Built-in pattern-based prompt-injection filter | Beta |
+| **Capability redaction/result-size controls** — `redact_params` and `max_result_size` | Beta |
+| **Output Parser** — JSON and Struct-mapped parsers for structured LLM responses | Stable |
+| **Tracing** — Pluggable span-based observability | Stable |
+| **Error Taxonomy** — Provider errors translated to Phronomy transport/authentication/rate-limit/context errors | Beta |
+
+## Knowledge and integration
+
+| Feature | Stability |
+|---|---|
+| **Knowledge** — Journal-backed persistent Agent context registered with `knowledge:` / `add_knowledge`, selected per LLM call by Context Policy | Beta |
+| **`VectorStore#size`** — Document count for InMemory, RedisSearch, and Pgvector backends | Beta |
+| **`VectorStore::AsyncBackend`** — Pluggable async VectorStore interface with pool-backed defaults and native-async override points | Beta |
+| **Model Context Protocol (MCP) Tool** — `Phronomy::Tools::Mcp` integration through the official `mcp` gem | Beta |
+| **Agent Tool** — `Phronomy::Tools::Agent.from_agent` exposes a child Agent as a Tool without occupying a worker while waiting | Beta |
+| **Vector Search Tool** — `Phronomy::Tools::VectorSearch` wraps VectorStore and Embeddings adapters | Beta |
+
+## Execution and reliability
+
+| Feature | Stability |
+|---|---|
+| **EventLoop** — Runtime-owned event-driven execution core shared by Agent, ToolInvocation, Workflow, and MultiAgent sessions | Beta |
+| **`invoke` / `invoke_async`** — Blocking and non-blocking Agent/Workflow entry points | Stable |
+| **Agent async events** — `invoke_async(..., on_event:)` and `stream_async(..., on_event:)`; streaming additionally emits `:token` | Beta |
+| **`stream` / `stream_async`** — Event callbacks execute on EventLoop and must return quickly | Beta |
+| **`stream_callback_error_policy`** — Terminal event callback error policy (`:report` / `:fail_task`) | Beta |
+| **`Task#map`** — Application-level Task result transformation and error propagation | Stable |
+| **CancellationToken** — Cooperative cancellation with explicit `cancel!`, lazy monotonic deadlines, and callback registration | Experimental |
+| **Tool `execution_mode`** — `:cooperative` for short EventLoop-safe work; `:offloaded` for synchronous work that must stay off EventLoop | Experimental |
+| **OffloadPool sizing** — `offload_pool_size` / `offload_queue_size`; named pools available for application-owned isolation | Beta |
+| **InvocationContext** — Explicit correlation/cancellation/deadline context for Agent and Workflow invocations | Beta |
+| **Metrics** — OffloadPool active/queue/abandoned metrics plus EventLoop queue/lag metrics | Beta |
+| **Runtime lifecycle** — Runtime-owned EventLoop and terminal `Runtime#shutdown` | Beta |
+
+## Agent and workflow patterns
+
+| Feature | Stability |
+|---|---|
+| **Workflow asynchronous pattern** — Start async work, return immediately, and continue through `Workflow#signal` | Beta |
+| **Multi-agent** — Agent-as-Tool and hub-and-spoke handoff routing | Beta |
+| **GeneratorVerifier** — Generator-Verifier loop with injectable prompts/parsers | Beta |
+| **`Phronomy::MultiAgent::Orchestrator`** — Parallel subagent dispatch, fan-out, and `subagent` DSL | Beta |
+| **`Phronomy::MultiAgent::TeamCoordinator`** — LLM coordinator with stateful worker Agents | Beta |
+| **SharedState** — Peer-agent shared-state coordination | Experimental |
+| **Human-in-the-loop approval** — Suspension and approval/resume of Tool requests | Beta |
+| **`tool_approval_policy`** — Application-defined allow/approve/reject policy | Beta |
+
+## Public API boundary
+
+The feature tables above describe the primary APIs intended for gem consumers.
+Source declarations marked `@api private`, including most EventLoop/FSMSession and
+OffloadPool internals, are implementation details and may change without the same
+compatibility guarantees.
+
+## Advanced and internal APIs
+
+| Feature | Stability |
+|---|---|
+| **`Phronomy::Diagnostics`** — Snapshot of EventLoop lag/queue state and OffloadPool activity | Experimental |
+| **`Phronomy::Testing::FakeClock`** — Test-only deterministic clock helper | Beta |
+
+For runtime ownership and the distinction between public lifecycle APIs and
+private execution machinery, see [Runtime and concurrency](runtime-and-concurrency.md).

data/docs/getting-started.md

@@ -0,0 +1,351 @@
+# Getting started
+
+This guide contains the setup and introductory examples that were previously
+embedded in the repository README. The README remains the project entry point;
+this document carries the longer operational examples.
+
+## Install
+
+Add Phronomy to your Gemfile:
+
+```ruby
+gem "phronomy"
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+Phronomy uses RubyLLM for Large Language Model (LLM) access. Configure provider credentials and the
+transport retry/timeout policy on RubyLLM itself:
+
+```ruby
+RubyLLM.configure do |c|
+  c.openai_api_key = ENV["OPENAI_API_KEY"]
+  # c.anthropic_api_key = ENV["ANTHROPIC_API_KEY"]
+
+  c.request_timeout = 120
+  c.max_retries = 3
+  c.retry_interval = 0.1
+  c.retry_backoff_factor = 2
+  c.retry_interval_randomness = 0.5
+end
+```
+
+Phronomy does not add a second LLM transport timeout/retry layer on top of the
+configured adapter.
+
+## Optional dependencies
+
+Install only the backend gems required by your application:
+
+| Gem | Required for |
+|---|---|
+| `pgvector` | `Phronomy::VectorStore::Pgvector` |
+| `redis` | `Phronomy::VectorStore::RedisSearch` |
+| `opentelemetry-api` | `Phronomy::Tracing::OpenTelemetryTracer` |
+
+## Define a Tool and Agent
+
+```ruby
+class WebSearch < Phronomy::Agent::Context::Capability::Base
+  description "Search the web"
+  param :query, type: :string, desc: "Search query"
+
+  def execute(query:)
+    "Mock search result for: #{query}"
+  end
+end
+
+class ResearchAgent < Phronomy::Agent::Base
+  agent_definition id: "research-agent", version: 1
+  model "gpt-4o"
+  instructions "You are a research assistant. Use tools to answer questions."
+  tools(WebSearch => nil)
+  max_iterations 5
+end
+
+result = ResearchAgent.new.invoke("Research Ruby AI frameworks")
+puts result[:output]
+```
+
+Every concrete stateful Agent definition declares a stable definition ID and
+version. The definition identity is checked when persisted Agent state is loaded.
+
+## Stateful Agent persistence
+
+Phronomy Agents own their conversation history and persistent Knowledge. The
+application does not need to pass the previous `messages` array back into every
+invocation.
+
+```ruby
+persistence = Phronomy::Persistence::InMemory.new
+
+agent = ResearchAgent.create(
+  agent_id: "research-session-42",
+  knowledge: ["Customer tier: enterprise"],
+  persistence: persistence
+)
+
+agent.invoke("My name is Alice.")
+agent.add_knowledge("Customer locale: ja-JP")
+result = agent.invoke("What is my name?")
+
+puts result[:output]
+```
+
+Load the same Agent again when the same Persistence backend is available:
+
+```ruby
+agent = ResearchAgent.load(
+  "research-session-42",
+  persistence: persistence
+)
+
+agent.invoke("Continue our previous discussion.")
+```
+
+The active transcript and Knowledge views can be advanced independently without
+deleting the append-only canonical Journal:
+
+```ruby
+agent.clear_transcript!
+agent.clear_knowledge!
+agent.reset_context!
+```
+
+`purge!` is different: it permanently removes the Agent and persisted execution
+history from the configured Persistence backend.
+
+## Sync and async Agent APIs
+
+Use synchronous APIs at an external/top-level application boundary and async
+APIs when the caller must remain non-blocking.
+
+```ruby
+result = agent.invoke("Hello")
+
+task = agent.invoke_async("Hello")
+result = task.wait_result
+```
+
+`Task#wait_result` is for an external caller. Do not block EventLoop waiting for
+a Task that can only complete through that same EventLoop.
+
+Streaming follows the same split:
+
+```ruby
+agent.stream("Explain the design") do |event|
+  puts event.payload if event.type == :token
+end
+```
+
+```ruby
+task = agent.stream_async(
+  "Explain the design",
+  on_event: ->(event) { puts event.payload if event.type == :token }
+)
+```
+
+Streaming callbacks execute on EventLoop and therefore should return quickly.
+
+## Human-in-the-loop approval
+
+A Tool requiring approval can suspend an Agent invocation. Resume it with the
+approval request identifier returned by the suspension result.
+
+At a top-level synchronous boundary:
+
+```ruby
+result = agent.invoke("Perform the requested protected action")
+
+if result[:suspended]
+  request = result[:approval_request]
+  result = agent.approve(
+    result[:execution_id],
+    approval_request_id: request.id,
+    approved: true
+  )
+end
+```
+
+From an EventLoop callback, use `approve_async` rather than blocking EventLoop.
+
+## Workflow basics
+
+A Workflow is state-machine-driven and can halt at an explicit wait state:
+
+```ruby
+class ReviewContext
+  include Phronomy::WorkflowContext
+  field :draft,    type: :replace
+  field :feedback, type: :replace
+  field :approved, type: :replace, default: false
+end
+
+write_draft  = ->(state) { state.merge(draft: "Draft content") }
+review_draft = ->(state) { state.merge(feedback: "Feedback on: #{state.draft}") }
+
+workflow = Phronomy::Workflow.define(ReviewContext) do
+  initial :write
+  state :write, action: write_draft
+  state :review, action: review_draft
+  wait_state :awaiting_approval
+  state :finalize, action: ->(s) { s.merge(approved: true) }
+
+  transition from: :write, to: :review
+  transition from: :review, to: :awaiting_approval
+  transition from: :awaiting_approval, on: :approve, to: :finalize
+  transition from: :awaiting_approval, on: :reject, to: :write
+  transition from: :finalize, to: :__finish__
+end
+
+state = workflow.invoke({draft: ""}, config: {thread_id: "doc-1"})
+final = workflow.send_event(state: state, event: :approve)
+puts final.approved
+```
+
+Workflow entry and transition actions are synchronous Run-to-Completion
+callbacks. If a Workflow needs an Agent or another asynchronous lifecycle, start
+it asynchronously, return the Workflow context immediately, and deliver its
+completion later with `Workflow#signal`.
+
+```ruby
+class AnswerContext
+  include Phronomy::WorkflowContext
+
+  field :question, type: :replace, default: ""
+  field :answer,   type: :replace, default: nil
+  field :thread_id, type: :replace, default: nil
+end
+
+class ResearchAgent < Phronomy::Agent::Base
+  agent_definition id: "research-agent", version: 1
+  model "gpt-4o-mini"
+  instructions "Research the question and return a concise answer."
+end
+
+my_agent = ResearchAgent.new
+workflow = nil
+
+workflow = Phronomy::Workflow.define(AnswerContext) do
+  initial :asking
+  state :asking
+  state :done
+
+  entry :asking, ->(ctx) {
+    thread_id = ctx.thread_id
+
+    my_agent.invoke_async(
+      ctx.question,
+      on_event: ->(event) {
+        next unless event.type == :done
+
+        workflow.signal(
+          thread_id: thread_id,
+          event: :answer_ready,
+          payload: {answer: event.payload[:output]}
+        )
+      }
+    )
+
+    ctx
+  }
+
+  transition(
+    from: :asking,
+    on: :answer_ready,
+    to: :done,
+    action: ->(ctx, event) { ctx.merge(answer: event.payload[:answer]) }
+  )
+end
+```
+
+Returning a `Phronomy::Task` from a Workflow entry/transition action is not an
+implicit await mechanism and is rejected.
+
+## Agent as Tool
+
+Expose a child Agent using `Phronomy::Tools::Agent.from_agent` rather than calling
+a synchronous child Agent from a Tool worker:
+
+```ruby
+ResearchTool = Phronomy::Tools::Agent.from_agent(
+  ResearchAgent,
+  tool_name: "research",
+  description: "Delegate research to the research Agent"
+)
+
+class OrchestratorAgent < Phronomy::Agent::Base
+  agent_definition id: "orchestrator-agent", version: 1
+  model "gpt-4o"
+  instructions "Use the research Tool when research is required."
+  tools(ResearchTool => nil)
+end
+```
+
+Agent-backed Tools return control to EventLoop while the child lifecycle is
+waiting. They do not occupy an OffloadPool worker merely to wait for a child
+Agent result.
+
+## Filters
+
+Filters can transform or reject values at Agent boundaries:
+
+```ruby
+class NoCreditCardFilter < Phronomy::Filter::Base
+  def call(value, **_context)
+    block!("Credit card numbers are not allowed") if value.match?(/\d{4}-\d{4}-\d{4}-\d{4}/)
+    value
+  end
+end
+
+agent.add_input_filter(NoCreditCardFilter.new)
+```
+
+Phronomy includes `PromptInjectionFilter` as a baseline pattern filter. It is not
+a complete security policy for untrusted input.
+
+## Persistent Knowledge and per-call context
+
+Register durable Knowledge on the Agent:
+
+```ruby
+agent.add_knowledge(
+  "Customer locale: ja-JP",
+  metadata: {"origin" => "customer_profile"}
+)
+```
+
+Request-scoped context can instead be supplied through `before_llm_input` using
+`LLMInputPatch#segment_candidates`; those candidates enter Context Policy for the
+specific call and are not persisted to the Journal.
+
+## Model Context Protocol (MCP)
+
+Phronomy targets MCP 1.x through the official `mcp` gem:
+
+```ruby
+search_tool = Phronomy::Tools::Mcp.from_server(
+  "stdio://./mcp-server",
+  tool_name: "web_search"
+)
+
+begin
+  # use search_tool
+ensure
+  search_tool.close
+end
+```
+
+See [MCP client](mcp-client.md) for schema, error, cancellation, and lifecycle
+contracts.
+
+## Next steps
+
+- [Features and API stability](features.md)
+- [Runtime and concurrency](runtime-and-concurrency.md)
+- [Architecture decisions](decisions/)
+- [Migration guides](migrations/)

data/docs/migrations/0.15.md

@@ -0,0 +1,35 @@
+# Migration from 0.15-era execution-policy APIs
+
+This guide preserves the execution-policy migration information that previously
+lived in the top-level README. It is historical upgrade guidance; new code should
+follow the current contracts documented in
+[Runtime and concurrency](../runtime-and-concurrency.md).
+
+| Removed Phronomy setting/API | Replacement |
+|---|---|
+| `retry_policy` | RubyLLM transport retry, or explicit application orchestration |
+| `invoke_timeout` | `InvocationContext#deadline` or `cancellation_token` when the caller needs a root deadline |
+| `config[:llm_timeout]` | `RubyLLM.configure { |c| c.request_timeout = ... }` |
+| Tool `retry_on` | Tool/client-specific retry with explicit idempotency guarantees |
+| `config[:tool_timeout]` | Tool/client-native timeout |
+| `max_parallel_tools` | No replacement; `parallel_tool_execution` remains an on/off mode |
+| `InvocationContext#provider_limits` | Configure the provider client directly |
+| `stream_queue_max_size` | No replacement; the shared EventLoop queue is unbounded by design. Monitor `Metrics.snapshot[:event_loop_queue_depth]` instead |
+
+## Timeout ownership
+
+Phronomy owns logical lifecycle deadlines and cooperative cancellation. Provider,
+HTTP, database, MCP, and other transport-level timeouts belong to the relevant
+client library.
+
+A caller-defined root deadline can be represented through `InvocationContext` or
+a cancellation token, but that does not replace operation-native socket/request/
+statement timeouts for a synchronous call already in progress.
+
+## Retry ownership
+
+Automatic replay of an entire Agent invocation is not a generic framework retry
+policy. Configure transport retry on RubyLLM or another adapter, and implement
+application orchestration explicitly when replay is semantically safe.
+
+See [`CHANGELOG.md`](../../CHANGELOG.md) for the release-by-release history.

data/docs/migrations/0.16.md

@@ -0,0 +1,43 @@
+# 0.16 cleanup migration
+
+This guide preserves the compatibility cleanup information that previously lived
+in the top-level README. It is intended for applications upgrading older Phronomy
+code; new applications should use the current documentation directly.
+
+| Removed API | Current contract |
+|---|---|
+| `context_overhead` | Manifest-first assembly budgets actual mandatory + selected content |
+| Tool `on_error :return_empty` | Use `:raise` or `:suppress` |
+| `dispatch_parallel(..., force_kill:)` / `fan_out(..., force_kill:)` | Cooperative cancellation; no force-kill switch |
+| `runtime_backend` | Removed. Phronomy has one control model: EventLoop/FSMSession for lifecycle coordination and OffloadPool for synchronous work that must stay off EventLoop |
+| `Runtime.instance = ...` | Runtime replacement is test/internal infrastructure, not a public setter |
+| `Runtime#spawn` / `TaskGroup` | Removed. Start framework async work through domain async APIs (`invoke_async`, Workflow events, ToolInvocation, FanOut) |
+| `tools ToolA, ToolB` | Use `tools(ToolA => nil, ToolB => nil)` |
+| `CancellationToken.new(deadline: Time...)` | Use `CancellationToken.timeout_after(seconds)` for a lazy token deadline, or a Runtime-backed cancellation scope when callback delivery on deadline is required |
+| `StaticKnowledge` / `EntityKnowledge` / `Knowledge::Base` / `Phronomy::KnowledgeSource` | Register plain persistent Knowledge with `knowledge:` or `add_knowledge` |
+| `static_knowledge*` class APIs | Persistent Knowledge belongs to Agent instances and is Journal-backed |
+| `clear_memory!` | Use `clear_knowledge!`; conversation history is controlled independently with `clear_transcript!` |
+
+The legacy `build_context` / `LlmContextWindow::Assembler` extension path is no
+longer an active API. Stateful Agent input is assembled through the canonical:
+
+```text
+Journal
+  → Context candidates
+  → Context Policy
+  → Large Language Model (LLM) Input Manifest
+  → runtime projection
+```
+
+## Concurrency model
+
+The active concurrency model is EventLoop/FSMSession for logical lifecycle
+coordination plus OffloadPool for synchronous work that cannot run on EventLoop.
+There is no configurable runtime backend and no generic framework task-spawn
+backend.
+
+For the current design, see
+[Runtime and concurrency](../runtime-and-concurrency.md) and
+[ADR-010](../decisions/010-cooperative-first-concurrency.md).
+
+See [`CHANGELOG.md`](../../CHANGELOG.md) for release details.