phronomy 0.15.1 → 0.16.0

data/docs/decisions/012-canonical-execution-log-and-context-policy.md

@@ -0,0 +1,69 @@
+# 012 — Canonical Complete Execution Log and Context Policy
+
+## Status
+
+Accepted for the stateful Agent refactor. This decision supersedes the parts of ADR-011 that made the legacy `build_context`/Assembler path the long-term LLM-input authority.
+
+## Decision
+
+Phronomy records the logical execution facts it observes as an append-only Canonical Complete Execution Log. Context selection, pruning and compaction do not rewrite or delete those raw facts. They decide only which representation is materialized into one LLM Call Manifest.
+
+Journal and Manifest are separate authorities:
+
+- **Journal** is the authoritative record of logical execution facts that Phronomy observed.
+- **Manifest** is the authoritative record of the logical input fixed for one particular LLM Call.
+- Runtime Projection is derived from the Manifest and must not add semantic content that the Manifest did not select.
+
+The persistent identity axes have narrow responsibilities:
+
+- `agent_id` identifies the owning Agent.
+- `execution_id` identifies one AgentExecution. It is provenance, not a Context-selection atom.
+- `llm_call_id` identifies one runtime Provider LLM Call. It is allocated before transport starts and correlates that call's outcome. It is provenance, not a semantic-compaction boundary.
+- `tool_call_id` links an assistant message's Tool Call with the corresponding Tool execution/message.
+- Journal `sequence` is canonical chronology.
+
+No `message_group_id`, import-only source provenance ID, or synthetic imported `execution_id` / `llm_call_id` is introduced.
+
+### Message preservation
+
+A logical message that Phronomy receives is not flattened merely to make later Context assembly convenient.
+
+- A Provider assistant response is captured as one complete assistant message containing its observable `content` and all Tool Calls.
+- An imported assistant message is journaled as one assistant message with the structure supplied by the Import contract.
+- A Tool value returned by Phronomy Tool execution is an execution fact (`tool_result`).
+- The Tool-role message actually appended to the LLM conversation is a separate logical fact (`tool_message`).
+- Imported Tool-role messages are journaled directly as `tool_message` records; Phronomy does not invent a separate raw Tool execution result for an execution it did not observe.
+
+The Journal therefore does not need to infer or reconstruct a source message boundary that Phronomy already observed. Context Policy can inspect Tool Call IDs contained in an assistant message and form protocol-safe selection units with the corresponding Tool messages.
+
+### Import boundary
+
+The application supplying imported history is responsible for satisfying Phronomy's Import contract. Phronomy interprets valid input according to that contract and rejects only data that is invalid under the contract, such as unsupported roles, missing Tool Call IDs, orphan/duplicate Tool results, unresolved Tool calls, or malformed message structure.
+
+External resource acquisition is not a `ContextImporter` responsibility. Files, URLs and similar resources are obtained and interpreted by the Application or Tool that owns that capability; Phronomy journals the logical content/results it actually receives. `ContextImporter` therefore does not introduce RubyLLM-specific attachment handling or an attachment-specific reject path.
+
+Phronomy must not reject an otherwise valid input merely because an internal flattened representation would lose information. In particular, two separately supplied assistant messages remain two separate Journal messages.
+
+### Manifest boundary
+
+The Manifest fixes what one LLM Call will actually receive after Context Policy selection. A historical raw Tool return value and the Tool message produced from it are not interchangeable: the raw result belongs to the execution log, while the message selected for an LLM Call belongs to the Manifest input path.
+
+One Provider response is captured as a Phronomy-owned `ProviderCallOutcome` before Agent-owned Tool execution starts. The canonical assistant-message record is produced from that outcome, not from Application callback delivery.
+
+Context selection is expressed separately through `ContextCandidate`, dependency-aware `ContextSelectionUnit`, `ContextRequest`, `ContextPolicy`, validated `ContextPlan`, and final token-budget validation. An assistant message containing Tool Calls and the corresponding Tool messages form an atomic protocol unit. Ordinary messages in the same `execution_id` remain independently selectable.
+
+## RubyLLM boundary
+
+Agent-owned Tool execution requires RubyLLM's additive callback contract introduced in RubyLLM 1.15. Phronomy therefore requires `ruby_llm >= 1.15, < 2`.
+
+RubyLLM 1.15 adds the complete assistant message to `Chat#messages` before `before_tool_call` callbacks run. Phronomy captures the immutable Provider outcome at that boundary and raises `ToolCallIntercepted` only as an internal control transfer so approval, suspension, parallel dispatch and durable state remain Phronomy-owned.
+
+## Consequences
+
+- Canonical execution history is independent of the current Context budget or policy.
+- Import and runtime histories converge on the same canonical assistant/tool-message model without synthetic grouping identity.
+- Old optional working history may be excluded from a follow-up Manifest without being deleted.
+- Raw Tool results remain available as execution facts even when the corresponding Tool message is omitted from a later Manifest.
+- Tool protocol dependencies are validated independently from semantic selection policy.
+- Context Policy can later introduce deterministic derived/compacted records without replacing their raw sources.
+- Public custom Context Policy APIs, transaction-boundary restructuring, deterministic compaction, Manifest v2/tool subsets, and legacy Assembler removal remain later phases.

data/lib/phronomy/agent/activation_registry.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+module Phronomy
+  module Agent
+    class ActivationRegistry
+      def initialize
+        @mutex = Mutex.new
+        @records = {}
+      end
+
+      def register(activation)
+        @mutex.synchronize do
+          raise ArgumentError, "activation already registered: #{activation.execution_id}" if @records.key?(activation.execution_id)
+          @records[activation.execution_id] = activation
+        end
+        activation
+      end
+
+      def fetch(execution_id)
+        @mutex.synchronize { @records[execution_id.to_s] }
+      end
+
+      def delete(execution_id)
+        @mutex.synchronize { @records.delete(execution_id.to_s) }
+      end
+    end
+  end
+end

data/lib/phronomy/agent/agent_execution.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+require "securerandom"
+require "time"
+
+module Phronomy
+  module Agent
+    class AgentExecution
+      ACTIVE_STATUSES = %i[preparing active suspended].freeze
+      TERMINAL_STATUSES = %i[completed failed cancelled rejected blocked].freeze
+      TRANSITIONS = {
+        preparing: %i[preparing active failed cancelled blocked],
+        active: %i[active suspended completed failed cancelled rejected blocked],
+        suspended: %i[suspended active failed cancelled],
+        completed: %i[completed],
+        failed: %i[failed],
+        cancelled: %i[cancelled],
+        rejected: %i[rejected],
+        blocked: %i[blocked]
+      }.freeze
+
+      ATTRIBUTES = %i[
+        execution_id agent_id execution_revision status phase
+        base_agent_revision base_context_revision base_journal_position
+        working_records llm_calls approval_request result_ref error_ref
+        created_at updated_at terminal_reason metadata
+      ].freeze
+      attr_reader(*ATTRIBUTES)
+
+      def self.start(agent_root:, input_record:, metadata: {})
+        now = Time.now.utc.iso8601(6)
+        new(
+          execution_id: SecureRandom.uuid,
+          agent_id: agent_root.agent_id,
+          execution_revision: 0,
+          status: :preparing,
+          phase: :preparing,
+          base_agent_revision: agent_root.agent_revision,
+          base_context_revision: agent_root.context_revision,
+          base_journal_position: agent_root.journal_position,
+          working_records: [input_record],
+          llm_calls: [],
+          approval_request: nil,
+          result_ref: nil,
+          error_ref: nil,
+          created_at: now,
+          updated_at: now,
+          terminal_reason: nil,
+          metadata: metadata
+        )
+      end
+
+      def initialize(**attributes)
+        ATTRIBUTES.each do |name|
+          value = attributes.fetch(name)
+          value = value.to_sym if %i[status phase].include?(name)
+          instance_variable_set("@#{name}", Immutable.copy(value))
+        end
+        raise ArgumentError, "unknown execution status: #{status.inspect}" unless TRANSITIONS.key?(status)
+        raise ArgumentError, "execution_revision must be non-negative" if execution_revision.negative?
+        Immutable.validate_canonical_json!(metadata, label: "Execution metadata")
+        if approval_request
+          Immutable.validate_canonical_json!(approval_request, label: "Approval request")
+        end
+        freeze
+      end
+
+      def active?
+        ACTIVE_STATUSES.include?(status)
+      end
+
+      def terminal?
+        TERMINAL_STATUSES.include?(status)
+      end
+
+      def with(**changes)
+        next_status = changes.fetch(:status, status).to_sym
+        unless TRANSITIONS.fetch(status).include?(next_status)
+          raise ArgumentError, "illegal AgentExecution transition: #{status} -> #{next_status}"
+        end
+        values = ATTRIBUTES.to_h { |name| [name, public_send(name)] }.merge(changes)
+        values[:execution_revision] = execution_revision + 1 unless changes.key?(:execution_revision)
+        values[:updated_at] = Time.now.utc.iso8601(6) unless changes.key?(:updated_at)
+        self.class.new(**values)
+      end
+
+      def to_h
+        ATTRIBUTES.to_h do |name|
+          value = public_send(name)
+          value = value.map(&:to_h) if name == :working_records
+          value = value.map(&:to_h) if name == :llm_calls
+          [name.to_s, value]
+        end
+      end
+    end
+  end
+end

data/lib/phronomy/agent/agent_execution_activation.rb

@@ -0,0 +1,172 @@
+# frozen_string_literal: true
+
+require "securerandom"
+require "time"
+
+module Phronomy
+  module Agent
+    class AgentExecutionActivation
+      attr_reader :execution_id, :agent, :application_listener, :coordinator,
+        :base_manifest
+      attr_accessor :invocation, :session
+
+      def initialize(
+        execution:,
+        agent:,
+        runtime_projection:,
+        coordinator:,
+        application_listener: nil
+      )
+        @execution_id = execution.execution_id
+        @execution = execution
+        @agent = agent
+        @runtime_projection = runtime_projection
+        @base_manifest = runtime_projection.manifest
+        @coordinator = coordinator
+        @application_listener = application_listener
+        @mutex = Mutex.new
+        @active_call = nil
+        @llm_results = []
+        @runtime_events = []
+        @callback_errors = []
+      end
+
+      def execution
+        @mutex.synchronize { @execution }
+      end
+
+      def replace_execution(value)
+        @mutex.synchronize { @execution = value }
+      end
+
+      def runtime_projection
+        @mutex.synchronize { @runtime_projection }
+      end
+
+      def replace_runtime_projection(value)
+        @mutex.synchronize { @runtime_projection = value }
+      end
+
+      # Allocates Provider Call identity before transport begins. This identity is
+      # provenance only; Context selection must not use it as a semantic boundary.
+      def begin_llm_call(projection)
+        call_context = {
+          llm_call_id: SecureRandom.uuid,
+          manifest_ref: projection.manifest_ref,
+          started_at: Time.now.utc.iso8601(6)
+        }.freeze
+        @mutex.synchronize do
+          if @active_call
+            raise Phronomy::Error,
+              "cannot start a Provider Call while another Provider Call is active"
+          end
+          @runtime_projection = projection
+          @active_call = call_context
+        end
+        call_context
+      end
+
+      def record_llm_result(response:, error:, streaming:)
+        @mutex.synchronize do
+          active_call = @active_call
+          unless active_call
+            raise Phronomy::Error, "LLM result arrived without an active Provider Call"
+          end
+          @llm_results << {
+            llm_call_id: active_call.fetch(:llm_call_id),
+            response: response,
+            error: error,
+            streaming: streaming,
+            manifest_ref: active_call.fetch(:manifest_ref),
+            started_at: active_call.fetch(:started_at)
+          }
+          @active_call = nil
+        end
+      end
+
+      def runtime_snapshot
+        @mutex.synchronize do
+          {
+            llm_results: @llm_results.dup,
+            runtime_events: @runtime_events.dup,
+            active_call: @active_call&.dup
+          }
+        end
+      end
+
+      def acknowledge_runtime_snapshot(snapshot)
+        @mutex.synchronize do
+          @llm_results.shift(snapshot.fetch(:llm_results).length)
+          @runtime_events.shift(snapshot.fetch(:runtime_events).length)
+          if snapshot[:active_call] && @active_call == snapshot[:active_call]
+            @active_call = nil
+          end
+        end
+      end
+
+      ApplicationCallbackFailure = Data.define(:event_type, :error) do
+        def to_stream_callback_error
+          wrapped = Phronomy::StreamCallbackError.new(
+            event_type: event_type, original_error: error, result: nil
+          )
+          begin
+            raise wrapped, cause: error
+          rescue Phronomy::StreamCallbackError => caught
+            caught.set_backtrace(error.backtrace)
+            caught
+          end
+        end
+      end
+
+      attr_reader :callback_failure
+
+      def callback_failed?
+        @mutex.synchronize { !@callback_failure.nil? }
+      end
+
+      # Canonical runtime recording is independent of Application callback health.
+      # Once an event is observed it is appended even after a listener has failed.
+      def record_event(event)
+        listener = @mutex.synchronize do
+          @runtime_events << event
+          @callback_failure ? nil : @application_listener
+        end
+        return unless listener
+
+        listener.call(event)
+      rescue => callback_error
+        failure = ApplicationCallbackFailure.new(
+          event_type: event.type, error: callback_error
+        )
+        @mutex.synchronize do
+          @callback_failure ||= failure
+          @application_listener = nil
+        end
+        notify_callback_failure(failure)
+      end
+
+      private
+
+      def notify_callback_failure(failure)
+        invocation = @mutex.synchronize { @invocation }
+        if invocation
+          accepted = Phronomy::Runtime.instance.event_loop.post_to_session(
+            Phronomy::Event.new(
+              type: :application_callback_failed,
+              target_id: invocation.id,
+              payload: {failure: failure}
+            )
+          )
+          unless accepted
+            Phronomy.configuration.logger&.warn(
+              "[Phronomy] Callback failure recorded but could not notify FSM: execution_id=#{@execution_id}"
+            )
+          end
+        end
+        Phronomy.configuration.logger&.warn(
+          "[Phronomy] Application event listener failed: #{failure.error.class}: #{failure.error.message}"
+        )
+      end
+    end
+  end
+end

data/lib/phronomy/agent/agent_invocation.rb

@@ -26,6 +26,8 @@ module Phronomy
         llm_failed
       ].freeze
 
+      CALLBACK_FAILED_EVENTS = %i[application_callback_failed].freeze
+
       attr_accessor :input,
         :messages,
         :chat,
@@ -50,7 +52,9 @@ module Phronomy
         :tool_invocations,
         :session_id,
         :phase,
-        :mode
+        :mode,
+        :current_llm_call_id,
+        :tool_batch_llm_call_id
 
       def initialize(
         agent:,
@@ -96,9 +100,10 @@ module Phronomy
         @error = nil
         @session_id = nil
         @phase = nil
+        @current_llm_call_id = nil
+        @tool_batch_llm_call_id = nil
       end
 
-      # Compatibility aliases for existing internal callers.
       def stream_listener
         @event_listener
       end
@@ -116,6 +121,11 @@ module Phronomy
         @phase = phase
       end
 
+      def begin_llm_call!(llm_call_id)
+        @current_llm_call_id = llm_call_id.to_s
+        self
+      end
+
       def pending_tool_calls=(calls)
         @pending_tool_calls = Array(calls)
       end
@@ -128,6 +138,7 @@ module Phronomy
         @pending_tool_calls = []
         @tool_invocations = []
         @approval_request = nil
+        @tool_batch_llm_call_id = nil
       end
 
       def handle_fsm_event(event)
@@ -146,6 +157,11 @@ module Phronomy
           return true
         end
 
+        if CALLBACK_FAILED_EVENTS.include?(event.type)
+          @error ||= event.payload.fetch(:failure).to_stream_callback_error
+          return true
+        end
+
         return false unless TOOL_EVENT_TYPES.include?(event.type)
 
         invocation = tool_invocation(
@@ -164,8 +180,6 @@ module Phronomy
         true
       end
 
-      # Deprecated internal compatibility hook. FSMSession no longer calls
-      # this method; asynchronous results enter through explicit events.
       def apply_fsm_action_result(result)
         event_type =
           if result.respond_to?(:error) &&
@@ -185,15 +199,20 @@ module Phronomy
         self
       end
 
-      def accept_tool_calls!(tool_calls)
+      def accept_tool_calls!(tool_calls, llm_call_id: nil)
         @user_message_sent = true
         @pending_tool_calls = Array(tool_calls)
         @messages = @chat.messages
+        @tool_batch_llm_call_id = (llm_call_id || @current_llm_call_id)&.to_s
+        @current_llm_call_id = nil
         @pending_tool_calls.each do |tool_call|
           deliver_event(
             StreamEvent.new(
               type: :tool_call,
-              payload: {tool_call: tool_call}
+              payload: {
+                tool_call: tool_call,
+                llm_call_id: @tool_batch_llm_call_id
+              }.compact
             )
           )
         end
@@ -210,6 +229,7 @@ module Phronomy
         @usage = Phronomy::TokenUsage.from_tokens(response.tokens)
         @messages = @chat.messages
         @pending_tool_calls = []
+        @current_llm_call_id = nil
         self
       end
 
@@ -255,9 +275,10 @@ module Phronomy
 
       def record_tool_results!
         @tool_invocations.each do |invocation|
+          tool_content = invocation.result.to_s
           @chat.add_message(
             role: :tool,
-            content: invocation.result.to_s,
+            content: tool_content,
             tool_call_id: invocation.tool_call_id
           )
           deliver_event(
@@ -266,8 +287,14 @@ module Phronomy
               payload: {
                 tool_call_id: invocation.tool_call_id,
                 tool_name: invocation.tool_name,
-                tool_result: invocation.result
-              }
+                tool_result: invocation.result,
+                tool_message: {
+                  "role" => "tool",
+                  "content" => tool_content,
+                  "tool_call_id" => invocation.tool_call_id.to_s
+                },
+                llm_call_id: @tool_batch_llm_call_id
+              }.compact
             )
           )
         end
@@ -361,6 +388,7 @@ module Phronomy
         end
 
         if event.type == :llm_failed
+          @current_llm_call_id = nil
           @error = result.error ||
             Phronomy::Error.new("LLM operation failed without an error")
           return
@@ -368,8 +396,12 @@ module Phronomy
 
         if result.error
           if result.error.is_a?(ToolCallIntercepted)
-            accept_tool_calls!(result.error.tool_calls)
+            accept_tool_calls!(
+              result.error.tool_calls,
+              llm_call_id: result.error.llm_call_id
+            )
           else
+            @current_llm_call_id = nil
             @error = result.error
           end
         else

data/lib/phronomy/agent/agent_invocation_session_builder.rb

@@ -132,6 +132,17 @@ module Phronomy
           ],
           resume: [
             {from: :suspended, to: :waiting_for_tools, guard: nil}
+          ],
+          application_callback_failed: [
+            {from: :filtering_input, to: :failed, guard: nil},
+            {from: :building_context, to: :failed, guard: nil},
+            {from: :calling_llm, to: :failed, guard: nil},
+            {from: :starting_tools, to: :failed, guard: nil},
+            {from: :evaluating_tools, to: :failed, guard: nil},
+            {from: :waiting_for_tools, to: :failed, guard: nil},
+            {from: :dispatching_tools, to: :failed, guard: nil},
+            {from: :recording_tool_results, to: :failed, guard: nil},
+            {from: :output_filtering, to: :failed, guard: nil}
           ]
         )
       end
@@ -206,30 +217,58 @@ module Phronomy
           invocation.chat,
           invocation.config
         )
-        install_tool_interceptors(invocation.chat)
+        install_tool_interceptors(invocation.chat, invocation)
         invocation
       end
       private_class_method :building_context_action
 
-      def self.install_tool_interceptors(chat)
+      # RubyLLM >= 1.15 adds the complete assistant Message to chat.messages
+      # before before_tool_call is fired. Capture that Message at the control
+      # boundary instead of trying to recover it after the exception unwinds.
+      def self.install_tool_interceptors(chat, invocation = nil)
+        unless chat.respond_to?(:before_tool_call)
+          raise Phronomy::ConfigurationError,
+            "Agent-owned Tool execution requires RubyLLM >= 1.15 (before_tool_call callback)"
+        end
+
         if chat.respond_to?(:on_tool_call_batch)
           chat.on_tool_call_batch do |tool_calls|
-            raise Phronomy::Agent::ToolCallIntercepted.new(tool_calls)
+            raise build_tool_interception(chat, tool_calls, invocation)
           end
         end
 
-        if chat.respond_to?(:before_tool_call)
-          chat.before_tool_call do |tool_call|
-            raise Phronomy::Agent::ToolCallIntercepted.new(tool_call)
-          end
-        else
-          chat.on_tool_call do |tool_call|
-            raise Phronomy::Agent::ToolCallIntercepted.new(tool_call)
-          end
+        chat.before_tool_call do |tool_call|
+          raise build_tool_interception(chat, [tool_call], invocation)
         end
       end
       private_class_method :install_tool_interceptors
 
+      def self.build_tool_interception(chat, fallback_tool_calls, invocation)
+        assistant_message = chat.messages.last
+        unless assistant_message&.respond_to?(:role) &&
+            assistant_message.role.to_sym == :assistant &&
+            assistant_message.respond_to?(:tool_calls)
+          raise Phronomy::Error,
+            "RubyLLM Tool callback fired before the complete assistant message was observable"
+        end
+
+        message_tool_calls = assistant_message.tool_calls
+        tool_calls = if message_tool_calls.respond_to?(:values)
+          message_tool_calls.values
+        else
+          Array(message_tool_calls)
+        end
+        tool_calls = Array(fallback_tool_calls) if tool_calls.empty?
+
+        ToolCallIntercepted.new(
+          tool_calls,
+          assistant_message: assistant_message,
+          assistant_outcome: ProviderCallOutcome.capture(assistant_message),
+          llm_call_id: invocation&.current_llm_call_id
+        )
+      end
+      private_class_method :build_tool_interception
+
       def self.calling_llm_action(agent, runtime, invocation)
         user_message = invocation.user_message_sent ?
           nil :

data/lib/phronomy/agent/agent_root.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+require "time"
+
+module Phronomy
+  module Agent
+    class AgentRoot
+      LIFECYCLE_STATUSES = %i[idle active suspended closed invalidated].freeze
+
+      ATTRIBUTES = %i[
+        agent_id agent_definition_id definition_version agent_revision
+        context_revision journal_position lifecycle_status transcript_generation memory_generation
+        created_at updated_at metadata
+      ].freeze
+
+      attr_reader(*ATTRIBUTES)
+
+      def self.create(agent_id:, agent_definition_id:, definition_version:, metadata: {})
+        now = Time.now.utc.iso8601(6)
+        new(
+          agent_id: agent_id,
+          agent_definition_id: agent_definition_id,
+          definition_version: definition_version,
+          agent_revision: 0,
+          context_revision: 0,
+          journal_position: 0,
+          lifecycle_status: :idle,
+          transcript_generation: 0,
+          memory_generation: 0,
+          created_at: now,
+          updated_at: now,
+          metadata: metadata
+        )
+      end
+
+      def initialize(**attributes)
+        ATTRIBUTES.each do |name|
+          value = attributes.fetch(name)
+          value = value.to_sym if name == :lifecycle_status
+          instance_variable_set("@#{name}", Immutable.copy(value))
+        end
+        unless LIFECYCLE_STATUSES.include?(lifecycle_status)
+          raise ArgumentError, "unknown Agent lifecycle status: #{lifecycle_status.inspect}"
+        end
+        raise ArgumentError, "agent_revision must be non-negative" if agent_revision.negative?
+        raise ArgumentError, "context_revision must be non-negative" if context_revision.negative?
+        raise ArgumentError, "journal_position must be non-negative" if journal_position.negative?
+        Immutable.validate_canonical_json!(metadata, label: "Agent metadata")
+        freeze
+      end
+
+      def with(**changes)
+        values = to_h.transform_keys(&:to_sym).merge(changes)
+        values[:updated_at] = Time.now.utc.iso8601(6) unless changes.key?(:updated_at)
+        self.class.new(**values)
+      end
+
+      def to_h
+        ATTRIBUTES.to_h { |name| [name.to_s, public_send(name)] }
+      end
+
+      def self.from_h(hash)
+        new(**ATTRIBUTES.to_h { |name| [name, hash.fetch(name.to_s)] })
+      end
+    end
+  end
+end