turnkit 0.7.2 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fb1f58cfea02169c68e433d8a8b84dc5eb2a8a54f18ea9e89853e4ad85445ccd
-  data.tar.gz: '085c56655a5e11670528b157926a6003a04a074944adaf7cf61730cc18c5d6a3'
+  metadata.gz: 73e85ee51e846b0d62828d964ad53b00fd0f6ce1e11d56611b47b45462d49d9f
+  data.tar.gz: f4c4591902e4a9a9fc4c2a50c6e768ad6a4191ac242418434ff5d8b22e99143b
 SHA512:
-  metadata.gz: 38ef2c605e8ee0d5a15417db75a6efce91b045103f68f86a241f7db788806b63ab6dbdd16cc40b1bee24b54ed93f6b83fb9331e13dbf6d06900bc79f4e7305d8
-  data.tar.gz: a9a752449f11422b599b9eaba4598441a295366fc4ab1dddccb136689d9279c2e8b426de9d1f2f8b45fe74011209bd22942e388033892547b5f8355a65d8e56c
+  metadata.gz: cd39ba14aa1e48ce133d8b4cffc76d63939494a90466cfd62a5b70cb612f76f2caf8a5ffafc371272f00644ff1842a9b96768fea0724788721c45c3d36163655
+  data.tar.gz: a77ae5606a43be2790b54cf2a6c8eb78225b5f8b41d0f7b166b8c67b6568a357e65d1ae8f690b446c55afde370aab33d1d3f21dad30a7aec493eb2eb6007224c

data/CHANGELOG.md

@@ -1,5 +1,31 @@
 # Changelog
 
+## 0.9.0 - 2026-09-19
+
+- Add `Turn#internal_evaluation` and a dependency-free Cloudflare Jev adapter for
+  Noul/Choice/Score judgments, separate from chat. Persist request/attempt receipts
+  with fenced atomic usage/cost accounting and bounded opt-in retries.
+- Add optional runtime-owned `output_metadata` to child results so output-audit
+  callables can report assessments without changing strict generated schemas.
+- Preserve unknown evaluation costs in cost summaries and retain total-only
+  provider charges when aggregating them with itemized costs.
+
+## 0.8.0 - 2026-09-18
+
+- Add typed delegation to `SubAgentTool`: subclasses declare their own
+  parameters, an `agent` class macro (or instance `agent`), and `task_for` to
+  build the child task inside the runtime so bulk data never enters the parent
+  context. Emit `sub_agent.delegated` with `task_chars` per delegation, and
+  auto-register agents owned by sub-agent tools alongside `sub_agents`.
+- Add `Agent#tool_policy`, a per-agent routing/cost gate that runs after
+  authorization and returns `:allow` or `[:block, reason]`; blocked calls return
+  the reason to the model with `details["tool_policy_blocked"]`.
+- Add `examples/shunt`, a Spotify Portal-style context router with
+  `bulk_read`/`code_write` worker agents, a large-file `read_file` gate, and
+  measured savings.
+- Breaking: `SubAgentTool#build_child` is an instance method, and the `task`
+  parameter is declared only on `SubAgentTool.for` classes.
+
 ## 0.7.2 - 2026-09-10
 
 - Add explicit `Tool.budget_completion!` for replay-safe local terminal saves

data/README.md

@@ -18,6 +18,10 @@ For GPT-6 Astra tools, opt into the pinned RubyLLM 2 release candidate and
 The [live validation app](examples/interactive_validation/README.md) exercises
 these controls with Rails 8.1, PostgreSQL, Sidekiq, and actual Astra/xhigh requests.
 
+For non-generative judgments, use [structured evaluations](docs/structured-evaluations.md).
+Cloudflare Jev supports Noul, Choice, and Score through a separate typed API with
+durable receipts, usage/cost accounting, and output-audit integration—not chat messages.
+
 ## Installation
 
 Add this line to your application's **Gemfile**:
@@ -590,6 +594,28 @@ puts turn.output_text
 
 Rely on TurnKit to validate tools and model-provided arguments.
 
+#### Tool policies
+
+Gate tool calls per agent for routing or cost, separately from identity
+authorization:
+
+```ruby
+agent = TurnKit::Agent.new(
+  name: "reporter",
+  tools: [ReadFile, BulkRead],
+  tool_policy: lambda do |tool:, arguments:, context:|
+    next :allow unless tool.is_a?(ReadFile) && File.foreach(arguments["path"]).count > 350
+    [:block, "File is large. Use `bulk_read` with a question, or re-read with `offset`/`limit`."]
+  end
+)
+```
+
+The policy runs after authorization and before the tool executes. Return
+`:allow` (or `nil`) to proceed, or `[:block, reason]` to return the reason to
+the model as a tool error with `details["tool_policy_blocked"] = true`. Keep
+`authorization_policy` for who may call what; use `tool_policy` for how much and
+which way.
+
 ### Images
 
 Generate images inside a durable turn with `turn.paint`. The image call uses the
@@ -839,6 +865,32 @@ puts turn.output_text
 
 Use sub-agents for isolated child conversations.
 
+#### Typed delegation
+
+Subclass `SubAgentTool` to build the child task from typed arguments so bulk
+data never enters the parent's context:
+
+```ruby
+class BulkRead < TurnKit::SubAgentTool
+  agent reader
+  description "Read files and answer a question about them."
+
+  parameter :question, :string, required: true
+  parameter :paths, :array, required: true, items: :string
+
+  def task_for(question:, paths:)
+    files = paths.map { |path| "<file path=\"#{path}\">\n#{File.read(path)}\n</file>" }
+    "#{question}\n\n#{files.join("\n")}"
+  end
+end
+```
+
+The parent model supplies `question` and `paths`; `task_for` runs inside the
+runtime, and only the child's answer returns to the parent. Override `agent` on
+an instance when the child agent is configured at runtime. Each delegation emits
+`sub_agent.delegated` with `task_chars`, so avoided parent context is
+measurable. See [`examples/shunt`](examples/shunt) for a complete routing setup.
+
 #### Oracle- and Librarian-style specialists
 
 Use ordinary agents, not a second specialist runtime. Give each specialist its

data/lib/turnkit/adapters/cloudflare_jev.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+require "net/http"
+require "timeout"
+
+module TurnKit
+  module Adapters
+    class CloudflareJev
+      MODEL = "typesafe/jev"
+
+      def initialize(account_id:, api_token:)
+        raise ConfigError, "Cloudflare account ID is required" unless account_id.to_s.match?(/\A[a-zA-Z0-9_-]+\z/)
+        raise ConfigError, "Cloudflare API token is required" if api_token.to_s.empty?
+        @account_id, @api_token = account_id, api_token
+      end
+
+      def inspect = "#<#{self.class.name}>"
+
+      # Include route/account, but never the bearer credential, in receipt identity.
+      def identity = { "provider" => "cloudflare", "account" => @account_id, "schema" => 1 }
+      def cost_model(model:) = "cloudflare/#{model}"
+
+      def validate!(model:, state:, questions:)
+        raise InputError, "unsupported Cloudflare evaluation model" unless model == MODEL
+        Evaluation.input!(state: state, questions: questions)
+      end
+
+      def evaluate(model:, state:, questions:, timeout:)
+        raise ArgumentError, "timeout must be positive and finite" unless timeout.is_a?(Numeric) && timeout.finite? && timeout.positive?
+        input = validate!(model: model, state: state, questions: questions)
+        uri = URI("https://api.cloudflare.com/client/v4/accounts/#{@account_id}/ai/run")
+        request = Net::HTTP::Post.new(uri)
+        request["Authorization"] = "Bearer #{@api_token}"
+        request["Content-Type"] = "application/json"
+        request.body = JSON.generate(model: model, input: input)
+        response = Timeout.timeout(timeout) do
+          Net::HTTP.start(uri.host, uri.port, use_ssl: true, open_timeout: timeout, read_timeout: timeout, write_timeout: timeout) do |http|
+            http.max_retries = 0
+            http.request(request)
+          end
+        end
+        code = response.code.to_i
+        unless code.between?(200, 299)
+          transient = [408, 429, 500, 502, 503, 504].include?(code)
+          # Quota exhaustion is not transient capacity pressure.
+          if code == 429
+            errors = JSON.parse(response.body).fetch("errors", []) rescue []
+            transient = false if Array(errors).any? { |error| error.is_a?(Hash) && error["code"] == 3036 }
+          end
+          delay = retry_delay(response["Retry-After"])
+          raise EvaluationError.new(code >= 500 || code == 408 ? :uncertain : :unavailable,
+            retryable: transient, retry_after: delay, http_status: code)
+        end
+        value = JSON.parse(response.body)
+        if value.is_a?(Hash) && value.key?("result")
+          raise EvaluationError.new(:unavailable) unless value["success"] == true
+          value = value["result"]
+        end
+        Evaluation.result!(value, questions: input.fetch("questions"))
+      rescue JSON::ParserError
+        raise EvaluationError.new(:malformed), cause: nil
+      rescue Timeout::Error, IOError, SystemCallError, OpenSSL::SSL::SSLError, SocketError, Net::HTTPBadResponse, Net::ProtocolError
+        raise EvaluationError.new(:uncertain, retryable: true), cause: nil
+      end
+
+      private
+        def retry_delay(value)
+          return unless value
+          return value.to_f if value.match?(/\A\d+(\.\d+)?\z/)
+          [Time.httpdate(value) - Clock.now, 0].max
+        rescue ArgumentError
+          nil
+        end
+    end
+  end
+end

data/lib/turnkit/agent.rb

@@ -24,10 +24,10 @@ module TurnKit
     attr_reader :name, :description, :model, :instructions, :tools, :skills, :available_skills, :sub_agents
     attr_reader :client, :store, :max_iterations, :timeout, :max_spend, :max_depth, :max_tool_executions, :max_tool_executions_by_name
     attr_reader :prompt_sections, :system_prompt, :prompt_mode, :thinking, :compaction, :output_schema, :input_schema, :on_event
-    attr_reader :output_policy, :output_policy_mode, :output_policy_model, :output_retries, :context_contributors
+    attr_reader :output_policy, :output_policy_mode, :output_policy_model, :output_retries, :context_contributors, :tool_policy
 
     def initialize(name:, description: "", model: nil, instructions: "", orchestrator: false, tools: [], skills: [], available_skills: [], sub_agents: [],
-      system_prompt: nil, prompt_sections: nil, prompt_mode: nil, client: nil, store: nil,
+      tool_policy: nil, system_prompt: nil, prompt_sections: nil, prompt_mode: nil, client: nil, store: nil,
       max_iterations: nil, timeout: nil, max_spend: nil, max_depth: nil, max_tool_executions: nil, max_tool_executions_by_name: nil, thinking: nil, compaction: nil,
       output_schema: nil, input_schema: nil, output_policy: nil, output_policy_mode: nil, output_policy_model: nil, output_policy_thinking: nil, output_retries: 0, on_event: nil, context_contributors: [], inherit_globals: true)
       @name = name.to_s
@@ -39,6 +39,7 @@ module TurnKit
       @skills = Array(skills).dup.freeze
       @available_skills = ((inherit_globals ? Array(TurnKit.available_skills) : []) + Array(available_skills)).uniq { |skill| skill.key }.freeze
       @sub_agents = Array(sub_agents).dup.freeze
+      @tool_policy = tool_policy
       @system_prompt = system_prompt
       @prompt_sections = prompt_sections
       @prompt_mode = prompt_mode&.to_sym || (:task if @orchestrator)

data/lib/turnkit/background.rb

@@ -264,7 +264,7 @@ module TurnKit
             loaded = Background.load_turn(record.fetch("id"), store: store)
             tool = loaded.agent.effective_tools(turn: loaded).find { |candidate| candidate.tool_name == execution["tool_name"] }
             recovery = tool.is_a?(Class) ? tool.recovery : tool&.class&.recovery
-            ordinary = tool && !(tool.is_a?(Class) && tool < SubAgentTool) &&
+            ordinary = tool && !SubAgentTool.delegates?(tool) &&
               ![WaitTool, LaunchAgentTool, SendMessageTool].include?(tool)
             if ordinary && recovery == :replay_safe
               store.claim_tool_execution(execution.fetch("id"), to: "pending", started_at: nil)

data/lib/turnkit/coordination_tools.rb

@@ -39,7 +39,7 @@ module TurnKit
         if existing
           child = existing
         else
-          built = SubAgentTool.for(agent).build_child(task: task, context: context)
+          built = SubAgentTool.for(agent).new.build_child(task: task, context: context)
           options = parent.store.load_turn(built.id).fetch("options")
           options = options.merge("callback_conversation_id" => parent.conversation.id) if callback
           child = parent.store.update_turn(built.id, submitted_at: Clock.now, options: options)

data/lib/turnkit/cost.rb

@@ -10,6 +10,13 @@ module TurnKit
     def self.aggregate(costs)
       costs = costs.compact
       return new unless costs.any?
+      return new(unknown: true) if costs.any?(&:unknown?)
+      # A provider-supplied total cannot be assigned to a token component.
+      # Preserve it when combining chat totals with itemized evaluation prices.
+      if costs.any? { |cost| cost.total && COMPONENTS.all? { |component| cost.public_send(component).nil? } }
+        totals = costs.map(&:total)
+        return totals.any?(&:nil?) ? new : new(total: totals.sum)
+      end
 
       if costs.any? { |cost| COMPONENTS.any? { |component| !cost.public_send(component).nil? } }
         values = COMPONENTS.to_h do |component|
@@ -41,6 +48,10 @@ module TurnKit
 
     def self.from_record(record)
       attrs = record.transform_keys(&:to_s)
+      evaluations = attrs.dig("options", "state", "evaluations") || {}
+      if evaluations.values.any? { |receipt| receipt.fetch("attempts", []).any? { |attempt| attempt["cost"].nil? } }
+        return new(unknown: true)
+      end
       usage = attrs["usage"] || {}
       return from_hash(usage["cost_details"] || usage[:cost_details]) if usage["cost_details"] || usage[:cost_details]
       return new(total: attrs["cost"]) if attrs["cost"]
@@ -93,7 +104,8 @@ module TurnKit
         cache_read: hash[:cache_read],
         cache_write: hash[:cache_write],
         thinking: hash[:thinking],
-        total: hash[:total]
+        total: hash[:total],
+        unknown: hash[:unknown] || false
       )
     end
 
@@ -120,7 +132,7 @@ module TurnKit
       tokens.to_i * price.to_f / PER_MILLION
     end
 
-    def initialize(input: nil, output: nil, cache_read: nil, cache_write: nil, thinking: nil, total: nil, strict: false)
+    def initialize(input: nil, output: nil, cache_read: nil, cache_write: nil, thinking: nil, total: nil, strict: false, unknown: false)
       @input = number(input)
       @output = number(output)
       @cache_read = number(cache_read)
@@ -128,9 +140,13 @@ module TurnKit
       @thinking = number(thinking)
       @total = number(total)
       @strict = strict
+      @unknown = unknown
     end
 
+    def unknown? = @unknown
+
     def total
+      return nil if unknown?
       return @total if @total
       return nil if @strict && COMPONENTS.any? { |component| public_send(component).nil? }
 
@@ -145,7 +161,8 @@ module TurnKit
         "cache_read" => cache_read,
         "cache_write" => cache_write,
         "thinking" => thinking,
-        "total" => total
+        "total" => total,
+        "unknown" => (true if unknown?)
       }.compact
     end
 

data/lib/turnkit/evaluation.rb

@@ -0,0 +1,119 @@
+# frozen_string_literal: true
+
+module TurnKit
+  # Deliberately separate from Result: evaluations have no message parts or text.
+  class EvaluationResult
+    attr_reader :model, :answers, :usage, :receipt_id
+
+    def initialize(model:, answers:, usage:, receipt_id: nil)
+      @model, @answers, @usage, @receipt_id = model, answers, usage, receipt_id
+    end
+
+    def to_h
+      { "model" => model, "answers" => answers, "usage" => usage.to_h }
+    end
+
+    def self.from_h(value, receipt_id: nil)
+      new(model: value.fetch("model"), answers: value.fetch("answers"),
+        usage: Usage.from_h(value.fetch("usage")), receipt_id: receipt_id)
+    end
+  end
+
+  class EvaluationError < Error
+    attr_reader :status, :retry_after, :usage, :model, :http_status
+
+    def initialize(status, retryable: false, retry_after: nil, usage: nil, model: nil, http_status: nil)
+      @status, @retryable, @retry_after, @usage, @model = status.to_s, retryable, retry_after, usage, model
+      @http_status = http_status
+      # Never include provider bodies, URLs, state, question names, or credentials.
+      super("evaluation #{@status}")
+    end
+
+    def retryable? = @retryable
+  end
+
+  module Evaluation
+    module_function
+
+    # Canonical JSON also rejects Ruby objects which JSON.generate would stringify.
+    def json(value)
+      case value
+      when Hash
+        raise InputError, "evaluation keys must be strings or symbols" unless value.keys.all? { |k| k.is_a?(String) || k.is_a?(Symbol) }
+        raise InputError, "duplicate evaluation keys" unless value.keys.map(&:to_s).uniq.size == value.size
+        value.transform_keys(&:to_s).sort.to_h.transform_values { |v| json(v) }
+      when Array then value.map { |v| json(v) }
+      when String, Integer, TrueClass, FalseClass, NilClass then value
+      when Float
+        raise InputError, "evaluation numbers must be finite" unless value.finite?
+        value
+      else raise InputError, "evaluation values must be JSON"
+      end
+    end
+
+    def text?(value) = value.nil? || value.is_a?(String) || value.is_a?(Hash) || value.is_a?(Array)
+
+    def input!(state:, questions:)
+      input = json({ state: state, questions: questions })
+      valid = text?(input["state"]) && input["questions"].is_a?(Hash)
+      raise InputError, "invalid evaluation input" unless valid
+      input["questions"].each do |id, question|
+        valid = !id.empty? && question.is_a?(Hash) && (question.keys - %w[type instructions criteria]).empty? &&
+          question.key?("instructions") && text?(question["instructions"])
+        raise InputError, "invalid evaluation question" unless valid
+        criteria = question["criteria"]
+        valid = case question["type"]
+        when "noul"
+          criteria.nil? || (criteria.is_a?(Hash) && (criteria.keys - %w[true false]).empty? && criteria.values.all? { |v| text?(v) })
+        when "choice"
+          criteria.is_a?(Hash) && criteria.values.all? { |v| text?(v) }
+        when "score"
+          criteria.is_a?(Array) && criteria.size >= 2 && criteria.all? { |v| text?(v) }
+        else false
+        end
+        raise InputError, "invalid evaluation criteria" unless valid
+      end
+      input
+    end
+
+    def probability?(value) = value.is_a?(Numeric) && value.finite? && value.between?(0, 1)
+
+    def usage(value)
+      return unless value.is_a?(Hash) && value.keys.sort == %w[input_tokens output_tokens]
+      return unless value.values.all? { |v| v.is_a?(Integer) && v.between?(0, 9_007_199_254_740_991) }
+      Usage.new(input_tokens: value["input_tokens"], output_tokens: value["output_tokens"])
+    end
+
+    def result!(value, questions:)
+      observed = usage(value["usage"]) if value.is_a?(Hash)
+      model = value["model"] if value.is_a?(Hash) && value["model"].is_a?(String) && !value["model"].empty?
+      invalid = -> { raise EvaluationError.new(:malformed, usage: observed, model: model) }
+      invalid.call unless value.is_a?(Hash) && value.keys.sort == %w[answers model usage] && model && observed
+      answers = value["answers"]
+      invalid.call unless answers.is_a?(Hash) && answers.keys.sort == questions.keys.sort
+      answers.each do |id, answer|
+        question = questions.fetch(id)
+        type = question.fetch("type")
+        invalid.call unless answer.is_a?(Hash) && answer["type"] == type
+        if type == "noul"
+          invalid.call unless answer.keys.sort == %w[noul type] && probability?(answer["noul"])
+          next
+        end
+        keys = type == "choice" ? %w[choice confidence probabilities type] : %w[confidence legend probabilities score type]
+        invalid.call unless answer.keys.sort == keys && probability?(answer["confidence"])
+        expected = type == "choice" ? question.fetch("criteria").keys : question.fetch("criteria").each_index.map(&:to_s)
+        probabilities = answer["probabilities"]
+        invalid.call unless probabilities.is_a?(Hash) && probabilities.keys.sort == expected.sort &&
+          probabilities.values.all? { |p| probability?(p) } && (probabilities.values.sum - 1).abs <= 0.01
+        if type == "choice"
+          invalid.call unless expected.include?(answer["choice"])
+        else
+          score, legend = answer.values_at("score", "legend")
+          invalid.call unless score.is_a?(Numeric) && score.finite? && score.between?(0, expected.size - 1) &&
+            legend.is_a?(Hash) && legend.keys.sort == expected.sort && legend.values.all? { |v| v.is_a?(String) }
+        end
+      end
+      EvaluationResult.new(model: model, answers: answers, usage: observed)
+    end
+  end
+end

data/lib/turnkit/internal_evaluation.rb

@@ -0,0 +1,142 @@
+# frozen_string_literal: true
+
+module TurnKit
+  module InternalEvaluation
+    # Call from a tool or output-audit callable, never from a persistence callback.
+    # before_dispatch is application transport/spend policy, not model input.
+    def internal_evaluation(evaluator:, model:, purpose:, state:, questions:, policy_version:,
+      candidate: nil, max_attempts: 1, timeout: 30, before_dispatch: nil)
+      raise LostClaim, "evaluation requires an owned execution" unless store.is_a?(ExecutionStore)
+      raise ArgumentError, "max_attempts must be 1..3" unless max_attempts.is_a?(Integer) && (1..3).include?(max_attempts)
+      raise ArgumentError, "timeout must be positive and finite" unless timeout.is_a?(Numeric) && timeout.finite? && timeout.positive?
+      input = evaluator.validate!(model: model, state: state, questions: questions)
+      reload
+      identity = Evaluation.json(provider: evaluator.identity, model: model, purpose: purpose.to_s,
+        policy_version: policy_version.to_s, candidate: candidate, input: input, schema: 1,
+        output_candidate: @record.dig("options", "state", "candidate"),
+        output_data: @record.dig("options", "state", "output_data"),
+        timeout: timeout, max_attempts: max_attempts)
+      key = Digest::SHA256.hexdigest(JSON.generate(identity))
+      cost_model = evaluator.cost_model(model: model)
+      deadline = Clock.now + timeout
+      loop do
+        receipt = store.atomic do
+          reload
+          raise LostClaim, "evaluation requires an owned execution" unless store.is_a?(ExecutionStore) && running?
+          (evaluation_receipts[key] || { "id" => key, "model" => model,
+            "provider" => identity["provider"], "purpose" => purpose.to_s,
+            "policy_version" => policy_version.to_s, "cost_model" => cost_model, "attempts" => [] })
+        end
+        return EvaluationResult.from_h(receipt.fetch("result"), receipt_id: key) if receipt["status"] == "completed"
+        last = receipt["attempts"].last
+        if last && (!last["retryable"] || receipt["attempts"].size >= max_attempts)
+          raise EvaluationError.new(last.fetch("status"), http_status: last["http_status"])
+        end
+        if last && last["owner"] == @record["claim_token"] && !last["finished_at"]
+          raise EvaluationError.new(:in_progress)
+        end
+        delay = last ? [last.fetch("retry_at", 0) - Clock.now.to_f, 0].max : 0
+        remaining = check_evaluation_dispatch!(deadline)
+        raise BudgetError, "evaluation deadline exceeded" if delay >= remaining
+        sleep(delay) if delay.positive?
+        Authorization.authorize!(:evaluate, principal: @record.dig("options", "principal"),
+          turn: self, evaluator: evaluator, model: model, purpose: purpose)
+        before_dispatch.call(turn: self, model: model, purpose: purpose) if before_dispatch
+        attempt = nil
+        remaining = nil
+        store.atomic do
+          reload
+          remaining = check_evaluation_dispatch!(deadline)
+          current = evaluation_receipts[key]
+          # A second caller may have committed or dispatched while policy ran.
+          raise EvaluationError.new(:in_progress) if current && current != receipt
+          attempt = { "id" => SecureRandom.uuid, "owner" => @record["claim_token"],
+            "status" => "uncertain", "retryable" => true, "started_at" => Clock.now.iso8601(6),
+            "cost" => nil }
+          receipt["attempts"] << attempt
+          receipt["status"] = "uncertain"
+          save_evaluation_receipt!(key, receipt)
+        end
+        event = { receipt_id: key, attempt_id: attempt["id"], model: model, purpose: purpose.to_s,
+          question_count: input.fetch("questions").size }
+        emit("evaluation.requested", event)
+        result = nil
+        failure = nil
+        begin
+          result = evaluator.evaluate(model: model, state: input.fetch("state"),
+            questions: input.fetch("questions"), timeout: remaining)
+        rescue EvaluationError => error
+          failure = error
+        end
+        observed_usage = result ? result.usage : failure.usage
+        observed_cost = Cost.from_usage(observed_usage, model: cost_model) if observed_usage
+        store.atomic do
+          reload
+          attempt.merge!("status" => result ? "completed" : failure.status,
+            "finished_at" => Clock.now.iso8601(6), "retryable" => failure&.retryable? || false,
+            "http_status" => failure&.http_status,
+            "usage" => observed_usage&.to_h, "cost" => observed_cost&.total,
+            "returned_model" => result ? result.model : failure.model)
+          if failure&.retryable?
+            attempt["retry_at"] = Clock.now.to_f + (failure.retry_after || rand * (2 ** (receipt["attempts"].size - 1)))
+          end
+          receipt["status"] = attempt["status"]
+          receipt["result"] = result.to_h if result
+          add_usage!(observed_usage, cost: observed_cost) if observed_usage
+          save_evaluation_receipt!(key, receipt)
+        end
+        emit(result ? "evaluation.completed" : "evaluation.failed", event.merge(
+          status: attempt["status"], returned_model: attempt["returned_model"], http_status: attempt["http_status"],
+          usage: observed_usage&.to_h, cost: observed_cost&.to_h,
+          duration: Clock.now - Time.iso8601(attempt["started_at"])))
+        # Keep the shared inline budget in sync without turning a paid successful
+        # assessment into a retry just because its own cost exhausted the budget.
+        begin
+          budget.add_cost!(observed_cost&.total)
+        rescue BudgetError
+          # The next dispatch checks the persisted root ledger.
+        end
+        root_budget = execution_budget
+        root_budget.check!(depth: depth, allow_exhausted_spend: true)
+        raise BudgetError, "evaluation deadline exceeded" if Clock.now >= deadline
+        return EvaluationResult.from_h(receipt.fetch("result"), receipt_id: key) if result
+        raise failure unless failure.retryable? && receipt["attempts"].size < max_attempts
+      end
+    end
+
+    # Copies, not references to live state. Attempts without usage have unknown
+    # cost, even if the turn's known-cost subtotal is zero.
+    def evaluation_receipts
+      JSON.parse(JSON.generate(@record.dig("options", "state", "evaluations") || {}))
+    end
+
+    def output_metadata = JSON.parse(JSON.generate(@record.dig("options", "state", "output_metadata") || {}))
+
+    def output_metadata=(value)
+      value = Evaluation.json(value)
+      raise InputError, "output metadata must be an object" unless value.is_a?(Hash)
+      raise LostClaim, "output metadata requires an owned execution" unless store.is_a?(ExecutionStore)
+      store.atomic do
+        reload
+        raise LostClaim, "output metadata requires an owned execution" unless store.is_a?(ExecutionStore) && running?
+        update_state!("output_metadata" => value)
+      end
+    end
+
+    private
+      def save_evaluation_receipt!(key, receipt)
+        update_state!("evaluations" => evaluation_receipts.merge(key => receipt))
+      end
+
+      def check_evaluation_dispatch!(deadline)
+        current_budget = execution_budget
+        current_budget.check!(depth: depth)
+        raise LostClaim, "evaluation execution is no longer running" unless running?
+        raise EvaluationError.new(:interrupted) if @record.dig("options", "controls", "pause_requested")
+        root_deadline = current_budget.root_started_at + current_budget.timeout if current_budget.timeout
+        remaining = [(deadline - Clock.now), (root_deadline - Clock.now if root_deadline)].compact.min
+        raise BudgetError, "evaluation deadline exceeded" unless remaining.positive?
+        remaining
+      end
+  end
+end

data/lib/turnkit/sub_agent_tool.rb

@@ -1,24 +1,49 @@
 # frozen_string_literal: true
 
 module TurnKit
+  # Runs a child agent in a fresh conversation and returns only its final result.
+  # `SubAgentTool.for(agent)` exposes an agent as a tool taking `task`. Subclasses
+  # declare their own parameters and override `task_for` to assemble the task in
+  # Ruby, so bulk data (file contents, records) reaches the child without ever
+  # entering the parent model's context. The child comes from the `agent` class
+  # macro or an instance-level `agent` override.
   class SubAgentTool < Tool
-    parameter :task, :string, required: true, description: "The complete task for the sub-agent, including all relevant context."
-
-    def self.for(agent)
-      Class.new(self) do
-        @agent = agent
-        tool_name agent.name
-        description agent.description.empty? ? "Delegate work to #{agent.name}." : agent.description
-        usage_hint "Use when work can be delegated independently to #{agent.name}. Pass a complete task and only relevant context."
+    class << self
+      def agent(value = nil)
+        @agent = value if value
+        @agent || (superclass < SubAgentTool ? superclass.agent : nil)
+      end
 
-        class << self
-          attr_reader :agent
+      def for(agent)
+        sub_agent = agent
+        Class.new(self) do
+          agent sub_agent
+          tool_name sub_agent.name
+          description sub_agent.description.empty? ? "Delegate work to #{sub_agent.name}." : sub_agent.description
+          usage_hint "Use when work can be delegated independently to #{sub_agent.name}. Pass a complete task and only relevant context."
+          parameter :task, :string, required: true, description: "The complete task for the sub-agent, including all relevant context."
         end
       end
+
+      def delegates?(tool)
+        tool.is_a?(self) || (tool.is_a?(Class) && tool <= self)
+      end
+
+      def result(record)
+        { "conversation_id" => record.fetch("conversation_id"), "turn_id" => record.fetch("id"),
+          "status" => record.fetch("status"), "result" => record["output_text"].to_s,
+          "output_metadata" => record.dig("options", "state", "output_metadata"),
+          "output_data" => record["output_data"], "error" => record["error"] }.compact
+      end
     end
 
-    def self.build_child(task:, context:)
-      sub_agent = agent
+    def agent = self.class.agent
+
+    def task_for(**arguments)
+      arguments.fetch(:task)
+    end
+
+    def build_child(task:, context:)
       parent_turn = context.turn
       lineage = {
         "parent_conversation_id" => parent_turn.conversation.id,
@@ -27,32 +52,28 @@ module TurnKit
         "principal" => context.principal
       }
       store = parent_turn.store
-      record = store.create_conversation("agent_name" => sub_agent.name, "model" => sub_agent.effective_model, "metadata" => lineage)
-      conversation = Conversation.new(agent: sub_agent, record: record, store: store, model: sub_agent.effective_model, metadata: lineage)
+      record = store.create_conversation("agent_name" => agent.name, "model" => agent.effective_model, "metadata" => lineage)
+      conversation = Conversation.new(agent: agent, record: record, store: store, model: agent.effective_model, metadata: lineage)
       trigger = conversation.say(task, metadata: lineage)
+      parent_turn.emit("sub_agent.delegated", id: context.execution.tool_call_id, name: agent.name,
+        conversation_id: record.fetch("id"), task_chars: task.length)
       conversation.build_turn(
         trigger_message_id: trigger.id,
         budget: parent_turn.budget,
         parent_turn: parent_turn,
         parent_tool_execution: context.execution,
         depth: parent_turn.depth + 1,
-        model: sub_agent.effective_model,
-        agent: sub_agent,
+        model: agent.effective_model,
+        agent: agent,
         principal: context.principal,
         on_event: parent_turn.agent.effective_on_event
       )
     end
 
-    def self.result(record)
-      { "conversation_id" => record.fetch("conversation_id"), "turn_id" => record.fetch("id"),
-        "status" => record.fetch("status"), "result" => record["output_text"].to_s,
-        "output_data" => record["output_data"], "error" => record["error"] }.compact
-    end
-
-    def call(task:, context:)
+    def call(context:, **arguments)
       Authorization.authorize!(:launch_agent, principal: context.principal, turn: context.turn,
-        agent: self.class.agent, arguments: { "task" => task })
-      child = self.class.build_child(task: task, context: context)
+        agent: agent, arguments: arguments.transform_keys(&:to_s))
+      child = build_child(task: task_for(**arguments), context: context)
       child.run!
       SubAgentTool.result(child.store.load_turn(child.id))
     end

data/lib/turnkit/tool_runner.rb

@@ -93,6 +93,8 @@ module TurnKit
         context = ToolContext.new(turn: turn, execution: execution)
         payload = begin
           Authorization.authorize!(:tool, principal: context.principal, turn: turn, tool: tool, arguments: tool_call.arguments)
+          blocked = tool_policy_block(tool, tool_call.arguments, context)
+          return finish_error(execution, tool_call, blocked, details: { "tool_policy_blocked" => true }) if blocked
           # Observe cancellation/reconciliation immediately before crossing the
           # external-effect boundary. Calls already sent cannot be recalled.
           control = turn.control_boundary!
@@ -191,20 +193,30 @@ module TurnKit
       end
 
       def subagent?(tool)
-        tool.is_a?(Class) && tool < SubAgentTool
+        SubAgentTool.delegates?(tool)
+      end
+
+      # Agent-owned routing/cost policy, distinct from identity authorization.
+      # Returns the block reason the model sees, or nil to proceed.
+      def tool_policy_block(tool, arguments, context)
+        decision, reason = turn.agent.tool_policy&.call(tool: tool, arguments: arguments, context: context)
+        return reason.to_s if decision == :block
+        raise ArgumentError, "tool_policy must return :allow or [:block, reason]" unless [ nil, :allow ].include?(decision)
       end
 
       def delegate(tool, call, context)
-        arguments = tool.validate_arguments(call.arguments)
+        tool = tool.new if tool.is_a?(Class)
+        arguments = tool.class.validate_arguments(call.arguments)
         Authorization.authorize!(:launch_agent, principal: context.principal, turn: turn, agent: tool.agent, arguments: arguments)
         TurnKit.resolve_agent(tool.agent.name)
+        task = tool.task_for(**arguments.transform_keys(&:to_sym))
         child = turn.store.atomic_graph do
           turn.store.atomic(Background.root_conversation(turn.store, turn.store.load_turn(turn.id))) do
             control = turn.control_boundary!
             next control if control
             row = turn.store.list_turns(root_turn_id: turn.root_turn_id).find { |candidate| candidate["parent_tool_execution_id"] == context.execution.id }
             unless row
-              built = tool.build_child(task: arguments.fetch("task"), context: context)
+              built = tool.build_child(task: task, context: context)
               row = turn.store.update_turn(built.id, submitted_at: Clock.now)
             end
             Background.wait(turn, [row.fetch("id")])

data/lib/turnkit/turn.rb

@@ -3,6 +3,7 @@
 module TurnKit
   class Turn
     include TurnControls
+    include InternalEvaluation
     STATUSES = Record::TURN_STATUSES
 
     attr_reader :agent, :conversation, :store, :budget, :depth
@@ -325,6 +326,7 @@ module TurnKit
               add_usage!(result.usage, cost: cost)
               persist_assistant_message(result)
               update_state!("phase" => result.tool_calls? ? "tools" : "output", "parts" => result.parts,
+                "output_metadata" => nil,
                 "candidate" => result.text, "output_data" => result.output_data, "terminal_tool_name" => nil,
                 "budget_completion_call_id" => select_budget_completion(result))
             end

data/lib/turnkit/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module TurnKit
-  VERSION = "0.7.2"
+  VERSION = "0.9.0"
 end

data/lib/turnkit.rb

@@ -46,9 +46,12 @@ require_relative "turnkit/load_skill_tool"
 require_relative "turnkit/message_projection"
 require_relative "turnkit/tool_runner"
 require_relative "turnkit/turn_controls"
+require_relative "turnkit/evaluation"
+require_relative "turnkit/internal_evaluation"
 require_relative "turnkit/turn"
 require_relative "turnkit/usage"
 require_relative "turnkit/run"
+require_relative "turnkit/adapters/cloudflare_jev"
 require_relative "turnkit/adapters/codex"
 require_relative "turnkit/adapters/ruby_llm"
 require_relative "turnkit/active_record_store"
@@ -78,7 +81,8 @@ module TurnKit
 
   def self.register(agent)
     @agents[agent.name] = agent
-    agent.sub_agents.each { |child| register(child) }
+    tools = agent.effective_tools + agent.available_skills.flat_map(&:tools)
+    tools.each { |tool| register(tool.agent) if SubAgentTool.delegates?(tool) }
     agent
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: turnkit
 version: !ruby/object:Gem::Version
-  version: 0.7.2
+  version: 0.9.0
 platform: ruby
 authors:
 - Sam Couch
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2026-09-10 00:00:00.000000000 Z
+date: 2026-09-19 00:00:00.000000000 Z
 dependencies: []
 description: TurnKit is a Ruby/Rails agent runtime for durable AI conversations, application
   runs, orchestrator agents, tool calling, skills, sub-agents, context compaction,
@@ -37,6 +37,7 @@ files:
 - lib/generators/turnkit/upgrade_generator.rb
 - lib/turnkit.rb
 - lib/turnkit/active_record_store.rb
+- lib/turnkit/adapters/cloudflare_jev.rb
 - lib/turnkit/adapters/codex.rb
 - lib/turnkit/adapters/ruby_llm.rb
 - lib/turnkit/agent.rb
@@ -50,11 +51,13 @@ files:
 - lib/turnkit/coordination_tools.rb
 - lib/turnkit/cost.rb
 - lib/turnkit/error.rb
+- lib/turnkit/evaluation.rb
 - lib/turnkit/event.rb
 - lib/turnkit/execution_store.rb
 - lib/turnkit/id.rb
 - lib/turnkit/image_result.rb
 - lib/turnkit/image_tool.rb
+- lib/turnkit/internal_evaluation.rb
 - lib/turnkit/job.rb
 - lib/turnkit/load_skill_tool.rb
 - lib/turnkit/media_analysis_result.rb