agent-harness 0.41.0 → 0.42.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c4406422d8b90ec6406bb4cdcc6e1d7496b708f43aad306d258c2570ed4f710e
-  data.tar.gz: cd23fe2f7d4b05e2ac18ac9aa2a631a0b292bc2703ae092f4f55a1ca6f3febbb
+  metadata.gz: 9c2a3cf60957839163bbda4f4e202f663ee68915e49ba3853abce85b5f6f16e9
+  data.tar.gz: b5a9950414fc72fe2626abbb3ef50275e2e28267346414d04d330d3ea37c1297
 SHA512:
-  metadata.gz: efb5e61347d70b28e03bb0cd2843429928843147bf8f196d1292af97be97c8ec67ca4ef81e6f169562d1fac5e88e277b06d07a7cd44bf3d85a772effb2c46051
-  data.tar.gz: 7c5fb37c7e14edbc3da68f74dc3f1a6d10e04630e70509a79bd04b9ebbfde2397d6a4753e288dd33d962efae79003728d30cdb0ae113d1bd4864de5995daa112
+  metadata.gz: b6147210553af7975d99dbf53cc52c0b37e7ba25c8424a9cfb24a290bae9172d25bcbc8a6e5b489d1b2ae363251facc756ea0eaae55a6efd3904cdb64fcda025
+  data.tar.gz: f869749041ff450642b8ba940a3f0f9f8ea2cd7a3d788b2414ae7f0115dcb5fde4dcb7f5aa725b4e1ba2216568a1412e2bf5439c47b61bf75b2d06390faebfae

data/.release-please-manifest.json

@@ -1,3 +1,3 @@
 {
-  ".": "0.41.0"
+  ".": "0.42.0"
 }

data/CHANGELOG.md

@@ -5,6 +5,13 @@
 * add runner model compatibility contract (`AgentHarness.model_compatibility`) with structured `ModelCompatibility::Result` outcomes. Codex exposes static facts for CLI-gated models (e.g. `gpt-5.5` requires Codex CLI `>= 0.116.0`), a baseline supported-model list, supported auth modes, and a `DEFAULT_COMPATIBLE_MODEL_ID` fallback so downstream orchestrators can validate tier/model assignments before scheduling agent runs ([#259](https://github.com/viamin/agent-harness/issues/259)).
 * **auth:** add provider-owned PKCE code-exchange API for Claude OAuth (`AgentHarness::Authentication.exchange_code`). Takes an authorization code plus PKCE verifier (and `redirect_uri`/`client_id`), posts an `authorization_code` grant to the Claude token endpoint, and persists the resulting access/refresh tokens in the native `claudeAiOauth` shape. Adds `exchange_code_supported?` and a `code_exchange` key to `auth_capabilities` ([#266](https://github.com/viamin/agent-harness/issues/266)).
 
+## [0.42.0](https://github.com/viamin/agent-harness/compare/agent-harness/v0.41.0...agent-harness/v0.42.0) (2026-09-25)
+
+
+### Features
+
+* Expose Attempt-Level Usage and Cost (RDR-072) ([#443](https://github.com/viamin/agent-harness/issues/443)) ([818f8ca](https://github.com/viamin/agent-harness/commit/818f8ca0f873f002fc4d26b2a864238ae41fa866))
+
 ## [0.41.0](https://github.com/viamin/agent-harness/compare/agent-harness/v0.40.0...agent-harness/v0.41.0) (2026-09-25)
 
 

data/docs/provider-neutral-api-execution-contract.md

@@ -8,8 +8,8 @@ usage, and optional conversation-persistence work.
 ## Status and rollout boundary
 
 RDR-072's rollout guard was **docs-only** for the design phase. The normalized
-chat capability described in "Shipped normalized chat surface" below is now
-implemented. Other capabilities remain design contracts and each still needs
+chat and attempt-accounting capabilities described below are now implemented
+behind `AgentHarness::Api::ChatTransport`. Other capabilities remain design contracts and each still needs
 its own failing-first contract tests, implementation, release evidence, and
 downstream adoption evidence before a caller enables it.
 
@@ -336,6 +336,23 @@ observer so durable accounting can persist an attempt even when no message is
 created. Callers deduplicate on `attempt_id`. Cost identifies its source as
 provider-reported or harness-estimated; unknown cost remains `nil`.
 
+The observer receives `attempt_completed` after every physical request, with
+the same report later returned in `result[:attempts]`. Delivery is at least
+once across process recovery: consumers MUST enforce a unique key on
+`attempt_id` and treat a repeated report as an idempotent upsert, not another
+charge. `AttemptReport.from_h` restores JSON-decoded reports. Stored cost
+includes USD component amounts, `source` (`provider_reported` or `estimated`),
+and `priced_at`; restoration never consults current prices. Missing counts and
+prices remain absent/`nil`, while a reported zero remains zero. Usage may also
+include `cache_read_tokens`, `cache_write_tokens`, and `thinking_tokens`.
+
+Attempt events contain only identifiers, provider/model, outcome, normalized
+usage/cost, timestamps, and sanitized classified errors. They do not include
+prompts, messages, request headers, endpoints, or credentials. RubyLLM API chat
+requests are in this ledger. Existing CLI providers, `TextTransport`,
+`OpenAICompatibleTransport`, token trackers, embeddings, and other
+non-`Api::ChatTransport` paths remain outside it.
+
 Only errors classified `transient` are eligible for bounded request retry:
 connection failure, timeout before a partial stream, rate limit, server error,
 service unavailable, and overload. Authentication, authorization, billing,
@@ -569,6 +586,27 @@ types stay behind the harness boundary.
 
 ## Compatibility and release evidence
 
+### Attempt-accounting capability evidence
+
+- Release: pending the first published version containing issue #435; a Git
+  branch or tag alone is not downstream adoption evidence.
+- Scope: normalized chat through `Api::ChatTransport` for Anthropic Messages
+  and OpenAI Responses/Chat Completions (including compatible endpoints), with
+  request-local API-key authentication. It is stacked on the normalized chat
+  capability from #433.
+- Verification: the API contract specs cover request-local credentials,
+  endpoint/header isolation, error classification, bounded non-nested retry,
+  fallback, cancellation, partial usage, cache usage, repeated identity,
+  observer redaction, and JSON reload with preserved pricing. The full upstream
+  suite and lint run on the repository's supported Ruby environment.
+- Migration: no Rails tables or migrations are loaded or required. Paid and
+  agent-image consumer versions remain unverified and MUST NOT adopt this
+  capability until their integration suites record the exact released gem and
+  image versions.
+- Retained paths: CLI/subscription providers, legacy HTTP transports,
+  embeddings, and token trackers remain outside this ledger and require
+  separate migration issues.
+
 AgentHarness currently supports Ruby 3.2 and later and must remain usable as a
 plain Ruby gem. RubyLLM 2.0.0 itself supports Ruby 3.1.3 and later, but adding it
 would introduce Faraday, event-stream parsing, Schematist, Marcel, and Zeitwerk

data/lib/agent_harness/api/attempt_report.rb

@@ -0,0 +1,89 @@
+# frozen_string_literal: true
+
+require "time"
+
+module AgentHarness
+  module Api
+    # Serializable accounting facts for one physical provider request.
+    class AttemptReport
+      STATUSES = %i[succeeded failed cancelled partial].freeze
+      USAGE_KEYS = %i[input_tokens output_tokens cache_read_tokens cache_write_tokens thinking_tokens total_tokens].freeze
+      COST_KEYS = %i[input output cache_read cache_write thinking total].freeze
+
+      attr_reader :attributes
+
+      def self.from_h(attributes)
+        new(**symbolize(attributes))
+      end
+
+      def self.symbolize(value)
+        return value.to_h { |key, child| [key.to_sym, symbolize(child)] } if value.is_a?(Hash)
+        return value.map { |child| symbolize(child) } if value.is_a?(Array)
+
+        value
+      end
+      private_class_method :symbolize
+
+      def initialize(attempt_id:, request_id:, number:, provider:, model:, status:, started_at:, finished_at:,
+        usage: nil, cost: nil, provider_reported: false, error: nil)
+        @attributes = {
+          attempt_id: attempt_id.to_s, request_id: request_id.to_s, number: number,
+          provider: provider.to_sym, model: model&.to_s, status: status.to_sym,
+          started_at: timestamp(started_at), finished_at: timestamp(finished_at),
+          usage: normalize_usage(usage), cost: normalize_cost(cost),
+          provider_reported: provider_reported == true, error: error
+        }
+        validate!
+        deep_freeze(@attributes)
+      end
+
+      def to_h
+        deep_dup(attributes)
+      end
+
+      private
+
+      def timestamp(value)
+        value.respond_to?(:iso8601) ? value.iso8601(6) : Time.iso8601(value.to_s).utc.iso8601(6)
+      end
+
+      def normalize_usage(usage)
+        return unless usage
+
+        normalized = self.class.send(:symbolize, usage)
+        USAGE_KEYS.each_with_object({}) do |key, result|
+          result[key] = normalized[key] if normalized.key?(key)
+        end
+      end
+
+      def normalize_cost(cost)
+        return unless cost
+
+        normalized = self.class.send(:symbolize, cost)
+        amounts = COST_KEYS.to_h { |key| [key, normalized[key]] }
+        amounts.merge(currency: normalized.fetch(:currency, "USD"), source: normalized.fetch(:source).to_sym,
+          priced_at: timestamp(normalized.fetch(:priced_at)))
+      end
+
+      def validate!
+        raise ArgumentError, "attempt_id is required" if attributes[:attempt_id].empty?
+        raise ArgumentError, "request_id is required" if attributes[:request_id].empty?
+        raise ArgumentError, "number must be a positive integer" unless attributes[:number].is_a?(Integer) && attributes[:number].positive?
+        raise ArgumentError, "unknown attempt status" unless STATUSES.include?(attributes[:status])
+      end
+
+      def deep_freeze(value)
+        value.each_value { |child| deep_freeze(child) } if value.is_a?(Hash)
+        value.each { |child| deep_freeze(child) } if value.is_a?(Array)
+        value.freeze
+      end
+
+      def deep_dup(value)
+        return value.to_h { |key, child| [key, deep_dup(child)] } if value.is_a?(Hash)
+        return value.map { |child| deep_dup(child) } if value.is_a?(Array)
+
+        value
+      end
+    end
+  end
+end

data/lib/agent_harness/api/chat_transport.rb

@@ -2,6 +2,7 @@
 
 require "securerandom"
 require "ruby_llm"
+require_relative "attempt_report"
 require_relative "ruby_llm_chat_adapter"
 
 module AgentHarness
@@ -85,6 +86,18 @@ module AgentHarness
         end
 
         def perform_attempt(candidate)
+          accounting = nil
+          provider_usage = -> { !streamed_usage.nil? }
+          prepared_chat = @adapter.prepare(
+            candidate: candidate,
+            messages: request[:messages],
+            tools: request[:tools] || [],
+            max_output_tokens: request[:max_output_tokens],
+            temperature: request[:temperature],
+            timeout: request[:timeout],
+            on_accounting: ->(facts) { accounting = facts },
+            provider_usage: provider_usage
+          )
           attempt_id = @id_generator.call
           started_at = Time.now.utc
           emitted = false
@@ -99,7 +112,9 @@ module AgentHarness
             temperature: request[:temperature],
             stream: request[:stream] == true,
             timeout: request[:timeout],
-            cancellation: request[:cancellation]
+            cancellation: request[:cancellation],
+            on_accounting: ->(facts) { accounting = facts },
+            prepared_chat: prepared_chat
           ) do |event|
             event = normalize_stream_event(event)
             emitted = true if output_event?(event)
@@ -109,15 +124,21 @@ module AgentHarness
           end
           raise RubyLLM::CancelledError unless active?
 
-          success(candidate, attempt_id, started_at, adapter_result)
+          success(candidate, attempt_id, started_at, adapter_result, accounting)
         rescue ObserverError
           raise
         rescue => error
-          failure(candidate, attempt_id, started_at, classify(error), partial: emitted, usage: streamed_usage)
+          return failed_result(classify(error), candidate) unless attempt_id
+
+          failure(candidate, attempt_id, started_at, classify(error), partial: emitted,
+            accounting: accounting || {
+              usage: streamed_usage, provider_reported: !streamed_usage.nil?
+            })
         end
 
-        def success(candidate, attempt_id, started_at, adapter_result)
-          attempts << attempt_report(candidate, attempt_id, started_at, :succeeded, usage: adapter_result[:usage])
+        def success(candidate, attempt_id, started_at, adapter_result, accounting)
+          accounting ||= {usage: adapter_result[:usage], provider_reported: !adapter_result[:usage].nil?}
+          append_attempt(candidate, attempt_id, started_at, :succeeded, **accounting)
           result = base_result(candidate).merge(
             status: :succeeded,
             content: adapter_result[:content] || "",
@@ -130,10 +151,10 @@ module AgentHarness
           result
         end
 
-        def failure(candidate, attempt_id, started_at, error, partial:, usage:)
+        def failure(candidate, attempt_id, started_at, error, partial:, accounting:)
           status = failure_status(error, partial)
           error = error.merge(retryable: false) if partial
-          attempts << attempt_report(candidate, attempt_id, started_at, status, error: error, usage: usage)
+          append_attempt(candidate, attempt_id, started_at, status, error: error, **accounting)
           result = base_result(candidate).merge(
             status: status,
             content: partial ? @partial_content.dup : "",
@@ -178,8 +199,9 @@ module AgentHarness
         def backoff
           exponent = [attempts.length - 1, 0].max
           delay = retry_config[:base_delay_seconds] * (2**exponent)
+          delay = [delay, @last_error[:retry_after_seconds].to_f].max
           cap = retry_config[:max_delay_seconds]
-          remaining = cap&.positive? ? [delay, cap].min : delay
+          remaining = cap.nil? ? delay : [delay, cap].min
           while remaining.positive? && active?
             interval = [remaining, 0.05].min
             @sleeper.call(interval)
@@ -227,7 +249,8 @@ module AgentHarness
 
         def normalize_tool_calls(tool_calls)
           Array(tool_calls).map do |call|
-            call.merge(id: tool_id(call[:provider_id]), status: :completed)
+            id = call[:provider_id].nil? ? @id_generator.call : tool_id(call[:provider_id])
+            call.merge(id: id, status: :completed)
           end
         end
 
@@ -265,8 +288,10 @@ module AgentHarness
             usage: aggregate_usage, error: error)
         end
 
-        def attempt_report(candidate, attempt_id, started_at, status, error: nil, usage: nil)
-          {
+        def append_attempt(candidate, attempt_id, started_at, status, error: nil, usage: nil, cost: nil,
+          provider_reported: false)
+          cost = cost.merge(priced_at: Time.now.utc) if cost
+          report = AttemptReport.new(
             attempt_id: attempt_id,
             request_id: request[:request_id],
             number: attempts.length + 1,
@@ -276,20 +301,22 @@ module AgentHarness
             started_at: started_at.iso8601(6),
             finished_at: Time.now.utc.iso8601(6),
             usage: usage,
-            cost: nil,
-            provider_reported: false,
+            cost: cost,
+            provider_reported: provider_reported,
             error: error
-          }
+          ).attributes
+          attempts << report
+          emit(:attempt_completed, attempt_id:, attempt: report)
         end
 
         def aggregate_usage
           reports = attempts.filter_map { |attempt| attempt[:usage] }
           return if reports.empty?
 
-          %i[input_tokens output_tokens total_tokens].to_h do |key|
+          %i[input_tokens output_tokens cache_read_tokens cache_write_tokens thinking_tokens total_tokens].to_h do |key|
             values = reports.filter_map { |usage| usage[key] }
             [key, values.empty? ? nil : values.sum]
-          end
+          end.compact
         end
 
         def classify(error)
@@ -402,10 +429,23 @@ module AgentHarness
       end
 
       def self.payload(error, category, code, retryable)
-        {category: category, code: code, retryable: retryable, message: safe_message(category, code)}
+        payload = {category: category, code: code, retryable: retryable, message: safe_message(category, code)}
+        retry_after = retry_after_seconds(error)
+        payload[:retry_after_seconds] = retry_after if retry_after
+        payload
       end
       private_class_method :payload
 
+      def self.retry_after_seconds(error)
+        return unless error.respond_to?(:response) && error.response
+
+        headers = error.response.respond_to?(:headers) ? error.response.headers : error.response[:response_headers]
+        value = headers&.find { |key, _| key.to_s.casecmp?("retry-after") }&.last
+        delay = Float(value, exception: false)
+        delay if delay&.finite? && delay >= 0
+      end
+      private_class_method :retry_after_seconds
+
       def self.safe_message(category, code)
         "Chat request failed (#{category}/#{code})"
       end

data/lib/agent_harness/api/ruby_llm_chat_adapter.rb

@@ -14,18 +14,31 @@ module AgentHarness
         anthropic: %i[anthropic_api_key anthropic_api_base],
         openai: %i[openai_api_key openai_api_base]
       }.freeze
+      DEFAULT_REQUEST_TIMEOUT = 300
       OPENAI_UNSUPPLIED_CONFIG = %i[openai_organization_id openai_project_id openai_use_system_role].freeze
 
-      def call(candidate:, messages:, tools:, max_output_tokens:, temperature:, stream:, timeout:, cancellation:, &on_event)
-        context = build_context(candidate, timeout)
-        chat = context.chat(model: candidate[:model], provider: candidate[:provider], protocol: ruby_llm_protocol(candidate),
-          assume_model_exists: true)
-        configure_chat(chat, candidate, messages, tools, max_output_tokens, temperature)
-        response = generate(chat, stream, cancellation, &on_event)
+      def call(candidate:, messages:, tools:, max_output_tokens:, temperature:, stream:, timeout:, cancellation:,
+        on_accounting: nil, prepared_chat: nil, &on_event)
+        provider_usage_reported = false
+        chat = prepared_chat || prepare(candidate:, messages:, tools:, max_output_tokens:, temperature:, timeout:,
+          on_accounting:, provider_usage: -> { provider_usage_reported })
+        response = generate(chat, stream, cancellation) do |event|
+          provider_usage_reported = true if event[:type] == :usage_updated
+          on_event&.call(event)
+        end
         emit_completed_tool_calls(response, &on_event) if stream
         normalize_response(response)
       end
 
+      def prepare(candidate:, messages:, tools:, max_output_tokens:, temperature:, timeout:, on_accounting: nil,
+        provider_usage: -> { false })
+        context = build_context(candidate, timeout, on_accounting, provider_usage)
+        context.chat(model: candidate[:model], provider: candidate[:provider], protocol: ruby_llm_protocol(candidate),
+          assume_model_exists: true).tap do |chat|
+          configure_chat(chat, candidate, messages, tools, max_output_tokens, temperature)
+        end
+      end
+
       # RubyLLM keys streamed tool-call chunks by a stream index while
       # continuation chunks carry no call id, so correlation state must live
       # between chunks. Also remembers the latest cumulative token counts so
@@ -69,7 +82,7 @@ module AgentHarness
         candidate[:protocol]
       end
 
-      def build_context(candidate, timeout)
+      def build_context(candidate, timeout, on_accounting, provider_usage)
         provider = candidate[:provider].to_sym
         config_keys = PROVIDER_CONFIG[provider]
         raise RubyLLM::ConfigurationError, "Unsupported chat provider: #{provider}" unless config_keys
@@ -80,10 +93,51 @@ module AgentHarness
           config.public_send("#{config_keys[1]}=", candidate[:endpoint])
           clear_unsupplied_openai_config(config) if provider == :openai
           config.max_retries = 0
+          config.instrumenter = UsageInstrumenter.new(on_accounting, provider_usage) if on_accounting
           apply_timeout(config, timeout)
         end
       end
 
+      # Retains only accounting instrumentation; other events may contain
+      # request content and are intentionally discarded.
+      class UsageInstrumenter
+        def initialize(callback, provider_usage)
+          @callback = callback
+          @provider_usage = provider_usage
+        end
+
+        def instrument(name, payload)
+          @callback.call(normalize(payload)) if name == "usage.ruby_llm"
+          yield(payload) if block_given?
+        end
+
+        private
+
+        def normalize(payload)
+          tokens = payload.fetch(:tokens)
+          cost = payload.fetch(:cost)
+          usage = tokens.to_h
+          usage[:total_tokens] = tokens.input + tokens.output if tokens.input && tokens.output
+          {
+            usage: usage,
+            cost: cost.total.nil? ? nil : cost.to_h.merge(source: cost_source(tokens), currency: "USD"),
+            provider_reported: provider_reported?(payload, usage)
+          }
+        end
+
+        def provider_reported?(payload, usage)
+          return false if usage.empty?
+          return true unless payload[:status]&.to_sym == :failed
+          return true if @provider_usage.call
+
+          usage.values.compact.any?(&:positive?)
+        end
+
+        def cost_source(tokens)
+          tokens.reported_cost.nil? ? :estimated : :provider_reported
+        end
+      end
+
       def clear_unsupplied_openai_config(config)
         OPENAI_UNSUPPLIED_CONFIG.each { |key| config.public_send("#{key}=", nil) }
       end
@@ -93,8 +147,7 @@ module AgentHarness
           raise UnsupportedOptionError, "RubyLLM does not support request-local connect timeouts"
         end
 
-        seconds = timeout&.dig(:read_seconds)
-        config.request_timeout = seconds if seconds
+        config.request_timeout = timeout&.dig(:read_seconds) || DEFAULT_REQUEST_TIMEOUT
       end
 
       def configure_chat(chat, candidate, messages, tools, max_output_tokens, temperature)
@@ -262,11 +315,9 @@ module AgentHarness
       end
 
       def normalize_usage(tokens)
-        input = tokens&.input
-        output = tokens&.output
-        return unless input || output
+        return unless tokens&.to_h&.any?
 
-        {input_tokens: input, output_tokens: output, total_tokens: (input && output) ? input + output : nil}
+        tokens.to_h.merge(total_tokens: (tokens.input && tokens.output) ? tokens.input + tokens.output : nil)
       end
     end
   end

data/lib/agent_harness/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module AgentHarness
-  VERSION = "0.41.0"
+  VERSION = "0.42.0"
 end

data/lib/agent_harness.rb

@@ -471,6 +471,7 @@ require_relative "agent_harness/error_taxonomy"
 require_relative "agent_harness/text_transport"
 require_relative "agent_harness/openai_compatible_transport"
 require_relative "agent_harness/conversation"
+require_relative "agent_harness/api/attempt_report"
 require_relative "agent_harness/api/chat_transport"
 require_relative "agent_harness/quota_status"
 require_relative "agent_harness/authentication"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: agent-harness
 version: !ruby/object:Gem::Version
-  version: 0.41.0
+  version: 0.42.0
 platform: ruby
 authors:
 - Bart Agapinan
@@ -135,6 +135,7 @@ files:
 - json-2.18.1.gem
 - lib/agent-harness.rb
 - lib/agent_harness.rb
+- lib/agent_harness/api/attempt_report.rb
 - lib/agent_harness/api/chat_transport.rb
 - lib/agent_harness/api/ruby_llm_chat_adapter.rb
 - lib/agent_harness/authentication.rb