llm_logs 0.5.1 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a687cddb64dc51d51cdf1bfa833833523b54194054ae32fe631354f187c9f256
-  data.tar.gz: ec0375795a8d27afa1266c1a8ebacd0984395fd8d41753c6ebd4c2658cfef026
+  metadata.gz: 5d03f4e8b8e5607855655d0d756f0c906395e42e808d3bfe255ced558a639d5e
+  data.tar.gz: bb172cb33c5d864ec6dde523cfe64e14c9fe4783a0a4b9c3e856dc58717b0cb3
 SHA512:
-  metadata.gz: fe87b0ed824568ba3c1104e600fdc670a43919c5b061159a7e1e94bffe59257272b74914fd83256633236fd02da3b1ae99efd240ea8a6e645c93880e07d3a7af
-  data.tar.gz: 81986848eec310e3c2360ceecfbe894a58fc5836adc1a5a6832c90a2c60abbc72898521c6a7b44cf9a1aa118c0837aea861d2e919aeb4fd33b5d83c251e6b22d
+  metadata.gz: 904aefd977972ab4bb67534972656d9abd55b3d4120c2ed99f961b661b5dfc9c1d2f7861d1afe9b74b8b6f175d86d9a5315ac2a468488ae58f23e737ef8d7c4c
+  data.tar.gz: 3e8e0f902575f2e2cbaf656965267343bf2124f781de41c036fe426d1bb6382551bbaede74f5f794ab52cdcaedba858c8adc6452554728ffbcf28786ccd0f53a

data/README.md

@@ -209,16 +209,7 @@ Running the task creates missing prompts, updates metadata, and creates a new pr
 
 Send latency-insensitive requests through a provider's Batch API for roughly half the cost. LlmLogs persists each request, groups pending requests into a provider batch, reconciles results, and records a trace per request — so batched work shows up in the dashboard alongside synchronous calls.
 
-Two batch backends are supported, selected **per model**:
-
-- **[OpenAI Responses Batch API](https://platform.openai.com/docs/guides/batch)** via [`ruby_llm-responses_api`](https://rubygems.org/gems/ruby_llm-responses_api) — the default for OpenAI models.
-- **[AWS Bedrock Batch API](#aws-bedrock-batches)** (`CreateModelInvocationJob`) for Anthropic Claude models.
-
-Add the OpenAI provider to your app's Gemfile:
-
-```ruby
-gem "ruby_llm-responses_api"
-```
+Batching runs on the **[AWS Bedrock Batch API](#aws-bedrock-batches)** (`CreateModelInvocationJob`) for Anthropic Claude models. Models Bedrock does not serve are not batchable; run them synchronously.
 
 ### Enqueue a Request
 
@@ -227,10 +218,10 @@ Requests are persisted immediately and grouped by `purpose` + `model` when submi
 ```ruby
 LlmLogs::Batch.enqueue(
   purpose: "chat_summary",
-  model: "gpt-4.1-mini",
+  model: "anthropic.claude-haiku-4-5-20251001-v1:0",
   instructions: "Summarize the conversation in two sentences.",
   input: conversation_text,
-  schema: SummarySchema,          # optional RubyLLM::Schema for structured output
+  schema: SUMMARY_SCHEMA,          # optional JSON schema Hash for structured output
   routing: { conversation_id: 42 }, # your keys, echoed into the trace metadata
   temperature: 0.2                  # optional
 )
@@ -247,7 +238,8 @@ Register one handler per `purpose`. The gem owns the batch lifecycle; your app o
 LlmLogs.register_batch_handler("chat_summary", ChatSummaryHandler.new)
 
 class ChatSummaryHandler
-  # Called once a request succeeds. `message` is the RubyLLM::Message.
+  # Called once a request succeeds. `message` is a `LlmLogs::Batch::Adapters::Bedrock::Result`
+  # (content, input_tokens, output_tokens, model_id).
   def call(request, message)
     Conversation.find(request.routing["conversation_id"])
       .update!(summary: message.content)
@@ -302,7 +294,7 @@ LlmLogs.configuration.bedrock_batch = LlmLogs::Configuration::BedrockBatch.new(
 LlmLogs.register_batch_adapter(:bedrock, LlmLogs::Batch::Adapters::Bedrock.new)
 ```
 
-Provider selection is per model: `LlmLogs::Batch.batch_provider_for(model)` returns `:bedrock` when the adapter is registered and `model_matcher` matches, otherwise the OpenAI backend when the model resolves there, otherwise `nil` (not batchable — run it synchronously). Bedrock enforces a **minimum records per job**, so check `LlmLogs::Batch.min_records_for(model)` and fall back to a synchronous call when a batch would be under the floor.
+Provider selection is per model: `LlmLogs::Batch.batch_provider_for(model)` returns `:bedrock` when the adapter is registered and `model_matcher` matches, otherwise `nil` (not batchable — run it synchronously). Bedrock enforces a **minimum records per job**, so check `LlmLogs::Batch.min_records_for(model)` and fall back to a synchronous call when a batch would be under the floor.
 
 The adapter builds its AWS clients from the ambient credential chain by default; inject your own to authenticate explicitly:
 
@@ -330,20 +322,31 @@ Browse traces and manage prompts at `/llm_logs`.
 ```ruby
 LlmLogs.setup do |config|
   config.enabled = true                                      # master switch for logging
-  config.auto_instrument = true                              # auto-prepend on RubyLLM::Chat
+  config.auto_instrument = true                              # subscribe to ruby_llm notifications (needs RubyLLM.config.instrumenter = ActiveSupport::Notifications)
   config.retention_days = 30                                 # for future cleanup job
   config.prompts_source_path = Rails.root.join("db/data/prompts")
   config.prompt_subfolders = %w[skills fragments templates]
   config.batch_enabled = true                                # enable the batch API integration
-  config.batch_provider = :openai_responses                  # default (OpenAI) backend; Bedrock is registered separately (see Batches)
   config.page_size = 50                                      # rows per page on all index pages
 end
 ```
 
+## ruby_llm 2.0 integration
+
+With `auto_instrument` on, LlmLogs subscribes to ruby_llm's `chat.ruby_llm` and `tool_call.ruby_llm` notifications: one `llm` span per provider round and one tool span per tool call, all siblings under the trace. Tokens and cost come from each round's usage entries. This requires `RubyLLM.config.instrumenter` to be `ActiveSupport::Notifications`; the Railtie sets that default, and `LlmLogs::Instrumentation::RubyLlmChat.install!` sets it when unset.
+
+### Bedrock fixes in ruby_llm
+
+llm_logs no longer patches ruby_llm. The Bedrock bugs it used to patch are fixed in ruby_llm 2.1: [crmne/ruby_llm#1024](https://github.com/crmne/ruby_llm/pull/1024) (re-sign each attempt, retry TLS errors), [#1025](https://github.com/crmne/ruby_llm/pull/1025) (`reasoning_config` for GPT, adaptive thinking for adaptive-only Claude) and [ab4a4f06](https://github.com/crmne/ruby_llm/commit/ab4a4f060fc4760053105b92500c042c52c3d968) (reasoning replays only to the model that produced it). llm_logs 0.7.0 requires ruby_llm 2.1 or later. Two behaviours differ from the old patches:
+
+- GPT effort goes out as `reasoning_config` only when the model's registry entry publishes that schema. ruby_llm's registry does for the GPT-6 and GPT-5.6 ids; a catalog entry of your own for a GPT id needs `metadata.converse.additionalRequestFieldsSchema`, or the effort goes out as `reasoning_effort`, which Bedrock rejects.
+- Reasoning replays only to the exact model that produced it, not to any model of the same family.
+
 ## Requirements
 
 - Rails 8.0+
 - Ruby 3.3+
+- ruby_llm 2.1+
 - PostgreSQL
 
 ## License

data/app/models/llm_logs/batch/submitter.rb

@@ -1,12 +1,13 @@
 module LlmLogs
   class Batch
-    # Groups pending BatchRequests of one purpose+model into a single OpenAI batch via
-    # ruby_llm-responses_api. To prevent two concurrent FlushJobs from double-submitting
-    # the same requests, it first CLAIMS the pending rows in a `FOR UPDATE SKIP LOCKED`
-    # transaction (assigning them to a placeholder Batch with no openai_batch_id, which
-    # flips them out of the `pending` scope and which PollJob ignores). It then submits to
-    # OpenAI and records the batch id. If submission fails, the claim is released (requests
-    # return to `pending`) and the placeholder batch is dropped, so the work retries next flush.
+    # Groups pending BatchRequests of one purpose+model into a single provider batch (Bedrock).
+    # To keep two concurrent FlushJobs from double-submitting the same requests, it first
+    # CLAIMS the pending rows in a `FOR UPDATE SKIP LOCKED` transaction, assigning them to a
+    # placeholder Batch with no provider batch id (stored in the legacy `openai_batch_id`
+    # column), which flips them out of the `pending` scope and which PollJob ignores. It then
+    # submits the batch and records its id. If submission fails, the claim is released
+    # (requests return to `pending`) and the placeholder batch is dropped, so the work retries
+    # on the next flush.
     class Submitter
       def initialize(purpose:, model:, metadata: {})
         @purpose = purpose

data/app/models/llm_logs/batch/trace_recorder.rb

@@ -27,23 +27,35 @@ module LlmLogs
             output: { "content" => span.serialize_content(message.content) },
             input_tokens: message.input_tokens,
             output_tokens: message.output_tokens,
-            cost: compute_cost(message)
+            cost: compute_cost(message, request: request, provider: provider)
           )
           span.finish
         end
         trace
       end
 
-      def compute_cost(message)
-        model_info = RubyLLM.models.find(message.model_id)
-        return nil unless model_info&.input_price_per_million && model_info&.output_price_per_million
+      # Prices the request at the submitted model's standard rates for this provider (the
+      # Bedrock id, e.g. us.anthropic.claude-haiku-4-5-...), falling back to the id the result
+      # reports (a native Anthropic id resolves to the anthropic provider's rates). nil when
+      # neither is in the registry or the registry has no price for the tokens used.
+      def compute_cost(message, request:, provider:)
+        model = pricing_model(request.model, provider) || pricing_model(message.model_id, nil)
+        return nil unless model
 
-        raw = (message.input_tokens.to_f * model_info.input_price_per_million +
-               message.output_tokens.to_f * model_info.output_price_per_million) / 1_000_000
-        (raw * BATCH_COST_MULTIPLIER).round(6)
+        tokens = RubyLLM::Tokens.new(input: message.input_tokens.to_i, output: message.output_tokens.to_i)
+        total = model.cost_for(tokens).total
+        total && (total * BATCH_COST_MULTIPLIER).round(6)
       rescue StandardError
         nil
       end
+
+      def pricing_model(model_id, provider)
+        return nil if model_id.blank?
+
+        provider.present? ? RubyLLM.models.find(model_id, provider: provider) : RubyLLM.models.find(model_id)
+      rescue RubyLLM::ModelNotFoundError
+        nil
+      end
     end
   end
 end

data/app/models/llm_logs/batch.rb

@@ -66,14 +66,10 @@ module LlmLogs
       !batch_provider_for(model).nil?
     end
 
-    # Which batch provider (if any) serves this model. Bedrock wins for Claude models when
-    # the Bedrock adapter is configured; otherwise fall back to the OpenAI provider when the
-    # model resolves there; otherwise nil (run synchronously).
+    # Which batch provider (if any) serves this model: :bedrock when the Bedrock adapter is
+    # registered and its model_matcher matches, otherwise nil (run synchronously).
     def self.batch_provider_for(model)
-      return :bedrock if bedrock_serves?(model)
-      return :openai_responses if openai_serves?(model)
-
-      nil
+      bedrock_serves?(model) ? :bedrock : nil
     end
 
     # The Bedrock minimum records-per-job floor for this model (0 when Bedrock does not serve it).
@@ -89,25 +85,5 @@ module LlmLogs
       matcher = config.model_matcher
       matcher.respond_to?(:call) ? matcher.call(model.to_s) : matcher.match?(model.to_s)
     end
-
-    def self.openai_serves?(model)
-      return false unless defined?(RubyLLM::Providers::OpenAIResponses)
-
-      servable_by_batch_provider?(model)
-    end
-
-    # The batch path submits via RubyLLM.batch(provider: batch_provider). A model is
-    # only batchable if that provider can actually serve it -- i.e. the model resolves
-    # under batch_provider. Models that belong to a different provider (e.g. Bedrock /
-    # Anthropic) don't resolve there, so they return false and the caller runs them
-    # synchronously instead of enqueueing work that would fail at submit time.
-    def self.servable_by_batch_provider?(model)
-      RubyLLM::Models.resolve(
-        model, provider: LlmLogs.batch_provider, assume_exists: false, config: RubyLLM.config
-      )
-      true
-    rescue RubyLLM::ModelNotFoundError
-      false
-    end
   end
 end

data/app/models/llm_logs/span.rb

@@ -19,11 +19,23 @@ module LlmLogs
       LlmLogs::Tracer.current_span = parent_span
     end
 
+    # +message+ is a RubyLLM 2.x Message (or anything with +content+ and +tokens+).
     def record_response(message)
       self.output = { content: serialize_content(message.content) }
-      self.input_tokens = message.input_tokens
-      self.output_tokens = message.output_tokens
-      self.cached_tokens = message.cached_tokens
+      record_tokens(message.tokens)
+    end
+
+    # Copies a RubyLLM::Tokens onto the span. +input+ is non-cached input (RubyLLM 2.x no
+    # longer subtracts cache buckets from it); cache writes and thinking have no column and
+    # go to metadata.
+    def record_tokens(tokens)
+      return unless tokens
+
+      self.input_tokens = tokens.input
+      self.output_tokens = tokens.output
+      self.cached_tokens = tokens.cache_read
+      set_attribute("cache_write_tokens", tokens.cache_write) unless tokens.cache_write.nil?
+      set_attribute("thinking_tokens", tokens.thinking) unless tokens.thinking.nil?
     end
 
     # Structured (schema) responses arrive as a Hash/Array; keep them as-is so the

data/lib/llm_logs/configuration.rb

@@ -3,7 +3,7 @@ module LlmLogs
     BedrockBatch = Struct.new(:role_arn, :s3_bucket, :s3_prefix, :min_records, :model_matcher, :region, keyword_init: true)
 
     attr_accessor :enabled, :auto_instrument, :retention_days, :prompts_source_path, :prompt_subfolders,
-                  :batch_enabled, :batch_provider, :page_size, :bedrock_batch, :reasoning_effort_options
+                  :batch_enabled, :page_size, :bedrock_batch, :reasoning_effort_options
 
     def initialize
       @enabled             = true
@@ -12,7 +12,6 @@ module LlmLogs
       @prompts_source_path = nil
       @prompt_subfolders   = %w[skills fragments templates]
       @batch_enabled       = true
-      @batch_provider      = :openai_responses
       @page_size           = 50
       @bedrock_batch       = nil
       # Effort tiers offered in the prompt form. Union of what current providers

data/lib/llm_logs/engine.rb

@@ -6,7 +6,7 @@ module LlmLogs
       ActiveSupport.on_load(:active_record) do
         if LlmLogs.auto_instrument && defined?(RubyLLM::Chat)
           require "llm_logs/instrumentation/ruby_llm_chat"
-          RubyLLM::Chat.prepend(LlmLogs::Instrumentation::RubyLlmChat)
+          LlmLogs::Instrumentation::RubyLlmChat.install!
         end
       end
     end

data/lib/llm_logs/instrumentation/ruby_llm_chat.rb

@@ -1,105 +1,213 @@
 module LlmLogs
   module Instrumentation
+    # Records RubyLLM 2.x chat activity as llm_logs spans by subscribing to the
+    # events RubyLLM emits for observability adapters (docs/_advanced/instrumentation.md):
+    #
+    #   chat.ruby_llm      -- one per model request (RubyLLM::Chat#generate_once): an "llm" span
+    #                         per provider round. Wraps every transport retry of that round, so a
+    #                         round that fails after its retries surfaces as :exception_object.
+    #                         With fallbacks, each model tried gets its own span.
+    #   tool_call.ruby_llm -- one per local tool execution (RubyLLM::Chat#execute_tool): a "tool" span.
+    #
+    # Rounds and tools are siblings under the current span/trace:
+    #   trace -> llm (round 1, asks for a tool) -> tool.x -> llm (round 2, final answer)
+    #
+    # The subscription is global (every RubyLLM::Chat, including acts_as_chat's to_llm and
+    # RubyLLM::Agent), but only fires while RubyLLM's instrumenter is ActiveSupport::Notifications,
+    # which RubyLLM's Railtie sets by default.
     module RubyLlmChat
-      def complete(&block)
-        return super unless LlmLogs.enabled?
-
-        span = LlmLogs::Tracer.start_span(
-          name: "chat.complete",
-          span_type: "llm",
-          model: @model&.id,
-          provider: @model&.provider,
-          input: messages.map { |m| { role: m.role, content: llm_logs_serialize_content(m.content) } }
-        )
-
-        messages_before = messages.size
-
-        begin
-          result = super(&block)
-          span.record_response(result)
-          span.cost = llm_logs_compute_cost(result)
-          result
-        rescue => e
-          span.record_error(e)
-          llm_logs_capture_partial_tokens(span, messages_before)
-          raise
-        ensure
-          span.finish
+      SPAN_KEY = :llm_logs_span
+      USAGE_START_KEY = :llm_logs_usage_start
+
+      class << self
+        def install!
+          return if installed?
+
+          RubyLLM.config.instrumenter ||= ActiveSupport::Notifications
+          @subscribers = [
+            ActiveSupport::Notifications.subscribe("chat.ruby_llm", ChatSubscriber.new),
+            ActiveSupport::Notifications.subscribe("tool_call.ruby_llm", ToolSubscriber.new)
+          ]
         end
-      end
 
-      private
+        def uninstall!
+          Array(@subscribers).each { |subscriber| ActiveSupport::Notifications.unsubscribe(subscriber) }
+          @subscribers = nil
+        end
 
-      def llm_logs_serialize_content(content)
-        content.is_a?(Hash) || content.is_a?(Array) ? content.to_json : content.to_s
-      end
+        def installed?
+          !@subscribers.nil?
+        end
 
-      def llm_logs_serialize_tool_result(result)
-        if result.is_a?(Hash)
-          result
-        elsif defined?(RubyLLM::Tool::Halt) && result.is_a?(RubyLLM::Tool::Halt)
-          { halted: result.message }
-        else
-          { result: result.to_s }
+        # Instrumentation must never break the LLM call it observes. ActiveSupport re-raises
+        # subscriber exceptions into the instrumented block, so every hook rescues.
+        def guard
+          yield
+        rescue StandardError => e
+          Rails.logger&.error("[llm_logs] instrumentation error: #{e.class}: #{e.message}")
+          nil
         end
       end
 
-      def llm_logs_capture_partial_tokens(span, messages_before)
-        assistant_msg = messages[messages_before..].find { |m| m.role == :assistant }
-        return unless assistant_msg&.input_tokens
+      # Serializers shared by both subscribers.
+      module Serialize
+        module_function
 
-        span.input_tokens = assistant_msg.input_tokens
-        span.output_tokens = assistant_msg.output_tokens
-        span.cached_tokens = assistant_msg.cached_tokens
-        span.cost = llm_logs_compute_cost(assistant_msg)
-      rescue StandardError
-        nil
-      end
+        def messages(list)
+          Array(list).map { |message| message_entry(message) }
+        end
+
+        def message_entry(message)
+          entry = {role: message.role, content: message.content}
+          calls = tool_calls(message.tool_calls)
+          entry[:tool_calls] = calls if calls
+          entry[:tool_call_id] = message.tool_call_id if message.tool_call_id
+          entry
+        end
+
+        def tool_calls(calls)
+          return nil if calls.nil? || calls.empty?
+
+          calls.values.map { |call| {id: call.id, name: call.name, arguments: call.arguments} }
+        end
 
-      def llm_logs_compute_cost(message)
-        model_info = llm_logs_resolve_pricing_model
-        return unless model_info
+        # Structured (schema) answers are JSON text in 2.x; store the parsed Hash so the
+        # UI renders nested fields instead of an escaped string.
+        def response(message, schema:)
+          content = message.content
+          content = parse_json(content) if schema && content.is_a?(String) && !content.empty?
+          output = {content: content}
+          calls = tool_calls(message.tool_calls)
+          output[:tool_calls] = calls if calls
+          output[:thinking] = message.thinking.text if message.thinking&.text.present?
+          output
+        end
 
-        input_price = model_info.input_price_per_million
-        output_price = model_info.output_price_per_million
-        return unless input_price && output_price
+        def parse_json(text)
+          JSON.parse(text)
+        rescue JSON::ParserError
+          text
+        end
 
-        (message.input_tokens.to_f * input_price + message.output_tokens.to_f * output_price) / 1_000_000
+        def tool_result(result)
+          case result
+          when Hash then result
+          when Array then {result: result}
+          else {result: result.to_s}
+          end
+        end
       end
 
-      def llm_logs_resolve_pricing_model
-        return @model if @model&.input_price_per_million
+      class ChatSubscriber
+        def start(_name, _id, payload)
+          RubyLlmChat.guard do
+            next unless LlmLogs.enabled?
+
+            chat = payload[:chat]
+            payload[USAGE_START_KEY] = chat.usage_entries.length if chat.respond_to?(:usage_entries)
+            payload[SPAN_KEY] = LlmLogs::Tracer.start_span(
+              name: "chat.complete",
+              span_type: "llm",
+              model: payload[:model],
+              provider: payload[:provider],
+              input: Serialize.messages(payload[:input_messages]),
+              metadata: request_metadata(payload)
+            )
+          end
+        end
 
-        RubyLLM.models.find(@model.id) if @model&.id
-      rescue StandardError
-        nil
+        def finish(_name, _id, payload)
+          span = payload.delete(SPAN_KEY)
+          return unless span
+
+          RubyLlmChat.guard do
+            if (error = payload[:exception_object])
+              span.record_error(error)
+            else
+              span.output = Serialize.response(payload[:response], schema: payload[:schema])
+              span.set_attribute("finish_reason", payload[:response].finish_reason&.to_s)
+              response_model = payload[:response_model]
+              span.set_attribute("response_model", response_model) if response_model && response_model != payload[:model]
+            end
+            record_usage(span, payload)
+          end
+        ensure
+          RubyLlmChat.guard { span&.finish }
+        end
+
+        private
+
+        def request_metadata(payload)
+          {
+            "streaming" => payload[:streaming],
+            "tools" => Array(payload[:tools]).map(&:to_s),
+            "temperature" => payload[:temperature],
+            "thinking" => thinking_metadata(payload[:thinking]),
+            "schema" => payload[:schema] && payload[:schema][:name]
+          }.compact
+        end
+
+        def thinking_metadata(thinking)
+          return nil unless thinking
+
+          {"effort" => thinking.effort&.to_s, "budget" => thinking.budget, "enabled" => thinking.enabled}.compact.presence
+        end
+
+        # Tokens and cost of every transport attempt this round made (retries included), read
+        # from the chat's usage ledger. Falls back to the event's tokens/cost when the ledger is
+        # unavailable. A failed attempt whose usage is unknown (e.g. a timeout that may have
+        # been billed) is left out of the cost and flagged with cost_complete: false.
+        def record_usage(span, payload)
+          entries = round_usage_entries(payload)
+          if entries
+            tokens = RubyLLM::Tokens.aggregate(entries.map(&:tokens))
+            cost = RubyLLM::Cost.aggregate(entries.map(&:cost))
+            span.set_attribute("attempts", entries.size) if entries.size > 1
+            span.set_attribute("cost_complete", false) unless entries.all?(&:cost_available?)
+          else
+            tokens = payload[:tokens]
+            cost = payload[:cost]
+          end
+          span.record_tokens(tokens)
+          span.cost = cost&.total
+        end
+
+        def round_usage_entries(payload)
+          start = payload[USAGE_START_KEY]
+          chat = payload[:chat]
+          return nil unless start && chat.respond_to?(:usage_entries)
+
+          chat.usage_entries.drop(start)
+        end
       end
 
-      public
-
-      def execute_tool(tool_call)
-        return super unless LlmLogs.enabled?
-
-        span = LlmLogs::Tracer.start_span(
-          name: "tool.#{tool_call.name}",
-          span_type: "tool",
-          input: tool_call.arguments,
-          metadata: { tool_name: tool_call.name }
-        )
-
-        begin
-          result = super
-          span.output = llm_logs_serialize_tool_result(result)
-          span.set_attribute("tool.halted", true) if defined?(RubyLLM::Tool::Halt) && result.is_a?(RubyLLM::Tool::Halt)
-          # Convert hash/array results to JSON so RubyLLM stores clean JSON
-          # in message content, not Ruby hash syntax from Hash#to_s
-          result = result.to_json if result.is_a?(Hash) || result.is_a?(Array)
-          result
-        rescue => e
-          span.record_error(e)
-          raise
+      class ToolSubscriber
+        def start(_name, _id, payload)
+          RubyLlmChat.guard do
+            next unless LlmLogs.enabled?
+
+            payload[SPAN_KEY] = LlmLogs::Tracer.start_span(
+              name: "tool.#{payload[:tool_name]}",
+              span_type: "tool",
+              input: payload[:tool_arguments],
+              metadata: {tool_name: payload[:tool_name].to_s, tool_call_id: payload[:tool_call_id]}.compact
+            )
+          end
+        end
+
+        def finish(_name, _id, payload)
+          span = payload.delete(SPAN_KEY)
+          return unless span
+
+          RubyLlmChat.guard do
+            if (error = payload[:exception_object])
+              span.record_error(error)
+            else
+              span.output = Serialize.tool_result(payload[:result])
+            end
+          end
         ensure
-          span.finish
+          RubyLlmChat.guard { span&.finish }
         end
       end
     end

data/lib/llm_logs/version.rb

@@ -1,3 +1,3 @@
 module LlmLogs
-  VERSION = "0.5.1"
+  VERSION = "0.7.0"
 end

data/lib/llm_logs.rb

@@ -54,10 +54,6 @@ module LlmLogs
     configuration.batch_enabled
   end
 
-  def self.batch_provider
-    configuration.batch_provider
-  end
-
   def self.bedrock_batch
     configuration.bedrock_batch
   end
@@ -71,7 +67,7 @@ module LlmLogs
   end
 
   def self.batch_adapters
-    @batch_adapters ||= {openai_responses: LlmLogs::Batch::Adapters::OpenaiResponses.new}
+    @batch_adapters ||= {}
   end
 
   def self.register_batch_adapter(provider, adapter)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm_logs
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.7.0
 platform: ruby
 authors:
 - Anton
@@ -97,30 +97,36 @@ dependencies:
   name: ruby_llm
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.16'
-  type: :development
+        version: '2.1'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '3'
+  type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '2.1'
+    - - "<"
       - !ruby/object:Gem::Version
-        version: '1.16'
+        version: '3'
 - !ruby/object:Gem::Dependency
-  name: ruby_llm-responses_api
+  name: aws-eventstream
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: webmock
   requirement: !ruby/object:Gem::Requirement
@@ -186,10 +192,8 @@ files:
 - app/models/llm_logs/application_record.rb
 - app/models/llm_logs/batch.rb
 - app/models/llm_logs/batch/adapters/bedrock.rb
-- app/models/llm_logs/batch/adapters/openai_responses.rb
 - app/models/llm_logs/batch/handler_registry.rb
 - app/models/llm_logs/batch/reconciler.rb
-- app/models/llm_logs/batch/schema_format.rb
 - app/models/llm_logs/batch/submitter.rb
 - app/models/llm_logs/batch/trace_recorder.rb
 - app/models/llm_logs/batch_request.rb
@@ -261,7 +265,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.20
+rubygems_version: 3.7.2
 specification_version: 4
 summary: Rails engine for LLM logging and prompt management
 test_files: []

data/app/models/llm_logs/batch/adapters/openai_responses.rb

@@ -1,63 +0,0 @@
-module LlmLogs
-  class Batch
-    module Adapters
-      # Wraps ruby_llm-responses_api's OpenAI Batch API. Contains the remote-API logic
-      # previously inline in Submitter/Reconciler; the DB/claim lifecycle stays in those.
-      class OpenaiResponses
-        PROVIDER = :openai_responses
-
-        def submit(_batch, requests)
-          rubyllm_batch = RubyLLM.batch(model: requests.first.model, provider: PROVIDER)
-          requests.each do |request|
-            payload = request.payload
-            rubyllm_batch.add(
-              payload["input"],
-              id: request.custom_id,
-              instructions: payload["instructions"],
-              temperature: payload["temperature"],
-              **reasoning_extra(payload["reasoning_effort"]),
-              **schema_extra(payload["schema"])
-            )
-          end
-          rubyllm_batch.create!
-          {provider_batch_id: rubyllm_batch.id, openai_batch_id: rubyllm_batch.id, provider_metadata: {}}
-        end
-
-        def terminal_status(batch)
-          resume(batch).status
-        end
-
-        def results(batch)
-          resume(batch).results
-        end
-
-        def error_ids(batch)
-          resume(batch).errors.filter_map { |e| e["custom_id"] }
-        end
-
-        private
-
-        # The Responses API nests effort under `reasoning:`; `reasoning_effort` is the
-        # Chat Completions spelling and is silently ignored here.
-        def reasoning_extra(effort)
-          return {} if effort.blank?
-
-          {reasoning: {effort: effort}}
-        end
-
-        # No memoization: LlmLogs.batch_adapters holds one shared instance of this adapter
-        # for the life of the process, so caching the resumed handle here would go stale
-        # for a long-running PollJob worker (in-progress batches would never re-fetch a
-        # later terminal status) and would grow unbounded. Build a fresh handle every call.
-        def resume(batch)
-          RubyLLM.batch(id: batch.provider_batch_id, provider: PROVIDER)
-        end
-
-        def schema_extra(schema)
-          format = SchemaFormat.call(schema)
-          format ? {text: format} : {}
-        end
-      end
-    end
-  end
-end

data/app/models/llm_logs/batch/schema_format.rb

@@ -1,25 +0,0 @@
-module LlmLogs
-  class Batch
-    # Translates a {name:, schema:, strict:} schema (the shape RubyLLM::Chat#with_schema
-    # produces) into the OpenAI Responses API `text.format` block. The batch path builds
-    # request bodies directly via RubyLLM.batch#add(**extra), bypassing with_schema, so
-    # we must hand the json_schema block in ourselves.
-    module SchemaFormat
-      module_function
-
-      def call(schema)
-        return nil if schema.nil?
-
-        schema = schema.symbolize_keys if schema.respond_to?(:symbolize_keys)
-        {
-          format: {
-            type: "json_schema",
-            name: schema[:name] || "response",
-            schema: schema[:schema] || schema,
-            strict: schema.key?(:strict) ? schema[:strict] : true
-          }
-        }
-      end
-    end
-  end
-end