railwatch 0.1.3 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 41585458ddfaf2036a184f5afca05be24749ed73ba54bb41f38ca7e881694ac9
-  data.tar.gz: f4bd36a26ec1eb80ee729596c116015f3159f6ecb4306d64ebce9ae3f1eb4335
+  metadata.gz: 0b78b98999447f64098f912cdfdcbb6202c9331ac4e07988fb350080d084f79c
+  data.tar.gz: e230f37a0a728c381fc4b84849bc0db762ad14e2b9c37aa5802850b97cba7271
 SHA512:
-  metadata.gz: 67197b5696dd6d041e8f8a311ac7b5ca6b9f989347dd1c45b542e5cc3a5ad70207af7c507d135c721a522755dde589b8bbe16c912e163bf7796ed2bb6a6fcbed
-  data.tar.gz: e3e3746abd37ebfec16b09dae18d67f8ba1ef425b1a4571c805102f313eb93dca19558d64ba3212d18fb7eba4e7cbdc18c670f264cc9288ca65d3fea81b919c7
+  metadata.gz: 27c7659440a7e1592b99d10ad2a409c711a2b5a44cafee9a4707ff8672e0952fcf6819ad5b98c96f4e9d25ae6e1c5b86193ebb70a74b52904f8aee808977c72d
+  data.tar.gz: c1101b7933f5703f4dfd7469b07bf9ac3f39cfaee7bd2a9d451e30126921c33a0326c5976d8605e7dce39f13b10b8d65fd8052007051775bc9e46f0b5cbb6368

data/CHANGELOG.md

@@ -1,5 +1,31 @@
 # Changelog
 
+## 0.1.4 (2026-09-15)
+
+- `llm_call` records what the call carried and how it was configured, not
+  just what it cost. `finish_reason` shows when an answer was cut off
+  (`max_tokens`) or filtered, which previously read exactly like a complete
+  one. `attachments` and `attachment_types` show that a call carried two
+  images and a PDF, so a document read is no longer indistinguishable from
+  an expensive prompt -- on a document-reading call the attachments are most
+  of the input tokens. `params` holds the settings that produced the answer
+  (temperature, max_output_tokens, tool_choice, thinking, caching, whether a
+  schema was used, and the per-operation ones) so a surprising result can be
+  reproduced. `tools` names what the model could reach; `tool_call_id` joins
+  a tool call to the turn that asked for it.
+- `provider_request_id` is read from the response headers (`request-id`,
+  `x-request-id`, `x-amzn-requestid`). It is the only key that joins a
+  Railwatch record to the provider's own record of the same call, and it is
+  what a provider support ticket asks for.
+- `cost_reported` distinguishes a price the provider stated from one
+  estimated against the model registry.
+- `provider_options` is filtered twice before it is stored: through the
+  app's own parameter filter, and again against the credential-name matcher
+  that catches `X-Api-Key` on a header. The default parameter filter is
+  password-shaped, so an `api_key` passed per call went through it
+  untouched. Attachment filenames stay behind `capture_llm_content`, since
+  a filename is business data rather than metadata.
+
 ## 0.1.3 (2026-09-15)
 
 - LLM calls are recorded from RubyLLM's own instrumentation. Every model

data/docs/records.md

@@ -512,7 +512,13 @@ not a dependency — an app without it never emits these. Requires RubyLLM
 
 The model call also appears as an `outgoing_request`, since it is an HTTP
 call like any other. The two are different grains on purpose: the
-`outgoing_request` is the HTTP truth, the `llm_call` is what it cost.
+`outgoing_request` is the HTTP truth, the `llm_call` is what it cost. That
+difference is useful: RubyLLM retries through Faraday, so one `llm_call`
+with several `outgoing_request` rows against it in the same execution is a
+call that was retried. Over a window, `outgoing_requests - llm_calls` to the
+same provider host is the number of *extra attempts*, not a rate -- the
+share of calls that were retried needs counting the calls with more than one
+request against them, which the execution id supports.
 
 **Token counts and cost differ by RubyLLM version.** 1.16 reports token
 counts and no cost at all. 2.0 reports both, from its usage ledger, and
@@ -556,6 +562,15 @@ so cost is always complete.
 | `workflow_step_id` | Step identifier within the workflow (2.0+). |
 | `workflow_step_name` | Step name (2.0+). |
 | `workflow_step_parent_id` | Enclosing step, for nested steps — what reconstructs the tree (2.0+). |
+| `finish_reason` | Why the model stopped: `stop`, `max_tokens`, `tool_calls`, `content_filter`, or whatever the provider spelled it. `max_tokens` means the answer was cut off -- without this a truncated extraction reads exactly like a complete one. |
+| `provider_request_id` | The provider's own id for the request, read from the response headers (`request-id`, `x-request-id`, `x-amzn-requestid`). The only key that joins this record to the provider's side of it, and what a support ticket asks for. |
+| `tools` | Comma-separated names of the tools the model could reach, first 50. `tool_count` says how many; retracing needs which. |
+| `cost_reported` | Whether the provider priced the call itself, or the amount is an estimate from the model registry. Null on gems or operations that report no cost. |
+| `attachments` | How many files the last user turn carried. Absent when it carried none. Only the last turn is measured: earlier turns were counted by the calls that sent them. |
+| `attachment_types` | What they were, by category and count, e.g. `imagex2,pdf`. Categories are RubyLLM's: image, pdf, audio, video, text, document, unknown. On a document-reading call the attachments are most of the input tokens, so without this an expensive scan is indistinguishable from an expensive prompt. |
+| `attachment_names` | Filenames, only when `config.capture_llm_content` is on. A filename like `ACME_invoice_88231.pdf` is business data, not metadata, so it follows the same switch as prompts. |
+| `params` | JSON of the settings that produced the answer, so a surprising one can be reproduced: `temperature`, `max_output_tokens`, `tool_choice`, `tool_call_limit`, `thinking`, `caching`, `citations`, whether a `schema` was used, plus the per-operation ones (`dimensions`, `task_type`, `size`, `count`, `voice`, `format`, `language`, `pages`, `document_count`, `top_n`), `server_tools` and the provider's `server_tool_use` counters. `provider_options` is included, filtered twice: through the app's own parameter filter, and again against the credential-name matcher that catches `X-Api-Key` on a header -- an `api_key` passed per call sails straight through a password-shaped filter. For `operation: "tool"` this holds the tool result's class instead. |
+| `tool_call_id` | The provider's id for a tool invocation, for joining a tool call to the assistant turn that asked for it. |
 | `prompt` | Last user turn, only when `config.capture_llm_content` is on (off by default). Capped at 4 KiB of bytes. |
 | `completion` | The reply, same condition and cap. For `operation: "tool"` these two hold the tool's arguments and result instead. |
 

data/lib/railwatch/subscribers/llm.rb

@@ -67,7 +67,13 @@ module Railwatch
           streaming: p[:streaming] == true,
           message_count: p[:message_count],
           tool_count: Array(p[:tools]).size,
+          tools: tool_names(p),
           cost_nanos: cost_nanos(p),
+          cost_reported: cost_reported(p),
+          finish_reason: finish_reason(p),
+          provider_request_id: provider_request_id(p),
+          params: params(operation, p),
+          **attachments(p),
           **tokens(p),
           **workflow(p),
           **outcome(p),
@@ -103,6 +109,8 @@ module Railwatch
           provider: p[:provider].to_s,
           model: p[:model].to_s,
           tool_name: tool_name[0, 255],
+          tool_call_id: p[:tool_call_id]&.to_s&.slice(0, 128),
+          params: ({ result_class: p[:result_class].to_s[0, 128] } if p[:result_class]),
           duration: micros(event),
           **workflow(p),
           **outcome(p),
@@ -110,6 +118,145 @@ module Railwatch
           completion: content(p[:result_content]))
       end
 
+      # Why the model stopped. :max_tokens means the answer was cut off --
+      # a truncated extraction reads exactly like a complete one without
+      # this, which is the failure most worth being able to see.
+      def finish_reason(payload)
+        response = payload[:response]
+        return nil unless response.respond_to?(:finish_reason)
+
+        response.finish_reason&.to_s&.slice(0, 32)
+      end
+
+      # Message#raw is the Faraday response (protocol.rb hands it in), so the
+      # provider's own request id is in its headers. It is what a provider
+      # support ticket asks for, and the only key that joins our record to
+      # theirs.
+      REQUEST_ID_HEADERS = %w[request-id x-request-id x-amzn-requestid].freeze
+
+      def provider_request_id(payload)
+        raw = payload[:response]
+        raw = raw.raw if raw.respond_to?(:raw)
+        headers = raw.respond_to?(:headers) ? raw.headers : nil
+        return nil unless headers.respond_to?(:[])
+
+        REQUEST_ID_HEADERS.each do |name|
+          value = headers[name]
+          return value.to_s[0, 128] if value.present?
+        end
+        nil
+      end
+
+      # Which tools the model could reach on this call. tool_count alone says
+      # how many; retracing needs which.
+      def tool_names(payload)
+        names = Array(payload[:tools]).map(&:to_s)
+        names.empty? ? nil : names.first(50).join(",")[0, 1024]
+      end
+
+      # Whether the provider priced the call itself, or we estimated it from
+      # the registry. The difference matters when a total is queried against
+      # an invoice.
+      def cost_reported(payload)
+        tokens = payload[:tokens]
+        return nil unless tokens.respond_to?(:reported_cost)
+        # No cost means no provenance to report. false would claim the
+        # registry priced it, which is the same false certainty cost_nanos
+        # avoids by being nil rather than zero.
+        return nil if cost_nanos(payload).nil?
+
+        !tokens.reported_cost.nil?
+      end
+
+      # What the call carried besides text. Images and PDFs are most of the
+      # input tokens on a document-reading call, and without this an
+      # expensive scan is indistinguishable from an expensive prompt.
+      # Only the last user turn is measured: earlier turns were counted by
+      # the calls that sent them, and walking the whole history would both
+      # double-count and cost O(messages) on every call.
+      def attachments(payload)
+        message = last_user_message(payload)
+        list = message.respond_to?(:attachments) ? Array(message.attachments) : []
+        return {} if list.empty?
+
+        types = list.filter_map { |a| a.type.to_s if a.respond_to?(:type) }.tally
+          .sort_by { |_, n| -n }.map { |type, n| n > 1 ? "#{type}x#{n}" : type }.join(",")
+        { attachments: list.size, attachment_types: types[0, 128],
+          attachment_names: content(list.filter_map { |a| a.filename if a.respond_to?(:filename) }.join(", ")) }
+      end
+
+      # The knobs that change what a call costs and what it returns, so a
+      # surprising result can be reproduced with the settings that produced
+      # it. Provider options go through the app's own parameter filter: they
+      # are request configuration, but an app can put anything in them.
+      COMMON_PARAMS = %i[temperature max_output_tokens tool_choice tool_call_limit
+                         thinking caching citations dimensions task_type size count
+                         voice format language pages document_count top_n].freeze
+
+      def params(operation, payload)
+        out = {}
+        COMMON_PARAMS.each do |key|
+          value = payload[key]
+          next if value.nil?
+          # false is kept, not dropped: `caching` defaults to nil, so
+          # caching: false is a deliberate choice, and reproducing a call
+          # needs the settings it ran with. RubyLLM does not distinguish a
+          # boolean that was set from one that defaulted, so record both
+          # rather than guess which mattered.
+          out[key] = value.is_a?(Numeric) || [ true, false ].include?(value) ? value : value.to_s[0, 128]
+        end
+        out[:schema] = true if payload[:schema]
+        out[:server_tools] = Array(payload[:server_tools]).map(&:to_s).first(20) if payload[:server_tools].present?
+        if (usage = payload[:tokens]).respond_to?(:server_tool_use) && usage.server_tool_use.present?
+          out[:server_tool_use] = usage.server_tool_use
+        end
+        if (options = payload[:provider_options]).is_a?(Hash) && !options.empty?
+          out[:provider_options] = provider_options(options)
+        end
+        out[:operation] = operation unless out.empty?
+        out.empty? ? nil : out
+      end
+
+      # Two filters, because one is not enough here. The app's parameter
+      # filter defaults to password-shaped names only, and provider_options
+      # is the one place in this payload where a per-request credential
+      # plausibly lives -- an api_key passed per call sails straight through
+      # a password filter. The redactor's credential-name matcher (the same
+      # one that catches X-Api-Key on a header) closes that.
+      def provider_options(options)
+        redact_credentials(Railwatch.redactor.params(options.transform_keys(&:to_s)))
+      end
+
+      # Recursive, because provider options nest: extra_headers carrying an
+      # authorization value is a hash inside the hash, and a top-level scan
+      # walks straight past it.
+      def redact_credentials(value, depth = 0)
+        return value if depth > 4
+
+        case value
+        when Hash
+          value.each_with_object({}) do |(key, item), out|
+            # The matcher is written for header names, which are hyphenated;
+            # provider options are Ruby-ish and use underscores, so api_key
+            # would sail past a pattern expecting api-key.
+            out[key] = if Railwatch.redactor.redact_header?(key.to_s.tr("_", "-"))
+              Redactor::FILTERED
+            else
+              redact_credentials(item, depth + 1)
+            end
+          end
+        when Array then value.map { |item| redact_credentials(item, depth + 1) }
+        else value
+        end
+      end
+
+      def last_user_message(payload)
+        messages = payload[:input_messages]
+        return nil unless messages.respond_to?(:reverse_each)
+
+        messages.reverse_each.find { |m| m.respond_to?(:role) && m.role.to_s == "user" }
+      end
+
       # 2.0 sends a RubyLLM::Tokens; 1.16 sends bare counts on the event.
       def tokens(payload)
         counts = payload[:tokens]
@@ -161,11 +308,9 @@ module Railwatch
       # The last thing the app asked, which is the half of a conversation
       # worth seeing next to a cost. Earlier turns are the app's own records.
       def prompt_text(payload)
-        messages = payload[:input_messages]
-        return payload[:input] || payload[:prompt] || payload[:query] unless messages.respond_to?(:reverse_each)
+        return payload[:input] || payload[:prompt] || payload[:query] unless payload[:input_messages].respond_to?(:reverse_each)
 
-        last = messages.reverse_each.find { |m| m.respond_to?(:role) && m.role.to_s == "user" }
-        message_text(last)
+        message_text(last_user_message(payload))
       end
 
       def message_text(message)

data/lib/railwatch/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Railwatch
-  VERSION = "0.1.3"
+  VERSION = "0.1.4"
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: railwatch
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.1.4
 platform: ruby
 authors:
 - Cole Robertson
 bindir: bin
 cert_chain: []
-date: 2026-09-15 00:00:00.000000000 Z
+date: 2026-09-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails