railwatch 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 50160a5f39e8b422581a3b6e6cf0b07eddc3565cfc17a8ba522bc49e15cd93df
-  data.tar.gz: 4f2e2fdac0c85df89a9d595fb33f02ebcf5c02a6f88ed81cb6dc19a163a2b90d
+  metadata.gz: 41585458ddfaf2036a184f5afca05be24749ed73ba54bb41f38ca7e881694ac9
+  data.tar.gz: f4bd36a26ec1eb80ee729596c116015f3159f6ecb4306d64ebce9ae3f1eb4335
 SHA512:
-  metadata.gz: 584a8545680ad9c2d24ef05f6fcfd331a933cb87a310859a6c6e7df6c447b1d1f1d1ace62ccf11b899ae9bbcf0255d3b397562e57ba402e2410ba5790a99b144
-  data.tar.gz: 0fd7c798ae6d8b745e5e19ed9a894331b09eb6b15a179d9d6cda6d6bbb4c1d8a6b8bf15eefecdb24db57fd2b956b1eabba1085e0aba7fa074810aa2d5f41374c
+  metadata.gz: 67197b5696dd6d041e8f8a311ac7b5ca6b9f989347dd1c45b542e5cc3a5ad70207af7c507d135c721a522755dde589b8bbe16c912e163bf7796ed2bb6a6fcbed
+  data.tar.gz: e3e3746abd37ebfec16b09dae18d67f8ba1ef425b1a4571c805102f313eb93dca19558d64ba3212d18fb7eba4e7cbdc18c670f264cc9288ca65d3fea81b919c7

data/CHANGELOG.md

@@ -1,5 +1,26 @@
 # Changelog
 
+## 0.1.3 (2026-09-15)
+
+- LLM calls are recorded from RubyLLM's own instrumentation. Every model
+  call it emits -- `chat`, `compaction`, `embedding`, `image`, `speech`,
+  `transcription`, `moderation`, `rerank`, `ocr` -- plus each tool
+  invocation becomes one
+  `llm_call` child record on the request, job, or command that made it,
+  carrying provider, model, duration, token counts per bucket, and cost.
+  Nothing is patched: RubyLLM publishes ActiveSupport::Notifications events
+  and Railwatch subscribes to them like any Rails event.
+- Both RubyLLM generations are read from the same subscriber. 1.16 puts
+  token counts on the event as scalars and reports no cost; 2.0 sends its
+  `Tokens` and `Cost` objects, and stamps `workflow_id` and step identity on
+  every event inside `RubyLLM.workflow`, which is recorded so an agent run
+  can be reassembled from its steps. A 1.16 app has no cost rather than a
+  cost of zero, and a model the registry cannot price is unpriced, not free.
+- `capture_llm_content` (default off, `RAILWATCH_CAPTURE_LLM_CONTENT`)
+  records the last user turn and the reply, capped at 4 KiB of bytes each. Token
+  counts, model, and cost are always captured; prompts are not, because
+  they are whatever the app sent a provider.
+
 ## 0.1.2 (2026-09-14)
 
 - A failed job's exception is reported once. Solid Queue re-raises it out

data/docs/records.md

@@ -502,6 +502,63 @@ never double-recorded.
 | `response_body` | First 4 KiB of the response body, but only when `config.capture_response_body_on_error` is on (off by default) *and* the response was an error. A body that parses as a JSON object is run through the same parameter filter as request params and re-serialized; anything else is stored as it arrived. nil in every other case — including a connection failure, where there is no response (on the Net::HTTP path a body is read only if Net::HTTP already buffered it, so a response being streamed through `read_body` is never consumed; on the Faraday path the body is taken only once a status came back, so an outgoing request payload can never be filed as a response). |
 | `source` | App-code call site (Net::HTTP path only). |
 
+### `llm_call`
+
+Every RubyLLM model call and tool invocation, from
+`lib/railwatch/subscribers/llm.rb`. RubyLLM publishes its own
+`ActiveSupport::Notifications` events, so nothing is patched and RubyLLM is
+not a dependency — an app without it never emits these. Requires RubyLLM
+1.16 or later, which is where its instrumentation landed.
+
+The model call also appears as an `outgoing_request`, since it is an HTTP
+call like any other. The two are different grains on purpose: the
+`outgoing_request` is the HTTP truth, the `llm_call` is what it cost.
+
+**Token counts and cost differ by RubyLLM version.** 1.16 reports token
+counts and no cost at all. 2.0 reports both, from its usage ledger, and
+adds the `workflow_*` fields. `cost_nanos` is null rather than zero
+whenever RubyLLM reported no cost or the model registry could not price
+it — an unpriced call is not a free one.
+
+**Concurrent tool calls are not recorded.** RubyLLM's opt-in
+`tool_concurrency` (`:threads` or `:fibers`) runs each tool in a fresh
+thread or fiber. `Railwatch::Current` is backed by
+`ActiveSupport::IsolatedExecutionState`, which a new thread does not
+inherit, so the `tool_call.ruby_llm` event fires with no execution to
+attach to and the record is dropped rather than misattributed. This
+affects every Railwatch subscriber in an app-spawned thread, not just this
+one. Tool concurrency is off by default; with it off, tool calls are
+recorded normally. The model calls themselves are unaffected either way,
+so cost is always complete.
+
+| Field | Meaning |
+|---|---|
+| `group` | Hash of provider + model + operation, or of `"tool"` + tool name. |
+| `operation` | `"chat"`, `"compaction"`, `"embedding"`, `"image"`, `"speech"`, `"transcription"`, `"moderation"`, `"rerank"`, `"ocr"`, or `"tool"`. |
+| `provider` | Provider slug, e.g. `"anthropic"`. |
+| `model` | Model the call was made with. Empty for a provider that selects its own (moderation). |
+| `response_model` | Model the provider says answered, which can differ from the one asked for. |
+| `tool_name` | Tool name, for `operation: "tool"`. |
+| `duration` | Microseconds. |
+| `status` | `"ok"`, or `"failed"` if the call raised. |
+| `error` | `"Class: message"`, truncated to 255 chars, if the call raised. |
+| `streaming` | Whether the call was streamed. |
+| `message_count` | Conversation length at the time of the call. |
+| `tool_count` | Number of tools the model was offered. |
+| `input_tokens` | Standard (non-cached) input tokens. |
+| `output_tokens` | Billable output tokens. |
+| `cache_read_tokens` | Tokens served from the provider's prompt cache. |
+| `cache_write_tokens` | Tokens written to the provider's prompt cache. |
+| `thinking_tokens` | Reasoning tokens, where the provider reports them separately. |
+| `cost_nanos` | Cost in billionths of a US dollar. Null when unpriced — see above. Nanodollars because a cheap call is well under a microdollar and floats do not sum to an invoice. |
+| `workflow_id` | `RubyLLM.workflow` identifier (2.0+). Null outside a workflow. |
+| `workflow_name` | Workflow name (2.0+). |
+| `workflow_step_id` | Step identifier within the workflow (2.0+). |
+| `workflow_step_name` | Step name (2.0+). |
+| `workflow_step_parent_id` | Enclosing step, for nested steps — what reconstructs the tree (2.0+). |
+| `prompt` | Last user turn, only when `config.capture_llm_content` is on (off by default). Capped at 4 KiB of bytes. |
+| `completion` | The reply, same condition and cap. For `operation: "tool"` these two hold the tool's arguments and result instead. |
+
 ### `storage_op`
 
 Every Active Storage service operation. See

data/lib/railwatch/configuration.rb

@@ -5,7 +5,8 @@ module Railwatch
   # Laravel Nightwatch's config so the two products document the same knobs.
   class Configuration
     RECORD_TYPES = %i[queries cache_events mail broadcasts notifications outgoing_requests
-                      storage_ops view_renders logs transactions deprecations sessions].freeze
+                      storage_ops view_renders logs transactions deprecations sessions
+                      llm_calls].freeze
 
     # Framework/vendor noise excluded by default so a fresh install isn't
     # dominated by Rails' own housekeeping. Both lists are opt-in to disable
@@ -78,7 +79,8 @@ module Railwatch
                   :profile_sample, :profile_slow_ms, :profile_interval_us, :profiler,
                   :capture_job_arguments, :capture_job_retry_errors, :capture_response_body_on_error, :max_attachment_bytes,
                   :track_sessions, :session_flush_interval, :session_timeout,
-                  :capture_console, :interactive_runner_paths, :ignored_request_paths
+                  :capture_console, :interactive_runner_paths, :ignored_request_paths,
+                  :capture_llm_content
 
     attr_reader :deploy, :deploy_source, :detect_deploy, :user_resolver, :beacon_user_resolver,
                 :fingerprint_resolver, :redactors, :rejectors, :before_ingest, :backpressure_high_water
@@ -175,6 +177,10 @@ module Railwatch
       @capture_job_arguments = env_bool("RAILWATCH_CAPTURE_JOB_ARGUMENTS", false)
       @capture_job_retry_errors = env_bool("RAILWATCH_CAPTURE_JOB_RETRY_ERRORS", false)
       @capture_response_body_on_error = env_bool("RAILWATCH_CAPTURE_RESPONSE_BODY_ON_ERROR", false)
+      # Prompts and completions are whatever the app sent a provider, so
+      # they are off until an operator opts in. Token counts, model, and
+      # cost -- the reason the record exists -- are always captured.
+      @capture_llm_content = env_bool("RAILWATCH_CAPTURE_LLM_CONTENT", false)
       @max_attachment_bytes = env_int("RAILWATCH_MAX_ATTACHMENT_BYTES", 1_048_576)
       # Release health: one `session` record per browser tab (the beacon
       # client) and per authenticated/cookied server session (Railwatch::Sessions).

data/lib/railwatch/execution.rb

@@ -12,7 +12,7 @@ module Railwatch
     MAX_RECORDS = 10_000
     COUNTERS = %i[queries cached_queries exceptions logs cache_events jobs_enqueued mail
                   broadcasts notifications outgoing_requests storage_ops view_renders
-                  transactions hydrated_models lazy_loads deprecations spans].freeze
+                  transactions hydrated_models lazy_loads deprecations spans llm_calls].freeze
     # GC.stat with no key builds the whole stat hash; whether this Ruby
     # reports GC time never changes, so ask once.
     GC_TIME_SUPPORTED = GC.stat.key?(:time)

data/lib/railwatch/record.rb

@@ -10,7 +10,7 @@ module Railwatch
       query: 1, n_plus_one: 1, transaction: 1, exception: 1, cache_event: 1, mail: 1,
       broadcast: 1, notification: 1, outgoing_request: 1, storage_op: 1, view_render: 1,
       log: 1, enqueued_job: 1, user: 1, deprecation: 1, visit: 1, process: 1, span: 1, health: 1,
-      profile: 1, attachment: 1, session: 1
+      profile: 1, attachment: 1, session: 1, llm_call: 1
     }.freeze
 
     # Used in place of an execution's envelope when there is no execution, so

data/lib/railwatch/subscribers/llm.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+module Railwatch
+  module Subscribers
+    # RubyLLM's own instrumentation. It emits an ActiveSupport::Notifications
+    # event per model call -- chat, embedding, image, and the rest -- plus one
+    # per tool invocation, so nothing here patches RubyLLM; we subscribe the
+    # same way we subscribe to Rails.
+    #
+    # One `llm_call` record per event, with `operation` naming which kind it
+    # was. Tool calls are the same record with operation "tool": they sit in
+    # the same execution waterfall and carry no tokens or cost.
+    #
+    # Two RubyLLM generations are supported. 1.16 puts token counts on the
+    # event as scalars and reports no cost at all; 2.0 sends Tokens and Cost
+    # objects built from its usage ledger. Both normalise to the same wire
+    # record, so a 1.16 app simply has no cost. Subscribing to an event
+    # RubyLLM never emits costs nothing, so the operations 2.0 added are
+    # subscribed unconditionally rather than behind a version check.
+    module Llm
+      extend Base
+
+      module_function
+
+      # Every usage-bearing operation in RubyLLM 2.0. The ones 1.16 knows
+      # about (chat, embedding, image, transcription, moderation) emit the
+      # same event names, so this list needs no version branch. `compaction`
+      # is a chat call by another name -- same payload, same tokens, same
+      # cost -- and is billed, so it belongs here rather than being invisible
+      # spend.
+      OPERATIONS = %w[chat compaction embedding image speech transcription
+                      moderation rerank ocr].freeze
+
+      # Costs are fractions of a cent: a cheap model's call is well under a
+      # microdollar, and floats summed across a month of rollups do not add
+      # up to an invoice. Nanodollars keep it exact in an integer column
+      # ($1,000 is 1e12, comfortably inside i64).
+      NANOS_PER_DOLLAR = 1_000_000_000
+
+      # Matches capture_response_body_on_error's cap. Prompts and completions
+      # are free text an app controls, so this is a size bound, not redaction.
+      CONTENT_MAX = 4096
+
+      def install!(_app)
+        OPERATIONS.each do |operation|
+          subscribe("#{operation}.ruby_llm") { |event| record_call(operation, event) }
+        end
+        subscribe("tool_call.ruby_llm") { |event| record_tool(event) }
+      end
+
+      def record_call(operation, event)
+        exe = execution
+        exe&.count(:llm_calls)
+        return unless recording?
+
+        p = event.payload
+        provider = p[:provider].to_s
+        model = p[:model].to_s
+        Railwatch.record(:llm_call,
+          group: Record.group_hash(provider, model, operation),
+          timestamp: started_at(event),
+          operation: operation,
+          provider: provider,
+          model: model,
+          response_model: p[:response_model]&.to_s&.slice(0, 255),
+          duration: micros(event),
+          streaming: p[:streaming] == true,
+          message_count: p[:message_count],
+          tool_count: Array(p[:tools]).size,
+          cost_nanos: cost_nanos(p),
+          **tokens(p),
+          **workflow(p),
+          **outcome(p),
+          prompt: content(prompt_text(p)),
+          completion: content(message_text(p[:response])))
+      end
+
+      # RubyLLM's opt-in tool_concurrency (:threads or :fibers) runs each
+      # tool in a fresh thread or fiber, and Current is backed by
+      # IsolatedExecutionState, which a new thread does not inherit. So this
+      # fires with no execution and the record is dropped.
+      #
+      # It cannot be fixed from here: by the time the event is delivered we
+      # are already inside the worker, with no reference to the execution
+      # that spawned it, and the thread is RubyLLM's to create
+      # (chat/tool_concurrency.rb propagates its own workflow context across
+      # that boundary, but knows nothing of ours). The same is true of every
+      # subscriber in an app-spawned thread. Dropping beats guessing: a
+      # process-wide fallback would file one request's tool call under
+      # another's execution. Concurrency is off by default, and the model
+      # calls are unaffected either way, so cost stays complete.
+      def record_tool(event)
+        exe = execution
+        exe&.count(:llm_calls)
+        return unless recording?
+
+        p = event.payload
+        tool_name = p[:tool_name].to_s
+        Railwatch.record(:llm_call,
+          group: Record.group_hash("tool", tool_name),
+          timestamp: started_at(event),
+          operation: "tool",
+          provider: p[:provider].to_s,
+          model: p[:model].to_s,
+          tool_name: tool_name[0, 255],
+          duration: micros(event),
+          **workflow(p),
+          **outcome(p),
+          prompt: content(p[:tool_arguments]),
+          completion: content(p[:result_content]))
+      end
+
+      # 2.0 sends a RubyLLM::Tokens; 1.16 sends bare counts on the event.
+      def tokens(payload)
+        counts = payload[:tokens]
+        if counts.respond_to?(:input)
+          { input_tokens: counts.input, output_tokens: counts.output,
+            cache_read_tokens: counts.cache_read, cache_write_tokens: counts.cache_write,
+            thinking_tokens: counts.thinking }
+        else
+          { input_tokens: payload[:input_tokens], output_tokens: payload[:output_tokens],
+            cache_read_tokens: payload[:cached_tokens], cache_write_tokens: payload[:cache_creation_tokens],
+            thinking_tokens: payload[:thinking_tokens] }
+        end
+      end
+
+      # nil in three distinct cases that all mean "we do not know": RubyLLM
+      # 1.16 (which reports no cost), a model the registry has no pricing
+      # for, and an operation that used no tokens. Never zero -- an unpriced
+      # call is not a free call, and the difference matters on a bill.
+      def cost_nanos(payload)
+        cost = payload[:cost]
+        return nil unless cost.respond_to?(:total)
+
+        total = cost.total
+        total && (total * NANOS_PER_DOLLAR).round
+      end
+
+      # Present only on 2.0, and only inside RubyLLM.workflow. Stamped on
+      # every event the block emits, which is what lets an agent run be
+      # reassembled from its steps.
+      def workflow(payload)
+        return {} unless payload[:workflow_id]
+
+        { workflow_id: payload[:workflow_id].to_s[0, 64],
+          workflow_name: payload[:workflow_name].to_s[0, 255],
+          workflow_step_id: payload[:workflow_step_id]&.to_s&.slice(0, 64),
+          workflow_step_name: payload[:workflow_step_name]&.to_s&.slice(0, 255),
+          workflow_step_parent_id: payload[:workflow_step_parent_id]&.to_s&.slice(0, 64) }
+      end
+
+      # Rails adds :exception to the payload when the instrumented block
+      # raised; RubyLLM leaves the rest of the event untouched in that case.
+      def outcome(payload)
+        error = payload[:exception]
+        return { status: "ok" } unless error
+
+        { status: "failed", error: "#{Array(error).first}: #{Array(error).last}"[0, 255] }
+      end
+
+      # The last thing the app asked, which is the half of a conversation
+      # worth seeing next to a cost. Earlier turns are the app's own records.
+      def prompt_text(payload)
+        messages = payload[:input_messages]
+        return payload[:input] || payload[:prompt] || payload[:query] unless messages.respond_to?(:reverse_each)
+
+        last = messages.reverse_each.find { |m| m.respond_to?(:role) && m.role.to_s == "user" }
+        message_text(last)
+      end
+
+      def message_text(message)
+        return nil if message.nil?
+
+        message.respond_to?(:content) ? message.content : message
+      end
+
+      def content(value)
+        return nil unless Railwatch.config.capture_llm_content
+        return nil if value.nil?
+
+        text = value.to_s
+        return nil if text.empty?
+        # byteslice, not [0, n]: the cap bounds what is buffered and shipped,
+        # and 4096 characters of CJK is three times that in bytes. scrub
+        # repairs the multibyte character the slice may have cut in half.
+        text.bytesize > CONTENT_MAX ? text.byteslice(0, CONTENT_MAX).scrub("") : text
+      end
+    end
+  end
+end

data/lib/railwatch/subscribers.rb

@@ -12,6 +12,7 @@ require "railwatch/subscribers/storage"
 require "railwatch/subscribers/views"
 require "railwatch/subscribers/logs"
 require "railwatch/subscribers/jobs"
+require "railwatch/subscribers/llm"
 require "railwatch/subscribers/deprecations"
 require "railwatch/subscribers/users"
 require "railwatch/subscribers/process_info"
@@ -19,7 +20,7 @@ require "railwatch/subscribers/process_info"
 module Railwatch
   module Subscribers
     ALL = [ Requests, Queries, Exceptions, Cache, Mail, Broadcasts, Notifications,
-            Storage, Views, Logs, Jobs, Deprecations, Users, ProcessInfo ].freeze
+            Storage, Views, Logs, Jobs, Deprecations, Users, ProcessInfo, Llm ].freeze
 
     module_function
 

data/lib/railwatch/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Railwatch
-  VERSION = "0.1.2"
+  VERSION = "0.1.3"
 end

data/lib/railwatch.rb

@@ -496,7 +496,7 @@ module Railwatch
       query: :queries, n_plus_one: :queries, transaction: :transactions, cache_event: :cache_events,
       mail: :mail, broadcast: :broadcasts, notification: :notifications, outgoing_request: :outgoing_requests,
       storage_op: :storage_ops, view_render: :view_renders, log: :logs, deprecation: :deprecations,
-      session: :sessions
+      session: :sessions, llm_call: :llm_calls
     }.freeze
 
     def type_plural(type)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: railwatch
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - Cole Robertson
@@ -115,6 +115,7 @@ files:
 - lib/railwatch/subscribers/deprecations.rb
 - lib/railwatch/subscribers/exceptions.rb
 - lib/railwatch/subscribers/jobs.rb
+- lib/railwatch/subscribers/llm.rb
 - lib/railwatch/subscribers/logs.rb
 - lib/railwatch/subscribers/mail.rb
 - lib/railwatch/subscribers/notifications.rb