railwatch 0.1.1 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +41 -0
- data/docs/records.md +60 -1
- data/docs/replacing-sentry.md +6 -6
- data/docs/testing.md +2 -5
- data/docs/troubleshooting.md +36 -0
- data/lib/railwatch/configuration.rb +8 -2
- data/lib/railwatch/execution.rb +1 -1
- data/lib/railwatch/minitest.rb +4 -3
- data/lib/railwatch/record.rb +1 -1
- data/lib/railwatch/subscribers/exceptions.rb +20 -0
- data/lib/railwatch/subscribers/llm.rb +190 -0
- data/lib/railwatch/subscribers/process_info.rb +5 -2
- data/lib/railwatch/subscribers.rb +2 -1
- data/lib/railwatch/version.rb +1 -1
- data/lib/railwatch.rb +1 -1
- metadata +3 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 41585458ddfaf2036a184f5afca05be24749ed73ba54bb41f38ca7e881694ac9
|
|
4
|
+
data.tar.gz: f4bd36a26ec1eb80ee729596c116015f3159f6ecb4306d64ebce9ae3f1eb4335
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 67197b5696dd6d041e8f8a311ac7b5ca6b9f989347dd1c45b542e5cc3a5ad70207af7c507d135c721a522755dde589b8bbe16c912e163bf7796ed2bb6a6fcbed
|
|
7
|
+
data.tar.gz: e3e3746abd37ebfec16b09dae18d67f8ba1ef425b1a4571c805102f313eb93dca19558d64ba3212d18fb7eba4e7cbdc18c670f264cc9288ca65d3fea81b919c7
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,46 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.3 (2026-09-15)
|
|
4
|
+
|
|
5
|
+
- LLM calls are recorded from RubyLLM's own instrumentation. Every model
|
|
6
|
+
call it emits -- `chat`, `compaction`, `embedding`, `image`, `speech`,
|
|
7
|
+
`transcription`, `moderation`, `rerank`, `ocr` -- plus each tool
|
|
8
|
+
invocation becomes one
|
|
9
|
+
`llm_call` child record on the request, job, or command that made it,
|
|
10
|
+
carrying provider, model, duration, token counts per bucket, and cost.
|
|
11
|
+
Nothing is patched: RubyLLM publishes ActiveSupport::Notifications events
|
|
12
|
+
and Railwatch subscribes to them like any Rails event.
|
|
13
|
+
- Both RubyLLM generations are read from the same subscriber. 1.16 puts
|
|
14
|
+
token counts on the event as scalars and reports no cost; 2.0 sends its
|
|
15
|
+
`Tokens` and `Cost` objects, and stamps `workflow_id` and step identity on
|
|
16
|
+
every event inside `RubyLLM.workflow`, which is recorded so an agent run
|
|
17
|
+
can be reassembled from its steps. A 1.16 app has no cost rather than a
|
|
18
|
+
cost of zero, and a model the registry cannot price is unpriced, not free.
|
|
19
|
+
- `capture_llm_content` (default off, `RAILWATCH_CAPTURE_LLM_CONTENT`)
|
|
20
|
+
records the last user turn and the reply, capped at 4 KiB of bytes each. Token
|
|
21
|
+
counts, model, and cost are always captured; prompts are not, because
|
|
22
|
+
they are whatever the app sent a provider.
|
|
23
|
+
|
|
24
|
+
## 0.1.2 (2026-09-14)
|
|
25
|
+
|
|
26
|
+
- A failed job's exception is reported once. Solid Queue re-raises it out
|
|
27
|
+
of the worker thread, where its app executor reports the same error object
|
|
28
|
+
to `Rails.error` again after the `job_attempt` execution has finished; that
|
|
29
|
+
second report (source `application.solid_queue`, unlinked) doubled every
|
|
30
|
+
failed job's occurrence count. An error object now remembers that its
|
|
31
|
+
unhandled report has shipped, across executions.
|
|
32
|
+
- `require "railwatch/minitest"` includes the assertions into
|
|
33
|
+
`ActiveSupport::TestCase` through its load hook, so `assert_railwatch_queries`
|
|
34
|
+
works after the generator's one-line edit to `test/test_helper.rb` without
|
|
35
|
+
a manual `include`.
|
|
36
|
+
- Solid Queue's supervisor, dispatcher, scheduler, and forked workers report
|
|
37
|
+
role `worker` after Solid Queue renames the process. They were classified
|
|
38
|
+
`web` by every health sample taken after boot, with empty Puma thread
|
|
39
|
+
stats.
|
|
40
|
+
- Troubleshooting entries for `json` 3.0 against Rails 8.1.3.1 (`bin/jobs`
|
|
41
|
+
crash loop, not a Railwatch fault) and for deprecations that are counted
|
|
42
|
+
but never listed because the app's deprecation behavior lacks `:notify`.
|
|
43
|
+
|
|
3
44
|
## 0.1.1 (2026-09-14)
|
|
4
45
|
|
|
5
46
|
- Token prefixes are `rw_` for an environment's ingest token and `rwp_`
|
data/docs/records.md
CHANGED
|
@@ -502,6 +502,63 @@ never double-recorded.
|
|
|
502
502
|
| `response_body` | First 4 KiB of the response body, but only when `config.capture_response_body_on_error` is on (off by default) *and* the response was an error. A body that parses as a JSON object is run through the same parameter filter as request params and re-serialized; anything else is stored as it arrived. nil in every other case — including a connection failure, where there is no response (on the Net::HTTP path a body is read only if Net::HTTP already buffered it, so a response being streamed through `read_body` is never consumed; on the Faraday path the body is taken only once a status came back, so an outgoing request payload can never be filed as a response). |
|
|
503
503
|
| `source` | App-code call site (Net::HTTP path only). |
|
|
504
504
|
|
|
505
|
+
### `llm_call`
|
|
506
|
+
|
|
507
|
+
Every RubyLLM model call and tool invocation, from
|
|
508
|
+
`lib/railwatch/subscribers/llm.rb`. RubyLLM publishes its own
|
|
509
|
+
`ActiveSupport::Notifications` events, so nothing is patched and RubyLLM is
|
|
510
|
+
not a dependency — an app without it never emits these. Requires RubyLLM
|
|
511
|
+
1.16 or later, which is where its instrumentation landed.
|
|
512
|
+
|
|
513
|
+
The model call also appears as an `outgoing_request`, since it is an HTTP
|
|
514
|
+
call like any other. The two are different grains on purpose: the
|
|
515
|
+
`outgoing_request` is the HTTP truth, the `llm_call` is what it cost.
|
|
516
|
+
|
|
517
|
+
**Token counts and cost differ by RubyLLM version.** 1.16 reports token
|
|
518
|
+
counts and no cost at all. 2.0 reports both, from its usage ledger, and
|
|
519
|
+
adds the `workflow_*` fields. `cost_nanos` is null rather than zero
|
|
520
|
+
whenever RubyLLM reported no cost or the model registry could not price
|
|
521
|
+
it — an unpriced call is not a free one.
|
|
522
|
+
|
|
523
|
+
**Concurrent tool calls are not recorded.** RubyLLM's opt-in
|
|
524
|
+
`tool_concurrency` (`:threads` or `:fibers`) runs each tool in a fresh
|
|
525
|
+
thread or fiber. `Railwatch::Current` is backed by
|
|
526
|
+
`ActiveSupport::IsolatedExecutionState`, which a new thread does not
|
|
527
|
+
inherit, so the `tool_call.ruby_llm` event fires with no execution to
|
|
528
|
+
attach to and the record is dropped rather than misattributed. This
|
|
529
|
+
affects every Railwatch subscriber in an app-spawned thread, not just this
|
|
530
|
+
one. Tool concurrency is off by default; with it off, tool calls are
|
|
531
|
+
recorded normally. The model calls themselves are unaffected either way,
|
|
532
|
+
so cost is always complete.
|
|
533
|
+
|
|
534
|
+
| Field | Meaning |
|
|
535
|
+
|---|---|
|
|
536
|
+
| `group` | Hash of provider + model + operation, or of `"tool"` + tool name. |
|
|
537
|
+
| `operation` | `"chat"`, `"compaction"`, `"embedding"`, `"image"`, `"speech"`, `"transcription"`, `"moderation"`, `"rerank"`, `"ocr"`, or `"tool"`. |
|
|
538
|
+
| `provider` | Provider slug, e.g. `"anthropic"`. |
|
|
539
|
+
| `model` | Model the call was made with. Empty for a provider that selects its own (moderation). |
|
|
540
|
+
| `response_model` | Model the provider says answered, which can differ from the one asked for. |
|
|
541
|
+
| `tool_name` | Tool name, for `operation: "tool"`. |
|
|
542
|
+
| `duration` | Microseconds. |
|
|
543
|
+
| `status` | `"ok"`, or `"failed"` if the call raised. |
|
|
544
|
+
| `error` | `"Class: message"`, truncated to 255 chars, if the call raised. |
|
|
545
|
+
| `streaming` | Whether the call was streamed. |
|
|
546
|
+
| `message_count` | Conversation length at the time of the call. |
|
|
547
|
+
| `tool_count` | Number of tools the model was offered. |
|
|
548
|
+
| `input_tokens` | Standard (non-cached) input tokens. |
|
|
549
|
+
| `output_tokens` | Billable output tokens. |
|
|
550
|
+
| `cache_read_tokens` | Tokens served from the provider's prompt cache. |
|
|
551
|
+
| `cache_write_tokens` | Tokens written to the provider's prompt cache. |
|
|
552
|
+
| `thinking_tokens` | Reasoning tokens, where the provider reports them separately. |
|
|
553
|
+
| `cost_nanos` | Cost in billionths of a US dollar. Null when unpriced — see above. Nanodollars because a cheap call is well under a microdollar and floats do not sum to an invoice. |
|
|
554
|
+
| `workflow_id` | `RubyLLM.workflow` identifier (2.0+). Null outside a workflow. |
|
|
555
|
+
| `workflow_name` | Workflow name (2.0+). |
|
|
556
|
+
| `workflow_step_id` | Step identifier within the workflow (2.0+). |
|
|
557
|
+
| `workflow_step_name` | Step name (2.0+). |
|
|
558
|
+
| `workflow_step_parent_id` | Enclosing step, for nested steps — what reconstructs the tree (2.0+). |
|
|
559
|
+
| `prompt` | Last user turn, only when `config.capture_llm_content` is on (off by default). Capped at 4 KiB of bytes. |
|
|
560
|
+
| `completion` | The reply, same condition and cap. For `operation: "tool"` these two hold the tool's arguments and result instead. |
|
|
561
|
+
|
|
505
562
|
### `storage_op`
|
|
506
563
|
|
|
507
564
|
Every Active Storage service operation. See
|
|
@@ -652,7 +709,9 @@ ship. Forked workers start with an empty cache.
|
|
|
652
709
|
|
|
653
710
|
### `deprecation`
|
|
654
711
|
|
|
655
|
-
`deprecation.rails`. See `lib/railwatch/subscribers/deprecations.rb`.
|
|
712
|
+
`deprecation.rails`. See `lib/railwatch/subscribers/deprecations.rb`. Rails
|
|
713
|
+
only emits that notification when `config.active_support.deprecation`
|
|
714
|
+
includes `:notify`; see [Troubleshooting](troubleshooting.md#deprecations-are-counted-but-never-listed).
|
|
656
715
|
|
|
657
716
|
| Field | Meaning |
|
|
658
717
|
|---|---|
|
data/docs/replacing-sentry.md
CHANGED
|
@@ -12,12 +12,12 @@ the other.
|
|
|
12
12
|
|
|
13
13
|
## Decide whether Railwatch covers your workload
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
broader, multi-language managed platform with browser replay,
|
|
19
|
-
SDKs, a large integration catalog, and generic tracing.
|
|
20
|
-
|
|
15
|
+
Railwatch and Sentry aim at different targets. Railwatch is an
|
|
16
|
+
application-monitoring product built around Rails executions: a request,
|
|
17
|
+
job, scheduled task, command, or channel action and everything it did.
|
|
18
|
+
Sentry is a broader, multi-language managed platform with browser replay,
|
|
19
|
+
native/mobile SDKs, a large integration catalog, and generic tracing.
|
|
20
|
+
Railwatch does not claim those broader capabilities.
|
|
21
21
|
|
|
22
22
|
| Workload or capability | Status | Boundary |
|
|
23
23
|
|---|---|---|
|
data/docs/testing.md
CHANGED
|
@@ -18,15 +18,12 @@ require "railwatch/rspec"
|
|
|
18
18
|
That requires `railwatch/spec_helper`, includes `Railwatch::SpecHelper` into every
|
|
19
19
|
example group, and defines the matchers below.
|
|
20
20
|
|
|
21
|
-
Minitest: the same thing in `test/test_helper.rb`.
|
|
21
|
+
Minitest: the same thing in `test/test_helper.rb`. The require includes
|
|
22
|
+
`Railwatch::Minitest` into `ActiveSupport::TestCase` on its own.
|
|
22
23
|
|
|
23
24
|
```ruby
|
|
24
25
|
require "rails/test_help"
|
|
25
26
|
require "railwatch/minitest"
|
|
26
|
-
|
|
27
|
-
class ActiveSupport::TestCase
|
|
28
|
-
include Railwatch::Minitest
|
|
29
|
-
end
|
|
30
27
|
```
|
|
31
28
|
|
|
32
29
|
Railwatch must be *enabled* in the test environment or every block would look
|
data/docs/troubleshooting.md
CHANGED
|
@@ -81,6 +81,42 @@ to stderr prefixed `[railwatch]`. They never go to `Rails.logger`, so they
|
|
|
81
81
|
can't become `log` records about themselves. `Railwatch.on_unrecoverable
|
|
82
82
|
{ |e| ... }` gets the same failures as a callback.
|
|
83
83
|
|
|
84
|
+
## `bin/jobs` dies in a loop with `wrong number of arguments (given 2, expected 1)`
|
|
85
|
+
|
|
86
|
+
**Symptom.** Every Solid Queue process crashes at boot from
|
|
87
|
+
`json/common.rb` in `parse`, and Railwatch reports the same
|
|
88
|
+
`ArgumentError` from source `application.solid_queue` hundreds of times.
|
|
89
|
+
The web server still answers until it touches a session or a JSON column.
|
|
90
|
+
|
|
91
|
+
**Cause.** Not Railwatch. `json` 3.0 (September 2026) made `JSON.parse`'s
|
|
92
|
+
options keyword-only, and Active Support up to 8.1.3.1 still passes them
|
|
93
|
+
positionally ([rails/rails#58685](https://github.com/rails/rails/issues/58685)).
|
|
94
|
+
A fresh `rails new` resolves the newest `json`, so a new app hits this
|
|
95
|
+
before Railwatch is even installed.
|
|
96
|
+
|
|
97
|
+
**Fix.** Pin `json` below 3 until a Rails release includes the fix:
|
|
98
|
+
|
|
99
|
+
```sh
|
|
100
|
+
bundle add json --version "< 3"
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
## Deprecations are counted but never listed
|
|
104
|
+
|
|
105
|
+
**Symptom.** The `deprecations` counter on an execution is non-zero, but
|
|
106
|
+
the Deprecations page stays empty.
|
|
107
|
+
|
|
108
|
+
**Cause.** Railwatch listens to the `deprecation.rails` notification, and
|
|
109
|
+
Rails only emits it when the app's deprecation behavior includes
|
|
110
|
+
`:notify`. The default in development is `:log` and in production
|
|
111
|
+
`:silence`, neither of which notifies.
|
|
112
|
+
|
|
113
|
+
**Fix.** Add `:notify` alongside whatever the environment already does:
|
|
114
|
+
|
|
115
|
+
```ruby
|
|
116
|
+
# config/environments/production.rb
|
|
117
|
+
config.active_support.deprecation = [:silence, :notify]
|
|
118
|
+
```
|
|
119
|
+
|
|
84
120
|
## Doubled scheduled_task records
|
|
85
121
|
|
|
86
122
|
**Symptom.** Every recurring task shows twice on the Scheduled tasks
|
|
@@ -5,7 +5,8 @@ module Railwatch
|
|
|
5
5
|
# Laravel Nightwatch's config so the two products document the same knobs.
|
|
6
6
|
class Configuration
|
|
7
7
|
RECORD_TYPES = %i[queries cache_events mail broadcasts notifications outgoing_requests
|
|
8
|
-
storage_ops view_renders logs transactions deprecations sessions
|
|
8
|
+
storage_ops view_renders logs transactions deprecations sessions
|
|
9
|
+
llm_calls].freeze
|
|
9
10
|
|
|
10
11
|
# Framework/vendor noise excluded by default so a fresh install isn't
|
|
11
12
|
# dominated by Rails' own housekeeping. Both lists are opt-in to disable
|
|
@@ -78,7 +79,8 @@ module Railwatch
|
|
|
78
79
|
:profile_sample, :profile_slow_ms, :profile_interval_us, :profiler,
|
|
79
80
|
:capture_job_arguments, :capture_job_retry_errors, :capture_response_body_on_error, :max_attachment_bytes,
|
|
80
81
|
:track_sessions, :session_flush_interval, :session_timeout,
|
|
81
|
-
:capture_console, :interactive_runner_paths, :ignored_request_paths
|
|
82
|
+
:capture_console, :interactive_runner_paths, :ignored_request_paths,
|
|
83
|
+
:capture_llm_content
|
|
82
84
|
|
|
83
85
|
attr_reader :deploy, :deploy_source, :detect_deploy, :user_resolver, :beacon_user_resolver,
|
|
84
86
|
:fingerprint_resolver, :redactors, :rejectors, :before_ingest, :backpressure_high_water
|
|
@@ -175,6 +177,10 @@ module Railwatch
|
|
|
175
177
|
@capture_job_arguments = env_bool("RAILWATCH_CAPTURE_JOB_ARGUMENTS", false)
|
|
176
178
|
@capture_job_retry_errors = env_bool("RAILWATCH_CAPTURE_JOB_RETRY_ERRORS", false)
|
|
177
179
|
@capture_response_body_on_error = env_bool("RAILWATCH_CAPTURE_RESPONSE_BODY_ON_ERROR", false)
|
|
180
|
+
# Prompts and completions are whatever the app sent a provider, so
|
|
181
|
+
# they are off until an operator opts in. Token counts, model, and
|
|
182
|
+
# cost -- the reason the record exists -- are always captured.
|
|
183
|
+
@capture_llm_content = env_bool("RAILWATCH_CAPTURE_LLM_CONTENT", false)
|
|
178
184
|
@max_attachment_bytes = env_int("RAILWATCH_MAX_ATTACHMENT_BYTES", 1_048_576)
|
|
179
185
|
# Release health: one `session` record per browser tab (the beacon
|
|
180
186
|
# client) and per authenticated/cookied server session (Railwatch::Sessions).
|
data/lib/railwatch/execution.rb
CHANGED
|
@@ -12,7 +12,7 @@ module Railwatch
|
|
|
12
12
|
MAX_RECORDS = 10_000
|
|
13
13
|
COUNTERS = %i[queries cached_queries exceptions logs cache_events jobs_enqueued mail
|
|
14
14
|
broadcasts notifications outgoing_requests storage_ops view_renders
|
|
15
|
-
transactions hydrated_models lazy_loads deprecations spans].freeze
|
|
15
|
+
transactions hydrated_models lazy_loads deprecations spans llm_calls].freeze
|
|
16
16
|
# GC.stat with no key builds the whole stat hash; whether this Ruby
|
|
17
17
|
# reports GC time never changes, so ask once.
|
|
18
18
|
GC_TIME_SUPPORTED = GC.stat.key?(:time)
|
data/lib/railwatch/minitest.rb
CHANGED
|
@@ -7,10 +7,9 @@ module Railwatch
|
|
|
7
7
|
# assert_/refute_. Add to test/test_helper.rb:
|
|
8
8
|
#
|
|
9
9
|
# require "railwatch/minitest"
|
|
10
|
-
# class ActiveSupport::TestCase
|
|
11
|
-
# include Railwatch::Minitest
|
|
12
|
-
# end
|
|
13
10
|
#
|
|
11
|
+
# That includes the module into ActiveSupport::TestCase (through its load
|
|
12
|
+
# hook, so the require order against rails/test_help does not matter).
|
|
14
13
|
# Includes Railwatch::SpecHelper, so `railwatch_records(:query)` is available
|
|
15
14
|
# too. See docs/testing.md.
|
|
16
15
|
module Minitest
|
|
@@ -41,3 +40,5 @@ module Railwatch
|
|
|
41
40
|
end
|
|
42
41
|
end
|
|
43
42
|
end
|
|
43
|
+
|
|
44
|
+
ActiveSupport.on_load(:active_support_test_case) { include Railwatch::Minitest } if defined?(ActiveSupport.on_load)
|
data/lib/railwatch/record.rb
CHANGED
|
@@ -10,7 +10,7 @@ module Railwatch
|
|
|
10
10
|
query: 1, n_plus_one: 1, transaction: 1, exception: 1, cache_event: 1, mail: 1,
|
|
11
11
|
broadcast: 1, notification: 1, outgoing_request: 1, storage_op: 1, view_render: 1,
|
|
12
12
|
log: 1, enqueued_job: 1, user: 1, deprecation: 1, visit: 1, process: 1, span: 1, health: 1,
|
|
13
|
-
profile: 1, attachment: 1, session: 1
|
|
13
|
+
profile: 1, attachment: 1, session: 1, llm_call: 1
|
|
14
14
|
}.freeze
|
|
15
15
|
|
|
16
16
|
# Used in place of an execution's envelope when there is no execution, so
|
|
@@ -87,6 +87,13 @@ module Railwatch
|
|
|
87
87
|
def capture(error, handled:, severity:, context: {}, source: nil, fingerprint: nil)
|
|
88
88
|
return unless Railwatch.enabled?
|
|
89
89
|
return if ignored?(error)
|
|
90
|
+
# Solid Queue re-raises a failed job's error out of the worker thread,
|
|
91
|
+
# where its executor reports it to Rails.error a second time
|
|
92
|
+
# (source application.solid_queue) after the job_attempt execution
|
|
93
|
+
# has already finished and reported it. The execution's own
|
|
94
|
+
# first_exception_report? bookkeeping is gone by then, so the error
|
|
95
|
+
# object itself remembers that its unhandled report has shipped.
|
|
96
|
+
return if !handled && reported_unhandled?(error)
|
|
90
97
|
|
|
91
98
|
exe = execution
|
|
92
99
|
if exe&.first_exception_observation?(error, handled)
|
|
@@ -151,9 +158,22 @@ module Railwatch
|
|
|
151
158
|
# children (Railwatch.tail_keep?).
|
|
152
159
|
exe.exception_reported = true if exe
|
|
153
160
|
Railwatch.record_now(:exception, group: group, **rec)
|
|
161
|
+
remember_reported(error)
|
|
154
162
|
end
|
|
155
163
|
end
|
|
156
164
|
|
|
165
|
+
def reported_unhandled?(error)
|
|
166
|
+
error.instance_variable_defined?(:@__railwatch_reported)
|
|
167
|
+
rescue StandardError
|
|
168
|
+
false
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
def remember_reported(error)
|
|
172
|
+
error.instance_variable_set(:@__railwatch_reported, true)
|
|
173
|
+
rescue StandardError
|
|
174
|
+
nil
|
|
175
|
+
end
|
|
176
|
+
|
|
157
177
|
# The group hash `capture` would assign this error. Public so
|
|
158
178
|
# Railwatch.attach can file an attachment against the same issue without
|
|
159
179
|
# having to re-derive the bucketing rule (source snippets are skipped:
|
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Railwatch
|
|
4
|
+
module Subscribers
|
|
5
|
+
# RubyLLM's own instrumentation. It emits an ActiveSupport::Notifications
|
|
6
|
+
# event per model call -- chat, embedding, image, and the rest -- plus one
|
|
7
|
+
# per tool invocation, so nothing here patches RubyLLM; we subscribe the
|
|
8
|
+
# same way we subscribe to Rails.
|
|
9
|
+
#
|
|
10
|
+
# One `llm_call` record per event, with `operation` naming which kind it
|
|
11
|
+
# was. Tool calls are the same record with operation "tool": they sit in
|
|
12
|
+
# the same execution waterfall and carry no tokens or cost.
|
|
13
|
+
#
|
|
14
|
+
# Two RubyLLM generations are supported. 1.16 puts token counts on the
|
|
15
|
+
# event as scalars and reports no cost at all; 2.0 sends Tokens and Cost
|
|
16
|
+
# objects built from its usage ledger. Both normalise to the same wire
|
|
17
|
+
# record, so a 1.16 app simply has no cost. Subscribing to an event
|
|
18
|
+
# RubyLLM never emits costs nothing, so the operations 2.0 added are
|
|
19
|
+
# subscribed unconditionally rather than behind a version check.
|
|
20
|
+
module Llm
|
|
21
|
+
extend Base
|
|
22
|
+
|
|
23
|
+
module_function
|
|
24
|
+
|
|
25
|
+
# Every usage-bearing operation in RubyLLM 2.0. The ones 1.16 knows
|
|
26
|
+
# about (chat, embedding, image, transcription, moderation) emit the
|
|
27
|
+
# same event names, so this list needs no version branch. `compaction`
|
|
28
|
+
# is a chat call by another name -- same payload, same tokens, same
|
|
29
|
+
# cost -- and is billed, so it belongs here rather than being invisible
|
|
30
|
+
# spend.
|
|
31
|
+
OPERATIONS = %w[chat compaction embedding image speech transcription
|
|
32
|
+
moderation rerank ocr].freeze
|
|
33
|
+
|
|
34
|
+
# Costs are fractions of a cent: a cheap model's call is well under a
|
|
35
|
+
# microdollar, and floats summed across a month of rollups do not add
|
|
36
|
+
# up to an invoice. Nanodollars keep it exact in an integer column
|
|
37
|
+
# ($1,000 is 1e12, comfortably inside i64).
|
|
38
|
+
NANOS_PER_DOLLAR = 1_000_000_000
|
|
39
|
+
|
|
40
|
+
# Matches capture_response_body_on_error's cap. Prompts and completions
|
|
41
|
+
# are free text an app controls, so this is a size bound, not redaction.
|
|
42
|
+
CONTENT_MAX = 4096
|
|
43
|
+
|
|
44
|
+
def install!(_app)
|
|
45
|
+
OPERATIONS.each do |operation|
|
|
46
|
+
subscribe("#{operation}.ruby_llm") { |event| record_call(operation, event) }
|
|
47
|
+
end
|
|
48
|
+
subscribe("tool_call.ruby_llm") { |event| record_tool(event) }
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def record_call(operation, event)
|
|
52
|
+
exe = execution
|
|
53
|
+
exe&.count(:llm_calls)
|
|
54
|
+
return unless recording?
|
|
55
|
+
|
|
56
|
+
p = event.payload
|
|
57
|
+
provider = p[:provider].to_s
|
|
58
|
+
model = p[:model].to_s
|
|
59
|
+
Railwatch.record(:llm_call,
|
|
60
|
+
group: Record.group_hash(provider, model, operation),
|
|
61
|
+
timestamp: started_at(event),
|
|
62
|
+
operation: operation,
|
|
63
|
+
provider: provider,
|
|
64
|
+
model: model,
|
|
65
|
+
response_model: p[:response_model]&.to_s&.slice(0, 255),
|
|
66
|
+
duration: micros(event),
|
|
67
|
+
streaming: p[:streaming] == true,
|
|
68
|
+
message_count: p[:message_count],
|
|
69
|
+
tool_count: Array(p[:tools]).size,
|
|
70
|
+
cost_nanos: cost_nanos(p),
|
|
71
|
+
**tokens(p),
|
|
72
|
+
**workflow(p),
|
|
73
|
+
**outcome(p),
|
|
74
|
+
prompt: content(prompt_text(p)),
|
|
75
|
+
completion: content(message_text(p[:response])))
|
|
76
|
+
end
|
|
77
|
+
|
|
78
|
+
# RubyLLM's opt-in tool_concurrency (:threads or :fibers) runs each
|
|
79
|
+
# tool in a fresh thread or fiber, and Current is backed by
|
|
80
|
+
# IsolatedExecutionState, which a new thread does not inherit. So this
|
|
81
|
+
# fires with no execution and the record is dropped.
|
|
82
|
+
#
|
|
83
|
+
# It cannot be fixed from here: by the time the event is delivered we
|
|
84
|
+
# are already inside the worker, with no reference to the execution
|
|
85
|
+
# that spawned it, and the thread is RubyLLM's to create
|
|
86
|
+
# (chat/tool_concurrency.rb propagates its own workflow context across
|
|
87
|
+
# that boundary, but knows nothing of ours). The same is true of every
|
|
88
|
+
# subscriber in an app-spawned thread. Dropping beats guessing: a
|
|
89
|
+
# process-wide fallback would file one request's tool call under
|
|
90
|
+
# another's execution. Concurrency is off by default, and the model
|
|
91
|
+
# calls are unaffected either way, so cost stays complete.
|
|
92
|
+
def record_tool(event)
|
|
93
|
+
exe = execution
|
|
94
|
+
exe&.count(:llm_calls)
|
|
95
|
+
return unless recording?
|
|
96
|
+
|
|
97
|
+
p = event.payload
|
|
98
|
+
tool_name = p[:tool_name].to_s
|
|
99
|
+
Railwatch.record(:llm_call,
|
|
100
|
+
group: Record.group_hash("tool", tool_name),
|
|
101
|
+
timestamp: started_at(event),
|
|
102
|
+
operation: "tool",
|
|
103
|
+
provider: p[:provider].to_s,
|
|
104
|
+
model: p[:model].to_s,
|
|
105
|
+
tool_name: tool_name[0, 255],
|
|
106
|
+
duration: micros(event),
|
|
107
|
+
**workflow(p),
|
|
108
|
+
**outcome(p),
|
|
109
|
+
prompt: content(p[:tool_arguments]),
|
|
110
|
+
completion: content(p[:result_content]))
|
|
111
|
+
end
|
|
112
|
+
|
|
113
|
+
# 2.0 sends a RubyLLM::Tokens; 1.16 sends bare counts on the event.
|
|
114
|
+
def tokens(payload)
|
|
115
|
+
counts = payload[:tokens]
|
|
116
|
+
if counts.respond_to?(:input)
|
|
117
|
+
{ input_tokens: counts.input, output_tokens: counts.output,
|
|
118
|
+
cache_read_tokens: counts.cache_read, cache_write_tokens: counts.cache_write,
|
|
119
|
+
thinking_tokens: counts.thinking }
|
|
120
|
+
else
|
|
121
|
+
{ input_tokens: payload[:input_tokens], output_tokens: payload[:output_tokens],
|
|
122
|
+
cache_read_tokens: payload[:cached_tokens], cache_write_tokens: payload[:cache_creation_tokens],
|
|
123
|
+
thinking_tokens: payload[:thinking_tokens] }
|
|
124
|
+
end
|
|
125
|
+
end
|
|
126
|
+
|
|
127
|
+
# nil in three distinct cases that all mean "we do not know": RubyLLM
|
|
128
|
+
# 1.16 (which reports no cost), a model the registry has no pricing
|
|
129
|
+
# for, and an operation that used no tokens. Never zero -- an unpriced
|
|
130
|
+
# call is not a free call, and the difference matters on a bill.
|
|
131
|
+
def cost_nanos(payload)
|
|
132
|
+
cost = payload[:cost]
|
|
133
|
+
return nil unless cost.respond_to?(:total)
|
|
134
|
+
|
|
135
|
+
total = cost.total
|
|
136
|
+
total && (total * NANOS_PER_DOLLAR).round
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
# Present only on 2.0, and only inside RubyLLM.workflow. Stamped on
|
|
140
|
+
# every event the block emits, which is what lets an agent run be
|
|
141
|
+
# reassembled from its steps.
|
|
142
|
+
def workflow(payload)
|
|
143
|
+
return {} unless payload[:workflow_id]
|
|
144
|
+
|
|
145
|
+
{ workflow_id: payload[:workflow_id].to_s[0, 64],
|
|
146
|
+
workflow_name: payload[:workflow_name].to_s[0, 255],
|
|
147
|
+
workflow_step_id: payload[:workflow_step_id]&.to_s&.slice(0, 64),
|
|
148
|
+
workflow_step_name: payload[:workflow_step_name]&.to_s&.slice(0, 255),
|
|
149
|
+
workflow_step_parent_id: payload[:workflow_step_parent_id]&.to_s&.slice(0, 64) }
|
|
150
|
+
end
|
|
151
|
+
|
|
152
|
+
# Rails adds :exception to the payload when the instrumented block
|
|
153
|
+
# raised; RubyLLM leaves the rest of the event untouched in that case.
|
|
154
|
+
def outcome(payload)
|
|
155
|
+
error = payload[:exception]
|
|
156
|
+
return { status: "ok" } unless error
|
|
157
|
+
|
|
158
|
+
{ status: "failed", error: "#{Array(error).first}: #{Array(error).last}"[0, 255] }
|
|
159
|
+
end
|
|
160
|
+
|
|
161
|
+
# The last thing the app asked, which is the half of a conversation
|
|
162
|
+
# worth seeing next to a cost. Earlier turns are the app's own records.
|
|
163
|
+
def prompt_text(payload)
|
|
164
|
+
messages = payload[:input_messages]
|
|
165
|
+
return payload[:input] || payload[:prompt] || payload[:query] unless messages.respond_to?(:reverse_each)
|
|
166
|
+
|
|
167
|
+
last = messages.reverse_each.find { |m| m.respond_to?(:role) && m.role.to_s == "user" }
|
|
168
|
+
message_text(last)
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
def message_text(message)
|
|
172
|
+
return nil if message.nil?
|
|
173
|
+
|
|
174
|
+
message.respond_to?(:content) ? message.content : message
|
|
175
|
+
end
|
|
176
|
+
|
|
177
|
+
def content(value)
|
|
178
|
+
return nil unless Railwatch.config.capture_llm_content
|
|
179
|
+
return nil if value.nil?
|
|
180
|
+
|
|
181
|
+
text = value.to_s
|
|
182
|
+
return nil if text.empty?
|
|
183
|
+
# byteslice, not [0, n]: the cap bounds what is buffered and shipped,
|
|
184
|
+
# and 4096 characters of CJK is three times that in bytes. scrub
|
|
185
|
+
# repairs the multibyte character the slice may have cut in half.
|
|
186
|
+
text.bytesize > CONTENT_MAX ? text.byteslice(0, CONTENT_MAX).scrub("") : text
|
|
187
|
+
end
|
|
188
|
+
end
|
|
189
|
+
end
|
|
190
|
+
end
|
|
@@ -84,9 +84,12 @@ module Railwatch
|
|
|
84
84
|
|
|
85
85
|
# Puma is loaded in every process of an app that bundles it, so a Solid
|
|
86
86
|
# Queue worker is recognised first, by how it was started (bin/jobs or
|
|
87
|
-
# `rake solid_queue:start`)
|
|
87
|
+
# `rake solid_queue:start`) or by the procline Solid Queue gives every
|
|
88
|
+
# process it forks ("solid-queue-worker(1.7.0): ..."), which replaces
|
|
89
|
+
# $PROGRAM_NAME after boot and would otherwise turn the supervisor,
|
|
90
|
+
# dispatcher, and scheduler into "web" on every health sample.
|
|
88
91
|
def role
|
|
89
|
-
if defined?(::SolidQueue) && ($PROGRAM_NAME.include?("jobs") || ARGV.first.to_s.start_with?("solid_queue:")) then "worker"
|
|
92
|
+
if defined?(::SolidQueue) && ($PROGRAM_NAME.include?("jobs") || $PROGRAM_NAME.start_with?("solid-queue-") || ARGV.first.to_s.start_with?("solid_queue:")) then "worker"
|
|
90
93
|
elsif defined?(::Rails::Console) then "console"
|
|
91
94
|
elsif $PROGRAM_NAME.end_with?("rake") then "command"
|
|
92
95
|
elsif defined?(::Puma) then "web"
|
|
@@ -12,6 +12,7 @@ require "railwatch/subscribers/storage"
|
|
|
12
12
|
require "railwatch/subscribers/views"
|
|
13
13
|
require "railwatch/subscribers/logs"
|
|
14
14
|
require "railwatch/subscribers/jobs"
|
|
15
|
+
require "railwatch/subscribers/llm"
|
|
15
16
|
require "railwatch/subscribers/deprecations"
|
|
16
17
|
require "railwatch/subscribers/users"
|
|
17
18
|
require "railwatch/subscribers/process_info"
|
|
@@ -19,7 +20,7 @@ require "railwatch/subscribers/process_info"
|
|
|
19
20
|
module Railwatch
|
|
20
21
|
module Subscribers
|
|
21
22
|
ALL = [ Requests, Queries, Exceptions, Cache, Mail, Broadcasts, Notifications,
|
|
22
|
-
Storage, Views, Logs, Jobs, Deprecations, Users, ProcessInfo ].freeze
|
|
23
|
+
Storage, Views, Logs, Jobs, Deprecations, Users, ProcessInfo, Llm ].freeze
|
|
23
24
|
|
|
24
25
|
module_function
|
|
25
26
|
|
data/lib/railwatch/version.rb
CHANGED
data/lib/railwatch.rb
CHANGED
|
@@ -496,7 +496,7 @@ module Railwatch
|
|
|
496
496
|
query: :queries, n_plus_one: :queries, transaction: :transactions, cache_event: :cache_events,
|
|
497
497
|
mail: :mail, broadcast: :broadcasts, notification: :notifications, outgoing_request: :outgoing_requests,
|
|
498
498
|
storage_op: :storage_ops, view_render: :view_renders, log: :logs, deprecation: :deprecations,
|
|
499
|
-
session: :sessions
|
|
499
|
+
session: :sessions, llm_call: :llm_calls
|
|
500
500
|
}.freeze
|
|
501
501
|
|
|
502
502
|
def type_plural(type)
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: railwatch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.3
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Cole Robertson
|
|
8
8
|
bindir: bin
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date: 2026-09-
|
|
10
|
+
date: 2026-09-15 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: rails
|
|
@@ -115,6 +115,7 @@ files:
|
|
|
115
115
|
- lib/railwatch/subscribers/deprecations.rb
|
|
116
116
|
- lib/railwatch/subscribers/exceptions.rb
|
|
117
117
|
- lib/railwatch/subscribers/jobs.rb
|
|
118
|
+
- lib/railwatch/subscribers/llm.rb
|
|
118
119
|
- lib/railwatch/subscribers/logs.rb
|
|
119
120
|
- lib/railwatch/subscribers/mail.rb
|
|
120
121
|
- lib/railwatch/subscribers/notifications.rb
|