railwatch 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3620348235aae6e5733728931b85a9c0edd0ac31367a4fd143eceee1de28cbc0
-  data.tar.gz: 273a578b3059f4390854d7452df971d8223c9e4f6c8c6a9e7e8d9f46a27ef1f3
+  metadata.gz: 072de4fbe3a2b0854d5e00b3b41b033a901d05d9f681f8e6083ae9c6882e4751
+  data.tar.gz: f24164cbf663302b6df3e302aadf59e7328316a4e6d1c86d1b8b5ad49ee6f85f
 SHA512:
-  metadata.gz: 82640c29a57b4c99d890b40ef75179d635b53308879516ba4973aaed0cfa1179696e0ee14e6660be2eb59c935611eb395e9f508e583c2604fbe8a7e57da0f8ad
-  data.tar.gz: 53ff388054d12658c7d0cb99c3d0c3dd01be8579ac2227efbb07d65f008a766851761655904b081c8f344aa5db6f3ad9d2cb1e25a349bf30b943e85cb74489bc
+  metadata.gz: 9d3907a21539f08dd1e71d4eb83e0305d749673a51e9eb4d8404e4cff2924665d0c0dd28c5f4e8d62b50174ff0618f7e43d41a5984cd52b24a1d6fe63745dae7
+  data.tar.gz: 1bc7449ec96e9c85432d9f75746a28e4f5152ba71b97d3d32e6b7a9ee4b53044abdf34281a78bf4bf546436b0b8168376b51635daf640fe396fd853163cde638

data/CHANGELOG.md

@@ -6,6 +6,49 @@
      filled in by the release commit, which is also the only commit that
      touches lib/railwatch/version.rb and Gemfile.lock. See CONTRIBUTING.md. -->
 
+## 1.2.0 (2026-10-25)
+
+- Executions send only the counters that fired. Every parent record
+  carried all 18 of `Execution::COUNTERS`, most of them zero: 281 bytes of
+  `counters` JSON on every request, job and command. A request that ran
+  two queries, a log line and a view now sends 59 bytes; a channel action
+  that did nothing sends `{}`. A counter missing from the hash is zero.
+  Needs a railwatch-rs that reads it that way (Rebulk/railwatch-rs#43);
+  older servers show a blank query count on the route page and `null`
+  from the MCP tools for a request that ran none.
+
+- Health samples stop repeating the recurring task manifest. `detail`
+  carried every Solid Queue recurring task key and schedule on every
+  sample, every 15 seconds, from every process: one unchanging string
+  that was 94% of the column (1,312 of 1,417 bytes a row on a 28-task
+  app). It is now sent when it changes, on a process's first sample, and
+  otherwise once every five minutes per process. railwatch-rs's
+  scheduled-task check reads the newest sample that carries one from the
+  last ten minutes, so it always finds one.
+
+- An execution whose record tree outgrew `execution_buffer_bytes` (8 MiB)
+  says so. The buffer keeps the earliest child records and drops the
+  rest, and the parent said nothing: a job that issued 5,048 queries
+  showed a plausible 3,155-query trace, and the loss was visible only as
+  the batch's dropped count. The parent record now carries
+  `dropped_records` and `dropped_bytes` when the tree lost anything.
+  railwatch-rs shows them on the execution page (Rebulk/railwatch-rs#44).
+- Stop building records the execution is about to drop. Once its buffer
+  has overflowed, `Execution#recording?` answers no and counts the record
+  the subscriber was about to build, so the SQL normalisation, redaction,
+  backtrace walk and weighing for it never run.
+
+## 1.1.1 (2026-10-24)
+
+- The browser client names the first page load after its Inertia
+  component again on inertia_rails 3.23+, which renders the page object as
+  `<script data-page="app" type="application/json">` instead of a
+  `data-page` attribute on `<div id="app">`. The client only read the
+  attribute, so every initial load (the only visits that carry LCP, CLS,
+  INP and TTFB) arrived with no component: per-page web vitals were empty.
+  Apps that copied the client (`app/frontend/lib/railwatch.ts`) need the
+  same change in their copy.
+
 ## 1.1.0 (2026-10-22)
 
 - A receiver that refuses the connection (nothing listening: a Railwatch

data/docs/records.md

@@ -56,12 +56,14 @@ these fields, from `Railwatch.build_parent` in `lib/railwatch.rb`:
 |---|---|
 | `duration` | Wall time in microseconds, execution start to finish. |
 | `stages` | Hash of stage name → microseconds spent in it (e.g. `{"middleware_before" => 120, "action" => 4300, "render" => 900}`). |
-| `counters` | Hash of child-record counts for this execution — `queries`, `cached_queries`, `exceptions`, `logs`, `cache_events`, `jobs_enqueued`, `mail`, `broadcasts`, `notifications`, `outgoing_requests`, `storage_ops`, `view_renders`, `transactions`, `hydrated_models`, `lazy_loads`, `deprecations`, `spans` (`Execution::COUNTERS`). Counted even when the execution is sampled out, so aggregate rates don't depend on the sample rate. |
+| `counters` | Hash of child-record counts for this execution — `queries`, `cached_queries`, `exceptions`, `logs`, `cache_events`, `jobs_enqueued`, `mail`, `broadcasts`, `notifications`, `outgoing_requests`, `storage_ops`, `view_renders`, `transactions`, `hydrated_models`, `lazy_loads`, `deprecations`, `spans` (`Execution::COUNTERS`). Counted even when the execution is sampled out, so aggregate rates don't depend on the sample rate. Only the counters that fired are sent: a counter missing from the hash is zero, and readers must treat it that way. |
 | `peak_memory` | RSS in bytes, sampled at most once per second process-wide (`Execution.sampled_memory`) — cheap enough to read but not per-execution-accurate to the microsecond. |
 | `allocations` | Objects allocated during the execution (`GC.stat(:total_allocated_objects)` delta). |
 | `gc_time` | GC time in the execution's window, when the Ruby build exposes `GC.stat(:time)`. |
 | `exception_preview` | First unhandled exception's `"Class: message"`, truncated to 255 chars, or nil. |
 | `context` | Serialized `Railwatch.context(...)` key/values active for this execution, parameter-filtered like request params. `"_railwatch_truncated": true` when it did not fit in 64KB. |
+| `dropped_records` | Child records this execution's tree lost to `execution_buffer_bytes` (or `Execution::MAX_RECORDS`): the earliest records are kept and everything after the first overflow is counted here instead of built. Present only when non-zero, and only on a tree that shipped. View renders past `max_view_renders_per_execution` are capped, not dropped, and not counted here. |
+| `dropped_bytes` | Estimated in-memory weight of those dropped records (records refused after the overflow are weighed at the buffer's mean). Present alongside `dropped_records`. |
 
 A sampled-out execution still ships its parent record if it raised an
 unhandled exception. See `Railwatch.finish_execution`. Sampling controls
@@ -870,7 +872,7 @@ watches. The record still ships with whatever it did manage to read.
 | `pool_waiting` | Threads blocked waiting for a connection — sustained non-zero means the pool is undersized for the thread count. |
 | `queue_depth` | `SolidQueue::ReadyExecution.count` — jobs ready to run right now. |
 | `queue_latency` | Microseconds since the oldest ready job was created, i.e. the backlog's head-of-line wait. nil when the queue is empty. |
-| `detail` | JSON string: `queues` (ready count per queue name), `workers` (`SolidQueue::Process` rows of kind `Worker`), `requests_count` (Puma's lifetime request count for this process), `running` (threads Puma has spawned), `max_threads_reached` (true when Puma's `pool_capacity` was 0 at sample time, i.e. no spare thread), `recurring_tasks` (Solid Queue recurring task key => schedule, from `SolidQueue::RecurringTask`; left out when there are none or the table could not be read, so the platform can tell a task removed from `config/recurring.yml` apart from one that stopped running). |
+| `detail` | JSON string: `queues` (ready count per queue name), `workers` (`SolidQueue::Process` rows of kind `Worker`), `requests_count` (Puma's lifetime request count for this process), `running` (threads Puma has spawned), `max_threads_reached` (true when Puma's `pool_capacity` was 0 at sample time, i.e. no spare thread), `recurring_tasks` (Solid Queue recurring task key => schedule, from `SolidQueue::RecurringTask`; left out when there are none or the table could not be read, so the platform can tell a task removed from `config/recurring.yml` apart from one that stopped running). **Not on every sample.** The manifest is identical in every process and changes only with `config/recurring.yml`, so repeating it made it 94% of this column; it is re-sent when it changes and otherwise once every five minutes per process. The reader takes the newest sample that carries one, from inside a ten-minute window. |
 
 Every Puma field is nil when no `Puma::Server` exists in the process, and
 every Solid Queue field is nil when `SolidQueue` isn't loaded.

data/lib/generators/railwatch/install/templates/railwatch.ts

@@ -659,13 +659,19 @@ function startVitals() {
   )
 }
 
-// The component the server rendered this page with, off the root element's
-// serialized page object. Inertia's own events take over from here.
+// The component the server rendered this page with, off the serialized page
+// object: inertia_rails 3.23+ renders it as <script data-page="app"
+// type="application/json">, older ones as <div id="app" data-page="...">.
+// Inertia's own events take over from here.
 function pageComponent(): string | undefined {
   try {
-    const page = JSON.parse(
-      document.getElementById("app")?.dataset.page ?? "{}",
-    ) as { component?: string }
+    const script = document.querySelector(
+      'script[data-page="app"][type="application/json"]',
+    )
+    const raw = script
+      ? script.textContent
+      : document.getElementById("app")?.dataset.page
+    const page = JSON.parse(raw ?? "{}") as { component?: string }
     return page.component
   } catch {
     // Not an Inertia-rendered page, or the payload moved. Report it anyway.

data/lib/railwatch/execution.rb

@@ -114,6 +114,7 @@ module Railwatch
       @buffered_bytes = 0
       @dropped_records = 0
       @dropped_bytes = 0
+      @full = false
       @keep = false
       # Tail sampling keeps buffering child records for a head-sampled-out
       # execution so the ship/discard decision can be made at the end. Read
@@ -145,8 +146,28 @@ module Railwatch
       @paused_depth.positive?
     end
 
+    # The gate every subscriber asks before it builds a child record. Once
+    # the tree has overflowed (#buffer) the answer is no, and the record the
+    # caller was about to build is counted as dropped right here: building
+    # it -- SQL normalisation, redaction, a backtrace walk, weighing -- only
+    # to throw it away cost ~15us per query, and a job issuing 5,000 queries
+    # into an 8 MiB buffer drops most of them. A gate with a side effect,
+    # deliberately: each caller asks once per record it would build, so the
+    # count stays exact. Bytes are estimated at the buffer's mean record
+    # weight (never a division by zero: @full is only set once a record is
+    # held), so the dropped-bytes figure keeps meaning something.
     def recording?
-      (sampled? || @tail_buffering) && !paused?
+      return false unless (sampled? || @tail_buffering) && !paused?
+      return true unless @full
+
+      @dropped_records += 1
+      @dropped_bytes += @buffered_bytes / @records.size
+      false
+    end
+
+    # Whether the buffer has overflowed and #recording? is now refusing.
+    def full?
+      @full
     end
 
     # Ship this execution's whole tree regardless of the head sampling
@@ -222,13 +243,16 @@ module Railwatch
 
       # A failure-context ring keeps the LAST record_limit records: the ones
       # just before the exception are the ones worth having. Every other
-      # buffer keeps the earliest and rejects the overflow. Either way the
-      # loss is counted onto the parent's batch (Railwatch.finish_execution).
+      # buffer keeps the earliest and rejects the overflow -- and from the
+      # first rejection on, refuses at the gate (#recording?) so nothing
+      # else is built for it. Either way the loss is counted onto the parent
+      # record and its batch (Railwatch.finish_execution).
       if @failure_context
         drop_oldest while @records.any? && (@records.size >= @record_limit || @buffered_bytes + bytes > @byte_limit)
       elsif @records.size >= @record_limit || @buffered_bytes + bytes > @byte_limit
         @dropped_records += 1
         @dropped_bytes += bytes
+        @full = true
         return
       end
 
@@ -279,6 +303,14 @@ module Railwatch
       @counters[counter] += by
     end
 
+    # The counters that actually fired. Most executions touch a handful of the
+    # 18, so sending the rest as explicit zeros costs roughly 240 bytes on the
+    # wire and in storage, per execution, to say nothing happened. A reader
+    # treats an absent counter as zero, exactly as it treats a zero.
+    def counted
+      @counters.reject { |_, value| value.zero? }
+    end
+
     # Rails.error and an outer middleware can observe the same error. Count
     # that occurrence once even when sampling or pause prevents its report.
     # Reporting has separate flags so a suppressed observation can still be

data/lib/railwatch/health.rb

@@ -19,6 +19,8 @@ module Railwatch
     @thread = nil
     @pid = nil
     @stopping = false
+    @manifest_digest = nil
+    @manifest_sent_at = nil
 
     module_function
 
@@ -51,6 +53,8 @@ module Railwatch
       @thread = nil
       @pid = nil
       @stopping = false
+      @manifest_digest = nil
+      @manifest_sent_at = nil
       remove_instance_variable(:@puma_server) if defined?(@puma_server)
       start!
     end
@@ -81,6 +85,7 @@ module Railwatch
         puma = puma_stats
         pool = pool_stats
         queue = solid_queue_stats
+        fields = detail(puma, queue)
         Railwatch.record(:health,
           pid: Process.pid,
           role: Subscribers::ProcessInfo.role,
@@ -93,7 +98,10 @@ module Railwatch
           pool_waiting: pool[:waiting],
           queue_depth: queue[:queue_depth],
           queue_latency: queue[:queue_latency],
-          detail: JSON.generate(detail(puma, queue)))
+          detail: JSON.generate(fields))
+        # Only once the sample is recorded: a sample that raised must not
+        # count as having sent the manifest, or the next one leaves it out.
+        manifest_sent(fields[:recurring_tasks]) if fields[:recurring_tasks]
       end
     rescue StandardError => e
       Railwatch.debug { "health sample failed: #{e.class}: #{e.message}" }
@@ -122,13 +130,41 @@ module Railwatch
     # with scheduled-task detection). Left out rather than sent empty when
     # there are none or the table could not be read: Jobs folds a failed
     # read into an empty set, and "no manifest" must not read as "no tasks".
+    #
+    # The manifest is a property of the deploy, not of the sample: it is
+    # identical in every process and changes only when config/recurring.yml
+    # does. Sending it every interval made it 94% of every health sample --
+    # 1,312 of 1,417 bytes per row on a 28-task app, the same string stored
+    # 32,694 times over nineteen hours. So it is re-sent when it changes, and
+    # otherwise only once per MANIFEST_INTERVAL. Nothing is lost: the
+    # server's scheduled-task check takes the newest sample that *carries* a
+    # manifest from the last ten minutes, and the floor is half that.
     def recurring_tasks
       schedules = Subscribers::Jobs.recurring_tasks[:schedules]
-      schedules.empty? ? nil : schedules
+      return nil if schedules.empty?
+      return nil unless manifest_due?(schedules)
+
+      schedules
     rescue StandardError
       nil
     end
 
+    # Five minutes, as a duration rather than a count of samples:
+    # health_interval is configurable, so any count would outrun the reader's
+    # ten-minute window once an app slowed its sampling down.
+    MANIFEST_INTERVAL = 300
+
+    # Monotonic, so a clock step cannot park the manifest for five minutes or
+    # put it on every sample. Read and written only from the health thread.
+    def manifest_due?(schedules)
+      !(schedules.hash == @manifest_digest && @manifest_sent_at && Clock.monotonic - @manifest_sent_at < MANIFEST_INTERVAL)
+    end
+
+    def manifest_sent(schedules)
+      @manifest_digest = schedules.hash
+      @manifest_sent_at = Clock.monotonic
+    end
+
     # Puma::Server#stats is the only public API exposing busy_threads, so it is
     # used in preference to the individual readers. It also resets Puma's
     # own since-last-read backlog_max/reactor_max gauges, which Railwatch does

data/lib/railwatch/subscribers/views.rb

@@ -33,8 +33,10 @@ module Railwatch
           subscribe("#{kind}.action_view") do |event|
             exe = execution
             exe&.count(:view_renders)
-            next unless recording?
+            # The cap first: a render past it is never built, so a full
+            # buffer's gate (Execution#recording?) must not count it dropped.
             next if exe && exe.counters[:view_renders] > Railwatch.config.max_view_renders_per_execution
+            next unless recording?
             p = event.payload
             identifier = p[:identifier].to_s.delete_prefix(Backtrace.app_root)
             Railwatch.record(:view_render,

data/lib/railwatch/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Railwatch
-  VERSION = "1.1.0"
+  VERSION = "1.2.0"
 end

data/lib/railwatch/wire_fixtures.json

@@ -23,23 +23,9 @@
     },
     "counters": {
       "queries": 2,
-      "cached_queries": 0,
-      "exceptions": 0,
       "logs": 1,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
       "view_renders": 1,
-      "transactions": 0,
-      "hydrated_models": 1,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "hydrated_models": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -99,23 +85,7 @@
     },
     "counters": {
       "queries": 1,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 1,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 0,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "logs": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -160,23 +130,7 @@
     },
     "counters": {
       "queries": 1,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 1,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 0,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "logs": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -222,23 +176,7 @@
     "stages": {},
     "counters": {
       "queries": 1,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 0,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 1,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "transactions": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -267,26 +205,7 @@
     "tenant": null,
     "duration": 1,
     "stages": {},
-    "counters": {
-      "queries": 0,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 0,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 0,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
-    },
+    "counters": {},
     "peak_memory": 1,
     "allocations": 1,
     "gc_time": 1,

data/lib/railwatch.rb

@@ -167,6 +167,13 @@ module Railwatch
         group ||= fields.delete(:group)
         fields[:tail_sampled] = true if tail
         fields[:profiled] = true if profiled
+        # What the tree lost to its own buffer (Execution#buffer), so the
+        # parent says so where someone opening it will look. Only on a tree
+        # that ships: one that is discarded lost nothing anyone would see.
+        if shipping && exe.dropped_records.positive?
+          fields[:dropped_records] = exe.dropped_records
+          fields[:dropped_bytes] = exe.dropped_bytes
+        end
         parent = build_parent(parent_type, exe, group: group, **fields)
       end
       if shipping
@@ -224,7 +231,7 @@ module Railwatch
     def build_parent(type, exe, group: nil, **fields)
       rec = Record.build(type, exe, group: group, timestamp: exe.started_at,
                          duration: exe.duration, stages: exe.stage_durations.compact.transform_keys(&:to_s),
-                         counters: exe.counters, peak_memory: exe.peak_memory, allocations: exe.allocations,
+                         counters: exe.counted, peak_memory: exe.peak_memory, allocations: exe.allocations,
                          gc_time: exe.gc_time, exception_preview: exe.exception_preview,
                          context: Context.serialized, **fields)
       run_redactors(type, rec)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: railwatch
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Cole Robertson