railwatch 1.1.1 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c905c857b9616e0eb4a38518afa97d787f0ee12eab644c950671278ec28ef0e3
-  data.tar.gz: 98e4c9103ac4231d2a676b55ceca51ff36d9c84c8570c97391c0eb4d07675b27
+  metadata.gz: 072de4fbe3a2b0854d5e00b3b41b033a901d05d9f681f8e6083ae9c6882e4751
+  data.tar.gz: f24164cbf663302b6df3e302aadf59e7328316a4e6d1c86d1b8b5ad49ee6f85f
 SHA512:
-  metadata.gz: 1b31a6f1ae9843426a8480c3f6c9ee8551c886083d6643069cfbd35e5cc7275416fa047d06270d80ce8410619bf0b7431f5ebb0cf9204a59be50ce1f3a30fc08
-  data.tar.gz: ed4a1f14aef82b6586b8fd86cd86071b5741906de30f89e12f28327d9e9aeaac664fcbbfc6a2ae1d674ec2ee2599fbcd141fb86c30b44db58c84900f7d21b841
+  metadata.gz: 9d3907a21539f08dd1e71d4eb83e0305d749673a51e9eb4d8404e4cff2924665d0c0dd28c5f4e8d62b50174ff0618f7e43d41a5984cd52b24a1d6fe63745dae7
+  data.tar.gz: 1bc7449ec96e9c85432d9f75746a28e4f5152ba71b97d3d32e6b7a9ee4b53044abdf34281a78bf4bf546436b0b8168376b51635daf640fe396fd853163cde638

data/CHANGELOG.md

@@ -6,6 +6,38 @@
      filled in by the release commit, which is also the only commit that
      touches lib/railwatch/version.rb and Gemfile.lock. See CONTRIBUTING.md. -->
 
+## 1.2.0 (2026-10-25)
+
+- Executions send only the counters that fired. Every parent record
+  carried all 18 of `Execution::COUNTERS`, most of them zero: 281 bytes of
+  `counters` JSON on every request, job and command. A request that ran
+  two queries, a log line and a view now sends 59 bytes; a channel action
+  that did nothing sends `{}`. A counter missing from the hash is zero.
+  Needs a railwatch-rs that reads it that way (Rebulk/railwatch-rs#43);
+  older servers show a blank query count on the route page and `null`
+  from the MCP tools for a request that ran none.
+
+- Health samples stop repeating the recurring task manifest. `detail`
+  carried every Solid Queue recurring task key and schedule on every
+  sample, every 15 seconds, from every process: one unchanging string
+  that was 94% of the column (1,312 of 1,417 bytes a row on a 28-task
+  app). It is now sent when it changes, on a process's first sample, and
+  otherwise once every five minutes per process. railwatch-rs's
+  scheduled-task check reads the newest sample that carries one from the
+  last ten minutes, so it always finds one.
+
+- An execution whose record tree outgrew `execution_buffer_bytes` (8 MiB)
+  says so. The buffer keeps the earliest child records and drops the
+  rest, and the parent said nothing: a job that issued 5,048 queries
+  showed a plausible 3,155-query trace, and the loss was visible only as
+  the batch's dropped count. The parent record now carries
+  `dropped_records` and `dropped_bytes` when the tree lost anything.
+  railwatch-rs shows them on the execution page (Rebulk/railwatch-rs#44).
+- Stop building records the execution is about to drop. Once its buffer
+  has overflowed, `Execution#recording?` answers no and counts the record
+  the subscriber was about to build, so the SQL normalisation, redaction,
+  backtrace walk and weighing for it never run.
+
 ## 1.1.1 (2026-10-24)
 
 - The browser client names the first page load after its Inertia

data/docs/records.md

@@ -56,12 +56,14 @@ these fields, from `Railwatch.build_parent` in `lib/railwatch.rb`:
 |---|---|
 | `duration` | Wall time in microseconds, execution start to finish. |
 | `stages` | Hash of stage name → microseconds spent in it (e.g. `{"middleware_before" => 120, "action" => 4300, "render" => 900}`). |
-| `counters` | Hash of child-record counts for this execution — `queries`, `cached_queries`, `exceptions`, `logs`, `cache_events`, `jobs_enqueued`, `mail`, `broadcasts`, `notifications`, `outgoing_requests`, `storage_ops`, `view_renders`, `transactions`, `hydrated_models`, `lazy_loads`, `deprecations`, `spans` (`Execution::COUNTERS`). Counted even when the execution is sampled out, so aggregate rates don't depend on the sample rate. |
+| `counters` | Hash of child-record counts for this execution — `queries`, `cached_queries`, `exceptions`, `logs`, `cache_events`, `jobs_enqueued`, `mail`, `broadcasts`, `notifications`, `outgoing_requests`, `storage_ops`, `view_renders`, `transactions`, `hydrated_models`, `lazy_loads`, `deprecations`, `spans` (`Execution::COUNTERS`). Counted even when the execution is sampled out, so aggregate rates don't depend on the sample rate. Only the counters that fired are sent: a counter missing from the hash is zero, and readers must treat it that way. |
 | `peak_memory` | RSS in bytes, sampled at most once per second process-wide (`Execution.sampled_memory`) — cheap enough to read but not per-execution-accurate to the microsecond. |
 | `allocations` | Objects allocated during the execution (`GC.stat(:total_allocated_objects)` delta). |
 | `gc_time` | GC time in the execution's window, when the Ruby build exposes `GC.stat(:time)`. |
 | `exception_preview` | First unhandled exception's `"Class: message"`, truncated to 255 chars, or nil. |
 | `context` | Serialized `Railwatch.context(...)` key/values active for this execution, parameter-filtered like request params. `"_railwatch_truncated": true` when it did not fit in 64KB. |
+| `dropped_records` | Child records this execution's tree lost to `execution_buffer_bytes` (or `Execution::MAX_RECORDS`): the earliest records are kept and everything after the first overflow is counted here instead of built. Present only when non-zero, and only on a tree that shipped. View renders past `max_view_renders_per_execution` are capped, not dropped, and not counted here. |
+| `dropped_bytes` | Estimated in-memory weight of those dropped records (records refused after the overflow are weighed at the buffer's mean). Present alongside `dropped_records`. |
 
 A sampled-out execution still ships its parent record if it raised an
 unhandled exception. See `Railwatch.finish_execution`. Sampling controls
@@ -870,7 +872,7 @@ watches. The record still ships with whatever it did manage to read.
 | `pool_waiting` | Threads blocked waiting for a connection — sustained non-zero means the pool is undersized for the thread count. |
 | `queue_depth` | `SolidQueue::ReadyExecution.count` — jobs ready to run right now. |
 | `queue_latency` | Microseconds since the oldest ready job was created, i.e. the backlog's head-of-line wait. nil when the queue is empty. |
-| `detail` | JSON string: `queues` (ready count per queue name), `workers` (`SolidQueue::Process` rows of kind `Worker`), `requests_count` (Puma's lifetime request count for this process), `running` (threads Puma has spawned), `max_threads_reached` (true when Puma's `pool_capacity` was 0 at sample time, i.e. no spare thread), `recurring_tasks` (Solid Queue recurring task key => schedule, from `SolidQueue::RecurringTask`; left out when there are none or the table could not be read, so the platform can tell a task removed from `config/recurring.yml` apart from one that stopped running). |
+| `detail` | JSON string: `queues` (ready count per queue name), `workers` (`SolidQueue::Process` rows of kind `Worker`), `requests_count` (Puma's lifetime request count for this process), `running` (threads Puma has spawned), `max_threads_reached` (true when Puma's `pool_capacity` was 0 at sample time, i.e. no spare thread), `recurring_tasks` (Solid Queue recurring task key => schedule, from `SolidQueue::RecurringTask`; left out when there are none or the table could not be read, so the platform can tell a task removed from `config/recurring.yml` apart from one that stopped running). **Not on every sample.** The manifest is identical in every process and changes only with `config/recurring.yml`, so repeating it made it 94% of this column; it is re-sent when it changes and otherwise once every five minutes per process. The reader takes the newest sample that carries one, from inside a ten-minute window. |
 
 Every Puma field is nil when no `Puma::Server` exists in the process, and
 every Solid Queue field is nil when `SolidQueue` isn't loaded.

data/lib/railwatch/execution.rb

@@ -114,6 +114,7 @@ module Railwatch
       @buffered_bytes = 0
       @dropped_records = 0
       @dropped_bytes = 0
+      @full = false
       @keep = false
       # Tail sampling keeps buffering child records for a head-sampled-out
       # execution so the ship/discard decision can be made at the end. Read
@@ -145,8 +146,28 @@ module Railwatch
       @paused_depth.positive?
     end
 
+    # The gate every subscriber asks before it builds a child record. Once
+    # the tree has overflowed (#buffer) the answer is no, and the record the
+    # caller was about to build is counted as dropped right here: building
+    # it -- SQL normalisation, redaction, a backtrace walk, weighing -- only
+    # to throw it away cost ~15us per query, and a job issuing 5,000 queries
+    # into an 8 MiB buffer drops most of them. A gate with a side effect,
+    # deliberately: each caller asks once per record it would build, so the
+    # count stays exact. Bytes are estimated at the buffer's mean record
+    # weight (never a division by zero: @full is only set once a record is
+    # held), so the dropped-bytes figure keeps meaning something.
     def recording?
-      (sampled? || @tail_buffering) && !paused?
+      return false unless (sampled? || @tail_buffering) && !paused?
+      return true unless @full
+
+      @dropped_records += 1
+      @dropped_bytes += @buffered_bytes / @records.size
+      false
+    end
+
+    # Whether the buffer has overflowed and #recording? is now refusing.
+    def full?
+      @full
     end
 
     # Ship this execution's whole tree regardless of the head sampling
@@ -222,13 +243,16 @@ module Railwatch
 
       # A failure-context ring keeps the LAST record_limit records: the ones
       # just before the exception are the ones worth having. Every other
-      # buffer keeps the earliest and rejects the overflow. Either way the
-      # loss is counted onto the parent's batch (Railwatch.finish_execution).
+      # buffer keeps the earliest and rejects the overflow -- and from the
+      # first rejection on, refuses at the gate (#recording?) so nothing
+      # else is built for it. Either way the loss is counted onto the parent
+      # record and its batch (Railwatch.finish_execution).
       if @failure_context
         drop_oldest while @records.any? && (@records.size >= @record_limit || @buffered_bytes + bytes > @byte_limit)
       elsif @records.size >= @record_limit || @buffered_bytes + bytes > @byte_limit
         @dropped_records += 1
         @dropped_bytes += bytes
+        @full = true
         return
       end
 
@@ -279,6 +303,14 @@ module Railwatch
       @counters[counter] += by
     end
 
+    # The counters that actually fired. Most executions touch a handful of the
+    # 18, so sending the rest as explicit zeros costs roughly 240 bytes on the
+    # wire and in storage, per execution, to say nothing happened. A reader
+    # treats an absent counter as zero, exactly as it treats a zero.
+    def counted
+      @counters.reject { |_, value| value.zero? }
+    end
+
     # Rails.error and an outer middleware can observe the same error. Count
     # that occurrence once even when sampling or pause prevents its report.
     # Reporting has separate flags so a suppressed observation can still be

data/lib/railwatch/health.rb

@@ -19,6 +19,8 @@ module Railwatch
     @thread = nil
     @pid = nil
     @stopping = false
+    @manifest_digest = nil
+    @manifest_sent_at = nil
 
     module_function
 
@@ -51,6 +53,8 @@ module Railwatch
       @thread = nil
       @pid = nil
       @stopping = false
+      @manifest_digest = nil
+      @manifest_sent_at = nil
       remove_instance_variable(:@puma_server) if defined?(@puma_server)
       start!
     end
@@ -81,6 +85,7 @@ module Railwatch
         puma = puma_stats
         pool = pool_stats
         queue = solid_queue_stats
+        fields = detail(puma, queue)
         Railwatch.record(:health,
           pid: Process.pid,
           role: Subscribers::ProcessInfo.role,
@@ -93,7 +98,10 @@ module Railwatch
           pool_waiting: pool[:waiting],
           queue_depth: queue[:queue_depth],
           queue_latency: queue[:queue_latency],
-          detail: JSON.generate(detail(puma, queue)))
+          detail: JSON.generate(fields))
+        # Only once the sample is recorded: a sample that raised must not
+        # count as having sent the manifest, or the next one leaves it out.
+        manifest_sent(fields[:recurring_tasks]) if fields[:recurring_tasks]
       end
     rescue StandardError => e
       Railwatch.debug { "health sample failed: #{e.class}: #{e.message}" }
@@ -122,13 +130,41 @@ module Railwatch
     # with scheduled-task detection). Left out rather than sent empty when
     # there are none or the table could not be read: Jobs folds a failed
     # read into an empty set, and "no manifest" must not read as "no tasks".
+    #
+    # The manifest is a property of the deploy, not of the sample: it is
+    # identical in every process and changes only when config/recurring.yml
+    # does. Sending it every interval made it 94% of every health sample --
+    # 1,312 of 1,417 bytes per row on a 28-task app, the same string stored
+    # 32,694 times over nineteen hours. So it is re-sent when it changes, and
+    # otherwise only once per MANIFEST_INTERVAL. Nothing is lost: the
+    # server's scheduled-task check takes the newest sample that *carries* a
+    # manifest from the last ten minutes, and the floor is half that.
     def recurring_tasks
       schedules = Subscribers::Jobs.recurring_tasks[:schedules]
-      schedules.empty? ? nil : schedules
+      return nil if schedules.empty?
+      return nil unless manifest_due?(schedules)
+
+      schedules
     rescue StandardError
       nil
     end
 
+    # Five minutes, as a duration rather than a count of samples:
+    # health_interval is configurable, so any count would outrun the reader's
+    # ten-minute window once an app slowed its sampling down.
+    MANIFEST_INTERVAL = 300
+
+    # Monotonic, so a clock step cannot park the manifest for five minutes or
+    # put it on every sample. Read and written only from the health thread.
+    def manifest_due?(schedules)
+      !(schedules.hash == @manifest_digest && @manifest_sent_at && Clock.monotonic - @manifest_sent_at < MANIFEST_INTERVAL)
+    end
+
+    def manifest_sent(schedules)
+      @manifest_digest = schedules.hash
+      @manifest_sent_at = Clock.monotonic
+    end
+
     # Puma::Server#stats is the only public API exposing busy_threads, so it is
     # used in preference to the individual readers. It also resets Puma's
     # own since-last-read backlog_max/reactor_max gauges, which Railwatch does

data/lib/railwatch/subscribers/views.rb

@@ -33,8 +33,10 @@ module Railwatch
           subscribe("#{kind}.action_view") do |event|
             exe = execution
             exe&.count(:view_renders)
-            next unless recording?
+            # The cap first: a render past it is never built, so a full
+            # buffer's gate (Execution#recording?) must not count it dropped.
             next if exe && exe.counters[:view_renders] > Railwatch.config.max_view_renders_per_execution
+            next unless recording?
             p = event.payload
             identifier = p[:identifier].to_s.delete_prefix(Backtrace.app_root)
             Railwatch.record(:view_render,

data/lib/railwatch/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Railwatch
-  VERSION = "1.1.1"
+  VERSION = "1.2.0"
 end

data/lib/railwatch/wire_fixtures.json

@@ -23,23 +23,9 @@
     },
     "counters": {
       "queries": 2,
-      "cached_queries": 0,
-      "exceptions": 0,
       "logs": 1,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
       "view_renders": 1,
-      "transactions": 0,
-      "hydrated_models": 1,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "hydrated_models": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -99,23 +85,7 @@
     },
     "counters": {
       "queries": 1,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 1,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 0,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "logs": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -160,23 +130,7 @@
     },
     "counters": {
       "queries": 1,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 1,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 0,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "logs": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -222,23 +176,7 @@
     "stages": {},
     "counters": {
       "queries": 1,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 0,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 1,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
+      "transactions": 1
     },
     "peak_memory": 1,
     "allocations": 1,
@@ -267,26 +205,7 @@
     "tenant": null,
     "duration": 1,
     "stages": {},
-    "counters": {
-      "queries": 0,
-      "cached_queries": 0,
-      "exceptions": 0,
-      "logs": 0,
-      "cache_events": 0,
-      "jobs_enqueued": 0,
-      "mail": 0,
-      "broadcasts": 0,
-      "notifications": 0,
-      "outgoing_requests": 0,
-      "storage_ops": 0,
-      "view_renders": 0,
-      "transactions": 0,
-      "hydrated_models": 0,
-      "lazy_loads": 0,
-      "deprecations": 0,
-      "spans": 0,
-      "llm_calls": 0
-    },
+    "counters": {},
     "peak_memory": 1,
     "allocations": 1,
     "gc_time": 1,

data/lib/railwatch.rb

@@ -167,6 +167,13 @@ module Railwatch
         group ||= fields.delete(:group)
         fields[:tail_sampled] = true if tail
         fields[:profiled] = true if profiled
+        # What the tree lost to its own buffer (Execution#buffer), so the
+        # parent says so where someone opening it will look. Only on a tree
+        # that ships: one that is discarded lost nothing anyone would see.
+        if shipping && exe.dropped_records.positive?
+          fields[:dropped_records] = exe.dropped_records
+          fields[:dropped_bytes] = exe.dropped_bytes
+        end
         parent = build_parent(parent_type, exe, group: group, **fields)
       end
       if shipping
@@ -224,7 +231,7 @@ module Railwatch
     def build_parent(type, exe, group: nil, **fields)
       rec = Record.build(type, exe, group: group, timestamp: exe.started_at,
                          duration: exe.duration, stages: exe.stage_durations.compact.transform_keys(&:to_s),
-                         counters: exe.counters, peak_memory: exe.peak_memory, allocations: exe.allocations,
+                         counters: exe.counted, peak_memory: exe.peak_memory, allocations: exe.allocations,
                          gc_time: exe.gc_time, exception_preview: exe.exception_preview,
                          context: Context.serialized, **fields)
       run_redactors(type, rec)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: railwatch
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.2.0
 platform: ruby
 authors:
 - Cole Robertson