tj-scale 1.0.2 → 1.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bfa02cccd18ac22e845267ddac5835d60c620e793e65aa34b648d1cc7889eab3
-  data.tar.gz: 28a78e5c2643a400910aaca6d535f191f521c1633a8eaab48d28365aa40835e6
+  metadata.gz: a15d2cfe793359d5678938c3f6cf974f496ac716b173acbf4f9b408668b88da4
+  data.tar.gz: '0490a176f40d292a3868a135ddd8ecdddfad30025a5d7458163c7159f2dbe144'
 SHA512:
-  metadata.gz: a269ea3a6087b6bdf54f40a6ac98960d7e70219049ec02dea5d7de592277fba106c5e77769277bd04b3623e356ad90a03926cb0bfcc7edef0867f37ef1bc11d3
-  data.tar.gz: e2ed378beb3196a9e9508132f04fea691a828ee6359650b8bb8e6daa96c8693199cbb091e06c8ee1d9aa31353596f7af305c23fbe7703ecbf8a35c9fceb78bb7
+  metadata.gz: d5978c29aeaac97ade3368b4c693ab5d445c9c43716490d8299fe22c2785a0014b335df040c3836db385de4f28093aa4c74b0abe2642f731eeb4295e69f4bbbc
+  data.tar.gz: 4e3ff1c12ff935b3f211c9698a718e54f1e5b41501dcf565cbdc8a64c08f7df1e50258cb4444b029040f117fce5cac9385e36a995dbe8b714c8f684baf6b52fb

data/CHANGELOG.md

@@ -7,13 +7,72 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+
+## [1.1.1] (tj-scale) - 2026-07-20
+
+### Fixed
+
+- **Web `queue_time_ms` is now captured on Heroku.** The Rack queue-time
+  middleware only parsed the nginx-style `X-Request-Start: t=<seconds>` form, but
+  Heroku's router sends a **bare millisecond epoch** (e.g. `1731523200000`). The
+  header is now parsed in both formats, so `queue_time_ms` is actually produced on
+  Heroku when `TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1` (previously always `nil`).
+- **Stale `queue_time_ms` is no longer re-reported.** The captured router
+  queue-time snapshot is now consumed each ingest tick, so after web traffic stops
+  the gem stops resending the last elevated value (which kept the web dyno scaled
+  up and blocked downscaling on the `queue_time_ms` metric).
+- **Delayed Job counting no longer depends on PostgreSQL.** The runnable-jobs
+  query used a Postgres-only `interval '5 seconds'` SQL literal that raises on
+  MySQL/SQLite; the exception was swallowed to `0`, silently masking the entire
+  worker backlog. The 5-second buffer is now computed in Ruby (`run_at < now - 5`),
+  so counting works on any Delayed Job-supported database.
+
+### Added
+
+- Integration coverage for the Sidekiq backend against the real `Sidekiq::Queue`
+  API and a live Redis, alongside the existing double-based specs — those only
+  asserted our assumptions about `size` and `latency`, so an upstream API change
+  would have surfaced as broken autoscaling in a host app rather than a red test.
+  `sidekiq` is now a development dependency. The specs use
+  `TJ_SCALE_TEST_REDIS_URL` (default `redis://localhost:6379/15`), `FLUSHDB`
+  between examples, and skip themselves when no Redis is reachable, so
+  `rake spec` still passes without one.
+- An explicit test that enqueued counts exclude the retry and scheduled sets,
+  pinning that as a deliberate limitation: a backlog waiting in `retry` or
+  `schedule` does not by itself scale workers up.
+
+
+## [1.1.0] (tj-scale) - 2026-06-12
+
+### Changed
+
+- **Monitor process is now auto-detected from Heroku's `DYNO` env var**: `web.1`
+  monitors web and `worker.1` monitors worker, so one app monitors both process
+  types with no Procfile env prefixes or extra config. `TJ_SCALE_MONITOR_PROCESS`
+  still pins a single process type when set explicitly; off-Heroku (no `DYNO`)
+  the default remains `worker`. Non-web/worker dynos (`release.*`, `run.*`,
+  `scheduler.*`) never start the metrics loop.
+- `target_process` (payload) follows the auto-detected process, so web samples
+  land on web scale rules and worker samples on worker rules automatically.
+
+### Added
+
+- **Custom process types**: any Heroku process type (e.g. `bulk-worker.1`) now
+  runs the monitor and reports job metrics under its own `target_process`, not
+  just `web`/`worker`. Platform dynos (`release`, `run`, `scheduler`, `console`,
+  `rake`) are excluded.
+- **Per-process Sidekiq queue filter**: `TJ_SCALE_SIDEKIQ_QUEUES_<PROCESS>`
+  (process name upcased, non-alphanumerics → `_`, e.g.
+  `TJ_SCALE_SIDEKIQ_QUEUES_BULK_WORKER=low`) overrides the global
+  `TJ_SCALE_SIDEKIQ_QUEUES` for that process, so a bulk fleet can scale on its
+  own queue's backlog while the worker fleet scales on the rest.
+
 ## [1.0.2] (tj-scale) - 2026-06-12
 
 ### Changed
 
 - README fully updated for the `tj-scale` name: install/build/publish instructions,
   repository URLs (`Untechnickle/tj-scale-gem`), and generic example app names.
-- Supersedes 1.0.0 and 1.0.1, which are yanked (same code, stale docs/description).
 
 ## [1.0.1] (tj-scale) - 2026-06-12
 

data/README.md

@@ -153,7 +153,7 @@ Then `bundle install`.
 
 3. **No extra `require`** is needed in `application.rb`: the gem registers a **Rails Railtie** and loads with Rails. Ensure your app already bundles its job backend — **`delayed_job_active_record`** or **`sidekiq`** (the gem auto-detects which one is loaded, or set `TJ_SCALE_JOB_BACKEND` explicitly).
 
-4. Deploy or run the server as usual. The monitoring thread starts only on **`web.1`** or **`worker.1`**, depending on `TJ_SCALE_MONITOR_PROCESS` (see below).
+4. Deploy or run the server as usual. The monitoring thread starts on **`web.1`** and **`worker.1`** automatically (process type auto-detected from Heroku's `DYNO`), so one app reports both web and worker metrics with no extra config. Set `TJ_SCALE_MONITOR_PROCESS` to pin a single process type (see below).
 
 ### 5. Configure environment variables
 
@@ -165,7 +165,7 @@ Configuration is **entirely via environment variables** (and Heroku **config var
 | `TJ_SCALE_API_URL` | Yes | Full URL of the Agent API ingest path: **`POST /api/v1/metrics`** or **`POST /api/v1/sys-logs`** (same handler). Example local: `http://127.0.0.1:5001/api/v1/metrics`. |
 | `TJ_SCALE_TARGET_APP` | No | Heroku app name to scale; defaults to **`HEROKU_APP_NAME`** if unset. |
 | `TJ_SCALE_TARGET_PROCESS` | No | `web` or `worker` to scale; defaults to `TJ_SCALE_MONITOR_PROCESS`. |
-| `TJ_SCALE_MONITOR_PROCESS` | No | `web` or `worker` — only that process’s **`.1`** dyno runs the reporter (default **`worker`**). |
+| `TJ_SCALE_MONITOR_PROCESS` | No | `web` or `worker` to pin the reporter to one process type. Default: auto-detected from `DYNO` (`web.1` reports web, `worker.1` reports worker); **`worker`** when `DYNO` is unset (non-Heroku). |
 | `TJ_SCALE_INTERVAL_SECONDS` | No | Seconds between posts (default **20**). |
 | `TJ_SCALE_JOB_BACKEND` | No | Queue backend for the **worker** reporter: `delayed_job`, `sidekiq`, or `auto` (default). `auto` uses Sidekiq when the `sidekiq` gem is loaded, otherwise Delayed Job. |
 | `TJ_SCALE_SIDEKIQ_QUEUES` | No | Sidekiq backend only: comma-separated queue names to count (empty = all queues). |
@@ -272,7 +272,7 @@ heroku config:set TJ_SCALE_API_TOKEN=your_api_token_here
 heroku config:set TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics
 ```
 
-The `DYNO` environment variable is automatically set by Heroku and is used to determine which dyno should run the monitoring loop (only `web.1` or `worker.1`, depending on `TJ_SCALE_MONITOR_PROCESS`).
+The `DYNO` environment variable is automatically set by Heroku and is used to determine which dynos run the monitoring loop: `web.1` and `worker.1` each report their own process type (override with `TJ_SCALE_MONITOR_PROCESS`).
 
 ### Example: web on `my-app-web`, workers on `my-app-workers`
 
@@ -307,7 +307,7 @@ Ensure both apps have the `web` / `worker` process types in the `Procfile` as ne
 
 ### Automatic Monitoring
 
-Once installed and configured, the gem starts monitoring on the first dyno of the configured process type (`web.1` or `worker.1`, see `TJ_SCALE_MONITOR_PROCESS`). The loop runs every `TJ_SCALE_INTERVAL_SECONDS` (default 20) and POSTs metrics to the configured API.
+Once installed and configured, the gem starts monitoring on the first dyno of each process type (`web.1` and `worker.1`; pin one with `TJ_SCALE_MONITOR_PROCESS`). The loop runs every `TJ_SCALE_INTERVAL_SECONDS` (default 20) and POSTs metrics to the configured API.
 
 ### Manual Usage
 
@@ -338,7 +338,7 @@ ENV['TJ_MAX_PRIORITY'] = '10'
 
 ## How It Works
 
-1. **Initialization**: When your Rails application starts, the `Railtie` checks if it's running on the first monitored dyno (`web.1` or `worker.1`, per `TJ_SCALE_MONITOR_PROCESS`)
+1. **Initialization**: When your Rails application starts, the `Railtie` checks if it's running on the first dyno of a monitored process type (`web.1` or `worker.1`, auto-detected from `DYNO`; `TJ_SCALE_MONITOR_PROCESS` pins one)
 2. **Background Thread**: If conditions are met, a background thread is started
 3. **Monitoring Loop**: Every N seconds (`TJ_SCALE_INTERVAL_SECONDS`), the thread POSTs a payload: on **worker** reporters it counts waiting jobs on the active backend; on **web** reporters it sends router queue time (middleware / env) plus request volume and average response time for the interval.
 4. **Job counting (worker reporters, Delayed Job backend)**: `pending_job_count` filters jobs based on:

data/lib/tj-scale.rb

@@ -11,8 +11,10 @@ require "tj_scale_ruby/models/tj_scale_ruby"
 # TJ Scale (the control plane) scales Heroku formation when ingested metrics cross rules
 # you configure there. This gem only **reports signals**; it never calls Heroku directly.
 #
-# **Where it runs:** A single background thread on **one** dyno — the first dyno of the
-# process you choose (`web.1` or `worker.1` via +TJ_SCALE_MONITOR_PROCESS+). Heroku sets +DYNO+.
+# **Where it runs:** A single background thread on the **first dyno of each process type**.
+# The process is auto-detected from Heroku's +DYNO+ (`web.1` monitors web, `worker.1` monitors
+# worker — so one app monitors both with no extra config). Set +TJ_SCALE_MONITOR_PROCESS+ to
+# pin a single process type instead; off-Heroku (no +DYNO+) the default is `worker`.
 #
 # **What it sends:** On each tick (+TJ_SCALE_INTERVAL_SECONDS+), it POSTs JSON to
 # +TJ_SCALE_API_URL+ (e.g. +/api/v1/metrics+ or +/api/v1/sys-logs+ — same handler on the
@@ -21,12 +23,12 @@ require "tj_scale_ruby/models/tj_scale_ruby"
 # includes +heroku_app+ and +target_process+; scaling limits (min/max dynos) live in the
 # control plane's settings, not in this gem.
 #
-# - **Worker monitor** (+TJ_SCALE_MONITOR_PROCESS=worker+): counts waiting jobs
+# - **Worker monitor** (on +worker.1+): counts waiting jobs
 #   (+job_count+, +queue_time_s+) on the configured queue backend — **Delayed Job** or **Sidekiq**,
 #   toggled with +TJ_SCALE_JOB_BACKEND+ (+delayed_job+ / +sidekiq+ / +auto+, the default, which
 #   prefers Sidekiq when loaded). Delayed Job needs +delayed_job_active_record+ and a shared DB;
 #   Sidekiq needs the +sidekiq+ gem and Redis (+TJ_SCALE_SIDEKIQ_QUEUES+ limits counted queues).
-# - **Web monitor** (+TJ_SCALE_MONITOR_PROCESS=web+): sends +queue_time_ms+ from Rack middleware
+# - **Web monitor** (on +web.1+): sends +queue_time_ms+ from Rack middleware
 #   (router queue via +X-Request-Start+) when +TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1+, plus
 #   request volume and average +response_time_ms+ for the tick. Without middleware, set
 #   +TJ_SCALE_QUEUE_TIME_MS+ or the platform will not scale on queue time until data exists.
@@ -69,7 +71,8 @@ module TjScaleRuby
     end
   end
 
-  # True when this dyno should run the metrics loop (first of TJ_SCALE_MONITOR_PROCESS).
+  # True when this dyno should run the metrics loop: the first dyno (.1) of the
+  # monitored process type (auto-detected from DYNO, or TJ_SCALE_MONITOR_PROCESS).
   # Result is memoized; cleared by {.reset_configuration!}.
   def self.monitor_on_this_dyno?
     return @monitor_on_this_dyno if defined?(@monitor_on_this_dyno)

data/lib/tj_scale_ruby/configuration.rb

@@ -5,14 +5,21 @@ module TjScaleRuby
   class Configuration
     DEFAULT_INTERVAL_SECONDS = 20
     JOB_BACKENDS = %w[auto delayed_job sidekiq].freeze
+    # Any Heroku process type (web, worker, bulk-worker, …) is monitorable;
+    # platform dynos never run app monitoring.
+    PROCESS_NAME = /\A[a-z][a-z0-9_-]*\z/
+    NON_APP_PROCESSES = %w[release run scheduler console rake].freeze
 
     attr_reader :monitor_process, :interval_seconds,
       :min_priority, :max_priority, :job_backend, :sidekiq_queues
 
     def initialize(env = ENV)
-      @monitor_process = normalize_process(env["TJ_SCALE_MONITOR_PROCESS"]) || "worker"
+      @monitor_process = normalize_process(env["TJ_SCALE_MONITOR_PROCESS"]) ||
+        process_from_dyno(env["DYNO"]) || "worker"
       @job_backend = normalize_job_backend(env["TJ_SCALE_JOB_BACKEND"])
-      @sidekiq_queues = parse_list(env["TJ_SCALE_SIDEKIQ_QUEUES"])
+      @sidekiq_queues = parse_list(
+        env[sidekiq_queues_key(@monitor_process)] || env["TJ_SCALE_SIDEKIQ_QUEUES"]
+      )
       @interval_seconds = parse_positive_int(env["TJ_SCALE_INTERVAL_SECONDS"], DEFAULT_INTERVAL_SECONDS)
       @target_app = nonempty_string(env["TJ_SCALE_TARGET_APP"])
       @heroku_app_name = nonempty_string(env["HEROKU_APP_NAME"])
@@ -33,11 +40,24 @@ module TjScaleRuby
 
     private
 
+    # Auto-detect from Heroku's DYNO (web.1 → web, worker.1 → worker) so one app
+    # can monitor both process types without per-process env. Nil off-Heroku and
+    # for non-web/worker dynos (release.1, run.x, scheduler.x).
+    def process_from_dyno(dyno)
+      normalize_process(dyno.to_s.split(".").first)
+    end
+
     def normalize_process(value)
       p = value.to_s.strip.downcase
-      return nil if p.empty?
+      return nil if p.empty? || NON_APP_PROCESSES.include?(p)
+
+      PROCESS_NAME.match?(p) ? p : nil
+    end
 
-      %w[web worker].include?(p) ? p : nil
+    # Per-process Sidekiq queue filter, e.g. TJ_SCALE_SIDEKIQ_QUEUES_BULK_WORKER=low
+    # for a +bulk-worker+ process; falls back to TJ_SCALE_SIDEKIQ_QUEUES.
+    def sidekiq_queues_key(process)
+      "TJ_SCALE_SIDEKIQ_QUEUES_#{process.to_s.upcase.gsub(/[^A-Z0-9]/, '_')}"
     end
 
     # "delayed_job" | "sidekiq" | "auto" (default; detects from loaded gems).
@@ -91,6 +111,17 @@ module TjScaleRuby
       web_queue_mutex.synchronize { @web_queue_time_ms }
     end
 
+    # Reads and clears the latest captured router queue time (ms). Consuming it each
+    # ingest tick prevents a stale elevated value from being re-reported (and keeping
+    # the web dyno scaled up) after traffic stops. Thread-safe.
+    def flush_web_queue_time_ms!
+      web_queue_mutex.synchronize do
+        v = @web_queue_time_ms
+        @web_queue_time_ms = nil
+        v
+      end
+    end
+
     def clear_web_queue_time_snapshot!
       web_queue_mutex.synchronize { @web_queue_time_ms = nil }
     end

data/lib/tj_scale_ruby/job_backends/delayed_job.rb

@@ -16,7 +16,10 @@ module TjScaleRuby
           min_priority = TjScaleRuby.configuration.min_priority
           max_priority = TjScaleRuby.configuration.max_priority
 
-          query = ::Delayed::Job.where("run_at + interval '5 seconds' < ?", now)
+          # 5-second buffer past run_at. Compute the cutoff in Ruby (run_at < now - 5s)
+          # instead of a SQL "interval" literal, which is PostgreSQL-only and would raise
+          # on MySQL/SQLite — silently masking the whole backlog as 0.
+          query = ::Delayed::Job.where("run_at < ?", now - 5)
           query = query.where("failed_at IS NULL")
           query = query.where("locked_at IS NULL")
           query = query.where("priority >= ?", min_priority) unless min_priority.zero?

data/lib/tj_scale_ruby/models/tj_scale_ruby.rb

@@ -80,7 +80,7 @@ module TjScaleRuby
       private
 
       def web_queue_time_ms_for_payload
-        snap = TjScaleRuby.last_web_queue_time_ms
+        snap = TjScaleRuby.flush_web_queue_time_ms!
         return snap unless snap.nil?
 
         env = ENV["TJ_SCALE_QUEUE_TIME_MS"]

data/lib/tj_scale_ruby/rack_queue_time_middleware.rb

@@ -29,14 +29,27 @@ module TjScaleRuby
       [now_ms - start_ms, 0].max
     end
 
+    # Heroku's router sends a bare unix-millisecond integer ("1751376000123"); nginx
+    # sends a "t=" prefixed value in seconds ("t=1751376000.123"). Match the prefixed
+    # form first, then fall back to a bare leading number, so the primary deployment
+    # target (Heroku) is not silently unparsed — a nil here means +queue_time_ms+ is
+    # never reported and web autoscaling on that metric can never fire.
     def self.request_start_epoch_ms(env)
       raw = env["HTTP_X_REQUEST_START"]
-      return nil if raw.nil? || raw.strip.empty?
+      return nil if raw.nil?
 
-      m = raw.to_s.match(/t=(\d+(?:\.\d+)?)/i)
+      s = raw.to_s.strip
+      return nil if s.empty?
+
+      # Heroku's router sends a bare millisecond epoch ("1731523200000"); nginx and
+      # some proxies send "t=<seconds>[.<frac>]". Accept both — prefer the +t=+ capture
+      # when present, otherwise fall back to the leading numeric timestamp.
+      m = s.match(/t=(\d+(?:\.\d+)?)/i) || s.match(/(\d+(?:\.\d+)?)/)
       return nil unless m
 
       v = m[1].to_f
+      return nil unless v.positive?
+
       v *= 1000 if v < 1_000_000_000_000 # seconds (or fractional) -> ms
       v.round
     end

data/lib/tj_scale_ruby/version.rb

@@ -2,5 +2,5 @@
 
 # Version module for TjScaleRuby gem
 module TjScaleRuby
-  VERSION = "1.0.2"
+  VERSION = "1.1.1"
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: tj-scale
 version: !ruby/object:Gem::Version
-  version: 1.0.2
+  version: 1.1.1
 platform: ruby
 authors:
 - Tanuj
 bindir: exe
 cert_chain: []
-date: 2026-06-12 00:00:00.000000000 Z
+date: 2026-07-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -57,6 +57,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '4.1'
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '7.3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '7.3'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement