tj-scale 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: e070c5d82758136d8df85df78f7f6b021f7494cfa8a2b64ff417aff9ad386333
4
+ data.tar.gz: d4cdd0e53528d48a634b8ee929c498cf63756ae7fd8be239ad6aaf277174052f
5
+ SHA512:
6
+ metadata.gz: 45b1804fc8ee06924b28d7230645fcb34d2d26fa4c37a412183cbfac6ff9bf82da108876d8012f275c774fc027e312ca0095b9bb6d0b89af6303023d0ae302fa
7
+ data.tar.gz: 46758b912e0a3ede622ed4ceae6ddb0b78c74b6dfe2cc4022fdf484892320db56204eea15600285bd7cbb6a09286974030b5ba73f983250cf3c3194ebd24632c
data/CHANGELOG.md ADDED
@@ -0,0 +1,77 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ### Changed
11
+
12
+ - **Gem renamed to `tj-scale`** (was `tj-scale-ruby`); require path is now `require "tj-scale"`. Internal `TjScaleRuby` module and `lib/tj_scale_ruby/` paths are unchanged.
13
+
14
+ ### Removed
15
+
16
+ - **`min_dynos` / `max_dynos` removed from the gem.** The control plane's dashboard settings are the single source of truth for scaling bounds. `TJ_SCALE_MIN_DYNOS` / `TJ_SCALE_MAX_DYNOS` are ignored, and payloads no longer include `min_dynos` / `max_dynos`.
17
+
18
+ ## [1.2.0] - 2026-06-12
19
+
20
+ ### Added
21
+
22
+ - **Sidekiq support.** Worker metrics now come from a pluggable queue backend: **Delayed Job** or **Sidekiq**, selected with `TJ_SCALE_JOB_BACKEND` (`delayed_job` / `sidekiq` / `auto`). The default `auto` prefers Sidekiq when the host app has it loaded, otherwise Delayed Job — existing Delayed Job apps need no changes.
23
+ - `TJ_SCALE_SIDEKIQ_QUEUES` — comma-separated list of Sidekiq queues to count (empty = all queues). Sidekiq `queue_time_s` is the latency of the slowest monitored queue.
24
+ - Worker payloads include a `job_backend` field (`"delayed_job"` or `"sidekiq"`).
25
+ - `TjScaleRuby::JobMonitor.backend` exposes the active backend; `TjScaleRuby::JobBackends.resolve/detect` for manual use.
26
+ - **Control plane (tj-scale-dashboard):** Slack notifications. Configure an incoming webhook under Settings → Slack notifications; the control plane posts a message whenever web or worker dynos scale up or down (auto or manual) and, optionally, when a scale attempt fails (throttled to one per app/process per 15 minutes). Includes per-event toggles, channel/username overrides, and a "Send test message" button.
27
+
28
+ ### Changed (repository layout)
29
+
30
+ - The Rails control plane moved out of this repo's `platform/` directory into the sibling **`tj-scale-dashboard`** project (Rails module renamed `Platform` → `TjScaleDashboard`).
31
+
32
+ ### Changed
33
+
34
+ - **`delayed_job_active_record` is no longer a runtime dependency.** Host apps bring their own queue gem (`delayed_job_active_record` or `sidekiq`); the gem lazily loads whichever backend is selected. If you rely on Delayed Job, make sure it is in your app's Gemfile (it almost certainly already is).
35
+ - Ingest requests send **`Authorization: Bearer`** as well as **`LOGPLEX_DRAIN_TOKEN`** (TJ Scale Agent API accepts either).
36
+
37
+ ## [1.1.0] - 2026-04-27
38
+
39
+ ### Changed
40
+
41
+ - **Breaking:** `TJ_SCALE_API_URL` is **required**; hardcoded default ingest URLs were removed.
42
+ - Documented end-to-end behavior in `lib/tj-scale-ruby.rb` (reporter vs control plane).
43
+ - Depend on **`openssl` >= 3.2** and `require "openssl"` before HTTP so metric POSTs avoid CRL verification failures on newer OpenSSL/Ruby combinations (same class of issue as ruby/openssl#949).
44
+
45
+ ### Removed
46
+
47
+ - `TjScaleRuby::JobMonitor.restart_dynos` and `TJ_SCALE_RESTART_URL` (platform ingest never performed restarts).
48
+ - `TJ_SCALE_METRIC_SOURCE` / heartbeat mode (sent misleading zeros and could interact badly with autoscale rules). Web metrics use the web reporter path + queue middleware instead of skipping DJ queries.
49
+
50
+ ## [1.0.1] - 2026-04-14
51
+
52
+ ### Changed
53
+
54
+ - Gemspec packaging includes only `lib/**` plus `README.md`, `CHANGELOG.md`, and `LICENSE.txt` (no Gemfile, Rakefile, or other dev-only root files).
55
+ - The monorepo `platform/` app and test directories stay out of the gem.
56
+ - When `.git` is missing, `gem build` still works using the same file list.
57
+ - Require `active_support/core_ext/module/delegation` before `rails/railtie` so the entrypoint loads reliably when Active Support is not yet loaded.
58
+
59
+ ## [1.0.0] - 2026-04-14
60
+
61
+ First public release of **tj-scale-ruby**.
62
+
63
+ ### Added
64
+
65
+ - Automatic Delayed Job queue depth reporting to a configurable HTTP endpoint (default interval 20s).
66
+ - Heroku-aware monitoring on the first `web.1` or `worker.1` dyno via `TJ_SCALE_MONITOR_PROCESS`.
67
+ - `TjScaleRuby::Configuration` for target app, process type, min/max dynos (defaults 2–8), interval, priority filters (`TJ_MIN_PRIORITY` / `TJ_MAX_PRIORITY`), and optional heartbeat-only mode (`TJ_SCALE_METRIC_SOURCE=none`; **removed in 1.1.0**).
68
+ - JSON payloads including `job_count`, `timestamp`, and when set `heroku_app`, `target_process`, `min_dynos`, `max_dynos`.
69
+ - `TjScaleRuby::JobMonitor.restart_dynos` for restart requests via the scaling service (**removed in 1.1.0**).
70
+ - `TjScaleRuby.monitor_on_this_dyno?` and `TjScaleRuby.start_monitoring_loop!` (Railtie starts the loop after Rails initializes).
71
+
72
+ ### Performance
73
+
74
+ - Single `Time.zone.now` per `send_log` for consistent timestamps and queue counts.
75
+ - Cached configuration for priority filters and Heroku app name (no repeated `ENV` reads in the hot path).
76
+ - Memoized `monitor_on_this_dyno?` and parsed API URIs; `Net::HTTP.start` for bounded connection lifecycle per request.
77
+ - Shared `http_post` helper for log and restart calls.
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2023-2024 Untechnickle
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,504 @@
1
+ # TjScaleRuby
2
+
3
+ [![Ruby Version](https://img.shields.io/badge/ruby-%3E%3D%203.0.0-red.svg)](https://www.ruby-lang.org/)
4
+ [![Rails Version](https://img.shields.io/badge/rails-%3E%3D%206.1-red.svg)](https://rubyonrails.org/)
5
+ [![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE.txt)
6
+
7
+ TjScaleRuby is a Ruby gem that automatically monitors your background job queues — **Delayed Job or Sidekiq** — and your web traffic, and sends metrics to a remote monitoring service for auto-scaling purposes. It integrates seamlessly with Rails applications and Heroku deployments.
8
+
9
+ ## Features
10
+
11
+ - 🔍 **Automatic Queue Monitoring**: Continuously monitors **Delayed Job or Sidekiq** queues for pending jobs
12
+ - 🔀 **Pluggable Job Backend**: Flip between Delayed Job and Sidekiq with one env var (`TJ_SCALE_JOB_BACKEND`); auto-detected by default
13
+ - 📊 **Metrics Reporting**: Sends job counts, queue latency, and web queue/response times to a remote monitoring service
14
+ - 🚀 **Heroku Integration**: Automatically runs on the first web or worker dyno
15
+ - ⚙️ **Configurable**: Customizable priority/queue filters and API endpoints
16
+ - 🔔 **Slack Notifications** (control plane): message your team whenever web or worker dynos scale up/down, or when scaling fails
17
+ - 🔒 **Error Handling**: Robust error handling with comprehensive logging
18
+ - 🧪 **Well Tested**: Comprehensive test suite with RSpec
19
+
20
+ ## Control plane (Rails + PostgreSQL)
21
+
22
+ The control plane lives in its own project, **`tj-scale-dashboard`** (a sibling directory / repository to this gem): dashboard, per-app scaling rules (web queue time + worker job counts), metrics ingest API compatible with the gem, Heroku formation updates, and **Slack notifications** for every scale event (Settings → Slack notifications). It uses **PostgreSQL** — see its `README.md` and `.env.example` in `../tj-scale-dashboard/`.
23
+
24
+ ## Installation
25
+
26
+ ### As a Gem
27
+
28
+ Add this line to your application's Gemfile:
29
+
30
+ ```ruby
31
+ gem "tj-scale-ruby", "~> 1.2"
32
+ ```
33
+
34
+ And then execute:
35
+
36
+ ```bash
37
+ $ bundle install
38
+ ```
39
+
40
+ Or install it yourself as:
41
+
42
+ ```bash
43
+ $ gem install tj-scale-ruby
44
+ ```
45
+
46
+ ### From Source
47
+
48
+ 1. Clone the repository:
49
+
50
+ ```bash
51
+ git clone https://github.com/untechnickle/tj-scale-ruby.git
52
+ cd tj-scale-ruby
53
+ ```
54
+
55
+ 2. Install dependencies:
56
+
57
+ ```bash
58
+ bundle install
59
+ ```
60
+
61
+ 3. Run the setup script:
62
+
63
+ ```bash
64
+ bin/setup
65
+ ```
66
+
67
+ ## End-to-end guide
68
+
69
+ Build the gem, publish or reference it, add it to your Rails app, and set configuration (env / Heroku). Follow these steps in order the first time you ship and adopt the gem.
70
+
71
+ ### 1. Prerequisites
72
+
73
+ - **Ruby** 3.0+ and **Bundler** 2+
74
+ - A **Rails** 6.1+ app with **one job backend**:
75
+ - **`delayed_job_active_record`** (and a DB-backed `delayed_jobs` table), **or**
76
+ - **`sidekiq`** (and Redis)
77
+ - **Heroku CLI** (if you deploy to Heroku)
78
+ - A **RubyGems.org** account (if you publish publicly)
79
+
80
+ ### 2. Prepare and build the gem (in this repository)
81
+
82
+ 1. **Track files for the gem package** — the gemspec uses `git ls-files`, so the repo must be a git repository and files must be committed:
83
+
84
+ ```bash
85
+ cd /path/to/tj-scale
86
+ git init # if needed
87
+ git add .
88
+ git commit -m "Release tj-scale-ruby"
89
+ ```
90
+
91
+ 2. **Install dependencies and verify quality**:
92
+
93
+ ```bash
94
+ bundle install
95
+ bundle exec rspec
96
+ bundle exec rake standard
97
+ ```
98
+
99
+ 3. **Set the version** in `lib/tj_scale_ruby/version.rb` (for example `1.0.0`).
100
+
101
+ 4. **Build the package**:
102
+
103
+ ```bash
104
+ gem build tj-scale-ruby.gemspec
105
+ ```
106
+
107
+ This produces `tj-scale-ruby-<version>.gem` in the current directory.
108
+
109
+ ### 3. Publish the gem (choose one)
110
+
111
+ **RubyGems.org (public)**
112
+
113
+ 1. Create an API key at [rubygems.org/api_keys](https://rubygems.org/api_keys).
114
+ 2. Push the file you built (version must match `VERSION`):
115
+
116
+ ```bash
117
+ gem push tj-scale-ruby-1.0.0.gem
118
+ ```
119
+
120
+ **Private gem host**
121
+
122
+ ```bash
123
+ gem push tj-scale-ruby-1.0.0.gem --host https://your-gem-server.com
124
+ ```
125
+
126
+ Configure Bundler in the app if your host needs authentication (see your provider’s docs).
127
+
128
+ **Without publishing — use the repo directly**
129
+
130
+ In the app’s `Gemfile`:
131
+
132
+ ```ruby
133
+ gem "tj-scale-ruby", git: "https://github.com/untechnickle/tj-scale-ruby.git", branch: "main"
134
+ # or path for local development:
135
+ # gem "tj-scale-ruby", path: "../tj-scale"
136
+ ```
137
+
138
+ Then `bundle install`.
139
+
140
+ ### 4. Add the gem to your Rails application
141
+
142
+ 1. Open the app’s **`Gemfile`** and add (adjust version or source as above):
143
+
144
+ ```ruby
145
+ gem "tj-scale-ruby", "~> 1.2"
146
+ ```
147
+
148
+ 2. Run:
149
+
150
+ ```bash
151
+ bundle install
152
+ ```
153
+
154
+ 3. **No extra `require`** is needed in `application.rb`: the gem registers a **Rails Railtie** and loads with Rails. Ensure your app already bundles its job backend — **`delayed_job_active_record`** or **`sidekiq`** (the gem auto-detects which one is loaded, or set `TJ_SCALE_JOB_BACKEND` explicitly).
155
+
156
+ 4. Deploy or run the server as usual. The monitoring thread starts only on **`web.1`** or **`worker.1`**, depending on `TJ_SCALE_MONITOR_PROCESS` (see below).
157
+
158
+ ### 5. Configure environment variables
159
+
160
+ Configuration is **entirely via environment variables** (and Heroku **config vars** in production). Copy [`.env.example`](.env.example) to `.env` for local use if you use **dotenv-rails** or similar.
161
+
162
+ | Variable | Required | Purpose |
163
+ |----------|----------|---------|
164
+ | `TJ_SCALE_API_TOKEN` | Yes | Sent as **`LOGPLEX_DRAIN_TOKEN`** and **`Authorization: Bearer …`** on each ingest POST. |
165
+ | `TJ_SCALE_API_URL` | Yes | Full URL of the Agent API ingest path: **`POST /api/v1/metrics`** or **`POST /api/v1/sys-logs`** (same handler). Example local: `http://127.0.0.1:5001/api/v1/metrics`. |
166
+ | `TJ_SCALE_TARGET_APP` | No | Heroku app name to scale; defaults to **`HEROKU_APP_NAME`** if unset. |
167
+ | `TJ_SCALE_TARGET_PROCESS` | No | `web` or `worker` to scale; defaults to `TJ_SCALE_MONITOR_PROCESS`. |
168
+ | `TJ_SCALE_MONITOR_PROCESS` | No | `web` or `worker` — only that process’s **`.1`** dyno runs the reporter (default **`worker`**). |
169
+ | `TJ_SCALE_INTERVAL_SECONDS` | No | Seconds between posts (default **20**). |
170
+ | `TJ_SCALE_JOB_BACKEND` | No | Queue backend for the **worker** reporter: `delayed_job`, `sidekiq`, or `auto` (default). `auto` uses Sidekiq when the `sidekiq` gem is loaded, otherwise Delayed Job. |
171
+ | `TJ_SCALE_SIDEKIQ_QUEUES` | No | Sidekiq backend only: comma-separated queue names to count (empty = all queues). |
172
+ | `TJ_MIN_PRIORITY` / `TJ_MAX_PRIORITY` | No | Limit which job priorities are counted (`0` = no filter; **worker** reporter, Delayed Job backend only). |
173
+ | `TJ_SCALE_QUEUE_TIME_MS` | No | When `TJ_SCALE_MONITOR_PROCESS=web`, supplies **`queue_time_ms`** if no middleware snapshot exists. |
174
+ | `TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE` | No | Set to **`1`**, **`true`**, **`yes`**, or **`on`** to enable Rack middleware that records **`queue_time_ms`** from **`X-Request-Start`** (Heroku). |
175
+
176
+ **Payload by reporter:** **`web`** — `queue_time_ms` (when known), `job_count` (requests since last tick), optional `response_time_ms`. **`worker`** — `job_count`, `queue_time_s` (oldest waiting job age / queue latency, or **0**), and `job_backend` (`delayed_job` or `sidekiq`). Both send `heroku_app` and `target_process`. Scaling limits (min/max dynos) are configured in the control plane's dashboard settings, not by this gem. Autoscaling runs on the **control plane** after ingest; this gem does not call Heroku.
177
+
178
+ Heroku sets **`DYNO`** (for example `web.1`, `worker.1`) and typically **`HEROKU_APP_NAME`**; do not set **`DYNO`** manually in production.
179
+
180
+ **Example: `revize-prod` (web) and `revize-prod-mirror` (workers)**
181
+
182
+ Use the same token on both apps if one backend handles scaling.
183
+
184
+ ```bash
185
+ # revize-prod — reporter on web.1, scale web dynos
186
+ heroku config:set TJ_SCALE_API_TOKEN=your_token \
187
+ TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics \
188
+ TJ_SCALE_TARGET_APP=revize-prod \
189
+ TJ_SCALE_TARGET_PROCESS=web \
190
+ TJ_SCALE_MONITOR_PROCESS=web \
191
+ -a revize-prod
192
+
193
+ # revize-prod-mirror — reporter on worker.1, scale worker dynos
194
+ heroku config:set TJ_SCALE_API_TOKEN=your_token \
195
+ TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics \
196
+ TJ_SCALE_TARGET_APP=revize-prod-mirror \
197
+ TJ_SCALE_TARGET_PROCESS=worker \
198
+ TJ_SCALE_MONITOR_PROCESS=worker \
199
+ -a revize-prod-mirror
200
+ ```
201
+
202
+ On the **web** app, enable **`TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1`** (and restart) so `queue_time_ms` is populated from Heroku’s router; the web dyno does not need a shared Delayed Job DB for that path.
203
+
204
+ ### 6. Wire your monitoring service (Agent API)
205
+
206
+ The TJ Scale **Agent API** accepts the same JSON on either route:
207
+
208
+ - **`POST /api/v1/metrics`**
209
+ - **`POST /api/v1/sys-logs`**
210
+
211
+ Set **`TJ_SCALE_API_URL`** to the full URL of one of them (for example `http://127.0.0.1:5001/api/v1/metrics` in development). Authenticate with the ingest token using **`LOGPLEX_DRAIN_TOKEN`** or **`Authorization: Bearer`** (same secret as **`TJ_SCALE_API_TOKEN`**); the gem sends **both** headers.
212
+
213
+ The gem POSTs JSON with **`timestamp`**, **`heroku_app`**, **`target_process`**, plus **web** metrics (`queue_time_ms`, `job_count`, optional `response_time_ms`) or **worker** metrics (`job_count`, `queue_time_s`). Your control plane ingests samples and applies scaling rules (thresholds, cooldowns) when calling the Heroku Platform API.
214
+
215
+ ### 7. Verify
216
+
217
+ - At least one **web** dyno on `revize-prod` and one **worker** dyno on `revize-prod-mirror` so **`web.1`** and **`worker.1`** exist.
218
+ - Check **Rails logs** for `TjScaleRuby` lines (debug/warn/error).
219
+ - Confirm your API receives periodic requests with the expected payload.
220
+
221
+ ### Optional: manual calls in Ruby
222
+
223
+ ```ruby
224
+ TjScaleRuby::JobMonitor.pending_job_count
225
+ TjScaleRuby::JobMonitor.oldest_pending_queue_seconds
226
+ TjScaleRuby.record_web_queue_time_ms!(42) # optional; middleware does this on each request
227
+ TjScaleRuby::JobMonitor.send_log
228
+ ```
229
+
230
+ ## Configuration
231
+
232
+ ### Environment Variables
233
+
234
+ Create a `.env` file in your project root (or set these in your Heroku config vars):
235
+
236
+ ```bash
237
+ # Required: API token and ingest URL for your control plane
238
+ TJ_SCALE_API_TOKEN=your_api_token_here
239
+ TJ_SCALE_API_URL=http://127.0.0.1:5001/api/v1/metrics
240
+
241
+ # --- Scaling metadata (sent to your monitoring API; defaults shown) ---
242
+ # Heroku app name to scale (defaults to HEROKU_APP_NAME when unset).
243
+ TJ_SCALE_TARGET_APP=my-heroku-app
244
+
245
+ # Process type to scale on that app: "web" or "worker" (defaults to TJ_SCALE_MONITOR_PROCESS).
246
+ TJ_SCALE_TARGET_PROCESS=web
247
+
248
+ # Which process type runs the reporter loop: only the first dyno of this type (web.1 or worker.1).
249
+ # Use "web" on the web app, "worker" on the worker app.
250
+ TJ_SCALE_MONITOR_PROCESS=worker
251
+
252
+ # Seconds between metric posts (default 20).
253
+ TJ_SCALE_INTERVAL_SECONDS=20
254
+
255
+ # Optional: Minimum priority filter for jobs (0 = no filter)
256
+ # Only count jobs with priority >= this value
257
+ TJ_MIN_PRIORITY=0
258
+
259
+ # Optional: Maximum priority filter for jobs (0 = no filter)
260
+ # Only count jobs with priority <= this value
261
+ TJ_MAX_PRIORITY=0
262
+ ```
263
+
264
+ Heroku sets `HEROKU_APP_NAME` for the current app. If you omit `TJ_SCALE_TARGET_APP`, the payload uses that value so each app reports scaling targets for itself.
265
+
266
+ ### Heroku Configuration
267
+
268
+ Set the environment variables on Heroku:
269
+
270
+ ```bash
271
+ heroku config:set TJ_SCALE_API_TOKEN=your_api_token_here
272
+ heroku config:set TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics
273
+ ```
274
+
275
+ The `DYNO` environment variable is automatically set by Heroku and is used to determine which dyno should run the monitoring loop (only `web.1` or `worker.1`, depending on `TJ_SCALE_MONITOR_PROCESS`).
276
+
277
+ ### Example: web on `revize-prod`, workers on `revize-prod-mirror`
278
+
279
+ Use two Heroku apps (or one app with both process types—same idea). Configure each app’s config vars so the **monitoring service** receives `heroku_app`, `target_process`, and `job_count` on each POST. Scaling bounds (min/max dynos) are set in the control plane's dashboard.
280
+
281
+ **App `revize-prod` (web dynos)** — run the reporter on the first web dyno and scale `web` dynos:
282
+
283
+ ```bash
284
+ heroku config:set TJ_SCALE_API_TOKEN=your_token -a revize-prod
285
+ heroku config:set TJ_SCALE_MONITOR_PROCESS=web -a revize-prod
286
+ heroku config:set TJ_SCALE_TARGET_APP=revize-prod -a revize-prod
287
+ heroku config:set TJ_SCALE_TARGET_PROCESS=web -a revize-prod
288
+ heroku config:set TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics -a revize-prod
289
+ heroku config:set TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1 -a revize-prod
290
+ ```
291
+
292
+ **App `revize-prod-mirror` (worker dynos)** — run the reporter on `worker.1` and scale `worker` dynos:
293
+
294
+ ```bash
295
+ heroku config:set TJ_SCALE_API_TOKEN=your_token -a revize-prod-mirror
296
+ heroku config:set TJ_SCALE_MONITOR_PROCESS=worker -a revize-prod-mirror
297
+ heroku config:set TJ_SCALE_TARGET_APP=revize-prod-mirror -a revize-prod-mirror
298
+ heroku config:set TJ_SCALE_TARGET_PROCESS=worker -a revize-prod-mirror
299
+ heroku config:set TJ_SCALE_API_URL=https://your-control-plane.example.com/api/v1/metrics -a revize-prod-mirror
300
+ ```
301
+
302
+ Ensure both apps have the `web` / `worker` process types in the `Procfile` as needed, deploy, and scale at least one dyno of each monitored type so `web.1` / `worker.1` exist.
303
+
304
+ **Monitoring API:** Your receiver at `TJ_SCALE_API_URL` should read the JSON body and enforce the min/max dyno bounds from its own dashboard settings when calling the Heroku Platform API (or your own scaler). Fields sent include `job_count`, `timestamp`, and when set, `heroku_app` and `target_process`.
305
+
306
+ ## Usage
307
+
308
+ ### Automatic Monitoring
309
+
310
+ Once installed and configured, the gem starts monitoring on the first dyno of the configured process type (`web.1` or `worker.1`, see `TJ_SCALE_MONITOR_PROCESS`). The loop runs every `TJ_SCALE_INTERVAL_SECONDS` (default 20) and POSTs metrics to the configured API.
311
+
312
+ ### Manual Usage
313
+
314
+ You can also use the gem's methods manually in your code:
315
+
316
+ ```ruby
317
+ # Get the count of pending jobs
318
+ count = TjScaleRuby::JobMonitor.pending_job_count
319
+ puts "Pending jobs: #{count}"
320
+
321
+ # Send current job count to the monitoring service
322
+ response = TjScaleRuby::JobMonitor.send_log
323
+ if response
324
+ puts "Log sent successfully"
325
+ end
326
+
327
+ ```
328
+
329
+ ### Priority Filtering
330
+
331
+ You can filter jobs by priority using environment variables:
332
+
333
+ ```ruby
334
+ # Only count jobs with priority between 5 and 10
335
+ ENV['TJ_MIN_PRIORITY'] = '5'
336
+ ENV['TJ_MAX_PRIORITY'] = '10'
337
+ ```
338
+
339
+ ## How It Works
340
+
341
+ 1. **Initialization**: When your Rails application starts, the `Railtie` checks if it's running on the first monitored dyno (`web.1` or `worker.1`, per `TJ_SCALE_MONITOR_PROCESS`)
342
+ 2. **Background Thread**: If conditions are met, a background thread is started
343
+ 3. **Monitoring Loop**: Every N seconds (`TJ_SCALE_INTERVAL_SECONDS`), the thread POSTs a payload: on **worker** reporters it counts waiting jobs on the active backend; on **web** reporters it sends router queue time (middleware / env) plus request volume and average response time for the interval.
344
+ 4. **Job counting (worker reporters, Delayed Job backend)**: `pending_job_count` filters jobs based on:
345
+ - Jobs past their `run_at` time (with 5 second buffer)
346
+ - Jobs that haven't failed (`failed_at IS NULL`)
347
+ - Jobs that aren't locked (`locked_at IS NULL`)
348
+ - Optional priority range filters
349
+ 5. **Job counting (worker reporters, Sidekiq backend)**: `pending_job_count` sums enqueued jobs across queues (all queues, or only those in `TJ_SCALE_SIDEKIQ_QUEUES`), and `queue_time_s` is the latency of the slowest monitored queue.
350
+
351
+ ## Development
352
+
353
+ After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
354
+
355
+ ### Running Tests
356
+
357
+ ```bash
358
+ # Run all tests
359
+ bundle exec rspec
360
+
361
+ # Run tests with documentation format
362
+ bundle exec rspec --format documentation
363
+
364
+ # Run a specific test file
365
+ bundle exec rspec spec/lib/tj_scale_ruby/models/tj_scale_ruby_spec.rb
366
+ ```
367
+
368
+ ### Code Quality
369
+
370
+ The project uses [Standard](https://github.com/testdouble/standard) for code formatting:
371
+
372
+ ```bash
373
+ # Check code style
374
+ bundle exec rake standard
375
+
376
+ # Auto-fix code style issues
377
+ bundle exec rake standard:fix
378
+ ```
379
+
380
+ ## Building and Publishing the Gem
381
+
382
+ For a full checklist (git, tests, version, integrate in the app, Heroku), see **[End-to-end guide](#end-to-end-guide)** above.
383
+
384
+ ### Building the Gem
385
+
386
+ 1. Update the version in `lib/tj_scale_ruby/version.rb`
387
+ 2. Ensure files are tracked by git (the gemspec uses `git ls-files`).
388
+ 3. Build the gem:
389
+
390
+ ```bash
391
+ gem build tj-scale-ruby.gemspec
392
+ ```
393
+
394
+ This will create a `.gem` file in the current directory.
395
+
396
+ ### Publishing to RubyGems
397
+
398
+ 1. Make sure you have a RubyGems account
399
+ 2. Get your API key from https://rubygems.org/api_keys
400
+ 3. Configure your credentials:
401
+
402
+ ```bash
403
+ gem push --key your_api_key tj-scale-ruby-1.0.0.gem
404
+ ```
405
+
406
+ Or use the credentials file:
407
+
408
+ ```bash
409
+ # Create ~/.gem/credentials
410
+ # Add your API key:
411
+ # ---
412
+ # :rubygems_api_key: your_api_key_here
413
+
414
+ gem push tj-scale-ruby-1.0.0.gem
415
+ ```
416
+
417
+ ### Publishing to a Private Gem Server
418
+
419
+ If you're using a private gem server:
420
+
421
+ ```bash
422
+ gem push tj-scale-ruby-1.0.0.gem --host https://your-gem-server.com
423
+ ```
424
+
425
+ ## Deployment
426
+
427
+ ### Heroku Deployment
428
+
429
+ 1. Add the gem to your `Gemfile`:
430
+
431
+ ```ruby
432
+ gem "tj-scale-ruby", "~> 1.2"
433
+ ```
434
+
435
+ 2. Set the required environment variables:
436
+
437
+ ```bash
438
+ heroku config:set TJ_SCALE_API_TOKEN=your_token
439
+ ```
440
+
441
+ 3. Deploy your application:
442
+
443
+ ```bash
444
+ git push heroku main
445
+ ```
446
+
447
+ The gem will automatically start monitoring on the first worker dyno.
448
+
449
+ ### Docker Deployment
450
+
451
+ 1. Add the gem to your `Gemfile`
452
+ 2. Set environment variables in your Docker configuration
453
+ 3. The gem will automatically start when the Rails application initializes
454
+
455
+ ### Other Platforms
456
+
457
+ The gem works with any Rails application. Just ensure:
458
+ - Delayed Job or Sidekiq is configured
459
+ - Environment variables are set
460
+ - The application has worker dynos (for automatic monitoring)
461
+
462
+ ## Requirements
463
+
464
+ - Ruby >= 3.0.0
465
+ - Rails >= 6.1
466
+ - One job backend (worker reporters only; bring your own gem):
467
+ - delayed_job_active_record >= 4.1 and PostgreSQL, **or**
468
+ - sidekiq and Redis
469
+
470
+ ## Contributing
471
+
472
+ Bug reports and pull requests are welcome on GitHub at https://github.com/untechnickle/tj-scale-ruby.
473
+
474
+ 1. Fork the repository
475
+ 2. Create your feature branch (`git checkout -b feature/amazing-feature`)
476
+ 3. Make your changes
477
+ 4. Add tests for your changes
478
+ 5. Ensure all tests pass (`bundle exec rspec`)
479
+ 6. Ensure code style is correct (`bundle exec rake standard`)
480
+ 7. Commit your changes (`git commit -am 'Add some amazing feature'`)
481
+ 8. Push to the branch (`git push origin feature/amazing-feature`)
482
+ 9. Open a Pull Request
483
+
484
+ ## License
485
+
486
+ The gem is available as open source under the terms of the [MIT License](LICENSE.txt).
487
+
488
+ ## Changelog
489
+
490
+ See [CHANGELOG.md](CHANGELOG.md) for a list of changes and version history.
491
+
492
+ ## Support
493
+
494
+ For issues, questions, or contributions, please open an issue on the [GitHub repository](https://github.com/untechnickle/tj-scale-ruby/issues).
495
+
496
+ ## Authors
497
+
498
+ - **Tanuj** - *Initial work* - [tanuj@untechnickle.com](mailto:tanuj@untechnickle.com)
499
+
500
+ ## Acknowledgments
501
+
502
+ - Built for Untechnickle
503
+ - Works with [Delayed Job](https://github.com/collectiveidea/delayed_job) and [Sidekiq](https://github.com/sidekiq/sidekiq) for background job processing
504
+
data/lib/tj-scale.rb ADDED
@@ -0,0 +1,94 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Railties expect ActiveSupport core extensions (e.g. `delegate_missing_to`) before load.
4
+ require "active_support/core_ext/module/delegation"
5
+ require "rails/railtie"
6
+ require "tj_scale_ruby/configuration"
7
+ require "tj_scale_ruby/models/tj_scale_ruby"
8
+
9
+ # ## How this gem participates in autoscaling
10
+ #
11
+ # TJ Scale (the control plane) scales Heroku formation when ingested metrics cross rules
12
+ # you configure there. This gem only **reports signals**; it never calls Heroku directly.
13
+ #
14
+ # **Where it runs:** A single background thread on **one** dyno — the first dyno of the
15
+ # process you choose (`web.1` or `worker.1` via +TJ_SCALE_MONITOR_PROCESS+). Heroku sets +DYNO+.
16
+ #
17
+ # **What it sends:** On each tick (+TJ_SCALE_INTERVAL_SECONDS+), it POSTs JSON to
18
+ # +TJ_SCALE_API_URL+ (e.g. +/api/v1/metrics+ or +/api/v1/sys-logs+ — same handler on the
19
+ # platform) with +LOGPLEX_DRAIN_TOKEN+ and +Authorization: Bearer+ set from +TJ_SCALE_API_TOKEN+.
20
+ # Payload always
21
+ # includes +heroku_app+ and +target_process+; scaling limits (min/max dynos) live in the
22
+ # control plane's settings, not in this gem.
23
+ #
24
+ # - **Worker monitor** (+TJ_SCALE_MONITOR_PROCESS=worker+): counts waiting jobs
25
+ # (+job_count+, +queue_time_s+) on the configured queue backend — **Delayed Job** or **Sidekiq**,
26
+ # toggled with +TJ_SCALE_JOB_BACKEND+ (+delayed_job+ / +sidekiq+ / +auto+, the default, which
27
+ # prefers Sidekiq when loaded). Delayed Job needs +delayed_job_active_record+ and a shared DB;
28
+ # Sidekiq needs the +sidekiq+ gem and Redis (+TJ_SCALE_SIDEKIQ_QUEUES+ limits counted queues).
29
+ # - **Web monitor** (+TJ_SCALE_MONITOR_PROCESS=web+): sends +queue_time_ms+ from Rack middleware
30
+ # (router queue via +X-Request-Start+) when +TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1+, plus
31
+ # request volume and average +response_time_ms+ for the tick. Without middleware, set
32
+ # +TJ_SCALE_QUEUE_TIME_MS+ or the platform will not scale on queue time until data exists.
33
+ #
34
+ # **Platform side:** You create a monitored app, set thresholds (e.g. upscale above N jobs or ms),
35
+ # and supply a Heroku API token there. Scaling runs in jobs after each accepted ingest.
36
+ #
37
+ # Main entry point for TjScaleRuby gem
38
+ module TjScaleRuby
39
+ # Rails integration: optional Rack middleware (web queue timing) + metrics loop on +web.1+ / +worker.1+.
40
+ class Railtie < Rails::Railtie
41
+ initializer "tj_scale_ruby.rack_queue_time_middleware" do |app|
42
+ flag = ENV.fetch("TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE", "").to_s.strip.downcase
43
+ next unless %w[1 true yes on].include?(flag)
44
+
45
+ require "tj_scale_ruby/rack_queue_time_middleware"
46
+ app.middleware.insert(0, RackQueueTimeMiddleware)
47
+ end
48
+
49
+ config.after_initialize do
50
+ TjScaleRuby.start_monitoring_loop!
51
+ end
52
+ end
53
+
54
+ # Starts a background thread on the designated first dyno (see {#monitor_on_this_dyno?}).
55
+ def self.start_monitoring_loop!
56
+ return unless monitor_on_this_dyno?
57
+
58
+ interval = configuration.interval_seconds
59
+ Thread.new do
60
+ loop do
61
+ begin
62
+ JobMonitor.send_log
63
+ rescue StandardError => e
64
+ Rails.logger.error("TjScaleRuby: Error in monitoring loop: #{e.message}")
65
+ end
66
+
67
+ sleep(interval)
68
+ end
69
+ end
70
+ end
71
+
72
+ # True when this dyno should run the metrics loop (first of TJ_SCALE_MONITOR_PROCESS).
73
+ # Result is memoized; cleared by {.reset_configuration!}.
74
+ def self.monitor_on_this_dyno?
75
+ return @monitor_on_this_dyno if defined?(@monitor_on_this_dyno)
76
+
77
+ dyno = ENV["DYNO"]
78
+ @monitor_on_this_dyno = if dyno.nil? || dyno.empty?
79
+ false
80
+ else
81
+ parts = dyno.split(".", 2)
82
+ if parts.size != 2
83
+ false
84
+ else
85
+ process, index = parts
86
+ process == configuration.monitor_process && index.to_i == 1
87
+ end
88
+ end
89
+ end
90
+
91
+ def self.clear_monitoring_memo!
92
+ remove_instance_variable(:@monitor_on_this_dyno) if defined?(@monitor_on_this_dyno)
93
+ end
94
+ end
@@ -0,0 +1,140 @@
1
+ # frozen_string_literal: true
2
+
3
+ module TjScaleRuby
4
+ # Reads Heroku / scaling settings from environment variables.
5
+ class Configuration
6
+ DEFAULT_INTERVAL_SECONDS = 20
7
+ JOB_BACKENDS = %w[auto delayed_job sidekiq].freeze
8
+
9
+ attr_reader :monitor_process, :interval_seconds,
10
+ :min_priority, :max_priority, :job_backend, :sidekiq_queues
11
+
12
+ def initialize(env = ENV)
13
+ @monitor_process = normalize_process(env["TJ_SCALE_MONITOR_PROCESS"]) || "worker"
14
+ @job_backend = normalize_job_backend(env["TJ_SCALE_JOB_BACKEND"])
15
+ @sidekiq_queues = parse_list(env["TJ_SCALE_SIDEKIQ_QUEUES"])
16
+ @interval_seconds = parse_positive_int(env["TJ_SCALE_INTERVAL_SECONDS"], DEFAULT_INTERVAL_SECONDS)
17
+ @target_app = nonempty_string(env["TJ_SCALE_TARGET_APP"])
18
+ @heroku_app_name = nonempty_string(env["HEROKU_APP_NAME"])
19
+ @target_process = normalize_process(env["TJ_SCALE_TARGET_PROCESS"]) || @monitor_process
20
+ @min_priority = env["TJ_MIN_PRIORITY"].to_i
21
+ @max_priority = env["TJ_MAX_PRIORITY"].to_i
22
+ end
23
+
24
+ # Heroku app name the scaler should act on (defaults to HEROKU_APP_NAME when unset).
25
+ def target_app
26
+ @target_app || @heroku_app_name
27
+ end
28
+
29
+ # Process type to scale on that app: "web" or "worker".
30
+ def target_process
31
+ @target_process
32
+ end
33
+
34
+ private
35
+
36
+ def normalize_process(value)
37
+ p = value.to_s.strip.downcase
38
+ return nil if p.empty?
39
+
40
+ %w[web worker].include?(p) ? p : nil
41
+ end
42
+
43
+ # "delayed_job" | "sidekiq" | "auto" (default; detects from loaded gems).
44
+ def normalize_job_backend(value)
45
+ b = value.to_s.strip.downcase
46
+ JOB_BACKENDS.include?(b) ? b : "auto"
47
+ end
48
+
49
+ def parse_list(value)
50
+ value.to_s.split(",").map(&:strip).reject(&:empty?)
51
+ end
52
+
53
+ def parse_positive_int(raw, default)
54
+ n = raw.to_i
55
+ n.positive? ? n : default
56
+ end
57
+
58
+ def nonempty_string(value)
59
+ s = value.to_s.strip
60
+ s.empty? ? nil : s
61
+ end
62
+ end
63
+
64
+ class << self
65
+ def configuration
66
+ @configuration ||= Configuration.new
67
+ end
68
+
69
+ def configuration=(config)
70
+ @configuration = config
71
+ end
72
+
73
+ def reset_configuration!
74
+ @configuration = nil
75
+ TjScaleRuby.clear_monitoring_memo!
76
+ TjScaleRuby.clear_web_queue_time_snapshot!
77
+ TjScaleRuby.clear_web_traffic_stats!
78
+ end
79
+
80
+ # Latest web router queue time (ms) captured by {RackQueueTimeMiddleware}; thread-safe.
81
+ def record_web_queue_time_ms!(ms)
82
+ v = Integer(ms)
83
+ return if v.negative?
84
+
85
+ web_queue_mutex.synchronize { @web_queue_time_ms = v }
86
+ rescue ArgumentError, TypeError
87
+ nil
88
+ end
89
+
90
+ def last_web_queue_time_ms
91
+ web_queue_mutex.synchronize { @web_queue_time_ms }
92
+ end
93
+
94
+ def clear_web_queue_time_snapshot!
95
+ web_queue_mutex.synchronize { @web_queue_time_ms = nil }
96
+ end
97
+
98
+ # Request volume + response time for +web+ metrics (flushed each ingest tick).
99
+ def record_web_request_finished!(duration_ms)
100
+ v = Integer(duration_ms)
101
+ return if v.negative?
102
+
103
+ web_traffic_mutex.synchronize do
104
+ @web_request_count = (@web_request_count || 0) + 1
105
+ @web_response_time_sum_ms = (@web_response_time_sum_ms || 0) + v
106
+ end
107
+ rescue ArgumentError, TypeError
108
+ nil
109
+ end
110
+
111
+ # @return [Hash] +:job_count+ requests since last flush; +:response_time_ms+ average ms or +nil+
112
+ def flush_web_traffic_stats_for_payload!
113
+ web_traffic_mutex.synchronize do
114
+ c = @web_request_count || 0
115
+ sum = @web_response_time_sum_ms || 0
116
+ @web_request_count = 0
117
+ @web_response_time_sum_ms = 0
118
+ avg = c.positive? ? (sum.to_f / c).round : nil
119
+ { job_count: c, response_time_ms: avg }
120
+ end
121
+ end
122
+
123
+ def clear_web_traffic_stats!
124
+ web_traffic_mutex.synchronize do
125
+ @web_request_count = 0
126
+ @web_response_time_sum_ms = 0
127
+ end
128
+ end
129
+
130
+ def web_queue_mutex
131
+ @web_queue_mutex ||= Mutex.new
132
+ end
133
+ private :web_queue_mutex
134
+
135
+ def web_traffic_mutex
136
+ @web_traffic_mutex ||= Mutex.new
137
+ end
138
+ private :web_traffic_mutex
139
+ end
140
+ end
@@ -0,0 +1,50 @@
1
+ # frozen_string_literal: true
2
+
3
+ module TjScaleRuby
4
+ module JobBackends
5
+ # Counts runnable +Delayed::Job+ rows in the shared database.
6
+ # Requires the host app to bundle +delayed_job_active_record+.
7
+ class DelayedJob
8
+ class << self
9
+ def backend_name
10
+ "delayed_job"
11
+ end
12
+
13
+ # Relation for runnable Delayed::Job rows (priority filters applied).
14
+ def pending_jobs_relation(now = Time.zone.now)
15
+ ensure_loaded!
16
+ min_priority = TjScaleRuby.configuration.min_priority
17
+ max_priority = TjScaleRuby.configuration.max_priority
18
+
19
+ query = ::Delayed::Job.where("run_at + interval '5 seconds' < ?", now)
20
+ query = query.where("failed_at IS NULL")
21
+ query = query.where("locked_at IS NULL")
22
+ query = query.where("priority >= ?", min_priority) unless min_priority.zero?
23
+ query = query.where("priority <= ?", max_priority) unless max_priority.zero?
24
+
25
+ query
26
+ end
27
+
28
+ def pending_job_count(now = Time.zone.now)
29
+ pending_jobs_relation(now).count
30
+ end
31
+
32
+ # Approximate age in seconds of the oldest runnable job (+run_at+ vs +now+). Zero if none.
33
+ def oldest_pending_queue_seconds(now = Time.zone.now)
34
+ oldest = pending_jobs_relation(now).minimum(:run_at)
35
+ return 0 if oldest.nil?
36
+
37
+ [(now - oldest).to_f.ceil, 0].max
38
+ end
39
+
40
+ private
41
+
42
+ def ensure_loaded!
43
+ return if defined?(::Delayed::Job)
44
+
45
+ require "delayed/backend/active_record"
46
+ end
47
+ end
48
+ end
49
+ end
50
+ end
@@ -0,0 +1,43 @@
1
+ # frozen_string_literal: true
2
+
3
+ module TjScaleRuby
4
+ module JobBackends
5
+ # Reads queue depth and latency from the Sidekiq API (Redis).
6
+ # Requires the host app to bundle +sidekiq+; priority filters do not apply —
7
+ # use +TJ_SCALE_SIDEKIQ_QUEUES+ to limit which queues are counted.
8
+ class Sidekiq
9
+ class << self
10
+ def backend_name
11
+ "sidekiq"
12
+ end
13
+
14
+ def pending_job_count(_now = nil)
15
+ ensure_loaded!
16
+ queues.sum(&:size)
17
+ end
18
+
19
+ # Latency (seconds since the oldest enqueued job) of the slowest monitored queue.
20
+ def oldest_pending_queue_seconds(_now = nil)
21
+ ensure_loaded!
22
+ latency = queues.map(&:latency).max
23
+ latency.nil? ? 0 : latency.ceil
24
+ end
25
+
26
+ private
27
+
28
+ def queues
29
+ names = TjScaleRuby.configuration.sidekiq_queues
30
+ return ::Sidekiq::Queue.all if names.empty?
31
+
32
+ names.map { |name| ::Sidekiq::Queue.new(name) }
33
+ end
34
+
35
+ def ensure_loaded!
36
+ return if defined?(::Sidekiq::Queue)
37
+
38
+ require "sidekiq/api"
39
+ end
40
+ end
41
+ end
42
+ end
43
+ end
@@ -0,0 +1,27 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "tj_scale_ruby/job_backends/delayed_job"
4
+ require "tj_scale_ruby/job_backends/sidekiq"
5
+
6
+ module TjScaleRuby
7
+ # Pluggable queue backends for worker metrics. Selected with
8
+ # +TJ_SCALE_JOB_BACKEND+ (+delayed_job+, +sidekiq+, or +auto+ — the default).
9
+ # +auto+ prefers Sidekiq when the host app has it loaded, else Delayed Job.
10
+ module JobBackends
11
+ def self.current
12
+ resolve(TjScaleRuby.configuration.job_backend)
13
+ end
14
+
15
+ def self.resolve(name)
16
+ case name.to_s
17
+ when "sidekiq" then Sidekiq
18
+ when "delayed_job" then DelayedJob
19
+ else detect
20
+ end
21
+ end
22
+
23
+ def self.detect
24
+ defined?(::Sidekiq) ? Sidekiq : DelayedJob
25
+ end
26
+ end
27
+ end
@@ -0,0 +1,134 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "openssl"
4
+ require "net/http"
5
+ require "uri"
6
+ require "json"
7
+ require "tj_scale_ruby/job_backends"
8
+
9
+ module TjScaleRuby
10
+ # Posts metrics to your TJ Scale (or compatible) ingest URL. Scaling decisions happen on the server.
11
+ class JobMonitor
12
+ class << self
13
+ # Active queue backend (Delayed Job or Sidekiq) per +TJ_SCALE_JOB_BACKEND+.
14
+ def backend
15
+ JobBackends.current
16
+ end
17
+
18
+ # Relation for runnable Delayed::Job rows (Delayed Job backend only; kept for manual use).
19
+ def pending_jobs_relation(now = Time.zone.now)
20
+ JobBackends::DelayedJob.pending_jobs_relation(now)
21
+ end
22
+
23
+ # Counts jobs waiting to run on the active backend (DJ: runnable rows with priority
24
+ # filters; Sidekiq: enqueued jobs across monitored queues).
25
+ def pending_job_count(now = Time.zone.now)
26
+ backend.pending_job_count(now)
27
+ rescue StandardError => e
28
+ Rails.logger.error("TjScaleRuby: Error counting pending jobs: #{e.message}")
29
+ 0
30
+ end
31
+
32
+ # Age in seconds of the oldest waiting job (DJ: +run_at+ vs +now+; Sidekiq: queue latency).
33
+ def oldest_pending_queue_seconds(now = Time.zone.now)
34
+ backend.oldest_pending_queue_seconds(now)
35
+ rescue StandardError => e
36
+ Rails.logger.error("TjScaleRuby: Error computing queue_time_s: #{e.message}")
37
+ 0
38
+ end
39
+
40
+ def send_log
41
+ token = ENV["TJ_SCALE_API_TOKEN"]
42
+ unless token
43
+ Rails.logger.warn("TjScaleRuby: TJ_SCALE_API_TOKEN not configured")
44
+ return false
45
+ end
46
+
47
+ api_url = ENV["TJ_SCALE_API_URL"].to_s.strip
48
+ if api_url.empty?
49
+ Rails.logger.warn("TjScaleRuby: TJ_SCALE_API_URL not configured")
50
+ return false
51
+ end
52
+
53
+ now = Time.zone.now
54
+ cfg = TjScaleRuby.configuration
55
+ payload = build_log_payload(now)
56
+
57
+ response = http_post(URI.parse(api_url), body: payload.compact.to_json, token: token)
58
+
59
+ if response.is_a?(Net::HTTPSuccess)
60
+ if cfg.monitor_process == "web"
61
+ Rails.logger.debug do
62
+ "TjScaleRuby: Sent web metrics queue_time_ms=#{payload[:queue_time_ms].inspect}"
63
+ end
64
+ else
65
+ Rails.logger.debug do
66
+ "TjScaleRuby: Sent worker metrics job_count=#{payload[:job_count]} " \
67
+ "queue_time_s=#{payload[:queue_time_s]}"
68
+ end
69
+ end
70
+ else
71
+ Rails.logger.warn("TjScaleRuby: API returned #{response.code}: #{response.message}")
72
+ end
73
+
74
+ response
75
+ rescue StandardError => e
76
+ Rails.logger.error("TjScaleRuby: Error sending log: #{e.message}")
77
+ false
78
+ end
79
+
80
+ private
81
+
82
+ def web_queue_time_ms_for_payload
83
+ snap = TjScaleRuby.last_web_queue_time_ms
84
+ return snap unless snap.nil?
85
+
86
+ env = ENV["TJ_SCALE_QUEUE_TIME_MS"]
87
+ return nil if env.blank?
88
+
89
+ env.to_i
90
+ end
91
+
92
+ def build_log_payload(now)
93
+ cfg = TjScaleRuby.configuration
94
+ base = {
95
+ timestamp: now.iso8601,
96
+ heroku_app: cfg.target_app,
97
+ target_process: cfg.target_process
98
+ }
99
+
100
+ if cfg.monitor_process == "web"
101
+ payload = base.dup
102
+ qms = web_queue_time_ms_for_payload
103
+ payload[:queue_time_ms] = qms unless qms.nil?
104
+ traffic = TjScaleRuby.flush_web_traffic_stats_for_payload!
105
+ payload[:job_count] = traffic[:job_count]
106
+ payload[:response_time_ms] = traffic[:response_time_ms] if traffic[:response_time_ms]
107
+ payload
108
+ else
109
+ job_count = pending_job_count(now)
110
+ queue_time_s = oldest_pending_queue_seconds(now)
111
+ base.merge(job_count: job_count, queue_time_s: queue_time_s, job_backend: backend.backend_name)
112
+ end
113
+ end
114
+
115
+ def http_post(uri, body:, token:)
116
+ Net::HTTP.start(
117
+ uri.host,
118
+ uri.port,
119
+ use_ssl: uri.scheme == "https",
120
+ read_timeout: 10,
121
+ open_timeout: 10
122
+ ) do |http|
123
+ request = Net::HTTP::Post.new(uri)
124
+ request["Content-Type"] = "application/json"
125
+ request["Accept"] = "application/vnd.heroku+json; version=3"
126
+ request["LOGPLEX_DRAIN_TOKEN"] = token
127
+ request["Authorization"] = "Bearer #{token}"
128
+ request.body = body if body
129
+ http.request(request)
130
+ end
131
+ end
132
+ end
133
+ end
134
+ end
@@ -0,0 +1,45 @@
1
+ # frozen_string_literal: true
2
+
3
+ module TjScaleRuby
4
+ # Records approximate router queue wait from {Heroku's X-Request-Start}[https://devcenter.heroku.com/articles/http-routing#request-queueing]
5
+ # so +web.1+ metrics can send +queue_time_ms+ without relying on a static ENV value.
6
+ #
7
+ # Enable with +TJ_SCALE_ENABLE_QUEUE_TIME_MIDDLEWARE=1+ (see Railtie).
8
+ class RackQueueTimeMiddleware
9
+ def initialize(app)
10
+ @app = app
11
+ end
12
+
13
+ def call(env)
14
+ t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_millisecond)
15
+ ms = self.class.queue_wait_ms(env)
16
+ TjScaleRuby.record_web_queue_time_ms!(ms) unless ms.nil?
17
+ status, headers, body = @app.call(env)
18
+ elapsed = (Process.clock_gettime(Process::CLOCK_MONOTONIC, :float_millisecond) - t0).round
19
+ TjScaleRuby.record_web_request_finished!(elapsed)
20
+ [status, headers, body]
21
+ end
22
+
23
+ # @return [Integer, nil] milliseconds since request entered the router, or nil if header missing/invalid
24
+ def self.queue_wait_ms(env)
25
+ start_ms = request_start_epoch_ms(env)
26
+ return nil unless start_ms
27
+
28
+ now_ms = (Time.now.utc.to_f * 1000).round
29
+ [now_ms - start_ms, 0].max
30
+ end
31
+
32
+ def self.request_start_epoch_ms(env)
33
+ raw = env["HTTP_X_REQUEST_START"]
34
+ return nil if raw.nil? || raw.strip.empty?
35
+
36
+ m = raw.to_s.match(/t=(\d+(?:\.\d+)?)/i)
37
+ return nil unless m
38
+
39
+ v = m[1].to_f
40
+ v *= 1000 if v < 1_000_000_000_000 # seconds (or fractional) -> ms
41
+ v.round
42
+ end
43
+ private_class_method :request_start_epoch_ms
44
+ end
45
+ end
@@ -0,0 +1,6 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Version module for TjScaleRuby gem
4
+ module TjScaleRuby
5
+ VERSION = "1.0.0"
6
+ end
metadata ADDED
@@ -0,0 +1,209 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: tj-scale
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.0.0
5
+ platform: ruby
6
+ authors:
7
+ - Tanuj
8
+ bindir: exe
9
+ cert_chain: []
10
+ date: 2026-06-12 00:00:00.000000000 Z
11
+ dependencies:
12
+ - !ruby/object:Gem::Dependency
13
+ name: rails
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - ">="
17
+ - !ruby/object:Gem::Version
18
+ version: '6.1'
19
+ - - "<"
20
+ - !ruby/object:Gem::Version
21
+ version: '9'
22
+ type: :runtime
23
+ prerelease: false
24
+ version_requirements: !ruby/object:Gem::Requirement
25
+ requirements:
26
+ - - ">="
27
+ - !ruby/object:Gem::Version
28
+ version: '6.1'
29
+ - - "<"
30
+ - !ruby/object:Gem::Version
31
+ version: '9'
32
+ - !ruby/object:Gem::Dependency
33
+ name: openssl
34
+ requirement: !ruby/object:Gem::Requirement
35
+ requirements:
36
+ - - "~>"
37
+ - !ruby/object:Gem::Version
38
+ version: '3.2'
39
+ type: :runtime
40
+ prerelease: false
41
+ version_requirements: !ruby/object:Gem::Requirement
42
+ requirements:
43
+ - - "~>"
44
+ - !ruby/object:Gem::Version
45
+ version: '3.2'
46
+ - !ruby/object:Gem::Dependency
47
+ name: delayed_job_active_record
48
+ requirement: !ruby/object:Gem::Requirement
49
+ requirements:
50
+ - - "~>"
51
+ - !ruby/object:Gem::Version
52
+ version: '4.1'
53
+ type: :development
54
+ prerelease: false
55
+ version_requirements: !ruby/object:Gem::Requirement
56
+ requirements:
57
+ - - "~>"
58
+ - !ruby/object:Gem::Version
59
+ version: '4.1'
60
+ - !ruby/object:Gem::Dependency
61
+ name: bundler
62
+ requirement: !ruby/object:Gem::Requirement
63
+ requirements:
64
+ - - ">="
65
+ - !ruby/object:Gem::Version
66
+ version: '2.0'
67
+ - - "<"
68
+ - !ruby/object:Gem::Version
69
+ version: '5'
70
+ type: :development
71
+ prerelease: false
72
+ version_requirements: !ruby/object:Gem::Requirement
73
+ requirements:
74
+ - - ">="
75
+ - !ruby/object:Gem::Version
76
+ version: '2.0'
77
+ - - "<"
78
+ - !ruby/object:Gem::Version
79
+ version: '5'
80
+ - !ruby/object:Gem::Dependency
81
+ name: rake
82
+ requirement: !ruby/object:Gem::Requirement
83
+ requirements:
84
+ - - "~>"
85
+ - !ruby/object:Gem::Version
86
+ version: '13.0'
87
+ type: :development
88
+ prerelease: false
89
+ version_requirements: !ruby/object:Gem::Requirement
90
+ requirements:
91
+ - - "~>"
92
+ - !ruby/object:Gem::Version
93
+ version: '13.0'
94
+ - !ruby/object:Gem::Dependency
95
+ name: rspec
96
+ requirement: !ruby/object:Gem::Requirement
97
+ requirements:
98
+ - - "~>"
99
+ - !ruby/object:Gem::Version
100
+ version: '3.0'
101
+ type: :development
102
+ prerelease: false
103
+ version_requirements: !ruby/object:Gem::Requirement
104
+ requirements:
105
+ - - "~>"
106
+ - !ruby/object:Gem::Version
107
+ version: '3.0'
108
+ - !ruby/object:Gem::Dependency
109
+ name: standard
110
+ requirement: !ruby/object:Gem::Requirement
111
+ requirements:
112
+ - - "~>"
113
+ - !ruby/object:Gem::Version
114
+ version: '1.3'
115
+ type: :development
116
+ prerelease: false
117
+ version_requirements: !ruby/object:Gem::Requirement
118
+ requirements:
119
+ - - "~>"
120
+ - !ruby/object:Gem::Version
121
+ version: '1.3'
122
+ - !ruby/object:Gem::Dependency
123
+ name: webmock
124
+ requirement: !ruby/object:Gem::Requirement
125
+ requirements:
126
+ - - "~>"
127
+ - !ruby/object:Gem::Version
128
+ version: '3.0'
129
+ type: :development
130
+ prerelease: false
131
+ version_requirements: !ruby/object:Gem::Requirement
132
+ requirements:
133
+ - - "~>"
134
+ - !ruby/object:Gem::Version
135
+ version: '3.0'
136
+ - !ruby/object:Gem::Dependency
137
+ name: pry
138
+ requirement: !ruby/object:Gem::Requirement
139
+ requirements:
140
+ - - "~>"
141
+ - !ruby/object:Gem::Version
142
+ version: '0.14'
143
+ type: :development
144
+ prerelease: false
145
+ version_requirements: !ruby/object:Gem::Requirement
146
+ requirements:
147
+ - - "~>"
148
+ - !ruby/object:Gem::Version
149
+ version: '0.14'
150
+ description: |
151
+ TjScaleRuby reports metrics from your Rails app to an TJ Scale (or compatible)
152
+ control plane so it can autoscale Heroku formation from rules you configure there.
153
+
154
+ Behavior at a glance:
155
+ - Runs a background loop on exactly one dyno: web.1 or worker.1 (configurable).
156
+ - Worker reporter: waiting job count and oldest-job age (queue_time_s) from Delayed Job
157
+ or Sidekiq — pick with TJ_SCALE_JOB_BACKEND (auto-detected by default).
158
+ - Web reporter: router queue_time_ms (Rack middleware + X-Request-Start), request volume,
159
+ and average response time per reporting interval.
160
+ - Sends target Heroku app and process type on every POST; scaling limits
161
+ (min/max dynos) are configured in the control plane's dashboard settings.
162
+ - Configure with environment variables; no Rails initializer required.
163
+
164
+ Requires Rails 6.1+ and Ruby 3.0+. Bring your own queue gem: delayed_job_active_record
165
+ or sidekiq (worker reporters only).
166
+ email:
167
+ - tanuj@untechnickle.com
168
+ executables: []
169
+ extensions: []
170
+ extra_rdoc_files: []
171
+ files:
172
+ - CHANGELOG.md
173
+ - LICENSE.txt
174
+ - README.md
175
+ - lib/tj-scale.rb
176
+ - lib/tj_scale_ruby/configuration.rb
177
+ - lib/tj_scale_ruby/job_backends.rb
178
+ - lib/tj_scale_ruby/job_backends/delayed_job.rb
179
+ - lib/tj_scale_ruby/job_backends/sidekiq.rb
180
+ - lib/tj_scale_ruby/models/tj_scale_ruby.rb
181
+ - lib/tj_scale_ruby/rack_queue_time_middleware.rb
182
+ - lib/tj_scale_ruby/version.rb
183
+ homepage: https://github.com/Untechnickle/tj-scale-gem
184
+ licenses:
185
+ - MIT
186
+ metadata:
187
+ source_code_uri: https://github.com/Untechnickle/tj-scale-gem
188
+ changelog_uri: https://github.com/Untechnickle/tj-scale-gem/blob/main/CHANGELOG.md
189
+ bug_tracker_uri: https://github.com/Untechnickle/tj-scale-gem/issues
190
+ documentation_uri: https://github.com/Untechnickle/tj-scale-gem#readme
191
+ rdoc_options: []
192
+ require_paths:
193
+ - lib
194
+ required_ruby_version: !ruby/object:Gem::Requirement
195
+ requirements:
196
+ - - ">="
197
+ - !ruby/object:Gem::Version
198
+ version: 3.0.0
199
+ required_rubygems_version: !ruby/object:Gem::Requirement
200
+ requirements:
201
+ - - ">="
202
+ - !ruby/object:Gem::Version
203
+ version: '0'
204
+ requirements: []
205
+ rubygems_version: 3.6.2
206
+ specification_version: 4
207
+ summary: 'Rails gem: Delayed Job / Sidekiq queue metrics for Heroku auto-scaling (web
208
+ or worker dynos)'
209
+ test_files: []