solid_terminator 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +7 -0
- data/LICENSE.txt +21 -0
- data/README.md +262 -0
- data/lib/generators/solid_terminator/install_generator.rb +49 -0
- data/lib/generators/solid_terminator/templates/create_solid_queue_terminated_executions.rb +15 -0
- data/lib/generators/solid_terminator/templates/create_solid_queue_terminations.rb +10 -0
- data/lib/generators/solid_terminator/templates/initializer.rb +14 -0
- data/lib/solid_terminator/claimed_execution_patch.rb +32 -0
- data/lib/solid_terminator/configuration.rb +43 -0
- data/lib/solid_terminator/engine.rb +21 -0
- data/lib/solid_terminator/job_terminated.rb +5 -0
- data/lib/solid_terminator/monitor.rb +98 -0
- data/lib/solid_terminator/terminable.rb +48 -0
- data/lib/solid_terminator/terminated_execution.rb +27 -0
- data/lib/solid_terminator/termination.rb +11 -0
- data/lib/solid_terminator/thread_registry.rb +46 -0
- data/lib/solid_terminator/version.rb +5 -0
- data/lib/solid_terminator.rb +52 -0
- metadata +93 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: eebd10e8944a7298814632a848c9f0d9d858591384b0787c714a5f7e958d98a2
|
|
4
|
+
data.tar.gz: 072a76c8bc904536248aa0eeb2b3ed7ed9a0bb3e9f37c24655112bcf0894b190
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 4d6bc234cdd8edf4427e378bf155425492b45100115705ef289fba803a63fff236c7458560b00057c2e18c17384809654bca333b4e8e323f6862095e6869c720
|
|
7
|
+
data.tar.gz: 7928aa7bf48a113df6d440736556a24fb75b400e42221f57f9bda607ae952f61a4324aef4b15d4e9e5ee7eefba2b37975a22533bc2368b9f8f364f74f244781f
|
data/CHANGELOG.md
ADDED
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Meszaros Istvan-Abel
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,262 @@
|
|
|
1
|
+
# SolidTerminator
|
|
2
|
+
|
|
3
|
+
*Hasta la vista, ActiveJob.*
|
|
4
|
+
|
|
5
|
+
Terminate a specific in-progress [SolidQueue](https://github.com/rails/solid_queue) job without stopping the worker process or affecting any other running jobs.
|
|
6
|
+
|
|
7
|
+
SolidQueue has no native per-job termination. Sending `TERM`/`QUIT` to a worker kills every job running on it. SolidTerminator fills that gap by raising a custom exception on exactly the thread running the target job.
|
|
8
|
+
|
|
9
|
+
## How it works
|
|
10
|
+
|
|
11
|
+
1. **Web server** — `SolidTerminator.terminate!(active_job_id)` inserts a row into `solid_queue_terminations`.
|
|
12
|
+
2. **Worker** — one monitor thread per worker process polls the table every 5 seconds (configurable). When it finds a row matching a locally-running job, it raises `SolidTerminator::JobTerminated` on that job's thread via `Thread#raise`.
|
|
13
|
+
3. **Job** — jobs that include `SolidTerminator::Terminable` rescue the exception, write a `TerminatedExecution` audit record, and stop cleanly.
|
|
14
|
+
|
|
15
|
+
The monitor skips the DB query entirely when no jobs are running — zero overhead at idle.
|
|
16
|
+
|
|
17
|
+
If the job catches the exception internally and keeps running, the termination row stays in the table and the monitor raises again on the next poll. Termination keeps retrying until the job actually stops.
|
|
18
|
+
|
|
19
|
+
## Requirements
|
|
20
|
+
|
|
21
|
+
- Ruby >= 3.1
|
|
22
|
+
- Rails >= 7.1
|
|
23
|
+
- SolidQueue >= 1.0
|
|
24
|
+
|
|
25
|
+
## Installation
|
|
26
|
+
|
|
27
|
+
Add to your `Gemfile`:
|
|
28
|
+
|
|
29
|
+
```ruby
|
|
30
|
+
gem "solid_terminator"
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Run the install generator:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
bin/rails generate solid_terminator:install
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
This copies two migrations and an initializer. Then run migrations.
|
|
40
|
+
|
|
41
|
+
**Single-database apps** (SolidQueue shares the main database):
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
bin/rails db:migrate
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
**Multi-database apps** (SolidQueue uses a dedicated database, typically keyed `queue` in `database.yml`):
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
bin/rails db:migrate:queue
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
> If your queue database key is not `queue`, pass `--database=<key>` to the generator:
|
|
54
|
+
> ```bash
|
|
55
|
+
> bin/rails generate solid_terminator:install --database=jobs
|
|
56
|
+
> ```
|
|
57
|
+
|
|
58
|
+
The generator creates two tables:
|
|
59
|
+
|
|
60
|
+
| Table | Purpose |
|
|
61
|
+
|---|---|
|
|
62
|
+
| `solid_queue_terminations` | Termination signal — almost always empty |
|
|
63
|
+
| `solid_queue_terminated_executions` | Audit trail of every terminated job |
|
|
64
|
+
|
|
65
|
+
## Usage
|
|
66
|
+
|
|
67
|
+
### Make a job terminable
|
|
68
|
+
|
|
69
|
+
Include `SolidTerminator::Terminable` in any job you want to be able to terminate:
|
|
70
|
+
|
|
71
|
+
```ruby
|
|
72
|
+
class MyJob < ApplicationJob
|
|
73
|
+
include SolidTerminator::Terminable
|
|
74
|
+
|
|
75
|
+
def perform
|
|
76
|
+
loop do
|
|
77
|
+
do_some_work
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
end
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
The concern registers the job's thread when execution starts and deregisters it when execution ends — whether the job succeeds, fails, or is terminated.
|
|
84
|
+
|
|
85
|
+
### Graceful cleanup on termination
|
|
86
|
+
|
|
87
|
+
`JobTerminated` propagates through your job's `perform` like any other exception. Use `ensure` for cleanup that must always run, and `rescue` if you need to react to termination specifically:
|
|
88
|
+
|
|
89
|
+
```ruby
|
|
90
|
+
class MyJob < ApplicationJob
|
|
91
|
+
include SolidTerminator::Terminable
|
|
92
|
+
|
|
93
|
+
def perform
|
|
94
|
+
acquire_lock
|
|
95
|
+
do_long_work
|
|
96
|
+
rescue SolidTerminator::JobTerminated
|
|
97
|
+
release_lock # clean up before the job stops
|
|
98
|
+
raise # always re-raise so SolidTerminator can finish
|
|
99
|
+
ensure
|
|
100
|
+
close_connections # runs on success, failure, and termination
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
> **Important:** Always re-raise `JobTerminated` after your cleanup. If you swallow it, the termination row stays in the database and the monitor will raise again on the next poll until the job stops.
|
|
106
|
+
|
|
107
|
+
### Request termination
|
|
108
|
+
|
|
109
|
+
From a controller, background job, console, or anywhere in your app:
|
|
110
|
+
|
|
111
|
+
```ruby
|
|
112
|
+
SolidTerminator.terminate!(active_job_id)
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The job's thread receives `SolidTerminator::JobTerminated` within the next polling interval (default 5 seconds) and stops cleanly. Calling `terminate!` twice for the same job is safe — the duplicate is silently ignored.
|
|
116
|
+
|
|
117
|
+
**From a controller:**
|
|
118
|
+
|
|
119
|
+
```ruby
|
|
120
|
+
class JobsController < ApplicationController
|
|
121
|
+
def destroy
|
|
122
|
+
SolidTerminator.terminate!(params[:active_job_id])
|
|
123
|
+
head :accepted
|
|
124
|
+
end
|
|
125
|
+
end
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
**From the Rails console:**
|
|
129
|
+
|
|
130
|
+
```ruby
|
|
131
|
+
SolidTerminator.terminate!("9a7b3c2d-1234-5678-abcd-ef0123456789")
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### Find the active_job_id
|
|
135
|
+
|
|
136
|
+
`active_job_id` is the UUID assigned by ActiveJob. Capture it at enqueue time:
|
|
137
|
+
|
|
138
|
+
```ruby
|
|
139
|
+
job = MyJob.perform_later(args)
|
|
140
|
+
job.job_id # => "9a7b3c2d-..."
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
Find currently running jobs via SolidQueue's claimed executions:
|
|
144
|
+
|
|
145
|
+
```ruby
|
|
146
|
+
SolidQueue::ClaimedExecution
|
|
147
|
+
.joins(:job)
|
|
148
|
+
.where(solid_queue_jobs: { class_name: 'MyJob' })
|
|
149
|
+
.pluck('solid_queue_jobs.active_job_id')
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Or find any enqueued/running job by class:
|
|
153
|
+
|
|
154
|
+
```ruby
|
|
155
|
+
SolidQueue::Job.where(class_name: 'MyJob').pluck(:active_job_id)
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
## Terminated jobs
|
|
159
|
+
|
|
160
|
+
When a job is terminated, SolidTerminator writes a `SolidQueue::TerminatedExecution` record so you have a queryable audit trail.
|
|
161
|
+
|
|
162
|
+
### Query
|
|
163
|
+
|
|
164
|
+
```ruby
|
|
165
|
+
# All terminated jobs, newest first
|
|
166
|
+
SolidQueue::TerminatedExecution.ordered
|
|
167
|
+
|
|
168
|
+
# Filter by queue
|
|
169
|
+
SolidQueue::TerminatedExecution.where(queue_name: 'critical')
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Each record exposes: `job_id`, `queue_name`, `priority`, `terminated_at`.
|
|
173
|
+
|
|
174
|
+
### Retry a terminated job
|
|
175
|
+
|
|
176
|
+
Re-enqueue the original job with its original arguments:
|
|
177
|
+
|
|
178
|
+
```ruby
|
|
179
|
+
SolidQueue::TerminatedExecution.find_by(job_id: sq_job_id).retry
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
This destroys the `TerminatedExecution` record and enqueues a new job in a single transaction.
|
|
183
|
+
|
|
184
|
+
## Configuration
|
|
185
|
+
|
|
186
|
+
The generator creates `config/initializers/solid_terminator.rb` with all available options:
|
|
187
|
+
|
|
188
|
+
```ruby
|
|
189
|
+
SolidTerminator.configure do |config|
|
|
190
|
+
# How often the monitor thread polls for termination requests (seconds).
|
|
191
|
+
# config.polling_interval = 5 # default: 5
|
|
192
|
+
|
|
193
|
+
# Route logs to Rails.logger instead of a dedicated file.
|
|
194
|
+
# config.logger = Rails.logger
|
|
195
|
+
|
|
196
|
+
# Log file path. Ignored when config.logger is set.
|
|
197
|
+
# config.log_file = Rails.root.join("log", "solid_terminator.log") # default
|
|
198
|
+
|
|
199
|
+
# Log verbosity. Default: Logger::INFO.
|
|
200
|
+
# Logger::DEBUG adds per-poll trace lines; Logger::WARN suppresses routine messages.
|
|
201
|
+
# config.log_level = Logger::INFO
|
|
202
|
+
end
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
## Deployment topologies
|
|
206
|
+
|
|
207
|
+
Works across all SolidQueue topologies without any topology-specific configuration:
|
|
208
|
+
|
|
209
|
+
| Topology | Behaviour |
|
|
210
|
+
|---|---|
|
|
211
|
+
| Puma plugin (in-process) | Monitor starts with the worker thread pool |
|
|
212
|
+
| Separate `bin/jobs` process | One monitor per `bin/jobs` instance |
|
|
213
|
+
| Multiple worker processes on one host | One monitor per process; registry is process-local |
|
|
214
|
+
| Multiple hosts | Works automatically — each host's workers manage their own jobs |
|
|
215
|
+
|
|
216
|
+
The thread registry and `Thread#raise` always operate within the same OS process, so no cross-host signalling is needed. A termination row created on any host is picked up by whichever worker is running the job, because each monitor only queries for jobs in its own process-local registry.
|
|
217
|
+
|
|
218
|
+
## Design decisions
|
|
219
|
+
|
|
220
|
+
**Why a separate table instead of Redis or PostgreSQL LISTEN/NOTIFY?**
|
|
221
|
+
Works with PostgreSQL, MySQL, and SQLite out of the box — no extra infrastructure required. The table is almost always empty, so polling is essentially free.
|
|
222
|
+
|
|
223
|
+
**Why `Thread#raise` instead of process signals?**
|
|
224
|
+
Workers run a thread pool. A signal would interrupt every thread in the process; `Thread#raise` targets exactly the right one without disturbing other jobs.
|
|
225
|
+
|
|
226
|
+
**Why does `JobTerminated` inherit from `Exception` instead of `StandardError`?**
|
|
227
|
+
`rescue => e` only catches `StandardError`. Inheriting from `Exception` means the termination signal propagates through typical job error-handling code without being accidentally swallowed. Jobs that do have a `rescue Exception` handler will simply receive the exception again on the next poll until they stop.
|
|
228
|
+
|
|
229
|
+
**Why inherit from `SolidQueue::Record`?**
|
|
230
|
+
Ensures both tables live in the same database as the rest of SolidQueue, including multi-database setups that use `connects_to`.
|
|
231
|
+
|
|
232
|
+
## Roadmap
|
|
233
|
+
|
|
234
|
+
Planned improvements — no particular order:
|
|
235
|
+
|
|
236
|
+
**Signaling adapters**
|
|
237
|
+
- Redis pub/sub adapter — lower latency for stacks that already have Redis
|
|
238
|
+
- PostgreSQL `LISTEN/NOTIFY` adapter — zero extra infra for PG-only apps
|
|
239
|
+
- Pluggable adapter interface so custom adapters can be dropped in
|
|
240
|
+
|
|
241
|
+
**Termination control**
|
|
242
|
+
- Timeout-based auto-termination — automatically terminate a job that exceeds a configured runtime
|
|
243
|
+
- Grace period — wait N seconds after signaling before re-raising, allowing the job to reach a natural checkpoint
|
|
244
|
+
- Bulk termination — `SolidTerminator.terminate_all!(class_name:)` or by queue name
|
|
245
|
+
|
|
246
|
+
**Observability**
|
|
247
|
+
- ActiveSupport notifications (`solid_terminator.terminated`, `solid_terminator.monitor_error`) for APM integration and custom hooks
|
|
248
|
+
- `TerminatedExecution` retention policy and auto-cleanup of old records
|
|
249
|
+
- Termination status query — `SolidTerminator.termination_pending?(active_job_id)`
|
|
250
|
+
|
|
251
|
+
## Development
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
bundle install
|
|
255
|
+
bundle exec rspec # run tests
|
|
256
|
+
bundle exec rubocop # lint
|
|
257
|
+
bundle exec rubocop -A # lint with auto-fix
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
## License
|
|
261
|
+
|
|
262
|
+
MIT
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'rails/generators'
|
|
4
|
+
require 'rails/generators/active_record'
|
|
5
|
+
|
|
6
|
+
module SolidTerminator
|
|
7
|
+
module Generators
|
|
8
|
+
class InstallGenerator < Rails::Generators::Base
|
|
9
|
+
include Rails::Generators::Migration
|
|
10
|
+
|
|
11
|
+
source_root File.expand_path('templates', __dir__)
|
|
12
|
+
|
|
13
|
+
class_option :database,
|
|
14
|
+
type: :string,
|
|
15
|
+
aliases: '-d',
|
|
16
|
+
desc: 'Target database key from database.yml (default: auto-detected from queue DB config).'
|
|
17
|
+
|
|
18
|
+
def self.next_migration_number(dirname)
|
|
19
|
+
ActiveRecord::Generators::Base.next_migration_number(dirname)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def copy_migrations
|
|
23
|
+
migration_template \
|
|
24
|
+
'create_solid_queue_terminations.rb',
|
|
25
|
+
File.join(target_migrations_dir, 'create_solid_queue_terminations.rb')
|
|
26
|
+
|
|
27
|
+
migration_template \
|
|
28
|
+
'create_solid_queue_terminated_executions.rb',
|
|
29
|
+
File.join(target_migrations_dir, 'create_solid_queue_terminated_executions.rb')
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def create_initializer
|
|
33
|
+
template 'initializer.rb', 'config/initializers/solid_terminator.rb'
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
private
|
|
37
|
+
|
|
38
|
+
def target_migrations_dir
|
|
39
|
+
db_key = options[:database] || 'queue'
|
|
40
|
+
db_config = ActiveRecord::Base.configurations.find_db_config(db_key)
|
|
41
|
+
Array(db_config&.migrations_paths).first || 'db/migrate'
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def migration_version
|
|
45
|
+
"[#{ActiveRecord::Migration.current_version}]"
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
end
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
class CreateSolidQueueTerminatedExecutions < ActiveRecord::Migration<%= migration_version %>
|
|
2
|
+
def change
|
|
3
|
+
create_table :solid_queue_terminated_executions do |t|
|
|
4
|
+
t.bigint :job_id, null: false
|
|
5
|
+
t.integer :priority, default: 0, null: false
|
|
6
|
+
t.string :queue_name, null: false
|
|
7
|
+
t.datetime :terminated_at, null: false
|
|
8
|
+
t.datetime :created_at, null: false
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
add_index :solid_queue_terminated_executions, :job_id, unique: true
|
|
12
|
+
add_index :solid_queue_terminated_executions, %i[queue_name priority job_id],
|
|
13
|
+
name: 'index_solid_queue_terminated_executions_for_filtering'
|
|
14
|
+
end
|
|
15
|
+
end
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
class CreateSolidQueueTerminations < ActiveRecord::Migration<%= migration_version %>
|
|
2
|
+
def change
|
|
3
|
+
create_table :solid_queue_terminations do |t|
|
|
4
|
+
t.string :active_job_id, null: false
|
|
5
|
+
t.datetime :created_at, null: false
|
|
6
|
+
end
|
|
7
|
+
|
|
8
|
+
add_index :solid_queue_terminations, :active_job_id, unique: true
|
|
9
|
+
end
|
|
10
|
+
end
|
|
@@ -0,0 +1,14 @@
|
|
|
1
|
+
SolidTerminator.configure do |config|
|
|
2
|
+
# How often the monitor thread polls for termination requests (seconds).
|
|
3
|
+
# config.polling_interval = 5
|
|
4
|
+
|
|
5
|
+
# Use Rails.logger (or any Logger-compatible object) instead of a dedicated file.
|
|
6
|
+
# config.logger = Rails.logger
|
|
7
|
+
|
|
8
|
+
# Log file path. Defaults to log/solid_terminator.log. Ignored when config.logger is set.
|
|
9
|
+
# config.log_file = Rails.root.join("log", "solid_terminator.log")
|
|
10
|
+
|
|
11
|
+
# Log level. Defaults to Logger::INFO.
|
|
12
|
+
# Use Logger::DEBUG for verbose output, Logger::WARN to suppress routine messages.
|
|
13
|
+
# config.log_level = Logger::INFO
|
|
14
|
+
end
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidTerminator
|
|
4
|
+
module ClaimedExecutionPatch
|
|
5
|
+
def perform
|
|
6
|
+
result = execute
|
|
7
|
+
handle_result(result)
|
|
8
|
+
ensure
|
|
9
|
+
unblock_next_job
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
private
|
|
13
|
+
|
|
14
|
+
def handle_result(result)
|
|
15
|
+
if result.success?
|
|
16
|
+
finished
|
|
17
|
+
elsif result.error.is_a?(SolidTerminator::JobTerminated)
|
|
18
|
+
finish_terminated
|
|
19
|
+
else
|
|
20
|
+
failed_with(result.error)
|
|
21
|
+
raise result.error
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def finish_terminated
|
|
26
|
+
transaction do
|
|
27
|
+
job.finished!
|
|
28
|
+
destroy!
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
end
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'logger'
|
|
4
|
+
|
|
5
|
+
module SolidTerminator
|
|
6
|
+
class Configuration
|
|
7
|
+
attr_accessor :polling_interval
|
|
8
|
+
attr_reader :log_file, :log_level
|
|
9
|
+
|
|
10
|
+
def initialize
|
|
11
|
+
@polling_interval = 5
|
|
12
|
+
@log_level = Logger::INFO
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
def log_file=(path)
|
|
16
|
+
@log_file = path
|
|
17
|
+
@logger = nil
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def log_level=(level)
|
|
21
|
+
@log_level = level
|
|
22
|
+
@logger.level = level if @logger && !@external_logger
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def logger=(logger)
|
|
26
|
+
@logger = logger
|
|
27
|
+
@external_logger = true
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def logger
|
|
31
|
+
@logger ||= Logger.new(log_file || default_log_path).tap do |l|
|
|
32
|
+
l.progname = 'SolidTerminator'
|
|
33
|
+
l.level = @log_level
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
private
|
|
38
|
+
|
|
39
|
+
def default_log_path
|
|
40
|
+
defined?(Rails) && Rails.root ? Rails.root.join('log', 'solid_terminator.log') : $stdout
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'rails'
|
|
4
|
+
|
|
5
|
+
module SolidTerminator
|
|
6
|
+
class Engine < Rails::Engine
|
|
7
|
+
config.to_prepare do
|
|
8
|
+
require 'solid_terminator/termination'
|
|
9
|
+
require 'solid_terminator/terminated_execution'
|
|
10
|
+
require 'solid_terminator/claimed_execution_patch'
|
|
11
|
+
SolidQueue::ClaimedExecution.prepend(SolidTerminator::ClaimedExecutionPatch)
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
config.after_initialize do
|
|
15
|
+
if defined?(SolidQueue)
|
|
16
|
+
SolidQueue.on_worker_start { SolidTerminator.start_monitor }
|
|
17
|
+
SolidQueue.on_worker_stop { SolidTerminator.stop_monitor }
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidTerminator
|
|
4
|
+
class Monitor
|
|
5
|
+
def initialize(interval: SolidTerminator.configuration.polling_interval)
|
|
6
|
+
@interval = interval
|
|
7
|
+
@mutex = Mutex.new
|
|
8
|
+
@stopped = false
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def start
|
|
12
|
+
logger.info("Monitor starting (polling every #{@interval}s)")
|
|
13
|
+
@thread = Thread.new { run }
|
|
14
|
+
@thread.name = 'SolidTerminator::Monitor'
|
|
15
|
+
@thread
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def stop
|
|
19
|
+
logger.info('Monitor stopping')
|
|
20
|
+
@mutex.synchronize { @stopped = true }
|
|
21
|
+
begin
|
|
22
|
+
@thread&.wakeup
|
|
23
|
+
rescue ThreadError
|
|
24
|
+
nil
|
|
25
|
+
end
|
|
26
|
+
@thread&.join(@interval + 2)
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
private
|
|
30
|
+
|
|
31
|
+
def stopped?
|
|
32
|
+
@mutex.synchronize { @stopped }
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def run
|
|
36
|
+
retries = 0
|
|
37
|
+
until stopped?
|
|
38
|
+
check_for_terminations
|
|
39
|
+
sleep @interval
|
|
40
|
+
end
|
|
41
|
+
rescue ActiveRecord::StatementInvalid => e
|
|
42
|
+
logger.error("Monitor stopped — table missing, run migrations. #{e.message}")
|
|
43
|
+
rescue StandardError => e
|
|
44
|
+
retries = on_run_error(e, retries)
|
|
45
|
+
retry unless stopped?
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def on_run_error(error, retries)
|
|
49
|
+
retries += 1
|
|
50
|
+
logger.error("Monitor error (attempt #{retries}): #{error.class}: #{error.message}. Retrying in #{@interval}s.")
|
|
51
|
+
sleep @interval
|
|
52
|
+
retries
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def check_for_terminations
|
|
56
|
+
job_ids = ThreadRegistry.snapshot
|
|
57
|
+
if job_ids.nil?
|
|
58
|
+
logger.debug('Skipping — no jobs registered')
|
|
59
|
+
else
|
|
60
|
+
process_terminations(job_ids)
|
|
61
|
+
ThreadRegistry.purge_dead_threads
|
|
62
|
+
end
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
def process_terminations(job_ids)
|
|
66
|
+
return if job_ids.empty?
|
|
67
|
+
|
|
68
|
+
logger.debug("Checking for terminations (#{job_ids.size} job(s) registered)")
|
|
69
|
+
Termination.for_jobs(job_ids).each { |t| signal_termination(t) }
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
def signal_termination(termination)
|
|
73
|
+
thread = ThreadRegistry.thread_for(termination.active_job_id)
|
|
74
|
+
return if thread.nil?
|
|
75
|
+
|
|
76
|
+
return clean_stale_entry(termination) unless thread.alive?
|
|
77
|
+
|
|
78
|
+
logger.info("Raising termination on job #{termination.active_job_id}")
|
|
79
|
+
begin
|
|
80
|
+
thread.raise(JobTerminated, "Job #{termination.active_job_id} terminated")
|
|
81
|
+
rescue ThreadError
|
|
82
|
+
logger.warn("Thread for job #{termination.active_job_id} died before signal; will retry next poll")
|
|
83
|
+
end
|
|
84
|
+
# Row stays: job's ensure deletes it on clean exit; if the job swallows the exception
|
|
85
|
+
# the row persists so the monitor retries next poll rather than silently giving up.
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
def clean_stale_entry(termination)
|
|
89
|
+
logger.warn("Thread for job #{termination.active_job_id} is dead; cleaning up stale entry")
|
|
90
|
+
ThreadRegistry.deregister(termination.active_job_id)
|
|
91
|
+
termination.destroy
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def logger
|
|
95
|
+
SolidTerminator.configuration.logger
|
|
96
|
+
end
|
|
97
|
+
end
|
|
98
|
+
end
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidTerminator
|
|
4
|
+
module Terminable
|
|
5
|
+
extend ActiveSupport::Concern
|
|
6
|
+
|
|
7
|
+
included do
|
|
8
|
+
around_perform :with_termination_handling
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
private
|
|
12
|
+
|
|
13
|
+
def with_termination_handling
|
|
14
|
+
current_thread = Thread.current
|
|
15
|
+
st_logger.debug("Job #{job_id} registered for termination monitoring")
|
|
16
|
+
ThreadRegistry.register(job_id)
|
|
17
|
+
yield
|
|
18
|
+
rescue JobTerminated => e
|
|
19
|
+
on_job_terminated(current_thread, e)
|
|
20
|
+
ensure
|
|
21
|
+
ThreadRegistry.deregister(job_id, current_thread)
|
|
22
|
+
st_logger.debug("Job #{job_id} deregistered from termination monitoring")
|
|
23
|
+
cleanup_termination_row
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def on_job_terminated(current_thread, error)
|
|
27
|
+
# Deregister before cleanup so the monitor does not re-raise mid-cleanup.
|
|
28
|
+
ThreadRegistry.deregister(job_id, current_thread)
|
|
29
|
+
handle_job_terminated(error)
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def handle_job_terminated(error)
|
|
33
|
+
st_logger.info("Job #{job_id} terminated: #{error.message}")
|
|
34
|
+
SolidQueue::TerminatedExecution.record_termination(self)
|
|
35
|
+
raise
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def cleanup_termination_row
|
|
39
|
+
Termination.where(active_job_id: job_id).delete_all
|
|
40
|
+
rescue StandardError => e
|
|
41
|
+
st_logger.error("Failed to delete termination row for job #{job_id}: #{e.class}: #{e.message}")
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
def st_logger
|
|
45
|
+
SolidTerminator.configuration.logger
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidQueue
|
|
4
|
+
class TerminatedExecution < SolidQueue::Execution
|
|
5
|
+
scope :ordered, -> { order(terminated_at: :desc) }
|
|
6
|
+
|
|
7
|
+
def self.record_termination(active_job)
|
|
8
|
+
sq_job = SolidQueue::Job.find_by(active_job_id: active_job.job_id)
|
|
9
|
+
return nil unless sq_job
|
|
10
|
+
|
|
11
|
+
create!(job_id: sq_job.id, queue_name: sq_job.queue_name, priority: sq_job.priority, terminated_at: Time.current)
|
|
12
|
+
rescue StandardError => e
|
|
13
|
+
SolidTerminator.configuration.logger.error(
|
|
14
|
+
"Failed to record termination for job #{active_job.job_id}: #{e.class}: #{e.message}"
|
|
15
|
+
)
|
|
16
|
+
nil
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def retry
|
|
20
|
+
active_job = ActiveJob::Base.deserialize(job.arguments)
|
|
21
|
+
transaction do
|
|
22
|
+
destroy!
|
|
23
|
+
active_job.enqueue
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidTerminator
|
|
4
|
+
class Termination < SolidQueue::Record
|
|
5
|
+
self.table_name = 'solid_queue_terminations'
|
|
6
|
+
|
|
7
|
+
validates :active_job_id, presence: true, uniqueness: true
|
|
8
|
+
|
|
9
|
+
scope :for_jobs, ->(job_ids) { where(active_job_id: job_ids) }
|
|
10
|
+
end
|
|
11
|
+
end
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidTerminator
|
|
4
|
+
module ThreadRegistry
|
|
5
|
+
@mutex = Mutex.new
|
|
6
|
+
@registry = {}
|
|
7
|
+
|
|
8
|
+
class << self
|
|
9
|
+
def register(active_job_id, thread = Thread.current)
|
|
10
|
+
@mutex.synchronize { @registry[active_job_id] = thread }
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
def deregister(active_job_id, thread = nil)
|
|
14
|
+
@mutex.synchronize do
|
|
15
|
+
return if thread && !@registry[active_job_id].equal?(thread)
|
|
16
|
+
|
|
17
|
+
@registry.delete(active_job_id)
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def thread_for(active_job_id)
|
|
22
|
+
@mutex.synchronize { @registry[active_job_id] }
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def active_job_ids
|
|
26
|
+
@mutex.synchronize { @registry.keys.dup }
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
def snapshot
|
|
30
|
+
@mutex.synchronize do
|
|
31
|
+
return nil if @registry.empty?
|
|
32
|
+
|
|
33
|
+
@registry.keys.dup
|
|
34
|
+
end
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
def purge_dead_threads
|
|
38
|
+
@mutex.synchronize { @registry.delete_if { |_, t| !t.alive? } }
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def empty?
|
|
42
|
+
@mutex.synchronize { @registry.empty? }
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'active_record'
|
|
4
|
+
require 'active_job'
|
|
5
|
+
require 'active_support'
|
|
6
|
+
|
|
7
|
+
require 'solid_terminator/version'
|
|
8
|
+
require 'solid_terminator/job_terminated'
|
|
9
|
+
require 'solid_terminator/configuration'
|
|
10
|
+
require 'solid_terminator/thread_registry'
|
|
11
|
+
require 'solid_terminator/terminable'
|
|
12
|
+
require 'solid_terminator/monitor'
|
|
13
|
+
require 'solid_terminator/engine' if defined?(Rails::Engine)
|
|
14
|
+
|
|
15
|
+
module SolidTerminator
|
|
16
|
+
@monitor_mutex = Mutex.new
|
|
17
|
+
|
|
18
|
+
class << self
|
|
19
|
+
def configuration
|
|
20
|
+
@configuration ||= Configuration.new
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def configure
|
|
24
|
+
yield configuration
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
def terminate!(active_job_id)
|
|
28
|
+
configuration.logger.info("Termination requested for job #{active_job_id}")
|
|
29
|
+
Termination.create!(active_job_id: active_job_id)
|
|
30
|
+
rescue ActiveRecord::RecordNotUnique, ActiveRecord::RecordInvalid => e
|
|
31
|
+
raise if e.is_a?(ActiveRecord::RecordInvalid) && !e.record.errors.of_kind?(:active_job_id, :taken)
|
|
32
|
+
|
|
33
|
+
configuration.logger.warn("Termination already requested for job #{active_job_id}")
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def start_monitor
|
|
37
|
+
@monitor_mutex.synchronize do
|
|
38
|
+
return if @monitor
|
|
39
|
+
|
|
40
|
+
@monitor = Monitor.new
|
|
41
|
+
@monitor.start
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def stop_monitor
|
|
46
|
+
@monitor_mutex.synchronize do
|
|
47
|
+
@monitor&.stop
|
|
48
|
+
@monitor = nil
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: solid_terminator
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Meszaros Istvan-Abel
|
|
8
|
+
autorequire:
|
|
9
|
+
bindir: bin
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 2026-06-07 00:00:00.000000000 Z
|
|
12
|
+
dependencies:
|
|
13
|
+
- !ruby/object:Gem::Dependency
|
|
14
|
+
name: rails
|
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
|
16
|
+
requirements:
|
|
17
|
+
- - ">="
|
|
18
|
+
- !ruby/object:Gem::Version
|
|
19
|
+
version: '7.1'
|
|
20
|
+
type: :runtime
|
|
21
|
+
prerelease: false
|
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
23
|
+
requirements:
|
|
24
|
+
- - ">="
|
|
25
|
+
- !ruby/object:Gem::Version
|
|
26
|
+
version: '7.1'
|
|
27
|
+
- !ruby/object:Gem::Dependency
|
|
28
|
+
name: solid_queue
|
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
|
30
|
+
requirements:
|
|
31
|
+
- - ">="
|
|
32
|
+
- !ruby/object:Gem::Version
|
|
33
|
+
version: '1.0'
|
|
34
|
+
type: :runtime
|
|
35
|
+
prerelease: false
|
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
37
|
+
requirements:
|
|
38
|
+
- - ">="
|
|
39
|
+
- !ruby/object:Gem::Version
|
|
40
|
+
version: '1.0'
|
|
41
|
+
description: Adds per-job termination to SolidQueue via a monitor thread and Thread#raise
|
|
42
|
+
email:
|
|
43
|
+
- meszarosistvan97@gmail.com
|
|
44
|
+
executables: []
|
|
45
|
+
extensions: []
|
|
46
|
+
extra_rdoc_files: []
|
|
47
|
+
files:
|
|
48
|
+
- CHANGELOG.md
|
|
49
|
+
- LICENSE.txt
|
|
50
|
+
- README.md
|
|
51
|
+
- lib/generators/solid_terminator/install_generator.rb
|
|
52
|
+
- lib/generators/solid_terminator/templates/create_solid_queue_terminated_executions.rb
|
|
53
|
+
- lib/generators/solid_terminator/templates/create_solid_queue_terminations.rb
|
|
54
|
+
- lib/generators/solid_terminator/templates/initializer.rb
|
|
55
|
+
- lib/solid_terminator.rb
|
|
56
|
+
- lib/solid_terminator/claimed_execution_patch.rb
|
|
57
|
+
- lib/solid_terminator/configuration.rb
|
|
58
|
+
- lib/solid_terminator/engine.rb
|
|
59
|
+
- lib/solid_terminator/job_terminated.rb
|
|
60
|
+
- lib/solid_terminator/monitor.rb
|
|
61
|
+
- lib/solid_terminator/terminable.rb
|
|
62
|
+
- lib/solid_terminator/terminated_execution.rb
|
|
63
|
+
- lib/solid_terminator/termination.rb
|
|
64
|
+
- lib/solid_terminator/thread_registry.rb
|
|
65
|
+
- lib/solid_terminator/version.rb
|
|
66
|
+
homepage: https://github.com/IstvanMs/solid_terminator
|
|
67
|
+
licenses:
|
|
68
|
+
- MIT
|
|
69
|
+
metadata:
|
|
70
|
+
rubygems_mfa_required: 'true'
|
|
71
|
+
source_code_uri: https://github.com/IstvanMs/solid_terminator
|
|
72
|
+
bug_tracker_uri: https://github.com/IstvanMs/solid_terminator/issues
|
|
73
|
+
changelog_uri: https://github.com/IstvanMs/solid_terminator/blob/main/CHANGELOG.md
|
|
74
|
+
post_install_message:
|
|
75
|
+
rdoc_options: []
|
|
76
|
+
require_paths:
|
|
77
|
+
- lib
|
|
78
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
79
|
+
requirements:
|
|
80
|
+
- - ">="
|
|
81
|
+
- !ruby/object:Gem::Version
|
|
82
|
+
version: '3.1'
|
|
83
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
84
|
+
requirements:
|
|
85
|
+
- - ">="
|
|
86
|
+
- !ruby/object:Gem::Version
|
|
87
|
+
version: '0'
|
|
88
|
+
requirements: []
|
|
89
|
+
rubygems_version: 3.3.27
|
|
90
|
+
signing_key:
|
|
91
|
+
specification_version: 4
|
|
92
|
+
summary: Terminate specific in-progress SolidQueue jobs without stopping the worker
|
|
93
|
+
test_files: []
|