solid-jobs 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +47 -7
- data/README.md +16 -4
- data/docs/reliability.md +7 -7
- data/exe/solid-jobs +2 -3
- data/lib/active_job/queue_adapters/solid_jobs_adapter.rb +2 -2
- data/lib/solid_jobs/{config.rb → blueprint.rb} +1 -1
- data/lib/solid_jobs/{api.rb → catalog.rb} +11 -18
- data/lib/solid_jobs/{server.rb → conductor.rb} +8 -8
- data/lib/solid_jobs/{cli.rb → console.rb} +3 -3
- data/lib/solid_jobs/{processor.rb → engine.rb} +3 -3
- data/lib/solid_jobs/errors.rb +1 -1
- data/lib/solid_jobs/executor.rb +2 -2
- data/lib/solid_jobs/{testing.rb → lab.rb} +1 -1
- data/lib/solid_jobs/publisher.rb +5 -5
- data/lib/solid_jobs/{active_job.rb → rails_adapter.rb} +3 -3
- data/lib/solid_jobs/{rails.rb → railtie.rb} +1 -2
- data/lib/solid_jobs/task.rb +12 -12
- data/lib/solid_jobs/utilities.rb +5 -11
- data/lib/solid_jobs/version.rb +1 -1
- data/lib/solid_jobs.rb +10 -10
- metadata +9 -9
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7f95c0daf98dbfa77203eaa6d182f0b0a5ce4c22738a29f7d81b587c9f3abe3a
|
|
4
|
+
data.tar.gz: bf5524aa572a45f439732ec30e5ad12d5ad25b37a98462381d39bd90b5dd7c91
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: cecbfb4efd3abe1598e66b1764ca07da25e4d103c5e59b66c14f84f5f8f3af6b45e91710d535b0b15ee69636a9396629a0e980451202f390a8aebb8a6b41091b
|
|
7
|
+
data.tar.gz: cf40a09b495d1ea62e0789ea8688b530b26bfc7414e77cd2d4cc30f8c874c1a8f6c19b20f2eb2c493c8ca27bbeb2a319a62024dff639d6748d720b1ab13c04d6
|
data/CHANGELOG.md
CHANGED
|
@@ -2,12 +2,51 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.4.0] - 2026-10-03
|
|
6
|
+
|
|
7
|
+
### Changed
|
|
8
|
+
|
|
9
|
+
- Keep the independent SolidJobs runtime, envelope, `solid_jobs:` keyspace,
|
|
10
|
+
interceptors, claims, and renamed component vocabulary from 0.3.0.
|
|
11
|
+
- Remaining shared names with other Ruby job libraries are limited to
|
|
12
|
+
`version.rb`, Rails Active Job adapter contracts (`ActiveJob`,
|
|
13
|
+
`QueueAdapters`, `enqueue_at`), and generic Ruby methods.
|
|
14
|
+
|
|
15
|
+
## [0.3.0] - 2026-10-03
|
|
16
|
+
|
|
17
|
+
### Breaking
|
|
18
|
+
|
|
19
|
+
- Rename the runtime components to SolidJobs-specific names:
|
|
20
|
+
`Config` to `Blueprint`, `Server` to `Conductor`, `Processor` to `Engine`,
|
|
21
|
+
`CLI` to `Console`, `Testing` to `Lab`, and `Metrics` to `Counters`.
|
|
22
|
+
- Rename the task execution callback from `perform` to `execute_task`.
|
|
23
|
+
- Rename `Task.enqueue_at` to `Task.schedule_at`. The Active Job adapter keeps
|
|
24
|
+
its Rails-required `enqueue_at` method.
|
|
25
|
+
- Rename the Active Job execution task from `ActiveJob::Wrapper` to
|
|
26
|
+
`RailsAdapter::AdapterTask`.
|
|
27
|
+
- Rename generic implementation files to match the new component vocabulary.
|
|
28
|
+
`version.rb` remains unchanged as the standard Ruby gem version entry point.
|
|
29
|
+
- Remove the remaining generic internal constants `ClassMethods`,
|
|
30
|
+
`EmptyQueueError`, and `Shutdown` in favor of `TaskMethods`,
|
|
31
|
+
`NoCapturedTask`, and `ExecutionHalt`.
|
|
32
|
+
- Replace `Node#quiet`, `Node#stop`, and `Node#dump_threads` with
|
|
33
|
+
`Node#request_control(:pause)`, `:shutdown`, or `:backtraces`.
|
|
34
|
+
|
|
35
|
+
### Changed
|
|
36
|
+
|
|
37
|
+
- Update the executable, Rails adapter, documentation, benchmarks, and test
|
|
38
|
+
suite to use the new vocabulary consistently.
|
|
39
|
+
- Keep the SolidJobs task API, envelope, `solid_jobs:` Redis keyspace, claims,
|
|
40
|
+
interceptors, and failure model introduced in 0.2.0.
|
|
41
|
+
- Convert configuration keys using Ruby hash transformations, preserving
|
|
42
|
+
nested values and leaving the original input unchanged.
|
|
43
|
+
|
|
5
44
|
## [0.2.0] - 2026-10-03
|
|
6
45
|
|
|
7
46
|
### Breaking
|
|
8
47
|
|
|
9
48
|
- Replace the previous job API with the independent `SolidJobs::Task` API:
|
|
10
|
-
`enqueue`, `enqueue_after`, `
|
|
49
|
+
`enqueue`, `enqueue_after`, `schedule_at`, `enqueue_many`, `execute`, and
|
|
11
50
|
`with_options`.
|
|
12
51
|
- Replace the previous payload with a SolidJobs envelope using `id`, `task`,
|
|
13
52
|
`arguments`, `channel`, `run_at`, `created_ms`, and `queued_ms`.
|
|
@@ -15,7 +54,7 @@ All notable changes to this project will be documented in this file.
|
|
|
15
54
|
not read or migrated automatically.
|
|
16
55
|
- Replace middleware chains with `publish_interceptors` and
|
|
17
56
|
`execute_interceptors`, whose interceptors implement `around(context)`.
|
|
18
|
-
- Replace the administrative API with `
|
|
57
|
+
- Replace the administrative API with `Counters`, `Channel`, `StoredTask`,
|
|
19
58
|
`PlannedTasks`, `RetryingTasks`, `DiscardedTasks`, `Node`, `Nodes`,
|
|
20
59
|
`Execution`, and `Claims`.
|
|
21
60
|
- Replace queue configuration with channels and the `:weighted`, `:priority`,
|
|
@@ -28,7 +67,7 @@ All notable changes to this project will be documented in this file.
|
|
|
28
67
|
and requeue operations.
|
|
29
68
|
- Failure handling uses `max_failures`, `retry_within`, `retry_delay`, and
|
|
30
69
|
`after_final_failure`.
|
|
31
|
-
-
|
|
70
|
+
- Lab modes are now `:capture` and `:execute`.
|
|
32
71
|
|
|
33
72
|
## [0.1.3] - 2026-10-01
|
|
34
73
|
|
|
@@ -41,9 +80,9 @@ All notable changes to this project will be documented in this file.
|
|
|
41
80
|
|
|
42
81
|
### Added
|
|
43
82
|
|
|
44
|
-
- `
|
|
83
|
+
- `Conductor#start` logs a warning on Ruby < 4 when `concurrency > 1`, pointing
|
|
45
84
|
to the Ruby 3.4 Ractor GC-barrier deadlock and the recommended setups.
|
|
46
|
-
- `
|
|
85
|
+
- `Conductor#stop` is bounded: components get `shutdown_timeout + STOP_GRACE`
|
|
47
86
|
to return, `:stop` is re-sent up to `STOP_RESENDS` times, then the
|
|
48
87
|
component is abandoned with an error log instead of hanging shutdown.
|
|
49
88
|
- `test/support/ractor_barrier_repro.rb`: standalone reproducer of the Ruby
|
|
@@ -51,7 +90,7 @@ All notable changes to this project will be documented in this file.
|
|
|
51
90
|
|
|
52
91
|
### Changed
|
|
53
92
|
|
|
54
|
-
-
|
|
93
|
+
- Conductor-based stress tests are skipped on Ruby < 4: Ruby 3.4's Ractor
|
|
55
94
|
scheduler deadlocks the VM on a GC barrier under cross-Ractor `move:`
|
|
56
95
|
traffic. Multi-Ractor servers are recommended on Ruby ≥ 4.0 (see
|
|
57
96
|
`docs/reliability.md`). CI jobs now time out after 20 minutes.
|
|
@@ -74,7 +113,7 @@ All notable changes to this project will be documented in this file.
|
|
|
74
113
|
|
|
75
114
|
### Added
|
|
76
115
|
|
|
77
|
-
- Ractor-oriented client,
|
|
116
|
+
- Ractor-oriented client, Engine, Scheduler, Heartbeat, and Conductor runtime.
|
|
78
117
|
- Sidekiq-compatible Redis queue keys and open-source job payload format.
|
|
79
118
|
- Immediate, scheduled, retry, and dead-job handling.
|
|
80
119
|
- At-least-once reservations with atomic journaling, generation-fenced ACK
|
|
@@ -89,6 +128,7 @@ All notable changes to this project will be documented in this file.
|
|
|
89
128
|
- Reliable-hot-path and CPU-scaling profilers plus the independent Sidekiq
|
|
90
129
|
comparison benchmark.
|
|
91
130
|
|
|
131
|
+
[0.3.0]: https://github.com/nicolasva/solid-jobs/compare/v0.2.0...v0.3.0
|
|
92
132
|
[0.2.0]: https://github.com/nicolasva/solid-jobs/compare/v0.1.3...v0.2.0
|
|
93
133
|
[0.1.3]: https://github.com/nicolasva/solid-jobs/compare/v0.1.2...v0.1.3
|
|
94
134
|
[0.1.2]: https://github.com/nicolasva/solid-jobs/compare/v0.1.1...v0.1.2
|
data/README.md
CHANGED
|
@@ -27,14 +27,14 @@ class RecalculateAccount
|
|
|
27
27
|
|
|
28
28
|
task_options channel: "critical", max_failures: 10
|
|
29
29
|
|
|
30
|
-
def
|
|
30
|
+
def execute_task(account_id)
|
|
31
31
|
Account.find(account_id).recalculate!
|
|
32
32
|
end
|
|
33
33
|
end
|
|
34
34
|
|
|
35
35
|
RecalculateAccount.enqueue(42)
|
|
36
36
|
RecalculateAccount.enqueue_after(30, 42)
|
|
37
|
-
RecalculateAccount.
|
|
37
|
+
RecalculateAccount.schedule_at(Time.now + 300, 42)
|
|
38
38
|
RecalculateAccount.enqueue_many([[42], [43], [44]])
|
|
39
39
|
```
|
|
40
40
|
|
|
@@ -131,7 +131,7 @@ override, SolidJobs uses capped exponential backoff with equal jitter.
|
|
|
131
131
|
## Delivery semantics
|
|
132
132
|
|
|
133
133
|
SolidJobs provides **at-least-once** delivery. An executor atomically claims a
|
|
134
|
-
task before execution and completes the claim only after `
|
|
134
|
+
task before execution and completes the claim only after `execute_task` returns.
|
|
135
135
|
Graceful shutdown requeues unfinished tasks. Claims owned by a crashed node are
|
|
136
136
|
recovered into their original channels.
|
|
137
137
|
|
|
@@ -142,7 +142,19 @@ application-level deduplication when duplicate effects are unsafe.
|
|
|
142
142
|
See [docs/reliability.md](docs/reliability.md) for the state machine, claim
|
|
143
143
|
fencing, recovery rules, and Ruby Ractor caveats.
|
|
144
144
|
|
|
145
|
-
##
|
|
145
|
+
## Node control
|
|
146
|
+
|
|
147
|
+
```ruby
|
|
148
|
+
node = SolidJobs::Nodes.new.first
|
|
149
|
+
node.request_control(:pause)
|
|
150
|
+
node.request_control(:backtraces)
|
|
151
|
+
node.request_control(:shutdown)
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Control requests are asynchronous and use the selected node's namespaced Redis
|
|
155
|
+
mailbox. Unknown actions raise `KeyError` without writing a request.
|
|
156
|
+
|
|
157
|
+
## Lab
|
|
146
158
|
|
|
147
159
|
```ruby
|
|
148
160
|
SolidJobs.testing!(:capture) do
|
data/docs/reliability.md
CHANGED
|
@@ -40,7 +40,7 @@ age alone, so a legitimate long-running task is not stolen.
|
|
|
40
40
|
## Failure boundaries
|
|
41
41
|
|
|
42
42
|
- Before claim: the task remains in `solid_jobs:channel:<name>`.
|
|
43
|
-
- After claim or during `
|
|
43
|
+
- After claim or during `execute_task`: the task remains claimed.
|
|
44
44
|
- After the application effect but before completion: recovery replays it.
|
|
45
45
|
- Redis unavailable during completion: the task remains claimed and is replayed.
|
|
46
46
|
- Graceful shutdown: the active task may finish within the configured timeout;
|
|
@@ -53,7 +53,7 @@ effects require application-level idempotency.
|
|
|
53
53
|
## Startup isolation
|
|
54
54
|
|
|
55
55
|
The server does not claim work while components are booting. Heartbeat,
|
|
56
|
-
|
|
56
|
+
Engine, and Timer Ractors initialize their local configuration and Redis
|
|
57
57
|
pool, report `READY` exactly once, and wait behind
|
|
58
58
|
`SolidJobs::StartupBarrier`:
|
|
59
59
|
|
|
@@ -74,14 +74,14 @@ barrier runs while several Ractors exchange moved messages.
|
|
|
74
74
|
Redis. Ruby 4.0.1 completes the equivalent reproducer.
|
|
75
75
|
|
|
76
76
|
Run multi-Ractor SolidJobs servers on Ruby >= 4.0. On Ruby 3.4, prefer one
|
|
77
|
-
process per
|
|
78
|
-
Ruby versions affected by this runtime issue, and `
|
|
77
|
+
process per Engine (`concurrency: 1`). Conductor stress tests are skipped on
|
|
78
|
+
Ruby versions affected by this runtime issue, and `Conductor#start` warns when it
|
|
79
79
|
detects a multi-Ractor configuration there.
|
|
80
80
|
|
|
81
81
|
## Bounded shutdown
|
|
82
82
|
|
|
83
|
-
`
|
|
84
|
-
`shutdown_timeout +
|
|
83
|
+
`Conductor#stop` never waits forever for a component. Each Ractor gets
|
|
84
|
+
`shutdown_timeout + Conductor::STOP_GRACE` to return after `:stop`; the signal is
|
|
85
85
|
re-sent up to `STOP_RESENDS` times before the component is abandoned with an
|
|
86
86
|
error log.
|
|
87
87
|
|
|
@@ -111,7 +111,7 @@ retains it so recovery increments the same attempt sequence.
|
|
|
111
111
|
|
|
112
112
|
## Backpressure and channels
|
|
113
113
|
|
|
114
|
-
Each
|
|
114
|
+
Each Engine owns at most one claim and claims only immediately before
|
|
115
115
|
execution. A node with concurrency `N` therefore holds at most `N` active
|
|
116
116
|
claims, regardless of channel depth.
|
|
117
117
|
|
data/exe/solid-jobs
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require "solid_jobs"
|
|
4
|
-
require "solid_jobs/
|
|
4
|
+
require "solid_jobs/rails_adapter"
|
|
5
5
|
|
|
6
6
|
module ActiveJob
|
|
7
7
|
module QueueAdapters
|
|
@@ -28,7 +28,7 @@ module ActiveJob
|
|
|
28
28
|
|
|
29
29
|
def push(job, at: nil)
|
|
30
30
|
envelope = {
|
|
31
|
-
"task" => SolidJobs::
|
|
31
|
+
"task" => SolidJobs::RailsAdapter::AdapterTask,
|
|
32
32
|
"wrapped" => job.class.name,
|
|
33
33
|
"channel" => job.queue_name,
|
|
34
34
|
"arguments" => [job.serialize],
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
require "json"
|
|
4
4
|
|
|
5
5
|
module SolidJobs
|
|
6
|
-
class
|
|
6
|
+
class Counters
|
|
7
7
|
attr_reader :config
|
|
8
8
|
|
|
9
9
|
def initialize(config: SolidJobs.config)
|
|
@@ -245,6 +245,12 @@ module SolidJobs
|
|
|
245
245
|
end
|
|
246
246
|
|
|
247
247
|
class Node
|
|
248
|
+
CONTROL_SIGNALS = {
|
|
249
|
+
pause: "TSTP",
|
|
250
|
+
shutdown: "TERM",
|
|
251
|
+
backtraces: "TTIN",
|
|
252
|
+
}.freeze
|
|
253
|
+
|
|
248
254
|
attr_reader :identity
|
|
249
255
|
|
|
250
256
|
def initialize(identity, config: SolidJobs.config)
|
|
@@ -283,16 +289,10 @@ module SolidJobs
|
|
|
283
289
|
Time.at(Float(@attributes["beat"])) if @attributes["beat"]
|
|
284
290
|
end
|
|
285
291
|
|
|
286
|
-
def
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
def stop
|
|
291
|
-
signal("TERM")
|
|
292
|
-
end
|
|
293
|
-
|
|
294
|
-
def dump_threads
|
|
295
|
-
signal("TTIN")
|
|
292
|
+
def request_control(action)
|
|
293
|
+
name = CONTROL_SIGNALS.fetch(action)
|
|
294
|
+
@config.redis_pool.call("LPUSH", Keyspace.node_signals(identity), name)
|
|
295
|
+
true
|
|
296
296
|
end
|
|
297
297
|
|
|
298
298
|
def executions
|
|
@@ -301,13 +301,6 @@ module SolidJobs
|
|
|
301
301
|
Execution.new(identity, executor_id, raw)
|
|
302
302
|
end
|
|
303
303
|
end
|
|
304
|
-
|
|
305
|
-
private
|
|
306
|
-
|
|
307
|
-
def signal(name)
|
|
308
|
-
@config.redis_pool.call("LPUSH", Keyspace.node_signals(identity), name)
|
|
309
|
-
true
|
|
310
|
-
end
|
|
311
304
|
end
|
|
312
305
|
|
|
313
306
|
class Nodes
|
|
@@ -5,7 +5,7 @@ require "socket"
|
|
|
5
5
|
require "timeout"
|
|
6
6
|
|
|
7
7
|
module SolidJobs
|
|
8
|
-
class
|
|
8
|
+
class Conductor
|
|
9
9
|
# Extra time granted beyond shutdown_timeout before a component is
|
|
10
10
|
# considered stuck, and how many times :stop is re-sent before giving up.
|
|
11
11
|
STOP_GRACE = 5.0
|
|
@@ -25,7 +25,7 @@ module SolidJobs
|
|
|
25
25
|
end
|
|
26
26
|
|
|
27
27
|
def start
|
|
28
|
-
raise Error, "
|
|
28
|
+
raise Error, "Conductor is already running" if @started
|
|
29
29
|
|
|
30
30
|
warn_ruby34_multi_ractor
|
|
31
31
|
snapshot = Utilities.shareable_copy(
|
|
@@ -38,7 +38,7 @@ module SolidJobs
|
|
|
38
38
|
local_config = heartbeat = nil
|
|
39
39
|
begin
|
|
40
40
|
started = SolidJobs::StartupBarrier.boot(ready) do
|
|
41
|
-
local_config = SolidJobs::
|
|
41
|
+
local_config = SolidJobs::Blueprint.from_ractor_snapshot(settings)
|
|
42
42
|
SolidJobs.use_config(local_config)
|
|
43
43
|
heartbeat = SolidJobs::Heartbeat.new(
|
|
44
44
|
local_config,
|
|
@@ -119,9 +119,9 @@ module SolidJobs
|
|
|
119
119
|
local_config = processor = nil
|
|
120
120
|
begin
|
|
121
121
|
started = SolidJobs::StartupBarrier.boot(ready_port) do
|
|
122
|
-
local_config = SolidJobs::
|
|
122
|
+
local_config = SolidJobs::Blueprint.from_ractor_snapshot(settings)
|
|
123
123
|
SolidJobs.use_config(local_config)
|
|
124
|
-
processor = SolidJobs::
|
|
124
|
+
processor = SolidJobs::Engine.new(
|
|
125
125
|
local_config,
|
|
126
126
|
identity: settings.fetch(:identity),
|
|
127
127
|
processor_id: id,
|
|
@@ -161,7 +161,7 @@ module SolidJobs
|
|
|
161
161
|
local_config = scheduler = nil
|
|
162
162
|
begin
|
|
163
163
|
started = SolidJobs::StartupBarrier.boot(ready) do
|
|
164
|
-
local_config = SolidJobs::
|
|
164
|
+
local_config = SolidJobs::Blueprint.from_ractor_snapshot(settings)
|
|
165
165
|
SolidJobs.use_config(local_config)
|
|
166
166
|
scheduler = SolidJobs::Timer.new(local_config)
|
|
167
167
|
local_config.redis_pool.call("PING")
|
|
@@ -227,8 +227,8 @@ module SolidJobs
|
|
|
227
227
|
|
|
228
228
|
RUBY34_MULTI_RACTOR_WARNING =
|
|
229
229
|
"SolidJobs: Ruby %s can deadlock the whole VM on a Ractor GC barrier " \
|
|
230
|
-
"under multi-
|
|
231
|
-
"servers on Ruby >= 4.0, or use concurrency: 1 (one process per
|
|
230
|
+
"under multi-Engine load (see docs/reliability.md). Run multi-Ractor " \
|
|
231
|
+
"servers on Ruby >= 4.0, or use concurrency: 1 (one process per Engine)."
|
|
232
232
|
|
|
233
233
|
def warn_ruby34_multi_ractor
|
|
234
234
|
return if RUBY_VERSION >= "4" || config.concurrency <= 1
|
|
@@ -4,7 +4,7 @@ require "optparse"
|
|
|
4
4
|
require "yaml"
|
|
5
5
|
|
|
6
6
|
module SolidJobs
|
|
7
|
-
class
|
|
7
|
+
class Console
|
|
8
8
|
SIGNALS = %w[INT TERM TSTP TTIN INFO].freeze
|
|
9
9
|
|
|
10
10
|
def initialize
|
|
@@ -16,7 +16,7 @@ module SolidJobs
|
|
|
16
16
|
load_configuration
|
|
17
17
|
load_application
|
|
18
18
|
apply_options
|
|
19
|
-
server =
|
|
19
|
+
server = Conductor.new(config: SolidJobs.config).start
|
|
20
20
|
read_io, write_io = IO.pipe
|
|
21
21
|
handlers = install_signal_handlers(write_io)
|
|
22
22
|
wait_for_shutdown(server, read_io)
|
|
@@ -38,7 +38,7 @@ module SolidJobs
|
|
|
38
38
|
options.banner = "Usage: solid-jobs [options]"
|
|
39
39
|
options.on("-r", "--require PATH", "Require an application file") { |value| @options[:require] = value }
|
|
40
40
|
options.on("-C", "--config PATH", "Load YAML configuration") { |value| @options[:config] = value }
|
|
41
|
-
options.on("-c", "--concurrency N", Integer, "
|
|
41
|
+
options.on("-c", "--concurrency N", Integer, "Engine Ractor count") { |value| @options[:concurrency] = value }
|
|
42
42
|
options.on("--channel CHANNEL", "Channel name or name,weight") do |value|
|
|
43
43
|
@options[:channels] << channel(value)
|
|
44
44
|
end
|
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
require "json"
|
|
4
4
|
|
|
5
5
|
module SolidJobs
|
|
6
|
-
class
|
|
6
|
+
class Engine
|
|
7
7
|
def initialize(config, identity: nil, processor_id: nil, heartbeat: nil)
|
|
8
8
|
@config = config
|
|
9
9
|
@claims = Claim.new(config, identity: identity, processor_id: processor_id)
|
|
@@ -46,7 +46,7 @@ module SolidJobs
|
|
|
46
46
|
register_work(claim, envelope)
|
|
47
47
|
begin
|
|
48
48
|
@executor.execute(envelope)
|
|
49
|
-
rescue
|
|
49
|
+
rescue ExecutionHalt
|
|
50
50
|
requeue(claim)
|
|
51
51
|
break
|
|
52
52
|
rescue StandardError => error
|
|
@@ -73,7 +73,7 @@ module SolidJobs
|
|
|
73
73
|
|
|
74
74
|
def interrupt_current
|
|
75
75
|
runner = @state_mutex.synchronize { @runner if @busy }
|
|
76
|
-
runner&.raise(
|
|
76
|
+
runner&.raise(ExecutionHalt, "SolidJobs shutdown timeout exceeded")
|
|
77
77
|
end
|
|
78
78
|
|
|
79
79
|
private
|
data/lib/solid_jobs/errors.rb
CHANGED
data/lib/solid_jobs/executor.rb
CHANGED
|
@@ -9,7 +9,7 @@ module SolidJobs
|
|
|
9
9
|
attr_reader :config
|
|
10
10
|
|
|
11
11
|
def initialize(redis_config:, config: nil)
|
|
12
|
-
@config = config ||
|
|
12
|
+
@config = config || Blueprint.new(redis: redis_config)
|
|
13
13
|
end
|
|
14
14
|
|
|
15
15
|
def execute(envelope)
|
|
@@ -25,7 +25,7 @@ module SolidJobs
|
|
|
25
25
|
|
|
26
26
|
task = Utilities.constantize(task_name).new
|
|
27
27
|
task.task_id = envelope["id"] if task.respond_to?(:task_id=)
|
|
28
|
-
operation = -> { task.
|
|
28
|
+
operation = -> { task.execute_task(*envelope.fetch("arguments")) }
|
|
29
29
|
return operation.call if config.execute_interceptors.empty?
|
|
30
30
|
|
|
31
31
|
config.execute_interceptors.call(Execution.new(task, envelope), &operation)
|
data/lib/solid_jobs/publisher.rb
CHANGED
|
@@ -19,7 +19,7 @@ module SolidJobs
|
|
|
19
19
|
|
|
20
20
|
def initialize(config: nil, configuration: nil)
|
|
21
21
|
selected = config || configuration || SolidJobs.config
|
|
22
|
-
@config = selected.is_a?(SolidRedis::Config) ?
|
|
22
|
+
@config = selected.is_a?(SolidRedis::Config) ? Blueprint.new(redis: selected) : selected
|
|
23
23
|
end
|
|
24
24
|
|
|
25
25
|
def publish(envelope)
|
|
@@ -88,15 +88,15 @@ module SolidJobs
|
|
|
88
88
|
def persist(envelopes)
|
|
89
89
|
return true if envelopes.empty?
|
|
90
90
|
|
|
91
|
-
if defined?(
|
|
92
|
-
case
|
|
91
|
+
if defined?(Lab)
|
|
92
|
+
case Lab.mode
|
|
93
93
|
when :capture
|
|
94
94
|
envelopes.each do |envelope|
|
|
95
|
-
|
|
95
|
+
Lab.captured_for(Utilities.constantize(envelope.fetch("task"))) << envelope
|
|
96
96
|
end
|
|
97
97
|
return true
|
|
98
98
|
when :execute
|
|
99
|
-
envelopes.each { |envelope|
|
|
99
|
+
envelopes.each { |envelope| Lab.execute(envelope) }
|
|
100
100
|
return true
|
|
101
101
|
end
|
|
102
102
|
end
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module SolidJobs
|
|
4
|
-
module
|
|
5
|
-
class
|
|
4
|
+
module RailsAdapter
|
|
5
|
+
class AdapterTask
|
|
6
6
|
include SolidJobs::Task
|
|
7
7
|
|
|
8
|
-
def
|
|
8
|
+
def execute_task(job_data)
|
|
9
9
|
::ActiveJob::Base.execute(job_data.merge("provider_job_id" => task_id))
|
|
10
10
|
end
|
|
11
11
|
end
|
data/lib/solid_jobs/task.rb
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
module SolidJobs
|
|
4
4
|
module Task
|
|
5
5
|
def self.included(base)
|
|
6
|
-
base.extend(
|
|
6
|
+
base.extend(TaskMethods)
|
|
7
7
|
base.include(InstanceMethods)
|
|
8
8
|
base.task_options
|
|
9
9
|
end
|
|
@@ -25,7 +25,7 @@ module SolidJobs
|
|
|
25
25
|
end
|
|
26
26
|
end
|
|
27
27
|
|
|
28
|
-
module
|
|
28
|
+
module TaskMethods
|
|
29
29
|
def task_options(options = nil)
|
|
30
30
|
if options
|
|
31
31
|
merged = current_task_options.merge(Utilities.stringify_keys(options))
|
|
@@ -64,16 +64,16 @@ module SolidJobs
|
|
|
64
64
|
end
|
|
65
65
|
|
|
66
66
|
def enqueue_after(delay, *arguments)
|
|
67
|
-
|
|
67
|
+
schedule_at(Time.now.to_f + Float(delay), *arguments)
|
|
68
68
|
end
|
|
69
69
|
|
|
70
|
-
def
|
|
70
|
+
def schedule_at(time, *arguments)
|
|
71
71
|
timestamp = time.is_a?(Time) ? time.to_f : Float(time)
|
|
72
72
|
publish("task" => self, "arguments" => arguments, "run_at" => timestamp)
|
|
73
73
|
end
|
|
74
74
|
|
|
75
75
|
def execute(*arguments)
|
|
76
|
-
|
|
76
|
+
Lab.execute(current_task_options.merge("task" => self, "arguments" => arguments))
|
|
77
77
|
end
|
|
78
78
|
|
|
79
79
|
def with_options(options)
|
|
@@ -81,7 +81,7 @@ module SolidJobs
|
|
|
81
81
|
end
|
|
82
82
|
|
|
83
83
|
def captured
|
|
84
|
-
|
|
84
|
+
Lab.captured_for(self)
|
|
85
85
|
end
|
|
86
86
|
|
|
87
87
|
def clear_captured
|
|
@@ -90,9 +90,9 @@ module SolidJobs
|
|
|
90
90
|
|
|
91
91
|
def execute_next
|
|
92
92
|
envelope = captured.shift
|
|
93
|
-
raise
|
|
93
|
+
raise NoCapturedTask, "No captured task for #{name}" unless envelope
|
|
94
94
|
|
|
95
|
-
|
|
95
|
+
Lab.execute(envelope)
|
|
96
96
|
end
|
|
97
97
|
|
|
98
98
|
private
|
|
@@ -114,17 +114,17 @@ module SolidJobs
|
|
|
114
114
|
if (delay = @options.delete("delay"))
|
|
115
115
|
enqueue_after(delay, *arguments)
|
|
116
116
|
elsif (time = @options.delete("run_at"))
|
|
117
|
-
|
|
117
|
+
schedule_at(time, *arguments)
|
|
118
118
|
else
|
|
119
119
|
publish(arguments)
|
|
120
120
|
end
|
|
121
121
|
end
|
|
122
122
|
|
|
123
123
|
def enqueue_after(delay, *arguments)
|
|
124
|
-
|
|
124
|
+
schedule_at(Time.now.to_f + Float(delay), *arguments)
|
|
125
125
|
end
|
|
126
126
|
|
|
127
|
-
def
|
|
127
|
+
def schedule_at(time, *arguments)
|
|
128
128
|
timestamp = time.is_a?(Time) ? time.to_f : Float(time)
|
|
129
129
|
publish(arguments, "run_at" => timestamp)
|
|
130
130
|
end
|
|
@@ -143,5 +143,5 @@ module SolidJobs
|
|
|
143
143
|
end
|
|
144
144
|
end
|
|
145
145
|
|
|
146
|
-
class
|
|
146
|
+
class NoCapturedTask < Error; end
|
|
147
147
|
end
|
data/lib/solid_jobs/utilities.rb
CHANGED
|
@@ -5,20 +5,14 @@ module SolidJobs
|
|
|
5
5
|
module_function
|
|
6
6
|
|
|
7
7
|
def stringify_keys(value)
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
end
|
|
13
|
-
when Array
|
|
14
|
-
value.map { |nested| stringify_keys(nested) }
|
|
15
|
-
else
|
|
16
|
-
value
|
|
17
|
-
end
|
|
8
|
+
return stringify_hash_keys(value).transform_values { |nested| stringify_keys(nested) } if value.is_a?(Hash)
|
|
9
|
+
return value.map { |nested| stringify_keys(nested) } if value.is_a?(Array)
|
|
10
|
+
|
|
11
|
+
value
|
|
18
12
|
end
|
|
19
13
|
|
|
20
14
|
def stringify_hash_keys(hash)
|
|
21
|
-
hash.
|
|
15
|
+
hash.transform_keys(&:to_s)
|
|
22
16
|
end
|
|
23
17
|
|
|
24
18
|
def shareable_copy(value)
|
data/lib/solid_jobs/version.rb
CHANGED
data/lib/solid_jobs.rb
CHANGED
|
@@ -11,22 +11,22 @@ require_relative "solid_jobs/keyspace"
|
|
|
11
11
|
require_relative "solid_jobs/ractor_support"
|
|
12
12
|
require_relative "solid_jobs/startup_barrier"
|
|
13
13
|
require_relative "solid_jobs/interceptor_registry"
|
|
14
|
-
require_relative "solid_jobs/
|
|
14
|
+
require_relative "solid_jobs/blueprint"
|
|
15
15
|
require_relative "solid_jobs/publisher"
|
|
16
|
-
require_relative "solid_jobs/
|
|
16
|
+
require_relative "solid_jobs/lab"
|
|
17
17
|
require_relative "solid_jobs/task"
|
|
18
18
|
require_relative "solid_jobs/executor"
|
|
19
19
|
require_relative "solid_jobs/claim"
|
|
20
20
|
require_relative "solid_jobs/failure_policy"
|
|
21
21
|
require_relative "solid_jobs/timer"
|
|
22
|
-
require_relative "solid_jobs/
|
|
22
|
+
require_relative "solid_jobs/engine"
|
|
23
23
|
require_relative "solid_jobs/heartbeat"
|
|
24
24
|
require_relative "solid_jobs/recovery"
|
|
25
25
|
require_relative "solid_jobs/integrity_check"
|
|
26
|
-
require_relative "solid_jobs/
|
|
27
|
-
require_relative "solid_jobs/
|
|
28
|
-
require_relative "solid_jobs/
|
|
29
|
-
require_relative "solid_jobs/
|
|
26
|
+
require_relative "solid_jobs/conductor"
|
|
27
|
+
require_relative "solid_jobs/catalog"
|
|
28
|
+
require_relative "solid_jobs/rails_adapter" if defined?(::ActiveJob::Base)
|
|
29
|
+
require_relative "solid_jobs/railtie" if defined?(::Rails::Railtie)
|
|
30
30
|
|
|
31
31
|
module SolidJobs
|
|
32
32
|
CONFIG_KEY = :solid_jobs_config
|
|
@@ -36,7 +36,7 @@ module SolidJobs
|
|
|
36
36
|
# Configuration is Ractor-local: every thread and fiber inside a Ractor
|
|
37
37
|
# shares it, while each Ractor keeps its own isolated configuration.
|
|
38
38
|
def config
|
|
39
|
-
Ractor.current[CONFIG_KEY] ||=
|
|
39
|
+
Ractor.current[CONFIG_KEY] ||= Blueprint.new
|
|
40
40
|
end
|
|
41
41
|
|
|
42
42
|
def configure
|
|
@@ -51,11 +51,11 @@ module SolidJobs
|
|
|
51
51
|
|
|
52
52
|
def reset!
|
|
53
53
|
Ractor.current[CONFIG_KEY]&.close
|
|
54
|
-
Ractor.current[CONFIG_KEY] =
|
|
54
|
+
Ractor.current[CONFIG_KEY] = Blueprint.new
|
|
55
55
|
end
|
|
56
56
|
|
|
57
57
|
def testing!(mode, &block)
|
|
58
|
-
|
|
58
|
+
Lab.testing!(mode, &block)
|
|
59
59
|
end
|
|
60
60
|
|
|
61
61
|
def enqueue(task, arguments, channel: "default", **options)
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: solid-jobs
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Nicolas Vandenbogaerde
|
|
@@ -119,11 +119,12 @@ files:
|
|
|
119
119
|
- exe/solid-jobs
|
|
120
120
|
- lib/active_job/queue_adapters/solid_jobs_adapter.rb
|
|
121
121
|
- lib/solid_jobs.rb
|
|
122
|
-
- lib/solid_jobs/
|
|
123
|
-
- lib/solid_jobs/
|
|
122
|
+
- lib/solid_jobs/blueprint.rb
|
|
123
|
+
- lib/solid_jobs/catalog.rb
|
|
124
124
|
- lib/solid_jobs/claim.rb
|
|
125
|
-
- lib/solid_jobs/
|
|
126
|
-
- lib/solid_jobs/
|
|
125
|
+
- lib/solid_jobs/conductor.rb
|
|
126
|
+
- lib/solid_jobs/console.rb
|
|
127
|
+
- lib/solid_jobs/engine.rb
|
|
127
128
|
- lib/solid_jobs/errors.rb
|
|
128
129
|
- lib/solid_jobs/executor.rb
|
|
129
130
|
- lib/solid_jobs/failure_policy.rb
|
|
@@ -131,15 +132,14 @@ files:
|
|
|
131
132
|
- lib/solid_jobs/integrity_check.rb
|
|
132
133
|
- lib/solid_jobs/interceptor_registry.rb
|
|
133
134
|
- lib/solid_jobs/keyspace.rb
|
|
134
|
-
- lib/solid_jobs/
|
|
135
|
+
- lib/solid_jobs/lab.rb
|
|
135
136
|
- lib/solid_jobs/publisher.rb
|
|
136
137
|
- lib/solid_jobs/ractor_support.rb
|
|
137
|
-
- lib/solid_jobs/
|
|
138
|
+
- lib/solid_jobs/rails_adapter.rb
|
|
139
|
+
- lib/solid_jobs/railtie.rb
|
|
138
140
|
- lib/solid_jobs/recovery.rb
|
|
139
|
-
- lib/solid_jobs/server.rb
|
|
140
141
|
- lib/solid_jobs/startup_barrier.rb
|
|
141
142
|
- lib/solid_jobs/task.rb
|
|
142
|
-
- lib/solid_jobs/testing.rb
|
|
143
143
|
- lib/solid_jobs/timer.rb
|
|
144
144
|
- lib/solid_jobs/utilities.rb
|
|
145
145
|
- lib/solid_jobs/version.rb
|