solid_queue 1.4.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +25 -4
- data/UPGRADING.md +11 -0
- data/app/models/solid_queue/claimed_execution.rb +23 -14
- data/app/models/solid_queue/job/executable.rb +6 -5
- data/app/models/solid_queue/job/schedulable.rb +2 -1
- data/app/models/solid_queue/queue.rb +1 -3
- data/app/models/solid_queue/queue_selector.rb +2 -2
- data/app/models/solid_queue/record/distinct_values.rb +48 -0
- data/app/models/solid_queue/record.rb +3 -0
- data/app/models/solid_queue/recurring_task.rb +30 -4
- data/app/models/solid_queue/semaphore.rb +7 -1
- data/lib/active_job/queue_adapters/solid_queue_adapter.rb +9 -1
- data/lib/generators/solid_queue/install/templates/config/queue.yml +1 -1
- data/lib/solid_queue/cli.rb +6 -0
- data/lib/solid_queue/configuration.rb +44 -22
- data/lib/solid_queue/engine.rb +6 -0
- data/lib/solid_queue/processes/runnable.rb +4 -2
- data/lib/solid_queue/scheduler/recurring_schedule.rb +7 -5
- data/lib/solid_queue/supervisor/signals.rb +1 -1
- data/lib/solid_queue/supervisor.rb +2 -0
- data/lib/solid_queue/tasks.rb +6 -0
- data/lib/solid_queue/version.rb +1 -1
- data/lib/solid_queue.rb +9 -0
- metadata +4 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: d40f055103084c9aaa9efe6ded0e675ff967a88200bf9f5579b1287e29d48818
|
|
4
|
+
data.tar.gz: af6ab85d5c674dbceff32768b5aa7b4d87bc1aeea24476e3fb469d639c4fad46
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 926a7f82ebca8389d5d6d6116bd6539d65715704e739f66083f1e21411737da72972f4b643bd4291e95ef5bb107417cb11fde6c535d6e4013434ac589cc2fc6e
|
|
7
|
+
data.tar.gz: 684e5f32bbe7aa4ce0645687a92c2538016431151208156ec144b574e1d48e8d04144f28a2eacca86ff5bd4449a2c6bdd439bdcd74bdf72788ae4430abd284c2
|
data/README.md
CHANGED
|
@@ -23,6 +23,7 @@ Solid Queue can be used with SQL databases such as MySQL, PostgreSQL, or SQLite,
|
|
|
23
23
|
- [Threads, processes, and signals](#threads-processes-and-signals)
|
|
24
24
|
- [Database configuration](#database-configuration)
|
|
25
25
|
- [Other configuration settings](#other-configuration-settings)
|
|
26
|
+
- [Validating the configuration](#validating-the-configuration)
|
|
26
27
|
- [Lifecycle hooks](#lifecycle-hooks)
|
|
27
28
|
- [Errors when enqueuing](#errors-when-enqueuing)
|
|
28
29
|
- [Concurrency controls](#concurrency-controls)
|
|
@@ -44,7 +45,7 @@ Solid Queue is configured by default in new Rails 8 applications. If you're runn
|
|
|
44
45
|
1. `bundle add solid_queue`
|
|
45
46
|
2. `bin/rails solid_queue:install`
|
|
46
47
|
|
|
47
|
-
(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.
|
|
48
|
+
(Note: The minimum supported version of Rails is 7.1 and Ruby is 3.2.)
|
|
48
49
|
|
|
49
50
|
This will configure Solid Queue as the production Active Job backend, create the configuration files `config/queue.yml` and `config/recurring.yml`, and create the `db/queue_schema.rb`. It'll also create a `bin/jobs` executable wrapper that you can use to start Solid Queue.
|
|
50
51
|
|
|
@@ -338,7 +339,7 @@ FROM solid_queue_ready_executions
|
|
|
338
339
|
WHERE queue_name LIKE 'beta%';
|
|
339
340
|
```
|
|
340
341
|
|
|
341
|
-
This type of `DISTINCT` query on a column that's the leftmost column in an index can be performed very fast in MySQL thanks to a technique called [Loose Index Scan](https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html#loose-index-scan). PostgreSQL
|
|
342
|
+
This type of `DISTINCT` query on a column that's the leftmost column in an index can be performed very fast in MySQL thanks to a technique called [Loose Index Scan](https://dev.mysql.com/doc/refman/8.0/en/group-by-optimization.html#loose-index-scan). PostgreSQL doesn't implement this technique natively, so Solid Queue uses a [recursive CTE](https://www.postgresql.org/docs/current/queries-with.html#QUERIES-WITH-RECURSIVE) to emulate it, achieving similar performance by walking the B-tree index and jumping between distinct values. SQLite doesn't implement loose index scan either, but this is unlikely to be a problem since SQLite is typically used in development with small datasets.
|
|
342
343
|
|
|
343
344
|
Similarly to using prefixes, the same will happen if you have paused queues, because we need to get a list of all queues with a query like
|
|
344
345
|
```sql
|
|
@@ -372,6 +373,8 @@ The supervisor is in charge of managing these processes, and it responds to the
|
|
|
372
373
|
|
|
373
374
|
When receiving a `QUIT` signal, if workers still have jobs in-flight, these will be returned to the queue when the processes are deregistered.
|
|
374
375
|
|
|
376
|
+
On Windows, the `QUIT` signal can't be trapped, so the supervisor only responds to `TERM` and `INT` there.
|
|
377
|
+
|
|
375
378
|
If processes have no chance of cleaning up before exiting (e.g. if someone pulls a cable somewhere), in-flight jobs might remain claimed by the processes executing them. Processes send heartbeats, and the supervisor checks and prunes processes with expired heartbeats. Jobs that were claimed by processes with an expired heartbeat will be marked as failed with a `SolidQueue::Processes::ProcessPrunedError`. You can configure both the frequency of heartbeats and the threshold to consider a process dead. See the section below for this.
|
|
376
379
|
|
|
377
380
|
In a similar way, if a worker is terminated in any other way not initiated by the above signals (e.g. a worker is sent a `KILL` signal), jobs in progress will be marked as failed so that they can be inspected, with a `SolidQueue::Processes::ProcessExitError`. Sometimes a job in particular is responsible for this, for example, if it has a memory leak and you have a mechanism to kill processes over a certain memory threshold, so this will help identifying this kind of situation.
|
|
@@ -408,6 +411,22 @@ There are several settings that control how Solid Queue works that you can set a
|
|
|
408
411
|
- `clear_finished_jobs_after`: period to keep finished jobs around, in case `preserve_finished_jobs` is true — defaults to 1 day. When installing Solid Queue, [a recurring job](#recurring-tasks) is automatically configured to clear finished jobs every hour on the 12th minute in batches. You can edit the `recurring.yml` configuration to change this as you see fit.
|
|
409
412
|
- `default_concurrency_control_period`: the value to be used as the default for the `duration` parameter in [concurrency controls](#concurrency-controls). It defaults to 3 minutes.
|
|
410
413
|
|
|
414
|
+
### Validating the configuration
|
|
415
|
+
|
|
416
|
+
You can validate the Solid Queue configuration ahead of time, without starting any process. This is handy in deploy scripts or CI to catch mistakes—a typo in `recurring.yml`, no processes configured, and so on—before they cause a supervisor to boot into a broken state:
|
|
417
|
+
|
|
418
|
+
```bash
|
|
419
|
+
# Using the bin/jobs binstub
|
|
420
|
+
bin/jobs check
|
|
421
|
+
|
|
422
|
+
# Or via rake
|
|
423
|
+
bin/rails solid_queue:check
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
Both commands validate the configuration for the current Rails environment. On success they print `Solid Queue configuration is valid.` and exit `0`; otherwise they print the errors and exit non-zero. When the number of threads is larger than the [database connection pool](#database-configuration), they also print an advisory warning about it—the same one the supervisor logs on boot. They're tolerant of a missing database connection, so they can run on CI or deploy hosts without database credentials.
|
|
427
|
+
|
|
428
|
+
`bin/jobs check` accepts the same options as `bin/jobs start` (e.g. `--config_file`, `--recurring_schedule_file`, `--skip-recurring`). The rake task honors the same environment variables Solid Queue already uses: `SOLID_QUEUE_CONFIG`, `SOLID_QUEUE_RECURRING_SCHEDULE`, and `SOLID_QUEUE_SKIP_RECURRING`. To validate a specific environment's configuration, set `RAILS_ENV`, for example `RAILS_ENV=production bin/jobs check`.
|
|
429
|
+
|
|
411
430
|
|
|
412
431
|
## Lifecycle hooks
|
|
413
432
|
|
|
@@ -471,7 +490,7 @@ Solid Queue extends Active Job with concurrency controls, that allows you to lim
|
|
|
471
490
|
|
|
472
491
|
```ruby
|
|
473
492
|
class MyJob < ApplicationJob
|
|
474
|
-
limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2,
|
|
493
|
+
limits_concurrency to: max_concurrent_executions, key: ->(arg1, arg2, *) { ... }, duration: max_interval_to_guarantee_concurrency_limit, group: concurrency_group, on_conflict: on_conflict_behaviour
|
|
475
494
|
|
|
476
495
|
# ...
|
|
477
496
|
```
|
|
@@ -705,7 +724,9 @@ production:
|
|
|
705
724
|
|
|
706
725
|
Tasks are specified as a hash/dictionary, where the key will be the task's key internally. Each task needs to either have a `class`, which will be the job class to enqueue, or a `command`, which will be eval'ed in the context of a job (`SolidQueue::RecurringJob`) that will be enqueued according to its schedule, in the `solid_queue_recurring` queue.
|
|
707
726
|
|
|
708
|
-
Each task needs to have also a schedule, which is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). You can
|
|
727
|
+
Each task needs to have also a schedule, which is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). Schedules can include a time zone (e.g. `0 9 * * * America/New_York` or `every day at 9am America/New_York`). When a schedule doesn't specify one, it's interpreted in the application's configured time zone (`config.time_zone`) by default. You can change or disable this default with `config.solid_queue.time_zone`; setting it to `nil` falls back to the system's local time.
|
|
728
|
+
|
|
729
|
+
You can optionally supply the following for each task:
|
|
709
730
|
- `args`: the arguments to be passed to the job, as a single argument, a hash, or an array of arguments that can also include kwargs as the last element in the array.
|
|
710
731
|
|
|
711
732
|
The job in the example configuration above will be enqueued every second as:
|
data/UPGRADING.md
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
|
+
# Upgrading to version 1.5.x
|
|
2
|
+
Ruby 3.1 is no longer supported, as it reached end-of-life in March 2025. Solid Queue now requires Ruby 3.2 or newer. If you're still on Ruby 3.1, Bundler will continue to resolve solid_queue 1.4.x for you, but you won't receive any new versions until you upgrade Ruby.
|
|
3
|
+
|
|
4
|
+
Recurring schedules that don't specify a time zone are now interpreted in your application's configured time zone (`config.time_zone`) by default, instead of the system's local time. This only affects schedules without an explicit time zone (e.g. `every day at 9am`); schedules that already include one (e.g. `0 9 * * * America/New_York`) are unaffected.
|
|
5
|
+
|
|
6
|
+
If your `config.time_zone` differs from the system time where your processes run, recurring jobs may fire at a different wall-clock time than before. To keep the previous behavior, set:
|
|
7
|
+
|
|
8
|
+
```ruby
|
|
9
|
+
config.solid_queue.time_zone = nil
|
|
10
|
+
```
|
|
11
|
+
|
|
1
12
|
# Upgrading to version 1.x
|
|
2
13
|
The value returned for `enqueue_after_transaction_commit?` has changed to `true`, and it's no longer configurable. If you want to change this, you need to use Active Job's configuration options.
|
|
3
14
|
|
|
@@ -43,7 +43,6 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
|
|
|
43
43
|
SolidQueue.instrument(:fail_many_claimed) do |payload|
|
|
44
44
|
executions.each do |execution|
|
|
45
45
|
execution.failed_with(error)
|
|
46
|
-
execution.unblock_next_job
|
|
47
46
|
end
|
|
48
47
|
|
|
49
48
|
payload[:process_ids] = executions.map(&:process_id).uniq
|
|
@@ -71,13 +70,11 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
|
|
|
71
70
|
failed_with(result.error)
|
|
72
71
|
raise result.error
|
|
73
72
|
end
|
|
74
|
-
ensure
|
|
75
|
-
unblock_next_job
|
|
76
73
|
end
|
|
77
74
|
|
|
78
75
|
def release
|
|
79
76
|
SolidQueue.instrument(:release_claimed, job_id: job.id, process_id: process_id) do
|
|
80
|
-
|
|
77
|
+
unless_already_finalized do
|
|
81
78
|
job.dispatch_bypassing_concurrency_limits
|
|
82
79
|
destroy!
|
|
83
80
|
end
|
|
@@ -89,14 +86,7 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
|
|
|
89
86
|
end
|
|
90
87
|
|
|
91
88
|
def failed_with(error)
|
|
92
|
-
|
|
93
|
-
job.failed_with(error)
|
|
94
|
-
destroy!
|
|
95
|
-
end
|
|
96
|
-
end
|
|
97
|
-
|
|
98
|
-
def unblock_next_job
|
|
99
|
-
job.unblock_next_blocked_job
|
|
89
|
+
finalize { job.failed_with(error) }
|
|
100
90
|
end
|
|
101
91
|
|
|
102
92
|
private
|
|
@@ -108,9 +98,28 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
|
|
|
108
98
|
end
|
|
109
99
|
|
|
110
100
|
def finished
|
|
111
|
-
|
|
112
|
-
|
|
101
|
+
finalize { job.finished! }
|
|
102
|
+
end
|
|
103
|
+
|
|
104
|
+
def finalize
|
|
105
|
+
finalized = unless_already_finalized do
|
|
106
|
+
yield
|
|
113
107
|
destroy!
|
|
108
|
+
true
|
|
109
|
+
end
|
|
110
|
+
|
|
111
|
+
# Unblock the next job outside the finalize transaction so a failure while
|
|
112
|
+
# releasing the concurrency lock or dispatching the next job can't roll back
|
|
113
|
+
# a job that already finished or failed. Only the actor that owned and
|
|
114
|
+
# finalized the claim gets here, so the lock is released exactly once.
|
|
115
|
+
job.unblock_next_blocked_job if finalized
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
def unless_already_finalized
|
|
119
|
+
transaction do
|
|
120
|
+
return false unless self.class.unscoped.lock.find_by(id: id)
|
|
121
|
+
|
|
122
|
+
yield
|
|
114
123
|
end
|
|
115
124
|
end
|
|
116
125
|
end
|
|
@@ -41,15 +41,16 @@ module SolidQueue
|
|
|
41
41
|
end
|
|
42
42
|
|
|
43
43
|
def successfully_dispatched(jobs)
|
|
44
|
-
|
|
44
|
+
jobs_by_id = jobs.index_by(&:id)
|
|
45
|
+
dispatched_and_ready(jobs_by_id) + dispatched_and_blocked(jobs_by_id)
|
|
45
46
|
end
|
|
46
47
|
|
|
47
|
-
def dispatched_and_ready(
|
|
48
|
-
|
|
48
|
+
def dispatched_and_ready(jobs_by_id)
|
|
49
|
+
ReadyExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] }
|
|
49
50
|
end
|
|
50
51
|
|
|
51
|
-
def dispatched_and_blocked(
|
|
52
|
-
|
|
52
|
+
def dispatched_and_blocked(jobs_by_id)
|
|
53
|
+
BlockedExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] }
|
|
53
54
|
end
|
|
54
55
|
end
|
|
55
56
|
|
|
@@ -23,7 +23,8 @@ module SolidQueue
|
|
|
23
23
|
end
|
|
24
24
|
|
|
25
25
|
def successfully_scheduled(jobs)
|
|
26
|
-
|
|
26
|
+
jobs_by_id = jobs.index_by(&:id)
|
|
27
|
+
ScheduledExecution.where(job_id: jobs_by_id.keys).pluck(:job_id).map { |id| jobs_by_id[id] }
|
|
27
28
|
end
|
|
28
29
|
end
|
|
29
30
|
|
|
@@ -43,7 +43,7 @@ module SolidQueue
|
|
|
43
43
|
end
|
|
44
44
|
|
|
45
45
|
def all_queues
|
|
46
|
-
relation.
|
|
46
|
+
relation.distinct_values_of(:queue_name)
|
|
47
47
|
end
|
|
48
48
|
|
|
49
49
|
def exact_names
|
|
@@ -53,7 +53,7 @@ module SolidQueue
|
|
|
53
53
|
def prefixed_names
|
|
54
54
|
if prefixes.empty? then []
|
|
55
55
|
else
|
|
56
|
-
relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).
|
|
56
|
+
relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct_values_of(:queue_name)
|
|
57
57
|
end
|
|
58
58
|
end
|
|
59
59
|
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SolidQueue
|
|
4
|
+
class Record
|
|
5
|
+
module DistinctValues
|
|
6
|
+
extend ActiveSupport::Concern
|
|
7
|
+
|
|
8
|
+
# PostgreSQL has no native loose index scan, so a plain DISTINCT on a leading
|
|
9
|
+
# index column degrades to a full index scan on large tables. We emulate one
|
|
10
|
+
# with a recursive CTE that walks the index jumping between distinct values.
|
|
11
|
+
class_methods do
|
|
12
|
+
def distinct_values_of(column)
|
|
13
|
+
if loose_index_scan_emulation_needed?
|
|
14
|
+
loose_distinct_via_recursive_cte(column)
|
|
15
|
+
else
|
|
16
|
+
distinct.pluck(column)
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
private
|
|
21
|
+
def loose_index_scan_emulation_needed?
|
|
22
|
+
connection.adapter_name == "PostgreSQL"
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Emulates a loose index scan, honoring the current scope (e.g. LIKE prefixes)
|
|
26
|
+
# by building the anchor and the recursive step as scoped relations, whose
|
|
27
|
+
# #to_sql inlines any bind parameters so they can be embedded in the raw CTE.
|
|
28
|
+
def loose_distinct_via_recursive_cte(column)
|
|
29
|
+
col = connection.quote_column_name(column)
|
|
30
|
+
|
|
31
|
+
connection.select_values(<<~SQL.squish)
|
|
32
|
+
WITH RECURSIVE t AS (
|
|
33
|
+
(#{next_distinct_value(col, "#{col} IS NOT NULL")})
|
|
34
|
+
UNION ALL
|
|
35
|
+
SELECT (#{next_distinct_value(col, "#{col} > t.#{col}")}) FROM t WHERE t.#{col} IS NOT NULL
|
|
36
|
+
)
|
|
37
|
+
SELECT #{col} FROM t WHERE #{col} IS NOT NULL
|
|
38
|
+
SQL
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
# Smallest value of `col` within the current scope that matches `condition`.
|
|
42
|
+
def next_distinct_value(col, condition)
|
|
43
|
+
all.where(Arel.sql(condition)).reorder(Arel.sql(col)).limit(1).select(Arel.sql(col)).to_sql
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
@@ -56,16 +56,17 @@ module SolidQueue
|
|
|
56
56
|
end
|
|
57
57
|
end
|
|
58
58
|
|
|
59
|
-
|
|
60
|
-
|
|
59
|
+
|
|
60
|
+
def next_time_after(time)
|
|
61
|
+
parsed_schedule_with_time_zone.next_time(time).utc
|
|
61
62
|
end
|
|
62
63
|
|
|
63
64
|
def next_time
|
|
64
|
-
|
|
65
|
+
parsed_schedule_with_time_zone.next_time.utc
|
|
65
66
|
end
|
|
66
67
|
|
|
67
68
|
def previous_time
|
|
68
|
-
|
|
69
|
+
parsed_schedule_with_time_zone.previous_time.utc
|
|
69
70
|
end
|
|
70
71
|
|
|
71
72
|
def last_enqueued_time
|
|
@@ -85,6 +86,7 @@ module SolidQueue
|
|
|
85
86
|
|
|
86
87
|
perform_later.tap do |job|
|
|
87
88
|
unless job.successfully_enqueued?
|
|
89
|
+
report_enqueue_error(job.enqueue_error, at: at)
|
|
88
90
|
payload[:enqueue_error] = job.enqueue_error&.message
|
|
89
91
|
end
|
|
90
92
|
end
|
|
@@ -97,6 +99,7 @@ module SolidQueue
|
|
|
97
99
|
payload[:skipped] = true
|
|
98
100
|
false
|
|
99
101
|
rescue Job::EnqueueError => error
|
|
102
|
+
report_enqueue_error(error, at: at)
|
|
100
103
|
payload[:enqueue_error] = error.message
|
|
101
104
|
false
|
|
102
105
|
end
|
|
@@ -168,11 +171,24 @@ module SolidQueue
|
|
|
168
171
|
end
|
|
169
172
|
end
|
|
170
173
|
|
|
174
|
+
def parsed_schedule_with_time_zone
|
|
175
|
+
@parsed_schedule_with_time_zone ||= apply_default_time_zone_to(parsed_schedule)
|
|
176
|
+
end
|
|
171
177
|
|
|
172
178
|
def parsed_schedule
|
|
173
179
|
@parsed_schedule ||= Fugit.parse(schedule, multi: :fail)
|
|
174
180
|
end
|
|
175
181
|
|
|
182
|
+
def apply_default_time_zone_to(schedule)
|
|
183
|
+
if schedule.respond_to?(:zone) && schedule.zone.nil? && default_time_zone.present?
|
|
184
|
+
Fugit.parse("#{schedule.to_cron_s} #{default_time_zone}", multi: :fail)
|
|
185
|
+
else
|
|
186
|
+
schedule
|
|
187
|
+
end
|
|
188
|
+
rescue ArgumentError
|
|
189
|
+
schedule
|
|
190
|
+
end
|
|
191
|
+
|
|
176
192
|
def job_class
|
|
177
193
|
@job_class ||= class_name.present? ? class_name.safe_constantize : self.class.default_job_class
|
|
178
194
|
end
|
|
@@ -180,5 +196,15 @@ module SolidQueue
|
|
|
180
196
|
def enqueue_options
|
|
181
197
|
{ queue: queue_name, priority: priority }.compact
|
|
182
198
|
end
|
|
199
|
+
|
|
200
|
+
def default_time_zone
|
|
201
|
+
SolidQueue.time_zone
|
|
202
|
+
end
|
|
203
|
+
|
|
204
|
+
def report_enqueue_error(error, at:)
|
|
205
|
+
if error
|
|
206
|
+
Rails.error.report(error, handled: true, source: "application.solid_queue", context: { task: key, at: at })
|
|
207
|
+
end
|
|
208
|
+
end
|
|
183
209
|
end
|
|
184
210
|
end
|
|
@@ -32,7 +32,13 @@ module SolidQueue
|
|
|
32
32
|
|
|
33
33
|
class Proxy
|
|
34
34
|
def self.signal_all(jobs)
|
|
35
|
-
|
|
35
|
+
# Guard against incrementing a semaphore's value beyond its limit. Jobs can
|
|
36
|
+
# have different limits, so group them and cap each group with `value < limit`.
|
|
37
|
+
jobs.group_by { |job| job.concurrency_limit || 1 }.each do |limit, grouped_jobs|
|
|
38
|
+
Semaphore.where(key: grouped_jobs.map(&:concurrency_key))
|
|
39
|
+
.where(value: ...limit)
|
|
40
|
+
.update_all("value = value + 1")
|
|
41
|
+
end
|
|
36
42
|
end
|
|
37
43
|
|
|
38
44
|
def initialize(job)
|
|
@@ -8,9 +8,17 @@ module ActiveJob
|
|
|
8
8
|
#
|
|
9
9
|
# Rails.application.config.active_job.queue_adapter = :solid_queue
|
|
10
10
|
class SolidQueueAdapter < (Rails::VERSION::MAJOR == 7 && Rails::VERSION::MINOR == 1 ? Object : AbstractAdapter)
|
|
11
|
-
class_attribute :stopping, default: false, instance_writer: false
|
|
11
|
+
class_attribute :stopping, default: false, instance_writer: false, instance_predicate: false
|
|
12
12
|
SolidQueue.on_worker_stop { self.stopping = true }
|
|
13
13
|
|
|
14
|
+
# Accept an optional job argument for compatibility with Rails main, which
|
|
15
|
+
# began passing the running job to +queue_adapter.stopping?+ so adapters can
|
|
16
|
+
# decide whether to checkpoint based on it. We rely solely on the worker
|
|
17
|
+
# shutdown flag, so the argument is ignored.
|
|
18
|
+
def stopping?(_job = nil)
|
|
19
|
+
self.class.stopping
|
|
20
|
+
end
|
|
21
|
+
|
|
14
22
|
def enqueue_after_transaction_commit?
|
|
15
23
|
true
|
|
16
24
|
end
|
data/lib/solid_queue/cli.rb
CHANGED
|
@@ -30,5 +30,11 @@ module SolidQueue
|
|
|
30
30
|
def start
|
|
31
31
|
SolidQueue::Supervisor.start(**options.symbolize_keys)
|
|
32
32
|
end
|
|
33
|
+
|
|
34
|
+
desc :check, "Validates the Solid Queue configuration for the current Rails env without starting anything. Exits non-zero on errors."
|
|
35
|
+
def check
|
|
36
|
+
configuration = SolidQueue::Configuration.new(**options.symbolize_keys)
|
|
37
|
+
exit 1 unless configuration.check
|
|
38
|
+
end
|
|
33
39
|
end
|
|
34
40
|
end
|
|
@@ -3,10 +3,12 @@
|
|
|
3
3
|
module SolidQueue
|
|
4
4
|
class Configuration
|
|
5
5
|
include ActiveModel::Model
|
|
6
|
+
include ActiveModel::Validations::Callbacks
|
|
6
7
|
|
|
7
|
-
validate :ensure_configured_processes
|
|
8
|
-
validate :
|
|
9
|
-
|
|
8
|
+
validate :ensure_configured_processes, :ensure_valid_recurring_tasks
|
|
9
|
+
validate :warn_about_incorrectly_sized_thread_pool, :warn_about_missing_config_files
|
|
10
|
+
|
|
11
|
+
before_validation { warnings.clear }
|
|
10
12
|
|
|
11
13
|
class Process < Struct.new(:kind, :attributes)
|
|
12
14
|
def instantiate
|
|
@@ -47,26 +49,32 @@ module SolidQueue
|
|
|
47
49
|
end
|
|
48
50
|
end
|
|
49
51
|
|
|
50
|
-
def
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
else
|
|
54
|
-
error_messages = invalid_tasks.map do |task|
|
|
55
|
-
all_messages = task.errors.full_messages.map { |msg| "\t#{msg}" }.join("\n")
|
|
56
|
-
"#{task.key}:\n#{all_messages}"
|
|
57
|
-
end
|
|
58
|
-
.join("\n")
|
|
52
|
+
def mode
|
|
53
|
+
options[:mode].to_s.inquiry
|
|
54
|
+
end
|
|
59
55
|
|
|
60
|
-
|
|
61
|
-
|
|
56
|
+
def standalone?
|
|
57
|
+
mode.fork? || options[:standalone]
|
|
62
58
|
end
|
|
63
59
|
|
|
64
|
-
def
|
|
65
|
-
@
|
|
60
|
+
def warnings
|
|
61
|
+
@warnings ||= ActiveModel::Errors.new(self)
|
|
66
62
|
end
|
|
67
63
|
|
|
68
|
-
def
|
|
69
|
-
|
|
64
|
+
def check
|
|
65
|
+
if valid?
|
|
66
|
+
warnings.full_messages.each { |warning| $stderr.puts warning }
|
|
67
|
+
$stdout.puts "Solid Queue configuration is valid."
|
|
68
|
+
|
|
69
|
+
true
|
|
70
|
+
else
|
|
71
|
+
$stderr.puts "Solid Queue configuration is invalid:"
|
|
72
|
+
(warnings.full_messages + errors.full_messages).each do |message|
|
|
73
|
+
message.each_line { |line| $stderr.puts " #{line.chomp}" }
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
false
|
|
77
|
+
end
|
|
70
78
|
end
|
|
71
79
|
|
|
72
80
|
private
|
|
@@ -88,11 +96,26 @@ module SolidQueue
|
|
|
88
96
|
end
|
|
89
97
|
end
|
|
90
98
|
|
|
91
|
-
def
|
|
92
|
-
|
|
93
|
-
|
|
99
|
+
def warn_about_incorrectly_sized_thread_pool
|
|
100
|
+
db_pool_size = SolidQueue::Record.connection_pool&.size
|
|
101
|
+
|
|
102
|
+
if db_pool_size && db_pool_size < estimated_number_of_threads
|
|
103
|
+
warnings.add(:base, "Warning: Solid Queue is configured to use #{estimated_number_of_threads} threads but the " \
|
|
94
104
|
"database connection pool is #{db_pool_size}. Increase it in `config/database.yml`")
|
|
95
105
|
end
|
|
106
|
+
rescue ActiveRecord::ActiveRecordError
|
|
107
|
+
# No usable database connection. Skip the pool-size warning in that case.
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
def warn_about_missing_config_files
|
|
111
|
+
files = [ options[:config_file] ]
|
|
112
|
+
files << options[:recurring_schedule_file] unless skip_recurring_tasks?
|
|
113
|
+
|
|
114
|
+
files.compact.each do |file|
|
|
115
|
+
unless Pathname.new(file).exist?
|
|
116
|
+
warnings.add(:base, "Warning: provided configuration file '#{file}' does not exist. Falling back to default configuration.")
|
|
117
|
+
end
|
|
118
|
+
end
|
|
96
119
|
end
|
|
97
120
|
|
|
98
121
|
def default_options
|
|
@@ -221,7 +244,6 @@ module SolidQueue
|
|
|
221
244
|
if file.exist?
|
|
222
245
|
ActiveSupport::ConfigurationFile.parse(file).deep_symbolize_keys
|
|
223
246
|
else
|
|
224
|
-
puts "[solid_queue] WARNING: Provided configuration file '#{file}' does not exist. Falling back to default configuration."
|
|
225
247
|
{}
|
|
226
248
|
end
|
|
227
249
|
end
|
data/lib/solid_queue/engine.rb
CHANGED
|
@@ -16,6 +16,12 @@ module SolidQueue
|
|
|
16
16
|
end
|
|
17
17
|
end
|
|
18
18
|
|
|
19
|
+
initializer "solid_queue.time_zone" do |app|
|
|
20
|
+
unless config.solid_queue.key?(:time_zone)
|
|
21
|
+
SolidQueue.time_zone = app.config.time_zone
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
|
|
19
25
|
initializer "solid_queue.app_executor", before: :run_prepare_callbacks do |app|
|
|
20
26
|
config.solid_queue.app_executor ||= app.executor
|
|
21
27
|
config.solid_queue.on_thread_error ||= ->(exception) { Rails.error.report(exception, handled: false) }
|
|
@@ -4,7 +4,9 @@ module SolidQueue::Processes
|
|
|
4
4
|
module Runnable
|
|
5
5
|
include Supervised
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
def mode=(value)
|
|
8
|
+
@mode = (value || DEFAULT_MODE).to_s.inquiry
|
|
9
|
+
end
|
|
8
10
|
|
|
9
11
|
def start
|
|
10
12
|
run_in_mode do
|
|
@@ -33,7 +35,7 @@ module SolidQueue::Processes
|
|
|
33
35
|
DEFAULT_MODE = :async
|
|
34
36
|
|
|
35
37
|
def mode
|
|
36
|
-
|
|
38
|
+
@mode ||= DEFAULT_MODE.to_s.inquiry
|
|
37
39
|
end
|
|
38
40
|
|
|
39
41
|
def run_in_mode(&block)
|
|
@@ -33,8 +33,8 @@ module SolidQueue
|
|
|
33
33
|
end
|
|
34
34
|
end
|
|
35
35
|
|
|
36
|
-
def schedule_task(task)
|
|
37
|
-
scheduled_tasks[task.key] = schedule(task)
|
|
36
|
+
def schedule_task(task, run_at: task.next_time)
|
|
37
|
+
scheduled_tasks[task.key] = schedule(task, run_at: run_at)
|
|
38
38
|
end
|
|
39
39
|
|
|
40
40
|
def unschedule_tasks
|
|
@@ -99,9 +99,11 @@ module SolidQueue
|
|
|
99
99
|
dynamic_tasks_enabled? ? RecurringTask.dynamic.to_a : []
|
|
100
100
|
end
|
|
101
101
|
|
|
102
|
-
def schedule(task)
|
|
103
|
-
|
|
104
|
-
|
|
102
|
+
def schedule(task, run_at: task.next_time)
|
|
103
|
+
delay = [ (run_at - Time.current).to_f, 0.1 ].max
|
|
104
|
+
|
|
105
|
+
scheduled_task = Concurrent::ScheduledTask.new(delay, args: [ self, task, run_at ]) do |thread_schedule, thread_task, thread_task_run_at|
|
|
106
|
+
thread_schedule.schedule_task(thread_task, run_at: thread_task.next_time_after(thread_task_run_at))
|
|
105
107
|
|
|
106
108
|
wrap_in_app_executor do
|
|
107
109
|
thread_task.enqueue(at: thread_task_run_at)
|
|
@@ -13,6 +13,8 @@ module SolidQueue
|
|
|
13
13
|
configuration = Configuration.new(**options)
|
|
14
14
|
|
|
15
15
|
if configuration.valid?
|
|
16
|
+
configuration.warnings.full_messages.each { |warning| SolidQueue.logger.warn(warning) }
|
|
17
|
+
|
|
16
18
|
klass = configuration.mode.fork? ? ForkSupervisor : AsyncSupervisor
|
|
17
19
|
klass.new(configuration).tap(&:start)
|
|
18
20
|
else
|
data/lib/solid_queue/tasks.rb
CHANGED
|
@@ -8,4 +8,10 @@ namespace :solid_queue do
|
|
|
8
8
|
task start: :environment do
|
|
9
9
|
SolidQueue::Supervisor.start
|
|
10
10
|
end
|
|
11
|
+
|
|
12
|
+
desc "validate the Solid Queue configuration for the current Rails env without starting any process"
|
|
13
|
+
task check: :environment do
|
|
14
|
+
configuration = SolidQueue::Configuration.new
|
|
15
|
+
exit 1 unless configuration.check
|
|
16
|
+
end
|
|
11
17
|
end
|
data/lib/solid_queue/version.rb
CHANGED
data/lib/solid_queue.rb
CHANGED
|
@@ -41,6 +41,15 @@ module SolidQueue
|
|
|
41
41
|
mattr_accessor :clear_finished_jobs_after, default: 1.day
|
|
42
42
|
mattr_accessor :default_concurrency_control_period, default: 3.minutes
|
|
43
43
|
|
|
44
|
+
mattr_reader :time_zone
|
|
45
|
+
|
|
46
|
+
def time_zone=(zone)
|
|
47
|
+
@@time_zone = if zone
|
|
48
|
+
resolved = zone.respond_to?(:tzinfo) ? zone : ActiveSupport::TimeZone[zone]
|
|
49
|
+
resolved&.tzinfo&.name || zone.to_s
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
|
|
44
53
|
delegate :on_start, :on_stop, :on_exit, to: Supervisor
|
|
45
54
|
|
|
46
55
|
def schedule_recurring_task(key, **options)
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: solid_queue
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.5.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Rosa Gutierrez
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-07-23 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: activerecord
|
|
@@ -294,6 +294,7 @@ files:
|
|
|
294
294
|
- app/models/solid_queue/queue_selector.rb
|
|
295
295
|
- app/models/solid_queue/ready_execution.rb
|
|
296
296
|
- app/models/solid_queue/record.rb
|
|
297
|
+
- app/models/solid_queue/record/distinct_values.rb
|
|
297
298
|
- app/models/solid_queue/recurring_execution.rb
|
|
298
299
|
- app/models/solid_queue/recurring_task.rb
|
|
299
300
|
- app/models/solid_queue/recurring_task/arguments.rb
|
|
@@ -360,7 +361,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
360
361
|
requirements:
|
|
361
362
|
- - ">="
|
|
362
363
|
- !ruby/object:Gem::Version
|
|
363
|
-
version: '3.
|
|
364
|
+
version: '3.2'
|
|
364
365
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
365
366
|
requirements:
|
|
366
367
|
- - ">="
|