marj 5.0.0 → 6.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +171 -92
- data/lib/marj/mission_control.rb +119 -0
- data/lib/marj/record.rb +2 -2
- data/lib/marj.rb +53 -16
- data/lib/marj_adapter.rb +64 -34
- metadata +5 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6d0039931eb3f0bd43ce090b0b581d6544b26fda022729f6371f4c8ac393076e
|
4
|
+
data.tar.gz: 6abb66afe0bc56064be4ecee3f3e59e78adcf10afb83654529387b785f817335
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ec25e50e772072eb6d13ebcb04a67e8ee4602782b62edab07e9521817993f6d92bac376bff90a6aeca1c724fa61dc4ba04ac1fe8ce9659139d2dcd98c397bce4
|
7
|
+
data.tar.gz: f78b5ba741202818a5374c95fddcb6258b091d4feba4b2de1d1f9de8d1422b99348bf5e354619d215b32b9b7f8d4055ef95d271055cf489e4049b06b40d3463c
|
data/README.md
CHANGED
@@ -2,69 +2,88 @@
|
|
2
2
|
|
3
3
|
A minimal database-backed ActiveJob queueing backend.
|
4
4
|
|
5
|
-
## Purpose
|
6
|
-
|
7
|
-
To provide a database-backed ActiveJob queueing backend with as few features
|
8
|
-
as possible and the minimum backend-specific API required.
|
9
|
-
|
10
5
|
## Quick Links
|
11
6
|
|
12
|
-
API docs: https://gemdocs.org/gems/marj/
|
7
|
+
API docs: https://gemdocs.org/gems/marj/6.0.0/ <br>
|
13
8
|
RubyGems: https://rubygems.org/gems/marj <br>
|
14
9
|
Changelog: https://github.com/nicholasdower/marj/releases <br>
|
15
10
|
Issues: https://github.com/nicholasdower/marj/issues <br>
|
16
11
|
Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
|
17
12
|
|
18
|
-
##
|
13
|
+
## Motivation
|
19
14
|
|
20
|
-
|
15
|
+
There are already several great database-backed ActiveJob queueing backends:
|
16
|
+
- [Delayed::Job](https://github.com/collectiveidea/delayed_job)
|
17
|
+
- [Que](https://github.com/que-rb/que)
|
18
|
+
- [GoodJob](https://github.com/bensheldon/good_job)
|
19
|
+
- [Solid Queue](https://github.com/basecamp/solid_queue)
|
21
20
|
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
- An method is provided to query enqueued jobs.
|
27
|
-
- An method is provided to discard enqueued jobs.
|
28
|
-
- An `ActiveRecord` class is provided to query the database directly.
|
21
|
+
Any of these which support your RDBMS is likely to work well for you. But you
|
22
|
+
may find them to be more featureful and complex than what you require.
|
23
|
+
|
24
|
+
Marj aims to be a minimal alternative.
|
29
25
|
|
30
|
-
|
26
|
+
## Goal
|
31
27
|
|
32
|
-
-
|
33
|
-
-
|
34
|
-
-
|
35
|
-
-
|
36
|
-
-
|
28
|
+
To be the database-backend ActiveJob queueing backend with:
|
29
|
+
- The simplest setup
|
30
|
+
- The fewest configuration options
|
31
|
+
- The fewest features
|
32
|
+
- The fewest backend-specific APIs
|
33
|
+
- The fewest lines of code
|
37
34
|
|
38
|
-
|
39
|
-
|
35
|
+
## Features
|
36
|
+
|
37
|
+
Marj supports and has been tested with MySQL, PostgreSQL and SQLite.
|
38
|
+
|
39
|
+
It provides the following features:
|
40
|
+
- Enqueued jobs are written to the database.
|
41
|
+
- Enqueued jobs can be queried, executed and discarded.
|
42
|
+
- Executed jobs are re-enqueued or discarded, depending on the result.
|
43
|
+
- Compatibility with [Mission Control Jobs](https://github.com/basecamp/mission_control-jobs).
|
44
|
+
|
45
|
+
## Extensions
|
46
|
+
|
47
|
+
Marj does not provide the following features by default, but each can easily
|
48
|
+
be added to your application with a few lines of code. See [Example Usage](#example-usage)
|
49
|
+
and [Extension Examples](#extension-examples) for suggestions.
|
50
|
+
- [Automatic job execution](#example-usage)
|
51
|
+
- [Concurrency protection](#concurrency-protection)
|
52
|
+
- [Job timeouts](#job-timeouts)
|
53
|
+
- [Error storage](#error-storage)
|
54
|
+
- [Discarded job retention](#discarded-job-retention)
|
55
|
+
- [Observability](#observability)
|
56
|
+
- [Multi-DB Support](#multi-db-support)
|
40
57
|
|
41
58
|
## API
|
42
59
|
|
43
|
-
|
60
|
+
Marj relies on existing ActiveJob APIs, for example:
|
44
61
|
|
45
62
|
```ruby
|
46
|
-
|
47
|
-
job.
|
48
|
-
job.perform_now # Perform
|
63
|
+
job.enqueue # Enqueues a job
|
64
|
+
job.perform_now # Performs a job
|
49
65
|
```
|
50
66
|
|
51
|
-
|
52
|
-
|
67
|
+
Additionally, it extends the ActiveJob API with methods required for a
|
68
|
+
minimal queueing backend implementaion:
|
53
69
|
|
54
70
|
```ruby
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
job.discard # Discard
|
71
|
+
SomeJob.query(args) # Queries for enqeueued jobs
|
72
|
+
job.discard # Runs discard callbacks and, by default, deletes the job
|
73
|
+
job.delete # Deletes the job
|
59
74
|
```
|
60
75
|
|
76
|
+
## Requirements
|
77
|
+
|
78
|
+
- `activejob >= 7.1`
|
79
|
+
- `activerecord >= 7.1`
|
80
|
+
|
61
81
|
## Setup
|
62
82
|
|
63
83
|
### 1. Install
|
64
84
|
|
65
85
|
```shell
|
66
|
-
bundle add
|
67
|
-
gem install activejob activerecord marj # or globally
|
86
|
+
bundle add marj
|
68
87
|
```
|
69
88
|
|
70
89
|
### 2. Create the database table
|
@@ -101,29 +120,26 @@ end
|
|
101
120
|
```ruby
|
102
121
|
require 'marj'
|
103
122
|
|
123
|
+
# Choose one of the following:
|
104
124
|
Rails.configuration.active_job.queue_adapter = :marj # Globally, with Rails
|
105
125
|
ActiveJob::Base.queue_adapter = :marj # Globally, without Rails
|
106
126
|
SomeJob.queue_adapter = :marj # Single job
|
107
127
|
```
|
108
128
|
|
109
|
-
### 4.
|
129
|
+
### 4. Optionally, include the Marj module
|
110
130
|
|
111
|
-
|
112
|
-
|
131
|
+
Without any additional setup, jobs can be queried and discarded via the `Marj`
|
132
|
+
module:
|
113
133
|
|
114
134
|
```ruby
|
115
135
|
Marj.query(:all)
|
116
|
-
ActiveJob::Base.queue_adapter.query(:all)
|
117
136
|
Marj.discard(job)
|
118
|
-
ActiveJob::Base.queue_adapter.discard(job)
|
119
137
|
```
|
120
138
|
|
121
139
|
But it is also convenient to query or discard via job classes:
|
122
140
|
|
123
141
|
```ruby
|
124
|
-
ApplicationJob.query(:all)
|
125
142
|
SomeJob.query(:all)
|
126
|
-
ApplicationJob.discard(job)
|
127
143
|
SomeJob.discard(job)
|
128
144
|
job.discard
|
129
145
|
```
|
@@ -131,11 +147,9 @@ job.discard
|
|
131
147
|
In order to enable this functionality, you must include the `Marj` module:
|
132
148
|
|
133
149
|
```ruby
|
134
|
-
class
|
150
|
+
class SomeJob < ActiveJob::Base
|
135
151
|
include Marj
|
136
|
-
end
|
137
152
|
|
138
|
-
class SomeJob < ApplicationJob
|
139
153
|
def perform; end
|
140
154
|
end
|
141
155
|
```
|
@@ -143,27 +157,29 @@ end
|
|
143
157
|
## Example Usage
|
144
158
|
|
145
159
|
```ruby
|
146
|
-
# Enqueue
|
160
|
+
# Enqueue a job
|
147
161
|
job = SomeJob.perform_later('foo')
|
148
|
-
job.perform_now
|
149
162
|
|
150
|
-
#
|
151
|
-
|
163
|
+
# Query jobs
|
164
|
+
SomeJob.query(:all)
|
165
|
+
SomeJob.query(:due)
|
166
|
+
SomeJob.query(:due, queue: :foo)
|
167
|
+
SomeJob.query('8720417d-8fff-4fcf-bc16-22aaef8543d2')
|
152
168
|
|
153
|
-
#
|
154
|
-
|
169
|
+
# Execute a job
|
170
|
+
job.perform_now
|
155
171
|
|
156
|
-
#
|
157
|
-
|
158
|
-
break unless Marj.query(:due, :first)&.tap(&:perform_now)
|
159
|
-
end
|
172
|
+
# Execute jobs which are due (single query)
|
173
|
+
SomeJob.query(:due).map(&:perform_now)
|
160
174
|
|
161
|
-
#
|
162
|
-
|
175
|
+
# Execute jobs which are due (multiple queries)
|
176
|
+
loop {
|
177
|
+
SomeJob.query(:due, :first)&.tap(&:perform_now) || break
|
178
|
+
end
|
163
179
|
|
164
|
-
#
|
180
|
+
# Execute jobs as they become due
|
165
181
|
loop do
|
166
|
-
|
182
|
+
SomeJob.query(:due).each(&:perform_now) rescue logger.error($!)
|
167
183
|
ensure
|
168
184
|
sleep 5.seconds
|
169
185
|
end
|
@@ -175,17 +191,54 @@ By default, jobs enqeued during tests will be written to the database. Enqueued
|
|
175
191
|
jobs can be executed via:
|
176
192
|
|
177
193
|
```ruby
|
178
|
-
|
194
|
+
SomeJob.query(:due).map(&:perform_now)
|
179
195
|
```
|
180
196
|
|
181
|
-
Alternatively, to use [
|
197
|
+
Alternatively, to use ActiveJob's [TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
|
182
198
|
```ruby
|
183
199
|
ActiveJob::Base.queue_adapter = :test
|
184
200
|
```
|
185
201
|
|
186
202
|
## Extension Examples
|
187
203
|
|
188
|
-
|
204
|
+
Most missing features can easily be added to your application with a few lines
|
205
|
+
of code.
|
206
|
+
|
207
|
+
### Concurrency Protection
|
208
|
+
|
209
|
+
To prevent two threads from executing the same job simultaneously, consider
|
210
|
+
moving jobs to a different queue before executing them.
|
211
|
+
|
212
|
+
```ruby
|
213
|
+
class ApplicationJob < ActiveJob::Base
|
214
|
+
around_perform do |job, block|
|
215
|
+
if job.queue_name.start_with?('claimed')
|
216
|
+
raise "Job #{job.job_id} already claimed"
|
217
|
+
end
|
218
|
+
|
219
|
+
updated = Marj::Record
|
220
|
+
.where(job_id: job.job_id, queue_name: job.queue_name)
|
221
|
+
.update_all(
|
222
|
+
queue_name: "claimed-#{job.queue_name}",
|
223
|
+
scheduled_at: Time.now.utc
|
224
|
+
)
|
225
|
+
unless updated == 1
|
226
|
+
raise "Failed to claim job #{job.job_id}. #{updated} records updated"
|
227
|
+
end
|
228
|
+
|
229
|
+
begin
|
230
|
+
block.call
|
231
|
+
rescue StandardError
|
232
|
+
Marj::Record
|
233
|
+
.where(job_id: job.job_id, queue_name: "claimed-#{job.queue_name}")
|
234
|
+
.update_all(queue_name: job.queue_name, scheduled_at: job.scheduled_at)
|
235
|
+
raise
|
236
|
+
end
|
237
|
+
end
|
238
|
+
end
|
239
|
+
```
|
240
|
+
|
241
|
+
### Job Timeouts
|
189
242
|
|
190
243
|
```ruby
|
191
244
|
class ApplicationJob < ActiveJob::Base
|
@@ -205,7 +258,7 @@ class ApplicationJob < ActiveJob::Base
|
|
205
258
|
end
|
206
259
|
```
|
207
260
|
|
208
|
-
###
|
261
|
+
### Error Storage
|
209
262
|
|
210
263
|
```ruby
|
211
264
|
class AddLastErrorToJobs < ActiveRecord::Migration[7.1]
|
@@ -246,51 +299,77 @@ class ApplicationJob < ActiveJob::Base
|
|
246
299
|
end
|
247
300
|
```
|
248
301
|
|
249
|
-
###
|
302
|
+
### Discarded Job Retention
|
250
303
|
|
251
|
-
|
252
|
-
|
304
|
+
By default, discarded jobs are deleted from the database. If desired, this
|
305
|
+
behavior can be overridden, for instance to move jobs to a differnt queue:
|
253
306
|
|
254
307
|
```ruby
|
255
|
-
|
256
|
-
|
257
|
-
|
258
|
-
|
259
|
-
|
260
|
-
|
261
|
-
|
262
|
-
|
263
|
-
|
264
|
-
|
265
|
-
|
266
|
-
table.string :locale, null: false
|
267
|
-
table.string :timezone, null: false
|
268
|
-
end
|
308
|
+
ActiveJob::Base.queue_adapter = MarjAdapter.new(
|
309
|
+
discard: proc { _1.enqueue(queue: 'discarded') }
|
310
|
+
)
|
311
|
+
|
312
|
+
# To retrieve a discarded job, query the discarded queue:
|
313
|
+
job = Marj.query(:first, queue_name: 'discarded')
|
314
|
+
|
315
|
+
# To delete, use one of the following:
|
316
|
+
Marj.delete(job)
|
317
|
+
job.delete
|
318
|
+
```
|
269
319
|
|
270
|
-
|
271
|
-
|
272
|
-
|
320
|
+
### Observability
|
321
|
+
|
322
|
+
For instance with Prometheus metrics:
|
323
|
+
|
324
|
+
```ruby
|
325
|
+
class ApplicationJob < ActiveJob::Base
|
326
|
+
counter = Prometheus::Client::Counter.new(
|
327
|
+
:job_events_total, docstring: '...', labels: [:job, :event]
|
328
|
+
)
|
329
|
+
|
330
|
+
around_enqueue do |job, block|
|
331
|
+
counter.increment(labels: { job: job.class.name, event: 'before_enqueue' })
|
332
|
+
block.call
|
333
|
+
counter.increment(labels: { job: job.class.name, event: 'after_enqueue' })
|
334
|
+
rescue Exception
|
335
|
+
counter.increment(labels: { job: job.class.name, event: 'enqueue_error' })
|
336
|
+
raise
|
273
337
|
end
|
274
338
|
|
275
|
-
|
276
|
-
|
339
|
+
around_perform do |job, block|
|
340
|
+
counter.increment(labels: { job: job.class.name, event: 'before_perform' })
|
341
|
+
block.call
|
342
|
+
counter.increment(labels: { job: job.class.name, event: 'after_perform' })
|
343
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
344
|
+
counter.increment(labels: { job: job.class.name, event: 'perform_error' })
|
345
|
+
raise
|
346
|
+
end
|
347
|
+
|
348
|
+
after_discard do |job|
|
349
|
+
counter.increment(labels: { job: job.class.name, event: 'after_discard' })
|
277
350
|
end
|
278
351
|
end
|
352
|
+
```
|
353
|
+
|
354
|
+
### Multi-DB Support
|
355
|
+
|
356
|
+
It is possible to create a custom record class in order to, for instance,
|
357
|
+
use a different class or table name, or write jobs to multiple
|
358
|
+
databases/tables within a single application.
|
359
|
+
|
360
|
+
Assuming you have a jobs tabled named `my_jobs`:
|
279
361
|
|
362
|
+
```ruby
|
280
363
|
class MyRecord < Marj::Record
|
281
364
|
self.table_name = 'my_jobs'
|
282
365
|
end
|
283
366
|
|
284
|
-
CreateMyJobs.migrate(:up)
|
285
|
-
|
286
367
|
class MyJob < ActiveJob::Base
|
287
|
-
self.queue_adapter = MarjAdapter.new('MyRecord')
|
288
|
-
|
289
368
|
include Marj
|
290
369
|
|
291
|
-
|
292
|
-
|
293
|
-
end
|
370
|
+
self.queue_adapter = MarjAdapter.new('MyRecord')
|
371
|
+
|
372
|
+
def perform; end
|
294
373
|
end
|
295
374
|
|
296
375
|
MyJob.perform_later('oh, hi')
|
@@ -0,0 +1,119 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative '../marj_adapter'
|
4
|
+
require 'mission_control/jobs'
|
5
|
+
require 'mission_control/jobs/adapter'
|
6
|
+
|
7
|
+
# :nocov:
|
8
|
+
module Marj
|
9
|
+
module MissionControl
|
10
|
+
include ::MissionControl::Jobs::Adapter
|
11
|
+
|
12
|
+
def queues
|
13
|
+
record_class.group(:queue_name).count(:queue_name).map { |k, v| { name: k, size: v, active: true } }
|
14
|
+
end
|
15
|
+
|
16
|
+
def queue_size(queue_name)
|
17
|
+
record_class.where(queue_name: queue_name).count
|
18
|
+
end
|
19
|
+
|
20
|
+
def clear_queue(queue_name)
|
21
|
+
Marj::Record.where(queue_name: queue_name).delete_all
|
22
|
+
end
|
23
|
+
|
24
|
+
def pause_queue(_queue_name)
|
25
|
+
raise 'not supported: pause queue'
|
26
|
+
end
|
27
|
+
|
28
|
+
def resume_queue(_queue_name)
|
29
|
+
raise 'not supported: resume queue'
|
30
|
+
end
|
31
|
+
|
32
|
+
def queue_paused?(_queue_name)
|
33
|
+
false
|
34
|
+
end
|
35
|
+
|
36
|
+
def supported_statuses
|
37
|
+
%i[pending failed scheduled]
|
38
|
+
end
|
39
|
+
|
40
|
+
def supported_filters(_jobs_relation)
|
41
|
+
%i[queue_name job_class_name]
|
42
|
+
end
|
43
|
+
|
44
|
+
def exposes_workers?
|
45
|
+
false
|
46
|
+
end
|
47
|
+
|
48
|
+
def workers
|
49
|
+
raise 'not supported: workers'
|
50
|
+
end
|
51
|
+
|
52
|
+
def find_worker(_worker_id)
|
53
|
+
raise 'not supported: find workers'
|
54
|
+
end
|
55
|
+
|
56
|
+
def jobs_count(jobs_relation)
|
57
|
+
ar_relation(jobs_relation).count
|
58
|
+
end
|
59
|
+
|
60
|
+
def fetch_jobs(jobs_relation)
|
61
|
+
ar_relation(jobs_relation).each_with_index.map { |record, index| to_job(record, jobs_relation, index) }
|
62
|
+
end
|
63
|
+
|
64
|
+
def retry_all_jobs(jobs_relation)
|
65
|
+
ar_relation(jobs_relation).map { |record| record.to_job.perform_now }
|
66
|
+
end
|
67
|
+
|
68
|
+
def retry_job(job, _jobs_relation)
|
69
|
+
Marj::Record.find(job.job_id).to_job.perform_now
|
70
|
+
end
|
71
|
+
|
72
|
+
def discard_all_jobs(jobs_relation)
|
73
|
+
ar_relation(jobs_relation).map { |record| discard(record.to_job) }
|
74
|
+
end
|
75
|
+
|
76
|
+
def discard_job(job, _jobs_relation)
|
77
|
+
discard(Marj::Record.find(job.job_id).to_job)
|
78
|
+
end
|
79
|
+
|
80
|
+
def find_job(job_id, jobs_relation)
|
81
|
+
to_job(record_class.find_by(job_id: job_id), jobs_relation)
|
82
|
+
end
|
83
|
+
|
84
|
+
private
|
85
|
+
|
86
|
+
def ar_relation(jobs_relation)
|
87
|
+
relation = Marj::Record.all.offset(jobs_relation.offset_value).limit(jobs_relation.limit_value)
|
88
|
+
relation = relation.where.not(executions: 0) if jobs_relation.status == :failed
|
89
|
+
relation = relation.where.not(scheduled_at: nil) if jobs_relation.status == :scheduled
|
90
|
+
relation = relation.where(job_class: jobs_relation.job_class_name) if jobs_relation.job_class_name
|
91
|
+
relation = relation.where(queue_name: jobs_relation.queue_name) if jobs_relation.queue_name
|
92
|
+
relation
|
93
|
+
end
|
94
|
+
|
95
|
+
def to_job(record, jobs_relation, index = 0)
|
96
|
+
return nil unless record
|
97
|
+
|
98
|
+
job = record.to_job
|
99
|
+
job_data = job.serialize
|
100
|
+
ActiveJob::JobProxy.new(job_data).tap do |proxy|
|
101
|
+
if job.executions.positive?
|
102
|
+
proxy.last_execution_error = ActiveJob::ExecutionError.new(
|
103
|
+
error_class: Exception, message: 'unknown', backtrace: []
|
104
|
+
)
|
105
|
+
proxy.failed_at = job.enqueued_at
|
106
|
+
proxy.status = :failed
|
107
|
+
elsif job.scheduled_at
|
108
|
+
proxy.status = :scheduled
|
109
|
+
else
|
110
|
+
proxy.status = :pending
|
111
|
+
end
|
112
|
+
proxy.raw_data = job_data
|
113
|
+
proxy.position = jobs_relation.offset_value + index
|
114
|
+
proxy.arguments = job.arguments # For some reason MissionControl sets the arguments to the entire job data
|
115
|
+
end
|
116
|
+
end
|
117
|
+
end
|
118
|
+
end
|
119
|
+
# :nocov:
|
data/lib/marj/record.rb
CHANGED
@@ -84,8 +84,8 @@ module Marj
|
|
84
84
|
# - If a job succeeds, after_perform will be called.
|
85
85
|
# - If a job fails and should be retried, enqueue will be called. This is handled by the queue adapter.
|
86
86
|
# - If a job exceeds its max attempts, after_discard will be called.
|
87
|
-
job.singleton_class.after_perform { |_j|
|
88
|
-
job.singleton_class.after_discard { |_j, _exception|
|
87
|
+
job.singleton_class.after_perform { |_j| job.queue_adapter.delete(job) }
|
88
|
+
job.singleton_class.after_discard { |_j, _exception| job.queue_adapter.discard(job, run_callbacks: false) }
|
89
89
|
job.singleton_class.instance_variable_set(:@record, record)
|
90
90
|
|
91
91
|
job
|
data/lib/marj.rb
CHANGED
@@ -6,13 +6,15 @@ require_relative 'marj_adapter'
|
|
6
6
|
#
|
7
7
|
# The {Marj} module provides the following methods:
|
8
8
|
# - +query+ - Queries enqueued jobs
|
9
|
-
# - +discard+ - Discards a job
|
9
|
+
# - +discard+ - Discards a job, by default by executing after_discard callbacks and delegating to delete
|
10
|
+
# - +delete+ - Deletes a job
|
10
11
|
#
|
11
12
|
# It is possible to call the above methods on the {Marj} module itself or on any class which includes it.
|
12
13
|
#
|
13
14
|
# Example usage:
|
14
15
|
# Marj.query(:first) # Returns the first job
|
15
16
|
# Marj.discard(job) # Discards the specified job
|
17
|
+
# Marj.delete(job) # Deletes the specified job
|
16
18
|
#
|
17
19
|
# class ApplicationJob < ActiveJob::Base
|
18
20
|
# include Marj
|
@@ -25,33 +27,37 @@ require_relative 'marj_adapter'
|
|
25
27
|
# job = ApplicationJob.query(:first) # Returns the first enqueued job
|
26
28
|
# job = SomeJob.query(:first) # Returns the first enqueued job with job_class SomeJob
|
27
29
|
# ApplicationJob.discard(job) # Discards the specified job
|
30
|
+
# ApplicationJob.delete(job) # Deletes the specified job
|
28
31
|
# job.discard # Discards the job
|
32
|
+
# job.delete # Deletes the job
|
29
33
|
#
|
30
34
|
# See https://github.com/nicholasdower/marj
|
31
35
|
module Marj
|
32
36
|
# The Marj version.
|
33
|
-
VERSION = '
|
37
|
+
VERSION = '6.0.0'
|
34
38
|
|
35
39
|
Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
|
36
40
|
|
37
41
|
# Provides the {query} and {discard} class methods.
|
38
42
|
module ClassMethods
|
39
|
-
# Queries enqueued jobs.
|
43
|
+
# Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
|
44
|
+
# - Symbol arguments are treated as +ActiveRecord+ scopes.
|
45
|
+
# - If only a job ID is specified, the corresponding job is returned.
|
46
|
+
# - If +:limit+ is specified, the maximum number of jobs is limited.
|
47
|
+
# - If +:order+ is specified, the jobs are ordered by the given attribute.
|
40
48
|
#
|
41
|
-
#
|
49
|
+
# By default jobs are ordered by when they should be executed.
|
42
50
|
#
|
43
51
|
# Example usage:
|
44
|
-
# query
|
45
|
-
# query(:
|
46
|
-
# query(:
|
47
|
-
# query(
|
48
|
-
#
|
49
|
-
# query(
|
50
|
-
# query(
|
51
|
-
# query(job_id: '123')
|
52
|
-
#
|
53
|
-
# query(queue: 'foo') # Returns all jobs in the 'foo' queue
|
54
|
-
# query(job_queue: 'foo') # Same as above
|
52
|
+
# query # Returns all jobs
|
53
|
+
# query(:all) # Returns all jobs
|
54
|
+
# query(:due) # Returns jobs which are due to be executed
|
55
|
+
# query(:due, limit: 10) # Returns at most 10 jobs which are due to be executed
|
56
|
+
# query(job_class: Foo) # Returns all jobs with job_class Foo
|
57
|
+
# query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
|
58
|
+
# query(queue_name: 'foo') # Returns all jobs in the 'foo' queue
|
59
|
+
# query(job_id: '123') # Returns the job with job_id '123' or nil if no such job exists
|
60
|
+
# query('123') # Returns the job with job_id '123' or nil if no such job exists
|
55
61
|
def query(*args, **kwargs)
|
56
62
|
kwargs[:job_class] ||= self if self < ActiveJob::Base && name != 'ApplicationJob'
|
57
63
|
queue_adapter.query(*args, **kwargs)
|
@@ -63,6 +69,13 @@ module Marj
|
|
63
69
|
def discard(job)
|
64
70
|
queue_adapter.discard(job)
|
65
71
|
end
|
72
|
+
|
73
|
+
# Deletes the record associated with the specified job.
|
74
|
+
#
|
75
|
+
# @return [ActiveJob::Base] the deleted job
|
76
|
+
def delete(job)
|
77
|
+
queue_adapter.delete(job)
|
78
|
+
end
|
66
79
|
end
|
67
80
|
|
68
81
|
# (see ClassMethods#query)
|
@@ -75,13 +88,25 @@ module Marj
|
|
75
88
|
queue_adapter.discard(job)
|
76
89
|
end
|
77
90
|
|
78
|
-
#
|
91
|
+
# (see ClassMethods#delete)
|
92
|
+
def self.delete(job)
|
93
|
+
queue_adapter.delete(job)
|
94
|
+
end
|
95
|
+
|
96
|
+
# Deletes this job.
|
79
97
|
#
|
80
98
|
# @return [ActiveJob::Base] this job
|
81
99
|
def discard
|
82
100
|
self.class.queue_adapter.discard(self)
|
83
101
|
end
|
84
102
|
|
103
|
+
# Deletes the record associated with this job.
|
104
|
+
#
|
105
|
+
# @return [ActiveJob::Base] this job
|
106
|
+
def delete
|
107
|
+
self.class.queue_adapter.delete(self)
|
108
|
+
end
|
109
|
+
|
85
110
|
def self.included(clazz)
|
86
111
|
clazz.extend(ClassMethods)
|
87
112
|
end
|
@@ -92,3 +117,15 @@ module Marj
|
|
92
117
|
end
|
93
118
|
private_class_method :queue_adapter
|
94
119
|
end
|
120
|
+
|
121
|
+
# :nocov:
|
122
|
+
if defined?(Rails)
|
123
|
+
begin
|
124
|
+
require 'mission_control/jobs'
|
125
|
+
require_relative 'marj/mission_control'
|
126
|
+
MarjAdapter.include(Marj::MissionControl)
|
127
|
+
rescue LoadError
|
128
|
+
# ignore
|
129
|
+
end
|
130
|
+
end
|
131
|
+
# :nocov:
|
data/lib/marj_adapter.rb
CHANGED
@@ -2,11 +2,13 @@
|
|
2
2
|
|
3
3
|
# ActiveJob queue adapter for Marj.
|
4
4
|
#
|
5
|
-
# In addition to the standard +ActiveJob+ queue adapter API, this adapter provides
|
6
|
-
#
|
5
|
+
# In addition to the standard +ActiveJob+ queue adapter API, this adapter provides:
|
6
|
+
# - A +query+ method which can be used to query enqueued jobs
|
7
|
+
# - A +discard+ method which can be used to discard enqueued jobs.
|
8
|
+
# - A +delete+ method which can be used to delete enqueued jobs.
|
7
9
|
#
|
8
|
-
# Although it is possible to access the adapter directly in order to query or
|
9
|
-
# {Marj} module.
|
10
|
+
# Although it is possible to access the adapter directly in order to query, discard or delete, it is recommended to use
|
11
|
+
# the {Marj} module.
|
10
12
|
#
|
11
13
|
# See https://github.com/nicholasdower/marj
|
12
14
|
class MarjAdapter
|
@@ -15,9 +17,11 @@ class MarjAdapter
|
|
15
17
|
|
16
18
|
# Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ class.
|
17
19
|
#
|
18
|
-
# @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use to
|
19
|
-
|
20
|
+
# @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use, defaults to +Marj::Record+
|
21
|
+
# @param discard [Proc] the proc to use to discard jobs, defaults to delegating to {delete}
|
22
|
+
def initialize(record_class: 'Marj::Record', discard: proc { |job| delete(job) })
|
20
23
|
@record_class = record_class
|
24
|
+
@discard_proc = discard
|
21
25
|
end
|
22
26
|
|
23
27
|
# Enqueue a job for immediate execution.
|
@@ -77,25 +81,25 @@ class MarjAdapter
|
|
77
81
|
end
|
78
82
|
|
79
83
|
# Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
|
80
|
-
# -
|
84
|
+
# - Symbol arguments are treated as +ActiveRecord+ scopes.
|
81
85
|
# - If only a job ID is specified, the corresponding job is returned.
|
82
86
|
# - If +:limit+ is specified, the maximum number of jobs is limited.
|
87
|
+
# - If +:order+ is specified, the jobs are ordered by the given attribute.
|
83
88
|
#
|
84
|
-
#
|
85
|
-
# query(:all) # Delegates to Marj::Record.all
|
86
|
-
# query(:due) # Delegates to Marj::Record.due
|
87
|
-
# query(:all, limit: 10) # Returns a maximum of 10 jobs
|
88
|
-
# query(job_class: Foo) # Returns all jobs with job_class Foo
|
89
|
-
#
|
90
|
-
# query('123') # Returns the job with id '123' or nil if no such job exists
|
91
|
-
# query(id: '123') # Same as above
|
92
|
-
# query(job_id: '123') # Same as above
|
89
|
+
# By default jobs are ordered by when they should be executed.
|
93
90
|
#
|
94
|
-
#
|
95
|
-
# query
|
91
|
+
# Example usage:
|
92
|
+
# query # Returns all jobs
|
93
|
+
# query(:all) # Returns all jobs
|
94
|
+
# query(:due) # Returns jobs which are due to be executed
|
95
|
+
# query(:due, limit: 10) # Returns at most 10 jobs which are due to be executed
|
96
|
+
# query(job_class: Foo) # Returns all jobs with job_class Foo
|
97
|
+
# query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
|
98
|
+
# query(queue_name: 'foo') # Returns all jobs in the 'foo' queue
|
99
|
+
# query(job_id: '123') # Returns the job with job_id '123' or nil if no such job exists
|
100
|
+
# query('123') # Returns the job with job_id '123' or nil if no such job exists
|
96
101
|
def query(*args, **kwargs)
|
97
102
|
args, kwargs = args.dup, kwargs.dup.symbolize_keys
|
98
|
-
kwargs = kwargs.merge(queue_name: kwargs.delete(:queue)) if kwargs.key?(:queue)
|
99
103
|
kwargs = kwargs.merge(job_id: kwargs.delete(:id)) if kwargs.key?(:id)
|
100
104
|
kwargs[:job_id] = args.shift if args.size == 1 && args.first.is_a?(String) && args.first.match(JOB_ID_REGEX)
|
101
105
|
|
@@ -103,18 +107,16 @@ class MarjAdapter
|
|
103
107
|
return record_class.find_by(job_id: kwargs[:job_id])&.to_job
|
104
108
|
end
|
105
109
|
|
106
|
-
symbol_args =
|
107
|
-
symbol_args
|
108
|
-
order_by = kwargs.delete(:order)
|
109
|
-
order_by = :queue_name if [:queue, 'queue'].include?(order_by)
|
110
|
+
symbol_args, args = args.partition { _1.is_a?(Symbol) }
|
111
|
+
symbol_args.delete(:all)
|
110
112
|
limit = kwargs.delete(:limit)
|
111
|
-
symbol_args.shift if symbol_args.first == :all
|
112
113
|
relation = record_class.all
|
113
|
-
relation = relation.order(
|
114
|
-
relation = relation.by_due_date unless relation.order_values.any?
|
114
|
+
relation = relation.order(kwargs.delete(:order)) if kwargs.key?(:order)
|
115
115
|
relation = relation.where(*args, **kwargs) if args.any? || kwargs.any?
|
116
116
|
relation = relation.limit(limit) if limit
|
117
117
|
relation = relation.send(symbol_args.shift) while symbol_args.any?
|
118
|
+
relation = relation.by_due_date if relation.is_a?(ActiveRecord::Relation) && relation.order_values.empty?
|
119
|
+
|
118
120
|
if relation.is_a?(Enumerable)
|
119
121
|
relation.map(&:to_job)
|
120
122
|
elsif relation.is_a?(record_class)
|
@@ -126,12 +128,46 @@ class MarjAdapter
|
|
126
128
|
|
127
129
|
# Discards the specified job.
|
128
130
|
#
|
131
|
+
# @param job [ActiveJob::Base] the job being discarded
|
132
|
+
# @param run_callbacks [Boolean] whether to run the +after_discard+ callbacks
|
129
133
|
# @return [ActiveJob::Base] the discarded job
|
130
|
-
def discard(job)
|
134
|
+
def discard(job, run_callbacks: true)
|
135
|
+
job.tap do
|
136
|
+
@discard_proc.call(job)
|
137
|
+
run_after_discard_callbacks(job) if run_callbacks
|
138
|
+
end
|
139
|
+
end
|
140
|
+
|
141
|
+
# Deletes the record associated with the specified job.
|
142
|
+
#
|
143
|
+
# @return [ActiveJob::Base] the deleted job
|
144
|
+
def delete(job)
|
145
|
+
job.tap { destroy_record(job) }
|
146
|
+
end
|
147
|
+
|
148
|
+
private
|
149
|
+
|
150
|
+
# Returns the +ActiveRecord+ class to use to store jobs.
|
151
|
+
#
|
152
|
+
# @return [Class] the +ActiveRecord+ class
|
153
|
+
def record_class
|
154
|
+
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
155
|
+
end
|
156
|
+
|
157
|
+
# Destroys the record associated with the given job if it exists.
|
158
|
+
#
|
159
|
+
# @return [ActiveRecord::Base, NilClass] the destroyed record or +nil+ if no such record exists
|
160
|
+
def destroy_record(job)
|
131
161
|
record = job.singleton_class.instance_variable_get(:@record)
|
132
162
|
record ||= Marj::Record.find_by(job_id: job.job_id)&.tap { _1.send(:register_callbacks, job) }
|
133
163
|
record&.destroy
|
164
|
+
end
|
134
165
|
|
166
|
+
# Invokes the specified job's +after_discard+ callbacks.
|
167
|
+
#
|
168
|
+
# @param job [ActiveJob::Base] the job being discarded
|
169
|
+
# @return [NilClass] the given job
|
170
|
+
def run_after_discard_callbacks(job)
|
135
171
|
# Copied from ActiveJob::Exceptions#run_after_discard_procs
|
136
172
|
exceptions = []
|
137
173
|
job.after_discard_procs.each do |blk|
|
@@ -141,12 +177,6 @@ class MarjAdapter
|
|
141
177
|
end
|
142
178
|
raise exceptions.last if exceptions.any?
|
143
179
|
|
144
|
-
|
145
|
-
end
|
146
|
-
|
147
|
-
private
|
148
|
-
|
149
|
-
def record_class
|
150
|
-
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
180
|
+
nil
|
151
181
|
end
|
152
182
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: marj
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 6.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Nick Dower
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-
|
11
|
+
date: 2024-02-05 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -48,6 +48,7 @@ files:
|
|
48
48
|
- LICENSE.txt
|
49
49
|
- README.md
|
50
50
|
- lib/marj.rb
|
51
|
+
- lib/marj/mission_control.rb
|
51
52
|
- lib/marj/record.rb
|
52
53
|
- lib/marj_adapter.rb
|
53
54
|
homepage: https://github.com/nicholasdower/marj
|
@@ -55,8 +56,8 @@ licenses:
|
|
55
56
|
- MIT
|
56
57
|
metadata:
|
57
58
|
bug_tracker_uri: https://github.com/nicholasdower/marj/issues
|
58
|
-
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/
|
59
|
-
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/
|
59
|
+
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v6.0.0
|
60
|
+
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v6.0.0
|
60
61
|
homepage_uri: https://github.com/nicholasdower/marj
|
61
62
|
rubygems_mfa_required: 'true'
|
62
63
|
source_code_uri: https://github.com/nicholasdower/marj
|