marj 4.1.0 → 6.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +219 -133
- data/lib/marj/mission_control.rb +119 -0
- data/lib/marj/record.rb +49 -6
- data/lib/marj.rb +95 -128
- data/lib/marj_adapter.rb +151 -5
- metadata +5 -6
- data/lib/marj/jobs_interface.rb +0 -113
- data/lib/marj/relation.rb +0 -72
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6d0039931eb3f0bd43ce090b0b581d6544b26fda022729f6371f4c8ac393076e
|
4
|
+
data.tar.gz: 6abb66afe0bc56064be4ecee3f3e59e78adcf10afb83654529387b785f817335
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ec25e50e772072eb6d13ebcb04a67e8ee4602782b62edab07e9521817993f6d92bac376bff90a6aeca1c724fa61dc4ba04ac1fe8ce9659139d2dcd98c397bce4
|
7
|
+
data.tar.gz: f78b5ba741202818a5374c95fddcb6258b091d4feba4b2de1d1f9de8d1422b99348bf5e354619d215b32b9b7f8d4055ef95d271055cf489e4049b06b40d3463c
|
data/README.md
CHANGED
@@ -4,42 +4,89 @@ A minimal database-backed ActiveJob queueing backend.
|
|
4
4
|
|
5
5
|
## Quick Links
|
6
6
|
|
7
|
-
API docs: https://gemdocs.org/gems/marj/
|
7
|
+
API docs: https://gemdocs.org/gems/marj/6.0.0/ <br>
|
8
8
|
RubyGems: https://rubygems.org/gems/marj <br>
|
9
9
|
Changelog: https://github.com/nicholasdower/marj/releases <br>
|
10
10
|
Issues: https://github.com/nicholasdower/marj/issues <br>
|
11
11
|
Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
|
12
12
|
|
13
|
+
## Motivation
|
14
|
+
|
15
|
+
There are already several great database-backed ActiveJob queueing backends:
|
16
|
+
- [Delayed::Job](https://github.com/collectiveidea/delayed_job)
|
17
|
+
- [Que](https://github.com/que-rb/que)
|
18
|
+
- [GoodJob](https://github.com/bensheldon/good_job)
|
19
|
+
- [Solid Queue](https://github.com/basecamp/solid_queue)
|
20
|
+
|
21
|
+
Any of these which support your RDBMS is likely to work well for you. But you
|
22
|
+
may find them to be more featureful and complex than what you require.
|
23
|
+
|
24
|
+
Marj aims to be a minimal alternative.
|
25
|
+
|
26
|
+
## Goal
|
27
|
+
|
28
|
+
To be the database-backend ActiveJob queueing backend with:
|
29
|
+
- The simplest setup
|
30
|
+
- The fewest configuration options
|
31
|
+
- The fewest features
|
32
|
+
- The fewest backend-specific APIs
|
33
|
+
- The fewest lines of code
|
34
|
+
|
13
35
|
## Features
|
14
36
|
|
37
|
+
Marj supports and has been tested with MySQL, PostgreSQL and SQLite.
|
38
|
+
|
39
|
+
It provides the following features:
|
15
40
|
- Enqueued jobs are written to the database.
|
16
|
-
-
|
17
|
-
-
|
18
|
-
-
|
19
|
-
|
20
|
-
|
41
|
+
- Enqueued jobs can be queried, executed and discarded.
|
42
|
+
- Executed jobs are re-enqueued or discarded, depending on the result.
|
43
|
+
- Compatibility with [Mission Control Jobs](https://github.com/basecamp/mission_control-jobs).
|
44
|
+
|
45
|
+
## Extensions
|
46
|
+
|
47
|
+
Marj does not provide the following features by default, but each can easily
|
48
|
+
be added to your application with a few lines of code. See [Example Usage](#example-usage)
|
49
|
+
and [Extension Examples](#extension-examples) for suggestions.
|
50
|
+
- [Automatic job execution](#example-usage)
|
51
|
+
- [Concurrency protection](#concurrency-protection)
|
52
|
+
- [Job timeouts](#job-timeouts)
|
53
|
+
- [Error storage](#error-storage)
|
54
|
+
- [Discarded job retention](#discarded-job-retention)
|
55
|
+
- [Observability](#observability)
|
56
|
+
- [Multi-DB Support](#multi-db-support)
|
57
|
+
|
58
|
+
## API
|
59
|
+
|
60
|
+
Marj relies on existing ActiveJob APIs, for example:
|
61
|
+
|
62
|
+
```ruby
|
63
|
+
job.enqueue # Enqueues a job
|
64
|
+
job.perform_now # Performs a job
|
65
|
+
```
|
66
|
+
|
67
|
+
Additionally, it extends the ActiveJob API with methods required for a
|
68
|
+
minimal queueing backend implementaion:
|
69
|
+
|
70
|
+
```ruby
|
71
|
+
SomeJob.query(args) # Queries for enqeueued jobs
|
72
|
+
job.discard # Runs discard callbacks and, by default, deletes the job
|
73
|
+
job.delete # Deletes the job
|
74
|
+
```
|
21
75
|
|
22
|
-
##
|
76
|
+
## Requirements
|
23
77
|
|
24
|
-
-
|
25
|
-
-
|
26
|
-
- Concurrency Controls
|
27
|
-
- Observability
|
28
|
-
- A User Interace
|
78
|
+
- `activejob >= 7.1`
|
79
|
+
- `activerecord >= 7.1`
|
29
80
|
|
30
81
|
## Setup
|
31
82
|
|
32
83
|
### 1. Install
|
33
84
|
|
34
85
|
```shell
|
35
|
-
bundle add
|
36
|
-
|
37
|
-
# or
|
38
|
-
|
39
|
-
gem install activejob activerecord marj
|
86
|
+
bundle add marj
|
40
87
|
```
|
41
88
|
|
42
|
-
###
|
89
|
+
### 2. Create the database table
|
43
90
|
|
44
91
|
```ruby
|
45
92
|
class CreateJobs < ActiveRecord::Migration[7.1]
|
@@ -68,149 +115,74 @@ class CreateJobs < ActiveRecord::Migration[7.1]
|
|
68
115
|
end
|
69
116
|
```
|
70
117
|
|
71
|
-
|
72
|
-
table name, set `Marj.table_name` before loading `ActiveRecord`.
|
73
|
-
|
74
|
-
### 4. Configure the queue adapter
|
118
|
+
### 3. Configure the queue adapter
|
75
119
|
|
76
120
|
```ruby
|
77
121
|
require 'marj'
|
78
122
|
|
123
|
+
# Choose one of the following:
|
79
124
|
Rails.configuration.active_job.queue_adapter = :marj # Globally, with Rails
|
80
125
|
ActiveJob::Base.queue_adapter = :marj # Globally, without Rails
|
81
126
|
SomeJob.queue_adapter = :marj # Single job
|
82
127
|
```
|
83
128
|
|
84
|
-
|
85
|
-
|
86
|
-
```ruby
|
87
|
-
# Enqueue and manually run a job:
|
88
|
-
job = SomeJob.perform_later('foo')
|
89
|
-
job.perform_now
|
129
|
+
### 4. Optionally, include the Marj module
|
90
130
|
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
# Run all due jobs (single DB query)
|
95
|
-
Marj.due.perform_all
|
96
|
-
|
97
|
-
# Run all due jobs (multiple DB queries)
|
98
|
-
Marj.due.perform_all(batch_size: 1)
|
99
|
-
|
100
|
-
# Run all due jobs in a specific queue:
|
101
|
-
Marj.queue('foo').due.perform_all
|
102
|
-
|
103
|
-
# Run jobs as they become due:
|
104
|
-
loop do
|
105
|
-
Marj.due.perform_all rescue logger.error($!)
|
106
|
-
ensure
|
107
|
-
sleep 5.seconds
|
108
|
-
end
|
109
|
-
```
|
110
|
-
|
111
|
-
## Jobs Interface
|
112
|
-
|
113
|
-
The `Marj` module provides methods for interacting with enqueued jobs. These
|
114
|
-
methods accept, return and yield +ActiveJob+ objects rather than +ActiveRecord+
|
115
|
-
objects. Returned jobs are orderd by due date. To query the database directly,
|
116
|
-
use `Marj::Record`.
|
117
|
-
|
118
|
-
Example usage:
|
131
|
+
Without any additional setup, jobs can be queried and discarded via the `Marj`
|
132
|
+
module:
|
119
133
|
|
120
134
|
```ruby
|
121
|
-
Marj.all
|
122
|
-
Marj.
|
123
|
-
Marj.due # Returns jobs which are due to be executed.
|
124
|
-
Marj.next # Returns the next job(s) to be executed.
|
125
|
-
Marj.count # Returns the number of enqueued jobs.
|
126
|
-
Marj.where # Returns jobs matching the specified criteria.
|
127
|
-
Marj.perform_all # Executes all jobs.
|
128
|
-
Marj.discard_all # Discards all jobs.
|
129
|
-
Marj.discard # Discards the specified job.
|
135
|
+
Marj.query(:all)
|
136
|
+
Marj.discard(job)
|
130
137
|
```
|
131
138
|
|
132
|
-
|
139
|
+
But it is also convenient to query or discard via job classes:
|
133
140
|
|
134
141
|
```ruby
|
135
|
-
|
142
|
+
SomeJob.query(:all)
|
143
|
+
SomeJob.discard(job)
|
144
|
+
job.discard
|
136
145
|
```
|
137
146
|
|
138
|
-
|
139
|
-
|
140
|
-
The `Marj::JobsInterface` can be added to any class or module. For example, to
|
141
|
-
add it to all jobs classes:
|
147
|
+
In order to enable this functionality, you must include the `Marj` module:
|
142
148
|
|
143
149
|
```ruby
|
144
|
-
class
|
145
|
-
|
150
|
+
class SomeJob < ActiveJob::Base
|
151
|
+
include Marj
|
146
152
|
|
147
|
-
def
|
148
|
-
Marj::Relation.new(
|
149
|
-
self == ApplicationJob ?
|
150
|
-
Marj::Record.ordered : Marj::Record.where(job_class: self)
|
151
|
-
)
|
152
|
-
end
|
153
|
+
def perform; end
|
153
154
|
end
|
154
|
-
|
155
|
-
class SomeJob < ApplicationJob; end
|
156
|
-
|
157
|
-
ApplicationJob.due # Returns all jobs which are due to be executed.
|
158
|
-
SomeJob.due # Returns SomeJobs which are due to be executed.
|
159
155
|
```
|
160
156
|
|
161
|
-
##
|
162
|
-
|
163
|
-
It is possible to create a custom record class in order to, for instance,
|
164
|
-
write jobs to multiple databases/tables within a single application.
|
157
|
+
## Example Usage
|
165
158
|
|
166
159
|
```ruby
|
167
|
-
|
168
|
-
|
169
|
-
create_table :my_jobs, id: :string, primary_key: :job_id do |table|
|
170
|
-
table.string :job_class, null: false
|
171
|
-
table.text :arguments, null: false
|
172
|
-
table.string :queue_name, null: false
|
173
|
-
table.integer :priority
|
174
|
-
table.integer :executions, null: false
|
175
|
-
table.text :exception_executions, null: false
|
176
|
-
table.datetime :enqueued_at, null: false
|
177
|
-
table.datetime :scheduled_at
|
178
|
-
table.string :locale, null: false
|
179
|
-
table.string :timezone, null: false
|
180
|
-
end
|
181
|
-
|
182
|
-
add_index :my_jobs, %i[enqueued_at]
|
183
|
-
add_index :my_jobs, %i[scheduled_at]
|
184
|
-
add_index :my_jobs, %i[priority scheduled_at enqueued_at]
|
185
|
-
end
|
186
|
-
|
187
|
-
def self.down
|
188
|
-
drop_table :my_jobs
|
189
|
-
end
|
190
|
-
end
|
191
|
-
|
192
|
-
class MyRecord < Marj::Record
|
193
|
-
self.table_name = 'my_jobs'
|
194
|
-
end
|
195
|
-
|
196
|
-
CreateMyJobs.migrate(:up)
|
160
|
+
# Enqueue a job
|
161
|
+
job = SomeJob.perform_later('foo')
|
197
162
|
|
198
|
-
|
199
|
-
|
163
|
+
# Query jobs
|
164
|
+
SomeJob.query(:all)
|
165
|
+
SomeJob.query(:due)
|
166
|
+
SomeJob.query(:due, queue: :foo)
|
167
|
+
SomeJob.query('8720417d-8fff-4fcf-bc16-22aaef8543d2')
|
200
168
|
|
201
|
-
|
169
|
+
# Execute a job
|
170
|
+
job.perform_now
|
202
171
|
|
203
|
-
|
204
|
-
|
205
|
-
end
|
172
|
+
# Execute jobs which are due (single query)
|
173
|
+
SomeJob.query(:due).map(&:perform_now)
|
206
174
|
|
207
|
-
|
208
|
-
|
209
|
-
|
175
|
+
# Execute jobs which are due (multiple queries)
|
176
|
+
loop {
|
177
|
+
SomeJob.query(:due, :first)&.tap(&:perform_now) || break
|
210
178
|
end
|
211
179
|
|
212
|
-
|
213
|
-
|
180
|
+
# Execute jobs as they become due
|
181
|
+
loop do
|
182
|
+
SomeJob.query(:due).each(&:perform_now) rescue logger.error($!)
|
183
|
+
ensure
|
184
|
+
sleep 5.seconds
|
185
|
+
end
|
214
186
|
```
|
215
187
|
|
216
188
|
## Testing
|
@@ -219,17 +191,54 @@ By default, jobs enqeued during tests will be written to the database. Enqueued
|
|
219
191
|
jobs can be executed via:
|
220
192
|
|
221
193
|
```ruby
|
222
|
-
|
194
|
+
SomeJob.query(:due).map(&:perform_now)
|
223
195
|
```
|
224
196
|
|
225
|
-
Alternatively, to use [
|
197
|
+
Alternatively, to use ActiveJob's [TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
|
226
198
|
```ruby
|
227
199
|
ActiveJob::Base.queue_adapter = :test
|
228
200
|
```
|
229
201
|
|
230
202
|
## Extension Examples
|
231
203
|
|
232
|
-
|
204
|
+
Most missing features can easily be added to your application with a few lines
|
205
|
+
of code.
|
206
|
+
|
207
|
+
### Concurrency Protection
|
208
|
+
|
209
|
+
To prevent two threads from executing the same job simultaneously, consider
|
210
|
+
moving jobs to a different queue before executing them.
|
211
|
+
|
212
|
+
```ruby
|
213
|
+
class ApplicationJob < ActiveJob::Base
|
214
|
+
around_perform do |job, block|
|
215
|
+
if job.queue_name.start_with?('claimed')
|
216
|
+
raise "Job #{job.job_id} already claimed"
|
217
|
+
end
|
218
|
+
|
219
|
+
updated = Marj::Record
|
220
|
+
.where(job_id: job.job_id, queue_name: job.queue_name)
|
221
|
+
.update_all(
|
222
|
+
queue_name: "claimed-#{job.queue_name}",
|
223
|
+
scheduled_at: Time.now.utc
|
224
|
+
)
|
225
|
+
unless updated == 1
|
226
|
+
raise "Failed to claim job #{job.job_id}. #{updated} records updated"
|
227
|
+
end
|
228
|
+
|
229
|
+
begin
|
230
|
+
block.call
|
231
|
+
rescue StandardError
|
232
|
+
Marj::Record
|
233
|
+
.where(job_id: job.job_id, queue_name: "claimed-#{job.queue_name}")
|
234
|
+
.update_all(queue_name: job.queue_name, scheduled_at: job.scheduled_at)
|
235
|
+
raise
|
236
|
+
end
|
237
|
+
end
|
238
|
+
end
|
239
|
+
```
|
240
|
+
|
241
|
+
### Job Timeouts
|
233
242
|
|
234
243
|
```ruby
|
235
244
|
class ApplicationJob < ActiveJob::Base
|
@@ -249,7 +258,7 @@ class ApplicationJob < ActiveJob::Base
|
|
249
258
|
end
|
250
259
|
```
|
251
260
|
|
252
|
-
###
|
261
|
+
### Error Storage
|
253
262
|
|
254
263
|
```ruby
|
255
264
|
class AddLastErrorToJobs < ActiveRecord::Migration[7.1]
|
@@ -290,6 +299,83 @@ class ApplicationJob < ActiveJob::Base
|
|
290
299
|
end
|
291
300
|
```
|
292
301
|
|
302
|
+
### Discarded Job Retention
|
303
|
+
|
304
|
+
By default, discarded jobs are deleted from the database. If desired, this
|
305
|
+
behavior can be overridden, for instance to move jobs to a differnt queue:
|
306
|
+
|
307
|
+
```ruby
|
308
|
+
ActiveJob::Base.queue_adapter = MarjAdapter.new(
|
309
|
+
discard: proc { _1.enqueue(queue: 'discarded') }
|
310
|
+
)
|
311
|
+
|
312
|
+
# To retrieve a discarded job, query the discarded queue:
|
313
|
+
job = Marj.query(:first, queue_name: 'discarded')
|
314
|
+
|
315
|
+
# To delete, use one of the following:
|
316
|
+
Marj.delete(job)
|
317
|
+
job.delete
|
318
|
+
```
|
319
|
+
|
320
|
+
### Observability
|
321
|
+
|
322
|
+
For instance with Prometheus metrics:
|
323
|
+
|
324
|
+
```ruby
|
325
|
+
class ApplicationJob < ActiveJob::Base
|
326
|
+
counter = Prometheus::Client::Counter.new(
|
327
|
+
:job_events_total, docstring: '...', labels: [:job, :event]
|
328
|
+
)
|
329
|
+
|
330
|
+
around_enqueue do |job, block|
|
331
|
+
counter.increment(labels: { job: job.class.name, event: 'before_enqueue' })
|
332
|
+
block.call
|
333
|
+
counter.increment(labels: { job: job.class.name, event: 'after_enqueue' })
|
334
|
+
rescue Exception
|
335
|
+
counter.increment(labels: { job: job.class.name, event: 'enqueue_error' })
|
336
|
+
raise
|
337
|
+
end
|
338
|
+
|
339
|
+
around_perform do |job, block|
|
340
|
+
counter.increment(labels: { job: job.class.name, event: 'before_perform' })
|
341
|
+
block.call
|
342
|
+
counter.increment(labels: { job: job.class.name, event: 'after_perform' })
|
343
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
344
|
+
counter.increment(labels: { job: job.class.name, event: 'perform_error' })
|
345
|
+
raise
|
346
|
+
end
|
347
|
+
|
348
|
+
after_discard do |job|
|
349
|
+
counter.increment(labels: { job: job.class.name, event: 'after_discard' })
|
350
|
+
end
|
351
|
+
end
|
352
|
+
```
|
353
|
+
|
354
|
+
### Multi-DB Support
|
355
|
+
|
356
|
+
It is possible to create a custom record class in order to, for instance,
|
357
|
+
use a different class or table name, or write jobs to multiple
|
358
|
+
databases/tables within a single application.
|
359
|
+
|
360
|
+
Assuming you have a jobs tabled named `my_jobs`:
|
361
|
+
|
362
|
+
```ruby
|
363
|
+
class MyRecord < Marj::Record
|
364
|
+
self.table_name = 'my_jobs'
|
365
|
+
end
|
366
|
+
|
367
|
+
class MyJob < ActiveJob::Base
|
368
|
+
include Marj
|
369
|
+
|
370
|
+
self.queue_adapter = MarjAdapter.new('MyRecord')
|
371
|
+
|
372
|
+
def perform; end
|
373
|
+
end
|
374
|
+
|
375
|
+
MyJob.perform_later('oh, hi')
|
376
|
+
MyJob.query(:due, :first).perform_now
|
377
|
+
```
|
378
|
+
|
293
379
|
## ActiveJob Cheatsheet
|
294
380
|
|
295
381
|
For more information on ActiveJob, see:
|
@@ -0,0 +1,119 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative '../marj_adapter'
|
4
|
+
require 'mission_control/jobs'
|
5
|
+
require 'mission_control/jobs/adapter'
|
6
|
+
|
7
|
+
# :nocov:
|
8
|
+
module Marj
|
9
|
+
module MissionControl
|
10
|
+
include ::MissionControl::Jobs::Adapter
|
11
|
+
|
12
|
+
def queues
|
13
|
+
record_class.group(:queue_name).count(:queue_name).map { |k, v| { name: k, size: v, active: true } }
|
14
|
+
end
|
15
|
+
|
16
|
+
def queue_size(queue_name)
|
17
|
+
record_class.where(queue_name: queue_name).count
|
18
|
+
end
|
19
|
+
|
20
|
+
def clear_queue(queue_name)
|
21
|
+
Marj::Record.where(queue_name: queue_name).delete_all
|
22
|
+
end
|
23
|
+
|
24
|
+
def pause_queue(_queue_name)
|
25
|
+
raise 'not supported: pause queue'
|
26
|
+
end
|
27
|
+
|
28
|
+
def resume_queue(_queue_name)
|
29
|
+
raise 'not supported: resume queue'
|
30
|
+
end
|
31
|
+
|
32
|
+
def queue_paused?(_queue_name)
|
33
|
+
false
|
34
|
+
end
|
35
|
+
|
36
|
+
def supported_statuses
|
37
|
+
%i[pending failed scheduled]
|
38
|
+
end
|
39
|
+
|
40
|
+
def supported_filters(_jobs_relation)
|
41
|
+
%i[queue_name job_class_name]
|
42
|
+
end
|
43
|
+
|
44
|
+
def exposes_workers?
|
45
|
+
false
|
46
|
+
end
|
47
|
+
|
48
|
+
def workers
|
49
|
+
raise 'not supported: workers'
|
50
|
+
end
|
51
|
+
|
52
|
+
def find_worker(_worker_id)
|
53
|
+
raise 'not supported: find workers'
|
54
|
+
end
|
55
|
+
|
56
|
+
def jobs_count(jobs_relation)
|
57
|
+
ar_relation(jobs_relation).count
|
58
|
+
end
|
59
|
+
|
60
|
+
def fetch_jobs(jobs_relation)
|
61
|
+
ar_relation(jobs_relation).each_with_index.map { |record, index| to_job(record, jobs_relation, index) }
|
62
|
+
end
|
63
|
+
|
64
|
+
def retry_all_jobs(jobs_relation)
|
65
|
+
ar_relation(jobs_relation).map { |record| record.to_job.perform_now }
|
66
|
+
end
|
67
|
+
|
68
|
+
def retry_job(job, _jobs_relation)
|
69
|
+
Marj::Record.find(job.job_id).to_job.perform_now
|
70
|
+
end
|
71
|
+
|
72
|
+
def discard_all_jobs(jobs_relation)
|
73
|
+
ar_relation(jobs_relation).map { |record| discard(record.to_job) }
|
74
|
+
end
|
75
|
+
|
76
|
+
def discard_job(job, _jobs_relation)
|
77
|
+
discard(Marj::Record.find(job.job_id).to_job)
|
78
|
+
end
|
79
|
+
|
80
|
+
def find_job(job_id, jobs_relation)
|
81
|
+
to_job(record_class.find_by(job_id: job_id), jobs_relation)
|
82
|
+
end
|
83
|
+
|
84
|
+
private
|
85
|
+
|
86
|
+
def ar_relation(jobs_relation)
|
87
|
+
relation = Marj::Record.all.offset(jobs_relation.offset_value).limit(jobs_relation.limit_value)
|
88
|
+
relation = relation.where.not(executions: 0) if jobs_relation.status == :failed
|
89
|
+
relation = relation.where.not(scheduled_at: nil) if jobs_relation.status == :scheduled
|
90
|
+
relation = relation.where(job_class: jobs_relation.job_class_name) if jobs_relation.job_class_name
|
91
|
+
relation = relation.where(queue_name: jobs_relation.queue_name) if jobs_relation.queue_name
|
92
|
+
relation
|
93
|
+
end
|
94
|
+
|
95
|
+
def to_job(record, jobs_relation, index = 0)
|
96
|
+
return nil unless record
|
97
|
+
|
98
|
+
job = record.to_job
|
99
|
+
job_data = job.serialize
|
100
|
+
ActiveJob::JobProxy.new(job_data).tap do |proxy|
|
101
|
+
if job.executions.positive?
|
102
|
+
proxy.last_execution_error = ActiveJob::ExecutionError.new(
|
103
|
+
error_class: Exception, message: 'unknown', backtrace: []
|
104
|
+
)
|
105
|
+
proxy.failed_at = job.enqueued_at
|
106
|
+
proxy.status = :failed
|
107
|
+
elsif job.scheduled_at
|
108
|
+
proxy.status = :scheduled
|
109
|
+
else
|
110
|
+
proxy.status = :pending
|
111
|
+
end
|
112
|
+
proxy.raw_data = job_data
|
113
|
+
proxy.position = jobs_relation.offset_value + index
|
114
|
+
proxy.arguments = job.arguments # For some reason MissionControl sets the arguments to the entire job data
|
115
|
+
end
|
116
|
+
end
|
117
|
+
end
|
118
|
+
end
|
119
|
+
# :nocov:
|
data/lib/marj/record.rb
CHANGED
@@ -6,7 +6,7 @@ require 'active_record'
|
|
6
6
|
module Marj
|
7
7
|
# The default +ActiveRecord+ class.
|
8
8
|
class Record < ActiveRecord::Base
|
9
|
-
self.table_name =
|
9
|
+
self.table_name = :jobs
|
10
10
|
|
11
11
|
# Order by +enqueued_at+ rather than +job_id+ (the default).
|
12
12
|
self.implicit_order_column = 'enqueued_at'
|
@@ -50,9 +50,47 @@ module Marj
|
|
50
50
|
# discarded.
|
51
51
|
#
|
52
52
|
# @return [ActiveJob::Base]
|
53
|
-
def
|
54
|
-
|
53
|
+
def to_job
|
54
|
+
# See register_callbacks for details on how callbacks are used.
|
55
|
+
job = job_class.new.tap { register_callbacks(_1) }
|
56
|
+
|
57
|
+
# ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
|
58
|
+
# deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
|
59
|
+
job_data = attributes.merge('arguments' => JSON.parse(read_attribute_before_type_cast(:arguments)))
|
60
|
+
|
61
|
+
# ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
|
62
|
+
job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
|
63
|
+
|
64
|
+
job.deserialize(job_data)
|
65
|
+
|
66
|
+
# ActiveJob deserializes arguments on demand when a job is performed. Until then they are empty. That's strange.
|
67
|
+
# Instead, deserialize them now. Also, clear `serialized_arguments` to prevent ActiveJob from overwriting changes
|
68
|
+
# to arguments when serializing later.
|
69
|
+
job.arguments = arguments
|
70
|
+
job.serialized_arguments = nil
|
71
|
+
|
72
|
+
job
|
73
|
+
end
|
74
|
+
|
75
|
+
# Registers callbacks for the given job which destroy this record when the job succeeds or is discarded.
|
76
|
+
#
|
77
|
+
# @param job [ActiveJob::Base]
|
78
|
+
# @return [ActiveJob::Base]
|
79
|
+
def register_callbacks(job)
|
80
|
+
raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
|
81
|
+
|
82
|
+
record = self
|
83
|
+
# We need to detect three cases:
|
84
|
+
# - If a job succeeds, after_perform will be called.
|
85
|
+
# - If a job fails and should be retried, enqueue will be called. This is handled by the queue adapter.
|
86
|
+
# - If a job exceeds its max attempts, after_discard will be called.
|
87
|
+
job.singleton_class.after_perform { |_j| job.queue_adapter.delete(job) }
|
88
|
+
job.singleton_class.after_discard { |_j, _exception| job.queue_adapter.discard(job, run_callbacks: false) }
|
89
|
+
job.singleton_class.instance_variable_set(:@record, record)
|
90
|
+
|
91
|
+
job
|
55
92
|
end
|
93
|
+
private :register_callbacks
|
56
94
|
|
57
95
|
class << self
|
58
96
|
# Returns an +ActiveRecord::Relation+ scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in
|
@@ -63,11 +101,16 @@ module Marj
|
|
63
101
|
where('scheduled_at IS NULL OR scheduled_at <= ?', Time.now.utc)
|
64
102
|
end
|
65
103
|
|
66
|
-
# Returns an +ActiveRecord::Relation+ scope for jobs ordered by
|
67
|
-
#
|
104
|
+
# Returns an +ActiveRecord::Relation+ scope for jobs ordered by due date.
|
105
|
+
#
|
106
|
+
# Jobs are ordered by the following criteria, in order:
|
107
|
+
# 1. past or null scheduled_at before future scheduled_at
|
108
|
+
# 2. ascending priority, nulls last
|
109
|
+
# 3. ascending scheduled_at, nulls last
|
110
|
+
# 4. ascending enqueued_at
|
68
111
|
#
|
69
112
|
# @return [ActiveRecord::Relation]
|
70
|
-
def
|
113
|
+
def by_due_date
|
71
114
|
order(
|
72
115
|
Arel.sql(<<~SQL.squish, Time.now.utc)
|
73
116
|
CASE WHEN scheduled_at IS NULL OR scheduled_at <= ? THEN 0 ELSE 1 END,
|
data/lib/marj.rb
CHANGED
@@ -1,164 +1,131 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require_relative 'marj_adapter'
|
4
|
-
require_relative 'marj/jobs_interface'
|
5
|
-
require_relative 'marj/relation'
|
6
4
|
|
7
5
|
# A minimal database-backed ActiveJob queueing backend.
|
8
6
|
#
|
9
|
-
# The {Marj} module provides
|
10
|
-
# +
|
11
|
-
#
|
7
|
+
# The {Marj} module provides the following methods:
|
8
|
+
# - +query+ - Queries enqueued jobs
|
9
|
+
# - +discard+ - Discards a job, by default by executing after_discard callbacks and delegating to delete
|
10
|
+
# - +delete+ - Deletes a job
|
11
|
+
#
|
12
|
+
# It is possible to call the above methods on the {Marj} module itself or on any class which includes it.
|
12
13
|
#
|
13
14
|
# Example usage:
|
14
|
-
# Marj.
|
15
|
-
# Marj.
|
16
|
-
# Marj.
|
17
|
-
#
|
18
|
-
#
|
19
|
-
#
|
20
|
-
#
|
21
|
-
# Marj.discard_all # Discards all jobs.
|
22
|
-
# Marj.discard # Discards the specified job.
|
15
|
+
# Marj.query(:first) # Returns the first job
|
16
|
+
# Marj.discard(job) # Discards the specified job
|
17
|
+
# Marj.delete(job) # Deletes the specified job
|
18
|
+
#
|
19
|
+
# class ApplicationJob < ActiveJob::Base
|
20
|
+
# include Marj
|
21
|
+
# end
|
23
22
|
#
|
24
|
-
#
|
25
|
-
#
|
23
|
+
# class SomeJob < ApplicationJob;
|
24
|
+
# def perform; end
|
25
|
+
# end
|
26
26
|
#
|
27
|
-
#
|
28
|
-
#
|
27
|
+
# job = ApplicationJob.query(:first) # Returns the first enqueued job
|
28
|
+
# job = SomeJob.query(:first) # Returns the first enqueued job with job_class SomeJob
|
29
|
+
# ApplicationJob.discard(job) # Discards the specified job
|
30
|
+
# ApplicationJob.delete(job) # Deletes the specified job
|
31
|
+
# job.discard # Discards the job
|
32
|
+
# job.delete # Deletes the job
|
29
33
|
#
|
30
34
|
# See https://github.com/nicholasdower/marj
|
31
35
|
module Marj
|
32
36
|
# The Marj version.
|
33
|
-
VERSION = '
|
37
|
+
VERSION = '6.0.0'
|
34
38
|
|
35
39
|
Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
|
36
40
|
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
#
|
44
|
-
# The name of the +ActiveRecord+ class. Defaults to +Marj::Record+.
|
45
|
-
# @return [Class, String]
|
46
|
-
|
47
|
-
attr_writer :record_class
|
48
|
-
|
49
|
-
def record_class
|
50
|
-
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
51
|
-
end
|
52
|
-
|
53
|
-
# @!attribute table_name
|
54
|
-
# The name of the database table. Defaults to +:jobs+.
|
55
|
-
# @return [Symbol, String]
|
56
|
-
attr_accessor :table_name
|
57
|
-
|
58
|
-
# Returns a {Marj::Relation} for all jobs in the order they should be executed.
|
41
|
+
# Provides the {query} and {discard} class methods.
|
42
|
+
module ClassMethods
|
43
|
+
# Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
|
44
|
+
# - Symbol arguments are treated as +ActiveRecord+ scopes.
|
45
|
+
# - If only a job ID is specified, the corresponding job is returned.
|
46
|
+
# - If +:limit+ is specified, the maximum number of jobs is limited.
|
47
|
+
# - If +:order+ is specified, the jobs are ordered by the given attribute.
|
59
48
|
#
|
60
|
-
#
|
61
|
-
|
62
|
-
|
49
|
+
# By default jobs are ordered by when they should be executed.
|
50
|
+
#
|
51
|
+
# Example usage:
|
52
|
+
# query # Returns all jobs
|
53
|
+
# query(:all) # Returns all jobs
|
54
|
+
# query(:due) # Returns jobs which are due to be executed
|
55
|
+
# query(:due, limit: 10) # Returns at most 10 jobs which are due to be executed
|
56
|
+
# query(job_class: Foo) # Returns all jobs with job_class Foo
|
57
|
+
# query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
|
58
|
+
# query(queue_name: 'foo') # Returns all jobs in the 'foo' queue
|
59
|
+
# query(job_id: '123') # Returns the job with job_id '123' or nil if no such job exists
|
60
|
+
# query('123') # Returns the job with job_id '123' or nil if no such job exists
|
61
|
+
def query(*args, **kwargs)
|
62
|
+
kwargs[:job_class] ||= self if self < ActiveJob::Base && name != 'ApplicationJob'
|
63
|
+
queue_adapter.query(*args, **kwargs)
|
63
64
|
end
|
64
65
|
|
65
66
|
# Discards the specified job.
|
66
67
|
#
|
67
|
-
# @return [
|
68
|
+
# @return [ActiveJob::Base] the discarded job
|
68
69
|
def discard(job)
|
69
|
-
|
70
|
+
queue_adapter.discard(job)
|
70
71
|
end
|
71
72
|
|
72
|
-
|
73
|
-
|
74
|
-
# Creates a job instance for the given record which will update the database when successfully executed, enqueued or
|
75
|
-
# discarded.
|
73
|
+
# Deletes the record associated with the specified job.
|
76
74
|
#
|
77
|
-
# @
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
# ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
|
84
|
-
# deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
|
85
|
-
job_data = record.attributes.merge('arguments' => JSON.parse(record.read_attribute_before_type_cast(:arguments)))
|
86
|
-
|
87
|
-
# ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
|
88
|
-
job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
|
75
|
+
# @return [ActiveJob::Base] the deleted job
|
76
|
+
def delete(job)
|
77
|
+
queue_adapter.delete(job)
|
78
|
+
end
|
79
|
+
end
|
89
80
|
|
90
|
-
|
81
|
+
# (see ClassMethods#query)
|
82
|
+
def self.query(*args, **kwargs)
|
83
|
+
queue_adapter.query(*args, **kwargs)
|
84
|
+
end
|
91
85
|
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
job.serialized_arguments = nil
|
86
|
+
# (see ClassMethods#discard)
|
87
|
+
def self.discard(job)
|
88
|
+
queue_adapter.discard(job)
|
89
|
+
end
|
97
90
|
|
98
|
-
|
99
|
-
|
91
|
+
# (see ClassMethods#delete)
|
92
|
+
def self.delete(job)
|
93
|
+
queue_adapter.delete(job)
|
94
|
+
end
|
100
95
|
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
def register_callbacks(job, record)
|
108
|
-
raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
|
96
|
+
# Deletes this job.
|
97
|
+
#
|
98
|
+
# @return [ActiveJob::Base] this job
|
99
|
+
def discard
|
100
|
+
self.class.queue_adapter.discard(self)
|
101
|
+
end
|
109
102
|
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
job.singleton_class.instance_variable_set(:@record, record)
|
103
|
+
# Deletes the record associated with this job.
|
104
|
+
#
|
105
|
+
# @return [ActiveJob::Base] this job
|
106
|
+
def delete
|
107
|
+
self.class.queue_adapter.delete(self)
|
108
|
+
end
|
117
109
|
|
118
|
-
|
119
|
-
|
110
|
+
def self.included(clazz)
|
111
|
+
clazz.extend(ClassMethods)
|
112
|
+
end
|
113
|
+
private_class_method :included
|
120
114
|
|
121
|
-
|
122
|
-
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
# @return [ActiveJob::Base] the enqueued job
|
127
|
-
def enqueue(job, record_class, time = nil)
|
128
|
-
job.scheduled_at = time
|
129
|
-
# Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
|
130
|
-
serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
|
115
|
+
def self.queue_adapter
|
116
|
+
ActiveJob::Base.queue_adapter
|
117
|
+
end
|
118
|
+
private_class_method :queue_adapter
|
119
|
+
end
|
131
120
|
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
# This job instance has already been associated with a database row.
|
141
|
-
if record_class.exists?(job_id: job.job_id)
|
142
|
-
# The database row still exists, we simply need to update it.
|
143
|
-
existing_record.update!(serialized)
|
144
|
-
else
|
145
|
-
# Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
|
146
|
-
# want to register the new instance because someone might still have a reference to the existing one.
|
147
|
-
record_class.create!(serialized)
|
148
|
-
existing_record.reload
|
149
|
-
end
|
150
|
-
else
|
151
|
-
# This job instance has not been associated with a database row.
|
152
|
-
if (new_record = record_class.find_by(job_id: job.job_id))
|
153
|
-
# The database row already exists. Update it.
|
154
|
-
new_record.update!(serialized)
|
155
|
-
else
|
156
|
-
# The database row does not exist. Create it.
|
157
|
-
new_record = record_class.create!(serialized)
|
158
|
-
end
|
159
|
-
register_callbacks(job, new_record)
|
160
|
-
end
|
161
|
-
job
|
162
|
-
end
|
121
|
+
# :nocov:
|
122
|
+
if defined?(Rails)
|
123
|
+
begin
|
124
|
+
require 'mission_control/jobs'
|
125
|
+
require_relative 'marj/mission_control'
|
126
|
+
MarjAdapter.include(Marj::MissionControl)
|
127
|
+
rescue LoadError
|
128
|
+
# ignore
|
163
129
|
end
|
164
130
|
end
|
131
|
+
# :nocov:
|
data/lib/marj_adapter.rb
CHANGED
@@ -2,13 +2,26 @@
|
|
2
2
|
|
3
3
|
# ActiveJob queue adapter for Marj.
|
4
4
|
#
|
5
|
+
# In addition to the standard +ActiveJob+ queue adapter API, this adapter provides:
|
6
|
+
# - A +query+ method which can be used to query enqueued jobs
|
7
|
+
# - A +discard+ method which can be used to discard enqueued jobs.
|
8
|
+
# - A +delete+ method which can be used to delete enqueued jobs.
|
9
|
+
#
|
10
|
+
# Although it is possible to access the adapter directly in order to query, discard or delete, it is recommended to use
|
11
|
+
# the {Marj} module.
|
12
|
+
#
|
5
13
|
# See https://github.com/nicholasdower/marj
|
6
14
|
class MarjAdapter
|
15
|
+
JOB_ID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.freeze
|
16
|
+
private_constant :JOB_ID_REGEX
|
17
|
+
|
7
18
|
# Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ class.
|
8
19
|
#
|
9
|
-
# @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use to
|
10
|
-
|
20
|
+
# @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use, defaults to +Marj::Record+
|
21
|
+
# @param discard [Proc] the proc to use to discard jobs, defaults to delegating to {delete}
|
22
|
+
def initialize(record_class: 'Marj::Record', discard: proc { |job| delete(job) })
|
11
23
|
@record_class = record_class
|
24
|
+
@discard_proc = discard
|
12
25
|
end
|
13
26
|
|
14
27
|
# Enqueue a job for immediate execution.
|
@@ -16,7 +29,7 @@ class MarjAdapter
|
|
16
29
|
# @param job [ActiveJob::Base] the job to enqueue
|
17
30
|
# @return [ActiveJob::Base] the enqueued job
|
18
31
|
def enqueue(job)
|
19
|
-
|
32
|
+
enqueue_at(job)
|
20
33
|
end
|
21
34
|
|
22
35
|
# Enqueue a job for execution at the specified time.
|
@@ -24,13 +37,146 @@ class MarjAdapter
|
|
24
37
|
# @param job [ActiveJob::Base] the job to enqueue
|
25
38
|
# @param timestamp [Numeric, NilClass] optional number of seconds since Unix epoch at which to execute the job
|
26
39
|
# @return [ActiveJob::Base] the enqueued job
|
27
|
-
def enqueue_at(job, timestamp)
|
28
|
-
|
40
|
+
def enqueue_at(job, timestamp = nil)
|
41
|
+
job.scheduled_at = timestamp ? Time.at(timestamp).utc : nil
|
42
|
+
|
43
|
+
# Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
|
44
|
+
serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
|
45
|
+
|
46
|
+
# Serialize sets locale to I18n.locale.to_s and enqueued_at to Time.now.utc.iso8601(9).
|
47
|
+
# Update the job to reflect what is being enqueued.
|
48
|
+
job.locale = serialized[:locale]
|
49
|
+
job.enqueued_at = Time.iso8601(serialized[:enqueued_at]).utc
|
50
|
+
|
51
|
+
# When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
|
52
|
+
# are registered on the job instance so that when the job is executed, the database record is deleted or updated
|
53
|
+
# (depending on the result).
|
54
|
+
#
|
55
|
+
# We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
|
56
|
+
# singleton class. This holds a reference to the record. This ensures that if execute is called on a record
|
57
|
+
# instance, any updates to the database are reflected on that record instance.
|
58
|
+
if (existing_record = job.singleton_class.instance_variable_get(:@record))
|
59
|
+
# This job instance has already been associated with a database row.
|
60
|
+
if record_class.exists?(job_id: job.job_id)
|
61
|
+
# The database row still exists, we simply need to update it.
|
62
|
+
existing_record.update!(serialized)
|
63
|
+
else
|
64
|
+
# Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
|
65
|
+
# want to register the new instance because someone might still have a reference to the existing one.
|
66
|
+
record_class.create!(serialized)
|
67
|
+
existing_record.reload
|
68
|
+
end
|
69
|
+
else
|
70
|
+
# This job instance has not been associated with a database row.
|
71
|
+
if (new_record = record_class.find_by(job_id: job.job_id))
|
72
|
+
# The database row already exists. Update it.
|
73
|
+
new_record.update!(serialized)
|
74
|
+
else
|
75
|
+
# The database row does not exist. Create it.
|
76
|
+
new_record = record_class.create!(serialized)
|
77
|
+
end
|
78
|
+
new_record.send(:register_callbacks, job)
|
79
|
+
end
|
80
|
+
job
|
81
|
+
end
|
82
|
+
|
83
|
+
# Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
|
84
|
+
# - Symbol arguments are treated as +ActiveRecord+ scopes.
|
85
|
+
# - If only a job ID is specified, the corresponding job is returned.
|
86
|
+
# - If +:limit+ is specified, the maximum number of jobs is limited.
|
87
|
+
# - If +:order+ is specified, the jobs are ordered by the given attribute.
|
88
|
+
#
|
89
|
+
# By default jobs are ordered by when they should be executed.
|
90
|
+
#
|
91
|
+
# Example usage:
|
92
|
+
# query # Returns all jobs
|
93
|
+
# query(:all) # Returns all jobs
|
94
|
+
# query(:due) # Returns jobs which are due to be executed
|
95
|
+
# query(:due, limit: 10) # Returns at most 10 jobs which are due to be executed
|
96
|
+
# query(job_class: Foo) # Returns all jobs with job_class Foo
|
97
|
+
# query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
|
98
|
+
# query(queue_name: 'foo') # Returns all jobs in the 'foo' queue
|
99
|
+
# query(job_id: '123') # Returns the job with job_id '123' or nil if no such job exists
|
100
|
+
# query('123') # Returns the job with job_id '123' or nil if no such job exists
|
101
|
+
def query(*args, **kwargs)
|
102
|
+
args, kwargs = args.dup, kwargs.dup.symbolize_keys
|
103
|
+
kwargs = kwargs.merge(job_id: kwargs.delete(:id)) if kwargs.key?(:id)
|
104
|
+
kwargs[:job_id] = args.shift if args.size == 1 && args.first.is_a?(String) && args.first.match(JOB_ID_REGEX)
|
105
|
+
|
106
|
+
if args.empty? && kwargs.size == 1 && kwargs.key?(:job_id)
|
107
|
+
return record_class.find_by(job_id: kwargs[:job_id])&.to_job
|
108
|
+
end
|
109
|
+
|
110
|
+
symbol_args, args = args.partition { _1.is_a?(Symbol) }
|
111
|
+
symbol_args.delete(:all)
|
112
|
+
limit = kwargs.delete(:limit)
|
113
|
+
relation = record_class.all
|
114
|
+
relation = relation.order(kwargs.delete(:order)) if kwargs.key?(:order)
|
115
|
+
relation = relation.where(*args, **kwargs) if args.any? || kwargs.any?
|
116
|
+
relation = relation.limit(limit) if limit
|
117
|
+
relation = relation.send(symbol_args.shift) while symbol_args.any?
|
118
|
+
relation = relation.by_due_date if relation.is_a?(ActiveRecord::Relation) && relation.order_values.empty?
|
119
|
+
|
120
|
+
if relation.is_a?(Enumerable)
|
121
|
+
relation.map(&:to_job)
|
122
|
+
elsif relation.is_a?(record_class)
|
123
|
+
relation.to_job
|
124
|
+
else
|
125
|
+
relation
|
126
|
+
end
|
127
|
+
end
|
128
|
+
|
129
|
+
# Discards the specified job.
|
130
|
+
#
|
131
|
+
# @param job [ActiveJob::Base] the job being discarded
|
132
|
+
# @param run_callbacks [Boolean] whether to run the +after_discard+ callbacks
|
133
|
+
# @return [ActiveJob::Base] the discarded job
|
134
|
+
def discard(job, run_callbacks: true)
|
135
|
+
job.tap do
|
136
|
+
@discard_proc.call(job)
|
137
|
+
run_after_discard_callbacks(job) if run_callbacks
|
138
|
+
end
|
139
|
+
end
|
140
|
+
|
141
|
+
# Deletes the record associated with the specified job.
|
142
|
+
#
|
143
|
+
# @return [ActiveJob::Base] the deleted job
|
144
|
+
def delete(job)
|
145
|
+
job.tap { destroy_record(job) }
|
29
146
|
end
|
30
147
|
|
31
148
|
private
|
32
149
|
|
150
|
+
# Returns the +ActiveRecord+ class to use to store jobs.
|
151
|
+
#
|
152
|
+
# @return [Class] the +ActiveRecord+ class
|
33
153
|
def record_class
|
34
154
|
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
35
155
|
end
|
156
|
+
|
157
|
+
# Destroys the record associated with the given job if it exists.
|
158
|
+
#
|
159
|
+
# @return [ActiveRecord::Base, NilClass] the destroyed record or +nil+ if no such record exists
|
160
|
+
def destroy_record(job)
|
161
|
+
record = job.singleton_class.instance_variable_get(:@record)
|
162
|
+
record ||= Marj::Record.find_by(job_id: job.job_id)&.tap { _1.send(:register_callbacks, job) }
|
163
|
+
record&.destroy
|
164
|
+
end
|
165
|
+
|
166
|
+
# Invokes the specified job's +after_discard+ callbacks.
|
167
|
+
#
|
168
|
+
# @param job [ActiveJob::Base] the job being discarded
|
169
|
+
# @return [NilClass] the given job
|
170
|
+
def run_after_discard_callbacks(job)
|
171
|
+
# Copied from ActiveJob::Exceptions#run_after_discard_procs
|
172
|
+
exceptions = []
|
173
|
+
job.after_discard_procs.each do |blk|
|
174
|
+
instance_exec(job, nil, &blk)
|
175
|
+
rescue StandardError => e
|
176
|
+
exceptions << e
|
177
|
+
end
|
178
|
+
raise exceptions.last if exceptions.any?
|
179
|
+
|
180
|
+
nil
|
181
|
+
end
|
36
182
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: marj
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 6.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Nick Dower
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-
|
11
|
+
date: 2024-02-05 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -48,17 +48,16 @@ files:
|
|
48
48
|
- LICENSE.txt
|
49
49
|
- README.md
|
50
50
|
- lib/marj.rb
|
51
|
-
- lib/marj/
|
51
|
+
- lib/marj/mission_control.rb
|
52
52
|
- lib/marj/record.rb
|
53
|
-
- lib/marj/relation.rb
|
54
53
|
- lib/marj_adapter.rb
|
55
54
|
homepage: https://github.com/nicholasdower/marj
|
56
55
|
licenses:
|
57
56
|
- MIT
|
58
57
|
metadata:
|
59
58
|
bug_tracker_uri: https://github.com/nicholasdower/marj/issues
|
60
|
-
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/
|
61
|
-
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/
|
59
|
+
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v6.0.0
|
60
|
+
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v6.0.0
|
62
61
|
homepage_uri: https://github.com/nicholasdower/marj
|
63
62
|
rubygems_mfa_required: 'true'
|
64
63
|
source_code_uri: https://github.com/nicholasdower/marj
|
data/lib/marj/jobs_interface.rb
DELETED
@@ -1,113 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
module Marj
|
4
|
-
# The interface provided by {Marj} and {Marj::Relation}.
|
5
|
-
#
|
6
|
-
# To create a custom jobs interface, for example for all job classes in your application:
|
7
|
-
# class ApplicationJob < ActiveJob::Base
|
8
|
-
# extend Marj::JobsInterface
|
9
|
-
#
|
10
|
-
# def self.all
|
11
|
-
# Marj::Relation.new(self == ApplicationJob ? Marj::Record.ordered : Marj::Record.where(job_class: self))
|
12
|
-
# end
|
13
|
-
# end
|
14
|
-
#
|
15
|
-
# class SomeJob < ApplicationJob
|
16
|
-
# def perform(msg)
|
17
|
-
# puts msg
|
18
|
-
# end
|
19
|
-
# end
|
20
|
-
#
|
21
|
-
# This will allow you to query jobs via the +ApplicationJob+ class:
|
22
|
-
# ApplicationJob.next # Returns the next job of any type
|
23
|
-
#
|
24
|
-
# Or to query jobs via a specific job class:
|
25
|
-
# SomeJob.next # Returns the next SomeJob
|
26
|
-
#
|
27
|
-
# Alternatively, to create a jobs interface for a single job class:
|
28
|
-
# class SomeJob < ActiveJob::Base
|
29
|
-
# extend Marj::JobsInterface
|
30
|
-
#
|
31
|
-
# def self.all
|
32
|
-
# Marj::Relation.new(Marj::Record.where(job_class: self).ordered)
|
33
|
-
# end
|
34
|
-
# end
|
35
|
-
module JobsInterface
|
36
|
-
def self.included(clazz)
|
37
|
-
return if clazz == Marj::Relation
|
38
|
-
|
39
|
-
clazz.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
|
40
|
-
end
|
41
|
-
private_class_method :included
|
42
|
-
|
43
|
-
def self.extended(clazz)
|
44
|
-
clazz.singleton_class.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
|
45
|
-
end
|
46
|
-
private_class_method :extended
|
47
|
-
|
48
|
-
# Returns a {Marj::Relation} for jobs in the specified queue(s).
|
49
|
-
#
|
50
|
-
# @param queue [String, Symbol] the queue to query
|
51
|
-
# @param queues [Array<String>, Array<Symbol>] more queues to query
|
52
|
-
# @return [Marj::Relation]
|
53
|
-
def queue(queue, *queues)
|
54
|
-
Marj::Relation.new(all.where(queue_name: queues.dup.unshift(queue)))
|
55
|
-
end
|
56
|
-
|
57
|
-
# Returns the next job or the next N jobs if +limit+ is specified. If no jobs exist, returns +nil+.
|
58
|
-
#
|
59
|
-
# @param limit [Integer, NilClass]
|
60
|
-
# @return [ActiveJob::Base, NilClass]
|
61
|
-
def next(limit = nil)
|
62
|
-
all.first(limit)&.then { _1.is_a?(Array) ? _1.map(&:as_job) : _1.as_job }
|
63
|
-
end
|
64
|
-
|
65
|
-
# Returns a count of jobs, optionally either matching the specified column name criteria or where the specified
|
66
|
-
# block returns +true+.
|
67
|
-
#
|
68
|
-
# @param column_name [String, Symbol, NilClass]
|
69
|
-
# @param block [Proc, NilClass]
|
70
|
-
# @return [Integer]
|
71
|
-
def count(column_name = nil, &block)
|
72
|
-
block_given? ? all.count(column_name) { |r| block.call(r.as_job) } : all.count(column_name)
|
73
|
-
end
|
74
|
-
|
75
|
-
# Returns a {Marj::Relation} for jobs matching the specified criteria.
|
76
|
-
#
|
77
|
-
# @param args [Array]
|
78
|
-
# @return [Marj::Relation]
|
79
|
-
def where(*args)
|
80
|
-
Marj::Relation.new(all.where(*args))
|
81
|
-
end
|
82
|
-
|
83
|
-
# Returns a {Marj::Relation} for enqueued jobs with a +scheduled_at+ that is either +null+ or in the past.
|
84
|
-
#
|
85
|
-
# @return [Marj::Relation]
|
86
|
-
def due
|
87
|
-
Marj::Relation.new(all.due)
|
88
|
-
end
|
89
|
-
|
90
|
-
# Calls +perform_now+ on each job.
|
91
|
-
#
|
92
|
-
# @param batch_size [Integer, NilClass] the number of jobs to fetch at a time, or +nil+ to fetch all jobs at once
|
93
|
-
# @return [Array] the results returned by each job
|
94
|
-
def perform_all(batch_size: nil)
|
95
|
-
if batch_size
|
96
|
-
[].tap do |results|
|
97
|
-
while (jobs = all.limit(batch_size).map(&:as_job)).any?
|
98
|
-
results.concat(jobs.map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } })
|
99
|
-
end
|
100
|
-
end
|
101
|
-
else
|
102
|
-
all.map(&:as_job).map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } }
|
103
|
-
end
|
104
|
-
end
|
105
|
-
|
106
|
-
# Discards all jobs.
|
107
|
-
#
|
108
|
-
# @return [Numeric] the number of discarded jobs
|
109
|
-
def discard_all
|
110
|
-
all.delete_all
|
111
|
-
end
|
112
|
-
end
|
113
|
-
end
|
data/lib/marj/relation.rb
DELETED
@@ -1,72 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require_relative 'jobs_interface'
|
4
|
-
|
5
|
-
module Marj
|
6
|
-
# Returned by {Marj::JobsInterface} query methods to enable chaining and +Enumerable+ methods.
|
7
|
-
class Relation
|
8
|
-
include Enumerable
|
9
|
-
include Marj::JobsInterface
|
10
|
-
|
11
|
-
attr_reader :all
|
12
|
-
private :all
|
13
|
-
|
14
|
-
# Returns a {Marj::Relation} which wraps the specified +ActiveRecord+ relation.
|
15
|
-
def initialize(ar_relation)
|
16
|
-
@all = ar_relation
|
17
|
-
end
|
18
|
-
|
19
|
-
# Yields each job in this relation.
|
20
|
-
#
|
21
|
-
# @param block [Proc]
|
22
|
-
# @return [Array] the jobs in this relation
|
23
|
-
def each(&block)
|
24
|
-
all.map(&:as_job).each(&block)
|
25
|
-
end
|
26
|
-
|
27
|
-
# Provides +pretty_inspect+ output containing arrays of jobs rather than arrays of records, similar to the output
|
28
|
-
# produced when calling +pretty_inspect+ on +ActiveRecord::Relation+.
|
29
|
-
#
|
30
|
-
# Instead of the default +pretty_inspect+ output:
|
31
|
-
# > Marj.all
|
32
|
-
# =>
|
33
|
-
# #<Marj::Relation:0x000000012728bd88
|
34
|
-
# @ar_relation=
|
35
|
-
# [#<Marj::Record:0x0000000126c42080
|
36
|
-
# job_id: "1382cb98-c518-46ca-a0cc-d831e11a0714",
|
37
|
-
# job_class: TestJob,
|
38
|
-
# arguments: ["foo"],
|
39
|
-
# queue_name: "default",
|
40
|
-
# priority: nil,
|
41
|
-
# executions: 0,
|
42
|
-
# exception_executions: {},
|
43
|
-
# enqueued_at: 2024-01-25 15:31:06.115773 UTC,
|
44
|
-
# scheduled_at: nil,
|
45
|
-
# locale: "en",
|
46
|
-
# timezone: "UTC">]>
|
47
|
-
#
|
48
|
-
# Produces:
|
49
|
-
# > Marj.all
|
50
|
-
# =>
|
51
|
-
# [#<TestJob:0x000000010b63cef8
|
52
|
-
# @_scheduled_at_time=nil,
|
53
|
-
# @arguments=[],
|
54
|
-
# @enqueued_at=2024-01-25 15:31:06 UTC,
|
55
|
-
# @exception_executions={},
|
56
|
-
# @executions=0,
|
57
|
-
# @job_id="1382cb98-c518-46ca-a0cc-d831e11a0714",
|
58
|
-
# @locale="en",
|
59
|
-
# @priority=nil,
|
60
|
-
# @provider_job_id=nil,
|
61
|
-
# @queue_name="default",
|
62
|
-
# @scheduled_at=nil,
|
63
|
-
# @serialized_arguments=["foo"],
|
64
|
-
# @timezone="UTC">]
|
65
|
-
#
|
66
|
-
# @param pp [PP]
|
67
|
-
# @return [NilClass]
|
68
|
-
def pretty_print(pp)
|
69
|
-
pp.pp(to_a)
|
70
|
-
end
|
71
|
-
end
|
72
|
-
end
|