marj 4.1.0 → 6.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +219 -133
- data/lib/marj/mission_control.rb +119 -0
- data/lib/marj/record.rb +49 -6
- data/lib/marj.rb +95 -128
- data/lib/marj_adapter.rb +151 -5
- metadata +5 -6
- data/lib/marj/jobs_interface.rb +0 -113
- data/lib/marj/relation.rb +0 -72
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6d0039931eb3f0bd43ce090b0b581d6544b26fda022729f6371f4c8ac393076e
|
4
|
+
data.tar.gz: 6abb66afe0bc56064be4ecee3f3e59e78adcf10afb83654529387b785f817335
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: ec25e50e772072eb6d13ebcb04a67e8ee4602782b62edab07e9521817993f6d92bac376bff90a6aeca1c724fa61dc4ba04ac1fe8ce9659139d2dcd98c397bce4
|
7
|
+
data.tar.gz: f78b5ba741202818a5374c95fddcb6258b091d4feba4b2de1d1f9de8d1422b99348bf5e354619d215b32b9b7f8d4055ef95d271055cf489e4049b06b40d3463c
|
data/README.md
CHANGED
@@ -4,42 +4,89 @@ A minimal database-backed ActiveJob queueing backend.
|
|
4
4
|
|
5
5
|
## Quick Links
|
6
6
|
|
7
|
-
API docs: https://gemdocs.org/gems/marj/
|
7
|
+
API docs: https://gemdocs.org/gems/marj/6.0.0/ <br>
|
8
8
|
RubyGems: https://rubygems.org/gems/marj <br>
|
9
9
|
Changelog: https://github.com/nicholasdower/marj/releases <br>
|
10
10
|
Issues: https://github.com/nicholasdower/marj/issues <br>
|
11
11
|
Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
|
12
12
|
|
13
|
+
## Motivation
|
14
|
+
|
15
|
+
There are already several great database-backed ActiveJob queueing backends:
|
16
|
+
- [Delayed::Job](https://github.com/collectiveidea/delayed_job)
|
17
|
+
- [Que](https://github.com/que-rb/que)
|
18
|
+
- [GoodJob](https://github.com/bensheldon/good_job)
|
19
|
+
- [Solid Queue](https://github.com/basecamp/solid_queue)
|
20
|
+
|
21
|
+
Any of these which support your RDBMS is likely to work well for you. But you
|
22
|
+
may find them to be more featureful and complex than what you require.
|
23
|
+
|
24
|
+
Marj aims to be a minimal alternative.
|
25
|
+
|
26
|
+
## Goal
|
27
|
+
|
28
|
+
To be the database-backend ActiveJob queueing backend with:
|
29
|
+
- The simplest setup
|
30
|
+
- The fewest configuration options
|
31
|
+
- The fewest features
|
32
|
+
- The fewest backend-specific APIs
|
33
|
+
- The fewest lines of code
|
34
|
+
|
13
35
|
## Features
|
14
36
|
|
37
|
+
Marj supports and has been tested with MySQL, PostgreSQL and SQLite.
|
38
|
+
|
39
|
+
It provides the following features:
|
15
40
|
- Enqueued jobs are written to the database.
|
16
|
-
-
|
17
|
-
-
|
18
|
-
-
|
19
|
-
|
20
|
-
|
41
|
+
- Enqueued jobs can be queried, executed and discarded.
|
42
|
+
- Executed jobs are re-enqueued or discarded, depending on the result.
|
43
|
+
- Compatibility with [Mission Control Jobs](https://github.com/basecamp/mission_control-jobs).
|
44
|
+
|
45
|
+
## Extensions
|
46
|
+
|
47
|
+
Marj does not provide the following features by default, but each can easily
|
48
|
+
be added to your application with a few lines of code. See [Example Usage](#example-usage)
|
49
|
+
and [Extension Examples](#extension-examples) for suggestions.
|
50
|
+
- [Automatic job execution](#example-usage)
|
51
|
+
- [Concurrency protection](#concurrency-protection)
|
52
|
+
- [Job timeouts](#job-timeouts)
|
53
|
+
- [Error storage](#error-storage)
|
54
|
+
- [Discarded job retention](#discarded-job-retention)
|
55
|
+
- [Observability](#observability)
|
56
|
+
- [Multi-DB Support](#multi-db-support)
|
57
|
+
|
58
|
+
## API
|
59
|
+
|
60
|
+
Marj relies on existing ActiveJob APIs, for example:
|
61
|
+
|
62
|
+
```ruby
|
63
|
+
job.enqueue # Enqueues a job
|
64
|
+
job.perform_now # Performs a job
|
65
|
+
```
|
66
|
+
|
67
|
+
Additionally, it extends the ActiveJob API with methods required for a
|
68
|
+
minimal queueing backend implementaion:
|
69
|
+
|
70
|
+
```ruby
|
71
|
+
SomeJob.query(args) # Queries for enqeueued jobs
|
72
|
+
job.discard # Runs discard callbacks and, by default, deletes the job
|
73
|
+
job.delete # Deletes the job
|
74
|
+
```
|
21
75
|
|
22
|
-
##
|
76
|
+
## Requirements
|
23
77
|
|
24
|
-
-
|
25
|
-
-
|
26
|
-
- Concurrency Controls
|
27
|
-
- Observability
|
28
|
-
- A User Interace
|
78
|
+
- `activejob >= 7.1`
|
79
|
+
- `activerecord >= 7.1`
|
29
80
|
|
30
81
|
## Setup
|
31
82
|
|
32
83
|
### 1. Install
|
33
84
|
|
34
85
|
```shell
|
35
|
-
bundle add
|
36
|
-
|
37
|
-
# or
|
38
|
-
|
39
|
-
gem install activejob activerecord marj
|
86
|
+
bundle add marj
|
40
87
|
```
|
41
88
|
|
42
|
-
###
|
89
|
+
### 2. Create the database table
|
43
90
|
|
44
91
|
```ruby
|
45
92
|
class CreateJobs < ActiveRecord::Migration[7.1]
|
@@ -68,149 +115,74 @@ class CreateJobs < ActiveRecord::Migration[7.1]
|
|
68
115
|
end
|
69
116
|
```
|
70
117
|
|
71
|
-
|
72
|
-
table name, set `Marj.table_name` before loading `ActiveRecord`.
|
73
|
-
|
74
|
-
### 4. Configure the queue adapter
|
118
|
+
### 3. Configure the queue adapter
|
75
119
|
|
76
120
|
```ruby
|
77
121
|
require 'marj'
|
78
122
|
|
123
|
+
# Choose one of the following:
|
79
124
|
Rails.configuration.active_job.queue_adapter = :marj # Globally, with Rails
|
80
125
|
ActiveJob::Base.queue_adapter = :marj # Globally, without Rails
|
81
126
|
SomeJob.queue_adapter = :marj # Single job
|
82
127
|
```
|
83
128
|
|
84
|
-
|
85
|
-
|
86
|
-
```ruby
|
87
|
-
# Enqueue and manually run a job:
|
88
|
-
job = SomeJob.perform_later('foo')
|
89
|
-
job.perform_now
|
129
|
+
### 4. Optionally, include the Marj module
|
90
130
|
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
# Run all due jobs (single DB query)
|
95
|
-
Marj.due.perform_all
|
96
|
-
|
97
|
-
# Run all due jobs (multiple DB queries)
|
98
|
-
Marj.due.perform_all(batch_size: 1)
|
99
|
-
|
100
|
-
# Run all due jobs in a specific queue:
|
101
|
-
Marj.queue('foo').due.perform_all
|
102
|
-
|
103
|
-
# Run jobs as they become due:
|
104
|
-
loop do
|
105
|
-
Marj.due.perform_all rescue logger.error($!)
|
106
|
-
ensure
|
107
|
-
sleep 5.seconds
|
108
|
-
end
|
109
|
-
```
|
110
|
-
|
111
|
-
## Jobs Interface
|
112
|
-
|
113
|
-
The `Marj` module provides methods for interacting with enqueued jobs. These
|
114
|
-
methods accept, return and yield +ActiveJob+ objects rather than +ActiveRecord+
|
115
|
-
objects. Returned jobs are orderd by due date. To query the database directly,
|
116
|
-
use `Marj::Record`.
|
117
|
-
|
118
|
-
Example usage:
|
131
|
+
Without any additional setup, jobs can be queried and discarded via the `Marj`
|
132
|
+
module:
|
119
133
|
|
120
134
|
```ruby
|
121
|
-
Marj.all
|
122
|
-
Marj.
|
123
|
-
Marj.due # Returns jobs which are due to be executed.
|
124
|
-
Marj.next # Returns the next job(s) to be executed.
|
125
|
-
Marj.count # Returns the number of enqueued jobs.
|
126
|
-
Marj.where # Returns jobs matching the specified criteria.
|
127
|
-
Marj.perform_all # Executes all jobs.
|
128
|
-
Marj.discard_all # Discards all jobs.
|
129
|
-
Marj.discard # Discards the specified job.
|
135
|
+
Marj.query(:all)
|
136
|
+
Marj.discard(job)
|
130
137
|
```
|
131
138
|
|
132
|
-
|
139
|
+
But it is also convenient to query or discard via job classes:
|
133
140
|
|
134
141
|
```ruby
|
135
|
-
|
142
|
+
SomeJob.query(:all)
|
143
|
+
SomeJob.discard(job)
|
144
|
+
job.discard
|
136
145
|
```
|
137
146
|
|
138
|
-
|
139
|
-
|
140
|
-
The `Marj::JobsInterface` can be added to any class or module. For example, to
|
141
|
-
add it to all jobs classes:
|
147
|
+
In order to enable this functionality, you must include the `Marj` module:
|
142
148
|
|
143
149
|
```ruby
|
144
|
-
class
|
145
|
-
|
150
|
+
class SomeJob < ActiveJob::Base
|
151
|
+
include Marj
|
146
152
|
|
147
|
-
def
|
148
|
-
Marj::Relation.new(
|
149
|
-
self == ApplicationJob ?
|
150
|
-
Marj::Record.ordered : Marj::Record.where(job_class: self)
|
151
|
-
)
|
152
|
-
end
|
153
|
+
def perform; end
|
153
154
|
end
|
154
|
-
|
155
|
-
class SomeJob < ApplicationJob; end
|
156
|
-
|
157
|
-
ApplicationJob.due # Returns all jobs which are due to be executed.
|
158
|
-
SomeJob.due # Returns SomeJobs which are due to be executed.
|
159
155
|
```
|
160
156
|
|
161
|
-
##
|
162
|
-
|
163
|
-
It is possible to create a custom record class in order to, for instance,
|
164
|
-
write jobs to multiple databases/tables within a single application.
|
157
|
+
## Example Usage
|
165
158
|
|
166
159
|
```ruby
|
167
|
-
|
168
|
-
|
169
|
-
create_table :my_jobs, id: :string, primary_key: :job_id do |table|
|
170
|
-
table.string :job_class, null: false
|
171
|
-
table.text :arguments, null: false
|
172
|
-
table.string :queue_name, null: false
|
173
|
-
table.integer :priority
|
174
|
-
table.integer :executions, null: false
|
175
|
-
table.text :exception_executions, null: false
|
176
|
-
table.datetime :enqueued_at, null: false
|
177
|
-
table.datetime :scheduled_at
|
178
|
-
table.string :locale, null: false
|
179
|
-
table.string :timezone, null: false
|
180
|
-
end
|
181
|
-
|
182
|
-
add_index :my_jobs, %i[enqueued_at]
|
183
|
-
add_index :my_jobs, %i[scheduled_at]
|
184
|
-
add_index :my_jobs, %i[priority scheduled_at enqueued_at]
|
185
|
-
end
|
186
|
-
|
187
|
-
def self.down
|
188
|
-
drop_table :my_jobs
|
189
|
-
end
|
190
|
-
end
|
191
|
-
|
192
|
-
class MyRecord < Marj::Record
|
193
|
-
self.table_name = 'my_jobs'
|
194
|
-
end
|
195
|
-
|
196
|
-
CreateMyJobs.migrate(:up)
|
160
|
+
# Enqueue a job
|
161
|
+
job = SomeJob.perform_later('foo')
|
197
162
|
|
198
|
-
|
199
|
-
|
163
|
+
# Query jobs
|
164
|
+
SomeJob.query(:all)
|
165
|
+
SomeJob.query(:due)
|
166
|
+
SomeJob.query(:due, queue: :foo)
|
167
|
+
SomeJob.query('8720417d-8fff-4fcf-bc16-22aaef8543d2')
|
200
168
|
|
201
|
-
|
169
|
+
# Execute a job
|
170
|
+
job.perform_now
|
202
171
|
|
203
|
-
|
204
|
-
|
205
|
-
end
|
172
|
+
# Execute jobs which are due (single query)
|
173
|
+
SomeJob.query(:due).map(&:perform_now)
|
206
174
|
|
207
|
-
|
208
|
-
|
209
|
-
|
175
|
+
# Execute jobs which are due (multiple queries)
|
176
|
+
loop {
|
177
|
+
SomeJob.query(:due, :first)&.tap(&:perform_now) || break
|
210
178
|
end
|
211
179
|
|
212
|
-
|
213
|
-
|
180
|
+
# Execute jobs as they become due
|
181
|
+
loop do
|
182
|
+
SomeJob.query(:due).each(&:perform_now) rescue logger.error($!)
|
183
|
+
ensure
|
184
|
+
sleep 5.seconds
|
185
|
+
end
|
214
186
|
```
|
215
187
|
|
216
188
|
## Testing
|
@@ -219,17 +191,54 @@ By default, jobs enqeued during tests will be written to the database. Enqueued
|
|
219
191
|
jobs can be executed via:
|
220
192
|
|
221
193
|
```ruby
|
222
|
-
|
194
|
+
SomeJob.query(:due).map(&:perform_now)
|
223
195
|
```
|
224
196
|
|
225
|
-
Alternatively, to use [
|
197
|
+
Alternatively, to use ActiveJob's [TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
|
226
198
|
```ruby
|
227
199
|
ActiveJob::Base.queue_adapter = :test
|
228
200
|
```
|
229
201
|
|
230
202
|
## Extension Examples
|
231
203
|
|
232
|
-
|
204
|
+
Most missing features can easily be added to your application with a few lines
|
205
|
+
of code.
|
206
|
+
|
207
|
+
### Concurrency Protection
|
208
|
+
|
209
|
+
To prevent two threads from executing the same job simultaneously, consider
|
210
|
+
moving jobs to a different queue before executing them.
|
211
|
+
|
212
|
+
```ruby
|
213
|
+
class ApplicationJob < ActiveJob::Base
|
214
|
+
around_perform do |job, block|
|
215
|
+
if job.queue_name.start_with?('claimed')
|
216
|
+
raise "Job #{job.job_id} already claimed"
|
217
|
+
end
|
218
|
+
|
219
|
+
updated = Marj::Record
|
220
|
+
.where(job_id: job.job_id, queue_name: job.queue_name)
|
221
|
+
.update_all(
|
222
|
+
queue_name: "claimed-#{job.queue_name}",
|
223
|
+
scheduled_at: Time.now.utc
|
224
|
+
)
|
225
|
+
unless updated == 1
|
226
|
+
raise "Failed to claim job #{job.job_id}. #{updated} records updated"
|
227
|
+
end
|
228
|
+
|
229
|
+
begin
|
230
|
+
block.call
|
231
|
+
rescue StandardError
|
232
|
+
Marj::Record
|
233
|
+
.where(job_id: job.job_id, queue_name: "claimed-#{job.queue_name}")
|
234
|
+
.update_all(queue_name: job.queue_name, scheduled_at: job.scheduled_at)
|
235
|
+
raise
|
236
|
+
end
|
237
|
+
end
|
238
|
+
end
|
239
|
+
```
|
240
|
+
|
241
|
+
### Job Timeouts
|
233
242
|
|
234
243
|
```ruby
|
235
244
|
class ApplicationJob < ActiveJob::Base
|
@@ -249,7 +258,7 @@ class ApplicationJob < ActiveJob::Base
|
|
249
258
|
end
|
250
259
|
```
|
251
260
|
|
252
|
-
###
|
261
|
+
### Error Storage
|
253
262
|
|
254
263
|
```ruby
|
255
264
|
class AddLastErrorToJobs < ActiveRecord::Migration[7.1]
|
@@ -290,6 +299,83 @@ class ApplicationJob < ActiveJob::Base
|
|
290
299
|
end
|
291
300
|
```
|
292
301
|
|
302
|
+
### Discarded Job Retention
|
303
|
+
|
304
|
+
By default, discarded jobs are deleted from the database. If desired, this
|
305
|
+
behavior can be overridden, for instance to move jobs to a differnt queue:
|
306
|
+
|
307
|
+
```ruby
|
308
|
+
ActiveJob::Base.queue_adapter = MarjAdapter.new(
|
309
|
+
discard: proc { _1.enqueue(queue: 'discarded') }
|
310
|
+
)
|
311
|
+
|
312
|
+
# To retrieve a discarded job, query the discarded queue:
|
313
|
+
job = Marj.query(:first, queue_name: 'discarded')
|
314
|
+
|
315
|
+
# To delete, use one of the following:
|
316
|
+
Marj.delete(job)
|
317
|
+
job.delete
|
318
|
+
```
|
319
|
+
|
320
|
+
### Observability
|
321
|
+
|
322
|
+
For instance with Prometheus metrics:
|
323
|
+
|
324
|
+
```ruby
|
325
|
+
class ApplicationJob < ActiveJob::Base
|
326
|
+
counter = Prometheus::Client::Counter.new(
|
327
|
+
:job_events_total, docstring: '...', labels: [:job, :event]
|
328
|
+
)
|
329
|
+
|
330
|
+
around_enqueue do |job, block|
|
331
|
+
counter.increment(labels: { job: job.class.name, event: 'before_enqueue' })
|
332
|
+
block.call
|
333
|
+
counter.increment(labels: { job: job.class.name, event: 'after_enqueue' })
|
334
|
+
rescue Exception
|
335
|
+
counter.increment(labels: { job: job.class.name, event: 'enqueue_error' })
|
336
|
+
raise
|
337
|
+
end
|
338
|
+
|
339
|
+
around_perform do |job, block|
|
340
|
+
counter.increment(labels: { job: job.class.name, event: 'before_perform' })
|
341
|
+
block.call
|
342
|
+
counter.increment(labels: { job: job.class.name, event: 'after_perform' })
|
343
|
+
rescue Exception => e # rubocop:disable Lint/RescueException
|
344
|
+
counter.increment(labels: { job: job.class.name, event: 'perform_error' })
|
345
|
+
raise
|
346
|
+
end
|
347
|
+
|
348
|
+
after_discard do |job|
|
349
|
+
counter.increment(labels: { job: job.class.name, event: 'after_discard' })
|
350
|
+
end
|
351
|
+
end
|
352
|
+
```
|
353
|
+
|
354
|
+
### Multi-DB Support
|
355
|
+
|
356
|
+
It is possible to create a custom record class in order to, for instance,
|
357
|
+
use a different class or table name, or write jobs to multiple
|
358
|
+
databases/tables within a single application.
|
359
|
+
|
360
|
+
Assuming you have a jobs tabled named `my_jobs`:
|
361
|
+
|
362
|
+
```ruby
|
363
|
+
class MyRecord < Marj::Record
|
364
|
+
self.table_name = 'my_jobs'
|
365
|
+
end
|
366
|
+
|
367
|
+
class MyJob < ActiveJob::Base
|
368
|
+
include Marj
|
369
|
+
|
370
|
+
self.queue_adapter = MarjAdapter.new('MyRecord')
|
371
|
+
|
372
|
+
def perform; end
|
373
|
+
end
|
374
|
+
|
375
|
+
MyJob.perform_later('oh, hi')
|
376
|
+
MyJob.query(:due, :first).perform_now
|
377
|
+
```
|
378
|
+
|
293
379
|
## ActiveJob Cheatsheet
|
294
380
|
|
295
381
|
For more information on ActiveJob, see:
|
@@ -0,0 +1,119 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative '../marj_adapter'
|
4
|
+
require 'mission_control/jobs'
|
5
|
+
require 'mission_control/jobs/adapter'
|
6
|
+
|
7
|
+
# :nocov:
|
8
|
+
module Marj
|
9
|
+
module MissionControl
|
10
|
+
include ::MissionControl::Jobs::Adapter
|
11
|
+
|
12
|
+
def queues
|
13
|
+
record_class.group(:queue_name).count(:queue_name).map { |k, v| { name: k, size: v, active: true } }
|
14
|
+
end
|
15
|
+
|
16
|
+
def queue_size(queue_name)
|
17
|
+
record_class.where(queue_name: queue_name).count
|
18
|
+
end
|
19
|
+
|
20
|
+
def clear_queue(queue_name)
|
21
|
+
Marj::Record.where(queue_name: queue_name).delete_all
|
22
|
+
end
|
23
|
+
|
24
|
+
def pause_queue(_queue_name)
|
25
|
+
raise 'not supported: pause queue'
|
26
|
+
end
|
27
|
+
|
28
|
+
def resume_queue(_queue_name)
|
29
|
+
raise 'not supported: resume queue'
|
30
|
+
end
|
31
|
+
|
32
|
+
def queue_paused?(_queue_name)
|
33
|
+
false
|
34
|
+
end
|
35
|
+
|
36
|
+
def supported_statuses
|
37
|
+
%i[pending failed scheduled]
|
38
|
+
end
|
39
|
+
|
40
|
+
def supported_filters(_jobs_relation)
|
41
|
+
%i[queue_name job_class_name]
|
42
|
+
end
|
43
|
+
|
44
|
+
def exposes_workers?
|
45
|
+
false
|
46
|
+
end
|
47
|
+
|
48
|
+
def workers
|
49
|
+
raise 'not supported: workers'
|
50
|
+
end
|
51
|
+
|
52
|
+
def find_worker(_worker_id)
|
53
|
+
raise 'not supported: find workers'
|
54
|
+
end
|
55
|
+
|
56
|
+
def jobs_count(jobs_relation)
|
57
|
+
ar_relation(jobs_relation).count
|
58
|
+
end
|
59
|
+
|
60
|
+
def fetch_jobs(jobs_relation)
|
61
|
+
ar_relation(jobs_relation).each_with_index.map { |record, index| to_job(record, jobs_relation, index) }
|
62
|
+
end
|
63
|
+
|
64
|
+
def retry_all_jobs(jobs_relation)
|
65
|
+
ar_relation(jobs_relation).map { |record| record.to_job.perform_now }
|
66
|
+
end
|
67
|
+
|
68
|
+
def retry_job(job, _jobs_relation)
|
69
|
+
Marj::Record.find(job.job_id).to_job.perform_now
|
70
|
+
end
|
71
|
+
|
72
|
+
def discard_all_jobs(jobs_relation)
|
73
|
+
ar_relation(jobs_relation).map { |record| discard(record.to_job) }
|
74
|
+
end
|
75
|
+
|
76
|
+
def discard_job(job, _jobs_relation)
|
77
|
+
discard(Marj::Record.find(job.job_id).to_job)
|
78
|
+
end
|
79
|
+
|
80
|
+
def find_job(job_id, jobs_relation)
|
81
|
+
to_job(record_class.find_by(job_id: job_id), jobs_relation)
|
82
|
+
end
|
83
|
+
|
84
|
+
private
|
85
|
+
|
86
|
+
def ar_relation(jobs_relation)
|
87
|
+
relation = Marj::Record.all.offset(jobs_relation.offset_value).limit(jobs_relation.limit_value)
|
88
|
+
relation = relation.where.not(executions: 0) if jobs_relation.status == :failed
|
89
|
+
relation = relation.where.not(scheduled_at: nil) if jobs_relation.status == :scheduled
|
90
|
+
relation = relation.where(job_class: jobs_relation.job_class_name) if jobs_relation.job_class_name
|
91
|
+
relation = relation.where(queue_name: jobs_relation.queue_name) if jobs_relation.queue_name
|
92
|
+
relation
|
93
|
+
end
|
94
|
+
|
95
|
+
def to_job(record, jobs_relation, index = 0)
|
96
|
+
return nil unless record
|
97
|
+
|
98
|
+
job = record.to_job
|
99
|
+
job_data = job.serialize
|
100
|
+
ActiveJob::JobProxy.new(job_data).tap do |proxy|
|
101
|
+
if job.executions.positive?
|
102
|
+
proxy.last_execution_error = ActiveJob::ExecutionError.new(
|
103
|
+
error_class: Exception, message: 'unknown', backtrace: []
|
104
|
+
)
|
105
|
+
proxy.failed_at = job.enqueued_at
|
106
|
+
proxy.status = :failed
|
107
|
+
elsif job.scheduled_at
|
108
|
+
proxy.status = :scheduled
|
109
|
+
else
|
110
|
+
proxy.status = :pending
|
111
|
+
end
|
112
|
+
proxy.raw_data = job_data
|
113
|
+
proxy.position = jobs_relation.offset_value + index
|
114
|
+
proxy.arguments = job.arguments # For some reason MissionControl sets the arguments to the entire job data
|
115
|
+
end
|
116
|
+
end
|
117
|
+
end
|
118
|
+
end
|
119
|
+
# :nocov:
|
data/lib/marj/record.rb
CHANGED
@@ -6,7 +6,7 @@ require 'active_record'
|
|
6
6
|
module Marj
|
7
7
|
# The default +ActiveRecord+ class.
|
8
8
|
class Record < ActiveRecord::Base
|
9
|
-
self.table_name =
|
9
|
+
self.table_name = :jobs
|
10
10
|
|
11
11
|
# Order by +enqueued_at+ rather than +job_id+ (the default).
|
12
12
|
self.implicit_order_column = 'enqueued_at'
|
@@ -50,9 +50,47 @@ module Marj
|
|
50
50
|
# discarded.
|
51
51
|
#
|
52
52
|
# @return [ActiveJob::Base]
|
53
|
-
def
|
54
|
-
|
53
|
+
def to_job
|
54
|
+
# See register_callbacks for details on how callbacks are used.
|
55
|
+
job = job_class.new.tap { register_callbacks(_1) }
|
56
|
+
|
57
|
+
# ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
|
58
|
+
# deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
|
59
|
+
job_data = attributes.merge('arguments' => JSON.parse(read_attribute_before_type_cast(:arguments)))
|
60
|
+
|
61
|
+
# ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
|
62
|
+
job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
|
63
|
+
|
64
|
+
job.deserialize(job_data)
|
65
|
+
|
66
|
+
# ActiveJob deserializes arguments on demand when a job is performed. Until then they are empty. That's strange.
|
67
|
+
# Instead, deserialize them now. Also, clear `serialized_arguments` to prevent ActiveJob from overwriting changes
|
68
|
+
# to arguments when serializing later.
|
69
|
+
job.arguments = arguments
|
70
|
+
job.serialized_arguments = nil
|
71
|
+
|
72
|
+
job
|
73
|
+
end
|
74
|
+
|
75
|
+
# Registers callbacks for the given job which destroy this record when the job succeeds or is discarded.
|
76
|
+
#
|
77
|
+
# @param job [ActiveJob::Base]
|
78
|
+
# @return [ActiveJob::Base]
|
79
|
+
def register_callbacks(job)
|
80
|
+
raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
|
81
|
+
|
82
|
+
record = self
|
83
|
+
# We need to detect three cases:
|
84
|
+
# - If a job succeeds, after_perform will be called.
|
85
|
+
# - If a job fails and should be retried, enqueue will be called. This is handled by the queue adapter.
|
86
|
+
# - If a job exceeds its max attempts, after_discard will be called.
|
87
|
+
job.singleton_class.after_perform { |_j| job.queue_adapter.delete(job) }
|
88
|
+
job.singleton_class.after_discard { |_j, _exception| job.queue_adapter.discard(job, run_callbacks: false) }
|
89
|
+
job.singleton_class.instance_variable_set(:@record, record)
|
90
|
+
|
91
|
+
job
|
55
92
|
end
|
93
|
+
private :register_callbacks
|
56
94
|
|
57
95
|
class << self
|
58
96
|
# Returns an +ActiveRecord::Relation+ scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in
|
@@ -63,11 +101,16 @@ module Marj
|
|
63
101
|
where('scheduled_at IS NULL OR scheduled_at <= ?', Time.now.utc)
|
64
102
|
end
|
65
103
|
|
66
|
-
# Returns an +ActiveRecord::Relation+ scope for jobs ordered by
|
67
|
-
#
|
104
|
+
# Returns an +ActiveRecord::Relation+ scope for jobs ordered by due date.
|
105
|
+
#
|
106
|
+
# Jobs are ordered by the following criteria, in order:
|
107
|
+
# 1. past or null scheduled_at before future scheduled_at
|
108
|
+
# 2. ascending priority, nulls last
|
109
|
+
# 3. ascending scheduled_at, nulls last
|
110
|
+
# 4. ascending enqueued_at
|
68
111
|
#
|
69
112
|
# @return [ActiveRecord::Relation]
|
70
|
-
def
|
113
|
+
def by_due_date
|
71
114
|
order(
|
72
115
|
Arel.sql(<<~SQL.squish, Time.now.utc)
|
73
116
|
CASE WHEN scheduled_at IS NULL OR scheduled_at <= ? THEN 0 ELSE 1 END,
|
data/lib/marj.rb
CHANGED
@@ -1,164 +1,131 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require_relative 'marj_adapter'
|
4
|
-
require_relative 'marj/jobs_interface'
|
5
|
-
require_relative 'marj/relation'
|
6
4
|
|
7
5
|
# A minimal database-backed ActiveJob queueing backend.
|
8
6
|
#
|
9
|
-
# The {Marj} module provides
|
10
|
-
# +
|
11
|
-
#
|
7
|
+
# The {Marj} module provides the following methods:
|
8
|
+
# - +query+ - Queries enqueued jobs
|
9
|
+
# - +discard+ - Discards a job, by default by executing after_discard callbacks and delegating to delete
|
10
|
+
# - +delete+ - Deletes a job
|
11
|
+
#
|
12
|
+
# It is possible to call the above methods on the {Marj} module itself or on any class which includes it.
|
12
13
|
#
|
13
14
|
# Example usage:
|
14
|
-
# Marj.
|
15
|
-
# Marj.
|
16
|
-
# Marj.
|
17
|
-
#
|
18
|
-
#
|
19
|
-
#
|
20
|
-
#
|
21
|
-
# Marj.discard_all # Discards all jobs.
|
22
|
-
# Marj.discard # Discards the specified job.
|
15
|
+
# Marj.query(:first) # Returns the first job
|
16
|
+
# Marj.discard(job) # Discards the specified job
|
17
|
+
# Marj.delete(job) # Deletes the specified job
|
18
|
+
#
|
19
|
+
# class ApplicationJob < ActiveJob::Base
|
20
|
+
# include Marj
|
21
|
+
# end
|
23
22
|
#
|
24
|
-
#
|
25
|
-
#
|
23
|
+
# class SomeJob < ApplicationJob;
|
24
|
+
# def perform; end
|
25
|
+
# end
|
26
26
|
#
|
27
|
-
#
|
28
|
-
#
|
27
|
+
# job = ApplicationJob.query(:first) # Returns the first enqueued job
|
28
|
+
# job = SomeJob.query(:first) # Returns the first enqueued job with job_class SomeJob
|
29
|
+
# ApplicationJob.discard(job) # Discards the specified job
|
30
|
+
# ApplicationJob.delete(job) # Deletes the specified job
|
31
|
+
# job.discard # Discards the job
|
32
|
+
# job.delete # Deletes the job
|
29
33
|
#
|
30
34
|
# See https://github.com/nicholasdower/marj
|
31
35
|
module Marj
|
32
36
|
# The Marj version.
|
33
|
-
VERSION = '
|
37
|
+
VERSION = '6.0.0'
|
34
38
|
|
35
39
|
Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
|
36
40
|
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
#
|
44
|
-
# The name of the +ActiveRecord+ class. Defaults to +Marj::Record+.
|
45
|
-
# @return [Class, String]
|
46
|
-
|
47
|
-
attr_writer :record_class
|
48
|
-
|
49
|
-
def record_class
|
50
|
-
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
51
|
-
end
|
52
|
-
|
53
|
-
# @!attribute table_name
|
54
|
-
# The name of the database table. Defaults to +:jobs+.
|
55
|
-
# @return [Symbol, String]
|
56
|
-
attr_accessor :table_name
|
57
|
-
|
58
|
-
# Returns a {Marj::Relation} for all jobs in the order they should be executed.
|
41
|
+
# Provides the {query} and {discard} class methods.
|
42
|
+
module ClassMethods
|
43
|
+
# Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
|
44
|
+
# - Symbol arguments are treated as +ActiveRecord+ scopes.
|
45
|
+
# - If only a job ID is specified, the corresponding job is returned.
|
46
|
+
# - If +:limit+ is specified, the maximum number of jobs is limited.
|
47
|
+
# - If +:order+ is specified, the jobs are ordered by the given attribute.
|
59
48
|
#
|
60
|
-
#
|
61
|
-
|
62
|
-
|
49
|
+
# By default jobs are ordered by when they should be executed.
|
50
|
+
#
|
51
|
+
# Example usage:
|
52
|
+
# query # Returns all jobs
|
53
|
+
# query(:all) # Returns all jobs
|
54
|
+
# query(:due) # Returns jobs which are due to be executed
|
55
|
+
# query(:due, limit: 10) # Returns at most 10 jobs which are due to be executed
|
56
|
+
# query(job_class: Foo) # Returns all jobs with job_class Foo
|
57
|
+
# query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
|
58
|
+
# query(queue_name: 'foo') # Returns all jobs in the 'foo' queue
|
59
|
+
# query(job_id: '123') # Returns the job with job_id '123' or nil if no such job exists
|
60
|
+
# query('123') # Returns the job with job_id '123' or nil if no such job exists
|
61
|
+
def query(*args, **kwargs)
|
62
|
+
kwargs[:job_class] ||= self if self < ActiveJob::Base && name != 'ApplicationJob'
|
63
|
+
queue_adapter.query(*args, **kwargs)
|
63
64
|
end
|
64
65
|
|
65
66
|
# Discards the specified job.
|
66
67
|
#
|
67
|
-
# @return [
|
68
|
+
# @return [ActiveJob::Base] the discarded job
|
68
69
|
def discard(job)
|
69
|
-
|
70
|
+
queue_adapter.discard(job)
|
70
71
|
end
|
71
72
|
|
72
|
-
|
73
|
-
|
74
|
-
# Creates a job instance for the given record which will update the database when successfully executed, enqueued or
|
75
|
-
# discarded.
|
73
|
+
# Deletes the record associated with the specified job.
|
76
74
|
#
|
77
|
-
# @
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
# ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
|
84
|
-
# deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
|
85
|
-
job_data = record.attributes.merge('arguments' => JSON.parse(record.read_attribute_before_type_cast(:arguments)))
|
86
|
-
|
87
|
-
# ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
|
88
|
-
job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
|
75
|
+
# @return [ActiveJob::Base] the deleted job
|
76
|
+
def delete(job)
|
77
|
+
queue_adapter.delete(job)
|
78
|
+
end
|
79
|
+
end
|
89
80
|
|
90
|
-
|
81
|
+
# (see ClassMethods#query)
|
82
|
+
def self.query(*args, **kwargs)
|
83
|
+
queue_adapter.query(*args, **kwargs)
|
84
|
+
end
|
91
85
|
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
job.serialized_arguments = nil
|
86
|
+
# (see ClassMethods#discard)
|
87
|
+
def self.discard(job)
|
88
|
+
queue_adapter.discard(job)
|
89
|
+
end
|
97
90
|
|
98
|
-
|
99
|
-
|
91
|
+
# (see ClassMethods#delete)
|
92
|
+
def self.delete(job)
|
93
|
+
queue_adapter.delete(job)
|
94
|
+
end
|
100
95
|
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
def register_callbacks(job, record)
|
108
|
-
raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
|
96
|
+
# Deletes this job.
|
97
|
+
#
|
98
|
+
# @return [ActiveJob::Base] this job
|
99
|
+
def discard
|
100
|
+
self.class.queue_adapter.discard(self)
|
101
|
+
end
|
109
102
|
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
job.singleton_class.instance_variable_set(:@record, record)
|
103
|
+
# Deletes the record associated with this job.
|
104
|
+
#
|
105
|
+
# @return [ActiveJob::Base] this job
|
106
|
+
def delete
|
107
|
+
self.class.queue_adapter.delete(self)
|
108
|
+
end
|
117
109
|
|
118
|
-
|
119
|
-
|
110
|
+
def self.included(clazz)
|
111
|
+
clazz.extend(ClassMethods)
|
112
|
+
end
|
113
|
+
private_class_method :included
|
120
114
|
|
121
|
-
|
122
|
-
|
123
|
-
|
124
|
-
|
125
|
-
|
126
|
-
# @return [ActiveJob::Base] the enqueued job
|
127
|
-
def enqueue(job, record_class, time = nil)
|
128
|
-
job.scheduled_at = time
|
129
|
-
# Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
|
130
|
-
serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
|
115
|
+
def self.queue_adapter
|
116
|
+
ActiveJob::Base.queue_adapter
|
117
|
+
end
|
118
|
+
private_class_method :queue_adapter
|
119
|
+
end
|
131
120
|
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
136
|
-
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
# This job instance has already been associated with a database row.
|
141
|
-
if record_class.exists?(job_id: job.job_id)
|
142
|
-
# The database row still exists, we simply need to update it.
|
143
|
-
existing_record.update!(serialized)
|
144
|
-
else
|
145
|
-
# Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
|
146
|
-
# want to register the new instance because someone might still have a reference to the existing one.
|
147
|
-
record_class.create!(serialized)
|
148
|
-
existing_record.reload
|
149
|
-
end
|
150
|
-
else
|
151
|
-
# This job instance has not been associated with a database row.
|
152
|
-
if (new_record = record_class.find_by(job_id: job.job_id))
|
153
|
-
# The database row already exists. Update it.
|
154
|
-
new_record.update!(serialized)
|
155
|
-
else
|
156
|
-
# The database row does not exist. Create it.
|
157
|
-
new_record = record_class.create!(serialized)
|
158
|
-
end
|
159
|
-
register_callbacks(job, new_record)
|
160
|
-
end
|
161
|
-
job
|
162
|
-
end
|
121
|
+
# :nocov:
|
122
|
+
if defined?(Rails)
|
123
|
+
begin
|
124
|
+
require 'mission_control/jobs'
|
125
|
+
require_relative 'marj/mission_control'
|
126
|
+
MarjAdapter.include(Marj::MissionControl)
|
127
|
+
rescue LoadError
|
128
|
+
# ignore
|
163
129
|
end
|
164
130
|
end
|
131
|
+
# :nocov:
|
data/lib/marj_adapter.rb
CHANGED
@@ -2,13 +2,26 @@
|
|
2
2
|
|
3
3
|
# ActiveJob queue adapter for Marj.
|
4
4
|
#
|
5
|
+
# In addition to the standard +ActiveJob+ queue adapter API, this adapter provides:
|
6
|
+
# - A +query+ method which can be used to query enqueued jobs
|
7
|
+
# - A +discard+ method which can be used to discard enqueued jobs.
|
8
|
+
# - A +delete+ method which can be used to delete enqueued jobs.
|
9
|
+
#
|
10
|
+
# Although it is possible to access the adapter directly in order to query, discard or delete, it is recommended to use
|
11
|
+
# the {Marj} module.
|
12
|
+
#
|
5
13
|
# See https://github.com/nicholasdower/marj
|
6
14
|
class MarjAdapter
|
15
|
+
JOB_ID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.freeze
|
16
|
+
private_constant :JOB_ID_REGEX
|
17
|
+
|
7
18
|
# Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ class.
|
8
19
|
#
|
9
|
-
# @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use to
|
10
|
-
|
20
|
+
# @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use, defaults to +Marj::Record+
|
21
|
+
# @param discard [Proc] the proc to use to discard jobs, defaults to delegating to {delete}
|
22
|
+
def initialize(record_class: 'Marj::Record', discard: proc { |job| delete(job) })
|
11
23
|
@record_class = record_class
|
24
|
+
@discard_proc = discard
|
12
25
|
end
|
13
26
|
|
14
27
|
# Enqueue a job for immediate execution.
|
@@ -16,7 +29,7 @@ class MarjAdapter
|
|
16
29
|
# @param job [ActiveJob::Base] the job to enqueue
|
17
30
|
# @return [ActiveJob::Base] the enqueued job
|
18
31
|
def enqueue(job)
|
19
|
-
|
32
|
+
enqueue_at(job)
|
20
33
|
end
|
21
34
|
|
22
35
|
# Enqueue a job for execution at the specified time.
|
@@ -24,13 +37,146 @@ class MarjAdapter
|
|
24
37
|
# @param job [ActiveJob::Base] the job to enqueue
|
25
38
|
# @param timestamp [Numeric, NilClass] optional number of seconds since Unix epoch at which to execute the job
|
26
39
|
# @return [ActiveJob::Base] the enqueued job
|
27
|
-
def enqueue_at(job, timestamp)
|
28
|
-
|
40
|
+
def enqueue_at(job, timestamp = nil)
|
41
|
+
job.scheduled_at = timestamp ? Time.at(timestamp).utc : nil
|
42
|
+
|
43
|
+
# Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
|
44
|
+
serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
|
45
|
+
|
46
|
+
# Serialize sets locale to I18n.locale.to_s and enqueued_at to Time.now.utc.iso8601(9).
|
47
|
+
# Update the job to reflect what is being enqueued.
|
48
|
+
job.locale = serialized[:locale]
|
49
|
+
job.enqueued_at = Time.iso8601(serialized[:enqueued_at]).utc
|
50
|
+
|
51
|
+
# When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
|
52
|
+
# are registered on the job instance so that when the job is executed, the database record is deleted or updated
|
53
|
+
# (depending on the result).
|
54
|
+
#
|
55
|
+
# We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
|
56
|
+
# singleton class. This holds a reference to the record. This ensures that if execute is called on a record
|
57
|
+
# instance, any updates to the database are reflected on that record instance.
|
58
|
+
if (existing_record = job.singleton_class.instance_variable_get(:@record))
|
59
|
+
# This job instance has already been associated with a database row.
|
60
|
+
if record_class.exists?(job_id: job.job_id)
|
61
|
+
# The database row still exists, we simply need to update it.
|
62
|
+
existing_record.update!(serialized)
|
63
|
+
else
|
64
|
+
# Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
|
65
|
+
# want to register the new instance because someone might still have a reference to the existing one.
|
66
|
+
record_class.create!(serialized)
|
67
|
+
existing_record.reload
|
68
|
+
end
|
69
|
+
else
|
70
|
+
# This job instance has not been associated with a database row.
|
71
|
+
if (new_record = record_class.find_by(job_id: job.job_id))
|
72
|
+
# The database row already exists. Update it.
|
73
|
+
new_record.update!(serialized)
|
74
|
+
else
|
75
|
+
# The database row does not exist. Create it.
|
76
|
+
new_record = record_class.create!(serialized)
|
77
|
+
end
|
78
|
+
new_record.send(:register_callbacks, job)
|
79
|
+
end
|
80
|
+
job
|
81
|
+
end
|
82
|
+
|
83
|
+
# Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
|
84
|
+
# - Symbol arguments are treated as +ActiveRecord+ scopes.
|
85
|
+
# - If only a job ID is specified, the corresponding job is returned.
|
86
|
+
# - If +:limit+ is specified, the maximum number of jobs is limited.
|
87
|
+
# - If +:order+ is specified, the jobs are ordered by the given attribute.
|
88
|
+
#
|
89
|
+
# By default jobs are ordered by when they should be executed.
|
90
|
+
#
|
91
|
+
# Example usage:
|
92
|
+
# query # Returns all jobs
|
93
|
+
# query(:all) # Returns all jobs
|
94
|
+
# query(:due) # Returns jobs which are due to be executed
|
95
|
+
# query(:due, limit: 10) # Returns at most 10 jobs which are due to be executed
|
96
|
+
# query(job_class: Foo) # Returns all jobs with job_class Foo
|
97
|
+
# query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
|
98
|
+
# query(queue_name: 'foo') # Returns all jobs in the 'foo' queue
|
99
|
+
# query(job_id: '123') # Returns the job with job_id '123' or nil if no such job exists
|
100
|
+
# query('123') # Returns the job with job_id '123' or nil if no such job exists
|
101
|
+
def query(*args, **kwargs)
|
102
|
+
args, kwargs = args.dup, kwargs.dup.symbolize_keys
|
103
|
+
kwargs = kwargs.merge(job_id: kwargs.delete(:id)) if kwargs.key?(:id)
|
104
|
+
kwargs[:job_id] = args.shift if args.size == 1 && args.first.is_a?(String) && args.first.match(JOB_ID_REGEX)
|
105
|
+
|
106
|
+
if args.empty? && kwargs.size == 1 && kwargs.key?(:job_id)
|
107
|
+
return record_class.find_by(job_id: kwargs[:job_id])&.to_job
|
108
|
+
end
|
109
|
+
|
110
|
+
symbol_args, args = args.partition { _1.is_a?(Symbol) }
|
111
|
+
symbol_args.delete(:all)
|
112
|
+
limit = kwargs.delete(:limit)
|
113
|
+
relation = record_class.all
|
114
|
+
relation = relation.order(kwargs.delete(:order)) if kwargs.key?(:order)
|
115
|
+
relation = relation.where(*args, **kwargs) if args.any? || kwargs.any?
|
116
|
+
relation = relation.limit(limit) if limit
|
117
|
+
relation = relation.send(symbol_args.shift) while symbol_args.any?
|
118
|
+
relation = relation.by_due_date if relation.is_a?(ActiveRecord::Relation) && relation.order_values.empty?
|
119
|
+
|
120
|
+
if relation.is_a?(Enumerable)
|
121
|
+
relation.map(&:to_job)
|
122
|
+
elsif relation.is_a?(record_class)
|
123
|
+
relation.to_job
|
124
|
+
else
|
125
|
+
relation
|
126
|
+
end
|
127
|
+
end
|
128
|
+
|
129
|
+
# Discards the specified job.
|
130
|
+
#
|
131
|
+
# @param job [ActiveJob::Base] the job being discarded
|
132
|
+
# @param run_callbacks [Boolean] whether to run the +after_discard+ callbacks
|
133
|
+
# @return [ActiveJob::Base] the discarded job
|
134
|
+
def discard(job, run_callbacks: true)
|
135
|
+
job.tap do
|
136
|
+
@discard_proc.call(job)
|
137
|
+
run_after_discard_callbacks(job) if run_callbacks
|
138
|
+
end
|
139
|
+
end
|
140
|
+
|
141
|
+
# Deletes the record associated with the specified job.
|
142
|
+
#
|
143
|
+
# @return [ActiveJob::Base] the deleted job
|
144
|
+
def delete(job)
|
145
|
+
job.tap { destroy_record(job) }
|
29
146
|
end
|
30
147
|
|
31
148
|
private
|
32
149
|
|
150
|
+
# Returns the +ActiveRecord+ class to use to store jobs.
|
151
|
+
#
|
152
|
+
# @return [Class] the +ActiveRecord+ class
|
33
153
|
def record_class
|
34
154
|
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
35
155
|
end
|
156
|
+
|
157
|
+
# Destroys the record associated with the given job if it exists.
|
158
|
+
#
|
159
|
+
# @return [ActiveRecord::Base, NilClass] the destroyed record or +nil+ if no such record exists
|
160
|
+
def destroy_record(job)
|
161
|
+
record = job.singleton_class.instance_variable_get(:@record)
|
162
|
+
record ||= Marj::Record.find_by(job_id: job.job_id)&.tap { _1.send(:register_callbacks, job) }
|
163
|
+
record&.destroy
|
164
|
+
end
|
165
|
+
|
166
|
+
# Invokes the specified job's +after_discard+ callbacks.
|
167
|
+
#
|
168
|
+
# @param job [ActiveJob::Base] the job being discarded
|
169
|
+
# @return [NilClass] the given job
|
170
|
+
def run_after_discard_callbacks(job)
|
171
|
+
# Copied from ActiveJob::Exceptions#run_after_discard_procs
|
172
|
+
exceptions = []
|
173
|
+
job.after_discard_procs.each do |blk|
|
174
|
+
instance_exec(job, nil, &blk)
|
175
|
+
rescue StandardError => e
|
176
|
+
exceptions << e
|
177
|
+
end
|
178
|
+
raise exceptions.last if exceptions.any?
|
179
|
+
|
180
|
+
nil
|
181
|
+
end
|
36
182
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: marj
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 6.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Nick Dower
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-
|
11
|
+
date: 2024-02-05 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -48,17 +48,16 @@ files:
|
|
48
48
|
- LICENSE.txt
|
49
49
|
- README.md
|
50
50
|
- lib/marj.rb
|
51
|
-
- lib/marj/
|
51
|
+
- lib/marj/mission_control.rb
|
52
52
|
- lib/marj/record.rb
|
53
|
-
- lib/marj/relation.rb
|
54
53
|
- lib/marj_adapter.rb
|
55
54
|
homepage: https://github.com/nicholasdower/marj
|
56
55
|
licenses:
|
57
56
|
- MIT
|
58
57
|
metadata:
|
59
58
|
bug_tracker_uri: https://github.com/nicholasdower/marj/issues
|
60
|
-
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/
|
61
|
-
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/
|
59
|
+
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v6.0.0
|
60
|
+
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v6.0.0
|
62
61
|
homepage_uri: https://github.com/nicholasdower/marj
|
63
62
|
rubygems_mfa_required: 'true'
|
64
63
|
source_code_uri: https://github.com/nicholasdower/marj
|
data/lib/marj/jobs_interface.rb
DELETED
@@ -1,113 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
module Marj
|
4
|
-
# The interface provided by {Marj} and {Marj::Relation}.
|
5
|
-
#
|
6
|
-
# To create a custom jobs interface, for example for all job classes in your application:
|
7
|
-
# class ApplicationJob < ActiveJob::Base
|
8
|
-
# extend Marj::JobsInterface
|
9
|
-
#
|
10
|
-
# def self.all
|
11
|
-
# Marj::Relation.new(self == ApplicationJob ? Marj::Record.ordered : Marj::Record.where(job_class: self))
|
12
|
-
# end
|
13
|
-
# end
|
14
|
-
#
|
15
|
-
# class SomeJob < ApplicationJob
|
16
|
-
# def perform(msg)
|
17
|
-
# puts msg
|
18
|
-
# end
|
19
|
-
# end
|
20
|
-
#
|
21
|
-
# This will allow you to query jobs via the +ApplicationJob+ class:
|
22
|
-
# ApplicationJob.next # Returns the next job of any type
|
23
|
-
#
|
24
|
-
# Or to query jobs via a specific job class:
|
25
|
-
# SomeJob.next # Returns the next SomeJob
|
26
|
-
#
|
27
|
-
# Alternatively, to create a jobs interface for a single job class:
|
28
|
-
# class SomeJob < ActiveJob::Base
|
29
|
-
# extend Marj::JobsInterface
|
30
|
-
#
|
31
|
-
# def self.all
|
32
|
-
# Marj::Relation.new(Marj::Record.where(job_class: self).ordered)
|
33
|
-
# end
|
34
|
-
# end
|
35
|
-
module JobsInterface
|
36
|
-
def self.included(clazz)
|
37
|
-
return if clazz == Marj::Relation
|
38
|
-
|
39
|
-
clazz.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
|
40
|
-
end
|
41
|
-
private_class_method :included
|
42
|
-
|
43
|
-
def self.extended(clazz)
|
44
|
-
clazz.singleton_class.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
|
45
|
-
end
|
46
|
-
private_class_method :extended
|
47
|
-
|
48
|
-
# Returns a {Marj::Relation} for jobs in the specified queue(s).
|
49
|
-
#
|
50
|
-
# @param queue [String, Symbol] the queue to query
|
51
|
-
# @param queues [Array<String>, Array<Symbol>] more queues to query
|
52
|
-
# @return [Marj::Relation]
|
53
|
-
def queue(queue, *queues)
|
54
|
-
Marj::Relation.new(all.where(queue_name: queues.dup.unshift(queue)))
|
55
|
-
end
|
56
|
-
|
57
|
-
# Returns the next job or the next N jobs if +limit+ is specified. If no jobs exist, returns +nil+.
|
58
|
-
#
|
59
|
-
# @param limit [Integer, NilClass]
|
60
|
-
# @return [ActiveJob::Base, NilClass]
|
61
|
-
def next(limit = nil)
|
62
|
-
all.first(limit)&.then { _1.is_a?(Array) ? _1.map(&:as_job) : _1.as_job }
|
63
|
-
end
|
64
|
-
|
65
|
-
# Returns a count of jobs, optionally either matching the specified column name criteria or where the specified
|
66
|
-
# block returns +true+.
|
67
|
-
#
|
68
|
-
# @param column_name [String, Symbol, NilClass]
|
69
|
-
# @param block [Proc, NilClass]
|
70
|
-
# @return [Integer]
|
71
|
-
def count(column_name = nil, &block)
|
72
|
-
block_given? ? all.count(column_name) { |r| block.call(r.as_job) } : all.count(column_name)
|
73
|
-
end
|
74
|
-
|
75
|
-
# Returns a {Marj::Relation} for jobs matching the specified criteria.
|
76
|
-
#
|
77
|
-
# @param args [Array]
|
78
|
-
# @return [Marj::Relation]
|
79
|
-
def where(*args)
|
80
|
-
Marj::Relation.new(all.where(*args))
|
81
|
-
end
|
82
|
-
|
83
|
-
# Returns a {Marj::Relation} for enqueued jobs with a +scheduled_at+ that is either +null+ or in the past.
|
84
|
-
#
|
85
|
-
# @return [Marj::Relation]
|
86
|
-
def due
|
87
|
-
Marj::Relation.new(all.due)
|
88
|
-
end
|
89
|
-
|
90
|
-
# Calls +perform_now+ on each job.
|
91
|
-
#
|
92
|
-
# @param batch_size [Integer, NilClass] the number of jobs to fetch at a time, or +nil+ to fetch all jobs at once
|
93
|
-
# @return [Array] the results returned by each job
|
94
|
-
def perform_all(batch_size: nil)
|
95
|
-
if batch_size
|
96
|
-
[].tap do |results|
|
97
|
-
while (jobs = all.limit(batch_size).map(&:as_job)).any?
|
98
|
-
results.concat(jobs.map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } })
|
99
|
-
end
|
100
|
-
end
|
101
|
-
else
|
102
|
-
all.map(&:as_job).map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } }
|
103
|
-
end
|
104
|
-
end
|
105
|
-
|
106
|
-
# Discards all jobs.
|
107
|
-
#
|
108
|
-
# @return [Numeric] the number of discarded jobs
|
109
|
-
def discard_all
|
110
|
-
all.delete_all
|
111
|
-
end
|
112
|
-
end
|
113
|
-
end
|
data/lib/marj/relation.rb
DELETED
@@ -1,72 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require_relative 'jobs_interface'
|
4
|
-
|
5
|
-
module Marj
|
6
|
-
# Returned by {Marj::JobsInterface} query methods to enable chaining and +Enumerable+ methods.
|
7
|
-
class Relation
|
8
|
-
include Enumerable
|
9
|
-
include Marj::JobsInterface
|
10
|
-
|
11
|
-
attr_reader :all
|
12
|
-
private :all
|
13
|
-
|
14
|
-
# Returns a {Marj::Relation} which wraps the specified +ActiveRecord+ relation.
|
15
|
-
def initialize(ar_relation)
|
16
|
-
@all = ar_relation
|
17
|
-
end
|
18
|
-
|
19
|
-
# Yields each job in this relation.
|
20
|
-
#
|
21
|
-
# @param block [Proc]
|
22
|
-
# @return [Array] the jobs in this relation
|
23
|
-
def each(&block)
|
24
|
-
all.map(&:as_job).each(&block)
|
25
|
-
end
|
26
|
-
|
27
|
-
# Provides +pretty_inspect+ output containing arrays of jobs rather than arrays of records, similar to the output
|
28
|
-
# produced when calling +pretty_inspect+ on +ActiveRecord::Relation+.
|
29
|
-
#
|
30
|
-
# Instead of the default +pretty_inspect+ output:
|
31
|
-
# > Marj.all
|
32
|
-
# =>
|
33
|
-
# #<Marj::Relation:0x000000012728bd88
|
34
|
-
# @ar_relation=
|
35
|
-
# [#<Marj::Record:0x0000000126c42080
|
36
|
-
# job_id: "1382cb98-c518-46ca-a0cc-d831e11a0714",
|
37
|
-
# job_class: TestJob,
|
38
|
-
# arguments: ["foo"],
|
39
|
-
# queue_name: "default",
|
40
|
-
# priority: nil,
|
41
|
-
# executions: 0,
|
42
|
-
# exception_executions: {},
|
43
|
-
# enqueued_at: 2024-01-25 15:31:06.115773 UTC,
|
44
|
-
# scheduled_at: nil,
|
45
|
-
# locale: "en",
|
46
|
-
# timezone: "UTC">]>
|
47
|
-
#
|
48
|
-
# Produces:
|
49
|
-
# > Marj.all
|
50
|
-
# =>
|
51
|
-
# [#<TestJob:0x000000010b63cef8
|
52
|
-
# @_scheduled_at_time=nil,
|
53
|
-
# @arguments=[],
|
54
|
-
# @enqueued_at=2024-01-25 15:31:06 UTC,
|
55
|
-
# @exception_executions={},
|
56
|
-
# @executions=0,
|
57
|
-
# @job_id="1382cb98-c518-46ca-a0cc-d831e11a0714",
|
58
|
-
# @locale="en",
|
59
|
-
# @priority=nil,
|
60
|
-
# @provider_job_id=nil,
|
61
|
-
# @queue_name="default",
|
62
|
-
# @scheduled_at=nil,
|
63
|
-
# @serialized_arguments=["foo"],
|
64
|
-
# @timezone="UTC">]
|
65
|
-
#
|
66
|
-
# @param pp [PP]
|
67
|
-
# @return [NilClass]
|
68
|
-
def pretty_print(pp)
|
69
|
-
pp.pp(to_a)
|
70
|
-
end
|
71
|
-
end
|
72
|
-
end
|