marj 2.1.0 → 4.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +183 -86
- data/lib/marj/jobs_interface.rb +113 -0
- data/lib/marj/record.rb +82 -0
- data/lib/marj/relation.rb +72 -0
- data/lib/marj.rb +151 -3
- data/lib/marj_adapter.rb +15 -2
- metadata +8 -7
- data/lib/marj_config.rb +0 -15
- data/lib/marj_record.rb +0 -157
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 4eb6129fa4948ed6e108469f44f380cda908620c6d08392cfa1e8c3bc84a9875
|
4
|
+
data.tar.gz: 28faed9a10fa3a8c885b521d6c3465196f814e97af07018d5eea88e580f322a9
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5f984f366a9cefb3187cbac7129144c8241d46d00357d719ec442dfce62d5c833c95ec30363a5af867e111499cc91cd2c60768da55d04c9c7fc401cce8ac3f83
|
7
|
+
data.tar.gz: e86029649f11ca05f96efcae59c1c60212b29335d9b154c3288a15aa81cd432b46f960166680b5bb0a534346852ccc73970cbaf8a298eb52893c37e260854f61
|
data/README.md
CHANGED
@@ -1,10 +1,10 @@
|
|
1
1
|
# Marj - Minimal ActiveRecord Jobs
|
2
2
|
|
3
|
-
|
3
|
+
A minimal database-backed ActiveJob queueing backend.
|
4
4
|
|
5
5
|
## Quick Links
|
6
6
|
|
7
|
-
API docs: https://
|
7
|
+
API docs: https://gemdocs.org/gems/marj/latest <br>
|
8
8
|
RubyGems: https://rubygems.org/gems/marj <br>
|
9
9
|
Changelog: https://github.com/nicholasdower/marj/releases <br>
|
10
10
|
Issues: https://github.com/nicholasdower/marj/issues <br>
|
@@ -16,6 +16,8 @@ Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
|
|
16
16
|
- Successfully executed jobs are deleted from the database.
|
17
17
|
- Failed jobs which should be retried are updated in the database.
|
18
18
|
- Failed jobs which should not be retried are deleted from the database.
|
19
|
+
- An interface is provided to retrieve, execute, discard and re-enqueue jobs.
|
20
|
+
- An `ActiveRecord` class is provided to query the database directly.
|
19
21
|
|
20
22
|
## Features Not Provided
|
21
23
|
|
@@ -24,35 +26,17 @@ Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
|
|
24
26
|
- Concurrency Controls
|
25
27
|
- Observability
|
26
28
|
- A User Interace
|
27
|
-
- Anything else you might dream up.
|
28
|
-
|
29
|
-
## Interface
|
30
|
-
|
31
|
-
- `Marj` - An ActiveRecord model class
|
32
|
-
- `Marj.ready` - Used to retrieve jobs ready to be executed
|
33
|
-
- `Marj#execute` - Used to execute jobs retrieved from the database
|
34
|
-
- `MarjConfig.table_name=` - Used to override the default table name
|
35
29
|
|
36
30
|
## Setup
|
37
31
|
|
38
32
|
### 1. Install
|
39
33
|
|
40
34
|
```shell
|
41
|
-
bundle add marj
|
35
|
+
bundle add activejob activerecord marj
|
42
36
|
|
43
37
|
# or
|
44
38
|
|
45
|
-
gem install marj
|
46
|
-
```
|
47
|
-
|
48
|
-
### 2. Configure
|
49
|
-
|
50
|
-
By default, the database table is named "jobs". To use a different table name:
|
51
|
-
|
52
|
-
```ruby
|
53
|
-
require 'marj'
|
54
|
-
|
55
|
-
MarjConfig.table_name = 'some_name'
|
39
|
+
gem install activejob activerecord marj
|
56
40
|
```
|
57
41
|
|
58
42
|
### 3. Create the database table
|
@@ -84,23 +68,17 @@ class CreateJobs < ActiveRecord::Migration[7.1]
|
|
84
68
|
end
|
85
69
|
```
|
86
70
|
|
71
|
+
Note that by default, Marj uses a table named `jobs`. To override the default
|
72
|
+
table name, set `Marj.table_name` before loading `ActiveRecord`.
|
73
|
+
|
87
74
|
### 4. Configure the queue adapter
|
88
75
|
|
89
76
|
```ruby
|
90
77
|
require 'marj'
|
91
78
|
|
92
|
-
#
|
93
|
-
|
94
|
-
|
95
|
-
end
|
96
|
-
|
97
|
-
# Without Rails:
|
98
|
-
ActiveJob::Base.queue_adapter = :marj
|
99
|
-
|
100
|
-
# Or for specific jobs (with or without Rails):
|
101
|
-
class SomeJob < ActiveJob::Base
|
102
|
-
self.queue_adapter = :marj
|
103
|
-
end
|
79
|
+
Rails.configuration.active_job.queue_adapter = :marj # Globally, with Rails
|
80
|
+
ActiveJob::Base.queue_adapter = :marj # Globally, without Rails
|
81
|
+
SomeJob.queue_adapter = :marj # Single job
|
104
82
|
```
|
105
83
|
|
106
84
|
## Example Usage
|
@@ -110,35 +88,139 @@ end
|
|
110
88
|
job = SomeJob.perform_later('foo')
|
111
89
|
job.perform_now
|
112
90
|
|
113
|
-
#
|
114
|
-
|
115
|
-
Marj.first.execute
|
91
|
+
# Retrieve and execute a job
|
92
|
+
Marj.due.next.perform_now
|
116
93
|
|
117
|
-
# Run all
|
118
|
-
Marj.
|
94
|
+
# Run all due jobs (single DB query)
|
95
|
+
Marj.due.perform_all
|
119
96
|
|
120
|
-
# Run all
|
121
|
-
|
97
|
+
# Run all due jobs (multiple DB queries)
|
98
|
+
Marj.due.perform_all(batch_size: 1)
|
122
99
|
|
123
|
-
# Run all
|
124
|
-
|
100
|
+
# Run all due jobs in a specific queue:
|
101
|
+
Marj.queue('foo').due.perform_all
|
125
102
|
|
126
|
-
# Run jobs as they become
|
103
|
+
# Run jobs as they become due:
|
127
104
|
loop do
|
128
|
-
|
129
|
-
rescue Exception => e
|
130
|
-
logger.error(e)
|
105
|
+
Marj.due.perform_all rescue logger.error($!)
|
131
106
|
ensure
|
132
107
|
sleep 5.seconds
|
133
108
|
end
|
134
109
|
```
|
135
110
|
|
111
|
+
# Jobs Interface
|
112
|
+
|
113
|
+
The `Marj` module provides methods for interacting with enqueued jobs. These
|
114
|
+
methods accept, return and yield +ActiveJob+ objects rather than +ActiveRecord+
|
115
|
+
objects. Returned jobs are orderd by due date. To query the database directly,
|
116
|
+
use `Marj::Record`.
|
117
|
+
|
118
|
+
Example usage:
|
119
|
+
|
120
|
+
```ruby
|
121
|
+
Marj.all # Returns all enqueued jobs.
|
122
|
+
Marj.queue # Returns jobs in the specified queue(s).
|
123
|
+
Marj.due # Returns jobs which are due to be executed.
|
124
|
+
Marj.next # Returns the next job(s) to be executed.
|
125
|
+
Marj.count # Returns the number of enqueued jobs.
|
126
|
+
Marj.where # Returns jobs matching the specified criteria.
|
127
|
+
Marj.perform_all # Executes all jobs.
|
128
|
+
Marj.discard_all # Discards all jobs.
|
129
|
+
Marj.discard # Discards the specified job.
|
130
|
+
```
|
131
|
+
|
132
|
+
Query methods can also be chained:
|
133
|
+
|
134
|
+
```ruby
|
135
|
+
Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
|
136
|
+
```
|
137
|
+
|
138
|
+
# Custom Jobs Interface
|
139
|
+
|
140
|
+
The `Marj::JobsInterface` can be added to any class or module. For example, to
|
141
|
+
add it to all jobs classes:
|
142
|
+
|
143
|
+
```ruby
|
144
|
+
class ApplicationJob < ActiveJob::Base
|
145
|
+
extend Marj::JobsInterface
|
146
|
+
|
147
|
+
def self.all
|
148
|
+
Marj::Relation.new(
|
149
|
+
self == ApplicationJob ?
|
150
|
+
Marj::Record.ordered : Marj::Record.where(job_class: self)
|
151
|
+
)
|
152
|
+
end
|
153
|
+
end
|
154
|
+
|
155
|
+
class SomeJob < ApplicationJob; end
|
156
|
+
|
157
|
+
ApplicationJob.due # Returns all jobs which are due to be executed.
|
158
|
+
SomeJob.due # Returns SomeJobs which are due to be executed.
|
159
|
+
```
|
160
|
+
|
161
|
+
## Customization
|
162
|
+
|
163
|
+
It is possible to create a custom record class and jobs interface. This enables,
|
164
|
+
for instance, writing jobs to multiple databases/tables within a single
|
165
|
+
application.
|
166
|
+
|
167
|
+
```ruby
|
168
|
+
class CreateMyJobs < ActiveRecord::Migration[7.1]
|
169
|
+
def self.up
|
170
|
+
create_table :my_jobs, id: :string, primary_key: :job_id do |table|
|
171
|
+
table.string :job_class, null: false
|
172
|
+
table.text :arguments, null: false
|
173
|
+
table.string :queue_name, null: false
|
174
|
+
table.integer :priority
|
175
|
+
table.integer :executions, null: false
|
176
|
+
table.text :exception_executions, null: false
|
177
|
+
table.datetime :enqueued_at, null: false
|
178
|
+
table.datetime :scheduled_at
|
179
|
+
table.string :locale, null: false
|
180
|
+
table.string :timezone, null: false
|
181
|
+
end
|
182
|
+
|
183
|
+
add_index :my_jobs, %i[enqueued_at]
|
184
|
+
add_index :my_jobs, %i[scheduled_at]
|
185
|
+
add_index :my_jobs, %i[priority scheduled_at enqueued_at]
|
186
|
+
end
|
187
|
+
|
188
|
+
def self.down
|
189
|
+
drop_table :my_jobs
|
190
|
+
end
|
191
|
+
end
|
192
|
+
|
193
|
+
class MyRecord < Marj::Record
|
194
|
+
self.table_name = 'my_jobs'
|
195
|
+
end
|
196
|
+
|
197
|
+
CreateMyJobs.migrate(:up)
|
198
|
+
|
199
|
+
class MyJob < ActiveJob::Base
|
200
|
+
self.queue_adapter = MarjAdapter.new('MyRecord')
|
201
|
+
|
202
|
+
extend Marj::JobsInterface
|
203
|
+
|
204
|
+
def self.all
|
205
|
+
Marj::Relation.new(MyRecord.all)
|
206
|
+
end
|
207
|
+
|
208
|
+
def perform(msg)
|
209
|
+
puts msg
|
210
|
+
end
|
211
|
+
end
|
212
|
+
|
213
|
+
MyJob.perform_later('oh, hi')
|
214
|
+
MyJob.due.next.perform_now
|
215
|
+
```
|
216
|
+
|
136
217
|
## Testing
|
137
218
|
|
138
|
-
By default, jobs enqeued during tests will be written to the database. Enqueued
|
219
|
+
By default, jobs enqeued during tests will be written to the database. Enqueued
|
220
|
+
jobs can be executed via:
|
139
221
|
|
140
222
|
```ruby
|
141
|
-
Marj.
|
223
|
+
Marj.due.perform_all
|
142
224
|
```
|
143
225
|
|
144
226
|
Alternatively, to use [ActiveJob::QueueAdapters::TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
|
@@ -158,7 +240,9 @@ class ApplicationJob < ActiveJob::Base
|
|
158
240
|
|
159
241
|
around_perform do |job, block|
|
160
242
|
if (timeout = job.class.instance_variable_get(:@timeout))
|
161
|
-
::Timeout.timeout(timeout, StandardError, 'execution expired')
|
243
|
+
::Timeout.timeout(timeout, StandardError, 'execution expired') do
|
244
|
+
block.call
|
245
|
+
end
|
162
246
|
else
|
163
247
|
block.call
|
164
248
|
end
|
@@ -185,7 +269,9 @@ class ApplicationJob < ActiveJob::Base
|
|
185
269
|
def last_error=(error)
|
186
270
|
if error.is_a?(Exception)
|
187
271
|
backtrace = error.backtrace&.map { |line| "\t#{line}" }&.join("\n")
|
188
|
-
error = backtrace ?
|
272
|
+
error = backtrace ?
|
273
|
+
"#{error.class}: #{error.message}\n#{backtrace}" :
|
274
|
+
"#{error.class}: #{error.message}"
|
189
275
|
end
|
190
276
|
|
191
277
|
@last_error = error&.truncate(10_000, omission: '… (truncated)')
|
@@ -217,10 +303,8 @@ For more information on ActiveJob, see:
|
|
217
303
|
|
218
304
|
```ruby
|
219
305
|
# With Rails
|
220
|
-
|
221
|
-
|
222
|
-
config.active_job.queue_adapter = FooAdapter.new # Uses FooAdapter directly
|
223
|
-
end
|
306
|
+
Rails.configuration.active_job.queue_adapter = :foo # Instantiates FooAdapter
|
307
|
+
Rails.configuration.active_job.queue_adapter = FooAdapter.new
|
224
308
|
|
225
309
|
# Without Rails
|
226
310
|
ActiveJob::Base.queue_adapter = :foo # Instantiates FooAdapter
|
@@ -233,40 +317,48 @@ SomeJob.queue_adapter = FooAdapter.new # Uses FooAdapter directly
|
|
233
317
|
|
234
318
|
### Configuration
|
235
319
|
|
236
|
-
|
237
|
-
|
238
|
-
|
239
|
-
|
240
|
-
|
241
|
-
|
242
|
-
|
243
|
-
|
244
|
-
|
320
|
+
```ruby
|
321
|
+
config.active_job.default_queue_name
|
322
|
+
config.active_job.queue_name_prefix
|
323
|
+
config.active_job.queue_name_delimiter
|
324
|
+
config.active_job.retry_jitter
|
325
|
+
SomeJob.queue_name
|
326
|
+
SomeJob.queue_as
|
327
|
+
SomeJob.queue_name_prefix
|
328
|
+
SomeJob.queue_name_delimiter
|
329
|
+
SomeJob.retry_jitter
|
330
|
+
```
|
245
331
|
|
246
332
|
### Options
|
247
333
|
|
248
|
-
|
249
|
-
|
250
|
-
|
251
|
-
|
334
|
+
```ruby
|
335
|
+
:wait # Enqueues the job with the specified delay
|
336
|
+
:wait_until # Enqueues the job at the time specified
|
337
|
+
:queue # Enqueues the job on the specified queue
|
338
|
+
:priority # Enqueues the job with the specified priority
|
339
|
+
```
|
252
340
|
|
253
341
|
### Callbacks
|
254
342
|
|
255
|
-
|
256
|
-
|
257
|
-
|
258
|
-
|
259
|
-
|
260
|
-
|
261
|
-
|
262
|
-
|
263
|
-
|
343
|
+
```ruby
|
344
|
+
SomeJob.before_enqueue
|
345
|
+
SomeJob.after_enqueue
|
346
|
+
SomeJob.around_enqueue
|
347
|
+
SomeJob.before_perform
|
348
|
+
SomeJob.after_perform
|
349
|
+
SomeJob.around_perform
|
350
|
+
ActiveJob::Callbacks.singleton_class.set_callback(:execute, :before, &block)
|
351
|
+
ActiveJob::Callbacks.singleton_class.set_callback(:execute, :after, &block)
|
352
|
+
ActiveJob::Callbacks.singleton_class.set_callback(:execute, :around, &block)
|
353
|
+
```
|
264
354
|
|
265
355
|
### Handling Exceptions
|
266
356
|
|
267
|
-
|
268
|
-
|
269
|
-
|
357
|
+
```ruby
|
358
|
+
SomeJob.retry_on
|
359
|
+
SomeJob.discard_on
|
360
|
+
SomeJob.after_discard
|
361
|
+
```
|
270
362
|
|
271
363
|
### Creating Jobs
|
272
364
|
|
@@ -274,35 +366,40 @@ SomeJob.queue_adapter = FooAdapter.new # Uses FooAdapter directly
|
|
274
366
|
# Create without enqueueing
|
275
367
|
job = SomeJob.new
|
276
368
|
job = SomeJob.new(args)
|
369
|
+
job = SomeJob.new.deserialize(other_job.serialize)
|
277
370
|
|
278
371
|
# Create and enqueue
|
279
372
|
job = SomeJob.perform_later
|
280
373
|
job = SomeJob.perform_later(args)
|
281
374
|
|
282
|
-
# Create and run (enqueued on failure)
|
375
|
+
# Create without enqueueing and run (only enqueued on failure if retryable)
|
283
376
|
SomeJob.perform_now
|
284
377
|
SomeJob.perform_now(args)
|
285
378
|
```
|
286
379
|
|
287
380
|
### Enqueueing Jobs
|
288
381
|
|
382
|
+
Jobs are enqueued via the `ActiveJob::Base#enqueue` method. This method returns
|
383
|
+
the job on success. If an error is raised during enqueueing, that error will
|
384
|
+
propagate to the caller, unless the error is an `ActiveJob::EnqueueError`. In
|
385
|
+
this case, `enqueue` will return `false` and `job.enqueue_error` will be set.
|
386
|
+
|
289
387
|
```ruby
|
290
388
|
SomeJob.new(args).enqueue
|
291
389
|
SomeJob.new(args).enqueue(options)
|
292
390
|
|
391
|
+
# Via perform_later
|
293
392
|
SomeJob.perform_later(SomeJob.new(args))
|
294
|
-
|
295
393
|
SomeJob.perform_later(args)
|
296
394
|
SomeJob.set(options).perform_later(args)
|
297
395
|
|
298
|
-
#
|
396
|
+
# After a failure during execution
|
299
397
|
SomeJob.perform_now(args)
|
398
|
+
ActiveJob::Base.execute(SomeJob.new(args).serialize)
|
300
399
|
|
301
400
|
# Enqueue multiple
|
302
401
|
ActiveJob.perform_all_later(SomeJob.new, SomeJob.new)
|
303
402
|
ActiveJob.perform_all_later(SomeJob.new, SomeJob.new, options:)
|
304
|
-
|
305
|
-
# Enqueue multiple
|
306
403
|
SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new)
|
307
404
|
SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new, options:)
|
308
405
|
```
|
@@ -310,7 +407,7 @@ SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new, options:)
|
|
310
407
|
### Executing Jobs
|
311
408
|
|
312
409
|
```ruby
|
313
|
-
# Executed without enqueueing, enqueued on failure if
|
410
|
+
# Executed without enqueueing, enqueued on failure if retryable
|
314
411
|
SomeJob.new(args).perform_now
|
315
412
|
SomeJob.perform_now(args)
|
316
413
|
ActiveJob::Base.execute(SomeJob.new(args).serialize)
|
@@ -0,0 +1,113 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Marj
|
4
|
+
# The interface provided by {Marj} and {Marj::Relation}.
|
5
|
+
#
|
6
|
+
# To create a custom jobs interface, for example for all job classes in your application:
|
7
|
+
# class ApplicationJob < ActiveJob::Base
|
8
|
+
# extend Marj::JobsInterface
|
9
|
+
#
|
10
|
+
# def self.all
|
11
|
+
# Marj::Relation.new(self == ApplicationJob ? Marj::Record.ordered : Marj::Record.where(job_class: self))
|
12
|
+
# end
|
13
|
+
# end
|
14
|
+
#
|
15
|
+
# class SomeJob < ApplicationJob
|
16
|
+
# def perform(msg)
|
17
|
+
# puts msg
|
18
|
+
# end
|
19
|
+
# end
|
20
|
+
#
|
21
|
+
# This will allow you to query jobs via the +ApplicationJob+ class:
|
22
|
+
# ApplicationJob.next # Returns the next job of any type
|
23
|
+
#
|
24
|
+
# Or to query jobs via a specific job class:
|
25
|
+
# SomeJob.next # Returns the next SomeJob
|
26
|
+
#
|
27
|
+
# Alternatively, to create a jobs interface for a single job class:
|
28
|
+
# class SomeJob < ActiveJob::Base
|
29
|
+
# extend Marj::JobsInterface
|
30
|
+
#
|
31
|
+
# def self.all
|
32
|
+
# Marj::Relation.new(Marj::Record.where(job_class: self).ordered)
|
33
|
+
# end
|
34
|
+
# end
|
35
|
+
module JobsInterface
|
36
|
+
def self.included(clazz)
|
37
|
+
return if clazz == Marj::Relation
|
38
|
+
|
39
|
+
clazz.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
|
40
|
+
end
|
41
|
+
private_class_method :included
|
42
|
+
|
43
|
+
def self.extended(clazz)
|
44
|
+
clazz.singleton_class.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
|
45
|
+
end
|
46
|
+
private_class_method :extended
|
47
|
+
|
48
|
+
# Returns a {Marj::Relation} for jobs in the specified queue(s).
|
49
|
+
#
|
50
|
+
# @param queue [String, Symbol] the queue to query
|
51
|
+
# @param queues [Array<String>, Array<Symbol>] more queues to query
|
52
|
+
# @return [Marj::Relation]
|
53
|
+
def queue(queue, *queues)
|
54
|
+
Marj::Relation.new(all.where(queue_name: queues.dup.unshift(queue)))
|
55
|
+
end
|
56
|
+
|
57
|
+
# Returns the next job or the next N jobs if +limit+ is specified. If no jobs exist, returns +nil+.
|
58
|
+
#
|
59
|
+
# @param limit [Integer, NilClass]
|
60
|
+
# @return [ActiveJob::Base, NilClass]
|
61
|
+
def next(limit = nil)
|
62
|
+
all.first(limit)&.then { _1.is_a?(Array) ? _1.map(&:as_job) : _1.as_job }
|
63
|
+
end
|
64
|
+
|
65
|
+
# Returns a count of jobs, optionally either matching the specified column name criteria or where the specified
|
66
|
+
# block returns +true+.
|
67
|
+
#
|
68
|
+
# @param column_name [String, Symbol, NilClass]
|
69
|
+
# @param block [Proc, NilClass]
|
70
|
+
# @return [Integer]
|
71
|
+
def count(column_name = nil, &block)
|
72
|
+
block_given? ? all.count(column_name) { |r| block.call(r.as_job) } : all.count(column_name)
|
73
|
+
end
|
74
|
+
|
75
|
+
# Returns a {Marj::Relation} for jobs matching the specified criteria.
|
76
|
+
#
|
77
|
+
# @param args [Array]
|
78
|
+
# @return [Marj::Relation]
|
79
|
+
def where(*args)
|
80
|
+
Marj::Relation.new(all.where(*args))
|
81
|
+
end
|
82
|
+
|
83
|
+
# Returns a {Marj::Relation} for enqueued jobs with a +scheduled_at+ that is either +null+ or in the past.
|
84
|
+
#
|
85
|
+
# @return [Marj::Relation]
|
86
|
+
def due
|
87
|
+
Marj::Relation.new(all.due)
|
88
|
+
end
|
89
|
+
|
90
|
+
# Calls +perform_now+ on each job.
|
91
|
+
#
|
92
|
+
# @param batch_size [Integer, NilClass] the number of jobs to fetch at a time, or +nil+ to fetch all jobs at once
|
93
|
+
# @return [Array] the results returned by each job
|
94
|
+
def perform_all(batch_size: nil)
|
95
|
+
if batch_size
|
96
|
+
[].tap do |results|
|
97
|
+
while (jobs = all.limit(batch_size).map(&:as_job)).any?
|
98
|
+
results.concat(jobs.map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } })
|
99
|
+
end
|
100
|
+
end
|
101
|
+
else
|
102
|
+
all.map(&:as_job).map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } }
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
106
|
+
# Discards all jobs.
|
107
|
+
#
|
108
|
+
# @return [Numeric] the number of discarded jobs
|
109
|
+
def discard_all
|
110
|
+
all.delete_all
|
111
|
+
end
|
112
|
+
end
|
113
|
+
end
|
data/lib/marj/record.rb
ADDED
@@ -0,0 +1,82 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'active_job'
|
4
|
+
require 'active_record'
|
5
|
+
|
6
|
+
module Marj
|
7
|
+
# The default +ActiveRecord+ class.
|
8
|
+
class Record < ActiveRecord::Base
|
9
|
+
self.table_name = Marj.table_name
|
10
|
+
|
11
|
+
# Order by +enqueued_at+ rather than +job_id+ (the default).
|
12
|
+
self.implicit_order_column = 'enqueued_at'
|
13
|
+
|
14
|
+
# Using a custom serializer for exception_executions so that we can interact with it as a hash rather than a
|
15
|
+
# string.
|
16
|
+
serialize(:exception_executions, coder: JSON)
|
17
|
+
|
18
|
+
# Using a custom serializer for arguments so that we can interact with as an array rather than a string.
|
19
|
+
# This enables code like:
|
20
|
+
# Marj::Record.next.arguments.first
|
21
|
+
# Marj::Record.next.update!(arguments: ['foo', 1, Time.now])
|
22
|
+
serialize(:arguments, coder: Class.new do
|
23
|
+
def self.dump(arguments)
|
24
|
+
return ActiveJob::Arguments.serialize(arguments).to_json if arguments.is_a?(Array)
|
25
|
+
return arguments if arguments.is_a?(String) || arguments.nil?
|
26
|
+
|
27
|
+
raise "invalid arguments: #{arguments}"
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.load(arguments)
|
31
|
+
arguments ? ActiveJob::Arguments.deserialize(JSON.parse(arguments)) : nil
|
32
|
+
end
|
33
|
+
end)
|
34
|
+
|
35
|
+
# Using a custom serializer for job_class so that we can interact with it as a class rather than a string.
|
36
|
+
serialize(:job_class, coder: Class.new do
|
37
|
+
def self.dump(clazz)
|
38
|
+
return clazz.name if clazz.is_a?(Class)
|
39
|
+
return clazz if clazz.is_a?(String) || clazz.nil?
|
40
|
+
|
41
|
+
raise "invalid class: #{clazz}"
|
42
|
+
end
|
43
|
+
|
44
|
+
def self.load(str)
|
45
|
+
str&.constantize
|
46
|
+
end
|
47
|
+
end)
|
48
|
+
|
49
|
+
# Returns a job object for this record which will update the database when successfully executed, enqueued or
|
50
|
+
# discarded.
|
51
|
+
#
|
52
|
+
# @return [ActiveJob::Base]
|
53
|
+
def as_job
|
54
|
+
Marj.send(:to_job, self)
|
55
|
+
end
|
56
|
+
|
57
|
+
class << self
|
58
|
+
# Returns an +ActiveRecord::Relation+ scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in
|
59
|
+
# the past.
|
60
|
+
#
|
61
|
+
# @return [ActiveRecord::Relation]
|
62
|
+
def due
|
63
|
+
where('scheduled_at IS NULL OR scheduled_at <= ?', Time.now.utc)
|
64
|
+
end
|
65
|
+
|
66
|
+
# Returns an +ActiveRecord::Relation+ scope for jobs ordered by +priority+ (+null+ last), then +scheduled_at+
|
67
|
+
# (+null+ last), then +enqueued_at+.
|
68
|
+
#
|
69
|
+
# @return [ActiveRecord::Relation]
|
70
|
+
def ordered
|
71
|
+
order(
|
72
|
+
Arel.sql(<<~SQL.squish, Time.now.utc)
|
73
|
+
CASE WHEN scheduled_at IS NULL OR scheduled_at <= ? THEN 0 ELSE 1 END,
|
74
|
+
CASE WHEN priority IS NULL THEN 1 ELSE 0 END, priority,
|
75
|
+
CASE WHEN scheduled_at IS NULL THEN 1 ELSE 0 END, scheduled_at,
|
76
|
+
enqueued_at
|
77
|
+
SQL
|
78
|
+
)
|
79
|
+
end
|
80
|
+
end
|
81
|
+
end
|
82
|
+
end
|
@@ -0,0 +1,72 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative 'jobs_interface'
|
4
|
+
|
5
|
+
module Marj
|
6
|
+
# Returned by {Marj::JobsInterface} query methods to enable chaining and +Enumerable+ methods.
|
7
|
+
class Relation
|
8
|
+
include Enumerable
|
9
|
+
include Marj::JobsInterface
|
10
|
+
|
11
|
+
attr_reader :all
|
12
|
+
private :all
|
13
|
+
|
14
|
+
# Returns a {Marj::Relation} which wraps the specified +ActiveRecord+ relation.
|
15
|
+
def initialize(ar_relation)
|
16
|
+
@all = ar_relation
|
17
|
+
end
|
18
|
+
|
19
|
+
# Yields each job in this relation.
|
20
|
+
#
|
21
|
+
# @param block [Proc]
|
22
|
+
# @return [Array] the jobs in this relation
|
23
|
+
def each(&block)
|
24
|
+
all.map(&:as_job).each(&block)
|
25
|
+
end
|
26
|
+
|
27
|
+
# Provides +pretty_inspect+ output containing arrays of jobs rather than arrays of records, similar to the output
|
28
|
+
# produced when calling +pretty_inspect+ on +ActiveRecord::Relation+.
|
29
|
+
#
|
30
|
+
# Instead of the default +pretty_inspect+ output:
|
31
|
+
# > Marj.all
|
32
|
+
# =>
|
33
|
+
# #<Marj::Relation:0x000000012728bd88
|
34
|
+
# @ar_relation=
|
35
|
+
# [#<Marj::Record:0x0000000126c42080
|
36
|
+
# job_id: "1382cb98-c518-46ca-a0cc-d831e11a0714",
|
37
|
+
# job_class: TestJob,
|
38
|
+
# arguments: ["foo"],
|
39
|
+
# queue_name: "default",
|
40
|
+
# priority: nil,
|
41
|
+
# executions: 0,
|
42
|
+
# exception_executions: {},
|
43
|
+
# enqueued_at: 2024-01-25 15:31:06.115773 UTC,
|
44
|
+
# scheduled_at: nil,
|
45
|
+
# locale: "en",
|
46
|
+
# timezone: "UTC">]>
|
47
|
+
#
|
48
|
+
# Produces:
|
49
|
+
# > Marj.all
|
50
|
+
# =>
|
51
|
+
# [#<TestJob:0x000000010b63cef8
|
52
|
+
# @_scheduled_at_time=nil,
|
53
|
+
# @arguments=[],
|
54
|
+
# @enqueued_at=2024-01-25 15:31:06 UTC,
|
55
|
+
# @exception_executions={},
|
56
|
+
# @executions=0,
|
57
|
+
# @job_id="1382cb98-c518-46ca-a0cc-d831e11a0714",
|
58
|
+
# @locale="en",
|
59
|
+
# @priority=nil,
|
60
|
+
# @provider_job_id=nil,
|
61
|
+
# @queue_name="default",
|
62
|
+
# @scheduled_at=nil,
|
63
|
+
# @serialized_arguments=["foo"],
|
64
|
+
# @timezone="UTC">]
|
65
|
+
#
|
66
|
+
# @param pp [PP]
|
67
|
+
# @return [NilClass]
|
68
|
+
def pretty_print(pp)
|
69
|
+
pp.pp(to_a)
|
70
|
+
end
|
71
|
+
end
|
72
|
+
end
|
data/lib/marj.rb
CHANGED
@@ -1,8 +1,156 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require_relative 'marj_adapter'
|
4
|
+
require_relative 'marj/jobs_interface'
|
5
|
+
require_relative 'marj/relation'
|
6
|
+
|
7
|
+
# A minimal database-backed ActiveJob queueing backend.
|
8
|
+
#
|
9
|
+
# The {Marj} module provides methods for interacting with enqueued jobs. These methods accept, return and yield
|
10
|
+
# +ActiveJob+ objects rather than +ActiveRecord+ objects. Returned jobs are ordered by due date. To query the database
|
11
|
+
# directly, use {Record}.
|
12
|
+
#
|
13
|
+
# Example usage:
|
14
|
+
# Marj.all # Returns all enqueued jobs.
|
15
|
+
# Marj.queue # Returns jobs in the specified queue(s).
|
16
|
+
# Marj.due # Returns jobs which are due to be executed.
|
17
|
+
# Marj.next # Returns the next job(s) to be executed.
|
18
|
+
# Marj.count # Returns the number of enqueued jobs.
|
19
|
+
# Marj.where # Returns jobs matching the specified criteria.
|
20
|
+
# Marj.perform_all # Executes all jobs.
|
21
|
+
# Marj.discard_all # Discards all jobs.
|
22
|
+
# Marj.discard # Discards the specified job.
|
23
|
+
#
|
24
|
+
# Query methods can also be chained:
|
25
|
+
# Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
|
26
|
+
#
|
27
|
+
# Note that by default, Marj uses {Marj::Record} to interact with the +jobs+ table. To use a different record class, set
|
28
|
+
# {record_class}. To simply override the table name, set {table_name} before loading +ActiveRecord+.
|
29
|
+
#
|
3
30
|
# See https://github.com/nicholasdower/marj
|
31
|
+
module Marj
|
32
|
+
# The Marj version.
|
33
|
+
VERSION = '4.0.0'
|
4
34
|
|
5
|
-
|
6
|
-
|
35
|
+
Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
|
36
|
+
|
37
|
+
@table_name = :jobs
|
38
|
+
@record_class = 'Marj::Record'
|
39
|
+
|
40
|
+
class << self
|
41
|
+
include Marj::JobsInterface
|
42
|
+
|
43
|
+
# @!attribute record_class
|
44
|
+
# The name of the +ActiveRecord+ class. Defaults to +Marj::Record+.
|
45
|
+
# @return [Class, String]
|
46
|
+
|
47
|
+
attr_writer :record_class
|
48
|
+
|
49
|
+
def record_class
|
50
|
+
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
51
|
+
end
|
52
|
+
|
53
|
+
# @!attribute table_name
|
54
|
+
# The name of the database table. Defaults to +:jobs+.
|
55
|
+
# @return [Symbol, String]
|
56
|
+
attr_accessor :table_name
|
57
|
+
|
58
|
+
# Returns a {Marj::Relation} for all jobs in the order they should be executed.
|
59
|
+
#
|
60
|
+
# @return [Marj::Relation]
|
61
|
+
def all
|
62
|
+
Marj::Relation.new(Marj.record_class.ordered)
|
63
|
+
end
|
64
|
+
|
65
|
+
# Discards the specified job.
|
66
|
+
#
|
67
|
+
# @return [Integer] the number of discarded jobs
|
68
|
+
def discard(job)
|
69
|
+
all.where(job_id: job.job_id).discard_all
|
70
|
+
end
|
71
|
+
|
72
|
+
private
|
73
|
+
|
74
|
+
# Creates a job instance for the given record which will update the database when successfully executed, enqueued or
|
75
|
+
# discarded.
|
76
|
+
#
|
77
|
+
# @param record [ActiveRecord::Base]
|
78
|
+
# @return [ActiveJob::Base] the new job instance
|
79
|
+
def to_job(record)
|
80
|
+
# See register_callbacks for details on how callbacks are used.
|
81
|
+
job = record.job_class.new.tap { register_callbacks(_1, record) }
|
82
|
+
|
83
|
+
# ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
|
84
|
+
# deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
|
85
|
+
job_data = record.attributes.merge('arguments' => JSON.parse(record.read_attribute_before_type_cast(:arguments)))
|
86
|
+
|
87
|
+
# ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
|
88
|
+
job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
|
89
|
+
|
90
|
+
job.tap { job.deserialize(job_data) }
|
91
|
+
end
|
92
|
+
|
93
|
+
# Registers callbacks for the given job which destroy the given database record when the job succeeds or is
|
94
|
+
# discarded.
|
95
|
+
#
|
96
|
+
# @param job [ActiveJob::Base]
|
97
|
+
# @param record [ActiveRecord::Base]
|
98
|
+
# @return [ActiveJob::Base]
|
99
|
+
def register_callbacks(job, record)
|
100
|
+
raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
|
101
|
+
|
102
|
+
# We need to detect three cases:
|
103
|
+
# - If a job succeeds, after_perform will be called.
|
104
|
+
# - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
|
105
|
+
# - If a job exceeds its max attempts, after_discard will be called.
|
106
|
+
job.singleton_class.after_perform { |_j| record.destroy! }
|
107
|
+
job.singleton_class.after_discard { |_j, _exception| record.destroy! }
|
108
|
+
job.singleton_class.instance_variable_set(:@record, record)
|
109
|
+
|
110
|
+
job
|
111
|
+
end
|
112
|
+
|
113
|
+
# Enqueue a job for execution at the specified time.
|
114
|
+
#
|
115
|
+
# @param job [ActiveJob::Base] the job to enqueue
|
116
|
+
# @param record_class [Class] the +ActiveRecord+ class
|
117
|
+
# @param time [Time, NilClass] optional time at which to execute the job
|
118
|
+
# @return [ActiveJob::Base] the enqueued job
|
119
|
+
def enqueue(job, record_class, time = nil)
|
120
|
+
job.scheduled_at = time
|
121
|
+
# Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
|
122
|
+
serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
|
7
123
|
|
8
|
-
|
124
|
+
# When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
|
125
|
+
# are registered on the job instance so that when the job is executed, the database record is deleted or updated
|
126
|
+
# (depending on the result).
|
127
|
+
#
|
128
|
+
# We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
|
129
|
+
# singleton class. This holds a reference to the record. This ensures that if execute is called on a record
|
130
|
+
# instance, any updates to the database are reflected on that record instance.
|
131
|
+
if (existing_record = job.singleton_class.instance_variable_get(:@record))
|
132
|
+
# This job instance has already been associated with a database row.
|
133
|
+
if record_class.exists?(job_id: job.job_id)
|
134
|
+
# The database row still exists, we simply need to update it.
|
135
|
+
existing_record.update!(serialized)
|
136
|
+
else
|
137
|
+
# Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
|
138
|
+
# want to register the new instance because someone might still have a reference to the existing one.
|
139
|
+
record_class.create!(serialized)
|
140
|
+
existing_record.reload
|
141
|
+
end
|
142
|
+
else
|
143
|
+
# This job instance has not been associated with a database row.
|
144
|
+
if (new_record = record_class.find_by(job_id: job.job_id))
|
145
|
+
# The database row already exists. Update it.
|
146
|
+
new_record.update!(serialized)
|
147
|
+
else
|
148
|
+
# The database row does not exist. Create it.
|
149
|
+
new_record = record_class.create!(serialized)
|
150
|
+
end
|
151
|
+
register_callbacks(job, new_record)
|
152
|
+
end
|
153
|
+
job
|
154
|
+
end
|
155
|
+
end
|
156
|
+
end
|
data/lib/marj_adapter.rb
CHANGED
@@ -4,12 +4,19 @@
|
|
4
4
|
#
|
5
5
|
# See https://github.com/nicholasdower/marj
|
6
6
|
class MarjAdapter
|
7
|
+
# Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ class.
|
8
|
+
#
|
9
|
+
# @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use to store jobs
|
10
|
+
def initialize(record_class = 'Marj::Record')
|
11
|
+
@record_class = record_class
|
12
|
+
end
|
13
|
+
|
7
14
|
# Enqueue a job for immediate execution.
|
8
15
|
#
|
9
16
|
# @param job [ActiveJob::Base] the job to enqueue
|
10
17
|
# @return [ActiveJob::Base] the enqueued job
|
11
18
|
def enqueue(job)
|
12
|
-
Marj.send(:enqueue, job)
|
19
|
+
Marj.send(:enqueue, job, record_class)
|
13
20
|
end
|
14
21
|
|
15
22
|
# Enqueue a job for execution at the specified time.
|
@@ -18,6 +25,12 @@ class MarjAdapter
|
|
18
25
|
# @param timestamp [Numeric, NilClass] optional number of seconds since Unix epoch at which to execute the job
|
19
26
|
# @return [ActiveJob::Base] the enqueued job
|
20
27
|
def enqueue_at(job, timestamp)
|
21
|
-
Marj.send(:enqueue, job, timestamp ? Time.at(timestamp).utc : nil)
|
28
|
+
Marj.send(:enqueue, job, record_class, timestamp ? Time.at(timestamp).utc : nil)
|
29
|
+
end
|
30
|
+
|
31
|
+
private
|
32
|
+
|
33
|
+
def record_class
|
34
|
+
@record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
|
22
35
|
end
|
23
36
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: marj
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 4.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Nick Dower
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-01-
|
11
|
+
date: 2024-01-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -38,7 +38,7 @@ dependencies:
|
|
38
38
|
- - ">="
|
39
39
|
- !ruby/object:Gem::Version
|
40
40
|
version: '7.1'
|
41
|
-
description: Marj (Minimal ActiveRecord Jobs) is
|
41
|
+
description: Marj (Minimal ActiveRecord Jobs) is a minimal database-backed ActiveJob
|
42
42
|
queueing backend.
|
43
43
|
email: nicholasdower@gmail.com
|
44
44
|
executables: []
|
@@ -48,16 +48,17 @@ files:
|
|
48
48
|
- LICENSE.txt
|
49
49
|
- README.md
|
50
50
|
- lib/marj.rb
|
51
|
+
- lib/marj/jobs_interface.rb
|
52
|
+
- lib/marj/record.rb
|
53
|
+
- lib/marj/relation.rb
|
51
54
|
- lib/marj_adapter.rb
|
52
|
-
- lib/marj_config.rb
|
53
|
-
- lib/marj_record.rb
|
54
55
|
homepage: https://github.com/nicholasdower/marj
|
55
56
|
licenses:
|
56
57
|
- MIT
|
57
58
|
metadata:
|
58
59
|
bug_tracker_uri: https://github.com/nicholasdower/marj/issues
|
59
|
-
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/
|
60
|
-
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/
|
60
|
+
changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v4.0.0
|
61
|
+
documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v4.0.0
|
61
62
|
homepage_uri: https://github.com/nicholasdower/marj
|
62
63
|
rubygems_mfa_required: 'true'
|
63
64
|
source_code_uri: https://github.com/nicholasdower/marj
|
data/lib/marj_config.rb
DELETED
@@ -1,15 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
# Marj configuration.
|
4
|
-
#
|
5
|
-
# See https://github.com/nicholasdower/marj
|
6
|
-
class MarjConfig
|
7
|
-
@table_name = 'jobs'
|
8
|
-
|
9
|
-
class << self
|
10
|
-
# The name of the database table. Defaults to "jobs".
|
11
|
-
#
|
12
|
-
# @return [String]
|
13
|
-
attr_accessor :table_name
|
14
|
-
end
|
15
|
-
end
|
data/lib/marj_record.rb
DELETED
@@ -1,157 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require 'active_job'
|
4
|
-
require 'active_record'
|
5
|
-
require_relative 'marj_config'
|
6
|
-
|
7
|
-
# The Marj ActiveRecord model class.
|
8
|
-
#
|
9
|
-
# See https://github.com/nicholasdower/marj
|
10
|
-
class Marj < ActiveRecord::Base
|
11
|
-
# The Marj version.
|
12
|
-
VERSION = '2.1.0'
|
13
|
-
|
14
|
-
# Executes the job associated with this record and returns the result.
|
15
|
-
def execute
|
16
|
-
# Normally we would call ActiveJob::Base#execute which has the following implementation:
|
17
|
-
# ActiveJob::Callbacks.run_callbacks(:execute) do
|
18
|
-
# job = deserialize(job_data)
|
19
|
-
# job.perform_now
|
20
|
-
# end
|
21
|
-
# However, we need to instantiate the job ourselves in order to register callbacks before execution.
|
22
|
-
ActiveJob::Callbacks.run_callbacks(:execute) do
|
23
|
-
# See register_callbacks for details on how callbacks are used.
|
24
|
-
job = job_class.new.tap { Marj.send(:register_callbacks, _1, self) }
|
25
|
-
|
26
|
-
# ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
|
27
|
-
# deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
|
28
|
-
job_data = attributes.merge('arguments' => JSON.parse(read_attribute_before_type_cast(:arguments)))
|
29
|
-
|
30
|
-
# ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
|
31
|
-
job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
|
32
|
-
|
33
|
-
job.deserialize(job_data)
|
34
|
-
job.perform_now
|
35
|
-
end
|
36
|
-
end
|
37
|
-
|
38
|
-
# Returns an ActiveRecord::Relation scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in the
|
39
|
-
# past. Jobs are ordered by +priority+ (+null+ last), then +scheduled_at+ (+null+ last), then +enqueued_at+.
|
40
|
-
#
|
41
|
-
# @return [ActiveRecord::Relation]
|
42
|
-
def self.ready
|
43
|
-
where('scheduled_at is null or scheduled_at <= ?', Time.now.utc).order(
|
44
|
-
Arel.sql(<<~SQL.squish)
|
45
|
-
CASE WHEN priority IS NULL THEN 1 ELSE 0 END, priority,
|
46
|
-
CASE WHEN scheduled_at IS NULL THEN 1 ELSE 0 END, scheduled_at,
|
47
|
-
enqueued_at
|
48
|
-
SQL
|
49
|
-
)
|
50
|
-
end
|
51
|
-
|
52
|
-
self.table_name = MarjConfig.table_name
|
53
|
-
|
54
|
-
# Order by +enqueued_at+ rather than +job_id+ (the default)
|
55
|
-
self.implicit_order_column = 'enqueued_at'
|
56
|
-
|
57
|
-
# Using a custom serializer for exception_executions so that we can interact with it as a hash rather than a string.
|
58
|
-
serialize(:exception_executions, coder: JSON)
|
59
|
-
|
60
|
-
# Using a custom serializer for arguments so that we can interact with as an array rather than a string.
|
61
|
-
# This enables code like:
|
62
|
-
# Marj.first.arguments.first
|
63
|
-
# Marj.first.update!(arguments: ['foo', 1, Time.now])
|
64
|
-
serialize(:arguments, coder: Class.new do
|
65
|
-
def self.dump(arguments)
|
66
|
-
return ActiveJob::Arguments.serialize(arguments).to_json if arguments.is_a?(Array)
|
67
|
-
return arguments if arguments.is_a?(String) || arguments.nil?
|
68
|
-
|
69
|
-
raise "invalid arguments: #{arguments}"
|
70
|
-
end
|
71
|
-
|
72
|
-
def self.load(arguments)
|
73
|
-
arguments ? ActiveJob::Arguments.deserialize(JSON.parse(arguments)) : nil
|
74
|
-
end
|
75
|
-
end)
|
76
|
-
|
77
|
-
# Using a custom serializer for job_class so that we can interact with it as a class rather than a string.
|
78
|
-
serialize(:job_class, coder: Class.new do
|
79
|
-
def self.dump(clazz)
|
80
|
-
return clazz.name if clazz.is_a?(Class)
|
81
|
-
return clazz if clazz.is_a?(String) || clazz.nil?
|
82
|
-
|
83
|
-
raise "invalid class: #{clazz}"
|
84
|
-
end
|
85
|
-
|
86
|
-
def self.load(str)
|
87
|
-
str&.constantize
|
88
|
-
end
|
89
|
-
end)
|
90
|
-
|
91
|
-
# Registers job callbacks used to keep the database record for the specified job in sync.
|
92
|
-
#
|
93
|
-
# @param job [ActiveJob::Base]
|
94
|
-
# @return [ActiveJob::Base]
|
95
|
-
def self.register_callbacks(job, record)
|
96
|
-
if job.singleton_class.instance_variable_get(:@__marj)
|
97
|
-
# Callbacks already registered. We just need to update the record.
|
98
|
-
job.singleton_class.instance_variable_set(:@__marj, record)
|
99
|
-
return
|
100
|
-
end
|
101
|
-
|
102
|
-
# We need to detect three cases:
|
103
|
-
# - If a job succeeds, after_perform will be called.
|
104
|
-
# - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
|
105
|
-
# - If a job exceeds its max attempts, after_discard will be called.
|
106
|
-
job.singleton_class.after_perform { |_j| job.singleton_class.instance_variable_get(:@__marj).destroy! }
|
107
|
-
job.singleton_class.after_discard { |_j, _exception| job.singleton_class.instance_variable_get(:@__marj).destroy! }
|
108
|
-
job.singleton_class.instance_variable_set(:@__marj, record)
|
109
|
-
|
110
|
-
job
|
111
|
-
end
|
112
|
-
private_class_method :register_callbacks
|
113
|
-
|
114
|
-
# Enqueue a job for execution at the specified time.
|
115
|
-
#
|
116
|
-
# @param job [ActiveJob::Base] the job to enqueue
|
117
|
-
# @param time [Time, NilClass] optional time at which to execute the job
|
118
|
-
# @return [ActiveJob::Base] the enqueued job
|
119
|
-
def self.enqueue(job, time = nil)
|
120
|
-
job.scheduled_at = time
|
121
|
-
# Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
|
122
|
-
serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
|
123
|
-
|
124
|
-
# When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks are
|
125
|
-
# registered on the job instance so that when the job is executed, the database record is deleted or updated
|
126
|
-
# (depending on the result).
|
127
|
-
#
|
128
|
-
# There are two normal cases:
|
129
|
-
# - The first time a job is enqueued, we need to create the record and register callbacks.
|
130
|
-
# - If a previously enqueued job instance is re-enqueued, for instance after execution fails, callbacks have
|
131
|
-
# already been registered. In this case we only need to update the record.
|
132
|
-
#
|
133
|
-
# We keep track of whether callbacks have been registered by setting the @__marj instance variable on the job's
|
134
|
-
# singleton class. This holds a reference to the record. This allows us to update the record without re-fetching it
|
135
|
-
# and also ensures that if execute is called on a record any updates to the database are reflected on that record
|
136
|
-
# instance.
|
137
|
-
#
|
138
|
-
# There are also two edge cases:
|
139
|
-
# - It is possible for new job instance to be created for a job that is already in the database. In this case
|
140
|
-
# we need to update the record and register callbacks.
|
141
|
-
# - It is possible for the underlying row corresponding to an existing job to have been deleted. In this case we
|
142
|
-
# need to create a new record and update the reference on the job's singleton class.
|
143
|
-
if (record = job.singleton_class.instance_variable_get(:@__marj))
|
144
|
-
if Marj.exists?(job_id: job.job_id)
|
145
|
-
record.update!(serialized)
|
146
|
-
else
|
147
|
-
record = Marj.create!(serialized)
|
148
|
-
register_callbacks(job, record)
|
149
|
-
end
|
150
|
-
else
|
151
|
-
record = Marj.find_or_create_by!(job_id: job.job_id) { _1.assign_attributes(serialized) }
|
152
|
-
register_callbacks(job, record)
|
153
|
-
end
|
154
|
-
job
|
155
|
-
end
|
156
|
-
private_class_method :enqueue
|
157
|
-
end
|