belated 0.7.0 → 0.8.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/workflows/main.yml +2 -2
- data/CHANGELOG.md +5 -0
- data/Gemfile.lock +3 -1
- data/README.md +8 -10
- data/belated.gemspec +1 -0
- data/lib/active_job/queue_adapters/belated_adapter.rb +7 -0
- data/lib/belated/client.rb +6 -1
- data/lib/belated/job_wrapper.rb +7 -7
- data/lib/belated/queue.rb +40 -14
- data/lib/belated/version.rb +1 -1
- data/lib/belated/worker.rb +20 -0
- data/lib/belated.rb +1 -18
- metadata +16 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9bd1b76847288931abea65941cf12792557ce92d930f07132d3d78f40cd736e0
|
4
|
+
data.tar.gz: 264e80282918062e70948fcc6f9357b56e49170eabab1c963ecf6ab4767401a2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5f96e1e24d51fea623952ac7d0dba6333bd7f2ec114fe271a7dc56a37b768f3bad58f0bcec1323038d0cf1fd0cb5104a0e66e93757ad7e9d4efe579596c95fac
|
7
|
+
data.tar.gz: ceab192601a13e0e8413f6205e5d82e1ebca1a483f4201cabeda670f8f18d0cf7225fc1442eec96474d7cafc561e4c4808e14ebbffae436b144a3d4805f2770f
|
data/.github/workflows/main.yml
CHANGED
@@ -7,9 +7,9 @@ jobs:
|
|
7
7
|
strategy:
|
8
8
|
fail-fast: false
|
9
9
|
matrix:
|
10
|
-
os: [ubuntu-latest
|
10
|
+
os: [ubuntu-latest]
|
11
11
|
# Due to https://github.com/actions/runner/issues/849, we have to use quotes for '3.0'
|
12
|
-
ruby: [2.7, '3.0']
|
12
|
+
ruby: [2.6, 2.7, '3.0']
|
13
13
|
runs-on: ${{ matrix.os }}
|
14
14
|
steps:
|
15
15
|
- uses: actions/checkout@v2
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,10 @@
|
|
1
1
|
## [Unreleased]
|
2
2
|
|
3
|
+
## [0.8] - 2021-09-08
|
4
|
+
- Using PStore for future jobs backup and job history. Job history is rotated daily, future jobs cannot at the moment.
|
5
|
+
- PStore has ultrasafe mode, should test whether that makes things very slow. If not, might be worth using it for the peace of mind. Or at least have it as an option.
|
6
|
+
- ActiveJob support is a bit more natural now code-wise.
|
7
|
+
|
3
8
|
## [0.7] - 2021-09-04
|
4
9
|
- ActiveJob support! Retries, exception rescuing should work as expected.
|
5
10
|
- Second Moderna jab took me out for a while... sorry for the long wait.
|
data/Gemfile.lock
CHANGED
@@ -1,9 +1,10 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
belated (0.
|
4
|
+
belated (0.8.0)
|
5
5
|
drb
|
6
6
|
dry-configurable
|
7
|
+
pstore
|
7
8
|
ruby2_keywords
|
8
9
|
sorted_set
|
9
10
|
|
@@ -111,6 +112,7 @@ GEM
|
|
111
112
|
parallel (1.20.1)
|
112
113
|
parser (3.0.2.0)
|
113
114
|
ast (~> 2.4.1)
|
115
|
+
pstore (0.1.1)
|
114
116
|
racc (1.5.2)
|
115
117
|
rack (2.2.3)
|
116
118
|
rack-test (1.1.0)
|
data/README.md
CHANGED
@@ -2,11 +2,9 @@
|
|
2
2
|
|
3
3
|
[![CodeFactor](https://www.codefactor.io/repository/github/sampokuokkanen/belated/badge)](https://www.codefactor.io/repository/github/sampokuokkanen/belated) [![Gem Version](https://badge.fury.io/rb/belated.svg)](https://badge.fury.io/rb/belated)
|
4
4
|
|
5
|
-
This is Belated, a new Ruby backend job library! It supports running procs, lambdas and classes in the background. To deal with restarts, it uses YAML to load the
|
5
|
+
This is Belated, a new Ruby backend job library! It supports running procs, lambdas and classes in the background. To deal with restarts, it uses YAML for the current jobs in the queue waiting to be processed and PStore for the future jobs to load the queues into a file, which it then calls at startup to find the previous jobs. There is no way in Ruby to save procs or lambdas to a file, so they are discarded when the process restarts.
|
6
6
|
|
7
|
-
Belated uses the Ruby Queue class, so it's First In, First Out (FIFO).
|
8
|
-
|
9
|
-
Note that Belated used to be called HardWorker. That name was already in use in Sidekiq documentation and a bit too generic anyway.
|
7
|
+
Belated uses the Ruby Queue class, so it's First In, First Out (FIFO), unless of course you want to run the job in the future. In that case the order is decided by the time the job is scheduled to be executed.
|
10
8
|
|
11
9
|
It uses dRuby to do the communication! Which is absolute great. No need for Redis or PostgreSQL, just Ruby standard libraries.
|
12
10
|
|
@@ -16,14 +14,14 @@ Can be used if you're on a normal instance such as EC2 or Digital Ocean drop. No
|
|
16
14
|
|
17
15
|
TODO LIST:
|
18
16
|
|
19
|
-
-
|
20
|
-
- Have a web UI.
|
21
|
-
- Have a job history
|
22
|
-
- Do some performance testing.
|
17
|
+
- Have a web UI with job history.
|
23
18
|
- Deploy a Rails app to production that is using Belated
|
24
19
|
and mention it in the readme. (Capistrano support?)
|
25
|
-
([Wasurechatta](https://wasurechatta.com/))
|
26
|
-
|
20
|
+
([Wasurechatta](https://wasurechatta.com/) deployed, still need to setup Capistrano)
|
21
|
+
|
22
|
+
# Why not Sidekiq?
|
23
|
+
|
24
|
+
If you can, definitely use Sidekiq!!! Belated is supposed to be used if you can't get anything else to work. Like if you want to use SQLite in a Rails app and don't want to have Redis running. Or maybe you just want to run procs in the background?
|
27
25
|
|
28
26
|
## Installation
|
29
27
|
|
data/belated.gemspec
CHANGED
@@ -22,6 +22,13 @@ module ActiveJob # :nodoc:
|
|
22
22
|
Rails.logger.info "Belated got job #{job} to be performed at #{Time.at(timestamp)}"
|
23
23
|
instance.perform_belated(job, at: timestamp, active_job: true)
|
24
24
|
end
|
25
|
+
|
26
|
+
# JobWrapper that overwrites perform for ActiveJob
|
27
|
+
class JobWrapper < Belated::JobWrapper
|
28
|
+
def perform
|
29
|
+
Base.execute job.serialize
|
30
|
+
end
|
31
|
+
end
|
25
32
|
end
|
26
33
|
end
|
27
34
|
end
|
data/lib/belated/client.rb
CHANGED
@@ -101,7 +101,12 @@ class Belated
|
|
101
101
|
def wrap_job(job, at:, max_retries:, active_job:)
|
102
102
|
return job if job.is_a?(JobWrapper)
|
103
103
|
|
104
|
-
|
104
|
+
wrapper = if active_job
|
105
|
+
ActiveJob::QueueAdapters::BelatedAdapter::JobWrapper
|
106
|
+
else
|
107
|
+
JobWrapper
|
108
|
+
end
|
109
|
+
wrapper.new(job: job, at: at, max_retries: max_retries, active_job: active_job)
|
105
110
|
end
|
106
111
|
|
107
112
|
def drb_connected?
|
data/lib/belated/job_wrapper.rb
CHANGED
@@ -13,21 +13,23 @@ class Belated
|
|
13
13
|
class JobWrapper
|
14
14
|
include Comparable
|
15
15
|
include Logging
|
16
|
-
attr_accessor :retries, :max_retries, :id, :job, :at,
|
16
|
+
attr_accessor :retries, :max_retries, :id, :job, :at,
|
17
|
+
:completed, :proc_klass, :error, :active_job
|
17
18
|
|
18
19
|
def initialize(job:, max_retries: 5, at: nil, active_job: false)
|
20
|
+
raise 'JobError' unless job.respond_to?(:call) || job.respond_to?(:perform)
|
21
|
+
|
19
22
|
self.retries = 0
|
20
23
|
self.max_retries = max_retries
|
21
24
|
self.id = job.respond_to?(:job_id) ? job.job_id : SecureRandom.uuid
|
22
25
|
self.job = job
|
23
26
|
self.at = at
|
24
|
-
self.completed = false
|
25
27
|
self.proc_klass = job.instance_of?(Proc)
|
26
28
|
self.active_job = active_job
|
27
29
|
end
|
28
30
|
|
29
31
|
def <=>(other)
|
30
|
-
at <=> other
|
32
|
+
at <=> (other&.at || other&.scheduled_at)
|
31
33
|
end
|
32
34
|
|
33
35
|
# rubocop:disable Lint/RescueException
|
@@ -37,7 +39,7 @@ class Belated
|
|
37
39
|
resp
|
38
40
|
rescue Exception => e
|
39
41
|
case e.class
|
40
|
-
when Interrupt, SignalException
|
42
|
+
when Interrupt, SignalException, NoMethodError
|
41
43
|
raise e
|
42
44
|
else
|
43
45
|
retry_job(e)
|
@@ -47,9 +49,7 @@ class Belated
|
|
47
49
|
|
48
50
|
# rubocop:enable Lint/RescueException
|
49
51
|
def execute
|
50
|
-
if
|
51
|
-
ActiveJob::Base.execute job.serialize
|
52
|
-
elsif job.respond_to?(:call)
|
52
|
+
if job.respond_to?(:call)
|
53
53
|
job.call
|
54
54
|
elsif job.respond_to?(:arguments)
|
55
55
|
job.perform(*job.arguments)
|
data/lib/belated/queue.rb
CHANGED
@@ -4,7 +4,7 @@ require 'belated/job'
|
|
4
4
|
require 'belated/logging'
|
5
5
|
require 'belated/job_wrapper'
|
6
6
|
require 'sorted_set'
|
7
|
-
|
7
|
+
require 'pstore'
|
8
8
|
class Belated
|
9
9
|
# Job queues that Belated uses.
|
10
10
|
# queue is the jobs that are currenly
|
@@ -13,7 +13,7 @@ class Belated
|
|
13
13
|
# to be added to queue at some point in the future.
|
14
14
|
class Queue
|
15
15
|
include Logging
|
16
|
-
attr_accessor :future_jobs
|
16
|
+
attr_accessor :future_jobs, :future_jobs_db
|
17
17
|
|
18
18
|
FILE_NAME = 'belated_dump'
|
19
19
|
|
@@ -21,6 +21,24 @@ class Belated
|
|
21
21
|
@queue = queue
|
22
22
|
@mutex = Mutex.new
|
23
23
|
self.future_jobs = future_jobs
|
24
|
+
self.future_jobs_db = PStore.new('future_jobs.pstore', true) # pass true for thread safety
|
25
|
+
end
|
26
|
+
|
27
|
+
def enqueue_future_jobs
|
28
|
+
loop do
|
29
|
+
job = future_jobs.min
|
30
|
+
if job.nil?
|
31
|
+
sleep Belated.heartbeat
|
32
|
+
next
|
33
|
+
end
|
34
|
+
if job.at <= Time.now.to_f
|
35
|
+
delete_job(job)
|
36
|
+
push(job)
|
37
|
+
end
|
38
|
+
rescue DRb::DRbConnError
|
39
|
+
error 'DRb connection error!!!!!!'
|
40
|
+
log stats
|
41
|
+
end
|
24
42
|
end
|
25
43
|
|
26
44
|
def push(job)
|
@@ -30,6 +48,7 @@ class Belated
|
|
30
48
|
else
|
31
49
|
@mutex.synchronize do
|
32
50
|
@future_jobs << job
|
51
|
+
insert_into_future_jobs_db(job) unless job.proc_klass
|
33
52
|
end
|
34
53
|
end
|
35
54
|
end
|
@@ -52,16 +71,16 @@ class Belated
|
|
52
71
|
end
|
53
72
|
|
54
73
|
def load_jobs
|
55
|
-
|
74
|
+
future_jobs_db.transaction(true) do
|
75
|
+
future_jobs_db.roots.each do |id|
|
76
|
+
future_jobs << future_jobs_db[id]
|
77
|
+
end
|
78
|
+
end
|
56
79
|
return unless File.exist?(FILE_NAME)
|
57
80
|
|
58
81
|
jobs = YAML.load(File.binread(FILE_NAME))
|
59
82
|
jobs.each do |job|
|
60
|
-
|
61
|
-
future_jobs.push(job)
|
62
|
-
else
|
63
|
-
@queue.push(job)
|
64
|
-
end
|
83
|
+
@queue.push(job)
|
65
84
|
end
|
66
85
|
File.delete(FILE_NAME)
|
67
86
|
end
|
@@ -73,12 +92,6 @@ class Belated
|
|
73
92
|
class_array << klass
|
74
93
|
end
|
75
94
|
end
|
76
|
-
future_jobs.each do |_job|
|
77
|
-
unless proc_or_shutdown?(klass = future_jobs.pop)
|
78
|
-
class_array << klass
|
79
|
-
end
|
80
|
-
end
|
81
|
-
|
82
95
|
pp File.open(FILE_NAME, 'wb') { |f| f.write(YAML.dump(class_array)) }
|
83
96
|
end
|
84
97
|
|
@@ -91,5 +104,18 @@ class Belated
|
|
91
104
|
def proc_or_shutdown?(job)
|
92
105
|
job.is_a?(Symbol) || job.job.instance_of?(Proc)
|
93
106
|
end
|
107
|
+
|
108
|
+
def delete_job(job)
|
109
|
+
log "Deleting #{future_jobs.delete(job)} from future jobs"
|
110
|
+
future_jobs_db.transaction do
|
111
|
+
future_jobs_db.delete(job.id)
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
115
|
+
def insert_into_future_jobs_db(job)
|
116
|
+
future_jobs_db.transaction do
|
117
|
+
future_jobs_db[job.id] = job
|
118
|
+
end
|
119
|
+
end
|
94
120
|
end
|
95
121
|
end
|
data/lib/belated/version.rb
CHANGED
data/lib/belated/worker.rb
CHANGED
@@ -1,4 +1,6 @@
|
|
1
1
|
require_relative 'logging'
|
2
|
+
require 'pstore'
|
3
|
+
|
2
4
|
class Belated
|
3
5
|
# The worker class that actually gets the jobs from the queue
|
4
6
|
# and calls them. Expects the jobs to be procs or
|
@@ -20,9 +22,27 @@ class Belated
|
|
20
22
|
|
21
23
|
log "Worker #{@number} got job: #{job.inspect}"
|
22
24
|
log job.perform
|
25
|
+
history_insert(job) unless job.proc_klass || !job.completed
|
23
26
|
rescue DRb::DRbConnError, Errno::ECONNREFUSED, RangeError => e
|
24
27
|
log e
|
25
28
|
end
|
26
29
|
end
|
30
|
+
|
31
|
+
private
|
32
|
+
|
33
|
+
def history_insert(job)
|
34
|
+
store.transaction do
|
35
|
+
store[job.id] = job
|
36
|
+
end
|
37
|
+
rescue StandardError => e
|
38
|
+
error e
|
39
|
+
end
|
40
|
+
|
41
|
+
def store
|
42
|
+
today = Time.now.strftime('%F')
|
43
|
+
return @store if @store&.path&.include?(today)
|
44
|
+
|
45
|
+
@store = PStore.new("history-#{today}.pstore", true)
|
46
|
+
end
|
27
47
|
end
|
28
48
|
end
|
data/lib/belated.rb
CHANGED
@@ -51,7 +51,7 @@ class Belated
|
|
51
51
|
connect!
|
52
52
|
banner_and_info
|
53
53
|
trap_signals
|
54
|
-
enqueue_future_jobs
|
54
|
+
@@queue.enqueue_future_jobs
|
55
55
|
end
|
56
56
|
alias initialize start
|
57
57
|
|
@@ -95,23 +95,6 @@ class Belated
|
|
95
95
|
Belated.config.rails
|
96
96
|
end
|
97
97
|
|
98
|
-
def enqueue_future_jobs
|
99
|
-
loop do
|
100
|
-
job = @@queue.future_jobs.min
|
101
|
-
if job.nil?
|
102
|
-
sleep Belated.heartbeat
|
103
|
-
next
|
104
|
-
end
|
105
|
-
if job.at <= Time.now.to_f
|
106
|
-
log "Deleting #{@@queue.future_jobs.delete(job)} from future jobs"
|
107
|
-
@@queue.push(job)
|
108
|
-
end
|
109
|
-
rescue DRb::DRbConnError
|
110
|
-
error 'DRb connection error!!!!!!'
|
111
|
-
log stats
|
112
|
-
end
|
113
|
-
end
|
114
|
-
|
115
98
|
def reload
|
116
99
|
log 'reloading...'
|
117
100
|
@@queue.load_jobs
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: belated
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.8.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Sampo Kuokkanen
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-09-
|
11
|
+
date: 2021-09-08 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: drb
|
@@ -52,6 +52,20 @@ dependencies:
|
|
52
52
|
- - ">="
|
53
53
|
- !ruby/object:Gem::Version
|
54
54
|
version: '0'
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: pstore
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: '0'
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: '0'
|
55
69
|
- !ruby/object:Gem::Dependency
|
56
70
|
name: sorted_set
|
57
71
|
requirement: !ruby/object:Gem::Requirement
|