acidic_job 1.0.0.pre3 → 1.0.0.pre6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile.lock +3 -1
- data/README.md +1 -1
- data/UPGRADE_GUIDE.md +25 -15
- data/lib/acidic_job/run.rb +13 -0
- data/lib/acidic_job/upgrade_service.rb +112 -109
- data/lib/acidic_job/version.rb +1 -1
- data/lib/generators/acidic_job/drop_tables_generator.rb +19 -24
- data/lib/generators/acidic_job/install_generator.rb +27 -0
- data/lib/generators/{templates → acidic_job/templates}/create_acidic_job_runs_migration.rb.erb +1 -1
- data/lib/generators/{templates → acidic_job/templates}/drop_acidic_job_keys_migration.rb.erb +1 -1
- metadata +5 -5
- data/lib/generators/acidic_job_generator.rb +0 -25
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9247c761078c570dfbacac2542bc8b7417c816bb9928f975bf7b92722a29f6a4
|
4
|
+
data.tar.gz: 8d36bb7517db0493d37eaf44aee901339a5d8d77411cad0e55fefe2d940137c6
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 47697f8b833d7061bacb2b6eaf1e6c15a9d6fec700db0891f881b8f945cf271e1c37adba9cde4572aed5d0e272767a690c172ece0937dbc212a90bc0dd02e8e5
|
7
|
+
data.tar.gz: cda551ac7a76a63199f03b19e7787ba0bfae414253b1e71b098b8304705458d0c8f21b1755f9f10c15b651d55b849469284754d6e585d13159ee4c28a17ed723
|
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
acidic_job (1.0.0.
|
4
|
+
acidic_job (1.0.0.pre6)
|
5
5
|
activerecord (>= 6.1.0)
|
6
6
|
activesupport
|
7
7
|
|
@@ -122,6 +122,8 @@ GEM
|
|
122
122
|
nokogiri (1.12.3)
|
123
123
|
mini_portile2 (~> 2.6.1)
|
124
124
|
racc (~> 1.4)
|
125
|
+
nokogiri (1.12.3-x86_64-darwin)
|
126
|
+
racc (~> 1.4)
|
125
127
|
noticed (1.5.7)
|
126
128
|
http (>= 4.0.0)
|
127
129
|
rails (>= 5.2.0)
|
data/README.md
CHANGED
data/UPGRADE_GUIDE.md
CHANGED
@@ -1,19 +1,19 @@
|
|
1
1
|
# AcidicJob Upgrade Guide
|
2
2
|
|
3
|
-
1. Update version requirements in `Gemfile`
|
3
|
+
## 1. Update version requirements in `Gemfile`
|
4
4
|
|
5
|
-
```
|
6
|
-
-gem "acidic_job"
|
7
|
-
+gem "acidic_job", "~> 1.0.0.pre1"
|
5
|
+
```diff
|
6
|
+
- gem "acidic_job"
|
7
|
+
+ gem "acidic_job", "~> 1.0.0.pre1"
|
8
8
|
```
|
9
9
|
|
10
10
|
result:
|
11
11
|
```
|
12
|
-
Installing acidic_job 1.0.0.
|
12
|
+
Installing acidic_job 1.0.0.pre4 (was 0.7.7)
|
13
13
|
Bundle updated!
|
14
14
|
```
|
15
15
|
|
16
|
-
2. Generate migration for new `AcidicJob::Run` model
|
16
|
+
## 2. Generate migration for new `AcidicJob::Run` model
|
17
17
|
|
18
18
|
```bash
|
19
19
|
rails generate acidic_job
|
@@ -24,7 +24,7 @@ result:
|
|
24
24
|
create db/migrate/#{yyyymmddhhmmss}_create_acidic_job_runs.rb
|
25
25
|
```
|
26
26
|
|
27
|
-
3. Delete any unneeded `AcidicJob::Key` records
|
27
|
+
## 3. Delete any unneeded `AcidicJob::Key` records
|
28
28
|
|
29
29
|
Typically, records that are already finished do not need to be retained. Sometimes, however, applications key finished records around for some amount of time for debugging or metrics aggregation. Whatever your application's logic is for whether or not an `AcidicJob::Key` record is still needed, for all unneeded records, delete them.
|
30
30
|
|
@@ -34,7 +34,7 @@ For example, this would delete all finished `Key` records over 1 month old:
|
|
34
34
|
AcidicJob::Key.where(recovery_point: AcidicJob::Key::RECOVERY_POINT_FINISHED, last_run_at: ..1.month.ago).delete_all
|
35
35
|
```
|
36
36
|
|
37
|
-
4. Migrate `AcidicJob::Key` to `AcidicJob::Run`
|
37
|
+
## 4. Migrate `AcidicJob::Key` to `AcidicJob::Run`
|
38
38
|
|
39
39
|
`AcidicJob` ships with an upgrade module that provides a script to migrate older `Key` records to the new `Run` model.
|
40
40
|
|
@@ -47,25 +47,35 @@ This script will prepare an `insert_all` command for `Run` records by mapping th
|
|
47
47
|
result:
|
48
48
|
```
|
49
49
|
{
|
50
|
-
|
51
|
-
|
52
|
-
|
50
|
+
run_records: <Integer>,
|
51
|
+
key_records: <Integer>,
|
52
|
+
errored_keys: <Array>
|
53
53
|
}
|
54
54
|
```
|
55
55
|
|
56
|
-
5. Triage remaining `AcidicJob::Key` records
|
56
|
+
## 5. Triage remaining `AcidicJob::Key` records
|
57
57
|
|
58
58
|
If there were any `AcidicJob::Key` records that failed to be mapped to the new `Run` model, you will need to manually triage whatever the exception was. In all likelihood, the exception would be relating to the translation of the `Key#job_args` field to the `Run#serialized_job` field, as all other fields have a fairly straight-forward mapping. If you can't resolve the issue, please open an Issue in GitHub.
|
59
59
|
|
60
|
-
6. Ensure all `AcidicJob::Staged` records are processed
|
60
|
+
## 6. Ensure all `AcidicJob::Staged` records are processed
|
61
61
|
|
62
62
|
`AcidicJob` still ships with an upgrade module that provides the older `Key` and `Staged` records, so this functionality will still be present to handle any existing records in your database when you deploy the updated version.
|
63
63
|
|
64
|
-
7. Remove the old tables
|
64
|
+
## 7. Remove the old tables
|
65
65
|
|
66
66
|
Once you have successfully migrated everything over and the new system has been running smoothly for some time, you should drop the old `acidic_job_keys` and `staged_acidic_jobs` tables. We provide a migration generator just for this purpose:
|
67
67
|
|
68
68
|
```bash
|
69
69
|
rails generate acidic_job:drop_tables
|
70
|
+
```
|
71
|
+
|
72
|
+
result:
|
73
|
+
```
|
74
|
+
create db/migrate/#{yyyymmddhhmmss}_drop_old_acidic_job_tables.rb
|
75
|
+
```
|
76
|
+
|
77
|
+
You can then run the migration to have those tables removed:
|
78
|
+
|
79
|
+
```bash
|
70
80
|
rails db:migrate
|
71
|
-
```
|
81
|
+
```
|
data/lib/acidic_job/run.rb
CHANGED
@@ -35,6 +35,19 @@ module AcidicJob
|
|
35
35
|
validates :workflow, presence: true
|
36
36
|
end
|
37
37
|
|
38
|
+
def self.purge
|
39
|
+
successfully_completed = where(
|
40
|
+
recovery_point: FINISHED_RECOVERY_POINT,
|
41
|
+
error_object: nil
|
42
|
+
)
|
43
|
+
count = successfully_completed.count
|
44
|
+
|
45
|
+
return 0 if count.zero?
|
46
|
+
|
47
|
+
Rails.logger.info("Deleting #{count} successfully completed AcidicJob runs")
|
48
|
+
successfully_completed.delete_all
|
49
|
+
end
|
50
|
+
|
38
51
|
def finished?
|
39
52
|
recovery_point == FINISHED_RECOVERY_POINT
|
40
53
|
end
|
@@ -3,113 +3,116 @@
|
|
3
3
|
require "active_support/concern"
|
4
4
|
|
5
5
|
module AcidicJob
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
6
|
+
# recreate the original `Key` model
|
7
|
+
class Key < ::ActiveRecord::Base
|
8
|
+
RECOVERY_POINT_FINISHED = "FINISHED"
|
9
|
+
|
10
|
+
self.table_name = "acidic_job_keys"
|
11
|
+
|
12
|
+
serialize :error_object
|
13
|
+
serialize :job_args
|
14
|
+
serialize :workflow
|
15
|
+
store :attr_accessors
|
16
|
+
end
|
17
|
+
|
18
|
+
# recreate the original `Staged` model
|
19
|
+
class Staged < ActiveRecord::Base
|
20
|
+
self.table_name = "staged_acidic_jobs"
|
21
|
+
|
22
|
+
serialize :job_args
|
23
|
+
|
24
|
+
after_create_commit :enqueue_job
|
25
|
+
|
26
|
+
private
|
27
|
+
|
28
|
+
def enqueue_job
|
29
|
+
gid = { "staged_job_gid" => to_global_id.to_s }
|
30
|
+
|
31
|
+
if job_args.is_a?(Hash) && job_args.key?("arguments")
|
32
|
+
job_args["arguments"].concat([gid])
|
33
|
+
else
|
34
|
+
job_args.concat([gid])
|
35
|
+
end
|
36
|
+
|
37
|
+
case adapter
|
38
|
+
when "activejob"
|
39
|
+
::ActiveJob::Base.deserialize(job_args).enqueue
|
40
|
+
when "sidekiq"
|
41
|
+
job_name.constantize.perform_async(*job_args)
|
42
|
+
else
|
43
|
+
raise UnknownJobAdapter.new(adapter: adapter)
|
44
|
+
end
|
45
|
+
|
46
|
+
# NOTE: record will be deleted after the job has successfully been performed
|
47
|
+
true
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
module UpgradeService
|
52
|
+
def self.execute
|
53
|
+
# prepare an array to hold the attribute hashes to be passed to `insert_all`
|
54
|
+
run_attributes = []
|
55
|
+
# prepare an array to hold any `Key` records that we couldn't successfully map to `Run` records
|
56
|
+
errored_keys = []
|
57
|
+
|
58
|
+
# iterate over all `AcidicJob::Key` records in batches,
|
59
|
+
# preparing a `Run` attribute hash to be passed to `insert_all`
|
60
|
+
::AcidicJob::Key.find_each do |key|
|
61
|
+
# map all of the simple attributes directly
|
62
|
+
attributes = {
|
63
|
+
id: key.id,
|
64
|
+
staged: false,
|
65
|
+
idempotency_key: key.idempotency_key,
|
66
|
+
job_class: key.job_name,
|
67
|
+
last_run_at: key.last_run_at,
|
68
|
+
locked_at: key.locked_at,
|
69
|
+
recovery_point: key.recovery_point,
|
70
|
+
error_object: key.error_object,
|
71
|
+
attr_accessors: key.attr_accessors,
|
72
|
+
workflow: key.workflow,
|
73
|
+
created_at: key.created_at,
|
74
|
+
updated_at: key.updated_at
|
75
|
+
}
|
76
|
+
|
77
|
+
# prepare the more complicated `job_args` -> `serialized_job` translation
|
78
|
+
job_class = key.job_name.constantize
|
79
|
+
if defined?(::Sidekiq) && job_class.include?(::Sidekiq::Worker)
|
80
|
+
unless job_class.include?(::AcidicJob::Extensions::Sidekiq)
|
81
|
+
job_class.include(::AcidicJob::Extensions::Sidekiq)
|
82
|
+
end
|
83
|
+
job_instance = job_class.new
|
84
|
+
serialized_job = job_instance.serialize_job(*key.job_args)
|
85
|
+
elsif defined?(::ActiveJob) && job_class < ::ActiveJob::Base
|
86
|
+
unless job_class.include?(::AcidicJob::Extensions::ActiveJob)
|
87
|
+
job_class.include(::AcidicJob::Extensions::ActiveJob)
|
88
|
+
end
|
89
|
+
job_args = begin
|
90
|
+
::ActiveJob::Arguments.deserialize(key.job_args)
|
91
|
+
rescue ::ActiveJob::DeserializationError
|
92
|
+
key.job_args
|
93
|
+
end
|
94
|
+
job_instance = job_class.new(*job_args)
|
95
|
+
serialized_job = job_instance.serialize_job
|
96
|
+
end
|
97
|
+
|
98
|
+
attributes[:serialized_job] = serialized_job
|
99
|
+
run_attributes << attributes
|
100
|
+
rescue StandardError => e
|
101
|
+
errored_keys << [e, key]
|
102
|
+
end
|
103
|
+
|
104
|
+
# insert all of the `Run` records
|
105
|
+
::AcidicJob::Run.insert_all(run_attributes)
|
106
|
+
|
107
|
+
# delete all successfully migrated `Key` record
|
108
|
+
::AcidicJob::Key.where(id: ::AcidicJob::Run.select(:id)).delete_all
|
109
|
+
|
110
|
+
# return a report of the upgrade migration
|
111
|
+
{
|
112
|
+
run_records: ::AcidicJob::Run.count,
|
113
|
+
key_records: ::AcidicJob::Key.count,
|
114
|
+
errored_keys: errored_keys
|
115
|
+
}
|
116
|
+
end
|
117
|
+
end
|
115
118
|
end
|
data/lib/acidic_job/version.rb
CHANGED
@@ -1,31 +1,26 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
require "rails/generators"
|
4
3
|
require "rails/generators/active_record"
|
5
4
|
|
6
5
|
module AcidicJob
|
7
|
-
|
8
|
-
|
9
|
-
|
6
|
+
module Generators
|
7
|
+
class DropTablesGenerator < ::Rails::Generators::Base
|
8
|
+
include ActiveRecord::Generators::Migration
|
9
|
+
source_root File.expand_path("templates", __dir__)
|
10
10
|
|
11
|
-
|
12
|
-
|
13
|
-
# ActiveRecord::Generators::Base inherits from Rails::Generators::NamedBase
|
14
|
-
# which requires a NAME parameter for the new table name.
|
15
|
-
# Our generator always uses "acidic_job_runs", so we just set a random name here.
|
16
|
-
argument :name, type: :string, default: "random_name"
|
11
|
+
desc "Drops the pre-1.0 tables for the AcidicJob::Key and AcidicJob::Staged models."
|
17
12
|
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
end
|
13
|
+
def copy_migration
|
14
|
+
migration_template "drop_acidic_job_keys_migration.rb.erb",
|
15
|
+
"db/migrate/drop_old_acidic_job_tables.rb",
|
16
|
+
migration_version: migration_version
|
17
|
+
end
|
18
|
+
|
19
|
+
protected
|
20
|
+
|
21
|
+
def migration_version
|
22
|
+
"[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]"
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
@@ -0,0 +1,27 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "rails/generators/active_record"
|
4
|
+
|
5
|
+
module AcidicJob
|
6
|
+
module Generators
|
7
|
+
class InstallGenerator < ::Rails::Generators::Base
|
8
|
+
include ActiveRecord::Generators::Migration
|
9
|
+
source_root File.expand_path("templates", __dir__)
|
10
|
+
|
11
|
+
desc "Generates a migration for the AcidicJob::Run table."
|
12
|
+
|
13
|
+
# Copies the migration template to db/migrate.
|
14
|
+
def copy_acidic_job_runs_migration_files
|
15
|
+
migration_template "create_acidic_job_runs_migration.rb.erb",
|
16
|
+
"db/migrate/create_acidic_job_runs.rb",
|
17
|
+
migration_version: migration_version
|
18
|
+
end
|
19
|
+
|
20
|
+
protected
|
21
|
+
|
22
|
+
def migration_version
|
23
|
+
"[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]"
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: acidic_job
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.0.
|
4
|
+
version: 1.0.0.pre6
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- fractaledmind
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2022-03-
|
11
|
+
date: 2022-03-10 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|
@@ -90,9 +90,9 @@ files:
|
|
90
90
|
- lib/acidic_job/upgrade_service.rb
|
91
91
|
- lib/acidic_job/version.rb
|
92
92
|
- lib/generators/acidic_job/drop_tables_generator.rb
|
93
|
-
- lib/generators/
|
94
|
-
- lib/generators/templates/create_acidic_job_runs_migration.rb.erb
|
95
|
-
- lib/generators/templates/drop_acidic_job_keys_migration.rb.erb
|
93
|
+
- lib/generators/acidic_job/install_generator.rb
|
94
|
+
- lib/generators/acidic_job/templates/create_acidic_job_runs_migration.rb.erb
|
95
|
+
- lib/generators/acidic_job/templates/drop_acidic_job_keys_migration.rb.erb
|
96
96
|
homepage: https://github.com/fractaledmind/acidic_job
|
97
97
|
licenses:
|
98
98
|
- MIT
|
@@ -1,25 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require "rails/generators"
|
4
|
-
require "rails/generators/active_record"
|
5
|
-
|
6
|
-
class AcidicJobGenerator < ActiveRecord::Generators::Base
|
7
|
-
# ActiveRecord::Generators::Base inherits from Rails::Generators::NamedBase
|
8
|
-
# which requires a NAME parameter for the new table name.
|
9
|
-
# Our generator always uses "acidic_job_runs", so we just set a random name here.
|
10
|
-
argument :name, type: :string, default: "random_name"
|
11
|
-
|
12
|
-
source_root File.expand_path("templates", __dir__)
|
13
|
-
|
14
|
-
# Copies the migration template to db/migrate.
|
15
|
-
def copy_acidic_job_runs_migration_files
|
16
|
-
migration_template "create_acidic_job_runs_migration.rb.erb",
|
17
|
-
"db/migrate/create_acidic_job_runs.rb"
|
18
|
-
end
|
19
|
-
|
20
|
-
protected
|
21
|
-
|
22
|
-
def migration_class
|
23
|
-
ActiveRecord::Migration["#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}"]
|
24
|
-
end
|
25
|
-
end
|