acidic_job 1.0.0.rc3 → 1.0.0.rc5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +6 -6
- data/app/models/acidic_job/entry.rb +4 -2
- data/app/models/acidic_job/execution.rb +25 -12
- data/app/models/acidic_job/value.rb +2 -0
- data/lib/acidic_job/builder.rb +2 -2
- data/lib/acidic_job/context.rb +36 -20
- data/lib/acidic_job/errors.rb +2 -3
- data/lib/acidic_job/log_subscriber.rb +4 -6
- data/lib/acidic_job/plugin_context.rb +26 -3
- data/lib/acidic_job/plugins/transactional_step.rb +4 -4
- data/lib/acidic_job/serializer.rb +28 -0
- data/lib/acidic_job/serializers/job_serializer.rb +1 -1
- data/lib/acidic_job/serializers/range_serializer.rb +1 -3
- data/lib/acidic_job/testing.rb +10 -12
- data/lib/acidic_job/version.rb +1 -1
- data/lib/acidic_job/workflow.rb +44 -32
- data/lib/acidic_job.rb +3 -0
- data/lib/generators/acidic_job/install_generator.rb +5 -5
- data/lib/generators/acidic_job/templates/create_acidic_job_tables_migration.rb.erb +6 -6
- metadata +7 -55
- data/.codacy.yml +0 -4
- data/.github/FUNDING.yml +0 -13
- data/.github/workflows/main.yml +0 -38
- data/.gitignore +0 -18
- data/.rubocop.yml +0 -83
- data/.ruby-version +0 -1
- data/Gemfile +0 -5
- data/Gemfile.lock +0 -201
- data/Rakefile +0 -16
- data/TODO +0 -77
- data/UPGRADE_GUIDE.md +0 -81
- data/acidic_job.gemspec +0 -52
- data/bin/console +0 -21
- data/bin/setup +0 -8
- data/bin/test_all +0 -26
- data/blog_post.md +0 -28
- data/combustion/log/test.log +0 -0
- data/gemfiles/rails_7.0.gemfile +0 -11
- data/gemfiles/rails_7.1.gemfile +0 -11
- data/gemfiles/rails_7.2.gemfile +0 -11
- data/gemfiles/rails_8.0.gemfile +0 -11
data/UPGRADE_GUIDE.md
DELETED
@@ -1,81 +0,0 @@
|
|
1
|
-
# AcidicJob Upgrade Guide
|
2
|
-
|
3
|
-
## 1. Update version requirements in `Gemfile`
|
4
|
-
|
5
|
-
```diff
|
6
|
-
- gem "acidic_job"
|
7
|
-
+ gem "acidic_job", "~> 1.0.0.pre1"
|
8
|
-
```
|
9
|
-
|
10
|
-
result:
|
11
|
-
```
|
12
|
-
Installing acidic_job 1.0.0.pre4 (was 0.7.7)
|
13
|
-
Bundle updated!
|
14
|
-
```
|
15
|
-
|
16
|
-
## 2. Generate migration for new `AcidicJob::Run` model
|
17
|
-
|
18
|
-
```bash
|
19
|
-
rails generate acidic_job
|
20
|
-
```
|
21
|
-
|
22
|
-
result:
|
23
|
-
```
|
24
|
-
create db/migrate/#{yyyymmddhhmmss}_create_acidic_job_runs.rb
|
25
|
-
```
|
26
|
-
|
27
|
-
## 3. Delete any unneeded `AcidicJob::Key` records
|
28
|
-
|
29
|
-
Typically, records that are already finished do not need to be retained. Sometimes, however, applications key finished records around for some amount of time for debugging or metrics aggregation. Whatever your application's logic is for whether or not an `AcidicJob::Key` record is still needed, for all unneeded records, delete them.
|
30
|
-
|
31
|
-
For example, this would delete all finished `Key` records over 1 month old:
|
32
|
-
|
33
|
-
```ruby
|
34
|
-
AcidicJob::Key.where(recovery_point: AcidicJob::Key::RECOVERY_POINT_FINISHED, last_run_at: ..1.month.ago).delete_all
|
35
|
-
```
|
36
|
-
|
37
|
-
## 4. Migrate `AcidicJob::Key` to `AcidicJob::Run`
|
38
|
-
|
39
|
-
`AcidicJob` ships with an upgrade module that provides a script to migrate older `Key` records to the new `Run` model.
|
40
|
-
|
41
|
-
```ruby
|
42
|
-
AcidicJob::UpgradeService.execute
|
43
|
-
```
|
44
|
-
|
45
|
-
This script will prepare an `insert_all` command for `Run` records by mapping the older `Key` data to the new `Run` schema. It also creates the new `Run` records with the same `id` as their `Key` counterparts, and then deletes all `Key` records successfully mapped over. Any `Key` records that were failed to be mapped over will be reported, along with the exception, in the `errored_keys` portion of the resulting hash.
|
46
|
-
|
47
|
-
result:
|
48
|
-
```
|
49
|
-
{
|
50
|
-
run_records: <Integer>,
|
51
|
-
key_records: <Integer>,
|
52
|
-
errored_keys: <Array>
|
53
|
-
}
|
54
|
-
```
|
55
|
-
|
56
|
-
## 5. Triage remaining `AcidicJob::Key` records
|
57
|
-
|
58
|
-
If there were any `AcidicJob::Key` records that failed to be mapped to the new `Run` model, you will need to manually triage whatever the exception was. In all likelihood, the exception would be relating to the translation of the `Key#job_args` field to the `Run#serialized_job` field, as all other fields have a fairly straight-forward mapping. If you can't resolve the issue, please open an Issue in GitHub.
|
59
|
-
|
60
|
-
## 6. Ensure all `AcidicJob::Staged` records are processed
|
61
|
-
|
62
|
-
`AcidicJob` still ships with an upgrade module that provides the older `Key` and `Staged` records, so this functionality will still be present to handle any existing records in your database when you deploy the updated version.
|
63
|
-
|
64
|
-
## 7. Remove the old tables
|
65
|
-
|
66
|
-
Once you have successfully migrated everything over and the new system has been running smoothly for some time, you should drop the old `acidic_job_keys` and `staged_acidic_jobs` tables. We provide a migration generator just for this purpose:
|
67
|
-
|
68
|
-
```bash
|
69
|
-
rails generate acidic_job:drop_tables
|
70
|
-
```
|
71
|
-
|
72
|
-
result:
|
73
|
-
```
|
74
|
-
create db/migrate/#{yyyymmddhhmmss}_drop_old_acidic_job_tables.rb
|
75
|
-
```
|
76
|
-
|
77
|
-
You can then run the migration to have those tables removed:
|
78
|
-
|
79
|
-
```bash
|
80
|
-
rails db:migrate
|
81
|
-
```
|
data/acidic_job.gemspec
DELETED
@@ -1,52 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require_relative "lib/acidic_job/version"
|
4
|
-
|
5
|
-
Gem::Specification.new do |spec|
|
6
|
-
spec.name = "acidic_job"
|
7
|
-
spec.version = AcidicJob::VERSION
|
8
|
-
spec.authors = ["fractaledmind"]
|
9
|
-
spec.email = ["stephen.margheim@gmail.com"]
|
10
|
-
|
11
|
-
spec.summary = "Idempotent operations for Rails apps, built on top of ActiveJob."
|
12
|
-
spec.description = "Idempotent operations for Rails apps, built on top of ActiveJob."
|
13
|
-
spec.homepage = "https://github.com/fractaledmind/acidic_job"
|
14
|
-
spec.license = "MIT"
|
15
|
-
spec.required_ruby_version = Gem::Requirement.new(">= 2.7.0")
|
16
|
-
|
17
|
-
spec.metadata["homepage_uri"] = spec.homepage
|
18
|
-
spec.metadata["source_code_uri"] = "https://github.com/fractaledmind/acidic_job"
|
19
|
-
spec.metadata["changelog_uri"] = "https://github.com/fractaledmind/acidic_job/CHANGELOG.md"
|
20
|
-
|
21
|
-
# Specify which files should be added to the gem when it is released.
|
22
|
-
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
23
|
-
spec.files = Dir.chdir(File.expand_path(__dir__)) do
|
24
|
-
`git ls-files -z`.split("\x0").reject { |f| f.match(%r{\A(?:test|spec|features)/}) }
|
25
|
-
end
|
26
|
-
spec.bindir = "exe"
|
27
|
-
spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
|
28
|
-
spec.require_paths = ["lib"]
|
29
|
-
|
30
|
-
spec.add_dependency "json", ">= 2.7.0" # see: https://github.com/ruby/json/pull/519
|
31
|
-
">= 7.1".tap do |rails_version|
|
32
|
-
spec.add_dependency "activejob", rails_version
|
33
|
-
spec.add_dependency "activerecord", rails_version
|
34
|
-
spec.add_dependency "activesupport", rails_version
|
35
|
-
spec.add_dependency "railties", rails_version
|
36
|
-
spec.add_development_dependency "actionmailer", rails_version
|
37
|
-
end
|
38
|
-
|
39
|
-
spec.add_development_dependency "chaotic_job"
|
40
|
-
spec.add_development_dependency "combustion"
|
41
|
-
spec.add_development_dependency "minitest"
|
42
|
-
spec.add_development_dependency "rake"
|
43
|
-
spec.add_development_dependency "rubocop"
|
44
|
-
spec.add_development_dependency "rubocop-minitest"
|
45
|
-
spec.add_development_dependency "rubocop-rake"
|
46
|
-
spec.add_development_dependency "simplecov"
|
47
|
-
spec.add_development_dependency "sqlite3"
|
48
|
-
|
49
|
-
# For more information and examples about making a new gem, checkout our
|
50
|
-
# guide at: https://bundler.io/guides/creating_gem.html
|
51
|
-
spec.metadata["rubygems_mfa_required"] = "true"
|
52
|
-
end
|
data/bin/console
DELETED
@@ -1,21 +0,0 @@
|
|
1
|
-
#!/usr/bin/env ruby
|
2
|
-
# frozen_string_literal: true
|
3
|
-
|
4
|
-
require "bundler/setup"
|
5
|
-
require "rails"
|
6
|
-
require "acidic_job"
|
7
|
-
|
8
|
-
# You can add fixtures and/or initialization code here to make experimenting
|
9
|
-
# with your gem easier. You can also use a different console, if you like.
|
10
|
-
|
11
|
-
require "combustion"
|
12
|
-
require "sqlite3"
|
13
|
-
Combustion.path = "test/combustion"
|
14
|
-
Combustion.initialize! :active_record
|
15
|
-
|
16
|
-
# (If you use this, don't forget to add pry to your Gemfile!)
|
17
|
-
# require "pry"
|
18
|
-
# Pry.start
|
19
|
-
|
20
|
-
require "irb"
|
21
|
-
IRB.start(__FILE__)
|
data/bin/setup
DELETED
data/bin/test_all
DELETED
@@ -1,26 +0,0 @@
|
|
1
|
-
run_tests() {
|
2
|
-
if [ "$#" -ne 2 ]; then
|
3
|
-
echo "Usage: run_tests RUBY_VERSION RAILS_VERSION"
|
4
|
-
echo "Example: run_tests 3.0.7 7.1"
|
5
|
-
return 1
|
6
|
-
fi
|
7
|
-
|
8
|
-
local ruby="$1"
|
9
|
-
local gemfile="gemfiles/rails_$2.gemfile"
|
10
|
-
|
11
|
-
echo "**************************************************"
|
12
|
-
echo "Running tests with Ruby $1 and Rails $2..."
|
13
|
-
ASDF_RUBY_VERSION=$ruby BUNDLE_GEMFILE=$gemfile bundle check ||
|
14
|
-
ASDF_RUBY_VERSION=$ruby BUNDLE_GEMFILE=$gemfile bundle install &&
|
15
|
-
ASDF_RUBY_VERSION=$ruby BUNDLE_GEMFILE=$gemfile bundle exec rake
|
16
|
-
}
|
17
|
-
|
18
|
-
run_tests "3.0.7" "7.1"
|
19
|
-
run_tests "3.1.6" "7.1"
|
20
|
-
run_tests "3.1.6" "7.2"
|
21
|
-
run_tests "3.2.5" "7.1"
|
22
|
-
run_tests "3.2.5" "7.2"
|
23
|
-
run_tests "3.2.5" "8.0"
|
24
|
-
run_tests "3.3.5" "7.1"
|
25
|
-
run_tests "3.3.5" "7.2"
|
26
|
-
run_tests "3.3.5" "8.0"
|
data/blog_post.md
DELETED
@@ -1,28 +0,0 @@
|
|
1
|
-
# ACIDic Operations in Rails
|
2
|
-
|
3
|
-
At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. At the horizon of basically any software is the goal to make that sofware _robust_. Typically, one makes a software system robust by making each of its operations robust. Moreover, typically, robustness in software is considered as the software being "ACIDic"—atomic, consistent, isolated, durable.
|
4
|
-
|
5
|
-
In a loosely collected series of articles, Brandur Leach lays out the core techniques and principles required to make an HTTP API properly ACIDic:
|
6
|
-
|
7
|
-
1. https://brandur.org/acid
|
8
|
-
2. https://brandur.org/http-transactions
|
9
|
-
3. https://brandur.org/job-drain
|
10
|
-
4. https://brandur.org/idempotency-keys
|
11
|
-
|
12
|
-
With these techniques and principles in mind, our challenge is bring them into the world of a standard Rails application. This will require us to conceptually map the concepts of an HTTP request, an API server action, and an HTTP response into the world of a running Rails process.
|
13
|
-
|
14
|
-
We can begin to make this mapping by observing that an API server action is a specific instantiation of the general concept of an "operation". Like all operations, it has a "trigger" (the HTTP request) and a "response" (the HTTP response). So, what we need is a foundation upon which to build our Rails "operations".
|
15
|
-
|
16
|
-
In order to help us find that tool, let us consider the necessary characteristics we need. We need something that we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.). It should also be able to be run both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response). It should then also be able to retry a specific operation (in much the way that an API consumer can "retry an operation" by hitting the same endpoint with the same request).
|
17
|
-
|
18
|
-
As we lay out these characteristics, I imagine your mind is going where mine went—`ActiveJob` gives us a solid foundation upon which we can build "ACIDic" operations.
|
19
|
-
|
20
|
-
So, our challenge to build tooling which will allow us to make "operational" jobs _robust_.
|
21
|
-
|
22
|
-
What we need primarily is to be able to make our jobs *idempotent*, and one of the simplest yet still most powerful tools for making an operation idempotent is the idempotency key. As laid out in the article linked above, an idempotency key is a record that we store in our database to uniquely identify a particular execution of an operation and a related "recovery point" for where we are in the process of that operation.
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
data/combustion/log/test.log
DELETED
File without changes
|
data/gemfiles/rails_7.0.gemfile
DELETED
data/gemfiles/rails_7.1.gemfile
DELETED
data/gemfiles/rails_7.2.gemfile
DELETED
data/gemfiles/rails_8.0.gemfile
DELETED
@@ -1,11 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
source "https://rubygems.org"
|
4
|
-
|
5
|
-
gem "activejob", "~> 8.0.0.rc1"
|
6
|
-
gem "activerecord", "~> 8.0.0.rc1"
|
7
|
-
gem "activesupport", "~> 8.0.0.rc1"
|
8
|
-
gem "railties", "~> 8.0.0.rc1"
|
9
|
-
gem "sqlite3", ">= 2.1.0"
|
10
|
-
|
11
|
-
gemspec path: "../"
|