acidic_job 0.9.0 → 1.0.0.beta.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (42) hide show
  1. checksums.yaml +4 -4
  2. data/.github/workflows/main.yml +4 -5
  3. data/.gitignore +1 -1
  4. data/.rubocop.yml +0 -10
  5. data/Gemfile.lock +117 -112
  6. data/README.md +123 -140
  7. data/acidic_job.gemspec +2 -0
  8. data/bin/sandbox +1958 -0
  9. data/gemfiles/{rails_6.1_sidekiq_6.4.gemfile → rails_6.1.gemfile} +0 -2
  10. data/gemfiles/{rails_7.0_sidekiq_6.4.gemfile → rails_7.0.gemfile} +0 -2
  11. data/lib/acidic_job/active_kiq.rb +15 -44
  12. data/lib/acidic_job/arguments.rb +0 -8
  13. data/lib/acidic_job/errors.rb +0 -1
  14. data/lib/acidic_job/mixin.rb +19 -30
  15. data/lib/acidic_job/perform_wrapper.rb +5 -5
  16. data/lib/acidic_job/processor.rb +9 -8
  17. data/lib/acidic_job/run.rb +6 -27
  18. data/lib/acidic_job/serializer.rb +2 -2
  19. data/lib/acidic_job/serializers/exception_serializer.rb +18 -23
  20. data/lib/acidic_job/serializers/job_serializer.rb +14 -6
  21. data/lib/acidic_job/serializers/worker_serializer.rb +6 -4
  22. data/lib/acidic_job/version.rb +1 -1
  23. data/lib/acidic_job/workflow.rb +8 -0
  24. data/lib/acidic_job.rb +10 -10
  25. metadata +35 -23
  26. data/.github/FUNDING.yml +0 -13
  27. data/gemfiles/rails_6.1_sidekiq_6.5.gemfile +0 -10
  28. data/gemfiles/rails_6.1_sidekiq_7.0.gemfile +0 -10
  29. data/gemfiles/rails_7.0_sidekiq_6.5.gemfile +0 -10
  30. data/gemfiles/rails_7.0_sidekiq_7.0.gemfile +0 -10
  31. data/gemfiles/rails_7.1_sidekiq_6.4.gemfile +0 -10
  32. data/gemfiles/rails_7.1_sidekiq_6.5.gemfile +0 -10
  33. data/gemfiles/rails_7.1_sidekiq_7.0.gemfile +0 -10
  34. data/lib/acidic_job/configured_job.rb +0 -11
  35. data/lib/acidic_job/extensions/action_mailer.rb +0 -19
  36. data/lib/acidic_job/extensions/noticed.rb +0 -46
  37. data/lib/acidic_job/perform_acidicly.rb +0 -23
  38. data/lib/acidic_job/railtie.rb +0 -44
  39. data/lib/acidic_job/serializers/active_kiq_serializer.rb +0 -25
  40. data/lib/acidic_job/serializers/new_record_serializer.rb +0 -25
  41. data/lib/acidic_job/test_case.rb +0 -9
  42. data/lib/acidic_job/testing.rb +0 -73
data/README.md CHANGED
@@ -1,14 +1,11 @@
1
1
  # AcidicJob
2
2
 
3
- [![Gem Version](https://badge.fury.io/rb/acidic_job.svg)](https://rubygems.org/gems/acidic_job)
4
- [![Gem Downloads](https://img.shields.io/gem/dt/acidic_job)](https://rubygems.org/gems/acidic_job)
5
- ![Tests](https://github.com/fractaledmind/acidic_job/actions/workflows/main.yml/badge.svg)
6
- ![Coverage](https://img.shields.io/badge/code%20coverage-98%25-success)
7
- [![Codacy Badge](https://app.codacy.com/project/badge/Grade/e0df63f7a6f141d4aecc3c477314fdb2)](https://www.codacy.com/gh/fractaledmind/acidic_job/dashboard?utm_source=github.com&utm_medium=referral&utm_content=fractaledmind/acidic_job&utm_campaign=Badge_Grade)
3
+ [![Gem Version](https://badge.fury.io/rb/acidic_job.svg)](https://badge.fury.io/rb/acidic_job)
4
+ ![main workflow](https://github.com/fractaledmind/acidic_job/actions/workflows/main.yml/badge.svg)
8
5
 
9
- ## Idempotent operations for Rails apps (for ActiveJob or Sidekiq)
6
+ ### Idempotent operations for Rails apps (for ActiveJob or Sidekiq)
10
7
 
11
- At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`, or we Rubyists can use the tried and true power of pure `Sidekiq`. With either we can easily trigger operations from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
8
+ At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`, or we Rubyists can use the tried and true power of pure `Sidekiq`. With either we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
12
9
 
13
10
  However, in order to ensure that our operational jobs are _robust_, we need to ensure that they are properly [idempotent and transactional](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional). As stated in the [GitLab Sidekiq Style Guide](https://docs.gitlab.com/ee/development/sidekiq_style_guide.html#idempotent-jobs):
14
11
 
@@ -18,33 +15,14 @@ However, in order to ensure that our operational jobs are _robust_, we need to e
18
15
 
19
16
  This is, of course, far easier said than done. Thus, `AcidicJob`.
20
17
 
21
- `AcidicJob` provides a framework to help you make your operational jobs atomic ⚛️, consistent 🤖, isolated 🕴🏼, and durable ⛰️. Its conceptual framework is directly inspired by a truly wonderful loosely collected series of articles written by [Brandur Leach](https://twitter.com/brandur), which together lay out core techniques and principles required to make an HTTP API properly ACIDic:
18
+ `AcidicJob` provides a framework to help you make your operational jobs atomic ⚛️, consistent 🤖, isolated 🕴🏼, and durable ⛰️. Its conceptual framework is directly inspired by a truly wonderful loosely collected series of articles written by Brandur Leach, which together lay out core techniques and principles required to make an HTTP API properly ACIDic:
22
19
 
23
- 1. [Building Robust Systems with ACID and Constraints](https://brandur.org/acid)
24
- 2. [Using Atomic Transactions to Power an Idempotent API](https://brandur.org/http-transactions)
25
- 3. [Transactionally Staged Job Drains in Postgres](https://brandur.org/job-drain)
26
- 4. [Implementing Stripe-like Idempotency Keys in Postgres](https://brandur.org/idempotency-keys)
20
+ 1. https://brandur.org/acid
21
+ 2. https://brandur.org/http-transactions
22
+ 3. https://brandur.org/job-drain
23
+ 4. https://brandur.org/idempotency-keys
27
24
 
28
- Seriously, go and read these articles. `AcidicJob` brings these techniques and principles into the world of a standard Rails application, treating your background jobs like an internal API of sorts. It provides a suite of functionality that empowers you to create complex, robust, and _acidic_ jobs.
29
-
30
- ## Key Features
31
-
32
- * **Transactional Steps**
33
- break your job into a series of steps, each of which will be run within an acidic database transaction, allowing retries to jump back to the last "recovery point".
34
- * **Steps that Await Jobs**
35
- have workflow steps await other jobs, which will be enqueued and processed independently, and only when they all have finished will the parent job be re-enqueued to continue the workflow
36
- * **Iterable Steps**
37
- define steps that iterate over some collection fully until moving on to the next step
38
- * **Persisted Attributes**
39
- when retrying jobs at later steps, we need to ensure that data created in previous steps is still available to later steps on retry.
40
- * **Transactionally Staged Jobs**
41
- enqueue additional jobs within the acidic transaction safely
42
- * **Custom Idempotency Keys**
43
- use something other than the job ID for the idempotency key of the job run
44
- * **Sidekiq Callbacks**
45
- bring ActiveJob-like callbacks into your pure Sidekiq Workers
46
- * **Run Finished Callbacks**
47
- set callbacks for when a job run finishes fully
25
+ `AcidicJob` brings these techniques and principles into the world of a standard Rails application.
48
26
 
49
27
  ## Installation
50
28
 
@@ -70,45 +48,59 @@ rails generate acidic_job:install
70
48
 
71
49
  ## Usage
72
50
 
73
- `AcidicJob` brings the most seamless experience when you inject it into every job in your application. This can be done most easily by simply having your `ApplicationJob` inherit from `AcidicJob::Base` (if using `ActiveJob`; inherit from `AcidicJob::ActiveKiq` if using pure Sidekiq workers):
51
+ `AcidicJob` is a concern that you `include` into your base `ApplicationJob`.
74
52
 
75
53
  ```ruby
76
- class ApplicationJob < AcidicJob::Base
54
+ class ApplicationJob < ActiveJob::Base
55
+ include AcidicJob
77
56
  end
78
57
  ```
79
58
 
80
- This is useful because the module needs to be mixed into any and all jobs that you want to either [1] make acidic or [2] enqueue acidicly.
59
+ This is useful because the module needs to be mixed into any and all jobs that you want to either make acidic or enqueue acidicly.
81
60
 
82
- If you only want to inject `AcidicJob` into a single job, you can include our concern `AcidicJob::Mixin` instead:
61
+ It provides a suite of functionality that empowers you to create complex, robust, and _acidic_ jobs.
83
62
 
84
- ```ruby
85
- class SomeJob < ApplicationJob
86
- include AcidicJob::Mixin
87
- end
88
- ```
63
+ ### TL;DR
64
+
65
+ #### Key Features
89
66
 
90
- ## Key Features (in depth)
67
+ * **Transactional Steps**
68
+ break your job into a series of steps, each of which will be run within an acidic database transaction, allowing retries to jump back to the last "recovery point".
69
+ * **Steps that Await Jobs**
70
+ have workflow steps await other jobs, which will be enqueued and processed independently, and only when they all have finished will the parent job be re-enqueued to continue the workflow
71
+ * **Iterable Steps**
72
+ define steps that iterate over some collection fully until moving on to the next step
73
+ * **Persisted Attributes**
74
+ when retrying jobs at later steps, we need to ensure that data created in previous steps is still available to later steps on retry.
75
+ * **Transactionally Staged Jobs**
76
+ enqueue additional jobs within the acidic transaction safely
77
+ * **Custom Idempotency Keys**
78
+ use something other than the job ID for the idempotency key of the job run
79
+ * **Sidekiq Callbacks**
80
+ bring ActiveJob-like callbacks into your pure Sidekiq Workers
81
+ * **Run Finished Callbacks**
82
+ set callbacks for when a job run finishes fully
91
83
 
92
84
 
93
85
  ### Transactional Steps
94
86
 
95
- The first and foundational feature `acidic_job` provides is the `with_acidic_workflow` method, which takes a block of transactional step methods (defined via the `step`) method:
87
+ The first and foundational feature `acidic_job` provides is the `with_acidity` method, which takes a block of transactional step methods (defined via the `step`) method:
96
88
 
97
89
  ```ruby
98
- class RideCreateJob < AcidicJob::Base
90
+ class RideCreateJob < ActiveJob::Base
91
+ include AcidicJob
92
+
99
93
  def perform(user_id, ride_params)
100
94
  @user = User.find(user_id)
101
95
  @params = ride_params
102
96
 
103
- with_acidic_workflow persisting: { ride: nil } do |workflow|
104
- workflow.step :create_ride_and_audit_record
105
- workflow.step :create_stripe_charge
106
- workflow.step :send_receipt
97
+ with_acidity providing: { ride: nil } do
98
+ step :create_ride_and_audit_record
99
+ step :create_stripe_charge
100
+ step :send_receipt
107
101
  end
108
102
  end
109
103
 
110
- private
111
-
112
104
  def create_ride_and_audit_record
113
105
  # ...
114
106
  end
@@ -123,42 +115,46 @@ class RideCreateJob < AcidicJob::Base
123
115
  end
124
116
  ```
125
117
 
126
- `with_acidic_workflow` takes only the `persisting:` named parameter (optionally) and a block (required) where you define the steps of this operation. `step` simply takes the name of a method available in the job. That's all!
118
+ `with_acidity` takes only the `providing:` named parameter and a block where you define the steps of this operation. `step` simply takes the name of a method available in the job. That's all!
127
119
 
128
- Now, each execution of this job will find or create an `AcidicJob::Run` record, which we leverage to wrap every step in a database transaction. Moreover, this database record allows us to ensure that if your job fails on step 3, when it retries, it will simply jump right back to trying to execute the method defined for the 3rd step, _**and won't even execute the first two step methods**_. This means your step methods only need to be idempotent on failure, not on success, since they will never be run again if they succeed.
120
+ Now, each execution of this job will find or create an `AcidicJob::Run` record, which we leverage to wrap every step in a database transaction. Moreover, this database record allows `acidic_job` to ensure that if your job fails on step 3, when it retries, it will simply jump right back to trying to execute the method defined for the 3rd step, and won't even execute the first two step methods. This means your step methods only need to be idempotent on failure, not on success, since they will never be run again if they succeed.
129
121
 
130
122
 
131
123
  ### Steps that Await Jobs
132
124
 
133
- By simply adding the `awaits` option to your step declarations, you can attach any number of additional, asynchronous jobs to your step. This is profoundly powerful, as it means that you can define a workflow where step 2 is started _if and only if_ step 1 succeeds, but step 1 can have 3 different jobs enqueued on 3 different queues, each running in parallel. Once (and only once) all 3 jobs succeed, `AcidicJob` will re-enqueue the parent job and it will move on to step 2. That's right, you can have workers that are _executed in parallel_, **on separate queues**, and _asynchronously_, but are still **blocking**—as a group—the next step in your workflow! This unlocks incredible power and flexibility for defining and structuring complex workflows and operations.
125
+ By simply adding the `awaits` option to your step declarations, you can attach any number of additional, asynchronous jobs to your step. This is profoundly powerful, as it means that you can define a workflow where step 2 is started _if and only if_ step 1 succeeds, but step 1 can have 3 different jobs enqueued on 3 different queues, each running in parallel. Once all 3 jobs succeed, `acidic_job` will re-enqueue the parent job and it will move on to step 2. That's right, you can have workers that are _executed in parallel_, **on separate queues**, and _asynchronously_, but are still **blocking**—as a group—the next step in your workflow! This unlocks incredible power and flexibility for defining and structuring complex workflows and operations.
134
126
 
135
127
  ```ruby
136
- class RideCreateJob < AcidicJob::Base
128
+ class RideCreateJob < ActiveJob::Base
129
+ include AcidicJob
130
+
137
131
  def perform(user_id, ride_params)
138
132
  @user = User.find(user_id)
139
133
  @params = ride_params
140
134
 
141
- with_acidic_workflow persisting: { ride: nil } do |workflow|
142
- workflow.step :create_ride_and_audit_record, awaits: [SomeJob, AnotherJob]
143
- workflow.step :create_stripe_charge
144
- workflow.step :send_receipt
135
+ with_acidity providing: { ride: nil } do
136
+ step :create_ride_and_audit_record, awaits: [SomeJob, AnotherJob]
137
+ step :create_stripe_charge
138
+ step :send_receipt
145
139
  end
146
140
  end
147
141
  end
148
142
  ```
149
143
 
150
- If you need to await a job that takes arguments, you can prepare that job along with its arguments using the `with` class method that `AcidicJob` will add to your jobs:
144
+ If you need to await a job that takes arguments, you can prepare that job along with its arguments using the `with` class method that `acidic_job` will add to your jobs:
151
145
 
152
146
  ```ruby
153
- class RideCreateJob < AcidicJob::Base
147
+ class RideCreateJob < ActiveJob::Base
148
+ include AcidicJob
149
+
154
150
  def perform(user_id, ride_params)
155
151
  @user = User.find(user_id)
156
152
  @params = ride_params
157
153
 
158
- with_acidic_workflow persisting: { ride: nil } do |workflow|
159
- workflow.step :create_ride_and_audit_record, awaits: awaits: [SomeJob.with('argument_1', keyword: 'value'), AnotherJob.with(1, 2, 3, some: 'thing')]
160
- workflow.step :create_stripe_charge
161
- workflow.step :send_receipt
154
+ with_acidity providing: { ride: nil } do
155
+ step :create_ride_and_audit_record, awaits: awaits: [SomeJob.with('argument_1', keyword: 'value'), AnotherJob.with(1, 2, 3, some: 'thing')]
156
+ step :create_stripe_charge
157
+ step :send_receipt
162
158
  end
163
159
  end
164
160
  end
@@ -169,20 +165,21 @@ If your step awaits multiple jobs (e.g. `awaits: [SomeJob, AnotherJob.with('argu
169
165
  In some cases, you may need to _dynamically_ determine the collection of jobs that the step should wait for; in these cases, you can pass the name of a method to the `awaits` option:
170
166
 
171
167
  ```ruby
172
- class RideCreateJob < AcidicJob::Base
168
+ class RideCreateJob < ActiveJob::Base
169
+ include AcidicJob
170
+ set_callback :finish, :after, :delete_run_record
171
+
173
172
  def perform(user_id, ride_params)
174
173
  @user = User.find(user_id)
175
174
  @params = ride_params
176
175
 
177
- with_acidic_workflow persisting: { ride: nil } do |workflow|
178
- workflow.step :create_ride_and_audit_record, awaits: :dynamic_awaits
179
- workflow.step :create_stripe_charge
180
- workflow.step :send_receipt
176
+ with_acidity providing: { ride: nil } do
177
+ step :create_ride_and_audit_record, awaits: :dynamic_awaits
178
+ step :create_stripe_charge
179
+ step :send_receipt
181
180
  end
182
181
  end
183
-
184
- private
185
-
182
+
186
183
  def dynamic_awaits
187
184
  if @params["key"].present?
188
185
  [SomeJob.with('argument_1', keyword: 'value')]
@@ -196,49 +193,49 @@ end
196
193
 
197
194
  ### Iterable Steps
198
195
 
199
- Sometimes our workflows have steps that need to iterate over a collection and perform an action for each item in the collection before moving on to the next step in the workflow. In these cases, we can use the `for_each` option when defining our step to bind that method to a specific the collection, and `AcidicJob` will pass each item into your step method for processing, keeping the same transactional guarantees as for any step. This means that if your step encounters an error in processing any item in the collection, when your job is retried, the job will jump right back to that step and right back to that item in the collection to try again.
196
+ Sometimes our workflows have steps that need to iterate over a collection and perform an action for each item in the collection before moving on to the next step in the workflow. In these cases, we can use the `for_each` option when defining our step to specific the collection, and `acidic_job` will pass each item into your step method for processing, keeping the same transactional guarantees as for any step. This means that if your step encounters an error in processing any item in the collection, when your job is retried, the job will jump right back to that step and right back to that item in the collection to try again.
200
197
 
201
198
  ```ruby
202
- class ExampleJob < AcidicJob::Base
199
+ class ExampleJob < ActiveJob::Base
200
+ include AcidicJob
201
+
203
202
  def perform(record:)
204
- with_acidic_workflow persisting: { collection: [1, 2, 3, 4, 5] } do |workflow|
205
- workflow.step :process_item, for_each: :collection
206
- workflow.step :next_step
203
+ with_acidity providing: { collection: [1, 2, 3, 4, 5] } do
204
+ step :process_item, for_each: :collection
205
+ step :next_step
207
206
  end
208
207
  end
209
-
210
- private
211
-
208
+
212
209
  def process_item(item)
213
- # do whatever work needs to be done with an individual item from `collection`
210
+ # do whatever work needs to be done with this individual item
214
211
  end
215
212
  end
216
213
  ```
217
214
 
218
- **Note:** This feature relies on the "Persisted Attributes" feature detailed below. This means that you can only iterate over collections that ActiveJob can serialize. See [the Rails Guide on `ActiveJob`](https://edgeguides.rubyonrails.org/active_job_basics.html#supported-types-for-arguments) for more info.
215
+ **Note:** This feature relies on the "Persisted Attributes" feature detailed below. This means that you can only iterate over collections that ActiveRecord can serialize.
219
216
 
220
217
 
221
218
  ### Persisted Attributes
222
219
 
223
- The `persisting` option on the `with_acidic_workflow` method allows you to create a cross-step, cross-retry context. This means that you can set an attribute in step 1, access it in step 2, have step 2 fail, have the job retry, jump directly back to step 2 on retry, and have that object still accessible. This is done by serializing all objects to a field on the `AcidicJob::Run` and manually persisting getters and setters that sync with the database record.
224
-
225
- The default pattern you should follow when defining your `perform` method is to make any values that your `step` methods need access to, but are present at the start of the `perform` method simply instance variables. You only need to mark attributes that will be set _during a step_ via `persisting`. This means, the initial value will almost always be `nil`. If you need a default initial value, however, you can always provide that value to `persisting`.
220
+ Any objects passed to the `providing` option on the `with_acidity` method are made available across retries. This means that you can set an attribute in step 1, access it in step 2, have step 2 fail, have the job retry, jump directly back to step 2 on retry, and have that object still accessible. This is done by serializing all objects to a field on the `AcidicJob::Run` and manually providing getters and setters that sync with the database record.
226
221
 
227
222
  ```ruby
228
- class RideCreateJob < AcidicJob::Base
223
+ class RideCreateJob < ActiveJob::Base
224
+ include AcidicJob
225
+
229
226
  def perform(user_id, ride_params)
230
227
  @user = User.find(user_id)
231
228
  @params = ride_params
232
229
 
233
- with_acidic_workflow persisting: { ride: nil } do |workflow|
234
- workflow.step :create_ride_and_audit_record
235
- workflow.step :create_stripe_charge
236
- workflow.step :send_receipt
230
+ with_acidity providing: { ride: nil } do
231
+ step :create_ride_and_audit_record
232
+ step :create_stripe_charge
233
+ step :send_receipt
237
234
  end
238
235
  end
239
236
 
240
237
  def create_ride_and_audit_record
241
- self.ride = @user.rides.create(@params)
238
+ self.ride = Ride.create!
242
239
  end
243
240
 
244
241
  def create_stripe_charge
@@ -249,27 +246,31 @@ class RideCreateJob < AcidicJob::Base
249
246
  end
250
247
  ```
251
248
 
252
- **Note:** This does mean that you are restricted to objects that can be serialized by **`ActiveJob`** (for more info, see [the Rails Guide on `ActiveJob`](https://edgeguides.rubyonrails.org/active_job_basics.html#supported-types-for-arguments)). This means you can persist ActiveRecord models, and any simple Ruby data types, but you can't persist things like Procs or custom class instances, for example.
249
+ **Note:** This does mean that you are restricted to objects that can be serialized by **`ActiveJob`** (for more info, see [here](https://edgeguides.rubyonrails.org/active_job_basics.html#supported-types-for-arguments)). This means you can persist ActiveRecord models, and any simple Ruby data types, but you can't persist things like Procs or custom class instances, for example.
253
250
 
254
251
  **Note:** You will note the use of `self.ride = ...` in the code sample above. In order to call the attribute setter method that will sync with the database record, you _must_ use this style. `@ride = ...` and/or `ride = ...` will both fail to sync the value with the database record.
255
252
 
253
+ The default pattern you should follow when defining your `perform` method is to make any values that your `step` methods need access to, but are present at the start of the `perform` method simply instance variables. You only need to `provide` attributes that will be set _during a step_. This means, the initial value will almost always be `nil`.
254
+
256
255
 
257
256
  ### Transactionally Staged Jobs
258
257
 
259
258
  A standard problem when inside of database transactions is enqueuing other jobs. On the one hand, you could enqueue a job inside of a transaction that then rollbacks, which would leave that job to fail and retry and fail. On the other hand, you could enqueue a job that is picked up before the transaction commits, which would mean the records are not yet available to this job.
260
259
 
261
- In order to mitigate against such issues without forcing you to use a database-backed job queue, `AcidicJob` provides `perform_acidicly` and `deliver_acidicly` methods to "transactionally stage" enqueuing other jobs from within a step (whether another `ActiveJob` or a `Sidekiq::Worker` or an `ActionMailer` delivery). These methods will create a new `AcidicJob::Run` record, but inside of the database transaction of the `step`. Upon commit of that transaction, a model callback pushes the job to your actual job queue. Once the job has been successfully performed, the `AcidicJob::Run` record is deleted so that this table doesn't grow unbounded and unnecessarily.
260
+ In order to mitigate against such issues without forcing you to use a database-backed job queue, `acidic_job` provides `perform_acidicly` and `deliver_acidicly` methods to "transactionally stage" enqueuing other jobs from within a step (whether another ActiveJob or a Sidekiq::Worker or an ActionMailer delivery). These methods will create a new `AcidicJob::Run` record, but inside of the database transaction of the `step`. Upon commit of that transaction, a model callback pushes the job to your actual job queue. Once the job has been successfully performed, the `AcidicJob::Run` record is deleted so that this table doesn't grow unbounded and unnecessarily.
262
261
 
263
262
  ```ruby
264
- class RideCreateJob < AcidicJob::Base
263
+ class RideCreateJob < ActiveJob::Base
264
+ include AcidicJob
265
+
265
266
  def perform(user_id, ride_params)
266
267
  @user = User.find(user_id)
267
268
  @params = ride_params
268
269
 
269
- with_acidic_workflow persisting: { ride: nil } do |workflow|
270
- workflow.step :create_ride_and_audit_record
271
- workflow.step :create_stripe_charge
272
- workflow.step :send_receipt
270
+ with_acidity providing: { ride: nil } do
271
+ step :create_ride_and_audit_record
272
+ step :create_stripe_charge
273
+ step :send_receipt
273
274
  end
274
275
  end
275
276
 
@@ -289,7 +290,8 @@ By default, `AcidicJob` uses the job identifier provided by the queueing system
289
290
  Firstly, you can configure your job class to explicitly use either the job identifier or the job arguments as the foundation for the idempotency key. A job class that calls the `acidic_by_job_id` class method (which is the default behavior) will simply make the job run's idempotency key the job's identifier:
290
291
 
291
292
  ```ruby
292
- class ExampleJob < AcidicJob::Base
293
+ class ExampleJob < ActiveJob::Base
294
+ include AcidicJob
293
295
  acidic_by_job_id
294
296
 
295
297
  def perform
@@ -297,11 +299,12 @@ class ExampleJob < AcidicJob::Base
297
299
  end
298
300
  ```
299
301
 
300
- Conversely, a job class can use the `acidic_by_job_arguments` method to configure that job class to use the arguments passed to the job as the foundation for the job run's idempotency key:
302
+ Conversely, a job class can use the `acidic_by_job_args` method to configure that job class to use the arguments passed to the job as the foundation for the job run's idempotency key:
301
303
 
302
304
  ```ruby
303
- class ExampleJob < AcidicJob::Base
304
- acidic_by_job_arguments
305
+ class ExampleJob < ActiveJob::Base
306
+ include AcidicJob
307
+ acidic_by_job_args
305
308
 
306
309
  def perform(arg_1, arg_2)
307
310
  # the idempotency key will be based on whatever the values of `arg_1` and `arg_2` are
@@ -309,54 +312,51 @@ class ExampleJob < AcidicJob::Base
309
312
  end
310
313
  ```
311
314
 
312
- These options cover the two common situations, but sometimes our systems need finer-grained control. For example, our job might take some record as the job argument, but we need to use a combination of the record identifier and record status as the foundation for the idempotency key. In these cases you can pass a `Proc` or a `Block` to an `acidic_by` class method. This code will be executed in the context of the newly initialized job instance, so you will have access to whatever data the job is initialized with (like the `arguments`, for example):
315
+ These options cover the two common situations, but sometimes our systems need finer-grained control. For example, our job might take some record as the job argument, but we need to use a combination of the record identifier and record status as the foundation for the idempotency key. In these cases you can pass a `Proc` to an `acidic_by` class method:
313
316
 
314
317
  ```ruby
315
- class ExampleJob < AcidicJob::Base
316
- acidic_by do
317
- record = arguments.first[:record]
318
- [record.id, record.status]
319
- end
318
+ class ExampleJob < ActiveJob::Base
319
+ include AcidicJob
320
+ acidic_by ->(record:) { [record.id, record.status] }
320
321
 
321
322
  def perform(record:)
322
- # ...
323
+ # the idempotency key will be based on whatever the values of `record.id` and `record.status` are
323
324
  end
324
325
  end
325
326
  ```
326
327
 
327
- > **Note:** The `acidic_by` proc/block _will be executed in the context of the job instance_ at the moment the job is initialized. This means it will **not** have access to any instance variables defined in your `perform` method.
328
+ > **Note:** The signature of the `acidic_by` proc _needs to match the signature_ of the job's `perform` method.
328
329
 
329
330
 
330
331
  ### Sidekiq Callbacks
331
332
 
332
- In order to ensure that staged `AcidicJob::Run` records are only destroyed once the related job has been successfully performed, whether it is an ActiveJob or a Sidekiq Worker, `AcidicJob` also extends Sidekiq to support the [ActiveJob callback interface](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
333
+ In order to ensure that `AcidicJob::Staged` records are only destroyed once the related job has been successfully performed, whether it is an ActiveJob or a Sidekiq Worker, `acidic_job` also extends Sidekiq to support the [ActiveJob callback interface](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
333
334
 
334
- This allows us to use an `after_perform` callback to delete the `AcidicJob::Run` record, whether you are using the gem with ActiveJob or pure Sidekiq Workers. Of course, this means that you can add your own callbacks to any jobs or workers that include the `AcidicJob` module as well.
335
+ This allows `acidic_job` to use an `after_perform` callback to delete the `AcidicJob::Staged` record, whether you are using the gem with ActiveJob or pure Sidekiq Workers. Of course, this means that you can add your own callbacks to any jobs or workers that include the `AcidicJob` module as well.
335
336
 
336
337
 
337
338
  ### Run Finished Callbacks
338
339
 
339
- When working with workflow jobs that make use of the `awaits` feature for a step, it is important to remember that the `after_perform` callback will be called _as soon as the first `awaits` step has enqueued job_, and **not** when the entire job run has finished. `AcidicJob` allows the `perform` method to finish so that the queue for the workflow job is cleared to pick up new work while the `awaits` jobs are running. `AcidicJob` will automatically re-enqueue the workflow job and progress to the next step when all of the `awaits` jobs have successfully finished. However, this means that `after_perform` **is not necessarily** the same as `after_finish`. In order to provide the opportunity for you to execute callback logic _if and only if_ a job run has finished, we provide callback hooks for the `finish` event.
340
+ When working with workflow jobs that make use of the `awaits` feature for a step, it is important to remember that the `after_perform` callback will be called _as soon as the first `awaits` step has enqueued job_, and **not** when the entire job run has finished. `acidic_job` allows the `perform` method to finish so that the queue for the workflow job is cleared to pick up new work while the `awaits` jobs are running. `acidic_job` will automatically re-enqueue the workflow job and progress to the next step when all of the `awaits` jobs have successfully finished. However, this means that `after_perform` **is not necessarily** the same as `after_finish`. In order to provide the opportunity for you to execute callback logic _if and only if_ a job run has finished, we provide callback hooks for the `finish` event.
340
341
 
341
342
  For example, you could use this hook to immediately clean up the `AcidicJob::Run` database record whenever the workflow job finishes successfully like so:
342
343
 
343
344
  ```ruby
344
- class RideCreateJob < AcidicJob::Base
345
+ class RideCreateJob < ActiveJob::Base
346
+ include AcidicJob
345
347
  set_callback :finish, :after, :delete_run_record
346
348
 
347
349
  def perform(user_id, ride_params)
348
350
  @user = User.find(user_id)
349
351
  @params = ride_params
350
352
 
351
- with_acidic_workflow persisting: { ride: nil } do |workflow|
352
- workflow.step :create_ride_and_audit_record, awaits: [SomeJob.with('argument_1', keyword: 'value')]
353
- workflow.step :create_stripe_charge, args: [1, 2, 3], kwargs: { some: 'thing' }
354
- workflow.step :send_receipt
353
+ with_acidity providing: { ride: nil } do
354
+ step :create_ride_and_audit_record, awaits: [SomeJob.with('argument_1', keyword: 'value')]
355
+ step :create_stripe_charge, args: [1, 2, 3], kwargs: { some: 'thing' }
356
+ step :send_receipt
355
357
  end
356
358
  end
357
-
358
- private
359
-
359
+
360
360
  def delete_run_record
361
361
  return unless acidic_job_run.succeeded?
362
362
 
@@ -374,11 +374,11 @@ When testing acidic jobs, you are likely to run into `ActiveRecord::TransactionI
374
374
  ActiveRecord::TransactionIsolationError: cannot set transaction isolation in a nested transaction
375
375
  ```
376
376
 
377
- This error is thrown because by default RSpec and most MiniTest test suites use database transactions to keep the test database clean between tests. The database transaction that is wrapping all of the code executed in your test is run at the standard isolation level, but `AcidicJob` then tries to create another transaction at a more conservative isolation level. You cannot have a nested transaction that runs at a different isolation level, thus, this error.
377
+ This error is thrown because by default RSpec and most MiniTest test suites use database transactions to keep the test database clean between tests. The database transaction that is wrapping all of the code executed in your test is run at the standard isolation level, but acidic jobs then try to create another transaction run at a more conservative isolation level. You cannot have a nested transaction that runs at a different isolation level, thus, this error.
378
378
 
379
379
  In order to avoid this error, you need to ensure firstly that your tests that run your acidic jobs are not using a database transaction and secondly that they use some different strategy to keep your test database clean. The [DatabaseCleaner](https://github.com/DatabaseCleaner/database_cleaner) gem is a commonly used tool to manage different strategies for keeping your test database clean. As for which strategy to use, `truncation` and `deletion` are both safe, but their speed varies based on our app's table structure (see https://github.com/DatabaseCleaner/database_cleaner#what-strategy-is-fastest). Either is fine; use whichever is faster for your app.
380
380
 
381
- In order to make this test setup simpler, `AcidicJob` provides a `Testing` module that your job tests can include. It is simple; it sets `use_transactional_tests` to `false` (if the test is an `ActiveJob::TestCase`), and ensures a transaction-safe `DatabaseCleaner` strategy is run for each of your tests. Moreover, it ensures that the system's original DatabaseCleaner configuration is maintained, options included, except that any `transaction` strategies for any ORMs are replaced with a `deletion` strategy. It does so by storing whatever the system DatabaseCleaner configuration is at the start of `before_setup` phase in an instance variable and then restores that configuration at the end of `after_teardown` phase. In between, it runs the configuration thru a pipeline that selectively replaces any `transaction` strategies with a corresponding `deletion` strategy, leaving any other configured strategies untouched.
381
+ In order to make this test setup simpler, `AcidicJob` provides a `TestCase` class that your MiniTest jobs tests can inherit from. It is simple; it inherits from `ActiveJob::TestCase`, sets `use_transactional_tests` to `false`, and ensures `DatabaseCleaner` is run for each of your tests. Moreover, it ensures that the system's original DatabaseCleaner configuration is maintained, options included, except that any `transaction` strategies for any ORMs are replaced with a `deletion` strategy. It does so by storing whatever the system DatabaseCleaner configuration is at the start of `before_setup` phase in an instance variable and then restores that configuration at the end of `after_teardown` phase. In between, it runs the configuration thru a pipeline that selectively replaces any `transaction` strategies with a corresponding `deletion` strategy, leaving any other configured strategies untouched.
382
382
 
383
383
  For those of you using RSpec, you can require the `acidic_job/rspec_configuration` file, which will configure RSpec in the exact same way I have used in my RSpec projects to allow me to test acidic jobs with either the `deletion` strategy but still have all of my other tests use the fast `transaction` strategy:
384
384
 
@@ -419,23 +419,6 @@ After checking out the repo, run `bin/setup` to install dependencies. Then, run
419
419
 
420
420
  To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
421
421
 
422
- You can run a specific combination of Rails version and Sidekiq version using one of the Gemfiles defined in the `/gemfiles` directory via the `BUNDLE_GEMFILE` ENV variable, e.g.:
423
- ```sh
424
- BUNDLE_GEMFILE=gemfiles/rails_7.0_sidekiq_6.5.gemfile bundle exec rake test
425
- ```
426
-
427
- You can likewise test only one particular test file using the `TEST` ENV variable, e.g.:
428
- ```sh
429
- TEST=test/acidic_job/extensions/noticed_test.rb
430
- ```
431
-
432
- Finally, if you need to only run one particular test case itself, use the `TESTOPTS` ENV variable with the `--name` option, e.g.:
433
- ```sh
434
- TESTOPTS="--name=test_deliver_acidicly_on_noticed_notification_with_only_database_delivery"
435
- ```
436
-
437
- These options can of course be combined to help narrow down your debugging when you find a failing test in CI.
438
-
439
422
  ## Contributing
440
423
 
441
424
  Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/acidic_job.
data/acidic_job.gemspec CHANGED
@@ -34,6 +34,7 @@ Gem::Specification.new do |spec|
34
34
  spec.add_development_dependency "minitest"
35
35
  spec.add_development_dependency "net-smtp"
36
36
  spec.add_development_dependency "noticed"
37
+ spec.add_development_dependency "psych", "> 4.0"
37
38
  spec.add_development_dependency "railties"
38
39
  spec.add_development_dependency "rake"
39
40
  spec.add_development_dependency "rubocop"
@@ -42,6 +43,7 @@ Gem::Specification.new do |spec|
42
43
  spec.add_development_dependency "sidekiq"
43
44
  spec.add_development_dependency "simplecov"
44
45
  spec.add_development_dependency "sqlite3"
46
+ spec.add_development_dependency "warning"
45
47
 
46
48
  # For more information and examples about making a new gem, checkout our
47
49
  # guide at: https://bundler.io/guides/creating_gem.html