acidic_job 0.7.7 → 1.0.0.beta.1

Sign up to get free protection for your applications and to get access to all the features.
Files changed (53) hide show
  1. checksums.yaml +4 -4
  2. data/.github/workflows/main.yml +35 -12
  3. data/.gitignore +6 -0
  4. data/.rubocop.yml +21 -0
  5. data/.tool-versions +1 -0
  6. data/Gemfile +0 -25
  7. data/Gemfile.lock +181 -73
  8. data/README.md +274 -20
  9. data/UPGRADE_GUIDE.md +81 -0
  10. data/acidic_job.gemspec +17 -2
  11. data/bin/console +5 -2
  12. data/bin/sandbox +1958 -0
  13. data/combustion/log/test.log +0 -0
  14. data/gemfiles/rails_6.1.gemfile +8 -0
  15. data/gemfiles/rails_7.0.gemfile +8 -0
  16. data/lib/acidic_job/active_kiq.rb +114 -0
  17. data/lib/acidic_job/arguments.rb +22 -0
  18. data/lib/acidic_job/base.rb +11 -0
  19. data/lib/acidic_job/errors.rb +10 -17
  20. data/lib/acidic_job/{response.rb → finished_point.rb} +5 -7
  21. data/lib/acidic_job/idempotency_key.rb +27 -0
  22. data/lib/acidic_job/logger.rb +31 -0
  23. data/lib/acidic_job/mixin.rb +253 -0
  24. data/lib/acidic_job/perform_wrapper.rb +13 -26
  25. data/lib/acidic_job/processor.rb +96 -0
  26. data/lib/acidic_job/recovery_point.rb +4 -5
  27. data/lib/acidic_job/run.rb +299 -0
  28. data/lib/acidic_job/serializer.rb +24 -0
  29. data/lib/acidic_job/serializers/exception_serializer.rb +41 -0
  30. data/lib/acidic_job/serializers/finished_point_serializer.rb +24 -0
  31. data/lib/acidic_job/serializers/job_serializer.rb +35 -0
  32. data/lib/acidic_job/serializers/range_serializer.rb +28 -0
  33. data/lib/acidic_job/serializers/recovery_point_serializer.rb +25 -0
  34. data/lib/acidic_job/serializers/worker_serializer.rb +27 -0
  35. data/lib/acidic_job/version.rb +1 -1
  36. data/lib/acidic_job/workflow.rb +78 -0
  37. data/lib/acidic_job/workflow_builder.rb +35 -0
  38. data/lib/acidic_job/workflow_step.rb +103 -0
  39. data/lib/acidic_job.rb +29 -334
  40. data/lib/generators/acidic_job/drop_tables_generator.rb +26 -0
  41. data/lib/generators/acidic_job/install_generator.rb +27 -0
  42. data/lib/generators/acidic_job/templates/create_acidic_job_runs_migration.rb.erb +19 -0
  43. data/lib/generators/{templates/create_acidic_job_keys_migration.rb.erb → acidic_job/templates/drop_acidic_job_keys_migration.rb.erb} +10 -3
  44. metadata +235 -20
  45. data/.ruby_version +0 -1
  46. data/lib/acidic_job/deliver_transactionally_extension.rb +0 -26
  47. data/lib/acidic_job/key.rb +0 -33
  48. data/lib/acidic_job/no_op.rb +0 -11
  49. data/lib/acidic_job/perform_transactionally_extension.rb +0 -33
  50. data/lib/acidic_job/sidekiq_callbacks.rb +0 -45
  51. data/lib/acidic_job/staged.rb +0 -50
  52. data/lib/generators/acidic_job_generator.rb +0 -44
  53. data/lib/generators/templates/create_staged_acidic_jobs_migration.rb.erb +0 -10
data/README.md CHANGED
@@ -1,5 +1,8 @@
1
1
  # AcidicJob
2
2
 
3
+ [![Gem Version](https://badge.fury.io/rb/acidic_job.svg)](https://badge.fury.io/rb/acidic_job)
4
+ ![main workflow](https://github.com/fractaledmind/acidic_job/actions/workflows/main.yml/badge.svg)
5
+
3
6
  ### Idempotent operations for Rails apps (for ActiveJob or Sidekiq)
4
7
 
5
8
  At the conceptual heart of basically any software are "operations"—the discrete actions the software performs. Rails provides a powerful abstraction layer for building operations in the form of `ActiveJob`, or we Rubyists can use the tried and true power of pure `Sidekiq`. With either we can easily trigger from other Ruby code throughout our Rails application (controller actions, model methods, model callbacks, etc.); we can run operations both synchronously (blocking execution and then returning its response to the caller) and asychronously (non-blocking and the caller doesn't know its response); and we can also retry a specific operation if needed seamlessly.
@@ -37,24 +40,48 @@ Or simply execute to install the gem yourself:
37
40
 
38
41
  $ bundle add acidic_job
39
42
 
40
- Then, use the following command to copy over the `AcidicJob::Key` migration file as well as the `AcidicJob::Staged` migration file.
43
+ Then, use the following command to copy over the `AcidicJob::Run` migration file.
41
44
 
42
45
  ```
43
- rails generate acidic_job
46
+ rails generate acidic_job:install
44
47
  ```
45
48
 
46
49
  ## Usage
47
50
 
48
- `AcidicJob` is a concern that you `include` into your operation jobs.
51
+ `AcidicJob` is a concern that you `include` into your base `ApplicationJob`.
49
52
 
50
53
  ```ruby
51
- class RideCreateJob < ActiveJob::Base
54
+ class ApplicationJob < ActiveJob::Base
52
55
  include AcidicJob
53
56
  end
54
57
  ```
55
58
 
59
+ This is useful because the module needs to be mixed into any and all jobs that you want to either make acidic or enqueue acidicly.
60
+
56
61
  It provides a suite of functionality that empowers you to create complex, robust, and _acidic_ jobs.
57
62
 
63
+ ### TL;DR
64
+
65
+ #### Key Features
66
+
67
+ * **Transactional Steps**
68
+ break your job into a series of steps, each of which will be run within an acidic database transaction, allowing retries to jump back to the last "recovery point".
69
+ * **Steps that Await Jobs**
70
+ have workflow steps await other jobs, which will be enqueued and processed independently, and only when they all have finished will the parent job be re-enqueued to continue the workflow
71
+ * **Iterable Steps**
72
+ define steps that iterate over some collection fully until moving on to the next step
73
+ * **Persisted Attributes**
74
+ when retrying jobs at later steps, we need to ensure that data created in previous steps is still available to later steps on retry.
75
+ * **Transactionally Staged Jobs**
76
+ enqueue additional jobs within the acidic transaction safely
77
+ * **Custom Idempotency Keys**
78
+ use something other than the job ID for the idempotency key of the job run
79
+ * **Sidekiq Callbacks**
80
+ bring ActiveJob-like callbacks into your pure Sidekiq Workers
81
+ * **Run Finished Callbacks**
82
+ set callbacks for when a job run finishes fully
83
+
84
+
58
85
  ### Transactional Steps
59
86
 
60
87
  The first and foundational feature `acidic_job` provides is the `with_acidity` method, which takes a block of transactional step methods (defined via the `step`) method:
@@ -63,8 +90,11 @@ The first and foundational feature `acidic_job` provides is the `with_acidity` m
63
90
  class RideCreateJob < ActiveJob::Base
64
91
  include AcidicJob
65
92
 
66
- def perform(ride_params)
67
- with_acidity given: { user: current_user, params: ride_params, ride: nil } do
93
+ def perform(user_id, ride_params)
94
+ @user = User.find(user_id)
95
+ @params = ride_params
96
+
97
+ with_acidity providing: { ride: nil } do
68
98
  step :create_ride_and_audit_record
69
99
  step :create_stripe_charge
70
100
  step :send_receipt
@@ -85,20 +115,119 @@ class RideCreateJob < ActiveJob::Base
85
115
  end
86
116
  ```
87
117
 
88
- `with_acidity` takes only the `given:` named parameter and a block where you define the steps of this operation. `step` simply takes the name of a method available in the job. That's all!
118
+ `with_acidity` takes only the `providing:` named parameter and a block where you define the steps of this operation. `step` simply takes the name of a method available in the job. That's all!
119
+
120
+ Now, each execution of this job will find or create an `AcidicJob::Run` record, which we leverage to wrap every step in a database transaction. Moreover, this database record allows `acidic_job` to ensure that if your job fails on step 3, when it retries, it will simply jump right back to trying to execute the method defined for the 3rd step, and won't even execute the first two step methods. This means your step methods only need to be idempotent on failure, not on success, since they will never be run again if they succeed.
121
+
122
+
123
+ ### Steps that Await Jobs
124
+
125
+ By simply adding the `awaits` option to your step declarations, you can attach any number of additional, asynchronous jobs to your step. This is profoundly powerful, as it means that you can define a workflow where step 2 is started _if and only if_ step 1 succeeds, but step 1 can have 3 different jobs enqueued on 3 different queues, each running in parallel. Once all 3 jobs succeed, `acidic_job` will re-enqueue the parent job and it will move on to step 2. That's right, you can have workers that are _executed in parallel_, **on separate queues**, and _asynchronously_, but are still **blocking**—as a group—the next step in your workflow! This unlocks incredible power and flexibility for defining and structuring complex workflows and operations.
126
+
127
+ ```ruby
128
+ class RideCreateJob < ActiveJob::Base
129
+ include AcidicJob
130
+
131
+ def perform(user_id, ride_params)
132
+ @user = User.find(user_id)
133
+ @params = ride_params
134
+
135
+ with_acidity providing: { ride: nil } do
136
+ step :create_ride_and_audit_record, awaits: [SomeJob, AnotherJob]
137
+ step :create_stripe_charge
138
+ step :send_receipt
139
+ end
140
+ end
141
+ end
142
+ ```
143
+
144
+ If you need to await a job that takes arguments, you can prepare that job along with its arguments using the `with` class method that `acidic_job` will add to your jobs:
145
+
146
+ ```ruby
147
+ class RideCreateJob < ActiveJob::Base
148
+ include AcidicJob
149
+
150
+ def perform(user_id, ride_params)
151
+ @user = User.find(user_id)
152
+ @params = ride_params
153
+
154
+ with_acidity providing: { ride: nil } do
155
+ step :create_ride_and_audit_record, awaits: awaits: [SomeJob.with('argument_1', keyword: 'value'), AnotherJob.with(1, 2, 3, some: 'thing')]
156
+ step :create_stripe_charge
157
+ step :send_receipt
158
+ end
159
+ end
160
+ end
161
+ ```
162
+
163
+ If your step awaits multiple jobs (e.g. `awaits: [SomeJob, AnotherJob.with('argument_1', keyword: 'value')]`), your top level workflow job will only continue to the next step once **all** of the jobs in your `awaits` array have finished.
164
+
165
+ In some cases, you may need to _dynamically_ determine the collection of jobs that the step should wait for; in these cases, you can pass the name of a method to the `awaits` option:
166
+
167
+ ```ruby
168
+ class RideCreateJob < ActiveJob::Base
169
+ include AcidicJob
170
+ set_callback :finish, :after, :delete_run_record
171
+
172
+ def perform(user_id, ride_params)
173
+ @user = User.find(user_id)
174
+ @params = ride_params
175
+
176
+ with_acidity providing: { ride: nil } do
177
+ step :create_ride_and_audit_record, awaits: :dynamic_awaits
178
+ step :create_stripe_charge
179
+ step :send_receipt
180
+ end
181
+ end
182
+
183
+ def dynamic_awaits
184
+ if @params["key"].present?
185
+ [SomeJob.with('argument_1', keyword: 'value')]
186
+ else
187
+ [AnotherJob.with(1, 2, 3, some: 'thing')]
188
+ end
189
+ end
190
+ end
191
+ ```
192
+
193
+
194
+ ### Iterable Steps
195
+
196
+ Sometimes our workflows have steps that need to iterate over a collection and perform an action for each item in the collection before moving on to the next step in the workflow. In these cases, we can use the `for_each` option when defining our step to specific the collection, and `acidic_job` will pass each item into your step method for processing, keeping the same transactional guarantees as for any step. This means that if your step encounters an error in processing any item in the collection, when your job is retried, the job will jump right back to that step and right back to that item in the collection to try again.
197
+
198
+ ```ruby
199
+ class ExampleJob < ActiveJob::Base
200
+ include AcidicJob
201
+
202
+ def perform(record:)
203
+ with_acidity providing: { collection: [1, 2, 3, 4, 5] } do
204
+ step :process_item, for_each: :collection
205
+ step :next_step
206
+ end
207
+ end
208
+
209
+ def process_item(item)
210
+ # do whatever work needs to be done with this individual item
211
+ end
212
+ end
213
+ ```
214
+
215
+ **Note:** This feature relies on the "Persisted Attributes" feature detailed below. This means that you can only iterate over collections that ActiveRecord can serialize.
89
216
 
90
- Now, each execution of this job will find or create an `AcidicJob::Key` record, which we leverage to wrap every step in a database transaction. Moreover, this database record allows `acidic_job` to ensure that if your job fails on step 3, when it retries, it will simply jump right back to trying to execute the method defined for the 3rd step, and won't even execute the first two step methods. This means your step methods only need to be idempotent on failure, not on success, since they will never be run again if they succeed.
91
217
 
92
218
  ### Persisted Attributes
93
219
 
94
- Any objects passed to the `given` option on the `with_acidity` method are not just made available to each of your step methods, they are made available across retries. This means that you can set an attribute in step 1, access it in step 2, have step 2 fail, have the job retry, jump directly back to step 2 on retry, and have that object still accessible. This is done by serializing all objects to a field on the `AcidicJob::Key` and manually providing getters and setters that sync with the database record.
220
+ Any objects passed to the `providing` option on the `with_acidity` method are made available across retries. This means that you can set an attribute in step 1, access it in step 2, have step 2 fail, have the job retry, jump directly back to step 2 on retry, and have that object still accessible. This is done by serializing all objects to a field on the `AcidicJob::Run` and manually providing getters and setters that sync with the database record.
95
221
 
96
222
  ```ruby
97
223
  class RideCreateJob < ActiveJob::Base
98
224
  include AcidicJob
99
225
 
100
- def perform(ride_params)
101
- with_acidity given: { ride: nil } do
226
+ def perform(user_id, ride_params)
227
+ @user = User.find(user_id)
228
+ @params = ride_params
229
+
230
+ with_acidity providing: { ride: nil } do
102
231
  step :create_ride_and_audit_record
103
232
  step :create_stripe_charge
104
233
  step :send_receipt
@@ -117,22 +246,28 @@ class RideCreateJob < ActiveJob::Base
117
246
  end
118
247
  ```
119
248
 
120
- **Note:** This does mean that you are restricted to objects that can be serialized by ActiveRecord, thus no Procs, for example.
249
+ **Note:** This does mean that you are restricted to objects that can be serialized by **`ActiveJob`** (for more info, see [here](https://edgeguides.rubyonrails.org/active_job_basics.html#supported-types-for-arguments)). This means you can persist ActiveRecord models, and any simple Ruby data types, but you can't persist things like Procs or custom class instances, for example.
250
+
251
+ **Note:** You will note the use of `self.ride = ...` in the code sample above. In order to call the attribute setter method that will sync with the database record, you _must_ use this style. `@ride = ...` and/or `ride = ...` will both fail to sync the value with the database record.
252
+
253
+ The default pattern you should follow when defining your `perform` method is to make any values that your `step` methods need access to, but are present at the start of the `perform` method simply instance variables. You only need to `provide` attributes that will be set _during a step_. This means, the initial value will almost always be `nil`.
121
254
 
122
- **Note:** You will note the use of `self.ride = ...` in the code sample above. In order to call the attribute setter method that will sync with the database record, you _must_ use this style. `@ride = ...` and/or `ride = ...` will both fail to sync the value with the datbase record.
123
255
 
124
256
  ### Transactionally Staged Jobs
125
257
 
126
258
  A standard problem when inside of database transactions is enqueuing other jobs. On the one hand, you could enqueue a job inside of a transaction that then rollbacks, which would leave that job to fail and retry and fail. On the other hand, you could enqueue a job that is picked up before the transaction commits, which would mean the records are not yet available to this job.
127
259
 
128
- In order to mitigate against such issues without forcing you to use a database-backed job queue, `acidic_job` provides `perform_transactionally` and `deliver_transactionally` methods to "transactionally stage" enqueuing other jobs from within a step (whether another ActiveJob or a Sidekiq::Worker or an ActionMailer delivery). These methods will create a new `AcidicJob::Staged` record, but inside of the database transaction of the `step`. Upon commit of that transaction, a model callback pushes the job to your actual job queue. Once the job has been successfully performed, the `AcidicJob::Staged` record is deleted so that this table doesn't grow unbounded and unnecessarily.
260
+ In order to mitigate against such issues without forcing you to use a database-backed job queue, `acidic_job` provides `perform_acidicly` and `deliver_acidicly` methods to "transactionally stage" enqueuing other jobs from within a step (whether another ActiveJob or a Sidekiq::Worker or an ActionMailer delivery). These methods will create a new `AcidicJob::Run` record, but inside of the database transaction of the `step`. Upon commit of that transaction, a model callback pushes the job to your actual job queue. Once the job has been successfully performed, the `AcidicJob::Run` record is deleted so that this table doesn't grow unbounded and unnecessarily.
129
261
 
130
262
  ```ruby
131
263
  class RideCreateJob < ActiveJob::Base
132
264
  include AcidicJob
133
265
 
134
- def perform(ride_params)
135
- with_acidity given: { user: current_user, params: ride_params, ride: nil } do
266
+ def perform(user_id, ride_params)
267
+ @user = User.find(user_id)
268
+ @params = ride_params
269
+
270
+ with_acidity providing: { ride: nil } do
136
271
  step :create_ride_and_audit_record
137
272
  step :create_stripe_charge
138
273
  step :send_receipt
@@ -142,22 +277,141 @@ class RideCreateJob < ActiveJob::Base
142
277
  # ...
143
278
 
144
279
  def send_receipt
145
- RideMailer.with(ride: @ride, user: @user).confirm_charge.delivery_transactionally
280
+ RideMailer.with(user: @user, ride: @ride).confirm_charge.delivery_acidicly
146
281
  end
147
282
  end
148
283
  ```
149
284
 
285
+
286
+ ### Custom Idempotency Keys
287
+
288
+ By default, `AcidicJob` uses the job identifier provided by the queueing system (ActiveJob or Sidekiq) as the idempotency key for the job run. The idempotency key is what is used to guarantee that no two runs of the same job occur. However, sometimes we need particular jobs to be idempotent based on some other criteria. In these cases, `AcidicJob` provides a collection of tools to allow you to ensure the idempotency of your jobs.
289
+
290
+ Firstly, you can configure your job class to explicitly use either the job identifier or the job arguments as the foundation for the idempotency key. A job class that calls the `acidic_by_job_id` class method (which is the default behavior) will simply make the job run's idempotency key the job's identifier:
291
+
292
+ ```ruby
293
+ class ExampleJob < ActiveJob::Base
294
+ include AcidicJob
295
+ acidic_by_job_id
296
+
297
+ def perform
298
+ end
299
+ end
300
+ ```
301
+
302
+ Conversely, a job class can use the `acidic_by_job_args` method to configure that job class to use the arguments passed to the job as the foundation for the job run's idempotency key:
303
+
304
+ ```ruby
305
+ class ExampleJob < ActiveJob::Base
306
+ include AcidicJob
307
+ acidic_by_job_args
308
+
309
+ def perform(arg_1, arg_2)
310
+ # the idempotency key will be based on whatever the values of `arg_1` and `arg_2` are
311
+ end
312
+ end
313
+ ```
314
+
315
+ These options cover the two common situations, but sometimes our systems need finer-grained control. For example, our job might take some record as the job argument, but we need to use a combination of the record identifier and record status as the foundation for the idempotency key. In these cases you can pass a `Proc` to an `acidic_by` class method:
316
+
317
+ ```ruby
318
+ class ExampleJob < ActiveJob::Base
319
+ include AcidicJob
320
+ acidic_by ->(record:) { [record.id, record.status] }
321
+
322
+ def perform(record:)
323
+ # the idempotency key will be based on whatever the values of `record.id` and `record.status` are
324
+ end
325
+ end
326
+ ```
327
+
328
+ > **Note:** The signature of the `acidic_by` proc _needs to match the signature_ of the job's `perform` method.
329
+
330
+
150
331
  ### Sidekiq Callbacks
151
332
 
152
333
  In order to ensure that `AcidicJob::Staged` records are only destroyed once the related job has been successfully performed, whether it is an ActiveJob or a Sidekiq Worker, `acidic_job` also extends Sidekiq to support the [ActiveJob callback interface](https://edgeguides.rubyonrails.org/active_job_basics.html#callbacks).
153
334
 
154
335
  This allows `acidic_job` to use an `after_perform` callback to delete the `AcidicJob::Staged` record, whether you are using the gem with ActiveJob or pure Sidekiq Workers. Of course, this means that you can add your own callbacks to any jobs or workers that include the `AcidicJob` module as well.
155
336
 
156
- ### Sidekiq Batches
157
337
 
158
- One final feature for those of you using Sidekiq Pro: an integrated DSL for Sidekiq Batches. By simply adding the `awaits` option to your step declarations, you can attach any number of additional, asynchronous workers to your step. This is profoundly powerful, as it means that you can define a workflow where step 2 is started _if and only if_ step 1 succeeds, but step 1 can have 3 different workers enqueued on 3 different queues, each running in parallel. Once all 3 workers succeed, `acidic_job` will move on to step 2. That's right, by leveraging the power of Sidekiq Batches, you can have workers that are executed in parallel, on separate queues, and asynchronously, but are still blocking—as a group—the next step in your workflow! This unlocks incredible power and flexibility for defining and structuring complex workflows and operations, and in my mind is the number one selling point for Sidekiq Pro.
338
+ ### Run Finished Callbacks
159
339
 
160
- In my opinion, any commercial software using Sidekiq should get Sidekiq Pro; it is _absolutely_ worth the money. If, however, you are using `acidic_job` in a non-commercial application, you could use the open-source dropin replacement for this functionality: https://github.com/breamware/sidekiq-batch
340
+ When working with workflow jobs that make use of the `awaits` feature for a step, it is important to remember that the `after_perform` callback will be called _as soon as the first `awaits` step has enqueued job_, and **not** when the entire job run has finished. `acidic_job` allows the `perform` method to finish so that the queue for the workflow job is cleared to pick up new work while the `awaits` jobs are running. `acidic_job` will automatically re-enqueue the workflow job and progress to the next step when all of the `awaits` jobs have successfully finished. However, this means that `after_perform` **is not necessarily** the same as `after_finish`. In order to provide the opportunity for you to execute callback logic _if and only if_ a job run has finished, we provide callback hooks for the `finish` event.
341
+
342
+ For example, you could use this hook to immediately clean up the `AcidicJob::Run` database record whenever the workflow job finishes successfully like so:
343
+
344
+ ```ruby
345
+ class RideCreateJob < ActiveJob::Base
346
+ include AcidicJob
347
+ set_callback :finish, :after, :delete_run_record
348
+
349
+ def perform(user_id, ride_params)
350
+ @user = User.find(user_id)
351
+ @params = ride_params
352
+
353
+ with_acidity providing: { ride: nil } do
354
+ step :create_ride_and_audit_record, awaits: [SomeJob.with('argument_1', keyword: 'value')]
355
+ step :create_stripe_charge, args: [1, 2, 3], kwargs: { some: 'thing' }
356
+ step :send_receipt
357
+ end
358
+ end
359
+
360
+ def delete_run_record
361
+ return unless acidic_job_run.succeeded?
362
+
363
+ acidic_job_run.destroy!
364
+ end
365
+ end
366
+ ```
367
+
368
+
369
+ ## Testing
370
+
371
+ When testing acidic jobs, you are likely to run into `ActiveRecord::TransactionIsolationError`s:
372
+
373
+ ```
374
+ ActiveRecord::TransactionIsolationError: cannot set transaction isolation in a nested transaction
375
+ ```
376
+
377
+ This error is thrown because by default RSpec and most MiniTest test suites use database transactions to keep the test database clean between tests. The database transaction that is wrapping all of the code executed in your test is run at the standard isolation level, but acidic jobs then try to create another transaction run at a more conservative isolation level. You cannot have a nested transaction that runs at a different isolation level, thus, this error.
378
+
379
+ In order to avoid this error, you need to ensure firstly that your tests that run your acidic jobs are not using a database transaction and secondly that they use some different strategy to keep your test database clean. The [DatabaseCleaner](https://github.com/DatabaseCleaner/database_cleaner) gem is a commonly used tool to manage different strategies for keeping your test database clean. As for which strategy to use, `truncation` and `deletion` are both safe, but their speed varies based on our app's table structure (see https://github.com/DatabaseCleaner/database_cleaner#what-strategy-is-fastest). Either is fine; use whichever is faster for your app.
380
+
381
+ In order to make this test setup simpler, `AcidicJob` provides a `TestCase` class that your MiniTest jobs tests can inherit from. It is simple; it inherits from `ActiveJob::TestCase`, sets `use_transactional_tests` to `false`, and ensures `DatabaseCleaner` is run for each of your tests. Moreover, it ensures that the system's original DatabaseCleaner configuration is maintained, options included, except that any `transaction` strategies for any ORMs are replaced with a `deletion` strategy. It does so by storing whatever the system DatabaseCleaner configuration is at the start of `before_setup` phase in an instance variable and then restores that configuration at the end of `after_teardown` phase. In between, it runs the configuration thru a pipeline that selectively replaces any `transaction` strategies with a corresponding `deletion` strategy, leaving any other configured strategies untouched.
382
+
383
+ For those of you using RSpec, you can require the `acidic_job/rspec_configuration` file, which will configure RSpec in the exact same way I have used in my RSpec projects to allow me to test acidic jobs with either the `deletion` strategy but still have all of my other tests use the fast `transaction` strategy:
384
+
385
+ ```ruby
386
+ require "database_cleaner/active_record"
387
+
388
+ # see https://github.com/DatabaseCleaner/database_cleaner#how-to-use
389
+ RSpec.configure do |config|
390
+ config.use_transactional_fixtures = false
391
+
392
+ config.before(:suite) do
393
+ DatabaseCleaner.clean_with :truncation
394
+
395
+ # Here we are defaulting to :transaction but swapping to deletion for some specs;
396
+ # if your spec or its code-under-test uses
397
+ # nested transactions then specify :transactional e.g.:
398
+ # describe "SomeWorker", :transactional do
399
+ #
400
+ DatabaseCleaner.strategy = :transaction
401
+
402
+ config.before(:context, transactional: true) { DatabaseCleaner.strategy = :deletion }
403
+ config.after(:context, transactional: true) { DatabaseCleaner.strategy = :transaction }
404
+ config.before(:context, type: :system) { DatabaseCleaner.strategy = :deletion }
405
+ config.after(:context, type: :system) { DatabaseCleaner.strategy = :transaction }
406
+ end
407
+
408
+ config.around(:each) do |example|
409
+ DatabaseCleaner.cleaning do
410
+ example.run
411
+ end
412
+ end
413
+ end
414
+ ```
161
415
 
162
416
  ## Development
163
417
 
data/UPGRADE_GUIDE.md ADDED
@@ -0,0 +1,81 @@
1
+ # AcidicJob Upgrade Guide
2
+
3
+ ## 1. Update version requirements in `Gemfile`
4
+
5
+ ```diff
6
+ - gem "acidic_job"
7
+ + gem "acidic_job", "~> 1.0.0.pre1"
8
+ ```
9
+
10
+ result:
11
+ ```
12
+ Installing acidic_job 1.0.0.pre4 (was 0.7.7)
13
+ Bundle updated!
14
+ ```
15
+
16
+ ## 2. Generate migration for new `AcidicJob::Run` model
17
+
18
+ ```bash
19
+ rails generate acidic_job
20
+ ```
21
+
22
+ result:
23
+ ```
24
+ create db/migrate/#{yyyymmddhhmmss}_create_acidic_job_runs.rb
25
+ ```
26
+
27
+ ## 3. Delete any unneeded `AcidicJob::Key` records
28
+
29
+ Typically, records that are already finished do not need to be retained. Sometimes, however, applications key finished records around for some amount of time for debugging or metrics aggregation. Whatever your application's logic is for whether or not an `AcidicJob::Key` record is still needed, for all unneeded records, delete them.
30
+
31
+ For example, this would delete all finished `Key` records over 1 month old:
32
+
33
+ ```ruby
34
+ AcidicJob::Key.where(recovery_point: AcidicJob::Key::RECOVERY_POINT_FINISHED, last_run_at: ..1.month.ago).delete_all
35
+ ```
36
+
37
+ ## 4. Migrate `AcidicJob::Key` to `AcidicJob::Run`
38
+
39
+ `AcidicJob` ships with an upgrade module that provides a script to migrate older `Key` records to the new `Run` model.
40
+
41
+ ```ruby
42
+ AcidicJob::UpgradeService.execute
43
+ ```
44
+
45
+ This script will prepare an `insert_all` command for `Run` records by mapping the older `Key` data to the new `Run` schema. It also creates the new `Run` records with the same `id` as their `Key` counterparts, and then deletes all `Key` records successfully mapped over. Any `Key` records that were failed to be mapped over will be reported, along with the exception, in the `errored_keys` portion of the resulting hash.
46
+
47
+ result:
48
+ ```
49
+ {
50
+ run_records: <Integer>,
51
+ key_records: <Integer>,
52
+ errored_keys: <Array>
53
+ }
54
+ ```
55
+
56
+ ## 5. Triage remaining `AcidicJob::Key` records
57
+
58
+ If there were any `AcidicJob::Key` records that failed to be mapped to the new `Run` model, you will need to manually triage whatever the exception was. In all likelihood, the exception would be relating to the translation of the `Key#job_args` field to the `Run#serialized_job` field, as all other fields have a fairly straight-forward mapping. If you can't resolve the issue, please open an Issue in GitHub.
59
+
60
+ ## 6. Ensure all `AcidicJob::Staged` records are processed
61
+
62
+ `AcidicJob` still ships with an upgrade module that provides the older `Key` and `Staged` records, so this functionality will still be present to handle any existing records in your database when you deploy the updated version.
63
+
64
+ ## 7. Remove the old tables
65
+
66
+ Once you have successfully migrated everything over and the new system has been running smoothly for some time, you should drop the old `acidic_job_keys` and `staged_acidic_jobs` tables. We provide a migration generator just for this purpose:
67
+
68
+ ```bash
69
+ rails generate acidic_job:drop_tables
70
+ ```
71
+
72
+ result:
73
+ ```
74
+ create db/migrate/#{yyyymmddhhmmss}_drop_old_acidic_job_tables.rb
75
+ ```
76
+
77
+ You can then run the migration to have those tables removed:
78
+
79
+ ```bash
80
+ rails db:migrate
81
+ ```
data/acidic_job.gemspec CHANGED
@@ -27,10 +27,25 @@ Gem::Specification.new do |spec|
27
27
  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
28
28
  spec.require_paths = ["lib"]
29
29
 
30
- spec.add_dependency "activerecord", ">= 4.0.0"
30
+ spec.add_dependency "activejob"
31
+ spec.add_dependency "activerecord"
31
32
  spec.add_dependency "activesupport"
32
- spec.add_development_dependency "railties", ">= 4.0"
33
+ spec.add_development_dependency "combustion"
34
+ spec.add_development_dependency "minitest"
35
+ spec.add_development_dependency "net-smtp"
36
+ spec.add_development_dependency "noticed"
37
+ spec.add_development_dependency "psych", "> 4.0"
38
+ spec.add_development_dependency "railties"
39
+ spec.add_development_dependency "rake"
40
+ spec.add_development_dependency "rubocop"
41
+ spec.add_development_dependency "rubocop-minitest"
42
+ spec.add_development_dependency "rubocop-rake"
43
+ spec.add_development_dependency "sidekiq"
44
+ spec.add_development_dependency "simplecov"
45
+ spec.add_development_dependency "sqlite3"
46
+ spec.add_development_dependency "warning"
33
47
 
34
48
  # For more information and examples about making a new gem, checkout our
35
49
  # guide at: https://bundler.io/guides/creating_gem.html
50
+ spec.metadata["rubygems_mfa_required"] = "true"
36
51
  end
data/bin/console CHANGED
@@ -3,12 +3,15 @@
3
3
 
4
4
  require "bundler/setup"
5
5
  require "acidic_job"
6
- require_relative "../test/support/setup"
7
- require_relative "../test/support/ride_create_job"
8
6
 
9
7
  # You can add fixtures and/or initialization code here to make experimenting
10
8
  # with your gem easier. You can also use a different console, if you like.
11
9
 
10
+ require "combustion"
11
+ require "sqlite3"
12
+ Combustion.path = "test/combustion"
13
+ Combustion.initialize! :active_record
14
+
12
15
  # (If you use this, don't forget to add pry to your Gemfile!)
13
16
  # require "pry"
14
17
  # Pry.start