job-iteration 0.9.0 → 0.9.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.yardopts +3 -0
- data/Gemfile +1 -0
- data/Gemfile.lock +4 -2
- data/README.md +15 -8
- data/guides/custom-enumerator.md +74 -0
- data/guides/iteration-how-it-works.md +9 -9
- data/lib/job-iteration.rb +23 -13
- data/lib/job-iteration/active_record_cursor.rb +1 -1
- data/lib/job-iteration/active_record_enumerator.rb +2 -0
- data/lib/job-iteration/csv_enumerator.rb +21 -0
- data/lib/job-iteration/integrations/resque.rb +4 -12
- data/lib/job-iteration/integrations/sidekiq.rb +6 -12
- data/lib/job-iteration/iteration.rb +4 -4
- data/lib/job-iteration/test_helper.rb +17 -5
- data/lib/job-iteration/version.rb +1 -1
- metadata +4 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 744271afc9d1de8f525a2228d4969e182118affa
|
4
|
+
data.tar.gz: fa9d46fc8fbafba32134de730525d2ada4efc784
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 83e1ca25924c4d273b1ed5a19eed81565479f90aaebdadd50c12c5c19643437bfc330798328dc10abc1a10f7fc63b755cbcbd2c5d34e162a4ff723979757edb9
|
7
|
+
data.tar.gz: 34b83d31905e210800ca4237bb786c1492842e80e3aeadad137dae40e96b9eb92d675521c8941c4c43ae419e2e0967d2b1389220164c803954eb800abc449ef8
|
data/.yardopts
ADDED
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
job-iteration (0.9.
|
4
|
+
job-iteration (0.9.1)
|
5
5
|
activejob (~> 5.2)
|
6
6
|
|
7
7
|
GEM
|
@@ -88,6 +88,7 @@ GEM
|
|
88
88
|
unicode-display_width (1.4.0)
|
89
89
|
vegas (0.1.11)
|
90
90
|
rack (>= 1.0.0)
|
91
|
+
yard (0.9.15)
|
91
92
|
|
92
93
|
PLATFORMS
|
93
94
|
ruby
|
@@ -108,6 +109,7 @@ DEPENDENCIES
|
|
108
109
|
resque
|
109
110
|
rubocop
|
110
111
|
sidekiq
|
112
|
+
yard
|
111
113
|
|
112
114
|
BUNDLED WITH
|
113
|
-
1.16.
|
115
|
+
1.16.3
|
data/README.md
CHANGED
@@ -1,5 +1,7 @@
|
|
1
1
|
# Job Iteration API
|
2
2
|
|
3
|
+
[![Build Status](https://travis-ci.com/Shopify/job-iteration.svg?branch=master)](https://travis-ci.com/Shopify/job-iteration)
|
4
|
+
|
3
5
|
Meet Iteration, an extension for [ActiveJob](https://github.com/rails/rails/tree/master/activejob) that makes your jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).
|
4
6
|
|
5
7
|
## Background
|
@@ -16,15 +18,15 @@ class SimpleJob < ActiveJob::Base
|
|
16
18
|
end
|
17
19
|
```
|
18
20
|
|
19
|
-
The job would run fairly quickly when you only have a hundred User records. But as the number of records grows, it will take longer for a job to iterate over all Users. Eventually, there will be millions of records to iterate and the job will end up taking hours
|
21
|
+
The job would run fairly quickly when you only have a hundred `User` records. But as the number of records grows, it will take longer for a job to iterate over all Users. Eventually, there will be millions of records to iterate and the job will end up taking hours or even days.
|
20
22
|
|
21
23
|
With frequent deploys and worker restarts, it would mean that a job will be either lost of started from the beginning. Some records (especially those in the beginning of the relation) will be processed more than once.
|
22
24
|
|
23
|
-
Cloud environments are also unpredictable, and there's no way to guarantee that a single job will have reserved hardware to run for hours and days. What if AWS diagnosed the instance as unhealthy and will restart it in 5 minutes? What if Kubernetes pod is getting [evicted](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)? Again, all job progress will be lost.
|
25
|
+
Cloud environments are also unpredictable, and there's no way to guarantee that a single job will have reserved hardware to run for hours and days. What if AWS diagnosed the instance as unhealthy and will restart it in 5 minutes? What if a Kubernetes pod is getting [evicted](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/)? Again, all job progress will be lost. At Shopify, we also use it to interrupt workloads safely when moving tenants between shards and move shards between regions.
|
24
26
|
|
25
|
-
Software that is designed for high availability [must be
|
27
|
+
Software that is designed for high availability [must be resilient](https://12factor.net/disposability) to interruptions that come from the infrastructure. That's exactly what Iteration brings to ActiveJob. It's been developed at Shopify to safely process long-running jobs, in Cloud, and has been working in production since May 2017.
|
26
28
|
|
27
|
-
We recommend you
|
29
|
+
We recommend that you watch one of our [conference talks](https://www.youtube.com/watch?v=XvnWjsmAl60) about the ideas and history behind Iteration API.
|
28
30
|
|
29
31
|
## Getting started
|
30
32
|
|
@@ -103,10 +105,15 @@ class CsvJob < ActiveJob::Iteration
|
|
103
105
|
end
|
104
106
|
```
|
105
107
|
|
108
|
+
Iteration hooks into Sidekiq and Resque out of the box to support graceful interruption. No extra configuration is required.
|
109
|
+
|
106
110
|
## Guides
|
107
111
|
|
108
|
-
* [Iteration: how it works](guides/iteration-how-it-works.md)
|
109
|
-
* [Best practices](guides/best-practices.md)
|
112
|
+
* [Iteration: how it works](guides/iteration-how-it-works.md)
|
113
|
+
* [Best practices](guides/best-practices.md)
|
114
|
+
* [Writing custom enumerator](guides/custom-enumerator.md)
|
115
|
+
|
116
|
+
For more detailed documentation, see [rubydoc](https://www.rubydoc.info/github/Shopify/job-iteration).
|
110
117
|
|
111
118
|
## Requirements
|
112
119
|
|
@@ -132,11 +139,11 @@ There a few configuration assumptions that are required for Iteration to work wi
|
|
132
139
|
|
133
140
|
**What happens with retries?** An interruption of a job does not count as a retry. The iteration of job that caused the job to fail will be retried and progress will continue from there on.
|
134
141
|
|
135
|
-
**What happens if my iteration takes a long time?** We recommend that a single `each_iteration` should take no longer than 30 seconds. In the future, this may raise.
|
142
|
+
**What happens if my iteration takes a long time?** We recommend that a single `each_iteration` should take no longer than 30 seconds. In the future, this may raise an exception.
|
136
143
|
|
137
144
|
**Why is it important that `each_iteration` takes less than 30 seconds?** When the job worker is scheduled for restart or shutdown, it gets a notice to finish remaining unit of work. To guarantee that no progress is lost we need to make sure that `each_iteration` completes within a reasonable amount of time.
|
138
145
|
|
139
|
-
**What do I do if each iteration takes a long time, because it's doing nested operations?** If your `each_iteration` is complex, we recommend enqueuing another job. We may expose primitives in the future to do this more effectively, but this is not terribly common today. We recommend to read https://goo.gl/UobaaU to learn more about nested operations.
|
146
|
+
**What do I do if each iteration takes a long time, because it's doing nested operations?** If your `each_iteration` is complex, we recommend enqueuing another job, which will run your nested business logic. We may expose primitives in the future to do this more effectively, but this is not terribly common today. We recommend to read https://goo.gl/UobaaU to learn more about nested operations.
|
140
147
|
|
141
148
|
**Why do I use have to use this ugly helper in `build_enumerator`? Why can't you automatically infer it?** This is how the first version of the API worked. We checked the type of object returned by `build_enumerable`, and whether it was ActiveRecord Relation or an Array, we used the matching adapter. This caused opaque type branching in Iteration internals and it didn’t allow developers to craft their own Enumerators and control the cursor value. We made a decision to _always_ return Enumerator instance from `build_enumerator`. Now we provide explicit helpers to convert ActiveRecord Relation or an Array to Enumerator, and for more complex iteration flows developers can build their own `Enumerator` objects.
|
142
149
|
|
@@ -0,0 +1,74 @@
|
|
1
|
+
Iteration leverages the [Enumerator](http://ruby-doc.org/core-2.5.1/Enumerator.html) pattern from the Ruby standard library, which allows us to use almost any resource as a collection to iterate.
|
2
|
+
|
3
|
+
Consider a custom Enumerator that takes items from a Redis list. Because a Redis List is essentially a queue, we can ignore the cursor:
|
4
|
+
|
5
|
+
```ruby
|
6
|
+
class ListJob < ActiveJob::Base
|
7
|
+
include JobIteration::Iteration
|
8
|
+
|
9
|
+
def build_enumerator(*)
|
10
|
+
@redis = Redis.new
|
11
|
+
Enumerator.new |yielder|
|
12
|
+
yielder.yield @redis.lpop(key), nil
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
def each_iteration(item_from_redis)
|
17
|
+
# ...
|
18
|
+
end
|
19
|
+
end
|
20
|
+
```
|
21
|
+
|
22
|
+
But what about iterating based on a cursor? Consider this Enumerator that wraps third party API (Stripe) for paginated iteration:
|
23
|
+
|
24
|
+
```ruby
|
25
|
+
class StripeListEnumerator
|
26
|
+
# @param resource [Stripe::APIResource] The type of Stripe object to request
|
27
|
+
# @param params [Hash] Query parameters for the request
|
28
|
+
# @param options [Hash] Request options, such as API key or version
|
29
|
+
# @param cursor [String]
|
30
|
+
def initialize(resource, params: {}, options: {}, cursor:)
|
31
|
+
pagination_params = {}
|
32
|
+
pagination_params[:starting_after] = cursor unless cursor.nil?
|
33
|
+
|
34
|
+
@list = resource.public_send(:list, params.merge(pagination_params), options)
|
35
|
+
.auto_paging_each.lazy
|
36
|
+
end
|
37
|
+
|
38
|
+
def to_enumerator
|
39
|
+
to_enum(:each).lazy
|
40
|
+
end
|
41
|
+
|
42
|
+
private
|
43
|
+
|
44
|
+
# We yield our enumerator with the object id as the index so it is persisted
|
45
|
+
# as the cursor on the job. This allows us to properly set the
|
46
|
+
# `starting_after` parameter for the API request when resuming.
|
47
|
+
def each
|
48
|
+
@list.each do |item, _index|
|
49
|
+
yield item, item.id
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
```
|
54
|
+
|
55
|
+
```ruby
|
56
|
+
class StripeJob < ActiveJob::Base
|
57
|
+
include JobIteration::Iteration
|
58
|
+
|
59
|
+
def build_enumerator(params, cursor:)
|
60
|
+
StripeListEnumerator.new(
|
61
|
+
Stripe::Refund,
|
62
|
+
params: { charge: "ch_123" },
|
63
|
+
options: { api_key: "sk_test_123", stripe_version: "2018-01-18" },
|
64
|
+
cursor: cursor
|
65
|
+
).to_enumerator
|
66
|
+
end
|
67
|
+
|
68
|
+
def each_iteration(stripe_refund, _params)
|
69
|
+
# ...
|
70
|
+
end
|
71
|
+
end
|
72
|
+
```
|
73
|
+
|
74
|
+
We recommend that you read the implementation of the other enumerators that come with the library (`CsvEnumerator`, `ActiveRecordEnumerator`) to gain a better understanding of building Enumerator objects.
|
@@ -1,8 +1,8 @@
|
|
1
1
|
# Iteration: how it works
|
2
2
|
|
3
|
-
The main idea behind Iteration is to provide an API to describe jobs in interruptible manner,
|
3
|
+
The main idea behind Iteration is to provide an API to describe jobs in an interruptible manner, in contrast with implementing one massive `#perform` method that is impossible to interrupt safely.
|
4
4
|
|
5
|
-
Exposing the enumerator and the action to apply allows us to keep
|
5
|
+
Exposing the enumerator and the action to apply allows us to keep a cursor and interrupt between iterations. Let's see what this looks like with an ActiveRecord relation (and Enumerator).
|
6
6
|
|
7
7
|
1. `build_enumerator` is called, which constructs `ActiveRecordEnumerator` from an ActiveRecord relation (`Product.all`)
|
8
8
|
2. The first batch of records is loaded:
|
@@ -11,9 +11,9 @@ Exposing the enumerator and the action to apply allows us to keep the cursor and
|
|
11
11
|
SELECT `products`.* FROM `products` ORDER BY products.id LIMIT 100
|
12
12
|
```
|
13
13
|
|
14
|
-
3.
|
15
|
-
4.
|
16
|
-
5. After the last iteration is completed, we will check `job_should_exit?` which now returns `true
|
14
|
+
3. The job iterates over two records of the relation and then receives `SIGTERM` (graceful termination signal) caused by a deploy.
|
15
|
+
4. The signal handler sets a flag that makes `job_should_exit?` return `true`.
|
16
|
+
5. After the last iteration is completed, we will check `job_should_exit?` which now returns `true`.
|
17
17
|
6. The job stops iterating and pushes itself back to the queue, with the latest `cursor_position` value.
|
18
18
|
7. Next time when the job is taken from the queue, we'll load records starting from the last primary key that was processed:
|
19
19
|
|
@@ -23,18 +23,18 @@ SELECT `products`.* FROM `products` WHERE (products.id > 2) ORDER BY products.i
|
|
23
23
|
|
24
24
|
## Signals
|
25
25
|
|
26
|
-
It's critical to know UNIX signals in order to understand how interruption works. There are two main signals that Sidekiq and Resque use: `SIGTERM` and `SIGKILL`. `SIGTERM` is the graceful termination signal which means that the process should exit _soon_, not immediately. For Iteration, it means that we have time to wait for the last iteration to finish and to push job back to the queue with the last cursor position.
|
26
|
+
It's critical to know [UNIX signals](https://www.tutorialspoint.com/unix/unix-signals-traps.htm) in order to understand how interruption works. There are two main signals that Sidekiq and Resque use: `SIGTERM` and `SIGKILL`. `SIGTERM` is the graceful termination signal which means that the process should exit _soon_, not immediately. For Iteration, it means that we have time to wait for the last iteration to finish and to push job back to the queue with the last cursor position.
|
27
27
|
`SIGTERM` is what allows Iteration to work. In contrast, `SIGKILL` means immediate exit. It doesn't let the worker terminate gracefully, instead it will drop the job and exit as soon as possible.
|
28
28
|
|
29
|
-
Most of deploy strategies (Kubernetes, Heroku, Capistrano) send `SIGTERM` before
|
29
|
+
Most of the deploy strategies (Kubernetes, Heroku, Capistrano) send `SIGTERM` before shutting down a node, then wait for a timeout (usually from 30 seconds to a minute) to send `SIGKILL` if the process has not terminated yet.
|
30
30
|
|
31
31
|
Further reading: [Sidekiq signals](https://github.com/mperham/sidekiq/wiki/Signals).
|
32
32
|
|
33
33
|
## Enumerators
|
34
34
|
|
35
|
-
In the early versions of Iteration, `build_enumerator` used to return ActiveRecord relations directly, and we would infer the Enumerator based on the type of object. We used to support ActiveRecord relations, arrays and CSVs. This
|
35
|
+
In the early versions of Iteration, `build_enumerator` used to return ActiveRecord relations directly, and we would infer the Enumerator based on the type of object. We used to support ActiveRecord relations, arrays and CSVs. This made it hard to add support for other types of enumerations, and it was easy for developers to make mistakes and return an array of ActiveRecord objects, and for us starting to treat that as an array instead of as an ActiveRecord relation.
|
36
36
|
|
37
|
-
|
37
|
+
The current version of Iteration supports _any_ Enumerator. We expose helpers to build enumerators conveniently (`enumerator_builder.active_record_on_records`), but it's up for a developer to implement a custom Enumerator. Consider this example:
|
38
38
|
|
39
39
|
```ruby
|
40
40
|
class MyJob < ActiveJob::Base
|
data/lib/job-iteration.rb
CHANGED
@@ -1,33 +1,43 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
|
4
|
-
|
5
|
-
|
3
|
+
require_relative "./job-iteration/version"
|
4
|
+
require_relative "./job-iteration/enumerator_builder"
|
5
|
+
require_relative "./job-iteration/iteration"
|
6
6
|
|
7
7
|
module JobIteration
|
8
|
+
IntegrationLoadError = Class.new(StandardError)
|
9
|
+
|
8
10
|
INTEGRATIONS = [:resque, :sidekiq]
|
9
11
|
|
10
12
|
extend self
|
11
13
|
|
12
14
|
attr_accessor :max_job_runtime, :interruption_adapter
|
13
|
-
|
14
|
-
module AlwaysRunningInterruptionAdapter
|
15
|
-
extend self
|
16
|
-
|
17
|
-
def shutdown?
|
18
|
-
false
|
19
|
-
end
|
20
|
-
end
|
21
|
-
self.interruption_adapter = AlwaysRunningInterruptionAdapter
|
15
|
+
self.interruption_adapter = -> { false }
|
22
16
|
|
23
17
|
def load_integrations
|
18
|
+
loaded = nil
|
24
19
|
INTEGRATIONS.each do |integration|
|
25
20
|
begin
|
26
|
-
|
21
|
+
load_integration(integration)
|
22
|
+
if loaded
|
23
|
+
raise IntegrationLoadError,
|
24
|
+
"#{loaded} integration has already been loaded, but #{integration} is also available. " \
|
25
|
+
"Iteration will only work with one integration."
|
26
|
+
end
|
27
|
+
loaded = integration
|
27
28
|
rescue LoadError
|
28
29
|
end
|
29
30
|
end
|
30
31
|
end
|
32
|
+
|
33
|
+
def load_integration(integration)
|
34
|
+
unless INTEGRATIONS.include?(integration)
|
35
|
+
raise IntegrationLoadError,
|
36
|
+
"#{integration} integration is not supported. Available integrations: #{INTEGRATIONS.join(', ')}"
|
37
|
+
end
|
38
|
+
|
39
|
+
require_relative "./job-iteration/integrations/#{integration}"
|
40
|
+
end
|
31
41
|
end
|
32
42
|
|
33
43
|
JobIteration.load_integrations unless ENV['ITERATION_DISABLE_AUTOCONFIGURE']
|
@@ -1,6 +1,8 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
require_relative "./active_record_cursor"
|
3
3
|
module JobIteration
|
4
|
+
# Builds Enumerator based on ActiveRecord Relation. Supports enumerating on rows and batches.
|
5
|
+
# @see EnumeratorBuilder
|
4
6
|
class ActiveRecordEnumerator
|
5
7
|
def initialize(relation, columns: nil, batch_size: 100, cursor: nil)
|
6
8
|
@relation = relation
|
@@ -1,7 +1,24 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module JobIteration
|
4
|
+
# CsvEnumerator makes it possible to write an Iteration job
|
5
|
+
# that uses CSV file as a collection to Iterate.
|
6
|
+
# @example
|
7
|
+
# def build_enumerator(cursor:)
|
8
|
+
# csv = CSV.open('tmp/files', { converters: :integer, headers: true })
|
9
|
+
# JobIteration::CsvEnumerator.new(csv).rows(cursor: cursor)
|
10
|
+
# end
|
11
|
+
#
|
12
|
+
# def each_iteration(row)
|
13
|
+
# ...
|
14
|
+
# end
|
4
15
|
class CsvEnumerator
|
16
|
+
# Constructs CsvEnumerator instance based on a CSV file.
|
17
|
+
# @param [CSV] csv An instance of CSV object
|
18
|
+
# @return [JobIteration::CsvEnumerator]
|
19
|
+
# @example
|
20
|
+
# csv = CSV.open('tmp/files', { converters: :integer, headers: true })
|
21
|
+
# JobIteration::CsvEnumerator.new(csv).rows(cursor: cursor)
|
5
22
|
def initialize(csv)
|
6
23
|
unless csv.instance_of?(CSV)
|
7
24
|
raise ArgumentError, "CsvEnumerator.new takes CSV object"
|
@@ -10,6 +27,8 @@ module JobIteration
|
|
10
27
|
@csv = csv
|
11
28
|
end
|
12
29
|
|
30
|
+
# Constructs a enumerator on CSV rows
|
31
|
+
# @return [Enumerator] Enumerator instance
|
13
32
|
def rows(cursor:)
|
14
33
|
@csv.lazy
|
15
34
|
.each_with_index
|
@@ -17,6 +36,8 @@ module JobIteration
|
|
17
36
|
.to_enum { count_rows_in_file }
|
18
37
|
end
|
19
38
|
|
39
|
+
# Constructs a enumerator on batches of CSV rows
|
40
|
+
# @return [Enumerator] Enumerator instance
|
20
41
|
def batches(batch_size:, cursor:)
|
21
42
|
@csv.lazy
|
22
43
|
.each_slice(batch_size)
|
@@ -4,23 +4,15 @@ require 'resque'
|
|
4
4
|
|
5
5
|
module JobIteration
|
6
6
|
module Integrations
|
7
|
-
|
8
|
-
|
9
|
-
def initialize(*)
|
7
|
+
module ResqueIterationExtension # @private
|
8
|
+
def initialize(*) # @private
|
10
9
|
$resque_worker = self
|
11
10
|
super
|
12
11
|
end
|
13
12
|
end
|
13
|
+
# The patch is required in order to call shutdown? on a Resque::Worker instance
|
14
14
|
Resque::Worker.prepend(ResqueIterationExtension)
|
15
15
|
|
16
|
-
|
17
|
-
extend self
|
18
|
-
|
19
|
-
def shutdown?
|
20
|
-
$resque_worker.try!(:shutdown?)
|
21
|
-
end
|
22
|
-
end
|
23
|
-
|
24
|
-
JobIteration.interruption_adapter = ResqueInterruptionAdapter
|
16
|
+
JobIteration.interruption_adapter = -> { $resque_worker.try!(:shutdown?) }
|
25
17
|
end
|
26
18
|
end
|
@@ -3,19 +3,13 @@
|
|
3
3
|
require 'sidekiq'
|
4
4
|
|
5
5
|
module JobIteration
|
6
|
-
module Integrations
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
Sidekiq::CLI.instance.launcher.stopping?
|
13
|
-
else
|
14
|
-
false
|
15
|
-
end
|
6
|
+
module Integrations # @private
|
7
|
+
JobIteration.interruption_adapter = -> do
|
8
|
+
if defined?(Sidekiq::CLI) && Sidekiq::CLI.instance
|
9
|
+
Sidekiq::CLI.instance.launcher.stopping?
|
10
|
+
else
|
11
|
+
false
|
16
12
|
end
|
17
13
|
end
|
18
|
-
|
19
|
-
JobIteration.interruption_adapter = SidekiqInterruptionAdapter
|
20
14
|
end
|
21
15
|
end
|
@@ -53,7 +53,7 @@ module JobIteration
|
|
53
53
|
self.total_time = 0.0
|
54
54
|
end
|
55
55
|
|
56
|
-
def serialize
|
56
|
+
def serialize # @private
|
57
57
|
super.merge(
|
58
58
|
'cursor_position' => cursor_position,
|
59
59
|
'times_interrupted' => times_interrupted,
|
@@ -61,14 +61,14 @@ module JobIteration
|
|
61
61
|
)
|
62
62
|
end
|
63
63
|
|
64
|
-
def deserialize(job_data)
|
64
|
+
def deserialize(job_data) # @private
|
65
65
|
super
|
66
66
|
self.cursor_position = job_data['cursor_position']
|
67
67
|
self.times_interrupted = job_data['times_interrupted'] || 0
|
68
68
|
self.total_time = job_data['total_time'] || 0
|
69
69
|
end
|
70
70
|
|
71
|
-
def perform(*params)
|
71
|
+
def perform(*params) # @private
|
72
72
|
interruptible_perform(*params)
|
73
73
|
end
|
74
74
|
|
@@ -198,7 +198,7 @@ module JobIteration
|
|
198
198
|
return true
|
199
199
|
end
|
200
200
|
|
201
|
-
JobIteration.interruption_adapter.
|
201
|
+
JobIteration.interruption_adapter.call || (defined?(super) && super)
|
202
202
|
end
|
203
203
|
end
|
204
204
|
end
|
@@ -1,40 +1,52 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module JobIteration
|
4
|
+
# Include JobIteration::TestHelper to mock interruption when testing your jobs.
|
4
5
|
module TestHelper
|
5
|
-
class StoppingSupervisor
|
6
|
+
class StoppingSupervisor # @private
|
6
7
|
def initialize(stop_after_count)
|
7
8
|
@stop_after_count = stop_after_count
|
8
9
|
@calls = 0
|
9
10
|
end
|
10
11
|
|
11
|
-
def
|
12
|
+
def call
|
12
13
|
@calls += 1
|
13
14
|
(@calls % @stop_after_count) == 0
|
14
15
|
end
|
15
16
|
end
|
16
17
|
|
17
|
-
|
18
|
-
|
18
|
+
# Stubs interruption adapter to interrupt the job after every N iterations.
|
19
|
+
# @param [Integer] n_times Number of times before the job is interrupted
|
20
|
+
# @example
|
21
|
+
# test "this stuff interrupts" do
|
22
|
+
# iterate_exact_times(3.times)
|
23
|
+
# MyJob.perform_now
|
24
|
+
# end
|
19
25
|
def iterate_exact_times(n_times)
|
20
26
|
JobIteration.stubs(:interruption_adapter).returns(StoppingSupervisor.new(n_times.size))
|
21
27
|
end
|
22
28
|
|
29
|
+
# Stubs interruption adapter to interrupt the job after every sing iteration.
|
30
|
+
# @see #iterate_exact_times
|
23
31
|
def iterate_once
|
24
32
|
iterate_exact_times(1.times)
|
25
33
|
end
|
26
34
|
|
35
|
+
# Removes previous stubs and tells the job to iterate until the end.
|
27
36
|
def continue_iterating
|
28
37
|
stub_shutdown_adapter_to_return(false)
|
29
38
|
end
|
30
39
|
|
40
|
+
# Stubs the worker as already interrupted.
|
31
41
|
def mark_job_worker_as_interrupted
|
32
42
|
stub_shutdown_adapter_to_return(true)
|
33
43
|
end
|
34
44
|
|
45
|
+
private
|
46
|
+
|
35
47
|
def stub_shutdown_adapter_to_return(_value)
|
36
48
|
adapter = mock
|
37
|
-
adapter.stubs(:
|
49
|
+
adapter.stubs(:call).returns(false)
|
38
50
|
JobIteration.stubs(:interruption_adapter).returns(adapter)
|
39
51
|
end
|
40
52
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: job-iteration
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.9.
|
4
|
+
version: 0.9.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Shopify
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-
|
11
|
+
date: 2018-08-14 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -76,6 +76,7 @@ files:
|
|
76
76
|
- ".gitignore"
|
77
77
|
- ".rubocop.yml"
|
78
78
|
- ".travis.yml"
|
79
|
+
- ".yardopts"
|
79
80
|
- CODE_OF_CONDUCT.md
|
80
81
|
- Gemfile
|
81
82
|
- Gemfile.lock
|
@@ -83,6 +84,7 @@ files:
|
|
83
84
|
- README.md
|
84
85
|
- Rakefile
|
85
86
|
- guides/best-practices.md
|
87
|
+
- guides/custom-enumerator.md
|
86
88
|
- guides/iteration-how-it-works.md
|
87
89
|
- job-iteration.gemspec
|
88
90
|
- lib/job-iteration.rb
|