sidekiq 7.2.4 → 7.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Changes.md +43 -0
- data/README.md +1 -1
- data/lib/generators/sidekiq/job_generator.rb +2 -0
- data/lib/sidekiq/api.rb +10 -4
- data/lib/sidekiq/capsule.rb +5 -0
- data/lib/sidekiq/cli.rb +1 -0
- data/lib/sidekiq/client.rb +4 -1
- data/lib/sidekiq/config.rb +7 -1
- data/lib/sidekiq/deploy.rb +2 -0
- data/lib/sidekiq/embedded.rb +2 -0
- data/lib/sidekiq/fetch.rb +1 -1
- data/lib/sidekiq/iterable_job.rb +55 -0
- data/lib/sidekiq/job/interrupt_handler.rb +24 -0
- data/lib/sidekiq/job/iterable/active_record_enumerator.rb +53 -0
- data/lib/sidekiq/job/iterable/csv_enumerator.rb +47 -0
- data/lib/sidekiq/job/iterable/enumerators.rb +135 -0
- data/lib/sidekiq/job/iterable.rb +231 -0
- data/lib/sidekiq/job.rb +13 -2
- data/lib/sidekiq/job_logger.rb +22 -11
- data/lib/sidekiq/job_retry.rb +6 -1
- data/lib/sidekiq/job_util.rb +2 -0
- data/lib/sidekiq/metrics/query.rb +2 -0
- data/lib/sidekiq/metrics/shared.rb +2 -0
- data/lib/sidekiq/metrics/tracking.rb +13 -5
- data/lib/sidekiq/middleware/current_attributes.rb +29 -11
- data/lib/sidekiq/middleware/modules.rb +2 -0
- data/lib/sidekiq/monitor.rb +2 -1
- data/lib/sidekiq/processor.rb +11 -1
- data/lib/sidekiq/redis_client_adapter.rb +8 -5
- data/lib/sidekiq/redis_connection.rb +33 -2
- data/lib/sidekiq/ring_buffer.rb +2 -0
- data/lib/sidekiq/systemd.rb +2 -0
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/action.rb +2 -1
- data/lib/sidekiq/web/application.rb +9 -4
- data/lib/sidekiq/web/helpers.rb +53 -7
- data/lib/sidekiq/web.rb +48 -1
- data/lib/sidekiq.rb +2 -1
- data/sidekiq.gemspec +2 -1
- data/web/assets/javascripts/application.js +6 -1
- data/web/assets/javascripts/dashboard-charts.js +22 -12
- data/web/assets/javascripts/dashboard.js +1 -1
- data/web/assets/stylesheets/application.css +13 -1
- data/web/locales/tr.yml +101 -0
- data/web/views/dashboard.erb +6 -6
- data/web/views/layout.erb +6 -6
- data/web/views/metrics.erb +4 -4
- data/web/views/metrics_for_job.erb +4 -4
- metadata +26 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9f5b830734b904971e3969d347d0447564003a2afc743d681df9c2285e5e8681
|
4
|
+
data.tar.gz: 462e557d3093492eed4f4fb234f6015e52bb024e44398e617e5d8b6baebdf1ac
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d2b3bac182022693b4de09fe220fdae54a2e9e1ac68f0ce5d12d8e5b08f55d873cd62d7de4b40f0170e918f79c389501d29cc57ea691e78c98d9bb9ba4d4b709
|
7
|
+
data.tar.gz: 57d2312adca3de676e9c5fb30cd5ee8ed44270a0e0f460b2720d26cf5771c3144a7815db7694563d4d1d34185d6c3382448cdfa750a5a72449ffdcba1c889ab1
|
data/Changes.md
CHANGED
@@ -2,6 +2,49 @@
|
|
2
2
|
|
3
3
|
[Sidekiq Changes](https://github.com/sidekiq/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/sidekiq/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/sidekiq/sidekiq/blob/main/Ent-Changes.md)
|
4
4
|
|
5
|
+
7.3.1
|
6
|
+
----------
|
7
|
+
|
8
|
+
- Don't count job interruptions as failures in metrics [#6386]
|
9
|
+
- Add frozen string literal to a number of .rb files.
|
10
|
+
- Fix frozen string error with style_tag and script_tag [#6371]
|
11
|
+
- Fix an error on Ruby 2.7 because of usage of `Hash#except` [#6376]
|
12
|
+
|
13
|
+
7.3.0
|
14
|
+
----------
|
15
|
+
|
16
|
+
- **NEW FEATURE** Add `Sidekiq::IterableJob`, iteration support for long-running jobs. [#6286, fatkodima]
|
17
|
+
Iterable jobs are interruptible and can restart quickly if
|
18
|
+
running during a deploy. You must ensure that `each_iteration`
|
19
|
+
doesn't take more than Sidekiq's `-t` timeout (default: 25 seconds). Iterable jobs must not implement `perform`.
|
20
|
+
```ruby
|
21
|
+
class ProcessArrayJob
|
22
|
+
include Sidekiq::IterableJob
|
23
|
+
def build_enumerator(*args, **kwargs)
|
24
|
+
array_enumerator(args, **kwargs)
|
25
|
+
end
|
26
|
+
def each_iteration(arg)
|
27
|
+
puts arg
|
28
|
+
end
|
29
|
+
end
|
30
|
+
ProcessArrayJob.perform_async(1, 2, 3)
|
31
|
+
```
|
32
|
+
See the [Iteration](//github.com/sidekiq/sidekiq/wiki/Iteration) wiki page and the RDoc in `Sidekiq::IterableJob`.
|
33
|
+
This feature should be considered BETA until the next minor release.
|
34
|
+
- **SECURITY** The Web UI no longer allows extensions to use `<script>`.
|
35
|
+
Adjust CSP to disallow inline scripts within the Web UI. Please see
|
36
|
+
`examples/webui-ext` for how to register Web UI extensions and use
|
37
|
+
dynamic CSS and JS. This will make Sidekiq immune to XSS attacks. [#6270]
|
38
|
+
- Add config option, `:skip_default_job_logging` to disable Sidekiq's default
|
39
|
+
start/finish job logging. [#6200]
|
40
|
+
- Allow `Sidekiq::Limiter.redis` to use Redis Cluster [#6288]
|
41
|
+
- Retain CurrentAttributeѕ after inline execution [#6307]
|
42
|
+
- Ignore non-existent CurrentAttributes attributes when restoring [#6341]
|
43
|
+
- Raise default Redis {read,write,connect} timeouts from 1 to 3 seconds
|
44
|
+
to minimize ReadTimeoutErrors [#6162]
|
45
|
+
- Add `logger` as a dependency since it will become bundled in Ruby 3.5 [#6320]
|
46
|
+
- Ignore unsupported locales in the Web UI [#6313]
|
47
|
+
|
5
48
|
7.2.4
|
6
49
|
----------
|
7
50
|
|
data/README.md
CHANGED
@@ -86,7 +86,7 @@ Useful resources:
|
|
86
86
|
* Occasional announcements are made to the [@sidekiq](https://ruby.social/@sidekiq) Mastodon account.
|
87
87
|
* The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
|
88
88
|
|
89
|
-
Every
|
89
|
+
Every Thursday morning is Sidekiq office hour: I video chat and answer questions.
|
90
90
|
See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
|
91
91
|
|
92
92
|
Contributing
|
data/lib/sidekiq/api.rb
CHANGED
@@ -813,6 +813,8 @@ module Sidekiq
|
|
813
813
|
|
814
814
|
# Add the given job to the Dead set.
|
815
815
|
# @param message [String] the job data as JSON
|
816
|
+
# @option opts :notify_failure [Boolean] (true) Whether death handlers should be called
|
817
|
+
# @option opts :ex [Exception] (RuntimeError) An exception to pass to the death handlers
|
816
818
|
def kill(message, opts = {})
|
817
819
|
now = Time.now.to_f
|
818
820
|
Sidekiq.redis do |conn|
|
@@ -825,10 +827,14 @@ module Sidekiq
|
|
825
827
|
|
826
828
|
if opts[:notify_failure] != false
|
827
829
|
job = Sidekiq.load_json(message)
|
828
|
-
|
829
|
-
|
830
|
+
if opts[:ex]
|
831
|
+
ex = opt[:ex]
|
832
|
+
else
|
833
|
+
ex = RuntimeError.new("Job killed by API")
|
834
|
+
ex.set_backtrace(caller)
|
835
|
+
end
|
830
836
|
Sidekiq.default_configuration.death_handlers.each do |handle|
|
831
|
-
handle.call(job,
|
837
|
+
handle.call(job, ex)
|
832
838
|
end
|
833
839
|
end
|
834
840
|
true
|
@@ -1199,7 +1205,7 @@ module Sidekiq
|
|
1199
1205
|
@hsh.send(*all)
|
1200
1206
|
end
|
1201
1207
|
|
1202
|
-
def respond_to_missing?(name)
|
1208
|
+
def respond_to_missing?(name, *args)
|
1203
1209
|
@hsh.respond_to?(name)
|
1204
1210
|
end
|
1205
1211
|
end
|
data/lib/sidekiq/capsule.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
require "sidekiq/component"
|
2
4
|
|
3
5
|
module Sidekiq
|
@@ -17,6 +19,7 @@ module Sidekiq
|
|
17
19
|
# end
|
18
20
|
class Capsule
|
19
21
|
include Sidekiq::Component
|
22
|
+
extend Forwardable
|
20
23
|
|
21
24
|
attr_reader :name
|
22
25
|
attr_reader :queues
|
@@ -24,6 +27,8 @@ module Sidekiq
|
|
24
27
|
attr_reader :mode
|
25
28
|
attr_reader :weights
|
26
29
|
|
30
|
+
def_delegators :@config, :[], :[]=, :fetch, :key?, :has_key?, :merge!, :dig
|
31
|
+
|
27
32
|
def initialize(name, config)
|
28
33
|
@name = name
|
29
34
|
@config = config
|
data/lib/sidekiq/cli.rb
CHANGED
data/lib/sidekiq/client.rb
CHANGED
@@ -248,9 +248,12 @@ module Sidekiq
|
|
248
248
|
def atomic_push(conn, payloads)
|
249
249
|
if payloads.first.key?("at")
|
250
250
|
conn.zadd("schedule", payloads.flat_map { |hash|
|
251
|
-
at = hash
|
251
|
+
at = hash["at"].to_s
|
252
252
|
# ActiveJob sets this but the job has not been enqueued yet
|
253
253
|
hash.delete("enqueued_at")
|
254
|
+
# TODO: Use hash.except("at") when support for Ruby 2.7 is dropped
|
255
|
+
hash = hash.dup
|
256
|
+
hash.delete("at")
|
254
257
|
[at, Sidekiq.dump_json(hash)]
|
255
258
|
})
|
256
259
|
else
|
data/lib/sidekiq/config.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
require "forwardable"
|
2
4
|
|
3
5
|
require "set"
|
@@ -17,6 +19,10 @@ module Sidekiq
|
|
17
19
|
poll_interval_average: nil,
|
18
20
|
average_scheduled_poll_interval: 5,
|
19
21
|
on_complex_arguments: :raise,
|
22
|
+
iteration: {
|
23
|
+
max_job_runtime: nil,
|
24
|
+
retry_backoff: 0
|
25
|
+
},
|
20
26
|
error_handlers: [],
|
21
27
|
death_handlers: [],
|
22
28
|
lifecycle_events: {
|
@@ -52,7 +58,7 @@ module Sidekiq
|
|
52
58
|
@capsules = {}
|
53
59
|
end
|
54
60
|
|
55
|
-
def_delegators :@options, :[], :[]=, :fetch, :key?, :has_key?, :merge
|
61
|
+
def_delegators :@options, :[], :[]=, :fetch, :key?, :has_key?, :merge!, :dig
|
56
62
|
attr_reader :capsules
|
57
63
|
|
58
64
|
def to_json(*)
|
data/lib/sidekiq/deploy.rb
CHANGED
data/lib/sidekiq/embedded.rb
CHANGED
data/lib/sidekiq/fetch.rb
CHANGED
@@ -44,7 +44,7 @@ module Sidekiq # :nodoc:
|
|
44
44
|
return nil
|
45
45
|
end
|
46
46
|
|
47
|
-
queue, job = redis { |conn| conn.blocking_call(
|
47
|
+
queue, job = redis { |conn| conn.blocking_call(TIMEOUT, "brpop", *qs, TIMEOUT) }
|
48
48
|
UnitOfWork.new(queue, job, config) if queue
|
49
49
|
end
|
50
50
|
|
@@ -0,0 +1,55 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "sidekiq/job/iterable"
|
4
|
+
|
5
|
+
# Iterable jobs are ones which provide a sequence to process using
|
6
|
+
# `build_enumerator(*args, cursor: cursor)` and then process each
|
7
|
+
# element of that sequence in `each_iteration(item, *args)`.
|
8
|
+
#
|
9
|
+
# The job is kicked off as normal:
|
10
|
+
#
|
11
|
+
# ProcessUserSet.perform_async(123)
|
12
|
+
#
|
13
|
+
# but instead of calling `perform`, Sidekiq will call:
|
14
|
+
#
|
15
|
+
# enum = ProcessUserSet#build_enumerator(123, cursor:nil)
|
16
|
+
#
|
17
|
+
# Your Enumerator must yield `(object, updated_cursor)` and
|
18
|
+
# Sidekiq will call your `each_iteration` method:
|
19
|
+
#
|
20
|
+
# ProcessUserSet#each_iteration(object, 123)
|
21
|
+
#
|
22
|
+
# After every iteration, Sidekiq will check for shutdown. If we are
|
23
|
+
# stopping, the cursor will be saved to Redis and the job re-queued
|
24
|
+
# to pick up the rest of the work upon restart. Your job will get
|
25
|
+
# the updated_cursor so it can pick up right where it stopped.
|
26
|
+
#
|
27
|
+
# enum = ProcessUserSet#build_enumerator(123, cursor: updated_cursor)
|
28
|
+
#
|
29
|
+
# The cursor object must be serializable to JSON.
|
30
|
+
#
|
31
|
+
# Note there are several APIs to help you build enumerators for
|
32
|
+
# ActiveRecord Relations, CSV files, etc. See sidekiq/job/iterable/*.rb.
|
33
|
+
module Sidekiq
|
34
|
+
module IterableJob
|
35
|
+
def self.included(base)
|
36
|
+
base.include Sidekiq::Job
|
37
|
+
base.include Sidekiq::Job::Iterable
|
38
|
+
end
|
39
|
+
|
40
|
+
# def build_enumerator(*args, cursor:)
|
41
|
+
# def each_iteration(item, *args)
|
42
|
+
|
43
|
+
# Your job can also define several callbacks during points
|
44
|
+
# in each job's lifecycle.
|
45
|
+
#
|
46
|
+
# def on_start
|
47
|
+
# def on_resume
|
48
|
+
# def on_stop
|
49
|
+
# def on_complete
|
50
|
+
# def around_iteration
|
51
|
+
#
|
52
|
+
# To keep things simple and compatible, this is the same
|
53
|
+
# API as the `sidekiq-iteration` gem.
|
54
|
+
end
|
55
|
+
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Sidekiq
|
4
|
+
module Job
|
5
|
+
class InterruptHandler
|
6
|
+
include Sidekiq::ServerMiddleware
|
7
|
+
|
8
|
+
def call(instance, hash, queue)
|
9
|
+
yield
|
10
|
+
rescue Interrupted
|
11
|
+
logger.debug "Interrupted, re-queueing..."
|
12
|
+
c = Sidekiq::Client.new
|
13
|
+
c.push(hash)
|
14
|
+
raise Sidekiq::JobRetry::Skip
|
15
|
+
end
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
Sidekiq.configure_server do |config|
|
21
|
+
config.server_middleware do |chain|
|
22
|
+
chain.add Sidekiq::Job::InterruptHandler
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,53 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Sidekiq
|
4
|
+
module Job
|
5
|
+
module Iterable
|
6
|
+
# @api private
|
7
|
+
class ActiveRecordEnumerator
|
8
|
+
def initialize(relation, cursor: nil, **options)
|
9
|
+
@relation = relation
|
10
|
+
@cursor = cursor
|
11
|
+
@options = options
|
12
|
+
end
|
13
|
+
|
14
|
+
def records
|
15
|
+
Enumerator.new(-> { @relation.count }) do |yielder|
|
16
|
+
@relation.find_each(**@options, start: @cursor) do |record|
|
17
|
+
yielder.yield(record, record.id)
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
def batches
|
23
|
+
Enumerator.new(-> { @relation.count }) do |yielder|
|
24
|
+
@relation.find_in_batches(**@options, start: @cursor) do |batch|
|
25
|
+
yielder.yield(batch, batch.last.id)
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
def relations
|
31
|
+
Enumerator.new(-> { relations_size }) do |yielder|
|
32
|
+
# Convenience to use :batch_size for all the
|
33
|
+
# ActiveRecord batching methods.
|
34
|
+
options = @options.dup
|
35
|
+
options[:of] ||= options.delete(:batch_size)
|
36
|
+
|
37
|
+
@relation.in_batches(**options, start: @cursor) do |relation|
|
38
|
+
last_record = relation.last
|
39
|
+
yielder.yield(relation, last_record.id)
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
private
|
45
|
+
|
46
|
+
def relations_size
|
47
|
+
batch_size = @options[:batch_size] || 1000
|
48
|
+
(@relation.count + batch_size - 1) / batch_size # ceiling division
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Sidekiq
|
4
|
+
module Job
|
5
|
+
module Iterable
|
6
|
+
# @api private
|
7
|
+
class CsvEnumerator
|
8
|
+
def initialize(csv)
|
9
|
+
unless defined?(CSV) && csv.instance_of?(CSV)
|
10
|
+
raise ArgumentError, "CsvEnumerator.new takes CSV object"
|
11
|
+
end
|
12
|
+
|
13
|
+
@csv = csv
|
14
|
+
end
|
15
|
+
|
16
|
+
def rows(cursor:)
|
17
|
+
@csv.lazy
|
18
|
+
.each_with_index
|
19
|
+
.drop(cursor || 0)
|
20
|
+
.to_enum { count_of_rows_in_file }
|
21
|
+
end
|
22
|
+
|
23
|
+
def batches(cursor:, batch_size: 100)
|
24
|
+
@csv.lazy
|
25
|
+
.each_slice(batch_size)
|
26
|
+
.with_index
|
27
|
+
.drop(cursor || 0)
|
28
|
+
.to_enum { (count_of_rows_in_file.to_f / batch_size).ceil }
|
29
|
+
end
|
30
|
+
|
31
|
+
private
|
32
|
+
|
33
|
+
def count_of_rows_in_file
|
34
|
+
filepath = @csv.path
|
35
|
+
return unless filepath
|
36
|
+
|
37
|
+
count = IO.popen(["wc", "-l", filepath]) do |out|
|
38
|
+
out.read.strip.to_i
|
39
|
+
end
|
40
|
+
|
41
|
+
count -= 1 if @csv.headers
|
42
|
+
count
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
@@ -0,0 +1,135 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require_relative "active_record_enumerator"
|
4
|
+
require_relative "csv_enumerator"
|
5
|
+
|
6
|
+
module Sidekiq
|
7
|
+
module Job
|
8
|
+
module Iterable
|
9
|
+
module Enumerators
|
10
|
+
# Builds Enumerator object from a given array, using +cursor+ as an offset.
|
11
|
+
#
|
12
|
+
# @param array [Array]
|
13
|
+
# @param cursor [Integer] offset to start iteration from
|
14
|
+
#
|
15
|
+
# @return [Enumerator]
|
16
|
+
#
|
17
|
+
# @example
|
18
|
+
# array_enumerator(['build', 'enumerator', 'from', 'any', 'array'], cursor: cursor)
|
19
|
+
#
|
20
|
+
def array_enumerator(array, cursor:)
|
21
|
+
raise ArgumentError, "array must be an Array" unless array.is_a?(Array)
|
22
|
+
|
23
|
+
x = array.each_with_index.drop(cursor || 0)
|
24
|
+
x.to_enum { x.size }
|
25
|
+
end
|
26
|
+
|
27
|
+
# Builds Enumerator from `ActiveRecord::Relation`.
|
28
|
+
# Each Enumerator tick moves the cursor one row forward.
|
29
|
+
#
|
30
|
+
# @param relation [ActiveRecord::Relation] relation to iterate
|
31
|
+
# @param cursor [Object] offset id to start iteration from
|
32
|
+
# @param options [Hash] additional options that will be passed to relevant
|
33
|
+
# ActiveRecord batching methods
|
34
|
+
#
|
35
|
+
# @return [ActiveRecordEnumerator]
|
36
|
+
#
|
37
|
+
# @example
|
38
|
+
# def build_enumerator(cursor:)
|
39
|
+
# active_record_records_enumerator(User.all, cursor: cursor)
|
40
|
+
# end
|
41
|
+
#
|
42
|
+
# def each_iteration(user)
|
43
|
+
# user.notify_about_something
|
44
|
+
# end
|
45
|
+
#
|
46
|
+
def active_record_records_enumerator(relation, cursor:, **options)
|
47
|
+
ActiveRecordEnumerator.new(relation, cursor: cursor, **options).records
|
48
|
+
end
|
49
|
+
|
50
|
+
# Builds Enumerator from `ActiveRecord::Relation` and enumerates on batches of records.
|
51
|
+
# Each Enumerator tick moves the cursor `:batch_size` rows forward.
|
52
|
+
# @see #active_record_records_enumerator
|
53
|
+
#
|
54
|
+
# @example
|
55
|
+
# def build_enumerator(product_id, cursor:)
|
56
|
+
# active_record_batches_enumerator(
|
57
|
+
# Comment.where(product_id: product_id).select(:id),
|
58
|
+
# cursor: cursor,
|
59
|
+
# batch_size: 100
|
60
|
+
# )
|
61
|
+
# end
|
62
|
+
#
|
63
|
+
# def each_iteration(batch_of_comments, product_id)
|
64
|
+
# comment_ids = batch_of_comments.map(&:id)
|
65
|
+
# CommentService.call(comment_ids: comment_ids)
|
66
|
+
# end
|
67
|
+
#
|
68
|
+
def active_record_batches_enumerator(relation, cursor:, **options)
|
69
|
+
ActiveRecordEnumerator.new(relation, cursor: cursor, **options).batches
|
70
|
+
end
|
71
|
+
|
72
|
+
# Builds Enumerator from `ActiveRecord::Relation` and enumerates on batches,
|
73
|
+
# yielding `ActiveRecord::Relation`s.
|
74
|
+
# @see #active_record_records_enumerator
|
75
|
+
#
|
76
|
+
# @example
|
77
|
+
# def build_enumerator(product_id, cursor:)
|
78
|
+
# active_record_relations_enumerator(
|
79
|
+
# Product.find(product_id).comments,
|
80
|
+
# cursor: cursor,
|
81
|
+
# batch_size: 100,
|
82
|
+
# )
|
83
|
+
# end
|
84
|
+
#
|
85
|
+
# def each_iteration(batch_of_comments, product_id)
|
86
|
+
# # batch_of_comments will be a Comment::ActiveRecord_Relation
|
87
|
+
# batch_of_comments.update_all(deleted: true)
|
88
|
+
# end
|
89
|
+
#
|
90
|
+
def active_record_relations_enumerator(relation, cursor:, **options)
|
91
|
+
ActiveRecordEnumerator.new(relation, cursor: cursor, **options).relations
|
92
|
+
end
|
93
|
+
|
94
|
+
# Builds Enumerator from a CSV file.
|
95
|
+
#
|
96
|
+
# @param csv [CSV] an instance of CSV object
|
97
|
+
# @param cursor [Integer] offset to start iteration from
|
98
|
+
#
|
99
|
+
# @example
|
100
|
+
# def build_enumerator(import_id, cursor:)
|
101
|
+
# import = Import.find(import_id)
|
102
|
+
# csv_enumerator(import.csv, cursor: cursor)
|
103
|
+
# end
|
104
|
+
#
|
105
|
+
# def each_iteration(csv_row)
|
106
|
+
# # insert csv_row into database
|
107
|
+
# end
|
108
|
+
#
|
109
|
+
def csv_enumerator(csv, cursor:)
|
110
|
+
CsvEnumerator.new(csv).rows(cursor: cursor)
|
111
|
+
end
|
112
|
+
|
113
|
+
# Builds Enumerator from a CSV file and enumerates on batches of records.
|
114
|
+
#
|
115
|
+
# @param csv [CSV] an instance of CSV object
|
116
|
+
# @param cursor [Integer] offset to start iteration from
|
117
|
+
# @option options :batch_size [Integer] (100) size of the batch
|
118
|
+
#
|
119
|
+
# @example
|
120
|
+
# def build_enumerator(import_id, cursor:)
|
121
|
+
# import = Import.find(import_id)
|
122
|
+
# csv_batches_enumerator(import.csv, cursor: cursor)
|
123
|
+
# end
|
124
|
+
#
|
125
|
+
# def each_iteration(batch_of_csv_rows)
|
126
|
+
# # ...
|
127
|
+
# end
|
128
|
+
#
|
129
|
+
def csv_batches_enumerator(csv, cursor:, **options)
|
130
|
+
CsvEnumerator.new(csv).batches(cursor: cursor, **options)
|
131
|
+
end
|
132
|
+
end
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|