sidekiq-iteration 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 82316cffa840b2c9619792b6f0c5bb7ec696f964ad81edf6b0ed5861339ca064
+  data.tar.gz: 8337c0e87e6be8858d5b9d868c8a04b05f91de315be2e9aca9c40e8447c78644
+SHA512:
+  metadata.gz: 63712780bca873613cbe3ef89ff0037c3eaa5633a28382eb02427311360714a89f092e5efef29873efa75067d22f32745eea3f04d7606442e29da33e2e2e6a08
+  data.tar.gz: 4ded7fc6ab772c019154e6559027c87d389a356a16915d2c869e318fb26f5339dd98355a0f1eb422358c1cd9f7fe31bc2a70d5627a577f8f45394db15d0593b7

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## master (unreleased)
+
+## 0.1.0 (2022-11-02)
+
+- First release

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2022 fatkodima
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,265 @@
+# Sidekiq Iteration
+
+[![Build Status](https://github.com/fatkodima/sidekiq-iteration/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/fatkodima/sidekiq-iteration/actions/workflows/ci.yml)
+
+Meet Iteration, an extension for [Sidekiq](https://github.com/mperham/sidekiq) that makes your jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).
+
+## Background
+
+Imagine the following job:
+
+```ruby
+class SimpleJob
+  include Sidekiq::Job
+
+  def perform
+    User.find_each do |user|
+      user.notify_about_something
+    end
+  end
+end
+```
+
+The job would run fairly quickly when you only have a hundred `User` records. But as the number of records grows, it will take longer for a job to iterate over all Users. Eventually, there will be millions of records to iterate and the job will end up taking hours or even days.
+
+With frequent deploys and worker restarts, it would mean that a job will be either lost or restarted from the beginning. Some records (especially those in the beginning of the relation) will be processed more than once.
+
+Cloud environments are also unpredictable, and there's no way to guarantee that a single job will have reserved hardware to run for hours and days. What if AWS diagnosed the instance as unhealthy and will restart it in 5 minutes? All job progress will be lost.
+
+Software that is designed for high availability [must be resilient](https://12factor.net/disposability) to interruptions that come from the infrastructure. That's exactly what Iteration brings to Sidekiq.
+
+## Requirements
+
+- Ruby 2.7+ (if you need support for older ruby, [open an issue](https://github.com/fatkodima/sidekiq-iteration/issues/new))
+- Sidekiq 6+
+
+## Getting started
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'sidekiq-iteration'
+```
+
+And then execute:
+
+    $ bundle
+
+In the job, include `SidekiqIteration::Iteration` module and start describing the job with two methods (`build_enumerator` and `each_iteration`) instead of `perform`:
+
+```ruby
+class NotifyUsersJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(cursor:)
+    active_record_records_enumerator(User.all, cursor: cursor)
+  end
+
+  def each_iteration(user)
+    user.notify_about_something
+  end
+end
+```
+
+`each_iteration` will be called for each `User` model in `User.all` relation. The relation will be ordered by primary key, exactly like `find_each` does.
+Iteration hooks into Sidekiq out of the box to support graceful interruption. No extra configuration is required.
+
+## Examples
+
+### Job with custom arguments
+
+```ruby
+class ArgumentsJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(arg1, arg2, cursor:)
+    active_record_records_enumerator(User.all, cursor: cursor)
+  end
+
+  def each_iteration(user, arg1, arg2)
+    user.notify_about_something
+  end
+end
+
+ArgumentsJob.perform_async(arg1, arg2)
+```
+
+### Job with custom lifecycle callbacks
+
+```ruby
+class NotifyUsersJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def on_start
+    # Will be called when the job starts iterating. Called only once, for the first time.
+  end
+
+  def on_resume
+    # Called when the job resumes iterating.
+  end
+
+  def on_shutdown
+    # Called each time the job is interrupted.
+    # This can be due to throttling, `max_job_runtime` configuration, or sidekiq restarting.
+  end
+
+  def on_complete
+    # Called when the job finished iterating.
+  end
+
+  # ...
+end
+```
+
+### Iterating over batches of Active Record objects
+
+```ruby
+class BatchesJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(product_id, cursor:)
+    active_record_batches_enumerator(
+      Comment.where(product_id: product_id).select(:id),
+      cursor: cursor,
+      batch_size: 100,
+    )
+  end
+
+  def each_iteration(batch_of_comments, product_id)
+    comment_ids = batch_of_comments.map(&:id)
+    CommentService.call(comment_ids: comment_ids)
+  end
+end
+```
+
+### Iterating over batches of Active Record Relations
+
+```ruby
+class BatchesAsRelationJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(product_id, cursor:)
+    active_record_relations_enumerator(
+      Product.find(product_id).comments,
+      cursor: cursor,
+      batch_size: 100,
+    )
+  end
+
+  def each_iteration(batch_of_comments, product_id)
+    # batch_of_comments will be a Comment::ActiveRecord_Relation
+    batch_of_comments.update_all(deleted: true)
+  end
+end
+```
+
+### Iterating over arrays
+
+```ruby
+class ArrayJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(cursor:)
+    array_enumerator(['build', 'enumerator', 'from', 'any', 'array'], cursor: cursor)
+  end
+
+  def each_iteration(array_element)
+    # use array_element
+  end
+end
+```
+
+### Iterating over CSV
+
+```ruby
+class CsvJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(import_id, cursor:)
+    import = Import.find(import_id)
+    csv_enumereator(import.csv, cursor: cursor)
+  end
+
+  def each_iteration(csv_row)
+    # insert csv_row to database
+  end
+end
+```
+
+### Nested iteration
+
+```ruby
+class NestedIterationJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(cursor:)
+    nested_enumerator(
+      [
+        ->(cursor) { active_record_records_enumerator(Shop.all, cursor: cursor) },
+        ->(shop, cursor) { active_record_records_enumerator(shop.products, cursor: cursor) },
+        ->(_shop, product, cursor) { active_record_relations_enumerator(product.product_variants, cursor: cursor) }
+      ],
+      cursor: cursor
+    )
+  end
+
+  def each_iteration(product_variants_relation)
+    # do something
+  end
+end
+```
+
+## Guides
+
+* [Iteration: how it works](guides/iteration-how-it-works.md)
+* [Best practices](guides/best-practices.md)
+* [Writing custom enumerator](guides/custom-enumerator.md)
+* [Throttling](guides/throttling.md)
+
+For more detailed documentation, see [rubydoc](https://rubydoc.info/gems/sidekiq-iteration).
+
+## API
+
+Iteration job must respond to `build_enumerator` and `each_iteration` methods. `build_enumerator` must return [`Enumerator`](https://ruby-doc.org/core-3.1.2/Enumerator.htmll) object that respects the `cursor` value.
+
+## FAQ
+
+**Why can't I just iterate in `#perform` method and do whatever I want?** You can, but then your job has to comply with a long list of requirements, such as the ones above. This creates leaky abstractions more easily, when instead we can expose a more powerful abstraction for developers without exposing the underlying infrastructure.
+
+**What happens when my job is interrupted?** A checkpoint will be persisted to Redis after the current `each_iteration`, and the job will be re-enqueued. Once it's popped off the queue, the worker will work off from the next iteration.
+
+**What happens with retries?** An interruption of a job does not count as a retry. The iteration of job that caused the job to fail will be retried and progress will continue from there on.
+
+**What happens if my iteration takes a long time?** We recommend that a single `each_iteration` should take no longer than 30 seconds. In the future, this may raise an exception.
+
+**Why is it important that `each_iteration` takes less than 30 seconds?** When the job worker is scheduled for restart or shutdown, it gets a notice to finish remaining unit of work. To guarantee that no progress is lost we need to make sure that `each_iteration` completes within a reasonable amount of time.
+
+**What do I do if each iteration takes a long time, because it's doing nested operations?** If your `each_iteration` is complex, we recommend enqueuing another job, which will run your nested business logic. If `each_iteration` performs some other iterations, like iterating over child records, consider using [nested iterations](#nested-iteration).
+
+**My job has a complex flow. How do I write my own Enumerator?** See [the guide on Custom Enumerators](guides/custom-enumerator.md) for details.
+
+## Credits
+
+Thanks to [`job-iteration` gem](https://github.com/Shopify/job-iteration) for the original implementation and inspiration.
+
+## Development
+
+After checking out the repo, run `bundle install` to install dependencies and start Redis. Run `bundle exec rake` to run the linter and tests. This project uses multiple Gemfiles to test against multiple versions of Sidekiq; you can run the tests against the specific version with `BUNDLE_GEMFILE=gemfiles/sidekiq_6.gemfile bundle exec rake test`.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/fatkodima/sidekiq-iteration.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/guides/best-practices.md

@@ -0,0 +1,71 @@
+# Best practices
+
+## Batch iteration
+
+Regardless of the active record enumerator used in the task, `sidekiq-iteration` gem loads records in batches of 100 (by default).
+The following two tasks produce equivalent database queries, however `RecordsJob` task allows for more frequent interruptions by doing just one thing in the `each_iteration` method.
+
+```ruby
+# bad
+class BatchesJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(product_id, cursor:)
+    active_record_batches_enumerator(
+      Comment.where(product_id: product_id),
+      cursor: cursor,
+      batch_size: 5,
+    )
+  end
+
+  def each_iteration(batch_of_comments, product_id)
+    batch_of_comments.each(&:destroy)
+  end
+end
+
+# good
+class RecordsJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(product_id, cursor:)
+    active_record_records_enumerator(
+      Comment.where(product_id: product_id),
+      cursor: cursor,
+      batch_size: 5,
+    )
+  end
+
+  def each_iteration(comment, product_id)
+    comment.destroy
+  end
+end
+```
+
+## Max job runtime
+
+If a job is supposed to have millions of iterations and you expect it to run for hours and days, it's still a good idea to sometimes interrupt the job even if there are no interruption signals coming from deploys or the infrastructure.
+
+```ruby
+SidekiqIteration.max_job_runtime = 5.minutes # nil by default
+```
+
+Use this accessor to tweak how often you'd like the job to interrupt itself.
+
+### Per job max job runtime
+
+For more granular control, `max_job_runtime` can be set **per-job class**. This allows both incremental adoption, as well as using a conservative global setting, and an aggressive setting on a per-job basis.
+
+```ruby
+class MyJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  self.max_job_runtime = 3.minutes
+
+  # ...
+end
+```
+
+This setting will be inherited by any child classes, although it can be further overridden.

data/guides/custom-enumerator.md

@@ -0,0 +1,98 @@
+# Custom Enumerator
+
+Iteration leverages the [`Enumerator`](https://ruby-doc.org/core-3.1.2/Enumerator.html) pattern from the Ruby standard library, which allows us to use almost any resource as a collection to iterate.
+
+## Cursorless Enumerator
+
+Consider a custom Enumerator that takes items from a Redis list. Because a Redis list is essentially a queue, we can ignore the cursor:
+
+```ruby
+class ListJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(*)
+    @redis = Redis.new
+    Enumerator.new do |yielder|
+      loop do
+        item = @redis.lpop(key)
+        break unless item
+
+        yielder.yield(item, nil)
+      end
+    end
+  end
+
+  def each_iteration(item)
+    # ...
+  end
+end
+```
+
+## Enumerator with cursor
+
+But what about iterating based on a cursor? Consider this Enumerator that wraps third party API (Stripe) for paginated iteration:
+
+```ruby
+class StripeListEnumerator
+  # @param resource [Stripe::APIResource] The type of Stripe object to request
+  # @param params [Hash] Query parameters for the request
+  # @param options [Hash] Request options, such as API key or version
+  # @param cursor [String]
+  def initialize(resource, params: {}, options: {}, cursor:)
+    pagination_params = {}
+    pagination_params[:starting_after] = cursor unless cursor.nil?
+
+    # The following line makes a request, consider adding your rate limiter here.
+    @list = resource.public_send(:list, params.merge(pagination_params), options)
+  end
+
+  def to_enumerator
+    to_enum(:each).lazy
+  end
+
+  private
+
+  # We yield our enumerator with the object id as the index so it is persisted
+  # as the cursor on the job. This allows us to properly set the
+  # `starting_after` parameter for the API request when resuming.
+  def each
+    loop do
+      @list.each do |item, _index|
+        yield item, item.id
+      end
+
+      # The following line makes a request, consider adding your rate limiter here.
+      @list = @list.next_page
+
+      break if @list.empty?
+    end
+  end
+end
+```
+
+```ruby
+class StripeJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(params, cursor:)
+    StripeListEnumerator.new(
+      Stripe::Refund,
+      params: { charge: "ch_123" },
+      options: { api_key: "sk_test_123", stripe_version: "2018-01-18" },
+      cursor: cursor
+    ).to_enumerator
+  end
+
+  def each_iteration(stripe_refund, _params)
+    # ...
+  end
+end
+```
+
+## Notes
+
+We recommend that you read the implementation of the other enumerators that come with the library (`CsvEnumerator`, `ActiveRecordEnumerator`) to gain a better understanding of building Enumerator objects.
+
+Code that is written after the `yield` in a custom enumerator is not guaranteed to execute. In the case that a job is forced to exit (`max_job_runtime` is reached or sidekiq is topping), then the job is re-enqueued during the yield and the rest of the code in the enumerator does not run.

data/guides/iteration-how-it-works.md

@@ -0,0 +1,71 @@
+# Iteration: how it works
+
+The main idea behind Iteration is to provide an API to describe jobs in an interruptible manner, in contrast with implementing one massive `#perform` method that is impossible to interrupt safely.
+
+Exposing the enumerator and the action to apply allows us to keep a cursor and interrupt between iterations. Let's see what this looks like with an `ActiveRecord::Relation` (and `Enumerator`).
+
+```ruby
+class NotifyUsers
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(cursor:)
+    active_record_records_enumerator(User.all, cursor: cursor)
+  end
+
+  def each_iteration(user)
+    user.notify_about_something
+  end
+end
+```
+
+1. `build_enumerator` is called, which constructs `ActiveRecordEnumerator` from an ActiveRecord relation (`User.all`)
+2. The first batch of records is loaded:
+
+```sql
+SELECT "users".* FROM "users" ORDER BY "users"."id" LIMIT 100
+```
+
+3. The job iterates over two records of the relation and then receives `SIGTERM` (graceful termination signal) caused by a deploy.
+4. The signal handler sets a flag that makes `job_should_exit?` return `true`.
+5. After the last iteration is completed, we will check `job_should_exit?` which now returns `true`.
+6. The job stops iterating and pushes itself back to the queue, with the latest `cursor_position` value.
+7. Next time when the job is taken from the queue, we'll load records starting from the last primary key that was processed:
+
+```sql
+SELECT "users".* FROM "users" WHERE "users"."id" > 2 ORDER BY "products"."id" LIMIT 100
+```
+
+## Signals
+
+It's critical to know [UNIX signals](https://www.tutorialspoint.com/unix/unix-signals-traps.htm) in order to understand how interruption works. There are two main signals that Sidekiq use: `SIGTERM` and `SIGKILL`. `SIGTERM` is the graceful termination signal which means that the process should exit _soon_, not immediately. For Iteration, it means that we have time to wait for the last iteration to finish and to push job back to the queue with the last cursor position.
+`SIGTERM` is what allows Iteration to work. In contrast, `SIGKILL` means immediate exit. It doesn't let the worker terminate gracefully, instead it will drop the job and exit as soon as possible.
+
+Most of the deploy strategies (Heroku, Capistrano) send `SIGTERM` before shutting down a node, then wait for a timeout (usually from 30 seconds to a minute) to send `SIGKILL` if the process has not terminated yet.
+
+Further reading: [Sidekiq signals](https://github.com/mperham/sidekiq/wiki/Signals).
+
+## Enumerators
+
+Iteration supports _any_ `Enumerator`. We expose helpers to build enumerators conveniently (`active_record_records_enumerator`), but it's up for a developer to implement a custom `Enumerator`.
+
+Consider this example:
+
+```ruby
+class MyJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  def build_enumerator(cursor:)
+    Enumerator.new do
+      Redis.lpop("mylist") # or: Kafka.poll(timeout: 10.seconds)
+    end
+  end
+
+  def each_iteration(element_from_redis)
+    # ...
+  end
+end
+```
+
+Further reading: [ruby-doc](https://ruby-doc.org/core-3.1.2/Enumerator.html), [a great post about Enumerators](http://blog.arkency.com/2014/01/ruby-to-enum-for-enumerator/).

data/guides/throttling.md

@@ -0,0 +1,42 @@
+# Throttling
+
+The gem provides a throttling mechanism that can be used to throttle a job when a given condition is met.
+If a job is throttled, it will be interrupted and retried after a backoff period has passed.
+The default backoff is 30 seconds.
+
+Specify the throttle condition as a block:
+
+```ruby
+class DeleteAccountsThrottledJob
+  include Sidekiq::Job
+  include SidekiqIteration::Iteration
+
+  throttle_on(backoff: 1.minute) do
+    DatabaseStatus.unhealthy?
+  end
+
+  def build_enumerator(cursor:)
+    active_record_relations_enumerator(Account.inactive, cursor: cursor)
+  end
+
+  def each_iteration(accounts)
+    accounts.delete_all
+  end
+end
+```
+
+Note that it’s up to you to define a throttling condition that makes sense for your app.
+For example, `DatabaseStatus.healthy?` can check various MySQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
+
+Jobs can define multiple throttle conditions. Throttle conditions are inherited by descendants, and new conditions will be appended without impacting existing conditions.
+
+The backoff can also be specified as a Proc:
+
+```ruby
+class DeleteAccountsThrottledJob
+  throttle_on(backoff: -> { RandomBackoffGenerator.generate_duration } ) do
+    DatabaseStatus.unhealthy?
+  end
+  # ...
+end
+```

data/lib/sidekiq-iteration.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require "sidekiq_iteration"