job-iteration 1.9.0 → 1.11.0

data/guides/custom-enumerator.md

@@ -1,140 +0,0 @@
-# Custom Enumerator
-
-`Iteration` leverages the [Enumerator](https://ruby-doc.org/3.2.1/Enumerator.html) pattern from the Ruby standard library,
-which allows us to use almost any resource as a collection to iterate.
-
-Before writing an enumerator, it is important to understand [how Iteration works](iteration-how-it-works.md) and how
-your enumerator will be used by it. An enumerator must `yield` two things in the following order as positional
-arguments:
-- An object to be processed in a job `each_iteration` method
-- A cursor position, which `Iteration` will persist if `each_iteration` returns successfully and the job is forced to shut
-  down. It can be any data type your job backend can serialize and deserialize correctly.
-
-A job that includes `Iteration` is first started with `nil` as the cursor. When resuming an interrupted job, `Iteration`
-will deserialize the persisted cursor and pass it to the job's `build_enumerator` method, which your enumerator uses to
-find objects that come _after_ the last successfully processed object. The [array enumerator](https://github.com/Shopify/job-iteration/blob/v1.3.6/lib/job-iteration/enumerator_builder.rb#L50-L67)
-is a simple example which uses the array index as the cursor position.
-
-In addition to the remainder of this guide, we recommend you read the implementation of the other enumerators that come with the library (`CsvEnumerator`, `ActiveRecordEnumerator`) to gain a better understanding of building enumerators.
-
-## Enumerator with cursor
-
-For a more complex example, consider this `Enumerator` that wraps a third party API (Stripe) for paginated iteration and
-stores a string as the cursor position:
-
-```ruby
-class StripeListEnumerator
-  # @see https://stripe.com/docs/api/pagination
-  # @param resource [Stripe::APIResource] The type of Stripe object to request
-  # @param params [Hash] Query parameters for the request
-  # @param options [Hash] Request options, such as API key or version
-  # @param cursor [nil, String] The Stripe ID of the last item iterated over
-  def initialize(resource, params: {}, options: {}, cursor:)
-    pagination_params = {}
-    pagination_params[:starting_after] = cursor unless cursor.nil?
-
-    # The following line makes a request, consider adding your rate limiter here.
-    @list = resource.public_send(:list, params.merge(pagination_params), options)
-  end
-
-  def to_enumerator
-    to_enum(:each).lazy
-  end
-
-  private
-
-  # We yield our enumerator with the object id as the index so it is persisted
-  # as the cursor on the job. This allows us to properly set the
-  # `starting_after` parameter for the API request when resuming.
-  def each
-    loop do
-      @list.each do |item, _index|
-        # The first argument is what gets passed to `each_iteration`.
-        # The second argument (item.id) is going to be persisted as the cursor,
-        # it doesn't get passed to `each_iteration`.
-        yield item, item.id
-      end
-
-      # The following line makes a request, consider adding your rate limiter here.
-      @list = @list.next_page
-
-      break if @list.empty?
-    end
-  end
-end
-```
-
-### Usage
-
-Here we leverage the Stripe cursor pagination where the cursor is an ID of a specific item in the collection. The job
-which uses such an `Enumerator` would then look like so:
-
-```ruby
-class LoadRefundsForChargeJob < ActiveJob::Base
-  include JobIteration::Iteration
-
-  # If you added your own rate limiting above, handle it here. For example:
-  # retry_on(MyRateLimiter::LimitExceededError, wait: 30.seconds, attempts: :unlimited)
-  # Use an exponential back-off strategy when Stripe's API returns errors.
-
-  def build_enumerator(charge_id, cursor:)
-    enumerator_builder.wrap(
-      StripeListEnumerator.new(
-        Stripe::Refund,
-        params: { charge: charge_id}, # "charge_id" will be a prefixed Stripe ID such as "chrg_123"
-        options: { api_key: "sk_test_123", stripe_version: "2018-01-18" },
-        cursor: cursor
-      ).to_enumerator
-    )
-  end
-
-  # Note that in this case `each_iteration` will only receive one positional argument per iteration.
-  # If what your enumerator yields is a composite object you will need to unpack it yourself
-  # inside the `each_iteration`.
-  def each_iteration(stripe_refund, charge_id)
-    # ...
-  end
-end
-```
-
-and you initiate the job with
-
-```ruby
-LoadRefundsForChargeJob.perform_later(charge_id = "chrg_345")
-```
-
-## Cursorless enumerator
-
-Sometimes you can ignore the cursor. Consider the following custom `Enumerator` that takes items from a Redis list, which
-is essentially a queue. Even if this job doesn't need to persist a cursor in order to resume, it can still use
-`Iteration`'s signal handling to finish `each_iteration` and gracefully terminate.
-
-```ruby
-class RedisPopListJob < ActiveJob::Base
-  include JobIteration::Iteration
-
-  # @see https://redis.io/commands/lpop/
-  def build_enumerator(*)
-    @redis = Redis.new
-    enumerator_builder.wrap(
-      Enumerator.new do |yielder|
-        yielder.yield @redis.lpop(key), nil
-      end
-    )
-  end
-
-  def each_iteration(item_from_redis)
-    # ...
-  end
-end
-```
-
-## Caveats
-
-### Post-`yield` code
-
-Code that is written after the `yield` in a custom enumerator is not guaranteed to execute. In the case that a job is
-forced to exit ie `job_should_exit?` is true, then the job is re-enqueued during the yield and the rest of the code in
-the enumerator does not run. You can follow that logic
-[here](https://github.com/Shopify/job-iteration/blob/v1.3.6/lib/job-iteration/iteration.rb#L161-L165) and
-[here](https://github.com/Shopify/job-iteration/blob/v1.3.6/lib/job-iteration/iteration.rb#L131-L143)

data/guides/iteration-how-it-works.md

@@ -1,51 +0,0 @@
-# Iteration: how it works
-
-The main idea behind Iteration is to provide an API to describe jobs in an interruptible manner, in contrast with implementing one massive `#perform` method that is impossible to interrupt safely.
-
-Exposing the enumerator and the action to apply allows us to keep a cursor and interrupt between iterations. Let's see what this looks like with an  ActiveRecord relation (and Enumerator).
-
-1. `build_enumerator` is called, which constructs `ActiveRecordEnumerator` from an ActiveRecord relation (`Product.all`)
-2. The first batch of records is loaded:
-
-```sql
-SELECT  `products`.* FROM `products` ORDER BY products.id LIMIT 100
-```
-
-3. The job iterates over two records of the relation and then receives `SIGTERM` (graceful termination signal) caused by a deploy.
-4. The signal handler sets a flag that makes `job_should_exit?` return `true`.
-5. After the last iteration is completed, we will check `job_should_exit?` which now returns `true`.
-6. The job stops iterating and pushes itself back to the queue, with the latest `cursor_position` value.
-7. Next time when the job is taken from the queue, we'll load records starting from the last primary key that was processed:
-
-```sql
-SELECT  `products`.* FROM `products` WHERE (products.id > 2) ORDER BY products.id LIMIT 100
-```
-
-## Exceptions inside `each_iteration`
-
-Unrescued exceptions inside the `each_iteration` block are handled the same way as exceptions occuring in `perform` for a regular Active Job subclass, meaning you need to configure it to retry using [`retry_on`](https://api.rubyonrails.org/classes/ActiveJob/Exceptions/ClassMethods.html#method-i-retry_on) or manually call [`retry_job`](https://api.rubyonrails.org/classes/ActiveJob/Exceptions.html#method-i-retry_job). The job will re-enqueue itself with the last successful cursor, the iteration that failed will be retried with the same parameters and the cursor will only move if that iteration succeeds. This behaviour may be enough for intermittent errors, such as network connection failures, but if your execution is deterministic and you have an error, subsequent iterations will never run.
-
-In other words, if you are trying to process 100 records but the job consistently fails on the 61st, only the first 60 will be processed and the job will try to process the 61st record until retries are exhausted.
-
-If no retries are configured or retries are exhausted, Active Job 'bubbles up' the exception to the job backend. Retries by the backend (e.g. Sidekiq) are not supported, meaning that jobs retried by the job backend instead of Active Job will restart from the beginning.
-
-## Stopping a job
-
-Because jobs typically retry when exceptions are thrown, there is a special mechanism to fully stop a job that still has iterations remaining. To do this, you can `throw(:abort)`. This is then caught by job-iteration and signals that the job should complete now, regardless of its iteration state.
-
-## Signals
-
-It's critical to know [UNIX signals](https://www.tutorialspoint.com/unix/unix-signals-traps.htm) in order to understand how interruption works. There are two main signals that Sidekiq and Resque use: `SIGTERM` and `SIGKILL`. `SIGTERM` is the graceful termination signal which means that the process should exit _soon_, not immediately. For Iteration, it means that we have time to wait for the last iteration to finish and to push job back to the queue with the last cursor position.
-`SIGTERM` is what allows Iteration to work. In contrast, `SIGKILL` means immediate exit. It doesn't let the worker terminate gracefully, instead it will drop the job and exit as soon as possible.
-
-Most of the deploy strategies (Kubernetes, Heroku, Capistrano) send `SIGTERM` before shutting down a node, then wait for a timeout (usually from 30 seconds to a minute) to send `SIGKILL` if the process has not terminated yet.
-
-Further reading: [Sidekiq signals](https://github.com/mperham/sidekiq/wiki/Signals).
-
-## Enumerators
-
-In the early versions of Iteration, `build_enumerator` used to return ActiveRecord relations directly, and we would infer the Enumerator based on the type of object. We used to support ActiveRecord relations, arrays and CSVs. This made it hard to add support for other types of enumerations, and it was easy for developers to make mistakes and return an array of ActiveRecord objects, and for us starting to treat that as an array instead of as an ActiveRecord relation.
-
-The current version of Iteration supports _any_ Enumerator. We expose helpers to build common enumerators conveniently (`enumerator_builder.active_record_on_records`), but it's up to a developer to implement [a custom Enumerator](custom-enumerator.md).
-
-Further reading: [ruby-doc](https://ruby-doc.org/3.2.1/Enumerator.html), [a great post about Enumerators](http://blog.arkency.com/2014/01/ruby-to-enum-for-enumerator/).

data/guides/throttling.md

@@ -1,68 +0,0 @@
-Iteration comes with a special wrapper enumerator that allows you to throttle iterations based on external signal (e.g. database health).
-
-Consider this example:
-
-```ruby
-class InactiveAccountDeleteJob < ActiveJob::Base
-  include JobIteration::Iteration
-
-  def build_enumerator(_params, cursor:)
-    enumerator_builder.active_record_on_batches(
-      Account.inactive,
-      cursor: cursor
-    )
-  end
-
-  def each_iteration(batch, _params)
-    Account.where(id: batch.map(&:id)).delete_all
-  end
-end
-```
-
-For an app that keeps track of customer accounts, it's typical to purge old data that's no longer relevant for storage.
-
-At the same time, if you've got a lot of DB writes to perform, this can cause extra load on the database and slow down other parts of your service.
-
-You can change `build_enumerator` to wrap enumeration on DB rows into a throttle enumerator, which takes signal as a proc and enqueues the job for later in case the proc returned `true`.
-
-```ruby
-def build_enumerator(_params, cursor:)
-  enumerator_builder.build_throttle_enumerator(
-    enumerator_builder.active_record_on_batches(
-      Account.inactive,
-      cursor: cursor
-    ),
-    throttle_on: -> { DatabaseStatus.unhealthy? },
-    backoff: 30.seconds
-  )
-end
-```
-
-If you want to apply throttling on all jobs, you can subclass your own EnumeratorBuilder and override the default
-enumerator builder. The builder always wraps the returned enumerators from `build_enumerator`
-
-```ruby
-class MyOwnBuilder < JobIteration::EnumeratorBuilder
-  class Wrapper < Enumerator
-    class << self
-      def wrap(_builder, enum)
-        ThrottleEnumerator.new(
-          enum,
-          nil,
-          throttle_on: -> { DatabaseStatus.unhealthy? },
-          backoff: 30.seconds
-        )
-      end
-    end
-  end
-end
-
-JobIteration.enumerator_builder = MyOwnBuilder
-```
-
-Note that it's up to you to implement `DatabaseStatus.unhealthy?` that works for your database choice. At Shopify, a helper like `DatabaseStatus` checks the following MySQL metrics:
-
-* Replication lag across all regions
-* DB threads
-* DB is available for writes (otherwise indicates a failover happening)
-* [Semian](https://github.com/shopify/semian) open circuits