super_spreader 0.1.0 → 0.2.1

data/README.md

@@ -0,0 +1,240 @@
+# SuperSpreader
+
+SuperSpreader is a library for massive, memory- and compute-efficient backfills of ActiveRecord models using ActiveJob.
+
+This tool is built to backfill many millions of records in a resource-efficient way.  When paired with a properly written job, it can drastically reduce the wall time of a backfill through parallelization.  Jobs are enqueued in small batches so that the ActiveJob backend is not overwhelmed.  These jobs can also be stopped at a moment's notice, if needed.
+
+## Example use cases
+
+- Re-encrypt data
+- Make API calls to fill in missing data
+- Restructuring complex data
+
+## Warnings
+
+> [!WARNING]
+>
+> **Please be aware:** SuperSpreader is still fairly early in development.  While it can be used effectively by experienced hands, we are aware that it could have a better developer experience (DevX).  It was written to solve a specific problem (see "History").  We are working to generalize the tool as the need arises.  Pull requests are welcome!
+
+Please also see "Roadmap" for other known limitations that may be relevant to you.
+
+## History
+
+SuperSpreader was originally written to re-encrypt the Dialer database, a key component of Doximity's telehealth offerings.  Without SuperSpreader, it would have taken **several months** to handle many millions of records using a Key Management Service (KMS) that adds an overhead of 11 ms per record.  Using SuperSpreader took the time to backfill down to a couple of weeks.  This massive backfill happened safely during very high Dialer usage during the winter of 2020.  Of course, the name came from the coronavirus pandemic, which had a number of super-spreader events in the news around the same time.  Rather than spreading disease, the SuperSpreader gem spreads out telehealth background jobs to support the healthcare professionals that fight disease.
+
+Since that time, our team has started to use SuperSpreader in many other situations.  Our hope is that other teams, internal and external, can use it if they have similar problems to solve.
+
+## When should I use it?
+
+SuperSpreader was built for backfills.  If you need to touch every record and you have _a lot_ of records, it may be a good fit.
+
+That said, it's **not** common to need a tool like SuperSpreader.  Many backfills are better handled through SQL or Rake tasks.  SuperSpreader should only be used when the additional complexity is warranted.  Before using a shiny tool, **please stop and consider the tradeoffs**.
+
+For some use cases, a pure-SQL migration or Rake task may be a better fit.  It may also make sense to use **both** background jobs and foreground task (Rails migration or Rake task).  (Consider the database size and number of instances between production and staging.)  For that, you might consider [SuperSpreader::BatchHelper](https://github.com/doximity/super_spreader/blob/master/lib/super_spreader/batch_helper.rb).
+
+The primary criterion to consider is whether the backfill in question is _long-running_.  If you estimate it would take at least a couple of days to complete, it makes sense to consider SuperSpreader.  Another good reason to consider this tool is _code reuse_.  If you already have Ruby-land code that would be difficult or impossible to replicate in SQL, it makes sense to use SuperSpreader, assuming the equivalent Rake task would be impractical.
+
+## How does it work?
+
+SuperSpreader enqueues a configurable number of background jobs on a set schedule.  These background jobs are executed in small batches such that only a small number of jobs are enqueued at any given time.  The jobs start at the most recent record and work back to the first record, based on the auto-incrementing primary key.
+
+The configuration is able to be tuned for the needs of an individual problem.  If the backfill would require months of compute time, it can be run in parallel so that it takes much less time.  The resource utilization can be spread out so that shared resources, such as a database, are not overwhelmed with requests.  Finally, there is also support for running more jobs during off-peak usage based on a schedule.
+
+Backfills are implemented using ActiveJob classes.  SuperSpreader orchestrates running those jobs.  Each set of jobs is enqueued by a scheduler using the supplied configuration.
+
+As an example, assume that there's a table with 100,000,000 rows which need Ruby-land logic to be applied using `ExampleBackfillJob`.  The rate (e.g., how many jobs per second) is configurable.  Once configured, SuperSpreader would enqueue job in batches like:
+
+    ExampleBackfillJob run_at: "2020-11-16T22:51:59Z", begin_id: 99_999_901, end_id: 100_000_000
+    ExampleBackfillJob run_at: "2020-11-16T22:51:59Z", begin_id: 99_999_801, end_id:  99_999_900
+    ExampleBackfillJob run_at: "2020-11-16T22:51:59Z", begin_id: 99_999_701, end_id:  99_999_800
+    ExampleBackfillJob run_at: "2020-11-16T22:52:00Z", begin_id: 99_999_601, end_id:  99_999_700
+    ExampleBackfillJob run_at: "2020-11-16T22:52:00Z", begin_id: 99_999_501, end_id:  99_999_600
+    ExampleBackfillJob run_at: "2020-11-16T22:52:00Z", begin_id: 99_999_401, end_id:  99_999_500
+
+Notice that there are 3 jobs per second, 2 seconds of work were enqueued, and the batch size is 100.  Again, this is just an example for illustration, and the configuration can be modified to suit the needs of the problem.
+
+After running out of work, SuperSpreader will enqueue more work:
+
+    SuperScheduler::SchedulerJob run_at: "2020-11-16T22:52:01Z"
+
+And the work continues:
+
+    ExampleBackfillJob run_at: "2020-11-16T22:52:01Z", begin_id: 99_999_401, end_id:  99_999_500
+    ExampleBackfillJob run_at: "2020-11-16T22:52:01Z", begin_id: 99_999_301, end_id:  99_999_400
+    ExampleBackfillJob run_at: "2020-11-16T22:52:01Z", begin_id: 99_999_201, end_id:  99_999_300
+    ExampleBackfillJob run_at: "2020-11-16T22:52:02Z", begin_id: 99_999_101, end_id:  99_999_200
+    ExampleBackfillJob run_at: "2020-11-16T22:52:02Z", begin_id: 99_999_001, end_id:  99_999_100
+    ExampleBackfillJob run_at: "2020-11-16T22:52:02Z", begin_id: 99_998_901, end_id:  99_999_000
+
+This process continues until there is no more work to be done.  For more detail, please see [Spreader](https://github.com/doximity/super_spreader/blob/master/lib/super_spreader/spreader.rb) and [its spec](https://github.com/doximity/super_spreader/blob/master/spec/spreader_spec.rb).
+
+Additionally, the configuration can be tuned while SuperSpreader is running.  The configuration is read each time `SchedulerJob` runs.  As it stands, each run of SuperSpreader is hand-tuned.  It is highly recommended that SuperSpreader resource utilization is monitored during runs.  That said, it is designed to run autonomously once a good configuration is found.
+
+Example tuning:
+
+- Does the process need to go faster?  Increase the number of jobs per second.
+- Are batches taking too long to complete?  Decrease the batch size.
+- Is `SchedulerJob` taking a long time to complete?  Decrease the duration so that less work is enqueued in each cycle.
+
+Finally, SuperSpreader can be stopped instantly and resumed at a later time, if a need ever arises.
+
+## How do I use it?
+
+To repeat an earlier disclaimer:
+
+> **Please be aware:** SuperSpreader is still fairly early in development.  While it can be used  by experienced hands, we are aware that it could have a better developer experience (DevX).  It was written to solve a specific problem (see "History").  We are working to generalize the tool as the need arises.  Pull requests are welcome!
+
+If you haven't yet, please read the "How does it work?" section.  This basic workflow is tested in `spec/integration/backfill_spec.rb`.
+
+First, write a backfill job.  Please see [this example for details](https://github.com/doximity/super_spreader/blob/master/spec/support/example_backfill_job.rb).
+
+Next, configure `SuperSpreader` from the console by saving `SchedulerConfig` to Redis.  For documentation on each attribute, please see [SchedulerConfig](https://github.com/doximity/super_spreader/blob/master/lib/super_spreader/scheduler_config.rb).  It is recommended that you start slow, with small batches, short durations, and low per-second rates.
+
+**Important:** SuperSpreader currently only supports a _single_ configuration, though removing that limitation is our Roadmap (please see below).
+
+```ruby
+# NOTE: This is an example.  You should take your situation into account when
+# setting these values.
+config = SuperSpreader::SchedulerConfig.new
+
+config.batch_size = 10
+config.duration = 10
+config.job_class_name = "ExampleBackfillJob"
+
+config.per_second_on_peak = 3.0
+config.per_second_off_peak = 3.0
+
+config.on_peak_timezone = "America/Los_Angeles"
+config.on_peak_wday_begin = 1
+config.on_peak_wday_end = 5
+config.on_peak_hour_begin = 5
+config.on_peak_hour_end = 17
+
+config.save
+```
+
+Now the `SchedulerJob` can be started.  It will run until it is stopped or runs out of work.
+
+```ruby
+SuperSpreader::SchedulerJob.perform_now
+```
+
+At this point, you should monitor your database and worker instances using the tooling you have available.  You should make adjustments based on the metrics you have available.
+
+Based on those metrics, slowly step up `per_second_on_peak` and `batch_size` while continuing to monitor:
+
+```ruby
+config.batch_size = 20
+config.save
+```
+
+```ruby
+config.per_second_on_peak = 4.0
+config.save
+```
+
+Continue to step up the rates, until you arrive at a rate that is acceptable for your situation.
+For our re-encryption project as an example, our jobs ran at this rate:
+
+```ruby
+# NOTE: This is an example.  You should take your situation into account when
+# setting these values.
+config = SuperSpreader::SchedulerConfig.new
+
+config.batch_size = 70
+config.duration = 180
+config.job_class_name = "ReencryptJob"
+
+config.per_second_on_peak = 3.0
+config.per_second_off_peak = 7.5
+
+config.on_peak_timezone = "America/Los_Angeles"
+config.on_peak_wday_begin = 1
+config.on_peak_wday_end = 5
+config.on_peak_hour_begin = 5
+config.on_peak_hour_end = 17
+
+config.save
+```
+
+### Disaster recovery
+
+If at any point you need to stop the background jobs, stop all scheduling using:
+
+```ruby
+SuperSpreader::SchedulerJob.stop!
+```
+
+Optionally, if it is acceptable to have a partially-processed cycle, you can stop the backfill jobs as well:
+
+```ruby
+ExampleBackfillJob.stop!
+```
+
+(Recovering from a partially-processed cycle requires manually setting the correct `initial_id` in `SpreadTracker`.)
+
+The jobs will still be present in the job runner, but will all execute instantly because of the early return as demonstrated in [the example job](https://github.com/doximity/super_spreader/blob/master/spec/support/example_backfill_job.rb).  After the last scheduler job, the process will be paused.
+
+### Restarting
+
+If you stop the jobs but you wish to restart them later, use the `go!` method and *then* call `SuperSpreader::SchedulerJob.perform_now`.  Otherwise, the jobs will not do any work.
+
+```ruby
+ExampleBackfillJob.go!
+SuperSpreader::SchedulerJob.go!
+SuperSpreader::SchedulerJob.perform_now
+```
+
+## Installation
+
+If you've gotten this far and think SuperSpreader is a good fit for your problem, these are the instructions for installing it.
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'super_spreader'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install super_spreader
+
+SuperSpreader requires an ActiveRecord-compatible database, an ActiveJob-compatible job runner, and Redis for bookkeeping.
+
+For Rails, please set up SuperSpreader using an initializer:
+
+```ruby
+# config/initializers/super_spreader.rb
+
+SuperSpreader.logger = Rails.logger
+SuperSpreader.redis = Redis.new(url: ENV["REDIS_URL"])
+```
+
+## Roadmap
+
+Please see [the Milestones on GitHub](https://github.com/doximity/super_spreader/milestones?direction=asc&sort=title&state=open).
+
+## Development
+
+You'll need [Redis](https://redis.io/docs/getting-started/) and [Ruby](https://www.ruby-lang.org/en/downloads/) installed.  Please ensure both are set up before continuing.
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+1. See [CONTRIBUTING.md](./CONTRIBUTING.md)
+2. Fork it ( https://github.com/doximity/super_spreader/fork )
+3. Create your feature branch (`git checkout -b my-new-feature`)
+4. Commit your changes (`git commit -am 'Add some feature'`)
+5. Push to the branch (`git push origin my-new-feature`)
+6. Create a new Pull Request
+
+## License
+
+`super_spreader` is licensed under an Apache 2 license. Contributors are required to sign a contributor license agreement. See LICENSE.txt and CONTRIBUTING.md for more information.

data/Rakefile

@@ -0,0 +1,35 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "yard"
+
+RSpec::Core::RakeTask.new(:spec)
+YARD::Rake::YardocTask.new(:doc)
+
+task default: :spec
+
+desc "Run a REPL with access to this library"
+task :console do
+  sh("irb -I lib -r super_spreader")
+end
+
+namespace :check do
+  desc "Run all checks"
+  task all: %i[redis]
+
+  desc "Confirm Redis is accessible"
+  task :redis do
+    require "redis"
+
+    inaccessible_error_message = "Redis: inaccessible (please confirm that a Redis server is installed and running, e.g. redis-server)"
+
+    redis = Redis.new(url: ENV["REDIS_URL"])
+
+    if redis.ping == "PONG"
+      puts "Redis: OK"
+    else
+      raise inaccessible_error_message
+    end
+  rescue Redis::CannotConnectError
+    raise inaccessible_error_message
+  end
+end

data/bin/console

@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "super_spreader"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+rake check:all

data/lib/super_spreader/batch_helper.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+require "active_record"
+
+module SuperSpreader
+  # Methods in this module are suitable for use in Rails migrations.  It is
+  # expected that their interface will remain stable.  If breaking changes are
+  # introduced, a new module will be introduced so existing migrations will not
+  # be affected.
+  module BatchHelper
+    # Execute SQL in small batches for an entire table.
+    #
+    # It is assumed that the table has a primary key named +id+.
+    #
+    # Recommendation for migrations: Use this in combination with +disable_ddl_transaction!+.
+    # See also: https://github.com/ankane/strong_migrations#backfilling-data
+    #
+    # @param table_name [String] the name of the table
+    # @param step_size [Integer] how many records to process in each batch
+    # @yield [minimum_id, maximum_id] block that returns SQL to migrate records between minimum_id and maximum_id
+    def batch_execute(table_name:, step_size:)
+      result = execute(<<~SQL).to_a.flatten
+        SELECT MIN(id) AS min_id, MAX(id) AS max_id FROM #{quote_table_name(table_name)}
+      SQL
+      min_id = result[0]["min_id"]
+      max_id = result[0]["max_id"]
+      return unless min_id && max_id
+
+      lower_id = min_id
+      loop do
+        sql = yield(lower_id, lower_id + step_size)
+
+        execute(sql)
+
+        lower_id += step_size
+        break if lower_id > max_id
+      end
+    end
+  end
+end

data/lib/super_spreader/peak_schedule.rb

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "active_support/core_ext/time"
+
+module SuperSpreader
+  class PeakSchedule
+    def initialize(on_peak_wday_range:, on_peak_hour_range:, timezone:)
+      @on_peak_wday_range = on_peak_wday_range
+      @on_peak_hour_range = on_peak_hour_range
+      @timezone = timezone
+    end
+
+    def on_peak?(time = Time.current)
+      time_in_zone = time.in_time_zone(@timezone)
+
+      is_on_peak_day = @on_peak_wday_range.cover?(time_in_zone.wday)
+      is_on_peak_hour = @on_peak_hour_range.cover?(time_in_zone.hour)
+
+      is_on_peak_day && is_on_peak_hour
+    end
+  end
+end

data/lib/super_spreader/redis_model.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+require "active_model"
+require "redis"
+
+module SuperSpreader
+  class RedisModel
+    include ActiveModel::Model
+    include ActiveModel::Attributes
+    include ActiveModel::Serialization
+
+    def initialize(values = default_values)
+      super
+    end
+
+    def default_values
+      redis.hgetall(redis_key)
+    end
+
+    def persisted?
+      redis.get(redis_key).present?
+    end
+
+    def delete
+      redis.del(redis_key)
+    end
+
+    def save
+      redis.multi do |pipeline|
+        pipeline.del(redis_key)
+
+        serializable_hash.each do |key, value|
+          pipeline.hset(redis_key, key, value)
+        end
+      end
+    end
+
+    # Primarily for factory_bot
+    alias_method :save!, :save
+
+    private
+
+    def redis_key
+      self.class.name
+    end
+
+    def redis
+      SuperSpreader.redis
+    end
+  end
+end

data/lib/super_spreader/scheduler_config.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+require "super_spreader/peak_schedule"
+require "super_spreader/redis_model"
+
+module SuperSpreader
+  class SchedulerConfig < RedisModel
+    # The job class to enqueue on each run of the scheduler.
+    attribute :job_class_name, :string
+    # The number of records to process in each invocation of the job class.
+    attribute :batch_size, :integer
+    # The amount of work to enqueue, in seconds.
+    attribute :duration, :integer
+
+    # The number of jobs to enqueue per second, allowing for fractional amounts
+    # such as 1 job every other second using `0.5`.
+    attribute :per_second_on_peak, :float
+    # The same as per_second_on_peak, but for times that are not identified as
+    # on-peak.
+    attribute :per_second_off_peak, :float
+
+    # This section manages the definition "on peak."  Compare this terminology
+    # to bus or train schedules.
+
+    # The timezone to use for time calculations.
+    #
+    # Example: "America/Los_Angeles" for Pacific time
+    attribute :on_peak_timezone, :string
+    # The 24-hour hour on which on-peak application usage starts.
+    #
+    # Example: 5 for 5 AM
+    attribute :on_peak_hour_begin, :integer
+    # The 24-hour hour on which on-peak application usage ends.
+    #
+    # Example: 17 for 5 PM
+    attribute :on_peak_hour_end, :integer
+    # The wday value on which on-peak application usage starts.
+    #
+    # Example: 1 for Monday
+    attribute :on_peak_wday_begin, :integer
+    # The wday value on which on-peak application usage ends.
+    #
+    # Example: 5 for Friday
+    attribute :on_peak_wday_end, :integer
+
+    attr_writer :schedule
+
+    def job_class
+      job_class_name.constantize
+    end
+
+    def super_spreader_config
+      [job_class, job_class.super_spreader_model_class]
+    end
+
+    def spread_options
+      {
+        batch_size: batch_size,
+        duration: duration,
+        per_second: per_second
+      }
+    end
+
+    def per_second
+      schedule.on_peak? ? per_second_on_peak : per_second_off_peak
+    end
+
+    private
+
+    def schedule
+      @schedule ||=
+        PeakSchedule.new(
+          on_peak_wday_range: on_peak_wday_begin..on_peak_wday_end,
+          on_peak_hour_range: on_peak_hour_begin..on_peak_hour_end,
+          timezone: on_peak_timezone
+        )
+    end
+  end
+end

data/lib/super_spreader/scheduler_job.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require "active_job"
+require "json"
+require "super_spreader/scheduler_config"
+require "super_spreader/spreader"
+require "super_spreader/stop_signal"
+
+module SuperSpreader
+  class SchedulerJob < ActiveJob::Base
+    extend StopSignal
+
+    def perform
+      return if self.class.stopped?
+
+      log(started_at: Time.current.iso8601)
+      log(config.serializable_hash)
+
+      super_spreader = Spreader.new(*config.super_spreader_config)
+      next_id = super_spreader.enqueue_spread(**config.spread_options)
+      log(next_id: next_id)
+
+      return if next_id.zero?
+
+      self.class.set(wait_until: next_run_at).perform_later
+      log(next_run_at: next_run_at.iso8601)
+    end
+
+    def next_run_at
+      config.duration.seconds.from_now
+    end
+
+    def config
+      @config ||= SchedulerConfig.new
+    end
+
+    private
+
+    def log(hash)
+      SuperSpreader.logger.info({subject: self.class.name}.merge(hash).to_json)
+    end
+  end
+end

data/lib/super_spreader/spread_tracker.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+require "active_record"
+require "redis"
+
+module SuperSpreader
+  class SpreadTracker
+    def initialize(job_class, model_class)
+      @job_class = job_class
+      @model_class = model_class
+    end
+
+    def initial_id
+      redis_value = redis.hget(initial_id_key, @model_class.name)
+
+      value = redis_value || @model_class.maximum(:id)
+
+      value.to_i
+    end
+
+    def initial_id=(value)
+      if value.nil?
+        redis.hdel(initial_id_key, @model_class.name)
+      else
+        redis.hset(initial_id_key, @model_class.name, value)
+      end
+    end
+
+    private
+
+    def redis
+      SuperSpreader.redis
+    end
+
+    def initial_id_key
+      "#{@job_class.name}:initial_id"
+    end
+  end
+end

data/lib/super_spreader/spreader.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+require "super_spreader/spread_tracker"
+
+module SuperSpreader
+  class Spreader
+    def initialize(job_class, model_class, spread_tracker: nil)
+      @job_class = job_class
+      @model_class = model_class
+      @spread_tracker = spread_tracker || SpreadTracker.new(job_class, model_class)
+    end
+
+    def spread(batch_size:, duration:, per_second:, initial_id:, begin_at: Time.now.utc)
+      end_id = initial_id
+      segment_duration = 1.0 / per_second
+      time_index = 0.0
+      batches = []
+
+      while time_index < duration
+        break if end_id <= 0
+
+        # Use floor to prevent subsecond times
+        run_at = begin_at + time_index.floor
+        begin_id = clamp(end_id - batch_size + 1)
+        batches << {run_at: run_at, begin_id: begin_id, end_id: end_id}
+
+        break if begin_id == 1
+
+        end_id = begin_id - 1
+        time_index += segment_duration
+      end
+
+      batches
+    end
+
+    def enqueue_spread(**opts)
+      initial_id = @spread_tracker.initial_id
+      return 0 if initial_id.zero?
+
+      batches = spread(**opts.merge(initial_id: initial_id))
+
+      batches.each do |batch|
+        @job_class
+          .set(wait_until: batch[:run_at])
+          .perform_later(batch[:begin_id], batch[:end_id])
+      end
+
+      last_begin_id = batches.last[:begin_id]
+      next_id = last_begin_id - 1
+      @spread_tracker.initial_id = next_id
+
+      next_id
+    end
+
+    private
+
+    def clamp(value)
+      (value <= 0) ? 1 : value
+    end
+  end
+end

data/lib/super_spreader/stop_signal.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require "redis"
+
+module SuperSpreader
+  module StopSignal
+    def stop!
+      redis.set(stop_key, true)
+    end
+
+    def go!
+      redis.del(stop_key)
+    end
+
+    def stopped?
+      redis.exists(stop_key).positive?
+    end
+
+    private
+
+    def redis
+      SuperSpreader.redis
+    end
+
+    def stop_key
+      "#{name}:stop"
+    end
+  end
+end

data/lib/super_spreader/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module SuperSpreader
+  VERSION = "0.2.1"
+end