stepper_motor 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 503c1ad7d876deba41afff2e576d2b981271bdc52ec1d801c6f9be7fd2040108
+  data.tar.gz: b3a59f3dd38663fcaa7c09e0ce13860d9f243c2009233e7389d90fcd236b9b51
+SHA512:
+  metadata.gz: da032fcdb7b4d4dd0baeb0efb37c536e8b496d197007dd8d578a50abf606fb3cf75b7ab2e2a79ce8ba5bd3a3ba257f1a329eea66ef6384f7f5b2de51e6016e15
+  data.tar.gz: 7588f66ed4eb021b45657a98aea2273a7e2b996a68b6d997980b9d69c93f69e49909a5c9c7a871f4ef39409eebb7fc13650f3cf924fa09ed1278fad922908401

data/.rspec

@@ -0,0 +1,3 @@
+--format documentation
+--color
+--require spec_helper

data/.ruby-version

@@ -0,0 +1 @@
+2.7.7

data/.standard.yml

@@ -0,0 +1,8 @@
+ruby_version: 2.7
+ignore:
+  - 'spec/**/*':
+    - Lint/ConstantDefinitionInBlock
+    - Style/GlobalVars
+  - 'lib/zip_kit/rack_body.rb': # Erroneous detection
+    - Lint/OrderedMagicComments
+    - Layout/EmptyLineAfterMagicComment

data/.yardopts

@@ -0,0 +1 @@
+--markup markdown - docs/MANUAL.md

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+## [0.1.0] - 2025-01-01
+
+- Initial release. Expect bugs of mysterious kinds.

data/LICENSE.md

@@ -0,0 +1,3 @@
+Copyright (c) Julik Tarkhanov
+
+stepper_motor is an Open Source project licensed under the terms of the LGPLv3 license. Please see http://www.gnu.org/licenses/lgpl-3.0.html for license text.

data/README.md

@@ -0,0 +1,421 @@
+# StepperMotor
+
+Is a useful tool for running stepped or iterative workflows inside your Rails application.
+
+## Usage
+
+Define a workflow and launch your user into it:
+
+```ruby
+class SignupJourney < StepperMotor::Journey
+  step :after_signup do
+    WelcomeMailer.welcome_email(hero).deliver_later
+  end
+
+  step :remind_of_tasks, wait: 2.days do
+    ServiceUpdateMailer.two_days_spent_email(hero).deliver_later
+  end
+
+  step :onboarding_complete_, wait: 15.days do
+    OnboardingCompleteMailer.onboarding_complete_email(hero).deliver_later
+  end
+end
+
+SignupJourney.create!(hero: current_user)
+```
+
+## Installation
+
+Add the gem to the application's Gemfile, and then generate and run the migration
+
+    $ bundle add stepper_motor
+    $ bundle install
+    $ bin/rails g stepper_motor:install --uuid # Pass "uuid" if you are using UUID for your primary and foreign keys
+    $ bin/rails db:migrate
+
+
+## Intro
+
+`stepper_motor` solves a real, tangible problem in Rails apps - tracking activities over long periods of time. It does so in a durable, reentrant and consistent manner, utilizing the guarantees provided by your relational database you already have.
+
+## Philosophy behind StepperMotor
+
+Most of our applications have workflows which have to happen in steps. They pretty much always have some things in common:
+
+* We want just one workflow of a certain type per user or per business transaction
+* We want only one parallel execution of a unique workflow at a time
+* We want the steps to be explicitly idempotent
+* We want visibility into the step our workflow is in, what step it is going to enter, what step it has left
+
+While Rails provides great abstractions for "inline" actions induced via APIs or web requests in the form of ActionController, and great abstractions for single "unit of work" tasks via ActiveJob - these are lacking if one wants true idempotency and correct state tracking throughout multiple steps. When a workflow like this has to be implemented in a system, the choice usually goes out to a number of possible solutions:
+
+* Trying ActiveJob-specific "batch" workflows, such as [Sidekiq Pro's batches](https://github.com/sidekiq/sidekiq/wiki/Batches) or [good_job batches](https://github.com/bensheldon/good_job?tab=readme-ov-file#batches)
+* State machines attached to an ActiveRecord, via tools like [aasm](https://github.com/aasm/aasm) or [state_machine_enum](https://github.com/cheddar-me/state_machine_enum) - locking and controlling transitions then usually falls on the developer
+* Adopting a complex solution like [Temporal.io](https://temporal.io/), with the app relegated to just executing parts of the workflow
+
+We believe all of these solutions do not quite hit the "sweet spot" where step workflows would integrate well with Rails.
+
+* Most Rails apps already have a perfectly fit transactional, durable data store - the main database
+* The devloper should not have intimate understanding of DB atomicity and ActiveRecord `with_lock` and `reload` to have step workflows
+* It should not be necessary to configure a whole extra service (like Temporal.io) just for supporting those workflows. A service like that should be a part of your monolith, not an external application. It should not be necessary to talk to that service using complex, Ruby-unfriendly protocols and interfaces like gRPC.
+
+So, StepperMotor aims to give you "just enough of Temporal-like functionality" for most Rails-bound workflows. Without extra dependencies, network calls, services or having to learn extra languages. So let's dive in.
+
+## How we do it
+
+StepperMotor is built around the concept of a `Journey`. A `Journey` [is a sequence of steps happening to a `hero`](https://en.wikipedia.org/wiki/Hero%27s_journey) - once launched, the journey will run until it either finishes or cancels. A `Journey` is just an `ActiveRecord` model, with all the persistence methods you already know and use.
+
+Steps are defined inside the Journey as blocks, and they run in the context of the `Journey` model. The following constraints apply:
+
+* For any one Journey, only one Fiber/Thread/Process may be performing a step on it
+* For any one Journey, only one step can be executing at any given time
+* For any `hero`, multiple different Journeys may exist and be in different stages of completion
+
+The `step` blocks get executed in the context of the `Journey` model instance. This is done so that you can define helper methods in the `Journey` subclass, and make good use of them. A Journey links to just one record - the `hero`.
+
+## Installation
+
+Add the gem to the application's Gemfile, and then generate and run the migration
+
+    $ bundle add stepper_motor
+    $ bundle install
+    $ bin/rails g stepper_motor:install
+    $ bin/rails db:migrate
+
+## Usage
+
+Define a workflow and launch your user into it:
+
+```ruby
+class SignupJourney < StepperMotor::Journey
+  step :after_signup do
+    WelcomeMailer.welcome_email(hero).deliver_later
+  end
+
+  step :remind_of_tasks, wait: 2.days do
+    ServiceUpdateMailer.two_days_spent_email(hero).deliver_later
+  end
+
+  step :onboarding_complete_, wait: 15.days do
+    OnboardingCompleteMailer.onboarding_complete_email(hero).deliver_later
+  end
+end
+
+class SignupController
+  def create
+    # ...your other business actions
+    SignupJourney.create!(hero: current_user)
+    redirect_to user_root_path(current_user)
+  end
+end
+```
+
+## A few sample journeys: single step with repeats
+
+Let's examine a simple single-step journey. Imagine you have a user that is about to churn, and you want to keep sending them drip emails until they churn in the hope that they will reconvert. The Journey will likely look like this:
+
+```ruby
+class ChurnPreventionJourney < StepperMotor::Journey
+  step do
+    cancel! if hero.subscription_lapses_at > 120.days.from_now
+
+    time_remaining_until_expiry_ = hero.subscription_lapses_at - Time.current
+    if time_remaining_until_expiry > 1.days
+      ResubscribeReminderMailer.extend_subscription_reminder(hero).deliver_later
+      send_next_reminder_after = (time_remaining_until_expiry / 2).in_days.floor
+      reattempt!(wait: send_next_reminder_after.days)
+    else
+      # If the user has churned - let the journey finish, as there is nothing to do
+      SadToSeeYouGoMailer.farewell(hero).deliver_later
+    end
+  end
+end
+
+ChurnPreventionJourney.create(hero: user)
+```
+
+In this case we have just one `step` which is going to be repeated. When we decide to repeat a step (if the user still has time to reconnect with the business), we postpone its execution by a certain amount of time - in this case, half the days remaining on the user's subscription. If a user rescubscribes, we `cancel!` the only step of the `Journey`, after which it gets marked `finished` in the database.
+
+## More sample journeys: email drip campaign
+
+As our second example, let's check out a drip campaign which inceitivises a user with bonuses as their account nears termination.
+
+```ruby
+class ReengagementJourney < StepperMotor::Journey
+  step :first do
+    cancel! if reengaged?
+    hero.bonus_programs.create!(type: BonusProgram::REENGAGEMENT)
+    hero.push_anayltics_event!(event: "reengagement_touchpoint", properties: {step: 1})
+  end
+
+  step :second, wait: 14.days do
+    cancel! if reengaged?
+    hero.bonus_programs.create!(type: BonusProgram::DISCOUNT)
+    hero.push_anayltics_event!(event: "reengagement_touchpoint", properties: {step: 2})
+  end
+
+  step :third, wait: 7.days do
+    cancel! if reengaged?
+    hero.bonus_programs.create!(type: BonusProgram::DOUBLE_DISCOUNT)
+    hero.push_anayltics_event!(event: "reengagement_touchpoint", properties: {step: 3})
+  end
+
+  step :final, wait: 3.days do
+    cancel! if reengaged?
+    hero.close_account!
+    hero.push_anayltics_event!(event: "reengagement_touchpoint", properties: {step: 4})
+  end
+
+  def reengaged?
+    # If the user purchased anything after this journey started,
+    # consider them "re-engaged"
+    hero.purchases.where("created_at > ?", created_at).any?
+  end
+end
+```
+
+In this instance, we split our workflow in a number of steps - 4 in total. After the first step (`:first`) we wait for 14 days before executing the next one. 7 days later - we run another one. We end with closing the user's account. If the user has reengaged at any step, we mark the `Journey` as `canceled`.
+
+## More sample journeys: archiving and deleting user data
+
+Imagine a user on your platform has requested their account to be deleted. Usually you do some archiving before deletion, to preserve some data that can be useful in aggregate - just scrubbing the PII. You also change the user information so that the user does not show up in the normal application flows anymore.
+
+```ruby
+class AccountErasureJourney < StepperMotor::Journey
+  step :deactivate_user do
+    hero.deactivated!
+  end
+
+  step :remove_authentication_tokens do
+    hero.sessions.destroy_all
+    hero.authentication_tokens.destroy_all
+  end
+
+  step :archive_pseudonymized_data do
+    DatapointArchive.create(name> "user-#{hero.id}-datapoints.gz") do |io|
+      CSV(io) do |csv|
+        csv << hero.datapoints.first.attributes.keys
+        hero.datapoints.each do |datapoint|
+          csv << Pseudonymizer.scrub(datapoint.attributes.values)
+        end
+      end
+    end
+  end
+
+  step :delete_data do
+    hero.datapoints.in_batches.destroy_all
+  end
+
+  step :send_deletion_email do
+    AccountErasureCompleteMailer.erasure_complete(hero).deliver_later
+  end
+end
+```
+
+While this is seemingly overkill to have steps defined for this type of workflow, the basic premise of a `Journey` still offers you substantial benefits. For example, you never want to enter `delete_data` before `archive_pseudonymized_data` has completed. Same with the `send_deletion_email` - you do not want to notify the user berore their data is actually gone. Neither do you want there to ever be more than 1 process executing any of those steps.
+
+## More sample journeys: performing an outgoing payment
+
+Another fairly widely known use case for step workflows is initiating a payment. We first initiate a payment through an external provider, and then poll for its state to revert or complete the payment.
+
+```ruby
+class PaymentInitiationJourney < StepperMotor::Journey
+  step :initiate_payment do
+    ik = hero.idempotency_key # The `hero` in this case is a Payment, not the User
+    result = PaymentProvider.transfer!(
+      from_account: hero.sender.bank_account_details,
+      to_account: hero.recipient.bank_account_details,
+      amount: hero.amount,
+      idempotency_key: ik
+    )
+    if result.intermittent_error?
+      reattempt!(wait: 5.seconds)
+    elsif result.invalid_request?
+      hero.failed!
+      cancel!
+    else
+      hero.processing!
+      # and then do nothing and proceed to the next step
+    end
+  end
+
+  step :confirm_payment do
+    ik = hero.idempotency_key # The `hero` in this case is a Payment, not the User
+    payment_details = PaymentProvider.details(idempotency_key: ik)
+    case payment_details.state
+    when :complete
+      hero.complete!
+      PaymentSentNotification.notify_sender_of_success(hero.sender).deliver_later
+    when :failed
+      hero.failed!
+      PaymentSentNotification.notify_sender_of_failure(hero.sender).deliver_later
+    else
+      logger.info {"Payment #{hero} still confirming" }
+      reattempt!(wait: 30.seconds) if payment_details.state == :processing
+    end
+  end
+end
+```
+
+Here, we first initiate a payment using an idempotency key, and then poll for its completion or failure repeatedly. When a payment fails or succeeds, we notify the sender and finish the `Journey`. Note that this `Journey` is of a _payment,_ not of the user. A user may have multiple Payments in flight, each with their own `Journey` being tracket transactionally and correctly.
+
+## Transactional semantics
+
+Getting the transactional semantics _right_ with a system like StepperMotor is crucial. We strike a decent balance between reliability/durability and performance, namely:
+
+* The initial "checkout" of a `Journey` for performing a step is lock-guarded
+* Inside the lock guard the `state` of the `Journey` gets set to `performing` - you can see that a journey is currently being performed, and no other processes will evern checkout that same `Journey`
+* The transaction is only applied at the start of the step, _outside_ of that step's block. This means that you can perform long-running operations in your steps, as long as they are idempotent - and manage transactions inside of the steps.
+
+We chose to make StepperMotor "transactionless" inside the steps because the operations and side effects we usually care about would be long-running and performing HTTP or RPC requests. Had the step been wrapped with a transaction, the transaction could become very long - creating a potential for a fairly large rollback in case the step fails.
+
+Another reason why we avoid forced transactions is that if, for whatever reason, you need multiple idempotent actions _inside_ of a step the outer transaction would not permit you to have those. We prefer leaving that flexibility to the end application.
+
+## Saving side-effects of steps
+
+Right now, StepperMotor does not provide any specific tools for saving side-effects or inputs of a step or of the entire `Journey` except for the related `hero` record. The reason for that is that side effects can take many shapes. A side effect may be a file output to S3, a record saved into your database, a file on the filesystem, or a blob of JSON carried around. The way this data has to be persisted can also vary. For the moment, we don't see a good _generalized_ way to persist those side effects aside of the factual outputs. So:
+
+* A record of the fact that a step has been performed to completion is sufficient to not re-enter that step
+* If you need repeatable, but idempotent steps - idempotency is on you
+
+## Unique Journeys
+
+By default, StepperMotor will only allow you to have one active `Journey` per journey type for any given specific `hero`. This will fail, either with a uniqueness constraint violation or a validation error:
+
+```ruby
+SomeJourney.create!(hero: user)
+SomeJourney.create!(hero: user)
+```
+
+Once a `Journey` becomes `canceled` or `finished`, another `Journey` of the same class can be created again for the same `hero`. If you need to create multiple `Journeys` of the same class for the same `hero`, pass the `allow_multiple` attribute set to `true`. This value gets persisted and affects the inclusion of the `Journey` into a partial index that enforces uniqueness:
+
+```ruby
+SomeJourney.create!(hero: user, allow_multiple: true)
+SomeJourney.create!(hero: user, allow_multiple: true)
+```
+
+## Querying for Journeys already created
+
+Frequently, you will encounter the need to select `heroes` to create `Journeys` for. You will likely want to create `Journeys` only for those `heroes` who do not have these `Journeys` yet. You can use a shortcut to generate you the SQL query to use in a `WHERE NOT EXISTS` SQL clause. Usually, your query will look something like this:
+
+```sql
+SELECT users.* FROM users WHERE NOT EXISTS (SELECT 1 FROM stepper_motor_journeys WHERE type = 'YourJourney' AND hero_id = users.id)
+```
+
+To make this simpler, we offer a special helper method:
+
+```ruby
+YourJourney.presence_sql_for(User) # => SELECT 1 FROM stepper_motor_journeys WHERE type = 'YourJourney' AND hero_id = users.id
+```
+
+## What to pick as the hero
+
+If your use case requires complex associations, you may want to make your `hero` a record representing the business process that the `Journey` tracks, instead of making the "actor" (say, an `Account`) the hero. This will allow for better granularity and better-looking code that will be easier to understand.
+
+So instead of doing this:
+
+```ruby
+class PurchaseJourney < StepperMotor::Journey
+  step :start_checkout do
+    hero.purchases.create!(sku: ...)
+  end
+end
+
+PurchaseJourney.create!(hero: user, allow_multiple: true)
+```
+
+try this:
+
+```ruby
+class PurchaseJourney < StepperMotor::Journey
+  step :start_checkout do
+    hero.checkout_started!
+  end
+end
+
+purchase = user.purchases.create!(sku: ...)
+PurchaseJourney.create!(hero: purchase)
+```
+
+## Forward-scheduling or in-time scheduling
+
+There are two known approaches for scheduling jobs far into the future. One approach is "in-time scheduling" - regularly run a _scheduling task_ which performs the steps that are up for execution. The code for such process would look roughly looks like this:
+
+```ruby
+Journey.where("state = 'ready' AND next_step_to_be_performed_at <= NOW()").find_each(&:perform_next_step!)
+```` 
+
+This scheduling task needs to be run with a high-enough frequency which matches your scheduling patterns.
+
+Another is "forward-scheduling" - when it is known that a step of a journey will have to be performed at a certain point in time, enqueue a job which is going to perform the step:
+
+```ruby
+PerformStepJob.set(wait: journey.next_step_to_be_performed_at).perform_later(journey)
+```
+
+This creates a large number of jobs on your queue, but will be easier to manage. StepperMotor supports both approaches, and you can configure the one you like using the configuration:
+
+```ruby
+StepperMotor.configure do |c|
+  # Use jobs per journey step and enqueue them early
+  c.schedule_via = :waiting_jobs
+end
+```
+
+or, for in-time scheduling
+
+```ruby
+StepperMotor.configure do |c|
+  # Use jobs per journey step and enqueue them early
+  c.schedule_via = :central_task
+end
+```
+
+If you use in-time scheduling you will need to add the `StepperMotor::ScheduleLoopJob` to your cron jobs, and perform it frequently enough. Note that having just the granularity of your cron jobs (minutes) may not be enough as reattempts of the steps may get scheduled with a smaller delay - of a few seconds, for instance.
+
+## Naming steps
+
+stepper_motor will name steps for you. However, using named steps is useful because you then can insert steps between existing ones, and have your `Journey` correctly identify the right step. Steps are performed in the order they are defined. Imagine you start with this step sequence:
+
+```ruby
+step :one do
+  # perform some action
+end
+
+step :two do
+  # perform some other action
+end
+```
+
+You have a `Journey` which is about to start step `one`. When the step gets performed, StepperMotor will do a lookup to find _the next step in order of definition._ In this case the step will be step `two`, so the name of that step will be saved with the `Journey`. Imagine you then edit the code to add an extra step between those:
+
+```ruby
+step :one do
+  # perform some action
+end
+
+step :one_bis_ do
+  # some compliance action
+end
+
+step :two do
+  # perform some other action
+end
+```
+
+Your existing `Journey` is already primed to perform step `two`. However, a `Journey` which is about to perform step `one` will now set `one_bis` as the next step to perform. This allows limited reordering and editing of `Journey` definitions after they have already begun.
+
+So, rules of thumb:
+
+* When steps are recalled to be performed, they get recalled _by name._
+* When preparing for the next step, _the next step from the current in order of definition_ is going to be used.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/stepper-motor/stepper_motor.

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "standard/rake"
+
+task default: %i[spec standard]

data/bin/console

@@ -0,0 +1,11 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "bundler/setup"
+require "stepper_motor"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/generators/install_generator.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+
+module StepperMotor
+  # The generator is used to install StepperMotor. It adds an example Journey, a configing
+  # initializer and the migration that creates tables.
+  # Run it with `bin/rails g stepper_motor:install` in your console.
+  class InstallGenerator < Rails::Generators::Base
+    include ActiveRecord::Generators::Migration
+
+    source_paths << File.join(File.dirname(File.dirname(__FILE__)))
+    class_option :uuid, type: :boolean, desc: "The foreign key type to use for hero_id. Can be either bigint or uuid"
+
+    # Generates monolithic migration file that contains all database changes.
+    def create_migration_file
+      # Migration files are named "...migration_001.rb" etc. This allows them to be emitted
+      # as they get added, and the order of the migrations can be controlled using predictable sorting.
+      # Adding a new migration to the gem is then just adding a file.
+      migration_file_paths_in_order = Dir.glob(__dir__ + "/*_migration_*.rb.erb").sort
+      migration_file_paths_in_order.each do |migration_template_path|
+        untemplated_migration_filename = File.basename(migration_template_path).gsub(/\.erb$/, "")
+        migration_template(migration_template_path, File.join(db_migrate_path, untemplated_migration_filename))
+      end
+    end
+
+    private
+
+    def uuid_fk?
+      options["uuid"].present?
+    end
+
+    def migration_version
+      ActiveRecord::VERSION::STRING.split(".").take(2).join(".")
+    end
+  end
+end

data/lib/generators/stepper_motor_migration_001.rb.erb

@@ -0,0 +1,41 @@
+class StepperMotorMigration001 < ActiveRecord::Migration[<%= migration_version %>]
+  def change
+    <% if uuid_fk? %>
+    create_table :stepper_motor_journeys, id: :uuid do |t|
+    <% else %>
+    create_table :stepper_motor_journeys do |t|
+    <% end %>
+      t.string :type, null: false, index: true
+      t.string :state, default: "ready"
+      t.string :hero_type, null: true
+      <% if uuid_fk? %>
+      t.uuid :hero_id
+      <% else %>
+      t.bigint :hero_id
+      <% end %>
+      t.boolean :allow_multiple, default: false
+      t.string :previous_step_name
+      t.string :next_step_name
+      t.datetime :next_step_to_be_performed_at
+      t.bigint :steps_entered, default: 0, null: false
+      t.bigint :steps_completed, default: 0, null: false
+      t.timestamps
+    end
+    # Foreign key needs to be indexed for rapid lookups of journeys for a specific hero
+    add_index :stepper_motor_journeys, [:hero_type, :hero_id]
+
+    # An index is needed on the type/hero as well to check whether there is a journey
+    # for a specific hero of a specific class
+    add_index :stepper_motor_journeys, [:type, :hero_type, :hero_id]
+
+    # A unique index prevents multiple journeys of the same type from being created for a particular hero
+    add_index :stepper_motor_journeys, [:type, :hero_id, :hero_type], where: "allow_multiple = 'f' AND state IN ('ready', 'performing')", unique: true, name: :one_per_hero_index
+
+    # An index is also needed for cleaning up finished and canceled journeys quickly
+    # for a specific hero of a specific class
+    add_index :stepper_motor_journeys, [:updated_at], where: "state = 'canceled' OR state = 'finished'"
+
+    # An extra index is needed to speed up select-to-perform in case of central scheduling
+    add_index :stepper_motor_journeys, [:next_step_to_be_performed_at], where: "state = 'ready'"
+  end
+end

data/lib/stepper_motor/cyclic_scheduler.rb

@@ -0,0 +1,63 @@
+# The cyclic scheduler is designed to be run regularly via a cron job. On every
+# cycle, it is going to look for Journeys which are going to come up for step execution
+# before the next cycle is supposed to run. Then it is going to enqueue jobs to perform
+# steps on those journeys. Since the scheduler gets run at a discrete interval, but we
+# still them to be processed on time, if we only picked up the journeys which have the
+# step execution time set to now or earlier, we will always have delays. This is why
+# this scheduler enqueues jobs for journeys whose time to run is between now and the
+# next cycle.
+#
+# Once the job gets created, it then gets enqueued and gets picked up by the ActiveJob
+# worker normally. If you are using SQS, which has a limit of 900 seconds for the `wait:`
+# value, you need to run the scheduler at least (!) every 900 seconds, and preferably
+# more frequently (for example, once every 5 minutes). This scheduler is also going to be
+# more gentle with ActiveJob adapters that may get slower with large queue depths, such as
+# good_job. This scheduler is a good fit if you are using an ActiveJob adapter which:
+#
+# * Does not allow easy introspection of jobs in the future (like Redis-based queues)
+# * Limits the value of the `wait:` parameter
+#
+# The scheduler needs to be configured in your cron table.
+class StepperMotor::CyclicScheduler < StepperMotor::ForwardScheduler
+  class RunSchedulingCycleJob < ActiveJob::Base
+    def perform
+      scheduler = StepperMotor.scheduler
+      return unless scheduler.is_a?(StepperMotor::CyclicScheduler)
+      scheduler.run_scheduling_cycle
+    end
+  end
+
+  # Creates a new scheduler. The scheduler needs to know how frequently it is going to be running -
+  # you define that frequency when you configure your cron job that calls `run_scheduling_cycle`. Journeys which
+  # have to perform their steps between the runs of the cycles will generate jobs. The more frequent the scheduling
+  # cycle, the fewer jobs are going to be created per cycle.
+  #
+  # @param cycle_duration[ActiveSupport::Duration] how frequently the scheduler runs
+  def initialize(cycle_duration:)
+    @cycle_duration = cycle_duration
+  end
+
+  # Run a scheduling cycle. This should be called from your ActiveJob that runs on a regular Cron cadence. Ideally you
+  # would call the instance of the scheduler configured for the whole StepperMotor (so that the `cycle_duration` gets
+  # correctly applied, as it is necessary to pick the journeys to step). Normally, you would do this:
+  #
+  # @return [void]
+  def run_scheduling_cycle
+    # Find all the journeys that have to step before the next scheduling cycle. This also picks up journeys
+    # which haven't been scheduled or weren't scheduled on time. We don't want to only schedule
+    # journeys which are "past due", because this would make the timing of the steps very lax.
+    scope = StepperMotor::Journey.where("state = 'ready' AND next_step_name IS NOT NULL AND next_step_to_be_performed_at < ?", Time.current + @cycle_duration)
+    scope.find_each do |journey|
+      schedule(journey)
+    end
+  end
+
+  def schedule(journey)
+    # We assume that the previous `run_scheduling_cycle` has occured recently. The longest time it will take
+    # until the next `run_scheduling_cycle` is the duration of the cycle (`run_scheduling_cycle` did run
+    # just before `schedule` gets called). Therefore, it should be sufficient to assume that if the step
+    # has to run before the next `run_scheduling_cycle`, we have to schedule a job for it right now.
+    time_remaining = journey.next_step_to_be_performed_at - Time.current
+    super if time_remaining <= @cycle_duration
+  end
+end

data/lib/stepper_motor/forward_scheduler.rb

@@ -0,0 +1,20 @@
+# The forward scheduler enqueues a job for every Journey that
+# gets sent to the `#schedule`. The job is then stored in the queue
+# and gets picked up by the ActiveJob worker normally. This is the simplest
+# option if your ActiveJob adapter supports far-ahead scheduling. Some adapters,
+# such as SQS, have limitations regarding the maximum delay after which a message
+# will become visible. For SQS, the limit is 900 seconds. If the job is further in the future,
+# it is likely going to fail to get enqueued. If you are working with a queue adapter
+# either:
+#
+# * Does not allow easy introspection of jobs in the future (like Redis-based queues)
+# * Limits the value of the `wait:` parameter
+#
+# this scheduler is not a good fit for you, and you will need to use the {CyclicScheduler} instead.
+class StepperMotor::ForwardScheduler
+  def schedule(journey)
+    wait = journey.next_step_to_be_performed_at - Time.current
+    wait = 0 if wait.negative?
+    StepperMotor::PerformStepJob.set(wait: wait).perform_later(journey.to_global_id.to_s)
+  end
+end