scheddy 0.2.1 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 86885e07595c5cc31d8a5963684a2618de1505986399fd93e0f54f6ff248bbac
-  data.tar.gz: 409a482e3d28dbdda1f2ec48e06ed7a52ff020e961ce89b89aef976698e9e400
+  metadata.gz: aaf5fc0d82ffab054b047932e5b346d027f3e58ff9c79b1942acf5bf6be38c73
+  data.tar.gz: 6477ad6b6bacb90f76b915146daacd771ec042d70787f4bca6d09e01ffe1f22f
 SHA512:
-  metadata.gz: fa861cb109163a3a86e8b327436441ed4f4f4dbee4b95d1ea605f67500b91110465aded2062b00934d1621211625665f735f07d5a9f8c14d0b8d0d9a9800f560
-  data.tar.gz: 33acabd55cc1ecce55ec9cd0a30e547dad4570d19414382839a02510e6d7543b2d12f5b136aa9951550dbf96a0b7bb7b3085ed974cd5b41e1f157380f642ded1
+  metadata.gz: e09bad65eecbb09c235c8410bf0e3c3f4b402ffbaa2f1c09153189ff5fac187b93a8b1524dafa636b4bc70fa69ebadcfc015884cd924f7f5cc3cdfa8c5d1ffca
+  data.tar.gz: b1075009c017fc0b13fc5399c81f6d2cea4f3f09295de94d781fcb06623e9962e2fc9b08701c566dbe34cacaf601cc48aec444da5fddf4edb760610b5af056a7

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright 2023 thomas morgan
+Copyright 2023-2024 thomas morgan
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -7,6 +7,7 @@ Scheddy is a batteries-included task scheduler for Rails. It is intended as a re
 * Catch up missed tasks. Designed for environments with frequent deploys. Also useful in dev where the scheduler isn't always running.
 * Job-queue agnostic. Works great with various ActiveJob adapters and non-ActiveJob queues too.
 * Minimal dependencies. Uses your existing database (or no database at all). Redis not required either.
+* Built-in cluster support by running 2+ Scheddy instances. (Database required in this case.)
 * Tasks and their schedules are versioned as part of your code.
 
 
@@ -18,13 +19,13 @@ Add to your application's `Gemfile`:
 gem "scheddy"
 ```
 
-After running `bundle install`, add the migration to your app:
+After running `bundle install`, add the migrations to your app:
 ```bash
 bin/rails scheddy:install:migrations
 bin/rails db:migrate
 ```
 
-FYI, if all tasks set `track_runs false`, the migration can be skipped.
+FYI, if all tasks set `track_runs false` and running only a single Scheddy instance, the migrations are optional.
 
 
 
@@ -129,7 +130,16 @@ Database transactions are valid. These can increase use of database connections
 
 Each task runs in its own thread which helps ensure all tasks perform on time. However, Scheddy is not intended as a job executor and doesn't have a robust mechanism for retrying failed jobs--that belongs to your background job queue.
 
-A given task will only ever be executed once at a time. Mostly relevant when using tiny intervals, if a prior execution is still going when the next execution is scheduled, Scheddy will skip the next execution and log an error message to that effect.
+A given task will only ever be executed once at a time. Mostly relevant when using tiny intervals, if a prior execution is still going when the next execution is scheduled to start, Scheddy will skip the next execution and log an error message to that effect.
+
+
+#### Running 2+ Scheddy instances in a cluster
+
+Running multiple Scheddy instances will automatically form a cluster with one leader and however many standbys. Tasks are only executed by the leader.
+
+Clean and unclean handoffs are both supported, taking 1-2 and 4-5 minutes respectively. After a handoff, task executions will be caught up by the new leader, same as if a single instance had been stopped and restarted.
+
+Note that the prior section's promise that a task will only be executed once at a time cannot be guaranteed in the case of a handoff between Scheddy instances. Since tasks are intended to run very quickly, and long tasks should run via ActiveJob instead, this should not be an issue in practice.
 
 
 #### Task context
@@ -193,24 +203,21 @@ You can also check your tasks configuration with:
   bundle exec scheddy tasks
 ```
 
+When running 2+ Scheddy's, a handoff may be initiated with:
+```bash
+  scheddy stepdown
+  # OR
+  bundle exec scheddy stepdown
+```
+
 
 ### In production
 
-Scheddy runs as its own process. It is intended to be run only once. Because Scheddy has the ability to catch up missed tasks, redundancy should be achieved through automatic restarts via `systemd`, `dockerd`, Kubernetes, or whatever supervisory system you use.
-
-During deployment, shutdown the old instance before starting the new one. In Kubernetes this might look like:
-```yaml
-kind: Deployment
-spec:
-  replicas: 1
-  strategy:
-    rollingUpdate:
-      maxSurge: 0
-      maxUnavailable: 1
-  template:
-    spec:
-      terminationGracePeriodSeconds: 60
-```
+Scheddy runs as its own process. During app deployments, it is recommended to shutdown the old Scheddy instance before starting the new one.
+
+Since Scheddy has the ability to catch up missed tasks, it is often viable to run just a single Scheddy instance, especially when that instance is automatically restarted by `systemd`, `dockerd`, Kubernetes, or whatever supervisory system you use.
+
+It is also possible to run 2+ Scheddy instances as a cluster. One instance will be selected as the leader and execute tasks. The other instance(s) will operate as standbys, ready to take over if the leader steps down or fails.
 
 
 ### In development (and `Procfile` in production)

data/app/models/scheddy/task_scheduler.rb

@@ -0,0 +1,64 @@
+module Scheddy
+  class TaskScheduler < ApplicationRecord
+
+    validates :leader_expires_at,
+      presence: {if: :leader_state},
+      absence: {unless: :leader_state}
+
+    validates :leader_state,
+      inclusion: [nil, 'leader']
+
+
+    scope :leader,     ->{ where(leader_state: 'leader') }
+    scope :not_leader, ->{ where.not(leader_state: 'leader') }
+    scope :stale,      ->{ where(last_seen_at: ..2.hours.ago).not_leader }
+
+
+    def expired?
+      leader_expires_at && leader_expires_at < Time.current
+    end
+
+    def leader?
+      leader_state == 'leader'
+    end
+
+
+    def clear_leader(only_if_expired: false)
+      reload if changed?
+      with_lock do
+        if !only_if_expired || expired?
+          update! leader_state: nil, leader_expires_at: nil
+        end
+      end
+    end
+
+    def mark_seen
+      if last_seen_at < (LEASE_RENEWAL_INTERVAL - 5.seconds).ago
+        update! last_seen_at: Time.current
+      end
+    end
+
+    def renew_leadership
+      if last_seen_at < (LEASE_RENEWAL_INTERVAL - 5.seconds).ago
+        update! leader_expires_at: LEASE_DURATION.from_now, last_seen_at: Time.current
+      else
+        true
+      end
+    rescue ActiveRecord::StaleObjectError
+      reload
+      false
+    end
+
+    def take_leadership
+      update! leader_state: 'leader', leader_expires_at: LEASE_DURATION.from_now, last_seen_at: Time.current
+    rescue ActiveRecord::RecordNotUnique
+      false
+    end
+
+    def request_stepdown
+      # intentionally leaves self.lock_version behind
+      self.class.increment_counter :lock_version, id, touch: true if leader?
+    end
+
+  end
+end

data/db/migrate/20230607201527_create_scheddy_task_histories.rb

@@ -1,6 +1,5 @@
 class CreateScheddyTaskHistories < ActiveRecord::Migration[6.0]
   def change
-    # feel free to modify to  id: :uuid  or another :id format if you prefer
     create_table :scheddy_task_histories do |t|
       t.string     :name,        null: false, index: {unique: true}
       t.datetime   :last_run_at

data/db/migrate/20240822165904_create_scheddy_task_schedulers.rb

@@ -0,0 +1,15 @@
+class CreateScheddyTaskSchedulers < ActiveRecord::Migration[6.0]
+  def change
+    create_table :scheddy_task_schedulers, id: :string do |t|
+      t.string     :hostname,          null: false
+      t.datetime   :last_seen_at,      null: false
+      t.datetime   :leader_expires_at
+      t.string     :leader_state
+      t.integer    :lock_version,      null: false, default: 0
+      t.integer    :pid,               null: false
+      t.timestamps
+
+      t.index :leader_state, unique: true
+    end
+  end
+end

data/lib/scheddy/cli.rb

@@ -17,6 +17,14 @@ module Scheddy
     end
 
 
+    desc :stepdown, 'Ask current Scheddy leader to step down'
+    def stepdown
+      load_app!
+      puts 'Requesting step down...'
+      Scheddy::TaskScheduler.leader.first&.request_stepdown
+    end
+
+
     desc :tasks, 'Show configured tasks'
     def tasks
       load_app!

data/lib/scheddy/config.rb

@@ -2,13 +2,6 @@ module Scheddy
   # default task list for when running standalone
   mattr_accessor :tasks, default: []
 
-  # called from within task's execution thread; must be multi-thread safe
-  # task is allowed to be nil
-  mattr_accessor :error_handler, default: lambda {|e, task|
-    logger.error "Exception in Scheddy task '#{task&.name}': #{e.inspect}\n  #{e.backtrace.join("\n  ")}"
-    Rails.error.report(e, handled: true, severity: :error)
-  }
-
   def self.config(&block)
     Config.new(tasks, &block)
   end
@@ -37,8 +30,8 @@ module Scheddy
     def run_at(cron, name:, tag: :auto, track: :auto, &task)
       task(name) do
         run_at     cron
-        logger_tag tag if tag!=:auto
-        track_runs track if track!=:auto
+        logger_tag tag if tag != :auto
+        track_runs track if track != :auto
         perform(&task)
       end
     end
@@ -48,8 +41,8 @@ module Scheddy
       task(name) do
         run_every     interval
         initial_delay delay if delay
-        logger_tag    tag if tag!=:auto
-        track_runs    track if track!=:auto
+        logger_tag    tag if tag != :auto
+        track_runs    track if track != :auto
         perform(&task)
       end
     end

data/lib/scheddy/error_handler.rb

@@ -0,0 +1,16 @@
+module Scheddy
+  # called from within task's execution thread; must be multi-thread safe
+  # task is allowed to be nil
+  mattr_accessor :error_handler, default: lambda {|e, task|
+    task &&= "task '#{task&.name}' "
+    logger.error "Exception in Scheddy #{task}: #{e.inspect}\n  #{e.backtrace.join("\n  ")}"
+    Rails.error.report(e, handled: true, severity: :error)
+  }
+
+  def self.handle_error(e, task=nil)
+    if h = Scheddy.error_handler
+      h.call(*[e, task].take(h.arity.abs))
+    end
+  end
+
+end

data/lib/scheddy/scheduler.rb

@@ -1,4 +1,7 @@
 module Scheddy
+  LEASE_RENEWAL_INTERVAL = 1.minute
+  LEASE_DURATION         = 4.minutes
+    # must be > 2x the renewal interval
 
   def self.run
     Scheduler.new(tasks).run
@@ -7,69 +10,149 @@ module Scheddy
   class Scheduler
 
     def run
-      puts "[Scheddy] Starting scheduler with #{tasks.size} #{'task'.pluralize tasks.size}"
+      puts "[scheddy] Hello. This is Scheddy v#{VERSION}."
+      puts "[scheddy] hostname=#{hostname}, pid=#{pid}, id=#{scheduler_id}"
       trap_signals!
-      cleanup_task_history
+      puts "[scheddy] Starting scheduler with #{tasks.size} #{'task'.pluralize tasks.size}"
+      unless register_process
+        puts '[scheddy] No scheddy_task_schedulers table found; disabling cluster support'
+      end
 
       until stop?
-        next_cycle = run_once
-        wait_until next_cycle unless stop?
+        with_leader do |new_leader|
+          reset_tasks if new_leader
+          cleanup_task_history
+          cleanup_task_scheduler
+
+          next_cycle = run_once
+          if tasks.any? && scheduler_record
+            next_cycle = [next_cycle, LEASE_RENEWAL_INTERVAL.from_now].compact.min
+          end
+          wait_until next_cycle unless stop?
+        end
       end
 
+      stepdown_as_leader
+
       running = tasks.select(&:running?).count
       if running > 0
-        puts "[Scheddy] Waiting for #{running} tasks to complete"
-        wait_until(45.seconds.from_now, skip_stop: true) do
+        puts "[scheddy] Waiting for #{running} tasks to complete"
+        wait_for(45.seconds, skip_stop: true) do
           tasks.none?(&:running?)
         end
         tasks.select(&:running?).each do |task|
-          $stderr.puts "[Scheddy] Killing task #{task.name}"
+          $stderr.puts "[scheddy] Killing task #{task.name}"
           task.kill
         end
       end
 
-      puts '[Scheddy] Done'
+    ensure
+      unregister_process
+      puts '[scheddy] Goodbye'
     end
 
     # return : Time of next cycle
     def run_once
-      tasks.flat_map do |task|
+      if tasks.empty?
+        logger.warn 'No tasks found; doing nothing'
+        return 1.hour.from_now
+      end
+      tasks.filter_map do |task|
         task.perform(self) unless stop?
       end.min
     end
 
+    def stepdown? ; @stepdown ; end
     def stop? ; @stop ; end
 
+    def hostname
+      @hostname ||= Socket.gethostname.force_encoding(Encoding::UTF_8)
+    end
+
+    def pid
+      @pid ||= Process.pid
+    end
+
+    def scheduler_id
+      @scheduler_id ||= SecureRandom.alphanumeric 12
+    end
+
+    def logger
+      @logger ||= Scheddy.logger.tagged "scheddy-#{scheduler_id}"
+    end
+
 
     private
 
-    attr_reader :tasks
-    attr_writer :stop
+    attr_reader :scheduler_record, :tasks
+    attr_writer :stepdown, :stop
+    attr_accessor :leader_state
 
     def initialize(tasks)
       @tasks = tasks
+      self.leader_state = :standby
     end
 
     def cleanup_task_history
-      known_tasks = tasks.select(&:track_runs).map(&:name)
-      return if known_tasks.empty?  # table doesn't have to exist if track_runs always disabled
-      Scheddy::TaskHistory.find_each do |r|
-        r.destroy if known_tasks.exclude? r.name
+      return if @cleaned_tasks
+      @cleaned_tasks = true
+      return unless Scheddy::TaskHistory.table_exists?
+      trackable_tasks = tasks.select(&:track_runs).map(&:name)
+      return if trackable_tasks.empty?
+      Scheddy::TaskHistory.where(updated_at: ..1.day.ago).where.not(name: trackable_tasks).find_each do |r|
+        r.destroy
+      end
+    end
+
+    def cleanup_task_scheduler
+      return if @cleaned_schedulers
+      @cleaned_schedulers = true
+      return unless Scheddy::TaskScheduler.table_exists?
+      Scheddy::TaskScheduler.stale.find_each do |r|
+        logger.debug "Removing stale scheduler record for id=#{r.id}"
+        r.destroy
+      end
+    end
+
+    def register_process
+      return false unless Scheddy::TaskScheduler.table_exists?
+      @scheduler_record ||= Scheddy::TaskScheduler.create!(
+        id:           scheduler_id,
+        hostname:     hostname,
+        last_seen_at: Time.current,
+        pid:          pid
+      )
+    end
+
+    def unregister_process
+      Scheddy::TaskScheduler.delete scheduler_record.id if scheduler_record
+    end
+
+    def reset_tasks
+      tasks.each(&:reset)
+    end
+
+    def stepdown!(sig=nil)
+      if scheduler_record&.leader?
+        puts '[scheddy] Requesting step down'
+        self.stepdown = true
       end
-    rescue ActiveRecord::StatementInvalid => e
-      return if e.message =~ /relation "scheddy_task_histories" does not exist/
-      raise
     end
 
     def stop!(sig=nil)
-      puts '[Scheddy] Stopping'
+      puts '[scheddy] Stopping'
       self.stop = true
     end
 
     def trap_signals!
-      trap 'INT', &method(:stop!)
+      trap 'INT',  &method(:stop!)
       trap 'QUIT', &method(:stop!)
       trap 'TERM', &method(:stop!)
+      trap 'USR1', &method(:stepdown!)
+    end
+
+    def wait_for(duration, skip_stop: false, &block)
+      wait_until duration.from_now, skip_stop:, &block
     end
 
     # &block - optional block - return truthy to end prematurely
@@ -81,5 +164,65 @@ module Scheddy
       end
     end
 
+    def with_leader
+      return yield(false) if !scheduler_record || tasks.empty?
+
+      wait_t = LEASE_RENEWAL_INTERVAL.from_now
+      if leader_state == :standby
+        if current_leader = Scheddy::TaskScheduler.leader.first
+          if current_leader.expired?
+            logger.error "Forcefully clearing expired leader status for id=#{current_leader.id}"
+            current_leader.clear_leader(only_if_expired: true)
+            wait_t = 5.seconds.from_now
+          end
+        else
+          if scheduler_record.take_leadership
+            self.leader_state = :new_leader
+          end
+        end
+      end
+
+      case leader_state
+      when :new_leader
+        logger.info 'We are now cluster leader'
+        self.leader_state = :existing_leader
+        return yield true
+      when :existing_leader
+        if !stepdown? && scheduler_record.renew_leadership
+          return yield false
+        else
+          if stepdown_as_leader
+            scheduler_record.mark_seen
+          end
+          wait_t += 5.seconds # improve odds of another daemon taking over
+        end
+      when :standby
+        scheduler_record.mark_seen
+      end
+
+      wait_until wait_t
+    rescue Exception => e
+      logger.error 'Error in scheduler; retrying in 1 minute'
+      Scheddy.handle_error(e)
+      if leader_state != :standby
+        logger.warn 'Due to prior error, stepping down as leader'
+        stepdown_as_leader skip_msg: true
+      end
+      wait_for 1.minute
+    end
+
+    def stepdown_as_leader(skip_msg: false)
+      return true if leader_state == :standby
+      logger.info 'Stepping down as leader' unless skip_msg
+      scheduler_record.clear_leader
+      self.stepdown = false
+      self.leader_state = :standby
+      true
+    rescue Exception => e
+      logger.error 'Failed to step down as leader'
+      Scheddy.handle_error(e)
+      false
+    end
+
   end
 end

data/lib/scheddy/task.rb

@@ -23,9 +23,7 @@ module Scheddy
                 task.call(*[context].take(task.arity.abs))
               end
             rescue Exception => e
-              if h = Scheddy.error_handler
-                h.call(*[e, self].take(h.arity.abs))
-              end
+              Scheddy.handle_error(e, self)
             end
           end
         ensure
@@ -34,9 +32,7 @@ module Scheddy
       next_cycle!
     rescue Exception => e
       logger.error "Scheddy: error scheduling task '#{name}'; retrying in 5 seconds"
-      if h = Scheddy.error_handler
-        h.call(*[e, self].take(h.arity.abs))
-      end
+      Scheddy.handle_error(e, self)
       return self.next_cycle = 5.seconds.from_now
     end
 
@@ -53,6 +49,11 @@ module Scheddy
       @next_cycle
     end
 
+    def reset
+      self.next_cycle = :initial
+      @task_history = nil
+    end
+
 
     def to_h
       attrs = {
@@ -169,6 +170,12 @@ module Scheddy
 
     def record_this_run
       return unless track_runs
+      begin
+        Scheddy::TaskHistory.connection.verify!
+      rescue => e
+        logger.error "Database offline while updating task history for Scheddy task '#{name}': #{e.inspect}"
+        return
+      end
       Scheddy::TaskHistory.logger.silence(Logger::INFO) do
         task_history.update last_run_at: Time.current
       end

data/lib/scheddy/version.rb

@@ -1,3 +1,3 @@
 module Scheddy
-  VERSION = "0.2.1"
+  VERSION = '0.3.0'
 end

data/lib/scheddy.rb

@@ -6,6 +6,7 @@ end
 %w(
   config
   context
+  error_handler
   logger
   scheduler
   task

data/lib/tasks/scheddy_tasks.rake

@@ -5,6 +5,12 @@ namespace :scheddy do
     Scheddy.run
   end
 
+  desc 'Ask current Scheddy leader to step down'
+  task stepdown: :environment do
+    puts 'Requesting step down...'
+    Scheddy::TaskScheduler.leader.first&.request_stepdown
+  end
+
   task :migrate do
     `bin/rails db:migrate SCOPE=scheddy`
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: scheddy
 version: !ruby/object:Gem::Version
-  version: 0.2.1
+  version: 0.3.0
 platform: ruby
 authors:
 - thomas morgan
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-06-09 00:00:00.000000000 Z
+date: 2024-08-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fugit
@@ -54,7 +54,8 @@ dependencies:
         version: '1.0'
 description: Scheddy is a batteries-included task scheduler for Rails. It is intended
   as a replacement for cron and cron-like functionality (including job queue specific
-  schedulers). It is job-queue agnostic and can catch up missed tasks.
+  schedulers). It is job-queue agnostic, can catch up missed tasks, and has native
+  clustering.
 email:
 - tm@iprog.com
 executables:
@@ -67,13 +68,16 @@ files:
 - Rakefile
 - app/models/scheddy/application_record.rb
 - app/models/scheddy/task_history.rb
+- app/models/scheddy/task_scheduler.rb
 - db/migrate/20230607201527_create_scheddy_task_histories.rb
+- db/migrate/20240822165904_create_scheddy_task_schedulers.rb
 - exe/scheddy
 - lib/scheddy.rb
 - lib/scheddy/cli.rb
 - lib/scheddy/config.rb
 - lib/scheddy/context.rb
 - lib/scheddy/engine.rb
+- lib/scheddy/error_handler.rb
 - lib/scheddy/logger.rb
 - lib/scheddy/scheduler.rb
 - lib/scheddy/task.rb
@@ -101,9 +105,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
+rubygems_version: 3.5.11
 signing_key:
 specification_version: 4
 summary: Job-queue agnostic, cron-like task scheduler for Rails apps, with missed
-  task catch-ups and other features.
+  task catch-ups, clustering, and other features.
 test_files: []