solid_queue 0.2.2 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d907fb9133f2c72a61b586038f05b248a29e7c55f506aa541ca39f35a15d40ff
-  data.tar.gz: 887ecd7d0a15159ae10845466993699165a531367dbb21151e3ae704e56fa8e9
+  metadata.gz: 2c5bed020eb4391f1885c95f2f4373e30d1eacd544e2cfcadf131c8c09079fd0
+  data.tar.gz: 749ac036b072622cd2cc299f1fc5b1a7353a44347beccaa8407157eed8af02ef
 SHA512:
-  metadata.gz: 665bb353c9cc8c557952ca5e2f63120b0541eef7a20008b6c288147e0350564200be2e6272f9924448867b9b2d09a7e48dc5b2267ec35b91f5dcd38bc409fff1
-  data.tar.gz: 9e911e1a270b5da57f75011e40ad6949480a8b87a2eb391934697159701273e1f05da9ac5c114e8d9dac492bb7685f6884eacac592ed44a460f513fd7349f85a
+  metadata.gz: f39c8aadb36bc8ae5556f3cbd190e6d123ca441b14374b56c207530dd2d37d8dc4880f6e977f6bc2e047b47c886768a3c86d6fefd040a447f6d8f21e28e8a273
+  data.tar.gz: d95c4be96c4388c6558a32f44690e9d92913e47bd01a379f09ef8b3e65a9098a2e578dd68885c36374b27e389c65d67e1100ad4d1c6908e6f675ede75bce8d3a

data/README.md

@@ -2,7 +2,7 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, unique jobs and recurring, cron-like tasks are coming very soon._
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, and some way of specifying unique jobs are coming very soon._
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
@@ -66,6 +66,8 @@ $ bundle exec rake solid_queue:start
 
 This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
+For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bundle exec rake solid_queue:start` on multiple machines at the same time. If you'd like to designate some machines to be only dispatchers or only workers, use `bundle exec rake solid_queue:dispatch` or `bundle exec rake solid_queue:work`, respectively.
+
 ## Requirements
 Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
 
@@ -75,7 +77,7 @@ Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as t
 
 We have three types of processes in Solid Queue:
 - _Workers_ are in charge of picking jobs ready to run from queues and processing them. They work off the `solid_queue_ready_executions` table.
-- _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They also do some maintenance work related to concurrency controls.
+- _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They're also in charge of managing [recurring tasks](#recurring-tasks), dispatching jobs to process them according to their schedule. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
 - The _supervisor_ forks workers and dispatchers according to the configuration, controls their heartbeats, and sends them signals to stop and start them when needed.
 
 By default, Solid Queue will try to find your configuration under `config/solid_queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG`. This is what this configuration looks like:
@@ -115,8 +117,10 @@ Everything is optional. If no configuration is provided, Solid Queue will run wi
   This will create a worker fetching jobs from all queues starting with `staging`. The wildcard `*` is only allowed on its own or at the end of a queue name; you can't specify queue names such as `*_some_queue`. These will be ignored.
 
   Finally, you can combine prefixes with exact names, like `[ staging*, background ]`, and the behaviour with respect to order will be the same as with only exact names.
-- `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `5`. Only workers have this setting.
+- `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
 - `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
+- `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
+- `recurring_tasks`: a list of recurring tasks the dispatcher will manage. Read more details about this one in the [Recurring tasks](#recurring-tasks) section.
 
 
 ### Queue order and priorities
@@ -131,7 +135,7 @@ We recommend not mixing queue order with priorities but either choosing one or t
 
 ### Threads, processes and signals
 
-Workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Besides this, parallelism can be achieved via multiple processes, configurable via different workers or the `processes` parameter above.
+Workers in Solid Queue use a thread pool to run work in multiple threads, configurable via the `threads` parameter above. Besides this, parallelism can be achieved via multiple processes on one machine (configurable via different workers or the `processes` parameter above) or by horizontal scaling.
 
 The supervisor is in charge of managing these processes, and it responds to the following signals:
 - `TERM`, `INT`: starts graceful termination. The supervisor will send a `TERM` signal to its supervised processes, and it'll wait up to `SolidQueue.shutdown_timeout` time until they're done. If any supervised processes are still around by then, it'll send a `QUIT` signal to them to indicate they must exit.
@@ -263,3 +267,48 @@ Solid Queue has been inspired by [resque](https://github.com/resque/resque) and
 
 ## License
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Recurring tasks
+Solid Queue supports defining recurring tasks that run at specific times in the future, on a regular basis like cron jobs. These are managed by dispatcher processes and as such, they can be defined in the dispatcher's configuration like this:
+```yml
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+      recurring_tasks:
+        my_periodic_job:
+          class: MyJob
+          args: [ 42, { status: "custom_status" } ]
+          schedule: every second
+```
+`recurring_tasks` is a hash/dictionary, and the key will be the task key internally. Each task needs to have a class, which will be the job class to enqueue, and a schedule. The schedule is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). You can also provide arguments to be passed to the job, as a single argument, a hash, or an array of arguments that can also include kwargs as the last element in the array.
+
+The job in the example configuration above will be enqueued every second as:
+```ruby
+MyJob.perform_later(42, status: "custom_status")
+```
+
+Tasks are enqueued at their corresponding times by the dispatcher that owns them, and each task schedules the next one. This is pretty much [inspired by what GoodJob does](https://github.com/bensheldon/good_job/blob/994ecff5323bf0337e10464841128fda100750e6/lib/good_job/cron_manager.rb).
+
+It's possible to run multiple dispatchers with the same `recurring_tasks` configuration. To avoid enqueuing duplicate tasks at the same time, an entry in a new `solid_queue_recurring_executions` table is created in the same transaction as the job is enqueued. This table has a unique index on `task_key` and `run_at`, ensuring only one entry per task per time will be created. This only works if you have `preserve_finished_jobs` set to `true` (the default), and the guarantee applies as long as you keep the jobs around.
+
+Finally, it's possible to configure jobs that aren't handled by Solid Queue. That's it, you can a have a job like this in your app:
+```ruby
+class MyResqueJob < ApplicationJob
+  self.queue_adapter = :resque
+
+  def perform(arg)
+    # ..
+  end
+end
+```
+
+You can still configure this in Solid Queue:
+```yml
+  dispatchers:
+    - recurring_tasks:
+        my_periodic_resque_job:
+          class: MyResqueJob
+          args: 22
+          schedule: "*/5 * * * *"
+```
+and the job will be enqueued via `perform_later` so it'll run in Resque. However, in this case we won't track any `solid_queue_recurring_execution` record for it and there won't be any guarantees that the job is enqueued only once each time.

data/app/models/solid_queue/blocked_execution.rb

@@ -43,7 +43,7 @@ module SolidQueue
           promote_to_ready
           destroy!
 
-          SolidQueue.logger.info("[SolidQueue] Unblocked job #{job.id} under #{concurrency_key}")
+          SolidQueue.logger.debug("[SolidQueue] Unblocked job #{job.id} under #{concurrency_key}")
         end
       end
     end

data/app/models/solid_queue/claimed_execution.rb

@@ -16,7 +16,6 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
       insert_all!(job_data)
       where(job_id: job_ids, process_id: process_id).load.tap do |claimed|
         block.call(claimed)
-        SolidQueue.logger.info("[SolidQueue] Claimed #{claimed.size} jobs")
       end
     end
 

data/app/models/solid_queue/job/clearable.rb

@@ -6,13 +6,13 @@ module SolidQueue
       extend ActiveSupport::Concern
 
       included do
-        scope :clearable, ->(finished_before: SolidQueue.clear_finished_jobs_after.ago) { where.not(finished_at: nil).where(finished_at: ...finished_before) }
+        scope :clearable, ->(finished_before: SolidQueue.clear_finished_jobs_after.ago, class_name: nil) { where.not(finished_at: nil).where(finished_at: ...finished_before).where(class_name.present? ? { class_name: class_name } : {}) }
       end
 
       class_methods do
-        def clear_finished_in_batches(batch_size: 500, finished_before: SolidQueue.clear_finished_jobs_after.ago)
+        def clear_finished_in_batches(batch_size: 500, finished_before: SolidQueue.clear_finished_jobs_after.ago, class_name: nil)
           loop do
-            records_deleted = clearable(finished_before: finished_before).limit(batch_size).delete_all
+            records_deleted = clearable(finished_before: finished_before, class_name: class_name).limit(batch_size).delete_all
             break if records_deleted == 0
           end
         end

data/app/models/solid_queue/job/executable.rb

@@ -6,7 +6,7 @@ module SolidQueue
       extend ActiveSupport::Concern
 
       included do
-        include Clearable, ConcurrencyControls, Schedulable
+        include ConcurrencyControls, Schedulable
 
         has_one :ready_execution
         has_one :claimed_execution
@@ -78,7 +78,7 @@ module SolidQueue
       end
 
       def finished!
-        if preserve_finished_jobs?
+        if SolidQueue.preserve_finished_jobs?
           touch(:finished_at)
         else
           destroy!
@@ -117,10 +117,6 @@ module SolidQueue
         def execution
           %w[ ready claimed failed ].reduce(nil) { |acc, status| acc || public_send("#{status}_execution") }
         end
-
-        def preserve_finished_jobs?
-          SolidQueue.preserve_finished_jobs
-        end
     end
   end
 end

data/app/models/solid_queue/job/recurrable.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Job
+    module Recurrable
+      extend ActiveSupport::Concern
+
+      included do
+        has_one :recurring_execution, dependent: :destroy
+      end
+    end
+  end
+end

data/app/models/solid_queue/job.rb

@@ -2,7 +2,7 @@
 
 module SolidQueue
   class Job < Record
-    include Executable
+    include Executable, Clearable, Recurrable
 
     serialize :arguments, coder: JSON
 

data/app/models/solid_queue/recurring_execution.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class RecurringExecution < Execution
+    scope :clearable, -> { where.missing(:job) }
+
+    class << self
+      def record(task_key, run_at, &block)
+        transaction do
+          if job_id = block.call
+            create!(job_id: job_id, task_key: task_key, run_at: run_at)
+          end
+        end
+      rescue ActiveRecord::RecordNotUnique
+        SolidQueue.logger.info("[SolidQueue] Skipped recurring task #{task_key} at #{run_at} — already dispatched")
+      end
+
+      def clear_in_batches(batch_size: 500)
+        loop do
+          records_deleted = clearable.limit(batch_size).delete_all
+          break if records_deleted == 0
+        end
+      end
+    end
+  end
+end

data/db/migrate/20240218110712_create_recurring_executions.rb

@@ -0,0 +1,14 @@
+class CreateRecurringExecutions < ActiveRecord::Migration[7.1]
+  def change
+    create_table :solid_queue_recurring_executions do |t|
+      t.references :job, index: { unique: true }, null: false
+      t.string :task_key, null: false
+      t.datetime :run_at, null: false
+      t.datetime :created_at, null: false
+
+      t.index [ :task_key, :run_at ], unique: true
+    end
+
+    add_foreign_key :solid_queue_recurring_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade
+  end
+end

data/lib/solid_queue/configuration.rb

@@ -4,7 +4,7 @@ module SolidQueue
   class Configuration
     WORKER_DEFAULTS = {
       queues: "*",
-      threads: 5,
+      threads: 3,
       processes: 1,
       polling_interval: 0.1
     }
@@ -12,7 +12,9 @@ module SolidQueue
     DISPATCHER_DEFAULTS = {
       batch_size: 500,
       polling_interval: 1,
-      concurrency_maintenance_interval: 600
+      concurrency_maintenance: true,
+      concurrency_maintenance_interval: 600,
+      recurring_tasks: []
     }
 
     def initialize(mode: :work, load_from: nil)
@@ -33,7 +35,7 @@ module SolidQueue
       if mode.in? %i[ work all]
         workers_options.flat_map do |worker_options|
           processes = worker_options.fetch(:processes, WORKER_DEFAULTS[:processes])
-          processes.times.collect { SolidQueue::Worker.new(**worker_options.with_defaults(WORKER_DEFAULTS)) }
+          processes.times.map { Worker.new(**worker_options.with_defaults(WORKER_DEFAULTS)) }
         end
       else
         []
@@ -42,8 +44,10 @@ module SolidQueue
 
     def dispatchers
       if mode.in? %i[ dispatch all]
-        dispatchers_options.flat_map do |dispatcher_options|
-          SolidQueue::Dispatcher.new(**dispatcher_options)
+        dispatchers_options.map do |dispatcher_options|
+          recurring_tasks = parse_recurring_tasks dispatcher_options[:recurring_tasks]
+
+          Dispatcher.new **dispatcher_options.merge(recurring_tasks: recurring_tasks).with_defaults(DISPATCHER_DEFAULTS)
         end
       end
     end
@@ -73,6 +77,11 @@ module SolidQueue
           .map { |options| options.dup.symbolize_keys }
       end
 
+      def parse_recurring_tasks(tasks)
+        Array(tasks).map do |id, options|
+          Dispatcher::RecurringTask.from_configuration(id, **options)
+        end.select(&:valid?)
+      end
 
       def load_config_from(file_or_hash)
         case file_or_hash

data/lib/solid_queue/dispatcher/concurrency_maintenance.rb

@@ -0,0 +1,44 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Dispatcher::ConcurrencyMaintenance
+    include AppExecutor
+
+    attr_reader :interval, :batch_size
+
+    def initialize(interval, batch_size)
+      @interval = interval
+      @batch_size = batch_size
+    end
+
+    def start
+      @concurrency_maintenance_task = Concurrent::TimerTask.new(run_now: true, execution_interval: interval) do
+        expire_semaphores
+        unblock_blocked_executions
+      end
+
+      @concurrency_maintenance_task.add_observer do |_, _, error|
+        handle_thread_error(error) if error
+      end
+
+      @concurrency_maintenance_task.execute
+    end
+
+    def stop
+      @concurrency_maintenance_task.shutdown
+    end
+
+    private
+      def expire_semaphores
+        wrap_in_app_executor do
+          Semaphore.expired.in_batches(of: batch_size, &:delete_all)
+        end
+      end
+
+      def unblock_blocked_executions
+        wrap_in_app_executor do
+          BlockedExecution.unblock(batch_size)
+        end
+      end
+  end
+end

data/lib/solid_queue/dispatcher/recurring_schedule.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Dispatcher::RecurringSchedule
+    include AppExecutor
+
+    attr_reader :configured_tasks, :scheduled_tasks
+
+    def initialize(tasks)
+      @configured_tasks = Array(tasks).map { |task| Dispatcher::RecurringTask.wrap(task) }
+      @scheduled_tasks = Concurrent::Hash.new
+    end
+
+    def load_tasks
+      configured_tasks.each do |task|
+        load_task(task)
+      end
+    end
+
+    def load_task(task)
+      scheduled_tasks[task.key] = schedule(task)
+    end
+
+    def unload_tasks
+      scheduled_tasks.values.each(&:cancel)
+      scheduled_tasks.clear
+    end
+
+    def tasks
+      configured_tasks.each_with_object({}) { |task, hsh| hsh[task.key] = task.to_h }
+    end
+
+    def inspect
+      configured_tasks.map(&:to_s).join(" | ")
+    end
+
+    private
+      def schedule(task)
+        scheduled_task = Concurrent::ScheduledTask.new(task.delay_from_now, args: [ self, task, task.next_time ]) do |thread_schedule, thread_task, thread_task_run_at|
+          thread_schedule.load_task(thread_task)
+
+          wrap_in_app_executor do
+            thread_task.enqueue(at: thread_task_run_at)
+          end
+        end
+
+        scheduled_task.add_observer do |_, _, error|
+          # Don't notify on task cancellation before execution, as this will happen normally
+          # as part of unloading tasks
+          handle_thread_error(error) if error && !error.is_a?(Concurrent::CancelledOperationError)
+        end
+
+        scheduled_task.tap(&:execute)
+      end
+  end
+end

data/lib/solid_queue/dispatcher/recurring_task.rb

@@ -0,0 +1,85 @@
+require "fugit"
+
+module SolidQueue
+  class Dispatcher::RecurringTask
+    class << self
+      def wrap(args)
+        args.is_a?(self) ? args : from_configuration(args.first, **args.second)
+      end
+
+      def from_configuration(key, **options)
+        new(key, class_name: options[:class], schedule: options[:schedule], arguments: options[:args])
+      end
+    end
+
+    attr_reader :key, :schedule, :class_name, :arguments
+
+    def initialize(key, class_name:, schedule:, arguments: nil)
+      @key = key
+      @class_name = class_name
+      @schedule = schedule
+      @arguments = Array(arguments)
+    end
+
+    def delay_from_now
+      [ (next_time - Time.current).to_f, 0 ].max
+    end
+
+    def next_time
+      parsed_schedule.next_time.utc
+    end
+
+    def enqueue(at:)
+      if using_solid_queue_adapter?
+        perform_later_and_record(run_at: at)
+      else
+        perform_later
+      end
+    end
+
+    def valid?
+      parsed_schedule.instance_of?(Fugit::Cron)
+    end
+
+    def to_s
+      "#{class_name}.perform_later(#{arguments.map(&:inspect).join(",")}) [ #{parsed_schedule.original.to_s} ]"
+    end
+
+    def to_h
+      {
+        schedule: schedule,
+        class_name: class_name,
+        arguments: arguments
+      }
+    end
+
+    private
+      def using_solid_queue_adapter?
+        job_class.queue_adapter_name.inquiry.solid_queue?
+      end
+
+      def perform_later_and_record(run_at:)
+        RecurringExecution.record(key, run_at) { perform_later.provider_job_id }
+      end
+
+      def perform_later
+        job_class.perform_later(*arguments_with_kwargs)
+      end
+
+      def arguments_with_kwargs
+        if arguments.last.is_a?(Hash)
+          arguments[0...-1] + [ Hash.ruby2_keywords_hash(arguments.last) ]
+        else
+          arguments
+        end
+      end
+
+      def parsed_schedule
+        @parsed_schedule ||= Fugit.parse(schedule)
+      end
+
+      def job_class
+        @job_class ||= class_name.safe_constantize
+      end
+  end
+end

data/lib/solid_queue/dispatcher.rb

@@ -2,72 +2,57 @@
 
 module SolidQueue
   class Dispatcher < Processes::Base
-    include Processes::Runnable, Processes::Poller
+    include Processes::Poller
 
-    attr_accessor :batch_size, :concurrency_maintenance_interval
+    attr_accessor :batch_size, :concurrency_maintenance, :recurring_schedule
 
-    set_callback :boot, :after, :launch_concurrency_maintenance
-    set_callback :shutdown, :before, :stop_concurrency_maintenance
+    after_boot :start_concurrency_maintenance, :load_recurring_schedule
+    before_shutdown :stop_concurrency_maintenance, :unload_recurring_schedule
 
     def initialize(**options)
       options = options.dup.with_defaults(SolidQueue::Configuration::DISPATCHER_DEFAULTS)
 
       @batch_size = options[:batch_size]
       @polling_interval = options[:polling_interval]
-      @concurrency_maintenance_interval = options[:concurrency_maintenance_interval]
+
+      @concurrency_maintenance = ConcurrencyMaintenance.new(options[:concurrency_maintenance_interval], options[:batch_size]) if options[:concurrency_maintenance]
+      @recurring_schedule = RecurringSchedule.new(options[:recurring_tasks])
     end
 
     private
-      def run
+      def poll
         batch = dispatch_next_batch
-
-        unless batch.size > 0
-          procline "waiting"
-          interruptible_sleep(polling_interval)
-        end
+        batch.size
       end
 
       def dispatch_next_batch
         with_polling_volume do
-          SolidQueue::ScheduledExecution.dispatch_next_batch(batch_size)
+          ScheduledExecution.dispatch_next_batch(batch_size)
         end
       end
 
-      def launch_concurrency_maintenance
-        @concurrency_maintenance_task = Concurrent::TimerTask.new(run_now: true, execution_interval: concurrency_maintenance_interval) do
-          expire_semaphores
-          unblock_blocked_executions
-        end
-
-        @concurrency_maintenance_task.add_observer do |_, _, error|
-          handle_thread_error(error) if error
-        end
-
-        @concurrency_maintenance_task.execute
+      def start_concurrency_maintenance
+        concurrency_maintenance&.start
       end
 
-      def stop_concurrency_maintenance
-        @concurrency_maintenance_task.shutdown
+      def load_recurring_schedule
+        recurring_schedule.load_tasks
       end
 
-      def expire_semaphores
-        wrap_in_app_executor do
-          Semaphore.expired.in_batches(of: batch_size, &:delete_all)
-        end
+      def stop_concurrency_maintenance
+        concurrency_maintenance&.stop
       end
 
-      def unblock_blocked_executions
-        wrap_in_app_executor do
-          BlockedExecution.unblock(batch_size)
-        end
+      def unload_recurring_schedule
+        recurring_schedule.unload_tasks
       end
 
-      def initial_jitter
-        Kernel.rand(0...polling_interval)
+      def set_procline
+        procline "waiting"
       end
 
       def metadata
-        super.merge(batch_size: batch_size)
+        super.merge(batch_size: batch_size, concurrency_maintenance_interval: concurrency_maintenance&.interval, recurring_schedule: recurring_schedule.tasks.presence )
       end
   end
 end

data/lib/solid_queue/processes/base.rb

@@ -3,25 +3,8 @@
 module SolidQueue
   module Processes
     class Base
-      include ActiveSupport::Callbacks
-      define_callbacks :boot, :shutdown
-
+      include Callbacks # Defines callbacks needed by other concerns
       include AppExecutor, Registrable, Interruptible, Procline
-
-      private
-        def observe_initial_delay
-          interruptible_sleep(initial_jitter)
-        end
-
-        def boot
-        end
-
-        def shutdown
-        end
-
-        def initial_jitter
-          0
-        end
     end
   end
 end

data/lib/solid_queue/processes/callbacks.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module SolidQueue::Processes
+  module Callbacks
+    extend ActiveSupport::Concern
+
+    included do
+      extend ActiveModel::Callbacks
+      define_model_callbacks :boot, :shutdown
+    end
+
+    private
+      def boot
+      end
+
+      def shutdown
+      end
+  end
+end

data/lib/solid_queue/processes/poller.rb

@@ -4,11 +4,39 @@ module SolidQueue::Processes
   module Poller
     extend ActiveSupport::Concern
 
+    include Runnable
+
     included do
       attr_accessor :polling_interval
     end
 
     private
+      def run
+        if mode.async?
+          @thread = Thread.new { start_loop }
+        else
+          start_loop
+        end
+      end
+
+      def start_loop
+        loop do
+          break if shutting_down?
+
+          wrap_in_app_executor do
+            unless poll > 0
+              interruptible_sleep(polling_interval)
+            end
+          end
+        end
+      ensure
+        run_callbacks(:shutdown) { shutdown }
+      end
+
+      def poll
+        raise NotImplementedError
+      end
+
       def with_polling_volume
         if SolidQueue.silence_polling?
           ActiveRecord::Base.logger.silence { yield }

data/lib/solid_queue/processes/registrable.rb

@@ -5,11 +5,10 @@ module SolidQueue::Processes
     extend ActiveSupport::Concern
 
     included do
-      set_callback :boot, :after, :register
-      set_callback :boot, :after, :launch_heartbeat
+      after_boot :register, :launch_heartbeat
 
-      set_callback :shutdown, :before, :stop_heartbeat
-      set_callback :shutdown, :after, :deregister
+      before_shutdown :stop_heartbeat
+      after_shutdown :deregister
     end
 
     def inspect
@@ -26,7 +25,7 @@ module SolidQueue::Processes
           pid: process_pid,
           hostname: hostname,
           supervisor: try(:supervisor),
-          metadata: metadata
+          metadata: metadata.compact
       end
 
       def deregister

data/lib/solid_queue/processes/runnable.rb

@@ -8,11 +8,9 @@ module SolidQueue::Processes
 
     def start
       @stopping = false
-
-      observe_initial_delay
       run_callbacks(:boot) { boot }
 
-      start_loop
+      run
     end
 
     def stop
@@ -20,60 +18,47 @@ module SolidQueue::Processes
       @thread&.join
     end
 
-  private
-    DEFAULT_MODE = :async
+    private
+      DEFAULT_MODE = :async
 
-    def mode
-      (@mode || DEFAULT_MODE).to_s.inquiry
-    end
+      def mode
+        (@mode || DEFAULT_MODE).to_s.inquiry
+      end
 
-    def boot
-      register_signal_handlers if supervised?
-      SolidQueue.logger.info("[SolidQueue] Starting #{self}")
-    end
+      def boot
+        if supervised?
+          register_signal_handlers
+          set_procline
+        end
 
-    def start_loop
-      if mode.async?
-        @thread = Thread.new { do_start_loop }
-      else
-        do_start_loop
+        SolidQueue.logger.info("[SolidQueue] Starting #{self}")
       end
-    end
 
-    def do_start_loop
-      loop do
-        break if shutting_down?
-
-        wrap_in_app_executor do
-          run
-        end
+      def shutting_down?
+        stopping? || supervisor_went_away? || finished?
       end
-    ensure
-      run_callbacks(:shutdown) { shutdown }
-    end
 
-    def shutting_down?
-      stopping? || supervisor_went_away? || finished?
-    end
+      def run
+        raise NotImplementedError
+      end
 
-    def run
-      raise NotImplementedError
-    end
+      def stopping?
+        @stopping
+      end
 
-    def stopping?
-      @stopping
-    end
+      def finished?
+        running_inline? && all_work_completed?
+      end
 
-    def finished?
-      running_inline? && all_work_completed?
-    end
+      def all_work_completed?
+        false
+      end
 
-    def all_work_completed?
-      false
-    end
+      def set_procline
+      end
 
-    def running_inline?
-      mode.inline?
-    end
+      def running_inline?
+        mode.inline?
+      end
   end
 end

data/lib/solid_queue/processes/supervised.rb

@@ -14,6 +14,10 @@ module SolidQueue::Processes
     end
 
     private
+      def set_procline
+        procline "waiting"
+      end
+
       def supervisor_went_away?
         supervised? && supervisor&.pid != ::Process.ppid
       end

data/lib/solid_queue/recurring_tasks/manager.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module RecurringTasks
+    class Manager < Processes::Base
+      include Processes::Runnable
+
+      attr_accessor :schedule
+
+      after_boot :load_schedule
+      before_shutdown :unload_schedule
+
+      def initialize(tasks)
+        @schedule = Schedule.new(tasks)
+      end
+
+      private
+        def load_schedule
+          schedule.load_tasks
+        end
+
+        def unload_schedule
+          schedule.unload_tasks
+        end
+
+        def metadata
+          super.merge(schedule: schedule.tasks)
+        end
+    end
+  end
+end

data/lib/solid_queue/recurring_tasks/schedule.rb

@@ -0,0 +1,58 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  module RecurringTasks
+    class Schedule
+      include AppExecutor
+
+      attr_reader :configured_tasks, :scheduled_tasks
+
+      def initialize(tasks)
+        @configured_tasks = Array(tasks).map { |task| Task.wrap(task) }
+        @scheduled_tasks = Concurrent::Hash.new
+      end
+
+      def load_tasks
+        configured_tasks.each do |task|
+          load_task(task)
+        end
+      end
+
+      def load_task(task)
+        scheduled_tasks[task.key] = schedule(task)
+      end
+
+      def unload_tasks
+        scheduled_tasks.values.each(&:cancel)
+        scheduled_tasks.clear
+      end
+
+      def tasks
+        configured_tasks.each_with_object({}) { |task, hsh| hsh[task.key] = task.to_h }
+      end
+
+      def inspect
+        configured_tasks.map(&:to_s).join(" | ")
+      end
+
+      private
+        def schedule(task)
+          scheduled_task = Concurrent::ScheduledTask.new(task.delay_from_now, args: [ self, task, task.next_time ]) do |thread_schedule, thread_task, thread_task_run_at|
+            thread_schedule.load_task(thread_task)
+
+            wrap_in_app_executor do
+              thread_task.enqueue(at: thread_task_run_at)
+            end
+          end
+
+          scheduled_task.add_observer do |_, _, error|
+            # Don't notify on task cancellation before execution, as this will happen normally
+            # as part of unloading tasks
+            handle_thread_error(error) if error && !error.is_a?(Concurrent::CancelledOperationError)
+          end
+
+          scheduled_task.tap(&:execute)
+        end
+    end
+  end
+end

data/lib/solid_queue/recurring_tasks/task.rb

@@ -0,0 +1,87 @@
+require "fugit"
+
+module SolidQueue
+  module RecurringTasks
+    class Task
+      class << self
+        def wrap(args)
+          args.is_a?(self) ? args : from_configuration(args.first, **args.second)
+        end
+
+        def from_configuration(key, **options)
+          new(key, class_name: options[:class], schedule: options[:schedule], arguments: options[:args])
+        end
+      end
+
+      attr_reader :key, :schedule, :class_name, :arguments
+
+      def initialize(key, class_name:, schedule:, arguments: nil)
+        @key = key
+        @class_name = class_name
+        @schedule = schedule
+        @arguments = Array(arguments)
+      end
+
+      def delay_from_now
+        [ (next_time - Time.current).to_f, 0 ].max
+      end
+
+      def next_time
+        parsed_schedule.next_time.utc
+      end
+
+      def enqueue(at:)
+        if using_solid_queue_adapter?
+          perform_later_and_record(run_at: at)
+        else
+          perform_later
+        end
+      end
+
+      def valid?
+        parsed_schedule.instance_of?(Fugit::Cron)
+      end
+
+      def to_s
+        "#{class_name}.perform_later(#{arguments.map(&:inspect).join(",")}) [ #{parsed_schedule.original.to_s} ]"
+      end
+
+      def to_h
+        {
+          schedule: schedule,
+          class_name: class_name,
+          arguments: arguments
+        }
+      end
+
+      private
+        def using_solid_queue_adapter?
+          job_class.queue_adapter_name.inquiry.solid_queue?
+        end
+
+        def perform_later_and_record(run_at:)
+          RecurringExecution.record(key, run_at) { perform_later.provider_job_id }
+        end
+
+        def perform_later
+          job_class.perform_later(*arguments_with_kwargs)
+        end
+
+        def arguments_with_kwargs
+          if arguments.last.is_a?(Hash)
+            arguments[0...-1] + [ Hash.ruby2_keywords_hash(arguments.last) ]
+          else
+            arguments
+          end
+        end
+
+        def parsed_schedule
+          @parsed_schedule ||= Fugit.parse(schedule)
+        end
+
+        def job_class
+          @job_class ||= class_name.safe_constantize
+        end
+    end
+  end
+end

data/lib/solid_queue/supervisor.rb

@@ -4,7 +4,7 @@ module SolidQueue
   class Supervisor < Processes::Base
     include Processes::Signals
 
-    set_callback :boot, :after, :launch_process_prune
+    after_boot :launch_process_prune
 
     class << self
       def start(mode: :work, load_configuration_from: nil)

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.2.2"
+  VERSION = "0.3.0"
 end

data/lib/solid_queue/worker.rb

@@ -2,7 +2,7 @@
 
 module SolidQueue
   class Worker < Processes::Base
-    include Processes::Runnable, Processes::Poller
+    include Processes::Poller
 
     attr_accessor :queues, :pool
 
@@ -15,22 +15,17 @@ module SolidQueue
     end
 
     private
-      def run
-        polled_executions = poll
-
-        if polled_executions.size > 0
-          procline "performing #{polled_executions.count} jobs"
-
-          polled_executions.each do |execution|
+      def poll
+        claim_executions.then do |executions|
+          executions.each do |execution|
             pool.post(execution)
           end
-        else
-          procline "waiting for jobs in #{queues.join(",")}"
-          interruptible_sleep(polling_interval)
+
+          executions.size
         end
       end
 
-      def poll
+      def claim_executions
         with_polling_volume do
           SolidQueue::ReadyExecution.claim(queues, pool.idle_threads, process.id)
         end
@@ -47,6 +42,10 @@ module SolidQueue
         SolidQueue::ReadyExecution.aggregated_count_across(queues).zero?
       end
 
+      def set_procline
+        procline "waiting for jobs in #{queues.join(",")}"
+      end
+
       def metadata
         super.merge(queues: queues.join(","), thread_pool_size: pool.size)
       end

data/lib/solid_queue.rb

@@ -3,24 +3,16 @@
 require "solid_queue/version"
 require "solid_queue/engine"
 
-require "active_job/queue_adapters/solid_queue_adapter"
-require "active_job/concurrency_controls"
-
-require "solid_queue/app_executor"
-require "solid_queue/processes/supervised"
-require "solid_queue/processes/registrable"
-require "solid_queue/processes/interruptible"
-require "solid_queue/processes/pidfile"
-require "solid_queue/processes/procline"
-require "solid_queue/processes/poller"
-require "solid_queue/processes/base"
-require "solid_queue/processes/runnable"
-require "solid_queue/processes/signals"
-require "solid_queue/configuration"
-require "solid_queue/pool"
-require "solid_queue/worker"
-require "solid_queue/dispatcher"
-require "solid_queue/supervisor"
+require "active_job"
+require "active_job/queue_adapters"
+
+require "zeitwerk"
+
+loader = Zeitwerk::Loader.for_gem(warn_on_extra_files: false)
+loader.ignore("#{__dir__}/solid_queue/tasks.rb")
+loader.ignore("#{__dir__}/generators")
+loader.ignore("#{__dir__}/puma")
+loader.setup
 
 module SolidQueue
   mattr_accessor :logger, default: ActiveSupport::Logger.new($stdout)
@@ -42,11 +34,17 @@ module SolidQueue
   mattr_accessor :clear_finished_jobs_after, default: 1.day
   mattr_accessor :default_concurrency_control_period, default: 3.minutes
 
-  def self.supervisor?
-    supervisor
-  end
+  class << self
+    def supervisor?
+      supervisor
+    end
+
+    def silence_polling?
+      silence_polling
+    end
 
-  def self.silence_polling?
-    silence_polling
+    def preserve_finished_jobs?
+      preserve_finished_jobs
+    end
   end
 end

metadata

@@ -1,29 +1,85 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.3.0
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-03-05 00:00:00.000000000 Z
+date: 2024-03-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: rails
+  name: activerecord
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: '7.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+- !ruby/object:Gem::Dependency
+  name: activejob
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+- !ruby/object:Gem::Dependency
+  name: railties
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
       - !ruby/object:Gem::Version
         version: '7.1'
+- !ruby/object:Gem::Dependency
+  name: concurrent-ruby
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.2.2
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.2.2
+- !ruby/object:Gem::Dependency
+  name: fugit
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.9.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 1.9.0
 - !ruby/object:Gem::Dependency
   name: debug
   requirement: !ruby/object:Gem::Requirement
@@ -66,6 +122,48 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: mysql2
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Database-backed Active Job backend.
 email:
 - rosa@37signals.com
@@ -86,6 +184,7 @@ files:
 - app/models/solid_queue/job/clearable.rb
 - app/models/solid_queue/job/concurrency_controls.rb
 - app/models/solid_queue/job/executable.rb
+- app/models/solid_queue/job/recurrable.rb
 - app/models/solid_queue/job/schedulable.rb
 - app/models/solid_queue/pause.rb
 - app/models/solid_queue/process.rb
@@ -94,11 +193,13 @@ files:
 - app/models/solid_queue/queue_selector.rb
 - app/models/solid_queue/ready_execution.rb
 - app/models/solid_queue/record.rb
+- app/models/solid_queue/recurring_execution.rb
 - app/models/solid_queue/scheduled_execution.rb
 - app/models/solid_queue/semaphore.rb
 - config/routes.rb
 - db/migrate/20231211200639_create_solid_queue_tables.rb
 - db/migrate/20240110143450_add_missing_index_to_blocked_executions.rb
+- db/migrate/20240218110712_create_recurring_executions.rb
 - lib/active_job/concurrency_controls.rb
 - lib/active_job/queue_adapters/solid_queue_adapter.rb
 - lib/generators/solid_queue/install/USAGE
@@ -109,9 +210,13 @@ files:
 - lib/solid_queue/app_executor.rb
 - lib/solid_queue/configuration.rb
 - lib/solid_queue/dispatcher.rb
+- lib/solid_queue/dispatcher/concurrency_maintenance.rb
+- lib/solid_queue/dispatcher/recurring_schedule.rb
+- lib/solid_queue/dispatcher/recurring_task.rb
 - lib/solid_queue/engine.rb
 - lib/solid_queue/pool.rb
 - lib/solid_queue/processes/base.rb
+- lib/solid_queue/processes/callbacks.rb
 - lib/solid_queue/processes/interruptible.rb
 - lib/solid_queue/processes/pidfile.rb
 - lib/solid_queue/processes/poller.rb
@@ -120,6 +225,9 @@ files:
 - lib/solid_queue/processes/runnable.rb
 - lib/solid_queue/processes/signals.rb
 - lib/solid_queue/processes/supervised.rb
+- lib/solid_queue/recurring_tasks/manager.rb
+- lib/solid_queue/recurring_tasks/schedule.rb
+- lib/solid_queue/recurring_tasks/task.rb
 - lib/solid_queue/supervisor.rb
 - lib/solid_queue/tasks.rb
 - lib/solid_queue/version.rb