solid_queue 0.8.2 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b975d1366456d8f53686e0cee4f14e35dfc5461b206de7553818a109f6ae0088
-  data.tar.gz: 3d1084ecd73a3cd5c557d89b9edf36bac7db5c1f9ce2e995811c31370c261b6b
+  metadata.gz: b0a582a0c2b6b0025baed737e35b41dbe2dc687b0fbd7a221909b6350321a717
+  data.tar.gz: 85053c1ab6f8b7d5b66855631a003e789306499eba71d100ae91a32e39757b97
 SHA512:
-  metadata.gz: e06a86af9d282440726a2dfd81d03637d093e7a12fca7f43be3a1f28312d9117c90ce778a7c04923eb5a57feb2909721d1da9c32f0d5b640dc3c09cf3a3c98db
-  data.tar.gz: 888f76de27a21e99cc64835b005a140b555b812a06f32b7c7588c5b423e8f84301760c653bcfa2a4020def4d6aa3f8c5464684277bf26833355e4cc82cf684d5
+  metadata.gz: 630b5c980be4b1e152544e46eb38922706b620765ffb1b9c18ff81a9c00a221be2d0ed2f723ecd7d39391a01119785b3be42db408f7e48a6b0afd67ab62088eb
+  data.tar.gz: 4451b96428dfbaa8213d9fff225dfeaa5826aa10d6ab68151effbba01df866c8c19110e4c6f147a25b70fa34827c6a6c03381b1fb7681934b036791ba4f16faa

data/README.md

@@ -2,9 +2,9 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`).
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, recurring jobs, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`).
 
-Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
+Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails's multi-threading.
 
 ## Installation
 
@@ -13,9 +13,9 @@ Solid Queue is configured by default in new Rails 8 applications. But if you're
 1. `bundle add solid_queue`
 2. `bin/rails solid_queue:install`
 
-This will configure Solid Queue as the production Active Job backend, create `config/solid_queue.yml`, and create the `db/queue_schema.rb`.
+This will configure Solid Queue as the production Active Job backend, create the configuration files `config/queue.yml` and `config/recurring.yml`, and create the `db/queue_schema.rb`. It'll also create a `bin/jobs` executable wrapper that you can use to start Solid Queue.
 
-You will then have to add the configuration for the queue database in `config/database.yml`. If you're using sqlite, it'll look like this:
+Once you've done that, you will then have to add the configuration for the queue database in `config/database.yml`. If you're using sqlite, it'll look like this:
 
 ```yaml
 production:
@@ -43,12 +43,24 @@ production:
     migrations_paths: db/queue_migrate
 ```
 
+Note: Calling `bin/rails solid_queue:install` will automatically add `config.solid_queue.connects_to = { database: { writing: :queue } }` to `config/environments/production.rb`, so no additional configuration is needed there (although you must make sure that you use the `queue` name in `database.yml` for this to match!). But if you want to use Solid Queue in a different environment (like staging or even development), you'll have to manually add that `config.solid_queue.connects_to` line to the respective environment file. And, as always, make sure that the name you're using for the database in `config/database.yml` matches the name you use in `config.solid_queue.connects_to`.
+
 Then run `db:prepare` in production to ensure the database is created and the schema is loaded.
 
 Now you're ready to start processing jobs by running `bin/jobs` on the server that's doing the work. This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
 For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
 
+### Single database configuration
+
+It's also possibile to use one single database for both production data:
+
+1. Copy the contents of `db/queue_schema.rb` into a normal migration and delete `db/queue_schema.rb`
+2. Remove `config.solid_queue.connects_to` from `production.rb`
+3. Migrate your database. You are ready to run `bin/jobs`
+
+You won't have multiple databases, so `database.yml` doesn't need to have primary and queue database.
+
 ## Incremental adoption
 
 If you're planning to adopt Solid Queue incrementally by switching one job at the time, you can do so by leaving the `config.active_job.queue_adapter` set to your old backend, and then set the `queue_adapter` directly in the jobs you're moving:
@@ -61,22 +73,31 @@ class MyJob < ApplicationJob
   # ...
 end
 ```
+
 ## High performance requirements
 
 Solid Queue was designed for the highest throughput when used with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue. You can also use it with SQLite on smaller applications.
 
 ## Configuration
 
-### Workers and dispatchers
+### Workers, dispatchers and scheduler
 
-We have three types of actors in Solid Queue:
+We have several types of actors in Solid Queue:
 - _Workers_ are in charge of picking jobs ready to run from queues and processing them. They work off the `solid_queue_ready_executions` table.
-- _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They're also in charge of managing [recurring tasks](#recurring-tasks), dispatching jobs to process them according to their schedule. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
+- _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
+- The _scheduler_ manages [recurring tasks](#recurring-tasks), enqueuing jobs for them when they're due.
 - The _supervisor_ runs workers and dispatchers according to the configuration, controls their heartbeats, and stops and starts them when needed.
 
-Solid Queue's supervisor will fork a separate process for each supervised worker/dispatcher.
+Solid Queue's supervisor will fork a separate process for each supervised worker/dispatcher/scheduler.
 
-By default, Solid Queue will try to find your configuration under `config/solid_queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG`. This is what this configuration looks like:
+By default, Solid Queue will try to find your configuration under `config/queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG` or by using the `-c/--config_file` option with `bin/jobs`, like this:
+
+```
+bin/jobs -c config/calendar.yml
+```
+
+
+This is what this configuration looks like:
 
 ```yml
 production:
@@ -105,6 +126,7 @@ production:
 ```
 the supervisor will run 1 dispatcher and no workers.
 
+
 Here's an overview of the different options:
 
 - `polling_interval`: the time interval in seconds that workers and dispatchers will wait before checking for more jobs. This time defaults to `1` second for dispatchers and `0.1` seconds for workers.
@@ -127,7 +149,7 @@ Here's an overview of the different options:
 - `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
 - `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
-- `recurring_tasks`: a list of recurring tasks the dispatcher will manage. Read more details about this one in the [Recurring tasks](#recurring-tasks) section.
+
 
 ### Queue order and priorities
 
@@ -234,6 +256,9 @@ class MyJob < ApplicationJob
 
 When a job includes these controls, we'll ensure that, at most, the number of jobs (indicated as `to`) that yield the same `key` will be performed concurrently, and this guarantee will last for `duration` for each job enqueued. Note that there's no guarantee about _the order of execution_, only about jobs being performed at the same time (overlapping).
 
+The concurrency limits use the concept of semaphores when enqueuing, and work as follows: when a job is enqueued, we check if it specifies concurrency controls. If it does, we check the semaphore for the computed concurrency key. If the semaphore is open, we claim it and we set the job as _ready_. Ready means it can be picked up by workers for execution. When the job finishes executing (be it successfully or unsuccesfully, resulting in a failed execution), we signal the semaphore and try to unblock the next job with the same key, if any. Unblocking the next job doesn't mean running that job right away, but moving it from _blocked_ to _ready_. Since something can happen that prevents the first job from releasing the semaphore and unblocking the next job (for example, someone pulling a plug in the machine where the worker is running), we have the `duration` as a failsafe. Jobs that have been blocked for more than duration are candidates to be released, but only as many of them as the concurrency rules allow, as each one would need to go through the semaphore dance check. This means that the `duration` is not really about the job that's enqueued or being run, it's about the jobs that are blocked waiting.
+
+
 For example:
 ```ruby
 class DeliverAnnouncementToContactJob < ApplicationJob
@@ -242,7 +267,7 @@ class DeliverAnnouncementToContactJob < ApplicationJob
   def perform(contact)
     # ...
 ```
-Where `contact` and `account` are `ActiveRecord` records. In this case, we'll ensure that at most two jobs of the kind `DeliverAnnouncementToContact` for the same account will run concurrently. If, for any reason, one of those jobs takes longer than 5 minutes or doesn't release its concurrency lock within 5 minutes of acquiring it, a new job with the same key might gain the lock.
+Where `contact` and `account` are `ActiveRecord` records. In this case, we'll ensure that at most two jobs of the kind `DeliverAnnouncementToContact` for the same account will run concurrently. If, for any reason, one of those jobs takes longer than 5 minutes or doesn't release its concurrency lock (signals the semaphore) within 5 minutes of acquiring it, a new job with the same key might gain the lock.
 
 Let's see another example using `group`:
 
@@ -268,7 +293,7 @@ Note that the `duration` setting depends indirectly on the value for `concurrenc
 
 Jobs are unblocked in order of priority but queue order is not taken into account for unblocking jobs. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.
 
-Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for gaining the lock, and whenever they get it, they'll be run. It doesn't matter if they had gained the lock already in the past.
+Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for getting an open semaphore, and whenever they get it, they'll be run. It doesn't matter if they had already gotten an open semaphore in the past.
 
 ## Failed jobs and retries
 
@@ -293,27 +318,42 @@ to your `puma.rb` configuration.
 
 ## Recurring tasks
 
-Solid Queue supports defining recurring tasks that run at specific times in the future, on a regular basis like cron jobs. These are managed by dispatcher processes and as such, they can be defined in the dispatcher's configuration like this:
+Solid Queue supports defining recurring tasks that run at specific times in the future, on a regular basis like cron jobs. These are managed by the scheduler process and are defined in their own configuration file. By default, the file is located in `config/recurring.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_RECURRING_SCHEDULE` or by using the `--recurring_schedule_file` option with `bin/jobs`, like this:
+
+```
+bin/jobs --recurring_schedule_file=config/schedule.yml
+```
+
+The configuration itself looks like this:
+
 ```yml
-  dispatchers:
-    - polling_interval: 1
-      batch_size: 500
-      recurring_tasks:
-        my_periodic_job:
-          class: MyJob
-          args: [ 42, { status: "custom_status" } ]
-          schedule: every second
+a_periodic_job:
+  class: MyJob
+  args: [ 42, { status: "custom_status" } ]
+  schedule: every second
+a_cleanup_task:
+  command: "DeletedStuff.clear_all"
+  schedule: every day at 9am
 ```
-`recurring_tasks` is a hash/dictionary, and the key will be the task key internally. Each task needs to have a class, which will be the job class to enqueue, and a schedule. The schedule is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). You can also provide arguments to be passed to the job, as a single argument, a hash, or an array of arguments that can also include kwargs as the last element in the array.
+
+Tasks are specified as a hash/dictionary, where the key will be the task's key internally. Each task needs to either have a `class`, which will be the job class to enqueue, or a `command`, which will be eval'ed in the context of a job (`SolidQueue::RecurringJob`) that will be enqueued according to its schedule, in the `solid_queue_recurring` queue.
+
+Each task needs to have also a schedule, which is parsed using [Fugit](https://github.com/floraison/fugit), so it accepts anything [that Fugit accepts as a cron](https://github.com/floraison/fugit?tab=readme-ov-file#fugitcron). You can optionally supply the following for each task:
+- `args`: the arguments to be passed to the job, as a single argument, a hash, or an array of arguments that can also include kwargs as the last element in the array.
 
 The job in the example configuration above will be enqueued every second as:
 ```ruby
 MyJob.perform_later(42, status: "custom_status")
 ```
 
-Tasks are enqueued at their corresponding times by the dispatcher that owns them, and each task schedules the next one. This is pretty much [inspired by what GoodJob does](https://github.com/bensheldon/good_job/blob/994ecff5323bf0337e10464841128fda100750e6/lib/good_job/cron_manager.rb).
+- `queue`: a different queue to be used when enqueuing the job. If none, the queue set up for the job class.
+
+- `priority`: a numeric priority value to be used when enqueuing the job.
+
 
-It's possible to run multiple dispatchers with the same `recurring_tasks` configuration. To avoid enqueuing duplicate tasks at the same time, an entry in a new `solid_queue_recurring_executions` table is created in the same transaction as the job is enqueued. This table has a unique index on `task_key` and `run_at`, ensuring only one entry per task per time will be created. This only works if you have `preserve_finished_jobs` set to `true` (the default), and the guarantee applies as long as you keep the jobs around.
+Tasks are enqueued at their corresponding times by the scheduler, and each task schedules the next one. This is pretty much [inspired by what GoodJob does](https://github.com/bensheldon/good_job/blob/994ecff5323bf0337e10464841128fda100750e6/lib/good_job/cron_manager.rb).
+
+It's possible to run multiple schedulers with the same `recurring_tasks` configuration, for example, if you have multiple servers for redundancy, and you run the `scheduler` in more than one of them. To avoid enqueuing duplicate tasks at the same time, an entry in a new `solid_queue_recurring_executions` table is created in the same transaction as the job is enqueued. This table has a unique index on `task_key` and `run_at`, ensuring only one entry per task per time will be created. This only works if you have `preserve_finished_jobs` set to `true` (the default), and the guarantee applies as long as you keep the jobs around.
 
 Finally, it's possible to configure jobs that aren't handled by Solid Queue. That is, you can have a job like this in your app:
 ```ruby
@@ -328,13 +368,12 @@ end
 
 You can still configure this in Solid Queue:
 ```yml
-  dispatchers:
-    - recurring_tasks:
-        my_periodic_resque_job:
-          class: MyResqueJob
-          args: 22
-          schedule: "*/5 * * * *"
+my_periodic_resque_job:
+  class: MyResqueJob
+  args: 22
+  schedule: "*/5 * * * *"
 ```
+
 and the job will be enqueued via `perform_later` so it'll run in Resque. However, in this case we won't track any `solid_queue_recurring_execution` record for it and there won't be any guarantees that the job is enqueued only once each time.
 
 ## Inspiration

data/Rakefile

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require "bundler/setup"
 
 APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
@@ -6,3 +8,14 @@ load "rails/tasks/engine.rake"
 load "rails/tasks/statistics.rake"
 
 require "bundler/gem_tasks"
+
+def databases
+  %w[ mysql postgres sqlite ]
+end
+
+task :test do
+  databases.each do |database|
+    sh("TARGET_DB=#{database} bin/setup")
+    sh("TARGET_DB=#{database} bin/rails test")
+  end
+end

data/UPGRADING.md

@@ -1,3 +1,8 @@
+# Upgrading to version 0.9.x
+This version has two breaking changes regarding configuration:
+- The default configuration file has changed from `config/solid_queue.yml` to `config/queue.yml`.
+- Recurring tasks are now defined in `config/recurring.yml` (by default). Before, they would be defined as part of the _dispatcher_ configuration. Now they've been upgraded to their own configuration file, and a dedicated process (the _scheduler_) to manage them. Check the _Recurring tasks_ section in the `README` to learn how to configure them in detail. They still follow the same format as before when they lived under `dispatchers > recurring_tasks`.
+
 # Upgrading to version 0.8.x
 *IMPORTANT*: This version collapsed all migrations into a single `db/queue_schema.rb`, that will use a separate `queue` database. If you're upgrading from a version < 0.6.0, you need to upgrade to 0.6.0 first, ensure all migrations are up-to-date, and then upgrade further.
 

data/app/jobs/solid_queue/recurring_job.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+class SolidQueue::RecurringJob < ActiveJob::Base
+  queue_as :solid_queue_recurring
+
+  def perform(command)
+    eval(command)
+  end
+end

data/app/models/solid_queue/process/prunable.rb

@@ -10,9 +10,9 @@ module SolidQueue
       end
 
       class_methods do
-        def prune
+        def prune(excluding: nil)
           SolidQueue.instrument :prune_processes, size: 0 do |payload|
-            prunable.non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
+            prunable.excluding(excluding).non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
               payload[:size] += batch.size
 
               batch.each(&:prune)

data/app/models/solid_queue/recurring_task.rb

@@ -7,17 +7,30 @@ module SolidQueue
     serialize :arguments, coder: Arguments, default: []
 
     validate :supported_schedule
+    validate :ensure_command_or_class_present
     validate :existing_job_class
 
     scope :static, -> { where(static: true) }
 
+    mattr_accessor :default_job_class
+    self.default_job_class = RecurringJob
+
     class << self
       def wrap(args)
         args.is_a?(self) ? args : from_configuration(args.first, **args.second)
       end
 
       def from_configuration(key, **options)
-        new(key: key, class_name: options[:class], schedule: options[:schedule], arguments: options[:args])
+        new \
+          key: key,
+          class_name: options[:class],
+          command: options[:command],
+          arguments: options[:args],
+          schedule: options[:schedule],
+          queue_name: options[:queue].presence,
+          priority: options[:priority].presence,
+          description: options[:description],
+          static: true
       end
 
       def create_or_update_all(tasks)
@@ -47,7 +60,7 @@ module SolidQueue
         else
           payload[:other_adapter] = true
 
-          perform_later do |job|
+          perform_later.tap do |job|
             unless job.successfully_enqueued?
               payload[:enqueue_error] = job.enqueue_error&.message
             end
@@ -77,8 +90,14 @@ module SolidQueue
         end
       end
 
+      def ensure_command_or_class_present
+        unless command.present? || class_name.present?
+          errors.add :base, :command_and_class_blank, message: "either command or class_name must be present"
+        end
+      end
+
       def existing_job_class
-        unless job_class.present?
+        if class_name.present? && job_class.nil?
           errors.add :class_name, :undefined, message: "doesn't correspond to an existing class"
         end
       end
@@ -89,7 +108,7 @@ module SolidQueue
 
       def enqueue_and_record(run_at:)
         RecurringExecution.record(key, run_at) do
-          job_class.new(*arguments_with_kwargs).tap do |active_job|
+          job_class.new(*arguments_with_kwargs).set(enqueue_options).tap do |active_job|
             active_job.run_callbacks(:enqueue) do
               Job.enqueue(active_job)
             end
@@ -98,12 +117,16 @@ module SolidQueue
         end
       end
 
-      def perform_later(&block)
-        job_class.perform_later(*arguments_with_kwargs, &block)
+      def perform_later
+        job_class.new(*arguments_with_kwargs).tap do |active_job|
+          active_job.enqueue(enqueue_options)
+        end
       end
 
       def arguments_with_kwargs
-        if arguments.last.is_a?(Hash)
+        if class_name.nil?
+          command
+        elsif arguments.last.is_a?(Hash)
           arguments[0...-1] + [ Hash.ruby2_keywords_hash(arguments.last) ]
         else
           arguments
@@ -116,7 +139,11 @@ module SolidQueue
       end
 
       def job_class
-        @job_class ||= class_name&.safe_constantize
+        @job_class ||= class_name.present? ? class_name.safe_constantize : self.class.default_job_class
+      end
+
+      def enqueue_options
+        { queue: queue_name, priority: priority }.compact
       end
   end
 end

data/lib/generators/solid_queue/install/USAGE

@@ -6,5 +6,6 @@ Example:
 
     This will perform the following:
         Adds solid_queue db schema
+        Adds default configurations
         Replaces Active Job's adapter in environment configuration
         Installs bin/jobs binstub to start the supervisor

data/lib/generators/solid_queue/install/install_generator.rb

@@ -4,7 +4,8 @@ class SolidQueue::InstallGenerator < Rails::Generators::Base
   source_root File.expand_path("templates", __dir__)
 
   def copy_files
-    template "config/solid_queue.yml"
+    template "config/queue.yml"
+    template "config/recurring.yml"
     template "db/queue_schema.rb"
     template "bin/jobs"
     chmod "bin/jobs", 0755 & ~File.umask, verbose: false

data/lib/generators/solid_queue/install/templates/config/recurring.yml

@@ -0,0 +1,9 @@
+# periodic_cleanup:
+#   class: CleanSoftDeletedRecordsJob
+#   queue: background
+#   args: [ 1000, { batch_size: 500 } ]
+#   schedule: every hour
+# periodic_command:
+#   command: "SoftDeletedRecord.due.delete_all"
+#   priority: 2
+#   schedule: at 5am every day

data/lib/solid_queue/cli.rb

@@ -4,7 +4,17 @@ require "thor"
 
 module SolidQueue
   class Cli < Thor
-    class_option :config_file, type: :string, aliases: "-c", default: Configuration::DEFAULT_CONFIG_FILE_PATH, desc: "Path to config file"
+    class_option :config_file, type: :string, aliases: "-c",
+      default: Configuration::DEFAULT_CONFIG_FILE_PATH,
+      desc: "Path to config file",
+      banner: "SOLID_QUEUE_CONFIG"
+
+    class_option :recurring_schedule_file, type: :string,
+      default: Configuration::DEFAULT_RECURRING_SCHEDULE_FILE_PATH,
+      desc: "Path to recurring schedule definition",
+      banner: "SOLID_QUEUE_RECURRING_SCHEDULE"
+
+    class_option :skip_recurring, type: :boolean, default: false
 
     def self.exit_on_failure?
       true
@@ -14,7 +24,7 @@ module SolidQueue
     default_command :start
 
     def start
-      SolidQueue::Supervisor.start(load_configuration_from: options["config_file"])
+      SolidQueue::Supervisor.start(**options.symbolize_keys)
     end
   end
 end

data/lib/solid_queue/configuration.rb

@@ -19,21 +19,21 @@ module SolidQueue
       batch_size: 500,
       polling_interval: 1,
       concurrency_maintenance: true,
-      concurrency_maintenance_interval: 600,
-      recurring_tasks: []
+      concurrency_maintenance_interval: 600
     }
 
-    DEFAULT_CONFIG = {
-      workers: [ WORKER_DEFAULTS ],
-      dispatchers: [ DISPATCHER_DEFAULTS ]
-    }
+    DEFAULT_CONFIG_FILE_PATH = "config/queue.yml"
+    DEFAULT_RECURRING_SCHEDULE_FILE_PATH = "config/recurring.yml"
 
-    def initialize(load_from: nil)
-      @raw_config = config_from(load_from)
+    def initialize(**options)
+      @options = options.with_defaults(default_options)
     end
 
     def configured_processes
-      dispatchers + workers
+      if only_work? then workers
+      else
+        dispatchers + workers + schedulers
+      end
     end
 
     def max_number_of_threads
@@ -42,9 +42,29 @@ module SolidQueue
     end
 
     private
-      attr_reader :raw_config
+      attr_reader :options
+
+      def default_options
+        {
+          config_file: Rails.root.join(ENV["SOLID_QUEUE_CONFIG"] || DEFAULT_CONFIG_FILE_PATH),
+          recurring_schedule_file: Rails.root.join(ENV["SOLID_QUEUE_RECURRING_SCHEDULE"] || DEFAULT_RECURRING_SCHEDULE_FILE_PATH),
+          only_work: false,
+          only_dispatch: false,
+          skip_recurring: false
+        }
+      end
 
-      DEFAULT_CONFIG_FILE_PATH = "config/solid_queue.yml"
+      def only_work?
+        options[:only_work]
+      end
+
+      def only_dispatch?
+        options[:only_dispatch]
+      end
+
+      def skip_recurring_tasks?
+        options[:skip_recurring] || only_work?
+      end
 
       def workers
         workers_options.flat_map do |worker_options|
@@ -55,41 +75,58 @@ module SolidQueue
 
       def dispatchers
         dispatchers_options.map do |dispatcher_options|
-          recurring_tasks = parse_recurring_tasks dispatcher_options[:recurring_tasks]
-          Process.new :dispatcher, dispatcher_options.merge(recurring_tasks: recurring_tasks).with_defaults(DISPATCHER_DEFAULTS)
+          Process.new :dispatcher, dispatcher_options.with_defaults(DISPATCHER_DEFAULTS)
         end
       end
 
-      def config_from(file_or_hash, env: Rails.env)
-        load_config_from(file_or_hash).then do |config|
-          config = config[env.to_sym] ? config[env.to_sym] : config
-          if (config.keys & DEFAULT_CONFIG.keys).any? then config
-          else
-            DEFAULT_CONFIG
-          end
+      def schedulers
+        if !skip_recurring_tasks? && recurring_tasks.any?
+          [ Process.new(:scheduler, recurring_tasks: recurring_tasks) ]
+        else
+          []
         end
       end
 
       def workers_options
-        @workers_options ||= options_from_raw_config(:workers)
+        @workers_options ||= processes_config.fetch(:workers, [])
           .map { |options| options.dup.symbolize_keys }
       end
 
       def dispatchers_options
-        @dispatchers_options ||= options_from_raw_config(:dispatchers)
+        @dispatchers_options ||= processes_config.fetch(:dispatchers, [])
           .map { |options| options.dup.symbolize_keys }
       end
 
-      def options_from_raw_config(key)
-        Array(raw_config[key])
-      end
-
-      def parse_recurring_tasks(tasks)
-        Array(tasks).map do |id, options|
+      def recurring_tasks
+        @recurring_tasks ||= recurring_tasks_config.map do |id, options|
           RecurringTask.from_configuration(id, **options)
         end.select(&:valid?)
       end
 
+      def processes_config
+        @processes_config ||= config_from \
+          options.slice(:workers, :dispatchers).presence || options[:config_file],
+          keys: [ :workers, :dispatchers ],
+          fallback: { workers: [ WORKER_DEFAULTS ], dispatchers: [ DISPATCHER_DEFAULTS ] }
+      end
+
+      def recurring_tasks_config
+        @recurring_tasks ||= config_from options[:recurring_schedule_file]
+      end
+
+
+      def config_from(file_or_hash, keys: [], fallback: {}, env: Rails.env)
+        load_config_from(file_or_hash).then do |config|
+          config = config[env.to_sym] ? config[env.to_sym] : config
+          config = config.slice(*keys) if keys.any? && config.present?
+
+          if config.empty? then fallback
+          else
+            config
+          end
+        end
+      end
+
       def load_config_from(file_or_hash)
         case file_or_hash
         when Hash
@@ -97,29 +134,17 @@ module SolidQueue
         when Pathname, String
           load_config_from_file Pathname.new(file_or_hash)
         when NilClass
-          load_config_from_env_location || load_config_from_default_location
+          {}
         else
           raise "Solid Queue cannot be initialized with #{file_or_hash.inspect}"
         end
       end
 
-      def load_config_from_env_location
-        if ENV["SOLID_QUEUE_CONFIG"].present?
-          load_config_from_file Rails.root.join(ENV["SOLID_QUEUE_CONFIG"])
-        end
-      end
-
-      def load_config_from_default_location
-        Rails.root.join(DEFAULT_CONFIG_FILE_PATH).then do |config_file|
-          config_file.exist? ? load_config_from_file(config_file) : {}
-        end
-      end
-
       def load_config_from_file(file)
         if file.exist?
           ActiveSupport::ConfigurationFile.parse(file).deep_symbolize_keys
         else
-          raise "Configuration file for Solid Queue not found in #{file}"
+          {}
         end
       end
   end

data/lib/solid_queue/dispatcher.rb

@@ -2,10 +2,10 @@
 
 module SolidQueue
   class Dispatcher < Processes::Poller
-    attr_accessor :batch_size, :concurrency_maintenance, :recurring_schedule
+    attr_accessor :batch_size, :concurrency_maintenance
 
-    after_boot :start_concurrency_maintenance, :schedule_recurring_tasks
-    before_shutdown :stop_concurrency_maintenance, :unschedule_recurring_tasks
+    after_boot :start_concurrency_maintenance
+    before_shutdown :stop_concurrency_maintenance
 
     def initialize(**options)
       options = options.dup.with_defaults(SolidQueue::Configuration::DISPATCHER_DEFAULTS)
@@ -13,13 +13,12 @@ module SolidQueue
       @batch_size = options[:batch_size]
 
       @concurrency_maintenance = ConcurrencyMaintenance.new(options[:concurrency_maintenance_interval], options[:batch_size]) if options[:concurrency_maintenance]
-      @recurring_schedule = RecurringSchedule.new(options[:recurring_tasks])
 
       super(**options)
     end
 
     def metadata
-      super.merge(batch_size: batch_size, concurrency_maintenance_interval: concurrency_maintenance&.interval, recurring_schedule: recurring_schedule.task_keys.presence)
+      super.merge(batch_size: batch_size, concurrency_maintenance_interval: concurrency_maintenance&.interval)
     end
 
     private

data/lib/solid_queue/log_subscriber.rb

@@ -94,7 +94,7 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
     if error = event.payload[:error]
       warn formatted_event(event, action: "Error registering #{process_kind}", **attributes.merge(error: formatted_error(error)))
     else
-      info formatted_event(event, action: "Register #{process_kind}", **attributes)
+      debug formatted_event(event, action: "Register #{process_kind}", **attributes)
     end
   end
 
@@ -114,7 +114,7 @@ class SolidQueue::LogSubscriber < ActiveSupport::LogSubscriber
     if error = event.payload[:error]
       warn formatted_event(event, action: "Error deregistering #{process.kind}", **attributes.merge(error: formatted_error(error)))
     else
-      info formatted_event(event, action: "Deregister #{process.kind}", **attributes)
+      debug formatted_event(event, action: "Deregister #{process.kind}", **attributes)
     end
   end
 

data/lib/solid_queue/processes/base.rb

@@ -10,6 +10,7 @@ module SolidQueue
 
       def initialize(*)
         @name = generate_name
+        @stopped = false
       end
 
       def kind
@@ -28,10 +29,18 @@ module SolidQueue
         {}
       end
 
+      def stop
+        @stopped = true
+      end
+
       private
         def generate_name
           [ kind.downcase, SecureRandom.hex(10) ].join("-")
         end
+
+        def stopped?
+          @stopped
+        end
     end
   end
 end

data/lib/solid_queue/processes/poller.rb

@@ -41,9 +41,6 @@ module SolidQueue::Processes
         raise NotImplementedError
       end
 
-      def shutdown
-      end
-
       def with_polling_volume
         SolidQueue.instrument(:polling) do
           if SolidQueue.silence_polling? && ActiveRecord::Base.logger

data/lib/solid_queue/processes/registrable.rb

@@ -29,11 +29,11 @@ module SolidQueue::Processes
       end
 
       def deregister
-        process.deregister if registered?
+        process&.deregister
       end
 
       def registered?
-        process&.persisted?
+        process.present?
       end
 
       def launch_heartbeat
@@ -53,7 +53,10 @@ module SolidQueue::Processes
       end
 
       def heartbeat
-        process.heartbeat
+        process.with_lock { process.heartbeat }
+      rescue ActiveRecord::RecordNotFound
+        self.process = nil
+        wake_up
       end
   end
 end

data/lib/solid_queue/processes/runnable.rb

@@ -17,7 +17,9 @@ module SolidQueue::Processes
     end
 
     def stop
-      @stopped = true
+      super
+
+      wake_up
       @thread&.join
     end
 
@@ -31,8 +33,6 @@ module SolidQueue::Processes
       def boot
         SolidQueue.instrument(:start_process, process: self) do
           run_callbacks(:boot) do
-            @stopped = false
-
             if running_as_fork?
               register_signal_handlers
               set_procline
@@ -41,18 +41,18 @@ module SolidQueue::Processes
         end
       end
 
+      def run
+        raise NotImplementedError
+      end
+
       def shutting_down?
-        stopped? || (running_as_fork? && supervisor_went_away?) || finished?
+        stopped? || (running_as_fork? && supervisor_went_away?) || finished? || !registered?
       end
 
       def run
         raise NotImplementedError
       end
 
-      def stopped?
-        @stopped
-      end
-
       def finished?
         running_inline? && all_work_completed?
       end
@@ -61,6 +61,9 @@ module SolidQueue::Processes
         false
       end
 
+      def shutdown
+      end
+
       def set_procline
       end
 

data/lib/solid_queue/processes/supervised.rb

@@ -29,7 +29,6 @@ module SolidQueue::Processes
         %w[ INT TERM ].each do |signal|
           trap(signal) do
             stop
-            interrupt
           end
         end
 

data/lib/solid_queue/{dispatcher → scheduler}/recurring_schedule.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module SolidQueue
-  class Dispatcher::RecurringSchedule
+  class Scheduler::RecurringSchedule
     include AppExecutor
 
     attr_reader :configured_tasks, :scheduled_tasks

data/lib/solid_queue/scheduler.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Scheduler < Processes::Base
+    include Processes::Runnable
+
+    attr_accessor :recurring_schedule
+
+    after_boot :schedule_recurring_tasks
+    before_shutdown :unschedule_recurring_tasks
+
+    def initialize(recurring_tasks:, **options)
+      @recurring_schedule = RecurringSchedule.new(recurring_tasks)
+
+      super(**options)
+    end
+
+    def metadata
+      super.merge(recurring_schedule: recurring_schedule.task_keys.presence)
+    end
+
+    private
+      SLEEP_INTERVAL = 60 # Right now it doesn't matter, can be set to 1 in the future for dynamic tasks
+
+      def run
+        loop do
+          break if shutting_down?
+
+          interruptible_sleep(SLEEP_INTERVAL)
+        end
+      ensure
+        SolidQueue.instrument(:shutdown_process, process: self) do
+          run_callbacks(:shutdown) { shutdown }
+        end
+      end
+
+      def schedule_recurring_tasks
+        recurring_schedule.schedule_tasks
+      end
+
+      def unschedule_recurring_tasks
+        recurring_schedule.unschedule_tasks
+      end
+
+      def all_work_completed?
+        recurring_schedule.empty?
+      end
+
+      def set_procline
+        procline "scheduling #{recurring_schedule.task_keys.join(",")}"
+      end
+  end
+end

data/lib/solid_queue/supervisor/maintenance.rb

@@ -24,7 +24,7 @@ module SolidQueue
       end
 
       def prune_dead_processes
-        wrap_in_app_executor { SolidQueue::Process.prune }
+        wrap_in_app_executor { SolidQueue::Process.prune(excluding: process) }
       end
 
       def fail_orphaned_executions

data/lib/solid_queue/supervisor.rb

@@ -6,9 +6,9 @@ module SolidQueue
     include Maintenance, Signals, Pidfiled
 
     class << self
-      def start(load_configuration_from: nil)
+      def start(**options)
         SolidQueue.supervisor = true
-        configuration = Configuration.new(load_from: load_configuration_from)
+        configuration = Configuration.new(**options)
 
         if configuration.configured_processes.any?
           new(configuration).tap(&:start)
@@ -37,7 +37,7 @@ module SolidQueue
     end
 
     def stop
-      @stopped = true
+      super
       run_stop_hooks
     end
 
@@ -47,7 +47,6 @@ module SolidQueue
       def boot
         SolidQueue.instrument(:start_process, process: self) do
           run_callbacks(:boot) do
-            @stopped = false
             sync_std_streams
           end
         end
@@ -87,10 +86,6 @@ module SolidQueue
         forks[pid] = process_instance
       end
 
-      def stopped?
-        @stopped
-      end
-
       def set_procline
         procline "supervising #{supervised_processes.join(", ")}"
       end

data/lib/solid_queue/version.rb

@@ -1,3 +1,3 @@
 module SolidQueue
-  VERSION = "0.8.2"
+  VERSION = "0.9.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: solid_queue
 version: !ruby/object:Gem::Version
-  version: 0.8.2
+  version: 0.9.0
 platform: ruby
 authors:
 - Rosa Gutierrez
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-09-06 00:00:00.000000000 Z
+date: 2024-09-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -203,6 +203,7 @@ files:
 - README.md
 - Rakefile
 - UPGRADING.md
+- app/jobs/solid_queue/recurring_job.rb
 - app/models/solid_queue/blocked_execution.rb
 - app/models/solid_queue/claimed_execution.rb
 - app/models/solid_queue/execution.rb
@@ -235,7 +236,8 @@ files:
 - lib/generators/solid_queue/install/USAGE
 - lib/generators/solid_queue/install/install_generator.rb
 - lib/generators/solid_queue/install/templates/bin/jobs
-- lib/generators/solid_queue/install/templates/config/solid_queue.yml
+- lib/generators/solid_queue/install/templates/config/queue.yml
+- lib/generators/solid_queue/install/templates/config/recurring.yml
 - lib/generators/solid_queue/install/templates/db/queue_schema.rb
 - lib/puma/plugin/solid_queue.rb
 - lib/solid_queue.rb
@@ -244,7 +246,6 @@ files:
 - lib/solid_queue/configuration.rb
 - lib/solid_queue/dispatcher.rb
 - lib/solid_queue/dispatcher/concurrency_maintenance.rb
-- lib/solid_queue/dispatcher/recurring_schedule.rb
 - lib/solid_queue/engine.rb
 - lib/solid_queue/lifecycle_hooks.rb
 - lib/solid_queue/log_subscriber.rb
@@ -260,6 +261,8 @@ files:
 - lib/solid_queue/processes/registrable.rb
 - lib/solid_queue/processes/runnable.rb
 - lib/solid_queue/processes/supervised.rb
+- lib/solid_queue/scheduler.rb
+- lib/solid_queue/scheduler/recurring_schedule.rb
 - lib/solid_queue/supervisor.rb
 - lib/solid_queue/supervisor/maintenance.rb
 - lib/solid_queue/supervisor/pidfile.rb
@@ -276,11 +279,15 @@ metadata:
   homepage_uri: https://github.com/rails/solid_queue
   source_code_uri: https://github.com/rails/solid_queue
 post_install_message: |
-  Upgrading to Solid Queue 0.8.0 from < 0.6.0? You need to upgrade to 0.6.0 first. Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
-  for upgrade instructions.
+  Upgrading to Solid Queue 0.9.0? There are some breaking changes about how recurring tasks are configured.
+
+  Upgrading to Solid Queue 0.8.0 from < 0.6.0? You need to upgrade to 0.6.0 first.
 
   Upgrading to Solid Queue 0.4.x, 0.5.x, 0.6.x or 0.7.x? There are some breaking changes about how Solid Queue is started,
   configuration and new migrations.
+
+  --> Check https://github.com/rails/solid_queue/blob/main/UPGRADING.md
+  for upgrade instructions.
 rdoc_options: []
 require_paths:
 - lib