solid_queue 0.4.1 → 0.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cb47bf3ee9dffa1300c093abb260394d088f100b3c16a026d755f706d9f7a852
-  data.tar.gz: 10cc1b6f866d148d0c1cefce7587614aaca3e938f3f04370c15318e5b8d3509f
+  metadata.gz: a7234dc4430998648196bf2d3905b66bb85e265c2393121a7c5343fed36b1996
+  data.tar.gz: 216a0918e29194e6d11fe4bf365183228127f8390658a11ace329523ad41468c
 SHA512:
-  metadata.gz: 71054017fcd26421f25140db9c929518ee011210fd6fea51e77352abf2a128289e965843dc1cb1fe97b62fa003d571e1de3631fcab329914b069ef9e0621806b
-  data.tar.gz: d08497fc98f9498aeaa3449182fc13cb90f71c859c67d55e0e5c14803bb1d5445e7717e1e3aca578fcb22a09c47126d465306e0220f54d6af4a8f8027fef0072
+  metadata.gz: a080aedf20f39940d8c25e8a14628811801b8fd4832fbcb926beecd0d1ce4c694ec8d80b608656f9de80cc8cc69b525f62d4fe7bc7258408aa6d6af62292d4fd
+  data.tar.gz: 66d24c1bb1c7cb1b9afbcf8a0b667df624dd3f47cea92887fbcd69d93b1832af3c44afb570d1cfb459c37e2b3c163900e0dd2febb067ee9179b4c49514e4b359

data/README.md

@@ -2,7 +2,7 @@
 
 Solid Queue is a DB-based queuing backend for [Active Job](https://edgeguides.rubyonrails.org/active_job_basics.html), designed with simplicity and performance in mind.
 
-Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`). _Improvements to logging and instrumentation, a better CLI tool, a way to run within an existing process in "async" mode, and some way of specifying unique jobs are coming very soon._
+Besides regular job enqueuing and processing, Solid Queue supports delayed jobs, concurrency controls, pausing queues, numeric priorities per job, priorities by queue order, and bulk enqueuing (`enqueue_all` for Active Job's `perform_all_later`).
 
 Solid Queue can be used with SQL databases such as MySQL, PostgreSQL or SQLite, and it leverages the `FOR UPDATE SKIP LOCKED` clause, if available, to avoid blocking and waiting on locks when polling jobs. It relies on Active Job for retries, discarding, error handling, serialization, or delays, and it's compatible with Ruby on Rails multi-threading.
 
@@ -31,9 +31,9 @@ $ bin/rails generate solid_queue:install
 
 This will set `solid_queue` as the Active Job's adapter in production, and will copy the required migration over to your app.
 
-Alternatively, you can add only the migration to your app:
+Alternatively, you can skip setting the Active Job's adapter with:
 ```bash
-$ bin/rails solid_queue:install:migrations
+$ bin/rails generate solid_queue:install --skip_adapter
 ```
 
 And set Solid Queue as your Active Job's queue backend manually, in your environment config:
@@ -42,7 +42,7 @@ And set Solid Queue as your Active Job's queue backend manually, in your environ
 config.active_job.queue_adapter = :solid_queue
 ```
 
-Alternatively, you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
+Or you can set only specific jobs to use Solid Queue as their backend if you're migrating from another adapter and want to move jobs progressively:
 
 ```ruby
 # app/jobs/my_job.rb
@@ -59,14 +59,14 @@ Finally, you need to run the migrations:
 $ bin/rails db:migrate
 ```
 
-After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them.
+After this, you'll be ready to enqueue jobs using Solid Queue, but you need to start Solid Queue's supervisor to run them. You can use the provided binstub:`
 ```bash
-$ bundle exec rake solid_queue:start
+$ bin/jobs
 ```
 
 This will start processing jobs in all queues using the default configuration. See [below](#configuration) to learn more about configuring Solid Queue.
 
-For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bundle exec rake solid_queue:start` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
+For small projects, you can run Solid Queue on the same machine as your webserver. When you're ready to scale, Solid Queue supports horizontal scaling out-of-the-box. You can run Solid Queue on a separate server from your webserver, or even run `bin/jobs` on multiple machines at the same time. Depending on the configuration, you can designate some machines to run only dispatchers or only workers. See the [configuration](#configuration) section for more details on this.
 
 ## Requirements
 Besides Rails 7.1, Solid Queue works best with MySQL 8+ or PostgreSQL 9.5+, as they support `FOR UPDATE SKIP LOCKED`. You can use it with older versions, but in that case, you might run into lock waits if you run multiple workers for the same queue.
@@ -80,7 +80,7 @@ We have three types of actors in Solid Queue:
 - _Dispatchers_ are in charge of selecting jobs scheduled to run in the future that are due and _dispatching_ them, which is simply moving them from the `solid_queue_scheduled_executions` table over to the `solid_queue_ready_executions` table so that workers can pick them up. They're also in charge of managing [recurring tasks](#recurring-tasks), dispatching jobs to process them according to their schedule. On top of that, they do some maintenance work related to [concurrency controls](#concurrency-controls).
 - The _supervisor_ runs workers and dispatchers according to the configuration, controls their heartbeats, and stops and starts them when needed.
 
-By default, Solid Queue runs in `fork` mode. This means the supervisor will fork a separate process for each supervised worker/dispatcher. There's also an `async` mode where each worker and dispatcher will be run as a thread of the supervisor process. This can be used with [the provided Puma plugin](#puma-plugin)
+Solid Queue's supervisor will fork a separate process for each supervised worker/dispatcher.
 
 By default, Solid Queue will try to find your configuration under `config/solid_queue.yml`, but you can set a different path using the environment variable `SOLID_QUEUE_CONFIG`. This is what this configuration looks like:
 
@@ -131,7 +131,7 @@ Here's an overview of the different options:
 
   Finally, you can combine prefixes with exact names, like `[ staging*, background ]`, and the behaviour with respect to order will be the same as with only exact names.
 - `threads`: this is the max size of the thread pool that each worker will have to run jobs. Each worker will fetch this number of jobs from their queue(s), at most and will post them to the thread pool to be run. By default, this is `3`. Only workers have this setting.
-- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting. **Note**: this option will be ignored if [running in `async` mode](#running-as-a-fork-or-asynchronously).
+- `processes`: this is the number of worker processes that will be forked by the supervisor with the settings given. By default, this is `1`, just a single process. This setting is useful if you want to dedicate more than one CPU core to a queue or queues with the same configuration. Only workers have this setting.
 - `concurrency_maintenance`: whether the dispatcher will perform the concurrency maintenance work. This is `true` by default, and it's useful if you don't use any [concurrency controls](#concurrency-controls) and want to disable it or if you run multiple dispatchers and want some of them to just dispatch jobs without doing anything else.
 - `recurring_tasks`: a list of recurring tasks the dispatcher will manage. Read more details about this one in the [Recurring tasks](#recurring-tasks) section.
 
@@ -194,13 +194,13 @@ development:
     # ...
 ```
 
-Install migrations and specify the dedicated database name with the `DATABASE` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
+Install migrations and specify the dedicated database name with the `--database` option. This will create the Solid Queue migration files in a separate directory, matching the value provided in `migrations_paths` in `config/database.yml`.
 
 ```bash
-$ bin/rails solid_queue:install:migrations DATABASE=solid_queue
+$ bin/rails g solid_queue:install --database solid_queue
 ```
 
-Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`), the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
+Note: If you've already run the solid queue install command (`bin/rails generate solid_queue:install`) without a `--database` option, the migration files will have already been generated under the primary database's `db/migrate/` directory. You can remove these files and keep the ones generated by the database-specific migration installation above.
 
 Finally, run the migrations:
 
@@ -208,17 +208,47 @@ Finally, run the migrations:
 $ bin/rails db:migrate
 ```
 
+## Lifecycle hooks
+
+In Solid queue, you can hook into two different points in the supervisor's life:
+- `start`: after the supervisor has finished booting and right before it forks workers and dispatchers.
+- `stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown.
+
+And into two different points in a worker's life:
+- `worker_start`: after the worker has finished booting and right before it starts the polling loop.
+- `worker_stop`: after receiving a signal (`TERM`, `INT` or `QUIT`) and right before starting graceful or immediate shutdown (which is just `exit!`).
+
+You can use the following methods with a block to do this:
+```ruby
+SolidQueue.on_start
+SolidQueue.on_stop
+
+SolidQueue.on_worker_start
+SolidQueue.on_worker_stop
+```
+
+For example:
+```ruby
+SolidQueue.on_start { start_metrics_server }
+SolidQueue.on_stop { stop_metrics_server }
+```
+
+These can be called several times to add multiple hooks, but it needs to happen before Solid Queue is started. An initializer would be a good place to do this.
+
+
 ### Other configuration settings
 _Note_: The settings in this section should be set in your `config/application.rb` or your environment config like this: `config.solid_queue.silence_polling = true`
 
 There are several settings that control how Solid Queue works that you can set as well:
 - `logger`: the logger you want Solid Queue to use. Defaults to the app logger.
 - `app_executor`: the [Rails executor](https://guides.rubyonrails.org/threading_and_code_execution.html#executor) used to wrap asynchronous operations, defaults to the app executor
-- `on_thread_error`: custom lambda/Proc to call when there's an error within a thread that takes the exception raised as argument. Defaults to
+- `on_thread_error`: custom lambda/Proc to call when there's an error within a Solid Queue thread that takes the exception raised as argument. Defaults to
 
   ```ruby
   -> (exception) { Rails.error.report(exception, handled: false) }
   ```
+  **This is not used for errors raised within a job execution**. Errors happening in jobs are handled by Active Job's `retry_on` or `discard_on`, and ultimately will result in [failed jobs](#failed-jobs-and-retries). This is for errors happening within Solid Queue itself.
+
 - `use_skip_locked`: whether to use `FOR UPDATE SKIP LOCKED` when performing locking reads. This will be automatically detected in the future, and for now, you'd only need to set this to `false` if your database doesn't support it. For MySQL, that'd be versions < 8, and for PostgreSQL, versions < 9.5. If you use SQLite, this has no effect, as writes are sequential.
 - `process_heartbeat_interval`: the heartbeat interval that all processes will follow—defaults to 60 seconds.
 - `process_alive_threshold`: how long to wait until a process is considered dead after its last heartbeat—defaults to 5 minutes.
@@ -283,6 +313,8 @@ In this case, if we have a `Box::MovePostingsByContactToDesignatedBoxJob` job en
 
 Note that the `duration` setting depends indirectly on the value for `concurrency_maintenance_interval` that you set for your dispatcher(s), as that'd be the frequency with which blocked jobs are checked and unblocked. In general, you should set `duration` in a way that all your jobs would finish well under that duration and think of the concurrency maintenance task as a failsafe in case something goes wrong.
 
+Jobs are unblocked in order of priority but queue order is not taken into account for unblocking jobs. That means that if you have a group of jobs that share a concurrency group but are in different queues, or jobs of the same class that you enqueue in different queues, the queue order you set for a worker is not taken into account when unblocking blocked ones. The reason is that a job that runs unblocks the next one, and the job itself doesn't know about a particular worker's queue order (you could even have different workers with different queue orders), it can only know about priority. Once blocked jobs are unblocked and available for polling, they'll be picked up by a worker following its queue order.
+
 Finally, failed jobs that are automatically or manually retried work in the same way as new jobs that get enqueued: they get in the queue for gaining the lock, and whenever they get it, they'll be run. It doesn't matter if they had gained the lock already in the past.
 
 ## Failed jobs and retries
@@ -305,18 +337,6 @@ plugin :solid_queue
 ```
 to your `puma.rb` configuration.
 
-### Running as a fork or asynchronously
-
-By default, the Puma plugin will fork additional processes for each worker and dispatcher so that they run in different processes. This provides the best isolation and performance, but can have additional memory usage.
-
-Alternatively, workers and dispatchers can be run within the same Puma process(s). To do so just configure the plugin as:
-
-```ruby
-plugin :solid_queue
-solid_queue_mode :async
-```
-
-Note that in this case, the `processes` configuration option will be ignored.
 
 ## Jobs and transactional integrity
 :warning: Having your jobs in the same ACID-compliant database as your application data enables a powerful yet sharp tool: taking advantage of transactional integrity to ensure some action in your app is not committed unless your job is also committed. This can be very powerful and useful, but it can also backfire if you base some of your logic on this behaviour, and in the future, you move to another active job backend, or if you simply move Solid Queue to its own database, and suddenly the behaviour changes under you.

data/UPGRADING.md

@@ -0,0 +1,102 @@
+# Upgrading to version 0.7.x
+
+This version removed the new async mode introduced in version 0.4.0 and introduced a new binstub that can be used to start Solid Queue's supervisor. It includes also a minor migration.
+
+To install both the binstub `bin/jobs` and the migration, you can just run
+```
+bin/rails generate solid_queue:install
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails generate solid_queue:install --database <the_name_of_your_solid_queue_db>
+```
+
+
+# Upgrading to version 0.6.x
+
+## New migration in 3 steps
+This version adds two new migrations to modify the `solid_queue_processes` table. The goal of that migration is to add a new column that needs to be `NOT NULL`. This needs to be done with two migrations and the following steps to ensure it happens without downtime and with new processes being able to register just fine:
+1. Run the first migration that adds the new column, nullable
+2. Deploy the updated Solid Queue code that uses this column
+2. Run the second migration. This migration does two things:
+  - Backfill existing rows that would have the column as NULL
+  - Make the column not nullable and add a new index
+
+Besides, it adds another migration with no effects to the `solid_queue_recurring_tasks` table. This one can be run just fine whenever, as the column affected is not used.
+
+To install the migrations:
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then follow the steps above, running first one, then deploying the code, then running the second one.
+
+## New behaviour when workers are killed
+From this version onwards, when a worker is killed and the supervisor can detect that, it'll fail in-progress jobs claimed by that worker. For this to work correctly, you need to run the above migration and ensure you restart any supervisors you'd have. 
+
+
+# Upgrading to version 0.5.x
+This version includes a new migration to improve recurring tasks. To install it, just run:
+
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then run the migrations.
+
+
+# Upgrading to version 0.4.x
+This version introduced an _async_ mode (this mode has been removed in version 0.7.0) to run the supervisor and have all workers and dispatchers run as part of the same process as the supervisor, instead of separate, forked, processes. Together with this, we introduced some changes in how the supervisor is started. Prior this change, you could choose whether you wanted to run workers, dispatchers or both, by starting Solid Queue as `solid_queue:work` or `solid_queue:dispatch`. From version 0.4.0, the only option available is:
+
+```
+$ bundle exec rake solid_queue:start
+```
+Whether the supervisor starts workers, dispatchers or both will depend on your configuration. For example, if you don't configure any dispatchers, only workers will be started. That is, with this configuration:
+
+```yml
+production:
+  workers:
+    - queues: [ real_time, background ]
+      threads: 5
+      polling_interval: 0.1
+      processes: 3
+```
+the supervisor will run 3 workers, each one with 5 threads, and no supervisors. With this configuration:
+```yml
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+      concurrency_maintenance_interval: 300
+```
+the supervisor will run 1 dispatcher and no workers.
+
+
+# Upgrading to version 0.3.x
+This version introduced support for [recurring (cron-style) jobs](https://github.com/rails/solid_queue/blob/main/README.md#recurring-tasks), and it needs a new DB migration for it. To install it, just run:
+
+```bash
+$ bin/rails solid_queue:install:migrations
+```
+
+Or, if you're using a different database for Solid Queue:
+
+```bash
+$ bin/rails solid_queue:install:migrations DATABASE=<the_name_of_your_solid_queue_db>
+```
+
+And then run the migrations.

data/app/jobs/solid_queue/recurring_job.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+class SolidQueue::RecurringJob < ActiveJob::Base
+  def perform(command)
+    SolidQueue.instrument(:run_command, command: command) do
+      eval(command, TOPLEVEL_BINDING, __FILE__, __LINE__)
+    end
+  end
+end

data/app/models/solid_queue/claimed_execution.rb

@@ -29,8 +29,21 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     def release_all
       SolidQueue.instrument(:release_many_claimed) do |payload|
         includes(:job).tap do |executions|
-          payload[:size] = executions.size
           executions.each(&:release)
+
+          payload[:size] = executions.size
+        end
+      end
+    end
+
+    def fail_all_with(error)
+      SolidQueue.instrument(:fail_many_claimed) do |payload|
+        includes(:job).tap do |executions|
+          executions.each { |execution| execution.failed_with(error) }
+
+          payload[:process_ids] = executions.map(&:process_id).uniq
+          payload[:job_ids] = executions.map(&:job_id).uniq
+          payload[:size] = executions.size
         end
       end
     end
@@ -69,6 +82,13 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
     raise UndiscardableError, "Can't discard a job in progress"
   end
 
+  def failed_with(error)
+    transaction do
+      job.failed_with(error)
+      destroy!
+    end
+  end
+
   private
     def execute
       ActiveJob::Base.execute(job.arguments)
@@ -83,11 +103,4 @@ class SolidQueue::ClaimedExecution < SolidQueue::Execution
         destroy!
       end
     end
-
-    def failed_with(error)
-      transaction do
-        job.failed_with(error)
-        destroy!
-      end
-    end
 end

data/app/models/solid_queue/process/executor.rb

@@ -8,7 +8,19 @@ module SolidQueue
       included do
         has_many :claimed_executions
 
-        after_destroy -> { claimed_executions.release_all }, if: :claims_executions?
+        after_destroy :release_all_claimed_executions
+      end
+
+      def fail_all_claimed_executions_with(error)
+        if claims_executions?
+          claimed_executions.fail_all_with(error)
+        end
+      end
+
+      def release_all_claimed_executions
+        if claims_executions?
+          claimed_executions.release_all
+        end
       end
 
       private

data/app/models/solid_queue/process/prunable.rb

@@ -15,11 +15,18 @@ module SolidQueue
             prunable.non_blocking_lock.find_in_batches(batch_size: 50) do |batch|
               payload[:size] += batch.size
 
-              batch.each { |process| process.deregister(pruned: true) }
+              batch.each(&:prune)
             end
           end
         end
       end
+
+      def prune
+        error = Processes::ProcessPrunedError.new(last_heartbeat_at)
+        fail_all_claimed_executions_with(error)
+
+        deregister(pruned: true)
+      end
     end
   end
 end

data/app/models/solid_queue/process.rb

@@ -4,7 +4,7 @@ class SolidQueue::Process < SolidQueue::Record
   include Executor, Prunable
 
   belongs_to :supervisor, class_name: "SolidQueue::Process", optional: true, inverse_of: :supervisees
-  has_many :supervisees, class_name: "SolidQueue::Process", inverse_of: :supervisor, foreign_key: :supervisor_id, dependent: :destroy
+  has_many :supervisees, class_name: "SolidQueue::Process", inverse_of: :supervisor, foreign_key: :supervisor_id
 
   store :metadata, coder: JSON
 
@@ -13,10 +13,10 @@ class SolidQueue::Process < SolidQueue::Record
       create!(attributes.merge(last_heartbeat_at: Time.current)).tap do |process|
         payload[:process_id] = process.id
       end
+    rescue Exception => error
+      payload[:error] = error
+      raise
     end
-  rescue Exception => error
-    SolidQueue.instrument :register_process, **attributes.merge(error: error)
-    raise
   end
 
   def heartbeat
@@ -25,12 +25,19 @@ class SolidQueue::Process < SolidQueue::Record
 
   def deregister(pruned: false)
     SolidQueue.instrument :deregister_process, process: self, pruned: pruned do |payload|
-      payload[:claimed_size] = claimed_executions.size if claims_executions?
-
       destroy!
+
+      unless supervised? || pruned
+        supervisees.each(&:deregister)
+      end
     rescue Exception => error
       payload[:error] = error
       raise
     end
   end
+
+  private
+    def supervised?
+      supervisor_id.present?
+    end
 end

data/app/models/solid_queue/recurring_execution.rb

@@ -7,16 +7,29 @@ module SolidQueue
     scope :clearable, -> { where.missing(:job) }
 
     class << self
+      def create_or_insert!(**attributes)
+        if connection.supports_insert_conflict_target?
+          # PostgreSQL fails and aborts the current transaction when it hits a duplicate key conflict
+          # during two concurrent INSERTs for the same value of an unique index. We need to explicitly
+          # indicate unique_by to ignore duplicate rows by this value when inserting
+          unless insert(attributes, unique_by: [ :task_key, :run_at ]).any?
+            raise AlreadyRecorded
+          end
+        else
+          create!(**attributes)
+        end
+      rescue ActiveRecord::RecordNotUnique
+        raise AlreadyRecorded
+      end
+
       def record(task_key, run_at, &block)
         transaction do
           block.call.tap do |active_job|
-            if active_job
-              create!(job_id: active_job.provider_job_id, task_key: task_key, run_at: run_at)
+            if active_job && active_job.successfully_enqueued?
+              create_or_insert!(job_id: active_job.provider_job_id, task_key: task_key, run_at: run_at)
             end
           end
         end
-      rescue ActiveRecord::RecordNotUnique => e
-        raise AlreadyRecorded
       end
 
       def clear_in_batches(batch_size: 500)

data/app/models/solid_queue/recurring_task/arguments.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require "active_job/arguments"
+
+module SolidQueue
+  class RecurringTask::Arguments
+    class << self
+      def load(data)
+        data.nil? ? [] : ActiveJob::Arguments.deserialize(ActiveSupport::JSON.load(data))
+      end
+
+      def dump(data)
+        ActiveSupport::JSON.dump(ActiveJob::Arguments.serialize(Array(data)))
+      end
+    end
+  end
+end

data/app/models/solid_queue/recurring_task.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+require "fugit"
+
+module SolidQueue
+  class RecurringTask < Record
+    serialize :arguments, coder: Arguments, default: []
+
+    validate :supported_schedule
+    validate :existing_job_class
+
+    scope :static, -> { where(static: true) }
+
+    class << self
+      def wrap(args)
+        args.is_a?(self) ? args : from_configuration(args.first, **args.second)
+      end
+
+      def from_configuration(key, **options)
+        new(key: key, class_name: options[:class], schedule: options[:schedule], arguments: options[:args])
+      end
+
+      def create_or_update_all(tasks)
+        if connection.supports_insert_conflict_target?
+          # PostgreSQL fails and aborts the current transaction when it hits a duplicate key conflict
+          # during two concurrent INSERTs for the same value of an unique index. We need to explicitly
+          # indicate unique_by to ignore duplicate rows by this value when inserting
+          upsert_all tasks.map(&:attributes_for_upsert), unique_by: :key
+        else
+          upsert_all tasks.map(&:attributes_for_upsert)
+        end
+      end
+    end
+
+    def delay_from_now
+      [ (next_time - Time.current).to_f, 0 ].max
+    end
+
+    def next_time
+      parsed_schedule.next_time.utc
+    end
+
+    def enqueue(at:)
+      SolidQueue.instrument(:enqueue_recurring_task, task: key, at: at) do |payload|
+        active_job = if using_solid_queue_adapter?
+          enqueue_and_record(run_at: at)
+        else
+          payload[:other_adapter] = true
+
+          perform_later do |job|
+            unless job.successfully_enqueued?
+              payload[:enqueue_error] = job.enqueue_error&.message
+            end
+          end
+        end
+
+        payload[:active_job_id] = active_job.job_id if active_job
+      rescue RecurringExecution::AlreadyRecorded
+        payload[:skipped] = true
+      rescue Job::EnqueueError => error
+        payload[:enqueue_error] = error.message
+      end
+    end
+
+    def to_s
+      "#{class_name}.perform_later(#{arguments.map(&:inspect).join(",")}) [ #{parsed_schedule.original} ]"
+    end
+
+    def attributes_for_upsert
+      attributes.without("id", "created_at", "updated_at")
+    end
+
+    private
+      def supported_schedule
+        unless parsed_schedule.instance_of?(Fugit::Cron)
+          errors.add :schedule, :unsupported, message: "is not a supported recurring schedule"
+        end
+      end
+
+      def existing_job_class
+        unless job_class.present?
+          errors.add :class_name, :undefined, message: "doesn't correspond to an existing class"
+        end
+      end
+
+      def using_solid_queue_adapter?
+        job_class.queue_adapter_name.inquiry.solid_queue?
+      end
+
+      def enqueue_and_record(run_at:)
+        RecurringExecution.record(key, run_at) do
+          job_class.new(*arguments_with_kwargs).tap do |active_job|
+            active_job.run_callbacks(:enqueue) do
+              Job.enqueue(active_job)
+            end
+            active_job.successfully_enqueued = true
+          end
+        end
+      end
+
+      def perform_later(&block)
+        job_class.perform_later(*arguments_with_kwargs, &block)
+      end
+
+      def arguments_with_kwargs
+        if arguments.last.is_a?(Hash)
+          arguments[0...-1] + [ Hash.ruby2_keywords_hash(arguments.last) ]
+        else
+          arguments
+        end
+      end
+
+
+      def parsed_schedule
+        @parsed_schedule ||= Fugit.parse(schedule)
+      end
+
+      def job_class
+        @job_class ||= class_name&.safe_constantize
+      end
+  end
+end

data/app/models/solid_queue/semaphore.rb

@@ -17,6 +17,17 @@ module SolidQueue
       def signal_all(jobs)
         Proxy.signal_all(jobs)
       end
+
+      # Requires a unique index on key
+      def create_unique_by(attributes)
+        if connection.supports_insert_conflict_target?
+          insert({ **attributes }, unique_by: :key).any?
+        else
+          create!(**attributes)
+        end
+      rescue ActiveRecord::RecordNotUnique
+        false
+      end
     end
 
     class Proxy
@@ -44,15 +55,17 @@ module SolidQueue
         attr_accessor :job
 
         def attempt_creation
-          Semaphore.create!(key: key, value: limit - 1, expires_at: expires_at)
-          true
-        rescue ActiveRecord::RecordNotUnique
-          if limit == 1 then false
+          if Semaphore.create_unique_by(key: key, value: limit - 1, expires_at: expires_at)
+            true
           else
-            attempt_decrement
+            check_limit_or_decrement
           end
         end
 
+        def check_limit_or_decrement
+          limit == 1 ? false : attempt_decrement
+        end
+
         def attempt_decrement
           Semaphore.available.where(key: key).update_all([ "value = value - 1, expires_at = ?", expires_at ]) > 0
         end

data/db/migrate/20240719134516_create_recurring_tasks.rb

@@ -0,0 +1,20 @@
+class CreateRecurringTasks < ActiveRecord::Migration[7.1]
+  def change
+    create_table :solid_queue_recurring_tasks do |t|
+      t.string :key, null: false, index: { unique: true }
+      t.string :schedule, null: false
+      t.string :command, limit: 2048
+      t.string :class_name
+      t.text :arguments
+
+      t.string :queue_name
+      t.integer :priority, default: 0
+
+      t.boolean :static, default: true, index: true
+
+      t.text :description
+
+      t.timestamps
+    end
+  end
+end

data/db/migrate/20240811173327_add_name_to_processes.rb

@@ -0,0 +1,5 @@
+class AddNameToProcesses < ActiveRecord::Migration[7.1]
+  def change
+    add_column :solid_queue_processes, :name, :string
+  end
+end