solid_queue_mongoid 0.3.0

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Sal Scotto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,249 @@
+# SolidQueueMongoid
+
+[![CI](https://github.com/washu/solid_queue_mongoid/actions/workflows/ci.yml/badge.svg)](https://github.com/washu/solid_queue_mongoid/actions/workflows/ci.yml)
+
+A MongoDB/Mongoid adapter for [SolidQueue](https://github.com/basecamp/solid_queue) that allows you to use MongoDB as the backend instead of ActiveRecord/PostgreSQL/MySQL.
+
+This gem provides a drop-in replacement for SolidQueue's ActiveRecord models, using Mongoid documents instead. All SolidQueue features are supported including job scheduling, concurrency controls, recurring tasks, and more.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'solid_queue_mongoid'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+## How It Works
+
+`solid_queue_mongoid` defines its Mongoid models in the `SolidQueue::` namespace and then requires `solid_queue` for you. This means you only need one gem in your Gemfile — `solid_queue` is pulled in automatically as a dependency.
+
+In Rails, the gem's Railtie runs before Rails freezes `eager_load_paths` and tells Zeitwerk to ignore SolidQueue's `app/models` directory, so the ActiveRecord model files are never autoloaded.
+
+No special require ordering is needed in your application.
+
+## Configuration
+
+### 1. Configure Mongoid
+
+First, ensure you have Mongoid configured in your application. Create or update `config/mongoid.yml`:
+
+```yaml
+development:
+  clients:
+    default:
+      database: my_app_development
+      hosts:
+        - localhost:27017
+```
+
+### 2. Configure SolidQueueMongoid
+
+Create an initializer at `config/initializers/solid_queue_mongoid.rb`:
+
+```ruby
+# frozen_string_literal: true
+
+SolidQueueMongoid.configure do |config|
+  # Optional: Specify which Mongoid client to use for SolidQueue collections
+  # Default is :default
+  config.client = :default  # or :secondary, :solid_queue, etc.
+
+  # Optional: Set a collection prefix to avoid conflicts with existing collections
+  # Default is "solid_queue_"
+  config.collection_prefix = "solid_queue_"
+end
+```
+
+#### Using a Separate MongoDB Client
+
+If you want to store SolidQueue data in a separate MongoDB database, configure a separate client in `config/mongoid.yml`:
+
+```yaml
+development:
+  clients:
+    default:
+      database: my_app_development
+      hosts:
+        - localhost:27017
+    solid_queue:
+      database: my_app_jobs
+      hosts:
+        - localhost:27017
+```
+
+Then configure SolidQueueMongoid to use it:
+
+```ruby
+SolidQueueMongoid.configure do |config|
+  config.client = :solid_queue
+  config.collection_prefix = "solid_queue_"
+end
+```
+
+### 3. Create Indexes
+
+After configuration, create the necessary MongoDB indexes:
+
+```bash
+# Using rake task (recommended)
+bundle exec rake solid_queue_mongoid:create_indexes
+
+# Or in Ruby/Rails console
+SolidQueueMongoid.create_indexes
+```
+
+### How Client Configuration Works
+
+All SolidQueue models automatically use the configured Mongoid client for all queries and operations. The gem overrides Mongoid's query methods to ensure:
+
+- All queries (`where`, `find`, `create`, etc.) use the configured client
+- Cross-model associations work correctly within the same client
+- Index creation happens on the correct database
+- No manual `with(client:)` calls are needed in your code
+
+This means you can safely use multiple MongoDB databases without any special handling - just configure the client and everything works automatically.
+
+### Collection Naming
+
+With the default `collection_prefix` of `"solid_queue_"`, your collections will be named:
+
+- `solid_queue_jobs`
+- `solid_queue_ready_executions`
+- `solid_queue_claimed_executions`
+- `solid_queue_blocked_executions`
+- `solid_queue_scheduled_executions`
+- `solid_queue_failed_executions`
+- `solid_queue_recurring_executions`
+- `solid_queue_processes`
+- `solid_queue_pauses`
+- `solid_queue_semaphores`
+- `solid_queue_recurring_tasks`
+
+This prefix ensures that SolidQueue collections won't conflict with any existing collections in your database.
+
+To see all collection names:
+
+```bash
+bundle exec rake solid_queue_mongoid:show_collections
+```
+
+## Usage
+
+### 4. Configure the ActiveJob Adapter
+
+In `config/application.rb` (or the appropriate environment file):
+
+```ruby
+config.active_job.queue_adapter = :solid_queue
+```
+
+### Enqueuing Jobs
+
+Once configured, use SolidQueue exactly as you would with ActiveRecord:
+
+```ruby
+# In your ActiveJob
+class MyJob < ApplicationJob
+  queue_as :default
+
+  def perform(*args)
+    # Your job logic
+  end
+end
+
+# Enqueue jobs
+MyJob.perform_later(arg1, arg2)
+
+# Schedule jobs
+MyJob.set(wait: 1.hour).perform_later(arg1, arg2)
+MyJob.set(wait_until: Date.tomorrow.noon).perform_later(arg1, arg2)
+```
+
+### Configuration in Rails
+
+Configure SolidQueue in `config/queue.yml` or through `config.solid_queue` in your Rails configuration:
+
+```yaml
+# config/queue.yml
+production:
+  dispatchers:
+    - polling_interval: 1
+      batch_size: 500
+  workers:
+    - queues: "*"
+      threads: 3
+      processes: 2
+      polling_interval: 0.1
+```
+
+## Rake Tasks
+
+The gem provides several Rake tasks for managing indexes:
+
+```bash
+# Create all indexes
+bundle exec rake solid_queue_mongoid:create_indexes
+
+# Remove all indexes
+bundle exec rake solid_queue_mongoid:remove_indexes
+
+# Show collection names and configuration
+bundle exec rake solid_queue_mongoid:show_collections
+```
+
+## Index Management
+
+Unlike ActiveRecord migrations, MongoDB uses indexes that can be created on-demand. The gem provides a convenient way to manage these indexes similar to `db:migrate`:
+
+### Creating Indexes
+
+Always run this after:
+- Initial installation
+- Upgrading the gem
+- Changing configuration
+
+```bash
+bundle exec rake solid_queue_mongoid:create_indexes
+```
+
+### In Production
+
+Add index creation to your deployment process:
+
+```ruby
+# In a Rails initializer or deployment script
+if Rails.env.production?
+  SolidQueueMongoid.create_indexes
+end
+```
+
+Or use the Rake task in your deployment pipeline:
+
+```bash
+bundle exec rake solid_queue_mongoid:create_indexes
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/washu/solid_queue_mongoid. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/washu/solid_queue_mongoid/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the SolidQueueMongoid project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/washu/solid_queue_mongoid/blob/main/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/lib/solid_queue_mongoid/models/blocked_execution.rb

@@ -0,0 +1,125 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class BlockedExecution < Execution
+    assumes_attributes_from_job :concurrency_key
+
+    field :concurrency_key, type: String
+    field :expires_at, type: Time
+
+    before_create :set_expires_at
+
+    index({ concurrency_key: 1, priority: 1, job_id: 1 })
+    index({ expires_at: 1, concurrency_key: 1 })
+
+    INDEX_HINTS = {
+      index_solid_queue_blocked_executions_for_release: { concurrency_key: 1, priority: 1, job_id: 1 },
+      index_solid_queue_blocked_executions_for_maintenance: { expires_at: 1, concurrency_key: 1 }
+    }.freeze
+
+    scope :expired, -> { where(:expires_at.lt => Time.current) }
+
+    class << self
+      # Make create! idempotent: if a BlockedExecution for this job already exists
+      # (e.g. created by auto-dispatch), return the existing record instead of raising.
+      def create!(**attrs, &block)
+        super
+      rescue Mongo::Error::OperationFailure => e
+        raise unless e.message.to_s.include?("E11000") || e.message.to_s.include?("duplicate key")
+
+        job_id = attrs[:job_id] || attrs[:job]&.id
+        where(job_id: job_id).first || raise(e)
+      rescue Mongoid::Errors::Validations => e
+        return where(job_id: attrs[:job_id] || attrs[:job]&.id).first || raise(e) if uniqueness_only_error?(e.document)
+
+        raise
+      end
+
+      def unblock(limit)
+        SolidQueue.instrument(:release_many_blocked, limit: limit) do |payload|
+          expired_keys = expired.order(:concurrency_key).distinct(:concurrency_key).first(limit)
+          payload[:size] = release_many(releasable(expired_keys))
+        end
+      end
+
+      # Convenience method: release blocked executions for a given concurrency key.
+      # +limit+ is the maximum number to unblock (default: all).
+      def unblock_all(concurrency_key, limit = nil)
+        scope = ordered.where(concurrency_key: concurrency_key)
+        scope = scope.limit(limit) if limit
+        count = 0
+        scope.each do |execution|
+          break if limit && count >= limit
+
+          count += 1 if execution.release
+        end
+        count
+      end
+
+      def release_many(concurrency_keys)
+        Array(concurrency_keys).count { |key| release_one(key) }
+      end
+
+      def release_one(concurrency_key)
+        # NOTE: no outer Mongoid.transaction here — #release already wraps its work
+        # in a transaction. MongoDB does not support nested sessions, so wrapping
+        # again would cause InvalidSessionNesting errors.
+        execution = ordered.where(concurrency_key: concurrency_key).limit(1).first
+        execution ? execution.release : false
+      end
+
+      private
+
+      def releasable(concurrency_keys)
+        semaphores = Semaphore.where(:key.in => concurrency_keys).pluck(:key, :value, :limit)
+        # Build hash of key => [value, limit]
+        sem_map = semaphores.each_with_object({}) do |(key, value, limit), h|
+          h[key] = { value: value, limit: limit }
+        end
+
+        # Keys without semaphore (never acquired) + keys where value < limit (slot available)
+        (concurrency_keys - sem_map.keys) +
+          sem_map.select { |_key, s| s[:value] < s[:limit] }.keys
+      end
+    end
+
+    def unblock
+      release
+    end
+
+    def release
+      SolidQueue.instrument(:release_blocked, job_id: job.id, concurrency_key: concurrency_key,
+                                              released: false) do |payload|
+        Mongoid.transaction do
+          if acquire_concurrency_lock
+            promote_to_ready
+            destroy!
+            payload[:released] = true
+            true
+          else
+            false
+          end
+        end
+      end
+    end
+
+    private
+
+    def set_expires_at
+      self.expires_at = job.concurrency_duration.from_now
+    end
+
+    def acquire_concurrency_lock
+      Semaphore.wait(job)
+    end
+
+    def promote_to_ready
+      existing = ReadyExecution.where(job_id: job_id).first
+      return existing if existing
+
+      ReadyExecution.create!(job_id: job_id, queue_name: queue_name, priority: priority)
+    rescue Mongoid::Errors::Validations, Mongo::Error::OperationFailure
+      ReadyExecution.where(job_id: job_id).first
+    end
+  end
+end

data/lib/solid_queue_mongoid/models/claimed_execution.rb

@@ -0,0 +1,134 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class ClaimedExecution < Execution
+    assumes_attributes_from_job # inherits queue_name and priority from job
+
+    field :process_id, type: BSON::ObjectId
+
+    belongs_to :process, class_name: "SolidQueue::Process", optional: true
+
+    # Executions whose process_id references a process that no longer exists.
+    scope :orphaned, lambda {
+      existing_process_ids = SolidQueue::Process.all.pluck(:id)
+      existing_process_ids.empty? ? all : where(:process_id.nin => existing_process_ids)
+    }
+
+    index({ process_id: 1 })
+
+    Result = Struct.new(:success, :error) do
+      def success?
+        success
+      end
+    end
+
+    class << self
+      # Atomically creates ClaimedExecution records for the given job_ids and
+      # yields the claimed set to the block (which deletes the ReadyExecutions).
+      def claiming(job_ids, process_id, &block)
+        job_data = Array(job_ids).map { |job_id| { job_id: job_id, process_id: process_id } }
+
+        SolidQueue.instrument(:claim, process_id: process_id, job_ids: job_ids) do |payload|
+          claimed = job_data.filter_map do |attrs|
+            create!(attrs)
+          rescue Mongoid::Errors::Validations, Mongo::Error::OperationFailure
+            nil
+          end
+
+          block.call(claimed)
+
+          payload[:size] = claimed.size
+          payload[:claimed_job_ids] = claimed.map(&:job_id)
+        end
+      end
+
+      def release_all
+        SolidQueue.instrument(:release_many_claimed) do |payload|
+          executions = all.to_a
+          executions.each do |execution|
+            execution.release
+          rescue Mongoid::Errors::Validations, Mongo::Error::OperationFailure
+            # If ReadyExecution already exists, that's fine
+          end
+          payload[:size] = executions.size
+        end
+      end
+
+      def fail_all_with(error)
+        executions = includes(:job).to_a
+        return if executions.empty?
+
+        SolidQueue.instrument(:fail_many_claimed) do |payload|
+          executions.each do |execution|
+            execution.failed_with(error)
+            execution.unblock_next_job
+          end
+          payload[:process_ids] = executions.map(&:process_id).uniq
+          payload[:job_ids] = executions.map(&:job_id).uniq
+          payload[:size] = executions.size
+        end
+      end
+
+      def discard_all_in_batches(*)
+        raise Execution::UndiscardableError, "Can't discard jobs in progress"
+      end
+
+      def discard_all_from_jobs(*)
+        raise Execution::UndiscardableError, "Can't discard jobs in progress"
+      end
+    end
+
+    # Called by Pool thread — executes the job and marks it finished or failed.
+    def perform
+      result = execute
+
+      if result.success?
+        finished
+      else
+        failed_with(result.error)
+        raise result.error
+      end
+    ensure
+      unblock_next_job
+    end
+
+    # Release this execution back to ready (called by process deregister / prune).
+    def release
+      SolidQueue.instrument(:release_claimed, job_id: job.id, process_id: process_id) do
+        job.dispatch_bypassing_concurrency_limits
+        destroy!
+      end
+    end
+
+    def discard
+      raise Execution::UndiscardableError, "Can't discard a job in progress"
+    end
+
+    def failed_with(error)
+      Mongoid.transaction do
+        job.failed_with(error)
+        destroy!
+      end
+    end
+
+    def unblock_next_job
+      job.unblock_next_blocked_job
+    end
+
+    private
+
+    def execute
+      ActiveJob::Base.execute(job.arguments.merge("provider_job_id" => job.id.to_s))
+      Result.new(true, nil)
+    rescue Exception => e # rubocop:disable Lint/RescueException
+      Result.new(false, e)
+    end
+
+    def finished
+      Mongoid.transaction do
+        job.finished!
+        destroy!
+      end
+    end
+  end
+end

data/lib/solid_queue_mongoid/models/classes.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+# This file pre-declares all the main model classes without their concerns.
+# Must be loaded BEFORE concern files to avoid superclass mismatch errors.
+
+module SolidQueue
+  class Execution < Record; end
+
+  class Job < Record; end
+
+  class ReadyExecution < Execution; end
+  class ClaimedExecution < Execution; end
+  class BlockedExecution < Execution; end
+  class ScheduledExecution < Execution; end
+  class FailedExecution < Execution; end
+
+  class RecurringExecution < Record; end
+
+  class Process < Record
+    module Executor; end
+    module Prunable; end
+  end
+
+  class Pause < Record; end
+  # plain Ruby class — not Mongoid-backed
+  class Queue; end
+  class Semaphore < Record; end
+
+  class RecurringTask < Record
+    module Arguments; end
+  end
+end

data/lib/solid_queue_mongoid/models/execution/dispatching.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Execution < Record
+    module Dispatching
+      extend ActiveSupport::Concern
+
+      class_methods do
+        # Called by ScheduledExecution.dispatch_next_batch and FailedExecution.retry_all.
+        # Dispatches jobs by id: promotes them to ready/blocked, then removes the
+        # source execution records.
+        def dispatch_jobs(job_ids)
+          jobs = Job.where(:id.in => job_ids)
+
+          Job.dispatch_all(jobs).map(&:id).then do |dispatched_job_ids|
+            where(:job_id.in => dispatched_job_ids).delete_all
+            dispatched_job_ids.size
+          end
+        end
+      end
+    end
+  end
+end

data/lib/solid_queue_mongoid/models/execution/job_attributes.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module SolidQueue
+  class Execution < Record
+    module JobAttributes
+      extend ActiveSupport::Concern
+
+      included do
+        class_attribute :assumable_attributes_from_job, instance_accessor: false,
+                                                        default: %i[queue_name priority]
+
+        field :queue_name, type: String
+        field :priority, type: Integer, default: 0
+        field :job_id, type: BSON::ObjectId
+
+        index({ queue_name: 1, priority: 1, created_at: 1 })
+
+        belongs_to :job, class_name: "SolidQueue::Job", optional: false
+
+        # NOTE: uniqueness is enforced by the MongoDB unique index on job_id
+        # (added per-subclass by assumes_attributes_from_job), not by a Rails
+        # validator.  Rails-level uniqueness checks are susceptible to stale
+        # QueryCache reads and are redundant given the DB constraint.
+      end
+
+      class_methods do
+        # Subclasses call this to declare which additional job attributes they mirror.
+        # It registers a before_create callback that copies those attributes from the
+        # associated job at creation time.
+        # Also ensures a unique index on job_id for the calling class's collection.
+        def assumes_attributes_from_job(*attribute_names)
+          self.assumable_attributes_from_job = (assumable_attributes_from_job + attribute_names).uniq
+          before_create :assume_attributes_from_job
+          # Add unique job_id index to THIS subclass's collection (not parent's)
+          return if index_specifications.any? { |s| s.spec.keys.map(&:to_s) == ["job_id"] && s.options[:unique] }
+
+          index({ job_id: 1 }, { unique: true })
+        end
+
+        def attributes_from_job(job)
+          job.attributes.symbolize_keys.slice(*assumable_attributes_from_job)
+        end
+      end
+
+      private
+
+      def assume_attributes_from_job
+        self.class.assumable_attributes_from_job.each do |attr|
+          send("#{attr}=", job.send(attr)) if job.respond_to?(attr)
+        end
+      end
+    end
+  end
+end