marj 4.1.0 → 6.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8ea99cb466811e6f77fb6476e7e8ca02b7345c8d297ebdb1954b9109f772bfd9
-  data.tar.gz: 35a1ed99e2afe6934a5a7ea9b40375a93d5c79b9c46d304aac49457913794fba
+  metadata.gz: 6d0039931eb3f0bd43ce090b0b581d6544b26fda022729f6371f4c8ac393076e
+  data.tar.gz: 6abb66afe0bc56064be4ecee3f3e59e78adcf10afb83654529387b785f817335
 SHA512:
-  metadata.gz: 0a1568a19c7faeedc80d0a17cf71c536a5dcb9b497645c87772213b5a20ca22b8ae250099826a75654d36260b5d34d77c896b81b28cc87e3ec1e4e181b5440bb
-  data.tar.gz: 701727d777c8f9c990bdca6d66f9cf3d76d41eddddd77f1dbcbb1c75cff5b75b7acce065cb5c83b01c23dff3a6878218b4b6f3c4528d0e307ae8bc8e192d91e9
+  metadata.gz: ec25e50e772072eb6d13ebcb04a67e8ee4602782b62edab07e9521817993f6d92bac376bff90a6aeca1c724fa61dc4ba04ac1fe8ce9659139d2dcd98c397bce4
+  data.tar.gz: f78b5ba741202818a5374c95fddcb6258b091d4feba4b2de1d1f9de8d1422b99348bf5e354619d215b32b9b7f8d4055ef95d271055cf489e4049b06b40d3463c

data/README.md

@@ -4,42 +4,89 @@ A minimal database-backed ActiveJob queueing backend.
 
 ## Quick Links
 
-API docs: https://gemdocs.org/gems/marj/latest <br>
+API docs: https://gemdocs.org/gems/marj/6.0.0/ <br>
 RubyGems: https://rubygems.org/gems/marj <br>
 Changelog: https://github.com/nicholasdower/marj/releases <br>
 Issues: https://github.com/nicholasdower/marj/issues <br>
 Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
 
+## Motivation
+
+There are already several great database-backed ActiveJob queueing backends:
+- [Delayed::Job](https://github.com/collectiveidea/delayed_job)
+- [Que](https://github.com/que-rb/que)
+- [GoodJob](https://github.com/bensheldon/good_job)
+- [Solid Queue](https://github.com/basecamp/solid_queue)
+
+Any of these which support your RDBMS is likely to work well for you. But you
+may find them to be more featureful and complex than what you require.
+
+Marj aims to be a minimal alternative.
+
+## Goal
+
+To be the database-backend ActiveJob queueing backend with:
+- The simplest setup
+- The fewest configuration options
+- The fewest features
+- The fewest backend-specific APIs
+- The fewest lines of code
+
 ## Features
 
+Marj supports and has been tested with MySQL, PostgreSQL and SQLite.
+
+It provides the following features:
 - Enqueued jobs are written to the database.
-- Successfully executed jobs are deleted from the database.
-- Failed jobs which should be retried are updated in the database.
-- Failed jobs which should not be retried are deleted from the database.
-- An interface is provided to retrieve, execute, discard and re-enqueue jobs.
-- An `ActiveRecord` class is provided to query the database directly.
+- Enqueued jobs can be queried, executed and discarded.
+- Executed jobs are re-enqueued or discarded, depending on the result.
+- Compatibility with [Mission Control Jobs](https://github.com/basecamp/mission_control-jobs).
+
+## Extensions
+
+Marj does not provide the following features by default, but each can easily
+be added to your application with a few lines of code. See [Example Usage](#example-usage)
+and [Extension Examples](#extension-examples) for suggestions.
+- [Automatic job execution](#example-usage)
+- [Concurrency protection](#concurrency-protection)
+- [Job timeouts](#job-timeouts)
+- [Error storage](#error-storage)
+- [Discarded job retention](#discarded-job-retention)
+- [Observability](#observability)
+- [Multi-DB Support](#multi-db-support)
+
+## API
+
+Marj relies on existing ActiveJob APIs, for example:
+
+```ruby
+job.enqueue         # Enqueues a job
+job.perform_now     # Performs a job
+```
+
+Additionally, it extends the ActiveJob API with methods required for a
+minimal queueing backend implementaion:
+
+```ruby
+SomeJob.query(args) # Queries for enqeueued jobs
+job.discard         # Runs discard callbacks and, by default, deletes the job
+job.delete          # Deletes the job
+```
 
-## Features Not Provided
+## Requirements
 
-- Workers
-- Timeouts
-- Concurrency Controls
-- Observability
-- A User Interace
+- `activejob >= 7.1`
+- `activerecord >= 7.1`
 
 ## Setup
 
 ### 1. Install
 
 ```shell
-bundle add activejob activerecord marj
-
-# or
-
-gem install activejob activerecord marj
+bundle add marj
 ```
 
-### 3. Create the database table
+### 2. Create the database table
 
 ```ruby
 class CreateJobs < ActiveRecord::Migration[7.1]
@@ -68,149 +115,74 @@ class CreateJobs < ActiveRecord::Migration[7.1]
 end
 ```
 
-Note that by default, Marj uses a table named `jobs`. To override the default
-table name, set `Marj.table_name` before loading `ActiveRecord`.
-
-### 4. Configure the queue adapter
+### 3. Configure the queue adapter
 
 ```ruby
 require 'marj'
 
+# Choose one of the following:
 Rails.configuration.active_job.queue_adapter = :marj # Globally, with Rails
 ActiveJob::Base.queue_adapter = :marj                # Globally, without Rails
 SomeJob.queue_adapter = :marj                        # Single job
 ```
 
-## Example Usage
-
-```ruby
-# Enqueue and manually run a job:
-job = SomeJob.perform_later('foo')
-job.perform_now
+### 4. Optionally, include the Marj module
 
-# Retrieve and execute a job
-Marj.due.next.perform_now
-
-# Run all due jobs (single DB query)
-Marj.due.perform_all
-
-# Run all due jobs (multiple DB queries)
-Marj.due.perform_all(batch_size: 1)
-
-# Run all due jobs in a specific queue:
-Marj.queue('foo').due.perform_all
-
-# Run jobs as they become due:
-loop do
-  Marj.due.perform_all rescue logger.error($!)
-ensure
-  sleep 5.seconds
-end
-```
-
-## Jobs Interface
-
-The `Marj` module provides methods for interacting with enqueued jobs. These
-methods accept, return and yield +ActiveJob+ objects rather than +ActiveRecord+
-objects. Returned jobs are orderd by due date. To query the database directly,
-use `Marj::Record`.
-
-Example usage:
+Without any additional setup, jobs can be queried and discarded via the `Marj`
+module:
 
 ```ruby
-Marj.all         # Returns all enqueued jobs.
-Marj.queue       # Returns jobs in the specified queue(s).
-Marj.due         # Returns jobs which are due to be executed.
-Marj.next        # Returns the next job(s) to be executed.
-Marj.count       # Returns the number of enqueued jobs.
-Marj.where       # Returns jobs matching the specified criteria.
-Marj.perform_all # Executes all jobs.
-Marj.discard_all # Discards all jobs.
-Marj.discard     # Discards the specified job.
+Marj.query(:all)
+Marj.discard(job)
 ```
 
-Query methods can also be chained:
+But it is also convenient to query or discard via job classes:
 
 ```ruby
-Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
+SomeJob.query(:all)
+SomeJob.discard(job)
+job.discard
 ```
 
-## Custom Jobs Interface
-
-The `Marj::JobsInterface` can be added to any class or module. For example, to
-add it to all jobs classes:
+In order to enable this functionality, you must include the `Marj` module:
 
 ```ruby
-class ApplicationJob < ActiveJob::Base
-  extend Marj::JobsInterface
+class SomeJob < ActiveJob::Base
+  include Marj
 
-  def self.all
-    Marj::Relation.new(
-      self == ApplicationJob ?
-        Marj::Record.ordered : Marj::Record.where(job_class: self)
-   )
-  end
+  def perform; end
 end
-
-class SomeJob < ApplicationJob; end
-
-ApplicationJob.due # Returns all jobs which are due to be executed.
-SomeJob.due        # Returns SomeJobs which are due to be executed.
 ```
 
-## Multiple Tables
-
-It is possible to create a custom record class in order to, for instance,
-write jobs to multiple databases/tables within a single application.
+## Example Usage
 
 ```ruby
-class CreateMyJobs < ActiveRecord::Migration[7.1]
-  def self.up
-    create_table :my_jobs, id: :string, primary_key: :job_id do |table|
-      table.string   :job_class,            null: false
-      table.text     :arguments,            null: false
-      table.string   :queue_name,           null: false
-      table.integer  :priority
-      table.integer  :executions,           null: false
-      table.text     :exception_executions, null: false
-      table.datetime :enqueued_at,          null: false
-      table.datetime :scheduled_at
-      table.string   :locale,               null: false
-      table.string   :timezone,             null: false
-    end
-
-    add_index :my_jobs, %i[enqueued_at]
-    add_index :my_jobs, %i[scheduled_at]
-    add_index :my_jobs, %i[priority scheduled_at enqueued_at]
-  end
-
-  def self.down
-    drop_table :my_jobs
-  end
-end
-
-class MyRecord < Marj::Record
-  self.table_name = 'my_jobs'
-end
-
-CreateMyJobs.migrate(:up)
+# Enqueue a job
+job = SomeJob.perform_later('foo')
 
-class MyJob < ActiveJob::Base
-  self.queue_adapter = MarjAdapter.new('MyRecord')
+# Query jobs
+SomeJob.query(:all)
+SomeJob.query(:due)
+SomeJob.query(:due, queue: :foo)
+SomeJob.query('8720417d-8fff-4fcf-bc16-22aaef8543d2')
 
-  extend Marj::JobsInterface
+# Execute a job
+job.perform_now
 
-  def self.all
-    Marj::Relation.new(MyRecord.all)
-  end
+# Execute jobs which are due (single query)
+SomeJob.query(:due).map(&:perform_now)
 
-  def perform(msg)
-    puts msg
-  end
+# Execute jobs which are due (multiple queries)
+loop {
+  SomeJob.query(:due, :first)&.tap(&:perform_now) || break
 end
 
-MyJob.perform_later('oh, hi')
-MyJob.due.next.perform_now
+# Execute jobs as they become due
+loop do
+  SomeJob.query(:due).each(&:perform_now) rescue logger.error($!)
+ensure
+  sleep 5.seconds
+end
 ```
 
 ## Testing
@@ -219,17 +191,54 @@ By default, jobs enqeued during tests will be written to the database. Enqueued
 jobs can be executed via:
 
 ```ruby
-Marj.due.perform_all
+SomeJob.query(:due).map(&:perform_now)
 ```
 
-Alternatively, to use [ActiveJob::QueueAdapters::TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
+Alternatively, to use ActiveJob's [TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
 ```ruby
 ActiveJob::Base.queue_adapter = :test
 ```
 
 ## Extension Examples
 
-### Timeouts
+Most missing features can easily be added to your application with a few lines
+of code.
+
+### Concurrency Protection
+
+To prevent two threads from executing the same job simultaneously, consider
+moving jobs to a different queue before executing them.
+
+```ruby
+class ApplicationJob < ActiveJob::Base
+  around_perform do |job, block|
+    if job.queue_name.start_with?('claimed')
+      raise "Job #{job.job_id} already claimed"
+    end
+
+    updated = Marj::Record
+      .where(job_id: job.job_id, queue_name: job.queue_name)
+      .update_all(
+        queue_name: "claimed-#{job.queue_name}",
+        scheduled_at: Time.now.utc
+      )
+    unless updated == 1
+      raise "Failed to claim job #{job.job_id}. #{updated} records updated"
+    end
+    
+    begin
+      block.call
+    rescue StandardError
+      Marj::Record
+        .where(job_id: job.job_id, queue_name: "claimed-#{job.queue_name}")
+        .update_all(queue_name: job.queue_name, scheduled_at: job.scheduled_at)
+      raise
+    end
+  end
+end
+```
+
+### Job Timeouts
 
 ```ruby
 class ApplicationJob < ActiveJob::Base
@@ -249,7 +258,7 @@ class ApplicationJob < ActiveJob::Base
 end
 ```
 
-### Last Error
+### Error Storage
 
 ```ruby
 class AddLastErrorToJobs < ActiveRecord::Migration[7.1]
@@ -290,6 +299,83 @@ class ApplicationJob < ActiveJob::Base
 end
 ```
 
+### Discarded Job Retention
+
+By default, discarded jobs are deleted from the database. If desired, this
+behavior can be overridden, for instance to move jobs to a differnt queue:
+
+```ruby
+ActiveJob::Base.queue_adapter = MarjAdapter.new(
+  discard: proc { _1.enqueue(queue: 'discarded') }
+)
+
+# To retrieve a discarded job, query the discarded queue:
+job = Marj.query(:first, queue_name: 'discarded')
+
+# To delete, use one of the following:
+Marj.delete(job)
+job.delete
+```
+
+### Observability
+
+For instance with Prometheus metrics:
+
+```ruby
+class ApplicationJob < ActiveJob::Base
+  counter = Prometheus::Client::Counter.new(
+    :job_events_total, docstring: '...', labels: [:job, :event]
+  )
+
+  around_enqueue do |job, block|
+    counter.increment(labels: { job: job.class.name, event: 'before_enqueue' })
+    block.call
+    counter.increment(labels: { job: job.class.name, event: 'after_enqueue' })
+  rescue Exception
+    counter.increment(labels: { job: job.class.name, event: 'enqueue_error' })
+    raise
+  end
+
+  around_perform do |job, block|
+    counter.increment(labels: { job: job.class.name, event: 'before_perform' })
+    block.call
+    counter.increment(labels: { job: job.class.name, event: 'after_perform' })
+  rescue Exception => e # rubocop:disable Lint/RescueException
+    counter.increment(labels: { job: job.class.name, event: 'perform_error' })
+    raise
+  end
+
+  after_discard do |job|
+    counter.increment(labels: { job: job.class.name, event: 'after_discard' })
+  end
+end
+```
+
+### Multi-DB Support
+
+It is possible to create a custom record class in order to, for instance,
+use a different class or table name, or write jobs to multiple
+databases/tables within a single application.
+
+Assuming you have a jobs tabled named `my_jobs`:
+
+```ruby
+class MyRecord < Marj::Record
+  self.table_name = 'my_jobs'
+end
+
+class MyJob < ActiveJob::Base
+  include Marj
+
+  self.queue_adapter = MarjAdapter.new('MyRecord')
+
+  def perform; end
+end
+
+MyJob.perform_later('oh, hi')
+MyJob.query(:due, :first).perform_now
+```
+
 ## ActiveJob Cheatsheet
 
 For more information on ActiveJob, see:

data/lib/marj/mission_control.rb

@@ -0,0 +1,119 @@
+# frozen_string_literal: true
+
+require_relative '../marj_adapter'
+require 'mission_control/jobs'
+require 'mission_control/jobs/adapter'
+
+# :nocov:
+module Marj
+  module MissionControl
+    include ::MissionControl::Jobs::Adapter
+
+    def queues
+      record_class.group(:queue_name).count(:queue_name).map { |k, v| { name: k, size: v, active: true } }
+    end
+
+    def queue_size(queue_name)
+      record_class.where(queue_name: queue_name).count
+    end
+
+    def clear_queue(queue_name)
+      Marj::Record.where(queue_name: queue_name).delete_all
+    end
+
+    def pause_queue(_queue_name)
+      raise 'not supported: pause queue'
+    end
+
+    def resume_queue(_queue_name)
+      raise 'not supported: resume queue'
+    end
+
+    def queue_paused?(_queue_name)
+      false
+    end
+
+    def supported_statuses
+      %i[pending failed scheduled]
+    end
+
+    def supported_filters(_jobs_relation)
+      %i[queue_name job_class_name]
+    end
+
+    def exposes_workers?
+      false
+    end
+
+    def workers
+      raise 'not supported: workers'
+    end
+
+    def find_worker(_worker_id)
+      raise 'not supported: find workers'
+    end
+
+    def jobs_count(jobs_relation)
+      ar_relation(jobs_relation).count
+    end
+
+    def fetch_jobs(jobs_relation)
+      ar_relation(jobs_relation).each_with_index.map { |record, index| to_job(record, jobs_relation, index) }
+    end
+
+    def retry_all_jobs(jobs_relation)
+      ar_relation(jobs_relation).map { |record| record.to_job.perform_now }
+    end
+
+    def retry_job(job, _jobs_relation)
+      Marj::Record.find(job.job_id).to_job.perform_now
+    end
+
+    def discard_all_jobs(jobs_relation)
+      ar_relation(jobs_relation).map { |record| discard(record.to_job) }
+    end
+
+    def discard_job(job, _jobs_relation)
+      discard(Marj::Record.find(job.job_id).to_job)
+    end
+
+    def find_job(job_id, jobs_relation)
+      to_job(record_class.find_by(job_id: job_id), jobs_relation)
+    end
+
+    private
+
+    def ar_relation(jobs_relation)
+      relation = Marj::Record.all.offset(jobs_relation.offset_value).limit(jobs_relation.limit_value)
+      relation = relation.where.not(executions: 0) if jobs_relation.status == :failed
+      relation = relation.where.not(scheduled_at: nil) if jobs_relation.status == :scheduled
+      relation = relation.where(job_class: jobs_relation.job_class_name) if jobs_relation.job_class_name
+      relation = relation.where(queue_name: jobs_relation.queue_name) if jobs_relation.queue_name
+      relation
+    end
+
+    def to_job(record, jobs_relation, index = 0)
+      return nil unless record
+
+      job = record.to_job
+      job_data = job.serialize
+      ActiveJob::JobProxy.new(job_data).tap do |proxy|
+        if job.executions.positive?
+          proxy.last_execution_error = ActiveJob::ExecutionError.new(
+            error_class: Exception, message: 'unknown', backtrace: []
+          )
+          proxy.failed_at = job.enqueued_at
+          proxy.status = :failed
+        elsif job.scheduled_at
+          proxy.status = :scheduled
+        else
+          proxy.status = :pending
+        end
+        proxy.raw_data = job_data
+        proxy.position = jobs_relation.offset_value + index
+        proxy.arguments = job.arguments # For some reason MissionControl sets the arguments to the entire job data
+      end
+    end
+  end
+end
+# :nocov:

data/lib/marj/record.rb

@@ -6,7 +6,7 @@ require 'active_record'
 module Marj
   # The default +ActiveRecord+ class.
   class Record < ActiveRecord::Base
-    self.table_name = Marj.table_name
+    self.table_name = :jobs
 
     # Order by +enqueued_at+ rather than +job_id+ (the default).
     self.implicit_order_column = 'enqueued_at'
@@ -50,9 +50,47 @@ module Marj
     # discarded.
     #
     # @return [ActiveJob::Base]
-    def as_job
-      Marj.send(:to_job, self)
+    def to_job
+      # See register_callbacks for details on how callbacks are used.
+      job = job_class.new.tap { register_callbacks(_1) }
+
+      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
+      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
+      job_data = attributes.merge('arguments' => JSON.parse(read_attribute_before_type_cast(:arguments)))
+
+      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
+      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
+
+      job.deserialize(job_data)
+
+      # ActiveJob deserializes arguments on demand when a job is performed. Until then they are empty. That's strange.
+      # Instead, deserialize them now. Also, clear `serialized_arguments` to prevent ActiveJob from overwriting changes
+      # to arguments when serializing later.
+      job.arguments = arguments
+      job.serialized_arguments = nil
+
+      job
+    end
+
+    # Registers callbacks for the given job which destroy this record when the job succeeds or is discarded.
+    #
+    # @param job [ActiveJob::Base]
+    # @return [ActiveJob::Base]
+    def register_callbacks(job)
+      raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
+
+      record = self
+      # We need to detect three cases:
+      #  - If a job succeeds, after_perform will be called.
+      #  - If a job fails and should be retried, enqueue will be called. This is handled by the queue adapter.
+      #  - If a job exceeds its max attempts, after_discard will be called.
+      job.singleton_class.after_perform { |_j| job.queue_adapter.delete(job) }
+      job.singleton_class.after_discard { |_j, _exception| job.queue_adapter.discard(job, run_callbacks: false) }
+      job.singleton_class.instance_variable_set(:@record, record)
+
+      job
     end
+    private :register_callbacks
 
     class << self
       # Returns an +ActiveRecord::Relation+ scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in
@@ -63,11 +101,16 @@ module Marj
         where('scheduled_at IS NULL OR scheduled_at <= ?', Time.now.utc)
       end
 
-      # Returns an +ActiveRecord::Relation+ scope for jobs ordered by +priority+ (+null+ last), then +scheduled_at+
-      # (+null+ last), then +enqueued_at+.
+      # Returns an +ActiveRecord::Relation+ scope for jobs ordered by due date.
+      #
+      # Jobs are ordered by the following criteria, in order:
+      # 1. past or null scheduled_at before future scheduled_at
+      # 2. ascending priority, nulls last
+      # 3. ascending scheduled_at, nulls last
+      # 4. ascending enqueued_at
       #
       # @return [ActiveRecord::Relation]
-      def ordered
+      def by_due_date
         order(
           Arel.sql(<<~SQL.squish, Time.now.utc)
             CASE WHEN scheduled_at IS NULL OR scheduled_at <= ? THEN 0 ELSE 1 END,

data/lib/marj.rb

@@ -1,164 +1,131 @@
 # frozen_string_literal: true
 
 require_relative 'marj_adapter'
-require_relative 'marj/jobs_interface'
-require_relative 'marj/relation'
 
 # A minimal database-backed ActiveJob queueing backend.
 #
-# The {Marj} module provides methods for interacting with enqueued jobs. These methods accept, return and yield
-# +ActiveJob+ objects rather than +ActiveRecord+ objects. Returned jobs are ordered by due date. To query the database
-# directly, use {Record}.
+# The {Marj} module provides the following methods:
+# - +query+ - Queries enqueued jobs
+# - +discard+ - Discards a job, by default by executing after_discard callbacks and delegating to delete
+# - +delete+ - Deletes a job
+#
+# It is possible to call the above methods on the {Marj} module itself or on any class which includes it.
 #
 # Example usage:
-#   Marj.all         # Returns all enqueued jobs.
-#   Marj.queue       # Returns jobs in the specified queue(s).
-#   Marj.due         # Returns jobs which are due to be executed.
-#   Marj.next        # Returns the next job(s) to be executed.
-#   Marj.count       # Returns the number of enqueued jobs.
-#   Marj.where       # Returns jobs matching the specified criteria.
-#   Marj.perform_all # Executes all jobs.
-#   Marj.discard_all # Discards all jobs.
-#   Marj.discard     # Discards the specified job.
+#   Marj.query(:first) # Returns the first job
+#   Marj.discard(job)  # Discards the specified job
+#   Marj.delete(job)  # Deletes the specified job
+#
+#   class ApplicationJob < ActiveJob::Base
+#     include Marj
+#   end
 #
-# Query methods can also be chained:
-#   Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
+#   class SomeJob < ApplicationJob;
+#     def perform; end
+#   end
 #
-# Note that by default, Marj uses {Marj::Record} to interact with the +jobs+ table. To use a different record class, set
-# {record_class}. To simply override the table name, set {table_name} before loading +ActiveRecord+.
+#   job = ApplicationJob.query(:first) # Returns the first enqueued job
+#   job = SomeJob.query(:first)        # Returns the first enqueued job with job_class SomeJob
+#   ApplicationJob.discard(job)        # Discards the specified job
+#   ApplicationJob.delete(job)         # Deletes the specified job
+#   job.discard                        # Discards the job
+#   job.delete                         # Deletes the job
 #
 # See https://github.com/nicholasdower/marj
 module Marj
   # The Marj version.
-  VERSION = '4.1.0'
+  VERSION = '6.0.0'
 
   Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
 
-  @table_name = :jobs
-  @record_class = 'Marj::Record'
-
-  class << self
-    include Marj::JobsInterface
-
-    # @!attribute record_class
-    #   The name of the +ActiveRecord+ class. Defaults to +Marj::Record+.
-    #   @return [Class, String]
-
-    attr_writer :record_class
-
-    def record_class
-      @record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
-    end
-
-    # @!attribute table_name
-    #   The name of the database table. Defaults to +:jobs+.
-    #   @return [Symbol, String]
-    attr_accessor :table_name
-
-    # Returns a {Marj::Relation} for all jobs in the order they should be executed.
+  # Provides the {query} and {discard} class methods.
+  module ClassMethods
+    # Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
+    # - Symbol arguments are treated as +ActiveRecord+ scopes.
+    # - If only a job ID is specified, the corresponding job is returned.
+    # - If +:limit+ is specified, the maximum number of jobs is limited.
+    # - If +:order+ is specified, the jobs are ordered by the given attribute.
     #
-    # @return [Marj::Relation]
-    def all
-      Marj::Relation.new(Marj.record_class.ordered)
+    # By default jobs are ordered by when they should be executed.
+    #
+    # Example usage:
+    #   query                       # Returns all jobs
+    #   query(:all)                 # Returns all jobs
+    #   query(:due)                 # Returns jobs which are due to be executed
+    #   query(:due, limit: 10)      # Returns at most 10 jobs which are due to be executed
+    #   query(job_class: Foo)       # Returns all jobs with job_class Foo
+    #   query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
+    #   query(queue_name: 'foo')    # Returns all jobs in the 'foo' queue
+    #   query(job_id: '123')        # Returns the job with job_id '123' or nil if no such job exists
+    #   query('123')                # Returns the job with job_id '123' or nil if no such job exists
+    def query(*args, **kwargs)
+      kwargs[:job_class] ||= self if self < ActiveJob::Base && name != 'ApplicationJob'
+      queue_adapter.query(*args, **kwargs)
     end
 
     # Discards the specified job.
     #
-    # @return [Integer] the number of discarded jobs
+    # @return [ActiveJob::Base] the discarded job
     def discard(job)
-      all.where(job_id: job.job_id).discard_all
+      queue_adapter.discard(job)
     end
 
-    private
-
-    # Creates a job instance for the given record which will update the database when successfully executed, enqueued or
-    # discarded.
+    # Deletes the record associated with the specified job.
     #
-    # @param record [ActiveRecord::Base]
-    # @return [ActiveJob::Base] the new job instance
-    def to_job(record)
-      # See register_callbacks for details on how callbacks are used.
-      job = record.job_class.new.tap { register_callbacks(_1, record) }
-
-      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
-      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
-      job_data = record.attributes.merge('arguments' => JSON.parse(record.read_attribute_before_type_cast(:arguments)))
-
-      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
-      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
+    # @return [ActiveJob::Base] the deleted job
+    def delete(job)
+      queue_adapter.delete(job)
+    end
+  end
 
-      job.deserialize(job_data)
+  # (see ClassMethods#query)
+  def self.query(*args, **kwargs)
+    queue_adapter.query(*args, **kwargs)
+  end
 
-      # ActiveJob deserializes arguments on demand when a job is performed. Until then they are empty. That's strange.
-      # Instead, deserialize them now. Also, clear `serialized_arguments` to prevent ActiveJob from overwriting changes
-      # to arguments when serializing later.
-      job.arguments = record.arguments
-      job.serialized_arguments = nil
+  # (see ClassMethods#discard)
+  def self.discard(job)
+    queue_adapter.discard(job)
+  end
 
-      job
-    end
+  # (see ClassMethods#delete)
+  def self.delete(job)
+    queue_adapter.delete(job)
+  end
 
-    # Registers callbacks for the given job which destroy the given database record when the job succeeds or is
-    # discarded.
-    #
-    # @param job [ActiveJob::Base]
-    # @param record [ActiveRecord::Base]
-    # @return [ActiveJob::Base]
-    def register_callbacks(job, record)
-      raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
+  # Deletes this job.
+  #
+  # @return [ActiveJob::Base] this job
+  def discard
+    self.class.queue_adapter.discard(self)
+  end
 
-      # We need to detect three cases:
-      #  - If a job succeeds, after_perform will be called.
-      #  - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
-      #  - If a job exceeds its max attempts, after_discard will be called.
-      job.singleton_class.after_perform { |_j| record.destroy! }
-      job.singleton_class.after_discard { |_j, _exception| record.destroy! }
-      job.singleton_class.instance_variable_set(:@record, record)
+  # Deletes the record associated with this job.
+  #
+  # @return [ActiveJob::Base] this job
+  def delete
+    self.class.queue_adapter.delete(self)
+  end
 
-      job
-    end
+  def self.included(clazz)
+    clazz.extend(ClassMethods)
+  end
+  private_class_method :included
 
-    # Enqueue a job for execution at the specified time.
-    #
-    # @param job [ActiveJob::Base] the job to enqueue
-    # @param record_class [Class] the +ActiveRecord+ class
-    # @param time [Time, NilClass] optional time at which to execute the job
-    # @return [ActiveJob::Base] the enqueued job
-    def enqueue(job, record_class, time = nil)
-      job.scheduled_at = time
-      # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
-      serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
+  def self.queue_adapter
+    ActiveJob::Base.queue_adapter
+  end
+  private_class_method :queue_adapter
+end
 
-      # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
-      # are registered on the job instance so that when the job is executed, the database record is deleted or updated
-      # (depending on the result).
-      #
-      # We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
-      # singleton class. This holds a reference to the record. This ensures that if execute is called on a record
-      # instance, any updates to the database are reflected on that record instance.
-      if (existing_record = job.singleton_class.instance_variable_get(:@record))
-        # This job instance has already been associated with a database row.
-        if record_class.exists?(job_id: job.job_id)
-          # The database row still exists, we simply need to update it.
-          existing_record.update!(serialized)
-        else
-          # Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
-          # want to register the new instance because someone might still have a reference to the existing one.
-          record_class.create!(serialized)
-          existing_record.reload
-        end
-      else
-        # This job instance has not been associated with a database row.
-        if (new_record = record_class.find_by(job_id: job.job_id))
-          # The database row already exists. Update it.
-          new_record.update!(serialized)
-        else
-          # The database row does not exist. Create it.
-          new_record = record_class.create!(serialized)
-        end
-        register_callbacks(job, new_record)
-      end
-      job
-    end
+# :nocov:
+if defined?(Rails)
+  begin
+    require 'mission_control/jobs'
+    require_relative 'marj/mission_control'
+    MarjAdapter.include(Marj::MissionControl)
+  rescue LoadError
+    # ignore
   end
 end
+# :nocov:

data/lib/marj_adapter.rb

@@ -2,13 +2,26 @@
 
 # ActiveJob queue adapter for Marj.
 #
+# In addition to the standard +ActiveJob+ queue adapter API, this adapter provides:
+# - A +query+ method which can be used to query enqueued jobs
+# - A +discard+ method which can be used to discard enqueued jobs.
+# - A +delete+ method which can be used to delete enqueued jobs.
+#
+# Although it is possible to access the adapter directly in order to query, discard or delete, it is recommended to use
+# the {Marj} module.
+#
 # See https://github.com/nicholasdower/marj
 class MarjAdapter
+  JOB_ID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.freeze
+  private_constant :JOB_ID_REGEX
+
   # Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ class.
   #
-  # @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use to store jobs
-  def initialize(record_class = 'Marj::Record')
+  # @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use, defaults to +Marj::Record+
+  # @param discard [Proc] the proc to use to discard jobs, defaults to delegating to {delete}
+  def initialize(record_class: 'Marj::Record', discard: proc { |job| delete(job) })
     @record_class = record_class
+    @discard_proc = discard
   end
 
   # Enqueue a job for immediate execution.
@@ -16,7 +29,7 @@ class MarjAdapter
   # @param job [ActiveJob::Base] the job to enqueue
   # @return [ActiveJob::Base] the enqueued job
   def enqueue(job)
-    Marj.send(:enqueue, job, record_class)
+    enqueue_at(job)
   end
 
   # Enqueue a job for execution at the specified time.
@@ -24,13 +37,146 @@ class MarjAdapter
   # @param job [ActiveJob::Base] the job to enqueue
   # @param timestamp [Numeric, NilClass] optional number of seconds since Unix epoch at which to execute the job
   # @return [ActiveJob::Base] the enqueued job
-  def enqueue_at(job, timestamp)
-    Marj.send(:enqueue, job, record_class, timestamp ? Time.at(timestamp).utc : nil)
+  def enqueue_at(job, timestamp = nil)
+    job.scheduled_at = timestamp ? Time.at(timestamp).utc : nil
+
+    # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
+    serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
+
+    # Serialize sets locale to I18n.locale.to_s and enqueued_at to Time.now.utc.iso8601(9).
+    # Update the job to reflect what is being enqueued.
+    job.locale = serialized[:locale]
+    job.enqueued_at = Time.iso8601(serialized[:enqueued_at]).utc
+
+    # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
+    # are registered on the job instance so that when the job is executed, the database record is deleted or updated
+    # (depending on the result).
+    #
+    # We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
+    # singleton class. This holds a reference to the record. This ensures that if execute is called on a record
+    # instance, any updates to the database are reflected on that record instance.
+    if (existing_record = job.singleton_class.instance_variable_get(:@record))
+      # This job instance has already been associated with a database row.
+      if record_class.exists?(job_id: job.job_id)
+        # The database row still exists, we simply need to update it.
+        existing_record.update!(serialized)
+      else
+        # Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
+        # want to register the new instance because someone might still have a reference to the existing one.
+        record_class.create!(serialized)
+        existing_record.reload
+      end
+    else
+      # This job instance has not been associated with a database row.
+      if (new_record = record_class.find_by(job_id: job.job_id))
+        # The database row already exists. Update it.
+        new_record.update!(serialized)
+      else
+        # The database row does not exist. Create it.
+        new_record = record_class.create!(serialized)
+      end
+      new_record.send(:register_callbacks, job)
+    end
+    job
+  end
+
+  # Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
+  # - Symbol arguments are treated as +ActiveRecord+ scopes.
+  # - If only a job ID is specified, the corresponding job is returned.
+  # - If +:limit+ is specified, the maximum number of jobs is limited.
+  # - If +:order+ is specified, the jobs are ordered by the given attribute.
+  #
+  # By default jobs are ordered by when they should be executed.
+  #
+  # Example usage:
+  #   query                       # Returns all jobs
+  #   query(:all)                 # Returns all jobs
+  #   query(:due)                 # Returns jobs which are due to be executed
+  #   query(:due, limit: 10)      # Returns at most 10 jobs which are due to be executed
+  #   query(job_class: Foo)       # Returns all jobs with job_class Foo
+  #   query(:due, job_class: Foo) # Returns jobs which are due to be executed with job_class Foo
+  #   query(queue_name: 'foo')    # Returns all jobs in the 'foo' queue
+  #   query(job_id: '123')        # Returns the job with job_id '123' or nil if no such job exists
+  #   query('123')                # Returns the job with job_id '123' or nil if no such job exists
+  def query(*args, **kwargs)
+    args, kwargs = args.dup, kwargs.dup.symbolize_keys
+    kwargs = kwargs.merge(job_id: kwargs.delete(:id)) if kwargs.key?(:id)
+    kwargs[:job_id] = args.shift if args.size == 1 && args.first.is_a?(String) && args.first.match(JOB_ID_REGEX)
+
+    if args.empty? && kwargs.size == 1 && kwargs.key?(:job_id)
+      return record_class.find_by(job_id: kwargs[:job_id])&.to_job
+    end
+
+    symbol_args, args = args.partition { _1.is_a?(Symbol) }
+    symbol_args.delete(:all)
+    limit = kwargs.delete(:limit)
+    relation = record_class.all
+    relation = relation.order(kwargs.delete(:order)) if kwargs.key?(:order)
+    relation = relation.where(*args, **kwargs) if args.any? || kwargs.any?
+    relation = relation.limit(limit) if limit
+    relation = relation.send(symbol_args.shift) while symbol_args.any?
+    relation = relation.by_due_date if relation.is_a?(ActiveRecord::Relation) && relation.order_values.empty?
+
+    if relation.is_a?(Enumerable)
+      relation.map(&:to_job)
+    elsif relation.is_a?(record_class)
+      relation.to_job
+    else
+      relation
+    end
+  end
+
+  # Discards the specified job.
+  #
+  # @param job [ActiveJob::Base] the job being discarded
+  # @param run_callbacks [Boolean] whether to run the +after_discard+ callbacks
+  # @return [ActiveJob::Base] the discarded job
+  def discard(job, run_callbacks: true)
+    job.tap do
+      @discard_proc.call(job)
+      run_after_discard_callbacks(job) if run_callbacks
+    end
+  end
+
+  # Deletes the record associated with the specified job.
+  #
+  # @return [ActiveJob::Base] the deleted job
+  def delete(job)
+    job.tap { destroy_record(job) }
   end
 
   private
 
+  # Returns the +ActiveRecord+ class to use to store jobs.
+  #
+  # @return [Class] the +ActiveRecord+ class
   def record_class
     @record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
   end
+
+  # Destroys the record associated with the given job if it exists.
+  #
+  # @return [ActiveRecord::Base, NilClass] the destroyed record or +nil+ if no such record exists
+  def destroy_record(job)
+    record = job.singleton_class.instance_variable_get(:@record)
+    record ||= Marj::Record.find_by(job_id: job.job_id)&.tap { _1.send(:register_callbacks, job) }
+    record&.destroy
+  end
+
+  # Invokes the specified job's +after_discard+ callbacks.
+  #
+  # @param job [ActiveJob::Base] the job being discarded
+  # @return [NilClass] the given job
+  def run_after_discard_callbacks(job)
+    # Copied from ActiveJob::Exceptions#run_after_discard_procs
+    exceptions = []
+    job.after_discard_procs.each do |blk|
+      instance_exec(job, nil, &blk)
+    rescue StandardError => e
+      exceptions << e
+    end
+    raise exceptions.last if exceptions.any?
+
+    nil
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: marj
 version: !ruby/object:Gem::Version
-  version: 4.1.0
+  version: 6.0.0
 platform: ruby
 authors:
 - Nick Dower
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-01-28 00:00:00.000000000 Z
+date: 2024-02-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -48,17 +48,16 @@ files:
 - LICENSE.txt
 - README.md
 - lib/marj.rb
-- lib/marj/jobs_interface.rb
+- lib/marj/mission_control.rb
 - lib/marj/record.rb
-- lib/marj/relation.rb
 - lib/marj_adapter.rb
 homepage: https://github.com/nicholasdower/marj
 licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/nicholasdower/marj/issues
-  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v4.1.0
-  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v4.1.0
+  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v6.0.0
+  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v6.0.0
   homepage_uri: https://github.com/nicholasdower/marj
   rubygems_mfa_required: 'true'
   source_code_uri: https://github.com/nicholasdower/marj

data/lib/marj/jobs_interface.rb

@@ -1,113 +0,0 @@
-# frozen_string_literal: true
-
-module Marj
-  # The interface provided by {Marj} and {Marj::Relation}.
-  #
-  # To create a custom jobs interface, for example for all job classes in your application:
-  #   class ApplicationJob < ActiveJob::Base
-  #     extend Marj::JobsInterface
-  #
-  #     def self.all
-  #       Marj::Relation.new(self == ApplicationJob ? Marj::Record.ordered : Marj::Record.where(job_class: self))
-  #     end
-  #   end
-  #
-  #   class SomeJob < ApplicationJob
-  #     def perform(msg)
-  #       puts msg
-  #     end
-  #   end
-  #
-  # This will allow you to query jobs via the +ApplicationJob+ class:
-  #   ApplicationJob.next # Returns the next job of any type
-  #
-  # Or to query jobs via a specific job class:
-  #   SomeJob.next # Returns the next SomeJob
-  #
-  # Alternatively, to create a jobs interface for a single job class:
-  #   class SomeJob < ActiveJob::Base
-  #     extend Marj::JobsInterface
-  #
-  #     def self.all
-  #       Marj::Relation.new(Marj::Record.where(job_class: self).ordered)
-  #     end
-  #   end
-  module JobsInterface
-    def self.included(clazz)
-      return if clazz == Marj::Relation
-
-      clazz.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
-    end
-    private_class_method :included
-
-    def self.extended(clazz)
-      clazz.singleton_class.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
-    end
-    private_class_method :extended
-
-    # Returns a {Marj::Relation} for jobs in the specified queue(s).
-    #
-    # @param queue [String, Symbol] the queue to query
-    # @param queues [Array<String>, Array<Symbol>] more queues to query
-    # @return [Marj::Relation]
-    def queue(queue, *queues)
-      Marj::Relation.new(all.where(queue_name: queues.dup.unshift(queue)))
-    end
-
-    # Returns the next job or the next N jobs if +limit+ is specified. If no jobs exist, returns +nil+.
-    #
-    # @param limit [Integer, NilClass]
-    # @return [ActiveJob::Base, NilClass]
-    def next(limit = nil)
-      all.first(limit)&.then { _1.is_a?(Array) ? _1.map(&:as_job) : _1.as_job }
-    end
-
-    # Returns a count of jobs, optionally either matching the specified column name criteria or where the specified
-    # block returns +true+.
-    #
-    # @param column_name [String, Symbol, NilClass]
-    # @param block [Proc, NilClass]
-    # @return [Integer]
-    def count(column_name = nil, &block)
-      block_given? ? all.count(column_name) { |r| block.call(r.as_job) } : all.count(column_name)
-    end
-
-    # Returns a {Marj::Relation} for jobs matching the specified criteria.
-    #
-    # @param args [Array]
-    # @return [Marj::Relation]
-    def where(*args)
-      Marj::Relation.new(all.where(*args))
-    end
-
-    # Returns a {Marj::Relation} for enqueued jobs with a +scheduled_at+ that is either +null+ or in the past.
-    #
-    # @return [Marj::Relation]
-    def due
-      Marj::Relation.new(all.due)
-    end
-
-    # Calls +perform_now+ on each job.
-    #
-    # @param batch_size [Integer, NilClass] the number of jobs to fetch at a time, or +nil+ to fetch all jobs at once
-    # @return [Array] the results returned by each job
-    def perform_all(batch_size: nil)
-      if batch_size
-        [].tap do |results|
-          while (jobs = all.limit(batch_size).map(&:as_job)).any?
-            results.concat(jobs.map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } })
-          end
-        end
-      else
-        all.map(&:as_job).map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } }
-      end
-    end
-
-    # Discards all jobs.
-    #
-    # @return [Numeric] the number of discarded jobs
-    def discard_all
-      all.delete_all
-    end
-  end
-end

data/lib/marj/relation.rb

@@ -1,72 +0,0 @@
-# frozen_string_literal: true
-
-require_relative 'jobs_interface'
-
-module Marj
-  # Returned by {Marj::JobsInterface} query methods to enable chaining and +Enumerable+ methods.
-  class Relation
-    include Enumerable
-    include Marj::JobsInterface
-
-    attr_reader :all
-    private :all
-
-    # Returns a {Marj::Relation} which wraps the specified +ActiveRecord+ relation.
-    def initialize(ar_relation)
-      @all = ar_relation
-    end
-
-    # Yields each job in this relation.
-    #
-    # @param block [Proc]
-    # @return [Array] the jobs in this relation
-    def each(&block)
-      all.map(&:as_job).each(&block)
-    end
-
-    # Provides +pretty_inspect+ output containing arrays of jobs rather than arrays of records, similar to the output
-    # produced when calling +pretty_inspect+ on +ActiveRecord::Relation+.
-    #
-    # Instead of the default +pretty_inspect+ output:
-    #   > Marj.all
-    #    =>
-    #   #<Marj::Relation:0x000000012728bd88
-    #    @ar_relation=
-    #     [#<Marj::Record:0x0000000126c42080
-    #       job_id: "1382cb98-c518-46ca-a0cc-d831e11a0714",
-    #       job_class: TestJob,
-    #       arguments: ["foo"],
-    #       queue_name: "default",
-    #       priority: nil,
-    #       executions: 0,
-    #       exception_executions: {},
-    #       enqueued_at: 2024-01-25 15:31:06.115773 UTC,
-    #       scheduled_at: nil,
-    #       locale: "en",
-    #       timezone: "UTC">]>
-    #
-    # Produces:
-    #   > Marj.all
-    #    =>
-    #   [#<TestJob:0x000000010b63cef8
-    #     @_scheduled_at_time=nil,
-    #     @arguments=[],
-    #     @enqueued_at=2024-01-25 15:31:06 UTC,
-    #     @exception_executions={},
-    #     @executions=0,
-    #     @job_id="1382cb98-c518-46ca-a0cc-d831e11a0714",
-    #     @locale="en",
-    #     @priority=nil,
-    #     @provider_job_id=nil,
-    #     @queue_name="default",
-    #     @scheduled_at=nil,
-    #     @serialized_arguments=["foo"],
-    #     @timezone="UTC">]
-    #
-    # @param pp [PP]
-    # @return [NilClass]
-    def pretty_print(pp)
-      pp.pp(to_a)
-    end
-  end
-end