marj 4.0.0 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4eb6129fa4948ed6e108469f44f380cda908620c6d08392cfa1e8c3bc84a9875
-  data.tar.gz: 28faed9a10fa3a8c885b521d6c3465196f814e97af07018d5eea88e580f322a9
+  metadata.gz: 450ae15f6cf892215323b4eeeccdbf8a75c2db6162b0a75378c411f6e0bdac9c
+  data.tar.gz: 5b1fabd1000b8439e158aabc943deef507045252efe6805cd26a89629e39efe0
 SHA512:
-  metadata.gz: 5f984f366a9cefb3187cbac7129144c8241d46d00357d719ec442dfce62d5c833c95ec30363a5af867e111499cc91cd2c60768da55d04c9c7fc401cce8ac3f83
-  data.tar.gz: e86029649f11ca05f96efcae59c1c60212b29335d9b154c3288a15aa81cd432b46f960166680b5bb0a534346852ccc73970cbaf8a298eb52893c37e260854f61
+  metadata.gz: 1e49dcbc43034f454fb57bf1dca72972eba67719cd8c9aa1aa43b10f91287d2bab41bb97025dbca688de9e6630f0c0d494e6607100382165f56a0f93a38fc5e4
+  data.tar.gz: d075bd8f29864ea83be5ce0099f925f9bb527f9373940427c98cfce3545ee25a565a1fb6293023938bbd39b320ab55015b924281a31bdabb43085398551342cc

data/README.md

@@ -2,9 +2,14 @@
 
 A minimal database-backed ActiveJob queueing backend.
 
+## Purpose
+
+To provide a database-backed ActiveJob queueing backend with as few features
+as possible and the minimum backend-specific API required.
+
 ## Quick Links
 
-API docs: https://gemdocs.org/gems/marj/latest <br>
+API docs: https://gemdocs.org/gems/marj/5.0.0/ <br>
 RubyGems: https://rubygems.org/gems/marj <br>
 Changelog: https://github.com/nicholasdower/marj/releases <br>
 Issues: https://github.com/nicholasdower/marj/issues <br>
@@ -12,34 +17,57 @@ Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
 
 ## Features
 
+### Provided
+
 - Enqueued jobs are written to the database.
 - Successfully executed jobs are deleted from the database.
 - Failed jobs which should be retried are updated in the database.
 - Failed jobs which should not be retried are deleted from the database.
-- An interface is provided to retrieve, execute, discard and re-enqueue jobs.
+- An method is provided to query enqueued jobs.
+- An method is provided to discard enqueued jobs.
 - An `ActiveRecord` class is provided to query the database directly.
 
-## Features Not Provided
+### Not Provided
 
-- Workers
+- Automatic job execution
 - Timeouts
-- Concurrency Controls
+- Concurrency controls
 - Observability
-- A User Interace
+- A user interace
+
+Note that because Marj does not automatically execute jobs, clients are
+responsible for retrieving and either executing or discarding jobs.
+
+## API
+
+The ActiveJob API already provides methods for enqueueing and performing jobs:
+
+```ruby
+queue_adapter.enqueue(job)  # Enqueue
+job.enqueue                 # Enqueue
+job.perform_now             # Perform
+```
+
+Marj works with these existing methods and additionally extends the ActiveJob API
+with methods for querying and discarding jobs:
+
+```ruby
+queue_adapter.query(args)   # Query
+SomeJob.query(args)         # Query
+queue_adapter.discard(job)  # Discard
+job.discard                 # Discard
+```
 
 ## Setup
 
 ### 1. Install
 
 ```shell
-bundle add activejob activerecord marj
-
-# or
-
-gem install activejob activerecord marj
+bundle add activejob activerecord marj  # via Bundler
+gem install activejob activerecord marj # or globally
 ```
 
-### 3. Create the database table
+### 2. Create the database table
 
 ```ruby
 class CreateJobs < ActiveRecord::Migration[7.1]
@@ -68,10 +96,7 @@ class CreateJobs < ActiveRecord::Migration[7.1]
 end
 ```
 
-Note that by default, Marj uses a table named `jobs`. To override the default
-table name, set `Marj.table_name` before loading `ActiveRecord`.
-
-### 4. Configure the queue adapter
+### 3. Configure the queue adapter
 
 ```ruby
 require 'marj'
@@ -81,137 +106,67 @@ ActiveJob::Base.queue_adapter = :marj                # Globally, without Rails
 SomeJob.queue_adapter = :marj                        # Single job
 ```
 
-## Example Usage
-
-```ruby
-# Enqueue and manually run a job:
-job = SomeJob.perform_later('foo')
-job.perform_now
-
-# Retrieve and execute a job
-Marj.due.next.perform_now
-
-# Run all due jobs (single DB query)
-Marj.due.perform_all
-
-# Run all due jobs (multiple DB queries)
-Marj.due.perform_all(batch_size: 1)
-
-# Run all due jobs in a specific queue:
-Marj.queue('foo').due.perform_all
-
-# Run jobs as they become due:
-loop do
-  Marj.due.perform_all rescue logger.error($!)
-ensure
-  sleep 5.seconds
-end
-```
-
-# Jobs Interface
-
-The `Marj` module provides methods for interacting with enqueued jobs. These
-methods accept, return and yield +ActiveJob+ objects rather than +ActiveRecord+
-objects. Returned jobs are orderd by due date. To query the database directly,
-use `Marj::Record`.
+### 4. Include the Marj module (optional)
 
-Example usage:
+By default, jobs can be queried and discarded via the `MarjAdapter` or the
+`Marj` module:
 
 ```ruby
-Marj.all         # Returns all enqueued jobs.
-Marj.queue       # Returns jobs in the specified queue(s).
-Marj.due         # Returns jobs which are due to be executed.
-Marj.next        # Returns the next job(s) to be executed.
-Marj.count       # Returns the number of enqueued jobs.
-Marj.where       # Returns jobs matching the specified criteria.
-Marj.perform_all # Executes all jobs.
-Marj.discard_all # Discards all jobs.
-Marj.discard     # Discards the specified job.
+Marj.query(:all)
+ActiveJob::Base.queue_adapter.query(:all)
+Marj.discard(job)
+ActiveJob::Base.queue_adapter.discard(job)
 ```
 
-Query methods can also be chained:
+But it is also convenient to query or discard via job classes:
 
 ```ruby
-Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
+ApplicationJob.query(:all)
+SomeJob.query(:all)
+ApplicationJob.discard(job)
+SomeJob.discard(job)
+job.discard
 ```
 
-# Custom Jobs Interface
-
-The `Marj::JobsInterface` can be added to any class or module. For example, to
-add it to all jobs classes:
+In order to enable this functionality, you must include the `Marj` module:
 
 ```ruby
 class ApplicationJob < ActiveJob::Base
-  extend Marj::JobsInterface
-
-  def self.all
-    Marj::Relation.new(
-      self == ApplicationJob ?
-        Marj::Record.ordered : Marj::Record.where(job_class: self)
-   )
-  end
+  include Marj
 end
 
-class SomeJob < ApplicationJob; end
-
-ApplicationJob.due # Returns all jobs which are due to be executed.
-SomeJob.due        # Returns SomeJobs which are due to be executed.
+class SomeJob < ApplicationJob
+  def perform; end
+end
 ```
 
-## Customization
-
-It is possible to create a custom record class and jobs interface. This enables,
-for instance, writing jobs to multiple databases/tables within a single
-application.
+## Example Usage
 
 ```ruby
-class CreateMyJobs < ActiveRecord::Migration[7.1]
-  def self.up
-    create_table :my_jobs, id: :string, primary_key: :job_id do |table|
-      table.string   :job_class,            null: false
-      table.text     :arguments,            null: false
-      table.string   :queue_name,           null: false
-      table.integer  :priority
-      table.integer  :executions,           null: false
-      table.text     :exception_executions, null: false
-      table.datetime :enqueued_at,          null: false
-      table.datetime :scheduled_at
-      table.string   :locale,               null: false
-      table.string   :timezone,             null: false
-    end
+# Enqueue and manually run a job
+job = SomeJob.perform_later('foo')
+job.perform_now
 
-    add_index :my_jobs, %i[enqueued_at]
-    add_index :my_jobs, %i[scheduled_at]
-    add_index :my_jobs, %i[priority scheduled_at enqueued_at]
-  end
+# Retrieve and execute a job
+Marj.query(:due, :first).perform_now
 
-  def self.down
-    drop_table :my_jobs
-  end
-end
+# Run all due jobs (single DB query)
+Marj.query(:due).map(&:perform_now)
 
-class MyRecord < Marj::Record
-  self.table_name = 'my_jobs'
+# Run all due jobs (multiple DB queries)
+loop do
+  break unless Marj.query(:due, :first)&.tap(&:perform_now)
 end
 
-CreateMyJobs.migrate(:up)
+# Run all jobs in a specific queue which are due to be executed
+Marj.query(:due, queue: :foo).map(&:perform_now)
 
-class MyJob < ActiveJob::Base
-  self.queue_adapter = MarjAdapter.new('MyRecord')
-
-  extend Marj::JobsInterface
-
-  def self.all
-    Marj::Relation.new(MyRecord.all)
-  end
-
-  def perform(msg)
-    puts msg
-  end
+# Run jobs as they become due
+loop do
+  Marj.query(:due).each(&:perform_now) rescue logger.error($!)
+ensure
+  sleep 5.seconds
 end
-
-MyJob.perform_later('oh, hi')
-MyJob.due.next.perform_now
 ```
 
 ## Testing
@@ -291,6 +246,57 @@ class ApplicationJob < ActiveJob::Base
 end
 ```
 
+### Multiple Tables/Databases
+
+It is possible to create a custom record class in order to, for instance,
+write jobs to multiple databases/tables within a single application.
+
+```ruby
+class CreateMyJobs < ActiveRecord::Migration[7.1]
+  def self.up
+    create_table :my_jobs, id: :string, primary_key: :job_id do |table|
+      table.string   :job_class,            null: false
+      table.text     :arguments,            null: false
+      table.string   :queue_name,           null: false
+      table.integer  :priority
+      table.integer  :executions,           null: false
+      table.text     :exception_executions, null: false
+      table.datetime :enqueued_at,          null: false
+      table.datetime :scheduled_at
+      table.string   :locale,               null: false
+      table.string   :timezone,             null: false
+    end
+
+    add_index :my_jobs, %i[enqueued_at]
+    add_index :my_jobs, %i[scheduled_at]
+    add_index :my_jobs, %i[priority scheduled_at enqueued_at]
+  end
+
+  def self.down
+    drop_table :my_jobs
+  end
+end
+
+class MyRecord < Marj::Record
+  self.table_name = 'my_jobs'
+end
+
+CreateMyJobs.migrate(:up)
+
+class MyJob < ActiveJob::Base
+  self.queue_adapter = MarjAdapter.new('MyRecord')
+
+  include Marj
+
+  def perform(msg)
+    puts msg
+  end
+end
+
+MyJob.perform_later('oh, hi')
+MyJob.query(:due, :first).perform_now
+```
+
 ## ActiveJob Cheatsheet
 
 For more information on ActiveJob, see:

data/lib/marj/record.rb

@@ -6,7 +6,7 @@ require 'active_record'
 module Marj
   # The default +ActiveRecord+ class.
   class Record < ActiveRecord::Base
-    self.table_name = Marj.table_name
+    self.table_name = :jobs
 
     # Order by +enqueued_at+ rather than +job_id+ (the default).
     self.implicit_order_column = 'enqueued_at'
@@ -50,9 +50,47 @@ module Marj
     # discarded.
     #
     # @return [ActiveJob::Base]
-    def as_job
-      Marj.send(:to_job, self)
+    def to_job
+      # See register_callbacks for details on how callbacks are used.
+      job = job_class.new.tap { register_callbacks(_1) }
+
+      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
+      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
+      job_data = attributes.merge('arguments' => JSON.parse(read_attribute_before_type_cast(:arguments)))
+
+      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
+      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
+
+      job.deserialize(job_data)
+
+      # ActiveJob deserializes arguments on demand when a job is performed. Until then they are empty. That's strange.
+      # Instead, deserialize them now. Also, clear `serialized_arguments` to prevent ActiveJob from overwriting changes
+      # to arguments when serializing later.
+      job.arguments = arguments
+      job.serialized_arguments = nil
+
+      job
+    end
+
+    # Registers callbacks for the given job which destroy this record when the job succeeds or is discarded.
+    #
+    # @param job [ActiveJob::Base]
+    # @return [ActiveJob::Base]
+    def register_callbacks(job)
+      raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
+
+      record = self
+      # We need to detect three cases:
+      #  - If a job succeeds, after_perform will be called.
+      #  - If a job fails and should be retried, enqueue will be called. This is handled by the queue adapter.
+      #  - If a job exceeds its max attempts, after_discard will be called.
+      job.singleton_class.after_perform { |_j| record.destroy! }
+      job.singleton_class.after_discard { |_j, _exception| record.destroy! }
+      job.singleton_class.instance_variable_set(:@record, record)
+
+      job
     end
+    private :register_callbacks
 
     class << self
       # Returns an +ActiveRecord::Relation+ scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in
@@ -63,11 +101,16 @@ module Marj
         where('scheduled_at IS NULL OR scheduled_at <= ?', Time.now.utc)
       end
 
-      # Returns an +ActiveRecord::Relation+ scope for jobs ordered by +priority+ (+null+ last), then +scheduled_at+
-      # (+null+ last), then +enqueued_at+.
+      # Returns an +ActiveRecord::Relation+ scope for jobs ordered by due date.
+      #
+      # Jobs are ordered by the following criteria, in order:
+      # 1. past or null scheduled_at before future scheduled_at
+      # 2. ascending priority, nulls last
+      # 3. ascending scheduled_at, nulls last
+      # 4. ascending enqueued_at
       #
       # @return [ActiveRecord::Relation]
-      def ordered
+      def by_due_date
         order(
           Arel.sql(<<~SQL.squish, Time.now.utc)
             CASE WHEN scheduled_at IS NULL OR scheduled_at <= ? THEN 0 ELSE 1 END,

data/lib/marj.rb

@@ -1,156 +1,94 @@
 # frozen_string_literal: true
 
 require_relative 'marj_adapter'
-require_relative 'marj/jobs_interface'
-require_relative 'marj/relation'
 
 # A minimal database-backed ActiveJob queueing backend.
 #
-# The {Marj} module provides methods for interacting with enqueued jobs. These methods accept, return and yield
-# +ActiveJob+ objects rather than +ActiveRecord+ objects. Returned jobs are ordered by due date. To query the database
-# directly, use {Record}.
+# The {Marj} module provides the following methods:
+# - +query+ - Queries enqueued jobs
+# - +discard+ - Discards a job
+#
+# It is possible to call the above methods on the {Marj} module itself or on any class which includes it.
 #
 # Example usage:
-#   Marj.all         # Returns all enqueued jobs.
-#   Marj.queue       # Returns jobs in the specified queue(s).
-#   Marj.due         # Returns jobs which are due to be executed.
-#   Marj.next        # Returns the next job(s) to be executed.
-#   Marj.count       # Returns the number of enqueued jobs.
-#   Marj.where       # Returns jobs matching the specified criteria.
-#   Marj.perform_all # Executes all jobs.
-#   Marj.discard_all # Discards all jobs.
-#   Marj.discard     # Discards the specified job.
+#   Marj.query(:first) # Returns the first job
+#   Marj.discard(job)  # Discards the specified job
+#
+#   class ApplicationJob < ActiveJob::Base
+#     include Marj
+#   end
 #
-# Query methods can also be chained:
-#   Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
+#   class SomeJob < ApplicationJob;
+#     def perform; end
+#   end
 #
-# Note that by default, Marj uses {Marj::Record} to interact with the +jobs+ table. To use a different record class, set
-# {record_class}. To simply override the table name, set {table_name} before loading +ActiveRecord+.
+#   job = ApplicationJob.query(:first) # Returns the first enqueued job
+#   job = SomeJob.query(:first)        # Returns the first enqueued job with job_class SomeJob
+#   ApplicationJob.discard(job)        # Discards the specified job
+#   job.discard                        # Discards the job
 #
 # See https://github.com/nicholasdower/marj
 module Marj
   # The Marj version.
-  VERSION = '4.0.0'
+  VERSION = '5.0.0'
 
   Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
 
-  @table_name = :jobs
-  @record_class = 'Marj::Record'
-
-  class << self
-    include Marj::JobsInterface
-
-    # @!attribute record_class
-    #   The name of the +ActiveRecord+ class. Defaults to +Marj::Record+.
-    #   @return [Class, String]
-
-    attr_writer :record_class
-
-    def record_class
-      @record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
-    end
-
-    # @!attribute table_name
-    #   The name of the database table. Defaults to +:jobs+.
-    #   @return [Symbol, String]
-    attr_accessor :table_name
-
-    # Returns a {Marj::Relation} for all jobs in the order they should be executed.
+  # Provides the {query} and {discard} class methods.
+  module ClassMethods
+    # Queries enqueued jobs.
+    #
+    # Similar to +ActiveRecord.where+ with a few additional features.
+    #
+    # Example usage:
+    #   query(:all)             # Delegates to Marj::Record.all
+    #   query(:due)             # Delegates to Marj::Record.due
+    #   query(:all, limit: 10)  # Returns a maximum of 10 jobs
+    #   query(job_class: Foo)   # Returns all jobs with job_class Foo
     #
-    # @return [Marj::Relation]
-    def all
-      Marj::Relation.new(Marj.record_class.ordered)
+    #   query('123')            # Returns the job with id '123' or nil if no such job exists
+    #   query(id: '123')        # Same as above
+    #   query(job_id: '123')    # Same as above
+    #
+    #   query(queue: 'foo')     # Returns all jobs in the 'foo' queue
+    #   query(job_queue: 'foo') # Same as above
+    def query(*args, **kwargs)
+      kwargs[:job_class] ||= self if self < ActiveJob::Base && name != 'ApplicationJob'
+      queue_adapter.query(*args, **kwargs)
     end
 
     # Discards the specified job.
     #
-    # @return [Integer] the number of discarded jobs
+    # @return [ActiveJob::Base] the discarded job
     def discard(job)
-      all.where(job_id: job.job_id).discard_all
-    end
-
-    private
-
-    # Creates a job instance for the given record which will update the database when successfully executed, enqueued or
-    # discarded.
-    #
-    # @param record [ActiveRecord::Base]
-    # @return [ActiveJob::Base] the new job instance
-    def to_job(record)
-      # See register_callbacks for details on how callbacks are used.
-      job = record.job_class.new.tap { register_callbacks(_1, record) }
-
-      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
-      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
-      job_data = record.attributes.merge('arguments' => JSON.parse(record.read_attribute_before_type_cast(:arguments)))
-
-      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
-      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
-
-      job.tap { job.deserialize(job_data) }
+      queue_adapter.discard(job)
     end
+  end
 
-    # Registers callbacks for the given job which destroy the given database record when the job succeeds or is
-    # discarded.
-    #
-    # @param job [ActiveJob::Base]
-    # @param record [ActiveRecord::Base]
-    # @return [ActiveJob::Base]
-    def register_callbacks(job, record)
-      raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
+  # (see ClassMethods#query)
+  def self.query(*args, **kwargs)
+    queue_adapter.query(*args, **kwargs)
+  end
 
-      # We need to detect three cases:
-      #  - If a job succeeds, after_perform will be called.
-      #  - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
-      #  - If a job exceeds its max attempts, after_discard will be called.
-      job.singleton_class.after_perform { |_j| record.destroy! }
-      job.singleton_class.after_discard { |_j, _exception| record.destroy! }
-      job.singleton_class.instance_variable_set(:@record, record)
+  # (see ClassMethods#discard)
+  def self.discard(job)
+    queue_adapter.discard(job)
+  end
 
-      job
-    end
+  # Discards this job.
+  #
+  # @return [ActiveJob::Base] this job
+  def discard
+    self.class.queue_adapter.discard(self)
+  end
 
-    # Enqueue a job for execution at the specified time.
-    #
-    # @param job [ActiveJob::Base] the job to enqueue
-    # @param record_class [Class] the +ActiveRecord+ class
-    # @param time [Time, NilClass] optional time at which to execute the job
-    # @return [ActiveJob::Base] the enqueued job
-    def enqueue(job, record_class, time = nil)
-      job.scheduled_at = time
-      # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
-      serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
+  def self.included(clazz)
+    clazz.extend(ClassMethods)
+  end
+  private_class_method :included
 
-      # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
-      # are registered on the job instance so that when the job is executed, the database record is deleted or updated
-      # (depending on the result).
-      #
-      # We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
-      # singleton class. This holds a reference to the record. This ensures that if execute is called on a record
-      # instance, any updates to the database are reflected on that record instance.
-      if (existing_record = job.singleton_class.instance_variable_get(:@record))
-        # This job instance has already been associated with a database row.
-        if record_class.exists?(job_id: job.job_id)
-          # The database row still exists, we simply need to update it.
-          existing_record.update!(serialized)
-        else
-          # Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
-          # want to register the new instance because someone might still have a reference to the existing one.
-          record_class.create!(serialized)
-          existing_record.reload
-        end
-      else
-        # This job instance has not been associated with a database row.
-        if (new_record = record_class.find_by(job_id: job.job_id))
-          # The database row already exists. Update it.
-          new_record.update!(serialized)
-        else
-          # The database row does not exist. Create it.
-          new_record = record_class.create!(serialized)
-        end
-        register_callbacks(job, new_record)
-      end
-      job
-    end
+  def self.queue_adapter
+    ActiveJob::Base.queue_adapter
   end
+  private_class_method :queue_adapter
 end

data/lib/marj_adapter.rb

@@ -2,8 +2,17 @@
 
 # ActiveJob queue adapter for Marj.
 #
+# In addition to the standard +ActiveJob+ queue adapter API, this adapter provides a +query+ method which can be used to
+# query enqueued jobs and a +discard+ method which can be used to discard enqueued jobs.
+#
+# Although it is possible to access the adapter directly in order to query or discard, it is recommended to use the
+# {Marj} module.
+#
 # See https://github.com/nicholasdower/marj
 class MarjAdapter
+  JOB_ID_REGEX = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.freeze
+  private_constant :JOB_ID_REGEX
+
   # Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ class.
   #
   # @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use to store jobs
@@ -16,7 +25,7 @@ class MarjAdapter
   # @param job [ActiveJob::Base] the job to enqueue
   # @return [ActiveJob::Base] the enqueued job
   def enqueue(job)
-    Marj.send(:enqueue, job, record_class)
+    enqueue_at(job)
   end
 
   # Enqueue a job for execution at the specified time.
@@ -24,8 +33,115 @@ class MarjAdapter
   # @param job [ActiveJob::Base] the job to enqueue
   # @param timestamp [Numeric, NilClass] optional number of seconds since Unix epoch at which to execute the job
   # @return [ActiveJob::Base] the enqueued job
-  def enqueue_at(job, timestamp)
-    Marj.send(:enqueue, job, record_class, timestamp ? Time.at(timestamp).utc : nil)
+  def enqueue_at(job, timestamp = nil)
+    job.scheduled_at = timestamp ? Time.at(timestamp).utc : nil
+
+    # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
+    serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
+
+    # Serialize sets locale to I18n.locale.to_s and enqueued_at to Time.now.utc.iso8601(9).
+    # Update the job to reflect what is being enqueued.
+    job.locale = serialized[:locale]
+    job.enqueued_at = Time.iso8601(serialized[:enqueued_at]).utc
+
+    # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
+    # are registered on the job instance so that when the job is executed, the database record is deleted or updated
+    # (depending on the result).
+    #
+    # We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
+    # singleton class. This holds a reference to the record. This ensures that if execute is called on a record
+    # instance, any updates to the database are reflected on that record instance.
+    if (existing_record = job.singleton_class.instance_variable_get(:@record))
+      # This job instance has already been associated with a database row.
+      if record_class.exists?(job_id: job.job_id)
+        # The database row still exists, we simply need to update it.
+        existing_record.update!(serialized)
+      else
+        # Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
+        # want to register the new instance because someone might still have a reference to the existing one.
+        record_class.create!(serialized)
+        existing_record.reload
+      end
+    else
+      # This job instance has not been associated with a database row.
+      if (new_record = record_class.find_by(job_id: job.job_id))
+        # The database row already exists. Update it.
+        new_record.update!(serialized)
+      else
+        # The database row does not exist. Create it.
+        new_record = record_class.create!(serialized)
+      end
+      new_record.send(:register_callbacks, job)
+    end
+    job
+  end
+
+  # Queries enqueued jobs. Similar to +ActiveRecord.where+ with a few additional features:
+  # - Leading symbol arguments are treated as +ActiveRecord+ scopes.
+  # - If only a job ID is specified, the corresponding job is returned.
+  # - If +:limit+ is specified, the maximum number of jobs is limited.
+  #
+  # Example usage:
+  #   query(:all)             # Delegates to Marj::Record.all
+  #   query(:due)             # Delegates to Marj::Record.due
+  #   query(:all, limit: 10)  # Returns a maximum of 10 jobs
+  #   query(job_class: Foo)   # Returns all jobs with job_class Foo
+  #
+  #   query('123')            # Returns the job with id '123' or nil if no such job exists
+  #   query(id: '123')        # Same as above
+  #   query(job_id: '123')    # Same as above
+  #
+  #   query(queue: 'foo')     # Returns all jobs in the 'foo' queue
+  #   query(job_queue: 'foo') # Same as above
+  def query(*args, **kwargs)
+    args, kwargs = args.dup, kwargs.dup.symbolize_keys
+    kwargs = kwargs.merge(queue_name: kwargs.delete(:queue)) if kwargs.key?(:queue)
+    kwargs = kwargs.merge(job_id: kwargs.delete(:id)) if kwargs.key?(:id)
+    kwargs[:job_id] = args.shift if args.size == 1 && args.first.is_a?(String) && args.first.match(JOB_ID_REGEX)
+
+    if args.empty? && kwargs.size == 1 && kwargs.key?(:job_id)
+      return record_class.find_by(job_id: kwargs[:job_id])&.to_job
+    end
+
+    symbol_args = []
+    symbol_args << args.shift while args.first.is_a?(Symbol)
+    order_by = kwargs.delete(:order)
+    order_by = :queue_name if [:queue, 'queue'].include?(order_by)
+    limit = kwargs.delete(:limit)
+    symbol_args.shift if symbol_args.first == :all
+    relation = record_class.all
+    relation = relation.order(order_by) if order_by
+    relation = relation.by_due_date unless relation.order_values.any?
+    relation = relation.where(*args, **kwargs) if args.any? || kwargs.any?
+    relation = relation.limit(limit) if limit
+    relation = relation.send(symbol_args.shift) while symbol_args.any?
+    if relation.is_a?(Enumerable)
+      relation.map(&:to_job)
+    elsif relation.is_a?(record_class)
+      relation.to_job
+    else
+      relation
+    end
+  end
+
+  # Discards the specified job.
+  #
+  # @return [ActiveJob::Base] the discarded job
+  def discard(job)
+    record = job.singleton_class.instance_variable_get(:@record)
+    record ||= Marj::Record.find_by(job_id: job.job_id)&.tap { _1.send(:register_callbacks, job) }
+    record&.destroy
+
+    # Copied from ActiveJob::Exceptions#run_after_discard_procs
+    exceptions = []
+    job.after_discard_procs.each do |blk|
+      instance_exec(job, nil, &blk)
+    rescue StandardError => e
+      exceptions << e
+    end
+    raise exceptions.last if exceptions.any?
+
+    job
   end
 
   private

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: marj
 version: !ruby/object:Gem::Version
-  version: 4.0.0
+  version: 5.0.0
 platform: ruby
 authors:
 - Nick Dower
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-01-28 00:00:00.000000000 Z
+date: 2024-01-31 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -48,17 +48,15 @@ files:
 - LICENSE.txt
 - README.md
 - lib/marj.rb
-- lib/marj/jobs_interface.rb
 - lib/marj/record.rb
-- lib/marj/relation.rb
 - lib/marj_adapter.rb
 homepage: https://github.com/nicholasdower/marj
 licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/nicholasdower/marj/issues
-  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v4.0.0
-  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v4.0.0
+  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v5.0.0
+  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v5.0.0
   homepage_uri: https://github.com/nicholasdower/marj
   rubygems_mfa_required: 'true'
   source_code_uri: https://github.com/nicholasdower/marj

data/lib/marj/jobs_interface.rb

@@ -1,113 +0,0 @@
-# frozen_string_literal: true
-
-module Marj
-  # The interface provided by {Marj} and {Marj::Relation}.
-  #
-  # To create a custom jobs interface, for example for all job classes in your application:
-  #   class ApplicationJob < ActiveJob::Base
-  #     extend Marj::JobsInterface
-  #
-  #     def self.all
-  #       Marj::Relation.new(self == ApplicationJob ? Marj::Record.ordered : Marj::Record.where(job_class: self))
-  #     end
-  #   end
-  #
-  #   class SomeJob < ApplicationJob
-  #     def perform(msg)
-  #       puts msg
-  #     end
-  #   end
-  #
-  # This will allow you to query jobs via the +ApplicationJob+ class:
-  #   ApplicationJob.next # Returns the next job of any type
-  #
-  # Or to query jobs via a specific job class:
-  #   SomeJob.next # Returns the next SomeJob
-  #
-  # Alternatively, to create a jobs interface for a single job class:
-  #   class SomeJob < ActiveJob::Base
-  #     extend Marj::JobsInterface
-  #
-  #     def self.all
-  #       Marj::Relation.new(Marj::Record.where(job_class: self).ordered)
-  #     end
-  #   end
-  module JobsInterface
-    def self.included(clazz)
-      return if clazz == Marj::Relation
-
-      clazz.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
-    end
-    private_class_method :included
-
-    def self.extended(clazz)
-      clazz.singleton_class.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
-    end
-    private_class_method :extended
-
-    # Returns a {Marj::Relation} for jobs in the specified queue(s).
-    #
-    # @param queue [String, Symbol] the queue to query
-    # @param queues [Array<String>, Array<Symbol>] more queues to query
-    # @return [Marj::Relation]
-    def queue(queue, *queues)
-      Marj::Relation.new(all.where(queue_name: queues.dup.unshift(queue)))
-    end
-
-    # Returns the next job or the next N jobs if +limit+ is specified. If no jobs exist, returns +nil+.
-    #
-    # @param limit [Integer, NilClass]
-    # @return [ActiveJob::Base, NilClass]
-    def next(limit = nil)
-      all.first(limit)&.then { _1.is_a?(Array) ? _1.map(&:as_job) : _1.as_job }
-    end
-
-    # Returns a count of jobs, optionally either matching the specified column name criteria or where the specified
-    # block returns +true+.
-    #
-    # @param column_name [String, Symbol, NilClass]
-    # @param block [Proc, NilClass]
-    # @return [Integer]
-    def count(column_name = nil, &block)
-      block_given? ? all.count(column_name) { |r| block.call(r.as_job) } : all.count(column_name)
-    end
-
-    # Returns a {Marj::Relation} for jobs matching the specified criteria.
-    #
-    # @param args [Array]
-    # @return [Marj::Relation]
-    def where(*args)
-      Marj::Relation.new(all.where(*args))
-    end
-
-    # Returns a {Marj::Relation} for enqueued jobs with a +scheduled_at+ that is either +null+ or in the past.
-    #
-    # @return [Marj::Relation]
-    def due
-      Marj::Relation.new(all.due)
-    end
-
-    # Calls +perform_now+ on each job.
-    #
-    # @param batch_size [Integer, NilClass] the number of jobs to fetch at a time, or +nil+ to fetch all jobs at once
-    # @return [Array] the results returned by each job
-    def perform_all(batch_size: nil)
-      if batch_size
-        [].tap do |results|
-          while (jobs = all.limit(batch_size).map(&:as_job)).any?
-            results.concat(jobs.map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } })
-          end
-        end
-      else
-        all.map(&:as_job).map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } }
-      end
-    end
-
-    # Discards all jobs.
-    #
-    # @return [Numeric] the number of discarded jobs
-    def discard_all
-      all.delete_all
-    end
-  end
-end

data/lib/marj/relation.rb

@@ -1,72 +0,0 @@
-# frozen_string_literal: true
-
-require_relative 'jobs_interface'
-
-module Marj
-  # Returned by {Marj::JobsInterface} query methods to enable chaining and +Enumerable+ methods.
-  class Relation
-    include Enumerable
-    include Marj::JobsInterface
-
-    attr_reader :all
-    private :all
-
-    # Returns a {Marj::Relation} which wraps the specified +ActiveRecord+ relation.
-    def initialize(ar_relation)
-      @all = ar_relation
-    end
-
-    # Yields each job in this relation.
-    #
-    # @param block [Proc]
-    # @return [Array] the jobs in this relation
-    def each(&block)
-      all.map(&:as_job).each(&block)
-    end
-
-    # Provides +pretty_inspect+ output containing arrays of jobs rather than arrays of records, similar to the output
-    # produced when calling +pretty_inspect+ on +ActiveRecord::Relation+.
-    #
-    # Instead of the default +pretty_inspect+ output:
-    #   > Marj.all
-    #    =>
-    #   #<Marj::Relation:0x000000012728bd88
-    #    @ar_relation=
-    #     [#<Marj::Record:0x0000000126c42080
-    #       job_id: "1382cb98-c518-46ca-a0cc-d831e11a0714",
-    #       job_class: TestJob,
-    #       arguments: ["foo"],
-    #       queue_name: "default",
-    #       priority: nil,
-    #       executions: 0,
-    #       exception_executions: {},
-    #       enqueued_at: 2024-01-25 15:31:06.115773 UTC,
-    #       scheduled_at: nil,
-    #       locale: "en",
-    #       timezone: "UTC">]>
-    #
-    # Produces:
-    #   > Marj.all
-    #    =>
-    #   [#<TestJob:0x000000010b63cef8
-    #     @_scheduled_at_time=nil,
-    #     @arguments=[],
-    #     @enqueued_at=2024-01-25 15:31:06 UTC,
-    #     @exception_executions={},
-    #     @executions=0,
-    #     @job_id="1382cb98-c518-46ca-a0cc-d831e11a0714",
-    #     @locale="en",
-    #     @priority=nil,
-    #     @provider_job_id=nil,
-    #     @queue_name="default",
-    #     @scheduled_at=nil,
-    #     @serialized_arguments=["foo"],
-    #     @timezone="UTC">]
-    #
-    # @param pp [PP]
-    # @return [NilClass]
-    def pretty_print(pp)
-      pp.pp(to_a)
-    end
-  end
-end