marj 2.1.0 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '078c3d48b27784e09425ba8eda2207038d6ce5c7a711953e8567c76aac689435'
-  data.tar.gz: c129b4848de1e2fe55de0de3a4893586a052cd814df798879dc196a81c1a94d5
+  metadata.gz: 7afb73261695ca8eb0005d59b292ec659a9c10124a2b3984242e3f8e209fc897
+  data.tar.gz: 5f5fea852ab9a8e7eb9ca6fa59ed6cb4e0537651bf3edf6713ec0cad4a1c1ed0
 SHA512:
-  metadata.gz: 4cef48fb7c862123f4a5ad7ed5bf7a9d95eb3d7036831db79ad3dc9cf94f074a112ce58f0189484f991a52185cdf8e8ceed48dd9237bf4a04d5e3bbe30b2d274
-  data.tar.gz: 7631b1a0fef1ece1edcf15541dc85d25b2d5d3ba1cb8ca716133975126e5ce9bb4f28d627a4c2a5d751029ce8f67ba4db47c6fa9d197467fade4c09d20a04716
+  metadata.gz: 348506cb34c7956e4c17554223c81d47353905ebdfb684c965d4deeb2b3a47727b5cb971211e2f2c48e039c8c5c1d9a1b02a6e9647be247ca0d86472a357f1b0
+  data.tar.gz: 94069a23d1882ff62c5b630e5e50357c63f230f8fc4a86828d3e4d785402f6c325fb82fb2d398c5ec937a572018b8c4027f5569fb818dcbc1dd2258e1cc9ac54

data/README.md

@@ -1,6 +1,6 @@
 # Marj - Minimal ActiveRecord Jobs
 
-The simplest database-backed ActiveJob queueing backend.
+A minimal database-backed ActiveJob queueing backend.
 
 ## Quick Links
 
@@ -16,6 +16,8 @@ Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
 - Successfully executed jobs are deleted from the database.
 - Failed jobs which should be retried are updated in the database.
 - Failed jobs which should not be retried are deleted from the database.
+- An interface is provided to retrieve, execute, discard and re-enqueue jobs.
+- An `ActiveRecord` model class is provided to query the database directly.
 
 ## Features Not Provided
 
@@ -24,38 +26,20 @@ Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
 - Concurrency Controls
 - Observability
 - A User Interace
-- Anything else you might dream up.
-
-## Interface
-
-- `Marj` - An ActiveRecord model class
-- `Marj.ready` - Used to retrieve jobs ready to be executed
-- `Marj#execute` - Used to execute jobs retrieved from the database
-- `MarjConfig.table_name=` - Used to override the default table name
 
 ## Setup
 
 ### 1. Install
 
 ```shell
-bundle add marj
+bundle add activejob activerecord marj
 
 # or
 
-gem install marj
-```
-
-### 2. Configure
-
-By default, the database table is named "jobs". To use a different table name:
-
-```ruby
-require 'marj'
-
-MarjConfig.table_name = 'some_name'
+gem install activejob activerecord marj
 ```
 
-### 3. Create the database table
+### 3. Create the jobs table
 
 ```ruby
 class CreateJobs < ActiveRecord::Migration[7.1]
@@ -89,18 +73,56 @@ end
 ```ruby
 require 'marj'
 
-# With rails:
-class MyApplication < Rails::Application
-  config.active_job.queue_adapter = :marj
-end
+Rails.configuration.active_job.queue_adapter = :marj # Globally, with Rails
+ActiveJob::Base.queue_adapter = :marj                # Globally, without Rails
+SomeJob.queue_adapter = :marj                        # Single job
+```
+
+## Jobs Interface
+
+`Marj::Jobs` provides a query interface (`Marj::JobsInterface`) which can be
+used to retrieve, execute and discard enqueued jobs. It returns, yields and
+accepts `ActiveJob` objects rather than `ActiveRecord` objects. Jobs are
+orderd by due date. To query the database directly, use `Marj::Record`.
 
-# Without Rails:
-ActiveJob::Base.queue_adapter = :marj
+```ruby
+Marj::Jobs.all         # Returns all enqueued jobs.
+Marj::Jobs.queue       # Returns jobs in the specified queue(s).
+Marj::Jobs.due         # Returns jobs which are due to be executed.
+Marj::Jobs.next        # Returns the next job(s) to be executed.
+Marj::Jobs.count       # Returns the number of enqueued jobs.
+Marj::Jobs.where       # Returns jobs matching the specified criteria.
+Marj::Jobs.perform_all # Executes all jobs.
+Marj::Jobs.discard_all # Discards all jobs.
+Marj::Jobs.discard     # Discards the specified job.
+```
+
+`all`, `queue`, `due` and `where` return a `Marj::Relation` which provides
+the same `Marj::JobsInterface`. This can be used to chain query methods like:
+
+```ruby
+Marj::Jobs.due.where(job_class: SomeJob).next
+```
 
-# Or for specific jobs (with or without Rails):
-class SomeJob < ActiveJob::Base
-  self.queue_adapter = :marj
+Note that the `Marj::JobsInterface` can be added to any class or module. For
+example, to add it to all jobs classes:
+
+```ruby
+class ApplicationJob < ActiveJob::Base
+  extend Marj::JobsInterface
+
+  def self.all
+    Marj::Relation.new(
+      self == ApplicationJob ?
+        Marj::Record.ordered : Marj::Record.where(job_class: self)
+   )
+  end
 end
+
+class SomeJob < ApplicationJob; end
+
+ApplicationJob.due # Returns all jobs which are due to be executed.
+SomeJob.due        # Returns SomeJobs which are due to be executed.
 ```
 
 ## Example Usage
@@ -110,35 +132,96 @@ end
 job = SomeJob.perform_later('foo')
 job.perform_now
 
-# Enqueue, retrieve and manually run a job:
-SomeJob.perform_later('foo')
-Marj.first.execute
+# Retrieve and execute a job
+Marj::Jobs.due.next.perform_now
 
-# Run all ready jobs:
-Marj.ready.each(&:execute)
+# Run all due jobs (single DB query)
+Marj::Jobs.due.perform_all
 
-# Run all ready jobs, querying each time:
-loop { Marj.ready.first&.tap(&:execute) || break }
+# Run all due jobs (multiple DB queries)
+Marj::Jobs.due.perform_all(batch_size: 1)
 
-# Run all ready jobs in a specific queue:
-loop { Marj.where(queue_name: 'foo').ready.first&.tap(&:execute) || break }
+# Run all due jobs in a specific queue:
+Marj::Jobs.queue('foo').due.perform_all
 
-# Run jobs as they become ready:
+# Run all jobs indefinitely, as they become due:
 loop do
-  loop { Marj.ready.first&.tap(&:execute) || break }
-rescue Exception => e
-  logger.error(e)
+  Marj::Jobs.due.perform_all rescue logger.error($!)
 ensure
   sleep 5.seconds
 end
 ```
 
+## Customization
+
+It is possible to create a custom record class and jobs interface. This enables,
+for instance, writing jobs to multiple databases/tables within a single
+application.
+
+```
+class CreateMyJobs < ActiveRecord::Migration[7.1]
+  def self.up
+    create_table :my_jobs, id: :string, primary_key: :job_id do |table|
+      table.string   :job_class,            null: false
+      table.text     :arguments,            null: false
+      table.string   :queue_name,           null: false
+      table.integer  :priority
+      table.integer  :executions,           null: false
+      table.text     :exception_executions, null: false
+      table.datetime :enqueued_at,          null: false
+      table.datetime :scheduled_at
+      table.string   :locale,               null: false
+      table.string   :timezone,             null: false
+    end
+
+    add_index :my_jobs, %i[enqueued_at]
+    add_index :my_jobs, %i[scheduled_at]
+    add_index :my_jobs, %i[priority scheduled_at enqueued_at]
+  end
+
+  def self.down
+    drop_table :my_jobs
+  end
+end
+
+class MyRecord < ActiveRecord::Base
+  include Marj::RecordInterface
+
+  self.table_name = 'my_jobs'
+end
+
+CreateMyJobs.migrate(:up)
+
+class ApplicationJob < ActiveJob::Base
+  self.queue_adapter = MarjAdapter.new('MyRecord')
+
+  extend Marj::JobsInterface
+
+  def self.all
+    Marj::Relation.new(
+      self == ApplicationJob ?
+        MyRecord.ordered : MyRecord.where(job_class: self)
+    )
+  end
+end
+
+class MyJob < ApplicationJob
+  def perform(msg)
+    puts msg
+  end
+end
+
+MyJob.perform_later('oh, hi')
+MyJob.due.next.perform_now
+```
+
 ## Testing
 
-By default, jobs enqeued during tests will be written to the database. Enqueued jobs can be executed via:
+By default, jobs enqeued during tests will be written to the database. Enqueued
+jobs can be executed via:
 
 ```ruby
-Marj.ready.each(&:execute)
+Marj::Jobs.due.perform_all
 ```
 
 Alternatively, to use [ActiveJob::QueueAdapters::TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
@@ -158,7 +241,9 @@ class ApplicationJob < ActiveJob::Base
 
   around_perform do |job, block|
     if (timeout = job.class.instance_variable_get(:@timeout))
-      ::Timeout.timeout(timeout, StandardError, 'execution expired') { block.call }
+      ::Timeout.timeout(timeout, StandardError, 'execution expired') do
+        block.call
+      end
     else
       block.call
     end
@@ -185,7 +270,9 @@ class ApplicationJob < ActiveJob::Base
   def last_error=(error)
     if error.is_a?(Exception)
       backtrace = error.backtrace&.map { |line| "\t#{line}" }&.join("\n")
-      error = backtrace ? "#{error.class}: #{error.message}\n#{backtrace}" : "#{error.class}: #{error.message}"
+      error = backtrace ?
+        "#{error.class}: #{error.message}\n#{backtrace}" :
+        "#{error.class}: #{error.message}"
     end
 
     @last_error = error&.truncate(10_000, omission: '… (truncated)')
@@ -217,10 +304,8 @@ For more information on ActiveJob, see:
 
 ```ruby
 # With Rails
-class MyApplication < Rails::Application
-  config.active_job.queue_adapter = :foo           # Instantiates FooAdapter
-  config.active_job.queue_adapter = FooAdapter.new # Uses FooAdapter directly
-end
+Rails.configuration.active_job.queue_adapter = :foo # Instantiates FooAdapter
+Rails.configuration.active_job.queue_adapter = FooAdapter.new
 
 # Without Rails
 ActiveJob::Base.queue_adapter = :foo               # Instantiates FooAdapter
@@ -237,11 +322,11 @@ SomeJob.queue_adapter = FooAdapter.new             # Uses FooAdapter directly
 - `config.active_job.queue_name_prefix`
 - `config.active_job.queue_name_delimiter`
 - `config.active_job.retry_jitter`
+- `SomeJob.queue_name`
+- `SomeJob.queue_as`
 - `SomeJob.queue_name_prefix`
 - `SomeJob.queue_name_delimiter`
 - `SomeJob.retry_jitter`
-- `SomeJob.queue_name`
-- `SomeJob.queue_as`
 
 ### Options
 
@@ -274,35 +359,40 @@ SomeJob.queue_adapter = FooAdapter.new             # Uses FooAdapter directly
 # Create without enqueueing
 job = SomeJob.new
 job = SomeJob.new(args)
+job = SomeJob.new.deserialize(other_job.serialize)
 
 # Create and enqueue
 job = SomeJob.perform_later
 job = SomeJob.perform_later(args)
 
-# Create and run (enqueued on failure)
+# Create without enqueueing and run (only enqueued on failure if retryable)
 SomeJob.perform_now
 SomeJob.perform_now(args)
 ```
 
 ### Enqueueing Jobs
 
+Jobs are enqueued via the `ActiveJob::Base#enqueue` method. This method returns
+the job on success. If an error is raised during enqueueing, that error will
+propagate to the caller, unless the error is an `ActiveJob::EnqueueError`. In
+this case, `enqueue` will return `false` and `job.enqueue_error` will be set.
+
 ```ruby
 SomeJob.new(args).enqueue
 SomeJob.new(args).enqueue(options)
 
+# Via perform_later
 SomeJob.perform_later(SomeJob.new(args))
-
 SomeJob.perform_later(args)
 SomeJob.set(options).perform_later(args)
 
-# Enqueued on failure
+# After a failure during execution
 SomeJob.perform_now(args)
+ActiveJob::Base.execute(SomeJob.new(args).serialize)
 
 # Enqueue multiple
 ActiveJob.perform_all_later(SomeJob.new, SomeJob.new)
 ActiveJob.perform_all_later(SomeJob.new, SomeJob.new, options:)
-
-# Enqueue multiple
 SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new)
 SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new, options:)
 ```
@@ -310,7 +400,7 @@ SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new, options:)
 ### Executing Jobs
 
 ```ruby
-# Executed without enqueueing, enqueued on failure if retries configured
+# Executed without enqueueing, enqueued on failure if retryable
 SomeJob.new(args).perform_now
 SomeJob.perform_now(args)
 ActiveJob::Base.execute(SomeJob.new(args).serialize)

data/lib/marj/jobs.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require_relative 'jobs_interface'
+require_relative 'relation'
+
+module Marj
+  # Provides methods for querying, performing and discarding jobs. Returns, yields and accepts
+  # +ActiveJob+ objects rather than +ActiveRecord+ objects. To query the database directly, use
+  # {Marj::Record}.
+  #
+  # To create a custom jobs interface, see {Marj::JobsInterface}.
+  module Jobs
+    singleton_class.include Marj::JobsInterface
+
+    # Returns a {Marj::Relation} for all jobs in the order they should be executed.
+    #
+    # @return [Marj::Relation]
+    def self.all
+      Marj::Relation.new(Marj::Record.ordered)
+    end
+
+    # Discards the specified job.
+    #
+    # @return [Integer] the number of discarded jobs
+    def self.discard(job)
+      all.where(job_id: job.job_id).discard_all
+    end
+  end
+end

data/lib/marj/jobs_interface.rb

@@ -0,0 +1,86 @@
+# frozen_string_literal: true
+
+module Marj
+  # The interface provided by {Marj::Jobs} and {Marj::Relation}. Include to create a custom jobs interface.
+  #
+  # To create a jobs interface for all job classes:
+  #   class ApplicationJob < ActiveJob::Base
+  #     extend Marj::JobsInterface
+  #
+  #     def self.all
+  #       Marj::Relation.new(self == ApplicationJob ? Marj::Record.ordered : Marj::Record.where(job_class: self))
+  #     end
+  #   end
+  #
+  #   ApplicationJob.next
+  #   SomeJob.next
+  #
+  # To create a jobs interface for a single job class:
+  #   class SomeJob < ActiveJob::Base
+  #     extend Marj::JobsInterface
+  #
+  #     def self.all
+  #       Marj::Relation.new(Marj::Record.where(job_class: self).ordered)
+  #     end
+  #   end
+  #
+  #   SomeJob.next
+  module JobsInterface
+    # Returns a {Marj::Relation} for jobs in the specified queue(s).
+    #
+    # @param queue [String, Symbol] the queue to query
+    # @param queues [Array<String, Array<Symbol>] more queues to query
+    # @return [Marj::Relation]
+    def queue(queue, *queues)
+      all.queue(queue, *queues)
+    end
+
+    # Returns the next job or the next N jobs if +limit+ is specified. If no jobs exist, returns +nil+.
+    #
+    # @param limit [Integer, NilClass]
+    # @return [ActiveJob::Base, NilClass]
+    def next(limit = nil)
+      all.next(limit)
+    end
+
+    # Returns a count of jobs, optionally either matching the specified column name criteria or where the specified
+    # block returns +true+.
+    #
+    # @param column_name [String, Symbol, NilClass]
+    # @param block [Proc, NilClass]
+    # @return [Integer]
+    def count(column_name = nil, &block)
+      all.count(column_name, &block)
+    end
+
+    # Returns a {Marj::Relation} for jobs matching the specified criteria.
+    #
+    # @param args [Array]
+    # @return [Marj::Relation]
+    def where(*args)
+      all.where(*args)
+    end
+
+    # Returns a {Marj::Relation} for enqueued jobs with a +scheduled_at+ that is either +null+ or in the past.
+    #
+    # @return [Marj::Relation]
+    def due
+      all.due
+    end
+
+    # Calls +perform_now+ on each job.
+    #
+    # @param batch_size [Integer, NilClass] the number of jobs to fetch at a time, or +nil+ to fetch all jobs at once
+    # @return [Array] the results returned by each job
+    def perform_all(batch_size: nil)
+      all.perform_all(batch_size: batch_size)
+    end
+
+    # Discards all jobs.
+    #
+    # @return [Numeric] the number of discarded jobs
+    def discard_all
+      all.discard_all
+    end
+  end
+end

data/lib/marj/record.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require 'active_job'
+require 'active_record'
+require_relative 'record_interface'
+
+module Marj
+  # The Marj ActiveRecord model class.
+  #
+  # See https://github.com/nicholasdower/marj
+  class Record < ActiveRecord::Base
+    include Marj::RecordInterface
+    extend Marj::RecordInterface::ClassMethods # Added explicitly to generate docs
+
+    self.table_name = 'jobs'
+  end
+end

data/lib/marj/record_interface.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+require 'active_job/arguments'
+
+module Marj
+  # Provides base functionality for {Marj::Record}. Can be used to create a custom +ActiveRecord+ model class.
+  #
+  # Example Usage:
+  #   class MyRecord < ActiveRecord::Base
+  #     include Marj::RecordInterface
+  #
+  #     self.table_name = 'my_jobs'
+  #   end
+  module RecordInterface
+    # Adds {ClassMethods}, custom serializers and an implicit order column to the including class.
+    #
+    # @param clazz [Class] the including class
+    def self.included(clazz)
+      clazz.extend(ClassMethods)
+
+      # Order by +enqueued_at+ rather than +job_id+ (the default).
+      clazz.implicit_order_column = 'enqueued_at'
+
+      # Using a custom serializer for exception_executions so that we can interact with it as a hash rather than a
+      # string.
+      clazz.serialize(:exception_executions, coder: JSON)
+
+      # Using a custom serializer for arguments so that we can interact with as an array rather than a string.
+      # This enables code like:
+      #   Marj::Record.next.arguments.first
+      #   Marj::Record.next.update!(arguments: ['foo', 1, Time.now])
+      clazz.serialize(:arguments, coder: Class.new do
+        def self.dump(arguments)
+          return ActiveJob::Arguments.serialize(arguments).to_json if arguments.is_a?(Array)
+          return arguments if arguments.is_a?(String) || arguments.nil?
+
+          raise "invalid arguments: #{arguments}"
+        end
+
+        def self.load(arguments)
+          arguments ? ActiveJob::Arguments.deserialize(JSON.parse(arguments)) : nil
+        end
+      end)
+
+      # Using a custom serializer for job_class so that we can interact with it as a class rather than a string.
+      clazz.serialize(:job_class, coder: Class.new do
+        def self.dump(clazz)
+          return clazz.name if clazz.is_a?(Class)
+          return clazz if clazz.is_a?(String) || clazz.nil?
+
+          raise "invalid class: #{clazz}"
+        end
+
+        def self.load(str)
+          str&.constantize
+        end
+      end)
+    end
+
+    # Class methods for {Marj::RecordInterface}.
+    module ClassMethods
+      # Returns an +ActiveRecord::Relation+ scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in
+      # the past.
+      #
+      # @return [ActiveRecord::Relation]
+      def due
+        where('scheduled_at IS NULL OR scheduled_at <= ?', Time.now.utc)
+      end
+
+      # Returns an +ActiveRecord::Relation+ scope for jobs ordered by +priority+ (+null+ last), then +scheduled_at+
+      # (+null+ last), then +enqueued_at+.
+      #
+      # @return [ActiveRecord::Relation]
+      def ordered
+        order(
+          Arel.sql(<<~SQL.squish, Time.now.utc)
+            CASE WHEN scheduled_at IS NULL OR scheduled_at <= ? THEN 0 ELSE 1 END,
+            CASE WHEN priority IS NULL THEN 1 ELSE 0 END, priority,
+            CASE WHEN scheduled_at IS NULL THEN 1 ELSE 0 END, scheduled_at,
+            enqueued_at
+          SQL
+        )
+      end
+    end
+
+    # Returns a job object for this record which will update the database when successfully executed, enqueued or
+    # discarded.
+    #
+    # @return [ActiveJob::Base]
+    def as_job
+      Marj.send(:to_job, self)
+    end
+  end
+end

data/lib/marj/relation.rb

@@ -0,0 +1,112 @@
+# frozen_string_literal: true
+
+require_relative 'jobs_interface'
+
+module Marj
+  # Returned by {Marj::JobsInterface} query methods to enable chaining and +Enumerable+ methods.
+  class Relation
+    include Enumerable
+    include Marj::JobsInterface
+
+    # Returns a Marj::Relation which wraps the specified +ActiveRecord+ relation.
+    def initialize(ar_relation)
+      @ar_relation = ar_relation
+    end
+
+    # (see Marj::JobsInterface#queue)
+    def queue(queue, *queues)
+      Marj::Relation.new(@ar_relation.where(queue_name: queues.dup.unshift(queue)))
+    end
+
+    # (see Marj::JobsInterface#next)
+    def next(limit = nil)
+      @ar_relation.first(limit)&.then { _1.is_a?(Array) ? _1.map(&:as_job) : _1.as_job }
+    end
+
+    # (see Marj::JobsInterface#count)
+    def count(column_name = nil, &block)
+      block_given? ? @ar_relation.count(column_name) { |r| block.call(r.as_job) } : @ar_relation.count(column_name)
+    end
+
+    # (see Marj::JobsInterface#where)
+    def where(*args)
+      Marj::Relation.new(@ar_relation.where(*args))
+    end
+
+    # (see Marj::JobsInterface#due)
+    def due
+      Marj::Relation.new(@ar_relation.due)
+    end
+
+    # (see Marj::JobsInterface#perform_all)
+    def perform_all(batch_size: nil)
+      if batch_size
+        [].tap do |results|
+          while (jobs = @ar_relation.limit(batch_size).map(&:as_job)).any?
+            results.concat(jobs.map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } })
+          end
+        end
+      else
+        @ar_relation.map(&:as_job).map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } }
+      end
+    end
+
+    # (see Marj::JobsInterface#discard_all)
+    def discard_all
+      @ar_relation.delete_all
+    end
+
+    # Yields each job in this relation.
+    #
+    # @param block [Proc]
+    # @return [Array] the jobs in this relation
+    def each(&block)
+      @ar_relation.map(&:as_job).each(&block)
+    end
+
+    # Provides +pretty_inspect+ output containing arrays of jobs rather than arrays of records, similar to the output
+    # produced when calling +pretty_inspect+ on +ActiveRecord::Relation+.
+    #
+    # Instead of the default +pretty_inspect+ output:
+    #   > Marj::Jobs.all
+    #    =>
+    #   #<Marj::Relation:0x000000012728bd88
+    #    @ar_relation=
+    #     [#<Marj::Record:0x0000000126c42080
+    #       job_id: "1382cb98-c518-46ca-a0cc-d831e11a0714",
+    #       job_class: TestJob,
+    #       arguments: ["foo"],
+    #       queue_name: "default",
+    #       priority: nil,
+    #       executions: 0,
+    #       exception_executions: {},
+    #       enqueued_at: 2024-01-25 15:31:06.115773 UTC,
+    #       scheduled_at: nil,
+    #       locale: "en",
+    #       timezone: "UTC">]>
+    #
+    # Produces:
+    #   > Marj::Jobs.all
+    #    =>
+    #   [#<TestJob:0x000000010b63cef8
+    #     @_scheduled_at_time=nil,
+    #     @arguments=[],
+    #     @enqueued_at=2024-01-25 15:31:06 UTC,
+    #     @exception_executions={},
+    #     @executions=0,
+    #     @job_id="1382cb98-c518-46ca-a0cc-d831e11a0714",
+    #     @locale="en",
+    #     @priority=nil,
+    #     @provider_job_id=nil,
+    #     @queue_name="default",
+    #     @scheduled_at=nil,
+    #     @serialized_arguments=["foo"],
+    #     @timezone="UTC">]
+    #
+    # @param pp [PP]
+    # @return [NilClass]
+    def pretty_print(pp)
+      pp.pp(to_a)
+    end
+  end
+end

data/lib/marj.rb

@@ -1,8 +1,106 @@
 # frozen_string_literal: true
 
+require_relative 'marj_adapter'
+require_relative 'marj/jobs'
+require_relative 'marj/jobs_interface'
+require_relative 'marj/record_interface'
+require_relative 'marj/relation'
+
+# A minimal database-backed ActiveJob queueing backend.
+#
 # See https://github.com/nicholasdower/marj
+module Marj
+  # The Marj version.
+  VERSION = '3.0.0'
 
-require_relative 'marj_adapter'
-require_relative 'marj_config'
+  Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
+
+  class << self
+    private
+
+    # Creates a job instance for the given record which will update the database when successfully executed, enqueued or
+    # discarded.
+    #
+    # @param record [ActiveRecord::Base]
+    # @return [ActiveJob::Base] the new job instance
+    def to_job(record)
+      # See register_callbacks for details on how callbacks are used.
+      job = record.job_class.new.tap { register_callbacks(_1, record) }
+
+      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
+      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
+      job_data = record.attributes.merge('arguments' => JSON.parse(record.read_attribute_before_type_cast(:arguments)))
+
+      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
+      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
+
+      job.tap { job.deserialize(job_data) }
+    end
+    private :to_job
+
+    # Registers callbacks for the given job which destroy the given database record when the job succeeds or is
+    # discarded.
+    #
+    # @param job [ActiveJob::Base]
+    # @param record [ActiveRecord::Base]
+    # @return [ActiveJob::Base]
+    def register_callbacks(job, record)
+      raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
+
+      # We need to detect three cases:
+      #  - If a job succeeds, after_perform will be called.
+      #  - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
+      #  - If a job exceeds its max attempts, after_discard will be called.
+      job.singleton_class.after_perform { |_j| record.destroy! }
+      job.singleton_class.after_discard { |_j, _exception| record.destroy! }
+      job.singleton_class.instance_variable_set(:@record, record)
+
+      job
+    end
+    private :register_callbacks
+
+    # Enqueue a job for execution at the specified time.
+    #
+    # @param job [ActiveJob::Base] the job to enqueue
+    # @param record_class [Class] the +ActiveRecord+ model class
+    # @param time [Time, NilClass] optional time at which to execute the job
+    # @return [ActiveJob::Base] the enqueued job
+    def enqueue(job, record_class, time = nil)
+      job.scheduled_at = time
+      # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
+      serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
 
-Kernel.autoload(:Marj, File.expand_path('marj_record.rb', __dir__))
+      # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
+      # are registered on the job instance so that when the job is executed, the database record is deleted or updated
+      # (depending on the result).
+      #
+      # We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
+      # singleton class. This holds a reference to the record. This ensures that if execute is called on a record
+      # instance, any updates to the database are reflected on that record instance.
+      if (existing_record = job.singleton_class.instance_variable_get(:@record))
+        # This job instance has already been associated with a database row.
+        if record_class.exists?(job_id: job.job_id)
+          # The database row still exists, we simply need to update it.
+          existing_record.update!(serialized)
+        else
+          # Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
+          # want to register the new instance because someone might still have a reference to the existing one.
+          record_class.create!(serialized)
+          existing_record.reload
+        end
+      else
+        # This job instance has not been associated with a database row.
+        if (new_record = record_class.find_by(job_id: job.job_id))
+          # The database row already exists. Update it.
+          new_record.update!(serialized)
+        else
+          # The database row does not exist. Create it.
+          new_record = record_class.create!(serialized)
+        end
+        register_callbacks(job, new_record)
+      end
+      job
+    end
+    private :enqueue
+  end
+end

data/lib/marj_adapter.rb

@@ -4,12 +4,19 @@
 #
 # See https://github.com/nicholasdower/marj
 class MarjAdapter
+  # Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ model class.
+  #
+  # @param record_class [Class, String] the +ActiveRecord+ model class (or its name) to use to store jobs
+  def initialize(record_class = 'Marj::Record')
+    @record_class = record_class
+  end
+
   # Enqueue a job for immediate execution.
   #
   # @param job [ActiveJob::Base] the job to enqueue
   # @return [ActiveJob::Base] the enqueued job
   def enqueue(job)
-    Marj.send(:enqueue, job)
+    Marj.send(:enqueue, job, record_class)
   end
 
   # Enqueue a job for execution at the specified time.
@@ -18,6 +25,12 @@ class MarjAdapter
   # @param timestamp [Numeric, NilClass] optional number of seconds since Unix epoch at which to execute the job
   # @return [ActiveJob::Base] the enqueued job
   def enqueue_at(job, timestamp)
-    Marj.send(:enqueue, job, timestamp ? Time.at(timestamp).utc : nil)
+    Marj.send(:enqueue, job, record_class, timestamp ? Time.at(timestamp).utc : nil)
+  end
+
+  private
+
+  def record_class
+    @record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: marj
 version: !ruby/object:Gem::Version
-  version: 2.1.0
+  version: 3.0.0
 platform: ruby
 authors:
 - Nick Dower
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-01-24 00:00:00.000000000 Z
+date: 2024-01-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -38,7 +38,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '7.1'
-description: Marj (Minimal ActiveRecord Jobs) is the simplest database-backed ActiveJob
+description: Marj (Minimal ActiveRecord Jobs) is a minimal database-backed ActiveJob
   queueing backend.
 email: nicholasdower@gmail.com
 executables: []
@@ -48,16 +48,19 @@ files:
 - LICENSE.txt
 - README.md
 - lib/marj.rb
+- lib/marj/jobs.rb
+- lib/marj/jobs_interface.rb
+- lib/marj/record.rb
+- lib/marj/record_interface.rb
+- lib/marj/relation.rb
 - lib/marj_adapter.rb
-- lib/marj_config.rb
-- lib/marj_record.rb
 homepage: https://github.com/nicholasdower/marj
 licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/nicholasdower/marj/issues
-  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v2.1.0
-  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v2.1.0
+  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v3.0.0
+  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v3.0.0
   homepage_uri: https://github.com/nicholasdower/marj
   rubygems_mfa_required: 'true'
   source_code_uri: https://github.com/nicholasdower/marj

data/lib/marj_config.rb

@@ -1,15 +0,0 @@
-# frozen_string_literal: true
-
-# Marj configuration.
-#
-# See https://github.com/nicholasdower/marj
-class MarjConfig
-  @table_name = 'jobs'
-
-  class << self
-    # The name of the database table. Defaults to "jobs".
-    #
-    # @return [String]
-    attr_accessor :table_name
-  end
-end

data/lib/marj_record.rb

@@ -1,157 +0,0 @@
-# frozen_string_literal: true
-
-require 'active_job'
-require 'active_record'
-require_relative 'marj_config'
-
-# The Marj ActiveRecord model class.
-#
-# See https://github.com/nicholasdower/marj
-class Marj < ActiveRecord::Base
-  # The Marj version.
-  VERSION = '2.1.0'
-
-  # Executes the job associated with this record and returns the result.
-  def execute
-    # Normally we would call ActiveJob::Base#execute which has the following implementation:
-    #   ActiveJob::Callbacks.run_callbacks(:execute) do
-    #     job = deserialize(job_data)
-    #     job.perform_now
-    #   end
-    # However, we need to instantiate the job ourselves in order to register callbacks before execution.
-    ActiveJob::Callbacks.run_callbacks(:execute) do
-      # See register_callbacks for details on how callbacks are used.
-      job = job_class.new.tap { Marj.send(:register_callbacks, _1, self) }
-
-      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
-      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
-      job_data = attributes.merge('arguments' => JSON.parse(read_attribute_before_type_cast(:arguments)))
-
-      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
-      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
-
-      job.deserialize(job_data)
-      job.perform_now
-    end
-  end
-
-  # Returns an ActiveRecord::Relation scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in the
-  # past. Jobs are ordered by +priority+ (+null+ last), then +scheduled_at+ (+null+ last), then +enqueued_at+.
-  #
-  # @return [ActiveRecord::Relation]
-  def self.ready
-    where('scheduled_at is null or scheduled_at <= ?', Time.now.utc).order(
-      Arel.sql(<<~SQL.squish)
-        CASE WHEN priority IS NULL THEN 1 ELSE 0 END, priority,
-        CASE WHEN scheduled_at IS NULL THEN 1 ELSE 0 END, scheduled_at,
-        enqueued_at
-    SQL
-    )
-  end
-
-  self.table_name = MarjConfig.table_name
-
-  # Order by +enqueued_at+ rather than +job_id+ (the default)
-  self.implicit_order_column = 'enqueued_at'
-
-  # Using a custom serializer for exception_executions so that we can interact with it as a hash rather than a string.
-  serialize(:exception_executions, coder: JSON)
-
-  # Using a custom serializer for arguments so that we can interact with as an array rather than a string.
-  # This enables code like:
-  #   Marj.first.arguments.first
-  #   Marj.first.update!(arguments: ['foo', 1, Time.now])
-  serialize(:arguments, coder: Class.new do
-    def self.dump(arguments)
-      return ActiveJob::Arguments.serialize(arguments).to_json if arguments.is_a?(Array)
-      return arguments if arguments.is_a?(String) || arguments.nil?
-
-      raise "invalid arguments: #{arguments}"
-    end
-
-    def self.load(arguments)
-      arguments ? ActiveJob::Arguments.deserialize(JSON.parse(arguments)) : nil
-    end
-  end)
-
-  # Using a custom serializer for job_class so that we can interact with it as a class rather than a string.
-  serialize(:job_class, coder: Class.new do
-    def self.dump(clazz)
-      return clazz.name if clazz.is_a?(Class)
-      return clazz if clazz.is_a?(String) || clazz.nil?
-
-      raise "invalid class: #{clazz}"
-    end
-
-    def self.load(str)
-      str&.constantize
-    end
-  end)
-
-  # Registers job callbacks used to keep the database record for the specified job in sync.
-  #
-  # @param job [ActiveJob::Base]
-  # @return [ActiveJob::Base]
-  def self.register_callbacks(job, record)
-    if job.singleton_class.instance_variable_get(:@__marj)
-      # Callbacks already registered. We just need to update the record.
-      job.singleton_class.instance_variable_set(:@__marj, record)
-      return
-    end
-
-    # We need to detect three cases:
-    #  - If a job succeeds, after_perform will be called.
-    #  - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
-    #  - If a job exceeds its max attempts, after_discard will be called.
-    job.singleton_class.after_perform { |_j| job.singleton_class.instance_variable_get(:@__marj).destroy! }
-    job.singleton_class.after_discard { |_j, _exception| job.singleton_class.instance_variable_get(:@__marj).destroy! }
-    job.singleton_class.instance_variable_set(:@__marj, record)
-
-    job
-  end
-  private_class_method :register_callbacks
-
-  # Enqueue a job for execution at the specified time.
-  #
-  # @param job [ActiveJob::Base] the job to enqueue
-  # @param time [Time, NilClass] optional time at which to execute the job
-  # @return [ActiveJob::Base] the enqueued job
-  def self.enqueue(job, time = nil)
-    job.scheduled_at = time
-    # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
-    serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
-
-    # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks are
-    # registered on the job instance so that when the job is executed, the database record is deleted or updated
-    # (depending on the result).
-    #
-    # There are two normal cases:
-    #  - The first time a job is enqueued, we need to create the record and register callbacks.
-    #  - If a previously enqueued job instance is re-enqueued, for instance after execution fails, callbacks have
-    #    already been registered. In this case we only need to update the record.
-    #
-    # We keep track of whether callbacks have been registered by setting the @__marj instance variable on the job's
-    # singleton class. This holds a reference to the record. This allows us to update the record without re-fetching it
-    # and also ensures that if execute is called on a record any updates to the database are reflected on that record
-    # instance.
-    #
-    # There are also two edge cases:
-    #  - It is possible for new job instance to be created for a job that is already in the database. In this case
-    #    we need to update the record and register callbacks.
-    #  - It is possible for the underlying row corresponding to an existing job to have been deleted. In this case we
-    #    need to create a new record and update the reference on the job's singleton class.
-    if (record = job.singleton_class.instance_variable_get(:@__marj))
-      if Marj.exists?(job_id: job.job_id)
-        record.update!(serialized)
-      else
-        record = Marj.create!(serialized)
-        register_callbacks(job, record)
-      end
-    else
-      record = Marj.find_or_create_by!(job_id: job.job_id) { _1.assign_attributes(serialized) }
-      register_callbacks(job, record)
-    end
-    job
-  end
-  private_class_method :enqueue
-end