marj 2.1.0 → 4.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '078c3d48b27784e09425ba8eda2207038d6ce5c7a711953e8567c76aac689435'
-  data.tar.gz: c129b4848de1e2fe55de0de3a4893586a052cd814df798879dc196a81c1a94d5
+  metadata.gz: 4eb6129fa4948ed6e108469f44f380cda908620c6d08392cfa1e8c3bc84a9875
+  data.tar.gz: 28faed9a10fa3a8c885b521d6c3465196f814e97af07018d5eea88e580f322a9
 SHA512:
-  metadata.gz: 4cef48fb7c862123f4a5ad7ed5bf7a9d95eb3d7036831db79ad3dc9cf94f074a112ce58f0189484f991a52185cdf8e8ceed48dd9237bf4a04d5e3bbe30b2d274
-  data.tar.gz: 7631b1a0fef1ece1edcf15541dc85d25b2d5d3ba1cb8ca716133975126e5ce9bb4f28d627a4c2a5d751029ce8f67ba4db47c6fa9d197467fade4c09d20a04716
+  metadata.gz: 5f984f366a9cefb3187cbac7129144c8241d46d00357d719ec442dfce62d5c833c95ec30363a5af867e111499cc91cd2c60768da55d04c9c7fc401cce8ac3f83
+  data.tar.gz: e86029649f11ca05f96efcae59c1c60212b29335d9b154c3288a15aa81cd432b46f960166680b5bb0a534346852ccc73970cbaf8a298eb52893c37e260854f61

data/README.md

@@ -1,10 +1,10 @@
 # Marj - Minimal ActiveRecord Jobs
 
-The simplest database-backed ActiveJob queueing backend.
+A minimal database-backed ActiveJob queueing backend.
 
 ## Quick Links
 
-API docs: https://www.rubydoc.info/github/nicholasdower/marj <br>
+API docs: https://gemdocs.org/gems/marj/latest <br>
 RubyGems: https://rubygems.org/gems/marj <br>
 Changelog: https://github.com/nicholasdower/marj/releases <br>
 Issues: https://github.com/nicholasdower/marj/issues <br>
@@ -16,6 +16,8 @@ Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
 - Successfully executed jobs are deleted from the database.
 - Failed jobs which should be retried are updated in the database.
 - Failed jobs which should not be retried are deleted from the database.
+- An interface is provided to retrieve, execute, discard and re-enqueue jobs.
+- An `ActiveRecord` class is provided to query the database directly.
 
 ## Features Not Provided
 
@@ -24,35 +26,17 @@ Development: https://github.com/nicholasdower/marj/blob/master/CONTRIBUTING.md
 - Concurrency Controls
 - Observability
 - A User Interace
-- Anything else you might dream up.
-
-## Interface
-
-- `Marj` - An ActiveRecord model class
-- `Marj.ready` - Used to retrieve jobs ready to be executed
-- `Marj#execute` - Used to execute jobs retrieved from the database
-- `MarjConfig.table_name=` - Used to override the default table name
 
 ## Setup
 
 ### 1. Install
 
 ```shell
-bundle add marj
+bundle add activejob activerecord marj
 
 # or
 
-gem install marj
-```
-
-### 2. Configure
-
-By default, the database table is named "jobs". To use a different table name:
-
-```ruby
-require 'marj'
-
-MarjConfig.table_name = 'some_name'
+gem install activejob activerecord marj
 ```
 
 ### 3. Create the database table
@@ -84,23 +68,17 @@ class CreateJobs < ActiveRecord::Migration[7.1]
 end
 ```
 
+Note that by default, Marj uses a table named `jobs`. To override the default
+table name, set `Marj.table_name` before loading `ActiveRecord`.
+
 ### 4. Configure the queue adapter
 
 ```ruby
 require 'marj'
 
-# With rails:
-class MyApplication < Rails::Application
-  config.active_job.queue_adapter = :marj
-end
-
-# Without Rails:
-ActiveJob::Base.queue_adapter = :marj
-
-# Or for specific jobs (with or without Rails):
-class SomeJob < ActiveJob::Base
-  self.queue_adapter = :marj
-end
+Rails.configuration.active_job.queue_adapter = :marj # Globally, with Rails
+ActiveJob::Base.queue_adapter = :marj                # Globally, without Rails
+SomeJob.queue_adapter = :marj                        # Single job
 ```
 
 ## Example Usage
@@ -110,35 +88,139 @@ end
 job = SomeJob.perform_later('foo')
 job.perform_now
 
-# Enqueue, retrieve and manually run a job:
-SomeJob.perform_later('foo')
-Marj.first.execute
+# Retrieve and execute a job
+Marj.due.next.perform_now
 
-# Run all ready jobs:
-Marj.ready.each(&:execute)
+# Run all due jobs (single DB query)
+Marj.due.perform_all
 
-# Run all ready jobs, querying each time:
-loop { Marj.ready.first&.tap(&:execute) || break }
+# Run all due jobs (multiple DB queries)
+Marj.due.perform_all(batch_size: 1)
 
-# Run all ready jobs in a specific queue:
-loop { Marj.where(queue_name: 'foo').ready.first&.tap(&:execute) || break }
+# Run all due jobs in a specific queue:
+Marj.queue('foo').due.perform_all
 
-# Run jobs as they become ready:
+# Run jobs as they become due:
 loop do
-  loop { Marj.ready.first&.tap(&:execute) || break }
-rescue Exception => e
-  logger.error(e)
+  Marj.due.perform_all rescue logger.error($!)
 ensure
   sleep 5.seconds
 end
 ```
 
+# Jobs Interface
+
+The `Marj` module provides methods for interacting with enqueued jobs. These
+methods accept, return and yield +ActiveJob+ objects rather than +ActiveRecord+
+objects. Returned jobs are orderd by due date. To query the database directly,
+use `Marj::Record`.
+
+Example usage:
+
+```ruby
+Marj.all         # Returns all enqueued jobs.
+Marj.queue       # Returns jobs in the specified queue(s).
+Marj.due         # Returns jobs which are due to be executed.
+Marj.next        # Returns the next job(s) to be executed.
+Marj.count       # Returns the number of enqueued jobs.
+Marj.where       # Returns jobs matching the specified criteria.
+Marj.perform_all # Executes all jobs.
+Marj.discard_all # Discards all jobs.
+Marj.discard     # Discards the specified job.
+```
+
+Query methods can also be chained:
+
+```ruby
+Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
+```
+
+# Custom Jobs Interface
+
+The `Marj::JobsInterface` can be added to any class or module. For example, to
+add it to all jobs classes:
+
+```ruby
+class ApplicationJob < ActiveJob::Base
+  extend Marj::JobsInterface
+
+  def self.all
+    Marj::Relation.new(
+      self == ApplicationJob ?
+        Marj::Record.ordered : Marj::Record.where(job_class: self)
+   )
+  end
+end
+
+class SomeJob < ApplicationJob; end
+
+ApplicationJob.due # Returns all jobs which are due to be executed.
+SomeJob.due        # Returns SomeJobs which are due to be executed.
+```
+
+## Customization
+
+It is possible to create a custom record class and jobs interface. This enables,
+for instance, writing jobs to multiple databases/tables within a single
+application.
+
+```ruby
+class CreateMyJobs < ActiveRecord::Migration[7.1]
+  def self.up
+    create_table :my_jobs, id: :string, primary_key: :job_id do |table|
+      table.string   :job_class,            null: false
+      table.text     :arguments,            null: false
+      table.string   :queue_name,           null: false
+      table.integer  :priority
+      table.integer  :executions,           null: false
+      table.text     :exception_executions, null: false
+      table.datetime :enqueued_at,          null: false
+      table.datetime :scheduled_at
+      table.string   :locale,               null: false
+      table.string   :timezone,             null: false
+    end
+
+    add_index :my_jobs, %i[enqueued_at]
+    add_index :my_jobs, %i[scheduled_at]
+    add_index :my_jobs, %i[priority scheduled_at enqueued_at]
+  end
+
+  def self.down
+    drop_table :my_jobs
+  end
+end
+
+class MyRecord < Marj::Record
+  self.table_name = 'my_jobs'
+end
+
+CreateMyJobs.migrate(:up)
+
+class MyJob < ActiveJob::Base
+  self.queue_adapter = MarjAdapter.new('MyRecord')
+
+  extend Marj::JobsInterface
+
+  def self.all
+    Marj::Relation.new(MyRecord.all)
+  end
+
+  def perform(msg)
+    puts msg
+  end
+end
+
+MyJob.perform_later('oh, hi')
+MyJob.due.next.perform_now
+```
+
 ## Testing
 
-By default, jobs enqeued during tests will be written to the database. Enqueued jobs can be executed via:
+By default, jobs enqeued during tests will be written to the database. Enqueued
+jobs can be executed via:
 
 ```ruby
-Marj.ready.each(&:execute)
+Marj.due.perform_all
 ```
 
 Alternatively, to use [ActiveJob::QueueAdapters::TestAdapter](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/TestAdapter.html):
@@ -158,7 +240,9 @@ class ApplicationJob < ActiveJob::Base
 
   around_perform do |job, block|
     if (timeout = job.class.instance_variable_get(:@timeout))
-      ::Timeout.timeout(timeout, StandardError, 'execution expired') { block.call }
+      ::Timeout.timeout(timeout, StandardError, 'execution expired') do
+        block.call
+      end
     else
       block.call
     end
@@ -185,7 +269,9 @@ class ApplicationJob < ActiveJob::Base
   def last_error=(error)
     if error.is_a?(Exception)
       backtrace = error.backtrace&.map { |line| "\t#{line}" }&.join("\n")
-      error = backtrace ? "#{error.class}: #{error.message}\n#{backtrace}" : "#{error.class}: #{error.message}"
+      error = backtrace ?
+        "#{error.class}: #{error.message}\n#{backtrace}" :
+        "#{error.class}: #{error.message}"
     end
 
     @last_error = error&.truncate(10_000, omission: '… (truncated)')
@@ -217,10 +303,8 @@ For more information on ActiveJob, see:
 
 ```ruby
 # With Rails
-class MyApplication < Rails::Application
-  config.active_job.queue_adapter = :foo           # Instantiates FooAdapter
-  config.active_job.queue_adapter = FooAdapter.new # Uses FooAdapter directly
-end
+Rails.configuration.active_job.queue_adapter = :foo # Instantiates FooAdapter
+Rails.configuration.active_job.queue_adapter = FooAdapter.new
 
 # Without Rails
 ActiveJob::Base.queue_adapter = :foo               # Instantiates FooAdapter
@@ -233,40 +317,48 @@ SomeJob.queue_adapter = FooAdapter.new             # Uses FooAdapter directly
 
 ### Configuration
 
-- `config.active_job.default_queue_name`
-- `config.active_job.queue_name_prefix`
-- `config.active_job.queue_name_delimiter`
-- `config.active_job.retry_jitter`
-- `SomeJob.queue_name_prefix`
-- `SomeJob.queue_name_delimiter`
-- `SomeJob.retry_jitter`
-- `SomeJob.queue_name`
-- `SomeJob.queue_as`
+```ruby
+config.active_job.default_queue_name
+config.active_job.queue_name_prefix
+config.active_job.queue_name_delimiter
+config.active_job.retry_jitter
+SomeJob.queue_name
+SomeJob.queue_as
+SomeJob.queue_name_prefix
+SomeJob.queue_name_delimiter
+SomeJob.retry_jitter
+```
 
 ### Options
 
-- `:wait` - Enqueues the job with the specified delay
-- `:wait_until` - Enqueues the job at the time specified
-- `:queue` - Enqueues the job on the specified queue
-- `:priority` - Enqueues the job with the specified priority
+```ruby
+:wait       # Enqueues the job with the specified delay
+:wait_until # Enqueues the job at the time specified
+:queue      # Enqueues the job on the specified queue
+:priority   # Enqueues the job with the specified priority
+```
 
 ### Callbacks
 
-- `SomeJob.before_enqueue`
-- `SomeJob.after_enqueue`
-- `SomeJob.around_enqueue`
-- `SomeJob.before_perform`
-- `SomeJob.after_perform`
-- `SomeJob.around_perform`
-- `ActiveJob::Callbacks.singleton_class.set_callback(:execute, :before, &block)`
-- `ActiveJob::Callbacks.singleton_class.set_callback(:execute, :after, &block)`
-- `ActiveJob::Callbacks.singleton_class.set_callback(:execute, :around, &block)`
+```ruby
+SomeJob.before_enqueue
+SomeJob.after_enqueue
+SomeJob.around_enqueue
+SomeJob.before_perform
+SomeJob.after_perform
+SomeJob.around_perform
+ActiveJob::Callbacks.singleton_class.set_callback(:execute, :before, &block)
+ActiveJob::Callbacks.singleton_class.set_callback(:execute, :after, &block)
+ActiveJob::Callbacks.singleton_class.set_callback(:execute, :around, &block)
+```
 
 ### Handling Exceptions
 
-- `SomeJob.retry_on`
-- `SomeJob.discard_on`
-- `SomeJob.after_discard`
+```ruby
+SomeJob.retry_on
+SomeJob.discard_on
+SomeJob.after_discard
+```
 
 ### Creating Jobs
 
@@ -274,35 +366,40 @@ SomeJob.queue_adapter = FooAdapter.new             # Uses FooAdapter directly
 # Create without enqueueing
 job = SomeJob.new
 job = SomeJob.new(args)
+job = SomeJob.new.deserialize(other_job.serialize)
 
 # Create and enqueue
 job = SomeJob.perform_later
 job = SomeJob.perform_later(args)
 
-# Create and run (enqueued on failure)
+# Create without enqueueing and run (only enqueued on failure if retryable)
 SomeJob.perform_now
 SomeJob.perform_now(args)
 ```
 
 ### Enqueueing Jobs
 
+Jobs are enqueued via the `ActiveJob::Base#enqueue` method. This method returns
+the job on success. If an error is raised during enqueueing, that error will
+propagate to the caller, unless the error is an `ActiveJob::EnqueueError`. In
+this case, `enqueue` will return `false` and `job.enqueue_error` will be set.
+
 ```ruby
 SomeJob.new(args).enqueue
 SomeJob.new(args).enqueue(options)
 
+# Via perform_later
 SomeJob.perform_later(SomeJob.new(args))
-
 SomeJob.perform_later(args)
 SomeJob.set(options).perform_later(args)
 
-# Enqueued on failure
+# After a failure during execution
 SomeJob.perform_now(args)
+ActiveJob::Base.execute(SomeJob.new(args).serialize)
 
 # Enqueue multiple
 ActiveJob.perform_all_later(SomeJob.new, SomeJob.new)
 ActiveJob.perform_all_later(SomeJob.new, SomeJob.new, options:)
-
-# Enqueue multiple
 SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new)
 SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new, options:)
 ```
@@ -310,7 +407,7 @@ SomeJob.set(options).perform_all_later(SomeJob.new, SomeJob.new, options:)
 ### Executing Jobs
 
 ```ruby
-# Executed without enqueueing, enqueued on failure if retries configured
+# Executed without enqueueing, enqueued on failure if retryable
 SomeJob.new(args).perform_now
 SomeJob.perform_now(args)
 ActiveJob::Base.execute(SomeJob.new(args).serialize)

data/lib/marj/jobs_interface.rb

@@ -0,0 +1,113 @@
+# frozen_string_literal: true
+
+module Marj
+  # The interface provided by {Marj} and {Marj::Relation}.
+  #
+  # To create a custom jobs interface, for example for all job classes in your application:
+  #   class ApplicationJob < ActiveJob::Base
+  #     extend Marj::JobsInterface
+  #
+  #     def self.all
+  #       Marj::Relation.new(self == ApplicationJob ? Marj::Record.ordered : Marj::Record.where(job_class: self))
+  #     end
+  #   end
+  #
+  #   class SomeJob < ApplicationJob
+  #     def perform(msg)
+  #       puts msg
+  #     end
+  #   end
+  #
+  # This will allow you to query jobs via the +ApplicationJob+ class:
+  #   ApplicationJob.next # Returns the next job of any type
+  #
+  # Or to query jobs via a specific job class:
+  #   SomeJob.next # Returns the next SomeJob
+  #
+  # Alternatively, to create a jobs interface for a single job class:
+  #   class SomeJob < ActiveJob::Base
+  #     extend Marj::JobsInterface
+  #
+  #     def self.all
+  #       Marj::Relation.new(Marj::Record.where(job_class: self).ordered)
+  #     end
+  #   end
+  module JobsInterface
+    def self.included(clazz)
+      return if clazz == Marj::Relation
+
+      clazz.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
+    end
+    private_class_method :included
+
+    def self.extended(clazz)
+      clazz.singleton_class.delegate :queue, :next, :count, :where, :due, :perform_all, :discard_all, to: :all
+    end
+    private_class_method :extended
+
+    # Returns a {Marj::Relation} for jobs in the specified queue(s).
+    #
+    # @param queue [String, Symbol] the queue to query
+    # @param queues [Array<String>, Array<Symbol>] more queues to query
+    # @return [Marj::Relation]
+    def queue(queue, *queues)
+      Marj::Relation.new(all.where(queue_name: queues.dup.unshift(queue)))
+    end
+
+    # Returns the next job or the next N jobs if +limit+ is specified. If no jobs exist, returns +nil+.
+    #
+    # @param limit [Integer, NilClass]
+    # @return [ActiveJob::Base, NilClass]
+    def next(limit = nil)
+      all.first(limit)&.then { _1.is_a?(Array) ? _1.map(&:as_job) : _1.as_job }
+    end
+
+    # Returns a count of jobs, optionally either matching the specified column name criteria or where the specified
+    # block returns +true+.
+    #
+    # @param column_name [String, Symbol, NilClass]
+    # @param block [Proc, NilClass]
+    # @return [Integer]
+    def count(column_name = nil, &block)
+      block_given? ? all.count(column_name) { |r| block.call(r.as_job) } : all.count(column_name)
+    end
+
+    # Returns a {Marj::Relation} for jobs matching the specified criteria.
+    #
+    # @param args [Array]
+    # @return [Marj::Relation]
+    def where(*args)
+      Marj::Relation.new(all.where(*args))
+    end
+
+    # Returns a {Marj::Relation} for enqueued jobs with a +scheduled_at+ that is either +null+ or in the past.
+    #
+    # @return [Marj::Relation]
+    def due
+      Marj::Relation.new(all.due)
+    end
+
+    # Calls +perform_now+ on each job.
+    #
+    # @param batch_size [Integer, NilClass] the number of jobs to fetch at a time, or +nil+ to fetch all jobs at once
+    # @return [Array] the results returned by each job
+    def perform_all(batch_size: nil)
+      if batch_size
+        [].tap do |results|
+          while (jobs = all.limit(batch_size).map(&:as_job)).any?
+            results.concat(jobs.map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } })
+          end
+        end
+      else
+        all.map(&:as_job).map { |job| ActiveJob::Callbacks.run_callbacks(:execute) { job.perform_now } }
+      end
+    end
+
+    # Discards all jobs.
+    #
+    # @return [Numeric] the number of discarded jobs
+    def discard_all
+      all.delete_all
+    end
+  end
+end

data/lib/marj/record.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+require 'active_job'
+require 'active_record'
+
+module Marj
+  # The default +ActiveRecord+ class.
+  class Record < ActiveRecord::Base
+    self.table_name = Marj.table_name
+
+    # Order by +enqueued_at+ rather than +job_id+ (the default).
+    self.implicit_order_column = 'enqueued_at'
+
+    # Using a custom serializer for exception_executions so that we can interact with it as a hash rather than a
+    # string.
+    serialize(:exception_executions, coder: JSON)
+
+    # Using a custom serializer for arguments so that we can interact with as an array rather than a string.
+    # This enables code like:
+    #   Marj::Record.next.arguments.first
+    #   Marj::Record.next.update!(arguments: ['foo', 1, Time.now])
+    serialize(:arguments, coder: Class.new do
+      def self.dump(arguments)
+        return ActiveJob::Arguments.serialize(arguments).to_json if arguments.is_a?(Array)
+        return arguments if arguments.is_a?(String) || arguments.nil?
+
+        raise "invalid arguments: #{arguments}"
+      end
+
+      def self.load(arguments)
+        arguments ? ActiveJob::Arguments.deserialize(JSON.parse(arguments)) : nil
+      end
+    end)
+
+    # Using a custom serializer for job_class so that we can interact with it as a class rather than a string.
+    serialize(:job_class, coder: Class.new do
+      def self.dump(clazz)
+        return clazz.name if clazz.is_a?(Class)
+        return clazz if clazz.is_a?(String) || clazz.nil?
+
+        raise "invalid class: #{clazz}"
+      end
+
+      def self.load(str)
+        str&.constantize
+      end
+    end)
+
+    # Returns a job object for this record which will update the database when successfully executed, enqueued or
+    # discarded.
+    #
+    # @return [ActiveJob::Base]
+    def as_job
+      Marj.send(:to_job, self)
+    end
+
+    class << self
+      # Returns an +ActiveRecord::Relation+ scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in
+      # the past.
+      #
+      # @return [ActiveRecord::Relation]
+      def due
+        where('scheduled_at IS NULL OR scheduled_at <= ?', Time.now.utc)
+      end
+
+      # Returns an +ActiveRecord::Relation+ scope for jobs ordered by +priority+ (+null+ last), then +scheduled_at+
+      # (+null+ last), then +enqueued_at+.
+      #
+      # @return [ActiveRecord::Relation]
+      def ordered
+        order(
+          Arel.sql(<<~SQL.squish, Time.now.utc)
+            CASE WHEN scheduled_at IS NULL OR scheduled_at <= ? THEN 0 ELSE 1 END,
+            CASE WHEN priority IS NULL THEN 1 ELSE 0 END, priority,
+            CASE WHEN scheduled_at IS NULL THEN 1 ELSE 0 END, scheduled_at,
+            enqueued_at
+          SQL
+        )
+      end
+    end
+  end
+end

data/lib/marj/relation.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+require_relative 'jobs_interface'
+
+module Marj
+  # Returned by {Marj::JobsInterface} query methods to enable chaining and +Enumerable+ methods.
+  class Relation
+    include Enumerable
+    include Marj::JobsInterface
+
+    attr_reader :all
+    private :all
+
+    # Returns a {Marj::Relation} which wraps the specified +ActiveRecord+ relation.
+    def initialize(ar_relation)
+      @all = ar_relation
+    end
+
+    # Yields each job in this relation.
+    #
+    # @param block [Proc]
+    # @return [Array] the jobs in this relation
+    def each(&block)
+      all.map(&:as_job).each(&block)
+    end
+
+    # Provides +pretty_inspect+ output containing arrays of jobs rather than arrays of records, similar to the output
+    # produced when calling +pretty_inspect+ on +ActiveRecord::Relation+.
+    #
+    # Instead of the default +pretty_inspect+ output:
+    #   > Marj.all
+    #    =>
+    #   #<Marj::Relation:0x000000012728bd88
+    #    @ar_relation=
+    #     [#<Marj::Record:0x0000000126c42080
+    #       job_id: "1382cb98-c518-46ca-a0cc-d831e11a0714",
+    #       job_class: TestJob,
+    #       arguments: ["foo"],
+    #       queue_name: "default",
+    #       priority: nil,
+    #       executions: 0,
+    #       exception_executions: {},
+    #       enqueued_at: 2024-01-25 15:31:06.115773 UTC,
+    #       scheduled_at: nil,
+    #       locale: "en",
+    #       timezone: "UTC">]>
+    #
+    # Produces:
+    #   > Marj.all
+    #    =>
+    #   [#<TestJob:0x000000010b63cef8
+    #     @_scheduled_at_time=nil,
+    #     @arguments=[],
+    #     @enqueued_at=2024-01-25 15:31:06 UTC,
+    #     @exception_executions={},
+    #     @executions=0,
+    #     @job_id="1382cb98-c518-46ca-a0cc-d831e11a0714",
+    #     @locale="en",
+    #     @priority=nil,
+    #     @provider_job_id=nil,
+    #     @queue_name="default",
+    #     @scheduled_at=nil,
+    #     @serialized_arguments=["foo"],
+    #     @timezone="UTC">]
+    #
+    # @param pp [PP]
+    # @return [NilClass]
+    def pretty_print(pp)
+      pp.pp(to_a)
+    end
+  end
+end

data/lib/marj.rb

@@ -1,8 +1,156 @@
 # frozen_string_literal: true
 
+require_relative 'marj_adapter'
+require_relative 'marj/jobs_interface'
+require_relative 'marj/relation'
+
+# A minimal database-backed ActiveJob queueing backend.
+#
+# The {Marj} module provides methods for interacting with enqueued jobs. These methods accept, return and yield
+# +ActiveJob+ objects rather than +ActiveRecord+ objects. Returned jobs are ordered by due date. To query the database
+# directly, use {Record}.
+#
+# Example usage:
+#   Marj.all         # Returns all enqueued jobs.
+#   Marj.queue       # Returns jobs in the specified queue(s).
+#   Marj.due         # Returns jobs which are due to be executed.
+#   Marj.next        # Returns the next job(s) to be executed.
+#   Marj.count       # Returns the number of enqueued jobs.
+#   Marj.where       # Returns jobs matching the specified criteria.
+#   Marj.perform_all # Executes all jobs.
+#   Marj.discard_all # Discards all jobs.
+#   Marj.discard     # Discards the specified job.
+#
+# Query methods can also be chained:
+#   Marj.due.where(job_class: SomeJob).next # Returns the next SomeJob that is due
+#
+# Note that by default, Marj uses {Marj::Record} to interact with the +jobs+ table. To use a different record class, set
+# {record_class}. To simply override the table name, set {table_name} before loading +ActiveRecord+.
+#
 # See https://github.com/nicholasdower/marj
+module Marj
+  # The Marj version.
+  VERSION = '4.0.0'
 
-require_relative 'marj_adapter'
-require_relative 'marj_config'
+  Kernel.autoload(:Record, File.expand_path(File.join('marj', 'record.rb'), __dir__))
+
+  @table_name = :jobs
+  @record_class = 'Marj::Record'
+
+  class << self
+    include Marj::JobsInterface
+
+    # @!attribute record_class
+    #   The name of the +ActiveRecord+ class. Defaults to +Marj::Record+.
+    #   @return [Class, String]
+
+    attr_writer :record_class
+
+    def record_class
+      @record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
+    end
+
+    # @!attribute table_name
+    #   The name of the database table. Defaults to +:jobs+.
+    #   @return [Symbol, String]
+    attr_accessor :table_name
+
+    # Returns a {Marj::Relation} for all jobs in the order they should be executed.
+    #
+    # @return [Marj::Relation]
+    def all
+      Marj::Relation.new(Marj.record_class.ordered)
+    end
+
+    # Discards the specified job.
+    #
+    # @return [Integer] the number of discarded jobs
+    def discard(job)
+      all.where(job_id: job.job_id).discard_all
+    end
+
+    private
+
+    # Creates a job instance for the given record which will update the database when successfully executed, enqueued or
+    # discarded.
+    #
+    # @param record [ActiveRecord::Base]
+    # @return [ActiveJob::Base] the new job instance
+    def to_job(record)
+      # See register_callbacks for details on how callbacks are used.
+      job = record.job_class.new.tap { register_callbacks(_1, record) }
+
+      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
+      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
+      job_data = record.attributes.merge('arguments' => JSON.parse(record.read_attribute_before_type_cast(:arguments)))
+
+      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
+      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
+
+      job.tap { job.deserialize(job_data) }
+    end
+
+    # Registers callbacks for the given job which destroy the given database record when the job succeeds or is
+    # discarded.
+    #
+    # @param job [ActiveJob::Base]
+    # @param record [ActiveRecord::Base]
+    # @return [ActiveJob::Base]
+    def register_callbacks(job, record)
+      raise 'callbacks already registered' if job.singleton_class.instance_variable_get(:@record)
+
+      # We need to detect three cases:
+      #  - If a job succeeds, after_perform will be called.
+      #  - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
+      #  - If a job exceeds its max attempts, after_discard will be called.
+      job.singleton_class.after_perform { |_j| record.destroy! }
+      job.singleton_class.after_discard { |_j, _exception| record.destroy! }
+      job.singleton_class.instance_variable_set(:@record, record)
+
+      job
+    end
+
+    # Enqueue a job for execution at the specified time.
+    #
+    # @param job [ActiveJob::Base] the job to enqueue
+    # @param record_class [Class] the +ActiveRecord+ class
+    # @param time [Time, NilClass] optional time at which to execute the job
+    # @return [ActiveJob::Base] the enqueued job
+    def enqueue(job, record_class, time = nil)
+      job.scheduled_at = time
+      # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
+      serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
 
-Kernel.autoload(:Marj, File.expand_path('marj_record.rb', __dir__))
+      # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks
+      # are registered on the job instance so that when the job is executed, the database record is deleted or updated
+      # (depending on the result).
+      #
+      # We keep track of whether callbacks have been registered by setting the @record instance variable on the job's
+      # singleton class. This holds a reference to the record. This ensures that if execute is called on a record
+      # instance, any updates to the database are reflected on that record instance.
+      if (existing_record = job.singleton_class.instance_variable_get(:@record))
+        # This job instance has already been associated with a database row.
+        if record_class.exists?(job_id: job.job_id)
+          # The database row still exists, we simply need to update it.
+          existing_record.update!(serialized)
+        else
+          # Someone else deleted the database row, we need to recreate and reload the existing record instance. We don't
+          # want to register the new instance because someone might still have a reference to the existing one.
+          record_class.create!(serialized)
+          existing_record.reload
+        end
+      else
+        # This job instance has not been associated with a database row.
+        if (new_record = record_class.find_by(job_id: job.job_id))
+          # The database row already exists. Update it.
+          new_record.update!(serialized)
+        else
+          # The database row does not exist. Create it.
+          new_record = record_class.create!(serialized)
+        end
+        register_callbacks(job, new_record)
+      end
+      job
+    end
+  end
+end

data/lib/marj_adapter.rb

@@ -4,12 +4,19 @@
 #
 # See https://github.com/nicholasdower/marj
 class MarjAdapter
+  # Creates a new adapter which will enqueue jobs using the given +ActiveRecord+ class.
+  #
+  # @param record_class [Class, String] the +ActiveRecord+ class (or its name) to use to store jobs
+  def initialize(record_class = 'Marj::Record')
+    @record_class = record_class
+  end
+
   # Enqueue a job for immediate execution.
   #
   # @param job [ActiveJob::Base] the job to enqueue
   # @return [ActiveJob::Base] the enqueued job
   def enqueue(job)
-    Marj.send(:enqueue, job)
+    Marj.send(:enqueue, job, record_class)
   end
 
   # Enqueue a job for execution at the specified time.
@@ -18,6 +25,12 @@ class MarjAdapter
   # @param timestamp [Numeric, NilClass] optional number of seconds since Unix epoch at which to execute the job
   # @return [ActiveJob::Base] the enqueued job
   def enqueue_at(job, timestamp)
-    Marj.send(:enqueue, job, timestamp ? Time.at(timestamp).utc : nil)
+    Marj.send(:enqueue, job, record_class, timestamp ? Time.at(timestamp).utc : nil)
+  end
+
+  private
+
+  def record_class
+    @record_class = @record_class.is_a?(String) ? @record_class.constantize : @record_class
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: marj
 version: !ruby/object:Gem::Version
-  version: 2.1.0
+  version: 4.0.0
 platform: ruby
 authors:
 - Nick Dower
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-01-24 00:00:00.000000000 Z
+date: 2024-01-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -38,7 +38,7 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '7.1'
-description: Marj (Minimal ActiveRecord Jobs) is the simplest database-backed ActiveJob
+description: Marj (Minimal ActiveRecord Jobs) is a minimal database-backed ActiveJob
   queueing backend.
 email: nicholasdower@gmail.com
 executables: []
@@ -48,16 +48,17 @@ files:
 - LICENSE.txt
 - README.md
 - lib/marj.rb
+- lib/marj/jobs_interface.rb
+- lib/marj/record.rb
+- lib/marj/relation.rb
 - lib/marj_adapter.rb
-- lib/marj_config.rb
-- lib/marj_record.rb
 homepage: https://github.com/nicholasdower/marj
 licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/nicholasdower/marj/issues
-  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v2.1.0
-  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v2.1.0
+  changelog_uri: https://github.com/nicholasdower/marj/releases/tag/v4.0.0
+  documentation_uri: https://www.rubydoc.info/github/nicholasdower/marj/v4.0.0
   homepage_uri: https://github.com/nicholasdower/marj
   rubygems_mfa_required: 'true'
   source_code_uri: https://github.com/nicholasdower/marj

data/lib/marj_config.rb

@@ -1,15 +0,0 @@
-# frozen_string_literal: true
-
-# Marj configuration.
-#
-# See https://github.com/nicholasdower/marj
-class MarjConfig
-  @table_name = 'jobs'
-
-  class << self
-    # The name of the database table. Defaults to "jobs".
-    #
-    # @return [String]
-    attr_accessor :table_name
-  end
-end

data/lib/marj_record.rb

@@ -1,157 +0,0 @@
-# frozen_string_literal: true
-
-require 'active_job'
-require 'active_record'
-require_relative 'marj_config'
-
-# The Marj ActiveRecord model class.
-#
-# See https://github.com/nicholasdower/marj
-class Marj < ActiveRecord::Base
-  # The Marj version.
-  VERSION = '2.1.0'
-
-  # Executes the job associated with this record and returns the result.
-  def execute
-    # Normally we would call ActiveJob::Base#execute which has the following implementation:
-    #   ActiveJob::Callbacks.run_callbacks(:execute) do
-    #     job = deserialize(job_data)
-    #     job.perform_now
-    #   end
-    # However, we need to instantiate the job ourselves in order to register callbacks before execution.
-    ActiveJob::Callbacks.run_callbacks(:execute) do
-      # See register_callbacks for details on how callbacks are used.
-      job = job_class.new.tap { Marj.send(:register_callbacks, _1, self) }
-
-      # ActiveJob::Base#deserialize expects serialized arguments. But the record arguments have already been
-      # deserialized by a custom ActiveRecord serializer (see below). So instead we use the raw arguments string.
-      job_data = attributes.merge('arguments' => JSON.parse(read_attribute_before_type_cast(:arguments)))
-
-      # ActiveJob::Base#deserialize expects dates to be strings rather than Time objects.
-      job_data = job_data.to_h { |k, v| [k, %w[enqueued_at scheduled_at].include?(k) ? v&.iso8601 : v] }
-
-      job.deserialize(job_data)
-      job.perform_now
-    end
-  end
-
-  # Returns an ActiveRecord::Relation scope for enqueued jobs with a +scheduled_at+ that is either +null+ or in the
-  # past. Jobs are ordered by +priority+ (+null+ last), then +scheduled_at+ (+null+ last), then +enqueued_at+.
-  #
-  # @return [ActiveRecord::Relation]
-  def self.ready
-    where('scheduled_at is null or scheduled_at <= ?', Time.now.utc).order(
-      Arel.sql(<<~SQL.squish)
-        CASE WHEN priority IS NULL THEN 1 ELSE 0 END, priority,
-        CASE WHEN scheduled_at IS NULL THEN 1 ELSE 0 END, scheduled_at,
-        enqueued_at
-    SQL
-    )
-  end
-
-  self.table_name = MarjConfig.table_name
-
-  # Order by +enqueued_at+ rather than +job_id+ (the default)
-  self.implicit_order_column = 'enqueued_at'
-
-  # Using a custom serializer for exception_executions so that we can interact with it as a hash rather than a string.
-  serialize(:exception_executions, coder: JSON)
-
-  # Using a custom serializer for arguments so that we can interact with as an array rather than a string.
-  # This enables code like:
-  #   Marj.first.arguments.first
-  #   Marj.first.update!(arguments: ['foo', 1, Time.now])
-  serialize(:arguments, coder: Class.new do
-    def self.dump(arguments)
-      return ActiveJob::Arguments.serialize(arguments).to_json if arguments.is_a?(Array)
-      return arguments if arguments.is_a?(String) || arguments.nil?
-
-      raise "invalid arguments: #{arguments}"
-    end
-
-    def self.load(arguments)
-      arguments ? ActiveJob::Arguments.deserialize(JSON.parse(arguments)) : nil
-    end
-  end)
-
-  # Using a custom serializer for job_class so that we can interact with it as a class rather than a string.
-  serialize(:job_class, coder: Class.new do
-    def self.dump(clazz)
-      return clazz.name if clazz.is_a?(Class)
-      return clazz if clazz.is_a?(String) || clazz.nil?
-
-      raise "invalid class: #{clazz}"
-    end
-
-    def self.load(str)
-      str&.constantize
-    end
-  end)
-
-  # Registers job callbacks used to keep the database record for the specified job in sync.
-  #
-  # @param job [ActiveJob::Base]
-  # @return [ActiveJob::Base]
-  def self.register_callbacks(job, record)
-    if job.singleton_class.instance_variable_get(:@__marj)
-      # Callbacks already registered. We just need to update the record.
-      job.singleton_class.instance_variable_set(:@__marj, record)
-      return
-    end
-
-    # We need to detect three cases:
-    #  - If a job succeeds, after_perform will be called.
-    #  - If a job fails and should be retried, enqueue will be called. This is handled by the enqueue method.
-    #  - If a job exceeds its max attempts, after_discard will be called.
-    job.singleton_class.after_perform { |_j| job.singleton_class.instance_variable_get(:@__marj).destroy! }
-    job.singleton_class.after_discard { |_j, _exception| job.singleton_class.instance_variable_get(:@__marj).destroy! }
-    job.singleton_class.instance_variable_set(:@__marj, record)
-
-    job
-  end
-  private_class_method :register_callbacks
-
-  # Enqueue a job for execution at the specified time.
-  #
-  # @param job [ActiveJob::Base] the job to enqueue
-  # @param time [Time, NilClass] optional time at which to execute the job
-  # @return [ActiveJob::Base] the enqueued job
-  def self.enqueue(job, time = nil)
-    job.scheduled_at = time
-    # Argument serialization is done by ActiveJob. ActiveRecord expects deserialized arguments.
-    serialized = job.serialize.symbolize_keys!.without(:provider_job_id).merge(arguments: job.arguments)
-
-    # When a job is enqueued, we must create/update the corresponding database record. We also must ensure callbacks are
-    # registered on the job instance so that when the job is executed, the database record is deleted or updated
-    # (depending on the result).
-    #
-    # There are two normal cases:
-    #  - The first time a job is enqueued, we need to create the record and register callbacks.
-    #  - If a previously enqueued job instance is re-enqueued, for instance after execution fails, callbacks have
-    #    already been registered. In this case we only need to update the record.
-    #
-    # We keep track of whether callbacks have been registered by setting the @__marj instance variable on the job's
-    # singleton class. This holds a reference to the record. This allows us to update the record without re-fetching it
-    # and also ensures that if execute is called on a record any updates to the database are reflected on that record
-    # instance.
-    #
-    # There are also two edge cases:
-    #  - It is possible for new job instance to be created for a job that is already in the database. In this case
-    #    we need to update the record and register callbacks.
-    #  - It is possible for the underlying row corresponding to an existing job to have been deleted. In this case we
-    #    need to create a new record and update the reference on the job's singleton class.
-    if (record = job.singleton_class.instance_variable_get(:@__marj))
-      if Marj.exists?(job_id: job.job_id)
-        record.update!(serialized)
-      else
-        record = Marj.create!(serialized)
-        register_callbacks(job, record)
-      end
-    else
-      record = Marj.find_or_create_by!(job_id: job.job_id) { _1.assign_attributes(serialized) }
-      register_callbacks(job, record)
-    end
-    job
-  end
-  private_class_method :enqueue
-end