gouda 0.1.3 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a8d5d39b730d429fe834fc88ecfe67e3a1bad397022715f6f52a1aa5e050df8b
-  data.tar.gz: b6cb85e539a387f75e3e84ffe756d54a991755dbd3c4335d67b476d70770eb97
+  metadata.gz: 0be474cea3bc87b9c0a41c9c654a31e4dc865e46876249a1db615bf56e494a40
+  data.tar.gz: '048574e736404c6c76f437fdfded93d7e810a23c83bb4f28ede8dffe9761a215'
 SHA512:
-  metadata.gz: 919a70fddf01f87880289e977b853aad5b01470e4f9b46ef1e8c5867ee50c4751f1ccd6ddc05146c40af83f8c82106d980feb93e022c4871114d7bb01361cee5
-  data.tar.gz: 466fb4a016ff74b481b936445ea66ed85981520147831b4cb11ec08446b7e7f7962974041f153448248558190ff0b0afb427b37e21af8b915337885ceb37f250
+  metadata.gz: 00e687d2fc384484cf2c07bf6c661e8d5ff1287a96f4e4f0abd98df2933538d7d81946ee51fe8e7abd6f2e5b285b6762aba231d359926ba5ee09d07a8b7e526c
+  data.tar.gz: cbd09ad83be4b9e6b9b35e20b2c80b7c8a5b8de1e20f8adfa6d0701d1aa31be7e5ea8edd2e64fab8f67e69cdc9757d1bcf79908fac9a81e9eccf9497f0a1c073

data/.ruby-version

@@ -1 +1 @@
-3.3.1
+3.3.3

data/CHANGELOG.md

@@ -15,3 +15,15 @@
 ## [0.1.3] - 2023-06-11
 
 - Allow the Rails app to boot even if there is no database yet
+
+## [0.1.4] - 2023-06-14
+
+- Rescue NoDatabaseError at scheduler update.
+- Include tests in gem, for sake of easier debugging.
+- Reduce logging in local test runs.
+- Bump local ruby version to 3.3.3
+
+## [0.1.5] - 2023-06-18
+
+- Update documentation
+- Don't pass on scheduler keys to retries

data/README.md

@@ -1,7 +1,10 @@
 Gouda is an ActiveJob adapter used at Cheddar. It requires PostgreSQL and a recent version of Rails.
 
-⚠️ At the moment Gouda is only used internally at Cheddar. We do not provide support for it, nor do we accept
-issues or feature requests. This is likely to change in the future.
+> [!CAUTION]
+> At the moment Gouda is only used internally at Cheddar. Any support to external parties is on best-effort
+> basis. While we are happy to see issues and pull requests, we can't guarantee that those will be addressed
+> quickly. The library does receive rapid updates which may break your application if you come to depend on
+> the library. That is to be expected.
 
 ## Installation
 
@@ -11,10 +14,10 @@ $ bundle install
 $ bin/rails g gouda:install
 ```
 
-Gouda is build as a lightweight alternative to [good_job](https://github.com/bensheldon/good_job) and has been created before [solid_queue.](https://github.com/rails/solid_queue/)
-It is _smaller_ than solid_queue though.
-
-It was designed to enable job processing using `SELECT ... FOR UPDATE SKIP LOCKED` on Postgres so that we could use pg_bouncer in our system setup. 
+Gouda is a lightweight alternative to [good_job](https://github.com/bensheldon/good_job) and [solid_queue.](https://github.com/rails/solid_queue/) - while
+more similar to the latter. It has been created prior to solid_queue and is smaller. It was designed to enable job processing using `SELECT ... FOR UPDATE SKIP LOCKED`
+on Postgres so that we could use pg_bouncer in our system setup. We have also observed that `SKIP LOCKED` causes less load on our database than advisory locking,
+especially as queue depths would grow. 
 
 
 ## Key concepts in Gouda: Workload

data/gouda.gemspec

@@ -7,17 +7,17 @@ Gem::Specification.new do |spec|
   spec.description = "Job Scheduler for Rails and PostgreSQL"
   spec.authors = ["Sebastian van Hesteren", "Julik Tarkhanov"]
   spec.email = ["sebastian@cheddar.me", "me@julik.nl"]
-  spec.homepage = "https://rubygems.org/gems/gouda"
+  spec.homepage = "https://github.com/cheddar-me/gouda"
   spec.license = "MIT"
   spec.required_ruby_version = Gem::Requirement.new(">= 2.7.0")
   spec.require_paths = ["lib"]
 
-  spec.metadata["homepage_uri"] = spec.homepage
-  spec.metadata["source_code_uri"] = "https://github.com/cheddar-me/gouda"
+  spec.metadata["homepage_uri"] = 
+  spec.metadata["source_code_uri"] = spec.homepage
   spec.metadata["changelog_uri"] = "https://github.com/cheddar-me/gouda/CHANGELOG.md"
 
   spec.files = Dir.chdir(File.expand_path(__dir__)) do
-    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{\A(?:test|spec|features)/}) }
+    `git ls-files -z`.split("\x0")
   end
 
   spec.add_dependency "activerecord", "~> 7"

data/lib/active_job/queue_adapters/gouda_adapter.rb

@@ -1,8 +1,10 @@
 # frozen_string_literal: true
 
-module ActiveJob # :nodoc:
-  module QueueAdapters # :nodoc:
-    class GoudaAdapter < Gouda::Adapter
-    end
-  end
+# The sole purpose of this class is so that you can do
+# `config.active_job.queue_adapter = :gouda` in your Rails
+# config, as Rails insists on resolving the adapter module
+# name from the symbol automatically. If Rails ever allows
+# us to "register" an adapter to a symbol this module can
+# be removed later.
+class ActiveJob::QueueAdapters::GoudaAdapter < Gouda::Adapter
 end

data/lib/gouda/adapter.rb

@@ -57,10 +57,11 @@ class Gouda::Adapter
       # We can't tell Postgres to ignore conflicts on _both_ the scheduler key and the enqueue concurrency key but not on
       # the ID - it is either "all indexes" or "just one", but never "this index and that index". MERGE https://www.postgresql.org/docs/current/sql-merge.html
       # is in theory capable of solving this but let's not complicate things all to hastily, the hour is getting late
+      scheduler_key = active_job.try(:executions) == 0 ? active_job.scheduler_key : nil # only enforce scheduler key on first workload
       {
         active_job_id: active_job.job_id, # Multiple jobs can have the same ID due to retries, job-iteration etc.
         scheduled_at: active_job.scheduled_at || t_now,
-        scheduler_key: active_job.scheduler_key, # So that the scheduler_key gets retained between retries
+        scheduler_key: scheduler_key,
         priority: active_job.priority,
         execution_concurrency_key: extract_execution_concurrency_key(active_job),
         enqueue_concurrency_key: extract_enqueue_concurrency_key(active_job),

data/lib/gouda/bulk.rb

@@ -1,6 +1,22 @@
 # frozen_string_literal: true
 
 module Gouda
+  # Inside this call, all `perform_later` calls on ActiveJob subclasses
+  # (including mailers) will be buffered. The call is reentrant, so you
+  # can have multiple `in_bulk` calls with arbitrary nesting. At the end
+  # of the block, the buffered jobs will be enqueued using their respective
+  # adapters. If an adapter supports `enqueue_all` (Sidekiq does in recent
+  # releases of Rails, for example), this functionality will be used. This
+  # method is especially useful when doing things such as mass-emails, or
+  # maintenance tasks where a large swath of jobs gets enqueued at once.
+  #
+  # @example
+  #    Gouda.in_bulk do
+  #      User.recently_joined.find_each do |recently_joined_user|
+  #        GreetingJob.perform_later(recently_joined_user)
+  #      end
+  #    end
+  # @return [Object] the return value of the block
   def self.in_bulk(&blk)
     if Thread.current[:gouda_bulk_buffer].nil?
       Thread.current[:gouda_bulk_buffer] = []

data/lib/gouda/queue_constraints.rb

@@ -1,44 +1,48 @@
 # frozen_string_literal: true
 
 module Gouda
+  # A queue constraint supports just one method, `to_sql`, which returns
+  # a condition based on the `queue_name` value of the `gouda_workloads`
+  # table. The minimal constraint is just a no-op - it allows execution
+  # of jobs from all queues in the system.
   module AnyQueue
     def self.to_sql
       "1=1"
     end
   end
 
+  # Allows execution of jobs only from specified queues
+  # For example, if you have a queue named "gpu", and you run
+  # jobs requiring a GPU on this queue, on your worker script
+  # running on GPU-equipped machines you could use
+  # `OnlyQueuesConstraint.new([:gpu])`
   class OnlyQueuesConstraint < Struct.new(:queue_names)
     def to_sql
       placeholders = (["?"] * queue_names.length).join(",")
-      ActiveRecord::Base.sanitize_sql_array([<<~SQL, *queue_names])
+      Gouda::Workload.sanitize_sql_array([<<~SQL, *queue_names])
         queue_name IN (#{placeholders})
       SQL
     end
   end
 
+  # Allows execution of jobs from queues except the given ones
+  # For example, if you have a queue named "emails" which is time-critical,
+  # on all other machines your worker script can specify
+  # `ExceptQueueConstraint.new([:emails])`
   class ExceptQueueConstraint < Struct.new(:queue_names)
     def to_sql
       placeholders = (["?"] * queue_names.length).join(",")
-      ActiveRecord::Base.sanitize_sql_array([<<~SQL, *queue_names])
+      Gouda::Workload.sanitize_sql_array([<<~SQL, *queue_names])
         queue_name NOT IN (#{placeholders})
       SQL
     end
   end
 
-  def self.parse_queue_constraint(constraint_str_from_envvar)
-    parsed = queue_parser(constraint_str_from_envvar)
-    if parsed[:include]
-      OnlyQueuesConstraint.new(parsed[:include])
-    elsif parsed[:exclude]
-      ExceptQueueConstraint.new(parsed[:exclude])
-    else
-      AnyQueue
-    end
-  end
-
-  # Parse a string representing a group of queues into a more readable data
-  # structure.
-  # @param string [String] Queue string
+  # Parse a string representing a group of queues into a queue constraint
+  # Note that this works similar to good_job. For example, the
+  # constraints do not necessarily compose all that well.
+  #
+  # @param queue_constraint_str[String] Queue string
   # @return [Hash]
   #   How to match a given queue. It can have the following keys and values:
   #   - +{ all: true }+ indicates that all queues match.
@@ -49,8 +53,8 @@ module Gouda
   # @example
   #   Gouda::QueueConstraints.queue_parser('-queue1,queue2')
   #   => { exclude: [ 'queue1', 'queue2' ] }
-  def self.queue_parser(string)
-    string = string.presence || "*"
+  def self.parse_queue_constraint(queue_constraint_str)
+    string = queue_constraint_str.presence || "*"
 
     case string.first
     when "-"
@@ -63,11 +67,11 @@ module Gouda
     queues = string.split(",").map(&:strip)
 
     if queues.include?("*")
-      {all: true}
+      AnyQueue
     elsif exclude_queues
-      {exclude: queues}
+      ExceptQueueConstraint.new([queues])
     else
-      {include: queues}
+      OnlyQueuesConstraint.new([queues])
     end
   end
 end

data/lib/gouda/railtie.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 module Gouda
+  UNINITIALISED_DATABASE_EXCEPTIONS = [ActiveRecord::NoDatabaseError, ActiveRecord::StatementInvalid, ActiveRecord::ConnectionNotEstablished]
+
   class Railtie < Rails::Railtie
     rake_tasks do
       task preload: :setup do
@@ -54,7 +56,7 @@ module Gouda
       Gouda::Scheduler.build_scheduler_entries_list!
       begin
         Gouda::Scheduler.upsert_workloads_from_entries_list!
-      rescue ActiveRecord::NoDatabaseError
+      rescue *Gouda::UNINITIALISED_DATABASE_EXCEPTIONS
         # Do nothing. On a freshly checked-out Rails app, running even unrelated Rails tasks
         # (such as asset compilation) - or, more importantly, initial db:create -
         # will cause a NoDatabaseError, as this is a chicken-and-egg problem. That error

data/lib/gouda/scheduler.rb

@@ -11,6 +11,9 @@ module Gouda::Scheduler
       [name, interval_seconds, cron, job_class].compact.join("_")
     end
 
+    # Tells when this particular task should next run
+    #
+    # @return [Time]
     def next_at
       if interval_seconds
         first_existing = Gouda::Workload.where(scheduler_key: scheduler_key).where("scheduled_at > NOW()").order("scheduled_at DESC").pluck(:scheduled_at).first
@@ -22,6 +25,9 @@ module Gouda::Scheduler
       end
     end
 
+    # Builds the ActiveJob which can be enqueued for this entry
+    #
+    # @return [ActiveJob::Base]
     def build_active_job
       next_at = self.next_at
       return unless next_at

data/lib/gouda/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Gouda
-  VERSION = "0.1.3"
+  VERSION = "0.1.5"
 end

data/test/gouda/concurrency_extension_test.rb

@@ -0,0 +1,160 @@
+# frozen_string_literal: true
+
+require "gouda/test_helper"
+
+class GoudaConcurrencyExtensionTest < ActiveSupport::TestCase
+  include AssertHelper
+  class TestJobWithoutConcurrency < ActiveJob::Base
+    self.queue_adapter = Gouda::Adapter.new
+  end
+
+  class TestJobWithPerformConcurrency < ActiveJob::Base
+    self.queue_adapter = Gouda::Adapter.new
+    include Gouda::ActiveJobExtensions::Concurrency
+    gouda_control_concurrency_with(perform_limit: 1)
+
+    def perform(*args)
+    end
+  end
+
+  setup do
+    @adapter ||= Gouda::Adapter.new
+    Gouda::Railtie.initializers.each(&:run)
+  end
+
+  test "gouda_control_concurrency_with with just perform_limit sets a perform concurrency key and no enqueue concurrency key" do
+    job = TestJobWithPerformConcurrency.new
+    assert_nil job.enqueue_concurrency_key
+    assert job.execution_concurrency_key
+  end
+
+  test "gouda_control_concurrency_with with just perform_limit makes the perform concurrency key dependent on job params" do
+    job1 = TestJobWithPerformConcurrency.new(1, 2, :something)
+    assert job1.execution_concurrency_key
+
+    job2 = TestJobWithPerformConcurrency.new(1, 2, :something)
+    assert_equal job2.execution_concurrency_key, job1.execution_concurrency_key
+
+    job3 = TestJobWithPerformConcurrency.new(1, 2, :something_else)
+    refute_equal job3.execution_concurrency_key, job1.execution_concurrency_key
+  end
+
+  class TestJobWithCommonConcurrency < ActiveJob::Base
+    self.queue_adapter = Gouda::Adapter.new
+    include Gouda::ActiveJobExtensions::Concurrency
+    gouda_control_concurrency_with(total_limit: 1)
+
+    def perform(*args)
+    end
+  end
+
+  test "gouda_control_concurrency_with with total_limit sets a perform concurrency key and an enqueue concurrency key" do
+    job = TestJobWithCommonConcurrency.new
+    assert job.enqueue_concurrency_key
+    assert job.execution_concurrency_key
+  end
+
+  test "gouda_control_concurrency_with with total_limit makes the perform concurrency key dependent on job params" do
+    job1 = TestJobWithCommonConcurrency.new(1, 2, :something)
+    assert job1.execution_concurrency_key
+
+    job2 = TestJobWithCommonConcurrency.new(1, 2, :something)
+    assert_equal job2.execution_concurrency_key, job1.execution_concurrency_key
+
+    job3 = TestJobWithCommonConcurrency.new(1, 2, :something_else)
+    refute_equal job3.execution_concurrency_key, job1.execution_concurrency_key
+  end
+
+  test "gouda_control_concurrency_with with total_limit makes the enqueue concurrency key dependent on job params" do
+    job1 = TestJobWithCommonConcurrency.new(1, 2, :something)
+    assert job1.enqueue_concurrency_key
+
+    job2 = TestJobWithCommonConcurrency.new(1, 2, :something)
+    assert_equal job2.enqueue_concurrency_key, job1.enqueue_concurrency_key
+
+    job3 = TestJobWithCommonConcurrency.new(1, 2, :something_else)
+    refute_equal job3.enqueue_concurrency_key, job1.enqueue_concurrency_key
+  end
+
+  class TestJobWithEnqueueConcurrency < ActiveJob::Base
+    self.queue_adapter = Gouda::Adapter.new
+    include Gouda::ActiveJobExtensions::Concurrency
+    gouda_control_concurrency_with(enqueue_limit: 1)
+
+    def perform(*args)
+    end
+  end
+
+  test "gouda_control_concurrency_with with enqueue_limit sets a perform concurrency key and an enqueue concurrency key" do
+    job = TestJobWithEnqueueConcurrency.new
+    assert job.enqueue_concurrency_key
+    assert_nil job.execution_concurrency_key
+  end
+
+  test "gouda_control_concurrency_with with enqueue_limit makes the enqueue concurrency key dependent on job params" do
+    job1 = TestJobWithEnqueueConcurrency.new(1, 2, :something)
+    assert job1.enqueue_concurrency_key
+
+    job2 = TestJobWithEnqueueConcurrency.new(1, 2, :something)
+    assert_equal job2.enqueue_concurrency_key, job1.enqueue_concurrency_key
+
+    job3 = TestJobWithEnqueueConcurrency.new(1, 2, :something_else)
+    refute_equal job3.enqueue_concurrency_key, job1.enqueue_concurrency_key
+  end
+
+  class TestJobWithCustomKey < ActiveJob::Base
+    self.queue_adapter = Gouda::Adapter.new
+    include Gouda::ActiveJobExtensions::Concurrency
+    gouda_control_concurrency_with total_limit: 1, key: "42"
+  end
+
+  test "can use an arbitrary string as the custom key" do
+    job = TestJobWithCustomKey.new
+    assert_equal "42", job.enqueue_concurrency_key
+    assert_equal "42", job.execution_concurrency_key
+  end
+
+  class TestJobWithCustomKeyProc < ActiveJob::Base
+    self.queue_adapter = Gouda::Adapter.new
+    include Gouda::ActiveJobExtensions::Concurrency
+    gouda_control_concurrency_with total_limit: 1, key: -> { @ivar }
+
+    def initialize(...)
+      super
+      @ivar = "123"
+    end
+  end
+
+  test "can use a proc that gets instance_exec'd as the custom key" do
+    job = TestJobWithCustomKeyProc.new
+    assert_equal "123", job.enqueue_concurrency_key
+    assert_equal "123", job.execution_concurrency_key
+  end
+
+  class TestJobWithWithUnconfiguredConcurrency < ActiveJob::Base
+    self.queue_adapter = Gouda::Adapter.new
+    include Gouda::ActiveJobExtensions::Concurrency
+  end
+
+  test "validates arguments" do
+    assert_raises ArgumentError do
+      TestJobWithWithUnconfiguredConcurrency.gouda_control_concurrency_with
+    end
+
+    assert_raises ArgumentError do
+      TestJobWithWithUnconfiguredConcurrency.gouda_control_concurrency_with total_limit: 2
+    end
+
+    assert_raises ArgumentError do
+      TestJobWithWithUnconfiguredConcurrency.gouda_control_concurrency_with perform_limit: 2
+    end
+
+    assert_raises ArgumentError do
+      TestJobWithWithUnconfiguredConcurrency.gouda_control_concurrency_with enqueue_limit: 2
+    end
+
+    assert_raises ArgumentError do
+      TestJobWithWithUnconfiguredConcurrency.gouda_control_concurrency_with total_limit: 2, bollocks: 4
+    end
+  end
+end