postqueue 0.2.1 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 7b6f8a8ad0f1aef1b565a223153f6280a9361314
-  data.tar.gz: 4dbf3727e217a8c248ce37fc78715aaf9080f0ea
+  metadata.gz: 5fd15294351f2f6e2be7d4c15d7ac509e097f3c5
+  data.tar.gz: cbe1f9ca4e5bf7bbfa6027794e2c0ff26ad14c03
 SHA512:
-  metadata.gz: 569162000045134eb3107231ed94c6b52e15f5aed0e47ec2d681502037d142fe0cbd7ac9e85b2361e5fbeced0a0bd01fce6bee8cea4ef258cf0b2b19c46afb4e
-  data.tar.gz: 6c50d7e414a6b411e4066f6d2351d4d68310f8df847299911db0c7fbba2b25f82550a2db9f3ed8b0612c7777e4a415da2fbdcba9e96bcf92badb0bc602c24e60
+  metadata.gz: 5192d32ebb44a46f60df9303edb8cf2065edb42194e77e28576d2deba16dfbcc703c2c4f384f895a190cec68a327426936d1bfe48c157e8763e275a641945820
+  data.tar.gz: c27d77120587844546d644d5bdfa38897a771080fd89654242b39c00ca95125c8916b5dbb751ff47ed8def82ba50cded1daf2090dc6424854c2bac5343f366f3

data/README.md

@@ -1,18 +1,37 @@
 # Postqueue
 
-The postqueue gem implements a simple to use queue on top of postgresql. Since this implementation is using the SKIP LOCKED
-syntax, it needs PostgresQL >= 9.5.
+## Intro
 
-Why not using another queue implementation? postqueue comes with some extras:
+The postqueue gem implements a simple to use queue on top of postgresql. Note that while 
+a queue like this is typically used in a job queueing scenario, this document does not 
+talk about jobs, it talks about **queue items**; it also does not schedule a job, 
+it **enqueues** an item, and it does not executes a job, it **processes** queue items.
 
-- support for optimal handling of idempotent operations 
-- batch processing
-- searchable via SQL
+Why building an additional queue implementation? Compared to delayed_job or the other 
+usual suspects postqueue implements these features:
+
+- The item structure is intentionally kept super simple: an item is described by an
+  `op` field - a string - and an `id` field, an integer. In a typical usecase a 
+  queue item would describe an operation on a specific entity, where `op` names 
+  both the operation and the entity type and the `id` field would describe the 
+  individual entity.
+
+- With such a simplistic item structure the queue itself can be searched or 
+  otherwise evaluated using SQL. This also allows for **skipping duplicate entries** 
+  when enqueuing items (managed via a duplicate: argument when enqueuing) and for 
+  **batch processing** multple items in one go. 
+
+- With data being kept in a Postgresql database processing provides **transactional semantics**: 
+  an item failing to process stays in the queue. Error handling is kept simpe to a 
+  strategy of rescheduling items up to a specific maximum number of processing attemps.
+
+Please be aware that postqueue is using the SELECT .. FOR UPDATE SKIP LOCKED Postgresql syntax, 
+and therefore needs at least PostgresQL >= 9.5.
 
 ## Basic usage
 
 ```ruby
-queue = PostgresQL::Base.new
+queue = Postqueue.new
 queue.enqueue op: "product/reindex", entity_id: [12,13,14,15]
 queue.process do |op, entity_ids|
   # note: entity_ids is always an Array of ids.
@@ -27,12 +46,11 @@ end
 
 The process call will select a number of queue items for processing. They will all have 
 the same `op` attribute. The callback will receive the `op` attribute and the `entity_ids`
-of all queue entries selected for processing. The `processing` method will return the 
-return value of the block.
+of all queue entries selected for processing. The `processing` method will return the number
+of processed items.
 
-If no callback is given the return value will be the `[op, entity_ids]` values
-that would have been sent to the block. This is highly unrecommended though, since 
-when using a block to do processing errors and exceptions can properly be dealt with.
+If no callback is given the matching items are only removed from the queue without
+any processing.
 
 Postqueue.process also accepts the following arguments:
 
@@ -45,56 +63,90 @@ Example:
       # only handle up to 10 "product/reindex" entries
     end
 
-If the block fails, by either returning `false` or by raising an exception the queue will
-postpone processing these entries by an increasing amount of time, up until 
-`Postqueue::MAX_ATTEMPTS` failed attempts. The current MAX_ATTEMPTS definition
-leads to a maximum postpone interval (currently up to 190 seconds).
+If the block raises an exception the queue will postpone processing these entries 
+by an increasing amount of time, up until `queue.max_attempts` failed attempts.
+That value defaults to 5.
 
 If the queue is empty or no matching queue entry could be found, `Postqueue.process` 
-returns nil.
+returns 0.
+
+## Advanced usage
+
+### Concurrency
+
+Postqueue implements the following concurrency guarantees:
+
+- catastrophic DB failure and communication breakdown aside a queue item which is enqueued will eventually be processed successfully exactly once;
+- multiple consumers can work in parallel.
+
+Note that you should not share a Postqueue ruby object across threads - instead you should create
+process objects with the identical configuration.
+
+### Idempotent operations
+
+When enqueueing items duplicate idempotent operations are not enqueued. Whether or not an operation
+should be considered idempotent is defined when configuring the queue:
+
+     Postqueue.new do |queue|
+       queue.idempotent_operation "idempotent"
+     end
 
-### process a single entry
+### Processing a single entry
 
 Postqueue implements a shortcut to process only a single entry. Under the hood this 
 calls `Postqueue.process` with `batch_size` set to `1`:
 
-    Postqueue.process_one do |op, entity_ids|
-    end
+    queue.process_one
 
 Note that even though `process_one` will only ever process a single entry the 
-`entity_ids` parameter to the block is still an array (holding a single ID 
+`entity_ids` parameter to the callback is still an array (with a single ID entry 
 in that case).
 
-## idempotent operations
+### Migrating
+
+Postqueue comes with migration helpers:
+
+    # set up a table for use with postqueue.
+    Postqueue.migrate!(table_name = "postqueue")
 
-Postqueue comes with simple support for idempotent operations: if an operation is deemed
-idempotent it is not enqueued again if it can be found in the queue already. Note that 
-a queue item will be created if another item is currently being processed.
+    # set up a table for use with postqueue.
+    Postqueue.unmigrate!(table_name = "postqueue")
 
-    class Testqueue < Postqueue::Base
-      def idempotent?(entity_type:,op:)
-        op == "reindex"
-      end
+You can also set up your own table, as long as it is compatible.
+
+To use a non-default table or a non-default database, change the `item_class`
+attribute of the queue:
+
+    Postqueue.new do |queue|
+      queue.item_class = MyItemClass
     end
 
-## batch processing
+`MyItemClass` should inherit from Postqueue::Item and use the same or a compatible database
+structure.
+
+## Batch processing
 
-Often queue items can be processed in batches for a better performance of the entire system. 
-To allow batch processing for some items subclass `Postqueue::Base` and reimplement the 
-`batch_size?` method to return a suggested batch size for a specific operation. 
-The following implements a batch_size of 100 for all queue entries: 
+Often queue items can be batched together for a performant operation. To allow batch 
+processing for some items, configure the Postqueue to either set a `default_batch_size`
+or an operation-specific batch_size:
 
-    class Batchqueue < Postqueue::Base
-      def batch_size(op:)
-        100
-      end
+    Postqueue.new do |queue|
+      queue.default_batch_size = 100
+      queue.batch_sizes["batchable"] = 10
     end
 
-## Searchable via SQL
+## Test mode
 
-In contrast to other queue implementations available for Rubyists this queue formats
-entries in a way that makes it possible to query the queue via SQL. On the other 
-hand this queue also does not allow to enqueue arbitrary entries as these others do.
+During unit tests it is likely preferrable to process queue items in synchronous fashion (i.e. as they come in).
+You can enable this mode via:
+
+    Postqueue.async_processing = false
+
+You can also enable this on a queue-by-queue base via:
+
+    Postqueue.new do |queue|
+      queue.async_processing = false
+    end
 
 ## Installation
 
@@ -112,19 +164,21 @@ Or install it yourself as:
 
     $ gem install postqueue
 
-## Usage
-
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Make sure you have a local postgresql implementation of
-at least version 9.5. Add a `postqueue` user with a `postqueue` password, and create a `postqueue_test` database for it. 
-The script `./scripts/prepare_pg` can be helpful in establishing that.
+After checking out the repo, run `bin/setup` to install dependencies. Make sure you have 
+a local postgresql implementation of at least version 9.5. Add a `postqueue` user with 
+a `postqueue` password, and create a `postqueue_test` database for it. The script 
+`./scripts/prepare_pg` can be somewhat helpful in establishing that.
 
-Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive
+prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. 
 
-To release a new version, run `./scripts/release`, which will bump the version number, create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+To release a new version, run `./scripts/release`, which will bump the version number, 
+create a git tag for the version, push git commits and tags, and push the `.gem` file 
+to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 

data/lib/postqueue/item/enqueue.rb

@@ -0,0 +1,33 @@
+module Postqueue
+  class Item
+    # Enqueues an queue item. If the operation is duplicate, and an entry with
+    # the same combination of op and entity_id exists already, no new entry will
+    # be added to the queue.
+    #
+    # Returns the number of items that have been enqueued.
+    def self.enqueue(op:, entity_id:, ignore_duplicates: false)
+      if entity_id.is_a?(Enumerable)
+        return enqueue_many(op: op, entity_ids: entity_id, ignore_duplicates: ignore_duplicates)
+      end
+
+      if ignore_duplicates && where(op: op, entity_id: entity_id).present?
+        return 0
+      end
+
+      insert_item op: op, entity_id: entity_id
+      return 1
+    end
+
+    def self.enqueue_many(op:, entity_ids:, ignore_duplicates:) #:nodoc:
+      entity_ids.uniq! if ignore_duplicates
+
+      transaction do
+        entity_ids.each do |entity_id|
+          enqueue(op: op, entity_id: entity_id, ignore_duplicates: ignore_duplicates)
+        end
+      end
+
+      entity_ids.count
+    end
+  end
+end

data/lib/postqueue/item/inserter.rb

@@ -0,0 +1,29 @@
+require "active_record"
+
+module Postqueue
+  #
+  # An item class.
+  class Item < ActiveRecord::Base
+    module ActiveRecordInserter
+      def insert_item(op:, entity_id:)
+        create!(op: op, entity_id: entity_id)
+      end
+    end
+
+    module RawInserter
+      def prepared_inserter_statement
+        @prepared_inserter_statement ||= begin
+          name = "postqueue-insert-{table_name}-#{Thread.current.object_id}"
+          connection.raw_connection.prepare(name, "INSERT INTO #{table_name}(op, entity_id) VALUES($1, $2)")
+          name
+        end
+      end
+
+      def insert_item(op:, entity_id:)
+        connection.raw_connection.exec_prepared(prepared_inserter_statement, [op, entity_id])
+      end
+    end
+
+    extend RawInserter
+  end
+end

data/lib/postqueue/item.rb

@@ -1,8 +1,19 @@
 require "active_record"
 
 module Postqueue
+  #
+  # An item class.
   class Item < ActiveRecord::Base
     self.table_name = :postqueue
+
+    def self.postpone(ids)
+      connection.exec_query <<-SQL
+        UPDATE #{table_name}
+          SET failed_attempts = failed_attempts+1,
+              next_run_at = next_run_at + power(failed_attempts + 1, 1.5) * interval '10 second'
+          WHERE id IN (#{ids.join(',')})
+      SQL
+    end
   end
 
   def self.unmigrate!(table_name = "postqueue")
@@ -33,3 +44,6 @@ module Postqueue
     SQL
   end
 end
+
+require_relative "item/inserter"
+require_relative "item/enqueue"

data/lib/postqueue/logger.rb

@@ -0,0 +1,9 @@
+module Postqueue
+  def self.logger=(logger)
+    @logger ||= logger
+  end
+
+  def self.logger
+    @logger ||= Logger.new(STDOUT)
+  end
+end

data/lib/postqueue/queue/callback.rb

@@ -0,0 +1,60 @@
+module Postqueue
+  class MissingHandler < RuntimeError
+    attr_reader :queue, :op, :entity_ids
+
+    def initialize(queue:, op:, entity_ids:)
+      @queue = queue
+      @op = op
+      @entity_ids = entity_ids
+    end
+
+    def to_s
+      "#{queue.item_class.table_name}: Unknown operation #{op} with #{entity_ids.count} entities"
+    end
+  end
+
+  class Queue
+    Timing = Struct.new(:avg_queue_time, :max_queue_time, :total_processing_time, :processing_time)
+
+    def on(op, &block)
+      raise ArgumentError, "Invalid op #{op.inspect}, must be a string" unless op.is_a?(String)
+      callbacks[op] = block
+      self
+    end
+
+    private
+
+    def callbacks
+      @callbacks ||= {}
+    end
+
+    def callback_for(op:)
+      callbacks[op] || callbacks['*']
+    end
+
+    def on_missing_handler(op:, entity_ids:)
+      raise MissingHandler.new(queue: self, op: op, entity_ids: entity_ids)
+    end
+
+    private
+
+    def run_callback(op:, entity_ids:)
+      queue_times = item_class.find_by_sql <<-SQL
+        SELECT extract('epoch' from AVG(now() - created_at)) AS avg,
+               extract('epoch' from MAX(now() - created_at)) AS max
+        FROM #{item_class.table_name} WHERE entity_id IN (#{entity_ids.join(',')})
+      SQL
+      queue_time = queue_times.first
+
+      total_processing_time = Benchmark.realtime do
+        if callback = callback_for(op: op)
+          callback.call(op, entity_ids)
+        else
+          on_missing_handler(op: op, entity_ids: entity_ids)
+        end
+      end
+
+      Timing.new(queue_time.avg, queue_time.max, total_processing_time, total_processing_time / entity_ids.length)
+    end
+  end
+end

data/lib/postqueue/queue/logging.rb

@@ -0,0 +1,22 @@
+module Postqueue
+  # The Postqueue processor processes items in a single Postqueue table.
+  class Queue
+    private
+
+    def on_processing(op, entity_ids, timing)
+      msg = "processing '#{op}' for id(s) #{entity_ids.join(',')}: "
+      msg += "processing #{entity_ids.length} items took #{'%.3f secs' % timing.total_processing_time}"
+
+      msg += ", queue_time: avg: #{'%.3f secs' % timing.avg_queue_time}/max: #{'%.3f secs' % timing.max_queue_time}"
+      logger.info msg
+    end
+
+    def on_exception(exception, op, entity_ids)
+      logger.warn "processing '#{op}' for id(s) #{entity_ids.inspect}: caught #{exception}"
+    end
+
+    def logger
+      Postqueue.logger
+    end
+  end
+end

data/lib/postqueue/queue/processing.rb

@@ -0,0 +1,57 @@
+module Postqueue
+  # The Postqueue processor processes items in a single Postqueue table.
+  class Queue
+    # Processes up to batch_size entries
+    #
+    # process batch_size: 100
+    def process(op: nil, batch_size: 100)
+      item_class.transaction do
+        process_inside_transaction(op: op, batch_size: batch_size)
+      end
+    end
+
+    # processes a single entry
+    def process_one(op: nil, &block)
+      process(op: op, batch_size: 1)
+    end
+
+    def process_until_empty(op: nil, batch_size: 100)
+      count = 0
+      loop do
+        processed_items = process(op: op, batch_size: batch_size)
+        break if processed_items == 0
+        count += processed_items
+      end
+      count
+    end
+
+    private
+
+    # The actual processing. Returns [ op, [ ids-of-processed-items ] ] or nil
+    def process_inside_transaction(op:, batch_size:)
+      items = select_and_lock_batch(op: op, max_batch_size: batch_size)
+      match = items.first
+      return 0 unless match
+
+      entity_ids = items.map(&:entity_id)
+      timing = run_callback(op: match.op, entity_ids: entity_ids)
+
+      on_processing(match.op, entity_ids, timing)
+      item_class.where(id: items.map(&:id)).delete_all
+
+      # even though we try not to enqueue duplicates we cannot guarantee that,
+      # since concurrent enqueue transactions might still insert duplicates.
+      # That's why we explicitely remove all non-failed duplicates here.
+      if idempotent_operation?(match.op)
+        duplicates = select_and_lock_duplicates(op: match.op, entity_ids: entity_ids)
+        item_class.where(id: duplicates.map(&:id)).delete_all unless duplicates.empty?
+      end
+
+      entity_ids.length
+    rescue => e
+      on_exception(e, match.op, entity_ids)
+      item_class.postpone items.map(&:id)
+      raise
+    end
+  end
+end

data/lib/postqueue/{base → queue}/select_and_lock.rb

@@ -1,15 +1,20 @@
 module Postqueue
-  class Base
-    # Select and lock up to \a limit unlocked items in the queue.
+  class Queue
+    # Select and lock up to \a limit unlocked items in the queue. Used by
+    # select_and_lock_batch.
     def select_and_lock(relation, limit:)
       # Ordering by next_run_at and id should not strictly be necessary, but helps
       # processing entries in the passed in order when enqueued at the same time.
-      relation = relation.where("failed_attempts < ? AND next_run_at < ?", MAX_ATTEMPTS, Time.now).order(:next_run_at, :id)
+      relation = relation
+                 .select(:id, :entity_id, :op)
+                 .where("failed_attempts < ? AND next_run_at < ?", max_attemps, Time.now)
+                 .order(:next_run_at, :id)
 
       # FOR UPDATE SKIP LOCKED selects and locks entries, but skips those that
       # are already locked - preventing this transaction from being locked.
       sql = relation.to_sql + " FOR UPDATE SKIP LOCKED"
       sql += " LIMIT #{limit}" if limit
+
       item_class.find_by_sql(sql)
     end
 
@@ -20,25 +25,35 @@ module Postqueue
     # passed in, that one is chosen as a filter condition, otherwise the op value
     # of the first queue entry is used insteatd.
     #
-    # This method will at maximum select and lock batch_size items. If the batch_size
-    # returned by the #batch_size method is smaller than the passed in value here
-    # that one is used instead.
-    def select_and_lock_batch(op:, batch_size:, &_block)
+    # This method will at maximum select and lock \a batch_size items. 
+    # If the \a batch_size configured in the queue is smaller than the value
+    # passed in here that one is used instead.
+    #
+    # Returns an array of item objects.
+    def select_and_lock_batch(op:, max_batch_size:)
       relation = item_class.all
       relation = relation.where(op: op) if op
 
       match = select_and_lock(relation, limit: 1).first
       return [] unless match
 
-      batch_size = calculate_batch_size(op: match.op, max_batch_size: batch_size)
+      batch_size = calculate_batch_size(op: match.op, max_batch_size: max_batch_size)
       return [ match ] if batch_size <= 1
 
       batch_relation = relation.where(op: match.op)
       select_and_lock(batch_relation, limit: batch_size)
     end
 
+    def select_and_lock_duplicates(op:, entity_ids:)
+      raise ArgumentError, "Missing op argument" unless op
+      return [] if entity_ids.empty?
+
+      relation = item_class.where(op: op, entity_id: entity_ids)
+      select_and_lock(relation, limit: nil)
+    end
+
     def calculate_batch_size(op:, max_batch_size:)
-      recommended_batch_size = batch_size(op: op) || 1
+      recommended_batch_size = batch_size(op: op)
       return 1 if recommended_batch_size < 2
       return recommended_batch_size unless max_batch_size
       max_batch_size < recommended_batch_size ? max_batch_size : recommended_batch_size

data/lib/postqueue/queue.rb

@@ -0,0 +1,61 @@
+module Postqueue
+  class Queue
+    # The AR::Base class to use. You would only change this if you want to run
+    # the queue in a different database or in a different table.
+    attr_accessor :item_class
+
+    # The default batch size. Will be used if no specific batch size is defined
+    # for an operation.
+    attr_accessor :default_batch_size
+
+    # batch size for a given op
+    attr_reader :batch_sizes
+
+    # maximum number of processing attempts.
+    attr_reader :max_attemps
+
+    def async_processing?
+      @async_processing
+    end
+
+    attr_writer :async_processing
+
+    def initialize(&block)
+      @batch_sizes = {}
+      @item_class = ::Postqueue::Item
+      @default_batch_size = 1
+      @max_attemps = 5
+      @async_processing = Postqueue.async_processing?
+
+      yield self if block
+    end
+
+    def batch_size(op:)
+      batch_sizes[op] || default_batch_size || 1
+    end
+
+    def idempotent_operations
+      @idempotent_operations ||= {}
+    end
+
+    def idempotent_operation?(op)
+      idempotent_operations.fetch(op) { idempotent_operations.fetch('*', false)  }
+    end
+
+    def idempotent_operation(op, flag = true)
+      idempotent_operations[op] = flag
+    end
+
+    def enqueue(op:, entity_id:)
+      enqueued_items = item_class.enqueue op: op, entity_id: entity_id, ignore_duplicates: idempotent_operation?(op)
+      return unless enqueued_items > 0
+
+      process_until_empty(op: op) unless async_processing?
+    end
+  end
+end
+
+require_relative "queue/select_and_lock"
+require_relative "queue/processing"
+require_relative "queue/callback"
+require_relative "queue/logging"

data/lib/postqueue/version.rb

@@ -1,3 +1,3 @@
 module Postqueue
-  VERSION = "0.2.1"
+  VERSION = "0.4.0"
 end

data/lib/postqueue.rb

@@ -1,8 +1,22 @@
-require "postqueue/item"
-require "postqueue/base"
-require "postqueue/version"
+require_relative "postqueue/logger"
+require_relative "postqueue/item"
+require_relative "postqueue/version"
+require_relative "postqueue/queue"
 
 module Postqueue
+  def self.new(*args, &block)
+    ::Postqueue::Queue.new(*args, &block)
+  end
+
+  def self.async_processing=(async_processing)
+    @async_processing = async_processing
+  end
+
+  def self.async_processing?
+    @async_processing
+  end
+
+  self.async_processing = true
 end
 
-# require 'postqueue/railtie' if defined?(Rails)
+# require_relative 'postqueue/railtie' if defined?(Rails)

data/lib/tracker/advisory_lock.rb

@@ -0,0 +1,39 @@
+module AdvisoryLock
+  ADVISORY_LOCK = self
+
+  def self.included(base)
+    base.extend ClassMethods
+  end
+
+  module ClassMethods
+    def exclusive(lock_identifier = self.class.name.hash, &block)
+      ADVISORY_LOCK.exclusive(lock_identifier, connection, &block)
+    end
+  end
+
+  module Implementation
+    def exclusive(lock_identifier, connection = ActiveRecord::Base.connection, &_block)
+      if obtained_lock?(lock_identifier, connection)
+        begin
+          yield
+        ensure
+          release_lock(lock_identifier, connection)
+        end
+      else
+        raise "Cannot get lock #{lock_identifier.inspect}"
+      end
+    end
+
+    private
+
+    def obtained_lock?(lock_identifier, connection)
+      connection.select_value("select pg_try_advisory_lock(#{lock_identifier})")
+    end
+
+    def release_lock(lock_identifier, connection)
+      connection.execute "select pg_advisory_unlock(#{lock_identifier})"
+    end
+  end
+
+  extend Implementation
+end