postqueue 0.1.0 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: e0428f06b20e8b4558deecc920b4b02e625b4195
-  data.tar.gz: 7c295c6e46e1cc1cf18ef11e9dd2f3425d3fe9d3
+  metadata.gz: 7b6f8a8ad0f1aef1b565a223153f6280a9361314
+  data.tar.gz: 4dbf3727e217a8c248ce37fc78715aaf9080f0ea
 SHA512:
-  metadata.gz: 54ec8abb1c5ddb4a77d9676a944fda97f77f97b6c617c497929df93d523cdcdcb08cd2d8d79d1ba56745d7244d2fb6def5bc1692290aedf71d7fe1f495c23d0b
-  data.tar.gz: 2620f5068b77289253313a4de4d98882f18ae8d7c3c80a54d397006f007cf8b36f87cc951449614cb6fe49dd625eebb1cd0378e63b9f5e2eb8b20f32bc79db8d
+  metadata.gz: 569162000045134eb3107231ed94c6b52e15f5aed0e47ec2d681502037d142fe0cbd7ac9e85b2361e5fbeced0a0bd01fce6bee8cea4ef258cf0b2b19c46afb4e
+  data.tar.gz: 6c50d7e414a6b411e4066f6d2351d4d68310f8df847299911db0c7fbba2b25f82550a2db9f3ed8b0612c7777e4a415da2fbdcba9e96bcf92badb0bc602c24e60

data/README.md

@@ -13,56 +13,63 @@ Why not using another queue implementation? postqueue comes with some extras:
 
 ```ruby
 queue = PostgresQL::Base.new
-queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-queue.process do |op, entity_type, entity_ids|
+queue.enqueue op: "product/reindex", entity_id: [12,13,14,15]
+queue.process do |op, entity_ids|
   # note: entity_ids is always an Array of ids.
-  case "#{op}/#{entity_type}"
-  when "reindex/product"
+  case op
+  when "product/reindex"
     Product.index_many(Product.where(id: entity_ids))
   else
-    raise "Unsupported op/entity_type: #{op}/#{entity_type}"
+    raise "Unsupported op: #{op}"
   end
 end
 ```
 
-The callback will receive the `op` and `entity_type` attributes and the `entity_ids` of all queue entries
-selected for processing. If the block fails, by either returning `false` or by raising an exception the
-queue entries are postponed a bit, up until `Postqueue::MAX_ATTEMPTS` times (which currently is defined as 5).
+The process call will select a number of queue items for processing. They will all have 
+the same `op` attribute. The callback will receive the `op` attribute and the `entity_ids`
+of all queue entries selected for processing. The `processing` method will return the 
+return value of the block.
 
-The method will return the return value of the block. 
-
-If no callback is given the return value will be the `[op, entity_type, entity_ids]` values that would have
-been sent to the block. This is highly unrecommended though, since when using a block to do processing errors
-and exceptions can properly be dealt with.
+If no callback is given the return value will be the `[op, entity_ids]` values
+that would have been sent to the block. This is highly unrecommended though, since 
+when using a block to do processing errors and exceptions can properly be dealt with.
 
 Postqueue.process also accepts the following arguments:
 
-- `entity_type`: only process entries with this `entity_type`;
 - `op`: only process entries with this `op` value;
-- `batch_size`: when the first matching entry supports batch processing limit the size of the batch to the bassed in size.
+- `batch_size`: maximum number of items to process in one go.
 
 Example:
 
-    Postqueue.process(entity_type: 'Product', batch_size: 10) do |op, entity_type, entity_ids|
-      # only handle Product entries
+    Postqueue.process(op: 'product/reindex', batch_size: 10) do |op, entity_ids|
+      # only handle up to 10 "product/reindex" entries
     end
 
-If the queue is empty or no matching queue entry could be found, both `Postqueue.process` and `Postqueue.process_one` will return nil.
+If the block fails, by either returning `false` or by raising an exception the queue will
+postpone processing these entries by an increasing amount of time, up until 
+`Postqueue::MAX_ATTEMPTS` failed attempts. The current MAX_ATTEMPTS definition
+leads to a maximum postpone interval (currently up to 190 seconds).
+
+If the queue is empty or no matching queue entry could be found, `Postqueue.process` 
+returns nil.
 
 ### process a single entry
 
-For the simple just-give-me-the-next use case there is a shortcut, which only processed the first matching entry. Under the hood this calls Postqueue.process with `batch_size` set to `1`.
+Postqueue implements a shortcut to process only a single entry. Under the hood this 
+calls `Postqueue.process` with `batch_size` set to `1`:
 
-    Postqueue.process_one do |op, entity_type, entity_ids|
+    Postqueue.process_one do |op, entity_ids|
     end
 
-Note that even though `process_one` will only ever process a single entry the `entity_ids` parameter to the block is still an array (holding a single ID in that case).
+Note that even though `process_one` will only ever process a single entry the 
+`entity_ids` parameter to the block is still an array (holding a single ID 
+in that case).
 
 ## idempotent operations
 
-If queue items represent idempotent operations they need not be run repeatedly, but only once. To implement idempotent
-operations, subclass `Postqueue::Base` and reimplement the `idempotent?` method. The following marks all "reindex" 
-ops as idempotent:
+Postqueue comes with simple support for idempotent operations: if an operation is deemed
+idempotent it is not enqueued again if it can be found in the queue already. Note that 
+a queue item will be created if another item is currently being processed.
 
     class Testqueue < Postqueue::Base
       def idempotent?(entity_type:,op:)
@@ -72,22 +79,22 @@ ops as idempotent:
 
 ## batch processing
 
-Often queue items can be processed in batches for a better performance of the entire system. To enable
-batch processing for some items subclass `Postqueue::Base` and reimplement the `batch_size?` method
-to return a suggested batch_size for a specific operation. The following implements a batch_size of 100
-for all queue entries: 
+Often queue items can be processed in batches for a better performance of the entire system. 
+To allow batch processing for some items subclass `Postqueue::Base` and reimplement the 
+`batch_size?` method to return a suggested batch size for a specific operation. 
+The following implements a batch_size of 100 for all queue entries: 
 
-    class Testqueue < Postqueue::Base
-      def batch_size(entity_type:,op:)
+    class Batchqueue < Postqueue::Base
+      def batch_size(op:)
         100
       end
     end
 
 ## Searchable via SQL
 
-In contrast to other queue implementations available for Rubyists this queue formats entries in a way that
-makes it possible to query the queue via SQL. On the other hand this queue also does not allow to 
-enqueue arbitrary entries as these others do.
+In contrast to other queue implementations available for Rubyists this queue formats
+entries in a way that makes it possible to query the queue via SQL. On the other 
+hand this queue also does not allow to enqueue arbitrary entries as these others do.
 
 ## Installation
 

data/lib/postqueue/base/callback.rb

@@ -0,0 +1,23 @@
+module Postqueue
+  class Base
+    def run_callback(op:, entity_ids:, &_block)
+      queue_times = item_class.find_by_sql <<-SQL
+        SELECT extract('epoch' from AVG(now() - created_at)) AS avg,
+               extract('epoch' from MAX(now() - created_at)) AS max
+        FROM #{item_class.table_name} WHERE entity_id IN (#{entity_ids.join(',')})
+      SQL
+      queue_time = queue_times.first
+
+      # run callback.
+      result = [ op, entity_ids ]
+
+      total_processing_time = Benchmark.realtime do
+        result = yield(*result) if block_given?
+      end
+
+      timing = Timing.new(queue_time.avg, queue_time.max, total_processing_time, total_processing_time / entity_ids.length)
+
+      [ result, timing ]
+    end
+  end
+end

data/lib/postqueue/base/enqueue.rb

@@ -1,19 +1,31 @@
 module Postqueue
   class Base
-    def enqueue(op:, entity_type:, entity_id:)
-      # An optimized code path, as laid out below, is 4 times as fast.
-      # However, exec_query changed from Rails 4 to Rails 5.
+    # Enqueues an queue item. If the operation is duplicate, and an entry with
+    # the same combination of op and entity_id exists already, no new entry will
+    # be added to the queue.
+    #
+    # [TODO] An optimized code path, talking directly to PG, might be faster by a factor of 4 or so.
+    def enqueue(op:, entity_id:, duplicate: true)
+      if entity_id.is_a?(Array)
+        enqueue_many(op: op, entity_ids: entity_id, duplicate: duplicate)
+        return
+      end
 
-      # sql = "INSERT INTO #{item_class.table_name} (op, entity_type, entity_id) VALUES($1, $2, $3)"
-      # binds = [ ]
-      #
-      # binds << ActiveRecord::Attribute.from_user("name", op,  ::ActiveRecord::Type::String.new)
-      # binds << ActiveRecord::Attribute.from_user("entity_type", entity_type, ::ActiveRecord::Type::String.new)
-      # binds << ActiveRecord::Attribute.from_user("entity_id", entity_id, ::ActiveRecord::Type::Integer.new)
-      # # Note: Rails 4 does not understand prepare: true
-      # db.exec_query(sql, 'SQL', binds, prepare: true)
+      if !duplicate && item_class.where(op: op, entity_id: entity_id).present?
+        return
+      end
 
-      item_class.create!(op: op, entity_type: entity_type, entity_id: entity_id)
+      item_class.create!(op: op, entity_id: entity_id)
+    end
+
+    private
+
+    def enqueue_many(op:, entity_ids:, duplicate:) #:nodoc:
+      item_class.transaction do
+        entity_ids.each do |entity_id|
+          enqueue(op: op, entity_id: entity_id, duplicate: duplicate)
+        end
+      end
     end
   end
 end

data/lib/postqueue/base/processing.rb

@@ -2,125 +2,55 @@ module Postqueue
   MAX_ATTEMPTS = 5
 
   class Base
-
     # Processes many entries
     #
     # process batch_size: 100
-    def process(entity_type:nil, op:nil, batch_size:100, &block)
+    def process(op: nil, batch_size: 100, &block)
       status, result = item_class.transaction do
-        process_inside_transaction(entity_type: entity_type, op: op, batch_size: batch_size, &block)
+        process_inside_transaction(op: op, batch_size: batch_size, &block)
       end
 
       raise result if status == :err
       result
     end
 
-    def process_one(entity_type:nil, op:nil, &block)
-      process(entity_type: entity_type, op: op, batch_size: 1, &block)
-    end
-
-    def idempotent?(entity_type:, op:)
-      false
-    end
-
-    def batch_size(entity_type:, op:)
-      10
+    def process_one(op: nil, &block)
+      process(op: op, batch_size: 1, &block)
     end
 
     private
 
-    # Select and lock up to \a limit unlocked items in the queue.
-    def select_and_lock(relation, limit:)
-      relation = relation.where("failed_attempts < ? AND next_run_at < ?", MAX_ATTEMPTS, Time.now).order(:next_run_at, :id)
-
-      sql = relation.to_sql + " FOR UPDATE SKIP LOCKED"
-      sql += " LIMIT #{limit}" if limit
-      items = item_class.find_by_sql(sql)
-
-      items
-    end
-
-    def calculate_batch_size(op:, entity_type:, batch_size:)
-      processor_batch_size = self.batch_size(op: op, entity_type: entity_type)
-      if !processor_batch_size || processor_batch_size < 2
-        1
-      elsif(!batch_size)
-        processor_batch_size
-      else
-        [ processor_batch_size, batch_size ].min
-      end
-    end
-
     # The actual processing. Returns [ :ok, number-of-items ] or  [ :err, exception ]
-    def process_inside_transaction(entity_type:, op:, batch_size:, &block)
-      relation = item_class.all
-      relation = relation.where(entity_type: entity_type) if entity_type
-      relation = relation.where(op: op) if op
-
-      first_match = select_and_lock(relation, limit: 1).first
-      return [ :ok, nil ] unless first_match
-      op, entity_type = first_match.op, first_match.entity_type
+    def process_inside_transaction(op:, batch_size:, &block)
+      batch = select_and_lock_batch(op: op, batch_size: batch_size)
 
-      # determine batch to process. Whether or not an operation can be batched is defined
-      # by the Base#batch_size method. If that signals batch processing by returning a
-      # number > 0, then the passed in batch_size provides an additional upper limit.
-      batch_size = calculate_batch_size(op: op, entity_type: entity_type, batch_size: batch_size)
-      if batch_size > 1
-        batch_relation = relation.where(entity_type: entity_type, op: op)
-        batch = select_and_lock(batch_relation, limit: batch_size)
-      else
-        batch = [ first_match ]
-      end
+      match = batch.first
+      return [ :ok, nil ] unless match
 
       entity_ids = batch.map(&:entity_id)
-
-      # If the current operation is idempotent we will mark additional queue items as
-      # in process.
-      if idempotent?(op: op, entity_type: entity_type)
-        entity_ids.uniq!
-        process_relations   = relation.where(entity_type: entity_type, op: op, entity_id: entity_ids)
-        items_in_processing = select_and_lock(process_relations, limit: nil)
-      else
-        items_in_processing = batch
-      end
-
-      items_in_processing_ids = items_in_processing.map(&:id)
-
-      queue_times = item_class.find_by_sql <<-SQL
-        SELECT extract('epoch' from AVG(now() - created_at)) AS avg, 
-               extract('epoch' from MAX(now() - created_at)) AS max
-        FROM #{item_class.table_name} WHERE entity_id IN (#{entity_ids.join(",")})
-      SQL
-      queue_time = queue_times.first
-      
-      # run callback.
-      result = [ op, entity_type, entity_ids ]
-
-      processing_time = Benchmark.realtime do
-        result = yield *result if block_given?
-      end
+      result, timing = run_callback(op: match.op, entity_ids: entity_ids, &block)
 
       # Depending on the result either reprocess or delete all items
       if result == false
-        postpone items_in_processing_ids
+        postpone batch.map(&:id)
       else
-        on_processing(op, entity_type, entity_ids, processing_time, queue_time)
-        item_class.where(id: items_in_processing_ids).delete_all 
+        on_processing(match.op, entity_ids, timing)
+        item_class.where(id: batch.map(&:id)).delete_all
       end
 
       [ :ok, result ]
     rescue => e
-      on_exception(e, op, entity_type, entity_ids)
-      postpone items_in_processing_ids
+      on_exception(e, match.op, entity_ids)
+      postpone batch.map(&:id)
       [ :err, e ]
     end
 
     def postpone(ids)
       item_class.connection.exec_query <<-SQL
-        UPDATE #{item_class.table_name} 
-          SET failed_attempts = failed_attempts+1, 
-              next_run_at = next_run_at + power(failed_attempts + 1, 1.5) * interval '10 second' 
-          WHERE id IN (#{ids.join(",")})
+        UPDATE #{item_class.table_name}
+          SET failed_attempts = failed_attempts+1,
+              next_run_at = next_run_at + power(failed_attempts + 1, 1.5) * interval '10 second'
+          WHERE id IN (#{ids.join(',')})
       SQL
     end
   end

data/lib/postqueue/base/select_and_lock.rb

@@ -0,0 +1,47 @@
+module Postqueue
+  class Base
+    # Select and lock up to \a limit unlocked items in the queue.
+    def select_and_lock(relation, limit:)
+      # Ordering by next_run_at and id should not strictly be necessary, but helps
+      # processing entries in the passed in order when enqueued at the same time.
+      relation = relation.where("failed_attempts < ? AND next_run_at < ?", MAX_ATTEMPTS, Time.now).order(:next_run_at, :id)
+
+      # FOR UPDATE SKIP LOCKED selects and locks entries, but skips those that
+      # are already locked - preventing this transaction from being locked.
+      sql = relation.to_sql + " FOR UPDATE SKIP LOCKED"
+      sql += " LIMIT #{limit}" if limit
+      item_class.find_by_sql(sql)
+    end
+
+    # returns a batch of queue items for processing. These queue items are choosen
+    # depending on the passed in op: and batch_size: settings (if any).
+    #
+    # All selected queue items will have the same op value. If an op: value is
+    # passed in, that one is chosen as a filter condition, otherwise the op value
+    # of the first queue entry is used insteatd.
+    #
+    # This method will at maximum select and lock batch_size items. If the batch_size
+    # returned by the #batch_size method is smaller than the passed in value here
+    # that one is used instead.
+    def select_and_lock_batch(op:, batch_size:, &_block)
+      relation = item_class.all
+      relation = relation.where(op: op) if op
+
+      match = select_and_lock(relation, limit: 1).first
+      return [] unless match
+
+      batch_size = calculate_batch_size(op: match.op, max_batch_size: batch_size)
+      return [ match ] if batch_size <= 1
+
+      batch_relation = relation.where(op: match.op)
+      select_and_lock(batch_relation, limit: batch_size)
+    end
+
+    def calculate_batch_size(op:, max_batch_size:)
+      recommended_batch_size = batch_size(op: op) || 1
+      return 1 if recommended_batch_size < 2
+      return recommended_batch_size unless max_batch_size
+      max_batch_size < recommended_batch_size ? max_batch_size : recommended_batch_size
+    end
+  end
+end

data/lib/postqueue/base.rb

@@ -1,7 +1,14 @@
 module Postqueue
+  Timing = Struct.new(:avg_queue_time, :max_queue_time, :total_processing_time, :processing_time)
+
   class Base
     private
 
+    def batch_size(op:)
+      _ = op
+      1
+    end
+
     def item_class
       Postqueue::Item
     end
@@ -10,23 +17,25 @@ module Postqueue
       Postqueue.logger
     end
 
-    def on_processing(op, entity_type, entity_ids, runtime, queue_time)
-      logger.info "processing '#{op}/#{entity_type}' for id(s) #{entity_ids.join(",")}: processing #{entity_ids.length} items took #{"%.3f msecs" % runtime}, queue_time: avg: #{"%.3f msecs" % queue_time.avg}/max: #{"%.3f msecs" % queue_time.max}"
+    def on_processing(op, entity_ids, timing)
+      msg = "processing '#{op}' for id(s) #{entity_ids.join(',')}: "
+      msg += "processing #{entity_ids.length} items took #{'%.3f msecs' % timing.total_processing_time}"
+
+      msg += ", queue_time: avg: #{'%.3f msecs' % timing.avg_queue_time}/max: #{'%.3f msecs' % timing.max_queue_time}"
+      logger.info msg
     end
 
-    def on_exception(exception, op, entity_type, entity_ids)
-      logger.warn "processing '#{op}/#{entity_type}' for id(s) #{entity_ids.join(",")}: caught #{exception}"
+    def on_exception(exception, op, entity_ids)
+      logger.warn "processing '#{op}' for id(s) #{entity_ids.inspect}: caught #{exception}"
     end
   end
 
   def self.logger
     Logger.new(STDERR)
   end
-
-  def self.new
-    Base.new
-  end
 end
 
-require 'postqueue/base/enqueue'
-require 'postqueue/base/processing'
+require "postqueue/base/enqueue"
+require "postqueue/base/select_and_lock"
+require "postqueue/base/processing"
+require "postqueue/base/callback"

data/lib/postqueue/item.rb

@@ -14,16 +14,21 @@ module Postqueue
   def self.migrate!(table_name = "postqueue")
     Item.connection.execute <<-SQL
     CREATE TABLE #{table_name} (
-      id          SERIAL PRIMARY KEY, 
+      id          SERIAL PRIMARY KEY,
       op          VARCHAR,
-      entity_type VARCHAR,
       entity_id   INTEGER NOT NULL DEFAULT 0,
       created_at  timestamp without time zone NOT NULL DEFAULT (now() at time zone 'utc'),
       next_run_at timestamp without time zone NOT NULL DEFAULT (now() at time zone 'utc'),
       failed_attempts INTEGER NOT NULL DEFAULT 0
     );
 
+    -- This index should be usable to find duplicate duplicates in the table. While
+    -- we search for entries with matching op and entity_id, we assume that entity_id
+    -- has a much higher cardinality.
     CREATE INDEX #{table_name}_idx1 ON #{table_name}(entity_id);
+
+    -- This index should help picking the next entries to run. Otherwise a full tablescan
+    -- would be necessary whenevr we check out items.
     CREATE INDEX #{table_name}_idx2 ON #{table_name}(next_run_at);
     SQL
   end

data/lib/postqueue/version.rb

@@ -1,3 +1,3 @@
 module Postqueue
-  VERSION = '0.1.0'
+  VERSION = "0.2.1"
 end

data/lib/postqueue.rb

@@ -1,6 +1,6 @@
-require 'postqueue/item'
-require 'postqueue/base'
-require 'postqueue/version'
+require "postqueue/item"
+require "postqueue/base"
+require "postqueue/version"
 
 module Postqueue
 end

data/spec/postqueue/enqueue_spec.rb

@@ -1,23 +1,58 @@
-require 'spec_helper'
+require "spec_helper"
 
-describe 'enqueuing' do
-  let(:queue) { Postqueue.new }
+describe "enqueuing" do
+  let(:queue) { Postqueue::Base.new }
+  let(:item)  { queue.items.first }
 
-  before do
-    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+  context "when enqueueing entries" do
+    before do
+      queue.enqueue op: "myop", entity_id: 12
+    end
+
+    it "enqueues items" do
+      expect(item.op).to eq("myop")
+      expect(item.entity_id).to eq(12)
+    end
+
+    it "sets defaults" do
+      expect(item.created_at).to be > (Time.now - 1.second)
+      expect(item.next_run_at).to be > (Time.now - 1.second)
+      expect(item.failed_attempts).to eq(0)
+    end
   end
 
-  let(:item) { Postqueue::Item.first }
+  context "when enqueueing identical duplicate entries" do
+    before do
+      queue.enqueue op: "duplicate", entity_id: 12, duplicate: duplicate
+      queue.enqueue op: "duplicate", entity_id: 13, duplicate: duplicate
+      queue.enqueue op: "duplicate", entity_id: 12, duplicate: duplicate
+      queue.enqueue op: "duplicate", entity_id: 12, duplicate: duplicate
+      queue.enqueue op: "duplicate", entity_id: 12, duplicate: duplicate
+      queue.enqueue op: "no-duplicate", entity_id: 13, duplicate: duplicate
+    end
+
+    context "when duplicates are permitted" do
+      let(:duplicate) { true }
+
+      it "does not skip duplicates" do
+        expect(items.map(&:entity_id)).to eq([12, 13, 12, 12, 12, 13])
+      end
+    end
+
+    context "when duplicates are not permitted" do
+      let(:duplicate) { false }
 
-  it 'enqueues items' do
-    expect(item.op).to eq("myop")
-    expect(item.entity_type).to eq("mytype")
-    expect(item.entity_id).to eq(12)
+      it "skips later duplicates" do
+        expect(items.map(&:entity_id)).to eq([12, 13, 13])
+      end
+    end
   end
 
-  it 'sets defaults' do
-    expect(item.created_at).to be > (Time.now - 1.second)
-    expect(item.next_run_at).to be > (Time.now - 1.second)
-    expect(item.failed_attempts).to eq(0)
+  context "when enqueueing many entries" do
+    it "adds all entries skipping duplicates" do
+      queue.enqueue op: "duplicate", entity_id: 12, duplicate: false
+      queue.enqueue op: "duplicate", entity_id: [13, 12, 12, 13, 14], duplicate: false
+      expect(items.map(&:entity_id)).to eq([12, 13, 14])
+    end
   end
 end

data/spec/postqueue/postqueue_spec.rb

@@ -1,4 +1,4 @@
-require 'spec_helper'
+require "spec_helper"
 
 describe "::Postqueue" do
   it "reports a version" do

data/spec/postqueue/process_errors_spec.rb

@@ -1,17 +1,17 @@
-require 'spec_helper'
+require "spec_helper"
 
 describe "::queue.process_one" do
-  let(:queue) { Postqueue.new }
+  let(:queue) { Postqueue::Base.new }
 
   class E < RuntimeError; end
 
   before do
-    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+    queue.enqueue op: "mytype", entity_id: 12
   end
 
   context "block raises an exception" do
     before do
-      expect { queue.process_one do |op, type, ids| raise E end }.to raise_error(E)
+      expect { queue.process_one { |_op, _type, _ids| raise E } }.to raise_error(E)
     end
 
     it "reraises the exception" do
@@ -29,7 +29,7 @@ describe "::queue.process_one" do
 
   context "block returns false" do
     before do
-      @result = queue.process_one do |op, type, ids| false end
+      @result = queue.process_one { |_op, _type, _ids| false }
     end
 
     it "returns false" do
@@ -51,7 +51,10 @@ describe "::queue.process_one" do
       items.update_all(failed_attempts: Postqueue::MAX_ATTEMPTS)
 
       @called_block = 0
-      @result = queue.process_one do called_block += 1; false end
+      @result = queue.process_one do
+        @called_block += 1
+        false
+      end
     end
 
     it "does not call the block" do

data/spec/postqueue/process_spec.rb

@@ -0,0 +1,80 @@
+require "spec_helper"
+
+describe "::queue.process" do
+  class Testqueue < Postqueue::Base
+    def batch_size(op:)
+      _ = op
+      10
+    end
+  end
+
+  let(:queue) { Testqueue.new }
+
+  describe "basics" do
+    before do
+      queue.enqueue op: "myop", entity_id: 12
+      queue.enqueue op: "myop", entity_id: 13
+      queue.enqueue op: "myop", entity_id: 14
+    end
+
+    it "processes the first entry" do
+      r = queue.process_one
+      expect(r).to eq(["myop", [12]])
+      expect(items.map(&:entity_id)).to contain_exactly(13, 14)
+    end
+
+    it "honors search conditions" do
+      queue.enqueue(op: "otherop", entity_id: 112)
+
+      r = queue.process_one(op: "otherop")
+      expect(r).to eq(["otherop", [112]])
+      expect(items.map(&:entity_id)).to contain_exactly(12, 13, 14)
+    end
+
+    it "yields a block and returns its return value" do
+      queue.enqueue op: "otherop", entity_id: 112
+      r = queue.process_one(op: "otherop") do |op, ids|
+        expect(op).to eq("otherop")
+        expect(ids).to eq([112])
+        "yihaa"
+      end
+
+      expect(r).to eq("yihaa")
+      expect(items.map(&:entity_id)).to contain_exactly(12, 13, 14)
+    end
+  end
+
+  context "when having entries with different entity_type and op" do
+    before do
+      queue.enqueue op: "myop", entity_id: 12
+      queue.enqueue op: "myop", entity_id: 13
+      queue.enqueue op: "otherop", entity_id: 14
+      queue.enqueue op: "myop", entity_id: 15
+      queue.enqueue op: "otherop", entity_id: 16
+    end
+
+    it "processes one entries" do
+      r = queue.process batch_size: 1
+      expect(r).to eq(["myop", [12]])
+      expect(items.map(&:entity_id)).to contain_exactly(13, 14, 15, 16)
+    end
+
+    it "processes two entries" do
+      r = queue.process batch_size: 2
+      expect(r).to eq(["myop", [12, 13]])
+      expect(items.map(&:entity_id)).to contain_exactly(14, 15, 16)
+    end
+
+    it "processes only matching entries when asked for more" do
+      r = queue.process
+      expect(r).to eq(["myop", [12, 13, 15]])
+      expect(items.map(&:entity_id)).to contain_exactly(14, 16)
+    end
+
+    it "honors search conditions" do
+      r = queue.process(op: "otherop")
+      expect(r).to eq(["otherop", [14, 16]])
+      expect(items.map(&:entity_id)).to contain_exactly(12, 13, 15)
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -1,24 +1,22 @@
-path = File.expand_path('../../mpx/lib', __FILE__)
+path = File.expand_path("../../mpx/lib", __FILE__)
 $LOAD_PATH.unshift(path) unless $LOAD_PATH.include?(path)
 
-ENV['RACK_ENV'] = 'test'
+ENV["RACK_ENV"] = "test"
 
-require 'rspec'
-require 'pry'
-require 'simplecov'
+require "rspec"
+require "pry"
+require "simplecov"
 
 SimpleCov.start do
   minimum_coverage 94
 end
 
-require 'postqueue'
-require './spec/support/configure_active_record'
-
-$logger = Logger.new(File.open("log/test.log", "a"))
+require "postqueue"
+require "./spec/support/configure_active_record"
 
 module Postqueue
   def self.logger
-    $logger
+    @logger ||= Logger.new(File.open("log/test.log", "a"))
   end
 end
 
@@ -28,10 +26,10 @@ end
 
 RSpec.configure do |config|
   config.run_all_when_everything_filtered = true
-  config.filter_run focus: (ENV['CI'] != 'true')
+  config.filter_run focus: (ENV["CI"] != "true")
   config.expect_with(:rspec) { |c| c.syntax = :expect }
-  config.order = 'random'
+  config.order = "random"
 
-  config.before(:all) { }
-  config.after { }
+  config.before(:all) {}
+  config.after {}
 end

data/spec/support/configure_active_record.rb

@@ -1,15 +1,14 @@
-require 'active_record'
-require_relative './models'
+require "active_record"
 
 $LOAD_PATH << File.dirname(__FILE__)
 
-ActiveRecord::Base.establish_connection(adapter: 'postgresql', 
-                                        database: 'postqueue_test', 
-                                        username: 'postqueue', 
-                                        password: 'postqueue')
+ActiveRecord::Base.establish_connection(adapter: "postgresql",
+                                        database: "postqueue_test",
+                                        username: "postqueue",
+                                        password: "postqueue")
 
-require_relative 'schema.rb'
-require_relative 'models.rb'
+# require_relative "schema.rb"
+# require_relative "models.rb"
 
 Postqueue.unmigrate!
 Postqueue.migrate!
@@ -18,7 +17,7 @@ RSpec.configure do |config|
   config.around(:each) do |example|
     ActiveRecord::Base.connection.transaction do
       example.run
-      raise ActiveRecord::Rollback, 'Clean up'
+      raise ActiveRecord::Rollback, "Clean up"
     end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: postqueue
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.1
 platform: ruby
 authors:
 - radiospiel
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-12-10 00:00:00.000000000 Z
+date: 2016-12-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -108,6 +108,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: pg
   requirement: !ruby/object:Gem::Requirement
@@ -132,19 +146,18 @@ files:
 - README.md
 - lib/postqueue.rb
 - lib/postqueue/base.rb
+- lib/postqueue/base/callback.rb
 - lib/postqueue/base/enqueue.rb
 - lib/postqueue/base/processing.rb
+- lib/postqueue/base/select_and_lock.rb
 - lib/postqueue/item.rb
 - lib/postqueue/version.rb
 - spec/postqueue/enqueue_spec.rb
-- spec/postqueue/idempotent_queue_spec.rb
 - spec/postqueue/postqueue_spec.rb
 - spec/postqueue/process_errors_spec.rb
-- spec/postqueue/process_one_spec.rb
+- spec/postqueue/process_spec.rb
 - spec/spec_helper.rb
 - spec/support/configure_active_record.rb
-- spec/support/models.rb
-- spec/support/schema.rb
 homepage: https://github.com/radiospiel/postqueue
 licenses:
 - MIT

data/spec/postqueue/idempotent_queue_spec.rb

@@ -1,89 +0,0 @@
-require 'spec_helper'
-
-describe "Idempotent queue" do
-  class Testqueue < Postqueue::Base
-    def idempotent?(entity_type:,op:)
-      true
-    end
-
-    def batch_size(entity_type:,op:)
-      100
-    end
-  end
-
-  let(:queue) { Testqueue.new }
-
-  context 'when having entries with the same entity_type and op' do
-    before do
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 14
-    end
-
-    it "processes one entries" do
-      r = queue.process batch_size: 1
-      expect(r).to eq(["myop", "mytype", [12]])
-      expect(items.map(&:entity_id)).to contain_exactly(13, 14)
-    end
-
-    it "processes two entries" do
-      r = queue.process batch_size: 2
-      expect(r).to eq(["myop", "mytype", [12, 13]])
-      expect(items.map(&:entity_id)).to contain_exactly(14)
-    end
-
-    it "processes many entries" do
-      r = queue.process
-      expect(r).to eq(["myop", "mytype", [12, 13, 14]])
-      expect(items.map(&:entity_id)).to contain_exactly()
-    end
-  end
-
-  context 'when having entries with different entity_type and op' do
-    before do
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-      queue.enqueue op: "otherop", entity_type: "mytype", entity_id: 14
-      queue.enqueue op: "myop", entity_type: "othertype", entity_id: 15
-      queue.enqueue op: "otherop", entity_type: "othertype", entity_id: 16
-    end
-
-    it "processes one entries" do
-      r = queue.process batch_size: 1
-      expect(r).to eq(["myop", "mytype", [12]])
-      expect(items.map(&:entity_id)).to contain_exactly(13, 14, 15, 16)
-    end
-
-    it "processes two entries" do
-      r = queue.process batch_size: 2
-      expect(r).to eq(["myop", "mytype", [12, 13]])
-      expect(items.map(&:entity_id)).to contain_exactly(14, 15, 16)
-    end
-
-    it "processes only matching entries when asked for more" do
-      r = queue.process
-      expect(r).to eq(["myop", "mytype", [12, 13]])
-      expect(items.map(&:entity_id)).to contain_exactly(14, 15, 16)
-    end
-
-    it "honors search conditions" do
-      r = queue.process(op: "otherop")
-      expect(r).to eq(["otherop", "mytype", [14]])
-      expect(items.map(&:entity_id)).to contain_exactly(12, 13, 15, 16)
-    end
-  end
-  
-  context 'when having duplicate entries' do
-    before do
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-    end
-
-    it "removes duplicates from the queue" do
-      r = queue.process batch_size: 1
-      expect(r).to eq(["myop", "mytype", [12]])
-      expect(items.map(&:entity_id)).to contain_exactly(13)
-    end
-  end
-end

data/spec/postqueue/process_one_spec.rb

@@ -1,40 +0,0 @@
-require 'spec_helper'
-
-describe "::queue.process_one" do
-  let(:queue) { Postqueue.new }
-
-  before do
-    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 14
-  end
-
-  let(:processor) { Postqueue::Processor.new }
-
-  it "processes one entry" do
-    r = queue.process_one
-    expect(r).to eq(["myop", "mytype", [12]])
-    expect(items.map(&:entity_id)).to contain_exactly(13, 14)
-  end
-
-  it "honors search conditions" do
-    queue.enqueue op: "otherop", entity_type: "mytype", entity_id: 112
-
-    r = queue.process_one(op: "otherop")
-    expect(r).to eq(["otherop", "mytype", [112]])
-    expect(items.map(&:entity_id)).to contain_exactly(12, 13, 14)
-  end
-
-  it "yields a block and returns it" do
-    queue.enqueue op: "otherop", entity_type: "mytype", entity_id: 112 
-    r = queue.process_one(op: "otherop") do |op, type, ids|
-      expect(op).to eq("otherop")
-      expect(type).to eq("mytype")
-      expect(ids).to eq([112])
-      "yihaa"
-    end
-
-    expect(r).to eq("yihaa")
-    expect(items.map(&:entity_id)).to contain_exactly(12, 13, 14)
-  end
-end

data/spec/support/models.rb

@@ -1,123 +0,0 @@
-__END__
-
-require 'ostruct'
-
-class MockAssocationInfo < OpenStruct
-  def virtual?; mode == :virtual; end
-  def belongs_to?; mode == :belongs_to; end
-  def habtm?; mode == :has_and_belongs_to_many; end
-end
-
-module AnalyticsReflectionStub
-  attr :analytics_reflection
-  def set_analytics_reflection(hsh)
-    @analytics_reflection = OpenStruct.new(hsh)
-  end
-
-  attr :analytics_parent
-  def set_analytics_parent(analytics_parent)
-    @analytics_parent = analytics_parent
-  end
-end
-
-class Unicorn < ActiveRecord::Base
-  extend AnalyticsReflectionStub
-
-  def self.name_without_prefix
-    'UnicornWithoutPrefix'
-  end
-
-  validates_presence_of :name
-
-  set_analytics_reflection associations_by_foreign_keys: {}, analytics_keys: [:name]
-end
-
-class Manticore < ActiveRecord::Base
-  def self.name_without_prefix
-    'ManticoreWithoutPrefix'
-  end
-end
-
-class Rider < ActiveRecord::Base
-end
-
-class Foal < ActiveRecord::Base
-  extend AnalyticsReflectionStub
-
-  belongs_to :parent, class_name: 'Unicorn'
-  has_and_belongs_to_many :riders, class_name: 'Rider'
-
-  set_analytics_reflection associations_by_foreign_keys: 
-    { 
-    parent_id: MockAssocationInfo.new(mode: :belongs_to, klass: Unicorn, name: :parent),
-    rider_ids: MockAssocationInfo.new(mode: :has_and_belongs_to_many, klass: Rider, name: :riders),
-    stable_ids: MockAssocationInfo.new(mode: :virtual, name: :stables)
-  },
-  analytics_keys: [:nick_name, :age, :parent]
-  set_analytics_parent :parent
-end
-
-class Pegasus < ActiveRecord::Base
-  extend AnalyticsReflectionStub
-
-  belongs_to :parent, class_name: 'Foal'
-
-  set_analytics_reflection associations_by_foreign_keys: { :parent_id => OpenStruct.new(klass: Foal, name: :parent) },
-    analytics_keys: [:nick_name, :age, :parent]
-  set_analytics_parent :parent
-
-  attr_reader :affiliation_id
-end
-
-class Dragon < ActiveRecord::Base
-  def self.name_without_prefix
-    'DragonWithoutPrefix'
-  end
-
-  def describe
-    'yihaa'
-  end
-
-  extend AnalyticsReflectionStub
-end
-
-class Asset < ActiveRecord::Base
-  extend AnalyticsReflectionStub
-  attr_reader :affiliation_id
-end
-
-class Product < ActiveRecord::Base
-  extend AnalyticsReflectionStub
-  attr_reader :affiliation_id
-end
-
-class User < ActiveRecord::Base
-  extend AnalyticsReflectionStub
-  attr_reader :affiliation_id
-
-  def analytics_title
-    "my analytics_title"
-  end
-end
-
-class Grouping < ActiveRecord::Base
-  extend AnalyticsReflectionStub
-  attr_reader :affiliation_id
-end
-
-class Group < Grouping
-end
-
-class MockProductAsset < ActiveRecord::Base
-  has_one :asset
-  has_one :product
-
-  attr_accessor :asset, :product
-end
-
-class MockGroupUser < ActiveRecord::Base
-  has_one :group
-  has_one :user
-
-  attr_accessor :group, :user
-end

data/spec/support/schema.rb

@@ -1,88 +0,0 @@
-__END__
-
-ActiveRecord::Schema.define do
-  self.verbose = false
-
-  create_table :unicorns, force: true do |t|
-    t.string  :name, null: false
-    t.string :affiliation_id
-  end
-
-  create_table :foals, force: true do |t|
-    t.integer   :parent_id, null: false
-    t.string    :nick_name
-    t.integer   :age
-    t.datetime  :created_at
-  end
-
-  create_table :riders, force: true do |t|
-    t.string    :nick_name
-  end
-
-  create_table :foal_riders, force: true do |t|
-    t.integer :rider_id
-    t.integer :foal_id
-  end
-
-  create_table :pegasus, force: true do |t|
-    t.integer   :parent_id, null: false
-    t.string    :nick_name
-    t.integer   :age
-    t.datetime  :created_at
-  end
-
-  create_table :mock_product_assets, force: true do |t|
-    t.integer   :asset_id
-    t.string    :access_level
-    t.integer   :product_id
-  end
-
-  create_table :mock_group_users, force: true do |t|
-    t.integer   :user_id
-    t.string    :access_level
-    t.integer   :group_id
-  end
-
-  create_table :assets, force: true
-
-  create_table :products, force: true
-
-  create_table :users, force: true do |t|
-    t.string    :title
-  end
-
-  create_table :groupings, force: true
-
-  create_table :manticores, force: true do |t|
-    t.string  :dummy_field
-  end
-
-  create_table :dragons, force: true do |t|
-    t.string  :full_name
-  end
-
-  execute "INSERT INTO unicorns(affiliation_id, id, name) VALUES('mpx', 1, 'Faith')"
-  execute "INSERT INTO unicorns(affiliation_id, id, name) VALUES('mpx', 2, 'Faery')"
-  execute "INSERT INTO unicorns(affiliation_id, id, name) VALUES('mpx', 3, 'Yaser')"
-
-  execute "INSERT INTO foals(id, parent_id, nick_name, age, created_at) VALUES(1, 1, 'Little Faith', 12, 0)"
-  execute "INSERT INTO foals(id, parent_id, nick_name, age, created_at) VALUES(2, 1, 'Faith Nick',   9, 0)"
-
-  execute "INSERT INTO riders(id, nick_name) VALUES(1, 'Storm Rider')"
-  execute "INSERT INTO riders(id, nick_name) VALUES(2, 'Desert Rider')"
-
-  execute "INSERT INTO foal_riders(rider_id, foal_id) VALUES(1, 1)"
-  execute "INSERT INTO foal_riders(rider_id, foal_id) VALUES(2, 1)"
-
-  execute "INSERT INTO pegasus(parent_id, nick_name, age, created_at) VALUES(1, 'Derpy', 12, 0)"
-
-  execute "INSERT INTO dragons(full_name) VALUES('Chrysophylax Dives')"
-  execute "INSERT INTO dragons(full_name) VALUES('Nepomuk')"
-  execute "INSERT INTO dragons(full_name) VALUES('Smaug')"
-
-  execute "INSERT INTO manticores(dummy_field) VALUES('Dumb Manticore')"
-
-  execute "INSERT INTO users(id, title) VALUES(67, 'sixtyseven')"
-
-  execute "INSERT INTO groupings(id) VALUES(42)"
-end