postqueue 0.0.10 → 0.0.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5ec904d462a89e0a99db3738cdeaac46cda9364d
-  data.tar.gz: b45a746b40cfe5521aea1ce3d96dab1b746e3972
+  metadata.gz: 6cd82270501112669519b0a37ae116d9b5f2ae36
+  data.tar.gz: ae81665e57301d5998ff4493d5db54d393d2fe68
 SHA512:
-  metadata.gz: d623d3d9bc87f4d17231b5954cb518d78f395171baa41ed4260edd757560f44b6aea98bf707f1db29708c85da95d9036476d06fc7acd72300e6fae465c5f6175
-  data.tar.gz: e3ffd8e1933f5de86446314e69f15540cb99a045c3a783db241fb5e699a58fbd13f4e86193445b4e79803699a7eca2ad711aa1f8b834293fcb1407429aa61685
+  metadata.gz: 0d79ebc63c90da1834c3b867aabbcf05cb03d8d45e0a99e9c043620a61770958b186a0c98ea80f5a4786358581f7d4b5f0671ecd1fee87b2b1c893211477fd2e
+  data.tar.gz: 0366cf7aa5322d3becd0f9c5fa801fda0a8950cd433303b3a10265f41d3d93e480617a27aa8e8ea64f9f47212931977c8c2e210dbb227998f844639779cbb87d

data/README.md

@@ -1,8 +1,93 @@
 # Postqueue
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/postqueue`. To experiment with that code, run `bin/console` for an interactive prompt.
+The postqueue gem implements a simple to use queue on top of postgresql. Since this implementation is using the SKIP LOCKED
+syntax, it needs PostgresQL >= 9.5.
 
-TODO: Delete this and the text above, and describe your gem
+Why not using another queue implementation? postqueue comes with some extras:
+
+- support for optimal handling of idempotent operations 
+- batch processing
+- searchable via SQL
+
+## Basic usage
+
+```ruby
+queue = PostgresQL::Base.new
+queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+queue.process do |op, entity_type, entity_ids|
+  # note: entity_ids is always an Array of ids.
+  case "#{op}/#{entity_type}"
+  when "reindex/product"
+    Product.index_many(Product.where(id: entity_ids))
+  else
+    raise "Unsupported op/entity_type: #{op}/#{entity_type}"
+  end
+end
+```
+
+The callback will receive the `op` and `entity_type` attributes and the `entity_ids` of all queue entries
+selected for processing. If the block fails, by either returning `false` or by raising an exception the
+queue entries are postponed a bit, up until `Postqueue::MAX_ATTEMPTS` times (which currently is defined as 5).
+
+The method will return the return value of the block. 
+
+If no callback is given the return value will be the `[op, entity_type, entity_ids]` values that would have
+been sent to the block. This is highly unrecommended though, since when using a block to do processing errors
+and exceptions can properly be dealt with.
+
+Postqueue.process also accepts the following arguments:
+
+- `entity_type`: only process entries with this `entity_type`;
+- `op`: only process entries with this `op` value;
+- `batch_size`: when the first matching entry supports batch processing limit the size of the batch to the bassed in size.
+
+Example:
+
+    Postqueue.process(entity_type: 'Product', batch_size: 10) do |op, entity_type, entity_ids|
+      # only handle Product entries
+    end
+
+If the queue is empty or no matching queue entry could be found, both `Postqueue.process` and `Postqueue.process_one` will return nil.
+
+### process a single entry
+
+For the simple just-give-me-the-next use case there is a shortcut, which only processed the first matching entry. Under the hood this calls Postqueue.process with `batch_size` set to `1`.
+
+    Postqueue.process_one do |op, entity_type, entity_ids|
+    end
+
+Note that even though `process_one` will only ever process a single entry the `entity_ids` parameter to the block is still an array (holding a single ID in that case).
+
+## idempotent operations
+
+If queue items represent idempotent operations they need not be run repeatedly, but only once. To implement idempotent
+operations, subclass `Postqueue::Base` and reimplement the `idempotent?` method. The following marks all "reindex" 
+ops as idempotent:
+
+    class Testqueue < Postqueue::Base
+      def idempotent?(entity_type:,op:)
+        op == "reindex"
+      end
+    end
+
+## batch processing
+
+Often queue items can be processed in batches for a better performance of the entire system. To enable
+batch processing for some items subclass `Postqueue::Base` and reimplement the `batch_size?` method
+to return a suggested batch_size for a specific operation. The following implements a batch_size of 100
+for all queue entries: 
+
+    class Testqueue < Postqueue::Base
+      def batch_size(entity_type:,op:)
+        100
+      end
+    end
+
+## Searchable via SQL
+
+In contrast to other queue implementations available for Rubyists this queue formats entries in a way that
+makes it possible to query the queue via SQL. On the other hand this queue also does not allow to 
+enqueue arbitrary entries as these others do.
 
 ## Installation
 
@@ -22,13 +107,17 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
-
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+After checking out the repo, run `bin/setup` to install dependencies. Make sure you have a local postgresql implementation of
+at least version 9.5. Add a `postqueue` user with a `postqueue` password, and create a `postqueue_test` database for it. 
+The script `./scripts/prepare_pg` can be helpful in establishing that.
+
+Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. 
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+To release a new version, run `./scripts/release`, which will bump the version number, create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 

data/lib/postqueue.rb

@@ -1,6 +1,5 @@
 require 'postqueue/item'
-require 'postqueue/enqueue'
-require 'postqueue/processing'
+require 'postqueue/base'
 require 'postqueue/version'
 
 module Postqueue

data/lib/postqueue/base.rb

@@ -0,0 +1,32 @@
+module Postqueue
+  class Base
+    private
+
+    def item_class
+      Postqueue::Item
+    end
+
+    def logger
+      Postqueue.logger
+    end
+
+    def on_processing(op, entity_type, entity_ids, runtime, queue_time)
+      logger.info "processing '#{op}/#{entity_type}' for id(s) #{entity_ids.join(",")}: processing #{entity_ids.length} items took #{"%.3f msecs" % runtime}, queue_time: avg: #{"%.3f msecs" % queue_time.avg}/max: #{"%.3f msecs" % queue_time.max}"
+    end
+
+    def on_exception(exception, op, entity_type, entity_ids)
+      logger.warn "processing '#{op}/#{entity_type}' for id(s) #{entity_ids.join(",")}: caught #{exception}"
+    end
+  end
+
+  def self.logger
+    Logger.new(STDERR)
+  end
+
+  def self.new
+    Base.new
+  end
+end
+
+require 'postqueue/base/enqueue'
+require 'postqueue/base/processing'

data/lib/postqueue/{enqueue.rb → base/enqueue.rb}

@@ -1,12 +1,10 @@
 module Postqueue
-  module Enqueue
-    Item = ::Postqueue::Item
-
+  class Base
     def enqueue(op:, entity_type:, entity_id:)
       # An optimized code path, as laid out below, is 4 times as fast.
       # However, exec_query changed from Rails 4 to Rails 5.
 
-      # sql = "INSERT INTO postqueue (op, entity_type, entity_id) VALUES($1, $2, $3)"
+      # sql = "INSERT INTO #{item_class.table_name} (op, entity_type, entity_id) VALUES($1, $2, $3)"
       # binds = [ ]
       #
       # binds << ActiveRecord::Attribute.from_user("name", op,  ::ActiveRecord::Type::String.new)
@@ -15,9 +13,7 @@ module Postqueue
       # # Note: Rails 4 does not understand prepare: true
       # db.exec_query(sql, 'SQL', binds, prepare: true)
 
-      Item.create!(op: op, entity_type: entity_type, entity_id: entity_id)
+      item_class.create!(op: op, entity_type: entity_type, entity_id: entity_id)
     end
   end
-
-  extend Enqueue
 end

data/lib/postqueue/base/processing.rb

@@ -0,0 +1,127 @@
+module Postqueue
+  MAX_ATTEMPTS = 5
+
+  class Base
+
+    # Processes many entries
+    #
+    # process batch_size: 100
+    def process(entity_type:nil, op:nil, batch_size:100, &block)
+      status, result = item_class.transaction do
+        process_inside_transaction(entity_type: entity_type, op: op, batch_size: batch_size, &block)
+      end
+
+      raise result if status == :err
+      result
+    end
+
+    def process_one(entity_type:nil, op:nil, &block)
+      process(entity_type: entity_type, op: op, batch_size: 1, &block)
+    end
+
+    def idempotent?(entity_type:, op:)
+      false
+    end
+
+    def batch_size(entity_type:, op:)
+      10
+    end
+
+    private
+
+    # Select and lock up to \a limit unlocked items in the queue.
+    def select_and_lock(relation, limit:)
+      relation = relation.where("failed_attempts < ? AND next_run_at < ?", MAX_ATTEMPTS, Time.now).order(:next_run_at, :id)
+
+      sql = relation.to_sql + " FOR UPDATE SKIP LOCKED"
+      sql += " LIMIT #{limit}" if limit
+      items = item_class.find_by_sql(sql)
+
+      items
+    end
+
+    def calculate_batch_size(op:, entity_type:, batch_size:)
+      processor_batch_size = self.batch_size(op: op, entity_type: entity_type)
+      if !processor_batch_size || processor_batch_size < 2
+        1
+      elsif(!batch_size)
+        processor_batch_size
+      else
+        [ processor_batch_size, batch_size ].min
+      end
+    end
+
+    # The actual processing. Returns [ :ok, number-of-items ] or  [ :err, exception ]
+    def process_inside_transaction(entity_type:, op:, batch_size:, &block)
+      relation = item_class.all
+      relation = relation.where(entity_type: entity_type) if entity_type
+      relation = relation.where(op: op) if op
+
+      first_match = select_and_lock(relation, limit: 1).first
+      return [ :ok, nil ] unless first_match
+      op, entity_type = first_match.op, first_match.entity_type
+
+      # determine batch to process. Whether or not an operation can be batched is defined
+      # by the Base#batch_size method. If that signals batch processing by returning a
+      # number > 0, then the passed in batch_size provides an additional upper limit.
+      batch_size = calculate_batch_size(op: op, entity_type: entity_type, batch_size: batch_size)
+      if batch_size > 1
+        batch_relation = relation.where(entity_type: entity_type, op: op)
+        batch = select_and_lock(batch_relation, limit: batch_size)
+      else
+        batch = [ first_match ]
+      end
+
+      entity_ids = batch.map(&:entity_id)
+
+      # If the current operation is idempotent we will mark additional queue items as
+      # in process.
+      if idempotent?(op: op, entity_type: entity_type)
+        entity_ids.uniq!
+        process_relations   = relation.where(entity_type: entity_type, op: op, entity_id: entity_ids)
+        items_in_processing = select_and_lock(process_relations, limit: nil)
+      else
+        items_in_processing = batch
+      end
+
+      items_in_processing_ids = items_in_processing.map(&:id)
+
+      queue_times = item_class.find_by_sql <<-SQL
+        SELECT extract('epoch' from AVG(now() - created_at)) AS avg, 
+               extract('epoch' from MAX(now() - created_at)) AS max
+        FROM #{item_class.table_name} WHERE entity_id IN (#{entity_ids.join(",")})
+      SQL
+      queue_time = queue_times.first
+      
+      # run callback.
+      result = [ op, entity_type, entity_ids ]
+
+      processing_time = Benchmark.realtime do
+        result = yield *result if block_given?
+      end
+
+      # Depending on the result either reprocess or delete all items
+      if result == false
+        postpone items_in_processing_ids
+      else
+        on_processing(op, entity_type, entity_ids, processing_time, queue_time)
+        item_class.where(id: items_in_processing_ids).delete_all 
+      end
+
+      [ :ok, result ]
+    rescue => e
+      on_exception(e, op, entity_type, entity_ids)
+      postpone items_in_processing_ids
+      [ :err, e ]
+    end
+
+    def postpone(ids)
+      item_class.connection.exec_query <<-SQL
+        UPDATE #{item_class.table_name} 
+          SET failed_attempts = failed_attempts+1, 
+              next_run_at = next_run_at + power(failed_attempts + 1, 1.5) * interval '10 second' 
+          WHERE id IN (#{ids.join(",")})
+      SQL
+    end
+  end
+end

data/lib/postqueue/item.rb

@@ -5,15 +5,15 @@ module Postqueue
     self.table_name = :postqueue
   end
 
-  def self.unmigrate!
+  def self.unmigrate!(table_name = "postqueue")
     Item.connection.execute <<-SQL
-      DROP TABLE IF EXISTS postqueue;
+      DROP TABLE IF EXISTS #{table_name};
     SQL
   end
 
-  def self.migrate!
+  def self.migrate!(table_name = "postqueue")
     Item.connection.execute <<-SQL
-    CREATE TABLE postqueue (
+    CREATE TABLE #{table_name} (
       id          SERIAL PRIMARY KEY, 
       op          VARCHAR,
       entity_type VARCHAR,
@@ -23,8 +23,8 @@ module Postqueue
       failed_attempts INTEGER NOT NULL DEFAULT 0
     );
 
-    CREATE INDEX postqueue_idx1 ON postqueue(entity_id);
-    CREATE INDEX postqueue_idx2 ON postqueue(next_run_at);
+    CREATE INDEX #{table_name}_idx1 ON #{table_name}(entity_id);
+    CREATE INDEX #{table_name}_idx2 ON #{table_name}(next_run_at);
     SQL
   end
 end

data/lib/postqueue/version.rb

@@ -1,3 +1,3 @@
 module Postqueue
-  VERSION = '0.0.10'
+  VERSION = '0.0.11'
 end

data/spec/postqueue/enqueue_spec.rb

@@ -1,8 +1,10 @@
 require 'spec_helper'
 
-describe ::Postqueue::Enqueue do
+describe 'enqueuing' do
+  let(:queue) { Postqueue.new }
+
   before do
-    Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
   end
 
   let(:item) { Postqueue::Item.first }

data/spec/postqueue/{process_spec.rb → idempotent_queue_spec.rb}

@@ -1,27 +1,39 @@
 require 'spec_helper'
 
-describe "::Postqueue.process" do
+describe "Idempotent queue" do
+  class Testqueue < Postqueue::Base
+    def idempotent?(entity_type:,op:)
+      true
+    end
+
+    def batch_size(entity_type:,op:)
+      100
+    end
+  end
+
+  let(:queue) { Testqueue.new }
+
   context 'when having entries with the same entity_type and op' do
     before do
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 14
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 14
     end
 
     it "processes one entries" do
-      r = Postqueue.process limit: 1
+      r = queue.process batch_size: 1
       expect(r).to eq(["myop", "mytype", [12]])
       expect(items.map(&:entity_id)).to contain_exactly(13, 14)
     end
 
     it "processes two entries" do
-      r = Postqueue.process limit: 2
+      r = queue.process batch_size: 2
       expect(r).to eq(["myop", "mytype", [12, 13]])
       expect(items.map(&:entity_id)).to contain_exactly(14)
     end
 
     it "processes many entries" do
-      r = Postqueue.process
+      r = queue.process
       expect(r).to eq(["myop", "mytype", [12, 13, 14]])
       expect(items.map(&:entity_id)).to contain_exactly()
     end
@@ -29,33 +41,33 @@ describe "::Postqueue.process" do
 
   context 'when having entries with different entity_type and op' do
     before do
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-      Postqueue.enqueue op: "otherop", entity_type: "mytype", entity_id: 14
-      Postqueue.enqueue op: "myop", entity_type: "othertype", entity_id: 15
-      Postqueue.enqueue op: "otherop", entity_type: "othertype", entity_id: 16
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
+      queue.enqueue op: "otherop", entity_type: "mytype", entity_id: 14
+      queue.enqueue op: "myop", entity_type: "othertype", entity_id: 15
+      queue.enqueue op: "otherop", entity_type: "othertype", entity_id: 16
     end
 
     it "processes one entries" do
-      r = Postqueue.process limit: 1
+      r = queue.process batch_size: 1
       expect(r).to eq(["myop", "mytype", [12]])
       expect(items.map(&:entity_id)).to contain_exactly(13, 14, 15, 16)
     end
 
     it "processes two entries" do
-      r = Postqueue.process limit: 2
+      r = queue.process batch_size: 2
       expect(r).to eq(["myop", "mytype", [12, 13]])
       expect(items.map(&:entity_id)).to contain_exactly(14, 15, 16)
     end
 
     it "processes only matching entries when asked for more" do
-      r = Postqueue.process
+      r = queue.process
       expect(r).to eq(["myop", "mytype", [12, 13]])
       expect(items.map(&:entity_id)).to contain_exactly(14, 15, 16)
     end
 
     it "honors search conditions" do
-      r = Postqueue.process(where: { op: "otherop" })
+      r = queue.process(op: "otherop")
       expect(r).to eq(["otherop", "mytype", [14]])
       expect(items.map(&:entity_id)).to contain_exactly(12, 13, 15, 16)
     end
@@ -63,21 +75,15 @@ describe "::Postqueue.process" do
   
   context 'when having duplicate entries' do
     before do
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-      Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
+      queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
     end
 
     it "removes duplicates from the queue" do
-      r = Postqueue.process limit: 1
+      r = queue.process batch_size: 1
       expect(r).to eq(["myop", "mytype", [12]])
       expect(items.map(&:entity_id)).to contain_exactly(13)
     end
-
-    it "does not remove duplicates when skip_duplicates is set to false" do
-      r = Postqueue.process limit: 1, skip_duplicates: false
-      expect(r).to eq(["myop", "mytype", [12]])
-      expect(items.map(&:entity_id)).to contain_exactly(13, 12)
-    end
   end
 end

data/spec/postqueue/process_errors_spec.rb

@@ -1,41 +1,73 @@
 require 'spec_helper'
 
-describe "::Postqueue.process_one" do
+describe "::queue.process_one" do
+  let(:queue) { Postqueue.new }
+
   class E < RuntimeError; end
 
   before do
-    Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
   end
 
-  it "fails when block raises an exception and reraises the exception" do
-    expect { Postqueue.process_one do |op, type, ids| raise E end }.to raise_error(E)
-    expect(items.map(&:entity_id)).to contain_exactly(12)
-  end
+  context "block raises an exception" do
+    before do
+      expect { queue.process_one do |op, type, ids| raise E end }.to raise_error(E)
+    end
+
+    it "reraises the exception" do
+      # checked in before block
+    end
 
-  it "fails when block returns false" do
-    Postqueue.process_one do |op, type, ids| false end
-    expect(items.map(&:entity_id)).to contain_exactly(12)
+    it "keeps the item in the queue" do
+      expect(items.map(&:entity_id)).to contain_exactly(12)
+    end
+
+    it "increments the failed_attempt count" do
+      expect(items.map(&:failed_attempts)).to contain_exactly(1)
+    end
   end
 
-  it "keeps item in the queue after failure, with an increased failed_attempt count" do
-    called_block = 0
-    Postqueue.process_one do called_block += 1; false end
-    expect(called_block).to eq(1)
+  context "block returns false" do
+    before do
+      @result = queue.process_one do |op, type, ids| false end
+    end
+
+    it "returns false" do
+      expect(@result).to be false
+    end
 
-    expect(items.map(&:entity_id)).to contain_exactly(12)
-    expect(items.first.failed_attempts).to eq(1)
+    it "keeps the item in the queue" do
+      expect(items.map(&:entity_id)).to contain_exactly(12)
+    end
+
+    it "increments the failed_attempt count" do
+      expect(items.map(&:failed_attempts)).to contain_exactly(1)
+    end
   end
 
-  it "ignores items with a failed_attempt count > MAX_ATTEMPTS" do
-    expect(Postqueue::MAX_ATTEMPTS).to be >= 3
-    items.update_all(failed_attempts: 3)
+  context "failed_attempts reached MAX_ATTEMPTS" do
+    before do
+      expect(Postqueue::MAX_ATTEMPTS).to be >= 3
+      items.update_all(failed_attempts: Postqueue::MAX_ATTEMPTS)
+
+      @called_block = 0
+      @result = queue.process_one do called_block += 1; false end
+    end
+
+    it "does not call the block" do
+      expect(@called_block).to eq(0)
+    end
+
+    it "returns nil" do
+      expect(@result).to eq(nil)
+    end
 
-    called_block = 0
-    r = Postqueue.process_one do called_block += 1; false end
-    expect(r).to eq(nil)
-    expect(called_block).to eq(0)
+    it "does not remove the item" do
+      expect(items.map(&:entity_id)).to contain_exactly(12)
+    end
 
-    expect(items.map(&:entity_id)).to contain_exactly(12)
-    expect(items.first.failed_attempts).to eq(3)
+    it "does not increment the failed_attempts count" do
+      expect(items.first.failed_attempts).to eq(Postqueue::MAX_ATTEMPTS)
+    end
   end
 end

data/spec/postqueue/process_one_spec.rb

@@ -1,29 +1,33 @@
 require 'spec_helper'
 
-describe "::Postqueue.process_one" do
+describe "::queue.process_one" do
+  let(:queue) { Postqueue.new }
+
   before do
-    Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
-    Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
-    Postqueue.enqueue op: "myop", entity_type: "mytype", entity_id: 14
+    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 12
+    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 13
+    queue.enqueue op: "myop", entity_type: "mytype", entity_id: 14
   end
 
+  let(:processor) { Postqueue::Processor.new }
+
   it "processes one entry" do
-    r = Postqueue.process_one
+    r = queue.process_one
     expect(r).to eq(["myop", "mytype", [12]])
     expect(items.map(&:entity_id)).to contain_exactly(13, 14)
   end
 
   it "honors search conditions" do
-    Postqueue.enqueue op: "otherop", entity_type: "mytype", entity_id: 112
+    queue.enqueue op: "otherop", entity_type: "mytype", entity_id: 112
 
-    r = Postqueue.process_one(where: { op: "otherop" })
+    r = queue.process_one(op: "otherop")
     expect(r).to eq(["otherop", "mytype", [112]])
     expect(items.map(&:entity_id)).to contain_exactly(12, 13, 14)
   end
 
   it "yields a block and returns it" do
-    Postqueue.enqueue op: "otherop", entity_type: "mytype", entity_id: 112 
-    r = Postqueue.process_one(where: { op: "otherop" }) do |op, type, ids|
+    queue.enqueue op: "otherop", entity_type: "mytype", entity_id: 112 
+    r = queue.process_one(op: "otherop") do |op, type, ids|
       expect(op).to eq("otherop")
       expect(type).to eq("mytype")
       expect(ids).to eq([112])

data/spec/spec_helper.rb

@@ -14,6 +14,14 @@ end
 require 'postqueue'
 require './spec/support/configure_active_record'
 
+$logger = Logger.new(File.open("log/test.log", "a"))
+
+module Postqueue
+  def self.logger
+    $logger
+  end
+end
+
 def items
   Postqueue::Item.all
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: postqueue
 version: !ruby/object:Gem::Version
-  version: 0.0.10
+  version: 0.0.11
 platform: ruby
 authors:
 - radiospiel
@@ -131,15 +131,16 @@ extra_rdoc_files: []
 files:
 - README.md
 - lib/postqueue.rb
-- lib/postqueue/enqueue.rb
+- lib/postqueue/base.rb
+- lib/postqueue/base/enqueue.rb
+- lib/postqueue/base/processing.rb
 - lib/postqueue/item.rb
-- lib/postqueue/processing.rb
 - lib/postqueue/version.rb
 - spec/postqueue/enqueue_spec.rb
+- spec/postqueue/idempotent_queue_spec.rb
 - spec/postqueue/postqueue_spec.rb
 - spec/postqueue/process_errors_spec.rb
 - spec/postqueue/process_one_spec.rb
-- spec/postqueue/process_spec.rb
 - spec/spec_helper.rb
 - spec/support/configure_active_record.rb
 - spec/support/models.rb

data/lib/postqueue/processing.rb

@@ -1,97 +0,0 @@
-module Postqueue
-  MAX_ATTEMPTS = 3
-
-  module Processing
-    # Processes many entries
-    #
-    # process limit: 100, skip_duplicates: true
-    def process(options = {}, &block)
-      limit           = options.fetch(:limit, 100)
-      skip_duplicates = options.fetch(:skip_duplicates, true)
-      options.delete :limit
-      options.delete :skip_duplicates
-
-      status, result = Item.transaction do
-        process_inside_transaction options, limit: limit, skip_duplicates: skip_duplicates, &block
-      end
-
-      raise result if status == :err
-      result
-    end
-
-    # Process a single entry from the queue
-    #
-    # Example:
-    #
-    #   process_one do |op, entity_type
-    #   end
-    def process_one(options = {}, &block)
-      options = options.merge(limit: 1, skip_duplicates: false)
-      process options, &block
-    end
-
-    private
-
-    def select_and_lock(relation, limit:)
-      relation = relation.where("failed_attempts < ? AND next_run_at < ?", MAX_ATTEMPTS, Time.now).order(:next_run_at, :id)
-
-      sql = relation.to_sql + " FOR UPDATE SKIP LOCKED"
-      sql += " LIMIT #{limit}" if limit
-      items = Item.find_by_sql(sql)
-
-      items
-    end
-
-    # The actual processing. Returns [ :ok, number-of-items ] or  [ :err, exception ]
-    def process_inside_transaction(options, limit:, skip_duplicates:, &block)
-      relation = Item.all
-      relation = relation.where(options[:where]) if options[:where]
-
-      first_match = select_and_lock(relation, limit: 1).first
-      return [ :ok, nil ] unless first_match
-
-      # find all matching entries with the same entity_type/op value
-      if limit > 1
-        batch_relation = relation.where(entity_type: first_match.entity_type, op: first_match.op)
-        matches = select_and_lock(batch_relation, limit: limit)
-      else
-        matches = [ first_match ]
-      end
-
-      entity_ids = matches.map(&:entity_id)
-
-      # When skipping dupes we'll find and lock all entries that match entity_type,
-      # op, and one of the entity_ids in the first batch of matches
-      if skip_duplicates
-        entity_ids.uniq!
-        process_relations = relation.where(entity_type: first_match.entity_type, op: first_match.op, entity_id: entity_ids)
-        process_items = select_and_lock process_relations, limit: nil
-      else
-        process_items = matches
-      end
-
-      # Actually process the queue items
-      result = [ first_match.op, first_match.entity_type, entity_ids ]
-      result = yield *result if block_given?
-
-      if result == false
-        postpone process_items
-      else
-        Item.where(id: process_items.map(&:id)).delete_all 
-      end
-
-      [ :ok, result ]
-    rescue => e
-      postpone process_items
-      [ :err, e ]
-    end
-
-    def postpone(items)
-      ids = items.map(&:id)
-      sql = "UPDATE postqueue SET failed_attempts = failed_attempts+1, next_run_at = next_run_at + interval '10 second' WHERE id IN (#{ids.join(",")})"
-      Item.connection.exec_query(sql)
-    end
-  end
-  
-  extend Processing
-end