postqueue 0.5.3 → 0.5.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ad176bce5f7bba7fda654b053e5fa32acd41abdd
-  data.tar.gz: 646743647918dbc637ea6e740f4015b3ef942f82
+  metadata.gz: 323c108f2c06eaaa629dd218e0e2483252b44c3e
+  data.tar.gz: c3c5521cf196e4b68bf835d1a00da1cd40c2e612
 SHA512:
-  metadata.gz: ffd319be64f642202f294c85b7c2d89d310b4d42dc4654cbb6d3d4d2bf65ffcd57ab80a6aa9aa328062a7243ca5d2696d03e120993b46db1f49a578c65a124e2
-  data.tar.gz: 587ab795e8b9e239f8e126d6325418dca8d9014999c1da4d696ddbc19115074497be3d5ee00fd457ca2dac234f7a2ff21801be154457c5b03f34428c14466ce1
+  metadata.gz: 2e0991a54661b6ca09325e1a490ddb5f4dd71fb320188cd8d3aea49890aa03ffe8740f31f1e98a1015b4d7634e3735c2af0b68747bbaa52d21ddce2003e81e38
+  data.tar.gz: 645df66d57f05fd2f7f7a33d0798b4aa33faacfe72f84a1bebbe3d7678e520024806801aaa90ddd672400cf0b8ee2b0dda7d18755ace559a602e8e2e96d27c6f

data/README.md

@@ -1,116 +1,181 @@
 # Postqueue
 
-## Intro
+The `postqueue` gem implements a simple to use queue on top of postgresql.
 
-The postqueue gem implements a simple to use queue on top of postgresql. Note that while 
-a queue like this is typically used in a job queueing scenario, this document does not 
-talk about jobs, it talks about **queue items**; it also does not schedule a job, 
-it **enqueues** an item, and it does not executes a job, it **processes** queue items.
+Lets have a word about words first: while a queue like this is typically used for job queues, this document does not talk about jobs, it talks about **queue items**; we also do not schedule jobs, we **enqueues** items, and we don't executes a job, we **processes** queue items instead.
 
-Why building an additional queue implementation? Compared to delayed_job or the other 
-usual suspects postqueue implements these features:
+So, why building an additional queue implementation? Compared to the usual suspects this is what postqueue brings to the table:
 
-- The item structure is intentionally kept super simple: an item is described by an
-  `op` field - a string - and an `id` field, an integer. In a typical usecase a 
-  queue item would describe an operation on a specific entity, where `op` names 
-  both the operation and the entity type and the `id` field would describe the 
-  individual entity.
+- The **item structure** is intentionally kept very **simple**: an item is described by an `op` field - a string - and an `id` field, an integer. In a typical usecase a queue item would describe an operation on a specific entity, and `op` would name both the operation and the entity type (say: `"product/invalidate"`) and the `id` field would hold the id of the product to invalidate.
 
-- With such a simplistic item structure the queue itself can be searched or 
-  otherwise evaluated using SQL. This also allows for **skipping duplicate entries** 
-  when enqueuing items (managed via a duplicate: argument when enqueuing) and for 
-  **batch processing** multple items in one go. 
+- Such a simplistic item structure lends quite well to **querying** the queue **using SQL**. While this is great for monitoring purposes - it is quite easy to fetch metrics regarding upcoming items - it also allows special handling of idempotent operations and automatic batching.
 
-- With data being kept in a Postgresql database processing provides **transactional semantics**: 
-  an item failing to process stays in the queue. Error handling is kept simpe to a 
-  strategy of rescheduling items up to a specific maximum number of processing attemps.
+- Some tasks typically handled by queues are of **idempotent** nature. For example, reindexing a document into a NoSQL search index needs not be done twice in a row, since only the last change to the primary object should and ultimately will be stored in the search index. Such tasks can therefore be run only once. postqueue supports this feature by optionally skipping duplicates when enqueuing tasks.
 
-Please be aware that postqueue is using the SELECT .. FOR UPDATE SKIP LOCKED Postgresql syntax, 
-and therefore needs at least PostgresQL >= 9.5.
+- Other tasks can be handled much more efficient when **run in batches**. Typical examples include processing a larger number of entities that need to be read from a database, but could be pulled much more efficient in a single query instead of in N queries. postqueue automatically batches such items.
+
+- Being based on Postgresql postqueue provides **transactional semantics**: an item written into the database in a transaction that fails afterwards is never processed.
+
+- **automatic retries**: like delayed job postqueue implements a rudimentary form of error processing. A failing item - this is an item which does not have a handler registered, or whose handler fails by raising an exception - is kept in the queue and reprocessed later. It is reprocessed up to N times (currently 5 times by default) until it is "doomed" ultimately. This is similar to delayed job's error handling, with some differences, however: 
+
+  - no backtrace is kept in the database
+  - the waiting time doesn't ramp up as fast (postqueue does `1.5 ** <number of retries>`)
+
+Please be aware that postqueue is using the `SELECT .. FOR UPDATE SKIP LOCKED` Postgresql syntax, and therefore needs at least PostgresQL >= 9.5.
 
 ## Basic usage
 
+Postqueue is able to run queues that use separate tables as their backstore, or use a preexisting table. However, basic usage should cover most scenarios. 
+
+Hence we cover the basic scenario here: in that scenario a single table *"postqueue"* is used to store queue items, and the `Postqueue` default queue holds all configuration. We also assume that you want to integrate Postqueue with a Rails application.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'postqueue'
+```
+
+And then execute:
+
+    $ bundle
+
+### Adding a migration
+
+The following migration creates a postqueue table with all necessary entries:
+
 ```ruby
-queue = Postqueue.new
-queue.enqueue op: "product/reindex", entity_id: [12,13,14,15]
-queue.on "product/reindex" do |op, entity_ids|
+class AddPostqueue < ActiveRecord::Migration
+  def up
+    Postqueue.migrate!
+  end
+
+  def down
+    Postqueue.unmigrate!
+  end
+end
+```
+
+### Configuring Postqueue
+
+The postqueue configuration descrives all possible operations and their features:
+
+- is it possible to batch these operationss? In that case multiple queue items will be combined and processed in one go. Set `batch_size:` to a sensible size.
+- is this an idempotent operation? Set `idempotent:` to true.
+
+The configuration file should live in `config/initializer/postqueue.rb` for Rails apps, and in `config/postqueue.rb` in other Ruby applications.
+
+```ruby
+# config/initializer/postqueue.rb
+Postqueue.on "refresh", batch_size: 10, idempotent: true do |op, entity_ids|
   Product.index_many(Product.where(id: entity_ids))
 end
+```
+
+Note that you could define an operation without a handler callback:
+
+```ruby
+Postqueue.on "foo"
+```
+
+In this case the `foo` ops are just removed from the queue during procesing.
+
+### Postqueue database configuration
+
+When run from inside a Rails application Postqueue will reuse the applications database connection. When run outside a Rails application postqueue will use a
+`config/database.yml` file to determine database connection settings. It will use the `RAILS_ENV` environment value, defaulting to `"development"`, to choose from entries in that file.
+
+### Enqueueing items
+
+Enqueuing items can be done using code like this:
+
+```ruby
+# enqueue a single op
+Postqueue.enqueue op: "refresh", entity_id: 12
 
-queue.process
+# enqueue multiple ops in one go.
+Postqueue.enqueue op: "refresh", entity_id: [12,13]
+Postqueue.enqueue op: "refresh", entity_id: [13,14]
 ```
 
-The process call will select a number of queue items for processing. They will all have 
-the same `op` attribute. The callback will receive the `op` attribute and the `entity_ids`
-of all queue entries selected for processing. The `processing` method will return the number
-of processed items.
+Note that enqueueing is pretty fast. My developer machine is able to enqueue ~20000 items per second.
 
-If no callback is given the matching items are only removed from the queue without
-any processing.
+### Processing items
 
-Postqueue.process also accepts the following arguments:
+While we recommend to use the command line interface to process postqueue items you can certainly process these *from within ruby code* by using one of these methods:
+
+    # process the next batch of items
+    Postqueue.process
+    
+    # process a single item, do not batch
+    Postqueue.process_one
+    
+    # process batches of items until there are none left
+    Postqueue.process_until_empty
+
+These calls will select a one or more queue items for processing (with the same `op` attribute). The `process_*` methods will then call the callback for that operation with the `entity_ids` of all queue entries. After processing they will return the number of processed items, or 0 if no items could be found or be processed.
+
+The `process_*` methods also accept the following arguments:
 
 - `op`: only process entries with this `op` value;
 - `batch_size`: maximum number of items to process in one go.
 
 Example:
 
-    Postqueue.process(op: 'product/reindex', batch_size: 10) do |op, entity_ids|
-      # only handle up to 10 "product/reindex" entries
-    end
-
-If the block raises an exception the queue will postpone processing these entries 
-by an increasing amount of time, up until `queue.max_attempts` failed attempts.
-That value defaults to 5.
+    # process only `product` queue items
+    Postqueue.process(op: 'product', batch_size: 10)
 
-If the queue is empty or no matching queue entry could be found, `Postqueue.process` 
-returns 0.
+### processing from the command line
 
-## Advanced usage
-
-### Concurrency
+```ruby
+bundle exec postqueue run
+```
 
-Postqueue implements the following concurrency guarantees:
+starts a single postqueue runner. Note that there is intentionally no option to daemonize this process or to run these in parallel.
 
-- catastrophic DB failure and communication breakdown aside a queue item which is enqueued will eventually be processed successfully exactly once;
-- multiple consumers can work in parallel.
+The postqueue CLI has additional commands, see below.
 
-Note that you should not share a Postqueue ruby object across threads - instead you should create
-process objects with the identical configuration.
+## The postqueue CLI
 
-### Idempotent operations
+postqueue comes with a command line interface:
 
-When enqueueing items duplicate idempotent operations are not enqueued. Whether or not an operation
-should be considered idempotent is defined when configuring the queue:
+```
+~/postqueue[master] > ./bin/postqueue --help
+This is postqueue 0.5.3. Usage examples:
+
+  postqueue [ stats ]
+  postqueue peek
+  postqueue enqueue op entity_id,entity_id,entity_id
+  postqueue run
+  postqueue help
+  postqueue process
+```
 
-     Postqueue.new do |queue|
-       queue.on "idempotent", idempotent: true do ]op, entity_ids|
-         # .. handle queue item
-       end
-     end
+You can use the postqueue CLI to
 
-### Processing a single entry
+- enqueue an item, e.g. `bundle exec postqueue enqueue foo 1,2,3`
+- start a runner to process the queue: `bundle exec postqueue run`
+- process a single item off the queue: `bundle exec postqueue process`
+- get some stats for the queue: `bundle exec postqueue stats`
+- get a list of the next 100 queue items: `bundle exec postqueue peek`
 
-Postqueue implements a shortcut to process only a single entry. Under the hood this 
-calls `Postqueue.process` with `batch_size` set to `1`:
+## Additional  notes
 
-    queue.process_one
+### Concurrency
 
-Note that even though `process_one` will only ever process a single entry the 
-`entity_ids` parameter to the callback is still an array (with a single ID entry 
-in that case).
+Postqueue implements the following concurrency guarantees:
 
-### Migrating
+- catastrophic DB failure and communication breakdown aside a queue item which is enqueued will eventually be processed successfully exactly once;
+- multiple consumers can work in parallel.
 
-Postqueue comes with migration helpers:
+Note that you should not share a Postqueue instance across threads - instead you should create process objects with the identical configuration.
 
-    # set up a table for use with postqueue.
-    Postqueue.migrate!(table_name = "postqueue")
+### Idempotent operations
 
-    # set up a table for use with postqueue.
-    Postqueue.unmigrate!(table_name = "postqueue")
+If an operation was configured as idempotent (using the `Postqueue.on "op", idempotent: true` configuration) duplicate idempotent operations are not enqueued. However, if multiple transactions are enqueueing items at the same time, or when an idempotent item is processing while another item is being enqueued an additional queue item will still be enqueued. Therefore we also remove duplicate items during processing.
 
-You can also set up your own table, as long as it is compatible.
+### Using non-default tables or databases
 
 To use a non-default table or a non-default database, change the `item_class`
 attribute of the queue:
@@ -119,72 +184,76 @@ attribute of the queue:
       queue.item_class = MyItemClass
     end
 
-`MyItemClass` should inherit from Postqueue::Item and use the same or a compatible database
-structure.
+`MyItemClass` should inherit from Postqueue::Item and use the same or a compatible database structure.
 
-## Batch processing
+### Special Ops
 
-Often queue items can be batched together for a performant operation. To allow batch 
-processing for some items, configure the Postqueue to either set a `default_batch_size`
-or an operation-specific batch_size:
+Postqueue always registers the following operations:
 
-    Postqueue.new do |queue|
-      queue.default_batch_size = 100
-      queue.on "batchable", batch_size: 10 do
-        ...
-      end
+- `"test"` will write an output to the Postqueue.logger. Use this to test your infrastructure.
+- `"fail"` will always raise an exception. Use this to test your error handling integration.
+
+### Unknown operations
+
+You can define a handler to handle unknown operations like this:
+
+    on :missing_handler do |op, entity_ids|
+      raise MissingHandler, queue: self, op: op, entity_ids: entity_ids
     end
 
-## Test mode
+### Exception handling
 
-Postqueue works usually in an async mode: queue items that are enqueued are kept in a queue,
-and must be picked up later explicitely for processing (via one of the `process`, `process_one` or `process_until_empty` methods).
+You can define a handler to handle any exceptions. This is the integration point for your exception handling framework like rollbar.com or so.
 
-During unit tests it is likely preferrable to process queue items in synchronous fashion -
-if you are interested in actual processing - or, at least, in a mode which validates that
-the `op` value is valid (that means that a handler is registered for that op). You can
-change the processing mode via
+The default exception handler is:
 
-    # can be :sync, :async, :verify
-    Postqueue.processing = :sync
+    Postqueue.on_exception do |e, _, _|
+      e.send :raise
+    end
 
-You can also change the processing mnode on a queue-by-queue base:
+The following would report exceptions to STDOUT and to rollbar:
 
-    Postqueue.new do |queue|
-      queue.processing = :sync
+    Postqueue.on_exception do |e, op, entity_ids|
+      msg =  "Caught error #{e.to_s.inspect}"
+      msg += " on op: #{op.inspect} "
+      msg += " w/entity_ids: #{entity_ids.inspect}"
+      Rollbar.error(e)
     end
 
-## Installation
+### `after_processing` callback
 
-Add this line to your application's Gemfile:
+After a batch of operations is processed, Postqueue calls the `after_processing` callback. It receives the `op` and `entity_ids` from the current processing run, and a `timing` object, which has these attributes:
 
-```ruby
-gem 'postqueue'
-```
+- Timing#avg_queue_time: the average queueing time for all item in the batch;
+- Timing#max_queue_time: the maximum queueing time of any item in the batch;
+- Timing#processing_time: elapsed time for processing the batch.
 
-And then execute:
+The default `after_processing` callback simply logs all information. You can
+easily use your own:
 
-    $ bundle
+    Postqueue.after_processing do |op, entity_ids, timing|
+      processing_time = timing.processing_time
+      Postqueue.logger.info "#{op] processing #{entity_ids.length}: #{'%.3f secs' % processing_time}"
+    end
+
+## Testing postqueue applications
+
+Postqueue works usually in an async mode: queue items that are enqueued are kept in a queue, and must be picked up later explicitely for processing (via one of the `process`, `process_one` or `process_until_empty` methods).
 
-Or install it yourself as:
+During unit tests it is likely preferrable to process queue items synchronously - if you are interested in actual processing - or, at least, in a mode which validates that the `op` value is actually configured in your application (i.e. that a handler is registered for that op). You can change the processing mode via
 
-    $ gem install postqueue
+    # can be :sync, :async, :verify
+    Postqueue.processing = :sync
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Make sure you have 
-a local postgresql implementation of at least version 9.5. Add a `postqueue` user with 
-a `postqueue` password, and create a `postqueue_test` database for it. The script 
-`./scripts/prepare_pg` can be somewhat helpful in establishing that.
+After checking out the repo, run `bin/setup` to install dependencies. Make sure you have a local postgresql implementation of at least version 9.5. Add a `postqueue` user with a `postqueue` password, and create a `postqueue_test` database for it. The script `./scripts/prepare_pg` can be somewhat helpful in establishing that.
 
-Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive
-prompt that will allow you to experiment.
+Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. 
 
-To release a new version, run `./scripts/release`, which will bump the version number, 
-create a git tag for the version, push git commits and tags, and push the `.gem` file 
-to [rubygems.org](https://rubygems.org).
+To release a new version, run `./scripts/release`, which will bump the version number, create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 

data/lib/postqueue/queue.rb

@@ -45,11 +45,27 @@ module Postqueue
         raise MissingHandler, queue: self, op: op, entity_ids: entity_ids
       end
 
+      set_default_callbacks
+
+      yield self if block
+    end
+
+    def set_default_callbacks
       on_exception do |e, _, _|
         e.send :raise
       end
 
-      yield self if block
+      after_processing do |op, entity_ids, timing|
+        processing_time = timing.processing_time
+        avg_queue_time  = timing.avg_queue_time
+        max_queue_time  = timing.max_queue_time
+
+        msg = "processing '#{op}' for id(s) #{entity_ids.join(',')}: "
+        msg += "processing #{entity_ids.length} items took #{'%.3f secs' % processing_time}"
+        msg += ", queue_time: #{'%.3f secs (avg)' % avg_queue_time}/#{'%.3f secs (max)' % max_queue_time}"
+
+        Postqueue.logger.info msg
+      end
     end
 
     def batch_size(op:)
@@ -83,3 +99,4 @@ require_relative "queue/processing"
 require_relative "queue/callback"
 require_relative "queue/logging"
 require_relative "queue/runner"
+require_relative "queue/timing"

data/lib/postqueue/queue/callback.rb

@@ -14,8 +14,6 @@ module Postqueue
   end
 
   class Queue
-    Timing = Struct.new(:avg_queue_time, :max_queue_time, :total_processing_time, :processing_time)
-
     def assert_valid_op!(op)
       return if op == :missing_handler
       return if op.is_a?(String)
@@ -58,12 +56,12 @@ module Postqueue
       SQL
       queue_time = queue_times.first
 
-      total_processing_time = Benchmark.realtime do
+      processing_time = Benchmark.realtime do
         callback = callback_for(op: op) || callbacks.fetch(:missing_handler)
         callback.call(op, entity_ids)
       end
 
-      Timing.new(queue_time.avg, queue_time.max, total_processing_time, total_processing_time / entity_ids.length)
+      Timing.new(queue_time.avg, queue_time.max, processing_time)
     end
   end
 end

data/lib/postqueue/queue/logging.rb

@@ -3,12 +3,15 @@ module Postqueue
   class Queue
     private
 
-    def on_processing(op, entity_ids, timing)
-      msg = "processing '#{op}' for id(s) #{entity_ids.join(',')}: "
-      msg += "processing #{entity_ids.length} items took #{'%.3f secs' % timing.total_processing_time}"
+    def after_processing(&block)
+      if block
+        @after_processing = block
+        if block.arity > -3 && block.arity != 3
+          raise ArgumentError, "Invalid after_processing block: must accept 3 arguments"
+        end
+      end
 
-      msg += ", queue_time: avg: #{'%.3f secs' % timing.avg_queue_time}/max: #{'%.3f secs' % timing.max_queue_time}"
-      logger.info msg
+      @after_processing
     end
 
     def log_exception(exception, op, entity_ids)

data/lib/postqueue/queue/processing.rb

@@ -36,7 +36,7 @@ module Postqueue
       entity_ids = items.map(&:entity_id)
       timing = run_callback(op: match.op, entity_ids: entity_ids)
 
-      on_processing(match.op, entity_ids, timing)
+      after_processing.call(match.op, entity_ids, timing)
       item_class.where(id: items.map(&:id)).delete_all
 
       # even though we try not to enqueue duplicates we cannot guarantee that,

data/lib/postqueue/queue/timing.rb

@@ -0,0 +1,3 @@
+module Postqueue
+  Timing = Struct.new(:avg_queue_time, :max_queue_time, :processing_time)
+end

data/lib/postqueue/version.rb

@@ -1,3 +1,3 @@
 module Postqueue
-  VERSION = "0.5.3"
+  VERSION = "0.5.4"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: postqueue
 version: !ruby/object:Gem::Version
-  version: 0.5.3
+  version: 0.5.4
 platform: ruby
 authors:
 - radiospiel
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-12-20 00:00:00.000000000 Z
+date: 2016-12-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -150,7 +150,8 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: a postgres based queue implementation
+description: simplistic postgresql based queue with support for batching and idempotent
+  operations
 email:
 - radiospiel@open-lab.org
 executables:
@@ -175,7 +176,7 @@ files:
 - lib/postqueue/queue/processing.rb
 - lib/postqueue/queue/runner.rb
 - lib/postqueue/queue/select_and_lock.rb
-- lib/postqueue/railtie.rb
+- lib/postqueue/queue/timing.rb
 - lib/postqueue/version.rb
 - lib/tracker.rb
 - lib/tracker/advisory_lock.rb
@@ -217,5 +218,6 @@ rubyforge_project:
 rubygems_version: 2.5.1
 signing_key: 
 specification_version: 4
-summary: a postgres based queue implementation
+summary: simplistic postgresql based queue with support for batching and idempotent
+  operations
 test_files: []

data/lib/postqueue/railtie.rb

@@ -1,17 +0,0 @@
-module Postqueue
-  class Railtie < Rails::Railtie
-    # initializer "postqueue.configure_rails_initialization" do
-    #   Postqueue.load_config_from "config/postqueue.rb"
-    # end
-  end
-
-  # def self.load_config_from(path)
-  #   @config_file = path
-  # end
-  #
-  # def self.load_config
-  #   return unless @config_file
-  #   load @config_file
-  #   @config_file = nil
-  # end
-end