ost 0.0.3 → 0.1.0

data/README.md

@@ -0,0 +1,158 @@
+Ost
+===
+
+Redis based queues and workers.
+
+![Ost Cafe, by Arancia Project](http://farm4.static.flickr.com/3255/3161710005_36566b8a9e.jpg)
+
+Description
+-----------
+
+**Ost** makes it easy to enqueue object ids and process them with
+workers.
+
+Say you want to process video uploads. In your application you will
+have something like this:
+
+``` ruby
+Ost[:videos_to_process].push(@video.id)
+```
+
+Then, you will have a worker that will look like this:
+
+``` ruby
+require "ost"
+
+Ost[:videos_to_process].each do |id|
+  # Do something with it!
+end
+```
+
+Usage
+-----
+
+**Ost** connects to Redis automatically with the default options
+(localhost:6379, database 0).
+
+You can customize the connection by calling `connect`:
+
+``` ruby
+Ost.connect port: 6380, db: 2
+```
+
+Then you only need to refer to a queue for it to pop into existence:
+
+``` ruby
+Ost[:rss_feeds] << @feed.id
+```
+
+A worker is a Ruby file with this basic code:
+
+``` ruby
+require "ost"
+
+Ost[:rss_feeds].each do |id|
+  # ...
+end
+```
+
+It will pop items from the queue as soon as they become available. It
+uses BRPOPLPUSH with a timeout that can be specified with the
+`OST_TIMEOUT` environment variable.
+
+Note that in these examples we are pushing numbers to the queue. As
+we have unlimited queues, each queue should be specialized and the
+workers must be smart enough to know what to do with the numbers they
+pop.
+
+Available methods
+=================
+
+`Ost.connect`: configure the connection to Redis. By default, it
+connects to localhost in port 6379 and uses the database 0. It accepts
+the same options as [redis-rb](https://github.com/redis/redis-rb).
+
+`Ost.stop`: halt processing for all queues.
+
+`Ost[:example].push item`, `Ost[:some_queue] << item`: add `item` to
+the `:example` queue.
+
+`Ost[:example].push { |item| ... }`, `Ost[:example].each { |item| ...
+}`: consume `item` from the `:example` queue. If the block doesn't
+complete successfully, the item will be left at a backup queue.
+
+`Ost[:example].stop`: halt processing for the `example` queue.
+
+Failures
+========
+
+**Ost** stores in-process items in backup queues. That allows the
+developer to deal with exceptions in a way that results adequate
+for his application.
+
+There is one backup queue for each worker, with the following
+convention for naming the key in Redis: given a worker using the
+`:events` queue, running in the hostname `domU-12-31-39-04-49-C7`
+with the process id `28431`, the key for the backup queue will be
+`ost:events:domU-12-31-39-04-49-C7:28431`.
+
+Here's the explanation for each part:
+
+* `ost`: namespace for all **Ost** related keys.
+* `events`: name of the queue.
+* `domU-12-31-39-04-49-C7`: hostname of the worker.
+* `28431`: process id of the worker.
+
+Priorities
+----------
+
+There's no concept of priorities, as each queue is specialized and you
+can create as many as you want. For example, nothing prevents the
+creation of the `:example_high_priority` or the
+`:example_low_priority` queues.
+
+Differences with Delayed::Job and Resque
+----------------------------------------
+
+Both [Delayed::Job](http://github.com/tobi/delayed_job) and
+[Resque](http://github.com/defunkt/resque) provide queues and workers
+(the latter using Redis). They provide dumb workers that process jobs,
+which are specialized for each task. The specialization takes place
+in the application side, and the job is serialized and pushed into a
+queue.
+
+**Ost**, by contrast, just pushes numbers into specialized queues, and
+uses workers that are subscribed to specific queues and know what to
+do with the items they get. The total sum of logic is about the same,
+but there's less communication and less data transfer with **Ost**.
+
+Installation
+------------
+
+    $ gem install ost
+
+License
+-------
+
+Copyright (c) 2010, 2011, 2012 Michel Martens
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.

data/lib/ost.rb

@@ -1,18 +1,20 @@
 require "nest"
 
 module Ost
-  VERSION = "0.0.3"
-  TIMEOUT = ENV["OST_TIMEOUT"] || 2
+  VERSION = "0.1.0"
+  TIMEOUT = ENV["OST_TIMEOUT"] || 0
 
   class Queue
-    attr :ns
+    attr :key
+    attr :backup
 
     def initialize(name)
-      @ns = Nest.new(:ost)[name]
+      @key = Nest.new(:ost)[name]
+      @backup = @key[Socket.gethostname][Process.pid]
     end
 
     def push(value)
-      redis.lpush(ns, value)
+      key.lpush(value)
     end
 
     def each(&block)
@@ -21,28 +23,22 @@ module Ost
       loop do
         break if @stopping
 
-        _, item = redis.brpop(ns, TIMEOUT)
-        next if item.nil? or item.empty?
+        item = @key.brpoplpush(@backup, TIMEOUT)
 
-        begin
-          block.call(item)
-        rescue Exception => e
-          error = "#{Time.now} #{ns[item]} => #{e.inspect}"
+        block.call(item)
 
-          redis.rpush   ns[:errors], error
-          redis.publish ns[:errors], error
-        end
+        @backup.del
       end
     end
 
-    def errors
-      redis.lrange ns[:errors], 0, -1
-    end
-
     def stop
       @stopping = true
     end
 
+    def items
+      key.lrange(0, -1)
+    end
+
     alias << push
     alias pop each
 

data/ost.gemspec

@@ -11,7 +11,7 @@ Gem::Specification.new do |s|
 
   s.files = Dir[
     "LICENSE",
-    "README.markdown",
+    "README.md",
     "Rakefile",
     "lib/**/*.rb",
     "*.gemspec",

data/test/ost_test.rb

@@ -1,14 +1,20 @@
+ENV["OST_TIMEOUT"] = "1"
+
 require File.expand_path("test_helper", File.dirname(__FILE__))
 
 scope do
   def ost(&job)
     thread = Thread.new do
-      Ost[:events].each(&job)
+      Ost[:events].each do |item|
+        begin
+          yield(item)
+        ensure
+          thread.kill
+        end
+      end
     end
 
-    sleep 0.1
-
-    thread.kill
+    thread.join
   end
 
   def enqueue(id)
@@ -24,9 +30,14 @@ scope do
     Redis.new
   end
 
-  test "insert items in the queue" do |redis|
+  test "allows access to the queued items" do
     enqueue(1)
-    assert_equal ["1"], redis.lrange("ost:events", 0, -1)
+
+    assert_equal ["1"], Ost[:events].items
+  end
+
+  test "allows access to the underlying key" do
+    assert_equal 0, Ost[:events].key.llen
   end
 
   test "process items from the queue" do |redis|
@@ -38,50 +49,10 @@ scope do
       results << item
     end
 
-    assert_equal [], redis.lrange("ost:events", 0, -1)
+    assert_equal [], Ost[:events].items
     assert_equal ["1"], results
   end
 
-  test "add failures to special lists" do |redis|
-    enqueue(1)
-
-    ost do |item|
-      raise "Wrong answer"
-    end
-
-    assert_equal 0, redis.llen("ost:events")
-    assert_equal 1, redis.llen("ost:events:errors")
-
-    assert redis.rpop("ost:events:errors").match(/ost:events:1 => #<RuntimeError: Wrong answer/)
-  end
-
-  test "publish the error to a specific channel" do |redis|
-    enqueue(1)
-    results = []
-
-    t1 = Thread.new do
-      redis.subscribe("ost:events:errors") do |on|
-        on.message do |channel, message|
-          if message[/ost:events:1 => #<RuntimeError: Wrong answer/]
-            results << message
-            redis.unsubscribe
-          end
-        end
-      end
-    end
-
-    ost do |item|
-      raise "Wrong answer"
-    end
-
-    t1.join
-
-    assert_equal 0, redis.llen("ost:events")
-    assert_equal 1, redis.llen("ost:events:errors")
-
-    assert results.pop.match(/ost:events:1 => #<RuntimeError: Wrong answer/)
-  end
-
   test "halt processing a queue" do
     Thread.new do
       sleep 0.5
@@ -107,4 +78,38 @@ scope do
 
     assert true
   end
+
+  test "maintains a backup queue for when worker dies" do
+    enqueue(1)
+
+    assert_equal 0, Ost[:events].backup.llen
+
+    begin
+      Ost[:events].each do |item|
+        item.some_error
+      end
+    rescue
+    end
+
+    assert_equal ["1"], Ost[:events].backup.lrange(0, -1)
+  end
+
+  test "cleans up the backup queue on success" do
+    enqueue(1)
+
+    done = false
+
+    Thread.new do
+      Ost[:events].each do |item|
+        assert_equal ["1"], Ost[:events].backup.lrange(0, -1)
+        done = true
+      end
+    end
+
+    until done; end
+
+    Ost[:events].stop
+
+    assert_equal 0, Ost[:events].backup.llen
+  end
 end

metadata

@@ -1,51 +1,47 @@
---- !ruby/object:Gem::Specification 
+--- !ruby/object:Gem::Specification
 name: ost
-version: !ruby/object:Gem::Version 
+version: !ruby/object:Gem::Version
+  version: 0.1.0
   prerelease: 
-  version: 0.0.3
 platform: ruby
-authors: 
+authors:
 - Michel Martens
 autorequire: 
 bindir: bin
 cert_chain: []
-
-date: 2011-05-13 00:00:00 Z
-dependencies: 
-- !ruby/object:Gem::Dependency 
+date: 2012-03-17 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
   name: nest
-  prerelease: false
-  requirement: &id001 !ruby/object:Gem::Requirement 
+  requirement: &2160446460 !ruby/object:Gem::Requirement
     none: false
-    requirements: 
+    requirements:
     - - ~>
-      - !ruby/object:Gem::Version 
-        version: "1.0"
+      - !ruby/object:Gem::Version
+        version: '1.0'
   type: :runtime
-  version_requirements: *id001
-- !ruby/object:Gem::Dependency 
-  name: cutest
   prerelease: false
-  requirement: &id002 !ruby/object:Gem::Requirement 
+  version_requirements: *2160446460
+- !ruby/object:Gem::Dependency
+  name: cutest
+  requirement: &2160445920 !ruby/object:Gem::Requirement
     none: false
-    requirements: 
+    requirements:
     - - ~>
-      - !ruby/object:Gem::Version 
-        version: "1.0"
+      - !ruby/object:Gem::Version
+        version: '1.0'
   type: :development
-  version_requirements: *id002
+  prerelease: false
+  version_requirements: *2160445920
 description: Ost lets you manage queues and workers with Redis.
-email: 
+email:
 - michel@soveran.com
 executables: []
-
 extensions: []
-
 extra_rdoc_files: []
-
-files: 
+files:
 - LICENSE
-- README.markdown
+- README.md
 - Rakefile
 - lib/ost.rb
 - ost.gemspec
@@ -53,30 +49,26 @@ files:
 - test/test_helper.rb
 homepage: http://github.com/soveran/ost
 licenses: []
-
 post_install_message: 
 rdoc_options: []
-
-require_paths: 
+require_paths:
 - lib
-required_ruby_version: !ruby/object:Gem::Requirement 
+required_ruby_version: !ruby/object:Gem::Requirement
   none: false
-  requirements: 
-  - - ">="
-    - !ruby/object:Gem::Version 
-      version: "0"
-required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
-  requirements: 
-  - - ">="
-    - !ruby/object:Gem::Version 
-      version: "0"
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
 requirements: []
-
 rubyforge_project: 
-rubygems_version: 1.8.1
+rubygems_version: 1.8.10
 signing_key: 
 specification_version: 3
 summary: Redis based queues and workers.
 test_files: []
-

data/README.markdown

@@ -1,111 +0,0 @@
-Ost
-===
-
-Redis based queues and workers.
-
-![Ost Cafe, by Arancia Project](http://farm4.static.flickr.com/3255/3161710005_36566b8a9e.jpg)
-
-Description
------------
-
-**Ost** makes it easy to enqueue object ids and process them with workers.
-
-Say you want to process video uploads. In your application you will have something like this:
-
-    Ost[:videos_to_process].push(@video.id)
-
-Then, you will have a worker that will look like this:
-
-    require "ost"
-
-    Ost[:videos_to_process].each do |id|
-      # Do something with it!
-    end
-
-Usage
------
-
-**Ost** connects to Redis automatically with the default options (localhost:6379, database 0).
-
-You can customize the connection by calling `connect`:
-
-    Ost.connect port: 6380, db: 2
-
-Then you only need to refer to a queue for it to pop into existence:
-
-    Ost[:rss_feeds] << @feed.id
-
-A worker is a Ruby file with this basic code:
-
-    require "ost"
-
-    Ost[:rss_feeds].each do |id|
-      ...
-    end
-
-It will pop items from the queue with a timeout of two seconds and retry indefinitely. If you want to configure the timeout, set the environment variable `OST_TIMEOUT`.
-
-Available methods for a queue are `push` (aliased to `<<`) and `pop` (aliased to `each`).
-
-Note that in these examples we are pushing numbers to the queue. As we have unlimited queues, each queue should be specialized and the workers must be smart enough to know what to do with the numbers they pop.
-
-Failures
-========
-
-If the block raises an error, it is captured by **Ost** and the exception is logged in Redis.
-
-Consider this example:
-
-    require "ost"
-
-    Ost[:rss_feeds].each do |id|
-      ...
-      raise "Invalid format"
-    end
-
-Then, in the console you can do:
-
-    >> Ost[:rss_feeds].push 1
-    => 1
-
-    >> Ost[:rss_feeds].errors
-    => ["2010-04-12 21:57:23 -0300 ost:rss_feeds:1 => #<RuntimeError: Invalid format>"]
-
-Differences with Delayed::Job and Resque
---------------------------------------
-
-Both [Delayed::Job](http://github.com/tobi/delayed_job) and [Resque](http://github.com/defunkt/resque)
-provide queues and workers (the latter using Redis). They provide dumb workers that process jobs, which are specialized for each task. The specialization takes place in the application side, and the job is serialized and pushed into a queue.
-
-**Ost**, by contrast, just pushes numbers into specialized queues, and uses workers that are subscribed to specific queues and know what to do with the items they get. The total sum of logic is almost the same.
-
-Installation
-------------
-
-    $ gem install ost
-
-License
--------
-
-Copyright (c) 2010 Michel Martens
-
-Permission is hereby granted, free of charge, to any person
-obtaining a copy of this software and associated documentation
-files (the "Software"), to deal in the Software without
-restriction, including without limitation the rights to use,
-copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the
-Software is furnished to do so, subject to the following
-conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
-OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
-NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
-HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
-WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
-FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
-OTHER DEALINGS IN THE SOFTWARE.