pipes 0.1.0 → 0.2.0

data/.travis.yml

@@ -0,0 +1,5 @@
+rvm:
+  - 1.9.2
+  - 1.9.3
+services:
+  - redis-server

data/README.md

@@ -1,12 +1,16 @@
-# Pipes
+# Pipes [![Build Status](https://secure.travis-ci.org/mikepack/pipes.png)](http://travis-ci.org/mikepack/pipes) [![Dependency Status](https://gemnasium.com/mikepack/pipes.png)](https://gemnasium.com/mikepack/pipes) [![Code Climate](https://codeclimate.com/badge.png)](https://codeclimate.com/github/mikepack/pipes)
+
+[RDoc](http://rubydoc.info/github/mikepack/pipes)
 
 ![Pipes](http://i.imgur.com/MND26.png)
 
 Pipes is a Redis-backed concurrency management system designed around Resque. It provides a DSL for defining "stages" of a process. Each (Resque) job in the stage can be run concurrently, but all must finish before subsequent stages are run.
 
+Conceivably, Pipes is a lightweight, advanced Resque queue. It can be dropped right in place of Resque.
+
 ## Example
 
-At Factory Code Labs, we work on a system for which we must deploy static HTML files. We must render any number of HTML pages, assets, .htaccess files, etc so the static HTML-based site can run on Apache.
+At Factory Labs, we work on a system for which we must deploy static HTML files. We must render any number of HTML pages, assets, .htaccess files, etc so the static HTML-based site can run on Apache.
 
 Here's a simplified look at our stages:
 
@@ -25,6 +29,10 @@ Here's a simplified look at our stages:
 
 We want to ensure that all of **Stage 1** is finished before **Stage 2** begins, and likewise for **Stage 3**. However, the individual components of each stage can execute asynchronously, we just want to make sure they converge when all is finished.
 
+This can be visualized as follows:
+
+![Architecture](http://i.imgur.com/0CEmm.png)
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -41,7 +49,7 @@ Or install it yourself as:
 
 ## Usage
 
-Pipes assumes your conforming to the Resque API in your jobs, so you might have the following:
+Pipes assumes you're conforming to the Resque API in your jobs, so you might have the following:
 
 ```ruby
 module Writers
@@ -99,10 +107,12 @@ The name of the stage is arbitrary. Above, we have `content_writers`, `publisher
 
 ### Running The Jobs
 
-Once your configuration is set up, you can fire off the jobs:
+Once your configuration is set up, you can fire off the jobs.
+
+The Pipes API is designed to mimic Resque:
 
 ```ruby
-Pipes::Runner.run([Writers::HTMLWriter, Publishers::Rsyncer])
+Pipes.enqueue([Writers::HTMLWriter, Publishers::Rsyncer])
 ```
 
 The above line essentially says "here's the jobs I'm looking to run", at which point Pipes takes over to determine how to partition them into their appropriate stages. Pipes will break these two jobs up as you would expect:
@@ -115,10 +125,13 @@ Writers::HTMLWriter
 Publishers::Rsyncer
 ```
 
-You can also pass arguments to the jobs, just like Resque:
+You can also pass arguments to the jobs, just like Resque. In fact, any call to `Resque.enqueue` can be safely replaced with `Pipes.enqueue`, but the reverse is not true:
 
 ```ruby
-Pipes::Runner.run([Writers::HTMLWriter], 'http://localhost:3000/page')
+# If you currently have:
+# Resque.enqueue(Writers::HTMLWriter, 'http://localhost:3000/page')
+# ...you can replace it with:
+Pipes.enqueue(Writers::HTMLWriter, 'http://localhost:3000/page')
 ```
 
 In the above case, all jobs' `.perform` methods would receive the `http://localhost:3000/page` argument. You can, of course, pass multiple arguments:
@@ -134,10 +147,10 @@ module Writers
   end
 end
 
-Pipes::Runner.run([Writers::HTMLWriter], 'google.com', 80)
+Pipes.enqueue(Writers::HTMLWriter, 'google.com', 80)
 ```
 
-## Defining Stage Dependencies
+## Defining Job Dependencies
 
 Pipes makes it easy to define dependencies between jobs.
 
@@ -239,18 +252,125 @@ In the above example, `Notifiers::FileActivator` will also be a dependency of `W
 Running jobs with dependencies is the same as before:
 
 ```ruby
-Pipes::Runner.run([Writers::HTMLWriter], 'http://localhost:3000/page')
+Pipes.enqueue(Writers::HTMLWriter, 'http://localhost:3000/page')
 ```
 
 The above code will run `Writers::HTMLWriter` in **Stage 1**, `Publishers::Rsyncer` and `Publishers::CDNUploader` in **Stage 2**, and `Notifiers::FileActivator` in **Stage 3**, all receiving the `http://localhost:3000/page' argument.
 
-You can turn off dependency resolution by passing in some additional Pipes options as the third argument:
+## Defining Stage Dependencies
+
+Just as jobs can have dependencies, stages can as well.
+
+Imagine you have multiple jobs in a given stage, all of which have the same dependencies:
+
+```ruby
+Pipes.configure do |config|
+  config.stages do
+    content_writers [
+      {Writers::HTMLWriter => :publishers},
+      {Writers::AssetWriter => :publishers}
+    ]
+
+    publishers [
+      Publishers::Rsyncer
+    ]
+
+    notifiers [
+      Notifiers::FileActivator
+    ]
+  end
+end
+```
+
+This isn't so DRY. You would be better off adding a stage dependency:
+
+```ruby
+Pipes.configure do |config|
+  config.stages do
+    content_writers [
+      Writers::HTMLWriter,
+      Writers::AssetWriter
+    ] => :publishers
+
+    publishers [
+      Publishers::Rsyncer
+    ]
+
+    notifiers [
+      Notifiers::FileActivator
+    ]
+  end
+end
+```
+
+This tells Pipes that you would like all jobs in the `:content_writers` stage to have a depencency on all `:publishers`.
+
+You can intermix types for stage dependencies, just like with job dependencies:
+
+```ruby
+Pipes.configure do |config|
+  config.stages do
+    content_writers [
+      Writers::HTMLWriter,
+      Writers::AssetWriter
+    ] => [:publishers, Notifiers::FileActivator]
+
+    publishers [
+      Publishers::Rsyncer
+    ]
+
+    notifiers [
+      Notifiers::FileActivator
+    ]
+  end
+end
+```
+
+This will ensure that all `:publishers` and the `Notifiers::FileActivator` get run when either of the `:content_writers` are run.
+
+As you would expect, Pipes will resolve deep dependencies for you as well:
 
 ```ruby
-Pipes::Runner.run([Writers::HTMLWriter], 'http://localhost:3000/page', {resolve: false})
+Pipes.configure do |config|
+  config.stages do
+    content_writers [
+      Writers::HTMLWriter,
+      Writers::AssetWriter
+    ] => :publishers
+
+    publishers [
+      Publishers::Rsyncer
+    ] => :notifiers
+
+    notifiers [
+      Notifiers::FileActivator
+    ]
+  end
+end
 ```
 
-In the above code, only `Writers::HTMLWriter` will be run.
+The above will add `Publishers::Rsyncer` and `Notifiers::FileActivator` as dependencies of both `Writers::HTMLWriter` and `Writers::AssetWriter`.
+
+Intermixing job and stage dependencies works, too, resulting in the same dependency graph as the above example:
+
+```ruby
+Pipes.configure do |config|
+  config.stages do
+    content_writers [
+      Writers::HTMLWriter,
+      Writers::AssetWriter
+    ] => :publishers
+
+    publishers [
+      {Publishers::Rsyncer => :notifiers}
+    ]
+
+    notifiers [
+      Notifiers::FileActivator
+    ]
+  end
+end
+```
 
 ## Acceptable Formats for Jobs
 
@@ -258,16 +378,16 @@ Pipes allows you to specify your jobs in a variety of ways:
 
 ```ruby
 # A single job
-Pipes::Runner.run(Writers::HTMLWriter)
+Pipes.enqueue(Writers::HTMLWriter)
 
 # A single job as a string. Might be helpful if accepting params from a form
-Pipes::Runner.run('Writers::HTMLWriter')
+Pipes.enqueue('Writers::HTMLWriter')
 
 # An entire stage
-Pipes::Runner.run(:content_writers)
+Pipes.enqueue(:content_writers)
 
 # You can pass an array of any of the above, intermixing types
-Pipes::Runner.run([:content_writers, 'Publishers::CDNUploader', Notifiers::FileActivator])
+Pipes.enqueue([:content_writers, 'Publishers::CDNUploader', Notifiers::FileActivator])
 ```
 
 ## Configuring Pipes
@@ -284,7 +404,7 @@ Pipes.configure do |config|
   # config.namespace will specify a Redis namespace to use (default nil):
   config.namespace = 'my_project'
 
-  # config.resolve tells Pipes to resolve dependencies when calling Pipes::Runner.run(...) (default true):
+  # config.resolve tells Pipes to resolve dependencies when calling Pipes.enqueue(...) (default true):
   config.resolve = false
 
   config.stages do
@@ -295,9 +415,196 @@ end
 
 If you're using Pipes in a Rails app, stick your configuration in `config/initializers/pipes.rb`.
 
+## Pipes Options
+
+You can pass a hash of options when enqueueing workers through Pipes.
+
+**resolve**
+
+By default, Pipes will resolve and queue up all dependencies of the jobs you are requesting. You can turn off dependency resolution by passing in some additional Pipes options as the third argument:
+
+```ruby
+Pipes.enqueue(Writers::HTMLWriter, 'http://localhost:3000/page', {resolve: false})
+```
+
+When **resolve** is false, only `Writers::HTMLWriter` will be run, ignoring dependencies.
+
+**allow_duplicates**
+
+If jobs are already queued up in Pipes and you'd like to enqueue more jobs, you may need to specify that only certain jobs be duplicated in the queue.
+
+```ruby
+Pipes.enqueue(Writers::HTMLWriter, 'http://localhost:3000/page', {allow_duplicates: :content_writers})
+# ..or an array of stages..
+Pipes.enqueue(Writers::HTMLWriter, 'http://localhost:3000/page', {allow_duplicates: [:content_writers, :publishers]})
+```
+
+When Pipes enqueues `Writers::HTMLWriter` and all its dependencies, it will check whether any jobs with the same class name already exist in the queue. If a job has already been queued up with the same class name and **does not** belong to one of the stages provided to **allow_duplicates**, it is ignored.
+
+This option helps prevent adding redundant jobs to the queue. See the section *Queueing Up Additional Jobs*.
+
+## Working With Resque Priorities
+
+Pipes is designed to work on top of Resque's already-existing queueing system. That is, the queue priorities Resque provides will continue to be honored.
+
+Combining Resque's priority queues with Pipe's stages can produce fine-grain control over how your jobs get processed. By using the normal `@queue` instance variable, and specifying priorities when starting up your Resque workers, you can control the order in which jobs get processed for each individual stage.
+
+Say we had the following jobs:
+
+```ruby
+module Writers
+  class HTMLWriter
+    @queue = :priority_1
+
+    def self.perform; end
+  end
+end
+
+module Writers
+  class AssetWriter
+    @queue = :priority_2
+
+    def self.perform; end
+  end
+end
+```
+
+Both `Writer::HTMLWriter` and `Writer::AssetWriter` are configured for the same stage:
+
+```ruby
+Pipes.configure do |config|
+  config.stages do
+    content_writers [
+      Writers::HTMLWriter,
+      Writers::AssetWriter
+    ]
+  end
+end
+```
+
+Start up Resque with the usual priority list:
+
+```bash
+$ QUEUES=priority_1,priority_2 rake resque:work
+```
+
+Run the jobs through Pipes:
+
+```ruby
+Pipes.enqueue(:content_writers)
+```
+
+Pipes will queue up both `Writer::HTMLWriter` and `Writer::AssetWriter` in Resque. Resque takes over and respects the queue priorities, first running `Writers::HTMLWriter`, then `Writers::AssetWriter`.
+
+## Queueing Up Additional Jobs
+
+Say you have a job, `Writers::HTMLWriter`, whose purpose is to fire off additional jobs to accomplish the real work. This is actual the case for us at Factory Labs. Our `HTMLWriter` fires off additional jobs who do the heavy lifting of parsing page contents, and writing to a file.
+
+Our `HTMLWriter` fires off additional writers:
+
+```ruby
+module Writers
+  class HTMLWriter
+    @queue = :content_writers
+
+    def self.perform(locale)
+      Pages.all.each do |page|
+        # Enqueue additional jobs to do the real work
+        Pipes.enqueue(Writers::PageWriter, page.id, locale, {allow_duplicates: [:content_writers]})
+      end
+    end
+  end
+
+  class PageWriter
+    @queue = :content_writers
+
+    def self.perform(page_id, locale)
+      # We would normally do stuff with the locale...
+      url     = page_url(Page.find(page_id))
+      content = URI.parse(url).read
+
+      File.new('index.html', 'w') do |f|
+        f.write(content)
+      end
+    end
+  end
+end
+```
+
+Both jobs are configured for the same stage, with a dependency on `:publishers`:
+
+```ruby
+Pipes.configure do |config|
+  config.stages do
+    content_writers [
+      {Writers::HTMLWriter => :publishers},
+      {Writers::PageWriter => :publishers}
+    ]
+
+    publishers [
+      Publishers::Rsyncer
+    ]
+  end
+end
+```
+
+We fire off just the `HTMLWriter`:
+
+```ruby
+Pipes.enqueue(Writer::HTMLWriter, 'en-US')
+```
+
+Pipes queues up the `Writer::HTMLWriter` and its dependent, `Publishers::Rsyncer`. So, our queue looks like this:
+
+```ruby
+# Stage 1 (content_writers)
+Writers::HTMLWriter.perform('en-US')
+
+# Stage 2 (publishers)
+Publishers::Rsyncer.perform('en-US')
+```
+
+After processing the first job, `HTMLWriter`, the Pipes queue looks like this:
+
+```ruby
+# Stage 1 (content_writers)
+Writers::PageWriter.perform(1, 'en-US')
+Writers::PageWriter.perform(2, 'en-US')
+Writers::PageWriter.perform(3, 'en-US')
+...
+
+# Stage 2 (publishers)
+Publishers::Rsyncer.perform('en-US')
+```
+
+Pipes will ensure your stages stay intact when enqueueing additional jobs mid-pipe. That is, **Stage 2** jobs are still queued *after* additional jobs have been added to **Stage 1**. This applies to jobs added to any stages. You can continue to add jobs to any stage while Pipes is working.
+
+**The allow_duplicates option**
+
+By default, Pipes will check for exact duplicate jobs in the queue (eg `Writer::HTMLWriter` with argument `en-US`). If we don't provide the `allow_duplicates` option within the `HTMLWriter`'s `#perform` method, the Pipes queue would look like this:
+
+```ruby
+# Stage 1 (content_writers)
+(DONE) Writers::HTMLWriter.perform('en-US')
+Writers::PageWriter.perform(1, 'en-US')
+Writers::PageWriter.perform(2, 'en-US')
+Writers::PageWriter.perform(3, 'en-US')
+...
+
+# Stage 2 (publishers)
+Publishers::Rsyncer.perform('en-US')
+Publishers::Rsyncer.perform(1, 'en-US')
+Publishers::Rsyncer.perform(2, 'en-US')
+Publishers::Rsyncer.perform(3, 'en-US')
+```
+
+We only want to run rsync once, so this is incorrect. To prevent this from happening, we indicate that we only want duplicate `:content_writers` to the **allow_duplicates** option.
+
+By telling Pipes that we want to only allow duplicate `:content_writers`, we prevent duplicate `Rsyncer`s from being queued up, even though `PageWriter` has a `Rsyncer` dependency. **allow_duplicates** will force Pipes to check whether the `Rsyncer` class already exists in the queue (ignoring job arguments), and if so, skips that job.
+
 ## Support
 
-Pipes is currently tested under Ruby 1.9.3.
+Pipes makes use of Ruby 1.9's ordered hashes. No deliberate support for Ruby 1.8.
 
 ## Known Caveats
 
@@ -305,13 +612,13 @@ If your job is expecting a hash as the last argument, you'll need to pass an add
 
 ```ruby
 # Pipes will assume {follow_links: true} is options for Pipes, not your job:
-Pipes::Runner.run([Writers::HTMLWriter], {follow_links: true})
+Pipes.enqueue([Writers::HTMLWriter], {follow_links: true})
 
 # So you should pass a trailing hash to denote that there are no Pipes options:
-Pipes::Runner.run([Writers::HTMLWriter], {follow_links: true}, {})
+Pipes.enqueue([Writers::HTMLWriter], {follow_links: true}, {})
 
 # Of course, if you do specify options for Pipes, everything will work fine:
-Pipes::Runner.run([Writers::HTMLWriter], {follow_links: true}, {resolve: true})
+Pipes.enqueue([Writers::HTMLWriter], {follow_links: true}, {resolve: true})
 ```
 
 ## Future Improvements

data/lib/pipes.rb

@@ -8,13 +8,17 @@ module Pipes
     attr_accessor :namespace, :resolve
   end
 
+  # Open up the configuration for multiple values.
+  #   eg: Pipes.configuration { |config| ... }
+  #
   def self.configure(*args, &block)
     yield self
   end
 
-  # config.redis can be a string or a redis connection
+  # config.redis can be a string or a redis connection.
   #   eg: config.redis = 'localhost:6379'
   #   or  config.redis = $MY_REDIS
+  #
   def self.redis=(redis)
     if redis.is_a? String
       host, port = redis.split(':')
@@ -24,12 +28,21 @@ module Pipes
     end
   end
 
+  # Stages, defined in the configuration.
+  #
   def self.stages(*args, &block)
     Abyss.configure(*args) do
       stages &block
     end
   end
 
+  # Shortcut to queue up a Pipes job. Designed to look
+  # similar to Resque.
+  #
+  def self.enqueue(*args, &block)
+    Runner.run(*args, &block)
+  end
+
   private
 
   def self.set_redis(redis)

data/lib/pipes/stage_parser.rb

@@ -25,7 +25,7 @@ module Pipes
       if !@dependencies[job] or @dependencies[job].empty?
         []
       else
-        recursive_dependencies = @dependencies[job].map{ |strat| dependents_for(strat) }
+        recursive_dependencies = @dependencies[job].map{ |klass| dependents_for(klass) }
         (@dependencies[job] + recursive_dependencies).flatten.uniq
       end
     end
@@ -45,9 +45,12 @@ module Pipes
     #
     def stages_with_resolved_dependencies
       # Looping over all stages...
-      @stages.inject({}) do |resolved_stages, (name, stage)|
+      @stages.inject({}) do |resolved_stages, (name, jobs)|
+        # If it's defined with a stage dependency
+        jobs, _ = jobs.to_a.first if jobs.is_a? Hash
+
         # Looping over all jobs...
-        resolved_stages[name] = stage.inject([]) do |resolved_stage, job|
+        resolved_stages[name] = jobs.inject([]) do |resolved_stage, job|
           job = job.keys[0] if job.is_a? Hash
           # Normalze to new hash form
           resolved_stage << {job => @dependencies[job]}
@@ -73,16 +76,25 @@ module Pipes
       @dependencies = {}
 
       reversed = Hash[@stages.to_a.reverse]
-      reversed.each do |name, stage|
-        stage.each do |job|
+      reversed.each do |name, jobs|
+        if jobs.is_a? Hash
+          # Stage dependency present
+          jobs, stage_dependents = jobs.to_a.first
+        end
+
+        jobs.each do |job|
           # Does the job have dependents?
           if job.is_a? Hash
-            strat, dependents = job.to_a.first
-            @dependencies[strat] = dependencies_for_job(dependents)
+            job, dependents = job.to_a.first
+            @dependencies[job] = dependencies_for_job(dependents)
           else
             # Defined job is a simple class (eg Publisher)
             @dependencies[job] = []
           end
+
+          if stage_dependents
+            @dependencies[job] += dependencies_for_job(stage_dependents)
+          end
         end
       end
     end
@@ -105,17 +117,17 @@ module Pipes
     # Iterate over all jobs for this stage and find dependents.
     #
     def dependents_for_stage(stage_name)
-      stage = @stages[stage_name.to_sym]
+      stage = array_for_stage(@stages[stage_name.to_sym])
 
-      stage.inject([]) do |klasses, job|
+      stage.inject([]) do |jobs, job|
         # Does the job have dependents?
         if job.is_a? Hash
-          strat, dependents = job.to_a.first
-          klasses << strat
-          klasses << dependencies_for_job(dependents)
+          job, dependents = job.to_a.first
+          jobs << job
+          jobs << dependencies_for_job(dependents)
         else
           # Defined job is a simple class (eg Publisher)
-          klasses << [job] + dependents_for(job)
+          jobs << [job] + dependents_for(job)
         end
       end.flatten.uniq
     end
@@ -138,8 +150,10 @@ module Pipes
 
     # Just list the jobs in the stage, ignoring dependencies.
     #
-    def array_for_stage(stage)
-      stage.inject([]) do |arr, job|
+    def array_for_stage(jobs)
+      jobs, _ = jobs.to_a.first if jobs.is_a? Hash
+
+      jobs.inject([]) do |arr, job|
         arr << if job.is_a? Hash
           # Take just the job class, without any dependents
           job.keys[0]

data/lib/pipes/version.rb

@@ -1,3 +1,3 @@
 module Pipes
-  VERSION = '0.1.0'
+  VERSION = '0.2.0'
 end

data/pipes.gemspec

@@ -21,4 +21,5 @@ Gem::Specification.new do |gem|
   gem.add_dependency 'abyss', '~> 0.4.0'
 
   gem.add_development_dependency 'rspec'
+  gem.add_development_dependency 'rake'
 end

data/spec/pipes/runner_spec.rb

@@ -1,5 +1,4 @@
 require 'spec_helper'
-require 'mock_jobs'
 
 describe Pipes::Runner do
 

data/spec/pipes/stage_parser_spec.rb

@@ -1,5 +1,4 @@
 require 'spec_helper'
-require 'mock_jobs'
 
 describe Pipes::StageParser do
   subject { Pipes::StageParser }
@@ -133,27 +132,69 @@ describe Pipes::StageParser do
       end
     end
 
-    context 'with a comlex configuration, intermixing dependent types' do
+    context 'with the stage having a dependency on another stage (stage dependencies)' do
+      let(:stages) {
+        {
+          writers:    {[ Writers::ContentWriter, Writers::AnotherContentWriter ] => :publishers},
+          publishers: {[ Publishers::Publisher ] => :messengers},
+          messengers:  [ Messengers::SMS ]
+        }
+      }
+
+      it 'adds all classes within the dependent stages to the list' do
+        expected = {
+                     writers:    [ { Writers::ContentWriter        => [Publishers::Publisher, Messengers::SMS] },
+                                   { Writers::AnotherContentWriter => [Publishers::Publisher, Messengers::SMS] } ],
+                     publishers: [ { Publishers::Publisher => [Messengers::SMS] } ],
+                     messengers: [ { Messengers::SMS       => [] } ]
+                   }
+
+        subject.new(stages).stages_with_resolved_dependencies.should == expected
+      end
+    end
+
+    context 'with the stage dependency as a mixed array' do
+      let(:stages) {
+        {
+          writers:    {[ Writers::ContentWriter, Writers::AnotherContentWriter ] => [:publishers, Messengers::SMS]},
+          publishers:  [ Publishers::Publisher ],
+          messengers:  [ Messengers::SMS ]
+        }
+      }
+
+      it 'adds all classes within the dependent stages to the list' do
+        expected = {
+                     writers:    [ { Writers::ContentWriter        => [Publishers::Publisher, Messengers::SMS] },
+                                   { Writers::AnotherContentWriter => [Publishers::Publisher, Messengers::SMS] } ],
+                     publishers: [ { Publishers::Publisher => [] } ],
+                     messengers: [ { Messengers::SMS       => [] } ]
+                   }
+
+        subject.new(stages).stages_with_resolved_dependencies.should == expected
+      end
+    end
+
+    context 'with a complex configuration, intermixing dependent types' do
       let (:stages) {
         {
-          writers:    [ { Writers::ContentWriter        => [:publishers, Uploaders::Rsync] },
-                        { Writers::AnotherContentWriter => [Emailers::Email] }
-                      ],
-          publishers: [ { Publishers::Publisher => :emailers } ],
-          messengers: [ { Messengers::SMS => :uploaders } ],
-          uploaders:  [ { Uploaders::Rsync => Notifiers::Twitter } ],
-          emailers:   [ Emailers::Email, Emailers::AnotherEmail ],
-          notifiers:  [ Notifiers::Twitter ]
+          writers:     [ { Writers::ContentWriter        => [:publishers, Uploaders::Rsync] },
+                         { Writers::AnotherContentWriter => [Emailers::Email] }
+                       ],
+          publishers: {[ { Publishers::Publisher => :emailers } ] => :notifiers},
+          messengers:  [ { Messengers::SMS => :uploaders } ],
+          uploaders:   [ { Uploaders::Rsync => Notifiers::Twitter } ],
+          emailers:    [ Emailers::Email, Emailers::AnotherEmail ],
+          notifiers:   [ Notifiers::Twitter ]
         }
       }
 
       it 'resolves all dependencies' do
         expected = {
                      writers:    [ { Writers::ContentWriter        => [Publishers::Publisher, Emailers::Email, Emailers::AnotherEmail,
-                                                                       Uploaders::Rsync, Notifiers::Twitter] },
+                                                                       Notifiers::Twitter, Uploaders::Rsync] },
                                    { Writers::AnotherContentWriter => [Emailers::Email] }
                                  ],
-                     publishers: [ { Publishers::Publisher  => [Emailers::Email, Emailers::AnotherEmail] } ],
+                     publishers: [ { Publishers::Publisher  => [Emailers::Email, Emailers::AnotherEmail, Notifiers::Twitter] } ],
                      messengers: [ { Messengers::SMS        => [Uploaders::Rsync, Notifiers::Twitter] } ],
                      uploaders:  [ { Uploaders::Rsync       => [Notifiers::Twitter] } ],
                      emailers:   [ { Emailers::Email        => [] },

data/spec/pipes/store_spec.rb

@@ -1,5 +1,4 @@
 require 'spec_helper'
-require 'mock_jobs'
 
 describe Pipes::Store do
 

data/spec/pipes_spec.rb

@@ -43,4 +43,13 @@ describe Pipes do
 
   end
 
+  describe '.enqueue' do
+
+    it 'delegates to Runner.run' do
+      Pipes::Runner.should_receive(:run).with(Writers::ContentWriter, 'some arg', {resolve: true})
+      Pipes.enqueue(Writers::ContentWriter, 'some arg', {resolve: true})
+    end
+
+  end
+
 end

data/spec/spec_helper.rb

@@ -1,6 +1,8 @@
 require 'rspec'
 RSpec::Mocks.setup(Object)
 
+require 'mock_jobs'
+
 require 'pipes'
 
 Pipes.namespace = 'test'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pipes
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-09-21 00:00:00.000000000 Z
+date: 2012-09-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: resque
-  requirement: !ruby/object:Gem::Requirement
+  requirement: &70304995066400 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -21,15 +21,10 @@ dependencies:
         version: 1.22.0
   type: :runtime
   prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 1.22.0
+  version_requirements: *70304995066400
 - !ruby/object:Gem::Dependency
   name: redis-objects
-  requirement: !ruby/object:Gem::Requirement
+  requirement: &70304995064640 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -37,15 +32,10 @@ dependencies:
         version: 0.5.3
   type: :runtime
   prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 0.5.3
+  version_requirements: *70304995064640
 - !ruby/object:Gem::Dependency
   name: abyss
-  requirement: !ruby/object:Gem::Requirement
+  requirement: &70304995063140 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -53,15 +43,10 @@ dependencies:
         version: 0.4.0
   type: :runtime
   prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    none: false
-    requirements:
-    - - ~>
-      - !ruby/object:Gem::Version
-        version: 0.4.0
+  version_requirements: *70304995063140
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: !ruby/object:Gem::Requirement
+  requirement: &70304995061340 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -69,12 +54,18 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
+  version_requirements: *70304995061340
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: &70304995059980 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70304995059980
 description: A Redis-backed concurrency management system
 email:
 - mikepackdev@gmail.com
@@ -85,6 +76,7 @@ files:
 - .gitignore
 - .rspec
 - .rvmrc
+- .travis.yml
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -125,7 +117,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 1.8.10
 signing_key: 
 specification_version: 3
 summary: A Redis-backed concurrency management system
@@ -138,3 +130,4 @@ test_files:
 - spec/pipes/utils_spec.rb
 - spec/pipes_spec.rb
 - spec/spec_helper.rb
+has_rdoc: