resque-fanout 0.5

data/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) Frederik Fix
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,110 @@
+Resque Plugin for pubsub routing between queues
+===============================================
+
+This plugin allows you to define "exchanges" which look like regular Resque queues, but rather than processing jobs they distribute them to one or more other queues. The mapping between exchanges and queues can change at runtime. A code based configuration method as well as a web frontend are provided.
+
+This is useful for loosely coupled asynchronous communication between multiple applications.
+
+
+Consider a system consisting of three applications. The main frontend application, an internal one processing billing and another handling user accounts. When a new user registers work needs to be performed in the last two.
+
+
+In the billing application define the following worker:
+
+``` ruby
+class BillingListener
+
+  @queue = :payroll
+
+  def self.perform(user_hash)
+    
+  end
+
+end
+
+Resque.subscribe :new_user, :class => BillingListener
+```
+
+In the user accounts application define this worker:
+
+``` ruby
+class AccountListener
+
+  @queue = :accounts
+
+  def self.perform(user_hash)
+    
+  end
+
+end
+
+Resque.subscribe :new_user, :class => AccountListener
+```
+
+When a new user registers in the frontend application execute:
+
+``` ruby
+Resque.publish :new_user, :user_name => :feynman, :account_type => :qed
+```
+
+The job will then be processed by both the BillingListener and the AccountListener in the context of the respective applications.
+
+
+PubSub semantics
+================
+
+The mapping between exchange and queues is maintained in the Redis server and can therefore change at runtime. The job distribution is performed upon job submission. Exchanges override queues with the same name.
+
+The mapping is written to Redis by the following call:
+
+``` ruby
+Resque.subscribe :new_user, :class => AccountListener
+```
+
+This call will do two things
+
+1. Add the queue defined for AccountListener to the outbound list for the exchange "new_user"
+2. Set the class AccountListener as the class handling the request on the queue defined in step 1
+
+The second step is necessary because the class handling the jobs sent to the exchange "new_user" might be different in the respective receiving applications. The receiving class is set by setting the "class" attribute in the Job definition when it is pushed to the queue.
+
+The above semantics allow you to reroute an existing queue. Consider the above example but the frontend application also defines the following worker:
+
+``` ruby
+class NewUserListener
+
+  @queue = :new_user
+
+  def self.perform(user_hash)
+    
+  end
+
+end
+
+Resque.subscribe :new_user, :class => NewUserListener
+```
+
+If you then do the following in the frontend application,
+
+``` ruby
+Resque.enqueue(NewUserListener, :user_name => :feynman, :account_type => :qed)
+```
+
+the job will be performed by all three workers in their respective application contexts.
+
+
+The `Resque.subscribe` method provides another mode which does not set the handling class:
+
+``` ruby
+Resque.subscribe :new_user, :queue => :accounts
+```
+
+This will of course only work if the receiving applications have a worker called `NewUserListener`. Using `Resque.publish` with this exchange will cause an exception.
+
+
+Managing the Exchange -> Queue mapping
+======================================
+
+The mapping can be defined either by the `Resque.subscribe` method or through the web frontend. A relevant mapping is created by the `Resque.subscribe` call. Therefore if you delete a mapping in the web frontend but rerun `Resque.subscribe` it will reappear. Depending on your circumstances this may or may not be what you expect. It is recommended to place your mapping definitions in a Rake task similar to  `rake db:migrate`
+
+

data/Rakefile

@@ -0,0 +1,9 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = ["-c", "-f progress"]
+end
+
+task :default => :spec

data/lib/resque-fanout/fanout_server.rb

@@ -0,0 +1,54 @@
+module Resque
+  module FanoutServer
+
+    VIEW_PATH = File.join(File.dirname(__FILE__), 'views')
+
+    def self.registered(app)
+      # index
+      app.get "/fanout/?" do
+        @exchanges = Resque.exchanges
+        @queues = Resque.queues
+        erb(File.read(File.join(::Resque::FanoutServer::VIEW_PATH, "index.erb")))
+      end
+
+      # create new mapping
+      app.post '/fanout/?' do
+        if valid_mapping? params
+          Resque.subscribe params[:exchange], :queue => params[:queue], :class => params[:class]
+          redirect u(:fanout, "?notice=#{URI.escape 'Mapping created'}")
+        else
+          redirect u(:fanout, "?error=#{URI.escape 'Invalid Mapping'}")
+        end
+      end
+
+      # delete mapping
+      app.post '/fanout/:exchange/:queue/remove' do
+        if valid_mapping? params
+          Resque.unsubscribe params[:exchange], :queue => params[:queue]
+          redirect u(:fanout, "?notice=#{URI.escape 'Mapping removed'}")
+        else
+          redirect u(:fanout, "?error=#{URI.escape 'Invalid Parameters'}")
+        end
+      end
+
+      app.helpers do
+        def valid_mapping?(params)
+          params[:exchange].to_s != "" and params[:queue].to_s != ""
+        end
+
+        def display_flash
+          [:error, :notice].map do |field|
+            if params[field].to_s != ""
+              "<p class='flash #{field}'>#{params[field]}</p>"
+            end
+          end.compact.join
+        end
+      end
+
+      app.tabs << "Fanout"
+    end
+
+  end
+end
+
+Resque::Server.register Resque::FanoutServer

data/lib/resque-fanout/job_extension.rb

@@ -0,0 +1,17 @@
+class Resque::Job
+
+  def self.create(queue, klass, *args)
+    Resque.validate(klass, queue)
+
+    if Resque.inline?
+      constantize(klass).perform(*decode(encode(args)))
+    elsif queues = Resque.queues_for(queue)
+      queues.each do |_queue|
+        Resque.push(_queue[:queue], :class => (_queue[:class] || klass.to_s), :args => args)
+      end
+    else
+      Resque.push(queue, :class => klass.to_s, :args => args)
+    end
+  end
+
+end

data/lib/resque-fanout/resque_extension.rb

@@ -0,0 +1,57 @@
+module ResqueExtension
+
+  def self.apply
+    Resque.extend ResqueExtension
+  end
+
+
+  def subscribe(exchange, options={})
+    raise ArgumentError, "either class or queue param must be supplier" unless options[:queue] || options[:class]
+
+    queue = options[:queue] || queue_from_class(options[:class])
+    redis.sadd("exchanges:#{exchange.to_s}", queue.to_s)
+    redis.hset("exchanges:class:#{exchange.to_s}", queue.to_s, options[:class].to_s) if options[:class]
+    redis.sadd(:exchanges, exchange.to_s)
+  end
+
+  def unsubscribe(exchange, options={})
+    raise ArgumentError, "either class or queue param must be supplier" unless options[:queue] || options[:class]
+
+    queue = options[:queue] || queue_from_class(options[:class])
+    redis.srem("exchanges:#{exchange.to_s}", queue.to_s)
+    redis.hdel("exchanges:class:#{exchange.to_s}", queue.to_s)
+    if redis.scard("exchanges:#{exchange.to_s}") == 0
+      redis.srem(:exchanges, exchange.to_s)
+    end
+  end
+
+  def queues_for(exchange)
+    if redis.sismember(:exchanges, exchange.to_s)
+      klasses = Hash[redis.hgetall("exchanges:class:#{exchange.to_s}")]
+      redis.smembers("exchanges:#{exchange.to_s}").map do |queue|
+        res = { :queue => queue }
+        res[:class] = klasses[queue] if klasses[queue]
+        res
+      end
+    end
+  end
+
+  def exchanges
+    redis.smembers(:exchanges).map do |exchange|
+      {
+        :exchange => exchange,
+        :queues => queues_for(exchange)
+      }
+    end
+  end
+
+
+  def publish(exchange, *args)
+    (queues_for(exchange) || []).each do |queue|
+      push queue[:queue], :class => queue[:class], :args => args
+    end
+  end
+
+end
+
+ResqueExtension.apply

data/lib/resque-fanout/views/index.erb

@@ -0,0 +1,98 @@
+<style type="text/css" media="screen">
+  .flash {
+    font-size: 190%;
+    font-weight: bold;
+    text-align: center;
+    padding: 20px;
+  }
+
+  .flash.notice {
+    background: #61BF55;
+  }
+
+  .flash.error {
+    background: #E47E74;
+  }
+
+  #main form.fanout-remove {
+    float: left;
+    margin-left: 20px;
+    margin-top: -5px;
+  }
+  .mapping span {
+    float: left;
+  }
+
+  #main #new-mapping {
+    width: 100%;
+    margin-top: 10px;
+  }
+
+  #new-mapping p {
+    clear: left;
+  }
+
+  #new-mapping label {
+    width: 15em;
+    display: block;
+    float: left;
+  }
+</style>
+
+<%= display_flash %>
+
+<h1 class='wi'>Fanout Exchanges</h1>
+
+<% if @exchanges.empty? %>
+  <p class='intro'>There are no exchanges defined</p>
+<% else %>
+  <p class='intro'>There are <%= @exchanges.size %> exchanges defined</p>
+
+  <table>
+    <tr>
+      <th>Exchange</th>
+      <th>Queues</th>
+    </tr>
+    <% for exchange in @exchanges %>
+      <tr>
+        <td><%= exchange[:exchange] %></td>
+        <td>
+          <ul>
+            <% for queue in exchange[:queues] %>
+              <li class="mapping">
+                <span><%= queue[:queue] %><%= "(=> #{queue[:class]})" if queue[:class] %></span>
+                <form method="POST" action="<%= u(:fanout) %>/<%= exchange[:exchange] %>/<%= queue[:queue] %>/remove" class='fanout-remove'>
+                  <input type='submit' name='' value='Remove Mapping' onclick='return confirm("Are you absolutely sure? This cannot be undone.");' />
+                </form>
+              </li>
+            <% end %>
+          </ul>
+        </td>
+      </tr>
+    <% end %>
+  </table>
+<% end %>
+
+<h2>Add new mapping</h2>
+
+<form method="POST" action="<%= u(:pubsub) %>" id='new-mapping'>
+  <p>
+    <label>Exchange Name</label>
+    <input type="text" name="exchange">
+  </p>
+  <p>
+    <label>Queue Name</label>
+    <select name="queue">
+      <% for queue in @queues %>
+        <option><%= queue[:name] %></option>
+      <% end %>
+    </select>
+  </p>
+  <p>
+    <label>Class Name (optional)</label>
+    <input type="text" name="class">
+  </p>
+  <input type='submit' name='' value='Create Mapping' />
+</form>
+
+

data/lib/resque-fanout.rb

@@ -0,0 +1,8 @@
+$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + '/../lib'))
+
+require 'resque'
+require 'resque/server'
+
+require 'resque-fanout/resque_extension'
+require 'resque-fanout/job_extension'
+require 'resque-fanout/pubsub_server'

metadata

@@ -0,0 +1,120 @@
+--- !ruby/object:Gem::Specification
+name: resque-fanout
+version: !ruby/object:Gem::Version
+  version: '0.5'
+  prerelease: 
+platform: ruby
+authors:
+- Frederik Fix
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-04-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.3.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.3.0
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.1
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.1
+- !ruby/object:Gem::Dependency
+  name: resque
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.19.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 1.19.0
+description: ! 'A Resque plugin that provides endpoints which distributes Jobs submitted
+  to them to (multiple) subscribing queues. Useful for loosely coupled inter-application
+  communication.
+
+'
+email: ich@derfred.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- Rakefile
+- LICENSE
+- lib/resque-fanout/fanout_server.rb
+- lib/resque-fanout/job_extension.rb
+- lib/resque-fanout/resque_extension.rb
+- lib/resque-fanout/views/index.erb
+- lib/resque-fanout.rb
+homepage: http://github.com/derfred/resque-fanout
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: Resque Plugin for fanout routing to multiple queues
+test_files: []