ruote-amqp 2.1.5 → 2.2.0

data/{History.txt → CHANGELOG.txt}

@@ -1,22 +1,36 @@
-=== 2.0.0 (WIP)
+
+= ruote-amqp
+
+
+== ruote-amqp - 2.2.0    released 2011/03/01
+
+- receiver : exposing #decode_workitem for overwriting
+- options[:queue] patch by Mario Camou
+- :durable path by belucid
+- 1 MQ per dispatch thread 'leak' fix. Thanks weifeng365
+- RuoteAMQP::Participant --> RuoteAMQP::ParticipantProxy
+
+
+== 2.0.0 not released
 
 * Compatible with ruote 2.0
   * Thanks to John Mettraux (http://github.com/jmettraux/ruote-amqp)
   * Thanks to Jason & Jordan (http://github.com/asm/ruote-amqp)
   * Thanks to Charles Magid (http://github.com/ChasManRors/ruote-amqp)
 
-=== 0.9.21.1 2009-08-03
+== 0.9.21.1 2009-08-03
 
 * Switch to using persistent AMQP messages by default
 
-=== 0.9.21 2009-07-13
+== 0.9.21 2009-07-13
 
 * Depend on ruote-0.9.21 for flexible JSON backends
 * Support for default queues
 * Support for mapping participant names to queue names
 * Plenty of RDOC fixed
 
-=== 0.9.20 2009-07-13
+== 0.9.20 2009-07-13
 
 * 1 major enhancement:
   * Initial release
+

data/CREDITS.txt

@@ -0,0 +1,29 @@
+
+= CREDITS
+
+(probably incomplete, don't hesitate to tell us if a mention is missing)
+
+
+== AUTHORS
+
+* Kenneth Kalmer - http://www.opensourcery.co.za/
+* John Mettraux - https://github.com/jmettraux
+
+
+== CONTRIBUTORS
+
+* Mario Camou
+* Sean Johnson - https://github.com/belucid
+* Victor Liu - https://github.com/pennymax
+* weifeng - https://github.com/weifeng365
+* David Greaves - https://github.com/lbt
+* Hartog C. de Mik - https://github.com/coffeeaddict
+* Torsten Schoenebaum - https://github.com/tosch
+* Jordan Ritter - https://github.com/jpr5
+* Charles Magid - https://github.com/ChasManRors
+* Marc Mauger - https://github.com/simianarmy
+* Jason - https://github.com/asm
+
+
+== FEEDBACK
+

data/README.rdoc

@@ -1,3 +1,4 @@
+
 = ruote-amqp
 
 * http://github.com/kennethkalmer/ruote-amqp
@@ -16,7 +17,7 @@ http://ruote.rubyforge.org/part_implementations.html
 == FEATURES/PROBLEMS:
 
 * Flexible participant for sending workitems
-* Flexible listener for receiving replies
+* Flexible receiver for receiving replies
 * Flexible launch item listener for launching processes over AMQP
 * Fully evented (thanks to the amqp gem)
 
@@ -26,9 +27,8 @@ Please review the rdoc in RuoteAMQP::Participant and Ruote::AMQP::Listener
 
 == REQUIREMENTS:
 
-* ruote[http://ruote.rubyforge.org] 2.1.4 or later
-* amqp[http://github.com/tmm1/amqp] 0.6.6 or later
-* rufus-json[http://github.com/jmettraux/rufus-json] 0.1.0 or later
+* ruote[http://ruote.rubyforge.org] 2.2.0 or later
+* amqp[http://github.com/tmm1/amqp] 0.6.7 or later
 * rabbitmq[http://www.rabbitmq.com/] 1.6.0 or later
 
 == INSTALL:
@@ -40,7 +40,7 @@ Please be sure to have read the requirements section above
 == TESTS:
 
 To run the tests you need the following requirements met, or the testing environment
-will fail horribly.
+will fail horribly (or simply get stuck without output).
 
 === RabbitMQ vhost
 
@@ -69,7 +69,7 @@ DaemonKit doesn't currently support ruote 2.1, support is forthcoming.
 
 (The MIT License)
 
-Copyright (c) 2010 Kenneth Kalmer
+Copyright (c) 2010-2011 Kenneth Kalmer
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/Rakefile

@@ -1,50 +1,101 @@
-require 'rubygems'
-require 'rake'
-
-require 'lib/ruote-amqp'
-
-begin
-  require 'jeweler'
-  Jeweler::Tasks.new do |gemspec|
-    gemspec.name = 'ruote-amqp'
-    gemspec.version = RuoteAMQP::VERSION
-    gemspec.summary = 'AMQP participant/listener pair for ruote 2.1'
-    gemspec.email = 'kenneth.kalmer@gmail.com'
-    gemspec.homepage = 'http://github.com/kennethkalmer/ruote-amqp'
-    gemspec.authors = ['kenneth.kalmer@gmail.com']
-    gemspec.extra_rdoc_files.include '*.txt'
-
-    gemspec.add_dependency 'rufus-json', '>= 0.1.0'
-    gemspec.add_dependency 'amqp', '>= 0.6.6'
-    gemspec.add_dependency 'ruote', '>= 2.1.5'
-    gemspec.add_development_dependency 'rspec'
-  end
-  Jeweler::GemcutterTasks.new
-rescue LoadError
-  puts "Jeweler not available. Install it with 'gem install jeweler'"
-end
-
-require 'spec/rake/spectask'
-Spec::Rake::SpecTask.new(:spec) do |spec|
-  spec.libs << 'lib' << 'spec'
-  spec.spec_files = FileList['spec/**/*_spec.rb']
-end
-
-Spec::Rake::SpecTask.new(:rcov) do |spec|
-  spec.libs << 'lib' << 'spec'
-  spec.pattern = 'spec/**/*_spec.rb'
-  spec.rcov = true
-end
-
-task :spec #=> :check_dependencies
-
-task :default => :spec
-
-begin
-  require 'yard'
-  YARD::Rake::YardocTask.new
-rescue LoadError
-  task :yardoc do
-    abort "YARD is not available. In order to run yardoc, you must: sudo gem install yard"
-  end
-end
+
+$:.unshift('.') # 1.9.2
+
+require 'rubygems'
+require 'rubygems/user_interaction' if Gem::RubyGemsVersion == '1.5.0'
+
+require 'rake'
+require 'rake/clean'
+require 'rake/rdoctask'
+
+
+#
+# clean
+
+CLEAN.include('pkg', 'rdoc')
+
+
+#
+# test / spec
+
+task :spec do
+
+  sh 'rspec spec/'
+end
+
+task :test => [ :spec ]
+task :default => [ :spec ]
+
+
+#
+# gem
+
+GEMSPEC_FILE = Dir['*.gemspec'].first
+GEMSPEC = eval(File.read(GEMSPEC_FILE))
+GEMSPEC.validate
+
+
+desc %{
+  builds the gem and places it in pkg/
+}
+task :build do
+
+  sh "gem build #{GEMSPEC_FILE}"
+  sh "mkdir pkg" rescue nil
+  sh "mv #{GEMSPEC.name}-#{GEMSPEC.version}.gem pkg/"
+end
+
+desc %{
+  builds the gem and pushes it to rubygems.org
+}
+task :push => :build do
+
+  sh "gem push pkg/#{GEMSPEC.name}-#{GEMSPEC.version}.gem"
+end
+
+
+#
+# rabbitmq preparation
+
+desc %{
+  prepare RabbitMQ (vhost, user, perms)
+}
+task :prepare do
+
+  sh "rabbitmqctl add_vhost ruote-test"
+  sh "rabbitmqctl add_user ruote ruote"
+  sh "rabbitmqctl set_permissions -p ruote-test ruote '.*' '.*' '.*'"
+end
+
+
+#
+# rdoc
+#
+# make sure to have rdoc 2.5.x to run that
+
+Rake::RDocTask.new do |rd|
+
+  rd.main = 'README.rdoc'
+  rd.rdoc_dir = 'rdoc'
+
+  rd.rdoc_files.include(
+    'README.rdoc', 'CHANGELOG.txt', 'CREDITS.txt', 'lib/**/*.rb')
+
+  rd.title = "#{GEMSPEC.name} #{GEMSPEC.version}"
+end
+
+
+#
+# upload_rdoc
+
+desc %{
+  upload the rdoc to rubyforge
+}
+task :upload_rdoc => [ :clean, :rdoc ] do
+
+  account = 'jmettraux@rubyforge.org'
+  webdir = '/var/www/gforge-projects/ruote'
+
+  sh "rsync -azv -e ssh rdoc/#{GEMSPEC.name}_rdoc #{account}:#{webdir}/"
+end
+

data/TODO.txt

@@ -1,4 +1,8 @@
-ruote-amqp TODO
-===============
 
-[ ] Launch processes over AMQP
+[o] Launch processes over AMQP (LaunchitemListener)
+[o] Receiver
+[o] ParticipantProxy ?
+
+[ ] have a class method ParticipantProxy.stop_all ?
+[ ] use Ruote::Workitem #as_json and #from_json(s)
+

data/lib/ruote-amqp.rb

@@ -1,5 +1,10 @@
+
 require 'mq'
 
+require 'ruote-amqp/version'
+
+
+#
 # AMQP participant and listener pair for ruote.
 #
 # == Documentation
@@ -15,9 +20,9 @@ require 'mq'
 #
 module RuoteAMQP
 
-  VERSION = '2.1.5'
+  autoload 'ParticipantProxy',   'ruote-amqp/participant'
 
-  autoload 'Participant',        'ruote-amqp/participant'
+  autoload 'Receiver',           'ruote-amqp/receiver'
   autoload 'WorkitemListener',   'ruote-amqp/workitem_listener'
   autoload 'LaunchitemListener', 'ruote-amqp/launchitem_listener'
 
@@ -70,6 +75,6 @@ module RuoteAMQP
       Thread.main[:ruote_amqp_connection].join
       Thread.main[:ruote_amqp_started] = false
     end
-
   end
 end
+

data/lib/ruote-amqp/launchitem_listener.rb

@@ -1,88 +1,22 @@
+
 module RuoteAMQP
 
-  # = AMQP Launchitem Listener
-  #
-  # Used on its own, the RuoteAMQP::LaunchitemListener provides the engine with
-  # a way to launch process definitions over an AMQP direct exchange.
-  #
-  # == Message Format
-  #
-  # The LaunchitemListener expects JSON formatted messages that look like this:
-  #
-  #   {
-  #     "definition" : "process definition",
-  #     "fields" : { "key" : "value" },
-  #     "variables" : { "key" : "value" }
-  #  }
-  #
-  # The definition key is a complete string representation of a business process.
-  #
-  # == Configuration
   #
-  # AMQP configuration is handled by directly manipulating the values of the
-  # +AMQP.settings+ hash, as provided by the AMQP gem. No defaults are set by
-  # the listener. The only +option+ parsed by the initializer is the +queue+
-  # key (in the optional hash). If no +queue+ key is provided, the listener
-  # will subscribe to the +ruote_launchitems+ direct exchange for launchitems.
+  # Got replaced by RuoteAMQP::Receiver
   #
-  # The listener requires version 0.6.6 or later of the amqp gem.
+  # This class is kept for backward compatibility.
   #
-  # == Usage
-  #
-  # Register the engine with the listener:
-  #
-  #   RuoteAMQP::LaunchitemListener.new( engine_instance )
-  #
-  # The workitem listener leverages the asynchronous nature of the amqp gem,
-  # so no timers are setup when initialized.
-  class LaunchitemListener < Ruote::Receiver
-
-    class << self
-
-      # Listening queue - set this before initialization
-      attr_writer :queue
-
-      def queue
-        @queue ||= 'ruote_launchitems'
-      end
-
-    end
+  class LaunchitemListener < ::RuoteAMQP::Receiver
 
     # Start a new LaunchItem listener
     #
-    # @param [ Ruote::Engine ] An instance of a ruote engine
-    # @param [ String ] Optional queue name
-    def initialize( engine, queue = nil )
-
-      self.class.queue = queue if queue
-
-      RuoteAMQP.start!
-
-      MQ.queue( self.class.queue, :durable => true ).subscribe do |message|
-        if AMQP.closing?
-          # Do nothing, we're going down
-        else
-          launchitem = decode_launchitem( message )
-          engine.launch( *launchitem )
-        end
-      end
-    end
-
-    def stop
-      RuoteAMQP.stop!
-    end
-
-    private
-
-    # Complicated guesswork that needs to happen here to detect the format
-    def decode_launchitem( msg )
-      hash = Rufus::Json.decode( msg )
-      opts = {}
-      definition = hash.delete('definition')
-      fields = hash.delete('fields') || {}
-      variables = hash.delete('variables') || {}
+    # @param [ Ruote::Engine, Ruote::Storage ] A configured ruote engine or storage instance
+    # @param opts :queue / :unsubscribe
+    #
+    def initialize(engine_or_storage, opts={})
 
-      [ definition, fields, variables ]
+      super(engine_or_storage, opts.merge(:launchitems => :only))
     end
   end
 end
+

data/lib/ruote-amqp/participant.rb

@@ -1,19 +1,37 @@
+
 require 'ruote/part/local_participant'
+require 'ruote-amqp'
+
 
 module RuoteAMQP
 
+  #
   # = AMQP Participants
   #
-  # The RuoteAMQP::Participant allows you to send workitems (serialized as
+  # The RuoteAMQP::ParticipantProxy allows you to send workitems (serialized as
   # JSON) or messages to any AMQP queues right from the process
-  # definition. When combined with the RuoteAMQP::Listener you can easily
+  # definition. When combined with the RuoteAMQP::Receiver you can easily
   # leverage an extremely powerful local/remote participant
   # combinations.
   #
-  # By default the participant relies on the presence of an AMQP
-  # listener. Workitems are sent and no replies are given to the
-  # engine. The participant can be configured to reply to the engine
-  # immediately after queueing a message, see the usage section below.
+  # For local/remote participants The local part of the
+  # RuoteAMQP::ParticipantProxy relies on the presence of a
+  # RuoteAMQP::Receiver. Workitems are sent to the remote participant
+  # and the local part does not normally reply to the engine.  Instead
+  # the engine will continue when a reply is received on the
+  # 'ruote_workitems' queue (see RuoteAMQP::Receiver).
+  #
+  # Of course, the standard :forget => true format can be used even
+  # with remote particpants and :forget can even be set as a default in
+  # the options.
+  #
+  # A simple way to create a remote participant to act upon workitems
+  # is to use the daemon-kit ruote responder.
+  #
+  # Simple AMQP messages are treated as 'fire and forget' and the flow
+  # will continue when the local participant has queued the message
+  # for sending. (As there is no meaningful way to receive a workitem
+  # in reply).
   #
   # == Configuration
   #
@@ -21,86 +39,59 @@ module RuoteAMQP
   # values of the +AMQP.settings+ hash, as provided by the AMQP
   # gem. No AMQP defaults are set by the participant.
   #
-  # The participant requires version 0.6.1 or later of the amqp gem.
-  #
   # == Usage
   #
-  # Currently it's possible to send either workitems or messages
-  # directly to a specific queue, and have the engine wait for
-  # replies on another queue (see AMQPListener).
-  #
-  # Setting up the participant
+  # Define the queue used by an AMQP participant :
   #
   #   engine.register_participant(
-  #     :amqp, RuoteAMQP::Participant )
+  #     :delete_user, RuoteAMQP::ParticipantProxy, 'queue' => 'user_manager')
   #
-  # Setup a participant that always replies to the engine
+  # Sending a workitem to the remote participant defined above:
   #
-  #   engine.register_participant(
-  #     :amqp, RuoteAMQP::Participant.new(:reply_by_default => true ) )
-  #
-  # Sending a message example
-  #
-  #   class AmqpMessageExample0 < OpenWFE::ProcessDefinition
+  #   Ruote.process_definition do
   #     sequence do
-  #       amqp :queue => 'test', :message => 'foo'
+  #       delete_user
   #     end
   #   end
   #
-  # Sending a workitem
+  # Let the local participant reply to the engine without involving
+  # the receiver
   #
-  #   class AmqpWorkitemExample0 < OpenWFE::ProcessDefinition
+  #   Ruote.process_definition do
   #     sequence do
-  #       amqp :queue => 'test'
+  #       delete_user :forget => true
   #     end
   #   end
   #
-  # Let the participant reply to the engine without involving the listener
+  # Setting up the participant in a slightly more 'raw' way:
+  #
+  #   engine.register_participant(
+  #     :amqp, RuoteAMQP::ParticipantProxy )
+  #
+  # Sending a workitem to a specific queue:
   #
-  #   class AmqpWaitExample < OpenWFE::ProcessDefinition
+  #   Ruote.process_definition do
   #     sequence do
-  #       amqp :queue => 'test', :reply_anyway => true
+  #       amqp :queue => 'test', 'command' => '/run/regression_test'
   #     end
   #   end
   #
-  # When waiting for a reply it only makes sense to send a workitem.
-  #
-  # Keeping things DRY with participant name to queue maps:
-  #
-  #   amqp = RuoteAMQP::Participant.new( :default_queue => 'test' )
-  #   amqp.map_participant 'george', 'whitehouse'
-  #   amqp.map_participant 'barak', 'whitehouse'
-  #   amqp.map_participant 'greenspan', 'treasury'
+  # Setup a 'fire and forget' participant that always replies to the
+  # engine:
   #
-  #   engine.register_participant( :george, amqp )
-  #   engine.register_participant( :barak, amqp )
-  #   engine.register_participant( :greespan, amqp )
-  #   engine.register_participant( :amqp, amqp )
-  #
-  #   class DryAmqProcess0 < OpenWFE::ProcessDefinition
-  #     cursor :break_if => "${f:economy_recovered}" do
-  #       # Workitem sent to 'whitehouse' queue
-  #       george :activity => 'Tank economy'
-  #
-  #       # Workitem sent to 'treasury' queue
-  #       greenspan :activity => 'Resign'
+  #   engine.register_participant(
+  #     :jfdi, RuoteAMQP::ParticipantProxy, 'forget' => true )
   #
-  #       # Workitem sent to 'whitehouse' queue
-  #       barak :activity => 'Cleanup mess'
+  # Sending a message example to a specific queue (both steps are
+  # equivalent):
   #
-  #       # Workitem sent to default 'test' queue
-  #       amqp :activity => 'Notify CNN'
+  #   Ruote.process_definition do
+  #     sequence do
+  #       amqp :queue => 'test', :message => 'foo'
+  #       amqp :queue => 'test', :message => 'foo', :forget => true
   #     end
   #   end
   #
-  # == Workitem modifications
-  #
-  # To ease replies, and additional workitem attribute is set:
-  #
-  #   'reply_queue'
-  #
-  # +reply_queue+ has the name of the queue where the RuoteAMQP::Listener
-  # expects replies from remote participants
   #
   # == AMQP notes
   #
@@ -112,27 +103,33 @@ module RuoteAMQP
   # participant, and messages are marked as persistent by default (see
   # #RuoteAMQP)
   #
-  class Participant
+  class ParticipantProxy
 
     include Ruote::LocalParticipant
 
-    # Accepts an options hash with the following keys:
+    # The following parameters are used in the process definition.
     #
-    # * :reply_by_default => (bool) false by default
-    # * :default_queue => (string) nil by default
-    def initialize( options = {} )
+    # An options hash with the same keys to provide defaults is
+    # accepted at registration time (see above).
+    #
+    # * :queue => (string) The AMQP queue used by the remote participant.
+    #   nil by default.
+    # * :forget => (bool) Whether the flow should block until the remote
+    #   participant replies.
+    #   false by default
+    #
+    def initialize(options)
+
       RuoteAMQP.start!
 
       @options = {
-        :reply_by_default => false,
-        :default_queue => nil
-      }.merge( options )
-
-      @participant_maps = {}
-    end
-
-    def map_participant( name, queue )
-      @participant_maps[ name ] = queue
+        'queue' => nil,
+        'forget' => false,
+      }.merge(options.inject({}) { |h, (k, v)|
+        h[k.to_s] = v; h
+      })
+        #
+        # the inject is here to make sure that all options have String keys
     end
 
     # Process the workitem at hand. By default the workitem will be
@@ -140,47 +137,105 @@ module RuoteAMQP
     # workitem parameter. You can specify a +message+ workitem
     # parameter to have that sent instead of the workitem.
     #
-    # To force the participant to reply to the engine, set the
-    # +reply_anyway+ workitem parameter.
-    def consume( workitem )
-      if target_queue = determine_queue( workitem )
-
-        q = MQ.queue( target_queue, :durable => true )
-
-        # Message or workitem?
-        if message = ( workitem.fields['message'] || workitem.fields['params']['message'] )
-          q.publish( message, :persistent => RuoteAMQP.use_persistent_messages? )
-        else
-          q.publish( encode_workitem( workitem ), :persistent => RuoteAMQP.use_persistent_messages? )
-        end
+    def consume(workitem)
+
+      target_queue = determine_queue(workitem)
+
+      raise 'no queue specified (outbound delivery)' unless target_queue
+
+      q = MQ.queue(target_queue, :durable => true)
+      forget = determine_forget(workitem)
+
+      opts = {
+        :persistent => RuoteAMQP.use_persistent_messages?,
+        :content_type => 'application/json' }
+
+      if message = workitem.fields['message'] || workitem.params['message']
+
+        forget = true # sending a message implies 'forget' => true
+
+        q.publish(message, opts)
+
       else
-        raise "no queue in workitem params!"
-      end
 
-      if @options[:reply_by_default] || workitem.fields['params']['reply_anyway'] == true
-        reply_to_engine( workitem )
+        q.publish(encode_workitem(workitem), opts)
       end
+
+      reply_to_engine(workitem) if forget
     end
 
+    # (Stops the underlying queue subscription)
+    #
     def stop
+
       RuoteAMQP.stop!
     end
 
     def cancel(fei, flavour)
+      #
+      # TODO : sending a cancel item is not a bad idea, especially if the
+      #        job done over the amqp fence lasts...
+      #
+    end
+
+    # The current AMQP (0.6.7) has 1 queue per thread. If you let the default
+    # "one thread per participant consume call" kick in, you'll end up with
+    # 1 queue per consume call (and...)
+    #
+    # So, by returning true here, we force the queue to be always the same.
+    #
+    # Many thanks to https://github.com/weifeng365 for reporting this issue
+    # and suggesting the fix.
+    #
+    # TODO : should we have something to close queues when the engine / worker
+    #        shuts down ?
+    #        or is it already covered in the #stop ?
+    #
+    def do_not_thread
+
+      true
     end
 
     private
 
-    def determine_queue( workitem )
-      workitem.fields['params']['queue'] ||
-      @participant_maps[ workitem.participant_name ] ||
-      @options[:default_queue]
+    def determine_forget(workitem)
+
+      return workitem.params['forget'] if workitem.params.has_key?('forget')
+      return @options['forget'] if @options.has_key?('forget')
+      false
+    end
+
+    def determine_queue(workitem)
+
+      workitem.params['queue'] || @options['queue']
+    end
+
+    # Encodes the workitem as JSON. Makes sure to add to the field 'params'
+    # an entry named 'participant_options' which contains the options of
+    # this participant.
+    #
+    def encode_workitem(wi)
+
+      wi.params['participant_options'] = @options
+
+      Rufus::Json.encode(wi.to_h)
     end
+  end
+
+  #
+  # Kept for backward compatibility.
+  #
+  # You should use RuoteAMQP::ParticipantProxy.
+  #
+  class Participant < ParticipantProxy
 
-    # Encode (and extend) the workitem as JSON
-    def encode_workitem( wi )
-      wi.fields['params']['reply_queue'] = WorkitemListener.queue
-      wi.to_h.to_json
+    def initialize(options)
+      puts '=' * 80
+      puts "RuoteAMQP::Participant will be deprecated soon (2.1.12)"
+      puts "please use RuoteAMQP::ParticipantProxy instead"
+      puts '=' * 80
+      super
     end
   end
 end
+