ruote 2.1.9 → 2.1.10

data/CHANGELOG.txt

@@ -2,6 +2,38 @@
 = ruote - CHANGELOG.txt
 
 
+== ruote - 2.1.10    released 2010/06/15
+
+- storage#copy_to(other_storage) implemented
+- #launch moved from Engine to ReceiverMixin
+- participants without initialize(opts) are now allowed
+- engine.wait_for(:inactive)
+- engine.wait_for(*interests) unlocked
+- engine.wait_for(:empty)
+- fixed issue with participant 'x' and :on_error. Thanks Oleg.
+- receiver : reply and reply_to_engine : from aliases to wrappers
+- Ruote::StorageParticipant more flexibility for method args
+- Ruote::FlowExpressionId .extract_h and .extract
+- dropped fs_history (storage_history is better)
+- parser to_xml _if 'x == b' --> <if test="x == b">
+- workitem.sid shortcut for workitem.fei.to_storage_id
+- workitem.wfid shorcut for workitem.fei.wfid
+- new error_handler service
+- Receiver.new(x), x can be worker, engine, context or storage
+- Participant#on_reply(workitem) manipulating workitems when they come back
+- set '${v:customers.0.name}' => 'x' now OK, was limited to fields. Thanks Oleg
+- LocalParticipant#put(fei, hash) #get(fei, key) for stashing info
+- LocalParticipant#re_dispatch(wi, opts)
+- bug in HashStorage, apply over apply didn't raise a persist error. Fixed.
+- keeping track of workitem fields as they were right before a participant error
+- Workitem.error holds the error when on_error. Thanks Oleg.
+- Workitem#error and Workitem#timed_out shortcuts
+- participant :on_error => 'x' broken. Fixed. Thanks Oleg.
+- Engine#workitem(fei) for advanced users
+- LocalParticpant : added a reject(workitem) method
+- participant exp : dispatched = true set right after dispatch
+
+
 == ruote - 2.1.9    released 2010/03/22
 
 - made participant.cancel occur asynchronously (as should be)

data/CREDITS.txt

@@ -16,6 +16,7 @@ Torsten Schoenebaum - http://github.com/tosch
 Contributors
 ------------
 
+Oleg Pudeyev - http://github.com/p
 Brett Anthoine - http://github.com/anb
 Matt Nichols - http://github.com/mattnichols
 Nicholas Faiz - http://github.com/biv
@@ -38,6 +39,8 @@ Richard Jennings
 Feedback
 --------
 
+Oleg (foenixx) - many suggestions and bug reports
+Avishai Shalom - discussion and ideas about participant/worker locality
 Gonzalo Suarez - many help
 Francisco Kiko - many help
 David Goldhirsch - EM participant block

data/Rakefile

@@ -23,8 +23,8 @@ ruote is an open source ruby workflow engine.
     gem.rubyforge_project = 'ruote'
     gem.test_file = 'test/test.rb'
 
-    gem.add_dependency 'rufus-json', '>= 0.2.0'
-    gem.add_dependency 'rufus-cloche', '>= 0.1.16'
+    gem.add_dependency 'rufus-json', '>= 0.2.2'
+    gem.add_dependency 'rufus-cloche', '>= 0.1.17'
     gem.add_dependency 'rufus-dollar'
     gem.add_dependency 'rufus-lru'
     gem.add_dependency 'rufus-mnemo', '>= 1.1.0'
@@ -48,7 +48,7 @@ end
 begin
   require 'yard'
   YARD::Rake::YardocTask.new do |doc|
-    doc.options = [ '-o', 'ruote_rdoc', '--title', "ruote #{Ruote::VERSION}" ]
+    doc.options = [ '-o', 'rdoc', '--title', "ruote #{Ruote::VERSION}" ]
   end
 rescue LoadError
   task :yard do
@@ -65,7 +65,7 @@ desc 'Upload the documentation to rubyforge'
 task :upload_rdoc => :yard do
   sh %{
     rsync -azv -e ssh \
-      ruote_rdoc \
+      rdoc \
       jmettraux@rubyforge.org:/var/www/gforge-projects/ruote/
   }
 end

data/TODO.txt

@@ -193,6 +193,27 @@
 [o] part = engine.register_participant :alpha, StorageParticipant should work...
 [o] concurrence :merge_type => 'stack'
 [o] CompositeStorage.new('msgs' => AmqpStorage.new(''), ...)
+[x] let the storage participant leverage Ruote::FlowExpressionId.from_id(s)
+[o] Andrew's technique http://groups.google.com/group/openwferu-users/browse_thread/thread/c2aa4b53d1664d45/8523a1a5ee98fd71
+[o] Avishai : LocalParticipant : repost dispatch message
+[o] Rdoc Ruote::Engine.register_participant -> passing a block to a participant
+    and perhaps also on
+    http://ruote.rubyforge.org/implementing_participants.html (Avish)
+[o] wrap workitem in process error ? for on_error consumption (thanks Oleg)
+    doing workitem.fields['__error__'] = [ fei, time, error_message ]
+[o] HashStorage should emit 'init persist fail' messages as well !
+[o] Oleg's idea about participant on_reply
+    http://groups.google.com/group/openwferu-users/browse_thread/thread/2e6a95708c10847b
+    on_reply should be done in the receive action, not in Receiver
+    thus, in the ParticipantExpression
+[x] exp : step (jump to cursor tag ?) : there is already the jump expression
+[x] auto-participant re-apply
+[o] receiver should be OK with a storage or a context
+[x] Avishai : Worker : hook for rejecting the dispatch message
+[o] receiver / local participant : reply/forward/proceed/... mess : fix
+[o] storage participant : accept string for fei
+[o] => Ruote::FlowExpressionId.extract(x)
+[o] fei : place engine id in fei.to_storage_id (and back)
 
 [ ] exp : exp (restricted form of eval ?)
 [ ] exp : case (is it necessary ?)
@@ -200,19 +221,15 @@
 [ ] exp : filter-definition
 [x] exp : lose ?
 [x] exp : parameter
-[ ] exp : log
+[ ] exp : log : or could it be a participant ?
 
 [ ] exp : defined (not really necessary)
 [ ] exp : quote (not really necessary)
 [ ] exp : field / attribute (not really necessary)
 [ ] exp : variable (not really necessary)
 
-[ ] exp : step (jump to cursor tag ?)
-
 [ ] conditional : rprefix ! ${r:x} is perhaps sufficient
 
-[ ] auto-participant re-apply
-
 [ ] define without name (__result__)
 
 [ ] pooltool.ru
@@ -234,6 +251,9 @@
 
 [ ] pause engine
 [ ] pause process instance
+    |
+    would it mean something like placing a paused list in the storage
+    and fetching it all the time ?
 
 [ ] file/fs_listener [example] ?
 
@@ -293,24 +313,25 @@
 
 [ ] empty iterator or concurrent-iterator, log ? crash ? empty while...
 [ ] at expression ?
-[ ] listen to participants/errors/tags {in|out}
 
 [ ] remove abort_on_exception=true
 
 [ ] shell ? irb ? Shell.new(storage)
 [ ] focus on fulldup or json.dup (via fulldup ?)
 
-[ ] implement pause engine
-[ ] implement pause process
+[ ] listen to participants/errors/tags {in|out}
+
+[x] engine.on_error = 'participant_name' // 'subprocess_name'
+    done at : http://github.com/jmettraux/ruote/commit/50292d954ff877f1f6615022216f346a7001b483
+    `--> reverting that for now, too dangerous
 
-[ ] engine.on_error = 'participant_name' // 'subprocess_name'
+[ ]  should __error__ contain the tree ?
+[ ] engine.on_cancel = 'participant_name' // 'subprocess_name'
 
 [ ] "business days" plugin
 
 [ ] issue with ruote-kit and inpa participants...
 
-[ ] let the storage participant leverage Ruote::FlowExpressionId.from_id(s)
-
 [ ] participant :ref => '${f:nada}', :or => 'xyz'
     (look at OpenWFE manual, this feature already existed in there)
     http://www.openwfe.org/manual/ch06s02.html#expression_participant
@@ -323,3 +344,26 @@
 
 [ ] find better solution than "get all schedules"
 
+[ ] worker : minuteman, make it cron triggerable
+    trap SIGUSR1 or USR2
+    maybe it's expensive to fire a [worker] process each minute
+    have to write the $$ (pid) somewhere for cron to pick it up
+
+[ ] detach / attach segments of processes
+[ ] clone process ? (could be used by {de|at}tach)
+
+[ ] dollar.rb ${timestamp} ?
+
+[ ] toto :task => 'maw the lawn', :within => '3d'
+
+[ ] solve the ps#root_expression_for(fei) dilemma
+
+[ ] engine.noisy = true shortcut
+
+[ ] re_apply_stalled
+    http://groups.google.com/group/openwferu-users/browse_thread/thread/ff29f26d6b5fd135
+
+[ ] wait_for(:inactive) blocks until worker is inactive
+
+[ ] storage0.copy_to(storage1) / migrate_to as requested by Matt Nichols
+

data/examples/barley.rb

@@ -91,7 +91,8 @@ require 'cgi'
 require 'sinatra'
 require 'haml'
 
-use_in_file_templates!
+#use_in_file_templates!  # sinatra 0.9.x
+enable :inline_templates  # sinatra 1.0
 
 def h (s)
   Rack::Utils.escape_html(s)

data/examples/flickr_report.rb

@@ -2,14 +2,14 @@
 $:.unshift('lib')
 
 require 'rubygems'
-require 'ruote/engine' # sudo gem install ruote
+require 'ruote' # sudo gem install ruote
 require 'atom/feed' # sudo gem install atom-tools
 require 'prawn' # sudo gem install prawn
 
 #
 # starting a transient engine (no need to make it persistent)
 
-engine = Ruote::Engine.new(:definition_in_launchitem_allowed => true)
+engine = Ruote::Engine.new(Ruote::Worker.new(Ruote::HashStorage.new()))
 
 #
 # a process that fetches the latest pictures from flickr.com and submits
@@ -47,7 +47,7 @@ engine.register_participant :get_pictures do |workitem|
 
   workitem.fields['pictures'] = feed.entries.inject([]) do |a, entry|
     a << [
-      entry.title,
+      entry.title.to_s,
       entry.authors.first.name,
       entry.links.last.href
     ]
@@ -94,10 +94,9 @@ end
 #
 # launching the process, requesting pictures tagged 'cat' and 'fish'
 
-li = Ruote::Launchitem.new(pdef)
-li.fields['tags'] = [ 'cat', 'fish' ]
+initial_workitem_fields = { 'tags' => [ 'cat', 'fish' ] }
 
-fei = engine.launch(li)
+fei = engine.launch(pdef, initial_workitem_fields)
 
 #
 # workflow engines are asynchronous beasts, have to wait for them

data/examples/web_first_page.rb

@@ -1,4 +1,15 @@
 
+#
+# This is not a runnable example, it's just a ruby source file that contains
+# the samples found at
+#
+#   http://ruote.rubyforge.org/index.html
+#
+# NOTE : this example need some rework. The use of block participants will
+# probably be avoided
+#
+
+require 'rubygems'
 require 'ruote'
 
 pdef = Ruote.process_definition :name => 'work' do

data/lib/ruote/context.rb

@@ -40,29 +40,53 @@ module Ruote
     attr_accessor :worker
     attr_accessor :engine
 
-    def initialize (storage, worker_or_engine)
+    def initialize (storage, worker=nil)
 
       @storage = storage
+      @storage.context = self
 
-      @worker, @engine = if worker_or_engine.kind_of?(Ruote::Engine)
-        [ worker_or_engine.worker, worker_or_engine ]
-      else
-        [ worker_or_engine, nil ]
-      end
+      @engine = nil
+      @worker = worker
+
+      @services = {}
 
       initialize_services
     end
 
+    # A trick : in order to avoid
+    #
+    #   @context = o.respond_to?(:context) ? o.context : o
+    #   # or
+    #   #@context = o.is_a?(Ruote::Context) ? o : o.context
+    #
+    # simply letting a context say its context is itself.
+    #
+    def context
+
+      self
+    end
+
+    # Returns the engine_id (as set in the configuration under the key
+    # "engine_id"), or, by default, "engine".
+    #
     def engine_id
 
       get_conf['engine_id'] || 'engine'
     end
 
+    # Used for things like
+    #
+    #   if @context['ruby_eval_allowed']
+    #     # ...
+    #   end
+    #
     def [] (key)
 
       SERVICE_PREFIX.match(key) ? @services[key] : get_conf[key]
     end
 
+    # Mostly used by engine#configure
+    #
     def []= (key, value)
 
       raise(
@@ -105,15 +129,14 @@ module Ruote
       service
     end
 
+    # Takes care of shutting down every service registered in this context.
+    #
     def shutdown
 
       @storage.shutdown if @storage.respond_to?(:shutdown)
       @worker.shutdown if @worker
 
-      @services.values.each do |s|
-
-        s.shutdown if s.respond_to?(:shutdown)
-      end
+      @services.values.each { |s| s.shutdown if s.respond_to?(:shutdown) }
     end
 
     protected
@@ -125,8 +148,6 @@ module Ruote
 
     def initialize_services
 
-      @services = {}
-
       default_conf.merge(get_conf).each do |key, value|
 
         next unless SERVICE_PREFIX.match(key)
@@ -152,7 +173,9 @@ module Ruote
         's_dispatch_pool' => [
           'ruote/part/dispatch_pool', 'Ruote::DispatchPool' ],
         's_logger' => [
-          'ruote/log/wait_logger', 'Ruote::WaitLogger' ] }
+          'ruote/log/wait_logger', 'Ruote::WaitLogger' ],
+        's_error_handler' => [
+          'ruote/error_handler', 'Ruote::ErrorHandler' ] }
     end
   end
 end

data/lib/ruote/engine.rb

@@ -29,69 +29,63 @@ require 'ruote/receiver/base'
 
 module Ruote
 
+  #
+  # This class holds the 'engine' name, perhaps 'dashboard' would have been
+  # a better name. Anyway, the methods here allow to launch processes
+  # and to query about their status. There are also methods for fixing
+  # issues with stalled processes or processes stuck in errors.
+  #
+  # NOTE : the methods #launch and #reply are implemented in
+  # Ruote::ReceiverMixin
+  #
   class Engine
 
     include ReceiverMixin
 
-    attr_reader :storage
-    attr_reader :worker
     attr_reader :context
     attr_reader :variables
 
     def initialize (worker_or_storage, run=true)
 
-      if worker_or_storage.respond_to?(:storage)
+      @context = worker_or_storage.context
+      @context.engine = self
 
-        @worker = worker_or_storage
-        @storage = @worker.storage
-        @context = @worker.context
-        @context.engine = self
-      else
+      @variables = EngineVariables.new(@context.storage)
 
-        @worker = nil
-        @storage = worker_or_storage
-        @context = Ruote::Context.new(@storage, self)
-      end
-
-      @variables = EngineVariables.new(@storage)
-
-      @worker.run_in_thread if @worker && run
+      @context.worker.run_in_thread if @context.worker && run
+        # launch the worker if there is one
     end
 
-    def launch (process_definition, fields={}, variables={})
+    def storage
 
-      wfid = @context.wfidgen.generate
+      @context.storage
+    end
 
-      @storage.put_msg(
-        'launch',
-        'wfid' => wfid,
-        'tree' => @context.parser.parse(process_definition),
-        'workitem' => { 'fields' => fields },
-        'variables' => variables)
+    def worker
 
-      wfid
+      @context.worker
     end
 
     def cancel_process (wfid)
 
-      @storage.put_msg('cancel_process', 'wfid' => wfid)
+      @context.storage.put_msg('cancel_process', 'wfid' => wfid)
     end
 
     def kill_process (wfid)
 
-      @storage.put_msg('kill_process', 'wfid' => wfid)
+      @context.storage.put_msg('kill_process', 'wfid' => wfid)
     end
 
     def cancel_expression (fei)
 
       fei = fei.to_h if fei.respond_to?(:to_h)
-      @storage.put_msg('cancel', 'fei' => fei)
+      @context.storage.put_msg('cancel', 'fei' => fei)
     end
 
     def kill_expression (fei)
 
       fei = fei.to_h if fei.respond_to?(:to_h)
-      @storage.put_msg('cancel', 'fei' => fei, 'flavour' => 'kill')
+      @context.storage.put_msg('cancel', 'fei' => fei, 'flavour' => 'kill')
     end
 
     # Replays at a given error (hopefully you fixed the cause of the error
@@ -113,9 +107,9 @@ module Ruote
         exp.unpersist_or_raise if exp
       end
 
-      @storage.delete(err.to_h) # remove error
+      @context.storage.delete(err.to_h) # remove error
 
-      @storage.put_msg(action, msg) # trigger replay
+      @context.storage.put_msg(action, msg) # trigger replay
     end
 
     # Re-applies an expression (given via its FlowExpressionId).
@@ -148,7 +142,7 @@ module Ruote
     #
     def process (wfid)
 
-      exps = @storage.get_many('expressions', /!#{wfid}$/)
+      exps = @context.storage.get_many('expressions', /!#{wfid}$/)
       errs = self.errors( wfid )
 
       return nil if exps.empty? && errs.empty?
@@ -162,7 +156,7 @@ module Ruote
     #
     def processes
 
-      exps = @storage.get_many('expressions')
+      exps = @context.storage.get_many('expressions')
       errs = self.errors
 
       by_wfid = {}
@@ -181,10 +175,13 @@ module Ruote
     #
     def errors( wfid = nil )
       wfid.nil? ?
-        @storage.get_many('errors') :
-        @storage.get_many('errors', /!#{wfid}$/)
+        @context.storage.get_many('errors') :
+        @context.storage.get_many('errors', /!#{wfid}$/)
     end
 
+    # Shuts down the engine, mostly passes the shutdown message to the other
+    # services and hope they'll shut down properly.
+    #
     def shutdown
 
       @context.shutdown
@@ -207,7 +204,16 @@ module Ruote
     #     # will make the current thread block until 5 messages have been
     #     # processed on the workqueue...
     #
-    def wait_for (item)
+    #   engine.wait_for(:empty)
+    #     # will return as soon as the engine/storage is empty, ie as soon
+    #     # as there are no more processes running in the engine (no more
+    #     # expressions placed in the storage)
+    #
+    # It's OK to wait for multiple wfids :
+    #
+    #   engine.wait_for('20100612-bezerijozo', '20100612-yakisoba')
+    #
+    def wait_for (*items)
 
       logger = @context['s_logger']
 
@@ -215,7 +221,7 @@ module Ruote
         "can't wait_for, there is no logger that responds to that call"
       ) unless logger.respond_to?(:wait_for)
 
-      logger.wait_for(item)
+      logger.wait_for(items)
     end
 
     # Loads and parses the process definition at the given path.
@@ -255,37 +261,42 @@ module Ruote
     #   engine.register_participant /^moon-.+/, MyParticipant.new('Saturn-V')
     #
     #
-    # == passing a block to a participant
+    # == 'stateless' participants are preferred over 'stateful' ones
     #
-    # Usually only the BlockParticipant cares about being passed a block :
+    # Ruote 2.1 is OK with 1 storage and 1+ workers. The workers may be
+    # in other ruby runtimes. This implies that if you have registered a
+    # participant instance (instead of passing its classname and options),
+    # that participant will only run in the worker 'embedded' in the engine
+    # where it was registered... Let me rephrase it, participants instantiated
+    # at registration time (and that includes block participants) only runs
+    # in one worker, always the same.
     #
-    #   engine.register_participant 'compute_sum' do |workitem|
-    #     workitem.fields['kilroy'] = 'was here'
-    #   end
+    # 'stateless' participants, instantiated at each dispatch, are preferred.
+    # Any worker can handle them.
     #
-    # But it's OK to pass a block to a custom participant :
+    # Block participants are still fine for demos (where the worker is included
+    # in the engine (see all the quickstarts). And small engines with 1 worker
+    # are not that bad, not everybody is building huge systems).
     #
-    #   require 'ruote/part/local_participant'
+    # Here is a 'stateless' participant example :
     #
-    #   class MyParticipant
-    #     include Ruote::LocalParticipant
+    #   class MyStatelessParticipant
     #     def initialize (opts)
-    #       @name = opts[:name]
-    #       @block = opts[:block]
+    #       @opts = opts
     #     end
     #     def consume (workitem)
-    #       workitem.fields['prestamp'] = Time.now
-    #       workitem.fields['author'] = @name
-    #       @block.call(workitem)
-    #       reply_to_engine(workitem)
+    #       workitem.fields['rocket_name'] = @opts['name']
+    #       send_to_the_moon(workitem)
+    #     end
+    #     def cancel (fei, flavour)
+    #       # do nothing
     #     end
     #   end
     #
-    #   engine.register_participant 'al', MyParticipant, :name => 'toto' do |wi|
-    #     wi.fields['nada'] = surf
-    #   end
+    #   engine.register_participant 'moon', MyStatelessParticipant, 'name' => 'saturn5'
     #
-    # The block is available under the :block option.
+    # Remember that the options (the hash that follows the class name), must be
+    # serialisable via JSON.
     #
     def register_participant (regex, participant=nil, opts={}, &block)
 
@@ -341,6 +352,27 @@ module Ruote
 
       @context[config_key] = value
     end
+
+    # A convenience methods for advanced users (like Oleg).
+    #
+    # Given a fei (flow expression id), fetches the workitem as stored in
+    # the expression with that fei.
+    # This is the "applied workitem", if the workitem is currently handed to
+    # a participant, this method will return the workitem as applied, not
+    # the workitem as saved by the participant/user in whatever worklist it
+    # uses. If you need that workitem, do the vanilla thing and ask it to
+    # the [storage] participant or its worklist.
+    #
+    # The fei might be a string fei (result of fei.to_storage_id), a
+    # FlowExpressionId instance or a hash.
+    #
+    def workitem (fei)
+
+      fexp = Ruote::Exp::FlowExpression.fetch(
+        @context, Ruote::FlowExpressionId.extract_h(fei))
+
+      Ruote::Workitem.new(fexp.h.applied_workitem)
+    end
   end
 
   #