libis-workflow 2.0.beta.13 → 2.0.beta.14

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ee7675981468e33b70d8f03bb43ade771225a34b
-  data.tar.gz: 2770f1c82005d427bcc29c8aa08d84f395f971f8
+  metadata.gz: cf536dca7ee73ba80bf51318292bf07e006d10a7
+  data.tar.gz: 3a5bfd9e58632e117a86de20ea949c469b2f1801
 SHA512:
-  metadata.gz: 557aa30bcff78d439b02008c86c5a1a4f43264fb23d1405012e0f3430006581c5ee2ce092aa0421fa89357b9ff7576b51cca6788f0fc15d22ca5e99be8f3ab75
-  data.tar.gz: dd1c1fabe78d2ca9a7ad0f89ce6ae0975a478e1383a904119fa6a55fdc2111340f3aa2fbdc9deb56faf43a8d52ac88f9ec18bc61ae6925e17e7f49e7371cb9ee
+  metadata.gz: ef7cf24225f7f0e56615afd279ab54bc33ee484aa0bec30956b1db90179da92c30e4f856c548488da99ce26534cc20b2c7a3ea9826453dc621f090a212f216f2
+  data.tar.gz: c01a212ea18e595a25c79c1e55d49d94e8df43f431e8b5f8f79e1756aa8513beb0fe56f985853e00604588d137c6010dadc939358ee05e68a90fdb478dcacae0

data/README.md

@@ -49,7 +49,7 @@ name has to be provided to the job configuration so that the job can instantiate
 will be able to execute the tasks in proper order on all the WorkItems supplied/collected. Each task can be implemented 
 with code to run or simply contain a list of child tasks. 
 
-One tasks is predefined:
+One task is predefined:
 ::Libis::Workflow::Tasks::Analyzer - analyzes the workflow run and summarizes the results. It is always included as the
 last task by the workflow unless you supply a closing task called 'Analyzer' yourself.
 
@@ -103,15 +103,17 @@ is a Hash with:
   
 The ::Libis::Workflow::Task base class allready defines the following parameters:
 * quiet: Prevent generating log output. Default: false
-* abort_on_error: Stop all tasks when an error occurs. Default: false
-* always_run: Run this task, even if the item failed a previous task. Default: false
-* subitems: Do not process the given item, but only the subitems. Default: false
 * recursive: Run the task on all subitems recursively. Default: false
+* retry_count: Number of times to retry the task. Default: 0
+* retry_interval: Number of seconds to wait between retries. Default: 10
 
-If 'class' is not present, the default '::Libis::Workflow::Task' with the given name will be instantiated, which simply
-iterates over the child items of the given work item and performs each sub-task on each of the child items. If a 'class'
-value is given, an instance of that class will be created and the task will be handed the work item to process on. See 
-the chapter on 'Tasks' below for more information on tasks.
+If 'class' is not present, the default '::Libis::Workflow::TaskGroup' with the given name will be instantiated, which 
+performs each sub-task on the item. If the task is configured to be recursive, it will iterate over the child items and
+perform each sub-task on each of the child items. If a 'class' value is given, an instance of that class will be created 
+and the task will be handed the work item to process on. See the chapter on 'Tasks' below for more information on tasks.
+
+Note that a task with custom processing will not execute sub-tasks. If you configured a processing task with subtasks
+an exception will be thrown when trying to execute the job.
 
 ##### Input variable definition
 
@@ -276,17 +278,17 @@ perform on each work item:
 As seen above, the task should define a method called process_item that takes one argument. The argument will be a 
 reference to the work item that it needs to perform an action on. The task has several option to progress after 
 performing its actions:
-* return. This is considered a normal and successful operation result. If the item was replaced by a new item or if the 
- item tree has changed, it is recommended to return the new item. After a successful return the item's status will be 
- set to 'done' for the given task.
+* return. This is considered a normal and successful operation result. After a successful return the item's status will 
+  be set to 'done' for the given task.
 * raise a ::Libis::WorkflowError. Indicates that something went wrong during the processing of the item. The item's 
- status will be set to failed for the given task and the exception message will be printed in the error log. Processing 
- will continue with the next item. This action is recommended for temporary or recoverable errors.
+  status will be set to failed for the given task and the exception message will be printed in the error log. Processing 
+  will continue with the next item. This action is recommended for temporary or recoverable errors. The parent item will
+  be flagged as 'failed' if any of the child items failed.
 * raise a ::Libis::WorkflowAbort. A severe and fatal error has occured. Processing will abort immediately and the 
- failure status will be escalated to all items up the item hierarchy. Due to the escalating behaviour, no message is 
- printed in the error log automatically, so it is up to the task to an appropriate log the error itself.
+  failure status will be escalated to all items up the item hierarchy. Due to the escalating behaviour, no message is 
+  printed in the error log automatically, so it is up to the task to an appropriate log the error itself.
 * raise any other Exception. Should be avoided, but if it happens nevertheless, it will cause the item to fail for the 
- given task and the exception message to be logged in the error log. It will attempt to process the other items.
+  given task and the exception message to be logged in the error log. It will not attempt to process the other items.
 
 ### Controlling behavior with parameters
 
@@ -294,43 +296,32 @@ You have some options to control how the task will behave in special cases. Thes
 the task, which can be set (and fixed with the 'frozen' option) on the task, but can be configured at run-time with the 
 help of workflow input parameters and run options.
 
-#### Performing an action on the work item and all child items recursively
-
-With the 'recursive' parameter set to true, your task's process_item method will be called for the work item and then 
-once for each child and each child's children recursively.
-  
-#### Performing an action only on the child items, skipping the work item itself
-
-The parameter 'subitems' decides if the item handed over to the task will be processed or if it will process only it's 
-child items. This will only work once, not recursively, but by organizing tasks hierarchically with 'subitems' set to 
-true, it is possible to make sure that only items on a certain hierarchy level are performed. Recommended for workflows 
-where a well-known and fixed hierarchical structure has to be processed selectively.
-
-Alternatively the 'recursive' option can be set and the 'process_item' method could check for certain item types, 
-properties or hierarchy levels to decide to perform the operation. This approach is more flexible but harder to 
-understand unless well documented.
-
 #### Preventing any logging output from the task
 
 Logging output can be blocked on a task-by-task basis by setting the 'quiet' parameter to true. Probably not usefull 
 except for the Analyzer task where the parameter is fixed to true. Logging output would otherwise intervene with the 
 log summary processing performed by the task.
 
-#### Aborting the task whenever any item fails
- 
-When a task is so critical in the workflow process that any failure renders further processing useless, the parameter 
-'abort_on_error' can be turned on. Raising a ::Libis::WorkflowAbort exception would perform the same thing, but the 
-parameter makes the configuration more flexible.
+#### Performing an action on the work item and all child items recursively
 
-#### Always running a task, even on items that failed in an earlier stage in the workflow
+With the 'recursive' parameter set to true, your task's process_item method will be called for the work item and then 
+once for each child and each child's children recursively.
 
-The opposite of aborting on error, the parameter 'always_run' can be set to true to force a task to process the items 
-even if the item has a failed status from a previous task. Note that this will cause the item's status to no longer be 
-failing until a task fails on the item again. The old failed status will be tracable in the status history.
+Note: you should not make both parent and child tasks recursive as this will cause the subitems to be processed 
+multiple times. If you make the parent task recursive, all tasks and sub-tasks will be performed on each item in the
+tree. Making the child tasks recursive makes the parent task only perform on the top item and then performs each 
+sub-task one-by-one for the whole item tree. The last option is the most efficient.
+  
+Attention should be paid for the 
+  
+#### Retrying if task failed
 
-The parameter only configures the task on which it is set. If the task is a subtask it will only be forced to run if 
-any of it's previous sibling tasks failed. In order to force the run, even if the item failed in another branch of the 
-task hiearchy, the parent tasks should have their 'always_run' parameter also set to true.
+The parameters 'retry_count' and 'retry_interval' control the task's behaviour if a task has to wait for a result for an
+asynchonous job. A task could be waiting for a result from the other job which will be indicated by a 'ASYNC_WAIT'
+status. Alternatively the task may know that the job is halted and waiting for user interaction, indicated with the
+'ASYNC_HALT' status. Only when the status is 'ASYNC_WAIT', the task will retry its process. By default the 'retry_count'
+is 0, which causes the task not to retry. Before retrying the task will pause for the number of seconds given in the
+parameter 'retry_interval', which is 30 by default.
 
 ### Pre- and postprocessing
 
@@ -338,6 +329,13 @@ The default implementation of 'process' is to call 'pre_process' and then call '
 followed by calling 'post_process'. The methods 'pre_process' and 'post_process' are no-operation methods by default, 
 but can be overwritten if needed.
 
+The 'pre_process' is intended to re-initialize the task before processing a new item. It can also be used to force the
+task to skip processing the items altogether by calling the 'skip_processing_item' method or to prevent a recursive
+task from traveling further down the item tree by calling the 'stop_processing_subitems' method. The temporary locks
+behave as reset-on-read switches and are only active for the processing of the current item.
+
+The 'post_process' method can be used to update any object after the item processing.
+
 ### Convenience functions
 
 #### get_root_item()

data/lib/libis/workflow/action.rb

@@ -0,0 +1,24 @@
+module Libis
+  module Workflow
+    module Action
+
+      RUN = 0
+      CONTINUE = 1
+      RETRY = 2
+      ALWAYS_RUN = 3
+      FAILED = 4
+
+      def next_success_action(action)
+        case action
+          when :run, :continue, :retry, :always_run
+            :always_run
+          when :failed
+            :failed
+          else
+            :failed
+        end
+      end
+
+    end
+  end
+end

data/lib/libis/workflow/base/logging.rb

@@ -0,0 +1,76 @@
+module Libis
+  module Workflow
+    module Base
+      module Logging
+
+        # Helper function for the Tasks to add a log entry to the log_history.
+        #
+        # The supplied message structure is expected to contain the following fields:
+        # - :severity : ::Logger::Severity value
+        # - :id : optional message id
+        # - :text : message text
+        # - :task : list of tasks names (task hierarchy) that submits the message
+        #
+        # @param [Hash] message
+        def add_log(message = {})
+          msg = message_struct(message)
+          add_log_entry(msg)
+          self.save
+        end
+
+        def <=(message = {})
+          self.add_log(message)
+        end
+
+        # Add a structured message to the log history. The message text can be submitted as an integer or text. If an
+        # integer is submitted, it will be used to look up the text in the MessageRegistry. The message text will be
+        # passed to the % operator with the args parameter. If that failes (e.g. because the format string is not correct)
+        # the args value is appended to the message.
+        #
+        # @param [Symbol] severity
+        # @param [Hash] msg should contain message text as :id or :text and the hierarchical name of the task as :task
+        # @param [Array] args string format values
+        def log_message(severity, msg, *args)
+          # Prepare info from msg struct for use with string substitution
+          message_id, message_text = if msg[:id]
+                                       [msg[:id], MessageRegistry.instance.get_message(msg[:id])]
+                                     elsif msg[:text]
+                                       [0, msg[:text]]
+                                     else
+                                       [0, '']
+                                     end
+          task = msg[:task] || '*UNKNOWN*'
+          message_text = (message_text % args rescue "#{message_text} - #{args}")
+
+          self.add_log severity: severity, id: message_id.to_i, text: message_text, task: task
+          name = ''
+          begin
+            name = self.to_s
+            name = self.name
+            name = self.namepath
+          rescue
+            # do nothing
+          end
+          Config.logger.add(severity, message_text, ('%s - %s ' % [task, name]))
+        end
+
+        protected
+
+        SEV_LABEL = %w(DEBUG INFO WARN ERROR FATAL ANY) unless const_defined? :SEV_LABEL
+
+        # create and return a proper message structure
+        # @param [Hash] opts
+        def message_struct(opts = {})
+          opts.reverse_merge!(severity: ::Logger::INFO, code: nil, text: '')
+          {
+              severity: SEV_LABEL[opts[:severity]],
+              task: opts[:task],
+              code: opts[:code],
+              message: opts[:text]
+          }.cleanup
+        end
+
+      end
+    end
+  end
+end

data/lib/libis/workflow/base/run.rb

@@ -23,7 +23,7 @@ module Libis
       module Run
         include ::Libis::Workflow::Base::WorkItem
 
-        attr_accessor :tasks
+        attr_accessor :tasks, :action
 
         def work_dir
           # noinspection RubyResolve
@@ -49,7 +49,16 @@ module Libis
         end
 
         # Execute the workflow.
-        def run
+        #
+        # The action parameter defines how the execution of the tasks will behave:
+        #  - With the default :run action each task will be executed regardsless how the task performed on the item
+        #    previously.
+        #  - When using the :retry action a task will not perform on an item if it was successful the last time. This
+        #    allows you to retry a run when an temporary error (e.g. asynchronous wait or halt) occured.
+        #
+        # @param [Symbol] action the type of action to take during this run. :run or :retry
+        def run(action = :run)
+          self.action = action
 
           self.start_date = Time.now
 
@@ -58,16 +67,11 @@ module Libis
           self.tasks = workflow.tasks(self)
           configure_tasks self.options
 
-          self.status = :STARTED
 
           self.tasks.each do |task|
-            # note: do not return as we want to give any remaining task in the queue the oportunity to run
-            next if self.failed? and not task.parameter(:always_run)
             task.run self
           end
 
-          self.status = :DONE unless self.failed?
-
         end
 
         protected

data/lib/libis/workflow/base/work_item.rb

@@ -4,6 +4,8 @@ require 'backports/rails/hash'
 require 'libis/tools/extend/hash'
 
 require 'libis/workflow/config'
+require 'libis/workflow/status'
+require_relative 'logging'
 
 module Libis
   module Workflow
@@ -66,12 +68,11 @@ module Libis
       #   self.status_log << { c_at: ::Time.now, tasklist: tasklist, text: message }.cleanup
       # end
       #
-      # def status_label(status_entry)
-      #   "#{status_entry[:tasklist].last rescue nil}#{status_entry[:text] rescue nil}"
-      # end
       #
       module WorkItem
         include Enumerable
+        include Libis::Workflow::Status
+        include Libis::Workflow::Base::Logging
 
         # String representation of the identity of the work item.
         #
@@ -97,7 +98,9 @@ module Libis
           self.names.join('/');
         end
 
-        # File name save version of the to_s output. The output should be safe to use as a file name to store work item
+        # File name safe version of the to_s output.
+        #
+        # The output should be safe to use as a file name to store work item
         # data. Typical use is when extra file items are created by a task and need to be stored on disk. The default
         # implementation URL-encodes (%xx) all characters except alphanumeric, '.' and '-'.
         #
@@ -106,51 +109,6 @@ module Libis
           self.to_s.gsub(/[^\w.-]/) { |s| '%%%02x' % s.ord }
         end
 
-        # Gets the current status of the object.
-        #
-        # @return [Symbol] status code
-        def status
-          s = self.status_log.last
-          label = status_label(s)
-          label.empty? ? :NOT_STARTED : label.to_sym
-        end
-
-        # Changes the status of the object. The status changed is logged in the status_log with the current timestamp.
-        #
-        # @param [Symbol] s
-        def status=(s, tasklist = nil)
-          s, tasklist = s if s.is_a? Array
-          s = s.to_sym
-          add_status_log(s, tasklist)
-          self.save
-        end
-
-        # Check ingest status of the object. The status is checked to see if it ends in 'Failed'.
-        #
-        # @return [Boolean] true if the object failed, false otherwise
-        def failed?
-          self.status.to_s =~ /Failed$/i ? true : false
-        end
-
-        # Helper function for the Tasks to add a log entry to the log_history.
-        #
-        # The supplied message structure is expected to contain the following fields:
-        # - :severity : ::Logger::Severity value
-        # - :id : optional message id
-        # - :text : message text
-        # - :task : list of tasks names (task hierarchy) that submits the message
-        #
-        # @param [Hash] message
-        def add_log(message = {})
-          msg = message_struct(message)
-          add_log_entry(msg)
-          self.save
-        end
-
-        def <=(message = {})
-          ; self.add_log(message);
-        end
-
         # Iterates over the work item clients and invokes code on each of them.
         def each
           self.items.each { |item| yield item }
@@ -160,7 +118,7 @@ module Libis
         #
         # @param [WorkItem] item to be added to the child list :items
         def add_item(item)
-          return self unless item and item.is_a? WorkItem
+          return self unless item and item.is_a? Libis::Workflow::Base::WorkItem
           self.items << item
           item.parent = self
           self.save!
@@ -168,7 +126,28 @@ module Libis
           self
         end
 
-        alias :<< :add_item
+        alias_method :<<, :add_item
+
+        # Return item's parent
+        # @return [Libis::Workflow::Base::WorkItem]
+        def get_parent
+          self.parent
+        end
+
+        # go up the hierarchy and return the topmost work item
+        #
+        # @return [Libis::Workflow::Base::WorkItem]
+        def get_root
+          self.get_parent && self.get_parent.is_a?(Libis::Workflow::Base::WorkItem) && self.get_parent.get_root || self
+        end
+
+        # Get the top
+        #
+        # @return [Libis::Workflow::Base::Run]
+        def get_run
+          return self if self.is_a?(Libis::Workflow::Base::Run)
+          self.get_parent && self.get_parent.get_run || nil
+        end
 
         # Dummy method. It is a placeholder for DB backed implementations. Wherever appropriate WorkItem#save will be
         # called to save the current item's state. If state needs to persisted, you should override this method or make
@@ -182,63 +161,6 @@ module Libis
         def save!
         end
 
-        # Add a structured message to the log history. The message text can be submitted as an integer or text. If an
-        # integer is submitted, it will be used to look up the text in the MessageRegistry. The message text will be
-        # passed to the % operator with the args parameter. If that failes (e.g. because the format string is not correct)
-        # the args value is appended to the message.
-        #
-        # @param [Symbol] severity
-        # @param [Hash] msg should contain message text as :id or :text and the hierarchical name of the task as :task
-        # @param [Array] args string format values
-        def log_message(severity, msg, *args)
-          # Prepare info from msg struct for use with string substitution
-          message_id, message_text = if msg[:id]
-                                       [msg[:id], MessageRegistry.instance.get_message(msg[:id])]
-                                     elsif msg[:text]
-                                       [0, msg[:text]]
-                                     else
-                                       [0, '']
-                                     end
-          task = msg[:task] || '*UNKNOWN*'
-          message_text = (message_text % args rescue "#{message_text} - #{args}")
-
-          self.add_log severity: severity, id: message_id.to_i, text: message_text, task: task
-          name = ''
-          begin
-            name = self.to_s
-            name = self.name
-            name = self.namepath
-          rescue
-            # do nothing
-          end
-          Config.logger.add(severity, message_text, ('%s - %s ' % [task, name]))
-        end
-
-        protected
-
-        SEV_LABEL = %w(DEBUG INFO WARN ERROR FATAL ANY) unless const_defined? :SEV_LABEL
-
-        # go up the hierarchy and return the topmost work item
-        #
-        # @return [WorkItem] the root work item
-        def root
-          root = self
-          root = root.parent while root.parent and root.parent.is_a? WorkItem
-          root
-        end
-
-        # create and return a proper message structure
-        # @param [Hash] opts
-        def message_struct(opts = {})
-          opts.reverse_merge!(severity: ::Logger::INFO, code: nil, text: '')
-          {
-              severity: SEV_LABEL[opts[:severity]],
-              task: opts[:task],
-              code: opts[:code],
-              message: opts[:text]
-          }.cleanup
-        end
-
       end
 
     end

data/lib/libis/workflow/base/workflow.rb

@@ -1,6 +1,7 @@
 # encoding: utf-8
 
 require 'libis/tools/parameter'
+require 'libis/workflow/task_group'
 
 module Libis
   module Workflow
@@ -31,17 +32,14 @@ module Libis
       #       A task definition is a Hash with the following values:
       #       - class: [String] the class name of the task including the module names
       #       - name: [String] optional if class is present. A friendly name for the task that will be used in the logs.
-      #       - subitems: [Boolean] execute the task on the items in the current level or on the
-      #           child items of the current level. This parameter can be used in combination with the subtasks to
-      #           control what objects in the hierarchy the tasks are executed against.
-      #       - recursive: [Boolean] execute the task for the current level items only or automatically recurse through
-      #           the item's hierarchy and execute on all items below.
       #       - tasks: [Array] a list of subtask defintions for this task.
+      #
       #       Additionally the task definition Hash may specify values for any other parameter that the task knows of.
-      #       All tasks have parameters 'quiet', 'always_run', 'abort_on_error'. For more information about these see
-      #       the documentation of the task class.
+      #       All tasks have parameters 'quiet', 'recursive', 'retry_count' and 'retry_interval'. For more
+      #       information about these see the documentation of the task class.
+      #
       #       A task definition does not require to have a 'class' entry. If not present the default
-      #       ::Libis::Workflow::Task class will be instatiated. It will do nothing itself, but will execute the
+      #       ::Libis::Workflow::TaskGroup class will be instatiated. It will do nothing itself, but will execute the
       #       subtasks on the item(s). In such case a 'name' is mandatory.
       #
       # These values should be set by calling the #configure method which takes a Hash as argument with :name,
@@ -75,6 +73,7 @@ module Libis
         end
 
         def self.included(base)
+
           base.extend ClassMethods
         end
 
@@ -138,13 +137,13 @@ module Libis
         end
 
         def instantize_task(parent, cfg)
-          task_class = Task
+          task_class = Libis::Workflow::TaskGroup
           task_class = cfg[:class].constantize if cfg[:class]
           # noinspection RubyArgCount
           task_instance = task_class.new(parent, cfg)
-          cfg[:tasks].map do |task_cfg|
+          cfg[:tasks] && cfg[:tasks].map do |task_cfg|
             task_instance << instantize_task(task_instance, task_cfg)
-          end rescue nil
+          end
           task_instance
         end
 

data/lib/libis/workflow/base.rb

@@ -1,4 +1,5 @@
 require_relative 'base/logger'
+require_relative 'base/logging'
 require_relative 'base/work_item'
 require_relative 'base/file_item'
 require_relative 'base/dir_item'

data/lib/libis/workflow/status.rb

@@ -0,0 +1,83 @@
+module Libis
+  module Workflow
+    module Status
+
+      STATUS = {
+          NOT_STARTED: 0,
+          STARTED: 1,
+          DONE: 2,
+          ASYNC_WAIT: 3,
+          ASYNC_HALT: 4,
+          FAILED: 5
+      }
+
+      # Changes the status of the object. The status changed is logged in the status_log with the current timestamp.
+      #
+      # @param [Array] x Array with status and task
+      def status=(x)
+        s, task = x
+        self.add_status_log(task: task, status: s)
+        self.save
+      end
+
+      # Get last known status symbol for a given task
+      #
+      # @param [String] task task name to check item status for
+      # @return [Symbol] the status code
+      def status(task = nil)
+        entry = status_entry(task)
+        status_symbol(entry[:status]) rescue :NOT_STARTED
+      end
+
+      # Gets the last known status label of the object.
+      #
+      # @param [String] task name of task to get the status for
+      # @return [String] status label ( = task name + status )
+      def status_label(task = nil)
+        entry = self.status_entry(task)
+        "#{entry[:task] rescue nil}#{entry[:status].capitalize rescue nil}"
+      end
+
+      # Check status of the object.
+      #
+      # @param [Symbol] state status to look for
+      # @param [String] task name of task whose status to check
+      # @return [Boolean] true if the object status matches
+      def check_status(state, task = nil)
+        self.status(task) == state
+      end
+
+      # Compare status with current status of the object.
+      #
+      # @param [Symbol] state
+      # @return [Integer] 1, 0 or -1 depnding on which
+      def compare_status(state, task = nil)
+        STATUS[self.status(task)] <=> STATUS[state]
+      end
+
+      protected
+
+      # Get last known status entry for a given task
+      #
+      # @param [String] task task name to check item status for
+      # @return [Hash] the status entry
+      def status_entry(task = nil)
+        return self.status_log.last if task.blank?
+        self.status_log.select { |entry| entry[:task] == task }.last
+      end
+
+      # Convert String, Symbol or Integer to correct symbol for the status.
+      # If the input value is nil, the fist status entry is returned.
+      #
+      # @param [String|Symbol|Integer] x string, symbol or integer for status code.
+      # @return [Symbol] the corresponding STATUS symbol
+      def status_symbol(x)
+        return STATUS.key(x) if x.is_a?(Integer)
+        return x if STATUS.has_key?(x)
+        x = x.to_s.upcase.to_sym
+        STATUS.has_key?(x) ? x : nil
+      end
+
+    end
+  end
+end