libis-workflow 2.0.beta.11 → 2.0.beta.12

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: a86fb4847f5b02c16e1e6b71433f3ccba54ee721
-  data.tar.gz: a9c6c8c3a00b93216a6c08c00276b00f7a5886a8
+  metadata.gz: 3baa19f0fbc05d8254835124a68bf02dff63952d
+  data.tar.gz: 9a6f33b1e6de7ee1d699aaaca71e80a2194b1d9b
 SHA512:
-  metadata.gz: 37071948f241e661356d909e46d431219a7c088abc7867ebc04b9eff3250f4b41c3d7278eb5a42f9878398f071ff92a7bb26d099ea704e1cb84355f37a3d99b9
-  data.tar.gz: ef20686e433df64d613ef72bf09bd422b3d34943ee10d6b975789fab612bb98c6115724e241aff6256236019f469f43dcb4d5079c1e0b22cd62fe4df6bf17b61
+  metadata.gz: dd107fb0efd65099dda2c0215d93ad75fde65c0a2e3271f87ce505157ea1b1f321874d426f63a315214e3c09ba3ca3dc3d2f7d620edcdc02c16292912b38df41
+  data.tar.gz: 61199cfae002d01666ae76d76c2e5acdca3c837d99c59f57da0a972d9c21bdb011e59a89ebda3373f9b6387518cc0673f5d9282bf2516a72f7da5983cad36804

data/README.md

@@ -42,12 +42,12 @@ provided. It contains all the basic logic required for proper configuration and
 implementation is provided in the class ::Libis::Workflow::Workflow for your convenience to be used as-is or to derive
 your own from.
 
-The workflow object will be able to execute the tasks in proper order on all the WorkItems supplied/collected. Each 
-task can be implemented with code to run or simply contain a list of child tasks. When a workflow is executed a special
-run object is created that captures the configuration, logs and workitems generated while executing the tasks. Essential
-logic is provided in the module ::Libis::Workflow::Base::Run with a simple in-memory implementation in 
-::Libis::Workflow::Run. The run object's class name has to be provided to the workflow configuration so that the 
-workflow can instantiate the correct object.
+The Job class is responsible for instantiating a run-time workflow execution object - a Run - that captures the 
+configuration, logs and workitems generated while executing the tasks. Essential logic is provided in the module 
+::Libis::Workflow::Base::Run with a simple in-memory implementation in ::Libis::Workflow::Run. The run object's class 
+name has to be provided to the job configuration so that the job can instantiate the correct object. The run object 
+will be able to execute the tasks in proper order on all the WorkItems supplied/collected. Each task can be implemented 
+with code to run or simply contain a list of child tasks. 
 
 One tasks is predefined:
 ::Libis::Workflow::Tasks::Analyzer - analyzes the workflow run and summarizes the results. It is always included as the
@@ -67,22 +67,29 @@ You should start by including the following line in your source code:
 This will load all of the Libis Workflow framework into your environment, but including only the required parts is OK as
 well. This is shown in the examples below.
 
-### Workflows
+### Workflows and Jobs
 
 An implementation of ::Libis::Workflow::Base::Workflow contains the definition of a workflow. Once instantiated, it can 
-be run by calling the 'run' method. This will create an intance of an implementation of ::Libis::Workflow::Base::Run, 
-configure it and call the 'run' method on it. The Workflow constructor takes no arguments, but is should be configured 
-by calling the 'configure' method with the workflow configuration as an argument. The 'run' method takes an option Hash 
-as argument.
+be run by calling the 'execute' method on a job object created for that workflow. This will create an intance of an 
+implementation of ::Libis::Workflow::Base::Run, configure it and call the 'run' method on it. The Workflow constructor 
+takes no arguments, but is should be configured by calling the 'configure' method with the workflow configuration as an 
+argument. The job's 'execute' method takes an option Hash as argument with extra/overriding configuration values.
 
+### Job configuration
+A job configuration is a Hash with:
+* name: String to identify the workflow
+* description: String with detailed textual information
+* workflow: Object reference to a Workflow that contains the task configuration
+* run_object: String with class name of the ::Libis::Workflow::Base::Run implementation to be created. An istance of 
+  this class will be created for each run and serves as the root work item for that particular run.
+* input: Hash with input parameter values for the workflow
+  
 #### Workflow configuration
 
 A workflow configuration is a Hash with:
 * name: String to identify the workflow
 * description: String with detailed textual information
 * tasks: Array of task descriptions
-* run_object: String with class name of the ::Libis::Workflow::Base::Run implementation to be created. An istance of 
-  this class will be created for each run and serves as the root work item for that particular run. 
 * input: Hash with input variable definitions
 
 ##### Task description
@@ -108,8 +115,8 @@ the chapter on 'Tasks' below for more information on tasks.
 
 ##### Input variable definition
 
-The input variables define parameters for the workflow. When a workflow is run, it can give values for any of these
-input variable and the workflow run will use the new values instead of the defaults.
+The input variables define parameters for the workflow. When a job is executed, it can provide values for any of these
+input variables and the workflow run will use the new values instead of the defaults.
 
 The key of the input Hash is the unique name of the variable. The value is another Hash with the parameter definition.
 See ::Libis::Tools::Parameter for the content of this Hash.
@@ -122,8 +129,8 @@ if absent.
 
 #### Run-time configuration
 
-The 'run' method takes an optional Hash as argument which will complement and override the options Hash described in the
-previous chapter.
+The job's 'execute' method takes an optional Hash as argument which will complement and override the options Hash 
+described in the previous chapter.
  
 Once the workflow is configured and the root work item instantiated, the method will run each top-level task on the root
 work item in sequence until all tasks have completed successfully or a task has failed.

data/lib/libis/workflow/base/job.rb

@@ -0,0 +1,85 @@
+# encoding: utf-8
+
+require 'libis/tools/parameter'
+
+module Libis
+  module Workflow
+    module Base
+
+      # This is the base module for Jobs.
+      #
+      # This module lacks the implementation for the data attributes. It functions as an interface that describes the
+      # common functionality regardless of the storage implementation. These attributes require some implementation:
+      #
+      # - name: [String] the name of the Job. The name will be used to identify the job. Each time a job is executed,
+      #     a Run will be created for the associated workflow. The Run will get a name that starts with the job name
+      #     and ends with the date and time the Run was first started. As such this name attribute serves as an
+      #     identifier and should be treated as such. If possible it should be unique.
+      # - description: [String] optional information about the job.
+      # - workflow: [Object] the workflow containing the tasks that need to run.
+      # - run_obj: [String] the full class name of the Run implementation object that should be created when the
+      #     Job is executed.
+      # - input: [Hash] workflow input parameter values. Each input parameter of the workflow can be set by the entries
+      #     in this Hash.
+      #
+      # A minimal in-memory implementation could be:
+      #
+      # class Job
+      #   include ::Libis::Workflow::Base::Job
+      #
+      #   attr_accessor :name, :description, :workflow, :run_object, :input
+      #
+      #   def initialize
+      #     @name = ''
+      #     @description = ''
+      #     @input = Hash.new
+      #     @workflow = ::Libis::Workflow::Workflow.new
+      #     @run_object = ::Libis::Workflow::Run.new
+      #   end
+      #
+      # end
+      #
+      module Job
+
+        def run_name(timestamp = Time.now)
+          "#{self.name}-#{timestamp.strftime('%Y%m%d%H%M%S')}"
+        end
+
+        def configure(cfg = {})
+          self.name ||= ''
+          self.description ||= ''
+          self.input ||= {}
+
+          self.name = cfg[:name] if cfg.has_key?(:name)
+          self.description = cfg[:description] if cfg.has_key?(:description)
+          self.workflow = cfg[:workflow] if cfg.has_key?(:workflow)
+          self.run_object = cfg[:run_object] if cfg.has_key?(:run_object)
+          self.input.merge!(cfg[:input] || {})
+        end
+
+        # noinspection RubyResolve
+        # @param [Hash] opts optional extra conguration values for this particular run
+        def execute(opts = {})
+          run = self.create_run_object
+          raise RuntimeError.new "Could not create instance of run object '#{self.run_object}'" unless run
+
+          run.job = self
+          run.options = self.input.merge(opts)
+          run.save
+
+          run.run
+
+          run
+        end
+
+        protected
+
+        # noinspection RubyResolve
+        def create_run_object
+          self.run_object.constantize.new
+        end
+
+      end
+    end
+  end
+end

data/lib/libis/workflow/base/run.rb

@@ -14,7 +14,7 @@ module Libis
       # common functionality regardless of the storage implementation. These attributes require some implementation:
       #
       # - start_date: [Time] the timestamp of the execution of the run
-      # - workflow: [Object] a reference to the Workflow this Run belongs to
+      # - job: [Object] a reference to the Job this Run belongs to
       #
       # Note that ::Libis::Workflow::Base::WorkItem is a parent module and therefore requires implementation of the
       # attributes of that module too.
@@ -33,7 +33,7 @@ module Libis
         end
 
         def name
-          self.workflow.run_name(self.start_date)
+          self.job.run_name(self.start_date)
         end
 
         def names
@@ -44,16 +44,18 @@ module Libis
           self.name
         end
 
+        def workflow
+          self.job.workflow
+        end
+
         # Execute the workflow.
-        # @param [Hash] opts a list with parameter name and value tuples that specify the values for the workflow input
-        #     parameters.
-        def run(opts = {})
+        def run
 
           self.start_date = Time.now
 
-          self.options = workflow.prepare_input(self.options.merge(opts))
+          self.options = workflow.prepare_input(self.options)
 
-          self.tasks = self.workflow.tasks(self)
+          self.tasks = workflow.tasks(self)
           configure_tasks self.options
 
           self.status = :STARTED

data/lib/libis/workflow/base/workflow.rb

@@ -19,8 +19,6 @@ module Libis
       # - config: [Hash] detailed configuration for the workflow. The application assumes it behaves as a Hash and will
       #     access it with [], merge! and delete methods. If your implementation decides to implement it with another
       #     object, it should implement above methods. The config Hash requires the following keys:
-      #   - run_object: [String] the full class name of the Run implementation object that should be created when the
-      #       Workflow is executed.
       #   - input: [Hash] all input parameter definitions where the key is the parameter name and the value is another
       #       Hash with arguments for the parameter definition. It typically contains the following arguments:
       #       - default: default value if no value specified when the workflow is executed
@@ -46,6 +44,9 @@ module Libis
       #       ::Libis::Workflow::Task class will be instatiated. It will do nothing itself, but will execute the
       #       subtasks on the item(s). In such case a 'name' is mandatory.
       #
+      # These values should be set by calling the #configure method which takes a Hash as argument with :name,
+      # :description, :input and :tasks keys.
+      #
       # A minimal in-memory implementation could be:
       #
       # class Workflow
@@ -55,7 +56,7 @@ module Libis
       #
       #   def initialize
       #     @name = ''
-      #     @descripition = ''
+      #     @description = ''
       #     @config = Hash.new
       #   end
       #
@@ -105,44 +106,15 @@ module Libis
           {}
         end
 
-        def run_name(timestamp = Time.now)
-          "#{self.workflow.name}-#{timestamp.strftime('%Y%m%d%H%M%S')}"
-        end
-
-        def perform(opts = {})
-          self.run opts
-        end
-
-        def create_run_object
-          self.config[:run_object].constantize.new
-        end
-
-        # @param [Hash] opts
-        def run(opts = {})
-
-          run_object = self.create_run_object
-          raise RuntimeError.new "Could not create instance of run object '#{self.config[:run_object]}'" unless run_object
-
-          run_object.workflow = self
-          run_object.options = opts
-          run_object.save
-
-          run_object.run opts
-
-          run_object
-        end
-
         # @param [Hash] opts
         def prepare_input(opts)
           options = opts.dup
           self.input.each do |key, parameter|
-            key
-            # provided in opts
-            options[key] = parameter[:default] unless options.has_key? key
+            options[key] = parameter[:default] unless options.has_key?(key)
             options[key] = parameter.parse(options[key])
             propagate_to = []
             propagate_to = parameter[:propagate_to] if parameter[:propagate_to].is_a? Array
-            propagate_to = parameter[:propagate_to].split(/\s*,\s*/) if parameter[:propagate_to].is_a? String
+            propagate_to = parameter[:propagate_to].split(/[\s,;]+/) if parameter[:propagate_to].is_a? String
             propagate_to.each do |target|
               task_name, param_name = target.split('#')
               param_name ||= key

data/lib/libis/workflow/job.rb

@@ -0,0 +1,26 @@
+# encoding: utf-8
+
+require 'libis/workflow/base/job'
+require 'libis/workflow/workflow'
+require 'libis/workflow/run'
+
+module Libis
+  module Workflow
+
+    class Job
+      include ::Libis::Workflow::Base::Job
+
+      attr_accessor :name, :description, :workflow, :run_object, :input
+
+      def initialize
+        @name = ''
+        @description = ''
+        @input = Hash.new
+        @workflow = ::Libis::Workflow::Workflow.new
+        @run_object = ::Libis::Workflow::Run.new
+      end
+
+    end
+
+  end
+end

data/lib/libis/workflow/run.rb

@@ -12,11 +12,11 @@ module Libis
     class Run < ::Libis::Workflow::WorkItem
       include ::Libis::Workflow::Base::Run
 
-      attr_accessor :start_date, :workflow
+      attr_accessor :start_date, :job
 
       def initialize
         @start_date = Time.now
-        @workflow = nil
+        @job = nil
         super
       end
 

data/lib/libis/workflow/version.rb

@@ -2,6 +2,6 @@
 
 module Libis
   module Workflow
-    VERSION = '2.0.beta.11' unless const_defined? :VERSION # the guard is against a redefinition warning that happens on Travis
+    VERSION = '2.0.beta.12' unless const_defined? :VERSION # the guard is against a redefinition warning that happens on Travis
   end
 end

data/lib/libis/workflow/worker.rb

@@ -10,30 +10,30 @@ module Libis
     class Worker
       include Sidekiq::Worker
 
-      def perform(workflow_config, options = {})
-        workflow = self.class.configure(workflow_config, options)
+      def perform(job_config, options = {})
+        job = self.class.configure(job_config, options)
         options[:interactive] = false
-        workflow.run options
+        job.execute options
       end
 
-      def self.configure(workflow_config, options = {})
+      def self.configure(job_config, options = {})
         log_path = options.delete :log_path
         if log_path
           Config.logger = ::Logger.new(
-              File.join(log_path, "#{workflow_config}.log"),
+              File.join(log_path, "#{job_config[:name]}.log"),
               (options.delete(:log_shift_age) || 'daily'),
               (options.delete(:log_shift_size) || 1024 ** 2)
           )
           Config.logger.formatter = ::Logger::Formatter.new
           Config.logger.level = ::Logger::DEBUG
         end
-        get_workflow(workflow_config)
+        get_job(job_config)
       end
 
-      def get_workflow(workflow_config)
-        workflow = ::Libis::Workflow::Workflow.new
-        workflow.configure workflow_config
-        workflow
+      def get_job(job_config)
+        job = ::Libis::Workflow::Job.new
+        job.configure job_config
+        job
       end
 
     end

data/lib/libis/workflow/workflow.rb

@@ -1,12 +1,5 @@
 # encoding: utf-8
 
-require 'backports/rails/string'
-require 'backports/rails/hash'
-
-require 'libis/workflow/config'
-require 'libis/workflow/task'
-require 'libis/workflow/tasks/analyzer'
-
 require 'libis/workflow/base/workflow'
 
 module Libis
@@ -19,7 +12,7 @@ module Libis
 
       def initialize
         @name = ''
-        @descripition = ''
+        @description = ''
         @config = Hash.new
       end
 

data/lib/libis/workflow.rb

@@ -12,6 +12,7 @@ module Libis
       autoload :FileItem, 'libis/workflow/base/file_item'
       autoload :DirItem, 'libis/workflow/base/dir_item'
       autoload :Logger, 'libis/workflow/base/logger'
+      autoload :Job, 'libis/workflow/base/job'
       autoload :Run, 'libis/workflow/base/run'
       autoload :Workflow, 'libis/workflow/base/workflow'
     end
@@ -21,6 +22,7 @@ module Libis
     autoload :DirItem, 'libis/workflow/dir_item'
 
     autoload :Workflow, 'libis/workflow/workflow'
+    autoload :Job, 'libis/workflow/job'
     autoload :Run, 'libis/workflow/run'
     autoload :Task, 'libis/workflow/task'
 

data/spec/workflow_spec.rb

@@ -33,7 +33,6 @@ describe 'TestWorkflow' do
                 ]
             }
         ],
-        run_object: 'TestRun',
         input: {
             dirname: {default: '.', propagate_to: 'CollectFiles#location'},
             checksum_type: {default: 'SHA1', propagate_to: 'ChecksumTester'}
@@ -42,8 +41,20 @@ describe 'TestWorkflow' do
     workflow
   }
 
+  let(:job) {
+    job = ::Libis::Workflow::Job.new
+    job.configure(
+        name: 'TestJob',
+        description: 'Job for testing',
+        workflow: workflow,
+        run_object: 'TestRun',
+        input: {dirname: dirname, checksum_type: 'SHA256'},
+    )
+    job
+  }
+
   let!(:run) {
-    workflow.run(dirname: dirname, checksum_type: 'SHA256')
+    job.execute
   }
 
   it 'should contain three tasks' do

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: libis-workflow
 version: !ruby/object:Gem::Version
-  version: 2.0.beta.11
+  version: 2.0.beta.12
 platform: ruby
 authors:
 - Kris Dekeyser
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-10-22 00:00:00.000000000 Z
+date: 2015-11-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -127,6 +127,7 @@ files:
 - lib/libis/workflow/base.rb
 - lib/libis/workflow/base/dir_item.rb
 - lib/libis/workflow/base/file_item.rb
+- lib/libis/workflow/base/job.rb
 - lib/libis/workflow/base/logger.rb
 - lib/libis/workflow/base/run.rb
 - lib/libis/workflow/base/work_item.rb
@@ -134,6 +135,7 @@ files:
 - lib/libis/workflow/config.rb
 - lib/libis/workflow/dir_item.rb
 - lib/libis/workflow/file_item.rb
+- lib/libis/workflow/job.rb
 - lib/libis/workflow/message_registry.rb
 - lib/libis/workflow/run.rb
 - lib/libis/workflow/task.rb
@@ -173,7 +175,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: 1.3.1
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.2.2
+rubygems_version: 2.5.0
 signing_key: 
 specification_version: 4
 summary: LIBIS Workflow framework.