dynflow 0.6.2 → 0.7.0

data/README.md

@@ -12,6 +12,9 @@ written in Ruby that allows to:
 * extend the workflows from third-party libraries
 * keep consistency between local transactional database and
   external services
+* suspend the long-running steps, not blocking the thread pool
+* cancel steps when possible
+* extend the actions behavior with middlewares
 * define the input/output interface between the building blocks (planned)
 * define rollback for the workflow (planned)
 * have multiple workers for distributing the load (planned)
@@ -22,8 +25,12 @@ persistence, transaction layer or executor implementation, giving you
 the last word in choosing the right one (providing default
 implementations as well).
 
+![Screenshot](doc/images/screenshot.png)
+
 * [Current status](#current-status)
 * [How it works](#how-it-works)
+* [Examples](#examples)
+* [The Anatomy of Action Class](#the-anatomy-of-action-class)
 * [Glossary](#glossary)
 * [Related projects](#related-projects)
 
@@ -35,12 +42,6 @@ to support the services orchestration in the
 [Katello](http://katello.org) and [Foreman](http://theforeman.org/)
 projects, getting to production-ready state in couple of weeks.
 
-Requirements
-------------
-
--   Ruby MRI 1.9.3, 2.0, or 2.1.
--   It does not work on JRuby nor Rubinius yet.
-
 How it works
 ------------
 
@@ -90,14 +91,28 @@ The output of this phase is a set of actions and their inputs.
 
 Every action can participate in every phase.
 
-Example
--------
+Examples
+--------
 
-One code snippet is worth 1000 words:
+The `examples` directory contains simple ruby scripts different
+features in action. You can just run the example files and see the Dynflow
+in action.
+
+* `orchestrate.rb` - example worlflow of getting some infrastructure
+  up and running, with ability to rescue from some error states.
+
+* `orchestrate_evented.rb` - the same workflow using the ability to
+  suspend/wakeup actions while waiting for some external event.
+  It also demonstrates the ability to cancel actions that support it.
+
+* `remote_executor.rb` - example of executing the flows in external
+  process
 
-```ruby
-# The anatomy of action class
 
+The Anatomy of Action Class
+---------------------------
+
+```ruby
 # every action needs to inherit from Dynflow::Action
 class Action < Dynflow::Action
 
@@ -151,7 +166,6 @@ class Action < Dynflow::Action
   end
 end
 ```
-
 Every action should be as atomic as possible, providing better
 granularity when manipulating the process. Since every action can be
 subscribed by another one, adding new behaviour to an existing
@@ -160,8 +174,6 @@ workflow is really simple.
 The input and output format can be used for defining the interface
 that other developers can use when extending the workflows.
 
-See the examples directory for more complete examples.
-
 Glossary
 --------
 
@@ -208,6 +220,14 @@ Related projects
    for running system tasks with Dynflow, comes with simple Web-UI for
    testing it
 
+
+Requirements
+------------
+
+-   Ruby MRI 1.9.3, 2.0, or 2.1.
+-   It does not work on JRuby nor Rubinius yet.
+
+
 License
 -------
 

data/examples/example_helper.rb

@@ -0,0 +1,47 @@
+$:.unshift(File.expand_path('../../lib', __FILE__))
+
+require 'dynflow'
+
+class ExampleHelper
+  class << self
+    def world
+      @world ||= create_world
+    end
+
+    def create_world(options = {})
+      options = default_world_options.merge(options)
+      Dynflow::SimpleWorld.new(options)
+    end
+
+    def default_world_options
+      { logger_adapter: logger_adapter }
+    end
+
+    def logger_adapter
+      Dynflow::LoggerAdapters::Simple.new $stderr, 4
+    end
+
+
+    def run_web_console(world = ExampleHelper.world)
+      require 'dynflow/web_console'
+      dynflow_console = Dynflow::WebConsole.setup do
+        set :world, world
+      end
+      dynflow_console.run!
+    end
+
+    # for simulation of the execution failing for the first time
+    def something_should_fail!
+      @should_fail = true
+    end
+
+    # for simulation of the execution failing for the first time
+    def something_should_fail?
+      @should_fail
+    end
+
+    def nothing_should_fail!
+      @should_fail = false
+    end
+  end
+end

data/examples/orchestrate.rb

@@ -1,7 +1,30 @@
-# Simple example on how Dynflow could be used for simple infrastructure orchestration
+#!/usr/bin/env ruby
+
+example_description = <<DESC
+  Orchestrate Example
+  ===================
+
+  This example simulates a workflow of setting up an infrastructure, using
+  more high-level steps in CreateInfrastructure, that expand to smaller steps
+  of PrepareDisk, CraeteVM etc.
+
+  It shows the possibility to run the independend actions concurrently, chaining
+  the actions (passing the output of PrepareDisk action to CreateVM, automatically
+  detecting the dependency making sure to run the one before the other).
+
+  It also simulates a failure and demonstrates the Dynflow ability to rescue
+  from the error and consinue with the run.
+
+  Once the Sinatra web console starts, you can navigate to http://localhost:4567
+  to see what's happening in the Dynflow world.
+
+DESC
+
+require_relative 'example_helper'
 
 module Orchestrate
 
+
   class CreateInfrastructure < Dynflow::Action
 
     def plan
@@ -16,7 +39,6 @@ module Orchestrate
                     :db_machine => 'host1',
                     :storage_machine => 'host2')
       end
-      sleep 2
     end
   end
 
@@ -36,12 +58,19 @@ module Orchestrate
     end
 
     def finalize
-      puts "We've create a machine #{input[:name]}"
+      # this is called after run methods of the actions in the
+      # execution plan were finished
     end
 
   end
 
-  class PrepareDisk < Dynflow::Action
+  class Base < Dynflow::Action
+    def sleep!
+      sleep(rand(2))
+    end
+  end
+
+  class PrepareDisk < Base
 
     input_format do
       param :name
@@ -52,13 +81,13 @@ module Orchestrate
     end
 
     def run
-      sleep(rand(5))
+      sleep!
       output[:path] = "/var/images/#{input[:name]}.img"
     end
 
   end
 
-  class CreateVM < Dynflow::Action
+  class CreateVM < Base
 
     input_format do
       param :name
@@ -70,25 +99,25 @@ module Orchestrate
     end
 
     def run
-      sleep(rand(5))
+      sleep!
       output[:ip] = "192.168.100.#{rand(256)}"
     end
 
   end
 
-  class AddIPtoHosts < Dynflow::Action
+  class AddIPtoHosts < Base
 
     input_format do
       param :ip
     end
 
     def run
-      sleep(rand(5))
+      sleep!
     end
 
   end
 
-  class ConfigureMachine < Dynflow::Action
+  class ConfigureMachine < Base
 
     input_format do
       param :ip
@@ -98,24 +127,31 @@ module Orchestrate
 
     def run
       # for demonstration of resuming after error
-      if Orchestrate.should_fail?
-        Orchestrate.should_pass!
+      if ExampleHelper.something_should_fail?
+        ExampleHelper.nothing_should_fail!
+        puts <<-MSG.gsub(/^.*\|/, '')
+
+                | Execution plan #{execution_plan_id} is failing
+                | You can resume it at http://localhost:4567/#{execution_plan_id}
+
+        MSG
         raise "temporary unavailabe"
       end
 
-      sleep(rand(5))
+      sleep!
     end
 
   end
+end
 
-
-  # for simulation of the execution failing for the first time
-  def self.should_fail?
-    ! @should_pass
-  end
-
-  def self.should_pass!
-    @should_pass = true
+if $0 == __FILE__
+  ExampleHelper.something_should_fail!
+  ExampleHelper.world.trigger(Orchestrate::CreateInfrastructure)
+  Thread.new do
+    9.times do
+      ExampleHelper.world.trigger(Orchestrate::CreateInfrastructure)
+    end
   end
-
+  puts example_description
+  ExampleHelper.run_web_console
 end

data/examples/orchestrate_evented.rb

@@ -0,0 +1,174 @@
+#!/usr/bin/env ruby
+
+require_relative 'example_helper'
+
+example_description = <<DESC
+  Orchestrate Evented Example
+  ===========================
+
+  This example, how the `examples/orchestrate.rb` can be updated to not block
+  the threads while waiting for external tasks. In this cases, we usually wait
+  most of the time: and we can suspend the run of the action while waiting,
+  for the event. Therefore we suspend the action in the run, ask the world.clock
+  to wake us up few seconds later, so that the thread pool can do something useful
+  in the meantime.
+
+  Additional benefit besides being able to do more while waiting is allowing to
+  send external events to the action while it's suspended. One use case is being
+  able to cancel the action while it's running.
+
+  Once the Sinatra web console starts, you can navigate to http://localhost:4567
+  to see what's happening in the Dynflow world.
+
+DESC
+
+module OrchestrateEvented
+
+  class CreateInfrastructure < Dynflow::Action
+
+    def plan(get_stuck = false)
+      sequence do
+        concurrence do
+          plan_action(CreateMachine, 'host1', 'db', get_stuck: get_stuck)
+          plan_action(CreateMachine, 'host2', 'storage')
+        end
+        plan_action(CreateMachine,
+                    'host3',
+                    'web_server',
+                    :db_machine => 'host1',
+                    :storage_machine => 'host2')
+      end
+    end
+  end
+
+  class CreateMachine < Dynflow::Action
+
+    def plan(name, profile, config_options = {})
+      prepare_disk = plan_action(PrepareDisk, 'name' => name)
+      create_vm    = plan_action(CreateVM,
+                                 :name => name,
+                                 :disk => prepare_disk.output['path'])
+      plan_action(AddIPtoHosts, :name => name, :ip => create_vm.output[:ip])
+      plan_action(ConfigureMachine,
+                  :ip => create_vm.output[:ip],
+                  :profile => profile,
+                  :config_options => config_options)
+      plan_self(:name => name)
+    end
+
+    def finalize
+    end
+
+  end
+
+  class Base < Dynflow::Action
+
+    Finished = Algebrick.atom
+
+    def run(event = nil)
+      match(event,
+            (on Finished do
+               on_finish
+             end),
+            (on Dynflow::Action::Skip do
+               # do nothing
+             end),
+            (on nil do
+               suspend { |suspended_action| world.clock.ping suspended_action, rand(1), Finished }
+             end))
+    end
+
+    def on_finish
+      raise NotImplementedError
+    end
+
+  end
+
+  class PrepareDisk < Base
+
+    input_format do
+      param :name
+    end
+
+    output_format do
+      param :path
+    end
+
+    def on_finish
+      output[:path] = "/var/images/#{input[:name]}.img"
+    end
+
+  end
+
+  class CreateVM < Base
+
+    input_format do
+      param :name
+      param :disk
+    end
+
+    output_format do
+      param :ip
+    end
+
+    def on_finish
+      output[:ip] = "192.168.100.#{rand(256)}"
+    end
+
+  end
+
+  class AddIPtoHosts < Base
+
+    input_format do
+      param :ip
+    end
+
+    def on_finish
+    end
+
+  end
+
+  class ConfigureMachine < Base
+
+    # thanks to this Dynflow knows this action can be politely
+    # asked to get canceled
+    include ::Dynflow::Action::Cancellable
+
+    input_format do
+      param :ip
+      param :profile
+      param :config_options
+    end
+
+    def run(event = nil)
+      if event == Dynflow::Action::Cancellable::Cancel
+        output[:message] = "I was cancelled but we don't care"
+      else
+        super
+      end
+    end
+
+    def on_finish
+      if input[:config_options][:get_stuck]
+        puts <<-MSG.gsub(/^.*\|/, '')
+
+            | Execution plan #{execution_plan_id} got stuck
+            | You can cancel the stucked step at http://localhost:4567/#{execution_plan_id}
+
+        MSG
+        # we suspend the action but don't plan the wakeup event,
+        # causing it to wait forever (till we cancel it)
+        suspend
+      end
+    end
+
+  end
+
+end
+
+if $0 == __FILE__
+  ExampleHelper.world.trigger(OrchestrateEvented::CreateInfrastructure)
+  ExampleHelper.world.trigger(OrchestrateEvented::CreateInfrastructure, true)
+  puts example_description
+  ExampleHelper.run_web_console
+end