dynflow 0.1.0 → 0.2.0

data/.gitignore

@@ -1 +1,7 @@
 Gemfile.lock
+*.swp
+*.swn
+*.swo
+.bin
+.bundle
+.idea

data/.travis.yml

@@ -0,0 +1,9 @@
+language:
+  - ruby
+
+rvm:
+  - "1.9.3"
+  - "2.0.0"
+
+script:
+  - bundle exec rake test

data/Gemfile

@@ -5,13 +5,3 @@ gemspec
 group :development, :test do
   gem 'pry'
 end
-
-group :test do
-  gem 'database_cleaner'
-end
-
-group :engine do
-  gem "rails", "~> 3.2.8"
-  gem "jquery-rails"
-  gem "sqlite3"
-end

data/MIT-LICENSE

@@ -1,4 +1,4 @@
-Copyright 2012 Pavel Pokorný
+Copyright 2013 Ivan Nečas
 
 Permission is hereby granted, free of charge, to any person obtaining
 a copy of this software and associated documentation files (the

data/README.md

@@ -1,25 +1,60 @@
-DYNamic workFLOW
-================
+![Dynflow](doc/images/logo.png)
+=======
+
+Dynflow [DYN(amic work)FLOW] is a workflow engine
+written in Ruby that allows to:
+
+* keep track of the progress of running process
+* run the code asynchronously
+* resume the process when something goes wrong, skip some steps when needed
+* detect independent parts and run them concurrently
+* compose simple actions into more complex scenarios
+* extend the workflows from third-party libraries
+* keep consistency between local transactional database and
+  external services
+* define the input/output interface between the building blocks (planned)
+* define rollback for the workflow (planned)
+* have multiple workers for distributing the load (planned)
+
+Dynflow doesn't try to choose the best tool for the jobs, as the right
+tool depends on the context. Instead, it provides interfaces for
+persistence, transaction layer or executor implementation, giving you
+the last word in choosing the right one (providing default
+implementations as well).
+
+* [Current status](#current-status)
+* [How it works](#how-it-works)
+* [Glossary](#glossary)
+* [Related projects](#related-projects)
+
+Current status
+-------------
+
+Dynflow has been under heavy development for several months to be able
+to support the services orchestration in the
+[Katello](http://katello.org) and [Foreman](http://theforeman.org/)
+projects, getting to production-ready state in couple of weeks.
+
+How it works
+------------
 
 In traditional workflow engines, you specify a static workflow and
 then run it with various inputs. Dynflow takes different approach.
-
 You specify the inputs and the workflow is generated on the fly. You
 can either specify the steps explicitly or subscribe one action to
 another. This is suitable for plugin architecture, where you can't
 write the whole process on one place.
 
 Dynflow doesn't differentiate between workflow and action. Instead,
-every action can populate another actions, effectively producing the
-resulting set of steps.
+every action can populate another actions. This allows composing
+more simpler workflows into a big one.
 
 The whole execution is done in three phases:
 
-1. *Planning phase*
+1. *Plan phase*
 
-  Construct the execution plan for the workflow. It's invoked by
-  calling `trigger` on an action. Two mechanisms are used to get the set
-  of actions to be executed:
+  Construct the execution plan for the workflow. Two mechanisms are
+  used to get the set of actions to be executed:
 
     a. explicit calls of `plan_action` methods in the `plan` method
 
@@ -29,7 +64,7 @@ The whole execution is done in three phases:
 
 The output of this phase is a set of actions and their inputs.
 
-2. *Execution phase*
+2. *Run phase*
 
   The plan is being executed step by step, calling the run method of
   an action with corresponding input. The results of every action are
@@ -41,7 +76,7 @@ The output of this phase is a set of actions and their inputs.
   serialized therefore the workflow itself can be persisted. This makes
   it easy to recover from failed actions by rerunning it.
 
-3. *Finalization phase*
+3. *Finalize phase*
 
   Take the results from the execution phase and perform some additional
   tasks. This is suitable for example for recording the results into
@@ -91,45 +126,26 @@ class Action < Dynflow::Action
     plan_action SubAction, object_1
     # we can specify, where in the workflow this action should be
     # placed, as well as prepare the input.
-    plan_self {'id' => object_2.id, 'name' => object_2.name}
+    plan_self { id: object_2.id, name: object_2.name}
   end
 
   # OPTIONAL: run the execution part of this action. Transform the
   # data from +input+ to +output+. When not specified, the action is
   # not used in the execution phase.
   def run
-    output['uuid'] = "#{input['name']}-#{input['id']}"
+    output[:uuid] = "#{input[:name]}-#{input[:id]}"
   end
 
   # OPTIONAL: finalize the action after the execution phase finishes.
   # in the +input+ and +output+ attributes are available the data from
   # execution phase. in the +outputs+ argument, all the execution
   # phase actions are available, each providing its input and output.
-  def finalize(outputs)
-    puts output['uuid']
+  def finalize
+    puts output[:uuid]
   end
 end
 ```
 
-One can generate the execution plan for an action without actually
-running it:
-
-```ruby
-pp Publish.plan(short_article).actions
-# the expanded workflow is:
-# [
-#  Publish: {"title"=>"Short", "body"=>"Short"} ~> {},
-#  Review:  {"title"=>"Short", "body"=>"Short"} ~> {},
-#  Print:   {"title"=>"Short", "body"=>"Short", "color"=>false} ~> {}
-# ]
-```
-
-Therefore it's suitable for the plan methods to not have any side
-effects (except of database writes that can be roll-backed)
-
-In the finalization phase, `finalize` method is called on every action
-if defined. The order is the same as in the execution plan.
-
 Every action should be as atomic as possible, providing better
 granularity when manipulating the process. Since every action can be
 subscribed by another one, adding new behaviour to an existing
@@ -140,12 +156,58 @@ that other developers can use when extending the workflows.
 
 See the examples directory for more complete examples.
 
+Glossary
+--------
+
+* **action** - building block for the workflows: a Ruby class
+    inherited from `Dynflow::Action`. Defines code to be run in
+    plan/run/finalize phase. It has defined input and output data.
+* **execution plan** - definition of the workflow: product of the plan
+    phase
+* **trigger an action** - entering the plan phase, starting with the
+    `plan` method of the action. The execution follows immediately.
+* **plan_self** - converts the arguments of the `plan` method into
+    action input, that can be accessed from the `run`/`finalize`
+    phase.
+* **plan_action** - includes another action into the workflow, passing
+    the arguments into the `plan` method of the action
+* **step** - execution unit of the action. It represents the action in
+    specific phase (plan step, run step, finalize step).
+* **flow** - definition of the run/finalize phase, holding the
+    information about steps that can run concurrently/in sequence.
+    Part of execution plan.
+* **executor** - service that executes the run and finalize flows
+    based on the execution plan. It can run in the same process as the
+    plan phase or in different process (using the remote executor)
+* **world** - the universe where the Dynflow runs the code: it holds
+    all needed configuration.
+
+Related projects
+----------------
+
+* [Foreman](http://theforeman.org) - lifecycle management tool for
+  physical and virtual servers
+
+* [Katello](http://katello.org) - content management plugin for
+  Foreman: integrates couple of REST services for managing the
+  software updates in the infrastructure.
+
+* [Foreman-tasks](https://github.com/iNecas/foreman-tasks) - Foreman
+  plugin providing the tasks management with Dynflow on the back-end
+
+* [Dyntask](https://github.com/iNecas/dyntask) - generic Rails engine
+  providing the tasks management features with Dynflow on the back-end
+
+* [Sysflow](https://github.com/iNecas/sysflow) - set of reusable tools
+   for running system tasks with Dynflow, comes with simple Web-UI for
+   testing it
+
 License
 -------
 
 MIT
 
-Author
-------
+Authors
+-------
 
-Ivan Nečas
+Ivan Nečas, Petr Chalupa

data/Rakefile

@@ -1,15 +1,11 @@
 require 'rake/testtask'
 require 'fileutils'
 
-desc "Generic tests"
 Rake::TestTask.new do |t|
   t.libs << 'lib' << 'test'
   t.test_files = FileList['test/*_test.rb']
   t.verbose = true
 end
 
-namespace :test do
-  desc "All tests"
-  task :all => [:test] do
-  end
-end
+desc Rake::Task['test'].comment
+task :default => :test

data/dynflow.gemspec

@@ -10,7 +10,7 @@ Gem::Specification.new do |s|
   s.homepage    = "http://github.com/iNecas/dynflow"
   s.summary     = "DYNamic workFLOW engine"
   s.description = "Generate and executed workflows dynamically based "+
-                  "on input data and leave it open for others to jump into it as well"
+      "on input data and leave it open for others to jump into it as well"
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
@@ -19,5 +19,14 @@ Gem::Specification.new do |s|
   s.add_dependency "activesupport"
   s.add_dependency "multi_json"
   s.add_dependency "apipie-params"
+  s.add_dependency "algebrick", '~> 0.4.0'
+  s.add_dependency "uuidtools"
+
+  s.add_development_dependency "rack-test"
   s.add_development_dependency "minitest"
+  s.add_development_dependency "minitest-reporters"
+  s.add_development_dependency "activerecord"
+  s.add_development_dependency "sequel"
+  s.add_development_dependency "sqlite3"
+  s.add_development_dependency "sinatra"
 end

data/examples/generate_work_for_daemon.rb

@@ -0,0 +1,24 @@
+#!/usr/bin/env ruby
+
+root_path    = File.expand_path(File.join(File.dirname(__FILE__), '..'))
+dynflow_path = File.join(root_path, 'lib')
+$LOAD_PATH << dynflow_path unless $LOAD_PATH.include? dynflow_path
+
+require 'dynflow'
+require 'tmpdir'
+
+socket_path         = File.join(Dir.tmpdir, 'dynflow_socket')
+persistence_adapter = Dynflow::PersistenceAdapters::Sequel.new ARGV[0] || 'sqlite://db.sqlite'
+
+world = Dynflow::SimpleWorld.new do |world|
+  { persistence_adapter: persistence_adapter,
+    executor:            Dynflow::Executors::RemoteViaSocket.new(world, socket_path) }
+end
+
+load File.join(root_path, 'test', 'code_workflow_example.rb')
+
+loop do
+  world.trigger Dynflow::CodeWorkflowExample::Slow, 1
+  sleep 0.5
+  p 'tick'
+end

data/examples/orchestrate.rb

@@ -0,0 +1,121 @@
+# Simple example on how Dynflow could be used for simple infrastructure orchestration
+
+module Orchestrate
+
+  class CreateInfrastructure < Dynflow::Action
+
+    def plan
+      sequence do
+        concurrence do
+          plan_action(CreateMachine, 'host1', 'db')
+          plan_action(CreateMachine, 'host2', 'storage')
+        end
+        plan_action(CreateMachine,
+                    'host3',
+                    'web_server',
+                    :db_machine => 'host1',
+                    :storage_machine => 'host2')
+      end
+      sleep 2
+    end
+  end
+
+  class CreateMachine < Dynflow::Action
+
+    def plan(name, profile, config_options = {})
+      prepare_disk = plan_action(PrepareDisk, 'name' => name)
+      create_vm    = plan_action(CreateVM,
+                                 :name => name,
+                                 :disk => prepare_disk.output['path'])
+      plan_action(AddIPtoHosts, :name => name, :ip => create_vm.output[:ip])
+      plan_action(ConfigureMachine,
+                  :ip => create_vm.output[:ip],
+                  :profile => profile,
+                  :config_options => config_options)
+      plan_self(:name => name)
+    end
+
+    def finalize
+      puts "We've create a machine #{input[:name]}"
+    end
+
+  end
+
+  class PrepareDisk < Dynflow::Action
+
+    input_format do
+      param :name
+    end
+
+    output_format do
+      param :path
+    end
+
+    def run
+      sleep(rand(5))
+      output[:path] = "/var/images/#{input[:name]}.img"
+    end
+
+  end
+
+  class CreateVM < Dynflow::Action
+
+    input_format do
+      param :name
+      param :disk
+    end
+
+    output_format do
+      param :ip
+    end
+
+    def run
+      sleep(rand(5))
+      output[:ip] = "192.168.100.#{rand(256)}"
+    end
+
+  end
+
+  class AddIPtoHosts < Dynflow::Action
+
+    input_format do
+      param :ip
+    end
+
+    def run
+      sleep(rand(5))
+    end
+
+  end
+
+  class ConfigureMachine < Dynflow::Action
+
+    input_format do
+      param :ip
+      param :profile
+      param :config_options
+    end
+
+    def run
+      # for demonstration of resuming after error
+      if Orchestrate.should_fail?
+        Orchestrate.should_pass!
+        raise "temporary unavailabe"
+      end
+
+      sleep(rand(5))
+    end
+
+  end
+
+
+  # for simulation of the execution failing for the first time
+  def self.should_fail?
+    ! @should_pass
+  end
+
+  def self.should_pass!
+    @should_pass = true
+  end
+
+end

data/examples/run_daemon.rb

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+root_path    = File.expand_path(File.join(File.dirname(__FILE__), '..'))
+dynflow_path = File.join(root_path, 'lib')
+$LOAD_PATH << dynflow_path unless $LOAD_PATH.include? dynflow_path
+
+require 'dynflow'
+require 'tmpdir'
+
+socket              = File.join(Dir.tmpdir, 'dynflow_socket')
+persistence_adapter = Dynflow::PersistenceAdapters::Sequel.new ARGV[0] || 'sqlite://db.sqlite'
+world               = Dynflow::SimpleWorld.new persistence_adapter: persistence_adapter
+listener            = Dynflow::Listeners::Socket.new world, socket
+
+load File.join(root_path, 'test', 'code_workflow_example.rb')
+
+Dynflow::Daemon.new(listener, world).run