orchestrated 0.0.5 → 0.0.6

data/README.markdown

@@ -1,17 +1,32 @@
 Orchestrated
 ============
 
-The [delayed_job](https://github.com/collectiveidea/delayed_job) Ruby Gem provides a job queuing system for Ruby. It implements an elegant API for delaying execution of any object method. Not only is the execution of the method (message delivery) delayed in time, it is potentially shifted in space too. By shifting in space, i.e. running in a separate virtual machine, possibly on a separate computer, multiple CPUs can be brought to bear on a computing problem.
+The [delayed_job](https://github.com/collectiveidea/delayed_job) Ruby Gem provides a restartable queuing system for Ruby. It implements an elegant API for delaying execution of any Ruby object method invocation. Not only is the message delivery delayed in time, it is potentially shifted in space too. By shifting in space, i.e. running in a different virtual machine, possibly on a separate computer, multiple CPUs can be brought to bear on a computing problem.
 
 By breaking up otherwise serial execution into multiple queued jobs, a program can be made more scalable. This sort of distributed queue-processing architecture has a long and successful history in data processing.
 
-Queuing works well for simple tasks. By simple we mean the task can be done all at once, in one piece. It has no dependencies on other tasks. This works well for performing a file upload task in the background (to avoid tying up a Ruby virtual machine process/thread). More complex (compound) multi-part tasks, however, do not fit this model. Examples of complex (compound) tasks include:
+Queuing works well for simple, independent tasks. By simple we mean the task can be done all at once, in one piece, with no inter-task dependencies. This works well for performing a file upload task in the background (to avoid tying up a Ruby virtual machine process/thread). More complex (compound) multi-part tasks, however, do not fit this model. Examples of complex (compound) tasks include:
 
-1. pipelined (multi-step) generation of complex PDF documents
-2. extract/transfer/load (ETL) jobs that may load thousands of database records
+1. pipelined (multi-step) generation of a complex PDF document
+2. an extract/transfer/load (ETL) job that must acquire data from source systems, transform it and load it into the target system
 
 If we would like to scale these compound operations, breaking them into smaller parts, and managing the execution of those parts across many computers, we need an "orchestrator". This project implements just such a framework, called "Orchestrated".
 
+Orchestrated introduces the ```acts_as_orchestrated``` Object class method. When invoked on your class, this will define the ```orchestrated``` instance method. You use ```orchestrated``` in a mannner similar to [delayed_job](https://github.com/collectiveidea/delayed_job)'s ```delay```—the difference being that ```orchestrated``` takes a parameter that lets you specify dependencies between your jobs.
+
+The reason we refer to [delayed_job](https://github.com/collectiveidea/delayed_job) as a restartable queueing system is because, even if computers (database host, worker hosts) in the cluster crash, the work on the queues progresses. If no worker is servicing a particular queue, then work accumulates there. Once workers are available, they consume the jobs. This is a resilient architecture.
+
+With Orchestrated you can create restartable workflows, a workflow consisting of one or more dependent, queueable, tasks. This means that your workflows will continue to make progress even in the face of database and (queue) worker crashes.
+
+In summary, orchestrated workflows running atop [active_record](https://github.com/rails/rails/tree/master/activerecord) and [delayed_job](https://github.com/collectiveidea/delayed_job) have these characteristics:
+
+1. restartable—the workflows make progress even though (queue worker, and database) hosts are not always available
+2. scalable—compound workflows are broken into steps which can be executed on separate computers. Results can be accumulated from the disparate steps as needed.
+3. forgiving of external system failures—workflow steps that communicate with an external system can simply throw an exception when the system is unavailable, assured that the step will be automatically retried again later.
+4. composable—compound tasks can be defined in terms of simpler ones
+
+Read on to get started.
+
 Installation
 ------------
 
@@ -37,7 +52,13 @@ If you do not already have [delayed_job](https://github.com/collectiveidea/delay
 The API
 -------
 
-To orchestrate (methods) on your own classes you simply call ```acts_as_orchestrated``` in the class definition like this:
+To orchestrate (methods) on your own classes you simply call ```acts_as_orchestrated``` in the class definition. Declaring ```acts_as_orchestrated``` on your class defines the ```orchestrated``` method:
+
+* ```orchestrated```—call this to specify your workflow prerequisite, and designate a workflow step
+
+Use ```orchestrated``` to orchestrate any method on your class.
+
+Let's say for example you needed to download a couple files from remote systems (a slow process), merge their content and then load the results into your system. Imagine you have a ```Downloader``` class that knows how to download and an ```Xform``` class that knows how to merge the content and load the results into your system. Your ```Xform``` class might look something like this:
 
 ```ruby
 class Xform
@@ -55,12 +76,7 @@ class Xform
 end
 ```
 
-Declaring ```acts_as_orchestrated``` on your class gives it two methods:
-
-* ```orchestrated```—call this to specify your workflow prerequisite, and designate a workflow step
-* ```orchestration```—call this in the context of a workflow step (execution) to access orchestration (and prerequisite) context
-
-After that you can orchestrate any method on such a class. Let's say you needed to download a couple files from remote systems (a slow process), merge their content and then load the results into your system. Imagine you had had a Downloader class that knew how to download and an Xform class that knew how to merge the content and load the results into your system. You might write an orchestration like this:
+You might write an orchestration like this:
 
 ```ruby
 xform = Xform.new
@@ -78,15 +94,16 @@ xform.orchestrated(
 ).load('combined_records')
 ```
 
-The next time you process delayed jobs, the :download messages will be delivered to a couple Downloaders. After the last download completes, the next time a delayed job is processed, the :merge message will be delivered to an Xform object. And on it goes…
+The next time you process delayed jobs, the ```download``` messages will be delivered to a couple Downloaders. After the last download completes, the next time a delayed job is processed, the ```merge``` message will be delivered to an Xform object. And on it goes…
 
 What happened there? The pattern is:
 
 1. create an orchestrated object (instantiate it)
-2. call orchestrated on it: this returns an "orchestration"
-3. send a message to the orchestration (returned in the second step)
+2. call ```orchestrated``` on it: this returns a *magic proxy* object that can respond to any of the messages your object can respond to
+3. send any message to the *magic proxy* (returned in the second step) and the framework will delay delivery of that message and immediately return a "completion expression" you can use as a prerequisite for other orchestrations
+4. (optionally) use the "completion expression" returned in (3) as a prerequisite for other orchestrations
 
-Now the messages you can send in (3) are limited to the messages that your object can respond to. The message will be remembered by the framework and "replayed" (on a new instance of your object) somewhere on the network (later).
+Now the messages you can send in (3) can be anything that your object can respond to. The message will be remembered by the framework and "replayed" (on a new instance of your object) somewhere on the network (later).
 
 Not accidentally, this is similar to the way [delayed_job](https://github.com/collectiveidea/delayed_job)'s delay method works. Under the covers, orchestrated is conspiring with [delayed_job](https://github.com/collectiveidea/delayed_job) when it comes time to actually execute a workflow step. Before that time though, orchestrated keeps track of everything.
 
@@ -95,16 +112,15 @@ Key Concept: Prerequisites (Completion Expressions)
 
 Unlike [delayed_job](https://github.com/collectiveidea/delayed_job) ```delay```, the orchestrated ```orchestrated``` method takes an optional parameter: the prerequisite. The prerequisite determines when your workflow step is ready to run.
 
-The return value from "orchestrate" is itself a ready-to-use prerequisite. You saw this in the statement generation example above. The result of the first ```orchestrated``` call was sent as an argument to the second. In this way, the second workflow step was suspended until after the first one finished. You may have also noticed from that example that if you specify no prerequisite then the step will be ready to run immediately, as was the case for the "generate" call).
+The return value from messaging the *magic proxy* is itself a ready-to-use prerequisite. You saw this in the statement generation example above. The result of the first call to ```orchestrated``` calls (to "download") were sent as an argument to the third ("merge"). In this way, the "merge" workflow step was suspended until after the "download"s finished. You may have also noticed from that example that if you specify no prerequisite then the step will be ready to run immediately, as was the case for the "download" calls).
 
-There are five kinds of prerequisite in all. Some of them are used for combining others. The prerequisites types, also known as "completion expressions" are:
+Users of the framework deal directly with three kinds of prerequisite or "completion expression":
 
-1. ```OrchestrationCompletion```—returned by "orchestrate", complete when its associated orchestration is complete
-2. ```Complete```—always complete
-3. ```FirstCompletion```—aggregates other completions: complete after the first one completes
-4. ```LastCompletion```—aggregates other completions: complete after all of them are complete
+1. ```OrchestrationCompletion```—returned messages to the *magic proxy*, complete when its associated orchestration is complete
+2. ```FirstCompletion```—aggregates other completions: complete after the first one completes
+3. ```LastCompletion```—aggregates other completions: complete after all of them are complete
 
-See the completion_spec for examples of how to combine these different prerequisite types into completion expressions.
+There are other kinds of completion expression used internally by the framework but these three are the important ones for users to understand. See the completion_spec for examples of how to combine these different prerequisite types into completion expressions.
 
 Key Concept: Orchestration State
 --------------------------------
@@ -115,15 +131,15 @@ An orchestration can be in one of a few states:
 
 When you create a new orchestration that is waiting on a prerequisite that is not complete yet, the orchestration will be in the "waiting" state. Some time later, if that prerequisite completes, then your orchestration will become "ready". A "ready" orchestration is automatically queued to run by the framework (via [delayed_job](https://github.com/collectiveidea/delayed_job)).
 
-A "ready" orchestration will use [delayed_job](https://github.com/collectiveidea/delayed_job) to delivery its (delayed) message. In the context of such a message delivery (inside your object method e.g. StatementGenerator#generate or StatementGenerator#render) you can rely on the ability to access the current Orchestration (context) object via the "orchestration" accessor.
+A "ready" orchestration will use [delayed_job](https://github.com/collectiveidea/delayed_job) to deliver its (delayed) message. In the context of such a message delivery (inside your object method e.g. ```Xform#merge``` or ```Xform#load``` in our example) you can rely on the ability to access the current Orchestration (context) object via the ```orchestration``` accessor. Be careful with that one though. You really shouldn't need it very often, and to use it, you have to understand framework internals.
 
 After your workflow step executes, the orchestration moves into either the "succeeded" or "failed" state.
 
-When an orchestration is "ready" or "waiting" it may be canceled by sending it the ```cancel!``` message. This moves it to the "canceled" state and prevents delivery of the orchestrated message (in the future).
+When an orchestration is "ready" or "waiting" it may be canceled by sending it the ```cancel!``` message (i.e. a ```cancel!``` message to the ```OrchestrationCompletion```. This moves it to the orchestration to the "canceled" state and prevents delivery of the orchestrated message (in the future).
 
 It is important to understand that both of the states: "succeeded" and "failed" are part of a "super-state": "complete". When an orchestration is in either of those two states, it will return ```true``` in response to the ```complete?``` message.
 
-It is not just successful completion of orchestrated methods that causes dependent ones to run—a "failed" orchestration is complete too! If you have an orchestration that actually requires successful completion of its prerequisite then it can inspect the prerequisite as needed. It's accessible through the ```orchestration`` accessor (on the orchestrated object).
+It is not just successful completion of orchestrated methods that causes dependent ones to run—a "failed" orchestration is complete too! If you have an orchestration that actually requires successful completion of its prerequisite then your method can inspect the prerequisite as needed, by accessing it via ```self.orchestration.prerequisite.prerequisite```.
 
 Failure (An Option)
 -------------------

data/lib/orchestrated/base.rb

@@ -7,11 +7,7 @@ module Orchestrated
     end
     def method_missing(sym, *args)
       raise 'cannot orchestrate with blocks because they are not portable across processes' if block_given?
-      OrchestrationCompletion.new do |completion|
-        completion.orchestration = Orchestration.create( @target, sym, args, @prerequisite)
-      end.tap do |completion|
-        completion.save!
-      end
+      Orchestration.create( @target, sym, args, @prerequisite)
     end
   end
 

data/lib/orchestrated/completion.rb

@@ -44,10 +44,10 @@ module Orchestrated
     def +(c); self << c; end # synonymc
   end
   class LastCompletion < CompositeCompletion
-    def complete?; prerequisites.all?(&:complete?); end
-    def always_complete?; prerequisites.empty?; end
+    def complete?; prerequisite_associations.all?(&:complete?); end
+    def always_complete?; prerequisite_associations.empty?; end
     def never_complete?; prerequisites.any?(&:never_complete?); end
-    def canceled?; prerequisites.any?(&:canceled?); end
+    def canceled?; prerequisite_associations.any?(&:canceled?); end
     def <<(c)
       prerequisites << c unless c.always_complete?
       self
@@ -60,10 +60,10 @@ module Orchestrated
     end
   end
   class FirstCompletion < CompositeCompletion
-    def complete?; prerequisites.any?(&:complete?); end
+    def complete?; prerequisite_associations.any?(&:complete?); end
     def always_complete?; prerequisites.any?(&:always_complete?); end
-    def never_complete?; prerequisites.empty?; end
-    def canceled?; prerequisites.all?(&:canceled?); end
+    def never_complete?; prerequisite_associations.empty?; end
+    def canceled?; prerequisite_associations.all?(&:canceled?); end
     def <<(c)
       prerequisites << c unless c.never_complete?
       self

data/lib/orchestrated/dependency.rb

@@ -4,9 +4,12 @@ module Orchestrated
   class OrchestrationDependency < ActiveRecord::Base
     # TODO: figure out why Rails thinks I'm mass-assigning this over in Orchestration when I'm not really!
     attr_accessible :prerequisite_id
-    belongs_to :dependent, :class_name => 'CompletionExpression'
     belongs_to :prerequisite, :class_name => 'CompletionExpression'
-    state_machine :initial => :incomplete do
+    belongs_to :dependent, :class_name => 'CompletionExpression'
+
+    before_validation :constrain
+
+    state_machine :initial => :incomplete, :action => :save_avoiding_recursion do
       state :incomplete
       state :complete
       state :canceled
@@ -17,10 +20,50 @@ module Orchestrated
         transition :incomplete => :canceled
       end
       after_transition any => :complete do |dependency, transition|
-        dependency.dependent.prerequisite_complete
+        dependency.call_dependent{|d| d.prerequisite_complete}
       end
       after_transition any => :canceled do |dependency, transition|
-        dependency.dependent.prerequisite_canceled
+        dependency.call_dependent{|d| d.prerequisite_canceled}
+      end
+    end
+    def call_dependent(&block)
+      yield dependent unless dependent.nil?
+    end
+    def constrain
+      @saving = true
+      _constrain.tap{@saving = false}
+    end
+    def _constrain
+      if prerequisite.present?
+        if prerequisite_id_changed? || new_record?
+          # this may be our first prerequisite, or our prerequisite may
+          # have changed—either way we must initialize our state
+
+          # This method can be called more than once in general since it is called
+          # as part of validation. Rather than loosening the state machine (to allow
+          # e.g. complete=>complete) we explicitly avoid re-submitting events here.
+          prerequisite_completed if prerequisite.complete? && can_prerequisite_completed?
+          prerequisite_canceled if prerequisite.canceled? && can_prerequisite_canceled?
+        else
+          # prerequisite has not changed so our state is already correct
+          if dependent_id_changed?
+            # dependent has been set for the first time—propigate state
+            call_dependent{|d| d.prerequisite_complete} if prerequisite.complete?
+            call_dependent{|d| d.prerequisite_canceled} if prerequisite.canceled?
+          end
+        end
+      end
+      true
+    end
+    def save_avoiding_recursion
+      # Default action of state_machine is "save", however that is
+      # a problem when we need to transition state during validation
+      # (see constrain method above). If were are validating then we
+      # dursnt call save again.
+      if @saving
+        true # allow state transition but don't save ActiveRecord
+      else
+        save # save ActiveRecord as usual and return true/false
       end
     end
   end

data/lib/orchestrated/orchestration.rb

@@ -73,26 +73,23 @@ module Orchestrated
 
     end
 
+    # Actually creates a completion (wrapper). Not _exactly_ an orchestration—ssh…
     def self.create( value, sym, args, prerequisite)
-      # set prerequisite in new call so it is passed to state_machine :initial proc
-      new.tap do |orchestration|
-
-        orchestration.handler = Handler.new( value, sym, args)
-
-        # wee! static analysis FTW!
-        raise 'prerequisite can never be complete' if prerequisite.never_complete?
-
-        prerequisite.save!
-        orchestration.save!
-        interest = OrchestrationInterest.new.tap do |interest|
-          interest.prerequisite = prerequisite
-          interest.orchestration = orchestration
-        end
-        interest.save!
-
-        # prime the pump for a constant prerequisite
-        orchestration.prerequisite_complete! if prerequisite.complete?
-     end
+      # wee! static analysis FTW!
+      raise 'prerequisite can never be complete' if prerequisite.never_complete?
+      prerequisite.save!
+      OrchestrationCompletion.new.tap do |completion|
+        completion.orchestration = new.tap do |orchestration|
+          orchestration.handler = Handler.new( value, sym, args)
+          orchestration.save!
+          interest = OrchestrationInterest.new.tap do |interest|
+            interest.prerequisite = prerequisite
+            interest.orchestration = orchestration
+            interest.save!
+          end # interest
+        end # orchestration
+        completion.save!
+      end # completion
     end
 
     def enqueue

data/lib/orchestrated/version.rb

@@ -1,3 +1,3 @@
 module Orchestrated
-  VERSION = "0.0.5"
+  VERSION = "0.0.6"
 end

data/orchestrated.gemspec

@@ -18,19 +18,19 @@ Gem::Specification.new do |gem|
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
 
-  gem.add_runtime_dependency     'delayed_job_active_record', '>= 0.3'
-  gem.add_runtime_dependency     'activerecord', ['~> 3']
-  gem.add_runtime_dependency     'state_machine', ['>= 1']
+  gem.add_runtime_dependency     'delayed_job_active_record', '~> 0.3.3'
+  gem.add_runtime_dependency     'activerecord', ['~> 3.2']
+  gem.add_runtime_dependency     'state_machine', ['~> 1.1.2']
 
   gem.add_development_dependency 'rake', ['>= 10']
-  gem.add_development_dependency 'rails', ['~> 3'] # for rspec-rails
-  gem.add_development_dependency 'rspec-rails', ['>= 2.12']
+  gem.add_development_dependency 'rails', ['~> 3.2'] # for rspec-rails
+  gem.add_development_dependency 'rspec-rails', ['~> 2.12']
   # I couldn't get rspecs transactional fixtures setting to do savepoints
   # in this project (which is not _really_ a Rails app). database_cleaner
   # claims it'll help us clean up the database so let's try it!
-  gem.add_development_dependency 'database_cleaner', ['>= 0.9']
-  gem.add_development_dependency 'sqlite3', ['>= 1.3.6']
+  gem.add_development_dependency 'database_cleaner', ['~> 0.9']
+  gem.add_development_dependency 'sqlite3', ['~> 1.3']
   gem.add_development_dependency 'debugger'
   # The state_machine:draw rake task needs this
-  gem.add_development_dependency 'ruby-graphviz', ['>= 0.9.0']
+  gem.add_development_dependency 'ruby-graphviz', ['~> 0.9']
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: orchestrated
 version: !ruby/object:Gem::Version
-  version: 0.0.5
+  version: 0.0.6
   prerelease: 
 platform: ruby
 authors:
@@ -9,24 +9,24 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-01-04 00:00:00.000000000 Z
+date: 2013-01-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: delayed_job_active_record
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: 0.3.3
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: 0.3.3
 - !ruby/object:Gem::Dependency
   name: activerecord
   requirement: !ruby/object:Gem::Requirement
@@ -34,7 +34,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '3'
+        version: '3.2'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -42,23 +42,23 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '3'
+        version: '3.2'
 - !ruby/object:Gem::Dependency
   name: state_machine
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '1'
+        version: 1.1.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '1'
+        version: 1.1.2
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -82,7 +82,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '3'
+        version: '3.2'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -90,13 +90,13 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '3'
+        version: '3.2'
 - !ruby/object:Gem::Dependency
   name: rspec-rails
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
         version: '2.12'
   type: :development
@@ -104,7 +104,7 @@ dependencies:
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
         version: '2.12'
 - !ruby/object:Gem::Dependency
@@ -112,7 +112,7 @@ dependencies:
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
         version: '0.9'
   type: :development
@@ -120,7 +120,7 @@ dependencies:
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
         version: '0.9'
 - !ruby/object:Gem::Dependency
@@ -128,17 +128,17 @@ dependencies:
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 1.3.6
+        version: '1.3'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 1.3.6
+        version: '1.3'
 - !ruby/object:Gem::Dependency
   name: debugger
   requirement: !ruby/object:Gem::Requirement
@@ -160,17 +160,17 @@ dependencies:
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 0.9.0
+        version: '0.9'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 0.9.0
+        version: '0.9'
 description: a workflow orchestration framework running on delayed_job and active_record
 email:
 - bill@paydici.com