simpler_workflow 0.1.12 → 0.2.0

data/CHANGELOG.md

@@ -0,0 +1,50 @@
+0.2.0
+=====
+
+Switches from running activity and decision loops in a thread to running
+them as child process. This will simplify monitoring the different loops
+and provide insulation from the other loops.
+
+* Adds ```simpler_workflow:work``` rake task to locate, load and run the
+  activity and decision loops.
+* Adds the option to perform an action after forking a child process.
+* Adds a swf script used to manage the workflow. The following actions
+  are supported:
+  * *start* - Start the activity and decision loops located under
+    lib/workflow as a daemon
+  * *run* - runs the activity and decision loops without daemonizing
+    them.
+  * *stop* - stops the activity and decision loops.
+  * *pstree* - shows the child processes
+  * *shutdown* - shuts down the activity and decision loops
+  * *tail* - tails the logs associated with the workflow.
+* Preloads the Rails environment (if running under Rails) to reduce the
+  amount of time spent loading it after forking. May also reduce the
+  memory footprint of the application by sharing memory (on some
+  platform).
+
+Migrating to 0.2.0
+------------------
+
+The workflow definitions should stay the same. You can either choose to
+use the ```swf``` script to manage the workflows, build on top of the ```simpler_workflow:work``` 
+rake task or keep your existing scripts.
+
+You will need to add ```Process.waitall``` to prevent the parent
+process from exiting prematurely. 
+
+Using the ```SimplerWorkflow.after_fork``` method
+-------------------------------------------------
+
+You can use the ```SimplerWorkflow.after_form``` method to perform
+actions after the child processes have been forked. This is useful to
+re-establish ActiveRecord connections when using PostgreSQL for example.
+Here's an example of this usage:
+
+```ruby
+SimplerWorkflow.after_fork do
+  ActiveRecord::Base.establish_connection
+end
+```
+  
+It's usage is very similar to Resque's usage. 

data/README.md

@@ -1,6 +1,6 @@
 # SimplerWorkflow
 
-[![Build Status](https://secure.travis-ci.org/SnuggHome/simpler_workflow.png)](http://travis-ci.org/SnuggHome/simpler_workflow)
+[![Build Status](https://secure.travis-ci.org/fredjean/simpler_workflow.png)](http://travis-ci.org/fredjean/simpler_workflow)
 
 A wrapper around Amazon's Simple Workflow Service meant to simplify declaring and using activities and workflow. Provides some sane defaults
 and work around some idiosyncracies of the platform.
@@ -90,7 +90,9 @@ end
 my_activity.start_activity_loop
 ```
 
-The activity manages a loop that waits for messages from SWF.
+The activity manages a loop that waits for messages from SWF. It runs
+under it's own sub-process to make it easier to manage and isolate each
+of the activities.
 
 Activities are passed a task parameter. This parameter is provided to the activity by SWF and provides a lot of information about the task
 at end. One item passed is the input attribute.
@@ -150,6 +152,9 @@ The next step is to start the decision loop:
 my_workflow.decision_loop
 ```
 
+The decision loop will be launched in it's own sub-process and will
+proceed to poll SWF for decision tasks.
+
 #### Customizing the workflow
 
 There are hooks for different section of the decision loop. You can specify what happens when the workflow is started with the ```on_start_execution```
@@ -182,6 +187,16 @@ Here are a few recommendations on when to change the version of a workflow or ac
 2. You may want to bump a version if you have work in progress under an existing workflow and you need to introduce changes for new work. You will need to keep the older activity and or workflow around while it completes.
 3. You do not need to bump the version when you change the work performed by the activity or the decision loop itself. This is code that is directly managed by SimplerWorkflow and isn't communicated to AWS. This only works if you do not want previous workflows to finish using the previous version of the code though.
 
+## Running Workflows
+
+There is a new Rake task called ```simpler_workflow:work``` that will
+look for workflows located under the ```lib/workflow``` directory. This
+makes it much easier to put together a few workflow and manage them.
+
+Another addition in 0.2.0 is the swf command. This command provides a
+script that starts and stops the workflows and provide other monitoring
+tools.
+
 ## Contributing
 
 We welcome all kinds of contributions. This include code, fixes, issues, documentation, tests... Here's how you can contribute:

data/bin/swf

@@ -1,29 +1,27 @@
 #!/usr/bin/env ruby
 # encoding: utf-8
+# Mostly based on @ahoward's https://gist.github.com/3098500 with a few fixes and
+# adjustments.
 
 # setup
 #
   require 'fileutils'
 
+  pwd        = ENV['PWD']
   script     = File.expand_path(__FILE__).gsub(%r|\breleases/\d+\b|, 'current')
-  script_dir = File.dirname(script)
-  rails_root = File.dirname(script_dir)
+  rails_root = pwd.gsub(%r|\breleases/\d+\b|, 'current')
 
   Dir.chdir(rails_root)
 
-  require File.join(rails_root, 'config', 'env.rb')
-
-  FileUtils.mkdir_p(File.join(rails_root, 'log', 'jobs'))
-  FileUtils.touch(File.join(rails_root, 'log', 'jobs', 'pid'))
-
-  %w(
-    BACKGROUND PIDFILE QUEUE VVERBOSE INTERVAL COUNT
-  ).each{|key| ENV.delete(key)}
+  FileUtils.mkdir_p(File.join(rails_root, 'log', 'workflow'))
+  FileUtils.touch(File.join(rails_root, 'log', 'workflow', 'pid'))
 
   quiet   = ARGV.delete('-q') || ARGV.delete('--quiet')
   mode    = ARGV.shift || 'run'
 
-  pidfile = './log/jobs/pid'
+  pidfile = './log/workflow/pid'
+  lockfile = File.open('./log/workflow/lock', "w")
+
   pid     = Integer(IO.read(pidfile).strip) rescue nil
   running = begin; Process.kill(0, pid); true; rescue Object; false; end
 
@@ -34,15 +32,12 @@
       puts(pid) if running
 
     when 'run'
-      exit(42) unless DATA.flock(File::LOCK_EX | File::LOCK_NB)
+      exit(42) unless lockfile.flock(File::LOCK_EX | File::LOCK_NB)
 
-      ENV['PIDFILE']  = pidfile
-      ENV['QUEUE']    = 'jobs'
-      ENV['VVERBOSE'] = '1'
-      exec "rake environment resque:work"
+      exec "rake simpler_workflow:work"
 
     when 'start'
-      exit(42) unless DATA.flock(File::LOCK_EX | File::LOCK_NB)
+      exit(42) unless lockfile.flock(File::LOCK_EX | File::LOCK_NB)
 
       unless running
         FileUtils.rm_f(pidfile)
@@ -54,6 +49,7 @@
           pid = Integer(a.read.strip)
           a.close
           puts(pid) unless quiet
+          File.open(pidfile, "w") { |f| f.write(pid) }
           exit
         end
         exit!(0) if fork
@@ -66,7 +62,7 @@
           'stdout' => STDOUT,
           'stderr' => STDERR,
         }.each do |basename, io|
-          path = File.join("log/jobs/#{ basename }")
+          path = File.join("log/workflow/#{ basename }")
           begin
             open(path, 'a+'){|fd| io.reopen(fd)}
           rescue
@@ -77,18 +73,13 @@
         Process.setsid rescue nil
         File.umask(0) rescue nil
 
-# This is where Resque is started...
-# Looks like a good place to load and run a task that launches a workflow...
-        ENV['PIDFILE']  = pidfile
-        ENV['QUEUE']    = 'jobs'
-        ENV['VVERBOSE'] = '1' if ENV['RAILS_ENV'] != 'production'
-        exec "rake environment resque:work"
+        exec "rake simpler_workflow:work"
       end
 
     when 'stop'
       if running
         begin
-          Process.kill('-QUIT', pid)
+          Process.kill('QUIT', pid)
         rescue Errno::ESRCH
           nil
         end
@@ -135,10 +126,11 @@
       exec "#{ script } start -q >/dev/null 2>&1"
 
     when 'tail'
-      exec "tail -F ./log/jobs/*"
+      exec "tail -F ./log/workflow/*"
 
     when 'pstree'
       exec "pstree #{ pid }" if running
   end
-  
+
 __END__
+

data/lib/aws/simple_workflow/decision_task_additions.rb

@@ -3,10 +3,20 @@ require 'aws/simple_workflow/decision_task'
 module AWS
   class SimpleWorkflow
     module DecisionTaskAdditions
+      def self.extended(inst)
+        inst.class.__send__ :alias_method, :_original_events, :events
+      end
+
       def scheduled_event(event)
-        workflow_execution.events.reverse_order.find { |e| e.id == event.attributes.scheduled_event_id }
+        events.to_a[event.attributes.scheduled_event_id - 1]
+      end
+
+      def events
+        @__original_events ||= _original_events
       end
+
     end
+
   end
 end
 

data/lib/simpler_workflow/activity.rb

@@ -52,11 +52,8 @@ module SimplerWorkflow
       logger.info("Performing task #{name}")
       @perform_task.call(task)
     rescue => e
-      context = {}
-      context[:activity_type] = [name.to_s, version]
-      context[:input] = task.input
-      context[:activity_id] = task.activity_id
-      SimplerWorkflow.exception_reporter.report(e, context)
+      logger.error e.message
+      logger.error e.backtrace.join("\n")
       task.fail! :reason => e.message[0..250], :details => {:failure_policy => failure_policy}.to_json
     end
 
@@ -65,33 +62,50 @@ module SimplerWorkflow
     end
 
     def start_activity_loop
-      Thread.new do
+      SimplerWorkflow.child_processes << fork do
+
+        $0 = "Activity: #{name} #{version}"
+
+        Signal.trap('QUIT') do
+          logger.info("Received SIGQUIT")
+          @time_to_exit = true
+        end
+
+        Signal.trap('INT') do 
+          logger.info("Received SIGINT")
+          Process.exit!(0)
+        end
+
+
+        if SimplerWorkflow.after_fork
+          SimplerWorkflow.after_fork.call
+        end
+
         loop do
           begin
             logger.info("Starting activity_loop for #{name}")
-            domain.activity_tasks.poll(name.to_s) do |task|
-              begin
-                logger.info("Received task...")
-                perform_task(task)
-                unless task.responded?
-                  if next_activity
-                    result = {:next_activity => next_activity}.to_json
-                    task.complete!(:result => result)
-                  else
-                    task.complete!
-                  end
+            domain.activity_tasks.poll_for_single_task(name.to_s) do |task|
+              logger.info("Received task...")
+              perform_task(task)
+              unless task.responded?
+                if next_activity
+                  result = {:next_activity => next_activity}.to_json
+                  task.complete!(:result => result)
+                else
+                  task.complete!
                 end
-              rescue => e
-                context = {}
-                context[:activity_type] = [name.to_s, version]
-                context[:input] = task.input
-                context[:activity_id] = task.activity_id
-                SimplerWorkflow.exception_reporter.report(e, context)
-                task.fail! :reason => e.message, :details => { :failure_policy => :abort }.to_json unless task.responded?
               end
             end
-          rescue Timeout::Error => e
-            retry
+            Process.exit(0) if @time_to_exit
+          rescue Timeout::Error
+            if @time_to_exit
+              Process.exit(0)
+            else
+              retry
+            end
+          rescue => e
+            logger.error(e.message)
+            logger.error(e.backtrace.join("\n"))
           end
         end
       end

data/lib/simpler_workflow/tasks.rb

@@ -0,0 +1,47 @@
+# require 'simpler_workflow/tasks'
+# will give you the simpler_workflow tasks.
+# Much inspiration was derived from https://github.com/defunkt/resque
+
+namespace :simpler_workflow do
+  task :setup
+
+  desc "Runs the workflows."
+  task :work => [:preload, :setup] do
+    require 'simpler_workflow'
+
+    Signal.trap('QUIT') do
+      SimplerWorkflow.child_processes.each do |child|
+        Process.kill('QUIT', child)
+      end
+      exit(0)
+    end
+
+    Signal.trap('INT') do
+      SimplerWorkflow.child_processes.each do |child|
+        Process.kill('INT', child)
+      end
+      exit(0)
+    end
+
+    pattern = ENV['WORKFLOW'] || 'lib/workflow/*.rb'
+
+    FileList.new(pattern).each do |f|
+      load f
+    end
+
+    Process.waitall
+  end
+
+  desc "Preloads the Rails environment if this is running under Rails."
+  task :preload => :setup do
+    if defined?(Rails) && Rails.respond_to?(:application)
+      # Rails 3
+      Rails.application.eager_load!
+    elsif defined?(Rails::Initializer)
+      # Rails 2.3
+      $rails_rake_task = false
+      Rails::Initializer.run :load_application_classes
+    end
+  end
+end
+

data/lib/simpler_workflow/version.rb

@@ -1,3 +1,3 @@
 module SimplerWorkflow
-  VERSION = "0.1.12"
+  VERSION = "0.2.0"
 end

data/lib/simpler_workflow/workflow.rb

@@ -29,37 +29,56 @@ module SimplerWorkflow
     end
 
     def decision_loop
-      logger.info("Starting decision loop for #{name.to_s}, #{version} listening to #{task_list}")
-      domain.decision_tasks.poll(task_list) do |decision_task|
-        begin
-          start_time = DateTime.now
-          logger.info("Received decision task #{decision_task.id} at #{start_time}")
-          decision_task.extend AWS::SimpleWorkflow::DecisionTaskAdditions
-          decision_task.new_events.each do |event|
-            logger.info("Processing #{event.event_type}")
-            case event.event_type
-            when 'WorkflowExecutionStarted'
-              start_execution(decision_task, event)
-            when 'ActivityTaskCompleted'
-              activity_completed(decision_task, event)
-            when 'ActivityTaskFailed'
-              activity_failed(decision_task, event)
-            when 'ActivityTaskTimedOut'
-              activity_timed_out(decision_task, event)
+      SimplerWorkflow.child_processes << fork do
+
+        $0 = "Workflow: #{name} #{version}"
+
+        Signal.trap('QUIT') do
+          logger.info("Received SIGQUIT")
+          @time_to_exit = true
+        end
+
+        Signal.trap('INT') do 
+          logger.info("Received SIGINT")
+          Process.exit!(0)
+        end
+
+        if SimplerWorkflow.after_fork
+          SimplerWorkflow.after_fork.call
+        end
+
+
+        loop do
+          begin
+            logger.info("Waiting for a decision task for #{name.to_s}, #{version} listening to #{task_list}")
+            domain.decision_tasks.poll_for_single_task(task_list) do |decision_task|
+              decision_task.extend AWS::SimpleWorkflow::DecisionTaskAdditions
+              logger.info("Received decision task")
+              decision_task.new_events.each do |event|
+                logger.info("Processing #{event.event_type}")
+                case event.event_type
+                when 'WorkflowExecutionStarted'
+                  start_execution(decision_task, event)
+                when 'ActivityTaskCompleted'
+                  activity_completed(decision_task, event)
+                when 'ActivityTaskFailed'
+                  activity_failed(decision_task, event)
+                when 'ActivityTaskTimedOut'
+                  activity_timed_out(decision_task, event)
+                end
+              end
+            end
+            Process.exit 0 if @time_to_exit
+          rescue Timeout::Error => e
+            if @time_to_exit
+              Process.exit 0
+            else
+              retry
             end
           end
-        rescue => e
-          context = {
-            :workflow_execution => decision_task.workflow_execution,
-            :workflow => to_workflow_type,
-            :decision_task => decision_task
-          }
-          SimplerWorkflow.exception_reporter.report(e, context)
-          raise e
+          nil
         end
       end
-    rescue Timeout::Error => e
-      retry
     end
 
     def task_list
@@ -126,10 +145,6 @@ module SimplerWorkflow
       end
     end
 
-    def to_workflow_type
-      { :name => name, :version => version }
-    end
-
     def start_workflow(input, options = {})
       options[:input] = input
       domain.workflow_types[name.to_s, version].start_execution(options)

data/lib/simpler_workflow.rb

@@ -2,32 +2,40 @@ require 'aws-sdk'
 require 'aws/simple_workflow/decision_task_additions'
 require 'map'
 
+require 'simpler_workflow/tasks' if defined?(Rake)
+
 module SimplerWorkflow
   extend self
 
+  # Provides a handle to a domain.
   def domain(domain_name)
     @domains ||= {}
     @domains[domain_name.to_sym] ||= Domain.new(domain_name)
   end
 
+  # Provides a handle to the SimpleWorkflow underlying service.
   def swf
     @swf ||= ::AWS::SimpleWorkflow.new
   end
 
+  # The logger used. Falls back to the Rails logger.
   def logger
     $logger || Rails.logger
   end
 
-  def exception_reporter(&block)
-    if block_given?
-      @exception_reporter = DefaultExceptionReporter.new(&block)
-    end
-
-    @exception_reporter || DefaultExceptionReporter.new
+  # Sets the code to be called after a process fork when a block is provided.
+  # Returns the previously set block (or nil) otherwise.
+  #
+  # @param block The block that will be called after a process is forked.
+  # @return Proc the block that was passed earlier (or nil)
+  def after_fork(&block)
+    block ? (@after_fork = block) : @after_fork
   end
+  attr_writer :after_fork
 
-  def exception_reporter=(exception_handler)
-    @exception_reporter = exception_handler
+  # The list of child processes that have been forked from the main process.
+  def child_processes
+    @child_processes ||= []
   end
 
   autoload :Version,  'simpler_workflow/version'
@@ -35,7 +43,6 @@ module SimplerWorkflow
   autoload :Workflow, 'simpler_workflow/workflow'
   autoload :Activity, 'simpler_workflow/activity'
   autoload :OptionsAsMethods, 'simpler_workflow/options_as_methods'
-  autoload :DefaultExceptionReporter, 'simpler_workflow/default_exception_reporter'
 end
 
 class Map

data/lib/tasks/simpler_workflow.rake

@@ -0,0 +1,2 @@
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../../lib'
+require 'simpler_workflow/tasks'

data/simpler_workflow.gemspec

@@ -6,7 +6,19 @@ Gem::Specification.new do |gem|
   gem.email         = ["fred@snugghome.com"]
   gem.description   = %q{A wrapper around Amazon's Simple Workflow Service}
   gem.summary       = %q{A wrapper and DSL around Amazon's Simple Workflow Service with the goal of making it almost pleasant to define workflows.}
-  gem.homepage      = "https://github.com/SnuggHome/simpler_workflow"
+  gem.homepage      = "https://github.com/fredjean/simpler_workflow"
+  gem.post_install_message =<<EOM
+simpler_workflow #{SimplerWorkflow::VERSION}
+========================
+
+Have a look at https://github.com/fredjean/simpler_workflow/wiki/MIgrating-to-0.2.0 if you
+are upgrading from a 0.1.x version of the gem. There is a fundamental change in how the 
+activity and decision loops are run. You may need to adjust your application for this to work.
+
+Thank you for installing simpler_workflow. I hope you find it useful.
+
+-- Fred
+EOM
 
   gem.files         = `git ls-files`.split($\)
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: simpler_workflow
 version: !ruby/object:Gem::Version
-  version: 0.1.12
+  version: 0.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-10-03 00:00:00.000000000 Z
+date: 2012-08-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk
@@ -102,6 +102,7 @@ files:
 - .gitignore
 - .rspec
 - .travis.yml
+- CHANGELOG.md
 - Gemfile
 - LICENSE
 - README.md
@@ -111,19 +112,25 @@ files:
 - lib/aws/simple_workflow/decision_task_additions.rb
 - lib/simpler_workflow.rb
 - lib/simpler_workflow/activity.rb
-- lib/simpler_workflow/default_exception_reporter.rb
 - lib/simpler_workflow/domain.rb
 - lib/simpler_workflow/options_as_methods.rb
+- lib/simpler_workflow/tasks.rb
 - lib/simpler_workflow/version.rb
 - lib/simpler_workflow/workflow.rb
 - lib/simpler_workflow/workflow_collection.rb
+- lib/tasks/simpler_workflow.rake
 - simpler_workflow.gemspec
 - spec/domain_spec.rb
 - spec/simpler_workflow_spec.rb
 - spec/spec_helper.rb
-homepage: https://github.com/SnuggHome/simpler_workflow
+homepage: https://github.com/fredjean/simpler_workflow
 licenses: []
-post_install_message: 
+post_install_message: ! "simpler_workflow 0.2.0\n========================\n\nHave
+  a look at https://github.com/fredjean/simpler_workflow/wiki/MIgrating-to-0.2.0 if
+  you\nare upgrading from a 0.1.x version of the gem. There is a fundamental change
+  in how the \nactivity and decision loops are run. You may need to adjust your application
+  for this to work.\n\nThank you for installing simpler_workflow. I hope you find
+  it useful.\n\n-- Fred\n"
 rdoc_options: []
 require_paths:
 - lib
@@ -133,12 +140,18 @@ required_ruby_version: !ruby/object:Gem::Requirement
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -1625844284378708087
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
   - - ! '>='
     - !ruby/object:Gem::Version
       version: '0'
+      segments:
+      - 0
+      hash: -1625844284378708087
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23

data/lib/simpler_workflow/default_exception_reporter.rb

@@ -1,26 +0,0 @@
-# Default exception handler. Just logs to the logger and re-raise
-# so the exception can be managed as usual.
-
-module SimplerWorkflow
-  class DefaultExceptionReporter
-    attr_accessor :reporter, :tag
-
-    def initialize(&block)
-      @reporter = block if block_given?
-    end
-
-    def report(e, context = {})
-      if reporter
-        reporter.call(e, context)
-      else
-        SimplerWorkflow.logger.error("[#{tag}] Exception: #{e.message}")
-        SimplerWorkflow.logger.error("[#{tag}] Context: #{context.inspect}") unless context.empty?
-        SimplerWorkflow.logger.error("[#{tag}] Backtrace:\n#{e.backtrace.join("\n")}")
-      end
-    end
-
-    def tag
-      @tag || "SimplerWorkflow"
-    end
-  end
-end