ntswf 1.0.0

data/LICENSE

@@ -0,0 +1,165 @@
+GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

data/README.md

@@ -0,0 +1,68 @@
+NTSWF
+=====
+
+Not That Simple Workflow - A layer around AWS SWF
+
+Common denominator of our [infopark](http://infopark.com/) internal services using
+AWS Simple Workflow.
+
+Usage
+-----
+### Gemfile
+
+    gem 'ntswf', '~> 1.0'
+
+### Client
+```
+class WorkflowClient
+  include Ntswf::Client
+
+  def enqueue!
+    start_execution(
+      execution_id: 'my_singleton_task',
+      name: 'my_worker_name',
+      params: {my: :param},
+      unit: 'my_worker',
+    )
+  end
+end
+
+config = {domain: 'my_domain', unit: 'my_app'} # ...
+WorkflowClient.new(config).enqueue!
+```
+See {Ntswf::Base#initialize} for configuration options.
+
+### Decision worker
+```
+class Decider
+  include Ntswf::DecisionWorker
+end
+
+config = {domain: 'my_domain', unit: 'my_app'} # ...
+loop { Decider.new(config).process_decision_task }
+```
+
+### Activity worker
+```
+class Worker
+  include Ntswf::ActivityWorker
+
+  def process_activity_task
+    super do |task|
+      options = parse_input(task.input)
+      # ...
+      task.complete!(result: 'OK')
+    end
+  end
+end
+
+config = {domain: 'my_domain', unit: 'my_worker'} # ...
+loop { Worker.new(config).process_activity_task }
+```
+
+### Setup helpers
+See {Ntswf::Utils}
+
+License
+-------
+[LPGLv3](http://www.gnu.org/licenses/lgpl-3.0.html)

data/lib/ntswf/activity_worker.rb

@@ -0,0 +1,29 @@
+require_relative 'worker'
+
+module Ntswf
+  # Interface for a worker executing tasks
+  module ActivityWorker
+    include Ntswf::Worker
+
+    def process_activity_task
+      announce("polling for activity task #{activity_task_list}")
+      domain.activity_tasks.poll_for_single_task(activity_task_list) do |task|
+        announce("got activity task #{task.activity_type.inspect} #{task.input}")
+        begin
+          if task.activity_type == activity_type
+            yield task
+          else
+            raise "unknown activity type: #{task.activity_type.inspect}"
+          end
+        rescue => e
+          notify(e, activity_type: task.activity_type.inspect, input: task.input)
+          details = {
+            error: e.message[0, 1000],
+            exception: e.class.to_s[0, 1000],
+          }
+          task.fail!(details: details.to_json, reason: 'Exception')
+        end
+      end
+    end
+  end
+end

data/lib/ntswf/base.rb

@@ -0,0 +1,93 @@
+require 'aws'
+
+module Ntswf
+  module Base
+    # @param config [Hash] A configuration with the following keys:
+    # @option config [String] :access_key_id AWS credential
+    # @option config [Hash] :activity_task_lists The task list names for activities as hash (see also *:unit*)
+    # @option config [String] :decision_task_list The task list name for decisions
+    # @option config [String] :domain The SWF domain name
+    # @option config [Numeric] :execution_version Value allowing clients to reject future execution versions
+    # @option config [String] :secret_access_key AWS credential
+    # @option config [String] :unit This worker/client's activity task list key
+    def initialize(config)
+      @config = config.with_indifferent_access
+      raise_if_invalid_task_list
+    end
+
+    # @return [AWS::SimpleWorkflow]
+    def swf
+      @swf ||= AWS::SimpleWorkflow.new(@config.slice(:access_key_id, :secret_access_key).merge(
+          use_ssl: true))
+    end
+
+    def workflow_name
+      "#{default_unit}-workflow"
+    end
+
+    def workflow_version
+      "v1"
+    end
+
+    def domain
+      @domain ||= swf.domains[@config[:domain]]
+    end
+
+    def activity_task_lists
+      @config[:activity_task_lists]
+    end
+
+    def decision_task_list
+      @config[:decision_task_list] || "#{default_unit}-decision-task-list"
+    end
+
+    def activity_task_list
+      activity_task_lists[default_unit] || "#{default_unit}-activity-task-list"
+    end
+
+    def default_unit
+      @default_unit ||= @config[:unit].to_s
+    end
+
+    def execution_version
+      @config[:execution_version]
+    end
+
+    # Parse the options stored in a task's *input* value
+    # @param input [String] A task's input
+    # @return [Hash] Converted input string
+    # @see Ntswf::Client#start_execution Hash keys to be expected
+    def parse_input(input)
+      options, legacy_params = JSON.parse(input)
+      options = {name: options} unless options.kind_of? Hash
+      options.merge!(params: legacy_params) if legacy_params
+      options.with_indifferent_access
+    end
+
+    def notify(message, params)
+      log("#{message.message}\n  #{message.backtrace.join("\n  ")}") if message.kind_of? Exception
+    end
+
+    protected
+
+    def announce(s)
+      $0 = s
+      log(s)
+    end
+
+    def log(s)
+      $stdout.puts("#{Process.pid} #{s}")
+    end
+
+    def raise_if_invalid_task_list
+      for task_list in [*activity_task_lists.values, decision_task_list] do
+        if task_list.include?(separator)
+          raise "Invalid config '#{task_list}': Separator '#{separator}' is reserved for internal use."
+        end
+        if task_list.count(". ") > 0
+          raise "Invalid config '#{task_list}': Dots and spaces not allowed."
+        end
+      end
+    end
+  end
+end

data/lib/ntswf/client.rb

@@ -0,0 +1,45 @@
+require_relative 'base'
+
+module Ntswf
+  # Interface for an application that wishes to start a task
+  module Client
+    include Ntswf::Base
+
+    # Enqueue a new SWF task.
+    #
+    # The options configure the control flow of the task.
+    # Excluding *:execution_id* they will be stored in the *input* argument of the task as JSON.
+    # @param options [Hash] The task's options. Keys with special meaning:
+    # @option options [String] :execution_id Mandatory workflow ID suffix, allowed IDs are documented at docs.amazonwebservices.com (WorkflowId Property)
+    # @option options [Numeric] :interval Optional, in seconds. Enforces periodic re-run of the task, even in case of failure
+    # @option options [String] :name Identifies the kind of task for the executing unit
+    # @option options [Hash] :params Custom task parameters passed on to the executing unit
+    # @option options [String] :unit The executing unit's key, a corresponding activity task list must be configured
+    # @option options [Numeric] :version Optional minimum version of the client. The task may be rescheduled by older clients.
+    # @return [AWS::SimpleWorkflow::WorkflowExecution]
+    # @raise [AWS::SimpleWorkflow::Errors::WorkflowExecutionAlreadyStartedFault]
+    def start_execution(options)
+      execution_id = options.delete(:execution_id)
+      workflow_type.start_execution(
+        child_policy: :terminate,
+        execution_start_to_close_timeout: 48 * 3600,
+        input: options.to_json,
+        tag_list: [options[:unit].to_s, options[:name].to_s],
+        task_list: decision_task_list,
+        task_start_to_close_timeout: 10 * 60,
+        workflow_id: [activity_task_list, execution_id].join(separator),
+      )
+    end
+
+    # @return [String] separator part of the final *workflow_id*
+    def separator
+      ";"
+    end
+
+    protected
+
+    def workflow_type
+      @workflow_type ||= domain.workflow_types[workflow_name, workflow_version]
+    end
+  end
+end

data/lib/ntswf/decision_worker.rb

@@ -0,0 +1,105 @@
+require_relative 'worker'
+
+module Ntswf
+  # Interface for a worker arbitrating tasks, optionally for multiple apps
+  module DecisionWorker
+    include Ntswf::Worker
+
+    # Process a decision task.
+    # The following task values are interpreted:
+    # input:: see {Ntswf::Client#start_execution}
+    # reason:: reschedule if {RETRY}
+    # result:: Interpreted as {Hash}, see below for keys
+    # Result keys
+    # :seconds_until_retry:: Planned reschedule after task completion
+    def process_decision_task
+      announce("polling for decision task #{decision_task_list}")
+      domain.decision_tasks.poll_for_single_task(decision_task_list) do |task|
+        announce("got decision task #{task.workflow_execution.inspect}")
+        begin
+          task.new_events.each do |event|
+            log("processing event #{event.inspect}")
+            case event.event_type
+            when 'WorkflowExecutionStarted'
+              schedule(task, event)
+            when 'TimerFired'
+              start_event = task.events.first
+              keys = [
+                :child_policy,
+                :execution_start_to_close_timeout,
+                :input,
+                :tag_list,
+                :task_list,
+                :task_start_to_close_timeout,
+              ]
+              attributes = start_event.attributes.to_h.keep_if {|k| keys.include? k}
+              task.continue_as_new_workflow_execution(attributes)
+            when 'ActivityTaskCompleted'
+              result = parse_result(event.attributes.result)
+              start_timer(task, result[:seconds_until_retry]) or task.complete_workflow_execution(
+                  result: event.attributes.result)
+            when 'ActivityTaskFailed'
+              if (event.attributes.reason == RETRY)
+                schedule(task, task.events.first)
+              else
+                start_timer(task) or task.fail_workflow_execution(
+                    event.attributes.to_h.slice(:details, :reason))
+              end
+            when 'ActivityTaskTimedOut'
+              notify("Timeout in Simple Workflow. Possible cause: all workers busy",
+                  workflow_execution: task.workflow_execution.inspect)
+              start_timer(task) or task.cancel_workflow_execution(
+                  details: 'activity task timeout')
+            end
+          end
+        rescue => e
+          notify(e, workflow_execution: task.workflow_execution.inspect)
+          raise e
+        end
+      end
+    end
+
+    protected
+
+    def start_timer(task, interval = nil)
+      unless interval
+        options = parse_input(task.events.first.attributes.input)
+        interval = options['interval']
+      end
+      task.start_timer(interval.to_i) if interval
+      interval
+    end
+
+    def schedule(task, data_providing_event)
+      input = data_providing_event.attributes.input
+      options = parse_input(input)
+      app_in_charge = options['unit'] || guess_app_from(data_providing_event)
+      task_list = activity_task_lists[app_in_charge]
+      raise "Missing activity task list config for #{app_in_charge.inspect}" unless task_list
+
+      task.schedule_activity_task(activity_type, {
+        heartbeat_timeout: :none,
+        input: input,
+        task_list: task_list,
+        schedule_to_close_timeout: 12 * 3600,
+        schedule_to_start_timeout: 10 * 60,
+        start_to_close_timeout: 12 * 3600,
+      })
+    end
+
+    def parse_result(result)
+      if result
+        value = JSON.parse(result) rescue nil # expecting JSON::ParserError
+      end
+      value = {} unless value.kind_of? Hash
+      value.with_indifferent_access
+    end
+
+    private
+
+    # transitional, until all apps speak the input options protocol
+    def guess_app_from(data_providing_event)
+      data_providing_event.workflow_execution.workflow_type.name[/\w+/]
+    end
+  end
+end

data/lib/ntswf/utils.rb

@@ -0,0 +1,27 @@
+require_relative 'base'
+
+module Ntswf
+  module Utils
+    include Ntswf::Base
+
+    def create_domain(description)
+      swf.domains.create(@config[:domain], 3, description: description)
+    end
+
+    def activity_name
+      "#{default_unit}-activity"
+    end
+
+    def register_workflow_type
+      domain.workflow_types.register(workflow_name, workflow_version)
+    end
+
+    def register_activity_type
+      domain.activity_types.register(activity_name, workflow_version)
+    end
+
+    def activity_type
+      @activity_type ||= domain.activity_types[activity_name, workflow_version]
+    end
+  end
+end

data/lib/ntswf/worker.rb

@@ -0,0 +1,10 @@
+require_relative 'base'
+
+module Ntswf
+  module Worker
+    include Ntswf::Base
+
+    # *reason* value to force task reschedule, may be set if the worker is unable process the task
+    RETRY = "Retry"
+  end
+end

data/lib/ntswf.rb

@@ -0,0 +1,14 @@
+module Ntswf
+  AUTOLOAD = %w(
+    ActivityWorker
+    Client
+    DecisionWorker
+    Utils
+  )
+
+  AUTOLOAD.each { |c| autoload c.to_sym, "ntswf/#{c.gsub(/.(?=[A-Z])/, '\0_').downcase}.rb" }
+
+  def self.included(base)
+    AUTOLOAD.each { |c| include const_get c }
+  end
+end

metadata

@@ -0,0 +1,71 @@
+--- !ruby/object:Gem::Specification
+name: ntswf
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+  prerelease: 
+platform: ruby
+authors:
+- Infopark AG
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-01-10 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.8'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.8'
+description: A layer around AWS SWF
+email: info@infopark.de
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- LICENSE
+- lib/ntswf.rb
+- lib/ntswf/base.rb
+- lib/ntswf/utils.rb
+- lib/ntswf/client.rb
+- lib/ntswf/worker.rb
+- lib/ntswf/activity_worker.rb
+- lib/ntswf/decision_worker.rb
+homepage: https://github.com/infopark/ntswf
+licenses:
+- LGPLv3
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.9.3
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: Not That Simple Workflow
+test_files: []
+has_rdoc: