oflow 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 3bffd9fcf94e7e2e7bd1042d9eac1579debdbb15
+  data.tar.gz: 1606f17e500c5c533b7192dfb2308ffffb99a393
+SHA512:
+  metadata.gz: c9c6fac1a663adba34988a7caeaff5603054aaa7cf581186bcd199ff439650bc168e3ff65f9edf5a0cc2bf71e246aef5e032f97a11b70f7b2b92050170467053
+  data.tar.gz: a158841d007f3be5f1bddc54fb22686931ff42bdd033c5b097d2bcef6ceb351347e9c0709107926866de9ea0504c429d7bbe28ffeb1d9a2b2974075bb790de80

data/LICENSE

@@ -0,0 +1,20 @@
+The MIT License (MIT)
+
+Copyright (c) 2013 Peter Ohler
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of
+this software and associated documentation files (the "Software"), to deal in
+the Software without restriction, including without limitation the rights to
+use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
+the Software, and to permit persons to whom the Software is furnished to do so,
+subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
+FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
+COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,182 @@
+OFlow-ruby
+==========
+
+Operations Workflow in Ruby. This implements a workflow/process flow using
+multiple task nodes that each have their own queues and execution thread.
+
+## Installation
+    gem install oflow
+
+## Documentation
+
+*Documentation*: http://www.ohler.com/oflow
+
+## Source
+
+*GitHub* *repo*: https://github.com/ohler55/oflow
+
+*RubyGems* *repo*: https://rubygems.org/gems/oflow
+
+Follow [@peterohler on Twitter](http://twitter.com/#!/peterohler) for announcements and news about the OFlow gem.
+
+## Build Status
+
+[![Build Status](https://secure.travis-ci.org/ohler55/oflow.png?branch=master)](http://travis-ci.org/ohler55/oflow)
+
+### Current Release 0.3
+
+ - Initial release with minimal features.
+
+## Description
+
+Workflow or more accurately, process flow is associated with processing data
+based on a diagram of what the processing steps are. OFlow-Ruby implements that
+in Ruby. Each node in a flow diagram is processing unit with it's own
+thread. This allows highly parallel processing of data or at least as much as
+Ruby allows. The future C version will exploit parallel processing to a greater
+extent with much higher performance.
+
+One of the problem in any system that processes data in parallel is determing
+what is going on in the system. OFlow provides a run time inspector that can
+monitor and control the system and individual nodes in a flow.
+
+## How to Use / Example
+
+Flows are composed of individual building blocks referred to as Tasks. A Task
+wraps an Actor that is the custom portion of a Task. Once the individual Actors
+have been written they are assembled by linking Tasks together to define the
+process flow. The process flow can be defined in Ruby or in the near future with
+a drawing application such as OmniGraffle.
+
+The approach of using diagrams to define what is effectively a program allows
+less experience developers to assmeble pieces build by more experienced
+developers. It also make is much easier to describe a process to non-developers.
+
+OFlow uses a design pattern of queues and encapsulated processing units in the
+form of Tasks. It also forces isolation by freezing data that is passed from one
+Task to another to assure data that is sent to more than one Task is not
+modified by another Task.
+
+Putting it all together in a simple hello world flow with timmer triggers starts
+with defining a hello world Actor in a file called helloworld.rb.
+
+```ruby
+require 'oflow'
+
+class HelloWorld < ::OFlow::Actor
+  def initialize(task, options)
+    super
+  end
+
+  def perform(op, box)
+    puts 'Hello World!'
+  end
+end
+```
+
+Next build the flow using Ruby code.
+
+```ruby
+def hello_flow(period)
+    ::OFlow::Env.flow('hello_world') { |f|
+      f.task(:repeater, ::OFlow::Actors::Timer, repeat: 3, period: period) { |t|
+        t.link(nil, :hello, nil)
+      }
+      f.task(:hello, HelloWorld)
+    }
+end
+
+hello_flow(1.0)
+
+if $0 == __FILE__
+  ::OFlow::Env.flush()
+end
+```
+
+Running the helloworld.rb results in this output.
+
+```
+> helloworld.rb
+Hello World!
+Hello World!
+Hello World!
+>
+```
+
+## Future Features
+
+ - Balancer Actor that distributes to a set of others Tasks based on how busy each is.
+
+ - Merger Actor that waits for a criteria to be met before continuing.
+
+ - HTTP Server Actor
+
+ - Persister Actor that writes to disk and reads on start)
+
+ - CallOut Actor that uses pipes and fork to use a non-Ruby actor.
+
+ - Cross linking Tasks and Flows.
+
+ - Dynamic links to Tasks and Flows.
+
+ - OmniGraffle file input for configuration.
+
+ - .svg file input for configuration.
+
+ - Visio file input for configuration.
+
+ - High performance C version. Proof of concept puts the performance range at
+   around 10M operations per second where an operation is one task execution per
+   thread.
+
+ - HTTP based inpector.
+
+# Links
+
+## Links of Interest
+
+*Fast XML parser and marshaller on RubyGems*: https://rubygems.org/gems/ox
+
+*Fast XML parser and marshaller on GitHub*: https://github.com/ohler55/ox
+
+[Oj Object Encoding Format](http://www.ohler.com/dev/oj_misc/encoding_format.html) describes the OJ Object JSON encoding format.
+
+[Need for Speed](http://www.ohler.com/dev/need_for_speed/need_for_speed.html) for an overview of how Oj::Doc was designed.
+
+[Oj Strict Mode Performance](http://www.ohler.com/dev/oj_misc/performance_strict.html) compares Oj strict mode parser performance to other JSON parsers.
+
+[Oj Compat Mode Performance](http://www.ohler.com/dev/oj_misc/performance_compat.html) compares Oj compat mode parser performance to other JSON parsers.
+
+[Oj Object Mode Performance](http://www.ohler.com/dev/oj_misc/performance_object.html) compares Oj object mode parser performance to other marshallers.
+
+[Oj Callback Performance](http://www.ohler.com/dev/oj_misc/performance_callback.html) compares Oj callback parser performance to other JSON parsers.
+
+### License:
+
+    Copyright (c) 2014, Peter Ohler
+    All rights reserved.
+    
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions are met:
+    
+     - Redistributions of source code must retain the above copyright notice, this
+       list of conditions and the following disclaimer.
+    
+     - Redistributions in binary form must reproduce the above copyright notice,
+       this list of conditions and the following disclaimer in the documentation
+       and/or other materials provided with the distribution.
+    
+     - Neither the name of Peter Ohler nor the names of its contributors may be
+       used to endorse or promote products derived from this software without
+       specific prior written permission.
+    
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+    AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+    DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+    SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+    CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+    OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/lib/oflow/actor.rb

@@ -0,0 +1,76 @@
+
+module OFlow
+
+  # Actors provide the custom functionality for Tasks. Each Task creates an
+  # instance of some Actor. Actors are not shared between Tasks and each can be
+  # assure that the data they operate on is their own.
+  class Actor
+
+    # The enclosing task.
+    attr_reader :task
+
+    # Creates a new instance.
+    # @param task [Task] enclosing Task
+    # @param options [Hash] additional options
+    def initialize(task, options)
+      @task = task
+    end
+    
+    # Perform the primary functions for the Actor.
+    # @param op [Symbol] operation to perform
+    # @param box [Box] contents or data for the operation
+    def perform(op, box)
+    end
+
+    # Returns whether the Actor should have it's own thread or not. In almost
+    # all cases the Actor should have it's own thread. The exception is when the
+    # action is trivial such as a relay.
+    # @return [Boolean] indicator of whether a new thread should be created.
+    def with_own_thread()
+      true
+    end
+
+    # Return array of Specs.
+    # @return [Array] Array of Specs.
+    def inputs()
+      nil
+    end
+
+    # Return array of Specs.
+    # @return [Array] Array of Specs.
+    def outputs()
+      nil
+    end
+
+    # Return any options that should be displayed as part of a Task.describe().
+    # @return [Hash] Hash of options with String keys.
+    def options()
+      {}
+    end
+
+    class Spec
+      attr_reader :op
+      attr_reader :type
+
+      def initialize(op, type)
+        if op.nil? || op.is_a?(Symbol)
+          @op = op
+        else
+          @op = op.to_sym
+        end
+        @type = type
+      end
+
+      alias dest op
+
+      def to_s()
+        "Spec{op: #{@op}, type: #{@type}}"
+      end
+      alias inspect to_s
+
+    end # Spec
+
+  end
+
+end # OFlow
+

data/lib/oflow/actors/errorhandler.rb

@@ -0,0 +1,32 @@
+
+module OFlow
+  module Actors
+
+    # The default error handler.
+    class ErrorHandler < Actor
+
+      def initialize(task, options={})
+        super
+      end
+
+      # Open the box, form a reasonable message, then log that message.
+      # @param op [Symbol] ignores
+      # @param box [Box] data associated with the error
+      def perform(op, box)
+        contents = box.contents
+        return task.error(contents.to_s) unless contents.is_a?(Array)
+        e, where = contents
+        task.error(e.to_s) unless e.is_a?(Exception)
+        msg = ["#{e.class}: #{e.message}"]
+        e.backtrace.each { |line| msg << ('    ' + line) }
+        task.log_msg(:error, msg.join("\n"), where)
+      end
+
+      # Handle error immediately.
+      def with_own_thread()
+        false
+      end
+
+    end # ErrorHandler
+  end # Actors
+end # OFlow

data/lib/oflow/actors/ignore.rb

@@ -0,0 +1,22 @@
+
+module OFlow
+  module Actors
+    # Does nothing but can be used to terminate a Link to assure all output from
+    # a Task terminate somewhere.
+    class Ignore < Actor
+
+      def initialize(task, options)
+        super
+      end
+
+      def perform(op, box)
+        # ignore
+      end
+
+      def with_own_thread()
+        false
+      end
+
+    end # Ignore
+  end # Actors
+end # OFlow

data/lib/oflow/actors/log.rb

@@ -0,0 +1,175 @@
+
+require 'logger'
+
+module OFlow
+  module Actors
+
+    # An asynchronous logger build on top of the Actor class. It is able to log
+    # messages as well as forward calls to a Task.
+    class Log < Actor
+
+      SEVERITY_MAP = {
+        :fatal => Logger::Severity::FATAL,
+        :error => Logger::Severity::ERROR,
+        :warn => Logger::Severity::WARN,
+        :info => Logger::Severity::INFO,
+        :debug => Logger::Severity::DEBUG,
+      }
+      def initialize(task, options={})
+        @logger = nil
+        @formatter = nil
+        @name = nil
+        super
+        set_options(options)
+      end
+
+      # Returns the current severity level.
+      # @return [Fixnum] Logger severity level
+      def severity()
+        @logger.level
+      end
+      alias :level :severity
+
+      # Returns the current formatter.
+      # @return [Logger::Formatter] current formatter
+      def formatter()
+        @formatter
+      end
+
+      # Writes a log entry. op is the severity. The box contents is expected to be
+      # an Array with the first element as the full task name of the logging task
+      # and the second argument being the message to log
+      def perform(op, box)
+        op = op.to_sym unless op.nil?
+        a = box.contents
+        case op
+        when :severity
+          self.severity = a
+        when :formatter
+          # TBD
+        when :file, :filename
+          # TBD
+          # self.set_filename(filename, shift_age=7, shift_size=1048576)
+        else
+          level = SEVERITY_MAP.fetch(op, Logger::Severity::UNKNOWN)
+          if a.is_a?(Array)
+            log(level, a[0], a[1])
+          else
+            log(level, a.to_s, '')
+          end
+          # Forward to the next if there is a generic (nil) link.
+          task.ship(nil, box) if task.find_link(nil)
+        end
+      end
+
+      private
+
+      # Sets the logger, severity, and formatter if provided.
+      # @param [Hash] options options to be used for initialization
+      # @option options [String] :filename filename to write to
+      # @option options [Fixnum] :max_file_size maximum file size
+      # @option options [Fixnum] :max_file_count maximum number of log file
+      # @option options [IO] :stream IO stream
+      # @option options [String|Fixnum] :severity initial setting for severity
+      # @option options [Proc] :formatter initial setting for the formatter procedure
+      def set_options(options)
+        if !(filename = options[:filename]).nil?
+          max_file_size = options.fetch(:max_file_size, options.fetch(:shift_size, 1048576))
+          max_file_count = options.fetch(:max_file_count, options.fetch(:shift_age, 7))
+          @logger = Logger.new(filename, max_file_count, max_file_size)
+        elsif !(stream = options[:stream]).nil?
+          @logger = Logger.new(stream)
+        else
+          @logger = Logger.new(STDOUT)
+        end
+        @logger.level = options.fetch(:severity, Env.log_level)
+        @formatter = options.fetch(:formatter, nil)
+        @logger.formatter = proc { |s,t,p,m| m }
+        @name = 'Logger' if @name.nil?
+      end
+
+      # Writes a message if the severity is high enough. This method is
+      # executed asynchronously.
+      # @param level [Fixnum] one of the Logger levels
+      # @param message [String] string to log
+      # @param tid [Fixnum|String] Task id of the Task generating the message
+      def log(level, message, tid)
+        now = Time.now
+        ss = ['DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL'][level]
+        ss = '' if ss.nil?
+        if @formatter.nil?
+          msg = "#{ss[0]}, [#{now.strftime('%Y-%m-%dT%H:%M:%S.%6N')} #{tid}] #{ss} -- #{message}\n"
+        else
+          msg = @formatter.call(ss, now, tid, message)
+        end
+        @logger.add(level, msg)
+      end
+
+      # Sets the logger to use the stream specified. This method is executed
+      # asynchronously.
+      # @param [IO] stream stream to write log messages to
+      def stream=(stream)
+        logger = Logger.new(stream)
+        logger.level = @logger.level
+        logger.formatter = @logger.formatter
+        @logger = logger
+      end
+
+      # Creates a new Logger to write log messages to using the parameters
+      # specified. This method is executed asynchronously.
+      # @param filename [String] filename of active log file
+      # @param shift_age [Fixmun] maximum number of archive files to save
+      # @param shift_size [Fixmun] maximum file size
+      def set_filename(filename, shift_age=7, shift_size=1048576)
+        logger = Logger.new(filename, shift_age, shift_size)
+        logger.level = @logger.level
+        logger.formatter = @logger.formatter
+        @logger = logger
+      end
+
+      # Replace the logger with a new Logger Object. This method is executed
+      # asynchronously.
+      # @param logger [Logger] replacement logger
+      def logger=(logger)
+        @logger = logger
+      end
+
+      # Sets the severity level of the logger. This method is executed
+      # asynchronously.
+      # @param level [String|Fixnum] value to set the severity to
+      def severity=(level)
+        if level.is_a?(String)
+          sev = {
+            'FATAL' => Logger::Severity::FATAL,
+            'ERROR' => Logger::Severity::ERROR,
+            'WARN' => Logger::Severity::WARN,
+            'INFO' => Logger::Severity::INFO,
+            'DEBUG' => Logger::Severity::DEBUG,
+            '4' => Logger::Severity::FATAL,
+            '3' => Logger::Severity::ERROR,
+            '2' => Logger::Severity::WARN,
+            '1' => Logger::Severity::INFO,
+            '0' => Logger::Severity::DEBUG
+          }[level.upcase()]
+          raise "#{level} is not a severity" if sev.nil?
+          level = sev
+        elsif level.is_a?(Symbol)
+          sev = SEVERITY_MAP[level]
+          raise "#{level} is not a severity" if sev.nil?
+          level = sev
+        elsif !level.is_a?(Fixnum) || level < Logger::Severity::DEBUG || Logger::Severity::FATAL < level
+          raise "#{level} is not a severity"
+        end
+        @logger.level = level
+      end
+
+      # Sets the formatter procedure of the logger. This method is executed
+      # asynchronously.
+      # @param proc [Proc] value to set the formatter to
+      def formatter=(proc)
+        @formatter = proc
+      end
+
+    end # Log
+  end # Actors
+end # OFlow

data/lib/oflow/actors/relay.rb

@@ -0,0 +1,23 @@
+
+module OFlow
+  module Actors
+
+    # Relays a shipment to another location. This is useful for creating aliases
+    # for a Task.
+    class Relay < Actor
+
+      def initialize(task, options)
+        super
+      end
+
+      def perform(op, box)
+        task.ship(op, box)
+      end
+
+      def with_own_thread()
+        false
+      end
+
+    end # Relay
+  end # Actors
+end # OFlow

data/lib/oflow/actors/timer.rb

@@ -0,0 +1,126 @@
+
+require 'logger'
+
+module OFlow
+  module Actors
+
+    class Timer < Actor
+
+      MAX_SLEEP = 1.0
+
+      # When to trigger the first event. nil means start now.
+      attr_reader :start
+      # The stop time. If nil then there is not stopping unless the repeat limit
+      # kicks in.
+      attr_reader :stop
+      # How long to wait between each trigger. nil indicates as fast as possible,
+      attr_reader :period
+      # How many time to repeat before stopping. nil mean go forever.
+      attr_reader :repeat
+      # Label for the Tracker is used and for trigger content.
+      attr_reader :label
+      # The number of time the timer has fired or shipped.
+      attr_reader :count
+      # Boolean flag indicating a tracker should be added to the trigger content
+      # if true.
+      attr_reader :with_tracker
+      # Time of next or pending trigger.
+      attr_reader :pending
+
+      def initialize(task, options={})
+        @count = 0
+        @pending = nil
+        set_options(options)
+        @start = Time.now() if @start.nil?
+        @pending = @start
+        super
+        task.receive(:init, nil)
+      end
+
+      # The loop in the Task containing this Actor is the thread used for the
+      # timer. Mostly the perform() method sleeps but it will be woken when a
+      # new request is placed on the Task queue so it exits if there is a
+      # request on the queue even if it has not triggered a ship() know that it
+      # will be re-entered.
+      def perform(op, box)
+        op = op.to_sym unless op.nil?
+        case op
+        when :stop
+          # TBD if no arg (or earlier than now) then stop now else set to new stop time
+        when :start
+          # TBD if stopped then start if no arg, if arg then set start time
+        when :period
+          # TBD
+        when :repeat
+          # TBD
+        when :label
+          # TBD
+        when :with_tracker
+          # TBD
+        end
+        while true
+          now = Time.now()
+
+          # If past stop time then it is done. A future change in options can
+          # restart the timer.
+          return if !@stop.nil? && @stop < now
+          # Has repeat number been exceeded?
+          return if !@repeat.nil? && @repeat <= @count
+          # If there is nothing pending the timer has completed.
+          return if @pending.nil?
+          # If the Task is blocked or shutting down.
+          return if Task::CLOSING == task.state || Task::BLOCKED == task.state
+
+          if @pending <= now
+            # Skip if stopped but do not increment counter.
+            unless Task::STOPPED == task.state
+              @count += 1
+              now = Time.now()
+              tracker = @with_tracker ? Tracker.new(@label) : nil
+              box = Box.new([@label, @count, now.utc()], tracker)
+              task.links.each_key do |key|
+                begin
+                  task.ship(key, box)
+                rescue BlockedError => e
+                  task.warn("Failed to ship timer #{box.contents} to #{key}. Task blocked.")
+                rescue BusyError => e
+                  task.warn("Failed to ship timer #{box.contents} to #{key}. Task busy.")
+                end
+              end
+            end
+            if @period.nil? || @period == 0
+              @pending = now
+            else
+              @pending += period
+            end
+          end
+          # If there is a request waiting then return so it can be handled. It
+          # will come back here to allow more timer processing.
+          return if 0 < task.queue_count()
+
+          if Task::STOPPED == task.state
+            sleep(0.1)
+          else
+            now = Time.now()
+            if now < @pending
+              wait_time = @pending - now
+              wait_time = MAX_SLEEP if MAX_SLEEP < wait_time
+              sleep(wait_time)
+            end
+          end
+        end
+      end
+
+      def set_options(options)
+        @start = options[:start]
+        @stop = options[:stop]
+        @period = options[:period]
+        @repeat = options[:repeat]
+        @label = options[:label]
+        @with_tracker = options[:with_tracker]
+        # TBD check values for type and range
+      end
+
+    end # Timer
+  end # Actors
+end # OFlow

data/lib/oflow/actors.rb

@@ -0,0 +1,11 @@
+
+module OFlow
+  module Actors
+  end
+end
+
+require 'oflow/actors/ignore'
+require 'oflow/actors/relay'
+require 'oflow/actors/log'
+require 'oflow/actors/errorhandler'
+require 'oflow/actors/timer'