fbp 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f960d9070fecbed44e1f00cf0101600ba45ae40e
+  data.tar.gz: 421aaed3fb2d274f77654b085362d9078fb6aff8
+SHA512:
+  metadata.gz: 3be973bebeaca66839b8e3927895c2e872e547a83abf35615a39b4ab0b8f2a5ea6a56f633b111a923e7d96fa5ea4f61d475e8eaaa5a49af3ccca7604f0af334b
+  data.tar.gz: 940deecb561721366fed0ecaaa39af8cf95c7fd681441b8f3cf999b0aff1ae05d7a0614c1bb09431d8a7fba0c0ae4ec458fd8031dd50619b685a973a2dfbd731

data/lib/fbp.rb

@@ -0,0 +1,464 @@
+require "fbp/version"
+require "fbp/fpb-thread-pool"
+
+=begin rdoc
+== Description
+The Fbp module provides support for Flow Based Programming for the Ruby
+language.
+
+Flow Based Programming is described in the Book:
+<b>Flow-Based Programming, 2nd Edition: A New Approach to Application Development</b>
+Written by J. Paul Morrison
+ISBN-10: 1451542321
+ISBN-13: 978-1451542325
+
+Documentation about Flow Based Programing can also be found
+at http://www.jpaulmorrison.com/fbp/
+
+== Discussion
+The main idea behind Flow based Programming is that data should flow from 
+one asynchronous processing unit to another asynchronous processing unit.  
+These asynchronous processing units together would form a network that would
+constitute an application. One of the many benefits of this programming
+model is it makes it much easier to create multi-threaded and multi-process
+applications.  It also fosters the creation of small reusable components that
+maybe used in multiple applications. J. Paul Morrison does a great job of
+describing the benefits of this programming model and I highly recommend reading
+his book on the subject.
+
+Recently developers have rediscovered Flow Based Programming as a way to deal
+with managing multi-threaded applications and to deal with the complexity of
+web development. NoFlo is a company that developed a Flow Based Programming
+system for Javascript.
+
+While having used the precepts of Flow Based Programming over the years, there
+was not a unified way to do Flow Based Programming.  Given that Ruby is one of the 
+main development languages for web development, it seemed time was ripe for creating
+a Flow Based Programming system for Ruby
+
+== Concepts
+One of the basic concepts of FBP is an Information Packet (IP) An IP defines the data
+that flows between asynchronous processing units.  For Ruby Fbp an IP is a hash object.
+This allows for a common data type with infinite variations.
+
+For the Ruby implementation, each asynchronous unit is implemented by a Node object.
+Each Node object runs on a thread to ensure asynchronous execution.  To keep the
+number of threads to a reasonable amount, Node objects, execute on a thread in a
+thread pool managed by the Node_Pool class which in turn uses the Pool class developed
+by Kim Burgestrand kim@burgestrand.se.
+
+Nodes define an input queue and at least one output channel.  The input queue of a
+node uses a Queue object to manage the IPs that come into the node.  While every node
+is run on a thread, no processing occurs on that thread until an item has been pushed
+into the input queue of an node.  This is possible because the Queue class will block
+until it can get an item out of the queue.
+
+Nodes may need to be parameterized  .  In J. Paul Morrison's book, the data used to do
+parameterization is called an Initial Information Packed (IIP). For Ruby an IIP is a
+hash object.  A Node typically takes an IIP as an optional argument when creating a
+new instance of an object.  An IIP may also come to a node the node from its input queue,
+doing so means that the parameterization of a node can come from an upstream node.
+
+By default Nodes instances have a single output channel named :output.  For many Nodes
+this is sufficient.  There are however some nodes that need to have multiple output
+channels.  An example would be a Selector_node.  A Selector_node takes IPs and compares
+a value in the IP with a preset value.  Given the selector record the IP either matches
+a criteria or not. Depending on if the incoming IP matches the preset value or not
+determines which output channel.  So while the default is that this is only a single
+output channel, Nodes may have any number of output channels. For Nodes that do have
+a multiple output channels, it is possible to write to every output channel by
+specifying the output channel :all.
+
+== Current Implementation
+The current release must be consider a proof of concept. Only a handful of Nodes have
+been created and many more would need to be created before significant applications
+can be made.  Over time more nodes will be developed but there are enough nodes to
+start looking at Flow Based Programming for Ruby.
+=end
+module Fbp
+
+  protected
+	def self.create(name)  #:nodoc:
+	  class_object = nil
+	  
+	  class_object = name if name.is_a? Class
+	  if class_object.nil?
+	    begin
+        name_str = name.to_s
+        class_object =  Kernel.const_get(name_str)
+      rescue
+        class_object = nil
+      end
+      return nil if class_object.nil?
+    end
+    
+    instance = nil
+    begin
+      instance =  class_object.new()
+    rescue
+      instance = nil
+    end
+
+    instance
+  end
+
+=begin rdoc
+=== Description
+The Node class defines the base class for all Ruby Fbp Nodes.
+It defines the basic operations of all Nodes.
+=== Discussion
+When creating a new Node subclass is made it will need to at least
+override the do_node_work method.  If the subclass requires an IIP
+the subclass will also need to override the is_ready_to_run? method.
+
+The Node class basic behavior is to write its incoming IPs to its
+single output channel.
+
+=== Node object life cycle
+A node is created it is quiescent.  This is necessary as most node need
+other nodes in a network of node before it is useful.  Once all of the
+nodes have been created that are needed for an application, they need to
+be placed into a network using the register_for_output_from_node method
+to hve the output of one node become the input to another node. Once the
+network has been made.  The first node in the network needs to be executed.
+Calling the execute method on the first node will ensure that all nodes
+in the network will also be executed.  When a node is execute, it will
+be sent an IP {:start => true}. This means that when a new node subclass
+is made it needs to know that it will receive this IP in its
+def do_node_work method.  Once a node is executing, it will block until
+an IP is push into the input queue of the node unless the node has the :
+:requires_input option is set to false.  If that is the case the node will not
+block but will send an IP to the node of the form {:continue => true} for
+the node to process. The ability not to block is useful for nodes like
+the Test_file_reader_node node that reads input from a file and 
+creates IPs for down stream nodes to process.
+
+Each subclass of the Node class must override the do_node_work method.  If the node has
+not completed its work then the do_node_work method should return true.
+If the node has completed its work it should return false.  When a node
+has completed its work, it will be sent an IP of {:completed => true}.
+This :completed IP is handled in a special way.  It will push this IP into
+output_queue attribute of a Node object.  This allows the wait_until_completed
+to block until all of the work of a node is completed.  The {:completed => true}
+IP is also sent to every output channel for a node telling all of the
+down stream nodes that its upstream node has finished.
+
+=== Example Usage
+ # Need to require the Fbp gem
+ require 'Fbp'
+ # First Set the number of threads that should be used for this solution
+ Fbp::num_threads = 5
+ # Make the thread pool that will be used to run the nodes in the application
+ Fbp::make_pool
+ # Make the nodes needed for the application
+ read_node = Fbp::Test_file_reader_node.new(File.expand_path('~/input.txt'))
+ write_node = Fbp::Text_file_writer_node.new(File.expand_path('~/output.txt'))
+ # Hook up the nodes into the network needed for the application
+ write_node.register_for_output_from_node(read_node)
+ # Execute the first node in the network
+ read_node.execute
+ # Wait for the network to complete its work by checking to see if the last node
+ # in the network has completed
+ write_node.wait_until_completed
+ # With the work completed shutdown the thread pool
+ Fbp::shutdown
+=end
+	class Node
+
+    # The output attribute is an Array that holds all of the
+    # output channels for this Node
+		attr_accessor :output
+
+    # The executing attribute specifies if a Node is executing.
+		attr_reader :executing
+
+    # The options attribute hold the IIP data for a Node.
+    attr_reader :options
+
+    protected
+    # The input attribute is the Queue instance that holds incoming
+    # IPs for this Node.
+    attr_accessor :input
+
+    public
+		def initialize() #:nodoc:
+			# Initialize the input and out data types
+			@input_queue = Queue.new
+			@output_queue = Queue.new
+			
+			channel_output = Array.new
+			@output = {:output => channel_output}
+			
+			@mutex = Mutex.new
+
+			# Provide for parameterization for a node
+      @options = Hash.new
+			@options[:output] = :all
+      @options[:requires_input] = true
+
+			# Initialize the state variables for a node
+			@executing = false
+			@in_transaction = false
+			@continue_processing = true
+
+			# Provide for transaction support
+			@transactions_must_queue = true
+			@transaction_queue = Array.new
+		end
+=begin rdoc
+The  do_node_work method is where the work of a Node is done.
+Each subclass of the Node object will need to override
+this method to implement the behavior of the node. The
+base behavior of this method is to write its incoming IPs
+to it single output channel
+=end
+		def do_node_work(args)
+				write_to_output(args)
+        return false if args.has_key?(:stop)
+				true
+    end
+
+=begin rdoc
+The is_ready_to_run? method is used to ensure that a Node has
+received its required IIP before it is allowed to process incoming
+IPs.  Each subclass of the Node class that needs an IIP before it
+can process IPs needs to override this method and check to see if
+all of the required options have been received.  If all of the
+required options have been set then this method should return true
+otherwise it should return false.  The default behavior of the Node
+class is to simply return true.
+=end
+		def is_ready_to_run?()
+			true
+		end
+
+=begin rdoc
+The execute method will start the execution of this node on one of
+the threads in the thread pool.  It will block until data 
+comes into the input queue unless the node has has the :requires_input
+option it is false.  If the :requires_input option is set to false then 
+the input queue will be checked but if there are no IPs to process in the queue
+an IP will created of the form {:continue => true} and that will be sent to
+the node for processing.  
+=end
+		def execute
+			return if @executing || !Fbp.has_pool
+			
+			@output.each do |key, channel|
+			  next if channel.nil?
+        channel.each  {|n| n.execute if !n.executing}
+			end
+			
+			write_to_input({:start => true})
+			@executing = true
+			Fbp.schedule do
+				while @continue_processing
+					@mutex.synchronize do
+            if  @options[:requires_input]
+              @continue_processing = should_continue?(@input_queue.pop)
+            else
+              begin
+                ip =  @input_queue.pop(true) # non blocking
+              rescue
+                ip = nil
+              end
+              ip = ip.nil? ? {:continue => true} : ip
+              @continue_processing = should_continue?(ip)
+            end
+          end
+        end
+				@executing = false
+				@output.each_key {|channel| write_to_output({:completed => true}, channel)}
+			end
+		end
+		
+=begin rcod
+The write_to_input method will push an IP onto the input queue of a Node.
+=end
+		def write_to_input(obj)
+			@input_queue << obj if !obj.nil?
+		end
+
+=begin
+The write_to_output will write an IP into the input queue of all of
+the nodes in the specified output channel.  The default output 
+channel is the :all channel which write the IP to every channel.
+=end
+		def write_to_output(result, output_channel = :all)
+			return if result.nil?
+			@output_queue << result if result.has_key? :completed
+			
+			# Get the channels that will be written to
+			channels = nil
+			if :all == output_channel
+			  channels = @output.keys
+			else
+			  channels = [output_channel]
+			end
+			
+			# With the channel set, iterate each node and write to 
+			# that nodes output
+			channels.each do |channel_key|
+			  next if channel_key.nil?
+        channel_array = @output[channel_key]
+        next if channel_array.nil?
+			  channel_array.each do |a_node|
+			    next if a_node.nil?
+			    a_node.write_to_input(result)
+		    end
+			end
+			
+		end
+=begin rdoc
+The register_for_output_from_node method is how networks of nodes are created.
+A down stream node will register with an up stream node for the up stream's 
+node output on a specific output channel.  The default output channel is the
+:output channel.  Calling method will place the calling object into the array
+of nodes in the upstream node's output channel.  When the up stream node
+writes out it output, it will write it to all of the input queues of all of
+the down stream nodes that have registered for the output of the up steam node.
+=end
+		def register_for_output_from_node(node, output_channel = :output)
+			return if node.nil?
+			channel = node.output[output_channel]
+			if channel.nil?
+				channel = Array.new
+				node.output[output_channel] = channel
+			end
+			node.output[output_channel] << self if !node.output[output_channel].include? self
+		end
+
+=begin rdoc
+The unregister_for_output_from_node method will remove this node from
+an output queue of an up stream node.
+=end
+		def unregister_for_output_from_node(node, output_channel = :output)
+			return if node.nil?
+			channel = node.output[output_channel]
+			node.output[output_channel] = node.output[output_channel] - [self] if !channel.nil?
+		end
+
+		def merge_options!(options) #:nodoc:
+			@options.merge!(options)
+		end
+
+		def set_option (key, value) #:nodoc:
+			return if key.nil?
+			@options[key] = value
+		end
+
+		def clean_option(key) #:nodoc:
+			return if key.nil?
+			@options.delete key
+		end
+
+=begin rdoc
+The stop method will send an IP to this node of the form {:stop => true}.
+The default implementation would be to have the execution of the node stop
+though subclasses of the Node class could change that behavior
+=end
+		def stop
+			write_to_input({:stop => true})
+		end
+
+=begin rdoc
+The wait_until_completed provides a way to wait until a node has completed
+its work.  This is needed as all of the work of the nodes in a network of
+nodes is done asynchronously.  Typically this is called on the last node
+in a network of nodes to ensure that all processing has completed. This
+method works by waiting on the output_queue which is only written to when
+all work of the node has completed. The  Queue instance will cause the
+calling thread to block until an IP has been placed into the output
+queue.
+=end
+		def wait_until_completed
+			@output_queue.pop
+		end
+		
+		protected
+		def should_continue?(args) #:nodoc:
+			# If the IP contains the :stop key then stop execution
+			return false if args.has_key? :stop
+
+			# Options (IIP) support.
+			# If this node requires an IIP before it can process IPs
+			# check to see if all of the required IIPs have been received
+			# by the node.  This is done by calling is_ready_to_run?
+			# Each node that requires IIP(s) before it can processes IPs must
+			# re-implement the is_ready_to_run? method and have that
+			# method determine if all required options have been set.
+			#
+			# If an IP is sent before all of the required options then the
+			# node is put into a transaction and all of the IPs will be cached
+			# in order until all of the required options have been set. Once
+			# all of the required options have been set then the end
+			# of transaction is sent and the cached IPs will be processed.
+			if !is_ready_to_run?()
+				if args.has_key? :option
+					@options.merge!(args[:option])
+					args.delete :option
+				end
+				args[:end_transaction] = true if required_options_recieved? && @in_transaction
+				args[:begin_transaction] = true if !required_options_recieved? && !@in_transaction
+			end
+
+			# If the IP signals the beginning of a transaction, ensure that this node
+			# is not already in a transaction.  Nested transactions are not supported.
+			# Also if the node is marked with as a one shot that is incompatible with
+			# being in a transaction
+			return false if args.has_key? :begin_transaction && @in_transaction
+
+			# If the IP signals the end of a transaction and the node is not in a
+			# transaction, this is a programming error
+			return false if args.has_key? :end_transaction && !@in_transaction
+
+			# If the IP signals the beginning of a transaction then mark this node
+			# as being in a transaction and if there is no other data in the IP
+			# then return signaling that the node should continue
+			if args.has_key?(:begin_transaction)
+				@in_transaction = true
+				args.delete :begin_transaction
+				return true if args.empty?
+			end
+
+			# If the IP signals the end of a transaction then see if there is any other
+			# data in the IP.  If there is any additional data add it to the
+			# transaction queue before processing the transaction. Once processed mark
+			# this node as not being in a transaction
+			if args.has_key?(:end_transaction) || (args.has_key?(:completed) && !@transaction_queue.empty?)
+				args.delete :end_transaction
+				@transaction_queue << args if !args.empty?
+				args.clear
+				args[:ips] = @transaction_queue.clone if !@transaction_queue.empty?
+				@transaction_queue.clear
+				@in_transaction = false
+				return true  if !args.has_key? :ips
+      end
+
+			# If the node is in a transaction add the IP to the transaction queue,
+			# otherwise tell the node to process th IP
+			result = true
+
+			if @in_transaction 
+			  @transaction_queue << args
+			else
+			  result = do_node_work(args)
+			end
+
+			result
+		end
+	end
+end
+
+require "fbp/selector-node"
+require "fbp/concatenate-node"
+require "fbp/assign-node"
+require "fbp/encode-node"
+require "fbp/decode-node"
+require "fbp/sort-node"
+require "fbp/counter-node"
+require "fbp/text_file_reader_node"
+require "fbp/text_file_writer_node"
+require "fbp/flow-node"
+require "fbp/aggregator-node"
+