rflow 1.0.0a2 → 1.0.0a3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 01773e5fd8c83119dd4c8f699099848bf3584738
-  data.tar.gz: b57b8e3c110e826c8d5db30e1c43e416c56bc627
+  metadata.gz: 82d81c66d6a26aa814d88893e294951247434585
+  data.tar.gz: 03edc9684d65ffb4e0729000f8948f8dbe69c16c
 SHA512:
-  metadata.gz: 4c51ebe3e627932117535eeb8cfb0cb213e34fccf9938b78a27095aaf9da2792a2f1e16ddc66bb49a4556adbbe7ca37ed0d96a048319c18581546b91fcbf7d29
-  data.tar.gz: ad1162c1f0a2902300073cdccc0ccab3a67ace9fd99cfc47c5bd8449c8d551e94e089a4a73e15600910f08a87de7e34d2d8751ba204a891a62f0a35103618fee
+  metadata.gz: 7ca15c17aa77a9e15d5309c5e9dd22b533b0108ca6f8c29031899f8c824f3c4939c5c37e57bbe51db5834eefc40f0f52a840874af9660fbbe07729b8a66ea3d0
+  data.tar.gz: a8f7a76ca8eba6aec45cdf56433792938610e11332dd0523917795dd4a59edc84660d80b08a48a67c222f3a312ba6837b238a36e9db9e982b62fac2742f93417

data/.gitignore

@@ -6,3 +6,4 @@ pkg/*
 .\#*
 spec/tmp/
 *.swp
+/.vagrant

data/.ruby-version

@@ -1 +1 @@
-ruby-2.1.1
+ruby-2.1.2

data/README.VAGRANT

@@ -0,0 +1,63 @@
+How to build the vagrant boxes (VMWare Fusion 6):
+
+First:
+Add /Applications/VMware Fusion.app//Contents/Library/ to your PATH.
+
+CentOS 6.2 (http://cbednarski.com/articles/creating-vagrant-base-box-for-centos-62/)
+
+curl -O http://mirrors.usc.edu/pub/linux/distributions/centos/6.2/isos/x86_64/CentOS-6.2-x86_64-minimal.iso
+Create a new VMWare box with the iso. Start the image. Don't use Easy Install.
+  Hostname: centos62
+  Root password: vagrant
+  Disk partition: Replace Existing Linux Partition.
+  Reboot.
+Log in as root/vagrant.
+vi /etc/sysconfig/network-scripts/ifcfg-eth0
+  Change ONBOOT="no" to yes.
+  Remove HWADDR line.
+  Add BOOTPROTO="dhcp"
+vi /etc/udev/rules.d/70-persistent-net.rules
+  Replace the eth0 line with: SUBSYSTEM=="net", ACTION=="add", DRIVERS="?*", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
+shutdown -r now
+
+Log back in as root.
+VMWare Fusion > Virtual Machine > Install VMWare Tools
+mkdir /media/cdrom
+mount /dev/cdrom /media/cdrom
+cd /tmp
+tar -xzf /media/cdrom/VM[tab].tar.gz
+yum install -y perl eject
+/tmp/vmware-tools-distrib/vmware-install.pl --default
+
+yum install -y sudo
+useradd -m vagrant
+usermod -aG wheel vagrant
+echo vagrant | passwd vagrant --stdin
+echo "vagrant ALL=(ALL) ALL" >> /etc/sudoers
+echo "%wheel ALL=NOPASSWD: ALL" >> /etc/sudoers
+echo 'Defaults env_keep="SSH_AUTH_SOCK"' >> /etc/sudoers
+
+vi /etc/sudoers
+  Change requiretty to !requiretty
+
+yum install -y openssh-server
+echo "UseDNS no" >> /etc/ssh/sshd_config
+
+mkdir -m 0700 /home/vagrant/.ssh
+curl -s https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub > /home/vagrant/.ssh/authorized_keys
+chown -R vagrant:vagrant /home/vagrant/.ssh
+chmod -R 0600 /home/vagrant/.ssh/*
+shutdown -r now
+
+Verify logging in as vagrant/vagrant and `sudo ls /root` with no password.
+
+sudo shutdown -h now
+
+cd to wherever you have stored the VMWare box (~/Documents/Virtual Machines.localized/ by default).
+vmware-vdiskmanager -d Virtual\ Disk.vmdk   (ignore the warning message)
+vmware-vdiskmanager -k Virtual\ Disk.vmdk   (ignore the warning message)
+cat > metadata.json
+{
+  "provider":"vmware_fusion"
+}
+

data/README.md

@@ -20,21 +20,34 @@ however only two are in current use, namely ZeroMQ connections and
 Avro serialization.
 
 RFlow currently runs as a single-threaded, evented system on top of
-[Eventmachine](http://rubyeventmachine.com/), meaning that any code
+[EventMachine](http://rubyeventmachine.com/), meaning that any code
 should be coded in an asynchronous style so as to not block the
-Eventmachine reactor (and thus block all the other components). There
-is currently work being done to "shard" the workflow among multiple
-processes and/or threads.
+EventMachine reactor (and thus block all the other components). Use
+`EM.defer` and other such patterns, along with EventMachine plugins
+for various servers and clients, to work in this style and defer
+computation to background threads.
+
+RFlow component workflows may be split into `shards` to improve
+parallelism. Each shard is currently represented by a separate process,
+though threads may be supported in the future. Multiple copies of a
+shard may be instantiated, which will cooperate to round-robin
+incoming messages.
 
 Some of the long-term goals of RFlow are to allow for components and
 portions of the workflow to be defined in any language that supports
-Avro and ZeroMQ, which are numerous.
+Avro and ZeroMQ, which are numerous. For this reason, the official
+specification of an RFlow workflow is a SQLite database containing
+information on its components, connections, ports, settings, etc.
+There is a Ruby DSL that aids in populating the database but the intent
+is that multiple processes and languages could access and manipulate
+the database form.
 
 ## Developer Notes
 
 You will need ZeroMQ preinstalled. Currently, EventMachine only supports
 v3.2.4, not v4.x, so install that version. Older versions like 2.2 will not
-work.
+work. (You will probably get errors saying arcane things like
+`assertion failed, mailbox.cpp(84)`).
 
 ## Definitions
 
@@ -50,8 +63,8 @@ work.
   accessed by an array.
 
 * __Connection__ - a directed link between an output port and an input
-  port.  RFlow supports generalized connection types, however only
-  ZeroMQ IPC links are currently used.
+  port.  RFlow supports generalized connection types; however, only
+  ZeroMQ links are currently used.
 
 * __Message__ - a bit of serialized data that is sent out an output
   port and recieved on an input port. Due to the serialization,
@@ -63,7 +76,6 @@ work.
   components (nodes) are wired together via connections to their
   respective output/input ports.
 
-
 ## Component Examples
 
 The following describes the API of an RFlow component:
@@ -90,7 +102,7 @@ end
 * `configure!` (called with a hash configuration) is called after the
   component is instantiated but before the workflow has been wired or
   any messages have been sent. Note that this is called outside the
-  Eventmachine reactor.
+  EventMachine reactor.
 
 * `run!` is called after all the components have been wired together
   with connections and the entire workflow has been created. For a
@@ -98,11 +110,11 @@ end
   be sent. For example, if the component is reading from a file, this
   is where the file will be opened, the contents read into a message,
   and the message sent out the output port. `run!` is called within
-  the Eventmachine reactor.
+  the EventMachine reactor.
 
 * `process_message` is an evented callback that is called whenever the
   component receives a message on one of its input ports.
-  `process_message` is called withing the Eventmachine reactor
+  `process_message` is called within the EventMachine reactor
 
 * `shutdown!` is called when the flow is being terminated, and is
   meant to allow the components to do penultimate processing and send
@@ -186,20 +198,19 @@ configuration):
 class RFlow::Components::FileOutput < RFlow::Component
   input_port :in
 
-  attr_accessor :output_file_path, :output_file
+  attr_accessor :output_file_path
 
   def configure!(config)
     self.output_file_path = config['output_file_path']
-    self.output_file = File.new output_file_path, 'w+'
   end
 
   def process_message(input_port, input_port_key, connection, message)
-    output_file.puts message.data.data_object.inspect
-    output_file.flush
-  end
-
-  def cleanup
-    output_file.close
+    File.open(output_file_path, 'a') do |f|
+      f.flock(File::LOCK_EX)
+      f.puts message.data.data_object.inspect
+      f.flush
+      f.flock(File::LOCK_UN)
+    end
   end
 end
 ```
@@ -314,7 +325,7 @@ messaga.data.default?   # => false
 RFlow currently stores its configuration in a SQLite database which
 are internally accessed via ActiveRecord.  Given that SQLite is a
 rather simple and standard interface, non-RFlow components could
-access it and determine their respsective ZMQ connections.
+access it and determine their respective ZMQ connections.
 
 DB schemas for the configuration database are in
 [lib/rflow/configuration/migrations](lib/rflow/configuration/migrations)
@@ -323,26 +334,29 @@ tables uses a UUID primary key, and UUIDs are used within RFlow to
 identify specific components.
 
 * settings - general application settings, such as log levels, app
-  names, directories, etc
+  names, directories, etc.
+
+* shards - a list of the shards defined for the workflow, including
+  UUID, type, and number of workers for the shard
 
 * components - a list of the components including its name,
-  specification (Ruby class), and options. Note that the options are
+  specification (Ruby class), shard, and options. Note that the options are
   serialized to the database as YAML, and components should understand
   that the round-trip through the database might not be perfect (e.g.
   Ruby symbols might become strings). A component also has a number of
   input ports and output ports.
 
 * ports - belonging to a component (via `component_uuid` foreign key),
-  also has a `type` colum for ActiveRecord STI, which gets set to
+  also has a `type` column for ActiveRecord STI, which gets set to
   either a `RFlow::Configuration::InputPort` or
   `RFlow::Configuration::OutputPort`.
 
 * connections - a connection between two ports via foriegn keys
   `input_port_uuid` and `output_port_uuid`. Like ports, connections
-  are typed via AR STI (`RFlow::Configuration::ZMQConnection` or
-  `RFlow::Configuration::AMQPConnection`) and have a YAML serialized
-  `options` hash. A connection also (potentially) defines the port
-  keys.
+  are typed via AR STI (`RFlow::Configuration::ZMQConnection` and
+  'RFlow::Configuration::BrokeredZMGConnection` are the only
+  supported values for now) and have a YAML serialized `options`
+  hash. A connection also (potentially) defines the port keys.
 
 RFlow also provides a RubyDSL for configuration-like file to be used
 to load the database:
@@ -351,10 +365,9 @@ to load the database:
 RFlow::Configuration::RubyDSL.configure do |config|
   # Configure the settings, which include paths for various files, log
   # levels, and component specific stuffs
-  config.setting('rflow.log_level', 'DEBUG')
-  config.setting('rflow.application_directory_path', '../tmp')
-
-  config.setting('rflow.application_name', 'testapp')
+  config.setting 'rflow.log_level', 'DEBUG'
+  config.setting 'rflow.application_directory_path', '../tmp'
+  config.setting 'rflow.application_name', 'testapp'
 
   # Instantiate components
   config.component 'generate_ints1', 'RFlow::Components::GenerateIntegerSequence', {
@@ -386,10 +399,82 @@ RFlow::Configuration::RubyDSL.configure do |config|
 end
 ```
 
+## Parallelism
+
+RFlow supports parallelizing workflows and splitting them into multiple
+`shard`s. By default, components declared in the Ruby DSL exist in the
+default shard, named `DEFAULT`. There is only one worker for the default
+shard.
+
+ZeroMQ communication between components in the same shard uses ZeroMQ's
+`inproc` socket type for maximum performance. ZeroMQ communication between
+components in different shards is accomplished with a ZeroMQ `ipc` socket.
+In the case of a many-to-many connection (many workers in a producing
+shard and many workers in a consuming shard), a ZeroMQ message broker
+process is created to route the messages appropriately. Senders round-robin
+to receivers and receivers fair-queue the messages from the senders.
+Load balancing based on receiver responsiveness is not currently implemented.
+
+To define a custom shard in the Ruby DSL, use the `shard` method. For
+example:
+
+```ruby
+RFlow::Configuration::RubyDSL.configure do |config|
+  # Configure the settings, which include paths for various files, log
+  # levels, and component specific stuffs
+  config.setting 'rflow.log_level', 'DEBUG'
+  config.setting 'rflow.application_directory_path', '../tmp'
+  config.setting 'rflow.application_name', 'testapp'
+
+  config.shard 'integers', :process => 2 do |shard|
+    shard.component 'generate_ints1', 'RFlow::Components::GenerateIntegerSequence', {
+      'start' => 0,
+      'finish' => 10,
+      'step' => 3,
+      'interval_seconds' => 1
+    }
+    shard.component 'generate_ints2', 'RFlow::Components::GenerateIntegerSequence', {
+      'start' => 20,
+      'finish' => 30
+    }
+  end
+
+  # another style of specifying type and count; count defaults to 1
+  config.shard 'filters', :type => :process, :count => 1 do |shard|
+    shard.component 'filter', 'RFlow::Components::RubyProcFilter', {
+      'filter_proc_string' => 'lambda {|message| true}'
+    }
+  end
+
+  # another way of specifying type
+  config.process 'filters', :count => 2 do |shard|
+    shard.component 'output1', 'RFlow::Components::FileOutput', {
+      'output_file_path' => '/tmp/out1'
+    }
+  end
+
+  # this component will be created in the DEFAULT shard
+  config.component 'output2', 'RFlow::Components::FileOutput', {
+    'output_file_path' => '/tmp/out2'
+  }
+
+  # Wire components together
+  config.connect 'generate_ints1#out' => 'filter#in'
+  config.connect 'generate_ints2#out' => 'filter#in'
+  config.connect 'filter#filtered' => 'replicate#in'
+  config.connect 'filter#out' => 'output1#in'
+  config.connect 'filter#filtered' => 'output2#in'
+end
+```
+
+At runtime, shards with no components defined will have no workers and
+will not be started. (So, if you put all components in a custom shard,
+no `DEFAULT` workers will be seen.)
+
 ## Command-Line Operation
 
 RFlow includes the `rflow` binary that can load a database from a Ruby
-DSL, as well as start/stop the wokflow application as a daemon.
+DSL, as well as start/stop the workflow application as a daemon.
 Invoking the `rflow` binary without any options will give a brief help:
 
 ```

data/Vagrantfile

@@ -0,0 +1,53 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+VAGRANTFILE_API_VERSION = "2"
+
+Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
+  config.vm.define 'centos62' do |c|
+    c.vm.box = "jstoneham/rflow-centos62"
+  end
+  config.vm.define 'centos64' do |c|
+    c.vm.box = "box-cutter/centos64"
+  end
+  config.vm.define 'centos65' do |c|
+    c.vm.box = "chef/centos-6.5"
+  end
+
+  config.vm.synced_folder '.', '/rflow'
+  # bring over rflow examples; use rsync so it's safe to create IPCs in the rflow-examples directory
+  # (that is, avoid NFS)
+  # run 'vagrant rsync-auto' to get syncing to happen automatically
+  config.vm.synced_folder '../rflow_examples', '/rflow_examples', type: 'rsync', rsync__exclude: '.git/'
+  config.vm.synced_folder '../rflow-components-http', '/rflow-components-http'
+
+  # forward http for rflow testing
+  config.vm.network "forwarded_port", guest: 8000, host: 8000
+
+  # install RPM dependencies for rflow and zeromq
+  config.vm.provision "shell", privileged: true, inline: <<-EOS
+    curl -O https://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
+    rpm -ivh epel-release-6-8.noarch.rpm
+    yum -y install libyaml-devel patch libffi-devel glibc-headers autoconf gcc-c++ glibc-devel readline-devel zlib-devel openssl-devel automake libtool bison git sqlite-devel rpm-build libuuid-devel vim
+  EOS
+
+  # build zeromq as vagrant user
+  config.vm.provision "shell", privileged: false, inline: <<-EOS
+    curl -O http://download.zeromq.org/zeromq-3.2.4.tar.gz
+    rpmbuild -tb zeromq-3.2.4.tar.gz
+  EOS
+
+  # install zeromq
+  config.vm.provision "shell", privileged: true, inline: <<-EOS
+    rpm -ivh ~vagrant/rpmbuild/RPMS/x86_64/zeromq-*
+  EOS
+
+  # set up RVM and bundler
+  config.vm.provision "shell", privileged: false, inline: <<-EOS
+    rm -f .profile
+    curl -sSL https://get.rvm.io | bash -s stable
+    source .rvm/scripts/rvm
+    rvm install `cat /rflow/.ruby-version`
+    cd /rflow
+    bundle update
+  EOS
+end

data/bin/rflow

@@ -66,7 +66,12 @@ end
 
 # Now require rflow because the following parts of the startup require
 # pieces (usually RFlow::Configuration or RFlow.logger)
-require 'rflow'
+begin
+  require 'rflow'
+rescue Exception => e
+  STDERR.puts "Error loading RFlow: #{e.class} - #{e.message}"
+  exit 1
+end
 
 # Set up the startup logging, which is distinct from the runtime
 # logging that is defined in the config database.  The startup logging

data/example/basic_extensions.rb

@@ -14,21 +14,20 @@ end
 RFlow::Configuration.add_available_data_extension('RFlow::Message::Data::Integer', SimpleDataExtension)
 
 class RFlow::Components::FileOutput < RFlow::Component
-  attr_accessor :output_file_path, :output_file
+  attr_accessor :output_file_path
   input_port :in
 
   def configure!(config)
     self.output_file_path = config['output_file_path']
-    self.output_file = File.new output_file_path, 'w+'
   end
 
   def process_message(input_port, input_port_key, connection, message)
-    output_file.puts message.data.data_object.inspect
-    output_file.flush
-  end
-
-  def cleanup
-    output_file.close
+    File.open(output_file_path, 'a') do |f|
+      f.flock(File::LOCK_EX)
+      f.puts message.data.data_object.inspect
+      f.flush
+      f.flock(File::LOCK_UN)
+    end
   end
 end
 

data/example/http_extensions.rb

@@ -5,21 +5,20 @@ require 'eventmachine'
 require 'evma_httpserver'
 
 class RFlow::Components::FileOutput < RFlow::Component
-  attr_accessor :output_file_path, :output_file
+  attr_accessor :output_file_path
   input_port :in
 
   def configure!(config)
     self.output_file_path = config['output_file_path']
-    self.output_file = File.new output_file_path, 'w+'
   end
 
   def process_message(input_port, input_port_key, connection, message)
-    output_file.puts message.data.data_object.inspect
-    output_file.flush
-  end
-
-  def cleanup!
-    output_file.close
+    File.open(output_file_path, 'a') do |f|
+      f.flock(File::LOCK_EX)
+      f.puts message.data.data_object.inspect
+      f.flush
+      f.flock(File::LOCK_UN)
+    end
   end
 end