rflow 0.0.5 → 1.0.0a1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c322bc6cf9c3b4ccd46b13cf56d3c2460dc5f0be
-  data.tar.gz: af2d6fb3e7051a074c56fa10b70b5a02b23bb0d7
+  metadata.gz: 57c6f0b7c61b30886bbf0f4b2f65821aa5b1b0f9
+  data.tar.gz: 62f58d281509732effeca0c1a041df2668497b80
 SHA512:
-  metadata.gz: 5a3cd46af3c815d2cb5840d48a0e38f7e28dbe911276fac75d3466bef9d05d7d49e38baa010b5ce419a67aea5829d2f227d2ff74aa5e689f6c1e6109d882ad81
-  data.tar.gz: 539c61aca94e84e1ccb00acba1ef87c7a6556dc65180f9f905561b09e147eab83ff0a409dd01b84a1113ba820cf89c6470ee64c1ace68736232423ba3a1d9668
+  metadata.gz: 8d74949a15024641aef4123ca703d2f2ccf6fb5f97dca9829a282ca53bd6d36c347c844b189255955c7fa058bf903853d0c3acf13fda4dc2e2b3f40e49129310
+  data.tar.gz: f6233d9cc128220c886b6ed4970b544040cace77af6701b6f7429da304ad7de00b5536455469e21988f37d2fa1faaedf3f4324f08eee343669b03c6bdaece735

data/.ruby-gemset

@@ -0,0 +1 @@
+rflow-dev

data/.ruby-version

@@ -0,0 +1 @@
+ruby-2.1.1

data/.travis.yml

@@ -0,0 +1,21 @@
+language: ruby
+
+rvm:
+  - 1.9.3
+  - 2.0.0
+  - 2.1.0
+
+before_install:
+  - sudo apt-get install libtool autoconf automake uuid-dev build-essential
+  - wget http://download.zeromq.org/zeromq-3.2.4.tar.gz && tar zxvf zeromq-3.2.4.tar.gz && cd zeromq-3.2.4 && ./configure && make && sudo make install && cd ..
+# Only has 4.0.4, need 3.2 version due to old em-zeromq
+#  - sudo add-apt-repository -y ppa:chris-lea/zeromq
+#  - sudo apt-get update
+#  - sudo apt-get install libzmq3 libzmq3-dev
+
+script: bundle exec rspec spec
+
+notifications:
+  hipchat:
+    rooms:
+      secure: a4nrCmDPwhteJA65QFRlBdnsknT+4y/JtZL/sLPCObOahFWvLOXMggPXvHAOssCaa2ydYrMMvWNliOz63nuu3qAnR90H7aOU3o+2K3zeACy0cAjF27lDonLhaYHeUz07oPwr/iDlFC8bDfFDempjIFFnXSc/LhUWaCltnJ7W5vI=

data/.yardopts

@@ -0,0 +1 @@
+--output ./doc --main README.md --files schema/*.avsc lib/**/*.rb bin/*.rb - README.md LICENSE

data/Gemfile

@@ -1,5 +1,9 @@
-source "http://rubygems.org"
+source "https://rubygems.org"
 
 # Specify your gem's dependencies in rflow.gemspec
 gemspec
 
+group :development do
+  gem 'guard'
+  gem 'guard-rspec'
+end

data/Guardfile

@@ -0,0 +1,8 @@
+# A sample Guardfile
+# More info at https://github.com/guard/guard#readme
+
+guard :rspec do
+  watch(%r{^spec/.+_spec\.rb$})
+  watch(%r{^lib/(.+)\.rb$})     { |m| "spec/lib/#{m[1]}_spec.rb" }
+  watch('spec/spec_helper.rb')  { "spec" }
+end

data/LICENSE

@@ -0,0 +1,190 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2014 RedJack LLC
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/NOTES

@@ -1,3 +1,16 @@
+RFlow starts
+read in DB
+create new shards
+  - Create a set of workers with the shard configuration
+    - each worker creates a set of components
+
+  - create components
+
+
+
+
+
+
 RFlow Manager
 
   Components
@@ -20,12 +33,12 @@ rflow <config file>
     - place pid files in deployment's run directory
   Configure components via zmq
   Daemonize self
-  
+
 
 
 class Component
   def self.input_port
-  end 
+  end
 
   def self.output_port
   end
@@ -33,11 +46,11 @@ class Component
   attr_accessor :state
 
   def initialize(config, run_directory)
-    
+
   end
 
   def run
-    
+
   end
 
   def configure
@@ -56,7 +69,7 @@ class PassThrough < Component
     # This will initialize the ports
     super
     # Do stuff to initialize component
-  end   
+  end
 
 end
 
@@ -66,7 +79,7 @@ Computation Requirements:
     - management bus connection information
     - group and instance UUID
     - beacon interval
-    - run directory, containing 
+    - run directory, containing
       - PID files
       - log dir + logs
       - computation-specific configuration (conf dir)
@@ -90,7 +103,7 @@ Computation Requirements:
 
 External Computations:
   - Given (out-of-band) startup info (mgmt bus, UUIDs, beacon interval)
-  - 
+  -
 
 
 RFlow
@@ -100,7 +113,7 @@ RFlow
 
 Translate
   - Need to add <associated type="objtype" name="myname"> where name attr can be used in later XML templates
-      
+
 
 
 
@@ -112,7 +125,7 @@ Plugins:
     - necessary to tell system?
     - need a protocol for defining schema transfer
     - each message has attached schema
-  
+
 
 lib/rflow/message.rb
 
@@ -122,7 +135,7 @@ RFlow::Management
   - Somewhere for external people to register new computations with running system
   - computation says that its running and asks for Connection configuration
     - how will it specify where in the workflow it wants to run????
-  
+
 RFlow::Message(complete on-the-wire Avro message format)
   data_type, provenance, external_ids, empty, data (see below)
 
@@ -142,7 +155,7 @@ RFlow::Connection::AMQP
 
 RFlow::Connection::ZMQ
 
-    
+
 
 
 computation_a.output_port -> (connection.incoming -> connection.outgoing) -> computation_b.input_port
@@ -152,12 +165,12 @@ AMQP::Topic - responsible for setting up a topic -> queue binding
   r.outgoing = amqp connection, channel, vhost, login, password, queue name
   behavior -> n x m, "round-robin" among the connected outgoing
     incoming behavior will need to set topic/key, uses the data type in the RFlow::Message
-    
+
 
 ZMQ::PubSub - device-less, responsible for assigning ip/port and assigning one client to bind the port
   r.incoming = zmq connection string (tcp://ip:port), type pub
   r.outgoing = zmq connection string (tcp://ip:port), type sub
-  behavior -> n x m, broadcast sending, 
+  behavior -> n x m, broadcast sending,
 
 ZMQ::PushPull - device-less, responsible for assigning ip/port and assigning one client to bind the port
   r.incoming = zmq connection string (tcp://ip:port), type push

data/README.md

@@ -0,0 +1,448 @@
+# RFlow
+
+[![Build Status](https://travis-ci.org/redjack/rflow.png?branch=master)](https://travis-ci.org/redjack/rflow)
+
+RFlow is a Ruby framework inspired by
+[flow-based programming](http://en.wikipedia.org/wiki/Flow-based_programming)
+(FBP), which was previously inspired by
+[Communicating Sequential Processes](http://en.wikipedia.org/wiki/Communicating_sequential_processes)
+(CSP). It has some conceptual similarities to Javascript's
+[NoFlo](http://noflojs.org/) system, Java's
+[Storm](http://storm.incubator.apache.org/), and Clojure's
+[core.async](http://clojure.github.io/core.async/) library.
+
+In short, components communicate with each other by sending/receiving
+messages via their output/input ports over connections. Ports are
+"wired" together output->input with connections, and messages are
+explicitly serialized before being sent over the connection. RFlow
+supports generalized connection types and message serialization,
+however only two are in current use, namely ZeroMQ connections and
+Avro serialization.
+
+RFlow currently runs as a single-threaded, evented system on top of
+[Eventmachine](http://rubyeventmachine.com/), meaning that any code
+should be coded in an asynchronous style so as to not block the
+Eventmachine reactor (and thus block all the other components). There
+is currently work being done to "shard" the workflow among multiple
+processes and/or threads.
+
+Some of the long-term goals of RFlow are to allow for components and
+portions of the workflow to be defined in any language that supports
+Avro and ZeroMQ, which a numerous.
+
+
+## Definitions
+
+* __Component__ - the basic unit of RFlow computation. Each
+  component is a shared-nothing, individual computation module that
+  communicates with the rest of the system through explicit message
+  passing via input and output ports.
+
+* __Port__ - a named entity on each component that is responsible for
+  receiving data (and input port) or sending data (and output port).
+  Ports can be "keyed" or "indexed" to allow better multiplexing of
+  messages out/in a single port, as well as allow a single port to be
+  accessed by an array.
+
+* __Connection__ - a directed link between an output port and an input
+  port.  RFlow supports generalized connection types, however only
+  ZeroMQ IPC links are currently used.
+
+* __Message__ - a bit of serialized data that is sent out an output
+  port and recieved on an input port. Due to the serialization,
+  message types and schemas are explicitly defined. In a departure
+  from "pure" FBP, RFlow supports sending multiple message types via a
+  single connection.
+
+* __Workflow__ - the common name for the digraph created when the
+  components (nodes) are wired together via connections to their
+  respective output/input ports.
+
+
+## Component Examples
+
+The following describes the API of an RFlow component:
+
+```ruby
+class SimpleComponent < RFlow::Component
+  input_port :in
+  output_port :out
+
+  def configure!(config); end
+  def run!; end
+  def process_message(input_port, input_port_key, connection, message); end
+  def shutdown!; end
+  def cleanup!; end
+end
+```
+
+* `input_port` and `output_port` define the named ports that will
+  receive data or send data, respectively. These class methods create
+  accessors for their respective port names, to be used later in the
+  `process_message` or `run!` methods. There can be multiple (or no)
+  input and output ports.
+
+* `configure!` (called with a hash configuration) is called after the
+  component is instantiated but before the workflow has been wired or
+  any messages have been sent. Note that this is called outside the
+  Eventmachine reactor.
+
+* `run!` is called after all the components have been wired together
+  with connections and the entire workflow has been created. For a
+  component that is a source of messages, this is where messages will
+  be sent. For example, if the component is reading from a file, this
+  is where the file will be opened, the contents read into a message,
+  and the message sent out the output port. `run!` is called within
+  the Eventmachine reactor.
+
+* `process_message` is an evented callback that is called whenever the
+  component receives a message on one of its input ports.
+  `process_message` is called withing the Eventmachine reactor
+
+* `shutdown!` is called when the flow is being terminated, and is
+  meant to allow the components to do penultimate processing and send
+  any final messages. All components in a flow will be told to
+  `shutdown!` before they are told to `cleanup!`.
+
+* `cleanup!` is the final call to each component, and allow the
+  component to clean up any external resources that it might have
+  outstanding, such as file handles or network sockets.
+
+"Source" components will often do all of their work within the `run!`
+method, and often gather message data from an external source, such as
+file, database, or network socket. The following component generates a
+set of integers between a configured start/finish, incrementing by a
+configured step:
+
+```ruby
+class RFlow::Components::GenerateIntegerSequence < RFlow::Component
+  output_port :out
+
+  def configure!(config)
+    @start = config['start'].to_i
+    @finish = config['finish'].to_i
+    @step = config['step'] ? config['step'].to_i : 1
+    # If interval seconds is not given, it will default to 0
+    @interval_seconds = config['interval_seconds'].to_i
+  end
+
+  # Note that this uses the timer (sometimes with 0 interval) so as
+  # not to block the reactor
+  def run!
+    timer = EM::PeriodicTimer.new(@interval_seconds) do
+      message = RFlow::Message.new('RFlow::Message::Data::Integer')
+      message.data.data_object = @start
+      out.send_message message
+      @start += @step
+      timer.cancel if @start > @finish
+    end
+  end
+end
+```
+
+"Middle" components receive messages on input port(s), perform a bit
+of computation, and then send a message out the output port(s). The
+following component accepts a Ruby expression string via its config,
+and then uses that as an expression to determine what port to send an
+incoming message:
+
+```ruby
+class RFlow::Components::RubyProcFilter < RFlow::Component
+  input_port :in
+  output_port :filtered
+  output_port :dropped
+  output_port :errored
+
+  def configure!(config)
+    @filter_proc = eval("lambda {|message| #{config['filter_proc_string']} }")
+  end
+
+  def process_message(input_port, input_port_key, connection, message)
+    begin
+      if @filter_proc.call(message)
+        filtered.send_message message
+      else
+        dropped.send_message message
+      end
+    rescue Exception => e
+      errored.send_message message
+    end
+  end
+end
+```
+
+"Sink" components accept messages on an input port and do not have an
+output port. They often operate on external sinks, such as writing
+messages to a file, database, or network socket. The following
+component writes the inspected message to a file (defined via the
+configuration):
+
+```ruby
+class RFlow::Components::FileOutput < RFlow::Component
+  input_port :in
+
+  attr_accessor :output_file_path, :output_file
+
+  def configure!(config)
+    self.output_file_path = config['output_file_path']
+    self.output_file = File.new output_file_path, 'w+'
+  end
+
+  def process_message(input_port, input_port_key, connection, message)
+    output_file.puts message.data.data_object.inspect
+    output_file.flush
+  end
+
+  def cleanup
+    output_file.close
+  end
+end
+```
+
+## RFlow Messages
+
+RFlow messages are instances of
+[`RFlow::Message`](lib/rflow/message.rb), which are ultimately
+serialized via an Avro [schema](schema/message.zvsc).
+
+There are two parts of the message "envelope": a provenance and the
+embedded data object "payload".
+
+The `provenance` is a way for a component to annotate a message with a
+bit of data that should (by convention) be carried through the
+workflow with the message, as well as being copied to derived
+messages. For example, a TCP server component would spin up a TCP
+server and, upon recieving a connection and packets on a session, it
+would marshal the packets into `RFlow::Messsage`s and send them out
+its output ports. Messages received on its input port, however, need
+to have a way to be matched to the corresponding underlying TCP
+connection. `provenance` provides a method for the TCP server
+component to add a bit of metadata (namely an identifier for the TCP
+connection) such that later messages that contain the same provenance
+can be matched to the correct underlying TCP connection.
+
+
+The other parts of the message envelope are related to the embedded
+data object. In addition to the data object itself (which is encoded
+with a specific Avro schema), there are a few fields that describe the
+embedded data, namely the `data_type_name`, the
+`data_serialization_type`, and the `data_schema`. By including all
+this metadata in each message, the system is completely dynamic and
+allow for multiple message types to be included on a single
+connection, as well as enabling non-RFlow components to be created in
+any language. This does come at the expense of larger messages which
+results in greater message overhead.
+
+For example, if we have a simple integer data type that we would like
+to serialize via Avro, we can register the schema with the following
+`add_available_data_type` code shown below:
+
+```ruby
+long_integer_schema = '{"type": "long"}'
+RFlow::Configuration.add_available_data_type('RFlow::Message::Data::Integer', 'avro', long_integer_schema)
+```
+
+This will make the schema and message type available to RFlow, such
+that it will be able to create a new message with:
+
+```ruby
+message = RFlow::Message.new('RFlow::Message::Data::Integer')
+```
+
+and will automatically reconstitute a message from the connection and
+call a component's `process_message`.
+
+The deserialized Avro Ruby object is available as the `data_object`
+accessor on the `data` class, i.e.:
+
+```ruby
+message.data.data_object = 1024
+```
+
+The `data_object` is the deserialized Avro Ruby object and, as such,
+allows the Avro object to be accessed as a Ruby object. In order to
+provide a more convenient interface to the underlying Avro object,
+RFlow allows modules to be dynamically mixed in to the `data` class
+object.
+
+For example, the module below provides a bit of extra functionality to
+the above-mentioned `RFlow::Message::Data::Integer` message type,
+namely to default the integer to 0 upon being mixed in, provide a
+better named accessor, and add a `default?` method to the `data` object:
+
+```ruby
+module SimpleDataExtension
+  def self.extended(base_data)
+    base_data.data_object = 0
+  end
+
+  def int; data_object; end
+  def int=(new_int); data_object = new_int; end
+
+  def default?;
+    data_object == 0
+  end
+end
+```
+
+Once a module is defined, it needs to be registered to the appropriate
+message data type.  Note that multiple modules can be registered for a
+given message data type.
+
+```ruby
+RFlow::Configuration.add_available_data_extension('RFlow::Message::Data::Integer', SimpleDataExtension)
+```
+
+The result of this is that the following code will work:
+
+```ruby
+message = RFlow::Message.new('RFlow::Message::Data::Integer')
+message.data.int == 0   # => true
+message.data.default?   # => true
+message.data.int = 1024
+messaga.data.default?   # => false
+```
+
+
+## RFlow Workflow Configuration
+
+RFlow currently stores its configuration in a SQLite database which
+are internally accessed via ActiveRecord.  Given that SQLite is a
+rather simple and standard interface, non-RFlow components could
+access it and determine their respsective ZMQ connections.
+
+DB schemas for the configuration database are in
+[lib/rflow/configuration/migrations](lib/rflow/configuration/migrations)
+and define the complete workflow configuration.  Note that each of the
+tables uses a UUID primary key, and UUIDs are used within RFlow to
+identify specific components.
+
+* settings - general application settings, such as log levels, app
+  names, directories, etc
+
+* components - a list of the components including its name,
+  specification (Ruby class), and options. Note that the options are
+  serialized to the database as YAML, and components should understand
+  that the round-trip through the database might not be perfect (e.g.
+  Ruby symbols might become strings). A component also has a number of
+  input ports and output ports.
+
+* ports - belonging to a component (via `component_uuid` foreign key),
+  also has a `type` colum for ActiveRecord STI, which gets set to
+  either a `RFlow::Configuration::InputPort` or
+  `RFlow::Configuration::OutputPort`.
+
+* connections - a connection between two ports via foriegn keys
+  `input_port_uuid` and `output_port_uuid`. Like ports, connections
+  are typed via AR STI (`RFlow::Configuration::ZMQConnection` or
+  `RFlow::Configuration::AMQPConnection`) and have a YAML serialized
+  `options` hash. A connection also (potentially) defines the port
+  keys.
+
+RFlow also provides a RubyDSL for configuration-like file to be used
+to load the database:
+
+```ruby
+RFlow::Configuration::RubyDSL.configure do |config|
+  # Configure the settings, which include paths for various files, log
+  # levels, and component specific stuffs
+  config.setting('rflow.log_level', 'DEBUG')
+  config.setting('rflow.application_directory_path', '../tmp')
+
+  config.setting('rflow.application_name', 'testapp')
+
+  # Instantiate components
+  config.component 'generate_ints1', 'RFlow::Components::GenerateIntegerSequence', {
+    'start' => 0,
+    'finish' => 10,
+    'step' => 3,
+    'interval_seconds' => 1
+  }
+  config.component 'generate_ints2', 'RFlow::Components::GenerateIntegerSequence', {
+    'start' => 20,
+    'finish' => 30
+  }
+  config.component 'filter', 'RFlow::Components::RubyProcFilter', {
+    'filter_proc_string' => 'lambda {|message| true}'
+  }
+  config.component 'output1', 'RFlow::Components::FileOutput', {
+    'output_file_path' => '/tmp/out1'
+  }
+  config.component 'output2', 'RFlow::Components::FileOutput', {
+    'output_file_path' => '/tmp/out2'
+  }
+
+  # Wire components together
+  config.connect 'generate_ints1#out' => 'filter#in'
+  config.connect 'generate_ints2#out' => 'filter#in'
+  config.connect 'filter#filtered' => 'replicate#in'
+  config.connect 'filter#out' => 'output1#in'
+  config.connect 'filter#filtered' => 'output2#in'
+end
+```
+
+## Command-Line Operation
+
+RFlow includes the `rflow` binary that can load a database from a Ruby
+DSL, as well as start/stop the wokflow application as a daemon.
+Invoking the `rflow` binary without any options will give a brief help:
+
+```
+Usage: rflow [options] (start|stop|status|load)
+    -d, --database DB                Config database (sqlite) path (GENERALLY REQUIRED)
+    -c, --config CONFIG              Config file path (only valid for load)
+    -e, --extensions FILE1[,FILE_N]  Extension file paths (will load)
+    -g, --gems GEM1[,GEM_N]          Extension gems (will require)
+    -l, --log LOGFILE                Initial startup log file (in addition to stdout)
+    -v, --verbose [LEVEL]            Control the startup log (and stdout) verbosity (DEBUG, INFO, WARN) defaults to INFO
+    -f                               Run in the foreground
+        --version                    Show RFlow version and exit
+    -h, --help                       Show this message and exit
+```
+
+In general, the process for getting started is to first create a
+configuration database via `rflow load`:
+
+```
+rflow load -d my_config.sqlite -c my_ruby_dsl.rb
+```
+
+which will create the `my_config.sqlite` configuration database loaded
+with the `my_ruby_dsl.rb` configuration DSL.
+
+Once a config database exists, you can start up the application that
+it describes with `rflow start`. The `--extensions` argument allows
+loading of arbitrary Ruby code (via Ruby's `load`), which is usually
+where the component implementations are stored, as well as data type
+registrations.
+
+```
+rflow start -d my_config.sqlite -e my_component.rb,my_other_component.rb,my_data_type.rb
+```
+
+By default, RFlow will daemonize, write its pid file to
+`./run/app.pid` and write its log file to `./log/app.log`.  The `-f`
+flag will keep RFlow in the foreground.
+
+RFlow also supports two signals that allow for useful management of a
+running RFlow daemon's log. Sending a `SIGUSR1` to the running RFlow
+process will cause RFlow to close and reopen its log file, which
+allows for easy log management without restarting RFlow. In addition,
+sending a `SIGUSR2` will toggle RFlow's log-level to `DEBUG`, and a
+subsequent `SIGUSR2` will toggle the log-level back to what was
+originally set. This allows for easy debugging of a running RFlow
+process.
+
+   Copyright 2014 RedJack LLC
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.