deadly_serious 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 68d3c2f2fb290436ae2543707c16063f4747b7c4
-  data.tar.gz: 76d915c5453302c81cd2b9a0cb7171824212f340
+  metadata.gz: 3dfdbc7c45af05380d213ab4011d4228e2e8b6af
+  data.tar.gz: 089e715ec1cc97af1b01b43d6bc8133a33353bb3
 SHA512:
-  metadata.gz: 9796184360d511d9c3dc54ef94b31d965adbd25c54f347866118f3b6afd3b8beb73ce3a1a669c026bd6fc8426f8df0be7ff2f9deb25cdd9e5e29a5100d4a57e4
-  data.tar.gz: 27c775a1c743c74c34c707d1176302ccf9bd23a347d606dbe3e19b699953e871d7ca9f72dd9fab47c94d5747e2eb39570dbac7e4cdb84791959d363d3b2a34d9
+  metadata.gz: e1e87aca37f80261b0ab8f0d7e1786d5fa20d224bed7277a0eb72833200f9a47b198f888b1d77a6e15af3a2a479237d1715dccff0b460fafffe934012179e316
+  data.tar.gz: b5df200c7451bf4d058dea4b791b013c93c5ed0a5dbb3715a41deecee859c7a4249c42b5145a28fa3a8d6e881a46d9dffc046939f61a5d90492fc51e52485535

data/README.md

@@ -1,10 +1,26 @@
 # DeadlySerious
 
-Flow Based Programming Engine mechanically sympathetic to \*nix.
+Flow Based Programming Maestro!
 
-Flow Based Programming engine that relies on named pipes and Linux processes (sorry, it not works on Windows right now). That means it uses "mechanical sympathy" with the Operating System, i.e., the S.O. is *part* of the program, it's not something *below* it.
+This relies on [*named pipes*](http://linux.die.net/man/7/fifo) and *Linux processes* to create a program. Each component runs as a separate linux process and they exchange information through pipes.
 
-**REQUIRES** Ruby 2.0 and a \*nix based SO (tested on *Ubuntu* and *Arch Linux*)
+**REQUIRES** Ruby 2.0 and a \*nix based OS (Operating System, tested on *Ubuntu* and *Arch Linux*)
+
+Unlike [NoFlo](http://noflojs.org), this is not a real engine. It just "orchestrates" linux processes and pipes to create a flow based system.
+
+Overall, it's slower than a normal ruby program (the pipes add some overhead). However, there are 4 points where this approach is pretty interesting:
+
+1. High modifiabilty:
+  * The interfaces between each component is tiny and very clear: they are just streams of characteres. I usually use csv format or json when I need more structure than that.
+  * You can connect ruby process to anything else that deals with STDIN, STDOUT or files (which includes shell commands, of course).
+2. Cheap parallelism and distributed computation:
+  * Each component runs as a separated process. The OS is in charge here (and it does a amazing work running things in parallel).
+  * As any shell command can be use as a component, you can use a simple [ncat](http://nmap.org/ncat) (or something similar) to disttribute jobs between different boxes.
+  * It's really easy to avoid deadlocks and race conditions with the FBP paradigm.
+3. Low memory footprint
+  * As each component usually process things as they appear in the pipe, it's easy to crush tons of data using very little memory. Notable exceptions as components that needs to accumulate things to process, like "sort".
+4. Very easy to reason about (personal opinion):
+  * Of course, this is not a merit of this gem, but of Flow Based Programming in general. I dare do say (oh, blasphemy!) that Object Oriented and Functional programming paradigms are good ONLY for tiny systems. They make a huge mess on big ones (#prontofalei).
 
 ## Installation
 
@@ -22,7 +38,237 @@ Or install it yourself as:
 
 ## Usage
 
-TODO: Write usage instructions here
+### Basic pipeline
+
+Create a class that will orchestrate the pipeline:
+
+```ruby
+#!/usr/bin/env ruby
+# Assuming your are using RVM
+
+class Pipeline < DeadlySerious::Engine::Spawner
+  def run_pipeline
+    # Here comes the code
+  end
+end
+
+# This line will alow you to run
+# it directly from the shell.
+#
+# Please, note that you fires the pipeline
+# calling "run" not "run_pipeline".
+Pipeline.new.run if __FILE__ == $0
+```
+
+You can spawn process the following way:
+
+```ruby
+class Pipeline < DeadlySerious::Engine::Spawner
+  def run_pipeline
+
+    spawn_process(YourComponentClass,
+      readers: ['>an_awesome_text_file.txt'], # reads from a file
+      writers: ['your_first_output_pipe'])    # outputs to a pipe
+
+    spawn_process(YourOtherComponentClass,
+      readers: ['your_first_output_pipe'],
+      writers: ['more_pipe1', 'more_pipe2'])
+
+  end
+end
+```
+
+A component is any class with a "run" method and two named parameters "readers" and "writers":
+
+```ruby
+class EchoComponent
+  # "readers" and "writers" are both Array of IO objects.
+  def run(readers: [], writers: [])
+    reader = readers.first
+    writer = writers.first
+
+    reader.each_line do |line|
+      writer << line
+    end
+  end
+end
+```
+
+### Pipes and files
+
+The parameters you receive in the "def run(readers: [], writers: [])" method are [**IO**](http://www.ruby-doc.org/core-2.0/IO.html) objects.
+
+They are already opened when they are passed to your component, and they are properly closed when your component is done.
+
+In the Pipeline class, readers and writers are just pipe names *or* file names. If you want to read or write to a file instead of a pipe, prepend its name with ">", like this:
+
+```ruby
+spawn_process(YourComponentClass,
+  readers: ['>an_awesome_text_file.txt'], # reads from a file
+  writers: ['your_first_output_pipe'])    # outputs to a pipe
+
+spawn_process(YourComponentClass,
+  readers: ['an_awesome_pipe'],            # reads from a pipe
+  writers: ['>your_first_output_file'])    # outputs to a file
+```
+
+Files are read and created in the "./data" directory, "." being the directory where you fired the program.
+
+Pipes are created in the '/tmp/deadly_serious/&lt;pid&gt;/' directory and they live just during the program execution. Once it's done, the directory is deleted.
+
+### Shell commands
+
+Spawning shell commands are simples as that:
+
+```ruby
+spawn_command('cat ((>a_file_in_data_dir.csv)) | grep wabba > ((some_pipe))')
+spawn_command('cat ((some_pipe)) > ((>my_own_output_file.txt))')
+```
+
+The "((" and "))" are replaced by the actual pipe (or file) path before execution.
+
+### Preserve pipe directory and change data directory
+
+In the Pipeline class (the one you extended from Engine::Spawner), you can override the "initialize" method to pass some parameters:
+
+```ruby
+class Pipeline < DeadlySerious::Engine::Spawner
+  def initialize
+    super(
+      data_dir: './data',                             # Files directory
+      pipe_dir: "/tmp/deadly_serious/#{Process.pid}", # Pipes directory
+      preserve_pipe_dir: false)                       # Keeps the pipes directory after finish execution?
+  end
+end
+```
+
+You can overwrite any of them. The ones presented above are default.
+
+### JSON integration
+
+Yet to be explained.
+
+### Pre-made components
+
+  * Source components
+  * Splitter
+  * Joiner
+
+Yet to be explained.
+
+### Examples
+
+Here a simple program to deal with the "Telegram Problem" as first described by Peter Naur.
+
+> Write a program that takes a number **w**, then accepts lines of text and outputs lines of text, where the output lines have as many words as possible but are never longer than **w** characters. Words may not be split, but you may assume that no single word is too long for a line.
+
+```ruby
+require 'deadly_serious'
+
+module TelegramProblem
+  LINE_SIZE = 80
+  PARAGRAPH_PACKET = 'CONTROL PACKET: paragraph'
+  EOL_PACKET = 'CONTROL PACKET: end of line'
+
+  # Break text in words and "END OF LINE" packets (EOL)
+  class WordSplitter
+    def run(readers: [], writers: [])
+      reader = readers.first
+      writer = writers.first
+
+      reader.each do |line|
+        line.chomp!
+        line.scan(/$|\S+/) do |word|
+          packet = (word == '' ? EOL_PACKET : word)
+          writer << packet << "\n"
+        end
+      end
+    end
+  end
+
+  # Transform double "end of line" in "paragraph"
+  class EolToParagraph
+    def run(readers: [], writers: [])
+      reader = readers.first
+      writer = writers.first
+
+      last2 = ''
+      last1 = ''
+      reader.each do |packet|
+        packet.chomp!
+
+        if packet == EOL_PACKET
+          last2 = last1
+          last1 = packet
+          next
+        end
+
+        if last1 == EOL_PACKET && last2 == EOL_PACKET
+          writer << PARAGRAPH_PACKET << "\n"
+        end
+
+        writer << packet << "\n"
+        last2 = last1
+        last1 = packet
+      end
+    end
+  end
+
+  # Join words
+  class SentenceJoiner
+    def run(readers: [], writers: [])
+      reader = readers.first
+      writer = writers.first
+
+      line_size = 0
+
+      reader.each do |packet|
+        packet.chomp!
+
+        if packet == PARAGRAPH_PACKET
+          writer << "\n\n"
+          line_size = 0
+          next
+        end
+
+        if line_size + packet.size + 1 <= LINE_SIZE
+          writer << ' ' if line_size > 0
+          writer << packet
+          line_size += packet.size + 1
+        else
+          writer << "\n"
+          writer << packet
+          line_size = packet.size
+        end
+      end
+      writer << "\n" unless line_size == 0
+    end
+  end
+
+  class Pipeline < DeadlySerious::Engine::Spawner
+    include DeadlySerious
+    def run_pipeline
+      spawn_process(WordSplitter, readers: ['>war_and_peace.txt'], writers: ['words_and_eol'])
+      spawn_process(EolToParagraph, readers: ['words_and_eol'], writers: ['just_words'])
+      spawn_process(SentenceJoiner, readers: ['just_words'], writers: ['>output.data'])
+    end
+  end
+end
+
+if __FILE__ == $0
+  TelegramProblem::Pipeline.new.run
+end
+```
+
+Check the "examples" directory for other examples of use. There's even a version of this Telegram Problem program made without pipes and such, but using the same logic. I made it to have a "feeling" of the overhead of using pipes.
+
+In my findings, the overhead is roughly 100% in this very simple problem (same time, 2x cpu). Considering that each of the components above are *really* simple (just split, join words and an "if" and 2 pipes), I found the overhead not a great deal. However, I need more tests.
+
+## Future features (a.k.a. "The Wishlist")
+
+ * Socket connectors (pipe things through net)
+ * Remote coordination (create and running remote components from a master box)
+ * More pre-made components (using C?)
 
 ## Contributing
 

data/deadly_serious.gemspec

@@ -20,6 +20,7 @@ Gem::Specification.new do |spec|
 
   spec.add_development_dependency 'bundler', '~> 1.3'
   spec.add_development_dependency 'rake'
+  spec.add_development_dependency 'yard'
 
   spec.add_dependency 'json'
 end

data/lib/deadly_serious/engine/channel.rb

@@ -1,26 +1,28 @@
+require 'socket'
+
 module DeadlySerious
   module Engine
-    class Channel
-      def initialize(name, data_dir: nil, pipe_dir: nil)
-        matcher = name.match(/^(>?)(.*)$/)
-        @type = matcher[1] == '>' ? :file : :pipe
-        name = matcher[2]
-        @io_name = if @type == :file
-                     "#{data_dir}/#{name}"
-                   else
-                     "#{pipe_dir}/#{name}"
-                   end
-      end
-
-      # Create a pipe or file (acording to name)
-      # and returns the full name of the thing created.
-      def create
-        return @io_name if File.exist?(@io_name)
-        if @type == :file
-          `touch #{@io_name}`
+    # Fake class, it's actually a factory ¬¬
+    module Channel
+      def self.new(name, data_dir: nil, pipe_dir: nil)
+        matcher = name.match(/^(>)?(.*?)(?:(:)(\d{1,5}))?$/)
+        if matcher[1] == '>'
+          FileChannel.new(matcher[2], data_dir)
+        elsif matcher[3] == ':'
+          SocketChannel.new(matcher[2], matcher[4].to_i)
         else
-          `mkfifo #{@io_name}`
+          PipeChannel.new(matcher[2], pipe_dir)
         end
+      end
+    end
+
+    class FileChannel
+      def initialize(name, directory)
+        @io_name = File.join(directory, name)
+      end
+
+      def create
+        `touch #{@io_name}` unless File.exist?(@io_name)
         @io_name
       end
 
@@ -34,5 +36,53 @@ module DeadlySerious
         open(@io_name, 'w')
       end
     end
+
+    class PipeChannel
+      def initialize(name, directory)
+        @io_name = File.join(directory, name)
+      end
+
+      def create
+        `mkfifo #{@io_name}` unless File.exist?(@io_name)
+        @io_name
+      end
+
+      def open_reader
+        fail %(Pipe "#{@io_name}" not found) unless File.exist?(@io_name)
+        open(@io_name, 'r')
+      end
+
+      def open_writer
+        fail %(Pipe "#{@io_name}" not found) unless File.exist?(@io_name)
+        open(@io_name, 'w')
+      end
+    end
+
+    class SocketChannel
+      def initialize(host, port)
+        @host, @port = host, port
+        @retry_counter = 3
+      end
+
+      def create
+        # Do nothing
+      end
+
+      def open_reader
+        TCPSocket.new(@host, @port)
+      rescue Exception => e
+        @retry_counter -= 1
+        if @retry_counter > 0
+          sleep 1 and retry
+        else
+          raise e
+        end
+      end
+
+      def open_writer
+        server = TCPServer.new(@port)
+        server.accept
+      end
+    end
   end
 end

data/lib/deadly_serious/engine/spawner.rb

@@ -16,48 +16,12 @@ module DeadlySerious
         end
       end
 
-      def self.dasherize(a_string)
-        a_string.gsub(/(.)([A-Z])/, '\1-\2').downcase.gsub(/\W+/, '-')
-      end
-
-      def set_process_name(name)
-        $0 = "ruby #{self.class.dasherize(name)}"
-      end
-
-      def channel_for(pipe_name)
-        Channel.new(pipe_name, data_dir: @data_dir, pipe_dir: @pipe_dir)
-      end
-
-      def create_pipe(pipe_name)
-        channel_for(pipe_name).create
-      end
-
-      def read_pipe(pipe_name)
-        channel_for(pipe_name).open_reader
-      end
-
-      def write_pipe(pipe_name)
-        channel = channel_for(pipe_name)
-        return channel.open_writer unless block_given?
-
-        channel.open_writer do |io|
-          yield io
-        end
-      end
-
-      def fork_it
-        @ids << fork do
-          yield
-        end
-      end
-
-      def wait_children
-        @ids.each { |id| Process.wait(id) }
-      end
-
-      def kill_children
-        @ids.each { |id| Process.kill('SIGTERM', id) }
+      def run
+        run_pipeline
         wait_children
+      rescue Exception => e
+        kill_children
+        raise e
       end
 
       def spawn_source(a_class, *args, writer: self.class.dasherize(a_class.name))
@@ -94,12 +58,58 @@ module DeadlySerious
         @ids << spawn(command)
       end
 
-      def run
-        run_pipeline
+      private
+
+      # @!group Process Control
+
+      def fork_it
+        @ids << fork do
+          yield
+        end
+      end
+
+      def wait_children
+        @ids.each { |id| Process.wait(id) }
+      end
+
+      def kill_children
+        @ids.each { |id| Process.kill('SIGTERM', id) }
         wait_children
-      rescue Exception => e
-        kill_children
-        raise e
+      end
+
+      def set_process_name(name)
+        $0 = "ruby #{self.class.dasherize(name)}"
+      end
+
+      # @!endgroup
+      # @!group Channel Helpers
+
+      def channel_for(pipe_name)
+        Channel.new(pipe_name, data_dir: @data_dir, pipe_dir: @pipe_dir)
+      end
+
+      def create_pipe(pipe_name)
+        channel_for(pipe_name).create
+      end
+
+      def read_pipe(pipe_name)
+        channel_for(pipe_name).open_reader
+      end
+
+      def write_pipe(pipe_name)
+        channel = channel_for(pipe_name)
+        return channel.open_writer unless block_given?
+
+        channel.open_writer do |io|
+          yield io
+        end
+      end
+
+      # @!endgroup
+      # @!group Minor Helpers
+
+      def self.dasherize(a_string)
+        a_string.gsub(/(.)([A-Z])/, '\1-\2').downcase.gsub(/\W+/, '-')
       end
     end
   end

data/lib/deadly_serious/version.rb

@@ -1,3 +1,3 @@
 module DeadlySerious
-  VERSION = "0.3.0"
+  VERSION = "0.4.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: deadly_serious
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Ronie Uliana
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-10 00:00:00.000000000 Z
+date: 2013-08-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -38,6 +38,20 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: yard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: json
   requirement: !ruby/object:Gem::Requirement
@@ -105,3 +119,4 @@ summary: Flow Based Programming engine that relies on named pipes and Linux proc
   with the Operating System, i.e., the S.O. is *part* of the program, it's not something
   *below* it.
 test_files: []
+has_rdoc: