magritte 0.5.0

data/LICENSE

@@ -0,0 +1,26 @@
+Copyright (c) 2013 MyDrive Solutions Limited, All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+    * Redistributions of source code must retain the above copyright
+    notice, this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above
+    copyright notice, this list of conditions and the following
+    disclaimer in the documentation and/or other materials provided
+    with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
+BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
+LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
+ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,128 @@
+Magritte
+========
+This is a simple but powerful wrapper to Open3 pipes that makes it
+easy to handle two-way piping of data into and out of a sub-process.
+Various input IO wrappers are supported and output can either be
+to an IO or to a block. A simple line buffer class is also provided,
+to turn block writes to the output block into line-by-line output
+to make interacting with the sub-process easier.
+
+What it Does
+------------
+You have a sub-command that you want to put data into and from which
+you want to retrieve the output, much like a Unix command line pipe.
+This is a non-trivial operation involving non-blocking reads and writes, the
+checking of the state of the input and output IOs, etc.  Magritte
+abstracts all of that behind an easy to use, fluent interface.
+
+
+Usage
+-----
+
+####Simplest Use Case
+For purposes of showing a simple example, let's say you wanted to
+use the command line tool `grep` to filter some input data.  Yes,
+you can do this natively in Ruby, but it's a trivial and easy to
+understand example. The normal use case would be wrapping an existing
+custom command line tool with Ruby. 
+
+But, back to `grep`. To store the output into a `StringIO` you could
+do the following:
+
+```ruby
+buffer = StringIO.new
+Magritte::Pipe.from_input_file('some.txt')
+  .out_to(buffer)
+  .filtering_with('grep "relistan"')
+```
+
+This example will take the contents of `some.txt` and stream it through
+`grep "relistan"`, storing the results in `buffer`.
+
+####String as Input
+
+```ruby
+data   = "foo\nfoo\nrelistan\n"
+buffer = StringIO.new
+
+Magritte::Pipe.from_input_string(data)
+  .out_to(buffer)
+  .filtering_with('grep "relistan"')
+```
+
+####IO Stream as Input
+
+```ruby
+buffer = StringIO.new
+socket = Socket.new(xxx)
+
+Magritte::Pipe.from_input_stream(socket)
+  .out_to(buffer)
+  .filtering_with('grep "relistan"')
+```
+
+####Output to a Block
+
+Rather than outputting the results to a stream, you can provide a block
+to `out_to` which will be invoked on each read of output data from the
+sub-process.  This allows you to process the data in a stream-like
+manner without having to buffer all of the output and then process it
+
+NOTE: Like a call to `IO.write`, the block _must_ return the number of 
+bytes processed.  This is fed back to the buffering process to make
+sure that the next iteration of data will include any missed bytes 
+when sent to your output block.
+
+```ruby
+Magritte::Pipe.from_input_file('some.txt')
+  .out_to { |data| $stdout.write data; data.size }
+  .filtering_with('grep "relistan"')
+```
+
+####Line Buffering
+
+When passing data into your block, it's often much easier to work on
+it if you can access it line-by-line rather than as a stream of data.
+Magritte supports this with a provided `LineBuffer` class that is
+wrapped into the API. You simply call `line_by_line` and your `out_to`
+block will be invoked on each line, one at a time.  Note that when
+using the `LineBuffer` you do *not* need to specify the number of bytes
+written by your `out_to` block as the `LineBuffer` handles this for you.
+
+```ruby
+Magritte::Pipe.from_input_file('some.txt')
+  .line_by_line
+  .out_to { |data| puts data }
+  .filtering_with('grep "relistan"')
+```
+
+Exit Status
+-----------
+Magritte will raise an `Errno::EPIPE` in the event of a non-zero
+status code in the sub-process.
+
+Limitations
+-----------
+To simplify implementation, Magritte uses `Open3.popen2e` which combines
+`stderr` and `stdout` on the output stream.  This means that in the event
+of an error in the sub-process, any output will be contained in the same
+output stream as the rest of the data.  I've found that ordinarily this
+is what you want, but it might not work for all situations.  If there is
+enough interest, I may implement a more complicated alternative later.
+
+In line-by-line mode with an output block provded to `.out_to`, the output
+*must* provide a terminating record separator or the last line will not
+be passed to the block.
+
+Credits
+-------
+This software was written by [Karl Matthias](https://github.com/relistan).
+[The name](http://en.wikipedia.org/wiki/The_Treachery_of_Images) 
+was suggested by [Gavin Heavyside](https://github.com/gavinheavyside).
+Magritte was developed with the support of 
+[MyDrive Solutions Limited](http://mydrivesolutions.com).
+
+License
+-------
+This plugin is released under the BSD two clause license which is
+available in both the Ruby Gem and the source repository.

data/lib/magritte.rb

@@ -0,0 +1,3 @@
+Dir[File.join(File.dirname(__FILE__), 'magritte', '*')].each do |file|
+  require file
+end

data/lib/magritte/line_buffer.rb

@@ -0,0 +1,32 @@
+module Magritte
+  class LineBuffer
+    include Enumerable
+
+    attr_reader :buffer
+
+    def initialize(record_separator="\n")
+      @buffer = ""
+      @record_separator = record_separator
+    end
+
+    def write(data)
+      last_eol = data.rindex(@record_separator)
+      return 0 unless last_eol
+
+      data = data[0..(last_eol + @record_separator.size - 1)]
+      @buffer += data
+      data.size
+    end
+
+    def each_line(&block)
+      raise ArgumentError.new("No block passed to each_line!") unless block_given?
+      return if buffer.empty?
+
+      lines = @buffer.split(@record_separator)
+      lines.each(&block)
+      @buffer = ""
+    end
+
+    alias_method :each, :each_line
+  end
+end

data/lib/magritte/pipe.rb

@@ -0,0 +1,171 @@
+require 'open3'
+require 'stringio'
+require_relative 'line_buffer'
+
+# Acts as a two way pipe like the shell command line. We put
+# data into a sub-process and capture the output.
+
+module Magritte
+  class Pipe
+    READ_BLOCK_SIZE = 512
+
+    def initialize(input, output=nil)
+      @input  = input
+      @output = output
+      @line_by_line = false
+    end
+
+    def self.from_input_file(infile)
+      self.from_input_stream(File.open(infile))
+    end
+
+    def self.from_input_stream(io)
+      new(io)
+    end
+
+    def self.from_input_string(str)
+      new(StringIO.new(str || ""))
+    end
+
+    def out_to(io=nil, &block)
+      if block_given?
+        @output = Proc.new(&block)
+      else
+        @output = io if io
+      end
+
+      self
+    end
+
+    def line_by_line
+      @line_by_line = true
+      @buffer = LineBuffer.new
+      self
+    end
+
+    def filtering_with(command)
+      raise "No output IO is set! Invoke out_to first!" unless @output
+
+      Open3.popen2e(command) do |subproc_input, subproc_output, wait_thr|
+        @subproc_output = subproc_output
+        @subproc_input  = subproc_input
+
+        clear_buffers
+
+        while true do
+          read_ready, write_ready, = select
+
+          read_from_input
+
+          if write_ready
+            write_to_subproc
+          end
+
+          if read_ready
+            read_from_subproc
+            send_output
+          end
+
+          # We close the input to signal to the sub-process that we are
+          # done sending data. It will close its output when processsing
+          # is completed. That signals us to stop piping data.
+          if ready_to_close?
+            @subproc_input.flush
+            @subproc_input.close
+          end
+
+          break if @subproc_output.closed?
+        end
+
+        raise Errno::EPIPE.new("sub-process dirty exit!") unless wait_thr.value == 0
+      end
+    end
+
+    private
+
+    def clear_buffers
+      @write_data = ""
+      @read_data  = ""
+    end
+
+    def select
+      output = descriptor_array_for(@subproc_output)
+      input  = descriptor_array_for(@subproc_input)
+      IO.select(output, input, nil, 0.01)
+    end
+
+    def descriptor_array_for(stream)
+      stream.closed? ? nil : [stream]
+    end
+
+    def write_to(io, data)
+      return 0 if data.empty?
+      return 0 if io.closed?
+
+      begin
+        io.write_nonblock(data)
+      rescue Errno::EAGAIN
+        return 0
+      end
+    end
+
+    def write_to_subproc
+      bytes_written = write_to(@subproc_input, @write_data)
+      @write_data = @write_data[bytes_written..-1]
+    end
+
+    def read_from_subproc
+      @read_data += read_from(@subproc_output)
+    end
+
+    def read_from_input
+      @write_data += read_from(@input)
+    end
+
+    def send_output
+      bytes_written = if @line_by_line && @output.is_a?(Proc)
+        send_output_line_by_line
+      else
+        send_output_block
+      end
+
+      @read_data = @read_data[bytes_written..-1]
+    end
+
+    def send_output_line_by_line
+      bytes_written = @buffer.write(@read_data)
+      @buffer.each_line { |line| @output.call(line) }
+      bytes_written
+    end
+
+    def send_output_block
+      if @output.is_a?(Proc)
+        bytes_written = @output.call(@read_data)
+        raise 'output block must return number of bytes written!' unless bytes_written.is_a?(Fixnum)
+        bytes_written
+      else
+        write_to(@output, @read_data)
+      end
+
+    end
+
+    def read_from(io)
+      begin
+        data = io.read_nonblock(READ_BLOCK_SIZE) unless io.closed?
+      rescue EOFError
+        io.close
+      rescue Errno::EAGAIN
+      end
+
+      data || ""
+    end
+
+    def ready_to_close?
+      @input.closed? && @write_data.empty? && @read_data.empty? && !@subproc_input.closed?
+    end
+  end
+end
+
+#Magritte::Pipe.from_input_file(ARGV[0]).out_to($stdout).filtering_with("build/bin/snapper --dbname=nt2012q1")
+#Magritte::Pipe.from_input_stream($stdin).out_to($stdout).filtering_with("build/bin/snapper --dbname=nt2012q1")
+#Magritte::Pipe.from_input_stream($stdin).out_to { |input| asdf += input; input.size}.filtering_with("cat")

metadata

@@ -0,0 +1,77 @@
+--- !ruby/object:Gem::Specification
+name: magritte
+version: !ruby/object:Gem::Version
+  version: 0.5.0
+  prerelease: 
+platform: ruby
+authors:
+- Karl Matthias
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-05-11 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: &70302135231300 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: *70302135231300
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: &70302135230200 !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  type: :development
+  prerelease: false
+  version_requirements: *70302135230200
+description: ! '  Magritte is a simple but powerful wrapper to Open3 pipes that makes
+  it easy to handle two-way piping of data into and out of a sub-process. Various
+  input IO wrappers are supported and output can either be to an IO or to a block.
+  A simple line buffer class is also provided, to turn block writes to the output
+  block into line-by-line output to make interacting with the sub-process easier.
+
+'
+email: relistan@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/magritte/line_buffer.rb
+- lib/magritte/pipe.rb
+- lib/magritte.rb
+- README.md
+- LICENSE
+homepage: http://github.com/mydrive/magritte
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.11
+signing_key: 
+specification_version: 3
+summary: Simple but powerful wrapper of two-way pipes to/from a sub-process.
+test_files: []