filter_io 0.1.0

data/.gitignore

@@ -0,0 +1,2 @@
+*.gemspec
+pkg

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 Jason Weathered
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.markdown

@@ -0,0 +1,103 @@
+# `filter_io`
+## Filter IO streams with a block. Ruby's FilterInputStream.
+
+`filter_io` is analogous to Java's `FilterIOStream` in that it allows you to intercept and process data in an IO stream. This is particularly useful when used to clean up bad input data when using a CSV or XML parser.
+
+`filter_io` provides a one-pass approach to filtering data which can be much faster and memory efficient than doing two passes (cleaning the source file into a buffer and then calling the original parser).
+
+`filter_io` has been tested against Ruby 1.8.7 and Ruby 1.9.2.
+
+### Installation
+
+You can install from Gemcutter by running:
+
+    sudo gem install filter_io
+
+### Example Usage
+
+#### A Simple Example: ROT-13
+
+    io = FilterIO.new io do |data|
+      data.tr "A-Za-z", "N-ZA-Mn-za-m"
+    end
+
+#### A Useful Example: Line Ending Normalisation
+
+A common usage of `filter_io` is to normalise line endings before parsing CSV data:
+
+    # open source stream
+    File.open(filename) do |io|
+      
+      # apply filter to stream
+      io = FilterIO.new(io) do |data, state|
+        # grab another chunk if the last character is a delimiter
+        raise FilterIO::NeedMoreData if data =~ /[\r\n]\z/ && !state.eof?
+        # normalise line endings to LF
+        data.gsub /\r\n|\r|\n/, "\n"
+      end
+      
+      # process resulting stream normally
+      FasterCSV.parse(io) do |row|
+        pp row
+      end
+      
+    end
+
+### Reference
+
+Call `FilterIO.new` with the original IO stream, any options and the filtering block. The returned object pretends like a normal read-only non-seekable IO stream.
+
+#### Block `state` parameter
+
+An optional second parameter to the block is the `state` parameter which contains stream metadata which may be useful when processing the chuck. The methods currently available are:
+
+* `bof?`: Returns true if this is the *first* chuck of the stream.
+* `eof?`: Returns true if this is the *last* chunk of the stream.
+
+#### `NeedMoreData`
+
+If the filtering block needs more data to be able to return anything, you can raise a `FilterIO::NeedMoreData` exception and `filter_io` will read another block and pass the additional data to you. This can be repeated as necessary until enough data is retrieved.
+
+For example usage of `NeedMoreData`, see the line ending normalisation example above.
+
+#### Re-buffering Unprocessed Data
+
+If your block is unable to process the whole chunk of data immediately, it can return both the processed chuck and the remainder to be processed later. This is done by returning a 2 element array: `[processed, unprocessed]`. If output is empty and unprocessed data is returned, `filter_io` will grab some more data and call the block again.
+
+Here's an example which processes whole lines and prepends the line length to the beginning of each line.
+
+    io = FilterIO.new io do |data, state|
+      output = ''
+      # grab complete lines until we hit EOF
+      while data =~ /(.*)\n/ || (state.eof? && data =~ /(.+)/)
+        output << "#{$1.size} #{$1}\n"
+        data = $'
+      end
+      # `output` contains the processed lines, `data` contains any left over partial line
+      [output, data]
+    end
+
+#### Block Size
+
+When either `readline`, `gets` or `read(nil)` is called, `filter_io` will process the input stream in 1,024 byte chucks. You can adjust this by passing a `:block_size` option to `new`.
+
+#### Character Encodings
+
+Ruby 1.9 has character encoding support can convert between UTF-8, ISO-8859-1, ASCII-8BIT, etc. This is triggered in `IO` by using `:external_encoding` and `:internal_encoding` when opening the stream.
+`filter_io` will use the underlying stream's encoding settings when reading and filtering data. The processing block will be passed data in the internal encoding.
+As per the core `IO` object, if `read` is called with a length (in bytes), the data will be returned in the external encoding.
+In summary, everything should Just Work&trade;
+
+### Note on Patches/Pull Requests
+ 
+* Fork the project.
+* Make your feature addition or bug fix.
+* Add tests for it. This is important so I don't break it in a
+  future version unintentionally.
+* Commit, do not mess with rakefile, version, or history.
+  (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
+* Send me a pull request. Bonus points for topic branches.
+
+### Copyright
+
+Copyright (c) 2010 Jason Weathered. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,32 @@
+require 'rubygems'
+require 'rake'
+require 'rake/testtask'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test the filter_io plugin.'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.libs << 'test'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+task :test => :check_dependencies
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gem|
+    gem.name = "filter_io"
+    gem.summary = "Filter IO streams with a block. Ruby's FilterInputStream."
+    gem.email = "jason@jasoncodes.com"
+    gem.homepage = "http://github.com/jasoncodes/filter_io"
+    gem.authors = ["Jason Weathered"]
+    gem.has_rdoc = false
+    gem.add_dependency 'activesupport'
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
+end

data/VERSION

@@ -0,0 +1 @@
+0.1.0

data/lib/filter_io.rb

@@ -0,0 +1,304 @@
+require 'active_support'
+
+class FilterIO
+  
+  DEFAULT_BLOCK_SIZE = 1024
+  
+  class NeedMoreData < Exception
+  end
+  
+  class BlockState
+    attr_reader :bof, :eof
+    def initialize(bof, eof)
+      @bof = bof
+      @eof = eof
+    end
+    alias_method :bof?, :bof
+    alias_method :eof?, :eof
+  end
+  
+  def initialize(io, options = nil, &block)
+    @io = io
+    @options = options || {}
+    @block = block
+    @pos = 0
+    @buffer = empty_string
+    @buffer_raw = empty_string_raw
+    @options.assert_valid_keys :block_size
+  end
+  
+  def pos
+    @pos
+  end
+  
+  def bof?
+    @pos == 0
+  end
+  
+  def eof?
+    @buffer.empty? && source_eof?
+  end
+  
+  def source_eof?
+    @buffer_raw.empty? && @io.eof?
+  end
+  
+  def readchar
+    raise EOFError, 'end of file reached' if eof?
+    if @io.respond_to? :external_encoding
+      data = empty_string_raw
+      begin
+        data << read(1).force_encoding(@io.internal_encoding || @io.external_encoding)
+      end until data.valid_encoding? or source_eof?
+      data.encode! @io.internal_encoding if @io.internal_encoding
+      data
+    else
+      read(1).ord
+    end
+  end
+  
+  def getc
+    readchar
+  rescue EOFError
+    nil
+  end
+  
+  def read(length = nil)
+    
+    raise ArgumentError if length && length < 0
+    return '' if length == 0
+    
+    # fill the buffer up to the fill level (or whole input if length is nil)
+    while !source_eof? && (length.nil? || length > bytesize(@buffer))
+      buffer_data @options[:block_size] || length
+    end
+    
+    # we now have all the data in the buffer that we need (or can get if EOF)
+    case
+    when bytesize(@buffer) > 0
+      # limit length to the buffer size if we were asked for it all or have ran out (EOF)
+      read_length = if length.nil? or length > bytesize(@buffer)
+        bytesize @buffer
+      else
+        length
+      end
+      data = pop_bytes read_length
+      @pos += bytesize(data)
+      if length.nil? && data.respond_to?(:encoding)
+        data.force_encoding @io.external_encoding
+        data.encode! @io.internal_encoding if @io.internal_encoding
+      end
+      data
+    when source_eof?
+      # end of file, nothing in the buffer to return
+      length.nil? ? empty_string : nil
+    else
+      raise IOError, 'Read error'
+    end
+    
+  end
+  
+  def rewind
+    seek 0, IO::SEEK_SET
+  end
+  
+  def seek(offset, whence = IO::SEEK_SET)
+    
+    new_pos = case whence
+    when IO::SEEK_SET
+      offset
+    when IO::SEEK_CUR
+      pos + offset
+    when IO::SEEK_END
+      raise Errno::EINVAL, 'SEEK_END not supported'
+    else
+      raise Errno::EINVAL
+    end
+    
+    case new_pos
+    when pos
+      # noop
+    when 0
+      @io.rewind
+      @pos = 0
+      @buffer = empty_string
+      @buffer_raw = empty_string_raw
+    else
+      raise Errno::EINVAL, 'Random seek not supported'
+    end
+    
+    0
+  end
+  
+  def ungetc(char)
+    char = char.chr if char.respond_to? :chr
+    @pos -= bytesize(char)
+    @pos = 0 if @pos < 0
+    @buffer = char + @buffer
+  end
+  
+  def gets(sep_string = $/)
+    
+    return nil if eof?
+    return read if sep_string.nil?
+    
+    paragraph_mode = sep_string == ''
+    sep_string = "\n\n" if paragraph_mode
+    sep_string = sep_string.to_s unless sep_string.is_a? String
+    
+    if paragraph_mode
+      # consume any leading newlines
+      char = getc
+      char = getc while char && char.ord == 10
+      if char
+        ungetc char # push the first non-newline back onto the buffer
+      else
+        return nil # nothing left except newlines, bail out
+      end
+    end
+    
+    # fill the buffer until it contains the separator sequence
+    until source_eof? or @buffer.index(sep_string)
+      buffer_data
+    end
+    
+    # calculate how much of the buffer to return
+    length = if idx = @buffer.index(sep_string)
+      # we found the separator, include it in our output
+      length = idx + sep_string.size
+    else
+      # no separator found (must be EOF). return everything we've got
+      length = @buffer.size
+    end
+    
+    # increment the position and return the buffer fragment
+    data = @buffer.slice!(0, length)
+    @pos += bytesize(data)
+    
+    data
+  end
+  
+  def readline(sep_string = $/)
+    gets(sep_string) or raise EOFError, 'end of file reached'
+  end
+  
+  def each_line(sep_string = $/)
+    unless block_given?
+      klass = defined?(Enumerator) ? Enumerator : Enumerable::Enumerator
+      return klass.new(self, :each_line, sep_string)
+    end
+    while line = gets(sep_string)
+      yield line
+    end
+    self
+  end
+  alias :each :each_line
+  alias :lines :each_line
+  
+  def readlines(sep_string = $/)
+    lines = []
+    each_line(sep_string) { |line| lines << line }
+    lines
+  end
+  
+  protected
+  
+  def empty_string
+    str = String.new
+    if @io.respond_to?(:internal_encoding)
+      str.force_encoding @io.internal_encoding || @io.external_encoding
+    end
+    str
+  end
+  
+  def empty_string_raw
+    str = String.new
+    if @io.respond_to?(:external_encoding)
+      str.force_encoding @io.external_encoding
+    end
+    str
+  end
+  
+  def bytesize(str)
+    str.respond_to?(:bytesize) ? str.bytesize : str.size
+  end
+  
+  def pop_bytes(count)
+    data = begin
+      if @io.respond_to?(:internal_encoding)
+        @buffer.force_encoding 'ASCII-8BIT'
+      end
+      @buffer.slice!(0, count)
+    ensure
+      if @io.respond_to?(:internal_encoding)
+        @buffer.force_encoding @io.internal_encoding || @io.external_encoding
+      end
+    end
+    data
+  end
+  
+  def buffer_data(block_size = nil)
+    
+    block_size ||= DEFAULT_BLOCK_SIZE
+    
+    data = unless @buffer_raw.empty?
+     @buffer_raw.slice! 0, bytesize(@buffer_raw)
+    else
+     @io.read(block_size) or return
+    end
+    
+    initial_data_size = bytesize(data)
+    begin
+      
+      data = process_data data, initial_data_size
+      
+      # if no processed data was returned and there is unprocessed data...
+      if data.is_a?(Array) && data.size == 2 && data[0].size == 0 && data[1].size > 0
+        # restore the unprocessed data into the temporary buffer
+        data = data[1]
+        # and add some more data to the buffer
+        raise NeedMoreData
+      end
+      
+    rescue NeedMoreData => e
+      raise EOFError, 'end of file reached' if eof?
+      data << @io.read(block_size)
+      retry
+    end
+    
+    data = [data] unless data.is_a? Array
+    raise 'Block must have 1 or 2 values' unless data.size <= 2
+    @buffer << data[0]
+    if data[1]
+      if @io.respond_to?(:internal_encoding) && @io.internal_encoding
+        data[1].convert! @io.external_encoding
+      end
+      @buffer_raw = data[1]
+    end
+    
+  end
+  
+  def process_data(data, initial_data_size)
+    
+    if data.respond_to? :encoding
+      org_encoding = data.encoding
+      data.force_encoding @io.external_encoding
+      additional_data_size = bytesize(data) - initial_data_size
+      unless data.valid_encoding? or source_eof? or additional_data_size >= 4
+        data.force_encoding org_encoding
+        raise NeedMoreData
+      end
+      data.encode! @io.internal_encoding if @io.internal_encoding
+    end
+    
+    if data && @block
+      state = BlockState.new @io.pos == data.length, source_eof?
+      args = [data, state]
+      args = args.first(@block.arity > 0 ? @block.arity : 1)
+      data = @block.call(*args)
+    end
+    
+    data
+  end
+  
+end

data/test/filter_io_test.rb

@@ -0,0 +1,678 @@
+# -*- coding: utf-8 -*-
+
+require 'test_helper'
+require 'stringio'
+require 'tempfile'
+
+class FilterIOTest < ActiveSupport::TestCase
+  
+  def assert_equal_reference_io(input)
+    
+    expected_io = StringIO.new(input)
+    actual_io = FilterIO.new(StringIO.new(input))
+    
+    results = [expected_io, actual_io].map do |io|
+      results = []
+      errors = []
+      positions = []
+      
+      # call the block repeatedly until we get to EOF
+      # and once more at the end to check what happens at EOF
+      one_more_time = [true]
+      while !io.eof? || one_more_time.pop
+        begin
+          results << yield(io)
+          errors << nil
+        rescue Exception => e
+          results << nil
+          errors << [e.class, e.message]
+        end
+        positions << io.pos
+        raise 'Too many iterations' if results.size > 100
+      end
+      
+      [results, errors, positions]
+    end
+    
+    # compare the filtered output against the reference
+    results[0].zip(results[1]).each do |expected, actual|
+      assert_equal expected, actual
+    end
+    
+  end
+  
+  test "empty source" do
+    io = FilterIO.new(StringIO.new(''))
+    assert_true io.bof?
+    io = FilterIO.new(StringIO.new(''))
+    assert_true io.eof?
+    io = FilterIO.new(StringIO.new(''))
+    assert_raise EOFError do
+      io.readchar
+    end
+  end
+  
+  test "simple eof" do
+    io = FilterIO.new(StringIO.new('x'))
+    assert_false io.eof?
+    assert_equal 'x', io.readchar.chr
+    assert_true io.eof?
+    assert_equal '', io.read
+    assert_equal nil, io.read(8)
+  end
+  
+  test "simple bof" do
+    io = FilterIO.new(StringIO.new('x'))
+    assert_true io.bof?
+    assert_equal 'x', io.readchar.chr
+    assert_false io.bof?
+  end
+  
+  test "unicode readchar" do
+    assert_equal_reference_io('Résume') { |io| io.readchar }
+  end
+  
+  test "unicode read" do
+    (1..3).each do |read_size|
+      assert_equal_reference_io('Résume') { |io| io.read read_size }
+    end
+  end
+  
+  test "unicode read all" do
+    assert_equal_reference_io('Résume') { |io| io.read }
+  end
+  
+  test "unicode gets" do
+    assert_equal_reference_io("über\nrésumé") { |io| io.gets }
+  end
+  
+  test "unicode in block" do
+    input = 'Résumé Test'
+    expected = 'résumé test'
+    [2, nil].each do |block_size|
+      io = FilterIO.new(StringIO.new(input), :block_size => block_size) { |data| data.downcase }
+      actual = io.read
+      assert_equal expected, actual
+    end
+  end
+  
+  test "should not buffer forever on bad encoding" do
+    input = "123\xc3\xc34567890"
+    block_count = 0
+    io = FilterIO.new(StringIO.new(input), :block_size => 2) do |data|
+      block_count += 1
+      assert_operator data.size, :<=, 6
+      data
+    end
+    actual = io.read
+    if input.respond_to? :force_encoding
+      input.force_encoding 'ASCII-8BIT'
+      actual.force_encoding 'ASCII-8BIT'
+    end
+    assert_equal input, actual
+    assert_operator block_count, :>=, 3
+  end
+  
+  if IO.method_defined? :external_encoding
+    
+    def with_iso8859_1_test_file(internal_encoding)
+      Tempfile.open 'filter_io' do |tempfile|
+        File.open(tempfile.path, 'wb') do |io|
+          io.write "\xFCber\nR\xE9sum\xE9"
+        end
+        File.open(tempfile.path, :external_encoding => 'ISO-8859-1', :internal_encoding => internal_encoding) do |io|
+          yield io
+        end
+      end
+    end
+    
+    test "ISO-8859-1 sanity check to UTF-8" do
+      with_iso8859_1_test_file 'UTF-8' do |io_raw|
+        assert_equal 'ü', io_raw.readchar
+        assert_equal "ber\n", io_raw.gets
+        str = io_raw.gets
+        assert_equal 'résumé', str.downcase
+        assert_equal 'UTF-8', str.encoding.name
+      end
+    end
+    
+    test "ISO-8859-1 sanity check raw" do
+      with_iso8859_1_test_file nil do |io_raw|
+        assert_equal 'ü'.encode('ISO-8859-1'), io_raw.readchar
+        assert_equal "ber\n", io_raw.gets
+        str = io_raw.gets
+        assert_equal 'résumé'.encode('ISO-8859-1'), str.downcase
+        assert_equal 'ISO-8859-1', str.encoding.name
+      end
+    end
+    
+    test "iso-8859-1 readchar to UTF-8" do
+      with_iso8859_1_test_file 'UTF-8' do |io_raw|
+        io = FilterIO.new(io_raw)
+        "über\n".chars.each do |expected|
+          actual = io.readchar
+          assert_equal expected, actual
+          assert_equal 'UTF-8', actual.encoding.name
+        end
+      end
+    end
+    
+    test "iso-8859-1 readchar raw" do
+      with_iso8859_1_test_file nil do |io_raw|
+        io = FilterIO.new(io_raw)
+        "über\n".encode('ISO-8859-1').chars.each do |expected|
+          actual = io.readchar
+          assert_equal expected, actual
+          assert_equal 'ISO-8859-1', actual.encoding.name
+        end
+      end
+    end
+    
+    test "iso-8859-1 read to UTF-8" do
+      with_iso8859_1_test_file 'UTF-8' do |io_raw|
+        io = FilterIO.new(io_raw)
+        assert_equal 'ü'.force_encoding('ASCII-8BIT'), io.read(2)
+        assert_equal 'ASCII-8BIT', io.read(2).encoding.name
+      end
+    end
+    
+    test "iso-8859-1 read raw" do
+      with_iso8859_1_test_file nil do |io_raw|
+        io = FilterIO.new(io_raw)
+        assert_equal 'ü'.encode('ISO-8859-1').force_encoding('ASCII-8BIT'), io.read(1)
+        assert_equal 'ASCII-8BIT', io.read(2).encoding.name
+      end
+    end
+    
+    test "iso-8859-1 lines to UTF-8" do
+      with_iso8859_1_test_file 'UTF-8' do |io_raw|
+        io = FilterIO.new(io_raw)
+        expected = ["über\n", 'Résumé']
+        actual = io.lines.to_a
+        assert_equal expected, actual
+        assert_equal 'UTF-8', actual[0].encoding.name
+      end
+    end
+    
+    test "iso-8859-1 lines raw" do
+      with_iso8859_1_test_file nil do |io_raw|
+        io = FilterIO.new(io_raw)
+        expected = ["über\n", 'Résumé'].map { |str| str.encode('ISO-8859-1') }
+        actual = io.lines.to_a
+        assert_equal expected, actual
+        assert_equal 'ISO-8859-1', actual[0].encoding.name
+      end
+    end
+    
+    test "iso-8859-1 block to UTF-8" do
+      [1, 2, nil].each do |block_size|
+        expected = "über\nrésumé"
+        with_iso8859_1_test_file 'UTF-8' do |io_raw|
+          io = FilterIO.new(io_raw, :block_size => block_size) do |data, state|
+            assert_equal 'ü', data[0] if state.bof?
+            assert_equal 'UTF-8', data.encoding.name
+            data.downcase
+          end
+          assert_equal 'ü', io.readchar
+          assert_equal 'UTF-8', io.gets.encoding.name
+          assert_equal 'rés'.force_encoding('ASCII-8BIT'), io.read(4)
+          str = io.gets
+          assert_equal 'umé', str
+          assert_equal 'UTF-8', str.encoding.name
+        end
+      end
+    end
+    
+    test "iso-8859-1 block raw" do
+      [1, 2, nil].each do |block_size|
+        expected = "über\nrésumé".encode('ISO-8859-1')
+        with_iso8859_1_test_file 'ISO-8859-1' do |io_raw|
+          io = FilterIO.new(io_raw, :block_size => block_size) do |data, state|
+            assert_equal 'ü'.encode('ISO-8859-1'), data[0] if state.bof?
+            assert_equal 'ISO-8859-1', data.encoding.name
+            data.downcase
+          end
+          assert_equal 'ü'.encode('ISO-8859-1'), io.readchar
+          assert_equal 'ISO-8859-1', io.gets.encoding.name
+          assert_equal 'rés'.encode('ISO-8859-1').force_encoding('ASCII-8BIT'), io.read(3)
+          str = io.gets
+          assert_equal 'umé'.encode('ISO-8859-1'), str
+          assert_equal 'ISO-8859-1', str.encoding.name
+        end
+      end
+    end
+    
+  end
+  
+  test "read" do
+    input = 'Lorem ipsum dolor sit amet, consectetur adipisicing elit'
+    io_reference = StringIO.new(input)
+    io = FilterIO.new(StringIO.new(input))
+    [10,5,4,8,7,nil,nil].each do |read_len|
+      assert_equal io_reference.read(read_len), io.read(read_len)
+      assert_equal io_reference.pos, io.pos
+      if read_len
+        assert_equal io_reference.readchar, io.readchar
+      else
+        assert_raise(EOFError) { io_reference.readchar }
+        assert_raise(EOFError) { io.readchar }
+      end
+      assert_equal io_reference.pos, io.pos
+      assert_equal io_reference.eof?, io.eof?
+    end
+    assert_equal io_reference.read, io.read
+    assert_equal io_reference.read(4), io.read(4)
+    assert_true io_reference.eof?
+    assert_true io.eof?
+  end
+  
+  test "read zero before eof" do
+    io = FilterIO.new(StringIO.new('foo'))
+    assert_equal '', io.read(0)
+    assert_equal 0, io.pos
+    assert_false io.eof?
+  end
+  
+  test "read zero at eof" do
+    io = FilterIO.new(StringIO.new(''))
+    assert_equal '', io.read(0)
+    assert_equal 0, io.pos
+    assert_true io.eof?
+  end
+  
+  test "read negative" do
+    io = FilterIO.new(StringIO.new('foo'))
+    assert_equal 'fo', io.read(2)
+    assert_raise ArgumentError do
+      io.read(-1)
+    end
+    assert_equal 2, io.pos
+  end
+  
+  test "simple block" do
+    input = 'foo bar'
+    expected = 'FOO BAR'
+    io = FilterIO.new(StringIO.new(input)) do |data|
+      data.upcase
+    end
+    assert_equal expected, io.read
+  end
+  
+  test "block bof and eof" do
+    input = "Test String"
+    expected = ">>>*Test** Str**ing*<<<"
+    io = FilterIO.new(StringIO.new(input), :block_size => 4) do |data, state|
+      data = "*#{data}*"
+      data = ">>>#{data}" if state.bof?
+      data = "#{data}<<<" if state.eof?
+      data
+    end
+    assert_equal expected, io.read
+  end
+  
+  test "Symbol#to_proc" do
+    input = 'foo bar'
+    expected = 'FOO BAR'
+    io = FilterIO.new StringIO.new(input), &:upcase
+    assert_equal expected, io.read
+  end
+  
+  test "block size" do
+    [1,4,7,9,13,30].each do |block_size|
+      input = ('A'..'Z').to_a.join
+      expected = input.chars.enum_for(:each_slice, block_size).to_a.map(&:join).map { |x| "[#{x}]" }.join
+      io = FilterIO.new(StringIO.new(input), :block_size => block_size) do |data|
+        "[#{data}]"
+      end
+      assert_equal expected, io.read
+    end
+  end
+  
+  test "block size different to read size" do
+    (1..5).each do |block_size|
+      input_str = ('A'..'Z').to_a.join
+      expected_str = input_str.chars.enum_for(:each_slice, block_size).map { |x| "[#{x.join}]" }.join
+      (1..5).each do |read_size|
+        
+        expected = StringIO.new(expected_str)
+        actual = FilterIO.new(StringIO.new(input_str), :block_size => block_size) do |data|
+          "[#{data}]"
+        end
+        
+        until expected.eof?
+          assert_equal expected.read(read_size), actual.read(read_size)
+          assert_equal expected.pos, actual.pos
+        end
+        assert_equal expected.eof?, actual.eof?
+        
+      end
+    end
+  end
+  
+  test "rewind pass through" do
+    io = FilterIO.new(StringIO.new('foo bar baz'))
+    assert_equal 'foo b', io.read(5)
+    assert_equal 'ar b', io.read(4)
+    io.rewind
+    assert_equal 'foo', io.read(3)
+    assert_equal ' ', io.readchar.chr
+    io.rewind
+    assert_equal 'f', io.readchar.chr
+    assert_equal 'oo', io.read(2)
+  end
+  
+  test "rewind resets buffer" do
+    str = 'foobar'
+    io = FilterIO.new(StringIO.new(str))
+    assert_equal 'foo', io.read(3)
+    str.replace 'FooBar'
+    assert_equal 'Bar', io.read(3)
+    io.rewind
+    assert_equal 'Foo', io.read(3)
+  end
+  
+  test "rewind with block" do
+    input = 'abcdefghij'
+    expected = input[1..-1]
+    io = FilterIO.new(StringIO.new(input), :block_size => 4) do |data, state|
+      data = data[1..-1] if state.bof?
+      data
+    end
+    assert_equal 'bc', io.read(2)
+    assert_equal 'defg', io.read(4)
+    io.rewind
+    assert_equal 'bc', io.read(2)
+    assert_equal 'defg', io.read(4)
+  end
+  
+  test "ungetc" do
+    input = 'foobar'
+    io = FilterIO.new(StringIO.new(input))
+    assert_equal 'foo', io.read(3)
+    io.ungetc 'x'
+    io.ungetc 'y'[0].ord
+    assert_equal 'yxb', io.read(3)
+    (1..5).each do |i|
+      io.ungetc i.to_s
+    end
+    assert_equal '54321ar', io.read
+    assert_equal 'foobar', input
+  end
+  
+  test "need more data" do
+    input = '1ab123456cde78f9ghij0'
+    expected = input.gsub /\d+/, '[\0]'
+    (1..5).each do |block_size|
+      expected_size = 0
+      io = FilterIO.new(StringIO.new(input), :block_size => block_size) do |data, state|
+        expected_size += block_size
+        raise FilterIO::NeedMoreData if data =~ /\d\z/ && !state.eof?
+        assert_equal expected_size, data.size unless state.eof?
+        expected_size = 0
+        data.gsub /\d+/, '[\0]'
+      end
+      assert_equal expected, io.read
+    end
+  end
+  
+  test "line ending normalisation" do
+    input = "This\r\nis\r\ra\n\ntest\n\r\n\nstring\r\r\n.\n"
+    expected = "This\nis\n\na\n\ntest\n\n\nstring\n\n.\n"
+    (1..5).each do |block_size|
+      io = FilterIO.new(StringIO.new(input), :block_size => block_size) do |data, state|
+        raise FilterIO::NeedMoreData if data =~ /[\r\n]\z/ && !state.eof?
+        data.gsub /\r\n|\r|\n/, "\n"
+      end
+      assert_equal expected, io.read
+    end
+  end
+  
+  test "dropping characters" do
+    input = "ab1cde23f1g4hijklmno567pqr8stu9vw0xyz"
+    expected = input.gsub /\d+/, ''
+    (1..5).each do |block_size|
+      io = FilterIO.new(StringIO.new(input), :block_size => block_size) do |data|
+        data.gsub /\d+/, ''
+      end
+      assert_equal 0, io.pos
+      assert_equal expected, io.read
+      assert_equal expected.size, io.pos
+    end
+  end
+  
+  test "getc" do
+    assert_equal_reference_io('foo') { |io| io.getc }
+  end
+  
+  test "gets default" do
+    [
+      "",
+      "x",
+      "foo bar",
+      "foo\nbar",
+      "foo\nbar\nbaz\n"
+    ].each do |input|
+      assert_equal_reference_io(input) { |io| io.gets }
+    end
+  end
+  
+  test "gets all" do
+    [
+      "",
+      "x",
+      "foo bar",
+      "foo\nbar",
+      "foo\nbar\nbaz\n"
+    ].each do |input|
+      assert_equal_reference_io(input) { |io| io.gets(nil) }
+    end
+  end
+  
+  test "gets separator" do
+    [
+      "",
+      "x",
+      "foo\nbar\rbaz\n",
+      "abc\rdef\rghi\r",
+      "abcxyz",
+    ].each do |input|
+      ["\r", "x"].each do |sep_string|
+        assert_equal_reference_io(input) { |io| io.gets(sep_string) }
+      end
+    end
+  end
+  
+  test "gets 2 char separator" do
+    ["o", "oo"].each do |sep_string|
+      assert_equal_reference_io("foobarhelloworld") { |io| io.gets(sep_string) }
+    end
+  end
+  
+  test "gets paragraph" do
+    {
+      "" => [],
+      "x" => ['x'],
+      "foo bar" => ["foo bar"],
+      "foo bar\n" => ["foo bar\n"],
+      "foo bar\n\n" => ["foo bar\n\n"],
+      "foo bar\n\n\n" => ["foo bar\n\n"],
+      "foo bar\nbaz" => ["foo bar\nbaz"],
+      "foo bar\n\nbaz" => ["foo bar\n\n", "baz"],
+      "foo bar\n\n\nbaz" => ["foo bar\n\n", "baz"],
+      "foo bar\n\nbaz\n" => ["foo bar\n\n", "baz\n"],
+      "foo bar\n\nbaz\n\n" => ["foo bar\n\n", "baz\n\n"],
+      "foo bar\n\nbaz\n\n\n" => ["foo bar\n\n", "baz\n\n"],
+      "\n\n\nfoo bar\n\nbaz\n\n\nabc\ndef" => ["foo bar\n\n", "baz\n\n", "abc\ndef"],
+    }.each do |input, expected|
+      io = FilterIO.new(StringIO.new(input))
+      actual = []
+      while para = io.gets('')
+        actual << para
+      end
+      assert_equal expected, actual
+    end
+  end
+  
+  test "readline" do
+    [
+      "foo\nbar\n",
+      "foo\nbar\nbaz"
+    ].each do |input|
+      assert_equal_reference_io(input) { |io| io.readline }
+      assert_equal_reference_io(input) { |io| io.readline("o") }
+    end
+  end
+  
+  test "readlines" do
+    [
+      "foo\nbar\n",
+      "foo\nbar\nbaz"
+    ].each do |input|
+      assert_equal_reference_io(input) { |io| io.readlines }
+      assert_equal_reference_io(input) { |io| io.readlines("o") }
+    end
+  end
+  
+  test "lines with block" do
+    io = FilterIO.new(StringIO.new("foo\nbar\nbaz"))
+    expected = [ ["foo\n", "bar\n"], ["baz", nil] ]
+    actual = []
+    retval = io.lines do |line|
+      actual << [line, io.gets]
+    end
+    assert_equal io, retval
+    assert_equal expected, actual
+  end
+  
+  test "lines enumerator" do
+    io = FilterIO.new(StringIO.new("foo\nbar\nbaz"))
+    e = io.lines
+    expected = [ ["foo\n", "bar\n"], ["baz", nil] ]
+    actual = e.map { |line| [line, io.gets] }
+    assert_equal expected, actual
+  end
+  
+  test "seek set" do
+    
+    io = FilterIO.new(StringIO.new("abcdef"))
+    
+    # beginning
+    assert_equal 'a', io.readchar.chr
+    assert_equal 1, io.pos
+    io.seek 0, IO::SEEK_SET
+    assert_equal 'a', io.readchar.chr
+    assert_equal 1, io.pos
+    
+    # same position
+    io.seek 1, IO::SEEK_SET
+    assert_equal 'b', io.readchar.chr
+    assert_equal 2, io.pos
+    
+    # backwards fail
+    assert_raise Errno::EINVAL do
+      io.seek 1, IO::SEEK_SET
+    end
+    assert_equal 'c', io.readchar.chr
+    assert_equal 3, io.pos
+    
+  end
+    
+  test "seek current" do
+    
+    io = FilterIO.new(StringIO.new("abcdef"))
+    
+    # same pos
+    assert_equal 'ab', io.read(2)
+    assert_equal 2, io.pos
+    io.seek 0, IO::SEEK_CUR
+    assert_equal 2, io.pos
+    
+    # backwards fail
+    assert_equal 'c', io.read(1)
+    assert_equal 3, io.pos
+    assert_raise Errno::EINVAL do
+      io.seek -1, IO::SEEK_CUR
+    end
+    assert_equal 3, io.pos
+    
+    # forwards fail
+    assert_equal 3, io.pos
+    assert_raise Errno::EINVAL do
+      io.seek 2, IO::SEEK_CUR
+    end
+    assert_equal 3, io.pos
+    
+    # beginning
+    io.seek -io.pos, IO::SEEK_CUR
+    assert_equal 0, io.pos
+    
+  end
+  
+  test "seek end" do
+    io = FilterIO.new(StringIO.new("abcdef"))
+    assert_raise Errno::EINVAL do
+      io.seek 0, IO::SEEK_END
+    end
+    assert_raise Errno::EINVAL do
+      io.seek 6, IO::SEEK_END
+    end
+    assert_raise Errno::EINVAL do
+      io.seek -6, IO::SEEK_END
+    end
+  end
+  
+  test "need more data at eof" do
+    input = "foo"
+    io = FilterIO.new(StringIO.new(input), :block_size => 2) do |data|
+      raise FilterIO::NeedMoreData
+    end
+    assert_raise EOFError do
+      io.readline
+    end
+  end
+  
+  test "unget via block" do
+    # get consecutive unique characters from a feed
+    # this is similar to uniq(1) and STL's unique_copy
+    input = "122234435"
+    expected = "123435"
+    (1..5).each do |block_size|
+      io = FilterIO.new(StringIO.new(input), :block_size => block_size) do |data, state|
+        # grab all of the same character
+        data =~ /\A(.)\1*(?!\1)/ or raise 'No data'
+        # if there was nothing after it and we aren't at EOF...
+        # ...grab more data to make sure we're at the end
+        raise FilterIO::NeedMoreData if $'.empty? && !state.eof?
+        # return the matched character as data and re-buffer the rest
+        [$&[0], $']
+      end
+      assert_equal expected, io.read
+    end
+  end
+  
+  test "get more data via unget" do
+    
+    input = "foo\ntest\n\n12345\n678"
+    expected = input.gsub(/^.*$/) { |x| "#{$&.size} #{$&}" }
+    expected += "\n" unless expected =~ /\n\z/
+    
+    block_count = 0
+    io = FilterIO.new StringIO.new(input), :block_size => 2 do |data, state|
+      block_count += 1
+      raise 'Too many retries' if block_count > 100
+      raise "Expected less data: #{data.inspect}" if data.size > 6
+      output = ''
+      while data =~ /(.*)\n/ || (state.eof? && data =~ /(.+)/)
+        output << "#{$1.size} #{$1}\n"
+        data = $'
+      end
+      [output, data]
+    end
+    actual = io.read
+    
+    assert_equal expected, actual
+    assert_operator block_count, :>=, 10
+    
+  end
+  
+end

data/test/test_helper.rb

@@ -0,0 +1,6 @@
+require 'rubygems'
+gem 'test-unit'
+require 'test/unit'
+require 'active_support'
+require 'active_support/test_case'
+require 'filter_io'

metadata

@@ -0,0 +1,89 @@
+--- !ruby/object:Gem::Specification 
+name: filter_io
+version: !ruby/object:Gem::Version 
+  hash: 27
+  prerelease: false
+  segments: 
+  - 0
+  - 1
+  - 0
+  version: 0.1.0
+platform: ruby
+authors: 
+- Jason Weathered
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2010-06-14 00:00:00 +10:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: activesupport
+  prerelease: false
+  requirement: &id001 !ruby/object:Gem::Requirement 
+    none: false
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        hash: 3
+        segments: 
+        - 0
+        version: "0"
+  type: :runtime
+  version_requirements: *id001
+description: 
+email: jason@jasoncodes.com
+executables: []
+
+extensions: []
+
+extra_rdoc_files: 
+- LICENSE
+- README.markdown
+files: 
+- .gitignore
+- LICENSE
+- README.markdown
+- Rakefile
+- VERSION
+- lib/filter_io.rb
+- test/filter_io_test.rb
+- test/test_helper.rb
+has_rdoc: true
+homepage: http://github.com/jasoncodes/filter_io
+licenses: []
+
+post_install_message: 
+rdoc_options: 
+- --charset=UTF-8
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  none: false
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      hash: 3
+      segments: 
+      - 0
+      version: "0"
+requirements: []
+
+rubyforge_project: 
+rubygems_version: 1.3.7
+signing_key: 
+specification_version: 3
+summary: Filter IO streams with a block. Ruby's FilterInputStream.
+test_files: 
+- test/filter_io_test.rb
+- test/test_helper.rb