fastercsv 0.1.0

data/CHANGELOG

@@ -0,0 +1,7 @@
+= Change Log
+
+Below is a complete listing of changes for each revision of FasterCSV.
+
+== 0.1.0
+
+* Initial public release.

data/INSTALL

@@ -0,0 +1,23 @@
+= Installing FasterCSV
+
+RubyGems is the preferred easy install method for FasterCSV.  However, you can
+install FasterCSV manually as described below.
+
+== Installing the Gem
+
+FasterCSV is intended to be installed via the
+RubyGems[http://rubyforge.org/projects/rubygems/] system.  To get the latest
+version, simply enter the following into your command prompt:
+
+	$ sudo gem install fastercsv
+
+You must have RubyGems[http://rubyforge.org/projects/rubygems/] installed for
+the above to work.
+
+== Installing Manually
+
+Download the latest version of FasterCSV from the
+{RubyForge project page}[http://rubyforge.org/frs/?group_id=1102].  Navigate to
+the root project directory and enter:
+
+	$ sudo ruby setup.rb

data/LICENSE

@@ -0,0 +1,7 @@
+= License Terms
+
+Distributed under the user's choice of the GPL[http://www.gnu.org/copyleft/gpl.html] (see COPYING for details) or the
+{Ruby software license}[http://www.ruby-lang.org/en/LICENSE.txt] by
+James Edward Gray II.
+
+Please email James[mailto:james@grayproductions.net] with any questions.

data/README

@@ -0,0 +1,57 @@
+= Read Me
+
+by James Edward Gray II
+
+== Description
+
+Welcome to FasterCSV.
+
+FasterCSV is intended as a replacement to Ruby's standard CSV library.  It was designed to address concerns users of that library had and it has three primary goals:
+
+1.  Be significantly faster than CSV while remaining a pure Ruby library.
+2.  Use a smaller and easier to maintain code base.
+3.  Improve on the CSV interface.
+
+Obviously, the last one is subjective.  If you love CSV's interface, odds are
+good this one won't suit you.  I did try to defer to that interface whenever I
+didn't have a compelling reason to change it though, so hopefully this won't be
+too radically different.
+
+== What's Different From CSV?
+
+I'm sure I'll miss something, but I'll try to mention most of the major differences I am aware of, to help others quickly get up to speed:
+
+=== CSV Parsing
+
+* FasterCSV has a stricter parser and will throw MalformedCSVErrors on
+  problematic data.
+* FasterCSV has a less liberal idea of a line ending than CSV.  What you set as
+  the <tt>:row_sep</tt> is law.
+* CSV returns empty lines as <tt>[nil]</tt>.  FasterCSV calls them <tt>[]</tt>.
+* FasterCSV has a much faster parser.
+
+=== Interface
+
+* FasterCSV uses Hash-style parameters to set options.
+* FasterCSV does not have generate_row() or parse_row() from CSV.
+* FasterCSV does not have CSV's Reader and Writer classes.
+* FasterCSV::open() is more like Ruby's open() than CSV::open().
+* FasterCSV objects support most standard IO methods.
+* FasterCSV has a new() method used to wrap objects like String and IO for
+  reading and writing.
+* FasterCSV::generate() is different from CSV::generate().
+
+If you use this library and find yourself missing any functionality I have trimmed, please {let me know}[mailto:james@grayproductions.net].
+
+== Documentation
+
+See FasterCSV for documentation.
+
+== Installing
+
+See the INSTALL file for instructions.
+
+== Questions and/or Comments
+
+Feel free to email {James Edward Gray II}[mailto:james@grayproductions.net] with
+any questions.

data/Rakefile

@@ -0,0 +1,83 @@
+require "rake/rdoctask"
+require "rake/testtask"
+require "rake/gempackagetask"
+
+require "rubygems"
+
+task :default => [:test]
+
+Rake::TestTask.new do |test|
+	test.libs << "test"
+	test.test_files = [ "test/ts_all.rb" ]
+	test.verbose = true
+end
+
+Rake::RDocTask.new do |rdoc|
+	rdoc.main = "README"
+	rdoc.rdoc_files.include( "README",  "INSTALL",
+	                         "TODO",    "CHANGELOG",
+	                         "AUTHORS", "COPYING",
+	                         "LICENSE", "lib/" )
+	rdoc.rdoc_dir = "doc/html"
+	rdoc.title = "FasterCSV Documentation"
+end
+
+desc "Upload current documentation to Rubyforge"
+task :upload_docs => [:rdoc] do
+	sh "scp -r doc/html/* " +
+	   "bbazzarrakk@rubyforge.org:/var/www/gforge-projects/fastercsv/"
+end
+
+desc "Show library's code statistics"
+task :stats do
+	require 'code_statistics'
+	CodeStatistics.new( ["FasterCSV", "lib"], 
+	                    ["Units",     "test"] ).to_s
+end
+
+desc "Time FasterCSV and CSV"
+task :benchmark do
+  path = "test/test_data.csv"
+	sh %Q{time ruby -r csv -e 'CSV.foreach("#{path}") { |row| }'}
+	sh %Q{time ruby -r lib/faster_csv -e 'FasterCSV.foreach("#{path}") { |row| }'}
+end
+
+spec = Gem::Specification.new do |spec|
+	spec.name = "fastercsv"
+	spec.version = "0.1.0"
+	spec.platform = Gem::Platform::RUBY
+	spec.summary = "FasterCSV is CSV, but faster, smaller, and cleaner."
+
+	spec.files = Dir.glob("{lib,test}/**/*.rb").
+	                 reject { |item| item.include?(".svn") } +
+	                 ["Rakefile", "setup.rb"]
+	spec.test_suite_file = "test/ts_all.rb"
+
+	spec.has_rdoc = true
+	spec.extra_rdoc_files = %w{README INSTALL TODO CHANGELOG LICENSE}
+	spec.rdoc_options << "--title" << "FasterCSV Documentation" <<
+	                     "--main"  << "README"
+
+	spec.require_path = "lib"
+	spec.autorequire = "fastercsv"
+
+	spec.author = "James Edward Gray II"
+	spec.email = "james@grayproductions.net"
+	spec.rubyforge_project = "fastercsv"
+	spec.homepage = "http://fastercsv.rubyforge.org"
+	spec.description = <<END_DESC
+FasterCSV is intended as a complete replacement to the CSV standard library. It
+is significantly faster and smaller while still being pure Ruby code. It also
+strives for a better interface.
+END_DESC
+end
+
+Rake::GemPackageTask.new(spec) do |pkg|
+	pkg.need_zip = true
+	pkg.need_tar = true
+end
+
+desc "Add new files to Subversion"
+task :add_to_svn do
+  sh %Q{svn status | ruby -nae 'system "svn add \#{$F[1]}" if $F[0] == "?"' }
+end

data/TODO

@@ -0,0 +1,8 @@
+= To Do List
+
+The following is a list of planned expansions for FasterCSV, in no particular
+order.
+
+* Add support for accessing fields by headers (from first row of document).
+* Add "convertors" for switching numbers to Integers or Floats, dates to Date or
+  Time objects, etc.

data/lib/faster_csv.rb

@@ -0,0 +1,400 @@
+#!/usr/local/bin/ruby -w
+
+# = faster_csv.rb -- Faster CSV Reading and Writing
+#
+#  Created by James Edward Gray II on 2005-10-31.
+#  Copyright 2005 Gray Productions. All rights reserved.
+# 
+# See FasterCSV for documentation.
+
+require "stringio"
+require "forwardable"
+
+# 
+# This class provides a complete interface to CSV files and data.  It offers
+# tools to enable you to read and write to and from Strings or IO objects, as
+# needed.
+# 
+# == Reading
+# 
+# === From a File
+# 
+# ==== A Line at a Time
+# 
+#   FasterCSV.foreach("path/to/file.csv") do |row|
+#     # use row here...
+#   end
+# 
+# ==== All at Once
+# 
+#   arr_of_arrs = FasterCSV.read("path/to/file.csv")
+# 
+# === From a String
+# 
+# ==== A Line at a Time
+# 
+#   FasterCSV.parse("CSV,data,String") do |row|
+#     # use row here...
+#   end
+# 
+# ==== All at Once
+# 
+#   arr_of_arrs = FasterCSV.parse("CSV,data,String")
+# 
+# == Writing
+# 
+# === To a File
+# 
+#   FasterCSV.open("path/to/file.csv", "w") do |csv|
+#     csv << ["row", "of", "CSV", "data"]
+#     csv << ["another", "row"]
+#     # ...
+#   end
+# 
+# === To a String
+# 
+#   csv_string = FasterCSV.generate do |csv|
+#     csv << ["row", "of", "CSV", "data"]
+#     csv << ["another", "row"]
+#     # ...
+#   end
+# 
+# == Convert a Single Line
+# 
+#   csv_string = generate_line(["row", "of", "CSV", "data"])  # to CSV
+#   csv_array  = parse_line("CSV,data,String")                # from CSV
+# 
+class FasterCSV
+  # The error thrown when the parser encounters illegal CSV formatting.
+  class MalformedCSVError < RuntimeError; end
+  
+  # 
+  # The options used when no overrides are given by calling code.  They are:
+  # 
+  # <b><tt>:col_sep</tt></b>::  <tt>","</tt>
+  # <b><tt>:row_sep</tt></b>::  <tt>$/</tt>
+  # 
+  DEFAULT_OPTIONS = {:col_sep => ",", :row_sep => $/}
+  
+  # 
+  # This method is intended as the primary interface for reading CSV files.  You
+  # pass a +path+ and any +options+ you wish to set for the read.  Each row of
+  # file will be passed to the provided +block+ in turn.
+  # 
+  # The +options+ parameter can be anthing FasterCSV::new() understands.
+  # 
+  def self.foreach( path, options = Hash.new, &block )
+    open(path, options) do |csv|
+      csv.each(&block)
+    end
+  end
+
+  # 
+  # This method wraps a String in a FasterCSV object which is passed to the
+  # provided block.  You can use the block to append CSV rows to the String and
+  # when the block exits, the final String will be returned.
+  # 
+  # The +options+ parameter can be anthing FasterCSV::new() understands.
+  # 
+  def self.generate( options = Hash.new )
+    faster_csv = new("", options)
+    yield faster_csv
+    faster_csv.string
+  end
+
+  # 
+  # This method is a shortcut for converting a single row (Array) into a CSV 
+  # String.
+  # 
+  # The +options+ parameter can be anthing FasterCSV::new() understands.
+  # 
+  def self.generate_line( row, options = Hash.new )
+    (new("", options) << row).string
+  end
+  
+  # 
+  # :call-seq:
+  #   open( *args, options = Hash.new ) { |faster_csv| ... }
+  #   open( *args, options = Hash.new )
+  # 
+  # This method opens an IO object, and wraps that with FasterCSV.  This is
+  # intended as the primary interface for writing a CSV file.
+  # 
+  # You may pass any +args+ Ruby's open() understands followed by an optional
+  # Hash containing any +options+ FasterCSV::new() understands.
+  # 
+  # This method works like Ruby's open() call, in that it will pass a FasterCSV
+  # object to a provided block and close it when the block termminates, or it
+  # will return the FasterCSV object when no block is provided.  (*Note*: This
+  # is different from the standard CSV library which passes rows to the block.  
+  # Use FasterCSV::foreach() for that behavior.)
+  # 
+  # An opened FasterCSV object will delegate to many IO methods, for 
+  # convenience.  You may call:
+  # 
+  # * binmode()
+  # * close()
+  # * close_read()
+  # * close_write()
+  # * closed?()
+  # * eof()
+  # * eof?()
+  # * fcntl()
+  # * fileno()
+  # * flush()
+  # * fsync()
+  # * ioctl()
+  # * isatty()
+  # * lineno()
+  # * pid()
+  # * pos()
+  # * reopen()
+  # * rewind()
+  # * seek()
+  # * stat()
+  # * sync()
+  # * sync=()
+  # * tell()
+  # * to_i()
+  # * to_io()
+  # * tty?()
+  # 
+  def self.open( *args )
+    # find the +options+ Hash
+    options = if args.last.is_a? Hash then args.pop else Hash.new end
+    # wrap a File opened with the remaining +args+
+    csv     = new(File.open(*args), options)
+    
+    # handle blocks like Ruby's open(), not like the CSV library
+    if block_given?
+      begin
+        yield csv
+      ensure
+        csv.close
+      end
+    else
+      csv
+    end
+  end
+  
+  # 
+  # :call-seq:
+  #   parse( str, options ) { |row| ... }
+  #   parse( str, options )
+  # 
+  # This method can be used to easily parse CSV out of a String.  You may either
+  # provide a +block+ which will be called with each row of the String in turn,
+  # or just use the returned Array of Arrays (when no +block+ is given).
+  # 
+  # You pass your +str+ to read from, and an optional +options+ Hash containing
+  # anything FasterCSV::new() understands.
+  # 
+  def self.parse( *args, &block )
+    csv = new(*args)
+    if block.nil?  # slurp contents, if no block is given
+      begin
+        csv.read
+      ensure
+        csv.close
+      end
+    else           # or pass each row to a provided block
+      csv.each(&block)
+    end
+  end
+  
+  # 
+  # Use to slurp a CSV file into an Array of Arrays.  Pass the +path+ to the 
+  # file and any +options+ FasterCSV::new() understands.
+  # 
+  def self.read( path, options = Hash.new )
+    open(path, options) { |csv| csv.read }
+  end
+  
+  # Alias for FasterCSV::read().
+  def self.readlines( path, options = Hash.new )
+    open(path, options) { |csv| csv.readlines }
+  end
+  
+  # 
+  # This method is a shortcut for converting a single line of a CSV String into 
+  # a into an Array.  Note that if +line+ contains multiple rows, anything 
+  # beyond the first row is ignored.
+  # 
+  # The +options+ parameter can be anthing FasterCSV::new() understands.
+  # 
+  def self.parse_line( line, options = Hash.new )
+    new(line, options).shift
+  end
+  
+  # 
+  # This constructor will wrap either a String or IO object passed in +data+ for
+  # reading and/or writing.  In addition to the FasterCSV instance methods, 
+  # several IO methods are delegated.  (See FasterCSV::open() for a complete 
+  # list.)  If you pass a String for +data+, you can later retrieve it (after
+  # writing to it, for example) with FasterCSV.string().
+  # 
+  # You may set any reading and/or writing preferences in the +options+ Hash.  
+  # Available options are:
+  # 
+  # <b><tt>:col_sep</tt></b>:: The String placed between each field.
+  # <b><tt>:row_sep</tt></b>:: The String appended to the end of each row.
+  # 
+  # See FasterCSV::DEFAULT_OPTIONS for the default settings.
+  # 
+  # Options cannot be overriden in the instance methods for performance reasons,
+  # so be sure to set what you want here.
+  # 
+  def initialize( data, options = Hash.new )
+    # build the options for this read/write
+    options = DEFAULT_OPTIONS.merge(options)
+    
+    # create the IO object we will read from
+    @io = if data.is_a? String then StringIO.new(data) else data end
+    
+    # store the selected separators
+    @col_sep = options[:col_sep]
+    @row_sep = options[:row_sep]
+    
+    # prebuild Regexps for faster parsing
+    @parsers = [ /\A#{@col_sep}+/,                # for empty leading fields
+                 ### The Primary Parser ###
+                 / \G(?:^|#{Regexp.escape(@col_sep)})     # anchor the match
+                   (?: "((?>[^"]*)(?>""[^"]*)*)"          # find quoted fields
+                       |                                  # ... or ...
+                       ([^"#{Regexp.escape(@col_sep)}]*)  # unquoted fields
+                       )/x,
+                 ### End Primary Parser ###
+                 /#{@row_sep}\Z/ ]                 # safer than chomp!()
+  end
+  
+  ### IO and StringIO Delegation ###
+  
+  extend Forwardable
+  def_delegators :@io, :binmode, :close, :close_read, :close_write, :closed?,
+                       :eof, :eof?, :fcntl, :fileno, :flush, :fsync, :ioctl,
+                       :isatty, :lineno, :pid, :pos, :reopen, :rewind, :seek,
+                       :stat, :string, :sync, :sync=, :tell, :to_i, :to_io,
+                       :tty?
+
+  ### End Delegation ###
+  
+  # 
+  # The primary write method for wrapped Strings and IOs, +row+ (an Array) is
+  # converted to CSV and appended to the data source.
+  # 
+  # The data source must be open for writing.
+  # 
+  def <<( row )
+    @io << row.map do |field|
+      if field.nil?  # reverse +nil+ fields as empty unquoted fields
+        ""
+      else
+        field = String(field)  # Stringify fields
+        # reverse empty fields as empty quoted fields
+        if field.empty? or field.count(%Q{\r\n#{@col_sep}"}).nonzero?
+          %Q{"#{field.gsub('"', '""')}"}  # escape quoted fields
+        else
+          field  # unquoted field
+        end
+      end
+    end.join(@col_sep) + @row_sep  # add separators
+    
+    self  # for chaining
+  end
+  alias_method :add_row, :<<
+  alias_method :puts,    :<<
+  
+  include Enumerable
+  
+  # 
+  # Yields each row of the data source in turn.
+  # 
+  # Support for Enumerable.
+  # 
+  # The data source must be open for reading.
+  # 
+  def each
+    while row = shift
+      yield row
+    end
+  end
+  
+  # 
+  # Slurps the remaining rows and returns an Array of Arrays.
+  # 
+  # The data source must be open for reading.
+  # 
+  def read
+    to_a
+  end
+  alias_method :readlines, :read
+  
+  # 
+  # The primary read method for wrapped Strings and IOs, a single row is pulled
+  # from the data source, parsed and returned as an Array of fields.
+  # 
+  # The data source must be open for reading.
+  # 
+  def shift
+    # begin with a blank line, so we can always add to it
+    line = ""
+
+    # 
+    # it can take multiple calls to <tt>@io.gets()</tt> to get a full line,
+    # because of \r and/or \n characters embedded in quoted fields
+    # 
+    loop do
+      # add another read to the line
+      line  += @io.gets(@row_sep) rescue return nil
+      # copy the line so we can chop it up in parsing
+      parse = line.dup
+      parse.sub!(@parsers[2], "")
+      
+      # 
+      # I believe a blank line should be an <tt>Array.new</tt>, not 
+      # CSV's <tt>[nil]</tt>
+      # 
+      return Array.new if parse.empty?
+
+      # 
+      # shave leading empty fields if needed, because the main parser chokes 
+      # on these
+      # 
+      csv = if parse.sub!(@parsers[0], "")
+        [nil] * $&.length
+      else
+        Array.new
+      end
+      # 
+      # then parse the main fields with a hyper-tuned Regexp from 
+      # Mastering Regular Expressions, Second Edition
+      # 
+      parse.gsub!(@parsers[1]) do
+        csv << if $1.nil?     # we found an unquoted field
+          if $2.empty?        # switch empty unquoted fields to +nil+...
+            nil               # for CSV compatibility
+          else
+            # I decided to take a strict approach to CSV parsing...
+            if $2.count("\r\n").zero?  # verify correctness of field...
+              $2
+            else
+              # or throw an Exception
+              raise MalformedCSVError, 'Unquoted fields do not allow \r or \n.'
+            end
+          end
+        else                  # we found a quoted field...
+          $1.gsub('""', '"')  # unescape contents
+        end
+        ""  # gsub!'s replacement, clear the field
+      end
+
+      # if parse is empty?(), we found all the fields on the line...
+      break csv if parse.empty?
+      # if we're not empty?() but at eof?(), a quoted field wasn't closed...
+      raise MalformedCSVError, "Unclosed quoted field." if @io.eof?
+      # otherwise, we need to loop and pull some more data to complete the row
+    end
+  end
+  alias_method :gets,     :shift
+  alias_method :readline, :shift
+end