smarter_csv 1.0.0 → 1.0.1

data/README.md

@@ -3,15 +3,17 @@
 `smarter_csv` is a Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, suitable for direct processing with Mongoid or ActiveRecord, 
 and parallel processing with Resque or Sidekiq.
 
-`smarter_csv` has lots of optional features:
+`smarter_csv` has lots of features:
  * able to process large CSV-files
  * able to chunk the input from the CSV file to avoid loading the whole CSV file into memory
  * return a Hash for each line of the CSV file, so we can quickly use the results for either creating MongoDB or ActiveRecord entries, or further processing with Resque
- * able to pass a block to the method, so data from the CSV file can be directly processed (e.g. Resque.enqueue )
- * have a bit more flexible input format, where comments are possible, and col_sep,row_sep can be set to any character sequence, including control characters.
+ * able to pass a block to the `process` method, so data from the CSV file can be directly processed (e.g. Resque.enqueue )
+ * allows to have a bit more flexible input format, where comments are possible, and col_sep,row_sep can be set to any character sequence, including control characters.
  * able to re-map CSV "column names" to Hash-keys of your choice (normalization)
  * able to ignore "columns" in the input (delete columns)
- * able to eliminate nil or empty fields from the result hashes
+ * able to eliminate nil or empty fields from the result hashes (default)
+
+NOTE; This Gem is only for importing CSV files - writing of CSV files is not supported.
 
 ### Why?
 
@@ -20,14 +22,64 @@ Ruby's CSV library's API is pretty old, and it's processing of CSV-files returni
 As the existing CSV libraries didn't fit my needs, I was writing my own CSV processing - specifically for use in connection with Rails ORMs like Mongoid, MongoMapper or ActiveRecord. In those ORMs you can easily pass a hash with attribute/value pairs to the create() method. The lower-level Mongo driver and Moped also accept larger arrays of such hashes to create a larger amount of records quickly with just one call.
 
 ### Examples
-#### Example 1: Reading a CSV-File in one Chunk, returning one Array of Hashes:
+
+The two main choices you have in terms of how to call `SmarterCSV.process` are:
+ * calling `process` with or without a block
+ * passing a `:chunk_size` to the `process` method, and processing the CSV-file in chunks, rather than in one piece.
+
+#### Example 1a: How SmarterCSV processes CSV-files as array of hashes:
+Please note how each hash contains only the keys for columns with non-null values.
+
+     $ cat pets.csv
+     first name,last name,dogs,cats,birds,fish
+     Dan,McAllister,2,,,
+     Lucy,Laweless,,5,,
+     Miles,O'Brian,,,,21
+     Nancy,Homes,2,,1, 
+     $ irb
+     > require 'smarter_csv'
+      => true 
+     > pets_by_owner = SmarterCSV.process('/tmp/pets.csv')
+      => [ {:first_name=>"Dan", :last_name=>"McAllister", :dogs=>"2"},
+           {:first_name=>"Lucy", :last_name=>"Laweless", :cats=>"5"}, 
+           {:first_name=>"Miles", :last_name=>"O'Brian", :fish=>"21"}, 
+           {:first_name=>"Nancy", :last_name=>"Homes", :dogs=>"2", :birds=>"1"} 
+         ]
+
+
+#### Example 1b: How SmarterCSV processes CSV-files as chunks, returning arrays of hashes:
+Please note how the returned array contains two sub-arrays containing the chunks which were read, each chunk containing 2 hashes.
+In case the number of rows is not cleanly divisible by `:chunk_size`, the last chunk contains fewer hashes.
+
+     > pets_by_owner = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}})
+       => [ [ {:first=>"Dan", :last=>"McAllister", :dogs=>"2"}, {:first=>"Lucy", :last=>"Laweless", :cats=>"5"} ], 
+            [ {:first=>"Miles", :last=>"O'Brian", :fish=>"21"}, {:first=>"Nancy", :last=>"Homes", :dogs=>"2", :birds=>"1"} ]
+          ]
+
+#### Example 1c: How SmarterCSV processes CSV-files as chunks, and passes arrays of hashes to a given block:
+Please note how the given block is passed the data for each chunk as the parameter (array of hashes),
+and how the `process` method returns the number of chunks when called with a block
+
+     > total_chunks = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}}) do |chunk|
+         chunk.each do |h|   # you can post-process the data from each row to your heart's content, and also create virtual attributes:
+           h[:full_name] = [h[:first],h[:last]].join(' ')  # create a virtual attribute
+           h.delete(:first) ; h.delete(:last)              # remove two keys
+         end
+         puts chunk.inspect   # we could at this point pass the chunk to a Resque worker..
+       end
+
+       [{:dogs=>"2", :full_name=>"Dan McAllister"}, {:cats=>"5", :full_name=>"Lucy Laweless"}]
+       [{:fish=>"21", :full_name=>"Miles O'Brian"}, {:dogs=>"2", :birds=>"1", :full_name=>"Nancy Homes"}]
+        => 2 
+
+#### Example 2: Reading a CSV-File in one Chunk, returning one Array of Hashes:
 
     filename = '/tmp/input_file.txt' # TAB delimited file, each row ending with Control-M
-    recordsA = SmarterCSV.process(filename, {:col_sep => "\t", :row_sep => "\cM"}
+    recordsA = SmarterCSV.process(filename, {:col_sep => "\t", :row_sep => "\cM"})  # no block given
 
     => returns an array of hashes
 
-#### Example 2: Populate a MySQL or MongoDB Database with SmarterCSV:
+#### Example 3: Populate a MySQL or MongoDB Database with SmarterCSV:
 
     # without using chunks:
     filename = '/tmp/some.csv'
@@ -40,29 +92,71 @@ As the existing CSV libraries didn't fit my needs, I was writing my own CSV proc
      => returns number of chunks / rows we processed 
 
 
-#### Example 3: Populate a MongoDB Database in Chunks of 100 records with SmarterCSV:
+#### Example 4: Populate a MongoDB Database in Chunks of 100 records with SmarterCSV:
 
     # using chunks:
     filename = '/tmp/some.csv'
-    n = SmarterCSV.process(filename, {:key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}, :chunk_size => 100}) do |array|
+    n = SmarterCSV.process(filename, {:chunk_size => 100, :key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}}) do |chunk|
           # we're passing a block in, to process each resulting hash / row (block takes array of hashes)
-          # when chunking is enabled, there are up to :chunk_size hashes in each array
-          MyModel.collection.insert( array )   # insert up to 100 records at a time
+          # when chunking is enabled, there are up to :chunk_size hashes in each chunk
+          MyModel.collection.insert( chunk )   # insert up to 100 records at a time
     end
 
      => returns number of chunks we processed
 
 
-#### Example 4: Reading a CSV-like File, and Processing it with Resque:
+#### Example 5: Reading a CSV-like File, and Processing it with Resque:
 
     filename = '/tmp/strange_db_dump'   # a file with CRTL-A as col_separator, and with CTRL-B\n as record_separator (hello iTunes)
     n = SmarterCSV.process(filename, {:col_sep => "\cA", :row_sep => "\cB\n", :comment_regexp => /^#/,
-            :chunk_size => '5' , :key_mapping => {:export_date => nil, :name => :genre}}) do |x|
-        puts   "Resque.enque( ResqueWorkerClass, #{x.size}, #{x.inspect} )"   # simulate processing each chunk
+            :chunk_size => 100 , :key_mapping => {:export_date => nil, :name => :genre}}) do |chunk|
+        Resque.enque( ResqueWorkerClass, chunk ) # pass chunks of CSV-data to Resque workers for parallel processing
     end
     => returns number of chunks
 
 
+## Documentation
+
+The `process` method reads and processes a "generalized" CSV file and returns the contents either as an Array of Hashes,
+or an Array of Arrays, which contain Hashes, or processes Chunks of Hashes via a given block.
+
+    SmarterCSV.process(filename, options={}, &block)
+
+The options and the block are optional.
+
+`SmarterCSV.process` supports the following options:
+ * :col_sep : column separator , which defaults to ','
+ * :row_sep : row separator or record separator , defaults to system's $/ , which defaults to "\n"
+ * :quote_char : quotation character , defaults to '"'
+ * :comment_regexp : regular expression which matches comment lines , defaults to /^#/ (see NOTE about the CSV header)
+ * :chunk_size : if set, determines the desired chunk-size (defaults to nil, no chunk processing)
+ * :key_mapping : a hash which maps headers from the CSV file to keys in the result hash (default: nil)
+ * :downcase_header : downcase all column headers (default: true)
+ * :strings_as_keys : use strings instead of symbols as the keys in the result hashes (default: false)
+ * :remove_empty_values : remove values which have nil or empty strings as values (default: true)
+ * :remove_zero_values  : remove values which have a numeric value equal to zero / 0 (default: false)
+ * :remove_values_matching : removes key/value pairs if value matches given regular expressions (default: nil) , e.g. /^\$0\.0+$/ to match $0.00 , or /^#VALUE!$/ to match errors in Excel spreadsheets
+ * :convert_values_to_numeric : converts strings containing Integers or Floats to the appropriate class (default: true)
+ * :remove_empty_hashes : remove / ignore any hashes which don't have any key/value pairs (default: true)
+
+#### NOTES about CSV Headers:
+ * as this method parses CSV files, it is assumed that the first line of any file will contain a valid header
+ * the first line with the CSV header may or may not be commented out according to the :comment_regexp
+ * any occurences of :comment_regexp or :row_sep will be stripped from the first line with the CSV header
+ * any of the keys in the header line will be downcased, spaces replaced by underscore, and converted to Ruby symbols before being used as keys in the returned Hashes
+
+#### NOTES on Key Mapping:
+ * keys in the header line of the file can be re-mapped to a chosen set of symbols, so the resulting Hashes can be better used internally in your application (e.g. when directly creating MongoDB entries with them)
+ * if you want to completely delete a key, then map it to nil or to '', they will be automatically deleted from any result Hash
+
+#### NOTES on the use of Chunking and Blocks:
+ * chunking can be VERY USEFUL if used in combination with passing a block to File.read_csv FOR LARGE FILES
+ * if you pass a block to File.read_csv, that block will be executed and given an Array of Hashes as the parameter.
+ * if the chunk_size is not set, then the array will only contain one Hash.
+ * if the chunk_size is > 0 , then the array may contain up to chunk_size Hashes.
+ * this can be very useful when passing chunked data to a post-processing step, e.g. through Resque
+
+
 ## See also:
 
   http://www.unixgods.org/~tilo/Ruby/process_csv_as_hashes.html
@@ -83,9 +177,35 @@ Or install it yourself as:
 
     $ gem install smarter_csv
 
-## Usage
 
-TODO: Write usage instructions here
+## Changes
+
+#### 1.0.1 (2012-07-30)
+
+ * added the following options: 
+    * :downcase_header
+    * :strings_as_keys
+    * :remove_zero_values
+    * :remove_values_matching
+    * :remove_empty_hashes
+    * :convert_values_to_numeric
+
+ * renamed the following options:
+    * :remove_empty_fields => :remove_empty_values
+    
+
+#### 1.0.0 (2012-07-29)
+
+ * renamed `SmarterCSV.process_csv` to `SmarterCSV.process`.
+
+#### 1.0.0.pre1 (2012-07-29)
+
+
+## Reporting Bugs / Feature Requests
+
+Please [open an Issue on GitHub](https://github.com/tilo/smarter_csv/issues) if you have feedback, new feature requests, or want to report a bug. Thank you!
+
+
 
 ## Contributing
 

data/lib/smarter_csv/smarter_csv.rb

@@ -1,37 +1,9 @@
 module SmarterCSV
-  # this reads and processes a "generalized" CSV file and returns the contents either as an Array of Hashes,
-  # or an Array of Arrays, which contain Hashes, or processes Chunks of Hashes via a given block
-  #
-  # File.read_csv supports the following options:
-  #  * :col_sep : column separator , which defaults to ','
-  #  * :row_sep : row separator or record separator , defaults to system's $/ , which defaults to "\n"
-  #  * :quote_char : quotation character , defaults to '"' (currently not used)
-  #  * :comment_regexp : regular expression which matches comment lines , defaults to /^#/ (see NOTE about the CSV header)
-  #  * :chunk_size : if set, determines the desired chunk-size (defaults to nil, no chunk processing)
-  #  * :remove_empty_fields : remove fields which have nil or empty strings as values (default: true)
-  #
-  # NOTES about CSV Headers:
-  #  - as this method parses CSV files, it is assumed that the first line of any file will contain a valid header
-  #  - the first line with the CSV header may or may not be commented out according to the :comment_regexp
-  #  - any occurences of :comment_regexp or :row_sep will be stripped from the first line with the CSV header
-  #  - any of the keys in the header line will be converted to Ruby symbols before being used in the returned Hashes
-  #
-  # NOTES on Key Mapping:
-  #  - keys in the header line of the file can be re-mapped to a chosen set of symbols, so the resulting Hashes
-  #    can be better used internally in our application (e.g. when directly creating MongoDB entries with them)
-  #  - if you want to completely delete a key, then map it to nil or to '', they will be automatically deleted from any result Hash
-  #
-  # NOTES on the use of Chunking and Blocks:
-  #  - chunking can be VERY USEFUL if used in combination with passing a block to File.read_csv FOR LARGE FILES
-  #  - if you pass a block to File.read_csv, that block will be executed and given an Array of Hashes as the parameter.
-  #    If the chunk_size is not set, then the array will only contain one Hash.
-  #    If the chunk_size is > 0 , then the array may contain up to chunk_size Hashes.
-  #    This can be very useful when passing chunked data to a post-processing step, e.g. through Resque
-  #
-
   def SmarterCSV.process(filename, options={}, &block)
-    default_options = {:col_sep => ',' , :row_sep => $/ , :quote_char => '"', :remove_empty_fields => true,
-      :comment_regexp => /^#/, :chunk_size => nil , :key_mapping_hash => nil
+    default_options = {:col_sep => ',' , :row_sep => $/ , :quote_char => '"',
+      :remove_empty_values => true, :remove_zero_values => false , :remove_values_matching => nil , :remove_empty_hashes => true ,
+      :convert_values_to_numeric => true, :strip_chars_from_headers => nil ,
+      :comment_regexp => /^#/, :chunk_size => nil , :key_mapping_hash => nil , :downcase_header => true, :strings_as_keys => false 
     }
     options = default_options.merge(options)
     headerA = []
@@ -43,7 +15,11 @@ module SmarterCSV
       
       # process the header line in the CSV file..
       # the first line of a CSV file contains the header .. it might be commented out, so we need to read it anyhow
-      headerA = f.readline.sub(options[:comment_regexp],'').chomp(options[:row_sep]).split(options[:col_sep]).map{|x| x.gsub(%r/options[:quote_char]/,'').gsub(/\s+/,'_').to_sym}
+      headerA = f.readline.sub(options[:comment_regexp],'').chomp(options[:row_sep])
+      headerA = headerA.gsub(options[:strip_chars_from_headers], '') if options[:strip_chars_from_headers]
+      headerA = headerA.split(options[:col_sep]).map{|x| x.gsub(%r/options[:quote_char]/,'').gsub(/\s+/,'_')}
+      headerA.map!{|x| x.downcase }   if options[:downcase_header]
+      headerA.map!{|x| x.to_sym } unless options[:strings_as_keys]
       key_mappingH = options[:key_mapping]
       
       # do some key mapping on the keys in the file header
@@ -72,8 +48,21 @@ module SmarterCSV
         hash = Hash.zip(headerA,dataA)  # from Facets of Ruby library
         # make sure we delete any key/value pairs from the hash, which the user wanted to delete:
         hash.delete(nil); hash.delete(''); hash.delete(:"") # delete any hash keys which were mapped to be deleted
-        hash.delete_if{|k,v| v.nil? || v =~ /^\s*$/}  if options[:remove_empty_fields]
-        
+        hash.delete_if{|k,v| v.nil? || v =~ /^\s*$/}  if options[:remove_empty_values]
+        hash.delete_if{|k,v| ! v.nil? && v =~ /^(\d+|\d+\.\d+)$/ && v.to_f == 0} if options[:remove_zero_values]   # values are typically Strings!
+        hash.delete_if{|k,v| v =~ options[:remove_values_matching]} if options[:remove_values_matching]
+        if options[:convert_values_to_numeric]
+          hash.each do |k,v|
+            case v
+            when /^\d+$/
+              hash[k] = v.to_i 
+            when /^\d+\.\d+$/
+              hash[k] = v.to_f
+            end
+          end
+        end
+        next if hash.empty? if options[:remove_empty_hashes]
+
         if use_chunks
           chunk << hash  # append temp result to chunk
           

data/lib/smarter_csv/version.rb

@@ -1,3 +1,3 @@
 module SmarterCSV
-  VERSION = "1.0.0"
+  VERSION = "1.0.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: smarter_csv
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.0.1
   prerelease: 
 platform: ruby
 authors:
@@ -11,7 +11,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-07-29 00:00:00.000000000 Z
+date: 2012-07-30 00:00:00.000000000 Z
 dependencies: []
 description: Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, with
   optional features for processing large files in parallel, embedded comments, unusual