smarter_csv 1.3.0 → 1.5.0

data/README.md

@@ -1,19 +1,24 @@
-# SmarterCSV
-
-[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.org/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
 
----------------
 #### Service Announcement
 
-Work towards SmarterCSV 2.0 is on it's way, with much improved features, and more streamlined options.
+* Work towards SmarterCSV 2.0 is still on it's way, with much improved features, and more streamlined options.
+  Please check the [2.0-develop branch](https://github.com/tilo/smarter_csv/blob/master/README.md), open any issues and pull requests with mention of v2.0.
+
+* New versions of SmarterCSV 1.x will soon print a deprecation warning if you set :verbose to true
+  See below for list of deprecated options.
 
-Please check the 2.0-develop branch, and open issues marked v2.0 and leave your comments.
+#### Restructured Branches
 
-New versions on the 1.2 branch will soon print a deprecation warning if you set :verbose to true
-See below for list of deprecated options.
+* default branch is `main` for 1.x development
+* 2.x development is on `2.0-development`
 
 ---------------
-#### SmarterCSV
+
+# SmarterCSV
+
+[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.org/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
+
+#### SmarterCSV 1.x
 
 `smarter_csv` is a Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, suitable for direct processing with Mongoid or ActiveRecord,
 and parallel processing with Resque or Sidekiq.
@@ -56,6 +61,7 @@ You can also set the `:row_sep` manually! Checkout Example 5 for unusual `:row_s
 #### Example 1a: How SmarterCSV processes CSV-files as array of hashes:
 Please note how each hash contains only the keys for columns with non-null values.
 
+```ruby
      $ cat pets.csv
      first name,last name,dogs,cats,birds,fish
      Dan,McAllister,2,,,
@@ -71,21 +77,25 @@ Please note how each hash contains only the keys for columns with non-null value
            {:first_name=>"Miles", :last_name=>"O'Brian", :fish=>"21"},
            {:first_name=>"Nancy", :last_name=>"Homes", :dogs=>"2", :birds=>"1"}
          ]
+```
 
 
 #### Example 1b: How SmarterCSV processes CSV-files as chunks, returning arrays of hashes:
 Please note how the returned array contains two sub-arrays containing the chunks which were read, each chunk containing 2 hashes.
 In case the number of rows is not cleanly divisible by `:chunk_size`, the last chunk contains fewer hashes.
 
+```ruby
      > pets_by_owner = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}})
        => [ [ {:first=>"Dan", :last=>"McAllister", :dogs=>"2"}, {:first=>"Lucy", :last=>"Laweless", :cats=>"5"} ],
             [ {:first=>"Miles", :last=>"O'Brian", :fish=>"21"}, {:first=>"Nancy", :last=>"Homes", :dogs=>"2", :birds=>"1"} ]
           ]
+```
 
 #### Example 1c: How SmarterCSV processes CSV-files as chunks, and passes arrays of hashes to a given block:
 Please note how the given block is passed the data for each chunk as the parameter (array of hashes),
 and how the `process` method returns the number of chunks when called with a block
 
+```ruby
      > total_chunks = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}}) do |chunk|
          chunk.each do |h|   # you can post-process the data from each row to your heart's content, and also create virtual attributes:
            h[:full_name] = [h[:first],h[:last]].join(' ')  # create a virtual attribute
@@ -97,16 +107,16 @@ and how the `process` method returns the number of chunks when called with a blo
        [{:dogs=>"2", :full_name=>"Dan McAllister"}, {:cats=>"5", :full_name=>"Lucy Laweless"}]
        [{:fish=>"21", :full_name=>"Miles O'Brian"}, {:dogs=>"2", :birds=>"1", :full_name=>"Nancy Homes"}]
         => 2
-
+```
 #### Example 2: Reading a CSV-File in one Chunk, returning one Array of Hashes:
-
+```ruby
     filename = '/tmp/input_file.txt' # TAB delimited file, each row ending with Control-M
     recordsA = SmarterCSV.process(filename, {:col_sep => "\t", :row_sep => "\cM"})  # no block given
 
     => returns an array of hashes
-
+```
 #### Example 3: Populate a MySQL or MongoDB Database with SmarterCSV:
-
+```ruby
     # without using chunks:
     filename = '/tmp/some.csv'
     options = {:key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}}
@@ -117,9 +127,9 @@ and how the `process` method returns the number of chunks when called with a blo
     end
 
      => returns number of chunks / rows we processed
-
+```
 #### Example 4: Populate a MongoDB Database in Chunks of 100 records with SmarterCSV:
-
+```ruby
     # using chunks:
     filename = '/tmp/some.csv'
     options = {:chunk_size => 100, :key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}}
@@ -130,10 +140,10 @@ and how the `process` method returns the number of chunks when called with a blo
     end
 
      => returns number of chunks we processed
-
+```
 
 #### Example 5: Reading a CSV-like File, and Processing it with Resque:
-
+```ruby
     filename = '/tmp/strange_db_dump'   # a file with CRTL-A as col_separator, and with CTRL-B\n as record_separator (hello iTunes!)
     options = {
       :col_sep => "\cA", :row_sep => "\cB\n", :comment_regexp => /^#/,
@@ -143,11 +153,11 @@ and how the `process` method returns the number of chunks when called with a blo
         Resque.enque( ResqueWorkerClass, chunk ) # pass chunks of CSV-data to Resque workers for parallel processing
     end
     => returns number of chunks
-
+```
 #### Example 6: Using Value Converters
 
 NOTE: If you use `key_mappings` and `value_converters`, make sure that the value converters has references the keys based on the final mapped name, not the original name in the CSV file.
-
+```ruby
     $ cat spec/fixtures/with_dates.csv
     first,last,date,price
     Ben,Miller,10/30/1998,$44.50
@@ -180,7 +190,7 @@ NOTE: If you use `key_mappings` and `value_converters`, make sure that the value
       => 44.50
     data[0][:price].class
       => Float
-
+```
 ## Parallel Processing
 [Jack](https://github.com/xjlin0) wrote an interesting article about [Speeding up CSV parsing with parallel processing](http://xjlin0.github.io/tech/2015/05/25/faster-parsing-csv-with-parallel-processing)
 
@@ -205,9 +215,9 @@ The options and the block are optional.
      | :invalid_byte_sequence      |   ''     | what to replace invalid byte sequences with                                          |
      | :force_utf8                 |   false  | force UTF-8 encoding of all lines (including headers) in the CSV file                |
      | :skip_lines                 |   nil    | how many lines to skip before the first line or header line is processed             |
-     | :comment_regexp             |   /^#/   | regular expression which matches comment lines (see NOTE about the CSV header)       |
+     | :comment_regexp             |   nil    | regular expression to ignore comment lines (see NOTE on CSV header), e.g./\A#/       |
      ---------------------------------------------------------------------------------------------------------------------------------
-     | :col_sep                    |   ','    | column separator                                                                     |
+     | :col_sep                    |   ','    | column separator, can be set to :auto                                                |
      | :force_simple_split         |   false  | force simple splitting on :col_sep character for non-standard CSV-files.             |
      |                             |          | e.g. when :quote_char is not properly escaped                                        |
      | :row_sep                    | $/ ,"\n" | row separator or record separator , defaults to system's $/ , which defaults to "\n" |
@@ -222,7 +232,7 @@ The options and the block are optional.
      |                             |          | user provided Array of header strings or symbols, to define                          |
      |                             |          | what headers should be used, overriding any in-file headers.                         |
      |                             |          | You can not combine the :user_provided_headers and :key_mapping options              |
-     | :remove_empty_hashes        |   true   | remove / ignore any hashes which don't have any key/value pairs                      |
+     | :remove_empty_hashes        |   true   | remove / ignore any hashes which don't have any key/value pairs or all empty values  |
      | :verbose                    |   false  | print out line number while processing (to track down problems in input files)       |
      ---------------------------------------------------------------------------------------------------------------------------------
 
@@ -248,7 +258,7 @@ And header and data validations will also be supported in 2.x
      ---------------------------------------------------------------------------------------------------------------------------------
      | :value_converters           |   nil    | supply a hash of :header => KlassName; the class needs to implement self.convert(val)|
      | :remove_empty_values        |   true   | remove values which have nil or empty strings as values                              |
-     | :remove_zero_values         |   true   | remove values which have a numeric value equal to zero / 0                           |
+     | :remove_zero_values         |   false  | remove values which have a numeric value equal to zero / 0                           |
      | :remove_values_matching     |   nil    | removes key/value pairs if value matches given regular expressions. e.g.:            |
      |                             |          | /^\$0\.0+$/ to match $0.00 , or /^#VALUE!$/ to match errors in Excel spreadsheets    |
      | :convert_values_to_numeric  |   true   | converts strings containing Integers or Floats to the appropriate class              |
@@ -259,22 +269,23 @@ And header and data validations will also be supported in 2.x
 #### NOTES about File Encodings:
  * if you have a CSV file which contains unicode characters, you can process it as follows:
 
-
+```ruby
        File.open(filename, "r:bom|utf-8") do |f|
          data = SmarterCSV.process(f);
        end
-
+```
 * if the CSV file with unicode characters is in a remote location, similarly you need to give the encoding as an option to the `open` call:
-
+```ruby
        require 'open-uri'
        file_location = 'http://your.remote.org/sample.csv'
        open(file_location, 'r:utf-8') do |f|   # don't forget to specify the UTF-8 encoding!!
          data = SmarterCSV.process(f)
        end
-
+```
 #### NOTES about CSV Headers:
  * as this method parses CSV files, it is assumed that the first line of any file will contain a valid header
- * the first line with the CSV header may or may not be commented out according to the :comment_regexp
+ * the first line with the header might be commented out, in which case you will need to set `comment_regexp: /\A#/`
+   This is no longer handled automatically since 1.5.0.
  * any occurences of :comment_regexp or :row_sep will be stripped from the first line with the CSV header
  * any of the keys in the header line will be downcased, spaces replaced by underscore, and converted to Ruby symbols before being used as keys in the returned Hashes
  * you can not combine the :user_provided_headers and :key_mapping options
@@ -305,227 +316,27 @@ And header and data validations will also be supported in 2.x
 ## Installation
 
 Add this line to your application's Gemfile:
-
+```ruby
     gem 'smarter_csv'
-
+```
 And then execute:
-
+```ruby
     $ bundle
-
+```
 Or install it yourself as:
-
+```ruby
     $ gem install smarter_csv
-
-## Upcoming
-
-Planned in the next releases:
- * programmatic header transformations
- * CSV command line
-
-## Changes
-
-#### 1.3.0 (2022-01-06) Breaking code change if you used `--key_mappings`
- * fix bug for key_mappings (issue #181)   
-   The values of the `key_mappings` hash will now be used "as is", and no longer forced to be symbols
-
-   **Users with existing code with `--key_mappings` need to change their code** to 
-     * either use symbols in the `key_mapping` hash
-     * or change the expected keys from symbols to strings
-
-#### 1.2.9 (2021-11-22) (PULLED)
- * fix bug for key_mappings (issue #181)
-   The values of the `key_mappings` hash will now be used "as is", and no longer forced to be symbols
-
-#### 1.2.8 (2020-02-04)
- * fix deprecation warnings on Ruby 2.7 (thank to Diego Salido)
-
-#### 1.2.7 (2020-02-03)
-
-#### 1.2.6 (2018-11-13)
- * fixing error caused by calling f.close when we do not hand in a file
-
-#### 1.2.5 (2018-09-16)
- * fixing issue #136 with comments in CSV files
- * fixing error class hierarchy
-
-#### 1.2.4 (2018-08-06)
- * using Rails blank? if it's available
-
-#### 1.2.3 (2018-01-27)
- * fixed regression / test
- * fuxed quote_char interpolation for headers, but not data (thanks to Colin Petruno)
- * bugfix (thanks to Joshua Smith for reporting)
-
-#### 1.2.0 (2018-01-20)
- * add default validation that a header can only appear once
- * add option `required_headers`
-
-#### 1.1.5 (2017-11-05)
- * fix issue with invalid byte sequences in header (issue #103, thanks to Dave Myron)
- * fix issue with invalid byte sequences in multi-line data (thanks to Ivan Ushakov)
- * analyze only 500 characters by default when `:row_sep => :auto` is used.
-   added option `row_sep_auto_chars` to change the default if necessary. (thanks to Matthieu Paret)
-
-#### 1.1.4 (2017-01-16)
- * fixing UTF-8 related bug which was introduced in 1.1.2 (thanks to Tirdad C.)
-
-#### 1.1.3 (2016-12-30)
- * added warning when options indicate UTF-8 processing, but input filehandle is not opened with r:UTF-8 option
-
-#### 1.1.2 (2016-12-29)
- * added option `invalid_byte_sequence` (thanks to polycarpou)
- * added comments on handling of UTF-8 encoding when opening from File vs. OpenURI (thanks to KevinColemanInc)
-
-#### 1.1.1 (2016-11-26)
- * added option to `skip_lines` (thanks to wal)
- * added option to `force_utf8` encoding (thanks to jordangraft)
- * bugfix if no headers in input data (thanks to esBeee)
- * ensure input file is closed (thanks to waldyr)
- * improved verbose output (thankd to benmaher)
- * improved documentation
-
-#### 1.1.0 (2015-07-26)
- * added feature :value_converters, which allows parsing of dates, money, and other things (thanks to Raphaël Bleuse, Lucas Camargo de Almeida, Alejandro)
- * added error if :headers_in_file is set to false, and no :user_provided_headers are given (thanks to innhyu)
- * added support to convert dashes to underscore characters in headers (thanks to César Camacho)
- * fixing automatic detection of \r\n line-endings (thanks to feens)
-
-#### 1.0.19 (2014-10-29)
- * added option :keep_original_headers to keep CSV-headers as-is (thanks to Benjamin Thouret)
-
-#### 1.0.18 (2014-10-27)
- * added support for multi-line fields / csv fields containing CR (thanks to Chris Hilton) (issue #31)
-
-#### 1.0.17 (2014-01-13)
- * added option to set :row_sep to :auto , for automatic detection of the row-separator (issue #22)
-
-#### 1.0.16 (2014-01-13)
- * :convert_values_to_numeric option can now be qualified with :except or :only (thanks to Hugo Lepetit)
- * removed deprecated `process_csv` method
-
-#### 1.0.15 (2013-12-07)
- * new option:
-   * :remove_unmapped_keys  to completely ignore columns which were not mapped with :key_mapping (thanks to Dave Sanders)
-
-#### 1.0.14 (2013-11-01)
- * added GPL-2 and MIT license to GEM spec file; if you need another license contact me
-
-#### 1.0.12 (2013-10-15)
- * added RSpec tests
-
-#### 1.0.11 (2013-09-28)
- * bugfix : fixed issue #18 - fixing issue with last chunk not being properly returned (thanks to Jordan Running)
- * added RSpec tests
-
-#### 1.0.10 (2013-06-26)
- * bugfix : fixed issue #14 - passing options along to CSV.parse (thanks to Marcos Zimmermann)
-
-#### 1.0.9 (2013-06-19)
- * bugfix : fixed issue #13 with negative integers and floats not being correctly converted (thanks to Graham Wetzler)
-
-#### 1.0.8 (2013-06-01)
-
- * bugfix : fixed issue with nil values in inputs with quote-char (thanks to Félix Bellanger)
- * new options:
-    * :force_simple_split : to force simiple splitting on :col_sep character for non-standard CSV-files. e.g. without properly escaped :quote_char
-    * :verbose : print out line number while processing (to track down problems in input files)
-
-#### 1.0.7 (2013-05-20)
-
- * allowing process to work with objects with a 'readline' method (thanks to taq)
- * added options:
-    * :file_encoding : defaults to utf8  (thanks to MrTin, Paxa)
-
-#### 1.0.6 (2013-05-19)
-
- * bugfix : quoted fields are now correctly parsed
-
-#### 1.0.5 (2013-05-08)
-
- * bugfix : for :headers_in_file option
-
-#### 1.0.4 (2012-08-17)
-
- * renamed the following options:
-    * :strip_whitepace_from_values => :strip_whitespace   - removes leading/trailing whitespace from headers and values
-
-#### 1.0.3 (2012-08-16)
-
- * added the following options:
-    * :strip_whitepace_from_values   - removes leading/trailing whitespace from values
-
-#### 1.0.2 (2012-08-02)
-
- * added more options for dealing with headers:
-    * :user_provided_headers ,user provided Array with header strings or symbols, to precisely define what the headers should be, overriding any in-file headers (default: nil)
-    * :headers_in_file , if the file contains headers as the first line (default: true)
-
-#### 1.0.1 (2012-07-30)
-
- * added the following options:
-    * :downcase_header
-    * :strings_as_keys
-    * :remove_zero_values
-    * :remove_values_matching
-    * :remove_empty_hashes
-    * :convert_values_to_numeric
-
- * renamed the following options:
-    * :remove_empty_fields => :remove_empty_values
-
-
-#### 1.0.0 (2012-07-29)
-
- * renamed `SmarterCSV.process_csv` to `SmarterCSV.process`.
-
-#### 1.0.0.pre1 (2012-07-29)
-
+```
+## [ChangeLog](./CHANGELOG.md)
 
 ## Reporting Bugs / Feature Requests
 
 Please [open an Issue on GitHub](https://github.com/tilo/smarter_csv/issues) if you have feedback, new feature requests, or want to report a bug. Thank you!
 
+  * please include a small sample CSV file
+  * please mention your version of SmarterCSV, Ruby, Rails
 
-## Special Thanks
-
-Many thanks to people who have filed issues and sent comments.
-And a special thanks to those who contributed pull requests:
-
- * [Jack 0](https://github.com/xjlin0)
- * [Alejandro](https://github.com/agaviria)
- * [Lucas Camargo de Almeida](https://github.com/lcalmeida)
- * [Raphaël Bleuse](https://github.com/bleuse)
- * [feens](https://github.com/feens)
- * [César Camacho](https://github.com/chanko)
- * [innhyu](https://github.com/innhyu)
- * [Benjamin Thouret](https://github.com/benichu)
- * [Chris Hilton](https://github.com/chrismhilton)
- * [Sean Duckett](http://github.com/sduckett)
- * [Alex Ong](http://github.com/khaong)
- * [Martin Nilsson](http://github.com/MrTin)
- * [Eustáquio Rangel](http://github.com/taq)
- * [Pavel](http://github.com/paxa)
- * [Félix Bellanger](https://github.com/Keeguon)
- * [Graham Wetzler](https://github.com/grahamwetzler)
- * [Marcos G. Zimmermann](https://github.com/marcosgz)
- * [Jordan Running](https://github.com/jrunning)
- * [Dave Sanders](https://github.com/DaveSanders)
- * [Hugo Lepetit](https://github.com/giglemad)
- * [esBeee](https://github.com/esBeee)
- * [Waldyr de Souza](https://github.com/waldyr)
- * [Ben Maher](https://github.com/benmaher)
- * [Wal McConnell](https://github.com/wal)
- * [Jordan Graft](https://github.com/jordangraft)
- * [Michael](https://github.com/polycarpou)
- * [Kevin Coleman](https://github.com/KevinColemanInc)
- * [Tirdad C.](https://github.com/tridadc)
- * [Dave Myron](https://github.com/contentfree)
- * [Ivan Ushakov](https://github.com/IvanUshakov)
- * [Matthieu Paret](https://github.com/mtparet)
- * [Rohit Amarnath](https://github.com/ramarnat)
- * [Joshua Smith](https://github.com/enviable)
- * [Colin Petruno](https://github.com/colinpetruno)
- * [Diego Salido](https://github.com/salidux)
+## [A Special Thanks to all Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
 
 
 ## Contributing

data/Rakefile

@@ -1,26 +1,19 @@
 #!/usr/bin/env rake
 require "bundler/gem_tasks"
-
 require 'rubygems'
 require 'rake'
-
 require 'rspec/core/rake_task'
 
+task :default => :spec
+
 desc "Run RSpec"
 RSpec::Core::RakeTask.new do |t|
-  t.verbose = false
+  # t.verbose = false
 end
 
-desc "Run specs for all test cases"
-task :spec_all do 
-  system "rake spec"
+desc 'Run spec with coverage'
+task :coverage do
+  ENV['COVERAGE'] = 'true'
+  Rake::Task['spec'].execute
+  `open coverage/index.html`
 end
-
-# task :spec_all do
-#   %w[active_record data_mapper mongoid].each do |model_adapter|
-#     puts "MODEL_ADAPTER = #{model_adapter}"
-#     system "rake spec MODEL_ADAPTER=#{model_adapter}"
-#   end
-# end
-
-task :default => :spec