smarter_csv 1.4.0 → 1.5.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c8236e4cc8f0081efd9b74f12ad4b5342707d0a2f883414b07538160910008a3
-  data.tar.gz: b04a53b0030bf6c623aa19fb15c0c6c5ca123ce2ff85d47f176884fffa0f9811
+  metadata.gz: 352cf76ac0cd6b2eb4a1cac9e5056aa6e92a8a61b627d7c922e063dcf82ad675
+  data.tar.gz: 0c6e3ab1eaee02a9361fe0b418191244d81bc558dbcd10ee1d2c5f15390d91b6
 SHA512:
-  metadata.gz: f2ddaa7bf44362c8bb4439289172d40b6ca926a67a8a35fb335473ddf7349658a629f3008ece5314c6bc5fa17145a2ae89b4d706b9c130a1642a51f2434d5e21
-  data.tar.gz: b48908b657a07589886873fe251263dabbe6e2333a1fc025dfede085841544458d4498ba1d288a4a7c0de3875d1c14631cc584b2a1cb7fd0be1543b758781dd3
+  metadata.gz: 3763cd8e493e7da6560e8ce9adc58bd411f745f5af119c97d70c02667a524ccb1055b5c640ef795c3cb25b79fa5e17800018da6e76f9d358afa1c7a3513caae3
+  data.tar.gz: '039183fdece20e80007f3f0d3e395fac8d273df6c21928a35b685ade5915503b3c55918fc7015ead0a7d768a545bb76bcfb1087006af6118dc3d22df83e68ddb'

data/.gitignore

@@ -6,3 +6,5 @@
 .bundle
 Gemfile.lock
 pkg/*
+coverage/*
+.DS_Store

data/CHANGELOG.md

@@ -1,14 +1,36 @@
 
 # SmarterCSV 1.x Change Log
 
-## 1.4.0 (2022-01-11)
+## 1.5.1 (2022-04-26)
+  * added raising of `KeyMappingError` if `key_mapping` refers to a non-existent key
+  * added option `duplicate_header_suffix` (thanks to Skye Shaw)
+    When given a non-nil string, it uses the suffix to append numbering 2..n to duplicate headers.
+    If your code will need to process arbitrary CSV files, please set `duplicate_header_suffix`.
+
+## 1.5.0 (2022-04-25)
+  * fixed bug with trailing col_sep characters, introduced in 1.4.0
+  * Fix deprecation warning in Ruby 3.0.3 / $INPUT_RECORD_SEPARATOR (thanks to Joel Fouse )
+
+  * changed default for `comment_regexp` to be `nil` for a safer default behavior (thanks to David Lazar)
+  **Note**
+    This no longer assumes that lines starting with `#` are comments.
+    If you want to treat lines starting with '#' as comments, use `comment_regexp: /\A#/`
+
+## 1.4.2 (2022-02-12)
+  * fixed issue with simplecov
+
+## 1.4.1 (2022-02-12) (PULLED)
+  * minor fix: also support `col_sep: :auto`
+  * added simplecov
+
+## 1.4.0 (2022-02-11)
   * dropped GPL license, smarter_csv is now only using the MIT License
   * added experimental option `col_sep: 'auto` to auto-detect the column separator (issue #183)
     The default behavior is still to assume `,` is the column separator. 
   * fixed buggy behavior when using `remove_empty_values: false` (issue #168)
   * fixed Ruby 3.0 deprecation
 
-## 1.3.0 (2022-01-06) Breaking code change if you used `--key_mappings`
+## 1.3.0 (2022-02-06) Breaking code change if you used `--key_mappings`
  * fix bug for key_mappings (issue #181)   
    The values of the `key_mappings` hash will now be used "as is", and no longer forced to be symbols
 

data/CONTRIBUTORS.md

@@ -0,0 +1,46 @@
+# A Big Thank You to all the Contributors!!
+
+
+A Big Thank you to everyone who filed issues, sent comments, and who contributed with pull requests:
+
+ * [Jack 0](https://github.com/xjlin0)
+ * [Alejandro](https://github.com/agaviria)
+ * [Lucas Camargo de Almeida](https://github.com/lcalmeida)
+ * [Raphaël Bleuse](https://github.com/bleuse)
+ * [feens](https://github.com/feens)
+ * [César Camacho](https://github.com/chanko)
+ * [innhyu](https://github.com/innhyu)
+ * [Benjamin Thouret](https://github.com/benichu)
+ * [Chris Hilton](https://github.com/chrismhilton)
+ * [Sean Duckett](http://github.com/sduckett)
+ * [Alex Ong](http://github.com/khaong)
+ * [Martin Nilsson](http://github.com/MrTin)
+ * [Eustáquio Rangel](http://github.com/taq)
+ * [Pavel](http://github.com/paxa)
+ * [Félix Bellanger](https://github.com/Keeguon)
+ * [Graham Wetzler](https://github.com/grahamwetzler)
+ * [Marcos G. Zimmermann](https://github.com/marcosgz)
+ * [Jordan Running](https://github.com/jrunning)
+ * [Dave Sanders](https://github.com/DaveSanders)
+ * [Hugo Lepetit](https://github.com/giglemad)
+ * [esBeee](https://github.com/esBeee)
+ * [Waldyr de Souza](https://github.com/waldyr)
+ * [Ben Maher](https://github.com/benmaher)
+ * [Wal McConnell](https://github.com/wal)
+ * [Jordan Graft](https://github.com/jordangraft)
+ * [Michael](https://github.com/polycarpou)
+ * [Kevin Coleman](https://github.com/KevinColemanInc)
+ * [Tirdad C.](https://github.com/tridadc)
+ * [Dave Myron](https://github.com/contentfree)
+ * [Ivan Ushakov](https://github.com/IvanUshakov)
+ * [Matthieu Paret](https://github.com/mtparet)
+ * [Rohit Amarnath](https://github.com/ramarnat)
+ * [Joshua Smith](https://github.com/enviable)
+ * [Colin Petruno](https://github.com/colinpetruno)
+ * [Diego Salido](https://github.com/salidux)
+ * [Elie](https://github.com/elieteyssedou)
+ * [Chris Wong](https://github.com/lightwave)
+ * [Olle Jonsson](https://github.com/olleolleolle)
+ * [Nicolas Guillemain](https://github.com/Viiruus)
+ * [Sp6](https://github.com/sp6)
+ * [Joel Fouse](https://github.com/jfouse)

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2022 Tilo Sloboda
+Copyright (c) 2012..2022 Tilo Sloboda
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,17 +1,23 @@
-# SmarterCSV
-
-[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.org/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
 
----------------
 #### Service Announcement
 
 * Work towards SmarterCSV 2.0 is still on it's way, with much improved features, and more streamlined options.
-  Please check the 2.0-develop branch, open any issues and pull requests with mention of v2.0.
+  Please check the [2.0-develop branch](https://github.com/tilo/smarter_csv/blob/master/README.md), open any issues and pull requests with mention of v2.0.
 
-* New versions on the 1.2 branch will soon print a deprecation warning if you set :verbose to true
+* New versions of SmarterCSV 1.x will soon print a deprecation warning if you set :verbose to true
   See below for list of deprecated options.
 
+#### Restructured Branches
+
+* default branch is `main` for 1.x development
+* 2.x development is on `2.0-development`
+
 ---------------
+
+# SmarterCSV
+
+[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.org/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
+
 #### SmarterCSV 1.x
 
 `smarter_csv` is a Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, suitable for direct processing with Mongoid or ActiveRecord,
@@ -55,6 +61,7 @@ You can also set the `:row_sep` manually! Checkout Example 5 for unusual `:row_s
 #### Example 1a: How SmarterCSV processes CSV-files as array of hashes:
 Please note how each hash contains only the keys for columns with non-null values.
 
+```ruby
      $ cat pets.csv
      first name,last name,dogs,cats,birds,fish
      Dan,McAllister,2,,,
@@ -70,21 +77,25 @@ Please note how each hash contains only the keys for columns with non-null value
            {:first_name=>"Miles", :last_name=>"O'Brian", :fish=>"21"},
            {:first_name=>"Nancy", :last_name=>"Homes", :dogs=>"2", :birds=>"1"}
          ]
+```
 
 
 #### Example 1b: How SmarterCSV processes CSV-files as chunks, returning arrays of hashes:
 Please note how the returned array contains two sub-arrays containing the chunks which were read, each chunk containing 2 hashes.
 In case the number of rows is not cleanly divisible by `:chunk_size`, the last chunk contains fewer hashes.
 
+```ruby
      > pets_by_owner = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}})
        => [ [ {:first=>"Dan", :last=>"McAllister", :dogs=>"2"}, {:first=>"Lucy", :last=>"Laweless", :cats=>"5"} ],
             [ {:first=>"Miles", :last=>"O'Brian", :fish=>"21"}, {:first=>"Nancy", :last=>"Homes", :dogs=>"2", :birds=>"1"} ]
           ]
+```
 
 #### Example 1c: How SmarterCSV processes CSV-files as chunks, and passes arrays of hashes to a given block:
 Please note how the given block is passed the data for each chunk as the parameter (array of hashes),
 and how the `process` method returns the number of chunks when called with a block
 
+```ruby
      > total_chunks = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}}) do |chunk|
          chunk.each do |h|   # you can post-process the data from each row to your heart's content, and also create virtual attributes:
            h[:full_name] = [h[:first],h[:last]].join(' ')  # create a virtual attribute
@@ -96,16 +107,16 @@ and how the `process` method returns the number of chunks when called with a blo
        [{:dogs=>"2", :full_name=>"Dan McAllister"}, {:cats=>"5", :full_name=>"Lucy Laweless"}]
        [{:fish=>"21", :full_name=>"Miles O'Brian"}, {:dogs=>"2", :birds=>"1", :full_name=>"Nancy Homes"}]
         => 2
-
+```
 #### Example 2: Reading a CSV-File in one Chunk, returning one Array of Hashes:
-
+```ruby
     filename = '/tmp/input_file.txt' # TAB delimited file, each row ending with Control-M
     recordsA = SmarterCSV.process(filename, {:col_sep => "\t", :row_sep => "\cM"})  # no block given
 
     => returns an array of hashes
-
+```
 #### Example 3: Populate a MySQL or MongoDB Database with SmarterCSV:
-
+```ruby
     # without using chunks:
     filename = '/tmp/some.csv'
     options = {:key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}}
@@ -116,9 +127,9 @@ and how the `process` method returns the number of chunks when called with a blo
     end
 
      => returns number of chunks / rows we processed
-
+```
 #### Example 4: Populate a MongoDB Database in Chunks of 100 records with SmarterCSV:
-
+```ruby
     # using chunks:
     filename = '/tmp/some.csv'
     options = {:chunk_size => 100, :key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}}
@@ -129,10 +140,10 @@ and how the `process` method returns the number of chunks when called with a blo
     end
 
      => returns number of chunks we processed
-
+```
 
 #### Example 5: Reading a CSV-like File, and Processing it with Resque:
-
+```ruby
     filename = '/tmp/strange_db_dump'   # a file with CRTL-A as col_separator, and with CTRL-B\n as record_separator (hello iTunes!)
     options = {
       :col_sep => "\cA", :row_sep => "\cB\n", :comment_regexp => /^#/,
@@ -142,11 +153,11 @@ and how the `process` method returns the number of chunks when called with a blo
         Resque.enque( ResqueWorkerClass, chunk ) # pass chunks of CSV-data to Resque workers for parallel processing
     end
     => returns number of chunks
-
+```
 #### Example 6: Using Value Converters
 
 NOTE: If you use `key_mappings` and `value_converters`, make sure that the value converters has references the keys based on the final mapped name, not the original name in the CSV file.
-
+```ruby
     $ cat spec/fixtures/with_dates.csv
     first,last,date,price
     Ben,Miller,10/30/1998,$44.50
@@ -179,7 +190,7 @@ NOTE: If you use `key_mappings` and `value_converters`, make sure that the value
       => 44.50
     data[0][:price].class
       => Float
-
+```
 ## Parallel Processing
 [Jack](https://github.com/xjlin0) wrote an interesting article about [Speeding up CSV parsing with parallel processing](http://xjlin0.github.io/tech/2015/05/25/faster-parsing-csv-with-parallel-processing)
 
@@ -204,9 +215,9 @@ The options and the block are optional.
      | :invalid_byte_sequence      |   ''     | what to replace invalid byte sequences with                                          |
      | :force_utf8                 |   false  | force UTF-8 encoding of all lines (including headers) in the CSV file                |
      | :skip_lines                 |   nil    | how many lines to skip before the first line or header line is processed             |
-     | :comment_regexp             |   /^#/   | regular expression which matches comment lines (see NOTE about the CSV header)       |
+     | :comment_regexp             |   nil    | regular expression to ignore comment lines (see NOTE on CSV header), e.g./\A#/       |
      ---------------------------------------------------------------------------------------------------------------------------------
-     | :col_sep                    |   ','    | column separator, can be set to 'auto'                                               |
+     | :col_sep                    |   ','    | column separator, can be set to :auto                                                |
      | :force_simple_split         |   false  | force simple splitting on :col_sep character for non-standard CSV-files.             |
      |                             |          | e.g. when :quote_char is not properly escaped                                        |
      | :row_sep                    | $/ ,"\n" | row separator or record separator , defaults to system's $/ , which defaults to "\n" |
@@ -217,6 +228,7 @@ The options and the block are optional.
      | :headers_in_file            |   true   | Whether or not the file contains headers as the first line.                          |
      |                             |          | Important if the file does not contain headers,                                      |
      |                             |          | otherwise you would lose the first line of data.                                     |
+     | :duplicate_header_suffix    |   nil    | If set, adds numbers to duplicated headers and separates them by the given suffix    |
      | :user_provided_headers      |   nil    | *careful with that axe!*                                                             |
      |                             |          | user provided Array of header strings or symbols, to define                          |
      |                             |          | what headers should be used, overriding any in-file headers.                         |
@@ -258,27 +270,36 @@ And header and data validations will also be supported in 2.x
 #### NOTES about File Encodings:
  * if you have a CSV file which contains unicode characters, you can process it as follows:
 
-
+```ruby
        File.open(filename, "r:bom|utf-8") do |f|
          data = SmarterCSV.process(f);
        end
-
+```
 * if the CSV file with unicode characters is in a remote location, similarly you need to give the encoding as an option to the `open` call:
-
+```ruby
        require 'open-uri'
        file_location = 'http://your.remote.org/sample.csv'
        open(file_location, 'r:utf-8') do |f|   # don't forget to specify the UTF-8 encoding!!
          data = SmarterCSV.process(f)
        end
+```
 
 #### NOTES about CSV Headers:
  * as this method parses CSV files, it is assumed that the first line of any file will contain a valid header
- * the first line with the CSV header may or may not be commented out according to the :comment_regexp
+ * the first line with the header might be commented out, in which case you will need to set `comment_regexp: /\A#/`
+   This is no longer handled automatically since 1.5.0.
  * any occurences of :comment_regexp or :row_sep will be stripped from the first line with the CSV header
  * any of the keys in the header line will be downcased, spaces replaced by underscore, and converted to Ruby symbols before being used as keys in the returned Hashes
  * you can not combine the :user_provided_headers and :key_mapping options
  * if the incorrect number of headers are provided via :user_provided_headers, exception SmarterCSV::HeaderSizeMismatch is raised
 
+#### NOTES on Duplicate Headers:
+ As a corner case, it is possible that a CSV file contains multiple headers with the same name. 
+ * If that happens, by default `smarter_csv` will raise a `DuplicateHeaders` error.
+ * If you set `duplicate_header_suffix` to a non-nil string, it will use it to append numbers 2..n to the duplicate headers. To further disambiguate the headers, you can further use `key_mapping` to assign meaningful names.
+ * If your code will need to process arbitrary CSV files, please set `duplicate_header_suffix`.
+ * Another way to deal with duplicate headers it to use `user_assigned_headers` to ignore any headers in the file.
+
 #### NOTES on Key Mapping:
  * keys in the header line of the file can be re-mapped to a chosen set of symbols, so the resulting Hashes can be better used internally in your application (e.g. when directly creating MongoDB entries with them)
  * if you want to completely delete a key, then map it to nil or to '', they will be automatically deleted from any result Hash
@@ -304,64 +325,27 @@ And header and data validations will also be supported in 2.x
 ## Installation
 
 Add this line to your application's Gemfile:
-
+```ruby
     gem 'smarter_csv'
-
+```
 And then execute:
-
+```ruby
     $ bundle
-
+```
 Or install it yourself as:
-
+```ruby
     $ gem install smarter_csv
-
+```
 ## [ChangeLog](./CHANGELOG.md)
 
 ## Reporting Bugs / Feature Requests
 
 Please [open an Issue on GitHub](https://github.com/tilo/smarter_csv/issues) if you have feedback, new feature requests, or want to report a bug. Thank you!
 
+  * please include a small sample CSV file
+  * please mention your version of SmarterCSV, Ruby, Rails
 
-## Special Thanks
-
-Many thanks to people who have filed issues and sent comments.
-And a special thanks to those who contributed pull requests:
-
- * [Jack 0](https://github.com/xjlin0)
- * [Alejandro](https://github.com/agaviria)
- * [Lucas Camargo de Almeida](https://github.com/lcalmeida)
- * [Raphaël Bleuse](https://github.com/bleuse)
- * [feens](https://github.com/feens)
- * [César Camacho](https://github.com/chanko)
- * [innhyu](https://github.com/innhyu)
- * [Benjamin Thouret](https://github.com/benichu)
- * [Chris Hilton](https://github.com/chrismhilton)
- * [Sean Duckett](http://github.com/sduckett)
- * [Alex Ong](http://github.com/khaong)
- * [Martin Nilsson](http://github.com/MrTin)
- * [Eustáquio Rangel](http://github.com/taq)
- * [Pavel](http://github.com/paxa)
- * [Félix Bellanger](https://github.com/Keeguon)
- * [Graham Wetzler](https://github.com/grahamwetzler)
- * [Marcos G. Zimmermann](https://github.com/marcosgz)
- * [Jordan Running](https://github.com/jrunning)
- * [Dave Sanders](https://github.com/DaveSanders)
- * [Hugo Lepetit](https://github.com/giglemad)
- * [esBeee](https://github.com/esBeee)
- * [Waldyr de Souza](https://github.com/waldyr)
- * [Ben Maher](https://github.com/benmaher)
- * [Wal McConnell](https://github.com/wal)
- * [Jordan Graft](https://github.com/jordangraft)
- * [Michael](https://github.com/polycarpou)
- * [Kevin Coleman](https://github.com/KevinColemanInc)
- * [Tirdad C.](https://github.com/tridadc)
- * [Dave Myron](https://github.com/contentfree)
- * [Ivan Ushakov](https://github.com/IvanUshakov)
- * [Matthieu Paret](https://github.com/mtparet)
- * [Rohit Amarnath](https://github.com/ramarnat)
- * [Joshua Smith](https://github.com/enviable)
- * [Colin Petruno](https://github.com/colinpetruno)
- * [Diego Salido](https://github.com/salidux)
+## [A Special Thanks to all Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
 
 
 ## Contributing

data/Rakefile

@@ -1,26 +1,19 @@
 #!/usr/bin/env rake
 require "bundler/gem_tasks"
-
 require 'rubygems'
 require 'rake'
-
 require 'rspec/core/rake_task'
 
+task :default => :spec
+
 desc "Run RSpec"
 RSpec::Core::RakeTask.new do |t|
-  t.verbose = false
+  # t.verbose = false
 end
 
-desc "Run specs for all test cases"
-task :spec_all do 
-  system "rake spec"
+desc 'Run spec with coverage'
+task :coverage do
+  ENV['COVERAGE'] = 'true'
+  Rake::Task['spec'].execute
+  `open coverage/index.html`
 end
-
-# task :spec_all do
-#   %w[active_record data_mapper mongoid].each do |model_adapter|
-#     puts "MODEL_ADAPTER = #{model_adapter}"
-#     system "rake spec MODEL_ADAPTER=#{model_adapter}"
-#   end
-# end
-
-task :default => :spec