smarter_csv 1.1.0 → 1.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ec1442dad9c0f71dc3264f6df10f5fe1116f9d23
-  data.tar.gz: 2025c6c7c81fc6c94fed0ed7d391eb122a464bd1
+  metadata.gz: 34d6c592bebe9d6b1d8f87f9f59ecf4a7d3b3a9d
+  data.tar.gz: fe722f38c4962a312c4db7e14cb72e735426db82
 SHA512:
-  metadata.gz: 1dd00a098dba973b2f6e0303317bd11cdc527d05ca959dd21ae78f241a640e799b623f0b719da5d146c89856d0e50a557ffa8180725d29c6582e0e10231b091a
-  data.tar.gz: 7ab34fac0386ccef0ad13f1116b5ed0669d4ffe6270f8a4b9e39da8421132c4fac4f1c1764b78ca8ce5e702df29c9819ee1359bf0b18c98b735ea53a3935e4d8
+  metadata.gz: 75f5c56cfdeeef41be34f17bfbec30ae201c41463ad9aa6c7da7b5031c63fa1da27ef5120c245329eec108551fae722a5d156e09ead62c2e6aa34ce8edfe4cd8
+  data.tar.gz: 2d3b83fa5e7f4eada8d7f03df50890441f594b8db6d34df4d2c07164c73beda7b9963d596ce43d2ad74ffddb83efb4bdea8ca83d780d7f3129c258c6d21bdb70

data/README.md

@@ -20,7 +20,7 @@ One `smarter_csv` user wrote:
  * able to ignore "columns" in the input (delete columns)
  * able to eliminate nil or empty fields from the result hashes (default)
 
-NOTE; This Gem is only for importing CSV files - writing of CSV files is not supported.
+NOTE; This Gem is only for importing CSV files - writing of CSV files is not supported at this time.
 
 ### Why?
 
@@ -130,6 +130,8 @@ and how the `process` method returns the number of chunks when called with a blo
 
 #### Example 6: Using Value Converters
 
+NOTE: If you use `key_mappings` and `value_converters`, make sure that the value converters has references the keys based on the final mapped name, not the original name in the CSV file.
+
     $ cat spec/fixtures/with_dates.csv
     first,last,date,price
     Ben,Miller,10/30/1998,$44.50
@@ -163,6 +165,9 @@ and how the `process` method returns the number of chunks when called with a blo
     data[0][:price].class
       => Float
 
+## Parallel Processing
+[Jack](https://github.com/xjlin0) wrote an interesting article about [Speeding up CSV parsing with parallel processing](http://xjlin0.github.io/tech/2015/05/25/faster-parsing-csv-with-parallel-processing/)
+
 ## Documentation
 
 The `process` method reads and processes a "generalized" CSV file and returns the contents either as an Array of Hashes,
@@ -198,6 +203,8 @@ The options and the block are optional.
      | :headers_in_file            |   true   | Whether or not the file contains headers as the first line.                          |
      |                             |          | Important if the file does not contain headers,                                      |
      |                             |          | otherwise you would lose the first line of data.                                     |
+     | :skip_lines                 |   nil    | how many lines to skip before the first line or header line is processed             |
+     | :force_utf8                 |   false  | force UTF-8 encoding of all lines (including headers) in the CSV file                |     
      ---------------------------------------------------------------------------------------------------------------------------------
      | :value_converters           |   nil    | supply a hash of :header => KlassName; the class needs to implement self.convert(val)|
      | :remove_empty_values        |   true   | remove values which have nil or empty strings as values                              |
@@ -270,9 +277,22 @@ Or install it yourself as:
 
     $ gem install smarter_csv
 
+## Upcoming
+
+Planned in the next releases:
+ * programmatic header transformations
+ * CSV command line
 
 ## Changes
 
+#### 1.1.1 (2016-11-26)  
+ * added option to `skip_lines` (thanks to wal)
+ * added option to `force_utf8` encoding (thanks to jordangraft)
+ * bugfix if no headers in input data (thanks to esBeee)
+ * ensure input file is closed (thanks to waldyr)
+ * improved verbose output (thankd to benmaher)
+ * improved documentation
+ 
 #### 1.1.0 (2015-07-26)
  * added feature :value_converters, which allows parsing of dates, money, and other things (thanks to Raphaël Bleuse, Lucas Camargo de Almeida, Alejandro)
  * added error if :headers_in_file is set to false, and no :user_provided_headers are given (thanks to innhyu)
@@ -383,6 +403,7 @@ Please [open an Issue on GitHub](https://github.com/tilo/smarter_csv/issues) if
 Many thanks to people who have filed issues and sent comments.
 And a special thanks to those who contributed pull requests:
 
+ * [Jack 0](https://github.com/xjlin0)
  * [Alejandro](https://github.com/agaviria)
  * [Lucas Camargo de Almeida](https://github.com/lcalmeida)
  * [Raphaël Bleuse](https://github.com/bleuse)
@@ -402,6 +423,11 @@ And a special thanks to those who contributed pull requests:
  * [Jordan Running](https://github.com/jrunning)
  * [Dave Sanders](https://github.com/DaveSanders)
  * [Hugo Lepetit](https://github.com/giglemad)
+ * [esBeee](https://github.com/esBeee)
+ * [Waldyr de Souza](https://github.com/waldyr)
+ * [Ben Maher](https://github.com/benmaher)
+ * [Wal McConnell](https://github.com/wal)
+ * [Jordan Graft](https://github.com/jordangraft)
 
 
 ## Contributing

data/lib/smarter_csv/smarter_csv.rb

@@ -9,14 +9,15 @@ module SmarterCSV
       :remove_empty_values => true, :remove_zero_values => false , :remove_values_matching => nil , :remove_empty_hashes => true , :strip_whitespace => true,
       :convert_values_to_numeric => true, :strip_chars_from_headers => nil , :user_provided_headers => nil , :headers_in_file => true,
       :comment_regexp => /^#/, :chunk_size => nil , :key_mapping_hash => nil , :downcase_header => true, :strings_as_keys => false, :file_encoding => 'utf-8',
-      :remove_unmapped_keys => false, :keep_original_headers => false, :value_converters => nil,
+      :remove_unmapped_keys => false, :keep_original_headers => false, :value_converters => nil, :skip_lines => nil, :force_utf8 => false
     }
     options = default_options.merge(options)
     csv_options = options.select{|k,v| [:col_sep, :row_sep, :quote_char].include?(k)} # options.slice(:col_sep, :row_sep, :quote_char)
     headerA = []
     result = []
     old_row_sep = $/
-    line_count = 0
+    file_line_count = 0
+    csv_line_count = 0
     begin
       f = input.respond_to?(:readline) ? input : File.open(input, "r:#{options[:file_encoding]}")
 
@@ -26,14 +27,25 @@ module SmarterCSV
       end
       $/ = options[:row_sep]
 
+      if options[:skip_lines].to_i > 0
+        options[:skip_lines].to_i.times{f.readline}
+      end
+
       if options[:headers_in_file]        # extract the header line
         # process the header line in the CSV file..
         # the first line of a CSV file contains the header .. it might be commented out, so we need to read it anyhow
         header = f.readline.sub(options[:comment_regexp],'').chomp(options[:row_sep])
-        line_count += 1
+        header = header.encode!('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '') if options[:force_utf8]
+        file_line_count += 1
+        csv_line_count += 1
         header = header.gsub(options[:strip_chars_from_headers], '') if options[:strip_chars_from_headers]
+
         if (header =~ %r{#{options[:quote_char]}}) and (! options[:force_simple_split])
-          file_headerA = CSV.parse( header, csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
+          file_headerA = begin
+            CSV.parse( header, csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
+          rescue CSV::MalformedCSVError => e
+            raise $!, "#{$!} [SmarterCSV: csv line #{csv_line_count}]", $!.backtrace
+          end
         else
           file_headerA =  header.split(options[:col_sep])
         end
@@ -44,11 +56,9 @@ module SmarterCSV
           file_headerA.map!{|x| x.downcase }   if options[:downcase_header]
         end
 
-#        puts "HeaderA: #{file_headerA.join(' , ')}" if options[:verbose]
-
         file_header_size = file_headerA.size
       else
-        raise SmarterCSV::IncorrectOption , "ERROR [smarter_csv]: If :headers_in_file is set to false, you have to provide :user_provided_headers" if ! options.keys.include?(:user_provided_headers)
+        raise SmarterCSV::IncorrectOption , "ERROR [smarter_csv]: If :headers_in_file is set to false, you have to provide :user_provided_headers" if options[:user_provided_headers].nil?
       end
       if options[:user_provided_headers] && options[:user_provided_headers].class == Array && ! options[:user_provided_headers].empty?
         # use user-provided headers
@@ -88,22 +98,30 @@ module SmarterCSV
       # now on to processing all the rest of the lines in the CSV file:
       while ! f.eof?    # we can't use f.readlines() here, because this would read the whole file into memory at once, and eof => true
         line = f.readline  # read one line.. this uses the input_record_separator $/ which we set previously!
-        line_count += 1
-        print "processing line %10d\r" % line_count if options[:verbose]
+        line = line.encode!('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '') if options[:force_utf8]
+        file_line_count += 1
+        csv_line_count += 1
+        print "processing file line %10d, csv line %10d\r" % [file_line_count, csv_line_count] if options[:verbose]
         next  if  line =~ options[:comment_regexp]  # ignore all comment lines if there are any
 
         # cater for the quoted csv data containing the row separator carriage return character
         # in which case the row data will be split across multiple lines (see the sample content in spec/fixtures/carriage_returns_rn.csv)
         # by detecting the existence of an uneven number of quote characters
+        multiline = line.count(options[:quote_char])%2 == 1
         while line.count(options[:quote_char])%2 == 1
-          print "line contains uneven number of quote chars so including content of next line" if options[:verbose]
           line += f.readline
+          file_line_count += 1
         end
+        print "\nline contains uneven number of quote chars so including content through file line %d\n" % file_line_count if options[:verbose] && multiline
 
         line.chomp!    # will use $/ which is set to options[:col_sep]
 
         if (line =~ %r{#{options[:quote_char]}}) and (! options[:force_simple_split])
-          dataA = CSV.parse( line, csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
+          dataA = begin
+            CSV.parse( line, csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
+          rescue CSV::MalformedCSVError => e
+            raise $!, "#{$!} [SmarterCSV: csv line #{csv_line_count}]", $!.backtrace
+          end
         else
           dataA =  line.split(options[:col_sep])
         end
@@ -176,6 +194,10 @@ module SmarterCSV
           end
         end
       end
+
+      # print new line to retain last processing line message
+      print "\n" if options[:verbose]
+
       # last chunk:
       if ! chunk.nil? && chunk.size > 0
         # do something with the chunk
@@ -189,6 +211,7 @@ module SmarterCSV
       end
     ensure
       $/ = old_row_sep   # make sure this stupid global variable is always reset to it's previous value after we're done!
+      f.close
     end
     if block_given?
       return chunk_count  # when we do processing through a block we only care how many chunks we processed
@@ -197,11 +220,6 @@ module SmarterCSV
     end
   end
 
-#  def SmarterCSV.process_csv(*args)
-#    warn "[DEPRECATION] `process_csv` is deprecated.  Please use `process` instead."
-#    SmarterCSV.process(*args)
-#  end
-
   private
   # acts as a road-block to limit processing when iterating over all k/v pairs of a CSV-hash:
 
@@ -241,8 +259,7 @@ module SmarterCSV
     end
     counts["\r"] += 1 if last_char == "\r"
     # find the key/value pair with the largest counter:
-    k,v = counts.max_by{|k,v| v}
+    k,_ = counts.max_by{|_,v| v}
     return k                    # the most frequent one is it
   end
 end
-

data/lib/smarter_csv/version.rb

@@ -1,3 +1,3 @@
 module SmarterCSV
-  VERSION = "1.1.0"
+  VERSION = "1.1.1"
 end

data/spec/fixtures/malformed.csv

@@ -0,0 +1,3 @@
+"name","dob"
+"Arnold Schwarzenegger","1947-07-30"
+"Jeff "the dude" Bridges","1949-12-04"

data/spec/fixtures/malformed_header.csv

@@ -0,0 +1,3 @@
+"name","dob"dob""
+"Arnold Schwarzenegger","1947-07-30"
+"Jeff Bridges","1949-12-04"

data/spec/fixtures/skip_lines.csv

@@ -0,0 +1,8 @@
+Lines
+To
+Skip
+first name,last name,dogs,cats,birds,fish
+Dan,McAllister,2,,,
+Lucy,Laweless,,5,,
+Miles,O'Brian,,,,21
+Nancy,Homes,2,,1,

data/spec/smarter_csv/close_file_spec.rb

@@ -0,0 +1,15 @@
+require 'spec_helper'
+
+fixture_path = 'spec/fixtures'
+
+describe 'be_able_to' do
+  it 'close file after using it' do
+    options = {:col_sep => "\cA", :row_sep => "\cB", :comment_regexp => /^#/, :strings_as_keys => true}
+
+    file = File.new("#{fixture_path}/binary.csv")
+
+    SmarterCSV.process(file, options)
+
+    file.closed?.should == true
+  end
+end

data/spec/smarter_csv/malformed_spec.rb

@@ -0,0 +1,21 @@
+require 'spec_helper'
+
+fixture_path = 'spec/fixtures'
+
+describe 'malformed_csv' do
+  subject { lambda { SmarterCSV.process(csv_path) } }
+
+  context "malformed header" do
+    let(:csv_path) { "#{fixture_path}/malformed_header.csv" }
+    it { should raise_error(CSV::MalformedCSVError) }
+    it { should raise_error(/(Missing or stray quote in line 1|CSV::MalformedCSVError)/) }
+    it { should raise_error(CSV::MalformedCSVError) }
+  end
+
+  context "malformed content" do
+    let(:csv_path) { "#{fixture_path}/malformed.csv" }
+    it { should raise_error(CSV::MalformedCSVError) }
+    it { should raise_error(/(Missing or stray quote in line 1|CSV::MalformedCSVError)/) }
+    it { should raise_error(CSV::MalformedCSVError) }
+  end
+end

data/spec/smarter_csv/skip_lines_spec.rb

@@ -0,0 +1,29 @@
+require 'spec_helper'
+
+fixture_path = 'spec/fixtures'
+
+describe 'be_able_to' do
+  it 'loads_csv_file_skipping_lines' do
+    options = {skip_lines: 3}
+    data = SmarterCSV.process("#{fixture_path}/skip_lines.csv", options)
+    data.size.should == 4
+
+    data.each do |item|
+      item.keys.each do |key|
+        [:first_name,:last_name,:dogs,:cats,:birds,:fish].should include(key)
+      end
+    end
+  end
+
+  it 'loads_csv_with_user_defined_headers' do
+    options = {:skip_lines => 3, :headers_in_file => true, :user_provided_headers => [:a,:b,:c,:d,:e,:f]}
+    data = SmarterCSV.process("#{fixture_path}/skip_lines.csv", options)
+    data.size.should == 4
+  
+    data.each do |item|
+      item.keys.each do |key|
+        [:a,:b,:c,:d,:e,:f].should include( key )
+      end
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: smarter_csv
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.1.1
 platform: ruby
 authors:
 - |
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-07-27 00:00:00.000000000 Z
+date: 2016-11-26 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -59,18 +59,22 @@ files:
 - spec/fixtures/line_endings_r.csv
 - spec/fixtures/line_endings_rn.csv
 - spec/fixtures/lots_of_columns.csv
+- spec/fixtures/malformed.csv
+- spec/fixtures/malformed_header.csv
 - spec/fixtures/money.csv
 - spec/fixtures/no_header.csv
 - spec/fixtures/numeric.csv
 - spec/fixtures/pets.csv
 - spec/fixtures/quoted.csv
 - spec/fixtures/separator.csv
+- spec/fixtures/skip_lines.csv
 - spec/fixtures/with_dashes.csv
 - spec/fixtures/with_dates.csv
 - spec/smarter_csv/binary_file2_spec.rb
 - spec/smarter_csv/binary_file_spec.rb
 - spec/smarter_csv/carriage_return_spec.rb
 - spec/smarter_csv/chunked_reading_spec.rb
+- spec/smarter_csv/close_file_spec.rb
 - spec/smarter_csv/column_separator_spec.rb
 - spec/smarter_csv/convert_values_to_numeric_spec.rb
 - spec/smarter_csv/header_transformation_spec.rb
@@ -78,6 +82,7 @@ files:
 - spec/smarter_csv/key_mapping_spec.rb
 - spec/smarter_csv/line_ending_spec.rb
 - spec/smarter_csv/load_basic_spec.rb
+- spec/smarter_csv/malformed_spec.rb
 - spec/smarter_csv/no_header_spec.rb
 - spec/smarter_csv/not_downcase_header_spec.rb
 - spec/smarter_csv/quoted_spec.rb
@@ -86,6 +91,7 @@ files:
 - spec/smarter_csv/remove_not_mapped_keys_spec.rb
 - spec/smarter_csv/remove_values_matching_spec.rb
 - spec/smarter_csv/remove_zero_values_spec.rb
+- spec/smarter_csv/skip_lines_spec.rb
 - spec/smarter_csv/strings_as_keys_spec.rb
 - spec/smarter_csv/strip_chars_from_headers_spec.rb
 - spec/smarter_csv/value_converters_spec.rb
@@ -132,18 +138,22 @@ test_files:
 - spec/fixtures/line_endings_r.csv
 - spec/fixtures/line_endings_rn.csv
 - spec/fixtures/lots_of_columns.csv
+- spec/fixtures/malformed.csv
+- spec/fixtures/malformed_header.csv
 - spec/fixtures/money.csv
 - spec/fixtures/no_header.csv
 - spec/fixtures/numeric.csv
 - spec/fixtures/pets.csv
 - spec/fixtures/quoted.csv
 - spec/fixtures/separator.csv
+- spec/fixtures/skip_lines.csv
 - spec/fixtures/with_dashes.csv
 - spec/fixtures/with_dates.csv
 - spec/smarter_csv/binary_file2_spec.rb
 - spec/smarter_csv/binary_file_spec.rb
 - spec/smarter_csv/carriage_return_spec.rb
 - spec/smarter_csv/chunked_reading_spec.rb
+- spec/smarter_csv/close_file_spec.rb
 - spec/smarter_csv/column_separator_spec.rb
 - spec/smarter_csv/convert_values_to_numeric_spec.rb
 - spec/smarter_csv/header_transformation_spec.rb
@@ -151,6 +161,7 @@ test_files:
 - spec/smarter_csv/key_mapping_spec.rb
 - spec/smarter_csv/line_ending_spec.rb
 - spec/smarter_csv/load_basic_spec.rb
+- spec/smarter_csv/malformed_spec.rb
 - spec/smarter_csv/no_header_spec.rb
 - spec/smarter_csv/not_downcase_header_spec.rb
 - spec/smarter_csv/quoted_spec.rb
@@ -159,6 +170,7 @@ test_files:
 - spec/smarter_csv/remove_not_mapped_keys_spec.rb
 - spec/smarter_csv/remove_values_matching_spec.rb
 - spec/smarter_csv/remove_zero_values_spec.rb
+- spec/smarter_csv/skip_lines_spec.rb
 - spec/smarter_csv/strings_as_keys_spec.rb
 - spec/smarter_csv/strip_chars_from_headers_spec.rb
 - spec/smarter_csv/value_converters_spec.rb