smarter_csv 1.5.1 → 1.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 352cf76ac0cd6b2eb4a1cac9e5056aa6e92a8a61b627d7c922e063dcf82ad675
-  data.tar.gz: 0c6e3ab1eaee02a9361fe0b418191244d81bc558dbcd10ee1d2c5f15390d91b6
+  metadata.gz: 9be5f053e15e157d7d28555b4de894d2761d5918203da45f5fc4e6c5adcc2a3f
+  data.tar.gz: a47394f3d1f985960a64abf1a43ce6ebf9b8217af2c01a0c5f053af8c77c09ae
 SHA512:
-  metadata.gz: 3763cd8e493e7da6560e8ce9adc58bd411f745f5af119c97d70c02667a524ccb1055b5c640ef795c3cb25b79fa5e17800018da6e76f9d358afa1c7a3513caae3
-  data.tar.gz: '039183fdece20e80007f3f0d3e395fac8d273df6c21928a35b685ade5915503b3c55918fc7015ead0a7d768a545bb76bcfb1087006af6118dc3d22df83e68ddb'
+  metadata.gz: f27113af8a5771d89ac5c8783f1f69645c24bb576dafd97de17b0d8db8fff74dc396b42450802f9332c9f8b32a02ee18dabbc5dbc2daa91c75f957a678e99099
+  data.tar.gz: 5b2c2f3cbfc17b43c030c4c4c261962818bfbb2ce1a0ecd88f394682b75b75a64a2d8b6afcc0e4b99b97d39ef4b645de143cbb5f595e7aa9d661e66b1a53e98f

data/CHANGELOG.md

@@ -1,7 +1,17 @@
 
 # SmarterCSV 1.x Change Log
 
-## 1.5.1 (2022-04-26)
+## 1.6.1 (2022-05-06)
+  * unused keys in `key_mapping` generate a warning, no longer raise an exception
+
+## 1.6.0 (2022-05-03)
+  * completely rewrote line parser
+  * added methods `SmarterCSV.raw_headers` and `SmarterCSV.headers` to allow easy examination of how the headers are processed.
+
+## 1.5.2 (2022-04-29)
+  * added missing keys to the SmarterCSV::KeyMappingError exception message #189 (thanks to John Dell)
+  
+## 1.5.1 (2022-04-27)
   * added raising of `KeyMappingError` if `key_mapping` refers to a non-existent key
   * added option `duplicate_header_suffix` (thanks to Skye Shaw)
     When given a non-nil string, it uses the suffix to append numbering 2..n to duplicate headers.

data/CONTRIBUTORS.md

@@ -44,3 +44,4 @@ A Big Thank you to everyone who filed issues, sent comments, and who contributed
  * [Nicolas Guillemain](https://github.com/Viiruus)
  * [Sp6](https://github.com/sp6)
  * [Joel Fouse](https://github.com/jfouse)
+ * [John Dell](https://github.com/spovich)

data/README.md

@@ -16,10 +16,12 @@
 
 # SmarterCSV
 
-[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.org/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
+[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.com/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
 
 #### SmarterCSV 1.x
 
+`smarter_csv` is now 10 years old, and still kicking! 🎉🎉🎉
+
 `smarter_csv` is a Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, suitable for direct processing with Mongoid or ActiveRecord,
 and parallel processing with Resque or Sidekiq.
 
@@ -42,11 +44,13 @@ NOTE; This Gem is only for importing CSV files - writing of CSV files is not sup
 
 ### Why?
 
-Ruby's CSV library's API is pretty old, and it's processing of CSV-files returning Arrays of Arrays feels 'very close to the metal'. The output is not easy to use - especially not if you want to create database records from it. Another shortcoming is that Ruby's CSV library does not have good support for huge CSV-files, e.g. there is no support for 'chunking' and/or parallel processing of the CSV-content (e.g. with Resque or Sidekiq),
+Ruby's CSV library's API is pretty old, and it's processing of CSV-files returning Arrays of Arrays feels 'very close to the metal'. The output is not easy to use - especially not if you want to create database records or Sidekiq jobs with it. Another shortcoming is that Ruby's CSV library does not have good support for huge CSV-files, e.g. there is no support for 'chunking' and/or parallel processing of the CSV-content (e.g. with Sidekiq).
+
+As the existing CSV libraries didn't fit my needs, I was writing my own CSV processing - specifically for use in connection with Rails ORMs like Mongoid, MongoMapper and ActiveRecord. In those ORMs you can easily pass a hash with attribute/value pairs to the create() method. The lower-level Mongo driver and Moped also accept larger arrays of such hashes to create a larger amount of records quickly with just one call. The same patterns are used when you pass data to Sidekiq jobs.
 
-As the existing CSV libraries didn't fit my needs, I was writing my own CSV processing - specifically for use in connection with Rails ORMs like Mongoid, MongoMapper or ActiveRecord. In those ORMs you can easily pass a hash with attribute/value pairs to the create() method. The lower-level Mongo driver and Moped also accept larger arrays of such hashes to create a larger amount of records quickly with just one call.
+For processing large CSV files it is essential to process them in chunks, so the memory impact is minimized.
 
-### Examples
+### How?
 
 The two main choices you have in terms of how to call `SmarterCSV.process` are:
  * calling `process` with or without a block

data/lib/smarter_csv/smarter_csv.rb

@@ -6,6 +6,7 @@ module SmarterCSV
   class MissingHeaders < SmarterCSVException; end
   class NoColSepDetected < SmarterCSVException; end
   class KeyMappingError < SmarterCSVException; end
+  class MalformedCSVError < SmarterCSVException; end
 
   # first parameter: filename or input object which responds to readline method
   def SmarterCSV.process(input, options={}, &block)
@@ -18,24 +19,24 @@ module SmarterCSV
     @csv_line_count = 0
     has_rails = !! defined?(Rails)
     begin
-      f = input.respond_to?(:readline) ? input : File.open(input, "r:#{options[:file_encoding]}")
+      fh = input.respond_to?(:readline) ? input : File.open(input, "r:#{options[:file_encoding]}")
 
       # auto-detect the row separator
-      options[:row_sep] = SmarterCSV.guess_line_ending(f, options) if options[:row_sep].to_sym == :auto
+      options[:row_sep] = SmarterCSV.guess_line_ending(fh, options) if options[:row_sep].to_sym == :auto
       # attempt to auto-detect column separator
-      options[:col_sep] = guess_column_separator(f, options) if options[:col_sep].to_sym == :auto
-      # preserve options, in case we need to call the CSV class
-      csv_options = options.select{|k,v| [:col_sep, :row_sep, :quote_char].include?(k)} # options.slice(:col_sep, :row_sep, :quote_char)
-      csv_options.delete(:row_sep) if [nil, :auto].include?( options[:row_sep].to_sym )
-      csv_options.delete(:col_sep) if [nil, :auto].include?( options[:col_sep].to_sym )
+      options[:col_sep] = guess_column_separator(fh, options) if options[:col_sep].to_sym == :auto
 
-      if (options[:force_utf8] || options[:file_encoding] =~ /utf-8/i) && ( f.respond_to?(:external_encoding) && f.external_encoding != Encoding.find('UTF-8') || f.respond_to?(:encoding) && f.encoding != Encoding.find('UTF-8') )
+      if (options[:force_utf8] || options[:file_encoding] =~ /utf-8/i) && ( fh.respond_to?(:external_encoding) && fh.external_encoding != Encoding.find('UTF-8') || fh.respond_to?(:encoding) && fh.encoding != Encoding.find('UTF-8') )
         puts 'WARNING: you are trying to process UTF-8 input, but did not open the input with "b:utf-8" option. See README file "NOTES about File Encodings".'
       end
 
-      options[:skip_lines].to_i.times{f.readline(options[:row_sep])} if options[:skip_lines].to_i > 0
+      if options[:skip_lines].to_i > 0
+        options[:skip_lines].to_i.times do
+          readline_with_counts(fh, options)
+        end
+      end
 
-      headerA, header_size = process_headers(f, options, csv_options)
+      headerA, header_size = process_headers(fh, options)
 
       # in case we use chunking.. we'll need to set it up..
       if ! options[:chunk_size].nil? && options[:chunk_size].to_i > 0
@@ -48,10 +49,8 @@ module SmarterCSV
       end
 
       # now on to processing all the rest of the lines in the CSV file:
-      while ! f.eof?    # we can't use f.readlines() here, because this would read the whole file into memory at once, and eof => true
-        line = f.readline(options[:row_sep])  # read one line
-        @file_line_count += 1
-        @csv_line_count += 1
+      while ! fh.eof?    # we can't use fh.readlines() here, because this would read the whole file into memory at once, and eof => true
+        line = readline_with_counts(fh, options)
 
         # replace invalid byte sequence in UTF-8 with question mark to avoid errors
         line = line.force_encoding('utf-8').encode('utf-8', invalid: :replace, undef: :replace, replace: options[:invalid_byte_sequence]) if options[:force_utf8] || options[:file_encoding] !~ /utf-8/i
@@ -63,9 +62,10 @@ module SmarterCSV
         # cater for the quoted csv data containing the row separator carriage return character
         # in which case the row data will be split across multiple lines (see the sample content in spec/fixtures/carriage_returns_rn.csv)
         # by detecting the existence of an uneven number of quote characters
+
         multiline = line.count(options[:quote_char])%2 == 1 # should handle quote_char nil
         while line.count(options[:quote_char])%2 == 1 # should handle quote_char nil
-          next_line = f.readline(options[:row_sep])
+          next_line = fh.readline(options[:row_sep])
           next_line = next_line.force_encoding('utf-8').encode('utf-8', invalid: :replace, undef: :replace, replace: options[:invalid_byte_sequence]) if options[:force_utf8] || options[:file_encoding] !~ /utf-8/i
           line += next_line
           @file_line_count += 1
@@ -74,16 +74,8 @@ module SmarterCSV
 
         line.chomp!(options[:row_sep])
 
-        if (line =~ %r{#{options[:quote_char]}}) and (! options[:force_simple_split])
-          dataA = begin
-            CSV.parse( line, **csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
-          rescue CSV::MalformedCSVError => e
-            raise $!, "#{$!} [SmarterCSV: csv line #{@csv_line_count}]", $!.backtrace
-          end
-        else
-          dataA = line.split(options[:col_sep], header_size)
-        end
-        dataA.map!{|x| x.sub(/(#{options[:col_sep]})+\z/, '')} # remove any unwanted trailing col_sep characters at the end
+        dataA, data_size = parse(line, options, header_size)
+
         dataA.map!{|x| x.strip} if options[:strip_whitespace]
 
         # if all values are blank, then ignore this line
@@ -138,7 +130,7 @@ module SmarterCSV
         if use_chunks
           chunk << hash  # append temp result to chunk
 
-          if chunk.size >= chunk_size || f.eof?   # if chunk if full, or EOF reached
+          if chunk.size >= chunk_size || fh.eof?   # if chunk if full, or EOF reached
             # do something with the chunk
             if block_given?
               yield chunk  # do something with the hashes in the chunk in the block
@@ -179,7 +171,7 @@ module SmarterCSV
         chunk = []  # initialize for next chunk of data
       end
     ensure
-      f.close if f.respond_to?(:close)
+      fh.close if fh.respond_to?(:close)
     end
     if block_given?
       return chunk_count  # when we do processing through a block we only care how many chunks we processed
@@ -224,6 +216,62 @@ module SmarterCSV
     }
   end
 
+  def self.readline_with_counts(filehandle, options)
+    line  = filehandle.readline(options[:row_sep])
+    @file_line_count += 1
+    @csv_line_count += 1
+    line
+  end
+
+  # parses a single line: either a CSV header and body line
+  # - quoting rules compared to RFC-4180 are somewhat relaxed
+  # - we are not assuming that quotes inside a fields need to be doubled
+  # - we are not assuming that all fields need to be quoted (0 is even)
+  # - works with multi-char col_sep
+  # - if header_size is given, only up to header_size fields are parsed
+  #
+  # We use header_size for parsing the body lines to make sure we always match the number of headers
+  # in case there are trailing col_sep characters in line
+  #
+  # Our convention is that empty fields are returned as empty strings, not as nil.
+  #
+  def self.parse(line, options, header_size = nil)
+    return [] if line.nil?
+
+    col_sep = options[:col_sep]
+    quote = options[:quote_char]
+    quote_count = 0
+    elements = []
+    start = 0
+    i = 0
+
+    while i < line.size do
+      if line[i...i+col_sep.size] == col_sep && quote_count.even?
+        break if !header_size.nil? && elements.size >= header_size
+
+        elements << cleanup_quotes(line[start...i], quote)
+        i += col_sep.size
+        start = i
+      else
+        quote_count += 1 if line[i] == quote
+        i += 1
+      end
+    end
+    elements << cleanup_quotes(line[start..-1], quote) if header_size.nil? || elements.size < header_size
+    [elements, elements.size]
+  end
+
+  def self.cleanup_quotes(field, quote)
+    return field if field.nil? || field !~ /#{quote}/
+
+    if field.start_with?(quote) && field.end_with?(quote)
+      field.delete_prefix!(quote)
+      field.delete_suffix!(quote)
+    end
+    field.gsub!("#{quote}#{quote}", quote)
+    field
+  end
+
   def self.blank?(value)
     case value
     when Array
@@ -310,13 +358,22 @@ module SmarterCSV
     return k                    # the most frequent one is it
   end
 
-  def self.process_headers(filehandle, options, csv_options)
+  def self.raw_hearder
+    @raw_header
+  end
+
+  def self.headers
+    @headers
+  end
+
+  def self.process_headers(filehandle, options)
+    @raw_header = nil
+    @headers = nil
     if options[:headers_in_file]        # extract the header line
       # process the header line in the CSV file..
       # the first line of a CSV file contains the header .. it might be commented out, so we need to read it anyhow
-      header = filehandle.readline(options[:row_sep])
-      @file_line_count += 1
-      @csv_line_count += 1
+      header = readline_with_counts(filehandle, options)
+      @raw_header = header
 
       header = header.force_encoding('utf-8').encode('utf-8', invalid: :replace, undef: :replace, replace: options[:invalid_byte_sequence]) if options[:force_utf8] || options[:file_encoding] !~ /utf-8/i
       header = header.sub(options[:comment_regexp],'') if options[:comment_regexp]
@@ -324,16 +381,7 @@ module SmarterCSV
 
       header = header.gsub(options[:strip_chars_from_headers], '') if options[:strip_chars_from_headers]
 
-      if (header =~ %r{#{options[:quote_char]}}) and (! options[:force_simple_split])
-        file_headerA = begin
-          CSV.parse( header, **csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
-        rescue CSV::MalformedCSVError => e
-          raise $!, "#{$!} [SmarterCSV: csv line #{@csv_line_count}]", $!.backtrace
-        end
-      else
-        file_headerA =  header.split(options[:col_sep])
-      end
-      file_header_size = file_headerA.size # before mapping, which could delete keys
+      file_headerA, file_header_size = parse(header, options)
 
       file_headerA.map!{|x| x.gsub(%r/#{options[:quote_char]}/,'') }
       file_headerA.map!{|x| x.strip}  if options[:strip_whitespace]
@@ -371,7 +419,8 @@ module SmarterCSV
       #   if you want to completely delete a key, then map it to nil or to ''
       if ! key_mappingH.nil? && key_mappingH.class == Hash && key_mappingH.keys.size > 0
         # we can't map keys that are not there
-        raise SmarterCSV::KeyMappingError unless (key_mappingH.keys - headerA).empty?
+        missing_keys = key_mappingH.keys - headerA
+        puts "WARNING: missing header(s): #{missing_keys.join(",")}" unless missing_keys.empty?
 
         headerA.map!{|x| key_mappingH.has_key?(x) ? (key_mappingH[x].nil? ? nil : key_mappingH[x]) : (options[:remove_unmapped_keys] ? nil : x)}
       end
@@ -392,6 +441,7 @@ module SmarterCSV
       raise SmarterCSV::MissingHeaders , "ERROR: missing headers: #{missing_headers.join(',')}" unless missing_headers.empty?
     end
 
+    @headers = headerA
     [headerA, header_size]
   end
 

data/lib/smarter_csv/version.rb

@@ -1,3 +1,3 @@
 module SmarterCSV
-  VERSION = "1.5.1"
+  VERSION = "1.6.1"
 end

data/smarter_csv.gemspec

@@ -16,9 +16,9 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
-  spec.requirements  = ['csv'] # for CSV.parse() only needed in case we have quoted fields
   spec.add_development_dependency "rspec"
   spec.add_development_dependency "simplecov"
+  spec.add_development_dependency "awesome_print"
   #  spec.add_development_dependency "guard-rspec"
 
   spec.metadata["homepage_uri"] = spec.homepage

data/spec/smarter_csv/duplicate_headers_spec.rb

@@ -17,12 +17,12 @@ describe 'duplicate headers' do
       }.to raise_exception(SmarterCSV::DuplicateHeaders)
     end
 
-    it 'raises error on missing mapped headers' do
+    it 'does not raise error on missing mapped headers and includes missing headers in message' do
+      # the mapping is right, but the underlying csv file is bad
+      options = {:key_mapping => {:email => :a, :firstname => :b, :lastname => :c, :manager_email => :d, :age => :e} }
       expect {
-        # the mapping is right, but the underlying csv file is bad
-        options = {:key_mapping => {:email => :a, :firstname => :b, :lastname => :c, :manager_email => :d, :age => :e} }
         SmarterCSV.process("#{fixture_path}/duplicate_headers.csv", options)
-      }.to raise_exception(SmarterCSV::KeyMappingError)
+      }.not_to raise_exception(SmarterCSV::KeyMappingError)
     end
   end
 

data/spec/smarter_csv/invalid_headers_spec.rb

@@ -28,11 +28,11 @@ describe 'test exceptions for invalid headers' do
     }.to raise_exception(SmarterCSV::MissingHeaders)
   end
 
-  it 'raises error on missing mapped headers' do
+  it 'does not raise error on missing mapped headers and includes missing headers in message' do
+    # :age does not exist in the CSV header
+    options = {:key_mapping => {:email => :a, :firstname => :b, :lastname => :c, :manager_email => :d, :age => :e} }
     expect {
-      # :age does not exist in the CSV header
-      options = {:key_mapping => {:email => :a, :firstname => :b, :lastname => :c, :manager_email => :d, :age => :e} }
       SmarterCSV.process("#{fixture_path}/user_import.csv", options)
-    }.to raise_exception(SmarterCSV::KeyMappingError)
+    }.not_to raise_exception(SmarterCSV::KeyMappingError)
   end
 end

data/spec/smarter_csv/malformed_spec.rb

@@ -2,16 +2,24 @@ require 'spec_helper'
 
 fixture_path = 'spec/fixtures'
 
-describe 'malformed_csv' do
-  subject { lambda { SmarterCSV.process(csv_path) } }
-
-  context "malformed header" do
+# according to RFC-4180 quotes inside of "words" shouldbe doubled, but our parser is robust against that.
+describe 'malformed CSV quotes' do
+  context "malformed quotes in header" do
     let(:csv_path) { "#{fixture_path}/malformed_header.csv" }
-    it { should raise_error(CSV::MalformedCSVError) }
+    it 'should be resilient against single quotes' do
+      data = SmarterCSV.process(csv_path)
+      expect(data[0]).to eq({:name=>"Arnold Schwarzenegger", :dobdob=>"1947-07-30"})
+      expect(data[1]).to eq({:name=>"Jeff Bridges", :dobdob=>"1949-12-04"})
+    end
   end
 
-  context "malformed content" do
+  context "malformed quotes in content" do
     let(:csv_path) { "#{fixture_path}/malformed.csv" }
-    it { should raise_error(CSV::MalformedCSVError) }
+
+    it 'should be resilient against single quotes' do
+      data = SmarterCSV.process(csv_path)
+      expect(data[0]).to eq({:name=>"Arnold Schwarzenegger", :dob=>"1947-07-30"})
+      expect(data[1]).to eq({:name=>"Jeff \"the dude\" Bridges", :dob=>"1949-12-04"})
+    end
   end
 end

data/spec/smarter_csv/parse/column_separator_spec.rb

@@ -0,0 +1,61 @@
+require 'spec_helper'
+
+describe 'parse with col_sep' do
+  let(:options) { {quote_char: '"'} }
+
+  it 'parses with comma' do
+    line = "a,b,,d"
+    options.merge!({col_sep: ","})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', '', 'd']
+    expect(array_size).to eq 4
+  end
+
+  it 'parses trailing commas' do
+    line = "a,b,c,,"
+    options.merge!({col_sep: ","})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', 'c', '', '']
+    expect(array_size).to eq 5
+  end
+
+  it 'parses with space' do
+    line = "a b  d"
+    options.merge!({col_sep: " "})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', '', 'd']
+    expect(array_size).to eq 4
+  end
+
+  it 'parses with tab' do
+    line = "a\tb\t\td"
+    options.merge!({col_sep: "\t"})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', '', 'd']
+    expect(array_size).to eq 4
+  end
+
+  it 'parses with multiple space separator' do
+    line = "a b    d"
+    options.merge!({col_sep: "  "})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a b', '', 'd']
+    expect(array_size).to eq 3
+  end
+
+  it 'parses with multiple char separator' do
+    line = '<=><=>A<=>B<=>C'
+    options.merge!({col_sep: "<=>"})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ["", "", "A", "B", "C"]
+    expect(array_size).to eq 5
+  end
+
+  it 'parses trailing multiple char separator' do
+    line = '<=><=>A<=>B<=>C<=><=>'
+    options.merge!({col_sep: "<=>"})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ["", "", "A", "B", "C", "", ""]
+    expect(array_size).to eq 7
+  end
+end

data/spec/smarter_csv/parse/old_csv_library_spec.rb

@@ -0,0 +1,74 @@
+require 'spec_helper'
+
+describe 'old CSV library parsing tests' do
+  let(:options) { {quote_char: '"', col_sep: ","} }
+
+  [ ["\t", ["\t"]],
+    ["foo,\"\"\"\"\"\",baz", ["foo", "\"\"", "baz"]],
+    ["foo,\"\"\"bar\"\"\",baz", ["foo", "\"bar\"", "baz"]],
+    ["\"\"\"\n\",\"\"\"\n\"", ["\"\n", "\"\n"]],
+    ["foo,\"\r\n\",baz", ["foo", "\r\n", "baz"]],
+    ["\"\"", [""]],
+    ["foo,\"\"\"\",baz", ["foo", "\"", "baz"]],
+    ["foo,\"\r.\n\",baz", ["foo", "\r.\n", "baz"]],
+    ["foo,\"\r\",baz", ["foo", "\r", "baz"]],
+    ["foo,\"\",baz", ["foo", "", "baz"]],
+    ["\",\"", [","]],
+    ["foo", ["foo"]],
+    [",,", ['', '', '']],
+    [",", ['', '']],
+    ["foo,\"\n\",baz", ["foo", "\n", "baz"]],
+    ["foo,,baz", ["foo", '', "baz"]],
+    ["\"\"\"\r\",\"\"\"\r\"", ["\"\r", "\"\r"]],
+    ["\",\",\",\"", [",", ","]],
+    ["foo,bar,", ["foo", "bar", '']],
+    [",foo,bar", ['', "foo", "bar"]],
+    ["foo,bar", ["foo", "bar"]],
+    [";", [";"]],
+    ["\t,\t", ["\t", "\t"]],
+    ["foo,\"\r\n\r\",baz", ["foo", "\r\n\r", "baz"]],
+    ["foo,\"\r\n\n\",baz", ["foo", "\r\n\n", "baz"]],
+    ["foo,\"foo,bar\",baz", ["foo", "foo,bar", "baz"]],
+    [";,;", [";", ";"]]
+  ].each do |line, result|
+    it "parses #{line}" do
+      array, array_size = SmarterCSV.send(:parse, line, options)
+      expect(array).to eq result
+    end
+  end
+
+  [ ["foo,\"\"\"\"\"\",baz", ["foo", "\"\"", "baz"]],
+    ["foo,\"\"\"bar\"\"\",baz", ["foo", "\"bar\"", "baz"]],
+    ["foo,\"\r\n\",baz", ["foo", "\r\n", "baz"]],
+    ["\"\"", [""]],
+    ["foo,\"\"\"\",baz", ["foo", "\"", "baz"]],
+    ["foo,\"\r.\n\",baz", ["foo", "\r.\n", "baz"]],
+    ["foo,\"\r\",baz", ["foo", "\r", "baz"]],
+    ["foo,\"\",baz", ["foo", "", "baz"]],
+    ["foo", ["foo"]],
+    [",,", ['', '', '']],
+    [",", ['', '']],
+    ["foo,\"\n\",baz", ["foo", "\n", "baz"]],
+    ["foo,,baz", ["foo", '', "baz"]],
+    ["foo,bar", ["foo", "bar"]],
+    ["foo,\"\r\n\n\",baz", ["foo", "\r\n\n", "baz"]],
+    ["foo,\"foo,bar\",baz", ["foo", "foo,bar", "baz"]]
+  ].each do |line, result|
+    it "parses #{line}" do
+      array, array_size = SmarterCSV.send(:parse, line, options)
+      expect(array).to eq result
+    end
+  end
+
+  it 'mixed quotes' do
+    line = %Q{Ten Thousand,10000, 2710 ,,"10,000","It's ""10 Grand"", baby",10K}
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ["Ten Thousand", "10000", " 2710 ", "", "10,000", "It's \"10 Grand\", baby", "10K"]
+  end
+
+  it 'single quotes in fields' do
+    line = 'Indoor Chrome,49.2"" L x 49.2"" W x 20.5"" H,Chrome,"Crystal,Metal,Wood",23.12'
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['Indoor Chrome', '49.2" L x 49.2" W x 20.5" H', 'Chrome', 'Crystal,Metal,Wood', '23.12']
+  end
+end

data/spec/smarter_csv/parse/rfc4180_and_more_spec.rb

@@ -0,0 +1,170 @@
+require 'spec_helper'
+
+fixture_path = 'spec/fixtures'
+
+describe 'fulfills RFC-4180 and more' do
+  let(:options) { {col_sep: ',', row_sep: $INPUT_RECORD_SEPARATOR, quote_char: '"' } }
+
+  context 'parses simple CSV' do
+    context 'RFC-4180' do
+      it 'separating on col_sep' do
+        line = 'aaa,bbb,ccc'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [%w[aaa bbb ccc], 3]
+      end
+
+      it 'preserves whitespace' do
+        line = ' aaa , bbb , ccc '
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          [' aaa ', ' bbb ', ' ccc '], 3
+        ]
+      end
+    end
+
+    context 'extending RFC-4180' do
+      it 'with extra col_sep' do
+        line = 'aaa,bbb,ccc,'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', ''], 4
+        ]
+      end
+
+      it 'with extra col_sep with given header_size' do
+        line = 'aaa,bbb,ccc,'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+
+      it 'with multiple extra col_sep' do
+        line = 'aaa,bbb,ccc,,,'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple extra col_sep' do
+        line = 'aaa,bbb,ccc,,,'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+
+      it 'with multiple complex col_sep' do
+        line = 'aaa<=>bbb<=>ccc<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}))).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple complex col_sep with given header_size' do
+        line = 'aaa<=>bbb<=>ccc<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}), 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+    end
+  end
+
+  context 'parses quoted CSV' do
+    context 'RFC-4180' do
+      it 'separating on col_sep' do
+        line = '"aaa","bbb","ccc"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [%w[aaa bbb ccc], 3]
+      end
+
+      it 'parses corner case correctly' do
+        line = '"Board 4""","$17.40","10000003427"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['Board 4"', '$17.40', '10000003427'], 3
+        ]
+      end
+
+      it 'quoted parts can contain spaces' do
+        line = '" aaa1 aaa2 "," bbb1 bbb2 "," ccc1 ccc2 "'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          [' aaa1 aaa2 ', ' bbb1 bbb2 ', ' ccc1 ccc2 '], 3
+        ]
+      end
+
+      it 'quoted parts can contain row_sep' do
+        line = '"aaa1, aaa2","bbb1, bbb2","ccc1, ccc2"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa1, aaa2', 'bbb1, bbb2', 'ccc1, ccc2'], 3
+        ]
+      end
+
+      it 'quoted parts can contain row_sep' do
+        line = '"aaa1, ""aaa2"", aaa3","""bbb1"", bbb2","ccc1, ""ccc2"""'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa1, "aaa2", aaa3', '"bbb1", bbb2', 'ccc1, "ccc2"'], 3
+        ]
+      end
+
+      it 'some fields are quoted' do
+        line = '1,"board 4""",12.95'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['1', 'board 4"', '12.95'], 3
+        ]
+      end
+
+      it 'separating on col_sep' do
+        line = '"some","thing","""completely"" different"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['some', 'thing', '"completely" different'], 3
+        ]
+      end
+    end
+
+    context 'extending RFC-4180' do
+      it 'with extra col_sep, without given header_size' do
+        line = '"aaa","bbb","ccc",'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', ''], 4
+        ]
+      end
+
+      it 'with extra col_sep, with given header_size' do
+        line = '"aaa","bbb","ccc",'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [%w[aaa bbb ccc], 3]
+      end
+
+      it 'with multiple extra col_sep, without given header_size' do
+        line = '"aaa","bbb","ccc",,,'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple extra col_sep, with given header_size' do
+        line = '"aaa","bbb","ccc",,,'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+
+      it 'with multiple complex extra col_sep, without given header_size' do
+        line = '"aaa"<=>"bbb"<=>"ccc"<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}))).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple complex extra col_sep, with given header_size' do
+        line = '"aaa"<=>"bbb"<=>"ccc"<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}), 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+    end
+  end
+
+  # relaxed parsing compared to RFC-4180
+  context 'liberal_parsing' do
+    it 'parses corner case correctly' do
+      line = 'is,this "three, or four",fields'
+      expect( SmarterCSV.send(:parse, line, options)).to eq [
+        ['is', 'this "three, or four"', 'fields'], 3
+      ]
+    end
+  end
+end

data/spec/smarter_csv/quoted_spec.rb

@@ -3,7 +3,6 @@ require 'spec_helper'
 fixture_path = 'spec/fixtures'
 
 describe 'loading file with quoted fields' do
-
   it 'leaving the quotes in the data' do
     options = {}
     data = SmarterCSV.process("#{fixture_path}/quoted.csv", options)
@@ -12,6 +11,7 @@ describe 'loading file with quoted fields' do
     data[1][:description].should be_nil
     data[2][:model].should eq 'Venture "Extended Edition, Very Large"'
     data[2][:description].should be_nil
+    data[3][:description].should eq 'MUST SELL! air, moon roof, loaded'
     data.each do |h|
       h[:year].class.should eq Fixnum
       h[:make].should_not be_nil
@@ -20,17 +20,21 @@ describe 'loading file with quoted fields' do
     end
   end
 
-
+  # quotes inside quoted fields need to be escaped by another double-quote
   it 'removes quotes around quoted fields, but not inside data' do
     options = {}
     data = SmarterCSV.process("#{fixture_path}/quote_char.csv", options)
 
     data.length.should eq 6
+    data[0][:first_name].should eq "\"John"
+    data[0][:last_name].should eq "Cooke\""
     data[1][:first_name].should eq "Jam\ne\nson\""
     data[2][:first_name].should eq "\"Jean"
+    data[4][:first_name].should eq "Bo\"bbie"
+    data[5][:first_name].should eq 'Mica'
+    data[5][:last_name].should eq 'Copeland'
   end
 
-
   # NOTE: quotes inside headers need to be escaped by doubling them
   #       e.g. 'correct ""EXAMPLE""'
   #       this escaping is illegal: 'incorrect \"EXAMPLE\"' <-- this caused CSV parsing error
@@ -43,6 +47,6 @@ describe 'loading file with quoted fields' do
     data.length.should eq 3
     data.first.keys[2].should eq :isbn
     data.first.keys[3].should eq :discounted_price
+    data[1][:author].should eq 'Timothy "The Parser" Campbell'
   end
-
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: smarter_csv
 version: !ruby/object:Gem::Version
-  version: 1.5.1
+  version: 1.6.1
 platform: ruby
 authors:
 - Tilo Sloboda
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-04-27 00:00:00.000000000 Z
+date: 2022-05-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -38,6 +38,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: awesome_print
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, with
   optional features for processing large files in parallel, embedded comments, unusual
   field- and record-separators, flexible mapping of CSV-headers to Hash-keys
@@ -126,6 +140,9 @@ files:
 - spec/smarter_csv/malformed_spec.rb
 - spec/smarter_csv/no_header_spec.rb
 - spec/smarter_csv/not_downcase_header_spec.rb
+- spec/smarter_csv/parse/column_separator_spec.rb
+- spec/smarter_csv/parse/old_csv_library_spec.rb
+- spec/smarter_csv/parse/rfc4180_and_more_spec.rb
 - spec/smarter_csv/problematic.rb
 - spec/smarter_csv/quoted_spec.rb
 - spec/smarter_csv/remove_empty_values_spec.rb
@@ -161,8 +178,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
-requirements:
-- csv
+requirements: []
 rubygems_version: 3.1.6
 signing_key:
 specification_version: 4
@@ -233,6 +249,9 @@ test_files:
 - spec/smarter_csv/malformed_spec.rb
 - spec/smarter_csv/no_header_spec.rb
 - spec/smarter_csv/not_downcase_header_spec.rb
+- spec/smarter_csv/parse/column_separator_spec.rb
+- spec/smarter_csv/parse/old_csv_library_spec.rb
+- spec/smarter_csv/parse/rfc4180_and_more_spec.rb
 - spec/smarter_csv/problematic.rb
 - spec/smarter_csv/quoted_spec.rb
 - spec/smarter_csv/remove_empty_values_spec.rb