smarter_csv 1.5.2 → 1.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 88b9932c898320fb05d5697e155dc0bd3ade887d2fcfab7b660933e230007364
-  data.tar.gz: f0525d9c917aff44f910d4547b8e918faa3beb50d47adc29182df1fc1ec2be19
+  metadata.gz: fd2cf82aafc3b45257fbdfc594ed8e1d3bf2226e59cbee144b3003d8f79ec6cf
+  data.tar.gz: 95df862865e3123cf86194d47107f140f69f2fc91c20aba01d4004e8bffa5d74
 SHA512:
-  metadata.gz: 330ad44b9808150f6fdf96dec65d259c2d9cf5eb25e22dc80f63095f4014b065b8aa97a2ba9b814c6cea6f4c0361e04567be403ab78b54d0518b49dc072f36ac
-  data.tar.gz: 27531bd508b5b455a32947badfb85d7e95489ad282a837ef046864806ba7fa12539148ab2fe4c84174fba3ef085dd3adda5d7c070615d684cd99ed0f90b903a3
+  metadata.gz: df32ae9a380fa4fff0932d56e8a0cacadb8d4ebf7d8124e607f2ba389c3b60f875c300a2137fe04aac2b7eda77850b343af5e58c53e310f90460f96223f3228c
+  data.tar.gz: 107e1dbacdc6293a0c044a91cf237f50fbeab59eb5b032167f55d0fe6c2cf07b079c6cfe296368c03d2c84e64ce3c0e6ad744043397cdc217cd3ab51beb3ab09

data/CHANGELOG.md

@@ -1,6 +1,10 @@
 
 # SmarterCSV 1.x Change Log
 
+## 1.6.0 (2022-05-03)
+  * completely rewrote line parser
+  * added methods `SmarterCSV.raw_headers` and `SmarterCSV.headers` to allow easy examination of how the headers are processed.
+
 ## 1.5.2 (2022-04-29)
   * added missing keys to the SmarterCSV::KeyMappingError exception message #189 (thanks to John Dell)
   

data/README.md

@@ -16,10 +16,12 @@
 
 # SmarterCSV
 
-[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.org/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
+[![Build Status](https://secure.travis-ci.org/tilo/smarter_csv.svg?branch=master)](http://travis-ci.com/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
 
 #### SmarterCSV 1.x
 
+`smarter_csv` is now 10 years old, and still kicking! 🎉🎉🎉
+
 `smarter_csv` is a Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, suitable for direct processing with Mongoid or ActiveRecord,
 and parallel processing with Resque or Sidekiq.
 
@@ -42,11 +44,13 @@ NOTE; This Gem is only for importing CSV files - writing of CSV files is not sup
 
 ### Why?
 
-Ruby's CSV library's API is pretty old, and it's processing of CSV-files returning Arrays of Arrays feels 'very close to the metal'. The output is not easy to use - especially not if you want to create database records from it. Another shortcoming is that Ruby's CSV library does not have good support for huge CSV-files, e.g. there is no support for 'chunking' and/or parallel processing of the CSV-content (e.g. with Resque or Sidekiq),
+Ruby's CSV library's API is pretty old, and it's processing of CSV-files returning Arrays of Arrays feels 'very close to the metal'. The output is not easy to use - especially not if you want to create database records or Sidekiq jobs with it. Another shortcoming is that Ruby's CSV library does not have good support for huge CSV-files, e.g. there is no support for 'chunking' and/or parallel processing of the CSV-content (e.g. with Sidekiq).
+
+As the existing CSV libraries didn't fit my needs, I was writing my own CSV processing - specifically for use in connection with Rails ORMs like Mongoid, MongoMapper and ActiveRecord. In those ORMs you can easily pass a hash with attribute/value pairs to the create() method. The lower-level Mongo driver and Moped also accept larger arrays of such hashes to create a larger amount of records quickly with just one call. The same patterns are used when you pass data to Sidekiq jobs.
 
-As the existing CSV libraries didn't fit my needs, I was writing my own CSV processing - specifically for use in connection with Rails ORMs like Mongoid, MongoMapper or ActiveRecord. In those ORMs you can easily pass a hash with attribute/value pairs to the create() method. The lower-level Mongo driver and Moped also accept larger arrays of such hashes to create a larger amount of records quickly with just one call.
+For processing large CSV files it is essential to process them in chunks, so the memory impact is minimized.
 
-### Examples
+### How?
 
 The two main choices you have in terms of how to call `SmarterCSV.process` are:
  * calling `process` with or without a block

data/lib/smarter_csv/smarter_csv.rb

@@ -6,6 +6,7 @@ module SmarterCSV
   class MissingHeaders < SmarterCSVException; end
   class NoColSepDetected < SmarterCSVException; end
   class KeyMappingError < SmarterCSVException; end
+  class MalformedCSVError < SmarterCSVException; end
 
   # first parameter: filename or input object which responds to readline method
   def SmarterCSV.process(input, options={}, &block)
@@ -24,10 +25,6 @@ module SmarterCSV
       options[:row_sep] = SmarterCSV.guess_line_ending(fh, options) if options[:row_sep].to_sym == :auto
       # attempt to auto-detect column separator
       options[:col_sep] = guess_column_separator(fh, options) if options[:col_sep].to_sym == :auto
-      # preserve options, in case we need to call the CSV class
-      csv_options = options.select{|k,v| [:col_sep, :row_sep, :quote_char].include?(k)} # options.slice(:col_sep, :row_sep, :quote_char)
-      csv_options.delete(:row_sep) if [nil, :auto].include?( options[:row_sep].to_sym )
-      csv_options.delete(:col_sep) if [nil, :auto].include?( options[:col_sep].to_sym )
 
       if (options[:force_utf8] || options[:file_encoding] =~ /utf-8/i) && ( fh.respond_to?(:external_encoding) && fh.external_encoding != Encoding.find('UTF-8') || fh.respond_to?(:encoding) && fh.encoding != Encoding.find('UTF-8') )
         puts 'WARNING: you are trying to process UTF-8 input, but did not open the input with "b:utf-8" option. See README file "NOTES about File Encodings".'
@@ -39,7 +36,7 @@ module SmarterCSV
         end
       end
 
-      headerA, header_size = process_headers(fh, options, csv_options)
+      headerA, header_size = process_headers(fh, options)
 
       # in case we use chunking.. we'll need to set it up..
       if ! options[:chunk_size].nil? && options[:chunk_size].to_i > 0
@@ -76,15 +73,8 @@ module SmarterCSV
 
         line.chomp!(options[:row_sep])
 
-        if (line =~ %r{#{options[:quote_char]}}) and (! options[:force_simple_split])
-          dataA = begin
-            CSV.parse( line, **csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
-          rescue CSV::MalformedCSVError => e
-            raise $!, "#{$!} [SmarterCSV: csv line #{@csv_line_count}]", $!.backtrace
-          end
-        else
-          dataA = line.split(options[:col_sep], header_size)
-        end
+        dataA, data_size = parse(line, options, header_size)
+
         dataA.map!{|x| x.sub(/(#{options[:col_sep]})+\z/, '')} # remove any unwanted trailing col_sep characters at the end
         dataA.map!{|x| x.strip} if options[:strip_whitespace]
 
@@ -192,13 +182,6 @@ module SmarterCSV
 
   private
 
-  def self.readline_with_counts(filehandle, options)
-    line  = filehandle.readline(options[:row_sep])
-    @file_line_count += 1
-    @csv_line_count += 1
-    line
-  end
-
   def self.default_options
     {
       auto_row_sep_chars: 500,
@@ -233,6 +216,62 @@ module SmarterCSV
     }
   end
 
+  def self.readline_with_counts(filehandle, options)
+    line  = filehandle.readline(options[:row_sep])
+    @file_line_count += 1
+    @csv_line_count += 1
+    line
+  end
+
+  # parses a single line: either a CSV header and body line
+  # - quoting rules compared to RFC-4180 are somewhat relaxed
+  # - we are not assuming that quotes inside a fields need to be doubled
+  # - we are not assuming that all fields need to be quoted (0 is even)
+  # - works with multi-char col_sep
+  # - if header_size is given, only up to header_size fields are parsed
+  #
+  # We use header_size for parsing the body lines to make sure we always match the number of headers
+  # in case there are trailing col_sep characters in line
+  #
+  # Our convention is that empty fields are returned as empty strings, not as nil.
+  #
+  def self.parse(line, options, header_size = nil)
+    return [] if line.nil?
+
+    col_sep = options[:col_sep]
+    quote = options[:quote_char]
+    quote_count = 0
+    elements = []
+    start = 0
+    i = 0
+
+    while i < line.size do
+      if line[i...i+col_sep.size] == col_sep && quote_count.even?
+        break if !header_size.nil? && elements.size >= header_size
+
+        elements << cleanup_quotes(line[start...i], quote)
+        i += col_sep.size
+        start = i
+      else
+        quote_count += 1 if line[i] == quote
+        i += 1
+      end
+    end
+    elements << cleanup_quotes(line[start..-1], quote) if header_size.nil? || elements.size < header_size
+    [elements, elements.size]
+  end
+
+  def self.cleanup_quotes(field, quote)
+    return field if field.nil? || field !~ /#{quote}/
+
+    if field.start_with?(quote) && field.end_with?(quote)
+      field.delete_prefix!(quote)
+      field.delete_suffix!(quote)
+    end
+    field.gsub!("#{quote}#{quote}", quote)
+    field
+  end
+
   def self.blank?(value)
     case value
     when Array
@@ -319,11 +358,22 @@ module SmarterCSV
     return k                    # the most frequent one is it
   end
 
-  def self.process_headers(filehandle, options, csv_options)
+  def self.raw_hearder
+    @raw_header
+  end
+
+  def self.headers
+    @headers
+  end
+
+  def self.process_headers(filehandle, options)
+    @raw_header = nil
+    @headers = nil
     if options[:headers_in_file]        # extract the header line
       # process the header line in the CSV file..
       # the first line of a CSV file contains the header .. it might be commented out, so we need to read it anyhow
       header = readline_with_counts(filehandle, options)
+      @raw_header = header
 
       header = header.force_encoding('utf-8').encode('utf-8', invalid: :replace, undef: :replace, replace: options[:invalid_byte_sequence]) if options[:force_utf8] || options[:file_encoding] !~ /utf-8/i
       header = header.sub(options[:comment_regexp],'') if options[:comment_regexp]
@@ -331,16 +381,7 @@ module SmarterCSV
 
       header = header.gsub(options[:strip_chars_from_headers], '') if options[:strip_chars_from_headers]
 
-      if (header =~ %r{#{options[:quote_char]}}) and (! options[:force_simple_split])
-        file_headerA = begin
-          CSV.parse( header, **csv_options ).flatten.collect!{|x| x.nil? ? '' : x} # to deal with nil values from CSV.parse
-        rescue CSV::MalformedCSVError => e
-          raise $!, "#{$!} [SmarterCSV: csv line #{@csv_line_count}]", $!.backtrace
-        end
-      else
-        file_headerA =  header.split(options[:col_sep])
-      end
-      file_header_size = file_headerA.size # before mapping, which could delete keys
+      file_headerA, file_header_size = parse(header, options)
 
       file_headerA.map!{|x| x.gsub(%r/#{options[:quote_char]}/,'') }
       file_headerA.map!{|x| x.strip}  if options[:strip_whitespace]
@@ -400,6 +441,7 @@ module SmarterCSV
       raise SmarterCSV::MissingHeaders , "ERROR: missing headers: #{missing_headers.join(',')}" unless missing_headers.empty?
     end
 
+    @headers = headerA
     [headerA, header_size]
   end
 

data/lib/smarter_csv/version.rb

@@ -1,3 +1,3 @@
 module SmarterCSV
-  VERSION = "1.5.2"
+  VERSION = "1.6.0"
 end

data/smarter_csv.gemspec

@@ -16,9 +16,9 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
-  spec.requirements  = ['csv'] # for CSV.parse() only needed in case we have quoted fields
   spec.add_development_dependency "rspec"
   spec.add_development_dependency "simplecov"
+  spec.add_development_dependency "awesome_print"
   #  spec.add_development_dependency "guard-rspec"
 
   spec.metadata["homepage_uri"] = spec.homepage

data/spec/smarter_csv/malformed_spec.rb

@@ -2,16 +2,24 @@ require 'spec_helper'
 
 fixture_path = 'spec/fixtures'
 
-describe 'malformed_csv' do
-  subject { lambda { SmarterCSV.process(csv_path) } }
-
-  context "malformed header" do
+# according to RFC-4180 quotes inside of "words" shouldbe doubled, but our parser is robust against that.
+describe 'malformed CSV quotes' do
+  context "malformed quotes in header" do
     let(:csv_path) { "#{fixture_path}/malformed_header.csv" }
-    it { should raise_error(CSV::MalformedCSVError) }
+    it 'should be resilient against single quotes' do
+      data = SmarterCSV.process(csv_path)
+      expect(data[0]).to eq({:name=>"Arnold Schwarzenegger", :dobdob=>"1947-07-30"})
+      expect(data[1]).to eq({:name=>"Jeff Bridges", :dobdob=>"1949-12-04"})
+    end
   end
 
-  context "malformed content" do
+  context "malformed quotes in content" do
     let(:csv_path) { "#{fixture_path}/malformed.csv" }
-    it { should raise_error(CSV::MalformedCSVError) }
+
+    it 'should be resilient against single quotes' do
+      data = SmarterCSV.process(csv_path)
+      expect(data[0]).to eq({:name=>"Arnold Schwarzenegger", :dob=>"1947-07-30"})
+      expect(data[1]).to eq({:name=>"Jeff \"the dude\" Bridges", :dob=>"1949-12-04"})
+    end
   end
 end

data/spec/smarter_csv/parse/column_separator_spec.rb

@@ -0,0 +1,61 @@
+require 'spec_helper'
+
+describe 'parse with col_sep' do
+  let(:options) { {quote_char: '"'} }
+
+  it 'parses with comma' do
+    line = "a,b,,d"
+    options.merge!({col_sep: ","})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', '', 'd']
+    expect(array_size).to eq 4
+  end
+
+  it 'parses trailing commas' do
+    line = "a,b,c,,"
+    options.merge!({col_sep: ","})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', 'c', '', '']
+    expect(array_size).to eq 5
+  end
+
+  it 'parses with space' do
+    line = "a b  d"
+    options.merge!({col_sep: " "})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', '', 'd']
+    expect(array_size).to eq 4
+  end
+
+  it 'parses with tab' do
+    line = "a\tb\t\td"
+    options.merge!({col_sep: "\t"})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a', 'b', '', 'd']
+    expect(array_size).to eq 4
+  end
+
+  it 'parses with multiple space separator' do
+    line = "a b    d"
+    options.merge!({col_sep: "  "})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['a b', '', 'd']
+    expect(array_size).to eq 3
+  end
+
+  it 'parses with multiple char separator' do
+    line = '<=><=>A<=>B<=>C'
+    options.merge!({col_sep: "<=>"})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ["", "", "A", "B", "C"]
+    expect(array_size).to eq 5
+  end
+
+  it 'parses trailing multiple char separator' do
+    line = '<=><=>A<=>B<=>C<=><=>'
+    options.merge!({col_sep: "<=>"})
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ["", "", "A", "B", "C", "", ""]
+    expect(array_size).to eq 7
+  end
+end

data/spec/smarter_csv/parse/old_csv_library_spec.rb

@@ -0,0 +1,74 @@
+require 'spec_helper'
+
+describe 'old CSV library parsing tests' do
+  let(:options) { {quote_char: '"', col_sep: ","} }
+
+  [ ["\t", ["\t"]],
+    ["foo,\"\"\"\"\"\",baz", ["foo", "\"\"", "baz"]],
+    ["foo,\"\"\"bar\"\"\",baz", ["foo", "\"bar\"", "baz"]],
+    ["\"\"\"\n\",\"\"\"\n\"", ["\"\n", "\"\n"]],
+    ["foo,\"\r\n\",baz", ["foo", "\r\n", "baz"]],
+    ["\"\"", [""]],
+    ["foo,\"\"\"\",baz", ["foo", "\"", "baz"]],
+    ["foo,\"\r.\n\",baz", ["foo", "\r.\n", "baz"]],
+    ["foo,\"\r\",baz", ["foo", "\r", "baz"]],
+    ["foo,\"\",baz", ["foo", "", "baz"]],
+    ["\",\"", [","]],
+    ["foo", ["foo"]],
+    [",,", ['', '', '']],
+    [",", ['', '']],
+    ["foo,\"\n\",baz", ["foo", "\n", "baz"]],
+    ["foo,,baz", ["foo", '', "baz"]],
+    ["\"\"\"\r\",\"\"\"\r\"", ["\"\r", "\"\r"]],
+    ["\",\",\",\"", [",", ","]],
+    ["foo,bar,", ["foo", "bar", '']],
+    [",foo,bar", ['', "foo", "bar"]],
+    ["foo,bar", ["foo", "bar"]],
+    [";", [";"]],
+    ["\t,\t", ["\t", "\t"]],
+    ["foo,\"\r\n\r\",baz", ["foo", "\r\n\r", "baz"]],
+    ["foo,\"\r\n\n\",baz", ["foo", "\r\n\n", "baz"]],
+    ["foo,\"foo,bar\",baz", ["foo", "foo,bar", "baz"]],
+    [";,;", [";", ";"]]
+  ].each do |line, result|
+    it "parses #{line}" do
+      array, array_size = SmarterCSV.send(:parse, line, options)
+      expect(array).to eq result
+    end
+  end
+
+  [ ["foo,\"\"\"\"\"\",baz", ["foo", "\"\"", "baz"]],
+    ["foo,\"\"\"bar\"\"\",baz", ["foo", "\"bar\"", "baz"]],
+    ["foo,\"\r\n\",baz", ["foo", "\r\n", "baz"]],
+    ["\"\"", [""]],
+    ["foo,\"\"\"\",baz", ["foo", "\"", "baz"]],
+    ["foo,\"\r.\n\",baz", ["foo", "\r.\n", "baz"]],
+    ["foo,\"\r\",baz", ["foo", "\r", "baz"]],
+    ["foo,\"\",baz", ["foo", "", "baz"]],
+    ["foo", ["foo"]],
+    [",,", ['', '', '']],
+    [",", ['', '']],
+    ["foo,\"\n\",baz", ["foo", "\n", "baz"]],
+    ["foo,,baz", ["foo", '', "baz"]],
+    ["foo,bar", ["foo", "bar"]],
+    ["foo,\"\r\n\n\",baz", ["foo", "\r\n\n", "baz"]],
+    ["foo,\"foo,bar\",baz", ["foo", "foo,bar", "baz"]]
+  ].each do |line, result|
+    it "parses #{line}" do
+      array, array_size = SmarterCSV.send(:parse, line, options)
+      expect(array).to eq result
+    end
+  end
+
+  it 'mixed quotes' do
+    line = %Q{Ten Thousand,10000, 2710 ,,"10,000","It's ""10 Grand"", baby",10K}
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ["Ten Thousand", "10000", " 2710 ", "", "10,000", "It's \"10 Grand\", baby", "10K"]
+  end
+
+  it 'single quotes in fields' do
+    line = 'Indoor Chrome,49.2"" L x 49.2"" W x 20.5"" H,Chrome,"Crystal,Metal,Wood",23.12'
+    array, array_size = SmarterCSV.send(:parse, line, options)
+    expect(array).to eq ['Indoor Chrome', '49.2" L x 49.2" W x 20.5" H', 'Chrome', 'Crystal,Metal,Wood', '23.12']
+  end
+end

data/spec/smarter_csv/parse/rfc4180_and_more_spec.rb

@@ -0,0 +1,170 @@
+require 'spec_helper'
+
+fixture_path = 'spec/fixtures'
+
+describe 'fulfills RFC-4180 and more' do
+  let(:options) { {col_sep: ',', row_sep: $INPUT_RECORD_SEPARATOR, quote_char: '"' } }
+
+  context 'parses simple CSV' do
+    context 'RFC-4180' do
+      it 'separating on col_sep' do
+        line = 'aaa,bbb,ccc'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [%w[aaa bbb ccc], 3]
+      end
+
+      it 'preserves whitespace' do
+        line = ' aaa , bbb , ccc '
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          [' aaa ', ' bbb ', ' ccc '], 3
+        ]
+      end
+    end
+
+    context 'extending RFC-4180' do
+      it 'with extra col_sep' do
+        line = 'aaa,bbb,ccc,'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', ''], 4
+        ]
+      end
+
+      it 'with extra col_sep with given header_size' do
+        line = 'aaa,bbb,ccc,'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+
+      it 'with multiple extra col_sep' do
+        line = 'aaa,bbb,ccc,,,'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple extra col_sep' do
+        line = 'aaa,bbb,ccc,,,'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+
+      it 'with multiple complex col_sep' do
+        line = 'aaa<=>bbb<=>ccc<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}))).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple complex col_sep with given header_size' do
+        line = 'aaa<=>bbb<=>ccc<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}), 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+    end
+  end
+
+  context 'parses quoted CSV' do
+    context 'RFC-4180' do
+      it 'separating on col_sep' do
+        line = '"aaa","bbb","ccc"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [%w[aaa bbb ccc], 3]
+      end
+
+      it 'parses corner case correctly' do
+        line = '"Board 4""","$17.40","10000003427"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['Board 4"', '$17.40', '10000003427'], 3
+        ]
+      end
+
+      it 'quoted parts can contain spaces' do
+        line = '" aaa1 aaa2 "," bbb1 bbb2 "," ccc1 ccc2 "'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          [' aaa1 aaa2 ', ' bbb1 bbb2 ', ' ccc1 ccc2 '], 3
+        ]
+      end
+
+      it 'quoted parts can contain row_sep' do
+        line = '"aaa1, aaa2","bbb1, bbb2","ccc1, ccc2"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa1, aaa2', 'bbb1, bbb2', 'ccc1, ccc2'], 3
+        ]
+      end
+
+      it 'quoted parts can contain row_sep' do
+        line = '"aaa1, ""aaa2"", aaa3","""bbb1"", bbb2","ccc1, ""ccc2"""'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa1, "aaa2", aaa3', '"bbb1", bbb2', 'ccc1, "ccc2"'], 3
+        ]
+      end
+
+      it 'some fields are quoted' do
+        line = '1,"board 4""",12.95'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['1', 'board 4"', '12.95'], 3
+        ]
+      end
+
+      it 'separating on col_sep' do
+        line = '"some","thing","""completely"" different"'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['some', 'thing', '"completely" different'], 3
+        ]
+      end
+    end
+
+    context 'extending RFC-4180' do
+      it 'with extra col_sep, without given header_size' do
+        line = '"aaa","bbb","ccc",'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', ''], 4
+        ]
+      end
+
+      it 'with extra col_sep, with given header_size' do
+        line = '"aaa","bbb","ccc",'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [%w[aaa bbb ccc], 3]
+      end
+
+      it 'with multiple extra col_sep, without given header_size' do
+        line = '"aaa","bbb","ccc",,,'
+        expect( SmarterCSV.send(:parse, line, options)).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple extra col_sep, with given header_size' do
+        line = '"aaa","bbb","ccc",,,'
+        expect( SmarterCSV.send(:parse, line, options, 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+
+      it 'with multiple complex extra col_sep, without given header_size' do
+        line = '"aaa"<=>"bbb"<=>"ccc"<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}))).to eq [
+          ['aaa', 'bbb', 'ccc', '', '', ''], 6
+        ]
+      end
+
+      it 'with multiple complex extra col_sep, with given header_size' do
+        line = '"aaa"<=>"bbb"<=>"ccc"<=><=><=>'
+        expect( SmarterCSV.send(:parse, line, options.merge({col_sep: '<=>'}), 3)).to eq [
+          ['aaa', 'bbb', 'ccc'], 3
+        ]
+      end
+    end
+  end
+
+  # relaxed parsing compared to RFC-4180
+  context 'liberal_parsing' do
+    it 'parses corner case correctly' do
+      line = 'is,this "three, or four",fields'
+      expect( SmarterCSV.send(:parse, line, options)).to eq [
+        ['is', 'this "three, or four"', 'fields'], 3
+      ]
+    end
+  end
+end

data/spec/smarter_csv/quoted_spec.rb

@@ -3,7 +3,6 @@ require 'spec_helper'
 fixture_path = 'spec/fixtures'
 
 describe 'loading file with quoted fields' do
-
   it 'leaving the quotes in the data' do
     options = {}
     data = SmarterCSV.process("#{fixture_path}/quoted.csv", options)
@@ -12,6 +11,7 @@ describe 'loading file with quoted fields' do
     data[1][:description].should be_nil
     data[2][:model].should eq 'Venture "Extended Edition, Very Large"'
     data[2][:description].should be_nil
+    data[3][:description].should eq 'MUST SELL! air, moon roof, loaded'
     data.each do |h|
       h[:year].class.should eq Fixnum
       h[:make].should_not be_nil
@@ -20,17 +20,21 @@ describe 'loading file with quoted fields' do
     end
   end
 
-
+  # quotes inside quoted fields need to be escaped by another double-quote
   it 'removes quotes around quoted fields, but not inside data' do
     options = {}
     data = SmarterCSV.process("#{fixture_path}/quote_char.csv", options)
 
     data.length.should eq 6
+    data[0][:first_name].should eq "\"John"
+    data[0][:last_name].should eq "Cooke\""
     data[1][:first_name].should eq "Jam\ne\nson\""
     data[2][:first_name].should eq "\"Jean"
+    data[4][:first_name].should eq "Bo\"bbie"
+    data[5][:first_name].should eq 'Mica'
+    data[5][:last_name].should eq 'Copeland'
   end
 
-
   # NOTE: quotes inside headers need to be escaped by doubling them
   #       e.g. 'correct ""EXAMPLE""'
   #       this escaping is illegal: 'incorrect \"EXAMPLE\"' <-- this caused CSV parsing error
@@ -43,6 +47,6 @@ describe 'loading file with quoted fields' do
     data.length.should eq 3
     data.first.keys[2].should eq :isbn
     data.first.keys[3].should eq :discounted_price
+    data[1][:author].should eq 'Timothy "The Parser" Campbell'
   end
-
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: smarter_csv
 version: !ruby/object:Gem::Version
-  version: 1.5.2
+  version: 1.6.0
 platform: ruby
 authors:
 - Tilo Sloboda
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-04-29 00:00:00.000000000 Z
+date: 2022-05-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rspec
@@ -38,6 +38,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: awesome_print
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Ruby Gem for smarter importing of CSV Files as Array(s) of Hashes, with
   optional features for processing large files in parallel, embedded comments, unusual
   field- and record-separators, flexible mapping of CSV-headers to Hash-keys
@@ -126,6 +140,9 @@ files:
 - spec/smarter_csv/malformed_spec.rb
 - spec/smarter_csv/no_header_spec.rb
 - spec/smarter_csv/not_downcase_header_spec.rb
+- spec/smarter_csv/parse/column_separator_spec.rb
+- spec/smarter_csv/parse/old_csv_library_spec.rb
+- spec/smarter_csv/parse/rfc4180_and_more_spec.rb
 - spec/smarter_csv/problematic.rb
 - spec/smarter_csv/quoted_spec.rb
 - spec/smarter_csv/remove_empty_values_spec.rb
@@ -161,8 +178,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
-requirements:
-- csv
+requirements: []
 rubygems_version: 3.1.6
 signing_key:
 specification_version: 4
@@ -233,6 +249,9 @@ test_files:
 - spec/smarter_csv/malformed_spec.rb
 - spec/smarter_csv/no_header_spec.rb
 - spec/smarter_csv/not_downcase_header_spec.rb
+- spec/smarter_csv/parse/column_separator_spec.rb
+- spec/smarter_csv/parse/old_csv_library_spec.rb
+- spec/smarter_csv/parse/rfc4180_and_more_spec.rb
 - spec/smarter_csv/problematic.rb
 - spec/smarter_csv/quoted_spec.rb
 - spec/smarter_csv/remove_empty_values_spec.rb