dsv 0.12.4 → 0.12.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 94796d930eb9448d20a7ac7c4a43598e2cdeaef2e4744f2309414f48a3b73dd9
-  data.tar.gz: 6c9f3cf66144d9d9246080216e3200faf0eeaf169e3c6f023d07e3ae6b185a2b
+  metadata.gz: d6c8309ce7e75acd45ed55995d9c1ef8f04f1c6e945e56e80cb60d1fe4b7fa2a
+  data.tar.gz: 2f6a5693d250797bf9e9ce69f2d06c2fd144e5722c7f3e7996c0f4029b71fdea
 SHA512:
-  metadata.gz: 357af1737793d37e8134c96912cfe4a666b6b41eccbba31a9bedbb67bc282c9f42abe0752df3775f803f13d71fe0f25639a64eb7748cad0eed34dd078dfcad8d
-  data.tar.gz: 8c6abaa1867be5c7c9d649903f0d844be6a63971b4ac09551f3203a282be7bbb2739f669e9bbbaf33cc65ed3c99095b2c640a782c6291ce8e852524b93796075
+  metadata.gz: c91f3fd66f79475ec7c617f47545378a08af49e3a78e7b52b684496b3ab6cb22205e2c26a5c7cd025ca26956ac8730685e2a4344ce7505ea4e46ef737ef0d7e5
+  data.tar.gz: 3b2c669417438150a4fdfe421358e75362148ac7fa4fe8be1579c79b4c0f7f862283a2797dbef5bd02949cf9218daa61cdc194c0ca84844f443d39242e937245

data/CHANGELOG

@@ -1,5 +1,19 @@
 # dsv/CHANGELOG
 
+## 20261016
+
+0.12.5: Regexp separators and quote: :double made to do what the README says.
+
+1. ~ split_row(): a Regexp separator reads a quoted field holding it, rejoined with the text matched; + pieces_and_separators(), + separator_before(), + split_on(). It had raised TypeError.
+2. ~ split_row() under :double: + split_double_quoted(), wholly quoted or wholly unquoted rows split fast, any other row read as with no mode named; + split_quoted(), + unescaped_fields(), + quoted_separator. "a",b,"c" had been read as one field.
+3. ~ complete_quoted_row(): carries a quoted field across lines under :double too.
+4. ~ writing with a Regexp separator raises ArgumentError; + writable_column_separator. The row renderer is chosen at the first write.
+5. ~ test/dsv_test.rb: - the skip on a Regexp separator with a quoted field; + examples of the above.
+6. ~ README.md: writing with a Regexp raises; how :double reads a row it does not promise.
+7. ~ TODO: 5, - the Regexp separator as a reason for the scanner parser.
+8. ~ lib/DSV/VERSION.rb: /0.12.4/0.12.5/.
+
+
 ## 20260913, 1005
 
 0.12.4: + dsv(.rb).gemspec, Gemfile, Rakefile, .gitignore and LICENSE.
@@ -41,7 +55,7 @@
 
 0.11.18: + TODO, and the header's Todo, Ideas and Bugs sections retired into it.
 
-1. + TODO: what survives of the header's three sections and of _libraries/SimpleCSV/notes.txt, triaged: what is done or fixed is dropped with its version named in the reconciliation, and the rest is grouped as 0.12.0, decided; 0.13.0, speed; and features after, the guessing mode (separator, header, and a header made from the data where there is none), templated ingestion as a whole-row Regexp, the duplicate-header modes, the writing options, symbols and strings interchangeably, a strict: option, and a line number on parse errors.
+1. + TODO: what survives of the header's three sections and of SimpleCSV's notes, triaged: what is done or fixed is dropped with its version named in the reconciliation, and the rest is grouped as 0.12.0, decided; 0.13.0, speed; and features after, the guessing mode (separator, header, and a header made from the data where there is none), templated ingestion as a whole-row Regexp, the duplicate-header modes, the writing options, symbols and strings interchangeably, a strict: option, and a line number on parse errors.
 2. ~ lib/SimpleCSV.rb: - the Todo, Ideas and Bugs sections; the header is the name, the date, the version and the description. /0.11.17/0.11.18/.
 
 0.11.17: ~ SimpleCSV.header_row, the header row as names.
@@ -168,7 +182,7 @@
 
 1. + test/SimpleCSV_test.rb: MiniTest spec style, 102 examples. 76 pass and state the interface as it stands; 26 are skipped and state what the library should do and does not yet, each skip message naming its finding in the Handoff 0 closing note, so that the skip count is the fault list and each fix from here removes one skip. The RFC 4180 cases are rewritten from stdlib CSV's parsing tests. Run as `ruby test/SimpleCSV_test.rb`.
 2. ~ lib/SimpleCSV.rb: - the Changes section of the header, its entries being here from 0.10.0 on; the date and version lines stay. From here each entry is written by hand, one per commit.
-3. 0.10.4 was the last revision imported from the staging tree; from here every change is a commit.
+3. 0.10.4 was the last imported version; from here every change is a commit.
 
 ### Note that prior to 0.10.4 predates this being in git.
 

data/README.md

@@ -76,7 +76,7 @@ DSV.read('people.txt', headers: true, column_separator: /,\s*/)
 DSV.read('people.csv', headers: true, row_separator: "\r\n")
 ```
 
-`col_sep:` and `row_sep:` are accepted as spellings of those. Writing takes the same two options, so a file round-trips through whatever it was read with.
+`col_sep:` and `row_sep:` are accepted as spellings of those. Writing takes the same two options, so a file round-trips through whatever it was read with. A Regexp says what may be read as a separator, not what to write, so writing with one raises `ArgumentError`; give a String to write.
 
 ### Quoting
 
@@ -87,6 +87,8 @@ DSV.read('plain.csv', headers: true, quote: :none)    # no field is quoted; quot
 DSV.read('quoted.csv', headers: true, quote: :double)  # every field is quoted, as DSV writes them
 ```
 
+Under `:double` a row that is wholly unquoted, as a header row often is, is split plainly, and a row that is neither wholly quoted nor wholly unquoted is read as it would be with no mode named, so naming the mode never costs a correct read.
+
 ### Writing
 
 Set the columns, give the rows as Hashes, and write:

data/TODO

@@ -9,7 +9,7 @@
 
 ## 0.13.0, speed
 
-5. The saved-up speed change, with the evidence in the reconciliation: two string predicates in place of four regex matches per piece in split_csv; rows built by zip.to_h; a benchmark script in the repository against stdlib CSV, and the README's speed sentence written from its output. The scanner-based parser on the branch scanner-parser is judged here, for the Regexp separator with a quoted field if for nothing else.
+5. The saved-up speed change, with the evidence in the reconciliation: two string predicates in place of four regex matches per piece in split_csv; rows built by zip.to_h; a benchmark script in the repository against stdlib CSV, and the README's speed sentence written from its output. The scanner-based parser on the branch scanner-parser is judged here.
 
 ## Features, after
 

data/lib/DSV/VERSION.rb

@@ -2,5 +2,5 @@
 # DSV::VERSION
 
 class DSV
-  VERSION = '0.12.4'
+  VERSION = '0.12.5'
 end

data/lib/dsv.rb

@@ -156,7 +156,6 @@ class DSV
     @column_separator = options[:column_separator] || options[:col_sep] || ','
     @selected_columns = options[:selected_columns]
     @as_array = options[:as_array] || false
-    @render_row = row_renderer
     if options[:columns]
       self.columns = options[:columns]
     else
@@ -264,32 +263,84 @@ class DSV
     value.nil? || value.empty?
   end
 
-  # One row into its fields, by the quote mode: :none splits; :double splits a wholly quoted row on quote-separator-quote and a wholly unquoted one plainly; unspecified splits and reassembles a quoted field that held the separator, a doubled quote inside it reading as one. Direct string operations throughout; the scanner-based parser is on the branch scanner-parser.
+  # One row into its fields, by the quote mode: :none splits; :double splits a row as it promises, wholly quoted or wholly unquoted, and one that is neither as unspecified would; unspecified splits and reassembles a quoted field that held the separator, a doubled quote inside it reading as one. Direct string operations throughout; the scanner-based parser is on the branch scanner-parser.
   def split_row(raw_row)
     row = raw_row.chomp(@row_separator)
     case @quote
     when :none, :unquoted
-      row.split(@column_separator, -1)
+      split_on(row, @column_separator)
     when :double, :double_quoted, :double_quotes
-      if row.start_with?('"')
-        row.delete_prefix('"').delete_suffix('"').split('"' + @column_separator + '"', -1).collect{|field| field.gsub('""', '"')}
-      else
-        row.split(@column_separator, -1)
-      end
+      split_double_quoted(row) || split_quoted(row)
+    else
+      split_quoted(row)
+    end
+  end
+
+  # A row split on a separator, String or Regexp; a Regexp's matches divide it, since String#split would also return any groups it captures.
+  def split_on(row, separator)
+    separator.is_a?(::Regexp) ? pieces_and_separators(row, separator).first : row.split(separator, -1)
+  end
+
+  # A row's pieces between a Regexp's matches, with the text of each match, so that a quoted field may be joined back with the separator it held.
+  def pieces_and_separators(row, separator)
+    pieces, separators, position = [], [], 0
+    row.scan(separator) do
+      match = ::Regexp.last_match
+      pieces << row[position...match.begin(0)]
+      separators << match[0]
+      position = match.end(0)
+    end
+    [pieces << row[position..], separators]
+  end
+
+  # Unspecified quoting: the row split on the separator, a quoted field that held it joined back with the separator's own text.
+  def split_quoted(row)
+    if @column_separator.is_a?(::Regexp)
+      reassemble_quoted_fields(*pieces_and_separators(row, @column_separator))
     else
       reassemble_quoted_fields(row.split(@column_separator, -1))
     end
   end
 
-  # Pieces of a row split on the separator, the pieces of a quoted field that held the separator joined back with it; a piece that is a lone quote opens or closes a field.
-  def reassemble_quoted_fields(pieces)
+  # The :double contract: a wholly quoted row split on quote-separator-quote, a wholly unquoted one plainly; nil for a row that is neither.
+  def split_double_quoted(row)
+    if row.start_with?('"')
+      unescaped_fields(split_on(row.delete_prefix('"').delete_suffix('"'), quoted_separator))
+    else
+      split_on(row, @column_separator) unless row.include?('"')
+    end
+  end
+
+  # The fields of a row split as wholly quoted, a doubled quote read as one; nil where a field holds a lone quote, the mark of a row that was not. A field has twice the quotes it has unescaped exactly when every quote in it is one of a doubled pair.
+  def unescaped_fields(fields)
+    fields.collect do |field|
+      next field unless field.include?('"')
+      unescaped = field.gsub('""', '"')
+      return nil unless field.count('"') == 2 * unescaped.count('"')
+      unescaped
+    end
+  end
+
+  # Quote, separator, quote: what divides the fields of a wholly quoted row.
+  def quoted_separator
+    @quoted_separator ||= (
+      if @column_separator.is_a?(::Regexp)
+        ::Regexp.new('"(?:' + @column_separator.source + ')"', @column_separator.options)
+      else
+        '"' + @column_separator + '"'
+      end
+    )
+  end
+
+  # Pieces of a row split on the separator, the pieces of a quoted field that held the separator joined back with it, or with the text it matched where the separator is a Regexp; a piece that is a lone quote opens or closes a field.
+  def reassemble_quoted_fields(pieces, separators = nil)
     fields = []
     buffer = nil
-    pieces.each do |piece|
+    pieces.each_with_index do |piece, i|
       if buffer && !piece.end_with?('"')
-        buffer << @column_separator << piece
+        buffer << separator_before(i, separators) << piece
       elsif buffer
-        fields << (buffer << @column_separator << piece.delete_suffix('"')).gsub('""', '"')
+        fields << (buffer << separator_before(i, separators) << piece.delete_suffix('"')).gsub('""', '"')
         buffer = nil
       elsif piece.start_with?('"') && (piece.length == 1 || !piece.end_with?('"'))
         buffer = +piece.delete_prefix('"')
@@ -302,9 +353,14 @@ class DSV
     fields
   end
 
-  # A quoted field may hold the row separator: while the quotes in a row are unbalanced the next line belongs to it.
+  # What stood between a piece and the one before it: the separator, or the text a Regexp separator matched there.
+  def separator_before(i, separators)
+    separators ? separators[i - 1] : @column_separator
+  end
+
+  # A quoted field may hold the row separator: while the quotes in a row are unbalanced the next line belongs to it, in every quote mode which reads quotes as structure, :none and :unquoted reading them as data.
   def complete_quoted_row(raw_row)
-    raw_row << @source.gets(@row_separator).to_s while @quote.nil? && raw_row.count('"').odd? && !@source.eof?
+    raw_row << @source.gets(@row_separator).to_s while @quote != :none && @quote != :unquoted && raw_row.count('"').odd? && !@source.eof?
     raw_row
   end
 
@@ -377,10 +433,17 @@ class DSV
   alias_method :write_csv_row, :write_row
 
   def write_values(values)
+    @render_row ||= row_renderer
     @source.write(@render_row.call(values))
   end
 
-  # Chosen once from the quote mode: how a row of values is rendered, with the instance's separators, a quote inside a quoted value doubled as RFC 4180 has it. The spacey modes put a space after each separator, as Array#to_csv_row did.
+  # The column separator a row is written with: a Regexp says what may be read as one, not what to write.
+  def writable_column_separator
+    raise ::ArgumentError, "a Regexp column separator can be read with, not written with; give a String to write" if @column_separator.is_a?(::Regexp)
+    @column_separator
+  end
+
+  # Chosen once, at the first write, from the quote mode: how a row of values is rendered, with the instance's separators, a quote inside a quoted value doubled as RFC 4180 has it. The spacey modes put a space after each separator, as Array#to_csv_row did.
   def row_renderer
     quote, spacey = case @quote&.to_sym
                     when :none, :unquoted then [nil, false]
@@ -390,7 +453,7 @@ class DSV
                     when :spacey_double then ['"', true]
                     else ['"', false]
                     end
-    joiner = spacey ? @column_separator + ' ' : @column_separator
+    joiner = spacey ? writable_column_separator + ' ' : writable_column_separator
     if quote
       doubled = quote * 2
       ->(values){values.collect{|value| quote + value.to_s.gsub(quote, doubled) + quote}.join(joiner) + @row_separator}

data/test/dsv_test.rb

@@ -442,8 +442,21 @@ describe DSV do
     end
 
     it "reads a regular expression separator with a quoted field holding the separator" do
-      skip "Observation: reassembling a quoted field concatenates the separator back in, which a Regexp cannot be; the parked scanner parser handles it"
       _(DSV.read("a,b\n\"x, y\",2\n", headers: true, column_separator: /,\s*/)).must_equal [{'a' => 'x, y', 'b' => '2'}]
+      _(DSV.read("\"x,  y\",2\n", column_separator: /,\s*/)).must_equal [{0 => 'x,  y', 1 => '2'}]
+    end
+
+    it "reads a regular expression separator under quote: :double" do
+      _(DSV.read("\"a\"; \"b\"\n\"x; y\";\"2\"\n", headers: true, quote: :double, column_separator: /;\s*/)).must_equal [{'a' => 'x; y', 'b' => '2'}]
+    end
+
+    it "reads a regular expression separator that captures, without the captures becoming fields" do
+      _(DSV.read("1;2,3\n", quote: :none, column_separator: /(;|,)/)).must_equal [{0 => '1', 1 => '2', 2 => '3'}]
+      _(DSV.read("\"x;y\",2\n", column_separator: /(;|,)/)).must_equal [{0 => 'x;y', 1 => '2'}]
+    end
+
+    it "reads with a regular expression separator under the spacey quote modes" do
+      _(DSV.read("1, 2\n", quote: :spacey_none, column_separator: /,\s*/)).must_equal [{0 => '1', 1 => '2'}]
     end
   end
 
@@ -519,6 +532,20 @@ describe DSV do
     it "keeps a separator inside a quoted field under quote: :double" do
       _(DSV.read("a,b\n\"x, y\",\"2\"\n", headers: true, quote: :double)).must_equal [{'a' => 'x, y', 'b' => '2'}]
     end
+
+    it "reads a row that is neither wholly quoted nor wholly unquoted under quote: :double as it would unspecified" do
+      _(DSV.read("\"a\",b,\"c\"\n", quote: :double)).must_equal [{0 => 'a', 1 => 'b', 2 => 'c'}]
+      _(DSV.read("1,\"x\"\n", quote: :double)).must_equal [{0 => '1', 1 => 'x'}]
+      _(DSV.read("\"a\",\"b\",c\n", quote: :double)).must_equal [{0 => 'a', 1 => 'b', 2 => 'c'}]
+      _(DSV.read("a,b\n1,\"x, y\"\n", headers: true, quote: :double)).must_equal [{'a' => '1', 'b' => 'x, y'}]
+    end
+
+    it "reads a quoted field holding the row separator as one field across lines under quote: :double, as DSV writes it" do
+      _(DSV.read("\"x\ny\",\"2\"\n\"z\",\"3\"\n", quote: :double)).must_equal [{0 => "x\ny", 1 => '2'}, {0 => 'z', 1 => '3'}]
+      _(DSV.read("1,\"x\ny\"\n", quote: :double)).must_equal [{0 => '1', 1 => "x\ny"}]
+      output = written(headers: true, columns: ['a', 'b']){|csv| csv.rows = [{'a' => "x\ny", 'b' => '2'}]; csv.write}
+      _(DSV.read(output, headers: true, quote: :double)).must_equal [{'a' => "x\ny", 'b' => '2'}]
+    end
   end
 
   describe "#write" do
@@ -597,6 +624,11 @@ describe DSV do
       _(written(columns: [:a, :b], quote: :spacey_none){|csv| csv.write_row('a' => 1, 'b' => 2)}).must_equal "1, 2\n"
       _(written(columns: [:a, :b], quote: :spacey_double){|csv| csv.write_row('a' => 1, 'b' => 2)}).must_equal "\"1\", \"2\"\n"
     end
+
+    it "refuses to write with a regular expression separator, there being no one string to write between fields" do
+      error = _{written(columns: [:a, :b], column_separator: /,\s*/){|csv| csv.write_row('a' => 1, 'b' => 2)}}.must_raise ArgumentError
+      _(error.message).must_match(/Regexp/)
+    end
   end
 
   describe "DSV::String" do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dsv
 version: !ruby/object:Gem::Version
-  version: 0.12.4
+  version: 0.12.5
 platform: ruby
 authors:
 - thoran
@@ -91,7 +91,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.21
+rubygems_version: 4.0.23
 specification_version: 4
 summary: Read and write delimiter-separated values.
 test_files: []