dsv 0.12.7 → 0.12.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ac547ecbc72cfc0cfb3b4e2a1f9c0c5327ea1f4e064c126a28aa85f8c3a42ded
-  data.tar.gz: 5be7f5303ed207575c43dfacac8d7b4e68f259172b48d25aba60104af57b68f7
+  metadata.gz: 7171a2d2de4dba26f3e304ca3e0c216e9bb093a36cef1420629e9f936e644889
+  data.tar.gz: 4ccf92f6000c7e5b9d82315babc17c816edd3df7587583eb79f2c16eb19125cb
 SHA512:
-  metadata.gz: 022fbe8f4caf5baf90c785ba1ef586574d270331e25b6734e1e982766e1a8836a3182f1f92452734ce1d3316e834e0b07da4b8f9b89f6105c45222e4505dab7b
-  data.tar.gz: 19cd8f16d5b72349be0e066f63a5dcbac7cc9ea1ab4736e67cd07507415794ce2f9cc47d9bc747358125c252480018b1a0170efd820bdab8056611f46971cfd0
+  metadata.gz: 4727efe3e9571299914d4b6731169ac5a70eabd475176f06cb084dba35cfcc038420faf4740ad01bbedb9caea1b13ba21ed40a058f2699ca6f4fedf9ba47a0b1
+  data.tar.gz: 6b6143f85e9c6106344fed384228433057eff0641a38eff9e6f823a2d905cca416284b7068f80ff26529feb02045fe09f323e689614012bd267d0783aec469ba

data/CHANGELOG

@@ -1,5 +1,27 @@
 # dsv/CHANGELOG
 
+## 20261028
+
+0.12.9: + DSV::UnterminatedQuote, raised where a quoted field does not close.
+
+1. + lib/DSV/UnterminatedQuote.rb: DSV::UnterminatedQuote, with line, the line on which the field's row began.
+2. ~ row_fields(), parse_row(), columns(): raise it where a quoted field is still open at the end of the source, or of the line or header given. Such a field had been dropped silently, with its row.
+3. + line_of(): counts the lines before the row only when raising.
+4. ~ test/dsv_test.rb: - the skip on a quoted field which does not close; + the line it names; the malformed examples now state the default rule.
+5. ~ README.md: Quoting, the rule for input which isn't RFC 4180.
+6. ~ TODO: 12, as UnterminatedQuote finds its line; + 15, the rule as an option.
+7. ~ lib/DSV/VERSION.rb: /0.12.8/0.12.9/.
+
+0.12.8: a stray quote no longer takes in the next line.
+
+1. /complete_quoted_row()/row_fields()/: takes the next line into a row while a quoted field in it is open, rather than while its quotes are odd in number. A stray quote had made the count odd, and one beside an open field even.
+2. ~ reassemble_quoted_fields(), split_row(): nil where a quoted field is still open at the end of the row.
+3. ~ split_double_quoted(): takes a row as wholly quoted only where it ends in a closing quote.
+4. + parse_fields(), from parse_row(), so that the read loop splits each row once.
+5. ~ test/dsv_test.rb: - the skip on a stray quote; + a quoted field across lines after a stray quote.
+6. ~ lib/DSV/VERSION.rb: /0.12.7/0.12.8/.
+
+
 ## 20261027
 
 0.12.7: ~ reassemble_quoted_fields(): an escaped quote before a separator no longer closes a quoted field.

data/README.md

@@ -89,6 +89,8 @@ DSV.read('quoted.csv', headers: true, quote: :double)  # every field is quoted,
 
 Under `:double` a row that is wholly unquoted, as a header row often is, is split plainly, and a row that is neither wholly quoted nor wholly unquoted is read as it would be with no mode named, so naming the mode never costs a correct read.
 
+Input which isn't RFC 4180 is read to one rule. A quote opens a field only at the start of one, so a quote anywhere else, as in `12" pizza`, is data. A quoted field closes only at a quote followed by the separator or the end of the row, so one which doesn't close by the end of the source, as `"abc` or `"ab"c` doesn't, raises `DSV::UnterminatedQuote`, whose `line` is the line its row began on.
+
 ### Writing
 
 Set the columns, give the rows as Hashes, and write:

data/TODO

@@ -19,6 +19,7 @@
 9. Writing options: minimal quoting, a field quoted only where it holds a separator, a quote or a row terminator; a row terminator inside a field kept, quoted, or folded to a space; the encoding declared and a transcode offered with any loss marked.
 10. Symbols and Strings interchangeable for column names, in a selection, a columns: list and a row's keys; today a String is required.
 11. A strict: option, off by default, that raises on a row with more or fewer fields than the header; today a short row gets only the columns it has and a long row's excess lands under nil.
-12. A line number in each parse error, UnterminatedQuote first, found only when the error is raised, from the source's position at that moment; never counted as it goes, since counting costs time on every row for the sake of the rare one.
+12. A line number in each parse error, found only when the error is raised, from the source's position at that moment, as UnterminatedQuote finds its; never counted as it goes, since counting costs time on every row for the sake of the rare one.
 13. Chunked processing of a large file: a bounded number of rows at a time handed to a block, so that nothing needs the whole file in memory and a partial result can be acted on before the rest is read.
 14. Checks, all of them after the above: strict: (13), column types, and whatever else proves wanted; correct and published comes first.
+15. The rule for malformed quoting as an option, its name to be settled, perhaps one with strict: (11): the default, as now; a stricter one, under which a quote that does not open a field raises too, and :double raises on a row which is not what it promises rather than reading it as unspecified quoting does; and a lenient one, never raising, text after a closing quote kept and a field which does not close running to the end of the source.

data/lib/DSV/UnterminatedQuote.rb

@@ -0,0 +1,14 @@
+# DSV/UnterminatedQuote.rb
+# DSV::UnterminatedQuote
+
+class DSV
+  # A quoted field which does not close by the end of the source; line is the one on which its row began.
+  class UnterminatedQuote < ::StandardError
+    attr_reader :line
+
+    def initialize(line)
+      @line = line
+      super("A quoted field in the row beginning on line #{line} does not close.")
+    end
+  end
+end

data/lib/DSV/VERSION.rb

@@ -2,5 +2,5 @@
 # DSV::VERSION
 
 class DSV
-  VERSION = '0.12.7'
+  VERSION = '0.12.9'
 end

data/lib/dsv.rb

@@ -8,6 +8,7 @@ require 'stringio'
 require 'DSV/VERSION'
 require 'DSV/File'
 require 'DSV/String'
+require 'DSV/UnterminatedQuote'
 
 class DSV
 
@@ -180,7 +181,7 @@ class DSV
     else
       read_header
       @rows = []
-      @source.each(@row_separator){|raw_row| @rows << parse_row(complete_quoted_row(raw_row), *selected_columns)}
+      @source.each(@row_separator){|raw_row| @rows << parse_fields(row_fields(raw_row), *selected_columns)}
       @rows
     end
   end
@@ -211,7 +212,7 @@ class DSV
     @columns = (
       if header_row? && ['r', 'r+', 'a+'].include?(@mode) && (first_row = first_row?)
         columns, i = {}, -1
-        split_row(first_row).each do |column_name|
+        (split_row(first_row) || raise(UnterminatedQuote.new(1))).each do |column_name|
           columns[column_name] = columns.key?(column_name) ? [*columns[column_name], i += 1] : (i += 1)
         end
         columns
@@ -263,7 +264,7 @@ class DSV
     value.nil? || value.empty?
   end
 
-  # One row into its fields, by the quote mode: :none splits; :double splits a row as it promises, wholly quoted or wholly unquoted, and one that is neither as unspecified would; unspecified splits and reassembles a quoted field that held the separator, a doubled quote inside it reading as one. Direct string operations throughout; the scanner-based parser is on the branch scanner-parser.
+  # One row into its fields, by the quote mode: :none splits; :double splits a row as it promises, wholly quoted or wholly unquoted, and one that is neither as unspecified would; unspecified splits and reassembles a quoted field that held the separator, a doubled quote inside it reading as one. nil where a quoted field in the row is still open at its end. Direct string operations throughout; the scanner-based parser is on the branch scanner-parser.
   def split_row(raw_row)
     row = raw_row.chomp(@row_separator)
     case @quote
@@ -302,10 +303,10 @@ class DSV
     end
   end
 
-  # The :double contract: a wholly quoted row split on quote-separator-quote, a wholly unquoted one plainly; nil for a row that is neither.
+  # The :double contract: a wholly quoted row split on quote-separator-quote, a wholly unquoted one plainly; nil for a row that is neither, or whose last field does not close.
   def split_double_quoted(row)
     if row.start_with?('"')
-      unescaped_fields(split_on(row.delete_prefix('"').delete_suffix('"'), quoted_separator))
+      unescaped_fields(split_on(row.delete_prefix('"').delete_suffix('"'), quoted_separator)) if trailing_quotes(row, 0).odd?
     else
       split_on(row, @column_separator) unless row.include?('"')
     end
@@ -332,7 +333,7 @@ class DSV
     )
   end
 
-  # Pieces of a row split on the separator, the pieces of a quoted field that held the separator joined back with it, or with the text it matched where the separator is a Regexp; a piece closes a field where it ends in an odd run of quotes, a doubled quote being escaped, and a piece that is a lone quote opens or closes one.
+  # Pieces of a row split on the separator, the pieces of a quoted field that held the separator joined back with it, or with the text it matched where the separator is a Regexp; a piece closes a field where it ends in an odd run of quotes, a doubled quote being escaped, and a piece that is a lone quote opens or closes one; nil where a field is still open after the last piece.
   def reassemble_quoted_fields(pieces, separators = nil)
     fields = []
     buffer = nil
@@ -350,7 +351,7 @@ class DSV
         fields << piece
       end
     end
-    fields
+    fields unless buffer
   end
 
   # How many quotes a piece ends with, counting back to but not including the byte given, so that a field's opening quote is not taken for part of its closing. Read by byte, a quote being 34 in any ASCII-compatible encoding, so that nothing is allocated per character.
@@ -365,30 +366,41 @@ class DSV
     separators ? separators[i - 1] : @column_separator
   end
 
-  # A quoted field may hold the row separator: while the quotes in a row are unbalanced the next line belongs to it, in every quote mode which reads quotes as structure, :none and :unquoted reading them as data.
-  def complete_quoted_row(raw_row)
-    raw_row << @source.gets(@row_separator).to_s while @quote != :none && @quote != :unquoted && raw_row.count('"').odd? && !@source.eof?
-    raw_row
+  # A row's fields, the next line taken into the row while a quoted field in it is open, a quoted field being able to hold the row separator; a field still open at the end of the source raises UnterminatedQuote.
+  def row_fields(raw_row)
+    raw_row << @source.gets(@row_separator) until (fields = split_row(raw_row)) || @source.eof?
+    fields || raise(UnterminatedQuote.new(line_of(raw_row)))
+  end
+
+  # The line on which a row read up to the source's position began, counted only when an error wants it, by reading the source again up to the row, so that no row which closes pays for it.
+  def line_of(raw_row)
+    start = @source.pos - raw_row.bytesize
+    @source.rewind
+    @source.read(start).to_s.scan(@row_separator).length + 1
   end
 
   def parse_row(raw_row, *selected_columns)
+    parse_fields(split_row(raw_row) || raise(UnterminatedQuote.new(1)), *selected_columns)
+  end
+
+  def parse_fields(fields, *selected_columns)
     parsed_row = {}
     i = -1
     if selected_columns.empty?
       if blank?(@columns)
-        split_row(raw_row).each{|column_value| parsed_row[i += 1] = column_value}
+        fields.each{|column_value| parsed_row[i += 1] = column_value}
       elsif repeated_names.empty?
-        split_row(raw_row).each{|column_value| parsed_row[attributes[i += 1]] = column_value}
+        fields.each{|column_value| parsed_row[attributes[i += 1]] = column_value}
       else
-        split_row(raw_row).each{|column_value| store_field(parsed_row, attributes[i += 1], column_value)}
+        fields.each{|column_value| store_field(parsed_row, attributes[i += 1], column_value)}
       end
     else
       selected_columns.flatten!
       case selected_columns[0]
       when Integer
-        split_row(raw_row).each{|column_value| parsed_row[i] = column_value unless !selected_columns.include?(i += 1)}
+        fields.each{|column_value| parsed_row[i] = column_value unless !selected_columns.include?(i += 1)}
       else
-        split_row(raw_row).each{|column_value| store_field(parsed_row, attributes[i], column_value) unless !selected_columns.include?(attributes[i += 1])}
+        fields.each{|column_value| store_field(parsed_row, attributes[i], column_value) unless !selected_columns.include?(attributes[i += 1])}
       end
     end
     if @as_array

data/test/dsv_test.rb

@@ -554,7 +554,7 @@ describe DSV do
     end
   end
 
-  # Input which is not RFC 4180, read to one rule provisionally, until a policy for malformed quoting is decided: a quote is structure only where it opens a field, and a field it opens which does not close raises UnterminatedQuote.
+  # Input which is not RFC 4180, read to the default rule: a quote is structure only where it opens a field, and a field it opens which does not close raises UnterminatedQuote.
   describe "quoting on read, where the input is malformed" do
     it "reads a quote which does not open a field as data" do
       _(DSV.parse_line('a"b"c,d')).must_equal ['a"b"c', 'd']
@@ -562,15 +562,28 @@ describe DSV do
     end
 
     it "reads a stray quote in an unquoted field as data, without taking in the next line" do
-      skip "Observation: the quotes in a row are counted, so a stray quote leaves them unbalanced and the next line is taken into the row; unspecified since 0.11.4, :double since 0.12.5"
       _(DSV.read("12\" pizza,9.99\nnext,row\n", as_array: true)).must_equal [['12" pizza', '9.99'], ['next', 'row']]
       _(DSV.read("12\" pizza,9.99\nnext,row\n", as_array: true, quote: :double)).must_equal [['12" pizza', '9.99'], ['next', 'row']]
     end
 
+    it "reads a quoted field across lines after a stray quote in the same row" do
+      _(DSV.read("a\"b,\"c\nd\"\n", as_array: true)).must_equal [['a"b', "c\nd"]]
+      _(DSV.read("a\"b,\"c\nd\"\n", as_array: true, quote: :double)).must_equal [['a"b', "c\nd"]]
+    end
+
     it "raises UnterminatedQuote on a quoted field which does not close" do
-      skip "Observation: a quoted field which does not close is dropped silently, with the rest of its row"
       _{DSV.read("\"abc,d\n")}.must_raise DSV::UnterminatedQuote
       _{DSV.read("\"ab\"c,d\n")}.must_raise DSV::UnterminatedQuote
+      _{DSV.read("x,\"abc\n", quote: :double)}.must_raise DSV::UnterminatedQuote
+      _{DSV.parse_line('x,"abc')}.must_raise DSV::UnterminatedQuote
+    end
+
+    it "names the line on which the row holding the unterminated field began" do
+      error = _{DSV.read("a,b\n\"c\nd\",e\nf,\"g\nh\n")}.must_raise DSV::UnterminatedQuote
+      _(error.line).must_equal 4
+      _(error.message).must_include 'line 4'
+      error = _{DSV.read("name,note\nAda,\"unclosed\n", headers: true)}.must_raise DSV::UnterminatedQuote
+      _(error.line).must_equal 2
     end
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dsv
 version: !ruby/object:Gem::Version
-  version: 0.12.7
+  version: 0.12.9
 platform: ruby
 authors:
 - thoran
@@ -67,6 +67,7 @@ files:
 - dsv.gemspec
 - lib/DSV/File.rb
 - lib/DSV/String.rb
+- lib/DSV/UnterminatedQuote.rb
 - lib/DSV/VERSION.rb
 - lib/dsv.rb
 - test/VERSION_test.rb