dsv 0.12.8 → 0.12.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 190e66c0b25b64c18dac0b955ae59fe64591d14d369cafa950d31b1aa56a2b30
-  data.tar.gz: 6a7ea034b8f8da40716efb89a555951307ee312c38142b84c71f2712c57e0622
+  metadata.gz: f88116d5530f25575975779af46933ddd7c67af651b2c61a3894b237b000950a
+  data.tar.gz: cd03ed4c4b47c12044387544346d8be13ac3f256f1263f135302134bf1ce429d
 SHA512:
-  metadata.gz: e6f6cb3c8f04131f26bf26be33f7dc1e000a4e39a58537f3c327c250837d880477b07e478be8688a57b124dfd85f24f92d569109de15e95ec61b95631652e493
-  data.tar.gz: f0787b59e38dbac938fcd7474b815e24d2660c33438289a9720ca239b439a3d7435ce650db9dd5279b2ae2266ccfe763d1dfe8005d92cdb2dd1a8c1369f16cb8
+  metadata.gz: 0a6e2477a6ff620bba7e5317af9ec1482f0effa853f3ac6f6074e1460170f201b0649db16c1ba67b87621546b986948678d4f6610f2623e29766cb9d948c227d
+  data.tar.gz: b9ad0b92bcf1ec1caee1604217005250411ac318014ceafb9384ceb70ae276f5d2e457e097a287dcbb5c9d9b668144e8a2c35194d9e7e2a9d28377c5aa3bf8b9

data/CHANGELOG

@@ -1,7 +1,27 @@
 # dsv/CHANGELOG
 
+## 20261103
+
+0.12.10: ~ TODO, now grouped by release; + ROADMAP.md.
+
+1. ~ TODO: the jobs decided, by release: the fixes left for 0.12.x; modes as declarations in 0.13.0; streaming in 0.14.0; what 1.0 needs; speed and new options in 1.x.
+2. ~ TODO: 11, strict:, and 15, the malformed-quoting option, superseded by field_count: and :rfc_4180; 13, chunks, by a streaming each; 1, 6, 7 and the undecided parts of 2, to ROADMAP.md.
+3. + ROADMAP.md: the design, the boundary for 1.0, the version rule, the release sequence, and the themes after 1.0 with what is undecided in each.
+4. ~ lib/DSV/VERSION.rb: /0.12.9/0.12.10/.
+
+
 ## 20261028
 
+0.12.9: + DSV::UnterminatedQuote, raised where a quoted field does not close.
+
+1. + lib/DSV/UnterminatedQuote.rb: DSV::UnterminatedQuote, with line, the line on which the field's row began.
+2. ~ row_fields(), parse_row(), columns(): raise it where a quoted field is still open at the end of the source, or of the line or header given. Such a field had been dropped silently, with its row.
+3. + line_of(): counts the lines before the row only when raising.
+4. ~ test/dsv_test.rb: - the skip on a quoted field which does not close; + the line it names; the malformed examples now state the default rule.
+5. ~ README.md: Quoting, the rule for input which isn't RFC 4180.
+6. ~ TODO: 12, as UnterminatedQuote finds its line; + 15, the rule as an option.
+7. ~ lib/DSV/VERSION.rb: /0.12.8/0.12.9/.
+
 0.12.8: a stray quote no longer takes in the next line.
 
 1. /complete_quoted_row()/row_fields()/: takes the next line into a row while a quoted field in it is open, rather than while its quotes are odd in number. A stray quote had made the count odd, and one beside an open field even.

data/README.md

@@ -89,6 +89,8 @@ DSV.read('quoted.csv', headers: true, quote: :double)  # every field is quoted,
 
 Under `:double` a row that is wholly unquoted, as a header row often is, is split plainly, and a row that is neither wholly quoted nor wholly unquoted is read as it would be with no mode named, so naming the mode never costs a correct read.
 
+Input which isn't RFC 4180 is read to one rule. A quote opens a field only at the start of one, so a quote anywhere else, as in `12" pizza`, is data. A quoted field closes only at a quote followed by the separator or the end of the row, so one which doesn't close by the end of the source, as `"abc` or `"ab"c` doesn't, raises `DSV::UnterminatedQuote`, whose `line` is the line its row began on.
+
 ### Writing
 
 Set the columns, give the rows as Hashes, and write:

data/TODO

@@ -1,24 +1,46 @@
 # TODO
 
-## 0.12.x, before the gem is published
+## 0.12.x, fixes
 
-1. A path is told from content by the call, not by a guess: .read, .foreach, .open and the other path-taking class methods take a path, .parse takes text, new keeps the guess with a switch to say which, and a path under a read mode that names no file raises rather than being read as text.
-2. Selection as a keyword on the calls, only: or select:, columns: being taken; the positional form kept or retired; selected_columns: renamed to match.
-3. The instance-level header_row accessor renamed headers?, the class-level header_row now being the row.
-4. << to append a row and generate to build a string, the two writing calls a CSV user reaches for first.
+1. The class methods close what they open, an early exit included, and without a block return an Enumerator which opens the source afresh on each pass, as CSV.foreach does; collect, select, reject and detect without a block no longer raise.
+2. as_array: with a column selection: each('name') no longer raises TypeError, and read('name', as_array: true) gives only the columns selected. as_array: in the README, as the way to get Arrays as CSV gives them.
+3. Reading honours quote: given as a String, as writing does; DSV.parse_line honours quote:.
+4. A space as the column separator splits on each space, not as String#split(' ') does.
+5. :double back to 0.12.4's reading, undoing 0.12.5 item 2: a row not starting with a quote split plainly, one starting with a quote split on quote-separator-quote, and one which leaves a lone quote in a field read in full as wholly quoted. The README back to 0.12.3's wording.
+6. :single read, as :double is with a single quote.
+7. filter, a class-level alias of select.
+8. A String under a write mode taken as a path, so that a file not yet there can be written by name.
+9. The internals private.
 
-## 0.13.0, speed
+## 0.13.0, modes as declarations
 
-5. The saved-up speed change, with the evidence in the reconciliation: two string predicates in place of four regex matches per piece in split_csv; rows built by zip.to_h; a benchmark script in the repository against stdlib CSV, and the README's speed sentence written from its output. The scanner-based parser on the branch scanner-parser is judged here.
+10. An unknown quote mode, file mode or option raises ArgumentError, where each is ignored today.
+11. :mixed, the name of the default.
+12. A mode raises on a contradiction which costs nothing to see, every row alike, header included: under :double, a row not starting with a quote, and a bare field in a row read in full. Declaring :mixed reads such a row.
 
-## Features, after
+## 0.14.0, streaming
 
-6. Guessing, as a mode that runs once at construction on a sample of the first rows and feeds the same chooser as an explicit option, scanner only: the column separator, the one whose count is the same on every sampled row; whether the first row is a header, its columns' types differing from the rows beneath in the same way across columns; and, where there is no header, a header made from the data, each column named for what its values look like, integer, decimal, date, email, URL, phone, with a positional fallback, column_0, for text and a suffix where two columns share a class. Names come from what values look like, not what they mean; anything more is a schema, and another library's.
-7. Templated ingestion: a template: option taking a whole-row Regexp with a capture per field, the group names being the column names, one match per row; an Array of separators as its writable special case, compiled into such a Regexp at construction and invertible for writing where a general template is not.
-8. The duplicate-header modes as an option: keep all as an Array, the default; index, a, a.1; first; last; error. All derivations of the same columns map at header time, none costing anything per row.
-9. Writing options: minimal quoting, a field quoted only where it holds a separator, a quote or a row terminator; a row terminator inside a field kept, quoted, or folded to a space; the encoding declared and a transcode offered with any loss marked.
-10. Symbols and Strings interchangeable for column names, in a selection, a columns: list and a row's keys; today a String is required.
-11. A strict: option, off by default, that raises on a row with more or fewer fields than the header; today a short row gets only the columns it has and a long row's excess lands under nil.
-12. A line number in each parse error, UnterminatedQuote first, found only when the error is raised, from the source's position at that moment; never counted as it goes, since counting costs time on every row for the sake of the rare one.
-13. Chunked processing of a large file: a bounded number of rows at a time handed to a block, so that nothing needs the whole file in memory and a partial result can be acted on before the rest is read.
-14. Checks, all of them after the above: strict: (13), column types, and whatever else proves wanted; correct and published comes first.
+13. each yields each row as it is read and returns its receiver, so that detect and break stop reading and memory does not grow with the file; rows is no longer filled by each. Chunks come from each_slice. A source read once, start to end, with no rewind, so that a pipe or $stdin can be read: the header from the first row read, and the line of an error found by seeking only where the source can; a rewind left to r+ rewriting and first_row called alone.
+
+## 1.0
+
+14. A corpus in the tests, a day of NEMWEB's dispatch files trimmed and the RFC 4180 cases, each checked against stdlib CSV's rows.
+15. A benchmark script against stdlib CSV, and the README's speed sentence written from its output.
+16. README: every mode by example, and a table matching each to stdlib CSV's and Python's settings.
+
+## 1.x, speed, results unchanged
+
+17. A per-row fast path for the default: split, the fields bare or wrapped in two quotes unwrapped, any other row read in full.
+18. as_array: returning the split fields, rather than building a Hash and then an Array from it.
+19. The splitter chosen once, at construction, as row_renderer is for writing.
+20. Two string predicates in place of four regex matches per piece; rows built by zip.to_h; the scanner-based parser, at the tag scanner-parser-20261006, judged against the rest.
+
+## 1.x, features
+
+21. :rfc_4180, a quote mode reading what RFC 4180 allows as :mixed does and raising on what it forbids.
+22. field_count:, :header or an Integer, off by default: a row of another width raises.
+23. Writing: minimal quoting, a field quoted only where it holds a separator, a quote or a row terminator; a row terminator inside a field kept, quoted, or folded to a space; the encoding declared and a transcode offered with any loss marked.
+24. A line number in each parse error, found only when the error is raised, as UnterminatedQuote finds its, where the source can seek. line_numbers:, off by default, counts lines as it reads instead, for any source a pipe included: the read loop chosen once, at construction, as row_renderer is for writing, so that without it no row pays for the count.
+25. The instance-level header_row renamed headers?, kept as an alias; << to append a row and generate to build a string.
+26. Symbols and Strings interchangeable for column names, in a selection, a columns: list and a row's keys.
+27. The duplicate-header modes as an option: keep all as an Array, the default; index, a, a.1; first; last; error.

data/lib/DSV/UnterminatedQuote.rb

@@ -0,0 +1,14 @@
+# DSV/UnterminatedQuote.rb
+# DSV::UnterminatedQuote
+
+class DSV
+  # A quoted field which does not close by the end of the source; line is the one on which its row began.
+  class UnterminatedQuote < ::StandardError
+    attr_reader :line
+
+    def initialize(line)
+      @line = line
+      super("A quoted field in the row beginning on line #{line} does not close.")
+    end
+  end
+end

data/lib/DSV/VERSION.rb

@@ -2,5 +2,5 @@
 # DSV::VERSION
 
 class DSV
-  VERSION = '0.12.8'
+  VERSION = '0.12.10'
 end

data/lib/dsv.rb

@@ -8,6 +8,7 @@ require 'stringio'
 require 'DSV/VERSION'
 require 'DSV/File'
 require 'DSV/String'
+require 'DSV/UnterminatedQuote'
 
 class DSV
 
@@ -211,7 +212,7 @@ class DSV
     @columns = (
       if header_row? && ['r', 'r+', 'a+'].include?(@mode) && (first_row = first_row?)
         columns, i = {}, -1
-        (split_row(first_row) || []).each do |column_name|
+        (split_row(first_row) || raise(UnterminatedQuote.new(1))).each do |column_name|
           columns[column_name] = columns.key?(column_name) ? [*columns[column_name], i += 1] : (i += 1)
         end
         columns
@@ -365,14 +366,21 @@ class DSV
     separators ? separators[i - 1] : @column_separator
   end
 
-  # A row's fields, the next line taken into the row while a quoted field in it is open, a quoted field being able to hold the row separator; a field still open at the end of the source is dropped, with its row.
+  # A row's fields, the next line taken into the row while a quoted field in it is open, a quoted field being able to hold the row separator; a field still open at the end of the source raises UnterminatedQuote.
   def row_fields(raw_row)
     raw_row << @source.gets(@row_separator) until (fields = split_row(raw_row)) || @source.eof?
-    fields || []
+    fields || raise(UnterminatedQuote.new(line_of(raw_row)))
+  end
+
+  # The line on which a row read up to the source's position began, counted only when an error wants it, by reading the source again up to the row, so that no row which closes pays for it.
+  def line_of(raw_row)
+    start = @source.pos - raw_row.bytesize
+    @source.rewind
+    @source.read(start).to_s.scan(@row_separator).length + 1
   end
 
   def parse_row(raw_row, *selected_columns)
-    parse_fields(split_row(raw_row) || [], *selected_columns)
+    parse_fields(split_row(raw_row) || raise(UnterminatedQuote.new(1)), *selected_columns)
   end
 
   def parse_fields(fields, *selected_columns)

data/test/dsv_test.rb

@@ -554,7 +554,7 @@ describe DSV do
     end
   end
 
-  # Input which is not RFC 4180, read to one rule provisionally, until a policy for malformed quoting is decided: a quote is structure only where it opens a field, and a field it opens which does not close raises UnterminatedQuote.
+  # Input which is not RFC 4180, read to the default rule: a quote is structure only where it opens a field, and a field it opens which does not close raises UnterminatedQuote.
   describe "quoting on read, where the input is malformed" do
     it "reads a quote which does not open a field as data" do
       _(DSV.parse_line('a"b"c,d')).must_equal ['a"b"c', 'd']
@@ -572,9 +572,18 @@ describe DSV do
     end
 
     it "raises UnterminatedQuote on a quoted field which does not close" do
-      skip "Observation: a quoted field which does not close is dropped silently, with its row"
       _{DSV.read("\"abc,d\n")}.must_raise DSV::UnterminatedQuote
       _{DSV.read("\"ab\"c,d\n")}.must_raise DSV::UnterminatedQuote
+      _{DSV.read("x,\"abc\n", quote: :double)}.must_raise DSV::UnterminatedQuote
+      _{DSV.parse_line('x,"abc')}.must_raise DSV::UnterminatedQuote
+    end
+
+    it "names the line on which the row holding the unterminated field began" do
+      error = _{DSV.read("a,b\n\"c\nd\",e\nf,\"g\nh\n")}.must_raise DSV::UnterminatedQuote
+      _(error.line).must_equal 4
+      _(error.message).must_include 'line 4'
+      error = _{DSV.read("name,note\nAda,\"unclosed\n", headers: true)}.must_raise DSV::UnterminatedQuote
+      _(error.line).must_equal 2
     end
   end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dsv
 version: !ruby/object:Gem::Version
-  version: 0.12.8
+  version: 0.12.10
 platform: ruby
 authors:
 - thoran
@@ -67,6 +67,7 @@ files:
 - dsv.gemspec
 - lib/DSV/File.rb
 - lib/DSV/String.rb
+- lib/DSV/UnterminatedQuote.rb
 - lib/DSV/VERSION.rb
 - lib/dsv.rb
 - test/VERSION_test.rb