smarter_csv 1.17.0 → 1.17.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 702bd7049e83c0beb85f0ca11a122e6f1659eddef6afec66eaf1c37c5b30f43f
-  data.tar.gz: dd1915694d041c9b631324de7408f46fc8f426f9e1c60136c35a8f1e754d4590
+  metadata.gz: 2e665f0dc98db44950aa9cbb2cac430068e91df8886062068413dfbcefc74fc3
+  data.tar.gz: def43fb66886b16ec13bd429b4fd6923b09aa1a01757a696390a38c18b59fa31
 SHA512:
-  metadata.gz: fa00d07c21cffa711a43ecb4622ad3a09b667f1c1965ad26bee864ada6a3c168076ec04550781c75c4f0acbb28fcab60001278f459cf3065cceaef6820764e30
-  data.tar.gz: eab835a356e5343e20a5cc0784ffd9aafa8ab631256d412ec570a0192060b8f3e9c6f619db36e59494364f6c51cda16a9c434c2565ef5a5b6e23f4813a7eaaef
+  metadata.gz: 2fb7793ed4eca64cfef1f7dd82a417b44988832280b373c1748213d9f7c879cd0a2d17c4e3b72c82be6acedb01b0fec26b70e6daaefb645ee2c3bf64b7aedcd8
+  data.tar.gz: a0b8842d5a69d8526af81d4e2a64c31fd6a54d6d610c5ce0dbb16298dfd03c3d296546c049a4a76a710908390ec1ea1739bc7530b1ab652c7ead1ceaa02b431d

data/CHANGELOG.md

@@ -1,7 +1,25 @@
 
 # SmarterCSV 1.x Change Log
 
-## 1.17.0 (NOT RELEASED)
+## 1.17.2 (2026-05-21)
+
+RSpec tests: **2,220→ 2,274** (+54 tests)
+
+### Bug Fix
+
+  - fixed [Issue #334](https://github.com/tilo/smarter_csv/issues/334) with escaped double quote followed by comma. Thanks to [conorg](https://github.com/conorg)
+  - fixed bug when using `headers: { except: }`
+  - added more tests
+
+## 1.17.1 (2026-05-17)
+
+RSpec tests: **2,210→ 2,220** (+10 tests)
+
+### Bug Fix
+
+  - fixing issue with `remove_empty_hashes: false` not being honored in accelerated path (does not affect you when you use default settings)
+
+## 1.17.0 (2026-05-14)
 
 RSpec tests: **1,434 → 2,210** (+776 tests)
 
@@ -55,6 +73,13 @@ Measured against 1.16.4 (Apple M4, Ruby 3.4.7):
 
 Per-file breakdown: [`docs/releases/1.17.0/performance_notes.md`](docs/releases/1.17.0/performance_notes.md).
 
+## 1.16.5 (2026-05-17)
+
+### Bug Fix
+
+  - fixing issue with `remove_empty_hashes: false` not being honored in accelerated path (does not affect you when you use default settings)
+
+
 ## 1.16.4 (2026-04-21) — Bug Fixes
 
 RSpec tests: **1,434 → 1,467** (+33 tests)
@@ -206,6 +231,11 @@ Measured on 19 benchmark files, Apple M1, Ruby 3.4.7. See [benchmarks](docs/rele
  * **Writer temp file** no longer hardcoded to `/tmp` (fixes Windows); properly cleaned up with `Tempfile#close!`.
  * **Writer `StringIO`**: `finalize` no longer attempts to close a caller-owned `StringIO`.
 
+## 1.15.3 (2026-05-17)
+
+### Bug Fix
+
+  - fixing issue with `remove_empty_hashes: false` not being honored in accelerated path (does not affect you when you use default settings)
 
 ## 1.15.2 (2026-02-20)
 

data/CONTRIBUTORS.md

@@ -1,4 +1,4 @@
-# A Big Thank You to all 63 Contributors!!
+# A Big Thank You to all 64 Contributors!!
 
 
 A Big Thank you to everyone who filed issues, sent comments, and who contributed with pull requests:
@@ -66,3 +66,4 @@ A Big Thank you to everyone who filed issues, sent comments, and who contributed
  * [Dom Lebron](https://github.com/biglebronski)
  * [Paho Lurie-Gregg](https://github.com/paholg)
  * [Jonas Staškevičius](https://github.com/pirminis)
+ * [conorg](https://github.com/conorg)

data/README.md

@@ -1,8 +1,8 @@
 
 # SmarterCSV
 
-  ![Gem Version](https://img.shields.io/gem/v/smarter_csv) [![codecov](https://codecov.io/gh/tilo/smarter_csv/branch/main/graph/badge.svg?token=1L7OD80182)](https://codecov.io/gh/tilo/smarter_csv) [View on RubyGems](https://rubygems.org/gems/smarter_csv) [View on RubyToolbox](https://www.ruby-toolbox.com/search?q=smarter_csv)
-
+![Gem Version](https://img.shields.io/gem/v/smarter_csv) [![codecov](https://codecov.io/gh/tilo/smarter_csv/branch/main/graph/badge.svg?token=1L7OD80182)](https://codecov.io/gh/tilo/smarter_csv) [![Downloads](https://img.shields.io/gem/dt/smarter_csv)](https://rubygems.org/gems/smarter_csv) [![RubyGems](https://img.shields.io/badge/RubyGems-smarter__csv-brightgreen?logo=rubygems&logoColor=white)](https://rubygems.org/gems/smarter_csv) [![Ruby Toolbox](https://img.shields.io/badge/Ruby%20Toolbox-smarter__csv-brightgreen)](https://www.ruby-toolbox.com/projects/smarter_csv)
+        
   SmarterCSV is a high-performance CSV ingestion and generation for Ruby, focused on fast end-to-end CSV ingestion of real-world data — no silent failures, no surprises, not just tokenization.
 
   ⭐ If SmarterCSV saved you hours of import time, please star the repo, and consider sponsoring this project.
@@ -311,7 +311,7 @@ Or install it yourself as:
   * [Examples](docs/examples.md)
   * [Real-World CSV Files](docs/real_world_csv.md)
   * [SmarterCSV over the Years](docs/history.md)
-  * [Release Notes](docs/releases/1.16.0/changes.md)
+  * [Release Notes](docs/releases/1.17.0/changes.md)
 
 ## Articles
   * [Parsing CSV Files in Ruby with SmarterCSV](https://tilo-sloboda.medium.com/parsing-csv-files-in-ruby-with-smartercsv-6ce66fb6cf38)
@@ -333,7 +333,7 @@ For reporting issues, please:
   * open a pull-request adding a test that demonstrates the issue
   * mention your version of SmarterCSV, Ruby, Rails
 
-# [A Special Thanks to all 63 Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
+# [A Special Thanks to all 64 Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
 
 
 ## Contributing

data/ext/smarter_csv/smarter_csv.c

@@ -331,25 +331,40 @@ static VALUE rb_parse_csv_line(VALUE self, VALUE line, VALUE col_sep, VALUE quot
           if (!allow_escaped_quotes || backslash_count % 2 == 0) {
             if (__builtin_expect(quote_boundary_standard, 1)) {
               if (in_quotes) {
-                // closing quote: only valid if followed by col_sep, row_sep, or end of line
-                bool valid_close = (p + 1 >= endP);
-                if (!valid_close) {
-                  valid_close = true;
-                  for (long j = 0; j < col_sep_len; j++) {
-                    if (*(p + 1 + j) != *(col_sepP + j)) { valid_close = false; break; }
+                if (p + 2 < endP && *(p + 1) == quote_char_val) {
+                  /* RFC doubled quote inside a quoted field ("" → ").
+                   * Give this precedence over the closing-quote check, but only
+                   * when another byte follows the doubled pair.
+                   *
+                   * Compatibility note: we intentionally do NOT force terminal
+                   * "" to be consumed here. SmarterCSV has a long-standing lenient
+                   * behavior for malformed tails like ...\"" in :double_quotes mode:
+                   * the final quote may still close the field instead of turning the
+                   * row into an unclosed-quote error. Issue #334 needs doubled-quote
+                   * precedence for ..."",... (more content follows), but we keep the
+                   * historical leniency for terminal ..."". */
+                  p++;
+                } else {
+                  // closing quote: only valid if followed by col_sep, row_sep, or end of line
+                  bool valid_close = (p + 1 >= endP);
+                  if (!valid_close) {
+                    valid_close = true;
+                    for (long j = 0; j < col_sep_len; j++) {
+                      if (*(p + 1 + j) != *(col_sepP + j)) { valid_close = false; break; }
+                    }
                   }
-                }
-                if (!valid_close && row_sep_len > 0) {
-                  valid_close = true;
-                  for (long j = 0; j < row_sep_len; j++) {
-                    if (*(p + 1 + j) != *(row_sepP + j)) { valid_close = false; break; }
+                  if (!valid_close && row_sep_len > 0) {
+                    valid_close = true;
+                    for (long j = 0; j < row_sep_len; j++) {
+                      if (*(p + 1 + j) != *(row_sepP + j)) { valid_close = false; break; }
+                    }
                   }
+                  if (valid_close) {
+                    in_quotes = false;
+                    field_started = true;
+                  }
+                  // else: quote inside quoted field → literal
                 }
-                if (valid_close) {
-                  in_quotes = false;
-                  field_started = true;
-                }
-                // else: quote inside quoted field → literal (handles "" doubling)
               } else if (!field_started) {
                 in_quotes = true;     // opening quote at field boundary
                 field_started = true;
@@ -829,6 +844,11 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash(VALUE self, VALUE line,
    * the frame stays well below 4 KB and ___chkstk_darwin never fires on ARM64 macOS.
    */
   bool *keep_bitmap = NULL;
+  /* In THIS (non-ctx) function the bitmap is alloca'd to headers_len on every call (see the alloca
+   * sites below), so keep_bitmap[] is exactly headers_len long and headers_len is the correct bound
+   * at all access sites. Do NOT mirror rb_parse_line_to_hash_ctx's keep_bitmap_len here: that variant
+   * caches its bitmap across rows (where @headers can grow), so it must use the captured length; this
+   * one rebuilds per call and does not. */
   bool keep_extra_columns = true; /* extra cols (> headers_len): keep by default */
   bool has_only = false;          /* true when only_headers: filtering is active */
   long early_exit_after = -1;     /* column index after which we stop; -1 = no early exit */
@@ -1147,25 +1167,40 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash(VALUE self, VALUE line,
             if (!allow_escaped_quotes || backslash_count % 2 == 0) {
               if (__builtin_expect(quote_boundary_standard, 1)) {
                 if (in_quotes) {
-                  // closing quote: only valid if followed by col_sep, row_sep, or end of line
-                  bool valid_close = (p + 1 >= endP);
-                  if (!valid_close) {
-                    valid_close = true;
-                    for (long j = 0; j < col_sep_len; j++) {
-                      if (*(p + 1 + j) != *(col_sepP + j)) { valid_close = false; break; }
+                  if (p + 2 < endP && *(p + 1) == quote_char_val) {
+                    /* RFC doubled quote inside a quoted field ("" → ").
+                     * Give this precedence over the closing-quote check, but only
+                     * when another byte follows the doubled pair.
+                     *
+                     * Compatibility note: we intentionally do NOT force terminal
+                     * "" to be consumed here. SmarterCSV has a long-standing lenient
+                     * behavior for malformed tails like ...\"" in :double_quotes mode:
+                     * the final quote may still close the field instead of turning the
+                     * row into an unclosed-quote error. Issue #334 needs doubled-quote
+                     * precedence for ..."",... (more content follows), but we keep the
+                     * historical leniency for terminal ..."". */
+                    p++;
+                  } else {
+                    // closing quote: only valid if followed by col_sep, row_sep, or end of line
+                    bool valid_close = (p + 1 >= endP);
+                    if (!valid_close) {
+                      valid_close = true;
+                      for (long j = 0; j < col_sep_len; j++) {
+                        if (*(p + 1 + j) != *(col_sepP + j)) { valid_close = false; break; }
+                      }
                     }
-                  }
-                  if (!valid_close && row_sep_len2 > 0) {
-                    valid_close = true;
-                    for (long j = 0; j < row_sep_len2; j++) {
-                      if (*(p + 1 + j) != *(row_sepP2 + j)) { valid_close = false; break; }
+                    if (!valid_close && row_sep_len2 > 0) {
+                      valid_close = true;
+                      for (long j = 0; j < row_sep_len2; j++) {
+                        if (*(p + 1 + j) != *(row_sepP2 + j)) { valid_close = false; break; }
+                      }
                     }
+                    if (valid_close) {
+                      in_quotes = false;
+                      field_started = true;
+                    }
+                    // else: quote inside quoted field → literal
                   }
-                  if (valid_close) {
-                    in_quotes = false;
-                    field_started = true;
-                  }
-                  // else: quote inside quoted field → literal (handles "" doubling)
                 } else if (!field_started) {
                   in_quotes = true;     // opening quote at field boundary
                   field_started = true;
@@ -1242,12 +1277,20 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash(VALUE self, VALUE line,
    * return nil instead of the hash so the row can be skipped.
    * With lazy allocation, if all_blank is true, xform.hash is still Qnil —
    * no hash was ever allocated.
+   *
+   * If remove_empty_hashes is disabled, preserve the row as an empty hash.
+   * This keeps parity with the Ruby path without impacting the normal
+   * non-blank hot path.
    */
-  if (remove_empty && all_blank) {
-    VALUE result = rb_ary_new_capa(2);
-    rb_ary_push(result, Qnil);
-    rb_ary_push(result, LONG2FIX(element_count));
-    return result;
+  if (all_blank) {
+    if (remove_empty) {
+      VALUE result = rb_ary_new_capa(2);
+      rb_ary_push(result, Qnil);
+      rb_ary_push(result, LONG2FIX(element_count));
+      return result;
+    }
+
+    ensure_hash_allocated(&xform);
   }
 
   /* ----------------------------------------
@@ -1487,6 +1530,14 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
   int  numeric_mode        = ctx->numeric_mode;
   VALUE numeric_keys       = ctx->numeric_keys;
   bool *keep_bitmap         = ctx->keep_bitmap;
+  /* keep_bitmap is cached in the context (xmalloc'd once at construction, sized to the header count
+   * THEN). @headers can grow in place as undeclared extra columns appear, so the live headers_len
+   * (re-read each call below) may exceed the bitmap's length. Every keep_bitmap[] access in this
+   * function MUST be bounded by keep_bitmap_len, never headers_len — indices past the bitmap are
+   * extra columns and follow keep_extra_columns. Bounding by the grown headers_len was an
+   * out-of-bounds heap read (the bug). The sibling rb_parse_line_to_hash safely uses headers_len
+   * because it re-allocs its bitmap to headers_len on every call. */
+  long  keep_bitmap_len     = ctx->keep_bitmap_len;
   bool  keep_extra_columns  = ctx->keep_extra_columns;
   long  early_exit_after    = ctx->early_exit_after;
 
@@ -1588,7 +1639,7 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
           while (trim_end >= trim_start && (*trim_end == ' ' || *trim_end == '\t')) trim_end--;
         }
         long trimmed_len = (trim_end >= trim_start) ? (trim_end - trim_start + 1) : 0;
-        if (!keep_bitmap || (element_count < headers_len ? keep_bitmap[element_count] : keep_extra_columns)) {
+        if (!keep_bitmap || (element_count < keep_bitmap_len ? keep_bitmap[element_count] : keep_extra_columns)) {
           if (insert_field_into_hash(&xform, trim_start, trimmed_len, element_count, false, quote_char_val, encoding))
             all_blank = false;
         }
@@ -1609,7 +1660,7 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
           while (trim_end >= trim_start && (*trim_end == ' ' || *trim_end == '\t')) trim_end--;
         }
         long trimmed_len = (trim_end >= trim_start) ? (trim_end - trim_start + 1) : 0;
-        if (!keep_bitmap || (element_count < headers_len ? keep_bitmap[element_count] : keep_extra_columns)) {
+        if (!keep_bitmap || (element_count < keep_bitmap_len ? keep_bitmap[element_count] : keep_extra_columns)) {
           if (insert_field_into_hash(&xform, trim_start, trimmed_len, element_count, false, quote_char_val, encoding))
             all_blank = false;
         }
@@ -1672,7 +1723,7 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
 
         bool has_embedded_quotes = quoted || (trimmed_len > 0 && memchr(trim_start, quote_char_val, trimmed_len));
 
-        if (!keep_bitmap || (element_count < headers_len ? keep_bitmap[element_count] : keep_extra_columns)) {
+        if (!keep_bitmap || (element_count < keep_bitmap_len ? keep_bitmap[element_count] : keep_extra_columns)) {
           if (insert_field_into_hash(&xform, trim_start, trimmed_len, element_count, has_embedded_quotes, quote_char_val, encoding))
             all_blank = false;
         }
@@ -1706,25 +1757,40 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
             if (!allow_escaped_quotes || backslash_count % 2 == 0) {
               if (__builtin_expect(quote_boundary_standard, 1)) {
                 if (in_quotes) {
-                  /* closing quote: only valid if followed by col_sep, row_sep, or end */
-                  bool valid_close = (p + 1 >= endP);
-                  if (!valid_close) {
-                    valid_close = true;
-                    for (long j = 0; j < col_sep_len; j++) {
-                      if (*(p + 1 + j) != *(col_sepP + j)) { valid_close = false; break; }
+                  if (p + 2 < endP && *(p + 1) == quote_char_val) {
+                    /* RFC doubled quote inside a quoted field ("" → ").
+                     * Give this precedence over the closing-quote check, but only
+                     * when another byte follows the doubled pair.
+                     *
+                     * Compatibility note: we intentionally do NOT force terminal
+                     * "" to be consumed here. SmarterCSV has a long-standing lenient
+                     * behavior for malformed tails like ...\"" in :double_quotes mode:
+                     * the final quote may still close the field instead of turning the
+                     * row into an unclosed-quote error. Issue #334 needs doubled-quote
+                     * precedence for ..."",... (more content follows), but we keep the
+                     * historical leniency for terminal ..."". */
+                    p++;
+                  } else {
+                    /* closing quote: only valid if followed by col_sep, row_sep, or end */
+                    bool valid_close = (p + 1 >= endP);
+                    if (!valid_close) {
+                      valid_close = true;
+                      for (long j = 0; j < col_sep_len; j++) {
+                        if (*(p + 1 + j) != *(col_sepP + j)) { valid_close = false; break; }
+                      }
                     }
-                  }
-                  if (!valid_close && row_sep_len2 > 0) {
-                    valid_close = true;
-                    for (long j = 0; j < row_sep_len2; j++) {
-                      if (*(p + 1 + j) != *(row_sepP2 + j)) { valid_close = false; break; }
+                    if (!valid_close && row_sep_len2 > 0) {
+                      valid_close = true;
+                      for (long j = 0; j < row_sep_len2; j++) {
+                        if (*(p + 1 + j) != *(row_sepP2 + j)) { valid_close = false; break; }
+                      }
                     }
+                    if (valid_close) {
+                      in_quotes     = false;
+                      field_started = true;
+                    }
+                    /* else: quote inside quoted field → literal */
                   }
-                  if (valid_close) {
-                    in_quotes     = false;
-                    field_started = true;
-                  }
-                  /* else: quote inside quoted field → literal (handles "" doubling) */
                 } else if (!field_started) {
                   in_quotes     = true;   /* opening quote at field boundary */
                   field_started = true;
@@ -1783,7 +1849,7 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
 
       bool has_embedded_quotes = quoted || (trimmed_len > 0 && memchr(trim_start, quote_char_val, trimmed_len));
 
-      if (!keep_bitmap || (element_count < headers_len ? keep_bitmap[element_count] : keep_extra_columns)) {
+      if (!keep_bitmap || (element_count < keep_bitmap_len ? keep_bitmap[element_count] : keep_extra_columns)) {
         if (insert_field_into_hash(&xform, trim_start, trimmed_len, element_count, has_embedded_quotes, quote_char_val, encoding))
           all_blank = false;
       }
@@ -1794,11 +1860,15 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
   /* ----------------------------------------
    * SECTION 6: Handle blank rows
    * ---------------------------------------- */
-  if (remove_empty && all_blank) {
-    VALUE result = rb_ary_new_capa(2);
-    rb_ary_push(result, Qnil);
-    rb_ary_push(result, LONG2FIX(element_count));
-    return result;
+  if (all_blank) {
+    if (remove_empty) {
+      VALUE result = rb_ary_new_capa(2);
+      rb_ary_push(result, Qnil);
+      rb_ary_push(result, LONG2FIX(element_count));
+      return result;
+    }
+
+    ensure_hash_allocated(&xform);
   }
 
   /* ----------------------------------------
@@ -1807,7 +1877,7 @@ __attribute__((hot)) static VALUE rb_parse_line_to_hash_ctx(VALUE self, VALUE li
   if (!remove_empty_values) {
     ensure_hash_allocated(&xform);
     for (long i = element_count; i < headers_len; i++) {
-      if (!keep_bitmap || keep_bitmap[i]) {
+      if (!keep_bitmap || (i < keep_bitmap_len ? keep_bitmap[i] : keep_extra_columns)) {
         rb_hash_aset(xform.hash, rb_ary_entry(headers, i), Qnil);
       }
     }

data/lib/smarter_csv/parser.rb

@@ -410,15 +410,28 @@ module SmarterCSV
                 if !allow_escaped_quotes || backslash_count % 2 == 0
                   if quote_boundary_standard
                     if in_quotes
-                      # closing quote: only valid if followed by col_sep, row_sep, or end of line
                       next_i = i + 1
-                      if next_i >= bytesize ||
-                         line.getbyte(next_i) == col_sep_byte ||
-                         (row_sep_bytesize > 0 && line.byteslice(next_i, row_sep_bytesize) == row_sep)
+                      if next_i + 1 < bytesize && line.getbyte(next_i) == quote_byte
+                        # RFC doubled quote inside a quoted field ("" → ").
+                        # Give this precedence over the closing-quote check, but only
+                        # when another byte follows the doubled pair.
+                        #
+                        # Compatibility note: we intentionally do NOT force terminal
+                        # "" to be consumed here. SmarterCSV has a long-standing lenient
+                        # behavior for malformed tails like ...\"" in :double_quotes mode:
+                        # the final quote may still close the field instead of turning the
+                        # row into an unclosed-quote error. Issue #334 needs doubled-quote
+                        # precedence for ..."",... (more content follows), but we keep the
+                        # historical leniency for terminal ..."".
+                        i = next_i
+                      # closing quote: only valid if followed by col_sep, row_sep, or end of line
+                      elsif next_i >= bytesize ||
+                            line.getbyte(next_i) == col_sep_byte ||
+                            (row_sep_bytesize > 0 && line.byteslice(next_i, row_sep_bytesize) == row_sep)
                         in_quotes = false
                         field_started = true
                       end
-                      # else: quote inside quoted field → literal (handles "" doubling)
+                      # else: quote inside quoted field → literal
                     elsif !field_started # at field boundary: open quoted field
                       in_quotes = true
                       field_started = true
@@ -519,15 +532,28 @@ module SmarterCSV
                 if !allow_escaped_quotes || backslash_count % 2 == 0
                   if quote_boundary_standard
                     if in_quotes
-                      # closing quote: only valid if followed by col_sep, row_sep, or end of line
                       next_i = i + 1
-                      if next_i >= line_size ||
-                         line[next_i...next_i + col_sep_size] == col_sep ||
-                         (row_sep_size > 0 && line[next_i...next_i + row_sep_size] == row_sep)
+                      if next_i + 1 < line_size && line[next_i] == quote
+                        # RFC doubled quote inside a quoted field ("" → ").
+                        # Give this precedence over the closing-quote check, but only
+                        # when another character follows the doubled pair.
+                        #
+                        # Compatibility note: we intentionally do NOT force terminal
+                        # "" to be consumed here. SmarterCSV has a long-standing lenient
+                        # behavior for malformed tails like ...\"" in :double_quotes mode:
+                        # the final quote may still close the field instead of turning the
+                        # row into an unclosed-quote error. Issue #334 needs doubled-quote
+                        # precedence for ..."",... (more content follows), but we keep the
+                        # historical leniency for terminal ..."".
+                        i = next_i
+                      # closing quote: only valid if followed by col_sep, row_sep, or end of line
+                      elsif next_i >= line_size ||
+                            line[next_i...next_i + col_sep_size] == col_sep ||
+                            (row_sep_size > 0 && line[next_i...next_i + row_sep_size] == row_sep)
                         in_quotes = false
                         field_started = true
                       end
-                      # else: quote inside quoted field → literal (handles "" doubling)
+                      # else: quote inside quoted field → literal
                     elsif !field_started # at field boundary: open quoted field
                       in_quotes = true
                       field_started = true

data/lib/smarter_csv/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SmarterCSV
-  VERSION = "1.17.0"
+  VERSION = "1.17.2"
 end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: smarter_csv
 version: !ruby/object:Gem::Version
-  version: 1.17.0
+  version: 1.17.2
 platform: ruby
 authors:
 - Tilo Sloboda
 bindir: bin
 cert_chain: []
-date: 2026-05-14 00:00:00.000000000 Z
+date: 2026-05-21 00:00:00.000000000 Z
 dependencies: []
 description: |
   SmarterCSV is a high-performance CSV reader and writer for Ruby focused on
@@ -39,7 +39,6 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
-- TO_DO.md
 - docs/_introduction.md
 - docs/bad_row_quarantine.md
 - docs/basic_read_api.md

data/TO_DO.md

@@ -1,109 +0,0 @@
-# SmarterCSV v2.0 TO DO List
-
-DONE:
-[X] Don't call rewind on filehandle
-[X] use Procs for validations and transformatoins [issue #118](https://github.com/tilo/smarter_csv/issues/118)
-[X] skip file opening, allow reading from CSV string, e.g. reading from S3 file [issue #120](https://github.com/tilo/smarter_csv/issues/120). Or stream large file from S3 (linked in the issue)
-[X] [2.0 BUG]  convert_to_float saves Proc as @@convert_to_integer [issue #157](https://github.com/tilo/smarter_csv/issues/157)
-[X] add enumerable to speed up parallel processing [issue #66](https://github.com/tilo/smarter_csv/issues/66), [issue #32](https://github.com/tilo/smarter_csv/issues/32)
-[X] Provide an example for custom Procs for hash_transformations in the docs [issue #174](https://github.com/tilo/smarter_csv/issues/174)
-[X] Collect all Errors, before surfacing them. Avoid throwing an exception on the first error [issue #133](https://github.com/tilo/smarter_csv/issues/133)
-
-
-Partially Done:
-[ ] make @errors and @warnings work [issue #118](https://github.com/tilo/smarter_csv/issues/118)
-
-StilL TO DO:
-[ ] Replace remove_empty_values: false [issue #213](https://github.com/tilo/smarter_csv/issues/213)
-
-Arguably by design (e.g. exclude these columns from conversion and have them returned as a string)
-[ ] [2.0 BUG] :convert_values_to_numeric_unless_leading_zeros drops leading zeros [issue #151](https://github.com/tilo/smarter_csv/issues/151)
-
-
-## Numeric conversion: align the Ruby fallback path with the C path (permissive)
-
-Context: `convert_values_to_numeric` runs in two places that currently DISAGREE on edge cases:
-  - C path (`acceleration: true`, the default): `ext/smarter_csv/smarter_csv.c#try_numeric_conversion`
-    uses `strtol`/`strtod` (base 10; float branch only entered when the field contains a `.`).
-  - Ruby fallback (`acceleration: false`): `lib/smarter_csv/hash_transformations.rb` uses the
-    strict regex `NUMERIC_REGEX = /\A[+-]?\d+(?:\.\d+)?\z/` plus `to_i` / `to_f`.
-
-Divergence (verified empirically):
-  | value     | C path           | Ruby fallback     |
-  |-----------|------------------|-------------------|
-  | ".5"      | 0.5 (Float)      | ".5" (String)     |
-  | "3."      | 3.0 (Float)      | "3." (String)     |
-  | "1.5e3"   | 1500.0 (Float)   | "1.5e3" (String)  |
-  | "1.0e10"  | 10000000000.0    | "1.0e10" (String) |
-
-Decision: the C path's permissive behavior (corner cases + scientific notation) is the intended
-contract. Fix = make the Ruby fallback match the C path. Do NOT tighten the C path.
-
-Ruby-side changes (in `hash_transformations.rb`):
-  1. Swap NUMERIC_REGEX for a permissive one:
-       /\A[+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?\z/
-     matches 1, 1., 1.5, .5, 1e3, 1.5e3, -3.14e-2, etc.; still rejects ".", "e3", "1.2.3",
-     "1_000", "0x1F".
-  2. Add `DOT_BYTE = '.'.ord` (46) and include it in the first-byte fast-reject's allowed set
-     (the C pre-check already allows a leading `.`; without this, ".5" gets rejected on byte 0).
-  3. Int-vs-float decision: `(v.include?('.') || v.include?('e') || v.include?('E')) ? v.to_f : v.to_i`
-     (currently only checks for `.`).
-
-Stays a string on BOTH paths (no change needed, but worth characterization tests — there are
-currently NONE):
-  - "010" => 10  (NOT octal 8 — both paths use base-10 conversion: String#to_i / strtol(.,10).
-    A switch to Kernel#Integer() would break this. Lock it down with a test.)
-  - "0x1F", "0b101", "0o17"  => string  (radix prefixes not honored by base-10 conversion)
-  - "1_000"                  => string  (underscores)
-  - "1,200.00", "1.300,00"   => string  (thousands sep / decimal comma — strtod stops at the
-    separator → not fully consumed; regex rejects. This is the only safe behavior; "1,200" is
-    genuinely ambiguous. Locale-specific number formats are the caller's job via value_converters.)
-
-NOT doing: locale sniffing (read LC_NUMERIC at init and adjust the regexes). Rejected because
-the machine locale tells you nothing about the file's number format, it breaks reproducibility
-(same code + same file → different results on a US vs EU box), and `,` can't be both col_sep and
-decimal separator anyway. Note `strtod` IS locale-sensitive (LC_NUMERIC) but it's dormant — Ruby
-runs in the C/POSIX locale; don't deliberately activate it.
-
-When done: parity tests (`[true, false].each`) for the now-consistent set (.5, 3., 1.5e3, 1e3)
-plus characterization tests for the stays-a-string set above; CHANGELOG line noting the Ruby
-fallback's numeric conversion now accepts scientific notation and bare-dot forms, matching the
-accelerated path. Behavior change affects `acceleration: false` users only — and aligns them with
-the default.
-
-
-## Warn once when the C extension didn't load on a platform that supports it
-
-Context: `acceleration: true` is the default. When the C extension fails to build / isn't loaded,
-SmarterCSV silently falls back to the Ruby parser — graceful degradation by design (so the gem
-keeps working for users with broken toolchains, JRuby, TruffleRuby, etc.). Today there is no
-signal to the user that they're not getting the C path; their CSV parsing is just slower than
-they might have expected.
-
-Idea: emit a one-time warning when:
-  * the C extension is NOT loaded — `!SmarterCSV::Parser.respond_to?(:parse_csv_line_c)`, AND
-  * the platform is one where it *should* be available — `RUBY_ENGINE == 'ruby'` (MRI / CRuby).
-    JRuby and TruffleRuby don't load CRuby C extensions natively; nothing for the user to do.
-
-Where to fire:
-  * NOT at `require 'smarter_csv'` time — Rails.logger typically isn't set up yet, so any
-    "route through the warnings system" code would just fall through to `Kernel#warn` anyway,
-    and the warning would land in stderr instead of the Rails log where ops would see it.
-  * At first `Reader.new` / `SmarterCSV.process` call — Rails has booted, the existing
-    routing-through-Rails.logger-or-Kernel#warn infra works, and the existing deduped warnings
-    histogram means it fires once per process regardless of how many parse calls.
-
-Implementation sketch:
-  * Add a new warning code (e.g. `:c_extension_unavailable`) alongside the existing ones
-    (`:chunk_size_default`, `:header_a_method`, `:utf8_missing_binary_mode`, ...).
-  * Severity `:warn`. Suppressible via the existing `verbose: :quiet`.
-  * Message points at the fix — e.g. "C acceleration extension not loaded on this Ruby; using
-    Ruby parser. To enable acceleration, reinstall with `gem pristine smarter_csv` and check
-    the build log." Plus a link/pointer to a troubleshooting section in the docs.
-
-Bonus: add a public predicate `SmarterCSV.acceleration_available?` returning
-`Parser.respond_to?(:parse_csv_line_c)`. Zero noise, useful for scripts / CI / future spec
-files that want to branch on the environment fact rather than guess.
-
-NOT doing: a banner at `require` time (every Rails app would print it at boot, too noisy);
-warning when `acceleration: false` was explicitly chosen (the user knows what they're doing).