smarter_csv 1.16.0 → 1.16.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '082f1c3a20d98a975bc3992ab4dd71d30a3008308f8c8dcbc116f60a2206b824'
-  data.tar.gz: 936107710f1588d54b7655e96781a677567d83a302269b1ddf7f2e63d0a31b63
+  metadata.gz: 043745aedb1c63fd4a044b9ae46bb8e5d98324c14e609214ee3d895acfd5f501
+  data.tar.gz: c39a10521b767daf51887278c9020c9ff6d8d93c32c5ec3f95a17ec575ebdab5
 SHA512:
-  metadata.gz: 212ba64f768895c1250cbae766a6dd91dfe7e83e6ca856b5d94008c8c295bda7a1a1bdaa88e71b25fcacee46fe1400f68fe2574173043e0c1537fa40be90fb9d
-  data.tar.gz: 4dfdc5ce2275b3b4a4e82f2abcc45a112a4eb5e90e3324218688317ed72eadde9b59d64ffc9b4bbf108fcd8e36c702172b22600b766762d393831c3404d8a560
+  metadata.gz: 5f1d125138443f02e0276e964dac9e584b996de6acafe8b3856316852a38220094e69a2c14302f922bcc93b6d23cc594bbf926940ccd70a2bd65ab08c5a18b49
+  data.tar.gz: '0929051996781c8643c0239556d123c840e7041d9c12f7d867e3800dfb2c2eb92e6f6fb77b5fc08660a36f34f470cb826f255943fa445d3e29d231e648da51b4'

data/.rspec

@@ -1 +1,3 @@
 --require spec_helper
+--color
+--format documentation

data/CHANGELOG.md

@@ -1,10 +1,54 @@
 
 # SmarterCSV 1.x Change Log
 
+## 1.16.1 (2026-03-16) — Bug Fixes & New Features
+
+RSpec tests: **1,247 → 1,410** (+163 tests)
+
+### New Features
+
+* **`SmarterCSV.errors`** — class-level error access after any `process`, `parse`, `each`, or `each_chunk` call.
+  Exposes the same `reader.errors` hash without requiring access to the `Reader` instance.
+  Errors are cleared at the start of each call and stored per-thread (safe in Puma/Sidekiq).
+
+  ```ruby
+  # Previously — required Reader instance to access errors
+  reader = SmarterCSV::Reader.new('data.csv', on_bad_row: :skip)
+  reader.process
+  puts reader.errors[:bad_row_count]
+
+  # Now — works with the class-level API too
+  SmarterCSV.process('data.csv', on_bad_row: :skip)
+  puts SmarterCSV.errors[:bad_row_count]
+  ```
+
+> **Note:** `SmarterCSV.errors` only surfaces errors from the **most recent run on the
+> current thread**. In a multi-threaded environment (Puma, Sidekiq), each thread maintains
+> its own error state independently. If you call `SmarterCSV.process` twice in the same
+> thread, the second call's errors replace the first's. For long-running or complex
+> pipelines where you need to aggregate errors across multiple files, use the Reader API.
+>
+> ⚠️ **Fibers:** `SmarterCSV.errors` uses `Thread.current` for storage, which is **shared
+> across all fibers running in the same thread**. If you process CSV files concurrently
+> in fibers (e.g. with `Async`, `Falcon`, or manual `Fiber` scheduling), `SmarterCSV.errors`
+> may return stale or wrong results. **Use `SmarterCSV::Reader` directly** — errors are
+> scoped to the reader instance and are always correct regardless of fiber context.
+
+### Bug Fixes
+
+* fixed [#325](https://github.com/tilo/smarter_csv/issues/325): `col_sep` in quoted headers was handled incorrectly; Thanks to Paho Lurie-Gregg.
+* fixed issue with quoted numeric fields that were not converted to numeric
+
+### Tests
+
+* Added 163 tests covering new features and corner cases
+
 ## 1.16.0 (2026-03-12) — Minor Breaking Change
 
 [Full details](docs/releases/1.16.0/changes.md) · [Benchmarks](docs/releases/1.16.0/benchmarks.md) · [Performance notes](docs/releases/1.16.0/performance_notes.md)
 
+RSpec tests: **714 → 1,247** (+533 tests)
+
 ### Minor Breaking Change
 
 New option **`quote_boundary:`**

data/CONTRIBUTORS.md

@@ -64,3 +64,4 @@ A Big Thank you to everyone who filed issues, sent comments, and who contributed
  * [Mark Bumiller](https://github.com/makrsmark)
  * [Tophe](https://github.com/tophe)
  * [Dom Lebron](https://github.com/biglebronski)
+ * [Paho Lurie-Gregg](https://github.com/paholg)

data/README.md

@@ -3,9 +3,14 @@
 
   ![Gem Version](https://img.shields.io/gem/v/smarter_csv) [![codecov](https://codecov.io/gh/tilo/smarter_csv/branch/main/graph/badge.svg?token=1L7OD80182)](https://codecov.io/gh/tilo/smarter_csv) [View on RubyGems](https://rubygems.org/gems/smarter_csv) [View on RubyToolbox](https://www.ruby-toolbox.com/search?q=smarter_csv)
 
-  SmarterCSV is a high-performance CSV ingestion and generation for Ruby, focused on fastest end-to-end CSV ingestion — not just parsing.
+  SmarterCSV is a high-performance CSV ingestion and generation for Ruby, focused on fast end-to-end CSV ingestion of real-world data — no silent failures, no surprises, not just tokenization.
 
-  ⭐ If SmarterCSV saved you hours of import time, please star the repo.
+  ⭐ If SmarterCSV saved you hours of import time, please star the repo, and consider sponsoring this project.
+
+  Ruby's built-in CSV library has 10 documented failure modes that can silently corrupt or lose data — duplicate headers, blank header cells, extra columns, BOMs, whitespace, encoding issues, and more — all without raising an exception.
+  SmarterCSV handles 8 our of 10 by default, and the remaining 2 with a single option each.
+
+  > See [**Ruby CSV Pitfalls**](docs/ruby_csv_pitfalls.md) for 10 ways `CSV.read` silently corrupts or loses data, and how SmarterCSV handles them.
 
   Beyond raw speed, SmarterCSV is designed to provide a significantly more convenient and developer-friendly interface than traditional CSV libraries. Instead of returning raw arrays that require substantial post-processing, SmarterCSV produces Rails-ready hashes for each row, making the data immediately usable with ActiveRecord, Sidekiq pipelines, parallel processing, and JSON-based workflows such as S3.
 
@@ -66,7 +71,7 @@ rows = SmarterCSV.process('data.csv')
 data = SmarterCSV.parse(csv_string)
 ```
 
-See [**Migrating from Ruby CSV**](docs/migrating_from_csv.md) for a full comparison of options, behavior differences, and a quick-reference table.
+* See [**Migrating from Ruby CSV**](docs/migrating_from_csv.md) for a full comparison of options, behavior differences, and a quick-reference table.
 
 ## Examples
 
@@ -205,6 +210,7 @@ Or install it yourself as:
 
   * [Introduction](docs/_introduction.md)
   * [**Migrating from Ruby CSV**](docs/migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](docs/ruby_csv_pitfalls.md)
   * [Parsing Strategy](docs/parsing_strategy.md)
   * [The Basic Read API](docs/basic_read_api.md)
   * [The Basic Write API](docs/basic_write_api.md)
@@ -243,7 +249,7 @@ For reporting issues, please:
   * open a pull-request adding a test that demonstrates the issue
   * mention your version of SmarterCSV, Ruby, Rails
 
-# [A Special Thanks to all 61 Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
+# [A Special Thanks to all 62 Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
 
 
 ## Contributing

data/docs/_introduction.md

@@ -3,6 +3,7 @@
 
   * [**Introduction**](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)
@@ -29,7 +30,11 @@
 
 ## Why another CSV library?
 
-Ruby's built-in `csv` library is **slow** — up to 129× slower than SmarterCSV for equivalent work — and its API is inconvenient. It returns arrays of arrays, which means your application code must handle column indexing, header normalization, type conversion, and whitespace stripping manually. It also has no built-in support for chunked or parallel processing of large files.
+**Inconvenient.** Ruby's built-in `csv` library returns arrays of arrays, which means your application code must handle column indexing, header normalization, type conversion, and whitespace stripping manually. It also has no built-in support for chunked or parallel processing of large files.
+
+**Hidden failure modes.** `CSV.read` has 10 ways to silently corrupt or lose data — no exception, no warning, no log line. Duplicate headers, blank header cells, extra columns, BOMs, whitespace, inconsistent empty-field representation, runaway quoted fields, and encoding issues all fail silently. See [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md) for reproducible examples and the SmarterCSV fix for each.
+
+**Slow.** On top of everything else, it is up to 129× slower than SmarterCSV for equivalent end-to-end work.
 
 ![SmarterCSV 1.16.0 vs Ruby CSV 3.3.5 speedup](../images/SmarterCSV_1.16.0_vs_RubyCSV_3.3.5_speedup.png)
 

data/docs/bad_row_quarantine.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)
@@ -55,27 +56,33 @@ SmarterCSV.process('data.csv')
 # => raises SmarterCSV::MalformedCSV on the first bad row
 ```
 
-### `:skip`
+The `on_bad_row` option controls what happens when a bad row is encountered:
 
-Silently skip bad rows and continue. The count of skipped rows is available on
-`reader.errors[:bad_row_count]`. No error records are stored.
+* `on_bad_row: :raise` (default) fails fast.
+* `on_bad_row: :collect` quarantines them — error records available via `SmarterCSV.errors` or `reader.errors`.
+* `on_bad_row: ->(rec) { ... }` calls your lambda per bad row — works with both `SmarterCSV.process` and `SmarterCSV::Reader`.
+* `on_bad_row: :skip` discards bad rows silently — count available via `SmarterCSV.errors` or `reader.errors`.
 
-```ruby
-reader = SmarterCSV::Reader.new('data.csv', on_bad_row: :skip)
-result = reader.process
+### `:collect`
 
-puts "Processed: #{result.size} good rows"
-puts "Skipped:   #{reader.errors[:bad_row_count] || 0} bad rows"
-```
+Continue processing and store a structured error record for each bad row.
+Error records are available via `SmarterCSV.errors[:bad_rows]` (class-level API)
+or `reader.errors[:bad_rows]` (Reader API).
 
-### `:collect`
+```ruby
+# Class-level API — use SmarterCSV.errors after the call
+good_rows = SmarterCSV.process('data.csv', on_bad_row: :collect)
 
-Continue processing and store a structured error record for each bad row in
-`reader.errors[:bad_rows]`. Requires using `SmarterCSV::Reader` directly (the
-`SmarterCSV.process` convenience method discards the reader instance and cannot
-return the collected errors).
+good_rows.each { |row| MyModel.create!(row) }
+
+SmarterCSV.errors[:bad_rows].each do |rec|
+  Rails.logger.warn "Bad row at line #{rec[:csv_line_number]}: #{rec[:error_message]}"
+  Rails.logger.warn "Raw content: #{rec[:raw_logical_line]}"
+end
+```
 
 ```ruby
+# Reader API — use when you also need access to headers or other reader state
 reader = SmarterCSV::Reader.new('data.csv', on_bad_row: :collect)
 result = reader.process
 
@@ -90,35 +97,50 @@ end
 ### Callable (lambda / proc)
 
 Pass any object that responds to `#call`. It is invoked once per bad row with the
-error record hash, then processing continues. Useful for streaming errors to a
-dead-letter queue, a metrics system, or a separate file.
+error record hash, then processing continues. Because the lambda receives errors
+inline, **this works with both `SmarterCSV.process` and `SmarterCSV::Reader`** —
+you do not need a `Reader` instance to handle bad rows.
+
+```ruby
+# Works with SmarterCSV.process — no Reader instance needed
+bad_rows = []
+good_rows = SmarterCSV.process('data.csv',
+  on_bad_row: ->(rec) { bad_rows << rec })
+```
 
 ```ruby
 # Log to a dead-letter file
 quarantine = File.open('quarantine.csv', 'w')
-
-reader = SmarterCSV::Reader.new('data.csv',
-  on_bad_row: ->(rec) { quarantine.puts(rec[:raw_logical_line]) }
-)
-reader.process
+SmarterCSV.process('data.csv',
+  on_bad_row: ->(rec) { quarantine.puts(rec[:raw_logical_line]) })
 quarantine.close
 ```
 
 ```ruby
 # Send to a monitoring system
-reader = SmarterCSV::Reader.new('data.csv',
-  on_bad_row: ->(rec) { Metrics.increment('csv.bad_rows', tags: { error: rec[:error_class].name }) }
-)
-reader.process
+SmarterCSV.process('data.csv',
+  on_bad_row: ->(rec) { Metrics.increment('csv.bad_rows', tags: { error: rec[:error_class].name }) })
 ```
 
+### `:skip`
+
+Silently skip bad rows and continue. The count of skipped rows is available via
+`SmarterCSV.errors[:bad_row_count]` (class-level API) or `reader.errors[:bad_row_count]`
+(Reader API). No error records are stored.
+
 ```ruby
-# Collect into your own structure
-errors = []
-reader = SmarterCSV::Reader.new('data.csv',
-  on_bad_row: ->(rec) { errors << rec }
-)
+# Class-level API — use SmarterCSV.errors after the call
+SmarterCSV.process('data.csv', on_bad_row: :skip)
+puts "Skipped: #{SmarterCSV.errors[:bad_row_count] || 0} bad rows"
+```
+
+```ruby
+# Reader API — access reader.errors directly
+reader = SmarterCSV::Reader.new('data.csv', on_bad_row: :skip)
 result = reader.process
+
+puts "Processed: #{result.size} good rows"
+puts "Skipped:   #{reader.errors[:bad_row_count] || 0} bad rows"
 ```
 
 ## Error record structure
@@ -173,15 +195,50 @@ end
 
 ## Accessing errors
 
-Bad row data is stored on the `Reader` instance:
+There are two ways to access bad row data after processing:
+
+### Via `SmarterCSV.errors` (class-level API)
+
+`SmarterCSV.errors` returns the errors from the most recent call to `process`, `parse`,
+`each`, or `each_chunk` on the current thread. It is cleared at the start of each new call.
+
+```ruby
+SmarterCSV.process('data.csv', on_bad_row: :skip)
+puts SmarterCSV.errors[:bad_row_count]   # => 3
+
+SmarterCSV.process('data.csv', on_bad_row: :collect)
+puts SmarterCSV.errors[:bad_row_count]   # => 3
+puts SmarterCSV.errors[:bad_rows].size   # => 3
+```
+
+> **Note:** `SmarterCSV.errors` only surfaces errors from the **most recent run on the
+> current thread**. In a multi-threaded environment (Puma, Sidekiq), each thread maintains
+> its own error state independently. If you call `SmarterCSV.process` twice in the same
+> thread, the second call's errors replace the first's. For long-running or complex
+> pipelines where you need to aggregate errors across multiple files, use the Reader API.
+>
+> ⚠️ **Fibers:** `SmarterCSV.errors` uses `Thread.current` for storage, which is **shared
+> across all fibers running in the same thread**. If you process CSV files concurrently
+> in fibers (e.g. with `Async`, `Falcon`, or manual `Fiber` scheduling), `SmarterCSV.errors`
+> may return stale or wrong results. **Use `SmarterCSV::Reader` directly** — errors are
+> scoped to the reader instance and are always correct regardless of fiber context.
+
+### Via `reader.errors` (Reader API)
+
+For full control — including access to headers, raw headers, and errors from a specific
+call — use `SmarterCSV::Reader` directly:
 
 | Attribute | Description |
 |-----------|-------------|
 | `reader.errors[:bad_row_count]` | Total bad rows encountered (all modes) |
 | `reader.errors[:bad_rows]` | Array of error records (`:collect` mode only) |
 
-Note: `SmarterCSV.process` (the convenience method) discards the `Reader` instance after
-returning. To access `reader.errors`, always instantiate `SmarterCSV::Reader` directly.
+```ruby
+reader = SmarterCSV::Reader.new('data.csv', on_bad_row: :collect)
+reader.process
+puts reader.errors[:bad_row_count]
+puts reader.headers.inspect
+```
 
 ## Chunked processing
 

data/docs/basic_read_api.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [**The Basic Read API**](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)    

data/docs/basic_write_api.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [**The Basic Write API**](./basic_write_api.md)

data/docs/batch_processing.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)

data/docs/column_selection.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)

data/docs/data_transformations.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)

data/docs/examples.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)

data/docs/header_transformations.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)

data/docs/header_validations.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)

data/docs/history.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)
@@ -59,6 +60,7 @@ The first gem release was **v1.0.1 on 2012-07-30**.
 | 1.15.1  | 2026-02-17 | Fix for backslash in quoted fields (`quote_escaping:` option) |
 | 1.15.2  | 2026-02-20 | Further C-path optimisations; 5.4×–37.4× faster than 1.14.4 |
 | **1.16.0** | **2026-03-12** | **New `each`/`each_chunk` enumerator API; `SmarterCSV.parse`; bad row quarantine; column selection `headers: { only: }`; 1.8×–8.6× faster than Ruby CSV.read; new features for Reader and Writer; minor breaking: `quote_boundary: :standard`** |
+| 1.16.1  | 2026-03-16 | `SmarterCSV.errors` class-level error access; fix `col_sep` in quoted headers (#325); fix quoted numeric conversion |
 
 ---
 

data/docs/instrumentation.md

@@ -3,6 +3,7 @@
 
   * [Introduction](./_introduction.md)
   * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)