smarter_csv 1.16.4 → 1.17.0

data/docs/releases/1.17.0/changes.md

@@ -0,0 +1,161 @@
+
+### Contents
+
+  * [Introduction](../../_introduction.md)
+  * [Migrating from Ruby CSV](../../migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](../../ruby_csv_pitfalls.md)
+  * [Parsing Strategy](../../parsing_strategy.md)
+  * [The Basic Read API](../../basic_read_api.md)
+  * [The Basic Write API](../../basic_write_api.md)
+  * [Batch Processing](../../batch_processing.md)
+  * [Configuration Options](../../options.md)
+  * [Row and Column Separators](../../row_col_sep.md)
+  * [Header Transformations](../../header_transformations.md)
+  * [Header Validations](../../header_validations.md)
+  * [Column Selection](../../column_selection.md)
+  * [Data Transformations](../../data_transformations.md)
+  * [Value Converters](../../value_converters.md)
+  * [Bad Row Quarantine](../../bad_row_quarantine.md)
+  * [Warnings](../../warnings.md)
+  * [Instrumentation Hooks](../../instrumentation.md)
+  * [Examples](../../examples.md)
+  * [Real-World CSV Files](../../real_world_csv.md)
+  * [SmarterCSV over the Years](../../history.md)
+  * [**Release Notes**](./changes.md)
+
+--------------
+
+# SmarterCSV 1.17.0 — Changes
+
+RSpec tests: **1,434 → 2,210** (+776 tests since 1.16.4)
+
+1.17.0 is a **features-and-quality** release, focused on three things: streaming IO inputs, a structured warnings system, and Rails-friendly defaults. The C parser's core line-parsing — separator splitting, quote/escape handling, multiline stitching — is unchanged from 1.16.0 (see [`docs/releases/1.16.0/`](../1.16.0/changes.md) for the parser performance story); what changed in the C path this cycle is a faster code path for quoted-field-heavy files and Unicode-aware blank detection. On the C-accelerated path, 1.17.0 vs 1.16.4 is a **mixed picture**: quoted-field-heavy and wide files run meaningfully faster, a handful of short-line / many-small-field files run a little slower, and the rest are within noise. The Ruby path is parity throughout. The wins come from the faster quoted-field handling; the small regressions trace to the new auto-detection default (`auto_row_sep_chars` 500→4096) plus a tiny per-line overhead — see [performance_notes.md](performance_notes.md) and [benchmarks.md](benchmarks.md) for the per-file breakdown.
+
+---
+
+## Compatibility
+
+* **No breaking changes.** All 1.16.x code continues to work without modification.
+* **Behavior change worth noting:** `auto_row_sep_chars: nil` / `0` no longer means "scan whole file" — these values fall back to the default with a warning. The total scan is hard-capped at 64KB. If you relied on the previous undocumented "scan whole file" semantics, this is a visible change.
+
+---
+
+## Headline Features
+
+### 1. Non-Seekable Streaming Inputs
+
+SmarterCSV now reads directly from any IO source — including streams that don't support `rewind` or `seek`. No need to materialize the file on disk first.
+
+```ruby
+# Gzipped CSV — stream-decompressed, never written to disk
+require 'zlib'
+Zlib::GzipReader.open('huge.csv.gz') do |io|
+  SmarterCSV.process(io) { |row| MyModel.upsert(row.first) }
+end
+
+# STDIN / pipes
+SmarterCSV.process($stdin) { |row, _| MyModel.upsert(row.first) }
+
+# HTTP response body
+require 'open-uri'
+URI.open('https://example.com/data.csv') { |io| SmarterCSV.process(io) }
+
+# S3 — stream the response body directly
+require 'aws-sdk-s3'
+obj = Aws::S3::Client.new.get_object(bucket: 'data', key: 'imports/users.csv')
+SmarterCSV::Reader.new(obj.body, chunk_size: 500).each_chunk do |chunk, _|
+  MyModel.insert_all(chunk)
+end
+```
+
+Auto-detection of `row_sep` and `col_sep` works on these streaming sources thanks to internal buffering — the underlying source never needs to support `rewind` or `seek`. See [Real-World CSV Files → I/O Patterns](../../real_world_csv.md#io-patterns) and [Examples → Streaming Inputs](../../examples.md#example-14-streaming-inputs-non-seekable-io).
+
+### 2. Structured Warnings Collection
+
+Auto-detection and configuration warnings are now collected on the Reader as a deduped histogram, in addition to being emitted to a log sink:
+
+```ruby
+reader = SmarterCSV::Reader.new('data.csv')
+reader.process
+reader.warnings
+# => [
+#   { type: :config, code: :chunk_size_default, severity: :warn,
+#     message: "chunk_size not set, defaulting to 100. ...", count: 1 },
+#   ...
+# ]
+```
+
+Repeated warnings of the same `(type, code)` are deduped — `count` tracks occurrences across the run. This lets you surface warnings programmatically (dashboards, fail-deploys-on-codes, etc.) without parsing stderr text.
+
+**Warning codes available in 1.17.0:**
+
+| Code                          | Type           | Severity | Triggered when                                                                                |
+|-------------------------------|----------------|----------|-----------------------------------------------------------------------------------------------|
+| `:chunk_size_default`         | `:config`      | `:warn`  | `each_chunk` is called without `chunk_size:` and the default of `100` is used.                |
+| `:header_a_method`            | `:deprecation` | `:warn`  | The deprecated `Reader#headerA` accessor is called.                                           |
+| `:utf8_missing_binary_mode`   | `:encoding`    | `:warn`  | UTF-8 input is being processed but the IO was not opened with `"b:utf-8"`.                    |
+| `:no_clear_row_sep`           | `:row_sep`     | `:error` | Auto-detection found a true tie between separators after scanning 64KB. Silent miss-parse risk. |
+| `:no_row_sep_found`           | `:row_sep`     | `:error` | No known row separator was found in the first 64KB. Likely an exotic separator like `
`. |
+
+See [Warnings](../../warnings.md) for the full record shape, suppression options, and Rails integration details.
+
+### 3. Class-Level `SmarterCSV.warnings` Accessor
+
+Mirrors `SmarterCSV.errors`. Returns warnings from the most recent call to `process`, `parse`, `each`, or `each_chunk` on the current thread. Cleared at the start of each new call.
+
+```ruby
+SmarterCSV.process('data.csv')
+SmarterCSV.warnings.each do |w|
+  logger.warn("[#{w[:type]}/#{w[:code]}] #{w[:message]} (×#{w[:count]})")
+end
+```
+
+Per-thread (uses `Thread.current`) — safe under Puma and Sidekiq. Not fiber-safe; use `SmarterCSV::Reader` directly if processing CSV concurrently with `Async`/`Falcon`/manual `Fiber` scheduling.
+
+### 4. Rails.logger Auto-Routing
+
+When `Rails.logger` is present, warnings are routed through it at the severity declared at the call site (`:debug` / `:info` / `:warn` / `:error` / `:fatal`):
+
+```
+# In log/development.log
+[WARN]  SmarterCSV: chunk_size not set, defaulting to 100. ...
+```
+
+Without Rails, falls back to `Kernel#warn` (writes to `$stderr`). Detection is one-shot at Reader construction — no per-call overhead. The programmatic `reader.warnings` collection is identical in both modes.
+
+See [Warnings → Log sink routing](../../warnings.md#log-sink-routing).
+
+---
+
+## Improvements
+
+* **Better auto-detection of `row_sep` and `col_sep`** — more accurate results on files with comment headers and other irregularities at the start of the stream.
+
+* **`auto_row_sep_chars` default changed to `4096`** (was `500` in 1.16.x). Sized to cover wide-header CSVs in a single read. Out-of-range values, `nil`, or `0` fall back to the default with a warning. **Behavior change vs 1.16.x:** the previous undocumented "scan whole file" semantics on `nil`/`0` is removed; the total scan is hard-capped at 64KB.
+
+* **`buffer_size` is now a public option** — peek buffer chunk size for non-seekable inputs (pipes, gzip readers, HTTP/S3 bodies). Default `16_384`. Out-of-range values warn and clamp to the supported range rather than raising. Has no effect on seekable inputs (file paths, `File`, `StringIO`).
+
+* **Files ending in a lone `\r`** are now correctly detected as `\r`-terminated instead of falling through to a "no clear row separator" warning.
+
+* **`SmarterCSV.errors` mid-stream preservation** *(merged from 1.16.4)* — fixed a bug where collected error records could be lost when processing raised mid-stream (e.g. `bad_row_limit:` exceeded → `TooManyBadRows`, or a user block raising through `.process` / `.each` / `.each_chunk`).
+
+* **`enforce_utf8_encoding` for `ASCII-8BIT` inputs** *(merged from 1.16.4)* — fixed incorrect replacement of all non-ASCII bytes when the input was tagged binary. Encoding is now relabeled to UTF-8 before transcoding so only genuinely invalid byte sequences are replaced.
+
+---
+
+## Documentation
+
+Substantive expansion of the user-facing docs to match the new capabilities:
+
+* **`docs/examples.md`** — six new cookbook entries (Examples 14–19): Streaming Inputs, Resumable Plain-Ruby Import, CSV Files with Comment Lines, Tab-Separated Values (TSV), Multi-Line Fields, and Filtering and Transforming a CSV File (the `CSV.filter` replacement pattern).
+* **`docs/real_world_csv.md`** — expanded I/O Patterns section with worked examples for gzip, S3, HTTP, STDIN, and `IO.popen`. Added a Multi-Line Quoted Fields worked example.
+* **`docs/warnings.md`** *(new)* — full coverage of the structured warnings system: record shape, available codes, log-sink routing for Rails vs non-Rails, suppression via `verbose: :quiet`.
+* **`docs/header_transformations.md`** — added a worked example for `comment_regexp:` (CSV files with comment lines).
+* **`docs/row_col_sep.md`** — added a worked TSV example.
+* **`docs/batch_processing.md`** — added a Resumable Import (Plain Ruby) example using `chunk_index` + a JSON state file (companion to the Rails 8.1 ActiveJob version in `examples.md`).
+* **`docs/basic_read_api.md`** / **`docs/basic_write_api.md`** — cross-references to the read-transform-write composition pattern; added `$stdout` and S3 streaming write examples.
+* **`README.md`** — added inline examples for streaming inputs, value converters, header validation, and writing CSV; one-sentence note on Rails.logger auto-routing.
+
+---
+
+PREVIOUS: [SmarterCSV over the Years](../../history.md) | UP: [README](../../../README.md)

data/docs/releases/1.17.0/performance_notes.md

@@ -0,0 +1,126 @@
+# SmarterCSV 1.17.0 — Performance Notes
+
+The per-file tables below: Apple M4, Ruby 3.4.7 [arm64], 40 iterations per run × 8 runs, median across runs (p10-trimmed), measured 2026-05-11–12. 19-file corpus; `1.16.4 → 1.17.0`. Times in seconds — lower is better. (The "vs Ruby CSV" tables further down are from the earlier 2026-05-06 run — see Methodology.)
+
+---
+
+## 1.16.4 → 1.17.0 — C-accelerated path (the default)
+
+The C parser's core line-parsing (separator splitting, quote/escape handling, multiline stitching) is unchanged from 1.16.0. The C-path changes this cycle are a faster code path for quoted-field-heavy files — the big wins — and Unicode-aware blank detection.
+
+| file                           | 1.16.4 (s) | 1.17.0 (s) | 1.17.0 vs 1.16.4 |
+| ------------------------------ | ---------- | ---------- | ---------------- |
+| PEOPLE_IMPORT_B.csv            |    0.06255 |    0.06305 | ~1% noise        |
+| PEOPLE_IMPORT_C.csv            |    0.13072 |    0.13274 | ~2% noise        |
+| PEOPLE_IMPORT_NB.csv           |    0.05985 |    0.06079 | ~2% noise        |
+| PEOPLE_IMPORT_NC.csv           |    0.05273 |    0.05420 | ~3% noise        |
+| uscities.csv                   |    0.06325 |    0.05545 | 12.3% faster     |
+| uszips.csv                     |    0.06957 |    0.06255 | 10.1% faster     |
+| worldcities.csv                |    0.06824 |    0.06134 | 10.1% faster     |
+| embedded_newlines_60k.csv      |    0.12795 |    0.11951 | 6.6% faster      |
+| embedded_separators_60k.csv    |    0.05093 |    0.04591 | 9.9% faster      |
+| heavy_quoting_60k.csv          |    0.08926 |    0.07490 | 16.1% faster     |
+| long_fields_40k.csv            |    0.06375 |    0.04970 | 22.0% faster     |
+| many_empty_fields_60k.csv      |    0.06813 |    0.06888 | ~1% noise        |
+| multi_char_separator_60k.csv   |    0.07720 |    0.07830 | ~1% noise        |
+| sample_100k.csv                |    0.07051 |    0.07139 | ~1% noise        |
+| sensor_data_50krows_50cols.csv |    0.17839 |    0.17897 | ~1% noise        |
+| tab_separated_60k.tsv          |    0.06704 |    0.06798 | ~1% noise        |
+| utf8_multibyte_60k.csv         |    0.04391 |    0.04376 | ~ same           |
+| whitespace_heavy_60k.csv       |    0.06803 |    0.06897 | ~1% noise        |
+| wide_500_cols_20k.csv          |    1.07019 |    1.07348 | ~1% noise        |
+
+*`~N% noise` means the measured difference (≈N%, always a small slowdown here) is within the run-to-run variance of this setup (8 runs × 40 iterations, median across runs, p10-trimmed) — i.e. effectively unchanged, not a real regression. The raw per-version times are in the table for the exact figure.*
+
+Quote-heavy / large-field / wide files run **7–22% faster** than 1.16.4 (`long_fields_40k` 22%, `heavy_quoting_60k` 16%, the city files 10–12%, `embedded_separators` 10%, `embedded_newlines` 7%). Everything else is within ±3% of 1.16.4 — effectively unchanged. (The short-line / many-small-field files do show a small, *consistent* uptick at the bottom of that band, traceable to the larger default auto-detection scan window plus a tiny per-line overhead; if that matters for your workload, set `auto_row_sep_chars` lower. See [What's driving the mixed C-path picture](#whats-driving-the-mixed-c-path-picture) below.)
+
+---
+
+## 1.16.4 → 1.17.0 — Ruby fallback path (`acceleration: false`)
+
+Faster on nearly every file this cycle, from three changes: in-place stripping in the no-quote split path, a first-byte fast-reject before numeric conversion, and per-row / per-value overhead removed from the hash transformations.
+
+| file                           | 1.16.4 (s) | 1.17.0 (s) | 1.17.0 vs 1.16.4 |
+| ------------------------------ | ---------- | ---------- | ---------------- |
+| PEOPLE_IMPORT_B.csv            |    0.38220 |    0.35281 | 7.7% faster      |
+| PEOPLE_IMPORT_C.csv            |    0.99047 |    0.95728 | 3.4% faster      |
+| PEOPLE_IMPORT_NB.csv           |    0.36110 |    0.31716 | 12.2% faster     |
+| PEOPLE_IMPORT_NC.csv           |    0.28762 |    0.25849 | 10.1% faster     |
+| uscities.csv                   |    0.74246 |    0.71183 | 4.1% faster      |
+| uszips.csv                     |    0.90817 |    0.87628 | 3.5% faster      |
+| worldcities.csv                |    0.75714 |    0.72641 | 4.1% faster      |
+| embedded_newlines_60k.csv      |    0.88887 |    0.86252 | 3.0% faster      |
+| embedded_separators_60k.csv    |    0.57053 |    0.53401 | 6.4% faster      |
+| heavy_quoting_60k.csv          |    1.09395 |    1.02829 | 6.0% faster      |
+| long_fields_40k.csv            |    3.27964 |    3.29366 | ~ same           |
+| many_empty_fields_60k.csv      |    0.37815 |    0.33153 | 12.3% faster     |
+| multi_char_separator_60k.csv   |    0.45717 |    0.38380 | 16.0% faster     |
+| sample_100k.csv                |    0.34527 |    0.30690 | 11.1% faster     |
+| sensor_data_50krows_50cols.csv |    1.32705 |    1.33218 | ~ same           |
+| tab_separated_60k.tsv          |    0.38261 |    0.31359 | 18.0% faster     |
+| utf8_multibyte_60k.csv         |    0.24212 |    0.21281 | 12.1% faster     |
+| whitespace_heavy_60k.csv       |    0.37635 |    0.30848 | 18.0% faster     |
+| wide_500_cols_20k.csv          |    5.28395 |    4.23045 | 19.9% faster     |
+
+Gains run **3–20%** vs 1.16.4, biggest on wide / many-small-field files (`wide_500_cols` 20%, `whitespace_heavy` / `tab_separated` 18%, `multi_char_separator` 16%). Only `long_fields_40k` (dominated by large-field allocation, not per-field work) and `sensor_data` (numeric-heavy — the fast-reject's per-value cost and a saved per-value method call cancel out) sit at parity.
+
+---
+
+## What's driving the mixed C-path picture
+
+The C parser's core line-parsing — separator splitting, quote/escape handling, multiline stitching — is unchanged from 1.16.0; all of that hot-path work carries forward (see [the 1.16.0 changes](../1.16.0/changes.md) for the parser performance story). So why the split — some files faster, a band of small files a hair slower?
+
+**The wins are the quoted-field handling.** 1.17.0 added a faster path for fields wrapped in quotes: the common case — a quoted field with no doubled `""` inside — now skips a copy step. Files where most or all fields are quoted (city/address-style data, long quoted text, wide rows) pick up 7–22%.
+
+**The bigger default auto-detection window.** The benchmark leaves `row_sep` at `:auto` for every file, so each run reads `auto_row_sep_chars` bytes up front — now `4096`, was `500` — and scans them for the row separator.
+  * On tiny files where total parse time is only ~50–80 ms, that one-time scan shows up as a ≤3% uptick.
+  * On larger files it's noise (and often net-positive — the wider window usually settles the separator on the first read, avoiding the doubling-escalation loop).
+If you parse lots of very small files and care about that 1–3%, set `auto_row_sep_chars` lower, or pin `row_sep` explicitly to skip detection entirely. (The related `guess_line_ending` change — a chunked scan that doubles up to a 64 KB hard cap, replacing the old undocumented "scan whole file" on `nil`/`0` — is the same trade-off.)
+
+**Not a factor here:** the buffering layer for non-seekable streams. The benchmark passes file paths to `SmarterCSV.process`, which opens them as seekable `File` objects, so the seekable fast path is taken and no buffering wrapper is instantiated. That layer only runs for pipes / gzip readers / HTTP/S3 bodies, which have much higher latency anyway — any extra work the buffer does there is negligible.
+
+---
+
+## vs Ruby CSV 3.3.5 (1.17.0 reference)
+
+### vs `CSV.read` (raw arrays — minimum equivalent work)
+
+`CSV.read` is the *fastest* Ruby CSV mode: plain string arrays, no symbol keys, no numeric conversion. SmarterCSV/C delivers fully processed hashes — and still beats it on every file:
+
+| Range     | Files                                                                   |
+|-----------|-------------------------------------------------------------------------|
+| **7–8×**  | PEOPLE_IMPORT_C (7.8×), uszips (7.8×)                                   |
+| **6–7×**  | long_fields (6.9×), uscities (6.8×), worldcities (6.8×)                 |
+| **5–6×**  | embedded_separators (5.4×)                                              |
+| **3–4×**  | utf8_multibyte (3.9×), PEOPLE_IMPORT_NC (3.7×), many_empty (3.5×), heavy_quoting (3.4×), sample_100k (3.4×), PEOPLE_IMPORT_NB (3.2×) |
+| **2–3×**  | PEOPLE_IMPORT_B (2.9×), embedded_newlines (2.9×), whitespace_heavy (2.9×), sensor_data (2.5×) |
+| **1–2×**  | wide_500_cols (1.7×), tab_separated (1.6×), multi_char_separator (1.4×) |
+
+**Summary: 1.4×–7.8× faster than `CSV.read`, while returning fully processed hashes.**
+
+### vs `CSV.hashes` (string-keyed hashes — closer to SmarterCSV output)
+
+| Range      | Files                                                                  |
+|------------|------------------------------------------------------------------------|
+| **40–50×** | PEOPLE_IMPORT_C (47.3×)                                                |
+| **20–25×** | wide_500_cols (22.1×)                                                  |
+| **10–15×** | uszips (12.5×), PEOPLE_IMPORT_NC (12.1×), many_empty (11.8×), worldcities (11.4×), uscities (11.2×), sensor_data (11.1×) |
+| **7–10×**  | embedded_separators (8.3×), long_fields (8.1×), PEOPLE_IMPORT_NB (8.1×), PEOPLE_IMPORT_B (7.9×), heavy_quoting (7.0×) |
+| **5–7×**   | whitespace_heavy (6.9×), utf8_multibyte (6.7×), sample_100k (6.2×)     |
+| **4–5×**   | embedded_newlines (4.2×)                                               |
+| **2–3×**   | tab_separated (2.3×), multi_char_separator (2.2×)                      |
+
+**Summary: 2.2×–47.3× faster than `CSV.hashes`.**
+
+---
+
+## Methodology
+
+Same as 1.16.0:
+- Apple M4, Ruby 3.4.7
+- 40 iterations per run × 8 runs (2 warm-up), median across runs (p10-trimmed)
+- Raw .json captures preserved alongside the .md tables for reproducibility
+
+---
+
+PREVIOUS: [Changes](./changes.md) | UP: [README](../../../README.md)

data/docs/row_col_sep.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)
@@ -30,7 +31,7 @@
 
 Convenient defaults allow automatic detection of the column and row separators: `row_sep: :auto`, `col_sep: :auto`. This makes it easier to process any CSV files without having to examine the line endings or column separators, e.g. when users upload CSV files to your service and you have no control over the incoming files.
 
-You can change the setting `:auto_row_sep_chars` to only analyze the first N characters of the file (default is 500 characters); `nil` or `0` will check the whole file). Of course you can also set the `:row_sep` manually.
+The setting `:auto_row_sep_chars` controls the initial scan size used while detecting the row separator (default is `4096`). Detection stops as soon as one separator has a clear majority, up to a 64KB cap. Bump it higher if your files have very wide headers or long comment preambles; out-of-range values, `nil`, or `0` fall back to the default with a warning. Of course you can also set the `:row_sep` manually to skip auto-detection entirely.
 
 
 ## Column Separator `col_sep`
@@ -39,6 +40,25 @@ The automatic detection of column separators considers: `,`, `\t`, `;`, `:`, `|`
 
 Some CSV files may contain an unusual column separqator, which could even be a control character.
 
+### Tab-Separated Values (TSV)
+
+Tab-separated files are auto-detected by default — no options needed:
+
+```ruby
+$ cat data.tsv
+id<TAB>name<TAB>amount
+1<TAB>Alice<TAB>100
+2<TAB>Bob<TAB>200
+
+# Auto-detected — col_sep: :auto is the default
+SmarterCSV.process('data.tsv')
+
+# Or set the separator explicitly
+SmarterCSV.process('data.tsv', col_sep: "\t")
+```
+
+The default `col_sep: :auto` picks tab when it's the dominant delimiter in the first chunk of the file. The explicit form is useful in test fixtures or when you want to fail fast on unexpected formats.
+
 ## Row Separator `row_sep`
 
 The automatic detection of row separators considers: `\n`, `\r\n`, `\r`.

data/docs/ruby_csv_pitfalls.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)

data/docs/value_converters.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [**Value Converters**](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)
@@ -113,6 +114,29 @@ def self.convert(value)
 end
 ```
 
+## Handling Numeric Inputs
+
+Converters run **after** `convert_values_to_numeric`, so a field that looks like a
+number (e.g. `"42"`, `"3.14"`) will already be an `Integer` or `Float` by the time
+your converter sees it. If your converter expects a string, guard against this:
+
+```ruby
+# Safe: passes already-numeric values through unchanged
+dollar = ->(v) { v.is_a?(String) ? v.sub('$', '').to_f : v }
+
+# Unsafe: raises NoMethodError on Integer/Float (no #sub)
+dollar = ->(v) { v.sub('$', '').to_f }
+```
+
+Alternatively, exclude the column from numeric conversion so the converter always
+receives a string:
+
+```ruby
+SmarterCSV.process(file,
+  convert_values_to_numeric: { except: [:price] },
+  value_converters: { price: ->(v) { v&.sub('$', '')&.to_f } })
+```
+
 ## Class-Based Converters
 
 For converters you want to reuse across the codebase or test independently, define a class

data/docs/warnings.md

@@ -0,0 +1,141 @@
+
+### Contents
+
+  * [Introduction](./_introduction.md)
+  * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Ruby CSV Pitfalls](./ruby_csv_pitfalls.md)
+  * [Parsing Strategy](./parsing_strategy.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
+  * [Batch Processing](././batch_processing.md)
+  * [Configuration Options](./options.md)
+  * [Row and Column Separators](./row_col_sep.md)
+  * [Header Transformations](./header_transformations.md)
+  * [Header Validations](./header_validations.md)
+  * [Column Selection](./column_selection.md)
+  * [Data Transformations](./data_transformations.md)
+  * [Value Converters](./value_converters.md)
+  * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [**Warnings**](./warnings.md)
+  * [Instrumentation Hooks](./instrumentation.md)
+  * [Examples](./examples.md)
+  * [Real-World CSV Files](./real_world_csv.md)
+  * [SmarterCSV over the Years](./history.md)
+  * [Release Notes](./releases/1.16.0/changes.md)
+
+--------------
+
+# Warnings
+
+SmarterCSV records auto-detection and configuration warnings into a structured
+collection on the Reader, in addition to emitting them to a log sink. This lets
+you inspect warnings programmatically (e.g. surface them in dashboards, fail
+deploys on unexpected codes) without parsing stderr text.
+
+## Accessing warnings
+
+### Via the Reader API
+
+```ruby
+reader = SmarterCSV::Reader.new('data.csv')
+reader.process
+
+reader.warnings
+# => [
+#   { type: :config,   code: :chunk_size_default, severity: :warn,
+#     message: "chunk_size not set, defaulting to 100. ...", count: 1 },
+#   ...
+# ]
+```
+
+### Via the class-level API (`SmarterCSV.warnings`)
+
+Mirrors `SmarterCSV.errors`. Returns the warnings from the most recent call to
+`process`, `parse`, `each`, or `each_chunk` on the current thread. Cleared at
+the start of each new call.
+
+```ruby
+SmarterCSV.process('data.csv')
+SmarterCSV.warnings.each do |w|
+  logger.warn("[#{w[:type]}/#{w[:code]}] #{w[:message]} (×#{w[:count]})")
+end
+```
+
+> **Note:** `SmarterCSV.warnings` is per-thread (uses `Thread.current`). It is
+> safe in multi-threaded environments (Puma, Sidekiq), but **not fiber-safe**.
+> If you process CSV files concurrently in fibers (e.g. with `Async`, `Falcon`,
+> or manual `Fiber` scheduling), use `SmarterCSV::Reader` directly so warnings
+> are scoped to the reader instance.
+
+## Warning record shape
+
+| Field | Description |
+|---|---|
+| `type` | Coarse semantic grouping. Currently: `:config`, `:deprecation`, `:encoding`, `:row_sep`. |
+| `code` | Unique identifier for the specific warning. |
+| `severity` | Log level: `:debug` / `:info` / `:warn` / `:error` / `:fatal`. |
+| `message` | Human-readable description. |
+| `count` | Number of times this `(type, code)` was triggered during the run. |
+
+Repeated warnings of the same `(type, code)` are deduped — `count` tracks
+occurrences. The `message` is the first one emitted.
+
+## Available codes
+
+| Code | Type | Severity | Triggered when |
+|---|---|---|---|
+| `:chunk_size_default` | `:config` | `:warn` | `each_chunk` is called without `chunk_size:` and the default of `100` is used. |
+| `:header_a_method` | `:deprecation` | `:warn` | The deprecated `Reader#headerA` accessor is called. |
+| `:utf8_missing_binary_mode` | `:encoding` | `:warn` | UTF-8 input is being processed but the IO was not opened with `"b:utf-8"`. |
+| `:no_clear_row_sep` | `:row_sep` | `:error` | Auto-detection found a true tie between separators after scanning 64KB. Falls back to `"\n"` — silent miss-parse risk. |
+| `:no_row_sep_found` | `:row_sep` | `:error` | No known row separator was found in the first 64KB. Falls back to `"\n"`. Likely an exotic separator like `\u2028`. |
+
+## Log sink routing
+
+When the warning is emitted, the sink is selected at Reader construction time:
+
+* **Rails.logger present** — the warning is routed through `Rails.logger` at
+  the declared `severity`. `Rails.logger.warn(...)`, `Rails.logger.error(...)`,
+  etc.
+* **No Rails.logger** — falls back to `Kernel#warn` (writes to `$stderr`).
+
+Detection is one-shot at construct time, so there is no per-call overhead.
+
+### In a Rails app
+
+No setup needed. SmarterCSV detects `Rails.logger` automatically, and warnings appear in your Rails log at their declared severity:
+
+```ruby
+SmarterCSV.process('data.csv')
+# In log/development.log:
+# [WARN]  SmarterCSV: chunk_size not set, defaulting to 100. ...
+```
+
+### Without Rails (CLI scripts, plain Ruby, Sinatra, etc.)
+
+Falls back to `Kernel#warn`, which writes to `$stderr`:
+
+```ruby
+SmarterCSV.process('data.csv')
+# stderr:
+# SmarterCSV: chunk_size not set, defaulting to 100. ...
+```
+
+The programmatic `reader.warnings` / `SmarterCSV.warnings` collection is identical in both modes — you can always inspect warnings without parsing log output.
+
+## Suppressing warnings
+
+Pass `verbose: :quiet` to suppress both the recording and the log emission of
+all warnings. Currently this affects every code listed above.
+
+```ruby
+SmarterCSV.process('data.csv', verbose: :quiet)
+SmarterCSV.warnings   # => []
+```
+
+> ⚠️ Suppressing `:row_sep` warnings hides genuine silent miss-parse risk on
+> ambiguous files. Prefer passing `row_sep:` explicitly over silencing.
+
+----------------
+
+PREVIOUS: [Bad Row Quarantine](./bad_row_quarantine.md) | NEXT: [Instrumentation Hooks](./instrumentation.md) | UP: [README](../README.md)