smarter_csv 1.16.4 → 1.17.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f0dc97fe8b296d479efa58b5e404636fe66dbe768e032de987e4c2736b619a4
-  data.tar.gz: 6ffaa0b2f74fb6a48c22a28c21a254a0e9b962bcfb9d1b979e72e54ae446a5c1
+  metadata.gz: 702bd7049e83c0beb85f0ca11a122e6f1659eddef6afec66eaf1c37c5b30f43f
+  data.tar.gz: dd1915694d041c9b631324de7408f46fc8f426f9e1c60136c35a8f1e754d4590
 SHA512:
-  metadata.gz: 8e16f3d049432df188373da120fd4d5f04fd4a49d6a3bb3e91abf6f5722fa6fc90ee302e6db5348349f65290226e0da046cb10d71b2d267fc1be4d63af18107c
-  data.tar.gz: 473ce5f7d1b2bceb7a82898b90c9a19e4076675eea1d748af97d5c7c04abaf15e1cf95a8b774bab40ba5f36cad6ad517e874eab3e63854302ea9d6d4465fefc8
+  metadata.gz: fa00d07c21cffa711a43ecb4622ad3a09b667f1c1965ad26bee864ada6a3c168076ec04550781c75c4f0acbb28fcab60001278f459cf3065cceaef6820764e30
+  data.tar.gz: eab835a356e5343e20a5cc0784ffd9aafa8ab631256d412ec570a0192060b8f3e9c6f619db36e59494364f6c51cda16a9c434c2565ef5a5b6e23f4813a7eaaef

data/.rubocop.yml

@@ -13,6 +13,12 @@ Layout/SpaceInsideHashLiteralBraces:
 Layout/SpaceAroundOperators:
   Enabled: false
 
+Lint/ConstantDefinitionInBlock:
+  Enabled: false
+
+Lint/UnderscorePrefixedVariableName:
+  Enabled: false
+
 Metrics/AbcSize:
   Enabled: false
 
@@ -37,6 +43,9 @@ Metrics/ModuleLength:
 Metrics/PerceivedComplexity:
   Enabled: false
 
+Naming/MethodParameterName:
+  Enabled: false
+
 Naming/PredicateName:
   Enabled: false
 
@@ -156,7 +165,7 @@ Style/SymbolArray:
 Style/SymbolProc: # old Ruby versions can't do this
   Enabled: false
 
-Style/TernaryParentheses:
+Style/TernaryParentheses: # parentheses are good!
   Enabled: false
 
 Style/TrailingCommaInArrayLiteral:

data/CHANGELOG.md

@@ -1,6 +1,60 @@
 
 # SmarterCSV 1.x Change Log
 
+## 1.17.0 (NOT RELEASED)
+
+RSpec tests: **1,434 → 2,210** (+776 tests)
+
+### New Features
+
+* **Streaming IO support** — SmarterCSV now works with non-seekable IO sources such as pipes, STDIN, and Zlib streams.
+  A rewindable peek buffer transparently captures the first bytes of the stream so that `row_sep` and `col_sep` auto-detection can replay them without requiring the underlying source to support `rewind` or `seek`.
+
+* **Structured warnings** — auto-detection and configuration warnings are now collected on the Reader as a deduped histogram:
+
+  ```ruby
+  reader = SmarterCSV::Reader.new('data.csv')
+  reader.process
+  reader.warnings  # => [{ type:, code:, severity:, message:, count: }, ...]
+  ```
+
+  Repeated warnings of the same `(type, code)` are deduped — `count` tracks occurrences. Available codes today: `:chunk_size_default`, `:header_a_method`, `:utf8_missing_binary_mode`, `:no_clear_row_sep`, `:no_row_sep_found`.
+
+* **Class-level `SmarterCSV.warnings`** accessor — mirrors `SmarterCSV.errors`. Per-thread, cleared at the start of each `.process` / `.parse` / `.each` / `.each_chunk` call. Safe under Puma/Sidekiq.
+
+* **Rails.logger routing** — when `Rails.logger` is present, warnings are routed through it at the severity declared at the call site (`:debug` / `:info` / `:warn` / `:error` / `:fatal`); otherwise `Kernel#warn` is used as a fallback. Detection is cached at construct time, no per-call overhead.
+
+### Improvements
+
+* Improved auto-detection of `row_sep` and `col_sep` — giving more accurate results on files with comment headers.
+
+* Larger scan window for accurate row separator detection on files with wide headers or long first lines.
+
+* `guess_line_ending` now scans the input in chunks up to a 64KB hard cap, returning as soon as one separator has a clear majority. Near-tie chunk-boundary artifacts no longer cause spurious warnings; only true ties at the hard cap fall back to `"\n"` and emit a `:no_clear_row_sep` warning at `:error` severity (silent miss-parse risk).
+
+### New / Changed Options
+
+* **`buffer_size` is now a public option** — peek buffer chunk size for non-seekable inputs (pipes, gzip readers, HTTP/S3 bodies). Default `16_384`. Out-of-range values warn and clamp to the supported range rather than raising.
+
+* **`auto_row_sep_chars` default changed to `4096`** (was `500` in 1.16.x). Sized to cover wide-header CSVs in a single read. Bump it higher if your files have very wide headers or long comment preambles.
+
+### Bug Fixes
+
+* **Files ending in a lone `\r`** are now correctly detected as `\r`-terminated instead of falling through to a "no clear row separator" warning.
+
+* **`remove_empty_values` now treats Unicode whitespace as empty** — a field containing only whitespace, including characters like non-breaking space (U+00A0) or ideographic space (U+3000), is now dropped, the same way Ruby's `String#blank?` behaves. Previously only ASCII whitespace counted (and only Rails apps got the Unicode behavior, via `blank?` — an inconsistency that's now gone). Behavior is identical with or without the C extension.
+
+* **`remove_zero_values` now also removes signed zeros** — `+0`, `-0`, `-0.0`, `+0.00`, etc. are recognized as zero and dropped, just like `0` and `0.0`. (Only applies when `remove_zero_values: true`, which is off by default.)
+
+### Performance
+
+Measured against 1.16.4 (Apple M4, Ruby 3.4.7):
+
+* **C-accelerated path (the default):** quote-heavy, large-field, and wide CSVs parse meaningfully faster — roughly **7–22% faster** (city/address-style files ~10–12%; long-field and wide files the most). CSVs with very short lines and many tiny fields are up to ~3% slower — a side effect of the larger default auto-detection scan window (see `auto_row_sep_chars`); set it back to a smaller value if that matters for your workload. Net: solid wins where there's real per-row work, a small cost on the trivially-cheap cases.
+* **Ruby fallback path (`acceleration: false`):** faster on nearly every file — typically **3–20% faster** than 1.16.4, with the biggest gains on wide and many-small-field CSVs.
+
+Per-file breakdown: [`docs/releases/1.17.0/performance_notes.md`](docs/releases/1.17.0/performance_notes.md).
+
 ## 1.16.4 (2026-04-21) — Bug Fixes
 
 RSpec tests: **1,434 → 1,467** (+33 tests)

data/Gemfile

@@ -5,12 +5,17 @@ source 'https://rubygems.org'
 # Specify your gem's dependencies in smarter_csv.gemspec
 gemspec
 
-gem "rake"
-gem "rake-compiler"
+group :development do
+  gem "rake"
+  gem "rake-compiler"
+  gem "ostruct"          # silences rake's stdlib-deprecation warning during dev
+  gem "rubocop"
+end
 
-gem "awesome_print"
-gem 'pry'
-gem "rubocop"
+group :development, :test do
+  gem "awesome_print"
+  gem "pry"              # required in spec_helper.rb; also useful in dev console
+end
 
 group :test do
   gem "rspec"

data/README.md

@@ -14,9 +14,13 @@
 
   Beyond raw speed, SmarterCSV is designed to provide a significantly more convenient and developer-friendly interface than traditional CSV libraries. Instead of returning raw arrays that require substantial post-processing, SmarterCSV produces Rails-ready hashes for each row, making the data immediately usable with ActiveRecord, Sidekiq pipelines, parallel processing, and JSON-based workflows such as S3.
 
+  In a Rails app, warnings auto-route through `Rails.logger` and instrumentation hooks compose with `ActiveSupport::Notifications` — no setup required. Outside Rails, warnings fall back to `$stderr` and the same APIs work without any framework dependency.
+
   The library includes intelligent defaults, automatic detection of column and row separators, and flexible header/value transformations. These features eliminate much of the boilerplate typically required when working with CSV data and help keep ingestion code concise and maintainable.
 
-  For large files, SmarterCSV supports both chunked processing (arrays of hashes) and streaming via Enumerable APIs, enabling efficient batch jobs and low-memory pipelines. The C acceleration further optimizes the full ingestion path — including parsing, hash construction, and conversions — so performance gains reflect real-world workloads, not just tokenizer benchmarks.
+  For large files, SmarterCSV supports both chunked processing (arrays of hashes) and streaming via Enumerable APIs, enabling efficient batch jobs and low-memory pipelines.
+  As of 1.17.0, SmarterCSV also accepts **non-seekable streaming inputs** — pipes, `STDIN`, `Zlib::GzipReader`, and HTTP responses — with no need to materialize the file on disk first.
+  The C acceleration further optimizes the full ingestion path — including parsing, hash construction, and conversions — so performance gains reflect real-world workloads, not just tokenizer benchmarks.
 
   The interface is intentionally designed to robustly handle messy real-world CSV while keeping application code clean. Developers can easily map headers, skip unwanted rows, quarantine problematic data, and transform values on the fly without building custom post-processing pipelines. See [Real-World CSV Files](docs/real_world_csv.md) for a comprehensive guide to production CSV patterns.
 
@@ -33,22 +37,33 @@ SmarterCSV is designed for **real-world CSV processing**, returning fully usable
 
 For a fair comparison, `CSV.table` is the closest Ruby CSV equivalent to SmarterCSV.
 
-| Comparison (SmarterCSV 1.16.0, C-accelerated)  | Range                   |
+| Comparison (SmarterCSV 1.17.0, C-accelerated)  | Range                   |
 |-------------------------------------------------|-------------------------|
-| vs SmarterCSV 1.15.2 (with C acceleration)      | up to 2.4× faster       |
-| vs SmarterCSV 1.14.4 (with C acceleration)      | 9×–65× faster           |
-| vs SmarterCSV 1.14.4 (Ruby path)                | 1.7×–10.6× faster       |
-| vs CSV.read  (arrays of arrays)                 | 1.7×–8.6× faster        |
-| vs CSV.table (arrays of hashes)                 | 7×–129× faster          |
-| vs ZSV (arrays of hashes, equiv. output)        | 1.1×–6.6× faster †      |
+| vs SmarterCSV 1.15.2 (with C acceleration)      | up to 2.8× faster       |
+| vs SmarterCSV 1.14.4 (with C acceleration)      | 9×–82× faster           |
+| vs SmarterCSV 1.14.4 (Ruby path)                | 2.4×–19.8× faster       |
+| vs CSV.read  (arrays of arrays)                 | 1.3×–7.9× faster        |
+| vs CSV.table (arrays of hashes)                 | 4.9×–132× faster        |
+| vs ZSV 1.3.0 (arrays of hashes, equiv. output)  | 1.1×–6.6× faster †      |
+
+† SmarterCSV faster on 15 of 16 files. ZSV raw arrays (no hashes, no conversions) are 2×–14× faster — but that omits the post-processing work needed to produce usable output. ZSV row carried over from the 1.16.0 benchmark; not re-measured for 1.17.0.
+
+_Benchmarks: 19 CSV files (20k–240k rows), Ruby 3.4.7, Apple M4._
 
-† SmarterCSV faster on 15 of 16 files. ZSV raw arrays (no hashes, no conversions) are 2×–14× faster — but that omits the post-processing work needed to produce usable output.
+> ⁉️ **Why these numbers look a touch lower than 1.16.0 charts?**
+> TL;DR: because we use different statistic methods.
+>
+> Earlier versions of these benchmarks reported the best-of-N sample (the absolute `min` / fastest run) for each measurement. A single lucky run — empty caches lining up, no scheduler interrupts — could shave up to ~10% off and become the headline number. I think that would be misleading.
+> Because of that, we've switched to the 10th-percentile (`p10`) of multiple runs of 40 samples, which discards roughly the four luckiest runs and reports a time much closer to what you'll actually observe in production. On noisier fixtures `p10` is ~5–10% above `min`; on quiet ones it's within 1%. The relative ordering between versions and adapters is unchanged; the absolute speedup figures are simply more honest.
 
-_Benchmarks: 19 CSV files (20k–80k rows), Ruby 3.4.7, Apple M1._
+### SmarterCSV vs Ruby CSV
+![SmarterCSV 1.17.0 vs Ruby CSV 3.3.5 speedup](images/SmarterCSV_1.17.0_vs_RubyCSV_3.3.5_speedup.svg)
 
-![SmarterCSV 1.16.0 vs Ruby CSV 3.3.5 speedup](images/SmarterCSV_1.16.0_vs_RubyCSV_3.3.5_speedup.png)
+### SmarterCSV C Path
+![SmarterCSV 1.17.0 vs previous versions — C-accelerated path](images/SmarterCSV_1.17.0_vs_previous_C-speedup.svg)
 
-![SmarterCSV 1.16.0 vs previous versions — C-accelerated path](images/SmarterCSV_1.16.0_vs_previous_C-speedup.svg)
+### SmarterCSV Ruby Path
+![SmarterCSV 1.17.0 vs previous versions — Ruby path](images/SmarterCSV_1.17.0_vs_previous_Rb-speedup.svg)
 
 See [SmarterCSV 1.15.2: Faster Than Raw CSV Arrays](https://tilo-sloboda.medium.com/smartercsv-1-15-2-faster-than-raw-csv-arrays-benchmarks-zsv-and-the-full-pipeline-2c12a798032e) and [PR #319](https://github.com/tilo/smarter_csv/pull/319) for more details.
 
@@ -61,7 +76,7 @@ It's a one-line change:
 # Before
 rows = CSV.table('data.csv').map(&:to_h)
 
-# After — up to 129× faster, same symbol keys
+# After — up to 132× faster, same symbol keys
 rows = SmarterCSV.process('data.csv')
 ```
 
@@ -124,6 +139,23 @@ strip_whitespace → nil_values_matching → remove_empty_values → remove_zero
 
 Each step is individually configurable. See [Data Transformations](docs/data_transformations.md) and [Value Converters](docs/value_converters.md) for details.
 
+### Value Converters
+
+Per-column lambdas convert raw strings into typed values — dates, currency, booleans:
+
+```ruby
+require 'date'
+
+data = SmarterCSV.process('orders.csv',
+  value_converters: {
+    dob:    ->(v) { v && Date.strptime(v, '%m/%d/%Y') },
+    price:  ->(v) { v&.delete('$,')&.to_f },
+    active: ->(v) { v&.match?(/\Atrue\z/i) },
+  })
+```
+
+See [Value Converters](docs/value_converters.md).
+
 ### Batch Processing:
 
 Processing large CSV files in chunks minimizes memory usage and enables powerful workflows:
@@ -147,6 +179,8 @@ SmarterCSV.process(filename, chunk_size: 100) do |chunk|
 end
 ```
 
+See [Batch Processing](docs/batch_processing.md) for chunk sizing, `each_chunk`, and parallel-worker patterns.
+
 ### Modern Enumerator API:
 
 `Reader#each` is the modern, idiomatic way to process rows — `Reader` includes `Enumerable`, so all standard Ruby methods work:
@@ -166,6 +200,29 @@ first_ten = reader.lazy.select { |h| h[:active] }.first(10)
 reader.each_slice(500) { |batch| MyModel.insert_all(batch) }
 ```
 
+See [The Basic Read API](docs/basic_read_api.md) for the full `Reader` interface.
+
+### Streaming / Non-Seekable Inputs (1.17.0+):
+
+SmarterCSV reads directly from any IO — no need to materialize the file on disk first. Auto-detection works on streaming inputs without rewinding; the first chunk is buffered transparently.
+
+```ruby
+# Gzipped CSV — stream-decompressed, never written to disk
+require 'zlib'
+Zlib::GzipReader.open('huge.csv.gz') do |io|
+  SmarterCSV.process(io) { |row| MyModel.upsert(row.first) }
+end
+
+# STDIN / pipes
+SmarterCSV.process($stdin) { |row, _| ... }
+
+# HTTP response body
+require 'open-uri'
+URI.open('https://example.com/data.csv') { |io| SmarterCSV.process(io) }
+```
+
+See [Row and Column Separators](docs/row_col_sep.md) for how `:auto` detection works on non-seekable streams, and [Configuration Options](docs/options.md) for `buffer_size` (the peek-buffer chunk size).
+
 ### Bad Row Handling:
 
 SmarterCSV can quarantine malformed rows instead of crashing the entire import:
@@ -182,7 +239,33 @@ end
 
 See [Bad Row Quarantine](docs/bad_row_quarantine.md) for full details including `bad_row_limit` and `field_size_limit`.
 
-See [13 Examples](docs/examples.md) for more, including value converters, header validation, writing CSV, encoding handling, and resumable Rails ActiveJob imports.
+### Header Validation:
+
+Raise early if the file is missing required columns, before any data row is processed:
+
+```ruby
+begin
+  SmarterCSV.process('transactions.csv',
+    required_keys: [:account_id, :amount, :currency])
+rescue SmarterCSV::MissingKeys => e
+  abort "CSV missing columns: #{e.keys.join(', ')}"
+end
+```
+
+See [Header Validations](docs/header_validations.md).
+
+### Writing CSV:
+
+```ruby
+SmarterCSV.generate('output.csv') do |csv|
+  csv << { name: 'Alice', age: 30, city: 'New York' }
+  csv << { name: 'Bob',   age: 25, city: 'Chicago'  }
+end
+```
+
+Hashes (not arrays) make column-shift bugs impossible — adding a column never silently misaligns existing rows. See [The Basic Write API](docs/basic_write_api.md) for header renaming, value converters, and ordered output.
+
+See [18 Examples](docs/examples.md) for more, including encoding and preamble handling, key mapping, instrumentation hooks, and resumable Rails ActiveJob imports.
 
 ## Requirements
 
@@ -223,6 +306,7 @@ Or install it yourself as:
   * [Data Transformations](docs/data_transformations.md)
   * [Value Converters](docs/value_converters.md)
   * [Bad Row Quarantine](docs/bad_row_quarantine.md)
+  * [Warnings](docs/warnings.md)
   * [Instrumentation Hooks](docs/instrumentation.md)
   * [Examples](docs/examples.md)
   * [Real-World CSV Files](docs/real_world_csv.md)

data/TO_DO.md

@@ -0,0 +1,109 @@
+# SmarterCSV v2.0 TO DO List
+
+DONE:
+[X] Don't call rewind on filehandle
+[X] use Procs for validations and transformatoins [issue #118](https://github.com/tilo/smarter_csv/issues/118)
+[X] skip file opening, allow reading from CSV string, e.g. reading from S3 file [issue #120](https://github.com/tilo/smarter_csv/issues/120). Or stream large file from S3 (linked in the issue)
+[X] [2.0 BUG]  convert_to_float saves Proc as @@convert_to_integer [issue #157](https://github.com/tilo/smarter_csv/issues/157)
+[X] add enumerable to speed up parallel processing [issue #66](https://github.com/tilo/smarter_csv/issues/66), [issue #32](https://github.com/tilo/smarter_csv/issues/32)
+[X] Provide an example for custom Procs for hash_transformations in the docs [issue #174](https://github.com/tilo/smarter_csv/issues/174)
+[X] Collect all Errors, before surfacing them. Avoid throwing an exception on the first error [issue #133](https://github.com/tilo/smarter_csv/issues/133)
+
+
+Partially Done:
+[ ] make @errors and @warnings work [issue #118](https://github.com/tilo/smarter_csv/issues/118)
+
+StilL TO DO:
+[ ] Replace remove_empty_values: false [issue #213](https://github.com/tilo/smarter_csv/issues/213)
+
+Arguably by design (e.g. exclude these columns from conversion and have them returned as a string)
+[ ] [2.0 BUG] :convert_values_to_numeric_unless_leading_zeros drops leading zeros [issue #151](https://github.com/tilo/smarter_csv/issues/151)
+
+
+## Numeric conversion: align the Ruby fallback path with the C path (permissive)
+
+Context: `convert_values_to_numeric` runs in two places that currently DISAGREE on edge cases:
+  - C path (`acceleration: true`, the default): `ext/smarter_csv/smarter_csv.c#try_numeric_conversion`
+    uses `strtol`/`strtod` (base 10; float branch only entered when the field contains a `.`).
+  - Ruby fallback (`acceleration: false`): `lib/smarter_csv/hash_transformations.rb` uses the
+    strict regex `NUMERIC_REGEX = /\A[+-]?\d+(?:\.\d+)?\z/` plus `to_i` / `to_f`.
+
+Divergence (verified empirically):
+  | value     | C path           | Ruby fallback     |
+  |-----------|------------------|-------------------|
+  | ".5"      | 0.5 (Float)      | ".5" (String)     |
+  | "3."      | 3.0 (Float)      | "3." (String)     |
+  | "1.5e3"   | 1500.0 (Float)   | "1.5e3" (String)  |
+  | "1.0e10"  | 10000000000.0    | "1.0e10" (String) |
+
+Decision: the C path's permissive behavior (corner cases + scientific notation) is the intended
+contract. Fix = make the Ruby fallback match the C path. Do NOT tighten the C path.
+
+Ruby-side changes (in `hash_transformations.rb`):
+  1. Swap NUMERIC_REGEX for a permissive one:
+       /\A[+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?\z/
+     matches 1, 1., 1.5, .5, 1e3, 1.5e3, -3.14e-2, etc.; still rejects ".", "e3", "1.2.3",
+     "1_000", "0x1F".
+  2. Add `DOT_BYTE = '.'.ord` (46) and include it in the first-byte fast-reject's allowed set
+     (the C pre-check already allows a leading `.`; without this, ".5" gets rejected on byte 0).
+  3. Int-vs-float decision: `(v.include?('.') || v.include?('e') || v.include?('E')) ? v.to_f : v.to_i`
+     (currently only checks for `.`).
+
+Stays a string on BOTH paths (no change needed, but worth characterization tests — there are
+currently NONE):
+  - "010" => 10  (NOT octal 8 — both paths use base-10 conversion: String#to_i / strtol(.,10).
+    A switch to Kernel#Integer() would break this. Lock it down with a test.)
+  - "0x1F", "0b101", "0o17"  => string  (radix prefixes not honored by base-10 conversion)
+  - "1_000"                  => string  (underscores)
+  - "1,200.00", "1.300,00"   => string  (thousands sep / decimal comma — strtod stops at the
+    separator → not fully consumed; regex rejects. This is the only safe behavior; "1,200" is
+    genuinely ambiguous. Locale-specific number formats are the caller's job via value_converters.)
+
+NOT doing: locale sniffing (read LC_NUMERIC at init and adjust the regexes). Rejected because
+the machine locale tells you nothing about the file's number format, it breaks reproducibility
+(same code + same file → different results on a US vs EU box), and `,` can't be both col_sep and
+decimal separator anyway. Note `strtod` IS locale-sensitive (LC_NUMERIC) but it's dormant — Ruby
+runs in the C/POSIX locale; don't deliberately activate it.
+
+When done: parity tests (`[true, false].each`) for the now-consistent set (.5, 3., 1.5e3, 1e3)
+plus characterization tests for the stays-a-string set above; CHANGELOG line noting the Ruby
+fallback's numeric conversion now accepts scientific notation and bare-dot forms, matching the
+accelerated path. Behavior change affects `acceleration: false` users only — and aligns them with
+the default.
+
+
+## Warn once when the C extension didn't load on a platform that supports it
+
+Context: `acceleration: true` is the default. When the C extension fails to build / isn't loaded,
+SmarterCSV silently falls back to the Ruby parser — graceful degradation by design (so the gem
+keeps working for users with broken toolchains, JRuby, TruffleRuby, etc.). Today there is no
+signal to the user that they're not getting the C path; their CSV parsing is just slower than
+they might have expected.
+
+Idea: emit a one-time warning when:
+  * the C extension is NOT loaded — `!SmarterCSV::Parser.respond_to?(:parse_csv_line_c)`, AND
+  * the platform is one where it *should* be available — `RUBY_ENGINE == 'ruby'` (MRI / CRuby).
+    JRuby and TruffleRuby don't load CRuby C extensions natively; nothing for the user to do.
+
+Where to fire:
+  * NOT at `require 'smarter_csv'` time — Rails.logger typically isn't set up yet, so any
+    "route through the warnings system" code would just fall through to `Kernel#warn` anyway,
+    and the warning would land in stderr instead of the Rails log where ops would see it.
+  * At first `Reader.new` / `SmarterCSV.process` call — Rails has booted, the existing
+    routing-through-Rails.logger-or-Kernel#warn infra works, and the existing deduped warnings
+    histogram means it fires once per process regardless of how many parse calls.
+
+Implementation sketch:
+  * Add a new warning code (e.g. `:c_extension_unavailable`) alongside the existing ones
+    (`:chunk_size_default`, `:header_a_method`, `:utf8_missing_binary_mode`, ...).
+  * Severity `:warn`. Suppressible via the existing `verbose: :quiet`.
+  * Message points at the fix — e.g. "C acceleration extension not loaded on this Ruby; using
+    Ruby parser. To enable acceleration, reinstall with `gem pristine smarter_csv` and check
+    the build log." Plus a link/pointer to a troubleshooting section in the docs.
+
+Bonus: add a public predicate `SmarterCSV.acceleration_available?` returning
+`Parser.respond_to?(:parse_csv_line_c)`. Zero noise, useful for scripts / CI / future spec
+files that want to branch on the environment fact rather than guess.
+
+NOT doing: a banner at `require` time (every Rails app would print it at boot, too noisy);
+warning when `acceleration: false` was explicitly chosen (the user knows what they're doing).

data/docs/_introduction.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)

data/docs/bad_row_quarantine.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [**Bad Row Quarantine**](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)
@@ -339,4 +340,4 @@ Normal rows (where the entire line fits within the limit) bypass per-field check
 
 --------------------
 
-PREVIOUS: [Value Converters](./value_converters.md) | NEXT: [Instrumentation Hooks](./instrumentation.md) | UP: [README](../README.md)
+PREVIOUS: [Value Converters](./value_converters.md) | NEXT: [Warnings](./warnings.md) | UP: [README](../README.md)

data/docs/basic_read_api.md

@@ -123,8 +123,9 @@ reader.each do |hash|
   MyModel.upsert(hash)
 end
 
-puts reader.headers       # accessible after processing
+puts reader.headers        # accessible after processing
 puts reader.errors.inspect
+puts reader.warnings       # see [Warnings](./warnings.md)
 ```
 
 ### Returns an Enumerator when called without a block
@@ -185,6 +186,10 @@ reader.each { |hash| MyModel.upsert(hash) }
 reader.errors[:bad_rows].each { |rec| puts "Bad row: #{rec[:error_message]}" }
 ```
 
+### Read-Transform-Write Pipelines
+
+Composing `SmarterCSV.each` with `SmarterCSV.generate` is the idiomatic replacement for Ruby's `CSV.filter` — read CSV, mutate each row, write the result. See [Examples → Filtering and Transforming a CSV File](./examples.md#example-19-filtering-and-transforming-a-csv-file) for the full set of patterns (file → file, STDIN → STDOUT, gzip → gzip, header renaming).
+
 ---
 
 ## Value Transformation Pipeline

data/docs/basic_write_api.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)
@@ -188,6 +189,31 @@ File.open('output.csv', 'w') do |f|
 end
 ```
 
+**Write to STDOUT (e.g. piping to another process):**
+
+```ruby
+SmarterCSV.generate($stdout) do |csv|
+  records.each { |r| csv << r }
+end
+```
+
+Useful in CLI scripts: `ruby export.rb | gzip > out.csv.gz`.
+
+**Stream a CSV upload to S3 — never written to disk:**
+
+```ruby
+require 'aws-sdk-s3'
+
+obj = Aws::S3::Object.new(bucket_name: 'exports', key: 'reports/daily.csv')
+obj.upload_stream do |stream|
+  SmarterCSV.generate(stream) do |csv|
+    Order.find_each { |o| csv << o.attributes }
+  end
+end
+```
+
+`upload_stream` performs a multipart upload, so the CSV is sent to S3 incrementally as it's generated — memory usage stays flat regardless of result size.
+
 ### Full Interface
 
 The full interface gives you direct access to the `Writer` instance, which is useful when you
@@ -616,6 +642,10 @@ end
 > **Note:** `write_headers: false` only suppresses the header line. All other
 > options (`col_sep:`, `row_sep:`, `value_converters:`, etc.) apply as normal.
 
+## Read-Transform-Write Pipelines
+
+Pairing `SmarterCSV.generate` with `SmarterCSV.each` on the read side is the idiomatic replacement for Ruby's `CSV.filter`. See [Examples → Filtering and Transforming a CSV File](./examples.md#example-19-filtering-and-transforming-a-csv-file) for the full set of patterns, including streaming gzip → gzip pipelines.
+
 ## More Examples
 
 Check out the [RSpec tests](../spec/smarter_csv/writer_spec.rb) for more examples.

data/docs/batch_processing.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)
@@ -210,6 +211,30 @@ SmarterCSV::Reader.new('products.csv', chunk_size: 25).each_chunk do |chunk, _in
 end
 ```
 
+## Example: Resumable Import (Plain Ruby)
+
+Track the chunk cursor in a JSON state file so an interrupted import can resume where it left off — no Rails / ActiveJob required:
+
+```ruby
+require 'json'
+
+STATE_FILE = '/var/run/import.state.json'
+
+state = File.exist?(STATE_FILE) ? JSON.parse(File.read(STATE_FILE)) : { 'cursor' => 0 }
+
+SmarterCSV.process('import.csv', chunk_size: 500) do |chunk, chunk_index|
+  next if chunk_index < state['cursor']  # skip already-processed chunks on resume
+
+  MyModel.import!(chunk)
+  state['cursor'] = chunk_index + 1
+  File.write(STATE_FILE, JSON.dump(state))
+end
+
+File.delete(STATE_FILE)  # done — clear the cursor
+```
+
+If the process is killed at chunk 7, the next run skips chunks 0–6 quickly via `next` and resumes at chunk 7. For Rails 8.1+ projects, see [Examples → Resumable CSV Import with Rails ActiveJob](./examples.md#example-12-resumable-csv-import-with-rails-activejob-rails-81) for the framework-native version.
+
 ## Example: Reading a CSV from S3
 
 SmarterCSV accepts any IO-like object, so you can stream directly from S3 without

data/docs/column_selection.md

@@ -16,6 +16,7 @@
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)

data/docs/data_transformations.md

@@ -16,6 +16,7 @@
   * [**Data Transformations**](./data_transformations.md)
   * [Value Converters](./value_converters.md)
   * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Warnings](./warnings.md)
   * [Instrumentation Hooks](./instrumentation.md)
   * [Examples](./examples.md)
   * [Real-World CSV Files](./real_world_csv.md)