smarter_csv 1.17.0.pre5 → 1.17.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5d2154634f98b9df235995b9c6368e6208027d31da8d6d80ad09a526dd51fbf0
-  data.tar.gz: 62dd06196aef83b0e2c7dd6391ce9004fe95ba17310eec5f8ae9f2a74c2008a8
+  metadata.gz: 702bd7049e83c0beb85f0ca11a122e6f1659eddef6afec66eaf1c37c5b30f43f
+  data.tar.gz: dd1915694d041c9b631324de7408f46fc8f426f9e1c60136c35a8f1e754d4590
 SHA512:
-  metadata.gz: fdac413102754c859247b876f8a2130ff7dcf700c3531e4c58e72c36e0e53081c0ad7a93654d343428ed7c1c91dd0a35aa90e94877df72dab2b0aa5d9ae0bf65
-  data.tar.gz: c800d6e4c807ff2c502ac9c7f819de5b6eef582c587b505e1efe2b5ee284a941f809bdb7166d638ac3ee5595763116c72ccfbe76d30b83f83793a53b1f728a6c
+  metadata.gz: fa00d07c21cffa711a43ecb4622ad3a09b667f1c1965ad26bee864ada6a3c168076ec04550781c75c4f0acbb28fcab60001278f459cf3065cceaef6820764e30
+  data.tar.gz: eab835a356e5343e20a5cc0784ffd9aafa8ab631256d412ec570a0192060b8f3e9c6f619db36e59494364f6c51cda16a9c434c2565ef5a5b6e23f4813a7eaaef

data/.rubocop.yml

@@ -13,6 +13,9 @@ Layout/SpaceInsideHashLiteralBraces:
 Layout/SpaceAroundOperators:
   Enabled: false
 
+Lint/ConstantDefinitionInBlock:
+  Enabled: false
+
 Lint/UnderscorePrefixedVariableName:
   Enabled: false
 

data/CHANGELOG.md

@@ -1,9 +1,9 @@
 
 # SmarterCSV 1.x Change Log
 
-## 1.17.0.pre5 (2026-04-28)
+## 1.17.0 (NOT RELEASED)
 
-RSpec tests: **1,434 → 1,905** (+471 tests)
+RSpec tests: **1,434 → 2,210** (+776 tests)
 
 ### New Features
 
@@ -28,12 +28,33 @@ RSpec tests: **1,434 → 1,905** (+471 tests)
 
 * Improved auto-detection of `row_sep` and `col_sep` — giving more accurate results on files with comment headers.
 
-* Default value for `auto_row_sep_chars` changed from `500` to `8192`, providing a larger scan window for accurate row separator detection on files with wide headers or long first lines. 
-Values below `8192` (and `nil` / `0`) are now rejected and fall back to the default `8192` with a warning message.
-  This is a change from the previous `nil` / `0` were documented as "scan whole file".
+* Larger scan window for accurate row separator detection on files with wide headers or long first lines.
 
 * `guess_line_ending` now scans the input in chunks up to a 64KB hard cap, returning as soon as one separator has a clear majority. Near-tie chunk-boundary artifacts no longer cause spurious warnings; only true ties at the hard cap fall back to `"\n"` and emit a `:no_clear_row_sep` warning at `:error` severity (silent miss-parse risk).
 
+### New / Changed Options
+
+* **`buffer_size` is now a public option** — peek buffer chunk size for non-seekable inputs (pipes, gzip readers, HTTP/S3 bodies). Default `16_384`. Out-of-range values warn and clamp to the supported range rather than raising.
+
+* **`auto_row_sep_chars` default changed to `4096`** (was `500` in 1.16.x). Sized to cover wide-header CSVs in a single read. Bump it higher if your files have very wide headers or long comment preambles.
+
+### Bug Fixes
+
+* **Files ending in a lone `\r`** are now correctly detected as `\r`-terminated instead of falling through to a "no clear row separator" warning.
+
+* **`remove_empty_values` now treats Unicode whitespace as empty** — a field containing only whitespace, including characters like non-breaking space (U+00A0) or ideographic space (U+3000), is now dropped, the same way Ruby's `String#blank?` behaves. Previously only ASCII whitespace counted (and only Rails apps got the Unicode behavior, via `blank?` — an inconsistency that's now gone). Behavior is identical with or without the C extension.
+
+* **`remove_zero_values` now also removes signed zeros** — `+0`, `-0`, `-0.0`, `+0.00`, etc. are recognized as zero and dropped, just like `0` and `0.0`. (Only applies when `remove_zero_values: true`, which is off by default.)
+
+### Performance
+
+Measured against 1.16.4 (Apple M4, Ruby 3.4.7):
+
+* **C-accelerated path (the default):** quote-heavy, large-field, and wide CSVs parse meaningfully faster — roughly **7–22% faster** (city/address-style files ~10–12%; long-field and wide files the most). CSVs with very short lines and many tiny fields are up to ~3% slower — a side effect of the larger default auto-detection scan window (see `auto_row_sep_chars`); set it back to a smaller value if that matters for your workload. Net: solid wins where there's real per-row work, a small cost on the trivially-cheap cases.
+* **Ruby fallback path (`acceleration: false`):** faster on nearly every file — typically **3–20% faster** than 1.16.4, with the biggest gains on wide and many-small-field CSVs.
+
+Per-file breakdown: [`docs/releases/1.17.0/performance_notes.md`](docs/releases/1.17.0/performance_notes.md).
+
 ## 1.16.4 (2026-04-21) — Bug Fixes
 
 RSpec tests: **1,434 → 1,467** (+33 tests)

data/Gemfile

@@ -5,12 +5,17 @@ source 'https://rubygems.org'
 # Specify your gem's dependencies in smarter_csv.gemspec
 gemspec
 
-gem "rake"
-gem "rake-compiler"
+group :development do
+  gem "rake"
+  gem "rake-compiler"
+  gem "ostruct"          # silences rake's stdlib-deprecation warning during dev
+  gem "rubocop"
+end
 
-gem "awesome_print"
-gem 'pry'
-gem "rubocop"
+group :development, :test do
+  gem "awesome_print"
+  gem "pry"              # required in spec_helper.rb; also useful in dev console
+end
 
 group :test do
   gem "rspec"

data/README.md

@@ -14,6 +14,8 @@
 
   Beyond raw speed, SmarterCSV is designed to provide a significantly more convenient and developer-friendly interface than traditional CSV libraries. Instead of returning raw arrays that require substantial post-processing, SmarterCSV produces Rails-ready hashes for each row, making the data immediately usable with ActiveRecord, Sidekiq pipelines, parallel processing, and JSON-based workflows such as S3.
 
+  In a Rails app, warnings auto-route through `Rails.logger` and instrumentation hooks compose with `ActiveSupport::Notifications` — no setup required. Outside Rails, warnings fall back to `$stderr` and the same APIs work without any framework dependency.
+
   The library includes intelligent defaults, automatic detection of column and row separators, and flexible header/value transformations. These features eliminate much of the boilerplate typically required when working with CSV data and help keep ingestion code concise and maintainable.
 
   For large files, SmarterCSV supports both chunked processing (arrays of hashes) and streaming via Enumerable APIs, enabling efficient batch jobs and low-memory pipelines.
@@ -35,22 +37,33 @@ SmarterCSV is designed for **real-world CSV processing**, returning fully usable
 
 For a fair comparison, `CSV.table` is the closest Ruby CSV equivalent to SmarterCSV.
 
-| Comparison (SmarterCSV 1.16.0, C-accelerated)  | Range                   |
+| Comparison (SmarterCSV 1.17.0, C-accelerated)  | Range                   |
 |-------------------------------------------------|-------------------------|
-| vs SmarterCSV 1.15.2 (with C acceleration)      | up to 2.4× faster       |
-| vs SmarterCSV 1.14.4 (with C acceleration)      | 9×–65× faster           |
-| vs SmarterCSV 1.14.4 (Ruby path)                | 1.7×–10.6× faster       |
-| vs CSV.read  (arrays of arrays)                 | 1.7×–8.6× faster        |
-| vs CSV.table (arrays of hashes)                 | 7×–129× faster          |
-| vs ZSV (arrays of hashes, equiv. output)        | 1.1×–6.6× faster †      |
+| vs SmarterCSV 1.15.2 (with C acceleration)      | up to 2.8× faster       |
+| vs SmarterCSV 1.14.4 (with C acceleration)      | 9×–82× faster           |
+| vs SmarterCSV 1.14.4 (Ruby path)                | 2.4×–19.8× faster       |
+| vs CSV.read  (arrays of arrays)                 | 1.3×–7.9× faster        |
+| vs CSV.table (arrays of hashes)                 | 4.9×–132× faster        |
+| vs ZSV 1.3.0 (arrays of hashes, equiv. output)  | 1.1×–6.6× faster †      |
+
+† SmarterCSV faster on 15 of 16 files. ZSV raw arrays (no hashes, no conversions) are 2×–14× faster — but that omits the post-processing work needed to produce usable output. ZSV row carried over from the 1.16.0 benchmark; not re-measured for 1.17.0.
+
+_Benchmarks: 19 CSV files (20k–240k rows), Ruby 3.4.7, Apple M4._
 
-† SmarterCSV faster on 15 of 16 files. ZSV raw arrays (no hashes, no conversions) are 2×–14× faster — but that omits the post-processing work needed to produce usable output.
+> ⁉️ **Why these numbers look a touch lower than 1.16.0 charts?**
+> TL;DR: because we use different statistic methods.
+>
+> Earlier versions of these benchmarks reported the best-of-N sample (the absolute `min` / fastest run) for each measurement. A single lucky run — empty caches lining up, no scheduler interrupts — could shave up to ~10% off and become the headline number. I think that would be misleading.
+> Because of that, we've switched to the 10th-percentile (`p10`) of multiple runs of 40 samples, which discards roughly the four luckiest runs and reports a time much closer to what you'll actually observe in production. On noisier fixtures `p10` is ~5–10% above `min`; on quiet ones it's within 1%. The relative ordering between versions and adapters is unchanged; the absolute speedup figures are simply more honest.
 
-_Benchmarks: 19 CSV files (20k–80k rows), Ruby 3.4.7, Apple M1._
+### SmarterCSV vs Ruby CSV
+![SmarterCSV 1.17.0 vs Ruby CSV 3.3.5 speedup](images/SmarterCSV_1.17.0_vs_RubyCSV_3.3.5_speedup.svg)
 
-![SmarterCSV 1.16.0 vs Ruby CSV 3.3.5 speedup](images/SmarterCSV_1.16.0_vs_RubyCSV_3.3.5_speedup.png)
+### SmarterCSV C Path
+![SmarterCSV 1.17.0 vs previous versions — C-accelerated path](images/SmarterCSV_1.17.0_vs_previous_C-speedup.svg)
 
-![SmarterCSV 1.16.0 vs previous versions — C-accelerated path](images/SmarterCSV_1.16.0_vs_previous_C-speedup.svg)
+### SmarterCSV Ruby Path
+![SmarterCSV 1.17.0 vs previous versions — Ruby path](images/SmarterCSV_1.17.0_vs_previous_Rb-speedup.svg)
 
 See [SmarterCSV 1.15.2: Faster Than Raw CSV Arrays](https://tilo-sloboda.medium.com/smartercsv-1-15-2-faster-than-raw-csv-arrays-benchmarks-zsv-and-the-full-pipeline-2c12a798032e) and [PR #319](https://github.com/tilo/smarter_csv/pull/319) for more details.
 
@@ -63,7 +76,7 @@ It's a one-line change:
 # Before
 rows = CSV.table('data.csv').map(&:to_h)
 
-# After — up to 129× faster, same symbol keys
+# After — up to 132× faster, same symbol keys
 rows = SmarterCSV.process('data.csv')
 ```
 
@@ -126,6 +139,23 @@ strip_whitespace → nil_values_matching → remove_empty_values → remove_zero
 
 Each step is individually configurable. See [Data Transformations](docs/data_transformations.md) and [Value Converters](docs/value_converters.md) for details.
 
+### Value Converters
+
+Per-column lambdas convert raw strings into typed values — dates, currency, booleans:
+
+```ruby
+require 'date'
+
+data = SmarterCSV.process('orders.csv',
+  value_converters: {
+    dob:    ->(v) { v && Date.strptime(v, '%m/%d/%Y') },
+    price:  ->(v) { v&.delete('$,')&.to_f },
+    active: ->(v) { v&.match?(/\Atrue\z/i) },
+  })
+```
+
+See [Value Converters](docs/value_converters.md).
+
 ### Batch Processing:
 
 Processing large CSV files in chunks minimizes memory usage and enables powerful workflows:
@@ -149,6 +179,8 @@ SmarterCSV.process(filename, chunk_size: 100) do |chunk|
 end
 ```
 
+See [Batch Processing](docs/batch_processing.md) for chunk sizing, `each_chunk`, and parallel-worker patterns.
+
 ### Modern Enumerator API:
 
 `Reader#each` is the modern, idiomatic way to process rows — `Reader` includes `Enumerable`, so all standard Ruby methods work:
@@ -168,6 +200,29 @@ first_ten = reader.lazy.select { |h| h[:active] }.first(10)
 reader.each_slice(500) { |batch| MyModel.insert_all(batch) }
 ```
 
+See [The Basic Read API](docs/basic_read_api.md) for the full `Reader` interface.
+
+### Streaming / Non-Seekable Inputs (1.17.0+):
+
+SmarterCSV reads directly from any IO — no need to materialize the file on disk first. Auto-detection works on streaming inputs without rewinding; the first chunk is buffered transparently.
+
+```ruby
+# Gzipped CSV — stream-decompressed, never written to disk
+require 'zlib'
+Zlib::GzipReader.open('huge.csv.gz') do |io|
+  SmarterCSV.process(io) { |row| MyModel.upsert(row.first) }
+end
+
+# STDIN / pipes
+SmarterCSV.process($stdin) { |row, _| ... }
+
+# HTTP response body
+require 'open-uri'
+URI.open('https://example.com/data.csv') { |io| SmarterCSV.process(io) }
+```
+
+See [Row and Column Separators](docs/row_col_sep.md) for how `:auto` detection works on non-seekable streams, and [Configuration Options](docs/options.md) for `buffer_size` (the peek-buffer chunk size).
+
 ### Bad Row Handling:
 
 SmarterCSV can quarantine malformed rows instead of crashing the entire import:
@@ -184,7 +239,33 @@ end
 
 See [Bad Row Quarantine](docs/bad_row_quarantine.md) for full details including `bad_row_limit` and `field_size_limit`.
 
-See [13 Examples](docs/examples.md) for more, including value converters, header validation, writing CSV, encoding handling, and resumable Rails ActiveJob imports.
+### Header Validation:
+
+Raise early if the file is missing required columns, before any data row is processed:
+
+```ruby
+begin
+  SmarterCSV.process('transactions.csv',
+    required_keys: [:account_id, :amount, :currency])
+rescue SmarterCSV::MissingKeys => e
+  abort "CSV missing columns: #{e.keys.join(', ')}"
+end
+```
+
+See [Header Validations](docs/header_validations.md).
+
+### Writing CSV:
+
+```ruby
+SmarterCSV.generate('output.csv') do |csv|
+  csv << { name: 'Alice', age: 30, city: 'New York' }
+  csv << { name: 'Bob',   age: 25, city: 'Chicago'  }
+end
+```
+
+Hashes (not arrays) make column-shift bugs impossible — adding a column never silently misaligns existing rows. See [The Basic Write API](docs/basic_write_api.md) for header renaming, value converters, and ordered output.
+
+See [18 Examples](docs/examples.md) for more, including encoding and preamble handling, key mapping, instrumentation hooks, and resumable Rails ActiveJob imports.
 
 ## Requirements
 

data/TO_DO.md

@@ -0,0 +1,109 @@
+# SmarterCSV v2.0 TO DO List
+
+DONE:
+[X] Don't call rewind on filehandle
+[X] use Procs for validations and transformatoins [issue #118](https://github.com/tilo/smarter_csv/issues/118)
+[X] skip file opening, allow reading from CSV string, e.g. reading from S3 file [issue #120](https://github.com/tilo/smarter_csv/issues/120). Or stream large file from S3 (linked in the issue)
+[X] [2.0 BUG]  convert_to_float saves Proc as @@convert_to_integer [issue #157](https://github.com/tilo/smarter_csv/issues/157)
+[X] add enumerable to speed up parallel processing [issue #66](https://github.com/tilo/smarter_csv/issues/66), [issue #32](https://github.com/tilo/smarter_csv/issues/32)
+[X] Provide an example for custom Procs for hash_transformations in the docs [issue #174](https://github.com/tilo/smarter_csv/issues/174)
+[X] Collect all Errors, before surfacing them. Avoid throwing an exception on the first error [issue #133](https://github.com/tilo/smarter_csv/issues/133)
+
+
+Partially Done:
+[ ] make @errors and @warnings work [issue #118](https://github.com/tilo/smarter_csv/issues/118)
+
+StilL TO DO:
+[ ] Replace remove_empty_values: false [issue #213](https://github.com/tilo/smarter_csv/issues/213)
+
+Arguably by design (e.g. exclude these columns from conversion and have them returned as a string)
+[ ] [2.0 BUG] :convert_values_to_numeric_unless_leading_zeros drops leading zeros [issue #151](https://github.com/tilo/smarter_csv/issues/151)
+
+
+## Numeric conversion: align the Ruby fallback path with the C path (permissive)
+
+Context: `convert_values_to_numeric` runs in two places that currently DISAGREE on edge cases:
+  - C path (`acceleration: true`, the default): `ext/smarter_csv/smarter_csv.c#try_numeric_conversion`
+    uses `strtol`/`strtod` (base 10; float branch only entered when the field contains a `.`).
+  - Ruby fallback (`acceleration: false`): `lib/smarter_csv/hash_transformations.rb` uses the
+    strict regex `NUMERIC_REGEX = /\A[+-]?\d+(?:\.\d+)?\z/` plus `to_i` / `to_f`.
+
+Divergence (verified empirically):
+  | value     | C path           | Ruby fallback     |
+  |-----------|------------------|-------------------|
+  | ".5"      | 0.5 (Float)      | ".5" (String)     |
+  | "3."      | 3.0 (Float)      | "3." (String)     |
+  | "1.5e3"   | 1500.0 (Float)   | "1.5e3" (String)  |
+  | "1.0e10"  | 10000000000.0    | "1.0e10" (String) |
+
+Decision: the C path's permissive behavior (corner cases + scientific notation) is the intended
+contract. Fix = make the Ruby fallback match the C path. Do NOT tighten the C path.
+
+Ruby-side changes (in `hash_transformations.rb`):
+  1. Swap NUMERIC_REGEX for a permissive one:
+       /\A[+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?\z/
+     matches 1, 1., 1.5, .5, 1e3, 1.5e3, -3.14e-2, etc.; still rejects ".", "e3", "1.2.3",
+     "1_000", "0x1F".
+  2. Add `DOT_BYTE = '.'.ord` (46) and include it in the first-byte fast-reject's allowed set
+     (the C pre-check already allows a leading `.`; without this, ".5" gets rejected on byte 0).
+  3. Int-vs-float decision: `(v.include?('.') || v.include?('e') || v.include?('E')) ? v.to_f : v.to_i`
+     (currently only checks for `.`).
+
+Stays a string on BOTH paths (no change needed, but worth characterization tests — there are
+currently NONE):
+  - "010" => 10  (NOT octal 8 — both paths use base-10 conversion: String#to_i / strtol(.,10).
+    A switch to Kernel#Integer() would break this. Lock it down with a test.)
+  - "0x1F", "0b101", "0o17"  => string  (radix prefixes not honored by base-10 conversion)
+  - "1_000"                  => string  (underscores)
+  - "1,200.00", "1.300,00"   => string  (thousands sep / decimal comma — strtod stops at the
+    separator → not fully consumed; regex rejects. This is the only safe behavior; "1,200" is
+    genuinely ambiguous. Locale-specific number formats are the caller's job via value_converters.)
+
+NOT doing: locale sniffing (read LC_NUMERIC at init and adjust the regexes). Rejected because
+the machine locale tells you nothing about the file's number format, it breaks reproducibility
+(same code + same file → different results on a US vs EU box), and `,` can't be both col_sep and
+decimal separator anyway. Note `strtod` IS locale-sensitive (LC_NUMERIC) but it's dormant — Ruby
+runs in the C/POSIX locale; don't deliberately activate it.
+
+When done: parity tests (`[true, false].each`) for the now-consistent set (.5, 3., 1.5e3, 1e3)
+plus characterization tests for the stays-a-string set above; CHANGELOG line noting the Ruby
+fallback's numeric conversion now accepts scientific notation and bare-dot forms, matching the
+accelerated path. Behavior change affects `acceleration: false` users only — and aligns them with
+the default.
+
+
+## Warn once when the C extension didn't load on a platform that supports it
+
+Context: `acceleration: true` is the default. When the C extension fails to build / isn't loaded,
+SmarterCSV silently falls back to the Ruby parser — graceful degradation by design (so the gem
+keeps working for users with broken toolchains, JRuby, TruffleRuby, etc.). Today there is no
+signal to the user that they're not getting the C path; their CSV parsing is just slower than
+they might have expected.
+
+Idea: emit a one-time warning when:
+  * the C extension is NOT loaded — `!SmarterCSV::Parser.respond_to?(:parse_csv_line_c)`, AND
+  * the platform is one where it *should* be available — `RUBY_ENGINE == 'ruby'` (MRI / CRuby).
+    JRuby and TruffleRuby don't load CRuby C extensions natively; nothing for the user to do.
+
+Where to fire:
+  * NOT at `require 'smarter_csv'` time — Rails.logger typically isn't set up yet, so any
+    "route through the warnings system" code would just fall through to `Kernel#warn` anyway,
+    and the warning would land in stderr instead of the Rails log where ops would see it.
+  * At first `Reader.new` / `SmarterCSV.process` call — Rails has booted, the existing
+    routing-through-Rails.logger-or-Kernel#warn infra works, and the existing deduped warnings
+    histogram means it fires once per process regardless of how many parse calls.
+
+Implementation sketch:
+  * Add a new warning code (e.g. `:c_extension_unavailable`) alongside the existing ones
+    (`:chunk_size_default`, `:header_a_method`, `:utf8_missing_binary_mode`, ...).
+  * Severity `:warn`. Suppressible via the existing `verbose: :quiet`.
+  * Message points at the fix — e.g. "C acceleration extension not loaded on this Ruby; using
+    Ruby parser. To enable acceleration, reinstall with `gem pristine smarter_csv` and check
+    the build log." Plus a link/pointer to a troubleshooting section in the docs.
+
+Bonus: add a public predicate `SmarterCSV.acceleration_available?` returning
+`Parser.respond_to?(:parse_csv_line_c)`. Zero noise, useful for scripts / CI / future spec
+files that want to branch on the environment fact rather than guess.
+
+NOT doing: a banner at `require` time (every Rails app would print it at boot, too noisy);
+warning when `acceleration: false` was explicitly chosen (the user knows what they're doing).

data/docs/basic_read_api.md

@@ -186,6 +186,10 @@ reader.each { |hash| MyModel.upsert(hash) }
 reader.errors[:bad_rows].each { |rec| puts "Bad row: #{rec[:error_message]}" }
 ```
 
+### Read-Transform-Write Pipelines
+
+Composing `SmarterCSV.each` with `SmarterCSV.generate` is the idiomatic replacement for Ruby's `CSV.filter` — read CSV, mutate each row, write the result. See [Examples → Filtering and Transforming a CSV File](./examples.md#example-19-filtering-and-transforming-a-csv-file) for the full set of patterns (file → file, STDIN → STDOUT, gzip → gzip, header renaming).
+
 ---
 
 ## Value Transformation Pipeline

data/docs/basic_write_api.md

@@ -189,6 +189,31 @@ File.open('output.csv', 'w') do |f|
 end
 ```
 
+**Write to STDOUT (e.g. piping to another process):**
+
+```ruby
+SmarterCSV.generate($stdout) do |csv|
+  records.each { |r| csv << r }
+end
+```
+
+Useful in CLI scripts: `ruby export.rb | gzip > out.csv.gz`.
+
+**Stream a CSV upload to S3 — never written to disk:**
+
+```ruby
+require 'aws-sdk-s3'
+
+obj = Aws::S3::Object.new(bucket_name: 'exports', key: 'reports/daily.csv')
+obj.upload_stream do |stream|
+  SmarterCSV.generate(stream) do |csv|
+    Order.find_each { |o| csv << o.attributes }
+  end
+end
+```
+
+`upload_stream` performs a multipart upload, so the CSV is sent to S3 incrementally as it's generated — memory usage stays flat regardless of result size.
+
 ### Full Interface
 
 The full interface gives you direct access to the `Writer` instance, which is useful when you
@@ -617,6 +642,10 @@ end
 > **Note:** `write_headers: false` only suppresses the header line. All other
 > options (`col_sep:`, `row_sep:`, `value_converters:`, etc.) apply as normal.
 
+## Read-Transform-Write Pipelines
+
+Pairing `SmarterCSV.generate` with `SmarterCSV.each` on the read side is the idiomatic replacement for Ruby's `CSV.filter`. See [Examples → Filtering and Transforming a CSV File](./examples.md#example-19-filtering-and-transforming-a-csv-file) for the full set of patterns, including streaming gzip → gzip pipelines.
+
 ## More Examples
 
 Check out the [RSpec tests](../spec/smarter_csv/writer_spec.rb) for more examples.

data/docs/batch_processing.md

@@ -211,6 +211,30 @@ SmarterCSV::Reader.new('products.csv', chunk_size: 25).each_chunk do |chunk, _in
 end
 ```
 
+## Example: Resumable Import (Plain Ruby)
+
+Track the chunk cursor in a JSON state file so an interrupted import can resume where it left off — no Rails / ActiveJob required:
+
+```ruby
+require 'json'
+
+STATE_FILE = '/var/run/import.state.json'
+
+state = File.exist?(STATE_FILE) ? JSON.parse(File.read(STATE_FILE)) : { 'cursor' => 0 }
+
+SmarterCSV.process('import.csv', chunk_size: 500) do |chunk, chunk_index|
+  next if chunk_index < state['cursor']  # skip already-processed chunks on resume
+
+  MyModel.import!(chunk)
+  state['cursor'] = chunk_index + 1
+  File.write(STATE_FILE, JSON.dump(state))
+end
+
+File.delete(STATE_FILE)  # done — clear the cursor
+```
+
+If the process is killed at chunk 7, the next run skips chunks 0–6 quickly via `next` and resumes at chunk 7. For Rails 8.1+ projects, see [Examples → Resumable CSV Import with Rails ActiveJob](./examples.md#example-12-resumable-csv-import-with-rails-activejob-rails-81) for the framework-native version.
+
 ## Example: Reading a CSV from S3
 
 SmarterCSV accepts any IO-like object, so you can stream directly from S3 without

data/docs/examples.md

@@ -44,6 +44,12 @@
 11. [Batch Processing with Sidekiq](#example-11-batch-processing-with-sidekiq)
 12. [Resumable CSV Import with Rails ActiveJob](#example-12-resumable-csv-import-with-rails-activejob-rails-81)
 13. [Instrumentation](#example-13-instrumentation)
+14. [Streaming Inputs (Non-Seekable IO)](#example-14-streaming-inputs-non-seekable-io)
+15. [Resumable Import (Plain Ruby)](#example-15-resumable-import-plain-ruby)
+16. [CSV Files with Comment Lines](#example-16-csv-files-with-comment-lines)
+17. [Tab-Separated Values (TSV)](#example-17-tab-separated-values-tsv)
+18. [Multi-Line Fields](#example-18-multi-line-fields)
+19. [Filtering and Transforming a CSV File](#example-19-filtering-and-transforming-a-csv-file)
 
 ---
 
@@ -370,5 +376,124 @@ SmarterCSV.process('large_import.csv',
 
 See [Instrumentation Hooks](./instrumentation.md).
 
+---
+
+## Example 14: Streaming Inputs (Non-Seekable IO)
+
+*(1.17.0+)* SmarterCSV reads from gzipped files, HTTP responses, S3 objects, or piped STDIN — no need to materialize the file on disk first.
+
+```ruby
+require 'zlib'
+Zlib::GzipReader.open('huge.csv.gz') do |io|
+  SmarterCSV.process(io) { |row| MyModel.upsert(row.first) }
+end
+```
+
+See [Real-World CSV Files → I/O Patterns](./real_world_csv.md#io-patterns) for gzip, S3, HTTP, STDIN, and `IO.popen` worked examples.
+
+---
+
+## Example 15: Resumable Import (Plain Ruby)
+
+A non-Rails counterpart to Example 12 — track the chunk cursor in a JSON file so an interrupted import resumes where it left off.
+
+See [Batch Processing → Resumable Import (Plain Ruby)](./batch_processing.md#example-resumable-import-plain-ruby) for the worked example.
+
+---
+
+## Example 16: CSV Files with Comment Lines
+
+Strip lines matching a pattern (e.g. `#`-prefixed comments in DB dumps and log exports) using `comment_regexp`:
+
+```ruby
+SmarterCSV.process('data.csv', comment_regexp: /\A#/)
+```
+
+See [Header Transformations → CSV Files with Comment Lines](./header_transformations.md#csv-files-with-comment-lines) for the worked example.
+
+---
+
+## Example 17: Tab-Separated Values (TSV)
+
+```ruby
+SmarterCSV.process('data.tsv')                  # auto-detected
+SmarterCSV.process('data.tsv', col_sep: "\t")   # explicit
+```
+
+See [Row and Column Separators → Tab-Separated Values (TSV)](./row_col_sep.md#tab-separated-values-tsv) for details.
+
+---
+
+## Example 18: Multi-Line Fields
+
+Newlines inside `"..."` are preserved as part of the field — common in addresses, CRM notes, and free-text comments. No configuration needed.
+
+See [Real-World CSV Files → Multi-Line Quoted Fields](./real_world_csv.md#multi-line-quoted-fields) for the worked example.
+
+---
+
+## Example 19: Filtering and Transforming a CSV File
+
+The Ruby CSV library has `CSV.filter` for "read CSV, mutate each row, write CSV." In SmarterCSV this is a two-line composition of `SmarterCSV.each` and `SmarterCSV.generate`:
+
+```ruby
+SmarterCSV.generate('out.csv') do |csv|
+  SmarterCSV.each('in.csv') do |row|
+    row[:price] = (row[:price] * 1.1).round(2)
+    row.delete(:internal_notes)
+    csv << row
+  end
+end
+```
+
+The explicit `csv << row` is the win over `CSV.filter` — emission is intentional, not a side effect of mutating the block argument.
+
+### Pipeline (STDIN → STDOUT)
+
+```ruby
+# cat in.csv | ruby filter.rb > out.csv
+SmarterCSV.generate($stdout) do |csv|
+  SmarterCSV.each($stdin) { |row| csv << row }
+end
+```
+
+### Skipping rows
+
+```ruby
+SmarterCSV.generate('out.csv') do |csv|
+  SmarterCSV.each('in.csv') do |row|
+    next if row[:status] == 'archived'   # just skip — no emit
+    csv << row
+  end
+end
+```
+
+### Compressed in, compressed out
+
+```ruby
+require 'zlib'
+Zlib::GzipWriter.open('out.csv.gz') do |gz_out|
+  SmarterCSV.generate(gz_out) do |csv|
+    Zlib::GzipReader.open('in.csv.gz') do |gz_in|
+      SmarterCSV.each(gz_in) { |row| csv << row }
+    end
+  end
+end
+```
+
+Both endpoints are non-seekable streams — a pattern `CSV.filter` cannot handle, since it requires seekable input/output.
+
+### Header renaming on the way through
+
+```ruby
+SmarterCSV.generate('out.csv', headers: [:given_name, :family_name, :email]) do |csv|
+  SmarterCSV.each('in.csv',
+    key_mapping: { first_name: :given_name, last_name: :family_name }
+  ) { |row| csv << row }
+end
+```
+
+Use `key_mapping:` on the read side to rename columns and `headers:` on the write side to enforce output column order.
+
 --------------------
 PREVIOUS: [Instrumentation Hooks](./instrumentation.md) | NEXT: [Real-World CSV Files](./real_world_csv.md) | UP: [README](../README.md)

data/docs/header_transformations.md

@@ -62,6 +62,28 @@ See [Configuration Options](./options.md) for full option reference.
 
 ---
 
+## CSV Files with Comment Lines
+
+Strip comment lines anywhere in the file — including before the header — using `comment_regexp`:
+
+```ruby
+$ cat data.csv
+# Generated 2026-01-15 by exporter v3.2
+# Confidential — internal use only
+id,name,amount
+1,Alice,100
+2,Bob,200
+# end of file
+
+data = SmarterCSV.process('data.csv', comment_regexp: /\A#/)
+# => [{id: 1, name: "Alice", amount: 100},
+#     {id: 2, name: "Bob",   amount: 200}]
+```
+
+Common in database dumps, log exports, and pipelines that prepend provenance metadata. The regexp is applied per line — any line matching is dropped before parsing.
+
+---
+
 ## Header Normalization
 
 When processing the headers, it transforms them into Ruby symbols, stripping extra spaces, lower-casing them and replacing spaces with underscores. e.g. " \t Annual Sales  " becomes `:annual_sales`. (see Notes below)