smarter_csv 1.18.0 → 1.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3335e39a1c0792f01df9e95401c7f3885c49a0d64eeb9c76e5c20e25d01a62f5
-  data.tar.gz: e43f00228777b56fc1ee0814a74acaa6a23c51fe8da6f64e42ad92fe1b54002f
+  metadata.gz: dec0abc82b39809fd4822789f0e3f5bc80437eccf484435a540ea4c8ac83d500
+  data.tar.gz: 210f7a67b2a5e0470a97f7d7c05d8fb2ec644a945d96e3580a951fdf72ff2d2b
 SHA512:
-  metadata.gz: 2abcd136f30d284c3c27cbd2b6c9782aec4235ec62cc27ea7620380ae9efc889f9f1c05c10ad957d6e7c84d65be75d35d96d4ac59ba5780dc5e65e0151c661e6
-  data.tar.gz: 6c61062c08d0a89dea2c91a7faafd88f6d09c88015ad8c7f4facb2eb474b44b18e0c5eaffc36fa5e2885f0b2c02487497f2460d66ce2433f62c09bf42d92a4ce
+  metadata.gz: a31c7b13246f6abcb14fa882ad3cb962e74ad2d4b150dd034eb437dfb6a9c0eccb53c266d361b2b8d9bff1e461e4638b452d2c32868bd5e2e208dbaffc7bbd75
+  data.tar.gz: fe8d6066b2f25e6bb598ff0e93188ed77b0a409e412ebf1f03636d760abae8111eba63e291b943eacf7e2cac1504ebe818cb51e1ecd8eef2adf6396794748acf

data/CHANGELOG.md

@@ -4,6 +4,91 @@
 > [!TIP]
 > **Upgrading?** The [SmarterCSV Upgrade Wizard](https://tilo.github.io/smarter_csv/upgrade_wizard.html) walks you through what (if anything) you need to change for your specific version. Most steps do not require any changes.
 
+## 1.19.0 (2026-08-10)
+
+RSpec tests: **2,595 → 3,164** (+569 tests)
+
+
+### Reverted Behavior Changes
+
+  - **Exponent forms are no longer auto-converted to numbers ([#345](https://github.com/tilo/smarter_csv/issues/345)).**
+
+  Version 1.18.0 started converting scientific notation (`"1e3"`, `"12E5"`, `"1.5e3"`) to Floats. In real-world CSV data, digits-E-digits values are far more often identifiers (short codes, hex IDs) than scientific notation, and the auto-conversion corrupted them irreversibly — an ID like `"0047583311587E590003"` came back as `Infinity`.
+
+  As of 1.19.0, exponent forms always stay Strings — as they did in every version before 1.18.0. If a column really does contain scientific notation, convert it per-column with `value_converters`:
+
+  ```ruby
+  SmarterCSV.process(file, value_converters: { measurement: ->(v) { v.to_f } })
+  ```
+
+  Thanks to [Denis Sadomowski](https://github.com/sonicdes) for the report.
+
+### Behavior Changes
+
+  - **`field_size_limit` values below `4096` now raise a `ValidationError`** — the option is overrun protection (a hard upper bound against runaway fields), not per-field validation.
+
+  - **The Hash form of `convert_values_to_numeric` is now validated and normalized.** It requires exactly one of `only:`/`except:` with field name(s) (String/Symbol or an Array of them); an empty hash, unknown keys, both keys together, empty lists, or `nil`/boolean values raise a `ValidationError` (the two parser paths previously disagreed on these shapes). The listed names are normalized to the row-key type, so `only:`/`except:` now also works with `strings_as_keys` / `keep_original_headers`.
+
+  - **All empty field values are ONE shared, frozen, UTF-8 empty-string object — on both paths** (no String allocation per empty field). Mutating an empty value now raises `FrozenError` instead of silently changing every other empty value in the result. Relevant with `remove_empty_values: false`; the default `true` removes empties anyway.
+
+### Bug Fixes
+
+  - **Writer**: fields needing quoting are wrapped in the configured `quote_char`, not a hard-coded `"` — output with a custom `quote_char` round-trips again.
+  - **`Reader#each` without a block** no longer clears the configured `chunk_size` for a later `each_chunk` on the same Reader.
+  - **Duplicate-header disambiguation** no longer steals a real column's name — `name,name,name2` no longer raises `DuplicateHeaders` (the second `name` becomes `name3`).
+  - **The caller's `user_provided_headers` array** (and a reused options hash) is no longer mutated when rows have more columns than headers.
+  - **`headers: { only: }` / `{ except: }`** now works with `strings_as_keys` / `keep_original_headers` — selectors are normalized to the row-key type (with `only:`, nothing matched and every row came back empty: silent total data loss).
+  - **A quoted header containing an embedded newline** is stitched across physical lines like data rows (`"first\nname"` → `:first_name`); previously the first fragment was silently lost. An unclosed header quote at end-of-file raises `MalformedCSV`.
+  - **`quote_char: :auto`** raises a `ValidationError` instead of crashing with `NoMethodError` — quote_char has no auto-detection.
+
+### Bug Fixes — C/Ruby parity
+
+  The C-accelerated and pure-Ruby parsers now behave identically in all of the following cases (same input, same output — verified by differential fuzzing and by running every parsing spec on both paths):
+
+  - a partial multi-char separator at end-of-line is field content, not a separator — with `col_sep: '||'` the C path silently dropped the lone `|` from `"y|"` (also fixes an out-of-bounds read near end-of-line)
+  - a trailing `\r` before an LF row separator is part of the line terminator — a CRLF line with a quoted last field raised `MalformedCSV` on the C path, and `strip_whitespace: false` kept `"x\r"` / `"1\r"` as values
+  - with `strip_whitespace: true`, values are stripped of Ruby's full `String#strip` character set on the C path too (a stray `\r` from mixed LF/CRLF files survived before)
+  - an empty line yields `nil` for ALL columns with `remove_empty_values: false` (the C path gave the first column `""`)
+  - a row consisting only of NUL bytes counts as blank on the C path too (`String#strip` semantics; the NUL byte itself remains data with `remove_empty_hashes: false`)
+  - a `nil` entry in `user_provided_headers` drops that column on the C path too
+  - an empty-string header key (`strings_as_keys` + `duplicate_header_suffix: nil`) is dropped on the C path too
+  - non-ASCII `missing_header_prefix` (e.g. `"spalte_ä_"`) no longer raises `EncodingError` on the C path — extra-column keys are interned as UTF-8 symbols
+  - `col_sep` / `row_sep` / `missing_header_prefix` values longer than the C parser's internal buffers fall back to the pure-Ruby parser instead of being silently truncated
+  - `nil_values_matching` matches the RAW string value on the C path (a pattern like `/\A007\z/` only ever saw the converted `7`) — and no longer switches off numeric conversion and zero-removal for the non-matching values
+  - `field_size_limit` is checked against the raw field size BEFORE numeric conversion, so an oversized digit-only field raises on the C path too instead of being converted to a huge Integer (the exact overrun the option exists to prevent)
+  - `headers: { only: }` short-cuts on the pure-Ruby path too — parsing stops right after the last wanted column, matching the C path (no `:column_N` discovery behind it, and faster)
+  - a one-character multi-byte `col_sep` (e.g. `'é'`) no longer crashes the pure-Ruby parser
+  - a multi-byte character directly before a literal quote (`é"x`) no longer crashes the pure-Ruby parser (`IndexError` from a mid-character byte offset)
+  - invalid bytes in the input (typically Latin-1 data mislabeled as UTF-8) no longer crash the pure-Ruby parser — fields keep their raw bytes and encoding tag exactly (never transcoded, so the data stays recoverable via `force_encoding`); cleanup remains opt-in via `force_utf8` / `invalid_byte_sequence`
+  - the multiline stitch gate models the parser's rules exactly (doubled-quote precedence, backslash escapes, end-of-line chomp) — no more fabricated `MalformedCSV` on the pure-Ruby path for rows the parser can close
+
+### Tests
+
+  - **Every parsing spec now runs on BOTH the C-accelerated and the pure-Ruby path** via `[true, false]` acceleration loops (~420 additional examples), a seeded differential parity-fuzz spec (2,000 randomized inputs per run, including combined option sets) guards C/Ruby parity permanently, and line coverage is at **100%**.
+
+## 1.18.1 (2026-06-30)
+
+### Bug Fixes
+
+  - **Portable builds by default — fixes the "Illegal instruction" crash on heterogeneous CPUs ([#343](https://github.com/tilo/smarter_csv/issues/343)).**
+
+  Since 1.14.3 the C extension was compiled with `-march=native` on every platform except Apple Silicon, baking-in the build host's CPU instructions (e.g. AVX-512).
+  A binary built on one machine then could encounter `Illegal instruction` when run on a CPU lacking those instructions — common when the build host differs from the run host (CI/build servers, Docker images, mixed-hardware fleets).
+
+  The C extension is now built **portable** by default (no host-specific instructions). Thanks to [@paholg](https://github.com/paholg) for the report.
+
+### New Features
+
+  - **`SMARTER_CSV_PERFORMANCE` build option** (`portable` default, `tuned`, or `max`)
+
+  | Level                | Flags added                               | Portable?                        | Use when                              |
+  |----------------------|-------------------------------------------|----------------------------------|---------------------------------------|
+  | `portable` (default) | none                                      | Yes, any CPU of the arch         | Build host may differ from run host   |
+  | `tuned`              | `-mtune=native`                           | Yes, instruction scheduling only | Build and run hosts share a microarch |
+  | `max`                | `-march=native`, or `-mcpu=native` on ARM | No, host instruction optimization| Build host and run host are the same  |
+
+  See the [Introduction](docs/_introduction.md#build-time-performance-tuning-smarter_csv_performance) for details.
+
 ## 1.18.0 (2026-06-17)
 
 This release is focused on both performance and the introduction of automatic conversion of decimals to big_decimal or float, preserving the precision, and also supporting scientific notation.

data/CONTRIBUTORS.md

@@ -1,4 +1,4 @@
-# A Big Thank You to all 65 Contributors!!
+# A Big Thank You to all 66 Contributors!!
 
 
 A Big Thank you to everyone who filed issues, sent comments, and who contributed with pull requests:
@@ -68,3 +68,4 @@ A Big Thank you to everyone who filed issues, sent comments, and who contributed
  * [Jonas Staškevičius](https://github.com/pirminis)
  * [conorg](https://github.com/conorg)
  * [Alex Shenia](https://github.com/alexshenia)
+ * [Denis Sadomowski](https://github.com/sonicdes)

data/README.md

@@ -297,6 +297,29 @@ Or install it yourself as:
     $ gem install smarter_csv
 ```
 
+The C extension is built on install and used automatically. On platforms where it can't build, the pure-Ruby implementation runs instead and produces identical results.
+
+### CPU Optimization (`SMARTER_CSV_PERFORMANCE`)
+
+The C extension is compiled when the gem is installed. By default it is built **portable**: it uses no CPU-specific instructions, so a binary built on one machine runs on any other CPU of the same architecture. Set `SMARTER_CSV_PERFORMANCE` at install time to trade portability for speed:
+
+
+| Level                | Flags added                               | Portable?                        | Use when                              |
+|----------------------|-------------------------------------------|----------------------------------|---------------------------------------|
+| `portable` (default) | none                                      | Yes, any CPU of the arch         | Build host may differ from run host   |
+| `tuned`              | `-mtune=native`                           | Yes, instruction scheduling only | Build and run hosts share a microarch |
+| `max`                | `-march=native`, or `-mcpu=native` on ARM | No, host instruction optimization| Build host and run host are the same  |
+
+`max` enables host-specific instructions, so a binary built with it can crash with `Illegal instruction` if it later runs on a CPU that lacks them (for example, built on an AVX-512 machine and run on one without). `tuned` only changes instruction scheduling, never the instruction set, so it stays portable. Every flag is probed against your compiler at build time and skipped if unsupported, so an unavailable flag never breaks the build.
+
+```bash
+SMARTER_CSV_PERFORMANCE=tuned gem install smarter_csv   # portable, tuned for this machine's microarchitecture
+SMARTER_CSV_PERFORMANCE=max   gem install smarter_csv   # fastest, NOT portable — only when you build on the machine you run on
+SMARTER_CSV_PERFORMANCE=tuned bundle install            # same, under Bundler
+```
+
+For a fixed baseline instead of `native` (e.g. a portable-but-newer instruction set), pass flags directly via `CFLAGS`, which the build also honors: `CFLAGS="-march=x86-64-v2" gem install smarter_csv`.
+
 ## Documentation
 
   * [Introduction](docs/_introduction.md)
@@ -341,7 +364,7 @@ For reporting issues, please:
   * open a pull-request adding a test that demonstrates the issue
   * mention your version of SmarterCSV, Ruby, Rails
 
-# [A Special Thanks to all 65 Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
+# [A Special Thanks to all Contributors!](CONTRIBUTORS.md) 🎉🎉🎉
 
 
 ## Contributing

data/UPGRADING.md

@@ -2,7 +2,7 @@
 
 > [!TIP]
 > Prefer the interactive [Upgrade Wizard](https://tilo.github.io/smarter_csv/upgrade_wizard.html) for a guided walk-through with Yes/No questions.  
-> This document is auto-generated from `CHANGELOG.md` and `docs/upgrade_path.json` by `bin/gen-upgrading-md`.
+> This document is auto-generated from `CHANGELOG.md` and `docs/upgrade_path.json` by `bin/generate-upgrading-md`.
 
 ## How to use this guide
 
@@ -12,25 +12,40 @@
 
 Prefer an interactive walk-through? The [Upgrade Wizard](https://tilo.github.io/smarter_csv/upgrade_wizard.html) asks one question at a time and only shows the migration steps that apply to your code.
 
-**Latest release:** `1.17.3` (in the `1.17.x` series).
+**Latest release:** `1.18.1` (in the `1.18.x` series).
 
 ---
 
-## 1.17.x — latest series
+## 1.18.x — latest series
 
 **Versions in this series:**  
-[1.17.0, 1.17.1, 1.17.2, 1.17.3]
+[1.18.0, 1.18.1]
 
-**Latest release:** `1.17.3`
+> ⚠️ **In-series notes** worth checking:
+> - **1.18.0:** This version is particularly interesting if you have geolocation, scientific, or high-precision data.
+
+**Latest release:** `1.18.1`
 
 Update your Gemfile to:
 
 ```ruby
-gem 'smarter_csv', '~> 1.17.0'
+gem 'smarter_csv', '~> 1.18.0'
 ```
 
 Then run `bundle update smarter_csv`.
 
+## Series 1.17 → 1.18
+
+**Coming from any 1.17 version:**
+[1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4]
+
+**Upgrading to 1.18.x** (latest: `1.18.1`):
+
+- **If** you build and run the gem on the same machine (or a fleet of identical CPUs) and want the previous native-optimized build for maximum speed:
+  → set `SMARTER_CSV_PERFORMANCE=max` (or `tuned`) at install time — 1.18.1 builds <strong>portable</strong> by default (no host-specific CPU instructions) to fix an `Illegal instruction` crash when a binary built on one CPU is run on another (<a href="https://github.com/tilo/smarter_csv/issues/343">#343</a>). The default is safe everywhere; the env var opts back into host optimization.
+
+---
+
 ## Series 1.16 → 1.17
 
 **Coming from any 1.16 version:**  
@@ -40,7 +55,7 @@ Then run `bundle update smarter_csv`.
 > - **1.16.1:** **Fibers:** `SmarterCSV.errors` uses `Thread.current` for storage, which is **shared across all fibers running in the same thread**. If you process CSV files concurrently in fibers (e.g. with `Async`, `Falcon`, or manual `Fiber` scheduling), `SmarterCSV.errors` may return stale or wrong results. **Use `SmarterCSV::Reader` directly** — errors are scoped to the reader instance and are always correct regardless of fiber context.
 > - **1.16.2:** If your code references auto-generated keys for blank headers, update those to use the absolute column position.
 
-**Upgrading to 1.17.x** (latest: `1.17.3`): you can upgrade all the way — no code changes needed.
+**Upgrading to 1.17.x** (latest: `1.17.4`): you can upgrade all the way — no code changes needed.
 
 ---
 

data/docs/_introduction.md

@@ -75,6 +75,28 @@ SmarterCSV was created to solve exactly these problems: nightly imports of large
 * **CSV writing:**
   `SmarterCSV.generate` writes arrays of hashes to CSV, with support for header renaming and value converters on output. See [The Basic Write API](./basic_write_api.md).
 
+## Build-Time Performance Tuning (`SMARTER_CSV_PERFORMANCE`)
+
+The C extension is compiled when the gem is installed. By default it is built **portable**: it uses no CPU-specific instructions, so a binary compiled on one machine runs on any other CPU of the same architecture. This matters whenever the machine that builds the gem differs from the machine that runs it — a CI or build server, a Docker image moved between hosts, or a mixed-hardware fleet. A build that bakes in instructions the run host lacks (such as AVX-512) would otherwise crash with `Illegal instruction`.
+
+Set `SMARTER_CSV_PERFORMANCE` at install time to trade portability for speed:
+
+| Level                | Flags added                               | Portable?                        | Use when                              |
+|----------------------|-------------------------------------------|----------------------------------|---------------------------------------|
+| `portable` (default) | none                                      | Yes, any CPU of the arch         | Build host may differ from run host   |
+| `tuned`              | `-mtune=native`                           | Yes, instruction scheduling only | Build and run hosts share a microarch |
+| `max`                | `-march=native`, or `-mcpu=native` on ARM | No, host instruction optimization| Build host and run host are the same  |
+
+`tuned` only changes instruction scheduling, never the instruction set, so it stays portable — and it pays off when the build and run hosts share a microarchitecture (the same chip, or a fleet of identical instances). `max` enables host-specific instructions and is the fastest, but a binary built with it can crash on a different CPU. Every flag is probed against your compiler at build time and skipped if unsupported, so an unavailable flag never breaks the build.
+
+```bash
+SMARTER_CSV_PERFORMANCE=tuned gem install smarter_csv   # portable, tuned for this machine's microarchitecture
+SMARTER_CSV_PERFORMANCE=max   gem install smarter_csv   # fastest, NOT portable — only when you build on the machine you run on
+SMARTER_CSV_PERFORMANCE=tuned bundle install            # same, under Bundler
+```
+
+For a fixed baseline instead of `native` (e.g. a portable-but-newer instruction set), pass flags directly via `CFLAGS`, which the build also honors: `CFLAGS="-march=x86-64-v2" gem install smarter_csv`.
+
 ---------------
 
 NEXT: [Migrating from Ruby CSV](./migrating_from_csv.md) | UP: [README](../README.md)

data/docs/bad_row_quarantine.md

@@ -266,7 +266,9 @@ until it either finds the closing quote or reaches end-of-file, potentially cons
 of megabytes.
 
 `field_size_limit` sets a hard cap (in bytes) on the size of any individual extracted field.
-The default is `nil` (no limit). When a field exceeds the limit a
+The default is `nil` (no limit); the minimum allowed value is `4096` — this option is overrun
+protection against runaway or crafted fields, not a per-field validation tool, so small values
+are rejected with a `ValidationError`. When a field exceeds the limit a
 `SmarterCSV::FieldSizeLimitExceeded` exception is raised — and because it inherits from
 `SmarterCSV::Error`, the `on_bad_row` option handles it exactly like any other parse error.
 

data/docs/basic_read_api.md

@@ -223,7 +223,7 @@ comment_regexp → strip_chars_from_headers → split on col_sep → strip quote
     → disambiguate_headers → symbolize → key_mapping
 ```
 
-`user_provided_headers` bypasses the file header and all transformation steps — your array is used as-is.
+`user_provided_headers` bypasses the file header and all transformation steps — your array is used as-is; a `nil` entry drops that column.
 
 See [Header Transformations](./header_transformations.md) for the full step-by-step table and options.
 

data/docs/data_transformations.md

@@ -114,6 +114,8 @@ data = SmarterCSV.process(file, remove_empty_values: false)
 # => [{name: "Alice", score: 42, notes: nil}, {name: nil, score: nil, notes: "great player"}]
 ```
 
+With `remove_empty_values: false`, all kept empty-string values are ONE shared frozen String (an allocation optimization) — `dup` before mutating one in place *(1.19.0+)*.
+
 ---
 
 ## `remove_zero_values`
@@ -156,7 +158,9 @@ data = SmarterCSV.process(file,
   convert_values_to_numeric: { only: [:quantity, :price] })
 ```
 
-Scientific notation (e.g. `"1.5e3"`, `"6.022e23"`) is recognized and converted too. Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths.
+The Hash form requires exactly one of `only:`/`except:` with field name(s) — anything else (empty hash, unknown keys, both keys, empty lists, `nil`/boolean values) raises a `ValidationError` *(1.19.0+)*.
+
+Exponent forms (e.g. `"1e3"`, `"12E5"`, `"1.5e3"`) are NOT converted — they stay Strings *(changed in 1.19.0; only 1.18.x converted them)*. In real-world CSV data such values are far more often identifiers (short codes, hex IDs) than scientific notation, and auto-converting them corrupts data — e.g. an ID like `"0047583311587E590003"` became `Infinity`. If a column really does contain scientific notation, convert it per-column with [`value_converters`](./value_converters.md). Bare-dot forms like `".5"` and `"3."` are left as Strings (they are not valid numbers here). Integers and floats convert identically on the C-accelerated and pure-Ruby paths.
 
 ---
 
@@ -164,7 +168,7 @@ Scientific notation (e.g. `"1.5e3"`, `"6.022e23"`) is recognized and converted t
 
 **Default: `:auto`**
 
-Controls how decimal values (those with a `.` or an exponent) are converted. Integers are unaffected — they are always returned as `Integer`.
+Controls how decimal values (those with a `.`) are converted. Integers are unaffected — they are always returned as `Integer`.
 
 | Value         | Result                                                                                  |
 |---------------|-----------------------------------------------------------------------------------------|

data/docs/header_transformations.md

@@ -56,7 +56,9 @@ comment_regexp ──► strip_chars_from_headers ──► split on col_sep
 | 9 | `strings_as_keys` | `false` | Converts headers to symbols (skipped if `true` or `keep_original_headers`) |
 | 10 | `key_mapping` | `nil` | Renames or drops headers; use post-transformation key names as input |
 
-> `user_provided_headers` bypasses all file header reading and transformation entirely — your array is used as-is. Versions >1.13 automatically set `headers_in_file: false` when `user_provided_headers` is given; if the file has a header row you want to skip, set `headers_in_file: true` explicitly.
+> `user_provided_headers` bypasses all file header reading and transformation entirely — your array is used as-is; a `nil` entry drops that column. Versions >1.13 automatically set `headers_in_file: false` when `user_provided_headers` is given; if the file has a header row you want to skip, set `headers_in_file: true` explicitly.
+
+A quoted header containing an embedded newline is stitched across physical lines like a data row *(1.19.0+)* — the newline becomes `_` via the standard transformations (`"first\nname"` → `:first_name`).
 
 See [Configuration Options](./options.md) for full option reference.
 

data/docs/migrating_from_csv.md

@@ -223,7 +223,7 @@ rows = SmarterCSV.process('sample.csv',
   convert_values_to_numeric: { except: [:zip_code, :phone, :account_number] })
 ```
 
-**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved; scientific notation (`6.022e23`, `1.6e-19`) is recognized as numeric too.
+**High-precision decimals — scientific data and geo coordinates.** GPS/geo coordinates, scientific measurements, and financial figures routinely carry 16+ significant digits, where Ruby's `Float()`-based conversion (`converters: :numeric` / `:float`) silently rounds the value. SmarterCSV's default `decimal_precision: :auto` returns a `BigDecimal` once a value exceeds 16 significant digits (and a `Float` otherwise), so the full value is preserved. (Exponent forms like `6.022e23` are not auto-converted — in CSV data they are usually identifiers, not numbers; use `value_converters` for columns that really contain scientific notation.)
 
 **With Ruby CSV (precision lost):**
 ```ruby

data/docs/options.md

@@ -33,7 +33,7 @@
 |--------|---------|-------------|
 | `:row_sep` | `$/` | Separates rows. Defaults to your OS row separator: `\n` on UNIX, `\r\n` on Windows. |
 | `:col_sep` | `","` | Separates each value in a row. |
-| `:quote_char` | `'"'` | Character used to quote CSV fields. |
+| `:quote_char` | `'"'` | Character used to quote CSV fields. Must be a single byte. |
 | `:force_quotes` | `false` | Forces each individual value to be quoted. |
 | `:headers` | `[]` | List of keys from the input to use as headers in the CSV file. ⚠️ Disables automatic header detection! |
 | `:map_headers` | `{}` | Like `:headers`, but also maps each key to a user-specified header value. ⚠️ Disables automatic header detection! |
@@ -121,7 +121,7 @@ See [Parsing Strategy](./parsing_strategy.md) for full details on quote handling
 | Option | Default | Explanation |
 |--------|---------|-------------|
 | `:strip_whitespace` | `true` | Remove whitespace before/after values and headers. |
-| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats (including scientific notation like `1.5e3`) to the appropriate numeric type. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. |
+| `:convert_values_to_numeric` | `true` | Convert strings containing integers or floats to the appropriate numeric type. Exponent forms like `1.5e3` or `12E5` stay Strings (1.19.0+) — they are usually identifiers, not numbers. Accepts `{except: [:key1, :key2]}` or `{only: :key3}` to limit which columns. |
 | `:decimal_precision` | `:auto` | How decimals are converted: `:auto` returns `Float` but `BigDecimal` above 16 significant digits (no precision loss); `:float` always returns `Float`; `:bigdecimal` always returns `BigDecimal`. Integers are unaffected. |
 | `:value_converters` | `nil` | Hash of `:header => converter`; converter can be a lambda/Proc or a class implementing `self.convert(value)`. See [Value Converters](./value_converters.md). |
 | `:remove_empty_values` | `true` | Remove key/value pairs where the value is `nil`, empty, or whitespace-only — any Unicode whitespace, same as Ruby's `String#blank?`. |
@@ -138,7 +138,7 @@ See [Bad Row Quarantine](./bad_row_quarantine.md) for full details.
 | `:on_bad_row` | `:raise` | Behavior when a row raises a parse error. `:raise` (default): re-raise, stopping processing. `:skip`: skip the bad row and continue. `:collect`: skip and append an error record to `reader.errors[:bad_rows]`. callable: called with the error record per bad row; processing continues. |
 | `:collect_raw_lines` | `true` | When collecting bad rows, include the raw stitched line in the error record. |
 | `:bad_row_limit` | `nil` | If set, raises `SmarterCSV::TooManyBadRows` after this many bad rows. |
-| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit. Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). |
+| `:field_size_limit` | `nil` | Maximum size of any extracted field in bytes. `nil` means no limit; the minimum allowed value is `4096` (it is overrun protection, not per-field validation). Raises `SmarterCSV::FieldSizeLimitExceeded` (handled by `on_bad_row`) if a field or accumulating multiline buffer exceeds this size. Prevents DoS from runaway quoted fields or huge inline payloads. See [Bad Row Quarantine](./bad_row_quarantine.md#limiting-field-size-field_size_limit). |
 
 ### Output & Diagnostics
 

data/docs/real_world_csv.md

@@ -51,6 +51,7 @@ Real-world files come from dozens of different systems, each with their own defa
 | Windows-1252 / Latin-1 | 🔘 | Specify `file_encoding: 'windows-1252'`. Common in European financial exports, older SAP systems, QuickBooks. |
 | UTF-16 LE with BOM | 🔘 | Specify `file_encoding: 'utf-16le'`. Some Microsoft SQL Server and Access exports default to this. |
 | Shift-JIS / EUC-JP | 🔘 | Specify `file_encoding: 'shift_jis'` or `'euc-jp'`. Japanese ERP and POS systems. |
+| Invalid bytes / mislabeled encoding | ✅ | Never crashes — the affected field keeps its raw bytes exactly, recoverable via `force_encoding`. Opt into cleanup with `force_utf8` / `invalid_byte_sequence`. |
 
 ---
 

data/docs/upgrade_path.json

@@ -1,6 +1,6 @@
 {
-  "latest": "1.17",
-  "latest_release": "1.17.2",
+  "latest": "1.19",
+  "latest_release": "1.19.0",
   "path": {
     "1.0": {
       "to": "1.1",
@@ -170,6 +170,39 @@
       "to": "1.17",
       "latest_release": "1.16.6",
       "actions": []
+    },
+    "1.17": {
+      "to": "1.18",
+      "latest_release": "1.17.4",
+      "actions": [
+        {
+          "if": "you build and run the gem on the same machine (or a fleet of identical CPUs) and want the previous native-optimized build for maximum speed",
+          "then": "set <code>SMARTER_CSV_PERFORMANCE=max</code> (or <code>tuned</code>) at install time &mdash; 1.18.1 builds <strong>portable</strong> by default (no host-specific CPU instructions) to fix an <code>Illegal instruction</code> crash when a binary built on one CPU is run on another (<a href=\"https://github.com/tilo/smarter_csv/issues/343\">#343</a>). The default is safe everywhere; the env var opts back into host optimization."
+        }
+      ],
+      "note": "<strong>Bonus:</strong> 1.18 automatically converts scientific notation (e.g. <code>1e3</code>, <code>6.022e23</code>) to numbers, and preserves full precision on long decimals by returning a <code>BigDecimal</code> when a value carries more than 16 significant digits (<code>decimal_precision: :auto</code>, the default). Nothing to change &mdash; this just works."
+    },
+    "1.18": {
+      "to": "1.19",
+      "latest_release": "1.18.1",
+      "actions": [
+        {
+          "if": "you relied on 1.18.x auto-converting scientific notation (<code>\"1e3\"</code>, <code>\"1.5e3\"</code>) to numbers",
+          "then": "convert those columns explicitly with <code>value_converters: { column: ->(v) { v.to_f } }</code> &mdash; 1.19.0 keeps all exponent forms as Strings again (as every version before 1.18.0 did), because in real-world data they are usually identifiers, and IDs like <code>\"0047583311587E590003\"</code> were corrupted to <code>Infinity</code> (<a href=\"https://github.com/tilo/smarter_csv/issues/345\">#345</a>)."
+        },
+        {
+          "if": "you set <code>field_size_limit:</code> to a value below 4096",
+          "then": "raise it to at least <code>4096</code> &mdash; the option is overrun protection (a hard upper bound against runaway fields), not per-field validation, and 1.19.0 rejects smaller values with a <code>ValidationError</code>."
+        },
+        {
+          "if": "you pass <code>convert_values_to_numeric:</code> as a Hash with anything other than exactly one of <code>only:</code>/<code>except:</code> and field name(s)",
+          "then": "fix the option &mdash; an empty hash, unknown keys, both keys together, empty lists, or <code>nil</code>/boolean values now raise a <code>ValidationError</code> (the two parser paths previously disagreed silently on these shapes)."
+        },
+        {
+          "if": "you mutate empty-string values in the results (only possible with <code>remove_empty_values: false</code>)",
+          "then": "<code>dup</code> them first &mdash; all empty field values are now one shared, frozen empty string (an allocation optimization), so in-place mutation raises <code>FrozenError</code>."
+        }
+      ]
     }
   }
 }

data/docs/upgrade_wizard.html

@@ -94,6 +94,15 @@ input[type="text"] {
   color: var(--green);
   font-weight: 600;
 }
+.benefit {
+  background: var(--green-soft);
+  border: 1px solid #b6dab6;
+  border-radius: 8px;
+  padding: 1em 1.25em;
+  margin-bottom: 1em;
+  color: var(--fg);
+}
+.benefit strong { color: var(--green); }
 .check {
   padding: 0.85em 0;
   border-top: 1px solid var(--border);
@@ -244,6 +253,8 @@ function renderHop(series, originalVersion) {
       `).join("")}
     </div>`;
   }
+  // The bonus note renders below the actions (or below the drop-in box) regardless of whether this hop has actions.
+  if (hop.note) body += `<div class="benefit">${hop.note}</div>`;
 
   const nextLabel = hop.to === LATEST ? `Finish at ${targetRelease} &rarr;` : `Continue to ${targetRelease} &rarr;`;
   const reminder = hop.actions.length === 0 ?  `<p class="reminder">You can upgrade directly to version ${targetRelease}. No changes needed.</p>` :
@@ -324,6 +335,7 @@ function renderHop(series, originalVersion) {
       from:   series,
       to:     hop.to,
       dropIn: hop.actions.length === 0,
+      note:   hop.note || null,
       matched
     });
 
@@ -376,14 +388,15 @@ function renderSummary() {
   const list = decisions.map(d => {
     const targetRelease = latestReleaseFor(d.to);
     const heading = `<p class="summary-hop-heading"><strong>${d.from}.x &rarr; ${targetRelease}</strong></p>`;
+    const noteHTML = d.note ? `<p>${d.note}</p>` : "";
     if (d.dropIn) {
-      return `<div class="summary-hop">${heading}<p class="muted">No code changes needed for this step.</p></div>`;
+      return `<div class="summary-hop">${heading}<p class="muted">No code changes needed for this step.</p>${noteHTML}</div>`;
     }
     if (d.matched.length === 0) {
-      return `<div class="summary-hop">${heading}<p class="muted">None of the conditions in this step applied to your code.</p></div>`;
+      return `<div class="summary-hop">${heading}<p class="muted">None of the conditions in this step applied to your code.</p>${noteHTML}</div>`;
     }
     const items = d.matched.map(a => `<li><strong>If</strong> ${a["if"]}<br>&rarr; ${a.then}</li>`).join("");
-    return `<div class="summary-hop">${heading}<ul>${items}</ul></div>`;
+    return `<div class="summary-hop">${heading}<ul>${items}</ul>${noteHTML}</div>`;
   }).join("");
 
   return `

data/ext/smarter_csv/cpu_flags.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module SmarterCSV
+  # Pure (mkmf-free) selection of CPU-optimization flags from the
+  # SMARTER_CSV_PERFORMANCE environment variable. Kept separate from extconf.rb
+  # so the logic can be unit-tested without invoking a compiler.
+  #
+  # Levels:
+  #   portable - no host-specific flags. The binary runs on any CPU of the same
+  #              architecture. The safe default: a binary built here will not
+  #              crash with "Illegal instruction" on an older/different CPU.
+  #   tuned    - -mtune=native: tunes instruction scheduling for the build host's
+  #              microarchitecture WITHOUT changing the instruction set, so the
+  #              binary stays portable. A real win when build and run hosts share
+  #              a microarchitecture (same chip or a homogeneous fleet).
+  #   max      - host-specific instructions: -march=native, or -mcpu=native on
+  #              ARM/Clang where -march=native is rejected. Fastest, but NOT
+  #              portable -- may crash on a CPU lacking the build host's
+  #              instructions. Use only when build host and run host match.
+  #
+  # `accepts` is a predicate (in the real build, a wrapper over mkmf's
+  # try_compile) returning true when the compiler accepts a given flag; each
+  # candidate is probed so an unsupported flag is skipped rather than breaking
+  # the build.
+  module CpuFlags
+    LEVELS = %w[portable tuned max].freeze
+
+    # Candidate flags per level, in preference order. The first one the compiler
+    # accepts wins. `max` degrades march -> mcpu -> mtune; tuned only ever
+    # considers -mtune=native (never an instruction-set flag).
+    CANDIDATES = {
+      'portable' => [].freeze,
+      'tuned' => ['-mtune=native'].freeze,
+      'max' => ['-march=native', '-mcpu=native', '-mtune=native'].freeze,
+    }.freeze
+
+    # Returns a Hash: { level: String, flags: Array<String>, warning: String|nil }.
+    def self.select(raw_level, accepts:)
+      level, warning = normalize(raw_level)
+      chosen = CANDIDATES[level].find { |flag| accepts.call(flag) }
+      { level: level, flags: chosen ? [chosen] : [], warning: warning }
+    end
+
+    # Normalizes the env value to a known level. Unknown values fall back to
+    # 'portable' (a typo can then only ever be slower, never non-portable) and
+    # return a warning naming the bad value and the fallback.
+    def self.normalize(raw_level)
+      value = raw_level.to_s.strip.downcase
+      return ['portable', nil] if value.empty?
+      return [value, nil] if LEVELS.include?(value)
+
+      ['portable', "SMARTER_CSV_PERFORMANCE=#{raw_level.inspect} is not one of #{LEVELS.join('|')}; using 'portable'."]
+    end
+  end
+end

data/ext/smarter_csv/extconf.rb

@@ -2,6 +2,7 @@
 
 require 'mkmf'
 require "rbconfig"
+require_relative 'cpu_flags'
 
 if RbConfig::MAKEFILE_CONFIG["CFLAGS"].include?("-g -O3")
   fixed_CFLAGS = RbConfig::MAKEFILE_CONFIG["CFLAGS"].sub("-g -O3", "-O3 $(cflags)")
@@ -9,11 +10,31 @@ if RbConfig::MAKEFILE_CONFIG["CFLAGS"].include?("-g -O3")
   RbConfig::MAKEFILE_CONFIG["CFLAGS"] = fixed_CFLAGS
 end
 
+# Probe whether the compiler accepts a flag by compiling a trivial program with
+# it. Lets us skip flags the toolchain rejects (e.g. -march=native on Clang/ARM,
+# or GCC-only flags on MSVC) instead of breaking the build. Replaces the old
+# RUBY_PLATFORM string guesses: ask the actual compiler, don't infer from the OS.
+def compiler_accepts?(flag)
+  try_compile("int main(void){return 0;}", flag)
+end
+
 optflags = "-O3 -flto -fomit-frame-pointer -DNDEBUG".dup
-optflags << " -march=native" unless RUBY_PLATFORM.start_with?("arm64-darwin")
+
+# CPU optimization level, set via SMARTER_CSV_PERFORMANCE (default: portable).
+# See cpu_flags.rb for the full description of each level.
+#
+#   portable (default) - no host-specific flags; runs on any CPU of the same arch.
+#   tuned              - -mtune=native; host scheduling tuning, still portable.
+#   max                - host instruction set (-march/-mcpu native); fastest, but
+#                        NOT portable -- may crash on a CPU lacking those instructions.
+cpu = SmarterCSV::CpuFlags.select(ENV["SMARTER_CSV_PERFORMANCE"], accepts: method(:compiler_accepts?))
+warn(cpu[:warning]) if cpu[:warning]
+cpu[:flags].each { |flag| optflags << " #{flag}" }
+puts("SmarterCSV performance level: #{cpu[:level]} -- optflags: #{optflags}")
+
 # -fno-semantic-interposition: GCC/Clang only (not MSVC). Allows intra-library
 # calls to bypass the PLT on Linux and enables more aggressive LTO inlining.
-optflags << " -fno-semantic-interposition" unless RUBY_PLATFORM.include?("mswin")
+optflags << " -fno-semantic-interposition" if compiler_accepts?("-fno-semantic-interposition")
 
 append_cflags('-Wno-compound-token-split-by-macro')