smarter_csv 1.15.2 → 1.16.0

data/docs/batch_processing.md

@@ -2,6 +2,7 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
+  * [Migrating from Ruby CSV](./migrating_from_csv.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)
@@ -10,10 +11,17 @@
   * [Row and Column Separators](./row_col_sep.md)
   * [Header Transformations](./header_transformations.md)
   * [Header Validations](./header_validations.md)
+  * [Column Selection](./column_selection.md)
   * [Data Transformations](./data_transformations.md)
   * [Value Converters](./value_converters.md)
-    
---------------     
+  * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Instrumentation Hooks](./instrumentation.md)
+  * [Examples](./examples.md)
+  * [Real-World CSV Files](./real_world_csv.md)
+  * [SmarterCSV over the Years](./history.md)
+  * [Release Notes](./releases/1.16.0/changes.md)
+
+--------------
 
 # Batch Processing
 
@@ -64,7 +72,7 @@ The `process` method returns the number of chunks when called with a block.
         => 2
 ```
 
-## Example 3: Populate a MongoDB Database in Chunks of 100 records with SmarterCSV:
+## Example 3: ActiveRecord Bulk Insert in Chunks of 100 records with SmarterCSV:
 ```ruby
     # using chunks:
     filename = '/tmp/some.csv'
@@ -78,5 +86,154 @@ The `process` method returns the number of chunks when called with a block.
      => returns number of chunks we processed
 ```
 
+---
+
+# Modern Batch API — `each_chunk`
+
+`Reader#each_chunk` is the modern API for chunked batch processing. It yields `(Array<Hash>, chunk_index)` — the same shape as the `process` block — but returns an `Enumerator` when called without a block, enabling more flexible composition.
+
+## Configuration
+
+Set `chunk_size` in options when constructing the Reader. `each_chunk` reads this value automatically:
+
+```ruby
+reader = SmarterCSV::Reader.new('big.csv', chunk_size: 500)
+reader.each_chunk do |chunk, index|
+  puts "Processing chunk #{index} (#{chunk.size} rows)"
+  MyModel.insert_all(chunk)
+end
+```
+
+If `chunk_size` is not set, `each_chunk` defaults to `SmarterCSV::Reader::DEFAULT_CHUNK_SIZE` (100) and emits a warning to STDERR:
+
+```
+SmarterCSV: chunk_size not set, defaulting to 100. Set chunk_size explicitly to suppress this warning.
+```
+
+Set `chunk_size` explicitly to suppress the warning and choose the right batch size for your use case.
+
+## Simplified form
+
+```ruby
+SmarterCSV.each_chunk('big.csv', chunk_size: 500) do |chunk, index|
+  MyModel.insert_all(chunk)
+end
+```
+
+## Returns an Enumerator when called without a block
+
+```ruby
+reader = SmarterCSV::Reader.new('big.csv', chunk_size: 500)
+reader.each_chunk.with_index do |chunk, index|
+  puts "Chunk #{index}: #{chunk.size} rows"
+end
+```
+
+## Example: Sidekiq parallel import
+
+```ruby
+reader = SmarterCSV::Reader.new('users.csv', chunk_size: 100)
+reader.each_chunk do |chunk, index|
+  ImportWorker.perform_async(chunk)
+end
+```
+
+## Example: Resque parallel import
+
+```ruby
+reader = SmarterCSV::Reader.new('orders.csv', chunk_size: 200)
+reader.each_chunk do |chunk, index|
+  Resque.enqueue(OrderImportJob, chunk)
+end
+```
+
+## Example: ActiveRecord `insert_all` bulk insert
+
+```ruby
+reader = SmarterCSV::Reader.new('products.csv', chunk_size: 500)
+reader.each_chunk do |chunk, _index|
+  MyModel.insert_all(chunk)
+end
+```
+
+## Example: Progress tracking
+
+```ruby
+reader = SmarterCSV::Reader.new('big.csv', chunk_size: 1_000)
+total = File.foreach('big.csv').count - 1  # subtract header row
+
+reader.each_chunk do |chunk, index|
+  processed = [(index + 1) * 1_000, total].min
+  puts "#{processed}/#{total} rows processed"
+  MyModel.insert_all(chunk)
+end
+```
+
+## Interaction with `on_bad_row`
+
+`each_chunk` respects all `on_bad_row` options. Bad rows are excluded from chunks and counted or routed to your handler:
+
+```ruby
+reader = SmarterCSV::Reader.new('data.csv',
+  chunk_size: 500,
+  on_bad_row: :collect,
+)
+reader.each_chunk do |chunk, index|
+  MyModel.insert_all(chunk)
+end
+puts "Bad rows: #{reader.errors[:bad_row_count]}"
+reader.errors[:bad_rows].each { |rec| puts "Line #{rec[:csv_line_number]}: #{rec[:error_message]}" }
+```
+
+See [Bad Row Quarantine](./bad_row_quarantine.md) for full details.
+
+## Example: DynamoDB batch write
+
+DynamoDB's `batch_write_item` API accepts up to **25 items per request** — making
+`chunk_size: 25` the natural fit. SmarterCSV symbol keys map directly to DynamoDB
+attribute names after a simple `transform_keys(&:to_s)` call.
+
+```ruby
+require 'aws-sdk-dynamodb'
+
+client = Aws::DynamoDB::Client.new(region: 'us-east-1')
+
+SmarterCSV::Reader.new('products.csv', chunk_size: 25).each_chunk do |chunk, _index|
+  client.batch_write_item(
+    request_items: {
+      'ProductsTable' => chunk.map do |row|
+        { put_request: { item: row.transform_keys(&:to_s) } }
+      end
+    }
+  )
+end
+```
+
+## Example: Reading a CSV from S3
+
+SmarterCSV accepts any IO-like object, so you can stream directly from S3 without
+writing a temp file:
+
+```ruby
+require 'aws-sdk-s3'
+
+s3  = Aws::S3::Client.new(region: 'us-east-1')
+obj = s3.get_object(bucket: 'my-bucket', key: 'imports/products.csv')
+
+data = SmarterCSV.process(obj.body)
+MyModel.insert_all(data)
+```
+
+For large files, combine with chunked processing:
+
+```ruby
+obj = s3.get_object(bucket: 'my-bucket', key: 'imports/big.csv')
+
+SmarterCSV::Reader.new(obj.body, chunk_size: 500).each_chunk do |chunk, _index|
+  MyModel.insert_all(chunk)
+end
+```
+
 ----------------
-PREVIOUS: [The Basic Write API](./basic_write_api.md)  | NEXT: [Configuration Options](./options.md)
+
+PREVIOUS: [The Basic Write API](./basic_write_api.md) | NEXT: [Configuration Options](./options.md) | UP: [README](../README.md)

data/docs/column_selection.md

@@ -0,0 +1,183 @@
+
+### Contents
+
+  * [Introduction](./_introduction.md)
+  * [Migrating from Ruby CSV](./migrating_from_csv.md)
+  * [Parsing Strategy](./parsing_strategy.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
+  * [Batch Processing](././batch_processing.md)
+  * [Configuration Options](./options.md)
+  * [Row and Column Separators](./row_col_sep.md)
+  * [Header Transformations](./header_transformations.md)
+  * [Header Validations](./header_validations.md)
+  * [**Column Selection**](./column_selection.md)
+  * [Data Transformations](./data_transformations.md)
+  * [Value Converters](./value_converters.md)
+  * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Instrumentation Hooks](./instrumentation.md)
+  * [Examples](./examples.md)
+  * [Real-World CSV Files](./real_world_csv.md)
+  * [SmarterCSV over the Years](./history.md)
+  * [Release Notes](./releases/1.16.0/changes.md)
+
+--------------
+
+# Column Selection
+
+Wide CSV files often contain dozens or hundreds of columns, but a given application typically
+only needs a handful of them. The `headers: { only: }` and `headers: { except: }` options let
+you declare upfront which columns you want, so SmarterCSV skips allocation and hash insertion
+for everything else — both in the Ruby path and in the C-accelerated hot path.
+
+## Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `headers: { only: }` | `nil` | Keep only the listed columns in each result hash |
+| `headers: { except: }` | `nil` | Remove the listed columns from each result hash |
+
+You cannot use both options at the same time — doing so raises `SmarterCSV::ValidationError`.
+
+## Basic usage
+
+```ruby
+# Keep only two columns out of a wide file
+data = SmarterCSV.process('big.csv', headers: { only: [:id, :email] })
+# => [{id: 1, email: "alice@example.com"}, ...]
+
+# Keep everything except one noisy column
+data = SmarterCSV.process('big.csv', headers: { except: [:internal_notes] })
+```
+
+## Input flexibility
+
+Both options accept an Array of symbols or strings, or a single symbol or string — anything
+that makes sense as a column name. All values are normalized to symbols internally.
+
+```ruby
+headers: { only: :id }                     # single symbol — same as [:id]
+headers: { only: 'id' }                    # single string — normalized to :id
+headers: { only: [:id, :email] }           # array of symbols
+headers: { only: ['id', 'email'] }         # array of strings — normalized to symbols
+```
+
+## Names refer to post-mapping keys
+
+`headers: { only: }` and `headers: { except: }` use the **post-mapping** column name — the
+symbol that actually appears in the result hash after `key_mapping:` has been applied. You
+never need to know the original CSV header spelling.
+
+```ruby
+# CSV has header "First Name"; key_mapping renames it to :given_name
+data = SmarterCSV.process('contacts.csv',
+  key_mapping:  { first_name: :given_name },
+  headers:      { only: [:given_name] },   # post-mapping name
+)
+# => [{given_name: "Alice"}, ...]
+```
+
+## Interaction with `with_line_numbers:`
+
+`:csv_line_number` is added to each hash **after** column selection runs, so it is always
+present when `with_line_numbers: true` — even if it is not listed in `headers: { only: }`.
+
+```ruby
+data = SmarterCSV.process('data.csv',
+  headers:           { only: [:name] },
+  with_line_numbers: true,
+)
+data.each { |row| puts "#{row[:csv_line_number]}: #{row[:name]}" }
+```
+
+## Interaction with `strict:`
+
+`strict: true` raises `SmarterCSV::HeaderSizeMismatch` when a data row contains more fields
+than the header row. This check runs **before** column selection, so schema validation still
+catches malformed rows even when `headers: { only: }` is active.
+
+```ruby
+# Raises HeaderSizeMismatch on the row with extra fields, regardless of headers: { only: }
+SmarterCSV.process('data.csv', headers: { only: [:name] }, strict: true)
+```
+
+## Extra columns without `strict:`
+
+When `strict:` is false (the default) and a data row has more fields than the header,
+the extra columns are silently dropped — they cannot be in the `headers: { only: }` set, so
+the filter discards them naturally.
+
+> **Important:** `missing_headers: :auto` (auto-generating names like `column_7`,
+> `column_8` for extra data columns) does **not** work in combination with `headers: { only: }`.
+> `headers: { only: }` is a **performance improvement** that causes the parser to stop scanning
+> a row once all requested columns have been found — any extra columns beyond the header
+> count are never visited, so no auto-names are generated for them. If you need to capture
+> auto-named overflow columns, do not use `headers: { only: }` at the same time.
+
+## Unknown column names are silently ignored
+
+Listing a column name that doesn't exist in the file is not an error. The column simply
+never appears in any row hash.
+
+```ruby
+# :nonexistent_column is not in the file — no error, just absent from results
+data = SmarterCSV.process('data.csv', headers: { only: [:id, :nonexistent_column] })
+```
+
+## Performance
+
+Both options are implemented in the C extension (when acceleration is enabled). Excluded
+columns are skipped entirely inside the C parsing loop — no Ruby string is allocated, no
+numeric conversion runs, and no `rb_hash_aset` call is made for fields the caller doesn't
+need. This makes column selection a genuine performance option for wide CSV files, not just
+a post-processing filter.
+
+The Ruby fallback path applies the same filter via `hash.select!` / `hash.reject!` after
+parsing, giving correct results on all platforms.
+
+### `headers: { only: }` vs `headers: { except: }` — performance asymmetry
+
+**`headers: { only: }` enables early exit.** Once every requested column has been parsed,
+the parser stops scanning the current row entirely — the remaining fields are never visited.
+For a 500-column file where you only need 5 columns near the start, this can be
+**10–14× faster** than parsing all columns.
+
+**`headers: { except: }` cannot have early exit.** To know which columns to *keep*, the
+parser must scan every field in the row to the end. Skipping just a few columns out of many
+saves very little work, so benchmark results for `headers: { except: }` are typically flat
+(0.7×–1.0× vs full parse).
+
+**Rule of thumb:**
+- Use `headers: { only: }` when you want a small subset of a wide file — this is the fast path.
+- Use `headers: { except: }` only when you want *almost everything* and excluding a known
+  noisy column is more convenient than listing all the ones you want.
+- Avoid `headers: { except: }` as a performance tool on wide files — it provides no speed benefit.
+
+### `headers: { only: }` vs `remove_unmapped_keys:`
+
+If you are already using `key_mapping:` to rename headers, the `remove_unmapped_keys: true`
+option lets you implicitly drop everything not in the map — without listing each unwanted
+column explicitly. This is a convenient alternative to `headers: { only: }` when renaming
+and selecting go hand in hand:
+
+```ruby
+# With key_mapping + remove_unmapped_keys: convenient when renaming
+SmarterCSV.process('data.csv',
+  key_mapping:          { col_a: :name, col_b: :email },
+  remove_unmapped_keys: true,
+)
+
+# With headers: { only: }: better for pure selection — C-path early exit applies
+SmarterCSV.process('data.csv',
+  headers: { only: [:col_a, :col_b] },
+)
+```
+
+`headers: { only: }` is the faster choice for wide files since unneeded fields are skipped
+inside the C parser before any Ruby objects are created. `remove_unmapped_keys:` is a
+post-parse filter — all fields are parsed first, then the unwanted keys are deleted.
+See [Header Transformations](./header_transformations.md#key-mapping) for more details.
+
+---
+
+PREVIOUS: [Header Validations](./header_validations.md) | NEXT: [Data Transformations](./data_transformations.md) | UP: [README](../README.md)

data/docs/data_transformations.md

@@ -2,6 +2,7 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
+  * [Migrating from Ruby CSV](./migrating_from_csv.md)
   * [Parsing Strategy](./parsing_strategy.md)
   * [The Basic Read API](./basic_read_api.md)
   * [The Basic Write API](./basic_write_api.md)
@@ -10,52 +11,184 @@
   * [Row and Column Separators](./row_col_sep.md)
   * [Header Transformations](./header_transformations.md)
   * [Header Validations](./header_validations.md)
+  * [Column Selection](./column_selection.md)
   * [**Data Transformations**](./data_transformations.md)
   * [Value Converters](./value_converters.md)
-    
---------------    
+  * [Bad Row Quarantine](./bad_row_quarantine.md)
+  * [Instrumentation Hooks](./instrumentation.md)
+  * [Examples](./examples.md)
+  * [Real-World CSV Files](./real_world_csv.md)
+  * [SmarterCSV over the Years](./history.md)
+  * [Release Notes](./releases/1.16.0/changes.md)
+
+--------------
 
 # Data Transformations
 
-SmarterCSV automatically transforms the values in each colum in order to normalize the data.
-This behavior can be customized or disabled.
+SmarterCSV automatically normalizes the values in each row. All transformations are configurable — most are enabled by default because they're the right behavior for the vast majority of CSV files.
+
+## Transformation Pipeline
+
+Transformations run in this order for every row:
+
+| Step | Option | Default | What it does |
+|------|--------|---------|--------------|
+| 1 | `strip_whitespace` | `true` | Strips leading/trailing whitespace from all values (and headers) at parse time |
+| 2 | `nil_values_matching` | `nil` | Sets values matching the regexp to `nil` |
+| 3 | `remove_empty_values` | `true` | Removes keys whose value is `nil` or blank |
+| 4 | `remove_zero_values` | `false` | Removes keys whose value is numeric zero |
+| 5 | `convert_values_to_numeric` | `true` | Converts numeric-looking strings to `Integer` or `Float` |
+| 6 | `value_converters` | `nil` | Applies per-key custom converter lambdas or classes |
+| 7 | `remove_empty_hashes` | `true` | Drops rows that are entirely empty after all transformations |
+
+> Steps 2–6 run per field in order. `value_converters` receive the value **after** numeric conversion — guard against receiving `Integer`/`Float` if your converter expects a string.
+
+---
 
-## Remove Empty Values
-`remove_empty_values` is enabled by default
-It removes any values which are `nil` or would be empty strings.
+## `strip_whitespace`
 
-## Convert Values to Numeric
-`convert_values_to_numeric` is enabled by default. 
-SmarterCSV will convert strings containing Integers or Floats to the appropriate class.
+**Default: `true`**
 
-Here is an example of using `convert_values_to_numeric` for numbers with leading zeros, e.g. ZIP codes:
+Strips leading and trailing whitespace from all header names and all field values at parse time, before any other transformation runs.
 
+```ruby
+# CSV with padded values:
+# name,  score
+# Alice ,  42
+# Bob   ,  0
+
+data = SmarterCSV.process(file)
+# => [{name: "Alice", score: 42}, {name: "Bob", score: 0}]
+#  ↑ "Alice " stripped to "Alice", "  42" stripped to "42" then converted
+
+data = SmarterCSV.process(file, strip_whitespace: false)
+# => [{"name"=>"Alice ", " score"=>"  42"}, ...]
+#  ↑ whitespace preserved in both headers and values
 ```
-  data = SmarterCSV.process('/tmp/zip.csv',  convert_values_to_numeric: { except: [:zip] })
-   => [{:zip=>"00480"}, {:zip=>"51903"}, {:zip=>"12354"}, {:zip=>"02343"}]
-```   
 
-This will return the column `:zip` as a string with all digits intact.
+---
+
+## `nil_values_matching`
+
+**Default: `nil` (disabled)**
+
+Set values matching the given regular expression to `nil`. Combined with the default `remove_empty_values: true`, matching values are removed from the result hash. With `remove_empty_values: false`, the key is retained with a `nil` value — useful when you need to distinguish "field was absent" from "field had a sentinel value".
+
+```ruby
+# Treat common null sentinels as nil and remove them
+data = SmarterCSV.process(file, nil_values_matching: /\A(NULL|N\/A|NA|#N\/A|\(null\))\z/i)
 
-## Remove Zero Values
-`remove_zero_values` is disabled by default.
-When enabled, it removes key/value pairs which have a numeric value equal to zero.
+# Nil-ify but retain the key (don't remove)
+data = SmarterCSV.process(file,
+  nil_values_matching: /\A(NULL|N\/A)\z/i,
+  remove_empty_values: false)
+# => [{name: "Alice", score: nil}]  ← key retained with nil value
+
+# Remove Excel error values
+data = SmarterCSV.process(file, nil_values_matching: /\A(#VALUE!|#REF!|#DIV\/0!|NaN)\z/)
+```
+
+> **Deprecated:** `remove_values_matching:` still works but emits a deprecation warning.
+> Use `nil_values_matching:` instead.
+
+---
+
+## `remove_empty_values`
+
+**Default: `true`**
+
+Removes key/value pairs where the value is `nil` or an empty string after `strip_whitespace` and `nil_values_matching` have run. This is why SmarterCSV result hashes only contain keys with actual values — sparse CSV rows don't produce hashes cluttered with `nil` entries.
+
+```ruby
+# CSV: name,score,notes
+#      Alice,42,
+#      Bob,,great player
+
+data = SmarterCSV.process(file)
+# => [{name: "Alice", score: 42}, {name: "Bob", notes: "great player"}]
+#  ↑ empty :notes and :score keys are dropped automatically
+
+data = SmarterCSV.process(file, remove_empty_values: false)
+# => [{name: "Alice", score: 42, notes: nil}, {name: nil, score: nil, notes: "great player"}]
+```
 
-## Remove Values Matching
-`remove_values_matching` is disabled by default. 
-When enabled, this can help removing key/value pairs from result hashes which would cause problems. 
+---
 
-e.g.
- * `remove_values_matching: /^\$0\.0+$/` would remove $0.00 
- * `remove_values_matching: /^#VALUE!$/` would remove errors from Excel spreadsheets 
+## `remove_zero_values`
 
-## Empty Hashes
+**Default: `false`**
 
-It can happen that after all transformations, a row of the CSV file would produce a completely empty hash.
+When enabled, removes key/value pairs where the value is numeric zero (`0`, `0.0`, `"0"`, `"0.0"`). Useful when zero and absent mean the same thing in your domain.
 
-By default SmarterCSV uses `remove_empty_hashes: true` to remove these empty hashes from the result.
+```ruby
+# CSV: product,quantity,discount
+#      Widget,10,0
+#      Gadget,0,5
 
-This can be set to `false`, to keep these empty hashes in the results.
+data = SmarterCSV.process(file, remove_zero_values: true)
+# => [{product: "Widget", quantity: 10}, {product: "Gadget", discount: 5}]
+#  ↑ :discount=>0 and :quantity=>0 removed
+```
+
+---
+
+## `convert_values_to_numeric`
+
+**Default: `true`**
+
+Converts string values that look like integers or floats to the appropriate numeric type. This is one of the most common sources of silent data loss if not configured carefully — fields like ZIP codes, phone numbers, and account numbers with leading zeros will be silently corrupted if not excluded.
+
+```ruby
+data = SmarterCSV.process(file)
+# "42"     => 42    (Integer)
+# "3.14"   => 3.14  (Float)
+# "01234"  => 1234  ← leading zero lost! exclude this column
+
+# Exclude specific columns from numeric conversion
+data = SmarterCSV.process(file,
+  convert_values_to_numeric: { except: [:zip, :phone, :account_number] })
+# => [{zip: "01234", phone: "800-555-0100", amount: 99.99}]
+
+# Only convert specific columns (all others stay as strings)
+data = SmarterCSV.process(file,
+  convert_values_to_numeric: { only: [:quantity, :price] })
+```
+
+---
+
+## `remove_empty_hashes`
+
+**Default: `true`**
+
+After all per-field transformations, removes rows that have no remaining key/value pairs. This handles blank lines and rows where every field was empty or matched `nil_values_matching`.
+
+```ruby
+# CSV with a blank line between records:
+# name,score
+# Alice,42
+#
+# Bob,99
+
+data = SmarterCSV.process(file)
+# => [{name: "Alice", score: 42}, {name: "Bob", score: 99}]
+#  ↑ blank line silently dropped
+
+data = SmarterCSV.process(file, remove_empty_hashes: false)
+# => [{name: "Alice", score: 42}, {}, {name: "Bob", score: 99}]
+```
+
+---
+
+## Custom Transformations — `value_converters`
+
+For type conversions beyond numeric (dates, booleans, currency, etc.), use `value_converters`. They run last in the pipeline, after numeric conversion. See [Value Converters](./value_converters.md) for full documentation.
+
+```ruby
+data = SmarterCSV.process(file, value_converters: {
+  date:   ->(v) { v ? Date.strptime(v, '%m/%d/%Y') : nil },
+  active: ->(v) { v&.match?(/\Atrue\z/i) },
+})
+```
 
 -------------------
-PREVIOUS: [Header Validations](./header_validations.md) | NEXT: [Value Converters](./value_converters.md)
+PREVIOUS: [Column Selection](./column_selection.md) | NEXT: [Value Converters](./value_converters.md) | UP: [README](../README.md)