smarter_csv 1.13.0 → 1.14.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c28d21a143e743de4b21d8ad93d860b1d51424e525e9ec0a73bb640b170d9823
-  data.tar.gz: e7a16ae8494196b85d9a196d071f09b302583c9ef3a414f09f5ad6ae1f11c29b
+  metadata.gz: 7f7308ee39fc18fb93f23273b32559ae83139375bfc84c975a1b6cf048368a5e
+  data.tar.gz: a50d9ea0b40bf28adc829613d8d0b2744018b708453b96cc339653688a1c1f40
 SHA512:
-  metadata.gz: 319ee5aed33630e9670a1c95cc8da6fd57df9d1d7db57a00af79c1e5c10de56b4e9054c86b6b462ebdc693513a79aff2c881a1ede00ad28e5da58768b4a6f2cf
-  data.tar.gz: 6b36378d3a15ed9065c697f56f2cafc359d2c746e5796780a276c1a87c6a04be38616205f31ec9341412f3c9a4f52d150ce4ef95c0e76f340368d5683b1452e6
+  metadata.gz: baf9bad5261b6ab75d4d49fc7b0044536a9977955ef01d38b9d9431478649be8481a378a4b988b6091b7f2c5ba5f6c2a8899dfc2143f2bd11c834386453024d1
+  data.tar.gz: 7283acb30cbdde33c463f369a619cc93235ea2931153babdcaedabe468cf50e198ab0264e9c70ee166a81cc072b44ab874d89e4de835379495155784b7a31035

data/CHANGELOG.md

@@ -1,6 +1,12 @@
 
 # SmarterCSV 1.x Change Log
 
+## 1.14.0 (2025-04-07)
+ * adding advanced configuration options for writing CSV files. ([issue 297](https://github.com/tilo/smarter_csv/issues/297) thanks to Robert Reiz, [issue 296](https://github.com/tilo/smarter_csv/issues/296))
+
+## 1.13.1 (2024-12-12)
+  * fix bug with SmarterCSV.generate with `force_quotes: true` ([issue 294](https://github.com/tilo/smarter_csv/issues/294))
+
 ## 1.13.0 (2024-11-06) ⚡ POTENTIALLY BREAKING ⚡
   
   CHANGED DEFAULT BEHAVIOR
@@ -38,7 +44,7 @@
 
     IF you set `user_provided_headers` and the file has a header, then provide `headers_in_file: true` to avoid getting that extra record.
 
-   * handling of numeric columns with leading zeroes, e.g. ZIP codes. ([issue #151](https://github.com/tilo/smarter_csv/issues/151) thanks to David Moles). `convert_values_to_numeric: { except: [:zip] }` will now return a string for that column instead.
+   * improved documentation for handling of numeric columns with leading zeroes, e.g. ZIP codes. ([issue #151](https://github.com/tilo/smarter_csv/issues/151) thanks to David Moles). `convert_values_to_numeric: { except: [:zip] }` will  return a string for that column instead (since version 1.10.x)
 
 ## 1.12.1 (2024-07-10)
   * Improved column separator detection by ignoring quoted sections [#276](https://github.com/tilo/smarter_csv/pull/276) (thanks to Nicolas Castellanos)

data/CONTRIBUTORS.md

@@ -58,3 +58,4 @@ A Big Thank you to everyone who filed issues, sent comments, and who contributed
  * [Simon Rentzke](https://github.com/simonrentzke)
  * [Randall B](https://github.com/randall-coding)
  * [Matthew Kennedy](https://github.com/MattKitmanLabs)
+ * [Robert Reiz](https://github.com/reiz)

data/README.md

@@ -1,7 +1,7 @@
 
 # SmarterCSV
 
- [![codecov](https://codecov.io/gh/tilo/smarter_csv/branch/main/graph/badge.svg?token=1L7OD80182)](https://codecov.io/gh/tilo/smarter_csv) [![Gem Version](https://badge.fury.io/rb/smarter_csv.svg)](http://badge.fury.io/rb/smarter_csv)
+  ![Gem Version](https://img.shields.io/gem/v/smarter_csv) [![codecov](https://codecov.io/gh/tilo/smarter_csv/branch/main/graph/badge.svg?token=1L7OD80182)](https://codecov.io/gh/tilo/smarter_csv) [View on RubyGems](https://rubygems.org/gems/smarter_csv) [View on RubyToolbox](https://www.ruby-toolbox.com/search?q=smarter_csv)
 
  SmarterCSV provides a convenient interface for reading and writing CSV files and data.
 
@@ -35,7 +35,8 @@ Or install it yourself as:
 # Documentation
 
   * [Introduction](docs/_introduction.md)
-  * [The Basic API](docs/basic_api.md)
+  * [The Basic Read API](docs/basic_read_api.md)
+  * [The Basic Write API](docs/basic_write_api.md)
   * [Batch Processing](./docs/batch_processing.md)
   * [Configuration Options](docs/options.md)
   * [Row and Column Separators](docs/row_col_sep.md)
@@ -45,10 +46,10 @@ Or install it yourself as:
   * [Value Converters](docs/value_converters.md)
     
 # Articles
-* [Parsing CSV Files in Ruby with SmarterCSV](https://tilo-sloboda.medium.com/parsing-csv-files-in-ruby-with-smartercsv-6ce66fb6cf38)
-* [Processing 1.4 Million CSV Records in Ruby, fast ](https://lcx.wien/blog/processing-14-million-csv-records-in-ruby/)
-* [Speeding up CSV parsing with parallel processing](http://xjlin0.github.io/tech/2015/05/25/faster-parsing-csv-with-parallel-processing)
-* [The original post](http://www.unixgods.org/Ruby/process_csv_as_hashes.html) that started SmarterCSV
+  * [Parsing CSV Files in Ruby with SmarterCSV](https://tilo-sloboda.medium.com/parsing-csv-files-in-ruby-with-smartercsv-6ce66fb6cf38)
+  * [Processing 1.4 Million CSV Records in Ruby, fast ](https://lcx.wien/blog/processing-14-million-csv-records-in-ruby/)
+  * [Faster Parsing CSV with Parallel Processing](http://xjlin0.github.io/tech/2015/05/25/faster-parsing-csv-with-parallel-processing) by [Jack lin](https://github.com/xjlin0/)
+  * [The original post](http://www.unixgods.org/Ruby/process_csv_as_hashes.html) that started SmarterCSV
 
 # [ChangeLog](./CHANGELOG.md)
 

data/docs/_introduction.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [**Introduction**](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)
@@ -53,4 +54,4 @@ The CSV processing also needed to be robust against variations in the input data
   (planned feature)
 
 ---------------
-PREVIOUS [README](../README.md) | NEXT: [The Basic API](./basic_api.md)
+PREVIOUS [README](../README.md) | NEXT: [The Basic Read API](./basic_read_api.md)

data/docs/{basic_api.md → basic_read_api.md}

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [**The Basic API**](./basic_api.md)
+  * [**The Basic Read API**](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)    
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)
@@ -70,46 +71,6 @@ It cal also be used with a block:
 This allows you access to the internal state of the `reader` instance after processing.
 
 
-## Interface for Writing CSV
-
-To generate a CSV file, we use the `<<` operator to append new data to the file.
-
-The input operator for adding data to a CSV file `<<` can handle single hashes, array-of-hashes, or array-of-arrays-of-hashes, and can be called one or multiple times for each file.
-
-One smart feature of writing CSV data is the discovery of headers. 
-
-If you have hashes of data, where each hash can have different keys, the `SmarterCSV::Reader` automatically discovers the superset of keys as the headers of the CSV file. This can be disabled by either providing one of the options `headers`, `map_headers`, or `discover_headers: false`.
-
-
-### Simplified Interface
-
-The simplified interface takes a block:
-
-      ```
-        SmarterCSV.generate(filename, options) do |csv_writer|
-
-         MyModel.find_in_batches(batch_size: 100) do |batch|
-           batch.pluck(:name, :description, :instructor).each do |record|
-             csv_writer << record
-           end
-         end
-
-       end
-     ```
-
-### Full Interface
-
-      ```
-        writer = SmarterCSV::Writer.new(file_path, options)
-
-        MyModel.find_in_batches(batch_size: 100) do |batch|
-          batch.pluck(:name, :description, :instructor).each do |record|
-            csv_writer << record
-          end
-
-        writer.finalize
-      ```
-
 ## Rescue from Exceptions
 
 While SmarterCSV uses sensible defaults to process the most common CSV files, it will raise exceptions if it can not auto-detect `col_sep`, `row_sep`, or if it encounters other problems. Therefore please rescue from `SmarterCSV::Error`, and handle outliers according to your requirements.
@@ -154,4 +115,4 @@ $ hexdump -C spec/fixtures/bom_test_feff.csv
 ```
 
 ----------------
-PREVIOUS: [Introduction](./_introduction.md) | NEXT: [Batch Processing](./batch_processing.md)
+PREVIOUS: [Introduction](./_introduction.md) | NEXT: [The Basic Write API](./basic_write_api.md)

data/docs/basic_write_api.md

@@ -0,0 +1,160 @@
+
+### Contents
+
+  * [Introduction](./_introduction.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [**The Basic Write API**](./basic_write_api.md)
+  * [Batch Processing](././batch_processing.md)
+  * [Configuration Options](./options.md)
+  * [Row and Column Separators](./row_col_sep.md)
+  * [Header Transformations](./header_transformations.md)
+  * [Header Validations](./header_validations.md)
+  * [Data Transformations](./data_transformations.md)
+  * [Value Converters](./value_converters.md)
+    
+--------------  
+
+# SmarterCSV Basic Write API
+
+Let's explore the basic API for writing CSV files. There is a simplified API (backwards conpatible with previous SmarterCSV versions) and the full API, which allows you to access the internal state of the writer instance after processing.
+
+## Writing CSV Files
+
+To generate a CSV file, we use the `<<` operator to append new data to the file.
+
+The input operator for adding data to a CSV file `<<` can handle single hashes, array-of-hashes, or array-of-arrays-of-hashes, and can be called one or multiple times in order to create a file.
+
+### Auto-Discovery of Headers
+
+By default, the `SmarterCSV::Writer` discovers all keys that are present in the input data, and as they become know, appends them to the CSV headers. This ensures that all data will be included in the output CSV file.
+
+If you want to customize the output file, or only include select headers, check the section about Advanced Features below.
+
+### Auto-Quoting of Problematic Values
+
+CSV files use some special characters that are important for the CSV format to function:
+* @row_sep : typically `\n` the carriage return
+* @col_sep : typically `,` the comma
+* @quote_char : typically `"` the double-quote
+  
+When your data for a given field in a CSV row contains either of these characters, we need to prevent them to break the CSV file format.
+
+`SmarterCSV::Writer` automatically detects if a field contains either of these three characters. If a field contains the `@quote_char`, it will be prefixed by another `@qoute_char` as per CSV conventions.
+In either case the corresponding field will be put in double-quotes. 
+  
+
+### Simplified Interface
+
+The simplified interface takes a block:
+
+      ```
+        SmarterCSV.generate(filename, options) do |csv_writer|
+
+         MyModel.find_in_batches(batch_size: 100) do |batch|
+           batch.pluck(:name, :description, :instructor).each do |record|
+             csv_writer << record
+           end
+         end
+
+       end
+     ```
+
+### Full Interface
+
+      ```
+        writer = SmarterCSV::Writer.new(file_path, options)
+
+        MyModel.find_in_batches(batch_size: 100) do |batch|
+          batch.pluck(:name, :description, :instructor).each do |record|
+            csv_writer << record
+          end
+
+        writer.finalize
+      ```
+
+## Advanced Features: Customizing the Output Format
+
+You can customize the output format through different features.
+
+In the options, you can pass-in either of these parameters to customize your output format.
+* `headers`, which limits the CSV headers to just the specified list.
+* `map_header`, which maps a given list of Hash keys to custom strings, and limits the CSV headers to just those.
+* `value_converters`, which specifies a hash with more advanced value transformations.
+
+### Limited Headers
+
+You can use the `headers` option to limit the CSV headers to only a sub-set of Hash keys from your data.
+This will switch-off the automatic detection of headers, and limit the CSV output file to only the CSV headers you provide in this option.
+
+
+### Mapping Headers
+
+Similar to the `headers` option, you can define `map_headers` in order to rename a given set of Hash keys to some custom strings in order to rename them in the CSV header. This will switch-off the automatic detection of headers.
+
+
+### Per Key Value Converters
+
+
+Using per-key value converters, you can control how specific hash keys in your data are converted in the output.
+
+Example 1:
+
+```
+      options = {
+        value_converters: {
+          active: ->(v) { !!v ? 'YES' : 'NO' },
+        }
+      }
+```
+
+This maps the boolean value of the hash key `:active` into strings `"YES"`, `"NO"`.
+
+Example 2:
+
+```
+      options = {
+        value_converters: {
+          active: ->(v) { !!v ? '✅' : '❌' },
+          balance: ->(v) do
+            case v
+            when Float
+              '$%.2f' % v.round(2)
+            when Integer
+              "$#{v}"
+            else
+              v.to_s
+            end
+          end,
+        }
+      }
+```
+
+This maps the hash key `:balance` to a string. Floats are rounded and displayed with 2 decimals and prefixed by `$`. Integers are prefixed by `$`.
+The boolean value of the key `:active` is mapped into an emoji.
+
+### Global Value Converters
+
+You can also use the special keyword `:_all` to define transformations that are applied to each field of the CSV file.
+
+```
+      options = {
+        value_converters: {        
+          disable_auto_quoting: true, # ⚠️ Important: turn off auto-quoting because we're messing with it below
+          active: ->(v) { !!v ? 'YES' : 'NO' },
+          _all: ->(k, v) { v.is_a?(String) ? "\"#{v}\"" : v } # only double-quote string fields
+        }  
+      }
+```
+
+Using the `:_all` keyword, you can set up rules to convert all hash keys. This is applied after all per-key conversions are made.
+
+This example puts double-quotes around all String-value data, but leaves other types unchanged.
+
+Note that when you're customizing putting quote-chars around fields, you need to `disable_auto_quoting`.
+
+## More Examples
+
+Check out the [RSpec tests](../spec/smarter_csv/writer_spec.rb) for more examples.
+
+----------------
+PREVIOUS: [The Basic Read API](./basic_read_api.md) | NEXT: [Batch Processing](./batch_processing.md)

data/docs/batch_processing.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md) 
   * [**Batch Processing**](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)
@@ -65,4 +66,4 @@ and how the `process` method returns the number of chunks when called with a blo
 ```
 
 ----------------
-PREVIOUS: [The Basic API](./basic_api.md)  | NEXT: [Configuration Options](./options.md)
+PREVIOUS: [The Basic Write API](./basic_write_api.md)  | NEXT: [Configuration Options](./options.md)

data/docs/data_transformations.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)

data/docs/examples.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)

data/docs/header_transformations.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)
@@ -81,8 +82,8 @@ There is an additional option `remove_unmapped_keys` which can be enabled to onl
 
 ## CSV Files without Headers
 
-If you have CSV files without headers, it is important to set `headers_in_file: false`, otherwise you'll lose the first data line in your file.
-You then have to provide `user_provided_headers`, which takes an array of either symbols or strings.
+If you have CSV files without headers, it is important to set `headers_in_file: false`, otherwise you'll lose the first data line in your file. 
+You then have to provide `user_provided_headers`, which takes an array of either symbols or strings. Versions >1.13 now automatically set `headers_in_file: false` if you provide `user_provided_headers`. Also see next paragraph.
 
 
 ## CSV Files with Headers
@@ -93,6 +94,7 @@ For CSV files with headers, you can either:
 * map one or more headers into whatever you chose using the `map_headers` option.
   (if you map a header to `nil`, it will remove that column from the resulting row hash).
 * completely replace the headers using `user_provided_headers` (please be careful with this powerful option, as it is not robust against changes in input format).
+  When you use `user_provided_headers`, versions >1.13 will set `headers_in_file: false` -- so if you replace the headers for a file that has headers, you must set `headers_in_file: true` to override this and ignore the header row.
 * use the original unmodified headers from the CSV file, using `keep_original_headers`. This results in hash keys that are strings, and may be padded with spaces.
 
 
@@ -104,11 +106,10 @@ For CSV files with headers, you can either:
  * any occurences of :comment_regexp or :row_sep will be stripped from the first line with the CSV header
  * any of the keys in the header line will be downcased, spaces replaced by underscore, and converted to Ruby symbols before being used as keys in the returned Hashes
  * you can not combine the :user_provided_headers and :key_mapping options
- * if the incorrect number of headers are provided via :user_provided_headers, exception SmarterCSV::HeaderSizeMismatch is raised
+ * if the incorrect number of headers are provided via :user_provided_headers, versions >1.13 will automatically add column names `column_N` for additional unexpected columns. If you want to raise an error instead, add option `strict: true`, and it will raise `SmarterCSV::HeaderSizeMismatch`.
 
 ### NOTES on improper quotation and unwanted characters in headers:
- * some CSV files use un-escaped quotation characters inside fields. This can cause the import to break. To get around this, use the `:force_simple_split => true` option in combination with `:strip_chars_from_headers => /[\-"]/` . This will also significantly speed up the import.
-   If you would force a different :quote_char instead (setting it to a non-used character), then the import would be up to 5-times slower than using `:force_simple_split`.
+ * some CSV files use un-escaped quotation characters inside fields. This can cause the import to break. To get around this, set the `quote_char` to something different, e.g. `quote_char: "%"`, or try setting `:strip_chars_from_headers => /[\-"]/` 
 
 ---------------
 PREVIOUS: [Row and Column Separators](./row_col_sep.md) | NEXT: [Header Validations](./header_validations.md) 

data/docs/header_validations.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)

data/docs/options.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [**Configuration Options**](./options.md)
   * [Row and Column Separators](./row_col_sep.md)
@@ -20,14 +21,18 @@
      | Option                      | Default  |  Explanation                                                                         |
      ---------------------------------------------------------------------------------------------------------------------------------
      | :row_sep                    |   $/      | Separates rows; Defaults to your OS row separator. `/n` on UNIX, `/r/n` oon Windows | 
-     | :col_sep                    |   ","     | Separates each value in a row | 
-     | :quote_char                 |   '"'     | |
+     | :col_sep                    |   ","     | Separates each value in a row                                  | 
+     | :quote_char                 |   '"'     | To quote CSV fields.                                           |
      | :force_quotes               |   false   | Forces each individual value to be quoted |
-     | :discover_headers           |   true    | Automatically detects all keys in the input before writing the header |
-     |                             |           | This can be disabled by providing `headers` or `map_headers` options. |
      | :headers                    |    []     | You can provide the specific list of keys from the input you'd like to be used as headers in the CSV file |
+     |                             |           | ⚠️ This disables automatic header detection!                    |
      | :map_headers                |    {}     | Similar to `headers`, but also maps each desired key to a user-specified value that is uesd as the header. | 
-     |
+     |                             |           | ⚠️ This disables automatic header detection!                    |
+     | :discover_headers           |   true    | Automatically detects all keys in the input before writing the header |
+     |                             |           | Do not manually set this to `false`. ⚠️                         |
+     |                             |           | But you can set this to `true` when using `map_headers` option. |
+     | :disable_auto_quoting       |  false    | To manually disable auto-quoting of special characters. ⚠️ Be careful with this! |
+
 
 ## CSV Reading
 
@@ -41,9 +46,7 @@
      | :skip_lines                 |   nil    | how many lines to skip before the first line or header line is processed             |
      | :comment_regexp             |   nil    | regular expression to ignore comment lines (see NOTE on CSV header), e.g./\A#/       |
      ---------------------------------------------------------------------------------------------------------------------------------
-     | :col_sep                    |   :auto   | column separator (default was ',')                                                  |
-     | :force_simple_split         |   false  | force simple splitting on :col_sep character for non-standard CSV-files.             |
-     |                             |          | e.g. when :quote_char is not properly escaped                                        |
+     | :col_sep                    |  :auto   | column separator (default was ',')                                                   |
      | :row_sep                    |  :auto   | row separator or record separator (previous default was system's $/ , which defaulted to "\n") |
      |                             |          | This can also be set to :auto, but will process the whole cvs file first  (slow!)    |
      | :auto_row_sep_chars         |   500    | How many characters to analyze when using `:row_sep => :auto`. nil or 0 means whole file. |

data/docs/row_col_sep.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [**Row and Column Separators**](./row_col_sep.md)

data/docs/value_converters.md

@@ -2,7 +2,8 @@
 ### Contents
 
   * [Introduction](./_introduction.md)
-  * [The Basic API](./basic_api.md)
+  * [The Basic Read API](./basic_read_api.md)
+  * [The Basic Write API](./basic_write_api.md)
   * [Batch Processing](././batch_processing.md)
   * [Configuration Options](./options.md)
   * [Row and Column Separators](./row_col_sep.md)
@@ -13,7 +14,7 @@
     
 --------------  
 
-# Using Value Converters
+# Using Value Converters for Reading CSV
 
 Value Converters allow you to do custom transformations specific rows, to help you massage the data so it fits the expectations of your down-stream process, such as creating a DB record.
 

data/lib/smarter_csv/options.rb

@@ -20,7 +20,6 @@ module SmarterCSV
       downcase_header: true,
       duplicate_header_suffix: '', # was: nil,
       file_encoding: 'utf-8',
-      force_simple_split: false,
       force_utf8: false,
       headers_in_file: true,
       invalid_byte_sequence: '',

data/lib/smarter_csv/parser.rb

@@ -88,7 +88,9 @@ module SmarterCSV
 
       # Check for unclosed quotes at the end of the line
       if in_quotes
+        # :nocov:
         raise MalformedCSV, "Unclosed quoted field detected in line: #{line}"
+        # :nocov:
       end
 
       # Process the remaining field

data/lib/smarter_csv/reader.rb

@@ -112,7 +112,9 @@ module SmarterCSV
                 raise MalformedCSV, "Unclosed quoted field detected in multiline data"
               else
                 # Quotes are balanced; proceed without raising an error.
+                # :nocov:
                 break
+                # :nocov:
               end
             end
           end

data/lib/smarter_csv/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SmarterCSV
-  VERSION = "1.13.0"
+  VERSION = "1.14.0"
 end

data/lib/smarter_csv/writer.rb

@@ -1,15 +1,17 @@
 # frozen_string_literal: true
 
+require 'tempfile'
+
 module SmarterCSV
   #
   # Generate CSV files
   #
   # Create an instance of the Writer class with the filename and options.
-  # call `<<` one or mulltiple times to append data to the file.
+  # call `<<` one or multiple times to append data to the file.
   # call `finalize` to save the file.
   #
   # The `<<` method can take different arguments:
-  #  * a signle Hash
+  #  * a single Hash
   #  * an array of Hashes
   #  * nested arrays of arrays of Hashes
   #
@@ -29,6 +31,7 @@ module SmarterCSV
   #   headers : defaults to []
   #   force_quotes: defaults to false
   #   map_headers: defaults to {}, can be a hash of key -> value mappings
+  #   value_converters: optional hash of key -> lambda to control serialization
 
   # IMPORTANT NOTES:
   #  * Data hashes could contain strings or symbols as keys.
@@ -41,30 +44,32 @@ module SmarterCSV
     def initialize(file_path, options = {})
       @options = options
 
-      @row_sep = options[:row_sep] || $/ # Defaults to system's row separator. RFC4180 "\r\n"
+      @row_sep = options[:row_sep] || $/
       @col_sep = options[:col_sep] || ','
       @quote_char = options[:quote_char] || '"'
       @force_quotes = options[:force_quotes] == true
-      @discover_headers = true # defaults to true
+      @disable_auto_quoting = options[:disable_auto_quoting] == true
+      @value_converters = options[:value_converters] || {}
+      @map_all_keys = @value_converters.has_key?(:_all)
+      @mapped_keys = @value_converters.keys - [:_all]
+
+      @discover_headers = true
       if options.has_key?(:discover_headers)
-        # passing in the option overrides the default behavior
-        @discover_headers = options[:discover_headers] == true
+        @discover_headers = options[:discover_headers] == true # ⚠️ this option should not be exposed
       else
-        # disable discover_headers when headers are given explicitly
         @discover_headers = !(options.has_key?(:map_headers) || options.has_key?(:headers))
       end
-      @headers = [] # start with empty headers
-      @headers = options[:headers] if options.has_key?(:headers) # unless explicitly given
+
+      @headers = []
+      @headers = options[:headers] if options.has_key?(:headers)
       @headers = options[:map_headers].keys if options.has_key?(:map_headers) && !options.has_key?(:headers)
       @map_headers = options[:map_headers] || {}
 
       @output_file = File.open(file_path, 'w+')
-      # hidden state:
       @temp_file = Tempfile.new('tempfile', '/tmp')
       @quote_regex = Regexp.union(@col_sep, @row_sep, @quote_char)
     end
 
-    # this can be called many times in order to append lines to the csv file
     def <<(data)
       case data
       when Hash
@@ -74,13 +79,15 @@ module SmarterCSV
       when NilClass
         # ignore
       else
+        # :nocov:
         raise InvalidInputData, "Invalid data type: #{data.class}. Must be a Hash or an Array."
+        # :nocov:
       end
     end
 
     def finalize
-      # Map headers if :map_headers option is provided
       mapped_headers = @headers.map { |header| @map_headers[header] || header }
+      mapped_headers = mapped_headers.map { |x| escape_csv_field(x) } if @force_quotes
 
       @temp_file.rewind
       @output_file.write(mapped_headers.join(@col_sep) + @row_sep)
@@ -99,17 +106,43 @@ module SmarterCSV
         @headers.concat(new_keys)
       end
 
-      # Reorder the hash to match the current headers order and fill missing fields
-      ordered_row = @headers.map { |header| hash[header] || '' }
+      # Reorder the hash to match the current headers order and fill + map missing keys
+      ordered_row = @headers.map do |header|
+        has_header = hash.key?(header)
+        value = has_header ? hash[header] : '' # default to empty value
+
+        # first map individual keys
+        value = map_value(header, value) if @mapped_keys.include?(header)
+
+        # then apply general mapping rules
+        value = map_all_values(header, value) if @map_all_keys
+
+        escape_csv_field(value) # for backwards compatibility
+      end
 
-      @temp_file.write ordered_row.map { |value| escape_csv_field(value) }.join(@col_sep) + @row_sep
+      @temp_file.write ordered_row.join(@col_sep) + @row_sep
+    end
+
+    def map_value(key, value)
+      @value_converters[key].call(value)
+    end
+
+    def map_all_values(key, value)
+      @value_converters[:_all].call(key, value)
     end
 
     def escape_csv_field(field)
-      if @force_quotes || field.to_s.match(@quote_regex)
-        "\"#{field}\""
+      str = field.to_s
+      return str if @disable_auto_quoting
+
+      # double-quote fields if we force that, or if the field contains the comma, new-line, or quote character
+      contains_special_char = str.to_s.match(@quote_regex)
+      if @force_quotes || contains_special_char
+        str = str.gsub(@quote_char, @quote_char * 2) if contains_special_char # escape double-quote
+
+        "\"#{str}\""
       else
-        field.to_s
+        str
       end
     end
   end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: smarter_csv
 version: !ruby/object:Gem::Version
-  version: 1.13.0
+  version: 1.14.0
 platform: ruby
 authors:
 - Tilo Sloboda
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-11-05 00:00:00.000000000 Z
+date: 2025-04-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: awesome_print
@@ -117,7 +117,8 @@ files:
 - Rakefile
 - TO_DO_v2.md
 - docs/_introduction.md
-- docs/basic_api.md
+- docs/basic_read_api.md
+- docs/basic_write_api.md
 - docs/batch_processing.md
 - docs/data_transformations.md
 - docs/examples.md
@@ -165,7 +166,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.3
+rubygems_version: 3.5.4
 signing_key:
 specification_version: 4
 summary: Convenient CSV Reading and Writing