smarter_csv 1.11.2 → 1.12.0.pre1

data/docs/basic_api.md

@@ -0,0 +1,140 @@
+
+# SmarterCSV API
+
+Let's explore the basic APIs for reading and writing CSV files. There is a simplified API (backwards conpatible with previous SmarterCSV versions) and the full API, which allows you to access the internal state of the reader or writer instance after processing.
+
+## Reading CSV
+
+SmarterCSV has convenient defaults for automatically detecting row and column separators based on the given data. This provides more robust parsing of input files when you have no control over the data, e.g. when users upload CSV files.
+Learn more about this [in this section](docs/examples/row_col_sep.md).
+
+### Simplified Interface
+
+The simplified call to read CSV files is:
+
+      ```
+         array_of_hashes = SmarterCSV.process(file_or_input, options, &block)
+
+      ```
+It can also be used with a block:
+
+      ```
+         SmarterCSV.process(file_or_input, options, &block) do |hash|
+            # process one row of CSV
+         end
+      ```
+
+It can also be used for processing batches of rows:
+
+      ```
+         SmarterCSV.process(file_or_input, {chunk_size: 100}, &block) do |array_of_hashes|
+            # process one chunk of up to 100 rows of CSV data
+         end
+      ```
+
+### Full Interface
+
+The simplified API works in most cases, but if you need access to the internal state and detailed results of the CSV-parsing, you should use this form:
+
+      ```
+        reader = SmarterCSV::Reader.new(file_or_input, options)
+        data = reader.process
+
+        puts reader.raw_headers
+      ```
+It cal also be used with a block:
+
+      ```      
+        reader = SmarterCSV::Reader.new(file_or_input, options)
+        data = reader.process do 
+           # do something here
+        end
+
+        puts reader.raw_headers
+      ```
+
+This allows you access to the internal state of the `reader` instance after processing.
+
+
+## Interface for Writing CSV
+
+To generate a CSV file, we use the `<<` operator to append new data to the file.
+
+The input operator for adding data to a CSV file `<<` can handle single hashes, array-of-hashes, or array-of-arrays-of-hashes, and can be called one or multiple times for each file.
+
+One smart feature of writing CSV data is the discovery of headers. 
+
+If you have hashes of data, where each hash can have different keys, the `SmarterCSV::Reader` automatically discovers the superset of keys as the headers of the CSV file. This can be disabled by either providing one of the options `headers`, `map_headers`, or `discover_headers: false`.
+
+
+### Simplified Interface
+
+The simplified interface takes a block:
+
+      ```
+        SmarterCSV.generate(filename, options) do |csv_writer|
+
+         MyModel.find_in_batches(batch_size: 100) do |batch|
+           batch.pluck(:name, :description, :instructor).each do |record|
+             csv_writer << record
+           end
+         end
+
+       end
+     ```
+
+### Full Interface
+
+      ```
+        writer = SmarterCSV::Writer.new(file_path, options)
+
+        MyModel.find_in_batches(batch_size: 100) do |batch|
+          batch.pluck(:name, :description, :instructor).each do |record|
+            csv_writer << record
+          end
+
+        writer.finalize
+      ```
+
+## Rescue from Exceptions
+
+While SmarterCSV uses sensible defaults to process the most common CSV files, it will raise exceptions if it can not auto-detect `col_sep`, `row_sep`, or if it encounters other problems. Therefore please rescue from `SmarterCSV::Error`, and handle outliers according to your requirements.
+
+If you encounter unusual CSV files, please follow the tips in the Troubleshooting section below. You can use the options below to accomodate for unusual formats.
+
+## Troubleshooting
+
+In case your CSV file is not being parsed correctly, try to examine it in a text editor. For closer inspection  a tool like `hexdump` can help find otherwise hidden control character or byte sequences like [BOMs](https://en.wikipedia.org/wiki/Byte_order_mark).
+
+```
+$ hexdump -C spec/fixtures/bom_test_feff.csv
+00000000  fe ff 73 6f 6d 65 5f 69  64 2c 74 79 70 65 2c 66  |..some_id,type,f|
+00000010  75 7a 7a 62 6f 78 65 73  0d 0a 34 32 37 36 36 38  |uzzboxes..427668|
+00000020  30 35 2c 7a 69 7a 7a 6c  65 73 2c 31 32 33 34 0d  |05,zizzles,1234.|
+00000030  0a 33 38 37 35 39 31 35  30 2c 71 75 69 7a 7a 65  |.38759150,quizze|
+00000040  73 2c 35 36 37 38 0d 0a                           |s,5678..|
+```
+
+## Assumptions / Limitations
+
+* the escape character is `\`, as on UNIX and Windows systems.
+* quote charcters around fields are balanced, e.g. valid: `"field"`, invalid: `"field\"`
+  e.g. an escaped `quote_char` does not denote the end of a field.
+
+
+## NOTES about File Encodings:
+ * if you have a CSV file which contains unicode characters, you can process it as follows:
+
+```ruby
+       File.open(filename, "r:bom|utf-8") do |f|
+         data = SmarterCSV.process(f);
+       end
+```
+* if the CSV file with unicode characters is in a remote location, similarly you need to give the encoding as an option to the `open` call:
+```ruby
+       require 'open-uri'
+       file_location = 'http://your.remote.org/sample.csv'
+       open(file_location, 'r:utf-8') do |f|   # don't forget to specify the UTF-8 encoding!!
+         data = SmarterCSV.process(f)
+       end
+```

data/docs/batch_processing.md

@@ -0,0 +1,53 @@
+
+# Batch Processing
+
+Processing CSV data in batches (chunks), allows you to parallelize the workload of importing data.
+This can come in handy when you don't want to slow-down the CSV import of large files.
+
+Setting the option `chunk_size` sets the max batch size.
+
+
+## Example 1: How SmarterCSV processes CSV-files as chunks, returning arrays of hashes:
+Please note how the returned array contains two sub-arrays containing the chunks which were read, each chunk containing 2 hashes.
+In case the number of rows is not cleanly divisible by `:chunk_size`, the last chunk contains fewer hashes.
+
+```ruby
+     > pets_by_owner = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}})
+       => [ [ {:first=>"Dan", :last=>"McAllister", :dogs=>"2"}, {:first=>"Lucy", :last=>"Laweless", :cats=>"5"} ],
+            [ {:first=>"Miles", :last=>"O'Brian", :fish=>"21"}, {:first=>"Nancy", :last=>"Homes", :dogs=>"2", :birds=>"1"} ]
+          ]
+```
+
+## Example 2: How SmarterCSV processes CSV-files as chunks, and passes arrays of hashes to a given block:
+Please note how the given block is passed the data for each chunk as the parameter (array of hashes),
+and how the `process` method returns the number of chunks when called with a block
+
+```ruby
+     > total_chunks = SmarterCSV.process('/tmp/pets.csv', {:chunk_size => 2, :key_mapping => {:first_name => :first, :last_name => :last}}) do |chunk|
+         chunk.each do |h|   # you can post-process the data from each row to your heart's content, and also create virtual attributes:
+           h[:full_name] = [h[:first],h[:last]].join(' ')  # create a virtual attribute
+           h.delete(:first) ; h.delete(:last)              # remove two keys
+         end
+         puts chunk.inspect   # we could at this point pass the chunk to a Resque worker..
+       end
+
+       [{:dogs=>"2", :full_name=>"Dan McAllister"}, {:cats=>"5", :full_name=>"Lucy Laweless"}]
+       [{:fish=>"21", :full_name=>"Miles O'Brian"}, {:dogs=>"2", :birds=>"1", :full_name=>"Nancy Homes"}]
+        => 2
+```
+
+## Example 3: Populate a MongoDB Database in Chunks of 100 records with SmarterCSV:
+```ruby
+    # using chunks:
+    filename = '/tmp/some.csv'
+    options = {:chunk_size => 100, :key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}}
+    n = SmarterCSV.process(filename, options) do |chunk|
+          # we're passing a block in, to process each resulting hash / row (block takes array of hashes)
+          # when chunking is enabled, there are up to :chunk_size hashes in each chunk
+          MyModel.collection.insert( chunk )   # insert up to 100 records at a time
+    end
+
+     => returns number of chunks we processed
+```
+
+

data/docs/data_transformations.md

@@ -0,0 +1,32 @@
+# Data Transformations
+
+SmarterCSV automatically transforms the values in each colum in order to normalize the data.
+This behavior can be customized or disabled.
+
+## Remove Empty Values
+`remove_empty_values` is enabled by default
+It removes any values which are `nil` or would be empty strings.
+
+## Convert Values to Numeric
+`convert_values_to_numeric` is enabled by default. 
+SmarterCSV will convert strings containing Integers or Floats to the appropriate class.
+
+## Remove Zero Values
+`remove_zero_values` is disabled by default.
+When enabled, it removes key/value pairs which have a numeric value equal to zero.
+
+## Remove Values Matching
+`remove_values_matching` is disabled by default. 
+When enabled, this can help removing key/value pairs from result hashes which would cause problems. 
+
+e.g.
+ * `remove_values_matching: /^\$0\.0+$/` would remove $0.00 
+ * `remove_values_matching: /^#VALUE!$/` would remove errors from Excel spreadsheets 
+
+## Empty Hashes
+
+It can happen that after all transformations, a row of the CSV file would produce a completely empty hash.
+
+By default SmarterCSV uses `remove_empty_hashes: true` to remove these empty hashes from the result.
+
+This can be set to `true`, to keep these empty hashes in the results.

data/docs/examples.md

@@ -0,0 +1,61 @@
+
+# Examples
+
+Here are some examples to demonstrate the versatility of SmarterCSV.
+
+**It is generally recommended to rescue `SmarterCSV::Error` or it's sub-classes.**
+
+By default SmarterCSV determines the `row_sep` and `col_sep` values automatically. In cases where the automatic detection fails, an exception will be raised, e.g. `NoColSepDetected`. Rescuing from these exceptions will make sure that you don't miss processing CSV files, in case users upload CSV files with unexpected formats.
+
+In rare cases you may have to manually set these values, after going through the troubleshooting procedure described above.
+
+## Example 1a: How SmarterCSV processes CSV-files as array of hashes:
+Please note how each hash contains only the keys for columns with non-null values.
+
+```ruby
+     $ cat pets.csv
+     first name,last name,dogs,cats,birds,fish
+     Dan,McAllister,2,,,
+     Lucy,Laweless,,5,,
+     Miles,O'Brian,,,,21
+     Nancy,Homes,2,,1,
+     $ irb
+     > require 'smarter_csv'
+      => true
+     > pets_by_owner = SmarterCSV.process('/tmp/pets.csv')
+      => [ {:first_name=>"Dan", :last_name=>"McAllister", :dogs=>"2"},
+           {:first_name=>"Lucy", :last_name=>"Laweless", :cats=>"5"},
+           {:first_name=>"Miles", :last_name=>"O'Brian", :fish=>"21"},
+           {:first_name=>"Nancy", :last_name=>"Homes", :dogs=>"2", :birds=>"1"}
+         ]
+```
+
+
+## Example 3: Populate a MySQL or MongoDB Database with SmarterCSV:
+```ruby
+    # without using chunks:
+    filename = '/tmp/some.csv'
+    options = {:key_mapping => {:unwanted_row => nil, :old_row_name => :new_name}}
+    n = SmarterCSV.process(filename, options) do |array|
+          # we're passing a block in, to process each resulting hash / =row (the block takes array of hashes)
+          # when chunking is not enabled, there is only one hash in each array
+          MyModel.create( array.first )
+    end
+
+     => returns number of chunks / rows we processed
+```
+
+## Example 4: Processing a CSV File, and inserting batch jobs in Sidekiq:
+```ruby
+    filename = '/tmp/input.csv' # CSV file containing ids or data to process
+    options = { :chunk_size => 100 }
+    n = SmarterCSV.process(filename, options) do |chunk|
+      Sidekiq::Client.push_bulk(
+        'class' => SidekiqIndividualWorkerClass,
+        'args' => chunk,
+      )
+      # OR:
+      # SidekiqBatchWorkerClass.process_async(chunk ) # pass an array of hashes to Sidekiq workers for parallel processing
+    end
+    => returns number of chunks
+```

data/docs/header_transformations.md

@@ -0,0 +1,95 @@
+# Header Transformations
+
+By default SmarterCSV assumes that a CSV file has headers, and it automatically normalizes the headers and transforms them into Ruby symbols. You can completely customize or override this (see below).
+
+## Header Normalization
+
+When processing the headers, it transforms them into Ruby symbols, stripping extra spaces, lower-casing them and replacing spaces with underscores. e.g. " \t Annual Sales  " becomes `:annual_sales`. (see Notes below)
+
+## Duplicate Headers
+
+There can be a lot of variation in CSV files. It is possible that a CSV file contains multiple headers with the same name. 
+
+By default SmarterCSV handles duplicate headers by appending numbers 2..n to them.
+
+Consider this example:
+
+```
+$ cat > /tmp/dupe.csv
+name,name,name
+Carl,Edward,Sagan
+```
+
+When parsing these duplicate headers, SmarterCSV will return:
+
+```
+  data = SmarterCSV.process('/tmp/dupe.csv')
+   => [{:name=>"Carl", :name2=>"Edward", :name3=>"Sagan"}]
+```
+
+If you want to have an underscore between the header and the number, you can set `duplicate_header_suffix: '_'`.
+
+```
+  data = SmarterCSV.process('/tmp/dupe.csv', {duplicate_header_suffix: '_'})
+   => [{:name=>"Carl", :name_2=>"Edward", :name_3=>"Sagan"}]
+```
+ 
+ To further disambiguate the headers, you can further use `key_mapping` to assign meaningful names. Please note that the mapping uses the already transformed keys `name_2`, `name_3` as input.
+   
+```
+  options = {
+    duplicate_header_suffix: '_', 
+    key_mapping: {
+      name: :first_name, 
+      name_2: :middle_name, 
+      name_3: :last_name,
+    }
+  }
+  data = SmarterCSV.process('/tmp/dupe.csv', options)
+   => [{:first_name=>"Carl", :middle_name=>"Edward", :last_name=>"Sagan"}]
+```
+
+## Key Mapping
+
+The above example already illustrates how intermediate keys can be mapped into something different.
+This transfoms some of the keys in the input, but other keys are still present.
+
+There is an additional option `remove_unmapped_keys` which can be enabled to only produce the mapped keys in the resulting hashes, and drops any other columns.
+
+ 
+### NOTES on Key Mapping:
+ * keys in the header line of the file can be re-mapped to a chosen set of symbols, so the resulting Hashes can be better used internally in your application (e.g. when directly creating MongoDB entries with them)
+ * if you want to completely delete a key, then map it to nil or to '', they will be automatically deleted from any result Hash
+ * if you have input files with a large number of columns, and you want to ignore all columns which are not specifically mapped with :key_mapping, then use option :remove_unmapped_keys => true
+
+## CSV Files without Headers
+
+If you have CSV files without headers, it is important to set `headers_in_file: false`, otherwise you'll lose the first data line in your file.
+You then have to provide `user_provided_headers`, which takes an array of either symbols or strings.
+
+
+## CSV Files with Headers
+
+For CSV files with headers, you can either:
+
+* use the automatic header normalization
+* map one or more headers into whatever you chose using the `map_headers` option.
+  (if you map a header to `nil`, it will remove that column from the resulting row hash).
+* completely replace the headers using `user_provided_headers` (please be careful with this powerful option, as it is not robust against changes in input format).
+* use the original unmodified headers from the CSV file, using `keep_original_headers`. This results in hash keys that are strings, and may be padded with spaces.
+
+
+# Notes
+
+### NOTES about CSV Headers:
+ * as this method parses CSV files, it is assumed that the first line of any file will contain a valid header
+ * the first line with the header might be commented out, in which case you will need to set `comment_regexp: /\A#/`
+ * any occurences of :comment_regexp or :row_sep will be stripped from the first line with the CSV header
+ * any of the keys in the header line will be downcased, spaces replaced by underscore, and converted to Ruby symbols before being used as keys in the returned Hashes
+ * you can not combine the :user_provided_headers and :key_mapping options
+ * if the incorrect number of headers are provided via :user_provided_headers, exception SmarterCSV::HeaderSizeMismatch is raised
+
+### NOTES on improper quotation and unwanted characters in headers:
+ * some CSV files use un-escaped quotation characters inside fields. This can cause the import to break. To get around this, use the `:force_simple_split => true` option in combination with `:strip_chars_from_headers => /[\-"]/` . This will also significantly speed up the import.
+   If you would force a different :quote_char instead (setting it to a non-used character), then the import would be up to 5-times slower than using `:force_simple_split`.
+

data/docs/header_validations.md

@@ -0,0 +1,18 @@
+# Header Validations
+
+When you are importing data, it can be important to verify that all required data is present, to ensure consistent quality when importing data.
+
+You can use the `required_keys` option to specify an array of hash keys that you require to be present at a minimum for every data row (after header transformation).
+
+If these keys are not present, `SmarterCSV::MissingKeys` will be raised to inform you of the data inconsistency.
+
+## Example
+
+```ruby
+  options = {
+    required_keys: [:source_account, :destination_account, :amount]
+  }
+  data = SmarterCSV.process("/tmp/transactions.csv", options)
+
+  => this will raise SmarterCSV::MissingKeys if any row does not contain these three keys
+```

data/docs/notes.md

@@ -0,0 +1,29 @@
+
+# Notes
+
+
+
+
+## NOTES on the use of Chunking and Blocks:
+ * chunking can be VERY USEFUL if used in combination with passing a block to File.read_csv FOR LARGE FILES
+ * if you pass a block to File.read_csv, that block will be executed and given an Array of Hashes as the parameter.
+ * if the chunk_size is not set, then the array will only contain one Hash.
+ * if the chunk_size is > 0 , then the array may contain up to chunk_size Hashes.
+ * this can be very useful when passing chunked data to a post-processing step, e.g. through Sidekiq
+
+## NOTES about File Encodings:
+ * if you have a CSV file which contains unicode characters, you can process it as follows:
+
+```ruby
+       File.open(filename, "r:bom|utf-8") do |f|
+         data = SmarterCSV.process(f);
+       end
+```
+* if the CSV file with unicode characters is in a remote location, similarly you need to give the encoding as an option to the `open` call:
+```ruby
+       require 'open-uri'
+       file_location = 'http://your.remote.org/sample.csv'
+       open(file_location, 'r:utf-8') do |f|   # don't forget to specify the UTF-8 encoding!!
+         data = SmarterCSV.process(f)
+       end
+```

data/docs/options.md

@@ -0,0 +1,82 @@
+
+# SmarterCSV Options
+
+## CSV Writing
+
+     | Option                      | Default  |  Explanation                                                                         |
+     ---------------------------------------------------------------------------------------------------------------------------------
+     | :row_sep                    |   $/      | Separates rows; Defaults to your OS row separator. `/n` on UNIX, `/r/n` oon Windows | 
+     | :col_sep                    |   ","     | Separates each value in a row | 
+     | :quote_char                 |   '"'     | |
+     | :force_quotes               |   false   | Forces each individual value to be quoted |
+     | :discover_headers           |   true    | Automatically detects all keys in the input before writing the header |
+     |                             |           | This can be disabled by providing `headers` or `map_headers` options. |
+     | :headers                    |    []     | You can provide the specific list of keys from the input you'd like to be used as headers in the CSV file |
+     | :map_headers                |    {}     | Similar to `headers`, but also maps each desired key to a user-specified value that is uesd as the header. | 
+     |
+
+## CSV Reading
+
+     | Option                      | Default  |  Explanation                                                                         |
+     ---------------------------------------------------------------------------------------------------------------------------------
+     | :chunk_size                 |   nil    | if set, determines the desired chunk-size (defaults to nil, no chunk processing)     |
+     |                             |          |                                                                                      |
+     | :file_encoding              |   utf-8  | Set the file encoding eg.: 'windows-1252' or 'iso-8859-1'                            |
+     | :invalid_byte_sequence      |   ''     | what to replace invalid byte sequences with                                          |
+     | :force_utf8                 |   false  | force UTF-8 encoding of all lines (including headers) in the CSV file                |
+     | :skip_lines                 |   nil    | how many lines to skip before the first line or header line is processed             |
+     | :comment_regexp             |   nil    | regular expression to ignore comment lines (see NOTE on CSV header), e.g./\A#/       |
+     ---------------------------------------------------------------------------------------------------------------------------------
+     | :col_sep                    |   :auto   | column separator (default was ',')                                           |
+     | :force_simple_split         |   false  | force simple splitting on :col_sep character for non-standard CSV-files.             |
+     |                             |          | e.g. when :quote_char is not properly escaped                                        |
+     | :row_sep                    |  :auto   | row separator or record separator (previous default was system's $/ , which defaulted to "\n") |
+     |                             |          | This can also be set to :auto, but will process the whole cvs file first  (slow!)    |
+     | :auto_row_sep_chars         |   500    | How many characters to analyze when using `:row_sep => :auto`. nil or 0 means whole file. |
+     | :quote_char                 |   '"'    | quotation character                                                                  |
+     ---------------------------------------------------------------------------------------------------------------------------------
+     | :headers_in_file            |   true   | Whether or not the file contains headers as the first line.                          |
+     |                             |          | Important if the file does not contain headers,                                      |
+     |                             |          | otherwise you would lose the first line of data.                                     |
+     | :duplicate_header_suffix    |   ''     | Adds numbers to duplicated headers and separates them by the given suffix.           |
+     |                             |          | Set this to nil to raise `DuplicateHeaders` error instead (previous behavior)        |
+     | :user_provided_headers      |   nil    | *careful with that axe!*                                                             |
+     |                             |          | user provided Array of header strings or symbols, to define                          |
+     |                             |          | what headers should be used, overriding any in-file headers.                         |
+     |                             |          | You can not combine the :user_provided_headers and :key_mapping options              |
+     | :remove_empty_hashes        |   true   | remove / ignore any hashes which don't have any key/value pairs or all empty values  |
+     | :verbose                    |   false  | print out line number while processing (to track down problems in input files)       |
+     | :with_line_numbers          |   false  | add :csv_line_number to each data hash                                               |
+     ---------------------------------------------------------------------------------------------------------------------------------
+
+Additional 1.x Options which may be replaced in 2.0
+
+There have been a lot of 1-offs and feature creep around these options, and going forward we'll strive to have a simpler, but more flexible way to address these features.
+
+
+     | Option                      | Default  |  Explanation                                                                         |
+     ---------------------------------------------------------------------------------------------------------------------------------
+     | :key_mapping                |   nil    | a hash which maps headers from the CSV file to keys in the result hash               |
+     | :silence_missing_keys        |   false  | ignore missing keys in `key_mapping`                                   |
+     |                             |          | if set to true: makes all mapped keys optional                         |
+     |                             |          | if given an array, makes only the keys listed in it optional                         |
+     | :required_keys              |   nil    | An array. Specify the required names AFTER header transformation.                  |
+     | :required_headers           |   nil    | (DEPRECATED / renamed) Use `required_keys` instead                          |
+     |                             |          | or an exception is raised   No validation if nil is given.                           |
+     | :remove_unmapped_keys       |   false  | when using :key_mapping option, should non-mapped keys / columns be removed?         |
+     | :downcase_header            |   true   | downcase all column headers                                                          |
+     | :strings_as_keys            |   false  | use strings instead of symbols as the keys in the result hashes                      |
+     | :strip_whitespace           |   true   | remove whitespace before/after values and headers                                    |
+     | :keep_original_headers      |   false  | keep the original headers from the CSV-file as-is.                                   |
+     |                             |          | Disables other flags manipulating the header fields.                                 |
+     | :strip_chars_from_headers   |   nil    | RegExp to remove extraneous characters from the header line (e.g. if headers are quoted) |
+     ---------------------------------------------------------------------------------------------------------------------------------
+     | :value_converters           |   nil    | supply a hash of :header => KlassName; the class needs to implement self.convert(val)|
+     | :remove_empty_values        |   true   | remove values which have nil or empty strings as values                              |
+     | :remove_zero_values         |   false  | remove values which have a numeric value equal to zero / 0                           |
+     | :remove_values_matching     |   nil    | removes key/value pairs if value matches given regular expressions. e.g.:            |
+     |                             |          | /^\$0\.0+$/ to match $0.00 , or /^#VALUE!$/ to match errors in Excel spreadsheets    |
+     | :convert_values_to_numeric  |   true   | converts strings containing Integers or Floats to the appropriate class              |
+     |                             |          |      also accepts either {:except => [:key1,:key2]} or {:only => :key3}              |
+     ---------------------------------------------------------------------------------------------------------------------------------
+

data/docs/row_col_sep.md

@@ -0,0 +1,87 @@
+
+# Row and Column Separators
+
+## Automatic Detection
+
+Convenient defaults allow automatic detection of the column and row separators: `row_sep: :auto`, `col_sep: :auto`. This makes it easier to process any CSV files without having to examine the line endings or column separators, e.g. when users upload CSV files to your service and you have no control over the incoming files.
+
+You can change the setting `:auto_row_sep_chars` to only analyze the first N characters of the file (default is 500 characters); `nil` or `0` will check the whole file). Of course you can also set the `:row_sep` manually.
+
+
+## Column Separator `col_sep`
+
+The automatic detection of column separators considers: `,`, `\t`, `;`, `:`, `|`.
+
+Some CSV files may contain an unusual column separqator, which could even be a control character.
+
+## Row Separator `row_sep`
+
+The automatic detection of row separators considers: `\n`, `\r\n`, `\r`.
+
+Some CSV files may contain an unusual row separqator, which could even be a control character.
+
+
+## Custom / Non-Standard CSV Formats
+
+Besides custom values for `col_sep`, `row_sep`, some other customizations of CSV files are:
+*  the presence of a number of leading lines before the header or data section start.
+*  the presence of comment lines, e.g. lines starting with `#`
+
+To explore these special cases, please use the following examples.
+
+### Example 1: reading an iTunes DB dump
+
+This data format uses CTRL-A as the column separator, and CTRL-B as the record separator. It also has comment lines that start with a `#` character. This also maps the header `name` to `genre`, and ignores the column `export_date`.
+
+```ruby
+    filename = '/tmp/itunes_db_dump'   
+    options = {
+      :col_sep => "\cA", :row_sep => "\cB\n", :comment_regexp => /^#/,
+      :chunk_size => 100 , :key_mapping => {export_date: nil, name: :genre},
+    }
+    n = SmarterCSV.process(filename, options) do |chunk|
+      SidekiqWorkerClass.process_async(chunk) # pass an array of hashes to Sidekiq workers for parallel processing
+    end
+    => returns number of chunks
+```
+
+### Example 2: Reading a CSV-File with custom col_sep, row_sep
+In this example we have an unusual CSV file with `|` as the row separator, and `#` as the column separator.
+This unusual format needs explicit options `col_sep` and `row_sep`.
+
+```ruby
+    filename = '/tmp/input_file.txt'
+    recordsA = SmarterCSV.process(filename, {col_sep: "#", row_sep: "|"})
+
+    => returns an array of hashes
+```
+
+### Example 3:
+In this example, we use `skip_lines: 3` to skip and ignore the first 3 lines in the input
+
+
+```ruby
+    filename = '/tmp/input_file.txt'
+    recordsA = SmarterCSV.process(filename, {skip_lines: 3})
+
+    => returns an array of hashes
+```
+  
+
+### Example 4: reading an iTunes DB dump
+
+In this example, we use `comment_regexp` to filter out and ignore any lines starting with `#`
+
+
+```ruby
+    # Consider a file with CRTL-A as col_separator, and with CTRL-B\n as record_separator (hello iTunes!)
+    filename = '/tmp/strange_db_dump'   
+    options = {
+      :col_sep => "\cA", :row_sep => "\cB\n", :comment_regexp => /^#/,
+      :chunk_size => 100 , :key_mapping => {:export_date => nil, :name => :genre},
+    }
+    n = SmarterCSV.process(filename, options) do |chunk|
+      SidekiqWorkerClass.process_async(chunk) # pass an array of hashes to Sidekiq workers for parallel processing
+    end
+    => returns number of chunks
+```

data/docs/value_converters.md

@@ -0,0 +1,51 @@
+
+# Using Value Converters
+
+Value Converters allow you to do custom transformations specific rows, to help you massage the data so it fits the expectations of your down-stream process, such as creating a DB record.
+
+If you use `key_mappings` and `value_converters`, make sure that the value converters references the keys based on the final mapped name, not the original name in the CSV file.
+
+```ruby
+    $ cat spec/fixtures/with_dates.csv
+    first,last,date,price
+    Ben,Miller,10/30/1998,$44.50
+    Tom,Turner,2/1/2011,$15.99
+    Ken,Smith,01/09/2013,$199.99
+
+    $ irb
+    > require 'smarter_csv'
+    > require 'date'
+
+    # define a custom converter class, which implements self.convert(value)
+    class DateConverter
+      def self.convert(value)
+        Date.strptime( value, '%m/%d/%Y') # parses custom date format into Date instance
+      end
+    end
+
+    class DollarConverter
+      def self.convert(value)
+        value.sub('$','').to_f # strips the dollar sign and creates a Float value
+      end
+    end
+
+    require 'money'
+    class MoneyConverter
+      def self.convert(value)
+        # depending on locale you might want to also remove the indicator for thousands, e.g. comma 
+        Money.from_amount(value.gsub(/[\s\$]/,'').to_f) # creates a Money instance (based on cents)
+      end
+    end
+
+    options = {:value_converters => {:date => DateConverter, :price => DollarConverter}}
+    data = SmarterCSV.process("spec/fixtures/with_dates.csv", options)
+    first_record = data.first
+    first_record[:date]
+      => #<Date: 1998-10-30 ((2451117j,0s,0n),+0s,2299161j)>
+    first_record[:date].class
+      => Date
+    first_record[:price]
+      => 44.50
+    first_record[:price].class
+      => Float
+```