csv-utils 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 423eacb18512c595abe83765ccfd4d945f1e2e0d18a6370b6841997eb428ddb7
-  data.tar.gz: 3d2df7f1ee62eaebc25d00281401bf49f0fccdb809ec03cbac90e6bf5ba2fd62
+  metadata.gz: 9bae4a7090c060824d53643d89639336e44da08940ab6e9a629a2ed4fffd7837
+  data.tar.gz: 2e4ab9e02540822dbd64cbfd1ca9eebac2ddd24bb66592fb587e3a136e3a50f1
 SHA512:
-  metadata.gz: 2f9856120036ae6a87e2e758186221b7c0b48aa8d94365f7bdf4bffdf8f292868e43cf3c378eff7460902d710de2859d5406ac9ef5384c07f9728f2f1bad4ef4
-  data.tar.gz: 9e96e672cb1f4da44ae67de48d3e783cf1d2b180c0d8f4a9706ce1ace07b9ac456a689653dfe84537c0cab71e6c58cdb24e5caeec0bcc86bc73a374615ac4e0a
+  metadata.gz: 11b8b4ccd4cfaa040f537ebf62c8f835537e9822b253d8ae5af2ee8212a6692e2ad034166b504a9bd7d2845f2b731bb2405e10ba281dc73b82bdf4f37400c1ec
+  data.tar.gz: a16ae82b7e3016ab2b1c4e495bd882d083bf808eedd89c19e25b8adda91d6e123832096e3c60a9d28873a52cb65d203b1293f596ab3c1c3c0faa04a24a2bc35e

data/CHANGELOG.md

@@ -1,5 +1,38 @@
 # Changelog
 
+## [0.8.0](https://github.com/dougyouch/csv-utils/compare/v0.7.0...v0.8.0) (2026-10-05)
+
+
+### Features
+
+* **cli:** detect separators and encodings, one-pass splitter, lint scripts ([5bfc98b](https://github.com/dougyouch/csv-utils/commit/5bfc98b843c01fcc4a86f3516248942eb4e96b53))
+* **compare:** raise on unsorted input and missing update columns ([1ba7151](https://github.com/dougyouch/csv-utils/commit/1ba715178432570cdb0c3490d5f5f8e67ce895dc))
+* **iterator:** report the physical line rows start on and add CSVUtils errors ([242d747](https://github.com/dougyouch/csv-utils/commit/242d747c2e9a5a6bcb14a682751860142028c4aa))
+* **options:** detect separators by count, support semicolons, add full_scan ([1492864](https://github.com/dougyouch/csv-utils/commit/1492864075570065c108190a700edf6154a2dd48))
+
+
+### Performance Improvements
+
+* **sort:** merge up to 64 part files per pass and add sort_by and tmp_dir ([3245788](https://github.com/dougyouch/csv-utils/commit/3245788afabe2dbdf21b534065a54c3547f1183a))
+
+## [0.7.0](https://github.com/dougyouch/csv-utils/compare/v0.6.0...v0.7.0) (2026-10-05)
+
+
+### ⚠ BREAKING CHANGES
+
+* **encoding:** CSVIterator no longer takes a file mode, files are decoded as UTF-8 whatever the locale, and output is written in the decoded encoding. See UPGRADING.md.
+
+### Features
+
+* **encoding:** read every file with the encoding option and write the decoded encoding ([1a9ec4f](https://github.com/dougyouch/csv-utils/commit/1a9ec4f98cbaadfff7ef89fec0d586cea828cf91))
+* **iterator:** open files per call, build rows without zip, add auto_detect ([34ab89a](https://github.com/dougyouch/csv-utils/commit/34ab89ae49dad76507371d2bdf9a7cd310c2cf67))
+* **options:** detect windows-1252 and read utf-8 regardless of locale ([10eeca6](https://github.com/dougyouch/csv-utils/commit/10eeca69bf094fb8c2cb5b005bc74687bde61169))
+
+
+### Bug Fixes
+
+* **iterator:** strip the byte order mark after rewind and from explicit-header rows ([bf8804b](https://github.com/dougyouch/csv-utils/commit/bf8804b1c2ce16738779f3b9ecc7631c3b29a115))
+
 ## [0.6.0](https://github.com/dougyouch/csv-utils/compare/v0.5.0...v0.6.0) (2026-10-05)
 
 

data/README.md

@@ -6,7 +6,7 @@ Streaming tools for CSV files that are too big or too broken to load into memory
 [![Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/coverage.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
 [![Branch Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/branches.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
 
-[API reference](https://rubydoc.info/gems/csv-utils) · [Changelog](CHANGELOG.md) · [Architecture](ARCHITECTURE.md)
+[API reference](https://rubydoc.info/gems/csv-utils) · [Changelog](CHANGELOG.md) · [Upgrading](UPGRADING.md) · [Architecture](ARCHITECTURE.md)
 
 ## Features
 
@@ -19,6 +19,7 @@ Streaming tools for CSV files that are too big or too broken to load into memory
 - **CSV Iteration**: Efficient iteration over CSV files with batch support
 - **CSV Extension**: Extend CSV files with additional columns
 - **CSV Options**: Auto-detect CSV file properties (separators, encoding, BOM)
+- **Encodings**: Files are read as UTF-8 whatever the locale, or in the encoding you pass or `CSVOptions` detects
 - **Byte Order Marks**: UTF-8, UTF-16 and UTF-32 byte order marks are stripped from the first header
 - **CLI Tools**: Command-line utilities for CSV debugging and manipulation
 
@@ -75,7 +76,12 @@ CSVUtils::CSVCompare.new('primary.tsv', ['updated_at'], col_sep: "\t") { |src, d
 
 **Note**: Both CSV files must be sorted by the same key columns for comparison to work correctly, and the block must compare them in that same order.
 
-The block compares the key columns of a primary and a secondary record like `<=>`. Records only in the primary file are yielded as `:create`, records only in the secondary file as `:delete`, and matching records as `:update` when any of the update comparison columns differ. Without update comparison columns, no updates are yielded.
+The block compares the key columns of a primary and a secondary record like `<=>`. Records only in the primary file are yielded as `:create`, records only in the secondary file as `:delete`, and matching records as `:update` when any of the update comparison columns differ. Without update comparison columns, no updates are yielded. Records are hashes of header to value that also know the line they start on (`record.lineno`).
+
+`compare` checks its inputs instead of returning wrong results:
+
+- A file that isn't sorted the way the block expects raises `CSVUtils::UnsortedFileError` with the file and the line of the first record out of order. Each record is compared with the one before it in the same file, so the check only runs for a file whose first record the block compares equal to itself; a block written for different headers in each file (`src['id'] <=> dest['ID']`) skips it. Pass `check_order: false` to turn it off.
+- An update comparison column missing from either file raises `CSVUtils::HeaderNotFoundError`, rather than never reporting updates.
 
 ### Sorting CSV Files
 
@@ -87,11 +93,18 @@ require 'csv-utils'
 sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true)  # true = has headers
 sorter.sort(100_000) { |a, b| a.first.to_i <=> b.first.to_i }    # batch size, comparison block
 
+# faster: compute a key once per row instead of converting values on every comparison
+sorter.sort_by(100_000) { |row| row.first.to_i }
+sorter.sort_by { |row| [row[2], row[0].to_i] }                  # several columns
+
 # without a block, rows are compared as arrays of strings
 sorter.sort
+
+# write the temporary files somewhere other than next to output.csv
+sorter.sort_by(100_000, tmp_dir: '/mnt/scratch') { |row| row.first.to_i }
 ```
 
-Batches are sorted into temporary `output.csv.part.N` files next to the output, which are merged in pairs. The temporary files are removed when the sort finishes or fails.
+Batches are sorted into temporary files, which are then merged up to 64 at a time (`CSVSort::MERGE_WIDTH`), so files of up to 64 batches are merged in a single pass and every row is read and written about twice. Rows with equal keys keep the order of the batches they came from. The temporary files go next to the output unless you pass `tmp_dir:`, and are removed when the sort finishes or fails.
 
 ### Transforming CSV Data
 
@@ -230,7 +243,20 @@ lookup = iterator.to_hash('id', 'name')  # { 'id_value' => 'name_value', ... }
 iterator.each.with_index { |row, idx| puts "#{idx}: #{row['name']}" }
 ```
 
-Line numbers count the header row as line 1, so they match what an editor shows. `headers` returns `[]` for an empty file.
+Line numbers count the header row as line 1 and are the line each row starts on, so they match what an editor shows even when a quoted value has a line break in it (CSV's own line numbers count rows). `headers` returns `[]` for an empty file.
+
+A row CSV can't parse raises `CSVUtils::MalformedRowError`, a `CSV::MalformedCSVError` whose `line_number` is the line the row starts on and whose `prev_row` is the last row read. Bytes that don't match the encoding raise CSV's own `CSV::InvalidEncodingError`, which already has the right line.
+
+`to_hash` raises `CSVUtils::HeaderNotFoundError` for a header the file doesn't have. Every error this gem raises includes `CSVUtils::Error`, so `rescue CSVUtils::Error` catches them all, and also subclasses the error raised before it existed (`RuntimeError` or `CSV::MalformedCSVError`).
+
+Given a path, the iterator opens the file for each call (`each`, `headers`, `size`, ...) and closes it when the call returns, even when the block raises or stops early, so an idle iterator holds no file handle. A CSV object passed in is rewound, never closed.
+
+Let `CSVOptions` work out the separators and encoding:
+
+```ruby
+iterator = CSVUtils::CSVIterator.auto_detect('export.tsv')
+iterator = CSVUtils::CSVIterator.auto_detect('export.csv', full_scan: true) # check the encoding of the whole file
+```
 
 ### Matching CSV Rows
 
@@ -295,27 +321,44 @@ options = CSVUtils::CSVOptions.new('data.csv')
 options.valid?         # true if separators detected
 options.col_separator  # detected column separator
 options.row_separator  # detected row separator
-options.encoding       # detected encoding (UTF-8, UTF-16, UTF-32)
+options.encoding       # detected encoding (UTF-8, UTF-16, UTF-32, Windows-1252, ISO-8859-1)
 options.columns        # number of columns
 options.byte_order_mark # BOM if present
+
+# Open the file with what was detected, with CSV or any class in this gem
+CSV.open('data.csv', 'rb', **options.to_csv_options)
+CSVUtils::CSVSort.new('data.csv', 'sorted.csv', true, options.to_csv_options).sort
 ```
 
-Supported column separators: `\x02`, `\t`, `|`, `,` (the first one found in the header line wins)
+`to_csv_options` returns the detected `col_sep` and `row_sep`, leaving out any that weren't found, and an `encoding` that reads values as UTF-8 strings: `'bom|utf-8'` for UTF-8 files, `'BOM|UTF-16LE:UTF-8'` and the like for files with a UTF-16 or UTF-32 byte order mark (CSV detects the row separator for those), and `'Windows-1252:UTF-8'` for files that aren't valid UTF-8.
+
+Without a byte order mark, the encoding comes from the first megabyte (`CSVOptions::SAMPLE_SIZE`): valid UTF-8 is `UTF-8`, anything else is `Windows-1252`, the usual encoding of Excel exports, or `ISO-8859-1` when the sample has one of the five bytes Windows-1252 leaves undefined. A file whose first non-UTF-8 byte comes after the sample is read as UTF-8 unless you pass `full_scan: true`, which checks the whole file a megabyte at a time:
+
+```ruby
+CSVUtils::CSVOptions.new('data.csv', full_scan: true).encoding # => "Windows-1252"
+```
+
+Supported column separators: `\x02`, `\t`, `|`, `,`, `;`. The one found most often outside quotes in the header row wins, and a tie goes to the one listed first, so `cost|usd,name,id` is comma separated.
 Supported row separators: `\r\n`, `\n`, `\r`
 
-Headers are parsed as a CSV row, so a quoted header like `"Last, First"` counts as one column. An empty file isn't valid.
+The header row runs to the first line break outside quotes and is parsed as a CSV row, so a quoted header like `"Last, First"` counts as one column. An empty file isn't valid.
 
 ### Encodings and Byte Order Marks
 
-Files are opened with mode `'rb'`. With csv 3.3 and later, when Ruby's default external encoding is UTF-8 (the usual case), CSV reads `'rb'` files as UTF-8 and raises `CSV::InvalidEncodingError` on bytes that aren't valid UTF-8. To read a file's bytes as is, pass an explicit encoding:
+Every class handles encodings the same way:
+
+- **Reading** uses the `encoding:` CSV option, `'bom|utf-8'` by default, so values are UTF-8 strings whatever the locale (including `LANG=C` in containers and cron jobs). Bytes that aren't valid UTF-8 raise `CSV::InvalidEncodingError`.
+- **Writing** (`CSVExtender`, `CSVTransformer`, `CSVSort`) uses the encoding values were decoded to, so a file read with `'Windows-1252:UTF-8'` is written as UTF-8. `CSVReport` writes UTF-8 unless you pass `encoding:`.
+- **File modes never carry an encoding.** CSV raises `ArgumentError: encoding specified twice` when the mode and the `encoding:` option both have one.
 
 ```ruby
-iterator = CSVUtils::CSVIterator.new('latin1.csv', {}, 'rb:BINARY')
+iterator = CSVUtils::CSVIterator.new('export.csv', encoding: 'Windows-1252:UTF-8')
+iterator = CSVUtils::CSVIterator.new('latin1.csv', encoding: 'BINARY') # raw bytes
 comparator = CSVUtils::CSVCompare.new('primary.csv', ['updated_at'], encoding: 'BINARY') { |src, dest| src['id'] <=> dest['id'] }
-sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true, encoding: 'BINARY')
+sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true, CSVUtils::CSVOptions.new('input.csv').to_csv_options)
 ```
 
-`CSVSort`, `CSVExtender` and `CSVTransformer` take CSV options as their last argument and `CSVIterator` takes them second; they're passed to `CSV.open` for file paths.
+`CSVSort`, `CSVExtender`, `CSVTransformer`, `CSVCompare` and `CSVReport` take CSV options as their last argument and `CSVIterator` takes them second; they're passed to `CSV.open` for file paths. `CSVUtils::EncodingOptions` holds the rules above. Upgrading from 0.6? See [UPGRADING.md](UPGRADING.md).
 
 `CSVUtils::ByteOrderMark` detects and strips UTF-8, UTF-16 and UTF-32 byte order marks; `CSVIterator`, `CSVCompare` and `CSVOptions` use it to clean the first header.
 
@@ -360,6 +403,17 @@ csv-change-eol data.csv 7C5E7C0A
 
 `csv-validator` needs the `rchardet` gem (`gem install rchardet`) to guess the encoding of values that aren't UTF-8. It writes the converted values to `utf8-correction.csv`.
 
+`csv-find-error`, `csv-grep`, `csv-diff`, `csv-splitter`, `csv-explorer` and `csv-duplicate-finder` detect each file's separators, and all but `csv-duplicate-finder` its encoding too, with `CSVOptions`. Line numbers are the line a row starts on, so `csv-find-error` hands `csv-readline` the right line even after values with line breaks. `csv-splitter` reads the file once and writes UTF-8 parts. `csv-diff` needs both files to use the same column separator.
+
+## Benchmarks
+
+`script/benchmark.rb` times the iterator, sort and compare on a generated file and counts the objects they allocate. Set `LIB` to another checkout's `lib` to compare versions:
+
+```bash
+ROWS=500000 ruby script/benchmark.rb
+LIB=../csv-utils-0.6/lib ruby script/benchmark.rb
+```
+
 ## Development
 
 After checking out the repo, run `bundle install` to install dependencies. Then:

data/bin/csv-change-eol

@@ -1,26 +1,27 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'csv'
 
 def bold_string(str)
   "\033[1m#{str}\033[0m"
 end
-  
-USAGE = "Usage: #{bold_string('csv-change-eol')} <csv_file> <end of line character sequence in hex>"
+
+USAGE = "Usage: #{bold_string('csv-change-eol')} <csv_file> <end of line character sequence in hex>".freeze
 
 def exit_on_error(msg)
-  $stderr.print <<STR
-Error: #{bold_string(msg)}
+  $stderr.print <<~STR
+    Error: #{bold_string(msg)}
 
-#{USAGE}
+    #{USAGE}
 
-  End of line example: '7C5E7C0A' is '|^|\\n'
-   - 0A is new line
-   - 0D is carriage return
+      End of line example: '7C5E7C0A' is '|^|\\n'
+       - 0A is new line
+       - 0D is carriage return
 
-  Goto: #{bold_string('http://www.asciitable.com/')} for help with the character sequence
+      Goto: #{bold_string('http://www.asciitable.com/')} for help with the character sequence
 
-STR
+  STR
   exit 1
 end
 

data/bin/csv-diff

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'optparse'
 
@@ -15,7 +16,8 @@ OptionParser.new do |opts|
     exit
   end
 
-  opts.on('-u', '--unique HEADERS', 'Comma separated list of headers that generate a unique key per row, uses 1st column by default') do |v|
+  opts.on('-u', '--unique HEADERS',
+          'Comma separated list of headers that generate a unique key per row, uses 1st column by default') do |v|
     options[:unique_headers] = v.split(',')
   end
 
@@ -29,30 +31,38 @@ OptionParser.new do |opts|
 end.parse!
 
 unless ARGV.size == 2
-  $stderr.puts('Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>')
+  warn('Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>')
   exit 1
 end
 
 require 'csv-utils'
 
-csv1 = CSVUtils::CSVIterator.new(ARGV[0])
-csv2 = CSVUtils::CSVIterator.new(ARGV[1])
+csv1_options = CSVUtils::CSVOptions.new(ARGV[0]).to_csv_options
+csv2_options = CSVUtils::CSVOptions.new(ARGV[1]).to_csv_options
+unless csv1_options[:col_sep] == csv2_options[:col_sep]
+  warn("column separators do not match #{ARGV[0]} #{csv1_options[:col_sep].inspect}, " \
+       "#{ARGV[1]} #{csv2_options[:col_sep].inspect}")
+  exit 1
+end
+
+csv1 = CSVUtils::CSVIterator.new(ARGV[0], csv1_options)
+csv2 = CSVUtils::CSVIterator.new(ARGV[1], csv2_options)
 headers = csv1.headers
 
 unless headers == csv2.headers
-  $stderr.puts("headers do not match #{ARGV[0]} headers #{headers}, #{ARGV[1]} headers #{csv2.headers}")
+  warn("headers do not match #{ARGV[0]} headers #{headers}, #{ARGV[1]} headers #{csv2.headers}")
   exit 1
 end
 
 unknown_unique_headers = options[:unique_headers] - headers
 unless unknown_unique_headers.empty?
-  $stderr.puts("specified unique headers are unknown #{unknown_unique_headers}")
+  warn("specified unique headers are unknown #{unknown_unique_headers}")
   exit 1
 end
 
 unknown_ignore_headers = options[:ignore_headers] - headers
 unless unknown_ignore_headers.empty?
-  $stderr.puts("specified headers to ignore are unknown #{unknown_ignore_headers}")
+  warn("specified headers to ignore are unknown #{unknown_ignore_headers}")
   exit 1
 end
 
@@ -62,38 +72,40 @@ puts "uniqueness header(s) are #{options[:unique_headers].join(', ')}"
 
 unique_header_indexes = options[:unique_headers].map { |header| headers.index(header) }
 
-# empty cells are nil, so values are compared as strings
-sort_compare_proc = proc do |csv1_row, csv2_row|
-  result = unique_header_indexes.map { |idx| csv1_row[idx].to_s } <=> unique_header_indexes.map { |idx| csv2_row[idx].to_s }
-  result = csv1_row.map(&:to_s) <=> csv2_row.map(&:to_s) if result.zero?
-  result
+# empty cells are nil, so values are compared as strings; the key is computed once per row
+sort_key_proc = proc do |row|
+  [unique_header_indexes.map { |idx| row[idx].to_s }, row.map(&:to_s)]
 end
 
 csv1_sorted_file_name = ARGV[0] + '.sorted'
 csv2_sorted_file_name = ARGV[1] + '.sorted'
 
 puts "sorting #{ARGV[0]}"
-sorter = CSVUtils::CSVSort.new(ARGV[0], csv1_sorted_file_name)
-sorter.sort(options[:sort_batch_size], &sort_compare_proc)
+sorter = CSVUtils::CSVSort.new(ARGV[0], csv1_sorted_file_name, true, csv1_options)
+sorter.sort_by(options[:sort_batch_size], &sort_key_proc)
 
 puts "sorting #{ARGV[1]}"
-sorter = CSVUtils::CSVSort.new(ARGV[1], csv2_sorted_file_name)
-sorter.sort(options[:sort_batch_size], &sort_compare_proc)
+sorter = CSVUtils::CSVSort.new(ARGV[1], csv2_sorted_file_name, true, csv2_options)
+sorter.sort_by(options[:sort_batch_size], &sort_key_proc)
 
 update_comparison_columns = headers - options[:unique_headers] - options[:ignore_headers]
 
-comparer = CSVUtils::CSVCompare.new(csv1_sorted_file_name, update_comparison_columns) do |csv1_row, csv2_row|
-  options[:unique_headers].map { |header| csv1_row[header].to_s } <=> options[:unique_headers].map { |header| csv2_row[header].to_s }
+# the sorted files are written in the decoded encoding, with the same separators
+sorted_options = CSVUtils::EncodingOptions.write(csv1_options)
+comparer = CSVUtils::CSVCompare.new(csv1_sorted_file_name, update_comparison_columns, sorted_options) do |csv1_row, csv2_row|
+  options[:unique_headers].map { |header| csv1_row[header].to_s } <=> options[:unique_headers].map do |header|
+    csv2_row[header].to_s
+  end
 end
 
 stats = Hash.new(0)
 puts "comparing #{ARGV[0]} with #{ARGV[1]}"
 diff_file_name = 'diff-results-' + File.basename(ARGV[0])
 CSV.open(diff_file_name, 'wb') do |out|
-  out << ['Result'] + headers
+  out << (['Result'] + headers)
   comparer.compare(csv2_sorted_file_name) do |action, record|
     stats[action] += 1
-    out << [action] + record.values
+    out << ([action] + record.values)
   end
 end
 puts "differences found #{stats}"
@@ -102,7 +114,7 @@ File.unlink(csv1_sorted_file_name)
 File.unlink(csv2_sorted_file_name)
 
 if stats.empty?
-  puts "files were identical"
+  puts 'files were identical'
   File.unlink(diff_file_name)
 else
   puts "results can be found in #{diff_file_name}"

data/bin/csv-duplicate-finder

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'optparse'
 
@@ -23,13 +24,11 @@ require 'csv-utils'
 
 file = ARGV[0] || raise('no CSV file specified')
 
-# read bytes as is, so values that aren't valid UTF-8 can still be compared
-csv = CSVUtils::CSVIterator.new(file, {}, 'rb:BINARY')
+# detect the separators, but read bytes as is, so values that aren't valid UTF-8 can still be compared
+csv = CSVUtils::CSVIterator.new(file, CSVUtils::CSVOptions.new(file).to_csv_options.merge(encoding: 'BINARY'))
 
 missing_headers = options[:ignore_columns] - csv.headers
-unless missing_headers.empty?
-  raise("unknown headers #{missing_headers.join(', ')} in configured ignore headers")
-end
+raise("unknown headers #{missing_headers.join(', ')} in configured ignore headers") unless missing_headers.empty?
 
 hashed_rows = {}
 
@@ -45,7 +44,7 @@ end
 
 duplicate_rows = {}
 
-hashed_rows.reject! { |key, row_numbers| row_numbers.size <= 1 }
+hashed_rows.reject! { |_key, row_numbers| row_numbers.size <= 1 }
 
 hashed_rows.each do |key, row_numbers|
   hashed_rows[key] = {}
@@ -62,11 +61,11 @@ csv.each do |row|
 end
 
 CSV.open('duplicates-' + File.basename(file), 'wb:BINARY') do |out|
-  out << ['duplicate_key', 'line_no'] + csv.headers
+  out << (%w[duplicate_key line_no] + csv.headers)
 
   hashed_rows.each do |key, rows|
     rows.each do |lineno, row|
-      out << [key, lineno] + row.values
+      out << ([key, lineno] + row.values)
     end
   end
 end

data/bin/csv-explorer

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'csv-utils'
 require 'irb'
@@ -6,7 +7,7 @@ require 'irb'
 CSV_FILE = ARGV.shift
 
 def csv
-  @csv ||= CSVUtils::CSVIterator.new(CSV_FILE)
+  @csv ||= CSVUtils::CSVIterator.auto_detect(CSV_FILE)
 end
 
 IRB.start

data/bin/csv-find-error

@@ -1,22 +1,23 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
-require 'csv'
+require 'csv-utils'
 require 'rbconfig'
 
 file = ARGV[0] || abort('Usage: csv-find-error <csv file>')
 
-prev_row = nil
+iterator = CSVUtils::CSVIterator.auto_detect(file)
 begin
-  CSV.open(file, 'rb') { |csv| csv.each { |row| prev_row = row } }
+  iterator.each { |_row| nil }
 rescue CSV::MalformedCSVError => e
-  puts e.class.to_s + ': ' + e.message
-  puts "previous row was #{prev_row}"
-  if e.message =~ /line (\d+)/
-    lineno = $1.to_i
-    # run the csv-readline next to this script, with this gem's lib, rather than whichever is on the PATH
-    puts "running csv-readline #{file} #{lineno}"
-    system(RbConfig.ruby, '-I', File.expand_path('../lib', __dir__), File.join(__dir__, 'csv-readline'), file, lineno.to_s)
-  end
+  puts "#{e.class}: #{e.message}"
+  # CSV raises encoding errors ahead of the rows before them, so only a malformed row has a reliable previous row
+  puts "previous row was #{e.prev_row}" if e.is_a?(CSVUtils::MalformedRowError)
+  # line_number is the physical line the bad row starts on, which is what csv-readline reads
+  puts "running csv-readline #{file} #{e.line_number}"
+  # run the csv-readline next to this script, with this gem's lib, rather than whichever is on the PATH
+  system(RbConfig.ruby, '-I', File.expand_path('../lib', __dir__), File.join(__dir__, 'csv-readline'), file,
+         e.line_number.to_s)
   exit 1
 end
 

data/bin/csv-grep

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'optparse'
 
@@ -43,7 +44,7 @@ raise('no search specified') unless options[:search]
 
 require 'csv-utils'
 
-csv = CSVUtils::CSVIterator.new(ARGV[0])
+csv = CSVUtils::CSVIterator.auto_detect(ARGV[0])
 
 search_regex =
   if options[:exact_match]

data/bin/csv-readline

@@ -1,4 +1,5 @@
 #!/usr/bin/env ruby
+# frozen_string_literal: true
 
 require 'optparse'
 
@@ -8,62 +9,65 @@ def strip_byte_order_mark(line)
   CSVUtils::ByteOrderMark.strip(line)
 end
 
-def csv_parse_line(line)
-  columns = []
-
-  opened_quote = false
-  pos = -1
-  last_comma_pos = -1
-  column = 1
-
-  while pos = line.index(/([",\n])/, pos + 1)
-    case line[pos]
-    when '"'
-      if opened_quote
-        if line[pos+1] == ',' ||
-           line[pos+1] == "\r" ||
-           line[pos+1] == "\n" ||
-           line[pos+1].nil?
-          opened_quote = false
-        elsif line[pos+1] == '"'
-          pos += 1 # correctly escaped
-        else
-          col_end = line.index(/,/, pos + 1)
-          col_end ||= line.size - 1
-          # slice out the column value
-          val = line[last_comma_pos + 1, col_end - last_comma_pos - 1]
-          columns << [val, :stray_quote]
-          opened_quote = false
-          last_comma_pos = col_end
-          pos = col_end
-          # return columns
-        end
-      else
-        if (last_comma_pos == -1 && pos != 0) ||
-           (last_comma_pos != -1 && line[pos-1] != ',')
-          col_end = line.index(/,/, pos + 1)
-          col_end ||= line.size - 1
-          # slice out the column value
-          val = line[last_comma_pos + 1, col_end - last_comma_pos - 1]
-          columns << [val, :stray_quote]
-          opened_quote = false
-          last_comma_pos = col_end
-          pos = col_end
-        else
-          opened_quote = true
-        end
-      end
-    when ',',
-         "\n"
-      if ! opened_quote
-        column += 1
-        columns << [line[last_comma_pos + 1, pos - last_comma_pos - 1], :ok]
-        last_comma_pos = pos
-      end
+# Splits a raw line on commas, flagging columns with a quote CSV would reject, so a malformed line can be read.
+class LineParser
+  def initialize(line)
+    @line = line
+    @columns = []
+    @opened_quote = false
+    @pos = -1
+    @last_comma_pos = -1
+  end
+
+  # @return [Array<Array(String, Symbol)>] each column's raw value and :ok or :stray_quote
+  def parse
+    while (@pos = @line.index(/[",\n]/, @pos + 1))
+      @line[@pos] == '"' ? quote : separator
+    end
+    @columns
+  end
+
+  private
+
+  def quote
+    if @opened_quote
+      closing_quote
+    elsif column_start?
+      @opened_quote = true
+    else
+      stray_quote
+    end
+  end
+
+  def closing_quote
+    following = @line[@pos + 1]
+    if following.nil? || [',', "\r", "\n"].include?(following)
+      @opened_quote = false
+    elsif following == '"'
+      @pos += 1 # correctly escaped
+    else
+      stray_quote
     end
   end
 
-  columns
+  def column_start?
+    @last_comma_pos == -1 ? @pos.zero? : @line[@pos - 1] == ','
+  end
+
+  # the column runs to the next comma, quotes and all
+  def stray_quote
+    col_end = @line.index(',', @pos + 1) || (@line.size - 1)
+    @columns << [@line[@last_comma_pos + 1, col_end - @last_comma_pos - 1], :stray_quote]
+    @opened_quote = false
+    @last_comma_pos = @pos = col_end
+  end
+
+  def separator
+    return if @opened_quote
+
+    @columns << [@line[@last_comma_pos + 1, @pos - @last_comma_pos - 1], :ok]
+    @last_comma_pos = @pos
+  end
 end
 
 def empty_column?(col)
@@ -80,11 +84,12 @@ def parse_csv_row(file, lineno, number_of_lines)
   file.each do |line|
     cnt += 1
     next if cnt < lineno
+
     str << line
     number_of_lines -= 1
-    break if number_of_lines == 0
+    break if number_of_lines.zero?
   end
-  csv_parse_line(str)
+  LineParser.new(str).parse
 end
 
 options = {
@@ -106,7 +111,7 @@ end.parse!
 lineno = ARGV[1].to_i
 number_of_lines = (ARGV[2] || 1).to_i
 
-raise "no lineno specified" unless lineno > 0
+raise 'no lineno specified' unless lineno.positive?
 
 data = File.open(ARGV[0], 'rb') do |file|
   headers = strip_byte_order_mark(file.readline.strip).split(',')
@@ -117,9 +122,10 @@ cnt = 0
 data.each do |k, (v, status)|
   cnt += 1
   next if !options[:all_columns] && empty_column?(v)
+
   if status == :ok
-    puts sprintf('  %-3d %s: %s', cnt, k, v)
+    puts format('  %-3d %s: %s', cnt, k, v)
   else
-    puts sprintf('* %-3d (%s) %s: %s', cnt, status.to_s.gsub('_', ' '), k, v)
+    puts format('* %-3d (%s) %s: %s', cnt, status.to_s.gsub('_', ' '), k, v)
   end
 end