csv-utils 0.3.25 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9b30f6d77e44b6d4abca4ee199142b4efa6cb73895df87ba630b82f50b964fbc
-  data.tar.gz: 10337ebbb358c3e60f5f4d47756297577468b45bbaeaf4e1b0b355b12c9083a4
+  metadata.gz: 423eacb18512c595abe83765ccfd4d945f1e2e0d18a6370b6841997eb428ddb7
+  data.tar.gz: 3d2df7f1ee62eaebc25d00281401bf49f0fccdb809ec03cbac90e6bf5ba2fd62
 SHA512:
-  metadata.gz: 84432b88f8fc0aee4422fe46a36bab6e99cca1d604469608e2e4ced97c1f748be66796bc7f0c161b660d31e6424f5b155b38bfb034e9cee234fd8c71459bb2d3
-  data.tar.gz: ad864bb734d1c9f7d6a99d97aa2567811c66f8bf317571a947d56196776c61abf229d3d3aad0f5485f1a39b64d257e5be234ffa19dcc9c8a975a045fffcd95b1
+  metadata.gz: 2f9856120036ae6a87e2e758186221b7c0b48aa8d94365f7bdf4bffdf8f292868e43cf3c378eff7460902d710de2859d5406ac9ef5384c07f9728f2f1bad4ef4
+  data.tar.gz: 9e96e672cb1f4da44ae67de48d3e783cf1d2b180c0d8f4a9706ce1ace07b9ac456a689653dfe84537c0cab71e6c58cdb24e5caeec0bcc86bc73a374615ac4e0a

data/CHANGELOG.md

@@ -0,0 +1,37 @@
+# Changelog
+
+## [0.6.0](https://github.com/dougyouch/csv-utils/compare/v0.5.0...v0.6.0) (2026-10-05)
+
+
+### ⚠ BREAKING CHANGES
+
+* **gem:** Ruby 3.2 reached end of life in March 2026 and is no longer supported. The gemspec now declares required_ruby_version >= 3.3.
+
+### Features
+
+* **compare:** accept csv options for both files ([4336edf](https://github.com/dougyouch/csv-utils/commit/4336edf8ca928ebc73c7b58a533ee7280267907e))
+
+
+### Bug Fixes
+
+* **bom:** detect utf-32 le and strip marks from text in any encoding ([7c68517](https://github.com/dougyouch/csv-utils/commit/7c6851755b9ddfb6ca49b459d9ada578c35a634d))
+* **cli:** keep trailing whitespace and raw bytes in csv-change-eol ([a61e53a](https://github.com/dougyouch/csv-utils/commit/a61e53a3add068ebc49c1f6025f06185e8b8a702))
+* **cli:** match csv-diff rows on the --unique headers and allow empty cells ([3e0d1ae](https://github.com/dougyouch/csv-utils/commit/3e0d1ae3763090ade46068e056ee909711a2d4df))
+* **cli:** read raw bytes in csv-validator ([80c5002](https://github.com/dougyouch/csv-utils/commit/80c5002eb91474d58d7999f41385743c4f719c78))
+* **cli:** report line numbers and handle any bytes in csv-duplicate-finder ([f87144d](https://github.com/dougyouch/csv-utils/commit/f87144d07727c9a418c0a37a6e82ceedf55ecf59))
+* **cli:** run the bundled csv-readline from csv-find-error ([e6736c6](https://github.com/dougyouch/csv-utils/commit/e6736c6b2682b7611079a1bb5d10ea5df85f3f92))
+* **cli:** split single column files with csv-splitter ([ee00da6](https://github.com/dougyouch/csv-utils/commit/ee00da6f9128a744345583541850348490cabdfc))
+* **compare:** yield the last record when the other file ends first ([947d2b7](https://github.com/dougyouch/csv-utils/commit/947d2b73d4aa404c52b147782f5bc3c02f5afd42))
+* **io:** close files when a block raises ([5ffa159](https://github.com/dougyouch/csv-utils/commit/5ffa159255e6a134721075536f7b70de9f7c1244))
+* **iterator:** return an enumerator from each without a block ([751d034](https://github.com/dougyouch/csv-utils/commit/751d034d325279f4eaeca76b726e4a63a6443042))
+* **options:** count quoted headers correctly and handle empty files ([abbd084](https://github.com/dougyouch/csv-utils/commit/abbd084a4a2cf84bf27410d616c63e3679f07a14))
+* **row:** stop csv_column from changing the options hash passed in ([6a9f916](https://github.com/dougyouch/csv-utils/commit/6a9f916d1e3c9b0f27410a4cae795f80e5968cec))
+* **sort:** sort without a block and clean up temporary files on failure ([bc2d21a](https://github.com/dougyouch/csv-utils/commit/bc2d21ae63973323d12084de2710d07e06e70dbb))
+* **transformer:** allow append when headers were not read ([9d1e042](https://github.com/dougyouch/csv-utils/commit/9d1e04259e5819eaf5865895674b8aafc8d51c8f))
+
+
+### Build System
+
+* **gem:** require ruby 3.3 and add gem metadata ([e34a89a](https://github.com/dougyouch/csv-utils/commit/e34a89a473159fa4c70f5279d63ca95499a43c90))
+
+## Changelog

data/README.md

@@ -1,20 +1,30 @@
 # CSV Utils
 
-A Ruby library providing a comprehensive set of utilities for manipulating and processing CSV files. This library offers a robust set of tools for comparing, transforming, sorting, and managing CSV data efficiently.
+Streaming tools for CSV files that are too big or too broken to load into memory. Compare two sorted files into create, update and delete actions, sort with an external merge sort, transform and extend rows in batches, and build reports from Ruby objects. Command line tools pinpoint malformed rows and diff, grep, split and validate CSV files.
+
+[![CI](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+[![Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/coverage.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+[![Branch Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/branches.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+
+[API reference](https://rubydoc.info/gems/csv-utils) · [Changelog](CHANGELOG.md) · [Architecture](ARCHITECTURE.md)
 
 ## Features
 
 - **CSV Comparison**: Compare two CSV files and identify differences (creates, updates, and deletes)
-- **CSV Transformation**: Transform CSV data with customizable rules
-- **CSV Sorting**: Sort CSV files based on specified columns
-- **CSV Reporting**: Generate reports from CSV data
-- **CSV Iteration**: Efficient iteration over CSV files
-- **CSV Extension**: Extend CSV files with additional data
-- **CSV Wrapper**: Convenient wrapper for CSV operations
+- **CSV Transformation**: Transform CSV data with a chainable pipeline
+- **CSV Sorting**: Sort large CSV files using external merge sort
+- **CSV Reporting**: Generate CSV reports from Ruby objects
+- **CSV Row**: Mixin for defining CSV-serializable objects
+- **CSV Row Matcher**: Filter CSV rows using regex patterns across columns
+- **CSV Iteration**: Efficient iteration over CSV files with batch support
+- **CSV Extension**: Extend CSV files with additional columns
+- **CSV Options**: Auto-detect CSV file properties (separators, encoding, BOM)
+- **Byte Order Marks**: UTF-8, UTF-16 and UTF-32 byte order marks are stripped from the first header
+- **CLI Tools**: Command-line utilities for CSV debugging and manipulation
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Requires Ruby 3.3 or newer. Add this line to your application's Gemfile:
 
 ```ruby
 gem 'csv-utils'
@@ -36,8 +46,10 @@ $ gem install csv-utils
 
 ### Comparing CSV Files
 
+Compare two sorted CSV files to identify creates, updates, and deletes:
+
 ```ruby
-require 'csv_utils'
+require 'csv-utils'
 
 comparator = CSVUtils::CSVCompare.new('primary.csv', ['updated_at']) do |src, dest|
   src['id'] <=> dest['id']
@@ -55,32 +67,317 @@ comparator.compare('secondary.csv') do |action, record|
 end
 ```
 
+Pass CSV options as the third argument to read both files with them, e.g. tab separated files:
+
+```ruby
+CSVUtils::CSVCompare.new('primary.tsv', ['updated_at'], col_sep: "\t") { |src, dest| src['id'] <=> dest['id'] }
+```
+
+**Note**: Both CSV files must be sorted by the same key columns for comparison to work correctly, and the block must compare them in that same order.
+
+The block compares the key columns of a primary and a secondary record like `<=>`. Records only in the primary file are yielded as `:create`, records only in the secondary file as `:delete`, and matching records as `:update` when any of the update comparison columns differ. Without update comparison columns, no updates are yielded.
+
 ### Sorting CSV Files
 
+Sort large CSV files using external merge sort:
+
 ```ruby
-sorter = CSVUtils::CSVSort.new('input.csv')
-sorter.sort('output.csv', ['id', 'name'])
+require 'csv-utils'
+
+sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true)  # true = has headers
+sorter.sort(100_000) { |a, b| a.first.to_i <=> b.first.to_i }    # batch size, comparison block
+
+# without a block, rows are compared as arrays of strings
+sorter.sort
 ```
 
+Batches are sorted into temporary `output.csv.part.N` files next to the output, which are merged in pairs. The temporary files are removed when the sort finishes or fails.
+
 ### Transforming CSV Data
 
+Transform CSV data using a chainable pipeline:
+
+```ruby
+require 'csv-utils'
+
+CSVUtils::CSVTransformer.new('input.csv', 'output.csv')
+  .read_headers
+  .select { |row, headers, _| row[0].to_i > 100 }                    # filter rows
+  .map(['ID', 'Name']) { |row, headers, _| [row[0], row[1].upcase] } # transform rows
+  .append(['Email']) { |row, headers, _| ["#{row[1].downcase}@example.com"] }
+  .process(10_000)  # batch size
+```
+
+Available pipeline methods:
+- `select { |row, headers, additional_data| }` - Keep rows where block returns true
+- `reject { |row, headers, additional_data| }` - Remove rows where block returns true
+- `map(new_headers) { |row, headers, additional_data| }` - Transform rows
+- `append(additional_headers) { |row, headers, additional_data| }` - Add columns
+- `additional_data { |batch, headers| }` - Compute batch-level data for use in other steps
+- `each { |row, headers, additional_data| }` - Side effects without modification
+- `set_headers(headers)` - Override output headers
+
+Without `read_headers`, every row is treated as data and the output has no header row. `process` closes the files it opened, even when a step raises.
+
+### CSV Row and Report
+
+Define CSV-serializable objects and generate reports:
+
+```ruby
+require 'csv-utils'
+
+class User
+  include CSVUtils::CSVRow
+
+  attr_accessor :id, :name, :email
+
+  csv_column :id, header: 'ID'
+  csv_column :name
+  csv_column(:email) { email.downcase }
+
+  def initialize(id, name, email)
+    @id = id
+    @name = name
+    @email = email
+  end
+end
+
+users = [
+  User.new(1, 'Alice', 'ALICE@example.com'),
+  User.new(2, 'Bob', 'BOB@example.com')
+]
+
+# Generate CSV report
+CSVUtils::CSVReport.new('users.csv', User) do |report|
+  users.each { |user| report << user }
+end
+```
+
+The `csv_column` method accepts:
+- A symbol referencing a method: `csv_column :name`
+- A custom header: `csv_column :id, header: 'ID'`
+- A block for computed values: `csv_column(:email) { email.downcase }`
+- A proc: `csv_column :count, proc: Proc.new { data[:count] }`
+
+#### Generating Reports from ActiveRecord Models
+
+A powerful pattern is to subclass an ActiveRecord model with `CSVRow` for generating reports directly from database records:
+
 ```ruby
-transformer = CSVUtils::CSVTransformer.new('input.csv')
-transformer.transform('output.csv') do |row|
-  # Transform row data
-  row['new_column'] = row['old_column'].upcase
-  row
+require 'csv-utils'
+
+class UserCSVRow < User
+  include CSVUtils::CSVRow
+
+  csv_column :id
+  csv_column :name
+  csv_column :email
+  csv_column :num_orders      # computed column
+  csv_column :total_revenue   # computed column
+
+  def num_orders
+    orders.count
+  end
+
+  def total_revenue
+    orders.sum(:amount)
+  end
+
+  # free up memory during large iterations
+  def clear!
+    @association_cache = {}
+  end
+end
+
+# Generate report using ActiveRecord scopes
+CSVUtils::CSVReport.new('user_report.csv', UserCSVRow) do |report|
+  UserCSVRow.where(active: true).find_each do |user|
+    report << user
+    user.clear!
+  end
 end
 ```
 
+This pattern provides:
+- **Inherited attributes**: All model columns available without redefinition
+- **Association access**: Query related tables for computed columns
+- **ActiveRecord scopes**: Use `.where`, `.includes`, `.find_each` directly
+- **Memory efficiency**: The `clear!` method frees association cache during iteration
+
+### Iterating CSV Files
+
+Efficiently iterate over CSV files:
+
+```ruby
+require 'csv-utils'
+
+iterator = CSVUtils::CSVIterator.new('data.csv')
+
+# Iterate row by row
+iterator.each do |row|
+  puts "Line #{row.lineno}: #{row['name']}"
+end
+
+# Process in batches
+iterator.each_batch(1_000) do |batch|
+  # Process batch of rows
+end
+
+# Build a lookup hash
+lookup = iterator.to_hash('id', 'name')  # { 'id_value' => 'name_value', ... }
+
+# Without a block, each returns an Enumerator
+iterator.each.with_index { |row, idx| puts "#{idx}: #{row['name']}" }
+```
+
+Line numbers count the header row as line 1, so they match what an editor shows. `headers` returns `[]` for an empty file.
+
+### Matching CSV Rows
+
+Filter CSV rows using regex patterns:
+
+```ruby
+require 'csv-utils'
+
+# Match against all columns
+matcher = CSVUtils::CSVRowMatcher.new(/error/i)
+
+# Or match only specific columns
+matcher = CSVUtils::CSVRowMatcher.new(/error/i, ['status', 'message'])
+
+# Use with iteration
+iterator = CSVUtils::CSVIterator.new('logs.csv')
+error_rows = iterator.select(&matcher)
+
+# Use directly
+row = { 'id' => '123', 'status' => 'Error', 'message' => 'Connection failed' }
+matcher.match?(row)  # => true
+```
+
+The matcher can be used with any Enumerable method via `to_proc`:
+
+```ruby
+rows.select(&matcher)  # rows matching the pattern
+rows.reject(&matcher)  # rows not matching the pattern
+rows.find(&matcher)    # first matching row
+```
+
+### Extending CSV Files
+
+Add columns to an existing CSV:
+
+```ruby
+require 'csv-utils'
+
+extender = CSVUtils::CSVExtender.new('input.csv', 'output.csv')
+
+# Row by row
+extender.append(['new_column']) do |row, headers|
+  [row[0].upcase]  # return array of new column values
+end
+
+# Or in batches (useful for external lookups)
+extender.append_in_batches(['status'], 1_000) do |batch, headers|
+  # Return array of arrays, one per row in batch
+  batch.map { |row| ['active'] }
+end
+```
+
+### Auto-detecting CSV Options
+
+Detect CSV file properties automatically:
+
+```ruby
+require 'csv-utils'
+
+options = CSVUtils::CSVOptions.new('data.csv')
+
+options.valid?         # true if separators detected
+options.col_separator  # detected column separator
+options.row_separator  # detected row separator
+options.encoding       # detected encoding (UTF-8, UTF-16, UTF-32)
+options.columns        # number of columns
+options.byte_order_mark # BOM if present
+```
+
+Supported column separators: `\x02`, `\t`, `|`, `,` (the first one found in the header line wins)
+Supported row separators: `\r\n`, `\n`, `\r`
+
+Headers are parsed as a CSV row, so a quoted header like `"Last, First"` counts as one column. An empty file isn't valid.
+
+### Encodings and Byte Order Marks
+
+Files are opened with mode `'rb'`. With csv 3.3 and later, when Ruby's default external encoding is UTF-8 (the usual case), CSV reads `'rb'` files as UTF-8 and raises `CSV::InvalidEncodingError` on bytes that aren't valid UTF-8. To read a file's bytes as is, pass an explicit encoding:
+
+```ruby
+iterator = CSVUtils::CSVIterator.new('latin1.csv', {}, 'rb:BINARY')
+comparator = CSVUtils::CSVCompare.new('primary.csv', ['updated_at'], encoding: 'BINARY') { |src, dest| src['id'] <=> dest['id'] }
+sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true, encoding: 'BINARY')
+```
+
+`CSVSort`, `CSVExtender` and `CSVTransformer` take CSV options as their last argument and `CSVIterator` takes them second; they're passed to `CSV.open` for file paths.
+
+`CSVUtils::ByteOrderMark` detects and strips UTF-8, UTF-16 and UTF-32 byte order marks; `CSVIterator`, `CSVCompare` and `CSVOptions` use it to clean the first header.
+
+## CLI Tools
+
+The gem installs command-line utilities for CSV debugging:
+
+| Command | Description |
+|---------|-------------|
+| `csv-find-error` | Locate the first malformed row and show it with `csv-readline` |
+| `csv-readline` | Print the columns of a line, flagging stray quotes |
+| `csv-validator` | Report rows with the wrong number of columns and values that aren't UTF-8 |
+| `csv-diff` | Compare two CSV files by a unique key |
+| `csv-grep` | Search columns for a pattern |
+| `csv-splitter` | Split a large CSV file into parts |
+| `csv-explorer` | Open an IRB session with the file loaded as a `CSVIterator` |
+| `csv-duplicate-finder` | Find duplicate rows |
+| `csv-change-eol` | Rewrite a file with a different line ending |
+
+```bash
+# find the first malformed row
+csv-find-error data.csv
+
+# print the row on line 1042, read as 3 lines for values with embedded newlines, including empty columns
+csv-readline --all data.csv 1042 3
+
+# compare by the id column, ignoring updated_at; writes diff-results-old.csv
+csv-diff -u id -i updated_at old.csv new.csv
+
+# case-insensitive search of the email and name columns, first 10 matches
+csv-grep -s 'smith' -c email,name -i -l 10 data.csv
+
+# split into files of 100,000 rows, each with the header
+csv-splitter -r 100000 data.csv
+
+# find rows that are duplicates apart from id; writes duplicates-data.csv
+csv-duplicate-finder -i id data.csv
+
+# end every row with |^| and a newline
+csv-change-eol data.csv 7C5E7C0A
+```
+
+`csv-validator` needs the `rchardet` gem (`gem install rchardet`) to guess the encoding of values that aren't UTF-8. It writes the converted values to `utf8-correction.csv`.
+
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests.
+After checking out the repo, run `bundle install` to install dependencies. Then:
+
+```bash
+bundle exec rspec       # run the tests
+bundle exec rubocop     # run the linter
+bundle exec yard stats --list-undoc   # check the API docs
+```
+
+CI runs RuboCop, requires every public class, module, constant and method to have a YARD doc comment, and runs the specs on Ruby 3.3 and the `.ruby-version` Ruby with 100% line and branch coverage (`CI=1 bundle exec rspec` enforces it locally).
+
+Releases are automated by [release-please](https://github.com/googleapis/release-please) from [conventional commits](https://www.conventionalcommits.org/): merging its release PR tags the version and publishes the gem.
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/yourusername/csv-utils.
+Bug reports and pull requests are welcome on GitHub at https://github.com/dougyouch/csv-utils.
 
 ## License
 
-The gem is available as open source under the terms of the MIT License.
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/bin/csv-change-eol

@@ -28,12 +28,11 @@ csv_file = ARGV.shift || exit_on_error('no csv file specified')
 eol_sequence = ARGV.shift || exit_on_error('no EOL character sequence specified')
 
 exit_on_error("file #{csv_file} not found") unless File.exist?(csv_file)
-exit_on_error("not a HEX sequece (#{eol_sequence})") unless eol_sequence =~ /\A[0-9a-f]+\z/i
+exit_on_error("not a HEX sequence (#{eol_sequence})") unless eol_sequence =~ /\A[0-9a-f]+\z/i
 exit_on_error("incorrect number of characters in (#{eol_sequence}), should be even") unless eol_sequence.size.even?
 
 eol_sequence = [eol_sequence].pack('H*')
 
-
 escaped_csv_file =
   if csv_file =~ /\.csv$/i
     csv_file.sub(/(\.csv)$/i, '.escaped-eol\1')
@@ -41,13 +40,10 @@ escaped_csv_file =
     csv_file + '.escaped-eol'
   end
 
-
+# read and write bytes as is, so values in any encoding pass through unchanged
 File.open(escaped_csv_file, 'wb') do |out|
-  CSV.foreach(csv_file) do |row|
-    line = row.to_csv
-    line.rstrip!
-    line.concat(eol_sequence)
-    out.write line
+  CSV.foreach(csv_file, 'rb:BINARY') do |row|
+    out.write row.to_csv(row_sep: eol_sequence)
   end
 end
 

data/bin/csv-diff

@@ -8,72 +8,64 @@ options = {
   sort_batch_size: 1_000_000
 }
 OptionParser.new do |opts|
-  opts.banner = 'Usage: ' + File.basename(__FILE__) + ' [options] <csv file>'
+  opts.banner = 'Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>'
 
   opts.on('-h', '--help', 'Prints this help') do
     puts opts
     exit
   end
 
-  opts.on('-u', '--unique HEADERS', 'Comman separated list of headers that genrate a unique key per a row, use 1st column by default') do |v|
+  opts.on('-u', '--unique HEADERS', 'Comma separated list of headers that generate a unique key per row, uses 1st column by default') do |v|
     options[:unique_headers] = v.split(',')
   end
 
-  opts.on('-i', '--ignore HEADERS', 'Comman separated list of headers to ignore during row comparison') do |v|
+  opts.on('-i', '--ignore HEADERS', 'Comma separated list of headers to ignore during row comparison') do |v|
     options[:ignore_headers] = v.split(',')
   end
 
   opts.on('--sort-batch-size SIZE', Integer, 'Number of rows to load into memory while sorting') do |v|
-    opts[:sort_batch_size] = v
+    options[:sort_batch_size] = v
   end
 end.parse!
 
+unless ARGV.size == 2
+  $stderr.puts('Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>')
+  exit 1
+end
+
 require 'csv-utils'
 
 csv1 = CSVUtils::CSVIterator.new(ARGV[0])
 csv2 = CSVUtils::CSVIterator.new(ARGV[1])
+headers = csv1.headers
 
-unless csv1.first.keys == csv2.first.keys
-  $stderr.puts("headers do not match #{ARGV[0]} headers #{csv1.first.keys}, #{ARGV[1]} headers #{csv2.first.keys}")
+unless headers == csv2.headers
+  $stderr.puts("headers do not match #{ARGV[0]} headers #{headers}, #{ARGV[1]} headers #{csv2.headers}")
   exit 1
 end
 
-unknown_unique_headers = options[:unique_headers] - csv1.first.keys
+unknown_unique_headers = options[:unique_headers] - headers
 unless unknown_unique_headers.empty?
   $stderr.puts("specified unique headers are unknown #{unknown_unique_headers}")
   exit 1
 end
 
-unknown_ignore_headers = options[:ignore_headers] - csv1.first.keys
+unknown_ignore_headers = options[:ignore_headers] - headers
 unless unknown_ignore_headers.empty?
   $stderr.puts("specified headers to ignore are unknown #{unknown_ignore_headers}")
   exit 1
 end
 
-options[:unique_headers] = [csv1.first.keys.first] if options[:unique_headers].empty?
+options[:unique_headers] = [headers.first] if options[:unique_headers].empty?
 
 puts "uniqueness header(s) are #{options[:unique_headers].join(', ')}"
 
-unique_header_indexes = []
-csv1.first.keys.each_with_index do |header, idx|
-  unique_header_indexes << idx if options[:unique_headers].include?(header)
-end
+unique_header_indexes = options[:unique_headers].map { |header| headers.index(header) }
 
+# empty cells are nil, so values are compared as strings
 sort_compare_proc = proc do |csv1_row, csv2_row|
-  result = 0
-  unique_header_indexes.each do |idx|
-    result = csv1_row[idx] <=> csv2_row[idx]
-    break unless result == 0
-  end
-
-  if result == 0
-    csv1_row.each_with_index do |csv1_col, idx|
-      csv2_col = csv2_row[idx]
-      result = csv1_col <=> csv2_col
-      break unless result == 0
-    end
-  end
-
+  result = unique_header_indexes.map { |idx| csv1_row[idx].to_s } <=> unique_header_indexes.map { |idx| csv2_row[idx].to_s }
+  result = csv1_row.map(&:to_s) <=> csv2_row.map(&:to_s) if result.zero?
   result
 end
 
@@ -88,25 +80,17 @@ puts "sorting #{ARGV[1]}"
 sorter = CSVUtils::CSVSort.new(ARGV[1], csv2_sorted_file_name)
 sorter.sort(options[:sort_batch_size], &sort_compare_proc)
 
-options[:unique_headers] = [csv1.first.keys.first]
-update_comparison_columns = csv1.first.keys - options[:unique_headers]
-update_comparison_columns -= options[:ignore_headers]
+update_comparison_columns = headers - options[:unique_headers] - options[:ignore_headers]
 
 comparer = CSVUtils::CSVCompare.new(csv1_sorted_file_name, update_comparison_columns) do |csv1_row, csv2_row|
-  result = 0
-  options[:unique_headers].each do |header|
-    result = csv1_row[header] <=> csv2_row[header]
-    break unless result == 0
-  end
-
-  result
+  options[:unique_headers].map { |header| csv1_row[header].to_s } <=> options[:unique_headers].map { |header| csv2_row[header].to_s }
 end
 
 stats = Hash.new(0)
 puts "comparing #{ARGV[0]} with #{ARGV[1]}"
 diff_file_name = 'diff-results-' + File.basename(ARGV[0])
 CSV.open(diff_file_name, 'wb') do |out|
-  out << ['Result'] + csv1.first.keys
+  out << ['Result'] + headers
   comparer.compare(csv2_sorted_file_name) do |action, record|
     stats[action] += 1
     out << [action] + record.values

data/bin/csv-duplicate-finder

@@ -13,31 +13,34 @@ OptionParser.new do |opts|
     exit
   end
 
-  opts.on('-i', '--ignore HEADERS', 'Comman separated list of headers to ignore') do |v|
+  opts.on('-i', '--ignore HEADERS', 'Comma separated list of headers to ignore') do |v|
     options[:ignore_columns] = v.split(',')
   end
 end.parse!
 
 require 'digest/sha2'
-require 'json'
 require 'csv-utils'
 
-csv = CSVUtils::CSVIterator.new(ARGV[0])
+file = ARGV[0] || raise('no CSV file specified')
 
-missing_headers = options[:ignore_columns] - csv.first.keys
+# read bytes as is, so values that aren't valid UTF-8 can still be compared
+csv = CSVUtils::CSVIterator.new(file, {}, 'rb:BINARY')
+
+missing_headers = options[:ignore_columns] - csv.headers
 unless missing_headers.empty?
-  raise("unkown headers #{missing_headers.join(', ')} configured ingnore headers")
+  raise("unknown headers #{missing_headers.join(', ')} in configured ignore headers")
 end
 
 hashed_rows = {}
 
-csv.each_with_index do |row, idx|
+# rows are keyed by their line number in the file
+csv.each do |row|
   options[:ignore_columns].each do |ignore_column|
     row[ignore_column] = ''
   end
-  key = Digest::SHA256.hexdigest(row.to_json)
+  key = Digest::SHA256.hexdigest(row.values.to_csv)
   hashed_rows[key] ||= []
-  hashed_rows[key] << idx
+  hashed_rows[key] << row.lineno
 end
 
 duplicate_rows = {}
@@ -52,18 +55,18 @@ hashed_rows.each do |key, row_numbers|
   end
 end
 
-csv.each_with_index do |row, idx|
-  next unless (key = duplicate_rows[idx])
+csv.each do |row|
+  next unless (key = duplicate_rows[row.lineno])
 
-  hashed_rows[key][idx] = row
+  hashed_rows[key][row.lineno] = row
 end
 
-CSV.open('duplicates-' + File.basename(ARGV[0]), 'wb') do |out|
-  out << ['duplicate_key', 'row_no'] + csv.first.keys
+CSV.open('duplicates-' + File.basename(file), 'wb:BINARY') do |out|
+  out << ['duplicate_key', 'line_no'] + csv.headers
 
   hashed_rows.each do |key, rows|
-    rows.each do |idx, row|
-      out << [key, idx] + row.values
+    rows.each do |lineno, row|
+      out << [key, lineno] + row.values
     end
   end
 end