csv-utils 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 151a25f6d4d171b169ac194665f217151a96ef43aede166c6a62ec9f2b259765
-  data.tar.gz: b223b26ea97a29f58f532e6b737d481c4f144ce2e202592f8057300790156ed5
+  metadata.gz: c07e7e0ae21409f1fcac6d5b150e1577146d79a47ba9e13b8fb28d90d50fb9c6
+  data.tar.gz: cb27bd0e1edbfc74747c5107f6d740b86e6cedcd547857d0eff2f836cddd4b9d
 SHA512:
-  metadata.gz: 7afee881db16b4afce9cceb812d4994ee66bcf172db147eaf73911961983cd0d73f9c553ca549880793e1c9a2bb2904e8444f2618ff30d040808d1b4e5383cf7
-  data.tar.gz: 4811b6ff72153107bf029f36feef65ec68646e0365d3b3987bd3231b9842b20c8c27c72216ca25350bd469887315b139737eedeba3b7c7fa17d240c729ae9a5a
+  metadata.gz: b7402f247e5cead6fba5259df4465836778026e2abdfd2a40281c22ed21679f60e15b23d913d7bb1bb559345557b31d3e024ab181a11c131d9159a7d64794e10
+  data.tar.gz: 9fd6953261a7e29e2e7676ce770311921b447c899624f105e4d3496ed89018cc515a1b21ecf2f94b36def0a492808105da0953df48680d6beed03776b3d0e6a7

data/CHANGELOG.md

@@ -0,0 +1,55 @@
+# Changelog
+
+## [0.7.0](https://github.com/dougyouch/csv-utils/compare/v0.6.0...v0.7.0) (2026-10-05)
+
+
+### ⚠ BREAKING CHANGES
+
+* **encoding:** CSVIterator no longer takes a file mode, files are decoded as UTF-8 whatever the locale, and output is written in the decoded encoding. See UPGRADING.md.
+
+### Features
+
+* **encoding:** read every file with the encoding option and write the decoded encoding ([1a9ec4f](https://github.com/dougyouch/csv-utils/commit/1a9ec4f98cbaadfff7ef89fec0d586cea828cf91))
+* **iterator:** open files per call, build rows without zip, add auto_detect ([34ab89a](https://github.com/dougyouch/csv-utils/commit/34ab89ae49dad76507371d2bdf9a7cd310c2cf67))
+* **options:** detect windows-1252 and read utf-8 regardless of locale ([10eeca6](https://github.com/dougyouch/csv-utils/commit/10eeca69bf094fb8c2cb5b005bc74687bde61169))
+
+
+### Bug Fixes
+
+* **iterator:** strip the byte order mark after rewind and from explicit-header rows ([bf8804b](https://github.com/dougyouch/csv-utils/commit/bf8804b1c2ce16738779f3b9ecc7631c3b29a115))
+
+## [0.6.0](https://github.com/dougyouch/csv-utils/compare/v0.5.0...v0.6.0) (2026-10-05)
+
+
+### ⚠ BREAKING CHANGES
+
+* **gem:** Ruby 3.2 reached end of life in March 2026 and is no longer supported. The gemspec now declares required_ruby_version >= 3.3.
+
+### Features
+
+* **compare:** accept csv options for both files ([4336edf](https://github.com/dougyouch/csv-utils/commit/4336edf8ca928ebc73c7b58a533ee7280267907e))
+
+
+### Bug Fixes
+
+* **bom:** detect utf-32 le and strip marks from text in any encoding ([7c68517](https://github.com/dougyouch/csv-utils/commit/7c6851755b9ddfb6ca49b459d9ada578c35a634d))
+* **cli:** keep trailing whitespace and raw bytes in csv-change-eol ([a61e53a](https://github.com/dougyouch/csv-utils/commit/a61e53a3add068ebc49c1f6025f06185e8b8a702))
+* **cli:** match csv-diff rows on the --unique headers and allow empty cells ([3e0d1ae](https://github.com/dougyouch/csv-utils/commit/3e0d1ae3763090ade46068e056ee909711a2d4df))
+* **cli:** read raw bytes in csv-validator ([80c5002](https://github.com/dougyouch/csv-utils/commit/80c5002eb91474d58d7999f41385743c4f719c78))
+* **cli:** report line numbers and handle any bytes in csv-duplicate-finder ([f87144d](https://github.com/dougyouch/csv-utils/commit/f87144d07727c9a418c0a37a6e82ceedf55ecf59))
+* **cli:** run the bundled csv-readline from csv-find-error ([e6736c6](https://github.com/dougyouch/csv-utils/commit/e6736c6b2682b7611079a1bb5d10ea5df85f3f92))
+* **cli:** split single column files with csv-splitter ([ee00da6](https://github.com/dougyouch/csv-utils/commit/ee00da6f9128a744345583541850348490cabdfc))
+* **compare:** yield the last record when the other file ends first ([947d2b7](https://github.com/dougyouch/csv-utils/commit/947d2b73d4aa404c52b147782f5bc3c02f5afd42))
+* **io:** close files when a block raises ([5ffa159](https://github.com/dougyouch/csv-utils/commit/5ffa159255e6a134721075536f7b70de9f7c1244))
+* **iterator:** return an enumerator from each without a block ([751d034](https://github.com/dougyouch/csv-utils/commit/751d034d325279f4eaeca76b726e4a63a6443042))
+* **options:** count quoted headers correctly and handle empty files ([abbd084](https://github.com/dougyouch/csv-utils/commit/abbd084a4a2cf84bf27410d616c63e3679f07a14))
+* **row:** stop csv_column from changing the options hash passed in ([6a9f916](https://github.com/dougyouch/csv-utils/commit/6a9f916d1e3c9b0f27410a4cae795f80e5968cec))
+* **sort:** sort without a block and clean up temporary files on failure ([bc2d21a](https://github.com/dougyouch/csv-utils/commit/bc2d21ae63973323d12084de2710d07e06e70dbb))
+* **transformer:** allow append when headers were not read ([9d1e042](https://github.com/dougyouch/csv-utils/commit/9d1e04259e5819eaf5865895674b8aafc8d51c8f))
+
+
+### Build System
+
+* **gem:** require ruby 3.3 and add gem metadata ([e34a89a](https://github.com/dougyouch/csv-utils/commit/e34a89a473159fa4c70f5279d63ca95499a43c90))
+
+## Changelog

data/README.md

@@ -1,9 +1,12 @@
 # CSV Utils
 
-[![CI](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml/badge.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
-[![codecov](https://codecov.io/gh/dougyouch/csv-utils/graph/badge.svg)](https://codecov.io/gh/dougyouch/csv-utils)
+Streaming tools for CSV files that are too big or too broken to load into memory. Compare two sorted files into create, update and delete actions, sort with an external merge sort, transform and extend rows in batches, and build reports from Ruby objects. Command line tools pinpoint malformed rows and diff, grep, split and validate CSV files.
 
-A Ruby library providing a comprehensive set of utilities for manipulating and processing CSV files. This library offers a robust set of tools for comparing, transforming, sorting, and managing CSV data efficiently.
+[![CI](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+[![Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/coverage.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+[![Branch Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/branches.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+
+[API reference](https://rubydoc.info/gems/csv-utils) · [Changelog](CHANGELOG.md) · [Upgrading](UPGRADING.md) · [Architecture](ARCHITECTURE.md)
 
 ## Features
 
@@ -16,11 +19,13 @@ A Ruby library providing a comprehensive set of utilities for manipulating and p
 - **CSV Iteration**: Efficient iteration over CSV files with batch support
 - **CSV Extension**: Extend CSV files with additional columns
 - **CSV Options**: Auto-detect CSV file properties (separators, encoding, BOM)
+- **Encodings**: Files are read as UTF-8 whatever the locale, or in the encoding you pass or `CSVOptions` detects
+- **Byte Order Marks**: UTF-8, UTF-16 and UTF-32 byte order marks are stripped from the first header
 - **CLI Tools**: Command-line utilities for CSV debugging and manipulation
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Requires Ruby 3.3 or newer. Add this line to your application's Gemfile:
 
 ```ruby
 gem 'csv-utils'
@@ -63,7 +68,15 @@ comparator.compare('secondary.csv') do |action, record|
 end
 ```
 
-**Note**: Both CSV files must be sorted by the same key columns for comparison to work correctly.
+Pass CSV options as the third argument to read both files with them, e.g. tab separated files:
+
+```ruby
+CSVUtils::CSVCompare.new('primary.tsv', ['updated_at'], col_sep: "\t") { |src, dest| src['id'] <=> dest['id'] }
+```
+
+**Note**: Both CSV files must be sorted by the same key columns for comparison to work correctly, and the block must compare them in that same order.
+
+The block compares the key columns of a primary and a secondary record like `<=>`. Records only in the primary file are yielded as `:create`, records only in the secondary file as `:delete`, and matching records as `:update` when any of the update comparison columns differ. Without update comparison columns, no updates are yielded.
 
 ### Sorting CSV Files
 
@@ -74,8 +87,13 @@ require 'csv-utils'
 
 sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true)  # true = has headers
 sorter.sort(100_000) { |a, b| a.first.to_i <=> b.first.to_i }    # batch size, comparison block
+
+# without a block, rows are compared as arrays of strings
+sorter.sort
 ```
 
+Batches are sorted into temporary `output.csv.part.N` files next to the output, which are merged in pairs. The temporary files are removed when the sort finishes or fails.
+
 ### Transforming CSV Data
 
 Transform CSV data using a chainable pipeline:
@@ -100,6 +118,8 @@ Available pipeline methods:
 - `each { |row, headers, additional_data| }` - Side effects without modification
 - `set_headers(headers)` - Override output headers
 
+Without `read_headers`, every row is treated as data and the output has no header row. `process` closes the files it opened, even when a step raises.
+
 ### CSV Row and Report
 
 Define CSV-serializable objects and generate reports:
@@ -206,6 +226,19 @@ end
 
 # Build a lookup hash
 lookup = iterator.to_hash('id', 'name')  # { 'id_value' => 'name_value', ... }
+
+# Without a block, each returns an Enumerator
+iterator.each.with_index { |row, idx| puts "#{idx}: #{row['name']}" }
+```
+
+Line numbers count the header row as line 1, so they match what an editor shows. `headers` returns `[]` for an empty file.
+
+Given a path, the iterator opens the file for each call (`each`, `headers`, `size`, ...) and closes it when the call returns, even when the block raises or stops early, so an idle iterator holds no file handle. A CSV object passed in is rewound, never closed.
+
+Let `CSVOptions` work out the separators and encoding:
+
+```ruby
+iterator = CSVUtils::CSVIterator.auto_detect('export.tsv')
 ```
 
 ### Matching CSV Rows
@@ -271,33 +304,97 @@ options = CSVUtils::CSVOptions.new('data.csv')
 options.valid?         # true if separators detected
 options.col_separator  # detected column separator
 options.row_separator  # detected row separator
-options.encoding       # detected encoding (UTF-8, UTF-16, UTF-32)
+options.encoding       # detected encoding (UTF-8, UTF-16, UTF-32, Windows-1252, ISO-8859-1)
 options.columns        # number of columns
 options.byte_order_mark # BOM if present
+
+# Open the file with what was detected, with CSV or any class in this gem
+CSV.open('data.csv', 'rb', **options.to_csv_options)
+CSVUtils::CSVSort.new('data.csv', 'sorted.csv', true, options.to_csv_options).sort
 ```
 
-Supported column separators: `\x02`, `\t`, `|`, `,`
+`to_csv_options` returns the detected `col_sep` and `row_sep`, leaving out any that weren't found, and an `encoding` that reads values as UTF-8 strings: `'bom|utf-8'` for UTF-8 files, `'BOM|UTF-16LE:UTF-8'` and the like for files with a UTF-16 or UTF-32 byte order mark (CSV detects the row separator for those), and `'Windows-1252:UTF-8'` for files that aren't valid UTF-8.
+
+Without a byte order mark, the encoding comes from the first megabyte (`CSVOptions::SAMPLE_SIZE`): valid UTF-8 is `UTF-8`, anything else is `Windows-1252`, the usual encoding of Excel exports, or `ISO-8859-1` when the sample has one of the five bytes Windows-1252 leaves undefined. A file whose first non-UTF-8 byte comes after the sample is still read as UTF-8.
+
+Supported column separators: `\x02`, `\t`, `|`, `,` (the first one found in the header line wins)
 Supported row separators: `\r\n`, `\n`, `\r`
 
+Headers are parsed as a CSV row, so a quoted header like `"Last, First"` counts as one column. An empty file isn't valid.
+
+### Encodings and Byte Order Marks
+
+Every class handles encodings the same way:
+
+- **Reading** uses the `encoding:` CSV option, `'bom|utf-8'` by default, so values are UTF-8 strings whatever the locale (including `LANG=C` in containers and cron jobs). Bytes that aren't valid UTF-8 raise `CSV::InvalidEncodingError`.
+- **Writing** (`CSVExtender`, `CSVTransformer`, `CSVSort`) uses the encoding values were decoded to, so a file read with `'Windows-1252:UTF-8'` is written as UTF-8. `CSVReport` writes UTF-8 unless you pass `encoding:`.
+- **File modes never carry an encoding.** CSV raises `ArgumentError: encoding specified twice` when the mode and the `encoding:` option both have one.
+
+```ruby
+iterator = CSVUtils::CSVIterator.new('export.csv', encoding: 'Windows-1252:UTF-8')
+iterator = CSVUtils::CSVIterator.new('latin1.csv', encoding: 'BINARY') # raw bytes
+comparator = CSVUtils::CSVCompare.new('primary.csv', ['updated_at'], encoding: 'BINARY') { |src, dest| src['id'] <=> dest['id'] }
+sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true, CSVUtils::CSVOptions.new('input.csv').to_csv_options)
+```
+
+`CSVSort`, `CSVExtender`, `CSVTransformer`, `CSVCompare` and `CSVReport` take CSV options as their last argument and `CSVIterator` takes them second; they're passed to `CSV.open` for file paths. `CSVUtils::EncodingOptions` holds the rules above. Upgrading from 0.6? See [UPGRADING.md](UPGRADING.md).
+
+`CSVUtils::ByteOrderMark` detects and strips UTF-8, UTF-16 and UTF-32 byte order marks; `CSVIterator`, `CSVCompare` and `CSVOptions` use it to clean the first header.
+
 ## CLI Tools
 
-The gem includes command-line utilities for CSV debugging:
+The gem installs command-line utilities for CSV debugging:
 
 | Command | Description |
 |---------|-------------|
-| `csv-find-error` | Locate malformed CSV errors with context |
-| `csv-readline` | Read specific lines from a CSV file |
-| `csv-validator` | Validate CSV structure |
-| `csv-diff` | Compare two CSV files |
-| `csv-grep` | Search within CSV content |
-| `csv-splitter` | Split large CSV files into parts |
-| `csv-explorer` | Interactive CSV exploration |
+| `csv-find-error` | Locate the first malformed row and show it with `csv-readline` |
+| `csv-readline` | Print the columns of a line, flagging stray quotes |
+| `csv-validator` | Report rows with the wrong number of columns and values that aren't UTF-8 |
+| `csv-diff` | Compare two CSV files by a unique key |
+| `csv-grep` | Search columns for a pattern |
+| `csv-splitter` | Split a large CSV file into parts |
+| `csv-explorer` | Open an IRB session with the file loaded as a `CSVIterator` |
 | `csv-duplicate-finder` | Find duplicate rows |
-| `csv-change-eol` | Convert line endings |
+| `csv-change-eol` | Rewrite a file with a different line ending |
+
+```bash
+# find the first malformed row
+csv-find-error data.csv
+
+# print the row on line 1042, read as 3 lines for values with embedded newlines, including empty columns
+csv-readline --all data.csv 1042 3
+
+# compare by the id column, ignoring updated_at; writes diff-results-old.csv
+csv-diff -u id -i updated_at old.csv new.csv
+
+# case-insensitive search of the email and name columns, first 10 matches
+csv-grep -s 'smith' -c email,name -i -l 10 data.csv
+
+# split into files of 100,000 rows, each with the header
+csv-splitter -r 100000 data.csv
+
+# find rows that are duplicates apart from id; writes duplicates-data.csv
+csv-duplicate-finder -i id data.csv
+
+# end every row with |^| and a newline
+csv-change-eol data.csv 7C5E7C0A
+```
+
+`csv-validator` needs the `rchardet` gem (`gem install rchardet`) to guess the encoding of values that aren't UTF-8. It writes the converted values to `utf8-correction.csv`.
 
 ## Development
 
-After checking out the repo, run `bundle install` to install dependencies. Then, run `bundle exec rspec` to run the tests.
+After checking out the repo, run `bundle install` to install dependencies. Then:
+
+```bash
+bundle exec rspec       # run the tests
+bundle exec rubocop     # run the linter
+bundle exec yard stats --list-undoc   # check the API docs
+```
+
+CI runs RuboCop, requires every public class, module, constant and method to have a YARD doc comment, and runs the specs on Ruby 3.3 and the `.ruby-version` Ruby with 100% line and branch coverage (`CI=1 bundle exec rspec` enforces it locally).
+
+Releases are automated by [release-please](https://github.com/googleapis/release-please) from [conventional commits](https://www.conventionalcommits.org/): merging its release PR tags the version and publishes the gem.
 
 ## Contributing
 
@@ -305,4 +402,4 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/dougyo
 
 ## License
 
-The gem is available as open source under the terms of the MIT License.
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/bin/csv-change-eol

@@ -28,12 +28,11 @@ csv_file = ARGV.shift || exit_on_error('no csv file specified')
 eol_sequence = ARGV.shift || exit_on_error('no EOL character sequence specified')
 
 exit_on_error("file #{csv_file} not found") unless File.exist?(csv_file)
-exit_on_error("not a HEX sequece (#{eol_sequence})") unless eol_sequence =~ /\A[0-9a-f]+\z/i
+exit_on_error("not a HEX sequence (#{eol_sequence})") unless eol_sequence =~ /\A[0-9a-f]+\z/i
 exit_on_error("incorrect number of characters in (#{eol_sequence}), should be even") unless eol_sequence.size.even?
 
 eol_sequence = [eol_sequence].pack('H*')
 
-
 escaped_csv_file =
   if csv_file =~ /\.csv$/i
     csv_file.sub(/(\.csv)$/i, '.escaped-eol\1')
@@ -41,13 +40,10 @@ escaped_csv_file =
     csv_file + '.escaped-eol'
   end
 
-
+# read and write bytes as is, so values in any encoding pass through unchanged
 File.open(escaped_csv_file, 'wb') do |out|
-  CSV.foreach(csv_file) do |row|
-    line = row.to_csv
-    line.rstrip!
-    line.concat(eol_sequence)
-    out.write line
+  CSV.foreach(csv_file, 'rb:BINARY') do |row|
+    out.write row.to_csv(row_sep: eol_sequence)
   end
 end
 

data/bin/csv-diff

@@ -8,7 +8,7 @@ options = {
   sort_batch_size: 1_000_000
 }
 OptionParser.new do |opts|
-  opts.banner = 'Usage: ' + File.basename(__FILE__) + ' [options] <csv file>'
+  opts.banner = 'Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>'
 
   opts.on('-h', '--help', 'Prints this help') do
     puts opts
@@ -28,52 +28,44 @@ OptionParser.new do |opts|
   end
 end.parse!
 
+unless ARGV.size == 2
+  $stderr.puts('Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>')
+  exit 1
+end
+
 require 'csv-utils'
 
 csv1 = CSVUtils::CSVIterator.new(ARGV[0])
 csv2 = CSVUtils::CSVIterator.new(ARGV[1])
+headers = csv1.headers
 
-unless csv1.first.keys == csv2.first.keys
-  $stderr.puts("headers do not match #{ARGV[0]} headers #{csv1.first.keys}, #{ARGV[1]} headers #{csv2.first.keys}")
+unless headers == csv2.headers
+  $stderr.puts("headers do not match #{ARGV[0]} headers #{headers}, #{ARGV[1]} headers #{csv2.headers}")
   exit 1
 end
 
-unknown_unique_headers = options[:unique_headers] - csv1.first.keys
+unknown_unique_headers = options[:unique_headers] - headers
 unless unknown_unique_headers.empty?
   $stderr.puts("specified unique headers are unknown #{unknown_unique_headers}")
   exit 1
 end
 
-unknown_ignore_headers = options[:ignore_headers] - csv1.first.keys
+unknown_ignore_headers = options[:ignore_headers] - headers
 unless unknown_ignore_headers.empty?
   $stderr.puts("specified headers to ignore are unknown #{unknown_ignore_headers}")
   exit 1
 end
 
-options[:unique_headers] = [csv1.first.keys.first] if options[:unique_headers].empty?
+options[:unique_headers] = [headers.first] if options[:unique_headers].empty?
 
 puts "uniqueness header(s) are #{options[:unique_headers].join(', ')}"
 
-unique_header_indexes = []
-csv1.first.keys.each_with_index do |header, idx|
-  unique_header_indexes << idx if options[:unique_headers].include?(header)
-end
+unique_header_indexes = options[:unique_headers].map { |header| headers.index(header) }
 
+# empty cells are nil, so values are compared as strings
 sort_compare_proc = proc do |csv1_row, csv2_row|
-  result = 0
-  unique_header_indexes.each do |idx|
-    result = csv1_row[idx] <=> csv2_row[idx]
-    break unless result == 0
-  end
-
-  if result == 0
-    csv1_row.each_with_index do |csv1_col, idx|
-      csv2_col = csv2_row[idx]
-      result = csv1_col <=> csv2_col
-      break unless result == 0
-    end
-  end
-
+  result = unique_header_indexes.map { |idx| csv1_row[idx].to_s } <=> unique_header_indexes.map { |idx| csv2_row[idx].to_s }
+  result = csv1_row.map(&:to_s) <=> csv2_row.map(&:to_s) if result.zero?
   result
 end
 
@@ -88,25 +80,17 @@ puts "sorting #{ARGV[1]}"
 sorter = CSVUtils::CSVSort.new(ARGV[1], csv2_sorted_file_name)
 sorter.sort(options[:sort_batch_size], &sort_compare_proc)
 
-options[:unique_headers] = [csv1.first.keys.first]
-update_comparison_columns = csv1.first.keys - options[:unique_headers]
-update_comparison_columns -= options[:ignore_headers]
+update_comparison_columns = headers - options[:unique_headers] - options[:ignore_headers]
 
 comparer = CSVUtils::CSVCompare.new(csv1_sorted_file_name, update_comparison_columns) do |csv1_row, csv2_row|
-  result = 0
-  options[:unique_headers].each do |header|
-    result = csv1_row[header] <=> csv2_row[header]
-    break unless result == 0
-  end
-
-  result
+  options[:unique_headers].map { |header| csv1_row[header].to_s } <=> options[:unique_headers].map { |header| csv2_row[header].to_s }
 end
 
 stats = Hash.new(0)
 puts "comparing #{ARGV[0]} with #{ARGV[1]}"
 diff_file_name = 'diff-results-' + File.basename(ARGV[0])
 CSV.open(diff_file_name, 'wb') do |out|
-  out << ['Result'] + csv1.first.keys
+  out << ['Result'] + headers
   comparer.compare(csv2_sorted_file_name) do |action, record|
     stats[action] += 1
     out << [action] + record.values

data/bin/csv-duplicate-finder

@@ -13,31 +13,34 @@ OptionParser.new do |opts|
     exit
   end
 
-  opts.on('-i', '--ignore HEADERS', 'Comman separated list of headers to ignore') do |v|
+  opts.on('-i', '--ignore HEADERS', 'Comma separated list of headers to ignore') do |v|
     options[:ignore_columns] = v.split(',')
   end
 end.parse!
 
 require 'digest/sha2'
-require 'json'
 require 'csv-utils'
 
-csv = CSVUtils::CSVIterator.new(ARGV[0])
+file = ARGV[0] || raise('no CSV file specified')
 
-missing_headers = options[:ignore_columns] - csv.first.keys
+# read bytes as is, so values that aren't valid UTF-8 can still be compared
+csv = CSVUtils::CSVIterator.new(file, encoding: 'BINARY')
+
+missing_headers = options[:ignore_columns] - csv.headers
 unless missing_headers.empty?
   raise("unknown headers #{missing_headers.join(', ')} in configured ignore headers")
 end
 
 hashed_rows = {}
 
-csv.each_with_index do |row, idx|
+# rows are keyed by their line number in the file
+csv.each do |row|
   options[:ignore_columns].each do |ignore_column|
     row[ignore_column] = ''
   end
-  key = Digest::SHA256.hexdigest(row.to_json)
+  key = Digest::SHA256.hexdigest(row.values.to_csv)
   hashed_rows[key] ||= []
-  hashed_rows[key] << idx
+  hashed_rows[key] << row.lineno
 end
 
 duplicate_rows = {}
@@ -52,18 +55,18 @@ hashed_rows.each do |key, row_numbers|
   end
 end
 
-csv.each_with_index do |row, idx|
-  next unless (key = duplicate_rows[idx])
+csv.each do |row|
+  next unless (key = duplicate_rows[row.lineno])
 
-  hashed_rows[key][idx] = row
+  hashed_rows[key][row.lineno] = row
 end
 
-CSV.open('duplicates-' + File.basename(ARGV[0]), 'wb') do |out|
-  out << ['duplicate_key', 'row_no'] + csv.first.keys
+CSV.open('duplicates-' + File.basename(file), 'wb:BINARY') do |out|
+  out << ['duplicate_key', 'line_no'] + csv.headers
 
   hashed_rows.each do |key, rows|
-    rows.each do |idx, row|
-      out << [key, idx] + row.values
+    rows.each do |lineno, row|
+      out << [key, lineno] + row.values
     end
   end
 end

data/bin/csv-find-error

@@ -1,19 +1,21 @@
 #!/usr/bin/env ruby
 
 require 'csv'
-require 'shellwords'
+require 'rbconfig'
+
+file = ARGV[0] || abort('Usage: csv-find-error <csv file>')
 
 prev_row = nil
 begin
-  CSV.open(ARGV[0], 'rb').each { |row| prev_row = row }
+  CSV.open(file, 'rb') { |csv| csv.each { |row| prev_row = row } }
 rescue CSV::MalformedCSVError => e
   puts e.class.to_s + ': ' + e.message
   puts "previous row was #{prev_row}"
   if e.message =~ /line (\d+)/
     lineno = $1.to_i
-    cmd = "csv-readline #{Shellwords.escape(ARGV[0])} #{lineno}"
-    puts "running #{cmd}"
-    system(cmd)
+    # run the csv-readline next to this script, with this gem's lib, rather than whichever is on the PATH
+    puts "running csv-readline #{file} #{lineno}"
+    system(RbConfig.ruby, '-I', File.expand_path('../lib', __dir__), File.join(__dir__, 'csv-readline'), file, lineno.to_s)
   end
   exit 1
 end

data/bin/csv-readline

@@ -2,21 +2,10 @@
 
 require 'optparse'
 
-BYTE_ORDER_MARKS = {
-  "\xEF\xBB\xBF".force_encoding('ASCII-8BIT') => 'UTF-8',
-  "\xFE\xFF".force_encoding('ASCII-8BIT') => 'UTF-16',
-  "\xFF\xFE".force_encoding('ASCII-8BIT') => 'UTF-16',
-  "\x00\x00\xFE\xFF".force_encoding('ASCII-8BIT') => 'UTF-32',
-  "\xFF\xFE\x00\x00".force_encoding('ASCII-8BIT') => 'UTF-32'
-}
+require 'csv-utils'
 
 def strip_byte_order_mark(line)
-  BYTE_ORDER_MARKS.keys.detect do |bom|
-    if line =~ /\A#{bom}/
-      return line.sub(/\A#{bom}/, '')
-    end
-  end
-  line
+  CSVUtils::ByteOrderMark.strip(line)
 end
 
 def csv_parse_line(line)
@@ -87,7 +76,7 @@ end
 
 def parse_csv_row(file, lineno, number_of_lines)
   cnt = 1
-  str = ''
+  str = +''
   file.each do |line|
     cnt += 1
     next if cnt < lineno

data/bin/csv-splitter

@@ -15,6 +15,8 @@ OptionParser.new do |opts|
   end
 
   opts.on('-r', '--rows ROWS', Integer, 'Max number of rows per a CSV') do |v|
+    raise OptionParser::InvalidArgument, "--rows must be positive (#{v})" unless v.positive?
+
     options[:rows] = v
   end
 
@@ -36,20 +38,20 @@ def get_split_file_name(file, num_files, total_files)
 end
 
 csv_options = CSVUtils::CSVOptions.new(file)
+# a single column file has no separator to detect
+col_sep = csv_options.col_separator || ','
 
 csv = CSV.open(
   file,
   'rb',
-  col_sep: csv_options.col_separator,
+  col_sep: col_sep,
   liberal_parsing: true
 )
 
 headers = options[:no_header] ? nil : csv.shift
 
 total_rows = 0
-while csv.shift
-  total_rows +=1
-end
+total_rows += 1 while csv.shift
 
 csv.rewind
 csv.shift if headers
@@ -66,7 +68,7 @@ append_row_proc = proc do |row|
     out.close if out
     num_rows = 0
     num_files += 1
-    out = CSV.open(get_split_file_name(file, num_files, total_files), 'wb', col_sep: csv_options.col_separator)
+    out = CSV.open(get_split_file_name(file, num_files, total_files), 'wb', col_sep: col_sep)
     out << headers if headers
   end
 

data/bin/csv-validator

@@ -15,6 +15,8 @@ def utf8?(str)
 end
 
 def convert_to_utf8(str, current_encoding)
+  return nil unless current_encoding
+
   str.force_encoding(current_encoding)
   return nil unless str.valid_encoding?
 
@@ -30,12 +32,13 @@ def strip_bom!(col)
 end
 
 id_column_name = ARGV[1]
-csv = CSV.open(ARGV[0], 'rb')
+# read bytes as is; reading as UTF-8 would raise on the values this script looks for
+csv = CSV.open(ARGV[0] || abort('Usage: csv-validator <csv file> [<id header>]'), 'rb:BINARY')
 out = nil
 
 begin
-  headers = csv.shift
-  strip_bom!(headers[0])
+  headers = csv.shift || []
+  strip_bom!(headers[0]) if headers[0]
 
   id_column_name ||= headers[0]
   unless headers.include?(id_column_name)
@@ -68,9 +71,10 @@ begin
     row.each_with_index do |col, idx|
       next if col.nil? || utf8?(col)
 
-      $stderr.puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: none UTF-8 characters found in \"#{col}\""
-      if (col_utf8_encoded = convert_to_utf8(col, detect_encoding(col)))
-        puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: converted to UTF-8 from #{detect_encoding(col)} \"#{col_utf8_encoded}\""
+      $stderr.puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: non UTF-8 characters found in #{col.inspect}"
+      encoding = detect_encoding(col)
+      if (col_utf8_encoded = convert_to_utf8(col, encoding))
+        puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: converted to UTF-8 from #{encoding} \"#{col_utf8_encoded}\""
         out_proc.call [row[id_column_num], csv_lineno, (idx + 1), headers[idx], col_utf8_encoded]
       else
         $stderr.puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: unknown character encoding"

data/lib/csv-utils.rb

@@ -2,8 +2,12 @@
 
 require 'csv'
 
-# Collection of tools for working with CSV files.
+# Tools for comparing, sorting, transforming and debugging CSV files, including ones too large to load
+# into memory. Each class is autoloaded on first use.
 module CSVUtils
+  autoload :ByteOrderMark, 'csv_utils/byte_order_mark'
+  autoload :CharacterEncoding, 'csv_utils/character_encoding'
+  autoload :EncodingOptions, 'csv_utils/encoding_options'
   autoload :CSVCompare, 'csv_utils/csv_compare'
   autoload :CSVExtender, 'csv_utils/csv_extender'
   autoload :CSVIterator, 'csv_utils/csv_iterator'