csv-utils 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 151a25f6d4d171b169ac194665f217151a96ef43aede166c6a62ec9f2b259765
-  data.tar.gz: b223b26ea97a29f58f532e6b737d481c4f144ce2e202592f8057300790156ed5
+  metadata.gz: 423eacb18512c595abe83765ccfd4d945f1e2e0d18a6370b6841997eb428ddb7
+  data.tar.gz: 3d2df7f1ee62eaebc25d00281401bf49f0fccdb809ec03cbac90e6bf5ba2fd62
 SHA512:
-  metadata.gz: 7afee881db16b4afce9cceb812d4994ee66bcf172db147eaf73911961983cd0d73f9c553ca549880793e1c9a2bb2904e8444f2618ff30d040808d1b4e5383cf7
-  data.tar.gz: 4811b6ff72153107bf029f36feef65ec68646e0365d3b3987bd3231b9842b20c8c27c72216ca25350bd469887315b139737eedeba3b7c7fa17d240c729ae9a5a
+  metadata.gz: 2f9856120036ae6a87e2e758186221b7c0b48aa8d94365f7bdf4bffdf8f292868e43cf3c378eff7460902d710de2859d5406ac9ef5384c07f9728f2f1bad4ef4
+  data.tar.gz: 9e96e672cb1f4da44ae67de48d3e783cf1d2b180c0d8f4a9706ce1ace07b9ac456a689653dfe84537c0cab71e6c58cdb24e5caeec0bcc86bc73a374615ac4e0a

data/CHANGELOG.md

@@ -0,0 +1,37 @@
+# Changelog
+
+## [0.6.0](https://github.com/dougyouch/csv-utils/compare/v0.5.0...v0.6.0) (2026-10-05)
+
+
+### ⚠ BREAKING CHANGES
+
+* **gem:** Ruby 3.2 reached end of life in March 2026 and is no longer supported. The gemspec now declares required_ruby_version >= 3.3.
+
+### Features
+
+* **compare:** accept csv options for both files ([4336edf](https://github.com/dougyouch/csv-utils/commit/4336edf8ca928ebc73c7b58a533ee7280267907e))
+
+
+### Bug Fixes
+
+* **bom:** detect utf-32 le and strip marks from text in any encoding ([7c68517](https://github.com/dougyouch/csv-utils/commit/7c6851755b9ddfb6ca49b459d9ada578c35a634d))
+* **cli:** keep trailing whitespace and raw bytes in csv-change-eol ([a61e53a](https://github.com/dougyouch/csv-utils/commit/a61e53a3add068ebc49c1f6025f06185e8b8a702))
+* **cli:** match csv-diff rows on the --unique headers and allow empty cells ([3e0d1ae](https://github.com/dougyouch/csv-utils/commit/3e0d1ae3763090ade46068e056ee909711a2d4df))
+* **cli:** read raw bytes in csv-validator ([80c5002](https://github.com/dougyouch/csv-utils/commit/80c5002eb91474d58d7999f41385743c4f719c78))
+* **cli:** report line numbers and handle any bytes in csv-duplicate-finder ([f87144d](https://github.com/dougyouch/csv-utils/commit/f87144d07727c9a418c0a37a6e82ceedf55ecf59))
+* **cli:** run the bundled csv-readline from csv-find-error ([e6736c6](https://github.com/dougyouch/csv-utils/commit/e6736c6b2682b7611079a1bb5d10ea5df85f3f92))
+* **cli:** split single column files with csv-splitter ([ee00da6](https://github.com/dougyouch/csv-utils/commit/ee00da6f9128a744345583541850348490cabdfc))
+* **compare:** yield the last record when the other file ends first ([947d2b7](https://github.com/dougyouch/csv-utils/commit/947d2b73d4aa404c52b147782f5bc3c02f5afd42))
+* **io:** close files when a block raises ([5ffa159](https://github.com/dougyouch/csv-utils/commit/5ffa159255e6a134721075536f7b70de9f7c1244))
+* **iterator:** return an enumerator from each without a block ([751d034](https://github.com/dougyouch/csv-utils/commit/751d034d325279f4eaeca76b726e4a63a6443042))
+* **options:** count quoted headers correctly and handle empty files ([abbd084](https://github.com/dougyouch/csv-utils/commit/abbd084a4a2cf84bf27410d616c63e3679f07a14))
+* **row:** stop csv_column from changing the options hash passed in ([6a9f916](https://github.com/dougyouch/csv-utils/commit/6a9f916d1e3c9b0f27410a4cae795f80e5968cec))
+* **sort:** sort without a block and clean up temporary files on failure ([bc2d21a](https://github.com/dougyouch/csv-utils/commit/bc2d21ae63973323d12084de2710d07e06e70dbb))
+* **transformer:** allow append when headers were not read ([9d1e042](https://github.com/dougyouch/csv-utils/commit/9d1e04259e5819eaf5865895674b8aafc8d51c8f))
+
+
+### Build System
+
+* **gem:** require ruby 3.3 and add gem metadata ([e34a89a](https://github.com/dougyouch/csv-utils/commit/e34a89a473159fa4c70f5279d63ca95499a43c90))
+
+## Changelog

data/README.md

@@ -1,9 +1,12 @@
 # CSV Utils
 
-[![CI](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml/badge.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
-[![codecov](https://codecov.io/gh/dougyouch/csv-utils/graph/badge.svg)](https://codecov.io/gh/dougyouch/csv-utils)
+Streaming tools for CSV files that are too big or too broken to load into memory. Compare two sorted files into create, update and delete actions, sort with an external merge sort, transform and extend rows in batches, and build reports from Ruby objects. Command line tools pinpoint malformed rows and diff, grep, split and validate CSV files.
 
-A Ruby library providing a comprehensive set of utilities for manipulating and processing CSV files. This library offers a robust set of tools for comparing, transforming, sorting, and managing CSV data efficiently.
+[![CI](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+[![Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/coverage.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+[![Branch Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/branches.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
+
+[API reference](https://rubydoc.info/gems/csv-utils) · [Changelog](CHANGELOG.md) · [Architecture](ARCHITECTURE.md)
 
 ## Features
 
@@ -16,11 +19,12 @@ A Ruby library providing a comprehensive set of utilities for manipulating and p
 - **CSV Iteration**: Efficient iteration over CSV files with batch support
 - **CSV Extension**: Extend CSV files with additional columns
 - **CSV Options**: Auto-detect CSV file properties (separators, encoding, BOM)
+- **Byte Order Marks**: UTF-8, UTF-16 and UTF-32 byte order marks are stripped from the first header
 - **CLI Tools**: Command-line utilities for CSV debugging and manipulation
 
 ## Installation
 
-Add this line to your application's Gemfile:
+Requires Ruby 3.3 or newer. Add this line to your application's Gemfile:
 
 ```ruby
 gem 'csv-utils'
@@ -63,7 +67,15 @@ comparator.compare('secondary.csv') do |action, record|
 end
 ```
 
-**Note**: Both CSV files must be sorted by the same key columns for comparison to work correctly.
+Pass CSV options as the third argument to read both files with them, e.g. tab separated files:
+
+```ruby
+CSVUtils::CSVCompare.new('primary.tsv', ['updated_at'], col_sep: "\t") { |src, dest| src['id'] <=> dest['id'] }
+```
+
+**Note**: Both CSV files must be sorted by the same key columns for comparison to work correctly, and the block must compare them in that same order.
+
+The block compares the key columns of a primary and a secondary record like `<=>`. Records only in the primary file are yielded as `:create`, records only in the secondary file as `:delete`, and matching records as `:update` when any of the update comparison columns differ. Without update comparison columns, no updates are yielded.
 
 ### Sorting CSV Files
 
@@ -74,8 +86,13 @@ require 'csv-utils'
 
 sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true)  # true = has headers
 sorter.sort(100_000) { |a, b| a.first.to_i <=> b.first.to_i }    # batch size, comparison block
+
+# without a block, rows are compared as arrays of strings
+sorter.sort
 ```
 
+Batches are sorted into temporary `output.csv.part.N` files next to the output, which are merged in pairs. The temporary files are removed when the sort finishes or fails.
+
 ### Transforming CSV Data
 
 Transform CSV data using a chainable pipeline:
@@ -100,6 +117,8 @@ Available pipeline methods:
 - `each { |row, headers, additional_data| }` - Side effects without modification
 - `set_headers(headers)` - Override output headers
 
+Without `read_headers`, every row is treated as data and the output has no header row. `process` closes the files it opened, even when a step raises.
+
 ### CSV Row and Report
 
 Define CSV-serializable objects and generate reports:
@@ -206,8 +225,13 @@ end
 
 # Build a lookup hash
 lookup = iterator.to_hash('id', 'name')  # { 'id_value' => 'name_value', ... }
+
+# Without a block, each returns an Enumerator
+iterator.each.with_index { |row, idx| puts "#{idx}: #{row['name']}" }
 ```
 
+Line numbers count the header row as line 1, so they match what an editor shows. `headers` returns `[]` for an empty file.
+
 ### Matching CSV Rows
 
 Filter CSV rows using regex patterns:
@@ -276,28 +300,79 @@ options.columns        # number of columns
 options.byte_order_mark # BOM if present
 ```
 
-Supported column separators: `\x02`, `\t`, `|`, `,`
+Supported column separators: `\x02`, `\t`, `|`, `,` (the first one found in the header line wins)
 Supported row separators: `\r\n`, `\n`, `\r`
 
+Headers are parsed as a CSV row, so a quoted header like `"Last, First"` counts as one column. An empty file isn't valid.
+
+### Encodings and Byte Order Marks
+
+Files are opened with mode `'rb'`. With csv 3.3 and later, when Ruby's default external encoding is UTF-8 (the usual case), CSV reads `'rb'` files as UTF-8 and raises `CSV::InvalidEncodingError` on bytes that aren't valid UTF-8. To read a file's bytes as is, pass an explicit encoding:
+
+```ruby
+iterator = CSVUtils::CSVIterator.new('latin1.csv', {}, 'rb:BINARY')
+comparator = CSVUtils::CSVCompare.new('primary.csv', ['updated_at'], encoding: 'BINARY') { |src, dest| src['id'] <=> dest['id'] }
+sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true, encoding: 'BINARY')
+```
+
+`CSVSort`, `CSVExtender` and `CSVTransformer` take CSV options as their last argument and `CSVIterator` takes them second; they're passed to `CSV.open` for file paths.
+
+`CSVUtils::ByteOrderMark` detects and strips UTF-8, UTF-16 and UTF-32 byte order marks; `CSVIterator`, `CSVCompare` and `CSVOptions` use it to clean the first header.
+
 ## CLI Tools
 
-The gem includes command-line utilities for CSV debugging:
+The gem installs command-line utilities for CSV debugging:
 
 | Command | Description |
 |---------|-------------|
-| `csv-find-error` | Locate malformed CSV errors with context |
-| `csv-readline` | Read specific lines from a CSV file |
-| `csv-validator` | Validate CSV structure |
-| `csv-diff` | Compare two CSV files |
-| `csv-grep` | Search within CSV content |
-| `csv-splitter` | Split large CSV files into parts |
-| `csv-explorer` | Interactive CSV exploration |
+| `csv-find-error` | Locate the first malformed row and show it with `csv-readline` |
+| `csv-readline` | Print the columns of a line, flagging stray quotes |
+| `csv-validator` | Report rows with the wrong number of columns and values that aren't UTF-8 |
+| `csv-diff` | Compare two CSV files by a unique key |
+| `csv-grep` | Search columns for a pattern |
+| `csv-splitter` | Split a large CSV file into parts |
+| `csv-explorer` | Open an IRB session with the file loaded as a `CSVIterator` |
 | `csv-duplicate-finder` | Find duplicate rows |
-| `csv-change-eol` | Convert line endings |
+| `csv-change-eol` | Rewrite a file with a different line ending |
+
+```bash
+# find the first malformed row
+csv-find-error data.csv
+
+# print the row on line 1042, read as 3 lines for values with embedded newlines, including empty columns
+csv-readline --all data.csv 1042 3
+
+# compare by the id column, ignoring updated_at; writes diff-results-old.csv
+csv-diff -u id -i updated_at old.csv new.csv
+
+# case-insensitive search of the email and name columns, first 10 matches
+csv-grep -s 'smith' -c email,name -i -l 10 data.csv
+
+# split into files of 100,000 rows, each with the header
+csv-splitter -r 100000 data.csv
+
+# find rows that are duplicates apart from id; writes duplicates-data.csv
+csv-duplicate-finder -i id data.csv
+
+# end every row with |^| and a newline
+csv-change-eol data.csv 7C5E7C0A
+```
+
+`csv-validator` needs the `rchardet` gem (`gem install rchardet`) to guess the encoding of values that aren't UTF-8. It writes the converted values to `utf8-correction.csv`.
 
 ## Development
 
-After checking out the repo, run `bundle install` to install dependencies. Then, run `bundle exec rspec` to run the tests.
+After checking out the repo, run `bundle install` to install dependencies. Then:
+
+```bash
+bundle exec rspec       # run the tests
+bundle exec rubocop     # run the linter
+bundle exec yard stats --list-undoc   # check the API docs
+```
+
+CI runs RuboCop, requires every public class, module, constant and method to have a YARD doc comment, and runs the specs on Ruby 3.3 and the `.ruby-version` Ruby with 100% line and branch coverage (`CI=1 bundle exec rspec` enforces it locally).
+
+Releases are automated by [release-please](https://github.com/googleapis/release-please) from [conventional commits](https://www.conventionalcommits.org/): merging its release PR tags the version and publishes the gem.
 
 ## Contributing
 
@@ -305,4 +380,4 @@ Bug reports and pull requests are welcome on GitHub at https://github.com/dougyo
 
 ## License
 
-The gem is available as open source under the terms of the MIT License.
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/bin/csv-change-eol

@@ -28,12 +28,11 @@ csv_file = ARGV.shift || exit_on_error('no csv file specified')
 eol_sequence = ARGV.shift || exit_on_error('no EOL character sequence specified')
 
 exit_on_error("file #{csv_file} not found") unless File.exist?(csv_file)
-exit_on_error("not a HEX sequece (#{eol_sequence})") unless eol_sequence =~ /\A[0-9a-f]+\z/i
+exit_on_error("not a HEX sequence (#{eol_sequence})") unless eol_sequence =~ /\A[0-9a-f]+\z/i
 exit_on_error("incorrect number of characters in (#{eol_sequence}), should be even") unless eol_sequence.size.even?
 
 eol_sequence = [eol_sequence].pack('H*')
 
-
 escaped_csv_file =
   if csv_file =~ /\.csv$/i
     csv_file.sub(/(\.csv)$/i, '.escaped-eol\1')
@@ -41,13 +40,10 @@ escaped_csv_file =
     csv_file + '.escaped-eol'
   end
 
-
+# read and write bytes as is, so values in any encoding pass through unchanged
 File.open(escaped_csv_file, 'wb') do |out|
-  CSV.foreach(csv_file) do |row|
-    line = row.to_csv
-    line.rstrip!
-    line.concat(eol_sequence)
-    out.write line
+  CSV.foreach(csv_file, 'rb:BINARY') do |row|
+    out.write row.to_csv(row_sep: eol_sequence)
   end
 end
 

data/bin/csv-diff

@@ -8,7 +8,7 @@ options = {
   sort_batch_size: 1_000_000
 }
 OptionParser.new do |opts|
-  opts.banner = 'Usage: ' + File.basename(__FILE__) + ' [options] <csv file>'
+  opts.banner = 'Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>'
 
   opts.on('-h', '--help', 'Prints this help') do
     puts opts
@@ -28,52 +28,44 @@ OptionParser.new do |opts|
   end
 end.parse!
 
+unless ARGV.size == 2
+  $stderr.puts('Usage: ' + File.basename(__FILE__) + ' [options] <csv file 1> <csv file 2>')
+  exit 1
+end
+
 require 'csv-utils'
 
 csv1 = CSVUtils::CSVIterator.new(ARGV[0])
 csv2 = CSVUtils::CSVIterator.new(ARGV[1])
+headers = csv1.headers
 
-unless csv1.first.keys == csv2.first.keys
-  $stderr.puts("headers do not match #{ARGV[0]} headers #{csv1.first.keys}, #{ARGV[1]} headers #{csv2.first.keys}")
+unless headers == csv2.headers
+  $stderr.puts("headers do not match #{ARGV[0]} headers #{headers}, #{ARGV[1]} headers #{csv2.headers}")
   exit 1
 end
 
-unknown_unique_headers = options[:unique_headers] - csv1.first.keys
+unknown_unique_headers = options[:unique_headers] - headers
 unless unknown_unique_headers.empty?
   $stderr.puts("specified unique headers are unknown #{unknown_unique_headers}")
   exit 1
 end
 
-unknown_ignore_headers = options[:ignore_headers] - csv1.first.keys
+unknown_ignore_headers = options[:ignore_headers] - headers
 unless unknown_ignore_headers.empty?
   $stderr.puts("specified headers to ignore are unknown #{unknown_ignore_headers}")
   exit 1
 end
 
-options[:unique_headers] = [csv1.first.keys.first] if options[:unique_headers].empty?
+options[:unique_headers] = [headers.first] if options[:unique_headers].empty?
 
 puts "uniqueness header(s) are #{options[:unique_headers].join(', ')}"
 
-unique_header_indexes = []
-csv1.first.keys.each_with_index do |header, idx|
-  unique_header_indexes << idx if options[:unique_headers].include?(header)
-end
+unique_header_indexes = options[:unique_headers].map { |header| headers.index(header) }
 
+# empty cells are nil, so values are compared as strings
 sort_compare_proc = proc do |csv1_row, csv2_row|
-  result = 0
-  unique_header_indexes.each do |idx|
-    result = csv1_row[idx] <=> csv2_row[idx]
-    break unless result == 0
-  end
-
-  if result == 0
-    csv1_row.each_with_index do |csv1_col, idx|
-      csv2_col = csv2_row[idx]
-      result = csv1_col <=> csv2_col
-      break unless result == 0
-    end
-  end
-
+  result = unique_header_indexes.map { |idx| csv1_row[idx].to_s } <=> unique_header_indexes.map { |idx| csv2_row[idx].to_s }
+  result = csv1_row.map(&:to_s) <=> csv2_row.map(&:to_s) if result.zero?
   result
 end
 
@@ -88,25 +80,17 @@ puts "sorting #{ARGV[1]}"
 sorter = CSVUtils::CSVSort.new(ARGV[1], csv2_sorted_file_name)
 sorter.sort(options[:sort_batch_size], &sort_compare_proc)
 
-options[:unique_headers] = [csv1.first.keys.first]
-update_comparison_columns = csv1.first.keys - options[:unique_headers]
-update_comparison_columns -= options[:ignore_headers]
+update_comparison_columns = headers - options[:unique_headers] - options[:ignore_headers]
 
 comparer = CSVUtils::CSVCompare.new(csv1_sorted_file_name, update_comparison_columns) do |csv1_row, csv2_row|
-  result = 0
-  options[:unique_headers].each do |header|
-    result = csv1_row[header] <=> csv2_row[header]
-    break unless result == 0
-  end
-
-  result
+  options[:unique_headers].map { |header| csv1_row[header].to_s } <=> options[:unique_headers].map { |header| csv2_row[header].to_s }
 end
 
 stats = Hash.new(0)
 puts "comparing #{ARGV[0]} with #{ARGV[1]}"
 diff_file_name = 'diff-results-' + File.basename(ARGV[0])
 CSV.open(diff_file_name, 'wb') do |out|
-  out << ['Result'] + csv1.first.keys
+  out << ['Result'] + headers
   comparer.compare(csv2_sorted_file_name) do |action, record|
     stats[action] += 1
     out << [action] + record.values

data/bin/csv-duplicate-finder

@@ -13,31 +13,34 @@ OptionParser.new do |opts|
     exit
   end
 
-  opts.on('-i', '--ignore HEADERS', 'Comman separated list of headers to ignore') do |v|
+  opts.on('-i', '--ignore HEADERS', 'Comma separated list of headers to ignore') do |v|
     options[:ignore_columns] = v.split(',')
   end
 end.parse!
 
 require 'digest/sha2'
-require 'json'
 require 'csv-utils'
 
-csv = CSVUtils::CSVIterator.new(ARGV[0])
+file = ARGV[0] || raise('no CSV file specified')
 
-missing_headers = options[:ignore_columns] - csv.first.keys
+# read bytes as is, so values that aren't valid UTF-8 can still be compared
+csv = CSVUtils::CSVIterator.new(file, {}, 'rb:BINARY')
+
+missing_headers = options[:ignore_columns] - csv.headers
 unless missing_headers.empty?
   raise("unknown headers #{missing_headers.join(', ')} in configured ignore headers")
 end
 
 hashed_rows = {}
 
-csv.each_with_index do |row, idx|
+# rows are keyed by their line number in the file
+csv.each do |row|
   options[:ignore_columns].each do |ignore_column|
     row[ignore_column] = ''
   end
-  key = Digest::SHA256.hexdigest(row.to_json)
+  key = Digest::SHA256.hexdigest(row.values.to_csv)
   hashed_rows[key] ||= []
-  hashed_rows[key] << idx
+  hashed_rows[key] << row.lineno
 end
 
 duplicate_rows = {}
@@ -52,18 +55,18 @@ hashed_rows.each do |key, row_numbers|
   end
 end
 
-csv.each_with_index do |row, idx|
-  next unless (key = duplicate_rows[idx])
+csv.each do |row|
+  next unless (key = duplicate_rows[row.lineno])
 
-  hashed_rows[key][idx] = row
+  hashed_rows[key][row.lineno] = row
 end
 
-CSV.open('duplicates-' + File.basename(ARGV[0]), 'wb') do |out|
-  out << ['duplicate_key', 'row_no'] + csv.first.keys
+CSV.open('duplicates-' + File.basename(file), 'wb:BINARY') do |out|
+  out << ['duplicate_key', 'line_no'] + csv.headers
 
   hashed_rows.each do |key, rows|
-    rows.each do |idx, row|
-      out << [key, idx] + row.values
+    rows.each do |lineno, row|
+      out << [key, lineno] + row.values
     end
   end
 end

data/bin/csv-find-error

@@ -1,19 +1,21 @@
 #!/usr/bin/env ruby
 
 require 'csv'
-require 'shellwords'
+require 'rbconfig'
+
+file = ARGV[0] || abort('Usage: csv-find-error <csv file>')
 
 prev_row = nil
 begin
-  CSV.open(ARGV[0], 'rb').each { |row| prev_row = row }
+  CSV.open(file, 'rb') { |csv| csv.each { |row| prev_row = row } }
 rescue CSV::MalformedCSVError => e
   puts e.class.to_s + ': ' + e.message
   puts "previous row was #{prev_row}"
   if e.message =~ /line (\d+)/
     lineno = $1.to_i
-    cmd = "csv-readline #{Shellwords.escape(ARGV[0])} #{lineno}"
-    puts "running #{cmd}"
-    system(cmd)
+    # run the csv-readline next to this script, with this gem's lib, rather than whichever is on the PATH
+    puts "running csv-readline #{file} #{lineno}"
+    system(RbConfig.ruby, '-I', File.expand_path('../lib', __dir__), File.join(__dir__, 'csv-readline'), file, lineno.to_s)
   end
   exit 1
 end

data/bin/csv-readline

@@ -2,21 +2,10 @@
 
 require 'optparse'
 
-BYTE_ORDER_MARKS = {
-  "\xEF\xBB\xBF".force_encoding('ASCII-8BIT') => 'UTF-8',
-  "\xFE\xFF".force_encoding('ASCII-8BIT') => 'UTF-16',
-  "\xFF\xFE".force_encoding('ASCII-8BIT') => 'UTF-16',
-  "\x00\x00\xFE\xFF".force_encoding('ASCII-8BIT') => 'UTF-32',
-  "\xFF\xFE\x00\x00".force_encoding('ASCII-8BIT') => 'UTF-32'
-}
+require 'csv-utils'
 
 def strip_byte_order_mark(line)
-  BYTE_ORDER_MARKS.keys.detect do |bom|
-    if line =~ /\A#{bom}/
-      return line.sub(/\A#{bom}/, '')
-    end
-  end
-  line
+  CSVUtils::ByteOrderMark.strip(line)
 end
 
 def csv_parse_line(line)
@@ -87,7 +76,7 @@ end
 
 def parse_csv_row(file, lineno, number_of_lines)
   cnt = 1
-  str = ''
+  str = +''
   file.each do |line|
     cnt += 1
     next if cnt < lineno

data/bin/csv-splitter

@@ -15,6 +15,8 @@ OptionParser.new do |opts|
   end
 
   opts.on('-r', '--rows ROWS', Integer, 'Max number of rows per a CSV') do |v|
+    raise OptionParser::InvalidArgument, "--rows must be positive (#{v})" unless v.positive?
+
     options[:rows] = v
   end
 
@@ -36,20 +38,20 @@ def get_split_file_name(file, num_files, total_files)
 end
 
 csv_options = CSVUtils::CSVOptions.new(file)
+# a single column file has no separator to detect
+col_sep = csv_options.col_separator || ','
 
 csv = CSV.open(
   file,
   'rb',
-  col_sep: csv_options.col_separator,
+  col_sep: col_sep,
   liberal_parsing: true
 )
 
 headers = options[:no_header] ? nil : csv.shift
 
 total_rows = 0
-while csv.shift
-  total_rows +=1
-end
+total_rows += 1 while csv.shift
 
 csv.rewind
 csv.shift if headers
@@ -66,7 +68,7 @@ append_row_proc = proc do |row|
     out.close if out
     num_rows = 0
     num_files += 1
-    out = CSV.open(get_split_file_name(file, num_files, total_files), 'wb', col_sep: csv_options.col_separator)
+    out = CSV.open(get_split_file_name(file, num_files, total_files), 'wb', col_sep: col_sep)
     out << headers if headers
   end
 

data/bin/csv-validator

@@ -15,6 +15,8 @@ def utf8?(str)
 end
 
 def convert_to_utf8(str, current_encoding)
+  return nil unless current_encoding
+
   str.force_encoding(current_encoding)
   return nil unless str.valid_encoding?
 
@@ -30,12 +32,13 @@ def strip_bom!(col)
 end
 
 id_column_name = ARGV[1]
-csv = CSV.open(ARGV[0], 'rb')
+# read bytes as is; reading as UTF-8 would raise on the values this script looks for
+csv = CSV.open(ARGV[0] || abort('Usage: csv-validator <csv file> [<id header>]'), 'rb:BINARY')
 out = nil
 
 begin
-  headers = csv.shift
-  strip_bom!(headers[0])
+  headers = csv.shift || []
+  strip_bom!(headers[0]) if headers[0]
 
   id_column_name ||= headers[0]
   unless headers.include?(id_column_name)
@@ -68,9 +71,10 @@ begin
     row.each_with_index do |col, idx|
       next if col.nil? || utf8?(col)
 
-      $stderr.puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: none UTF-8 characters found in \"#{col}\""
-      if (col_utf8_encoded = convert_to_utf8(col, detect_encoding(col)))
-        puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: converted to UTF-8 from #{detect_encoding(col)} \"#{col_utf8_encoded}\""
+      $stderr.puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: non UTF-8 characters found in #{col.inspect}"
+      encoding = detect_encoding(col)
+      if (col_utf8_encoded = convert_to_utf8(col, encoding))
+        puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: converted to UTF-8 from #{encoding} \"#{col_utf8_encoded}\""
         out_proc.call [row[id_column_num], csv_lineno, (idx + 1), headers[idx], col_utf8_encoded]
       else
         $stderr.puts "row(#{csv_lineno}),col(#{idx + 1}) #{headers[idx]}: unknown character encoding"

data/lib/csv-utils.rb

@@ -2,8 +2,10 @@
 
 require 'csv'
 
-# Collection of tools for working with CSV files.
+# Tools for comparing, sorting, transforming and debugging CSV files, including ones too large to load
+# into memory. Each class is autoloaded on first use.
 module CSVUtils
+  autoload :ByteOrderMark, 'csv_utils/byte_order_mark'
   autoload :CSVCompare, 'csv_utils/csv_compare'
   autoload :CSVExtender, 'csv_utils/csv_extender'
   autoload :CSVIterator, 'csv_utils/csv_iterator'

data/lib/csv_utils/byte_order_mark.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module CSVUtils
+  # Detects and strips the byte order mark at the start of a CSV file.
+  module ByteOrderMark
+    # Byte order marks, as binary strings, to the encoding they indicate.
+    # This list is from https://en.wikipedia.org/wiki/Byte_order_mark.
+    # UTF-32 LE starts with the UTF-16 LE mark, so longer marks come first.
+    # @return [Hash{String => String}]
+    ENCODINGS = {
+      (+"\x00\x00\xFE\xFF").force_encoding('ASCII-8BIT').freeze => 'UTF-32',
+      (+"\xFF\xFE\x00\x00").force_encoding('ASCII-8BIT').freeze => 'UTF-32',
+      (+"\xEF\xBB\xBF").force_encoding('ASCII-8BIT').freeze => 'UTF-8',
+      (+"\xFE\xFF").force_encoding('ASCII-8BIT').freeze => 'UTF-16',
+      (+"\xFF\xFE").force_encoding('ASCII-8BIT').freeze => 'UTF-16'
+    }.freeze
+
+    # The byte order mark the string starts with. Compares bytes, so it works whatever the string's encoding is.
+    # @param str [String]
+    # @return [String, nil] the mark as a binary string
+    def self.detect(str)
+      prefix = str.byteslice(0, 4).b
+      ENCODINGS.keys.detect { |bom| prefix.start_with?(bom) }
+    end
+
+    # The string without its leading byte order mark, in its original encoding.
+    # @param str [String]
+    # @return [String]
+    def self.strip(str)
+      bom = detect(str)
+      bom ? str.byteslice(bom.bytesize..) : str
+    end
+  end
+end