csv-utils 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 423eacb18512c595abe83765ccfd4d945f1e2e0d18a6370b6841997eb428ddb7
-  data.tar.gz: 3d2df7f1ee62eaebc25d00281401bf49f0fccdb809ec03cbac90e6bf5ba2fd62
+  metadata.gz: c07e7e0ae21409f1fcac6d5b150e1577146d79a47ba9e13b8fb28d90d50fb9c6
+  data.tar.gz: cb27bd0e1edbfc74747c5107f6d740b86e6cedcd547857d0eff2f836cddd4b9d
 SHA512:
-  metadata.gz: 2f9856120036ae6a87e2e758186221b7c0b48aa8d94365f7bdf4bffdf8f292868e43cf3c378eff7460902d710de2859d5406ac9ef5384c07f9728f2f1bad4ef4
-  data.tar.gz: 9e96e672cb1f4da44ae67de48d3e783cf1d2b180c0d8f4a9706ce1ace07b9ac456a689653dfe84537c0cab71e6c58cdb24e5caeec0bcc86bc73a374615ac4e0a
+  metadata.gz: b7402f247e5cead6fba5259df4465836778026e2abdfd2a40281c22ed21679f60e15b23d913d7bb1bb559345557b31d3e024ab181a11c131d9159a7d64794e10
+  data.tar.gz: 9fd6953261a7e29e2e7676ce770311921b447c899624f105e4d3496ed89018cc515a1b21ecf2f94b36def0a492808105da0953df48680d6beed03776b3d0e6a7

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## [0.7.0](https://github.com/dougyouch/csv-utils/compare/v0.6.0...v0.7.0) (2026-10-05)
+
+
+### ⚠ BREAKING CHANGES
+
+* **encoding:** CSVIterator no longer takes a file mode, files are decoded as UTF-8 whatever the locale, and output is written in the decoded encoding. See UPGRADING.md.
+
+### Features
+
+* **encoding:** read every file with the encoding option and write the decoded encoding ([1a9ec4f](https://github.com/dougyouch/csv-utils/commit/1a9ec4f98cbaadfff7ef89fec0d586cea828cf91))
+* **iterator:** open files per call, build rows without zip, add auto_detect ([34ab89a](https://github.com/dougyouch/csv-utils/commit/34ab89ae49dad76507371d2bdf9a7cd310c2cf67))
+* **options:** detect windows-1252 and read utf-8 regardless of locale ([10eeca6](https://github.com/dougyouch/csv-utils/commit/10eeca69bf094fb8c2cb5b005bc74687bde61169))
+
+
+### Bug Fixes
+
+* **iterator:** strip the byte order mark after rewind and from explicit-header rows ([bf8804b](https://github.com/dougyouch/csv-utils/commit/bf8804b1c2ce16738779f3b9ecc7631c3b29a115))
+
 ## [0.6.0](https://github.com/dougyouch/csv-utils/compare/v0.5.0...v0.6.0) (2026-10-05)
 
 

data/README.md

@@ -6,7 +6,7 @@ Streaming tools for CSV files that are too big or too broken to load into memory
 [![Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/coverage.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
 [![Branch Coverage](https://raw.githubusercontent.com/dougyouch/csv-utils/badges/branches.svg)](https://github.com/dougyouch/csv-utils/actions/workflows/ci.yml)
 
-[API reference](https://rubydoc.info/gems/csv-utils) · [Changelog](CHANGELOG.md) · [Architecture](ARCHITECTURE.md)
+[API reference](https://rubydoc.info/gems/csv-utils) · [Changelog](CHANGELOG.md) · [Upgrading](UPGRADING.md) · [Architecture](ARCHITECTURE.md)
 
 ## Features
 
@@ -19,6 +19,7 @@ Streaming tools for CSV files that are too big or too broken to load into memory
 - **CSV Iteration**: Efficient iteration over CSV files with batch support
 - **CSV Extension**: Extend CSV files with additional columns
 - **CSV Options**: Auto-detect CSV file properties (separators, encoding, BOM)
+- **Encodings**: Files are read as UTF-8 whatever the locale, or in the encoding you pass or `CSVOptions` detects
 - **Byte Order Marks**: UTF-8, UTF-16 and UTF-32 byte order marks are stripped from the first header
 - **CLI Tools**: Command-line utilities for CSV debugging and manipulation
 
@@ -232,6 +233,14 @@ iterator.each.with_index { |row, idx| puts "#{idx}: #{row['name']}" }
 
 Line numbers count the header row as line 1, so they match what an editor shows. `headers` returns `[]` for an empty file.
 
+Given a path, the iterator opens the file for each call (`each`, `headers`, `size`, ...) and closes it when the call returns, even when the block raises or stops early, so an idle iterator holds no file handle. A CSV object passed in is rewound, never closed.
+
+Let `CSVOptions` work out the separators and encoding:
+
+```ruby
+iterator = CSVUtils::CSVIterator.auto_detect('export.tsv')
+```
+
 ### Matching CSV Rows
 
 Filter CSV rows using regex patterns:
@@ -295,11 +304,19 @@ options = CSVUtils::CSVOptions.new('data.csv')
 options.valid?         # true if separators detected
 options.col_separator  # detected column separator
 options.row_separator  # detected row separator
-options.encoding       # detected encoding (UTF-8, UTF-16, UTF-32)
+options.encoding       # detected encoding (UTF-8, UTF-16, UTF-32, Windows-1252, ISO-8859-1)
 options.columns        # number of columns
 options.byte_order_mark # BOM if present
+
+# Open the file with what was detected, with CSV or any class in this gem
+CSV.open('data.csv', 'rb', **options.to_csv_options)
+CSVUtils::CSVSort.new('data.csv', 'sorted.csv', true, options.to_csv_options).sort
 ```
 
+`to_csv_options` returns the detected `col_sep` and `row_sep`, leaving out any that weren't found, and an `encoding` that reads values as UTF-8 strings: `'bom|utf-8'` for UTF-8 files, `'BOM|UTF-16LE:UTF-8'` and the like for files with a UTF-16 or UTF-32 byte order mark (CSV detects the row separator for those), and `'Windows-1252:UTF-8'` for files that aren't valid UTF-8.
+
+Without a byte order mark, the encoding comes from the first megabyte (`CSVOptions::SAMPLE_SIZE`): valid UTF-8 is `UTF-8`, anything else is `Windows-1252`, the usual encoding of Excel exports, or `ISO-8859-1` when the sample has one of the five bytes Windows-1252 leaves undefined. A file whose first non-UTF-8 byte comes after the sample is still read as UTF-8.
+
 Supported column separators: `\x02`, `\t`, `|`, `,` (the first one found in the header line wins)
 Supported row separators: `\r\n`, `\n`, `\r`
 
@@ -307,15 +324,20 @@ Headers are parsed as a CSV row, so a quoted header like `"Last, First"` counts
 
 ### Encodings and Byte Order Marks
 
-Files are opened with mode `'rb'`. With csv 3.3 and later, when Ruby's default external encoding is UTF-8 (the usual case), CSV reads `'rb'` files as UTF-8 and raises `CSV::InvalidEncodingError` on bytes that aren't valid UTF-8. To read a file's bytes as is, pass an explicit encoding:
+Every class handles encodings the same way:
+
+- **Reading** uses the `encoding:` CSV option, `'bom|utf-8'` by default, so values are UTF-8 strings whatever the locale (including `LANG=C` in containers and cron jobs). Bytes that aren't valid UTF-8 raise `CSV::InvalidEncodingError`.
+- **Writing** (`CSVExtender`, `CSVTransformer`, `CSVSort`) uses the encoding values were decoded to, so a file read with `'Windows-1252:UTF-8'` is written as UTF-8. `CSVReport` writes UTF-8 unless you pass `encoding:`.
+- **File modes never carry an encoding.** CSV raises `ArgumentError: encoding specified twice` when the mode and the `encoding:` option both have one.
 
 ```ruby
-iterator = CSVUtils::CSVIterator.new('latin1.csv', {}, 'rb:BINARY')
+iterator = CSVUtils::CSVIterator.new('export.csv', encoding: 'Windows-1252:UTF-8')
+iterator = CSVUtils::CSVIterator.new('latin1.csv', encoding: 'BINARY') # raw bytes
 comparator = CSVUtils::CSVCompare.new('primary.csv', ['updated_at'], encoding: 'BINARY') { |src, dest| src['id'] <=> dest['id'] }
-sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true, encoding: 'BINARY')
+sorter = CSVUtils::CSVSort.new('input.csv', 'output.csv', true, CSVUtils::CSVOptions.new('input.csv').to_csv_options)
 ```
 
-`CSVSort`, `CSVExtender` and `CSVTransformer` take CSV options as their last argument and `CSVIterator` takes them second; they're passed to `CSV.open` for file paths.
+`CSVSort`, `CSVExtender`, `CSVTransformer`, `CSVCompare` and `CSVReport` take CSV options as their last argument and `CSVIterator` takes them second; they're passed to `CSV.open` for file paths. `CSVUtils::EncodingOptions` holds the rules above. Upgrading from 0.6? See [UPGRADING.md](UPGRADING.md).
 
 `CSVUtils::ByteOrderMark` detects and strips UTF-8, UTF-16 and UTF-32 byte order marks; `CSVIterator`, `CSVCompare` and `CSVOptions` use it to clean the first header.
 

data/bin/csv-duplicate-finder

@@ -24,7 +24,7 @@ require 'csv-utils'
 file = ARGV[0] || raise('no CSV file specified')
 
 # read bytes as is, so values that aren't valid UTF-8 can still be compared
-csv = CSVUtils::CSVIterator.new(file, {}, 'rb:BINARY')
+csv = CSVUtils::CSVIterator.new(file, encoding: 'BINARY')
 
 missing_headers = options[:ignore_columns] - csv.headers
 unless missing_headers.empty?

data/lib/csv-utils.rb

@@ -6,6 +6,8 @@ require 'csv'
 # into memory. Each class is autoloaded on first use.
 module CSVUtils
   autoload :ByteOrderMark, 'csv_utils/byte_order_mark'
+  autoload :CharacterEncoding, 'csv_utils/character_encoding'
+  autoload :EncodingOptions, 'csv_utils/encoding_options'
   autoload :CSVCompare, 'csv_utils/csv_compare'
   autoload :CSVExtender, 'csv_utils/csv_extender'
   autoload :CSVIterator, 'csv_utils/csv_iterator'

data/lib/csv_utils/character_encoding.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+module CSVUtils
+  # Guesses the character encoding of a file without a byte order mark from a sample of its bytes.
+  module CharacterEncoding
+    # Bytes Windows-1252 leaves undefined; converting them to UTF-8 raises.
+    # @return [Regexp]
+    WINDOWS_1252_UNDEFINED = /[\x81\x8D\x8F\x90\x9D]/n
+
+    # 'UTF-8' when the sample is valid UTF-8. Otherwise 'Windows-1252', the usual encoding of Excel exports,
+    # or 'ISO-8859-1' when the sample has a byte Windows-1252 leaves undefined, since every ISO-8859-1 byte
+    # converts to UTF-8.
+    # @param sample [String] bytes from the start of the file, ending on a whole character
+    # @return [String]
+    def self.detect(sample)
+      bytes = sample.b
+      return 'UTF-8' if bytes.dup.force_encoding(Encoding::UTF_8).valid_encoding?
+      return 'ISO-8859-1' if bytes.match?(WINDOWS_1252_UNDEFINED)
+
+      'Windows-1252'
+    end
+  end
+end

data/lib/csv_utils/csv_compare.rb

@@ -24,7 +24,7 @@ module CSVUtils
     # @param update_comparison_columns [Array<String>, nil] columns compared on matching records;
     #   without them no updates are yielded
     # @param csv_options [Hash] options passed to CSV.open for both files, ex: col_sep: "\t",
-    #   or encoding: 'BINARY' to read bytes that aren't valid UTF-8
+    #   or encoding: 'BINARY' to read bytes that aren't valid UTF-8; the :encoding defaults to 'bom|utf-8'
     # @yieldparam src [Hash{String => String}] record from the primary file
     # @yieldparam dest [Hash{String => String}] record from the secondary file
     # @yieldreturn [Integer] negative, 0 or positive like <=>, using the same order both files are sorted by
@@ -88,7 +88,7 @@ module CSVUtils
     end
 
     def open_csv(file)
-      csv = CSV.open(file, 'rb', **csv_options)
+      csv = CSV.open(file, 'rb', **EncodingOptions.read(csv_options))
       yield csv, read_headers(csv)
     ensure
       csv&.close

data/lib/csv_utils/csv_extender.rb

@@ -9,10 +9,11 @@ module CSVUtils
   class CSVExtender
     # @param src_csv [String, CSV, #shift] path or CSV to read; paths are opened and closed by the extender
     # @param dest_csv [String, CSV, #<<] path or CSV to write
-    # @param csv_options [Hash] options passed to CSV.open for paths
+    # @param csv_options [Hash] options passed to CSV.open for paths; the source is read with its :encoding,
+    #   'bom|utf-8' by default, and the output written in the encoding values were decoded to (see {EncodingOptions})
     def initialize(src_csv, dest_csv, csv_options = {})
-      @src_csv = CSVUtils::CSVWrapper.new(src_csv, 'rb', csv_options)
-      @dest_csv = CSVUtils::CSVWrapper.new(dest_csv, 'wb', csv_options)
+      @src_csv = CSVUtils::CSVWrapper.new(src_csv, 'rb', EncodingOptions.read(csv_options))
+      @dest_csv = CSVUtils::CSVWrapper.new(dest_csv, 'wb', EncodingOptions.write(csv_options))
     end
 
     # Appends columns one row at a time, then closes the files it opened.

data/lib/csv_utils/csv_iterator.rb

@@ -26,8 +26,15 @@ module CSVUtils
       # @param lineno [Integer]
       # @return [RowWrapper]
       def self.create(headers, row, lineno)
-        row_wrapper = RowWrapper[headers.zip(row)]
+        row_wrapper = new
         row_wrapper.lineno = lineno
+        # an index loop instead of headers.zip(row), which allocates an array per column for every row
+        idx = 0
+        size = headers.size
+        while idx < size
+          row_wrapper[headers[idx]] = row[idx]
+          idx += 1
+        end
         row_wrapper
       end
 
@@ -41,41 +48,40 @@ module CSVUtils
       end
     end
 
-    # @param src_csv [String, CSV] path or CSV to read; it must support rewind
-    # @param csv_options [Hash] options passed to CSV.open for a path
-    # @param mode [String] file mode for a path; 'rb:BINARY' reads bytes as is
-    def initialize(src_csv, csv_options = {}, mode = 'rb')
-      @src_csv = CSVUtils::CSVWrapper.new(src_csv, mode, csv_options)
+    # An iterator for a path, with the separators and encoding detected by {CSVOptions}.
+    # @example
+    #   CSVUtils::CSVIterator.auto_detect('export.tsv').each { |row| puts row['name'] }
+    # @param path [String]
+    # @return [CSVIterator]
+    def self.auto_detect(path)
+      options = CSVOptions.new(path)
+      new(path, options.to_csv_options)
+    end
+
+    # A path is opened by each method call and closed when the call returns,
+    # so an idle iterator holds no file handle.
+    # @param src_csv [String, CSV] path or CSV to read; a CSV must support rewind and is never closed
+    # @param csv_options [Hash] options passed to CSV.open for a path. The :encoding defaults to 'bom|utf-8'
+    #   (see {EncodingOptions}); 'BINARY' reads bytes as is and 'Windows-1252:UTF-8' converts.
+    def initialize(src_csv, csv_options = {})
+      @src_csv = src_csv
+      @csv_options = EncodingOptions.read(csv_options)
     end
 
     # Yields each row from the start of the file.
     # @param headers [Array<String>, nil] headers to use; by default the first row is read as the headers
     # @yieldparam row [RowWrapper]
     # @return [Enumerator, nil] an enumerator without a block
-    def each(headers = nil)
+    def each(headers = nil, &)
       return enum_for(:each, headers) unless block_given?
 
-      @src_csv.rewind
-
-      lineno = 0
-      unless headers
-        headers = read_headers
-        lineno += 1
-      end
-
-      @prev_row = nil
-      while (row = @src_csv.shift)
-        lineno += 1
-        yield RowWrapper.create(headers, row, lineno)
-        @prev_row = row
-      end
+      open_csv { |csv| each_row(csv, headers, &) }
     end
 
     # The first row, without the byte order mark.
     # @return [Array<String>] empty for an empty file
     def headers
-      @src_csv.rewind
-      read_headers
+      open_csv { |csv| read_headers(csv) }
     end
 
     # Builds a lookup hash from every row.
@@ -85,8 +91,9 @@ module CSVUtils
     # @return [Hash]
     # @raise [RuntimeError] when key or value isn't a header
     def to_hash(key, value = nil, &)
-      raise("header #{key} not found in #{headers}") unless headers.include?(key)
-      raise("headers #{value} not found in #{headers}") if value && !headers.include?(value)
+      file_headers = headers
+      raise("header #{key} not found in #{file_headers}") unless file_headers.include?(key)
+      raise("headers #{value} not found in #{file_headers}") if value && !file_headers.include?(value)
 
       value_proc =
         if value
@@ -103,11 +110,12 @@ module CSVUtils
     # Number of rows after the header row.
     # @return [Integer]
     def size
-      @src_csv.rewind
-      @src_csv.shift
-      cnt = 0
-      cnt += 1 while @src_csv.shift
-      cnt
+      open_csv do |csv|
+        csv.shift
+        cnt = 0
+        cnt += 1 while csv.shift
+        cnt
+      end
     end
 
     # Yields the rows in batches.
@@ -134,11 +142,43 @@ module CSVUtils
 
     private
 
+    # Opens a path for the duration of the block. Only a CSV that was passed in is rewound:
+    # rewinding goes back to byte 0, in front of a byte order mark the 'bom|' encoding skipped on open.
+    def open_csv
+      CSVWrapper.open(@src_csv, 'rb', @csv_options) do |csv|
+        csv.rewind unless @src_csv.is_a?(String)
+        yield csv
+      end
+    end
+
+    def each_row(csv, headers)
+      @prev_row = nil
+      row = shift_first_row(csv)
+      lineno = 1
+      unless headers
+        headers = row || []
+        row = csv.shift
+        lineno += 1
+      end
+
+      while row
+        yield RowWrapper.create(headers, row, lineno)
+        @prev_row = row
+        row = csv.shift
+        lineno += 1
+      end
+    end
+
     # an empty file has no headers, and an empty first header cell is nil
-    def read_headers
-      headers = @src_csv.shift || []
-      headers[0] = ByteOrderMark.strip(headers[0]) if headers[0]
-      headers
+    def read_headers(csv)
+      shift_first_row(csv) || []
+    end
+
+    # the first row without the byte order mark, whether it's read as headers or as data
+    def shift_first_row(csv)
+      row = csv.shift
+      row[0] = ByteOrderMark.strip(row[0]) if row && row[0]
+      row
     end
   end
 end

data/lib/csv_utils/csv_options.rb

@@ -25,25 +25,40 @@ module CSVUtils
       "\r"
     ].freeze
 
+    # @return [Hash{String => String}] UTF-16 and UTF-32 byte order marks, as binary strings, to the :encoding
+    #   option that decodes them to UTF-8
+    WIDE_ENCODINGS = {
+      (+"\x00\x00\xFE\xFF").force_encoding('ASCII-8BIT').freeze => 'BOM|UTF-32BE:UTF-8',
+      (+"\xFF\xFE\x00\x00").force_encoding('ASCII-8BIT').freeze => 'BOM|UTF-32LE:UTF-8',
+      (+"\xFE\xFF").force_encoding('ASCII-8BIT').freeze => 'BOM|UTF-16BE:UTF-8',
+      (+"\xFF\xFE").force_encoding('ASCII-8BIT').freeze => 'BOM|UTF-16LE:UTF-8'
+    }.freeze
+
+    # @return [Integer] bytes read to detect the encoding of a file without a byte order mark
+    SAMPLE_SIZE = 1024 * 1024
+
     # @return [Integer, nil] number of headers, nil without a column separator
     attr_reader :columns
     # @return [String, nil] the byte order mark the file starts with, as a binary string
     attr_reader :byte_order_mark
-    # @return [String] 'UTF-8', 'UTF-16' or 'UTF-32', from the byte order mark; 'UTF-8' without one
+    # @return [String] 'UTF-8', 'UTF-16' or 'UTF-32' from the byte order mark; without one, from the bytes of
+    #   the first {SAMPLE_SIZE} bytes: 'UTF-8', 'Windows-1252' or 'ISO-8859-1' (see {CharacterEncoding.detect})
     attr_reader :encoding
     # @return [String, nil] one of {COL_SEPARATORS}
     attr_reader :col_separator
     # @return [String, nil] one of {ROW_SEPARATORS}
     attr_reader :row_separator
 
-    # @param io [String, IO] path of the file, or an IO positioned at its first line
+    # @param io [String, IO] path of the file, or an IO positioned at its first line; up to {SAMPLE_SIZE}
+    #   bytes are read from it
     def initialize(io)
-      line = read_first_line(io)
+      sample = read_sample(io)
+      line = first_line(sample)
 
       @col_separator = auto_detect_col_sep(line)
       @row_separator = auto_detect_row_sep(line)
       @byte_order_mark = get_byte_order_mark(line)
-      @encoding = get_character_encoding(@byte_order_mark)
+      @encoding = get_character_encoding(@byte_order_mark, sample)
       @columns = get_number_of_columns(line) if @col_separator
     end
 
@@ -55,6 +70,21 @@ module CSVUtils
       true
     end
 
+    # Options for CSV.open, or any class in this library, with the detected separators and an :encoding that
+    # reads values as UTF-8 strings: 'bom|utf-8', 'Windows-1252:UTF-8', 'BOM|UTF-16LE:UTF-8' and so on.
+    # A separator that wasn't detected is left to CSV, and so is the row separator of UTF-16 and UTF-32 files,
+    # since it's detected from raw bytes.
+    # @example
+    #   CSV.open(path, 'rb', **options.to_csv_options)
+    #   CSVUtils::CSVSort.new(path, 'sorted.csv', true, options.to_csv_options).sort
+    # @return [Hash{Symbol => String}]
+    def to_csv_options
+      options = { encoding: encoding_option }
+      options[:col_sep] = col_separator if col_separator
+      options[:row_sep] = row_separator if row_separator && !wide_encoding?
+      options
+    end
+
     # @api private
     # @param line [String]
     # @return [String, nil]
@@ -96,9 +126,10 @@ module CSVUtils
 
     # @api private
     # @param bom [String, nil]
+    # @param sample [String] bytes from the start of the file
     # @return [String]
-    def get_character_encoding(bom)
-      BYTE_ORDER_MARKS[bom] || 'UTF-8'
+    def get_character_encoding(bom, sample)
+      BYTE_ORDER_MARKS[bom] || CharacterEncoding.detect(sample)
     end
 
     # @api private
@@ -110,10 +141,28 @@ module CSVUtils
 
     private
 
-    # an empty file reads as an empty line, which makes the options invalid
-    def read_first_line(io)
-      line = io.is_a?(String) ? File.open(io, 'rb', &:gets) : io.gets
-      line || ''
+    # an empty file reads as an empty sample, which makes the options invalid
+    def read_sample(io)
+      sample = (io.is_a?(String) ? File.open(io, 'rb') { |file| file.read(SAMPLE_SIZE) } : io.read(SAMPLE_SIZE)) || ''
+      sample.bytesize < SAMPLE_SIZE ? sample : drop_last_multibyte_char(sample)
+    end
+
+    # a full sample can end mid-character, so its last multibyte character, whole or not, is dropped
+    def drop_last_multibyte_char(sample)
+      sample.sub(/[\xC0-\xFF][\x80-\xBF]*\z/n, '')
+    end
+
+    def first_line(sample)
+      line_end = sample.index("\n")
+      line_end ? sample.byteslice(0, line_end + 1) : sample
+    end
+
+    def wide_encoding?
+      WIDE_ENCODINGS.key?(byte_order_mark)
+    end
+
+    def encoding_option
+      WIDE_ENCODINGS.fetch(byte_order_mark) { encoding == 'UTF-8' ? EncodingOptions::DEFAULT_ENCODING : "#{encoding}:UTF-8" }
     end
 
     def header_parse_options

data/lib/csv_utils/csv_report.rb

@@ -15,13 +15,14 @@ module CSVUtils
 
     # @param csv [String, CSV, #<<] path to write, opened in 'wb' mode by default, or a CSV to append to
     # @param headers [Array<String>, Class, nil] header row, or a {CSVRow} class to take the headers from
-    # @param csv_options [Hash] options passed to CSV.open for a path; :mode overrides the file mode
+    # @param csv_options [Hash] options passed to CSV.open for a path; :mode overrides the file mode and
+    #   :encoding, UTF-8 by default, is the encoding of the file written
     # @yieldparam report [CSVReport] when given, the block is passed to {#generate}
     def initialize(csv, headers = nil, csv_options = {}, &block)
       @csv =
         if csv.is_a?(String)
           @must_close = true
-          opts = csv_options.dup
+          opts = EncodingOptions.write(csv_options)
           mode = opts.delete(:mode) || 'wb'
           CSV.open(csv, mode, **opts)
         else

data/lib/csv_utils/csv_sort.rb

@@ -23,7 +23,8 @@ module CSVUtils
     # @param csv_file [String] path of the file to sort
     # @param new_csv_file [String] path of the sorted file to write
     # @param has_headers [Boolean] whether the first row is a header row
-    # @param csv_options [Hash] options passed to CSV.open
+    # @param csv_options [Hash] options passed to CSV.open; the file is read with its :encoding, 'bom|utf-8' by
+    #   default, and the sorted file written in the encoding values were decoded to (see {EncodingOptions})
     def initialize(csv_file, new_csv_file, has_headers = true, csv_options = {})
       @csv_file = csv_file
       @new_csv_file = new_csv_file
@@ -51,11 +52,12 @@ module CSVUtils
 
     # rubocop:disable-next Metrics/MethodLength
     def merge_sort_csv_files(src_csv_file1, src_csv_file2, dest_csv_file)
-      src1 = CSV.open(src_csv_file1, 'rb', **csv_options)
+      # part files were written with the write options, so reading them back with those changes nothing
+      src1 = CSV.open(src_csv_file1, 'rb', **write_options)
       begin
-        src2 = CSV.open(src_csv_file2, 'rb', **csv_options)
+        src2 = CSV.open(src_csv_file2, 'rb', **write_options)
         begin
-          dest = CSV.open(dest_csv_file, 'wb', **csv_options)
+          dest = CSV.open(dest_csv_file, 'wb', **write_options)
           begin
             if @headers
               dest << @headers
@@ -99,7 +101,7 @@ module CSVUtils
     end
 
     def create_sorted_csv_part_files(batch_size, &block)
-      src = CSV.open(csv_file, 'rb', **csv_options)
+      src = CSV.open(csv_file, 'rb', **EncodingOptions.read(csv_options))
       begin
         @headers = src.shift if has_headers
 
@@ -107,7 +109,7 @@ module CSVUtils
         create_batch_part_proc = proc do
           batch.sort!(&block)
           @csv_part_files << "#{new_csv_file}.part.#{@csv_part_files.size}"
-          CSV.open(@csv_part_files.last, 'wb', **csv_options) do |csv|
+          CSV.open(@csv_part_files.last, 'wb', **write_options) do |csv|
             csv << @headers if @headers
             batch.each { |row| csv << row }
           end
@@ -143,10 +145,21 @@ module CSVUtils
       if @csv_part_files.size.positive?
         FileUtils.mv(@csv_part_files.pop, new_csv_file)
       else
-        FileUtils.cp(@csv_file, new_csv_file)
+        write_headers_only
       end
     end
 
+    # a file without rows is written rather than copied, so it's in the same encoding as a sorted one
+    def write_headers_only
+      CSV.open(new_csv_file, 'wb', **write_options) do |csv|
+        csv << @headers if @headers
+      end
+    end
+
+    def write_options
+      EncodingOptions.write(csv_options)
+    end
+
     # removes the temporary files left behind when sorting fails part way through
     def delete_csv_part_files
       @csv_part_files.each { |file| FileUtils.rm_f(file) }

data/lib/csv_utils/csv_transformer.rb

@@ -19,10 +19,11 @@ module CSVUtils
 
     # @param src_csv [String, CSV, #shift] path or CSV to read; paths are opened and closed by the transformer
     # @param dest_csv [String, CSV, #<<] path or CSV to write
-    # @param csv_options [Hash] options passed to CSV.open for paths
+    # @param csv_options [Hash] options passed to CSV.open for paths; the source is read with its :encoding,
+    #   'bom|utf-8' by default, and the output written in the encoding values were decoded to (see {EncodingOptions})
     def initialize(src_csv, dest_csv, csv_options = {})
-      @src_csv = CSVUtils::CSVWrapper.new(src_csv, 'rb', csv_options)
-      @dest_csv = CSVUtils::CSVWrapper.new(dest_csv, 'wb', csv_options)
+      @src_csv = CSVUtils::CSVWrapper.new(src_csv, 'rb', EncodingOptions.read(csv_options))
+      @dest_csv = CSVUtils::CSVWrapper.new(dest_csv, 'wb', EncodingOptions.write(csv_options))
     end
 
     # Reads the first row as the headers. Without it, every row is data and the output has no header row.

data/lib/csv_utils/encoding_options.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module CSVUtils
+  # The one place the library decides how files are decoded and encoded. Every class reads with the
+  # :encoding CSV option, 'bom|utf-8' by default, so values are UTF-8 strings whatever the locale, and
+  # writes in the encoding the values were decoded to, so reading 'Windows-1252:UTF-8' writes UTF-8.
+  # File modes never carry an encoding: CSV raises when the mode and the :encoding option both have one.
+  #
+  # @example
+  #   CSV.open(path, 'rb', **CSVUtils::EncodingOptions.read(col_sep: "\t"))
+  module EncodingOptions
+    # @return [String] the :encoding option used for reading when none is given
+    DEFAULT_ENCODING = 'bom|utf-8'
+
+    # Options for reading: the given options with the default :encoding when they have none.
+    # @param csv_options [Hash]
+    # @return [Hash]
+    def self.read(csv_options)
+      { encoding: DEFAULT_ENCODING }.merge(csv_options)
+    end
+
+    # Options for writing values read with csv_options: the :encoding is the one values were decoded to.
+    # @param csv_options [Hash]
+    # @return [Hash]
+    def self.write(csv_options)
+      csv_options.merge(encoding: decoded_encoding(read(csv_options)[:encoding]))
+    end
+
+    # The encoding of strings read with an :encoding option: the internal encoding of 'external:internal',
+    # otherwise the external one, without a 'bom|' prefix.
+    # @example
+    #   CSVUtils::EncodingOptions.decoded_encoding('Windows-1252:UTF-8') # => 'UTF-8'
+    #   CSVUtils::EncodingOptions.decoded_encoding('bom|utf-8') # => 'utf-8'
+    # @param encoding [String, Encoding]
+    # @return [String]
+    def self.decoded_encoding(encoding)
+      encoding.to_s.split(':').last.sub(/\Abom\|/i, '')
+    end
+  end
+end

data/lib/csv_utils/version.rb

@@ -2,5 +2,5 @@
 
 module CSVUtils
   # Gem version, bumped by release-please
-  VERSION = '0.6.0'
+  VERSION = '0.7.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: csv-utils
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - Doug Youch
@@ -76,6 +76,7 @@ files:
 - bin/csv-validator
 - lib/csv-utils.rb
 - lib/csv_utils/byte_order_mark.rb
+- lib/csv_utils/character_encoding.rb
 - lib/csv_utils/csv_compare.rb
 - lib/csv_utils/csv_extender.rb
 - lib/csv_utils/csv_iterator.rb
@@ -86,6 +87,7 @@ files:
 - lib/csv_utils/csv_sort.rb
 - lib/csv_utils/csv_transformer.rb
 - lib/csv_utils/csv_wrapper.rb
+- lib/csv_utils/encoding_options.rb
 - lib/csv_utils/version.rb
 homepage: https://github.com/dougyouch/csv-utils
 licenses: