herringbone 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b6e9b4e12f2b83688e6ea7be2752dd389cd1a338489834160efb0df5f796916e
-  data.tar.gz: c69303ac028259092e2f22c0f5d635f0e428ae1950c1b4b42d4a6379fd6a5a87
+  metadata.gz: 826898400d423a36ccd89740d408da3d367726cf84c6cf33c4067a3c54b8861d
+  data.tar.gz: 0667170a3ede8376299aff04b4081259533c8877f9823ad8fcc9aee916a86650
 SHA512:
-  metadata.gz: 00a7d9246cb73211dcd5c46a5b0dab15b20b0fe1dc0026a2f02dcf3a9e13a11c91b8632bf1891fdd2f5a8deccf42f1c46918ee26ed9bf881208338f731cd12b6
-  data.tar.gz: 91edc03478f1d9aa24f4d40527b8d2ac8cbed53047e3870e2b17d52ba252bb29a354035f5f072105adf0122bcf93e24c424747552423f9f7ba345fedb98b4219
+  metadata.gz: 21984e5695bf75e086fa091a4c69456edd4fc7ef21ea2d73c89e0ad3de1316238d2f66cf9d8c86183c5e962a1ce965aaeac8036dc1a5c1a43aa6a3b85c983405
+  data.tar.gz: 9377178bd66028014087cc55ed255c4abc2ed1c0217cdfbd86d5f9fe647a4ac0d741a973415f393c31fc7af83d653ce14df33d7a7629dee69aaed554e1fa5719

data/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+## Unreleased
+
+## 0.4.0
+
+- herringbone is now Ractor-enabled! Reading and writing work from any Ractor on Ruby 3.1+, and schemas can be passed to Ractors with
+- **Breaking:** the optional gems (`zstd-ruby`, `brotli`, `snappy`, `xxhash`, `numo-narray-alt`)
+  are used only when loaded: require them (`Bundler.require` does) instead of relying on
+  Herringbone to require them on first use.
+- Files with LZO-compressed pages (written by parquet-mr with hadoop-lzo) can be read. Writing
+  LZO is not supported.
+- `Herringbone::SimpleWriter`: CSV-style writing with `headers!` and Array rows, types inferred.
+- `Herringbone.write` without `schema:` reads the rows once, holding back only the first 1000 for
+  inference, so cursors and one-shot Enumerators no longer lose rows.
+- `Herringbone.write(io, rows) { json :payload }` declares fields that replace inferred ones.
+- `Herringbone::SchemaMismatch` (an `EncodeError`) explains a row that doesn't fit an inferred
+  schema and stops the write; `EncodeError` has `row`, `column` and `value`.
+- `herringbone inspect --format=html` opens the page in the browser when run in a terminal.
+  `Ractor.make_shareable(schema)`.
+- **Breaking:** `herringbone inspect` takes `--format=text|json|html` instead of `--json`/`--html`.
+
+## 0.3.0
+
+- `read(as: :numo)` and `each_batch(as: :numo)` return Numo arrays (optional `numo-narray-alt`).
+- The `snappy` gem is used for Snappy when installed: 27x faster compression, 12x faster decompression.
+- Writes are forward-only, so files can be streamed into S3 multipart uploads.
+- YARD documentation for the whole library.
+
+## 0.2.0
+
+- Streaming reads: pages are decoded incrementally and rows assembled in batches.
+- `each_batch(as: :columns)` and `read(as: :columns)` for column-order reads.
+- `where:`, `from:` and `limit:` skip row groups and pages using statistics, bloom filters and the
+  page index.
+- Page indexes (ColumnIndex and OffsetIndex) are written for every column chunk.
+- Bloom filters are written and read, using the `xxhash` gem when it is installed.
+- `Herringbone::Inspector`, an HTML visualizer and `herringbone inspect`: page headers, page
+  indexes, CRC verification and the decoded ARROW:schema, without decompressing any data.
+- `Herringbone.write` takes ActiveRecord models and relations.
+- `zstd-ruby` and `brotli` are optional; a missing codec raises `MissingCodecError`.
+- **Breaking:** the public API was simplified: `Reader.new(io)` and `Writer.open(io, schema)`
+  only take IOs, Hash schemas and `Herringbone.export`/`open`/`read` are gone, and `DecodeError`
+  is now `FormatError`.
+
+## 0.1.0
+
+- Pure-Ruby Parquet reader and writer, with no Thrift gem or native extensions.
+- Full nesting support (structs, lists, maps) via Dremel shredding and assembly.
+- All standard encodings, v1 and v2 data pages, Snappy, LZ4 and GZIP in Ruby, ZSTD and Brotli via gems.
+- Schemas from a block DSL, inferred from rows, or built from ActiveRecord models.
+- Writer coercions for Rails values, byte-based row groups and compact buffers.

data/README.md

@@ -3,30 +3,71 @@
 A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/) files.
 
 - No Thrift gem and no native extensions of its own: the Thrift compact protocol, all encodings
-  and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems)
+  and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems).
+  LZO, from Hadoop-era files, can be read but not written
 - Full nesting support (structs, lists, maps, any depth) via Dremel record shredding/assembly
 - Reads files from parquet-mr, Arrow, Spark, Impala, DuckDB, Rust writers etc.
+- Optimized reads with batches and pages 
 - Ruby 3.0+
 
+## Diving in: dumping records in Rails
+
+Stream a relation straight into S3, no temp file needed:
+
+```ruby
+s3 = Aws::S3::TransferManager.new
+s3.upload_stream(bucket: "exports", key: "payments.parquet") do |io|
+  # Schema will be auto-inferred, find_each will be used automatically
+  Herringbone.write(io, Payment.where(status: "settled", created_at: 1.month.ago..))
+end
+```
+
 ## Installation
 
 ```ruby
 gem "herringbone"
-gem "zstd-ruby" # optional: ZSTD (faster writes and smaller files than the default Snappy)
-gem "brotli"    # optional: Brotli
-gem "xxhash"    # optional: faster bloom filters
 ```
 
-The only dependency is `bigdecimal`. Snappy, LZ4 and GZIP always work. `Herringbone.codecs` lists
-the codecs this process can use, e.g. `[:none, :snappy, :gzip, :lz4, :lz4_hadoop, :zstd]`. Using
-a missing one raises `Herringbone::MissingCodecError` naming the gem to add: a writer raises it
-before writing anything, a reader when it reaches the first such page (the schema and metadata
-are still readable). LZO is not supported.
+Add these libraries to speed up and enable certain functionality:
+
+```ruby
+gem "snappy"          # native Snappy, 2-3x faster reads and writes of typical files
+gem "zstd-ruby"       # ZSTD
+gem "brotli"          # Brotli
+gem "xxhash"          # faster bloom filters
+gem "numo-narray-alt" # read(as: :numo)
+```
+
+Herringbone never requires them itself: it uses whichever are loaded. Bundler.require (as in Rails)
+loads them, otherwise require them yourself:
+
+```ruby
+require "herringbone"
+require "zstd-ruby"
+require "numo/narray"
+```
+
+On Ruby 3.1 and later, Herringbone works from any Ractor. Pass a schema to a Ractor with
+`Ractor.make_shareable(schema)`. Rows made shareable the same way reach a writing Ractor by
+reference, without being copied:
+
+```ruby
+writer = Ractor.new("out.parquet", Ractor.make_shareable(schema)) do |path, schema|
+  File.open(path, "wb") do |f|
+    Herringbone::Writer.open(f, schema) do |w|
+      while (row = Ractor.receive) != :done
+        w << row
+      end
+    end
+  end
+end
+rows.each { |row| writer.send(Ractor.make_shareable(row)) }
+writer.send(:done)
+```
 
 ## Reading
 
-Herringbone never opens files by path: readers take a random-access IO (a `File` opened with
-`"rb"`, `StringIO`, `Tempfile`...), which stays open and belongs to the caller.
+Herringbone can read from any IO-ish object with random access (the IO should be seekable).
 
 ```ruby
 require "herringbone"
@@ -76,6 +117,34 @@ are exact. On a 1M-row file with 20k-row pages, looking up one `id` takes 0.1 s
 Filters help most on columns the data is sorted or clustered by. `reader.scan_plan(where: ...)`
 shows which row groups and row ranges a read would touch, without reading them.
 
+### Numo arrays
+
+`as: :numo` returns (or yields, with `each_batch`) a Hash of column name => Numo array, and takes
+the same `columns:`, `where:`, `from:` and `limit:`. Add `gem "numo-narray-alt"` (or
+`numo-narray`) to your Gemfile and `require "numo/narray"`: without it, `as: :numo` raises
+`Herringbone::UnsupportedError` naming the gem.
+
+```ruby
+cols = reader.read(as: :numo, columns: %w[id amount])  # { "id" => Numo::Int64, "amount" => Numo::DFloat }
+df = Rover::DataFrame.new(reader.read(as: :numo))       # no conversion needed
+```
+
+Flat numeric and boolean columns are decoded from the page bytes without a Ruby object per value:
+two numeric columns of 1M uncompressed rows read in about 25 ms, against 50 ms with `as: :columns`.
+
+| Parquet | Numo |
+|---|---|
+| INT32, INT64 (also TIME) | `Int32`, `Int64` |
+| INT(8/16) signed; INT(8/16/32/64) unsigned | `Int8`, `Int16`; `UInt8` … `UInt64` |
+| FLOAT, FLOAT16, DOUBLE | `SFloat`, `SFloat`, `DFloat` (nulls are NaN) |
+| integers with nulls | `DFloat` with NaN (exact up to 2**53) |
+| BOOLEAN | `Bit`; with nulls `RObject` of true/false/nil |
+| list of numbers, every row the same length and no nulls | 2-D `[rows, length]` (e.g. embeddings) |
+| strings, binary, decimals, dates, timestamps, UUIDs, structs, maps, other lists | `RObject` of the values `read` returns |
+
+Whether a column has nulls (or a list column is rectangular) is decided from the rows read, so
+with `each_batch` it can differ between batches.
+
 ## Writing
 
 ```ruby
@@ -108,12 +177,16 @@ end
 
 `Herringbone.write(io, rows)` writes an Enumerable of rows in one go, inferring the schema from the
 first 1000 rows unless `schema:` is given; fields declared in a block replace inferred ones:
-`Herringbone.write(io, rows, schema: Herringbone::Schema.infer(rows) { json :payload })`.
+`Herringbone.write(io, rows) { json :payload }`. The rows are iterated once, holding back only
+those first 1000, so lazy Enumerators and cursors that can't be rewound work. A later row that
+doesn't fit the inferred types raises `Herringbone::SchemaMismatch`, which explains what was
+inferred and how to declare the column, and leaves the file unfinished.
 
 The writer writes to any IO that responds to `#write` (a `File`, `StringIO`, `Tempfile`, socket or
-pipe), sequentially, and never seeks, rewinds or closes it. If the `Writer.open` block raises (or
-`#abort` is called), no footer is written and what was written so far is left for you to discard.
-To replace a file only once it is complete, write to a temporary file and rename it.
+pipe), sequentially, and never seeks, rewinds or closes it (it does switch it to binary mode). If
+the `Writer.open` block raises (or `#abort` is called), no footer is written and what was written
+so far is left for you to discard. To replace a file only once it is complete, write to a temporary
+file and rename it.
 
 Column types: `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double float16
 string binary json bson enum uuid date int96 time timestamp decimal fixed`, plus `struct`, `list`
@@ -159,6 +232,24 @@ Writer options:
 | `metadata` | `{}` | footer key/value metadata, read back with `reader.metadata` |
 | `bloom_filters` | none | `true`, an Array of column paths, or `{ "path" => { ndv:, fpp:, max_bytes: } }` |
 
+### Coming from CSV
+
+`SimpleWriter` writes like the CSV gem: name the columns, then append Arrays. The types are
+inferred from the first 1000 rows, as above.
+
+```ruby
+File.open("people.parquet", "wb") do |file|
+  Herringbone::SimpleWriter.open(file) do |sw|
+    sw.headers!(:id, :name, :age)
+    sw << [123, "John", 12]
+    sw << { id: 124, name: "Jane" } # Hashes work too
+  end
+end
+```
+
+Unlike CSV, headers are required. A column that holds more than one type can be declared up front:
+`Herringbone::SimpleWriter.new(io) { string :code }` (then call `close` when done).
+
 ### Statistics, page indexes and bloom filters
 
 Every column chunk gets min/max/null-count statistics and the Parquet page index (per-page min/max
@@ -171,10 +262,29 @@ emails or UUIDs; a bloom filter can. They are off by default. `bloom_filters: ["
 (or `true` for every non-boolean column) writes Parquet's split block bloom filters, sized from
 each row group's distinct values at a false positive probability of 1% (`fpp:`, up to `max_bytes:`,
 default 1MB) unless `ndv:` gives the number of distinct values. Nested leaves are named by their
-dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is installed:
+dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is loaded:
 a million rows take about 1.2 s longer to write with a filter on an INT64 column and 2.8 s longer
 with one on a ~22-byte string column, and about 0.5 s longer with `xxhash`.
 
+### Writing to S3
+
+Parquet keeps its metadata in a footer, so a file can be streamed into an S3 multipart upload
+without a local copy, using `upload_stream` from `aws-sdk-s3`:
+
+```ruby
+s3 = Aws::S3::TransferManager.new # aws-sdk-s3 1.197+; before that, Aws::S3::Object#upload_stream
+s3.upload_stream(bucket: "exports", key: "events.parquet", part_size: 16 * 1024 * 1024) do |io|
+  Herringbone::Writer.open(io, schema) do |w|
+    events.each { |event| w << event }
+  end
+end
+s3.upload_stream(bucket: "exports", key: "orders.parquet") { |io| Herringbone.write(io, Order.all) }
+```
+
+If the block raises, the SDK aborts the multipart upload and raises `Aws::S3::MultipartUploadError`,
+so no partial object is left. S3 allows at most 10,000 parts, which with the default 5MB parts caps
+the file at about 48GB; raise `part_size:` for bigger files.
+
 ## ActiveRecord
 
 `Herringbone.write` also takes a model or relation, which it reads with `find_each`, using a schema
@@ -229,7 +339,7 @@ follows `Model.columns`.
 
 > The inspector and its HTML view are modelled on
 > **[Parquet X-ray](https://huggingface.co/spaces/cfahlgren1/parquet-xray) by cfahlgren1** —
-> the design and the idea are theirs. Go check it out.
+> the design and the idea are theirs. Go check it out. It is amazing!
 
 `Herringbone::Inspector` examines a file using only its footer, page headers, page indexes and
 bloom filter headers. Nothing is decompressed, so it is fast on big files and works for ZSTD and
@@ -252,14 +362,20 @@ CRCs; the results then appear in `summary`, `report`, `to_h` and `to_html`.
 ## Command line
 
 ```
-bin/herringbone cat FILE [N]                    # rows as JSON lines
-bin/herringbone inspect FILE [--pages]          # text report (--pages lists every page header)
-bin/herringbone inspect FILE --json             # everything as JSON
-bin/herringbone inspect FILE --html > out.html  # the HTML page
+bin/herringbone cat FILE [N]                           # rows as JSON lines
+bin/herringbone inspect FILE [--pages]                 # text report (--pages lists every page header)
+bin/herringbone inspect FILE --format=json             # everything as JSON
+bin/herringbone inspect FILE --format=html             # the HTML page, opened in your browser
+bin/herringbone inspect FILE --format=html > out.html  # the HTML page, saved
 ```
 
 Add `--verify-checksums` to any `inspect` form to check page CRCs.
 
+`--format` is `text` (the default), `json` or `html`; `--pages` only goes with `text`.
+In a terminal, `--format=html` writes the page to a temp file, prints its path and opens it with `open` on
+macOS, `start` on Windows or `xdg-open` elsewhere. When stdout is redirected or piped it prints the
+page instead.
+
 ## Supported format features
 
 - Encodings (read and write): PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE, DELTA_BINARY_PACKED,

data/bin/herringbone

@@ -3,14 +3,21 @@
 
 # Usage:
 #   herringbone cat FILE [N]              print rows as JSON lines (optionally only the first N)
-#   herringbone inspect FILE [--pages]    print the schema, row groups, column chunks (and every page header)
-#   herringbone inspect FILE --json       print all of it as JSON
-#   herringbone inspect FILE --html       print a self-contained HTML page showing the file's layout
+#   herringbone inspect FILE [--pages]          print the schema, row groups, column chunks (and every
+#                                               page header); same as --format=text
+#   herringbone inspect FILE --format=json      print all of it as JSON
+#   herringbone inspect FILE --format=html      write a self-contained HTML page showing the file's layout
+#                                               and open it in the browser (printed when stdout is redirected)
 #   add --verify-checksums to any inspect form to read page bodies and check their CRCs
 # `inspect` reads only the footer, page headers and page indexes: no values are decompressed.
 $LOAD_PATH.unshift File.expand_path("../lib", __dir__)
 require "herringbone"
 require "json"
+%w[zstd-ruby brotli snappy xxhash].each do |lib|
+  require lib
+rescue LoadError
+end
+require "tmpdir"
 
 def jsonable(v)
   case v
@@ -25,8 +32,20 @@ def jsonable(v)
   end
 end
 
+# Opens a file with the platform's default application
+# @return [Boolean] whether the opener could be started
+def open_with_default_app(path)
+  command = case RbConfig::CONFIG["host_os"]
+  when /darwin/ then ["open", path]
+  when /mswin|mingw|cygwin/ then ["cmd", "/c", "start", "", path]
+  else ["xdg-open", path]
+  end
+  !!system(*command, out: File::NULL, err: File::NULL, exception: false)
+end
+
 USAGE = "usage: herringbone cat FILE [N]\n" \
-  "       herringbone inspect FILE [--pages | --json | --html] [--verify-checksums]"
+  "       herringbone inspect FILE [--format=text|json|html] [--pages] [--verify-checksums]\n" \
+  "       (--pages only with --format=text)"
 
 command, *rest = ARGV
 flags, args = rest.partition { |a| a.start_with?("--") }
@@ -34,8 +53,11 @@ path, limit = args
 valid = case command
 when "cat" then flags.empty? && path && (limit.nil? || limit.match?(/\A\d+\z/))
 when "inspect"
-  formats = flags & %w[--pages --json --html]
-  path && limit.nil? && (flags - %w[--pages --json --html --verify-checksums]).empty? && formats.size <= 1
+  format_flags, other_flags = flags.partition { |f| f.start_with?("--format=") }
+  format = format_flags.empty? ? "text" : format_flags.first.delete_prefix("--format=")
+  path && limit.nil? && format_flags.size <= 1 && %w[text json html].include?(format) &&
+    (other_flags - %w[--pages --verify-checksums]).empty? &&
+    (format == "text" || !other_flags.include?("--pages"))
 end
 unless valid
   warn USAGE
@@ -51,11 +73,19 @@ begin
     else
       inspector = Herringbone::Inspector.new(io)
       inspector.verify_checksums if flags.include?("--verify-checksums")
-      if flags.include?("--json")
+      case format
+      when "json"
         puts JSON.pretty_generate(inspector.to_h)
-      elsif flags.include?("--html")
-        $stdout.write(inspector.to_html)
-      else
+      when "html"
+        if $stdout.tty?
+          html_path = File.join(Dir.tmpdir, "#{File.basename(path, ".*")}-#{Process.pid}.html")
+          File.binwrite(html_path, inspector.to_html)
+          puts html_path
+          warn "herringbone: could not open the page, open #{html_path} in a browser" unless open_with_default_app(html_path)
+        else
+          $stdout.write(inspector.to_html)
+        end
+      when "text"
         puts inspector.report(pages: flags.include?("--pages"))
       end
     end

data/lib/herringbone/active_record.rb

@@ -11,13 +11,18 @@ module Herringbone
     # ActiveRecord is not required: this only uses what a model class exposes
     # (+columns+, +primary_key+ and, when present, +defined_enums+).
     #
-    # only:         attribute names to include (Strings or Symbols)
-    # except:       attribute names to leave out
-    # parquet_enum: Rails enum attributes are string columns holding the labels (the writer
-    #               rejects values outside the enum; stored values such as 0/1 are written as
-    #               their labels). true adds the Parquet ENUM annotation, see Builder#enum.
+    # Primary key columns are never nullable. Rails enum attributes are string columns holding the
+    # labels (the writer rejects values outside the enum; stored values such as 0/1 are written as
+    # their labels).
+    #
+    # @param model [Class] ActiveRecord model class, or anything exposing +columns+ the same way
+    # @param only [Array<String, Symbol>, String, Symbol, nil] attribute names to include
+    # @param except [Array<String, Symbol>, String, Symbol, nil] attribute names to leave out
+    # @param parquet_enum [Boolean] true adds the Parquet ENUM annotation to enum columns, see Builder#enum
+    # @return [Schema] schema with one top-level field per selected column, in model column order
+    # @raise [ArgumentError] when no columns are left after +only+ / +except+
     def self.from_active_record(model, only: nil, except: nil, parquet_enum: false)
-      only = only && Array(only).map(&:to_s)
+      only &&= Array(only).map(&:to_s)
       except = Array(except).map(&:to_s)
       defined_enums = model.respond_to?(:defined_enums) ? model.defined_enums.to_h { |k, v| [k.to_s, v] } : {}
       primary_keys = Array(model.respond_to?(:primary_key) ? model.primary_key : nil).map(&:to_s)
@@ -54,9 +59,14 @@ module Herringbone
     module ActiveRecordMapping
       module_function
 
+      # Precision for decimal columns that declare none (Postgres +numeric+ without arguments);
+      # 38 digits is the most a 16-byte FIXED_LEN_BYTE_ARRAY decimal holds
       DEFAULT_DECIMAL_PRECISION = 38
+      # Scale used together with DEFAULT_DECIMAL_PRECISION when the column declares neither
       DEFAULT_DECIMAL_SCALE = 9
 
+      # Bit width of named SQL integer types (MySQL, Postgres and SQLite spellings), keyed by the
+      # leading word of the lowercased +sql_type+
       INTEGER_SQL_TYPES = {
         "tinyint" => 8, "int1" => 8,
         "smallint" => 16, "int2" => 16, "smallserial" => 16, "serial2" => 16,
@@ -64,6 +74,16 @@ module Herringbone
         "bigint" => 64, "int8" => 64, "bigserial" => 64, "serial8" => 64
       }.freeze
 
+      # Declares +column+ on +builder+: hstore as map<string, string>, array columns as a list of
+      # the element type, everything else as the scalar type picked by #scalar_type.
+      #
+      # @param builder [Builder] builder to add the field to
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata (+name+, +type+, and
+      #   when available +sql_type+, +array+, +limit+, +precision+, +scale+)
+      # @param nullable [Boolean] whether the field may be null
+      # @param primary [Boolean] whether the column is (part of) the primary key
+      # @return [Node] the added schema node
+      # @raise [ArgumentError] when the builder rejects the field (e.g. a duplicate name)
       def add_column(builder, column, nullable, primary)
         sql_type = column.respond_to?(:sql_type) ? column.sql_type.to_s.downcase : ""
         array = (column.respond_to?(:array) && column.array) || sql_type.end_with?("[]")
@@ -83,10 +103,18 @@ module Herringbone
         end
       end
 
+      # Picks the DSL type for a non-array, non-hstore column, following the table above.
+      #
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata, read for
+      #   +limit+, +precision+ and +scale+
+      # @param type [Symbol, nil] ActiveRecord's abstract type (+column.type+)
+      # @param sql_type [String] lowercased database type with any array suffix removed
+      # @param primary [Boolean] whether the column is (part of) the primary key
+      # @return [Array(Symbol, Hash{Symbol => Object})] DSL type and its options for Builder#column
       def scalar_type(column, type, sql_type, primary)
         case type
         when :integer, :bigint then [integer_type(column, sql_type, primary), {}]
-        when :float then [sql_type == "float4" ? :float : :double, {}]
+        when :float then [(sql_type == "float4") ? :float : :double, {}]
         when :decimal, :money
           precision = column.respond_to?(:precision) ? column.precision : nil
           scale = column.respond_to?(:scale) ? column.scale : nil
@@ -94,12 +122,12 @@ module Herringbone
             precision = DEFAULT_DECIMAL_PRECISION
             scale ||= DEFAULT_DECIMAL_SCALE
           end
-          [:decimal, { precision: precision, scale: scale || 0 }]
+          [:decimal, {precision: precision, scale: scale || 0}]
         when :boolean then [:boolean, {}]
         when :binary then [:binary, {}]
         when :date then [:date, {}]
-        when :datetime, :timestamp, :timestamptz then [:timestamp, { unit: :micros, utc: true }]
-        when :time then [:time, { unit: :micros }]
+        when :datetime, :timestamp, :timestamptz then [:timestamp, {unit: :micros, utc: true}]
+        when :time then [:time, {unit: :micros}]
         when :json, :jsonb then [:json, {}]
         when :uuid then [:uuid, {}]
         else [:string, {}]
@@ -108,6 +136,11 @@ module Herringbone
 
       # Named SQL types (smallint, bigint, ...) decide the width; a generic "integer"/"int"
       # uses the column limit in bytes, since e.g. SQLite reports `t.integer limit: 2` as "integer(2)".
+      #
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata, read for +limit+
+      # @param sql_type [String] lowercased database type with any array suffix removed
+      # @param primary [Boolean] true forces int64, since row ids can outgrow the declared width
+      # @return [Symbol] one of +:int8+ .. +:int64+ or +:uint8+ .. +:uint64+
       def integer_type(column, sql_type, primary)
         bits = INTEGER_SQL_TYPES[sql_type[/\A[a-z0-9]+/]]
         bits ||= case (column.respond_to?(:limit) ? column.limit : nil)
@@ -120,9 +153,8 @@ module Herringbone
         bits = 64 if primary
         unsigned = sql_type.include?("unsigned")
         return :"uint#{bits}" if unsigned
-        { 8 => :int8, 16 => :int16, 32 => :int32, 64 => :int64 }.fetch(bits)
+        {8 => :int8, 16 => :int16, 32 => :int32, 64 => :int64}.fetch(bits)
       end
     end
   end
 end
-