herringbone 0.1.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1148482072b7346372f0ce7d404c237359dbda86443c0b12dd8200937f6336cc
-  data.tar.gz: 9c1abff1ff39a715b0cb32f9bb9385d3642cdf6a0f2f5f6c45e741f25832ca21
+  metadata.gz: '08770d40f03a2320d9aa60e6fe63fbfe40ea5ddeddc627bff3be2bf46f431795'
+  data.tar.gz: b18fef0b86fd290576d254f338da456ad80dc2600c68eb0c903e6085a024ebe2
 SHA512:
-  metadata.gz: 84e2d8192f4a0e9e15430cf8df20d1dd070028dc3561973d639ca887ff82c8673e7864063419a63b05649526ccf6dd0f2a6eb9d291067fd5c674a2ba76ec9771
-  data.tar.gz: 532f1deff266b6f4867f47c6e80b200cfa9bcdb1a5d45442ef8e40b5f25247087fa2a85975822c3f572c6a64575ae6b4f87bbd72a13df1143db2a0c82fde856b
+  metadata.gz: 5c941e4b5f31f278ff819d25d464dab518c46e50d10435c360faf2da5994a9cff891754d00a73d8834aa08f2cd00be0387d9f6c8536459389797ba7daa1a82d0
+  data.tar.gz: 4532d47095f0832bf4cebb86ef370882dd7287625d80b9b94f6bc49fffb1ad3317075ecf8c45a1fef3fdb2fa19c3a641c0a6c2c4664bf24f5aa664e5b686e03e

data/README.md

@@ -2,8 +2,8 @@
 
 A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/) files.
 
-- No native extensions, no Thrift gem: the Thrift compact protocol, all encodings and the
-  Snappy and LZ4 codecs are implemented in Ruby
+- No Thrift gem and no native extensions of its own: the Thrift compact protocol, all encodings
+  and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems)
 - Full nesting support (structs, lists, maps, any depth) via Dremel record shredding/assembly
 - Reads files from parquet-mr, Arrow, Spark, Impala, DuckDB, Rust writers etc.
 - Ruby 3.0+
@@ -12,32 +12,102 @@ A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/)
 
 ```ruby
 gem "herringbone"
+gem "snappy"    # optional: native Snappy, 2-3x faster reads and writes of typical files
+gem "zstd-ruby" # optional: ZSTD (faster writes and smaller files than the default Snappy)
+gem "brotli"    # optional: Brotli
+gem "xxhash"    # optional: faster bloom filters
+gem "numo-narray-alt" # optional: read(as: :numo)
 ```
 
-Snappy and LZ4 are implemented in Ruby, GZIP uses `zlib`, and ZSTD and Brotli come from the
-`zstd-ruby` and `brotli` gems (runtime dependencies, along with `bigdecimal`).
+The only dependency is `bigdecimal`. Snappy, LZ4 and GZIP always work. Snappy, Parquet's most
+common codec, is pure Ruby unless the `snappy` gem is installed (it needs libsnappy or cmake to
+build); Herringbone then uses it automatically. `Herringbone.codecs` lists
+the codecs this process can use, e.g. `[:none, :snappy, :gzip, :lz4, :lz4_hadoop, :zstd]`. Using
+a missing one raises `Herringbone::MissingCodecError` naming the gem to add: a writer raises it
+before writing anything, a reader when it reaches the first such page (the schema and metadata
+are still readable). LZO is not supported.
 
 ## Reading
 
+Herringbone never opens files by path: readers take a random-access IO (a `File` opened with
+`"rb"`, `StringIO`, `Tempfile`...), which stays open and belongs to the caller.
+
 ```ruby
 require "herringbone"
 
-Herringbone::Reader.open("data.parquet") do |reader|
-  reader.schema             # => #<Herringbone::Schema ...>
+File.open("data.parquet", "rb") do |file|
+  reader = Herringbone::Reader.new(file)
+  reader.schema
   reader.num_rows
 
-  reader.each_row do |row|  # Hash with String keys, nested values as Hash/Array
-    p row
-  end
-
-  reader.each_row(columns: ["id", "name"]) { |row| ... } # projection
-  reader.column("name")     # => all values of one top-level field
-  reader.read_row_group(0)  # => { "id" => [...], "name" => [...] }
+  reader.each_row { |row| p row }             # Hashes with String keys, nested values as Hash/Array
+  reader.each_batch(1000) { |rows| ... }      # Arrays of up to 1000 rows
+  reader.each_batch(10_000, as: :columns) { |batch| batch["amount"].sum } # { "id" => [...], ... }
+  reader.read                                 # all rows at once; read(as: :columns) for column Arrays
 end
+```
+
+Reads stream: pages are read and decoded one at a time per column and rows are assembled in
+batches, so memory depends on the batch and page sizes rather than on the row group size (1M
+rows in a single row group peak at about 60–160 MB RSS growth instead of 660 MB). Column-order
+batches (`as: :columns`) skip building a Hash per row and are 25–30% faster.
+
+`Reader.new` takes `keys: :symbol` for Symbol keys (in rows and in structs; map keys stay as
+stored) and `time_zone:` to return timestamps in a zone instead of UTC: a UTC offset (`"+02:00"`,
+or seconds), `"UTC"`, a `TZInfo::Timezone`, or anything responding to `#at` such as `Time.zone`
+in Rails (which yields `ActiveSupport::TimeWithZone`). Zone names like `"Europe/Amsterdam"` work
+when ActiveSupport or TZInfo is loaded. Timestamps stored with `isAdjustedToUTC=false` are
+wall-clock values and stay as they are.
+
+### Selecting rows
 
-Herringbone.read("data.parquet") # => Array of row Hashes
+`each_row`, `each_batch` and `read` take `columns:`, `where:`, `from:` and `limit:`:
+
+```ruby
+reader.read(columns: %w[id name])                                          # projection
+reader.read(where: { user_id: 42 })
+reader.read(where: { status: %w[paid shipped], created_at: 1.week.ago.. })  # IN, Ranges
+reader.read(where: { "address.city" => "Amsterdam", deleted_at: nil })     # struct members, IS NULL
+reader.read(where: { amount: ->(v) { v && v > 100 } })                      # any callable
+reader.read(from: 1_000_000, limit: 100)                                    # rows 1,000,000..1,000,099
 ```
 
+All conditions must hold; filtered columns need not be among `columns:`, and any column not inside
+a list or map can be filtered on. Row groups are skipped using min/max statistics, null counts and
+bloom filters, pages using the page index (written by Herringbone, parquet-mr, Arrow and others),
+and `from:` jumps to its row through the offset index; every row read is then checked, so results
+are exact. On a 1M-row file with 20k-row pages, looking up one `id` takes 0.1 s instead of 6.3 s.
+Filters help most on columns the data is sorted or clustered by. `reader.scan_plan(where: ...)`
+shows which row groups and row ranges a read would touch, without reading them.
+
+### Numo arrays
+
+`as: :numo` returns (or yields, with `each_batch`) a Hash of column name => Numo array, and takes
+the same `columns:`, `where:`, `from:` and `limit:`. Add `gem "numo-narray-alt"` (or
+`numo-narray`) to your Gemfile: Herringbone requires it on first use and raises
+`Herringbone::UnsupportedError` naming the gem when it is missing.
+
+```ruby
+cols = reader.read(as: :numo, columns: %w[id amount])  # { "id" => Numo::Int64, "amount" => Numo::DFloat }
+df = Rover::DataFrame.new(reader.read(as: :numo))       # no conversion needed
+```
+
+Flat numeric and boolean columns are decoded from the page bytes without a Ruby object per value:
+two numeric columns of 1M uncompressed rows read in about 25 ms, against 50 ms with `as: :columns`.
+
+| Parquet | Numo |
+|---|---|
+| INT32, INT64 (also TIME) | `Int32`, `Int64` |
+| INT(8/16) signed; INT(8/16/32/64) unsigned | `Int8`, `Int16`; `UInt8` … `UInt64` |
+| FLOAT, FLOAT16, DOUBLE | `SFloat`, `SFloat`, `DFloat` (nulls are NaN) |
+| integers with nulls | `DFloat` with NaN (exact up to 2**53) |
+| BOOLEAN | `Bit`; with nulls `RObject` of true/false/nil |
+| list of numbers, every row the same length and no nulls | 2-D `[rows, length]` (e.g. embeddings) |
+| strings, binary, decimals, dates, timestamps, UUIDs, structs, maps, other lists | `RObject` of the values `read` returns |
+
+Whether a column has nulls (or a list column is rectangular) is decided from the rows read, so
+with `each_batch` it can differ between batches.
+
 ## Writing
 
 ```ruby
@@ -56,38 +126,34 @@ schema = Herringbone::Schema.define do
   timestamp :created_at
 end
 
-Herringbone::Writer.open("out.parquet", schema) do |w|
-  w << { "id" => 1, "name" => "Anna", "status" => "paid", "tags" => ["a", "b"],
-         "scores" => { "x" => 1.5 }, "address" => { "city" => "Amsterdam" },
-         "price" => BigDecimal("9.99"), "payload" => { "any" => ["json"] }, "created_at" => Time.now }
-  w << { id: 2, status: :pending }  # Symbol keys and values work; missing keys are nulls
-  w << [3, "Bo", nil, nil, nil, nil, nil, nil, nil, nil] # Arrays in schema order
-  w << order                         # anything with #attributes (ActiveRecord) or #to_h (Struct, Data)
+File.open("out.parquet", "wb") do |file|
+  Herringbone::Writer.open(file, schema) do |w|
+    w << { "id" => 1, "name" => "Anna", "status" => "paid", "tags" => ["a", "b"],
+           "scores" => { "x" => 1.5 }, "address" => { "city" => "Amsterdam" },
+           "price" => BigDecimal("9.99"), "payload" => { "any" => ["json"] }, "created_at" => Time.now }
+    w << { id: 2, status: :pending }  # Symbol keys and values work; missing keys are nulls
+    w << [3, "Bo", nil, nil, nil, nil, nil, nil, nil, nil] # Arrays in schema order
+    w << order                         # anything with #attributes (ActiveRecord) or #to_h (Struct, Data)
+  end
 end
-
-# Or with an inferred schema, optionally overriding some columns
-Herringbone.write("out.parquet", rows, schema: Herringbone::Schema.infer(rows, types: { payload: :json }))
 ```
 
-Schemas can also be given as a Hash, anywhere a schema is accepted:
+`Herringbone.write(io, rows)` writes an Enumerable of rows in one go, inferring the schema from the
+first 1000 rows unless `schema:` is given; fields declared in a block replace inferred ones:
+`Herringbone.write(io, rows, schema: Herringbone::Schema.infer(rows) { json :payload })`.
 
-```ruby
-Herringbone::Schema.define(
-  id: { type: :int64, null: false },
-  name: :string,
-  tags: [:string],                                   # list of strings
-  address: { city: :string, zip: :string },          # struct
-  price: { type: :decimal, precision: 12, scale: 2 },
-  scores: { type: :map, key: :string, value: :double }
-)
-Herringbone::Writer.open("out.parquet", { id: :int64, name: :string }) { |w| w << [1, "x"] }
-```
+The writer writes to any IO that responds to `#write` (a `File`, `StringIO`, `Tempfile`, socket or
+pipe), sequentially, and never seeks, rewinds or closes it (it does switch it to binary mode). If
+the `Writer.open` block raises (or `#abort` is called), no footer is written and what was written
+so far is left for you to discard. To replace a file only once it is complete, write to a temporary
+file and rename it.
 
-Column types in the DSL: `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double
-float16 string binary json bson enum uuid date int96 time timestamp decimal fixed`, plus
-`struct`, `list` and `map`. Fields are nullable unless `null: false` is given.
-List elements are nullable unless `element_null: false`; map values unless `value_null: false`.
-Nested lists: `list :matrix do list :element, :double end`.
+Column types: `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double float16
+string binary json bson enum uuid date int96 time timestamp decimal fixed`, plus `struct`, `list`
+and `map`; `column :name, :int32` declares one by name. Fields are nullable unless `null: false`
+is given; list elements unless `element_null: false`, map values unless `value_null: false`.
+Nested lists: `list :matrix do list :element, :double end`. `time` and `timestamp` take `unit:`
+(`:millis`, `:micros`, `:nanos`) and `utc:`.
 
 `enum` is a string column. `values:` restricts what may be written, and also takes a Rails-style
 Hash (`values: Order.statuses`), in which case both labels and stored integers are accepted and
@@ -111,41 +177,72 @@ Columns accept the values Ruby and Rails code usually has at hand:
 Values that don't fit raise `Herringbone::EncodeError` naming the row number and column path; the
 failed row is discarded and the writer can carry on.
 
-When given a path, the writer writes to a temporary file next to it and renames it into place on
-close. If the block raises (or `#abort` is called), the temporary file is removed and any
-existing file at the path is left untouched. An IO (`File`, `StringIO`, a socket...) can be
-given instead of a path.
-
 Writer options:
 
 | option | default | |
 |---|---|---|
-| `compression` | `:zstd` | `:none`, `:snappy`, `:gzip`, `:lz4` (LZ4_RAW), `:lz4_hadoop`, `:zstd`, `:brotli` |
-| `row_group_bytes` | 16MB | flush a row group once the buffered values take about this much memory; bounds memory use. Low-cardinality string columns are dictionary-encoded as rows arrive and other strings are packed into byte buffers, so a 15-column table exported in 16MB groups peaks around 290 MB RSS (420 MB with 64MB groups) |
-| `row_group_size` | none | also flush after this many rows |
-| `page_size` | 1MB | approximate data page size |
+| `compression` | `:snappy` | `:none`, `:snappy`, `:gzip`, `:lz4` (LZ4_RAW), `:lz4_hadoop`, `:zstd`, `:brotli` |
+| `row_group_bytes` | 16MB | flush a row group once the buffered values take about this much memory, which bounds memory use (a 15-column table peaks around 290 MB RSS) |
+| `row_group_rows` | none | also flush after this many rows |
+| `page_bytes` | 1MB | approximate data page size |
+| `page_rows` | `20_000` | at most this many rows per data page |
 | `data_page_version` | `1` | `1` or `2` |
 | `dictionary` | `true` | `false`, or an Array of column paths to dictionary-encode |
 | `encodings` | `{}` | e.g. `{ "id" => :delta_binary_packed, "x" => :byte_stream_split }` |
-| `statistics` | `true` | write min/max/null_count |
-| `metadata` | `{}` | footer key/value metadata |
+| `metadata` | `{}` | footer key/value metadata, read back with `reader.metadata` |
+| `bloom_filters` | none | `true`, an Array of column paths, or `{ "path" => { ndv:, fpp:, max_bytes: } }` |
+
+### Statistics, page indexes and bloom filters
+
+Every column chunk gets min/max/null-count statistics and the Parquet page index (per-page min/max
+and row offsets), which Herringbone, DuckDB, Spark, Trino, Arrow and DataFusion use to skip pages.
+With 20,000 rows per page, `WHERE id BETWEEN ...` on a sorted column reads a handful of pages
+instead of the whole row group, so sort rows by the columns you filter on.
+
+Min/max cannot rule out a value inside a row group's range, the usual case for unsorted IDs,
+emails or UUIDs; a bloom filter can. They are off by default. `bloom_filters: ["user_id", "email"]`
+(or `true` for every non-boolean column) writes Parquet's split block bloom filters, sized from
+each row group's distinct values at a false positive probability of 1% (`fpp:`, up to `max_bytes:`,
+default 1MB) unless `ndv:` gives the number of distinct values. Nested leaves are named by their
+dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is installed:
+a million rows take about 1.2 s longer to write with a filter on an INT64 column and 2.8 s longer
+with one on a ~22-byte string column, and about 0.5 s longer with `xxhash`.
 
-## Exporting ActiveRecord models
+### Writing to S3
 
-`Herringbone::Schema.from_active_record` builds a schema from a model's columns, so that
-`record.attributes` can be written as-is. Rails is not a dependency: it only calls
-`columns`, `primary_key` and `defined_enums` on the model.
+Parquet keeps its metadata in a footer, so a file can be streamed into an S3 multipart upload
+without a local copy, using `upload_stream` from `aws-sdk-s3`:
 
 ```ruby
-schema = Herringbone::Schema.from_active_record(Order)
-Herringbone::Writer.open("orders.parquet", schema) do |w|
-  Order.find_each { |order| w << order.attributes }
+s3 = Aws::S3::TransferManager.new # aws-sdk-s3 1.197+; before that, Aws::S3::Object#upload_stream
+s3.upload_stream(bucket: "exports", key: "events.parquet", part_size: 16 * 1024 * 1024) do |io|
+  Herringbone::Writer.open(io, schema) do |w|
+    events.each { |event| w << event }
+  end
+end
+s3.upload_stream(bucket: "exports", key: "orders.parquet") { |io| Herringbone.write(io, Order.all) }
+```
+
+If the block raises, the SDK aborts the multipart upload and raises `Aws::S3::MultipartUploadError`,
+so no partial object is left. S3 allows at most 10,000 parts, which with the default 5MB parts caps
+the file at about 48GB; raise `part_size:` for bigger files.
+
+## ActiveRecord
+
+`Herringbone.write` also takes a model or relation, which it reads with `find_each`, using a schema
+built from the model's columns. It returns the number of rows written:
+
+```ruby
+File.open("orders.parquet", "wb") do |file|
+  Herringbone.write(file, Order.where(created_at: 1.year.ago..), compression: :zstd)
 end
+schema = Herringbone::Schema.from_active_record(Order, only: %w[id status total])
+Herringbone.write(io, Order, schema: schema)
 ```
 
-Options: `only:` and `except:` take attribute names; `enums: :enum` writes enum attributes with
-the Parquet ENUM annotation instead of as plain strings (the default, `enums: :string`). Enum
-attributes are written as their labels, and the writer rejects values outside the enum.
+`from_active_record` takes `only:`, `except:` and `parquet_enum: true`. Rails is not a dependency:
+it only calls `columns`, `primary_key` and `defined_enums` on the model. Enum attributes are
+written as their labels, and the writer rejects values outside the enum.
 
 | Column | Parquet |
 |---|---|
@@ -161,8 +258,8 @@ attributes are written as their labels, and the writer rejects values outside th
 | `string`, `text`, `citext`, anything else | `string` |
 | Postgres array columns | `list` of the element type |
 
-Columns declared `NOT NULL` (and primary keys) are required, all others nullable.
-Column order follows `Model.columns`.
+Columns declared `NOT NULL` (and primary keys) are required, all others nullable. Column order
+follows `Model.columns`.
 
 ## Type mapping
 
@@ -180,23 +277,50 @@ Column order follows `Model.columns`.
 | UUID | `String` like `"0f1e2d3c-..."` |
 | struct / list / map | `Hash` / `Array` / `Hash` |
 
-## Supported format features
+## Inspecting files
 
-- Encodings (read and write): PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE, DELTA_BINARY_PACKED,
-  DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT; legacy BIT_PACKED levels (read)
-- Data page v1 and v2, dictionary pages, page CRCs (written)
-- Legacy list and map layouts per the Parquet backward-compatibility rules
-- Not supported: encryption, column chunks in external files, bloom filters and page indexes
-  (ignored when reading, not written)
+> The inspector and its HTML view are modelled on
+> **[Parquet X-ray](https://huggingface.co/spaces/cfahlgren1/parquet-xray) by cfahlgren1** —
+> the design and the idea are theirs. Go check it out.
+
+`Herringbone::Inspector` examines a file using only its footer, page headers, page indexes and
+bloom filter headers. Nothing is decompressed, so it is fast on big files and works for ZSTD and
+Brotli files without the codec gems.
+
+```ruby
+File.open("data.parquet", "rb") do |file|
+  inspector = Herringbone::Inspector.new(file)
+  inspector.summary      # size, rows, row groups, codecs, page index / bloom filter presence...
+  inspector.row_groups[0].column("name").pages # page headers; also statistics, column_index...
+  puts inspector.report  # the schema, every column chunk, key/value metadata (Arrow schema decoded)
+  inspector.to_h         # all of it, JSON-serializable
+  inspector.to_html      # one self-contained HTML page with a to-scale byte map of the file
+end
+```
+
+`inspector.verify_checksums` reads every page body (still without decompressing) to check the page
+CRCs; the results then appear in `summary`, `report`, `to_h` and `to_html`.
 
 ## Command line
 
 ```
-bin/herringbone schema FILE
-bin/herringbone meta FILE
-bin/herringbone cat FILE [N]
+bin/herringbone cat FILE [N]                    # rows as JSON lines
+bin/herringbone inspect FILE [--pages]          # text report (--pages lists every page header)
+bin/herringbone inspect FILE --json             # everything as JSON
+bin/herringbone inspect FILE --html > out.html  # the HTML page
 ```
 
+Add `--verify-checksums` to any `inspect` form to check page CRCs.
+
+## Supported format features
+
+- Encodings (read and write): PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE, DELTA_BINARY_PACKED,
+  DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT; legacy BIT_PACKED levels (read)
+- Data page v1 and v2, dictionary pages, page CRCs (written)
+- Page indexes and split block bloom filters (read and written)
+- Legacy list and map layouts per the Parquet backward-compatibility rules
+- Not supported: encryption, column chunks in external files
+
 ## Development
 
 ```

data/bin/herringbone

@@ -2,9 +2,12 @@
 # frozen_string_literal: true
 
 # Usage:
-#   herringbone schema FILE        print the schema
-#   herringbone meta FILE          print row groups, codecs and encodings
-#   herringbone cat FILE [N]       print rows as JSON lines (optionally only the first N)
+#   herringbone cat FILE [N]              print rows as JSON lines (optionally only the first N)
+#   herringbone inspect FILE [--pages]    print the schema, row groups, column chunks (and every page header)
+#   herringbone inspect FILE --json       print all of it as JSON
+#   herringbone inspect FILE --html       print a self-contained HTML page showing the file's layout
+#   add --verify-checksums to any inspect form to read page bodies and check their CRCs
+# `inspect` reads only the footer, page headers and page indexes: no values are decompressed.
 $LOAD_PATH.unshift File.expand_path("../lib", __dir__)
 require "herringbone"
 require "json"
@@ -22,33 +25,42 @@ def jsonable(v)
   end
 end
 
-command, path, limit = ARGV
-unless %w[schema meta cat].include?(command) && path
-  warn "usage: herringbone (schema|meta|cat) FILE [N]"
+USAGE = "usage: herringbone cat FILE [N]\n" \
+  "       herringbone inspect FILE [--pages | --json | --html] [--verify-checksums]"
+
+command, *rest = ARGV
+flags, args = rest.partition { |a| a.start_with?("--") }
+path, limit = args
+valid = case command
+when "cat" then flags.empty? && path && (limit.nil? || limit.match?(/\A\d+\z/))
+when "inspect"
+  formats = flags & %w[--pages --json --html]
+  path && limit.nil? && (flags - %w[--pages --json --html --verify-checksums]).empty? && formats.size <= 1
+end
+unless valid
+  warn USAGE
   exit 1
 end
 
-Herringbone::Reader.open(path) do |r|
-  case command
-  when "schema"
-    puts r.schema.inspect
-  when "meta"
-    puts "rows: #{r.num_rows}, row groups: #{r.num_row_groups}, created by: #{r.created_by}"
-    r.key_value_metadata.each { |k, v| puts "  #{k} = #{v.to_s[0, 80]}" }
-    r.row_groups.each_with_index do |rg, i|
-      puts "row group #{i}: #{rg.num_rows} rows"
-      rg.columns.each do |cc|
-        m = cc.meta_data
-        encodings = m.encodings.map { |e| Herringbone::Format::Encoding::NAMES[e] }.join(",")
-        puts "  #{m.path_in_schema.join(".")}: #{Herringbone::Format::Codec::NAMES[m.codec]} #{encodings} " \
-          "#{m.total_compressed_size}/#{m.total_uncompressed_size} bytes"
+# Parquet and file errors are reported in one line, without a backtrace
+begin
+  File.open(path, "rb") do |io|
+    if command == "cat"
+      n = limit&.to_i
+      Herringbone::Reader.new(io).each_row(limit: n) { |row| puts JSON.generate(jsonable(row)) }
+    else
+      inspector = Herringbone::Inspector.new(io)
+      inspector.verify_checksums if flags.include?("--verify-checksums")
+      if flags.include?("--json")
+        puts JSON.pretty_generate(inspector.to_h)
+      elsif flags.include?("--html")
+        $stdout.write(inspector.to_html)
+      else
+        puts inspector.report(pages: flags.include?("--pages"))
       end
     end
-  when "cat"
-    n = limit&.to_i
-    r.each_row.with_index do |row, i|
-      break if n && i >= n
-      puts JSON.generate(jsonable(row))
-    end
   end
+rescue Herringbone::Error, SystemCallError => e
+  warn "herringbone: #{e.message}"
+  exit 1
 end

data/lib/herringbone/active_record.rb

@@ -5,23 +5,24 @@ module Herringbone
     # Builds a schema from an ActiveRecord model, so that the Hashes returned by
     # +record.attributes+ can be written directly:
     #
-    #   schema = Herringbone::Schema.from_active_record(Order)
-    #   Herringbone::Writer.open("orders.parquet", schema) do |w|
-    #     Order.find_each { |order| w << order.attributes }
-    #   end
+    #   schema = Herringbone::Schema.from_active_record(Order, except: %w[notes])
+    #   File.open("orders.parquet", "wb") { |f| Herringbone.write(f, Order, schema: schema) }
     #
     # ActiveRecord is not required: this only uses what a model class exposes
     # (+columns+, +primary_key+ and, when present, +defined_enums+).
     #
-    # only:   attribute names to include (Strings or Symbols)
-    # except: attribute names to leave out
-    # enums:  :string (default) writes enum attributes as STRING columns, :enum uses the
-    #         Parquet ENUM annotation (which pyarrow/pandas read as binary). Either way the
-    #         enum labels are written, and the writer rejects values outside the enum mapping
-    #         (stored values such as 0/1 are accepted and written as their labels).
-    def self.from_active_record(model, only: nil, except: nil, enums: :string)
-      raise ArgumentError, "enums: must be :string or :enum" unless %i[string enum].include?(enums)
-      only = only && Array(only).map(&:to_s)
+    # Primary key columns are never nullable. Rails enum attributes are string columns holding the
+    # labels (the writer rejects values outside the enum; stored values such as 0/1 are written as
+    # their labels).
+    #
+    # @param model [Class] ActiveRecord model class, or anything exposing +columns+ the same way
+    # @param only [Array<String, Symbol>, String, Symbol, nil] attribute names to include
+    # @param except [Array<String, Symbol>, String, Symbol, nil] attribute names to leave out
+    # @param parquet_enum [Boolean] true adds the Parquet ENUM annotation to enum columns, see Builder#enum
+    # @return [Schema] schema with one top-level field per selected column, in model column order
+    # @raise [ArgumentError] when no columns are left after +only+ / +except+
+    def self.from_active_record(model, only: nil, except: nil, parquet_enum: false)
+      only &&= Array(only).map(&:to_s)
       except = Array(except).map(&:to_s)
       defined_enums = model.respond_to?(:defined_enums) ? model.defined_enums.to_h { |k, v| [k.to_s, v] } : {}
       primary_keys = Array(model.respond_to?(:primary_key) ? model.primary_key : nil).map(&:to_s)
@@ -34,7 +35,7 @@ module Herringbone
         primary = primary_keys.include?(name)
         nullable = column.null != false && !primary
         if (mapping = defined_enums[name])
-          builder.enum(name, values: mapping.to_h, parquet_enum: enums == :enum, null: nullable)
+          builder.enum(name, values: mapping.to_h, parquet_enum: parquet_enum, null: nullable)
         else
           ActiveRecordMapping.add_column(builder, column, nullable, primary)
         end
@@ -58,9 +59,14 @@ module Herringbone
     module ActiveRecordMapping
       module_function
 
+      # Precision for decimal columns that declare none (Postgres +numeric+ without arguments);
+      # 38 digits is the most a 16-byte FIXED_LEN_BYTE_ARRAY decimal holds
       DEFAULT_DECIMAL_PRECISION = 38
+      # Scale used together with DEFAULT_DECIMAL_PRECISION when the column declares neither
       DEFAULT_DECIMAL_SCALE = 9
 
+      # Bit width of named SQL integer types (MySQL, Postgres and SQLite spellings), keyed by the
+      # leading word of the lowercased +sql_type+
       INTEGER_SQL_TYPES = {
         "tinyint" => 8, "int1" => 8,
         "smallint" => 16, "int2" => 16, "smallserial" => 16, "serial2" => 16,
@@ -68,6 +74,16 @@ module Herringbone
         "bigint" => 64, "int8" => 64, "bigserial" => 64, "serial8" => 64
       }.freeze
 
+      # Declares +column+ on +builder+: hstore as map<string, string>, array columns as a list of
+      # the element type, everything else as the scalar type picked by #scalar_type.
+      #
+      # @param builder [Builder] builder to add the field to
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata (+name+, +type+, and
+      #   when available +sql_type+, +array+, +limit+, +precision+, +scale+)
+      # @param nullable [Boolean] whether the field may be null
+      # @param primary [Boolean] whether the column is (part of) the primary key
+      # @return [Node] the added schema node
+      # @raise [ArgumentError] when the builder rejects the field (e.g. a duplicate name)
       def add_column(builder, column, nullable, primary)
         sql_type = column.respond_to?(:sql_type) ? column.sql_type.to_s.downcase : ""
         array = (column.respond_to?(:array) && column.array) || sql_type.end_with?("[]")
@@ -87,10 +103,18 @@ module Herringbone
         end
       end
 
+      # Picks the DSL type for a non-array, non-hstore column, following the table above.
+      #
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata, read for
+      #   +limit+, +precision+ and +scale+
+      # @param type [Symbol, nil] ActiveRecord's abstract type (+column.type+)
+      # @param sql_type [String] lowercased database type with any array suffix removed
+      # @param primary [Boolean] whether the column is (part of) the primary key
+      # @return [Array(Symbol, Hash{Symbol => Object})] DSL type and its options for Builder#column
       def scalar_type(column, type, sql_type, primary)
         case type
         when :integer, :bigint then [integer_type(column, sql_type, primary), {}]
-        when :float then [sql_type == "float4" ? :float : :double, {}]
+        when :float then [(sql_type == "float4") ? :float : :double, {}]
         when :decimal, :money
           precision = column.respond_to?(:precision) ? column.precision : nil
           scale = column.respond_to?(:scale) ? column.scale : nil
@@ -98,12 +122,12 @@ module Herringbone
             precision = DEFAULT_DECIMAL_PRECISION
             scale ||= DEFAULT_DECIMAL_SCALE
           end
-          [:decimal, { precision: precision, scale: scale || 0 }]
+          [:decimal, {precision: precision, scale: scale || 0}]
         when :boolean then [:boolean, {}]
         when :binary then [:binary, {}]
         when :date then [:date, {}]
-        when :datetime, :timestamp, :timestamptz then [:timestamp, { unit: :micros, utc: true }]
-        when :time then [:time, { unit: :micros }]
+        when :datetime, :timestamp, :timestamptz then [:timestamp, {unit: :micros, utc: true}]
+        when :time then [:time, {unit: :micros}]
         when :json, :jsonb then [:json, {}]
         when :uuid then [:uuid, {}]
         else [:string, {}]
@@ -112,6 +136,11 @@ module Herringbone
 
       # Named SQL types (smallint, bigint, ...) decide the width; a generic "integer"/"int"
       # uses the column limit in bytes, since e.g. SQLite reports `t.integer limit: 2` as "integer(2)".
+      #
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata, read for +limit+
+      # @param sql_type [String] lowercased database type with any array suffix removed
+      # @param primary [Boolean] true forces int64, since row ids can outgrow the declared width
+      # @return [Symbol] one of +:int8+ .. +:int64+ or +:uint8+ .. +:uint64+
       def integer_type(column, sql_type, primary)
         bits = INTEGER_SQL_TYPES[sql_type[/\A[a-z0-9]+/]]
         bits ||= case (column.respond_to?(:limit) ? column.limit : nil)
@@ -124,7 +153,7 @@ module Herringbone
         bits = 64 if primary
         unsigned = sql_type.include?("unsigned")
         return :"uint#{bits}" if unsigned
-        { 8 => :int8, 16 => :int16, 32 => :int32, 64 => :int64 }.fetch(bits)
+        {8 => :int8, 16 => :int16, 32 => :int32, 64 => :int64}.fetch(bits)
       end
     end
   end