herringbone 0.3.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '08770d40f03a2320d9aa60e6fe63fbfe40ea5ddeddc627bff3be2bf46f431795'
-  data.tar.gz: b18fef0b86fd290576d254f338da456ad80dc2600c68eb0c903e6085a024ebe2
+  metadata.gz: 4aca873b946a39da4bee947137c6914f0b321f95266f252f35487087e40ce3f4
+  data.tar.gz: 6d7f439f61d6d871cfbf8682e2af71bf2c502b5360355570aabeed373240d0d7
 SHA512:
-  metadata.gz: 5c941e4b5f31f278ff819d25d464dab518c46e50d10435c360faf2da5994a9cff891754d00a73d8834aa08f2cd00be0387d9f6c8536459389797ba7daa1a82d0
-  data.tar.gz: 4532d47095f0832bf4cebb86ef370882dd7287625d80b9b94f6bc49fffb1ad3317075ecf8c45a1fef3fdb2fa19c3a641c0a6c2c4664bf24f5aa664e5b686e03e
+  metadata.gz: 9cc7d70651be9426aec7608b1fa456545ea0c5318c4aaf193a67b14c7d2371a7173c62cca23a73f603d412dd2ddafa5d8f8e0833b08e6efc9c0632fe367e56d2
+  data.tar.gz: bb1b6ed8e32eb6bcbdb0c29ca1dbd773a74dd2f5d2587253a4a297d76a27ffddc7f0a508baabb9a3d45d0bcd40423f3f5ea531cdc4adf3b4564c3344f61f3fda

data/CHANGELOG.md

@@ -0,0 +1,66 @@
+# Changelog
+
+## Unreleased
+
+## 0.5.0
+
+- `Herringbone.redact` and `Herringbone::Redaction` rewrite a file with rows deleted, values
+  replaced or columns dropped, for GDPR erasure and pseudonymization, copying the row groups they
+  don't touch byte for byte.
+- `Herringbone::Redaction#affects?` says whether a redaction would change a file, reading only
+  statistics, bloom filters and the `where` columns.
+- `Herringbone::Writer#inspect` (and `SimpleWriter`, `ByteValues`) prints a one-line summary with
+  the writer's state and row count instead of dumping the buffered rows.
+- **Breaking:** schema DSL blocks receive the builder as a parameter instead of running with
+  `instance_eval`: `Herringbone::Schema.define { |s| s.int64 :id }`, likewise for `Schema.infer`,
+  `Herringbone.write`, `SimpleWriter.new` and nested `struct`/`list`/`map` blocks. A block without
+  a parameter raises `ArgumentError`.
+
+## 0.4.0
+
+- herringbone is now Ractor-enabled! Reading and writing work from any Ractor on Ruby 3.1+, and schemas can be passed to Ractors with
+- **Breaking:** the optional gems (`zstd-ruby`, `brotli`, `snappy`, `xxhash`, `numo-narray-alt`)
+  are used only when loaded: require them (`Bundler.require` does) instead of relying on
+  Herringbone to require them on first use.
+- Files with LZO-compressed pages (written by parquet-mr with hadoop-lzo) can be read. Writing
+  LZO is not supported.
+- `Herringbone::SimpleWriter`: CSV-style writing with `headers!` and Array rows, types inferred.
+- `Herringbone.write` without `schema:` reads the rows once, holding back only the first 1000 for
+  inference, so cursors and one-shot Enumerators no longer lose rows.
+- `Herringbone.write(io, rows) { json :payload }` declares fields that replace inferred ones.
+- `Herringbone::SchemaMismatch` (an `EncodeError`) explains a row that doesn't fit an inferred
+  schema and stops the write; `EncodeError` has `row`, `column` and `value`.
+- `herringbone inspect --format=html` opens the page in the browser when run in a terminal.
+  `Ractor.make_shareable(schema)`.
+- **Breaking:** `herringbone inspect` takes `--format=text|json|html` instead of `--json`/`--html`.
+
+## 0.3.0
+
+- `read(as: :numo)` and `each_batch(as: :numo)` return Numo arrays (optional `numo-narray-alt`).
+- The `snappy` gem is used for Snappy when installed: 27x faster compression, 12x faster decompression.
+- Writes are forward-only, so files can be streamed into S3 multipart uploads.
+- YARD documentation for the whole library.
+
+## 0.2.0
+
+- Streaming reads: pages are decoded incrementally and rows assembled in batches.
+- `each_batch(as: :columns)` and `read(as: :columns)` for column-order reads.
+- `where:`, `from:` and `limit:` skip row groups and pages using statistics, bloom filters and the
+  page index.
+- Page indexes (ColumnIndex and OffsetIndex) are written for every column chunk.
+- Bloom filters are written and read, using the `xxhash` gem when it is installed.
+- `Herringbone::Inspector`, an HTML visualizer and `herringbone inspect`: page headers, page
+  indexes, CRC verification and the decoded ARROW:schema, without decompressing any data.
+- `Herringbone.write` takes ActiveRecord models and relations.
+- `zstd-ruby` and `brotli` are optional; a missing codec raises `MissingCodecError`.
+- **Breaking:** the public API was simplified: `Reader.new(io)` and `Writer.open(io, schema)`
+  only take IOs, Hash schemas and `Herringbone.export`/`open`/`read` are gone, and `DecodeError`
+  is now `FormatError`.
+
+## 0.1.0
+
+- Pure-Ruby Parquet reader and writer, with no Thrift gem or native extensions.
+- Full nesting support (structs, lists, maps) via Dremel shredding and assembly.
+- All standard encodings, v1 and v2 data pages, Snappy, LZ4 and GZIP in Ruby, ZSTD and Brotli via gems.
+- Schemas from a block DSL, inferred from rows, or built from ActiveRecord models.
+- Writer coercions for Rails values, byte-based row groups and compact buffers.

data/README.md

@@ -3,34 +3,71 @@
 A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/) files.
 
 - No Thrift gem and no native extensions of its own: the Thrift compact protocol, all encodings
-  and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems)
+  and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems).
+  LZO, from Hadoop-era files, can be read but not written
 - Full nesting support (structs, lists, maps, any depth) via Dremel record shredding/assembly
 - Reads files from parquet-mr, Arrow, Spark, Impala, DuckDB, Rust writers etc.
+- Optimized reads with batches and pages 
 - Ruby 3.0+
 
+## Diving in: dumping records in Rails
+
+Stream a relation straight into S3, no temp file needed:
+
+```ruby
+s3 = Aws::S3::TransferManager.new
+s3.upload_stream(bucket: "exports", key: "payments.parquet") do |io|
+  # Schema will be auto-inferred, find_each will be used automatically
+  Herringbone.write(io, Payment.where(status: "settled", created_at: 1.month.ago..))
+end
+```
+
 ## Installation
 
 ```ruby
 gem "herringbone"
-gem "snappy"    # optional: native Snappy, 2-3x faster reads and writes of typical files
-gem "zstd-ruby" # optional: ZSTD (faster writes and smaller files than the default Snappy)
-gem "brotli"    # optional: Brotli
-gem "xxhash"    # optional: faster bloom filters
-gem "numo-narray-alt" # optional: read(as: :numo)
 ```
 
-The only dependency is `bigdecimal`. Snappy, LZ4 and GZIP always work. Snappy, Parquet's most
-common codec, is pure Ruby unless the `snappy` gem is installed (it needs libsnappy or cmake to
-build); Herringbone then uses it automatically. `Herringbone.codecs` lists
-the codecs this process can use, e.g. `[:none, :snappy, :gzip, :lz4, :lz4_hadoop, :zstd]`. Using
-a missing one raises `Herringbone::MissingCodecError` naming the gem to add: a writer raises it
-before writing anything, a reader when it reaches the first such page (the schema and metadata
-are still readable). LZO is not supported.
+Add these libraries to speed up and enable certain functionality:
+
+```ruby
+gem "snappy"          # native Snappy, 2-3x faster reads and writes of typical files
+gem "zstd-ruby"       # ZSTD
+gem "brotli"          # Brotli
+gem "xxhash"          # faster bloom filters
+gem "numo-narray-alt" # read(as: :numo)
+```
+
+Herringbone never requires them itself: it uses whichever are loaded. Bundler.require (as in Rails)
+loads them, otherwise require them yourself:
+
+```ruby
+require "herringbone"
+require "zstd-ruby"
+require "numo/narray"
+```
+
+On Ruby 3.1 and later, Herringbone works from any Ractor. Pass a schema to a Ractor with
+`Ractor.make_shareable(schema)`. Rows made shareable the same way reach a writing Ractor by
+reference, without being copied:
+
+```ruby
+writer = Ractor.new("out.parquet", Ractor.make_shareable(schema)) do |path, schema|
+  File.open(path, "wb") do |f|
+    Herringbone::Writer.open(f, schema) do |w|
+      while (row = Ractor.receive) != :done
+        w << row
+      end
+    end
+  end
+end
+rows.each { |row| writer.send(Ractor.make_shareable(row)) }
+writer.send(:done)
+```
 
 ## Reading
 
-Herringbone never opens files by path: readers take a random-access IO (a `File` opened with
-`"rb"`, `StringIO`, `Tempfile`...), which stays open and belongs to the caller.
+Herringbone can read from any IO-ish object with random access (the IO should be seekable).
 
 ```ruby
 require "herringbone"
@@ -84,8 +121,8 @@ shows which row groups and row ranges a read would touch, without reading them.
 
 `as: :numo` returns (or yields, with `each_batch`) a Hash of column name => Numo array, and takes
 the same `columns:`, `where:`, `from:` and `limit:`. Add `gem "numo-narray-alt"` (or
-`numo-narray`) to your Gemfile: Herringbone requires it on first use and raises
-`Herringbone::UnsupportedError` naming the gem when it is missing.
+`numo-narray`) to your Gemfile and `require "numo/narray"`: without it, `as: :numo` raises
+`Herringbone::UnsupportedError` naming the gem.
 
 ```ruby
 cols = reader.read(as: :numo, columns: %w[id amount])  # { "id" => Numo::Int64, "amount" => Numo::DFloat }
@@ -111,19 +148,19 @@ with `each_batch` it can differ between batches.
 ## Writing
 
 ```ruby
-schema = Herringbone::Schema.define do
-  int64 :id, null: false
-  string :name
-  enum :status, values: %w[pending paid shipped]
-  list :tags, :string
-  map :scores, :string, :double
-  struct :address do
-    string :city
-    string :zip
+schema = Herringbone::Schema.define do |s|
+  s.int64 :id, null: false
+  s.string :name
+  s.enum :status, values: %w[pending paid shipped]
+  s.list :tags, :string
+  s.map :scores, :string, :double
+  s.struct :address do |address|
+    address.string :city
+    address.string :zip
   end
-  decimal :price, precision: 12, scale: 2
-  json :payload
-  timestamp :created_at
+  s.decimal :price, precision: 12, scale: 2
+  s.json :payload
+  s.timestamp :created_at
 end
 
 File.open("out.parquet", "wb") do |file|
@@ -140,7 +177,10 @@ end
 
 `Herringbone.write(io, rows)` writes an Enumerable of rows in one go, inferring the schema from the
 first 1000 rows unless `schema:` is given; fields declared in a block replace inferred ones:
-`Herringbone.write(io, rows, schema: Herringbone::Schema.infer(rows) { json :payload })`.
+`Herringbone.write(io, rows) { |s| s.json :payload }`. The rows are iterated once, holding back only
+those first 1000, so lazy Enumerators and cursors that can't be rewound work. A later row that
+doesn't fit the inferred types raises `Herringbone::SchemaMismatch`, which explains what was
+inferred and how to declare the column, and leaves the file unfinished.
 
 The writer writes to any IO that responds to `#write` (a `File`, `StringIO`, `Tempfile`, socket or
 pipe), sequentially, and never seeks, rewinds or closes it (it does switch it to binary mode). If
@@ -148,11 +188,15 @@ the `Writer.open` block raises (or `#abort` is called), no footer is written and
 so far is left for you to discard. To replace a file only once it is complete, write to a temporary
 file and rename it.
 
+The block receives the schema builder, and the blocks of `struct`, `list` and `map` receive a
+builder of their own, so the block can still call your methods and read your instance variables.
+A block that takes no parameter raises `ArgumentError`.
+
 Column types: `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double float16
 string binary json bson enum uuid date int96 time timestamp decimal fixed`, plus `struct`, `list`
-and `map`; `column :name, :int32` declares one by name. Fields are nullable unless `null: false`
+and `map`; `s.column :name, :int32` declares one by name. Fields are nullable unless `null: false`
 is given; list elements unless `element_null: false`, map values unless `value_null: false`.
-Nested lists: `list :matrix do list :element, :double end`. `time` and `timestamp` take `unit:`
+Nested lists: `s.list :matrix do |matrix| matrix.list :element, :double end`. `time` and `timestamp` take `unit:`
 (`:millis`, `:micros`, `:nanos`) and `utc:`.
 
 `enum` is a string column. `values:` restricts what may be written, and also takes a Rails-style
@@ -192,6 +236,24 @@ Writer options:
 | `metadata` | `{}` | footer key/value metadata, read back with `reader.metadata` |
 | `bloom_filters` | none | `true`, an Array of column paths, or `{ "path" => { ndv:, fpp:, max_bytes: } }` |
 
+### Coming from CSV
+
+`SimpleWriter` writes like the CSV gem: name the columns, then append Arrays. The types are
+inferred from the first 1000 rows, as above.
+
+```ruby
+File.open("people.parquet", "wb") do |file|
+  Herringbone::SimpleWriter.open(file) do |sw|
+    sw.headers!(:id, :name, :age)
+    sw << [123, "John", 12]
+    sw << { id: 124, name: "Jane" } # Hashes work too
+  end
+end
+```
+
+Unlike CSV, headers are required. A column that holds more than one type can be declared up front:
+`Herringbone::SimpleWriter.new(io) { |s| s.string :code }` (then call `close` when done).
+
 ### Statistics, page indexes and bloom filters
 
 Every column chunk gets min/max/null-count statistics and the Parquet page index (per-page min/max
@@ -204,7 +266,7 @@ emails or UUIDs; a bloom filter can. They are off by default. `bloom_filters: ["
 (or `true` for every non-boolean column) writes Parquet's split block bloom filters, sized from
 each row group's distinct values at a false positive probability of 1% (`fpp:`, up to `max_bytes:`,
 default 1MB) unless `ndv:` gives the number of distinct values. Nested leaves are named by their
-dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is installed:
+dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is loaded:
 a million rows take about 1.2 s longer to write with a filter on an INT64 column and 2.8 s longer
 with one on a ~22-byte string column, and about 0.5 s longer with `xxhash`.
 
@@ -227,6 +289,136 @@ If the block raises, the SDK aborts the multipart upload and raises `Aws::S3::Mu
 so no partial object is left. S3 allows at most 10,000 parts, which with the default 5MB parts caps
 the file at about 48GB; raise `part_size:` for bigger files.
 
+## Redaction
+
+`Herringbone.redact` rewrites a file with rows removed or values replaced, for GDPR "forget me"
+requests and pseudonymization. One file goes in and one comes out, and the parts of the file
+nothing touches are copied byte for byte.
+
+```ruby
+# Forget me: remove the rows
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).delete
+end
+
+# Forget me, but keep the row for accounting: blank the personal columns
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).replace(email: nil, name: nil, address: nil)
+end
+
+# Forget me, but keep the row linkable: pseudonymize just that user's email
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).replace(:email) { |email| OpenSSL::HMAC.hexdigest("SHA256", KEY, email) }
+end
+
+# Pseudonymize a column across the whole file, mask another, drop a third
+Herringbone.redact(input, output) do |r|
+  r.replace(:email) { |email| OpenSSL::HMAC.hexdigest("SHA256", KEY, email.downcase) }
+  r.replace(:phone) { |phone| phone && "***#{phone[-3..]}" }
+  r.drop :ssn, :ip_address
+end
+```
+
+A `Herringbone::Redaction` is built once and applies itself to any number of files, which suits a
+forget-me job going over a bucket:
+
+```ruby
+forget = Herringbone::Redaction.new do |r|
+  r.where(user_id: 42).delete
+  r.where(email: "anna@example.com").delete     # separate statements OR together
+  r.where(created_at: ..2.years.ago).delete      # retention works the same way
+end
+
+if forget.affects?(input)      # statistics and bloom filters first, then the where columns only
+  report = forget.apply(input, output)
+  report.rows_deleted          # => 3
+  report.row_groups            # => { copied: 61, rewritten: 1 }
+end
+```
+
+The block receives the redaction being built, like `Writer.open` passes the writer, so it can use
+the methods and instance variables around it; a block without the `|r|` raises `ArgumentError`.
+`where` takes what `read(where:)` takes (values, Arrays, Ranges, `nil`, callables and dotted
+struct paths, but no columns inside lists or maps), and skips row groups and pages the same way.
+`where(...).delete` removes the matching rows. `replace` sets constants,
+`replace(email: nil, name: "[deleted]")`, or computes each value with a block,
+`replace(:email, :phone) { |value| ... }`. A block that takes two parameters also gets the whole
+row, as a Hash with String keys like `read` returns. Without `where`, `replace` applies to every
+row. A column is a top-level field or a struct member by dotted path (`"address.city"`; a member
+of a null struct is left alone). To change what is inside a list or map, replace the whole field
+with a block that receives the Array or Hash. `drop :a, :b` removes columns from the schema.
+
+Statements apply in declared order, row by row: a deleted row is gone for later statements, and
+later statements see what earlier ones replaced, in their conditions as well as in their blocks.
+Mistakes raise `ArgumentError` before anything is written: a `where` without a verb, a column that
+doesn't exist, `nil` (or a value of the wrong type) for a `null: false` column. A block that
+returns something its column can't store raises `Herringbone::EncodeError` when it gets there and
+leaves the output unfinished.
+
+`apply` and `Herringbone.redact` take IOs like `Reader` and `Writer` do: the input must be
+seekable, the output only needs `#write`, and neither is closed. To redact in place, write to a
+temporary file and rename it over the original; on S3, read the object and write the new one
+through `upload_stream`. The `Redaction::Report` that `apply` returns has `rows_read`,
+`rows_deleted`, `rows_changed` and `row_groups` (`{ copied:, rewritten: }`), which is the audit
+trail an erasure needs.
+
+Each row group is handled in the cheapest way that is still exact. If no statement can match it,
+judging by statistics, bloom filters and the page index, or by reading only the `where` columns,
+its column chunks are copied as they are and only their offsets are rebased. If rows match but
+none is deleted and only leaf columns change, just those column chunks are encoded again. If rows
+are deleted, or a nested field is replaced whole, the row group is rewritten, as one row group
+(or none, when every row goes). Rewritten chunks keep the codec of the original chunk (LZO, which
+Herringbone can't write, becomes Snappy) and get a new bloom filter if they had one. Writer
+options (`compression:`, `bloom_filters:`, `page_rows:`...) apply to the rewritten chunks;
+`row_group_bytes:` and `row_group_rows:` are refused, since row groups keep their boundaries. The
+footer's key/value metadata is copied (without `ARROW:schema` and `pandas` when columns are
+dropped, since those describe the columns) unless `metadata:` is given.
+
+No deleted or replaced value survives in the output: not in data pages, dictionary pages, column
+chunk min/max statistics, the page index or bloom filters. What Herringbone can't reach is up to
+you: the original file, its S3 versions and backups still hold the data until you delete them,
+only the columns you name are touched (an email that also sits in a free-text `notes` column stays
+there), and other files with the same person in them need their own pass.
+
+Some recipes. Keyed hashing keeps a column joinable without keeping the value; keep the key out
+of the data, and rotate or destroy it to cut the link:
+
+```ruby
+require "openssl"
+KEY = ENV.fetch("PSEUDONYM_KEY")
+Herringbone.redact(input, output) do |r|
+  r.replace(:email) { |email| email && OpenSSL::HMAC.hexdigest("SHA256", KEY, email.downcase) }
+end
+```
+
+Masking keeps enough to be recognizable to the person but not to anyone else:
+
+```ruby
+Herringbone.redact(input, output) do |r|
+  r.replace(:card_number) { |number| number && number[-4..].rjust(number.size, "*") }
+  r.replace(:email) { |email| email&.sub(/\A(.).*@/, '\1***@') }
+  r.replace(:birth_date) { |date| date && Date.new(date.year, 1, 1) }
+end
+```
+
+Fake values (with the `faker` gem) make a copy for staging that looks real. Seed Faker from the
+original value to get the same fake for the same person in every file:
+
+```ruby
+require "faker"
+require "zlib"
+Herringbone.redact(input, output) do |r|
+  r.replace(:name) do |name|
+    next nil unless name
+    Faker::Config.random = Random.new(Zlib.crc32(name))
+    Faker::Name.name
+  end
+  r.replace(:address) do |address|
+    address && { "city" => Faker::Address.city, "zip" => Faker::Address.zip_code }
+  end
+end
+```
+
 ## ActiveRecord
 
 `Herringbone.write` also takes a model or relation, which it reads with `find_each`, using a schema
@@ -281,7 +473,7 @@ follows `Model.columns`.
 
 > The inspector and its HTML view are modelled on
 > **[Parquet X-ray](https://huggingface.co/spaces/cfahlgren1/parquet-xray) by cfahlgren1** —
-> the design and the idea are theirs. Go check it out.
+> the design and the idea are theirs. Go check it out. It is amazing!
 
 `Herringbone::Inspector` examines a file using only its footer, page headers, page indexes and
 bloom filter headers. Nothing is decompressed, so it is fast on big files and works for ZSTD and
@@ -304,14 +496,20 @@ CRCs; the results then appear in `summary`, `report`, `to_h` and `to_html`.
 ## Command line
 
 ```
-bin/herringbone cat FILE [N]                    # rows as JSON lines
-bin/herringbone inspect FILE [--pages]          # text report (--pages lists every page header)
-bin/herringbone inspect FILE --json             # everything as JSON
-bin/herringbone inspect FILE --html > out.html  # the HTML page
+bin/herringbone cat FILE [N]                           # rows as JSON lines
+bin/herringbone inspect FILE [--pages]                 # text report (--pages lists every page header)
+bin/herringbone inspect FILE --format=json             # everything as JSON
+bin/herringbone inspect FILE --format=html             # the HTML page, opened in your browser
+bin/herringbone inspect FILE --format=html > out.html  # the HTML page, saved
 ```
 
 Add `--verify-checksums` to any `inspect` form to check page CRCs.
 
+`--format` is `text` (the default), `json` or `html`; `--pages` only goes with `text`.
+In a terminal, `--format=html` writes the page to a temp file, prints its path and opens it with `open` on
+macOS, `start` on Windows or `xdg-open` elsewhere. When stdout is redirected or piped it prints the
+page instead.
+
 ## Supported format features
 
 - Encodings (read and write): PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE, DELTA_BINARY_PACKED,

data/bin/herringbone

@@ -3,14 +3,21 @@
 
 # Usage:
 #   herringbone cat FILE [N]              print rows as JSON lines (optionally only the first N)
-#   herringbone inspect FILE [--pages]    print the schema, row groups, column chunks (and every page header)
-#   herringbone inspect FILE --json       print all of it as JSON
-#   herringbone inspect FILE --html       print a self-contained HTML page showing the file's layout
+#   herringbone inspect FILE [--pages]          print the schema, row groups, column chunks (and every
+#                                               page header); same as --format=text
+#   herringbone inspect FILE --format=json      print all of it as JSON
+#   herringbone inspect FILE --format=html      write a self-contained HTML page showing the file's layout
+#                                               and open it in the browser (printed when stdout is redirected)
 #   add --verify-checksums to any inspect form to read page bodies and check their CRCs
 # `inspect` reads only the footer, page headers and page indexes: no values are decompressed.
 $LOAD_PATH.unshift File.expand_path("../lib", __dir__)
 require "herringbone"
 require "json"
+%w[zstd-ruby brotli snappy xxhash].each do |lib|
+  require lib
+rescue LoadError
+end
+require "tmpdir"
 
 def jsonable(v)
   case v
@@ -25,8 +32,20 @@ def jsonable(v)
   end
 end
 
+# Opens a file with the platform's default application
+# @return [Boolean] whether the opener could be started
+def open_with_default_app(path)
+  command = case RbConfig::CONFIG["host_os"]
+  when /darwin/ then ["open", path]
+  when /mswin|mingw|cygwin/ then ["cmd", "/c", "start", "", path]
+  else ["xdg-open", path]
+  end
+  !!system(*command, out: File::NULL, err: File::NULL, exception: false)
+end
+
 USAGE = "usage: herringbone cat FILE [N]\n" \
-  "       herringbone inspect FILE [--pages | --json | --html] [--verify-checksums]"
+  "       herringbone inspect FILE [--format=text|json|html] [--pages] [--verify-checksums]\n" \
+  "       (--pages only with --format=text)"
 
 command, *rest = ARGV
 flags, args = rest.partition { |a| a.start_with?("--") }
@@ -34,8 +53,11 @@ path, limit = args
 valid = case command
 when "cat" then flags.empty? && path && (limit.nil? || limit.match?(/\A\d+\z/))
 when "inspect"
-  formats = flags & %w[--pages --json --html]
-  path && limit.nil? && (flags - %w[--pages --json --html --verify-checksums]).empty? && formats.size <= 1
+  format_flags, other_flags = flags.partition { |f| f.start_with?("--format=") }
+  format = format_flags.empty? ? "text" : format_flags.first.delete_prefix("--format=")
+  path && limit.nil? && format_flags.size <= 1 && %w[text json html].include?(format) &&
+    (other_flags - %w[--pages --verify-checksums]).empty? &&
+    (format == "text" || !other_flags.include?("--pages"))
 end
 unless valid
   warn USAGE
@@ -51,11 +73,19 @@ begin
     else
       inspector = Herringbone::Inspector.new(io)
       inspector.verify_checksums if flags.include?("--verify-checksums")
-      if flags.include?("--json")
+      case format
+      when "json"
         puts JSON.pretty_generate(inspector.to_h)
-      elsif flags.include?("--html")
-        $stdout.write(inspector.to_html)
-      else
+      when "html"
+        if $stdout.tty?
+          html_path = File.join(Dir.tmpdir, "#{File.basename(path, ".*")}-#{Process.pid}.html")
+          File.binwrite(html_path, inspector.to_html)
+          puts html_path
+          warn "herringbone: could not open the page, open #{html_path} in a browser" unless open_with_default_app(html_path)
+        else
+          $stdout.write(inspector.to_html)
+        end
+      when "text"
         puts inspector.report(pages: flags.include?("--pages"))
       end
     end

data/lib/herringbone/active_record.rb

@@ -92,7 +92,7 @@ module Herringbone
 
         if type == :hstore
           return builder.map(column.name, :string, :string, null: nullable) unless array
-          return builder.list(column.name, null: nullable) { map :element, :string, :string }
+          return builder.list(column.name, null: nullable) { |list| list.map :element, :string, :string }
         end
 
         dsl_type, opts = scalar_type(column, type, sql_type, primary)

data/lib/herringbone/byte_values.rb

@@ -45,6 +45,15 @@ module Herringbone
     # @return [Boolean] true while still dictionary-encoding (not yet switched to raw bytes)
     def dictionary? = !@indices.nil?
 
+    # Short summary for the console, without the values
+    #
+    # @return [String] value count, the dictionary size (or "raw" after switching to raw bytes),
+    #   the width for FIXED_LEN_BYTE_ARRAY and the estimated memory
+    def inspect
+      mode = dictionary? ? "dictionary=#{@dictionary.size}" : "raw"
+      "#<#{self.class.name} size=#{size} #{mode}#{" width=#{@width}" if @width} memory_bytes=#{memory_bytes}>"
+    end
+
     # Appends a value, switching to raw bytes when the dictionary limits are exceeded.
     # @param value [String] bytes of the value; for FIXED_LEN_BYTE_ARRAY it must be +width+ bytes
     #   long (not checked here)