herringbone 0.4.0 → 0.6.0

data/README.md

@@ -7,20 +7,13 @@ A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/)
   LZO, from Hadoop-era files, can be read but not written
 - Full nesting support (structs, lists, maps, any depth) via Dremel record shredding/assembly
 - Reads files from parquet-mr, Arrow, Spark, Impala, DuckDB, Rust writers etc.
-- Optimized reads with batches and pages 
+- Optimized reads with batches and pages
+- Parquet modular encryption (per column and of the footer), interoperable with parquet-mr and Arrow
 - Ruby 3.0+
 
-## Diving in: dumping records in Rails
-
-Stream a relation straight into S3, no temp file needed:
-
-```ruby
-s3 = Aws::S3::TransferManager.new
-s3.upload_stream(bucket: "exports", key: "payments.parquet") do |io|
-  # Schema will be auto-inferred, find_each will be used automatically
-  Herringbone.write(io, Payment.where(status: "settled", created_at: 1.month.ago..))
-end
-```
+This README covers the common cases. The [MANUAL](MANUAL.md) has everything else: reader and
+writer options, schemas and column types, bloom filters, Numo arrays, encryption, the redaction
+rules, type mapping and the inspector.
 
 ## Installation
 
@@ -28,370 +21,142 @@ end
 gem "herringbone"
 ```
 
-Add these libraries to speed up and enable certain functionality:
+Optionally add `snappy` (faster), `zstd-ruby` and `brotli` (more codecs), `xxhash` (faster bloom
+filters) or `numo-narray-alt` (`read(as: :numo)`). Herringbone uses whichever of them are loaded,
+see [Optional gems](MANUAL.md#optional-gems).
 
-```ruby
-gem "snappy"          # native Snappy, 2-3x faster reads and writes of typical files
-gem "zstd-ruby"       # ZSTD
-gem "brotli"          # Brotli
-gem "xxhash"          # faster bloom filters
-gem "numo-narray-alt" # read(as: :numo)
-```
+## Exporting from Rails
 
-Herringbone never requires them itself: it uses whichever are loaded. Bundler.require (as in Rails)
-loads them, otherwise require them yourself:
+Stream a relation straight into S3, no temp file needed:
 
 ```ruby
-require "herringbone"
-require "zstd-ruby"
-require "numo/narray"
+s3 = Aws::S3::TransferManager.new
+s3.upload_stream(bucket: "exports", key: "payments.parquet") do |io|
+  # Schema will be auto-inferred, find_each will be used automatically
+  Herringbone.write(io, Payment.where(status: "settled", created_at: 1.month.ago..))
+end
 ```
 
-On Ruby 3.1 and later, Herringbone works from any Ractor. Pass a schema to a Ractor with
-`Ractor.make_shareable(schema)`. Rows made shareable the same way reach a writing Ractor by
-reference, without being copied:
+Or to a local file:
 
 ```ruby
-writer = Ractor.new("out.parquet", Ractor.make_shareable(schema)) do |path, schema|
-  File.open(path, "wb") do |f|
-    Herringbone::Writer.open(f, schema) do |w|
-      while (row = Ractor.receive) != :done
-        w << row
-      end
-    end
-  end
+File.open("orders.parquet", "wb") do |file|
+  Herringbone.write(file, Order.where(created_at: 1.year.ago..), compression: :zstd)
 end
-rows.each { |row| writer.send(Ractor.make_shareable(row)) }
-writer.send(:done)
 ```
 
-## Reading
+Any Enumerable of Hashes, Structs or Arrays works too; the schema is inferred from the first 1000
+rows. See [ActiveRecord](MANUAL.md#activerecord) and [Writing to S3](MANUAL.md#writing-to-s3).
 
-Herringbone can read from any IO-ish object with random access (the IO should be seekable).
+## Reading
 
 ```ruby
-require "herringbone"
-
 File.open("data.parquet", "rb") do |file|
   reader = Herringbone::Reader.new(file)
-  reader.schema
   reader.num_rows
 
-  reader.each_row { |row| p row }             # Hashes with String keys, nested values as Hash/Array
-  reader.each_batch(1000) { |rows| ... }      # Arrays of up to 1000 rows
-  reader.each_batch(10_000, as: :columns) { |batch| batch["amount"].sum } # { "id" => [...], ... }
-  reader.read                                 # all rows at once; read(as: :columns) for column Arrays
-end
-```
+  reader.each_row { |row| p row }        # Hashes with String keys, nested values as Hash/Array
+  reader.each_batch(1000) { |rows| ... } # Arrays of up to 1000 rows
+  reader.read                            # all rows at once
 
-Reads stream: pages are read and decoded one at a time per column and rows are assembled in
-batches, so memory depends on the batch and page sizes rather than on the row group size (1M
-rows in a single row group peak at about 60–160 MB RSS growth instead of 660 MB). Column-order
-batches (`as: :columns`) skip building a Hash per row and are 25–30% faster.
-
-`Reader.new` takes `keys: :symbol` for Symbol keys (in rows and in structs; map keys stay as
-stored) and `time_zone:` to return timestamps in a zone instead of UTC: a UTC offset (`"+02:00"`,
-or seconds), `"UTC"`, a `TZInfo::Timezone`, or anything responding to `#at` such as `Time.zone`
-in Rails (which yields `ActiveSupport::TimeWithZone`). Zone names like `"Europe/Amsterdam"` work
-when ActiveSupport or TZInfo is loaded. Timestamps stored with `isAdjustedToUTC=false` are
-wall-clock values and stay as they are.
-
-### Selecting rows
-
-`each_row`, `each_batch` and `read` take `columns:`, `where:`, `from:` and `limit:`:
-
-```ruby
-reader.read(columns: %w[id name])                                          # projection
-reader.read(where: { user_id: 42 })
-reader.read(where: { status: %w[paid shipped], created_at: 1.week.ago.. })  # IN, Ranges
-reader.read(where: { "address.city" => "Amsterdam", deleted_at: nil })     # struct members, IS NULL
-reader.read(where: { amount: ->(v) { v && v > 100 } })                      # any callable
-reader.read(from: 1_000_000, limit: 100)                                    # rows 1,000,000..1,000,099
+  reader.read(columns: %w[id name], where: { user_id: 42 })
+  reader.read(where: { status: %w[paid shipped], created_at: 1.week.ago.. })
+end
 ```
 
-All conditions must hold; filtered columns need not be among `columns:`, and any column not inside
-a list or map can be filtered on. Row groups are skipped using min/max statistics, null counts and
-bloom filters, pages using the page index (written by Herringbone, parquet-mr, Arrow and others),
-and `from:` jumps to its row through the offset index; every row read is then checked, so results
-are exact. On a 1M-row file with 20k-row pages, looking up one `id` takes 0.1 s instead of 6.3 s.
-Filters help most on columns the data is sorted or clustered by. `reader.scan_plan(where: ...)`
-shows which row groups and row ranges a read would touch, without reading them.
+Reads stream page by page, and `where:` skips row groups and pages it can rule out using
+statistics, page indexes and bloom filters. See [Reading](MANUAL.md#reading).
 
-### Numo arrays
+## Writing CSV-style
 
-`as: :numo` returns (or yields, with `each_batch`) a Hash of column name => Numo array, and takes
-the same `columns:`, `where:`, `from:` and `limit:`. Add `gem "numo-narray-alt"` (or
-`numo-narray`) to your Gemfile and `require "numo/narray"`: without it, `as: :numo` raises
-`Herringbone::UnsupportedError` naming the gem.
+`SimpleWriter` writes like the CSV gem: name the columns, then append rows. The types are inferred.
 
 ```ruby
-cols = reader.read(as: :numo, columns: %w[id amount])  # { "id" => Numo::Int64, "amount" => Numo::DFloat }
-df = Rover::DataFrame.new(reader.read(as: :numo))       # no conversion needed
+File.open("people.parquet", "wb") do |file|
+  Herringbone::SimpleWriter.open(file) do |sw|
+    sw.headers!(:id, :name, :age)
+    sw << [123, "John", 12]
+    sw << { id: 124, name: "Jane" } # Hashes work too
+  end
+end
 ```
 
-Flat numeric and boolean columns are decoded from the page bytes without a Ruby object per value:
-two numeric columns of 1M uncompressed rows read in about 25 ms, against 50 ms with `as: :columns`.
+To control the types, declare a schema and use `Herringbone::Writer`, see
+[Writing](MANUAL.md#writing).
 
-| Parquet | Numo |
-|---|---|
-| INT32, INT64 (also TIME) | `Int32`, `Int64` |
-| INT(8/16) signed; INT(8/16/32/64) unsigned | `Int8`, `Int16`; `UInt8` … `UInt64` |
-| FLOAT, FLOAT16, DOUBLE | `SFloat`, `SFloat`, `DFloat` (nulls are NaN) |
-| integers with nulls | `DFloat` with NaN (exact up to 2**53) |
-| BOOLEAN | `Bit`; with nulls `RObject` of true/false/nil |
-| list of numbers, every row the same length and no nulls | 2-D `[rows, length]` (e.g. embeddings) |
-| strings, binary, decimals, dates, timestamps, UUIDs, structs, maps, other lists | `RObject` of the values `read` returns |
+## Encrypting
 
-Whether a column has nulls (or a list column is rectangular) is decided from the rows read, so
-with `each_batch` it can differ between batches.
-
-## Writing
+Parquet files tend to wander off: to S3 buckets, laptops, other teams. Encrypting them takes one
+key and one line, no KMS, no Hadoop configuration. Make a key once and keep it with your other
+secrets:
 
 ```ruby
-schema = Herringbone::Schema.define do
-  int64 :id, null: false
-  string :name
-  enum :status, values: %w[pending paid shipped]
-  list :tags, :string
-  map :scores, :string, :double
-  struct :address do
-    string :city
-    string :zip
-  end
-  decimal :price, precision: 12, scale: 2
-  json :payload
-  timestamp :created_at
-end
-
-File.open("out.parquet", "wb") do |file|
-  Herringbone::Writer.open(file, schema) do |w|
-    w << { "id" => 1, "name" => "Anna", "status" => "paid", "tags" => ["a", "b"],
-           "scores" => { "x" => 1.5 }, "address" => { "city" => "Amsterdam" },
-           "price" => BigDecimal("9.99"), "payload" => { "any" => ["json"] }, "created_at" => Time.now }
-    w << { id: 2, status: :pending }  # Symbol keys and values work; missing keys are nulls
-    w << [3, "Bo", nil, nil, nil, nil, nil, nil, nil, nil] # Arrays in schema order
-    w << order                         # anything with #attributes (ActiveRecord) or #to_h (Struct, Data)
-  end
-end
+Herringbone::Key.generate.hex # => "9f86d081884c7d65..." - put it in your credentials or ENV
 ```
 
-`Herringbone.write(io, rows)` writes an Enumerable of rows in one go, inferring the schema from the
-first 1000 rows unless `schema:` is given; fields declared in a block replace inferred ones:
-`Herringbone.write(io, rows) { json :payload }`. The rows are iterated once, holding back only
-those first 1000, so lazy Enumerators and cursors that can't be rewound work. A later row that
-doesn't fit the inferred types raises `Herringbone::SchemaMismatch`, which explains what was
-inferred and how to declare the column, and leaves the file unfinished.
-
-The writer writes to any IO that responds to `#write` (a `File`, `StringIO`, `Tempfile`, socket or
-pipe), sequentially, and never seeks, rewinds or closes it (it does switch it to binary mode). If
-the `Writer.open` block raises (or `#abort` is called), no footer is written and what was written
-so far is left for you to discard. To replace a file only once it is complete, write to a temporary
-file and rename it.
-
-Column types: `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double float16
-string binary json bson enum uuid date int96 time timestamp decimal fixed`, plus `struct`, `list`
-and `map`; `column :name, :int32` declares one by name. Fields are nullable unless `null: false`
-is given; list elements unless `element_null: false`, map values unless `value_null: false`.
-Nested lists: `list :matrix do list :element, :double end`. `time` and `timestamp` take `unit:`
-(`:millis`, `:micros`, `:nanos`) and `utc:`.
-
-`enum` is a string column. `values:` restricts what may be written, and also takes a Rails-style
-Hash (`values: Order.statuses`), in which case both labels and stored integers are accepted and
-the label is written. `parquet_enum: true` adds the Parquet ENUM annotation (pyarrow and pandas
-read such columns as binary, which is why it is off by default).
-
-Columns accept the values Ruby and Rails code usually has at hand:
-
-| column | accepts |
-|---|---|
-| `date` | `Date`, `Time`/`DateTime` (their date), `"2024-05-01"` |
-| `timestamp` | `Time`, `DateTime`, `ActiveSupport::TimeWithZone`, `Date` (midnight UTC), ISO-8601 strings, Integers in the column's unit |
-| `time` | `Time` (its time of day, as Rails returns for `time` columns), `"13:45:30.25"`, Integers |
-| `json` | Strings as-is, anything else through `JSON.generate` |
-| `string`, `enum` | Strings, Symbols, anything with `to_s` |
-| `boolean` | `true`/`false`, `1`/`0`, `"t"`/`"f"`, `"true"`/`"false"`, `"yes"`/`"no"` |
-| integers | Integers, whole-number Floats/BigDecimals/Rationals, numeric Strings; out-of-range values raise |
-| `decimal` | `BigDecimal`, Integer, Rational, Float, numeric Strings |
-| `uuid` | Strings with or without dashes, or 16 raw bytes |
-
-Values that don't fit raise `Herringbone::EncodeError` naming the row number and column path; the
-failed row is discarded and the writer can carry on.
-
-Writer options:
-
-| option | default | |
-|---|---|---|
-| `compression` | `:snappy` | `:none`, `:snappy`, `:gzip`, `:lz4` (LZ4_RAW), `:lz4_hadoop`, `:zstd`, `:brotli` |
-| `row_group_bytes` | 16MB | flush a row group once the buffered values take about this much memory, which bounds memory use (a 15-column table peaks around 290 MB RSS) |
-| `row_group_rows` | none | also flush after this many rows |
-| `page_bytes` | 1MB | approximate data page size |
-| `page_rows` | `20_000` | at most this many rows per data page |
-| `data_page_version` | `1` | `1` or `2` |
-| `dictionary` | `true` | `false`, or an Array of column paths to dictionary-encode |
-| `encodings` | `{}` | e.g. `{ "id" => :delta_binary_packed, "x" => :byte_stream_split }` |
-| `metadata` | `{}` | footer key/value metadata, read back with `reader.metadata` |
-| `bloom_filters` | none | `true`, an Array of column paths, or `{ "path" => { ndv:, fpp:, max_bytes: } }` |
-
-### Coming from CSV
-
-`SimpleWriter` writes like the CSV gem: name the columns, then append Arrays. The types are
-inferred from the first 1000 rows, as above.
+Then write with it:
 
 ```ruby
 File.open("people.parquet", "wb") do |file|
   Herringbone::SimpleWriter.open(file) do |sw|
-    sw.headers!(:id, :name, :age)
-    sw << [123, "John", 12]
-    sw << { id: 124, name: "Jane" } # Hashes work too
+    sw.encrypt!(key: ENV["PARQUET_KEY"])
+    sw.headers!(:id, :name, :email)
+    sw << [1, "John", "john@example.com"]
   end
 end
 ```
 
-Unlike CSV, headers are required. A column that holds more than one type can be declared up front:
-`Herringbone::SimpleWriter.new(io) { string :code }` (then call `close` when done).
-
-### Statistics, page indexes and bloom filters
-
-Every column chunk gets min/max/null-count statistics and the Parquet page index (per-page min/max
-and row offsets), which Herringbone, DuckDB, Spark, Trino, Arrow and DataFusion use to skip pages.
-With 20,000 rows per page, `WHERE id BETWEEN ...` on a sorted column reads a handful of pages
-instead of the whole row group, so sort rows by the columns you filter on.
-
-Min/max cannot rule out a value inside a row group's range, the usual case for unsorted IDs,
-emails or UUIDs; a bloom filter can. They are off by default. `bloom_filters: ["user_id", "email"]`
-(or `true` for every non-boolean column) writes Parquet's split block bloom filters, sized from
-each row group's distinct values at a false positive probability of 1% (`fpp:`, up to `max_bytes:`,
-default 1MB) unless `ndv:` gives the number of distinct values. Nested leaves are named by their
-dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is loaded:
-a million rows take about 1.2 s longer to write with a filter on an INT64 column and 2.8 s longer
-with one on a ~22-byte string column, and about 0.5 s longer with `xxhash`.
-
-### Writing to S3
-
-Parquet keeps its metadata in a footer, so a file can be streamed into an S3 multipart upload
-without a local copy, using `upload_stream` from `aws-sdk-s3`:
+and read with it:
 
 ```ruby
-s3 = Aws::S3::TransferManager.new # aws-sdk-s3 1.197+; before that, Aws::S3::Object#upload_stream
-s3.upload_stream(bucket: "exports", key: "events.parquet", part_size: 16 * 1024 * 1024) do |io|
-  Herringbone::Writer.open(io, schema) do |w|
-    events.each { |event| w << event }
-  end
-end
-s3.upload_stream(bucket: "exports", key: "orders.parquet") { |io| Herringbone.write(io, Order.all) }
+Herringbone::Reader.new(file, decryption: ENV["PARQUET_KEY"]).read
 ```
 
-If the block raises, the SDK aborts the multipart upload and raises `Aws::S3::MultipartUploadError`,
-so no partial object is left. S3 allows at most 10,000 parts, which with the default 5MB parts caps
-the file at about 48GB; raise `part_size:` for bigger files.
+The schema, the values and the statistics are all encrypted (AES-256-GCM); without the key the
+file is just noise. `encryption: key` does the same for `Herringbone.write` and
+`Herringbone::Writer`. Your colleagues can open the file with the same key in pyarrow 25+
+(`pq.read_table(path, decryption_properties=pyarrow.parquet.encryption.create_decryption_properties(bytes.fromhex(key)))`),
+Arrow, DataFusion, Trino or Spark, and `herringbone inspect people.parquet` asks for the key.
+When the time comes to rotate, read with all your keys, `decryption: [new_key, old_key]`, and
+each file finds its own. See [Encryption](MANUAL.md#encryption) for per-column keys, plaintext
+footers and which tools read what.
 
-## ActiveRecord
+## Redacting
 
-`Herringbone.write` also takes a model or relation, which it reads with `find_each`, using a schema
-built from the model's columns. It returns the number of rows written:
+`Herringbone.redact` rewrites a file with rows removed or values replaced, for GDPR "forget me"
+requests and pseudonymization. The parts of the file nothing touches are copied byte for byte.
 
 ```ruby
-File.open("orders.parquet", "wb") do |file|
-  Herringbone.write(file, Order.where(created_at: 1.year.ago..), compression: :zstd)
+# Forget me: remove the rows
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).delete
 end
-schema = Herringbone::Schema.from_active_record(Order, only: %w[id status total])
-Herringbone.write(io, Order, schema: schema)
-```
 
-`from_active_record` takes `only:`, `except:` and `parquet_enum: true`. Rails is not a dependency:
-it only calls `columns`, `primary_key` and `defined_enums` on the model. Enum attributes are
-written as their labels, and the writer rejects values outside the enum.
-
-| Column | Parquet |
-|---|---|
-| `integer` | `int16`/`int32`/`int64` by SQL type (`smallint`, `integer`, `bigint`...) or limit; `uintN` for `unsigned`; primary keys are always `int64` |
-| `float` | `double` (`float` for Postgres `float4`) |
-| `decimal` | `decimal(precision, scale)`; `decimal(38, 9)` when the column has no precision |
-| `boolean`, `date`, `binary`, `uuid` | same |
-| `json`, `jsonb` | `json` |
-| `datetime`, `timestamp`, `timestamptz` | `timestamp` (microseconds, UTC) |
-| `time` | `time` (microseconds) |
-| `hstore` | `map` of `string` to `string` |
-| enum attributes | `string` (or `enum`) |
-| `string`, `text`, `citext`, anything else | `string` |
-| Postgres array columns | `list` of the element type |
-
-Columns declared `NOT NULL` (and primary keys) are required, all others nullable. Column order
-follows `Model.columns`.
-
-## Type mapping
-
-| Parquet | Ruby |
-|---|---|
-| BOOLEAN | `true`/`false` |
-| INT32/INT64 (incl. signed/unsigned INTEGER) | `Integer` |
-| FLOAT, DOUBLE, FLOAT16 | `Float` |
-| STRING, ENUM, JSON | `String` (UTF-8) |
-| BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY, BSON | `String` (binary) |
-| DATE | `Date` (proleptic Gregorian) |
-| TIMESTAMP, INT96 | `Time` (UTC) |
-| TIME | `Integer` in the column's unit since midnight |
-| DECIMAL | `BigDecimal` |
-| UUID | `String` like `"0f1e2d3c-..."` |
-| struct / list / map | `Hash` / `Array` / `Hash` |
-
-## Inspecting files
-
-> The inspector and its HTML view are modelled on
-> **[Parquet X-ray](https://huggingface.co/spaces/cfahlgren1/parquet-xray) by cfahlgren1** —
-> the design and the idea are theirs. Go check it out. It is amazing!
-
-`Herringbone::Inspector` examines a file using only its footer, page headers, page indexes and
-bloom filter headers. Nothing is decompressed, so it is fast on big files and works for ZSTD and
-Brotli files without the codec gems.
+# Forget me, but keep the row for accounting: blank the personal columns
+Herringbone.redact(input, output) do |r|
+  r.where(user_id: 42).replace(email: nil, name: nil, address: nil)
+end
 
-```ruby
-File.open("data.parquet", "rb") do |file|
-  inspector = Herringbone::Inspector.new(file)
-  inspector.summary      # size, rows, row groups, codecs, page index / bloom filter presence...
-  inspector.row_groups[0].column("name").pages # page headers; also statistics, column_index...
-  puts inspector.report  # the schema, every column chunk, key/value metadata (Arrow schema decoded)
-  inspector.to_h         # all of it, JSON-serializable
-  inspector.to_html      # one self-contained HTML page with a to-scale byte map of the file
+# Pseudonymize a column across the whole file, mask another, drop a third
+Herringbone.redact(input, output) do |r|
+  r.replace(:email) { |email| OpenSSL::HMAC.hexdigest("SHA256", KEY, email.downcase) }
+  r.replace(:phone) { |phone| phone && "***#{phone[-3..]}" }
+  r.drop :ssn, :ip_address
 end
 ```
 
-`inspector.verify_checksums` reads every page body (still without decompressing) to check the page
-CRCs; the results then appear in `summary`, `report`, `to_h` and `to_html`.
+See [Redaction](MANUAL.md#redaction) for reusable redactions, reports and recipes.
 
-## Command line
+## Looking inside a file
 
 ```
-bin/herringbone cat FILE [N]                           # rows as JSON lines
-bin/herringbone inspect FILE [--pages]                 # text report (--pages lists every page header)
-bin/herringbone inspect FILE --format=json             # everything as JSON
-bin/herringbone inspect FILE --format=html             # the HTML page, opened in your browser
-bin/herringbone inspect FILE --format=html > out.html  # the HTML page, saved
+bin/herringbone cat FILE [N]                 # rows as JSON lines
+bin/herringbone inspect FILE                 # schema, row groups, column chunks
+bin/herringbone inspect FILE --format=html   # a byte map of the file, opened in your browser
 ```
 
-Add `--verify-checksums` to any `inspect` form to check page CRCs.
-
-`--format` is `text` (the default), `json` or `html`; `--pages` only goes with `text`.
-In a terminal, `--format=html` writes the page to a temp file, prints its path and opens it with `open` on
-macOS, `start` on Windows or `xdg-open` elsewhere. When stdout is redirected or piped it prints the
-page instead.
-
-## Supported format features
-
-- Encodings (read and write): PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE, DELTA_BINARY_PACKED,
-  DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY, BYTE_STREAM_SPLIT; legacy BIT_PACKED levels (read)
-- Data page v1 and v2, dictionary pages, page CRCs (written)
-- Page indexes and split block bloom filters (read and written)
-- Legacy list and map layouts per the Parquet backward-compatibility rules
-- Not supported: encryption, column chunks in external files
+See [Inspecting files](MANUAL.md#inspecting-files) and [Command line](MANUAL.md#command-line).
 
 ## Development
 
-```
-bundle install
-bundle exec rake test
-HERRINGBONE_PYTHON=/path/to/python-with-pyarrow bundle exec rake test   # also run pyarrow interop tests
-```
-
-`test/fixtures/parquet-testing` holds files from [apache/parquet-testing](https://github.com/apache/parquet-testing)
-(Apache-2.0); expectations for them were generated with pyarrow, see `test/fixtures/generate_expectations.py`.
+See [Development](MANUAL.md#development) and [CONTRIBUTING](CONTRIBUTING.md).

data/bin/herringbone

@@ -10,6 +10,13 @@
 #                                               and open it in the browser (printed when stdout is redirected)
 #   add --verify-checksums to any inspect form to read page bodies and check their CRCs
 # `inspect` reads only the footer, page headers and page indexes: no values are decompressed.
+#
+# Encrypted files: both commands take --key=KEY (a key matched by its fingerprint id, or used for
+# the footer when it is the only one), --key=KEY_ID=KEY, --footer-key=KEY and
+# --column-key=PATH=KEY (all repeatable but --footer-key), and --aad-prefix=PREFIX. KEY is hex, or
+# base64:..., or raw:... for the bytes as typed (a 16, 24 or 32-character String that is not
+# valid hex is taken as is too). Keys that are needed but not given are asked for on stdin
+# (without echo in a terminal; an empty answer skips the key), unless --no-prompt is given.
 $LOAD_PATH.unshift File.expand_path("../lib", __dir__)
 require "herringbone"
 require "json"
@@ -18,6 +25,7 @@ require "json"
 rescue LoadError
 end
 require "tmpdir"
+require "io/console"
 
 def jsonable(v)
   case v
@@ -43,12 +51,81 @@ def open_with_default_app(path)
   !!system(*command, out: File::NULL, err: File::NULL, exception: false)
 end
 
-USAGE = "usage: herringbone cat FILE [N]\n" \
-  "       herringbone inspect FILE [--format=text|json|html] [--pages] [--verify-checksums]\n" \
-  "       (--pages only with --format=text)"
+# A key as given on the command line or at the prompt: hex, base64:..., raw:..., or the bytes as
+# typed when they are not valid hex but have the size of a key
+# @return [String] the key
+# @raise [ArgumentError] when it is none of those, or not 16, 24 or 32 bytes long
+def parse_key(text, what)
+  text = text.strip
+  bytes = if text.start_with?("base64:")
+    text.delete_prefix("base64:").unpack1("m0")
+  elsif text.start_with?("raw:")
+    text.delete_prefix("raw:")
+  elsif text.match?(/\A(\h\h)+\z/) && Herringbone::Encryption::KEY_SIZES.include?(text.size / 2)
+    [text].pack("H*")
+  else
+    text
+  end
+  Herringbone::Encryption.check_key!(bytes, what)
+end
+
+# Asks for a key on stderr and reads it from stdin, without echo when stdin is a terminal
+# @return [String, nil] the key, nil when the answer is empty or stdin is exhausted
+def ask_key(metadata, owner)
+  what = (owner == :footer) ? "the footer" : "column #{owner}"
+  $stderr.print "Key for #{what}#{" (key metadata #{metadata.inspect})" if metadata}, hex: "
+  line = $stdin.tty? ? $stdin.noecho(&:gets) : $stdin.gets
+  $stderr.puts
+  return nil if line.nil? || line.strip.empty?
+  parse_key(line, "the key for #{what}")
+rescue ArgumentError => e
+  raise unless $stdin.tty?
+  warn "herringbone: #{e.message}"
+  retry
+end
+
+# The decryption: option from the key flags
+# @return [Hash{Symbol => Object}]
+def decryption_from(key_flags, prompt)
+  by_metadata = {}
+  plain = []
+  decryption = {columns: {}}
+  key_flags.each do |flag|
+    name, value = flag.split("=", 2)
+    case name
+    when "--footer-key" then decryption[:footer_key] = parse_key(value.to_s, "--footer-key")
+    when "--aad-prefix" then decryption[:aad_prefix] = value.to_s
+    when "--key"
+      target, key = value.to_s.split("=", 2)
+      if key.nil? || value.start_with?("base64:", "raw:")
+        plain << Herringbone::Key.new(parse_key(value.to_s, "--key"))
+        by_metadata[plain.last.id] = plain.last.bytes
+      else
+        by_metadata[target] = parse_key(key, "--key=#{target}")
+      end
+    when "--column-key"
+      target, key = value.to_s.split("=", 2)
+      raise ArgumentError, "--column-key expects --column-key=PATH=KEY" unless key
+      decryption[:columns][target] = parse_key(key, "--column-key=#{target}")
+    end
+  end
+  decryption[:footer_key] ||= plain.first.bytes if plain.size == 1
+  decryption[:keys] = ->(metadata, owner) { by_metadata.fetch(metadata) { prompt ? ask_key(metadata, owner) : nil } }
+  decryption
+end
+
+USAGE = "usage: herringbone cat FILE [N] [KEY OPTIONS]\n" \
+  "       herringbone inspect FILE [--format=text|json|html] [--pages] [--verify-checksums] [KEY OPTIONS]\n" \
+  "       (--pages only with --format=text)\n" \
+  "key options, for encrypted files: --key=KEY --key=KEY_ID=KEY --footer-key=KEY --column-key=PATH=KEY\n" \
+  "       --aad-prefix=PREFIX --no-prompt (KEY is hex, base64:... or raw:...)"
+
+KEY_FLAGS = %w[--footer-key --column-key --key --aad-prefix].freeze
 
 command, *rest = ARGV
 flags, args = rest.partition { |a| a.start_with?("--") }
+key_flags, flags = flags.partition { |f| KEY_FLAGS.include?(f.split("=", 2).first) }
+prompt = !flags.delete("--no-prompt")
 path, limit = args
 valid = case command
 when "cat" then flags.empty? && path && (limit.nil? || limit.match?(/\A\d+\z/))
@@ -64,14 +141,15 @@ unless valid
   exit 1
 end
 
-# Parquet and file errors are reported in one line, without a backtrace
+# Parquet, file and key errors are reported in one line, without a backtrace
 begin
+  decryption = decryption_from(key_flags, prompt)
   File.open(path, "rb") do |io|
     if command == "cat"
       n = limit&.to_i
-      Herringbone::Reader.new(io).each_row(limit: n) { |row| puts JSON.generate(jsonable(row)) }
+      Herringbone::Reader.new(io, decryption: decryption).each_row(limit: n) { |row| puts JSON.generate(jsonable(row)) }
     else
-      inspector = Herringbone::Inspector.new(io)
+      inspector = Herringbone::Inspector.new(io, decryption: decryption)
       inspector.verify_checksums if flags.include?("--verify-checksums")
       case format
       when "json"
@@ -90,7 +168,8 @@ begin
       end
     end
   end
-rescue Herringbone::Error, SystemCallError => e
+rescue Herringbone::Error, SystemCallError, ArgumentError => e
   warn "herringbone: #{e.message}"
+  warn "herringbone: give keys with --footer-key=KEY, --column-key=PATH=KEY or --key=KEY_METADATA=KEY" if e.is_a?(Herringbone::DecryptionError)
   exit 1
 end

data/lib/herringbone/active_record.rb

@@ -92,7 +92,7 @@ module Herringbone
 
         if type == :hstore
           return builder.map(column.name, :string, :string, null: nullable) unless array
-          return builder.list(column.name, null: nullable) { map :element, :string, :string }
+          return builder.list(column.name, null: nullable) { |list| list.map :element, :string, :string }
         end
 
         dsl_type, opts = scalar_type(column, type, sql_type, primary)