herringbone 0.9.1 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 14cacf678c274af811c0ba7cf8d2810f251655fe8c0f8a80411d714170ade738
-  data.tar.gz: c9ae864f1f2a15ba9972bd742556a909a534077aac7251c26841054b3e173e2e
+  metadata.gz: 6bcc4cf1c07a6664ec7a481ff9f889e8e9b3c63b24a4bb269bb8019562da3893
+  data.tar.gz: 6d8977ddd7b16c9796bbe5bf8e66951542959da197c4503b60b4fecbb95d9b55
 SHA512:
-  metadata.gz: 9767ea6bb316dd5271ab1ced6476587036aaaaae7439b95d9d39ad721f2ada4af4514d75d1274d030c7ebe041a2f76b22a69ee34bd35b6a49f6941f9ed75b3ce
-  data.tar.gz: 2487e14f78f16a8c2a823afc11b131f2bca9019ce54e4e6f52d423c1954715872ad2e9f1d9f5796674fdd70b34cd8780b5786b2b251bfa91f35f7a901d61d937
+  metadata.gz: ed4935d82ff796d9bbed85b44d67a36081569fab77bccfea3afc108349514809f20dc8f488c0145109c66b9ce8d5a5390d2b7888ddc07f417f826d338092c5c0
+  data.tar.gz: 75d11fc670e04725f7cb79314275fb32d9fb7b4c79e5530528a1300a3692591dfb6eca26f3d50f61bd706efa8ec15d74a53ac299b61a93d6c9e6604f28d9cbc1

data/CHANGELOG.md

@@ -2,6 +2,23 @@
 
 ## Unreleased
 
+- `Herringbone::Writer.open(io, headers: [:id, :name]) { |w| w << [1, "John"] }` infers the column
+  types when no schema is given, and `Herringbone.write(io, rows, headers: [:id, :name])` writes
+  Array rows with inferred types.
+- `Herringbone::FormatError` is raised for a page that cannot be decompressed, naming the codec and
+  the column, instead of `Zlib::DataError`, zstd-ruby's `RuntimeError` or an internal codec error.
+- `Herringbone::Reader` raises `Herringbone::FormatError` for a footer or page header that is
+  damaged in a way that still decodes, instead of a `NoMethodError`, `KeyError` or `TypeError`.
+- `Herringbone::Thrift::Error` is a `Herringbone::Error`, so `rescue Herringbone::Error` covers
+  footer metadata that cannot be written.
+- The gem ships its type signatures: `sig/herringbone.rbs` for RBS tools such as Steep (now valid
+  under `rbs validate`) and `rbi/herringbone.rbi` for Sorbet.
+- **Breaking:** `Herringbone::SimpleWriter` is removed: use `Herringbone::Writer.open` with
+  `headers:` (and `encryption:` instead of `encrypt!(key:)`).
+- **Breaking:** `Herringbone::Inspector::ArrowSchema`, `Herringbone::Redaction::Statement`,
+  `Redaction#statements` and `#drops`, `Schema::Column#converter` and `#encoder` and
+  `Schema::Builder#nodes` are marked `@api private`; nothing replaces them.
+
 ## 0.9.1
 
 - `Herringbone.combine` works in Ractors other than the main one, and so does reading ZSTD without

data/MANUAL.md

@@ -16,7 +16,7 @@ The full reference for Herringbone. For a quick start, see the [README](README.m
   - [Column types](#column-types)
   - [Accepted values](#accepted-values)
   - [Writer options](#writer-options)
-  - [SimpleWriter: coming from CSV](#simplewriter-coming-from-csv)
+  - [Coming from CSV](#coming-from-csv)
   - [Statistics, page indexes and bloom filters](#statistics-page-indexes-and-bloom-filters)
   - [Writing to S3](#writing-to-s3)
   - [Writing to Google Cloud Storage](#writing-to-google-cloud-storage)
@@ -41,6 +41,8 @@ The full reference for Herringbone. For a quick start, see the [README](README.m
 - [Inspecting files](#inspecting-files)
 - [Command line](#command-line)
 - [Supported format features](#supported-format-features)
+- [Errors](#errors)
+- [Public API and type signatures](#public-api-and-type-signatures)
 - [Development](#development)
 
 ## Installation
@@ -301,34 +303,27 @@ failed row is discarded and the writer can carry on.
 | `metadata` | `{}` | footer key/value metadata, read back with `reader.metadata` |
 | `bloom_filters` | none | `true`, an Array of column paths, or `{ "path" => { ndv:, fpp:, max_bytes: } }` |
 
-### SimpleWriter: coming from CSV
+### Coming from CSV
 
-`SimpleWriter` writes like the CSV gem: name the columns, then append Arrays. The types are
-inferred from the first 1000 rows, as with `Herringbone.write`.
+`Herringbone::Writer.open` without a schema infers the column types from the first 1000 rows, and
+`headers:` names the columns of Array rows, like the CSV gem's `headers:`:
 
 ```ruby
 File.open("people.parquet", "wb") do |file|
-  Herringbone::SimpleWriter.open(file) do |sw|
-    sw.headers!(:id, :name, :age)
-    sw << [123, "John", 12]
-    sw << { id: 124, name: "Jane" } # Hashes work too
+  Herringbone::Writer.open(file, headers: [:id, :name, :age]) do |w|
+    w << [123, "John", 12]
+    w << { id: 124, name: "Jane" } # Hashes work too; missing columns are nulls
   end
 end
 ```
 
-Unlike CSV, headers are required. A column that holds more than one type can be declared up front:
-`Herringbone::SimpleWriter.new(io) { |s| s.string :code }` (then call `close` when done).
+Without a block it returns the writer, to keep around and `close` when done. Unlike CSV, headers
+are required for Array rows: a Parquet file always has named columns. With a schema, Array rows
+list their values in schema order, so `headers:` can't be given as well. For a column that holds
+more than one type, pass the schema.
 
-`encrypt!` encrypts the file with one key (see [Encryption](#encryption)): give it the key's hex
-or a `Herringbone::Key` (`key: Herringbone::Key.generate` for a new one; keep it).
-
-```ruby
-Herringbone::SimpleWriter.open(file) do |sw|
-  sw.encrypt!(key: ENV["PARQUET_KEY"])
-  sw.headers!(:id, :name)
-  sw << [1, "John"]
-end
-```
+Rows you already have go in one call, where the block can declare just the columns that need it:
+`Herringbone.write(io, rows, headers: [:id, :code]) { |s| s.string :code }`.
 
 ### Statistics, page indexes and bloom filters
 
@@ -945,6 +940,37 @@ prints the page instead.
   footers, footer and column keys, AAD prefixes
 - Not supported: column chunks in external files
 
+## Errors
+
+Every error Herringbone raises on purpose is a `Herringbone::Error`:
+
+- `Herringbone::FormatError`: the file is not valid Parquet, or a part of it is damaged. A page
+  that cannot be decompressed raises it too, naming the codec and the column, and so does
+  metadata with a field missing; the error that gave the damage away (`Zlib::DataError`, a
+  `NoMethodError` on nil) is its `cause`.
+- `Herringbone::EncodeError`: a value does not fit its column; it carries `row`, `column` and
+  `value`. `Herringbone::SchemaMismatch` is the same for a schema that was inferred.
+- `Herringbone::UnsupportedError`: a Parquet feature Herringbone does not implement.
+  `Herringbone::MissingCodecError` says which gem a codec needs.
+- `Herringbone::IncompatibleSchema`: schemas that cannot be united, intersected or combined.
+- `Herringbone::DecryptionError`: a key is missing or wrong, or the encrypted bytes were changed.
+- `Herringbone::Thrift::Error`: footer metadata that cannot be written, such as a `metadata:`
+  value that is not valid UTF-8.
+
+Invalid arguments and options raise `ArgumentError`.
+
+## Public API and type signatures
+
+The public API is what the YARD docs show without an `@api private` tag: `Herringbone.write`,
+`redact`, `combine` and `codecs`, `Reader`, `Writer`, `Schema`, `Key`, the encryption
+configurations, `Redaction`, `Inspector`, the errors and the `herringbone` executable. Classes,
+modules and methods tagged `@api private` are there for the library itself and can change in any
+release.
+
+The gem ships type signatures for [RBS](https://github.com/ruby/rbs) (`sig/herringbone.rbs`, for
+Steep and other RBS tools) and for [Sorbet](https://sorbet.org) (`rbi/herringbone.rbi`), both
+generated from the YARD docs.
+
 ## Development
 
 ```

data/README.md

@@ -71,18 +71,21 @@ statistics, page indexes and bloom filters. See [Reading](MANUAL.md#reading).
 
 ## Writing CSV-style
 
-`SimpleWriter` writes like the CSV gem: name the columns, then append rows. The types are inferred.
+Name the columns with `headers:` and append the rows as Arrays, like with the CSV gem. The types are
+inferred.
 
 ```ruby
 File.open("people.parquet", "wb") do |file|
-  Herringbone::SimpleWriter.open(file) do |sw|
-    sw.headers!(:id, :name, :age)
-    sw << [123, "John", 12]
-    sw << { id: 124, name: "Jane" } # Hashes work too
+  Herringbone::Writer.open(file, headers: [:id, :name, :age]) do |w|
+    w << [123, "John", 12]
+    w << { id: 124, name: "Jane" } # Hashes work too
   end
 end
 ```
 
+Rows you already have in an Array or an Enumerable go in one call:
+`Herringbone.write(file, rows, headers: [:id, :name, :age])`.
+
 To control the types, declare a schema and use `Herringbone::Writer`, see
 [Writing](MANUAL.md#writing).
 
@@ -100,10 +103,8 @@ Then write with it:
 
 ```ruby
 File.open("people.parquet", "wb") do |file|
-  Herringbone::SimpleWriter.open(file) do |sw|
-    sw.encrypt!(key: ENV["PARQUET_KEY"])
-    sw.headers!(:id, :name, :email)
-    sw << [1, "John", "john@example.com"]
+  Herringbone::Writer.open(file, headers: [:id, :name, :email], encryption: ENV["PARQUET_KEY"]) do |w|
+    w << [1, "John", "john@example.com"]
   end
 end
 ```

data/lib/herringbone/compression.rb

@@ -147,7 +147,8 @@ module Herringbone
     # @return [String] decompressed bytes (binary)
     # @raise [UnsupportedError] for a codec herringbone does not implement
     # @raise [MissingCodecError] when the codec's gem is not loaded
-    # @raise [FormatError] when the decompressed size does not match +uncompressed_size+
+    # @raise [FormatError] when the data is corrupt, or the decompressed size does not match
+    #   +uncompressed_size+
     def decompress(codec, data, uncompressed_size)
       return "".b if uncompressed_size.zero? && data.empty?
       out = case codec
@@ -167,6 +168,13 @@ module Herringbone
         raise FormatError, "Decompressed #{out.bytesize} bytes, expected #{uncompressed_size}"
       end
       out
+    rescue Error
+      raise
+    rescue => e
+      # zstd-ruby raises a bare RuntimeError and zlib and brotli have errors of their own, so
+      # there is no narrower list to rescue
+      raise FormatError, "Cannot decompress #{data.bytesize} bytes of #{Format::Codec::NAMES[codec] || codec} data, " \
+        "they are corrupt or cut short: #{e.message} (#{e.class})"
     end
 
     # Compresses a page body. Given several Strings, compresses their concatenation; GZIP, ZSTD

data/lib/herringbone/encryption_configuration.rb

@@ -2,7 +2,7 @@
 
 module Herringbone
   # How Writer encrypts a file (Parquet modular encryption), checked as soon as it is built. The
-  # +encryption:+ option of Writer, Herringbone.write, SimpleWriter and Herringbone.redact takes
+  # +encryption:+ option of Writer, Herringbone.write and Herringbone.redact takes
   # one, or a Hash of the same keywords, which is turned into one with EncryptionConfiguration.from.
   #
   #   config = Herringbone::EncryptionConfiguration.new(

data/lib/herringbone/inferring_writer.rb

@@ -10,13 +10,18 @@ module Herringbone
   # A row that does not fit the inferred schema raises SchemaMismatch and stops the writer: the
   # file is left unfinished (no footer) rather than written with rows missing.
   #
-  # Used by Herringbone.write and SimpleWriter.
+  # With +headers:+, rows are Arrays of values in header order (or Hashes keyed by header), and
+  # only the held-back rows are turned into Hashes for inference.
   #
-  # @api private
+  # Writer.open returns one when it is given no schema, and Herringbone.write uses one when it has
+  # to infer the schema.
   class InferringWriter
+    # @api private
     # @param io [IO, #write] destination, passed to Writer.new
     # @param fix [String] how a caller declares a column, for SchemaMismatch messages; +%s+ is
     #   replaced by a declaration such as +string :age+
+    # @param headers [Array<String, Symbol>, nil] column names, in the order row Arrays list their
+    #   values; nil for rows the Writer takes as they are
     # @param options [Hash{Symbol => Object}] Writer options
     # @option options [Symbol] :compression (:snappy) codec, checked before any row is read
     #   (the other Writer options are passed on as well)
@@ -24,14 +29,15 @@ module Herringbone
     # @yieldparam sample [Array] the rows held back, at most Schema::INFER_SAMPLE
     # @yieldreturn [Schema]
     # @raise [MissingCodecError] when the codec's optional gem is not loaded
-    # @raise [ArgumentError] for a compression level the codec does not take, or invalid
-    #   encryption settings
-    def initialize(io, fix:, **options, &schema_for)
+    # @raise [ArgumentError] for a compression level the codec does not take, invalid encryption
+    #   settings, or +headers+ that are empty or name a column twice
+    def initialize(io, fix:, headers: nil, **options, &schema_for)
       # Fail before reading any rows if the codec's library is missing or the level is wrong
       codec = Compression.codec_id(options.fetch(:compression, :snappy))
       Compression.ensure_available!(codec)
       Compression.check_level!(codec, options[:compression_level])
       options = options.merge(encryption: EncryptionConfiguration.from(options[:encryption])) if options[:encryption]
+      @headers = headers && check_headers(headers)
       @io = io
       @fix = fix
       @options = options
@@ -42,12 +48,15 @@ module Herringbone
       @mismatch = nil
     end
 
-    # @param row [Object] row to write, in any form the Writer accepts
+    # @param row [Object] row to write, in any form the Writer accepts; with +headers+, an Array or
+    #   a Hash
     # @return [self]
+    # @raise [ArgumentError] when the row does not match the headers
     # @raise [SchemaMismatch] when the row does not fit the inferred schema
     # @raise [Error] after an earlier SchemaMismatch
     def <<(row)
       check_usable!
+      row = values_of(row) if @headers
       if @writer
         write(row)
       else
@@ -57,16 +66,6 @@ module Herringbone
       self
     end
 
-    # Sets the encryption before the first row
-    #
-    # @param config [EncryptionConfiguration] how to encrypt the file
-    # @return [void]
-    # @raise [Error] once rows were given
-    def encryption=(config)
-      raise Error, "Encryption must be set before the first row" if @writer || !@sample.empty?
-      @options = @options.merge(encryption: config)
-    end
-
     # @return [Integer] rows accepted so far, including the ones held back
     def rows_written = @writer ? @writer.rows_written : @sample.size
 
@@ -101,9 +100,10 @@ module Herringbone
     #
     # @return [void]
     # @raise [SchemaMismatch] when a held-back row does not fit
+    # @raise [ArgumentError] when a column declared in the block is not among the headers
     def start
       @sample_size = @sample.size
-      @writer = Writer.new(@io, @schema_for.call(@sample), **@options)
+      @writer = Writer.new(@io, infer_schema, **@options)
       sample, @sample = @sample, nil
       sample.each { |row| write(row) }
     end
@@ -119,6 +119,48 @@ module Herringbone
       raise @mismatch
     end
 
+    # @return [Schema] built by the block from the held-back rows, as Hashes when there are headers
+    # @raise [ArgumentError] when a column declared in the block is not among the headers
+    def infer_schema
+      return @schema_for.call(@sample) unless @headers
+      # Without rows, every column is nil and so becomes a string
+      rows = @sample.empty? ? [@headers.to_h { |name| [name, nil] }] : @sample.map { |values| @headers.zip(values).to_h }
+      schema = @schema_for.call(rows)
+      extra = schema.fields.map(&:name) - @headers
+      raise ArgumentError, "Declared columns not among the headers (#{@headers.join(", ")}): #{extra.join(", ")}" unless extra.empty?
+      schema
+    end
+
+    # @param headers [Array<String, Symbol>] column names given as +headers:+
+    # @return [Array<String>] the names as frozen Strings
+    # @raise [ArgumentError] when there are none or a name is given twice
+    def check_headers(headers)
+      names = Array(headers).map(&:to_s)
+      raise ArgumentError, "headers: needs at least one column name" if names.empty?
+      duplicates = names.tally.select { |_, n| n > 1 }.keys
+      raise ArgumentError, "headers: names a column more than once: #{duplicates.join(", ")}" unless duplicates.empty?
+      names.freeze
+    end
+
+    # @param row [Array, Hash] row given to #<<
+    # @return [Array] its values in header order
+    # @raise [ArgumentError] when it does not match the headers
+    def values_of(row)
+      case row
+      when Array
+        unless row.size == @headers.size
+          raise ArgumentError, "Expected #{@headers.size} values (#{@headers.join(", ")}), got #{row.size}"
+        end
+        row
+      when Hash
+        unknown = row.keys.map(&:to_s) - @headers
+        raise ArgumentError, "Not among the headers (#{@headers.join(", ")}): #{unknown.join(", ")}" unless unknown.empty?
+        @headers.map { |name| row.fetch(name) { row[name.to_sym] } }
+      else
+        raise ArgumentError, "With headers:, rows must be Arrays or Hashes, got #{row.class}"
+      end
+    end
+
     # @return [void]
     # @raise [Error] after a SchemaMismatch
     def check_usable!

data/lib/herringbone/inspector.rb

@@ -1155,6 +1155,8 @@ module Herringbone
     #
     # Fields carry :name, :type, :nullable and, when present, :children, :dictionary
     # ({ index_type:, ordered:, id: }), :extension (ARROW:extension:name) and :metadata.
+    #
+    # @api private
     module ArrowSchema
       # Raised for malformed or unsupported ARROW:schema values
       class Error < StandardError; end

data/lib/herringbone/key.rb

@@ -14,9 +14,8 @@ module Herringbone
   #   Herringbone.write(io, rows, encryption: key)
   #   Herringbone::Reader.new(io, decryption: key)            # or [new_key, old_key, ...]
   #
-  # Where a key may be given as a String instead (+encryption:+, +decryption:+,
-  # SimpleWriter#encrypt!), the String must be its hex: raw bytes are easy to mangle and to confuse
-  # with text, so they go through Key.new.
+  # Where a key may be given as a String instead (+encryption:+, +decryption:+), the String must be
+  # its hex: raw bytes are easy to mangle and to confuse with text, so they go through Key.new.
   #
   # Instances are frozen; #inspect and #to_s show the id but not the key.
   class Key

data/lib/herringbone/reader/column_chunk_reader.rb

@@ -301,11 +301,12 @@ module Herringbone
       # @param size [Integer] uncompressed size from the page header
       # @return [String] uncompressed bytes
       # @raise [UnsupportedError] for an unsupported codec, naming the column
+      # @raise [FormatError] for a body that cannot be decompressed, naming the column
       def decompress(body, size)
         Compression.decompress(@meta.codec, body, size)
       rescue MissingCodecError => e
         raise e.for_column(@column.dotted_path)
-      rescue UnsupportedError => e
+      rescue UnsupportedError, FormatError => e
         raise e, "#{e.message} (column #{@column.dotted_path})"
       end
 

data/lib/herringbone/reader.rb

@@ -67,7 +67,7 @@ module Herringbone
     #   to check the stored one)
     # @raise [ArgumentError] when +io+ cannot seek and read, or +keys+ / +time_zone+ /
     #   +decryption+ are invalid
-    # @raise [FormatError] when the footer is missing or cannot be decoded
+    # @raise [FormatError] when the footer is missing, cannot be decoded or holds a schema that makes no sense
     # @raise [DecryptionError] when the footer is encrypted and cannot be decrypted, or its
     #   signature or a column's metadata does not check out
     def initialize(io, keys: :string, time_zone: nil, decryption: nil)
@@ -83,7 +83,7 @@ module Herringbone
       @decryption = decryption.nil? ? nil : DecryptionConfiguration.from(decryption)
       @io = RestrictedReadableIO.wrap(io)
       @file_metadata = read_footer
-      @schema = Schema.from_elements(@file_metadata.schema)
+      @schema = damage_as_format_error("the schema") { Schema.from_elements(@file_metadata.schema) }
     end
 
     # Total number of rows, as stated in the footer (some writers store 0)
@@ -186,23 +186,27 @@ module Herringbone
       plan_rows(filter, from).each do |rg_index, ranges|
         rg = row_groups[rg_index]
         partial = ranges != [[0, rg.num_rows]]
-        cursors = fields.map do |f|
-          f.leaves.map do |col|
-            reader = chunk_reader(rg_index, col, converter: converters[col.index], lazy: true)
-            reader.locations = page_index(rg_index, col)[1]&.page_locations if partial
-            [col.index, ColumnCursor.new(reader)]
+        cursors = damage_as_format_error("row group #{rg_index}") do
+          fields.map do |f|
+            f.leaves.map do |col|
+              reader = chunk_reader(rg_index, col, converter: converters[col.index], lazy: true)
+              reader.locations = page_index(rg_index, col)[1]&.page_locations if partial
+              [col.index, ColumnCursor.new(reader)]
+            end
           end
         end
         ranges.each do |first, stop|
-          cursors.each { |cs| cs.each { |_, cursor| cursor.seek(first) } }
+          damage_as_format_error("row group #{rg_index}") { cursors.each { |cs| cs.each { |_, cursor| cursor.seek(first) } } }
           left = stop - first
           while left.positive?
             # Rows still missing from the current batch (pending_rows counts rows in both modes)
             k = pending ? size - pending_rows : size
             raise Error, "Internal error: batch needs #{k} rows" unless k.positive? # never loop without progress
             k = left if left < k
-            data = assemblers.each_with_index.map do |asm, j|
-              asm.read_rows(k, cursors[j].to_h { |idx, cursor| [idx, cursor.take(k)] })
+            data = damage_as_format_error("row group #{rg_index}") do
+              assemblers.each_with_index.map do |asm, j|
+                asm.read_rows(k, cursors[j].to_h { |idx, cursor| [idx, cursor.take(k)] })
+              end
             end
             left -= k
             kept = k
@@ -429,21 +433,25 @@ module Herringbone
           reader.locations = page_index(rg_index, col)[1]&.page_locations if partial
           reader
         end
-        ruby_cursors = ruby_fields.map { |f| f.leaves.map { |col| [col.index, ColumnCursor.new(open.call(col))] } }
-        fast_cursors = fast.map { |j| NumoCursor.new(open.call(out_fields[j].column), specs[j]) }
+        ruby_cursors, fast_cursors = damage_as_format_error("row group #{rg_index}") do
+          [ruby_fields.map { |f| f.leaves.map { |col| [col.index, ColumnCursor.new(open.call(col))] } },
+            fast.map { |j| NumoCursor.new(open.call(out_fields[j].column), specs[j]) }]
+        end
         ranges.each do |first, stop|
-          ruby_cursors.each { |cs| cs.each { |_, cursor| cursor.seek(first) } }
-          fast_cursors.each { |cursor| cursor.seek(first) }
+          damage_as_format_error("row group #{rg_index}") do
+            ruby_cursors.each { |cs| cs.each { |_, cursor| cursor.seek(first) } }
+            fast_cursors.each { |cursor| cursor.seek(first) }
+          end
           left = stop - first
           while left.positive?
             k = size - pending_rows
             raise Error, "Internal error: batch needs #{k} rows" unless k.positive? # never loop without progress
             k = step if k > step
             k = left if left < k
-            data = assemblers.each_with_index.map do |asm, j|
-              asm.read_rows(k, ruby_cursors[j].to_h { |idx, cursor| [idx, cursor.take(k)] })
+            data, numo = damage_as_format_error("row group #{rg_index}") do
+              [assemblers.each_with_index.map { |asm, j| asm.read_rows(k, ruby_cursors[j].to_h { |idx, cursor| [idx, cursor.take(k)] }) },
+                fast_cursors.map { |cursor| cursor.take(k) }]
             end
-            numo = fast_cursors.map { |cursor| cursor.take(k) }
             left -= k
             kept = k
             if filter
@@ -515,21 +523,45 @@ module Herringbone
       raise ArgumentError, "from: must not be negative" if from.negative?
       offset = 0
       plan = []
-      row_groups.each_with_index do |rg, i|
-        n = rg.num_rows
-        first = from - offset
-        offset += n
-        next if n.zero? || first >= n
-        ranges = [[first.positive? ? first : 0, n]]
-        if filter
-          next unless filter.row_group_may_match?(self, i)
-          ranges = Filter.intersect(ranges, filter.page_ranges(self, i))
+      damage_as_format_error("the list of row groups") do
+        row_groups.each_with_index do |rg, i|
+          n = rg.num_rows
+          first = from - offset
+          offset += n
+          next if n.zero? || first >= n
+          ranges = [[first.positive? ? first : 0, n]]
+          if filter
+            next unless filter.row_group_may_match?(self, i)
+            ranges = Filter.intersect(ranges, filter.page_ranges(self, i))
+          end
+          plan << [i, ranges] unless ranges.empty?
         end
-        plan << [i, ranges] unless ranges.empty?
       end
       plan
     end
 
+    # What code tripping over damaged metadata raises: a field the Thrift struct lacks is nil, a
+    # count or an offset is negative or far too large, an enum has a value nobody defined
+    DAMAGE_ERRORS = [NoMethodError, TypeError, ArgumentError, IndexError, RangeError, ZeroDivisionError].freeze
+    private_constant :DAMAGE_ERRORS
+
+    # Runs a block that decodes a part of the file, raising FormatError where damaged metadata
+    # makes it trip. Required fields are not checked when the metadata is decoded (that would cost
+    # on every page header), so a missing one shows up as a nil or a nonsensical number later on.
+    # The block must not call into the caller's code, whose errors are not ours to rename.
+    #
+    # @param what [String] the part of the file being read, for the message
+    # @yield decodes that part
+    # @yieldreturn [Object] returned as is
+    # @return [Object] the block's value
+    # @raise [FormatError] when the block raises one of DAMAGE_ERRORS, kept as the +cause+
+    def damage_as_format_error(what)
+      yield
+    rescue *DAMAGE_ERRORS => e
+      raise FormatError, "The file is damaged: #{what} cannot be read, because the metadata describing it has " \
+        "a field missing or holds a value that makes no sense (#{e.class}: #{e.message})"
+    end
+
     # Decodes one Thrift struct stored elsewhere in the file (ColumnIndex, OffsetIndex)
     #
     # @param klass [Class] Format struct class to decode with (responds to .decode)

data/lib/herringbone/redaction.rb

@@ -35,6 +35,8 @@ module Herringbone
     #   @return [Hash{String => Object}] column => value, for a replace without a block
     # @!attribute block
     #   @return [Proc, nil] computes the replacement values
+    #
+    # @api private
     Statement = Struct.new(:kind, :where, :targets, :constants, :block)
 
     # What #apply did, for the audit trail an erasure needs
@@ -52,6 +54,7 @@ module Herringbone
     # Rows selected with Redaction#where, waiting for a verb: #delete or #replace
     class Scope
       # @return [Hash{String, Symbol => Object}] the conditions given to Redaction#where
+      # @api private
       attr_reader :conditions
 
       # @param redaction [Redaction] the redaction the statement is added to
@@ -63,6 +66,7 @@ module Herringbone
       end
 
       # @return [Boolean] whether a verb was called on this scope
+      # @api private
       def used? = @used
 
       # Removes the matching rows
@@ -99,9 +103,11 @@ module Herringbone
     end
 
     # @return [Array<Statement>] the statements, in declared order
+    # @api private
     attr_reader :statements
 
     # @return [Array<String>] fields and struct members to remove from the schema
+    # @api private
     attr_reader :drops
 
     # Builds a redaction. The block receives the redaction and declares the statements on it,
@@ -275,6 +281,7 @@ module Herringbone
     # @param block [Proc, nil] computes the values
     # @return [Statement]
     # @raise [ArgumentError] when given both or neither of +constants+ and a block
+    # @api private
     def self.replace_statement(where, columns, constants, block)
       if block
         unless constants.empty?

data/lib/herringbone/schema.rb

@@ -163,12 +163,14 @@ module Herringbone
       #
       # @return [Proc, Method, nil] converts a physical value into its Ruby value, or nil when the physical
       #   value is used as-is
+      # @api private
       def converter = Types.reader_for(@node)
 
       # Types.writer_for of the node. Not memoized, like #converter.
       #
       # @return [Proc, Method] converts a Ruby value into the physical value to store, raising
       #   ArgumentError / TypeError / RangeError for values that do not fit
+      # @api private
       def encoder = Types.writer_for(@node)
     end
 
@@ -318,7 +320,7 @@ module Herringbone
       #   to one
       def row_hash(row)
         return row if row.is_a?(Hash)
-        raise ArgumentError, "Cannot infer a schema from Array rows (values in schema order); pass a schema" if row.is_a?(Array)
+        raise ArgumentError, "Cannot infer a schema from Array rows (values in schema order); pass headers: with the column names, or a schema" if row.is_a?(Array)
         return row.attributes if row.respond_to?(:attributes)
         return row.to_h if row.respond_to?(:to_h)
         raise ArgumentError, "Cannot infer a schema from a #{row.class}"
@@ -666,6 +668,7 @@ module Herringbone
     # a Builder of their own.
     class Builder
       # @return [Array<Node>] fields declared so far, in declaration order
+      # @api private
       attr_reader :nodes
 
       # Yields a new Builder to the block.
@@ -690,6 +693,7 @@ module Herringbone
       # @param usage [String] how the entry point is called with a block, for the error message
       # @return [void]
       # @raise [ArgumentError] when the block takes no parameter
+      # @api private
       def self.check_block!(block, usage)
         return if block.nil? || !block.parameters.empty?
         raise ArgumentError, "The block receives the schema builder as a parameter: #{usage}"

data/lib/herringbone/thrift.rb

@@ -7,8 +7,11 @@ module Herringbone
   #
   # @api private
   module Thrift
-    # Raised on malformed or truncated Thrift data, and on values the writer cannot encode
-    class Error < StandardError; end
+    # Raised on malformed or truncated Thrift data, and on values the writer cannot encode. Readers
+    # turn it into FormatError; from a writer it reaches the caller as it is.
+    #
+    # @api public
+    class Error < Herringbone::Error; end
 
     # Compact protocol wire type: end of a struct's fields
     T_STOP = 0

data/lib/herringbone/version.rb

@@ -2,5 +2,5 @@
 
 module Herringbone
   # Gem version, also written to the +created_by+ field of every file's footer
-  VERSION = "0.9.1"
+  VERSION = "1.0.0"
 end