herringbone 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b6e9b4e12f2b83688e6ea7be2752dd389cd1a338489834160efb0df5f796916e
-  data.tar.gz: c69303ac028259092e2f22c0f5d635f0e428ae1950c1b4b42d4a6379fd6a5a87
+  metadata.gz: '08770d40f03a2320d9aa60e6fe63fbfe40ea5ddeddc627bff3be2bf46f431795'
+  data.tar.gz: b18fef0b86fd290576d254f338da456ad80dc2600c68eb0c903e6085a024ebe2
 SHA512:
-  metadata.gz: 00a7d9246cb73211dcd5c46a5b0dab15b20b0fe1dc0026a2f02dcf3a9e13a11c91b8632bf1891fdd2f5a8deccf42f1c46918ee26ed9bf881208338f731cd12b6
-  data.tar.gz: 91edc03478f1d9aa24f4d40527b8d2ac8cbed53047e3870e2b17d52ba252bb29a354035f5f072105adf0122bcf93e24c424747552423f9f7ba345fedb98b4219
+  metadata.gz: 5c941e4b5f31f278ff819d25d464dab518c46e50d10435c360faf2da5994a9cff891754d00a73d8834aa08f2cd00be0387d9f6c8536459389797ba7daa1a82d0
+  data.tar.gz: 4532d47095f0832bf4cebb86ef370882dd7287625d80b9b94f6bc49fffb1ad3317075ecf8c45a1fef3fdb2fa19c3a641c0a6c2c4664bf24f5aa664e5b686e03e

data/README.md

@@ -12,12 +12,16 @@ A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/)
 
 ```ruby
 gem "herringbone"
+gem "snappy"    # optional: native Snappy, 2-3x faster reads and writes of typical files
 gem "zstd-ruby" # optional: ZSTD (faster writes and smaller files than the default Snappy)
 gem "brotli"    # optional: Brotli
 gem "xxhash"    # optional: faster bloom filters
+gem "numo-narray-alt" # optional: read(as: :numo)
 ```
 
-The only dependency is `bigdecimal`. Snappy, LZ4 and GZIP always work. `Herringbone.codecs` lists
+The only dependency is `bigdecimal`. Snappy, LZ4 and GZIP always work. Snappy, Parquet's most
+common codec, is pure Ruby unless the `snappy` gem is installed (it needs libsnappy or cmake to
+build); Herringbone then uses it automatically. `Herringbone.codecs` lists
 the codecs this process can use, e.g. `[:none, :snappy, :gzip, :lz4, :lz4_hadoop, :zstd]`. Using
 a missing one raises `Herringbone::MissingCodecError` naming the gem to add: a writer raises it
 before writing anything, a reader when it reaches the first such page (the schema and metadata
@@ -76,6 +80,34 @@ are exact. On a 1M-row file with 20k-row pages, looking up one `id` takes 0.1 s
 Filters help most on columns the data is sorted or clustered by. `reader.scan_plan(where: ...)`
 shows which row groups and row ranges a read would touch, without reading them.
 
+### Numo arrays
+
+`as: :numo` returns (or yields, with `each_batch`) a Hash of column name => Numo array, and takes
+the same `columns:`, `where:`, `from:` and `limit:`. Add `gem "numo-narray-alt"` (or
+`numo-narray`) to your Gemfile: Herringbone requires it on first use and raises
+`Herringbone::UnsupportedError` naming the gem when it is missing.
+
+```ruby
+cols = reader.read(as: :numo, columns: %w[id amount])  # { "id" => Numo::Int64, "amount" => Numo::DFloat }
+df = Rover::DataFrame.new(reader.read(as: :numo))       # no conversion needed
+```
+
+Flat numeric and boolean columns are decoded from the page bytes without a Ruby object per value:
+two numeric columns of 1M uncompressed rows read in about 25 ms, against 50 ms with `as: :columns`.
+
+| Parquet | Numo |
+|---|---|
+| INT32, INT64 (also TIME) | `Int32`, `Int64` |
+| INT(8/16) signed; INT(8/16/32/64) unsigned | `Int8`, `Int16`; `UInt8` … `UInt64` |
+| FLOAT, FLOAT16, DOUBLE | `SFloat`, `SFloat`, `DFloat` (nulls are NaN) |
+| integers with nulls | `DFloat` with NaN (exact up to 2**53) |
+| BOOLEAN | `Bit`; with nulls `RObject` of true/false/nil |
+| list of numbers, every row the same length and no nulls | 2-D `[rows, length]` (e.g. embeddings) |
+| strings, binary, decimals, dates, timestamps, UUIDs, structs, maps, other lists | `RObject` of the values `read` returns |
+
+Whether a column has nulls (or a list column is rectangular) is decided from the rows read, so
+with `each_batch` it can differ between batches.
+
 ## Writing
 
 ```ruby
@@ -111,9 +143,10 @@ first 1000 rows unless `schema:` is given; fields declared in a block replace in
 `Herringbone.write(io, rows, schema: Herringbone::Schema.infer(rows) { json :payload })`.
 
 The writer writes to any IO that responds to `#write` (a `File`, `StringIO`, `Tempfile`, socket or
-pipe), sequentially, and never seeks, rewinds or closes it. If the `Writer.open` block raises (or
-`#abort` is called), no footer is written and what was written so far is left for you to discard.
-To replace a file only once it is complete, write to a temporary file and rename it.
+pipe), sequentially, and never seeks, rewinds or closes it (it does switch it to binary mode). If
+the `Writer.open` block raises (or `#abort` is called), no footer is written and what was written
+so far is left for you to discard. To replace a file only once it is complete, write to a temporary
+file and rename it.
 
 Column types: `boolean int8 int16 int32 int64 uint8 uint16 uint32 uint64 float double float16
 string binary json bson enum uuid date int96 time timestamp decimal fixed`, plus `struct`, `list`
@@ -175,6 +208,25 @@ dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` ge
 a million rows take about 1.2 s longer to write with a filter on an INT64 column and 2.8 s longer
 with one on a ~22-byte string column, and about 0.5 s longer with `xxhash`.
 
+### Writing to S3
+
+Parquet keeps its metadata in a footer, so a file can be streamed into an S3 multipart upload
+without a local copy, using `upload_stream` from `aws-sdk-s3`:
+
+```ruby
+s3 = Aws::S3::TransferManager.new # aws-sdk-s3 1.197+; before that, Aws::S3::Object#upload_stream
+s3.upload_stream(bucket: "exports", key: "events.parquet", part_size: 16 * 1024 * 1024) do |io|
+  Herringbone::Writer.open(io, schema) do |w|
+    events.each { |event| w << event }
+  end
+end
+s3.upload_stream(bucket: "exports", key: "orders.parquet") { |io| Herringbone.write(io, Order.all) }
+```
+
+If the block raises, the SDK aborts the multipart upload and raises `Aws::S3::MultipartUploadError`,
+so no partial object is left. S3 allows at most 10,000 parts, which with the default 5MB parts caps
+the file at about 48GB; raise `part_size:` for bigger files.
+
 ## ActiveRecord
 
 `Herringbone.write` also takes a model or relation, which it reads with `find_each`, using a schema

data/lib/herringbone/active_record.rb

@@ -11,13 +11,18 @@ module Herringbone
     # ActiveRecord is not required: this only uses what a model class exposes
     # (+columns+, +primary_key+ and, when present, +defined_enums+).
     #
-    # only:         attribute names to include (Strings or Symbols)
-    # except:       attribute names to leave out
-    # parquet_enum: Rails enum attributes are string columns holding the labels (the writer
-    #               rejects values outside the enum; stored values such as 0/1 are written as
-    #               their labels). true adds the Parquet ENUM annotation, see Builder#enum.
+    # Primary key columns are never nullable. Rails enum attributes are string columns holding the
+    # labels (the writer rejects values outside the enum; stored values such as 0/1 are written as
+    # their labels).
+    #
+    # @param model [Class] ActiveRecord model class, or anything exposing +columns+ the same way
+    # @param only [Array<String, Symbol>, String, Symbol, nil] attribute names to include
+    # @param except [Array<String, Symbol>, String, Symbol, nil] attribute names to leave out
+    # @param parquet_enum [Boolean] true adds the Parquet ENUM annotation to enum columns, see Builder#enum
+    # @return [Schema] schema with one top-level field per selected column, in model column order
+    # @raise [ArgumentError] when no columns are left after +only+ / +except+
     def self.from_active_record(model, only: nil, except: nil, parquet_enum: false)
-      only = only && Array(only).map(&:to_s)
+      only &&= Array(only).map(&:to_s)
       except = Array(except).map(&:to_s)
       defined_enums = model.respond_to?(:defined_enums) ? model.defined_enums.to_h { |k, v| [k.to_s, v] } : {}
       primary_keys = Array(model.respond_to?(:primary_key) ? model.primary_key : nil).map(&:to_s)
@@ -54,9 +59,14 @@ module Herringbone
     module ActiveRecordMapping
       module_function
 
+      # Precision for decimal columns that declare none (Postgres +numeric+ without arguments);
+      # 38 digits is the most a 16-byte FIXED_LEN_BYTE_ARRAY decimal holds
       DEFAULT_DECIMAL_PRECISION = 38
+      # Scale used together with DEFAULT_DECIMAL_PRECISION when the column declares neither
       DEFAULT_DECIMAL_SCALE = 9
 
+      # Bit width of named SQL integer types (MySQL, Postgres and SQLite spellings), keyed by the
+      # leading word of the lowercased +sql_type+
       INTEGER_SQL_TYPES = {
         "tinyint" => 8, "int1" => 8,
         "smallint" => 16, "int2" => 16, "smallserial" => 16, "serial2" => 16,
@@ -64,6 +74,16 @@ module Herringbone
         "bigint" => 64, "int8" => 64, "bigserial" => 64, "serial8" => 64
       }.freeze
 
+      # Declares +column+ on +builder+: hstore as map<string, string>, array columns as a list of
+      # the element type, everything else as the scalar type picked by #scalar_type.
+      #
+      # @param builder [Builder] builder to add the field to
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata (+name+, +type+, and
+      #   when available +sql_type+, +array+, +limit+, +precision+, +scale+)
+      # @param nullable [Boolean] whether the field may be null
+      # @param primary [Boolean] whether the column is (part of) the primary key
+      # @return [Node] the added schema node
+      # @raise [ArgumentError] when the builder rejects the field (e.g. a duplicate name)
       def add_column(builder, column, nullable, primary)
         sql_type = column.respond_to?(:sql_type) ? column.sql_type.to_s.downcase : ""
         array = (column.respond_to?(:array) && column.array) || sql_type.end_with?("[]")
@@ -83,10 +103,18 @@ module Herringbone
         end
       end
 
+      # Picks the DSL type for a non-array, non-hstore column, following the table above.
+      #
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata, read for
+      #   +limit+, +precision+ and +scale+
+      # @param type [Symbol, nil] ActiveRecord's abstract type (+column.type+)
+      # @param sql_type [String] lowercased database type with any array suffix removed
+      # @param primary [Boolean] whether the column is (part of) the primary key
+      # @return [Array(Symbol, Hash{Symbol => Object})] DSL type and its options for Builder#column
       def scalar_type(column, type, sql_type, primary)
         case type
         when :integer, :bigint then [integer_type(column, sql_type, primary), {}]
-        when :float then [sql_type == "float4" ? :float : :double, {}]
+        when :float then [(sql_type == "float4") ? :float : :double, {}]
         when :decimal, :money
           precision = column.respond_to?(:precision) ? column.precision : nil
           scale = column.respond_to?(:scale) ? column.scale : nil
@@ -94,12 +122,12 @@ module Herringbone
             precision = DEFAULT_DECIMAL_PRECISION
             scale ||= DEFAULT_DECIMAL_SCALE
           end
-          [:decimal, { precision: precision, scale: scale || 0 }]
+          [:decimal, {precision: precision, scale: scale || 0}]
         when :boolean then [:boolean, {}]
         when :binary then [:binary, {}]
         when :date then [:date, {}]
-        when :datetime, :timestamp, :timestamptz then [:timestamp, { unit: :micros, utc: true }]
-        when :time then [:time, { unit: :micros }]
+        when :datetime, :timestamp, :timestamptz then [:timestamp, {unit: :micros, utc: true}]
+        when :time then [:time, {unit: :micros}]
         when :json, :jsonb then [:json, {}]
         when :uuid then [:uuid, {}]
         else [:string, {}]
@@ -108,6 +136,11 @@ module Herringbone
 
       # Named SQL types (smallint, bigint, ...) decide the width; a generic "integer"/"int"
       # uses the column limit in bytes, since e.g. SQLite reports `t.integer limit: 2` as "integer(2)".
+      #
+      # @param column [ActiveRecord::ConnectionAdapters::Column] column metadata, read for +limit+
+      # @param sql_type [String] lowercased database type with any array suffix removed
+      # @param primary [Boolean] true forces int64, since row ids can outgrow the declared width
+      # @return [Symbol] one of +:int8+ .. +:int64+ or +:uint8+ .. +:uint64+
       def integer_type(column, sql_type, primary)
         bits = INTEGER_SQL_TYPES[sql_type[/\A[a-z0-9]+/]]
         bits ||= case (column.respond_to?(:limit) ? column.limit : nil)
@@ -120,9 +153,8 @@ module Herringbone
         bits = 64 if primary
         unsigned = sql_type.include?("unsigned")
         return :"uint#{bits}" if unsigned
-        { 8 => :int8, 16 => :int16, 32 => :int32, 64 => :int64 }.fetch(bits)
+        {8 => :int8, 16 => :int16, 32 => :int32, 64 => :int64}.fetch(bits)
       end
     end
   end
 end
-

data/lib/herringbone/bloom_filter.rb

@@ -13,19 +13,27 @@ module Herringbone
   # String hashes the same bytes as the stored value. Nulls are never in a bloom filter.
   # The writer builds them (bloom_filters: option) and reads with where: consult them.
   class BloomFilter
+    # The spec's eight salt constants, one per word of a block
     SALT = [0x47b6137b, 0x44974d91, 0x8824ad5b, 0xa2b7289d, 0x705495c7, 0x2df1424b, 0x9efc4947, 0x5c6bfb31].freeze
     S0, S1, S2, S3, S4, S5, S6, S7 = SALT
     # Low 16 bits of the salts: (lo * salt) mod 2**32 is computed as
     # (lo & 0xFFFF) * salt + ((lo >> 16) * (salt & 0xFFFF) << 16), which stays a Fixnum
     L0, L1, L2, L3, L4, L5, L6, L7 = SALT.map { |s| s & 0xFFFF }
+    # Size of a block: eight 32-bit words
     BLOCK_BYTES = 32
+    # Smallest bitset: a single block
     MIN_BYTES = 32
+    # Largest bitset accepted when reading or building (128 MiB, the cap parquet-mr uses)
     MAX_BYTES = 128 * 1024 * 1024
     # Default cap for filters sized by the writer (as in parquet-mr)
     DEFAULT_MAX_BYTES = 1024 * 1024
+    # Default false positive probability for filters sized by the writer
     DEFAULT_FPP = 0.01
+    # 32-bit mask
     M32 = 0xFFFF_FFFF
+    # 64-bit mask
     M64 = 0xFFFF_FFFF_FFFF_FFFF
+    # Shorthand for the physical type constants
     T = Format::Type
     # Physical types a bloom filter can be built for (the spec does not define BOOLEAN hashing)
     TYPES = [T::INT32, T::INT64, T::INT96, T::FLOAT, T::DOUBLE, T::BYTE_ARRAY, T::FIXED_LEN_BYTE_ARRAY].freeze
@@ -33,10 +41,17 @@ module Herringbone
     # Bitset size in bytes for +ndv+ distinct values at false positive probability +fpp+, per the
     # spec's formula (m = -8 * ndv / ln(1 - fpp ** (1/8)) bits), rounded up to a power of two
     # and clamped to MIN_BYTES..max_bytes
+    #
+    # @param ndv [Integer] expected number of distinct values (values below 1 count as 1)
+    # @param fpp [Float] false positive probability, strictly between 0 and 1
+    # @param max_bytes [Integer] upper bound, clamped to MIN_BYTES..MAX_BYTES and rounded down to a
+    #   power of two
+    # @return [Integer] bitset size in bytes, a power of two
+    # @raise [ArgumentError] when +fpp+ is not between 0 and 1
     def self.optimal_num_bytes(ndv, fpp = DEFAULT_FPP, max_bytes: DEFAULT_MAX_BYTES)
       fpp = Float(fpp)
       raise ArgumentError, "fpp must be between 0 and 1, got #{fpp}" unless fpp > 0 && fpp < 1
-      max_bytes = [[Integer(max_bytes), MAX_BYTES].min, MIN_BYTES].max
+      max_bytes = Integer(max_bytes).clamp(MIN_BYTES, MAX_BYTES)
       max_bytes = 1 << (max_bytes.bit_length - 1) # a power of two
       ndv = [Integer(ndv), 1].max
       bits = -8.0 * ndv / Math.log(1 - fpp**(1.0 / 8))
@@ -47,6 +62,12 @@ module Herringbone
     end
 
     # XXH64 of the PLAIN encoding of a physical value of +type+ (as the column encoders produce it)
+    #
+    # @param value [Integer, Float, String, Array<Integer>] the physical value; for INT96 the
+    #   +[nanos_of_day, julian_day]+ pair the encoder produces
+    # @param type [Integer] Format::Type physical type
+    # @return [Integer] the unsigned 64-bit hash
+    # @raise [UnsupportedError] for BOOLEAN (or any type not in TYPES)
     def self.hash_physical(value, type)
       case type
       when T::INT32 then XXHash.xxh64_u32(value)
@@ -62,6 +83,12 @@ module Herringbone
     # Hashes of many physical values of +type+, converting them in bulk. With +distinct+, each
     # distinct physical value is hashed once (floats are compared by their bytes, so -0.0 and 0.0,
     # or NaNs with different payloads, stay apart as they hash differently).
+    #
+    # @param values [Array] physical values, as for .hash_physical
+    # @param type [Integer] Format::Type physical type
+    # @param distinct [Boolean] hash each distinct value once (the result is then shorter)
+    # @return [Array<Integer>] the unsigned 64-bit hashes
+    # @raise [UnsupportedError] for BOOLEAN (or any type not in TYPES)
     def self.hash_physical_all(values, type, distinct: false)
       case type
       when T::INT32 then XXHash.xxh64_u32_all(distinct ? values.uniq : values)
@@ -80,6 +107,13 @@ module Herringbone
 
     # Reads a filter (header and bitset) from +buf+ at +pos+. Returns nil for algorithms, hashes
     # or compressions this implementation does not know.
+    #
+    # @param buf [String] bytes holding the BloomFilterHeader followed by the bitset
+    # @param pos [Integer] byte offset of the header in +buf+
+    # @param column [Schema::Column, nil] column the filter belongs to, for converting values
+    # @return [BloomFilter, nil] the filter, or nil when it is of an unsupported kind
+    # @raise [FormatError] when the bitset is shorter than the header announces
+    # @raise [Thrift::Error] when the header cannot be decoded
     def self.decode(buf, pos = 0, column: nil)
       reader = Thrift::Reader.new(buf, pos)
       header = reader.read_struct(Format::BloomFilterHeader)
@@ -89,48 +123,78 @@ module Herringbone
       new(bitset: bitset, column: column)
     end
 
+    # Whether a header describes a filter this implementation can read: split block algorithm,
+    # XXH64, uncompressed, and a size that is a multiple of 32 bytes up to MAX_BYTES
+    #
+    # @param header [Format::BloomFilterHeader] decoded header
+    # @return [Boolean] true when supported
     def self.supported_header?(header)
       n = header.num_bytes
       n.is_a?(Integer) && n >= BLOCK_BYTES && (n % BLOCK_BYTES).zero? && n <= MAX_BYTES &&
-        header.algorithm&.block && header.hash_function&.xxhash &&
-        (header.compression.nil? || header.compression.uncompressed)
+        !(header.algorithm && header.algorithm.block).nil? &&
+        !(header.hash_function && header.hash_function.xxhash).nil? &&
+        (header.compression.nil? || !header.compression.uncompressed.nil?)
     end
 
+    # @return [Schema::Column, nil] column whose encoder converts values, nil for raw Strings
     attr_reader :column
 
     # A filter of +num_bytes+ (a multiple of 32, normally a power of two), or one using an existing
     # +bitset+ String. With a +column+ (a Schema::Column), values are Ruby values converted with
     # the column's encoder; without one, values must be Strings and their bytes are hashed.
+    #
+    # @param num_bytes [Integer, nil] bitset size for an empty filter (MIN_BYTES when nil); ignored
+    #   with +bitset+
+    # @param bitset [String, nil] existing bitset (little-endian 32-bit words)
+    # @param column [Schema::Column, nil] column the filter is for
+    # @raise [ArgumentError] when the size (or +bitset+'s size) is not a multiple of 32 bytes or
+    #   exceeds MAX_BYTES
+    # @raise [UnsupportedError] when the column's physical type cannot have a bloom filter
     def initialize(num_bytes = nil, bitset: nil, column: nil)
-      if bitset
-        @words = bitset.unpack("V*")
-      else
-        num_bytes = Integer(num_bytes || MIN_BYTES)
-        unless num_bytes >= BLOCK_BYTES && (num_bytes % BLOCK_BYTES).zero? && num_bytes <= MAX_BYTES
-          raise ArgumentError, "Bloom filter size must be a multiple of 32 bytes up to 128MB, got #{num_bytes}"
-        end
-        @words = Array.new(num_bytes / 4, 0)
+      num_bytes = bitset ? bitset.bytesize : Integer(num_bytes || MIN_BYTES)
+      unless num_bytes >= BLOCK_BYTES && (num_bytes % BLOCK_BYTES).zero? && num_bytes <= MAX_BYTES
+        raise ArgumentError, "Bloom filter size must be a multiple of 32 bytes up to 128MB, got #{num_bytes}"
       end
+      @words = bitset ? bitset.unpack("V*") : Array.new(num_bytes / 4, 0)
       @num_blocks = @words.size / 8
-      raise ArgumentError, "Bloom filter bitset must be a multiple of 32 bytes" if @num_blocks.zero? || @words.size % 8 != 0
       @column = column
       if column && !TYPES.include?(column.type)
         raise UnsupportedError, "Bloom filters are not supported for #{T::NAMES[column.type]} column #{column.dotted_path}"
       end
     end
 
+    # @return [Integer] bitset size in bytes
     def num_bytes = @words.size * 4
 
+    # Adds a value to the filter
+    #
+    # @param value [Object] Ruby value, converted with the column's encoder (a String without a
+    #   column)
+    # @return [BloomFilter] self
+    # @raise [ArgumentError] for nil or a value the column cannot encode
     def insert(value)
       insert_hash(hash_of(value))
       self
     end
 
+    # Whether the value may be in the filter. Never false for an inserted value; true for a
+    # value that was not inserted with about the false positive probability the filter was sized for.
+    #
+    # @param value [Object] Ruby value, converted with the column's encoder (a String without a
+    #   column)
+    # @return [Boolean] false when the value is definitely absent
+    # @raise [ArgumentError] for nil or a value the column cannot encode
     def might_contain?(value)
       might_contain_hash?(hash_of(value))
     end
 
     # XXH64 hash the filter uses for a Ruby +value+
+    #
+    # @param value [Object] Ruby value, converted with the column's encoder (a String without a
+    #   column)
+    # @return [Integer] the unsigned 64-bit hash
+    # @raise [ArgumentError] for nil, a value the column cannot encode, or (without a column) a
+    #   non-String
     def hash_of(value)
       raise ArgumentError, "Nulls are not recorded in bloom filters" if value.nil?
       if @column
@@ -146,6 +210,11 @@ module Herringbone
       end
     end
 
+    # Sets the bits for a hash: the upper 32 bits pick the block, the lower 32 bits (multiplied
+    # by each salt) one bit in each of its eight words
+    #
+    # @param h [Integer] unsigned 64-bit XXH64 hash
+    # @return [BloomFilter] self
     def insert_hash(h)
       i = (((h >> 32) * @num_blocks) >> 32) << 3
       x0 = h & 0xFFFF
@@ -162,10 +231,14 @@ module Herringbone
       self
     end
 
+    # Single-bit masks, +BITS[i] == 1 << i+
     BITS = Array.new(32) { |i| 1 << i }.freeze
 
     # Inserts many hashes at once: faster than insert_hash, as the hashes (mostly Bignums) are
     # split into 32-bit halves in bulk and the rest is Fixnum arithmetic
+    #
+    # @param hashes [Array<Integer>] unsigned 64-bit XXH64 hashes
+    # @return [BloomFilter] self
     def insert_hashes(hashes)
       halves = hashes.pack("Q<*").unpack("V*")
       w = @words
@@ -191,6 +264,10 @@ module Herringbone
       self
     end
 
+    # Whether all the bits for a hash are set (see #insert_hash)
+    #
+    # @param h [Integer] unsigned 64-bit XXH64 hash
+    # @return [Boolean] false when the hashed value is definitely absent
     def might_contain_hash?(h)
       i = (((h >> 32) * @num_blocks) >> 32) << 3
       x0 = h & 0xFFFF
@@ -207,19 +284,27 @@ module Herringbone
     end
 
     # The raw bitset (little-endian 32-bit words)
+    #
+    # @return [String] binary String of #num_bytes bytes
     def bitset = @words.pack("V*")
 
     # Header and bitset, as stored in a Parquet file
+    #
+    # @return [String] Thrift-encoded BloomFilterHeader followed by the bitset (binary)
     def encode
       header.encode << bitset
     end
 
+    # @return [String] size and, when set, the column's dotted path
     def inspect
       "#<#{self.class.name} #{num_bytes} bytes#{" for #{@column.dotted_path}" if @column}>"
     end
 
     private
 
+    # Header for this filter: split block algorithm, XXH64 hash, uncompressed
+    #
+    # @return [Format::BloomFilterHeader] the header
     def header
       Format::BloomFilterHeader.new(
         num_bytes: num_bytes,
@@ -234,6 +319,13 @@ module Herringbone
     # Internal (used by reads with where:): the bloom filter of a column chunk. +column+ is a
     # dotted path ("a.b"), an Array path or a Schema::Column. Returns a BloomFilter, or nil when
     # the chunk has none (or one of an unknown kind).
+    #
+    # @param row_group_index [Integer] index of the row group
+    # @param column [String, Array<String, Symbol>, Symbol, Schema::Column] the leaf column
+    # @return [BloomFilter, nil] the filter, or nil when there is none or it is unsupported
+    # @raise [IndexError] when there is no such row group
+    # @raise [ArgumentError] when there is no such column
+    # @raise [FormatError] when the filter is truncated or its header cannot be decoded
     def bloom_filter(row_group_index, column)
       col = bloom_filter_column(column)
       rg = row_groups.fetch(row_group_index) { raise IndexError, "No row group #{row_group_index}" }
@@ -256,10 +348,18 @@ module Herringbone
       bitset = @io.read(header.num_bytes)
       raise FormatError, "Truncated bloom filter" if bitset.nil? || bitset.bytesize != header.num_bytes
       BloomFilter.new(bitset: bitset, column: col)
+    rescue Thrift::Error => e
+      raise FormatError, "Corrupt bloom filter header for #{col.dotted_path}: #{e.message}"
     end
 
     private
 
+    # Resolves the +column+ argument of #bloom_filter to a leaf column
+    #
+    # @param column [String, Array<String, Symbol>, Symbol, Schema::Column] dotted path, path
+    #   Array or column
+    # @return [Schema::Column] the column
+    # @raise [ArgumentError] when the schema has no such column
     def bloom_filter_column(column)
       return column if column.is_a?(Schema::Column)
       col = schema.column(column.is_a?(Array) ? column.map(&:to_s) : column.to_s)

data/lib/herringbone/byte_values.rb

@@ -17,8 +17,11 @@ module Herringbone
     # After this many values, give up on the dictionary if more than half of them are distinct
     CARDINALITY_CHECK_AT = 4096
 
+    # Whether String#append_as_bytes (Ruby 3.4+) is available for appending without re-encoding
     APPEND_AS_BYTES = "".respond_to?(:append_as_bytes)
 
+    # @param width [Integer, nil] byte length of each value for FIXED_LEN_BYTE_ARRAY, nil for BYTE_ARRAY
+    # @param dictionary [Boolean] start in dictionary mode; false appends raw bytes from the start
     def initialize(width: nil, dictionary: true)
       @width = width
       @bytes = String.new(encoding: Encoding::BINARY)
@@ -31,14 +34,21 @@ module Herringbone
       end
     end
 
+    # @return [Integer] number of values held
     def size
       @indices ? @indices.size : @count
     end
 
+    # @return [Boolean] true when no values are held
     def empty? = size.zero?
 
+    # @return [Boolean] true while still dictionary-encoding (not yet switched to raw bytes)
     def dictionary? = !@indices.nil?
 
+    # Appends a value, switching to raw bytes when the dictionary limits are exceeded.
+    # @param value [String] bytes of the value; for FIXED_LEN_BYTE_ARRAY it must be +width+ bytes
+    #   long (not checked here)
+    # @return [ByteValues] self
     def <<(value)
       if @indices
         index = @dictionary[value]
@@ -61,16 +71,20 @@ module Herringbone
     end
 
     # Removes the last value
+    # @return [void]
     def pop
       truncate(size - 1) unless size.zero?
     end
 
     # Supports the `slice!(n..)` form used to roll back a failed row
+    # @param range [Range] endless range; values from +range.begin+ on are dropped
+    # @return [void]
     def slice!(range)
       truncate(range.begin)
     end
 
     # Approximate memory held, used to size row groups
+    # @return [Integer] estimated bytes
     def memory_bytes
       if @indices
         # Each distinct value is a String object plus a Hash entry
@@ -82,6 +96,11 @@ module Herringbone
 
     # Returns [:dictionary, values, indices] when the column is worth dictionary-encoding,
     # otherwise [:plain, values]
+    #
+    # Dictionary encoding is kept unless there are more than 16 values and more than about half
+    # of them are distinct.
+    # @return [Array(Symbol, Array<String>, Array<Integer>), Array(Symbol, Array<String>)]
+    #   distinct values and an index per value, or all values in order
     def materialize
       if @indices
         keys = @dictionary.keys
@@ -93,6 +112,9 @@ module Herringbone
 
     private
 
+    # Appends +value+ in raw-bytes mode, recording its length for BYTE_ARRAY.
+    # @param value [String] bytes of the value, in any encoding
+    # @return [ByteValues] self
     def append_bytes(value)
       if value.encoding == Encoding::BINARY || value.ascii_only?
         @bytes << value
@@ -106,6 +128,8 @@ module Herringbone
       self
     end
 
+    # Leaves dictionary mode, re-appending every value held so far as raw bytes.
+    # @return [void]
     def switch_to_bytes
       keys = @dictionary.keys
       indices = @indices
@@ -113,6 +137,9 @@ module Herringbone
       indices.each { |i| append_bytes(keys[i]) }
     end
 
+    # Drops all values after the first +n+.
+    # @param n [Integer] number of values to keep
+    # @return [void]
     def truncate(n)
       if @indices
         @indices.slice!(n..)
@@ -128,6 +155,8 @@ module Herringbone
       end
     end
 
+    # Splits the raw bytes back into one binary String per value.
+    # @return [Array<String>]
     def strings
       if @lengths
         pos = 0