herringbone 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '08770d40f03a2320d9aa60e6fe63fbfe40ea5ddeddc627bff3be2bf46f431795'
-  data.tar.gz: b18fef0b86fd290576d254f338da456ad80dc2600c68eb0c903e6085a024ebe2
+  metadata.gz: 826898400d423a36ccd89740d408da3d367726cf84c6cf33c4067a3c54b8861d
+  data.tar.gz: 0667170a3ede8376299aff04b4081259533c8877f9823ad8fcc9aee916a86650
 SHA512:
-  metadata.gz: 5c941e4b5f31f278ff819d25d464dab518c46e50d10435c360faf2da5994a9cff891754d00a73d8834aa08f2cd00be0387d9f6c8536459389797ba7daa1a82d0
-  data.tar.gz: 4532d47095f0832bf4cebb86ef370882dd7287625d80b9b94f6bc49fffb1ad3317075ecf8c45a1fef3fdb2fa19c3a641c0a6c2c4664bf24f5aa664e5b686e03e
+  metadata.gz: 21984e5695bf75e086fa091a4c69456edd4fc7ef21ea2d73c89e0ad3de1316238d2f66cf9d8c86183c5e962a1ce965aaeac8036dc1a5c1a43aa6a3b85c983405
+  data.tar.gz: 9377178bd66028014087cc55ed255c4abc2ed1c0217cdfbd86d5f9fe647a4ac0d741a973415f393c31fc7af83d653ce14df33d7a7629dee69aaed554e1fa5719

data/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+## Unreleased
+
+## 0.4.0
+
+- herringbone is now Ractor-enabled! Reading and writing work from any Ractor on Ruby 3.1+, and schemas can be passed to Ractors with
+- **Breaking:** the optional gems (`zstd-ruby`, `brotli`, `snappy`, `xxhash`, `numo-narray-alt`)
+  are used only when loaded: require them (`Bundler.require` does) instead of relying on
+  Herringbone to require them on first use.
+- Files with LZO-compressed pages (written by parquet-mr with hadoop-lzo) can be read. Writing
+  LZO is not supported.
+- `Herringbone::SimpleWriter`: CSV-style writing with `headers!` and Array rows, types inferred.
+- `Herringbone.write` without `schema:` reads the rows once, holding back only the first 1000 for
+  inference, so cursors and one-shot Enumerators no longer lose rows.
+- `Herringbone.write(io, rows) { json :payload }` declares fields that replace inferred ones.
+- `Herringbone::SchemaMismatch` (an `EncodeError`) explains a row that doesn't fit an inferred
+  schema and stops the write; `EncodeError` has `row`, `column` and `value`.
+- `herringbone inspect --format=html` opens the page in the browser when run in a terminal.
+  `Ractor.make_shareable(schema)`.
+- **Breaking:** `herringbone inspect` takes `--format=text|json|html` instead of `--json`/`--html`.
+
+## 0.3.0
+
+- `read(as: :numo)` and `each_batch(as: :numo)` return Numo arrays (optional `numo-narray-alt`).
+- The `snappy` gem is used for Snappy when installed: 27x faster compression, 12x faster decompression.
+- Writes are forward-only, so files can be streamed into S3 multipart uploads.
+- YARD documentation for the whole library.
+
+## 0.2.0
+
+- Streaming reads: pages are decoded incrementally and rows assembled in batches.
+- `each_batch(as: :columns)` and `read(as: :columns)` for column-order reads.
+- `where:`, `from:` and `limit:` skip row groups and pages using statistics, bloom filters and the
+  page index.
+- Page indexes (ColumnIndex and OffsetIndex) are written for every column chunk.
+- Bloom filters are written and read, using the `xxhash` gem when it is installed.
+- `Herringbone::Inspector`, an HTML visualizer and `herringbone inspect`: page headers, page
+  indexes, CRC verification and the decoded ARROW:schema, without decompressing any data.
+- `Herringbone.write` takes ActiveRecord models and relations.
+- `zstd-ruby` and `brotli` are optional; a missing codec raises `MissingCodecError`.
+- **Breaking:** the public API was simplified: `Reader.new(io)` and `Writer.open(io, schema)`
+  only take IOs, Hash schemas and `Herringbone.export`/`open`/`read` are gone, and `DecodeError`
+  is now `FormatError`.
+
+## 0.1.0
+
+- Pure-Ruby Parquet reader and writer, with no Thrift gem or native extensions.
+- Full nesting support (structs, lists, maps) via Dremel shredding and assembly.
+- All standard encodings, v1 and v2 data pages, Snappy, LZ4 and GZIP in Ruby, ZSTD and Brotli via gems.
+- Schemas from a block DSL, inferred from rows, or built from ActiveRecord models.
+- Writer coercions for Rails values, byte-based row groups and compact buffers.

data/README.md

@@ -3,34 +3,71 @@
 A pure-Ruby reader and writer for [Apache Parquet](https://parquet.apache.org/) files.
 
 - No Thrift gem and no native extensions of its own: the Thrift compact protocol, all encodings
-  and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems)
+  and the Snappy and LZ4 codecs are implemented in Ruby (ZSTD and Brotli use optional gems).
+  LZO, from Hadoop-era files, can be read but not written
 - Full nesting support (structs, lists, maps, any depth) via Dremel record shredding/assembly
 - Reads files from parquet-mr, Arrow, Spark, Impala, DuckDB, Rust writers etc.
+- Optimized reads with batches and pages 
 - Ruby 3.0+
 
+## Diving in: dumping records in Rails
+
+Stream a relation straight into S3, no temp file needed:
+
+```ruby
+s3 = Aws::S3::TransferManager.new
+s3.upload_stream(bucket: "exports", key: "payments.parquet") do |io|
+  # Schema will be auto-inferred, find_each will be used automatically
+  Herringbone.write(io, Payment.where(status: "settled", created_at: 1.month.ago..))
+end
+```
+
 ## Installation
 
 ```ruby
 gem "herringbone"
-gem "snappy"    # optional: native Snappy, 2-3x faster reads and writes of typical files
-gem "zstd-ruby" # optional: ZSTD (faster writes and smaller files than the default Snappy)
-gem "brotli"    # optional: Brotli
-gem "xxhash"    # optional: faster bloom filters
-gem "numo-narray-alt" # optional: read(as: :numo)
 ```
 
-The only dependency is `bigdecimal`. Snappy, LZ4 and GZIP always work. Snappy, Parquet's most
-common codec, is pure Ruby unless the `snappy` gem is installed (it needs libsnappy or cmake to
-build); Herringbone then uses it automatically. `Herringbone.codecs` lists
-the codecs this process can use, e.g. `[:none, :snappy, :gzip, :lz4, :lz4_hadoop, :zstd]`. Using
-a missing one raises `Herringbone::MissingCodecError` naming the gem to add: a writer raises it
-before writing anything, a reader when it reaches the first such page (the schema and metadata
-are still readable). LZO is not supported.
+Add these libraries to speed up and enable certain functionality:
+
+```ruby
+gem "snappy"          # native Snappy, 2-3x faster reads and writes of typical files
+gem "zstd-ruby"       # ZSTD
+gem "brotli"          # Brotli
+gem "xxhash"          # faster bloom filters
+gem "numo-narray-alt" # read(as: :numo)
+```
+
+Herringbone never requires them itself: it uses whichever are loaded. Bundler.require (as in Rails)
+loads them, otherwise require them yourself:
+
+```ruby
+require "herringbone"
+require "zstd-ruby"
+require "numo/narray"
+```
+
+On Ruby 3.1 and later, Herringbone works from any Ractor. Pass a schema to a Ractor with
+`Ractor.make_shareable(schema)`. Rows made shareable the same way reach a writing Ractor by
+reference, without being copied:
+
+```ruby
+writer = Ractor.new("out.parquet", Ractor.make_shareable(schema)) do |path, schema|
+  File.open(path, "wb") do |f|
+    Herringbone::Writer.open(f, schema) do |w|
+      while (row = Ractor.receive) != :done
+        w << row
+      end
+    end
+  end
+end
+rows.each { |row| writer.send(Ractor.make_shareable(row)) }
+writer.send(:done)
+```
 
 ## Reading
 
-Herringbone never opens files by path: readers take a random-access IO (a `File` opened with
-`"rb"`, `StringIO`, `Tempfile`...), which stays open and belongs to the caller.
+Herringbone can read from any IO-ish object with random access (the IO should be seekable).
 
 ```ruby
 require "herringbone"
@@ -84,8 +121,8 @@ shows which row groups and row ranges a read would touch, without reading them.
 
 `as: :numo` returns (or yields, with `each_batch`) a Hash of column name => Numo array, and takes
 the same `columns:`, `where:`, `from:` and `limit:`. Add `gem "numo-narray-alt"` (or
-`numo-narray`) to your Gemfile: Herringbone requires it on first use and raises
-`Herringbone::UnsupportedError` naming the gem when it is missing.
+`numo-narray`) to your Gemfile and `require "numo/narray"`: without it, `as: :numo` raises
+`Herringbone::UnsupportedError` naming the gem.
 
 ```ruby
 cols = reader.read(as: :numo, columns: %w[id amount])  # { "id" => Numo::Int64, "amount" => Numo::DFloat }
@@ -140,7 +177,10 @@ end
 
 `Herringbone.write(io, rows)` writes an Enumerable of rows in one go, inferring the schema from the
 first 1000 rows unless `schema:` is given; fields declared in a block replace inferred ones:
-`Herringbone.write(io, rows, schema: Herringbone::Schema.infer(rows) { json :payload })`.
+`Herringbone.write(io, rows) { json :payload }`. The rows are iterated once, holding back only
+those first 1000, so lazy Enumerators and cursors that can't be rewound work. A later row that
+doesn't fit the inferred types raises `Herringbone::SchemaMismatch`, which explains what was
+inferred and how to declare the column, and leaves the file unfinished.
 
 The writer writes to any IO that responds to `#write` (a `File`, `StringIO`, `Tempfile`, socket or
 pipe), sequentially, and never seeks, rewinds or closes it (it does switch it to binary mode). If
@@ -192,6 +232,24 @@ Writer options:
 | `metadata` | `{}` | footer key/value metadata, read back with `reader.metadata` |
 | `bloom_filters` | none | `true`, an Array of column paths, or `{ "path" => { ndv:, fpp:, max_bytes: } }` |
 
+### Coming from CSV
+
+`SimpleWriter` writes like the CSV gem: name the columns, then append Arrays. The types are
+inferred from the first 1000 rows, as above.
+
+```ruby
+File.open("people.parquet", "wb") do |file|
+  Herringbone::SimpleWriter.open(file) do |sw|
+    sw.headers!(:id, :name, :age)
+    sw << [123, "John", 12]
+    sw << { id: 124, name: "Jane" } # Hashes work too
+  end
+end
+```
+
+Unlike CSV, headers are required. A column that holds more than one type can be declared up front:
+`Herringbone::SimpleWriter.new(io) { string :code }` (then call `close` when done).
+
 ### Statistics, page indexes and bloom filters
 
 Every column chunk gets min/max/null-count statistics and the Parquet page index (per-page min/max
@@ -204,7 +262,7 @@ emails or UUIDs; a bloom filter can. They are off by default. `bloom_filters: ["
 (or `true` for every non-boolean column) writes Parquet's split block bloom filters, sized from
 each row group's distinct values at a false positive probability of 1% (`fpp:`, up to `max_bytes:`,
 default 1MB) unless `ndv:` gives the number of distinct values. Nested leaves are named by their
-dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is installed:
+dotted path (`"tags.list.element"`). Hashing is pure Ruby unless the `xxhash` gem is loaded:
 a million rows take about 1.2 s longer to write with a filter on an INT64 column and 2.8 s longer
 with one on a ~22-byte string column, and about 0.5 s longer with `xxhash`.
 
@@ -281,7 +339,7 @@ follows `Model.columns`.
 
 > The inspector and its HTML view are modelled on
 > **[Parquet X-ray](https://huggingface.co/spaces/cfahlgren1/parquet-xray) by cfahlgren1** —
-> the design and the idea are theirs. Go check it out.
+> the design and the idea are theirs. Go check it out. It is amazing!
 
 `Herringbone::Inspector` examines a file using only its footer, page headers, page indexes and
 bloom filter headers. Nothing is decompressed, so it is fast on big files and works for ZSTD and
@@ -304,14 +362,20 @@ CRCs; the results then appear in `summary`, `report`, `to_h` and `to_html`.
 ## Command line
 
 ```
-bin/herringbone cat FILE [N]                    # rows as JSON lines
-bin/herringbone inspect FILE [--pages]          # text report (--pages lists every page header)
-bin/herringbone inspect FILE --json             # everything as JSON
-bin/herringbone inspect FILE --html > out.html  # the HTML page
+bin/herringbone cat FILE [N]                           # rows as JSON lines
+bin/herringbone inspect FILE [--pages]                 # text report (--pages lists every page header)
+bin/herringbone inspect FILE --format=json             # everything as JSON
+bin/herringbone inspect FILE --format=html             # the HTML page, opened in your browser
+bin/herringbone inspect FILE --format=html > out.html  # the HTML page, saved
 ```
 
 Add `--verify-checksums` to any `inspect` form to check page CRCs.
 
+`--format` is `text` (the default), `json` or `html`; `--pages` only goes with `text`.
+In a terminal, `--format=html` writes the page to a temp file, prints its path and opens it with `open` on
+macOS, `start` on Windows or `xdg-open` elsewhere. When stdout is redirected or piped it prints the
+page instead.
+
 ## Supported format features
 
 - Encodings (read and write): PLAIN, PLAIN_DICTIONARY/RLE_DICTIONARY, RLE, DELTA_BINARY_PACKED,

data/bin/herringbone

@@ -3,14 +3,21 @@
 
 # Usage:
 #   herringbone cat FILE [N]              print rows as JSON lines (optionally only the first N)
-#   herringbone inspect FILE [--pages]    print the schema, row groups, column chunks (and every page header)
-#   herringbone inspect FILE --json       print all of it as JSON
-#   herringbone inspect FILE --html       print a self-contained HTML page showing the file's layout
+#   herringbone inspect FILE [--pages]          print the schema, row groups, column chunks (and every
+#                                               page header); same as --format=text
+#   herringbone inspect FILE --format=json      print all of it as JSON
+#   herringbone inspect FILE --format=html      write a self-contained HTML page showing the file's layout
+#                                               and open it in the browser (printed when stdout is redirected)
 #   add --verify-checksums to any inspect form to read page bodies and check their CRCs
 # `inspect` reads only the footer, page headers and page indexes: no values are decompressed.
 $LOAD_PATH.unshift File.expand_path("../lib", __dir__)
 require "herringbone"
 require "json"
+%w[zstd-ruby brotli snappy xxhash].each do |lib|
+  require lib
+rescue LoadError
+end
+require "tmpdir"
 
 def jsonable(v)
   case v
@@ -25,8 +32,20 @@ def jsonable(v)
   end
 end
 
+# Opens a file with the platform's default application
+# @return [Boolean] whether the opener could be started
+def open_with_default_app(path)
+  command = case RbConfig::CONFIG["host_os"]
+  when /darwin/ then ["open", path]
+  when /mswin|mingw|cygwin/ then ["cmd", "/c", "start", "", path]
+  else ["xdg-open", path]
+  end
+  !!system(*command, out: File::NULL, err: File::NULL, exception: false)
+end
+
 USAGE = "usage: herringbone cat FILE [N]\n" \
-  "       herringbone inspect FILE [--pages | --json | --html] [--verify-checksums]"
+  "       herringbone inspect FILE [--format=text|json|html] [--pages] [--verify-checksums]\n" \
+  "       (--pages only with --format=text)"
 
 command, *rest = ARGV
 flags, args = rest.partition { |a| a.start_with?("--") }
@@ -34,8 +53,11 @@ path, limit = args
 valid = case command
 when "cat" then flags.empty? && path && (limit.nil? || limit.match?(/\A\d+\z/))
 when "inspect"
-  formats = flags & %w[--pages --json --html]
-  path && limit.nil? && (flags - %w[--pages --json --html --verify-checksums]).empty? && formats.size <= 1
+  format_flags, other_flags = flags.partition { |f| f.start_with?("--format=") }
+  format = format_flags.empty? ? "text" : format_flags.first.delete_prefix("--format=")
+  path && limit.nil? && format_flags.size <= 1 && %w[text json html].include?(format) &&
+    (other_flags - %w[--pages --verify-checksums]).empty? &&
+    (format == "text" || !other_flags.include?("--pages"))
 end
 unless valid
   warn USAGE
@@ -51,11 +73,19 @@ begin
     else
       inspector = Herringbone::Inspector.new(io)
       inspector.verify_checksums if flags.include?("--verify-checksums")
-      if flags.include?("--json")
+      case format
+      when "json"
         puts JSON.pretty_generate(inspector.to_h)
-      elsif flags.include?("--html")
-        $stdout.write(inspector.to_html)
-      else
+      when "html"
+        if $stdout.tty?
+          html_path = File.join(Dir.tmpdir, "#{File.basename(path, ".*")}-#{Process.pid}.html")
+          File.binwrite(html_path, inspector.to_html)
+          puts html_path
+          warn "herringbone: could not open the page, open #{html_path} in a browser" unless open_with_default_app(html_path)
+        else
+          $stdout.write(inspector.to_html)
+        end
+      when "text"
         puts inspector.report(pages: flags.include?("--pages"))
       end
     end

data/lib/herringbone/codecs/lzo.rb

@@ -0,0 +1,224 @@
+# frozen_string_literal: true
+
+module Herringbone
+  module Codecs
+    # Pure-Ruby LZO1X decompressor, for reading Parquet files written by parquet-mr with
+    # hadoop-lzo. There is no compressor: nothing but hadoop-lzo writes LZO, and nothing
+    # outside the JVM reads it.
+    #
+    # Written from the bitstream description in the Linux kernel's
+    # Documentation/staging/lzo.rst, not from the (GPL) LZO sources.
+    module LZO
+      # Raised for corrupt or truncated LZO input.
+      class Error < StandardError; end
+
+      # Hadoop block header: big-endian 32-bit uncompressed size of the block.
+      HADOOP_BLOCK_PREFIX = 4
+      # Hadoop chunk header: big-endian 32-bit compressed size of the chunk.
+      HADOOP_CHUNK_PREFIX = 4
+      # Shorthand for Encoding::BINARY.
+      BINARY = Encoding::BINARY
+
+      module_function
+
+      # Parquet LZO (codec 3). hadoop-lzo writes Hadoop block framing; data that does not fit
+      # the framing is decoded as one bare LZO1X stream.
+      # @param input [String] compressed page data
+      # @param uncompressed_size [Integer] exact decompressed size, from the page header
+      # @return [String] decompressed bytes in ASCII-8BIT
+      # @raise [Error] if neither layout decodes
+      def decompress_hadoop(input, uncompressed_size)
+        src = binary(input)
+        try_hadoop(src, uncompressed_size) || decompress_block(src, uncompressed_size)
+      end
+
+      # Decompress one bare LZO1X stream that must expand to exactly uncompressed_size bytes.
+      # @param input [String] LZO1X stream, ending with the end-of-stream marker
+      # @param uncompressed_size [Integer] exact decompressed size
+      # @return [String] decompressed bytes in ASCII-8BIT
+      # @raise [Error] if the stream is corrupt or does not decode to +uncompressed_size+ bytes
+      def decompress_block(input, uncompressed_size)
+        src = binary(input)
+        out = String.new(capacity: uncompressed_size, encoding: BINARY)
+        ip = decode_block(src, 0, src.bytesize, out, uncompressed_size)
+        raise Error, "#{src.bytesize - ip} bytes after the end of the LZO stream" unless ip == src.bytesize
+        unless out.bytesize == uncompressed_size
+          raise Error, "LZO stream decoded to #{out.bytesize} bytes, expected #{uncompressed_size}"
+        end
+        out
+      end
+
+      # @param str [String] input in any encoding
+      # @return [String] +str+ itself if already binary, otherwise a binary copy
+      def binary(str)
+        (str.encoding == BINARY) ? str : str.b
+      end
+
+      # Hadoop's BlockCompressorStream layout: blocks of (uncompressed size, then chunks of
+      # (compressed size, LZO stream) until the block is filled). Each chunk is a separate stream.
+      # @param src [String] binary input
+      # @param uncompressed_size [Integer] exact decompressed size expected over all blocks
+      # @return [String, nil] decompressed bytes, or nil if +src+ is not valid Hadoop-framed LZO
+      def try_hadoop(src, uncompressed_size)
+        n = src.bytesize
+        out = String.new(capacity: uncompressed_size, encoding: BINARY)
+        ip = 0
+        while ip < n
+          return nil if n - ip < HADOOP_BLOCK_PREFIX
+          block_end = out.bytesize + src.byteslice(ip, HADOOP_BLOCK_PREFIX).unpack1("N")
+          ip += HADOOP_BLOCK_PREFIX
+          return nil if block_end > uncompressed_size
+
+          while out.bytesize < block_end
+            return nil if n - ip < HADOOP_CHUNK_PREFIX
+            csize = src.byteslice(ip, HADOOP_CHUNK_PREFIX).unpack1("N")
+            ip += HADOOP_CHUNK_PREFIX
+            return nil if csize > n - ip
+            begin
+              return nil unless decode_block(src, ip, ip + csize, out, block_end) == ip + csize
+            rescue Error
+              return nil
+            end
+            ip += csize
+          end
+        end
+        (out.bytesize == uncompressed_size) ? out : nil
+      end
+
+      # Decode one LZO1X stream from src[ip...iend], appending to out. Matches may only reach
+      # back to where this stream's output starts. out may not grow beyond limit.
+      #
+      # Every instruction is a match optionally followed by 0-3 literals (its low two bits, the
+      # "state"), or a run of 4+ literals. What an instruction byte below 16 means depends on
+      # the state the previous instruction left behind.
+      # @param src [String] binary input
+      # @param ip [Integer] offset of the stream in +src+
+      # @param iend [Integer] offset just past the stream
+      # @param out [String] binary output buffer, appended to
+      # @param limit [Integer] maximum total byte size of +out+
+      # @return [Integer] input offset just past the end-of-stream marker
+      # @raise [Error] if the stream is truncated, has an invalid distance or overflows +limit+
+      def decode_block(src, ip, iend, out, limit)
+        base = out.bytesize
+        raise Error, "Empty LZO stream" if ip >= iend
+
+        state = 0
+        t = src.getbyte(ip)
+        if t > 17 # the first byte may open with a literal run of its own
+          ip += 1
+          state = t - 17
+          ip = copy_literals(src, ip, iend, out, limit, state)
+          state = 4 if state > 4
+        end
+
+        while true
+          raise Error, "Truncated LZO stream" if ip >= iend
+          t = src.getbyte(ip)
+          ip += 1
+
+          if t < 16
+            if state == 0
+              len = t + 3
+              len, ip = extended_length(src, ip, iend, 18) if t == 0
+              ip = copy_literals(src, ip, iend, out, limit, len)
+              state = 4
+              next
+            end
+            raise Error, "Truncated LZO stream" if ip >= iend
+            dist = (src.getbyte(ip) << 2) + (t >> 2) + 1
+            ip += 1
+            if state == 4
+              dist += 2048
+              len = 3
+            else
+              len = 2
+            end
+            s = t & 3
+          elsif t < 32
+            len = (t & 7) + 2
+            len, ip = extended_length(src, ip, iend, 9) if t & 7 == 0
+            raise Error, "Truncated LZO stream" if ip + 2 > iend
+            le = src.getbyte(ip) | (src.getbyte(ip + 1) << 8)
+            ip += 2
+            dist = ((t & 8) << 11) + (le >> 2)
+            return ip if dist == 0 # end of stream
+            dist += 16_384
+            s = le & 3
+          elsif t < 64
+            len = (t & 31) + 2
+            len, ip = extended_length(src, ip, iend, 33) if t & 31 == 0
+            raise Error, "Truncated LZO stream" if ip + 2 > iend
+            le = src.getbyte(ip) | (src.getbyte(ip + 1) << 8)
+            ip += 2
+            dist = (le >> 2) + 1
+            s = le & 3
+          else
+            raise Error, "Truncated LZO stream" if ip >= iend
+            len = (t >> 5) + 1 # 3-4 bytes for 01LDDDSS, 5-8 for 1LLDDDSS
+            dist = (src.getbyte(ip) << 3) + ((t >> 2) & 7) + 1
+            ip += 1
+            s = t & 3
+          end
+
+          copy_match(out, base, dist, len, limit)
+          ip = copy_literals(src, ip, iend, out, limit, s) if s > 0
+          state = s
+        end
+      end
+
+      # Reads a length extension: a byte of 0 for each 255, then the nonzero remainder.
+      # @param src [String] binary input
+      # @param ip [Integer] offset of the first extension byte
+      # @param iend [Integer] offset just past the stream
+      # @param bias [Integer] added to the extension
+      # @return [Array(Integer, Integer)] the length, and the offset just past the extension
+      # @raise [Error] if the extension runs past +iend+
+      def extended_length(src, ip, iend, bias)
+        len = bias
+        while true
+          raise Error, "Truncated LZO length" if ip >= iend
+          b = src.getbyte(ip)
+          ip += 1
+          return [len + b, ip] unless b == 0
+          len += 255
+        end
+      end
+
+      # @param src [String] binary input
+      # @param ip [Integer] offset of the literals in +src+
+      # @param iend [Integer] offset just past the stream
+      # @param out [String] binary output buffer, appended to
+      # @param limit [Integer] maximum total byte size of +out+
+      # @param len [Integer] number of literal bytes
+      # @return [Integer] offset just past the literals
+      # @raise [Error] if the literals run past +iend+ or +out+ would exceed +limit+
+      def copy_literals(src, ip, iend, out, limit, len)
+        raise Error, "LZO literals run past end of input" if ip + len > iend
+        raise Error, "LZO output exceeds expected size" if out.bytesize + len > limit
+        out << src.byteslice(ip, len)
+        ip + len
+      end
+
+      # @param out [String] binary output buffer, appended to
+      # @param base [Integer] size of +out+ when the current stream started
+      # @param dist [Integer] how far back the match starts
+      # @param len [Integer] match length, which may exceed +dist+ (a repeating pattern)
+      # @param limit [Integer] maximum total byte size of +out+
+      # @return [String] +out+
+      # @raise [Error] if the match reaches before the stream's output or +out+ would exceed +limit+
+      def copy_match(out, base, dist, len, limit)
+        pos = out.bytesize
+        raise Error, "LZO match distance #{dist} reaches before start of output" if dist > pos - base
+        raise Error, "LZO output exceeds expected size" if pos + len > limit
+        if dist >= len
+          out << out.byteslice(pos - dist, len)
+        else
+          pattern = out.byteslice(pos - dist, dist)
+          out << (pattern * (len / dist)) << pattern.byteslice(0, len % dist)
+        end
+      end
+
+      private_class_method :binary, :try_hadoop, :decode_block, :extended_length, :copy_literals, :copy_match
+    end
+  end
+end

data/lib/herringbone/codecs/snappy.rb

@@ -7,7 +7,7 @@ module Herringbone
     # Pure-Ruby implementation of the raw Snappy block format (as used by Parquet),
     # see https://github.com/google/snappy/blob/main/format_description.txt
     #
-    # When the `snappy` gem (a binding to Google's libsnappy) can be loaded, it is used instead:
+    # When the `snappy` gem (a binding to Google's libsnappy) is loaded, it is used instead:
     # 12x faster decompression and 27x faster compression. It is optional and only a speedup: if
     # it is missing, the pure-Ruby code below is used silently. Both produce raw Snappy blocks the
     # other reads.
@@ -30,7 +30,7 @@ module Herringbone
       # The length preamble is a 32-bit varint, so no block may decompress to more than this.
       MAX_UNCOMPRESSED = (1 << 32) - 1
 
-      # Name of the optional native gem, passed to +require+.
+      # Name of the optional native gem.
       NATIVE_GEM = "snappy"
 
       module_function
@@ -57,41 +57,34 @@ module Herringbone
         native ? :native : :ruby
       end
 
-      # For tests and benchmarks: :ruby forces pure Ruby, :native requires the snappy gem
-      # (UnsupportedError if it cannot be loaded), nil goes back to the default
+      # For tests and benchmarks: :ruby forces pure Ruby, :native the snappy gem
+      # (UnsupportedError if it is not loaded), nil goes back to the default
       # @param name [Symbol, nil] +:ruby+, +:native+ or nil
       # @return [void]
-      # @raise [UnsupportedError] if +:native+ is requested and the gem cannot be loaded
+      # @raise [UnsupportedError] if +:native+ is requested and the gem is not loaded
       # @raise [ArgumentError] for any other backend name
       def backend=(name)
         @native = case name
         when :ruby then false
         when :native
-          native_library || raise(UnsupportedError, "The \"#{NATIVE_GEM}\" gem could not be loaded")
+          native_library || raise(UnsupportedError, "The \"#{NATIVE_GEM}\" gem is not loaded (require \"#{NATIVE_GEM}\")")
         when nil then nil
         else raise ArgumentError, "Unknown Snappy backend #{name.inspect} (expected :ruby, :native or nil)"
         end
       end
 
-      # The native library to use, resolved (and memoized) on first call unless forced by backend=.
+      # The native library to use: the snappy gem when it is loaded, unless backend= forced one.
       # @return [Module, nil] the +::Snappy+ module, or nil when the pure-Ruby code should run
       def native
-        @native = native_library || false if @native.nil?
-        @native || nil
+        native = @native
+        native = native_library if native.nil?
+        native || nil
       end
 
-      # Tries to load the snappy gem once and memoizes the result.
-      # @return [Module, false] the +::Snappy+ module, or false if it is missing or lacks inflate/deflate
+      # Not memoized, so that it works the same in every Ractor and once the gem is required later.
+      # @return [Module, false] the +::Snappy+ module, or false if it is not loaded or lacks inflate/deflate
       def native_library
-        if @native_lib.nil?
-          @native_lib = begin
-            require NATIVE_GEM
-            (::Snappy.respond_to?(:inflate) && ::Snappy.respond_to?(:deflate)) ? ::Snappy : false
-          rescue LoadError
-            false
-          end
-        end
-        @native_lib
+        (defined?(::Snappy.inflate) && defined?(::Snappy.deflate)) ? ::Snappy : false
       end
 
       # Decompresses into a preallocated IO::Buffer: copies do not allocate intermediate Strings.