zfp 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 74703e564c8c5a5b8f5acae9b29aeb8faaf494769a2be01216060f54f51e91d9
+  data.tar.gz: 84c36071e18794d6c8a174c6fe7508a7f5acad282dc7d574a8565cb3cad88082
+SHA512:
+  metadata.gz: 4bd205cf6669f1d47c9bdb16328b3d23558626bcdeb8066440ec5ef4ee1195822379bf2e13272aefd5ed9fba286873d0f4b796e260f7b0fa157cd521e3d582bd
+  data.tar.gz: 58c293a2b8d1d69740f56b82f6811819d59c50d017d4190d23ad3679e448a0099d2e966c264b5792b2f6dd65dd26470dab0b4c9f3d748148b63c55c9eab828fb

data/README.md

@@ -0,0 +1,346 @@
+# zfp — ZFP Floating-Point Compression for Ruby
+
+> _Because your floats deserve better than Base64._
+
+[![Gem Version](https://img.shields.io/gem/v/zfp)](https://rubygems.org/gems/zfp)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
+
+**`zfp`** brings [LLNL's battle-hardened ZFP compression library](https://computing.llnl.gov/projects/zfp) to Ruby. ZFP was built by national-lab scientists to compress petabytes of floating-point simulation data without losing the ability to do science on it. Now it's in a Ruby gem. You're welcome, science.
+
+Whether you're cramming ten years of OHLCV market data into Redis, shipping a million embedding vectors over the wire, or just deeply offended by how wasteful `Array#pack("E*")` is, this gem is for you.
+
+---
+
+## What ZFP Actually Does
+
+ZFP compresses *n*-dimensional arrays of floats, doubles, int32s, and int64s — up to 4 dimensions — using a floating-point-aware transform that exploits spatial correlation across array elements. Unlike general-purpose compressors, it understands the structure of numeric data.
+
+It offers four compression modes:
+
+| Mode | What it does | Good for |
+|---|---|---|
+| `:reversible` | Bit-exact lossless | Audit trails, exact P&L, storing anything you'll diff |
+| `:fixed_rate` | Guaranteed bits-per-value | Streaming, fixed-size storage slots |
+| `:fixed_precision` | Guaranteed significant bits | Scientific reproducibility |
+| `:fixed_accuracy` | Guaranteed absolute error bound | Financial data, ML embeddings, anything with a tolerance |
+
+---
+
+## Installation
+
+You need `libzfp` installed first. Install it with [upkg](https://github.com/seuros/upkg):
+
+```bash
+upkg install zfp
+```
+
+Then add the gem:
+
+```bash
+bundle add zfp
+```
+
+Or install directly:
+
+```bash
+gem install zfp
+```
+
+No native compilation. No waiting. The binding uses [ruby-ffi](https://github.com/ffi/ffi), so `gem install` is instant and the library loads at runtime.
+
+---
+
+## Five-Minute Quickstart
+
+```ruby
+require "zfp"
+
+prices = [174.21, 174.85, 173.40, 175.10, 176.33]  # ... 10,000 more of these
+
+# Lossless round-trip — bit-exact, no questions asked
+compressed = Zfp.compress(prices, type: :double, shape: [prices.size], mode: :reversible)
+restored   = Zfp.decompress(compressed, type: :double, shape: [prices.size], mode: :reversible)
+
+prices == restored  # => true
+
+# Self-describing pack — no metadata bookkeeping required
+packed   = Zfp.pack(prices, type: :double, shape: [prices.size], mode: :reversible)
+restored = Zfp.unpack(packed)  # type, shape, mode all embedded — no args needed
+
+prices == restored  # => true
+```
+
+That's it. The rest is just choosing how hard you want to squeeze.
+
+---
+
+## API Reference
+
+There are three ways to use the gem. Pick whichever fits your architecture.
+
+### Module methods — the easy path
+
+**Raw bytes** (`compress` / `decompress`)
+
+The caller manages metadata (type, shape, mode). Returns a plain `String` of compressed bytes.
+
+```ruby
+bytes = Zfp.compress(data, type: :double, shape: [1000], mode: :reversible)
+data  = Zfp.decompress(bytes, type: :double, shape: [1000], mode: :reversible)
+
+# Lossy — absolute error bounded to $0.001 per element
+bytes = Zfp.compress(prices, type: :double, shape: [1000], mode: :fixed_accuracy, tolerance: 0.001)
+data  = Zfp.decompress(bytes, type: :double, shape: [1000], mode: :fixed_accuracy, tolerance: 0.001)
+```
+
+**Self-describing bytes** (`pack` / `unpack`)
+
+Metadata is embedded in a 32-byte header. Pass bytes anywhere — Redis, S3, a message queue — and unpack without needing a schema.
+
+```ruby
+# Pack: type/shape/mode/params stored in the bytes themselves
+packed   = Zfp.pack(data, type: :double, shape: [1000], mode: :fixed_accuracy, tolerance: 0.001)
+
+# Unpack: zero arguments needed
+restored = Zfp.unpack(packed)
+```
+
+### `Zfp::Codec` — when you compress many arrays with the same config
+
+Build a codec once. Compress forever.
+
+```ruby
+codec = Zfp::Codec.new(type: :double, shape: [252], mode: :fixed_accuracy, tolerance: 0.001)
+
+# Same codec compresses each security's yearly price history
+securities.each do |ticker|
+  store[ticker] = codec.compress(daily_closes[ticker])
+end
+
+# Retrieve and decompress
+prices = codec.decompress(store["AAPL"])
+
+# Or pack (self-describing) via codec
+packed = codec.pack(daily_closes["AAPL"])
+```
+
+---
+
+## Multi-Dimensional Arrays
+
+ZFP natively understands 1-D through 4-D structure. Providing the actual shape (not just total element count) lets it exploit correlation across all axes simultaneously — the more structure you describe, the better it compresses.
+
+```ruby
+# 1-D time series
+Zfp.compress(prices, type: :double, shape: [252], mode: :reversible)
+
+# 2-D matrix — e.g. 50 securities × 252 days
+matrix = securities.flat_map { |s| daily_closes[s] }
+Zfp.compress(matrix, type: :double, shape: [50, 252], mode: :fixed_accuracy, tolerance: 0.001)
+
+# 3-D tensor — e.g. 10 portfolios × 30 securities × 252 days
+Zfp.compress(tensor, type: :double, shape: [10, 30, 252], mode: :reversible)
+
+# 4-D — ZFP supports up to 4 dimensions
+Zfp.compress(data4d, type: :double, shape: [4, 8, 16, 16], mode: :reversible)
+```
+
+---
+
+## All Four Scalar Types
+
+```ruby
+# Floating-point — all modes available
+Zfp.compress(floats,   type: :float,  shape: [n], mode: :reversible)
+Zfp.compress(doubles,  type: :double, shape: [n], mode: :fixed_accuracy, tolerance: 1e-6)
+
+# Integer — reversible mode only (already lossless by nature)
+Zfp.compress(counts,   type: :int32,  shape: [n], mode: :reversible)
+Zfp.compress(ids,      type: :int64,  shape: [n], mode: :reversible)
+```
+
+---
+
+## Numo::NArray Support
+
+If [numo-narray](https://github.com/ruby-numo/numo-narray) is loaded, the gem auto-detects type and shape from Numo arrays. No `type:` or `shape:` arguments needed on compress or pack.
+
+```ruby
+require "numo/narray"
+
+# Input: Numo::DFloat — type (:double) and shape inferred automatically
+closes = Numo::DFloat.cast(daily_prices).reshape(50, 252)
+bytes  = Zfp.compress(closes, mode: :fixed_accuracy, tolerance: 0.001)
+packed = Zfp.pack(closes,     mode: :reversible)
+
+# Output: get a Numo array back with numo: true
+result = Zfp.decompress(bytes, type: :double, shape: [50, 252], mode: :fixed_accuracy,
+                               tolerance: 0.001, numo: true)
+# => Numo::DFloat[50, 252]
+
+# pack/unpack preserves Numo in → Numo out automatically
+result = Zfp.unpack(packed)
+# => Numo::DFloat[50, 252]
+
+# All four Numo types supported:
+#   Numo::SFloat  → :float
+#   Numo::DFloat  → :double
+#   Numo::Int32   → :int32
+#   Numo::Int64   → :int64
+```
+
+---
+
+## Compression Mode Guide
+
+### `:reversible` — lossless, bit-exact
+
+Use this when correctness is non-negotiable. Compression ratios depend on data entropy:
+sequential integers and smooth time series compress well; high-entropy noise barely shrinks.
+
+```ruby
+bytes = Zfp.compress(data, type: :double, shape: [n], mode: :reversible)
+# Typical ratios: 2x–6x on financial/scientific data
+```
+
+### `:fixed_accuracy` — absolute error bound
+
+The workhorse for financial and ML workloads. You set a maximum per-element error;
+ZFP uses as few bits as needed to honor it.
+
+```ruby
+# Error on any element guaranteed ≤ tolerance
+bytes = Zfp.compress(prices, type: :double, shape: [n], mode: :fixed_accuracy, tolerance: 0.001)
+# $0.001 per price point — indistinguishable for P&L calculations
+# Typical ratios: 3x–8x
+```
+
+### `:fixed_precision` — significant bits
+
+Useful when you want to preserve a specific number of significant bits rather than
+an absolute error bound. Handy for scientific data where relative precision matters more than absolute.
+
+```ruby
+bytes = Zfp.compress(data, type: :double, shape: [n], mode: :fixed_precision, precision: 20)
+# 20 significant bits preserved (of 52 available in a double)
+```
+
+### `:fixed_rate` — guaranteed bytes per value
+
+Use when you need fixed-size storage slots — e.g., each block in a columnar store
+must be exactly the same size. The `rate` is bits per scalar value.
+
+```ruby
+bytes = Zfp.compress(data, type: :double, shape: [n], mode: :fixed_rate, rate: 16.0)
+# 16 bits per value → 4x smaller than raw double
+# WARNING: aggressive rates (< 8) can produce large errors on high-dynamic-range data.
+#          Always validate max_err against your tolerance before committing to a rate.
+```
+
+---
+
+## Real-World Performance
+
+Benchmarked on a portfolio of 5 × ~20 securities × 252 trading days of close prices
+(synthetic GBM data, see `examples/portfolio_performance.rb`):
+
+| Mode | Ratio | Max Error |
+|---|---|---|
+| `:reversible` | 1.1×–1.2× | 0 (exact) |
+| `:fixed_accuracy, tolerance: 0.001` | 3.0×–3.4× | < $0.001 |
+| `:fixed_accuracy, tolerance: 0.01` | 3.4× | < $0.01 |
+
+ZFP shines brightest on correlated data — real market prices (correlated sectors, macro
+moves, mean reversion) compress significantly better than the GBM baseline above.
+
+For ML embedding vectors (1536-dim float32, high spatial correlation):
+expect 4×–10× with `:fixed_accuracy` and a tolerance tuned to preserve cosine similarity.
+
+---
+
+## Error Handling
+
+```ruby
+Zfp::LibraryNotFound     # libzfp not installed or not found
+Zfp::InvalidType         # unrecognized :type symbol
+Zfp::InvalidMode         # unrecognized :mode symbol
+Zfp::InvalidShape        # empty shape, > 4 dimensions, or non-positive dimension
+Zfp::InvalidParams       # missing or invalid mode-specific param (rate/precision/tolerance)
+Zfp::CompressionFailed   # libzfp returned an error during compress
+Zfp::DecompressionFailed # libzfp returned an error during decompress
+Zfp::PackerError         # corrupt or truncated pack header
+```
+
+All inherit from `Zfp::Error < StandardError`.
+
+---
+
+## When to Use This Gem
+
+- **In-memory caches of numerical time series** — compress to a byte string, store in Redis,
+  decompress on demand. Transparent to the rest of your code.
+- **Columnar financial data** — years of OHLCV for thousands of tickers at a fraction of raw size.
+- **ML embedding storage** — vectors stay geometrically faithful under `:fixed_accuracy` at
+  tolerances that preserve cosine similarity.
+- **Message queues** — pack/unpack means the consumer needs no schema; bytes are self-describing.
+- **Checkpointing numerical computation** — save intermediate Numo arrays between steps.
+
+## When Not to Use This Gem
+
+- **Random or already-compressed data** — ZFP assumes spatial correlation. Noise, encrypted data,
+  and random floats will barely shrink and may grow.
+- **Text, blobs, or non-numeric data** — use MessagePack, zstd, or zlib.
+- **Tiny arrays (< ~50 elements)** — the 32-byte pack header plus ZFP block overhead dominates.
+- **Integer data needing lossy compression** — integer types support `:reversible` only.
+
+---
+
+## Examples
+
+```bash
+bundle exec ruby examples/01_basic_usage.rb
+bundle exec ruby examples/portfolio_performance.rb
+```
+
+---
+
+## Requirements
+
+- Ruby ≥ 3.0
+- libzfp 1.0+ — install with [upkg](https://github.com/seuros/upkg): `upkg install zfp`
+- [ffi](https://github.com/ffi/ffi) ~> 1.0 (installed automatically)
+- [numo-narray](https://github.com/ruby-numo/numo-narray) _(optional — loaded automatically if present)_
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/madbomber/zfp
+cd zfp
+bundle install
+bundle exec rake test
+```
+
+Tests that require libzfp are automatically skipped if the library isn't found,
+so the suite is safe to run anywhere.
+
+---
+
+## Contributing
+
+Bug reports and pull requests are welcome at https://github.com/madbomber/zfp.
+Please include a failing test for bugs and a passing test for features.
+
+---
+
+## References
+
+- [ZFP — Compressed Floating-Point and Integer Arrays](https://computing.llnl.gov/projects/zfp) — official project page at Lawrence Livermore National Laboratory
+- [ZFP on GitHub](https://github.com/LLNL/zfp) — source, documentation, and compression algorithm details
+
+---
+
+## License
+
+MIT. Go nuts.

data/lib/zfp/codec.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+
+module Zfp
+  class Codec
+    VALID_TYPES = %i[float double int32 int64].freeze
+    VALID_MODES = %i[fixed_rate fixed_precision fixed_accuracy reversible].freeze
+
+    def initialize(type:, shape:, mode:, numo: false, **params)
+      validate_type!(type)
+      validate_mode!(mode)
+      validate_shape!(shape)
+      validate_params!(mode, params)
+      @type   = type
+      @shape  = shape
+      @mode   = mode
+      @params = params
+      @numo   = numo
+    end
+
+    def compress(data)
+      buf   = TypeCoercion.to_buffer(data, @type)
+      field = Field.new(@type, @shape, buf)
+      Stream.new(@mode, @params).compress(field)
+    ensure
+      field&.free
+    end
+
+    def decompress(bytes)
+      count     = @shape.reduce(:*)
+      elem_size = TypeCoercion::ELEMENT_SIZE[@type]
+      out_buf   = ::FFI::MemoryPointer.new(:uint8, count * elem_size)
+      field     = Field.new(@type, @shape, out_buf)
+      Stream.new(@mode, @params).decompress(field, bytes)
+      TypeCoercion.from_buffer(out_buf, @type, @shape, @numo)
+    ensure
+      field&.free
+    end
+
+    def pack(data)
+      as_numo    = @numo || TypeCoercion.numo?(data)
+      compressed = compress(data)
+      Packer.encode(compressed, type: @type, shape: @shape, mode: @mode,
+                    params: @params, numo: as_numo)
+    end
+
+    private
+
+    def validate_type!(type)
+      raise Zfp::InvalidType, "Unknown type: #{type.inspect}. Valid: #{VALID_TYPES}" \
+        unless VALID_TYPES.include?(type)
+    end
+
+    def validate_mode!(mode)
+      raise Zfp::InvalidMode, "Unknown mode: #{mode.inspect}. Valid: #{VALID_MODES}" \
+        unless VALID_MODES.include?(mode)
+    end
+
+    def validate_shape!(shape)
+      unless shape.is_a?(Array) && shape.length.between?(1, 4) &&
+             shape.all? { |d| d.is_a?(Integer) && d > 0 }
+        raise Zfp::InvalidShape,
+          "shape must be Array of 1–4 positive integers, got #{shape.inspect}"
+      end
+    end
+
+    def validate_params!(mode, params)
+      case mode
+      when :fixed_rate
+        raise Zfp::InvalidParams, "fixed_rate requires rate: (Float > 0)" \
+          unless params[:rate]&.to_f&.positive?
+      when :fixed_precision
+        raise Zfp::InvalidParams, "fixed_precision requires precision: (Integer > 0)" \
+          unless params[:precision].is_a?(Integer) && params[:precision] > 0
+      when :fixed_accuracy
+        raise Zfp::InvalidParams, "fixed_accuracy requires tolerance: (Float > 0)" \
+          unless params[:tolerance]&.to_f&.positive?
+      end
+    end
+  end
+end

data/lib/zfp/errors.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module Zfp
+  class Error < StandardError; end
+  class LibraryNotFound     < Error; end
+  class InvalidType         < Error; end
+  class InvalidMode         < Error; end
+  class InvalidShape        < Error; end
+  class InvalidParams       < Error; end
+  class CompressionFailed   < Error; end
+  class DecompressionFailed < Error; end
+  class PackerError         < Error; end
+end

data/lib/zfp/ffi.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require "ffi"
+
+module Zfp
+  module FFI
+    extend ::FFI::Library
+
+    begin
+      ffi_lib "zfp"
+    rescue LoadError
+      raise Zfp::LibraryNotFound,
+        "ZFP library not found. Install with: brew install zfp"
+    end
+
+    ZFP_TYPE_INT32  = 1
+    ZFP_TYPE_INT64  = 2
+    ZFP_TYPE_FLOAT  = 3
+    ZFP_TYPE_DOUBLE = 4
+
+    attach_function :zfp_stream_open,           [:pointer],                                          :pointer
+    attach_function :zfp_stream_close,          [:pointer],                                          :void
+    attach_function :zfp_stream_set_bit_stream, [:pointer, :pointer],                                :void
+    attach_function :zfp_stream_rewind,         [:pointer],                                          :void
+    attach_function :zfp_stream_set_rate,       [:pointer, :double, :int, :uint, :int],              :double
+    attach_function :zfp_stream_set_precision,  [:pointer, :uint],                                   :uint
+    attach_function :zfp_stream_set_accuracy,   [:pointer, :double],                                 :double
+    attach_function :zfp_stream_set_reversible, [:pointer],                                          :void
+    attach_function :zfp_stream_maximum_size,   [:pointer, :pointer],                                :size_t
+    attach_function :zfp_compress,              [:pointer, :pointer],                                :size_t
+    attach_function :zfp_decompress,            [:pointer, :pointer],                                :size_t
+
+    attach_function :zfp_field_alloc,           [],                                                  :pointer
+    attach_function :zfp_field_free,            [:pointer],                                          :void
+    attach_function :zfp_field_set_pointer,     [:pointer, :pointer],                                :void
+    attach_function :zfp_field_set_type,        [:pointer, :int],                                    :void
+    attach_function :zfp_field_set_size_1d,     [:pointer, :size_t],                                 :void
+    attach_function :zfp_field_set_size_2d,     [:pointer, :size_t, :size_t],                        :void
+    attach_function :zfp_field_set_size_3d,     [:pointer, :size_t, :size_t, :size_t],               :void
+    attach_function :zfp_field_set_size_4d,     [:pointer, :size_t, :size_t, :size_t, :size_t],      :void
+
+    attach_function :stream_open,               [:pointer, :size_t],                                 :pointer
+    attach_function :stream_close,              [:pointer],                                          :void
+  end
+end

data/lib/zfp/field.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module Zfp
+  class Field
+    attr_reader :zfp_type_int, :shape
+
+    def initialize(type, shape, buffer)
+      validate_shape!(shape)
+      @type         = type
+      @shape        = shape
+      @buffer       = buffer  # hold reference — prevents GC while field is alive
+      @zfp_type_int = Zfp::TypeCoercion::RUBY_TO_ZFP_TYPE[type]
+      @ptr = Zfp::FFI.zfp_field_alloc
+      raise Zfp::Error, "Failed to allocate zfp_field" if @ptr.null?
+
+      Zfp::FFI.zfp_field_set_pointer(@ptr, buffer)
+      Zfp::FFI.zfp_field_set_type(@ptr, @zfp_type_int)
+      set_shape(shape)
+    end
+
+    def pointer
+      @ptr
+    end
+
+    def dims
+      @shape.length
+    end
+
+    def free
+      return if @ptr.nil? || @ptr.null?
+      Zfp::FFI.zfp_field_free(@ptr)
+      @ptr = ::FFI::Pointer::NULL
+    end
+
+    private
+
+    def validate_shape!(shape)
+      unless shape.is_a?(Array) && shape.length.between?(1, 4) &&
+             shape.all? { |d| d.is_a?(Integer) && d > 0 }
+        raise Zfp::InvalidShape,
+          "shape must be an Array of 1–4 positive integers, got #{shape.inspect}"
+      end
+    end
+
+    def set_shape(shape)
+      case shape.length
+      when 1 then Zfp::FFI.zfp_field_set_size_1d(@ptr, shape[0])
+      when 2 then Zfp::FFI.zfp_field_set_size_2d(@ptr, shape[0], shape[1])
+      when 3 then Zfp::FFI.zfp_field_set_size_3d(@ptr, shape[0], shape[1], shape[2])
+      when 4 then Zfp::FFI.zfp_field_set_size_4d(@ptr, shape[0], shape[1], shape[2], shape[3])
+      end
+    end
+  end
+end

data/lib/zfp/packer.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module Zfp
+  module Packer
+    MAGIC      = "ZFP\x01"
+    HEADER_LEN = 32
+    PACK_FMT   = "a4CCCCVVVVE"
+
+    TYPE_TO_BYTE = { float: 0, double: 1, int32: 2, int64: 3 }.freeze
+    BYTE_TO_TYPE = TYPE_TO_BYTE.invert.freeze
+
+    MODE_TO_BYTE = { fixed_rate: 0, fixed_precision: 1, fixed_accuracy: 2, reversible: 3 }.freeze
+    BYTE_TO_MODE = MODE_TO_BYTE.invert.freeze
+
+    def self.encode(compressed_bytes, type:, shape:, mode:, params: {}, numo: false)
+      flags  = numo ? 1 : 0
+      dims   = shape.length
+      dim0, dim1, dim2, dim3 = Array.new(4) { |i| shape[i].to_i }
+      param  = extract_param(mode, params)
+      header = [MAGIC, TYPE_TO_BYTE[type], MODE_TO_BYTE[mode], dims, flags,
+                dim0, dim1, dim2, dim3, param].pack(PACK_FMT)
+      header + compressed_bytes
+    end
+
+    def self.decode(bytes)
+      if bytes.bytesize < HEADER_LEN
+        raise Zfp::PackerError, "truncated header (#{bytes.bytesize} < #{HEADER_LEN} bytes)"
+      end
+
+      magic, type_b, mode_b, rank, flags, d0, d1, d2, d3, param =
+        bytes[0, HEADER_LEN].unpack(PACK_FMT)
+
+      raise Zfp::PackerError, "bad magic bytes (expected ZFP\\x01)" unless magic == MAGIC
+
+      type  = BYTE_TO_TYPE.fetch(type_b) { raise Zfp::PackerError, "unknown type byte #{type_b}" }
+      mode  = BYTE_TO_MODE.fetch(mode_b) { raise Zfp::PackerError, "unknown mode byte #{mode_b}" }
+      shape = [d0, d1, d2, d3].first(rank)
+      numo  = flags & 1 == 1
+      params = decode_params(mode, param)
+      data  = bytes[HEADER_LEN..]
+
+      [type, shape, mode, params, numo, data]
+    end
+
+    private_class_method def self.extract_param(mode, params)
+      case mode
+      when :fixed_rate      then params[:rate].to_f
+      when :fixed_precision then params[:precision].to_f
+      when :fixed_accuracy  then params[:tolerance].to_f
+      else 0.0
+      end
+    end
+
+    private_class_method def self.decode_params(mode, param)
+      case mode
+      when :fixed_rate      then { rate: param }
+      when :fixed_precision then { precision: param.round }
+      when :fixed_accuracy  then { tolerance: param }
+      else {}
+      end
+    end
+  end
+end

data/lib/zfp/stream.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+module Zfp
+  class Stream
+    def initialize(mode, params)
+      @mode   = mode
+      @params = params
+    end
+
+    def compress(field)
+      zfp_ptr = bitstream = output_buf = nil
+      zfp_ptr = Zfp::FFI.zfp_stream_open(nil)
+      apply_mode(zfp_ptr, field)
+      bufsize    = Zfp::FFI.zfp_stream_maximum_size(zfp_ptr, field.pointer)
+      output_buf = ::FFI::MemoryPointer.new(:uint8, bufsize)
+      bitstream  = Zfp::FFI.stream_open(output_buf, bufsize)
+      Zfp::FFI.zfp_stream_set_bit_stream(zfp_ptr, bitstream)
+      Zfp::FFI.zfp_stream_rewind(zfp_ptr)
+      written = Zfp::FFI.zfp_compress(zfp_ptr, field.pointer)
+      raise Zfp::CompressionFailed, "zfp_compress returned 0" if written == 0
+      output_buf.read_bytes(written)
+    ensure
+      Zfp::FFI.stream_close(bitstream) if bitstream && !bitstream.null?
+      Zfp::FFI.zfp_stream_close(zfp_ptr) if zfp_ptr && !zfp_ptr.null?
+    end
+
+    def decompress(field, compressed_bytes)
+      zfp_ptr = bitstream = input_buf = nil
+      input_buf = ::FFI::MemoryPointer.new(:uint8, compressed_bytes.bytesize)
+      input_buf.put_bytes(0, compressed_bytes)
+      zfp_ptr  = Zfp::FFI.zfp_stream_open(nil)
+      apply_mode(zfp_ptr, field)
+      bitstream = Zfp::FFI.stream_open(input_buf, compressed_bytes.bytesize)
+      Zfp::FFI.zfp_stream_set_bit_stream(zfp_ptr, bitstream)
+      Zfp::FFI.zfp_stream_rewind(zfp_ptr)
+      read = Zfp::FFI.zfp_decompress(zfp_ptr, field.pointer)
+      raise Zfp::DecompressionFailed, "zfp_decompress returned 0" if read == 0
+    ensure
+      Zfp::FFI.stream_close(bitstream) if bitstream && !bitstream.null?
+      Zfp::FFI.zfp_stream_close(zfp_ptr) if zfp_ptr && !zfp_ptr.null?
+    end
+
+    private
+
+    def apply_mode(zfp_ptr, field)
+      case @mode
+      when :fixed_rate
+        Zfp::FFI.zfp_stream_set_rate(zfp_ptr, @params[:rate].to_f,
+                                     field.zfp_type_int, field.dims, 0)
+      when :fixed_precision
+        Zfp::FFI.zfp_stream_set_precision(zfp_ptr, @params[:precision])
+      when :fixed_accuracy
+        Zfp::FFI.zfp_stream_set_accuracy(zfp_ptr, @params[:tolerance].to_f)
+      when :reversible
+        Zfp::FFI.zfp_stream_set_reversible(zfp_ptr)
+      else
+        raise Zfp::InvalidMode, "Unknown mode: #{@mode}"
+      end
+    end
+  end
+end

data/lib/zfp/type_coercion.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+module Zfp
+  module TypeCoercion
+    RUBY_TO_ZFP_TYPE = {
+      int32:  Zfp::FFI::ZFP_TYPE_INT32,
+      int64:  Zfp::FFI::ZFP_TYPE_INT64,
+      float:  Zfp::FFI::ZFP_TYPE_FLOAT,
+      double: Zfp::FFI::ZFP_TYPE_DOUBLE
+    }.freeze
+
+    PACK_FORMAT = {
+      float:  "e*",
+      double: "E*",
+      int32:  "l<*",
+      int64:  "q<*"
+    }.freeze
+
+    ELEMENT_SIZE = {
+      float:  4,
+      double: 8,
+      int32:  4,
+      int64:  8
+    }.freeze
+
+    class << self
+      def detect_type(data)
+        return nil unless numo?(data)
+        case data
+        when Numo::SFloat then :float
+        when Numo::DFloat then :double
+        when Numo::Int32  then :int32
+        when Numo::Int64  then :int64
+        else raise Zfp::InvalidType, "Unsupported Numo type: #{data.class}"
+        end
+      end
+
+      def detect_shape(data)
+        return nil unless numo?(data)
+        data.shape
+      end
+
+      def numo?(data)
+        defined?(Numo::NArray) && data.is_a?(Numo::NArray)
+      end
+
+      def to_buffer(data, type)
+        numo?(data) ? numo_to_buffer(data, type) : array_to_buffer(data, type)
+      end
+
+      def from_buffer(ptr, type, shape, as_numo)
+        as_numo ? buffer_to_numo(ptr, type, shape) : buffer_to_array(ptr, type, shape)
+      end
+
+      private
+
+      def array_to_buffer(array, type)
+        fmt = PACK_FORMAT.fetch(type) { raise Zfp::InvalidType, "Unknown type: #{type}" }
+        packed = array.flatten.pack(fmt)
+        ptr = ::FFI::MemoryPointer.new(:uint8, packed.bytesize)
+        ptr.put_bytes(0, packed)
+        ptr
+      end
+
+      def numo_to_buffer(narray, type)
+        fmt = PACK_FORMAT.fetch(type) { raise Zfp::InvalidType, "Unknown type: #{type}" }
+        packed = narray.to_a.flatten.pack(fmt)
+        ptr = ::FFI::MemoryPointer.new(:uint8, packed.bytesize)
+        ptr.put_bytes(0, packed)
+        ptr
+      end
+
+      def buffer_to_array(ptr, type, shape)
+        count = shape.reduce(:*)
+        bytes = ptr.read_bytes(count * ELEMENT_SIZE[type])
+        bytes.unpack(PACK_FORMAT[type])
+      end
+
+      def buffer_to_numo(ptr, type, shape)
+        count = shape.reduce(:*)
+        bytes = ptr.read_bytes(count * ELEMENT_SIZE[type])
+        array = bytes.unpack(PACK_FORMAT[type])
+        numo_class_for(type).cast(array).reshape(*shape)
+      end
+
+      def numo_class_for(type)
+        case type
+        when :float  then Numo::SFloat
+        when :double then Numo::DFloat
+        when :int32  then Numo::Int32
+        when :int64  then Numo::Int64
+        else raise Zfp::InvalidType, "Unknown type: #{type}"
+        end
+      end
+    end
+  end
+end

data/lib/zfp/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Zfp
+  VERSION = "0.1.0"
+end

data/lib/zfp.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require_relative "zfp/version"
+require_relative "zfp/errors"
+require_relative "zfp/ffi"
+require_relative "zfp/type_coercion"
+require_relative "zfp/field"
+require_relative "zfp/stream"
+require_relative "zfp/packer"
+require_relative "zfp/codec"
+
+module Zfp
+  def self.compress(data, type: nil, shape: nil, mode:, **params)
+    type  ||= TypeCoercion.detect_type(data)  or raise InvalidType,  "type: required for Ruby Array input"
+    shape ||= TypeCoercion.detect_shape(data) or raise InvalidShape, "shape: required for Ruby Array input"
+    Codec.new(type: type, shape: shape, mode: mode, **params).compress(data)
+  end
+
+  def self.decompress(bytes, type:, shape:, mode:, numo: false, **params)
+    Codec.new(type: type, shape: shape, mode: mode, numo: numo, **params).decompress(bytes)
+  end
+
+  def self.pack(data, type: nil, shape: nil, mode:, **params)
+    type  ||= TypeCoercion.detect_type(data)  or raise InvalidType,  "type: required for Ruby Array input"
+    shape ||= TypeCoercion.detect_shape(data) or raise InvalidShape, "shape: required for Ruby Array input"
+    as_numo = TypeCoercion.numo?(data)
+    Codec.new(type: type, shape: shape, mode: mode, numo: as_numo, **params).pack(data)
+  end
+
+  def self.unpack(bytes)
+    type, shape, mode, params, numo, compressed = Packer.decode(bytes)
+    Codec.new(type: type, shape: shape, mode: mode, numo: numo, **params).decompress(compressed)
+  end
+end

data/sig/zfp.rbs

@@ -0,0 +1,4 @@
+module Zfp
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,109 @@
+--- !ruby/object:Gem::Specification
+name: zfp
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Dewayne VanHoozer
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: ffi
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: numo-narray
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: debug_me
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: |
+  Compress and decompress arrays of float/double/int32/int64 in 1-4
+  dimensions using ZFP's fixed-rate, fixed-precision, fixed-accuracy,
+  or lossless reversible modes. Accepts Ruby Array and Numo::NArray.
+email:
+- dvanhoozer@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- lib/zfp.rb
+- lib/zfp/codec.rb
+- lib/zfp/errors.rb
+- lib/zfp/ffi.rb
+- lib/zfp/field.rb
+- lib/zfp/packer.rb
+- lib/zfp/stream.rb
+- lib/zfp/type_coercion.rb
+- lib/zfp/version.rb
+- sig/zfp.rbs
+homepage: https://github.com/madbomber/zfp
+licenses: []
+metadata: {}
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.10
+specification_version: 4
+summary: Ruby FFI bindings for the ZFP floating-point compression library
+test_files: []