probabilistic_machines 0.1.1-aarch64-linux

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 4dcf59b83144e678d83452a03c86846d5295254a1062439beaf8ea6cf0e86e83
+  data.tar.gz: 072644da326d2be1f1aa9f1f2d74b7040db8a9672b9932f38ef054ee84287d10
+SHA512:
+  metadata.gz: eb7669530fdf0f8d02f4db4f72b93139393d300de14031aa4e83a54410a427009d58531b4f1d79baf25a869406876bff873413bbedf95f23322e6ba1c747ff24
+  data.tar.gz: fa4c46cc6e900de7368d6ec5e559eb87726a319a64c8e077c3464d36226fe9362dafd09a420db2d8a0fdc9e26a1c85a8e5c2894e3e55c90a77e638cb0926d831

data/CHANGELOG.md

@@ -0,0 +1,58 @@
+# Changelog
+
+## [0.1.1](https://github.com/seuros/probabilistic_machines/compare/probabilistic_machines-v0.1.0...probabilistic_machines/v0.1.1) (2026-11-01)
+
+
+### Features
+
+* HeavyHitters codec and an infallible exact-size encode ([1cf9442](https://github.com/seuros/probabilistic_machines/commit/1cf944208e191f1cc7847221239204e14f0ad9a9))
+* probabilistic data structures in no_std Rust ([e8c9711](https://github.com/seuros/probabilistic_machines/commit/e8c97117a19e39ecfb2eff588724d9bd3d33be48))
+* Ruby gem and magnus FFI for the five structures ([83144bd](https://github.com/seuros/probabilistic_machines/commit/83144bd4553f2ea51fd8437ab966142b50122cec))
+
+
+### Bug Fixes
+
+* a loaded t-digest honours the compression its constructor requires ([dae877e](https://github.com/seuros/probabilistic_machines/commit/dae877eb1161dfc16f5238329c8c6fdd4b1c78e5))
+* a source install can build, and a failed gem push fails the release ([4b18e42](https://github.com/seuros/probabilistic_machines/commit/4b18e42a9b4c7d6264cc3cc9f685e0cb597de409))
+* a zero cluster budget spun the t-digest fold loop forever ([8f4f146](https://github.com/seuros/probabilistic_machines/commit/8f4f1466ca579a943d8a23aec4075b53314e3824))
+* expose and persist count-min saturation ([0ac83ac](https://github.com/seuros/probabilistic_machines/commit/0ac83ac77d02ddee5d7614d73e95e5d103e823e8))
+* fold t-digest clusters without overflowing their span ([2e33539](https://github.com/seuros/probabilistic_machines/commit/2e33539b0dbcc0eb8d288eedc936c0ece8d7f7bd))
+* interpolate a subnormal t-digest span without underflowing it ([1b05f4e](https://github.com/seuros/probabilistic_machines/commit/1b05f4e5515fa4468730bec3708daf8f57052726))
+* interpolate the t-digest cdf without overflowing the span ([48cd8a1](https://github.com/seuros/probabilistic_machines/commit/48cd8a11f1b2fdd56403af46614fd6a56b8b8309))
+* keep t-digest total weight finite and the cluster count bounded ([0a37d4b](https://github.com/seuros/probabilistic_machines/commit/0a37d4b52cd5a0abd7de862fe0f52e4f987c956e))
+* refuse a fractional compression too ([379c001](https://github.com/seuros/probabilistic_machines/commit/379c0012d2e5c21131ec8b9fb92534e796997be9))
+* refuse a fractional count instead of truncating it to zero ([0b88ae8](https://github.com/seuros/probabilistic_machines/commit/0b88ae8204ebbb761e8aab42df3e7d014501d135))
+* reject a compression a t-digest cannot honour ([f53e07b](https://github.com/seuros/probabilistic_machines/commit/f53e07b3af5377565253e14c7327b90c9975eb95))
+* reject a hyperloglog rank its precision cannot reach ([c41210b](https://github.com/seuros/probabilistic_machines/commit/c41210b66ccfc305240244fcf151990ea317392e))
+* reject a t-digest frame that lost whole clusters ([2e7e0b5](https://github.com/seuros/probabilistic_machines/commit/2e7e0b5594ff14297d98cb5c32380beed1f8192a))
+* reject a t-digest frame whose total weight cannot exist ([4bdf21f](https://github.com/seuros/probabilistic_machines/commit/4bdf21fa69949159aa30a8d400a8c7d11e64fbf1))
+* reject count-min counters its total cannot explain ([4413fe9](https://github.com/seuros/probabilistic_machines/commit/4413fe965cfdad617f46d9f2f2720e13a3aee6d0))
+* reject heavy-hitter records no stream could leave ([3a4172e](https://github.com/seuros/probabilistic_machines/commit/3a4172e730fd7c46b06d39bde2ef2c6034be19d4))
+* ship the core's test files so the packaged crate builds ([b567b39](https://github.com/seuros/probabilistic_machines/commit/b567b39144fd403f92efcb7ac48120be228d3b6f))
+* t-digest overflow and NaN, and report count-min saturation ([f3aa87c](https://github.com/seuros/probabilistic_machines/commit/f3aa87c018465de0618125b3f2f84e6848e57f37))
+* the t-digest arithmetic builds without std ([8009555](https://github.com/seuros/probabilistic_machines/commit/800955549c872ff7db8d257fbe63b18d326c1aa2))
+* validate HLL registers on decode, and the same hash collision ([496c077](https://github.com/seuros/probabilistic_machines/commit/496c0771f9a2c06766c1296c7123ea3b6bd7f4ae))
+* widen heavy-hitter bounds for what a merged table evicted ([41b0a9c](https://github.com/seuros/probabilistic_machines/commit/41b0a9c33f7433b915f448820628604c107d1cad))
+
+## 0.1.0
+
+- Initial release.
+- `ProbabilisticMachines::HyperLogLog`: distinct counts in 1024, 4096, 16384 or
+  65536 registers, with `merge` for union and `standard_error` for the bound.
+- `ProbabilisticMachines::CountMinSketch`: per-item frequencies in a width of
+  1024, 4096 or 16384 by a depth of 4 or 8.
+- `ProbabilisticMachines::TDigest`: quantiles and CDF from clusters, with a
+  runtime compression.
+- `ProbabilisticMachines::CuckooFilter`: membership that can forget, in 1024,
+  4096, 16384 or 65536 buckets, raising `ProbabilisticMachines::Full` when a
+  fingerprint has nowhere to go.
+- `ProbabilisticMachines::HeavyHitters`: Space-Saving top-k over 16, 64 or 256
+  counters, returning `Hitter` objects that carry the overestimate.
+- `dump`/`load` round-trip every sketch through the byte format of the
+  `probabilistic-machines` crate. A `HeavyHitters` blob is the crate's frame
+  followed by the extension's key table, so the tracked strings survive a
+  restart instead of loading back as hashes.
+- RBS signatures for the whole public API.
+- The `probabilistic-machines` crate codes `HeavyHitters<u64, K>`, and with
+  `alloc` hands every structure's frame back from `encoded` instead of filling
+  a caller's buffer, with `decode_into` to read one back in place.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2025 Abdelkader Boudih
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,203 @@
+# ProbabilisticMachines 🔭
+
+> **Survey the whole stream on a fixed fuel budget.** The tender that answers how
+> many, how often and how extreme without carrying the cargo it measured.
+
+Distinct counts, quantiles and frequencies in memory you pick up front.
+
+Counting exactly means keeping every key: a `Set` of user ids, a `Hash` of path
+counts, an array of latencies to sort. These five structures answer the same
+questions from state that was sized at construction time and never grows.
+**AMS Flajolet** takes a reading and discards the sample, which is why its hold
+is the same size on the way home as it was at launch.
+
+---
+
+## 🚀 Launch Sequence
+
+```ruby
+require 'probabilistic_machines'
+
+visitors = ProbabilisticMachines::HyperLogLog.new(registers: 4096)   # 4 KiB
+50_000.times { |id| visitors.add("user-#{id % 12_000}") }
+
+visitors.estimate       # => 12_361.2
+visitors.standard_error # => 0.01625
+```
+
+---
+
+## 🔩 The Native Extension Is Required
+
+There is no pure-Ruby fallback and no `DISABLE_*` switch. HyperLogLog, Count-Min
+Sketch, t-digest, cuckoo hashing and Space-Saving are five algorithms, and a
+second implementation would mean maintaining each of them twice, with two sets
+of accuracy tests and two byte formats to keep in step. The gem is a binding to
+the `probabilistic-machines` crate in `ext/probabilistic_machines_native/core`,
+and it needs Ruby 4.0 and a Rust toolchain (or a platform gem).
+
+---
+
+## 📐 Sizes Are a Fixed Menu
+
+The crate sizes every structure by a const generic, so each size is a separately
+compiled type. The extension carries these and raises `ArgumentError` for
+anything else:
+
+| Class            | Keyword              | Values                        |
+| ---------------- | -------------------- | ----------------------------- |
+| `HyperLogLog`    | `registers:`         | 1024, 4096, 16384, 65536      |
+| `CountMinSketch` | `width:` by `depth:` | 1024, 4096 by 4 or 8          |
+| `CuckooFilter`   | `buckets:`           | 1024, 4096, 16384, 65536      |
+| `HeavyHitters`   | `k:`                 | 16, 64, 256                   |
+| `TDigest`        | `compression:`       | any, it is a runtime argument |
+
+```ruby
+ProbabilisticMachines::HyperLogLog.new(registers: 3000)
+# ArgumentError: registers 3000 is not supported, expected one of [1024, 4096, 16384, 65536]
+```
+
+---
+
+## 🌌 How Many Distinct
+
+```ruby
+hll = ProbabilisticMachines::HyperLogLog.new(registers: 4096)
+hll.add('user-1')
+hll.add_hash(0x9e37_79b9_7f4a_7c15) # for streams that ship hashes, not keys
+hll.estimate                        # Float
+hll.standard_error                  # 1.04 / sqrt(registers)
+hll.merge(other)                    # exact union
+```
+
+Memory is one byte per register, so 4096 registers cost 4 KiB whether the stream
+held ten ids or ten billion.
+
+---
+
+## 📊 How Often Each
+
+```ruby
+cms = ProbabilisticMachines::CountMinSketch.new(width: 4096, depth: 4)
+cms.add('/login', count: 3)
+cms.count('/login')  # never below the truth
+cms.total            # every count added
+cms.merge(other)     # exact sum
+```
+
+A count is over the truth by at most `e * total / width`, with probability
+`1 - e ** -depth` of staying inside it. Width buys accuracy, depth buys
+confidence.
+
+---
+
+## 📏 Quantiles
+
+```ruby
+td = ProbabilisticMachines::TDigest.new(compression: 100)
+td.add(12.5)
+td.add(12.5, weight: 1_000.0) # one point standing for a thousand observations
+td.quantile(0.99)
+td.cdf(12.5)
+td.centroids                  # Array of frozen Centroid(mean:, weight:)
+td.minimum                    # exact, as is maximum
+td.merge(other)
+```
+
+A quantile is off by about `1 / compression` of the distribution in the middle
+and is exact at the extremes, in `compression * ln(n)` clusters rather than `n`
+samples. An empty digest answers `Float::NAN`.
+
+---
+
+## 🛰️ Have I Seen This Contact Before
+
+```ruby
+cf = ProbabilisticMachines::CuckooFilter.new(buckets: 4096)
+cf.insert('device-9')      # raises ProbabilisticMachines::Full when wedged
+cf.include?('device-9')    # contains? is the same question
+cf.remove('device-9')      # unlike a Bloom filter, it can forget
+```
+
+`buckets * 4` items in `buckets * 4` bytes. Never a false negative; about 3%
+false positives. Only remove items you inserted, or you take a colliding item's
+fingerprint with it. A cuckoo filter cannot merge, so there is no `merge` here.
+
+---
+
+## 👑 Who Dominates the Stream
+
+```ruby
+hh = ProbabilisticMachines::HeavyHitters.new(k: 64)
+hh.add('10.0.0.1')
+hh.add('10.0.0.1', count: 5)
+hh.top(10)   # [Hitter(key:, count:, overestimate:), ...] most frequent first
+hh.ranked    # every tracked key
+hh.observed  # observations fed in
+hh.merge(other)
+```
+
+After `n` observations every key seen more than `n / k` times is tracked, in `k`
+counters and no heap per key. A counter can inherit the count of the key it
+evicted, or what a merged-in table's own evictions may have hidden, so the true
+count lies between `count - overestimate` and `count`.
+
+Keys are UTF-8 strings here: Space-Saving needs `Copy + Eq` keys in Rust, so the
+table counts 64-bit key hashes and the extension keeps the string each tracked
+hash came from, to hand back from `top` and `ranked`. The other four classes
+take any byte string.
+
+---
+
+## 📦 Transmitting a Reading
+
+`dump` returns the crate's own byte frame, `load` reads one back. The frame
+names its own shape, so `load` needs no keyword and rejects a blob written with
+a size this build does not carry. It also reads the frame's fields against each
+other and refuses a state no stream could have left behind, so an edited blob
+raises instead of handing back a sketch that counts more than it ever saw.
+
+```ruby
+blob = hll.dump                                 # binary String
+ProbabilisticMachines::HyperLogLog.load(blob)
+ProbabilisticMachines::TDigest.load(blob)       # ArgumentError: codec: wrong structure tag
+```
+
+Anything that reads the crate reads these frames, so an embedded device can ship
+a sketch a Rails app folds in. `HeavyHitters` is the one blob that is not purely
+the crate's: the crate's frame counts 64-bit key hashes, so the extension appends
+its key table after it, one record of a 64-bit byte length and that many UTF-8
+bytes per tracked key. `load` rehashes each name, and rejects a blob whose key
+table is malformed or does not name every key the frame tracks.
+
+```ruby
+restored = ProbabilisticMachines::HeavyHitters.load(hh.dump)
+restored.top(1).first.key # => '10.0.0.1', the string, not its hash
+```
+
+---
+
+## ⚙️ The Rust Core
+
+The sketches themselves are the `probabilistic-machines` crate under
+`ext/probabilistic_machines_native/core`: `no_std`, zero dependencies, allocating
+only for `TDigest` and for the `encoded` frames this extension dumps, and
+verified on bare metal. The binding crate beside it under
+`ffi/` is the Ruby surface and is not published.
+
+---
+
+## 📡 Credits
+
+**AMS Flajolet**, NCC-0185, *Flajolet*-class tender.
+
+Named for Philippe Flajolet and G. Nigel Martin's *Probabilistic Counting
+Algorithms for Data Base Applications* (1985), and for Flajolet, Fusy, Gandouet
+and Meunier's *HyperLogLog* (2007), which established that you can count what
+you refuse to store.
+
+---
+
+## License
+
+MIT.

data/ext/probabilistic_machines_native/extconf.rb

@@ -0,0 +1,3 @@
+# frozen_string_literal: true
+
+require_relative 'ffi/extconf'

data/lib/probabilistic_machines/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module ProbabilisticMachines
+  VERSION = '0.1.1'
+end

data/lib/probabilistic_machines.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+require_relative 'probabilistic_machines/version'
+
+# Every class, the two `Data` result objects and both error classes are defined
+# by the native extension; there is no pure-Ruby implementation to fall back to.
+require 'probabilistic_machines_native/probabilistic_machines_native'

data/sig/probabilistic_machines.rbs

@@ -0,0 +1,182 @@
+# Sketches that answer a counting question in memory fixed at construction
+# time. Every class here is defined by the native extension.
+module ProbabilisticMachines
+  VERSION: String
+
+  class Error < StandardError
+  end
+
+  # Raised when a cuckoo filter has nowhere left to put a fingerprint.
+  class Full < Error
+  end
+
+  # A tracked key and its counter. The true count lies in
+  # `count - overestimate` through `count`.
+  class Hitter < ::Data
+    attr_reader key: String
+    attr_reader count: Integer
+    attr_reader overestimate: Integer
+  end
+
+  # One cluster of a digest: the mean of the values that collapsed into it.
+  class Centroid < ::Data
+    attr_reader mean: Float
+    attr_reader weight: Float
+  end
+
+  # Distinct counts in a fixed register array.
+  class HyperLogLog
+    # `registers` is 1024, 4096, 16384 or 65536.
+    def self.new: (registers: Integer) -> instance
+
+    def self.load: (String blob) -> instance
+
+    def registers: () -> Integer
+    def add: (String item) -> nil
+
+    # For streams that ship 64-bit hashes instead of keys.
+    def add_hash: (Integer hash) -> nil
+
+    def estimate: () -> Float
+
+    # Relative standard error of the estimate, `1.04 / sqrt(registers)`.
+    def standard_error: () -> Float
+
+    def empty?: () -> bool
+
+    # Unions `other` in. Raises `ArgumentError` on a different register count.
+    def merge: (HyperLogLog other) -> self
+
+    def dump: () -> String
+    def inspect: () -> String
+  end
+
+  # How often each item was seen, in `width * depth` counters.
+  class CountMinSketch
+    # `width` is 1024 or 4096; `depth` is 4 or 8.
+    def self.new: (width: Integer, depth: Integer) -> instance
+
+    def self.load: (String blob) -> instance
+
+    def width: () -> Integer
+    def depth: () -> Integer
+    def add: (String item, ?count: Integer) -> nil
+
+    # Never below the true count, and over it by at most
+    # `e * total / width` with probability `1 - e ** -depth`.
+    def count: (String item) -> Integer
+
+    # Sum of every count added.
+    def total: () -> Integer
+    def saturated?: () -> bool
+
+    # Sums `other` in. Raises `ArgumentError` on a different shape.
+    def merge: (CountMinSketch other) -> self
+
+    def dump: () -> String
+    def inspect: () -> String
+  end
+
+  # Quantiles of a stream, in clusters instead of samples.
+  class TDigest
+    def self.new: (compression: Integer) -> instance
+
+    def self.load: (String blob) -> instance
+
+    def compression: () -> Integer
+    def add: (Float value, ?weight: Float) -> nil
+
+    # Value at quantile `q`, `Float::NAN` when empty.
+    def quantile: (Float q) -> Float
+
+    # Fraction of the stream at or below `value`, `Float::NAN` when empty.
+    def cdf: (Float value) -> Float
+
+    # Total weight added.
+    def count: () -> Float
+
+    # Smallest value added, `Float::INFINITY` when empty.
+    def minimum: () -> Float
+
+    # Largest value added, `-Float::INFINITY` when empty.
+    def maximum: () -> Float
+
+    # Clusters currently held.
+    def size: () -> Integer
+
+    def empty?: () -> bool
+    def centroids: () -> Array[Centroid]
+    def merge: (TDigest other) -> self
+    def dump: () -> String
+    def inspect: () -> String
+  end
+
+  # Membership that can forget, in `buckets * 4` fingerprints.
+  class CuckooFilter
+    # `buckets` is 1024, 4096, 16384 or 65536.
+    def self.new: (buckets: Integer) -> instance
+
+    def self.load: (String blob) -> instance
+
+    def buckets: () -> Integer
+
+    # Fingerprint slots in the table, `buckets * 4`.
+    def capacity: () -> Integer
+
+    # Raises `Full` when the item has nowhere to go.
+    def insert: (String item) -> bool
+
+    # Never false for an item that was inserted; true for one that was not
+    # with probability about 3%.
+    def include?: (String item) -> bool
+
+    alias contains? include?
+
+    # Only for items that were inserted: removing an item that merely collides
+    # with one takes the other item's fingerprint with it.
+    def remove: (String item) -> bool
+
+    def size: () -> Integer
+    def empty?: () -> bool
+    def dump: () -> String
+    def inspect: () -> String
+  end
+
+  # Which keys dominate a stream, in `k` counters instead of one per key.
+  class HeavyHitters
+    # `k` is 16, 64 or 256.
+    def self.new: (k: Integer) -> instance
+
+    def self.load: (String blob) -> instance
+
+    def k: () -> Integer
+
+    # Keys are UTF-8 strings, so `top` and `ranked` can hand them back.
+    def add: (String key, ?count: Integer) -> nil
+
+    # Upper bound on how often `key` occurred, 0 when it is not tracked.
+    def count: (String key) -> Integer
+
+    # Observations fed in.
+    def observed: () -> Integer
+
+    # Keys currently tracked, at most `k`.
+    def size: () -> Integer
+
+    def empty?: () -> bool
+
+    # The `n` most frequent tracked keys, most frequent first.
+    def top: (Integer n) -> Array[Hitter]
+
+    # Every tracked key, most frequent first.
+    def ranked: () -> Array[Hitter]
+
+    # Folds `other` in. Raises `ArgumentError` on a different `k`.
+    def merge: (HeavyHitters other) -> self
+
+    # The crate's frame followed by the key table that carries the strings.
+    def dump: () -> String
+
+    def inspect: () -> String
+  end
+end

metadata

@@ -0,0 +1,63 @@
+--- !ruby/object:Gem::Specification
+name: probabilistic_machines
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: aarch64-linux
+authors:
+- Abdelkader Boudih
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2026-11-01 00:00:00.000000000 Z
+dependencies: []
+description: HyperLogLog, Count-Min Sketch, t-digest, cuckoo filter and Space-Saving
+  top-k, bound to Ruby from the no_std probabilistic-machines crate. Each sketch answers
+  a counting question in memory fixed at construction time, and serialises to a byte
+  string other languages reading the same crate can decode.
+email:
+- oss@seuros.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- ext/probabilistic_machines_native/extconf.rb
+- lib/probabilistic_machines.rb
+- lib/probabilistic_machines/version.rb
+- lib/probabilistic_machines_native/probabilistic_machines_native.so
+- sig/probabilistic_machines.rbs
+homepage: https://github.com/seuros/probabilistic_machines
+licenses:
+- MIT
+metadata:
+  source_code_uri: https://github.com/seuros/probabilistic_machines
+  changelog_uri: https://github.com/seuros/probabilistic_machines/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/seuros/probabilistic_machines/issues
+  rubygems_mfa_required: 'true'
+  cargo_crate_name: probabilistic_machines_native
+  cargo_manifest_path: ext/probabilistic_machines_native/ffi/Cargo.toml
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '4.0'
+  - - "<"
+    - !ruby/object:Gem::Version
+      version: 4.1.dev
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.23
+signing_key: 
+specification_version: 4
+summary: Distinct counts, quantiles and frequencies in bounded memory
+test_files: []