libdictenstein 4.0.0.rc.4

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 010cb3470993511a75d84982f225e366d1d9d31232b858539758a675d51132ba
+  data.tar.gz: 53a12f44a10d6fb1059e8322e5608f48d2a28f8bf76b2f9bc8889e4433b56656
+SHA512:
+  metadata.gz: 98bccac45121c8b16fbcefb357b0f9f95c67c2873c35f8e67cb66fa96ad1e5c99108598a3435c8bea008dfce8133765a7005c5a6513bd7474c648e8fbc148850
+  data.tar.gz: ff10f78c20b60cf9a150aa697e1072c34a6e9c20d5a92fc2af493fc517b93d856e2e8615ea8b3b59a39c1a766a3fe15912c7eb02d23a63d322d4af10861d05bd

data/README.md

@@ -0,0 +1,190 @@
+# Vinary Tree libdictenstein for Ruby
+
+The gem exposes full DynamicDAWG CRUD, immutable DoubleArrayTrie construction,
+SCDAWG substring search, persistent ARTrie CRUD/checkpoint/reopen, and persistent
+vocabulary reverse lookup. Every object implements `with_resource`, allowing an
+independently packaged liblevenshtein transducer to retain it in O(1).
+
+Calls acquire only a short lifetime lease; operations on the same dictionary
+are not serialized. The project-owned native resource advertises parallel and
+reentrant access automatically.
+
+Every dictionary is an `Enumerable` over immutable-revision `Entry` records:
+
+```ruby
+dictionary = VinaryTree::Libdictenstein::DynamicDawg.new
+dictionary.put("cat", 0)
+dictionary.put("cut", nil)
+
+dictionary.each do |entry|
+  p [entry.key, entry.value]
+  break if entry.key == "cat" # ensure closes the native cursor
+end
+
+keys = dictionary.keys
+values = dictionary.values
+snapshot = dictionary.entries
+```
+
+`each` returns an `Enumerator` without a block. `entry_stream` exposes manual
+`next`, `cancel`, and `close` for pull-driven bounded traversal.
+
+<!-- BEGIN GENERATED BINDING OPERATIONS; DO NOT EDIT -->
+
+## Support and package contract
+
+| Property | Contract |
+|---|---|
+| Binding | Ruby |
+| Languages/runtime | Ruby 3.3+ |
+| Support tier | Tier 2 |
+| Distribution | RubyGems `libdictenstein` |
+| Native boundary | Fiddle over the stable C ABI |
+| Canonical facade source | [`bindings/ruby/lib/vinary_tree/libdictenstein`](../../bindings/ruby/lib/vinary_tree/libdictenstein) |
+
+All tiers implement the same ownership, snapshot, status, and compatibility
+laws. The tier controls release gating rather than semantic quality. Start with
+the [producer documentation hub](../../docs/bindings/README.md), then use the
+[`ldict_*` C ABI reference](../../docs/bindings/c-abi-reference.md) for exact
+preconditions, statuses, thread-safety, and complexity.
+
+![A host facade owns a project handle while exported snapshots cross projects only as retained versioned resources.](../../docs/diagrams/abi-producer-component.svg)
+
+## Installation and native loading
+
+Install the distribution named above and its exact `vinary-tree-interop`
+dependency. Published managed packages carry or resolve supported native
+artifacts; source builds use the release library from `target/release` or the
+installed CMake/pkg-config package. Diagnose loading in this order: toolchain
+version, OS/CPU artifact, dependent package pin, loader path, then ABI/API
+handshake. Never silently load an arbitrary same-named system library.
+
+## Executable example and verification
+
+The canonical checked example is [`bindings/ruby/test/test_conformance.rb`](../../bindings/ruby/test/test_conformance.rb). CI runs
+the public package path with:
+
+```sh
+ruby -Ibindings/ruby/lib bindings/ruby/test/test_conformance.rb
+```
+
+The example is also conformance evidence: it uses public constructors, checks
+membership/value behavior, exports a retained resource, and closes every owned
+handle. Cross-project suites pass that resource to liblevenshtein without
+serialization.
+
+## Public API, backends, and data domains
+
+| Concept | Semantics |
+|---|---|
+| Dictionary handle | Owns one mutable or immutable backend instance and exposes kind/capability introspection. |
+| CRUD and batch mutation | Text and `u64` operations preserve optional values; empty-batch and partial-failure behavior follows the C reference. |
+| Persistent maintenance | `checkpoint`, `compact`, and `clear` are capability-gated and report unsupported operations explicitly. |
+| Retained resource | `resource()` lends `vt.dictionary.v1`; a consumer retains and snapshots it independently. |
+| Snapshot | Immutable revision with stable node identifiers, exact domains, bounded edge pages, and optional mapped values. |
+
+Dynamic DAWG supports mutable finite-term dictionaries; double-array tries are
+read-optimized static structures; SCDAWG indexes substrings; persistent ARTrie
+and vocabulary stores provide durable byte/Unicode/`u64` domains. Select by
+reported kind and capabilities rather than assuming every operation exists.
+
+Text APIs validate UTF-8 and traverse Unicode scalar values. Byte APIs retain
+arbitrary octets. Token APIs preserve the full `u64` range. Optional dictionary
+values are represented separately from terminal membership, so `None` is not a
+sentinel and empty terms remain valid when supported.
+
+## Native collection surface
+
+Every dictionary includes `Enumerable`. `each` returns an `Enumerator`
+without a block and yields host-owned `Entry` records in lexical order; its
+`ensure` path closes the cursor after exhaustion, `break`, or exception.
+`entry_stream` also exposes explicit `next`, `cancel`, and `close`, while
+`entries`, `keys`, and `values` provide materialized snapshot idioms. Binary
+strings, UTF-8 strings, and `Array<Integer>` preserve the three unit domains;
+`nil` remains distinct from every mapped integer.
+
+The gem executable keeps construction and warmup outside the timed drain and
+prints one JSON record. Run its materialized, streaming, and early-cancel arms
+over the shared deterministic corpus with:
+
+```sh
+ruby bindings/ruby/bin/libdictenstein-collection-profile --arm materialized --entries 4096
+ruby bindings/ruby/bin/libdictenstein-collection-profile --arm stream --entries 65536 --batch-size 256
+ruby bindings/ruby/bin/libdictenstein-collection-profile --arm stream-cancel --entries 65536 --batch-size 64 --early-cancel 64
+```
+
+The pure Rust producer is the semantic and performance baseline: generic
+snapshot traversal, borrowed and snapshot-owning `IntoIterator`, optimized bulk
+`FromIterator`/`Extend` where infallible, named fallible variants for persistent
+stores, deterministic order, and reusable fold/visitor paths. Read the local
+[Rust API audit](../../docs/bindings/rust-api-idioms.md) and the family
+[collection-protocol design](https://github.com/vinary-tree/liblevenshtein-rust/blob/master/docs/bindings/collection-protocols.md).
+
+Public spelling must remain native to this ecosystem. The shared engine
+standardizes laws and batching; it does not expose C handles, vtables, leases,
+or status codes to ordinary application code. Documentation may mark a protocol
+as shipped only after its language conformance and performance gates pass.
+
+## Ownership, snapshots, and resource handoff
+
+Prefer block forms or `ensure { dictionary.close }`; close persistent stores explicitly.
+
+An exported resource is borrowed until its base-vtable `retain` succeeds. A
+captured snapshot arrives owning one retain and may outlive the mutable
+dictionary. Later inserts, updates, removals, compaction, or checkpoints do not
+alter a pinned revision. Every successful retain has exactly one release, and
+failed construction transfers no ownership.
+
+## Errors and failure containment
+
+Failures become typed Ruby exceptions with status and diagnostic. Branch on the typed status or exception, not diagnostic text.
+Invalid UTF-8, domain mismatch, unsupported capability, closed handle, bad
+path, allocation failure, provider fault, I/O failure, and contained panic are
+distinct cases. Copy thread-local diagnostics before another native call.
+
+## Concurrency and reentrancy
+
+Independent handles and immutable snapshots are reentrant. Mutations follow
+the backend's advertised synchronization strategy; one host wrapper must not
+invent a stronger promise. Snapshot capture is a linearization point and never
+permits a torn root/count pair. Do not race close against another operation on
+the same handle, and do not retain callback/paging buffers after return.
+
+## Performance, durability, and marshalling
+
+- Use bulk construction for an initially empty dictionary and presorted input
+  when available; unordered construction uses the optimized sort-plus-minimal
+  path.
+- Batch mutations to amortize foreign-boundary crossings.
+- Export retained resources instead of serializing dictionaries between
+  Vinary packages.
+- Keep byte, Unicode, and `u64` domains explicit to avoid transcoding.
+- Treat `checkpoint` as durability, `compact` as representation maintenance,
+  and `close` as ownership release; they are not interchangeable.
+
+## Security model
+
+Treat paths, terms, values, page offsets, serialized files, and foreign callers
+as untrusted. Validate lengths before allocation, contain panics at the ABI,
+bound diagnostics and batches, prevent path traversal, and reject unknown enum
+values. See the [FFI boundary analysis](../../docs/security/ffi-boundary.md) and
+[family security model](https://github.com/vinary-tree/vinary-tree-interop/blob/master/docs/security-model.md).
+
+## Compatibility and troubleshooting
+
+The project ABI, project API revision, family ABI, interface version, package
+version, and persistent format version are separate counters. Negotiate each
+at its documented boundary. For unexpected behavior, record dictionary kind,
+capabilities, unit domain, persistence path, exact status, and copied diagnostic
+before reducing the operation sequence.
+
+## Maintainer checklist
+
+1. Update `bindings/api.json` before changing a public facade or package pin.
+2. Regenerate headers/constants and run the binding contract gate.
+3. Extend the executable, negative-path, leak, and cross-project tests.
+4. Update this guide when ownership, errors, capabilities, or platforms change.
+5. Render PlantUML headlessly and run math/link/documentation checks.
+6. Verify staged registry artifacts contain the guide and coherent pins.
+
+<!-- END GENERATED BINDING OPERATIONS -->

data/bin/libdictenstein-collection-profile

@@ -0,0 +1,157 @@
+#!/usr/bin/env ruby
+
+# Measures the public Ruby collection facade over a deterministic dictionary
+# revision. Corpus construction and warmup are outside the reported interval.
+
+$LOAD_PATH.unshift(File.expand_path("../lib", __dir__))
+
+require "json"
+require "optparse"
+require "vinary_tree/libdictenstein"
+
+LD = VinaryTree::Libdictenstein
+
+DEFAULT_ENTRIES = 65_536
+DEFAULT_BATCH_SIZE = 256
+DEFAULT_EARLY_CANCEL = 64
+KEY_UNITS = 38
+U64_MASK = (1 << 64) - 1
+ARMS = %w[materialized stream stream-cancel].freeze
+
+def parse_arguments(arguments)
+  config = {
+    arm: nil,
+    entries: DEFAULT_ENTRIES,
+    passes: 1,
+    warmup_passes: 1,
+    batch_size: DEFAULT_BATCH_SIZE,
+    early_cancel: DEFAULT_EARLY_CANCEL
+  }
+  parser = OptionParser.new do |options|
+    options.on("--arm ARM") { |value| config[:arm] = value }
+    options.on("--entries N", Integer) { |value| config[:entries] = value }
+    options.on("--passes N", Integer) { |value| config[:passes] = value }
+    options.on("--warmup-passes N", Integer) { |value| config[:warmup_passes] = value }
+    options.on("--batch-size N", Integer) { |value| config[:batch_size] = value }
+    options.on("--early-cancel N", Integer) { |value| config[:early_cancel] = value }
+  end
+  parser.parse!(arguments)
+  raise OptionParser::InvalidArgument, "unexpected positional arguments" unless arguments.empty?
+  raise OptionParser::InvalidArgument, "--arm must be #{ARMS.join(', ')}" unless ARMS.include?(config[:arm])
+  %i[entries passes batch_size early_cancel].each do |name|
+    raise OptionParser::InvalidArgument, "--#{name.to_s.tr('_', '-')} must be positive" unless config[name].positive?
+  end
+  raise OptionParser::InvalidArgument, "--warmup-passes must be nonnegative" if config[:warmup_passes].negative?
+  config
+end
+
+def make_corpus(size)
+  Array.new(size) do |index|
+    [format("collection/%04x/%08x/shared-suffix", index & 0x0fff, index).b, index]
+  end
+end
+
+def expected_checksum(corpus, limit)
+  corpus.sort_by(&:first).first(limit).sum { |key, value| key.bytesize ^ value } & U64_MASK
+end
+
+def entry_checksum(entry)
+  raise "benchmark expected a byte-domain entry" unless entry.domain == LD::BYTE
+  (entry.key.bytesize ^ (entry.value || 0)) & U64_MASK
+end
+
+def drain_materialized(dictionary)
+  entries = dictionary.entries
+  [entries.sum { |entry| entry_checksum(entry) } & U64_MASK, entries.length]
+end
+
+def drain_stream(dictionary, batch_size:, limit:, cancel:)
+  stream = dictionary.entry_stream(
+    max_entries: batch_size,
+    max_units: batch_size * KEY_UNITS,
+    max_values: batch_size
+  )
+  checksum = 0
+  processed = 0
+  begin
+    while processed < limit && (entry = stream.next)
+      checksum = (checksum + entry_checksum(entry)) & U64_MASK
+      processed += 1
+    end
+    if cancel
+      stream.cancel
+    elsif processed != limit || !stream.next.nil?
+      raise "stream cardinality differs from the generated corpus"
+    end
+    stream.close
+  ensure
+    stream.close
+  end
+  [checksum, processed]
+end
+
+def drain(dictionary, config)
+  case config[:arm]
+  when "materialized"
+    drain_materialized(dictionary)
+  when "stream"
+    drain_stream(dictionary, batch_size: config[:batch_size], limit: config[:entries], cancel: false)
+  when "stream-cancel"
+    drain_stream(
+      dictionary,
+      batch_size: config[:batch_size],
+      limit: [config[:entries], config[:early_cancel]].min,
+      cancel: true
+    )
+  end
+end
+
+def execute(arguments)
+  config = parse_arguments(arguments)
+  corpus = make_corpus(config[:entries])
+  dictionary = LD::DynamicDawg.new(domain: LD::BYTE)
+  begin
+    inserted = dictionary.put_all(corpus)
+    raise "inserted #{inserted} of #{corpus.length} generated entries" unless inserted == corpus.length
+
+    consumed = config[:arm] == "stream-cancel" ? [config[:entries], config[:early_cancel]].min : config[:entries]
+    expected = expected_checksum(corpus, consumed)
+    config[:warmup_passes].times do
+      checksum, count = drain(dictionary, config)
+      raise "warmup checksum or cardinality mismatch" unless count == consumed && checksum == expected
+    end
+
+    started = Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond)
+    checksum = 0
+    config[:passes].times do
+      pass_checksum, count = drain(dictionary, config)
+      raise "timed checksum or cardinality mismatch" unless count == consumed && pass_checksum == expected
+      checksum = (checksum + pass_checksum) & U64_MASK
+    end
+    elapsed_ns = Process.clock_gettime(Process::CLOCK_MONOTONIC, :nanosecond) - started
+    raise "aggregate checksum mismatch" unless checksum == (expected * config[:passes]) & U64_MASK
+
+    puts JSON.generate(
+      schema: "libdictenstein.host-collection-traversal.v1",
+      runtime: "ruby",
+      arm: config[:arm],
+      dictionary_entries: config[:entries],
+      consumed_entries_per_pass: consumed,
+      passes: config[:passes],
+      warmup_passes: config[:warmup_passes],
+      batch_size: config[:arm] == "materialized" ? nil : config[:batch_size],
+      early_cancel: config[:arm] == "stream-cancel" ? config[:early_cancel] : nil,
+      elapsed_ns: elapsed_ns,
+      checksum: checksum
+    )
+  ensure
+    dictionary.close
+  end
+end
+
+begin
+  execute(ARGV)
+rescue StandardError => error
+  warn error.message
+  exit 2
+end

data/lib/vinary_tree/libdictenstein/native.rb

@@ -0,0 +1,102 @@
+require "fiddle/import"
+require "rbconfig"
+
+module VinaryTree
+  module Libdictenstein
+    module Native
+      extend Fiddle::Importer
+
+      def self.candidates
+        explicit = ENV["LIBDICTENSTEIN_LIBRARY"]
+        os, filename = case RbConfig::CONFIG["host_os"]
+                       when /darwin/ then ["darwin", "liblibdictenstein.dylib"]
+                       when /mswin|mingw/ then ["windows", "libdictenstein.dll"]
+                       else ["linux", "liblibdictenstein.so"]
+                       end
+        cpu = case RbConfig::CONFIG["host_cpu"]
+              when /x86_64|amd64/ then "x64"
+              when /aarch64|arm64/ then "arm64"
+              else RbConfig::CONFIG["host_cpu"]
+              end
+        platform = "#{os}-#{cpu}"
+        [explicit, File.expand_path("native/#{platform}/#{filename}", __dir__), filename].compact
+      end
+      loaded = candidates.find do |candidate|
+        begin dlload candidate; true
+        rescue Fiddle::DLError then false
+        end
+      end
+      raise Fiddle::DLError, "unable to load libdictenstein (tried #{candidates.join(', ')})" unless loaded
+
+      Resource = struct ["void* context", "void* vtable"]
+      TextEntry = struct [
+        "void* data", "size_t len", "uint64_t value", "uint8_t has_value",
+        "uint8_t reserved[7]"
+      ]
+      DictionaryEntry = struct [
+        "size_t unit_offset", "size_t unit_len", "size_t value_offset",
+        "size_t value_len", "uint64_t reserved"
+      ]
+      EntryBatchLimits = struct [
+        "size_t max_entries", "size_t max_units", "size_t max_values",
+        "uint64_t reserved"
+      ]
+      EntryBatch = struct [
+        "void* entries", "size_t entry_count", "void* units",
+        "size_t unit_count", "void* values", "size_t value_count",
+        "uint64_t generation", "uint64_t reserved"
+      ]
+      EntriesInfo = struct [
+        "uint32_t unit_domain", "uint32_t value_domain", "uint32_t order",
+        "uint32_t reserved0", "uint64_t flags", "size_t exact_len",
+        "uint64_t identity_producer", "uint64_t identity_revision",
+        "uint64_t reserved[2]"
+      ]
+
+      extern "uint32_t ldict_abi_version(void)"
+      extern "uint32_t ldict_api_revision(void)"
+      extern "const char* ldict_last_error_message(void)"
+      extern "uint32_t ldict_dynamic_dawg_new(uint32_t, void*)"
+      extern "uint32_t ldict_double_array_trie_new(uint32_t, void*, size_t, void*)"
+      extern "uint32_t ldict_scdawg_new(uint32_t, void*)"
+      extern "uint32_t ldict_persistent_artrie_create(uint32_t, const void*, size_t, void*)"
+      extern "uint32_t ldict_persistent_artrie_open(uint32_t, const void*, size_t, void*)"
+      extern "uint32_t ldict_persistent_vocab_create(const void*, size_t, void*)"
+      extern "uint32_t ldict_persistent_vocab_open(const void*, size_t, void*)"
+      extern "void ldict_dictionary_free(void*)"
+      extern "uint32_t ldict_dictionary_resource(void*, void*)"
+      extern "uint32_t ldict_dictionary_entries_open(void*, void*, void*)"
+      extern "uint32_t ldict_entry_cursor_next(void*, void*, void*)"
+      extern "uint32_t ldict_entry_cursor_release(void*, uint64_t)"
+      extern "uint32_t ldict_entry_cursor_cancel(void*)"
+      extern "uint32_t ldict_entry_cursor_free(void*)"
+      extern "uint32_t ldict_dictionary_kind(void*, void*)"
+      extern "uint32_t ldict_dictionary_capabilities(void*, void*)"
+      extern "uint32_t ldict_dictionary_len(void*, void*)"
+      extern "uint32_t ldict_dictionary_insert_text_value(void*, const void*, size_t, uint64_t, uint8_t, void*)"
+      extern "uint32_t ldict_dictionary_remove_text(void*, const void*, size_t, void*)"
+      extern "uint32_t ldict_dictionary_contains_text(void*, const void*, size_t, void*)"
+      extern "uint32_t ldict_dictionary_get_text_value(void*, const void*, size_t, void*, void*, void*)"
+      extern "uint32_t ldict_dictionary_insert_u64_value(void*, const void*, size_t, uint64_t, uint8_t, void*)"
+      extern "uint32_t ldict_dictionary_remove_u64(void*, const void*, size_t, void*)"
+      extern "uint32_t ldict_dictionary_contains_u64(void*, const void*, size_t, void*)"
+      extern "uint32_t ldict_dictionary_get_u64_value(void*, const void*, size_t, void*, void*, void*)"
+      extern "uint32_t ldict_dictionary_insert_text_batch(void*, void*, size_t, void*)"
+      extern "uint32_t ldict_dictionary_clear(void*)"
+      extern "uint32_t ldict_dictionary_compact(void*, void*)"
+      extern "uint32_t ldict_dictionary_checkpoint(void*)"
+      extern "uint32_t ldict_scdawg_contains_substring(void*, const void*, size_t, void*)"
+      extern "uint32_t ldict_scdawg_substring_frequency(void*, const void*, size_t, void*)"
+      extern "uint32_t ldict_vocab_get_term(void*, uint64_t, void*, size_t, void*, void*)"
+
+      module_function
+      def pointer_output = Fiddle::Pointer.malloc(Fiddle::SIZEOF_VOIDP, Fiddle::RUBY_FREE)
+      def read_pointer(output) = output[0, Fiddle::SIZEOF_VOIDP].unpack1("J")
+      def size_output = Fiddle::Pointer.malloc(Fiddle::SIZEOF_SIZE_T, Fiddle::RUBY_FREE)
+      def read_size(output) = output[0, Fiddle::SIZEOF_SIZE_T].unpack1("J")
+      def u64_output = Fiddle::Pointer.malloc(8, Fiddle::RUBY_FREE)
+      def read_u64(output) = output[0, 8].unpack1("Q")
+      def byte_output = Fiddle::Pointer.malloc(1, Fiddle::RUBY_FREE)
+    end
+  end
+end

data/lib/vinary_tree/libdictenstein/version.rb

@@ -0,0 +1,5 @@
+module VinaryTree
+  module Libdictenstein
+    VERSION = "4.0.0.rc.4"
+  end
+end

data/lib/vinary_tree/libdictenstein.rb

@@ -0,0 +1,475 @@
+require "thread"
+require_relative "libdictenstein/version"
+require_relative "libdictenstein/native"
+
+module VinaryTree
+  module Libdictenstein
+    BYTE = 1
+    UNICODE_SCALAR = 2
+    U64 = 3
+
+    # Native ABI version (LDICT_ABI_VERSION); always 1 for this family.
+    def self.abi_version = Native.ldict_abi_version
+
+    # Compatible-additions revision within the ABI version (LDICT_API_REVISION).
+    def self.api_revision = Native.ldict_api_revision
+
+    Lookup = Data.define(:found?, :value)
+    Entry = Data.define(:key, :value, :domain)
+    EntryInfo = Data.define(:domain, :exact_length, :snapshot_identity)
+
+    class Error < StandardError
+      attr_reader :status
+      def initialize(status)
+        @status = status
+        super("libdictenstein status #{status}: #{Native.ldict_last_error_message.to_s}")
+      end
+    end
+    def self.check(status) = (raise Error, status unless status.zero?)
+
+    class ConcurrentHandle
+      def initialize(pointer)
+        @pointer = pointer
+        @active = 0
+        @closing = false
+        @mutex = Mutex.new
+        @condition = ConditionVariable.new
+      end
+
+      def with_pointer
+        pointer = @mutex.synchronize do
+          raise IOError, "dictionary is closed" if @closing || @pointer.zero?
+          @active += 1
+          @pointer
+        end
+        yield pointer
+      ensure
+        @mutex.synchronize do
+          @active -= 1
+          @condition.broadcast if @active.zero?
+        end if pointer
+      end
+
+      def close
+        pointer = @mutex.synchronize do
+          return 0 if @pointer.zero?
+          @closing = true
+          @condition.wait(@mutex) until @active.zero?
+          result = @pointer
+          @pointer = 0
+          result
+        end
+        Native.ldict_dictionary_free(pointer) unless pointer.zero?
+        pointer
+      end
+    end
+
+    # Shared low-level adapter over one opaque native entry cursor. Every key
+    # is copied before a leased batch is released.
+    class EntryCursorState
+      attr_reader :info
+
+      def initialize(handle, max_entries:, max_units:, max_values:)
+        raise ArgumentError, "max_entries must be positive" unless max_entries.positive?
+        raise ArgumentError, "entry batch limits must be nonnegative" if max_units.negative? || max_values.negative?
+
+        @mutex = Mutex.new
+        @cursor = 0
+        @leased = false
+        @ended = false
+        @index = 0
+        @limits_memory = Fiddle::Pointer.malloc(Native::EntryBatchLimits.size, Fiddle::RUBY_FREE)
+        @limits = Native::EntryBatchLimits.new(@limits_memory)
+        @limits.max_entries = max_entries
+        @limits.max_units = max_units
+        @limits.max_values = max_values
+        @limits.reserved = 0
+        @batch_memory = Fiddle::Pointer.malloc(Native::EntryBatch.size, Fiddle::RUBY_FREE)
+        @batch = Native::EntryBatch.new(@batch_memory)
+        info_memory = Fiddle::Pointer.malloc(Native::EntriesInfo.size, Fiddle::RUBY_FREE)
+        native_info = Native::EntriesInfo.new(info_memory)
+        cursor_output = Native.pointer_output
+        handle.with_pointer do |dictionary|
+          Libdictenstein.check(
+            Native.ldict_dictionary_entries_open(dictionary, cursor_output, info_memory)
+          )
+        end
+        @cursor = Native.read_pointer(cursor_output)
+        flags = native_info.flags
+        @info = EntryInfo.new(
+          native_info.unit_domain,
+          flags.anybits?(1) ? native_info.exact_len : nil,
+          flags.anybits?(2) ? [native_info.identity_producer, native_info.identity_revision].freeze : nil
+        )
+      end
+
+      def next_entry
+        @mutex.synchronize do
+          return nil if @cursor.zero? || @ended
+          begin
+            unless @leased
+              status = Native.ldict_entry_cursor_next(@cursor, @limits_memory, @batch_memory)
+              if status == 1
+                @ended = true
+                close_locked(cancel: false)
+                return nil
+              end
+              Libdictenstein.check(status)
+              @leased = true
+              @index = 0
+            end
+            result = copy_entry(@index)
+            @index += 1
+            release_locked if @index == @batch.entry_count
+            result
+          rescue Exception
+            close_locked(cancel: true) rescue nil
+            raise
+          end
+        end
+      end
+
+      def cancel
+        @mutex.synchronize do
+          return nil if @cursor.zero? || @ended
+          first_error = native_error(Native.ldict_entry_cursor_cancel(@cursor))
+          begin
+            release_locked
+          rescue Exception => error
+            first_error ||= error
+          end
+          @ended = true
+          raise first_error if first_error
+        end
+        nil
+      end
+
+      def close
+        @mutex.synchronize { close_locked(cancel: true) }
+      end
+
+      def closed?
+        @mutex.synchronize { @cursor.zero? }
+      end
+
+      private
+
+      def address(pointer)
+        return 0 if pointer.nil?
+        pointer.respond_to?(:to_i) ? pointer.to_i : Integer(pointer)
+      end
+
+      def checked_range(offset, length, total, name)
+        unless offset >= 0 && length >= 0 && offset <= total && length <= total - offset
+          raise RuntimeError, "invalid native #{name} arena range"
+        end
+        offset...(offset + length)
+      end
+
+      def copy_entry(index)
+        raise RuntimeError, "invalid native entry descriptor index" unless index.between?(0, @batch.entry_count - 1)
+        # `entries` collides with Fiddle::CStruct#entries; indexed field access
+        # selects the actual pointer member.
+        entries_address = address(@batch["entries"])
+        raise RuntimeError, "native entry descriptor array is null" if entries_address.zero?
+        descriptor = Native::DictionaryEntry.new(
+          Fiddle::Pointer.new(entries_address + index * Native::DictionaryEntry.size)
+        )
+        range = checked_range(descriptor.unit_offset, descriptor.unit_len, @batch.unit_count, "unit")
+        unit_address = address(@batch.units)
+        raise RuntimeError, "native unit arena is null" if range.size.positive? && unit_address.zero?
+        key = case @info.domain
+              when BYTE
+                range.size.zero? ? "".b : Fiddle::Pointer.new(unit_address + range.begin)[0, range.size].b
+              when UNICODE_SCALAR
+                scalars = range.size.zero? ? [] : Fiddle::Pointer.new(unit_address + range.begin * 4)[0, range.size * 4].unpack("L*")
+                scalars.pack("U*")
+              when U64
+                range.size.zero? ? [] : Fiddle::Pointer.new(unit_address + range.begin * 8)[0, range.size * 8].unpack("Q*")
+              else
+                raise RuntimeError, "unknown native entry unit domain #{@info.domain}"
+              end
+        value = case descriptor.value_len
+                when 0
+                  nil
+                when 1
+                  value_range = checked_range(descriptor.value_offset, 1, @batch.value_count, "value")
+                  values_address = address(@batch.values)
+                  raise RuntimeError, "native value arena is null" if values_address.zero?
+                  Fiddle::Pointer.new(values_address + value_range.begin * 8)[0, 8].unpack1("Q")
+                else
+                  raise RuntimeError, "invalid native optional-u64 descriptor"
+                end
+        Entry.new(key, value, @info.domain)
+      end
+
+      def release_locked
+        return nil unless @leased
+        Libdictenstein.check(Native.ldict_entry_cursor_release(@cursor, @batch.generation))
+        @leased = false
+        @index = 0
+        nil
+      end
+
+      def native_error(status)
+        status.zero? ? nil : Error.new(status)
+      end
+
+      def close_locked(cancel:)
+        return nil if @cursor.zero?
+        first_error = cancel ? native_error(Native.ldict_entry_cursor_cancel(@cursor)) : nil
+        begin
+          release_locked
+        rescue Exception => error
+          first_error ||= error
+        end
+        free_error = native_error(Native.ldict_entry_cursor_free(@cursor))
+        if free_error.nil?
+          @cursor = 0
+          @ended = true
+        else
+          first_error ||= free_error
+        end
+        raise first_error if first_error
+        nil
+      end
+    end
+
+    # Public bounded stream. Enumerable#each uses ensure so break and raised
+    # exceptions deterministically close the native cursor.
+    class EntryStream
+      include Enumerable
+
+      attr_reader :info
+
+      def initialize(handle, max_entries:, max_units:, max_values:)
+        @state = EntryCursorState.new(
+          handle,
+          max_entries: max_entries,
+          max_units: max_units,
+          max_values: max_values
+        )
+        @info = @state.info
+        ObjectSpace.define_finalizer(self, self.class.finalizer(@state))
+      end
+
+      def self.finalizer(state) = proc { state.close rescue nil }
+
+      def next = @state.next_entry
+
+      def each
+        return enum_for(__method__) unless block_given?
+        begin
+          while (entry = self.next)
+            yield entry
+          end
+        ensure
+          close
+        end
+        self
+      end
+
+      def cancel = @state.cancel
+
+      def close
+        @state.close
+        ObjectSpace.undefine_finalizer(self) if @state.closed?
+        nil
+      end
+    end
+
+    class Dictionary
+      include Enumerable
+
+      attr_reader :handle
+      def initialize(pointer)
+        @handle = ConcurrentHandle.new(pointer)
+        ObjectSpace.define_finalizer(self, self.class.finalizer(@handle))
+      end
+      def self.finalizer(handle) = proc { handle.close rescue nil }
+
+      def close
+        @handle.close
+        ObjectSpace.undefine_finalizer(self)
+        nil
+      end
+
+      def with_resource
+        @handle.with_pointer do |pointer|
+          resource = Native::Resource.malloc
+          Libdictenstein.check(Native.ldict_dictionary_resource(pointer, resource))
+          yield resource.context.to_i, resource.vtable.to_i
+        end
+      end
+
+      def kind = scalar(:ldict_dictionary_kind, Native.u64_output, ->(output) { Native.read_u64(output) & 0xffff_ffff })
+      def capabilities = scalar(:ldict_dictionary_capabilities, Native.u64_output, Native.method(:read_u64))
+      def length = scalar(:ldict_dictionary_len, Native.size_output, Native.method(:read_size))
+      alias size length
+
+      def entry_stream(max_entries: 256, max_units: 4096, max_values: 256)
+        EntryStream.new(
+          @handle,
+          max_entries: max_entries,
+          max_units: max_units,
+          max_values: max_values
+        )
+      end
+
+      def each
+        stream = entry_stream
+        return stream.each unless block_given?
+        stream.each { |entry| yield entry }
+        self
+      end
+
+      # Host-owned materialized snapshot idioms.
+      def entries = each.to_a
+      def keys = entries.map(&:key)
+      def values = entries.map(&:value)
+
+      def include?(term)
+        output = Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_contains_text(pointer, term.b, term.bytesize, output)) }
+        output[0].positive?
+      end
+
+      def get(term)
+        found, value, present = Native.byte_output, Native.u64_output, Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_get_text_value(pointer, term.b, term.bytesize, found, value, present)) }
+        Lookup.new(found[0].positive?, present[0].positive? ? Native.read_u64(value) : nil)
+      end
+
+      def include_u64?(tokens)
+        packed = tokens.pack("Q*"); output = Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_contains_u64(pointer, packed, tokens.length, output)) }
+        output[0].positive?
+      end
+
+      def get_u64(tokens)
+        packed = tokens.pack("Q*"); found, value, present = Native.byte_output, Native.u64_output, Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_get_u64_value(pointer, packed, tokens.length, found, value, present)) }
+        Lookup.new(found[0].positive?, present[0].positive? ? Native.read_u64(value) : nil)
+      end
+
+      private
+
+      def scalar(function, output, decode)
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.public_send(function, pointer, output)) }
+        decode.call(output)
+      end
+
+      def put_text(term, value)
+        output = Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_insert_text_value(pointer, term.b, term.bytesize, value || 0, value.nil? ? 0 : 1, output)) }
+        output[0].positive?
+      end
+
+      def remove_text(term)
+        output = Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_remove_text(pointer, term.b, term.bytesize, output)) }
+        output[0].positive?
+      end
+
+      def put_tokens(tokens, value)
+        packed = tokens.pack("Q*"); output = Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_insert_u64_value(pointer, packed, tokens.length, value || 0, value.nil? ? 0 : 1, output)) }
+        output[0].positive?
+      end
+
+      def remove_tokens(tokens)
+        packed = tokens.pack("Q*"); output = Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_remove_u64(pointer, packed, tokens.length, output)) }
+        output[0].positive?
+      end
+    end
+
+    class DynamicDawg < Dictionary
+      def initialize(domain: UNICODE_SCALAR)
+        output = Native.pointer_output; Libdictenstein.check(Native.ldict_dynamic_dawg_new(domain, output)); super(Native.read_pointer(output))
+      end
+      def put(term, value = nil) = put_text(term, value)
+      def remove(term) = remove_text(term)
+      def put_u64(tokens, value = nil) = put_tokens(tokens, value)
+      def remove_u64(tokens) = remove_tokens(tokens)
+      def clear = @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_clear(pointer)) }
+      def compact
+        output = Native.size_output; @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_compact(pointer, output)) }; Native.read_size(output)
+      end
+      def put_all(entries)
+        terms = entries.map { |term, _value| term.b }
+        memory = Fiddle::Pointer.malloc([1, entries.length * Native::TextEntry.size].max, Fiddle::RUBY_FREE)
+        entries.each_with_index do |(_term, value), index|
+          item = Native::TextEntry.new(memory + index * Native::TextEntry.size)
+          item.data = Fiddle::Pointer[terms[index]]
+          item.len = terms[index].bytesize
+          item.value = value || 0
+          item.has_value = value.nil? ? 0 : 1
+        end
+        output = Native.size_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_insert_text_batch(pointer, memory, entries.length, output)) }
+        Native.read_size(output)
+      end
+    end
+
+    class DoubleArrayTrie < Dictionary
+      def initialize(entries, domain: UNICODE_SCALAR)
+        terms = entries.map { |term, _value| term.b }
+        memory = Fiddle::Pointer.malloc([1, entries.length * Native::TextEntry.size].max, Fiddle::RUBY_FREE)
+        entries.each_with_index do |(_term, value), index|
+          item = Native::TextEntry.new(memory + index * Native::TextEntry.size)
+          item.data = Fiddle::Pointer[terms[index]]; item.len = terms[index].bytesize; item.value = value || 0; item.has_value = value.nil? ? 0 : 1
+        end
+        output = Native.pointer_output; Libdictenstein.check(Native.ldict_double_array_trie_new(domain, memory, entries.length, output)); super(Native.read_pointer(output))
+      end
+    end
+
+    class Scdawg < Dictionary
+      def initialize(domain: UNICODE_SCALAR)
+        output = Native.pointer_output; Libdictenstein.check(Native.ldict_scdawg_new(domain, output)); super(Native.read_pointer(output))
+      end
+      def put(term, value = nil) = put_text(term, value)
+      def include_substring?(term)
+        output = Native.byte_output; @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_scdawg_contains_substring(pointer, term.b, term.bytesize, output)) }; output[0].positive?
+      end
+      def substring_frequency(term)
+        output = Native.size_output; @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_scdawg_substring_frequency(pointer, term.b, term.bytesize, output)) }; Native.read_size(output)
+      end
+    end
+
+    class PersistentArtrie < Dictionary
+      def self.create(path, domain: UNICODE_SCALAR) = open_native(path, domain, true)
+      def self.open(path, domain: UNICODE_SCALAR) = open_native(path, domain, false)
+      def self.open_native(path, domain, create)
+        text = File.expand_path(path).b; output = Native.pointer_output
+        function = create ? :ldict_persistent_artrie_create : :ldict_persistent_artrie_open
+        Libdictenstein.check(Native.public_send(function, domain, text, text.bytesize, output)); new(Native.read_pointer(output))
+      end
+      def put(term, value = nil) = put_text(term, value)
+      def remove(term) = remove_text(term)
+      def put_u64(tokens, value = nil) = put_tokens(tokens, value)
+      def remove_u64(tokens) = remove_tokens(tokens)
+      def checkpoint = @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_checkpoint(pointer)) }
+    end
+
+    class PersistentVocabulary < Dictionary
+      def self.create(path) = open_native(path, true)
+      def self.open(path) = open_native(path, false)
+      def self.open_native(path, create)
+        text = File.expand_path(path).b; output = Native.pointer_output
+        function = create ? :ldict_persistent_vocab_create : :ldict_persistent_vocab_open
+        Libdictenstein.check(Native.public_send(function, text, text.bytesize, output)); new(Native.read_pointer(output))
+      end
+      def put(term, index) = put_text(term, index)
+      def term(index)
+        length, found = Native.size_output, Native.byte_output
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_vocab_get_term(pointer, index, 0, 0, length, found)) }
+        return nil if found[0].zero?
+        output = Fiddle::Pointer.malloc(Native.read_size(length), Fiddle::RUBY_FREE)
+        @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_vocab_get_term(pointer, index, output, output.size, length, found)) }
+        output[0, Native.read_size(length)].force_encoding(Encoding::UTF_8)
+      end
+      def checkpoint = @handle.with_pointer { |pointer| Libdictenstein.check(Native.ldict_dictionary_checkpoint(pointer)) }
+    end
+  end
+end

metadata

@@ -0,0 +1,52 @@
+--- !ruby/object:Gem::Specification
+name: libdictenstein
+version: !ruby/object:Gem::Version
+  version: 4.0.0.rc.4
+platform: ruby
+authors:
+- Dylon Edwards
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+description: DynamicDAWG, DAT, SCDAWG, and persistent ARTrie bindings for Ruby
+email:
+- dylon.devo@gmail.com
+executables:
+- libdictenstein-collection-profile
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- bin/libdictenstein-collection-profile
+- lib/vinary_tree/libdictenstein.rb
+- lib/vinary_tree/libdictenstein/native.rb
+- lib/vinary_tree/libdictenstein/native/darwin-arm64/liblibdictenstein.dylib
+- lib/vinary_tree/libdictenstein/native/linux-arm64/liblibdictenstein.so
+- lib/vinary_tree/libdictenstein/native/linux-x64/liblibdictenstein.so
+- lib/vinary_tree/libdictenstein/native/windows-x64/libdictenstein.dll
+- lib/vinary_tree/libdictenstein/version.rb
+homepage: https://github.com/vinary-tree/libdictenstein
+licenses:
+- Apache-2.0
+metadata:
+  source_code_uri: https://github.com/vinary-tree/libdictenstein
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.3'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.7.0.dev
+specification_version: 4
+summary: High-performance modular dictionary backends
+test_files: []