decompound 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: da38e3a4783149369844c3bf7ec4aaefffe650b24f600eea4c0d5dcf482e8fc9
-  data.tar.gz: 8e02323ca26643ac63f25b2403b2605a0a46707e67fa810614aa0f26e900ace7
+  metadata.gz: cfbd2354f4af5ae69ac9e258b52e658a4ad859b7882d5c60150902dfc1a68480
+  data.tar.gz: 27ff07f69553d8467c004967f584f2316c35dada2d738cd7cc6a6edfb269ca0e
 SHA512:
-  metadata.gz: '089027fdcbacbd946ffc6698ece09b843225e0956baea46c0f0bdb7f255517c54399a289d63f565a446b61f0332ed3522185a6c430a05605984ead964d3af338'
-  data.tar.gz: ffcef37ea872e33c59eb8d8d8620555b8631f085b0e9f6bad6ffced58e32966e15113a382a5db6cf32c0b23cbe9e2c48d508532adaf5d704812f04136d5951f8
+  metadata.gz: 327855e9992b2e955de67bd03147547ed113c30a60c30e7b0bc819b14224de8f39f660847bf317e491cb0d3c2720dd23efac2a9b142a1a011a2109831d875233
+  data.tar.gz: ca70c7dd58def1472d87882b3838099e7ae6ce9b8e85f37dd5278d85fad5c7184925eeda8353bb65c38330f81fafaa60d65415cfa1f6b339f1eda200f345563e

data/.ruby-version

@@ -0,0 +1 @@
+4.0.6

data/CHANGELOG.md

@@ -1,5 +1,9 @@
 ## [Unreleased]
 
+## [0.3.0] - 2026-07-16
+
+- Use binary model format: Reduce memory by searching a packed binary model instead of parsing JSON into hashes. Slower throughput though.
+
 ## [0.2.0] - 2026-07-15
 
 - Train bundled model from German Wiktionary nouns (~95% head-detection accuracy on GermaNet compounds)

data/README.md

@@ -17,48 +17,36 @@ The bundled model is trained on 127,106 German Wiktionary noun lemmas and their
 
 ## Installation
 
-Install the gem and add to the application's Gemfile by executing:
-
 ```bash
 bundle add decompound
 ```
 
-If bundler is not being used to manage dependencies, install the gem by executing:
+Or without bundler:
 
 ```bash
 gem install decompound
 ```
 
-## Usage
-
-```rb
-require "decompound"
-
-word = "Bodenbelag"
-parts = Decompound.split(word)
-# => ["boden", "belag"]
-```
-
 ## Roadmap
 
-- Improve memory efficiency
-- Provide different models (small, medium, large)
+- Provide different models (small, medium, large) (?)
+- Recursive decompounding
+  - `Fussbodenbelag` → `Fuss` + `Bodenbelag` → `Fuss` + `Boden` + `Belag`
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
-
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+After checking out the repo, run `bin/setup` to install dependencies, `rake test` to run the tests, and `bin/console` for an interactive prompt.
 
 ### Release process
 
 For each release:
 
 1. Update `lib/decompound/version.rb`
-2. Update `CHANGELOG.md`
-3. Run tests and RBS validation
-4. Commit with message `Prepare x.x.x release` and push
-5. Run `bundle exec rake release`
+2. Run `bundle install` to update `Gemfile.lock`
+3. Update `CHANGELOG.md`
+4. Run tests and RBS validation
+5. Commit with message `Prepare x.x.x release` and push
+6. Run `bundle exec rake release`
 
 ### Training CharSplit probabilities
 
@@ -73,11 +61,19 @@ ruby bin/train.rb \
 
 `--forms lemmas` uses only lemmas, while `--forms nominative` uses lemmas plus nominative inflections. Inflected forms provide the linking-element evidence (e.g. the *-s* in *Einkaufsliste*), so the bundled model uses `all`. The generated JSON contains CharSplit-compatible `prefix`, `infix`, and `suffix` probabilities.
 
+### Packing the runtime model
+
+The gem ships the probabilities as a binary table (`data/model.bin`), searched in place:
+
+```sh
+ruby bin/pack.rb data/ngram_probs.json data/model.bin
+```
+
 ## Provenance and credits
 
 The noun corpus used to train the bundled probabilities is derived from [German Wiktionary](https://de.wiktionary.org/). It is extracted by [Wiktextract](https://github.com/tatuylonen/wiktextract) and distributed by [Kaikki.org](https://kaikki.org/). Wiktionary textual content is licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/) and, where applicable, the GNU Free Documentation License. See [`data/vendor/german-nouns/SOURCE.md`](data/vendor/german-nouns/SOURCE.md) for the pinned source, extraction process, and reproducibility details.
 
-The splitting and training algorithms are Ruby ports of [CharSplit](https://github.com/dtuggener/CharSplit), created by Don Tuggener and described in *Incremental Coreference Resolution for German* (University of Zurich, 2016). CharSplit is MIT licensed; its copyright and license are reproduced in [`LICENSE-CharSplit.txt`](LICENSE-CharSplit.txt).
+The splitting and training algorithms are Ruby ports of [CharSplit](https://github.com/dtuggener/CharSplit), created by Don Tuggener and described in [Incremental Coreference Resolution for German](https://doi.org/10.5167/uzh-124915) (University of Zurich, 2016). CharSplit is MIT licensed; its copyright and license are reproduced in [`LICENSE-CharSplit.txt`](LICENSE-CharSplit.txt).
 
 See [`NOTICE.md`](NOTICE.md) for consolidated third-party notices.
 

data/lib/decompound/model.rb

@@ -0,0 +1,79 @@
+# frozen_string_literal: true
+
+module Decompound
+  # Reads the binary model format (decompound-binary-v1):
+  #
+  #   magic "DCPDv1\n"
+  #   count           uint32 LE
+  #   bucket_count    uint32 LE
+  #   buckets         bucket_count x (key prefix, NUL-padded to 3 bytes + uint32 LE first index)
+  #   offsets         (count + 1) x uint32 LE, byte offsets into the keys blob
+  #   probabilities   count x 3 bytes (prefix, infix, suffix), 255 = missing
+  #   keys            concatenated UTF-8 n-grams, sorted bytewise
+  #
+  # The file is held as a single string and searched in place, so loaded
+  # memory stays near file size instead of materializing millions of Ruby
+  # strings and hash entries. The buckets narrow each binary search to the
+  # keys sharing the query's first three bytes.
+
+  class Model
+    MAGIC = "DCPDv1\n"
+    PREFIX = 0
+    INFIX = 1
+    SUFFIX = 2
+    MISSING = 255
+    SCALE = 254.0
+
+    def self.load(path)
+      new(File.binread(path))
+    end
+
+    def initialize(data)
+      raise ArgumentError, "not a decompound binary model" unless data.byteslice(0, MAGIC.bytesize) == MAGIC
+
+      @data = data.freeze
+      @count, bucket_count = data.byteslice(MAGIC.bytesize, 8).unpack("VV")
+      buckets_at = MAGIC.bytesize + 8
+      @offsets_at = buckets_at + (bucket_count * 7)
+      @probabilities_at = @offsets_at + ((@count + 1) * 4)
+      @keys_at = @probabilities_at + (@count * 3)
+      @buckets = read_buckets(data, buckets_at, bucket_count)
+    end
+
+    def probability(ngram, position, default)
+      index = find(ngram.b)
+      return default unless index
+
+      code = @data.getbyte(@probabilities_at + (index * 3) + position)
+      (code == MISSING) ? default : code / SCALE
+    end
+
+    private
+
+    def read_buckets(data, buckets_at, bucket_count)
+      firsts = bucket_count.times.map { |i| data.byteslice(buckets_at + (i * 7) + 3, 4).unpack1("V") }
+      firsts << @count
+      buckets = {}
+      bucket_count.times do |i|
+        buckets[data.byteslice(buckets_at + (i * 7), 3)] = [firsts[i], firsts[i + 1] - 1]
+      end
+      buckets.freeze
+    end
+
+    def find(query)
+      low, high = @buckets[query.byteslice(0, 3).ljust(3, "\0")]
+      return nil unless low
+
+      while low <= high
+        mid = (low + high) / 2
+        from, upto = @data.byteslice(@offsets_at + (mid * 4), 8).unpack("VV")
+        case query <=> @data.byteslice(@keys_at + from, upto - from)
+        when 0 then return mid
+        when -1 then high = mid - 1
+        else low = mid + 1
+        end
+      end
+      nil
+    end
+  end
+end

data/lib/decompound/packer.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require_relative "model"
+
+module Decompound
+  # Compiles the trained JSON probability tables into the binary runtime model, see Decompound::Model
+
+  class Packer
+    POSITIONS = {"prefix" => Model::PREFIX, "infix" => Model::INFIX, "suffix" => Model::SUFFIX}.freeze
+
+    def pack(probabilities)
+      entries = Hash.new { |hash, key| hash[key] = [Model::MISSING] * 3 }
+      POSITIONS.each do |position, slot|
+        probabilities.fetch(position).each do |ngram, probability|
+          entries[ngram.b][slot] = (probability.clamp(0.0, 1.0) * Model::SCALE).round
+        end
+      end
+
+      keys = entries.keys.sort!
+      offsets = []
+      blob = +""
+      buckets = +""
+      bucket_count = 0
+      previous_prefix = nil
+      keys.each_with_index do |key, index|
+        prefix = key.byteslice(0, 3).ljust(3, "\0")
+        if prefix != previous_prefix
+          buckets << prefix << [index].pack("V")
+          bucket_count += 1
+          previous_prefix = prefix
+        end
+        offsets << blob.bytesize
+        blob << key
+      end
+      offsets << blob.bytesize
+
+      Model::MAGIC.b +
+        [keys.length, bucket_count].pack("VV") +
+        buckets +
+        offsets.pack("V*") +
+        keys.flat_map { |key| entries.fetch(key) }.pack("C*") +
+        blob
+    end
+  end
+end

data/lib/decompound/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Decompound
-  VERSION = "0.2.0"
+  VERSION = "0.3.0"
 end

data/lib/decompound.rb

@@ -1,10 +1,10 @@
 # frozen_string_literal: true
 
-require "json"
 require_relative "decompound/version"
+require_relative "decompound/model"
 
 module Decompound
-  NGRAM_PATH = File.expand_path("../data/ngram_probs.json", __dir__)
+  MODEL_PATH = File.expand_path("../data/model.bin", __dir__)
   FUGEN_S_ENDINGS = %w[ts gs ks hls ns].freeze
 
   def self.split(word)
@@ -20,7 +20,7 @@ module Decompound
   end
 
   def self.split_candidates(word)
-    probabilities = ngram_probabilities
+    model = self.model
 
     (3...(word.length - 2)).map do |position|
       left = word[0...position]
@@ -28,11 +28,12 @@ module Decompound
       left_ngram = without_fugen_s(left)
       right_ngram = without_fugen_s(right)
 
-      suffix_probability = probabilities.fetch("suffix").fetch(left_ngram, -1)
-      prefix_probability = probabilities.fetch("prefix").fetch(right_ngram, -1)
-      infix_probability = (3..(word.length + 1)).filter_map do |length|
-        ngram = right[0, length]
-        probabilities.fetch("infix").fetch(ngram, 1) unless ngram.empty?
+      suffix_probability = model.probability(left_ngram, Model::SUFFIX, -1)
+      prefix_probability = model.probability(right_ngram, Model::PREFIX, -1)
+      # CharSplit scans lengths up to word.length + 1; slices past the end of
+      # +right+ all yield +right+ itself, so stopping there is equivalent.
+      infix_probability = (3..right.length).map do |length|
+        model.probability(right[0, length], Model::INFIX, 1)
       end.min
 
       [prefix_probability - infix_probability + suffix_probability, left, right]
@@ -48,8 +49,8 @@ module Decompound
   end
   private_class_method :without_fugen_s
 
-  def self.ngram_probabilities
-    @ngram_probabilities ||= JSON.parse(File.read(NGRAM_PATH, encoding: "utf-8"))
+  def self.model
+    @model ||= Model.load(MODEL_PATH)
   end
-  private_class_method :ngram_probabilities
+  private_class_method :model
 end

metadata

@@ -1,34 +1,21 @@
 --- !ruby/object:Gem::Specification
 name: decompound
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Camillo Visini
 bindir: exe
 cert_chain: []
 date: 1980-01-02 00:00:00.000000000 Z
-dependencies:
-- !ruby/object:Gem::Dependency
-  name: json
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '2.9'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '2.9'
+dependencies: []
 description: A Ruby gem for splitting German compound words into their constituent
   parts
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- ".ruby-version"
 - CHANGELOG.md
 - Justfile
 - LICENSE-CharSplit.txt
@@ -36,11 +23,14 @@ files:
 - NOTICE.md
 - README.md
 - Rakefile
+- data/model.bin
 - data/ngram_probs.json
 - lib/charsplit/german_nouns.rb
 - lib/charsplit/ngram_counter.rb
 - lib/charsplit/trainer.rb
 - lib/decompound.rb
+- lib/decompound/model.rb
+- lib/decompound/packer.rb
 - lib/decompound/version.rb
 - script/download_german_nouns.rb
 - sig/decompound.rbs