lda-ruby 0.3.9 → 0.4.0

data/docs/release-runbook.md

@@ -0,0 +1,157 @@
+# Release Runbook (Phase 5A + 5B)
+
+This runbook defines the maintainer workflow for shipping `lda-ruby` source and precompiled platform gem releases.
+
+Authoritative platform/support policy is maintained in `docs/precompiled-platform-policy.md`.
+
+## Scope
+
+- Release artifact types:
+  - source gem: `pkg/lda-ruby-<version>.gem`
+  - precompiled gems (current targets are defined in `docs/precompiled-platform-policy.md`)
+- Release trigger: git tag (`vX.Y.Z`) with matching version files
+- Publish targets:
+  - RubyGems (`gem push`)
+  - GitHub Releases (gem + checksum attachment)
+
+## Prerequisites
+
+1. Access:
+   - push/tag rights on `master`
+   - access to GitHub Actions environments for release approvals
+   - RubyGems owner access for `lda-ruby`
+2. Local tooling:
+   - Ruby 3.2+ with Bundler
+   - Rust toolchain (`cargo`) for local precompiled-gem build checks
+   - `libclang` available to Rust bindgen
+   - Docker (recommended for reproducible checks)
+3. Repository state:
+   - release commit merged to `master`
+   - clean working tree
+   - version files in sync
+
+## Required Secrets and Environments
+
+GitHub repository secret:
+
+- `RUBYGEMS_API_KEY`: API key with push rights for `lda-ruby`.
+
+GitHub Actions environment:
+
+- `release`: protect this environment with required reviewer approval.
+- Both publish jobs in `.github/workflows/release.yml` are bound to `release`.
+
+## Release Preparation
+
+1. Prepare and update release files:
+
+   ```bash
+   ./bin/release-prepare 0.4.0
+   ```
+
+2. Review changes:
+   - `VERSION.yml`
+   - `lib/lda-ruby/version.rb`
+   - `CHANGELOG.md`
+
+3. Validate full release checks locally:
+
+   ```bash
+   SKIP_DOCKER=1 ./bin/release-preflight
+   ./bin/test-packaged-gem-manifest
+   ```
+
+4. Validate local precompiled gem flow for your current host platform:
+
+   ```bash
+   ./bin/release-precompiled-artifacts --tag v0.4.0 --skip-preflight
+   ```
+
+   Note: `release-precompiled-artifacts` only supports building for the current host platform (no cross-compilation).
+
+5. Commit and merge to `master`.
+
+## Dry-Run Path (No Publish)
+
+Use `workflow_dispatch` with `publish=false`.
+
+Behavior:
+
+- runs release validation and artifact build
+- uploads source + precompiled `pkg/lda-ruby-*.gem` and checksum files as workflow artifacts
+- does not push to RubyGems
+- does not create a GitHub release
+
+Latest verified dry-run reference:
+
+- date: 2026-02-25
+- workflow run: `https://github.com/ealdent/lda-ruby/actions/runs/22382692416`
+- dispatch parameters: `release_tag=v0.4.0`, `publish=false`
+- result: success across `validate`, `build_artifacts`, and full `build_precompiled_artifacts` matrix
+
+Optional local dry-run equivalent:
+
+```bash
+./bin/release-artifacts --tag v0.4.0
+./bin/release-precompiled-artifacts --tag v0.4.0 --skip-preflight
+```
+
+## Publish Path (Tag-Driven)
+
+1. Ensure the release commit is on `master`.
+2. Create and push the release tag:
+
+   ```bash
+   git checkout master
+   git pull --ff-only
+   git tag -a v0.4.0 -m "Release v0.4.0"
+   git push origin v0.4.0
+   ```
+
+3. Monitor `.github/workflows/release.yml`:
+   - `validate`
+   - `build_artifacts`
+   - `build_precompiled_artifacts` (linux + macOS matrix)
+   - environment-gated `publish_rubygems`
+   - environment-gated `publish_github_release`
+4. Approve the protected `release` environment when prompted.
+5. Confirm published outputs:
+   - RubyGems shows `lda-ruby` `0.4.0` source gem and platform gems
+   - GitHub release `v0.4.0` exists with all gem and `.sha256` attachments
+
+## Rollback and Recovery
+
+If publish fails before RubyGems push:
+
+1. Fix issue on `master`.
+2. Delete and recreate the tag only if the broken tag did not produce public artifacts:
+   - `git tag -d vX.Y.Z`
+   - `git push origin :refs/tags/vX.Y.Z`
+3. Re-tag and re-run release.
+
+If RubyGems push succeeds but GitHub release fails:
+
+1. Re-run only the GitHub release path by re-running the workflow job after fix.
+2. Do not re-push gem for the same version.
+
+If an incorrect gem is published:
+
+1. Yank from RubyGems:
+
+   ```bash
+   gem yank lda-ruby -v X.Y.Z
+   ```
+
+2. Publish a corrective version (for example `X.Y.(Z+1)`), do not re-use yanked version numbers.
+3. Update `CHANGELOG.md` and release notes to document the correction.
+
+## Troubleshooting
+
+- `Could not find 'bundler'`: install the Bundler version pinned in `Gemfile.lock`.
+- `cargo not found` in rust-enabled checks: ensure Rust toolchain is installed or run in Docker.
+- `libclang` not found while building precompiled gems: install LLVM/libclang and set `LIBCLANG_PATH` if needed.
+- Linux `Install Rust bindgen dependencies` can take several minutes on fresh runners due apt package index and package installs.
+- macOS Rust link errors (`symbol(s) not found` for Ruby APIs): ensure build path preserves `-C link-arg=-Wl,-undefined,dynamic_lookup` in `RUSTFLAGS`.
+- Tag/version mismatch: run `./bin/check-version-sync --tag vX.Y.Z`.
+- Artifact mismatch during release: rebuild with `./bin/release-artifacts --tag vX.Y.Z`.
+- Precompiled artifact mismatch: rebuild with `./bin/release-precompiled-artifacts --tag vX.Y.Z --skip-preflight`.

data/ext/lda-ruby/extconf.rb

@@ -1,9 +1,13 @@
-ENV["ARCHFLAGS"] = "-arch #{`uname -p` =~ /powerpc/ ? 'ppc' : 'i386'}"
+# frozen_string_literal: true
 
-require 'mkmf'
+require "mkmf"
 
-$CFLAGS << ' -Wall -ggdb -O0'
-$defs.push( "-D USE_RUBY" )
+extension_name = "lda-ruby/lda"
+dir_config(extension_name)
 
-dir_config('lda-ruby/lda')
-create_makefile("lda-ruby/lda")
+$defs << "-DUSE_RUBY"
+append_cflags("-Wall")
+append_cflags("-Wextra")
+append_cflags("-Wno-unused-parameter")
+
+create_makefile(extension_name)

data/ext/lda-ruby/lda-inference.c

@@ -614,7 +614,25 @@ void run_quiet_em(char* start, corpus* corpus) {
  *  * em_convergence
  *  * est_alpha
  */
-static VALUE wrap_set_config(VALUE self, VALUE init_alpha, VALUE num_topics, VALUE max_iter, VALUE convergence, VALUE em_max_iter, VALUE em_convergence, VALUE est_alpha) {
+static VALUE wrap_set_config(int argc, VALUE* argv, VALUE self) {
+  VALUE init_alpha = Qnil;
+  VALUE num_topics = Qnil;
+  VALUE max_iter = Qnil;
+  VALUE convergence = Qnil;
+  VALUE em_max_iter = Qnil;
+  VALUE em_convergence = Qnil;
+  VALUE est_alpha = Qnil;
+
+  rb_check_arity(argc, 5, 7);
+
+  init_alpha = argv[0];
+  num_topics = argv[1];
+  max_iter = argv[2];
+  convergence = argv[3];
+  em_max_iter = argv[4];
+  em_convergence = (argc >= 6) ? argv[5] : rb_float_new(EM_CONVERGED);
+  est_alpha = (argc == 7) ? argv[6] : rb_int_new(ESTIMATE_ALPHA);
+
 	INITIAL_ALPHA = NUM2DBL(init_alpha);
 	NTOPICS = NUM2INT(num_topics);
   if( NTOPICS < 0 ) { rb_raise(rb_eRuntimeError, "NTOPICS must be greater than 0 - %d", NTOPICS); }
@@ -954,13 +972,11 @@ static VALUE wrap_get_model_settings(VALUE self) {
 }
 
 
-void Init_lda() {
+void Init_lda(void) {
   corpus_loaded = FALSE;
   model_loaded = FALSE;
   VERBOSE = TRUE;
 
-  rb_require("lda-ruby");
-
   rb_cLdaModule   = rb_define_module("Lda");
   rb_cLda         = rb_define_class_under(rb_cLdaModule, "Lda", rb_cObject);
   rb_cLdaCorpus   = rb_define_class_under(rb_cLdaModule, "Corpus", rb_cObject);
@@ -977,7 +993,7 @@ void Init_lda() {
   rb_define_method(rb_cLda, "load_settings", wrap_load_settings, 1);
 
   // method to set all the config options at once
-  rb_define_method(rb_cLda, "set_config", wrap_set_config, 5);
+  rb_define_method(rb_cLda, "set_config", wrap_set_config, -1);
 
   // accessor stuff for main settings
   rb_define_method(rb_cLda, "max_iter", wrap_get_max_iter, 0);

data/ext/lda-ruby-rust/Cargo.toml

@@ -0,0 +1,12 @@
+[package]
+name = "lda_ruby_rust"
+version = "0.1.0"
+edition = "2021"
+rust-version = "1.74"
+
+[lib]
+name = "lda_ruby_rust"
+crate-type = ["cdylib"]
+
+[dependencies]
+magnus = "0.7"

data/ext/lda-ruby-rust/README.md

@@ -0,0 +1,48 @@
+# Experimental Rust Extension Scaffold
+
+This directory contains an experimental Rust extension scaffold built with `magnus`.
+
+Current scope:
+
+- Defines `Lda::RustBackend` module in Ruby.
+- Exposes capability hooks:
+  - `Lda::RustBackend.available?`
+  - `Lda::RustBackend.abi_version`
+  - `Lda::RustBackend.before_em(start, num_docs, num_terms)`
+  - `Lda::RustBackend.topic_weights_for_word(beta, gamma, word_index, min_probability)`
+  - `Lda::RustBackend.accumulate_topic_term_counts(topic_term_counts, phi_d, words, counts)`
+  - `Lda::RustBackend.infer_document(beta, gamma_initial, words, counts, max_iter, convergence, min_probability, init_alpha)`
+  - `Lda::RustBackend.infer_corpus_iteration(beta, document_words, document_counts, max_iter, convergence, min_probability, init_alpha)`
+  - `Lda::RustBackend.normalize_topic_term_counts(topic_term_counts, min_probability)`
+  - `Lda::RustBackend.average_gamma_shift(previous_gamma, current_gamma)`
+  - `Lda::RustBackend.topic_document_probability(phi_tensor, document_counts, num_topics, min_probability)`
+  - `Lda::RustBackend.seeded_topic_term_probabilities(document_words, document_counts, topics, terms, min_probability)`
+
+Hot-path kernels currently executed in Rust when `backend: :rust` is active:
+- topic weights for a word across topics
+- topic-term count accumulation from per-document `phi`
+- full per-document inference loop (batched inner EM updates)
+- full per-iteration corpus inference (batched document processing)
+- topic-term normalization and log-probability finalization for EM beta updates
+- gamma convergence shift reduction between EM iterations
+- topic-document average log-probability computation
+- seeded topic-term initialization
+
+Remaining numeric LDA kernels are still provided by the pure Ruby backend and will move incrementally.
+
+## Local build (optional)
+
+```bash
+cd ext/lda-ruby-rust
+cargo build --release
+```
+
+Then run Ruby with `require "lda_ruby_rust"` available on load path.
+
+## Install-time policy
+
+During source gem installs, `ext/lda-ruby-rust/extconf.rb` can optionally build this extension.
+
+- `LDA_RUBY_RUST_BUILD=auto` (default): build when `cargo` is available.
+- `LDA_RUBY_RUST_BUILD=always`: require a successful Rust build or fail installation.
+- `LDA_RUBY_RUST_BUILD=never`: always skip Rust build.

data/ext/lda-ruby-rust/extconf.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "rbconfig"
+
+require_relative "../../lib/lda-ruby/rust_build_policy"
+
+module Lda
+  module RustExtensionBuild
+    module_function
+
+    def run
+      policy = RustBuildPolicy.resolve
+      puts("Rust extension build policy: #{policy} (#{RustBuildPolicy::ENV_KEY})")
+
+      case policy
+      when RustBuildPolicy::NEVER
+        puts("Skipping Rust extension build (policy=#{RustBuildPolicy::NEVER}).")
+      when RustBuildPolicy::ALWAYS
+        ensure_cargo_available!
+        build_and_stage!
+      else
+        if cargo_available?
+          build_and_stage!
+        else
+          puts("cargo not found; skipping Rust extension build (policy=#{RustBuildPolicy::AUTO}).")
+        end
+      end
+
+      write_noop_makefile
+    rescue StandardError => e
+      if policy == RustBuildPolicy::ALWAYS
+        abort("Rust extension build failed with #{RustBuildPolicy::ENV_KEY}=#{RustBuildPolicy::ALWAYS}: #{e.message}")
+      end
+
+      warn("Rust extension build skipped after error in auto mode: #{e.message}")
+      write_noop_makefile
+    end
+
+    def ensure_cargo_available!
+      return if cargo_available?
+
+      abort("cargo not found in PATH but #{RustBuildPolicy::ENV_KEY}=#{RustBuildPolicy::ALWAYS} was requested.")
+    end
+
+    def cargo_available?
+      cargo = ENV.fetch("CARGO", "cargo")
+      system(cargo, "--version", out: File::NULL, err: File::NULL)
+    end
+
+    def build_and_stage!
+      cargo = ENV.fetch("CARGO", "cargo")
+      Dir.chdir(__dir__) do
+        env = rust_build_env
+        success =
+          if env.empty?
+            system(cargo, "build", "--release")
+          else
+            system(env, cargo, "build", "--release")
+          end
+        success or raise "cargo build --release failed"
+      end
+
+      source = File.join(__dir__, "target", "release", rust_cdylib_filename)
+      raise "Rust extension artifact not found at #{source}" unless File.exist?(source)
+
+      destination = File.expand_path("../../lib/lda_ruby_rust.#{RbConfig::CONFIG.fetch('DLEXT')}", __dir__)
+      FileUtils.cp(source, destination)
+      puts("Staged Rust extension to #{destination}")
+    end
+
+    def rust_cdylib_filename
+      host_os = RbConfig::CONFIG.fetch("host_os")
+      extension =
+        case host_os
+        when /darwin/
+          "dylib"
+        when /mswin|mingw|cygwin/
+          "dll"
+        else
+          "so"
+        end
+
+      "liblda_ruby_rust.#{extension}"
+    end
+
+    def rust_build_env
+      host_os = RbConfig::CONFIG.fetch("host_os")
+      return {} unless host_os.match?(/darwin/)
+
+      dynamic_lookup_flag = "-C link-arg=-Wl,-undefined,dynamic_lookup"
+      existing = ENV.fetch("RUSTFLAGS", "")
+      merged =
+        if existing.include?(dynamic_lookup_flag)
+          existing
+        else
+          [existing, dynamic_lookup_flag].reject(&:empty?).join(" ")
+        end
+
+      { "RUSTFLAGS" => merged }
+    end
+
+    def write_noop_makefile
+      File.write(
+        File.join(__dir__, "Makefile"),
+        <<~MAKEFILE
+          all:
+          \t@echo "Rust extension handled by extconf.rb"
+
+          install:
+          \t@echo "Rust extension handled by extconf.rb"
+
+          clean:
+          \t@true
+
+          distclean: clean
+        MAKEFILE
+      )
+    end
+  end
+end
+
+Lda::RustExtensionBuild.run