lda-ruby 0.3.9 → 0.5.0

data/docs/precompiled-platform-policy.md

@@ -0,0 +1,81 @@
+# Precompiled Platform Gem Policy (Phase 5B)
+
+This document defines the publish strategy and compatibility policy for `lda-ruby` precompiled gems.
+
+## Artifact Strategy
+
+Each release version publishes a split package set:
+
+- Source gem: `lda-ruby-<version>.gem`
+- Precompiled platform gems:
+  - `lda-ruby-<version>-x86_64-linux.gem`
+  - `lda-ruby-<version>-x86_64-darwin.gem`
+  - `lda-ruby-<version>-arm64-darwin.gem`
+  - `lda-ruby-<version>-x64-mingw-ucrt.gem`
+  - `lda-ruby-<version>-x86_64-linux-musl.gem`
+
+The source gem remains the universal fallback. Platform gems are additive and are expected to install without local build tools.
+Precompiled artifacts are built on matching host runners (no cross-compilation in current workflow).
+
+## Compatibility Policy
+
+- Supported Ruby versions: 3.2 and 3.3 (plus future versions validated by CI).
+- Release-blocking precompiled targets:
+  - Linux `x86_64-linux`
+  - Linux musl `x86_64-linux-musl`
+  - macOS Intel `x86_64-darwin`
+  - macOS Apple Silicon `arm64-darwin`
+  - Windows `x64-mingw-ucrt`
+- Other platforms:
+  - Install from source gem.
+  - Runtime remains supported through native/pure fallback paths.
+
+Backend behavior expectations:
+
+- Platform gem install:
+  - `auto` backend resolves to `rust` by default.
+  - `native` and `pure` overrides continue to work.
+- Source gem install:
+  - Rust build policy is controlled by `LDA_RUBY_RUST_BUILD=auto|always|never`.
+  - If Rust build is skipped/unavailable, `auto` falls back to `native`, then `pure_ruby`.
+
+## Guardrails
+
+Validation must pass before publish:
+
+- `./bin/release-preflight` (source-gem checks).
+- `./bin/release-precompiled-artifacts --platform <target>` for each release-blocking platform.
+
+Release automation requirements:
+
+- `.github/workflows/release.yml` builds source + precompiled artifacts.
+- Release workflow matrix must include all release-blocking precompiled targets.
+- Publish jobs push all built gems and attach checksums to GitHub releases.
+- Post-publish verification job must validate RubyGems entries and GitHub release assets for the tagged version.
+
+Continuous integration guardrail:
+
+- `.github/workflows/ci.yml` runs `release-precompiled-artifacts` for the full release-blocking precompiled matrix (Linux, Linux musl, macOS Intel, macOS Apple Silicon, Windows) on every branch/PR.
+- macOS precompiled lanes pin Homebrew `llvm@18` (falling back to `llvm` if unavailable) and export `LIBCLANG_PATH` from the selected prefix to keep bindgen stable across Homebrew formula updates.
+- `.github/workflows/precompiled-candidate-evaluation.yml` is used for additional platform candidate checks.
+- `.github/workflows/release.yml` dry-run validates the full release-blocking matrix before publish.
+
+Latest release-matrix validation:
+
+- [release dry-run 22556487788](https://github.com/ealdent/lda-ruby/actions/runs/22556487788) succeeded for Linux, Linux musl, macOS Intel, macOS Apple Silicon, and Windows targets.
+
+## Rollout / Expansion Rules
+
+When adding a new precompiled platform:
+
+1. Add target to release workflow matrix.
+2. Add or update CI coverage for that platform family.
+3. Update this policy and the release runbook support matrix.
+4. Record feasibility evidence and rollout notes in `docs/precompiled-target-evaluation.md`.
+5. Validate a dry-run release with `workflow_dispatch` before shipping.
+
+When deprecating a precompiled platform:
+
+1. Remove platform from release matrix.
+2. Keep source-gem path available unless the overall platform support policy changes.
+3. Document deprecation in `CHANGELOG.md` and release notes.

data/docs/precompiled-target-evaluation.md

@@ -0,0 +1,67 @@
+# Precompiled Target Evaluation (Priority 2)
+
+This document tracks current feasibility for expanding precompiled gem targets beyond the Phase 5B baseline.
+
+Current release-blocking precompiled targets:
+
+- `x86_64-linux`
+- `x86_64-darwin`
+- `arm64-darwin`
+- `x64-mingw-ucrt`
+- `x86_64-linux-musl`
+
+Reference implementation constraints:
+
+- `bin/release-precompiled-artifacts` only supports host-matching platform builds (no cross-compilation).
+- Release workflow currently uses matching host runners for each precompiled target.
+
+## Candidate: Windows (`x64-mingw-ucrt`)
+
+Status: promoted to release-blocking after release dry-run matrix success.
+
+Feasibility notes:
+
+- GitHub Actions provides Windows runners, so host-matching builds are possible in principle.
+- Existing release tooling is bash-first and assumes POSIX shell ergonomics throughout.
+- Runtime smoke and packaged-gem checks were validated in candidate runs before promotion.
+- Candidate runs:
+  - [run 22555475302](https://github.com/ealdent/lda-ruby/actions/runs/22555475302): failed in native extension compile (`rake compile`) with `cokus.h` macro collision and `time_t` mismatch.
+  - [run 22555550326](https://github.com/ealdent/lda-ruby/actions/runs/22555550326): progressed further, failed on `utils.c` `mkdir(name, mode)` mismatch (Windows `_mkdir` required).
+  - [run 22556009214](https://github.com/ealdent/lda-ruby/actions/runs/22556009214): Rust bindgen/toolchain parsing fixed; build then failed on Windows DLL name staging expectation.
+  - [run 22556129503](https://github.com/ealdent/lda-ruby/actions/runs/22556129503): Windows candidate build + artifact upload succeeded after GNU toolchain alignment, bindgen header/sysroot setup, and dual DLL name staging support.
+  - [run 22556206925](https://github.com/ealdent/lda-ruby/actions/runs/22556206925): Windows candidate remained green with packaged-gem runtime smoke checks enabled.
+  - [run 22556487788](https://github.com/ealdent/lda-ruby/actions/runs/22556487788): release workflow dry-run succeeded with `windows-x64-mingw-ucrt` included in release matrix.
+
+Required validation to promote:
+
+1. Completed: release dry-run matrix validation passed.
+
+## Candidate: musl Linux (`x86_64-linux-musl`)
+
+Status: promoted to release-blocking after release dry-run matrix success.
+
+Feasibility notes:
+
+- Current workflow uses `ubuntu-latest` (glibc), not musl.
+- Current artifact script rejects cross-platform builds, so a musl artifact requires either:
+  - a musl-hosted builder, or
+  - a dedicated musl-native build container/workflow path treated as host-equivalent for packaging.
+- Local validation signal (2026-03-01): Alpine container dry-run succeeded for host-matching `aarch64-linux-musl` with:
+  - `./bin/release-precompiled-artifacts --platform <detected-musl-platform> --skip-preflight --skip-runtime-checks`
+- Candidate workflow runs (2026-03-01):
+  - [run 22555475302](https://github.com/ealdent/lda-ruby/actions/runs/22555475302): built `x86_64-linux-musl` successfully but artifact upload path was misconfigured.
+  - [run 22555550326](https://github.com/ealdent/lda-ruby/actions/runs/22555550326): musl candidate built and uploaded artifacts successfully with corrected glob path (`pkg/lda-ruby-*-linux-musl.gem*`).
+  - [run 22556129503](https://github.com/ealdent/lda-ruby/actions/runs/22556129503): musl candidate build + artifact upload remained green alongside the fixed Windows lane.
+  - [run 22556206925](https://github.com/ealdent/lda-ruby/actions/runs/22556206925): musl candidate remained green with packaged-gem runtime smoke checks enabled.
+  - [run 22556487788](https://github.com/ealdent/lda-ruby/actions/runs/22556487788): release workflow dry-run succeeded with `linux-musl-x86_64` included in release matrix.
+
+Required validation to promote:
+
+1. Completed: release dry-run matrix validation passed.
+
+## Recommendation
+
+Current expansion step is complete for Windows and musl. Any additional target should follow the same sequence:
+1. Add candidate workflow coverage.
+2. Verify candidate runtime checks.
+3. Validate one release dry-run with the new matrix lane before promotion.

data/docs/release-runbook.md

@@ -0,0 +1,192 @@
+# Release Runbook (Phase 5A + 5B)
+
+This runbook defines the maintainer workflow for shipping `lda-ruby` source and precompiled platform gem releases.
+
+Authoritative platform/support policy is maintained in `docs/precompiled-platform-policy.md`; expansion feasibility notes live in `docs/precompiled-target-evaluation.md`.
+
+## Scope
+
+- Release artifact types:
+  - source gem: `pkg/lda-ruby-<version>.gem`
+  - precompiled gems (current targets are defined in `docs/precompiled-platform-policy.md`)
+- Release trigger: git tag (`vX.Y.Z`) with matching version files
+- Publish targets:
+  - RubyGems (`gem push`)
+  - GitHub Releases (gem + checksum attachment)
+
+## Prerequisites
+
+1. Access:
+   - push/tag rights on `master`
+   - access to GitHub Actions environments for release approvals
+   - RubyGems owner access for `lda-ruby`
+2. Local tooling:
+   - Ruby 3.2+ with Bundler
+   - Rust toolchain (`cargo`) for local precompiled-gem build checks
+   - `libclang` available to Rust bindgen
+   - Docker (recommended for reproducible checks)
+3. Repository state:
+   - release commit merged to `master`
+   - clean working tree
+   - version files in sync
+
+## Required Secrets and Environments
+
+GitHub repository secret:
+
+- `RUBYGEMS_API_KEY`: API key with push rights for `lda-ruby` and non-interactive publish support (no OTP prompt during `gem push`).
+
+GitHub Actions environment:
+
+- `release`: protect this environment with required reviewer approval.
+- Both publish jobs in `.github/workflows/release.yml` are bound to `release`.
+
+## Release Preparation
+
+1. Prepare and update release files:
+
+   ```bash
+   ./bin/release-prepare 0.4.0
+   ```
+
+2. Review changes:
+   - `VERSION.yml`
+   - `lib/lda-ruby/version.rb`
+   - `CHANGELOG.md`
+
+3. Validate full release checks locally:
+
+   ```bash
+   SKIP_DOCKER=1 ./bin/release-preflight
+   ./bin/test-packaged-gem-manifest
+   ```
+
+4. Validate local precompiled gem flow for your current host platform:
+
+   ```bash
+   ./bin/release-precompiled-artifacts --tag v0.4.0 --skip-preflight
+   ```
+
+   Note: `release-precompiled-artifacts` only supports building for the current host platform (no cross-compilation).
+
+5. Verify RubyGems API key behavior before tagging:
+
+   ```bash
+   ./bin/verify-rubygems-api-key
+   ```
+
+   This check intentionally attempts a duplicate push of an existing gem version. A duplicate-rejected response is expected and confirms non-interactive auth works.
+
+6. Commit and merge to `master`.
+
+## Dry-Run Path (No Publish)
+
+Use `workflow_dispatch` with `publish=false`.
+
+Behavior:
+
+- runs release validation and artifact build
+- uploads source + precompiled `pkg/lda-ruby-*.gem` and checksum files as workflow artifacts
+- does not push to RubyGems
+- does not create a GitHub release
+
+Latest verified dry-run reference:
+
+- date: 2026-03-02
+- workflow run: `https://github.com/ealdent/lda-ruby/actions/runs/22556487788`
+- dispatch parameters: `release_tag=v0.4.0`, `publish=false`
+- result: success across `validate`, `build_artifacts`, and full `build_precompiled_artifacts` matrix
+  - verified precompiled lanes:
+    - `linux-x86_64`
+    - `linux-musl-x86_64`
+    - `macos-x86_64`
+    - `macos-arm64`
+    - `windows-x64-mingw-ucrt`
+
+Optional local dry-run equivalent:
+
+```bash
+./bin/release-artifacts --tag v0.4.0
+./bin/release-precompiled-artifacts --tag v0.4.0 --skip-preflight
+```
+
+Candidate expansion workflow:
+
+- For future platform evaluation beyond current release-blocking targets, run `.github/workflows/precompiled-candidate-evaluation.yml` via `workflow_dispatch`.
+- Record outcome artifacts/logs in `docs/precompiled-target-evaluation.md`.
+
+## Known Publish Incident (`v0.4.0`)
+
+- date: 2026-02-25
+- release runs:
+  - `https://github.com/ealdent/lda-ruby/actions/runs/22383716372`
+  - `https://github.com/ealdent/lda-ruby/actions/runs/22383849236` (attempt 1 + rerun attempt 2 + rerun attempt 3)
+- result: artifact build stages passed, `publish to RubyGems` failed with OTP-required auth (`You have enabled multifactor authentication but no OTP code provided.`)
+- recovery action: rotated `release` environment secret `RUBYGEMS_API_KEY` to a CI-safe key and reran run `22383849236`.
+- recovery result: rerun attempt 3 succeeded; RubyGems `0.4.0` and GitHub release `v0.4.0` published.
+
+## Publish Path (Tag-Driven)
+
+1. Ensure the release commit is on `master`.
+2. Create and push the release tag:
+
+   ```bash
+   git checkout master
+   git pull --ff-only
+   git tag -a v0.4.0 -m "Release v0.4.0"
+   git push origin v0.4.0
+   ```
+
+3. Monitor `.github/workflows/release.yml`:
+   - `validate`
+   - `build_artifacts`
+   - `build_precompiled_artifacts` (linux + linux-musl + macOS + windows matrix)
+   - environment-gated `publish_rubygems`
+   - environment-gated `publish_github_release`
+   - `verify_published_artifacts`
+   - on failed tag-triggered `release.yml` runs, `.github/workflows/release-failure-alert.yml` opens a triage issue with failed job links
+   - if the same release run later succeeds (for example via rerun), the alert issue is auto-closed by `.github/workflows/release-failure-alert.yml`
+4. Approve the protected `release` environment when prompted.
+5. Confirm published outputs:
+   - RubyGems shows `lda-ruby` `0.4.0` source gem and platform gems
+   - GitHub release `v0.4.0` exists with all gem and `.sha256` attachments
+   - workflow job `verify_published_artifacts` succeeds
+
+## Rollback and Recovery
+
+If publish fails before RubyGems push:
+
+1. Fix issue on `master`.
+2. Delete and recreate the tag only if the broken tag did not produce public artifacts:
+   - `git tag -d vX.Y.Z`
+   - `git push origin :refs/tags/vX.Y.Z`
+3. Re-tag and re-run release.
+
+If RubyGems push succeeds but GitHub release fails:
+
+1. Re-run only the GitHub release path by re-running the workflow job after fix.
+2. Do not re-push gem for the same version.
+
+If an incorrect gem is published:
+
+1. Yank from RubyGems:
+
+   ```bash
+   gem yank lda-ruby -v X.Y.Z
+   ```
+
+2. Publish a corrective version (for example `X.Y.(Z+1)`), do not re-use yanked version numbers.
+3. Update `CHANGELOG.md` and release notes to document the correction.
+
+## Troubleshooting
+
+- `Could not find 'bundler'`: install the Bundler version pinned in `Gemfile.lock`.
+- `cargo not found` in rust-enabled checks: ensure Rust toolchain is installed or run in Docker.
+- `libclang` not found while building precompiled gems: install LLVM/libclang and set `LIBCLANG_PATH` if needed.
+- Linux `Install Rust bindgen dependencies` can take several minutes on fresh runners due apt package index and package installs.
+- RubyGems publish asks for OTP (`You have enabled multi-factor authentication but no OTP code provided`): run `./bin/verify-rubygems-api-key`, then rotate `RUBYGEMS_API_KEY` to a CI-safe key if OTP is requested.
+- Post-publish verification fails: run `./bin/verify-release-publish --tag vX.Y.Z` and fix missing RubyGems entries or GitHub release assets before considering the release complete.
+- macOS Rust link errors (`symbol(s) not found` for Ruby APIs): ensure build path preserves `-C link-arg=-Wl,-undefined,dynamic_lookup` in `RUSTFLAGS`.
+- Tag/version mismatch: run `./bin/check-version-sync --tag vX.Y.Z`.
+- Artifact mismatch during release: rebuild with `./bin/release-artifacts --tag vX.Y.Z`.
+- Precompiled artifact mismatch: rebuild with `./bin/release-precompiled-artifacts --tag vX.Y.Z --skip-preflight`.

data/docs/rust-orchestration-guardrails.md

@@ -0,0 +1,50 @@
+# Rust Orchestration Guardrails
+
+This document defines the minimum parity and performance gates for deeper Rust orchestration refactors.
+
+## Numeric parity guardrails
+
+Required tests:
+
+- `bundle exec ruby -Ilib:test test/backend_compatibility_test.rb`
+- `bundle exec ruby -Ilib:test test/rust_orchestration_test.rb`
+
+Current parity expectations:
+
+- Rust vs pure backend fixture parity remains exact within existing tolerances used by tests.
+- Session-based orchestration paths (`run_em_on_session`, `run_em_on_session_with_start_seed`, `run_em_on_session_start`, `run_em_on_session_with_corpus`) must match direct non-session orchestration for equivalent settings/seeds.
+- `Lda::Backends::Rust` cached-corpus EM should prefer the managed Rust session entrypoint (`run_em_on_session_with_corpus`) even when no active session id is cached locally, rather than branching in Ruby between session-only, recovery, and direct paths.
+- `Lda::Backends::Rust` non-session fallback should prefer Rust start-aware orchestration (`run_em_with_start_seed`) before legacy beta-input orchestration (`run_em`).
+- Direct non-session fallback should reuse the backend's cached Rust corpus snapshot rather than rebuilding corpus arrays from `@corpus` for each invocation.
+- Legacy beta-input compatibility fallback should also reuse the backend's cached Rust corpus snapshot rather than rebuilding full EM corpus input in Ruby.
+- Rust backend corpus/session lifecycle must not leak session count across corpus replacement.
+- Missing-session recovery in managed session orchestration (`run_em_on_session_with_corpus`) must recreate a usable session and keep parity with direct orchestration.
+- Managed Rust corpus orchestration (`run_em_on_session_with_corpus`) must keep parity with direct orchestration even when it falls back internally from session-backed execution to start-seeded array execution.
+- Corpus reassignment through Rust session replacement lifecycle (`replace_corpus_session`) must preserve stable session count and route subsequent EM runs over updated corpus data.
+- Unknown start-mode handling in seed-aware Rust orchestration must match Ruby's non-seeded fallback behavior when given the same explicit seed.
+
+## Benchmark guardrail
+
+Run:
+
+- `./bin/check-rust-benchmark`
+
+Default benchmark policy:
+
+- `BENCH_RUST_TO_PURE_MAX_RATIO=0.045`
+  - i.e., Rust mean runtime must be no worse than 4.5% of pure mean runtime on the benchmark fixture/config.
+- CI benchmark guardrail job enforces the same ratio with `BENCH_RUNS=1` for runtime stability.
+- latest tightening evidence (2026-03-05): local Docker guardrail check with `BENCH_RUNS=3` observed Rust/Pure ratio `0.0368` (`rust=0.0758s`, `pure=2.0569s`), and prior CI streak data on `codex/rust-orchestration-phase8` (`22555725309` .. `22557953998`) observed `[0.0252, 0.0288]`, supporting a tighter `0.045` threshold with headroom.
+
+Configurable environment knobs:
+
+- `BENCH_RUNS` (default `5`)
+- `BENCH_START` (default `seeded`)
+- `BENCH_TOPICS` (default `8`)
+- `BENCH_MAX_ITER` (default `20`)
+- `BENCH_EM_MAX_ITER` (default `40`)
+- `BENCH_RUST_TO_PURE_MAX_RATIO` (default `0.045`)
+
+## When to tighten thresholds
+
+Tighten benchmark thresholds only after collecting multiple stable runs on the same host/environment and updating this document with the new target ratio.

data/ext/lda-ruby/cokus.c

@@ -45,14 +45,14 @@
 
 #include "cokus.h"
 
-static uint32   state[N+1];     // state vector + 1 extra to not violate ANSI C
+static uint32   state[COKUS_N+1];     // state vector + 1 extra to not violate ANSI C
 static uint32   *next;          // next random value is computed from here
 static int      left = -1;      // can *next++ this many times before reloading
 
 void seedMT(uint32 seed)
  {
     //
-    // We initialize state[0..(N-1)] via the generator
+    // We initialize state[0..(COKUS_N-1)] via the generator
     //
     //   x_new = (69069 * x_old) mod 2^32
     //
@@ -100,28 +100,28 @@ void seedMT(uint32 seed)
     register uint32 x = (seed | 1U) & 0xFFFFFFFFU, *s = state;
     register int    j;
 
-    for(left=0, *s++=x, j=N; --j;
+    for(left=0, *s++=x, j=COKUS_N; --j;
         *s++ = (x*=69069U) & 0xFFFFFFFFU);
  }
 
 
 uint32 reloadMT(void)
 {
-    register uint32 *p0=state, *p2=state+2, *pM=state+M, s0, s1;
+    register uint32 *p0=state, *p2=state+2, *pM=state+COKUS_M, s0, s1;
     register int    j;
 
     if(left < -1)
         seedMT(4357U);
 
-    left=N-1, next=state+1;
+    left=COKUS_N-1, next=state+1;
 
-    for(s0=state[0], s1=state[1], j=N-M+1; --j; s0=s1, s1=*p2++)
-        *p0++ = *pM++ ^ (mixBits(s0, s1) >> 1) ^ (loBit(s1) ? K : 0U);
+    for(s0=state[0], s1=state[1], j=COKUS_N-COKUS_M+1; --j; s0=s1, s1=*p2++)
+        *p0++ = *pM++ ^ (mixBits(s0, s1) >> 1) ^ (loBit(s1) ? COKUS_K : 0U);
 
-    for(pM=state, j=M; --j; s0=s1, s1=*p2++)
-        *p0++ = *pM++ ^ (mixBits(s0, s1) >> 1) ^ (loBit(s1) ? K : 0U);
+    for(pM=state, j=COKUS_M; --j; s0=s1, s1=*p2++)
+        *p0++ = *pM++ ^ (mixBits(s0, s1) >> 1) ^ (loBit(s1) ? COKUS_K : 0U);
 
-    s1=state[0], *p0 = *pM ^ (mixBits(s0, s1) >> 1) ^ (loBit(s1) ? K : 0U);
+    s1=state[0], *p0 = *pM ^ (mixBits(s0, s1) >> 1) ^ (loBit(s1) ? COKUS_K : 0U);
     s1 ^= (s1 >> 11);
     s1 ^= (s1 <<  7) & 0x9D2C5680U;
     s1 ^= (s1 << 15) & 0xEFC60000U;
@@ -142,4 +142,3 @@ uint32 randomMT(void)
     y ^= (y >> 18);
     return(y);
  }
-

data/ext/lda-ruby/cokus.h

@@ -12,9 +12,9 @@
 
 typedef unsigned long uint32;
 
-#define N              (624)                 // length of state vector
-#define M              (397)                 // a period parameter
-#define K              (0x9908B0DFU)         // a magic constant
+#define COKUS_N        (624)                 // length of state vector
+#define COKUS_M        (397)                 // a period parameter
+#define COKUS_K        (0x9908B0DFU)         // a magic constant
 #define hiBit(u)       ((u) & 0x80000000U)   // mask all but highest   bit of u
 #define loBit(u)       ((u) & 0x00000001U)   // mask all but lowest    bit of u
 #define loBits(u)      ((u) & 0x7FFFFFFFU)   // mask     the highest   bit of u

data/ext/lda-ruby/extconf.rb

@@ -1,9 +1,13 @@
-ENV["ARCHFLAGS"] = "-arch #{`uname -p` =~ /powerpc/ ? 'ppc' : 'i386'}"
+# frozen_string_literal: true
 
-require 'mkmf'
+require "mkmf"
 
-$CFLAGS << ' -Wall -ggdb -O0'
-$defs.push( "-D USE_RUBY" )
+extension_name = "lda-ruby/lda"
+dir_config(extension_name)
 
-dir_config('lda-ruby/lda')
-create_makefile("lda-ruby/lda")
+$defs << "-DUSE_RUBY"
+append_cflags("-Wall")
+append_cflags("-Wextra")
+append_cflags("-Wno-unused-parameter")
+
+create_makefile(extension_name)

data/ext/lda-ruby/lda-inference.c

@@ -435,9 +435,9 @@ void infer(char* model_root, char* save, corpus* corpus) {
 int main(int argc, char* argv[]) {
 	corpus* corpus;
 
-	long t1;
+	time_t t1;
 	(void) time(&t1);
-	seedMT(t1);
+	seedMT((uint32) t1);
 		// seedMT(4357U);
 
 	if (argc > 1)
@@ -614,7 +614,25 @@ void run_quiet_em(char* start, corpus* corpus) {
  *  * em_convergence
  *  * est_alpha
  */
-static VALUE wrap_set_config(VALUE self, VALUE init_alpha, VALUE num_topics, VALUE max_iter, VALUE convergence, VALUE em_max_iter, VALUE em_convergence, VALUE est_alpha) {
+static VALUE wrap_set_config(int argc, VALUE* argv, VALUE self) {
+  VALUE init_alpha = Qnil;
+  VALUE num_topics = Qnil;
+  VALUE max_iter = Qnil;
+  VALUE convergence = Qnil;
+  VALUE em_max_iter = Qnil;
+  VALUE em_convergence = Qnil;
+  VALUE est_alpha = Qnil;
+
+  rb_check_arity(argc, 5, 7);
+
+  init_alpha = argv[0];
+  num_topics = argv[1];
+  max_iter = argv[2];
+  convergence = argv[3];
+  em_max_iter = argv[4];
+  em_convergence = (argc >= 6) ? argv[5] : rb_float_new(EM_CONVERGED);
+  est_alpha = (argc == 7) ? argv[6] : rb_int_new(ESTIMATE_ALPHA);
+
 	INITIAL_ALPHA = NUM2DBL(init_alpha);
 	NTOPICS = NUM2INT(num_topics);
   if( NTOPICS < 0 ) { rb_raise(rb_eRuntimeError, "NTOPICS must be greater than 0 - %d", NTOPICS); }
@@ -954,13 +972,11 @@ static VALUE wrap_get_model_settings(VALUE self) {
 }
 
 
-void Init_lda() {
+void Init_lda(void) {
   corpus_loaded = FALSE;
   model_loaded = FALSE;
   VERBOSE = TRUE;
 
-  rb_require("lda-ruby");
-
   rb_cLdaModule   = rb_define_module("Lda");
   rb_cLda         = rb_define_class_under(rb_cLdaModule, "Lda", rb_cObject);
   rb_cLdaCorpus   = rb_define_class_under(rb_cLdaModule, "Corpus", rb_cObject);
@@ -977,7 +993,7 @@ void Init_lda() {
   rb_define_method(rb_cLda, "load_settings", wrap_load_settings, 1);
 
   // method to set all the config options at once
-  rb_define_method(rb_cLda, "set_config", wrap_set_config, 5);
+  rb_define_method(rb_cLda, "set_config", wrap_set_config, -1);
 
   // accessor stuff for main settings
   rb_define_method(rb_cLda, "max_iter", wrap_get_max_iter, 0);

data/ext/lda-ruby/utils.c

@@ -1,5 +1,9 @@
 #include "utils.h"
 
+#ifdef _WIN32
+#include <direct.h>
+#endif
+
 /*
  * given log(a) and log(b), return log(a + b)
  *
@@ -85,7 +89,11 @@ double log_gamma(double x)
 
 void make_directory(char* name)
 {
+#ifdef _WIN32
+    _mkdir(name);
+#else
     mkdir(name, S_IRUSR|S_IWUSR|S_IXUSR);
+#endif
 }
 
 

data/ext/lda-ruby-rust/Cargo.toml

@@ -0,0 +1,12 @@
+[package]
+name = "lda_ruby_rust"
+version = "0.1.0"
+edition = "2021"
+rust-version = "1.74"
+
+[lib]
+name = "lda_ruby_rust"
+crate-type = ["cdylib"]
+
+[dependencies]
+magnus = "0.7"

data/ext/lda-ruby-rust/README.md

@@ -0,0 +1,73 @@
+# Experimental Rust Extension Scaffold
+
+This directory contains an experimental Rust extension scaffold built with `magnus`.
+
+Current scope:
+
+- Defines `Lda::RustBackend` module in Ruby.
+- Exposes capability hooks:
+  - `Lda::RustBackend.available?`
+  - `Lda::RustBackend.abi_version`
+  - `Lda::RustBackend.corpus_session_count`
+  - `Lda::RustBackend.corpus_session_exists(session_id)`
+  - `Lda::RustBackend.before_em(start, num_docs, num_terms)`
+  - `Lda::RustBackend.topic_weights_for_word(beta, gamma, word_index, min_probability)`
+  - `Lda::RustBackend.accumulate_topic_term_counts(topic_term_counts, phi_d, words, counts)`
+  - `Lda::RustBackend.infer_document(beta, gamma_initial, words, counts, max_iter, convergence, min_probability, init_alpha)`
+  - `Lda::RustBackend.infer_corpus_iteration(beta, document_words, document_counts, max_iter, convergence, min_probability, init_alpha)`
+  - `Lda::RustBackend.normalize_topic_term_counts(topic_term_counts, min_probability)`
+  - `Lda::RustBackend.average_gamma_shift(previous_gamma, current_gamma)`
+  - `Lda::RustBackend.topic_document_probability(phi_tensor, document_counts, num_topics, min_probability)`
+  - `Lda::RustBackend.seeded_topic_term_probabilities(document_words, document_counts, topics, terms, min_probability)`
+  - `Lda::RustBackend.random_topic_term_probabilities(topics, terms, min_probability, random_seed)`
+  - `Lda::RustBackend.create_corpus_session(document_words, document_counts, terms)`
+  - `Lda::RustBackend.replace_corpus_session(session_id, document_words, document_counts, terms)`
+  - `Lda::RustBackend.drop_corpus_session(session_id)`
+  - `Lda::RustBackend.configure_corpus_session(session_id, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability)`
+  - `Lda::RustBackend.run_em(initial_beta, document_words, document_counts, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability)`
+  - `Lda::RustBackend.run_em_with_start(start, document_words, document_counts, topics, terms, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability)`
+  - `Lda::RustBackend.run_em_with_start_seed(start, document_words, document_counts, topics, terms, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+  - `Lda::RustBackend.run_em_on_session(session_id, start, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+  - `Lda::RustBackend.run_em_on_session_start(session_id, start, random_seed)`
+  - `Lda::RustBackend.run_em_on_session_with_start_seed(session_id, start, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+  - `Lda::RustBackend.run_em_on_session_with_corpus(session_id, document_words, document_counts, terms, start, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+
+Hot-path kernels currently executed in Rust when `backend: :rust` is active:
+- topic weights for a word across topics
+- topic-term count accumulation from per-document `phi`
+- full per-document inference loop (batched inner EM updates)
+- full per-iteration corpus inference (batched document processing)
+- topic-term normalization and log-probability finalization for EM beta updates
+- gamma convergence shift reduction between EM iterations
+- topic-document average log-probability computation
+- seeded topic-term initialization
+- random topic-term initialization with explicit seed control
+- EM outer-loop orchestration with convergence checks (`run_em`)
+- start-aware deterministic EM orchestration (`run_em_with_start` for `seeded`/`deterministic`)
+- start-aware seeded and random EM orchestration with explicit seed control (`run_em_with_start_seed`)
+- unified session-settings orchestration (`run_em_on_session`) that applies settings and executes EM in one call
+- session-based EM orchestration against Rust-managed corpus lifecycle (`create_corpus_session` + `run_em_on_session_with_start_seed`)
+- settings-aware session orchestration (`configure_corpus_session` + `run_em_on_session_start`)
+- managed corpus orchestration (`run_em_on_session_with_corpus`) that can recreate missing sessions and, if session-backed execution cannot be used, falls back internally to direct start-aware execution inside Rust
+- `Lda::Backends::Rust` prefers `run_em_on_session_with_corpus` whenever a cached Rust corpus snapshot is available, even if no session id is currently cached locally
+- direct and legacy beta-input compatibility fallbacks both reuse the backend's cached Rust corpus snapshot instead of rebuilding corpus arrays in Ruby
+- unknown EM start modes in seed-aware orchestration follow Ruby's non-seeded fallback behavior (seeded by explicit `random_seed`)
+
+Remaining numeric LDA kernels are still provided by the pure Ruby backend and will move incrementally.
+
+## Local build (optional)
+
+```bash
+cd ext/lda-ruby-rust
+cargo build --release
+```
+
+Then run Ruby with `require "lda_ruby_rust"` available on load path.
+
+## Install-time policy
+
+During source gem installs, `ext/lda-ruby-rust/extconf.rb` can optionally build this extension.
+
+- `LDA_RUBY_RUST_BUILD=auto` (default): build when `cargo` is available.
+- `LDA_RUBY_RUST_BUILD=always`: require a successful Rust build or fail installation.
+- `LDA_RUBY_RUST_BUILD=never`: always skip Rust build.