lda-ruby 0.3.9 → 0.5.0

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    M2IzN2IwMmE1YWFhNTJjYjk0ZjMyYzhjMWQyZDdjZjY2YTA0OGNjZA==
-  data.tar.gz: !binary |-
-    OTNjZjE5MGNmOGI2YzY3YzhlNDRiYTBlNDM5NmUwYmY4Mjc2ZmNkNQ==
+SHA256:
+  metadata.gz: a2470a73c1ba2c8807574f34a606a429adbd20d94457990667b59d241b521171
+  data.tar.gz: 8c7aa9b952901e15b48b974d68b339ab20e41edfcc07fa391266591b219b5771
 SHA512:
-  metadata.gz: !binary |-
-    MDAxZmQ1N2U5MGQ1MDA2MGE4YTU0N2NkN2Y4N2Q4YzlmNTdmYTkxN2FlYzU0
-    ZmFjZDZiZWIwZmRiMDJlYjNjYTg5YWQ0N2RlOTY1MTg0YzZjZTc0NGQ0YmI1
-    ZDljMjljNTA3ODdlZjBjNjNjMTc0MGRlMjBhODQzZTg3YWM5OWE=
-  data.tar.gz: !binary |-
-    OGFiMzVhMzY4OTFmZTkzMmM3YjQxMzNkM2JlMjVjOTRjM2ZhNTc1YmUxZTRj
-    Y2M3YzZiNzNiYmVkNDI1ZTUxODhkMjhlYTQ4MmRhZjVkZjQxMDhjOTk5Njc4
-    OTZhOTM5ZTQ3OTFhY2U1YjRhODQyYzBkZWNlYzhjNzBjMWFlNGU=
+  metadata.gz: 3fbe0fe85517e82b63bb8f28358178fd2490e1c4162342efa62864b7a5d83cc81488c83b75443bc8a539319717c588b37a27504722947c361a08636884ab4133
+  data.tar.gz: c240216720322fb4a539a63cff5c85a37544fbbad6281bb2b5c89787016f9dab817ab6db8295083e7c218454b23065ec06836e454b24d47e85111696e2e5dd1c

data/CHANGELOG.md

@@ -1,3 +1,19 @@
+version 0.5.0
+=============
+
+- Expanded Rust-side EM orchestration with cached corpus snapshots, managed corpus sessions, and start-aware fallback paths.
+- Added Rust benchmark guardrails and deeper pure/Rust orchestration parity coverage.
+- Added release-blocking precompiled gem targets for Windows (`x64-mingw-ucrt`) and Linux musl (`x86_64-linux-musl`).
+- Hardened precompiled release workflows with full-matrix CI, post-publish verification, and release failure alerts.
+
+version 0.4.0
+=============
+
+- Ruby 3.2+/3.3+ modernization release.
+- Added Rust/native/pure backend selection and packaged-gem install policy validation.
+- Added tag-driven release automation and release runbook.
+- Added precompiled platform gem build/publish pipeline (Linux + macOS) and compatibility policy documentation.
+
 version 0.3.9
 =============
 

data/Gemfile

@@ -0,0 +1,9 @@
+source "https://rubygems.org"
+
+gemspec
+
+group :development, :test do
+  gem "rake", ">= 13.0"
+  gem "test-unit", ">= 3.6"
+  gem "shoulda-context", "~> 2.0"
+end

data/README.md

@@ -17,6 +17,119 @@ The original C code relied on files for the input and output. We felt it was nec
 
 If you have general questions about Latent Dirichlet Allocation, I urge you to use the [topic models mailing list][topic-models], since the people who monitor that are very knowledgeable.  If you encounter bugs specific to lda-ruby, please post an issue on the Github project.
 
+## Development
+
+### Local (Ruby 3.2+)
+
+```bash
+bundle install
+bundle exec rake test
+```
+
+### Docker (recommended for isolated setup)
+
+```bash
+./bin/docker-test
+```
+
+Rust backend runtime checks in Docker:
+
+```bash
+./bin/docker-test-rust
+```
+
+Install policy matrix checks in Docker:
+
+```bash
+./bin/docker-test-install-policies
+```
+
+For an interactive shell inside the dev container:
+
+```bash
+./bin/docker-shell
+```
+
+For an interactive shell with Rust toolchain + bindgen dependencies:
+
+```bash
+./bin/docker-shell-rust
+```
+
+### Build tasks
+
+- `bundle exec rake compile` builds the native extension.
+- `bundle exec rake compile_rust` builds the experimental Rust extension and stages a Ruby-loadable artifact (`lda_ruby_rust.<dlext>`).
+  - On macOS, build tasks automatically add the Rust linker flag for Ruby extension `dynamic_lookup`.
+- `bundle exec rake test` rebuilds the extension, then runs tests.
+- `bundle exec rake build` builds the gem package.
+- `bundle exec ruby -Ilib:test test/backend_compatibility_test.rb` runs backend compatibility fixtures.
+- `LDA_RUBY_BACKEND=rust bundle exec ruby -Ilib:test test/backend_compatibility_test.rb` runs parity checks in rust mode.
+- `./bin/benchmark-backends` benchmarks available backends (`pure`, `native`, `rust`) and prints JSON.
+- `./bin/check-rust-benchmark` enforces the Rust/pure benchmark ratio guardrail (configurable via env vars).
+- `./bin/docker-test-install-policies` verifies packaged-gem install behavior for `LDA_RUBY_RUST_BUILD=auto|always|never`, including runtime EM smoke checks.
+- `./bin/test-packaged-gem-fallback` verifies packaged-gem fallback behavior without Cargo (auto/never succeed, always fails) plus runtime smoke checks.
+- `./bin/test-packaged-gem-rust-enabled` verifies packaged-gem behavior with Cargo available (auto/always enable Rust, never disables Rust) plus runtime smoke checks.
+- `./bin/test-packaged-gem-manifest` verifies packaged-gem contents/metadata and rejects leaked build artifacts.
+- `./bin/release-preflight` runs unit tests + packaged-gem validation stack; set `SKIP_DOCKER=1` to skip Docker matrix checks.
+- `./bin/check-version-sync` verifies version parity between `VERSION.yml`, `lib/lda-ruby/version.rb`, and expected release tag.
+- `./bin/verify-rubygems-api-key` validates that your RubyGems API key can push non-interactively (required for CI release publishes).
+- `./bin/verify-release-publish --tag v0.4.0` verifies published RubyGems + GitHub release assets for a release tag.
+- `./bin/release-prepare 0.4.0` updates version/changelog files for a new release version.
+- `./bin/release-artifacts --tag v0.4.0` runs release checks, builds the source gem, and writes SHA256 checksums.
+- `./bin/release-precompiled-artifacts --tag v0.4.0 --platform x86_64-linux --skip-preflight` builds a precompiled platform gem and verifies install/runtime smoke checks.
+  - The `--platform` value must match the current host platform.
+
+Benchmark environment variables:
+- `BENCH_RUNS` (default: `3`)
+- `BENCH_START` (default: `seeded`)
+- `BENCH_TOPICS` (default: `8`)
+- `BENCH_MAX_ITER` (default: `20`)
+- `BENCH_EM_MAX_ITER` (default: `40`)
+
+### Install-time Rust build policy
+
+Source installs now run both extension setup scripts (`ext/lda-ruby/extconf.rb` and `ext/lda-ruby-rust/extconf.rb`).
+
+Rust build policy is controlled by `LDA_RUBY_RUST_BUILD`:
+- `auto` (default): build Rust extension if `cargo` is available, otherwise skip.
+- `always`: require Rust extension build and fail install if unavailable.
+- `never`: skip Rust extension build.
+
+Examples:
+- `LDA_RUBY_RUST_BUILD=always gem install lda-ruby`
+- `LDA_RUBY_RUST_BUILD=never bundle exec rake compile`
+
+### Precompiled platform gems
+
+Releases publish a source gem plus precompiled platform gems for:
+- `x86_64-linux`
+- `x86_64-darwin`
+- `arm64-darwin`
+
+On these platforms, installation should not require local C/Rust toolchains.
+Other platforms install from source gem and use the existing install-time fallback policy.
+
+For artifact strategy, compatibility targets, and rollout/deprecation rules, see `docs/precompiled-platform-policy.md`.
+
+### Backend selection
+
+- Default mode is `auto`: Rust backend when available, otherwise native extension, otherwise pure Ruby.
+- Force pure Ruby backend:
+  - `Lda::Lda.new(corpus, backend: :pure)`
+  - or `LDA_RUBY_BACKEND=pure`
+- Force native backend:
+  - `Lda::Lda.new(corpus, backend: :native)`
+- Force Rust backend (when extension is available):
+  - `Lda::Lda.new(corpus, backend: :rust)`
+  - or `LDA_RUBY_BACKEND=rust`
+
+`em("seeded")` is supported by both native and pure backends for deterministic fixture-oriented runs.
+
+Rust status: the extension hook layer is scaffolded in `ext/lda-ruby-rust`. Current Rust kernels include batched per-iteration corpus inference, batched per-document inference, topic-weights-per-word, topic-term-count accumulation, topic-term normalization/log-beta finalization, gamma-shift convergence reduction, topic-document average log-probability computation, seeded topic-term initialization, random topic-term initialization, and Rust-side EM orchestration paths (`run_em`, `run_em_with_start`, `run_em_with_start_seed`, session-based `run_em_on_session_with_start_seed`, settings-aware session `run_em_on_session_start`, and unified session-settings orchestration `run_em_on_session`) when `backend: :rust` is active. Session orchestration now uses shared Rust-side corpus storage and borrowed execution paths so EM session runs do not deep-clone corpus arrays per call, and the Ruby adapter now auto-recreates missing Rust sessions before EM to stay on the session path. The Rust backend still keeps the pure Ruby implementation as a compatibility fallback path if Rust orchestration is unavailable or returns invalid output. CI runs dedicated rust-runtime checks and numeric parity fixtures against the pure backend.
+`compile_rust` and `LDA_RUBY_RUST_BUILD=always` require a Rust toolchain plus Ruby development headers and `libclang`.
+Gem packaging excludes local Rust build artifacts (`ext/lda-ruby-rust/target/**`) so local cargo outputs do not leak into published gems.
+
 ## Resources
 
 + [Blog post about LDA-Ruby][lda-ruby]
@@ -29,9 +142,19 @@ If you have general questions about Latent Dirichlet Allocation, I urge you to u
 Blei, David M., Ng, Andrew Y., and Jordan, Michael I. 2003. Latent dirichlet allocation. Journal of Machine Learning Research. 3 (Mar. 2003), 993-1022 [[pdf][pdf]].
 
 [svmlight]: http://svmlight.joachims.org
-[lda-ruby]: http://mendicantbug.com/2008/11/17/lda-in-ruby/
-[blei]: http://www.cs.princeton.edu/~blei/lda-c/
+[lda-ruby]: http://web.archive.org/web/20120616115448/http://mendicantbug.com/2008/11/17/lda-in-ruby/
+[blei]: http://web.archive.org/web/20161126004857/http://www.cs.princeton.edu/~blei/lda-c/
 [wikipedia]: http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
-[ap-data]: http://www.cs.princeton.edu/~blei/lda-c/ap.tgz
+[ap-data]: http://web.archive.org/web/20160507090044/http://www.cs.princeton.edu/~blei/lda-c/ap.tgz
 [pdf]: http://www.cs.princeton.edu/picasso/mats/BleiNgJordan2003_blei.pdf
 [topic-models]: https://lists.cs.princeton.edu/mailman/listinfo/topic-models
+
+## Modernization
+
+For a Ruby 3.2+/3.3+ porting proposal, see `docs/porting-strategy.md`.
+
+For the latest implementation status and exact resume instructions, see `docs/modernization-handoff.md`.
+
+For release steps and rollback guidance, see `docs/release-runbook.md`.
+
+For precompiled gem strategy and compatibility policy, see `docs/precompiled-platform-policy.md`.

data/VERSION.yml

@@ -1,5 +1,5 @@
 ---
 :major: 0
-:minor: 3
-:patch: 9
-:build: 
+:minor: 5
+:patch: 0
+:build:

data/docs/modernization-handoff.md

@@ -0,0 +1,233 @@
+# Modernization Handoff (Resume Guide)
+
+This document is the canonical handoff state for continuing the Ruby 3.2+/3.3+ modernization in a new conversation.
+
+## Snapshot
+
+- Snapshot date: 2026-03-02
+- Active branch: `codex/rust-orchestration-phase8`
+- Repo status at snapshot start: clean working tree on `codex/rust-orchestration-phase8` (ahead of `master`)
+- Modernization branch: `codex/modernization` merged into `master`
+- Merge commit for modernization PR: `bc11269` (`Merge pull request #18 from ealdent/codex/modernization`)
+- Latest release dry-run validation (GitHub Actions):
+  - date: 2026-03-02
+  - workflow run: `release.yml` run `22556487788` (`workflow_dispatch`, `publish=false`)
+  - result: success (`validate release candidate`, `build release artifacts`, and all `build precompiled artifacts` matrix targets)
+  - publish jobs skipped by design (`publish=false`)
+- Modernization pull request:
+  - PR `#18`: `https://github.com/ealdent/lda-ruby/pull/18`
+  - state: merged (2026-02-25)
+- Release publish attempts (`v0.4.0`):
+  - `release.yml` run `22383716372`: failed at RubyGems publish (`No such API key`)
+  - `release.yml` run `22383849236`: failed at RubyGems publish (`OTP code required`)
+  - rerun attempt (`22383849236`, attempt 2): failed at RubyGems publish (`OTP code required`)
+  - rerun attempt (`22383849236`, attempt 3): success (RubyGems publish + GitHub release publish complete)
+- Release status:
+  - `v0.4.0` published to RubyGems (source + precompiled Linux/macOS gems)
+  - GitHub release `v0.4.0` published with gem + `.sha256` assets
+  - next release dry-run matrix is validated for Linux, Linux musl, macOS Intel, macOS Apple Silicon, and Windows targets (`22556487788`)
+
+## Project Goal
+
+Modernize `lda-ruby` for Ruby 3.2+/3.3+ with:
+
+- stable Ruby API compatibility
+- high-performance Rust backend for hot paths
+- pure Ruby fallback for portability
+- reliable packaging and release validation
+
+## Current Backend Behavior
+
+- `auto` selection order: `rust` -> `native` -> `pure_ruby`
+- `LDA_RUBY_BACKEND` override supported for `rust`, `native`, and `pure`
+- Rust build policy for source installs: `LDA_RUBY_RUST_BUILD=auto|always|never`
+
+## Phase Status
+
+### Phase 1 (API/tests stabilization)
+
+Status: complete.
+
+Delivered:
+
+- expanded compatibility fixtures
+- CI test coverage for multiple backend modes
+
+### Phase 2 (backend boundary extraction)
+
+Status: complete.
+
+Delivered:
+
+- `Lda::Lda` delegates through backend adapters
+- backend selection normalized through `Lda::Backends.build`
+
+### Phase 3 (pure Ruby reference backend)
+
+Status: complete.
+
+Delivered:
+
+- full pure Ruby backend path with EM/model outputs
+- pure backend compatibility in tests
+
+### Phase 4 (Rust native backend)
+
+Status: mostly complete.
+
+Delivered:
+
+- Rust extension scaffold with magnus/rb_sys
+- Rust kernels for the main hot loops:
+  - corpus iteration
+  - document inference
+  - topic weights
+  - topic-term accumulation
+  - topic-term finalization (`beta`/`log(beta)`)
+  - gamma shift reduction
+  - topic-document probability
+  - seeded initialization
+- trusted kernel-output fast path enabled in rust mode
+- Rust-side EM orchestration path (`Lda::RustBackend.run_em`) retained as a compatibility fallback for precomputed beta-input execution
+- Rust-side start-aware deterministic orchestration path (`Lda::RustBackend.run_em_with_start`) for `seeded`/`deterministic` EM starts
+- Rust-side random-start orchestration path (`Lda::RustBackend.run_em_with_start_seed`) using explicit seed-controlled random initialization (`Lda::RustBackend.random_topic_term_probabilities`)
+- Rust-managed corpus session lifecycle (`Lda::RustBackend.create_corpus_session` / `drop_corpus_session`) with session-based start-aware EM orchestration (`run_em_on_session_with_start_seed`) wired in `Lda::Backends::Rust`
+- Rust-managed session settings lifecycle (`configure_corpus_session`) with settings-aware orchestration (`run_em_on_session_start`) wired in `Lda::Backends::Rust`
+- Rust session execution refactor to shared session corpus storage + borrowed orchestration helpers, eliminating deep corpus clone overhead on each session EM run
+- unified Rust session orchestration API (`run_em_on_session`) that applies settings + runs EM in one call inside Rust session orchestration
+- `Lda::Backends::Rust` now routes cached-corpus EM through managed Rust orchestration (`run_em_on_session_with_corpus`), leaving session reuse/recovery decisions in Rust and preferring that managed path even when no active session id is cached locally; when the managed API is unavailable it still prefers direct Rust non-session orchestration (`run_em_with_start_seed`) before legacy Ruby-side beta-input fallback (`run_em`)
+- direct non-session Rust orchestration now reuses the backend's cached Rust corpus snapshot instead of rebuilding corpus arrays from `@corpus` on each fallback invocation
+- legacy Rust beta-input compatibility fallback now also reuses the backend's cached Rust corpus snapshot, only asking the pure-Ruby backend to synthesize the initial beta matrix
+- Rust managed-session orchestration API (`run_em_on_session_with_corpus`) added to recreate missing sessions and run EM in one Rust call, and now directly falls back to start-aware array execution inside Rust if session-backed execution cannot be used
+- Rust session lifecycle replacement API (`replace_corpus_session`) added so corpus reassignment can update existing Rust sessions in place (config reset + corpus swap) instead of Ruby-side drop/recreate
+- `Lda::Backends::Rust` now keeps session-based orchestration on the managed Rust path (`run_em_on_session_with_corpus`) even when sessions are dropped externally
+- parity/compatibility test coverage and rust runtime CI
+
+Open in Phase 4:
+
+- optional deeper Rust ownership beyond current unified session orchestration (for example additional control-plane logic and lifecycle APIs)
+
+### Phase 5 (packaging/release)
+
+Status: Phase 5A complete (source-gem release automation), Phase 5B complete for Linux/macOS/Windows/musl precompiled release matrix.
+
+Delivered:
+
+- clean gem file filtering (no local cargo/native artifacts)
+- Docker install-policy matrix checks
+- packaged gem runtime checks without Cargo (`bin/test-packaged-gem-fallback`)
+- packaged gem runtime checks with Cargo (`bin/test-packaged-gem-rust-enabled`)
+- packaged gem manifest/metadata gate (`bin/test-packaged-gem-manifest`)
+- single-command local gate (`bin/release-preflight`)
+- version/tag parity guard (`bin/check-version-sync`)
+- RubyGems CI credential preflight helper (`bin/verify-rubygems-api-key`)
+- post-publish artifact verification helper (`bin/verify-release-publish`)
+- deterministic release preparation helper (`bin/release-prepare`)
+- release artifact builder with checksum output (`bin/release-artifacts`)
+- precompiled artifact builder + runtime validator (`bin/release-precompiled-artifacts`)
+- gemspec precompiled variant support (`LDA_RUBY_GEM_VARIANT=precompiled`)
+- precompiled platform compatibility/publish policy (`docs/precompiled-platform-policy.md`)
+- precompiled target expansion tracker (`docs/precompiled-target-evaluation.md`)
+- macOS Rust build linker guardrail (`dynamic_lookup`) for precompiled packaging paths
+- tag-driven release workflow (`.github/workflows/release.yml`)
+- release failure alert workflow (`.github/workflows/release-failure-alert.yml`)
+  - tuned to create issues only for failed tag-triggered release runs, with failed job links in alert body
+  - now auto-closes matching alert issues when the same release run later reports success
+- maintainer release runbook (`docs/release-runbook.md`)
+- manual precompiled candidate workflow now validated on both lanes (`precompiled-candidate-evaluation.yml` run `22556129503`)
+- manual precompiled candidate workflow now validated with runtime checks on both lanes (`precompiled-candidate-evaluation.yml` run `22556206925`)
+- Windows candidate portability hardening across native + Rust build paths:
+  - C portability fixes (`cokus` macros, `time_t`, `_mkdir`)
+  - Rust toolchain alignment to GNU target for RubyInstaller (`x64-mingw-ucrt`)
+  - Rust bindgen header/sysroot compatibility wiring for Windows runners
+  - dual Rust DLL artifact-name staging support (`lda_ruby_rust.dll` and `liblda_ruby_rust.dll`)
+- CI jobs for packaged-gem fallback, rust-enabled checks, and manifest checks
+- CI precompiled gem build guardrail job (`precompiled-gem-build`) aligned to the full release-blocking precompiled matrix
+- macOS precompiled CI/release lanes now pin Homebrew `llvm@18` (with fallback to `llvm`) and export `LIBCLANG_PATH` from the selected prefix to avoid bindgen breakage from Homebrew LLVM drift
+- release workflow matrix for precompiled gems:
+  - `x86_64-linux`
+  - `x86_64-linux-musl`
+  - `x86_64-darwin`
+  - `arm64-darwin`
+  - `x64-mingw-ucrt`
+- release dry-run validation for expanded precompiled matrix (`release.yml` run `22556487788`)
+
+Open in Phase 5:
+
+- optional expansion of precompiled targets beyond current Linux/macOS/Windows/musl set
+
+## Validation Commands
+
+Core:
+
+- `./bin/docker-test`
+- `./bin/docker-test-rust`
+
+Packaging/release checks:
+
+- `./bin/check-version-sync`
+- `./bin/verify-rubygems-api-key`
+- `./bin/verify-release-publish --tag v0.4.0`
+- `./bin/test-packaged-gem-manifest`
+- `./bin/test-packaged-gem-fallback`
+- `./bin/test-packaged-gem-rust-enabled`
+- `SKIP_DOCKER=1 ./bin/release-preflight`
+- `./bin/release-artifacts --tag v0.4.0`
+- `./bin/release-precompiled-artifacts --tag v0.4.0 --skip-preflight`
+
+Optional full Docker matrix:
+
+- `./bin/docker-test-install-policies`
+
+Performance tracking:
+
+- `./bin/benchmark-backends`
+- `./bin/check-rust-benchmark`
+- `docs/rust-orchestration-guardrails.md`
+- CI currently enforces `BENCH_RUST_TO_PURE_MAX_RATIO=0.045` in `benchmark-guardrail`
+
+## CI Jobs Expected
+
+- native tests (`test-native`)
+- pure backend tests (`test-pure`)
+- rust runtime tests (`rust-runtime`)
+- Docker install policy matrix (`install-policy-matrix`)
+- packaged gem fallback checks (`packaged-gem-fallback`)
+- packaged gem rust-enabled checks (`packaged-gem-rust-enabled`)
+- packaged gem manifest checks (`packaged-gem-manifest`)
+- precompiled gem build checks (`precompiled-gem-build`)
+- rust scaffold check (`rust-scaffold`)
+- benchmark guardrail check (`benchmark-guardrail`)
+- release validation/build/publish pipeline on `v*` tags (`release.yml`)
+- post-publish artifact verification (`verify_published_artifacts` in `release.yml`)
+- release-failure issue alerting (`release-failure-alert`)
+
+## Remaining Work Queue
+
+Priority 1:
+
+- continue periodic Rust-orchestration benchmark guardrail tightening (`docs/rust-orchestration-guardrails.md`) as performance data remains stable
+- keep hybrid backend compatibility guarantees (Rust + native + pure fallback) while extending Rust orchestration only behind parity/guardrail checks
+
+Priority 2:
+
+- monitor expanded precompiled release lanes (Windows + musl Linux) now that release dry-run matrix validation is green in run `22556487788`
+- evaluate any additional precompiled targets beyond current release-blocking set using `docs/precompiled-target-evaluation.md`
+
+Priority 3:
+
+- monitor release-failure auto-alert and auto-close behavior (`.github/workflows/release-failure-alert.yml`) and adjust signal/noise as release cadence grows
+
+## Resume Instructions For A New Conversation
+
+1. Check out `master`.
+2. Open this file first: `docs/modernization-handoff.md`.
+3. Run `SKIP_DOCKER=1 ./bin/release-preflight`.
+4. Review `docs/release-runbook.md` for release flow/rollback details.
+5. Validate precompiled packaging locally for your host:
+   - `./bin/release-precompiled-artifacts --tag "$(./bin/check-version-sync --print-tag)" --skip-preflight`
+6. Continue with remaining modernization queue (`Priority 1` then `Priority 2/3`).
+
+If you want the next assistant to continue immediately, use:
+
+"Open `docs/modernization-handoff.md`, validate with `SKIP_DOCKER=1 ./bin/release-preflight`, run `./bin/release-precompiled-artifacts --skip-preflight`, and continue the `Priority 1/2/3` modernization queue."

data/docs/porting-strategy.md

@@ -0,0 +1,148 @@
+# Ruby 3.2+/3.3+ Porting Strategy (Experimental)
+
+## Recommendation
+
+For this gem, the best long-term path is:
+
+1. **Keep the public Ruby API.**
+2. **Replace the handwritten C extension bindings with Rust + magnus** (Ruby native extension bindings for modern CRuby).
+3. **Ship a pure-Ruby fallback backend** for portability and easier debugging.
+
+This gives a practical balance between maintenance and speed:
+
+- Pure Ruby only: easiest to maintain, but likely much slower for training.
+- Modern C extension rewrite: fast, but still painful to maintain against Ruby internals.
+- Rust extension: native speed with safer memory management and cleaner FFI layer.
+
+## Why this path fits this project
+
+- The current project already has a stable Ruby-facing object model (`Lda::Lda`, corpus/document classes).
+- The expensive parts are numeric loops in inference/training, which benefit from native code.
+- Ruby 3.2+ compatibility is much easier to preserve with a modern binding layer than with legacy C wrapper patterns.
+
+## Proposed architecture
+
+- `Lda::Backends::Rust` (preferred in `auto` mode when extension loads)
+- `Lda::Backends::Native` (fallback in `auto` mode when Rust is unavailable)
+- `Lda::Backends::PureRuby` (always available)
+- `Lda::Lda` delegates heavy operations to the selected backend.
+
+Suggested backend selection:
+
+- `ENV['LDA_RUBY_BACKEND']=pure_ruby` → force Ruby backend.
+- default (`auto`) → try Rust backend, then native backend, then pure Ruby.
+
+## Migration plan
+
+### Current status
+
+Completed in `codex/experiment-ruby3-modernization`:
+
+- Phase 1 baseline test capture expanded with backend compatibility fixtures.
+- Phase 2 backend boundary extraction (`Lda::Lda` now delegates through backend adapters).
+- Phase 3 pure Ruby backend implementation (available as `backend: :pure` or `LDA_RUBY_BACKEND=pure`).
+- CI matrix added for Ruby 3.2/3.3 with native and pure backend jobs.
+- Phase 4 started with Rust extension scaffolding (`ext/lda-ruby-rust`) and backend mode wiring (`backend: :rust` when extension is available).
+- Rust kernels ported so far:
+  - batched per-iteration corpus inference
+  - batched per-document inference loop (EM inner updates)
+  - per-word topic-weight computation
+  - topic-term accumulation from per-document `phi`
+  - topic-term normalization and log-beta finalization in EM
+  - gamma convergence shift reduction between EM iterations
+  - topic-document average log-probability computation
+  - seeded topic-term initialization
+- Rust runtime CI job added (compile + execute rust backend tests).
+- Rust/Pure numeric parity fixtures added for deterministic seeded runs.
+- `compile_rust` now stages a Ruby-loadable extension artifact to avoid `Init_` symbol mismatch from Cargo's `lib*` output naming.
+- Rust-side EM orchestration path added (`Lda::RustBackend.run_em`) and retained as legacy compatibility fallback for precomputed beta-input execution.
+- Rust-side deterministic-start orchestration path added (`Lda::RustBackend.run_em_with_start`) so `seeded`/`deterministic` startup can stay in Rust.
+- Rust-side seed-controlled random-start orchestration path added (`Lda::RustBackend.run_em_with_start_seed` + `random_topic_term_probabilities`) so random initialization can stay in Rust while preserving deterministic replay from an explicit seed.
+- Rust-side corpus session lifecycle added (`create_corpus_session`/`drop_corpus_session`) and `Lda::Backends::Rust` now prefers session-based EM orchestration (`run_em_on_session_with_start_seed`) before array-based fallback paths.
+- Rust-side session settings lifecycle added (`configure_corpus_session`) and `Lda::Backends::Rust` now prefers settings-aware session orchestration (`run_em_on_session_start`) before parameter-heavy session and array-based fallbacks.
+- Rust session orchestration now runs on shared Rust-side corpus session data via borrowed execution helpers, avoiding deep corpus array cloning on each session EM call.
+- Unified Rust session API added (`run_em_on_session`) to apply settings and execute EM in one call inside Rust session orchestration.
+- `Lda::Backends::Rust` now prefers direct Rust non-session orchestration (`run_em_with_start_seed`) before legacy `run_em(initial_beta, ...)` compatibility fallback when a session path is unavailable.
+- Direct non-session Rust orchestration now reuses the backend's cached Rust corpus snapshot instead of rebuilding corpus arrays from `@corpus` on each fallback invocation.
+- Rust managed-session orchestration API added (`run_em_on_session_with_corpus`) to recreate missing sessions and execute EM in one Rust call.
+- Rust session lifecycle replacement API added (`replace_corpus_session`) so corpus reassignment can update existing Rust sessions in place (config reset + corpus swap) instead of Ruby-side drop/recreate.
+- `Lda::Backends::Rust` now routes cached-corpus EM through `run_em_on_session_with_corpus`, leaving session reuse/recovery decisions in Rust, preferring that managed path even when no active session id is cached locally, and reducing Ruby-side fallback branching when sessions are externally dropped.
+- `run_em_on_session_with_corpus` now acts as a unified Rust managed-corpus entrypoint: it attempts session-backed execution first, then falls back to direct start-aware array execution inside Rust when a managed session cannot be used.
+- Legacy `run_em(initial_beta, ...)` compatibility fallback now reuses the Rust backend's cached corpus snapshot and only relies on the pure-Ruby backend to synthesize the initial beta matrix.
+- Dockerized rust runtime workflow added for local parity with CI (`Dockerfile.rust`, `bin/docker-test-rust`).
+- Gem packaging now excludes local Rust cargo build artifacts (`target/**`) for clean release builds.
+- Backend benchmark driver added (`bin/benchmark-backends`) to track pure/native/rust runtime deltas.
+- Rust orchestration guardrail policy documented (`docs/rust-orchestration-guardrails.md`) with benchmark threshold checker (`bin/check-rust-benchmark`).
+- CI benchmark guardrail job added (`benchmark-guardrail`) to enforce Rust/pure runtime ratio on Ubuntu (currently `BENCH_RUST_TO_PURE_MAX_RATIO=0.045`).
+- Source install path now has explicit Rust build policy via `LDA_RUBY_RUST_BUILD=auto|always|never`.
+- Docker install-policy matrix script added (`bin/docker-test-install-policies`) to verify source install behavior across environments.
+- CI now runs install-policy matrix checks on Ubuntu.
+- Install-policy matrix now runs packaged-gem runtime smoke checks (auto/pure/native/rust mode selection + EM pipeline) to validate release-time fallback behavior.
+- Cross-OS packaged-gem fallback CI job added (`bin/test-packaged-gem-fallback`) to validate auto/never/always install policy semantics without Cargo.
+- Packaged-gem Rust-enabled CI job added (`bin/test-packaged-gem-rust-enabled`) to validate auto/never/always install policy semantics with Cargo available.
+- Packaged-gem manifest CI job added (`bin/test-packaged-gem-manifest`) to enforce release artifact contents and metadata.
+- Local release preflight command added (`bin/release-preflight`) to run unit + packaged-gem validation checks in one pass.
+- Version/tag sync guard added (`bin/check-version-sync`) to enforce parity between `VERSION.yml`, `lib/lda-ruby/version.rb`, and release tags.
+- Release preparation helper added (`bin/release-prepare`) for deterministic version/changelog updates.
+- Release artifact helper added (`bin/release-artifacts`) to build source gem artifacts with SHA256 checksums.
+- Precompiled platform artifact helper added (`bin/release-precompiled-artifacts`) to build + validate native gems.
+- Tag-driven release workflow added (`.github/workflows/release.yml`) with dry-run support and environment-gated publish jobs.
+- RubyGems credential preflight helper added (`bin/verify-rubygems-api-key`) for CI-safe publish key validation.
+- Post-publish verification helper added (`bin/verify-release-publish`) to validate RubyGems + GitHub release artifacts by tag.
+- CI precompiled guardrail job added (`precompiled-gem-build`) for full release-blocking platform packaging checks (Linux, Linux musl, macOS Intel, macOS Apple Silicon, Windows).
+- macOS precompiled CI/release lanes now pin Homebrew `llvm@18` (with fallback to `llvm`) and export `LIBCLANG_PATH` from the selected prefix to avoid bindgen breakage from Homebrew LLVM drift.
+- Release workflow post-publish verification job added (`verify_published_artifacts`).
+- Release failure alert workflow added (`.github/workflows/release-failure-alert.yml`) to open issue alerts for failed tag-triggered `release.yml` runs and auto-close matching alerts when reruns succeed.
+- Maintainer release runbook added (`docs/release-runbook.md`) with publish and rollback/yank procedures.
+- Precompiled platform support policy added (`docs/precompiled-platform-policy.md`).
+
+For an up-to-date resume snapshot (phase status + exact remaining queue), see `docs/modernization-handoff.md`.
+
+### Phase 1: Stabilize API and tests
+
+- Capture current behavior with golden tests around:
+  - corpus loading
+  - EM convergence hooks
+  - topic-word output format
+  - `top_words`, `top_word_indices`, and `phi` shape
+- Add CI matrix for Ruby 3.2, 3.3, and latest.
+
+### Phase 2: Extract backend boundary
+
+- Introduce backend interface in Ruby.
+- Keep all existing high-level classes and output methods.
+- Route existing calls through one backend object.
+
+### Phase 3: Add pure Ruby reference backend
+
+- Implement a simple, correct (not necessarily fast) Gibbs or variational inference path.
+- Use this backend in tests as the compatibility baseline.
+
+### Phase 4: Add Rust native backend
+
+- Implement performance-critical loops in Rust.
+- Expose only minimal Ruby-facing methods.
+- Verify parity against pure Ruby backend on deterministic fixtures.
+
+### Phase 5: Packaging and release
+
+- Phase 5A (source-gem release automation): complete.
+- Keep source build path available.
+- Phase 5B (precompiled/native gem publishing): complete for Linux/macOS/Windows/musl release matrix targets via `bin/release-precompiled-artifacts` and release workflow matrix builds.
+
+## Tooling suggestions
+
+- Ruby test framework: Minitest (already present).
+- Native extension: `magnus` + `rb_sys`.
+- Performance checks: benchmark script comparing legacy behavior vs pure Ruby vs Rust.
+- CI: GitHub Actions with matrix (`ubuntu`, `macos`) and Ruby 3.2/3.3/latest.
+
+## What not to do first
+
+- Do not start by rewriting all algorithms at once.
+- Do not couple file parsing and inference internals.
+- Do not rely on old Ruby C API macros that changed across versions.
+
+## Decision summary
+
+If the goal is a future-proof gem with acceptable speed and much lower maintenance pain, **use a hybrid model (Rust native backend + pure Ruby fallback)** instead of a full pure-Ruby rewrite or another large handwritten C extension.