lda-ruby 0.3.9 → 0.4.0

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    M2IzN2IwMmE1YWFhNTJjYjk0ZjMyYzhjMWQyZDdjZjY2YTA0OGNjZA==
-  data.tar.gz: !binary |-
-    OTNjZjE5MGNmOGI2YzY3YzhlNDRiYTBlNDM5NmUwYmY4Mjc2ZmNkNQ==
+SHA256:
+  metadata.gz: d0b97c33082528d2f992c4686e85e2080746938cd898c2a0662dd357979ad650
+  data.tar.gz: e4bd8e49f0b3f295b0f3ebfd78fc32db4835f24d93837ad566069b83046538c2
 SHA512:
-  metadata.gz: !binary |-
-    MDAxZmQ1N2U5MGQ1MDA2MGE4YTU0N2NkN2Y4N2Q4YzlmNTdmYTkxN2FlYzU0
-    ZmFjZDZiZWIwZmRiMDJlYjNjYTg5YWQ0N2RlOTY1MTg0YzZjZTc0NGQ0YmI1
-    ZDljMjljNTA3ODdlZjBjNjNjMTc0MGRlMjBhODQzZTg3YWM5OWE=
-  data.tar.gz: !binary |-
-    OGFiMzVhMzY4OTFmZTkzMmM3YjQxMzNkM2JlMjVjOTRjM2ZhNTc1YmUxZTRj
-    Y2M3YzZiNzNiYmVkNDI1ZTUxODhkMjhlYTQ4MmRhZjVkZjQxMDhjOTk5Njc4
-    OTZhOTM5ZTQ3OTFhY2U1YjRhODQyYzBkZWNlYzhjNzBjMWFlNGU=
+  metadata.gz: cebd90cdaca9d030379105509325aac12f457eb9da75f8e2fc8f2d618d1ecbf201dc6b4407ec28e8f95b1dda4a71ba864b37ea16ad38a22b7605f2b7e281f908
+  data.tar.gz: fb452cc42435ff9382a39f878e8d7de5e3825e1b800e1cca4c2bec7684294d10fdec960f11cefc3f18a8c6381552e14ff058ecebd50884f91115d65e93880825

data/CHANGELOG.md

@@ -1,3 +1,11 @@
+version 0.4.0
+=============
+
+- Ruby 3.2+/3.3+ modernization release.
+- Added Rust/native/pure backend selection and packaged-gem install policy validation.
+- Added tag-driven release automation and release runbook.
+- Added precompiled platform gem build/publish pipeline (Linux + macOS) and compatibility policy documentation.
+
 version 0.3.9
 =============
 

data/Gemfile

@@ -0,0 +1,9 @@
+source "https://rubygems.org"
+
+gemspec
+
+group :development, :test do
+  gem "rake", ">= 13.0"
+  gem "test-unit", ">= 3.6"
+  gem "shoulda-context", "~> 2.0"
+end

data/README.md

@@ -17,6 +17,116 @@ The original C code relied on files for the input and output. We felt it was nec
 
 If you have general questions about Latent Dirichlet Allocation, I urge you to use the [topic models mailing list][topic-models], since the people who monitor that are very knowledgeable.  If you encounter bugs specific to lda-ruby, please post an issue on the Github project.
 
+## Development
+
+### Local (Ruby 3.2+)
+
+```bash
+bundle install
+bundle exec rake test
+```
+
+### Docker (recommended for isolated setup)
+
+```bash
+./bin/docker-test
+```
+
+Rust backend runtime checks in Docker:
+
+```bash
+./bin/docker-test-rust
+```
+
+Install policy matrix checks in Docker:
+
+```bash
+./bin/docker-test-install-policies
+```
+
+For an interactive shell inside the dev container:
+
+```bash
+./bin/docker-shell
+```
+
+For an interactive shell with Rust toolchain + bindgen dependencies:
+
+```bash
+./bin/docker-shell-rust
+```
+
+### Build tasks
+
+- `bundle exec rake compile` builds the native extension.
+- `bundle exec rake compile_rust` builds the experimental Rust extension and stages a Ruby-loadable artifact (`lda_ruby_rust.<dlext>`).
+  - On macOS, build tasks automatically add the Rust linker flag for Ruby extension `dynamic_lookup`.
+- `bundle exec rake test` rebuilds the extension, then runs tests.
+- `bundle exec rake build` builds the gem package.
+- `bundle exec ruby -Ilib:test test/backend_compatibility_test.rb` runs backend compatibility fixtures.
+- `LDA_RUBY_BACKEND=rust bundle exec ruby -Ilib:test test/backend_compatibility_test.rb` runs parity checks in rust mode.
+- `./bin/benchmark-backends` benchmarks available backends (`pure`, `native`, `rust`) and prints JSON.
+- `./bin/docker-test-install-policies` verifies packaged-gem install behavior for `LDA_RUBY_RUST_BUILD=auto|always|never`, including runtime EM smoke checks.
+- `./bin/test-packaged-gem-fallback` verifies packaged-gem fallback behavior without Cargo (auto/never succeed, always fails) plus runtime smoke checks.
+- `./bin/test-packaged-gem-rust-enabled` verifies packaged-gem behavior with Cargo available (auto/always enable Rust, never disables Rust) plus runtime smoke checks.
+- `./bin/test-packaged-gem-manifest` verifies packaged-gem contents/metadata and rejects leaked build artifacts.
+- `./bin/release-preflight` runs unit tests + packaged-gem validation stack; set `SKIP_DOCKER=1` to skip Docker matrix checks.
+- `./bin/check-version-sync` verifies version parity between `VERSION.yml`, `lib/lda-ruby/version.rb`, and expected release tag.
+- `./bin/release-prepare 0.4.0` updates version/changelog files for a new release version.
+- `./bin/release-artifacts --tag v0.4.0` runs release checks, builds the source gem, and writes SHA256 checksums.
+- `./bin/release-precompiled-artifacts --tag v0.4.0 --platform x86_64-linux --skip-preflight` builds a precompiled platform gem and verifies install/runtime smoke checks.
+  - The `--platform` value must match the current host platform.
+
+Benchmark environment variables:
+- `BENCH_RUNS` (default: `3`)
+- `BENCH_START` (default: `seeded`)
+- `BENCH_TOPICS` (default: `8`)
+- `BENCH_MAX_ITER` (default: `20`)
+- `BENCH_EM_MAX_ITER` (default: `40`)
+
+### Install-time Rust build policy
+
+Source installs now run both extension setup scripts (`ext/lda-ruby/extconf.rb` and `ext/lda-ruby-rust/extconf.rb`).
+
+Rust build policy is controlled by `LDA_RUBY_RUST_BUILD`:
+- `auto` (default): build Rust extension if `cargo` is available, otherwise skip.
+- `always`: require Rust extension build and fail install if unavailable.
+- `never`: skip Rust extension build.
+
+Examples:
+- `LDA_RUBY_RUST_BUILD=always gem install lda-ruby`
+- `LDA_RUBY_RUST_BUILD=never bundle exec rake compile`
+
+### Precompiled platform gems
+
+Releases publish a source gem plus precompiled platform gems for:
+- `x86_64-linux`
+- `x86_64-darwin`
+- `arm64-darwin`
+
+On these platforms, installation should not require local C/Rust toolchains.
+Other platforms install from source gem and use the existing install-time fallback policy.
+
+For artifact strategy, compatibility targets, and rollout/deprecation rules, see `docs/precompiled-platform-policy.md`.
+
+### Backend selection
+
+- Default mode is `auto`: Rust backend when available, otherwise native extension, otherwise pure Ruby.
+- Force pure Ruby backend:
+  - `Lda::Lda.new(corpus, backend: :pure)`
+  - or `LDA_RUBY_BACKEND=pure`
+- Force native backend:
+  - `Lda::Lda.new(corpus, backend: :native)`
+- Force Rust backend (when extension is available):
+  - `Lda::Lda.new(corpus, backend: :rust)`
+  - or `LDA_RUBY_BACKEND=rust`
+
+`em("seeded")` is supported by both native and pure backends for deterministic fixture-oriented runs.
+
+Rust status: the extension hook layer is scaffolded in `ext/lda-ruby-rust`. Current Rust kernels include batched per-iteration corpus inference, batched per-document inference, topic-weights-per-word, topic-term-count accumulation, topic-term normalization/log-beta finalization, gamma-shift convergence reduction, topic-document average log-probability computation, and seeded topic-term initialization when `backend: :rust` is active; remaining model math still delegates to the pure Ruby backend. CI now runs dedicated rust-runtime checks and numeric parity fixtures against the pure backend.
+`compile_rust` and `LDA_RUBY_RUST_BUILD=always` require a Rust toolchain plus Ruby development headers and `libclang`.
+Gem packaging excludes local Rust build artifacts (`ext/lda-ruby-rust/target/**`) so local cargo outputs do not leak into published gems.
+
 ## Resources
 
 + [Blog post about LDA-Ruby][lda-ruby]
@@ -29,9 +139,19 @@ If you have general questions about Latent Dirichlet Allocation, I urge you to u
 Blei, David M., Ng, Andrew Y., and Jordan, Michael I. 2003. Latent dirichlet allocation. Journal of Machine Learning Research. 3 (Mar. 2003), 993-1022 [[pdf][pdf]].
 
 [svmlight]: http://svmlight.joachims.org
-[lda-ruby]: http://mendicantbug.com/2008/11/17/lda-in-ruby/
-[blei]: http://www.cs.princeton.edu/~blei/lda-c/
+[lda-ruby]: http://web.archive.org/web/20120616115448/http://mendicantbug.com/2008/11/17/lda-in-ruby/
+[blei]: http://web.archive.org/web/20161126004857/http://www.cs.princeton.edu/~blei/lda-c/
 [wikipedia]: http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation
-[ap-data]: http://www.cs.princeton.edu/~blei/lda-c/ap.tgz
+[ap-data]: http://web.archive.org/web/20160507090044/http://www.cs.princeton.edu/~blei/lda-c/ap.tgz
 [pdf]: http://www.cs.princeton.edu/picasso/mats/BleiNgJordan2003_blei.pdf
 [topic-models]: https://lists.cs.princeton.edu/mailman/listinfo/topic-models
+
+## Modernization
+
+For a Ruby 3.2+/3.3+ porting proposal, see `docs/porting-strategy.md`.
+
+For the latest implementation status and exact resume instructions, see `docs/modernization-handoff.md`.
+
+For release steps and rollback guidance, see `docs/release-runbook.md`.
+
+For precompiled gem strategy and compatibility policy, see `docs/precompiled-platform-policy.md`.

data/VERSION.yml

@@ -1,5 +1,5 @@
 ---
 :major: 0
-:minor: 3
-:patch: 9
-:build: 
+:minor: 4
+:patch: 0
+:build:

data/docs/modernization-handoff.md

@@ -0,0 +1,190 @@
+# Modernization Handoff (Resume Guide)
+
+This document is the canonical handoff state for continuing the Ruby 3.2+/3.3+ modernization in a new conversation.
+
+## Snapshot
+
+- Snapshot date: 2026-02-25
+- Active branch: `codex/modernization`
+- Branch head at snapshot: `7f9b101` (`Fix macOS Rust extension linking for precompiled builds`)
+- Repo status at snapshot: clean working tree on `codex/modernization` (in sync with `origin/codex/modernization`)
+- Latest release dry-run validation (GitHub Actions):
+  - date: 2026-02-25
+  - workflow run: `release.yml` run `22382692416` (`workflow_dispatch`, `publish=false`)
+  - result: success (`validate release candidate`, `build release artifacts`, and all `build precompiled artifacts` matrix targets)
+  - publish jobs skipped by design (`publish=false`)
+- Open pull request:
+  - `codex/modernization` -> `master`
+  - PR `#18`: `https://github.com/ealdent/lda-ruby/pull/18`
+- Latest PR CI validation (GitHub Actions):
+  - date: 2026-02-25
+  - workflow run: `CI` run `22383301379` (trigger: `pull_request` for PR `#18`)
+  - result: success (all checks green, including `precompiled gem build` matrix and install policy/rust runtime jobs)
+
+## Project Goal
+
+Modernize `lda-ruby` for Ruby 3.2+/3.3+ with:
+
+- stable Ruby API compatibility
+- high-performance Rust backend for hot paths
+- pure Ruby fallback for portability
+- reliable packaging and release validation
+
+## Current Backend Behavior
+
+- `auto` selection order: `rust` -> `native` -> `pure_ruby`
+- `LDA_RUBY_BACKEND` override supported for `rust`, `native`, and `pure`
+- Rust build policy for source installs: `LDA_RUBY_RUST_BUILD=auto|always|never`
+
+## Phase Status
+
+### Phase 1 (API/tests stabilization)
+
+Status: complete.
+
+Delivered:
+
+- expanded compatibility fixtures
+- CI test coverage for multiple backend modes
+
+### Phase 2 (backend boundary extraction)
+
+Status: complete.
+
+Delivered:
+
+- `Lda::Lda` delegates through backend adapters
+- backend selection normalized through `Lda::Backends.build`
+
+### Phase 3 (pure Ruby reference backend)
+
+Status: complete.
+
+Delivered:
+
+- full pure Ruby backend path with EM/model outputs
+- pure backend compatibility in tests
+
+### Phase 4 (Rust native backend)
+
+Status: mostly complete.
+
+Delivered:
+
+- Rust extension scaffold with magnus/rb_sys
+- Rust kernels for the main hot loops:
+  - corpus iteration
+  - document inference
+  - topic weights
+  - topic-term accumulation
+  - topic-term finalization (`beta`/`log(beta)`)
+  - gamma shift reduction
+  - topic-document probability
+  - seeded initialization
+- trusted kernel-output fast path enabled in rust mode
+- parity/compatibility test coverage and rust runtime CI
+
+Open in Phase 4:
+
+- optional deeper Rust ownership of orchestration logic (current design still intentionally delegates control flow through Ruby fallback scaffolding)
+
+### Phase 5 (packaging/release)
+
+Status: Phase 5A complete (source-gem release automation), Phase 5B complete for initial Linux/macOS precompiled gems.
+
+Delivered:
+
+- clean gem file filtering (no local cargo/native artifacts)
+- Docker install-policy matrix checks
+- packaged gem runtime checks without Cargo (`bin/test-packaged-gem-fallback`)
+- packaged gem runtime checks with Cargo (`bin/test-packaged-gem-rust-enabled`)
+- packaged gem manifest/metadata gate (`bin/test-packaged-gem-manifest`)
+- single-command local gate (`bin/release-preflight`)
+- version/tag parity guard (`bin/check-version-sync`)
+- deterministic release preparation helper (`bin/release-prepare`)
+- release artifact builder with checksum output (`bin/release-artifacts`)
+- precompiled artifact builder + runtime validator (`bin/release-precompiled-artifacts`)
+- gemspec precompiled variant support (`LDA_RUBY_GEM_VARIANT=precompiled`)
+- precompiled platform compatibility/publish policy (`docs/precompiled-platform-policy.md`)
+- macOS Rust build linker guardrail (`dynamic_lookup`) for precompiled packaging paths
+- tag-driven release workflow (`.github/workflows/release.yml`)
+- maintainer release runbook (`docs/release-runbook.md`)
+- CI jobs for packaged-gem fallback, rust-enabled checks, and manifest checks
+- CI precompiled gem build guardrail job (`precompiled-gem-build`)
+- release workflow matrix for precompiled gems:
+  - `x86_64-linux`
+  - `x86_64-darwin`
+  - `arm64-darwin`
+
+Open in Phase 5:
+
+- optional expansion of precompiled targets (for example Windows and/or musl Linux)
+- tighter post-publish verification/alerting for multi-artifact release runs
+
+## Validation Commands
+
+Core:
+
+- `./bin/docker-test`
+- `./bin/docker-test-rust`
+
+Packaging/release checks:
+
+- `./bin/check-version-sync`
+- `./bin/test-packaged-gem-manifest`
+- `./bin/test-packaged-gem-fallback`
+- `./bin/test-packaged-gem-rust-enabled`
+- `SKIP_DOCKER=1 ./bin/release-preflight`
+- `./bin/release-artifacts --tag v0.4.0`
+- `./bin/release-precompiled-artifacts --tag v0.4.0 --skip-preflight`
+
+Optional full Docker matrix:
+
+- `./bin/docker-test-install-policies`
+
+Performance tracking:
+
+- `./bin/benchmark-backends`
+
+## CI Jobs Expected
+
+- native tests (`test-native`)
+- pure backend tests (`test-pure`)
+- rust runtime tests (`rust-runtime`)
+- Docker install policy matrix (`install-policy-matrix`)
+- packaged gem fallback checks (`packaged-gem-fallback`)
+- packaged gem rust-enabled checks (`packaged-gem-rust-enabled`)
+- packaged gem manifest checks (`packaged-gem-manifest`)
+- precompiled gem build checks (`precompiled-gem-build`)
+- rust scaffold check (`rust-scaffold`)
+- release validation/build/publish pipeline on `v*` tags (`release.yml`)
+
+## Remaining Work Queue
+
+Priority 1:
+
+- decide whether to keep current hybrid rust-kernel architecture or move more orchestration into Rust
+- if moving deeper into Rust, define parity guardrails and benchmark thresholds before refactors
+
+Priority 2:
+
+- evaluate additional precompiled targets (Windows and/or musl Linux)
+- add explicit post-publish verification checks for all uploaded release artifacts
+
+Priority 3:
+
+- define automated alerts/notifications for release artifact publish failures
+
+## Resume Instructions For A New Conversation
+
+1. Check out `codex/modernization`.
+2. Open this file first: `docs/modernization-handoff.md`.
+3. Run `SKIP_DOCKER=1 ./bin/release-preflight`.
+4. Review `docs/release-runbook.md` for release flow/rollback details.
+5. Validate precompiled packaging locally for your host:
+   - `./bin/release-precompiled-artifacts --tag "$(./bin/check-version-sync --print-tag)" --skip-preflight`
+6. Continue with `Priority 1` items under "Remaining Work Queue".
+
+If you want the next assistant to continue immediately, use:
+
+"Open `docs/modernization-handoff.md`, validate with `SKIP_DOCKER=1 ./bin/release-preflight`, run `./bin/release-precompiled-artifacts --skip-preflight`, and continue the remaining modernization queue."

data/docs/porting-strategy.md

@@ -0,0 +1,127 @@
+# Ruby 3.2+/3.3+ Porting Strategy (Experimental)
+
+## Recommendation
+
+For this gem, the best long-term path is:
+
+1. **Keep the public Ruby API.**
+2. **Replace the handwritten C extension bindings with Rust + magnus** (Ruby native extension bindings for modern CRuby).
+3. **Ship a pure-Ruby fallback backend** for portability and easier debugging.
+
+This gives a practical balance between maintenance and speed:
+
+- Pure Ruby only: easiest to maintain, but likely much slower for training.
+- Modern C extension rewrite: fast, but still painful to maintain against Ruby internals.
+- Rust extension: native speed with safer memory management and cleaner FFI layer.
+
+## Why this path fits this project
+
+- The current project already has a stable Ruby-facing object model (`Lda::Lda`, corpus/document classes).
+- The expensive parts are numeric loops in inference/training, which benefit from native code.
+- Ruby 3.2+ compatibility is much easier to preserve with a modern binding layer than with legacy C wrapper patterns.
+
+## Proposed architecture
+
+- `Lda::Backends::Rust` (preferred in `auto` mode when extension loads)
+- `Lda::Backends::Native` (fallback in `auto` mode when Rust is unavailable)
+- `Lda::Backends::PureRuby` (always available)
+- `Lda::Lda` delegates heavy operations to the selected backend.
+
+Suggested backend selection:
+
+- `ENV['LDA_RUBY_BACKEND']=pure_ruby` → force Ruby backend.
+- default (`auto`) → try Rust backend, then native backend, then pure Ruby.
+
+## Migration plan
+
+### Current status
+
+Completed in `codex/experiment-ruby3-modernization`:
+
+- Phase 1 baseline test capture expanded with backend compatibility fixtures.
+- Phase 2 backend boundary extraction (`Lda::Lda` now delegates through backend adapters).
+- Phase 3 pure Ruby backend implementation (available as `backend: :pure` or `LDA_RUBY_BACKEND=pure`).
+- CI matrix added for Ruby 3.2/3.3 with native and pure backend jobs.
+- Phase 4 started with Rust extension scaffolding (`ext/lda-ruby-rust`) and backend mode wiring (`backend: :rust` when extension is available).
+- Rust kernels ported so far:
+  - batched per-iteration corpus inference
+  - batched per-document inference loop (EM inner updates)
+  - per-word topic-weight computation
+  - topic-term accumulation from per-document `phi`
+  - topic-term normalization and log-beta finalization in EM
+  - gamma convergence shift reduction between EM iterations
+  - topic-document average log-probability computation
+  - seeded topic-term initialization
+- Rust runtime CI job added (compile + execute rust backend tests).
+- Rust/Pure numeric parity fixtures added for deterministic seeded runs.
+- `compile_rust` now stages a Ruby-loadable extension artifact to avoid `Init_` symbol mismatch from Cargo's `lib*` output naming.
+- Dockerized rust runtime workflow added for local parity with CI (`Dockerfile.rust`, `bin/docker-test-rust`).
+- Gem packaging now excludes local Rust cargo build artifacts (`target/**`) for clean release builds.
+- Backend benchmark driver added (`bin/benchmark-backends`) to track pure/native/rust runtime deltas.
+- Source install path now has explicit Rust build policy via `LDA_RUBY_RUST_BUILD=auto|always|never`.
+- Docker install-policy matrix script added (`bin/docker-test-install-policies`) to verify source install behavior across environments.
+- CI now runs install-policy matrix checks on Ubuntu.
+- Install-policy matrix now runs packaged-gem runtime smoke checks (auto/pure/native/rust mode selection + EM pipeline) to validate release-time fallback behavior.
+- Cross-OS packaged-gem fallback CI job added (`bin/test-packaged-gem-fallback`) to validate auto/never/always install policy semantics without Cargo.
+- Packaged-gem Rust-enabled CI job added (`bin/test-packaged-gem-rust-enabled`) to validate auto/never/always install policy semantics with Cargo available.
+- Packaged-gem manifest CI job added (`bin/test-packaged-gem-manifest`) to enforce release artifact contents and metadata.
+- Local release preflight command added (`bin/release-preflight`) to run unit + packaged-gem validation checks in one pass.
+- Version/tag sync guard added (`bin/check-version-sync`) to enforce parity between `VERSION.yml`, `lib/lda-ruby/version.rb`, and release tags.
+- Release preparation helper added (`bin/release-prepare`) for deterministic version/changelog updates.
+- Release artifact helper added (`bin/release-artifacts`) to build source gem artifacts with SHA256 checksums.
+- Precompiled platform artifact helper added (`bin/release-precompiled-artifacts`) to build + validate native gems.
+- Tag-driven release workflow added (`.github/workflows/release.yml`) with dry-run support and environment-gated publish jobs.
+- CI precompiled guardrail job added (`precompiled-gem-build`) for Linux/macOS packaging checks.
+- Maintainer release runbook added (`docs/release-runbook.md`) with publish and rollback/yank procedures.
+- Precompiled platform support policy added (`docs/precompiled-platform-policy.md`).
+
+For an up-to-date resume snapshot (phase status + exact remaining queue), see `docs/modernization-handoff.md`.
+
+### Phase 1: Stabilize API and tests
+
+- Capture current behavior with golden tests around:
+  - corpus loading
+  - EM convergence hooks
+  - topic-word output format
+  - `top_words`, `top_word_indices`, and `phi` shape
+- Add CI matrix for Ruby 3.2, 3.3, and latest.
+
+### Phase 2: Extract backend boundary
+
+- Introduce backend interface in Ruby.
+- Keep all existing high-level classes and output methods.
+- Route existing calls through one backend object.
+
+### Phase 3: Add pure Ruby reference backend
+
+- Implement a simple, correct (not necessarily fast) Gibbs or variational inference path.
+- Use this backend in tests as the compatibility baseline.
+
+### Phase 4: Add Rust native backend
+
+- Implement performance-critical loops in Rust.
+- Expose only minimal Ruby-facing methods.
+- Verify parity against pure Ruby backend on deterministic fixtures.
+
+### Phase 5: Packaging and release
+
+- Phase 5A (source-gem release automation): complete.
+- Keep source build path available.
+- Phase 5B (precompiled/native gem publishing): complete for initial Linux/macOS targets via `bin/release-precompiled-artifacts` and release workflow matrix builds.
+
+## Tooling suggestions
+
+- Ruby test framework: Minitest (already present).
+- Native extension: `magnus` + `rb_sys`.
+- Performance checks: benchmark script comparing legacy behavior vs pure Ruby vs Rust.
+- CI: GitHub Actions with matrix (`ubuntu`, `macos`) and Ruby 3.2/3.3/latest.
+
+## What not to do first
+
+- Do not start by rewriting all algorithms at once.
+- Do not couple file parsing and inference internals.
+- Do not rely on old Ruby C API macros that changed across versions.
+
+## Decision summary
+
+If the goal is a future-proof gem with acceptable speed and much lower maintenance pain, **use a hybrid model (Rust native backend + pure Ruby fallback)** instead of a full pure-Ruby rewrite or another large handwritten C extension.

data/docs/precompiled-platform-policy.md

@@ -0,0 +1,68 @@
+# Precompiled Platform Gem Policy (Phase 5B)
+
+This document defines the publish strategy and compatibility policy for `lda-ruby` precompiled gems.
+
+## Artifact Strategy
+
+Each release version publishes a split package set:
+
+- Source gem: `lda-ruby-<version>.gem`
+- Precompiled platform gems:
+  - `lda-ruby-<version>-x86_64-linux.gem`
+  - `lda-ruby-<version>-x86_64-darwin.gem`
+  - `lda-ruby-<version>-arm64-darwin.gem`
+
+The source gem remains the universal fallback. Platform gems are additive and are expected to install without local build tools.
+Precompiled artifacts are built on matching host runners (no cross-compilation in current workflow).
+
+## Compatibility Policy
+
+- Supported Ruby versions: 3.2 and 3.3 (plus future versions validated by CI).
+- Release-blocking precompiled targets:
+  - Linux `x86_64-linux`
+  - macOS Intel `x86_64-darwin`
+  - macOS Apple Silicon `arm64-darwin`
+- Other platforms:
+  - Install from source gem.
+  - Runtime remains supported through native/pure fallback paths.
+
+Backend behavior expectations:
+
+- Platform gem install:
+  - `auto` backend resolves to `rust` by default.
+  - `native` and `pure` overrides continue to work.
+- Source gem install:
+  - Rust build policy is controlled by `LDA_RUBY_RUST_BUILD=auto|always|never`.
+  - If Rust build is skipped/unavailable, `auto` falls back to `native`, then `pure_ruby`.
+
+## Guardrails
+
+Validation must pass before publish:
+
+- `./bin/release-preflight` (source-gem checks).
+- `./bin/release-precompiled-artifacts --platform <target>` for each release-blocking platform.
+
+Release automation requirements:
+
+- `.github/workflows/release.yml` builds source + precompiled artifacts.
+- Release workflow matrix must include all release-blocking precompiled targets.
+- Publish jobs push all built gems and attach checksums to GitHub releases.
+
+Continuous integration guardrail:
+
+- `.github/workflows/ci.yml` runs `release-precompiled-artifacts` for representative Linux/macOS targets on every branch/PR.
+
+## Rollout / Expansion Rules
+
+When adding a new precompiled platform:
+
+1. Add target to release workflow matrix.
+2. Add or update CI coverage for that platform family.
+3. Update this policy and the release runbook support matrix.
+4. Validate a dry-run release with `workflow_dispatch` before shipping.
+
+When deprecating a precompiled platform:
+
+1. Remove platform from release matrix.
+2. Keep source-gem path available unless the overall platform support policy changes.
+3. Document deprecation in `CHANGELOG.md` and release notes.