lda-ruby 0.5.0-x64-mingw-ucrt

data/ext/lda-ruby-rust/README.md

@@ -0,0 +1,73 @@
+# Experimental Rust Extension Scaffold
+
+This directory contains an experimental Rust extension scaffold built with `magnus`.
+
+Current scope:
+
+- Defines `Lda::RustBackend` module in Ruby.
+- Exposes capability hooks:
+  - `Lda::RustBackend.available?`
+  - `Lda::RustBackend.abi_version`
+  - `Lda::RustBackend.corpus_session_count`
+  - `Lda::RustBackend.corpus_session_exists(session_id)`
+  - `Lda::RustBackend.before_em(start, num_docs, num_terms)`
+  - `Lda::RustBackend.topic_weights_for_word(beta, gamma, word_index, min_probability)`
+  - `Lda::RustBackend.accumulate_topic_term_counts(topic_term_counts, phi_d, words, counts)`
+  - `Lda::RustBackend.infer_document(beta, gamma_initial, words, counts, max_iter, convergence, min_probability, init_alpha)`
+  - `Lda::RustBackend.infer_corpus_iteration(beta, document_words, document_counts, max_iter, convergence, min_probability, init_alpha)`
+  - `Lda::RustBackend.normalize_topic_term_counts(topic_term_counts, min_probability)`
+  - `Lda::RustBackend.average_gamma_shift(previous_gamma, current_gamma)`
+  - `Lda::RustBackend.topic_document_probability(phi_tensor, document_counts, num_topics, min_probability)`
+  - `Lda::RustBackend.seeded_topic_term_probabilities(document_words, document_counts, topics, terms, min_probability)`
+  - `Lda::RustBackend.random_topic_term_probabilities(topics, terms, min_probability, random_seed)`
+  - `Lda::RustBackend.create_corpus_session(document_words, document_counts, terms)`
+  - `Lda::RustBackend.replace_corpus_session(session_id, document_words, document_counts, terms)`
+  - `Lda::RustBackend.drop_corpus_session(session_id)`
+  - `Lda::RustBackend.configure_corpus_session(session_id, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability)`
+  - `Lda::RustBackend.run_em(initial_beta, document_words, document_counts, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability)`
+  - `Lda::RustBackend.run_em_with_start(start, document_words, document_counts, topics, terms, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability)`
+  - `Lda::RustBackend.run_em_with_start_seed(start, document_words, document_counts, topics, terms, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+  - `Lda::RustBackend.run_em_on_session(session_id, start, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+  - `Lda::RustBackend.run_em_on_session_start(session_id, start, random_seed)`
+  - `Lda::RustBackend.run_em_on_session_with_start_seed(session_id, start, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+  - `Lda::RustBackend.run_em_on_session_with_corpus(session_id, document_words, document_counts, terms, start, topics, max_iter, convergence, em_max_iter, em_convergence, init_alpha, min_probability, random_seed)`
+
+Hot-path kernels currently executed in Rust when `backend: :rust` is active:
+- topic weights for a word across topics
+- topic-term count accumulation from per-document `phi`
+- full per-document inference loop (batched inner EM updates)
+- full per-iteration corpus inference (batched document processing)
+- topic-term normalization and log-probability finalization for EM beta updates
+- gamma convergence shift reduction between EM iterations
+- topic-document average log-probability computation
+- seeded topic-term initialization
+- random topic-term initialization with explicit seed control
+- EM outer-loop orchestration with convergence checks (`run_em`)
+- start-aware deterministic EM orchestration (`run_em_with_start` for `seeded`/`deterministic`)
+- start-aware seeded and random EM orchestration with explicit seed control (`run_em_with_start_seed`)
+- unified session-settings orchestration (`run_em_on_session`) that applies settings and executes EM in one call
+- session-based EM orchestration against Rust-managed corpus lifecycle (`create_corpus_session` + `run_em_on_session_with_start_seed`)
+- settings-aware session orchestration (`configure_corpus_session` + `run_em_on_session_start`)
+- managed corpus orchestration (`run_em_on_session_with_corpus`) that can recreate missing sessions and, if session-backed execution cannot be used, falls back internally to direct start-aware execution inside Rust
+- `Lda::Backends::Rust` prefers `run_em_on_session_with_corpus` whenever a cached Rust corpus snapshot is available, even if no session id is currently cached locally
+- direct and legacy beta-input compatibility fallbacks both reuse the backend's cached Rust corpus snapshot instead of rebuilding corpus arrays in Ruby
+- unknown EM start modes in seed-aware orchestration follow Ruby's non-seeded fallback behavior (seeded by explicit `random_seed`)
+
+Remaining numeric LDA kernels are still provided by the pure Ruby backend and will move incrementally.
+
+## Local build (optional)
+
+```bash
+cd ext/lda-ruby-rust
+cargo build --release
+```
+
+Then run Ruby with `require "lda_ruby_rust"` available on load path.
+
+## Install-time policy
+
+During source gem installs, `ext/lda-ruby-rust/extconf.rb` can optionally build this extension.
+
+- `LDA_RUBY_RUST_BUILD=auto` (default): build when `cargo` is available.
+- `LDA_RUBY_RUST_BUILD=always`: require a successful Rust build or fail installation.
+- `LDA_RUBY_RUST_BUILD=never`: always skip Rust build.

data/ext/lda-ruby-rust/extconf.rb

@@ -0,0 +1,135 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "rbconfig"
+
+require_relative "../../lib/lda-ruby/rust_build_policy"
+
+module Lda
+  module RustExtensionBuild
+    module_function
+
+    def run
+      policy = RustBuildPolicy.resolve
+      puts("Rust extension build policy: #{policy} (#{RustBuildPolicy::ENV_KEY})")
+
+      case policy
+      when RustBuildPolicy::NEVER
+        puts("Skipping Rust extension build (policy=#{RustBuildPolicy::NEVER}).")
+      when RustBuildPolicy::ALWAYS
+        ensure_cargo_available!
+        build_and_stage!
+      else
+        if cargo_available?
+          build_and_stage!
+        else
+          puts("cargo not found; skipping Rust extension build (policy=#{RustBuildPolicy::AUTO}).")
+        end
+      end
+
+      write_noop_makefile
+    rescue StandardError => e
+      if policy == RustBuildPolicy::ALWAYS
+        abort("Rust extension build failed with #{RustBuildPolicy::ENV_KEY}=#{RustBuildPolicy::ALWAYS}: #{e.message}")
+      end
+
+      warn("Rust extension build skipped after error in auto mode: #{e.message}")
+      write_noop_makefile
+    end
+
+    def ensure_cargo_available!
+      return if cargo_available?
+
+      abort("cargo not found in PATH but #{RustBuildPolicy::ENV_KEY}=#{RustBuildPolicy::ALWAYS} was requested.")
+    end
+
+    def cargo_available?
+      cargo = ENV.fetch("CARGO", "cargo")
+      system(cargo, "--version", out: File::NULL, err: File::NULL)
+    end
+
+    def build_and_stage!
+      cargo = ENV.fetch("CARGO", "cargo")
+      Dir.chdir(__dir__) do
+        env = rust_build_env
+        success =
+          if env.empty?
+            system(cargo, "build", "--release")
+          else
+            system(env, cargo, "build", "--release")
+          end
+        success or raise "cargo build --release failed"
+      end
+
+      source = rust_cdylib_source
+      raise "Rust extension artifact not found at #{rust_cdylib_candidates.join(', ')}" unless source
+
+      destination = File.expand_path("../../lib/lda_ruby_rust.#{RbConfig::CONFIG.fetch('DLEXT')}", __dir__)
+      FileUtils.cp(source, destination)
+      puts("Staged Rust extension to #{destination}")
+    end
+
+    def rust_cdylib_source
+      rust_cdylib_candidates.find { |path| File.exist?(path) }
+    end
+
+    def rust_cdylib_candidates
+      rust_cdylib_filenames.map { |filename| File.join(__dir__, "target", "release", filename) }
+    end
+
+    def rust_cdylib_filenames
+      host_os = RbConfig::CONFIG.fetch("host_os")
+      case host_os
+      when /mswin|mingw|cygwin/
+        # On Windows cargo may emit either prefixed or unprefixed DLL names.
+        ["lda_ruby_rust.dll", "liblda_ruby_rust.dll"]
+      else
+        extension =
+          case host_os
+          when /darwin/
+            "dylib"
+          else
+            "so"
+          end
+
+        ["liblda_ruby_rust.#{extension}"]
+      end
+    end
+
+    def rust_build_env
+      host_os = RbConfig::CONFIG.fetch("host_os")
+      return {} unless host_os.match?(/darwin/)
+
+      dynamic_lookup_flag = "-C link-arg=-Wl,-undefined,dynamic_lookup"
+      existing = ENV.fetch("RUSTFLAGS", "")
+      merged =
+        if existing.include?(dynamic_lookup_flag)
+          existing
+        else
+          [existing, dynamic_lookup_flag].reject(&:empty?).join(" ")
+        end
+
+      { "RUSTFLAGS" => merged }
+    end
+
+    def write_noop_makefile
+      File.write(
+        File.join(__dir__, "Makefile"),
+        <<~MAKEFILE
+          all:
+          \t@echo "Rust extension handled by extconf.rb"
+
+          install:
+          \t@echo "Rust extension handled by extconf.rb"
+
+          clean:
+          \t@true
+
+          distclean: clean
+        MAKEFILE
+      )
+    end
+  end
+end
+
+Lda::RustExtensionBuild.run

data/ext/lda-ruby-rust/include/strings.h

@@ -0,0 +1,35 @@
+#ifndef LDA_RUBY_BINDGEN_STRINGS_H
+#define LDA_RUBY_BINDGEN_STRINGS_H
+
+#include <string.h>
+
+/*
+ * RubyInstaller headers may include <strings.h> on Windows, but Clang-based
+ * bindgen runs can miss that header in this environment. Provide compatibility
+ * aliases for bindgen preprocessing.
+ */
+#if defined(_WIN32) && !defined(__MINGW32__)
+#ifndef bzero
+#define bzero(ptr, size) memset((ptr), 0, (size))
+#endif
+#ifndef bcmp
+#define bcmp(a, b, n) memcmp((a), (b), (n))
+#endif
+#ifndef bcopy
+#define bcopy(src, dst, n) memmove((dst), (src), (n))
+#endif
+#ifndef index
+#define index strchr
+#endif
+#ifndef rindex
+#define rindex strrchr
+#endif
+#ifndef strcasecmp
+#define strcasecmp _stricmp
+#endif
+#ifndef strncasecmp
+#define strncasecmp _strnicmp
+#endif
+#endif
+
+#endif