pgn2 1.4.0 → 2.0.0

data/docs/superpowers/specs/2026-08-13-rust-bitboard-perft-design.md

@@ -0,0 +1,270 @@
+# Rust Bitboard Perft Backend (Magic Bitboards) — Design
+
+> **Status (2026-08-14):** The hand-rolled magic/pext engine described
+> below has been replaced by a thin adapter over the `chessie` crate
+> (MPL-2.0). See
+> `docs/superpowers/plans/2026-08-14-chessie-migration.md`. The "Engine
+> details (magic bitboards)" section is retained as the historical
+> design rationale for the since-removed hand-rolled engine; the shipped
+> engine is now `chessie` via the `pgn2-bitboard` adapter. The Ruby API
+> surface, FEN-keyed boundary, and global constraints below are
+> unchanged.
+
+**Goal:** Fast perft numbers, as the primary objective. A thin
+legal-move Ruby API is a secondary byproduct. Ship as a *required
+compiled Rust extension* distributed via **precompiled platform gems**
+(`rb_sys` + `magnus` + `rake-compiler-dock`), with source-build for
+local development. No pure-Ruby fallback for the shipped path.
+
+> **Decision context:** This supersedes the older Group 4 idea of an
+> optional bitboard/C backend with fallback. The chosen path is **3b**
+> (precompiled platform gems). **3a** (source build inside Docker) is
+> the documented interim until the prebuilt-gem CI is live.
+
+## Scope and non-goals
+
+- **Does NOT touch** the existing pure-Ruby 0x88 `Board` /
+  `MoveCalculator` / `Notation`. The Rust engine is a separate
+  bitboard engine keyed by FEN. Existing byte-identical FEN/PGN
+  guarantees stay intact.
+- **Does NOT replace** the in-progress `perf/attack-masks` work. That
+  branch is a pure-Ruby mailbox (knight/king offset tables) perf track
+  for the SAN/replay path, explicitly "no native." Magic bitboards live
+  in Rust and are decoupled — related only conceptually (both precompute
+  attack tables), not in code.
+- **NOT a full chess library:** no evaluation, no search, no UCI, no
+  Chess960, no tablebases. Perft-able move generator + make/unmake +
+  legal-move filter only.
+- Perft is verified against **published suites** (initial position,
+  Kiwipete, positions 3–6) to known node counts — not against the Ruby
+  engine, which has no legal-move API to cross-check against.
+
+## Architecture
+
+A Rust workspace at `ext/pgn2_native/` with two crates:
+
+1. `pgn2-bitboard` (**lib**, no Ruby dependency): a thin adapter over
+   the `chessie` crate (MPL-2.0). `Board` wraps `chessie::Game`; `perft`
+   and `legal_moves` delegate; `Move`/`MoveList`/`uci_parse` expose the
+   small surface the binding consumes. No chess logic lives in this
+   crate — `chessie` is the engine. **Unit-testable in pure Rust**
+   (`cargo test`) against published perft values — no Ruby in the loop.
+
+2. `pgn2_native` (**cdylib**): the magnus bindings, the only crate that
+   touches Ruby. Exposes `PGN::Bitboard::Engine` (constructed from a FEN
+   string) with:
+   - `#perft(depth) -> Integer`
+   - `#legal_moves -> Array<String>` (UCI: `e2e4`, `e1g1` for castling,
+     `e7e8q` for promotion)
+   - `#legal?(move) -> bool`
+
+**Boundary principle:** the engine crate knows nothing about Ruby; the
+binding crate is thin and is the only thing that touches magnus. This
+keeps chess logic testable without Ruby and the bridge trivial/fast.
+
+**Workspace layout (draft):**
+
+```
+ext/pgn2_native/
+├── Cargo.toml           # workspace manifest
+├── extconf.rb           # Ruby build entry point
+├── pgn2-bitboard/
+│   ├── Cargo.toml       # crate-type = ["lib"]
+│   └── src/
+│       ├── lib.rs
+│       ├── board.rs
+│       ├── moves.rs
+│       ├── magics.rs
+│       └── perft_tests.rs
+└── pgn2_native/
+    ├── Cargo.toml       # crate-type = ["cdylib"]
+    └── src/
+        └── lib.rs       # magnus init + class bindings
+```
+
+```toml
+# ext/pgn2_native/Cargo.toml
+[workspace]
+members = ["pgn2-bitboard", "pgn2_native"]
+resolver = "2"
+```
+
+```toml
+# ext/pgn2_native/pgn2_native/Cargo.toml
+[package]
+name = "pgn2_native"
+version = "0.1.0"
+edition = "2021"
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+magnus = "0.8"
+pgn2-bitboard = { path = "../pgn2-bitboard" }
+```
+
+## Ruby API surface (minimal)
+
+- `PGN::Bitboard::Engine.new(fen).perft(depth) -> Integer` — primary.
+- `PGN::Bitboard::Engine.new(fen).legal_moves -> Array<String>` — UCI
+  strings (e.g. `e2e4`, `e1g1`, `e7e8q`). Sorted lexicographically so
+  the output is stable across runs. UCI is unambiguous and trivial to
+  emit; SAN (disambiguation/check/checkmate) is intentionally left to
+  the existing pure-Ruby `PGN::Notation`, which can convert UCI→SAN
+  later if desired.
+- `PGN::Bitboard::Engine.new(fen).legal?(move) -> bool` — secondary.
+- A new `PGN::Bitboard` module keeps the native surface isolated from
+  the existing `PGN::Position`/`PGN::Board` (which stay pure Ruby). The
+  compiled library is loaded by a tiny Ruby shim such as
+  `lib/pgn/bitboard.rb`:
+  ```ruby
+  require "pgn2_native/pgn2_native"
+  ```
+- **Deferred:** having `PGN::Position#perft` delegate to the native
+  engine when loaded. Keep the surface isolated until the engine is
+  trusted.
+- Only strings and integers cross the Ruby↔Rust boundary — no Ruby
+  objects — so the bridge stays trivial and fast.
+
+## Bridge and packaging
+
+- `ext/pgn2_native/extconf.rb`:
+  ```ruby
+  require "mkmf"
+  require "rb_sys/mkmf"
+
+  create_rust_makefile("pgn2_native/pgn2_native")
+  ```
+- `pgn2.gemspec`:
+  ```ruby
+  spec.extensions = ["ext/pgn2_native/extconf.rb"]
+  spec.add_dependency "rb_sys", "~> 0.9.39" # build/runtime helper until
+                                            # RubyGems Rust support is stable
+  spec.add_development_dependency "rake-compiler", "~> 1.2"
+  spec.add_development_dependency "rake-compiler-dock", "~> 1.6"
+  ```
+  Because the gemspec uses `git ls-files`, make sure the Rust workspace
+  is checked in and includes `Cargo.lock` so source builds are
+  reproducible.
+- `magnus` `#[magnus::init] fn init(...)` defines the `PGN::Bitboard`
+  module + `Engine` class + methods. Wrap the Rust `Board`/`Engine`
+  struct as opaque `TypedData` so Ruby owns the lifetime and cannot
+  accidentally share mutable state across threads.
+- **Local dev build:** `bundle exec rake compile` (rake-compiler task)
+  invokes `cargo build --release`. Requires rustc/cargo locally.
+- **Prebuilt gems (the 3b path):** a GitHub Actions workflow using
+  `rake-compiler-dock` (rb_sys-compatible) cross-compiles
+  `pgn2-<ver>-<platform>.gem` for `x86_64-linux`, `aarch64-linux`,
+  `x86_64-darwin`, `aarch64-darwin` (Windows optional), pushed to
+  RubyGems on release. Bundler then pulls the platform gem on the
+  install host — **no toolchain required for end users.**
+
+## Deployment (chessellence / Azure)
+
+- chessellence `Gemfile` pins `pgn2` ≥ the native version.
+- With prebuilt **linux** gems published, the Docker build stage's
+  `bundle install` pulls the binary — **no Rust toolchain in the
+  Dockerfile at all** (the goal of choosing 3b). The final slim image
+  carries only the compiled `.so`.
+- If a platform binary is ever missing, the build stage falls back to a
+  source build (needs cargo). Mitigate by always publishing
+  `x86_64-linux` + `aarch64-linux` (the Azure targets).
+- **Interim** until the prebuilt-gem CI is live: chessellence deploys via
+  the 3a source-build path (rustup in the Docker build stage only). Keep
+  that documented in the rollout task.
+
+## Correctness strategy
+
+- Pure-Rust unit tests in `pgn2-bitboard` run on every CI: perft on the
+  six standard positions to published depths. Key anchors:
+  - Initial position, depth 6: **119,060,324** nodes.
+  - Kiwipete, depth 5: **193,690,690** nodes.
+  - Positions 3–6 to their known values (full table maintained in
+    `src/perft_tests.rs`).
+
+  These published counts are the **oracle**; there is no Ruby cross-check.
+- **make/unmake symmetry test:** iterate perft over every move, then
+  unmake each move, and assert the resulting FEN/bitboards match the
+  original exactly.
+- **Incremental legality test:** compared against a separate
+  pin-and-check-aware legal-move generator as soon as one is written.
+- A Ruby integration spec loads the ext (skip gracefully if not compiled)
+  and asserts perft on a couple of positions matches known values — this
+  guards the *binding*, not the engine.
+
+## Engine details (magic bitboards)
+
+- 12 `u64` bitboards (one per piece type per side); `white`/`black`/
+  `occupied` derived on demand. Slider attack computations treat the
+  moving slider's own square as empty (i.e. occupancy excludes that
+  square) so rooks/bishops/queens can attack along their ray through the
+  origin.
+- Precomputed knight/king attack tables; pawn attack tables; the 8 ray
+  directions.
+- **Slider attacks — BMI2 `pext` (implementation pivot):** the original
+  design called for magic bitboards with verified precomputed magic
+  numbers. Implementation instead uses the hardware `pext` instruction
+  to build a perfect bit-extract index (`((occ & mask) * magic) >> shift` is
+  replaced by `_pext_u64(occ & mask, mask)`), with a ray-walker fallback
+  on non-BMI2 targets (e.g. aarch64). Reasons: (1) random-candidate magic
+  search at `shift = 64 - bits` is effectively non-converging for a 12-bit
+  rook mask (~7.4M different-class subset pairs into 4096 bins → thousands
+  of collisions per trial); only specially-constructed magics work, and
+  transcribing published magics is a debugging hazard; (2) `pext` is
+  deterministic, search-free, instant to build, and collision-free by
+  construction; (3) the deploy target (Azure x86-64 Docker) has BMI2. The
+  magic-vs-reference validation tests are kept (now validating pext vs the
+  ray walker) for every square × subset. The public Rust/Ruby surface is
+  unchanged.
+- **Move encoding:** pack each move into a small integer (`from`:6 bits,
+  `to`:6 bits, `promotion`:3 bits, `capture`:1 bit, etc.) so move lists
+  and the make/unmake stack are cache-friendly and allocation-free.
+- **Move generation:** pseudo-legal from attack sets; legality by **make +
+  king-not-in-check filter** first (correctness-first, simplest to get
+  right), then optimize to pin-aware generation if needed to hit the nps
+  target.
+- **make/unmake:** all game-state mutations go onto a fixed-capacity
+  history stack inside the engine. A single `make`/`unmake` pair must
+  restore the board, side-to-move, castling rights, ep target, and
+  clocks bit-for-bit.
+- **Castling representation:** update the rook bitboard when a king move
+  crosses the castling line; legality of castling is checked by
+  verifying the king does not pass through or into check.
+- Zobrist **not** needed for perft — omit for now.
+
+## Risks and honest notes
+
+- Slider setup was originally the trickiest part; the `pext` pivot
+  (above) removes both the magic-search and the transcription hazard.
+  On non-BMI2 targets the ray-walker fallback is correct but slower —
+  add magic tables for aarch64 later if its perft nps matters.
+- The precompiled-gem CI (`rake-compiler-dock`) is real setup work;
+  budget it as its own task. Until live, deploy via 3a source-build
+  (documented interim).
+- The first PR brings the engine + `#perft`; `#legal_moves`/`#legal?`
+  follow once perft is trusted.
+- A required native extension changes the public gem contract: every
+  installer now gets a binary or builds from source. This is acceptable
+  per the decision, and is mitigated by prebuilt gems for common
+  platforms. Users on an unusual platform without a prebuilt need a
+  source build (cargo toolchain) — document this in the README.
+- Threading / GC: `Engine` wraps a Rust struct containing mutable game
+  state. Keep it tied to one Ruby object and do not expose
+  `Ractor`-shareable wrappers until the bindings explicitly support it.
+
+## Testing
+
+- **Rust:** `cargo test` — perft suites, make/unmake symmetry,
+  magic-table sanity.
+- **Ruby:** rspec, ext-gated; CI runs `cargo test` then `rspec`
+  (rspec compiles the ext first via the rake-compiler task).
+- **Bench:** `bench/perft.rb` runs perft on the standard positions,
+  reports nps; compare against published / chess.js baseline.
+
+## Global constraints
+
+- The existing pure-Ruby suite stays byte-identical and green; no
+  change to the 0x88 `Board` / `Notation` / `MoveCalculator`.
+- TDD on the Rust engine — the perft values *are* the tests.
+- Commit per task.

data/docs/superpowers/specs/2026-08-14-rust-integration-design.md

@@ -0,0 +1,217 @@
+# Rust Integration: `attacks::init` thread-safety + `Position#perft` / `#legal_moves` delegation — Design
+
+**Goal:** Land three incremental integrations of the new Rust bitboard
+engine into the Ruby surface, in increasing risk order:
+
+1. Make `attacks::init()` thread-safe and idempotent (correctness fix),
+   and drop the now-redundant per-method `attacks::init()` calls in the
+   magnus binding.
+2. Add `PGN::Position#perft(depth) -> Integer` delegating to
+   `PGN::Bitboard::Engine`.
+3. Add `PGN::Position#legal_moves -> Array<String>` (UCI) delegating to
+   the engine — **gated on an absolute-throughput bar** (see below).
+
+This extends the existing `2026-08-13-rust-bitboard-perft-design.md`
+backend; it does not revise it.
+
+## Non-goals
+
+- **No change to the pure-Ruby hot path.** `Position#move` / replay /
+  `MoveCalculator` / `Notation` stay byte-identical. `Position#perft`
+  and `#legal_moves` are *new* methods; they do not alter existing ones.
+- **No UCI→SAN conversion in this round.** SAN stays the existing
+  pure-Ruby `PGN::Notation`'s job. `#legal_moves` returns raw UCI. A
+  future round may compose `Notation.san(position, from, to, promo)` over
+  the UCI list; explicitly out of scope here.
+- **No pure-Ruby `legal_moves` fallback/comparator.** Per the chosen
+  gate, we measure absolute throughput only; no Ruby baseline is built.
+- **No `attacks::init`-removal across the whole crate.** The defensive
+  `attacks::init()` calls inside `pgn2-bitboard` (board/moves/perft/
+  legality) stay — they are cheap (atomic no-op after first) and keep
+  `cargo test` self-initializing. Only the *binding* (`pgn2_native`)
+  drops its redundant calls.
+
+## Task 1 — `attacks::init()` thread-safety
+
+### Problem
+
+`ext/pgn2_native/pgn2-bitboard/src/attacks.rs` holds four attack tables
+and a guard:
+
+```rust
+static mut KNIGHT: [Bitboard; 64] = [Bitboard::EMPTY; 64];
+static mut KING:   [Bitboard; 64] = [Bitboard::EMPTY; 64];
+static mut WP:     [Bitboard; 64] = [Bitboard::EMPTY; 64];
+static mut BP:     [Bitboard; 64] = [Bitboard::EMPTY; 64];
+static mut INIT: bool = false;
+
+pub fn init() {
+    unsafe {
+        if INIT { return; }
+        /* fill tables */
+        INIT = true;
+        crate::magics::build_all();
+    }
+}
+```
+
+`INIT` is a plain `bool` flipped inside `unsafe` with **no
+synchronization** — a data race (UB), and there is no happens-before
+guaranteeing later `unsafe` reads of the tables see the filled values.
+Under MRI the GIL masks this in practice, but it is a real defect and
+trips stricter tooling. (`magics.rs` is already correct: its `build_all`
+uses `Once`, so reads after `call_once` are happens-before-safe.)
+
+### Fix
+
+Drive `init()` through `std::sync::Once` so initialization is a single
+synchronized event and all subsequent reads are safe:
+
+```rust
+static INIT: Once = Once::new();
+
+pub fn init() {
+    INIT.call_once(|| unsafe {
+        // fill KNIGHT/KING/WP/BP (kept as `static mut`, written once here)
+        crate::magics::build_all();
+    });
+}
+```
+
+The `static mut` table arrays stay (they are written exactly once inside
+`call_once` and read after; `Once` establishes the happens-before). This
+is the minimal, correct change. Switching the arrays themselves to
+`OnceLock<[Bitboard; 64]>` is a larger cleanup explicitly deferred — it
+is not needed for correctness and would touch every accessor.
+
+`init()` remains idempotent and cheap: after first call it is one atomic
+load. The scattered `attacks::init()` calls inside `pgn2-bitboard` are
+therefore harmless and stay (keeps `cargo test` self-initializing).
+
+### Binding cleanup
+
+`pgn2_native/src/lib.rs` calls `attacks::init()` at the top of `perft`,
+`legal_moves_ruby`, and `legal_p`. These are **redundant**: every crate
+function they delegate to (`Board::perft`, `legal_moves`, `legality::*`)
+already calls `attacks::init()` itself. Remove the three binding-level
+calls. (`Engine::initialize`/`from_fen` needs no init — it only sets
+bitboards.)
+
+### Testing
+
+Existing `cargo test` perft oracle suite + Ruby `bitboard_spec.rb` must
+stay green. No new tests required for task 1 (behavior unchanged); the
+point is the safety/cheapness invariant, verified by inspection.
+
+## Task 2 — `PGN::Position#perft`
+
+### API
+
+```ruby
+# Returns the perft node count at +depth+ from this position.
+# Delegates to the native bitboard engine via a FEN round-trip.
+# @param depth [Integer] >= 0
+# @return [Integer]
+def perft(depth) -> Integer
+```
+
+### Implementation
+
+```ruby
+def perft(depth)
+  raise ArgumentError, "depth must be >= 0" unless depth.is_a?(Integer) && depth >= 0
+  PGN::Bitboard::Engine.new(to_fen.to_s).perft(depth)
+end
+```
+
+- Bridge is a **FEN round-trip**: `Position#to_fen` → `PGN::FEN#to_s`
+  → `Engine.new(fen)` → `#perft`. Only a String + Integer cross the
+  boundary (per the backend's boundary principle).
+- **Native availability:** the shipped gem is a precompiled native gem
+  (no fallback). `lib/pgn/bitboard.rb` loads the `.so` and rescues to
+  `LoadError` silently so the rest of the gem still loads. If the
+  extension is absent, `PGN::Bitboard` is undefined and `#perft` raises
+  `NameError` naturally — acceptable and consistent with the existing
+  `Engine` behavior. No new gating code; rely on `defined?`/NameError.
+
+### Testing
+
+New `spec/position_spec.rb` cases:
+- `Position.start.perft(0) == 1`, `perft(1) == 20`, `perft(2) == 400`,
+  `perft(3) == 8902`, `perft(4) == 197281` (published startpos values).
+- Kiwipete FEN → `perft(1) == 48`, `perft(2) == 2039`, `perft(3) == 97862`.
+- These double as cross-checks that the FEN round-trip is lossless for
+  perft-relevant state (placement/side/castling/ep).
+
+## Task 3 — `PGN::Position#legal_moves` (UCI), throughput-gated
+
+### API
+
+```ruby
+# All legal moves from this position as sorted UCI strings
+# (e.g. "e2e4", "e1g1" for castling, "e7e8q" for promotion).
+# Delegates to the native bitboard engine via a FEN round-trip.
+# @return [Array<String>] sorted lexicographically
+def legal_moves -> Array<String>
+```
+
+### Implementation
+
+```ruby
+def legal_moves
+  PGN::Bitboard::Engine.new(to_fen.to_s).legal_moves
+end
+```
+
+Same FEN round-trip bridge as task 2. Output is the engine's already-sorted
+UCI list; no Ruby post-processing.
+
+### Performance gate (decides whether task 3 ships)
+
+Per the chosen decision, **absolute throughput bar**, no Ruby baseline:
+
+- Benchmark `Position#legal_moves` end-to-end (FEN build + `Engine.new`
+  + native legal-gen + Ruby string materialization) on three positions:
+  startpos, a middlegame position, and Kiwipete.
+- **Ship iff** the middlegame case completes in **< 1 ms** end-to-end
+  (single call, warm). This makes the API practical: a user enumerating
+  legal moves per position pays under a millisecond, dominated by the
+  FEN round-trip and object construction, not by move generation.
+- If the middlegame case is **≥ 1 ms**, do **not** ship `#legal_moves`;
+  record the measured numbers and the breakdown (FEN build vs `Engine.new`
+  vs native gen vs string materialization) in the plan/changelog, and
+  stop. The gate is pass/fail; there is no partial ship.
+- The benchmark script lives at `bench/legal_moves.rb` and is run with
+  `bundle exec ruby bench/legal_moves.rb`; results committed to
+  `bench/` alongside the existing perft baselines.
+
+### Testing (only if it ships)
+
+New `spec/position_spec.rb` cases:
+- `Position.start.legal_moves.length == 20`.
+- After `Position.start.move("e4")`, the resulting `#legal_moves` equals
+  the engine's direct output for that FEN (delegation equivalence).
+- Spot-check a position with a promotion and one with castling, asserting
+  the UCI strings present (`e7e8q`, `e1g1`).
+
+## Risks
+
+- **FEN round-trip cost** is the dominant per-call cost for both new
+  methods. For `#perft` this is irrelevant (perft dwarfs it). For
+  `#legal_moves` it is *the* gate — if it blows the 1 ms bar, we don't
+  ship. No silent perf regression is possible: both methods are net-new.
+- **`attacks::init` race fix** is a one-line-ish change with no behavior
+  change; risk is a build break from `Once` import, caught by `cargo
+  test`.
+- **No regression to existing 233 specs**: all changes are additive
+  (new methods) or internal (init safety). Verified by running the full
+  suite after each task.
+
+## Out of scope / future
+
+- UCI→SAN via `PGN::Notation` over the `#legal_moves` list.
+- `Engine` reuse / caching across calls (a `Position`-held engine) to
+  amortize the FEN round-trip — only worth it if `#legal_moves` blows
+  the gate and we choose to optimize rather than drop.
+- Migrating `attacks.rs` table arrays to `OnceLock` (cleaner than
+  `static mut`, but not required for the race fix).