pgn2 1.0.0 → 1.1.0

data/docs/superpowers/specs/2026-08-13-pgn-performance-quick-wins-design.md

@@ -0,0 +1,227 @@
+# PGN2 performance quick wins — Approach A (quick, behavior-compatible micro-opts)
+
+> "Approach A" denotes the safe, allocation-focused track. A later, larger
+> design (Approach B) will cover architectural changes: piece-location indexes,
+> lazy position computation, and a coordinate-only internal board
+> representation.
+
+## Goal
+
+Make the gem faster on the combined **parse + replay** workload after the
+whittle → Racc migration, with only safe, behavior-compatible micro-optimizations
+(no public API changes, no serialized-output changes).
+
+## Scope
+
+This design covers the five highest-ROI quick wins identified in the profiles:
+
+1. Remove per-token `PGN::Lexer::Token` allocation on the parser hot path.
+2. Avoid double `MoveText` wrapping in `PGN::Game#moves=`.
+3. Replace `PGN::Move` SAN regex with hand-rolled parsing.
+4. Reduce small temporary allocations in `PGN::Position#move` and
+   `PGN::MoveCalculator`.
+5. Promote the Racc parse baseline to canonical and record new baselines in
+   `bench/baseline_*.txt` after verification.
+
+Out of scope: larger architecture changes such as piece-location indexes,
+lazy position computation, or coordinate-only internal board representation.
+
+## Context / baseline
+
+Measured on the current checkout with `bundle exec rake bench` (Racc parser);
+these are the canonical `bench/baseline_*.txt` values refreshed as the
+prerequisite in item 5:
+
+- **Parse-only:** ~125 k objects / 7.88 MB for 100 copies of the immortal game
+  (= 626,037 objects / 39.4 MB for the 500-game corpus in `bench/baseline_parse.txt`).
+- **Parse + replay:** ~337 k objects / 21.6 MB for the same per-100 ratio
+  (= 1,683,087 objects / 108 MB for 500 games).
+- **Replay only:** 2,177 objects / 142 kB for the 45-ply immortal game
+  (`bench/baseline_moves.txt`).
+
+Historical snapshots are kept in `bench/baseline_*.pre-quickwins.txt` and
+`bench/baseline_parse.racc.txt` for reference. The previously committed
+`baseline_parse.txt` held stale pre-migration whittle numbers and has been
+overwritten by the current Racc output.
+
+Top allocation hot spots:
+
+- `PGN::Lexer::Token` creation (parse-only).
+- `PGN::MoveText` created twice per parsed move (parser + `Game#moves=`).
+- `Move#initialize` SAN regex (`Regexp` + `MatchData` objects, replay).
+- Small throw-away arrays and string conversions in `MoveCalculator` and
+  `Position#move`.
+
+## Detailed changes
+
+### 1. Lexer: parser hot path avoids `Token` objects
+
+`lib/pgn/lexer.rb`
+
+- Add a fast public method `next_token_pair` that returns `[type, value]`
+  directly, without allocating a `PGN::Lexer::Token` Struct.
+  The byte offset is **not** expensive (`@ss.pos`) and is still required, so
+  the saving is the Struct allocation, not offset computation.
+- **Correctness requirement:** `next_token_pair` must invoke the same
+  `note_token(type, off)` and `advance_line(value)` side effects as
+  `next_token`. In particular `note_token` maintains `game_starts`, which
+  `PgnParser#assign_pgn!` relies on to slice each game's verbatim raw `pgn`
+  text — skipping it would corrupt `Game#pgn`.
+- Keep `next_token` and `tokens` unchanged (they continue to return `Token`
+  objects and are used by specs/other callers).
+- Implement both methods on top of a shared private scanning routine so the
+  scanning logic is not duplicated.
+- The single-byte literal path already returns frozen strings via
+  `LITERAL_BYTES` under `# frozen_string_literal: true`; preserve that in
+  `next_token_pair` (no change needed beyond routing through the shared
+  routine).
+
+`lib/pgn/pgn_parser.y`
+
+- Change `PgnParser#next_token` to call `@lexer.next_token_pair` and return the
+  resulting array directly.
+- Regenerate `lib/pgn/pgn_parser.rb` from the `.y` file with Racc.
+- No changes to grammar or semantics.
+
+### 2. Game#moves=: stop re-wrapping existing MoveText objects
+
+`lib/pgn/game.rb`
+
+- Detect when an element is already a `PGN::MoveText` instance and reuse it
+  directly instead of creating a new one.
+- Still perform castling normalization (`0` → `O`) for raw strings.
+- Preserve `clean_text` behavior for fresh comments only.
+- **Reuse-safety invariant:** reusing the parser's `MoveText` shares one object
+  between the parser's move tree and `Game#moves`. This is safe only because
+  nothing mutates a `MoveText` after construction; record that invariant in a
+  comment so a future mutation doesn't silently corrupt both holders.
+
+Expected effect: roughly halves `MoveText` allocations on the parse path while
+keeping serialized output identical.
+
+### 3. Move#initialize: hand-rolled SAN parser (highest-risk item — quantify first)
+
+`lib/pgn/move.rb`
+
+This is the highest-risk, and likely lowest-allocation-payoff, item: the SAN
+regex produces ~1 `MatchData` per ply (~2.1% of the 2,177-object replay
+baseline). Before committing to a full hand-roll, **measure #3 in isolation**
+and confirm the payoff; if marginal, ship #1, #2, #4 alone and defer #3.
+
+Trivial independent sub-win (do regardless of the full hand-roll):
+
+- In `Move#piece=`, replace `return if san.match('O-O')` (which allocates a
+  `MatchData` on every `Move.new`, castling or not) with a non-allocating
+  check such as `san.include?('O-O')` or `san.start_with?('O')`.
+
+Full hand-roll (if pursued):
+
+- Replace `move.match(SAN_REGEX)` with a manual parse over the byte/string
+  representation of the SAN notation.
+- Populate all existing attributes (`piece`, `destination`, `promotion`,
+  `check`, `capture`, `disambiguation`, `castle`) with the same values as
+  today, including the `piece=` early-return for castling.
+- Keep setter methods (`piece=`, `promotion=`, `capture=`, `disambiguation=`,
+  `castle=`) so external callers that assign attributes manually are unaffected.
+- Use frozen constants for frequently-checked piece sets (`pawn?` lives here,
+  in `move.rb` — see item 4).
+- **Upfront fixtures:** add a dedicated spec pinning every attribute for a
+  comprehensive set of SAN strings: `O-O`, `O-O-O`, `O-O+`, `O-O-O#`;
+  pawn moves and captures incl. promotion (`e4`, `exd5`, `e8=Q`, `exd8=Q+`,
+  `b1=N#`); piece moves with file/rank/full disambiguation (`Nbd2`, `R1e2`,
+  `Qh4e1`); captures with check/mate; and the `--` "don't care" move. Do this
+  before implementation, not only if ambiguity surfaces.
+
+### 4. Position / MoveCalculator: fewer throw-away allocations
+
+`lib/pgn/position.rb`
+
+- Inline `next_player` logic in `Position#move` (or simply rewrite its body to
+  `player == :white ? :black : :white`) to avoid the `(PLAYERS - [player])`
+  array allocation per ply.
+- Apply `castling_restrictions` only when non-empty; avoid array subtraction
+  when there is nothing to remove. (Safe to return the existing `castling`
+  array directly because castling arrays are replaced, never mutated.)
+
+`lib/pgn/move_calculator.rb`
+
+- Compute and cache destination coordinates in `initialize` instead of calling
+  `board.coordinates_for(move.destination)` repeatedly (it can be invoked 2–3
+  times per move across `direction_origins` / `move_origins` / `pawn_origins`).
+- Inline `valid_square?` boundary checks to remove the per-call method dispatch
+  (note: `(0..7)` are frozen range literals already cached by the VM, so the
+  win is call overhead, not Range allocation — don't claim the latter).
+- Replace run-time array/hash literals with frozen constants or `case`/`when`
+  where they appear on hot paths (e.g. rook-origin lookup in
+  `castling_restrictions`, the per-call hash literals `{ 'a1' => 'Q', ... }`).
+- Keep semantic behavior identical (same board updates, same disambiguation,
+  same `king_position` scan — Approach A deliberately defers indexing/caching
+  to a later design).
+
+`lib/pgn/move.rb` (relocated from move_calculator)
+
+- `Move#pawn?` uses `%w[P p].include?(piece)`, allocating an array each call;
+  it's invoked from `MoveCalculator#increment_halfmove?` and `pawn_origins`.
+  Replace with a frozen constant or `piece == 'P' || piece == 'p'`.
+
+## Testing & behavior preservation
+
+- All existing specs must pass without modification (`bundle exec rspec`).
+- The `bench/baseline_moves.txt` and `bench/baseline_parse.txt` files will be
+  regenerated with `bundle exec rake bench` and committed if they show lower
+  object/byte counts and equal or better throughput.
+- Public API remains unchanged: `PGN.parse`, `Game#moves=`, `Move.new`,
+  `Position#move`, etc. accept the same inputs and produce the same outputs and
+  side effects.
+- No changes to PGN/FEN serialization format.
+- **Lexer non-regression:** because `next_token_pair` shares the scanning
+  routine with `next_token`/`tokens`, the existing `tokens`/`next_token` specs
+  and the `game_starts`-driven `Game#pgn` raw-text slicing output must stay
+  byte-identical across the fixtures and inline inputs in
+  `spec/parser_explicit_spec.rb`.
+
+## Risks & mitigations
+
+| Risk | Mitigation |
+|---|---|
+| Hand-rolled SAN parser mishandles an edge case | Upfront comprehensive SAN fixture spec (castling both sides, promotion with/without check, pawn & piece captures, file/rank/full disambiguation, `--`) added before implementation; full suite must stay green. |
+| `next_token_pair` drifts from `next_token` behavior | Both methods share the private scanning routine, the same `RULES` ordering, and the same `note_token`/`advance_line` side effects (incl. `game_starts`). |
+| `Game#moves=` no longer applies castling normalization | Continue normalizing raw strings; only skip work when input is already `MoveText`. |
+| Performance wins smaller than expected | Measure before/after with the committed baseline harness. |
+
+## Acceptance criteria
+
+1. `bundle exec rspec` passes (0 failures).
+2. `lib/pgn/pgn_parser.rb` is regenerated and in sync with `.y`
+   (`git diff --stat` should reflect the generation, and CI racc-sync check
+   passes).
+3. `bundle exec rake bench` shows lower allocated objects/bytes than the
+   **post-Racc** committed baselines (`baseline_moves.txt` and
+   `baseline_parse.txt`, the latter promoted from `baseline_parse.racc.txt`
+   per item 5). Comparing against the old whittle `baseline_parse.txt` would
+   not be a meaningful gate.
+4. IPS numbers are equal or higher than current baselines.
+5. Baseline files are updated and committed as part of the change set.
+6. Lexer non-regression: `tokens`/`next_token` spec output and `Game#pgn`
+   raw-text slicing are byte-identical to the pre-change checkout.
+
+## Estimated impact
+
+Per-item expectations (to be confirmed by measurement, not assumed in
+aggregate):
+
+- **Items 1, 2, 4 (low-risk):** the largest reliable wins. Parse-side `Token`
+  elimination (#1) and halved `MoveText` allocation (#2) should show up directly
+  in parse-only object counts; `next_player`/`pawn?`/`castling_restrictions`
+  (#4) trim a handful of objects per ply.
+- **Item 3 (SAN hand-roll):** ~1 `MatchData` per ply (~2.1% of the 2,177-object
+  replay baseline) plus the `piece=` guard fix — modest, and the bulk of the
+  estimate does **not** come from this item.
+- **Replay allocations are dominated by `Position.new` + `Board#dup` +
+  `MoveCalculator` arrays**, which Approach A only trims at the edges; the big
+  replay win is deferred to Approach B.
+
+Conservative aggregate guess once #1/#2/#4 land: parse-only allocations down
+~10–15%, replay allocations down in the single-digit-percent range (lower than
+a naive reading of these micro-opts might suggest). Actual numbers will come
+from the baseline run; revise this section with measured per-item deltas.

data/lib/pgn/game.rb

@@ -77,15 +77,26 @@ module PGN
     #
     # Standardize castling moves to use O's instead of 0's
     #
+    # Reuse-safety invariant: when an element is already a MoveText we reuse
+    # it directly (sharing one object between the parser's move tree and
+    # Game#moves) whenever its comment is already fully cleaned. This is safe
+    # because nothing mutates a MoveText after this assignment returns. We
+    # still re-wrap when the comment carries braces: the parser's single-pass
+    # clean_text leaves braces on multi-line/nested comments, and the second
+    # clean_text that MoveText.new applies here is load-bearing for those
+    # (matches the legacy whittle byte-output). Skipping it would change
+    # serialized comments.
     def moves=(moves)
       @moves =
         moves.map do |m|
-          if m.is_a? String
+          if m.is_a?(String)
             MoveText.new(m.include?('0') ? m.gsub('0', 'O') : m)
+          elsif m.notation.include?('0')
+            MoveText.new(m.notation.gsub('0', 'O'), m.annotation, m.comment, m.variations)
+          elsif m.comment.nil? || !m.comment.include?('{')
+            m
           else
-            notation = m.notation
-            notation = notation.gsub('0', 'O') if notation.include?('0')
-            MoveText.new(notation, m.annotation, m.comment, m.variations)
+            MoveText.new(m.notation, m.annotation, m.comment, m.variations)
           end
         end
     end

data/lib/pgn/lexer.rb

@@ -24,10 +24,10 @@ module PGN
     end
 
     # Discarded: insignificant whitespace.
-    WSP = /\s+/.freeze
+    WSP = /\s+/
 
     # Discarded: a PGN "rest of line" comment beginning with `%`.
-    PGN_COMMENT = /% .*/.freeze
+    PGN_COMMENT = /% .*/
 
     # A tag value string. Allows unescaped double-quotes inside the value
     # (a form seen in real-world PGN files) — only a bare backslash starts
@@ -40,14 +40,14 @@ module PGN
         \\"                      # escaped quotation marks
       )*                         # zero or more of the above
       "                          # end of string
-    /x.freeze
+    /x
 
     # A brace-delimited comment, with recursive nesting via \g<1>.
     COMMENT = /
       (
         \{                           # beginning of comment
         (
-          [[:print:]&&[^\\\{\}]] |   # printing characters except brace and backslash
+          [[:print:]&&[^\\{}]] |   # printing characters except brace and backslash
           \n                     |
           \\\\                   |   # escaped backslashes
           \\\{|\\\}              |   # escaped braces
@@ -56,15 +56,15 @@ module PGN
         )*                           # zero or more of the above
           \}                           # end of comment
       )
-    /x.freeze
+    /x
 
     # Game termination marker.
     GAME_TERMINATION = %r{
       1-0       |    # white wins
       0-1       |    # black wins
-      1\/2-1\/2 |    # draw
+      1/2-1/2 |    # draw
       \*             # ?
-    }x.freeze
+    }x
 
     # A move in standard algebraic notation (incl. castling, promotion,
     # check/mate, the `--` "don't care" move).
@@ -83,43 +83,46 @@ module PGN
         \+                            |    # check (g5+)
         \#                                 # checkmate (Qe7#)
       )?
-    }x.freeze
+    }x
 
     # A move number indication, e.g. `1.`, `12.`, `1...`.
-    MOVE_NUMBER = /[[:digit:]]+\.*/.freeze
+    MOVE_NUMBER = /[[:digit:]]+\.*/
 
     # A tag name (letters, digits, underscores).
-    TAG_NAME = /[A-Za-z0-9_]+/.freeze
+    TAG_NAME = /[A-Za-z0-9_]+/
 
     # A numeric annotation glyph (`$1`) or a punctuation annotation (`?!`,
     # `!?`, `??`, ...).
     NAG = /
       \$\d+       | # dollar sign followed by an integer
-      [\?!][\?!]?   # support the most used annotations directly
-    /x.freeze
+      [?!][?!]?   # support the most used annotations directly
+    /x
 
     # Order matters: more specific / longer tokens are tried first so that
     # e.g. `1-0` (termination) wins over `1` (move number), and `0-0`
     # (castling) wins over `0` (move number). Whitespace and `%` comments
-    # are discarded (consumed but not emitted).
+    # are discarded (consumed but not emitted). Beyond that constraint,
+    # rules are ordered most- to least-frequent (one san_move/move_number
+    # per ply/full-move vs. a handful of comments/strings per game) so the
+    # common case fails the fewest regexes before matching.
     RULES = [
       [:wsp,              WSP,              true],   # discarded
       [:pgn_comment,      PGN_COMMENT,      true],   # discarded
-      [:comment,          COMMENT,          false],
-      [:string,           STRING,           false],
       [:game_termination, GAME_TERMINATION, false],
       [:san_move,         SAN_MOVE,         false],
-      [:nag,              NAG,              false],
       [:move_number,      MOVE_NUMBER,      false],
-      [:tag_name,         TAG_NAME,         false],
+      [:nag,              NAG,              false],
+      [:comment,          COMMENT,          false],
+      [:string,           STRING,           false],
+      [:tag_name,         TAG_NAME,         false]
     ].freeze.each(&:freeze)
 
-    # Single-character literals, matched by their byte value.
+    # Single-character literals, matched by their byte value: [type, frozen value].
     LITERAL_BYTES = {
-      91 => :lbracket,   # [
-      93 => :rbracket,   # ]
-      40 => :lparen,     # (
-      41 => :rparen,     # )
+      91 => [:lbracket, '['], # [
+      93 => [:rbracket, ']'], # ]
+      40 => [:lparen,   '('], # (
+      41 => [:rparen,   ')'] # )
     }.freeze
 
     def initialize(input)
@@ -127,7 +130,7 @@ module PGN
       @ss = StringScanner.new(input)
       @line = 1
       @game_starts = []
-      @between_games = true   # at start we are "between" games
+      @between_games = true # at start we are "between" games
     end
 
     attr_reader :game_starts
@@ -143,36 +146,55 @@ module PGN
 
     # Returns the next {Token}, or +nil+ at end of input.
     def next_token
+      type, value, off = scan_next
+      return nil unless type
+      Token.new(type: type, value: value, offset: off, line: @line)
+    end
+
+    # Fast path for the parser: returns [type, value] for the next
+    # non-discarded token, or +nil+ at end of input. Does not allocate a
+    # {Token} Struct. Shares the same scanning routine and the same
+    # +note_token+ / +advance_line+ side effects as +next_token+ (so
+    # +game_starts+ tracking is preserved).
+    def next_token_pair
+      type, value, = scan_next
+      return nil unless type
+      [type, value]
+    end
+
+    private
+
+    # Scan to the next non-discarded token and return [type, value, offset],
+    # or nil at end of input. Performs the exact +advance_line+ and
+    # +note_token+ side effects that +next_token+ historically did, so
+    # +game_starts+ (used for verbatim +Game#pgn+ slicing) stays correct.
+    def scan_next
       until @ss.eos?
         off = @ss.pos
 
         if (lit = LITERAL_BYTES[@input.getbyte(off)])
+          type, value = lit
           @ss.pos = off + 1
-          note_token(lit, off)
-          return Token.new(type: lit, value: @input.byteslice(off, 1),
-                           offset: off, line: @line)
+          note_token(type, off)
+          return [type, value, off]
         end
 
-        type, value = scan_one
+        type, value, discarded = scan_one
         advance_line(value)
-        if type == :wsp || type == :pgn_comment
-          # discarded: keep looping without emitting
-          next
-        end
+        next if discarded
+
         note_token(type, off)
-        return Token.new(type: type, value: value, offset: off, line: @line)
+        return [type, value, off]
       end
       nil
     end
 
-    private
-
-    # Try each terminal rule in order; return [type, matched_string] for
-    # the first match, or raise if nothing matches at the current position.
+    # Try each terminal rule in order; return [type, matched_string, discarded]
+    # for the first match, or raise if nothing matches at the current position.
     def scan_one
-      RULES.each do |(type, re, _discarded)|
+      RULES.each do |(type, re, discarded)|
         if (m = @ss.scan(re))
-          return [type, m]
+          return [type, m, discarded]
         end
       end
       raise UnconsumedInputError,

data/lib/pgn/move.rb

@@ -105,7 +105,10 @@ module PGN
     end
 
     def piece=(val)
-      return if san.match('O-O')
+      # Castling SAN (O-O / O-O-O) has no piece attribute; the castle attribute
+      # is set later in #initialize. Use a non-allocating prefix check instead
+      # of `san.match('O-O')`, which allocated a MatchData on every Move.new.
+      return if san.start_with?('O')
 
       val ||= 'P'
       @piece = black? ? val.downcase : val
@@ -161,7 +164,7 @@ module PGN
     # @return [Boolean] whether the piece being moved is a pawn
     #
     def pawn?
-      %w[P p].include?(piece)
+      piece == 'P' || piece == 'p'
     end
   end
 end

data/lib/pgn/move_calculator.rb

@@ -82,6 +82,11 @@ module PGN
       }
     }.freeze
 
+    # Frozen rook-origin -> castling-restriction lookup, shared by both
+    # white ('R') and black ('r') since their rook origins (a1/h1, a8/h8)
+    # are distinct keys. Replaces a per-call hash literal.
+    ROOK_RESTRICTIONS = { 'a1' => 'Q', 'h1' => 'K', 'a8' => 'q', 'h8' => 'k' }.freeze
+
     attr_accessor :board, :move, :origin
 
     # @param board [PGN::Board] the current board
@@ -114,9 +119,9 @@ module PGN
       when 'k'
         restrict += %w[k q]
       when 'R'
-        restrict << { 'a1' => 'Q', 'h1' => 'K' }[origin]
+        restrict << ROOK_RESTRICTIONS[origin]
       when 'r'
-        restrict << { 'a8' => 'q', 'h8' => 'k' }[origin]
+        restrict << ROOK_RESTRICTIONS[origin]
       end
 
       # when castling occurs
@@ -129,7 +134,7 @@ module PGN
       restrict << 'K' if move.destination == 'h1'
       restrict << 'k' if move.destination == 'h8'
 
-      restrict.compact.uniq
+      restrict.empty? ? restrict : restrict.compact.uniq
     end
 
     # @return [Boolean] whether to increment the halfmove clock
@@ -332,7 +337,7 @@ module PGN
     end
 
     def destination_coords
-      board.coordinates_for(move.destination)
+      @dest_coords ||= board.coordinates_for(move.destination)
     end
   end
 end

data/lib/pgn/parser.rb

@@ -1,28 +1,18 @@
+# frozen_string_literal: true
+
 require 'pgn/pgn_parser'
 
 module PGN
   # {PGN::Parser} is the public entry point for parsing PGN text into a list
-  # of game hashes. It delegates to a concrete backend parser (currently the
-  # stdlib Racc + StringScanner parser {PGN::PgnParser}).
-  #
-  # The +backend+ class attribute is retained so callers (tests, benchmarks)
-  # can substitute an alternative parser responding to +new.parse(pgn_string)+.
+  # of game hashes. It delegates to the stdlib Racc + StringScanner parser
+  # {PGN::PgnParser}.
   class Parser
-    class << self
-      # The backend parser class used by {.parse}.
-      # @return [Class]
-      attr_accessor :backend
-    end
-
-    # Default backend is the stdlib Racc + StringScanner parser.
-    self.backend = PGN::PgnParser
-
     # @param input [String] the raw PGN text (already force-encoded by the
     #   caller, e.g. via {PGN.parse}).
     # @return [Array<Hash>] one Hash per game, with keys +:tags+, +:result+,
     #   +:moves+, +:pgn+, +:comment+.
     def parse(input)
-      self.class.backend.new.parse(input)
+      PGN::PgnParser.new.parse(input)
     end
   end
 end