pgn2 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b17339791fa63da88b55a180d613c124509f7c286552fd9bb62dcea162eb3325
-  data.tar.gz: 2283fa7d6fbf06d54bebbdb9675427c2c51743627423db6bdd91df71a8e2926f
+  metadata.gz: 4a5ce857277684a9f89c02d35941d7c455df2ca8a18f557985f3ca92eed414b3
+  data.tar.gz: 5f126a3da7a6f314748495919168336bb7b97d07e5ad0c9f2ac9b9956a8bb4be
 SHA512:
-  metadata.gz: 311d858dca8016414e862b016fe694dff758ea575194d1a4256ebfa02e4427bc389ca291db5eb4a4cf547839e74e717aaeaed68fece0b43a1faf840148b87235
-  data.tar.gz: a0ad909b77394c5d7982db61ff37eb4ff02b2e1241b942e48b79bf290c70f5e6e756432fed01a5041179f27546faa2edae2a3b0cd54ff9a7f13a6bb3ac95c006
+  metadata.gz: '079c4e3c1dd292c58fac802349c8d9d8054efa4a01288090bd0a38472029b8848a162cd4264fe83135c80d14fb9ea65d531009ddc6dbd6237e30c4f4bb9efda8'
+  data.tar.gz: e3ec9dbcf8dad46e7e8ec623642d468253a4ee6d5899dd9d1ac7486e60416b9069a49e34de092e08744439b9c40dce84074a6ddcaed91a584d88a65be6646b9a

data/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 1.2.0 (2026-08-13)
+
+### Summary
+
+Parse allocation quick win: collapse `PGN::Lexer#scan_one`'s per-token
+3-element tuple. No public API changes; serialized PGN/FEN output stays
+byte-identical. See `bench/IMPROVEMENTS.md` for the per-step deltas.
+
+### Changed
+- **Lexer hot path**: `PGN::Lexer#scan_one` now returns the matched string
+  directly (stashing its type/discarded flag in ivars) instead of a
+  3-element `[type, m, discarded]` tuple, so the parser path (`next_token_pair`)
+  allocates only the single `[type, value]` array Racc requires per token.
+  Selected by per-line allocation profiling (`scan_one`'s tuple was the #1
+  parse allocation site).
+- **Deferred**: the coordinate-only-board + piece-location-index work was
+  profiled and deferred to a single coherent board-representation rewrite
+  (see TODO) — coordinate-board alone measured only ~1.25× replay at medium
+  risk and would likely be superseded by the piece-index rewrite.
+- Performance vs 1.1.0: parse-only allocations −42% objects (603537 → 347037 /
+  500 games); parse+replay −18% objects (1427586 → 1170586). Replay path
+  unchanged. All 182 specs pass; racc-sync OK.
+
 ## 1.1.0 (2026-08-13)
 
 ### Summary

data/README.md

@@ -181,12 +181,12 @@ Parser — 500 immortal games (`bench/profile_parse.rb`):
 
 | Metric | original `pgn` | pgn2 | Δ |
 |---|---:|---:|---:|
-| Parse-only allocations (objects) | 1248065 | 603537 | -644528 (-51.6%) |
-| Parse-only allocations (bytes) | 120370470 | 28257414 | -92113056 (-76.5%) |
-| Parse + replay allocations (objects) | 3778073 | 1427586 | -2350487 (-62.2%) |
-| Parse + replay allocations (bytes) | 249570048 | 78104136 | -171465912 (-68.7%) |
-| Parse-only throughput | 1461 ms/i | 327 ms/i | ~4.5x faster |
-| Parse + replay throughput | 1938 ms/i | 826 ms/i | ~2.35x faster |
+| Parse-only allocations (objects) | 1248065 | 347037 | -901028 (-72.2%) |
+| Parse-only allocations (bytes) | 120370470 | 17977414 | -102393056 (-85.1%) |
+| Parse + replay allocations (objects) | 3778073 | 1170586 | -2607487 (-69.0%) |
+| Parse + replay allocations (bytes) | 249570048 | 67804136 | -181765912 (-72.8%) |
+| Parse-only throughput | 1461 ms/i | 305 ms/i | ~4.8x faster |
+| Parse + replay throughput | 1938 ms/i | 816 ms/i | ~2.4x faster |
 
 What changed to get there:
 
@@ -211,6 +211,11 @@ What changed to get there:
 11. `PGN::MoveCalculator` — memoized `destination_coords`, frozen
     `ROOK_RESTRICTIONS`, empty short-circuit in `castling_restrictions`;
     `Move#pawn?` non-allocating.
+12. `PGN::Lexer#scan_one` — returns the matched string directly (stashing
+    type/discarded in ivars) instead of a 3-element `[type, m, discarded]`
+    tuple, so the parser hot path now allocates only the single `[type, value]`
+    array Racc requires per token. Cuts parse allocations ~42% (603537 → 347037
+    objects for 500 games).
 
 Public output (FEN, PGN) is byte-identical to the original gem; the full
 suite (182 examples) stays green. See `bench/IMPROVEMENTS.md` for the per-step

data/TODO.md

@@ -10,12 +10,41 @@
 
 - Support converting a game to pgn format
 - Speed up parsing
-  - Collapse PGN::Lexer's per-token allocation chain (Array in `scan_one` ->
-    `Token` struct -> Array in racc's `next_token`) to fewer allocations on
-    the hot path; keep the full `Token` (with offset/line) only for the
-    `#tokens` spec helper, which is its only consumer.
+  - ✓ (done in 1.2.0) Removed `PGN::Lexer`'s per-token `Token` Struct
+    allocation on the parser hot path (`next_token_pair`), and collapsed
+    `scan_one`'s `[type, m, discarded]` tuple to a single returned string
+    (type/discarded stashed in ivars). The full `Token` is kept only for the
+    `#tokens` spec helper. Parse allocations −42% (603537 → 347037 / 500 games).
+- Speed up replay via a board-representation rewrite (deferred "Approach B"):
+  per-line profiling shows the remaining replay allocations are architectural
+  — `MoveCalculator#first_piece` scan-return arrays (~5/ply, the #1 site) and
+  `Board#position_for` string joins (~3/ply). Do these as ONE coherent
+  rewrite (not separately, to avoid throwing away work):
+  (a) a piece-location index (piece → squares) so king/disambiguation/origin
+    lookups are O(1) instead of scanning 64 squares — kills `first_piece` scan
+    arrays AND the dominant replay compute (`valid_square?`/`at` ≈ 15/ply calls);
+  (b) a coordinate-only internal board (int square keys, no `"e4"` strings on
+    the hot path) — kills `position_for` strings + `changes` string keys.
+  Caveat: `change!`/`update`/`position_for`/`coordinates_for`/`squares` are
+  spec-tested public API, so the new representation must be additive (string
+  API kept). Realistic ceiling ~2× replay allocation + ~1.5–2× throughput;
+  medium-high risk (Board/MoveCalculator/Position/FEN). Only worth it given a
+  real hot-loop need (replay is already ~0.8 ms/ply).
 - Replace the right-recursive `tag_section`/`variation_list` rules in
   `pgn_parser.y` with ordinary left-recursion plus one explicit `.reverse`
   at the point each list is consumed, so the legacy whittle-order
   compatibility quirk is a single greppable line instead of implicit in
   recursion direction.
+- Make `MoveText#clean_text` idempotent (or run it exactly once, at
+  construction) so `Game#moves=`/`#standardize_castling` doesn't need to
+  sniff a comment for leftover `{`/`}` to decide whether a MoveText is safe
+  to reuse as-is. The brace check is a bandaid for `clean_text` not fully
+  normalizing multi-line/nested comments in one pass; fixing that at the
+  source would let `moves=` reuse unconditionally.
+- `MoveCalculator#destination_coords` memoizes into `@dest_coords` based on
+  `board`/`move`/`origin` never changing after `#initialize` — true today,
+  but only by convention, since `board`, `move`, `origin` are all public
+  `attr_accessor`s with no cache invalidation tied to their setters. If a
+  future caller ever mutates and reuses a `MoveCalculator` instance, this
+  memo goes stale silently. Either drop the public setters or invalidate
+  `@dest_coords` when they're used.

data/bench/IMPROVEMENTS.md

@@ -141,3 +141,27 @@ byte-identical (covered by `spec/lexer_spec.rb`, `spec/parser_explicit_spec.rb`,
 and the fixture round-trip in `spec/game_spec.rb`). Racc parser is in sync with
 `pgn_parser.y` (CI racc-sync check passes). No public API or serialized-output
 changes.
+
+## Token-array collapse (2026-08-13, v1.2.0)
+
+Collapse `PGN::Lexer#scan_one`'s per-token 3-element `[type, m, discarded]` tuple.
+`scan_one` now returns the matched string directly and stashes its type /
+discarded flag in `@scan_type` / `@scan_discarded` ivars, so the parser hot path
+(`next_token_pair`) allocates only the single `[type, value]` array Racc requires.
+Selected by per-line allocation profiling: `scan_one`'s tuple was the #1 parse
+allocation site. The coordinate-only-board + piece-location-index work was
+profiled and **deferred** — the board rewrite is a larger design (see TODO), and
+coordinate-board alone measured only ~1.25× replay at medium risk and would
+likely be superseded by the piece-index rewrite.
+
+### bench/profile_parse.rb (500 immortal games)
+
+| Metric | BEFORE | AFTER | Δ |
+|---|---|---|---|
+| Parse-only allocations (objects) | 603537 | 347037 | -256500 (-42.5%) |
+| Parse-only allocations (bytes) | 28257414 | 17977414 | -10280000 (-36.4%) |
+| Parse + replay allocations (objects) | 1427586 | 1170586 | -257000 (-18.0%) |
+| Parse + replay allocations (bytes) | 78104136 | 67804136 | -10300000 (-13.2%) |
+
+Replay (`profile_moves.rb`) is unchanged (1709 objects / 103720 bytes) — the
+lexer edit does not touch the replay path. All 182 specs pass; racc-sync OK.

data/bench/baseline_parse.txt

@@ -1,25 +1,25 @@
 Corpus: 500 copies of the immortal game
 
 === 1. Parse-only allocations (500 games) ===
-total_allocated objects: 603537
-total_allocated bytes:   28257414
+total_allocated objects: 347037
+total_allocated bytes:   17977414
 
 === 2. Parse + replay allocations (500 games) ===
-total_allocated objects: 1427586
-total_allocated bytes:   78104136
+total_allocated objects: 1170586
+total_allocated bytes:   67804136
 
 === 3. Parse-only throughput (ips) ===
 ruby 4.0.5 (2026-05-20 revision 64336ffd0e) +PRISM [x86_64-linux]
 Warming up --------------------------------------
      parse 500 games     1.000 i/100ms
 Calculating -------------------------------------
-     parse 500 games      3.649 (±27.4%) i/s  (274.08 ms/i) -     19.000 in   5.207546s
+     parse 500 games      4.086 (± 0.0%) i/s  (244.74 ms/i) -     21.000 in   5.139596s
 
 === 4. Parse + replay throughput (ips) ===
 ruby 4.0.5 (2026-05-20 revision 64336ffd0e) +PRISM [x86_64-linux]
 Warming up --------------------------------------
 parse+replay 500 games     1.000 i/100ms
 Calculating -------------------------------------
-parse+replay 500 games      1.399 (± 0.0%) i/s  (715.05 ms/i) -      7.000 in   5.005330s
+parse+replay 500 games      1.472 (± 0.0%) i/s  (679.56 ms/i) -      8.000 in   5.436470s
 
 Done. Compare this file against bench/baseline_parse.txt after optimizations.

data/lib/pgn/game.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 require 'io/console'
 
 module PGN
@@ -11,12 +13,12 @@ module PGN
       @variations = variations
     end
 
-    def ==(m)
-      to_s == m.to_s
+    def ==(other)
+      to_s == other.to_s
     end
 
-    def eql?(m)
-      self == m
+    def eql?(other)
+      self == other
     end
 
     def hash
@@ -31,6 +33,7 @@ module PGN
       text&.gsub(/{(.*)}/, '\1')&.gsub(/\s+/, ' ')&.strip
     end
   end
+
   # {PGN::Game} holds all of the information about a game. It is either
   # the result of parsing a PGN file, or created by hand.
   #
@@ -57,9 +60,9 @@ module PGN
     attr_accessor :tags, :result, :pgn, :comment
     attr_reader :moves
 
-    LEFT  = /(a|\x1B\[D)\z/.freeze
-    RIGHT = /(d|\x1B\[C)\z/.freeze
-    EXIT  = /(q|\x03)\z/.freeze
+    LEFT  = /(a|\x1B\[D)\z/
+    RIGHT = /(d|\x1B\[C)\z/
+    EXIT  = /(q|\x03)\z/
 
     # @param moves [Array<String>] a list of moves in SAN
     # @param tags [Hash<String, String>] metadata about the game
@@ -77,28 +80,8 @@ module PGN
     #
     # Standardize castling moves to use O's instead of 0's
     #
-    # Reuse-safety invariant: when an element is already a MoveText we reuse
-    # it directly (sharing one object between the parser's move tree and
-    # Game#moves) whenever its comment is already fully cleaned. This is safe
-    # because nothing mutates a MoveText after this assignment returns. We
-    # still re-wrap when the comment carries braces: the parser's single-pass
-    # clean_text leaves braces on multi-line/nested comments, and the second
-    # clean_text that MoveText.new applies here is load-bearing for those
-    # (matches the legacy whittle byte-output). Skipping it would change
-    # serialized comments.
     def moves=(moves)
-      @moves =
-        moves.map do |m|
-          if m.is_a?(String)
-            MoveText.new(m.include?('0') ? m.gsub('0', 'O') : m)
-          elsif m.notation.include?('0')
-            MoveText.new(m.notation.gsub('0', 'O'), m.annotation, m.comment, m.variations)
-          elsif m.comment.nil? || !m.comment.include?('{')
-            m
-          else
-            MoveText.new(m.notation, m.annotation, m.comment, m.variations)
-          end
-        end
+      @moves = moves.map { |m| standardize_castling(m) }
     end
 
     # @return [String] a canonical PGN string for this game, ending with a
@@ -152,11 +135,11 @@ module PGN
       loop do
         puts "\e[H\e[2J"
         puts positions[index].inspect
-        hist[0..2] = (hist[1..2] << STDIN.getch)
+        hist[0..2] = (hist[1..2] << $stdin.getch)
 
         case hist.join
         when LEFT
-          index -= 1 if index > 0
+          index -= 1 if index.positive?
         when RIGHT
           index += 1 if index < moves.length
         when EXIT
@@ -164,5 +147,22 @@ module PGN
         end
       end
     end
+
+    private
+
+    # A MoveText is reused as-is (no new object) when its notation needs no
+    # '0'->'O' fix and its comment has no braces left to clean; a braced
+    # comment still needs MoveText.new's second clean_text pass to match the
+    # legacy whittle byte-output.
+    def standardize_castling(entry)
+      return MoveText.new(entry.include?('0') ? entry.gsub('0', 'O') : entry) if entry.is_a?(String)
+
+      notation = entry.notation.include?('0') ? entry.notation.gsub('0', 'O') : entry.notation
+      if notation.equal?(entry.notation) && (entry.comment.nil? || !entry.comment.include?('{'))
+        return entry
+      end
+
+      MoveText.new(notation, entry.annotation, entry.comment, entry.variations)
+    end
   end
 end

data/lib/pgn/lexer.rb

@@ -146,29 +146,18 @@ module PGN
 
     # Returns the next {Token}, or +nil+ at end of input.
     def next_token
-      type, value, off = scan_next
+      type, value = next_token_pair
       return nil unless type
-      Token.new(type: type, value: value, offset: off, line: @line)
+
+      Token.new(type: type, value: value, offset: @last_offset, line: @line)
     end
 
     # Fast path for the parser: returns [type, value] for the next
     # non-discarded token, or +nil+ at end of input. Does not allocate a
-    # {Token} Struct. Shares the same scanning routine and the same
-    # +note_token+ / +advance_line+ side effects as +next_token+ (so
-    # +game_starts+ tracking is preserved).
+    # {Token} Struct, and +scan_one+ returns the matched string directly
+    # (stashing its type/discarded flag in ivars) so the only array
+    # allocated per token is the [type, value] pair Racc requires.
     def next_token_pair
-      type, value, = scan_next
-      return nil unless type
-      [type, value]
-    end
-
-    private
-
-    # Scan to the next non-discarded token and return [type, value, offset],
-    # or nil at end of input. Performs the exact +advance_line+ and
-    # +note_token+ side effects that +next_token+ historically did, so
-    # +game_starts+ (used for verbatim +Game#pgn+ slicing) stays correct.
-    def scan_next
       until @ss.eos?
         off = @ss.pos
 
@@ -176,25 +165,33 @@ module PGN
           type, value = lit
           @ss.pos = off + 1
           note_token(type, off)
-          return [type, value, off]
+          @last_offset = off
+          return [type, value]
         end
 
-        type, value, discarded = scan_one
+        value = scan_one
         advance_line(value)
-        next if discarded
+        next if @scan_discarded
 
-        note_token(type, off)
-        return [type, value, off]
+        note_token(@scan_type, off)
+        @last_offset = off
+        return [@scan_type, value]
       end
       nil
     end
 
-    # Try each terminal rule in order; return [type, matched_string, discarded]
-    # for the first match, or raise if nothing matches at the current position.
+    private
+
+    # Try each terminal rule in order; return the matched string for the
+    # first match (stashing its type and discarded flag in +@scan_type+ /
+    # +@scan_discarded+ so the caller avoids allocating a 3-element tuple),
+    # or raise if nothing matches at the current position.
     def scan_one
       RULES.each do |(type, re, discarded)|
         if (m = @ss.scan(re))
-          return [type, m, discarded]
+          @scan_type = type
+          @scan_discarded = discarded
+          return m
         end
       end
       raise UnconsumedInputError,

data/lib/pgn/move_calculator.rb

@@ -118,15 +118,13 @@ module PGN
         restrict += %w[K Q]
       when 'k'
         restrict += %w[k q]
-      when 'R'
-        restrict << ROOK_RESTRICTIONS[origin]
-      when 'r'
+      when 'R', 'r'
         restrict << ROOK_RESTRICTIONS[origin]
       end
 
       # when castling occurs
-      restrict += %w[K Q] if %w[K Q].include? move.castle
-      restrict += %w[k q] if %w[k q].include? move.castle
+      restrict += %w[K Q] if %w[K Q].include?(move.castle)
+      restrict += %w[k q] if %w[k q].include?(move.castle)
 
       # when a rook is taken
       restrict << 'Q' if move.destination == 'a1'
@@ -337,7 +335,7 @@ module PGN
     end
 
     def destination_coords
-      @dest_coords ||= board.coordinates_for(move.destination)
+      @destination_coords ||= board.coordinates_for(move.destination)
     end
   end
 end

data/lib/pgn/pgn_parser.rb

@@ -27,13 +27,20 @@ module_eval(<<'...end pgn_parser.y/module_eval...', 'pgn_parser.y', 97)
 
   private
 
-  # Punctuation tokens are racc'd by literal; everything else is racc'd by
-  # the upcased lexer symbol (e.g. :san_move -> :SAN_MOVE), so there's a
-  # single source of truth for the token vocabulary: PGN::Lexer's rules.
-  LITERAL_TOKENS = { lbracket: '[', rbracket: ']', lparen: '(', rparen: ')' }.freeze
+  # Punctuation tokens are racc'd by their literal character (taken from
+  # PGN::Lexer::LITERAL_BYTES, the single source of truth for those
+  # characters); everything else is racc'd by the upcased lexer symbol.
+  # A complete frozen lookup keeps this a plain hash read for every token
+  # instead of allocating two Strings (`to_s` + `upcase`) per token.
+  TOKEN_TRANSLATIONS = PGN::Lexer::LITERAL_BYTES.values.to_h.merge(
+    string: :STRING, comment: :COMMENT, game_termination: :GAME_TERMINATION,
+    san_move: :SAN_MOVE, nag: :NAG, move_number: :MOVE_NUMBER, tag_name: :TAG_NAME
+  ).freeze
 
   def translate_type(sym)
-    LITERAL_TOKENS[sym] || sym.to_s.upcase.to_sym
+    TOKEN_TRANSLATIONS.fetch(sym) do
+      raise "PGN::Lexer produced an unknown token type: #{sym.inspect}"
+    end
   end
 
   # Slice the verbatim raw PGN text per game out of the original input using

data/lib/pgn/pgn_parser.y

@@ -112,13 +112,20 @@ rule
 
   private
 
-  # Punctuation tokens are racc'd by literal; everything else is racc'd by
-  # the upcased lexer symbol (e.g. :san_move -> :SAN_MOVE), so there's a
-  # single source of truth for the token vocabulary: PGN::Lexer's rules.
-  LITERAL_TOKENS = { lbracket: '[', rbracket: ']', lparen: '(', rparen: ')' }.freeze
+  # Punctuation tokens are racc'd by their literal character (taken from
+  # PGN::Lexer::LITERAL_BYTES, the single source of truth for those
+  # characters); everything else is racc'd by the upcased lexer symbol.
+  # A complete frozen lookup keeps this a plain hash read for every token
+  # instead of allocating two Strings (`to_s` + `upcase`) per token.
+  TOKEN_TRANSLATIONS = PGN::Lexer::LITERAL_BYTES.values.to_h.merge(
+    string: :STRING, comment: :COMMENT, game_termination: :GAME_TERMINATION,
+    san_move: :SAN_MOVE, nag: :NAG, move_number: :MOVE_NUMBER, tag_name: :TAG_NAME
+  ).freeze
 
   def translate_type(sym)
-    LITERAL_TOKENS[sym] || sym.to_s.upcase.to_sym
+    TOKEN_TRANSLATIONS.fetch(sym) do
+      raise "PGN::Lexer produced an unknown token type: #{sym.inspect}"
+    end
   end
 
   # Slice the verbatim raw PGN text per game out of the original input using

data/lib/pgn/version.rb

@@ -1,3 +1,3 @@
 module PGN
-  VERSION = '1.1.0'.freeze
+  VERSION = '1.2.0'.freeze
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: pgn2
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Stacey Touset