racc-rbs-inline 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b215555957e98247fb76e1d65743e02e5b3ec8e84d4c143e40bb59c54b9ec51c
+  data.tar.gz: a809fc1c903ecbc510e63839f35d61dda7f349aa7abed5c5588acd4998f0fd47
+SHA512:
+  metadata.gz: ef470246d25e93fa2331241b931e441c13622bfec4afbec0caa2222f6f63a262da9558046c2d33d5497a03881dabb3888b4099ce7d1de02b4904cb08a8ea9467
+  data.tar.gz: b3878b62cc6659442d201861fb4db51a9609de9fe4e7d6331b1176f56f4f6f4787beacf3eab1a5953911d952c844300bbe96fe5a4f84e278b0a1b753fab975ad

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Yudai Takada
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,269 @@
+# racc-rbs-inline
+
+Add inline RBS annotations to a Racc-generated Ruby parser, without modifying Racc.
+The annotations work with **rbs-inline 0.14** and **Steep 2.0's native inline support**.
+Generated parsers keep their existing runtime dependencies.
+
+## Installation
+
+Until the first RubyGems release, add this development dependency to your Gemfile:
+
+```ruby
+group :development do
+  gem "racc-rbs-inline", git: "https://github.com/ydah/racc-rbs-inline"
+  gem "racc", "1.8.1"
+  gem "rbs-inline", "0.14.0" # only needed for the extraction workflow
+  gem "steep", "2.0.0"
+end
+```
+
+Run `bundle install`. The annotator requires Ruby 3.2+, Prism 1.0+, Racc 1.6.2+
+(below 2.0), and RBS 4.1+ (below 5.0).
+
+## Usage
+
+Add one command after your existing Racc invocation:
+
+```sh
+bundle exec racc -o lib/parser.rb parser.y
+bundle exec racc-rbs-inline annotate lib/parser.rb
+```
+
+Or generate and annotate together:
+
+```sh
+bundle exec racc-rbs-inline generate parser.y -o lib/parser.rb -- --embedded --frozen
+```
+
+`-o` is required. Arguments after `--` are passed to Racc, which runs as a separate
+process. Racc's exit status is preserved if generation fails. For executable
+output with Racc 1.8.1, specify a path such as `--executable=/usr/bin/ruby`;
+Racc itself rejects an omitted executable path on some Ruby versions.
+
+```sh
+# Annotate several files in place.
+bundle exec racc-rbs-inline annotate lib/parser.rb lib/other_parser.rb
+
+# Print the transformed source, leaving files alone.
+bundle exec racc-rbs-inline annotate --stdout lib/parser.rb
+
+# Add semantic value types declared in the grammar.
+bundle exec racc-rbs-inline annotate lib/parser.rb --grammar parser.y
+
+# List files needing annotation; exit 1 if any need changes, otherwise 0.
+bundle exec racc-rbs-inline annotate --check lib/parser.rb
+
+# Copy runtime signatures (defaults to sig/racc/parser.rbs).
+bundle exec racc-rbs-inline sig --output sig/racc
+
+bundle exec racc-rbs-inline --help
+bundle exec racc-rbs-inline --version
+```
+
+Invalid Ruby, including invalid Ruby exposed when action wrappers are removed,
+is rejected before writing the annotated file. `annotate` validates every input
+before writing any of them. Repeating annotation leaves the file unchanged.
+
+The library API returns source text without performing file I/O:
+
+```ruby
+require "racc_rbs_inline"
+
+annotated_source = RaccRbsInline.annotate(source)
+RaccRbsInline.annotated?(annotated_source) # => true
+
+# Supply grammar text to add or refresh semantic value types.
+annotated_source = RaccRbsInline.annotate(source,
+  grammar: File.read("parser.y"), grammar_filename: "parser.y")
+```
+
+`annotated?` detects the `# rbs_inline: enabled` comment. Without a grammar,
+files already carrying that marker are left untouched. Supplying a grammar
+validates the generated tables and refreshes reduction signatures, including
+files that already have generic annotations.
+
+## Semantic value types
+
+Declare terminal and nonterminal types in grammar comments:
+
+```ruby
+class TypedCalc
+  # @racc-type NUMBER exp target : Integer
+  # @racc-type '+' '(' ')' : String
+rule
+  target: exp { result = val[0] }
+  exp: NUMBER { result = val[0] }
+     | exp '+' NUMBER { result = val[0] + val[2] }
+     | '(' exp ')' { result = val[1] }
+end
+```
+
+`generate` and the Rake task read these declarations automatically. For an
+existing generated parser, pass `annotate --grammar parser.y`. Use the same
+Racc version that generated the parser: mismatched rule tables are rejected
+before writing. Invalid RBS types, unknown symbols, and conflicting declarations
+are also errors. Semantic type processing requires quoted grammar symbols to
+be literal strings without Ruby interpolation.
+
+Each reduction receives a tuple of the right-hand-side types and returns the
+left-hand-side type. Undeclared symbols stay `untyped`. The initial `result`
+parameter includes both the first right-hand-side type (or `nil` for an empty
+rule) and the left-hand-side type. `no_result_var` grammars receive two
+parameters instead. Racc 1.8's `?`, `*`, and `+` symbols become optional or array
+types; groups become `Array[untyped]`. Generated helper reductions and
+`_reduce_none` retain generic signatures.
+
+With the declarations above, returning `val[0]` from the parenthesized rule is
+reported as a `String`/`Integer` mismatch by both checking workflows. A runnable
+example, including token handling, is in
+[test/fixtures/typed_calc.y](test/fixtures/typed_calc.y).
+
+## Rake integration
+
+```ruby
+require "racc_rbs_inline/rake_task"
+
+RaccRbsInline::RakeTask.new(:parser) do |task|
+  task.grammar = "parser.y"
+  task.output = "lib/parser.rb"
+  task.options = ["--embedded", "--frozen"]
+end
+```
+
+`bundle exec rake parser` regenerates and annotates the output when the grammar
+is newer or the output is absent. The output is also available as a file task.
+
+## Type checking
+
+First copy the `Racc::Parser` runtime signatures:
+
+```sh
+bundle exec racc-rbs-inline sig
+```
+
+These signatures live in `data/racc` inside the gem, so they are not automatically
+loaded with this gem's own signatures. Racc signatures have been submitted in
+[gem_rbs_collection PR #1085](https://github.com/ruby/gem_rbs_collection/pull/1085).
+Until they are accepted, use the `sig` command. Once collection support is
+available, use `rbs collection install`
+and remove your copied `sig/racc` directory to avoid duplicate declarations.
+
+### Native inline support (Steep 2.0)
+
+```ruby
+# Steepfile
+target :lib do
+  signature "sig/racc"
+  check "lib", inline: true
+end
+```
+
+```sh
+bundle exec steep check
+```
+
+### Extract signatures with rbs-inline
+
+```ruby
+# Steepfile
+target :lib do
+  signature "sig"
+  check "lib"
+end
+```
+
+```sh
+bundle exec rbs-inline --output=sig/generated lib
+bundle exec steep check
+```
+
+Choose one workflow; do not load extracted declarations while also enabling
+native inline checking for the same parser.
+
+A complete small grammar is in [test/fixtures/sum.y](test/fixtures/sum.y).
+Copy that grammar to `parser.y` in a project with the Gemfile above:
+
+```sh
+mkdir -p lib
+bundle exec racc-rbs-inline generate parser.y -o lib/parser.rb
+bundle exec racc-rbs-inline sig
+# Save either Steepfile above, then run that workflow's commands.
+```
+
+The integration tests exercise this grammar through both workflows, including
+strict diagnostics. To enable strict diagnostics in your own target, add
+`configure_code_diagnostics(Steep::Diagnostic::Ruby.strict)`.
+
+## What gets annotated
+
+The annotator locates classes whose bodies assign `Racc_arg`. It unwraps their
+heredoc `module_eval` calls, then annotates ten tables, three constants, and all
+`_reduce_N` / `_reduce_none` methods. Reduction arguments and results use
+`untyped` unless grammar declarations supply semantic value types.
+
+User-written actions, `header`, `inner`, and `footer` code remain your
+responsibility to type. Existing annotations in `inner` become visible after
+unwrapping. Separate instance variable annotations from method definitions
+with a blank line:
+
+```ruby
+---- inner
+  # @rbs @tokens: Array[Array[untyped]]
+
+  #: (Array[Array[untyped]]) -> void
+  def initialize(tokens)
+    @tokens = tokens
+  end
+```
+
+**Backtrace locations change:** unwrapping `module_eval` means exceptions point
+to the generated `.rb` file instead of the grammar's `.y` lines, as with Racc's
+`--no-line-convert` option. Code relying on `__FILE__`, `__LINE__`, or lexical
+scope inside those wrappers should account for the same change.
+
+The implementation preserves other source bytes, including embedded runtime
+code, header/footer wrappers, encoding declarations, and line endings.
+
+## Development
+
+```sh
+bundle install
+bundle exec rbs collection install
+ruby script/fetch-racc
+bundle exec rake
+```
+
+`rake test` runs unit tests, 13 golden/behavior variants, both type-checking
+workflows, semantic value checks, deliberately corrupt table tests, and the
+upstream Racc corpus.
+The corpus is fetched from the pinned `v1.8.1` tag into ignored `tmp/racc` and is
+not redistributed. Set `RACC_CORPUS` to use another local copy. Without the
+corpus, that regression test reports a skip; CI always fetches it.
+
+The 58 upstream grammars include three intentional generator errors and two
+with invalid Ruby actions. All other 53 are checked separately to avoid class
+name collisions. User-code signature failures are documented in
+[test/regression_test.rb](test/regression_test.rb); generated table diagnostics
+must be absent. The embedded runtime is checked byte-for-byte while golden
+files normalize its representation, version banners, random delimiter IDs,
+and executable paths.
+
+`rake typecheck` regenerates this gem's own signatures and checks them with
+Steep. CI covers Ruby 3.2, 3.3, 3.4, and 4.0, with Racc 1.8.1 as the baseline
+and additional Racc 1.6.2 / 1.7.3 compatibility jobs.
+Racc 1.6.2 does not support `--frozen`; that variant is skipped there.
+Racc 1.6.2 and 1.7.3 cannot generate the four upstream grammars using Racc 1.8's
+group/optional/repetition syntax, so their corpus checks cover the remaining 49.
+
+The separate Lrama acceptance job checks the pinned upstream parser before and
+after generation with `--embedded --frozen` (79 examples each), then checks
+generated annotations with strict Steep diagnostics. Run it locally with
+`bundle exec ruby script/test-lrama`; it fetches Lrama into ignored `tmp/lrama`
+and writes reports to `tmp/lrama-acceptance`. It restores the original parser
+after the check and requires a clean checkout at the pinned revision.
+Application type errors remain in the report; two unrelated profiler signatures
+using unsupported `void` unions are excluded from the temporary checking project.
+
+## License
+
+[MIT](LICENSE.txt).

data/data/racc/parser.rbs

@@ -0,0 +1,99 @@
+# Type signatures for the Racc runtime (lib/racc/parser.rb, racc 1.8.x)
+module Racc
+  VERSION: String
+  Version: String
+  Copyright: String
+
+  Racc_No_Extensions: bool
+
+  class ParseError < StandardError
+  end
+
+  class Parser
+    # Token symbols as serialized in racc_token_table
+    type token_symbol = Symbol | String | false
+
+    # The 14-element Racc_arg tuple (same order as ParserFileGenerator emits)
+    type racc_arg = [
+      Array[Integer?],             # action_table
+      Array[Integer?],             # action_check
+      Array[Integer?],             # action_default
+      Array[Integer?],             # action_pointer
+      Array[Integer?],             # goto_table
+      Array[Integer?],             # goto_check
+      Array[Integer?],             # goto_default
+      Array[Integer?],             # goto_pointer
+      Integer,                     # nt_base
+      Array[Integer | Symbol],     # reduce_table
+      Hash[untyped, Integer],      # token_table
+      Integer,                     # shift_n
+      Integer,                     # reduce_n
+      bool                         # use_result_var
+    ]
+
+    interface _DebugOut
+      def print: (*untyped) -> untyped
+      def puts: (*untyped) -> untyped
+    end
+
+    Racc_Runtime_Version: String
+    Racc_Runtime_Core_Version_R: String
+    Racc_Runtime_Core_Version_C: String
+    Racc_Main_Parsing_Routine: Symbol
+    Racc_YY_Parse_Method: Symbol
+    Racc_Runtime_Core_Version: String
+    Racc_Runtime_Type: String
+
+    @yydebug: bool
+    @racc_debug_out: _DebugOut
+    @racc_state: Array[Integer]
+    @racc_tstack: Array[Integer]
+    @racc_vstack: Array[untyped]
+    @racc_t: Integer?
+    @racc_val: untyped
+    @racc_read_next: bool
+    @racc_user_yyerror: bool
+    @racc_error_status: Integer
+
+    def self.racc_runtime_type: () -> String
+
+    # Subclasses define their own constructors; keep it permissive so that
+    # unannotated overrides do not inherit `() -> void` from Object.
+    def initialize: (?) -> void
+
+    # Methods users commonly override: keep types permissive.
+    def next_token: () -> Array[untyped]?
+    def on_error: (Integer t, untyped val, Array[untyped] vstack) -> untyped
+
+    def do_parse: () -> untyped
+    def yyparse: (untyped recv, interned mid) -> untyped
+    def yyerror: () -> bot
+    def yyaccept: () -> bot
+    def yyerrok: () -> void
+    def token_to_str: (Integer t) -> String?
+
+    def racc_read_token: (Integer t, untyped tok, untyped val) -> void
+    def racc_shift: (Integer tok, Array[Integer] tstack, Array[untyped] vstack) -> void
+    def racc_reduce: (Array[Integer] toks, Integer sim, Array[Integer] tstack, Array[untyped] vstack) -> void
+    def racc_accept: () -> void
+    def racc_e_pop: (Array[Integer] state, Array[Integer] tstack, Array[untyped] vstack) -> void
+    def racc_next_state: (Integer curstate, Array[Integer] state) -> void
+    def racc_print_stacks: (Array[Integer] t, Array[untyped] v) -> void
+    def racc_print_states: (Array[Integer] s) -> void
+    def racc_token2str: (Integer tok) -> String
+
+    def _racc_setup: () -> racc_arg
+    def _racc_init_sysvars: () -> void
+    def _racc_do_parse_rb: (racc_arg arg, bool in_debug) -> untyped
+    def _racc_yyparse_rb: (untyped recv, interned mid, racc_arg arg, bool c_debug) -> untyped
+    def _racc_evalact: (Integer act, racc_arg arg) -> Integer?
+    def _racc_do_reduce: (racc_arg arg, Integer act) -> Integer?
+
+    private
+
+    def _racc_do_parse_c: (racc_arg arg, bool in_debug) -> untyped
+    def _racc_yyparse_c: (untyped recv, interned mid, racc_arg arg, bool c_debug) -> untyped
+  end
+end
+
+class ::ParseError = Racc::ParseError

data/exe/racc-rbs-inline

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "racc_rbs_inline/cli"
+
+exit RaccRbsInline::CLI.run(ARGV)

data/lib/racc_rbs_inline/annotator.rb

@@ -0,0 +1,180 @@
+# frozen_string_literal: true
+# rbs_inline: enabled
+
+require "prism"
+
+module RaccRbsInline
+  class Annotator
+    MAGIC = "# rbs_inline: enabled" #: String
+    INTEGER_TABLE = /\Aracc_(?:action|goto)_(?:table|check|pointer|default)\z/ #: Regexp
+    REDUCTION = /\A_reduce_(?:\d+|none)\z/ #: Regexp
+    CONSTANT_TYPES = {
+      Racc_arg: "::Racc::Parser::racc_arg",
+      Racc_token_to_s_table: "Array[String]",
+      Racc_debug_parser: "bool"
+    }.freeze #: Hash[Symbol, String]
+
+    #: (String source, ?grammar: String?, ?grammar_filename: String) -> String
+    def annotate(source, grammar: nil, grammar_filename: "grammar.y")
+      existing = annotated?(source)
+      return source if existing && !grammar
+
+      types = nil #: GrammarTypes?
+      if grammar
+        require_relative "grammar_types"
+        types = GrammarTypes.new(grammar, filename: grammar_filename)
+      end
+
+      source = unwrap(source)
+      source = add_annotations(source, types: types, existing: existing)
+      existing ? source : add_magic_comment(source)
+    end
+
+    #: (String source) -> bool
+    def annotated?(source)
+      return false unless source.include?(MAGIC)
+
+      Prism.parse(source).comments.any? { |comment| comment.location.slice.strip == MAGIC }
+    end
+
+    private
+
+    #: (String source) -> String
+    def unwrap(source)
+      edits = [] #: Array[[Integer, Integer, String]]
+      binary = source.b
+      parser_bodies(parse!(source)).each do |body|
+        body.body.each do |statement|
+          next unless statement.is_a?(Prism::CallNode) && statement.name == :module_eval && statement.receiver.nil?
+
+          heredoc = statement.arguments&.arguments&.first
+          next unless heredoc.is_a?(Prism::StringNode) && heredoc.heredoc?
+          closing = heredoc.closing_loc
+          next unless closing
+
+          start = line_start(binary, statement.location.start_offset)
+          edits << [start, closing.end_offset, heredoc.content_loc.slice.b]
+        end
+      end
+      apply(source, edits)
+    end
+
+    #: (String source, types: GrammarTypes?, existing: bool) -> String
+    def add_annotations(source, types:, existing:)
+      edits = [] #: Array[[Integer, Integer, String]]
+      binary = source.b
+      parser_bodies(parse!(source)).each do |body|
+        signatures = types ? types.signatures(body: body) : {} #: Hash[Symbol, String]
+        body.body.each do |statement|
+          case statement
+          when Prism::LocalVariableWriteNode
+            type = table_type(statement)
+            edits << trailing(statement, type) if type && !existing
+          when Prism::ConstantWriteNode
+            type = CONSTANT_TYPES[statement.name]
+            edits << trailing(statement, type) if type && !existing
+          when Prism::DefNode
+            next unless statement.name.to_s.match?(REDUCTION)
+
+            count = statement.parameters&.requireds&.size
+            unless [2, 3].include?(count)
+              raise Error, "Unsupported parameters for #{statement.name}: expected two or three required parameters"
+            end
+            parameters = "Array[untyped] val, Array[untyped] _values"
+            parameters += ", untyped result" if count == 3
+            start = line_start(binary, statement.location.start_offset)
+            indent = binary.byteslice(start, statement.location.start_offset - start)
+            signature = signatures.fetch(statement.name, "#: (#{parameters}) -> untyped")
+            finish = start
+            if existing && start.positive?
+              previous = line_start(binary, start - 1)
+              line = binary.byteslice(previous, start - previous)
+              start = previous if line&.match?(/\A[ \t]*#:/)
+            end
+            edits << [start, finish, "#{indent}#{signature}#{newline(source)}".b]
+          end
+        end
+      end
+      apply(source, edits)
+    end
+
+    #: (Prism::LocalVariableWriteNode node) -> String?
+    def table_type(node)
+      name = node.name.to_s
+      return "Array[Integer?]" if name.match?(INTEGER_TABLE)
+      return "Array[Integer | Symbol]" if name == "racc_reduce_table"
+      return unless name == "racc_token_table"
+
+      literal = node.value.is_a?(Prism::HashNode) && node.value.elements.all? do |element|
+        element.is_a?(Prism::AssocNode) &&
+          [Prism::SymbolNode, Prism::StringNode, Prism::FalseNode].any? { |type| element.key.is_a?(type) }
+      end
+      literal ? "Hash[::Racc::Parser::token_symbol, Integer]" : "Hash[untyped, Integer]"
+    end
+
+    #: (Prism::Node node, String type) -> [Integer, Integer, String]
+    def trailing(node, type)
+      offset = node.location.end_offset
+      [offset, offset, " #: #{type}".b]
+    end
+
+    #: (String source) -> String
+    def add_magic_comment(source)
+      offset = 0
+      source.b.each_line do |line|
+        break unless line.start_with?("#!") || line.match?(/\A#[^\r\n]*(?:frozen[_-]string[_-]literal\s*:|(?:coding|encoding)\s*[:=])/i)
+
+        offset += line.bytesize
+      end
+      apply(source, [[offset, offset, "#{MAGIC}#{newline(source)}".b]])
+    end
+
+    #: (String source) -> String
+    def newline(source)
+      source.include?("\r\n") ? "\r\n" : "\n"
+    end
+
+    #: (String source) -> Prism::ProgramNode
+    def parse!(source)
+      result = Prism.parse(source)
+      if result.failure?
+        details = result.errors.map { |error| "line #{error.location.start_line}: #{error.message}" }.join(", ")
+        raise Error, "Invalid Ruby source: #{details}"
+      end
+      result.value
+    end
+
+    #: (Prism::ProgramNode program) -> Array[Prism::StatementsNode]
+    def parser_bodies(program)
+      bodies = [] #: Array[Prism::StatementsNode]
+      pending = [program] #: Array[Prism::Node]
+      while (node = pending.pop)
+        if node.is_a?(Prism::ClassNode) && node.body.is_a?(Prism::StatementsNode) &&
+            node.body.body.any? { |statement| statement.is_a?(Prism::ConstantWriteNode) && statement.name == :Racc_arg }
+          bodies << node.body
+        end
+        pending.concat(node.compact_child_nodes)
+      end
+      raise Error, "Racc parser class not found" if bodies.empty?
+
+      bodies
+    end
+
+    #: (String binary, Integer offset) -> Integer
+    def line_start(binary, offset)
+      return 0 if offset.zero?
+
+      previous = binary.rindex("\n", offset - 1)
+      previous ? previous + 1 : 0
+    end
+
+    #: (String source, Array[[Integer, Integer, String]] edits) -> String
+    def apply(source, edits)
+      binary = source.b
+      edits.sort_by { |start, _, _| -start }.each do |start, finish, replacement|
+        binary[start...finish] = replacement
+      end
+      binary.force_encoding(source.encoding)
+    end
+  end
+end