ibex-runtime 0.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 05a9f9e7cb92bb15ef6efbfb632e4d846a3b3f8c4eb2fbccba549001b5e970ab
+  data.tar.gz: d8c262daf2e6525c80bf475aaed6500ee28fc054c6ef6628d772ae337b6f6a12
+SHA512:
+  metadata.gz: 20e654c2c2bac9452fa538a25918fd16475deb3bbd12a8491312709bc1623f7ac8f42381d3b191b061f244c617ca3b20c91528eda676d9ba3fe3f9af91c9dcc8
+  data.tar.gz: f69a58947420d425f9b6f957c680c16661272fb74f6bb9d3805584de74f66edb8ef141ee10c3eea7ab5fa84bb226101db63a14bd9e17f6c3690ed20c6374e46c

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Yudai Takada
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,337 @@
+# Ibex
+
+[![Gem Version](https://badge.fury.io/rb/ibex.svg)](https://badge.fury.io/rb/ibex)
+[![CI](https://github.com/ydah/ibex/actions/workflows/main.yml/badge.svg)](https://github.com/ydah/ibex/actions/workflows/main.yml)
+
+Ibex is a Pure Ruby LR parser generator. It reads racc-compatible grammar
+files, generates parsers with the familiar `do_parse` / `yyparse` API, and
+requires no C or Java extension.
+
+> [!IMPORTANT]
+> Ibex is pre-1.0. The current v1.0 decision is
+> [**HOLD**](docs/release-readiness.md): compatibility and scale evidence pass,
+> but cold generation and new-instance runtime still trail Racc's Ruby backend,
+> and the error UX evidence still needs independent review. Compatible mode is
+> the stable baseline; opt-in features have the maturity levels documented in
+> the [stability policy](docs/stability.md).
+
+- Try grammar analysis in the
+  [browser playground](https://ydah.github.io/ibex/playground/).
+- Start with the [grammar reference](docs/grammar-reference.md).
+- Build lossless and incremental tooling with the
+  [Red/Green CST guide](docs/cst.md).
+- Migrate an existing parser with the
+  [racc migration guide](docs/racc-migration.md).
+- Browse the [API reference](https://ydah.github.io/ibex/api/).
+
+The playground runs in a local Web Worker. It does not upload grammar source
+or execute semantic action bodies.
+
+## At a glance
+
+| Concern | Ibex default |
+| --- | --- |
+| Grammar input | racc-compatible `.y` source |
+| Construction | Direct LALR(1) |
+| Generated tables | Compact, immutable Ruby data |
+| Parser API | Pull parsing with `do_parse` / `yyparse` |
+| Generated dependency | The separately versioned `ibex-runtime` gem |
+| Ruby support | Ruby 3.0 or later |
+| Extensions | Disabled unless `pragma extended` or `--mode=extended` is used |
+| Concrete syntax | Stable format-v6 Red/Green CST with `pragma cst` |
+| Semantic actions | Opaque during generation and analysis; executed only by a running parser |
+
+The generator and runtime are deliberately separate:
+
+```text
+grammar.y + lexer contract
+          |
+          v
+        ibex  -----> parser.rb --------> ibex-runtime
+          |
+          +--------> optional RBS, IR, reports, diagrams, and docs
+```
+
+Use `-E` when a generated parser must embed the runtime instead of depending
+on `ibex-runtime`.
+
+## Quick start from a checkout
+
+Clone the repository and install its locked development dependencies:
+
+```sh
+git clone https://github.com/ydah/ibex.git
+cd ibex
+bundle install
+```
+
+Save this grammar as `calculator.y`:
+
+<!-- calculator-grammar:start -->
+```text
+class Calculator
+token NUM
+preclow
+  left '+'
+  left '*'
+prechigh
+rule
+  expr : expr '+' expr { result = val[0] + val[2] }
+       | expr '*' expr { result = val[0] * val[2] }
+       | NUM { result = val[0] }
+end
+---- inner
+def parse_tokens(tokens)
+  @tokens = tokens
+  do_parse
+end
+
+def next_token
+  @tokens.shift
+end
+---- footer
+if $PROGRAM_NAME == __FILE__
+  tokens = [[:NUM, 2], ['+', nil], [:NUM, 3], ['*', nil], [:NUM, 4]]
+  puts Calculator.new.parse_tokens(tokens)
+end
+```
+<!-- calculator-grammar:end -->
+
+Generate and run the parser without installing local gems:
+
+```sh
+bundle exec ruby -Ilib exe/ibex calculator.y
+bundle exec ruby -Ilib calculator.rb
+# 14
+```
+
+With the local gems installed, the equivalent commands are:
+
+```sh
+ibex calculator.y
+ruby calculator.rb
+```
+
+Ibex writes `calculator.rb`, uses compact tables, and generates a parser that
+requires `ibex-runtime`. Use `-o PATH` to choose the output, `--table=plain`
+for inspectable Hash rows, `--check` to compare without rewriting, or `-E` to
+embed the runtime.
+
+## Generated parser contract
+
+### Tokens and lexers
+
+A handwritten pull lexer implements `next_token` and returns:
+
+```ruby
+[token, value]
+[token, value, location]
+```
+
+`false` or `nil` marks EOF. Bare grammar tokens use Ruby symbols such as
+`:NUM`; quoted terminals use strings such as `'+'`. Locations are
+application-owned objects and remain optional.
+
+Existing lexers can use the [lexer migration guide](docs/lexer-migration.md).
+Extended grammars may instead generate a streaming lexer from declarative
+regular-expression rules.
+
+### Semantic actions and locations
+
+Actions read RHS values through `val`. `@1`, `@2`, and so on address parallel
+semantic locations, while `@$` is the span of the current reduction. Action
+source remains opaque Ruby: Ibex preserves its source mapping but does not
+parse, type-check, or sandbox it.
+
+### Parse lifecycles and errors
+
+Generated parsers support:
+
+- `do_parse` and yielding `yyparse` for racc-compatible pull parsing;
+- `push(token, value, location)` and `finish(location:)` for caller-driven
+  streaming;
+- structured `Ibex::ParseError` values containing the token, expected tokens,
+  state, location, and spelling suggestions;
+- yacc-style recovery through `on_error`, `yyerror`, `yyerrok`, and
+  `yyaccept`; and
+- immutable, shareable generated tables.
+
+Use one parser instance per concurrent parse. Isolated runtime sessions and
+resource budgets are opt-in when application code needs explicit execution
+boundaries.
+
+## Choose a grammar mode
+
+| Mode | Use it for | Contract |
+| --- | --- | --- |
+| `default` | Existing racc grammars and conservative migrations | Compatible syntax and runtime behavior |
+| `extended` | New grammars that need composition or generated tooling | Preview, explicitly enabled |
+
+Enable extended syntax with `pragma extended` in the grammar or
+`--mode=extended` on the command line:
+
+```text
+class ExtendedParser
+pragma extended
+token NUM KEY VALUE
+rule
+  arguments : separated_list(NUM, ',')
+  sum       : NUM:left '+' NUM:right { result = left + right }
+  maybe     : NUM?
+  many      : NUM*
+  groups    : (KEY VALUE)+
+  list(X)   : X | list(X) ',' X
+  %inline signed(X): '-' X { result = -val[1] }
+  negative  : signed(NUM)
+end
+```
+
+Extended mode includes nested EBNF groups, named references, parameterized and
+inline rules, multiple entry points, canonical fragment imports, generated
+lexers, CST or typed `Data` AST generation, declarative recovery, semantic
+types, and grammar-declared tests. These features do not silently change the
+default compatible frontend. Maturity is assigned per API: format-v6 batch
+Red/Green CST is Stable, while the wider extended grammar surface remains
+Preview and incremental CST sessions remain Experimental.
+
+## Choose a construction algorithm
+
+| Option | When to use it |
+| --- | --- |
+| `--algorithm=lalr` | Default direct LALR(1) construction |
+| `--algorithm=slr` | Grammars whose FOLLOW-set reductions are sufficient |
+| `--algorithm=ielr` | Preview backend for conflicts introduced by LALR state merging |
+| `--algorithm=lr1` | Canonical LR(1) analysis when the larger automaton is acceptable |
+
+`--suggest-ielr` checks whether IELR removes an unexpected LALR conflict and
+prints an advisory note when it does.
+That note does not change the selected algorithm or exit status.
+
+Conflict freedom is not the same as a proof of unambiguity. `ibex check
+--ambiguity` searches for a concrete sentence with two interpretations, but
+the token and configuration budgets make every result explicitly bounded.
+
+## Integrate generation into a build
+
+Treat grammar source as the input and generated Ruby as a reproducible build
+artifact:
+
+```sh
+ibex --warnings=error -o lib/calculator.rb grammar/calculator.y
+ibex --warnings=error --check -o lib/calculator.rb grammar/calculator.y
+```
+
+The first command writes the parser. The second is suitable for CI and fails
+when the committed output is stale without modifying it.
+
+Rake projects can declare the dependency directly:
+
+```ruby
+require "ibex/rake_task"
+
+Ibex::RakeTask.new("lib/calculator.rb") do |task|
+  task.grammar = "grammar/calculator.y"
+  task.options = ["--warnings=error"]
+end
+```
+
+The task follows the grammar's fragment dependency closure. Multi-output
+generation can additionally publish a manifest containing input identities,
+output paths, sizes, and SHA-256 digests. Generation renders all requested
+outputs before publishing them; `--watch` retains the last successful output
+when a later edit is invalid.
+
+Applications that only execute generated parsers need `ibex-runtime`, not the
+generator. To build and install both current local gems:
+
+```sh
+gem build ibex-runtime.gemspec
+gem build ibex.gemspec
+gem install ./ibex-runtime-0.1.0.gem
+gem install ./ibex-0.1.0.gem
+```
+
+## Diagnose, verify, and inspect
+
+| Goal | Command |
+| --- | --- |
+| Promote grammar warnings to errors | `ibex --warnings=error -C grammar.y` |
+| Collect structured frontend diagnostics | `ibex diagnose --format=json grammar.y` |
+| Explain one conflict | `ibex explain --state=7 --token=ELSE grammar.y` |
+| Search for a concrete ambiguity | `ibex check --ambiguity --algorithm=lr1 grammar.y` |
+| Run grammar-declared tests | `ibex test --coverage=100 grammar.y` |
+| Check canonical formatting | `ibex fmt --check grammar.y` |
+| Render grammar documentation | `ibex doc --format=html -o grammar.html grammar.y` |
+| Emit Automaton IR | `ibex --emit=automaton-ir grammar.y > automaton.json` |
+| Simulate table behavior without actions | `ibex debug automaton.json NUMBER PLUS NUMBER` |
+| Check a racc migration | `ibex migrate-check grammar.y` |
+| Start the language server | `ibex lsp --stdio` |
+
+Grammar IR, Automaton IR, generated table formats, diagnostic JSON, runtime
+events, coverage, and manifests are versioned contracts. Reports can also be
+rendered as text, DOT, Mermaid, HTML, railroad SVG, samples, or conflict
+counterexamples.
+
+## Safety boundaries and non-goals
+
+- Frontend parsing, diagnosis, formatting, documentation, LSP, static
+  migration checks, IR validation, and table simulation do not execute
+  semantic actions or user-code sections.
+- Generated parsers, grammar-declared tests, and explicit migration harnesses
+  do execute application Ruby. They are not sandboxes.
+- Fragment imports are confined to the declared source root and reject
+  traversal, cycles, and symlink escapes.
+- Counterexample, ambiguity, sample, repair, and runtime-budget searches are
+  bounded. Exhausting a budget is distinct from finding no witness within it.
+- Ibex generates deterministic LR parsers. It does not provide GLR or
+  generalized ambiguity handling. Incremental CST sessions are syntax-only
+  and experimental.
+- racc compatibility is a migration surface, not a claim that every generated
+  parser is a byte-for-byte or adapter-free replacement.
+- Preview and experimental features may have weaker compatibility guarantees
+  than the default mode.
+
+## Documentation by task
+
+| When you need to... | Start here |
+| --- | --- |
+| Write a grammar or use the runtime | [Grammar reference](docs/grammar-reference.md) |
+| Build, edit, serialize, or incrementally update syntax trees | [Red/Green CST guide](docs/cst.md) and [migration guide](docs/cst-migration.md) |
+| Learn from executable grammars | [Examples](examples/README.md) |
+| Migrate from racc | [racc migration guide](docs/racc-migration.md) |
+| Adapt a handwritten lexer | [Lexer migration guide](docs/lexer-migration.md) |
+| Configure an editor or LSP client | [Editor setup](docs/editor-setup.md) |
+| Understand the pipeline, IR, algorithms, or tables | [Architecture](docs/architecture.md) |
+| Check API maturity and deprecation rules | [Stability policy](docs/stability.md) |
+| Evaluate readiness, performance, or error UX | [Release readiness](docs/release-readiness.md), [benchmarks](benchmark/README.md), and [error UX](docs/error-ux.md) |
+| Contribute or run every quality gate | [Development guide](docs/development.md) |
+| Review implementation decisions | [Architecture decision records](docs/decisions/README.md) |
+
+The published documentation separates the
+[compatibility contract](https://ydah.github.io/ibex/compatibility/),
+[opt-in extensions](https://ydah.github.io/ibex/extensions/), and
+[experimental surfaces](https://ydah.github.io/ibex/experimental/). The
+[grammar gallery](https://ydah.github.io/ibex/gallery/) contains executable
+examples whose declared tests reach complete production coverage.
+
+## Development
+
+The [development guide](docs/development.md) contains frontend regeneration,
+RBS/Steep validation, grammar coverage, browser checks, mutation analysis, and
+workflow lint commands. The default local gate is:
+
+```sh
+bundle install
+bundle exec rake
+```
+
+<!-- type-stats:start -->
+The current whole-library `steep stats` result is 20,629 typed calls and 2,275 untyped calls out of 22,904 (90.1% typed).
+The generated signature tree contains 2,084 explicit `untyped` occurrences across 103 files.
+<!-- type-stats:end -->
+
+Performance measurements are evidence, not portable scores or CI timing
+thresholds. Follow the [benchmark guide](benchmark/README.md) to reproduce and
+compare results under the same environment and configuration.
+
+Ibex is available under the [MIT License](LICENSE.txt).

data/lib/ibex/runtime/ast_data.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+# rbs_inline: enabled
+
+module Ibex
+  module Runtime
+    # Builds Data classes on Ruby 3.2+ and an immutable Struct-compatible
+    # fallback on older supported Rubies.
+    module ASTData
+      # @rbs (*Symbol members) -> singleton(Object)
+      def define(*members)
+        return ::Data.define(*members) if defined?(::Data)
+
+        immutable = Module.new
+        immutable.send(:define_method, :initialize) do |*arguments, **keywords|
+          super(*arguments, **keywords)
+          freeze
+        end
+        Struct.new(*members, keyword_init: true).tap { |node| node.prepend(immutable) }
+      end
+      module_function :define
+    end
+  end
+end

data/lib/ibex/runtime/cst/annotation.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+# rbs_inline: enabled
+
+module Ibex
+  module Runtime
+    module CST
+      # Opaque identity used to track a syntax occurrence across path-copying edits.
+      class SyntaxAnnotation
+        # @rbs () -> void
+        def initialize
+          freeze
+        end
+      end
+    end
+  end
+end

data/lib/ibex/runtime/cst/cursor.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+# rbs_inline: enabled
+
+require_relative "syntax_node" unless defined?(Ibex::Runtime::CST::SyntaxNode)
+
+module Ibex
+  module Runtime
+    module CST
+      # Allocation-light cursor over Green elements and absolute offsets.
+      # rubocop:disable Naming/PredicateMethod -- cursor movement names follow the public tree-cursor convention.
+      class Cursor
+        # @rbs! type green = GreenNode | GreenToken
+        # @rbs! type frame = [GreenNode, Integer, Integer]
+
+        attr_reader :green #: green
+        attr_reader :offset #: Integer
+
+        # @rbs (SyntaxNode node) -> void
+        def initialize(node)
+          @green = node.green
+          @offset = node.offset
+          @frames = [] #: Array[frame]
+        end
+
+        # @rbs () -> bool
+        def goto_first_child
+          green = @green
+          return false unless green.is_a?(GreenNode)
+          return false if green.children.empty?
+
+          parent = green
+          @frames << [parent, 0, @offset]
+          @green = parent.children.fetch(0)
+          true
+        end
+
+        # @rbs () -> bool
+        def goto_next_sibling
+          frame = @frames.last
+          return false unless frame
+
+          parent, index, parent_offset = frame
+          next_index = index + 1
+          return false if next_index >= parent.children.length
+
+          frame[1] = next_index
+          @offset = parent_offset + parent.children.first(next_index).sum(&:full_width)
+          @green = parent.children.fetch(next_index)
+          true
+        end
+
+        # @rbs () -> bool
+        def goto_parent
+          frame = @frames.pop
+          return false unless frame
+
+          @green = frame.fetch(0)
+          @offset = frame.fetch(2)
+          true
+        end
+
+        # @rbs () -> Integer
+        def kind = @green.kind
+      end
+      # rubocop:enable Naming/PredicateMethod
+    end
+  end
+end

data/lib/ibex/runtime/cst/diff.rb

@@ -0,0 +1,80 @@
+# frozen_string_literal: true
+# rbs_inline: enabled
+
+require_relative "text_edit" unless defined?(Ibex::Runtime::CST::TextEdit)
+
+module Ibex
+  module Runtime
+    module CST
+      # Computes byte edits while skipping physically shared Green subtrees.
+      module Diff
+        # @rbs (SyntaxNode old_root, SyntaxNode new_root) -> Array[TextEdit]
+        def text_edits(old_root, new_root)
+          edits = [] #: Array[TextEdit]
+          compare(old_root.green, new_root.green, 0, edits)
+          edits.freeze
+        end
+        module_function :text_edits
+
+        # @rbs (GreenNode | GreenToken old_green, GreenNode | GreenToken new_green,
+        #   Integer offset, Array[TextEdit] edits) -> void
+        def compare(old_green, new_green, offset, edits)
+          return if old_green.equal?(new_green)
+
+          if old_green.is_a?(GreenNode) && new_green.is_a?(GreenNode) &&
+             compatible_nodes?(old_green, new_green)
+            child_offset = offset
+            old_green.children.each_with_index do |old_child, index|
+              new_child = new_green.children.fetch(index)
+              compare(old_child, new_child, child_offset, edits)
+              child_offset += old_child.full_width
+            end
+            return
+          end
+
+          append_trimmed_edit(old_green.to_source, new_green.to_source, offset, edits)
+        end
+        module_function :compare
+
+        # @rbs (GreenNode | GreenToken left, GreenNode | GreenToken right) -> bool
+        def compatible_nodes?(left, right)
+          left.is_a?(GreenNode) && right.is_a?(GreenNode) &&
+            left.kind == right.kind && left.children.length == right.children.length
+        end
+        module_function :compatible_nodes?
+
+        # @rbs (String old_text, String new_text, Integer offset, Array[TextEdit] edits) -> void
+        def append_trimmed_edit(old_text, new_text, offset, edits)
+          return if old_text == new_text
+
+          prefix = common_prefix(old_text, new_text)
+          suffix = common_suffix(old_text, new_text, prefix)
+          deleted = old_text.bytesize - prefix - suffix
+          inserted = new_text.byteslice(prefix, new_text.bytesize - prefix - suffix) || "".b
+          edits << TextEdit.new(start: offset + prefix, delete_length: deleted, insert_text: inserted)
+        end
+        module_function :append_trimmed_edit
+
+        # @rbs (String left, String right) -> Integer
+        def common_prefix(left, right)
+          limit = [left.bytesize, right.bytesize].min
+          index = 0
+          index += 1 while index < limit && left.getbyte(index) == right.getbyte(index)
+          index
+        end
+        module_function :common_prefix
+
+        # @rbs (String left, String right, Integer prefix) -> Integer
+        def common_suffix(left, right, prefix)
+          limit = [left.bytesize, right.bytesize].min - prefix
+          index = 0
+          index += 1 while index < limit && left.getbyte(-index - 1) == right.getbyte(-index - 1)
+          index
+        end
+        module_function :common_suffix
+
+        private_class_method :compare, :compatible_nodes?, :append_trimmed_edit, :common_prefix, :common_suffix
+      end
+    end
+  end
+end

data/lib/ibex/runtime/cst/editing.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+# rbs_inline: enabled
+
+require_relative "syntax_node" unless defined?(Ibex::Runtime::CST::SyntaxNode)
+
+module Ibex
+  module Runtime
+    module CST
+      # Shared path-copying operations for Red syntax elements.
+      module Editing
+        # @rbs! type element = SyntaxNode | SyntaxToken
+        # @rbs! type green = GreenNode | GreenToken
+
+        # @rbs (element target, green | element replacement) -> SyntaxNode
+        def replace(target, replacement)
+          green = green_element(replacement)
+          current = target
+          parent = target.parent
+          while parent
+            children = parent.green.children.dup
+            children[current.index] = green
+            green = GreenNode.new(
+              kind: parent.green.kind, children: children,
+              flags: parent.green.intrinsic_flags, annotations: parent.green.annotations
+            )
+            current = parent
+            parent = parent.parent
+          end
+          raise TypeError, "the root replacement must be a GreenNode" unless green.is_a?(GreenNode)
+
+          red_root(target.root, green)
+        end
+        module_function :replace
+
+        # @rbs (SyntaxNode previous, GreenNode green) -> SyntaxNode
+        def red_root(previous, green)
+          source = SourceText.new(green.to_source, file: previous.source_text.file)
+          SyntaxNode.new(
+            green: green, kinds: previous.kinds, trivia_policy: previous.trivia_policy,
+            source_text: source
+          )
+        end
+        module_function :red_root
+
+        # @rbs (green | element value) -> green
+        def green_element(value)
+          return value.green if value.is_a?(SyntaxNode) || value.is_a?(SyntaxToken)
+          return value if value.is_a?(GreenNode) || value.is_a?(GreenToken)
+
+          raise TypeError, "replacement must be a Green syntax element or Red syntax view"
+        end
+        module_function :green_element
+      end
+    end
+  end
+end