arpaka 0.1.1 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d9c4d3fb152bbd9bb8a5c00037371b5387d8d72027dfe9aacef3032364fff434
-  data.tar.gz: 39037d6380101b5f89d24d2c397428377fe44180e77215c59c5967349a48256e
+  metadata.gz: 93edcf594153e47990a52fa6b5645d19af6bea08b8eb1dba90c165737293561b
+  data.tar.gz: 7c02b70e4a01d155542c1d9db2302498c1d43b65a073ad9e8363bede2ebfe2f2
 SHA512:
-  metadata.gz: c0505eecc22775c6bb8f84932382d7274a80372541c0a68d65c6dc2c1203c37251ef91c063f94d38313d934b1d6e9143a27d34640413d5428344721d6449b209
-  data.tar.gz: 598e86005c31fbbd8cc98374c1c3dbedcb6feb1f97087dfaf3ab12461fc40b64e31c7b2f538eed382c726b19ab771f2edf2f1acfca25c1deef7f98fb74ef605c
+  metadata.gz: e0b845cc98cca04321ba644e8bb3b4bfe1974a0979faec672590110a63f8b825da0e34244fceb911dcd11027c57938a6dd9ad6e775ceb6826968c34d2bd7e2ab
+  data.tar.gz: 6e4b87c52c412c8dff7de7fe727d1ece2b4b831e07036c9e2619d48b4c66e59955d419122996ca491e5cca0eb5a32dd5434982c983ef24e3154da7a2bae01a74

data/CHANGELOG.md

@@ -1,5 +1,49 @@
 ## [Unreleased]
 
-## [0.1.0] - 2026-09-07
+## [2.0.0]
 
-- Initial release
+- Added optional structured AST output for lexical class, module, instance
+  method and singleton method definitions, with source-order nesting and
+  separate occurrences for reopened classes and modules.
+- Supported namespaced generated frontend class names through the API and CLI.
+- Improved Ruby AST compatibility for method parameters, local scopes, pattern
+  matching, splats, interpolation, control flow and command calls.
+- Added a usage guide and an AST comparison benchmark that separates parse
+  success from AST agreement and records unsupported comparisons explicitly.
+  Added optional Ractor checks and a `--disable-gc` switch for experiments.
+- Reduced the pinned grammar's generated standalone frontend from about 450 KB
+  to about 190 KB by compressing parser tables and omitting unused rule data.
+- Refactored Ruby parser and frontend generation into focused internal
+  components without changing the default detailed AST output.
+- Preserved Ruby 3.3 unsupported-rule metadata in generated frontends.
+- Fixed quote-specific escapes, escaped word boundaries and percent-regexp
+  delimiter escapes in generated Ruby frontends. Interpolated `%W` and `%I`
+  now fail explicitly instead of silently losing their interpolation.
+- Temporarily represent keyword arguments with Hash-shaped AST nodes while
+  RubyVM::AbstractSyntaxTree is used as the comparison oracle. This
+  representation may later be split into dedicated keyword-argument nodes as
+  Arpaka's semantic AST becomes the canonical model.
+
+## [1.0.0] - 2026-09-19
+
+- Added frontend generation directly from a Ruby `parse.y` file.
+- Added source profiles for the Ruby 3.3 and Ruby 3.4 grammars.
+- Loaded Ruby's canonical token definitions from `defs/id.def` when generating
+  frontends from a Ruby source tree.
+- Made generated runtime extensions depend on the selected source profile.
+
+## [0.1.1] - 2026-09-15
+
+- Changed Ruby source profile mismatches and unknown Action IDs from generation
+  errors to warnings.
+
+## [0.1.0] - 2026-09-15
+
+- Initial release of the `Lrama::Ruby` backend for generating standalone Ruby
+  parsers from Lrama grammars.
+- Added `Arpaka` frontend generation from a pinned Ruby source tree, including
+  its standalone lexer, AST, builder and Action mappings.
+- Added parser compilation in isolated Ruby Boxes, recognizer generation and
+  support for retaining error productions.
+- Added a Ruby action scanner and compatibility coverage using Rails and Ruby
+  Trick programs.

data/README.md

@@ -21,6 +21,13 @@ location and regenerate when updating the grammar or Actions.
 The Ruby frontend is experimental and partial. The gem contains generation
 support, not a bundled `parse.y` or a pre-generated Ruby frontend.
 
+Successful parsing does not guarantee a complete or correct AST. Known losses
+include block parameters and qualified constant paths. `%W` and `%I` also have
+incorrect Action mappings; their interpolated forms currently raise a lexer
+error instead of returning a flattened AST. Do not rely on these forms for AST
+analysis. The AST comparison task below records agreement separately from parse
+success.
+
 ## Requirements
 
 Ruby 4.0 or later. Start the generator with `RUBY_BOX=1`: Ruby Box isolates the
@@ -46,6 +53,11 @@ gem "arpaka", git: "https://github.com/katsyoshi/arpaka"
 
 ## Usage
 
+For a detailed, task-oriented guide to the `Arpaka.generate` and
+`Arpaka.compile` APIs, the CLI, standalone frontend runtime, limitations, and
+the AST compatibility benchmark, see [docs/usage.md](docs/usage.md). The
+summary below keeps the main README focused on the supported workflow.
+
 Write a Yacc-style grammar with Ruby actions, such as
 [examples/calculator.y](examples/calculator.y):
 
@@ -145,10 +157,36 @@ RUBY_BOX=1 bundle exec arpaka generate \
 ```
 
 The output directory must already exist. `--class-name` defaults to
-`RubyParser` and must be a single Ruby constant name. Existing output files are
-preserved unless `--force` is supplied. A failed generation leaves the previous
-file intact. The output includes its source profile, generator version and
-copyright/license notices.
+`RubyParser` and accepts a constant name or a namespace path such as
+`MyLibrary::RubyParser`; each component must be a valid Ruby constant name.
+Existing output files are preserved unless `--force` is supplied. A failed
+generation leaves the previous file intact. The output includes its source
+profile, generator version and copyright/license notices.
+
+When only `parse.y` is available, Arpaka can replace its `RUBY_TOKEN(NAME)`
+declarations with token names and generate a frontend without the rest of the
+Ruby source tree:
+
+```sh
+RUBY_BOX=1 bundle exec arpaka generate \
+  --parse-y /path/to/parse.y \
+  --class-name MyRubyParser \
+  --output lib/my_library/ruby_parser.rb
+```
+
+This mode embeds Arpaka's own license notices. The `--ruby-source` mode remains
+the compatibility-preserving path when the Ruby source tree is available.
+
+The experimental sample plugin demonstrates the plugin command shape. It is
+selected by the first command argument and runs in its own Ruby Box; the host
+process writes the generated source:
+
+```sh
+RUBY_BOX=1 bundle exec arpaka sample generate \
+  -y /path/to/parser.y \
+  -c MyParser \
+  -o lib/my_library/parser.rb
+```
 
 The consumer only needs the generated file and Ruby 4.0 or later:
 
@@ -164,6 +202,14 @@ internal; callers do not need to supply tokens. Each call has fresh parsing and
 local-variable state. No Arpaka/Lrama gem, Ruby Box, grammar files, network access
 or filesystem writes are needed at runtime.
 
+For a lightweight tree of lexical definitions, add `--ast-type structured` to
+`arpaka generate`, or pass `ast_type: :structured` to `Arpaka.generate` or
+`Arpaka.compile`. The generated class name stays the same. Its `parse` method
+returns `AST::Source` with a filename and children for class, module,
+instance method, and singleton method declarations. Reopened classes and modules
+remain separate occurrences. See [the usage guide](docs/usage.md#structured-ast-output)
+for the node fields and limitations.
+
 For generation from Ruby, `Arpaka.generate` returns that same standalone source.
 To try a frontend in memory, use `Arpaka.compile`; both require `RUBY_BOX=1`:
 
@@ -259,10 +305,10 @@ requires an explicit Ruby source tree. `ruby:check` verifies the source profile,
 Action inventory, grammar transformation and generated frontend without editing
 artifacts. `ruby:generate` writes the standalone frontend to `OUTPUT`.
 
-`lib/arpaka/actions.rb` holds the Ruby Action mappings;
-`lib/arpaka/upstream_rules.json` records the expected expanded upstream rules.
+`lib/arpaka/ruby/actions.rb` holds the Ruby Action mappings;
+`lib/arpaka/ruby/upstream_rules.json` records the expected expanded upstream rules.
 Update these and the source checksums together when adding support for a changed
-Ruby grammar. Runtime templates live under `lib/arpaka/runtime/`.
+Ruby grammar. Runtime templates live under `lib/arpaka/ruby/runtime/`.
 
 Tests exercise the generated frontend and the generic backend. Package checks
 install the gem, generate from external Ruby sources, and execute the resulting
@@ -297,6 +343,71 @@ Generation is a one-time build step per regeneration, not a cost of parsing an
 input file. The generated frontend is loaded and measured with Ruby Box disabled.
 Results vary with input and environment.
 
+### AST compatibility evidence
+
+The correctness benchmark compares the generated frontend's AST with
+`RubyVM::AbstractSyntaxTree` for every `.rb` file in explicitly supplied local
+source trees. It does not execute those files. Generate once with Ruby Box
+enabled; the comparison runs in a separate process with Ruby Box disabled:
+
+```sh
+RUBY_BOX=1 bundle exec ruby -Ilib benchmark/compatibility.rb \
+  --frontend local/tmp/compatibility-frontend.rb \
+  --frontend-class CompatibilityFrontend \
+  --corpus rails=/path/to/rails \
+  --corpus trick13=/path/to/trick13 \
+  --report local/tmp/arpaka-compatibility.jsonl
+```
+
+For a generated frontend named `A::B::Parser`, pass
+`--frontend-class A::B::Parser`. The default `CompatibilityFrontend` remains
+available for existing top-level frontends.
+
+The Rake entry point accepts the same corpus selections through `CORPORA`,
+with optional `REPORT` and `RUBY_SOURCE`:
+
+```sh
+RUBY_BOX=1 CORPORA='rails=/path/to/rails trick13=/path/to/trick13' \
+  REPORT=/tmp/arpaka-compatibility.jsonl bundle exec rake benchmark:compatibility
+```
+
+AST comparison currently covers a basic syntax subset. A file is a match only
+when both entire ASTs can be normalized and agree. Differences are mismatches,
+except that a call's explicit keyword arguments are reported as
+`accepted_mismatch`: Arpaka retains them as `KeywordArguments`, while
+`RubyVM::AbstractSyntaxTree` represents them as `HASH`. This exception applies
+only at a call argument position; Hash literals and `Pair` nodes are compared
+normally. Unknown or ambiguous nodes make the file unverified, including
+unknown nodes nested inside otherwise supported constructs. Parsing failures
+and per-parser 10-second timeouts are reported separately. A successful parse
+by itself never counts as AST agreement. Missing semantic information is not
+reconstructed from the input or the reference parser. When absent metadata
+would describe an empty parameter list or a regexp without flags, the result is
+unverified rather than assuming agreement or a defect.
+
+The JSONL report records runtime and revision information, source and generated
+frontend hashes, file-level outcomes, comparison diagnostics, and totals. These
+identify the exact inputs and distinguish complete matches, accepted
+representation differences and comparisons that remain unsupported. Internal
+comparison errors are also distinct from parser failures. The timeout includes
+worker startup and AST normalization. Exit status is 0 when every file is a
+match or accepted mismatch, 1 for any other mismatch, unverified file, parsing
+failure, timeout or internal error, and 2 for invalid configuration. The
+current frontend has known semantic losses, so a corpus run can legitimately
+exit with status 1. Reports are local artifacts; external
+corpora are not downloaded or included in the gem.
+
+The normal test suite checks the comparison machinery and known semantic-loss
+examples. External corpus comparison is an explicit task, separate from the
+speed benchmark and the default CI checks.
+
+The normal compatibility tests run the Thread path. Ruby Box/Ractor
+compatibility tests are currently omitted by default because of Ruby 4.0's
+Ruby Box/Ractor constraints; opt into those tests with
+`ARPAKA_RUN_RACTOR_TESTS=1`. This test-suite setting is separate from the
+compatibility runner's `--parallel ractor` mode, which remains available as an
+experimental, explicitly selected benchmark mode.
+
 ## License
 
 [MIT](LICENSE.txt).

data/Rakefile

@@ -2,11 +2,15 @@
 
 require "bundler/gem_tasks"
 require "rake/testtask"
+require "shellwords"
 
 Rake::TestTask.new(:test) do |t|
   t.libs << "test"
   t.libs << "lib"
-  t.test_files = FileList["test/**/*_test.rb"]
+  # The compatibility check exercises Ruby::Box and Ractor in a dedicated
+  # process. Keep it out of the regular suite so Box state cannot leak into
+  # the other tests.
+  t.test_files = FileList["test/**/*_test.rb"].exclude("test/benchmark_compatibility_test.rb")
 end
 
 namespace :ruby do
@@ -36,4 +40,14 @@ namespace :benchmark do
   task :parse do
     ruby "benchmark/parse.rb"
   end
+
+  desc "Compare generated Ruby ASTs with RubyVM (requires CORPORA; optional REPORT and RUBY_SOURCE)"
+  task :compatibility do
+    arguments = Shellwords.split(ENV.fetch("CORPORA", "")).flat_map do |corpus|
+      ["--corpus", corpus]
+    end
+    arguments += ["--report", ENV.fetch("REPORT")] if ENV["REPORT"]
+    arguments += ["--ruby-source", ENV.fetch("RUBY_SOURCE")] if ENV["RUBY_SOURCE"]
+    ruby "benchmark/compatibility.rb", *arguments
+  end
 end

data/docs/architecture.dot

@@ -0,0 +1,30 @@
+digraph arpaka_architecture {
+  graph [bgcolor="white", rankdir=TB, pad="0.25", nodesep="0.45", ranksep="0.65"];
+  node [shape=box, style="rounded,filled", fontname="Helvetica", fontsize=12,
+        color="#334155", fillcolor="#f8fafc", margin="0.16,0.10"];
+  edge [color="#64748b", penwidth=1.2, arrowsize=0.75, fontname="Helvetica", fontsize=10];
+
+  parse_y [label="Ruby parse.y\n(Yacc/Bison grammar)", fillcolor="#e0f2fe"];
+  lrama_parser [label="Lrama::Parser\nread grammar"];
+  grammar [label="Grammar model\nYaccRule / Actions"];
+  ruby_backend [label="Lrama::Ruby backend\ngenerate Ruby Parser", fillcolor="#dcfce7"];
+  generated_parser [label="Generated Ruby Parser\nparse table + reduce actions", fillcolor="#dcfce7"];
+
+  ruby_source [label="Ruby source", fillcolor="#fef3c7"];
+  lexer [label="Arpaka Lexer\nsource → tokens", fillcolor="#fef3c7"];
+  builder [label="Arpaka Builder\nsemantic values → AST", fillcolor="#fce7f3"];
+  ast [label="Arpaka AST\nindependent representation", fillcolor="#fce7f3"];
+  frontend [label="Generated Ruby entry file\nLexer + Parser + Builder", fillcolor="#ede9fe"];
+
+  oracle [label="RubyVM::AST\nvalidation oracle", shape=note, fillcolor="#f1f5f9"];
+  comparator [label="Compatibility comparator\nmatch / accepted / mismatch / unverified", shape=note, fillcolor="#f1f5f9"];
+
+  parse_y -> lrama_parser -> grammar -> ruby_backend -> generated_parser;
+  ruby_source -> lexer -> generated_parser;
+  generated_parser -> builder -> ast;
+  generated_parser -> frontend [style=dashed, label="embedded in"];
+  lexer -> frontend [style=dashed, label="embedded in"];
+  builder -> frontend [style=dashed, label="embedded in"];
+  ast -> comparator [style=dashed, label="compare"];
+  oracle -> comparator [style=dashed, label="reference"];
+}

data/docs/architecture.md

@@ -0,0 +1,96 @@
+# Arpaka architecture
+
+Arpaka uses Lrama to generate a Ruby parser from Ruby's `parse.y`. The generated
+frontend combines a Lexer, the generated Parser, and a Builder, then returns an
+Arpaka-specific AST.
+
+![Parser architecture](architecture.webp)
+
+## Generation pipeline
+
+The generation pipeline starts with the grammar definition rather than with a
+parser implementation written by hand:
+
+```text
+parse.y / Yacc grammar
+        ↓
+Lrama::Parser
+        ↓
+Grammar model / YaccRule
+        ↓
+Lrama::Ruby backend
+        ↓
+Generated Ruby Parser
+```
+
+Lrama owns grammar parsing, rule preparation, parser tables, and shift/reduce
+execution. The Ruby backend turns those structures into standalone Ruby source.
+Grammar actions are embedded as Ruby reduce actions; v1 does not introduce a
+language-independent semantic-action IR.
+
+## Runtime pipeline
+
+The generated Parser consumes tokens produced by the Arpaka Lexer. The Builder
+turns semantic values produced by reductions into Arpaka AST nodes.
+
+```text
+Ruby source → Lexer → Parser → Builder → Arpaka AST
+```
+
+The generated Ruby entry file packages these components together. It exposes a
+Ruby API for parsing source and does not require the Lrama or Arpaka gems at
+runtime.
+
+The Lexer is a separate component because Lrama generates parsers, not Ruby
+lexers. It handles Ruby's lexical rules, including strings, interpolation,
+regular expressions, percent literals, heredocs, escapes, and context-sensitive
+token boundaries.
+
+## Responsibilities
+
+- **Lrama** reads Yacc/Bison grammar, builds the grammar model, computes LALR or
+  IELR tables, and executes grammar actions through the generated parser.
+- **The Ruby backend** emits the parser table, parser runtime, and Ruby grammar
+  actions as Ruby source.
+- **The Arpaka Lexer** converts Ruby source into the token stream expected by
+  `parse.y`.
+- **The Builder** resolves scopes and semantic values and constructs AST nodes.
+- **The Arpaka AST** represents Ruby syntax and the semantic information needed
+  by later consumers. It is an independent representation, not a copy of
+  RubyVM's internal node layout.
+- **The compatibility comparator** uses RubyVM::AST as an Oracle for finding
+  missing or incorrectly interpreted information. It allows representation
+  differences when the meaning is equivalent.
+
+## Profiles and Actions
+
+Ruby's grammar changes across versions. Profiles select the source revision,
+runtime features, and Action mappings for a particular grammar. Rule signatures
+are used where numeric rule IDs are unstable; legacy profiles retain mappings
+for older Ruby grammars.
+
+An Action in v1 means the semantic action attached to a Yacc grammar rule. The
+profile maps that action to Ruby Builder code. A common semantic-action adapter
+between grammar actions and language-specific Builders is a possible future
+extension, but it is not required by the v1 architecture.
+
+## AST and validation
+
+Arpaka's AST is designed for a Ruby frontend, so node names and wrappers do not
+need to match RubyVM::AST exactly. RubyVM::AST provides an Oracle for checking
+whether the frontend preserves relevant syntax and meaning. The comparator
+classifies results as `match`, `accepted_mismatch`, `mismatch`, or `unverified`.
+
+Some RubyVM metadata is intentionally outside the v1 AST contract. Examples
+include information whose meaning is tied to RubyVM internals or cases where
+the reference AST does not expose enough information to distinguish two
+representations. Those cases remain explicitly classified instead of being
+silently accepted.
+
+## Scope of v1
+
+v1 targets Ruby 4.0's `parse.y` and provides a generated Ruby frontend with a
+Lexer, Parser, Builder, and independent AST. Ruby 3.3 and 3.4 profiles are
+additional compatibility targets. Broader version support, language-neutral
+semantic actions, and other-language backends can be introduced later without
+making them prerequisites for the v1 frontend.

data/docs/contributes.md

@@ -0,0 +1,115 @@
+# Compatibility checks
+
+This repository has two kinds of compatibility checks. They serve different
+purposes and should not be treated as interchangeable.
+
+## Pull request CI
+
+Pull request CI is the fast, required check for the repository itself. It
+verifies that the generated frontend and the test suite still work on the
+supported Ruby versions.
+
+The compatibility runner's Ractor mode is experimental. It is separate from
+the repository test suite: ordinary tests execute the Thread compatibility
+path, while Ruby Box/Ractor compatibility test cases are currently omitted by
+default. To opt into those test cases, set
+`ARPAKA_RUN_RACTOR_TESTS=1`. Generate a frontend first in a Ruby Box, then run
+the runner without a Ruby Box:
+
+```sh
+RUBY_BOX=1 OUTPUT=/tmp/arpaka-frontend.rb bundle exec rake ruby:generate
+
+bundle exec ruby benchmark/compatibility.rb \
+  --ruby-source vendor/ruby \
+  --corpus fixtures=benchmark/fixtures \
+  --frontend /tmp/arpaka-frontend.rb \
+  --reference-cache /path/to/reference.jsonl \
+  --threads 2 \
+  --parallel ractor
+```
+
+The runner's default mode remains Thread-based. Ractor checks should not make
+ordinary pull request validation depend on large external repositories, and
+the test-suite omission is not a change to the runner's `--parallel ractor`
+option.
+
+## Weekly compatibility inspection
+
+The weekly inspection is intended for larger real-world inputs, especially the
+Rails repository and the TRICK programs. It is a diagnostic and compatibility
+inspection, not a replacement for pull request CI.
+
+The external repositories should be checked out by the CI job at the commit
+being inspected. The RubyVM Oracle JSONL should then be generated in the same
+job and with the same Ruby runtime used for the comparison. The Oracle is
+temporary job data; it should not be committed to this repository.
+
+The inspection should record the external repository commits, Ruby runtime,
+Ractor count, GC setting and result summary. The comparison report should be
+uploaded as an Artifact. The much larger Oracle should normally remain in the
+job workspace and only be uploaded for a failed or explicitly requested run.
+
+### Reference benchmark
+
+The following measurement was made with Ruby 4.0.7 against the Rails
+repository and TRICK 2013, 2015, 2018, 2022 and 2025 sources. The corpus
+contained 3,525 Ruby files and both modes used `--threads 32`.
+
+| Mode | Elapsed time | Heap allocated pages | Heap live slots | Minor GC | Major GC |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Thread | 6.85 s | 1,483 | 131,390 | 5 | 3 |
+| Thread + `--disable-gc` | 6.62 s | 1,484 | 131,433 | 5 | 3 |
+| Ractor | 15.66 s | 4,795 | 4,445,551 | 128 | 10 |
+| Ractor + `--disable-gc` | 10.42 s | 101,705 | 131,481,029 | 2 | 1 |
+
+In this run, Ractor mode was about 2.3 times slower than Thread mode with GC
+enabled. Disabling GC reduced the Ractor time by about 33%, but increased heap
+usage substantially. These numbers are a reference point rather than a
+performance guarantee; Ruby versions, CPU count, corpus revisions and worker
+counts can change the result.
+
+## Ractor mode
+
+Ractor workers share the generated frontend class loaded by the parent Ractor.
+The frontend is loaded once per run so that each corpus does not redefine its
+constants.
+
+Use `--threads` to control the number of concurrent workers. Ractor mode has
+additional object-transfer and heap costs, so the best worker count may differ
+from the Thread mode.
+
+## Disabling GC
+
+`--disable-gc` is an opt-in performance experiment. It applies to both Thread
+and Ractor workers; GC remains enabled by default.
+
+```sh
+bundle exec ruby benchmark/compatibility.rb \
+  --corpus fixtures=benchmark/fixtures \
+  --frontend /tmp/arpaka-frontend.rb \
+  --reference-cache /path/to/reference.jsonl \
+  --parallel ractor \
+  --threads 32 \
+  --disable-gc
+```
+
+Disabling GC can improve throughput substantially, but it can also grow the
+Ruby heap dramatically. Use it only on machines with ample memory and inspect
+the `GC.stat` values in the report. It should not be enabled by default in CI.
+
+## Reading results
+
+The report distinguishes parser outcomes from tree comparison results:
+
+- `match`: normalized trees are equal.
+- `accepted_mismatch`: a known, explicitly accepted representation difference.
+- `mismatch`: normalized trees differ.
+- `unverified`: one side uses a construct outside the comparison model.
+- `parse_failure`: parsing failed.
+- `timeout`: a worker exceeded its time limit.
+- `error`: an unexpected worker or input error occurred.
+
+Existing `mismatch` and `unverified` results are not automatically evidence of
+a Ractor failure. Compare Thread and Ractor runs using the same corpus,
+reference cache and Ruby runtime; Ractor-specific failures are parse failures,
+timeouts, worker errors, or result differences that appear only in Ractor mode.