lrama 0.6.2 → 0.6.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e4158de45c42ff62eacfb00737261feaa49d8f0cc646004e30da74ba4e2e69c6
-  data.tar.gz: 734830227f701e18df2e9e8bc3da55d15f49c890e08530e6ac55ef87ae5f952d
+  metadata.gz: ce98751a4b4d20c20addf8eebf2a4eb22150b401abbb994941d73f2b2ef81a09
+  data.tar.gz: 94b83050b10ec5d01d61093cd2af595e52b3761cc8e86423fa8f30110e44e330
 SHA512:
-  metadata.gz: 52ebbe4d099ae63d73aa995bddc8e966f989a4d00ad3b39634d2abe2448da404dd9bff8f15e0dedd0577716089329c804ef2c4edcadd39ca6ba47f8d293d101d
-  data.tar.gz: 72e91c79618071b5850c85335cfe3f1b63ff89f11cd332b0141623a4e2a7e2c2c389dd0db9afe38a5a84b7aa891ac1834fc7a2d6c6eed8f62d87734f6b99cbbf
+  metadata.gz: e9e194703ec0b1657389f289b7c4d9cfd13326b1efe25e656080e5260ef2b7f3d4e93fd3d5d49e0934cfb394bc46b994c167a74c3dd2fb2ed3f3ebaaaf31647d
+  data.tar.gz: 59e1ed986bbea4ea4a56295d799f6ef5afee24e128fec66bc7ea8184f9bd849aa0807b5995b201e3dda274f6f0f023899a8c287e9a25818dae75c05e5c45cf6a

data/.github/workflows/test.yaml

@@ -124,9 +124,8 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        # '3.0' is the oldest living ruby version
-        # '2.7' is for BASERUBY
-        baseruby: ['head', '3.0', '2.7']
+        # '3.0' is the oldest living ruby version and minimal BASERUBY version
+        baseruby: ['head', '3.0']
         ruby_branch: ['master']
     defaults:
       run:

data/Gemfile

@@ -12,6 +12,6 @@ gem "stackprof", platforms: [:ruby] # stackprof doesn't support Windows
 # Recent steep requires Ruby >= 3.0.0.
 # Then skip install on some CI jobs.
 if !ENV['GITHUB_ACTION'] || ENV['INSTALL_STEEP'] == 'true'
-  gem "rbs", "3.4.1", require: false
+  gem "rbs", "3.4.4", require: false
   gem "steep", "1.6.0", require: false
 end

data/NEWS.md

@@ -1,5 +1,105 @@
 # NEWS for Lrama
 
+## Lrama 0.6.4 (2024-03-22)
+
+### Parameterizing rules (preceded, terminated, delimited)
+
+Support `preceded`, `terminated` and `delimited` rules.
+
+```
+program: preceded(opening, X)
+
+// Expanded to
+
+program: preceded_opening_X
+preceded_opening_X: opening X
+```
+
+```
+program: terminated(X, closing)
+
+// Expanded to
+
+program: terminated_X_closing
+terminated_X_closing: X closing
+```
+
+```
+program: delimited(opening, X, closing)
+
+// Expanded to
+
+program: delimited_opening_X_closing
+delimited_opening_X_closing: opening X closing
+```
+
+https://github.com/ruby/lrama/pull/382
+
+### Support `%destructor` declaration
+
+User can set codes for freeing semantic value resources by using `%destructor`.
+In general, these resources are freed by actions or after parsing.
+However if syntax error happens in parsing, these codes may not be executed.
+Codes associated to `%destructor` are executed when semantic value is popped from the stack by an error.
+
+```
+%token <val1> NUM
+%type <val2> expr2
+%type <val3> expr
+
+%destructor {
+    printf("destructor for val1: %d\n", $$);
+} <val1> // printer for TAG
+
+%destructor {
+    printf("destructor for val2: %d\n", $$);
+} <val2>
+
+%destructor {
+    printf("destructor for expr: %d\n", $$);
+} expr // printer for symbol
+```
+
+Bison supports this feature from 1.75b.
+
+https://github.com/ruby/lrama/pull/385
+
+## Lrama 0.6.3 (2024-02-15)
+
+### Bring Your Own Stack
+
+Provide functionalities for Bring Your Own Stack.
+
+Ruby’s Ripper library requires their own semantic value stack to manage Ruby Objects returned by user defined callback method. Currently Ripper uses semantic value stack (`yyvsa`) which is used by parser to manage Node. This hack introduces some limitation on Ripper. For example, Ripper can not execute semantic analysis depending on Node structure.
+
+Lrama introduces two features to support another semantic value stack by parser generator users.
+
+1. Callback entry points
+
+User can emulate semantic value stack by these callbacks.
+Lrama provides these five callbacks. Registered functions are called when each event happen. For example %after-shift function is called when shift happens on original semantic value stack.
+
+* `%after-shift` function_name
+* `%before-reduce` function_name
+* `%after-reduce` function_name
+* `%after-shift-error-token` function_name
+* `%after-pop-stack` function_name
+
+2. `$:n` variable to access index of each grammar symbols
+
+User also needs to access semantic value of their stack in grammar action. `$:n` provides the way to access to it. `$:n` is translated to the minus index from the top of the stack.
+For example
+
+```
+primary: k_if expr_value then compstmt if_tail k_end
+          {
+          /*% ripper: if!($:2, $:4, $:5) %*/
+          /* $:2 = -5, $:4 = -3, $:5 = -2. */
+          }
+```
+
+https://github.com/ruby/lrama/pull/367
+
 ## Lrama 0.6.2 (2024-01-27)
 
 ### %no-stdlib directive
@@ -17,7 +117,7 @@ Allow to pass an instantiated rule to other parameterizing rules.
 
 ```
 %rule constant(X) : X
-              ;
+                  ;
 
 %rule option(Y) : /* empty */
                 | Y

data/README.md

@@ -1,7 +1,23 @@
 # Lrama
 
+[![Gem Version](https://badge.fury.io/rb/lrama.svg)](https://badge.fury.io/rb/lrama)
+[![build](https://github.com/ruby/lrama/actions/workflows/test.yaml/badge.svg)](https://github.com/ruby/lrama/actions/workflows/test.yaml)
+
 Lrama is LALR (1) parser generator written by Ruby. The first goal of this project is providing error tolerant parser for CRuby with minimal changes on CRuby parse.y file.
 
+* [Features](#features)
+* [Installation](#installation)
+* [Usage](#usage)
+* [Versions and Branches](#versions-and-branches)
+* [Supported Ruby version](#supported-ruby-version)
+* [Development](#development)
+  * [How to generate parser.rb](#how-to-generate-parserrb)
+  * [Test](#test)
+  * [Profiling Lrama](#profiling-lrama)
+  * [Build Ruby](#build-ruby)
+* [Release flow](#release-flow)
+* [License](#license)
+
 ## Features
 
 * Bison style grammar file is supported with some assumptions
@@ -11,6 +27,9 @@ Lrama is LALR (1) parser generator written by Ruby. The first goal of this proje
   * b4_lac_if is always false
 * Error Tolerance parser
   * Subset of [Repairing Syntax Errors in LR Parsers (Corchuelo et al.)](https://idus.us.es/bitstream/handle/11441/65631/Repairing%20syntax%20errors.pdf) algorithm is supported
+* Parameterizing rules
+  * The definition of a non-terminal symbol can be parameterized with other (terminal or non-terminal) symbols.
+  * Providing a generic definition of parameterizing rules as a [standard library](lib/lrama/grammar/stdlib.y).
 
 ## Installation
 
@@ -85,6 +104,8 @@ Running tests:
 ```shell
 $ bundle install
 $ bundle exec rspec
+# or
+$ bundle exec rake spec
 ```
 
 Running type check:
@@ -93,6 +114,8 @@ Running type check:
 $ bundle install
 $ bundle exec rbs collection install
 $ bundle exec steep check
+# or
+$ bundle exec rake steep
 ```
 
 Running both of them:

data/Steepfile

@@ -5,21 +5,26 @@ target :lib do
   signature "sig"
 
   check "lib/lrama/grammar/binding.rb"
+  check "lib/lrama/grammar/code/destructor_code.rb"
   check "lib/lrama/grammar/code/printer_code.rb"
   check "lib/lrama/grammar/code.rb"
   check "lib/lrama/grammar/counter.rb"
   check "lib/lrama/grammar/error_token.rb"
   check "lib/lrama/grammar/parameterizing_rule"
   check "lib/lrama/grammar/parameterizing_rules"
+  check "lib/lrama/grammar/symbols"
   check "lib/lrama/grammar/percent_code.rb"
   check "lib/lrama/grammar/precedence.rb"
+  check "lib/lrama/grammar/destructor.rb"
   check "lib/lrama/grammar/printer.rb"
   check "lib/lrama/grammar/reference.rb"
   check "lib/lrama/grammar/rule_builder.rb"
   check "lib/lrama/grammar/symbol.rb"
+  check "lib/lrama/grammar/type.rb"
   check "lib/lrama/lexer"
   check "lib/lrama/report"
   check "lib/lrama/bitmap.rb"
   check "lib/lrama/digraph.rb"
+  check "lib/lrama/options.rb"
   check "lib/lrama/warning.rb"
 end

data/lib/lrama/context.rb

@@ -265,9 +265,9 @@ module Lrama
 
         s = actions.each_with_index.map do |n, i|
           [i, n]
-        end.select do |i, n|
+        end.reject do |i, n|
           # Remove default_reduction_rule entries
-          n != 0
+          n == 0
         end
 
         if s.count != 0
@@ -462,7 +462,7 @@ module Lrama
       @yylast = high
 
       # replace_ninf
-      @yypact_ninf = (@base.select {|i| i != BaseMin } + [0]).min - 1
+      @yypact_ninf = (@base.reject {|i| i == BaseMin } + [0]).min - 1
       @base.map! do |i|
         case i
         when BaseMin
@@ -472,7 +472,7 @@ module Lrama
         end
       end
 
-      @yytable_ninf = (@table.compact.select {|i| i != ErrorActionNumber } + [0]).min - 1
+      @yytable_ninf = (@table.compact.reject {|i| i == ErrorActionNumber } + [0]).min - 1
       @table.map! do |i|
         case i
         when nil

data/lib/lrama/grammar/code/destructor_code.rb

@@ -0,0 +1,40 @@
+module Lrama
+  class Grammar
+    class Code
+      class DestructorCode < Code
+        def initialize(type:, token_code:, tag:)
+          super(type: type, token_code: token_code)
+          @tag = tag
+        end
+
+        private
+
+        # * ($$) *yyvaluep
+        # * (@$) *yylocationp
+        # * ($:$) error
+        # * ($1) error
+        # * (@1) error
+        # * ($:1) error
+        def reference_to_c(ref)
+          case
+          when ref.type == :dollar && ref.name == "$" # $$
+            member = @tag.member
+            "((*yyvaluep).#{member})"
+          when ref.type == :at && ref.name == "$" # @$
+            "(*yylocationp)"
+          when ref.type == :index && ref.name == "$" # $:$
+            raise "$:#{ref.value} can not be used in #{type}."
+          when ref.type == :dollar # $n
+            raise "$#{ref.value} can not be used in #{type}."
+          when ref.type == :at # @n
+            raise "@#{ref.value} can not be used in #{type}."
+          when ref.type == :index # $:n
+            raise "$:#{ref.value} can not be used in #{type}."
+          else
+            raise "Unexpected. #{self}, #{ref}"
+          end
+        end
+      end
+    end
+  end
+end

data/lib/lrama/grammar/code/initial_action_code.rb

@@ -6,18 +6,24 @@ module Lrama
 
         # * ($$) yylval
         # * (@$) yylloc
+        # * ($:$) error
         # * ($1) error
         # * (@1) error
+        # * ($:1) error
         def reference_to_c(ref)
           case
           when ref.type == :dollar && ref.name == "$" # $$
             "yylval"
           when ref.type == :at && ref.name == "$" # @$
             "yylloc"
+          when ref.type == :index && ref.name == "$" # $:$
+            raise "$:#{ref.value} can not be used in initial_action."
           when ref.type == :dollar # $n
             raise "$#{ref.value} can not be used in initial_action."
           when ref.type == :at # @n
             raise "@#{ref.value} can not be used in initial_action."
+          when ref.type == :index # $:n
+            raise "$:#{ref.value} can not be used in initial_action."
           else
             raise "Unexpected. #{self}, #{ref}"
           end

data/lib/lrama/grammar/code/no_reference_code.rb

@@ -6,14 +6,18 @@ module Lrama
 
         # * ($$) error
         # * (@$) error
+        # * ($:$) error
         # * ($1) error
         # * (@1) error
+        # * ($:1) error
         def reference_to_c(ref)
           case
           when ref.type == :dollar # $$, $n
             raise "$#{ref.value} can not be used in #{type}."
           when ref.type == :at # @$, @n
             raise "@#{ref.value} can not be used in #{type}."
+          when ref.type == :index # $:$, $:n
+            raise "$:#{ref.value} can not be used in #{type}."
           else
             raise "Unexpected. #{self}, #{ref}"
           end

data/lib/lrama/grammar/code/printer_code.rb

@@ -11,8 +11,10 @@ module Lrama
 
         # * ($$) *yyvaluep
         # * (@$) *yylocationp
+        # * ($:$) error
         # * ($1) error
         # * (@1) error
+        # * ($:1) error
         def reference_to_c(ref)
           case
           when ref.type == :dollar && ref.name == "$" # $$
@@ -20,10 +22,14 @@ module Lrama
             "((*yyvaluep).#{member})"
           when ref.type == :at && ref.name == "$" # @$
             "(*yylocationp)"
+          when ref.type == :index && ref.name == "$" # $:$
+            raise "$:#{ref.value} can not be used in #{type}."
           when ref.type == :dollar # $n
             raise "$#{ref.value} can not be used in #{type}."
           when ref.type == :at # @n
             raise "@#{ref.value} can not be used in #{type}."
+          when ref.type == :index # $:n
+            raise "$:#{ref.value} can not be used in #{type}."
           else
             raise "Unexpected. #{self}, #{ref}"
           end

data/lib/lrama/grammar/code/rule_action.rb

@@ -11,8 +11,10 @@ module Lrama
 
         # * ($$) yyval
         # * (@$) yyloc
+        # * ($:$) error
         # * ($1) yyvsp[i]
         # * (@1) yylsp[i]
+        # * ($:1) i - 1
         #
         #
         # Consider a rule like
@@ -24,6 +26,8 @@ module Lrama
         # "Rule"                class: keyword_class { $1 } tSTRING { $2 + $3 } keyword_end { $class = $1 + $keyword_end }
         # "Position in grammar"                   $1     $2      $3          $4          $5
         # "Index for yyvsp"                       -4     -3      -2          -1           0
+        # "$:n"                                  $:1    $:2     $:3         $:4         $:5
+        # "index of $:n"                          -5     -4      -3          -2          -1
         #
         #
         # For the first midrule action:
@@ -31,6 +35,7 @@ module Lrama
         # "Rule"                class: keyword_class { $1 } tSTRING { $2 + $3 } keyword_end { $class = $1 + $keyword_end }
         # "Position in grammar"                   $1
         # "Index for yyvsp"                        0
+        # "$:n"                                  $:1
         def reference_to_c(ref)
           case
           when ref.type == :dollar && ref.name == "$" # $$
@@ -39,6 +44,8 @@ module Lrama
             "(yyval.#{tag.member})"
           when ref.type == :at && ref.name == "$" # @$
             "(yyloc)"
+          when ref.type == :index && ref.name == "$" # $:$
+            raise "$:$ is not supported"
           when ref.type == :dollar # $n
             i = -position_in_rhs + ref.index
             tag = ref.ex_tag || rhs[ref.index - 1].tag
@@ -47,6 +54,9 @@ module Lrama
           when ref.type == :at # @n
             i = -position_in_rhs + ref.index
             "(yylsp[#{i}])"
+          when ref.type == :index # $:n
+            i = -position_in_rhs + ref.index
+            "(#{i} - 1)"
           else
             raise "Unexpected. #{self}, #{ref}"
           end
@@ -70,7 +80,7 @@ module Lrama
         end
 
         def raise_tag_not_found_error(ref)
-          raise "Tag is not specified for '$#{ref.value}' in '#{@rule.to_s}'"
+          raise "Tag is not specified for '$#{ref.value}' in '#{@rule}'"
         end
       end
     end

data/lib/lrama/grammar/code.rb

@@ -1,4 +1,5 @@
 require "forwardable"
+require "lrama/grammar/code/destructor_code"
 require "lrama/grammar/code/initial_action_code"
 require "lrama/grammar/code/no_reference_code"
 require "lrama/grammar/code/printer_code"

data/lib/lrama/grammar/destructor.rb

@@ -0,0 +1,9 @@
+module Lrama
+  class Grammar
+    class Destructor < Struct.new(:ident_or_tags, :token_code, :lineno, keyword_init: true)
+      def translated_code(tag)
+        Code::DestructorCode.new(type: :destructor, token_code: token_code, tag: tag).translated_code
+      end
+    end
+  end
+end

data/lib/lrama/grammar/reference.rb

@@ -2,11 +2,12 @@ module Lrama
   class Grammar
     # type: :dollar or :at
     # name: String (e.g. $$, $foo, $expr.right)
-    # index: Integer (e.g. $1)
+    # number: Integer (e.g. $1)
+    # index: Integer
     # ex_tag: "$<tag>1" (Optional)
-    class Reference < Struct.new(:type, :name, :index, :ex_tag, :first_column, :last_column, keyword_init: true)
+    class Reference < Struct.new(:type, :name, :number, :index, :ex_tag, :first_column, :last_column, keyword_init: true)
       def value
-        name || index
+        name || number
       end
     end
   end

data/lib/lrama/grammar/rule_builder.rb

@@ -115,12 +115,12 @@ module Lrama
               @replaced_rhs << lhs_token
               parameterizing_rule_resolver.created_lhs_list << lhs_token
               parameterizing_rule.rhs_list.each do |r|
-                rule_builder = RuleBuilder.new(@rule_counter, @midrule_action_counter, i, lhs_tag: token.lhs_tag, skip_preprocess_references: true)
+                rule_builder = RuleBuilder.new(@rule_counter, @midrule_action_counter, lhs_tag: token.lhs_tag, skip_preprocess_references: true)
                 rule_builder.lhs = lhs_token
                 r.symbols.each { |sym| rule_builder.add_rhs(bindings.resolve_symbol(sym)) }
                 rule_builder.line = line
-                rule_builder.user_code = r.user_code
                 rule_builder.precedence_sym = r.precedence_sym
+                rule_builder.user_code = r.user_code
                 rule_builder.complete_input
                 rule_builder.setup_rules(parameterizing_rule_resolver)
                 @rule_builders_for_parameterizing_rules << rule_builder
@@ -181,11 +181,18 @@ module Lrama
                 if referring_symbol[1] == 0 # Refers to LHS
                   ref.name = '$'
                 else
-                  ref.index = referring_symbol[1]
+                  ref.number = referring_symbol[1]
                 end
               end
             end
 
+            if ref.number
+              # TODO: When Inlining is implemented, for example, if `$1` is expanded to multiple RHS tokens,
+              #       `$2` needs to access `$2 + n` to actually access it. So, after the Inlining implementation,
+              #       it needs resolves from number to index.
+              ref.index = ref.number
+            end
+
             # TODO: Need to check index of @ too?
             next if ref.type == :at
 

data/lib/lrama/grammar/stdlib.y

@@ -8,6 +8,9 @@
 
 **********************************************************************/
 
+// -------------------------------------------------------------------
+// Options
+
 /*
  * program: option(number)
  *
@@ -21,6 +24,45 @@
                | X
                ;
 
+// -------------------------------------------------------------------
+// Sequences
+
+/*
+ * program: preceded(opening, X)
+ *
+ * =>
+ *
+ * program: preceded_opening_X
+ * preceded_opening_X: opening X
+ */
+%rule preceded(opening, X): opening X { $$ = $2; }
+                          ;
+
+/*
+ * program: terminated(X, closing)
+ *
+ * =>
+ *
+ * program: terminated_X_closing
+ * terminated_X_closing: X closing
+ */
+%rule terminated(X, closing): X closing { $$ = $1; }
+                            ;
+
+/*
+ * program: delimited(opening, X, closing)
+ *
+ * =>
+ *
+ * program: delimited_opening_X_closing
+ * delimited_opening_X_closing: opening X closing
+ */
+%rule delimited(opening, X, closing): opening X closing { $$ = $2; }
+                                     ;
+
+// -------------------------------------------------------------------
+// Lists
+
 /*
  * program: list(number)
  *

data/lib/lrama/grammar/symbol.rb

@@ -7,11 +7,12 @@
 module Lrama
   class Grammar
     class Symbol
-      attr_accessor :id, :alias_name, :tag, :number, :token_id, :nullable, :precedence, :printer, :error_token, :first_set, :first_set_bitmap
+      attr_accessor :id, :alias_name, :tag, :number, :token_id, :nullable, :precedence,
+                    :printer, :destructor, :error_token, :first_set, :first_set_bitmap
       attr_reader :term
       attr_writer :eof_symbol, :error_symbol, :undef_symbol, :accept_symbol
 
-      def initialize(id:, alias_name: nil, number: nil, tag: nil, term:, token_id: nil, nullable: nil, precedence: nil, printer: nil)
+      def initialize(id:, term:, alias_name: nil, number: nil, tag: nil, token_id: nil, nullable: nil, precedence: nil, printer: nil, destructor: nil)
         @id = id
         @alias_name = alias_name
         @number = number
@@ -21,6 +22,7 @@ module Lrama
         @nullable = nullable
         @precedence = precedence
         @printer = printer
+        @destructor = destructor
       end
 
       def term?