prism 0.19.0 → 0.20.0

data/docs/build_system.md

@@ -53,7 +53,7 @@ Because of course those files are not part of the git repository.
 
 ### Building prism as part of CRuby
 
-[This script](https://github.com/ruby/ruby/blob/32e828bb4a6c65a392b2300f3bdf93008c7b6f25/tool/sync_default_gems.rb#L399-L426) imports prism sources in CRuby.
+[This script](https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337c430cb93caa151/tool/sync_default_gems.rb#L399-L422) imports prism sources in CRuby.
 
 The script generates the templates when importing.
 
@@ -71,4 +71,21 @@ and links to `libprism.a` (to avoid exporting symbols, so no conflict when insta
 
 ### Building prism as part of JRuby
 
-TODO, probably similar to TruffleRuby.
+TODO, similar to TruffleRuby.
+
+### Building prism from source as a C library
+
+All of the source files match `src/**/*.c` and all of the headers match `include/**/*.h`.
+
+If you want to build prism as a shared library and link against it, you should compile with:
+
+* `-fPIC -shared` - Compile as a shared library
+* `-DPRISM_EXPORT_SYMBOLS` - Export the symbols (by default nothing is exported)
+
+#### Flags
+
+`make` respects the `MAKEFLAGS` environment variable. As such, to speed up the build you can run:
+
+```
+MAKEFLAGS="-j10" bundle exec rake compile
+```

data/docs/cruby_compilation.md

@@ -0,0 +1,27 @@
+# Compiling Prism's AST
+
+One important class of consumers of Prism's AST is compilers. Currently [CRuby](https://github.com/ruby/ruby), [JRuby](https://github.com/jruby/jruby), [TruffleRuby](https://github.com/oracle/truffleruby), and [Natalie](https://github.com/natalie-lang/natalie) have all built compilation code on top of Prism's AST.
+
+This document will describe, at a high level, how CRuby's compilation of Prism's AST works.
+
+As described in the [build system documentation](build_system.md), there is a "push" Webhook set up within the Prism repo triggered on each new commit to send information about the commit to [git.ruby-lang.org](https://github.com/ruby/git.ruby-lang.org). This in turn runs [a script](https://github.com/ruby/ruby/blob/master/tool/sync_default_gems.rb) to sync over new changes in Prism to their corresponding files in Ruby. Any failures in this sync script will show alerts in the #alerts-sync channel in the RubyLang Slack. The result of this step is that files are synced from Prism into ruby/ruby for its use. It is also worth noting that [`common.mk`](https://github.com/ruby/ruby/blob/master/common.mk) contains a list of Prism files which it needs to correctly compile. If there are new Prism files added, this file should also be updated.
+
+ruby/ruby uses the Prism code to generate an AST from which it can generate instruction sequences. Compilation in ruby/ruby has three main steps:
+
+1. Compute an AST
+
+Syncing over the Prism code allows ruby/ruby to compute the AST using Prism. It currently does this within [`iseq.c`](https://github.com/ruby/ruby/blob/master/iseq.c) using the `pm_parser_init` fuction.
+
+2. Run a first pass of compilation
+
+Once the AST has been created, it is recursively descended in order to compute the appropriate instruction sequences. This is the crux of compilation, and we go into more detail about nuances in the following paragraphs.
+
+The code for this step is almost exclusively in [`prism_compile.c`](https://github.com/ruby/ruby/blob/master/prism_compile.c). The main function used for compilation is `pm_compile_node` which is essentially a huge switch statement over practically every node type which computes the appropriate instruction sequences for that node type. There are several convenience helpers, such as `PM_COMPILE`, `PM_COMPILE_POPPED`, `PM_COMPILE_NOT_POPPED` which all call into the `pm_compile_node` function.
+
+There are also several functions, like `parse_string`, `parse_integer` which consume Prism nodes and return CRuby values. These are all called for their relevant types within the big switch statement.
+
+The Prism compiler also uses a concept of "scope nodes" which are not standard Prism nodes in the AST, but instead nodes constructed within the compiler for the sole purpose of making compilation easier. Scope nodes are defined in [`prism_compile.h`](https://github.com/ruby/ruby/blob/master/prism_compile.h) and store information such as locals, local table size, local depth offset and the index lookup tables. Scope nodes can be generated for node types which have their own "scope".
+
+3. Run an optimization pass of compilation
+
+After the instruction sequences are initially computed, there is an existing (non-Prism based) optimization pass of the instruction sequences. There are several optimizations currently inlined into step 2, however, most of them happen in this step. Specifically, any peephole optimizations happen in this step. By the end of step 2, however, the instruction sequences take the same form regardless of if the initial AST was generated by Prism or not. Therefore, step 3 is agnostic to the parser, and should not require any Prism specific code.

data/docs/parser_translation.md

@@ -0,0 +1,34 @@
+# parser translation
+
+Prism ships with the ability to translate its syntax tree into the syntax tree used by the [whitequark/parser](https://github.com/whitequark/parser) gem. This allows you to use tools built on top of the `parser` gem with the `prism` parser.
+
+## Usage
+
+The `parser` gem provides multiple parsers to support different versions of the Ruby grammar. This includes all of the Ruby versions going back to 1.8, as well as third-party parsers like MacRuby and RubyMotion. The `prism` gem provides another parser that uses the `prism` parser to build the syntax tree.
+
+You can use the `prism` parser like you would any other. After requiring the parser, you should be able to call any of the regular `Parser::Base` APIs that you would normally use.
+
+```ruby
+require "prism/translation/parser"
+
+Prism::Translation::Parser.parse_file("path/to/file.rb")
+```
+
+### RuboCop
+
+To run RuboCop using the `prism` gem as the parser, you will need to require the `prism/translation/parser/rubocop` file. This file injects `prism` into the known options for both `rubocop` and `rubocop-ast`, such that you can specify it in your `.rubocop.yml` file. Unfortunately `rubocop` doesn't support any direct way to do this, so we have to get a bit hacky.
+
+First, set the `TargetRubyVersion` in your RuboCop configuration file to `80_82_73_83_77.33`. This is the version of Ruby that `prism` reports itself as. (The leading numbers are the ASCII values for `PRISM`.)
+
+```yaml
+AllCops:
+  TargetRubyVersion: 80_82_73_83_77.33
+```
+
+Now when you run `rubocop` you will need to require the `prism/translation/parser/rubocop` file before executing so that it can inject the `prism` parser into the known options.
+
+```
+bundle exec ruby -rprism/translation/parser/rubocop $(bundle exec which rubocop)
+```
+
+This should run RuboCop using the `prism` parser.

data/docs/parsing_rules.md

@@ -0,0 +1,19 @@
+# Rules
+
+This document contains information related to the rules of the parser for Ruby source code.
+
+As an example, in the documentation of many of the fields of nodes, it's mentioned that a field follows the lexing rules for `identifier` or `constant`. This document describes what those rules are.
+
+## Constants
+
+Constants in Ruby begin with an upper-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "upper-case letter" are encoding-dependent.
+
+## Non-void expression
+
+Most expressions in CRuby are non-void. This means the expression they represent resolves to a value. For example, `1 + 2` is a non-void expression, because it resolves to a method call. Even things like `class Foo; end` is a non-void expression, because it returns the last evaluated expression in the body of the class (or `nil`).
+
+Certain nodes, however, are void expressions, and cannot be combined to form larger expressions. For example, `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo`.
+
+## Identifiers
+
+Identifiers in Ruby begin with an underscore or lower-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "lower-case letter" are encoding-dependent.

data/docs/releasing.md

@@ -13,13 +13,13 @@ To release a new version of Prism, perform the following steps:
   * `ext/prism/extension.h` in the `EXPECTED_PRISM_VERSION` macro
   * `include/prism/version.h` in the version macros
   * `javascript/package.json` in the `version` field
-  * `rust/prism-sys/tests/utils_tests.rs` in the `version_test` function
+  * `rust/ruby-prism-sys/tests/utils_tests.rs` in the `version_test` function
   * `templates/java/org/prism/Loader.java.erb` in the `load` function
   * `templates/javascript/src/deserialize.js.erb` in the version constants
   * `templates/lib/prism/serialize.rb.erb` in the version constants
 * Run `bundle install` to update the `Gemfile.lock` file.
-* Update `rust/prism-sys/Cargo.toml` to match the new version and run `cargo build`
-* Update `rust/prism/Cargo.toml` to match the new version and run `cargo build`
+* Update `rust/ruby-prism-sys/Cargo.toml` to match the new version and run `cargo build`
+* Update `rust/ruby-prism/Cargo.toml` to match the new version and run `cargo build`
 * Commit all of the updated files.
 
 ## Publishing

data/docs/ruby_api.md

@@ -14,7 +14,7 @@ The full API is documented below.
 
 ## API
 
-* `Prism.dump(source, filepath)` - parse the syntax tree corresponding to the given source string and filepath, and serialize it to a string. Filepath can be nil.
+* `Prism.dump(source)` - parse the syntax tree corresponding to the given source string, and serialize it to a string
 * `Prism.dump_file(filepath)` - parse the syntax tree corresponding to the given source file and serialize it to a string
 * `Prism.lex(source)` - parse the tokens corresponding to the given source string and return them as an array within a parse result
 * `Prism.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result

data/docs/serialization.md

@@ -51,12 +51,21 @@ The comment type is one of:
 | location | the location of the key of the magic comment |
 | location | the location of the value of the magic comment |
 
-### diagnostic
+### error
 
 | # bytes | field |
 | --- | --- |
-| string | diagnostic message (ASCII-only characters) |
-| location | the location in the source this diagnostic applies to |
+| string | error message (ASCII-only characters) |
+| location | the location in the source this error applies to |
+| `1` | the level of the error: `0` for `fatal` |
+
+## warning
+
+| # bytes | field |
+| --- | --- |
+| string | warning message (ASCII-only characters) |
+| location | the location in the source this warning applies to |
+| `1` | the level of the warning: `0` for `default` and `1` for `verbose` |
 
 ## Structure
 
@@ -82,9 +91,9 @@ The header is structured like the following table:
 | magic comment* | magic comments |
 | location? | the optional location of the `__END__` keyword and its contents |
 | varuint | number of errors |
-| diagnostic* | errors |
+| error* | errors |
 | varuint | number of warnings |
-| diagnostic* | warnings |
+| warning* | warnings |
 | `4` | content pool offset |
 | varuint | content pool size |
 
@@ -175,9 +184,12 @@ The final argument to `pm_serialize_parse` is an optional string that controls t
 | ...     | the encoding bytes         |
 | `1`     | frozen string literal      |
 | `1`     | suppress warnings          |
+| `1`     | syntax version, see [pm_options_version_t](https://github.com/ruby/prism/blob/main/include/prism/options.h) for valid values |
 | `4`     | the number of scopes       |
 | ...     | the scopes                 |
 
+Scopes are ordered from the outermost scope to the innermost one.
+
 Each scope is layed out as follows:
 
 | # bytes | field                      |

data/ext/prism/api_node.c

@@ -171,7 +171,7 @@ pm_location_new(pm_parser_t *parser, const uint8_t *start, const uint8_t *end, V
 
 VALUE
 pm_token_new(pm_parser_t *parser, pm_token_t *token, rb_encoding *encoding, VALUE source) {
-    ID type = rb_intern(pm_token_type_to_str(token->type));
+    ID type = rb_intern(pm_token_type_name(token->type));
     VALUE location = pm_location_new(parser, token->start, token->end, source);
 
     VALUE argv[] = {
@@ -1362,23 +1362,27 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
 #line 144 "api_node.c.erb"
                 case PM_BLOCK_LOCAL_VARIABLE_NODE: {
                     pm_block_local_variable_node_t *cast = (pm_block_local_variable_node_t *) node;
-                    VALUE argv[2];
+                    VALUE argv[3];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
 #line 167 "api_node.c.erb"
                     assert(cast->name != 0);
-                    argv[0] = rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = rb_id2sym(constants[cast->name - 1]);
 
                     // location
-                    argv[1] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[2] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(2, argv, rb_cPrismBlockLocalVariableNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(3, argv, rb_cPrismBlockLocalVariableNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
                 case PM_BLOCK_NODE: {
                     pm_block_node_t *cast = (pm_block_node_t *) node;
-                    VALUE argv[7];
+                    VALUE argv[6];
 
                     // locals
 #line 173 "api_node.c.erb"
@@ -1388,52 +1392,52 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
                         rb_ary_push(argv[0], rb_id2sym(constants[cast->locals.ids[index] - 1]));
                     }
 
-                    // locals_body_index
-#line 189 "api_node.c.erb"
-                    argv[1] = ULONG2NUM(cast->locals_body_index);
-
                     // parameters
 #line 155 "api_node.c.erb"
-                    argv[2] = rb_ary_pop(value_stack);
+                    argv[1] = rb_ary_pop(value_stack);
 
                     // body
 #line 155 "api_node.c.erb"
-                    argv[3] = rb_ary_pop(value_stack);
+                    argv[2] = rb_ary_pop(value_stack);
 
                     // opening_loc
 #line 180 "api_node.c.erb"
-                    argv[4] = pm_location_new(parser, cast->opening_loc.start, cast->opening_loc.end, source);
+                    argv[3] = pm_location_new(parser, cast->opening_loc.start, cast->opening_loc.end, source);
 
                     // closing_loc
 #line 180 "api_node.c.erb"
-                    argv[5] = pm_location_new(parser, cast->closing_loc.start, cast->closing_loc.end, source);
+                    argv[4] = pm_location_new(parser, cast->closing_loc.start, cast->closing_loc.end, source);
 
                     // location
-                    argv[6] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[5] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(7, argv, rb_cPrismBlockNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(6, argv, rb_cPrismBlockNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
                 case PM_BLOCK_PARAMETER_NODE: {
                     pm_block_parameter_node_t *cast = (pm_block_parameter_node_t *) node;
-                    VALUE argv[4];
+                    VALUE argv[5];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
-                    argv[0] = cast->name == 0 ? Qnil : rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = cast->name == 0 ? Qnil : rb_id2sym(constants[cast->name - 1]);
 
                     // name_loc
 #line 183 "api_node.c.erb"
-                    argv[1] = cast->name_loc.start == NULL ? Qnil : pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
+                    argv[2] = cast->name_loc.start == NULL ? Qnil : pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
 
                     // operator_loc
 #line 180 "api_node.c.erb"
-                    argv[2] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
+                    argv[3] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
 
                     // location
-                    argv[3] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[4] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(4, argv, rb_cPrismBlockParameterNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(5, argv, rb_cPrismBlockParameterNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
@@ -2288,7 +2292,7 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
 #line 144 "api_node.c.erb"
                 case PM_DEF_NODE: {
                     pm_def_node_t *cast = (pm_def_node_t *) node;
-                    VALUE argv[14];
+                    VALUE argv[13];
 
                     // name
 #line 167 "api_node.c.erb"
@@ -2319,38 +2323,34 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
                         rb_ary_push(argv[5], rb_id2sym(constants[cast->locals.ids[index] - 1]));
                     }
 
-                    // locals_body_index
-#line 189 "api_node.c.erb"
-                    argv[6] = ULONG2NUM(cast->locals_body_index);
-
                     // def_keyword_loc
 #line 180 "api_node.c.erb"
-                    argv[7] = pm_location_new(parser, cast->def_keyword_loc.start, cast->def_keyword_loc.end, source);
+                    argv[6] = pm_location_new(parser, cast->def_keyword_loc.start, cast->def_keyword_loc.end, source);
 
                     // operator_loc
 #line 183 "api_node.c.erb"
-                    argv[8] = cast->operator_loc.start == NULL ? Qnil : pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
+                    argv[7] = cast->operator_loc.start == NULL ? Qnil : pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
 
                     // lparen_loc
 #line 183 "api_node.c.erb"
-                    argv[9] = cast->lparen_loc.start == NULL ? Qnil : pm_location_new(parser, cast->lparen_loc.start, cast->lparen_loc.end, source);
+                    argv[8] = cast->lparen_loc.start == NULL ? Qnil : pm_location_new(parser, cast->lparen_loc.start, cast->lparen_loc.end, source);
 
                     // rparen_loc
 #line 183 "api_node.c.erb"
-                    argv[10] = cast->rparen_loc.start == NULL ? Qnil : pm_location_new(parser, cast->rparen_loc.start, cast->rparen_loc.end, source);
+                    argv[9] = cast->rparen_loc.start == NULL ? Qnil : pm_location_new(parser, cast->rparen_loc.start, cast->rparen_loc.end, source);
 
                     // equal_loc
 #line 183 "api_node.c.erb"
-                    argv[11] = cast->equal_loc.start == NULL ? Qnil : pm_location_new(parser, cast->equal_loc.start, cast->equal_loc.end, source);
+                    argv[10] = cast->equal_loc.start == NULL ? Qnil : pm_location_new(parser, cast->equal_loc.start, cast->equal_loc.end, source);
 
                     // end_keyword_loc
 #line 183 "api_node.c.erb"
-                    argv[12] = cast->end_keyword_loc.start == NULL ? Qnil : pm_location_new(parser, cast->end_keyword_loc.start, cast->end_keyword_loc.end, source);
+                    argv[11] = cast->end_keyword_loc.start == NULL ? Qnil : pm_location_new(parser, cast->end_keyword_loc.start, cast->end_keyword_loc.end, source);
 
                     // location
-                    argv[13] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[12] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(14, argv, rb_cPrismDefNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(13, argv, rb_cPrismDefNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
@@ -3442,29 +3442,33 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
 #line 144 "api_node.c.erb"
                 case PM_KEYWORD_REST_PARAMETER_NODE: {
                     pm_keyword_rest_parameter_node_t *cast = (pm_keyword_rest_parameter_node_t *) node;
-                    VALUE argv[4];
+                    VALUE argv[5];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
-                    argv[0] = cast->name == 0 ? Qnil : rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = cast->name == 0 ? Qnil : rb_id2sym(constants[cast->name - 1]);
 
                     // name_loc
 #line 183 "api_node.c.erb"
-                    argv[1] = cast->name_loc.start == NULL ? Qnil : pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
+                    argv[2] = cast->name_loc.start == NULL ? Qnil : pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
 
                     // operator_loc
 #line 180 "api_node.c.erb"
-                    argv[2] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
+                    argv[3] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
 
                     // location
-                    argv[3] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[4] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(4, argv, rb_cPrismKeywordRestParameterNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(5, argv, rb_cPrismKeywordRestParameterNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
                 case PM_LAMBDA_NODE: {
                     pm_lambda_node_t *cast = (pm_lambda_node_t *) node;
-                    VALUE argv[8];
+                    VALUE argv[7];
 
                     // locals
 #line 173 "api_node.c.erb"
@@ -3474,34 +3478,30 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
                         rb_ary_push(argv[0], rb_id2sym(constants[cast->locals.ids[index] - 1]));
                     }
 
-                    // locals_body_index
-#line 189 "api_node.c.erb"
-                    argv[1] = ULONG2NUM(cast->locals_body_index);
-
                     // operator_loc
 #line 180 "api_node.c.erb"
-                    argv[2] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
+                    argv[1] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
 
                     // opening_loc
 #line 180 "api_node.c.erb"
-                    argv[3] = pm_location_new(parser, cast->opening_loc.start, cast->opening_loc.end, source);
+                    argv[2] = pm_location_new(parser, cast->opening_loc.start, cast->opening_loc.end, source);
 
                     // closing_loc
 #line 180 "api_node.c.erb"
-                    argv[4] = pm_location_new(parser, cast->closing_loc.start, cast->closing_loc.end, source);
+                    argv[3] = pm_location_new(parser, cast->closing_loc.start, cast->closing_loc.end, source);
 
                     // parameters
 #line 155 "api_node.c.erb"
-                    argv[5] = rb_ary_pop(value_stack);
+                    argv[4] = rb_ary_pop(value_stack);
 
                     // body
 #line 155 "api_node.c.erb"
-                    argv[6] = rb_ary_pop(value_stack);
+                    argv[5] = rb_ary_pop(value_stack);
 
                     // location
-                    argv[7] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[6] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(8, argv, rb_cPrismLambdaNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(7, argv, rb_cPrismLambdaNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
@@ -3989,53 +3989,61 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
 #line 144 "api_node.c.erb"
                 case PM_OPTIONAL_KEYWORD_PARAMETER_NODE: {
                     pm_optional_keyword_parameter_node_t *cast = (pm_optional_keyword_parameter_node_t *) node;
-                    VALUE argv[4];
+                    VALUE argv[5];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
 #line 167 "api_node.c.erb"
                     assert(cast->name != 0);
-                    argv[0] = rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = rb_id2sym(constants[cast->name - 1]);
 
                     // name_loc
 #line 180 "api_node.c.erb"
-                    argv[1] = pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
+                    argv[2] = pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
 
                     // value
 #line 155 "api_node.c.erb"
-                    argv[2] = rb_ary_pop(value_stack);
+                    argv[3] = rb_ary_pop(value_stack);
 
                     // location
-                    argv[3] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[4] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(4, argv, rb_cPrismOptionalKeywordParameterNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(5, argv, rb_cPrismOptionalKeywordParameterNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
                 case PM_OPTIONAL_PARAMETER_NODE: {
                     pm_optional_parameter_node_t *cast = (pm_optional_parameter_node_t *) node;
-                    VALUE argv[5];
+                    VALUE argv[6];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
 #line 167 "api_node.c.erb"
                     assert(cast->name != 0);
-                    argv[0] = rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = rb_id2sym(constants[cast->name - 1]);
 
                     // name_loc
 #line 180 "api_node.c.erb"
-                    argv[1] = pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
+                    argv[2] = pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
 
                     // operator_loc
 #line 180 "api_node.c.erb"
-                    argv[2] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
+                    argv[3] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
 
                     // value
 #line 155 "api_node.c.erb"
-                    argv[3] = rb_ary_pop(value_stack);
+                    argv[4] = rb_ary_pop(value_stack);
 
                     // location
-                    argv[4] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[5] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(5, argv, rb_cPrismOptionalParameterNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(6, argv, rb_cPrismOptionalParameterNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
@@ -4343,37 +4351,45 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
 #line 144 "api_node.c.erb"
                 case PM_REQUIRED_KEYWORD_PARAMETER_NODE: {
                     pm_required_keyword_parameter_node_t *cast = (pm_required_keyword_parameter_node_t *) node;
-                    VALUE argv[3];
+                    VALUE argv[4];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
 #line 167 "api_node.c.erb"
                     assert(cast->name != 0);
-                    argv[0] = rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = rb_id2sym(constants[cast->name - 1]);
 
                     // name_loc
 #line 180 "api_node.c.erb"
-                    argv[1] = pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
+                    argv[2] = pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
 
                     // location
-                    argv[2] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[3] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(3, argv, rb_cPrismRequiredKeywordParameterNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(4, argv, rb_cPrismRequiredKeywordParameterNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
                 case PM_REQUIRED_PARAMETER_NODE: {
                     pm_required_parameter_node_t *cast = (pm_required_parameter_node_t *) node;
-                    VALUE argv[2];
+                    VALUE argv[3];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
 #line 167 "api_node.c.erb"
                     assert(cast->name != 0);
-                    argv[0] = rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = rb_id2sym(constants[cast->name - 1]);
 
                     // location
-                    argv[1] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[2] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(2, argv, rb_cPrismRequiredParameterNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(3, argv, rb_cPrismRequiredParameterNode));
                     break;
                 }
 #line 144 "api_node.c.erb"
@@ -4440,23 +4456,27 @@ pm_ast_new(pm_parser_t *parser, pm_node_t *node, rb_encoding *encoding) {
 #line 144 "api_node.c.erb"
                 case PM_REST_PARAMETER_NODE: {
                     pm_rest_parameter_node_t *cast = (pm_rest_parameter_node_t *) node;
-                    VALUE argv[4];
+                    VALUE argv[5];
+
+                    // flags
+#line 192 "api_node.c.erb"
+                    argv[0] = ULONG2NUM(node->flags & ~PM_NODE_FLAG_COMMON_MASK);
 
                     // name
-                    argv[0] = cast->name == 0 ? Qnil : rb_id2sym(constants[cast->name - 1]);
+                    argv[1] = cast->name == 0 ? Qnil : rb_id2sym(constants[cast->name - 1]);
 
                     // name_loc
 #line 183 "api_node.c.erb"
-                    argv[1] = cast->name_loc.start == NULL ? Qnil : pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
+                    argv[2] = cast->name_loc.start == NULL ? Qnil : pm_location_new(parser, cast->name_loc.start, cast->name_loc.end, source);
 
                     // operator_loc
 #line 180 "api_node.c.erb"
-                    argv[2] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
+                    argv[3] = pm_location_new(parser, cast->operator_loc.start, cast->operator_loc.end, source);
 
                     // location
-                    argv[3] = pm_location_new(parser, node->location.start, node->location.end, source);
+                    argv[4] = pm_location_new(parser, node->location.start, node->location.end, source);
 
-                    rb_ary_push(value_stack, rb_class_new_instance(4, argv, rb_cPrismRestParameterNode));
+                    rb_ary_push(value_stack, rb_class_new_instance(5, argv, rb_cPrismRestParameterNode));
                     break;
                 }
 #line 144 "api_node.c.erb"