prism 0.19.0 → 0.24.0

data/config.yml

@@ -347,6 +347,8 @@ flags:
         comment: "a call that could have been a local variable"
       - name: ATTRIBUTE_WRITE
         comment: "a call that is an attribute write, so the value being written should be returned"
+      - name: IGNORE_VISIBILITY
+        comment: "a call that ignores method visibility"
     comment: Flags for call nodes.
   - name: EncodingFlags
     values:
@@ -368,14 +370,19 @@ flags:
     comment: Flags for integer nodes that correspond to the base of the integer.
   - name: KeywordHashNodeFlags
     values:
-      - name: STATIC_KEYS
-        comment: "a keyword hash which only has `AssocNode` elements all with static literal keys, which means the elements can be treated as keyword arguments"
+      - name: SYMBOL_KEYS
+        comment: "a keyword hash which only has `AssocNode` elements all with symbol keys, which means the elements can be treated as keyword arguments"
     comment: Flags for keyword hash nodes.
   - name: LoopFlags
     values:
       - name: BEGIN_MODIFIER
         comment: "a loop after a begin statement, so the body is executed first before the condition"
     comment: Flags for while and until loop nodes.
+  - name: ParameterFlags
+    values:
+      - name: REPEATED_PARAMETER
+        comment: "a parameter name that has been repeated in the method signature"
+    comment: Flags for parameter nodes.
   - name: RangeFlags
     values:
       - name: EXCLUDE_END
@@ -468,10 +475,31 @@ nodes:
     fields:
       - name: left
         type: node
+        comment: |
+          Represents the left side of the expression. It can be any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              left and right
+              ^^^^
+
+              1 && 2
+              ^
       - name: right
         type: node
+        comment: |
+          Represents the right side of the expression. It can be any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              left && right
+                      ^^^^^
+
+              1 and 2
+                    ^
       - name: operator_loc
         type: location
+        comment: |
+          The location of the `and` keyword or the `&&` operator.
+
+              left and right
+                   ^^^
     comment: |
       Represents the use of the `&&` operator or the `and` keyword.
 
@@ -501,8 +529,7 @@ nodes:
       - name: closing_loc
         type: location?
     comment: |
-      Represents an array literal. This can be a regular array using brackets or
-      a special array using % like %w or %i.
+      Represents an array literal. This can be a regular array using brackets or a special array using % like %w or %i.
 
           [1, 2, 3]
           ^^^^^^^^^
@@ -541,10 +568,34 @@ nodes:
     fields:
       - name: key
         type: node
+        comment: |
+          The key of the association. This can be any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              { a: b }
+                ^
+
+              { foo => bar }
+                ^^^
+
+              { def a; end => 1 }
+                ^^^^^^^^^^
       - name: value
-        type: node?
+        type: node
+        comment: |
+          The value of the association, if present. This can be any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              { foo => bar }
+                       ^^^
+
+              { x: 1 }
+                   ^
       - name: operator_loc
         type: location?
+        comment: |
+          The location of the `=>` operator, if present.
+
+              { foo => bar }
+                    ^^
     comment: |
       Represents a hash key/value pair.
 
@@ -554,8 +605,18 @@ nodes:
     fields:
       - name: value
         type: node?
+        comment: |
+          The value to be splatted, if present. Will be missing when keyword rest argument forwarding is used.
+
+              { **foo }
+                  ^^^
       - name: operator_loc
         type: location
+        comment: |
+          The location of the `**` operator.
+
+              { **x }
+                ^^
     comment: |
       Represents a splat in a hash literal.
 
@@ -565,6 +626,12 @@ nodes:
     fields:
       - name: name
         type: constant
+        comment: |
+          The name of the back-reference variable, including the leading `$`.
+
+              $& # name `:$&`
+
+              $+ # name `:$+`
     comment: |
       Represents reading a reference to a field in the previous match.
 
@@ -609,6 +676,9 @@ nodes:
           ^^^^^^^^^^
   - name: BlockLocalVariableNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant
     comment: |
@@ -620,8 +690,6 @@ nodes:
     fields:
       - name: locals
         type: constant[]
-      - name: locals_body_index
-        type: uint32
       - name: parameters
         type: node?
       - name: body
@@ -633,10 +701,13 @@ nodes:
     comment: |
       Represents a block of ruby code.
 
-      [1, 2, 3].each { |i| puts x }
-                     ^^^^^^^^^^^^^^
+          [1, 2, 3].each { |i| puts x }
+                         ^^^^^^^^^^^^^^
   - name: BlockParameterNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant?
       - name: name_loc
@@ -712,6 +783,17 @@ nodes:
         kind: CallNodeFlags
       - name: receiver
         type: node?
+        comment: |
+          The object that the method is being called on. This can be either `nil` or any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              foo.bar
+              ^^^
+
+              +foo
+               ^^^
+
+              foo + bar
+              ^^^
       - name: call_operator_loc
         type: location?
       - name: name
@@ -950,6 +1032,12 @@ nodes:
     fields:
       - name: name
         type: constant
+        comment: |
+          The name of the class variable, which is a `@@` followed by an [identifier](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#identifiers).
+
+              @@abc   # name `:@@abc`
+
+              @@_test # name `:@@_test`
     comment: |
       Represents referencing a class variable.
 
@@ -1120,6 +1208,12 @@ nodes:
     fields:
       - name: name
         type: constant
+        comment: |
+          The name of the [constant](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#constants).
+
+              X              # name `:X`
+
+              SOME_CONSTANT  # name `:SOME_CONSTANT`
     comment: |
       Represents referencing a constant.
 
@@ -1164,8 +1258,6 @@ nodes:
         type: node?
       - name: locals
         type: constant[]
-      - name: locals_body_index
-        type: uint32
       - name: def_keyword_loc
         type: location
       - name: operator_loc
@@ -1407,6 +1499,12 @@ nodes:
     fields:
       - name: name
         type: constant
+        comment: |
+          The name of the global variable, which is a `$` followed by an [identifier](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#identifier). Alternatively, it can be one of the special global variables designated by a symbol.
+
+              $foo   # name `:$foo`
+
+              $_Test # name `:$_Test`
     comment: |
       Represents referencing a global variable.
 
@@ -1440,10 +1538,28 @@ nodes:
     fields:
       - name: opening_loc
         type: location
+        comment: |
+          The location of the opening brace.
+
+              { a => b }
+              ^
       - name: elements
         type: node[]
+        comment: |
+          The elements of the hash. These can be either `AssocNode`s or `AssocSplatNode`s.
+
+              { a: b }
+                ^^^^
+
+              { **foo }
+                ^^^^^
       - name: closing_loc
         type: location
+        comment: |
+          The location of the closing brace.
+
+              { a => b }
+                       ^
     comment: |
       Represents a hash literal.
 
@@ -1507,14 +1623,16 @@ nodes:
       - name: value
         type: node
     comment: |
-      Represents a node that is implicitly being added to the tree but doesn't
-      correspond directly to a node in the source.
+      Represents a node that is implicitly being added to the tree but doesn't correspond directly to a node in the source.
 
           { foo: }
             ^^^^
 
           { Foo: }
             ^^^^
+
+          foo in { bar: }
+                   ^^^^
   - name: ImplicitRestNode
     comment: |
       Represents using a trailing comma to indicate an implicit rest parameter.
@@ -1709,6 +1827,12 @@ nodes:
     fields:
       - name: name
         type: constant
+        comment: |
+          The name of the instance variable, which is a `@` followed by an [identifier](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#identifiers).
+
+              @x     # name `:@x`
+
+              @_test # name `:@_test`
     comment: |
       Represents referencing an instance variable.
 
@@ -1761,9 +1885,7 @@ nodes:
         type: location
     newline: parts
     comment: |
-      Represents a regular expression literal that contains interpolation that
-      is being used in the predicate of a conditional to implicitly match
-      against the last line read by an IO object.
+      Represents a regular expression literal that contains interpolation that is being used in the predicate of a conditional to implicitly match against the last line read by an IO object.
 
           if /foo #{bar} baz/ then end
              ^^^^^^^^^^^^^^^^
@@ -1840,6 +1962,9 @@ nodes:
               ^^^^
   - name: KeywordRestParameterNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant?
       - name: name_loc
@@ -1856,8 +1981,6 @@ nodes:
     fields:
       - name: locals
         type: constant[]
-      - name: locals_body_index
-        type: uint32
       - name: operator_loc
         type: location
       - name: opening_loc
@@ -1930,12 +2053,33 @@ nodes:
     fields:
       - name: name
         type: constant
+        comment: |
+          The name of the local variable, which is an [identifier](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#identifiers).
+
+              x      # name `:x`
+
+              _Test  # name `:_Test`
+
+          Note that this can also be an underscore followed by a number for the default block parameters.
+
+              _1     # name `:_1`
+
+          Finally, for the default `it` block parameter, the name is `0it`. This is to distinguish it from an `it` local variable that is explicitly declared.
+
+              it     # name `:0it`
+
       - name: depth
         type: uint32
+        comment: |
+          The number of visible scopes that should be searched to find the origin of this local variable.
+
+              foo = 1; foo # depth 0
+
+              bar = 2; tap { bar } # depth 1
+
+          The specific rules for calculating the depth may differ from individual Ruby implementations, as they are not specified by the language. For more information, see [the Prism documentation](https://github.com/ruby/prism/blob/main/docs/local_variable_depth.md).
     comment: |
-      Represents reading a local variable. Note that this requires that a local
-      variable of the same name has already been written to in the same scope,
-      otherwise it is parsed as a method call.
+      Represents reading a local variable. Note that this requires that a local variable of the same name has already been written to in the same scope, otherwise it is parsed as a method call.
 
           foo
           ^^^
@@ -1981,9 +2125,7 @@ nodes:
       - name: unescaped
         type: string
     comment: |
-      Represents a regular expression literal used in the predicate of a
-      conditional to implicitly match against the last line read by an IO
-      object.
+      Represents a regular expression literal used in the predicate of a conditional to implicitly match against the last line read by an IO object.
 
           if /foo/i then end
              ^^^^^^
@@ -2021,15 +2163,13 @@ nodes:
       - name: targets
         type: node[]
     comment: |
-      Represents writing local variables using a regular expression match with
-      named capture groups.
+      Represents writing local variables using a regular expression match with named capture groups.
 
           /(?<foo>bar)/ =~ baz
           ^^^^^^^^^^^^^^^^^^^^
   - name: MissingNode
     comment: |
-      Represents a node that is missing from the source and results in a syntax
-      error.
+      Represents a node that is missing from the source and results in a syntax error.
   - name: ModuleNode
     fields:
       - name: locals
@@ -2122,8 +2262,7 @@ nodes:
       - name: maximum
         type: uint8
     comment: |
-      Represents an implicit set of parameters through the use of numbered
-      parameters within a block or lambda.
+      Represents an implicit set of parameters through the use of numbered parameters within a block or lambda.
 
           -> { _1 + _2 }
           ^^^^^^^^^^^^^^
@@ -2131,6 +2270,14 @@ nodes:
     fields:
       - name: number
         type: uint32
+        comment: |
+          The (1-indexed, from the left) number of the capture group. Numbered references that would overflow a `uint32`  result in a `number` of exactly `2**32 - 1`.
+
+              $1          # number `1`
+
+              $5432       # number `5432`
+
+              $4294967296 # number `4294967295`
     comment: |
       Represents reading a numbered reference to a capture in the previous match.
 
@@ -2138,6 +2285,9 @@ nodes:
           ^^
   - name: OptionalKeywordParameterNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant
       - name: name_loc
@@ -2152,6 +2302,9 @@ nodes:
           end
   - name: OptionalParameterNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant
       - name: name_loc
@@ -2170,10 +2323,31 @@ nodes:
     fields:
       - name: left
         type: node
+        comment: |
+          Represents the left side of the expression. It can be any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              left or right
+              ^^^^
+
+              1 || 2
+              ^
       - name: right
         type: node
+        comment: |
+          Represents the right side of the expression. It can be any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              left || right
+                      ^^^^^
+
+              1 or 2
+                   ^
       - name: operator_loc
         type: location
+        comment: |
+          The location of the `or` keyword or the `||` operator.
+
+              left or right
+                   ^^
     comment: |
       Represents the use of the `||` operator or the `or` keyword.
 
@@ -2227,8 +2401,7 @@ nodes:
       - name: rparen_loc
         type: location
     comment: |
-      Represents the use of the `^` operator for pinning an expression in a
-      pattern matching expression.
+      Represents the use of the `^` operator for pinning an expression in a pattern matching expression.
 
           foo in ^(bar)
                  ^^^^^^
@@ -2239,8 +2412,7 @@ nodes:
       - name: operator_loc
         type: location
     comment: |
-      Represents the use of the `^` operator for pinning a variable in a pattern
-      matching expression.
+      Represents the use of the `^` operator for pinning a variable in a pattern matching expression.
 
           foo in ^bar
                  ^^^^
@@ -2291,10 +2463,29 @@ nodes:
         kind: RangeFlags
       - name: left
         type: node?
+        comment: |
+          The left-hand side of the range, if present. It can be either `nil` or any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              1...
+              ^
+
+              hello...goodbye
+              ^^^^^
       - name: right
         type: node?
+        comment: |
+          The right-hand side of the range, if present. It can be either `nil` or any [non-void expression](https://github.com/ruby/prism/blob/main/docs/parsing_rules.md#non-void-expression).
+
+              ..5
+                ^
+
+              1...foo
+                  ^^^
+          If neither right-hand or left-hand side was included, this will be a MissingNode.
       - name: operator_loc
         type: location
+        comment: |
+          The location of the `..` or `...` operator.
     comment: |
       Represents the use of the `..` or `...` operators.
 
@@ -2338,6 +2529,9 @@ nodes:
           ^^^^^^
   - name: RequiredKeywordParameterNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant
       - name: name_loc
@@ -2350,6 +2544,9 @@ nodes:
           end
   - name: RequiredParameterNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant
     comment: |
@@ -2397,10 +2594,12 @@ nodes:
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
           end
 
-      `Foo, *splat, Bar` are in the `exceptions` field.
-      `ex` is in the `exception` field.
+      `Foo, *splat, Bar` are in the `exceptions` field. `ex` is in the `exception` field.
   - name: RestParameterNode
     fields:
+      - name: flags
+        type: flags
+        kind: ParameterFlags
       - name: name
         type: constant?
       - name: name_loc
@@ -2511,8 +2710,7 @@ nodes:
       - name: unescaped
         type: string
     comment: |
-      Represents a string literal, a string contained within a `%w` list, or
-      plain string content within an interpolated string.
+      Represents a string literal, a string contained within a `%w` list, or plain string content within an interpolated string.
 
           "foo"
           ^^^^^

data/docs/build_system.md

@@ -53,7 +53,7 @@ Because of course those files are not part of the git repository.
 
 ### Building prism as part of CRuby
 
-[This script](https://github.com/ruby/ruby/blob/32e828bb4a6c65a392b2300f3bdf93008c7b6f25/tool/sync_default_gems.rb#L399-L426) imports prism sources in CRuby.
+[This script](https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337c430cb93caa151/tool/sync_default_gems.rb#L399-L422) imports prism sources in CRuby.
 
 The script generates the templates when importing.
 
@@ -71,4 +71,21 @@ and links to `libprism.a` (to avoid exporting symbols, so no conflict when insta
 
 ### Building prism as part of JRuby
 
-TODO, probably similar to TruffleRuby.
+TODO, similar to TruffleRuby.
+
+### Building prism from source as a C library
+
+All of the source files match `src/**/*.c` and all of the headers match `include/**/*.h`.
+
+If you want to build prism as a shared library and link against it, you should compile with:
+
+* `-fPIC -shared` - Compile as a shared library
+* `-DPRISM_EXPORT_SYMBOLS` - Export the symbols (by default nothing is exported)
+
+#### Flags
+
+`make` respects the `MAKEFLAGS` environment variable. As such, to speed up the build you can run:
+
+```
+MAKEFLAGS="-j10" bundle exec rake compile
+```

data/docs/cruby_compilation.md

@@ -0,0 +1,27 @@
+# Compiling Prism's AST
+
+One important class of consumers of Prism's AST is compilers. Currently [CRuby](https://github.com/ruby/ruby), [JRuby](https://github.com/jruby/jruby), [TruffleRuby](https://github.com/oracle/truffleruby), and [Natalie](https://github.com/natalie-lang/natalie) have all built compilation code on top of Prism's AST.
+
+This document will describe, at a high level, how CRuby's compilation of Prism's AST works.
+
+As described in the [build system documentation](build_system.md), there is a "push" Webhook set up within the Prism repo triggered on each new commit to send information about the commit to [git.ruby-lang.org](https://github.com/ruby/git.ruby-lang.org). This in turn runs [a script](https://github.com/ruby/ruby/blob/master/tool/sync_default_gems.rb) to sync over new changes in Prism to their corresponding files in Ruby. Any failures in this sync script will show alerts in the #alerts-sync channel in the RubyLang Slack. The result of this step is that files are synced from Prism into ruby/ruby for its use. It is also worth noting that [`common.mk`](https://github.com/ruby/ruby/blob/master/common.mk) contains a list of Prism files which it needs to correctly compile. If there are new Prism files added, this file should also be updated.
+
+ruby/ruby uses the Prism code to generate an AST from which it can generate instruction sequences. Compilation in ruby/ruby has three main steps:
+
+1. Compute an AST
+
+Syncing over the Prism code allows ruby/ruby to compute the AST using Prism. It currently does this within [`iseq.c`](https://github.com/ruby/ruby/blob/master/iseq.c) using the `pm_parser_init` fuction.
+
+2. Run a first pass of compilation
+
+Once the AST has been created, it is recursively descended in order to compute the appropriate instruction sequences. This is the crux of compilation, and we go into more detail about nuances in the following paragraphs.
+
+The code for this step is almost exclusively in [`prism_compile.c`](https://github.com/ruby/ruby/blob/master/prism_compile.c). The main function used for compilation is `pm_compile_node` which is essentially a huge switch statement over practically every node type which computes the appropriate instruction sequences for that node type. There are several convenience helpers, such as `PM_COMPILE`, `PM_COMPILE_POPPED`, `PM_COMPILE_NOT_POPPED` which all call into the `pm_compile_node` function.
+
+There are also several functions, like `parse_string`, `parse_integer` which consume Prism nodes and return CRuby values. These are all called for their relevant types within the big switch statement.
+
+The Prism compiler also uses a concept of "scope nodes" which are not standard Prism nodes in the AST, but instead nodes constructed within the compiler for the sole purpose of making compilation easier. Scope nodes are defined in [`prism_compile.h`](https://github.com/ruby/ruby/blob/master/prism_compile.h) and store information such as locals, local table size, local depth offset and the index lookup tables. Scope nodes can be generated for node types which have their own "scope".
+
+3. Run an optimization pass of compilation
+
+After the instruction sequences are initially computed, there is an existing (non-Prism based) optimization pass of the instruction sequences. There are several optimizations currently inlined into step 2, however, most of them happen in this step. Specifically, any peephole optimizations happen in this step. By the end of step 2, however, the instruction sequences take the same form regardless of if the initial AST was generated by Prism or not. Therefore, step 3 is agnostic to the parser, and should not require any Prism specific code.

data/docs/parser_translation.md

@@ -0,0 +1,34 @@
+# parser translation
+
+Prism ships with the ability to translate its syntax tree into the syntax tree used by the [whitequark/parser](https://github.com/whitequark/parser) gem. This allows you to use tools built on top of the `parser` gem with the `prism` parser.
+
+## Usage
+
+The `parser` gem provides multiple parsers to support different versions of the Ruby grammar. This includes all of the Ruby versions going back to 1.8, as well as third-party parsers like MacRuby and RubyMotion. The `prism` gem provides another parser that uses the `prism` parser to build the syntax tree.
+
+You can use the `prism` parser like you would any other. After requiring the parser, you should be able to call any of the regular `Parser::Base` APIs that you would normally use.
+
+```ruby
+require "prism"
+
+Prism::Translation::Parser.parse_file("path/to/file.rb")
+```
+
+### RuboCop
+
+To run RuboCop using the `prism` gem as the parser, you will need to require the `prism/translation/parser/rubocop` file. This file injects `prism` into the known options for both `rubocop` and `rubocop-ast`, such that you can specify it in your `.rubocop.yml` file. Unfortunately `rubocop` doesn't support any direct way to do this, so we have to get a bit hacky.
+
+First, set the `TargetRubyVersion` in your RuboCop configuration file to `80_82_73_83_77.33`. This is the version of Ruby that `prism` reports itself as. (The leading numbers are the ASCII values for `PRISM`.)
+
+```yaml
+AllCops:
+  TargetRubyVersion: 80_82_73_83_77.33
+```
+
+Now when you run `rubocop` you will need to require the `prism/translation/parser/rubocop` file before executing so that it can inject the `prism` parser into the known options.
+
+```
+bundle exec ruby -rprism/translation/parser/rubocop $(bundle exec which rubocop)
+```
+
+This should run RuboCop using the `prism` parser.

data/docs/parsing_rules.md

@@ -0,0 +1,19 @@
+# Rules
+
+This document contains information related to the rules of the parser for Ruby source code.
+
+As an example, in the documentation of many of the fields of nodes, it's mentioned that a field follows the lexing rules for `identifier` or `constant`. This document describes what those rules are.
+
+## Constants
+
+Constants in Ruby begin with an upper-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "upper-case letter" are encoding-dependent.
+
+## Non-void expression
+
+Most expressions in CRuby are non-void. This means the expression they represent resolves to a value. For example, `1 + 2` is a non-void expression, because it resolves to a method call. Even things like `class Foo; end` is a non-void expression, because it returns the last evaluated expression in the body of the class (or `nil`).
+
+Certain nodes, however, are void expressions, and cannot be combined to form larger expressions. For example, `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo`.
+
+## Identifiers
+
+Identifiers in Ruby begin with an underscore or lower-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "lower-case letter" are encoding-dependent.