jruby-prism-parser 0.24.0-java → 1.4.0-java

data/docs/build_system.md

@@ -16,9 +16,9 @@ The main solution for the second point seems a Makefile, otherwise many of the u
 ## General Design
 
 1. Templates are generated by `templates/template.rb`
-4. The `Makefile` compiles both `libprism.a` and `libprism.{so,dylib,dll}` from the `src/**/*.c` and `include/**/*.h` files
-5. The `Rakefile` `:compile` task ensures the above prerequisites are done, then calls `make`,
-  and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`), which uses `libprism.a`
+2. The `Makefile` compiles both `libprism.a` and `libprism.{so,dylib,dll}` from the `src/**/*.c` and `include/**/*.h` files
+3. The `Rakefile` `:compile` task ensures the above prerequisites are done, then calls `make`,
+   and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`)
 
 This way there is minimal duplication, and each layer builds on the previous one and has its own responsibilities.
 
@@ -35,14 +35,11 @@ loaded per process (i.e., at most one version of the prism *gem* loaded in a pro
 ### Building the prism gem by `gem install/bundle install`
 
 The gem contains the pre-generated templates.
-When installing the gem, `extconf.rb` is used and that:
-* runs `make build/libprism.a`
-* compiles the C extension with mkmf
 
-When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead of the last step,
-there is Ruby code using FFI which uses `libprism.{so,dylib,dll}`
-to implement the same methods as the C extension, but using serialization instead of many native calls/accesses
-(JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
+When installing the gem on CRuby, `extconf.rb` is used and that compiles the C extension with mkmf, including both the extension files and the sources of prism itself.
+
+When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead the `extconf.rb` runs `make build/libprism.{so,dylib,dll}`.
+There is Ruby code using FFI which uses `libprism.{so,dylib,dll}` to implement the same methods as the C extension, but using serialization instead of many native calls/accesses (JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
 
 ### Building the prism gem from git, e.g. `gem "prism", github: "ruby/prism"`
 
@@ -66,13 +63,32 @@ The script generates the templates when importing.
 
 Then when `mx build` builds TruffleRuby and the `prism` mx project inside, it runs `make`.
 
-Then the `prism bindings` mx project is built, which contains the [bindings](https://github.com/oracle/truffleruby/blob/master/src/main/c/prism_bindings/src/prism_bindings.c)
+Then the `prism bindings` mx project is built, which contains the [bindings](https://github.com/oracle/truffleruby/blob/vm-24.1.1/src/main/c/yarp_bindings/src/yarp_bindings.c)
 and links to `libprism.a` (to avoid exporting symbols, so no conflict when installing the prism gem).
 
 ### Building prism as part of JRuby
 
 TODO, similar to TruffleRuby.
 
+### Building prism for embedded system
+
+For instance, you can build a static library `libprism.a` targeting the Arm Cortex-M0+ embedded system by the commands below:
+
+* `templates/template.rb`
+* `CFLAGS="-mcpu=cortex-m0plus" make static CC=arm-none-eabi-gcc`
+
+The build process internally looks up `_POSIX_MAPPED_FILES` and `_WIN32` macros to determine whether the functions of the memory map are available on the target platform.
+
+### Building prism with custom memory allocator
+
+If you need to use memory allocation functions implemented outside of the standard library, follow these steps:
+
+* Add `-D PRISM_XALLOCATOR` to the build options
+* Additionally, include `-I [path/to/custom_allocator]` where your `prism_xallocator.h` is located
+* Link the implementation of `prism_xallocator.c` that contains functions declared in `prism_xallocator.h`
+
+For further clarity, refer to `include/prism/defines.h`.
+
 ### Building prism from source as a C library
 
 All of the source files match `src/**/*.c` and all of the headers match `include/**/*.h`.
@@ -89,3 +105,15 @@ If you want to build prism as a shared library and link against it, you should c
 ```
 MAKEFLAGS="-j10" bundle exec rake compile
 ```
+
+## Build options
+
+* `PRISM_BUILD_DEBUG` - Will cause all file reading to copy into its own allocation to allow easier tracking of reading off the end of the buffer. By default this is off.
+* `PRISM_BUILD_MINIMAL` - Define all of the `PRISM_EXCLUDE_*` flags at once.
+* `PRISM_ENCODING_EXCLUDE_FULL` - Will cause the library to exclude the full encoding API, and only include the minimal number of encodings to support parsing Ruby code without encoding comments. By default this is off.
+* `PRISM_EXPORT_SYMBOLS` - Will cause the shared library to export symbols. By default this is off.
+* `PRISM_EXCLUDE_JSON` - Will cause the library to exclude the JSON API. By default this is off.
+* `PRISM_EXCLUDE_PACK` - Will cause the library to exclude the pack API. By default this is off.
+* `PRISM_EXCLUDE_PRETTYPRINT` - Will cause the library to exclude the prettyprint API. By default this is off.
+* `PRISM_EXCLUDE_SERIALIZATION` - Will cause the library to exclude the serialization API. By default this is off.
+* `PRISM_XALLOCATOR` - Will cause the library to use the custom memory allocator. By default this is off.

data/docs/configuration.md

@@ -4,6 +4,7 @@ A lot of code in prism's repository is templated from a single configuration fil
 
 * `ext/prism/api_node.c` - for defining how to build Ruby objects for the nodes out of C structs
 * `include/prism/ast.h` - for defining the C structs that represent the nodes
+* `include/prism/diagnostic.h` - for defining the diagnostics
 * `javascript/src/deserialize.js` - for defining how to deserialize the nodes in JavaScript
 * `javascript/src/nodes.js` - for defining the nodes in JavaScript
 * `java/org/prism/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
@@ -13,10 +14,13 @@ A lot of code in prism's repository is templated from a single configuration fil
 * `lib/prism/dispatcher.rb` - for defining the dispatch visitors for the nodes in Ruby
 * `lib/prism/dot_visitor.rb` - for defining the dot visitor for the nodes in Ruby
 * `lib/prism/dsl.rb` - for defining the DSL for the nodes in Ruby
+* `lib/prism/inspect_visitor.rb` - for defining the `#inspect` methods on nodes in Ruby
 * `lib/prism/mutation_compiler.rb` - for defining the mutation compiler for the nodes in Ruby
 * `lib/prism/node.rb` - for defining the nodes in Ruby
+* `lib/prism/reflection.rb` - for defining the reflection API in Ruby
 * `lib/prism/serialize.rb` - for defining how to deserialize the nodes in Ruby
 * `lib/prism/visitor.rb` - for defining the visitor interface for the nodes in Ruby
+* `src/diagnostic.c` - for defining how to build diagnostics
 * `src/node.c` - for defining how to free the nodes in C and calculate the size in memory in C
 * `src/prettyprint.c` - for defining how to prettyprint the nodes in C
 * `src/serialize.c` - for defining how to serialize the nodes in C

data/docs/cruby_compilation.md

@@ -10,7 +10,7 @@ ruby/ruby uses the Prism code to generate an AST from which it can generate inst
 
 1. Compute an AST
 
-Syncing over the Prism code allows ruby/ruby to compute the AST using Prism. It currently does this within [`iseq.c`](https://github.com/ruby/ruby/blob/master/iseq.c) using the `pm_parser_init` fuction.
+Syncing over the Prism code allows ruby/ruby to compute the AST using Prism. It currently does this within [`iseq.c`](https://github.com/ruby/ruby/blob/master/iseq.c) using the `pm_parser_init` function.
 
 2. Run a first pass of compilation
 

data/docs/fuzzing.md

@@ -26,7 +26,7 @@ fuzz
 There are currently three fuzzing targets
 
 - `pm_serialize_parse` (parse)
-- `pm_regexp_named_capture_group_names` (regexp)
+- `pm_regexp_parse` (regexp)
 
 Respectively, fuzzing can be performed with
 

data/docs/parser_translation.md

@@ -16,19 +16,24 @@ Prism::Translation::Parser.parse_file("path/to/file.rb")
 
 ### RuboCop
 
-To run RuboCop using the `prism` gem as the parser, you will need to require the `prism/translation/parser/rubocop` file. This file injects `prism` into the known options for both `rubocop` and `rubocop-ast`, such that you can specify it in your `.rubocop.yml` file. Unfortunately `rubocop` doesn't support any direct way to do this, so we have to get a bit hacky.
+Prism as a parser engine is directly supported since RuboCop 1.62. The class used for parsing is `Prism::Translation::Parser`.
 
-First, set the `TargetRubyVersion` in your RuboCop configuration file to `80_82_73_83_77.33`. This is the version of Ruby that `prism` reports itself as. (The leading numbers are the ASCII values for `PRISM`.)
+First, specify `prism` in your Gemfile:
 
-```yaml
-AllCops:
-  TargetRubyVersion: 80_82_73_83_77.33
+```ruby
+gem "prism"
 ```
 
-Now when you run `rubocop` you will need to require the `prism/translation/parser/rubocop` file before executing so that it can inject the `prism` parser into the known options.
+To use Prism with RuboCop, specify `ParserEngine` and `TargetRubyVersion` in your RuboCop configuration file:
 
+```yaml
+AllCops:
+  ParserEngine: parser_prism
+  TargetRubyVersion: 3.3
 ```
-bundle exec ruby -rprism/translation/parser/rubocop $(bundle exec which rubocop)
-```
 
-This should run RuboCop using the `prism` parser.
+The default value for `ParserEngine` is `parser_whitequark`, which indicates the Parser gem. You need to explicitly switch it to `parser_prism` to indicate Prism. Additionally, the value for `TargetRubyVersion` must be specified as `3.3` or higher, as Prism supports parsing versions of Ruby 3.3 and higher.
+The parser class is determined by the combination of values for `ParserEngine` and `TargetRubyVersion`. For example, if `TargetRubyVersion: 3.3`, parsing is performed by `Prism::Translation::Parser33`, and for `TargetRubyVersion 3.4`, parsing is performed by `Prism::Translation::Parser34`.
+
+For further information, please refer to the RuboCop documentation:
+https://docs.rubocop.org/rubocop/configuration.html#setting-the-parser-engine

data/docs/parsing_rules.md

@@ -12,7 +12,10 @@ Constants in Ruby begin with an upper-case letter. This is followed by any numbe
 
 Most expressions in CRuby are non-void. This means the expression they represent resolves to a value. For example, `1 + 2` is a non-void expression, because it resolves to a method call. Even things like `class Foo; end` is a non-void expression, because it returns the last evaluated expression in the body of the class (or `nil`).
 
-Certain nodes, however, are void expressions, and cannot be combined to form larger expressions. For example, `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo`.
+Certain nodes, however, are void expressions, and cannot be combined to form larger expressions.
+* `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo` can only be at a statement position.
+* The "jumps": `return`, `break`, `next`, `redo`, `retry` are void expressions.
+* `value => pattern` is also considered a void expression.
 
 ## Identifiers
 

data/docs/releasing.md

@@ -40,22 +40,14 @@ ruby -pi -e 'gsub(/^ruby-prism-sys = \{ version = ".+?"/, %Q{ruby-prism-sys = \{
 * Update the `Gemfile.lock` file:
 
 ```sh
-chruby ruby-3.4.0-dev
+chruby ruby-3.5.0-dev
 bundle install
 ```
 
 * Update the version-specific lockfiles:
 
 ```sh
-chruby ruby-2.7.8 && BUNDLE_GEMFILE=gemfiles/2.7/Gemfile bundle install
-chruby ruby-3.0.6 && BUNDLE_GEMFILE=gemfiles/3.0/Gemfile bundle install
-chruby ruby-3.1.4 && BUNDLE_GEMFILE=gemfiles/3.1/Gemfile bundle install
-chruby ruby-3.2.3 && BUNDLE_GEMFILE=gemfiles/3.2/Gemfile bundle install
-chruby ruby-3.3.0 && BUNDLE_GEMFILE=gemfiles/3.3/Gemfile bundle install
-chruby ruby-3.4.0-dev && BUNDLE_GEMFILE=gemfiles/3.4/Gemfile bundle install
-chruby jruby-9.4.5.0 && BUNDLE_GEMFILE=gemfiles/jruby/Gemfile bundle install
-chruby truffleruby-23.1.2 && BUNDLE_GEMFILE=gemfiles/truffleruby/Gemfile bundle install
-chruby ruby-3.4.0-dev
+bin/prism bundle install
 ```
 
 * Update the cargo lockfiles:
@@ -70,6 +62,12 @@ bundle exec rake cargo:build
 git commit -am "Bump to v$PRISM_VERSION"
 ```
 
+* Push up the changes:
+
+```sh
+git push
+```
+
 ## Publishing
 
 * Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file.

data/docs/relocation.md

@@ -0,0 +1,34 @@
+# Relocation
+
+Prism parses deterministically for the same input. This provides a nice property that is exposed through the `#node_id` API on nodes. Effectively this means that for the same input, these values will remain consistent every time the source is parsed. This means we can reparse the source same with a `#node_id` value and find the exact same node again.
+
+The `Relocation` module provides an API around this property. It allows you to "save" nodes and locations using a minimal amount of memory (just the node_id and a field identifier) and then reify them later. This minimizes the amount of memory you need to allocate to store this information because it does not keep around a pointer to the source string.
+
+## Getting started
+
+To get started with the `Relocation` module, you would first instantiate a `Repository` object. You do this through a DSL that chains method calls for configuration. For example, if for every entry in the repository you want to store the start and end lines, the start and end code unit columns for in UTF-16, and the leading comments, you would:
+
+```ruby
+repository = Prism::Relocation.filepath("path/to/file").lines.code_unit_columns(Encoding::UTF_16).leading_comments
+```
+
+Now that you have the repository, you can pass it into any of the `save*` APIs on nodes or locations to create entries in the repository that will be lazily reified.
+
+```ruby
+# assume that node is a Prism::ClassNode object
+entry = node.constant_path.save(repository)
+```
+
+Now that you have the entry object, you do not need to keep around a reference to the repository, it will be cleaned up on its own when the last entry is reified. Now, whenever you need to, you may call the associated field methods on the entry object, as in:
+
+```ruby
+entry.start_line
+entry.end_line
+
+entry.start_code_units_column
+entry.end_code_units_column
+
+entry.leading_comments
+```
+
+Note that if you had configured other fields to be saved, you would be able to access them as well. The first time one of these fields is accessed, the repository will reify every entry it knows about and then clean itself up. In this way, you can effectively treat them as if you had kept around lightweight versions of `Prism::Node` or `Prism::Location` objects.

data/docs/ripper_translation.md

@@ -0,0 +1,72 @@
+# Ripper translation
+
+Prism provides the ability to mirror the `Ripper` standard library. You can do this by:
+
+```ruby
+require "prism/translation/ripper/shim"
+```
+
+This provides the APIs like:
+
+```ruby
+Ripper.lex
+Ripper.parse
+Ripper.sexp_raw
+Ripper.sexp
+
+Ripper::SexpBuilder
+Ripper::SexpBuilderPP
+```
+
+Briefly, `Ripper` is a streaming parser that allows you to construct your own syntax tree. As an example:
+
+```ruby
+class ArithmeticRipper < Prism::Translation::Ripper
+  def on_binary(left, operator, right)
+    left.public_send(operator, right)
+  end
+
+  def on_int(value)
+    value.to_i
+  end
+
+  def on_program(stmts)
+    stmts
+  end
+
+  def on_stmts_new
+    []
+  end
+
+  def on_stmts_add(stmts, stmt)
+    stmts << stmt
+    stmts
+  end
+end
+
+ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
+```
+
+The exact names of the `on_*` methods are listed in the `Ripper` source.
+
+## Background
+
+It is helpful to understand the differences between the `Ripper` library and the `Prism` library. Both libraries perform parsing and provide you with APIs to manipulate and understand the resulting syntax tree. However, there are a few key differences.
+
+### Design
+
+`Ripper` is a streaming parser. This means as it is parsing Ruby code, it dispatches events back to the consumer. This allows quite a bit of flexibility. You can use it to build your own syntax tree or to find specific patterns in the code. `Prism` on the other hand returns to your the completed syntax tree _before_ it allows you to manipulate it. This means the tree that you get back is the only representation that can be generated by the parser _at parse time_ (but of course can be manipulated later).
+
+### Fields
+
+We use the term "field" to mean a piece of information on a syntax tree node. `Ripper` provides the minimal number of fields to accurately represent the syntax tree for the purposes of compilation/interpretation. For example, in the callbacks for nodes that are based on keywords (`class`, `module`, `for`, `while`, etc.) you are not given the keyword itself, you need to attach it on your own. In other cases, tokens are not necessarily dispatched at all, meaning you need to find them yourself. `Prism` provides the opposite: the maximum number of fields on nodes is provided. As a tradeoff, this requires more memory, but this is chosen to make it easier on consumers.
+
+### Maintainability
+
+The `Ripper` interface is not guaranteed in any way, and tends to change between patch versions of CRuby. This is largely due to the fact that `Ripper` is a by-product of the generated parser, as opposed to its own parser. As an example, in the expression `foo::bar = baz`, there are three different represents possible for the call operator, including:
+
+* `:"::"` - Ruby 1.9 to Ruby 3.1.4
+* `73` - Ruby 3.1.5 to Ruby 3.1.6
+* `[:@op, "::", [lineno, column]]` - Ruby 3.2.0 and later
+
+The `Prism` interface is guaranteed going forward to be the consistent, and the official Ruby syntax tree interface. This means you can rely on this interface without having to worry about individual changes between Ruby versions. It also is a gem, which means it is versioned based on the gem version, as opposed to being versioned based on the Ruby version. Finally, you can use `Prism` to parse multiple versions of Ruby, whereas `Ripper` is tied to the Ruby version it is running on.

data/docs/ruby_api.md

@@ -20,9 +20,10 @@ The full API is documented below.
 * `Prism.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
 * `Prism.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
 * `Prism.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
+* `Prism.parse_stream(io)` - parse the syntax tree corresponding to the source that is read out of the given IO object using the `#gets` method and return it within a parse result
 * `Prism.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
 * `Prism.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
-* `Prism.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree
+* `Prism.load(source, serialized, freeze = false)` - load the serialized syntax tree using the source as a reference into a syntax tree
 * `Prism.parse_comments(source)` - parse the comments corresponding to the given source string and return them
 * `Prism.parse_file_comments(source)` - parse the comments corresponding to the given source file and return them
 * `Prism.parse_success?(source)` - parse the syntax tree corresponding to the given source string and return true if it was parsed without errors

data/docs/serialization.md

@@ -55,18 +55,28 @@ The comment type is one of:
 
 | # bytes | field |
 | --- | --- |
+| varuint | type |
 | string | error message (ASCII-only characters) |
 | location | the location in the source this error applies to |
-| `1` | the level of the error: `0` for `fatal` |
+| `1` | the level of the error: `0` for `fatal`, `1` for `argument`, `2` for `load` |
 
-## warning
+### warning
 
 | # bytes | field |
 | --- | --- |
+| varuint | type |
 | string | warning message (ASCII-only characters) |
 | location | the location in the source this warning applies to |
 | `1` | the level of the warning: `0` for `default` and `1` for `verbose` |
 
+### integer
+
+| # bytes | field |
+| --- | --- |
+| `1` | `1` if the integer is negative, `0` if the integer is positive |
+| varuint | the number of words in this integer |
+| varuint+ | the words of the integer, least-significant to most-significant |
+
 ## Structure
 
 The serialized string representing the syntax tree is composed of three parts: the header, the body, and the constant pool.
@@ -106,16 +116,20 @@ Each node is structured like the following table:
 | # bytes | field |
 | --- | --- |
 | `1` | node type |
+| varuint | node identifier |
 | location | node location |
+| varuint | node flags |
 
 Every field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
 
+* `double` - A field that is a `double`. This is structured as a sequence of 8 bytes in native endian order.
 * `node` - A field that is a node. This is structured just as like parent node.
 * `node?` - A field that is a node that is optionally present. If the node is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just as like parent node.
 * `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, followed by the child nodes themselves.
 * `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string itself (_without_ a trailing null byte).
 * `constant` - A variable-length integer that represents an index in the constant pool.
 * `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
+* `integer` - A field that represents an arbitrary-sized integer. The structure is listed above.
 * `location` - A field that is a location. This is structured as a variable-length integer start followed by a variable-length integer length.
 * `location?` - A field that is a location that is optionally present. If the location is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just like the `location` child node.
 * `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
@@ -185,21 +199,31 @@ The final argument to `pm_serialize_parse` is an optional string that controls t
 | `4`     | the length the encoding    |
 | ...     | the encoding bytes         |
 | `1`     | frozen string literal      |
-| `1`     | suppress warnings          |
+| `1`     | command line flags         |
 | `1`     | syntax version, see [pm_options_version_t](https://github.com/ruby/prism/blob/main/include/prism/options.h) for valid values |
+| `1`     | whether or not the encoding is locked (should almost always be false) |
 | `4`     | the number of scopes       |
 | ...     | the scopes                 |
 
+Command line flags are a bitset. By default every flag is `0`. It includes the following values:
+
+* `0x1` - the `-a` option
+* `0x2` - the `-e` option
+* `0x4` - the `-l` option
+* `0x8` - the `-n` option
+* `0x10` - the `-p` option
+* `0x20` - the `-x` option
+
 Scopes are ordered from the outermost scope to the innermost one.
 
-Each scope is layed out as follows:
+Each scope is laid out as follows:
 
 | # bytes | field                      |
 | ------- | -------------------------- |
 | `4`     | the number of locals       |
 | ...     | the locals                 |
 
-Each local is layed out as follows:
+Each local is laid out as follows:
 
 | # bytes | field                      |
 | ------- | -------------------------- |