prism 0.24.0 → 0.25.0

data/docs/build_system.md

@@ -73,6 +73,25 @@ and links to `libprism.a` (to avoid exporting symbols, so no conflict when insta
 
 TODO, similar to TruffleRuby.
 
+### Building prism for embedded system
+
+For instance, you can build a static library `libprism.a` targeting the Arm Cortex-M0+ embedded system by the commands below:
+
+* `templates/template.rb`
+* `CFLAGS="-mcpu=cortex-m0plus" make static CC=arm-none-eabi-gcc`
+
+The build process internally looks up `_POSIX_MAPPED_FILES` and `_WIN32` macros to determine whether the functions of the memory map are available on the target platform.
+
+### Building prism with custom memory allocator
+
+If you need to use memory allocation functions implemented outside of the standard library, follow these steps:
+
+* Add `-D PRISM_XALLOCATOR` to the build options
+* Additionally, include `-I [path/to/custom_allocator]` where your `prism_xallocator.h` is located
+* Link the implementation of `prism_xallocator.c` that contains functions declared in `prism_xallocator.h`
+
+For further clarity, refer to `include/prism/defines.h`.
+
 ### Building prism from source as a C library
 
 All of the source files match `src/**/*.c` and all of the headers match `include/**/*.h`.
@@ -89,3 +108,15 @@ If you want to build prism as a shared library and link against it, you should c
 ```
 MAKEFLAGS="-j10" bundle exec rake compile
 ```
+
+## Build options
+
+* `PRISM_BUILD_DEBUG` - Will cause all file reading to copy into its own allocation to allow easier tracking of reading off the end of the buffer. By default this is off.
+* `PRISM_BUILD_MINIMAL` - Define all of the `PRISM_EXCLUDE_*` flags at once.
+* `PRISM_ENCODING_EXCLUDE_FULL` - Will cause the library to exclude the full encoding API, and only include the minimal number of encodings to support parsing Ruby code without encoding comments. By default this is off.
+* `PRISM_EXPORT_SYMBOLS` - Will cause the shared library to export symbols. By default this is off.
+* `PRISM_EXCLUDE_JSON` - Will cause the library to exclude the JSON API. By default this is off.
+* `PRISM_EXCLUDE_PACK` - Will cause the library to exclude the pack API. By default this is off.
+* `PRISM_EXCLUDE_PRETTYPRINT` - Will cause the library to exclude the prettyprint API. By default this is off.
+* `PRISM_EXCLUDE_SERIALIZATION` - Will cause the library to exclude the serialization API. By default this is off.
+* `PRISM_XALLOCATOR` - Will cause the library to use the custom memory allocator. By default this is off.

data/docs/configuration.md

@@ -4,6 +4,7 @@ A lot of code in prism's repository is templated from a single configuration fil
 
 * `ext/prism/api_node.c` - for defining how to build Ruby objects for the nodes out of C structs
 * `include/prism/ast.h` - for defining the C structs that represent the nodes
+* `include/prism/diagnostic.h` - for defining the diagnostics
 * `javascript/src/deserialize.js` - for defining how to deserialize the nodes in JavaScript
 * `javascript/src/nodes.js` - for defining the nodes in JavaScript
 * `java/org/prism/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
@@ -17,6 +18,7 @@ A lot of code in prism's repository is templated from a single configuration fil
 * `lib/prism/node.rb` - for defining the nodes in Ruby
 * `lib/prism/serialize.rb` - for defining how to deserialize the nodes in Ruby
 * `lib/prism/visitor.rb` - for defining the visitor interface for the nodes in Ruby
+* `src/diagnostic.c` - for defining how to build diagnostics
 * `src/node.c` - for defining how to free the nodes in C and calculate the size in memory in C
 * `src/prettyprint.c` - for defining how to prettyprint the nodes in C
 * `src/serialize.c` - for defining how to serialize the nodes in C

data/docs/cruby_compilation.md

@@ -10,7 +10,7 @@ ruby/ruby uses the Prism code to generate an AST from which it can generate inst
 
 1. Compute an AST
 
-Syncing over the Prism code allows ruby/ruby to compute the AST using Prism. It currently does this within [`iseq.c`](https://github.com/ruby/ruby/blob/master/iseq.c) using the `pm_parser_init` fuction.
+Syncing over the Prism code allows ruby/ruby to compute the AST using Prism. It currently does this within [`iseq.c`](https://github.com/ruby/ruby/blob/master/iseq.c) using the `pm_parser_init` function.
 
 2. Run a first pass of compilation
 

data/docs/parser_translation.md

@@ -16,19 +16,24 @@ Prism::Translation::Parser.parse_file("path/to/file.rb")
 
 ### RuboCop
 
-To run RuboCop using the `prism` gem as the parser, you will need to require the `prism/translation/parser/rubocop` file. This file injects `prism` into the known options for both `rubocop` and `rubocop-ast`, such that you can specify it in your `.rubocop.yml` file. Unfortunately `rubocop` doesn't support any direct way to do this, so we have to get a bit hacky.
+Prism as a parser engine is directly supported since RuboCop 1.62. The class used for parsing is `Prism::Translation::Parser`.
 
-First, set the `TargetRubyVersion` in your RuboCop configuration file to `80_82_73_83_77.33`. This is the version of Ruby that `prism` reports itself as. (The leading numbers are the ASCII values for `PRISM`.)
+First, specify `prism` in your Gemfile:
 
-```yaml
-AllCops:
-  TargetRubyVersion: 80_82_73_83_77.33
+```ruby
+gem "prism"
 ```
 
-Now when you run `rubocop` you will need to require the `prism/translation/parser/rubocop` file before executing so that it can inject the `prism` parser into the known options.
+To use Prism with RuboCop, specify `ParserEngine` and `TargetRubyVersion` in your RuboCop configuration file:
 
+```yaml
+AllCops:
+  ParserEngine: parser_prism
+  TargetRubyVersion: 3.3
 ```
-bundle exec ruby -rprism/translation/parser/rubocop $(bundle exec which rubocop)
-```
 
-This should run RuboCop using the `prism` parser.
+The default value for `ParserEngine` is `parser_whitequark`, which indicates the Parser gem. You need to explicitly switch it to `parser_prism` to indicate Prism. Additionally, the value for `TargetRubyVersion` must be specified as `3.3` or higher, as Prism supports parsing versions of Ruby 3.3 and higher.
+The parser class is determined by the combination of values for `ParserEngine` and `TargetRubyVersion`. For example, if `TargetRubyVersion: 3.3`, parsing is performed by `Prism::Translation::Parser33`, and for `TargetRubyVersion 3.4`, parsing is performed by `Prism::Translation::Parser34`.
+
+For further information, please refer to the RuboCop documentation:
+https://docs.rubocop.org/rubocop/configuration.html#setting-the-parser-engine

data/docs/releasing.md

@@ -54,8 +54,8 @@ chruby ruby-3.2.3 && BUNDLE_GEMFILE=gemfiles/3.2/Gemfile bundle install
 chruby ruby-3.3.0 && BUNDLE_GEMFILE=gemfiles/3.3/Gemfile bundle install
 chruby ruby-3.4.0-dev && BUNDLE_GEMFILE=gemfiles/3.4/Gemfile bundle install
 chruby jruby-9.4.5.0 && BUNDLE_GEMFILE=gemfiles/jruby/Gemfile bundle install
-chruby truffleruby-23.1.2 && BUNDLE_GEMFILE=gemfiles/truffleruby/Gemfile bundle install
-chruby ruby-3.4.0-dev
+chruby truffleruby-24.0.0 && BUNDLE_GEMFILE=gemfiles/truffleruby/Gemfile bundle install
+chruby ruby-3.4.0-dev && BUNDLE_GEMFILE=gemfiles/typecheck/Gemfile bundle install
 ```
 
 * Update the cargo lockfiles:

data/docs/ripper_translation.md

@@ -0,0 +1,50 @@
+# Ripper translation
+
+Prism provides the ability to mirror the `Ripper` standard library. You can do this by:
+
+```ruby
+require "prism/translation/ripper/shim"
+```
+
+This provides the APIs like:
+
+```ruby
+Ripper.lex
+Ripper.parse
+Ripper.sexp_raw
+Ripper.sexp
+
+Ripper::SexpBuilder
+Ripper::SexpBuilderPP
+```
+
+Briefly, `Ripper` is a streaming parser that allows you to construct your own syntax tree. As an example:
+
+```ruby
+class ArithmeticRipper < Prism::Translation::Ripper
+  def on_binary(left, operator, right)
+    left.public_send(operator, right)
+  end
+
+  def on_int(value)
+    value.to_i
+  end
+
+  def on_program(stmts)
+    stmts
+  end
+
+  def on_stmts_new
+    []
+  end
+
+  def on_stmts_add(stmts, stmt)
+    stmts << stmt
+    stmts
+  end
+end
+
+ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
+```
+
+The exact names of the `on_*` methods are listed in the `Ripper` source.

data/docs/ruby_api.md

@@ -20,6 +20,7 @@ The full API is documented below.
 * `Prism.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
 * `Prism.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
 * `Prism.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
+* `Prism.parse_stream(io)` - parse the syntax tree corresponding to the source that is read out of the given IO object using the `#gets` method and return it within a parse result
 * `Prism.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
 * `Prism.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
 * `Prism.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree

data/docs/serialization.md

@@ -55,18 +55,28 @@ The comment type is one of:
 
 | # bytes | field |
 | --- | --- |
+| varuint | type |
 | string | error message (ASCII-only characters) |
 | location | the location in the source this error applies to |
-| `1` | the level of the error: `0` for `fatal` |
+| `1` | the level of the error: `0` for `fatal`, `1` for `argument`, `2` for `load` |
 
-## warning
+### warning
 
 | # bytes | field |
 | --- | --- |
+| varuint | type |
 | string | warning message (ASCII-only characters) |
 | location | the location in the source this warning applies to |
 | `1` | the level of the warning: `0` for `default` and `1` for `verbose` |
 
+### integer
+
+| # bytes | field |
+| --- | --- |
+| `1` | `1` if the integer is negative, `0` if the integer is positive |
+| varuint | the number of words in this integer |
+| varuint+ | the words of the integer, least-significant to most-significant |
+
 ## Structure
 
 The serialized string representing the syntax tree is composed of three parts: the header, the body, and the constant pool.
@@ -110,12 +120,14 @@ Each node is structured like the following table:
 
 Every field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
 
+* `double` - A field that is a `double`. This is structured as a sequence of 8 bytes in native endian order.
 * `node` - A field that is a node. This is structured just as like parent node.
 * `node?` - A field that is a node that is optionally present. If the node is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just as like parent node.
 * `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, followed by the child nodes themselves.
 * `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string itself (_without_ a trailing null byte).
 * `constant` - A variable-length integer that represents an index in the constant pool.
 * `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
+* `integer` - A field that represents an arbitrary-sized integer. The structure is listed above.
 * `location` - A field that is a location. This is structured as a variable-length integer start followed by a variable-length integer length.
 * `location?` - A field that is a location that is optionally present. If the location is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just like the `location` child node.
 * `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
@@ -185,21 +197,30 @@ The final argument to `pm_serialize_parse` is an optional string that controls t
 | `4`     | the length the encoding    |
 | ...     | the encoding bytes         |
 | `1`     | frozen string literal      |
-| `1`     | suppress warnings          |
+| `1`     | command line flags         |
 | `1`     | syntax version, see [pm_options_version_t](https://github.com/ruby/prism/blob/main/include/prism/options.h) for valid values |
 | `4`     | the number of scopes       |
 | ...     | the scopes                 |
 
+Command line flags are a bitset. By default every flag is `0`. It includes the following values:
+
+* `0x1` - the `-a` option
+* `0x2` - the `-e` option
+* `0x4` - the `-l` option
+* `0x8` - the `-n` option
+* `0x10` - the `-p` option
+* `0x20` - the `-x` option
+
 Scopes are ordered from the outermost scope to the innermost one.
 
-Each scope is layed out as follows:
+Each scope is laid out as follows:
 
 | # bytes | field                      |
 | ------- | -------------------------- |
 | `4`     | the number of locals       |
 | ...     | the locals                 |
 
-Each local is layed out as follows:
+Each local is laid out as follows:
 
 | # bytes | field                      |
 | ------- | -------------------------- |