prism 0.29.0 → 1.1.0

data/docs/fuzzing.md

@@ -26,7 +26,7 @@ fuzz
 There are currently three fuzzing targets
 
 - `pm_serialize_parse` (parse)
-- `pm_regexp_named_capture_group_names` (regexp)
+- `pm_regexp_parse` (regexp)
 
 Respectively, fuzzing can be performed with
 

data/docs/parsing_rules.md

@@ -12,7 +12,10 @@ Constants in Ruby begin with an upper-case letter. This is followed by any numbe
 
 Most expressions in CRuby are non-void. This means the expression they represent resolves to a value. For example, `1 + 2` is a non-void expression, because it resolves to a method call. Even things like `class Foo; end` is a non-void expression, because it returns the last evaluated expression in the body of the class (or `nil`).
 
-Certain nodes, however, are void expressions, and cannot be combined to form larger expressions. For example, `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo`.
+Certain nodes, however, are void expressions, and cannot be combined to form larger expressions.
+* `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo` can only be at a statement position.
+* The "jumps": `return`, `break`, `next`, `redo`, `retry` are void expressions.
+* `value => pattern` is also considered a void expression.
 
 ## Identifiers
 

data/docs/ripper_translation.md

@@ -48,3 +48,25 @@ ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
 ```
 
 The exact names of the `on_*` methods are listed in the `Ripper` source.
+
+## Background
+
+It is helpful to understand the differences between the `Ripper` library and the `Prism` library. Both libraries perform parsing and provide you with APIs to manipulate and understand the resulting syntax tree. However, there are a few key differences.
+
+### Design
+
+`Ripper` is a streaming parser. This means as it is parsing Ruby code, it dispatches events back to the consumer. This allows quite a bit of flexibility. You can use it to build your own syntax tree or to find specific patterns in the code. `Prism` on the other hand returns to your the completed syntax tree _before_ it allows you to manipulate it. This means the tree that you get back is the only representation that can be generated by the parser _at parse time_ (but of course can be manipulated later).
+
+### Fields
+
+We use the term "field" to mean a piece of information on a syntax tree node. `Ripper` provides the minimal number of fields to accurately represent the syntax tree for the purposes of compilation/interpretation. For example, in the callbacks for nodes that are based on keywords (`class`, `module`, `for`, `while`, etc.) you are not given the keyword itself, you need to attach it on your own. In other cases, tokens are not necessarily dispatched at all, meaning you need to find them yourself. `Prism` provides the opposite: the maximum number of fields on nodes is provided. As a tradeoff, this requires more memory, but this is chosen to make it easier on consumers.
+
+### Maintainability
+
+The `Ripper` interface is not guaranteed in any way, and tends to change between patch versions of CRuby. This is largely due to the fact that `Ripper` is a by-product of the generated parser, as opposed to its own parser. As an example, in the expression `foo::bar = baz`, there are three different represents possible for the call operator, including:
+
+* `:"::"` - Ruby 1.9 to Ruby 3.1.4
+* `73` - Ruby 3.1.5 to Ruby 3.1.6
+* `[:@op, "::", [lineno, column]]` - Ruby 3.2.0 and later
+
+The `Prism` interface is guaranteed going forward to be the consistent, and the official Ruby syntax tree interface. This means you can rely on this interface without having to worry about individual changes between Ruby versions. It also is a gem, which means it is versioned based on the gem version, as opposed to being versioned based on the Ruby version. Finally, you can use `Prism` to parse multiple versions of Ruby, whereas `Ripper` is tied to the Ruby version it is running on.

data/docs/serialization.md

@@ -116,7 +116,9 @@ Each node is structured like the following table:
 | # bytes | field |
 | --- | --- |
 | `1` | node type |
+| varuint | node identifier |
 | location | node location |
+| varuint | node flags |
 
 Every field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
 
@@ -199,6 +201,7 @@ The final argument to `pm_serialize_parse` is an optional string that controls t
 | `1`     | frozen string literal      |
 | `1`     | command line flags         |
 | `1`     | syntax version, see [pm_options_version_t](https://github.com/ruby/prism/blob/main/include/prism/options.h) for valid values |
+| `1`     | whether or not the encoding is locked (should almost always be false) |
 | `4`     | the number of scopes       |
 | ...     | the scopes                 |