prism 2.0.0.rc1-x86_64-linux-gnu

data/docs/local_variable_depth.md

@@ -0,0 +1,229 @@
+# Local variable depth
+
+One feature of Prism is that it resolves local variables as it parses. It's necessary to do this because of ambiguities in the grammar. For example, consider the following code:
+
+```ruby
+foo / bar#/
+```
+
+If `foo` is a local variable, this is a call to `/` with `bar` as an argument, followed by a comment. If it's not a local variable, this is a method call to `foo` with a regular expression argument.
+
+"Depth" refers to the number of visible scopes that Prism has to go up to find the declaration of a local variable.
+Note that this follows the same scoping rules as Ruby, so a local variable is only visible in the scope it is declared in and in blocks nested in that scope.
+The rules for calculating the depth are very important to understand because they may differ from individual Ruby implementations since they are not specified by the language.
+
+Prism uses the minimum number of scopes, i.e., it only creates scopes when necessary semantically, in other words when there must be distinct scopes (which can be observed through `binding.local_variables`).
+That are no "transparent/invisible" scopes in Prism.
+Some Ruby implementations use those for some language constructs and need to adjust by maintaining a depth offset.
+
+Below are the places where a local variable can be written/targeted, along with how the depth is calculated at that point.
+
+## General
+
+In the course of general Ruby code when reading a local variable, the depth is equal to the number of scopes to go up to find the declaration of that variable. For example:
+
+```ruby
+foo = 1
+bar = 2
+baz = 3
+
+foo # depth 0
+tap { bar } # depth 1
+tap { tap { baz } } # depth 2
+```
+
+This also includes writing to a local variable, which could be writing to a local variable that is already declared. For example:
+
+```ruby
+foo = 1
+bar = 2
+
+foo = 3 # depth 0
+tap { bar = 4 } # depth 1
+```
+
+This includes multiple assignment, where the same principle applies. For example:
+
+```ruby
+foo = 1
+bar = 2
+
+foo, bar = 3, 4 # depth 0
+tap { foo, bar = 5, 6 } # depth 1
+```
+
+## `for` loops
+
+`for` loops in Ruby break down to calls to `.each` with a block.
+However in that case local variable reads and writes within the block will be in the same scope as the scope surrounding the `for` and not in a deeper/separate scope (surprising, but this is Ruby semantics).
+For example:
+
+```ruby
+foo = 1
+
+for e in baz
+  foo # depth 0
+  bar = 2 # depth 0
+end
+
+p bar # depth 0, prints 2
+```
+
+The local variable(s) used for the index of the `for` are also at the same depth (as variables inside and outside the `for`):
+
+```ruby
+for e in [1, 2] # depth 0
+  e # depth 0
+end
+
+p e # depth 0, prints 2
+```
+
+## Pattern matching captures
+
+You can target a local variable in a pattern matching expression using capture syntax. Using this syntax, you can target local variables in the current scope or in visible parent scopes. For example:
+
+```ruby
+42 => bar # depth 0
+```
+
+The example above writes to a local variable in the current scope. If the variable is already declared in a higher visible scope, it will be written to that scope instead. For example:
+
+```ruby
+foo = 1
+tap { 42 => foo } # depth 1
+```
+
+## Named capture groups
+
+You can target local variables through named capture groups in regular expressions if they are used on the left-hand side of a `=~` operator. For example:
+
+```ruby
+/(?<foo>\d+)/ =~ "42" # depth 0
+```
+
+This will write to a `foo` local variable. If the variable is already declared in a higher visible scope, it will be written to that scope instead. For example:
+
+```ruby
+foo = 1
+tap { /(?<foo>\d+)/ =~ "42" } # depth 1
+```
+
+## "interpolated once" regular expressions
+
+Regular expressions that interpolate local variables (unrelated to capture group local variables) and have the `o` flag will only interpolate the local variables once for the runtime of the program.
+In CRuby, this is implemented by compiling the regular expression within a nested instruction sequence, which means CRuby thinks the depth is one more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'foo = 1; /#{foo}/o'
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,18)> (catch: false)
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 once                                   block in <main>, <is:0>
+0006 leave
+
+== disasm: #<ISeq:block in <main>@-e:1 (1,9)-(1,18)> (catch: false)
+0000 putobject                              ""                        (   1)
+0002 getlocal_WC_1                          foo@0
+0004 dup
+0005 objtostring                            <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE>
+0007 anytostring
+0008 toregexp                               0, 2
+0011 leave
+```
+
+In this case CRuby fetches the local variable with `getlocal_WC_1` as the second instruction to the "once" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+## `rescue` clauses
+
+In CRuby, `rescue` clauses are implemented as their own instruction sequence, and therefore CRuby thinks the depth is one more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'begin; foo = 1; rescue; foo; end'
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,32)> (catch: true)
+== catch table
+| catch type: rescue st: 0000 ed: 0004 sp: 0000 cont: 0005
+| == disasm: #<ISeq:rescue in <main>@-e:1 (1,16)-(1,28)> (catch: true)
+| local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+| [ 1] $!@0
+| 0000 getlocal_WC_0                          $!@0                      (   1)
+| 0002 putobject                              StandardError
+| 0004 checkmatch                             3
+| 0006 branchunless                           11
+| 0008 getlocal_WC_1                          foo@0[Li]
+| 0010 leave
+| 0011 getlocal_WC_0                          $!@0
+| 0013 throw                                  0
+| catch type: retry  st: 0004 ed: 0005 sp: 0000 cont: 0000
+|------------------------------------------------------------------------
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 dup
+0002 setlocal_WC_0                          foo@0
+0004 nop
+0005 leave
+```
+
+In the catch table, CRuby is reading the `foo` local variable using `getlocal_WC_1` as the fifth instruction to the "rescue" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+Note that this includes the error reference, which can target local variables, as in:
+
+```
+$ ruby --dump=insns -e 'foo = 1; begin; rescue => foo; end'              
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,34)> (catch: true)
+== catch table
+| catch type: rescue st: 0003 ed: 0004 sp: 0000 cont: 0005
+| == disasm: #<ISeq:rescue in <main>@-e:1 (1,16)-(1,30)> (catch: true)
+| local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+| [ 1] $!@0
+| 0000 getlocal_WC_0                          $!@0                      (   1)
+| 0002 putobject                              StandardError
+| 0004 checkmatch                             3
+| 0006 branchunless                           14
+| 0008 getlocal_WC_0                          $!@0
+| 0010 setlocal_WC_1                          foo@0
+| 0012 putnil
+| 0013 leave
+| 0014 getlocal_WC_0                          $!@0
+| 0016 throw                                  0
+| catch type: retry  st: 0004 ed: 0005 sp: 0000 cont: 0003
+|------------------------------------------------------------------------
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 putnil
+0004 nop
+0005 leave
+```
+
+Note that CRuby is writing to the `foo` local variable using the `setlocal_WC_1` instruction as the sixth instruction to the "rescue" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+## Post execution blocks
+
+The `END {}` syntax allows executing code when the program exits. In CRuby, this is implemented as two nested instruction sequences. CRuby therefore thinks the depth is two more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'foo = 1; END { foo }'            
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,20)> (catch: false)
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 once                                   block in <main>, <is:0>
+0006 leave
+
+== disasm: #<ISeq:block in <main>@-e:0 (0,0)-(-1,-1)> (catch: false)
+0000 putspecialobject                       1                         (   1)
+0002 send                                   <calldata!mid:core#set_postexe, argc:0, FCALL>, block in <main>
+0005 leave
+
+== disasm: #<ISeq:block in <main>@-e:1 (1,9)-(1,20)> (catch: false)
+0000 getlocal                               foo@0, 2                  (   1)[LiBc]
+0003 leave                                  [Br]
+```
+
+In the instruction sequence corresponding to the code that gets executed inside the `END` block, CRuby is reading the `foo` local variable using `getlocal` as the second instruction to the `"block in <main>"` instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.

data/docs/mapping.md

@@ -0,0 +1,117 @@
+# Mapping
+
+When considering the previous CRuby parser versus prism, this document should be helpful to understand how various concepts are mapped.
+
+## Nodes
+
+The following table shows how the various CRuby nodes are mapped to prism nodes.
+
+| CRuby | prism |
+| --- | --- |
+| `NODE_SCOPE` | |
+| `NODE_BLOCK` | |
+| `NODE_IF` | `PM_IF_NODE` |
+| `NODE_UNLESS` | `PM_UNLESS_NODE` |
+| `NODE_CASE` | `PM_CASE_NODE` |
+| `NODE_CASE2` | `PM_CASE_NODE` (with a null predicate) |
+| `NODE_CASE3` | |
+| `NODE_WHEN` | `PM_WHEN_NODE` |
+| `NODE_IN` | `PM_IN_NODE` |
+| `NODE_WHILE` | `PM_WHILE_NODE` |
+| `NODE_UNTIL` | `PM_UNTIL_NODE` |
+| `NODE_ITER` | `PM_CALL_NODE` (with a non-null block) |
+| `NODE_FOR` | `PM_FOR_NODE` |
+| `NODE_FOR_MASGN` | `PM_FOR_NODE` (with a multi-write node as the index) |
+| `NODE_BREAK` | `PM_BREAK_NODE` |
+| `NODE_NEXT` | `PM_NEXT_NODE` |
+| `NODE_REDO` | `PM_REDO_NODE` |
+| `NODE_RETRY` | `PM_RETRY_NODE` |
+| `NODE_BEGIN` | `PM_BEGIN_NODE` |
+| `NODE_RESCUE` | `PM_RESCUE_NODE` |
+| `NODE_RESBODY` | |
+| `NODE_ENSURE` | `PM_ENSURE_NODE` |
+| `NODE_AND` | `PM_AND_NODE` |
+| `NODE_OR` | `PM_OR_NODE` |
+| `NODE_MASGN` | `PM_MULTI_WRITE_NODE` |
+| `NODE_LASGN` | `PM_LOCAL_VARIABLE_WRITE_NODE` |
+| `NODE_DASGN` | `PM_LOCAL_VARIABLE_WRITE_NODE` |
+| `NODE_GASGN` | `PM_GLOBAL_VARIABLE_WRITE_NODE` |
+| `NODE_IASGN` | `PM_INSTANCE_VARIABLE_WRITE_NODE` |
+| `NODE_CDECL` | `PM_CONSTANT_PATH_WRITE_NODE` |
+| `NODE_CVASGN` | `PM_CLASS_VARIABLE_WRITE_NODE` |
+| `NODE_OP_ASGN1` | |
+| `NODE_OP_ASGN2` | |
+| `NODE_OP_ASGN_AND` | `PM_OPERATOR_AND_ASSIGNMENT_NODE` |
+| `NODE_OP_ASGN_OR` | `PM_OPERATOR_OR_ASSIGNMENT_NODE` |
+| `NODE_OP_CDECL` | |
+| `NODE_CALL` | `PM_CALL_NODE` |
+| `NODE_OPCALL` | `PM_CALL_NODE` (with an operator as the method) |
+| `NODE_FCALL` | `PM_CALL_NODE` (with a null receiver and parentheses) |
+| `NODE_VCALL` | `PM_CALL_NODE` (with a null receiver and parentheses or arguments) |
+| `NODE_QCALL` | `PM_CALL_NODE` (with a &. operator) |
+| `NODE_SUPER` | `PM_SUPER_NODE` |
+| `NODE_ZSUPER` | `PM_FORWARDING_SUPER_NODE` |
+| `NODE_LIST` | `PM_ARRAY_NODE` |
+| `NODE_ZLIST` | `PM_ARRAY_NODE` (with no child elements) |
+| `NODE_VALUES` | `PM_ARGUMENTS_NODE` |
+| `NODE_HASH` | `PM_HASH_NODE` |
+| `NODE_RETURN` | `PM_RETURN_NODE` |
+| `NODE_YIELD` | `PM_YIELD_NODE` |
+| `NODE_LVAR` | `PM_LOCAL_VARIABLE_READ_NODE` |
+| `NODE_DVAR` | `PM_LOCAL_VARIABLE_READ_NODE` |
+| `NODE_GVAR` | `PM_GLOBAL_VARIABLE_READ_NODE` |
+| `NODE_IVAR` | `PM_INSTANCE_VARIABLE_READ_NODE` |
+| `NODE_CONST` | `PM_CONSTANT_PATH_READ_NODE` |
+| `NODE_CVAR` | `PM_CLASS_VARIABLE_READ_NODE` |
+| `NODE_NTH_REF` | `PM_NUMBERED_REFERENCE_READ_NODE` |
+| `NODE_BACK_REF` | `PM_BACK_REFERENCE_READ_NODE` |
+| `NODE_MATCH` | |
+| `NODE_MATCH2` | `PM_CALL_NODE` (with regular expression as receiver) |
+| `NODE_MATCH3` | `PM_CALL_NODE` (with regular expression as only argument) |
+| `NODE_LIT` | |
+| `NODE_STR` | `PM_STRING_NODE` |
+| `NODE_DSTR` | `PM_INTERPOLATED_STRING_NODE` |
+| `NODE_XSTR` | `PM_X_STRING_NODE` |
+| `NODE_DXSTR` | `PM_INTERPOLATED_X_STRING_NODE` |
+| `NODE_EVSTR` | `PM_STRING_INTERPOLATED_NODE` |
+| `NODE_DREGX` | `PM_INTERPOLATED_REGULAR_EXPRESSION_NODE` |
+| `NODE_ONCE` | |
+| `NODE_ARGS` | `PM_PARAMETERS_NODE` |
+| `NODE_ARGS_AUX` | |
+| `NODE_OPT_ARG` | `PM_OPTIONAL_PARAMETER_NODE` |
+| `NODE_KW_ARG` | `PM_KEYWORD_PARAMETER_NODE` |
+| `NODE_POSTARG` | `PM_REQUIRED_PARAMETER_NODE` |
+| `NODE_ARGSCAT` | |
+| `NODE_ARGSPUSH` | |
+| `NODE_SPLAT` | `PM_SPLAT_NODE` |
+| `NODE_BLOCK_PASS` | `PM_BLOCK_ARGUMENT_NODE` |
+| `NODE_DEFN` | `PM_DEF_NODE` (with a null receiver) |
+| `NODE_DEFS` | `PM_DEF_NODE` (with a non-null receiver) |
+| `NODE_ALIAS` | `PM_ALIAS_NODE` |
+| `NODE_VALIAS` | `PM_ALIAS_NODE` (with a global variable first argument) |
+| `NODE_UNDEF` | `PM_UNDEF_NODE` |
+| `NODE_CLASS` | `PM_CLASS_NODE` |
+| `NODE_MODULE` | `PM_MODULE_NODE` |
+| `NODE_SCLASS` | `PM_S_CLASS_NODE` |
+| `NODE_COLON2` | `PM_CONSTANT_PATH_NODE` |
+| `NODE_COLON3` | `PM_CONSTANT_PATH_NODE` (with a null receiver) |
+| `NODE_DOT2` | `PM_RANGE_NODE` (with a .. operator) |
+| `NODE_DOT3` | `PM_RANGE_NODE` (with a ... operator) |
+| `NODE_FLIP2` | `PM_RANGE_NODE` (with a .. operator) |
+| `NODE_FLIP3` | `PM_RANGE_NODE` (with a ... operator) |
+| `NODE_SELF` | `PM_SELF_NODE` |
+| `NODE_NIL` | `PM_NIL_NODE` |
+| `NODE_TRUE` | `PM_TRUE_NODE` |
+| `NODE_FALSE` | `PM_FALSE_NODE` |
+| `NODE_ERRINFO` | |
+| `NODE_DEFINED` | `PM_DEFINED_NODE` |
+| `NODE_POSTEXE` | `PM_POST_EXECUTION_NODE` |
+| `NODE_DSYM` | `PM_INTERPOLATED_SYMBOL_NODE` |
+| `NODE_ATTRASGN` | `PM_CALL_NODE` (with a message that ends with =) |
+| `NODE_LAMBDA` | `PM_LAMBDA_NODE` |
+| `NODE_ARYPTN` | `PM_ARRAY_PATTERN_NODE` |
+| `NODE_HSHPTN` | `PM_HASH_PATTERN_NODE` |
+| `NODE_FNDPTN` | `PM_FIND_PATTERN_NODE` |
+| `NODE_ERROR` | `PM_ERROR_RECOVERY_NODE` |
+| `NODE_LAST` | |
+```

data/docs/parser_translation.md

@@ -0,0 +1,24 @@
+# parser translation
+
+Prism ships with the ability to translate its syntax tree into the syntax tree used by the [whitequark/parser](https://github.com/whitequark/parser) gem. This allows you to use tools built on top of the `parser` gem with the `prism` parser.
+
+## Usage
+
+The `parser` gem provides multiple parsers to support different versions of the Ruby grammar. This includes all of the Ruby versions going back to 1.8, as well as third-party parsers like MacRuby and RubyMotion. The `prism` gem provides another parser that uses the `prism` parser to build the syntax tree.
+
+You can use the `prism` parser like you would any other. After requiring `prism`, you should be able to call any of the regular `Parser::Base` APIs that you would normally use.
+
+```ruby
+require "prism"
+
+# Same as `Parser::Ruby34`
+Prism::Translation::Parser34.parse_file("path/to/file.rb")
+
+# Same as `Parser::CurrentRuby`
+Prism::Translation::ParserCurrent.parse("puts 'Hello World!'")
+```
+
+All the parsers are autoloaded, so you don't have to worry about requiring them yourself.
+
+If you also need to parse Ruby versions below 3.3 (for which the `prism` translation layer does not have explicit support), check out
+[this guide](https://github.com/whitequark/parser/blob/master/doc/PRISM_TRANSLATION.md) from the `parser` gem on how to use both in conjunction.

data/docs/parsing_rules.md

@@ -0,0 +1,22 @@
+# Rules
+
+This document contains information related to the rules of the parser for Ruby source code.
+
+As an example, in the documentation of many of the fields of nodes, it's mentioned that a field follows the lexing rules for `identifier` or `constant`. This document describes what those rules are.
+
+## Constants
+
+Constants in Ruby begin with an upper-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "upper-case letter" are encoding-dependent.
+
+## Non-void expression
+
+Most expressions in CRuby are non-void. This means the expression they represent resolves to a value. For example, `1 + 2` is a non-void expression, because it resolves to a method call. Even things like `class Foo; end` is a non-void expression, because it returns the last evaluated expression in the body of the class (or `nil`).
+
+Certain nodes, however, are void expressions, and cannot be combined to form larger expressions.
+* `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo` can only be at a statement position.
+* The "jumps": `return`, `break`, `next`, `redo`, `retry` are void expressions.
+* `value => pattern` is also considered a void expression.
+
+## Identifiers
+
+Identifiers in Ruby begin with an underscore or lower-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "lower-case letter" are encoding-dependent.

data/docs/releasing.md

@@ -0,0 +1,147 @@
+# Releasing
+
+To release a new version of Prism, perform the following steps:
+
+## Preparation
+
+* Update the `CHANGELOG.md` file.
+  * Add a new section for the new version at the top of the file.
+  * Fill in the relevant changes — it may be easiest to click the link for the `Unreleased` heading to find the commits.
+  * Update the links at the bottom of the file.
+* Update the version numbers in the various files that reference them:
+
+```sh
+export PRISM_MAJOR="x"
+export PRISM_MINOR="y"
+export PRISM_PATCH="z"
+export PRISM_VERSION="$PRISM_MAJOR.$PRISM_MINOR.$PRISM_PATCH"
+ruby -pi -e 'gsub(/spec\.version = ".+?"/, %Q{spec.version = "#{ENV["PRISM_VERSION"]}"})' prism.gemspec
+ruby -pi -e 'gsub(/EXPECTED_PRISM_VERSION ".+?"/, %Q{EXPECTED_PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' ext/prism/extension.h
+ruby -pi -e 'gsub(/PRISM_VERSION_MAJOR \d+/, %Q{PRISM_VERSION_MAJOR #{ENV["PRISM_MAJOR"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION_MINOR \d+/, %Q{PRISM_VERSION_MINOR #{ENV["PRISM_MINOR"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION_PATCH \d+/, %Q{PRISM_VERSION_PATCH #{ENV["PRISM_PATCH"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION ".+?"/, %Q{PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' include/prism/version.h
+ruby -pi -e 'gsub(/"version": ".+?"/, %Q{"version": "#{ENV["PRISM_VERSION"]}"})' javascript/package.json
+ruby -pi -e 'gsub(/lossy\(\), ".+?"/, %Q{lossy(), "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/tests/utils_tests.rs
+ruby -pi -e 'gsub(/assertEquals\(".+?", prism.version\(\)\)/, %Q{assertEquals("#{ENV["PRISM_VERSION"]}", prism.version())})' java/wasm/src/test/java/org/jruby/parser/prism/WASMTest.java
+ruby -pi -e 'gsub(/\d+, "prism major/, %Q{#{ENV["PRISM_MAJOR"]}, "prism major})' templates/java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism minor/, %Q{#{ENV["PRISM_MINOR"]}, "prism minor})' templates/java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism patch/, %Q{#{ENV["PRISM_PATCH"]}, "prism patch})' templates/java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java.erb
+ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/Cargo.toml
+ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
+ruby -pi -e 'gsub(/^ruby-prism-sys = \{ version = ".+?"/, %Q{ruby-prism-sys = \{ version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
+```
+
+* Update the `Gemfile.lock` file:
+
+```sh
+chruby ruby-4.0.0-dev
+bundle install
+```
+
+* Update the version-specific lockfiles:
+
+```sh
+for VERSION in "2.7" "3.0" "3.1" "3.2" "3.3" "3.4" "4.0"; do docker run -it --rm -v "$PWD":/usr/src/app -w /usr/src/app -e BUNDLE_GEMFILE="gemfiles/$VERSION/Gemfile" "ruby:$VERSION" bundle update; done
+chruby ruby-4.1.0-dev && BUNDLE_GEMFILE=gemfiles/4.1/Gemfile bundle install
+```
+
+* Update the cargo lockfiles:
+
+```sh
+bundle exec rake cargo:build
+```
+
+* Regenerate `rust/ruby-prism-sys/src/bindings.rs` with `bundle exec rake cargo:bindings`. CI fails if the committed file is stale.
+
+* Commit all of the updated files:
+
+```sh
+git commit -am "Bump to v$PRISM_VERSION"
+```
+
+* Push up the changes:
+
+```sh
+git push
+```
+
+## Publishing
+
+### Automatic (Github Actions)
+
+The [`publish-gem.yml`](../.github/workflows/publish-gem.yml) workflow handles publishing. It triggers either on a `v*` tag push or via manual `workflow_dispatch`. In both cases it:
+
+- Builds the source gem and all native gems (calling the reusable [`build-gems.yml`](../.github/workflows/build-gems.yml) workflow with `--release`).
+- Pushes every gem to RubyGems.org using OIDC trusted publishing (no API key required).
+- Creates (or updates) a GitHub release for the tag, attaches the `.gem` files and a `CHECKSUMS.txt`, and auto-generates release notes.
+
+The `push` job is gated on `github.repository == 'ruby/prism'` and runs in the `rubygems.org` GitHub environment, which is the trust anchor for OIDC publishing to RubyGems.org. Pushes that already exist on RubyGems.org are reported as warnings rather than failing the run, so re-runs after a partial failure are safe.
+
+#### 1. Trigger the release pipeline
+
+After the preparation commit is on `main`, tag and push:
+
+```sh
+git tag "v${PRISM_VERSION}"
+git push origin "v${PRISM_VERSION}"
+```
+
+The `v*` tag push triggers `publish-gem.yml` automatically.
+
+If you need to re-trigger the workflow against an existing tag — for example, if the tag-push run failed partway through and you've fixed the cause — go to the Actions tab and use "Run workflow" on `Publish gem to rubygems.org`, passing the tag (e.g. `v1.10.0`) as the `version_tag` input. The workflow will check out that tag, rebuild, and push.
+
+#### 2. Edit the release notes
+
+The workflow creates the GitHub release with auto-generated notes. Edit it to paste in the relevant section of `CHANGELOG.md`.
+
+### Manually
+
+If for whatever reason (like an emergency), you need to release a version and the Github Actions pipelines aren't available, here's how you can build the full set of native gems.
+
+#### 1. Build gems
+
+```sh
+bin/build-gems
+```
+
+This script will:
+- Run `bundle update` and `bundle package` to vendor dependencies
+- Run a safety check (`compile` and `test`)
+- Build all gems via `rake gem:all` (source gem + native gems for all platforms using rake-compiler-dock)
+- Place all gems in `./gems/`
+- Verify all built gems with `test/prism/packaging/test-gem-file-contents`
+- Print SHA256 checksums for inclusion in release notes
+
+#### 2. Push gems
+
+Push the native gems first, then the source gem last (so that when users see the new version, native gems are already available):
+
+```sh
+# push native gems first
+for gem in gems/prism-*-*.gem ; do
+  gem push "$gem"
+done
+
+# push source gem last
+gem push gems/prism-${PRISM_VERSION}.gem
+```
+
+#### 3. Push tag
+
+Push a new tag to the GitHub repository, following the `vX.Y.Z` format:
+
+```sh
+git tag "v${PRISM_VERSION}"
+git push origin "v${PRISM_VERSION}"
+```
+
+#### 4. Create a release
+
+Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file, including the SHA256 checksums from the build output.

data/docs/relocation.md

@@ -0,0 +1,32 @@
+# Relocation
+
+Prism parses deterministically for the same input. This provides a nice property that is exposed through the `#node_id` API on nodes. Effectively this means that for the same input, these values will remain consistent every time the source is parsed. This means we can reparse the source same with a `#node_id` value and find the exact same node again.
+
+The `Relocation` module provides an API around this property. It allows you to "save" nodes and locations using a minimal amount of memory (just the node_id and a field identifier) and then reify them later. This minimizes the amount of memory you need to allocate to store this information because it does not keep around a pointer to the source string.
+
+## Getting started
+
+To get started with the `Relocation` module, you would first instantiate a `Repository` object. You do this through a DSL that chains method calls for configuration. For example, if for every entry in the repository you want to store the start and end lines, and the start and end code unit columns for in UTF-16, you would:
+
+```ruby
+repository = Prism::Relocation.filepath("path/to/file").lines.code_unit_columns(Encoding::UTF_16)
+```
+
+Now that you have the repository, you can pass it into any of the `save*` APIs on nodes or locations to create entries in the repository that will be lazily reified.
+
+```ruby
+# assume that node is a Prism::ClassNode object
+entry = node.constant_path.save(repository)
+```
+
+Now that you have the entry object, you do not need to keep around a reference to the repository, it will be cleaned up on its own when the last entry is reified. Now, whenever you need to, you may call the associated field methods on the entry object, as in:
+
+```ruby
+entry.start_line
+entry.end_line
+
+entry.start_code_units_column
+entry.end_code_units_column
+```
+
+Note that if you had configured other fields to be saved, you would be able to access them as well. The first time one of these fields is accessed, the repository will reify every entry it knows about and then clean itself up. In this way, you can effectively treat them as if you had kept around lightweight versions of `Prism::Node` or `Prism::Location` objects.

data/docs/ripper_translation.md

@@ -0,0 +1,63 @@
+# Ripper translation
+
+Prism provides the ability to mirror the `Ripper` standard library. It is available under `Prism::Translation::Ripper`. You can use the entire public API, and also some undocumented features that are commonly used.
+
+Briefly, `Ripper` is a streaming parser that allows you to construct your own syntax tree. As an example:
+
+```ruby
+class ArithmeticRipper < Prism::Translation::Ripper
+  def on_binary(left, operator, right)
+    left.public_send(operator, right)
+  end
+
+  def on_int(value)
+    value.to_i
+  end
+
+  def on_program(stmts)
+    stmts
+  end
+
+  def on_stmts_new
+    []
+  end
+
+  def on_stmts_add(stmts, stmt)
+    stmts << stmt
+    stmts
+  end
+end
+
+ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
+```
+
+The exact names of the `on_*` methods are listed in the `Ripper` source.
+
+You can can also automatically use the ripper translation in places that don't explicitly use the translation layer by doing the following:
+
+```ruby
+# Will redirect access of the `Ripper` constant to `Prism::Translation::Ripper`.
+require "prism/translation/ripper/shim"
+```
+
+## Background
+
+It is helpful to understand the differences between the `Ripper` library and the `Prism` library. Both libraries perform parsing and provide you with APIs to manipulate and understand the resulting syntax tree. However, there are a few key differences.
+
+### Design
+
+`Ripper` is a streaming parser. This means as it is parsing Ruby code, it dispatches events back to the consumer. This allows quite a bit of flexibility. You can use it to build your own syntax tree or to find specific patterns in the code. `Prism` on the other hand returns to you the completed syntax tree _before_ it allows you to manipulate it. This means the tree that you get back is the only representation that can be generated by the parser _at parse time_ (but of course can be manipulated later).
+
+### Fields
+
+We use the term "field" to mean a piece of information on a syntax tree node. `Ripper` provides the minimal number of fields to accurately represent the syntax tree for the purposes of compilation/interpretation. For example, in the callbacks for nodes that are based on keywords (`class`, `module`, `for`, `while`, etc.) you are not given the keyword itself, you need to attach it on your own. In other cases, tokens are not necessarily dispatched at all, meaning you need to find them yourself. `Prism` provides the opposite: the maximum number of fields on nodes is provided. As a tradeoff, this requires more memory, but this is chosen to make it easier on consumers.
+
+### Maintainability
+
+The `Ripper` interface is not guaranteed in any way, and tends to change between patch versions of CRuby. This is largely due to the fact that `Ripper` is a by-product of the generated parser, as opposed to its own parser. As an example, in the expression `foo::bar = baz`, there are three different represents possible for the call operator, including:
+
+* `:"::"` - Ruby 1.9 to Ruby 3.1.4
+* `73` - Ruby 3.1.5 to Ruby 3.1.6
+* `[:@op, "::", [lineno, column]]` - Ruby 3.2.0 and later
+
+The `Prism` interface is guaranteed going forward to be the consistent, and the official Ruby syntax tree interface. This means you can rely on this interface without having to worry about individual changes between Ruby versions. It also is a gem, which means it is versioned based on the gem version, as opposed to being versioned based on the Ruby version. Finally, you can use `Prism` to parse multiple versions of Ruby, whereas `Ripper` is tied to the Ruby version it is running on.