jruby-prism-parser 0.23.0.pre.SNAPSHOT-java

data/docs/local_variable_depth.md

@@ -0,0 +1,229 @@
+# Local variable depth
+
+One feature of Prism is that it resolves local variables as it parses. It's necessary to do this because of ambiguities in the grammar. For example, consider the following code:
+
+```ruby
+foo / bar#/
+```
+
+If `foo` is a local variable, this is a call to `/` with `bar` as an argument, followed by a comment. If it's not a local variable, this is a method call to `foo` with a regular expression argument.
+
+"Depth" refers to the number of visible scopes that Prism has to go up to find the declaration of a local variable.
+Note that this follows the same scoping rules as Ruby, so a local variable is only visible in the scope it is declared in and in blocks nested in that scope.
+The rules for calculating the depth are very important to understand because they may differ from individual Ruby implementations since they are not specified by the language.
+
+Prism uses the minimum number of scopes, i.e., it only creates scopes when necessary semantically, in other words when there must be distinct scopes (which can be observed through `binding.local_variables`).
+That are no "transparent/invisible" scopes in Prism.
+Some Ruby implementations use those for some language constructs and need to adjust by maintaining a depth offset.
+
+Below are the places where a local variable can be written/targeted, along with how the depth is calculated at that point.
+
+## General
+
+In the course of general Ruby code when reading a local variable, the depth is equal to the number of scopes to go up to find the declaration of that variable. For example:
+
+```ruby
+foo = 1
+bar = 2
+baz = 3
+
+foo # depth 0
+tap { bar } # depth 1
+tap { tap { baz } } # depth 2
+```
+
+This also includes writing to a local variable, which could be writing to a local variable that is already declared. For example:
+
+```ruby
+foo = 1
+bar = 2
+
+foo = 3 # depth 0
+tap { bar = 4 } # depth 1
+```
+
+This includes multiple assignment, where the same principle applies. For example:
+
+```ruby
+foo = 1
+bar = 2
+
+foo, bar = 3, 4 # depth 0
+tap { foo, bar = 5, 6 } # depth 1
+```
+
+## `for` loops
+
+`for` loops in Ruby break down to calls to `.each` with a block.
+However in that case local variable reads and writes within the block will be in the same scope as the scope surrounding the `for` and not in a deeper/separate scope (surprising, but this is Ruby semantics).
+For example:
+
+```ruby
+foo = 1
+
+for e in baz
+  foo # depth 0
+  bar = 2 # depth 0
+end
+
+p bar # depth 0, prints 2
+```
+
+The local variable(s) used for the index of the `for` are also at the same depth (as variables inside and outside the `for`):
+
+```ruby
+for e in [1, 2] # depth 0
+  e # depth 0
+end
+
+p e # depth 0, prints 2
+```
+
+## Pattern matching captures
+
+You can target a local variable in a pattern matching expression using capture syntax. Using this syntax, you can target local variables in the current scope or in visible parent scopes. For example:
+
+```ruby
+42 => bar # depth 0
+```
+
+The example above writes to a local variable in the current scope. If the variable is already declared in a higher visible scope, it will be written to that scope instead. For example:
+
+```ruby
+foo = 1
+tap { 42 => foo } # depth 1
+```
+
+## Named capture groups
+
+You can target local variables through named capture groups in regular expressions if they are used on the left-hand side of a `=~` operator. For example:
+
+```ruby
+/(?<foo>\d+)/ =~ "42" # depth 0
+```
+
+This will write to a `foo` local variable. If the variable is already declared in a higher visible scope, it will be written to that scope instead. For example:
+
+```ruby
+foo = 1
+tap { /(?<foo>\d+)/ =~ "42" } # depth 1
+```
+
+## "interpolated once" regular expressions
+
+Regular expressions that interpolate local variables (unrelated to capture group local variables) and have the `o` flag will only interpolate the local variables once for the runtime of the program.
+In CRuby, this is implemented by compiling the regular expression within a nested instruction sequence, which means CRuby thinks the depth is one more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'foo = 1; /#{foo}/o'
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,18)> (catch: false)
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 once                                   block in <main>, <is:0>
+0006 leave
+
+== disasm: #<ISeq:block in <main>@-e:1 (1,9)-(1,18)> (catch: false)
+0000 putobject                              ""                        (   1)
+0002 getlocal_WC_1                          foo@0
+0004 dup
+0005 objtostring                            <calldata!mid:to_s, argc:0, FCALL|ARGS_SIMPLE>
+0007 anytostring
+0008 toregexp                               0, 2
+0011 leave
+```
+
+In this case CRuby fetches the local variable with `getlocal_WC_1` as the second instruction to the "once" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+## `rescue` clauses
+
+In CRuby, `rescue` clauses are implemented as their own instruction sequence, and therefore CRuby thinks the depth is one more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'begin; foo = 1; rescue; foo; end'
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,32)> (catch: true)
+== catch table
+| catch type: rescue st: 0000 ed: 0004 sp: 0000 cont: 0005
+| == disasm: #<ISeq:rescue in <main>@-e:1 (1,16)-(1,28)> (catch: true)
+| local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+| [ 1] $!@0
+| 0000 getlocal_WC_0                          $!@0                      (   1)
+| 0002 putobject                              StandardError
+| 0004 checkmatch                             3
+| 0006 branchunless                           11
+| 0008 getlocal_WC_1                          foo@0[Li]
+| 0010 leave
+| 0011 getlocal_WC_0                          $!@0
+| 0013 throw                                  0
+| catch type: retry  st: 0004 ed: 0005 sp: 0000 cont: 0000
+|------------------------------------------------------------------------
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 dup
+0002 setlocal_WC_0                          foo@0
+0004 nop
+0005 leave
+```
+
+In the catch table, CRuby is reading the `foo` local variable using `getlocal_WC_1` as the fifth instruction to the "rescue" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+Note that this includes the error reference, which can target local variables, as in:
+
+```
+$ ruby --dump=insns -e 'foo = 1; begin; rescue => foo; end'              
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,34)> (catch: true)
+== catch table
+| catch type: rescue st: 0003 ed: 0004 sp: 0000 cont: 0005
+| == disasm: #<ISeq:rescue in <main>@-e:1 (1,16)-(1,30)> (catch: true)
+| local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+| [ 1] $!@0
+| 0000 getlocal_WC_0                          $!@0                      (   1)
+| 0002 putobject                              StandardError
+| 0004 checkmatch                             3
+| 0006 branchunless                           14
+| 0008 getlocal_WC_0                          $!@0
+| 0010 setlocal_WC_1                          foo@0
+| 0012 putnil
+| 0013 leave
+| 0014 getlocal_WC_0                          $!@0
+| 0016 throw                                  0
+| catch type: retry  st: 0004 ed: 0005 sp: 0000 cont: 0003
+|------------------------------------------------------------------------
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 putnil
+0004 nop
+0005 leave
+```
+
+Note that CRuby is writing to the `foo` local variable using the `setlocal_WC_1` instruction as the sixth instruction to the "rescue" instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.
+
+## Post execution blocks
+
+The `END {}` syntax allows executing code when the program exits. In CRuby, this is implemented as two nested instruction sequences. CRuby therefore thinks the depth is two more than prism does. For example:
+
+```
+$ ruby --dump=insns -e 'foo = 1; END { foo }'            
+== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,20)> (catch: false)
+local table (size: 1, argc: 0 [opts: 0, rest: -1, post: 0, block: -1, kw: -1@-1, kwrest: -1])
+[ 1] foo@0
+0000 putobject_INT2FIX_1_                                             (   1)[Li]
+0001 setlocal_WC_0                          foo@0
+0003 once                                   block in <main>, <is:0>
+0006 leave
+
+== disasm: #<ISeq:block in <main>@-e:0 (0,0)-(-1,-1)> (catch: false)
+0000 putspecialobject                       1                         (   1)
+0002 send                                   <calldata!mid:core#set_postexe, argc:0, FCALL>, block in <main>
+0005 leave
+
+== disasm: #<ISeq:block in <main>@-e:1 (1,9)-(1,20)> (catch: false)
+0000 getlocal                               foo@0, 2                  (   1)[LiBc]
+0003 leave                                  [Br]
+```
+
+In the instruction sequence corresponding to the code that gets executed inside the `END` block, CRuby is reading the `foo` local variable using `getlocal` as the second instruction to the `"block in <main>"` instruction sequence. When compiling CRuby, prism therefore will adjust the depth to account for this difference.

data/docs/mapping.md

@@ -0,0 +1,117 @@
+# Mapping
+
+When considering the previous CRuby parser versus prism, this document should be helpful to understand how various concepts are mapped.
+
+## Nodes
+
+The following table shows how the various CRuby nodes are mapped to prism nodes.
+
+| CRuby | prism |
+| --- | --- |
+| `NODE_SCOPE` | |
+| `NODE_BLOCK` | |
+| `NODE_IF` | `PM_IF_NODE` |
+| `NODE_UNLESS` | `PM_UNLESS_NODE` |
+| `NODE_CASE` | `PM_CASE_NODE` |
+| `NODE_CASE2` | `PM_CASE_NODE` (with a null predicate) |
+| `NODE_CASE3` | |
+| `NODE_WHEN` | `PM_WHEN_NODE` |
+| `NODE_IN` | `PM_IN_NODE` |
+| `NODE_WHILE` | `PM_WHILE_NODE` |
+| `NODE_UNTIL` | `PM_UNTIL_NODE` |
+| `NODE_ITER` | `PM_CALL_NODE` (with a non-null block) |
+| `NODE_FOR` | `PM_FOR_NODE` |
+| `NODE_FOR_MASGN` | `PM_FOR_NODE` (with a multi-write node as the index) |
+| `NODE_BREAK` | `PM_BREAK_NODE` |
+| `NODE_NEXT` | `PM_NEXT_NODE` |
+| `NODE_REDO` | `PM_REDO_NODE` |
+| `NODE_RETRY` | `PM_RETRY_NODE` |
+| `NODE_BEGIN` | `PM_BEGIN_NODE` |
+| `NODE_RESCUE` | `PM_RESCUE_NODE` |
+| `NODE_RESBODY` | |
+| `NODE_ENSURE` | `PM_ENSURE_NODE` |
+| `NODE_AND` | `PM_AND_NODE` |
+| `NODE_OR` | `PM_OR_NODE` |
+| `NODE_MASGN` | `PM_MULTI_WRITE_NODE` |
+| `NODE_LASGN` | `PM_LOCAL_VARIABLE_WRITE_NODE` |
+| `NODE_DASGN` | `PM_LOCAL_VARIABLE_WRITE_NODE` |
+| `NODE_GASGN` | `PM_GLOBAL_VARIABLE_WRITE_NODE` |
+| `NODE_IASGN` | `PM_INSTANCE_VARIABLE_WRITE_NODE` |
+| `NODE_CDECL` | `PM_CONSTANT_PATH_WRITE_NODE` |
+| `NODE_CVASGN` | `PM_CLASS_VARIABLE_WRITE_NODE` |
+| `NODE_OP_ASGN1` | |
+| `NODE_OP_ASGN2` | |
+| `NODE_OP_ASGN_AND` | `PM_OPERATOR_AND_ASSIGNMENT_NODE` |
+| `NODE_OP_ASGN_OR` | `PM_OPERATOR_OR_ASSIGNMENT_NODE` |
+| `NODE_OP_CDECL` | |
+| `NODE_CALL` | `PM_CALL_NODE` |
+| `NODE_OPCALL` | `PM_CALL_NODE` (with an operator as the method) |
+| `NODE_FCALL` | `PM_CALL_NODE` (with a null receiver and parentheses) |
+| `NODE_VCALL` | `PM_CALL_NODE` (with a null receiver and parentheses or arguments) |
+| `NODE_QCALL` | `PM_CALL_NODE` (with a &. operator) |
+| `NODE_SUPER` | `PM_SUPER_NODE` |
+| `NODE_ZSUPER` | `PM_FORWARDING_SUPER_NODE` |
+| `NODE_LIST` | `PM_ARRAY_NODE` |
+| `NODE_ZLIST` | `PM_ARRAY_NODE` (with no child elements) |
+| `NODE_VALUES` | `PM_ARGUMENTS_NODE` |
+| `NODE_HASH` | `PM_HASH_NODE` |
+| `NODE_RETURN` | `PM_RETURN_NODE` |
+| `NODE_YIELD` | `PM_YIELD_NODE` |
+| `NODE_LVAR` | `PM_LOCAL_VARIABLE_READ_NODE` |
+| `NODE_DVAR` | `PM_LOCAL_VARIABLE_READ_NODE` |
+| `NODE_GVAR` | `PM_GLOBAL_VARIABLE_READ_NODE` |
+| `NODE_IVAR` | `PM_INSTANCE_VARIABLE_READ_NODE` |
+| `NODE_CONST` | `PM_CONSTANT_PATH_READ_NODE` |
+| `NODE_CVAR` | `PM_CLASS_VARIABLE_READ_NODE` |
+| `NODE_NTH_REF` | `PM_NUMBERED_REFERENCE_READ_NODE` |
+| `NODE_BACK_REF` | `PM_BACK_REFERENCE_READ_NODE` |
+| `NODE_MATCH` | |
+| `NODE_MATCH2` | `PM_CALL_NODE` (with regular expression as receiver) |
+| `NODE_MATCH3` | `PM_CALL_NODE` (with regular expression as only argument) |
+| `NODE_LIT` | |
+| `NODE_STR` | `PM_STRING_NODE` |
+| `NODE_DSTR` | `PM_INTERPOLATED_STRING_NODE` |
+| `NODE_XSTR` | `PM_X_STRING_NODE` |
+| `NODE_DXSTR` | `PM_INTERPOLATED_X_STRING_NODE` |
+| `NODE_EVSTR` | `PM_STRING_INTERPOLATED_NODE` |
+| `NODE_DREGX` | `PM_INTERPOLATED_REGULAR_EXPRESSION_NODE` |
+| `NODE_ONCE` | |
+| `NODE_ARGS` | `PM_PARAMETERS_NODE` |
+| `NODE_ARGS_AUX` | |
+| `NODE_OPT_ARG` | `PM_OPTIONAL_PARAMETER_NODE` |
+| `NODE_KW_ARG` | `PM_KEYWORD_PARAMETER_NODE` |
+| `NODE_POSTARG` | `PM_REQUIRED_PARAMETER_NODE` |
+| `NODE_ARGSCAT` | |
+| `NODE_ARGSPUSH` | |
+| `NODE_SPLAT` | `PM_SPLAT_NODE` |
+| `NODE_BLOCK_PASS` | `PM_BLOCK_ARGUMENT_NODE` |
+| `NODE_DEFN` | `PM_DEF_NODE` (with a null receiver) |
+| `NODE_DEFS` | `PM_DEF_NODE` (with a non-null receiver) |
+| `NODE_ALIAS` | `PM_ALIAS_NODE` |
+| `NODE_VALIAS` | `PM_ALIAS_NODE` (with a global variable first argument) |
+| `NODE_UNDEF` | `PM_UNDEF_NODE` |
+| `NODE_CLASS` | `PM_CLASS_NODE` |
+| `NODE_MODULE` | `PM_MODULE_NODE` |
+| `NODE_SCLASS` | `PM_S_CLASS_NODE` |
+| `NODE_COLON2` | `PM_CONSTANT_PATH_NODE` |
+| `NODE_COLON3` | `PM_CONSTANT_PATH_NODE` (with a null receiver) |
+| `NODE_DOT2` | `PM_RANGE_NODE` (with a .. operator) |
+| `NODE_DOT3` | `PM_RANGE_NODE` (with a ... operator) |
+| `NODE_FLIP2` | `PM_RANGE_NODE` (with a .. operator) |
+| `NODE_FLIP3` | `PM_RANGE_NODE` (with a ... operator) |
+| `NODE_SELF` | `PM_SELF_NODE` |
+| `NODE_NIL` | `PM_NIL_NODE` |
+| `NODE_TRUE` | `PM_TRUE_NODE` |
+| `NODE_FALSE` | `PM_FALSE_NODE` |
+| `NODE_ERRINFO` | |
+| `NODE_DEFINED` | `PM_DEFINED_NODE` |
+| `NODE_POSTEXE` | `PM_POST_EXECUTION_NODE` |
+| `NODE_DSYM` | `PM_INTERPOLATED_SYMBOL_NODE` |
+| `NODE_ATTRASGN` | `PM_CALL_NODE` (with a message that ends with =) |
+| `NODE_LAMBDA` | `PM_LAMBDA_NODE` |
+| `NODE_ARYPTN` | `PM_ARRAY_PATTERN_NODE` |
+| `NODE_HSHPTN` | `PM_HASH_PATTERN_NODE` |
+| `NODE_FNDPTN` | `PM_FIND_PATTERN_NODE` |
+| `NODE_ERROR` | `PM_MISSING_NODE` |
+| `NODE_LAST` | |
+```

data/docs/parser_translation.md

@@ -0,0 +1,34 @@
+# parser translation
+
+Prism ships with the ability to translate its syntax tree into the syntax tree used by the [whitequark/parser](https://github.com/whitequark/parser) gem. This allows you to use tools built on top of the `parser` gem with the `prism` parser.
+
+## Usage
+
+The `parser` gem provides multiple parsers to support different versions of the Ruby grammar. This includes all of the Ruby versions going back to 1.8, as well as third-party parsers like MacRuby and RubyMotion. The `prism` gem provides another parser that uses the `prism` parser to build the syntax tree.
+
+You can use the `prism` parser like you would any other. After requiring the parser, you should be able to call any of the regular `Parser::Base` APIs that you would normally use.
+
+```ruby
+require "prism"
+
+Prism::Translation::Parser.parse_file("path/to/file.rb")
+```
+
+### RuboCop
+
+To run RuboCop using the `prism` gem as the parser, you will need to require the `prism/translation/parser/rubocop` file. This file injects `prism` into the known options for both `rubocop` and `rubocop-ast`, such that you can specify it in your `.rubocop.yml` file. Unfortunately `rubocop` doesn't support any direct way to do this, so we have to get a bit hacky.
+
+First, set the `TargetRubyVersion` in your RuboCop configuration file to `80_82_73_83_77.33`. This is the version of Ruby that `prism` reports itself as. (The leading numbers are the ASCII values for `PRISM`.)
+
+```yaml
+AllCops:
+  TargetRubyVersion: 80_82_73_83_77.33
+```
+
+Now when you run `rubocop` you will need to require the `prism/translation/parser/rubocop` file before executing so that it can inject the `prism` parser into the known options.
+
+```
+bundle exec ruby -rprism/translation/parser/rubocop $(bundle exec which rubocop)
+```
+
+This should run RuboCop using the `prism` parser.

data/docs/parsing_rules.md

@@ -0,0 +1,19 @@
+# Rules
+
+This document contains information related to the rules of the parser for Ruby source code.
+
+As an example, in the documentation of many of the fields of nodes, it's mentioned that a field follows the lexing rules for `identifier` or `constant`. This document describes what those rules are.
+
+## Constants
+
+Constants in Ruby begin with an upper-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "upper-case letter" are encoding-dependent.
+
+## Non-void expression
+
+Most expressions in CRuby are non-void. This means the expression they represent resolves to a value. For example, `1 + 2` is a non-void expression, because it resolves to a method call. Even things like `class Foo; end` is a non-void expression, because it returns the last evaluated expression in the body of the class (or `nil`).
+
+Certain nodes, however, are void expressions, and cannot be combined to form larger expressions. For example, `BEGIN {}`, `END {}`, `alias foo bar`, and `undef foo`.
+
+## Identifiers
+
+Identifiers in Ruby begin with an underscore or lower-case letter. This is followed by any number of underscores, alphanumeric, or non-ASCII characters. The definition of "alphanumeric" and "lower-case letter" are encoding-dependent.

data/docs/releasing.md

@@ -0,0 +1,98 @@
+# Releasing
+
+To release a new version of Prism, perform the following steps:
+
+## Preparation
+
+* Update the `CHANGELOG.md` file.
+  * Add a new section for the new version at the top of the file.
+  * Fill in the relevant changes — it may be easiest to click the link for the `Unreleased` heading to find the commits.
+  * Update the links at the bottom of the file.
+* Update the version numbers in the various files that reference them:
+
+```sh
+export PRISM_MAJOR="x"
+export PRISM_MINOR="y"
+export PRISM_PATCH="z"
+export PRISM_VERSION="$PRISM_MAJOR.$PRISM_MINOR.$PRISM_PATCH"
+ruby -pi -e 'gsub(/spec\.version = ".+?"/, %Q{spec.version = "#{ENV["PRISM_VERSION"]}"})' prism.gemspec
+ruby -pi -e 'gsub(/EXPECTED_PRISM_VERSION ".+?"/, %Q{EXPECTED_PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' ext/prism/extension.h
+ruby -pi -e 'gsub(/PRISM_VERSION_MAJOR \d+/, %Q{PRISM_VERSION_MAJOR #{ENV["PRISM_MAJOR"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION_MINOR \d+/, %Q{PRISM_VERSION_MINOR #{ENV["PRISM_MINOR"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION_PATCH \d+/, %Q{PRISM_VERSION_PATCH #{ENV["PRISM_PATCH"]}})' include/prism/version.h
+ruby -pi -e 'gsub(/PRISM_VERSION ".+?"/, %Q{PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' include/prism/version.h
+ruby -pi -e 'gsub(/"version": ".+?"/, %Q{"version": "#{ENV["PRISM_VERSION"]}"})' javascript/package.json
+ruby -pi -e 'gsub(/lossy\(\), ".+?"/, %Q{lossy(), "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/tests/utils_tests.rs
+ruby -pi -e 'gsub(/\d+, "prism major/, %Q{#{ENV["PRISM_MAJOR"]}, "prism major})' templates/java/org/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism minor/, %Q{#{ENV["PRISM_MINOR"]}, "prism minor})' templates/java/org/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism patch/, %Q{#{ENV["PRISM_PATCH"]}, "prism patch})' templates/java/org/prism/Loader.java.erb
+ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/javascript/src/deserialize.js.erb
+ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/lib/prism/serialize.rb.erb
+ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/Cargo.toml
+ruby -pi -e 'gsub(/^version = ".+?"/, %Q{version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
+ruby -pi -e 'gsub(/^ruby-prism-sys = { version = ".+?"/, %Q{ruby-prism-sys = { version = "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism/Cargo.toml
+```
+
+* Update the `Gemfile.lock` file:
+
+```sh
+chruby ruby-3.4.0-dev
+bundle install
+```
+
+* Update the version-specific lockfiles:
+
+```sh
+chruby ruby-2.7.8 && BUNDLE_GEMFILE=gemfiles/2.7/Gemfile bundle install
+chruby ruby-3.0.6 && BUNDLE_GEMFILE=gemfiles/3.0/Gemfile bundle install
+chruby ruby-3.1.4 && BUNDLE_GEMFILE=gemfiles/3.1/Gemfile bundle install
+chruby ruby-3.2.3 && BUNDLE_GEMFILE=gemfiles/3.2/Gemfile bundle install
+chruby ruby-3.3.0 && BUNDLE_GEMFILE=gemfiles/3.3/Gemfile bundle install
+chruby ruby-3.4.0-dev && BUNDLE_GEMFILE=gemfiles/3.4/Gemfile bundle install
+chruby jruby-9.4.5.0 && BUNDLE_GEMFILE=gemfiles/jruby/Gemfile bundle install
+chruby truffleruby-23.1.2 && BUNDLE_GEMFILE=gemfiles/truffleruby/Gemfile bundle install
+chruby ruby-3.4.0-dev
+```
+
+* Update the cargo lockfiles:
+
+```sh
+bundle exec rake cargo:build
+```
+
+* Commit all of the updated files:
+
+```sh
+git commit -am "Bump to v$PRISM_VERSION"
+```
+
+## Publishing
+
+* Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file.
+* Publish the gem to [rubygems.org](rubygems.org). Note that you must have access to the `prism` gem to do this.
+
+```sh
+bundle exec rake release
+```
+
+* Generate the `wasm` artifact (or download it from GitHub actions and put it in `javascript/src/prism.wasm`).
+
+```sh
+make wasm
+```
+
+* Publish the JavaScript package to [npmjs.com](npmjs.com). Note that you must have access to the `@ruby/prism` package to do this.
+
+```sh
+npm publish
+```
+
+* Publish the rust crate to [crates.io](crates.io). Note that you must have access to the `ruby-prism-sys` and `ruby-prism` crates to do this.
+
+```sh
+bundle exec rake cargo:publish:real
+```

data/docs/ripper.md

@@ -0,0 +1,36 @@
+# Ripper
+
+To test the parser, we compare against the output from `Ripper`, both for testing the lexer and testing the parser. The lexer test suite is much more feature complete at the moment.
+
+To lex source code using `prism`, you typically would run `Prism.lex(source)`. If you want to instead get output that `Ripper` would normally produce, you can run `Prism.lex_compat(source)`. This will produce tokens that should be equivalent to `Ripper`.
+
+To parse source code using `prism`, you typically would run `Prism.parse(source)`. If you want to instead using the `Ripper` streaming interface, you can inherit from `Prism::RipperCompat` and override the `on_*` methods. This will produce a syntax tree that should be equivalent to `Ripper`. That would look like:
+
+```ruby
+class ArithmeticRipper < Prism::RipperCompat
+  def on_binary(left, operator, right)
+    left.public_send(operator, right)
+  end
+
+  def on_int(value)
+    value.to_i
+  end
+
+  def on_program(stmts)
+    stmts
+  end
+
+  def on_stmts_new
+    []
+  end
+
+  def on_stmts_add(stmts, stmt)
+    stmts << stmt
+    stmts
+  end
+end
+
+ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
+```
+
+There are also APIs for building trees similar to the s-expression builders in `Ripper`. The method names are the same. These include `Prism::RipperCompat.sexp_raw(source)` and `Prism::RipperCompat.sexp(source)`.

data/docs/ruby_api.md

@@ -0,0 +1,43 @@
+# Ruby API
+
+The `prism` gem provides a Ruby API for accessing the syntax tree.
+
+For the most part, the API for accessing the tree mirrors that found in the [Syntax Tree](https://github.com/ruby-syntax-tree/syntax_tree) project. This means:
+
+* Walking the tree involves creating a visitor and passing it to the `#accept` method on any node in the tree
+* Nodes in the tree respond to named methods for accessing their children as well as `#child_nodes`
+* Nodes respond to the pattern matching interfaces `#deconstruct` and `#deconstruct_keys`
+
+Every entry in `config.yml` will generate a Ruby class as well as the code that builds the nodes themselves.
+Creating a syntax tree involves calling one of the class methods on the `Prism` module.
+The full API is documented below.
+
+## API
+
+* `Prism.dump(source)` - parse the syntax tree corresponding to the given source string, and serialize it to a string
+* `Prism.dump_file(filepath)` - parse the syntax tree corresponding to the given source file and serialize it to a string
+* `Prism.lex(source)` - parse the tokens corresponding to the given source string and return them as an array within a parse result
+* `Prism.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
+* `Prism.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
+* `Prism.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
+* `Prism.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
+* `Prism.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
+* `Prism.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree
+* `Prism.parse_comments(source)` - parse the comments corresponding to the given source string and return them
+* `Prism.parse_file_comments(source)` - parse the comments corresponding to the given source file and return them
+* `Prism.parse_success?(source)` - parse the syntax tree corresponding to the given source string and return true if it was parsed without errors
+* `Prism.parse_file_success?(filepath)` - parse the syntax tree corresponding to the given source file and return true if it was parsed without errors
+
+## Nodes
+
+Once you have nodes in hand coming out of a parse result, there are a number of common APIs that are available on each instance. They are:
+
+* `#accept(visitor)` - a method that will immediately call `visit_*` to specialize for the node type
+* `#child_nodes` - a positional array of the child nodes of the node, with `nil` values for any missing children
+* `#compact_child_nodes` - a positional array of the child nodes of the node with no `nil` values
+* `#copy(**keys)` - a method that allows creating a shallow copy of the node with the given keys overridden
+* `#deconstruct`/`#deconstruct_keys(keys)` - the pattern matching interface for nodes
+* `#inspect` - a string representation that looks like the syntax tree of the node
+* `#location` - a `Location` object that describes the location of the node in the source file
+* `#to_dot` - convert the node's syntax tree into graphviz dot notation
+* `#type` - a symbol that represents the type of the node, useful for quick comparisons

data/docs/ruby_parser_translation.md

@@ -0,0 +1,19 @@
+# ruby_parser translation
+
+Prism ships with the ability to translate its syntax tree into the syntax tree used by the [seattlerb/ruby_parser](https://github.com/seattlerb/ruby_parser) gem. This allows you to use tools built on top of the `ruby_parser` gem with the `prism` parser.
+
+## Usage
+
+You can call the `parse` and `parse_file` methods on the `Prism::Translation::RubyParser` module:
+
+```ruby
+filepath = "path/to/file.rb"
+Prism::Translation::RubyParser.parse_file(filepath)
+```
+
+This will return to you `Sexp` objects that mirror the result of calling `RubyParser` methods, as in:
+
+```ruby
+filepath = "path/to/file.rb"
+RubyParser.new.parse(File.read(filepath), filepath)
+```