prism 1.9.0 → 2.0.0.rc2

data/docs/build_system.md

@@ -87,7 +87,7 @@ If you need to use memory allocation functions implemented outside of the standa
 * Additionally, include `-I [path/to/custom_allocator]` where your `prism_xallocator.h` is located
 * Link the implementation of `prism_xallocator.c` that contains functions declared in `prism_xallocator.h`
 
-For further clarity, refer to `include/prism/defines.h`.
+For further clarity, refer to `include/prism/internal/allocator.h`.
 
 ### Building prism from source as a C library
 
@@ -97,6 +97,7 @@ If you want to build prism as a shared library and link against it, you should c
 
 * `-fPIC -shared` - Compile as a shared library
 * `-DPRISM_EXPORT_SYMBOLS` - Export the symbols (by default nothing is exported)
+* `-Wl,-Bsymbolic` (ELF platforms) - Bind the library's internal calls to its own exported functions at link time, so that they cannot be redirected to another copy of libprism loaded in the same process
 
 #### Flags
 
@@ -113,7 +114,6 @@ MAKEFLAGS="-j10" bundle exec rake compile
 * `PRISM_ENCODING_EXCLUDE_FULL` - Will cause the library to exclude the full encoding API, and only include the minimal number of encodings to support parsing Ruby code without encoding comments. By default this is off.
 * `PRISM_EXPORT_SYMBOLS` - Will cause the shared library to export symbols. By default this is off.
 * `PRISM_EXCLUDE_JSON` - Will cause the library to exclude the JSON API. By default this is off.
-* `PRISM_EXCLUDE_PACK` - Will cause the library to exclude the pack API. By default this is off.
 * `PRISM_EXCLUDE_PRETTYPRINT` - Will cause the library to exclude the prettyprint API. By default this is off.
 * `PRISM_EXCLUDE_SERIALIZATION` - Will cause the library to exclude the serialization API. By default this is off.
 * `PRISM_XALLOCATOR` - Will cause the library to use the custom memory allocator. By default this is off.

data/docs/configuration.md

@@ -5,11 +5,12 @@ A lot of code in prism's repository is templated from a single configuration fil
 * `ext/prism/api_node.c` - for defining how to build Ruby objects for the nodes out of C structs
 * `include/prism/ast.h` - for defining the C structs that represent the nodes
 * `include/prism/diagnostic.h` - for defining the diagnostics
+* `include/prism/node_new.h` - for defining the functions that create the nodes in C
 * `javascript/src/deserialize.js` - for defining how to deserialize the nodes in JavaScript
 * `javascript/src/nodes.js` - for defining the nodes in JavaScript
-* `java/org/prism/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
-* `java/org/prism/Loader.java` - for defining how to deserialize the nodes in Java
-* `java/org/prism/Nodes.java` - for defining the nodes in Java
+* `java/api/src/main/java-templates/org/ruby_lang/prism/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
+* `java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java` - for defining how to deserialize the nodes in Java
+* `java/api/src/main/java-templates/org/ruby_lang/prism/Nodes.java` - for defining the nodes in Java
 * `lib/prism/compiler.rb` - for defining the compiler for the nodes in Ruby
 * `lib/prism/dispatcher.rb` - for defining the dispatch visitors for the nodes in Ruby
 * `lib/prism/dot_visitor.rb` - for defining the dot visitor for the nodes in Ruby
@@ -61,7 +62,7 @@ The available values for `type` are:
 * `constant` - A field that is an integer that represents an index in the constant pool. This is a `pm_constant_id_t` in C.
 * `constant[]` - A field that is an array of constants. This is a `pm_constant_id_list_t` in C.
 * `location` - A field that is a location. This is a `pm_location_t` in C.
-* `location?` - A field that is a location that is optionally present. This is a `pm_location_t` in C, but if the value is not present then the `start` and `end` fields will be `NULL`.
+* `location?` - A field that is a location that is optionally present. This is a `pm_location_t` in C, but if the value is not present then the `length` field will be `0`.
 * `uint8` - A field that is an 8-bit unsigned integer. This is a `uint8_t` in C.
 * `uint32` - A field that is a 32-bit unsigned integer. This is a `uint32_t` in C.
 

data/docs/encoding.md

@@ -107,7 +107,7 @@ For each of these encodings, prism provides functions for checking if the subseq
 
 ## Getting notified when the encoding changes
 
-You may want to get notified when the encoding changes based on the result of parsing an encoding comment. We use this internally for our `lex` function in order to provide the correct encodings for the tokens that are returned. For that you can register a callback with `pm_parser_register_encoding_changed_callback`. The callback will be called with a pointer to the parser. The encoding can be accessed through `parser->encoding`.
+You may want to get notified when the encoding changes based on the result of parsing an encoding comment. We use this internally for our `lex` function in order to provide the correct encodings for the tokens that are returned. For that you can register a callback with `pm_parser_encoding_changed_callback_set`. The callback will be called with a pointer to the parser. The encoding can be accessed through `parser->encoding`.
 
 ```c
 // When the encoding that is being used to parse the source is changed by prism,
@@ -117,5 +117,5 @@ typedef void (*pm_encoding_changed_callback_t)(pm_parser_t *parser);
 // Register a callback that will be called whenever prism changes the encoding
 // it is using to parse based on the magic comment.
 PRISM_EXPORTED_FUNCTION void
-pm_parser_register_encoding_changed_callback(pm_parser_t *parser, pm_encoding_changed_callback_t callback);
+pm_parser_encoding_changed_callback_set(pm_parser_t *parser, pm_encoding_changed_callback_t callback);
 ```

data/docs/fuzzing.md

@@ -5,8 +5,7 @@ We use fuzzing to test the various entrypoints to the library. The fuzzer we use
 ```
 fuzz
 ├── corpus
-│   ├── parse             fuzzing corpus for parsing (a symlink to our fixtures)
-│   └── regexp            fuzzing corpus for regexp
+│   └── parse             fuzzing corpus for parsing (a symlink to our fixtures)
 ├── dict                  a AFL++ dictionary containing various tokens
 ├── docker
 │   └── Dockerfile        for building a container with the fuzzer toolchain
@@ -14,8 +13,6 @@ fuzz
 ├── heisenbug.c           entrypoint for reproducing a crash or hang
 ├── parse.c               fuzz handler for parsing
 ├── parse.sh              script to run parsing fuzzer
-├── regexp.c              fuzz handler for regular expression parsing
-├── regexp.sh             script to run regexp fuzzer
 └── tools
     ├── backtrace.sh      generates backtrace files for a crash directory
     └── minimize.sh       generates minimized crash or hang files
@@ -23,16 +20,14 @@ fuzz
 
 ## Usage
 
-There are currently three fuzzing targets
+There is currently one fuzz target:
 
 - `pm_serialize_parse` (parse)
-- `pm_regexp_parse` (regexp)
 
-Respectively, fuzzing can be performed with
+Fuzzing can be performed with
 
 ```
 make fuzz-run-parse
-make fuzz-run-regexp
 ```
 
 To end a fuzzing job, interrupt with CTRL+C. To enter a container with the fuzzing toolchain and debug utilities, run
@@ -43,8 +38,6 @@ make fuzz-debug
 
 # Out-of-bounds reads
 
-Currently, encoding functionality implementing the `pm_encoding_t` interface can read outside of inputs. For the time being, ASAN instrumentation is disabled for functions from src/enc. See `fuzz/asan.ignore`.
-
 To disable ASAN read instrumentation globally, use the `FUZZ_FLAGS` environment variable e.g.
 
 ```
@@ -55,7 +48,7 @@ Note, that this may make reproducing bugs difficult as they may depend on memory
 
 ```
 make fuzz-debug # enter the docker container with build tools
-make build/fuzz.heisenbug.parse # or .regexp
+make build/fuzz.heisenbug.parse
 ./build/fuzz.heisenbug.parse path-to-problem-input
 ```
 

data/docs/javascript.md

@@ -16,10 +16,25 @@ Then import the package:
 import { loadPrism } from "@ruby/prism";
 ```
 
-Then call the load function to get a parse function:
+Then call the load function to get a parse function. It takes the source as
+bytes, in the encoding the source is written in, because that is what the parser
+reads and what the locations in the result are offsets into. If you are starting
+from a string, encode it first:
 
 ```js
-const parse = await loadPrism();
+const parseArray = await loadPrism();
+
+function parse(source) {
+  return parseArray(new TextEncoder().encode(source));
+}
+```
+
+Reading a file gives you those bytes already, so there is nothing to encode:
+
+```js
+import { readFile } from "node:fs/promises";
+
+const parseResult = parseArray(await readFile("example.rb"));
 ```
 
 ## Browser
@@ -56,7 +71,7 @@ Finally, you can create a function that will parse a string of Ruby code.
 
 ```js
 function parse(source) {
-  return parsePrism(instance.exports, source);
+  return parsePrism(instance.exports, new TextEncoder().encode(source));
 }
 ```
 
@@ -84,7 +99,7 @@ Here's an example of a custom `FooCalls` visitor:
 import { loadPrism, Visitor } from "@ruby/prism"
 
 const parse = await loadPrism();
-const parseResult = parse("foo()");
+const parseResult = parse(new TextEncoder().encode("foo()"));
 
 class FooCalls extends Visitor {
   visitCallNode(node) {

data/docs/mapping.md

@@ -112,6 +112,6 @@ The following table shows how the various CRuby nodes are mapped to prism nodes.
 | `NODE_ARYPTN` | `PM_ARRAY_PATTERN_NODE` |
 | `NODE_HSHPTN` | `PM_HASH_PATTERN_NODE` |
 | `NODE_FNDPTN` | `PM_FIND_PATTERN_NODE` |
-| `NODE_ERROR` | `PM_MISSING_NODE` |
+| `NODE_ERROR` | `PM_ERROR_RECOVERY_NODE` |
 | `NODE_LAST` | |
 ```

data/docs/releasing.md

@@ -23,9 +23,10 @@ ruby -pi -e 'gsub(/PRISM_VERSION_PATCH \d+/, %Q{PRISM_VERSION_PATCH #{ENV["PRISM
 ruby -pi -e 'gsub(/PRISM_VERSION ".+?"/, %Q{PRISM_VERSION "#{ENV["PRISM_VERSION"]}"})' include/prism/version.h
 ruby -pi -e 'gsub(/"version": ".+?"/, %Q{"version": "#{ENV["PRISM_VERSION"]}"})' javascript/package.json
 ruby -pi -e 'gsub(/lossy\(\), ".+?"/, %Q{lossy(), "#{ENV["PRISM_VERSION"]}"})' rust/ruby-prism-sys/tests/utils_tests.rs
-ruby -pi -e 'gsub(/\d+, "prism major/, %Q{#{ENV["PRISM_MAJOR"]}, "prism major})' templates/java/org/prism/Loader.java.erb
-ruby -pi -e 'gsub(/\d+, "prism minor/, %Q{#{ENV["PRISM_MINOR"]}, "prism minor})' templates/java/org/prism/Loader.java.erb
-ruby -pi -e 'gsub(/\d+, "prism patch/, %Q{#{ENV["PRISM_PATCH"]}, "prism patch})' templates/java/org/prism/Loader.java.erb
+ruby -pi -e 'gsub(/assertEquals\(".+?", prism.version\(\)\)/, %Q{assertEquals("#{ENV["PRISM_VERSION"]}", prism.version())})' java/wasm/src/test/java/org/jruby/parser/prism/WASMTest.java
+ruby -pi -e 'gsub(/\d+, "prism major/, %Q{#{ENV["PRISM_MAJOR"]}, "prism major})' templates/java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism minor/, %Q{#{ENV["PRISM_MINOR"]}, "prism minor})' templates/java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java.erb
+ruby -pi -e 'gsub(/\d+, "prism patch/, %Q{#{ENV["PRISM_PATCH"]}, "prism patch})' templates/java/api/src/main/java-templates/org/ruby_lang/prism/Loader.java.erb
 ruby -pi -e 'gsub(/MAJOR_VERSION = \d+/, %Q{MAJOR_VERSION = #{ENV["PRISM_MAJOR"]}})' templates/javascript/src/deserialize.js.erb
 ruby -pi -e 'gsub(/MINOR_VERSION = \d+/, %Q{MINOR_VERSION = #{ENV["PRISM_MINOR"]}})' templates/javascript/src/deserialize.js.erb
 ruby -pi -e 'gsub(/PATCH_VERSION = \d+/, %Q{PATCH_VERSION = #{ENV["PRISM_PATCH"]}})' templates/javascript/src/deserialize.js.erb
@@ -57,6 +58,8 @@ chruby ruby-4.1.0-dev && BUNDLE_GEMFILE=gemfiles/4.1/Gemfile bundle install
 bundle exec rake cargo:build
 ```
 
+* Regenerate `rust/ruby-prism-sys/src/bindings.rs` with `bundle exec rake cargo:bindings`. CI fails if the committed file is stale.
+
 * Commit all of the updated files:
 
 ```sh
@@ -71,5 +74,74 @@ git push
 
 ## Publishing
 
-* Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file.
-* Push a new tag to the GitHub repository, following the `vX.Y.Z` format.
+### Automatic (Github Actions)
+
+The [`publish-gem.yml`](../.github/workflows/publish-gem.yml) workflow handles publishing. It triggers either on a `v*` tag push or via manual `workflow_dispatch`. In both cases it:
+
+- Builds the source gem and all native gems (calling the reusable [`build-gems.yml`](../.github/workflows/build-gems.yml) workflow with `--release`).
+- Pushes every gem to RubyGems.org using OIDC trusted publishing (no API key required).
+- Creates (or updates) a GitHub release for the tag, attaches the `.gem` files and a `CHECKSUMS.txt`, and auto-generates release notes.
+
+The `push` job is gated on `github.repository == 'ruby/prism'` and runs in the `rubygems.org` GitHub environment, which is the trust anchor for OIDC publishing to RubyGems.org. Pushes that already exist on RubyGems.org are reported as warnings rather than failing the run, so re-runs after a partial failure are safe.
+
+#### 1. Trigger the release pipeline
+
+After the preparation commit is on `main`, tag and push:
+
+```sh
+git tag "v${PRISM_VERSION}"
+git push origin "v${PRISM_VERSION}"
+```
+
+The `v*` tag push triggers `publish-gem.yml` automatically.
+
+If you need to re-trigger the workflow against an existing tag — for example, if the tag-push run failed partway through and you've fixed the cause — go to the Actions tab and use "Run workflow" on `Publish gem to rubygems.org`, passing the tag (e.g. `v1.10.0`) as the `version_tag` input. The workflow will check out that tag, rebuild, and push.
+
+#### 2. Edit the release notes
+
+The workflow creates the GitHub release with auto-generated notes. Edit it to paste in the relevant section of `CHANGELOG.md`.
+
+### Manually
+
+If for whatever reason (like an emergency), you need to release a version and the Github Actions pipelines aren't available, here's how you can build the full set of native gems.
+
+#### 1. Build gems
+
+```sh
+bin/build-gems
+```
+
+This script will:
+- Run `bundle update` and `bundle package` to vendor dependencies
+- Run a safety check (`compile` and `test`)
+- Build all gems via `rake gem:all` (source gem + native gems for all platforms using rake-compiler-dock)
+- Place all gems in `./gems/`
+- Verify all built gems with `test/prism/packaging/test-gem-file-contents`
+- Print SHA256 checksums for inclusion in release notes
+
+#### 2. Push gems
+
+Push the native gems first, then the source gem last (so that when users see the new version, native gems are already available):
+
+```sh
+# push native gems first
+for gem in gems/prism-*-*.gem ; do
+  gem push "$gem"
+done
+
+# push source gem last
+gem push gems/prism-${PRISM_VERSION}.gem
+```
+
+#### 3. Push tag
+
+Push a new tag to the GitHub repository, following the `vX.Y.Z` format:
+
+```sh
+git tag "v${PRISM_VERSION}"
+git push origin "v${PRISM_VERSION}"
+```
+
+#### 4. Create a release
+
+Update the GitHub release page with a copy of the latest entry in the `CHANGELOG.md` file, including the SHA256 checksums from the build output.

data/docs/relocation.md

@@ -6,10 +6,10 @@ The `Relocation` module provides an API around this property. It allows you to "
 
 ## Getting started
 
-To get started with the `Relocation` module, you would first instantiate a `Repository` object. You do this through a DSL that chains method calls for configuration. For example, if for every entry in the repository you want to store the start and end lines, the start and end code unit columns for in UTF-16, and the leading comments, you would:
+To get started with the `Relocation` module, you would first instantiate a `Repository` object. You do this through a DSL that chains method calls for configuration. For example, if for every entry in the repository you want to store the start and end lines, and the start and end code unit columns for in UTF-16, you would:
 
 ```ruby
-repository = Prism::Relocation.filepath("path/to/file").lines.code_unit_columns(Encoding::UTF_16).leading_comments
+repository = Prism::Relocation.filepath("path/to/file").lines.code_unit_columns(Encoding::UTF_16)
 ```
 
 Now that you have the repository, you can pass it into any of the `save*` APIs on nodes or locations to create entries in the repository that will be lazily reified.
@@ -27,8 +27,6 @@ entry.end_line
 
 entry.start_code_units_column
 entry.end_code_units_column
-
-entry.leading_comments
 ```
 
 Note that if you had configured other fields to be saved, you would be able to access them as well. The first time one of these fields is accessed, the repository will reify every entry it knows about and then clean itself up. In this way, you can effectively treat them as if you had kept around lightweight versions of `Prism::Node` or `Prism::Location` objects.

data/docs/serialization.md

@@ -44,6 +44,17 @@ The comment type is one of:
 | `1` | comment type |
 | location | the location in the source of this comment |
 
+### comment attachment
+
+A comment attachment lists the comments owned by a single node.
+Only nodes that own comments have an attachment, and there are none unless the `attach_comments` option was set.
+
+| # bytes | field |
+| --- | --- |
+| varuint | the node identifier of the node that owns the comments |
+| varuint | number of comments owned by the node |
+| varuint* | the index of each owned comment in the list of comments, ordered by start offset |
+
 ### magic comment
 
 | # bytes | field |
@@ -95,10 +106,13 @@ The header is structured like the following table:
 | `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
 | string | the encoding name |
 | varsint | the start line |
+| `8` | the checksum of the source up to the end of the `__END__` line in native endian order, see `pm_parser_checksum` |
 | varuint | number of newline offsets |
 | varuint* | newline offsets |
 | varuint | number of comments |
 | comment* | comments |
+| varuint | number of nodes with attached comments |
+| comment attachment* | comment attachments |
 | varuint | number of magic comments |
 | magic comment* | magic comments |
 | location? | the optional location of the `__END__` keyword and its contents |
@@ -106,6 +120,7 @@ The header is structured like the following table:
 | error* | errors |
 | varuint | number of warnings |
 | warning* | warnings |
+| `1` | `1` if the source is continuable (incomplete but could become valid with more input), `0` otherwise |
 | `4` | content pool offset |
 | varuint | content pool size |
 
@@ -126,7 +141,7 @@ Every field on the node is then appended to the serialized string. The fields ca
 * `node` - A field that is a node. This is structured just as like parent node.
 * `node?` - A field that is a node that is optionally present. If the node is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just as like parent node.
 * `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, followed by the child nodes themselves.
-* `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string itself (_without_ a trailing null byte).
+* `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string bytes (_without_ a trailing null byte).
 * `constant` - A variable-length integer that represents an index in the constant pool.
 * `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
 * `integer` - A field that represents an arbitrary-sized integer. The structure is listed above.
@@ -135,23 +150,14 @@ Every field on the node is then appended to the serialized string. The fields ca
 * `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
 * `uint32` - A field that is a 32-bit unsigned integer. This is structured as a variable-length integer.
 
-After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Constants can be either "owned" (in which case their contents are embedded in the serialization) or "shared" (in which case their contents represent a slice of the source string). The most significant bit of the constant indicates whether it is owned or shared.
-
-In the case that it is owned, the constant is structured as follows:
+After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Every constant is embedded in the serialization. Each constant is structured as follows:
 
 | # bytes | field |
 | --- | --- |
 | `4` | the byte offset in the serialization for the contents of the constant |
 | `4` | the byte length in the serialization |
 
-Note that you will need to mask off the most significant bit for the byte offset in the serialization. In the case that it is shared, the constant is structured as follows:
-
-| # bytes | field |
-| --- | --- |
-| `4` | the byte offset in the source string for the contents of the constant |
-| `4` | the byte length in the source string |
-
-After the constant pool, the contents of the owned constants are serialized. This is just a sequence of bytes that represent the contents of the constants. At the end of the serialization, the buffer is null terminated.
+After the constant pool, the contents of the constants are serialized. This is just a sequence of bytes that represent the contents of the constants. At the end of the serialization, the buffer is null terminated.
 
 ## APIs
 
@@ -161,31 +167,23 @@ The relevant APIs and struct definitions are listed below:
 // A pm_buffer_t is a simple memory buffer that stores data in a contiguous
 // block of memory. It is used to store the serialized representation of a
 // prism tree.
-typedef struct {
-  char *value;
-  size_t length;
-  size_t capacity;
-} pm_buffer_t;
-
-// Free the memory associated with the buffer.
-void pm_buffer_free(pm_buffer_t *);
 
 // Parse and serialize the AST represented by the given source to the given
 // buffer.
 void pm_serialize_parse(pm_buffer_t *buffer, const uint8_t *source, size_t length, const char *data);
 ```
 
-Typically you would use a stack-allocated `pm_buffer_t` and call `pm_serialize_parse`, as in:
+Typically you would allocate a `pm_buffer_t` and call `pm_serialize_parse`, as in:
 
 ```c
 void
 serialize(const uint8_t *source, size_t length) {
-  pm_buffer_t buffer = { 0 };
-  pm_serialize_parse(&buffer, source, length, NULL);
+  pm_buffer_t *buffer = pm_buffer_new();
+  pm_serialize_parse(buffer, source, length, NULL);
 
   // Do something with the serialized string.
 
-  pm_buffer_free(&buffer);
+  pm_buffer_free(buffer);
 }
 ```
 

data/docs/testing.md

@@ -14,42 +14,59 @@ These test specific prism implementation details like comments, errors, and regu
 
 Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/prism/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/prism/snapshots/` directory. For example, `test/prism/fixtures/strings.txt` has a corresponding `test/prism/snapshots/strings.txt`.
 
-If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/prism/snapshots/` directory, one will be created so that it exists for the next test run.
+If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/prism/snapshots/` directory, one will be created so that it exists for the next test run. If the snapshot already exists but differs you can run with `UPDATE_SNAPSHOTS=1` to regenerate them.
 
-### Testing against repositories
+When working on a specific example, you can limit tests to a single fixture via `FOCUS=strings.txt`.
 
-To test the parser against a repository, you can run `FILEPATHS='/path/to/repository/**/*.rb' rake lex`. This will run the parser against every file matched by the glob pattern and check its generated tokens against those generated by ripper.
+#### Versioning
 
-## Local testing
-
-As you are working, you will likely want to test your code locally. `test.rb` is ignored by git, so it can be used for local testing. There are also two executables which may help you:
-
-1. **bin/lex** takes a filepath and compares prism's lexed output to Ripper's lexed output. It prints any lexed output that doesn't match. It does some minor transformations to the lexed output in order to compare them, like split prism's heredoc tokens to mirror Ripper's.
-
-```
-$ bin/lex test.rb
-```
+Prism is capable of parsing code with different version specifiers. The test suite reflects this:
+* Test files under `test/prism/snapshots/*.txt` are validated against all supported versions.
+* Test files under `test/prism/snapshots/[MAJOR].[MINOR]/*.txt` are validated against all versions starting from the given version.
+* Test files under `test/prism/snapshots/[MAJOR].[MINOR]-[MAJOR].[MINOR]/*.txt` are validated against the given version range.
 
-If you would like to see the full lexed comparison, and not only the output that doesn't match, you can run with `VERBOSE=1`:
-
-```
-$ VERBOSE=1 bin/lex test.rb
-```
+Should you add logic specific to a new ruby version, validate the new behavior but also that old behavior remains the same.
 
-`bin/lex` can also be used with `-e` and then source code, like this:
+### Error tests
 
-```
-$ bin/lex -e "1 + 2"
-```
+Error tests are similar to snapshot tests and expectations are located under `test/prism/errors/`.
+They contain invalid syntax and rendered errors, similar to how `ruby -c` would show them.
+Error tests are versioned in the same way that snapshot tests are and also support `FOCUS`/`UPDATE_SNAPSHOTS`.
 
-2. **bin/parse** takes a filepath and outputs prism's parsed node structure generated from reading the file.
+## Local testing
 
+As you are working, you will likely want to test your code locally. `test.rb` is ignored by git, so it can be used for local testing. **bin/prism** contains a multitude of options to aid you, depending on which area
+of prism you are working on. `bin/prism --help` prints a list of all available commands.
+
+Commands that take a `[source]` can be called in three different ways:
+1. `bin/prism parse` will use `test.rb` as input.
+1. `bin/prism parse foo.rb` will read from the given path.
+1. `bin/prism parse -e "1 + 2"` will use the given source code.
+
+`bin/prism parse` is aliased as `bin/parse` for for convenience.
+
+### Validating your changes against real code
+
+If you create a change which is not expected to have a large impact on real code, you can use
+`bin/compare` to validate that assumption. It compiles two versions of prism (for example
+your feature branch and main), comparing ast, errors and warnings. Once done, a report is
+printed showcasing in what way files changed, if any (syntax valid -> syntax invalid, changed AST, etc.)
+
+```sh
+$ bin/compare main feature-branch ~/all-of-rubygems`
+...
+0/3157385
+1000/3157385
+...
+3157000/3157385
+Oops:
+intermine-1.05.00/lib/intermine/lists.rb changed from valid(true) to valid(false)
+whistle-0.1.1/lib/resource.rb changed from valid(true) to valid(false)
+Took 2661.5390081949999512 seconds
 ```
-$ bin/parse test.rb
-```
-
-`bin/parse` can also be used with `-e` and then source code, like this:
 
-```
-$ bin/parse -e "1 + 2"
-```
+It is up to you to aquire code samples. Here are some resources you can use for this purpose:
+* https://github.com/jeromedalbert/real-world-ruby-apps
+* https://github.com/eliotsykes/real-world-rails
+* https://github.com/rubygems/rubygems-mirror
+  Requires some work to unpack the downloaded gems. You should also run with `RUBYGEMS_MIRROR_ONLY_LATEST=TRUE`.