prism 2.0.0.rc1-arm64-darwin

data/docs/ruby_api.md

@@ -0,0 +1,45 @@
+# Ruby API
+
+The `prism` gem provides a Ruby API for accessing the syntax tree.
+
+For the most part, the API for accessing the tree mirrors that found in the [Syntax Tree](https://github.com/ruby-syntax-tree/syntax_tree) project. This means:
+
+* Walking the tree involves creating a visitor and passing it to the `#accept` method on any node in the tree
+* Nodes in the tree respond to named methods for accessing their children as well as `#child_nodes`
+* Nodes respond to the pattern matching interfaces `#deconstruct` and `#deconstruct_keys`
+
+Every entry in `config.yml` will generate a Ruby class as well as the code that builds the nodes themselves.
+Creating a syntax tree involves calling one of the class methods on the `Prism` module.
+The full API is documented below.
+
+## API
+
+* `Prism.dump(source)` - parse the syntax tree corresponding to the given source string, and serialize it to a string
+* `Prism.dump_file(filepath)` - parse the syntax tree corresponding to the given source file and serialize it to a string
+* `Prism.lex(source)` - parse the tokens corresponding to the given source string and return them as an array within a parse result
+* `Prism.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
+* `Prism.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
+* `Prism.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
+* `Prism.parse_stream(io)` - parse the syntax tree corresponding to the source that is read out of the given IO object using the `#gets` method and return it within a parse result
+* `Prism.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
+* `Prism.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
+* `Prism.load(source, serialized, freeze = false)` - load the serialized syntax tree using the source as a reference into a syntax tree
+* `Prism.parse_comments(source)` - parse the comments corresponding to the given source string and return them
+* `Prism.parse_file_comments(source)` - parse the comments corresponding to the given source file and return them
+* `Prism.parse_success?(source)` - parse the syntax tree corresponding to the given source string and return true if it was parsed without errors
+* `Prism.parse_file_success?(filepath)` - parse the syntax tree corresponding to the given source file and return true if it was parsed without errors
+
+## Nodes
+
+Once you have nodes in hand coming out of a parse result, there are a number of common APIs that are available on each instance. They are:
+
+* `#accept(visitor)` - a method that will immediately call `visit_*` to specialize for the node type
+* `#child_nodes` - a positional array of the child nodes of the node, with `nil` values for any missing children
+* `#compact_child_nodes` - a positional array of the child nodes of the node with no `nil` values
+* `#each_child_node` - with a block given yields all child nodes, without a block return an enumerator containing all child nodes
+* `#copy(**keys)` - a method that allows creating a shallow copy of the node with the given keys overridden
+* `#deconstruct`/`#deconstruct_keys(keys)` - the pattern matching interface for nodes
+* `#inspect` - a string representation that looks like the syntax tree of the node
+* `#location` - a `Location` object that describes the location of the node in the source file
+* `#to_dot` - convert the node's syntax tree into graphviz dot notation
+* `#type` - a symbol that represents the type of the node, useful for quick comparisons

data/docs/ruby_parser_translation.md

@@ -0,0 +1,19 @@
+# ruby_parser translation
+
+Prism ships with the ability to translate its syntax tree into the syntax tree used by the [seattlerb/ruby_parser](https://github.com/seattlerb/ruby_parser) gem. This allows you to use tools built on top of the `ruby_parser` gem with the `prism` parser.
+
+## Usage
+
+You can call the `parse` and `parse_file` methods on the `Prism::Translation::RubyParser` module:
+
+```ruby
+filepath = "path/to/file.rb"
+Prism::Translation::RubyParser.parse_file(filepath)
+```
+
+This will return to you `Sexp` objects that mirror the result of calling `RubyParser` methods, as in:
+
+```ruby
+filepath = "path/to/file.rb"
+RubyParser.new.parse(File.read(filepath), filepath)
+```

data/docs/serialization.md

@@ -0,0 +1,230 @@
+# Serialization
+
+Prism ships with the ability to serialize a syntax tree to a single string.
+The string can then be deserialized back into a syntax tree using a language other than C.
+This is useful for using the parsing logic in other tools without having to write a parser in that language.
+The syntax tree still requires a copy of the original source, as for the most part it just contains byte offsets into the source string.
+
+## Types
+
+Let us define some simple types for readability.
+
+### varuint
+
+A variable-length unsigned integer with the value fitting in `uint32_t` using between 1 and 5 bytes, using the [LEB128](https://en.wikipedia.org/wiki/LEB128) encoding.
+This drastically cuts down on the size of the serialized string, especially when the source file is large.
+
+### varsint
+
+A variable-length signed integer with the value fitting in `int32_t` using between 1 and 5 bytes, using [ZigZag encoding](https://protobuf.dev/programming-guides/encoding/#signed-ints) into [LEB128].
+
+### string
+
+| # bytes | field |
+| --- | --- |
+| varuint | the length of the string in bytes |
+| ... | the string bytes |
+
+### location
+
+| # bytes | field |
+| --- | --- |
+| varuint | byte offset into the source string where this location begins |
+| varuint | length of the location in bytes in the source string |
+
+### comment
+
+The comment type is one of:
+
+* 0=`INLINE` (`# comment`)
+* 1=`EMBEDDED_DOCUMENT` (`=begin`/`=end`)
+
+| # bytes | field |
+| --- | --- |
+| `1` | comment type |
+| location | the location in the source of this comment |
+
+### comment attachment
+
+A comment attachment lists the comments owned by a single node.
+Only nodes that own comments have an attachment, and there are none unless the `attach_comments` option was set.
+
+| # bytes | field |
+| --- | --- |
+| varuint | the node identifier of the node that owns the comments |
+| varuint | number of comments owned by the node |
+| varuint* | the index of each owned comment in the list of comments, ordered by start offset |
+
+### magic comment
+
+| # bytes | field |
+| --- | --- |
+| location | the location of the key of the magic comment |
+| location | the location of the value of the magic comment |
+
+### error
+
+| # bytes | field |
+| --- | --- |
+| varuint | type |
+| string | error message (ASCII-only characters) |
+| location | the location in the source this error applies to |
+| `1` | the level of the error: `0` for `fatal`, `1` for `argument`, `2` for `load` |
+
+### warning
+
+| # bytes | field |
+| --- | --- |
+| varuint | type |
+| string | warning message (ASCII-only characters) |
+| location | the location in the source this warning applies to |
+| `1` | the level of the warning: `0` for `default` and `1` for `verbose` |
+
+### integer
+
+| # bytes | field |
+| --- | --- |
+| `1` | `1` if the integer is negative, `0` if the integer is positive |
+| varuint | the number of words in this integer |
+| varuint+ | the words of the integer, least-significant to most-significant |
+
+## Structure
+
+The serialized string representing the syntax tree is composed of three parts: the header, the body, and the constant pool.
+The header contains information like the version of prism that serialized the tree.
+The body contains the actual nodes in the tree.
+The constant pool contains constants that were interned while parsing.
+
+The header is structured like the following table:
+
+| # bytes | field |
+| --- | --- |
+| `5` | "PRISM" |
+| `1` | major version number |
+| `1` | minor version number |
+| `1` | patch version number |
+| `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
+| string | the encoding name |
+| varsint | the start line |
+| varuint | number of newline offsets |
+| varuint* | newline offsets |
+| varuint | number of comments |
+| comment* | comments |
+| varuint | number of nodes with attached comments |
+| comment attachment* | comment attachments |
+| varuint | number of magic comments |
+| magic comment* | magic comments |
+| location? | the optional location of the `__END__` keyword and its contents |
+| varuint | number of errors |
+| error* | errors |
+| varuint | number of warnings |
+| warning* | warnings |
+| `1` | `1` if the source is continuable (incomplete but could become valid with more input), `0` otherwise |
+| `4` | content pool offset |
+| varuint | content pool size |
+
+After the header comes the body of the serialized string.
+The body consists of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
+Each node is structured like the following table:
+
+| # bytes | field |
+| --- | --- |
+| `1` | node type |
+| varuint | node identifier |
+| location | node location |
+| varuint | node flags |
+
+Every field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
+
+* `double` - A field that is a `double`. This is structured as a sequence of 8 bytes in native endian order.
+* `node` - A field that is a node. This is structured just as like parent node.
+* `node?` - A field that is a node that is optionally present. If the node is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just as like parent node.
+* `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, followed by the child nodes themselves.
+* `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string bytes (_without_ a trailing null byte).
+* `constant` - A variable-length integer that represents an index in the constant pool.
+* `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
+* `integer` - A field that represents an arbitrary-sized integer. The structure is listed above.
+* `location` - A field that is a location. This is structured as a variable-length integer start followed by a variable-length integer length.
+* `location?` - A field that is a location that is optionally present. If the location is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just like the `location` child node.
+* `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
+* `uint32` - A field that is a 32-bit unsigned integer. This is structured as a variable-length integer.
+
+After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Every constant is embedded in the serialization. Each constant is structured as follows:
+
+| # bytes | field |
+| --- | --- |
+| `4` | the byte offset in the serialization for the contents of the constant |
+| `4` | the byte length in the serialization |
+
+After the constant pool, the contents of the constants are serialized. This is just a sequence of bytes that represent the contents of the constants. At the end of the serialization, the buffer is null terminated.
+
+## APIs
+
+The relevant APIs and struct definitions are listed below:
+
+```c
+// A pm_buffer_t is a simple memory buffer that stores data in a contiguous
+// block of memory. It is used to store the serialized representation of a
+// prism tree.
+
+// Parse and serialize the AST represented by the given source to the given
+// buffer.
+void pm_serialize_parse(pm_buffer_t *buffer, const uint8_t *source, size_t length, const char *data);
+```
+
+Typically you would allocate a `pm_buffer_t` and call `pm_serialize_parse`, as in:
+
+```c
+void
+serialize(const uint8_t *source, size_t length) {
+  pm_buffer_t *buffer = pm_buffer_new();
+  pm_serialize_parse(buffer, source, length, NULL);
+
+  // Do something with the serialized string.
+
+  pm_buffer_free(buffer);
+}
+```
+
+The final argument to `pm_serialize_parse` is an optional string that controls the options to the parse function. This includes all of the normal options that could be passed to `pm_parser_init` through a `pm_options_t` struct, but serialized as a string to make it easier for callers through FFI. Note that no `varuint` are used here to make it easier to produce the data for the caller, and also serialized size is less important here. The format of the data is structured as follows:
+
+| # bytes | field                      |
+| ------- | -------------------------- |
+| `4`     | the length of the filepath |
+| ...     | the filepath bytes         |
+| `4`     | the line number            |
+| `4`     | the length the encoding    |
+| ...     | the encoding bytes         |
+| `1`     | frozen string literal      |
+| `1`     | command line flags         |
+| `1`     | syntax version, see [pm_options_version_t](https://github.com/ruby/prism/blob/main/include/prism/options.h) for valid values |
+| `1`     | whether or not the encoding is locked (should almost always be false) |
+| `4`     | the number of scopes       |
+| ...     | the scopes                 |
+
+Command line flags are a bitset. By default every flag is `0`. It includes the following values:
+
+* `0x1` - the `-a` option
+* `0x2` - the `-e` option
+* `0x4` - the `-l` option
+* `0x8` - the `-n` option
+* `0x10` - the `-p` option
+* `0x20` - the `-x` option
+
+Scopes are ordered from the outermost scope to the innermost one.
+
+Each scope is laid out as follows:
+
+| # bytes | field                      |
+| ------- | -------------------------- |
+| `4`     | the number of locals       |
+| ...     | the locals                 |
+
+Each local is laid out as follows:
+
+| # bytes | field                      |
+| ------- | -------------------------- |
+| `4`     | the length of the local    |
+| ...     | the local bytes            |
+
+The data can be `NULL` (as seen in the example above).

data/docs/testing.md

@@ -0,0 +1,72 @@
+# Testing
+
+This document explains how to test prism, both locally, and against existing test suites.
+
+## Test suite
+
+`rake test` will run all of the files in the `test/` directory. This can be conceived of as two parts: unit tests, and snapshot tests.
+
+### Unit tests
+
+These test specific prism implementation details like comments, errors, and regular expressions. There are corresponding files for each thing being tested (like `test/errors_test.rb`).
+
+### Snapshot tests
+
+Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/prism/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/prism/snapshots/` directory. For example, `test/prism/fixtures/strings.txt` has a corresponding `test/prism/snapshots/strings.txt`.
+
+If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/prism/snapshots/` directory, one will be created so that it exists for the next test run. If the snapshot already exists but differs you can run with `UPDATE_SNAPSHOTS=1` to regenerate them.
+
+When working on a specific example, you can limit tests to a single fixture via `FOCUS=strings.txt`.
+
+#### Versioning
+
+Prism is capable of parsing code with different version specifiers. The test suite reflects this:
+* Test files under `test/prism/snapshots/*.txt` are validated against all supported versions.
+* Test files under `test/prism/snapshots/[MAJOR].[MINOR]/*.txt` are validated against all versions starting from the given version.
+* Test files under `test/prism/snapshots/[MAJOR].[MINOR]-[MAJOR].[MINOR]/*.txt` are validated against the given version range.
+
+Should you add logic specific to a new ruby version, validate the new behavior but also that old behavior remains the same.
+
+### Error tests
+
+Error tests are similar to snapshot tests and expectations are located under `test/prism/errors/`.
+They contain invalid syntax and rendered errors, similar to how `ruby -c` would show them.
+Error tests are versioned in the same way that snapshot tests are and also support `FOCUS`/`UPDATE_SNAPSHOTS`.
+
+## Local testing
+
+As you are working, you will likely want to test your code locally. `test.rb` is ignored by git, so it can be used for local testing. **bin/prism** contains a multitude of options to aid you, depending on which area
+of prism you are working on. `bin/prism --help` prints a list of all available commands.
+
+Commands that take a `[source]` can be called in three different ways:
+1. `bin/prism parse` will use `test.rb` as input.
+1. `bin/prism parse foo.rb` will read from the given path.
+1. `bin/prism parse -e "1 + 2"` will use the given source code.
+
+`bin/prism parse` is aliased as `bin/parse` for for convenience.
+
+### Validating your changes against real code
+
+If you create a change which is not expected to have a large impact on real code, you can use
+`bin/compare` to validate that assumption. It compiles two versions of prism (for example
+your feature branch and main), comparing ast, errors and warnings. Once done, a report is
+printed showcasing in what way files changed, if any (syntax valid -> syntax invalid, changed AST, etc.)
+
+```sh
+$ bin/compare main feature-branch ~/all-of-rubygems`
+...
+0/3157385
+1000/3157385
+...
+3157000/3157385
+Oops:
+intermine-1.05.00/lib/intermine/lists.rb changed from valid(true) to valid(false)
+whistle-0.1.1/lib/resource.rb changed from valid(true) to valid(false)
+Took 2661.5390081949999512 seconds
+```
+
+It is up to you to aquire code samples. Here are some resources you can use for this purpose:
+* https://github.com/jeromedalbert/real-world-ruby-apps
+* https://github.com/eliotsykes/real-world-rails
+* https://github.com/rubygems/rubygems-mirror
+  Requires some work to unpack the downloaded gems. You should also run with `RUBYGEMS_MIRROR_ONLY_LATEST=TRUE`.