yarp 0.6.0

data/docs/build_system.md

@@ -0,0 +1,85 @@
+# Build System
+
+There are many ways to build YARP, which means the build system is a bit more complicated than usual.
+
+## Requirements
+
+* It must work to build YARP for all 6 uses-cases below.
+* It must be possible to build YARP without needing ruby/rake/etc.
+  Because once YARP is the single parser in TruffleRuby, JRuby or CRuby there won't be another Ruby parser around to parse such Ruby code.
+  Most/every Ruby implementations want to avoid depending on another Ruby during the build process as that is very brittle.
+* It is desirable to compile YARP with the same or very similar compiler flags for all use-cases (e.g. optimization level, warning flags, etc).
+  Otherwise, there is the risk YARP does not work correctly with those different compiler flags.
+
+The main solution for the second point seems a Makefile, otherwise many of the usages would have to duplicate the logic to build YARP.
+
+## General Design
+
+1. Templates are generated by `templates/template.rb`
+2. `autoconf` creates `./configure` and `autoheader` creates `config.h.in` (both files are platform-independent)
+3. `./configure` creates `include/yarp/config.h` (which contains `HAVE_*` macros, platform-specific) and the `Makefile`
+4. The `Makefile` compiles both `librubyparser.a` and `librubyparser.{so,dylib,dll}` from the `src/**/*.c` and `include/**/*.h` files
+5. The `Rakefile` `:compile` task ensures the above prerequisites are done, then calls `make`,
+  and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`), which uses `librubyparser.a`
+
+This way there is minimal duplication, and each layer builds on the previous one and has its own responsibilities.
+
+The static library exports no symbols, to avoid any conflict.
+The shared library exports some symbols, and this is fine since there should only be one librubyparser shared library
+loaded per process (i.e., at most one version of the yarp *gem* loaded in a process, only the gem uses the shared library).
+
+## The various ways to build YARP
+
+### Building from ruby/yarp repository with `bundle exec rake`
+
+`rake` calls `make` and then uses `Rake::ExtensionTask` to compile the C extension (see above).
+
+### Building the yarp gem by `gem install/bundle install`
+
+The gem contains the pre-generated templates, as well as `configure` and `config.h.in`
+When installing the gem, `extconf.rb` is used and that:
+* runs `./configure` which creates the `Makefile` and `include/yarp/config.h`
+* runs `make build/librubyparser.a`
+* compiles the C extension with mkmf
+
+When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead of the last step,
+there is Ruby code using Fiddle which uses `librubyparser.{so,dylib,dll}`
+to implement the same methods as the C extension, but using serialization instead of many native calls/accesses
+(JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
+
+### Building the yarp gem from git, e.g. `gem 'yarp', github: 'ruby/yarp'`
+
+The same as above, except the `extconf.rb` additionally runs first:
+* `templates/template.rb` to generate the templates
+* `autoconf` and `autoheader` to generate `configure` and `config.h.in`
+
+Because of course those files are not part of the git repository.
+
+### Building YARP as part of CRuby
+
+[This script](https://github.com/ruby/ruby/blob/32e828bb4a6c65a392b2300f3bdf93008c7b6f25/tool/sync_default_gems.rb#L399-L426) imports YARP sources in CRuby.
+
+The script generates the templates when importing.
+
+`include/yarp/config.h` is replaced by `#include "ruby/config.h"`.
+It is assumed that CRuby's `./configure` is a superset of YARP's configure checks.
+
+YARP's `autotools` is not used at all in CRuby and in fact YARP's `Makefile` is not used either.
+Instead, CRuby's `autotools` setup is used, and `CRuby`'s Makefiles are used.
+
+### Building YARP as part of TruffleRuby
+
+[This script](https://github.com/oracle/truffleruby/blob/master/tool/import-yarp.sh) imports YARP sources in TruffleRuby.
+The script generates the templates when importing.
+It also generates `configure` and `config.h.in` (to avoid needing `autotools` on every machine building TruffleRuby).
+
+Then when `mx build` builds TruffleRuby and the `yarp` mx project inside, it:
+* runs `./configure`
+* runs `make`
+
+Then the `yarp bindings` mx project is built, which contains the [bindings](https://github.com/oracle/truffleruby/blob/master/src/main/c/yarp_bindings/src/yarp_bindings.c)
+and links to `librubyparser.a` (to avoid exporting symbols, so no conflict when installing the yarp gem).
+
+### Building YARP as part of JRuby
+
+TODO, probably similar to TruffleRuby.

data/docs/building.md

@@ -0,0 +1,26 @@
+# Building
+
+The following describes how to build YARP from source.
+
+## Common
+
+All of the source files match `src/**/*.c` and all of the headers match `include/**/*.h`.
+
+The following flags should be used to compile YARP:
+
+* `-std=c99` - Use the C99 standard
+* `-Wall -Wconversion -Wextra -Wpedantic -Wundef` - Enable the warnings we care about
+* `-Werror` - Treat warnings as errors
+* `-fvisibility=hidden` - Hide all symbols by default
+
+The following flags can be used to compile YARP:
+
+* `-DHAVE_MMAP` - Should be passed if the system has the `mmap` function
+* `-DHAVE_SNPRINTF` - Should be passed if the system has the `snprintf` function
+
+## Shared
+
+If you want to build YARP as a shared library and link against it, you should compile with:
+
+* `-fPIC -shared` - Compile as a shared library
+* `-DYP_EXPORT_SYMBOLS` - Export the symbols (by default nothing is exported)

data/docs/configuration.md

@@ -0,0 +1,56 @@
+# Configuration
+
+A lot of code in YARP's repository is templated from a single configuration file, [config.yml](../config.yml). This file is used to generate the following files:
+
+* `ext/yarp/api_node.c` - for defining how to build Ruby objects for the nodes out of C structs
+* `include/yarp/ast.h` - for defining the C structs that represent the nodes
+* `java/org/yarp/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
+* `java/org/yarp/Loader.java` - for defining how to deserialize the nodes in Java
+* `java/org/yarp/Nodes.java` - for defining the nodes in Java
+* `lib/yarp/node.rb` - for defining the nodes in Ruby
+* `lib/yarp/serialize.rb` - for defining how to deserialize the nodes in Ruby
+* `src/node.c` - for defining how to free the nodes in C and calculate the size in memory in C
+* `src/prettyprint.c` - for defining how to prettyprint the nodes in C
+* `src/serialize.c` - for defining how to serialize the nodes in C
+* `src/token_type.c` - for defining the names of the token types
+
+Whenever the structure of the nodes changes, you can run `rake templates` to regenerate these files. Alternatively tasks like `rake test` should pick up on these changes automatically. Every file that is templated will include a comment at the top indicating that it was generated and that changes should be made to the template and not the generated file.
+
+`config.yml` has a couple of top level fields, which we'll describe below.
+
+## `tokens`
+
+This is a list of tokens to be used by the lexer. It is shared here so that it can be templated out into both an enum and a function that is used for debugging that returns the name of the token.
+
+Each token is expected to have a `name` key and a `comment` key (both as strings). Optionally they can have a `value` key (an integer) which is used to represent the value in the enum.
+
+In C these tokens will be templated out with the prefix `YP_TOKEN_`. For example, if you have a `name` key with the value `PERCENT`, you can access this in C through `YP_TOKEN_PERCENT`.
+
+## `flags`
+
+Sometimes we need to communicate more information in the tree than can be represented by the types of the nodes themselves. For example, we need to represent the flags passed to a regular expression or the type of call that a call node is performing. In these circumstances, it's helpful to reference a bitset of flags. This field is a list of flags that can be used in the nodes.
+
+Each flag is expected to have a `name` key (a string) and a `values` key (an array). Each value in the `values` key should be an object that contains both a `name` key (a string) that represents the name of the flag and a `comment` key (a string) that represents the comment for the flag.
+
+In C these flags will get templated out with a `YP_` prefix, then a snake-case version of the flag name, then the flag itself. For example, if you have a flag with the name `RegularExpressionFlags` and a value with the name `IGNORE_CASE`, you can access this in C through `YP_REGULAR_EXPRESSION_FLAGS_IGNORE_CASE`.
+
+## `nodes`
+
+Every node in the tree is defined in `config.yml`. Each node is expected to have a `name` key (a string) and a `comment` key (a string). By convention, the `comment` key uses the multi-line syntax of `: |` because the newlines will get templated into the comments of various files.
+
+Optionally, every node can define a `child_nodes` key that is an array. This array represents each part of the node that isn't communicated through the type and location of the node itself. Within the `child_nodes` key, each entry should be an object with a `name` key (a string) and a `type` key (a string). The `name` key represents the name of the child node and the `type` is used to determine how it should be represented in each language.
+
+The available values for `type` are:
+
+* `node` - A child node that is a node itself. This is a `yp_node_t *` in C.
+* `node?` - A child node that is optionally present. This is also a `yp_node_t *` in C, but can be `NULL`.
+* `node[]` - A child node that is an array of nodes. This is a `yp_node_list_t` in C.
+* `string` - A child node that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is a `yp_string_t` in C.
+* `constant` - A variable-length integer that represents an index in the constant pool. This is a `yp_constant_id_t` in C.
+* `constant[]` - A child node that is an array of constants. This is a `yp_constant_id_list_t` in C.
+* `location` - A child node that is a location. This is a `yp_location_t` in C.
+* `location?` - A child node that is a location that is optionally present. This is a `yp_location_t` in C, but if the value is not present then the `start` and `end` fields will be `NULL`.
+* `location[]` - A child node that is an array of locations. This is a `yp_location_list_t` in C.
+* `uint32` - A child node that is a 32-bit unsigned integer. This is a `uint32_t` in C.
+
+If the type is `node` or `node?` then the value also accepts an optional `kind` key (a string). This key is expected to match to the name of another node type within `config.yml`. This changes a couple of places where code is templated out to use the more specific struct name instead of the generic `yp_node_t`. For example, with `kind: StatementsNode` the `yp_node_t *` in C becomes a `yp_statements_node_t *`.

data/docs/design.md

@@ -0,0 +1,53 @@
+# Design
+
+There are three overall goals for this project:
+
+* to provide a documented and maintainable parser
+* to provide an error-tolerant parser suitable for use in an IDE
+* to provide a portable parser that can be used in projects that don't link against CRuby
+
+The design of the parser is based around these main goals.
+
+## Structure
+
+The first piece to understand about the parser is the design of its syntax tree. This is documented in `config.yml`. Every token and node is defined in that file, along with comments about where they are found in what kinds of syntax. This file is used to template out a lot of different files, all found in the `templates` directory. The `templates/template.rb` script performs the templating and outputs all files matching the directory structure found in the templates directory.
+
+The templated files contain all of the code required to allocate and initialize nodes, pretty print nodes, and serialize nodes. This means for the most part, you will only need to then hook up the parser to call the templated functions to create the nodes in the correct position. That means editing the parser itself, which is housed in `yarp.c`.
+
+## Pratt parsing
+
+In order to provide the best possible error tolerance, the parser is hand-written. It is structured using Pratt parsing, a technique developed by Vaughan Pratt back in the 1970s. Below are a bunch of links to articles and papers that explain Pratt parsing in more detail.
+
+* https://web.archive.org/web/20151223215421/http://hall.org.ua/halls/wizzard/pdf/Vaughan.Pratt.TDOP.pdf
+* https://tdop.github.io/
+* https://journal.stuffwithstuff.com/2011/03/19/pratt-parsers-expression-parsing-made-easy/
+* https://matklad.github.io/2020/04/13/simple-but-powerful-pratt-parsing.html
+* https://chidiwilliams.com/post/on-recursive-descent-and-pratt-parsing/
+
+You can find most of the functions that correspond to constructs in the Pratt parsing algorithm in `yarp.c`. As a couple of examples:
+
+* `parse` corresponds to the `parse_expression` function
+* `nud` (null denotation) corresponds to the `parse_expression_prefix` function
+* `led` (left denotation) corresponds to the `parse_expression_infix` function
+* `lbp` (left binding power) corresponds to accessing the `left` field of an element in the `binding_powers` array
+* `rbp` (right binding power) corresponds to accessing the `right` field of an element in the `binding_powers` array
+
+## Portability
+
+In order to enable using this parser in other projects, the parser is written in C99, and uses only the standard library. This means it can be embedded in most any other project without having to link against CRuby. It can be used directly through its C API to access individual fields, or it can used to parse a syntax tree and then serialize it to a single blob. For more information on serialization, see the [docs/serialization.md](serialization.md) file.
+
+## Error tolerance
+
+The design of the error tolerance of this parser is still very much in flux. We are experimenting with various approaches as the parser is being developed to try to determine the best approach. Below are a bunch of links to articles and papers that explain error tolerance in more detail, as well as document some of the approaches that we're evaluating.
+
+* https://tratt.net/laurie/blog/2020/automatic_syntax_error_recovery.html
+* https://diekmann.uk/diekmann_phd.pdf
+* https://eelcovisser.org/publications/2012/JongeKVS12.pdf
+* https://www.antlr.org/papers/allstar-techreport.pdf
+* https://github.com/microsoft/tolerant-php-parser/blob/main/docs/HowItWorks.md
+
+Currently, there are a couple of mechanisms for error tolerance that are in place:
+
+* If the parser expects a token in a particular position (for example the `in` keyword in a for loop or the `{` after `BEGIN` or `END`) then it will insert a missing token if one can't be found and continue parsing.
+* If the parser expects an expression in a particular position but encounters a token that can't be used as that expression, it checks up the stack to see if that token would close out a parent node. If so, it will close out all of its parent nodes using missing nodes wherever necessary and continue parsing.
+* If the parser cannot understand a token in any capacity, it will skip past the token.

data/docs/encoding.md

@@ -0,0 +1,116 @@
+# Encoding
+
+When parsing a Ruby file, there are times when the parser must parse identifiers. Identifiers are names of variables, methods, classes, etc. To determine the start of an identifier, the parser must be able to tell if the subsequent bytes form an alphabetic character. To determine the rest of the identifier, the parser must look forward through all alphanumeric characters.
+
+Determining if a set of bytes comprise an alphabetic or alphanumeric character is encoding-dependent. By default, the parser assumes that all source files are encoded UTF-8. If the file is not encoded in UTF-8, it must be encoded using an encoding that is "ASCII compatible" (i.e., all of the codepoints below 128 match the corresponding codepoints in ASCII and the minimum number of bytes required to represent a codepoint is 1 byte).
+
+If the file is not encoded in UTF-8, the user must specify the encoding in a "magic" comment at the top of the file. The comment looks like:
+
+```ruby
+# encoding: iso-8859-9
+```
+
+The key of the comment can be either "encoding" or "coding". The value of the comment must be a string that is a valid encoding name. The encodings that YARP supports by default are:
+
+* `ascii`
+* `ascii-8bit`
+* `big5`
+* `binary`
+* `cp932`
+* `euc-jp`
+* `gbk`
+* `iso-8859-1`
+* `iso-8859-2`
+* `iso-8859-3`
+* `iso-8859-4`
+* `iso-8859-5`
+* `iso-8859-6`
+* `iso-8859-7`
+* `iso-8859-8`
+* `iso-8859-9`
+* `iso-8859-10`
+* `iso-8859-11`
+* `iso-8859-13`
+* `iso-8859-14`
+* `iso-8859-15`
+* `iso-8859-16`
+* `koi8-r`
+* `shift_jis`
+* `sjis`
+* `us-ascii`
+* `utf-8`
+* `windows-31j`
+* `windows-1251`
+* `windows-1252`
+
+For each of these encodings, YARP provides a function for checking if the subsequent bytes form an alphabetic or alphanumeric character.
+
+## Support for other encodings
+
+If an encoding is encountered that is not supported by YARP, YARP will call a user-provided callback function with the name of the encoding if one is provided. That function can be registered with `yp_parser_register_encoding_decode_callback`. The user-provided callback function can then provide a pointer to an encoding struct that contains the requisite functions that YARP will use those to parse identifiers going forward.
+
+If the user-provided callback function returns `NULL` (the value also provided by the default implementation in case a callback was not registered), an error will be added to the parser's error list and parsing will continue on using the default UTF-8 encoding.
+
+```c
+// This struct defines the functions necessary to implement the encoding
+// interface so we can determine how many bytes the subsequent character takes.
+// Each callback should return the number of bytes, or 0 if the next bytes are
+// invalid for the encoding and type.
+typedef struct {
+    // Return the number of bytes that the next character takes if it is valid
+    // in the encoding. Does not read more than n bytes. It is assumed that n is
+    // at least 1.
+    size_t (*char_width)(const char *c, ptrdiff_t n);
+
+    // Return the number of bytes that the next character takes if it is valid
+    // in the encoding and is alphabetical. Does not read more than n bytes. It
+    // is assumed that n is at least 1.
+    size_t (*alpha_char)(const char *c, ptrdiff_t n);
+
+    // Return the number of bytes that the next character takes if it is valid
+    // in the encoding and is alphanumeric. Does not read more than n bytes. It
+    // is assumed that n is at least 1.
+    size_t (*alnum_char)(const char *c, ptrdiff_t n);
+
+    // Return true if the next character is valid in the encoding and is an
+    // uppercase character. Does not read more than n bytes. It is assumed that
+    // n is at least 1.
+    bool (*isupper_char)(const char *c, ptrdiff_t n);
+
+    // The name of the encoding. This should correspond to a value that can be
+    // passed to Encoding.find in Ruby.
+    const char *name;
+
+    // Return true if the encoding is a multibyte encoding.
+    bool multibyte;
+} yp_encoding_t;
+
+// When an encoding is encountered that isn't understood by YARP, we provide
+// the ability here to call out to a user-defined function to get an encoding
+// struct. If the function returns something that isn't NULL, we set that to
+// our encoding and use it to parse identifiers.
+typedef yp_encoding_t *(*yp_encoding_decode_callback_t)(yp_parser_t *parser, const char *name, size_t width);
+
+// Register a callback that will be called when YARP encounters a magic comment
+// with an encoding referenced that it doesn't understand. The callback should
+// return NULL if it also doesn't understand the encoding or it should return a
+// pointer to a yp_encoding_t struct that contains the functions necessary to
+// parse identifiers.
+YP_EXPORTED_FUNCTION void
+yp_parser_register_encoding_decode_callback(yp_parser_t *parser, yp_encoding_decode_callback_t callback);
+```
+
+## Getting notified when the encoding changes
+
+You may want to get notified when the encoding changes based on the result of parsing an encoding comment. We use this internally for our `lex` function in order to provide the correct encodings for the tokens that are returned. For that you can register a callback with `yp_parser_register_encoding_changed_callback`. The callback will be called with a pointer to the parser. The encoding can be accessed through `parser->encoding`.
+
+```c
+// When the encoding that is being used to parse the source is changed by YARP,
+// we provide the ability here to call out to a user-defined function.
+typedef void (*yp_encoding_changed_callback_t)(yp_parser_t *parser);
+
+// Register a callback that will be called whenever YARP changes the encoding it
+// is using to parse based on the magic comment.
+YP_EXPORTED_FUNCTION void
+yp_parser_register_encoding_changed_callback(yp_parser_t *parser, yp_encoding_changed_callback_t callback);
+```

data/docs/extension.md

@@ -0,0 +1,20 @@
+# Extension
+
+Part of this parser project is a native extension that provides a Ruby API that wraps calls to the C API. This allows you to invoke parsing from Ruby code. It also provides a Ruby API for accessing the syntax tree.
+
+For the most part, the API for accessing the tree mirrors that found in the [Syntax Tree](https://github.com/ruby-syntax-tree/syntax_tree) project. This means:
+
+* Walking the tree involves creating a visitor and passing it to the `#accept` method on any node in the tree
+* Nodes in the tree respond to named methods for accessing their children as well as `#child_nodes`
+* Nodes respond to the pattern matching interfaces `#deconstruct` and `#deconstruct_keys`
+
+Every entry in `config.yml` will generate a Ruby class as well as the code that builds the nodes themselves. Creating a syntax tree involves calling one of the class methods on the `YARP` module. The full API is documented below.
+
+## API
+
+* `YARP.dump(source, filepath)` - parse the syntax tree corresponding to the given source string and filepath, and serialize it to a string. Filepath can be nil.
+* `YARP.dump_file(filepath)` - parse the syntax tree corresponding to the given source file and serialize it to a string
+* `YARP.lex(source)` - parse the tokens corresponding to the given source string and return them as an array within a parse result
+* `YARP.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
+* `YARP.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
+* `YARP.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result

data/docs/fuzzing.md

@@ -0,0 +1,93 @@
+# Fuzzing
+
+We use fuzzing to test the various entrypoints to the library. The fuzzer we use is [AFL++](https://aflplus.plus). All files related to fuzzing live within the `fuzz` directory, which has the following structure:
+
+```
+fuzz
+├── corpus
+│   ├── parse             fuzzing corpus for parsing (a symlink to our fixtures)
+│   ├── regexp            fuzzing corpus for regexp
+│   └── unescape          fuzzing corpus for unescaping strings
+├── dict                  a AFL++ dictionary containing various tokens
+├── docker
+│   └── Dockerfile        for building a container with the fuzzer toolchain
+├── fuzz.c                generic entrypoint for fuzzing
+├── heisenbug.c           entrypoint for reproducing a crash or hang
+├── parse.c               fuzz handler for parsing
+├── parse.sh              script to run parsing fuzzer
+├── regexp.c              fuzz handler for regular expression parsing
+├── regexp.sh             script to run regexp fuzzer
+├── tools
+│   ├── backtrace.sh      generates backtrace files for a crash directory
+│   └── minimize.sh       generates minimized crash or hang files
+├── unescape.c            fuzz handler for unescape functionality
+└── unescape.sh           script to run unescape fuzzer
+```
+
+## Usage
+
+There are currently three fuzzing targets
+
+- `yp_parse_serialize` (parse)
+- `yp_regexp_named_capture_group_names` (regexp)
+- `yp_unescape_manipulate_string` (unescape)
+
+Respectively, fuzzing can be performed with
+
+```
+make fuzz-run-parse
+make fuzz-run-regexp
+make fuzz-run-unescape
+```
+
+To end a fuzzing job, interrupt with CTRL+C. To enter a container with the fuzzing toolchain and debug utilities, run
+
+```
+make fuzz-debug
+```
+
+# Out-of-bounds reads
+
+Currently, encoding functionality implementing the `yp_encoding_t` interface can read outside of inputs. For the time being, ASAN instrumentation is disabled for functions from src/enc. See `fuzz/asan.ignore`.
+
+To disable ASAN read instrumentation globally, use the `FUZZ_FLAGS` environment variable e.g.
+
+```
+FUZZ_FLAGS="-mllvm -asan-instrument-reads=false" make fuzz-run-parse
+```
+
+Note, that this may make reproducing bugs difficult as they may depend on memory outside of the input buffer. In that case, try
+
+```
+make fuzz-debug # enter the docker container with build tools
+make build/fuzz.heisenbug.parse # or .unescape or .regexp
+./build/fuzz.heisenbug.parse path-to-problem-input
+```
+
+# Triaging Crashes and Hangs
+
+Triaging crashes and hangs is easier when the inputs are as short as possible. In the fuzz container, an entire crash or hang directory can be minimized using
+
+```
+./fuzz/tools/minimize.sh <directory>
+```
+
+e.g.
+```
+./fuzz/tools/minimize.sh fuzz/output/parse/default/crashes
+```
+
+This may take a long time. In the crash/hang directory, for each input file there will appear a minimized version with the extension `.min` appended.
+
+Backtraces for crashes (not hangs) can be generated en masse with
+
+```
+./fuzz/tools/backtrace.sh <directory>
+```
+
+Files with basename equal to the input file name with extension `.bt` will be created e.g.
+
+```
+id:000000,sig:06,src:000006+000190,time:8480,execs:18929,op:splice,rep:4
+id:000000,sig:06,src:000006+000190,time:8480,execs:18929,op:splice,rep:4.bt
+```

data/docs/heredocs.md

@@ -0,0 +1,36 @@
+# Heredocs
+
+Heredocs are one of the most complicated pieces of this parser. There are many different forms, there can be multiple open at the same time, and they can be nested. In order to support parsing them, we keep track of a lot of metadata. Below is a basic overview of how it works.
+
+## 1. Lexing the identifier
+
+When a heredoc identifier is encountered in the regular process of lexing, we push the `YP_LEX_HEREDOC` mode onto the stack with the following metadata:
+
+* `ident_start`: A pointer to the start of the identifier for the heredoc. We need this to match against the end of the heredoc.
+* `ident_length`: The length of the identifier for the heredoc. We also need this to match.
+* `next_start`: A pointer to the place in source that the parser should resume lexing once it has completed this heredoc.
+
+We also set the special `parser.next_start` field which is a pointer to the place in the source where we should start lexing the next token. This is set to the pointer of the character immediately following the next newline.
+
+Note that if the `parser.heredoc_end` field is already set, then it means we have already encountered a heredoc on this line. In that case the `parser.next_start` field will be set to the `parser.heredoc_end` field. This is because we want to skip past the heredoc previous heredocs on this line and instead lex the body of this heredoc.
+
+## 2. Lexing the body
+
+The next time the lexer is asked for a token, it will be in the `YP_LEX_HEREDOC` mode. In this mode we are lexing the body of the heredoc. It will start by checking if the `next_start` field is set. If it is, then this is the first token within the body of the heredoc so we'll start lexing from there. Otherwise we'll start lexing from the end of the previous token.
+
+Lexing these fields is extremely similar to lexing an interpolated string. The only difference is that we also do an additional check at the beginning of each line to check if we have hit the terminator.
+
+## 3. Lexing the terminator
+
+On every newline within the body of a heredoc, we check to see if it matches the terminator followed by a newline or a carriage return and a newline. If it does, then we pop the lex mode off the stack and set a couple of fields on the parser:
+
+* `next_start`: This is set to the value that we previously stored on the heredoc to indicate where the lexer should resume lexing when it is done with this heredoc.
+* `heredoc_end`: This is set to the end of the heredoc. When a newline character is found, this indicates that the lexer should skip past to this next point.
+
+## 4. Lexing the rest of the line
+
+Once the heredoc has been lexed, the lexer will resume lexing from the `next_start` field. Lexing will continue until the next newline character. When the next newline character is found, it will check to see if the `heredoc_end` field is set. If it is it will skip to that point, unset the field, and continue lexing.
+
+## Compatibility with Ripper
+
+The order in which tokens are emitted is different from that of Ripper. Ripper emits each token in the file in the order in which it appears. YARP instead will emit the tokens that makes the most sense for the lexer, using the process described above. Therefore to line things up, `YARP.lex_compat` will shuffle the tokens around to match Ripper's output.

data/docs/mapping.md

@@ -0,0 +1,117 @@
+# Mapping
+
+When considering the previous CRuby parser versus YARP, this document should be helpful to understand how various concepts are mapped.
+
+## Nodes
+
+The following table shows how the various CRuby nodes are mapped to YARP nodes.
+
+| CRuby | YARP |
+| --- | --- |
+| `NODE_SCOPE` | |
+| `NODE_BLOCK` | |
+| `NODE_IF` | `YP_NODE_IF_NODE` |
+| `NODE_UNLESS` | `YP_NODE_UNLESS_NODE` |
+| `NODE_CASE` | `YP_NODE_CASE_NODE` |
+| `NODE_CASE2` | `YP_NODE_CASE_NODE` (with a null predicate) |
+| `NODE_CASE3` | |
+| `NODE_WHEN` | `YP_NODE_WHEN_NODE` |
+| `NODE_IN` | `YP_NODE_IN_NODE` |
+| `NODE_WHILE` | `YP_NODE_WHILE_NODE` |
+| `NODE_UNTIL` | `YP_NODE_UNTIL_NODE` |
+| `NODE_ITER` | `YP_NODE_CALL_NODE` (with a non-null block) |
+| `NODE_FOR` | `YP_NODE_FOR_NODE` |
+| `NODE_FOR_MASGN` | `YP_NODE_FOR_NODE` (with a multi-write node as the index) |
+| `NODE_BREAK` | `YP_NODE_BREAK_NODE` |
+| `NODE_NEXT` | `YP_NODE_NEXT_NODE` |
+| `NODE_REDO` | `YP_NODE_REDO_NODE` |
+| `NODE_RETRY` | `YP_NODE_RETRY_NODE` |
+| `NODE_BEGIN` | `YP_NODE_BEGIN_NODE` |
+| `NODE_RESCUE` | `YP_NODE_RESCUE_NODE` |
+| `NODE_RESBODY` | |
+| `NODE_ENSURE` | `YP_NODE_ENSURE_NODE` |
+| `NODE_AND` | `YP_NODE_AND_NODE` |
+| `NODE_OR` | `YP_NODE_OR_NODE` |
+| `NODE_MASGN` | `YP_NODE_MULTI_WRITE_NODE` |
+| `NODE_LASGN` | `YP_NODE_LOCAL_VARIABLE_WRITE_NODE` |
+| `NODE_DASGN` | `YP_NODE_LOCAL_VARIABLE_WRITE_NODE` |
+| `NODE_GASGN` | `YP_NODE_GLOBAL_VARIABLE_WRITE_NODE` |
+| `NODE_IASGN` | `YP_NODE_INSTANCE_VARIABLE_WRITE_NODE` |
+| `NODE_CDECL` | `YP_NODE_CONSTANT_PATH_WRITE_NODE` |
+| `NODE_CVASGN` | `YP_NODE_CLASS_VARIABLE_WRITE_NODE` |
+| `NODE_OP_ASGN1` | |
+| `NODE_OP_ASGN2` | |
+| `NODE_OP_ASGN_AND` | `YP_NODE_OPERATOR_AND_ASSIGNMENT_NODE` |
+| `NODE_OP_ASGN_OR` | `YP_NODE_OPERATOR_OR_ASSIGNMENT_NODE` |
+| `NODE_OP_CDECL` | |
+| `NODE_CALL` | `YP_NODE_CALL_NODE` |
+| `NODE_OPCALL` | `YP_NODE_CALL_NODE` (with an operator as the method) |
+| `NODE_FCALL` | `YP_NODE_CALL_NODE` (with a null receiver and parentheses) |
+| `NODE_VCALL` | `YP_NODE_CALL_NODE` (with a null receiver and parentheses or arguments) |
+| `NODE_QCALL` | `YP_NODE_CALL_NODE` (with a &. operator) |
+| `NODE_SUPER` | `YP_NODE_SUPER_NODE` |
+| `NODE_ZSUPER` | `YP_NODE_FORWARDING_SUPER_NODE` |
+| `NODE_LIST` | `YP_NODE_ARRAY_NODE` |
+| `NODE_ZLIST` | `YP_NODE_ARRAY_NODE` (with no child elements) |
+| `NODE_VALUES` | `YP_NODE_ARGUMENTS_NODE` |
+| `NODE_HASH` | `YP_NODE_HASH_NODE` |
+| `NODE_RETURN` | `YP_NODE_RETURN_NODE` |
+| `NODE_YIELD` | `YP_NODE_YIELD_NODE` |
+| `NODE_LVAR` | `YP_NODE_LOCAL_VARIABLE_READ_NODE` |
+| `NODE_DVAR` | `YP_NODE_LOCAL_VARIABLE_READ_NODE` |
+| `NODE_GVAR` | `YP_NODE_GLOBAL_VARIABLE_READ_NODE` |
+| `NODE_IVAR` | `YP_NODE_INSTANCE_VARIABLE_READ_NODE` |
+| `NODE_CONST` | `YP_NODE_CONSTANT_PATH_READ_NODE` |
+| `NODE_CVAR` | `YP_NODE_CLASS_VARIABLE_READ_NODE` |
+| `NODE_NTH_REF` | `YP_NODE_NUMBERED_REFERENCE_READ_NODE` |
+| `NODE_BACK_REF` | `YP_NODE_BACK_REFERENCE_READ_NODE` |
+| `NODE_MATCH` | |
+| `NODE_MATCH2` | `YP_NODE_CALL_NODE` (with regular expression as receiver) |
+| `NODE_MATCH3` | `YP_NODE_CALL_NODE` (with regular expression as only argument) |
+| `NODE_LIT` | |
+| `NODE_STR` | `YP_NODE_STRING_NODE` |
+| `NODE_DSTR` | `YP_NODE_INTERPOLATED_STRING_NODE` |
+| `NODE_XSTR` | `YP_NODE_X_STRING_NODE` |
+| `NODE_DXSTR` | `YP_NODE_INTERPOLATED_X_STRING_NODE` |
+| `NODE_EVSTR` | `YP_NODE_STRING_INTERPOLATED_NODE` |
+| `NODE_DREGX` | `YP_NODE_INTERPOLATED_REGULAR_EXPRESSION_NODE` |
+| `NODE_ONCE` | |
+| `NODE_ARGS` | `YP_NODE_PARAMETERS_NODE` |
+| `NODE_ARGS_AUX` | |
+| `NODE_OPT_ARG` | `YP_NODE_OPTIONAL_PARAMETER_NODE` |
+| `NODE_KW_ARG` | `YP_NODE_KEYWORD_PARAMETER_NODE` |
+| `NODE_POSTARG` | `YP_NODE_REQUIRED_PARAMETER_NODE` |
+| `NODE_ARGSCAT` | |
+| `NODE_ARGSPUSH` | |
+| `NODE_SPLAT` | `YP_NODE_SPLAT_NODE` |
+| `NODE_BLOCK_PASS` | `YP_NODE_BLOCK_ARGUMENT_NODE` |
+| `NODE_DEFN` | `YP_NODE_DEF_NODE` (with a null receiver) |
+| `NODE_DEFS` | `YP_NODE_DEF_NODE` (with a non-null receiver) |
+| `NODE_ALIAS` | `YP_NODE_ALIAS_NODE` |
+| `NODE_VALIAS` | `YP_NODE_ALIAS_NODE` (with a global variable first argument) |
+| `NODE_UNDEF` | `YP_NODE_UNDEF_NODE` |
+| `NODE_CLASS` | `YP_NODE_CLASS_NODE` |
+| `NODE_MODULE` | `YP_NODE_MODULE_NODE` |
+| `NODE_SCLASS` | `YP_NODE_S_CLASS_NODE` |
+| `NODE_COLON2` | `YP_NODE_CONSTANT_PATH_NODE` |
+| `NODE_COLON3` | `YP_NODE_CONSTANT_PATH_NODE` (with a null receiver) |
+| `NODE_DOT2` | `YP_NODE_RANGE_NODE` (with a .. operator) |
+| `NODE_DOT3` | `YP_NODE_RANGE_NODE` (with a ... operator) |
+| `NODE_FLIP2` | `YP_NODE_RANGE_NODE` (with a .. operator) |
+| `NODE_FLIP3` | `YP_NODE_RANGE_NODE` (with a ... operator) |
+| `NODE_SELF` | `YP_NODE_SELF_NODE` |
+| `NODE_NIL` | `YP_NODE_NIL_NODE` |
+| `NODE_TRUE` | `YP_NODE_TRUE_NODE` |
+| `NODE_FALSE` | `YP_NODE_FALSE_NODE` |
+| `NODE_ERRINFO` | |
+| `NODE_DEFINED` | `YP_NODE_DEFINED_NODE` |
+| `NODE_POSTEXE` | `YP_NODE_POST_EXECUTION_NODE` |
+| `NODE_DSYM` | `YP_NODE_INTERPOLATED_SYMBOL_NODE` |
+| `NODE_ATTRASGN` | `YP_NODE_CALL_NODE` (with a message that ends with =) |
+| `NODE_LAMBDA` | `YP_NODE_LAMBDA_NODE` |
+| `NODE_ARYPTN` | `YP_NODE_ARRAY_PATTERN_NODE` |
+| `NODE_HSHPTN` | `YP_NODE_HASH_PATTERN_NODE` |
+| `NODE_FNDPTN` | `YP_NODE_FIND_PATTERN_NODE` |
+| `NODE_ERROR` | `YP_NODE_MISSING_NODE` |
+| `NODE_LAST` | |
+```

data/docs/ripper.md

@@ -0,0 +1,36 @@
+# Ripper
+
+To test the parser, we compare against the output from `Ripper`, both for testing the lexer and testing the parser. The lexer test suite is much more feature complete at the moment.
+
+To lex source code using `YARP`, you typically would run `YARP.lex(source)`. If you want to instead get output that `Ripper` would normally produce, you can run `YARP.lex_compat(source)`. This will produce tokens that should be equivalent to `Ripper`.
+
+To parse source code using `YARP`, you typically would run `YARP.parse(source)`. If you want to instead using the `Ripper` streaming interface, you can inherit from `YARP::RipperCompat` and override the `on_*` methods. This will produce a syntax tree that should be equivalent to `Ripper`. That would look like:
+
+```ruby
+class ArithmeticRipper < YARP::RipperCompat
+  def on_binary(left, operator, right)
+    left.public_send(operator, right)
+  end
+
+  def on_int(value)
+    value.to_i
+  end
+
+  def on_program(stmts)
+    stmts
+  end
+
+  def on_stmts_new
+    []
+  end
+
+  def on_stmts_add(stmts, stmt)
+    stmts << stmt
+    stmts
+  end
+end
+
+ArithmeticRipper.new("1 + 2 - 3").parse # => [0]
+```
+
+There are also APIs for building trees similar to the s-expression builders in `Ripper`. The method names are the same. These include `YARP::RipperCompat.sexp_raw(source)` and `YARP::RipperCompat.sexp(source)`.