jruby-prism-parser 0.23.0.pre.SNAPSHOT-java

data/docs/build_system.md

@@ -0,0 +1,91 @@
+# Build System
+
+There are many ways to build prism, which means the build system is a bit more complicated than usual.
+
+## Requirements
+
+* It must work to build prism for all 6 uses-cases below.
+* It must be possible to build prism without needing ruby/rake/etc.
+  Because once prism is the single parser in TruffleRuby, JRuby or CRuby there won't be another Ruby parser around to parse such Ruby code.
+  Most/every Ruby implementations want to avoid depending on another Ruby during the build process as that is very brittle.
+* It is desirable to compile prism with the same or very similar compiler flags for all use-cases (e.g. optimization level, warning flags, etc).
+  Otherwise, there is the risk prism does not work correctly with those different compiler flags.
+
+The main solution for the second point seems a Makefile, otherwise many of the usages would have to duplicate the logic to build prism.
+
+## General Design
+
+1. Templates are generated by `templates/template.rb`
+4. The `Makefile` compiles both `libprism.a` and `libprism.{so,dylib,dll}` from the `src/**/*.c` and `include/**/*.h` files
+5. The `Rakefile` `:compile` task ensures the above prerequisites are done, then calls `make`,
+  and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`), which uses `libprism.a`
+
+This way there is minimal duplication, and each layer builds on the previous one and has its own responsibilities.
+
+The static library exports no symbols, to avoid any conflict.
+The shared library exports some symbols, and this is fine since there should only be one libprism shared library
+loaded per process (i.e., at most one version of the prism *gem* loaded in a process, only the gem uses the shared library).
+
+## The various ways to build prism
+
+### Building from ruby/prism repository with `bundle exec rake`
+
+`rake` calls `make` and then uses `Rake::ExtensionTask` to compile the C extension (see above).
+
+### Building the prism gem by `gem install/bundle install`
+
+The gem contains the pre-generated templates.
+When installing the gem, `extconf.rb` is used and that:
+* runs `make build/libprism.a`
+* compiles the C extension with mkmf
+
+When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead of the last step,
+there is Ruby code using FFI which uses `libprism.{so,dylib,dll}`
+to implement the same methods as the C extension, but using serialization instead of many native calls/accesses
+(JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
+
+### Building the prism gem from git, e.g. `gem "prism", github: "ruby/prism"`
+
+The same as above, except the `extconf.rb` additionally runs first:
+* `templates/template.rb` to generate the templates
+
+Because of course those files are not part of the git repository.
+
+### Building prism as part of CRuby
+
+[This script](https://github.com/ruby/ruby/blob/5124f9ac7513eb590c37717337c430cb93caa151/tool/sync_default_gems.rb#L399-L422) imports prism sources in CRuby.
+
+The script generates the templates when importing.
+
+prism's `Makefile` is not used at all in CRuby. Instead, CRuby's `Makefile` is used.
+
+### Building prism as part of TruffleRuby
+
+[This script](https://github.com/oracle/truffleruby/blob/master/tool/import-prism.sh) imports prism sources in TruffleRuby.
+The script generates the templates when importing.
+
+Then when `mx build` builds TruffleRuby and the `prism` mx project inside, it runs `make`.
+
+Then the `prism bindings` mx project is built, which contains the [bindings](https://github.com/oracle/truffleruby/blob/master/src/main/c/prism_bindings/src/prism_bindings.c)
+and links to `libprism.a` (to avoid exporting symbols, so no conflict when installing the prism gem).
+
+### Building prism as part of JRuby
+
+TODO, similar to TruffleRuby.
+
+### Building prism from source as a C library
+
+All of the source files match `src/**/*.c` and all of the headers match `include/**/*.h`.
+
+If you want to build prism as a shared library and link against it, you should compile with:
+
+* `-fPIC -shared` - Compile as a shared library
+* `-DPRISM_EXPORT_SYMBOLS` - Export the symbols (by default nothing is exported)
+
+#### Flags
+
+`make` respects the `MAKEFLAGS` environment variable. As such, to speed up the build you can run:
+
+```
+MAKEFLAGS="-j10" bundle exec rake compile
+```

data/docs/configuration.md

@@ -0,0 +1,64 @@
+# Configuration
+
+A lot of code in prism's repository is templated from a single configuration file, [config.yml](../config.yml). This file is used to generate the following files:
+
+* `ext/prism/api_node.c` - for defining how to build Ruby objects for the nodes out of C structs
+* `include/prism/ast.h` - for defining the C structs that represent the nodes
+* `javascript/src/deserialize.js` - for defining how to deserialize the nodes in JavaScript
+* `javascript/src/nodes.js` - for defining the nodes in JavaScript
+* `java/org/prism/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
+* `java/org/prism/Loader.java` - for defining how to deserialize the nodes in Java
+* `java/org/prism/Nodes.java` - for defining the nodes in Java
+* `lib/prism/compiler.rb` - for defining the compiler for the nodes in Ruby
+* `lib/prism/dispatcher.rb` - for defining the dispatch visitors for the nodes in Ruby
+* `lib/prism/dot_visitor.rb` - for defining the dot visitor for the nodes in Ruby
+* `lib/prism/dsl.rb` - for defining the DSL for the nodes in Ruby
+* `lib/prism/mutation_compiler.rb` - for defining the mutation compiler for the nodes in Ruby
+* `lib/prism/node.rb` - for defining the nodes in Ruby
+* `lib/prism/serialize.rb` - for defining how to deserialize the nodes in Ruby
+* `lib/prism/visitor.rb` - for defining the visitor interface for the nodes in Ruby
+* `src/node.c` - for defining how to free the nodes in C and calculate the size in memory in C
+* `src/prettyprint.c` - for defining how to prettyprint the nodes in C
+* `src/serialize.c` - for defining how to serialize the nodes in C
+* `src/token_type.c` - for defining the names of the token types
+
+Whenever the structure of the nodes changes, you can run `rake templates` to regenerate these files. Alternatively tasks like `rake test` should pick up on these changes automatically. Every file that is templated will include a comment at the top indicating that it was generated and that changes should be made to the template and not the generated file.
+
+`config.yml` has a couple of top level fields, which we'll describe below.
+
+## `tokens`
+
+This is a list of tokens to be used by the lexer. It is shared here so that it can be templated out into both an enum and a function that is used for debugging that returns the name of the token.
+
+Each token is expected to have a `name` key and a `comment` key (both as strings). Optionally they can have a `value` key (an integer) which is used to represent the value in the enum.
+
+In C these tokens will be templated out with the prefix `PM_TOKEN_`. For example, if you have a `name` key with the value `PERCENT`, you can access this in C through `PM_TOKEN_PERCENT`.
+
+## `flags`
+
+Sometimes we need to communicate more information in the tree than can be represented by the types of the nodes themselves. For example, we need to represent the flags passed to a regular expression or the type of call that a call node is performing. In these circumstances, it's helpful to reference a bitset of flags. This field is a list of flags that can be used in the nodes.
+
+Each flag is expected to have a `name` key (a string) and a `values` key (an array). Each value in the `values` key should be an object that contains both a `name` key (a string) that represents the name of the flag and a `comment` key (a string) that represents the comment for the flag.
+
+In C these flags will get templated out with a `PM_` prefix, then a snake-case version of the flag name, then the flag itself. For example, if you have a flag with the name `RegularExpressionFlags` and a value with the name `IGNORE_CASE`, you can access this in C through `PM_REGULAR_EXPRESSION_FLAGS_IGNORE_CASE`.
+
+## `nodes`
+
+Every node in the tree is defined in `config.yml`. Each node is expected to have a `name` key (a string) and a `comment` key (a string). By convention, the `comment` key uses the multi-line syntax of `: |` because the newlines will get templated into the comments of various files.
+
+Optionally, every node can define a `child_nodes` key that is an array. This array represents each part of the node that isn't communicated through the type and location of the node itself. Within the `child_nodes` key, each entry should be an object with a `name` key (a string) and a `type` key (a string). The `name` key represents the name of the child node and the `type` is used to determine how it should be represented in each language.
+
+The available values for `type` are:
+
+* `node` - A field that is a node. This is a `pm_node_t *` in C.
+* `node?` - A field that is a node that is optionally present. This is also a `pm_node_t *` in C, but can be `NULL`.
+* `node[]` - A field that is an array of nodes. This is a `pm_node_list_t` in C.
+* `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is a `pm_string_t` in C.
+* `constant` - A field that is an integer that represents an index in the constant pool. This is a `pm_constant_id_t` in C.
+* `constant[]` - A field that is an array of constants. This is a `pm_constant_id_list_t` in C.
+* `location` - A field that is a location. This is a `pm_location_t` in C.
+* `location?` - A field that is a location that is optionally present. This is a `pm_location_t` in C, but if the value is not present then the `start` and `end` fields will be `NULL`.
+* `uint8` - A field that is an 8-bit unsigned integer. This is a `uint8_t` in C.
+* `uint32` - A field that is a 32-bit unsigned integer. This is a `uint32_t` in C.
+
+If the type is `node` or `node?` then the value also accepts an optional `kind` key (a string). This key is expected to match to the name of another node type within `config.yml`. This changes a couple of places where code is templated out to use the more specific struct name instead of the generic `pm_node_t`. For example, with `kind: StatementsNode` the `pm_node_t *` in C becomes a `pm_statements_node_t *`.

data/docs/cruby_compilation.md

@@ -0,0 +1,27 @@
+# Compiling Prism's AST
+
+One important class of consumers of Prism's AST is compilers. Currently [CRuby](https://github.com/ruby/ruby), [JRuby](https://github.com/jruby/jruby), [TruffleRuby](https://github.com/oracle/truffleruby), and [Natalie](https://github.com/natalie-lang/natalie) have all built compilation code on top of Prism's AST.
+
+This document will describe, at a high level, how CRuby's compilation of Prism's AST works.
+
+As described in the [build system documentation](build_system.md), there is a "push" Webhook set up within the Prism repo triggered on each new commit to send information about the commit to [git.ruby-lang.org](https://github.com/ruby/git.ruby-lang.org). This in turn runs [a script](https://github.com/ruby/ruby/blob/master/tool/sync_default_gems.rb) to sync over new changes in Prism to their corresponding files in Ruby. Any failures in this sync script will show alerts in the #alerts-sync channel in the RubyLang Slack. The result of this step is that files are synced from Prism into ruby/ruby for its use. It is also worth noting that [`common.mk`](https://github.com/ruby/ruby/blob/master/common.mk) contains a list of Prism files which it needs to correctly compile. If there are new Prism files added, this file should also be updated.
+
+ruby/ruby uses the Prism code to generate an AST from which it can generate instruction sequences. Compilation in ruby/ruby has three main steps:
+
+1. Compute an AST
+
+Syncing over the Prism code allows ruby/ruby to compute the AST using Prism. It currently does this within [`iseq.c`](https://github.com/ruby/ruby/blob/master/iseq.c) using the `pm_parser_init` fuction.
+
+2. Run a first pass of compilation
+
+Once the AST has been created, it is recursively descended in order to compute the appropriate instruction sequences. This is the crux of compilation, and we go into more detail about nuances in the following paragraphs.
+
+The code for this step is almost exclusively in [`prism_compile.c`](https://github.com/ruby/ruby/blob/master/prism_compile.c). The main function used for compilation is `pm_compile_node` which is essentially a huge switch statement over practically every node type which computes the appropriate instruction sequences for that node type. There are several convenience helpers, such as `PM_COMPILE`, `PM_COMPILE_POPPED`, `PM_COMPILE_NOT_POPPED` which all call into the `pm_compile_node` function.
+
+There are also several functions, like `parse_string`, `parse_integer` which consume Prism nodes and return CRuby values. These are all called for their relevant types within the big switch statement.
+
+The Prism compiler also uses a concept of "scope nodes" which are not standard Prism nodes in the AST, but instead nodes constructed within the compiler for the sole purpose of making compilation easier. Scope nodes are defined in [`prism_compile.h`](https://github.com/ruby/ruby/blob/master/prism_compile.h) and store information such as locals, local table size, local depth offset and the index lookup tables. Scope nodes can be generated for node types which have their own "scope".
+
+3. Run an optimization pass of compilation
+
+After the instruction sequences are initially computed, there is an existing (non-Prism based) optimization pass of the instruction sequences. There are several optimizations currently inlined into step 2, however, most of them happen in this step. Specifically, any peephole optimizations happen in this step. By the end of step 2, however, the instruction sequences take the same form regardless of if the initial AST was generated by Prism or not. Therefore, step 3 is agnostic to the parser, and should not require any Prism specific code.

data/docs/design.md

@@ -0,0 +1,53 @@
+# Design
+
+There are three overall goals for this project:
+
+* to provide a documented and maintainable parser
+* to provide an error-tolerant parser suitable for use in an IDE
+* to provide a portable parser that can be used in projects that don't link against CRuby
+
+The design of the parser is based around these main goals.
+
+## Structure
+
+The first piece to understand about the parser is the design of its syntax tree. This is documented in `config.yml`. Every token and node is defined in that file, along with comments about where they are found in what kinds of syntax. This file is used to template out a lot of different files, all found in the `templates` directory. The `templates/template.rb` script performs the templating and outputs all files matching the directory structure found in the templates directory.
+
+The templated files contain all of the code required to allocate and initialize nodes, pretty print nodes, and serialize nodes. This means for the most part, you will only need to then hook up the parser to call the templated functions to create the nodes in the correct position. That means editing the parser itself, which is housed in `prism.c`.
+
+## Pratt parsing
+
+In order to provide the best possible error tolerance, the parser is hand-written. It is structured using Pratt parsing, a technique developed by Vaughan Pratt back in the 1970s. Below are a bunch of links to articles and papers that explain Pratt parsing in more detail.
+
+* https://web.archive.org/web/20151223215421/http://hall.org.ua/halls/wizzard/pdf/Vaughan.Pratt.TDOP.pdf
+* https://tdop.github.io/
+* https://journal.stuffwithstuff.com/2011/03/19/pratt-parsers-expression-parsing-made-easy/
+* https://matklad.github.io/2020/04/13/simple-but-powerful-pratt-parsing.html
+* https://chidiwilliams.com/post/on-recursive-descent-and-pratt-parsing/
+
+You can find most of the functions that correspond to constructs in the Pratt parsing algorithm in `prism.c`. As a couple of examples:
+
+* `parse` corresponds to the `parse_expression` function
+* `nud` (null denotation) corresponds to the `parse_expression_prefix` function
+* `led` (left denotation) corresponds to the `parse_expression_infix` function
+* `lbp` (left binding power) corresponds to accessing the `left` field of an element in the `binding_powers` array
+* `rbp` (right binding power) corresponds to accessing the `right` field of an element in the `binding_powers` array
+
+## Portability
+
+In order to enable using this parser in other projects, the parser is written in C99, and uses only the standard library. This means it can be embedded in most any other project without having to link against CRuby. It can be used directly through its C API to access individual fields, or it can used to parse a syntax tree and then serialize it to a single blob. For more information on serialization, see the [docs/serialization.md](serialization.md) file.
+
+## Error tolerance
+
+The design of the error tolerance of this parser is still very much in flux. We are experimenting with various approaches as the parser is being developed to try to determine the best approach. Below are a bunch of links to articles and papers that explain error tolerance in more detail, as well as document some of the approaches that we're evaluating.
+
+* https://tratt.net/laurie/blog/2020/automatic_syntax_error_recovery.html
+* https://diekmann.uk/diekmann_phd.pdf
+* https://eelcovisser.org/publications/2012/JongeKVS12.pdf
+* https://www.antlr.org/papers/allstar-techreport.pdf
+* https://github.com/microsoft/tolerant-php-parser/blob/main/docs/HowItWorks.md
+
+Currently, there are a couple of mechanisms for error tolerance that are in place:
+
+* If the parser expects a token in a particular position (for example the `in` keyword in a for loop or the `{` after `BEGIN` or `END`) then it will insert a missing token if one can't be found and continue parsing.
+* If the parser expects an expression in a particular position but encounters a token that can't be used as that expression, it checks up the stack to see if that token would close out a parent node. If so, it will close out all of its parent nodes using missing nodes wherever necessary and continue parsing.
+* If the parser cannot understand a token in any capacity, it will skip past the token.

data/docs/encoding.md

@@ -0,0 +1,121 @@
+# Encoding
+
+When parsing a Ruby file, there are times when the parser must parse identifiers. Identifiers are names of variables, methods, classes, etc. To determine the start of an identifier, the parser must be able to tell if the subsequent bytes form an alphabetic character. To determine the rest of the identifier, the parser must look forward through all alphanumeric characters.
+
+Determining if a set of bytes comprise an alphabetic or alphanumeric character is encoding-dependent. By default, the parser assumes that all source files are encoded UTF-8. If the file is not encoded in UTF-8, it must be encoded using an encoding that is "ASCII compatible" (i.e., all of the codepoints below 128 match the corresponding codepoints in ASCII and the minimum number of bytes required to represent a codepoint is 1 byte).
+
+If the file is not encoded in UTF-8, the user must specify the encoding in a "magic" comment at the top of the file. The comment looks like:
+
+```ruby
+# encoding: iso-8859-9
+```
+
+The key of the comment can be either "encoding" or "coding". The value of the comment must be a string that is a valid encoding name. The encodings that prism supports by default are:
+
+* `ASCII-8BIT`
+* `Big5`
+* `Big5-HKSCS`
+* `Big5-UAO`
+* `CESU-8`
+* `CP51932`
+* `CP850`
+* `CP852`
+* `CP855`
+* `CP949`
+* `CP950`
+* `CP951`
+* `Emacs-Mule`
+* `EUC-JP`
+* `eucJP-ms`
+* `EUC-JIS-2004`
+* `EUC-KR`
+* `EUC-TW`
+* `GB12345`
+* `GB18030`
+* `GB1988`
+* `GB2312`
+* `GBK`
+* `IBM437`
+* `IBM720`
+* `IBM737`
+* `IBM775`
+* `IBM852`
+* `IBM855`
+* `IBM857`
+* `IBM860`
+* `IBM861`
+* `IBM862`
+* `IBM863`
+* `IBM864`
+* `IBM865`
+* `IBM866`
+* `IBM869`
+* `ISO-8859-1`
+* `ISO-8859-2`
+* `ISO-8859-3`
+* `ISO-8859-4`
+* `ISO-8859-5`
+* `ISO-8859-6`
+* `ISO-8859-7`
+* `ISO-8859-8`
+* `ISO-8859-9`
+* `ISO-8859-10`
+* `ISO-8859-11`
+* `ISO-8859-13`
+* `ISO-8859-14`
+* `ISO-8859-15`
+* `ISO-8859-16`
+* `KOI8-R`
+* `KOI8-U`
+* `macCentEuro`
+* `macCroatian`
+* `macCyrillic`
+* `macGreek`
+* `macIceland`
+* `MacJapanese`
+* `macRoman`
+* `macRomania`
+* `macThai`
+* `macTurkish`
+* `macUkraine`
+* `Shift_JIS`
+* `SJIS-DoCoMo`
+* `SJIS-KDDI`
+* `SJIS-SoftBank`
+* `stateless-ISO-2022-JP`
+* `stateless-ISO-2022-JP-KDDI`
+* `TIS-620`
+* `US-ASCII`
+* `UTF-8`
+* `UTF8-MAC`
+* `UTF8-DoCoMo`
+* `UTF8-KDDI`
+* `UTF8-SoftBank`
+* `Windows-1250`
+* `Windows-1251`
+* `Windows-1252`
+* `Windows-1253`
+* `Windows-1254`
+* `Windows-1255`
+* `Windows-1256`
+* `Windows-1257`
+* `Windows-1258`
+* `Windows-31J`
+* `Windows-874`
+
+For each of these encodings, prism provides functions for checking if the subsequent bytes can be interpreted as a character, and then if that character is alphabetic, alphanumeric, or uppercase.
+
+## Getting notified when the encoding changes
+
+You may want to get notified when the encoding changes based on the result of parsing an encoding comment. We use this internally for our `lex` function in order to provide the correct encodings for the tokens that are returned. For that you can register a callback with `pm_parser_register_encoding_changed_callback`. The callback will be called with a pointer to the parser. The encoding can be accessed through `parser->encoding`.
+
+```c
+// When the encoding that is being used to parse the source is changed by prism,
+// we provide the ability here to call out to a user-defined function.
+typedef void (*pm_encoding_changed_callback_t)(pm_parser_t *parser);
+
+// Register a callback that will be called whenever prism changes the encoding
+// it is using to parse based on the magic comment.
+PRISM_EXPORTED_FUNCTION void
+pm_parser_register_encoding_changed_callback(pm_parser_t *parser, pm_encoding_changed_callback_t callback);
+```

data/docs/fuzzing.md

@@ -0,0 +1,88 @@
+# Fuzzing
+
+We use fuzzing to test the various entrypoints to the library. The fuzzer we use is [AFL++](https://aflplus.plus). All files related to fuzzing live within the `fuzz` directory, which has the following structure:
+
+```
+fuzz
+├── corpus
+│   ├── parse             fuzzing corpus for parsing (a symlink to our fixtures)
+│   └── regexp            fuzzing corpus for regexp
+├── dict                  a AFL++ dictionary containing various tokens
+├── docker
+│   └── Dockerfile        for building a container with the fuzzer toolchain
+├── fuzz.c                generic entrypoint for fuzzing
+├── heisenbug.c           entrypoint for reproducing a crash or hang
+├── parse.c               fuzz handler for parsing
+├── parse.sh              script to run parsing fuzzer
+├── regexp.c              fuzz handler for regular expression parsing
+├── regexp.sh             script to run regexp fuzzer
+└── tools
+    ├── backtrace.sh      generates backtrace files for a crash directory
+    └── minimize.sh       generates minimized crash or hang files
+```
+
+## Usage
+
+There are currently three fuzzing targets
+
+- `pm_serialize_parse` (parse)
+- `pm_regexp_named_capture_group_names` (regexp)
+
+Respectively, fuzzing can be performed with
+
+```
+make fuzz-run-parse
+make fuzz-run-regexp
+```
+
+To end a fuzzing job, interrupt with CTRL+C. To enter a container with the fuzzing toolchain and debug utilities, run
+
+```
+make fuzz-debug
+```
+
+# Out-of-bounds reads
+
+Currently, encoding functionality implementing the `pm_encoding_t` interface can read outside of inputs. For the time being, ASAN instrumentation is disabled for functions from src/enc. See `fuzz/asan.ignore`.
+
+To disable ASAN read instrumentation globally, use the `FUZZ_FLAGS` environment variable e.g.
+
+```
+FUZZ_FLAGS="-mllvm -asan-instrument-reads=false" make fuzz-run-parse
+```
+
+Note, that this may make reproducing bugs difficult as they may depend on memory outside of the input buffer. In that case, try
+
+```
+make fuzz-debug # enter the docker container with build tools
+make build/fuzz.heisenbug.parse # or .regexp
+./build/fuzz.heisenbug.parse path-to-problem-input
+```
+
+# Triaging Crashes and Hangs
+
+Triaging crashes and hangs is easier when the inputs are as short as possible. In the fuzz container, an entire crash or hang directory can be minimized using
+
+```
+./fuzz/tools/minimize.sh <directory>
+```
+
+e.g.
+```
+./fuzz/tools/minimize.sh fuzz/output/parse/default/crashes
+```
+
+This may take a long time. In the crash/hang directory, for each input file there will appear a minimized version with the extension `.min` appended.
+
+Backtraces for crashes (not hangs) can be generated en masse with
+
+```
+./fuzz/tools/backtrace.sh <directory>
+```
+
+Files with basename equal to the input file name with extension `.bt` will be created e.g.
+
+```
+id:000000,sig:06,src:000006+000190,time:8480,execs:18929,op:splice,rep:4
+id:000000,sig:06,src:000006+000190,time:8480,execs:18929,op:splice,rep:4.bt
+```

data/docs/heredocs.md

@@ -0,0 +1,36 @@
+# Heredocs
+
+Heredocs are one of the most complicated pieces of this parser. There are many different forms, there can be multiple open at the same time, and they can be nested. In order to support parsing them, we keep track of a lot of metadata. Below is a basic overview of how it works.
+
+## 1. Lexing the identifier
+
+When a heredoc identifier is encountered in the regular process of lexing, we push the `PM_LEX_HEREDOC` mode onto the stack with the following metadata:
+
+* `ident_start`: A pointer to the start of the identifier for the heredoc. We need this to match against the end of the heredoc.
+* `ident_length`: The length of the identifier for the heredoc. We also need this to match.
+* `next_start`: A pointer to the place in source that the parser should resume lexing once it has completed this heredoc.
+
+We also set the special `parser.next_start` field which is a pointer to the place in the source where we should start lexing the next token. This is set to the pointer of the character immediately following the next newline.
+
+Note that if the `parser.heredoc_end` field is already set, then it means we have already encountered a heredoc on this line. In that case the `parser.next_start` field will be set to the `parser.heredoc_end` field. This is because we want to skip past the previous heredocs on this line and instead lex the body of this heredoc.
+
+## 2. Lexing the body
+
+The next time the lexer is asked for a token, it will be in the `PM_LEX_HEREDOC` mode. In this mode we are lexing the body of the heredoc. It will start by checking if the `next_start` field is set. If it is, then this is the first token within the body of the heredoc so we'll start lexing from there. Otherwise we'll start lexing from the end of the previous token.
+
+Lexing these fields is extremely similar to lexing an interpolated string. The only difference is that we also do an additional check at the beginning of each line to check if we have hit the terminator.
+
+## 3. Lexing the terminator
+
+On every newline within the body of a heredoc, we check to see if it matches the terminator followed by a newline or a carriage return and a newline. If it does, then we pop the lex mode off the stack and set a couple of fields on the parser:
+
+* `next_start`: This is set to the value that we previously stored on the heredoc to indicate where the lexer should resume lexing when it is done with this heredoc.
+* `heredoc_end`: This is set to the end of the heredoc. When a newline character is found, this indicates that the lexer should skip past to this next point.
+
+## 4. Lexing the rest of the line
+
+Once the heredoc has been lexed, the lexer will resume lexing from the `next_start` field. Lexing will continue until the next newline character. When the next newline character is found, it will check to see if the `heredoc_end` field is set. If it is it will skip to that point, unset the field, and continue lexing.
+
+## Compatibility with Ripper
+
+The order in which tokens are emitted is different from that of Ripper. Ripper emits each token in the file in the order in which it appears. prism instead will emit the tokens that makes the most sense for the lexer, using the process described above. Therefore to line things up, `Prism.lex_compat` will shuffle the tokens around to match Ripper's output.

data/docs/javascript.md

@@ -0,0 +1,118 @@
+# JavaScript
+
+Prism provides bindings to JavaScript out of the box.
+
+## Node
+
+To use the package from node, install the `@ruby/prism` dependency:
+
+```sh
+npm install @ruby/prism
+```
+
+Then import the package:
+
+```js
+import { loadPrism } from "@ruby/prism";
+```
+
+Then call the load function to get a parse function:
+
+```js
+const parse = await loadPrism();
+```
+
+## Browser
+
+To use the package from the browser, you will need to do some additional work. The [javascript/example.html](../javascript/example.html) file shows an example of running Prism in the browser. You will need to instantiate the WebAssembly module yourself and then pass it to the `parsePrism` function.
+
+First, get a shim for WASI since not all browsers support it yet.
+
+```js
+import { WASI } from "https://unpkg.com/@bjorn3/browser_wasi_shim@latest/dist/index.js";
+```
+
+Next, import the `parsePrism` function from `@ruby/prism`, either through a CDN or by bundling it with your application.
+
+```js
+import { parsePrism } from "https://unpkg.com/@ruby/prism@latest/src/parsePrism.js";
+```
+
+Next, fetch and instantiate the WebAssembly module. You can access it through a CDN or by bundling it with your application.
+
+```js
+const wasm = await WebAssembly.compileStreaming(fetch("https://unpkg.com/@ruby/prism@latest/src/prism.wasm"));
+```
+
+Next, instantiate the module and initialize WASI.
+
+```js
+const wasi = new WASI([], [], []);
+const instance = await WebAssembly.instantiate(wasm, { wasi_snapshot_preview1: wasi.wasiImport });
+wasi.initialize(instance);
+```
+
+Finally, you can create a function that will parse a string of Ruby code.
+
+```js
+function parse(source) {
+  return parsePrism(instance.exports, source);
+}
+```
+
+## API
+
+Now that we have access to a `parse` function, we can use it to parse Ruby code:
+
+```js
+const parseResult = parse("1 + 2");
+```
+
+A ParseResult object is very similar to the Prism::ParseResult object from Ruby. It has the same properties: `value`, `comments`, `magicComments`, `errors`, and `warnings`. Here we can serialize the AST to JSON.
+
+```js
+console.log(JSON.stringify(parseResult.value, null, 2));
+```
+
+## Visitors
+
+Prism allows you to traverse the AST of parsed Ruby code using visitors.
+
+Here's an example of a custom `FooCalls` visitor:
+
+```js
+import { loadPrism, Visitor } from "@ruby/prism"
+
+const parse = await loadPrism();
+const parseResult = parse("foo()");
+
+class FooCalls extends Visitor {
+  visitCallNode(node) {
+    if (node.name === "foo") {
+      // Do something with the node
+    }
+
+    // Call super so that the visitor continues walking the tree
+    super.visitCallNode(node);
+  }
+}
+
+const fooVisitor = new FooCalls();
+
+parseResult.value.accept(fooVisitor);
+```
+
+## Building
+
+To build the WASM package yourself, first obtain a copy of `wasi-sdk`. You can retrieve this here: <https://github.com/WebAssembly/wasi-sdk>. Next, run:
+
+```sh
+make wasm WASI_SDK_PATH=path/to/wasi-sdk
+```
+
+This will generate `javascript/src/prism.wasm`. From there, you can run the tests to verify everything was generated correctly.
+
+```sh
+cd javascript
+node test
+```