yarp 0.6.0 → 0.8.0

data/docs/build_system.md

@@ -16,8 +16,6 @@ The main solution for the second point seems a Makefile, otherwise many of the u
 ## General Design
 
 1. Templates are generated by `templates/template.rb`
-2. `autoconf` creates `./configure` and `autoheader` creates `config.h.in` (both files are platform-independent)
-3. `./configure` creates `include/yarp/config.h` (which contains `HAVE_*` macros, platform-specific) and the `Makefile`
 4. The `Makefile` compiles both `librubyparser.a` and `librubyparser.{so,dylib,dll}` from the `src/**/*.c` and `include/**/*.h` files
 5. The `Rakefile` `:compile` task ensures the above prerequisites are done, then calls `make`,
   and uses `Rake::ExtensionTask` to compile the C extension (using its `extconf.rb`), which uses `librubyparser.a`
@@ -36,14 +34,13 @@ loaded per process (i.e., at most one version of the yarp *gem* loaded in a proc
 
 ### Building the yarp gem by `gem install/bundle install`
 
-The gem contains the pre-generated templates, as well as `configure` and `config.h.in`
+The gem contains the pre-generated templates.
 When installing the gem, `extconf.rb` is used and that:
-* runs `./configure` which creates the `Makefile` and `include/yarp/config.h`
 * runs `make build/librubyparser.a`
 * compiles the C extension with mkmf
 
 When installing the gem on JRuby and TruffleRuby, no C extension is built, so instead of the last step,
-there is Ruby code using Fiddle which uses `librubyparser.{so,dylib,dll}`
+there is Ruby code using FFI which uses `librubyparser.{so,dylib,dll}`
 to implement the same methods as the C extension, but using serialization instead of many native calls/accesses
 (JRuby does not support C extensions, serialization is faster on TruffleRuby than the C extension).
 
@@ -51,7 +48,6 @@ to implement the same methods as the C extension, but using serialization instea
 
 The same as above, except the `extconf.rb` additionally runs first:
 * `templates/template.rb` to generate the templates
-* `autoconf` and `autoheader` to generate `configure` and `config.h.in`
 
 Because of course those files are not part of the git repository.
 
@@ -61,21 +57,14 @@ Because of course those files are not part of the git repository.
 
 The script generates the templates when importing.
 
-`include/yarp/config.h` is replaced by `#include "ruby/config.h"`.
-It is assumed that CRuby's `./configure` is a superset of YARP's configure checks.
-
-YARP's `autotools` is not used at all in CRuby and in fact YARP's `Makefile` is not used either.
-Instead, CRuby's `autotools` setup is used, and `CRuby`'s Makefiles are used.
+YARP's `Makefile` is not used at all in CRuby. Instead, CRuby's `Makefile` is used.
 
 ### Building YARP as part of TruffleRuby
 
 [This script](https://github.com/oracle/truffleruby/blob/master/tool/import-yarp.sh) imports YARP sources in TruffleRuby.
 The script generates the templates when importing.
-It also generates `configure` and `config.h.in` (to avoid needing `autotools` on every machine building TruffleRuby).
 
-Then when `mx build` builds TruffleRuby and the `yarp` mx project inside, it:
-* runs `./configure`
-* runs `make`
+Then when `mx build` builds TruffleRuby and the `yarp` mx project inside, it runs `make`.
 
 Then the `yarp bindings` mx project is built, which contains the [bindings](https://github.com/oracle/truffleruby/blob/master/src/main/c/yarp_bindings/src/yarp_bindings.c)
 and links to `librubyparser.a` (to avoid exporting symbols, so no conflict when installing the yarp gem).

data/docs/building.md

@@ -1,6 +1,7 @@
 # Building
 
 The following describes how to build YARP from source.
+This comes directly from the [Makefile](../Makefile).
 
 ## Common
 
@@ -13,11 +14,6 @@ The following flags should be used to compile YARP:
 * `-Werror` - Treat warnings as errors
 * `-fvisibility=hidden` - Hide all symbols by default
 
-The following flags can be used to compile YARP:
-
-* `-DHAVE_MMAP` - Should be passed if the system has the `mmap` function
-* `-DHAVE_SNPRINTF` - Should be passed if the system has the `snprintf` function
-
 ## Shared
 
 If you want to build YARP as a shared library and link against it, you should compile with:

data/docs/encoding.md

@@ -39,6 +39,7 @@ The key of the comment can be either "encoding" or "coding". The value of the co
 * `sjis`
 * `us-ascii`
 * `utf-8`
+* `utf8-mac`
 * `windows-31j`
 * `windows-1251`
 * `windows-1252`

data/docs/{extension.md → ruby_api.md}

@@ -1,6 +1,6 @@
-# Extension
+# Ruby API
 
-Part of this parser project is a native extension that provides a Ruby API that wraps calls to the C API. This allows you to invoke parsing from Ruby code. It also provides a Ruby API for accessing the syntax tree.
+The `yarp` gem provides a Ruby API for accessing the syntax tree.
 
 For the most part, the API for accessing the tree mirrors that found in the [Syntax Tree](https://github.com/ruby-syntax-tree/syntax_tree) project. This means:
 
@@ -8,7 +8,9 @@ For the most part, the API for accessing the tree mirrors that found in the [Syn
 * Nodes in the tree respond to named methods for accessing their children as well as `#child_nodes`
 * Nodes respond to the pattern matching interfaces `#deconstruct` and `#deconstruct_keys`
 
-Every entry in `config.yml` will generate a Ruby class as well as the code that builds the nodes themselves. Creating a syntax tree involves calling one of the class methods on the `YARP` module. The full API is documented below.
+Every entry in `config.yml` will generate a Ruby class as well as the code that builds the nodes themselves.
+Creating a syntax tree involves calling one of the class methods on the `YARP` module.
+The full API is documented below.
 
 ## API
 
@@ -18,3 +20,4 @@ Every entry in `config.yml` will generate a Ruby class as well as the code that
 * `YARP.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
 * `YARP.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
 * `YARP.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
+* `YARP.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree

data/docs/serialization.md

@@ -1,10 +1,58 @@
 # Serialization
 
-YARP ships with the ability to serialize a syntax tree to a single string. The string can then be deserialized back into a syntax tree using a language other than C. This is useful for using the parsing logic in other tools without having to write a parser in that language. The syntax tree still requires a copy of the original source, as for the most part it just contains byte offsets into the source string.
+YARP ships with the ability to serialize a syntax tree to a single string.
+The string can then be deserialized back into a syntax tree using a language other than C.
+This is useful for using the parsing logic in other tools without having to write a parser in that language.
+The syntax tree still requires a copy of the original source, as for the most part it just contains byte offsets into the source string.
+
+## Types
+
+Let us define some simple types for readability.
+
+### varint
+
+A variable-length integer with the value fitting in `uint32_t` using between 1 and 5 bytes, using the [LEB128](https://en.wikipedia.org/wiki/LEB128) encoding.
+This drastically cuts down on the size of the serialized string, especially when the source file is large.
+
+### string
+
+| # bytes | field |
+| --- | --- |
+| varint | the length of the string in bytes |
+| ... | the string bytes |
+
+### location
+
+| # bytes | field |
+| --- | --- |
+| varint | byte offset into the source string where this location begins |
+| varint | length of the location in bytes in the source string |
+
+### comment
+
+The comment type is one of:
+* 0=`INLINE` (`# comment`)
+* 1=`EMBEDDED_DOCUMENT` (`=begin`/`=end`)
+* 2=`__END__` (after `__END__`)
+
+| # bytes | field |
+| --- | --- |
+| `1` | comment type |
+| location | the location in the source of this comment |
+
+### diagnostic
+
+| # bytes | field |
+| --- | --- |
+| string | diagnostic message (ASCII-only characters) |
+| location | the location in the source this diagnostic applies to |
 
 ## Structure
 
-The serialized string representing the syntax tree is composed of three parts: the header, the body, and the constant pool. The header contains information like the version of YARP that serialized the tree. The body contains the actual nodes in the tree. The constant pool contains constants that were interned while parsing.
+The serialized string representing the syntax tree is composed of three parts: the header, the body, and the constant pool.
+The header contains information like the version of YARP that serialized the tree.
+The body contains the actual nodes in the tree.
+The constant pool contains constants that were interned while parsing.
 
 The header is structured like the following table:
 
@@ -14,32 +62,28 @@ The header is structured like the following table:
 | `1` | major version number |
 | `1` | minor version number |
 | `1` | patch version number |
-| varint | the length of the encoding name |
 | string | the encoding name |
+| varint | number of comments |
+| comment* | comments |
 | varint | number of errors |
-| varint | byte length of error |
-| string | error string, as byte[] in source encoding |
-| varint | location in the source code - start |
-| varint | location in the source code - length |
-| ... | more errors |
+| diagnostic* | errors |
 | varint | number of warnings |
-| varint | byte length of warning |
-| string | warning string, as byte[] in source encoding |
-| varint | location in the source code - start |
-| varint | location in the source code - length |
-| ... | more warnings |
+| diagnostic* | warnings |
 | `4` | content pool offset |
 | varint | content pool size |
 
-After the header comes the body of the serialized string. The body consistents of a sequence of nodes that is built using a prefix traversal order of the syntax tree. Each node is structured like the following table:
+After the header comes the body of the serialized string.
+The body consistents of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
+Each node is structured like the following table:
 
 | # bytes | field |
 | --- | --- |
 | `1` | node type |
-| varint | byte offset into the source string where this node begins |
-| varint | length of the node in bytes in the source string |
+| location | node location |
 
-Each node's child is then appended to the serialized string. The child node types can be determined by referencing `config.yml`. Depending on the type of child node, it could take a couple of different forms, described below:
+Each node's child is then appended to the serialized string.
+The child node types can be determined by referencing `config.yml`.
+Depending on the type of child node, it could take a couple of different forms, described below:
 
 * `node` - A child node that is a node itself. This is structured just as like parent node.
 * `node?` - A child node that is optionally present. If the node is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just as like parent node.
@@ -52,7 +96,10 @@ Each node's child is then appended to the serialized string. The child node type
 * `location[]` - A child node that is an array of locations. This is structured as a `4` byte length, followed by the locations themselves.
 * `uint32` - A child node that is a 32-bit unsigned integer. This is structured as a variable-length integer.
 
-After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Each constant is structured as:
+After the syntax tree, the content pool is serialized.
+This is a list of constants that were referenced from within the tree.
+The content pool begins at the offset specified in the header.
+Each constant is structured as:
 
 | # bytes | field |
 | --- | --- |
@@ -61,10 +108,6 @@ After the syntax tree, the content pool is serialized. This is a list of constan
 
 At the end of the serialization, the buffer is null terminated.
 
-## Variable-length integers
-
-Variable-length integers are used throughout the serialized format, using the [LEB128](https://en.wikipedia.org/wiki/LEB128) encoding. This drastically cuts down on the size of the serialized string, especially when the source file is large.
-
 ## APIs
 
 The relevant APIs and struct definitions are listed below:
@@ -105,7 +148,10 @@ serialize(const char *source, size_t length) {
 }
 ```
 
-The final argument to `yp_parse_serialize` controls the metadata of the source. This includes the filepath that the source is associated with, and any nested local variables scopes that are necessary to properly parse the file (in the case of parsing an `eval`). The metadata is a serialized format itself, and is structured as follows:
+The final argument to `yp_parse_serialize` controls the metadata of the source.
+This includes the filepath that the source is associated with, and any nested local variables scopes that are necessary to properly parse the file (in the case of parsing an `eval`).
+Note that no `varint` are used here to make it easier to produce the metadata for the caller, and also serialized size is less important here.
+The metadata is a serialized format itself, and is structured as follows:
 
 | # bytes | field |
 | --- | --- |
@@ -127,4 +173,5 @@ Each local variable within each scope is encoded as:
 | `4` | the size of the local variable name |
 | | the local variable name |
 
-The metadata can be `NULL` (as seen in the example above). If it is not null, then a minimal metadata string would be `"\0\0\0\0\0\0\0\0"` which would use 4 bytes to indicate an empty filepath string and 4 bytes to indicate that there were no local variable scopes.
+The metadata can be `NULL` (as seen in the example above).
+If it is not null, then a minimal metadata string would be `"\0\0\0\0\0\0\0\0"` which would use 4 bytes to indicate an empty filepath string and 4 bytes to indicate that there were no local variable scopes.