yarp 0.8.0 → 0.10.0

data/docs/configuration.md

@@ -7,6 +7,7 @@ A lot of code in YARP's repository is templated from a single configuration file
 * `java/org/yarp/AbstractNodeVisitor.java` - for defining the visitor interface for the nodes in Java
 * `java/org/yarp/Loader.java` - for defining how to deserialize the nodes in Java
 * `java/org/yarp/Nodes.java` - for defining the nodes in Java
+* `lib/yarp/mutation_visitor.rb` - for defining the mutation visitor for the nodes in Ruby
 * `lib/yarp/node.rb` - for defining the nodes in Ruby
 * `lib/yarp/serialize.rb` - for defining how to deserialize the nodes in Ruby
 * `src/node.c` - for defining how to free the nodes in C and calculate the size in memory in C

data/docs/encoding.md

@@ -61,22 +61,22 @@ typedef struct {
     // Return the number of bytes that the next character takes if it is valid
     // in the encoding. Does not read more than n bytes. It is assumed that n is
     // at least 1.
-    size_t (*char_width)(const char *c, ptrdiff_t n);
+    size_t (*char_width)(const uint8_t *b, ptrdiff_t n);
 
     // Return the number of bytes that the next character takes if it is valid
     // in the encoding and is alphabetical. Does not read more than n bytes. It
     // is assumed that n is at least 1.
-    size_t (*alpha_char)(const char *c, ptrdiff_t n);
+    size_t (*alpha_char)(const uint8_t *b, ptrdiff_t n);
 
     // Return the number of bytes that the next character takes if it is valid
     // in the encoding and is alphanumeric. Does not read more than n bytes. It
     // is assumed that n is at least 1.
-    size_t (*alnum_char)(const char *c, ptrdiff_t n);
+    size_t (*alnum_char)(const uint8_t *b, ptrdiff_t n);
 
     // Return true if the next character is valid in the encoding and is an
     // uppercase character. Does not read more than n bytes. It is assumed that
     // n is at least 1.
-    bool (*isupper_char)(const char *c, ptrdiff_t n);
+    bool (*isupper_char)(const uint8_t *b, ptrdiff_t n);
 
     // The name of the encoding. This should correspond to a value that can be
     // passed to Encoding.find in Ruby.
@@ -90,7 +90,7 @@ typedef struct {
 // the ability here to call out to a user-defined function to get an encoding
 // struct. If the function returns something that isn't NULL, we set that to
 // our encoding and use it to parse identifiers.
-typedef yp_encoding_t *(*yp_encoding_decode_callback_t)(yp_parser_t *parser, const char *name, size_t width);
+typedef yp_encoding_t *(*yp_encoding_decode_callback_t)(yp_parser_t *parser, const uint8_t *name, size_t width);
 
 // Register a callback that will be called when YARP encounters a magic comment
 // with an encoding referenced that it doesn't understand. The callback should

data/docs/ruby_api.md

@@ -20,4 +20,6 @@ The full API is documented below.
 * `YARP.lex_file(filepath)` - parse the tokens corresponding to the given source file and return them as an array within a parse result
 * `YARP.parse(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result
 * `YARP.parse_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result
+* `YARP.parse_lex(source)` - parse the syntax tree corresponding to the given source string and return it within a parse result, along with the tokens
+* `YARP.parse_lex_file(filepath)` - parse the syntax tree corresponding to the given source file and return it within a parse result, along with the tokens
 * `YARP.load(source, serialized)` - load the serialized syntax tree using the source as a reference into a syntax tree

data/docs/serialization.md

@@ -73,7 +73,7 @@ The header is structured like the following table:
 | varint | content pool size |
 
 After the header comes the body of the serialized string.
-The body consistents of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
+The body consists of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
 Each node is structured like the following table:
 
 | # bytes | field |
@@ -130,14 +130,14 @@ void yp_buffer_free(yp_buffer_t *);
 
 // Parse and serialize the AST represented by the given source to the given
 // buffer.
-void yp_parse_serialize(const char *, size_t, yp_buffer_t *, const char *);
+void yp_parse_serialize(const uint8_t *source, size_t length, yp_buffer_t *buffer, const char *metadata);
 ```
 
 Typically you would use a stack-allocated `yp_buffer_t` and call `yp_parse_serialize`, as in:
 
 ```c
 void
-serialize(const char *source, size_t length) {
+serialize(const uint8_t *source, size_t length) {
   yp_buffer_t buffer;
   if (!yp_buffer_init(&buffer)) return;
 

data/docs/testing.md

@@ -12,9 +12,9 @@ These test specific YARP implementation details like comments, errors, and regul
 
 ### Snapshot tests
 
-Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/snapshots/` directory. For example, `test/fixtures/strings.txt` has a corresponding `test/snapshots/strings.txt`.
+Snapshot tests ensure that parsed output is equivalent to previous parsed output. There are many categorized examples of valid syntax within the `test/yarp/fixtures/` directory. When the test suite runs, it will parse all of this syntax, and compare it against corresponding files in the `test/yarp/snapshots/` directory. For example, `test/yarp/fixtures/strings.txt` has a corresponding `test/yarp/snapshots/strings.txt`.
 
-If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/snapshots/` directory, one will be created so that it exists for the next test run.
+If the parsed files do not match, it will raise an error. If there is not a corresponding file in the `test/yarp/snapshots/` directory, one will be created so that it exists for the next test run.
 
 ### Testing against repositories