yarp 0.7.0 → 0.9.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +54 -2
- data/Makefile +2 -0
- data/README.md +9 -5
- data/config.yml +160 -93
- data/docs/configuration.md +1 -0
- data/docs/ruby_api.md +2 -0
- data/docs/serialization.md +1 -1
- data/docs/testing.md +2 -2
- data/ext/yarp/api_node.c +361 -238
- data/ext/yarp/extension.c +75 -26
- data/ext/yarp/extension.h +2 -2
- data/include/yarp/ast.h +226 -175
- data/include/yarp/defines.h +5 -0
- data/include/yarp/node.h +10 -0
- data/include/yarp/unescape.h +4 -2
- data/include/yarp/util/yp_buffer.h +9 -1
- data/include/yarp/util/yp_constant_pool.h +3 -0
- data/include/yarp/util/yp_list.h +7 -7
- data/include/yarp/util/yp_newline_list.h +7 -0
- data/include/yarp/util/yp_state_stack.h +1 -1
- data/include/yarp/version.h +2 -2
- data/include/yarp.h +10 -0
- data/lib/yarp/desugar_visitor.rb +267 -0
- data/lib/yarp/ffi.rb +89 -48
- data/lib/yarp/lex_compat.rb +93 -25
- data/lib/yarp/mutation_visitor.rb +683 -0
- data/lib/yarp/node.rb +2061 -422
- data/lib/yarp/serialize.rb +162 -120
- data/lib/yarp.rb +54 -8
- data/src/node.c +360 -304
- data/src/prettyprint.c +190 -152
- data/src/serialize.c +382 -340
- data/src/token_type.c +2 -2
- data/src/unescape.c +89 -77
- data/src/util/yp_buffer.c +18 -0
- data/src/util/yp_list.c +7 -16
- data/src/util/yp_newline_list.c +10 -0
- data/src/util/yp_state_stack.c +0 -6
- data/src/yarp.c +941 -596
- data/yarp.gemspec +3 -1
- metadata +4 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: '028bafce371c205e4f9a845aa0328a8911519b064ec92be9a970c0077088ca5c'
|
4
|
+
data.tar.gz: ed653e7a3734884de2f42d1df304be7eb1532b29d7acf63214c747ee84426ec0
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e33c681424d19626c0ee1d5c21265bcdfce4ec56909a5707d4bf7025ea5356b5ac41b080a1e54a8490124065a1595bd5833af8e488558f0ff01c4f14a92d0430
|
7
|
+
data.tar.gz: d55c0483e944f9a33e76a2baeef11841da8f8bf8755512ff90497d07b0fedf2dc568f6ad8c384831b6b382b1482e34177c6dbd8314d6570542c59faca71ad23e
|
data/CHANGELOG.md
CHANGED
@@ -6,6 +6,56 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
6
6
|
|
7
7
|
## [Unreleased]
|
8
8
|
|
9
|
+
## [0.9.0] - 2023-08-25
|
10
|
+
|
11
|
+
### Added
|
12
|
+
|
13
|
+
- Regular expressions can now be bound by `\n`, `\r`, and a combination of `\r\n`.
|
14
|
+
- Strings delimited by `%`, `%q`, and `%Q` can now be bound by `\n`, `\r`, and a combination of `\r\n`.
|
15
|
+
- `IntegerNode#value` now returns the value of the integer as a Ruby `Integer`.
|
16
|
+
- `FloatNode#value` now returns the value of the float as a Ruby `Float`.
|
17
|
+
- `RationalNode#value` now returns the value of the rational as a Ruby `Rational`.
|
18
|
+
- `ImaginaryNode#value` now returns the value of the imaginary as a Ruby `Complex`.
|
19
|
+
- `ClassNode#name` is now a string that returns the name of just the class, without the namespace.
|
20
|
+
- `ModuleNode#name` is now a string that returns the name of just the module, without the namespace.
|
21
|
+
- Regular expressions and strings found after a heredoc declaration but before the heredoc body are now parsed correctly.
|
22
|
+
- The serialization API now supports shared strings, which should help reduce the size of the serialized AST.
|
23
|
+
- `*Node#copy` is introduced, which returns a copy of the node with the given overrides.
|
24
|
+
- `Location#copy` is introduced, which returns a copy of the location with the given overrides.
|
25
|
+
- `DesugarVisitor` is introduced, which provides a simpler AST for use in tools that want to process fewer node types.
|
26
|
+
- `{ClassVariable,Constant,ConstantPath,GlobalVariable,InstanceVariable,LocalVariable}TargetNode` are introduced. These nodes represent the target of writes in locations where a value cannot be provided, like a multi write or a rescue reference.
|
27
|
+
- `UntilNode#closing_loc` and `WhileNode#closing_loc` are now provided.
|
28
|
+
- `Location#join` is now provided, which joins two locations together.
|
29
|
+
- `YARP::parse_lex` and `YARP::parse_lex_file` are introduced to parse and lex in one result.
|
30
|
+
|
31
|
+
### Changed
|
32
|
+
|
33
|
+
- When there is a magic encoding comment, the encoding of the first token's source string is now properly reencoded.
|
34
|
+
- Constants followed by unary `&` are now properly parsed as a call with a passed block argument.
|
35
|
+
- Escaping multi-byte characters in a string literal will now properly escape the entire character.
|
36
|
+
- `YARP.lex_compat` now has more accurate behavior when a byte-order mark is present in the file.
|
37
|
+
- **BREAKING**: `AndWriteNode`, `OrWriteNode`, and `OperatorWriteNode` have been split back up into their `0.7.0` versions.
|
38
|
+
- We now properly support spaces between the `encoding` and `=`/`:` in a magic encoding comment.
|
39
|
+
- We now properly parse `-> foo: bar do end`.
|
40
|
+
|
41
|
+
## [0.8.0] - 2023-08-18
|
42
|
+
|
43
|
+
### Added
|
44
|
+
|
45
|
+
- Some performance improvements when converting from the C AST to the Ruby AST.
|
46
|
+
- Two rust crates have been added: `yarp-sys` and `yarp`. They are as yet unpublished.
|
47
|
+
|
48
|
+
### Changed
|
49
|
+
|
50
|
+
- Escaped newlines in strings and heredocs are now handled more correctly.
|
51
|
+
- Dedenting heredocs that result in empty string nodes will now drop those string nodes from the list.
|
52
|
+
- Beginless and endless ranges in conditional expressions now properly form a flip flop node.
|
53
|
+
- `%` at the end of files no longer crashes.
|
54
|
+
- Location information has been corrected for `if/elsif` chains that have no `else`.
|
55
|
+
- `__END__` at the very end of the file was previously parsed as an identifier, but is now correct.
|
56
|
+
- **BREAKING**: Nodes that reference `&&=`, `||=`, and other writing operators have been consolidated. Previously, they were separate individual nodes. Now they are a tree with the target being the left-hand side and the value being the right-hand side with a joining `AndWriteNode`, `OrWriteNode`, or `OperatorWriteNode` in the middle. This impacts all of the nodes that match this pattern: `{ClassVariable,Constant,ConstantPath,GlobalVariable,InstanceVariable,LocalVariable}Operator{And,Or,}WriteNode`.
|
57
|
+
- **BREAKING**: `BlockParametersNode`, `ClassNode`, `DefNode`, `LambdaNode`, `ModuleNode`, `ParenthesesNode`, and `SingletonClassNode` have had their `statements` field renamed to `body` to give a hint that it might not be a `StatementsNode` (it could also be a `BeginNode`).
|
58
|
+
|
9
59
|
## [0.7.0] - 2023-08-14
|
10
60
|
|
11
61
|
### Added
|
@@ -22,7 +72,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
22
72
|
|
23
73
|
- Autotools has been removed from the build system, so when the gem is installed it will no longer need to go through a configure step.
|
24
74
|
- The AST for `foo = *bar` has changed to have an explicit array on the right hand side, rather than a splat node. This is more consistent with how other parsers handle this.
|
25
|
-
- `RangeNodeFlags` has been renamed to `RangeFlags`.
|
75
|
+
- **BREAKING**: `RangeNodeFlags` has been renamed to `RangeFlags`.
|
26
76
|
- Unary minus on number literals is now parsed as part of the literal, rather than a call to a unary operator. This is more consistent with how other parsers handle this.
|
27
77
|
|
28
78
|
## [0.6.0] - 2023-08-09
|
@@ -31,6 +81,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
31
81
|
|
32
82
|
- 🎉 Initial release! 🎉
|
33
83
|
|
34
|
-
[unreleased]: https://github.com/ruby/yarp/compare/v0.
|
84
|
+
[unreleased]: https://github.com/ruby/yarp/compare/v0.9.0...HEAD
|
85
|
+
[0.9.0]: https://github.com/ruby/yarp/compare/v0.8.0...v0.9.0
|
86
|
+
[0.8.0]: https://github.com/ruby/yarp/compare/v0.7.0...v0.8.0
|
35
87
|
[0.7.0]: https://github.com/ruby/yarp/compare/v0.6.0...v0.7.0
|
36
88
|
[0.6.0]: https://github.com/ruby/yarp/compare/d60531...v0.6.0
|
data/Makefile
CHANGED
@@ -61,6 +61,8 @@ fuzz-docker-build: fuzz/docker/Dockerfile
|
|
61
61
|
$(Q) docker build -t yarp/fuzz fuzz/docker/
|
62
62
|
|
63
63
|
fuzz-run-%: FORCE fuzz-docker-build
|
64
|
+
$(ECHO) "generating templates"
|
65
|
+
$(Q) bundle exec rake templates
|
64
66
|
$(ECHO) "running $* fuzzer"
|
65
67
|
$(Q) docker run --rm -v $(shell pwd):/yarp yarp/fuzz /bin/bash -c "FUZZ_FLAGS=\"$(FUZZ_FLAGS)\" make build/fuzz.$*"
|
66
68
|
$(ECHO) "starting AFL++ run"
|
data/README.md
CHANGED
@@ -14,7 +14,6 @@ The repository contains the infrastructure for both a shared library (librubypar
|
|
14
14
|
│ ├── lex runs the lexer on a file or string, prints the tokens, and compares to ripper
|
15
15
|
│ └── parse runs the parser on a file or string and prints the syntax tree
|
16
16
|
├── config.yml specification for tokens and nodes in the tree
|
17
|
-
├── configure.ac configuration to generate the Makefile
|
18
17
|
├── docs documentation about the project
|
19
18
|
├── ext
|
20
19
|
│ └── yarp
|
@@ -29,6 +28,9 @@ The repository contains the infrastructure for both a shared library (librubypar
|
|
29
28
|
│ ├── yarp Ruby library files
|
30
29
|
│ └── yarp.rb main entrypoint for the Ruby library
|
31
30
|
├── rakelib various Rake tasks for the project
|
31
|
+
├── rust
|
32
|
+
│ ├── yarp Rustified crate for the shared library
|
33
|
+
│ └── yarp-sys FFI binding for Rust
|
32
34
|
├── src
|
33
35
|
│ ├── enc various encoding files
|
34
36
|
│ ├── util various utility files
|
@@ -36,8 +38,9 @@ The repository contains the infrastructure for both a shared library (librubypar
|
|
36
38
|
├── templates contains ERB templates generated by templates/template.rb
|
37
39
|
│ └── template.rb generates code from the nodes and tokens configured by config.yml
|
38
40
|
└── test
|
39
|
-
|
40
|
-
|
41
|
+
└── yarp
|
42
|
+
├── fixtures Ruby code used for testing
|
43
|
+
└── snapshots snapshots of generated syntax trees corresponding to fixtures
|
41
44
|
```
|
42
45
|
|
43
46
|
## Getting started
|
@@ -45,7 +48,7 @@ The repository contains the infrastructure for both a shared library (librubypar
|
|
45
48
|
To compile the shared library, you will need:
|
46
49
|
|
47
50
|
* A C99 compiler
|
48
|
-
* autotools
|
51
|
+
* autotools autoconf, automake, libtool)
|
49
52
|
* make
|
50
53
|
* Ruby 3.3.0-preview1 or later
|
51
54
|
|
@@ -77,10 +80,11 @@ See the [CONTRIBUTING.md](CONTRIBUTING.md) file for more information. We additio
|
|
77
80
|
* [Configuration](docs/configuration.md)
|
78
81
|
* [Design](docs/design.md)
|
79
82
|
* [Encoding](docs/encoding.md)
|
80
|
-
* [Extension](docs/extension.md)
|
81
83
|
* [Fuzzing](docs/fuzzing.md)
|
82
84
|
* [Heredocs](docs/heredocs.md)
|
83
85
|
* [Mapping](docs/mapping.md)
|
84
86
|
* [Ripper](docs/ripper.md)
|
87
|
+
* [Ruby API](docs/ruby_api.md)
|
85
88
|
* [Serialization](docs/serialization.md)
|
86
89
|
* [Testing](docs/testing.md)
|
90
|
+
|