prism 2.0.0.rc1-aarch64-linux-gnu → 2.1.0-aarch64-linux-gnu
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +61 -41
- data/CODE_OF_CONDUCT.md +1 -1
- data/LICENSE.md +1 -1
- data/Makefile +8 -8
- data/README.md +3 -1
- data/docs/fuzzing.md +5 -5
- data/docs/serialization.md +20 -6
- data/ext/prism/api_node.c +81 -81
- data/ext/prism/extension.c +54 -69
- data/ext/prism/extension.h +1 -1
- data/include/prism/arena.h +8 -0
- data/include/prism/arena.hpp +42 -0
- data/include/prism/ast.h +11 -1
- data/include/prism/ast.hpp +641 -0
- data/include/prism/buffer.h +8 -0
- data/include/prism/buffer.hpp +55 -0
- data/include/prism/comments.h +17 -0
- data/include/prism/compiler/accel.h +3 -1
- data/include/prism/compiler/branch.h +26 -0
- data/include/prism/compiler/bswap.h +46 -0
- data/include/prism/compiler/cplusplus.hpp +16 -0
- data/include/prism/compiler/endianness.h +34 -0
- data/include/prism/compiler/exceptions.hpp +17 -0
- data/include/prism/compiler/format.h +2 -0
- data/include/prism/compiler/lifetimebound.hpp +25 -0
- data/include/prism/constant_pool.h +25 -0
- data/include/prism/diagnostic.h +18 -0
- data/include/prism/errors_format.h +8 -0
- data/include/prism/internal/buffer.h +3 -0
- data/include/prism/internal/comments.h +2 -1
- data/include/prism/internal/node.h +8 -4
- data/include/prism/internal/parser.h +44 -13
- data/include/prism/internal/static_literals.h +24 -64
- data/include/prism/internal/stringy.h +1 -6
- data/include/prism/internal/xxh3.h +16 -0
- data/include/prism/json.h +8 -0
- data/include/prism/line_offset_list.h +8 -0
- data/include/prism/magic_comments.h +18 -0
- data/include/prism/node.h +55 -10
- data/include/prism/node.hpp +272 -0
- data/include/prism/node_type.hpp +38 -0
- data/include/prism/options.h +10 -1
- data/include/prism/options.hpp +43 -0
- data/include/prism/parser.h +93 -64
- data/include/prism/parser.hpp +274 -0
- data/include/prism/prettyprint.h +8 -0
- data/include/prism/serialize.h +8 -0
- data/include/prism/source.h +8 -0
- data/include/prism/stream.h +8 -0
- data/include/prism/string_query.h +19 -0
- data/include/prism/stringy.h +16 -27
- data/include/prism/stringy.hpp +31 -0
- data/include/prism/version.h +10 -2
- data/include/prism.h +0 -8
- data/include/prism.hpp +24 -0
- data/lib/prism/3.3/prism.so +0 -0
- data/lib/prism/3.4/prism.so +0 -0
- data/lib/prism/4.0/prism.so +0 -0
- data/lib/prism/ffi.rb +6 -0
- data/lib/prism/node.rb +736 -0
- data/lib/prism/parse_result.rb +23 -8
- data/lib/prism/serialize.rb +2568 -3287
- data/lib/prism/string_query.rb +8 -0
- data/prism.gemspec +21 -4
- data/rbi/generated/prism/node.rbi +2430 -968
- data/rbi/generated/prism/parse_result.rbi +16 -4
- data/rbi/generated/prism/serialize.rbi +8 -4
- data/rbi/generated/prism/string_query.rbi +7 -0
- data/sig/generated/prism/node.rbs +2439 -967
- data/sig/generated/prism/parse_result.rbs +16 -4
- data/sig/generated/prism/serialize.rbs +9 -4
- data/sig/generated/prism/string_query.rbs +7 -0
- data/src/arena.c +6 -1
- data/src/buffer.c +9 -0
- data/src/comments.c +11 -10
- data/src/diagnostic.c +24 -4
- data/src/integer.c +6 -1
- data/src/node.c +216 -368
- data/src/options.c +7 -13
- data/src/parser.c +70 -48
- data/src/prism.c +625 -462
- data/src/regexp.c +0 -1
- data/src/serialize.c +2301 -1950
- data/src/source.c +8 -2
- data/src/static_literals.c +280 -405
- data/src/string_query.c +14 -0
- data/src/stringy.c +3 -33
- data/src/xxh3.c +292 -0
- metadata +20 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 86b52d162f47d28e5a0803d370b2a24ef0f4dd3ec640fcd8e4363d612f97ccbd
|
|
4
|
+
data.tar.gz: 2a490da3e59c672f7b1e1c2ba642bc4d11059d3e6c6663e0bb7506690e51f40d
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2fecc54f3f2648ed61f4a346641a2426b2c5a0fd2182dff9d136a76451de5d52af723b89d0966fc6656b525418ed04800c6cc0bb8f641ca9897a53faf2d7bf5f
|
|
7
|
+
data.tar.gz: 7198dc64515c3ac7040c4c70cf96760b23c6c6c662a3f7349185729479b5ddcf0fd2db86f16e0891451d59fe7d03534710cde138cacfccd36a59ade27c4d0779
|
data/CHANGELOG.md
CHANGED
|
@@ -6,12 +6,24 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [2.1.0] - 2026-10-28
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- Add `Version::CRuby4_0` to the Rust bindings. `Version::CRuby3_5` remains available but is deprecated in favor of the new name; both select Ruby 4.0 syntax.
|
|
14
|
+
- Let embedders set the allocation size for the first block allocated in an arena via `PM_ARENA_INITIAL_SIZE`.
|
|
15
|
+
- Let embedders set the minimum initial size for the constant pool via `PM_CONSTANT_POOL_MIN_CAPACITY`.
|
|
16
|
+
|
|
17
|
+
## [2.0.0] - 2026-10-21
|
|
18
|
+
|
|
9
19
|
### Highlights
|
|
10
20
|
|
|
11
21
|
- Significant performance and memory usage improvements.
|
|
12
22
|
- Precompiled platform-specific variants on rubygems.org.
|
|
13
23
|
- Many documentation improvements, including a new playground.
|
|
14
24
|
- Prism now ships accurate and complete RBS and RBI definitions.
|
|
25
|
+
- The license is now owned by the Ruby developers.
|
|
26
|
+
- Prism now ships an official C++ interface.
|
|
15
27
|
|
|
16
28
|
### Added
|
|
17
29
|
|
|
@@ -29,6 +41,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
|
|
|
29
41
|
- Allow pattern matching with prism nodes to match on location slices, like `node in Prism::StringNode[content: "foo"]`.
|
|
30
42
|
- `Prism.parse_file` and similar now accept anything responding to `to_path` (like `Pathname`) directly.
|
|
31
43
|
- `attach_comments` was added as a valid option. Each comment is attached to the node that ends latest on the comment's line before it, otherwise to the innermost node that contains it (or to that node's statements, if only whitespace and comments come between the comment and the statements), otherwise to the top-level statements. The attached comments are available through `Node#comments` and through `pm_node_comments_size` and `pm_node_comment` in the C API, as well as `pm_parser_comment_attachment`, `pm_parser_comment_ranges_size`, and `pm_parser_comment_range`, which expose the attachments grouped by node. When the option is set, the comments on the parse result are sorted by start offset.
|
|
44
|
+
- `StringQuery#keyword?` was added to check if the string query matches a keyword.
|
|
45
|
+
- `Source#checksum` was added to retrieve the checksum of the source content.
|
|
32
46
|
|
|
33
47
|
### Changed
|
|
34
48
|
|
|
@@ -64,6 +78,9 @@ Note that for this major version, almost every change listed below is breaking.
|
|
|
64
78
|
- Most allocations are now performed through an arena.
|
|
65
79
|
- You may now build side tables of comment attachments through the `attach_comments` option.
|
|
66
80
|
- `newline_list` was renamed to `list_offset_list`.
|
|
81
|
+
- Visitor callbacks passed to `pm_visit_node` and `pm_visit_child_nodes` now return a `pm_visit_result_t` instead of a `bool`. Returning `PM_VISIT_DESCEND` or `PM_VISIT_SKIP` replaces returning `true` or `false`, and returning `PM_VISIT_STOP` ends the walk. Both functions now return `false` if the walk was stopped.
|
|
82
|
+
- `pm_string_t` is now a view of bytes owned by someone else. It no longer has a `type` field, `pm_string_owned_init` was removed, and `pm_string_constant_init` was renamed to `pm_string_init`, which takes the bytes as `const uint8_t *`.
|
|
83
|
+
- `pm_options_free` no longer frees the names of scope locals. The parser copies them when it is created, so they only need to remain valid until then.
|
|
67
84
|
|
|
68
85
|
#### Rust
|
|
69
86
|
|
|
@@ -71,6 +88,7 @@ Note that for this major version, almost every change listed below is breaking.
|
|
|
71
88
|
- Most allocations are now performed through an arena.
|
|
72
89
|
- `newline_list` was renamed to `list_offset_list`.
|
|
73
90
|
- `Diagnostic::message` now returns `&[u8]` instead of `&str`.
|
|
91
|
+
- `StringQuery` is now exposed.
|
|
74
92
|
|
|
75
93
|
#### JavaScript
|
|
76
94
|
|
|
@@ -827,44 +845,46 @@ Note that for this major version, almost every change listed below is breaking.
|
|
|
827
845
|
|
|
828
846
|
- 🎉 Initial release! 🎉
|
|
829
847
|
|
|
830
|
-
[unreleased]: https://github.com/ruby/
|
|
831
|
-
[1.
|
|
832
|
-
[
|
|
833
|
-
[1.
|
|
834
|
-
[1.
|
|
835
|
-
[1.
|
|
836
|
-
[1.
|
|
837
|
-
[1.
|
|
838
|
-
[1.5.
|
|
839
|
-
[1.
|
|
840
|
-
[1.
|
|
841
|
-
[1.
|
|
842
|
-
[1.
|
|
843
|
-
[1.
|
|
844
|
-
[
|
|
845
|
-
[0.
|
|
846
|
-
[0.
|
|
847
|
-
[0.
|
|
848
|
-
[0.
|
|
849
|
-
[0.
|
|
850
|
-
[0.
|
|
851
|
-
[0.
|
|
852
|
-
[0.
|
|
853
|
-
[0.
|
|
854
|
-
[0.
|
|
855
|
-
[0.
|
|
856
|
-
[0.
|
|
857
|
-
[0.
|
|
858
|
-
[0.
|
|
859
|
-
[0.
|
|
860
|
-
[0.
|
|
861
|
-
[0.
|
|
862
|
-
[0.
|
|
863
|
-
[0.
|
|
864
|
-
[0.
|
|
865
|
-
[0.
|
|
866
|
-
[0.
|
|
867
|
-
[0.
|
|
868
|
-
[0.
|
|
869
|
-
[0.
|
|
870
|
-
[0.
|
|
848
|
+
[unreleased]: https://github.com/ruby/prism3/compare/v2.1.0...HEAD
|
|
849
|
+
[2.1.0]: https://github.com/ruby/prism3/compare/v2.0.0...v2.1.0
|
|
850
|
+
[2.0.0]: https://github.com/ruby/prism3/compare/v1.9.0...v2.0.0
|
|
851
|
+
[1.9.0]: https://github.com/ruby/prism3/compare/v1.8.1...v1.9.0
|
|
852
|
+
[1.8.1]: https://github.com/ruby/prism3/compare/v1.8.0...v1.8.1
|
|
853
|
+
[1.8.0]: https://github.com/ruby/prism3/compare/v1.7.0...v1.8.0
|
|
854
|
+
[1.7.0]: https://github.com/ruby/prism3/compare/v1.6.0...v1.7.0
|
|
855
|
+
[1.6.0]: https://github.com/ruby/prism3/compare/v1.5.2...v1.6.0
|
|
856
|
+
[1.5.2]: https://github.com/ruby/prism3/compare/v1.5.1...v1.5.2
|
|
857
|
+
[1.5.1]: https://github.com/ruby/prism3/compare/v1.5.0...v1.5.1
|
|
858
|
+
[1.5.0]: https://github.com/ruby/prism3/compare/v1.4.0...v1.5.0
|
|
859
|
+
[1.4.0]: https://github.com/ruby/prism3/compare/v1.3.0...v1.4.0
|
|
860
|
+
[1.3.0]: https://github.com/ruby/prism3/compare/v1.2.0...v1.3.0
|
|
861
|
+
[1.2.0]: https://github.com/ruby/prism3/compare/v1.1.0...v1.2.0
|
|
862
|
+
[1.1.0]: https://github.com/ruby/prism3/compare/v1.0.0...v1.1.0
|
|
863
|
+
[1.0.0]: https://github.com/ruby/prism3/compare/v0.30.0...v1.0.0
|
|
864
|
+
[0.30.0]: https://github.com/ruby/prism3/compare/v0.29.0...v0.30.0
|
|
865
|
+
[0.29.0]: https://github.com/ruby/prism3/compare/v0.28.0...v0.29.0
|
|
866
|
+
[0.28.0]: https://github.com/ruby/prism3/compare/v0.27.0...v0.28.0
|
|
867
|
+
[0.27.0]: https://github.com/ruby/prism3/compare/v0.26.0...v0.27.0
|
|
868
|
+
[0.26.0]: https://github.com/ruby/prism3/compare/v0.25.0...v0.26.0
|
|
869
|
+
[0.25.0]: https://github.com/ruby/prism3/compare/v0.24.0...v0.25.0
|
|
870
|
+
[0.24.0]: https://github.com/ruby/prism3/compare/v0.23.0...v0.24.0
|
|
871
|
+
[0.23.0]: https://github.com/ruby/prism3/compare/v0.22.0...v0.23.0
|
|
872
|
+
[0.22.0]: https://github.com/ruby/prism3/compare/v0.21.0...v0.22.0
|
|
873
|
+
[0.21.0]: https://github.com/ruby/prism3/compare/v0.20.0...v0.21.0
|
|
874
|
+
[0.20.0]: https://github.com/ruby/prism3/compare/v0.19.0...v0.20.0
|
|
875
|
+
[0.19.0]: https://github.com/ruby/prism3/compare/v0.18.0...v0.19.0
|
|
876
|
+
[0.18.0]: https://github.com/ruby/prism3/compare/v0.17.1...v0.18.0
|
|
877
|
+
[0.17.1]: https://github.com/ruby/prism3/compare/v0.17.0...v0.17.1
|
|
878
|
+
[0.17.0]: https://github.com/ruby/prism3/compare/v0.16.0...v0.17.0
|
|
879
|
+
[0.16.0]: https://github.com/ruby/prism3/compare/v0.15.1...v0.16.0
|
|
880
|
+
[0.15.1]: https://github.com/ruby/prism3/compare/v0.15.0...v0.15.1
|
|
881
|
+
[0.15.0]: https://github.com/ruby/prism3/compare/v0.14.0...v0.15.0
|
|
882
|
+
[0.14.0]: https://github.com/ruby/prism3/compare/v0.13.0...v0.14.0
|
|
883
|
+
[0.13.0]: https://github.com/ruby/prism3/compare/v0.12.0...v0.13.0
|
|
884
|
+
[0.12.0]: https://github.com/ruby/prism3/compare/v0.11.0...v0.12.0
|
|
885
|
+
[0.11.0]: https://github.com/ruby/prism3/compare/v0.10.0...v0.11.0
|
|
886
|
+
[0.10.0]: https://github.com/ruby/prism3/compare/v0.9.0...v0.10.0
|
|
887
|
+
[0.9.0]: https://github.com/ruby/prism3/compare/v0.8.0...v0.9.0
|
|
888
|
+
[0.8.0]: https://github.com/ruby/prism3/compare/v0.7.0...v0.8.0
|
|
889
|
+
[0.7.0]: https://github.com/ruby/prism3/compare/v0.6.0...v0.7.0
|
|
890
|
+
[0.6.0]: https://github.com/ruby/prism3/compare/d60531...v0.6.0
|
data/CODE_OF_CONDUCT.md
CHANGED
|
@@ -55,7 +55,7 @@ further defined and clarified by project maintainers.
|
|
|
55
55
|
## Enforcement
|
|
56
56
|
|
|
57
57
|
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
|
58
|
-
reported by contacting the project team at
|
|
58
|
+
reported by contacting the project team at ruby-core@ruby-lang.org. All
|
|
59
59
|
complaints will be reviewed and investigated and will result in a response that
|
|
60
60
|
is deemed necessary and appropriate to the circumstances. The project team is
|
|
61
61
|
obligated to maintain confidentiality with regard to the reporter of an incident.
|
data/LICENSE.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
Copyright 2022-present,
|
|
1
|
+
Copyright 2022-present, Ruby developers
|
|
2
2
|
|
|
3
3
|
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
|
4
4
|
|
data/Makefile
CHANGED
|
@@ -6,7 +6,7 @@ Q1 = $(V:1=)
|
|
|
6
6
|
Q = $(Q1:0=@)
|
|
7
7
|
ECHO1 = $(V:1=@ :)
|
|
8
8
|
ECHO = $(ECHO1:0=@ echo)
|
|
9
|
-
FUZZ_OUTPUT_DIR = $(CURDIR)/fuzz/output
|
|
9
|
+
FUZZ_OUTPUT_DIR = $(CURDIR)/test/fuzz/output
|
|
10
10
|
|
|
11
11
|
SOEXT ?= $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
|
|
12
12
|
|
|
@@ -94,7 +94,7 @@ build/static/%.o: src/%.c Makefile $(HEADERS)
|
|
|
94
94
|
$(Q) $(MAKEDIRS) $(@D)
|
|
95
95
|
$(Q) $(CC) $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) -c -o $@ $<
|
|
96
96
|
|
|
97
|
-
build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
|
|
97
|
+
build/fuzz.%: $(SOURCES) test/fuzz/%.c test/fuzz/fuzz.c
|
|
98
98
|
$(ECHO) "building $* fuzzer"
|
|
99
99
|
$(Q) $(MAKEDIRS) $(@D)
|
|
100
100
|
$(ECHO) "building main fuzz binary"
|
|
@@ -102,16 +102,16 @@ build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
|
|
|
102
102
|
$(ECHO) "building cmplog binary"
|
|
103
103
|
$(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@.cmplog $^
|
|
104
104
|
|
|
105
|
-
build/fuzz.heisenbug.%: $(SOURCES) fuzz/%.c fuzz/heisenbug.c
|
|
105
|
+
build/fuzz.heisenbug.%: $(SOURCES) test/fuzz/%.c test/fuzz/heisenbug.c
|
|
106
106
|
$(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
|
|
107
107
|
|
|
108
108
|
fuzz-debug:
|
|
109
109
|
$(ECHO) "entering debug shell"
|
|
110
|
-
$(Q) docker run -it --rm -e HISTFILE=/prism/fuzz/output/.bash_history -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz
|
|
110
|
+
$(Q) docker run -it --rm -e HISTFILE=/prism/test/fuzz/output/.bash_history -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz
|
|
111
111
|
|
|
112
|
-
fuzz-docker-build: fuzz/docker/Dockerfile
|
|
112
|
+
fuzz-docker-build: test/fuzz/docker/Dockerfile
|
|
113
113
|
$(ECHO) "building docker image"
|
|
114
|
-
$(Q) docker build -t prism/fuzz fuzz/docker/
|
|
114
|
+
$(Q) docker build -t prism/fuzz test/fuzz/docker/
|
|
115
115
|
|
|
116
116
|
fuzz-run-%: FORCE fuzz-docker-build
|
|
117
117
|
$(ECHO) "generating templates"
|
|
@@ -120,11 +120,11 @@ fuzz-run-%: FORCE fuzz-docker-build
|
|
|
120
120
|
$(Q) docker run --rm -v $(CURDIR):/prism prism/fuzz /bin/bash -c "FUZZ_FLAGS=\"$(FUZZ_FLAGS)\" make build/fuzz.$*"
|
|
121
121
|
$(ECHO) "starting AFL++ run"
|
|
122
122
|
$(Q) $(MAKEDIRS) $(FUZZ_OUTPUT_DIR)/$*
|
|
123
|
-
$(Q) docker run -it --rm -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz /bin/bash -c "./fuzz/$*.sh /fuzz_output/$*"
|
|
123
|
+
$(Q) docker run -it --rm -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz /bin/bash -c "./test/fuzz/$*.sh /fuzz_output/$*"
|
|
124
124
|
FORCE:
|
|
125
125
|
|
|
126
126
|
fuzz-clean:
|
|
127
|
-
$(Q) $(RMALL) fuzz/output
|
|
127
|
+
$(Q) $(RMALL) test/fuzz/output
|
|
128
128
|
|
|
129
129
|
clean:
|
|
130
130
|
$(Q) $(RMALL) build
|
data/README.md
CHANGED
|
@@ -49,7 +49,6 @@ The repository contains the infrastructure for both a shared library (libprism)
|
|
|
49
49
|
│ └── prism
|
|
50
50
|
│ ├── extconf.rb configuration to generate the Makefile for the native extension
|
|
51
51
|
│ └── extension.c the native extension that interacts with libprism
|
|
52
|
-
├── fuzz files related to fuzz testing
|
|
53
52
|
├── gemfiles gemfiles used by different Ruby versions in CI
|
|
54
53
|
├── include
|
|
55
54
|
│ ├── prism header files for the shared library
|
|
@@ -75,6 +74,9 @@ The repository contains the infrastructure for both a shared library (libprism)
|
|
|
75
74
|
├── templates contains ERB templates generated by templates/template.rb
|
|
76
75
|
│ └── template.rb generates code from the nodes and tokens configured by config.yml
|
|
77
76
|
└── test
|
|
77
|
+
├── cpp tests for the C++ bindings
|
|
78
|
+
├── embedded smoke test for bare-metal targets
|
|
79
|
+
├── fuzz files related to fuzz testing
|
|
78
80
|
└── prism
|
|
79
81
|
├── fixtures Ruby code used for testing
|
|
80
82
|
└── snapshots snapshots of generated syntax trees corresponding to fixtures
|
data/docs/fuzzing.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
# Fuzzing
|
|
2
2
|
|
|
3
|
-
We use fuzzing to test the various entrypoints to the library. The fuzzer we use is [AFL++](https://aflplus.plus). All files related to fuzzing live within the `fuzz` directory, which has the following structure:
|
|
3
|
+
We use fuzzing to test the various entrypoints to the library. The fuzzer we use is [AFL++](https://aflplus.plus). All files related to fuzzing live within the `test/fuzz` directory, which has the following structure:
|
|
4
4
|
|
|
5
5
|
```
|
|
6
|
-
fuzz
|
|
6
|
+
test/fuzz
|
|
7
7
|
├── corpus
|
|
8
8
|
│ └── parse fuzzing corpus for parsing (a symlink to our fixtures)
|
|
9
9
|
├── dict a AFL++ dictionary containing various tokens
|
|
@@ -57,12 +57,12 @@ make build/fuzz.heisenbug.parse
|
|
|
57
57
|
Triaging crashes and hangs is easier when the inputs are as short as possible. In the fuzz container, an entire crash or hang directory can be minimized using
|
|
58
58
|
|
|
59
59
|
```
|
|
60
|
-
./fuzz/tools/minimize.sh <directory>
|
|
60
|
+
./test/fuzz/tools/minimize.sh <directory>
|
|
61
61
|
```
|
|
62
62
|
|
|
63
63
|
e.g.
|
|
64
64
|
```
|
|
65
|
-
./fuzz/tools/minimize.sh fuzz/output/parse/default/crashes
|
|
65
|
+
./test/fuzz/tools/minimize.sh test/fuzz/output/parse/default/crashes
|
|
66
66
|
```
|
|
67
67
|
|
|
68
68
|
This may take a long time. In the crash/hang directory, for each input file there will appear a minimized version with the extension `.min` appended.
|
|
@@ -70,7 +70,7 @@ This may take a long time. In the crash/hang directory, for each input file ther
|
|
|
70
70
|
Backtraces for crashes (not hangs) can be generated en masse with
|
|
71
71
|
|
|
72
72
|
```
|
|
73
|
-
./fuzz/tools/backtrace.sh <directory>
|
|
73
|
+
./test/fuzz/tools/backtrace.sh <directory>
|
|
74
74
|
```
|
|
75
75
|
|
|
76
76
|
Files with basename equal to the input file name with extension `.bt` will be created e.g.
|
data/docs/serialization.md
CHANGED
|
@@ -106,6 +106,7 @@ The header is structured like the following table:
|
|
|
106
106
|
| `1` | 1 indicates only semantics fields were serialized, 0 indicates all fields were serialized (including location fields) |
|
|
107
107
|
| string | the encoding name |
|
|
108
108
|
| varsint | the start line |
|
|
109
|
+
| `8` | the checksum of the source up to the end of the `__END__` line in native endian order, see `pm_parser_checksum` |
|
|
109
110
|
| varuint | number of newline offsets |
|
|
110
111
|
| varuint* | newline offsets |
|
|
111
112
|
| varuint | number of comments |
|
|
@@ -124,8 +125,14 @@ The header is structured like the following table:
|
|
|
124
125
|
| varuint | content pool size |
|
|
125
126
|
|
|
126
127
|
After the header comes the body of the serialized string.
|
|
127
|
-
The body consists of a sequence of nodes that is built using a
|
|
128
|
-
|
|
128
|
+
The body consists of a sequence of nodes that is built using a postfix traversal order of the syntax tree: the children of each node come before the node itself, so the root node comes last.
|
|
129
|
+
The body ends at the content pool offset specified in the header.
|
|
130
|
+
|
|
131
|
+
The children of a node are the values of its `node`, `node?`, and `node[]` fields, which are serialized in the order in which the fields are listed in `config.yml`.
|
|
132
|
+
A `node` field is serialized as its node, a `node?` field is serialized as its node or as a single `0` byte if the node is not present, and a `node[]` field is serialized as each of its nodes in order.
|
|
133
|
+
A loader can therefore keep a stack of loaded nodes: a `0` byte pushes an absent node, and each node pops its children off the top of the stack (they are in field order) and then pushes itself.
|
|
134
|
+
|
|
135
|
+
After its children, each node is structured like the following table:
|
|
129
136
|
|
|
130
137
|
| # bytes | field |
|
|
131
138
|
| --- | --- |
|
|
@@ -134,12 +141,11 @@ Each node is structured like the following table:
|
|
|
134
141
|
| location | node location |
|
|
135
142
|
| varuint | node flags |
|
|
136
143
|
|
|
137
|
-
Every field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
|
|
144
|
+
Every other field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
|
|
138
145
|
|
|
139
146
|
* `double` - A field that is a `double`. This is structured as a sequence of 8 bytes in native endian order.
|
|
140
|
-
* `node` -
|
|
141
|
-
* `node
|
|
142
|
-
* `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, followed by the child nodes themselves.
|
|
147
|
+
* `node`, `node?` - Nothing is appended here, since the node has already been serialized as a child.
|
|
148
|
+
* `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, which is the number of children that belong to this field.
|
|
143
149
|
* `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string bytes (_without_ a trailing null byte).
|
|
144
150
|
* `constant` - A variable-length integer that represents an index in the constant pool.
|
|
145
151
|
* `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
|
|
@@ -149,6 +155,14 @@ Every field on the node is then appended to the serialized string. The fields ca
|
|
|
149
155
|
* `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
|
|
150
156
|
* `uint32` - A field that is a 32-bit unsigned integer. This is structured as a variable-length integer.
|
|
151
157
|
|
|
158
|
+
The `body` field of a `DefNode` is lazy, which means that loaders can skip it and load it later. The children of a `DefNode` are preceded by the following, which loaders that do not skip lazy fields can ignore:
|
|
159
|
+
|
|
160
|
+
| # bytes | field |
|
|
161
|
+
| --- | --- |
|
|
162
|
+
| `1` | `255` |
|
|
163
|
+
| `4` | the byte length of the children that come before the `body` child |
|
|
164
|
+
| `4` | the byte length of the `body` child |
|
|
165
|
+
|
|
152
166
|
After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Every constant is embedded in the serialization. Each constant is structured as follows:
|
|
153
167
|
|
|
154
168
|
| # bytes | field |
|