prism 2.0.0.rc2 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (69) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +57 -41
  3. data/Makefile +8 -8
  4. data/README.md +3 -1
  5. data/docs/fuzzing.md +5 -5
  6. data/docs/serialization.md +19 -6
  7. data/ext/prism/api_node.c +80 -80
  8. data/ext/prism/extension.c +34 -67
  9. data/ext/prism/extension.h +1 -1
  10. data/include/prism/arena.h +8 -0
  11. data/include/prism/arena.hpp +42 -0
  12. data/include/prism/ast.h +11 -1
  13. data/include/prism/ast.hpp +641 -0
  14. data/include/prism/buffer.h +8 -0
  15. data/include/prism/buffer.hpp +55 -0
  16. data/include/prism/comments.h +17 -0
  17. data/include/prism/compiler/branch.h +26 -0
  18. data/include/prism/compiler/cplusplus.hpp +16 -0
  19. data/include/prism/compiler/exceptions.hpp +17 -0
  20. data/include/prism/compiler/format.h +2 -0
  21. data/include/prism/compiler/lifetimebound.hpp +25 -0
  22. data/include/prism/constant_pool.h +25 -0
  23. data/include/prism/diagnostic.h +18 -0
  24. data/include/prism/errors_format.h +8 -0
  25. data/include/prism/internal/comments.h +2 -1
  26. data/include/prism/internal/node.h +8 -4
  27. data/include/prism/internal/parser.h +26 -15
  28. data/include/prism/internal/static_literals.h +24 -64
  29. data/include/prism/internal/stringy.h +1 -6
  30. data/include/prism/json.h +8 -0
  31. data/include/prism/line_offset_list.h +8 -0
  32. data/include/prism/magic_comments.h +18 -0
  33. data/include/prism/node.h +55 -10
  34. data/include/prism/node.hpp +272 -0
  35. data/include/prism/node_type.hpp +38 -0
  36. data/include/prism/options.h +10 -1
  37. data/include/prism/options.hpp +43 -0
  38. data/include/prism/parser.h +72 -64
  39. data/include/prism/parser.hpp +274 -0
  40. data/include/prism/prettyprint.h +8 -0
  41. data/include/prism/serialize.h +8 -0
  42. data/include/prism/source.h +8 -0
  43. data/include/prism/stream.h +8 -0
  44. data/include/prism/string_query.h +8 -0
  45. data/include/prism/stringy.h +16 -27
  46. data/include/prism/stringy.hpp +31 -0
  47. data/include/prism/version.h +10 -2
  48. data/include/prism.h +0 -8
  49. data/include/prism.hpp +24 -0
  50. data/lib/prism/serialize.rb +2550 -3282
  51. data/prism.gemspec +14 -1
  52. data/rbi/generated/prism/node.rbi +736 -0
  53. data/rbi/generated/prism/serialize.rbi +6 -5
  54. data/sig/generated/prism/node.rbs +736 -0
  55. data/sig/generated/prism/serialize.rbs +7 -5
  56. data/src/arena.c +6 -1
  57. data/src/comments.c +11 -10
  58. data/src/diagnostic.c +24 -4
  59. data/src/integer.c +6 -1
  60. data/src/node.c +216 -368
  61. data/src/options.c +7 -13
  62. data/src/parser.c +52 -48
  63. data/src/prism.c +222 -240
  64. data/src/regexp.c +0 -1
  65. data/src/serialize.c +2300 -1950
  66. data/src/source.c +8 -2
  67. data/src/static_literals.c +280 -405
  68. data/src/stringy.c +3 -33
  69. metadata +14 -1
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: e07eb7cf91add89de3aa71e9707336a2a561c977b98f5a71ab31c9f0083623e4
4
- data.tar.gz: 684ec87f807ac1e17a3b251746745751707cf300b4d2f6429cdc852102261204
3
+ metadata.gz: 8e2f8907c10234cbde0a461b0674bc64139201a2d846bc2aa55e7ff416e3bac8
4
+ data.tar.gz: 396fd09bb9efc62dc076783701bb98fc9576b722ff9e2b4e89366eeef02f606e
5
5
  SHA512:
6
- metadata.gz: 86a9128a3c56bffa1b9c6684f83e2e320fc58113f0a778ff1139d1393573ed5c42db80d7b9665b558aef9af9fbe72d84d929c997ce2ebdbfe07eafaa00864a86
7
- data.tar.gz: 02c7498a1119b1106aa3a75bd07ca4d2aca13ba133e8c716693bbb33cca1d3e6004ccc6811155a4adff65a6758088066e817d7084727b80f7b6437cc280b6040
6
+ metadata.gz: 844a68c1e7f2ebed766ec0a2b9f68ce2849e0205fd8339a04310a5b8725cdb820c1c3d04b077c11f112dff974cd74a1d29a9c7dd1c8f5d3a61efc3cfc1dbc50c
7
+ data.tar.gz: 90ac3995ffabefe8943d3159ac6f6b5a215b214c7d74a3c42c4fbf9abd3a3bfb12af32cee27747b28b1c5467ff0edf956d1346caa065fc81a29b7c3daa24699f
data/CHANGELOG.md CHANGED
@@ -6,6 +6,16 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [2.1.0] - 2026-10-28
10
+
11
+ ### Added
12
+
13
+ - Add `Version::CRuby4_0` to the Rust bindings. `Version::CRuby3_5` remains available but is deprecated in favor of the new name; both select Ruby 4.0 syntax.
14
+ - Let embedders set the allocation size for the first block allocated in an arena via `PM_ARENA_INITIAL_SIZE`.
15
+ - Let embedders set the minimum initial size for the constant pool via `PM_CONSTANT_POOL_MIN_CAPACITY`.
16
+
17
+ ## [2.0.0] - 2026-10-21
18
+
9
19
  ### Highlights
10
20
 
11
21
  - Significant performance and memory usage improvements.
@@ -13,6 +23,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) a
13
23
  - Many documentation improvements, including a new playground.
14
24
  - Prism now ships accurate and complete RBS and RBI definitions.
15
25
  - The license is now owned by the Ruby developers.
26
+ - Prism now ships an official C++ interface.
16
27
 
17
28
  ### Added
18
29
 
@@ -67,6 +78,9 @@ Note that for this major version, almost every change listed below is breaking.
67
78
  - Most allocations are now performed through an arena.
68
79
  - You may now build side tables of comment attachments through the `attach_comments` option.
69
80
  - `newline_list` was renamed to `list_offset_list`.
81
+ - Visitor callbacks passed to `pm_visit_node` and `pm_visit_child_nodes` now return a `pm_visit_result_t` instead of a `bool`. Returning `PM_VISIT_DESCEND` or `PM_VISIT_SKIP` replaces returning `true` or `false`, and returning `PM_VISIT_STOP` ends the walk. Both functions now return `false` if the walk was stopped.
82
+ - `pm_string_t` is now a view of bytes owned by someone else. It no longer has a `type` field, `pm_string_owned_init` was removed, and `pm_string_constant_init` was renamed to `pm_string_init`, which takes the bytes as `const uint8_t *`.
83
+ - `pm_options_free` no longer frees the names of scope locals. The parser copies them when it is created, so they only need to remain valid until then.
70
84
 
71
85
  #### Rust
72
86
 
@@ -831,44 +845,46 @@ Note that for this major version, almost every change listed below is breaking.
831
845
 
832
846
  - 🎉 Initial release! 🎉
833
847
 
834
- [unreleased]: https://github.com/ruby/prism/compare/v1.9.0...HEAD
835
- [1.9.0]: https://github.com/ruby/prism/compare/v1.8.0...v1.9.0
836
- [1.8.1]: https://github.com/ruby/prism/compare/v1.8.0...v1.8.1
837
- [1.8.0]: https://github.com/ruby/prism/compare/v1.7.0...v1.8.0
838
- [1.7.0]: https://github.com/ruby/prism/compare/v1.6.0...v1.7.0
839
- [1.6.0]: https://github.com/ruby/prism/compare/v1.5.2...v1.6.0
840
- [1.5.2]: https://github.com/ruby/prism/compare/v1.5.1...v1.5.2
841
- [1.5.1]: https://github.com/ruby/prism/compare/v1.5.0...v1.5.1
842
- [1.5.0]: https://github.com/ruby/prism/compare/v1.4.0...v1.5.0
843
- [1.4.0]: https://github.com/ruby/prism/compare/v1.3.0...v1.4.0
844
- [1.3.0]: https://github.com/ruby/prism/compare/v1.2.0...v1.3.0
845
- [1.2.0]: https://github.com/ruby/prism/compare/v1.1.0...v1.2.0
846
- [1.1.0]: https://github.com/ruby/prism/compare/v1.0.0...v1.1.0
847
- [1.0.0]: https://github.com/ruby/prism/compare/v0.30.0...v1.0.0
848
- [0.30.0]: https://github.com/ruby/prism/compare/v0.29.0...v0.30.0
849
- [0.29.0]: https://github.com/ruby/prism/compare/v0.28.0...v0.29.0
850
- [0.28.0]: https://github.com/ruby/prism/compare/v0.27.0...v0.28.0
851
- [0.27.0]: https://github.com/ruby/prism/compare/v0.26.0...v0.27.0
852
- [0.26.0]: https://github.com/ruby/prism/compare/v0.25.0...v0.26.0
853
- [0.25.0]: https://github.com/ruby/prism/compare/v0.24.0...v0.25.0
854
- [0.24.0]: https://github.com/ruby/prism/compare/v0.23.0...v0.24.0
855
- [0.23.0]: https://github.com/ruby/prism/compare/v0.22.0...v0.23.0
856
- [0.22.0]: https://github.com/ruby/prism/compare/v0.21.0...v0.22.0
857
- [0.21.0]: https://github.com/ruby/prism/compare/v0.20.0...v0.21.0
858
- [0.20.0]: https://github.com/ruby/prism/compare/v0.19.0...v0.20.0
859
- [0.19.0]: https://github.com/ruby/prism/compare/v0.18.0...v0.19.0
860
- [0.18.0]: https://github.com/ruby/prism/compare/v0.17.1...v0.18.0
861
- [0.17.1]: https://github.com/ruby/prism/compare/v0.17.0...v0.17.1
862
- [0.17.0]: https://github.com/ruby/prism/compare/v0.16.0...v0.17.0
863
- [0.16.0]: https://github.com/ruby/prism/compare/v0.15.1...v0.16.0
864
- [0.15.1]: https://github.com/ruby/prism/compare/v0.15.0...v0.15.1
865
- [0.15.0]: https://github.com/ruby/prism/compare/v0.14.0...v0.15.0
866
- [0.14.0]: https://github.com/ruby/prism/compare/v0.13.0...v0.14.0
867
- [0.13.0]: https://github.com/ruby/prism/compare/v0.12.0...v0.13.0
868
- [0.12.0]: https://github.com/ruby/prism/compare/v0.11.0...v0.12.0
869
- [0.11.0]: https://github.com/ruby/prism/compare/v0.10.0...v0.11.0
870
- [0.10.0]: https://github.com/ruby/prism/compare/v0.9.0...v0.10.0
871
- [0.9.0]: https://github.com/ruby/prism/compare/v0.8.0...v0.9.0
872
- [0.8.0]: https://github.com/ruby/prism/compare/v0.7.0...v0.8.0
873
- [0.7.0]: https://github.com/ruby/prism/compare/v0.6.0...v0.7.0
874
- [0.6.0]: https://github.com/ruby/prism/compare/d60531...v0.6.0
848
+ [unreleased]: https://github.com/ruby/prism3/compare/v2.1.0...HEAD
849
+ [2.1.0]: https://github.com/ruby/prism3/compare/v2.0.0...v2.1.0
850
+ [2.0.0]: https://github.com/ruby/prism3/compare/v1.9.0...v2.0.0
851
+ [1.9.0]: https://github.com/ruby/prism3/compare/v1.8.1...v1.9.0
852
+ [1.8.1]: https://github.com/ruby/prism3/compare/v1.8.0...v1.8.1
853
+ [1.8.0]: https://github.com/ruby/prism3/compare/v1.7.0...v1.8.0
854
+ [1.7.0]: https://github.com/ruby/prism3/compare/v1.6.0...v1.7.0
855
+ [1.6.0]: https://github.com/ruby/prism3/compare/v1.5.2...v1.6.0
856
+ [1.5.2]: https://github.com/ruby/prism3/compare/v1.5.1...v1.5.2
857
+ [1.5.1]: https://github.com/ruby/prism3/compare/v1.5.0...v1.5.1
858
+ [1.5.0]: https://github.com/ruby/prism3/compare/v1.4.0...v1.5.0
859
+ [1.4.0]: https://github.com/ruby/prism3/compare/v1.3.0...v1.4.0
860
+ [1.3.0]: https://github.com/ruby/prism3/compare/v1.2.0...v1.3.0
861
+ [1.2.0]: https://github.com/ruby/prism3/compare/v1.1.0...v1.2.0
862
+ [1.1.0]: https://github.com/ruby/prism3/compare/v1.0.0...v1.1.0
863
+ [1.0.0]: https://github.com/ruby/prism3/compare/v0.30.0...v1.0.0
864
+ [0.30.0]: https://github.com/ruby/prism3/compare/v0.29.0...v0.30.0
865
+ [0.29.0]: https://github.com/ruby/prism3/compare/v0.28.0...v0.29.0
866
+ [0.28.0]: https://github.com/ruby/prism3/compare/v0.27.0...v0.28.0
867
+ [0.27.0]: https://github.com/ruby/prism3/compare/v0.26.0...v0.27.0
868
+ [0.26.0]: https://github.com/ruby/prism3/compare/v0.25.0...v0.26.0
869
+ [0.25.0]: https://github.com/ruby/prism3/compare/v0.24.0...v0.25.0
870
+ [0.24.0]: https://github.com/ruby/prism3/compare/v0.23.0...v0.24.0
871
+ [0.23.0]: https://github.com/ruby/prism3/compare/v0.22.0...v0.23.0
872
+ [0.22.0]: https://github.com/ruby/prism3/compare/v0.21.0...v0.22.0
873
+ [0.21.0]: https://github.com/ruby/prism3/compare/v0.20.0...v0.21.0
874
+ [0.20.0]: https://github.com/ruby/prism3/compare/v0.19.0...v0.20.0
875
+ [0.19.0]: https://github.com/ruby/prism3/compare/v0.18.0...v0.19.0
876
+ [0.18.0]: https://github.com/ruby/prism3/compare/v0.17.1...v0.18.0
877
+ [0.17.1]: https://github.com/ruby/prism3/compare/v0.17.0...v0.17.1
878
+ [0.17.0]: https://github.com/ruby/prism3/compare/v0.16.0...v0.17.0
879
+ [0.16.0]: https://github.com/ruby/prism3/compare/v0.15.1...v0.16.0
880
+ [0.15.1]: https://github.com/ruby/prism3/compare/v0.15.0...v0.15.1
881
+ [0.15.0]: https://github.com/ruby/prism3/compare/v0.14.0...v0.15.0
882
+ [0.14.0]: https://github.com/ruby/prism3/compare/v0.13.0...v0.14.0
883
+ [0.13.0]: https://github.com/ruby/prism3/compare/v0.12.0...v0.13.0
884
+ [0.12.0]: https://github.com/ruby/prism3/compare/v0.11.0...v0.12.0
885
+ [0.11.0]: https://github.com/ruby/prism3/compare/v0.10.0...v0.11.0
886
+ [0.10.0]: https://github.com/ruby/prism3/compare/v0.9.0...v0.10.0
887
+ [0.9.0]: https://github.com/ruby/prism3/compare/v0.8.0...v0.9.0
888
+ [0.8.0]: https://github.com/ruby/prism3/compare/v0.7.0...v0.8.0
889
+ [0.7.0]: https://github.com/ruby/prism3/compare/v0.6.0...v0.7.0
890
+ [0.6.0]: https://github.com/ruby/prism3/compare/d60531...v0.6.0
data/Makefile CHANGED
@@ -6,7 +6,7 @@ Q1 = $(V:1=)
6
6
  Q = $(Q1:0=@)
7
7
  ECHO1 = $(V:1=@ :)
8
8
  ECHO = $(ECHO1:0=@ echo)
9
- FUZZ_OUTPUT_DIR = $(CURDIR)/fuzz/output
9
+ FUZZ_OUTPUT_DIR = $(CURDIR)/test/fuzz/output
10
10
 
11
11
  SOEXT ?= $(shell ruby -e 'puts RbConfig::CONFIG["SOEXT"]')
12
12
 
@@ -94,7 +94,7 @@ build/static/%.o: src/%.c Makefile $(HEADERS)
94
94
  $(Q) $(MAKEDIRS) $(@D)
95
95
  $(Q) $(CC) $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) -c -o $@ $<
96
96
 
97
- build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
97
+ build/fuzz.%: $(SOURCES) test/fuzz/%.c test/fuzz/fuzz.c
98
98
  $(ECHO) "building $* fuzzer"
99
99
  $(Q) $(MAKEDIRS) $(@D)
100
100
  $(ECHO) "building main fuzz binary"
@@ -102,16 +102,16 @@ build/fuzz.%: $(SOURCES) fuzz/%.c fuzz/fuzz.c
102
102
  $(ECHO) "building cmplog binary"
103
103
  $(Q) AFL_LLVM_CMPLOG=1 afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@.cmplog $^
104
104
 
105
- build/fuzz.heisenbug.%: $(SOURCES) fuzz/%.c fuzz/heisenbug.c
105
+ build/fuzz.heisenbug.%: $(SOURCES) test/fuzz/%.c test/fuzz/heisenbug.c
106
106
  $(Q) afl-clang-lto $(DEBUG_FLAGS) $(CPPFLAGS) $(CFLAGS) $(FUZZ_FLAGS) -O0 -fsanitize=fuzzer,address -ggdb3 -std=c99 -Iinclude -o $@ $^
107
107
 
108
108
  fuzz-debug:
109
109
  $(ECHO) "entering debug shell"
110
- $(Q) docker run -it --rm -e HISTFILE=/prism/fuzz/output/.bash_history -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz
110
+ $(Q) docker run -it --rm -e HISTFILE=/prism/test/fuzz/output/.bash_history -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz
111
111
 
112
- fuzz-docker-build: fuzz/docker/Dockerfile
112
+ fuzz-docker-build: test/fuzz/docker/Dockerfile
113
113
  $(ECHO) "building docker image"
114
- $(Q) docker build -t prism/fuzz fuzz/docker/
114
+ $(Q) docker build -t prism/fuzz test/fuzz/docker/
115
115
 
116
116
  fuzz-run-%: FORCE fuzz-docker-build
117
117
  $(ECHO) "generating templates"
@@ -120,11 +120,11 @@ fuzz-run-%: FORCE fuzz-docker-build
120
120
  $(Q) docker run --rm -v $(CURDIR):/prism prism/fuzz /bin/bash -c "FUZZ_FLAGS=\"$(FUZZ_FLAGS)\" make build/fuzz.$*"
121
121
  $(ECHO) "starting AFL++ run"
122
122
  $(Q) $(MAKEDIRS) $(FUZZ_OUTPUT_DIR)/$*
123
- $(Q) docker run -it --rm -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz /bin/bash -c "./fuzz/$*.sh /fuzz_output/$*"
123
+ $(Q) docker run -it --rm -v $(CURDIR):/prism -v $(FUZZ_OUTPUT_DIR):/fuzz_output prism/fuzz /bin/bash -c "./test/fuzz/$*.sh /fuzz_output/$*"
124
124
  FORCE:
125
125
 
126
126
  fuzz-clean:
127
- $(Q) $(RMALL) fuzz/output
127
+ $(Q) $(RMALL) test/fuzz/output
128
128
 
129
129
  clean:
130
130
  $(Q) $(RMALL) build
data/README.md CHANGED
@@ -49,7 +49,6 @@ The repository contains the infrastructure for both a shared library (libprism)
49
49
  │   └── prism
50
50
  │   ├── extconf.rb configuration to generate the Makefile for the native extension
51
51
  │   └── extension.c the native extension that interacts with libprism
52
- ├── fuzz files related to fuzz testing
53
52
  ├── gemfiles gemfiles used by different Ruby versions in CI
54
53
  ├── include
55
54
  │   ├── prism header files for the shared library
@@ -75,6 +74,9 @@ The repository contains the infrastructure for both a shared library (libprism)
75
74
  ├── templates contains ERB templates generated by templates/template.rb
76
75
  │   └── template.rb generates code from the nodes and tokens configured by config.yml
77
76
  └── test
77
+ ├── cpp tests for the C++ bindings
78
+ ├── embedded smoke test for bare-metal targets
79
+ ├── fuzz files related to fuzz testing
78
80
  └── prism
79
81
  ├── fixtures Ruby code used for testing
80
82
  └── snapshots snapshots of generated syntax trees corresponding to fixtures
data/docs/fuzzing.md CHANGED
@@ -1,9 +1,9 @@
1
1
  # Fuzzing
2
2
 
3
- We use fuzzing to test the various entrypoints to the library. The fuzzer we use is [AFL++](https://aflplus.plus). All files related to fuzzing live within the `fuzz` directory, which has the following structure:
3
+ We use fuzzing to test the various entrypoints to the library. The fuzzer we use is [AFL++](https://aflplus.plus). All files related to fuzzing live within the `test/fuzz` directory, which has the following structure:
4
4
 
5
5
  ```
6
- fuzz
6
+ test/fuzz
7
7
  ├── corpus
8
8
  │   └── parse fuzzing corpus for parsing (a symlink to our fixtures)
9
9
  ├── dict a AFL++ dictionary containing various tokens
@@ -57,12 +57,12 @@ make build/fuzz.heisenbug.parse
57
57
  Triaging crashes and hangs is easier when the inputs are as short as possible. In the fuzz container, an entire crash or hang directory can be minimized using
58
58
 
59
59
  ```
60
- ./fuzz/tools/minimize.sh <directory>
60
+ ./test/fuzz/tools/minimize.sh <directory>
61
61
  ```
62
62
 
63
63
  e.g.
64
64
  ```
65
- ./fuzz/tools/minimize.sh fuzz/output/parse/default/crashes
65
+ ./test/fuzz/tools/minimize.sh test/fuzz/output/parse/default/crashes
66
66
  ```
67
67
 
68
68
  This may take a long time. In the crash/hang directory, for each input file there will appear a minimized version with the extension `.min` appended.
@@ -70,7 +70,7 @@ This may take a long time. In the crash/hang directory, for each input file ther
70
70
  Backtraces for crashes (not hangs) can be generated en masse with
71
71
 
72
72
  ```
73
- ./fuzz/tools/backtrace.sh <directory>
73
+ ./test/fuzz/tools/backtrace.sh <directory>
74
74
  ```
75
75
 
76
76
  Files with basename equal to the input file name with extension `.bt` will be created e.g.
data/docs/serialization.md CHANGED
@@ -125,8 +125,14 @@ The header is structured like the following table:
125
125
  | varuint | content pool size |
126
126
 
127
127
  After the header comes the body of the serialized string.
128
- The body consists of a sequence of nodes that is built using a prefix traversal order of the syntax tree.
129
- Each node is structured like the following table:
128
+ The body consists of a sequence of nodes that is built using a postfix traversal order of the syntax tree: the children of each node come before the node itself, so the root node comes last.
129
+ The body ends at the content pool offset specified in the header.
130
+
131
+ The children of a node are the values of its `node`, `node?`, and `node[]` fields, which are serialized in the order in which the fields are listed in `config.yml`.
132
+ A `node` field is serialized as its node, a `node?` field is serialized as its node or as a single `0` byte if the node is not present, and a `node[]` field is serialized as each of its nodes in order.
133
+ A loader can therefore keep a stack of loaded nodes: a `0` byte pushes an absent node, and each node pops its children off the top of the stack (they are in field order) and then pushes itself.
134
+
135
+ After its children, each node is structured like the following table:
130
136
 
131
137
  | # bytes | field |
132
138
  | --- | --- |
@@ -135,12 +141,11 @@ Each node is structured like the following table:
135
141
  | location | node location |
136
142
  | varuint | node flags |
137
143
 
138
- Every field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
144
+ Every other field on the node is then appended to the serialized string. The fields can be determined by referencing `config.yml`. Depending on the type of field, it could take a couple of different forms, described below:
139
145
 
140
146
  * `double` - A field that is a `double`. This is structured as a sequence of 8 bytes in native endian order.
141
- * `node` - A field that is a node. This is structured just as like parent node.
142
- * `node?` - A field that is a node that is optionally present. If the node is not present, then a single `0` byte will be written in its place. If it is present, then it will be structured just as like parent node.
143
- * `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, followed by the child nodes themselves.
147
+ * `node`, `node?` - Nothing is appended here, since the node has already been serialized as a child.
148
+ * `node[]` - A field that is an array of nodes. This is structured as a variable-length integer length, which is the number of children that belong to this field.
144
149
  * `string` - A field that is a string. For example, this is used as the name of the method in a call node, since it cannot directly reference the source string (as in `@-` or `foo=`). This is structured as a variable-length integer byte length, followed by the string bytes (_without_ a trailing null byte).
145
150
  * `constant` - A variable-length integer that represents an index in the constant pool.
146
151
  * `constant?` - An optional variable-length integer that represents an index in the constant pool. If it's not present, then a single `0` byte will be written in its place.
@@ -150,6 +155,14 @@ Every field on the node is then appended to the serialized string. The fields ca
150
155
  * `uint8` - A field that is an 8-bit unsigned integer. This is structured as a single byte.
151
156
  * `uint32` - A field that is a 32-bit unsigned integer. This is structured as a variable-length integer.
152
157
 
158
+ The `body` field of a `DefNode` is lazy, which means that loaders can skip it and load it later. The children of a `DefNode` are preceded by the following, which loaders that do not skip lazy fields can ignore:
159
+
160
+ | # bytes | field |
161
+ | --- | --- |
162
+ | `1` | `255` |
163
+ | `4` | the byte length of the children that come before the `body` child |
164
+ | `4` | the byte length of the `body` child |
165
+
153
166
  After the syntax tree, the content pool is serialized. This is a list of constants that were referenced from within the tree. The content pool begins at the offset specified in the header. Every constant is embedded in the serialization. Each constant is structured as follows:
154
167
 
155
168
  | # bytes | field |