@amritk/nish 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -23
- package/bin/launcher.js +45 -41
- package/bin/packaging.js +39 -33
- package/docs/AI.md +95 -10
- package/docs/INSTALL.md +13 -13
- package/package.json +9 -6
- package/runtime/nish.d.ts +20 -1
- package/runtime/nish.h +31 -11
- package/runtime/nish.mjs +3 -3
- package/runtime/{runtime_os.c → runtime-os.c} +1 -1
- package/runtime/{runtime_parallel.c → runtime-parallel.c} +112 -4
- package/runtime/{runtime_wasm.c → runtime-wasm.c} +6 -1
- package/runtime/runtime.c +5 -5
- package/runtime/shim.mjs +6 -6
- package/scripts/bootstrap.sh +29 -29
- package/scripts/build.sh +14 -9
- package/scripts/changelog-gen.mjs +260 -192
- package/scripts/ci-profile.mjs +80 -68
- package/scripts/codes-registry.js +25 -13
- package/scripts/gen-diagnostic-codes.mjs +86 -69
- package/scripts/nish-compiler.sh +2 -0
- package/scripts/platform-package.mjs +26 -26
- package/scripts/postinstall.mjs +46 -36
- package/scripts/size-report.sh +3 -3
- package/scripts/smoke.sh +1 -1
- package/std/README.md +2 -2
- package/std/collections.ts +194 -188
- package/std/json.ts +136 -136
- package/std/map.ts +9 -7
- package/std/pair.ts +2 -2
- package/std/testing.ts +67 -67
- package/std/text.ts +54 -54
- package/std/threads.ts +126 -38
package/scripts/bootstrap.sh
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
#!/usr/bin/env bash
|
|
2
|
-
# Build the self-hosted compiler: `
|
|
2
|
+
# Build the self-hosted compiler: `src/`, compiled by `src/`.
|
|
3
3
|
#
|
|
4
4
|
# scripts/bootstrap.sh [-o <exe>] [--stages 1|2|3] [--profile speed|size|debug]
|
|
5
5
|
# [--work <dir>] [--verify] [--quiet]
|
|
@@ -8,9 +8,9 @@
|
|
|
8
8
|
# parameter rather than a fixture (docs/wp19-stage0-retirement.md §3, G3):
|
|
9
9
|
#
|
|
10
10
|
# seed whatever compiles stage1 NISH_BOOTSTRAP, or build/seed
|
|
11
|
-
# stage1
|
|
12
|
-
# stage2
|
|
13
|
-
# stage3
|
|
11
|
+
# stage1 src/, built by the seed
|
|
12
|
+
# stage2 src/, built by stage1 the default output
|
|
13
|
+
# stage3 src/, built by stage2 --verify only
|
|
14
14
|
#
|
|
15
15
|
# NISH_BOOTSTRAP=<path> names the seed, the way GOROOT_BOOTSTRAP names the Go
|
|
16
16
|
# that builds Go. It is either a released `nish`, executed directly, or a Node
|
|
@@ -31,14 +31,14 @@
|
|
|
31
31
|
#
|
|
32
32
|
# --verify runs the equalities the proof is made of, byte for byte:
|
|
33
33
|
#
|
|
34
|
-
# IR(seed,
|
|
35
|
-
# IR(stage1,
|
|
34
|
+
# IR(seed, src/) == IR(stage1, src/) reported, never asserted
|
|
35
|
+
# IR(stage1, src/) == IR(stage2, src/) the fixed point: self-hosted
|
|
36
36
|
# stage3 == stage2 as files, on ELF and on Mach-O
|
|
37
37
|
# alike; see below for what the
|
|
38
38
|
# Mach-O half took
|
|
39
39
|
#
|
|
40
40
|
# The last two are properties of the working tree and of nothing else: whatever
|
|
41
|
-
# built stage1, the compiler `
|
|
41
|
+
# built stage1, the compiler `src/` describes has to agree with itself and
|
|
42
42
|
# then reproduce itself. Both are asserted whatever the seed is.
|
|
43
43
|
#
|
|
44
44
|
# The third is a raw byte comparison, and on Mach-O two links of the same input
|
|
@@ -59,22 +59,22 @@
|
|
|
59
59
|
# measurement, byte offsets and all.
|
|
60
60
|
#
|
|
61
61
|
# The first one is reported and not asserted. With a released seed it is one
|
|
62
|
-
# implementation at two points in time: the IR HEAD emits for `
|
|
62
|
+
# implementation at two points in time: the IR HEAD emits for `src/` is the
|
|
63
63
|
# IR the last release emitted for it. Nothing in that sentence is about
|
|
64
64
|
# bootstrapping. It is a freeze on codegen between releases, and it fails on
|
|
65
65
|
# exactly the changes a release cycle exists to carry: the first improvement to
|
|
66
|
-
# land broke it, when a flow-sensitive bounds analysis proved 46 of `
|
|
66
|
+
# land broke it, when a flow-sensitive bounds analysis proved 46 of `src/`'s
|
|
67
67
|
# 1,206 index checks redundant and 20 of 56 modules "differed" because the
|
|
68
68
|
# optimisation worked. So the difference is reported, and the report is
|
|
69
69
|
# information about this release rather than a verdict on the bootstrap.
|
|
70
70
|
#
|
|
71
|
-
# While `src/` existed, a stage0 seed made the same comparison the second half
|
|
71
|
+
# While stage0's `src/` existed, a stage0 seed made the same comparison the second half
|
|
72
72
|
# of Wheeler's diverse double-compiling, and it was asserted there
|
|
73
|
-
# (docs/wp19-stage0-retirement.md §1, G6). That claim went with `src/`.
|
|
73
|
+
# (docs/wp19-stage0-retirement.md §1, G6). That claim went with stage0's `src/`.
|
|
74
74
|
#
|
|
75
75
|
# What the seeded run buys is not that equality. The rolling freeze — "a
|
|
76
|
-
# construct added in 0.N cannot be used by `
|
|
77
|
-
# by stage1 being built at all: a `
|
|
76
|
+
# construct added in 0.N cannot be used by `src/` until 0.(N+1)" — is enforced
|
|
77
|
+
# by stage1 being built at all: a `src/` that reaches for something the seed
|
|
78
78
|
# has never heard of does not compile, does not link, and never gets as far as
|
|
79
79
|
# a comparison. That failure is loud here whatever the seed is, and it is the
|
|
80
80
|
# whole of what the seeded run proves about the freeze
|
|
@@ -115,7 +115,7 @@ usage: scripts/bootstrap.sh [-o <exe>] [--stages 1|2|3] [--profile speed|size|de
|
|
|
115
115
|
|
|
116
116
|
Builds the self-hosted compiler. The seed builds stage1, stage1 builds stage2
|
|
117
117
|
(the default output), stage2 builds stage3. --verify compares the IR each stage
|
|
118
|
-
emits for
|
|
118
|
+
emits for src/ and the stage2/stage3 binaries, byte for byte. The result is
|
|
119
119
|
the command line itself: -o, --link, --profile and the rest.
|
|
120
120
|
|
|
121
121
|
IR(seed) == IR(stage1) asks whether codegen has changed since the seed was
|
|
@@ -156,10 +156,10 @@ build_to=$stages
|
|
|
156
156
|
[ "$verify" -eq 1 ] && build_to=3
|
|
157
157
|
|
|
158
158
|
# The published npm package ships bin/, runtime/, scripts/ and std/ but not
|
|
159
|
-
#
|
|
159
|
+
# src/ -- and, since 0.6.0, no compiler of its own at all -- so say which file
|
|
160
160
|
# is missing rather than failing inside the compiler.
|
|
161
|
-
if [ ! -f
|
|
162
|
-
echo "bootstrap:
|
|
161
|
+
if [ ! -f src/compile.ts ]; then
|
|
162
|
+
echo "bootstrap: src/compile.ts is missing; run this from a checkout of the repository" >&2
|
|
163
163
|
exit 3
|
|
164
164
|
fi
|
|
165
165
|
|
|
@@ -211,7 +211,7 @@ else
|
|
|
211
211
|
[ -r "$seed" ] || seed_die "is not readable"
|
|
212
212
|
command -v node >/dev/null 2>&1 || seed_die "needs node on PATH, which is not there"
|
|
213
213
|
fi
|
|
214
|
-
# A seed that cannot answer `--version` cannot compile
|
|
214
|
+
# A seed that cannot answer `--version` cannot compile src/ either — a
|
|
215
215
|
# binary built for another platform, a .js that is not a compiler — and
|
|
216
216
|
# finding that out here names the variable and the path the caller set, where
|
|
217
217
|
# finding it out in the stage1 link names a temporary file three stages deep.
|
|
@@ -223,7 +223,7 @@ fi
|
|
|
223
223
|
|
|
224
224
|
say() { [ "$quiet" -eq 1 ] || printf '%s\n' "$*"; }
|
|
225
225
|
|
|
226
|
-
# `a` and `b` hold one `.ll` per module of `
|
|
226
|
+
# `a` and `b` hold one `.ll` per module of `src/`. Both the module set and
|
|
227
227
|
# every byte of every module must match: a stage that emitted one module fewer
|
|
228
228
|
# has not agreed about the rest.
|
|
229
229
|
compare_ir() {
|
|
@@ -260,11 +260,11 @@ survey_ir() {
|
|
|
260
260
|
done
|
|
261
261
|
if [ "$differing" -eq 0 ]; then
|
|
262
262
|
say " note: IR(seed) vs IR(stage1): all $total modules identical."
|
|
263
|
-
say " Not asserted: codegen simply has not moved for
|
|
263
|
+
say " Not asserted: codegen simply has not moved for src/ since"
|
|
264
264
|
say " the seed was built."
|
|
265
265
|
else
|
|
266
266
|
say " note: IR(seed) vs IR(stage1): $differing of $total modules differ."
|
|
267
|
-
say " Not a bootstrap failure. Codegen has moved for
|
|
267
|
+
say " Not a bootstrap failure. Codegen has moved for src/ since the"
|
|
268
268
|
say " seed was built, which is what a release carries. See the header"
|
|
269
269
|
say " of this script and docs/wp19-stage0-retirement.md §3, G3."
|
|
270
270
|
fi
|
|
@@ -295,8 +295,8 @@ survey_ir() {
|
|
|
295
295
|
link_stage() {
|
|
296
296
|
local by="$1" name="$2"
|
|
297
297
|
rm -rf "$work/stage" "$work/stage.modules"
|
|
298
|
-
"$by"
|
|
299
|
-
rm -rf "$work/$name" "$work/$name.modules"
|
|
298
|
+
"$by" src/compile.ts --link "$work/stage" --profile "$profile" >/dev/null
|
|
299
|
+
rm -rf "${work:?}/${name:?}" "${work:?}/${name:?}.modules"
|
|
300
300
|
mv "$work/stage" "$work/$name"
|
|
301
301
|
mv "$work/stage.modules" "$work/$name.modules"
|
|
302
302
|
}
|
|
@@ -310,28 +310,28 @@ outdir=$(dirname "$out")
|
|
|
310
310
|
# `scripts/build.sh` itself (docs/wp14-selfhost.md §7a). That leaves each
|
|
311
311
|
# stage's IR in `<exe>.modules/`, which is where the equalities read it, and
|
|
312
312
|
# means the chain exercises the same driver a user does.
|
|
313
|
-
say "stage1:
|
|
313
|
+
say "stage1: src/ compiled by $seed_label"
|
|
314
314
|
rm -rf "$work/stage1.modules"
|
|
315
315
|
# This is the step the rolling freeze is enforced by, so it says so when it
|
|
316
316
|
# fails rather than leaving a reader with the compiler's own diagnostic and no
|
|
317
317
|
# idea which rule it just met.
|
|
318
|
-
if ! run_seed
|
|
319
|
-
echo "bootstrap: $seed_label could not build stage1 from
|
|
318
|
+
if ! run_seed src/compile.ts --link "$work/stage1" --profile "$profile" >/dev/null; then
|
|
319
|
+
echo "bootstrap: $seed_label could not build stage1 from src/ (its output is above)" >&2
|
|
320
320
|
echo "bootstrap: if it refused the source, that is the freeze doing its job: a construct" >&2
|
|
321
|
-
echo "bootstrap: added in 0.N cannot be used by
|
|
321
|
+
echo "bootstrap: added in 0.N cannot be used by src/ until 0.(N+1) (docs/wp12-release.md," >&2
|
|
322
322
|
echo "bootstrap: \"The bootstrap seed\"; docs/wp19-stage0-retirement.md §3, G3)" >&2
|
|
323
323
|
exit 1
|
|
324
324
|
fi
|
|
325
325
|
|
|
326
326
|
if [ "$build_to" -ge 2 ]; then
|
|
327
|
-
say "stage2:
|
|
327
|
+
say "stage2: src/ compiled by stage1"
|
|
328
328
|
link_stage "$work/stage1" stage2
|
|
329
329
|
# The seed equality, reported and not asserted.
|
|
330
330
|
[ "$verify" -eq 1 ] && survey_ir "$work/stage1.modules" "$work/stage2.modules"
|
|
331
331
|
fi
|
|
332
332
|
|
|
333
333
|
if [ "$build_to" -ge 3 ]; then
|
|
334
|
-
say "stage3:
|
|
334
|
+
say "stage3: src/ compiled by stage2"
|
|
335
335
|
link_stage "$work/stage2" stage3
|
|
336
336
|
[ "$verify" -eq 1 ] && compare_ir "$work/stage2.modules" "$work/stage3.modules" "IR(stage1) == IR(stage2)"
|
|
337
337
|
# `stage3 == stage2`, in scripts/verify-binaries.sh: byte-identical, which
|
package/scripts/build.sh
CHANGED
|
@@ -4,9 +4,9 @@
|
|
|
4
4
|
# scripts/build.sh <module.ll> [more .ll/.c files...] -o <out> [--profile debug|speed|size|wasm]
|
|
5
5
|
#
|
|
6
6
|
# The C runtime is three translation units and is named as one: an input
|
|
7
|
-
# <dir>/runtime.c also compiles <dir>/
|
|
7
|
+
# <dir>/runtime.c also compiles <dir>/runtime-os.c, the half that wraps the
|
|
8
8
|
# system calls (files, directories, subprocesses, the environment, the clock),
|
|
9
|
-
# and <dir>/
|
|
9
|
+
# and <dir>/runtime-parallel.c, the half that divides a range of work across
|
|
10
10
|
# threads. Each of those files says why they are compiled and measured apart.
|
|
11
11
|
#
|
|
12
12
|
# Profiles:
|
|
@@ -15,7 +15,7 @@
|
|
|
15
15
|
# size -Oz + LTO + section GC + strip + no unwind tables. Rust
|
|
16
16
|
# `opt-level="z"`, `panic="abort"`, `strip=true` equivalent.
|
|
17
17
|
# wasm wasm32 freestanding module exporting every non-internal function;
|
|
18
|
-
# load it from Node. Add runtime/
|
|
18
|
+
# load it from Node. Add runtime/runtime-wasm.c to the inputs when a
|
|
19
19
|
# function uses arrays (the arena and the array cold paths, no libc);
|
|
20
20
|
# strings and I/O still need a WASI runtime and are not available.
|
|
21
21
|
# wasm wasm32 freestanding module exporting every non-internal function
|
|
@@ -84,12 +84,12 @@ done
|
|
|
84
84
|
[ -n "$out" ] || { echo "error: -o <out> is required" >&2; exit 2; }
|
|
85
85
|
|
|
86
86
|
# The runtime is three translation units, and a caller names one: whoever passes
|
|
87
|
-
# <dir>/runtime.c gets <dir>/
|
|
87
|
+
# <dir>/runtime.c gets <dir>/runtime-os.c and <dir>/runtime-parallel.c compiled
|
|
88
88
|
# beside it. They were one file until the operating-system half was split out
|
|
89
89
|
# for its own size budget, and the parallel half followed for the same reason
|
|
90
90
|
# (each file's header comment says why), and a link line is where those splits
|
|
91
91
|
# would otherwise leak: `nish --link` builds its command line in
|
|
92
|
-
#
|
|
92
|
+
# src/compile.ts, the published package's recipe in every document and
|
|
93
93
|
# README names runtime.c, and a user's own clang line does too. Pairing them
|
|
94
94
|
# here keeps every one of those correct, and keeps "the runtime" one thing to
|
|
95
95
|
# name from the outside. A caller that names one itself is left alone, because
|
|
@@ -97,7 +97,7 @@ done
|
|
|
97
97
|
for i in ${inputs[@]+"${inputs[@]}"}; do
|
|
98
98
|
case "$i" in
|
|
99
99
|
*/runtime.c|runtime.c)
|
|
100
|
-
for half in
|
|
100
|
+
for half in runtime-os.c runtime-parallel.c; do
|
|
101
101
|
side="${i%runtime.c}$half"
|
|
102
102
|
have=0
|
|
103
103
|
for j in "${inputs[@]}"; do
|
|
@@ -133,8 +133,12 @@ case "$(uname -s)" in
|
|
|
133
133
|
# of one input to one output path, and differing on a link to another. That
|
|
134
134
|
# is what scripts/bootstrap.sh links every comparable stage at one path for,
|
|
135
135
|
# and it is why `stage3 == stage2` holds on Mach-O with the load command in.
|
|
136
|
-
|
|
136
|
+
# shellcheck disable=SC2054 # -Wl,<flag> is one argument: the comma is the linker's
|
|
137
|
+
gc=(-Wl,-dead_strip)
|
|
138
|
+
# shellcheck disable=SC2054
|
|
139
|
+
strip_flag=(-Wl,-x) ;;
|
|
137
140
|
*)
|
|
141
|
+
# shellcheck disable=SC2054 # -Wl,<flag> is one argument: the comma is the linker's
|
|
138
142
|
gc=(-Wl,--gc-sections -Wl,--as-needed -Wl,-O2 -Wl,--build-id=none); strip_flag=(-s)
|
|
139
143
|
elf=(-fno-plt)
|
|
140
144
|
# GNU ld needs the gold plugin for LTO; prefer lld when clang can find it.
|
|
@@ -158,7 +162,7 @@ if [ "$debug" = 1 ]; then common+=(-g); strip_flag=(); fi
|
|
|
158
162
|
# own command line instead of using `common`; it is empty on every ordinary
|
|
159
163
|
# build, hence the bash 3.2 expansion spelling explained below.
|
|
160
164
|
#
|
|
161
|
-
# -pthread is for
|
|
165
|
+
# -pthread is for runtime-parallel.c, the translation unit that divides a range
|
|
162
166
|
# of work across threads: it is compiled in either configuration and only spawns
|
|
163
167
|
# under this macro, so this is the build where the flag has to be on the command
|
|
164
168
|
# line. On a current glibc the library half is already inside libc and the link
|
|
@@ -167,7 +171,7 @@ if [ "$debug" = 1 ]; then common+=(-g); strip_flag=(); fi
|
|
|
167
171
|
#
|
|
168
172
|
# It is deliberately in `common` and not in `tls`: `tls` is the wasm and wasi
|
|
169
173
|
# command lines, which have no threads to link against, and where
|
|
170
|
-
#
|
|
174
|
+
# runtime-parallel.c compiles to its sequential fallback because it tests
|
|
171
175
|
# __wasi__ and __wasm__ as well as the macro.
|
|
172
176
|
tls=()
|
|
173
177
|
if [ "$threads" = 1 ]; then
|
|
@@ -265,6 +269,7 @@ case "$profile" in
|
|
|
265
269
|
shared=(-shared -fPIC)
|
|
266
270
|
# macOS: the napi_* symbols come from the node binary at load time, so the
|
|
267
271
|
# linker must not insist on resolving them. ELF shared objects allow this.
|
|
272
|
+
# shellcheck disable=SC2054 # -Wl,<flag> is one argument: the comma is the linker's
|
|
268
273
|
case "$(uname -s)" in Darwin) shared+=(-Wl,-undefined,dynamic_lookup) ;; esac
|
|
269
274
|
# runtime/nish.h is the public ABI header the generated shim includes.
|
|
270
275
|
runtime_inc="$(cd "$(dirname "$0")/../runtime" && pwd)"
|