@amritk/nish-x86_64-linux 0.21.0 → 0.22.1

package/INSTALL.md

@@ -143,6 +143,36 @@ for — which catches a truncated download, a tarball built for another
 architecture, and an asset whose name does not match what is inside it — and
 only then swapped in. If any of that fails the install you had is untouched.
 
+**Every download is checked before it is unpacked.** install.sh compares the
+tarball's SHA-256 with the release's `SHA256SUMS` (or with the digest pinned
+for releases that predate it) and refuses a mismatch. For releases after
+0.21.0 it also checks the release's **build provenance attestation** when a
+signed-in GitHub CLI is present whose `gh attestation verify` takes the
+options below. The attestation
+is a Sigstore-signed statement, made by `actions/attest-build-provenance` in
+the release workflow, that every file in `SHA256SUMS` was built by
+`.github/workflows/release.yml` at the release's tag on a GitHub-hosted
+runner. A failed verification refuses the install; with no `gh`, a signed-out
+one or one too old for those options, the check is skipped with a note saying so, and the
+SHA-256 still applies. The same command by hand, on a tarball you downloaded:
+
+```bash
+gh attestation verify nish-<version>-<asset>.tar.gz \
+  --repo amritk/nish \
+  --signer-workflow amritk/nish/.github/workflows/release.yml \
+  --source-ref refs/tags/v<version> \
+  --deny-self-hosted-runners
+```
+
+It works on any file the release attaches, the npm tarballs included. Each
+release also attaches `nish-<version>.spdx.json`, an SPDX 2.3 software bill of
+materials: every attached file with its SHA-256, the clang, linker, Node.js
+and seed compiler that built each tarball (recorded by the build row itself),
+and the vendored code from
+[THIRD_PARTY_NOTICES.md](../THIRD_PARTY_NOTICES.md) that each of them contains.
+How these fit the rest of the release's defences is
+[docs/security/supply-chain.md](security/supply-chain.md).
+
 The npm route installs the **native** compiler, and it is a download rather than a
 build: nothing is compiled on your machine. The package declares one
 `nish-<os>-<arch>` package per supported platform as an `optionalDependencies`

package/docs/AI.md

@@ -93,6 +93,24 @@ compiles and the warning would be printed: `NL9007`'s inserts the guard that
 proves an index in range (not under `--no-warn-performance`), and `NL7002`'s
 migrates a site off `--unchecked-indexing` (below).
 
+When it compiles, **check the native build against the program's TypeScript
+reading**. Run the project's tests natively; where it has none, write the
+program back out and run both:
+
+```bash
+nish program.ts --emit-ts out/    # the program as TypeScript, plus out/entry.ts
+node out/entry.ts                 # Node 22.18 or newer; compare with the binary
+```
+
+The two must print the same and exit the same, a panic included: `--emit-ts`
+keeps the overflow and bounds checks the native build makes, but not a ranged
+integer's range (`integer<0, 255>` is a plain `number`). Where they differ,
+the difference is a row of [the user guide's table](guide/using-nish/to-typescript.md),
+or a bug in `nish` worth reporting. A program on `nish:net`, the signal
+descriptor or a C declaration (NL4010, NL4011), a compound assignment whose
+index would run twice (NL4012) or a generic whose element type decides its
+code (NL4013, NL4014) is refused and has no TypeScript to run.
+
 Two more surfaces worth knowing: `nish --emit-ast f.ts` prints what was parsed
 and `nish --emit-checked f.ts` prints the side tables the emitter reads. Both
 are one node per line, which is to say diffable.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@amritk/nish-x86_64-linux",
-  "version": "0.21.0",
+  "version": "0.22.1",
   "description": "The nish native compiler for Linux on x86_64",
   "license": "MIT",
   "os": [

package/runtime/browser.d.ts

@@ -3,12 +3,4 @@
  * in `nish/runtime` (docs/guide/using-nish/to-typescript.md).
  */
 
-/**
- * What the program's `exit(code)`, and every panic, throws: a page cannot end
- * itself. Tell it apart with `instanceof`, not by its `name`.
- */
-export class NishExit extends Error {
-  /** The 8-bit status a native exit would leave: `code & 0xff`. */
-  readonly code: number;
-  constructor(code: number);
-}
+export { NishExit } from "./core/host.js";

package/runtime/browser.mjs

@@ -16,20 +16,9 @@
  *   - Every file-system, process and machine call panics, naming itself and
  *     the browser.
  */
-import { installHost, panic } from "./core/host.mjs";
+import { installHost, NishExit, panic } from "./core/host.mjs";
 
-/**
- * What `exit(code)` throws: `code` is the 8-bit status a native exit would
- * leave. Exported so a caller tells it apart with `instanceof`, which a thrown
- * object that only carries the name does not pass.
- */
-export class NishExit extends Error {
-  constructor(code) {
-    super(`the program exited with ${code}`);
-    this.name = "NishExit";
-    this.code = code & 0xff;
-  }
-}
+export { NishExit };
 
 /**
  * One console stream, buffered to the next newline. The `"%s"` keeps a `%d`

package/runtime/core/arrays.d.ts

@@ -1,18 +1,36 @@
 /** The array half of the core: bounds-checked access, and the arena stubs. */
-import type { i32 } from "./numbers.js";
+import type { i32, i64 } from "./numbers.js";
 export declare function idx<T>(a: readonly T[], i: number | bigint): T;
 export declare function pop<T>(a: T[]): T;
 export declare function setIdx<T>(a: T[], i: number | bigint, v: T): T;
 export declare function updIdx<T>(a: T[], i: number | bigint, f: (old: T) => T): T;
-export declare function uncheckedGet<T extends number>(xs: readonly T[], i: i32): T;
-export declare function uncheckedSet<T extends number>(xs: T[], i: i32, v: T): void;
+export declare function uncheckedGet<T extends number | bigint>(xs: readonly T[], i: i32): T;
+export declare function uncheckedSet<T extends number | bigint>(xs: T[], i: i32, v: T): void;
 export declare function copyIn<T>(v: T): T;
 export declare function storeIdx<T>(a: T[], i: number | bigint, v: T): T;
 export declare function arraySet<T>(dst: T[], src: readonly T[], offset?: number): void;
-export declare function newArray<T>(n: number | bigint, zero: T): T[];
-export declare function arenaUsed(): number;
-export declare function arenaMark(): number;
+/** `dst.set(src, offset)` as `--emit-ts` writes it: a 64-bit offset is a bigint, checked exactly. */
+export declare function arraySetAt<T>(dst: T[], src: readonly T[], offset?: number | bigint): void;
+/** `new Array<T>(n)`: `--emit-ts` writes `T` out, so the zero it fills with is not what `T` is read from. */
+export declare function newArray<T>(n: number | bigint, zero: unknown): T[];
+export declare function arenaUsed(): i64;
+export declare function arenaMark(): i64;
 export declare function arenaRelease(m?: unknown): void;
 export declare function arenaReset(): void;
-/** `using a = arena()`: an object whose `Symbol.dispose` does nothing. */
-export declare function arena(): object;
+/**
+ * The protocol `using` reads, declared here as `nish.d.ts` does, so `using a =
+ * arena()` type-checks with no `"esnext.disposable"` in a program's `lib`.
+ */
+declare global {
+  interface SymbolConstructor {
+    readonly dispose: unique symbol;
+  }
+}
+
+/** What `arena()` answers: its `Symbol.dispose` does nothing. */
+export interface ArenaHandle {
+  [Symbol.dispose](): void;
+}
+
+/** `using a = arena()`: the arena is the collector's, so leaving it frees nothing. */
+export declare function arena(): ArenaHandle;

package/runtime/core/arrays.mjs

@@ -79,6 +79,22 @@ export function storeIdx(a, i, v) {
   return v;
 }
 
+/**
+ * `dst.set(src, offset)` as `nish --emit-ts` writes it: an `i64` or `u64`
+ * offset, a bigint here, is compared as one, so a value past 2^53 fails with
+ * the native panic rather than being rounded into range, and one in range is
+ * the number `arraySet` takes. `runtime/nish.mjs` keeps `arraySet`, whose
+ * bigint offset throws as `docs/LANGUAGE.md` says.
+ */
+export function arraySetAt(dst, src, offset) {
+  if (typeof offset === "bigint") {
+    const end = offset + BigInt(src.length);
+    if (offset < 0n || end > BigInt(dst.length)) panicSlice(offset, end, dst.length);
+    offset = Number(offset);
+  }
+  arraySet(dst, src, offset);
+}
+
 /**
  * `dst.set(src, offset)` (WP34 N2): `TypedArray.prototype.set`'s copy on the
  * plain array a `u8[]` is here. The source is copied first, so a self-copy or
@@ -94,13 +110,25 @@ export function arraySet(dst, src, offset) {
   // to the nearest double past 2^53 (docs/RUN_UNDER_NODE.md).
   const at = offset === undefined ? 0 : Math.trunc(offset) || 0;
   const end = at + src.length;
-  if (!(at >= 0 && end <= dst.length)) panicSlice(at, end, dst.length);
+  if (!(at >= 0 && end <= dst.length)) panicSlice(...saturatedRange(at, src.length), dst.length);
   // Two plain arrays overlap only when they are one array, which is the one
   // case that must read the source before writing it.
   const from = src === dst ? src.slice() : src;
   for (let i = 0; i < from.length; i++) dst[at + i] = from[i];
 }
 
+/**
+ * `[at, at + n)` as the native panic spells it: the offset saturated into an
+ * `i64`, as `llvm.fptosi.sat` leaves it, and the end wrapped, so `1e300` is
+ * `9223372036854775807` and its end `-9223372036854775808`.
+ */
+function saturatedRange(at, n) {
+  if (Number.isSafeInteger(at + n)) return [at, at + n];
+  const max = 2n ** 63n - 1n;
+  const start = at === Infinity || at >= 2 ** 63 ? max : at <= -(2 ** 63) ? -max - 1n : BigInt(at);
+  return [start, BigInt.asIntN(64, start + BigInt(n))];
+}
+
 /** `new Array<T>(n)`: `n` zero-filled elements (`0`, `0n`, or `false`). */
 export function newArray(n, zero) {
   return new Array(toIndex(n)).fill(zero);
@@ -115,8 +143,8 @@ export function newArray(n, zero) {
 // so "the arena of the calling thread" and "the arena" are the same object, and
 // these stubs answer for both. If T1 ever lands a spawn the rewrite can reach,
 // that is when this file grows a second arena to keep count of.
-export function arenaUsed() { return 0; }
-export function arenaMark() { return 0; }
+export function arenaUsed() { return 0n; }
+export function arenaMark() { return 0n; }
 export function arenaRelease() {}
 export function arenaReset() {}
 /** `using a = arena()`: a disposable whose disposal does nothing, because there is no arena to release. */

package/runtime/core/host.d.ts

@@ -20,6 +20,16 @@ export declare class NishNoHost extends Error {
   readonly requested: string;
 }
 
+/**
+ * What the browser host's `exit(code)`, and every panic there, throws: a page
+ * cannot end itself. Tell it apart with `instanceof`, not by its `name`.
+ */
+export declare class NishExit extends Error {
+  /** The 8-bit status a native exit would leave: `code & 0xff`. */
+  readonly code: number;
+  constructor(code: number);
+}
+
 export declare function installHost(next: Host): void;
 export declare function log(x: unknown): void;
 export declare function error(x: unknown): void;

package/runtime/core/host.mjs

@@ -48,6 +48,20 @@ const NO_HOST = {
 
 let host = NO_HOST;
 
+/**
+ * What `exit(code)` throws: `code` is the 8-bit status a native exit would
+ * leave. The browser host throws it, since a page cannot end itself; it is
+ * exported here and from `nish/runtime/browser` so a caller tells it apart with `instanceof`, which a thrown
+ * object that only carries the name does not pass.
+ */
+export class NishExit extends Error {
+  constructor(code) {
+    super(`the program exited with ${code}`);
+    this.name = "NishExit";
+    this.code = code & 0xff;
+  }
+}
+
 /** Make `next` the host every later call goes through. */
 export function installHost(next) {
   host = next;

package/runtime/core/numbers.d.ts

@@ -12,6 +12,16 @@ export type u32 = number;
 export type u64 = bigint;
 export type f32 = number;
 export type f64 = number;
+/**
+ * A ranged integer (docs/wp31-ranged-integers.md) is a `number`; its range is
+ * the native build's check, which Node does not make (wp33 §3.3, the range row).
+ */
+export type integer<Lo extends number, Hi extends number> = number;
+/** The typed-array aliases (docs/LANGUAGE.md): in Nish each is an ordinary array of its width. */
+export type Int32Array = i32[];
+export type Float32Array = f32[];
+export type Float64Array = f64[];
+export type BigInt64Array = i64[];
 
 /** A width as the rewriter names it, for `convert`. */
 export type NumericKind = "i32" | "i64" | "u8" | "u16" | "u32" | "u64" | "f32" | "f64";
@@ -46,7 +56,10 @@ export declare function shlU64(a: u64, b: u64): u64;
 export declare function lshrU64(a: u64, b: u64): u64;
 export declare function minU64(a: u64, b: u64): u64;
 export declare function maxU64(a: u64, b: u64): u64;
-export declare function convert(x: number | bigint, from: NumericKind, to: NumericKind): number | bigint;
+/** What `convert` answers for the width `K`: a `bigint` for the 64-bit integers. */
+export type Width<K extends NumericKind> = K extends "i64" | "u64" ? bigint : number;
+
+export declare function convert<K extends NumericKind>(x: number | bigint, from: NumericKind, to: K): Width<K>;
 export declare function parseInt(s: string): i32;
 export declare function parseFloat(s: string): f64;
 export declare function number(x: string | number | bigint | boolean): f64;

package/runtime/core/numbers.mjs

@@ -248,9 +248,36 @@ export function maxU64(a, b) {
  * what `sext`/`zext`/`trunc` do natively.
  */
 export function convert(x, from, to) {
-  // A zext of an unsigned 32-bit-or-narrower value to u64 keeps the value as it
-  // is, and skips the BigInt wrap below: a hot path in u64 arithmetic.
-  if (to === "u64" && (from === "u32" || from === "u16" || from === "u8")) return BigInt(x);
+  // The hot paths of u64 arithmetic written over i64 (`toU64(a) * toU64(b)`)
+  // stay in a function small enough for V8 to inline at the call: the two
+  // 64-bit kinds reinterpret each other's bits, and a zext of an unsigned
+  // 32-bit-or-narrower value to u64 keeps the value as it is.
+  if (typeof x === "bigint") {
+    if (to === "u64") return BigInt.asUintN(64, x);
+    if (to === "i64") return BigInt.asIntN(64, x);
+  } else if (to === "u64" && (from === "u32" || from === "u16" || from === "u8")) {
+    return BigInt(x);
+  }
+  return convertOther(x, from, to);
+}
+
+/** Every conversion `convert` does not answer inline. */
+function convertOther(x, from, to) {
+  if (typeof x === "bigint") {
+    // A BigInt source narrowed to 32 bits or less skips the `number` round trip.
+    const narrow = INT_KINDS[to];
+    if (narrow !== undefined) {
+      return Number(narrow.signed ? BigInt.asIntN(narrow.bits, x) : BigInt.asUintN(narrow.bits, x));
+    }
+  } else if (from === "i32" || from === "u32" || from === "u16" || from === "u8") {
+    // An integer `number` source narrowed to a `number` kind: the 32-bit wrap
+    // `|0` / `>>>0` and a mask do what trunc/sext/zext do, without a BigInt.
+    if (to === "i32") return x | 0;
+    if (to === "u32") return x >>> 0;
+    if (to === "u16") return x & 0xffff;
+    if (to === "u8") return x & 0xff;
+    if (to === "i64") return BigInt(x);
+  }
   const num = typeof x === "bigint" ? Number(x) : x;
   if (to === "f64") return num; // sitofp / uitofp / fpext
   if (to === "f32") return Math.fround(num); // ... / fptrunc, then rounded to a float

package/runtime/core/result.d.ts

@@ -25,6 +25,10 @@ export declare const Result: { prototype: ResultMethods<unknown, unknown> };
 export declare function Ok<T, E>(value: T): Result<T, E>;
 export declare function Ok<E>(): Result<void, E>;
 export declare function Err<T, E>(error: E): Result<T, E>;
+/** `r.expect(message())`, with `message` called on the `Err` path only. */
+export declare function expectLazy<T, E>(r: Result<T, E>, message: () => string): T;
+/** `r.unwrapOr(fallback())`, with `fallback` called on the `Err` path only. */
+export declare function unwrapOrLazy<T, E>(r: Result<T, E>, fallback: () => T): T;
 /** The `catch` of a body that calls `orReturn`: the `Err` to return, or `thrown` thrown again. */
 export declare function caught<T, E>(thrown: unknown): Result<T, E>;
 

package/runtime/core/result.mjs

@@ -27,6 +27,19 @@ export function exposeWith(s, arg, f) {
   return f(s.value, arg);
 }
 
+/**
+ * `r.expect(message)` and `r.unwrapOr(fallback)` as `--emit-ts` writes them
+ * when the argument is more than a name or a literal: natively it is
+ * evaluated on the `Err` path only (#182), so here it is a function of none.
+ */
+export function expectLazy(r, message) {
+  return r.ok ? r.value : r.expect(message());
+}
+
+export function unwrapOrLazy(r, fallback) {
+  return r.ok ? r.value : fallback();
+}
+
 /** Zero an array of integers, or every integer field of a record: `0` for a number, `0n` for an `i64` or `u64`. */
 function zeroFill(target) {
   if (Array.isArray(target) || ArrayBuffer.isView(target)) {

package/runtime/core/strings.d.ts

@@ -10,5 +10,5 @@ export declare function indexOf(s: string, sub: string, from?: number | bigint):
 export declare function startsWith(s: string, sub: string): boolean;
 export declare function endsWith(s: string, sub: string): boolean;
 export declare function fromCharCode(code: number | bigint): string;
-export declare function fromBytes(bytes: readonly u8[], start: number, end: number): string;
-export declare function utf8FromBytes(bytes: readonly u8[], start: number, end: number): string;
+export declare function fromBytes(bytes: readonly u8[], start: number | bigint, end: number | bigint): string;
+export declare function utf8FromBytes(bytes: readonly u8[], start: number | bigint, end: number | bigint): string;

package/runtime/nish.d.ts

@@ -353,7 +353,7 @@ declare function netWrite(fd: i32, buf: readonly u8[], off: i32, len: i32): i32;
 declare function netShutdown(fd: i32, how: i32): i32;
 /** Close the descriptor. */
 declare function netClose(fd: i32): i32;
-/** A TCP socket connecting to `addr`, answered while it connects: writable once it has, or has failed. */
+/** A TCP socket connecting to `addr`, answered while it connects: writable once it has, or has failed. It holds `TCP_NODELAY`, so a small write is sent at once. */
 declare function tcpConnect(addr: readonly u8[]): i32;
 /** Once `fd` is writable: 0 when the connection is made, else its failure (-111 refused). */
 declare function connectResult(fd: i32): i32;
@@ -510,7 +510,7 @@ declare module "nish:net" {
   /** Shut the read side (0), the write side (1) or both (2). */
   export function netShutdown(fd: i32, how: i32): i32;
   export function netClose(fd: i32): i32;
-  /** A TCP socket connecting to `addr`, answered while it connects: writable once it has, or has failed. */
+  /** A TCP socket connecting to `addr`, answered while it connects: writable once it has, or has failed. It holds `TCP_NODELAY`, so a small write is sent at once. */
   export function tcpConnect(addr: readonly u8[]): i32;
   /** Once `fd` is writable: 0 when the connection is made, else its failure (-111 refused). */
   export function connectResult(fd: i32): i32;

package/runtime/node.mjs

@@ -378,6 +378,9 @@ installHost({
   writeOut: (s) => fs.writeSync(1, s),
   writeErr: (s) => fs.writeSync(2, s),
   exit: (code) => process.exit(code),
+  // Index 0 is the program on both sides: the executable natively, the entry
+  // script here, which is one index later than Node's own `argv`. The global
+  // is left as Node set it; the output reads `argv()` (`src/emit-ts-records.ts`).
   argv: () => process.argv.slice(1),
   call: (name, args) => CALLS[name](...args),
 });

package/runtime/runtime-net.c

@@ -318,13 +318,24 @@ int32_t nish_tcp_accept(int32_t fd, nish_array *peer) {
    descriptor is answered all the same, and the connection is made once the
    loop reports it writable, when `connectResult` says how it went. Any other
    failure closes the socket and is answered instead of it. An `addr` shorter
-   than 18 bytes is -22 before any socket exists. */
+   than 18 bytes is -22 before any socket exists. The socket holds
+   `TCP_NODELAY` before it connects, as libpq and Go's net package set it on
+   every client socket: a client writes whole requests and waits on the
+   answer, and Nagle would hold a small write behind an unacknowledged one
+   for a delayed ACK (#942). A refused option closes the socket and is
+   answered instead of it. */
 int32_t nish_tcp_connect(const nish_array *addr) {
   if (addr->len < NISH_ADDR_BYTES) return -22;
   const unsigned char *p = (const unsigned char *)addr->data;
   int v4 = memcmp(p, nish_mapped_prefix, 12) == 0;
   int fd = nish_net_socket(v4 ? AF_INET : AF_INET6, SOCK_STREAM);
   if (fd < 0) return nish_net_fail();
+  int one = 1;
+  if (setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof one) != 0) {
+    int e = errno;
+    close(fd);
+    return nish_net_err(e);
+  }
   nish_sockaddr s;
   socklen_t n = nish_net_sockaddr(&s, p, (p[16] << 8) | p[17], v4);
   if (connect(fd, &s.sa, n) != 0 && errno != EINPROGRESS) {

package/scripts/build.sh

@@ -334,6 +334,39 @@ if [ "$debug" = 1 ]; then common+=(-g); strip_flag=(); fi
 # float than the baseline build (docs/LANGUAGE.md, "The CPU level").
 if [ -n "$cpu" ]; then common+=(-march=x86-64-v3 -ffp-contract=off); fi
 
+# Hardening, on every native profile (the wasm and wasi ones build their own
+# command line and have none of these to give). docs/security/runtime.md
+# records what each one closes and what it costs.
+#
+# The control-flow flag names the target's own scheme: -fcf-protection is an
+# error on arm64 and -mbranch-protection one on x86, so the triple decides.
+# Nish's IR carries both targets' module flags, and each backend reads its own;
+# these make clang's C units agree, because the linker marks the binary only
+# when every object asks for it. -fstack-clash-protection is ELF's alone: on
+# Mach-O clang ignores it with a warning, and the kernel's guard is the stack
+# itself. -z relro -z now leaves the GOT read-only after relocation, and
+# -z noexecstack says so even when an object without the note slips in.
+#
+# -fstack-protector-strong and _FORTIFY_SOURCE reach the C runtime alone: on a
+# .ll input clang adds no ssp attribute and the macro has no source to change,
+# so one flag list serves the one-command link too. FORTIFY wants an optimiser
+# and is left out of debug, where it would only warn. -U first, because a
+# distribution clang may already define it at another level.
+case "$("$CC" -dumpmachine 2>/dev/null)" in
+  x86_64*|amd64*) common+=(-fcf-protection=full); landing=1 ;;
+  aarch64*|arm64*) common+=(-mbranch-protection=standard); landing=1 ;;
+  *) landing=0 ;;
+esac
+common+=(-fstack-protector-strong)
+if [ "$profile" != debug ]; then common+=(-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3); fi
+case "$(uname -s)" in
+  Darwin) ;;
+  *)
+    common+=(-fstack-clash-protection)
+    # shellcheck disable=SC2054 # -Wl,<flag> is one argument: the comma is the linker's
+    common+=(-Wl,-z,relro,-z,now -Wl,-z,noexecstack) ;;
+esac
+
 tls=()
 if [ "$threads" = 1 ]; then
   common+=(-DNISH_THREADS=1 -ftls-model=initial-exec -pthread)
@@ -353,7 +386,7 @@ case "$profile" in
     optimise=(-Oz -flto -DNDEBUG ${pgo[@]+"${pgo[@]}"}
       -ffunction-sections -fdata-sections -fomit-frame-pointer
       -fno-asynchronous-unwind-tables -fno-unwind-tables ${elf[@]+"${elf[@]}"}
-      -fno-stack-protector -fvisibility=hidden) ;;
+      -fvisibility=hidden) ;;
   napi)
     optimise=(-O3 -flto -DNDEBUG ${pgo[@]+"${pgo[@]}"}
       -ffunction-sections -fdata-sections -fomit-frame-pointer
@@ -362,14 +395,18 @@ esac
 
 # The speed link splits full LTO's code generation into one partition per core,
 # run in parallel: lld's --lto-partitions, after the whole program is optimised
-# as one module, so inlining and the instructions a program runs do not change
-# (bench/instructions.json agrees to a few instructions in hundreds of
-# millions). The bytes depend on the count, so one machine links one input to
-# one output, as scripts/bootstrap.sh needs. ld64 and GNU ld have no such flag
-# and link as before, and so does every other profile. The count comes
-# from nproc (Linux), else getconf (both), else sysctl (macOS).
+# as one module, so inlining does not change. But a partition boundary promotes
+# the internal functions called across it to external ones, and under the
+# control-flow hardening above every external function opens with a landing pad
+# (endbr64, or bti c), which the hot calls then run: fib runs 6.5% more
+# instructions with two partitions than with one. So a target with landing pads
+# links in one partition, and the split is left to the rest. The bytes depend
+# on the count, so one machine links one input to one output, as
+# scripts/bootstrap.sh needs. ld64 and GNU ld have no such flag and link as
+# before, and so does every other profile. The count comes from nproc (Linux),
+# else getconf (both), else sysctl (macOS).
 lto_parts=()
-if [ "$profile" = speed ] && [ "$lld" = 1 ]; then
+if [ "$profile" = speed ] && [ "$lld" = 1 ] && [ "$landing" = 0 ]; then
   cores=$(nproc 2>/dev/null || getconf _NPROCESSORS_ONLN 2>/dev/null ||
     sysctl -n hw.ncpu 2>/dev/null || echo 1)
   # shellcheck disable=SC2054 # -Wl,<flag> is one argument: the comma is the linker's
@@ -379,15 +416,22 @@ fi
 # --runtime-objects and --runtime-from: the one-command link split in two, with
 # every compile it held run at once. A `$CC -c` takes `common` and `optimise`,
 # which is what the one command compiles each input with -- less
-# -fuse-ld=lld, which only names the linker -- so a runtime unit compiles to the
-# same object either way: native code for debug, -flto bitcode for speed and
-# size, which the link still optimises together with the program. The link
-# takes the same line the profile always uses, with objects for sources.
+# -fuse-ld=lld and the -Wl, flags, which only speak to the linker -- so a
+# runtime unit compiles to the same object either way: native code for debug,
+# -flto bitcode for speed and size, which the link still optimises together
+# with the program. The link takes the same line the profile always uses, with
+# objects for sources.
 compile=()
 for flag in "${common[@]}"; do
-  case "$flag" in -fuse-ld=*) ;; *) compile+=("$flag") ;; esac
+  case "$flag" in -fuse-ld=*|-Wl,*) ;; *) compile+=("$flag") ;; esac
 done
 compile+=(${optimise[@]+"${optimise[@]}"})
+# A program's modules are .ll alone, and clang warns that a -U it cannot
+# apply to one went unused, so their compile leaves the FORTIFY pair out.
+module_compile=()
+for flag in "${compile[@]}"; do
+  case "$flag" in -U_FORTIFY_SOURCE|-D_FORTIFY_SOURCE=*) ;; *) module_compile+=("$flag") ;; esac
+done
 # Waits for every job, not just the last, and fails when any of them did.
 # bash 3.2 has no `wait -n`, so the jobs are waited for in order.
 pids=()
@@ -542,7 +586,7 @@ if [ -n "$objects_from" ]; then
   for i in "${inputs[@]}"; do
     case "$i" in */runtime.c|runtime.c) echo "error: --runtime-from links the runtime in $objects_from, so $i is not an input" >&2; exit 2 ;; esac
     n=$((n + 1))
-    "$CC" "${compile[@]}" -c "$i" -o "$work/$n.o" &
+    "$CC" "${module_compile[@]}" -c "$i" -o "$work/$n.o" &
     pids+=("$!")
     objects+=("$work/$n.o")
   done

package/std/README.md

@@ -24,7 +24,7 @@ whatever program imports it, and subject to the same rules as `examples/` or
 | [`map.ts`](./map.ts) | `reserve(m, n)` and `getOrInsert(m, k, v)` for the global `Map`. Their bodies are the meaning, and what runs under Node: `reserve` does nothing, and `getOrInsert` is a `get`, and a `set` of `v` when the key was missing. Natively the compiler lowers every call in place — `reserve` to the table's `reserveSlots`, which grows the buckets once so that `n` entries fit without a rebuild, and `getOrInsert` to one `probe` and a `valueAt` or an `insertAt` through its answer — so, like `collections.ts`, it writes no `.ll` of its own ([`docs/wp32-map.md`](../docs/wp32-map.md) §9.2, [`docs/LANGUAGE.md`](../docs/LANGUAGE.md#map-and-set)) |
 | [`bytes.ts`](./bytes.ts) | `getU16LE` / `getU16BE` to `getF64LE` / `getF64BE` and the `set` twins: `u16`, `u32`, `u64`, `i32`, `i64` and `f64` read from and written into a `u8[]` at an offset, little- or big-endian. Each access is an ordinary index, so out of range is an `index` panic naming the first missing byte. In range the answers are those of the private copies `std/` keeps (`sha256LoadWord`, `chacha20Word`, `poly1305Le64`, sha512's `loadWord` / `storeWord`, `http2FramePut32`, `tlsPutU16`), none of which it replaces. A guard in the caller does not yet remove the checks after inlining. Under Node only the `u16` and `u32` functions answer as natively. Beside them are the runs a parser walks — `skipBlank`, `skipDigits`, `skipJsonText` and `skipJsonPlain`, the first index in a clamped range whose byte is not a JSON blank, a digit, a byte a JSON string holds as itself or a byte a JSON writer copies without an escape — and `copyBytes`, a window copied as one move that panics on a range outside its array. Their bodies call private walkers, which are what runs under Node; natively the compiler replaces each walker's call with a kernel of `runtime/runtime-bytes.c`, or `llvm.memmove` for the copy ([`docs/LANGUAGE.md`](../docs/LANGUAGE.md#byte-runs-skipblank-skipdigits-skipjsontext-skipjsonplain-and-copybytes-in-nishbytes)). The runs are variable-time. `tests/link/std_bytes*` are its programs |
 | [`base64.ts`](./base64.ts) | `nish/base64`: `base64Encode(bytes)`, `base64EncodeInto(src, off, len, dst, at)` and `base64EncodedLength(n)`, padded standard base64 (RFC 4648 §4) for kitty graphics frames. It only encodes, and it indexes its alphabet by the sextet, so it is not constant-time: it is for pixels, not secrets. A strict decoder, and an encoder for key material, are `crypto/base64url.ts`'s `base64StdPadDecode` and `base64StdPadEncode` (`tests/link/std_base64`; the two modules in one program: `tests/link/std_base64_two_modules`) |
-| [`random.ts`](./random.ts) | `Rng`, a seeded generator of pseudo-random numbers: `new Rng(seed)`, `nextI32()` and `nextF64()`. A Weyl sequence through all 2^32 states with MurmurHash3's finaliser on each, written with `nish:unsafe`'s `wrappingAdd` and `wrappingMul` so that every answer is the same bits natively and under Node (`tests/link/std_random`; [`docs/LANGUAGE.md`](../docs/LANGUAGE.md#random-numbers-rng-in-nishrandom)). A parallel body or a task that makes its own steps it without a shared write. Not for cryptography |
+| [`random.ts`](./random.ts) | `Rng`, a seeded generator of pseudo-random numbers: `new Rng(seed)`, `nextI32()`, `nextF64()` and `nextInt(lo, hi)`, an unbiased integer in [lo, hi] by rejection sampling. A Weyl sequence through all 2^32 states with MurmurHash3's finaliser on each, written with `nish:unsafe`'s `wrappingAdd`, `wrappingSub` and `wrappingMul` so that every answer is the same bits natively and under Node (`tests/link/std_random`; [`docs/LANGUAGE.md`](../docs/LANGUAGE.md#random-numbers-rng-in-nishrandom)). A parallel body or a task that makes its own steps it without a shared write. Not for cryptography |
 | [`cli.ts`](./cli.ts) | a command-line tool's `--help`. `HelpMenu` builds it a line at a time — `header`, `usage`, `section`, `row(flag, placeholder, description)`, `note`, `value` — with descriptions lined up at the caller's columns, counted on the plain text, and answers it with `text()`. `Style` paints it in the eight basic ANSI colours, bold and faint, which the terminal's theme defines, or in bold and underline alone when `TERM` declares no colours. `styleFor(fd, agent)` is the policy `nish --help` follows: plain unless the descriptor is a terminal, `NO_COLOR`, `FORCE_COLOR`, `CLICOLOR_FORCE`, `CLICOLOR` and `TERM=dumb` honoured, plain for an agent (`agentDriven`), forced colour always the theme's; it reads the environment and, for the terminal check, forks `/bin/sh` once, which runs `tput` by absolute path. `HelpMenu` and `Style` reach no capability. The compiler's own help is built with it through a relative import, so the module may use only what the last release compiles (`tests/link/std_cli`; [guide](../docs/guide/using-nish/standard-library.md#nishcli)) |
 | [`sqlite.ts`](./sqlite.ts) | SQLite, vendored in `runtime/sqlite/` and linked into a program that imports the module, through the C adapter `runtime/sqlite/nish-sqlite.c`. `openDatabase(path, flags)` answers a `Database`: `exec`, `begin` / `commit` / `rollback`, `prepare` into a `Statement`, `lastInsertRowid`, `changes` and `close`, which finalizes the connection's statements. A `Statement` binds (`bindI64`, `bindF64`, `bindNull`, `bindText`, `bindTextBytes`, `bindBlob`), steps, resets and reads columns (`columnType`, `columnI64`, `columnF64`, `columnBytes`, `columnText`, and `columnInto`, a copy into the caller's `u8[]`), so a row allocates nothing. Every failure is a `Result` of SQLite's extended code, and `error(code)` builds the `SqliteError` with its message. A handle is an `i32` into the adapter's thread-local tables, and one used after `close` or `finalize` answers `SQLITE_MISUSE`. Reaches `ffi`; links natively and under `--profile wasi` (`tests/link/std_sqlite*`; [`docs/LANGUAGE.md`](../docs/LANGUAGE.md#sqlite-nishsqlite)) |
 | [`base64.ts`](./base64.ts) | standard base64 (RFC 4648 §4) with `=` padding, encode only: `base64Encode(bytes)` answers a string, `base64EncodeInto(src, off, len, dst, at)` writes into a byte buffer allocating nothing, `base64EncodedLength(n)` sizes it; a range outside its array panics before a byte is written. What kitty graphics frames are sent in; not the constant-time `crypto/base64url` (`tests/link/std_base64`, `std_base64_f64`, `std_base64_negative_*`; [guide](../docs/guide/using-nish/standard-library.md#nishbase64)) |
@@ -81,7 +81,7 @@ is listed in [`SECURITY.md`](../SECURITY.md#threat-model).
 | [`crypto/hmac.ts`](./crypto/hmac.ts) | `hmacSha256` and `hmacSha384`, the streaming `HmacSha256` and `HmacSha384` (keyed in the constructor, then `update` and `digest`), and `hmacSha256Verify` / `hmacSha384Verify`, which compare a received tag with `timingSafeEqual`. The one-shot functions panic on a message longer than 2^31 − 1 bytes. `HmacSha256Scratch` / `HmacSha384Scratch` are the same tag over hashers the caller allocates once — `begin(key, off, len)`, `update`, `finishInto(out, at)`, `wipe` — so a loop of them inside `using a = arena()` leaves the arena where it was; `hmacCopySha256` / `hmacCopySha384` copy a hasher's state into one the caller holds. `pbkdf2HmacSha256(password, salt, iterations, dkLen)` is PBKDF2 with HMAC-SHA-256, for SCRAM-SHA-256: the password goes in and the key comes out as a `Secret<u8[]>`, every intermediate is wiped, and fewer than one iteration or a `dkLen` below one answers `null` | RFC 2104, RFC 4231 §4, RFC 8018 §5.2 |
 | [`crypto/hkdf.ts`](./crypto/hkdf.ts) | `hkdfExtractSha256` / `hkdfExtractSha384` (an empty salt is HashLen zeros) and `hkdfExpandSha256` / `hkdfExpandSha384`, which answer `null` for a length below zero or above 255 × HashLen, for an `info` longer than 2^31 − 1 bytes, and for a PRK shorter than HashLen. `hkdfExpandLabelSha256` / `hkdfExpandLabelSha384` are TLS 1.3's HKDF-Expand-Label, which QUIC uses too; the label is given without its `"tls13 "` prefix, and a length outside 0 to 255, a label outside 1 to 249 bytes, a context past 255 bytes or a secret shorter than HashLen panics, since each is the program's choice rather than the peer's. `hkdfExtractInto`, `hkdfExpandInto` (`false` where the others answer `null`) and `hkdfExpandLabelInto` compute the same bytes in an `HkdfScratch` the caller owns, over the hash a length of 32 or 48 names, into the caller's array, and wipe the scratch before they return. A key held as a `Secret<u8[]>` (`nish:secret`) goes through the `*Secret` twins, which read it only inside `exposeWith` and answer a `Secret`: `hkdfSha256Secret` / `hkdfSha384Secret`, each step's, and `hkdfExpandLabelSha256Secret(prk, label, context, length)` / `hkdfExpandLabelSha384Secret`, which panic where HKDF-Expand-Label does. A destination already held as a `Secret` is derived into in place by `hkdfExpandLabelSha256Held(dst, prk, label, context)` / `hkdfExpandLabelSha384Held`, as many bytes as `dst` holds, the plain output copied in inside `exposeWith` and wiped; `dst` may be `prk`. `hkdfExpandLabelHeldInto(hashLength, prk, label, context, out, at, length)` writes a plain window from a held PRK, for output that is plain by design, a record key or IV | RFC 5869 §2, Appendix A; RFC 8448 §3 and RFC 9001 A.1 for the label |
 | [`crypto/ct.ts`](./crypto/ct.ts) | `timingSafeEqual(a, b)`, which reads every byte whatever it holds, and `timingSafeEqualAt(a, aOff, b, bOff, len)` over two windows, which answers `false` for a window outside its array. Two lengths that differ answer `false` at once, because a length is public, and so does an array longer than 2^31 − 1 bytes | — |
-| [`crypto/bcrypt.ts`](./crypto/bcrypt.ts) | `bcryptHash(password, salt, cost)`, the 60-character `$2b$` hash of a `Secret<u8[]>` password under a 16-byte salt and a cost of 4 to 31, and `bcryptVerify(password, hash)`, which reads `$2a$`, `$2b$` and `$2y$` hashes, refuses `$2x$`, and compares in constant time. The first 72 bytes of a password count, up to its first NUL, as under libxcrypt's `crypt_r` (where other implementations differ: [the interoperability table](../docs/security/crypto-bcrypt.md#interoperability)); the Blowfish state is wiped before either returns. `BCRYPT_MAX_PASSWORD`, `BCRYPT_HASH_LENGTH`, `BCRYPT_SALT_SIZE`, `BCRYPT_MIN_COST`, `BCRYPT_MAX_COST`. A `std/threads` task cannot call them yet, since a task may not wipe | OpenBSD `bcrypt(3)`; crypt_blowfish 1.3 for `$2a$` and `$2y$` |
+| [`crypto/bcrypt.ts`](./crypto/bcrypt.ts) | `bcryptHash(password, salt, cost)`, the 60-character `$2b$` hash of a `Secret<u8[]>` password under a 16-byte salt and a cost of 4 to 31, and `bcryptVerify(password, hash)`, which reads `$2a$`, `$2b$` and `$2y$` hashes, refuses `$2x$`, and compares in constant time. The first 72 bytes of a password count, up to its first NUL, as under libxcrypt's `crypt_r` (where other implementations differ: [the interoperability table](../docs/security/crypto-bcrypt.md#interoperability)); the Blowfish state is wiped before either returns. `bcryptHash4(p0, p1, p2, p3, salts, cost)` and `bcryptVerify4(p0, p1, p2, p3, hashes)` do up to four at once, interleaved (four in the time of about 1.2 single hashes on one core), each lane the single path's answer; verify groups lanes by cost, and a malformed lane makes the whole call an `Err` naming it. `BCRYPT_MAX_PASSWORD`, `BCRYPT_HASH_LENGTH`, `BCRYPT_SALT_SIZE`, `BCRYPT_MIN_COST`, `BCRYPT_MAX_COST`. A `std/threads` task may call all four ([BCRYPT-9](../docs/security/crypto-bcrypt.md#findings)) | OpenBSD `bcrypt(3)`; crypt_blowfish 1.3 for `$2a$` and `$2y$` |
 | [`crypto/base64url.ts`](./crypto/base64url.ts) | `base64urlEncode(data)` and `base64urlDecode(text)`, unpadded; `base64StdPadEncode(data)` and `base64StdPadDecode(text)`, the padded standard alphabet that SCRAM and PEM use, constant-time in both directions. These are not `nish/base64`, which encodes only, is not constant-time and is for pixels; the names differ because std modules share the program's one namespace. Decoding is strict, so every byte string has one spelling. `base64urlDecode` answers `null` for any `=`, a character outside the alphabet, a length of 1 mod 4 or nonzero unused low bits; `base64StdPadDecode` requires the padding, answering `null` unless the text is whole four-character groups with `=` only as the last one or two characters, every other character in the alphabet and the unused bits zero. Both refuse a text longer than 2^31 − 1 bytes, and `base64urlEncode` panics on such an array | RFC 4648 §4, §5, §10 (`tests/link/std_base64_padded`) |
 | [`crypto/sha1.ts`](./crypto/sha1.ts) | `sha1(data)`, a fresh 20-byte digest, and `SHA1_SIZE` and `SHA1_BLOCK`. **Not for security**: SHA-1 is broken for collisions, and it is here only because RFC 6455 §4.2.2's `Sec-WebSocket-Accept` is defined on it (`nish/net/websocket` is its importer). One-shot only; it panics on an array longer than 2^31 − 1 bytes. It came after the #363 audit and has no record in `docs/security/` | FIPS 180-4 §6.1 |
 | [`crypto/x25519.ts`](./crypto/x25519.ts) | `x25519(scalar, u)`, the shared secret as a `Secret<u8[]>`, and `x25519Base(scalar)`, the public key, each from a `Secret<u8[]>` scalar, on ten 25.5-bit limbs in `i64`. Either answers `null` unless its arguments are `X25519_SIZE` (32) bytes; the scalar is clamped on a copy, which is wiped | RFC 7748 §5.2, §6.1, Wycheproof `x25519_test` |

package/std/cli.ts

@@ -273,9 +273,25 @@ export class HelpMenu {
 
   /** The opening line, the program's name, `version` and a one-line `summary`, and a blank after it. */
   header(version: string, summary: string): void {
+    this.title(version, summary)
+    this.lines.push("")
+  }
+
+  /**
+   * The opening line alone, without `header`'s blank after it, for a menu that
+   * puts something under its title (`line`) before the blank.
+   */
+  title(version: string, summary: string): void {
     const style = this.style
     this.lines.push(`${style.name(this.program)} ${style.dim(version)}  ${style.dim(summary)}`)
-    this.lines.push("")
+  }
+
+  /**
+   * A line exactly as given, painted (or not) by the caller with the menu's
+   * `style`: a rule, a banner, a blank. Its width is the caller's to keep.
+   */
+  line(text: string): void {
+    this.lines.push(text)
   }
 
   /**