buninu 0.3.0 → 0.4.0

package/apps/jsgotty/single-exe/README.md

@@ -1,80 +1,384 @@
-# This is completely optional
-- I still recommend using the methods in the root README.md
-- Bun's Android build is currently not supported
+# Single-file executable bootstrap
 
-# Usage
-- First run `bun ./packAssets.sh` to bundle the assets into `assets.tar`
-- Then run the build script like this
+Bundles a project's runtime assets into one `bun build --compile` binary, so
+the executable carries its own `static/`, templates, documentation and so on.
 
-  ```shell
-  bun build --compile --bytecode --minify ./entry.mjs --outfile=binname
-  ```
+Assets are declared once, in `package.json`:
 
-- You'll get a binname executable
+```json
+{
+  "assets": ["static", "README.md", "README.en.md"]
+}
+```
+
+Paths are relative to the project root. Folders are packed recursively; only
+regular files are stored, so symlinks and empty folders leave no trace.
+
+## The two back ends
+
+The default packs everything into a gzipped `assets.tar` that is embedded as a
+single file and unpacked into memory at startup. Setting `ASSETS_BUNFS=1` at
+build time switches to `bun build --compile --asset`, which stores the files
+individually inside the binary's virtual filesystem instead.
+
+Reading code never sees the difference — `assetsHelper` answers the same keys
+either way. The trade is binary size against startup cost:
+
+| | tar (default) | `ASSETS_BUNFS=1` |
+| --- | --- | --- |
+| Stored as | one gzipped archive | individual files, uncompressed |
+| Size, for jsgotty's 3.45 MB of assets | **0.70 MB** | 3.45 MB |
+| Work before `main` runs | decode the archive, copy every file into memory | **none** |
+| Startup cost, same assets | ~16 ms | **0 ms** |
+| Resident memory | whole payload, always | only what is read |
+| Maturity | in use | experimental |
+
+Text compresses well, so the tar usually wins on size by a wide margin — for
+jsgotty most of the payload is `static/`, whose JavaScript and CSS shrink to a
+fifth. What it costs is paid on every run, whether or not the program touches a
+single asset. Reach for `ASSETS_BUNFS=1` when that startup work, or holding the
+whole payload in memory, matters more than the extra megabytes on disk.
+
+## Adopting it in another project
+
+Copy `single-exe/` to the project root. Nothing inside it hard-codes its own
+name or depth, so it can be renamed or moved — but the importing code has to
+follow, so staying at the root is the easy path. Three edits point it at the
+project around it.
+
+1. **`assetsPacker.js`, the first two lines.** They are the only place the
+   asset root is written down, and a static import specifier cannot be
+   computed, so both must be literal:
+
+   ```js
+   import pkg from "../package.json" with { type: "json" };
+   export const ASSETS_ROOT = "..";
+   ```
+
+2. **`entry.mjs`, the import of the program.** It names what to start, and
+   entry.mjs is the compiled entry point in both back ends:
+
+   ```js
+   await import("../src/index.js");
+   ```
+
+   **Keep the `await`.** It reads like a formality — nothing here needs the
+   module object — but together with the `await globalThis.assetsLoaderPromise`
+   line above it, it is the only top-level `await` the compiled program writes
+   anywhere, and `--bytecode` reads exactly that to decide the program starts
+   asynchronously. Turn both into `.then(…)` and the build produces an
+   executable that exits 0 without running a line.
+   [Why](#the-await-in-entrymjs-is-load-bearing) explains the trap.
+
+3. **`package.json`.** Add the `assets` array shown above.
+
+Everything else — `compiled.js`, `assetsHelper.js`, `assetsLoader.mjs` — is
+project independent, with one exception: `buildHtmlBundleImageMap` defaults its
+attribute name to jsmdcui's `data-mdcui-src`, so pass your own.
+
+Node's own entry path must not import `assetsLoader.mjs` or `entry.mjs`; those
+are Bun only. `assetsHelper.js` and `compiled.js` run under Node 20.11+.
+
+## Reading assets
+
+Import from `assetsHelper.js`. Keys are the same package-relative paths that
+appear in `package.json`.
+
+| Function | Returns |
+| --- | --- |
+| `readInternalAssetText(path)` | string, or `null` when not embedded |
+| `readInternalAssetBytes(path)` | `Uint8Array`, or `null` |
+| `readAssetText(path)` | string, falling back to the file on disk |
+| `readAssetBytes(path)` | `Uint8Array`, same fallback |
+| `hasInternalAssets()` | whether this package has anything embedded |
+| `listInternalAssetPaths(prefix?)` | every embedded path under `prefix` |
+| `listInternalAssetDirs(prefix?)` | the immediate child names under `prefix` |
+| `assetPath(...parts)` | joins parts into a key |
+| `getAssetKey(path)` | the key as stored, namespace included |
+| `assetDiskPath(path)` | where the disk fallback reads that key from |
+| `SELF` | this package's namespace, `assets/<name>@<version>` |
+
+The read functions are synchronous for embedded assets on both back ends.
+`readAssetText` / `readAssetBytes` are async only because of the disk
+fallback, which is what makes the same code work from a source checkout.
+
+**Only the read functions fall back to disk.** The listing functions report
+what is embedded and nothing else, so from a source checkout they return an
+empty array. Branch on `hasInternalAssets()` and read the directory yourself:
+
+```js
+import { readdirSync } from "node:fs";
+import { join } from "node:path";
+import { readAssetText, hasInternalAssets, listInternalAssetDirs } from "../single-exe/assetsHelper.js";
+import { REPO_ROOT } from "../single-exe/compiled.js";
+
+//  Embedded or on disk, this one already works either way.
+const index = await readAssetText("static/index.html");
+
+//  Listing does not, so pick the source.
+const staticDirs = hasInternalAssets()
+  ? listInternalAssetDirs("static")
+  : readdirSync(join(REPO_ROOT, "static"));
+```
+
+## Building
+
+`compiled.js` exports the build entry points. Wire `buildEarlyExit` into the
+CLI and it handles `--build-exe` and `--build-for <target>`:
+
+```js
+import { buildEarlyExit, IS_COMPILED, REPO_ROOT } from "./single-exe/compiled.js";
+
+await buildEarlyExit(process.argv, "myapp");
+```
+
+`REPO_ROOT` is the project root in a checkout and the executable's own folder
+once compiled, which is where `--assets-extract` writes. `IS_COMPILED` tells
+the two apart.
+
+### Chaining file/folder assets across dependencies
+
+A build first bundles once to write a metafile, then treats every
+`assetsPacker.js` appearing in its `inputs` as a participant. Each package
+vendors its own `single-exe/`, and its `assetsHelper` imports its
+`assetsPacker`, so a packer lands in that graph exactly when the package is
+reachable from the entry point. There is no registration step, and anything
+tree-shaken away brings nothing with it.
+
+The packers then run one after another, each adding its files under its own
+`assets/<name>@<version>/` namespace — appending into the shared `assets.tar`
+by default, or copying into the shared `build/assets` tree under
+`ASSETS_BUNFS=1`. Only once every packer has finished does the real compile
+run, so the archive is complete and, by default, compressed by then.
+
+That namespace is why `ASSETS_BUNFS=1` stages every package into one fixed
+`build/assets`. `--asset` keeps only the **basename** of the path it is given
+and preserves everything below it, so the folder's own name becomes the root
+inside the binary:
+
+```
+--asset ./build/assets   with   build/assets/jsgotty@1.1.6/static/index.html
+                          ->    assets/jsgotty@1.1.6/static/index.html
+```
+
+Pass `./build/assets/jsgotty@1.1.6` instead and `build/assets/` is gone: the
+files land at `jsgotty@1.1.6/...` and every package gets its own root. One
+folder named `assets`, everyone copying into it, is what makes the keys come
+out the same as the ones the tar back end writes.
+
+`assetsPacker.js` also runs standalone; `--help` documents its flags.
+
+## Why the loader hands over a Promise
+
+`entry.mjs` imports `assetsLoader.mjs`, awaits `globalThis.assetsLoaderPromise`,
+and only then imports the main program. The indirection is deliberate: the
+loader carries a Bun-only import,
 
-# Single Executable Intro
+```js
+import assets from "./assets.tar" with { type: "file" };
+```
+
+and if the main program imported the loader, that dependency would land in its
+module graph. Node could then no longer load the program at all — not even to
+fall back to reading the assets from disk.
+
+Keeping the loader behind `entry.mjs` means the main program never mentions it,
+so Node 20.11+ runs the same file directly against on-disk assets while Bun
+runs it through the bootstrap. The Promise is that boundary, not a workaround
+for missing top-level await.
+
+`ASSETS_BUNFS=1` sidesteps the question: there is no tar to load, so the build
+writes `.bunfs-entry.mjs` next to `entry.mjs` — the same file with the loader
+import removed — compiles that, and deletes it afterwards. A copy rather than a
+stub, because entry.mjs may do more than import the program: a CommonJS main
+that starts itself from `require.main === module` needs the exported starter
+called instead, and that call has to survive. jsgotty is exactly that case; see
+[below](#the-entry-import-calls-an-exported-starter).
+
+## The `await` in entry.mjs is load-bearing
 
-This folder contains the Bun single-exe bootstrap used by the project.
+`--bytecode` decides whether a program starts asynchronously by looking for a
+top-level `await` **written in some file's own source**. The bundler moves a
+module into a lazy init function whenever something dynamically imports it, and
+a module moved that way takes its top-level `await` with it — what stays behind
+is an `await init_x()` the bundler wrote itself, which does not count. If no
+file wrote one, the executable is evaluated on the synchronous path, its
+suspended startup is dropped, and it exits 0 having run nothing. No error, at
+any `--minify` setting, however small the program.
 
-## Entry Flow
+`entry.mjs` is immune because it writes its own, and one is enough. Every other
+`await` in the graph — all of `assetsLoader.mjs`'s, for one — sits inside a
+function and cannot help. It is also why bunfs builds compile a copy of that
+file rather than handing the main program to `bun build` directly: a main
+program that reaches top-level await only through its imports writes none of
+its own.
 
-- `entry.mjs` imports `assetsLoader.mjs` first
-- `assetsLoader.mjs` loads `assets.tar` with `Bun.Archive` and mounts it as `globalThis.internalAssets`
-- `assetsLoaderPromise` is exposed on `globalThis`
-- The main program waits for `assetsLoaderPromise` if it exists
+Upstream: `src/bundler/linker_context/postProcessJSChunk.rs` skips exactly the
+modules the bundler moved, still true on oven-sh/bun `main` at 8eb5b6e2b5
+(2026-08-22). Until that changes, treat entry.mjs's `await`s as API.
 
-That keeps the main program bootable even if asset loading reports errors.
+## Environment variables
 
-## Assets Loading
+| Variable | When | Effect |
+| --- | --- | --- |
+| `ASSETS_BUNFS=1` | build | use `--asset` instead of the tar |
+| `ASSETS_NO_GZIP=1` | build | leave `assets.tar` uncompressed |
+| `ASSETS_DEBUG=1` | run | print how long unpacking took |
 
-- Bundled assets are loaded sequentially with `await file.bytes()`
-- Load failures are collected and printed to `stderr`
-- Asset loading never rejects the bootstrap promise
-- When loading succeeds, the archive is available through `globalThis.internalAssets`
+## Runtime flags
 
-## CLI Flags
+| Flag | Effect |
+| --- | --- |
+| `--assets-list` | print every embedded path and exit |
+| `--assets-extract` | write the assets next to the executable and exit |
+| `--assets-external` | ignore embedded assets and read from disk |
 
-- `--assets-list`
-  - Lists all entries inside bundled `assets.tar`
-  - Exits early before the main program starts
+These are handled by `assetsLoader.mjs`, so they exist in tar builds only.
 
-- `--assets-extract`
-  - Extracts bundled assets to the same directory as the executable
-  - Exits early before the main program starts
+## Advanced usage
 
-- `--assets-external`
-  - Skips loading bundled assets into `globalThis.internalAssets`
-  - Forces the main program and runtime helpers to use the external file tree
-  - Keeps the bootstrap alive while leaving `internalAssets` falsy
+### Forwarding arguments to `bun build`
 
-## Adapting this folder
+Anything after `--build-exe`, or after the target given to `--build-for`, is
+passed straight through to `bun build`:
 
-This single-exe folder is intended for reuse by other projects. Usually you
-only need to edit these project-specific files first:
+```shell
+bun ./gotty.js --build-exe --sourcemap
+bun ./gotty.js --build-for bun-linux-x64 --sourcemap
+```
+
+Order matters. The flags have to come after the build switch, or Bun consumes
+them before the program ever starts:
+
+```shell
+# Wrong: Bun parses this define before any of your code runs
+bun --define MY_APP=../app.md ./gotty.js --build-exe
 
-- `entry.mjs`
-- `packAssets.sh`
+# Right
+bun ./gotty.js --build-exe --define MY_APP=../app.md
+```
+
+### `--define` with a string value
 
-Then the main program can import `buildEarlyExit` from `compiled.js` and call
-it before normal CLI parsing. `--build-exe` and `--build-for <target>` will run
-`packAssets.sh` first, so `assets.tar` is generated as part of the build flow.
+**Bun expects `--define` values to be JSON literals, so a string has to arrive
+already quoted.** A bare path is read as an identifier and the build fails or
+inlines something unintended. Quoting it through a shell is awkward, so
+`compiled.js` exports a helper that does it:
 
 ```js
-const compiledHelper = await import("./single-exe/compiled.js").catch(() => null);
-await compiledHelper?.buildEarlyExit?.(process.argv, "my-bin");
+import { buildEarlyExit, stringifyNonPrimitiveDefineValues } from "./single-exe/compiled.js";
+
+stringifyNonPrimitiveDefineValues(process.argv, "MY_APP");
+await buildEarlyExit(process.argv, "my-bin");
 ```
 
-- `--build-exe` builds with no Bun target.
-- `--build-for <target>` passes `<target>` to `bun build --target=<target>`.
-- The second argument is the output filename. If omitted, it defaults to `single.exe`.
+Call it before `buildEarlyExit`, once per define name you want treated as a
+string. It rewrites the value in place:
 
-The main program should also switch its repo/resource root when running as a
-compiled binary. Use `IS_COMPILED` and `REPO_ROOT` from
-`compiled.js`; when compiled, treat the executable directory as the repo root
-used for external fallback files such as `static/`, `README.md`, or other files
-packed by `packAssets.sh`.
+```js
+["--define", "MY_APP=../app.md"]      // what the user typed
+["--define", 'MY_APP="../app.md"']    // what Bun receives
+```
+
+`--define=MY_APP=../app.md` works the same way. Numbers, booleans, `null` and
+`undefined` are left alone, so only genuine strings get quoted.
+
+### Images from an imported HTML bundle
+
+A compiled binary exposes an imported HTML entry as a `homepage` object:
+`homepage.index` is the compiled HTML and `homepage.files` lists the
+content-hashed assets by their path inside the binary. `assetsHelper` can map
+the *original* image references back to those paths, without copying any image
+bytes into memory.
+
+Bun rewrites `src` during the build, so keep the original reference in a custom
+attribute of your own choosing:
+
+```html
+<!-- source -->
+<img src="./images/photo.jpg" data-original-image="./images/photo.jpg">
+
+<!-- compiled -->
+<img src="/photo-abcd1234.jpg" data-original-image="./images/photo.jpg">
+```
+
+Then build the map once, in a compiled binary only:
 
 ```js
-const compiledHelper = await import("./single-exe/compiled.js").catch(() => null);
-const repoRoot = compiledHelper?.REPO_ROOT
+import { buildHtmlBundleImageMap, canonicalHtmlBundleImageHref } from "../single-exe/assetsHelper.js";
+import { IS_COMPILED } from "../single-exe/compiled.js";
+
+const images = IS_COMPILED
+  ? await buildHtmlBundleImageMap(homepage, "data-original-image")
+  : null;
+
+const path = images?.get(canonicalHtmlBundleImageHref(originalHref));
+const bytes = path ? await Bun.file(path).bytes() : null;
+```
+
+`buildHtmlBundleImageMap` returns a `Map` from canonical original reference to
+the path inside the binary; it decodes HTML entities and percent encoding so a
+reference written either way still matches. Reading stays lazy — the bytes are
+only touched when you use the returned path.
+
+Lower level, if you already hold a rewritten public path such as
+`/photo-abcd1234.jpg`: `findHtmlBundleAsset(homepage, publicPath, options)`
+finds its `homepage.files` entry, `findHtmlBundleImageAsset()` restricts that
+to `image/*`, and `htmlBundleImageAssetPath()` returns just the path.
+
+Never hard-code `/$bunfs/root/` or `B:/~BUN/`; always use the path that
+`homepage.files` reports.
+
+## jsgotty specifics
+
+The rest of this file is generic. These names belong to jsgotty itself and are
+not reserved by the bootstrap — an adopting project picks its own.
+
+Building is optional; running `bun gotty.js` from a checkout, as the root
+`README.md` describes, remains the usual way to run jsgotty.
+
+```shell
+bun ./gotty.js --build-exe
+bun ./gotty.js --build-for bun-linux-x64
+```
+
+Both produce `jsgt`: `gotty.js` calls `buildEarlyExit(process.argv, "jsgt")`
+from `bootstrap()`, before any CLI parsing, so the build switches exit before
+a server is ever started. Neither works from an already compiled binary.
+
+The three packed entries — `static`, `README.md`, `README.en.md` — are what the
+HTTP server hands out: `index.html`, `manifest.json`, `help.md`, the frontend
+JavaScript and CSS, and the two READMEs behind the help pages.
+
+### The entry import calls an exported starter
+
+`gotty.js` is CommonJS and starts itself from `require.main === module`. With
+`entry.mjs` as the bundle's entry point that guard is inlined to `false`, so
+the import cannot rely on it and calls the export instead:
+
+```js
+const { bootstrap } = await import("../gotty.js");
+
+await bootstrap();
+```
+
+Both back ends go through it: `ASSETS_BUNFS=1` compiles a copy of `entry.mjs`,
+so `gotty.js` is never the entry point and the guard stays `false` in either
+mode. Handing `gotty.js` to `bun build` directly instead would compile it as
+the entry, Bun would rewrite the guard to `import.meta.main` — true — and it
+would start itself; that path is not taken, and `bootstrap` is what starts the
+program in both builds.
+
+### `packAssets.sh`
+
+The original implementation, kept as a reference: a hand-written tar of the
+three asset entries, from before `assetsPacker.js` and its `package.json`
+declaration existed. `buildExecutable()` in `compiled.js` is the matching
+build path and still runs it, but `buildExe` now points at `buildExeAssets`,
+so nothing in the current flow calls either. It is the shortest way to see
+what the packer produces, and it still packs an `assets.tar` by hand:
+
+```shell
+bun ./single-exe/packAssets.sh
+bun build --compile --bytecode --minify ./single-exe/entry.mjs --outfile=jsgt
 ```