@ai-ecoverse/slicc-kernel 1.172.0 → 1.172.2

package/README.md

@@ -25,7 +25,7 @@ TypeScript declarations ship in `dist/types/`, and each entry (`.`, `./node`, `.
 
 | option | default | |
 | --- | --- | --- |
-| `root` | `navigator.storage.getDirectory()` | the `FileSystemDirectoryHandle` that becomes `/` |
+| `root` | `navigator.storage.getDirectory()` | the `FileSystemDirectoryHandle` that becomes `/`. WebKit cannot clone a handle, so there it must lie in OPFS, and the kernel worker reopens it by path |
 | `modules` | `'/node_modules'` | where installed packages are scanned for commands |
 | `commands` | none | `{ append: ['/opt/user/node_modules'] }`: more `node_modules` directories scanned for commands after all the others, so they never shadow one (see [Commands](#commands)) |
 | `env` | `{}` | environment added to every run |
@@ -635,6 +635,7 @@ WASIX programs get `AF_INET`/`AF_INET6` `SOCK_DGRAM` sockets, and `AF_UNIX` ones
 - **The proxy** listens on `127.0.0.1:3128` from the first connection on. It takes absolute-form `http:` requests and `CONNECT` tunnels, which it terminates with a certificate for the requested host, issued by the kernel's own CA, then hands each request to the transport. Programs start with `http_proxy`, `https_proxy` (and the upper-case names) pointing at it, `no_proxy` covering loopback and the [node name](#node-name), and `SSL_CERT_FILE`, `CURL_CA_BUNDLE` and `GIT_SSL_CAINFO` pointing at the CA bundle in `/etc/ssl/certs/slicc-kernel-ca.pem`; variables in `env` override them. The CA's key never leaves WebCrypto; it is kept in IndexedDB, in `<metadata>-ca`, or in `slicc-kernel-ca` when `metadata` is the default or `false`, so kernels on one origin share one CA whether or not they keep metadata. `ca` names another database, and `ca: false` keeps a CA in memory for the kernel's lifetime. Each kernel adds its CA to the bundle instead of replacing it, so every kernel on the origin stays trusted; the bundle keeps the newest 16.
 - **TLS** needs [`@ai-ecoverse/wasm-tls-engine`](https://www.npmjs.com/package/@ai-ecoverse/wasm-tls-engine) installed under `modules`, like a command. Without it, `CONNECT` to port 443 is answered `501` with the reason. A tunnel to another port is accepted, and what the program sends first decides: plain HTTP is served as `http://host:port`, without TLS, and a TLS handshake is terminated as above, or the tunnel closes when it cannot be.
 - **The transport** is an object on the page with `traits` (`manualRedirects`, `encodedBodies`, `maxRequestBody`, and `crossOrigin`: `'cors'` or `'any'`) and `fetch(request)`, which answers `{ status, statusText, headers, body, cancel }` with `body` an async iterable of `Uint8Array`, and optionally `url` and `redirected: true` when it followed redirects itself (`fetchTransport()` does), so a JS program's `fetch` reports where it ended. The kernel pulls the body one chunk at a time, so a slow program slows the download. A rejection with a numeric `status` is answered with that status; any other with `502`. A rejection with `detail: 'transport'` says the transport could not make the request at all. `fetchTransport()` sets it when its `fetch` rejects without a `cause.code`, as a browser's does, and a JS program's `fetch` gets it as `cause.detail` (above). Any other `detail` is dropped.
+- **`User-Agent` in a browser:** `fetchTransport()` leaves the header out when it uses the realm's own `fetch` in a page or a worker, or anywhere else `location` exists. Chromium drops it anyway, while WebKit sends it and preflights the request, which a registry refuses. A `fetch` you pass in gets it, and so does Node.
 - **Streamed request bodies** ([#238](https://github.com/ai-ecoverse/slicc-kernel/issues/238)): a request comes with its body whole, as `body` (a `Uint8Array`), unless the transport sets `traits.streamingBodies`. Then a body the kernel doesn't hold whole comes as `bodyStream`, an async iterable of `Uint8Array`, never together with `body`. That's an upload through the proxy (`Content-Length` or chunked, curl's `-T`, git's push) and a JS program's `fetch` with a stream body. The kernel reads from the program only as fast as the transport pulls, applies no `maxRequestBody` and reserves no memory for it. A transport with a limit of its own rejects with `status: 413`. When the transport answers before it has read the whole body, the proxy closes the program's connection after the response, since the rest of the body is still on it. A transport without the trait keeps getting whole bodies, capped at its `maxRequestBody`, so existing transports need no change. A transport that wraps another (a router, an injector, a logger) and passes on the inner `traits` must forward `bodyStream`, or not claim `streamingBodies`: a wrapper that rebuilds the request from `url`, `method`, `headers`, `body` and `signal` would drop a streamed upload and still answer. `nodeTransport()` streams (undici's `fetch` with `duplex: 'half'`). `fetchTransport({ streamingBodies: true })` turns it on for a page that knows its targets speak HTTP/2 or later, since Chrome streams a `fetch` upload only there, so it's off by default. The trait is also how an embedder's own transport, such as the `slicc` CLI link's exit, takes streamed uploads; across the page and the kernel's worker, and to attached clients, the body is pulled one chunk at a time. `fetchTransport()` is the plain `fetch` of the page, bound by CORS; its `hint` option is appended to the message of a request it could not make, so a program's 502 can name a way around CORS (`registry.npmjs.org` and jsDelivr allow it); an embedder with a way around CORS passes its own. A response body that delivers no data for `bodyIdleMs` (default 5 minutes; `0` or `Infinity` turns it off) fails with `ETIMEDOUT` instead of waiting forever, which matters because Chrome under heavy load can stop handing a page the rest of a body. A slow body that keeps delivering is never cut off.
 - **Traits of the embedder's own:** `traits` may carry more fields than the kernel reads. The kernel keeps them, and attached clients get them as they are (`client.transport.traits`, `onTraits`). In TypeScript, `NetworkTraits` types its own fields and leaves any other as `unknown`. One is typed: `exit`, the default exit an embedder routes through, `{ kind, caps }` (`NetworkExit`). `kind` is `'local'`, `'remote'` or `'tailnet'` (or another string), `caps` lists what the exit carries (`['net', 'http', 'ws']`), and further fields are `unknown`. When traits change later, call `kernel.updateTransportTraits()`.
 - **Redirect hops** ([#302](https://github.com/ai-ecoverse/slicc-kernel/issues/302)): a transport that follows redirects but can see them (a browser extension's `webRequest.onBeforeRedirect` can) sets `traits.redirectHops` and answers `redirects: [{ status, url, headers }]`, one entry per 3xx it followed, in order (`NetworkRedirectHop`). The kernel then treats it as one that does not follow. Programs get each 3xx as it was answered: through the proxy, curl, requests and `hf` see the real redirect, `X-Repo-Commit` and `ETag` included, and `ctx.fetch` with `redirect: 'manual'` answers it. The rest of the chain is held for the process that asked, at most 30 s and 8 chains per process (64 in all), with the final body unread. A follow-up from that process, with the method the redirect leads to and the same `Range`, `If-None-Match`, `If-Modified-Since`, `If-Range` and `Accept-Encoding` (and an `Authorization` if the first had one), is answered from it without a second request; any other request goes to the transport. A chain that expires or is pushed out is cancelled, and so is one whose follow-up arrives and does not match. After a redirect to another origin the follow-up is expected without `Authorization`, which HTTP clients drop when they follow one; a `Location` matches without its `#fragment`. The page's own `fetch` cannot show a redirect, so `fetchTransport()` in a page stays a transport that follows, and `redirect: 'manual'` fails there as before. `fetchTransport({ redirect: 'manual' })` hands redirects back unfollowed where the platform's `fetch` shows them (Node; a browser answers an opaque status 0), and `nodeTransport()` does so. A kernel socket also records who connected it (`SO_PEERCRED`): the connecting process's pid, uid and gid on the accepting side, and the listener's on the connecting side, kept when the socket passes to another process. The proxy uses this to keep each process's held chains its own.
@@ -915,6 +916,7 @@ An `fsa` mount is a drive for a local folder, from the File System Access API, a
 - **Insert request:** at the same time the kernel calls the page's `onMountPending({ target, source, insert })`, so the page can show "Insert a folder for /mnt/x" with a button.
 - **Inserting:** calling `insert()` from that button's click picks the folder with `requestDirectory()`, or asks for permission again on a folder the drive had before. The medium goes in without a remount, and the state becomes `ok`. `kernel.insert(target, handle)` inserts a handle the page already has.
 - **Persistence:** the drive's source is `fsa:<id>`, and the kernel keeps the folder's handle under that id in IndexedDB (`<metadata>:media`). Mounting `fsa:<id>` again, after a reboot too, inserts the folder at once while its permission holds, and asks again when it is back to `prompt`. In an Incognito window Chrome crashes a page that reads a folder handle back from IndexedDB ([#95](https://github.com/ai-ecoverse/slicc-kernel/issues/95)). Pass `media: false` there: the kernel then never opens the handle database, and keeps the handles in memory, so a remount finds its folder only within the same kernel, while POSIX metadata still persists. `metadata: false` keeps the handles in memory too.
+- **WebKit:** it has no folder picker and cannot clone a handle or keep one in IndexedDB, so a folder must lie in OPFS. The page sends its path, the kernel worker reopens it, and the media store keeps the path.
 - **Ejecting:** `umount` ejects. So does the permission going away: the next operation fails with `ENOMEDIUM` and the kernel asks again. The folder is untouched either way.
 - **Limits:** the folder has no symlinks or modes, and `df` reports no size for it either.
 

package/dist/index.js

@@ -546,6 +546,32 @@ function integrityUnknown(options) {
   return unknown;
 }
 
+// src/fs/portable.ts
+var privateTop = () => navigator.storage.getDirectory();
+function clonable(handle) {
+  const { port1, port2 } = new MessageChannel();
+  try {
+    port1.postMessage(handle);
+    return true;
+  } catch (err) {
+    if (err?.name === "DataCloneError") return false;
+    throw err;
+  } finally {
+    port1.close();
+    port2.close();
+  }
+}
+async function portable(handle, top = privateTop) {
+  if (clonable(handle)) return { handle };
+  const path = await (await top()).resolve(handle);
+  if (path === null) {
+    throw new Error(
+      "this browser cannot hand a folder to a worker unless it is in the origin private file system"
+    );
+  }
+  return { path };
+}
+
 // src/integrity-relay.ts
 function integrityAnswerer(port, lookup) {
   const ask = async ({ id, path }) => {
@@ -1314,6 +1340,11 @@ async function* chunks(body, idleMs) {
     await reader.cancel().catch(() => void 0);
   }
 }
+function sentHeaders(headers, inPage) {
+  const out = joinHeaders(headers);
+  if (inPage) out.delete("user-agent");
+  return out;
+}
 function streamOf(body) {
   const it = body[Symbol.asyncIterator]();
   return new ReadableStream(
@@ -1341,6 +1372,7 @@ function fetchTransport(options = {}) {
   const Socket = options.webSocket === true ? globalThis.WebSocket : options.webSocket || void 0;
   const headers = options.webSocketHeaders === true;
   const instance = options.redirect ?? "follow";
+  const inPage = !options.fetch && "location" in globalThis;
   return {
     traits: {
       manualRedirects: instance === "manual",
@@ -1357,7 +1389,7 @@ function fetchTransport(options = {}) {
       try {
         response = await send(request.url, {
           method: request.method,
-          headers: joinHeaders(request.headers),
+          headers: sentHeaders(request.headers, inPage),
           ...requestBody(request),
           signal: request.signal,
           redirect: request.redirect ?? instance,
@@ -2442,11 +2474,17 @@ function uplinkInit(uplink) {
   const nameserver = typeof uplink.nameserver === "string" ? { nameserver: uplink.nameserver } : {};
   return { uplink: { traits, ...uplink.routes ? { routes: uplink.routes } : {}, ...nameserver } };
 }
-function initCall(options, transport) {
+function rootField(at) {
+  return "path" in at ? { rootPath: at.path } : { root: at.handle };
+}
+async function insertCall(target, handle, source) {
+  return { op: "insert", target, source, ...await portable(handle) };
+}
+async function initCall(options, transport) {
   const hooks = sealedHooks(options.sealed);
   return {
     op: "init",
-    ...options.root ? { root: options.root } : {},
+    ...options.root ? rootField(await portable(options.root)) : {},
     ...options.modules ? { modules: options.modules } : {},
     ...options.commands ? { commands: options.commands } : {},
     ...options.env ? { env: options.env } : {},
@@ -2742,7 +2780,7 @@ async function createKernel(options = {}) {
   const pendingMedium = ({ target, source, handle }) => options.onMountPending?.({
     target,
     source,
-    insert: async () => void await call({ op: "insert", target, source, handle: await chosen(handle) })
+    insert: async () => void await call(await insertCall(target, await chosen(handle), source))
   });
   const aside = (data) => {
     if (files.event(data)) return true;
@@ -2815,7 +2853,7 @@ async function createKernel(options = {}) {
     };
   }
   const { dial } = dials;
-  await call(initCall(options, transport));
+  await call(await initCall(options, transport));
   return {
     async run(argv, runOptions = {}) {
       const { onStdout, onStderr, stdin, ...rest } = runOptions;
@@ -2841,7 +2879,7 @@ async function createKernel(options = {}) {
     mount: async (spec) => await call({ op: "mount", spec }),
     umount: async (target) => void await call({ op: "umount", target }),
     mounts: async () => await call({ op: "mounts" }),
-    insert: async (target, handle) => void await call({ op: "insert", target, handle }),
+    insert: async (target, handle) => void await call(await insertCall(target, handle)),
     ...pageOnly(call),
     watch: files.watch,
     terminate() {