@ai-ecoverse/slicc-kernel 1.53.4 → 1.54.1

package/README.md

@@ -34,7 +34,7 @@ const { status, stdout, stderr } = await kernel.run(['bash', '-c', 'echo hi > he
 | `worker` | `new URL('./kernel-worker.js', import.meta.url)` | the kernel worker script |
 | `network` | none | `{ transport, uplink }`: how programs reach the outside world (see [Network](#network) and [Uplink](#uplink)) |
 | `requestDirectory` | none | `() => Promise<FileSystemDirectoryHandle>`, typically `showDirectoryPicker`: how an `fsa` mount gets its folder (see [Removable media](#removable-media-fsa)) |
-| `hostfs` | none | `(source, { readonly }) => Promise<{ url, token, capabilities? }>`: a grant for a `hostfs` mount from the local proxy (see [Host folders](#host-folders-hostfs)) |
+| `hostfs` | none | `(source, { readonly }) => Promise<{ url, token, capabilities? } \| { port, token?, capabilities? }>`: a grant for a `hostfs` mount from the local proxy, or a `MessagePort` the page serves the folder on (see [Host folders](#host-folders-hostfs)) |
 | `onMountPending` | none | `({ target, source, insert }) => void`: an `fsa` mount needs a folder; call `insert()` from a user gesture |
 | `processMounts` | `true` | whether programs may call `mount(2)` and `umount2(2)`: `false`, or `(req) => boolean \| Promise<boolean>` to decide each call (see [Mounting from a program](#mounting-from-a-program)) |
 | `hostname` | `'slicc'` | the kernel's node name (see [Node name](#node-name)) |
@@ -253,7 +253,7 @@ await kernel.run(['curl', '-sS', 'https://registry.npmjs.org/@ai-ecoverse/wasm-b
 - **The proxy** listens on `127.0.0.1:3128` from the first connection on. It takes absolute-form `http:` requests and `CONNECT` tunnels, which it terminates with a certificate for the requested host, issued by the kernel's own CA, then hands each request to the transport. Programs start with `http_proxy`, `https_proxy` (and the upper-case names) pointing at it, `no_proxy` covering loopback and the [node name](#node-name), and `SSL_CERT_FILE`, `CURL_CA_BUNDLE` and `GIT_SSL_CAINFO` pointing at the CA bundle in `/etc/ssl/certs/slicc-kernel-ca.pem`; variables in `env` override them. The CA's key never leaves WebCrypto; it is kept in IndexedDB, in `<metadata>-ca`, or in `slicc-kernel-ca` when `metadata` is the default or `false`, so kernels on one origin share one CA whether or not they keep metadata. `ca` names another database, and `ca: false` keeps a CA in memory for the kernel's lifetime. Each kernel adds its CA to the bundle instead of replacing it, so every kernel on the origin stays trusted; the bundle keeps the newest 16.
 - **TLS** needs [`@ai-ecoverse/wasm-tls-engine`](https://www.npmjs.com/package/@ai-ecoverse/wasm-tls-engine) installed under `modules`, like a command. Without it, `CONNECT` to port 443 is answered `501` with the reason. A tunnel to another port is accepted, and what the program sends first decides: plain HTTP is served as `http://host:port`, without TLS, and a TLS handshake is terminated as above, or the tunnel closes when it cannot be.
 - **The transport** is an object on the page with `traits` (`manualRedirects`, `encodedBodies`, `maxRequestBody`, and `crossOrigin`: `'cors'` or `'any'`) and `fetch(request)`, which answers `{ status, statusText, headers, body, cancel }` with `body` an async iterable of `Uint8Array`, and optionally `url` and `redirected: true` when it followed redirects itself (`fetchTransport()` does), so a JS program's `fetch` reports where it ended. The kernel pulls the body one chunk at a time, so a slow program slows the download. A rejection with a numeric `status` is answered with that status; any other with `502`. `fetchTransport()` is the plain `fetch` of the page, bound by CORS; its `hint` option is appended to the message of a request it could not make, so a program's 502 can name a way around CORS (`registry.npmjs.org` and jsDelivr allow it); an embedder with a way around CORS passes its own. A response body that delivers no data for `bodyIdleMs` (default 5 minutes; `0` or `Infinity` turns it off) fails with `ETIMEDOUT` instead of waiting forever, which matters because Chrome under heavy load can stop handing a page the rest of a body. A slow body that keeps delivering is never cut off.
-- **Redirect hops** ([#302](https://github.com/ai-ecoverse/slicc-kernel/issues/302)): a transport that follows redirects but can see them (a browser extension's `webRequest.onBeforeRedirect` can) sets `traits.redirectHops` and answers `redirects: [{ status, url, headers }]`, one entry per 3xx it followed, in order (`NetworkRedirectHop`). The kernel then treats it as one that does not follow. Programs get each 3xx as it was answered: through the proxy, curl, requests and `hf` see the real redirect, `X-Repo-Commit` and `ETag` included, and `ctx.fetch` with `redirect: 'manual'` answers it. The rest of the chain is held for the process that asked, at most 30 s and 8 chains per process (64 in all), with the final body unread. A follow-up from that process, with the method the redirect leads to and the same `Range`, `If-None-Match`, `If-Modified-Since`, `If-Range` and `Accept-Encoding` (and an `Authorization` if the first had one), is answered from it without a second request; any other request goes to the transport. A chain that expires or is pushed out is cancelled, and so is one whose follow-up arrives and does not match. A redirect to another origin is usually fetched again rather than replayed: HTTP clients drop `Authorization` when they follow one, so the follow-up no longer matches. The page's own `fetch` cannot show a redirect, so `fetchTransport()` in a page stays a transport that follows, and `redirect: 'manual'` fails there as before. `fetchTransport({ redirect: 'manual' })` hands redirects back unfollowed where the platform's `fetch` shows them (Node; a browser answers an opaque status 0), and `nodeTransport()` does so. A kernel socket also records who connected it (`SO_PEERCRED`): the connecting process's pid, uid and gid on the accepting side, and the listener's on the connecting side, kept when the socket passes to another process. The proxy uses this to keep each process's held chains its own.
+- **Redirect hops** ([#302](https://github.com/ai-ecoverse/slicc-kernel/issues/302)): a transport that follows redirects but can see them (a browser extension's `webRequest.onBeforeRedirect` can) sets `traits.redirectHops` and answers `redirects: [{ status, url, headers }]`, one entry per 3xx it followed, in order (`NetworkRedirectHop`). The kernel then treats it as one that does not follow. Programs get each 3xx as it was answered: through the proxy, curl, requests and `hf` see the real redirect, `X-Repo-Commit` and `ETag` included, and `ctx.fetch` with `redirect: 'manual'` answers it. The rest of the chain is held for the process that asked, at most 30 s and 8 chains per process (64 in all), with the final body unread. A follow-up from that process, with the method the redirect leads to and the same `Range`, `If-None-Match`, `If-Modified-Since`, `If-Range` and `Accept-Encoding` (and an `Authorization` if the first had one), is answered from it without a second request; any other request goes to the transport. A chain that expires or is pushed out is cancelled, and so is one whose follow-up arrives and does not match. After a redirect to another origin the follow-up is expected without `Authorization`, which HTTP clients drop when they follow one; a `Location` matches without its `#fragment`. The page's own `fetch` cannot show a redirect, so `fetchTransport()` in a page stays a transport that follows, and `redirect: 'manual'` fails there as before. `fetchTransport({ redirect: 'manual' })` hands redirects back unfollowed where the platform's `fetch` shows them (Node; a browser answers an opaque status 0), and `nodeTransport()` does so. A kernel socket also records who connected it (`SO_PEERCRED`): the connecting process's pid, uid and gid on the accepting side, and the listener's on the connecting side, kept when the socket passes to another process. The proxy uses this to keep each process's held chains its own.
 - **WebSockets:** a `GET` with `Upgrade: websocket`, in absolute form or inside a tunnel, is bridged to the transport's optional `websocket({ url, protocols, headers, signal })`. It answers `{ protocol, buffered, send(data), messages, close(code, reason), closed }`, with `messages` an async iterable of strings and `Uint8Array`s and `closed` resolving with `{ code, reason }`; a transport that has it says so with `traits.websocket`. The program gets `101 Switching Protocols` with the far end's protocol, then its frames go out as messages and the far end's come back as frames. Pings are answered by the proxy, and close codes pass both ways (a code that may not go on the wire becomes 1000 or 1011). A message over 64 MiB, or more than 16 MiB of payload from the far end waiting for the program, closes with 1009. The proxy stops reading the program while 1 MiB of its messages wait to be sent. A refused connection is `502` with the reason. A transport without WebSockets answers `501` with a message naming the option below, and the kernel logs a warning once, so a program never hangs. `fetchTransport({ webSocket: true })` opens them with the page's `WebSocket` (or pass a constructor); they are off by default because a browser WebSocket carries the user's cookies for the target site, which `fetch` here never sends (`credentials: 'omit'`), so a program could reach a site as the user. A page also cannot send headers on a WebSocket, and an `https:` page cannot open `ws:` to anything but loopback, which is a `502` that says so. `nodeTransport()` has WebSockets on (Node keeps no cookies) and sends the request's headers. The credential-free path for a page is the coming slicc-node relay (`localProxyTransport()`, phase 2 of [#204](https://github.com/ai-ecoverse/slicc-kernel/issues/204)), which has no WebSockets yet.
 - **`localProxyTransport({ url, key })`** sends every request to a local proxy on loopback, such as [slicc-node](https://github.com/ai-ecoverse/slicc-node) or slicc-swift, which fetches it without CORS. Redirects reach the program unfollowed, with every `Set-Cookie`; bodies arrive decoded. `probeLocalProxy({ url, key })` resolves with the proxy's capabilities, or `null` when it is gone or refuses the key, so a page can fall back to `fetchTransport()`. `checkLocalProxy({ url, key })` says why: it resolves with `{ state }`, which is `ready` (with `probe`), `blocked` (the user denied Chrome's Local Network Access permission, `loopback-network`), `unreachable` (nothing answered; its `permission` is `granted`, `prompt` or `unknown`, and with `prompt` the page cannot tell a dismissed prompt from a proxy that is not running, since Chrome asks only once it has a connection), `refused` (with `status` and `error`, such as a stale key) or `incompatible`. The permission is not consulted when the page itself is on loopback. Both take a `fetch` option, and the transport `maxRequestBody` (default 64 MiB) and `bodyIdleMs` like `fetchTransport`.
 - **The host machine** is `host.slicc.internal` (`SLICC_HOST_LOOPBACK`), as `host.docker.internal` is in Docker, while `localhost` and `127.0.0.1` stay the kernel's own loopback. The proxy forwards `http(s)://host.slicc.internal:<port>/…` to the transport as `http(s)://127.0.0.1:<port>/…`, so it reaches the real machine's loopback through the local proxy or Node's `fetch` (through the page's own `fetch`, a request needs CORS from that server and Chrome's Local Network Access permission); other loopback addresses are still refused. This is HTTP and HTTPS through the proxy only: `/etc/hosts` and WASIX name resolution give the name `10.0.2.2`, QEMU's address for its host, so a program that connects a socket to it directly fails with `ENETUNREACH` instead of reaching the kernel's loopback.
@@ -428,6 +428,12 @@ A `hostfs` mount is a folder exported by the local proxy ([slicc-node](https://g
 - **Changes on the host** arrive as invalidations on a watch stream (`POST /api/hostfs/watch`), so an edit made outside shows up on the next access without a remount.
 - **Proxy loss:** when the watch stream ends, or is silent for 45 s, the kernel reconnects once at once. If that fails, the mount is `nomedium`, as with [removable media](#removable-media-fsa): operations fail with `ENOMEDIUM` (`ENODEV` for WASI), and the kernel keeps reconnecting with backoff (1 s to 30 s), so the folder comes back by itself when the proxy does. A mount made while the proxy is down starts as `nomedium`, and so does one whose grant or first watch takes more than 15 s; it comes up by itself when they arrive.
 - **Consistency:** a file that changes on the host while a program reads it fails with `ESTALE`, and the kernel restarts the whole read (up to 3 times) instead of stitching two versions together. On a ranged mount an open file keeps the version it was opened at (its ETag) until it writes, also when it is handed to a child process: a page fetched after the host changed the file fails the program's read with `ESTALE` ("Stale file handle") instead of mixing versions. A file deleted on the host while a program holds it open fails the next read that goes to the host with `ENOENT`, not with NFS's `ESTALE`.
+- **Over a port:** the hook may answer `{ port, token?, capabilities? }` instead, for a folder that doesn't come from slicc-node over HTTP: for example, a linked `slicc` CLI's export, which the page reaches over its own link. The page implements a `HostfsTransport` for the folder (`capabilities`, `token`, `maxIo`, `call(op, body)`, `read(fh, offset, size, ifMatch?)`, `write(fh, offset, bytes)`, `watch(signal, onEvent, onOpen)`, with slicc-node's ops and answers) and serves it with `serveHostfsPort(port, transport)`, exported from both entries, which returns a function that stops serving and closes the port.
+  - The kernel moves the port into its worker with the grant. Calls, reads and writes are request and reply messages there, and the watch is a stream of messages that the kernel ends by aborting.
+  - Errors keep their `code`. A transport that throws `ENOTCONN` (its link is down) makes the mount `nomedium` and reconnecting, as a lost proxy does: the kernel asks the hook again, and the page answers with a new port.
+  - A port that dies doesn't hang anyone: pending requests fail with `ENOTCONN` when the port closes (its `close` event), when its watch ends or goes silent, and, on engines that don't report a closed peer, when a request gets no answer within 60 s (plus 1 s per MiB of a read or write), which fails it with `ETIMEDOUT`. After any of these the port counts as gone and is closed, files opened through it are `ESTALE`, and the next operation reconnects. `serveHostfsPort` stops serving when its port closes.
+  - A read reply sends a copy of the transport's bytes, so a transport may return a view into storage it keeps (a cache, a pooled or shared buffer).
+  - `token` names the connection (a link id, say). When the next port's token differs, files that were open become `ESTALE`. Without a token, every port is a new connection.
 - **Node:** a page's requests carry its `Origin`, and slicc-node grants a token to that origin only. Node's `fetch` sends none, so a Node embedder passes `createNodeKernel` the origin its `hostfs` hook asked for the grant with, as `hostfsOrigin`, and the kernel adds it to every hostfs request. Without it the mount stays `nomedium`. `hostfsFetch(url, init)` replaces the `fetch` those requests use (for an agent or a test proxy), and gets the `Origin` too when both are given.
 
 ### Mounting from a program

package/dist/index.js

@@ -1632,6 +1632,79 @@ function localProxyTransport(options) {
   };
 }
 
+// src/mount/driver.ts
+function fsError(code, message = code) {
+  return Object.assign(new Error(message), { code });
+}
+
+// src/mount/hostfs.ts
+var HOSTFS_IO = 16 * 1024 * 1024;
+var codeOf = (err) => err?.code;
+var errorOf = (err) => ({
+  code: typeof codeOf(err) === "string" ? codeOf(err) : "EIO",
+  message: err?.message ?? String(err)
+});
+function serveHostfsPort(port, transport) {
+  const watches = /* @__PURE__ */ new Map();
+  const send2 = (reply, transfer = []) => port.postMessage(reply, transfer);
+  const listener = async ({
+    data
+  }) => {
+    const { id, op } = data;
+    if (op === "unwatch") return watches.get(id)?.abort();
+    if (op === "watch") {
+      const stop2 = new AbortController();
+      watches.set(id, stop2);
+      try {
+        await transport.watch(
+          stop2.signal,
+          (event) => send2({ id, event }),
+          () => send2({ id, open: true })
+        );
+        send2({ id, end: true });
+      } catch (err) {
+        send2({ id, error: errorOf(err) });
+      } finally {
+        watches.delete(id);
+      }
+      return;
+    }
+    try {
+      if (op === "read") {
+        const read = await transport.read(
+          data.fh,
+          data.offset,
+          data.size,
+          data.ifMatch
+        );
+        const bytes = read.slice();
+        return send2({ id, bytes }, [bytes.buffer]);
+      }
+      if (op === "write") {
+        await transport.write(data.fh, data.offset, data.bytes);
+        return send2({ id });
+      }
+      if (op !== "call") throw fsError("EINVAL", `unknown hostfs port op ${op}`);
+      send2({
+        id,
+        result: await transport.call(data.name, data.body)
+      });
+    } catch (err) {
+      send2({ id, error: errorOf(err) });
+    }
+  };
+  const stop = () => {
+    port.removeEventListener("message", listener);
+    port.removeEventListener("close", stop);
+    for (const watch of watches.values()) watch.abort();
+    port.close();
+  };
+  port.addEventListener("message", listener);
+  port.addEventListener("close", stop);
+  port.start();
+  return stop;
+}
+
 // src/index.ts
 var ISOLATION = "slicc-kernel needs a cross-origin isolated page (Cross-Origin-Opener-Policy: same-origin and Cross-Origin-Embedder-Policy: require-corp or credentialless)";
 function streamer(callback) {
@@ -1746,7 +1819,7 @@ async function createKernel(options = {}) {
     try {
       if (!options.hostfs) throw new Error("this page gives no host folders");
       const grant = await options.hostfs(source, { readonly });
-      worker.postMessage({ hostfsGrant: { id, grant } });
+      worker.postMessage({ hostfsGrant: { id, grant } }, "port" in grant ? [grant.port] : []);
     } catch (err) {
       worker.postMessage({ hostfsGrant: { id, error: err.message ?? String(err) } });
     }
@@ -1875,6 +1948,7 @@ export {
   createKernel,
   fetchTransport,
   localProxyTransport,
-  probeLocalProxy
+  probeLocalProxy,
+  serveHostfsPort
 };
 //# sourceMappingURL=index.js.map