@factoidal/core 0.29.0 → 0.31.0

package/CHANGELOG.md

@@ -1,6 +1,24 @@
 # Changelog
 
-## Unreleased
+## 0.31.0 (2026-11-05)
+
+- Stack use on phones. The engine failed on iPhone Safari with "Maximum call stack size exceeded". Two faults are fixed:
+  - The ordered regex matcher (`cmatchE`) nested stack frames for each character of a star. `REPLACE`, `fn:replace` and other capture-aware operations overflowed on long text: in 0.30.0, `REPLACE` with a star failed at 40,000 characters under Node's default stack. A star over one character class is now a loop, with the same order of outcomes. A star over a group still recurses.
+  - Engine start decoded a test fixture (`Codec/Archive.lean`, `realTar`) with one stack frame for each byte. In WebKit, start needed about 1 MB of stack. The decoder is now a loop.
+  - `test/stack-use.test.mjs` gates both faults in the package suite. All of its tests fail against 0.30.0.
+- SPARQL `SERVICE`: variable correlation, and scoped query provider grants (https://github.com/danbri/factoidal-core/pull/283).
+- XPath: `parse-QName` function values capture static namespaces (https://github.com/danbri/factoidal-core/pull/278).
+- VC-JOSE-COSE: draft plan and checklist (https://github.com/danbri/factoidal-core/pull/232).
+- Proof that checked normalization is preserved into the actual RDFS closure (https://github.com/danbri/factoidal-core/pull/288).
+
+## 0.30.0 (2026-11-04)
+
+- The Lean XMPP server now has an in-page host (`@factoidal/core/xmpp/inpage`): `openInPageXmppServer` exposes ordered message connections and `inPageWebSocket` adapts them for RFC 7395 clients. Protocol decisions remain in Lean, with shared routing for direct messages, MUC rooms, MIX channels and experimental GC3 groups. This provides an app-to-app transport; it does not include a chat UI or WebMCP bridge.
+- The Node XMPP host adds STARTTLS and WebSocket transport, IQ routing and sealed LWS-backed state for restart persistence. LWS writes enforce the configured content limit with a 413 response; component requests carry their principal through authorization.
+
+- `openInPageLwsStorage({ engine, baseIri, principal, rootAcl?, storageKey?, snapshot?, onSnapshot? })` (`@factoidal/core/lws/inpage`): one Lean LWS storage with no network, in a page, a worker or Node, reached through a `fetch`-shaped function that answers real `Response`s. Every request is the component principal's (issue 1238) and Web Access Control decides it; notifications are off. The storage is kept as one snapshot, sealed when a `storageKey` is given: after each change it is handed to `onSnapshot`, and the caller passes it back as `snapshot` to reopen (the engine refuses one sealed under another key). No Node API. Test: `test/lws-inpage.test.mjs`.
+- `openLwsDataset(storageRootUrl, { fetch, auth })` (`@factoidal/core/lws/dataset`): an LWS storage as an RDF dataset (https://github.com/danbri/factoidal/issues/1011, version 2 of the view). The default graph is empty. The first named graph is the table of contents, named by the storage description IRI (the target of the root's `https://www.w3.org/ns/lws#storage` link; the result member `toc`): the containment as `lws:items`, each listing member's `rdf:type` (`lws:Container` or `lws:DataResource`) with its `dcterms:format`, `schema:size` and `dcterms:modified`, and `<resource> <http://www.iana.org/assignments/relation/linkset> <linkset>` for each linkset read. Then one named graph per RDF resource (named by its IRI) and one per linkset (named by the linkset IRI, one triple per link; relation types become IRIs from the LWS vocabulary, else `http://www.iana.org/assignments/relation/<name>`). `query()` answers `{result, evidence}`; the evidence is each document read with its ETag and each linkset read with its linkset IRI and ETag. A full crawl on every open (one HEAD and one GET per resource for its linkset), no search index, no caching. The definition is Lean (`LWS/DatasetView.lean`, theorems `view_names_nodup`, `view_toc_first`, `view_replace`); the source theorem `lwsSource_query_sound` rests on the named assumption `lwsReadsAtSnapshot`. Needs a wasm build with version 2 of the ops `lwsViewPlan` (with `metadata`) and `lwsDatasetView`.
+- The experimental code notice now says "EXPERIMENTAL UNDER-REVIEWED CODE" (was "UNREVIEWED"), in the README, the package description, every help text, the warning on standard error when a server, client or key command starts, and the hub pages. Owner decision of 2026-10-28, confirmed 2026-11-03.
 
 ## 0.29.0 (2026-11-02)
 

package/README.md

@@ -1,6 +1,6 @@
 # @factoidal/core
 
-> ⚠️ **EXPERIMENTAL UNREVIEWED CODE - USE WITH CARE - DO NOT CREATE PUBLIC SERVICES USING THIS WITHOUT CAREFUL INDEPENDENT EXPERT REVIEW.**
+> ⚠️ **EXPERIMENTAL UNDER-REVIEWED CODE - USE WITH CARE - DO NOT CREATE PUBLIC SERVICES USING THIS WITHOUT CAREFUL INDEPENDENT EXPERT REVIEW.**
 >
 > This covers the whole core library and every command line tool, and in
 > particular all protocol, authentication and security code (the LWS, Solid,

package/bin/factoidal.mjs

@@ -94,7 +94,7 @@ function err (line) { console.error(line) }
 // Owner, 2026-10-27: every protocol, authentication and security component,
 // the CLIs and the core library carry this notice. It is printed with every
 // help text and on standard error when a server, client or key command starts.
-const REVIEW_WARNING = 'EXPERIMENTAL UNREVIEWED CODE - USE WITH CARE - DO NOT CREATE PUBLIC SERVICES USING THIS WITHOUT CAREFUL INDEPENDENT EXPERT REVIEW.'
+const REVIEW_WARNING = 'EXPERIMENTAL UNDER-REVIEWED CODE - USE WITH CARE - DO NOT CREATE PUBLIC SERVICES USING THIS WITHOUT CAREFUL INDEPENDENT EXPERT REVIEW.'
 
 /** The commands that serve, speak a protocol, or handle keys and tokens. */
 const WARNED_COMMANDS = new Set(['serve', 'lws-serve', 'solid-serve', 'solid-client', 'lws-client',
@@ -128,7 +128,7 @@ commands:
   authz-serve DIR             serve an OAuth 2.0 authorization server for LWS
                               (RFC 8414 metadata, JWK Set, RFC 8693 token exchange)
   authz-keys DIR              print the public JWK Set of an authorization server
-  xmpp-serve DIR              serve XMPP (RFC 6120/6121) over TCP or direct TLS
+  xmpp-serve DIR              serve XMPP (RFC 6120/6121): TCP, STARTTLS or direct TLS
   webdav-serve CONFIGDIR      serve local and remote LWS storages to file
                               browsers over WebDAV (RFC 4918)
   webdav-user FILE NAME       add or replace a user in a webdav-serve users
@@ -481,14 +481,8 @@ usage: factoidal lws-serve DIR (--authz DIR | --issuer IRI | --auth-config FILE
 Starts a Node HTTP server that answers every request through the engine's
 lwsStep operation.
 
-DIR names the storage directory. The storage is kept in the engine
-handle and written to DIR/lws-state.json after every request that
-changed it (temporary file, fsync, rename), and read back on the next
-start. The engine decides the snapshot format and refuses a snapshot
-path with a dot segment or other traversal form
-(formal/lean4/L4Factoidal/LWS/Snapshot.lean).
-
-With --storage files (off by default) a write costs the same whatever the
+DIR names the storage directory. By default (--storage files, since
+2026-11-03) a write costs the same whatever the
 size of the storage: DIR holds one file per resource body (DIR/bodies/),
 a journal of metadata changes (DIR/lws-journal) and a metadata file
 (DIR/lws-files.json) that the journal is folded into from time to time.
@@ -500,6 +494,12 @@ byte (formal/lean4/L4Factoidal/LWS/FileLayout.lean); the storage is still
 held in memory while the server runs
 (docs/designissues/2026-10-22-lws-storage-on-disk.md).
 
+With --storage snapshot the storage is written whole to DIR/lws-state.json
+after every request that changed it (temporary file, fsync, rename), and
+read back on the next start. The engine decides the snapshot format and
+refuses a snapshot path with a dot segment or other traversal form
+(formal/lean4/L4Factoidal/LWS/Snapshot.lean).
+
 Encryption at rest (off by default): when the environment variable
 FACTOIDAL_LWS_KEY holds a 32-byte key (64 hex digits, or 43 characters of
 base64url), the snapshot is sealed with ChaCha20-Poly1305 (HACL*) under a
@@ -577,9 +577,9 @@ The engine decides the file, its contents and every refusal
   --storage snapshot|files
                   how DIR keeps the storage (config member storage;
                   default the environment variable FACTOIDAL_LWS_STORAGE,
-                  else snapshot): one snapshot file written whole after
-                  each change, or one file per body with a journal, so a
-                  write costs the same whatever the storage size
+                  else files): one file per body with a journal, so a
+                  write costs the same whatever the storage size, or one
+                  snapshot file written whole after each change
                   https://github.com/danbri/factoidal/issues/918
   --cors-origin ORIGIN
                   answer CORS (Fetch Standard) only for this origin, written
@@ -744,10 +744,13 @@ those operations landed makes this command exit 3 and say so.
 
 Specification: https://solidproject.org/TR/protocol`,
 
-  'xmpp-serve': `factoidal xmpp-serve - serve XMPP (RFC 6120 and RFC 6121) over TCP or direct TLS
+  'xmpp-serve': `factoidal xmpp-serve - serve XMPP (RFC 6120 and RFC 6121) over TCP, STARTTLS or direct TLS
 
 usage: factoidal xmpp-serve DIR [--domain NAME] [--port N] [--host ADDRESS]
-                            [--accounts FILE] [--cert FILE --key FILE]
+                            [--accounts FILE] [--cert FILE --key FILE [--starttls]]
+                            [--ws-port N [--ws-path PATH] [--ws-origin ORIGIN]...]
+                            [--muc DOMAIN] [--mix DOMAIN] [--gc3 DOMAIN]
+                            [--state-key FILE]
 
 Starts a Node TCP server that runs one engine session per connection and
 answers every byte through the engine's xmppSessionFeed operation.
@@ -761,11 +764,41 @@ DIR/spool/<bare jid>/ is delivered to that account's bound session.
   --domain NAME    the domain the server is authoritative for (default localhost);
                    a stream for any other domain is answered host-unknown
   --port N         the port to bind; 0 takes an ephemeral one
-                   (default 5222, or 5223 with --cert and --key)
-  --host ADDRESS   the address to bind (default 127.0.0.1)
+                   (default 5222; 5223 with --cert and --key and no --starttls)
+  --host ADDRESS   the address to bind (default 127.0.0.1); without
+                   --cert and --key only a loopback address is accepted
   --accounts FILE  the accounts file (default DIR/accounts)
-  --cert FILE      PEM certificate chain: XEP-0368 direct TLS, no STARTTLS
+  --cert FILE      PEM certificate chain: XEP-0368 direct TLS, or STARTTLS with --starttls
   --key FILE       PEM private key for --cert
+  --starttls       listen in clear and REQUIRE STARTTLS (RFC 6120 section 5)
+                   before anything else; no SASL mechanism is offered and no
+                   credential is accepted until the TLS handshake finishes
+  --ws-port N      also serve XMPP over WebSocket (RFC 7395, the "xmpp"
+                   subprotocol of RFC 6455) for browser clients such as
+                   Strophe.js; wss: with --cert and --key, else loopback only
+  --ws-path PATH   the WebSocket endpoint path (default /xmpp-websocket)
+  --ws-origin ORIGIN  a browser Origin allowed to connect, e.g.
+                   http://localhost:5173 (repeatable); a request from any
+                   other Origin is refused 403, and one with no Origin
+                   (not a browser) is accepted only when no --ws-origin is given
+  --muc DOMAIN     also serve XEP-0045 multi-user chat rooms at DOMAIN, e.g.
+                   conference.example.com: temporary rooms, made by the first
+                   join and gone with the last occupant; no history, no
+                   configuration, no persistence (L4Factoidal/XMPP/Muc.lean)
+  --mix DOMAIN     also serve XEP-0369 MIX channels at DOMAIN, e.g. mix.example.com,
+                   joined through this server (XEP-0405 MIX-PAM); channels are
+                   made by an IQ <create/>, kept in memory only, JID-hidden; no
+                   PubSub nodes, no archive (L4Factoidal/XMPP/Mix.lean)
+  --state-key FILE keep the server's state across restarts: rosters, messages held
+                   for offline accounts, MIX channels and GC3 groups, in an LWS
+                   storage in DIR/state, sealed (ChaCha20-Poly1305) under the key
+                   in FILE (64 hex digits; keep FILE outside DIR). A wrong key
+                   stops the start. Plaintext roster files are migrated, then
+                   deleted. Without it, that state is lost at a restart.
+  --gc3 DOMAIN     also serve PROTOTYPE "Groupchat 3.0" groups at DOMAIN: persistent
+                   membership, stable participant ids, optional events. There is
+                   no published GC3 spec; every wire format is invented and listed
+                   in L4Factoidal/XMPP/Gc3.lean and danbri/factoidal issue 1236
 
 The server makes no protocol decision. Stream negotiation, SASL PLAIN and
 SCRAM-SHA-256, resource binding, the roster, presence subscription and
@@ -774,7 +807,7 @@ disco#info are answered by formal/lean4/L4Factoidal/XMPP/. A WebAssembly
 module built before the session operations landed makes this command
 exit 3 and say so.
 
-Not implemented: STARTTLS (use --cert/--key), server-to-server federation,
+Not implemented: SCRAM channel binding, server-to-server federation,
 MUC/MIX/GC3 rooms.
 
 Specifications: https://www.rfc-editor.org/rfc/rfc6120 and
@@ -1150,7 +1183,7 @@ function parseArguments (args, valueOptions) {
 class UsageError extends Error {}
 
 /** Options that may be given more than once; their value is the list. */
-const REPEATABLE_OPTIONS = new Set(['cors-origin', 'saml-idp', 'saml-decrypt-key', 'allow-host', 'allow-origin'])
+const REPEATABLE_OPTIONS = new Set(['cors-origin', 'saml-idp', 'saml-decrypt-key', 'allow-host', 'allow-origin', 'ws-origin'])
 
 const VALUE_OPTIONS = {
   xpath4: new Set(['expression', 'file', 'input', 'format']),
@@ -1178,7 +1211,7 @@ const VALUE_OPTIONS = {
   'authz-serve': new Set(['port', 'host', 'issuer', 'resource', 'alg', 'token-lifetime', 'rotate-after',
     'cors-origin', 'saml-idp', 'saml-decrypt-key']),
   'authz-keys': new Set([]),
-  'xmpp-serve': new Set(['port', 'host', 'domain', 'accounts', 'cert', 'key']),
+  'xmpp-serve': new Set(['port', 'host', 'domain', 'accounts', 'cert', 'key', 'ws-port', 'ws-path', 'ws-origin', 'muc', 'mix', 'gc3', 'state-key']),
   'webdav-serve': new Set(['port', 'host', 'users', 'guest', 'cert', 'key', 'login-limit', 'login-window', 'xmpp', 'xmpp-password-file',
     'xmpp-host', 'xmpp-port', 'xmpp-allow', 'xmpp-resource', 'xmpp-ca', 'allow-host', 'allow-origin']),
   'webdav-user': new Set(['iterations']),
@@ -2825,9 +2858,11 @@ async function commandXmppServe (positional, options) {
   if (useTls && (typeof options.cert !== 'string' || typeof options.key !== 'string')) {
     throw new UsageError('--cert and --key go together')
   }
+  const startTls = options.starttls === true
+  if (startTls && !useTls) throw new UsageError('--starttls needs --cert and --key')
   const settings = {
     dir: positional[0],
-    port: servePort(options, useTls ? 5223 : 5222),
+    port: servePort(options, useTls && !startTls ? 5223 : 5222),
     host: serveHost(options),
     onError: (error) => { if (!quiet) err(`xmpp-serve: ${error.message}`) }
   }
@@ -2837,6 +2872,51 @@ async function commandXmppServe (positional, options) {
     settings.cert = await fs.readFile(options.cert)
     settings.key = await fs.readFile(options.key)
   }
+  if (startTls) settings.startTls = true
+  if (options['state-key'] !== undefined) {
+    if (typeof options['state-key'] !== 'string') throw new UsageError('--state-key needs a FILE')
+    // 64 hex digits or base64url of 32 bytes; the engine checks it.
+    settings.stateKey = (await fs.readFile(options['state-key'], 'utf8')).trim()
+  }
+  if (options.muc !== undefined) {
+    if (typeof options.muc !== 'string' || !/^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$/i.test(options.muc)) {
+      throw new UsageError('--muc needs a domain name, e.g. conference.example.com')
+    }
+    settings.mucDomain = options.muc
+  }
+  if (options.mix !== undefined) {
+    if (typeof options.mix !== 'string' || !/^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$/i.test(options.mix)) {
+      throw new UsageError('--mix needs a domain name, e.g. mix.example.com')
+    }
+    if (options.mix === options.muc) throw new UsageError('--mix and --muc need different domains')
+    settings.mixDomain = options.mix
+  }
+  if (options.gc3 !== undefined) {
+    if (typeof options.gc3 !== 'string' || !/^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)+$/i.test(options.gc3)) {
+      throw new UsageError('--gc3 needs a domain name, e.g. gc3.example.com')
+    }
+    if (options.gc3 === options.muc || options.gc3 === options.mix) throw new UsageError('--gc3 needs a domain of its own')
+    settings.gc3Domain = options.gc3
+  }
+  if (options['ws-port'] !== undefined) {
+    const wsPort = Number(options['ws-port'])
+    if (!Number.isInteger(wsPort) || wsPort < 0 || wsPort > 65535) throw new UsageError('--ws-port needs a port number')
+    const origins = options['ws-origin'] === undefined ? []
+      : Array.isArray(options['ws-origin']) ? options['ws-origin'] : [options['ws-origin']]
+    settings.websocket = {
+      port: wsPort,
+      path: typeof options['ws-path'] === 'string' ? options['ws-path'] : '/xmpp-websocket',
+      allowedOrigins: origins,
+      allowNoOrigin: origins.length === 0
+    }
+    if (useTls) { settings.websocket.cert = settings.cert; settings.websocket.key = settings.key }
+  }
+  // Invariant I4 (docs/designissues/2026-10-28-feature-combination-safety.md,
+  // C4): with no TLS this listener offers PLAIN, which sends the password in
+  // clear, so it binds to loopback only.
+  if (!useTls && !/^(127\.\d+\.\d+\.\d+|::1|localhost)$/.test(settings.host)) {
+    throw new UsageError(`xmpp-serve: no TLS on ${settings.host}: give --cert and --key (with --starttls for port 5222), or bind to loopback`)
+  }
   let running
   try {
     running = await listen(settings)
@@ -2849,9 +2929,19 @@ async function commandXmppServe (positional, options) {
   if (!quiet) {
     err(`xmpp-serve: listening on ${running.origin} for domain ${running.xmpp.domain}` +
         ` (${running.xmpp.accounts} accounts, state ${positional[0]}` +
-        (useTls ? ', direct TLS)' : ', PLAINTEXT: no TLS on this listener)'))
+        (startTls ? ', STARTTLS required)' : useTls ? ', direct TLS)' : ', PLAINTEXT: no TLS on this listener)'))
+    if (settings.mucDomain) err(`xmpp-serve: multi-user chat (XEP-0045) at ${settings.mucDomain}`)
+    if (settings.mixDomain) err(`xmpp-serve: MIX channels (XEP-0369, joined by XEP-0405) at ${settings.mixDomain}`)
+    if (settings.gc3Domain) err(`xmpp-serve: GC3 PROTOTYPE groups at ${settings.gc3Domain} (no published spec; invented wire format)`)
+    if (settings.stateKey) err(`xmpp-serve: state kept across restarts in ${positional[0]}/state, sealed under the state key`)
+    else err('xmpp-serve: WARNING: no --state-key: held messages, MIX channels and GC3 groups are lost at a restart, and rosters are written in clear')
+  }
+  if (running.wsOrigin && !quiet) {
+    err(`xmpp-serve: XMPP over WebSocket on ${running.wsOrigin}` +
+        (settings.websocket.allowedOrigins.length ? ` for Origin ${settings.websocket.allowedOrigins.join(', ')}` : ' (no browser Origin allowed)'))
   }
   out(running.origin)
+  if (running.wsOrigin) out(running.wsOrigin)
   return untilInterrupted(running, quiet)
 }
 

package/l4-assets/l4factoidal.js

@@ -37,7 +37,7 @@
 // bytes change.
 
 // Stamped by formal/lean4/Wasm/build-wasm.sh step 9 -- do not hand-edit.
-const WASM_VERSION = "a020bab67e41";
+const WASM_VERSION = "14599aeba24a";
 
 import createModule from './l4factoidal.mjs';
 

package/l4-assets/version.json

@@ -2,17 +2,17 @@
   "engine": "lean4",
   "note": "The Lean 4 (L4Factoidal) engine build provenance and claims: the only engine of @factoidal/core since 0.9.0 (issue 713). This committed file is a placeholder. tools/fetch-builds.sh --set web (through tools/stamp-l4-version.mjs) writes the provenance members from the danbri/factoidal-builds release that built the wasm; formal/lean4/Wasm/build-wasm.sh step 9 writes them for a local build. Do not commit a stamped copy.",
   "stamped": true,
-  "release": "danbri/factoidal-builds build-72de36349ac0-web",
-  "releaseRun": "https://github.com/danbri/factoidal-builds/actions/runs/41472062630",
-  "gitSha": "72de36349ac003a402074cee9aa329828977f118",
-  "builtAt": "2026-11-02T22:15:10Z",
+  "release": "danbri/factoidal-builds build-ace53d982f83-web",
+  "releaseRun": "https://github.com/danbri/factoidal-builds/actions/runs/42050112010",
+  "gitSha": "ace53d982f83e6bf0a1a749a5e800481cc092b9f",
+  "builtAt": "2026-11-05T16:35:31Z",
   "leanToolchain": "leanprover/lean4:v4.33.1",
   "emscripten": "6.0.8",
   "abiVersion": "1",
   "engineVersion": "l4factoidal-wasm 0.2.0 (dispatch: RDF parse/serialize, SPARQL query+update, RDFS/OWL-RL closures, RDFC-1.0, CL/IKL)",
   "version": "l4factoidal-wasm 0.2.0 (dispatch: RDF parse/serialize, SPARQL query+update, RDFS/OWL-RL closures, RDFC-1.0, CL/IKL)",
-  "wasmSha256": "a020bab67e41d62992a520f4180a170cce0c4d419d687cd5eb227d2d14bba0b6",
-  "wasmBytes": 31003213,
+  "wasmSha256": "14599aeba24a46bf05e1c7ce8aa885df536fe1398a797034eff107e04fa4ebc3",
+  "wasmBytes": 31371405,
   "claims": {
     "source": "formal/lean4 (L4Factoidal): no sorry, no user axioms, no native_decide; W3C behaviour pinned by build-time #guard",
     "suitesAtBuildSha": "see docs/test-results and formal/lean4/PORT_NOTES.md at gitSha"

package/lib/version.mjs

@@ -2,4 +2,4 @@
 // needs import attributes, which not every runtime this package targets
 // reads). test/dts-drift.test.js checks that it equals package.json's
 // `version`.
-export const version = '0.29.0';
+export const version = '0.31.0';

package/lws/dataset.mjs

@@ -0,0 +1,160 @@
+// An LWS storage as an RDF dataset (https://github.com/danbri/factoidal/issues/1011,
+// version 2): a table-of-contents graph named by the storage description IRI
+// (the containment as lws:items, the listing members, the link from each
+// resource to its linkset), one named graph per RDF document, one named graph
+// per linkset, and the entity tag of every document and linkset read.
+// Design record: docs/designissues/2026-10-25-lws-dataset-view.md
+//
+// WHAT THIS FILE IS ALLOWED TO DO
+// Loop. Start at the storage root; ask the LWS client (lws/client.mjs) to
+// list a container; give the engine the listing and the container IRI (op
+// lwsViewPlan), which answers which members to list, which to read and whose
+// linksets to read; read those with the client (read, readLinkset); give the
+// engine every listing, read and linkset read (op lwsDatasetView), which
+// answers the dataset as N-Quads with the evidence and the name of the
+// table-of-contents graph; open that text as a Dataset. Every decision is
+// Lean: which members are containers, which media types have an RDF reader
+// and which resources have a linkset to read
+// (L4Factoidal/LWS/DatasetViewRead.lean), the entity tag and the graph of
+// each read, the links of each linkset, the storage description IRI, and the
+// view itself (L4Factoidal/LWS/DatasetView.lean). This file reads no header,
+// no link and chooses no media type.
+//
+// Limits (stated in API.md): a full crawl on every open; one HEAD and one GET
+// per resource for its linkset; no search index; no caching; the snapshot is
+// a sequence of GETs, not one read of the storage.
+
+import { loadEngine } from '../bin/engine.mjs'
+import { parse } from '../index.mjs'
+import { createLwsClient, LwsClientError } from './client.mjs'
+import { FactoidalError } from '../lib/errors.mjs'
+
+/** The engine ops this host needs, beyond the client's. */
+export const LWS_DATASET_OPS = ['lwsViewPlan', 'lwsDatasetView']
+
+function isUnknownOp (error) {
+  const text = error && error.message ? String(error.message) : String(error)
+  return text.indexOf('unknown op') >= 0
+}
+
+/**
+ * Whether the loaded engine carries the dataset-view ops.
+ *
+ * @param {object} engine the handle from `loadEngine()`
+ * @returns {Promise<{available: boolean, reason: string}>}
+ */
+export async function lwsDatasetOpsAvailable (engine) {
+  try {
+    const plan = engine.call('lwsViewPlan', ['{}'])
+    if (!plan || !Array.isArray(plan.metadata)) {
+      return {
+        available: false,
+        reason: 'the loaded l4factoidal WebAssembly module carries version 1 of lwsViewPlan ' +
+          '(no metadata member; rebuild formal/lean4/Wasm)'
+      }
+    }
+    return { available: true, reason: 'lwsViewPlan answered' }
+  } catch (error) {
+    if (isUnknownOp(error)) {
+      return {
+        available: false,
+        reason: 'the loaded l4factoidal WebAssembly module does not carry lwsViewPlan ' +
+          '(rebuild formal/lean4/Wasm)'
+      }
+    }
+    return { available: true, reason: 'lwsViewPlan answered with an engine error' }
+  }
+}
+
+/**
+ * Crawl an LWS storage and open it as a Dataset.
+ *
+ * @param {string} storageRootUrl the URL of the storage root container
+ * @param {object} [options]
+ * @param {Function} [options.fetch] the fetch to use (default `globalThis.fetch`)
+ * @param {object} [options.auth] passed to `createLwsClient`:
+ *   `{ subjectToken, subjectTokenType }`, `{ selfIssued }` or `{ session }`
+ * @param {object} [options.engine] an engine handle to reuse
+ * @param {number} [options.maxContainers] stop with an error after listing
+ *   this many containers (default 10000)
+ * @returns {Promise<object>} `{ dataset, toc, evidence, skipped, graphs,
+ *   query(sparql, options), dispose() }`
+ */
+export async function openLwsDataset (storageRootUrl, options = {}) {
+  if (typeof storageRootUrl !== 'string') {
+    throw new TypeError('openLwsDataset: storageRootUrl must be a string')
+  }
+  const engine = options.engine !== undefined ? options.engine : await loadEngine()
+  const probe = await lwsDatasetOpsAvailable(engine)
+  if (!probe.available) {
+    throw new FactoidalError(`openLwsDataset: ${probe.reason}`, { code: 'op.unknown' })
+  }
+  const auth = options.auth !== undefined && options.auth !== null ? options.auth : {}
+  const client = await createLwsClient({ ...auth, engine, fetch: options.fetch })
+  const maxContainers = typeof options.maxContainers === 'number' ? options.maxContainers : 10000
+
+  const queue = [storageRootUrl]
+  const listed = new Set()
+  const listings = []
+  const reads = []
+  const linksets = []
+  const skipped = []
+  while (queue.length > 0) {
+    const container = queue.shift()
+    if (listed.has(container)) continue
+    if (listed.size >= maxContainers) {
+      throw new FactoidalError(`openLwsDataset: more than ${maxContainers} containers`,
+        { code: 'lws.dataset.limit' })
+    }
+    listed.add(container)
+    const done = await client.list(container)
+    listings.push({ container, done })
+    const plan = engine.call('lwsViewPlan', [JSON.stringify(done), container])
+    for (const c of plan.containers) queue.push(c)
+    for (const s of plan.skipped) skipped.push(s)
+    for (const resource of plan.resources) {
+      try {
+        reads.push({ resource, done: await client.read(resource) })
+      } catch (error) {
+        if (!(error instanceof LwsClientError)) throw error
+        skipped.push({ resource, reason: `read: ${error.reason}` })
+      }
+    }
+    for (const resource of plan.metadata) {
+      try {
+        linksets.push({ resource, done: await client.readLinkset(resource) })
+      } catch (error) {
+        if (!(error instanceof LwsClientError)) throw error
+        skipped.push({ resource, reason: `linkset: ${error.reason}` })
+      }
+    }
+  }
+
+  const view = engine.call('lwsDatasetView', [JSON.stringify({ listings, reads, linksets })])
+  for (const s of view.skipped) skipped.push(s)
+  const dataset = await parse(view.nquads, { format: 'nquads' })
+  const evidence = view.evidence.map((e) => ({ ...e }))
+
+  return {
+    dataset,
+    /** The name of the table-of-contents graph: the storage description IRI. */
+    toc: view.toc,
+    /** Each RDF document read, `{ resource, etag, lastModified }`, then each
+     *  linkset read, `{ resource, linkset, etag }`. */
+    evidence,
+    /** Each member or read left out of the dataset: `{ resource, reason }`. */
+    skipped,
+    /** The number of named graphs (the table of contents, the documents and
+     *  the linksets). */
+    graphs: view.graphs,
+    /**
+     * Run a SPARQL query over the dataset.
+     * @returns {Promise<{result: *, evidence: Array<object>}>} `result` as
+     *   `Dataset.query` answers; `evidence` the documents and linksets read
+     */
+    async query (sparql, queryOptions) {
+      return { result: await dataset.query(sparql, queryOptions), evidence: evidence.slice() }
+    },
+    dispose () { dataset.dispose() }
+  }
+}

package/lws/inpage.mjs

@@ -0,0 +1,135 @@
+// factoidal/lws/inpage.mjs — one Lean LWS storage with no network: it runs
+// in the page (or a worker, or Node) beside the code that uses it, and is
+// reached through a `fetch`-shaped function.
+//
+//   import { loadEngine } from '@factoidal/core/browser'
+//   import { openInPageLwsStorage } from '@factoidal/core/lws/inpage'
+//   const pod = await openInPageLwsStorage({ engine: await loadEngine(),
+//     baseIri: 'https://pod.foafos.local/', principal: 'urn:foafos:shell',
+//     rootAcl, storageKey, snapshot: saved,
+//     onSnapshot: (s) => save(s) })
+//   const r = await pod.fetch('https://pod.foafos.local/notes/a.txt', { method: 'PUT', body: 'hi' })
+//
+// The storage has a component principal (issue 1238): every request is that
+// agent's, no token is read, and Web Access Control still decides it
+// (L4Factoidal/LWS/Authorization.lean). Whoever holds the object can use it,
+// so the host page decides which code gets it, as with xmpp/inpage.mjs.
+//
+// It is kept as one snapshot (`lwsSnapshot`), which the engine seals when a
+// `storageKey` is given. After each step that changed the storage this file
+// asks the engine for the snapshot and gives it to `onSnapshot`; the caller
+// keeps it (IndexedDB, a file) and passes it back as `snapshot` to reopen.
+// The engine checks that snapshot and refuses one sealed under another key.
+//
+// The engine makes every protocol decision (status, headers, bodies). This
+// file turns a fetch call into the request record of the wasm ABI
+// (docs/lws-solid-conformance.md, "wasm ABI") and the response record into a
+// Response. Notifications are off: there is no network to deliver them on.
+// No Node API; `crypto.getRandomValues`, `Request` and `Response` only.
+
+/** Random bytes for the engine, base64 (see lws/server.mjs, entropyBase64). */
+function entropyBase64 (size = 32) {
+  const bytes = new Uint8Array(size)
+  globalThis.crypto.getRandomValues(bytes)
+  let text = ''
+  for (const b of bytes) text += String.fromCharCode(b)
+  return btoa(text)
+}
+
+function handleOf (envelope) {
+  if (envelope === null || typeof envelope !== 'object') return null
+  for (const name of ['handle', 'id']) {
+    const value = envelope[name]
+    if (typeof value === 'string' && value.length > 0) return value
+    if (typeof value === 'number') return String(value)
+  }
+  return null
+}
+
+/**
+ * @param {object} options
+ * @param {object} options.engine a loaded engine handle (`loadEngine()`), with callBlobIO
+ * @param {string} options.baseIri the absolute IRI the storage root maps to; fetch
+ *   reaches only IRIs under it
+ * @param {string} options.principal the component principal (an IRI); every request is its
+ * @param {string} [options.rootAcl] the root ACL (Turtle) of a new storage
+ * @param {string} [options.owner] the WebID the storage advertises
+ * @param {string} [options.storageKey] key text (64 hex digits or base64url); seals the snapshot
+ * @param {object} [options.snapshot] a snapshot to continue from (from onSnapshot)
+ * @param {function} [options.onSnapshot] (snapshot) => void, after each change
+ * @param {number} [options.maxBody] the largest content of one write (the engine's default otherwise)
+ * @returns {Promise<{fetch: Function, snapshot: Function, close: Function, baseIri: string}>}
+ */
+export async function openInPageLwsStorage (options = {}) {
+  const { engine } = options
+  if (!engine || typeof engine.callBlobIO !== 'function') throw new TypeError('openInPageLwsStorage: an engine with callBlobIO is required')
+  if (typeof options.baseIri !== 'string') throw new TypeError('openInPageLwsStorage: baseIri is required')
+  if (typeof options.principal !== 'string') throw new TypeError('openInPageLwsStorage: principal is required')
+  const base = new URL(options.baseIri)
+  const config = { baseIri: base.href, principal: options.principal, notifications: false, entropy: entropyBase64() }
+  for (const name of ['rootAcl', 'owner', 'storageKey']) {
+    if (typeof options[name] === 'string') config[name] = options[name]
+  }
+  if (typeof options.maxBody === 'number') config.maxBody = options.maxBody
+  if (options.snapshot !== undefined && options.snapshot !== null) config.snapshot = options.snapshot
+  const opened = engine.call('lwsOpen', [JSON.stringify(config)])
+  const handle = handleOf(opened)
+  if (handle === null) throw new Error('lwsOpen answered with no handle')
+  const onSnapshot = typeof options.onSnapshot === 'function' ? options.onSnapshot : null
+  let closed = false
+
+  const snapshot = () => {
+    const answer = engine.call('lwsSnapshot', [handle, JSON.stringify({ nonce: entropyBase64(12) })])
+    if (answer === null || typeof answer !== 'object' || answer.snapshot === undefined) {
+      throw new Error('lwsSnapshot answered with no snapshot')
+    }
+    return answer.snapshot
+  }
+
+  /** A fetch to this storage: the same arguments as `fetch`, a real Response back. */
+  async function podFetch (input, init) {
+    if (closed) throw new TypeError('this storage is closed')
+    const request = new Request(input, init)
+    const url = new URL(request.url)
+    if (url.origin !== base.origin || !url.pathname.startsWith(base.pathname)) {
+      throw new TypeError('the IRI is not in this storage')
+    }
+    const target = '/' + url.pathname.slice(base.pathname.length) + url.search
+    const content = request.method === 'GET' || request.method === 'HEAD'
+      ? new Uint8Array(0)
+      : new Uint8Array(await request.arrayBuffer())
+    const record = {
+      method: request.method,
+      target,
+      headers: [...request.headers],
+      now: Math.floor(Date.now() / 1000),
+      random: entropyBase64(16)
+    }
+    const got = engine.callBlobIO('lwsStep', [handle, JSON.stringify(record)], content)
+    const envelope = got.envelope
+    if (envelope === null || typeof envelope !== 'object') throw new Error('lwsStep answered with no envelope')
+    // A storage with a component principal and no files has nothing to
+    // fetch, resolve or read; an engine that asks is not one this host runs.
+    for (const ask of ['fetch', 'resolve', 'read']) {
+      if (envelope[ask] !== undefined && envelope[ask] !== null) throw new Error(`lwsStep asked this host to ${ask}, which an in-page storage cannot`)
+    }
+    const answer = (envelope.response !== null && typeof envelope.response === 'object') ? envelope.response : envelope
+    if (typeof answer.status !== 'number') throw new Error('lwsStep answered with no status in its response record')
+    if (envelope.changed === true && onSnapshot !== null) onSnapshot(snapshot())
+    const headers = new Headers()
+    for (const [name, value] of Array.isArray(answer.headers) ? answer.headers : []) headers.append(name, value)
+    // No content is a null body: a Response refuses an empty one with 204 or 304.
+    const body = got.bytes instanceof Uint8Array && got.bytes.length > 0 ? got.bytes : null
+    return new Response(body, { status: answer.status, headers })
+  }
+
+  return {
+    baseIri: base.href,
+    fetch: podFetch,
+    snapshot,
+    close () {
+      closed = true
+      try { engine.call('lwsClose', [handle]) } catch (_error) { /* already gone */ }
+    }
+  }
+}

package/lws/persist.mjs

@@ -64,7 +64,7 @@ export const STORAGE_MODE_VARIABLE = 'FACTOIDAL_LWS_STORAGE'
 /**
  * The text of the storage mode: `explicit` when the caller gave a string,
  * and otherwise the environment variable, passed unread; the engine checks
- * it. An empty value is none (the engine's default, the snapshot).
+ * it. An empty value is none (the host's default with a DIR, "files").
  */
 export function storageModeText (explicit) {
   if (typeof explicit === 'string') return explicit