@kubohiroya/turbowarp-extension-loader 0.3.1 → 0.4.0

package/README.ja.md

@@ -122,6 +122,56 @@ const loader = createExtensionLoader({...hostPorts, cache});
 - 新たな Remote バイト列を取得し保存した場合も、権限承認は毎回別途必要です。コード内から追加取得される依存リソースや外部サービスはこのキャッシュの対象外です。
 - Web/Editor はこのポートを初期化して注入するだけとし、`scratch-gui` と `turbowarp-plus-editor` に別のキャッシュロジックを実装しないでください。
 
+## 機能拡張のバイト列がどこから来るか
+
+このファミリーは、機能拡張に限らず何を保持するにも3つの方式で整理しています。
+
+| 方式 | ブラウザー（IndexedDB） | 作品（SB3） |
+| --- | --- | --- |
+| `remote` | 残さない | 参照だけ |
+| `cached` | 検証したバイト列を残して使い回す | 参照だけ |
+| `embedded` | 使わない | 作品が中身を持つ |
+
+`embedded` には2通りあります。**`embedded-file`**（アーカイブ内に見えるファイル）と、
+**`embedded-data`**（作品の文書に `data:` で値を埋め込む形）です。
+
+読み込み時に選べるのは `remote` と `cached` だけです。`embedded` は作者の決定で、既に
+Manifest に入っているので尋ねることがありません。残り2つは開く人が自分のブラウザーについて
+決めることなので、作品にも Manifest にも載せず、操作ごとに渡します。
+
+```js
+await loader.load(manifest, {
+    // 指定の無いものは cached。この選択肢が無かった頃と同じ挙動になります。
+    caching: new Map([['someextension', 'remote']])
+});
+```
+
+`remote` は両方向に効きます。その拡張についてキャッシュを読まず、書き込みもしません。
+何も残したくないと言った人には、何も残りません。
+
+### 作品が保持の仕方をどう書くか
+
+`source.type` は表のとおり2値です。作品が中身を持つ `embedded` と、参照だけを持つ `remote`
+です。`embedded` は取得元 URL も記録できます。
+
+```json
+{"type": "embedded", "path": "extensions/thing-<sha256>.js", "url": "https://example.test/thing.js"}
+```
+
+この URL があるから、控えが欠けたときに取得へ落ちられます。無ければ、欠けた時点で終わりです。
+
+0.4.0 より前に書かれた Manifest は、同じものを `cached` と綴ります。**拒否します。** いまは何に
+あたるかをメッセージで示します。このファミリーは、古い版が書いたものを読み続けることを約束して
+いません。黙って読み替えると、直すべき人から変更が見えなくなります。古い版で保存した作品は、
+保存し直す必要があります。
+
+`prepareCatalogExtensionChanges` の `distribution` は `embedded` か `remote` です。**ブラウザー**の
+どこに控えるかは開く人の決定で、`load()` へ渡します。どちらでも Manifest は取得元を記録します。
+カタログの拡張はどこかから取得したものなので、embedded は分かっている取得元を捨てずに控えの横へ
+残します。URL は、どちらの形で保存されていても、固定したバイト列を返すか照合します。embedded の
+url が省略可能なのは一般の場合のためで、どこからも取得していないバイト列 — コスチューム、音、
+エディター上で書いた拡張 — には記録すべき取得元がありません。
+
 ## ソースとビルド
 
 ソースもテストも TypeScript で、`src/*.ts` と `test/*.test.ts` に置き、他の `@kubohiroya/turbowarp-*` パッケージと同じ構成にしています。`tsconfig.json` が出力なしの型検査、`tsconfig.build.json` がそれを継承して出力、`test/tsconfig.json` がテスト用の継承設定です。`npm run build`（`tsc -p tsconfig.build.json`）が ES2022 の ESM と型宣言を `dist/` に出力し、`exports`・`main`/`module`・webpack 4 用のルートファイル（`packager.js` など）はすべて `dist/` を指します。`dist/` は Git に含めず、Git 依存としてインストールされるときに `prepare` スクリプトがビルドします。npm パッケージにはビルド済みの `dist/` を含めます。webpack 4 では従来どおり `./webpack4` の Babel 設定が `dist/` を変換します。

package/README.md

@@ -121,6 +121,59 @@ const loader = createExtensionLoader({...hostPorts, cache});
 - Fetching and storing new Remote bytes still requires a separate authorization every time. Dependent resources and external services fetched from inside the code are outside the scope of this cache.
 - The Web and Editor hosts should only initialize and inject this port, and should not implement separate cache logic in `scratch-gui` or `turbowarp-plus-editor`.
 
+## Where an extension's bytes come from
+
+The family settled on three ways to hold anything, extensions included.
+
+| Way | Browser (IndexedDB) | Project (SB3) |
+| --- | --- | --- |
+| `remote` | nothing kept | reference only |
+| `cached` | verified bytes kept and reused | reference only |
+| `embedded` | not used | the project carries the bytes |
+
+`embedded` has two forms: **`embedded-file`**, a file inside the archive, and **`embedded-data`**,
+a `data:` value inside the project document.
+
+Only `remote` and `cached` are a choice at load time. `embedded` is the author's decision and is
+already in the manifest, so there is nothing left to ask. The other two are the opener's decision
+about their own browser, so they travel with neither the project nor the manifest:
+
+```js
+await loader.load(manifest, {
+    // Anything absent is cached, which is what every caller got before this option existed.
+    caching: new Map([['someextension', 'remote']])
+});
+```
+
+`remote` holds in both directions: the cache is not read for that extension and not written
+either, so a person who asked to leave nothing behind leaves nothing behind.
+
+### How a project says where it keeps an extension
+
+`source.type` has two values, matching the table: `embedded` for a project that carries the
+bytes, `remote` for one that carries only a reference. An embedded source may also record the
+URL the bytes came from:
+
+```json
+{"type": "embedded", "path": "extensions/thing-<sha256>.js", "url": "https://example.test/thing.js"}
+```
+
+That URL is what makes falling back possible when a retained copy is missing. Without it, a
+missing copy is the end of it.
+
+A manifest written before 0.4.0 spells that same thing `cached`. **It is refused**, with a
+message saying what it is now. This family does not promise to read what older versions wrote,
+and translating it silently would hide the change from whoever has to make it. A project saved
+by an older version has to be written again.
+
+`prepareCatalogExtensionChanges` takes `distribution: 'embedded' | 'remote'`, because where
+bytes are cached **in the browser** is the opener's decision and goes to `load()` instead.
+Either way the manifest records where the bytes came from: a catalog extension was fetched from
+an address, so embedding keeps that address beside the retained copy rather than discarding what
+is already known, and the URL is checked against the pinned bytes whichever form it is stored in.
+A url is optional on an embedded source for the general case, where bytes nobody fetched — a
+costume, a sound, an extension written in the editor — have no address to record.
+
 ## Source and build
 
 The source and the tests are TypeScript, in `src/*.ts` and `test/*.test.ts`, laid out the same way as the other `@kubohiroya/turbowarp-*` packages: `tsconfig.json` typechecks without emitting, `tsconfig.build.json` extends it to emit, and `test/tsconfig.json` extends it for the tests. `npm run build` (`tsc -p tsconfig.build.json`) emits ES2022 ESM and type declarations to `dist/`, and `exports`, `main`/`module` and the root files used by webpack 4 (`packager.js` and the rest) all point at `dist/`. `dist/` is not committed to Git; the `prepare` script builds it when the package is installed as a Git dependency, and the npm package ships the built `dist/`. Under webpack 4, the Babel configuration in `./webpack4` transpiles `dist/` as before.

package/dist/index.d.ts

@@ -6,16 +6,25 @@
  * weaker source — the Manifest pins the SHA-256, so a retained copy either is exactly the pinned
  * code or the load fails, which is why it may be read before the network.
  */
+/**
+ * Where a project keeps an extension.
+ *
+ * `embedded` is the project carrying the bytes; `remote` is the project carrying only a
+ * reference. An embedded source may also record the URL the bytes came from, which is what makes
+ * falling back possible when a retained copy is missing — and is the only difference between the
+ * two forms this used to spell as separate types.
+ *
+ * A manifest written before 0.4.0 spells that same embedded source `cached`. It is refused, not
+ * translated: this family does not promise to read what older versions wrote, and a refusal
+ * names the problem where a silent translation would hide it.
+ */
 export type ExtensionSource = Readonly<{
     type: 'embedded';
     path: string;
+    url?: string;
 } | {
     type: 'remote';
     url: string;
-} | {
-    type: 'cached';
-    url: string;
-    path: string;
 }>;
 export type ExecutionMode = 'sandboxed' | 'unsandboxed';
 export interface ExtensionSpec {
@@ -50,8 +59,33 @@ export interface AuthorizedCode extends ResolvedCode {
         sha256: string;
     }>;
 }
+/**
+ * Where one extension's bytes come from while the project runs.
+ *
+ * The family settled on three ways to hold anything, extensions included. `remote` fetches every
+ * time: nothing is kept in the browser and nothing is kept in the project. `cached` keeps
+ * verified bytes in the browser and reuses them, so opening again costs no network while the
+ * project still holds only a reference. `embedded` is the project carrying the bytes itself, in
+ * either of its two forms — `embedded-file`, a file inside the archive, and `embedded-data`, a
+ * `data:` value inside the project document.
+ *
+ * Only the first two are a choice here. `embedded` is the author's decision and is already in
+ * the manifest, so there is nothing to ask at load time. These two are the opener's decision,
+ * about this browser, so they travel with neither the project nor the manifest and are passed in
+ * for each operation.
+ *
+ * Note that `source.type` has a value also spelled `cached`, for a project that carries bytes
+ * and records where they came from. By the vocabulary above, a project that carries bytes is
+ * `embedded`, so that name is the one out of step; see the README.
+ */
+export type CachePolicy = 'cached' | 'remote';
 export interface OperationOptions {
     signal?: AbortSignal;
+    /**
+     * Per extension ID, where its bytes come from. Anything absent is `cached`, which is what
+     * every caller got before this existed.
+     */
+    caching?: ReadonlyMap<string, CachePolicy>;
 }
 export interface ReadOptions {
     signal: AbortSignal;

package/dist/index.js

@@ -56,14 +56,16 @@ function sourceOf(source) {
         return Object.freeze({ type: 'remote', url: remoteURL(string(source.url, 'URL', 4096)) });
     }
     if (source.type === 'embedded') {
-        keys(source, ['type', 'path'], 'source');
-        return Object.freeze({ type: 'embedded', path: embeddedPath(source.path) });
-    }
-    if (source.type === 'cached') {
-        keys(source, ['type', 'url', 'path'], 'source');
-        // The same HTTPS and path policies as the two single-location forms it combines.
-        return Object.freeze({ type: 'cached', url: remoteURL(string(source.url, 'URL', 4096)), path: embeddedPath(source.path) });
+        keys(source, ['type', 'path', 'url'], 'source');
+        const path = embeddedPath(source.path);
+        if (source.url === undefined)
+            return Object.freeze({ type: 'embedded', path });
+        return Object.freeze({ type: 'embedded', path, url: remoteURL(string(source.url, 'URL', 4096)) });
     }
+    // Name the one spelling that used to mean an embedded source, so a project written by an
+    // older version fails with something a reader can act on rather than "unsupported".
+    if (source.type === 'cached')
+        fail('INVALID_SOURCE', 'A cached source is now an embedded source with a url');
     fail('INVALID_SOURCE', 'Unsupported source type');
 }
 /** Validate and snapshot untrusted data; execution is only a request. */
@@ -320,10 +322,13 @@ export function createExtensionLoader(ports, options = {}) {
         ports.diagnostics?.(Object.freeze({ ...event }));
     }
     catch { /* Diagnostics must not alter authorization. */ } };
-    async function resolve(spec, signal, budget) {
+    async function resolve(spec, signal, budget, caching) {
         let bytes;
         let cached = false;
-        if (ports.cache) {
+        // Remote is the person's decision to leave nothing behind, so it holds in both
+        // directions: the cache is not read for this extension, and not written either.
+        const mayCache = caching?.get(spec.id) !== 'remote';
+        if (mayCache && ports.cache) {
             const entry = await guarded(() => ports.cache.get(spec.sha256, { signal }), signal);
             if (entry !== undefined && entry !== null) {
                 const candidate = snapshotBytes(entry, limits.maxBytes, spec.id);
@@ -336,24 +341,26 @@ export function createExtensionLoader(ports, options = {}) {
         if (!bytes) {
             // The manifest is frozen, so the narrowed source stays valid inside the port callbacks.
             const source = spec.source;
-            if (source.type !== 'remote') {
+            if (source.type === 'embedded') {
                 if (typeof ports.embeddedReader?.read !== 'function')
                     fail('INVALID_PORT', 'Embedded reader is unavailable', spec.id);
-                // A cached source may have lost its retained copy; its URL remains the canonical one.
                 try {
                     bytes = snapshotBytes(await guarded(() => ports.embeddedReader.read(source.path, { signal, maxBytes: limits.maxBytes }), signal), limits.maxBytes, spec.id);
                 }
                 catch (cause) {
-                    if (source.type === 'embedded' || signal.aborted)
+                    // Falling back is only possible where the project recorded where the bytes
+                    // came from; without that, a missing retained copy is the end of it.
+                    if (source.url === undefined || signal.aborted)
                         throw cause;
                     report({ type: 'retained-copy-unavailable', id: spec.id });
                 }
             }
-            if (!bytes && source.type !== 'embedded') {
+            const url = source.url;
+            if (!bytes && url !== undefined) {
                 if (typeof ports.fetcher?.fetch !== 'function')
                     fail('INVALID_PORT', 'Remote fetch port is unavailable', spec.id);
-                const response = await guarded(() => ports.fetcher.fetch(source.url, { signal, maxBytes: limits.maxBytes }), signal);
-                if (!response || !Array.isArray(response.redirects) || response.redirects.length || response.finalURL !== source.url)
+                const response = await guarded(() => ports.fetcher.fetch(url, { signal, maxBytes: limits.maxBytes }), signal);
+                if (!response || !Array.isArray(response.redirects) || response.redirects.length || response.finalURL !== url)
                     fail('REDIRECT_DENIED', 'Unproven or redirected remote source', spec.id);
                 bytes = snapshotBytes(response.bytes, limits.maxBytes, spec.id);
             }
@@ -373,13 +380,13 @@ export function createExtensionLoader(ports, options = {}) {
         catch (cause) {
             throw new LoaderError('INVALID_ENCODING', 'Extension must contain valid UTF-8', spec.id, cause);
         }
-        if (!cached && ports.cache)
+        if (!cached && mayCache && ports.cache)
             await guarded(() => ports.cache.put(spec.sha256, new Uint8Array(bytes), { signal }), signal);
         const result = Object.freeze({ ...spec, code, byteLength: bytes.byteLength, actualSHA256: actual, cached });
         report({ type: 'verified', id: spec.id, sha256: actual, cached });
         return result;
     }
-    async function load(input, { signal: externalSignal } = {}) {
+    async function load(input, { signal: externalSignal, caching } = {}) {
         if (state !== 'idle')
             fail('BUSY', 'Unload the previous project before another load');
         const manifest = validateManifest(input, limits);
@@ -397,7 +404,7 @@ export function createExtensionLoader(ports, options = {}) {
         try {
             const budget = { bytes: 0 };
             for (const spec of manifest.extensions) {
-                const code = await resolve(spec, signal, budget);
+                const code = await resolve(spec, signal, budget, caching);
                 const decision = await guarded(() => ports.trustProvider.authorize(code, { signal }), signal);
                 if (decision?.allowed !== true || decision.scope !== 'digest' || decision.sha256 !== code.sha256)
                     fail('TRUST_DENIED', 'Host did not approve this exact extension digest', spec.id);

package/dist/project-catalog.d.ts

@@ -3,7 +3,14 @@ import { type ArchiveLimits, type ZipConstructor } from './project-codec.js';
 export interface CatalogExtensionRequest {
     id: string;
     url: string;
-    distribution: 'embedded' | 'remote' | 'cached';
+    /**
+     * Where the project keeps it: carrying the bytes, or only a reference.
+     *
+     * Either way the manifest records where the bytes came from, so a project that carries them
+     * can still recover if its retained copy is lost. There is deliberately no third value:
+     * where bytes are cached in the browser is the opener's decision, passed to `load()`.
+     */
+    distribution: 'embedded' | 'remote';
     execution: ExecutionMode;
     dependencies?: string[];
     version?: string;

package/dist/project-catalog.js

@@ -63,7 +63,7 @@ export async function prepareCatalogExtensionChanges(input, requests, ports, { z
     try {
         const seen = new Set();
         const choices = requests.map(request => {
-            if (!request || !['remote', 'embedded', 'cached'].includes(request.distribution) || seen.has(request.id)) {
+            if (!request || !['remote', 'embedded'].includes(request.distribution) || seen.has(request.id)) {
                 fail('INVALID_ADDITION', 'Invalid or duplicate catalog selection', request?.id);
             }
             seen.add(request.id);
@@ -127,17 +127,22 @@ export async function prepareCatalogExtensionChanges(input, requests, ports, { z
             if (old && old.sha256 !== sha256)
                 fail('CONVERSION_INTEGRITY', 'Source conversion changed bytes', choice.id);
             const retainedPath = `extensions/${choice.id}-${sha256}.js`;
+            // A catalog extension was fetched from an address, so embedding keeps that address
+            // beside the copy rather than discarding what is already known. Bytes nobody fetched —
+            // a costume, a sound, an extension written here — have no address to record, which is
+            // why a url is optional on an embedded source.
             const source = choice.distribution === 'embedded' ?
-                { type: 'embedded', path: retainedPath } : choice.distribution === 'cached' ?
-                { type: 'cached', url: choice.source.url, path: retainedPath } : choice.source;
-            // A URL stays canonical for remote and cached, so it must still serve the pinned bytes.
-            if (source.type !== 'embedded') {
+                { type: 'embedded', path: retainedPath, url: choice.source.url } :
+                choice.source;
+            // A recorded URL is a promise about what it serves, so it must still serve exactly
+            // the pinned bytes — whether it is the only place they live or sits beside a copy.
+            if (source.url !== undefined) {
                 const remoteBytes = old ? await fetchBytes(source.url) : bytes;
                 if (await wait(() => digest(remoteBytes)) !== sha256) {
                     fail('DIGEST_MISMATCH', 'Remote source does not serve the currently pinned bytes', choice.id);
                 }
             }
-            // Embedded and cached both retain a copy in the project.
+            // An embedded choice retains a copy in the project.
             if (source.type !== 'remote') {
                 if (Object.hasOwn(files, source.path) && await wait(() => digest(files[source.path])) !== sha256) {
                     fail('SOURCE_PATH_COLLISION', 'The embedded path belongs to different content', choice.id);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@kubohiroya/turbowarp-extension-loader",
-  "version": "0.3.1",
+  "version": "0.4.0",
   "description": "Host-independent verified TurboWarp Plus extension loading core",
   "keywords": [
     "turbowarp",