@deepseek-ai/dsh-experimental-webworker-packer 0.2.1-alpha.3 → 0.2.1-alpha.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.i18n.yaml +4 -4
- package/README.md +5 -4
- package/README.zh.md +5 -4
- package/lib/bin.js +10 -11
- package/lib/index.js +2 -2
- package/lib/{repository-Bszxcf0w.js → repository-DhiU_Xz2.js} +222 -89
- package/lib/types/bin.js +11 -12
- package/lib/types/index.d.ts +2 -2
- package/lib/types/pack.d.ts +49 -10
- package/lib/types/repository.d.ts +41 -10
- package/lib/types/rules.d.ts +1 -1
- package/package.json +6 -6
package/README.i18n.yaml
CHANGED
|
@@ -9,14 +9,14 @@
|
|
|
9
9
|
en: 7ecd0d6282f6ccf1
|
|
10
10
|
zh: 782c000a57a5e3d8
|
|
11
11
|
/deepseek-ai-dsh-experimental-webworker-packer/summary:
|
|
12
|
-
en:
|
|
13
|
-
zh:
|
|
12
|
+
en: 511b69c241632e21
|
|
13
|
+
zh: a59afd79170fcae7
|
|
14
14
|
/deepseek-ai-dsh-experimental-webworker-packer/table-of-contents:
|
|
15
15
|
en: 4e53164159b5ce41
|
|
16
16
|
zh: 89582dc0294d0776
|
|
17
17
|
/deepseek-ai-dsh-experimental-webworker-packer/use-this-package:
|
|
18
|
-
en:
|
|
19
|
-
zh:
|
|
18
|
+
en: 4b6067a6ffd4a33f
|
|
19
|
+
zh: 8790e1a070a7a0f3
|
|
20
20
|
/deepseek-ai-dsh-experimental-webworker-packer/model-experience:
|
|
21
21
|
en: e16b355530dadc1c
|
|
22
22
|
zh: 8488e03b253b7b33
|
package/README.md
CHANGED
|
@@ -9,7 +9,7 @@ English | [中文](README.zh.md)
|
|
|
9
9
|
|
|
10
10
|
## Summary
|
|
11
11
|
|
|
12
|
-
The VFS image packer: turns one
|
|
12
|
+
The VFS image packer: turns one shipped profile's bundles and the installation's optional bundles into the gzip-compressed base tar the browser worker mounts as its filesystem, one archive per lazily loaded page chunk, and opaque data trees into ordered overlay tars ([experimental group](../README.md)). Nothing is compiled from source — the base image carries the repository's real build products, so a preview deployment debugs exactly what the served deployment ships. Read this page when packaging a preview image or diagnosing its contents.
|
|
13
13
|
|
|
14
14
|
## Table of Contents
|
|
15
15
|
|
|
@@ -23,17 +23,18 @@ The VFS image packer: turns one composed profile into the gzip-compressed base t
|
|
|
23
23
|
<a id="use-this-package"></a>
|
|
24
24
|
## Use this package
|
|
25
25
|
|
|
26
|
-
The public npm package exposes `packVfsImage` and `packVfsOverlay` through its library entry. Callers supply the
|
|
26
|
+
The public npm package exposes `packVfsImage` and `packVfsOverlay` through its library entry. Callers supply the profile name, the package roster, the installation manifest, package directories, and the resolution root; no profile layer is installed and no composed configuration is packed — the worker composes the bundle layers at boot. An optional `homePatch` is written to `home/cordis.patch.yml`, the home layer the image's app-boot reads. The runtime's library entry supplies the module-proxy and replacement tables used during packing.
|
|
27
27
|
|
|
28
28
|
The packer owns the internal `dsh.configTrees` declaration, its validation, and source directory resolution in [`src/repository.ts`](src/repository.ts). This field is not part of the public plugin manifest API.
|
|
29
29
|
|
|
30
30
|
The pack is a three-layer standard stack:
|
|
31
31
|
|
|
32
|
-
1. **Roster** — the
|
|
32
|
+
1. **Roster** — the shipped profile template's bundles and every bundle in app-boot's `OPTIONAL_BUNDLES`, each resolved from the CLI manifest as the launcher resolves it, plus every package their patch files name (insert lists and grouped rows included), plus the rows of every config tree the CLI declares in its `package.json` `dsh.configTrees`, materialized as a Node-style dependency closure. External peer edges never bind the worker; workspace peers stay on the chain. The CLI manifest itself enters the image alone at `node_modules/@deepseek-ai/dsh/package.json`, as the worker's bundle resolution anchor; the manifest records its name and the profile.
|
|
33
33
|
2. **Publish view** — each workspace or vendored package contributes its built npm slice (`files` through picomatch) without source or workspace `dist/`. External packages retain published JavaScript under both `src/` and `dist/` because their `main` or `exports` may point there; only generic test, map, declaration, and archive exclusions apply.
|
|
34
34
|
3. **Reachability sweep** — the runtime loader's own resolution walks from every non-wildcard workspace runtime export plus the worker assembly's seeds (`IMAGE_ENTRY_SEEDS`), lowering each reached module to the wrapper contract at pack time. Declaration-only condition trees do not seed the sweep; runtime requests for those exports still fail. The transform reports statically named imports, re-exports, and dynamic imports; calls through `require`; and module-scope direct calls of the form `createRequire(import.meta.url)('pkg')` through a named import from `node:module` or `module`, including an import alias. Page assets (`lib/client.js` behind `./client` exports) ship verbatim; an unresolvable request from our own code fails the pack, third-party ones are tolerated to fail loud at require time.
|
|
35
|
+
4. **Lazy chunks** — every page chunk a page asset requests as a literal `require.async("./client.<name>.js")`, the only form the dynamic Client build emits, leaves the base image unless a `./client` export names it. Each becomes its own deterministic archive named `lazy/<sha256 of the archive>.tar.gz` beside the base image, and the manifest registers its path, size, and archive for the worker's lazy filesystem. A request for a chunk the package does not publish fails the pack.
|
|
35
36
|
|
|
36
|
-
`repository.ts` owns the repo-shaped inputs (workspace scan of `vendor/`, `packages/`, `native/system/packages/`, and `apps/`; profile
|
|
37
|
+
`repository.ts` owns the repo-shaped inputs (workspace scan of `vendor/`, `packages/`, `native/system/packages/`, and `apps/`; the profile roster through app-boot's shipped templates, optional-bundle list, and bundle patch readers; the CLI manifest as the installation; the home patch, which sets the runtime's `PLATFORM_ENTRY_CONFIG` fields over each named row's config composed from the template's bundle layers); `pack.ts` owns none of them, so the same library packs a different tree by being called differently. The native scan makes the Landlock entry package an ordinary published-view dependency while its executable remains a Worker platform implementation. The CLI is `dsh-pack-vfs-image --out <file> [--profile web]`; it writes the lazy archives beside the base image, and `apps/web`'s `build:preview` runs it after the preview shell build.
|
|
37
38
|
|
|
38
39
|
The repository adapter also declares the preview-only fixture trees under `webworker-runtime/tests/fixtures/`. The CLI packs each named fixture into a separate deterministic overlay archive plus a browser-readable manifest. Overlay files bypass npm publish-view and module-reachability exclusions, so dot directories and example source files remain intact; their mounts are limited to `home/` and `workspace/`. `packVfsOverlay` treats all files as opaque bytes.
|
|
39
40
|
|
package/README.zh.md
CHANGED
|
@@ -9,7 +9,7 @@ kind: "package-library"
|
|
|
9
9
|
|
|
10
10
|
## 概述
|
|
11
11
|
|
|
12
|
-
VFS
|
|
12
|
+
VFS 镜像打包器:把一份随附 profile 的 bundle 和安装提供的 optional bundle 打成浏览器 worker 挂载为文件系统的 gzip 压缩基础 tar,把每个懒加载的页面 chunk 各打成一个归档,并把不透明数据目录变成按序应用的 overlay tar([实验组](../README.zh.md))。不做任何源码编译——基础镜像携带仓库真实构建产物,预览部署调试的正是 served 部署交付的字节。打包预览镜像或排查镜像内容时,请阅读本页。
|
|
13
13
|
|
|
14
14
|
## 目录
|
|
15
15
|
|
|
@@ -23,17 +23,18 @@ VFS 镜像打包器:把一份合成 profile 变成浏览器 worker 挂载为
|
|
|
23
23
|
<a id="use-this-package"></a>
|
|
24
24
|
## 使用本包
|
|
25
25
|
|
|
26
|
-
公开 npm 包通过库入口导出 `packVfsImage` 和 `packVfsOverlay
|
|
26
|
+
公开 npm 包通过库入口导出 `packVfsImage` 和 `packVfsOverlay`。调用方提供 profile 名称、包 roster、安装 manifest、包目录和解析根目录;本包不安装 profile 层,也不打包合成配置——worker 在启动时合成各 bundle 层。可选的 `homePatch` 写入 `home/cordis.patch.yml`,即镜像中 app-boot 读取的 home 层。运行时的库入口提供打包所用的模块代理表与替换表。
|
|
27
27
|
|
|
28
28
|
打包器在 [`src/repository.ts`](src/repository.ts) 中拥有内部 `dsh.configTrees` 声明、校验和源目录解析。该字段不属于公共插件 manifest API。
|
|
29
29
|
|
|
30
30
|
打包是三层标准栈:
|
|
31
31
|
|
|
32
|
-
1. **Roster
|
|
32
|
+
1. **Roster**——随附 profile 模板的 bundle 和 app-boot `OPTIONAL_BUNDLES` 中的每个 bundle(各自按启动器的方式从 CLI manifest 解析),加上它们 patch 文件点名的每个包(含 insert 列表与分组行),再加上 CLI(命令行界面)在 `package.json` `dsh.configTrees` 里声明的每棵配置树的行,按 Node 式依赖闭包物化。外部 peer 边绝不会绑定到 worker,workspace peer 保留在链上。CLI manifest 本身单独进入镜像,位于 `node_modules/@deepseek-ai/dsh/package.json`,作为 worker 解析 bundle 的锚点;manifest 记录它的名称和 profile。
|
|
33
33
|
2. **发布视图**——每个 workspace 或 vendored 包贡献其构建后的 npm 切片(`files` 走 picomatch),不带源码和 workspace `dist/`。外部包的 `main` 或 `exports` 可能指向 `src/` 或 `dist/`,因此两处发布 JavaScript 都会保留,只应用通用的测试、map、声明与归档排除规则。
|
|
34
34
|
3. **可达性 sweep**——用运行时加载器自己的解析, 从全部不含通配符的 workspace 运行时导出加 worker 装配种子 (`IMAGE_ENTRY_SEEDS`) 出发, 打包时把每个可达模块转换为符合包装层约定的形式. 仅含类型声明的条件树不作为扫描入口; 运行时代码请求这些导出仍会失败. 该转换会报告名称可静态确定的 import、re-export 与动态 import、经 `require` 发起的调用, 以及通过 `node:module` 或 `module` 具名导入 (含导入别名) 在模块作用域直接发起的 `createRequire(import.meta.url)('pkg')` 调用. 页面资产 (`./client` 导出背后的 `lib/client.js`) 原样直发; 自家代码的不可解析请求会让打包失败, 第三方不可解析请求则允许延后到 require 时明确失败.
|
|
35
|
+
4. **懒加载 chunk**——页面资产以字面量 `require.async("./client.<name>.js")`(动态 Client 构建产出的唯一形式)请求的每个页面 chunk,除非被 `./client` 导出点名,否则移出基础镜像。每个 chunk 成为基础镜像旁的一个独立确定性归档,命名为 `lazy/<归档的 sha256>.tar.gz`;manifest 为 worker 的懒加载文件系统登记其路径、大小和归档。请求包未发布的 chunk 会让打包失败。
|
|
35
36
|
|
|
36
|
-
`repository.ts` 拥有仓库形态输入(`vendor/`、`packages/`、`native/system/packages/` 与 `apps/` 的 workspace
|
|
37
|
+
`repository.ts` 拥有仓库形态输入(`vendor/`、`packages/`、`native/system/packages/` 与 `apps/` 的 workspace 扫描;经 app-boot 的随附模板、optional bundle 列表和 bundle patch 读取器得出 profile roster;以 CLI manifest 作为安装清单;home patch 在模板各 bundle 层合成出的行配置之上设置运行时 `PLATFORM_ENTRY_CONFIG` 字段);`pack.ts` 一概不拥有,同一库换参即可打另一棵树。Native 扫描使 Landlock 入口包成为普通发布视图依赖,其可执行文件仍由 Worker 平台实现。CLI 为 `dsh-pack-vfs-image --out <file> [--profile web]`;它把懒加载归档写在基础镜像旁,`apps/web` 的 `build:preview` 在预览壳构建后运行它。
|
|
37
38
|
|
|
38
39
|
仓库适配器还声明 `webworker-runtime/tests/fixtures/` 下仅用于 preview 的 fixture(测试前置数据) tree。CLI 会把每套具名 fixture 打成一份独立的确定性 overlay 归档,并写出浏览器可读的 manifest(元数据清单)。Overlay 文件绕过 npm 发布视图和模块可达性排除规则,因此点目录与示例源码会完整保留;其挂载位置仅限 `home/` 与 `workspace/`。`packVfsOverlay` 将所有文件视为不透明字节。
|
|
39
40
|
|
package/lib/bin.js
CHANGED
|
@@ -1,13 +1,14 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import { a as
|
|
2
|
+
import { a as packRepositoryImage, c as packPreviewFixture, n as describePack, o as previewFixtures } from "./repository-DhiU_Xz2.js";
|
|
3
3
|
import { PREVIEW_FIXTURE_MANIFEST_FILE, PREVIEW_FIXTURE_MANIFEST_VERSION } from "@deepseek-ai/dsh-experimental-webworker-runtime";
|
|
4
4
|
import { mkdirSync, writeFileSync } from "node:fs";
|
|
5
5
|
import { dirname, isAbsolute, join, resolve } from "node:path";
|
|
6
6
|
import { fileURLToPath } from "node:url";
|
|
7
7
|
//#region lib/types/bin.js
|
|
8
8
|
/**
|
|
9
|
-
* Pack a Preview deployment from this repository:
|
|
10
|
-
* image,
|
|
9
|
+
* Pack a Preview deployment from this repository: materialize and lower the
|
|
10
|
+
* base image for one shipped profile, write its lazy chunk archives beside it,
|
|
11
|
+
* then write each named fixture overlay and their manifest.
|
|
11
12
|
*
|
|
12
13
|
* Usage: dsh-pack-vfs-image --out <file> [--profile web] [--root /dsh]
|
|
13
14
|
* node --import tsx/esm src/bin.ts --out ../../apps/web/dist/preview/vfs-image.tar.gz
|
|
@@ -35,17 +36,15 @@ const repoRoot = fileURLToPath(new URL("../../../../", import.meta.url));
|
|
|
35
36
|
const profile = flag("profile", "web");
|
|
36
37
|
const out = flag("out");
|
|
37
38
|
const outputFile = isAbsolute(out) ? out : resolve(process.cwd(), out);
|
|
38
|
-
const result =
|
|
39
|
-
config: composeProfile(repoRoot, profile),
|
|
40
|
-
profile,
|
|
41
|
-
root: flag("root", "/dsh"),
|
|
42
|
-
workspaces: indexWorkspacePackages(repoRoot),
|
|
43
|
-
resolveFrom: repoRoot,
|
|
44
|
-
configTrees: configTrees(repoRoot)
|
|
45
|
-
});
|
|
39
|
+
const result = packRepositoryImage(repoRoot, profile, flag("root", "/dsh"));
|
|
46
40
|
if (result.missing.length > 0) throw new Error(`vfs image: ${String(result.missing.length)} dependencies did not resolve; the image would be incomplete`);
|
|
47
41
|
mkdirSync(dirname(outputFile), { recursive: true });
|
|
48
42
|
writeFileSync(outputFile, result.image);
|
|
43
|
+
for (const archive of result.lazyArchives) {
|
|
44
|
+
const path = join(dirname(outputFile), archive.name);
|
|
45
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
46
|
+
writeFileSync(path, archive.image);
|
|
47
|
+
}
|
|
49
48
|
const fixtureDefinitions = previewFixtures(repoRoot);
|
|
50
49
|
mkdirSync(join(dirname(outputFile), "fixtures"), { recursive: true });
|
|
51
50
|
const fixtureLines = [];
|
package/lib/index.js
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
|
-
import { a as
|
|
2
|
-
export {
|
|
1
|
+
import { a as packRepositoryImage, c as packPreviewFixture, d as collectModuleNames, f as packVfsImage, i as installationManifest, l as DEFAULT_ROOT, m as WRAPPER_CONTRACT, n as describePack, o as previewFixtures, p as packVfsOverlay, r as indexWorkspacePackages, s as profileRoster, t as configTrees, u as MANIFEST_PATH } from "./repository-DhiU_Xz2.js";
|
|
2
|
+
export { DEFAULT_ROOT, MANIFEST_PATH, WRAPPER_CONTRACT, collectModuleNames, configTrees, describePack, indexWorkspacePackages, installationManifest, packPreviewFixture, packRepositoryImage, packVfsImage, packVfsOverlay, previewFixtures, profileRoster };
|
|
@@ -1,13 +1,13 @@
|
|
|
1
|
-
import { DEFAULT_ROOT, DEFAULT_ROOT as DEFAULT_ROOT$1,
|
|
1
|
+
import { DEFAULT_ROOT, DEFAULT_ROOT as DEFAULT_ROOT$1, IMAGE_EMPTY_DIRECTORIES, IMAGE_HOME_DIRECTORY, IMAGE_LAZY_DIRECTORY, IMAGE_MANIFEST_PATH, IMAGE_OVERLAY_DIRECTORIES, LOWERING_VERSION, MODULE_PROXIES, MODULE_PROXY_PREFIXES, MemoryVfs, PLATFORM_ENTRY_CONFIG, REPLACED_EXTERNAL_PACKAGES, WorkerModuleLoader, lowerModuleSource, packTar } from "@deepseek-ai/dsh-experimental-webworker-runtime";
|
|
2
|
+
import { createHash } from "node:crypto";
|
|
2
3
|
import { existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, realpathSync, rmSync, writeFileSync } from "node:fs";
|
|
3
4
|
import { dirname, join, posix, relative } from "node:path";
|
|
4
5
|
import { gzipSync } from "node:zlib";
|
|
5
6
|
import picomatch from "picomatch";
|
|
6
7
|
import yaml from "js-yaml";
|
|
7
8
|
import { entryListSchema } from "@deepseek-ai/cordis-plugin-include";
|
|
8
|
-
import {
|
|
9
|
+
import { OPTIONAL_BUNDLES, PROFILE_PATCH_FILENAME, PROFILE_TEMPLATES, bundlePatchPaths, composeEntries, loadOverlayPatches, resolveBundleDir } from "@deepseek-ai/dsh-app-boot";
|
|
9
10
|
import { tmpdir } from "node:os";
|
|
10
|
-
import { DSH_HOME_ENV } from "@deepseek-ai/dsh-home-paths";
|
|
11
11
|
import { isSessionFormatJsonObject, parseSessionFormatLogFilename, sessionFormatLogFilename } from "@deepseek-ai/dsh-session-format";
|
|
12
12
|
import { createSessionFormatCatalogWithChildren, historicalSessionFormatCatalog, sessionFormatCatalog } from "@deepseek-ai/dsh-session-format-catalog";
|
|
13
13
|
import { historicalChildCatalogSource } from "@deepseek-ai/dsh-session-format-v3-to-v4";
|
|
@@ -80,7 +80,7 @@ const PAGE_ASSETS = [
|
|
|
80
80
|
"node_modules/@*/*/lib/client.*.js"
|
|
81
81
|
];
|
|
82
82
|
/**
|
|
83
|
-
* Image specifiers the worker assembly requires directly, beyond the
|
|
83
|
+
* Image specifiers the worker assembly requires directly, beyond the
|
|
84
84
|
* roster: they are requested by worker-bundle code, so no image file
|
|
85
85
|
* references them and the reachability sweep must seed them as roots. Keep in
|
|
86
86
|
* step with the literal `require`/`resolve` calls in the runtime's
|
|
@@ -89,36 +89,32 @@ const PAGE_ASSETS = [
|
|
|
89
89
|
const IMAGE_ENTRY_SEEDS = [
|
|
90
90
|
"@deepseek-ai/dsh-app-boot",
|
|
91
91
|
"@deepseek-ai/dsh-cmdline",
|
|
92
|
-
"@deepseek-ai/
|
|
93
|
-
"@deepseek-ai/cordis
|
|
94
|
-
"js-yaml"
|
|
92
|
+
"@deepseek-ai/dsh-launch-environment",
|
|
93
|
+
"@deepseek-ai/cordis"
|
|
95
94
|
];
|
|
96
95
|
//#endregion
|
|
97
96
|
//#region lib/types/pack.js
|
|
98
97
|
/**
|
|
99
|
-
* VFS image packer: turns
|
|
100
|
-
* gzip-compressed tar the browser runtime inflates and mounts as its filesystem
|
|
98
|
+
* VFS image packer: turns a package roster plus a package index into the single
|
|
99
|
+
* gzip-compressed tar the browser runtime inflates and mounts as its filesystem,
|
|
100
|
+
* and the lazy archives beside it.
|
|
101
101
|
*
|
|
102
102
|
* Nothing is compiled here. The image carries the repository's real build products,
|
|
103
103
|
* so a preview deployment debugs exactly what the served deployment ships. What the
|
|
104
104
|
* pass does add is the pack-time module transform and the manifest that records the
|
|
105
|
-
* wrapper contract it was transformed against.
|
|
105
|
+
* wrapper contract it was transformed against. No composed configuration is packed:
|
|
106
|
+
* the worker host composes the profile's bundle layers from the packages the roster
|
|
107
|
+
* materializes, as the served launcher does.
|
|
106
108
|
*
|
|
107
|
-
* This module holds no repository knowledge: paths, globs, and the
|
|
108
|
-
*
|
|
109
|
+
* This module holds no repository knowledge: paths, globs, and the roster come in as
|
|
110
|
+
* parameters, so the same library packs a different tree by being called
|
|
109
111
|
* differently. Locating those inputs is the CLI's job.
|
|
110
112
|
* @module @deepseek-ai/dsh-experimental-webworker-packer/src/pack
|
|
111
113
|
*/
|
|
112
114
|
/** Image path of the manifest; the layout contract's name, re-exported for callers. */
|
|
113
115
|
const MANIFEST_PATH = IMAGE_MANIFEST_PATH;
|
|
114
|
-
/** Image path of the
|
|
115
|
-
const
|
|
116
|
-
/**
|
|
117
|
-
* Manifest field the runtime judges the image by: the wrapper contract every packed
|
|
118
|
-
* body was emitted against. The runtime refuses an image whose value is not its own
|
|
119
|
-
* contract, because those bodies assume different wrapper semantics.
|
|
120
|
-
*/
|
|
121
|
-
const CONTRACT_FIELD = "lowered";
|
|
116
|
+
/** Image path of the home patch: `$DSH_HOME/cordis.patch.yml` under the Worker home. */
|
|
117
|
+
const HOME_PATCH_PATH = `${IMAGE_HOME_DIRECTORY}/${PROFILE_PATCH_FILENAME}`;
|
|
122
118
|
/** Exclude matcher over tree-root-relative paths ({@link EXCLUDE}). */
|
|
123
119
|
const excluded = picomatch([...EXCLUDE], { dot: true });
|
|
124
120
|
/** Workspace exclude matcher: {@link EXCLUDE} plus {@link EXCLUDE_WORKSPACE}. */
|
|
@@ -136,33 +132,25 @@ function packageNameOf(specifier) {
|
|
|
136
132
|
return first.startsWith("@") ? `${first}/${second}` : first;
|
|
137
133
|
}
|
|
138
134
|
/**
|
|
139
|
-
* Collect
|
|
140
|
-
*
|
|
141
|
-
*
|
|
142
|
-
*
|
|
143
|
-
*
|
|
135
|
+
* Collect package names from entry rows and patch lists, recursively through
|
|
136
|
+
* `insert` lists and nested `config` row lists (groups). Builtin rows
|
|
137
|
+
* (`cordis:group`), preset metadata documents, and file-relative entries carry
|
|
138
|
+
* names that are not package specifiers; only scoped names and bare names
|
|
139
|
+
* with a subpath count.
|
|
140
|
+
* @param rows - Parsed YAML value; anything but a row array is ignored.
|
|
144
141
|
* @param names - Package names collected so far.
|
|
145
142
|
*/
|
|
146
|
-
function
|
|
143
|
+
function collectModuleNames(rows, names) {
|
|
147
144
|
if (!Array.isArray(rows)) return;
|
|
148
145
|
for (const row of rows) {
|
|
149
146
|
if (typeof row !== "object" || row === null) continue;
|
|
150
|
-
const { name, config } = row;
|
|
151
|
-
if (typeof name === "string" && (name.startsWith("@") || name.includes("/"))) names.add(packageNameOf(name));
|
|
152
|
-
|
|
147
|
+
const { name, config, insert } = row;
|
|
148
|
+
if (typeof name === "string" && !name.startsWith(".") && (name.startsWith("@") || name.includes("/"))) names.add(packageNameOf(name));
|
|
149
|
+
collectModuleNames(config, names);
|
|
150
|
+
collectModuleNames(insert, names);
|
|
153
151
|
}
|
|
154
152
|
}
|
|
155
153
|
/**
|
|
156
|
-
* Package names the composition names.
|
|
157
|
-
* @param config - Composed profile; `!!js` scalars parse under Include's dialect.
|
|
158
|
-
* @returns Package names, deduplicated.
|
|
159
|
-
*/
|
|
160
|
-
function rosterOf(config) {
|
|
161
|
-
const names = /* @__PURE__ */ new Set();
|
|
162
|
-
moduleNamesOf(yaml.load(config, { schema: entryListSchema }), names);
|
|
163
|
-
return [...names];
|
|
164
|
-
}
|
|
165
|
-
/**
|
|
166
154
|
* Package names the compositions under one config tree name.
|
|
167
155
|
* @param root - Directory to walk.
|
|
168
156
|
* @returns Package names, deduplicated.
|
|
@@ -177,7 +165,7 @@ function treeRosterOf(root) {
|
|
|
177
165
|
continue;
|
|
178
166
|
}
|
|
179
167
|
if (!entry.name.endsWith(".yml") && !entry.name.endsWith(".yaml")) continue;
|
|
180
|
-
|
|
168
|
+
collectModuleNames(yaml.load(readFileSync(absolute, "utf8"), { schema: entryListSchema }), names);
|
|
181
169
|
}
|
|
182
170
|
};
|
|
183
171
|
walk(root);
|
|
@@ -499,6 +487,54 @@ function materialize(roster, options) {
|
|
|
499
487
|
missing
|
|
500
488
|
};
|
|
501
489
|
}
|
|
490
|
+
/**
|
|
491
|
+
* Request form the dynamic Client build emits for a package-local chunk: a
|
|
492
|
+
* literal `require.async("./client.<name>.js")`, served by the Host on demand.
|
|
493
|
+
*/
|
|
494
|
+
const CHUNK_REQUEST = /\brequire\.async\(\s*(["'])(\.\/client\.[A-Za-z0-9][A-Za-z0-9._-]*\.js)\1\s*\)/g;
|
|
495
|
+
/**
|
|
496
|
+
* Image paths every materialized package's `./client` export names, under any
|
|
497
|
+
* condition: those load eagerly, so a chunk request never makes them lazy.
|
|
498
|
+
* @param files - Image entries.
|
|
499
|
+
* @param packages - Materialized package names.
|
|
500
|
+
* @returns Eager Client entry paths.
|
|
501
|
+
*/
|
|
502
|
+
function clientEntriesOf(files, packages) {
|
|
503
|
+
const decoder = new TextDecoder();
|
|
504
|
+
const entries = /* @__PURE__ */ new Set();
|
|
505
|
+
const collect = (name, target) => {
|
|
506
|
+
if (typeof target === "string") entries.add(`node_modules/${name}/${target.replace(/^\.\//, "")}`);
|
|
507
|
+
else if (typeof target === "object" && target !== null) for (const value of Object.values(target)) collect(name, value);
|
|
508
|
+
};
|
|
509
|
+
for (const name of packages) {
|
|
510
|
+
const bytes = files[`node_modules/${name}/package.json`];
|
|
511
|
+
if (bytes === void 0) continue;
|
|
512
|
+
collect(name, JSON.parse(decoder.decode(bytes)).exports?.["./client"]);
|
|
513
|
+
}
|
|
514
|
+
return entries;
|
|
515
|
+
}
|
|
516
|
+
/**
|
|
517
|
+
* Page chunks requested only through `require.async`, in path order.
|
|
518
|
+
* @param files - Image entries after the sweep.
|
|
519
|
+
* @param eager - Eager Client entry paths, never lazy.
|
|
520
|
+
* @returns Image paths to move into lazy archives.
|
|
521
|
+
* @throws When a page asset requests a chunk its package does not publish.
|
|
522
|
+
*/
|
|
523
|
+
function lazyChunksOf(files, eager) {
|
|
524
|
+
const decoder = new TextDecoder();
|
|
525
|
+
const lazy = /* @__PURE__ */ new Set();
|
|
526
|
+
for (const [name, bytes] of Object.entries(files)) {
|
|
527
|
+
if (!pageAsset(name)) continue;
|
|
528
|
+
const directory = name.slice(0, name.lastIndexOf("/"));
|
|
529
|
+
for (const match of decoder.decode(bytes).matchAll(CHUNK_REQUEST)) {
|
|
530
|
+
const request = match[2];
|
|
531
|
+
const target = `${directory}/${request.slice(2)}`;
|
|
532
|
+
if (files[target] === void 0) throw new Error(`vfs image: ${name} requests chunk "${request}", which the package does not publish`);
|
|
533
|
+
if (!eager.has(target)) lazy.add(target);
|
|
534
|
+
}
|
|
535
|
+
}
|
|
536
|
+
return [...lazy].sort();
|
|
537
|
+
}
|
|
502
538
|
/** Gzip header byte that records the packing platform; RFC 1952 §2.3.1 spells 255 "unknown". */
|
|
503
539
|
const GZIP_OS_UNKNOWN = 255;
|
|
504
540
|
/** Offset of that byte in the gzip member header. */
|
|
@@ -527,42 +563,77 @@ function compressImage(archive) {
|
|
|
527
563
|
* The manifest's claim is all-or-nothing: it names the one contract every packed body
|
|
528
564
|
* was emitted against. A module the transform cannot express therefore fails the pack
|
|
529
565
|
* rather than downgrading the image, because a mostly-transformed image boots into
|
|
530
|
-
* errors far from their cause.
|
|
531
|
-
*
|
|
532
|
-
* @
|
|
566
|
+
* errors far from their cause. Page chunks requested through `require.async` leave
|
|
567
|
+
* the base image for deterministic archives named by their content digest, which the manifest registers.
|
|
568
|
+
* @param options - Roster, installation, package index, and paths.
|
|
569
|
+
* @returns The compressed image, its lazy archives, and what went into them.
|
|
533
570
|
* @throws When a config tree or workspace directory named in the options is missing,
|
|
534
|
-
* because a silently thinner image fails much later and much less clearly
|
|
571
|
+
* because a silently thinner image fails much later and much less clearly; when the
|
|
572
|
+
* installation is also a materialized package; or when a page asset requests a chunk
|
|
573
|
+
* its package does not publish.
|
|
535
574
|
*/
|
|
536
575
|
function packVfsImage(options) {
|
|
537
576
|
const root = options.root ?? DEFAULT_ROOT;
|
|
538
577
|
const encoder = new TextEncoder();
|
|
539
578
|
const configTrees = options.configTrees ?? [];
|
|
540
579
|
for (const tree of configTrees) if (!existsSync(tree.directory)) throw new Error(`vfs image: config tree ${tree.mount} is missing at ${tree.directory}`);
|
|
541
|
-
const roster = [...new Set([...
|
|
580
|
+
const roster = [...new Set([...options.roster, ...configTrees.filter((tree) => tree.scanRoster === true).flatMap((tree) => treeRosterOf(tree.directory))])];
|
|
542
581
|
const { files, packages, missing } = materialize(roster, options);
|
|
543
|
-
|
|
582
|
+
const installation = options.installation;
|
|
583
|
+
if (installation !== void 0) {
|
|
584
|
+
if (packages.has(installation.name)) throw new Error(`vfs image: installation ${installation.name} is also materialized; its manifest must enter the image alone`);
|
|
585
|
+
files[`node_modules/${installation.name}/package.json`] = readFileSync(installation.manifest);
|
|
586
|
+
}
|
|
587
|
+
if (options.homePatch !== void 0 && options.homePatch.length > 0) files[HOME_PATCH_PATH] = encoder.encode(yaml.dump(options.homePatch, {
|
|
588
|
+
schema: entryListSchema,
|
|
589
|
+
noRefs: true
|
|
590
|
+
}));
|
|
544
591
|
for (const tree of configTrees) collectTree(tree.directory, files, tree.mount, (relativePath) => !excluded(relativePath));
|
|
545
592
|
const executables = dropExecutables(files);
|
|
546
593
|
const { swept, transform, javascriptEntries, droppedJavascriptEntries, unresolvedExternalRequests } = sweepImage(files, options, [...packages.keys()].filter((name) => options.workspaces.has(name)), root);
|
|
547
|
-
|
|
594
|
+
const lazyArchives = [];
|
|
595
|
+
const lazy = [];
|
|
596
|
+
for (const path of lazyChunksOf(swept, clientEntriesOf(swept, packages.keys()))) {
|
|
597
|
+
const bytes = swept[path];
|
|
598
|
+
delete swept[path];
|
|
599
|
+
const image = compressImage(packTar({ [path]: bytes }));
|
|
600
|
+
const name = `${IMAGE_LAZY_DIRECTORY}/${createHash("sha256").update(image).digest("hex")}.tar.gz`;
|
|
601
|
+
lazyArchives.push({
|
|
602
|
+
name,
|
|
603
|
+
image,
|
|
604
|
+
files: [path]
|
|
605
|
+
});
|
|
606
|
+
lazy.push({
|
|
607
|
+
archive: name,
|
|
608
|
+
files: [{
|
|
609
|
+
path,
|
|
610
|
+
size: bytes.byteLength
|
|
611
|
+
}]
|
|
612
|
+
});
|
|
613
|
+
}
|
|
614
|
+
const manifest = {
|
|
548
615
|
root,
|
|
549
616
|
profile: options.profile,
|
|
550
|
-
|
|
551
|
-
|
|
617
|
+
...installation === void 0 ? {} : { installation: installation.name },
|
|
618
|
+
lowered: WRAPPER_CONTRACT,
|
|
619
|
+
javascriptEntries: javascriptEntries - lazy.length,
|
|
552
620
|
visitedEntries: transform.visited,
|
|
553
|
-
rewrittenEntries: transform.rewritten
|
|
554
|
-
|
|
621
|
+
rewrittenEntries: transform.rewritten,
|
|
622
|
+
lazy
|
|
623
|
+
};
|
|
624
|
+
swept[MANIFEST_PATH] = encoder.encode(`${JSON.stringify(manifest, null, 2)}\n`);
|
|
555
625
|
for (const directory of options.emptyDirectories ?? IMAGE_EMPTY_DIRECTORIES) swept[directory] = new Uint8Array(0);
|
|
556
626
|
return {
|
|
557
627
|
image: compressImage(packTar(swept)),
|
|
558
628
|
files: swept,
|
|
629
|
+
lazyArchives,
|
|
559
630
|
packages,
|
|
560
631
|
workspacePackages: [...packages.keys()].filter((name) => options.workspaces.has(name)).length,
|
|
561
632
|
roster,
|
|
562
633
|
missing,
|
|
563
634
|
executables,
|
|
564
635
|
pageBundles: Object.keys(swept).filter((name) => pageAsset(name)),
|
|
565
|
-
javascriptEntries,
|
|
636
|
+
javascriptEntries: manifest.javascriptEntries,
|
|
566
637
|
droppedJavascriptEntries,
|
|
567
638
|
unresolvedExternalRequests,
|
|
568
639
|
transform,
|
|
@@ -687,8 +758,8 @@ function selectedSessions(files) {
|
|
|
687
758
|
//#endregion
|
|
688
759
|
//#region lib/types/repository.js
|
|
689
760
|
/**
|
|
690
|
-
* Repository knowledge for the packer: where this tree's workspaces,
|
|
691
|
-
*
|
|
761
|
+
* Repository knowledge for the packer: where this tree's workspaces, installation
|
|
762
|
+
* manifest, profile bundles, and config trees are, and how to report a pack.
|
|
692
763
|
*
|
|
693
764
|
* The library half takes all of this as parameters. Keeping the lookup here is what
|
|
694
765
|
* lets the same library pack a different tree, and what keeps `pack.ts` free of
|
|
@@ -707,10 +778,10 @@ const WORKSPACE_SCAN_ROOTS = [
|
|
|
707
778
|
"native/system/packages",
|
|
708
779
|
"apps"
|
|
709
780
|
];
|
|
710
|
-
/**
|
|
781
|
+
/** The `dsh` installation package: its manifest anchors bundle resolution, as the launcher's does. */
|
|
711
782
|
const CLI_PACKAGE = "apps/cli";
|
|
712
|
-
/**
|
|
713
|
-
const
|
|
783
|
+
/** Diagnostic prefix for the app-boot readers this module calls. */
|
|
784
|
+
const BIN_NAME = "dsh-pack-vfs-image";
|
|
714
785
|
/** Repository-owned deterministic filesystem content offered by the preview. */
|
|
715
786
|
const PREVIEW_EXAMPLE_ROOT = "packages/experimental/webworker-runtime/tests/fixtures/vfs-example";
|
|
716
787
|
/**
|
|
@@ -740,42 +811,102 @@ function indexWorkspacePackages(repoRoot) {
|
|
|
740
811
|
return index;
|
|
741
812
|
}
|
|
742
813
|
/**
|
|
743
|
-
*
|
|
744
|
-
*
|
|
745
|
-
*
|
|
746
|
-
*
|
|
747
|
-
* bundles, and its patch files — would otherwise leak this machine's plugins
|
|
748
|
-
* into the image and break the same-tree-same-bytes guarantee.
|
|
814
|
+
* Read what one shipped profile offers from the installation the launcher boots:
|
|
815
|
+
* the template's bundles and every optional bundle, each resolved from the CLI
|
|
816
|
+
* manifest as the launcher resolves them, and the packages their patch files name.
|
|
817
|
+
* Machine state never contributes, so the same tree always packs the same roster.
|
|
749
818
|
* @param repoRoot - Absolute repository root.
|
|
750
|
-
* @param profile -
|
|
751
|
-
* @returns
|
|
819
|
+
* @param profile - Shipped profile template name.
|
|
820
|
+
* @returns Bundle names and the roster to materialize.
|
|
821
|
+
* @throws When the profile has no shipped template, or a bundle does not resolve or declares no patch.
|
|
752
822
|
*/
|
|
753
|
-
function
|
|
754
|
-
const
|
|
755
|
-
|
|
756
|
-
|
|
757
|
-
|
|
758
|
-
|
|
759
|
-
|
|
760
|
-
|
|
761
|
-
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
|
|
769
|
-
|
|
823
|
+
function profileRoster(repoRoot, profile) {
|
|
824
|
+
const template = shippedTemplate(profile);
|
|
825
|
+
const bundles = [...new Set([...template.bundles, ...OPTIONAL_BUNDLES])];
|
|
826
|
+
const names = new Set(bundles);
|
|
827
|
+
for (const bundle of bundles) collectModuleNames(bundlePatches(repoRoot, bundle), names);
|
|
828
|
+
return {
|
|
829
|
+
bundles,
|
|
830
|
+
roster: [...names]
|
|
831
|
+
};
|
|
832
|
+
}
|
|
833
|
+
/**
|
|
834
|
+
* The home patch one shipped profile's image carries: every row `PLATFORM_ENTRY_CONFIG` names, with the config the
|
|
835
|
+
* template's bundle layers compose for it and the Worker's required fields over that config. A patch replaces a row's
|
|
836
|
+
* whole config, so the composed fields stay; the image carries the same bundle files read here.
|
|
837
|
+
* @param repoRoot - Absolute repository root.
|
|
838
|
+
* @param profile - Shipped profile template name.
|
|
839
|
+
* @returns Home patch rows; a row the composition lacks contributes none.
|
|
840
|
+
* @throws When the profile has no shipped template, or a bundle does not resolve or declares no patch.
|
|
841
|
+
*/
|
|
842
|
+
function platformHomePatch(repoRoot, profile) {
|
|
843
|
+
const composed = composeEntries([shippedTemplate(profile).bundles.flatMap((bundle) => bundlePatches(repoRoot, bundle))]);
|
|
844
|
+
return Object.entries(PLATFORM_ENTRY_CONFIG).flatMap(([id, fields]) => {
|
|
845
|
+
const row = findRow(composed, id);
|
|
846
|
+
if (row === void 0) return [];
|
|
847
|
+
return [{
|
|
848
|
+
id,
|
|
849
|
+
config: {
|
|
850
|
+
...typeof row.config === "object" && row.config !== null && !Array.isArray(row.config) ? row.config : {},
|
|
851
|
+
...fields
|
|
770
852
|
}
|
|
771
|
-
}
|
|
772
|
-
}
|
|
773
|
-
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
853
|
+
}];
|
|
854
|
+
});
|
|
855
|
+
}
|
|
856
|
+
/** Find one Loader row by id, inside group configs too. */
|
|
857
|
+
function findRow(rows, id) {
|
|
858
|
+
if (!Array.isArray(rows)) return void 0;
|
|
859
|
+
for (const row of rows) {
|
|
860
|
+
if (row.id === id) return row;
|
|
861
|
+
const nested = findRow(row.config, id);
|
|
862
|
+
if (nested !== void 0) return nested;
|
|
777
863
|
}
|
|
778
864
|
}
|
|
865
|
+
/** The shipped template of one profile, or a refusal naming it. */
|
|
866
|
+
function shippedTemplate(profile) {
|
|
867
|
+
const template = Object.hasOwn(PROFILE_TEMPLATES, profile) ? PROFILE_TEMPLATES[profile] : void 0;
|
|
868
|
+
if (template === void 0) throw new Error(`vfs image: no shipped profile template named ${JSON.stringify(profile)}`);
|
|
869
|
+
return template;
|
|
870
|
+
}
|
|
871
|
+
/** One bundle's patch rows, resolved from the CLI manifest as the launcher resolves them. */
|
|
872
|
+
function bundlePatches(repoRoot, bundle) {
|
|
873
|
+
const directory = resolveBundleDir(BIN_NAME, bundle, installationManifest(repoRoot).manifest, join(repoRoot, CLI_PACKAGE));
|
|
874
|
+
const declaration = JSON.parse(readFileSync(join(directory, "package.json"), "utf8")).dsh?.bundle;
|
|
875
|
+
if (declaration === void 0) throw new Error(`vfs image: bundle ${bundle} declares no dsh.bundle`);
|
|
876
|
+
return bundlePatchPaths(directory, declaration).flatMap((file) => loadOverlayPatches(BIN_NAME, file));
|
|
877
|
+
}
|
|
878
|
+
/**
|
|
879
|
+
* The installation manifest the image anchors bundle resolution at.
|
|
880
|
+
* @param repoRoot - Absolute repository root.
|
|
881
|
+
* @returns The CLI package name and its manifest path.
|
|
882
|
+
*/
|
|
883
|
+
function installationManifest(repoRoot) {
|
|
884
|
+
const manifest = join(repoRoot, CLI_PACKAGE, "package.json");
|
|
885
|
+
const { name } = JSON.parse(readFileSync(manifest, "utf8"));
|
|
886
|
+
return {
|
|
887
|
+
name,
|
|
888
|
+
manifest
|
|
889
|
+
};
|
|
890
|
+
}
|
|
891
|
+
/**
|
|
892
|
+
* Pack this repository's base image for one shipped profile.
|
|
893
|
+
* @param repoRoot - Absolute repository root.
|
|
894
|
+
* @param profile - Shipped profile template name.
|
|
895
|
+
* @param root - Virtual root; the runtime default when omitted.
|
|
896
|
+
* @returns The packed image and its lazy archives.
|
|
897
|
+
*/
|
|
898
|
+
function packRepositoryImage(repoRoot, profile, root) {
|
|
899
|
+
return packVfsImage({
|
|
900
|
+
profile,
|
|
901
|
+
roster: profileRoster(repoRoot, profile).roster,
|
|
902
|
+
installation: installationManifest(repoRoot),
|
|
903
|
+
...root === void 0 ? {} : { root },
|
|
904
|
+
workspaces: indexWorkspacePackages(repoRoot),
|
|
905
|
+
resolveFrom: repoRoot,
|
|
906
|
+
configTrees: configTrees(repoRoot),
|
|
907
|
+
homePatch: platformHomePatch(repoRoot, profile)
|
|
908
|
+
});
|
|
909
|
+
}
|
|
779
910
|
/**
|
|
780
911
|
* Config trees the CLI package declares for deployment images
|
|
781
912
|
* (`dsh.configTrees` in its package.json): `path` is relative to the CLI
|
|
@@ -851,6 +982,8 @@ function describePack(result, repoRoot, outputFile) {
|
|
|
851
982
|
` raw ${megabytes(Object.values(result.files).reduce((sum, bytes) => sum + bytes.byteLength, 0))}`,
|
|
852
983
|
` compressed ${megabytes(result.image.byteLength)}`,
|
|
853
984
|
` config + presets ${megabytes(sizeOf("config/"))}`,
|
|
985
|
+
` lazy archives ${String(result.lazyArchives.length)} (${megabytes(result.lazyArchives.reduce((sum, archive) => sum + archive.image.byteLength, 0))} compressed)`,
|
|
986
|
+
...result.lazyArchives.map((archive) => ` ${megabytes(archive.image.byteLength).padStart(9)} ${archive.name} ${archive.files.join(", ")}`),
|
|
854
987
|
` javascript entries ${String(result.javascriptEntries)} (dropped ${String(result.executables.length)} executable scripts, ${String(result.pageBundles.length)} page bundles verbatim)`,
|
|
855
988
|
` wrapper contract ${result.contract}`,
|
|
856
989
|
` transform ${String(result.transform.rewritten)} of ${String(result.transform.visited)} reached entries rewritten, ${String(result.droppedJavascriptEntries)} unreachable dropped`,
|
|
@@ -862,4 +995,4 @@ function describePack(result, repoRoot, outputFile) {
|
|
|
862
995
|
];
|
|
863
996
|
}
|
|
864
997
|
//#endregion
|
|
865
|
-
export {
|
|
998
|
+
export { packRepositoryImage as a, packPreviewFixture as c, collectModuleNames as d, packVfsImage as f, installationManifest as i, DEFAULT_ROOT$1 as l, WRAPPER_CONTRACT as m, describePack as n, previewFixtures as o, packVfsOverlay as p, indexWorkspacePackages as r, profileRoster as s, configTrees as t, MANIFEST_PATH as u };
|
package/lib/types/bin.js
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
/**
|
|
3
|
-
* Pack a Preview deployment from this repository:
|
|
4
|
-
* image,
|
|
3
|
+
* Pack a Preview deployment from this repository: materialize and lower the
|
|
4
|
+
* base image for one shipped profile, write its lazy chunk archives beside it,
|
|
5
|
+
* then write each named fixture overlay and their manifest.
|
|
5
6
|
*
|
|
6
7
|
* Usage: dsh-pack-vfs-image --out <file> [--profile web] [--root /dsh]
|
|
7
8
|
* node --import tsx/esm src/bin.ts --out ../../apps/web/dist/preview/vfs-image.tar.gz
|
|
@@ -11,8 +12,7 @@ import { mkdirSync, writeFileSync } from 'node:fs';
|
|
|
11
12
|
import { dirname, isAbsolute, join, resolve } from 'node:path';
|
|
12
13
|
import { fileURLToPath } from 'node:url';
|
|
13
14
|
import { PREVIEW_FIXTURE_MANIFEST_FILE, PREVIEW_FIXTURE_MANIFEST_VERSION, } from '@deepseek-ai/dsh-experimental-webworker-runtime';
|
|
14
|
-
import {
|
|
15
|
-
import { composeProfile, configTrees, describePack, indexWorkspacePackages, packPreviewFixture, previewFixtures, } from "./repository.js";
|
|
15
|
+
import { describePack, packPreviewFixture, packRepositoryImage, previewFixtures, } from "./repository.js";
|
|
16
16
|
/**
|
|
17
17
|
* Read one `--flag value` pair.
|
|
18
18
|
* @param name - Flag name without dashes.
|
|
@@ -38,19 +38,18 @@ const repoRoot = fileURLToPath(new URL('../../../../', import.meta.url));
|
|
|
38
38
|
const profile = flag('profile', 'web');
|
|
39
39
|
const out = flag('out');
|
|
40
40
|
const outputFile = isAbsolute(out) ? out : resolve(process.cwd(), out);
|
|
41
|
-
const result =
|
|
42
|
-
config: composeProfile(repoRoot, profile),
|
|
43
|
-
profile,
|
|
44
|
-
root: flag('root', '/dsh'),
|
|
45
|
-
workspaces: indexWorkspacePackages(repoRoot),
|
|
46
|
-
resolveFrom: repoRoot,
|
|
47
|
-
configTrees: configTrees(repoRoot),
|
|
48
|
-
});
|
|
41
|
+
const result = packRepositoryImage(repoRoot, profile, flag('root', '/dsh'));
|
|
49
42
|
if (result.missing.length > 0) {
|
|
50
43
|
throw new Error(`vfs image: ${String(result.missing.length)} dependencies did not resolve; the image would be incomplete`);
|
|
51
44
|
}
|
|
52
45
|
mkdirSync(dirname(outputFile), { recursive: true });
|
|
53
46
|
writeFileSync(outputFile, result.image);
|
|
47
|
+
// Lazy archive names are relative to the base image, which the Worker resolves them against.
|
|
48
|
+
for (const archive of result.lazyArchives) {
|
|
49
|
+
const path = join(dirname(outputFile), archive.name);
|
|
50
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
51
|
+
writeFileSync(path, archive.image);
|
|
52
|
+
}
|
|
54
53
|
const fixtureDefinitions = previewFixtures(repoRoot);
|
|
55
54
|
const fixtureDirectory = join(dirname(outputFile), 'fixtures');
|
|
56
55
|
mkdirSync(fixtureDirectory, { recursive: true });
|
package/lib/types/index.d.ts
CHANGED
|
@@ -3,6 +3,6 @@
|
|
|
3
3
|
* @module @deepseek-ai/dsh-experimental-webworker-packer
|
|
4
4
|
*/
|
|
5
5
|
export { WRAPPER_CONTRACT, type ImageFiles, type TransformOutcome, } from './transform-image.ts';
|
|
6
|
-
export {
|
|
7
|
-
export {
|
|
6
|
+
export { collectModuleNames, DEFAULT_ROOT, MANIFEST_PATH, packVfsImage, packVfsOverlay, type ConfigTree, type ImageInstallation, type ImageTree, type PackedLazyArchive, type PackOptions, type PackOverlayResult, type PackResult, } from './pack.ts';
|
|
7
|
+
export { configTrees, describePack, indexWorkspacePackages, installationManifest, packPreviewFixture, packRepositoryImage, previewFixtures, profileRoster, type PreviewFixture, type ProfileRoster, } from './repository.ts';
|
|
8
8
|
//# sourceMappingURL=index.d.ts.map
|
package/lib/types/pack.d.ts
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
|
+
import { type PatchOptions } from '@deepseek-ai/cordis-plugin-include';
|
|
1
2
|
import { type ImageFiles, type TransformOutcome } from './transform-image.ts';
|
|
2
3
|
export { DEFAULT_ROOT } from '@deepseek-ai/dsh-experimental-webworker-runtime';
|
|
3
4
|
/** Image path of the manifest; the layout contract's name, re-exported for callers. */
|
|
4
5
|
export declare const MANIFEST_PATH: string;
|
|
5
|
-
/** Image path of the
|
|
6
|
-
export declare const
|
|
6
|
+
/** Image path of the home patch: `$DSH_HOME/cordis.patch.yml` under the Worker home. */
|
|
7
|
+
export declare const HOME_PATCH_PATH = "home/cordis.patch.yml";
|
|
7
8
|
/** One directory tree to copy into the image at a caller-selected mount. */
|
|
8
9
|
export interface ImageTree {
|
|
9
10
|
/** Image path to mount it at, relative to the virtual root. */
|
|
@@ -20,12 +21,24 @@ export interface ConfigTree extends ImageTree {
|
|
|
20
21
|
*/
|
|
21
22
|
readonly scanRoster?: boolean;
|
|
22
23
|
}
|
|
24
|
+
/** The installation whose manifest alone anchors bundle resolution inside the image. */
|
|
25
|
+
export interface ImageInstallation {
|
|
26
|
+
/** Package name the manifest is mounted under, `node_modules/<name>/package.json`. */
|
|
27
|
+
readonly name: string;
|
|
28
|
+
/** Absolute path of the manifest to copy. */
|
|
29
|
+
readonly manifest: string;
|
|
30
|
+
}
|
|
23
31
|
/** Everything the packer needs that it cannot know by itself. */
|
|
24
32
|
export interface PackOptions {
|
|
25
|
-
/**
|
|
26
|
-
readonly config: string;
|
|
27
|
-
/** Profile name, recorded in the manifest. */
|
|
33
|
+
/** Shipped profile template the worker host boots, recorded in the manifest. */
|
|
28
34
|
readonly profile: string;
|
|
35
|
+
/**
|
|
36
|
+
* Package names whose dependency closures the image materializes: the bundles the
|
|
37
|
+
* image offers and every package their patch rows name.
|
|
38
|
+
*/
|
|
39
|
+
readonly roster: readonly string[];
|
|
40
|
+
/** Installation manifest copied alone, without its package or dependency closure. */
|
|
41
|
+
readonly installation?: ImageInstallation;
|
|
29
42
|
/** Virtual root the image mounts under; defaults to {@link DEFAULT_ROOT}. */
|
|
30
43
|
readonly root?: string;
|
|
31
44
|
/** Package name to absolute directory, for workspace and vendored packages. */
|
|
@@ -36,18 +49,31 @@ export interface PackOptions {
|
|
|
36
49
|
readonly configTrees?: readonly ConfigTree[];
|
|
37
50
|
/** Empty directories to create; defaults to `home/`, `workspace/`, `tmp/`. */
|
|
38
51
|
readonly emptyDirectories?: readonly string[];
|
|
52
|
+
/** Patch rows written to {@link HOME_PATCH_PATH}, which the image's app-boot reads as the home layer. */
|
|
53
|
+
readonly homePatch?: readonly PatchOptions[];
|
|
39
54
|
/**
|
|
40
55
|
* Extra sweep roots: image specifiers requested by code outside the image.
|
|
41
56
|
* Defaults to the worker assembly's own entries.
|
|
42
57
|
*/
|
|
43
58
|
readonly entries?: readonly string[];
|
|
44
59
|
}
|
|
60
|
+
/** One lazy chunk archive to write beside the base image. */
|
|
61
|
+
export interface PackedLazyArchive {
|
|
62
|
+
/** Path relative to the base image, `lazy/<sha256 of image>.tar.gz`; the digest names the archive, and the Worker does not verify it. */
|
|
63
|
+
readonly name: string;
|
|
64
|
+
/** The gzip-compressed tar archive. */
|
|
65
|
+
readonly image: Uint8Array;
|
|
66
|
+
/** Image paths the archive carries. */
|
|
67
|
+
readonly files: readonly string[];
|
|
68
|
+
}
|
|
45
69
|
/** What one pack produced, for the caller to report or assert on. */
|
|
46
70
|
export interface PackResult {
|
|
47
71
|
/** The gzip-compressed tar archive to write; the runtime inflates it at mount. */
|
|
48
72
|
readonly image: Uint8Array;
|
|
49
|
-
/** Every entry, before zipping; the manifest is already among them. */
|
|
73
|
+
/** Every base-image entry, before zipping; the manifest is already among them. */
|
|
50
74
|
readonly files: ImageFiles;
|
|
75
|
+
/** Page chunks moved out of the base image, one archive each, in path order. */
|
|
76
|
+
readonly lazyArchives: readonly PackedLazyArchive[];
|
|
51
77
|
/** Package name to how many files it contributed, in materialization order. */
|
|
52
78
|
readonly packages: ReadonlyMap<string, number>;
|
|
53
79
|
/** How many of them came from the workspace rather than from `node_modules`. */
|
|
@@ -78,17 +104,30 @@ export interface PackOverlayResult {
|
|
|
78
104
|
/** Every path in the overlay before compression. */
|
|
79
105
|
readonly files: ImageFiles;
|
|
80
106
|
}
|
|
107
|
+
/**
|
|
108
|
+
* Collect package names from entry rows and patch lists, recursively through
|
|
109
|
+
* `insert` lists and nested `config` row lists (groups). Builtin rows
|
|
110
|
+
* (`cordis:group`), preset metadata documents, and file-relative entries carry
|
|
111
|
+
* names that are not package specifiers; only scoped names and bare names
|
|
112
|
+
* with a subpath count.
|
|
113
|
+
* @param rows - Parsed YAML value; anything but a row array is ignored.
|
|
114
|
+
* @param names - Package names collected so far.
|
|
115
|
+
*/
|
|
116
|
+
export declare function collectModuleNames(rows: unknown, names: Set<string>): void;
|
|
81
117
|
/**
|
|
82
118
|
* Pack one VFS image.
|
|
83
119
|
*
|
|
84
120
|
* The manifest's claim is all-or-nothing: it names the one contract every packed body
|
|
85
121
|
* was emitted against. A module the transform cannot express therefore fails the pack
|
|
86
122
|
* rather than downgrading the image, because a mostly-transformed image boots into
|
|
87
|
-
* errors far from their cause.
|
|
88
|
-
*
|
|
89
|
-
* @
|
|
123
|
+
* errors far from their cause. Page chunks requested through `require.async` leave
|
|
124
|
+
* the base image for deterministic archives named by their content digest, which the manifest registers.
|
|
125
|
+
* @param options - Roster, installation, package index, and paths.
|
|
126
|
+
* @returns The compressed image, its lazy archives, and what went into them.
|
|
90
127
|
* @throws When a config tree or workspace directory named in the options is missing,
|
|
91
|
-
* because a silently thinner image fails much later and much less clearly
|
|
128
|
+
* because a silently thinner image fails much later and much less clearly; when the
|
|
129
|
+
* installation is also a materialized package; or when a page asset requests a chunk
|
|
130
|
+
* its package does not publish.
|
|
92
131
|
*/
|
|
93
132
|
export declare function packVfsImage(options: PackOptions): PackResult;
|
|
94
133
|
/**
|
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
import type {
|
|
1
|
+
import type { PatchOptions } from '@deepseek-ai/cordis-plugin-include';
|
|
2
|
+
import { type ConfigTree, type ImageInstallation, type ImageTree, type PackResult } from './pack.ts';
|
|
2
3
|
export { packPreviewFixture } from './preview.ts';
|
|
3
4
|
/** One built-in Preview source and the trees packed into its overlay. */
|
|
4
5
|
export interface PreviewFixture {
|
|
@@ -17,18 +18,48 @@ export interface PreviewFixture {
|
|
|
17
18
|
* @returns Package name to absolute directory.
|
|
18
19
|
*/
|
|
19
20
|
export declare function indexWorkspacePackages(repoRoot: string): Map<string, string>;
|
|
21
|
+
/** The bundles one image offers and the packages it must materialize for them. */
|
|
22
|
+
export interface ProfileRoster {
|
|
23
|
+
/** The profile template's bundles, then the installation's optional bundles. */
|
|
24
|
+
readonly bundles: readonly string[];
|
|
25
|
+
/** Those bundle packages plus every package their patch rows name, deduplicated. */
|
|
26
|
+
readonly roster: readonly string[];
|
|
27
|
+
}
|
|
28
|
+
/**
|
|
29
|
+
* Read what one shipped profile offers from the installation the launcher boots:
|
|
30
|
+
* the template's bundles and every optional bundle, each resolved from the CLI
|
|
31
|
+
* manifest as the launcher resolves them, and the packages their patch files name.
|
|
32
|
+
* Machine state never contributes, so the same tree always packs the same roster.
|
|
33
|
+
* @param repoRoot - Absolute repository root.
|
|
34
|
+
* @param profile - Shipped profile template name.
|
|
35
|
+
* @returns Bundle names and the roster to materialize.
|
|
36
|
+
* @throws When the profile has no shipped template, or a bundle does not resolve or declares no patch.
|
|
37
|
+
*/
|
|
38
|
+
export declare function profileRoster(repoRoot: string, profile: string): ProfileRoster;
|
|
39
|
+
/**
|
|
40
|
+
* The home patch one shipped profile's image carries: every row `PLATFORM_ENTRY_CONFIG` names, with the config the
|
|
41
|
+
* template's bundle layers compose for it and the Worker's required fields over that config. A patch replaces a row's
|
|
42
|
+
* whole config, so the composed fields stay; the image carries the same bundle files read here.
|
|
43
|
+
* @param repoRoot - Absolute repository root.
|
|
44
|
+
* @param profile - Shipped profile template name.
|
|
45
|
+
* @returns Home patch rows; a row the composition lacks contributes none.
|
|
46
|
+
* @throws When the profile has no shipped template, or a bundle does not resolve or declares no patch.
|
|
47
|
+
*/
|
|
48
|
+
export declare function platformHomePatch(repoRoot: string, profile: string): PatchOptions[];
|
|
49
|
+
/**
|
|
50
|
+
* The installation manifest the image anchors bundle resolution at.
|
|
51
|
+
* @param repoRoot - Absolute repository root.
|
|
52
|
+
* @returns The CLI package name and its manifest path.
|
|
53
|
+
*/
|
|
54
|
+
export declare function installationManifest(repoRoot: string): ImageInstallation;
|
|
20
55
|
/**
|
|
21
|
-
*
|
|
22
|
-
* unevaluated. The dump runs against a throwaway Harness home, so the
|
|
23
|
-
* profile's own layer is the freshly initialized empty patch file and the
|
|
24
|
-
* machine's `$DSH_HOME` — its profile manifest with locally installed
|
|
25
|
-
* bundles, and its patch files — would otherwise leak this machine's plugins
|
|
26
|
-
* into the image and break the same-tree-same-bytes guarantee.
|
|
56
|
+
* Pack this repository's base image for one shipped profile.
|
|
27
57
|
* @param repoRoot - Absolute repository root.
|
|
28
|
-
* @param profile -
|
|
29
|
-
* @
|
|
58
|
+
* @param profile - Shipped profile template name.
|
|
59
|
+
* @param root - Virtual root; the runtime default when omitted.
|
|
60
|
+
* @returns The packed image and its lazy archives.
|
|
30
61
|
*/
|
|
31
|
-
export declare function
|
|
62
|
+
export declare function packRepositoryImage(repoRoot: string, profile: string, root?: string): PackResult;
|
|
32
63
|
/**
|
|
33
64
|
* Config trees the CLI package declares for deployment images
|
|
34
65
|
* (`dsh.configTrees` in its package.json): `path` is relative to the CLI
|
package/lib/types/rules.d.ts
CHANGED
|
@@ -33,7 +33,7 @@ export declare const EXCLUDE_WORKSPACE: readonly string[];
|
|
|
33
33
|
*/
|
|
34
34
|
export declare const PAGE_ASSETS: readonly string[];
|
|
35
35
|
/**
|
|
36
|
-
* Image specifiers the worker assembly requires directly, beyond the
|
|
36
|
+
* Image specifiers the worker assembly requires directly, beyond the
|
|
37
37
|
* roster: they are requested by worker-bundle code, so no image file
|
|
38
38
|
* references them and the reachability sweep must seed them as roots. Keep in
|
|
39
39
|
* step with the literal `require`/`resolve` calls in the runtime's
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@deepseek-ai/dsh-experimental-webworker-packer",
|
|
3
3
|
"description": "Build-time packer for the browser runtime's base VFS image and ordered data-overlay archives",
|
|
4
|
-
"version": "0.2.1-alpha.
|
|
4
|
+
"version": "0.2.1-alpha.4",
|
|
5
5
|
"publishConfig": {
|
|
6
6
|
"access": "public"
|
|
7
7
|
},
|
|
@@ -34,11 +34,11 @@
|
|
|
34
34
|
"license": "MIT",
|
|
35
35
|
"dependencies": {
|
|
36
36
|
"@deepseek-ai/cordis-plugin-include": "~1.0.10-alpha.2",
|
|
37
|
-
"@deepseek-ai/dsh-
|
|
38
|
-
"@deepseek-ai/dsh-
|
|
39
|
-
"@deepseek-ai/dsh-session-format": "0.2.1-alpha.
|
|
40
|
-
"@deepseek-ai/dsh-session-format-catalog": "0.2.1-alpha.
|
|
41
|
-
"@deepseek-ai/dsh-session-format-v3-to-v4": "0.2.1-alpha.
|
|
37
|
+
"@deepseek-ai/dsh-app-boot": "0.2.1-alpha.4",
|
|
38
|
+
"@deepseek-ai/dsh-experimental-webworker-runtime": "0.2.1-alpha.4",
|
|
39
|
+
"@deepseek-ai/dsh-session-format": "0.2.1-alpha.4",
|
|
40
|
+
"@deepseek-ai/dsh-session-format-catalog": "0.2.1-alpha.4",
|
|
41
|
+
"@deepseek-ai/dsh-session-format-v3-to-v4": "0.2.1-alpha.4",
|
|
42
42
|
"js-yaml": "^4.3.2",
|
|
43
43
|
"picomatch": "^4.0.7"
|
|
44
44
|
},
|