@davideasden/pi-undo 0.3.3 → 0.3.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/native-performance.md +91 -0
- package/native/bin/pi-undo-fs-darwin-arm64 +0 -0
- package/native/bin/pi-undo-fs-darwin-x64 +0 -0
- package/native/bin/pi-undo-fs-linux-arm64 +0 -0
- package/native/bin/pi-undo-fs-linux-x64 +0 -0
- package/native/bin/pi-undo-fs-win32-arm64.exe +0 -0
- package/native/bin/pi-undo-fs-win32-x64.exe +0 -0
- package/package.json +1 -1
- package/src/controller.ts +16 -1
- package/src/durable-pack.ts +137 -25
- package/src/encoding.ts +2 -2
- package/src/pi-runtime.ts +16 -2
- package/src/restore-engine.ts +101 -40
- package/src/root-discovery.ts +1 -0
- package/src/snapshot-store.ts +207 -22
|
@@ -2,6 +2,97 @@
|
|
|
2
2
|
|
|
3
3
|
本轮实现目录项缓存与 scoped restore 的可见路径检查,保留恢复前后的拓扑校验。目录扫描仍会逐个打开、核验工作区内的目录;缓存减少 `readdir`,没有把整次检查变成只访问 changedPaths 的增量操作。
|
|
4
4
|
|
|
5
|
+
## Durable pack 准备与批量写入
|
|
6
|
+
|
|
7
|
+
`createDurablePack` 的第三个参数和 `prepareDurableRestore` 的第四个参数支持可选的 `onTiming(phase, durationMs)` 与 `onDiagnostics(diagnostic)`。阶段标签包括 `prepare.plan`、`prepare.blobs`、`prepare.entries`、`pack.validate`、`pack.encode`、`pack.write`、`pack.sync`、`pack.publish`、`pack.checksum`、`pack.resident`、`prepare.pin`、`prepare.index` 和 `prepare.total`。其中 `prepare.total` 是完整准备时间,不能与各分阶段时间相加。默认 SnapshotStore 的 `prepare.plan` 包含对象完整性检查和 blob 预取;自定义 store 的兼容逐文件读取会计入 `prepare.entries`。计时在相应阶段成功结束时报告;没有可预制计划或发生异常时,不报告完整准备成功时间。
|
|
8
|
+
|
|
9
|
+
数量诊断报告 entries、fileVariants、payloadBytes、packBytes 和 writeCalls,不包含路径、内容或 operation ID。writeCalls 统计实际发起的 FileHandle 写入调用,包含短写重试,不代表操作系统内部 syscall 数量。诊断回调异常不会改变生成结果;这些回调可供基准或调用方采集,默认不另行写入运行时诊断文件。
|
|
10
|
+
|
|
11
|
+
写入使用最多 256 个 buffer、最多 4 MiB 的 `writev` 批次。单个大 header 或 payload 会切成视图分批写入,不拼接整包;空 payload 不发起写入。短写按实际写入字节数推进,覆盖 buffer 内、边界和跨 buffer 的情况;零进展、无效长度和 I/O 错误会失败。批次之间及发布前检查操作取消。持久化顺序仍为临时文件写完、文件同步并关闭、rename、父目录同步;发布前失败清理临时文件并保留已有 pack。
|
|
12
|
+
|
|
13
|
+
v1 格式不变。`test/fixtures/durable-pack-v1.bin` 由本轮批量写入之前的串行写入器生成,SHA-256 为 `d11a4364cee3b347116cf8c3f6cc598baddc593b778f3ee81f24568546f57b4e`;测试验证旧文件可加载,并且新写入器对同一输入产生逐字节一致的文件。原有损坏 pack 拒绝加载、native restore 和 packed recovery 测试继续覆盖恢复流程。
|
|
14
|
+
|
|
15
|
+
本机 macOS arm64 基准以 `a02dd580` 加入相同计时后的串行写入为基线,再只替换写入策略进行对比。每个场景冷、热缓存各三次,下面为完整 `prepareDurableRestore` 的中位数,单位毫秒;不包含造数、capture、apply 和清理。冷缓存指新 SnapshotStore/RestoreEngine 的进程内缓存,不清空操作系统页缓存;热缓存指同实例再次准备相同 pack。
|
|
16
|
+
|
|
17
|
+
| 场景 | 串行写入 | 批量写入 | 变化 |
|
|
18
|
+
| --- | ---: | ---: | ---: |
|
|
19
|
+
| 10,001 个文件,每版本 64 字节,冷缓存 | 2,503 | 2,267 | 降低约 9.4% |
|
|
20
|
+
| 同上,热缓存 | 1,310 | 1,075 | 降低约 17.9% |
|
|
21
|
+
| 8 个文件,每版本 4 MiB,冷缓存 | 265 | 275 | 增加约 10 ms |
|
|
22
|
+
| 同上,热缓存 | 187 | 196 | 增加约 9 ms |
|
|
23
|
+
|
|
24
|
+
小文件的串行写入调用为 20,005 次,批量写入为 81 次;写入阶段从约 233–239 ms 降到约 6–11.5 ms。大文件从 19 次变为 17 次,没有证明性能改善。小文件三次完整准备耗时分别为:基线冷 `[2517, 2503, 2488]`、热 `[1311, 1278, 1310]`;批量冷 `[2321, 2196, 2267]`、热 `[1075, 1094, 1052]`。这些是本机合成基准,不设 CI 墙钟门槛,不等同于完整 undo/redo 的收益。
|
|
25
|
+
|
|
26
|
+
基准还输出结束时 RSS 和整个测试进程的峰值 RSS;后者包含造数、capture 和所有前序样本,不能当作单次 prepare 的峰值。本批次整体高水位约为基线 824 MiB、批量 932 MiB,未证明内存改善。本轮只限制单次写入规模,仍保留原有的整批 payload 持有和复制策略,未实现流式 pack 构建。
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
PI_UNDO_PACK_BENCHMARK=1 npx vitest run test/durable-pack-benchmark.test.ts --reporter=verbose
|
|
30
|
+
npx vitest run test/durable-pack.test.ts test/durable-pack-write.test.ts test/packed-recovery.test.ts test/native-restore.test.ts
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
### Scoped plan 的冷 blob 校验复用
|
|
34
|
+
|
|
35
|
+
plan 额外报告 `plan.validate`、`plan.complete`、`plan.paths`、`plan.diff` 四段计时,分别对应入口校验、对象完整性与预取、路径展开、差异与计划生成。它们属于 `prepare.plan` 内部阶段,不能重复累加。按需基准同时记录 Git 子命令的次数与累计耗时;新旧两侧查询可能并行,Git 累计耗时也不能直接当作 plan wall-time。
|
|
36
|
+
|
|
37
|
+
带 scope 的完整性检查现在一次性将去重后的 blob 分为两组:缓存命中的 blob 与根 tree 继续用 `cat-file --batch-check` 检查磁盘;冷 blob 用原有按字节和数量分批的 `cat-file --batch` 同时完成校验与预取。每个读取响应仍验证 object ID、blob 类型、tree 中记录的长度、内容边界、条目顺序和无多余字节。冷 blob 不再先做一次仅查询类型/大小的检查,然后再读取相同对象。没有 scope 的完整性检查保持原来的完整 batch-check。
|
|
38
|
+
|
|
39
|
+
分组后不再按缓存状态重新过滤读取列表,防止其他请求在窗口内填入缓存后跳过本次磁盘校验。热缓存中的内容不能掩盖磁盘 blob 丢失;失败仍抛出 `object_missing`。测试覆盖冷热缓存对象缺失、并发填充、重复 blob 去重、scope 与路径归属,以及 missing、错误类型/ID/长度、截断和额外输出。
|
|
40
|
+
|
|
41
|
+
后续独立对比以 `29bdc0c7` 加相同分阶段诊断为基线,在两个 worktree 顺序运行相同基准,每组各三次。这里只比较本轮 scoped 校验变化,不与上面的批量写入收益累加:
|
|
42
|
+
|
|
43
|
+
| 场景 | 本轮基线 | 校验复用后 |
|
|
44
|
+
| --- | ---: | ---: |
|
|
45
|
+
| 10,001 个小文件,冷缓存完整准备 | 2,386 ms | 2,079 ms(减少约 12.9%) |
|
|
46
|
+
| 同上,仅 plan | 1,763 ms | 1,456 ms(减少约 17.4%) |
|
|
47
|
+
| 10,001 个小文件,热缓存完整准备 | 1,160 ms | 1,168 ms |
|
|
48
|
+
| 8 个大文件,冷缓存完整准备 | 281 ms | 281 ms |
|
|
49
|
+
| 8 个大文件,热缓存完整准备 | 201 ms | 201 ms |
|
|
50
|
+
|
|
51
|
+
小文件冷缓存三次完整准备耗时分别为基线 `[2370, 2386, 2401]`、优化后 `[2144, 2079, 2061]`。冷缓存仍有两次根 tree batch-check 和十次分块 blob 读取;减少的是两份检查请求中重复查询的 20,002 个 blob,不是省掉这些 blob 的读取与验证。热缓存仍完整检查全部所需对象,没有证明提速。测试进程整体峰值 RSS 约为基线 897 MiB、优化后 904 MiB,未证明内存改善;这仍包含造数、capture 和前序样本。
|
|
52
|
+
|
|
53
|
+
### Tree 范围合并、blob 有界并发与 pack 私有副本
|
|
54
|
+
|
|
55
|
+
对 `6a00f83d` 的进一步分析将 tree 查询拆成路径匹配与大小查询:10,001 个小文件的单次 Git 查询,带全部路径和大小约 459 ms;去掉路径列表约 228 ms;全量只列 tree 不查大小约 10 ms。这里是独立 Git 查询,不包含 capture、JS 解析或完整准备。
|
|
56
|
+
|
|
57
|
+
本轮保留以下三项优化:
|
|
58
|
+
|
|
59
|
+
- 大 scope 先在公共父目录内执行只列名称的探测,仅将完全受 scope 覆盖的目录合并为一个 literal pathspec。非 Windows 从 4,096 条路径起尝试,Windows 从 512 条起尝试;较小 scope 保持原查询。探测输出按 scope 字节数限制,最少 64 KiB、最多 2 MiB。超限、普通 Git 失败或无效提示回退精确查询;取消、超时和未确认进程退出不触发回退。最终结果及路径归属缓存仍按原 scope 过滤,未探测的祖先不能作为合并目标。`ls-tree -l` 的大小查询继续保留,以维持后续读取的字节预算。
|
|
60
|
+
- blob 仍按最多 2,048 个对象(Windows 为 256 个)、16 MiB 分批。只有所有批次都不超过 8 MiB 时才允许两路并行,因此每个读取任务的在途 payload 总预算仍不超过 16 MiB。大批次与超限单文件保持串行;一批失败后等待其他在途任务结束,并停止调度剩余批次。4,096 对象批次的实验没有显示足以保留的额外收益。
|
|
61
|
+
- pack 在校验前取得输入的私有副本,编码、写入和内存驻留共享这份副本;对外读取文件内容仍返回副本。调用方在诊断回调或异步写入期间改写输入,不会改变磁盘 pack 或驻留内容。SHA-256 直接同步读取 Uint8Array 有效区间,去掉计算 checksum 前的整段复制。v1 格式和原子发布顺序保持不变。
|
|
62
|
+
|
|
63
|
+
相同基准在基线与最终实现的两个 worktree 中顺序运行,每个场景冷、热各三次,以下为毫秒中位数。冷缓存仍指新 SnapshotStore 的内存缓存;各轮历史结果不合并计算:
|
|
64
|
+
|
|
65
|
+
| 场景 | `6a00f83d` | 本轮最终实现 |
|
|
66
|
+
| --- | ---: | ---: |
|
|
67
|
+
| 10,001 个小文件,冷缓存准备 | 1,938 | 1,526(减少约 21.2%) |
|
|
68
|
+
| 同上,仅 plan | 1,346 | 951(减少约 29.3%) |
|
|
69
|
+
| 10,001 个小文件,热缓存准备 | 1,099 | 1,103 |
|
|
70
|
+
| 10,001 个文件仅修改 1 个,冷缓存准备 | 84 | 87 |
|
|
71
|
+
| 同上,热缓存准备 | 72 | 72 |
|
|
72
|
+
| 8 个大文件,冷缓存准备 | 262 | 258 |
|
|
73
|
+
| 同上,热缓存准备 | 191 | 185 |
|
|
74
|
+
|
|
75
|
+
小文件冷缓存三次为基线 `[1980, 1922, 1938]`、最终 `[1560, 1526, 1511]`。热缓存、稀疏修改和大文件没有证明稳定提速。未充分覆盖目录的稀疏大 scope 可能增加一次有界探测开销;超过探测预算时回退精确查询,不复用部分结果。
|
|
76
|
+
|
|
77
|
+
真实 Pi 1.3.0 SDK、10,000 个 ignored 依赖包场景也分别进行了三次 undo/redo,验证每次内容、历史状态和五次拓扑发现。耗时中位数如下:
|
|
78
|
+
|
|
79
|
+
| 场景 | undo 基线 → 最终 | redo 基线 → 最终 |
|
|
80
|
+
| --- | ---: | ---: |
|
|
81
|
+
| 修改 1 个文件 | 2,649 → 2,285 ms | 2,386 → 2,394 ms |
|
|
82
|
+
| 修改 100 个文件,首次对比 | 2,030 → 2,314 ms | 1,954 → 2,474 ms |
|
|
83
|
+
| 修改 100 个文件,反向运行顺序复测 | 2,343 → 1,894 ms | 2,468 → 1,939 ms |
|
|
84
|
+
|
|
85
|
+
100 文件首次 undo 原始样本为基线 `[2030, 2555, 1885]`、最终 `[2702, 2314, 1863]`;反向顺序复测为基线 `[2011, 2802, 2343]`、最终 `[2692, 1838, 1894]`。两轮结果方向不同,因此没有证明端到端稳定提速,也不把准备阶段的 21.2% 外推到完整 undo/redo。复测留下的末次操作诊断中,plan 为 52/51 ms,而 apply 为 1608/1099 ms;当前小 scope 的主要耗时与波动仍在其他恢复阶段。
|
|
86
|
+
|
|
87
|
+
独立内存基准每次新开测试进程,输入为 64 MiB payload,在阶段边界记录相对入口的内存增量,分别运行三次。额外 ArrayBuffer 观测高值由约 124 MiB 降至 64 MiB;额外 RSS 由约 123 MiB 降至 67 MiB。它们不是整个进程的瞬时峰值,也不包含 SnapshotStore capture。输入与私有 payload 仍整体驻留,本轮没有实现流式 pack 构建。
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
PI_UNDO_TREE_BENCHMARK=1 npx vitest run test/tree-query-benchmark.test.ts
|
|
91
|
+
PI_UNDO_PACK_MEMORY_BENCHMARK=1 npx vitest run test/durable-pack-memory-benchmark.test.ts
|
|
92
|
+
PI_UNDO_PACK_BENCHMARK=1 npx vitest run test/durable-pack-benchmark.test.ts
|
|
93
|
+
PI_UNDO_LARGE_WORKSPACE=1 npx vitest run test/large-workspace.test.ts -t '10000 个依赖包'
|
|
94
|
+
```
|
|
95
|
+
|
|
5
96
|
## 性能诊断与批次调优
|
|
6
97
|
|
|
7
98
|
metadata inspect 默认每批最多处理 2,048 条路径,并受 1 MiB 路径请求字节上限约束。可使用 `PI_UNDO_METADATA_BATCH_MAX_PATHS=4096` 或 `8192` 做受控基准测试;有效范围为 1 到 8,192,超出范围会拒绝本次 capture。路径字节上限不会被环境变量覆盖,用于限制长路径工作区的单批请求规模。
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
package/package.json
CHANGED
package/src/controller.ts
CHANGED
|
@@ -250,6 +250,10 @@ export class UndoControllerImpl implements UndoController {
|
|
|
250
250
|
private lastSettlementDiagnostic: SettlementDiagnostic | undefined;
|
|
251
251
|
private promptDeferralInFlight = false;
|
|
252
252
|
private lastSafetyManifestId: ManifestId | null = null;
|
|
253
|
+
/** restore 成功后记录的 target manifest;settled capture 可用它替代 warmUpManifest 作为 baseline。 */
|
|
254
|
+
private lastAppliedTargetManifest: SnapshotManifest | undefined;
|
|
255
|
+
/** restore 完成时的 performance.now()(ms);2s racy-clean 窗口内不使用 lastAppliedTargetManifest。 */
|
|
256
|
+
private lastAppliedTargetAt: number | undefined;
|
|
253
257
|
private lastCaptureFailed = false;
|
|
254
258
|
private lastCaptureFailureMessage: string | undefined;
|
|
255
259
|
private warmUpInFlight: Promise<void> | undefined;
|
|
@@ -572,7 +576,11 @@ export class UndoControllerImpl implements UndoController {
|
|
|
572
576
|
|
|
573
577
|
private async settleRun(run?: RunCompletion): Promise<void> {
|
|
574
578
|
const staged = this.staged;
|
|
579
|
+
const appliedTargetManifest = this.lastAppliedTargetManifest;
|
|
580
|
+
const appliedTargetAt = this.lastAppliedTargetAt;
|
|
575
581
|
this.staged = undefined;
|
|
582
|
+
this.lastAppliedTargetManifest = undefined;
|
|
583
|
+
this.lastAppliedTargetAt = undefined;
|
|
576
584
|
if (this.disposed || this.locked || staged === undefined) return;
|
|
577
585
|
if (staged.startEntryId === undefined || staged.startEntryId === null) {
|
|
578
586
|
// Pi 没有提供已落盘的 start entry ID,不能把后续 assistant 输出归属到该 checkpoint。
|
|
@@ -611,7 +619,12 @@ export class UndoControllerImpl implements UndoController {
|
|
|
611
619
|
// 撕裂捕获(并发写撞上断言窗口)是暂态的:短暂退避后重试一次,成功则不清空历史。
|
|
612
620
|
const after = await measure("settled.capture", () =>
|
|
613
621
|
this.captureSettledBaselineWithRetry(
|
|
614
|
-
|
|
622
|
+
// racy-clean 窗口(2s)内文件 ctime 未稳定,isBaselineFresh 必然失败且会多跑一轮
|
|
623
|
+
// collectVisibleLeaves;只有超过 2s 后才能从 target manifest 受益。
|
|
624
|
+
(appliedTargetManifest !== undefined && appliedTargetAt !== undefined &&
|
|
625
|
+
performance.now() - appliedTargetAt >= 2_000)
|
|
626
|
+
? appliedTargetManifest
|
|
627
|
+
: staged.before,
|
|
615
628
|
(diagnostic) => captureDiagnostics.push(diagnostic),
|
|
616
629
|
recordCaptureTiming,
|
|
617
630
|
));
|
|
@@ -1023,6 +1036,8 @@ export class UndoControllerImpl implements UndoController {
|
|
|
1023
1036
|
await this.dependencies.journal.markCommitted(descriptor.opId);
|
|
1024
1037
|
});
|
|
1025
1038
|
this.lastSafetyManifestId = rollback.manifestId;
|
|
1039
|
+
this.lastAppliedTargetManifest = target;
|
|
1040
|
+
this.lastAppliedTargetAt = performance.now();
|
|
1026
1041
|
return done(this.advanceHistory(action, checkpoint, { code: "ok", changedFiles: applied.verifiedPaths }));
|
|
1027
1042
|
} catch (error) {
|
|
1028
1043
|
if (this.recordUnsafeExit(error)) return done(this.recoveryResult());
|
package/src/durable-pack.ts
CHANGED
|
@@ -5,12 +5,15 @@ import { dirname, join, resolve } from "node:path";
|
|
|
5
5
|
import { fsyncDirectory, fsyncFile } from "./atomic-fs.ts";
|
|
6
6
|
import { canonicalJson, checksum } from "./encoding.ts";
|
|
7
7
|
import type { MutationJournal } from "./mutation-journal.ts";
|
|
8
|
+
import { checkOperation } from "./operation-context.ts";
|
|
8
9
|
import { assertNoSymlinkEscape, relativeSafePath } from "./path-safety.ts";
|
|
9
10
|
import { fingerprintAbsent, fingerprintBytes, fingerprintFile, fingerprintLeaf, fingerprintSymlink } from "./quarantine.ts";
|
|
10
11
|
|
|
11
12
|
const PACK_FILE = "durable-pack-v1.bin";
|
|
12
13
|
const MAGIC = Buffer.from("PIUNDO-PACK-V1\0", "ascii");
|
|
13
14
|
const MAX_HEADER_BYTES = 16 * 1024 * 1024;
|
|
15
|
+
const WRITE_BATCH_MAX_BYTES = 4 * 1024 * 1024;
|
|
16
|
+
const WRITE_BATCH_MAX_BUFFERS = 256;
|
|
14
17
|
const FINALIZE_CONCURRENCY = 32;
|
|
15
18
|
|
|
16
19
|
export type DurableLeafInput =
|
|
@@ -33,6 +36,29 @@ export interface DurablePackInput {
|
|
|
33
36
|
readonly entries: readonly DurablePackEntryInput[];
|
|
34
37
|
}
|
|
35
38
|
|
|
39
|
+
export interface DurablePackPreparationOptions {
|
|
40
|
+
/** 阶段计时不包含路径或内容;回调异常不影响 pack 生成。 */
|
|
41
|
+
readonly onTiming?: (phase: string, durationMs: number) => void;
|
|
42
|
+
/** 成功生成 pack 后报告数量;writeCalls 是实际发起的写入调用次数(含短写重试)。 */
|
|
43
|
+
readonly onDiagnostics?: (diagnostic: DurablePackWriteDiagnostic) => void;
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
export interface DurablePackWriteDiagnostic {
|
|
47
|
+
readonly entries: number;
|
|
48
|
+
readonly fileVariants: number;
|
|
49
|
+
readonly payloadBytes: number;
|
|
50
|
+
readonly packBytes: number;
|
|
51
|
+
readonly writeCalls: number;
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
export function reportDurablePackTiming(
|
|
55
|
+
options: DurablePackPreparationOptions,
|
|
56
|
+
phase: string,
|
|
57
|
+
startedAt: number,
|
|
58
|
+
): void {
|
|
59
|
+
try { options.onTiming?.(phase, performance.now() - startedAt); } catch { /* 诊断失败不能改变事务结果。 */ }
|
|
60
|
+
}
|
|
61
|
+
|
|
36
62
|
interface PackedVariantHeader {
|
|
37
63
|
readonly kind: DurableLeafInput["kind"];
|
|
38
64
|
readonly fingerprint: string;
|
|
@@ -118,6 +144,10 @@ export class DurablePack {
|
|
|
118
144
|
return this.entries.get(path)?.targetFingerprint;
|
|
119
145
|
}
|
|
120
146
|
|
|
147
|
+
withStoragePath(storagePath: string): DurablePack {
|
|
148
|
+
return new DurablePack(this.opId, this.planDigest, storagePath, this.packChecksum, this.entries);
|
|
149
|
+
}
|
|
150
|
+
|
|
121
151
|
leaf(path: string, fingerprint: string): DurableLeaf | undefined {
|
|
122
152
|
const leaf = this.entries.get(path)?.variants.get(fingerprint);
|
|
123
153
|
if (leaf === undefined) return undefined;
|
|
@@ -132,10 +162,17 @@ export function durablePackPath(journal: MutationJournal): string {
|
|
|
132
162
|
export async function createDurablePack(
|
|
133
163
|
journal: MutationJournal,
|
|
134
164
|
input: DurablePackInput,
|
|
165
|
+
options: DurablePackPreparationOptions = {},
|
|
135
166
|
): Promise<DurablePack> {
|
|
136
|
-
|
|
137
|
-
|
|
167
|
+
const validationStarted = performance.now();
|
|
168
|
+
checkOperation();
|
|
169
|
+
const opId = input.opId;
|
|
170
|
+
const planDigest = input.planDigest;
|
|
171
|
+
if (opId !== journal.operationId) throw new Error("durable pack opId 与 mutation journal 不匹配");
|
|
172
|
+
assertDigest(planDigest, "planDigest");
|
|
138
173
|
const entries = canonicalEntries(input.entries);
|
|
174
|
+
reportDurablePackTiming(options, "pack.validate", validationStarted);
|
|
175
|
+
const encodingStarted = performance.now();
|
|
139
176
|
const payloads: Buffer[] = [];
|
|
140
177
|
let offset = 0;
|
|
141
178
|
const headerEntries: PackedEntryHeader[] = entries.map((entry) => ({
|
|
@@ -147,7 +184,8 @@ export async function createDurablePack(
|
|
|
147
184
|
variants: entry.variants.map((variant): PackedVariantHeader => {
|
|
148
185
|
if (variant.kind === "absent") return variant;
|
|
149
186
|
if (variant.kind === "symlink") return variant;
|
|
150
|
-
|
|
187
|
+
// canonicalEntries 已取得私有副本;编码、写入和驻留共享该副本。
|
|
188
|
+
const bytes = Buffer.from(variant.bytes.buffer, variant.bytes.byteOffset, variant.bytes.byteLength);
|
|
151
189
|
payloads.push(bytes);
|
|
152
190
|
const header = {
|
|
153
191
|
kind: "file" as const,
|
|
@@ -163,8 +201,8 @@ export async function createDurablePack(
|
|
|
163
201
|
}));
|
|
164
202
|
const headerPayload: PackHeaderPayload = {
|
|
165
203
|
schemaVersion: 1,
|
|
166
|
-
opId
|
|
167
|
-
planDigest
|
|
204
|
+
opId,
|
|
205
|
+
planDigest,
|
|
168
206
|
entries: headerEntries,
|
|
169
207
|
};
|
|
170
208
|
const header: PackHeader = {
|
|
@@ -177,30 +215,51 @@ export async function createDurablePack(
|
|
|
177
215
|
lengthBytes.writeUInt32BE(headerBytes.length);
|
|
178
216
|
const packPath = durablePackPath(journal);
|
|
179
217
|
const temporary = join(dirname(packPath), `.${PACK_FILE}.${process.pid}.${randomBytes(8).toString("hex")}.tmp`);
|
|
218
|
+
reportDurablePackTiming(options, "pack.encode", encodingStarted);
|
|
219
|
+
let writeCalls = 0;
|
|
180
220
|
let handle;
|
|
181
221
|
try {
|
|
222
|
+
const writeStarted = performance.now();
|
|
182
223
|
handle = await open(temporary, "wx", 0o600);
|
|
183
|
-
await writeAll(handle, MAGIC);
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
224
|
+
writeCalls = await writeAll(handle, [MAGIC, lengthBytes, headerBytes, ...payloads]);
|
|
225
|
+
reportDurablePackTiming(options, "pack.write", writeStarted);
|
|
226
|
+
checkOperation();
|
|
227
|
+
const syncStarted = performance.now();
|
|
187
228
|
await handle.sync();
|
|
188
229
|
await handle.close();
|
|
189
230
|
handle = undefined;
|
|
231
|
+
reportDurablePackTiming(options, "pack.sync", syncStarted);
|
|
232
|
+
const publishStarted = performance.now();
|
|
233
|
+
checkOperation();
|
|
190
234
|
await rename(temporary, packPath);
|
|
191
235
|
await fsyncDirectory(dirname(packPath));
|
|
236
|
+
reportDurablePackTiming(options, "pack.publish", publishStarted);
|
|
192
237
|
} catch (error) {
|
|
193
238
|
await handle?.close().catch(() => {});
|
|
194
239
|
await rm(temporary, { force: true }).catch(() => {});
|
|
195
240
|
throw error;
|
|
196
241
|
}
|
|
242
|
+
const checksumStarted = performance.now();
|
|
197
243
|
const digest = createHash("sha256");
|
|
198
244
|
digest.update(MAGIC);
|
|
199
245
|
digest.update(lengthBytes);
|
|
200
246
|
digest.update(headerBytes);
|
|
201
247
|
for (const payload of payloads) digest.update(payload);
|
|
202
248
|
const packChecksum = digest.digest("hex");
|
|
203
|
-
|
|
249
|
+
reportDurablePackTiming(options, "pack.checksum", checksumStarted);
|
|
250
|
+
const residentStarted = performance.now();
|
|
251
|
+
const pack = durablePackFromOwnedEntries(opId, planDigest, packPath, packChecksum, entries);
|
|
252
|
+
reportDurablePackTiming(options, "pack.resident", residentStarted);
|
|
253
|
+
try {
|
|
254
|
+
options.onDiagnostics?.({
|
|
255
|
+
entries: entries.length,
|
|
256
|
+
fileVariants: payloads.length,
|
|
257
|
+
payloadBytes: offset,
|
|
258
|
+
packBytes: MAGIC.length + lengthBytes.length + headerBytes.length + offset,
|
|
259
|
+
writeCalls,
|
|
260
|
+
});
|
|
261
|
+
} catch { /* 诊断失败不能改变事务结果。 */ }
|
|
262
|
+
return pack;
|
|
204
263
|
}
|
|
205
264
|
|
|
206
265
|
export async function loadDurablePack(
|
|
@@ -282,16 +341,36 @@ export async function hasDurablePack(journal: MutationJournal): Promise<boolean>
|
|
|
282
341
|
}
|
|
283
342
|
}
|
|
284
343
|
|
|
344
|
+
export async function publishResidentDurablePack(
|
|
345
|
+
cached: DurablePack,
|
|
346
|
+
journal: MutationJournal,
|
|
347
|
+
expectedPlanDigest: string,
|
|
348
|
+
): Promise<DurablePack> {
|
|
349
|
+
if (cached.planDigest !== expectedPlanDigest) throw new Error("durable cache planDigest 不匹配");
|
|
350
|
+
const target = durablePackPath(journal);
|
|
351
|
+
await rm(target, { force: true });
|
|
352
|
+
try {
|
|
353
|
+
await link(cached.storagePath, target);
|
|
354
|
+
await fsyncDirectory(dirname(target));
|
|
355
|
+
} catch (error) {
|
|
356
|
+
if (!hasErrorCode(error, "EXDEV")) throw error;
|
|
357
|
+
await copyFile(cached.storagePath, target);
|
|
358
|
+
await fsyncFile(target);
|
|
359
|
+
await fsyncDirectory(dirname(target));
|
|
360
|
+
}
|
|
361
|
+
return cached.withStoragePath(target);
|
|
362
|
+
}
|
|
363
|
+
|
|
285
364
|
export async function publishCachedDurablePack(
|
|
286
365
|
cachedPath: string,
|
|
287
366
|
journal: MutationJournal,
|
|
288
367
|
expectedPlanDigest: string,
|
|
289
368
|
expectedPackChecksum: string,
|
|
290
369
|
): Promise<DurablePack> {
|
|
291
|
-
const target = durablePackPath(journal);
|
|
292
370
|
if (checksum(await readFile(cachedPath)) !== expectedPackChecksum) {
|
|
293
371
|
throw new Error("durable cache pack checksum 不匹配");
|
|
294
372
|
}
|
|
373
|
+
const target = durablePackPath(journal);
|
|
295
374
|
await rm(target, { force: true });
|
|
296
375
|
try {
|
|
297
376
|
await link(cachedPath, target);
|
|
@@ -424,7 +503,10 @@ function canonicalEntries(
|
|
|
424
503
|
if (entry.targetArtifact !== null) assertArtifact(entry.path, entry.targetArtifact, "target");
|
|
425
504
|
assertDigest(entry.sourceFingerprint, "sourceFingerprint");
|
|
426
505
|
if (entry.targetFingerprint !== null) assertDigest(entry.targetFingerprint, "targetFingerprint");
|
|
427
|
-
|
|
506
|
+
// 在校验及任何回调、await 之前取得所有权;调用方之后改写输入不影响 pack。
|
|
507
|
+
const variants = entry.variants.map((variant): DurableLeafInput => variant.kind === "file"
|
|
508
|
+
? { ...variant, bytes: new Uint8Array(variant.bytes) }
|
|
509
|
+
: { ...variant }).sort((left, right) => left.fingerprint.localeCompare(right.fingerprint));
|
|
428
510
|
const fingerprints = new Set<string>();
|
|
429
511
|
for (const variant of variants) {
|
|
430
512
|
assertDigest(variant.fingerprint, "fingerprint");
|
|
@@ -519,7 +601,7 @@ function assertPackHeader(value: unknown): PackHeader {
|
|
|
519
601
|
return { ...payload, checksum: record.checksum };
|
|
520
602
|
}
|
|
521
603
|
|
|
522
|
-
function
|
|
604
|
+
function durablePackFromOwnedEntries(
|
|
523
605
|
opId: string,
|
|
524
606
|
planDigest: string,
|
|
525
607
|
storagePath: string,
|
|
@@ -538,24 +620,54 @@ function durablePackFromInput(
|
|
|
538
620
|
targetArtifact: entry.targetArtifact,
|
|
539
621
|
sourceFingerprint: entry.sourceFingerprint,
|
|
540
622
|
targetFingerprint: entry.targetFingerprint,
|
|
541
|
-
variants: new Map(entry.variants.map((variant) => [
|
|
542
|
-
variant.fingerprint,
|
|
543
|
-
variant.kind === "file"
|
|
544
|
-
? { ...variant, bytes: new Uint8Array(variant.bytes) }
|
|
545
|
-
: variant,
|
|
546
|
-
])),
|
|
623
|
+
variants: new Map(entry.variants.map((variant) => [variant.fingerprint, variant])),
|
|
547
624
|
},
|
|
548
625
|
])),
|
|
549
626
|
);
|
|
550
627
|
}
|
|
551
628
|
|
|
552
|
-
async function writeAll(handle: FileHandle,
|
|
553
|
-
let
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
629
|
+
async function writeAll(handle: FileHandle, buffers: readonly Uint8Array[]): Promise<number> {
|
|
630
|
+
let calls = 0;
|
|
631
|
+
let batch: Uint8Array[] = [];
|
|
632
|
+
let batchBytes = 0;
|
|
633
|
+
const flush = async (): Promise<void> => {
|
|
634
|
+
let remaining = batch;
|
|
635
|
+
let remainingBytes = batchBytes;
|
|
636
|
+
while (remaining.length > 0) {
|
|
637
|
+
checkOperation();
|
|
638
|
+
calls += 1;
|
|
639
|
+
const { bytesWritten } = await handle.writev(remaining);
|
|
640
|
+
if (!Number.isSafeInteger(bytesWritten) || bytesWritten <= 0 || bytesWritten > remainingBytes) {
|
|
641
|
+
throw new Error("durable pack 写入未推进或返回无效长度");
|
|
642
|
+
}
|
|
643
|
+
remainingBytes -= bytesWritten;
|
|
644
|
+
// 短写可能停在任意 buffer 内,也可能跨过多个 buffer。
|
|
645
|
+
let consumed = bytesWritten;
|
|
646
|
+
let index = 0;
|
|
647
|
+
while (index < remaining.length && consumed >= remaining[index]!.byteLength) {
|
|
648
|
+
consumed -= remaining[index]!.byteLength;
|
|
649
|
+
index += 1;
|
|
650
|
+
}
|
|
651
|
+
remaining = remaining.slice(index);
|
|
652
|
+
if (consumed > 0) remaining[0] = remaining[0]!.subarray(consumed);
|
|
653
|
+
}
|
|
654
|
+
batch = [];
|
|
655
|
+
batchBytes = 0;
|
|
656
|
+
};
|
|
657
|
+
for (const bytes of buffers) {
|
|
658
|
+
let offset = 0;
|
|
659
|
+
while (offset < bytes.byteLength) {
|
|
660
|
+
const length = Math.min(bytes.byteLength - offset, WRITE_BATCH_MAX_BYTES - batchBytes);
|
|
661
|
+
batch.push(bytes.subarray(offset, offset + length));
|
|
662
|
+
batchBytes += length;
|
|
663
|
+
offset += length;
|
|
664
|
+
if (batchBytes === WRITE_BATCH_MAX_BYTES || batch.length === WRITE_BATCH_MAX_BUFFERS) {
|
|
665
|
+
await flush();
|
|
666
|
+
}
|
|
667
|
+
}
|
|
558
668
|
}
|
|
669
|
+
await flush();
|
|
670
|
+
return calls;
|
|
559
671
|
}
|
|
560
672
|
|
|
561
673
|
function assertArtifact(path: string, artifact: string, role: "source" | "target"): void {
|
package/src/encoding.ts
CHANGED
|
@@ -40,8 +40,8 @@ export function canonicalJson(value: unknown): string {
|
|
|
40
40
|
}
|
|
41
41
|
|
|
42
42
|
export function checksum(value: string | Uint8Array): string {
|
|
43
|
-
|
|
44
|
-
return createHash("sha256").update(
|
|
43
|
+
// update 同步消费 Uint8Array 的有效区间,无需再复制完整 payload。
|
|
44
|
+
return createHash("sha256").update(value).digest("hex");
|
|
45
45
|
}
|
|
46
46
|
|
|
47
47
|
export function sameWorkspaceSnapshot(left: SnapshotManifest, right: SnapshotManifest): boolean {
|
package/src/pi-runtime.ts
CHANGED
|
@@ -61,7 +61,12 @@ async function initializePiUndoRuntime(context: ExtensionContext, pi: ExtensionA
|
|
|
61
61
|
if (topology.workspaceIdentity !== initialTopology.workspaceIdentity) {
|
|
62
62
|
throw new Error("workspace_identity_changed: workspace root canonical path changed; run /undo-recover");
|
|
63
63
|
}
|
|
64
|
-
return store.capture(topology, scopePaths, {
|
|
64
|
+
return store.capture(topology, scopePaths, {
|
|
65
|
+
topologyAlreadyValidated: true,
|
|
66
|
+
// captureWithWorkspaceLock 已持有工作区独占锁,且此处刚完成 discover;
|
|
67
|
+
// post-capture topology 校验在锁内是冗余的,跳过以节省一次全量目录扫描。
|
|
68
|
+
skipPostTopologyCheck: true,
|
|
69
|
+
});
|
|
65
70
|
};
|
|
66
71
|
const captureBaseline = async (
|
|
67
72
|
baseline: SnapshotManifest,
|
|
@@ -71,7 +76,7 @@ async function initializePiUndoRuntime(context: ExtensionContext, pi: ExtensionA
|
|
|
71
76
|
const discoveryStartedAt = performance.now();
|
|
72
77
|
let topology: RootTopology;
|
|
73
78
|
try {
|
|
74
|
-
topology = await discovery.discover(context.cwd);
|
|
79
|
+
topology = await discovery.discover(context.cwd, "settled-capture");
|
|
75
80
|
} finally {
|
|
76
81
|
try { onTiming?.("settled.rootDiscovery", performance.now() - discoveryStartedAt); } catch { /* diagnostics must not affect capture */ }
|
|
77
82
|
}
|
|
@@ -80,6 +85,9 @@ async function initializePiUndoRuntime(context: ExtensionContext, pi: ExtensionA
|
|
|
80
85
|
}
|
|
81
86
|
return store.captureBaseline(topology, baseline, undefined, {
|
|
82
87
|
topologyAlreadyValidated: true,
|
|
88
|
+
// captureBaselineWithWorkspaceLock 已持有工作区独占锁,且此处刚完成 discover;
|
|
89
|
+
// post-capture topology 校验在锁内是冗余的,跳过以节省一次全量目录扫描。
|
|
90
|
+
skipPostTopologyCheck: true,
|
|
83
91
|
onDiagnostics,
|
|
84
92
|
onTiming,
|
|
85
93
|
});
|
|
@@ -284,6 +292,12 @@ async function initializePiUndoRuntime(context: ExtensionContext, pi: ExtensionA
|
|
|
284
292
|
opId: operation.opId,
|
|
285
293
|
mutationJournal: journal.mutationJournal(operation.opId),
|
|
286
294
|
deferDurability: true,
|
|
295
|
+
onTiming: (phase, durationMs) => {
|
|
296
|
+
try {
|
|
297
|
+
phases.push({ phase, elapsedMs: Math.round(performance.now() - operationStarted) });
|
|
298
|
+
void durationMs;
|
|
299
|
+
} catch { /* 诊断回调不影响恢复。 */ }
|
|
300
|
+
},
|
|
287
301
|
}),
|
|
288
302
|
recoverPending: async () => {
|
|
289
303
|
await waitForFinalization();
|
package/src/restore-engine.ts
CHANGED
|
@@ -11,7 +11,10 @@ import {
|
|
|
11
11
|
hasDurablePack,
|
|
12
12
|
loadDurablePack,
|
|
13
13
|
publishCachedDurablePack,
|
|
14
|
+
publishResidentDurablePack,
|
|
14
15
|
removeDurablePack,
|
|
16
|
+
reportDurablePackTiming,
|
|
17
|
+
type DurablePackPreparationOptions,
|
|
15
18
|
type DurableLeafInput,
|
|
16
19
|
type DurablePack,
|
|
17
20
|
type DurablePackEntryInput,
|
|
@@ -67,7 +70,7 @@ export interface RestoreResult {
|
|
|
67
70
|
}
|
|
68
71
|
|
|
69
72
|
export interface RestoreEngine {
|
|
70
|
-
plan(current: SnapshotManifest, target: SnapshotManifest, scopePaths?: readonly string[]): Promise<RestorePlan>;
|
|
73
|
+
plan(current: SnapshotManifest, target: SnapshotManifest, scopePaths?: readonly string[], options?: DurablePackPreparationOptions): Promise<RestorePlan>;
|
|
71
74
|
apply(plan: RestorePlan, target: SnapshotManifest, options?: RestoreApplyOptions): Promise<RestoreResult>;
|
|
72
75
|
}
|
|
73
76
|
|
|
@@ -76,6 +79,8 @@ export interface RestoreApplyOptions {
|
|
|
76
79
|
readonly mutationJournal: MutationJournal;
|
|
77
80
|
readonly forceTargetArtifactSync?: boolean;
|
|
78
81
|
readonly deferDurability?: boolean;
|
|
82
|
+
/** 报告 apply 内部阶段耗时;回调异常不影响恢复结果。 */
|
|
83
|
+
readonly onTiming?: (phase: string, durationMs: number) => void;
|
|
79
84
|
}
|
|
80
85
|
|
|
81
86
|
export interface RestoreEngineOptions {
|
|
@@ -122,6 +127,12 @@ interface VisibleSubsetCheck {
|
|
|
122
127
|
* 枚举结束后的校验仍然执行,所以该窗口内的漂移依然被拒绝。
|
|
123
128
|
*/
|
|
124
129
|
readonly topologyValidated: boolean;
|
|
130
|
+
/**
|
|
131
|
+
* 调用方持有工作区独占锁、且枚举前已通过 discover 确认 topology 未变化时为 true。
|
|
132
|
+
* 允许跳过 listVisibleLeafPaths 结束后的二次 topology 校验(visible-paths-post 扫描),
|
|
133
|
+
* 以避免对整个工作区进行冗余全量目录枚举。
|
|
134
|
+
*/
|
|
135
|
+
readonly skipPostTopologyCheck?: boolean;
|
|
125
136
|
readonly extraExclusions?: readonly string[];
|
|
126
137
|
readonly scopePaths?: readonly string[];
|
|
127
138
|
readonly ownedPaths?: readonly (ReadonlyMap<string, OwnedPath> | undefined)[];
|
|
@@ -189,6 +200,7 @@ export class RestoreEngine {
|
|
|
189
200
|
private readonly preparedPlans = new Map<string, PreparedRestorePlan>();
|
|
190
201
|
private readonly durablePackCache = new Map<string, CachedDurablePack>();
|
|
191
202
|
private readonly durablePackByPair = new Map<string, CachedDurablePair>();
|
|
203
|
+
private readonly verifiedDurablePacks = new Set<string>();
|
|
192
204
|
private readonly durableIndexLock = new WorkspaceLock({
|
|
193
205
|
leaseMs: 10_000,
|
|
194
206
|
retryMs: 25,
|
|
@@ -230,22 +242,30 @@ export class RestoreEngine {
|
|
|
230
242
|
current: SnapshotManifest,
|
|
231
243
|
target: SnapshotManifest,
|
|
232
244
|
scopePaths?: readonly string[],
|
|
245
|
+
options: DurablePackPreparationOptions = {},
|
|
233
246
|
): Promise<RestorePlan> {
|
|
247
|
+
const validationStarted = performance.now();
|
|
234
248
|
assertManifest(current);
|
|
235
249
|
assertManifest(target);
|
|
236
250
|
const scope = scopePaths === undefined ? undefined : this.canonicalScope(scopePaths);
|
|
237
251
|
const canonicalScopePaths = scope === undefined ? undefined : [...scope];
|
|
238
252
|
assertCompatibleManifests(current, target, scope);
|
|
239
253
|
const isScopedPath = (path: string): boolean => scope === undefined || scope.has(path);
|
|
254
|
+
reportDurablePackTiming(options, "plan.validate", validationStarted);
|
|
255
|
+
const completeStarted = performance.now();
|
|
240
256
|
await allCompleted([
|
|
241
257
|
this.store.assertComplete(current.manifestId, canonicalScopePaths),
|
|
242
258
|
this.store.assertComplete(target.manifestId, canonicalScopePaths),
|
|
243
259
|
]);
|
|
260
|
+
reportDurablePackTiming(options, "plan.complete", completeStarted);
|
|
244
261
|
|
|
262
|
+
const pathsStarted = performance.now();
|
|
245
263
|
const [currentPaths, targetPaths] = await allCompleted([
|
|
246
264
|
this.readOwnedPaths(current, canonicalScopePaths),
|
|
247
265
|
this.readOwnedPaths(target, canonicalScopePaths),
|
|
248
266
|
]);
|
|
267
|
+
reportDurablePackTiming(options, "plan.paths", pathsStarted);
|
|
268
|
+
const diffStarted = performance.now();
|
|
249
269
|
const targetIgnoredProof = new IgnoredProofIndex(ignoredWorkspacePaths(target));
|
|
250
270
|
const deleteByRoot = new Map<string, string[]>();
|
|
251
271
|
const writeByRoot = new Map<string, string[]>();
|
|
@@ -294,6 +314,7 @@ export class RestoreEngine {
|
|
|
294
314
|
planDigest: checksum(canonicalJson(semanticPlan)),
|
|
295
315
|
};
|
|
296
316
|
this.rememberPreparedPlan(plan, currentPaths, targetPaths);
|
|
317
|
+
reportDurablePackTiming(options, "plan.diff", diffStarted);
|
|
297
318
|
return plan;
|
|
298
319
|
}
|
|
299
320
|
|
|
@@ -343,6 +364,7 @@ export class RestoreEngine {
|
|
|
343
364
|
false,
|
|
344
365
|
);
|
|
345
366
|
}
|
|
367
|
+
this.verifiedDurablePacks.add(`${pairKey}:${cached.planDigest}`);
|
|
346
368
|
return true;
|
|
347
369
|
} catch (error) {
|
|
348
370
|
rethrowOperationFailure(error);
|
|
@@ -354,9 +376,12 @@ export class RestoreEngine {
|
|
|
354
376
|
current: SnapshotManifest,
|
|
355
377
|
target: SnapshotManifest,
|
|
356
378
|
scopePaths: readonly string[],
|
|
379
|
+
options: DurablePackPreparationOptions = {},
|
|
357
380
|
): Promise<void> {
|
|
358
|
-
const
|
|
381
|
+
const started = performance.now();
|
|
382
|
+
const plan = await this.plan(current, target, scopePaths, options);
|
|
359
383
|
const prepared = this.takePreparedPlan(plan);
|
|
384
|
+
reportDurablePackTiming(options, "prepare.plan", started);
|
|
360
385
|
if (prepared === undefined || !this.canUseNativeFilePlan(plan, prepared.currentPaths, prepared.targetPaths)) return;
|
|
361
386
|
const cacheRoot = await this.store.durableCacheDirectory();
|
|
362
387
|
const cacheOpId = `cache-${plan.planDigest}`;
|
|
@@ -375,8 +400,10 @@ export class RestoreEngine {
|
|
|
375
400
|
prepared.targetPaths,
|
|
376
401
|
plan,
|
|
377
402
|
cacheOpId,
|
|
403
|
+
options,
|
|
378
404
|
),
|
|
379
|
-
});
|
|
405
|
+
}, options);
|
|
406
|
+
const pinStarted = performance.now();
|
|
380
407
|
const pinReason = `durable-cache:${plan.planDigest}`;
|
|
381
408
|
const pinned: ManifestId[] = [];
|
|
382
409
|
try {
|
|
@@ -389,6 +416,8 @@ export class RestoreEngine {
|
|
|
389
416
|
await rm(dirname(pack.storagePath), { recursive: true, force: true }).catch(() => {});
|
|
390
417
|
throw error;
|
|
391
418
|
}
|
|
419
|
+
reportDurablePackTiming(options, "prepare.pin", pinStarted);
|
|
420
|
+
const indexStarted = performance.now();
|
|
392
421
|
await this.rememberDurablePack(
|
|
393
422
|
plan.planDigest,
|
|
394
423
|
current,
|
|
@@ -398,6 +427,8 @@ export class RestoreEngine {
|
|
|
398
427
|
pinReason,
|
|
399
428
|
);
|
|
400
429
|
this.rememberPreparedPlan(plan, prepared.currentPaths, prepared.targetPaths);
|
|
430
|
+
reportDurablePackTiming(options, "prepare.index", indexStarted);
|
|
431
|
+
reportDurablePackTiming(options, "prepare.total", started);
|
|
401
432
|
}
|
|
402
433
|
|
|
403
434
|
private async rememberDurablePack(
|
|
@@ -802,6 +833,7 @@ export class RestoreEngine {
|
|
|
802
833
|
}
|
|
803
834
|
|
|
804
835
|
let topologyBefore: RootTopology;
|
|
836
|
+
const t0 = performance.now();
|
|
805
837
|
try {
|
|
806
838
|
if (await this.assertWorkspaceRootIdentity() !== current.workspaceIdentity) {
|
|
807
839
|
throw new Error("restore workspace root 必须使用 canonical identity");
|
|
@@ -811,6 +843,8 @@ export class RestoreEngine {
|
|
|
811
843
|
} catch (error) {
|
|
812
844
|
rethrowOperationFailure(error);
|
|
813
845
|
return { code: "restore_failed_safe", verifiedPaths: 0, totalPaths: 0 };
|
|
846
|
+
} finally {
|
|
847
|
+
reportRestoreTiming(options.onTiming, "apply.topologyPre", t0);
|
|
814
848
|
}
|
|
815
849
|
const [currentPaths, targetPaths] = prepared === undefined
|
|
816
850
|
? await allCompleted([
|
|
@@ -829,6 +863,7 @@ export class RestoreEngine {
|
|
|
829
863
|
} else if ((await options.mutationJournal.activeArtifacts()).size > 0) {
|
|
830
864
|
return { code: "recovery_required", verifiedPaths: 0, totalPaths: currentPaths.size };
|
|
831
865
|
}
|
|
866
|
+
const t1 = performance.now();
|
|
832
867
|
try {
|
|
833
868
|
// 兼容恢复路径会先在此窗口内补偿 pending mutations,因此只有非补偿路径可以跳过枚举入口的重复发现。
|
|
834
869
|
await this.assertCompleteVisibleSubset(
|
|
@@ -844,6 +879,8 @@ export class RestoreEngine {
|
|
|
844
879
|
} catch (error) {
|
|
845
880
|
rethrowOperationFailure(error);
|
|
846
881
|
return { code: "restore_failed_safe", verifiedPaths: 0, totalPaths: 0 };
|
|
882
|
+
} finally {
|
|
883
|
+
reportRestoreTiming(options.onTiming, "apply.visiblePre", t1);
|
|
847
884
|
}
|
|
848
885
|
let durablePack: DurablePack | undefined;
|
|
849
886
|
if (
|
|
@@ -859,12 +896,25 @@ export class RestoreEngine {
|
|
|
859
896
|
cached.currentManifestId === current.manifestId &&
|
|
860
897
|
cached.targetManifestId === target.manifestId
|
|
861
898
|
) {
|
|
862
|
-
|
|
863
|
-
|
|
864
|
-
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
)
|
|
899
|
+
const pairKey = durablePairKey(current.manifestId, target.manifestId, plan.scopePaths ?? []);
|
|
900
|
+
const residentKey = `${pairKey}:${plan.planDigest}`;
|
|
901
|
+
const resident = this.verifiedDurablePacks.has(residentKey)
|
|
902
|
+
? this.durablePackByPair.get(pairKey)
|
|
903
|
+
: undefined;
|
|
904
|
+
if (resident?.planDigest === plan.planDigest && resident.pack.packChecksum === cached.packChecksum) {
|
|
905
|
+
durablePack = await publishResidentDurablePack(
|
|
906
|
+
resident.pack,
|
|
907
|
+
options.mutationJournal,
|
|
908
|
+
plan.planDigest,
|
|
909
|
+
);
|
|
910
|
+
} else {
|
|
911
|
+
durablePack = await publishCachedDurablePack(
|
|
912
|
+
cached.path,
|
|
913
|
+
options.mutationJournal,
|
|
914
|
+
plan.planDigest,
|
|
915
|
+
cached.packChecksum,
|
|
916
|
+
);
|
|
917
|
+
}
|
|
868
918
|
} else {
|
|
869
919
|
durablePack = await createDurablePack(options.mutationJournal, {
|
|
870
920
|
opId: options.opId,
|
|
@@ -903,35 +953,27 @@ export class RestoreEngine {
|
|
|
903
953
|
this.canUseNativeFilePlan(plan, currentPaths, targetPaths)
|
|
904
954
|
) {
|
|
905
955
|
const result = await this.applyNativeFilePlan(
|
|
906
|
-
plan,
|
|
907
|
-
current,
|
|
908
|
-
target,
|
|
909
|
-
currentPaths,
|
|
910
|
-
targetPaths,
|
|
911
|
-
topologyBefore,
|
|
912
|
-
options,
|
|
913
|
-
durablePack,
|
|
914
|
-
nativeFileBatch.run,
|
|
956
|
+
plan, current, target, currentPaths, targetPaths, topologyBefore, options, durablePack, nativeFileBatch.run,
|
|
915
957
|
);
|
|
916
958
|
return result;
|
|
917
959
|
}
|
|
960
|
+
const t2 = performance.now();
|
|
918
961
|
try {
|
|
919
962
|
await this.prefetchCompleteRestoreBlobs(plan, current, target, currentPaths, targetPaths);
|
|
920
963
|
} catch (error) {
|
|
921
964
|
rethrowOperationFailure(error);
|
|
922
965
|
// 预取是性能优化;失败时继续走原有逐文件校验和可恢复 mutation 路径。
|
|
966
|
+
} finally {
|
|
967
|
+
reportRestoreTiming(options.onTiming, "apply.prefetch", t2);
|
|
923
968
|
}
|
|
924
969
|
const reusableTargetPaths = new Set(
|
|
925
970
|
[...targetPaths.keys()].filter((path) => !isPathAffectedByMutations(path, plan.deletePaths, plan.writePaths)),
|
|
926
971
|
);
|
|
972
|
+
const t3 = performance.now();
|
|
927
973
|
const preflight = await this.verifyKnownState(
|
|
928
|
-
current,
|
|
929
|
-
target,
|
|
930
|
-
currentPaths,
|
|
931
|
-
targetPaths,
|
|
932
|
-
plan.scopePaths,
|
|
933
|
-
reusableTargetPaths,
|
|
974
|
+
current, target, currentPaths, targetPaths, plan.scopePaths, reusableTargetPaths,
|
|
934
975
|
);
|
|
976
|
+
reportRestoreTiming(options.onTiming, "apply.preflight", t3);
|
|
935
977
|
if (!preflight.ok) {
|
|
936
978
|
return {
|
|
937
979
|
code: "restore_failed_safe",
|
|
@@ -958,34 +1000,30 @@ export class RestoreEngine {
|
|
|
958
1000
|
quarantine,
|
|
959
1001
|
};
|
|
960
1002
|
try {
|
|
961
|
-
|
|
962
|
-
|
|
963
|
-
targetPaths,
|
|
964
|
-
plan.deletePaths,
|
|
965
|
-
mutationContext,
|
|
966
|
-
);
|
|
1003
|
+
const t4 = performance.now();
|
|
1004
|
+
await this.deletePlannedPaths(target.manifestId, targetPaths, plan.deletePaths, mutationContext);
|
|
967
1005
|
await this.writePlannedPaths(target.manifestId, targetPaths, plan.writePaths, mutationContext);
|
|
968
|
-
|
|
1006
|
+
reportRestoreTiming(options.onTiming, "apply.mutation", t4);
|
|
1007
|
+
const t5 = performance.now();
|
|
969
1008
|
const topologyAfter = await this.discovery.discover(this.workspaceRoot, "restore-post");
|
|
970
1009
|
assertUnchangedTopology(topologyBefore, topologyAfter);
|
|
971
1010
|
await this.assertCompleteVisibleSubset(
|
|
972
|
-
topologyAfter,
|
|
973
|
-
[target],
|
|
974
|
-
options.mutationJournal,
|
|
1011
|
+
topologyAfter, [target], options.mutationJournal,
|
|
975
1012
|
{
|
|
976
1013
|
topologyValidated: true,
|
|
1014
|
+
// assertUnchangedTopology 已确认 topologyAfter 与 topologyBefore 完全一致,
|
|
1015
|
+
// 且 restore 在工作区独占锁内执行,无需再次全量扫描目录。
|
|
1016
|
+
skipPostTopologyCheck: true,
|
|
977
1017
|
scopePaths: plan.scopePaths,
|
|
978
1018
|
ownedPaths: [targetPaths],
|
|
979
1019
|
},
|
|
980
1020
|
);
|
|
1021
|
+
reportRestoreTiming(options.onTiming, "apply.topologyPost", t5);
|
|
1022
|
+
const t6 = performance.now();
|
|
981
1023
|
const verification = await this.verifyTarget(
|
|
982
|
-
target,
|
|
983
|
-
currentPaths,
|
|
984
|
-
targetPaths,
|
|
985
|
-
plan.deletePaths,
|
|
986
|
-
plan.scopePaths,
|
|
987
|
-
preflight.targetAttestations,
|
|
1024
|
+
target, currentPaths, targetPaths, plan.deletePaths, plan.scopePaths, preflight.targetAttestations,
|
|
988
1025
|
);
|
|
1026
|
+
reportRestoreTiming(options.onTiming, "apply.verify", t6);
|
|
989
1027
|
const result: RestoreResult = {
|
|
990
1028
|
code: "ok",
|
|
991
1029
|
verifiedPaths: verification.verifiedPaths,
|
|
@@ -1036,8 +1074,11 @@ export class RestoreEngine {
|
|
|
1036
1074
|
pack: DurablePack,
|
|
1037
1075
|
nativeRun: (pack: DurablePack) => Promise<void>,
|
|
1038
1076
|
): Promise<RestoreResult> {
|
|
1077
|
+
const t0 = performance.now();
|
|
1039
1078
|
try {
|
|
1040
1079
|
await nativeRun(pack);
|
|
1080
|
+
reportRestoreTiming(options.onTiming, "apply.nativeRun", t0);
|
|
1081
|
+
const t1 = performance.now();
|
|
1041
1082
|
const topologyAfter = await this.discovery.discover(this.workspaceRoot, "restore-post");
|
|
1042
1083
|
assertUnchangedTopology(topologyBefore, topologyAfter);
|
|
1043
1084
|
await this.assertCompleteVisibleSubset(
|
|
@@ -1046,6 +1087,9 @@ export class RestoreEngine {
|
|
|
1046
1087
|
options.mutationJournal,
|
|
1047
1088
|
{
|
|
1048
1089
|
topologyValidated: true,
|
|
1090
|
+
// assertUnchangedTopology 已确认 topologyAfter 与 topologyBefore 完全一致,
|
|
1091
|
+
// 且 restore 在工作区独占锁内执行,无需再次全量扫描目录。
|
|
1092
|
+
skipPostTopologyCheck: true,
|
|
1049
1093
|
extraExclusions: pack.paths().flatMap((path) => {
|
|
1050
1094
|
const artifacts = pack.artifacts(path);
|
|
1051
1095
|
return artifacts === undefined
|
|
@@ -1056,10 +1100,13 @@ export class RestoreEngine {
|
|
|
1056
1100
|
ownedPaths: [targetPaths],
|
|
1057
1101
|
},
|
|
1058
1102
|
);
|
|
1103
|
+
reportRestoreTiming(options.onTiming, "apply.topologyPost", t1);
|
|
1104
|
+
const t2 = performance.now();
|
|
1059
1105
|
const totalPaths = plan.deletePaths.length + plan.writePaths.length;
|
|
1060
1106
|
if ((await options.mutationJournal.load()).length !== 0) {
|
|
1061
1107
|
return { code: "recovery_required", verifiedPaths: 0, totalPaths };
|
|
1062
1108
|
}
|
|
1109
|
+
reportRestoreTiming(options.onTiming, "apply.verifyNative", t2);
|
|
1063
1110
|
return { code: "ok", verifiedPaths: totalPaths, totalPaths };
|
|
1064
1111
|
} catch (error) {
|
|
1065
1112
|
if (isUnconfirmedExit(error)) throw error;
|
|
@@ -1088,7 +1135,9 @@ export class RestoreEngine {
|
|
|
1088
1135
|
targetPaths: ReadonlyMap<string, OwnedPath>,
|
|
1089
1136
|
plan: RestorePlan,
|
|
1090
1137
|
opId: string,
|
|
1138
|
+
options: DurablePackPreparationOptions = {},
|
|
1091
1139
|
): Promise<DurablePackEntryInput[]> {
|
|
1140
|
+
const blobsStarted = performance.now();
|
|
1092
1141
|
const paths = [...new Set([...plan.deletePaths, ...plan.writePaths])].sort(comparePaths);
|
|
1093
1142
|
const useValidatedBatch = SnapshotStore.supportsValidatedBlobBatch(this.store);
|
|
1094
1143
|
const [currentBlobBytes, targetBlobBytes] = useValidatedBatch
|
|
@@ -1097,6 +1146,8 @@ export class RestoreEngine {
|
|
|
1097
1146
|
this.readDurableBlobBytes(target.manifestId, paths, targetPaths),
|
|
1098
1147
|
])
|
|
1099
1148
|
: [new Map<string, Uint8Array>(), new Map<string, Uint8Array>()];
|
|
1149
|
+
reportDurablePackTiming(options, "prepare.blobs", blobsStarted);
|
|
1150
|
+
const entriesStarted = performance.now();
|
|
1100
1151
|
const result: DurablePackEntryInput[] = [];
|
|
1101
1152
|
for (const path of paths) {
|
|
1102
1153
|
const variants = new Map<string, DurableLeafInput>();
|
|
@@ -1125,6 +1176,7 @@ export class RestoreEngine {
|
|
|
1125
1176
|
variants: [...variants.values()],
|
|
1126
1177
|
});
|
|
1127
1178
|
}
|
|
1179
|
+
reportDurablePackTiming(options, "prepare.entries", entriesStarted);
|
|
1128
1180
|
return result;
|
|
1129
1181
|
}
|
|
1130
1182
|
|
|
@@ -1880,6 +1932,7 @@ export class RestoreEngine {
|
|
|
1880
1932
|
excludePaths: exclusions.size === 0 ? undefined : [...exclusions],
|
|
1881
1933
|
includePaths: check.scopePaths,
|
|
1882
1934
|
topologyAlreadyValidated: check.topologyValidated,
|
|
1935
|
+
skipPostTopologyCheck: check.skipPostTopologyCheck,
|
|
1883
1936
|
});
|
|
1884
1937
|
for (const path of livePaths) {
|
|
1885
1938
|
if (!allowedPaths.has(path)) {
|
|
@@ -2124,6 +2177,14 @@ export class RestoreEngine {
|
|
|
2124
2177
|
}
|
|
2125
2178
|
}
|
|
2126
2179
|
|
|
2180
|
+
function reportRestoreTiming(
|
|
2181
|
+
onTiming: ((phase: string, durationMs: number) => void) | undefined,
|
|
2182
|
+
phase: string,
|
|
2183
|
+
startedAt: number,
|
|
2184
|
+
): void {
|
|
2185
|
+
try { onTiming?.(phase, performance.now() - startedAt); } catch { /* 诊断回调不影响恢复结果。 */ }
|
|
2186
|
+
}
|
|
2187
|
+
|
|
2127
2188
|
function preparedPlanKey(plan: RestorePlan): string {
|
|
2128
2189
|
return `${plan.currentManifestId}\0${plan.targetManifestId}\0${plan.planDigest}`;
|
|
2129
2190
|
}
|
package/src/root-discovery.ts
CHANGED
package/src/snapshot-store.ts
CHANGED
|
@@ -11,7 +11,7 @@ import {
|
|
|
11
11
|
ignoredPresentClosure,
|
|
12
12
|
topologyFingerprint,
|
|
13
13
|
} from "./encoding.ts";
|
|
14
|
-
import { GitRunner, type GitRunOptions } from "./git-runner.ts";
|
|
14
|
+
import { GitRunError, GitRunner, type GitRunOptions } from "./git-runner.ts";
|
|
15
15
|
import type {
|
|
16
16
|
DiscoveryRoot,
|
|
17
17
|
ManifestId,
|
|
@@ -35,6 +35,8 @@ const GC_RETENTION_MS = 7 * 24 * 60 * 60 * 1_000;
|
|
|
35
35
|
const IGNORE_POLICY = "git-check-ignore-v1";
|
|
36
36
|
const NULL_DEVICE = process.platform === "win32" ? "NUL" : "/dev/null";
|
|
37
37
|
const TREE_CACHE_LIMIT = 256;
|
|
38
|
+
const TREE_SCOPE_COMPACT_MIN_PATHS = process.platform === "win32" ? 512 : 4_096;
|
|
39
|
+
const TREE_SCOPE_PROBE_MAX_BYTES = 2 * 1024 * 1024;
|
|
38
40
|
const TREE_BLOB_MEMBERSHIP_LIMIT = 65_536;
|
|
39
41
|
const HASH_BATCH_MAX_PATHS = process.platform === "win32" ? 128 : 2_048;
|
|
40
42
|
const HASH_BATCH_MAX_ARGUMENT_BYTES = process.platform === "win32" ? 24 * 1024 : 128 * 1024;
|
|
@@ -232,6 +234,11 @@ export interface CaptureOptions {
|
|
|
232
234
|
* 因此校验窗口内出现的漂移依然会被拒绝。仅限调用方与本次枚举之间没有文件变更时使用。
|
|
233
235
|
*/
|
|
234
236
|
readonly topologyAlreadyValidated?: boolean;
|
|
237
|
+
/**
|
|
238
|
+
* 跳过捕获或枚举结束后的 topology 二次校验(safety-snapshot post 扫描)。
|
|
239
|
+
* 仅在调用方已持有工作区独占锁、且刚刚完成 discover 并确认 topology 未变化时传 true。
|
|
240
|
+
*/
|
|
241
|
+
readonly skipPostTopologyCheck?: boolean;
|
|
235
242
|
/** 只报告数量和错误类型,不包含路径、文件内容或异常消息。 */
|
|
236
243
|
readonly onDiagnostics?: (diagnostic: CaptureDiagnostic) => void;
|
|
237
244
|
/** 报告 capture 内部 wall-time 阶段;回调失败不影响捕获。 */
|
|
@@ -301,6 +308,8 @@ export class SnapshotStore {
|
|
|
301
308
|
private readonly treeBlobMembership = new Map<string, string>();
|
|
302
309
|
private readonly blobCache = new Map<string, CachedBlob>();
|
|
303
310
|
private readonly visibleLeafCache = new Map<string, Map<string, CachedVisibleLeaf>>();
|
|
311
|
+
/** check-ignore 探针结果缓存;同一进程内同一 cwd 只验证一次,失败不缓存。 */
|
|
312
|
+
private readonly validatedIgnoreQueries = new Set<string>();
|
|
304
313
|
private readonly leafCacheDirectoriesLoaded = new Set<string>();
|
|
305
314
|
private readonly configuredPrivateRepositories = new Set<string>();
|
|
306
315
|
private readonly leafCacheDirtyDirectories = new Set<string>();
|
|
@@ -433,7 +442,9 @@ export class SnapshotStore {
|
|
|
433
442
|
const cacheUpdates = capturedRoots.flatMap((captured) =>
|
|
434
443
|
captured.cacheUpdate === undefined ? [] : [captured.cacheUpdate]);
|
|
435
444
|
|
|
436
|
-
|
|
445
|
+
if (options.skipPostTopologyCheck !== true) {
|
|
446
|
+
await this.assertTopology(topology, "捕获期间 topology 已变化", "safety-snapshot");
|
|
447
|
+
}
|
|
437
448
|
const content = {
|
|
438
449
|
schemaVersion: SCHEMA_VERSION as 1,
|
|
439
450
|
workspaceIdentity: topology.workspaceIdentity,
|
|
@@ -518,7 +529,9 @@ export class SnapshotStore {
|
|
|
518
529
|
}
|
|
519
530
|
if (baselineFresh) {
|
|
520
531
|
if (diagnostic !== undefined) diagnostic.strategy = "baseline-reused";
|
|
521
|
-
|
|
532
|
+
if (options.skipPostTopologyCheck !== true) {
|
|
533
|
+
await this.assertTopology(topology, "捕获期间 topology 已变化", "safety-snapshot");
|
|
534
|
+
}
|
|
522
535
|
await this.touchStore(this.storeDirectory(topology));
|
|
523
536
|
return baseline;
|
|
524
537
|
}
|
|
@@ -526,7 +539,7 @@ export class SnapshotStore {
|
|
|
526
539
|
if (diagnostic !== undefined) diagnostic.strategy = "baseline-fallback";
|
|
527
540
|
const fullCaptureStartedAt = performance.now();
|
|
528
541
|
try {
|
|
529
|
-
return await this.captureLocked(topology, scope, { ...options, topologyAlreadyValidated: true }, diagnostic);
|
|
542
|
+
return await this.captureLocked(topology, scope, { ...options, topologyAlreadyValidated: true, skipPostTopologyCheck: options.skipPostTopologyCheck }, diagnostic);
|
|
530
543
|
} finally {
|
|
531
544
|
publishCaptureTiming(options.onTiming, "settled.fullCapture", fullCaptureStartedAt);
|
|
532
545
|
}
|
|
@@ -758,7 +771,9 @@ export class SnapshotStore {
|
|
|
758
771
|
result.add(workspaceRelativePath(root.relativeRoot, relativePath));
|
|
759
772
|
}
|
|
760
773
|
}
|
|
761
|
-
|
|
774
|
+
if (options.skipPostTopologyCheck !== true) {
|
|
775
|
+
await this.assertTopology(topology, "可见路径枚举期间 topology 已变化", "visible-paths-post");
|
|
776
|
+
}
|
|
762
777
|
return [...result].sort(comparePaths);
|
|
763
778
|
} catch (error) {
|
|
764
779
|
if (error instanceof SnapshotStoreError) throw error;
|
|
@@ -820,8 +835,7 @@ export class SnapshotStore {
|
|
|
820
835
|
)) {
|
|
821
836
|
throw new SnapshotStoreError("object_missing", "ignored-present proof 与 root tree 冲突");
|
|
822
837
|
}
|
|
823
|
-
await this.assertObjectsComplete(gitDirectory, root.treeId, entries);
|
|
824
|
-
if (scopePaths !== undefined) await this.preloadBlobBytes(gitDirectory, entries);
|
|
838
|
+
await this.assertObjectsComplete(gitDirectory, root.treeId, entries, scopePaths !== undefined);
|
|
825
839
|
if (rootScope === undefined && root.objectClosure !== treeObjectClosure(root.treeId, entries)) {
|
|
826
840
|
throw new SnapshotStoreError("object_missing", "root tree 对象闭包校验失败");
|
|
827
841
|
}
|
|
@@ -976,7 +990,9 @@ export class SnapshotStore {
|
|
|
976
990
|
const roots = new Map(manifest.roots.map((root) => [root.relativeRoot, root]));
|
|
977
991
|
const storeDirectory = dirname(dirname(manifestPath));
|
|
978
992
|
try {
|
|
979
|
-
|
|
993
|
+
// 先完成全部 membership 校验,再按 gitDirectory 分组;同一个私有 ODB 的多个 blob
|
|
994
|
+
// 合并成一次 cat-file --batch,省掉每次进程启动的固定开销。
|
|
995
|
+
const pending: { readonly gitDirectory: string; readonly objectId: string }[] = [];
|
|
980
996
|
for (const request of requests) {
|
|
981
997
|
const root = roots.get(request.rootPath);
|
|
982
998
|
if (root === undefined) {
|
|
@@ -1005,8 +1021,41 @@ export class SnapshotStore {
|
|
|
1005
1021
|
throw new SnapshotStoreError("object_missing", "blob 不属于指定 root tree path");
|
|
1006
1022
|
}
|
|
1007
1023
|
}
|
|
1008
|
-
|
|
1024
|
+
pending.push({ gitDirectory, objectId: request.blobId });
|
|
1025
|
+
}
|
|
1026
|
+
const bytesById = new Map<string, Uint8Array>();
|
|
1027
|
+
const byGitDirectory = new Map<string, string[]>();
|
|
1028
|
+
for (const { gitDirectory, objectId } of pending) {
|
|
1029
|
+
const ids = byGitDirectory.get(gitDirectory) ?? [];
|
|
1030
|
+
if (!ids.includes(objectId)) ids.push(objectId);
|
|
1031
|
+
byGitDirectory.set(gitDirectory, ids);
|
|
1032
|
+
}
|
|
1033
|
+
for (const [gitDirectory, objectIds] of byGitDirectory) {
|
|
1034
|
+
for (const objectId of objectIds) {
|
|
1035
|
+
const cached = this.blobCache.get(blobCacheKey(gitDirectory, objectId));
|
|
1036
|
+
if (cached === undefined) continue;
|
|
1037
|
+
bytesById.set(objectId, await cached.promise);
|
|
1038
|
+
}
|
|
1039
|
+
const missing = objectIds.filter((objectId) => !bytesById.has(objectId));
|
|
1040
|
+
if (missing.length === 0) continue;
|
|
1041
|
+
// batch 头部自带长度,无需预先知道 size;按响应自身长度解析后逐条校验。
|
|
1042
|
+
const loaded = parseBatchBlobOutputByOrder(
|
|
1043
|
+
await this.runGitBytes(["cat-file", "--batch"], {
|
|
1044
|
+
env: privateObjectEnvironment(gitDirectory),
|
|
1045
|
+
stdin: `${missing.join("\n")}\n`,
|
|
1046
|
+
}),
|
|
1047
|
+
missing,
|
|
1048
|
+
);
|
|
1049
|
+
for (const [objectId, bytes] of loaded) {
|
|
1050
|
+
this.rememberBlobBytes(gitDirectory, objectId, bytes);
|
|
1051
|
+
bytesById.set(objectId, bytes);
|
|
1052
|
+
}
|
|
1009
1053
|
}
|
|
1054
|
+
const result = pending.map(({ objectId }) => {
|
|
1055
|
+
const bytes = bytesById.get(objectId);
|
|
1056
|
+
if (bytes === undefined) throw new SnapshotStoreError("object_missing", "blob 无法读取");
|
|
1057
|
+
return new Uint8Array(bytes);
|
|
1058
|
+
});
|
|
1010
1059
|
if (revalidateManifest) await this.loadManifestFromPath(id, manifestPath);
|
|
1011
1060
|
return result;
|
|
1012
1061
|
} catch (error) {
|
|
@@ -1734,6 +1783,11 @@ export class SnapshotStore {
|
|
|
1734
1783
|
environment: Readonly<Record<string, string | undefined>>,
|
|
1735
1784
|
gitBacked: boolean,
|
|
1736
1785
|
): Promise<void> {
|
|
1786
|
+
// check-ignore 是探针:只验证忽略查询机制在当前 cwd 可用,不产生数据。
|
|
1787
|
+
// 同一进程内同一 cwd 的 git 环境不会变,重复启动进程没有任何信息价值;
|
|
1788
|
+
// 小工作区里这个固定开销占比很高。失败不缓存,仍会在下次重试。
|
|
1789
|
+
const cacheKey = `${cwd}\0${gitBacked ? "1" : "0"}`;
|
|
1790
|
+
if (this.validatedIgnoreQueries.has(cacheKey)) return;
|
|
1737
1791
|
try {
|
|
1738
1792
|
await this.runGit([
|
|
1739
1793
|
...(gitBacked ? ["-c", "core.fsmonitor=false"] : []),
|
|
@@ -1748,6 +1802,7 @@ export class SnapshotStore {
|
|
|
1748
1802
|
throw error;
|
|
1749
1803
|
}
|
|
1750
1804
|
}
|
|
1805
|
+
this.validatedIgnoreQueries.add(cacheKey);
|
|
1751
1806
|
}
|
|
1752
1807
|
|
|
1753
1808
|
private async ensurePrivateRepository(gitDirectory: string): Promise<void> {
|
|
@@ -1798,15 +1853,18 @@ export class SnapshotStore {
|
|
|
1798
1853
|
const key = `${gitDirectory}\0${treeId}\0${scope === undefined ? "*" : checksum(canonicalJson(scope))}`;
|
|
1799
1854
|
const cached = this.treeEntriesCache.get(key);
|
|
1800
1855
|
if (cached !== undefined) return cached;
|
|
1801
|
-
const pending =
|
|
1802
|
-
|
|
1803
|
-
|
|
1804
|
-
|
|
1805
|
-
|
|
1806
|
-
|
|
1807
|
-
|
|
1808
|
-
|
|
1809
|
-
const
|
|
1856
|
+
const pending = (async () => {
|
|
1857
|
+
const queryScope = scope === undefined || scope.length < TREE_SCOPE_COMPACT_MIN_PATHS
|
|
1858
|
+
? scope
|
|
1859
|
+
: await this.compactTreeQueryScope(gitDirectory, treeId, scope);
|
|
1860
|
+
const output = await this.runPrivateGitBytes(gitDirectory, [
|
|
1861
|
+
"ls-tree", "-r", "-l", "-z", treeId,
|
|
1862
|
+
...(queryScope === undefined ? [] : ["--", ...queryScope.map(literalPathspec)]),
|
|
1863
|
+
]);
|
|
1864
|
+
const scopeSet = queryScope === scope ? undefined : new Set(scope);
|
|
1865
|
+
// 查询范围合并只是优化提示;缓存和 path ownership 仍受原始 scope 约束。
|
|
1866
|
+
const entries = parseTreeEntries(output).filter((entry) =>
|
|
1867
|
+
scopeSet === undefined || treeScopeContains(scopeSet, entry.relativePath));
|
|
1810
1868
|
for (const entry of entries) {
|
|
1811
1869
|
const membershipKey = treeBlobMembershipKey(gitDirectory, treeId, entry.relativePath);
|
|
1812
1870
|
this.treeBlobMembership.delete(membershipKey);
|
|
@@ -1818,7 +1876,7 @@ export class SnapshotStore {
|
|
|
1818
1876
|
this.treeBlobMembership.delete(oldest);
|
|
1819
1877
|
}
|
|
1820
1878
|
return entries;
|
|
1821
|
-
});
|
|
1879
|
+
})();
|
|
1822
1880
|
this.treeEntriesCache.set(key, pending);
|
|
1823
1881
|
while (this.treeEntriesCache.size > TREE_CACHE_LIMIT) {
|
|
1824
1882
|
const oldest = this.treeEntriesCache.keys().next().value as string | undefined;
|
|
@@ -1833,14 +1891,90 @@ export class SnapshotStore {
|
|
|
1833
1891
|
}
|
|
1834
1892
|
}
|
|
1835
1893
|
|
|
1894
|
+
private async compactTreeQueryScope(
|
|
1895
|
+
gitDirectory: string,
|
|
1896
|
+
treeId: string,
|
|
1897
|
+
scope: readonly string[],
|
|
1898
|
+
): Promise<readonly string[] | undefined> {
|
|
1899
|
+
// 只探测 scope 的公共父目录,避免为合并路径而遍历无关子树。
|
|
1900
|
+
let parent = scope[0]!.slice(0, Math.max(0, scope[0]!.lastIndexOf("/")));
|
|
1901
|
+
for (const path of scope) {
|
|
1902
|
+
while (parent !== "" && !path.startsWith(`${parent}/`)) {
|
|
1903
|
+
parent = parent.slice(0, Math.max(0, parent.lastIndexOf("/")));
|
|
1904
|
+
}
|
|
1905
|
+
}
|
|
1906
|
+
let output: Uint8Array;
|
|
1907
|
+
try {
|
|
1908
|
+
output = await this.runGitBytes([
|
|
1909
|
+
"ls-tree", "-r", "--name-only", "-z", treeId,
|
|
1910
|
+
...(parent === "" ? [] : ["--", literalPathspec(parent)]),
|
|
1911
|
+
], {
|
|
1912
|
+
env: privateObjectEnvironment(gitDirectory),
|
|
1913
|
+
// 稀疏 scope 不应为优化提示枚举无界 tree;超限后使用原始精确查询。
|
|
1914
|
+
stdoutLimit: Math.min(TREE_SCOPE_PROBE_MAX_BYTES,
|
|
1915
|
+
Math.max(64 * 1024, scope.reduce((bytes, path) => bytes + Buffer.byteLength(path) + 1, 0) * 4)),
|
|
1916
|
+
});
|
|
1917
|
+
} catch (error) {
|
|
1918
|
+
if (error instanceof GitRunError && (error.code === "git_output_limit" || error.code === "git_failed")) return scope;
|
|
1919
|
+
throw error;
|
|
1920
|
+
}
|
|
1921
|
+
let paths: string[];
|
|
1922
|
+
try {
|
|
1923
|
+
paths = splitNulRecords(output).map((record) => relativeSafePath("/", decodeUtf8(record)));
|
|
1924
|
+
} catch {
|
|
1925
|
+
// 提示不能扩大校验范围:范围外的非法路径仍由原始精确查询处理。
|
|
1926
|
+
return scope;
|
|
1927
|
+
}
|
|
1928
|
+
const included = new Set(scope);
|
|
1929
|
+
const blocked = new Set<string>();
|
|
1930
|
+
// 未探测的祖先不能用作合并目标。
|
|
1931
|
+
if (parent !== "") {
|
|
1932
|
+
blocked.add(".");
|
|
1933
|
+
for (let end = parent.indexOf("/"); end >= 0; end = parent.indexOf("/", end + 1)) {
|
|
1934
|
+
blocked.add(parent.slice(0, end));
|
|
1935
|
+
}
|
|
1936
|
+
}
|
|
1937
|
+
for (const path of paths) {
|
|
1938
|
+
if (treeScopeContains(included, path)) continue;
|
|
1939
|
+
blocked.add(".");
|
|
1940
|
+
for (let end = path.indexOf("/"); end >= 0; end = path.indexOf("/", end + 1)) {
|
|
1941
|
+
blocked.add(path.slice(0, end));
|
|
1942
|
+
}
|
|
1943
|
+
}
|
|
1944
|
+
if (!blocked.has(".")) return undefined;
|
|
1945
|
+
const compacted = new Set<string>();
|
|
1946
|
+
for (const path of scope) {
|
|
1947
|
+
let candidate = path;
|
|
1948
|
+
for (let end = path.indexOf("/"); end >= 0; end = path.indexOf("/", end + 1)) {
|
|
1949
|
+
const parent = path.slice(0, end);
|
|
1950
|
+
if (!blocked.has(parent)) { candidate = parent; break; }
|
|
1951
|
+
}
|
|
1952
|
+
compacted.add(candidate);
|
|
1953
|
+
}
|
|
1954
|
+
return [...compacted].sort(comparePaths);
|
|
1955
|
+
}
|
|
1956
|
+
|
|
1836
1957
|
private async assertObjectsComplete(
|
|
1837
1958
|
gitDirectory: string,
|
|
1838
1959
|
treeId: string,
|
|
1839
1960
|
entries: readonly CapturedTreeEntry[],
|
|
1961
|
+
preloadBlobs = false,
|
|
1840
1962
|
): Promise<void> {
|
|
1963
|
+
const unique = new Map(entries.map((entry) => [entry.objectId, entry]));
|
|
1964
|
+
const toRead: CapturedTreeEntry[] = [];
|
|
1965
|
+
const toCheck: string[] = [];
|
|
1966
|
+
for (const entry of unique.values()) {
|
|
1967
|
+
if (preloadBlobs && !this.blobCache.has(blobCacheKey(gitDirectory, entry.objectId))) {
|
|
1968
|
+
toRead.push(entry);
|
|
1969
|
+
} else {
|
|
1970
|
+
toCheck.push(entry.objectId);
|
|
1971
|
+
}
|
|
1972
|
+
}
|
|
1973
|
+
// 冷 blob 由 --batch 的 ID、类型、长度和完整响应校验建立证据;
|
|
1974
|
+
// 缓存命中的 blob 仍查磁盘,避免内存内容掩盖对象缺失。分组只做一次。
|
|
1841
1975
|
const expected = [
|
|
1842
1976
|
{ objectId: treeId, type: "tree" },
|
|
1843
|
-
...
|
|
1977
|
+
...toCheck.map((objectId) => ({ objectId, type: "blob" })),
|
|
1844
1978
|
];
|
|
1845
1979
|
const output = await this.runGit(["cat-file", "--batch-check"], {
|
|
1846
1980
|
env: privateObjectEnvironment(gitDirectory),
|
|
@@ -1860,6 +1994,8 @@ export class SnapshotStore {
|
|
|
1860
1994
|
throw new Error(`Git object batch-check 校验失败:${object.objectId}`);
|
|
1861
1995
|
}
|
|
1862
1996
|
}
|
|
1997
|
+
// 不重新按 cache 过滤:并行请求可能在分组后填入缓存,仍须完成本次读取校验。
|
|
1998
|
+
await this.readBlobBatches(gitDirectory, toRead);
|
|
1863
1999
|
}
|
|
1864
2000
|
|
|
1865
2001
|
private async preloadBlobBytes(
|
|
@@ -1870,7 +2006,16 @@ export class SnapshotStore {
|
|
|
1870
2006
|
for (const entry of entries) {
|
|
1871
2007
|
if (!this.blobCache.has(blobCacheKey(gitDirectory, entry.objectId))) unique.set(entry.objectId, entry);
|
|
1872
2008
|
}
|
|
1873
|
-
|
|
2009
|
+
await this.readBlobBatches(gitDirectory, [...unique.values()]);
|
|
2010
|
+
}
|
|
2011
|
+
|
|
2012
|
+
private async readBlobBatches(gitDirectory: string, entries: readonly CapturedTreeEntry[]): Promise<void> {
|
|
2013
|
+
const batches = blobReadBatches(entries);
|
|
2014
|
+
// 只并行小批次:两个在途 payload 合计不超过原来的单批字节预算。
|
|
2015
|
+
// 存在大批次或超限单文件时保持串行,失败后等待所有在途任务退出。
|
|
2016
|
+
const concurrency = batches.every((batch) =>
|
|
2017
|
+
batch.reduce((bytes, entry) => bytes + entry.size, 0) <= BLOB_BATCH_MAX_BYTES / 2) ? 2 : 1;
|
|
2018
|
+
await mapConcurrentOrdered(batches, concurrency, async (batch) => {
|
|
1874
2019
|
const loaded = parseBatchBlobOutput(
|
|
1875
2020
|
await this.runGitBytes(["cat-file", "--batch"], {
|
|
1876
2021
|
env: privateObjectEnvironment(gitDirectory),
|
|
@@ -1879,7 +2024,7 @@ export class SnapshotStore {
|
|
|
1879
2024
|
batch,
|
|
1880
2025
|
);
|
|
1881
2026
|
for (const [objectId, bytes] of loaded) this.rememberBlobBytes(gitDirectory, objectId, bytes);
|
|
1882
|
-
}
|
|
2027
|
+
});
|
|
1883
2028
|
}
|
|
1884
2029
|
|
|
1885
2030
|
private async readBlobText(gitDirectory: string, objectId: string): Promise<string> {
|
|
@@ -2418,6 +2563,38 @@ function parseBatchBlobOutput(
|
|
|
2418
2563
|
return result;
|
|
2419
2564
|
}
|
|
2420
2565
|
|
|
2566
|
+
function parseBatchBlobOutputByOrder(
|
|
2567
|
+
output: Uint8Array,
|
|
2568
|
+
expected: readonly string[],
|
|
2569
|
+
): Map<string, Uint8Array> {
|
|
2570
|
+
const result = new Map<string, Uint8Array>();
|
|
2571
|
+
let offset = 0;
|
|
2572
|
+
for (const objectId of expected) {
|
|
2573
|
+
const lineEnd = output.indexOf(0x0a, offset);
|
|
2574
|
+
if (lineEnd < 0) throw new SnapshotStoreError("object_missing", "Git blob batch header 不完整");
|
|
2575
|
+
const header = decodeUtf8(output.subarray(offset, lineEnd));
|
|
2576
|
+
const match = header.match(/^([0-9a-f]{40,64}) blob ([0-9]+)$/);
|
|
2577
|
+
if (match === null || match[1] !== objectId) {
|
|
2578
|
+
throw new SnapshotStoreError("object_missing", `Git blob batch header 无效:${objectId}`);
|
|
2579
|
+
}
|
|
2580
|
+
const size = Number(match[2]);
|
|
2581
|
+
if (!Number.isSafeInteger(size) || size < 0) {
|
|
2582
|
+
throw new SnapshotStoreError("object_missing", `Git blob batch 长度无效:${objectId}`);
|
|
2583
|
+
}
|
|
2584
|
+
const contentStart = lineEnd + 1;
|
|
2585
|
+
const contentEnd = contentStart + size;
|
|
2586
|
+
if (contentEnd >= output.length || output[contentEnd] !== 0x0a) {
|
|
2587
|
+
throw new SnapshotStoreError("object_missing", `Git blob batch 内容不完整:${objectId}`);
|
|
2588
|
+
}
|
|
2589
|
+
result.set(objectId, output.slice(contentStart, contentEnd));
|
|
2590
|
+
offset = contentEnd + 1;
|
|
2591
|
+
}
|
|
2592
|
+
if (offset !== output.length) {
|
|
2593
|
+
throw new SnapshotStoreError("object_missing", "Git blob batch 输出包含多余内容");
|
|
2594
|
+
}
|
|
2595
|
+
return result;
|
|
2596
|
+
}
|
|
2597
|
+
|
|
2421
2598
|
function indexInfoBatches(
|
|
2422
2599
|
leaves: readonly VisibleLeaf[],
|
|
2423
2600
|
objectIds: ReadonlyMap<string, string>,
|
|
@@ -2672,6 +2849,14 @@ function treeDirectoryPaths(entries: readonly CapturedTreeEntry[]): string[] {
|
|
|
2672
2849
|
return directories;
|
|
2673
2850
|
}
|
|
2674
2851
|
|
|
2852
|
+
function treeScopeContains(scope: ReadonlySet<string>, path: string): boolean {
|
|
2853
|
+
if (scope.has(".") || scope.has(path)) return true;
|
|
2854
|
+
for (let end = path.indexOf("/"); end >= 0; end = path.indexOf("/", end + 1)) {
|
|
2855
|
+
if (scope.has(path.slice(0, end))) return true;
|
|
2856
|
+
}
|
|
2857
|
+
return false;
|
|
2858
|
+
}
|
|
2859
|
+
|
|
2675
2860
|
function parseTreeEntries(output: Uint8Array): CapturedTreeEntry[] {
|
|
2676
2861
|
const entries: CapturedTreeEntry[] = [];
|
|
2677
2862
|
for (const record of splitNulRecords(output)) {
|