@deepseek-ai/dsh-session-persistence-jsonl 0.1.6-alpha.2 → 0.1.7-alpha.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.i18n.yaml +2 -2
- package/README.md +6 -2
- package/README.zh.md +6 -2
- package/lib/index.js +210 -45
- package/lib/types/catalog-migration.d.ts +28 -0
- package/lib/types/generation.d.ts +16 -0
- package/lib/types/index.d.ts +5 -1
- package/lib/worker.cjs +1456 -313
- package/package.json +13 -13
package/README.i18n.yaml
CHANGED
|
@@ -2,5 +2,5 @@
|
|
|
2
2
|
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
|
3
3
|
# after editing either side, bring the other along and re-record with:
|
|
4
4
|
# pnpm run verify-translation-pairing --write packages/session/session-persistence-jsonl/README.md
|
|
5
|
-
README.md:
|
|
6
|
-
README.zh.md:
|
|
5
|
+
README.md: 5846404961caaef7a91fb4dda562655d15fd2be9
|
|
6
|
+
README.zh.md: b7dd646e57b75fe47f3683d2b230dace628f8eee
|
package/README.md
CHANGED
|
@@ -79,7 +79,11 @@ The current-generation scanner applies the current codec owner’s structural ad
|
|
|
79
79
|
|
|
80
80
|
### Reading the logs
|
|
81
81
|
|
|
82
|
-
`open(id, 'read'|'write')` selects the highest canonical generation. Current input follows the ordinary fast path. For historical input, a read open decodes and migrates the source once, validates the current logical result, and returns it without publishing a successor. A write open reuses that revision-keyed preparation when available, or performs the same preparation, then encodes a same-directory temporary file in bounded chunks, verifies it in a Worker Thread, rechecks the source revision, and publishes the current successor without overwrite before returning. The source remains byte-identical. Source drift after preparation rejects that write open without replacing the logical history already returned to readers; a later write open prepares the new revision. The backend marks decoded event graphs `shared-frozen` when it freezes them before memoization; handle reads and slices preserve that state, including empty slices. Only an unmaterialized pending log reports `detached`. `stat(id)` and `list()` select and translate only the highest generation header without reading event rows or starting migration; snapshots carry `sizeBytes` and a best-effort
|
|
82
|
+
`open(id, 'read'|'write')` selects the highest canonical generation. Current input follows the ordinary fast path. For historical input, a read open decodes and migrates the source once, validates the current logical result, and returns it without publishing a successor. A write open reuses that revision-keyed preparation when available, or performs the same preparation, then encodes a same-directory temporary file in bounded chunks, verifies it in a Worker Thread, rechecks the source revision, and publishes the current successor without overwrite before returning. The source remains byte-identical. Source drift after preparation rejects that write open without replacing the logical history already returned to readers; a later write open prepares the new revision. The backend marks decoded event graphs `shared-frozen` when it freezes them before memoization; every nested object and array is frozen, and handle reads and slices preserve that state, including empty slices. Only an unmaterialized pending log reports `detached`. `stat(id)` and `list()` select and translate only the highest generation header without reading event rows or starting migration; snapshots carry the selected file’s `sizeBytes` and a best-effort revision. Current revisions identify that file; historical revisions also fingerprint the selected files across the persistence root, so child changes invalidate cached logical events. Fingerprinting reads filesystem metadata only; unrelated changes conservatively invalidate historical revisions. One `list()` call shares a corpus fingerprint across its historical entries. With `compression: 'none'`, the log is newline-delimited text an external reader can consume directly; the compressed default must be read through the backend.
|
|
83
|
+
|
|
84
|
+
Historical body preparation completes the parent catalog through [V3→V4](../session-format-v3-to-v4/README.md). It finds candidate direct children from headers, reads each child's own descriptor through historical codecs, and retains compact evidence plus source revisions. It neither prepares child catalogs nor publishes child successors. Unreadable or unsupported headers, including corrupt Zstandard header frames, are omitted from discovery and `list()`. Direct access to a corrupt compressed header still rejects; header I/O and cancellation errors propagate. Child decoding and descriptor-field failures produce warnings naming the child path and retain header identity through `subagent/catalog` when the parent has no complete entry; healthy children and existing parent catalog entries remain available. Opening a damaged child still reports that child's error. Missing, unsupported, or multiple descriptors likewise produce unknown-mode membership without inventing a label. Published unknown entries remain browsable; child history reads retry the actual log and resolve its mode from a valid descriptor. Preparation rechecks membership and inspected source revisions before returning, reuse, and publication, including failed child reads so a repaired child invalidates stale preparation. Source drift retries a read open once and refuses a write open. Cancellation still aborts the operation. Current V4 opens bypass discovery and validate catalog fields, uniqueness, and current delivery ownership before exposing events.
|
|
85
|
+
|
|
86
|
+
Historical `stat` and `list` revisions require metadata work proportional to the root’s Session count. Fresh body preparation scans all selected headers and decodes direct-child bodies; memo reuse still scans membership and checks revisions. Read-only access never publishes an upgrade, so cold processes and evicted preparations repeat that work. Current V4 body reads and revisions avoid the historical corpus scan. See the [measured costs and diagnostic command](../../../.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.md#catalog-scan-measurements).
|
|
83
87
|
|
|
84
88
|
-----
|
|
85
89
|
|
|
@@ -93,7 +97,7 @@ This section explains the physical encoding and write path; the observable contr
|
|
|
93
97
|
|
|
94
98
|
### Design concept
|
|
95
99
|
|
|
96
|
-
The backend owns its complete storage runtime (`src/storage.ts`): `JsonlSessionHandle` carries the per-handle mutation chain, the routed live-event buffer with its fixed batching window and single-flight drain, monotonic reads, and idempotent close; a tracker holds the in-process single-writer claims, the open-handle set teardown sweeps, and the created-but-unmaterialized pending sessions the backend's own session listeners route into. Historical body reads share one per-session Decode/Migrate preparation, and a bounded revision-keyed memo lets an immediate observe-to-resume handoff reuse that parse; the backend deep-freezes each event graph once before memoization, so later handle reads reuse it without copying or freezing. Only a write open publishes the prepared successor. The package deliberately exposes only its default plugin export plus configuration types — the concrete class is not a named export, so consumers couple to `ctx.sessionPersistence`, and the shared seam suites (`runPersistenceContract`/`runLiveWritePathContract`) pin its observable behavior.
|
|
100
|
+
The backend owns its complete storage runtime (`src/storage.ts`): `JsonlSessionHandle` carries the per-handle mutation chain, the routed live-event buffer with its fixed batching window and single-flight drain, monotonic reads, and idempotent close; a tracker holds the in-process single-writer claims, the open-handle set teardown sweeps, and the created-but-unmaterialized pending sessions the backend's own session listeners route into. Historical body reads share one per-session Decode/Migrate preparation, and a bounded revision-keyed memo lets an immediate observe-to-resume handoff reuse that parse; the backend deep-freezes each event graph once before memoization, so later handle reads reuse it without copying or freezing. Only a write open publishes the prepared successor. The package deliberately exposes only its default plugin export plus configuration types — the concrete class is not a named export, so consumers couple to `ctx.sessionPersistence`, and the shared seam suites (`runPersistenceContract`/`runLiveWritePathContract`) pin its observable behavior. Physical revisions combine device, inode, size, and nanosecond timestamps for the preparation memo, stable-read retries, and publication checks. Historical public revisions add a SHA-256 fingerprint of sorted selected paths and their physical revisions; lock files and retained unselected generations do not contribute.
|
|
97
101
|
|
|
98
102
|
### Physical encoding
|
|
99
103
|
|
package/README.zh.md
CHANGED
|
@@ -79,7 +79,11 @@ kind: "package-reference"
|
|
|
79
79
|
|
|
80
80
|
### 读取日志
|
|
81
81
|
|
|
82
|
-
`open(id, 'read'|'write')` 选择最高规范 generation。当前格式输入走普通快速路径。对于历史输入,只读 open 会单遍解码并迁移源、校验当前逻辑结果,然后在不发布后继的情况下返回。写 open 会在可用时复用按 revision 为键的 preparation,否则执行同一套 preparation,再按有界分片编码同目录临时文件、在 Worker Thread 中校验、复查源修订,并在返回前以不覆盖方式发布当前后继。源保持逐字节不变。如果源在 preparation 后发生变化,该次写 open 会失败,已经返回给读方的逻辑历史不会被替换;后续写 open 会针对新的 revision 重新执行 preparation。后端在 memo 化前冻结已解码的 event graph,并在此时将其标记为 `shared-frozen
|
|
82
|
+
`open(id, 'read'|'write')` 选择最高规范 generation。当前格式输入走普通快速路径。对于历史输入,只读 open 会单遍解码并迁移源、校验当前逻辑结果,然后在不发布后继的情况下返回。写 open 会在可用时复用按 revision 为键的 preparation,否则执行同一套 preparation,再按有界分片编码同目录临时文件、在 Worker Thread 中校验、复查源修订,并在返回前以不覆盖方式发布当前后继。源保持逐字节不变。如果源在 preparation 后发生变化,该次写 open 会失败,已经返回给读方的逻辑历史不会被替换;后续写 open 会针对新的 revision 重新执行 preparation。后端在 memo 化前冻结已解码的 event graph,并在此时将其标记为 `shared-frozen`;每个嵌套对象和数组都会冻结,句柄读取和 slice 即使为空也保留该状态。只有尚未实体化的 pending 空日志报告 `detached`。`stat(id)` 与 `list()` 只选择并转换最高 generation 的 header,不读取事件行,也不启动迁移;快照携带所选文件的 `sizeBytes` 与尽力而为的修订号。当前格式修订号标识该文件;历史格式修订号还包含持久化根目录中所选文件的指纹,因此子日志变化会使缓存的逻辑事件失效。指纹只读取文件系统元数据;无关变化也会保守地使历史修订号失效。一次 `list()` 为其历史条目共用一个全库指纹。选择 `compression: 'none'` 后,日志是外部读取方可直接消费的换行分隔文本;压缩默认值必须经后端读取。
|
|
83
|
+
|
|
84
|
+
历史正文准备通过 [V3→V4](../session-format-v3-to-v4/README.zh.md) 补齐父目录:从 header 找到候选直属子 Session,通过历史编解码器逐个读取其自身 descriptor,保留紧凑证据与来源修订。此过程不准备子目录,也不发布子后继。不可读或不支持的 header(包括损坏的 Zstandard header 帧)不参与发现,也不出现在 `list()` 中。直接访问损坏的压缩 header 仍会失败;header 的 I/O 错误与取消错误继续传播。子日志解码或 descriptor 字段失败会产生带子路径的警告;父目录没有完整条目时,通过 `subagent/catalog` 保留 header 身份信息。健康子项和已有父目录项仍可使用。打开损坏子 Session 时仍报告该子会话的错误。缺失、不支持或多个 descriptor 同样生成模式未知的目录项,不编造标签。已发布的未知条目仍可浏览;读取子历史时会重试实际日志,并从有效 descriptor 确定模式。准备返回、复用与发布前会重新检查成员集合及已检查来源的修订,也包括读取失败的子日志,使修复后的子日志能够使旧准备缓存失效。来源变化时只读打开重试一次,写打开拒绝发布。取消仍会中止操作。当前 V4 打开跳过发现,并在暴露事件前校验目录字段、唯一性及当前投递归属。
|
|
85
|
+
|
|
86
|
+
历史格式的 `stat` 与 `list` 修订号需要与根目录 Session 数量成正比的元数据工作。首次正文准备扫描所有所选 header 并解码直属子正文;复用准备缓存仍扫描成员集合并检查修订。只读访问从不发布升级,因此冷进程与被淘汰的准备缓存会重复这些工作。当前 V4 正文读取和修订号避开历史全库扫描。见[实测成本与诊断命令](../../../.agents/notes/implemented/architecture/2026-08-31-released-session-format-migrations.zh.md#catalog-scan-measurements)。
|
|
83
87
|
|
|
84
88
|
-----
|
|
85
89
|
|
|
@@ -93,7 +97,7 @@ kind: "package-reference"
|
|
|
93
97
|
|
|
94
98
|
### 设计理念
|
|
95
99
|
|
|
96
|
-
该后端拥有自己完整的存储运行时(`src/storage.ts`):`JsonlSessionHandle` 承载逐句柄修改链、带固定批处理窗口与 single-flight 排空的已路由实时事件缓冲、单调读取与幂等 close;一个 tracker 持有进程内单写者认领、teardown 清扫所遍历的打开句柄集合,以及后端自己的会话监听器所路由进的已创建但未实体化待定会话。历史正文读取共享每个 Session 唯一的一次 Decode/Migrate preparation,按 revision 为键的有界 memo 让紧接的观察到恢复交接复用该解析;backend 在 memo 化前只对每个 event graph 深度冻结一次,因此后续 handle read 无需复制或再次冻结。只有写 open 才发布准备好的后继。本包有意只暴露默认插件导出与配置类型——具体类不是具名导出,因此消费方只耦合 `ctx.sessionPersistence`,其可观察行为由共享 seam 测试套件(`runPersistenceContract`/`runLiveWritePathContract
|
|
100
|
+
该后端拥有自己完整的存储运行时(`src/storage.ts`):`JsonlSessionHandle` 承载逐句柄修改链、带固定批处理窗口与 single-flight 排空的已路由实时事件缓冲、单调读取与幂等 close;一个 tracker 持有进程内单写者认领、teardown 清扫所遍历的打开句柄集合,以及后端自己的会话监听器所路由进的已创建但未实体化待定会话。历史正文读取共享每个 Session 唯一的一次 Decode/Migrate preparation,按 revision 为键的有界 memo 让紧接的观察到恢复交接复用该解析;backend 在 memo 化前只对每个 event graph 深度冻结一次,因此后续 handle read 无需复制或再次冻结。只有写 open 才发布准备好的后继。本包有意只暴露默认插件导出与配置类型——具体类不是具名导出,因此消费方只耦合 `ctx.sessionPersistence`,其可观察行为由共享 seam 测试套件(`runPersistenceContract`/`runLiveWritePathContract`)钉住。物理修订号组合 device、inode、size 与纳秒时间戳,供准备缓存、稳定读取重试及发布检查使用。历史格式的公开修订号还加入所选路径排序后及其物理修订号的 SHA-256 指纹;锁文件与保留但未被选中的代际不参与计算。
|
|
97
101
|
|
|
98
102
|
### 物理编码
|
|
99
103
|
|
package/lib/index.js
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import z from "@deepseek-ai/schemastery";
|
|
2
|
-
import { SessionFormatUnsupportedMigrationError, sessionFormatCatalog } from "@deepseek-ai/dsh-session-format-catalog";
|
|
2
|
+
import { SessionFormatUnsupportedMigrationError, createSessionFormatCatalogWithChildren, historicalSessionFormatCatalog, sessionFormatCatalog } from "@deepseek-ai/dsh-session-format-catalog";
|
|
3
3
|
import { readdirSync } from "node:fs";
|
|
4
4
|
import { link, lstat, mkdir, mkdtemp, open, readFile, readdir, realpath, rm, stat, truncate } from "node:fs/promises";
|
|
5
5
|
import { basename, dirname, isAbsolute, join, parse, resolve, toNamespacedPath } from "node:path";
|
|
@@ -9,9 +9,9 @@ import { createHash, randomBytes } from "node:crypto";
|
|
|
9
9
|
import { SessionAlreadyExistsError, SessionAlreadyOwnedError, SessionFormatUnsupportedError, SessionHandleClosedError, SessionPersistence, SessionPersistenceCorruptionError, SessionPersistenceNotFoundError, SessionPersistenceRevision, SessionReadOnlyError, assertContiguous, assertStoredId, materializeAppendBatch, materializeCreateHeader, sessionFormatVersionRefusal, validateStoredEvents } from "@deepseek-ai/dsh-session-persistence";
|
|
10
10
|
import { BlockAssembler, errorChain, expandAssistantStream } from "@deepseek-ai/dsh-llm";
|
|
11
11
|
import { tryLockExclusive } from "@deepseek-ai/node-addon-system/flock";
|
|
12
|
-
import { SESSION_FORMAT_VERSION, Session, SessionId, SessionLogOffset } from "@deepseek-ai/dsh-session";
|
|
13
|
-
import { SessionFormatUnsupportedMigrationError as SessionFormatUnsupportedMigrationError$1, parseSessionFormatLogFilename, sessionFormatLogFilename } from "@deepseek-ai/dsh-session-format";
|
|
14
|
-
import {
|
|
12
|
+
import { KNOWN_SESSION_EVENT_TYPES, SESSION_FORMAT_VERSION, Session, SessionId, SessionLogOffset } from "@deepseek-ai/dsh-session";
|
|
13
|
+
import { SessionFormatError, SessionFormatUnsupportedMigrationError as SessionFormatUnsupportedMigrationError$1, parseSessionFormatLogFilename, sessionFormatLogFilename } from "@deepseek-ai/dsh-session-format";
|
|
14
|
+
import { assertReleasedV4Relationships, assertV4RowAdmission, historicalChildCatalogSource } from "@deepseek-ai/dsh-session-format-v3-to-v4";
|
|
15
15
|
import { constants, createZstdCompress, createZstdDecompress, zstdCompress, zstdDecompress, zstdDecompressSync } from "node:zlib";
|
|
16
16
|
import { isDeepStrictEqual, promisify } from "node:util";
|
|
17
17
|
import { constants as constants$1 } from "node:buffer";
|
|
@@ -1069,6 +1069,7 @@ var SessionLogScanner = class {
|
|
|
1069
1069
|
finish() {
|
|
1070
1070
|
this.finished = true;
|
|
1071
1071
|
const artifact = this.restore.finish();
|
|
1072
|
+
assertReleasedV4Relationships(artifact, KNOWN_SESSION_EVENT_TYPES);
|
|
1072
1073
|
return {
|
|
1073
1074
|
meta: this.meta,
|
|
1074
1075
|
inheritedEventCount: SessionLogOffset(artifact.inheritedEventCount),
|
|
@@ -1089,7 +1090,7 @@ var SessionLogScanner = class {
|
|
|
1089
1090
|
return;
|
|
1090
1091
|
}
|
|
1091
1092
|
try {
|
|
1092
|
-
|
|
1093
|
+
assertV4RowAdmission(decoded, KNOWN_SESSION_EVENT_TYPES);
|
|
1093
1094
|
} catch (error) {
|
|
1094
1095
|
if (error instanceof SessionFormatUnsupportedMigrationError$1) throw new SessionFormatUnsupportedError(error.message);
|
|
1095
1096
|
throw error;
|
|
@@ -2078,6 +2079,7 @@ async function publishPreparedMigration(options, suffix, artifact, sourceIdentit
|
|
|
2078
2079
|
const verifiedStage = await verifyCurrentFile(staged.path, compression, artifact.header.id, eventCount);
|
|
2079
2080
|
if (verifiedStage.bytes !== staged.bytes || verifiedStage.digest !== staged.digest) throw new Error("staged session generation changed during verification");
|
|
2080
2081
|
await internals.barrier("before-source-check", 1);
|
|
2082
|
+
await options.validateRelatedSources?.();
|
|
2081
2083
|
if (identity(await internals.fs.stat(sourcePath)) !== identity(sourceIdentity)) throw new JsonlGenerationSourceChangedError(sourcePath);
|
|
2082
2084
|
const published = await publishCurrentExclusive(staged.path, currentPath, internals);
|
|
2083
2085
|
if (published && internals.platform === "win32") staged = {
|
|
@@ -2126,6 +2128,7 @@ async function prepareMigration(options, internals) {
|
|
|
2126
2128
|
throw error;
|
|
2127
2129
|
}
|
|
2128
2130
|
if (artifact.header.version !== format.currentVersion) throw new Error(`format migration returned v${artifact.header.version}, expected v${format.currentVersion}`);
|
|
2131
|
+
await options.validateRelatedSources?.();
|
|
2129
2132
|
const sourceIdentity = source.identity;
|
|
2130
2133
|
let publication;
|
|
2131
2134
|
return {
|
|
@@ -2159,6 +2162,111 @@ function createJsonlGenerationRuntime(overrides = {}) {
|
|
|
2159
2162
|
};
|
|
2160
2163
|
}
|
|
2161
2164
|
const defaultGenerationRuntime = createJsonlGenerationRuntime();
|
|
2165
|
+
/**
|
|
2166
|
+
* Read one stable source through the shared streaming parser without publishing a generation.
|
|
2167
|
+
* @param path - selected source generation path.
|
|
2168
|
+
* @param version - physical source version identified by its filename.
|
|
2169
|
+
* @param compression - source encoding.
|
|
2170
|
+
* @param format - codec/restore factory, independent of current-generation publication.
|
|
2171
|
+
* @param signal - cancellation observed during source reads and decode yields.
|
|
2172
|
+
* @returns decoded artifact and physical source identity for later revalidation.
|
|
2173
|
+
* @throws SessionFormatError for physical decoding failures; storage, cancellation, and unsupported migration errors retain their category.
|
|
2174
|
+
*/
|
|
2175
|
+
async function readDecodedJsonlSource(path, version, compression, format, signal) {
|
|
2176
|
+
const source = await readStableJsonlFile(path, signal);
|
|
2177
|
+
let artifact;
|
|
2178
|
+
try {
|
|
2179
|
+
artifact = await decodeStreamingMigration(source.bytes, compression, version, format, void 0, signal);
|
|
2180
|
+
} catch (error) {
|
|
2181
|
+
if (signal?.aborted || error instanceof SessionFormatError) throw error;
|
|
2182
|
+
throw new SessionFormatError(String(error), { cause: error });
|
|
2183
|
+
}
|
|
2184
|
+
return {
|
|
2185
|
+
artifact,
|
|
2186
|
+
identity: source.identity
|
|
2187
|
+
};
|
|
2188
|
+
}
|
|
2189
|
+
//#endregion
|
|
2190
|
+
//#region lib/types/catalog-migration.js
|
|
2191
|
+
/** Collect historical discovery facts without recursively preparing related current generations. */
|
|
2192
|
+
/**
|
|
2193
|
+
* Collect each related child's own descriptor through existing historical codecs.
|
|
2194
|
+
* @param parentId - parent whose incoming migration consumes these facts.
|
|
2195
|
+
* @param sources - header-indexed direct children in the selected source corpus.
|
|
2196
|
+
* @param compression - configured source encoding.
|
|
2197
|
+
* @param signal - cancellation forwarded through each source read.
|
|
2198
|
+
* @returns compact facts and child-local failures; complete child event arrays are released after extraction.
|
|
2199
|
+
*/
|
|
2200
|
+
async function prepareCatalogFacts(parentId, sources, compression, signal) {
|
|
2201
|
+
const facts = [];
|
|
2202
|
+
const failures = [];
|
|
2203
|
+
const witnesses = [];
|
|
2204
|
+
for (const source of sources) {
|
|
2205
|
+
signal.throwIfAborted();
|
|
2206
|
+
const version = parseGenerationLogFilename(basename(source.path), compression);
|
|
2207
|
+
if (version === void 0) throw new SessionFormatUnsupportedMigrationError$1(`unrecognized historical child generation ${source.path}`);
|
|
2208
|
+
const witness = {
|
|
2209
|
+
path: source.path,
|
|
2210
|
+
identity: await stat(source.path, { bigint: true })
|
|
2211
|
+
};
|
|
2212
|
+
witnesses.push(witness);
|
|
2213
|
+
const unavailable = {
|
|
2214
|
+
childId: source.header.id,
|
|
2215
|
+
childCreatedAt: source.header.createdAt,
|
|
2216
|
+
descriptorCount: 0,
|
|
2217
|
+
descriptor: null,
|
|
2218
|
+
sourcePath: source.path
|
|
2219
|
+
};
|
|
2220
|
+
let restored;
|
|
2221
|
+
try {
|
|
2222
|
+
restored = await readDecodedJsonlSource(source.path, version, compression, { createRestore: (header) => (version <= 3 ? historicalSessionFormatCatalog : sessionFormatCatalog).createRestore(header, {
|
|
2223
|
+
recovery: "recoverable",
|
|
2224
|
+
validation: "current"
|
|
2225
|
+
}) }, signal);
|
|
2226
|
+
} catch (error) {
|
|
2227
|
+
signal.throwIfAborted();
|
|
2228
|
+
failures.push({
|
|
2229
|
+
path: source.path,
|
|
2230
|
+
error
|
|
2231
|
+
});
|
|
2232
|
+
facts.push(unavailable);
|
|
2233
|
+
continue;
|
|
2234
|
+
}
|
|
2235
|
+
witness.identity = restored.identity;
|
|
2236
|
+
const header = restored.artifact.header;
|
|
2237
|
+
if (header.id !== source.header.id || header.createdAt !== source.header.createdAt || header.parentSession !== parentId || header.origin !== "subagent" || [
|
|
2238
|
+
"cwd",
|
|
2239
|
+
"isSeeded",
|
|
2240
|
+
"delegationDepth",
|
|
2241
|
+
"agentPreset"
|
|
2242
|
+
].some((key) => header[key] !== source.header[key])) throw new JsonlGenerationSourceChangedError(source.path);
|
|
2243
|
+
let fact;
|
|
2244
|
+
try {
|
|
2245
|
+
fact = historicalChildCatalogSource(restored.artifact);
|
|
2246
|
+
} catch (error) {
|
|
2247
|
+
failures.push({
|
|
2248
|
+
path: source.path,
|
|
2249
|
+
error
|
|
2250
|
+
});
|
|
2251
|
+
facts.push(unavailable);
|
|
2252
|
+
continue;
|
|
2253
|
+
}
|
|
2254
|
+
facts.push({
|
|
2255
|
+
...fact,
|
|
2256
|
+
sourcePath: source.path
|
|
2257
|
+
});
|
|
2258
|
+
}
|
|
2259
|
+
return {
|
|
2260
|
+
facts,
|
|
2261
|
+
failures,
|
|
2262
|
+
async validate() {
|
|
2263
|
+
for (const witness of witnesses) {
|
|
2264
|
+
const current = await stat(witness.path, { bigint: true });
|
|
2265
|
+
if (current.dev !== witness.identity.dev || current.ino !== witness.identity.ino || current.size !== witness.identity.size || current.mtimeNs !== witness.identity.mtimeNs || current.ctimeNs !== witness.identity.ctimeNs) throw new JsonlGenerationSourceChangedError(witness.path);
|
|
2266
|
+
}
|
|
2267
|
+
}
|
|
2268
|
+
};
|
|
2269
|
+
}
|
|
2162
2270
|
//#endregion
|
|
2163
2271
|
//#region lib/types/index.js
|
|
2164
2272
|
/**
|
|
@@ -2187,13 +2295,17 @@ function assertZstdHeaderFrame(plaintext) {
|
|
|
2187
2295
|
}
|
|
2188
2296
|
/** Loader schema for the JSONL artifact's physical encoding. */
|
|
2189
2297
|
const JsonlCompressionSchema = z.union([z.const("zstd"), z.const("none")]).default(DEFAULT_COMPRESSION);
|
|
2190
|
-
/** Deep-freeze
|
|
2298
|
+
/** Deep-freeze acyclic stored JSON; its arrays contain only indexed JSON values. */
|
|
2191
2299
|
function freezeStoredEvent(event) {
|
|
2192
2300
|
const pending = [event];
|
|
2193
2301
|
while (pending.length > 0) {
|
|
2194
2302
|
const current = pending.pop();
|
|
2195
2303
|
Object.freeze(current);
|
|
2196
|
-
for (
|
|
2304
|
+
if (Array.isArray(current)) for (let index = 0; index < current.length; index += 1) {
|
|
2305
|
+
const child = current[index];
|
|
2306
|
+
if (child !== null && typeof child === "object") pending.push(child);
|
|
2307
|
+
}
|
|
2308
|
+
else for (const key in current) {
|
|
2197
2309
|
const child = current[key];
|
|
2198
2310
|
if (child !== null && typeof child === "object") pending.push(child);
|
|
2199
2311
|
}
|
|
@@ -2289,10 +2401,6 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2289
2401
|
this.compression = config.compression ?? DEFAULT_COMPRESSION;
|
|
2290
2402
|
this.generationFormat = {
|
|
2291
2403
|
currentVersion: sessionFormatCatalog.currentVersion,
|
|
2292
|
-
createRestore: (header) => sessionFormatCatalog.createRestore(header, {
|
|
2293
|
-
recovery: "recoverable",
|
|
2294
|
-
validation: "transformed"
|
|
2295
|
-
}),
|
|
2296
2404
|
encodeHeader: (header, inheritedEventCount) => sessionFormatCatalog.encodeCurrentHeader(header, inheritedEventCount),
|
|
2297
2405
|
encodeEvent: (event) => sessionFormatCatalog.encodeCurrentEvent(event),
|
|
2298
2406
|
isUnsupportedMigrationError: (error) => error instanceof SessionFormatUnsupportedMigrationError
|
|
@@ -2354,7 +2462,13 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2354
2462
|
materialized: false,
|
|
2355
2463
|
inheritedEventCount: pending.inheritedEventCount
|
|
2356
2464
|
}));
|
|
2357
|
-
|
|
2465
|
+
let stored;
|
|
2466
|
+
try {
|
|
2467
|
+
stored = await this.requireStoredLog(id, options?.signal);
|
|
2468
|
+
} catch (error) {
|
|
2469
|
+
if (!(error instanceof JsonlGenerationSourceChangedError)) throw error;
|
|
2470
|
+
stored = await this.requireStoredLog(id, options?.signal);
|
|
2471
|
+
}
|
|
2358
2472
|
let state;
|
|
2359
2473
|
if (stored.status === "prepared") state = {
|
|
2360
2474
|
cursor: 0,
|
|
@@ -2437,7 +2551,7 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2437
2551
|
options?.signal?.throwIfAborted();
|
|
2438
2552
|
return {
|
|
2439
2553
|
header,
|
|
2440
|
-
revision: fileRevision(identity),
|
|
2554
|
+
revision: selected.sourceVersion < SESSION_FORMAT_VERSION ? SessionPersistenceRevision(`${fileRevision(identity)}:${await this.historicalCorpusRevision(options?.signal)}`) : fileRevision(identity),
|
|
2441
2555
|
sizeBytes: Number(identity.size)
|
|
2442
2556
|
};
|
|
2443
2557
|
} catch (error) {
|
|
@@ -2457,7 +2571,9 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2457
2571
|
const snapshots = [];
|
|
2458
2572
|
const listed = /* @__PURE__ */ new Set();
|
|
2459
2573
|
const pending = [...this.tracker.pendingEntries()];
|
|
2460
|
-
|
|
2574
|
+
const artifacts = await this.listArtifacts(signal);
|
|
2575
|
+
const corpusRevision = artifacts.some((artifact) => artifact.sourceVersion < SESSION_FORMAT_VERSION) ? await this.historicalCorpusRevision(signal) : void 0;
|
|
2576
|
+
for (const artifact of artifacts) {
|
|
2461
2577
|
signal?.throwIfAborted();
|
|
2462
2578
|
try {
|
|
2463
2579
|
const identity = await stat(artifact.path, { bigint: true });
|
|
@@ -2465,7 +2581,7 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2465
2581
|
listed.add(artifact.header.id);
|
|
2466
2582
|
snapshots.push({
|
|
2467
2583
|
header: artifact.header,
|
|
2468
|
-
revision: fileRevision(identity),
|
|
2584
|
+
revision: artifact.sourceVersion < SESSION_FORMAT_VERSION ? SessionPersistenceRevision(`${fileRevision(identity)}:${corpusRevision}`) : fileRevision(identity),
|
|
2469
2585
|
sizeBytes: Number(identity.size)
|
|
2470
2586
|
});
|
|
2471
2587
|
} catch (error) {
|
|
@@ -2534,6 +2650,12 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2534
2650
|
signal.throwIfAborted();
|
|
2535
2651
|
const memoized = this.coldLogMemo.get(id);
|
|
2536
2652
|
if (memoized?.status === "prepared" && memoized.revision === sourceRevision) {
|
|
2653
|
+
try {
|
|
2654
|
+
await memoized.validateRelatedSources();
|
|
2655
|
+
} catch (error) {
|
|
2656
|
+
this.coldLogMemo.delete(id);
|
|
2657
|
+
throw this.generationFailure(id, selected, error);
|
|
2658
|
+
}
|
|
2537
2659
|
this.coldLogMemo.delete(id);
|
|
2538
2660
|
this.coldLogMemo.set(id, memoized);
|
|
2539
2661
|
return memoized;
|
|
@@ -2557,13 +2679,34 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2557
2679
|
/** Decode one historical generation without publishing a successor. */
|
|
2558
2680
|
async prepareStoredMigration(id, selected, signal) {
|
|
2559
2681
|
let prepared;
|
|
2682
|
+
let validateRelatedSources;
|
|
2560
2683
|
try {
|
|
2684
|
+
const children = async () => (await this.listArtifacts(signal)).filter((source) => source.header.origin === "subagent" && source.header.parentSession === id);
|
|
2685
|
+
const sources = await children();
|
|
2686
|
+
const related = await prepareCatalogFacts(id, sources, this.compression, signal);
|
|
2687
|
+
for (const failure of related.failures) this.ctx.logger.warn(`${this.name}: session "${id}" catalog retained a child with unknown descriptor (raw log: ${failure.path}): ${String(failure.error)}`);
|
|
2688
|
+
const membership = sources.map((source) => source.path).sort();
|
|
2689
|
+
validateRelatedSources = async () => {
|
|
2690
|
+
const current = (await children()).map((source) => source.path).sort();
|
|
2691
|
+
const before = new Set(membership);
|
|
2692
|
+
const after = new Set(current);
|
|
2693
|
+
const changed = current.find((path) => !before.has(path)) ?? membership.find((path) => !after.has(path));
|
|
2694
|
+
if (changed !== void 0) throw new JsonlGenerationSourceChangedError(changed);
|
|
2695
|
+
await related.validate();
|
|
2696
|
+
};
|
|
2561
2697
|
prepared = await prepareJsonlMigration({
|
|
2562
2698
|
sourcePath: selected.sourcePath,
|
|
2563
2699
|
sourceVersion: selected.sourceVersion,
|
|
2564
2700
|
currentPath: selected.currentPath,
|
|
2565
2701
|
compression: this.compression,
|
|
2566
|
-
format:
|
|
2702
|
+
format: {
|
|
2703
|
+
...this.generationFormat,
|
|
2704
|
+
createRestore: (header) => createSessionFormatCatalogWithChildren(related.facts).createRestore(header, {
|
|
2705
|
+
recovery: "recoverable",
|
|
2706
|
+
validation: "transformed"
|
|
2707
|
+
})
|
|
2708
|
+
},
|
|
2709
|
+
validateRelatedSources,
|
|
2567
2710
|
verifyCurrentFile: verifyCurrentGenerationInWorker,
|
|
2568
2711
|
validateHistoricalHeader: (headerValue) => this.validateSourceIdentity(selected, headerValue, id, signal),
|
|
2569
2712
|
signal
|
|
@@ -2580,6 +2723,7 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2580
2723
|
});
|
|
2581
2724
|
const stored = {
|
|
2582
2725
|
status: "prepared",
|
|
2726
|
+
validateRelatedSources,
|
|
2583
2727
|
meta,
|
|
2584
2728
|
...freezeStoredEvents(events),
|
|
2585
2729
|
tornTruncateTo: void 0,
|
|
@@ -2856,33 +3000,57 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
2856
3000
|
decoder.close();
|
|
2857
3001
|
}
|
|
2858
3002
|
}
|
|
3003
|
+
/** Enumerate selected physical generations without interpreting their headers or bodies. */
|
|
3004
|
+
async listGenerations(signal) {
|
|
3005
|
+
const sources = [];
|
|
3006
|
+
for (const project of await this.listProjectDirs(signal)) for (const dir of await this.listSessionDirs(project, signal)) {
|
|
3007
|
+
signal?.throwIfAborted();
|
|
3008
|
+
const selected = await this.resolveGenerationInDirectory(dir, signal);
|
|
3009
|
+
if (selected !== void 0) sources.push(selected);
|
|
3010
|
+
}
|
|
3011
|
+
return sources;
|
|
3012
|
+
}
|
|
3013
|
+
/** Historical logical events depend on the corpus, including members with unreadable headers. */
|
|
3014
|
+
async historicalCorpusRevision(signal) {
|
|
3015
|
+
const paths = (await this.listGenerations(signal)).map((source) => source.sourcePath).sort();
|
|
3016
|
+
const hash = createHash("sha256");
|
|
3017
|
+
for (const path of paths) {
|
|
3018
|
+
signal?.throwIfAborted();
|
|
3019
|
+
let revision;
|
|
3020
|
+
try {
|
|
3021
|
+
revision = fileRevision(await stat(path, { bigint: true }));
|
|
3022
|
+
} catch (error) {
|
|
3023
|
+
if (!isENOENT(error)) throw error;
|
|
3024
|
+
revision = "missing";
|
|
3025
|
+
}
|
|
3026
|
+
hash.update(JSON.stringify([path, revision]));
|
|
3027
|
+
}
|
|
3028
|
+
signal?.throwIfAborted();
|
|
3029
|
+
return hash.digest("hex");
|
|
3030
|
+
}
|
|
2859
3031
|
async listArtifacts(signal) {
|
|
2860
3032
|
signal?.throwIfAborted();
|
|
2861
3033
|
await this.ensureRootEncoding();
|
|
2862
3034
|
signal?.throwIfAborted();
|
|
2863
3035
|
const artifacts = [];
|
|
2864
3036
|
const ids = /* @__PURE__ */ new Set();
|
|
2865
|
-
for (const
|
|
3037
|
+
for (const selected of await this.listGenerations(signal)) {
|
|
2866
3038
|
signal?.throwIfAborted();
|
|
2867
|
-
|
|
2868
|
-
|
|
2869
|
-
|
|
2870
|
-
|
|
2871
|
-
|
|
2872
|
-
|
|
2873
|
-
header = await this.readGenerationHeader(selected, void 0, signal);
|
|
2874
|
-
} catch (error) {
|
|
2875
|
-
if (error instanceof SessionFormatUnsupportedError) continue;
|
|
2876
|
-
throw error;
|
|
2877
|
-
}
|
|
2878
|
-
if (header === void 0) continue;
|
|
2879
|
-
if (ids.has(header.id)) throw new Error(`duplicate JSONL session id "${header.id}" appears in multiple project directories`);
|
|
2880
|
-
ids.add(header.id);
|
|
2881
|
-
artifacts.push({
|
|
2882
|
-
header,
|
|
2883
|
-
path: selected.sourcePath
|
|
2884
|
-
});
|
|
3039
|
+
let header;
|
|
3040
|
+
try {
|
|
3041
|
+
header = await this.readGenerationHeader(selected, void 0, signal);
|
|
3042
|
+
} catch (error) {
|
|
3043
|
+
if (error instanceof SessionFormatUnsupportedError || error instanceof SessionPersistenceCorruptionError) continue;
|
|
3044
|
+
throw error;
|
|
2885
3045
|
}
|
|
3046
|
+
if (header === void 0) continue;
|
|
3047
|
+
if (ids.has(header.id)) throw new Error(`duplicate JSONL session id "${header.id}" appears in multiple project directories`);
|
|
3048
|
+
ids.add(header.id);
|
|
3049
|
+
artifacts.push({
|
|
3050
|
+
header,
|
|
3051
|
+
path: selected.sourcePath,
|
|
3052
|
+
sourceVersion: selected.sourceVersion
|
|
3053
|
+
});
|
|
2886
3054
|
}
|
|
2887
3055
|
signal?.throwIfAborted();
|
|
2888
3056
|
return artifacts;
|
|
@@ -3122,7 +3290,7 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
3122
3290
|
await handle.close();
|
|
3123
3291
|
}
|
|
3124
3292
|
}
|
|
3125
|
-
/** Read
|
|
3293
|
+
/** Read only the header frame; compression failures reject as corruption, while I/O and cancellation propagate. */
|
|
3126
3294
|
async readFirstZstdLine(path, signal) {
|
|
3127
3295
|
signal?.throwIfAborted();
|
|
3128
3296
|
const handle = await open(path, "r");
|
|
@@ -3138,21 +3306,18 @@ var JsonlSessionPersistence = class extends SessionPersistence {
|
|
|
3138
3306
|
signal?.throwIfAborted();
|
|
3139
3307
|
content = Buffer.concat([content, chunk.subarray(0, bytesRead)]);
|
|
3140
3308
|
signal?.throwIfAborted();
|
|
3141
|
-
const first = scanZstdFrames(content, 1).frames[0];
|
|
3142
|
-
signal?.throwIfAborted();
|
|
3143
|
-
if (first === void 0) continue;
|
|
3144
|
-
let plaintext;
|
|
3145
3309
|
try {
|
|
3310
|
+
const first = scanZstdFrames(content, 1).frames[0];
|
|
3311
|
+
if (first === void 0) continue;
|
|
3312
|
+
const plaintext = await decompressZstdFrame(content.subarray(first.start, first.end));
|
|
3146
3313
|
signal?.throwIfAborted();
|
|
3147
|
-
plaintext
|
|
3314
|
+
assertZstdHeaderFrame(plaintext);
|
|
3315
|
+
return plaintext.subarray(0, -1).toString("utf8");
|
|
3148
3316
|
} catch (error) {
|
|
3149
3317
|
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
|
|
3150
3318
|
if (signal?.aborted) signal.throwIfAborted();
|
|
3151
|
-
throw new
|
|
3319
|
+
throw new SessionPersistenceCorruptionError(`corrupt Zstandard session log: header frame failed validation: ${String(error)} (raw log: ${path})`, { cause: error });
|
|
3152
3320
|
}
|
|
3153
|
-
signal?.throwIfAborted();
|
|
3154
|
-
assertZstdHeaderFrame(plaintext);
|
|
3155
|
-
return plaintext.subarray(0, -1).toString("utf8");
|
|
3156
3321
|
}
|
|
3157
3322
|
} finally {
|
|
3158
3323
|
await handle.close();
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
/** Collect historical discovery facts without recursively preparing related current generations. */
|
|
2
|
+
import type { SessionFormatJsonObject } from '@deepseek-ai/dsh-session-format';
|
|
3
|
+
import type { SessionHeader, SessionId } from '@deepseek-ai/dsh-session';
|
|
4
|
+
import type { JsonlCompression } from './format.ts';
|
|
5
|
+
/** Per-parent supplemental facts and the source checks required before serving or publishing them. */
|
|
6
|
+
interface PreparedCatalogFacts {
|
|
7
|
+
readonly facts: readonly SessionFormatJsonObject[];
|
|
8
|
+
readonly failures: readonly {
|
|
9
|
+
readonly path: string;
|
|
10
|
+
readonly error: unknown;
|
|
11
|
+
}[];
|
|
12
|
+
/** @returns resolves while every inspected child still has the captured physical revision. */
|
|
13
|
+
validate(): Promise<void>;
|
|
14
|
+
}
|
|
15
|
+
/**
|
|
16
|
+
* Collect each related child's own descriptor through existing historical codecs.
|
|
17
|
+
* @param parentId - parent whose incoming migration consumes these facts.
|
|
18
|
+
* @param sources - header-indexed direct children in the selected source corpus.
|
|
19
|
+
* @param compression - configured source encoding.
|
|
20
|
+
* @param signal - cancellation forwarded through each source read.
|
|
21
|
+
* @returns compact facts and child-local failures; complete child event arrays are released after extraction.
|
|
22
|
+
*/
|
|
23
|
+
export declare function prepareCatalogFacts(parentId: SessionId, sources: readonly {
|
|
24
|
+
readonly header: SessionHeader;
|
|
25
|
+
readonly path: string;
|
|
26
|
+
}[], compression: JsonlCompression, signal: AbortSignal): Promise<PreparedCatalogFacts>;
|
|
27
|
+
export {};
|
|
28
|
+
//# sourceMappingURL=catalog-migration.d.ts.map
|
|
@@ -24,6 +24,8 @@ export interface JsonlGenerationFormatAdapter {
|
|
|
24
24
|
}
|
|
25
25
|
/** Inputs for preparing one historical generation and publishing its current successor later. */
|
|
26
26
|
export interface PrepareJsonlMigrationOptions {
|
|
27
|
+
/** Revalidate related source facts before preparation returns and immediately before publication. */
|
|
28
|
+
readonly validateRelatedSources?: () => Promise<void>;
|
|
27
29
|
/** Immutable generation selected by the backend resolver. */
|
|
28
30
|
readonly sourcePath: string;
|
|
29
31
|
/** Version selected from the source filename and independently checked against its header. */
|
|
@@ -159,5 +161,19 @@ export declare function prepareJsonlMigration(options: PrepareJsonlMigrationOpti
|
|
|
159
161
|
* @returns bound generation operations.
|
|
160
162
|
*/
|
|
161
163
|
export declare function createJsonlGenerationRuntime(overrides?: JsonlGenerationRuntimeOverrides): JsonlGenerationRuntime;
|
|
164
|
+
/**
|
|
165
|
+
* Read one stable source through the shared streaming parser without publishing a generation.
|
|
166
|
+
* @param path - selected source generation path.
|
|
167
|
+
* @param version - physical source version identified by its filename.
|
|
168
|
+
* @param compression - source encoding.
|
|
169
|
+
* @param format - codec/restore factory, independent of current-generation publication.
|
|
170
|
+
* @param signal - cancellation observed during source reads and decode yields.
|
|
171
|
+
* @returns decoded artifact and physical source identity for later revalidation.
|
|
172
|
+
* @throws SessionFormatError for physical decoding failures; storage, cancellation, and unsupported migration errors retain their category.
|
|
173
|
+
*/
|
|
174
|
+
export declare function readDecodedJsonlSource(path: string, version: number, compression: JsonlCompression, format: Pick<JsonlGenerationFormatAdapter, 'createRestore'>, signal?: AbortSignal): Promise<{
|
|
175
|
+
artifact: SessionFormatArtifact;
|
|
176
|
+
identity: JsonlPhysicalIdentity;
|
|
177
|
+
}>;
|
|
162
178
|
export {};
|
|
163
179
|
//# sourceMappingURL=generation.d.ts.map
|
package/lib/types/index.d.ts
CHANGED
|
@@ -199,6 +199,10 @@ declare class JsonlSessionPersistence extends SessionPersistence {
|
|
|
199
199
|
acquireWriteLease(header: SessionHeader): Promise<SessionWriteLease>;
|
|
200
200
|
/** Decode complete frames and retain complete JSONL records from a torn final frame. */
|
|
201
201
|
private readZstdPrefix;
|
|
202
|
+
/** Enumerate selected physical generations without interpreting their headers or bodies. */
|
|
203
|
+
private listGenerations;
|
|
204
|
+
/** Historical logical events depend on the corpus, including members with unreadable headers. */
|
|
205
|
+
private historicalCorpusRevision;
|
|
202
206
|
private listArtifacts;
|
|
203
207
|
/** Read and translate one selected generation header without inspecting its body. */
|
|
204
208
|
private readGenerationHeader;
|
|
@@ -231,7 +235,7 @@ declare class JsonlSessionPersistence extends SessionPersistence {
|
|
|
231
235
|
* Reads in bounded chunks so a huge log costs only the header read.
|
|
232
236
|
*/
|
|
233
237
|
private readFirstLine;
|
|
234
|
-
/** Read
|
|
238
|
+
/** Read only the header frame; compression failures reject as corruption, while I/O and cancellation propagate. */
|
|
235
239
|
private readFirstZstdLine;
|
|
236
240
|
/** Select the numerically highest canonical generation in one Session directory. */
|
|
237
241
|
private resolveGenerationInDirectory;
|