driftseal 2.1.0 → 3.1.0

package/README.md

@@ -39,7 +39,8 @@ DriftSeal v2 is an outcome log rather than an intent-per-step log.
 
 ## Install
 
-DriftSeal requires Node.js 18 or newer.
+DriftSeal requires Node.js 22.13 or newer. Its derived outcome index uses the
+built-in `node:sqlite` module; no native database package is installed.
 
 ```sh
 npm install --global driftseal
@@ -65,8 +66,9 @@ fresh clone because Git config is local to each clone.
 
 Use `driftseal init --lang <BCP-47-tag>` to choose the prose language stored in
 outcome and MADR records. Use `--local-log` only when `.seal/` should remain
-untracked; DriftSeal reports tracked state but does not edit `.gitignore` or the
-Git index.
+untracked; DriftSeal reports tracked state but does not edit the repository-root
+`.gitignore` or the Git index. When the seal sits inside a Git worktree, `init`
+may still write `.seal/outcomes/.gitignore` so derived sidecars stay untracked.
 
 ## Core workflow
 
@@ -155,13 +157,25 @@ has, `status`, `log`, and `lane` fall back to `main` with a warning; `begin`
 still refuses until you `lane switch main` or add the lane. `log --last N` can
 return more than N records when an open outcome sits on another lane.
 
-A derived lane index caches fold state in Git metadata (or beside a custom
-seal). Incremental rebuilds follow `indexedThrough` and `indexedLines`; a full
-rebuild happens when the log identity changes. Per-lane heads, reverse links,
-and WAL byte ranges are stored for a seek path that is not consumed yet. The
-index is reconstructable and is not committed with the log. Custom-home
-sidecars sit next to `events.jsonl`. When that directory is inside a Git
-worktree, they are listed in its `.gitignore`.
+A derived SQLite outcome index stores folded records beside the WAL
+(`outcomes/.outcome-index.sqlite`). It is a disposable read model;
+`events.jsonl` remains the only canonical history. Incremental sync follows
+`indexedThrough` and `indexedLines`. File identity makes unchanged hot reads
+constant-time, while a full checksum of the previously indexed WAL prefix
+validates every incremental catch-up. A source rewrite, incompatible schema,
+malformed indexed row, or SQLite read error triggers a full rebuild.
+
+`log --last N` uses a `(lane, reclaimed, ordinal)` SQLite index and reads only
+the selected outcome rows plus open outcomes from other lanes. Parked events
+are checked and folded as a targeted overlay using indexed committed event
+identities, so an open outcome does not force a committed-WAL scan. When a
+lock-free read sees a missing or stale database, DriftSeal folds the canonical
+WAL in memory instead of serving stale index data. Stored WAL byte ranges remain
+available for future projections but are not required for recent-log lookup.
+The database is reconstructable and is not committed with the log. Sidecars sit
+next to `events.jsonl`; when that directory is inside a Git worktree, they are
+listed in its `.gitignore`. Keeping the file in the workspace avoids agent
+sandboxes that deny writes under `.git/`.
 
 ## Decisions and MADR
 
@@ -280,9 +294,11 @@ migrated state.
 
 ## Git and merge behavior
 
-In a Git worktree, `begin` parks the open outcome in Git metadata so it does not
-dirty the tracked log. `end` flushes the lineage to
-`.seal/outcomes/events.jsonl`. The event log is append-only during normal work.
+In a default Git-repository seal, `begin` parks the open outcome beside the WAL
+(`.seal/outcomes/.in-progress.jsonl`, gitignored) so it does not dirty the
+tracked log. Custom `$DRIFTSEAL_HOME` seals write that open outcome directly to
+`events.jsonl`. `end` flushes a parked lineage to `.seal/outcomes/events.jsonl`.
+The event log is append-only during normal work.
 
 After a merge collision, run:
 
@@ -329,11 +345,19 @@ MADR tools, and the three migration tools. Resources are:
   DriftSeal; use `reclaim`, `unreclaim`, `lane`, and `absorb` instead of manual edits.
 - `.seal/madr/` stores numbered MADR documents.
 - `$DRIFTSEAL_HOME` replaces the `.seal` root.
-- The current lane and derived lane index live in Git metadata for a default
-  repository seal, or beside a custom seal (`outcomes/.current-lane` and
-  `outcomes/.lane-index.json`). When the custom seal sits inside a Git
-  worktree, those sidecars are gitignored. They are reconstructable and are not
-  part of the committed WAL.
+- The derived SQLite outcome index and the current-lane pointer sit next to the
+  WAL (`outcomes/.outcome-index.sqlite`, `outcomes/.current-lane`). The index is
+  reconstructable from `events.jsonl`; a missing or stale current-lane pointer
+  falls back to `main`. A default repository seal also parks the open outcome and
+  local verification provenance beside the WAL
+  (`outcomes/.in-progress.jsonl`, `outcomes/.driftseal-local-outcome.json`). Those
+  two files are not reconstructable: deleting the park discards the still-open
+  outcome, and deleting provenance changes verifier trust. Custom
+  `$DRIFTSEAL_HOME` seals write open outcomes directly to the WAL and still keep
+  provenance beside that log. When the outcomes directory is inside a Git
+  worktree, the sidecars are gitignored. A default repository seal keeps one
+  ignored copy per worktree; a shared `$DRIFTSEAL_HOME` shares the files that
+  exist there.
 - Advisory hooks remind agents about lifecycle state but never broaden the
   repository's `AGENTS.md` policy.
 

package/README.zh-CN.md

@@ -33,7 +33,8 @@ DriftSeal v2 从“按步骤记录 intent”改为“按交付记录 outcome”
 
 ## 安装
 
-DriftSeal 需要 Node.js 18 或更高版本。
+DriftSeal 需要 Node.js 22.13 或更高版本。派生 outcome index 使用 Node 内置的
+`node:sqlite`，不会安装 native database package。
 
 ```sh
 npm install --global driftseal
@@ -59,7 +60,8 @@ attribute，并配置本地 Git merge driver。Git config 不会随 clone 传播
 
 `driftseal init --lang <BCP-47-tag>` 用来指定 outcome 与 MADR 正文的语言。
 `--local-log` 会让 `.seal/` 保持本地、不被跟踪；DriftSeal 只报告当前 tracked 状态，
-不会替你修改 `.gitignore` 或 Git index。
+不会替你修改仓库根目录的 `.gitignore` 或 Git index。seal 在 Git worktree 内时，
+`init` 仍可能写入 `.seal/outcomes/.gitignore`，用来忽略派生 sidecar。
 
 ## 基本工作流
 
@@ -140,11 +142,21 @@ lane 不能改名或删除。`lane add` 打错的名字会一直出现在 `drift
 直到 `lane switch main` 或把该 lane 加回来。`log --last N` 在 open outcome 属于
 别的 lane 时，返回条数可以多于 N。
 
-派生的 lane index 缓存 fold 状态，放在 Git metadata（或自定义 seal 旁边）。
-增量重建跟随 `indexedThrough` 和 `indexedLines`；log 身份变化时全量重建。每条
-lane 的 head、反向链接和 WAL byte range 会写入 index，供尚未接入的 seek 路径使用。
-它可以重建，不会随 log 一起提交。自定义 home 下的 sidecar 放在 `events.jsonl`
-旁边；该目录在 Git worktree 内时，由目录里的 `.gitignore` 忽略。
+派生的 SQLite outcome index 放在 WAL 旁边（`outcomes/.outcome-index.sqlite`）。
+它只是可以随时删除重建的 read model；`events.jsonl` 仍是唯一 canonical history。
+增量同步跟随 `indexedThrough` 和 `indexedLines`。未变化的 hot read 用 file
+identity 做常量时间校验；增量追赶前会校验此前全部 WAL prefix 的 checksum。
+source 被改写、schema 不兼容、indexed row 损坏或 SQLite read 失败时，DriftSeal
+会全量重建。
+
+`log --last N` 使用 `(lane, reclaimed, ordinal)` SQLite index，只读取命中的 outcome
+row，并补上其他 lane 的 open outcome。parked event 会依据 index 中保存的 committed
+event identity 做定向 overlay，不再因此重扫 committed WAL。无锁读取遇到缺失或 stale
+database 时，会在内存中 fold canonical WAL，绝不会返回 stale index。保存的 WAL byte
+range 留给后续 projection 使用，但 recent-log lookup 不依赖它。database 可以重建，
+不会随 log 一起提交。sidecar 放在 `events.jsonl` 旁边；该目录在 Git worktree
+内时，由目录里的 `.gitignore` 忽略。文件留在工作树里，避免被默认禁止写入
+`.git/` 的 agent sandbox 拦住。
 
 ## Decision 与 MADR
 
@@ -253,9 +265,11 @@ tools 可以立刻看到 migration 后的状态。
 
 ## Git 与 merge
 
-在 Git worktree 中，`begin` 会把 open outcome park 到 Git metadata，避免弄脏 tracked
-log；`end` 再把完整 lineage 写入 `.seal/outcomes/events.jsonl`。正常工作期间 event log
-保持 append-only。
+在默认 Git-repository seal 中，`begin` 会把 open outcome park 到 WAL 旁边
+（`.seal/outcomes/.in-progress.jsonl`，已被 gitignore），避免弄脏 tracked log。
+自定义 `$DRIFTSEAL_HOME` 会把该 open outcome 直接写入 `events.jsonl`。`end`
+再把 parked lineage flush 到 `.seal/outcomes/events.jsonl`。正常工作期间 event
+log 保持 append-only。
 
 发生 merge collision 后执行：
 
@@ -301,10 +315,16 @@ resources 为：
   `absorb`，不要手改。
 - `.seal/madr/` 保存编号化 MADR。
 - `$DRIFTSEAL_HOME` 替换整个 `.seal` root。
-- 当前 lane 与派生 lane index 对默认 repo seal 存在 Git metadata 里，对自定义 seal
-  则放在旁边（`outcomes/.current-lane` 与 `outcomes/.lane-index.json`）。自定义
-  seal 在 Git worktree 内时，这些 sidecar 会被 gitignore。它们可以重建，不是
-  committed WAL 的一部分。
+- 派生 SQLite outcome index 和当前 lane 指针放在 WAL 旁边
+  （`outcomes/.outcome-index.sqlite`、`outcomes/.current-lane`）。index 可以从
+  `events.jsonl` 重建；缺失或过期的 current-lane 会回退到 `main`。默认 repo
+  seal 还会把 open outcome 和本地 verification provenance park 在 WAL 旁边
+  （`outcomes/.in-progress.jsonl`、`outcomes/.driftseal-local-outcome.json`）。
+  这两份文件不能重建：删掉 park 会丢掉尚未 flush 的 open outcome，删掉
+  provenance 会改变 verifier 信任。自定义 `$DRIFTSEAL_HOME` 会把 open outcome
+  直接写入 WAL，provenance 仍放在该 log 旁边。outcomes 目录在 Git worktree 内时，
+  这些 sidecar 会被 gitignore。默认 repo seal 下每个 worktree 各自一份；共享的
+  `$DRIFTSEAL_HOME` 会共用那里实际存在的文件。
 - advisory hook 只提示 lifecycle 状态，不会扩大 repo 中 `AGENTS.md` 的政策边界。
 
 DriftSeal 不会替你判断 verification command 是否安全，也不会判断测试本身是否充分。

package/benchmark/recent-log.js

@@ -0,0 +1,181 @@
+'use strict';
+
+const assert = require('node:assert/strict');
+const { execFileSync } = require('node:child_process');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const { performance } = require('node:perf_hooks');
+const { openOutcomeIndex } = require('../lib/outcome-index-sqlite.js');
+
+const DRIFTSEAL = path.join(__dirname, '..', 'bin', 'driftseal.js');
+const quick = process.argv.includes('--quick');
+const large = process.argv.includes('--large');
+const unrelatedOutcomes = large ? 100_000 : quick ? 2_000 : 10_000;
+const laneOutcomes = large ? 1_000 : quick ? 20 : 100;
+const samples = quick ? 5 : 20;
+const root = fs.mkdtempSync(path.join(os.tmpdir(), 'driftseal-recent-log-benchmark-'));
+const home = path.join(root, 'seal');
+const outcomeDirectory = path.join(home, 'outcomes');
+const wal = path.join(outcomeDirectory, 'events.jsonl');
+const indexFile = path.join(outcomeDirectory, '.outcome-index.sqlite');
+const env = {
+  ...process.env,
+  DRIFTSEAL_HOME: home,
+  DRIFTSEAL_DECISION_HOME: path.join(home, 'madr'),
+};
+
+function runCli(args, envOverrides = {}) {
+  return execFileSync(process.execPath, [DRIFTSEAL, ...args], {
+    cwd: root,
+    encoding: 'utf8',
+    env: { ...env, ...envOverrides },
+    stdio: ['ignore', 'pipe', 'pipe'],
+    maxBuffer: 64 * 1024 * 1024,
+  });
+}
+
+function event(type, fields) {
+  return {
+    logVersion: 2,
+    schemaVersion: type === 'lane_add' || fields.lane ? 2 : 1,
+    type,
+    ...fields,
+  };
+}
+
+function outcomeEvents(id, outcome, lane, minute) {
+  const beganAt = new Date(Date.UTC(2026, 0, 1, 0, minute, 0)).toISOString();
+  const endedAt = new Date(Date.UTC(2026, 0, 1, 0, minute, 1)).toISOString();
+  return [
+    event('begin', {
+      id,
+      ts: beganAt,
+      outcome,
+      acceptance: [],
+      verify: null,
+      decisions: [],
+      ...(lane === 'main' ? {} : { lane }),
+    }),
+    event('end', {
+      id,
+      ts: endedAt,
+      status: 'abandoned',
+      note: `closed ${outcome}`,
+      verifyResult: null,
+      head: null,
+    }),
+  ];
+}
+
+function writeSyntheticWal() {
+  fs.mkdirSync(outcomeDirectory, { recursive: true });
+  const lines = [];
+  let sequence = 1;
+  for (let i = 0; i < unrelatedOutcomes; i++) {
+    const id = `2026-01-01-${String(sequence++).padStart(6, '0')}`;
+    lines.push(...outcomeEvents(id, `unrelated work ${i}`, 'main', i % 1440));
+  }
+  lines.push(
+    event('lane_add', {
+      id: 'lane:focus',
+      lane: 'focus',
+      description: 'Benchmark focus lane',
+      ts: '2026-01-02T00:00:00.000Z',
+    })
+  );
+  for (let i = 0; i < laneOutcomes; i++) {
+    const id = `2026-01-02-${String(sequence++).padStart(6, '0')}`;
+    lines.push(...outcomeEvents(id, `focus work ${i}`, 'focus', i));
+  }
+  fs.writeFileSync(wal, `${lines.map(JSON.stringify).join('\n')}\n`);
+  fs.writeFileSync(path.join(outcomeDirectory, '.current-lane'), 'focus\n');
+}
+
+function distribution(values) {
+  const sorted = [...values].sort((left, right) => left - right);
+  return {
+    p50: Number(sorted[Math.floor(sorted.length * 0.5)].toFixed(2)),
+    p95: Number(sorted[Math.min(sorted.length - 1, Math.floor(sorted.length * 0.95))].toFixed(2)),
+  };
+}
+
+function measure(action, count = samples) {
+  const durations = [];
+  let value;
+  for (let i = 0; i < count; i++) {
+    const started = performance.now();
+    value = action();
+    durations.push(performance.now() - started);
+  }
+  return { ...distribution(durations), value };
+}
+
+try {
+  writeSyntheticWal();
+  const cold = measure(() => runCli(['log', '--last', '3']), 1);
+  const hot = measure(() => runCli(['log', '--last', '3']));
+  const fullFold = measure(
+    () =>
+      runCli(['log', '--last', '3'], {
+        _DRIFTSEAL_TEST_DISABLE_OUTCOME_INDEX: '1',
+      }),
+    quick ? 1 : 3
+  );
+  assert.equal(hot.value, fullFold.value);
+  assert.match(hot.value, new RegExp(`focus work ${laneOutcomes - 1}`));
+  assert.doesNotMatch(hot.value, /unrelated work/);
+
+  const index = openOutcomeIndex(indexFile, { readOnly: true });
+  const direct = measure(() => index.queryRecent('focus', 3));
+  const queryPlan = index.explainRecent().map((row) => row.detail);
+  index.close();
+  assert.equal(direct.value.length, 3);
+  assert.ok(
+    queryPlan.some((detail) => /outcomes_lane_visible_ordinal/.test(detail)),
+    `expected recent query index in plan: ${queryPlan.join('; ')}`
+  );
+
+  const nextId = `2026-01-03-${String(unrelatedOutcomes + laneOutcomes + 1).padStart(6, '0')}`;
+  fs.appendFileSync(
+    wal,
+    `${outcomeEvents(nextId, 'incremental focus work', 'focus', 0)
+      .map(JSON.stringify)
+      .join('\n')}\n`
+  );
+  const incremental = measure(() => runCli(['log', '--last', '3']), 1);
+  assert.match(incremental.value, /incremental focus work/);
+
+  const startup = measure(() =>
+    execFileSync(process.execPath, ['-e', ''], {
+      cwd: root,
+      stdio: 'ignore',
+    })
+  );
+  const stats = {
+    mode: large ? 'large' : quick ? 'quick' : 'default',
+    node: process.version,
+    platform: `${process.platform}-${process.arch}`,
+    outcomes: {
+      unrelated: unrelatedOutcomes,
+      currentLane: laneOutcomes,
+    },
+    samples,
+    bytes: {
+      wal: fs.statSync(wal).size,
+      sqlite: fs.statSync(indexFile).size,
+    },
+    milliseconds: {
+      processStartup: { p50: startup.p50, p95: startup.p95 },
+      coldBuildAndRecent: { p50: cold.p50, p95: cold.p95 },
+      hotRecentCli: { p50: hot.p50, p95: hot.p95 },
+      hotRecentQuery: { p50: direct.p50, p95: direct.p95 },
+      fullFoldRecent: { p50: fullFold.p50, p95: fullFold.p95 },
+      incrementalAndRecent: { p50: incremental.p50, p95: incremental.p95 },
+    },
+    queryPlan,
+  };
+  process.stdout.write(`${JSON.stringify(stats, null, 2)}\n`);
+} finally {
+  fs.rmSync(root, { recursive: true, force: true });
+}