@tangle-network/harness-sessions 0.1.3 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +12 -2
- package/dist/catalog.d.ts +27 -0
- package/dist/catalog.js +39 -0
- package/dist/formats/claude-code.js +9 -11
- package/dist/formats/codex.js +82 -28
- package/dist/formats/factory.js +2 -2
- package/dist/formats/head.d.ts +5 -2
- package/dist/formats/head.js +59 -13
- package/dist/formats/opencode.d.ts +1 -1
- package/dist/formats/opencode.js +25 -15
- package/dist/formats/pi.js +2 -2
- package/dist/index.d.ts +2 -1
- package/dist/index.js +2 -1
- package/dist/schema.d.ts +1 -1
- package/dist/source.d.ts +5 -16
- package/dist/source.js +76 -33
- package/dist/stores.d.ts +13 -1
- package/dist/stores.js +71 -4
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -22,10 +22,19 @@ npm install @tangle-network/harness-sessions
|
|
|
22
22
|
| Harness | Format | Store under HOME | Served model from | Usage |
|
|
23
23
|
|---|---|---|---|---|
|
|
24
24
|
| `claude-code` | `claude-code.projects-jsonl` | `.claude/projects/*/{id}.jsonl`, subagents at `*/{id}/subagents/**/*.jsonl` | provider response (`message.model`) | per response, counted once per `message.id` |
|
|
25
|
-
| `codex` | `codex.rollout-jsonl` |
|
|
25
|
+
| `codex` | `codex.rollout-jsonl` | `rollout-*-{id}.jsonl` under `sessions/` and `archived_sessions/` of every Codex home (below) | turn context (the configured model) | per response (`last_token_usage`); session total from `total_token_usage` |
|
|
26
26
|
| `opencode` | `opencode.sqlite` | `.local/share/opencode/opencode.db` (shared by every session) | session record (`providerID/modelID`) | per step (`step-finish`) |
|
|
27
27
|
| `pi` | `pi.session-jsonl` | `.pi/agent/sessions/**/*_{id}.jsonl` | session record (`provider/model`) | per assistant message |
|
|
28
28
|
|
|
29
|
+
A HOME can hold several Codex homes, and `codexReader.locate` reads all of them (`codexHomes(home)` lists them):
|
|
30
|
+
|
|
31
|
+
- `.codex` and each sibling `.codex-<name>`;
|
|
32
|
+
- each acct account home, `.config/agent-accounts/codex/<account>`;
|
|
33
|
+
- `home` itself when it is a Codex home by its path (`isCodexHome`), such as a fleet seat's `codex` directory passed as a store root;
|
|
34
|
+
- for this process's own HOME only, `$CODEX_HOME` and `$ACCT_CONFIG/codex/<account>` in place of the default accounts directory. Another HOME, such as an archived host's store root or a capture, is read by its layout alone.
|
|
35
|
+
|
|
36
|
+
Homes that resolve to one directory count once. A session copied into several homes (`acct move`) is one ref, its largest copy. A lookup by id takes the rollout whose name ends in the id; when no name does, it reads the `session_meta` of each rollout modified since the id's UUIDv7 time.
|
|
37
|
+
|
|
29
38
|
Readers for Kimi, Factory, Gemini, Amp, Qwen, Copilot, Forge, Hermes, Prime and OpenClaw are added with the same real-data proof.
|
|
30
39
|
|
|
31
40
|
## Use
|
|
@@ -61,9 +70,10 @@ harness-sessions read <file|home|capture> [--harness h] [--session id] [--summar
|
|
|
61
70
|
- The served model comes only from the provider response or the session record, never from the profile or the request. A failed request has no served model; `requestedModel` says which model it was sent to.
|
|
62
71
|
- `usage` is `null` when the harness recorded none; it is never reported as zero. `input` excludes cache reads and cache writes, which have their own fields.
|
|
63
72
|
- A session with zero model calls is not the trace of a turn.
|
|
73
|
+
- `locate(home, { cwd })` keeps a session whose recorded cwd is `cwd` or lies under it on a path boundary (`withinPath`): `/w/agent` matches `/w/agent/sub`, never `/w/agent-dev-container`. Every store is resolved against the `home` given; only the Codex environment overrides above depend on the process.
|
|
64
74
|
- An unparsable record is counted in `integrity.unparsedRecords` and never stops a read (`corruption: 'strict'` throws instead). A record cut off at the end of a live copy is reported as a torn tail, not as corruption.
|
|
65
75
|
- `ending.status` is `completed`, `error`, `aborted` or `open`. `open` means the records stop mid-turn: the turn was still running or the process died.
|
|
66
|
-
- An OpenCode store is read
|
|
76
|
+
- An OpenCode store is opened read-only where it lies, with no copy: SQLite never writes the store or its write-ahead log. With a log present it reads the log's committed rows and keeps only its own shared-memory index (`-shm`) current; without one the store is opened immutable and nothing is created beside it. A store SQLite cannot open in place (a log with no `-shm` in a directory it cannot write, such as a capture on read-only storage, or any store under Node before 22.15, whose `node:sqlite` takes no URI file names) is read from a private copy. `integrity.sourceFiles` names the session as `<store>#session=<id>` with a digest of the rows read for it.
|
|
67
77
|
|
|
68
78
|
## Tests
|
|
69
79
|
|
package/dist/catalog.d.ts
CHANGED
|
@@ -153,6 +153,33 @@ export declare const EVIDENCE: {
|
|
|
153
153
|
readonly endingError: 'none';
|
|
154
154
|
};
|
|
155
155
|
};
|
|
156
|
+
/**
|
|
157
|
+
* Codex writes to whichever Codex home it runs with, so one HOME can hold several: `~/.codex`, a
|
|
158
|
+
* sibling such as `~/.codex-lane`, the `$CODEX_HOME` a process set, one acct home per account under
|
|
159
|
+
* `${ACCT_CONFIG:-~/.config/agent-accounts}/codex/<account>`, and on a fleet host a seat's `codex`
|
|
160
|
+
* directory. Every home keeps live rollouts in `sessions/` and archived ones in
|
|
161
|
+
* `archived_sessions/`. `STORES` declares the default home's live store, which is what a sandbox
|
|
162
|
+
* captures; the readers search every home. The same rule as tangle-tools
|
|
163
|
+
* `trace-archive/catalog.py` `classify`.
|
|
164
|
+
*/
|
|
165
|
+
export declare const CODEX_HOME_RULE: {
|
|
166
|
+
/** Directory names directly under HOME that are Codex homes: `.codex` and `.codex-<name>`. */
|
|
167
|
+
readonly siblings: RegExp;
|
|
168
|
+
/** Under HOME, unless `$ACCT_CONFIG` names it: each child is one account's Codex home. */
|
|
169
|
+
readonly accounts: '.config/agent-accounts/codex';
|
|
170
|
+
readonly sessionDirs: readonly ['sessions', 'archived_sessions'];
|
|
171
|
+
/** A rollout's file name, at any depth under a session dir (`sessions/` nests by date). */
|
|
172
|
+
readonly rollout: 'rollout-*.jsonl';
|
|
173
|
+
/** The name Codex gives a rollout, ending in its thread id. */
|
|
174
|
+
readonly named: 'rollout-*-{id}.jsonl';
|
|
175
|
+
};
|
|
176
|
+
/**
|
|
177
|
+
* True when `dir` is itself a Codex home by its path: `.codex`, `.codex-<name>`, an acct account
|
|
178
|
+
* home (`agent-accounts/codex/<account>`), or a fleet seat's `codex` directory.
|
|
179
|
+
*/
|
|
180
|
+
export declare function isCodexHome(dir: string): boolean;
|
|
181
|
+
/** True when `path` is `root` or lies under it on a path boundary: `/x/agent` is not under `/x/age`. */
|
|
182
|
+
export declare function withinPath(path: string, root: string): boolean;
|
|
156
183
|
/**
|
|
157
184
|
* Compile one store glob. `*` matches within a path segment, `**` matches any number of
|
|
158
185
|
* segments, and `{id}` matches the native session id: the literal id when one is given, any
|
package/dist/catalog.js
CHANGED
|
@@ -120,6 +120,45 @@ export const EVIDENCE = {
|
|
|
120
120
|
'prime.session-jsonl': { servedModel: 'session-record', usage: 'per-response', endingError: 'session-record' },
|
|
121
121
|
'openclaw.session-jsonl': { servedModel: 'session-record', usage: 'per-response', endingError: 'none' },
|
|
122
122
|
};
|
|
123
|
+
/**
|
|
124
|
+
* Codex writes to whichever Codex home it runs with, so one HOME can hold several: `~/.codex`, a
|
|
125
|
+
* sibling such as `~/.codex-lane`, the `$CODEX_HOME` a process set, one acct home per account under
|
|
126
|
+
* `${ACCT_CONFIG:-~/.config/agent-accounts}/codex/<account>`, and on a fleet host a seat's `codex`
|
|
127
|
+
* directory. Every home keeps live rollouts in `sessions/` and archived ones in
|
|
128
|
+
* `archived_sessions/`. `STORES` declares the default home's live store, which is what a sandbox
|
|
129
|
+
* captures; the readers search every home. The same rule as tangle-tools
|
|
130
|
+
* `trace-archive/catalog.py` `classify`.
|
|
131
|
+
*/
|
|
132
|
+
export const CODEX_HOME_RULE = {
|
|
133
|
+
/** Directory names directly under HOME that are Codex homes: `.codex` and `.codex-<name>`. */
|
|
134
|
+
siblings: /^\.codex(?:-.+)?$/u,
|
|
135
|
+
/** Under HOME, unless `$ACCT_CONFIG` names it: each child is one account's Codex home. */
|
|
136
|
+
accounts: '.config/agent-accounts/codex',
|
|
137
|
+
sessionDirs: ['sessions', 'archived_sessions'],
|
|
138
|
+
/** A rollout's file name, at any depth under a session dir (`sessions/` nests by date). */
|
|
139
|
+
rollout: 'rollout-*.jsonl',
|
|
140
|
+
/** The name Codex gives a rollout, ending in its thread id. */
|
|
141
|
+
named: 'rollout-*-{id}.jsonl',
|
|
142
|
+
};
|
|
143
|
+
/**
|
|
144
|
+
* True when `dir` is itself a Codex home by its path: `.codex`, `.codex-<name>`, an acct account
|
|
145
|
+
* home (`agent-accounts/codex/<account>`), or a fleet seat's `codex` directory.
|
|
146
|
+
*/
|
|
147
|
+
export function isCodexHome(dir) {
|
|
148
|
+
const parts = dir.replace(/\/+$/u, '').split('/');
|
|
149
|
+
const [name, parent, grandparent] = [parts.at(-1) ?? '', parts.at(-2) ?? '', parts.at(-3) ?? ''];
|
|
150
|
+
if (CODEX_HOME_RULE.siblings.test(name))
|
|
151
|
+
return true;
|
|
152
|
+
if (grandparent === 'agent-accounts' && parent === 'codex')
|
|
153
|
+
return true;
|
|
154
|
+
return name === 'codex' && parent !== 'agent-accounts';
|
|
155
|
+
}
|
|
156
|
+
/** True when `path` is `root` or lies under it on a path boundary: `/x/agent` is not under `/x/age`. */
|
|
157
|
+
export function withinPath(path, root) {
|
|
158
|
+
const p = path.replace(/\/+$/u, '');
|
|
159
|
+
const r = root.replace(/\/+$/u, '');
|
|
160
|
+
return p === r || p.startsWith(`${r}/`);
|
|
161
|
+
}
|
|
123
162
|
const REGEX_SPECIALS = /[.+^$()|[\]\\]/gu;
|
|
124
163
|
/**
|
|
125
164
|
* Compile one store glob. `*` matches within a path segment, `**` matches any number of
|
|
@@ -12,7 +12,7 @@ import { basename, join } from 'node:path';
|
|
|
12
12
|
import { SessionBuilder, count, isRecord, sessionError, str, textOf, usageOrNull } from '../builder.js';
|
|
13
13
|
import { fileRecords } from '../source.js';
|
|
14
14
|
import { runFold } from './fold.js';
|
|
15
|
-
import { EVIDENCE, STORES, mtimeMs, storeGlob, walkFiles } from '../stores.js';
|
|
15
|
+
import { EVIDENCE, LOCATE_READS, STORES, mapLimit, mtimeMs, storeGlob, walkFiles, withinPath } from '../stores.js';
|
|
16
16
|
import { headRecord } from './head.js';
|
|
17
17
|
import { claudeRecordIsMachine, claudeTypedPromptText } from './claude-records.js';
|
|
18
18
|
const FORMAT = 'claude-code.projects-jsonl';
|
|
@@ -254,27 +254,25 @@ async function locate(home, opts) {
|
|
|
254
254
|
const key = `${project}/${id}`;
|
|
255
255
|
subagentsBySession.set(key, [...(subagentsBySession.get(key) ?? []), join(root, rel)]);
|
|
256
256
|
}
|
|
257
|
-
const
|
|
258
|
-
for (const rel of all) {
|
|
259
|
-
if (!main.test(rel))
|
|
260
|
-
continue;
|
|
257
|
+
const sessions = await mapLimit(all.filter((rel) => main.test(rel)), LOCATE_READS, async (rel) => {
|
|
261
258
|
const path = join(root, rel);
|
|
262
259
|
const id = basename(rel, '.jsonl');
|
|
263
260
|
const mtime = await mtimeMs(path);
|
|
264
261
|
if (opts.sinceMs && mtime < opts.sinceMs)
|
|
265
|
-
|
|
262
|
+
return [];
|
|
266
263
|
const head = await headRecord(path, (r) => typeof r.cwd === 'string');
|
|
267
264
|
const cwd = head ? str(head.cwd) : null;
|
|
268
|
-
if (opts.cwd && (cwd === null || !cwd
|
|
269
|
-
|
|
265
|
+
if (opts.cwd && (cwd === null || !withinPath(cwd, opts.cwd)))
|
|
266
|
+
return [];
|
|
270
267
|
const subagents = subagentsBySession.get(`${rel.split('/')[0]}/${id}`) ?? [];
|
|
271
|
-
refs
|
|
268
|
+
const refs = [{ harness: HARNESS, format: FORMAT, nativeSessionId: id, path, files: [path, ...subagents], home, cwd, mtimeMs: mtime, parentNativeSessionId: null }];
|
|
272
269
|
for (const file of subagents) {
|
|
273
270
|
const agentId = basename(file, '.jsonl').replace(/^agent-/u, '');
|
|
274
271
|
refs.push({ harness: HARNESS, format: FORMAT, nativeSessionId: agentId, path: file, files: [file], home, cwd, mtimeMs: await mtimeMs(file), parentNativeSessionId: id });
|
|
275
272
|
}
|
|
276
|
-
|
|
277
|
-
|
|
273
|
+
return refs;
|
|
274
|
+
});
|
|
275
|
+
return sessions.flat().sort((a, b) => b.mtimeMs - a.mtimeMs);
|
|
278
276
|
}
|
|
279
277
|
export const claudeCodeReader = {
|
|
280
278
|
harness: HARNESS,
|
package/dist/formats/codex.js
CHANGED
|
@@ -11,10 +11,11 @@
|
|
|
11
11
|
*/
|
|
12
12
|
import { basename } from 'node:path';
|
|
13
13
|
import { SessionBuilder, argumentsOf, count, isRecord, sessionError, str, textOf, usageOrNull, } from '../builder.js';
|
|
14
|
-
import { fileRecords } from '../source.js';
|
|
14
|
+
import { fileRecords, isMissing } from '../source.js';
|
|
15
15
|
import { runFold } from './fold.js';
|
|
16
|
-
import { EVIDENCE, STORES,
|
|
16
|
+
import { CODEX_HOME_RULE, EVIDENCE, LOCATE_READS, STORES, codexHomes, mapLimit, storeGlob, walkFiles, withinPath } from '../stores.js';
|
|
17
17
|
import { headRecord } from './head.js';
|
|
18
|
+
import { stat } from 'node:fs/promises';
|
|
18
19
|
import { join } from 'node:path';
|
|
19
20
|
const FORMAT = 'codex.rollout-jsonl';
|
|
20
21
|
const STORE = STORES[FORMAT];
|
|
@@ -316,38 +317,91 @@ function httpStatusOf(payload) {
|
|
|
316
317
|
}
|
|
317
318
|
return null;
|
|
318
319
|
}
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
320
|
+
const ROLLOUT = storeGlob(CODEX_HOME_RULE.rollout);
|
|
321
|
+
const UUID_V7 = /^([0-9a-f]{8})-([0-9a-f]{4})-7[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/iu;
|
|
322
|
+
/** Every rollout file, live and archived, in every Codex home under `home`. */
|
|
323
|
+
async function rollouts(home) {
|
|
324
|
+
const files = [];
|
|
325
|
+
for (const codexHome of await codexHomes(home)) {
|
|
326
|
+
for (const dir of CODEX_HOME_RULE.sessionDirs) {
|
|
327
|
+
const root = join(codexHome, dir);
|
|
328
|
+
for (const rel of await walkFiles(root))
|
|
329
|
+
if (ROLLOUT.test(basename(rel)))
|
|
330
|
+
files.push(join(root, rel));
|
|
331
|
+
}
|
|
332
|
+
}
|
|
333
|
+
return files;
|
|
334
|
+
}
|
|
335
|
+
/** A UUIDv7's creation time: a session's rollout is written at or after its thread id is made. */
|
|
336
|
+
function uuidV7Ms(id) {
|
|
337
|
+
const match = UUID_V7.exec(id);
|
|
338
|
+
return match ? Number.parseInt(`${match[1]}${match[2]}`, 16) : 0;
|
|
339
|
+
}
|
|
340
|
+
/** A ref for each of `files` that passes the filters, from its `session_meta`, with the file's bytes. */
|
|
341
|
+
async function refsOf(files, home, opts) {
|
|
342
|
+
const found = await mapLimit(files, LOCATE_READS, async (path) => {
|
|
343
|
+
let info;
|
|
344
|
+
try {
|
|
345
|
+
info = await stat(path);
|
|
346
|
+
}
|
|
347
|
+
catch (error) {
|
|
348
|
+
if (isMissing(error))
|
|
349
|
+
return null;
|
|
350
|
+
throw error;
|
|
351
|
+
}
|
|
352
|
+
if (opts.sinceMs && info.mtimeMs < opts.sinceMs)
|
|
353
|
+
return null;
|
|
330
354
|
const meta = await headRecord(path, (r) => r.type === 'session_meta', 4);
|
|
331
355
|
const payload = meta && isRecord(meta.payload) ? meta.payload : {};
|
|
332
356
|
const cwd = str(payload.cwd);
|
|
333
|
-
if (opts.cwd && (cwd === null || !cwd
|
|
334
|
-
|
|
335
|
-
const id = str(payload.id) ?? ID_IN_NAME.exec(rel)?.[1] ?? basename(rel, '.jsonl');
|
|
357
|
+
if (opts.cwd && (cwd === null || !withinPath(cwd, opts.cwd)))
|
|
358
|
+
return null;
|
|
336
359
|
const source = isRecord(payload.source) ? payload.source : {};
|
|
337
360
|
const spawn = isRecord(source.subagent) && isRecord(source.subagent.thread_spawn) ? source.subagent.thread_spawn : {};
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
361
|
+
return {
|
|
362
|
+
bytes: info.size,
|
|
363
|
+
ref: {
|
|
364
|
+
harness: HARNESS,
|
|
365
|
+
format: FORMAT,
|
|
366
|
+
nativeSessionId: str(payload.id) ?? ID_IN_NAME.exec(path)?.[1] ?? basename(path, '.jsonl'),
|
|
367
|
+
path,
|
|
368
|
+
files: [path],
|
|
369
|
+
home,
|
|
370
|
+
cwd,
|
|
371
|
+
mtimeMs: info.mtimeMs,
|
|
372
|
+
parentNativeSessionId: str(payload.parent_thread_id) ?? str(spawn.parent_thread_id),
|
|
373
|
+
},
|
|
374
|
+
};
|
|
375
|
+
});
|
|
376
|
+
return found.filter((entry) => entry !== null);
|
|
377
|
+
}
|
|
378
|
+
/**
|
|
379
|
+
* Sessions in every Codex home under `home`. A lookup by id takes the rollout named for it; when no
|
|
380
|
+
* name carries the id, it reads the `session_meta` of each rollout written since the id was made.
|
|
381
|
+
* A session copied into several homes (`acct move`) is one ref, its largest copy, which holds
|
|
382
|
+
* every record.
|
|
383
|
+
*/
|
|
384
|
+
async function locate(home, opts) {
|
|
385
|
+
const files = await rollouts(home);
|
|
386
|
+
const id = opts.nativeSessionId;
|
|
387
|
+
let found;
|
|
388
|
+
if (id === undefined) {
|
|
389
|
+
found = await refsOf(files, home, opts);
|
|
390
|
+
}
|
|
391
|
+
else {
|
|
392
|
+
const named = storeGlob(CODEX_HOME_RULE.named, id);
|
|
393
|
+
const byName = files.filter((path) => named.test(basename(path)));
|
|
394
|
+
found = byName.length > 0
|
|
395
|
+
? await refsOf(byName, home, opts)
|
|
396
|
+
: (await refsOf(files, home, { ...opts, sinceMs: Math.max(opts.sinceMs ?? 0, uuidV7Ms(id)) })).filter((r) => r.ref.nativeSessionId === id);
|
|
397
|
+
}
|
|
398
|
+
const largest = new Map();
|
|
399
|
+
for (const copy of found) {
|
|
400
|
+
const kept = largest.get(copy.ref.nativeSessionId);
|
|
401
|
+
if (!kept || copy.bytes > kept.bytes || (copy.bytes === kept.bytes && copy.ref.mtimeMs > kept.ref.mtimeMs))
|
|
402
|
+
largest.set(copy.ref.nativeSessionId, copy);
|
|
349
403
|
}
|
|
350
|
-
return
|
|
404
|
+
return [...largest.values()].map((c) => c.ref).sort((a, b) => b.mtimeMs - a.mtimeMs);
|
|
351
405
|
}
|
|
352
406
|
export const codexReader = {
|
|
353
407
|
harness: HARNESS,
|
package/dist/formats/factory.js
CHANGED
|
@@ -12,7 +12,7 @@ import { readFile } from 'node:fs/promises';
|
|
|
12
12
|
import { basename, join } from 'node:path';
|
|
13
13
|
import { SessionBuilder, count, isRecord, str, textOf, usageOrNull } from '../builder.js';
|
|
14
14
|
import { fileRecords, isMissing } from '../source.js';
|
|
15
|
-
import { EVIDENCE, STORES, mtimeMs, storeGlob, walkFiles } from '../stores.js';
|
|
15
|
+
import { EVIDENCE, STORES, mtimeMs, storeGlob, walkFiles, withinPath } from '../stores.js';
|
|
16
16
|
import { runFold } from './fold.js';
|
|
17
17
|
import { headRecord } from './head.js';
|
|
18
18
|
const FORMAT = 'factory.session-jsonl';
|
|
@@ -128,7 +128,7 @@ async function locate(home, opts) {
|
|
|
128
128
|
const head = await headRecord(ref.path, (r) => r.type === 'session_start', 1);
|
|
129
129
|
const cwd = head ? str(head.cwd) : null;
|
|
130
130
|
ref.cwd = cwd;
|
|
131
|
-
if (!cwd || !cwd
|
|
131
|
+
if (!cwd || !withinPath(cwd, opts.cwd))
|
|
132
132
|
continue;
|
|
133
133
|
}
|
|
134
134
|
filtered.push(ref);
|
package/dist/formats/head.d.ts
CHANGED
|
@@ -1,5 +1,8 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* The first record among a session file's first `limit` records that satisfies `match
|
|
3
|
-
*
|
|
2
|
+
* The first record among a session file's first `limit` records that satisfies `match`. Locate
|
|
3
|
+
* reads one for every session in a store, so it reads the file a small step at a time and stops
|
|
4
|
+
* at the answer: never the rest of the file, and no digest of it. Lines parse as the session
|
|
5
|
+
* reader parses them: a blank, malformed or non-object line is skipped, and a last line with no
|
|
6
|
+
* newline counts when it is a whole record.
|
|
4
7
|
*/
|
|
5
8
|
export declare function headRecord(path: string, match: (record: Record<string, unknown>) => boolean, limit?: number): Promise<Record<string, unknown> | null>;
|
package/dist/formats/head.js
CHANGED
|
@@ -1,23 +1,69 @@
|
|
|
1
|
-
import {
|
|
1
|
+
import { open } from 'node:fs/promises';
|
|
2
|
+
import { StringDecoder } from 'node:string_decoder';
|
|
3
|
+
import { isMissing } from '../source.js';
|
|
4
|
+
/** Bytes a head read takes at a time. A session's identifying record is almost always in the first. */
|
|
5
|
+
const HEAD_CHUNK = 64 * 1024;
|
|
2
6
|
/**
|
|
3
|
-
* The first record among a session file's first `limit` records that satisfies `match
|
|
4
|
-
*
|
|
7
|
+
* The first record among a session file's first `limit` records that satisfies `match`. Locate
|
|
8
|
+
* reads one for every session in a store, so it reads the file a small step at a time and stops
|
|
9
|
+
* at the answer: never the rest of the file, and no digest of it. Lines parse as the session
|
|
10
|
+
* reader parses them: a blank, malformed or non-object line is skipped, and a last line with no
|
|
11
|
+
* newline counts when it is a whole record.
|
|
5
12
|
*/
|
|
6
13
|
export async function headRecord(path, match, limit = 64) {
|
|
7
|
-
let
|
|
14
|
+
let handle;
|
|
8
15
|
try {
|
|
9
|
-
|
|
10
|
-
if (match(record))
|
|
11
|
-
return record;
|
|
12
|
-
seen += 1;
|
|
13
|
-
if (seen >= limit)
|
|
14
|
-
return null;
|
|
15
|
-
}
|
|
16
|
+
handle = await open(path, 'r');
|
|
16
17
|
}
|
|
17
18
|
catch (error) {
|
|
18
|
-
if (error
|
|
19
|
+
if (isMissing(error))
|
|
19
20
|
return null;
|
|
20
21
|
throw error;
|
|
21
22
|
}
|
|
22
|
-
|
|
23
|
+
let seen = 0;
|
|
24
|
+
// undefined: read on; null: `limit` records without a match; a record: the match.
|
|
25
|
+
const take = (line) => {
|
|
26
|
+
const text = line.replace(/\r$/u, '');
|
|
27
|
+
if (text.trim().length === 0)
|
|
28
|
+
return undefined;
|
|
29
|
+
let value;
|
|
30
|
+
try {
|
|
31
|
+
value = JSON.parse(text);
|
|
32
|
+
}
|
|
33
|
+
catch {
|
|
34
|
+
return undefined;
|
|
35
|
+
}
|
|
36
|
+
if (value === null || typeof value !== 'object' || Array.isArray(value))
|
|
37
|
+
return undefined;
|
|
38
|
+
if (match(value))
|
|
39
|
+
return value;
|
|
40
|
+
seen += 1;
|
|
41
|
+
return seen >= limit ? null : undefined;
|
|
42
|
+
};
|
|
43
|
+
try {
|
|
44
|
+
const decoder = new StringDecoder('utf8');
|
|
45
|
+
const buffer = Buffer.allocUnsafe(HEAD_CHUNK);
|
|
46
|
+
// The line being read, in pieces, so a record longer than a step is joined once.
|
|
47
|
+
let pending = [];
|
|
48
|
+
let position = 0;
|
|
49
|
+
for (;;) {
|
|
50
|
+
const { bytesRead } = await handle.read(buffer, 0, HEAD_CHUNK, position);
|
|
51
|
+
if (bytesRead === 0)
|
|
52
|
+
return take(pending.join('') + decoder.end()) ?? null;
|
|
53
|
+
position += bytesRead;
|
|
54
|
+
const text = decoder.write(buffer.subarray(0, bytesRead));
|
|
55
|
+
let start = 0;
|
|
56
|
+
for (let end = text.indexOf('\n'); end !== -1; end = text.indexOf('\n', start)) {
|
|
57
|
+
const found = take(pending.join('') + text.slice(start, end));
|
|
58
|
+
pending = [];
|
|
59
|
+
if (found !== undefined)
|
|
60
|
+
return found;
|
|
61
|
+
start = end + 1;
|
|
62
|
+
}
|
|
63
|
+
pending.push(text.slice(start));
|
|
64
|
+
}
|
|
65
|
+
}
|
|
66
|
+
finally {
|
|
67
|
+
await handle.close();
|
|
68
|
+
}
|
|
23
69
|
}
|
|
@@ -1,4 +1,4 @@
|
|
|
1
1
|
import type { HarnessSessionReader, LocateOptions, SessionRef } from '../schema.js';
|
|
2
|
-
/** Every session in one store file,
|
|
2
|
+
/** Every session in one store file, read in place without writing it. */
|
|
3
3
|
export declare function opencodeSessionsInStore(path: string, home: string | null, opts?: LocateOptions): Promise<SessionRef[]>;
|
|
4
4
|
export declare const opencodeReader: HarnessSessionReader;
|
package/dist/formats/opencode.js
CHANGED
|
@@ -6,13 +6,15 @@
|
|
|
6
6
|
* `data` blob: role, the provider and model the message was sent to, tokens, cost, finish and the
|
|
7
7
|
* error that ended it. `part` rows carry the content: `text`, `reasoning`, `tool` (the call and
|
|
8
8
|
* its result in one row), `step-start` and `step-finish` (one model response, with its tokens and
|
|
9
|
-
* cost), `retry` (a failed attempt) and bookkeeping parts. The store is read
|
|
10
|
-
*
|
|
9
|
+
* cost), `retry` (a failed attempt) and bookkeeping parts. The store is opened read-only where it
|
|
10
|
+
* lies, so a read sees the rows a running OpenCode has committed to its write-ahead log and never
|
|
11
|
+
* writes the store, its log or a copy of it.
|
|
11
12
|
*/
|
|
13
|
+
import { createHash } from 'node:crypto';
|
|
12
14
|
import { join } from 'node:path';
|
|
13
15
|
import { SessionBuilder, count, isRecord, isoTime, sessionError, str, usageOrNull } from '../builder.js';
|
|
14
|
-
import { isMissing,
|
|
15
|
-
import { EVIDENCE, STORES, mtimeMs } from '../stores.js';
|
|
16
|
+
import { isMissing, withSqlite } from '../source.js';
|
|
17
|
+
import { EVIDENCE, STORES, mtimeMs, withinPath } from '../stores.js';
|
|
16
18
|
const FORMAT = 'opencode.sqlite';
|
|
17
19
|
const STORE = STORES[FORMAT];
|
|
18
20
|
const HARNESS = 'opencode';
|
|
@@ -55,11 +57,21 @@ function foldSession(db, ref, mode, builder) {
|
|
|
55
57
|
builder.cwd = session.directory;
|
|
56
58
|
builder.parentNativeSessionId = session.parent_id ?? null;
|
|
57
59
|
builder.seen(session.time_created);
|
|
60
|
+
// The session's source is the rows this fold reads, not the whole store: a digest of the store
|
|
61
|
+
// file would read every other session in it and change whenever any of them did.
|
|
62
|
+
const digest = createHash('sha256').update(JSON.stringify(session));
|
|
63
|
+
let bytes = 0;
|
|
64
|
+
const note = (row) => {
|
|
65
|
+
const data = String(row.data);
|
|
66
|
+
digest.update(`\n${row.id}\t${row.time_created}\t`).update(data);
|
|
67
|
+
bytes += Buffer.byteLength(data);
|
|
68
|
+
};
|
|
58
69
|
const messages = db.prepare('SELECT id, time_created, data FROM message WHERE session_id = ? ORDER BY time_created, id').all(ref.nativeSessionId);
|
|
59
70
|
const partsOf = db.prepare('SELECT id, time_created, data FROM part WHERE message_id = ? ORDER BY id');
|
|
60
71
|
let unparsed = 0;
|
|
61
72
|
let aborted = null;
|
|
62
73
|
for (const row of messages) {
|
|
74
|
+
note(row);
|
|
63
75
|
const data = parse(row.data);
|
|
64
76
|
if (data === null) {
|
|
65
77
|
unparsed += 1;
|
|
@@ -70,6 +82,7 @@ function foldSession(db, ref, mode, builder) {
|
|
|
70
82
|
builder.seen(time.completed);
|
|
71
83
|
const parts = [];
|
|
72
84
|
for (const partRow of partsOf.iterate(row.id)) {
|
|
85
|
+
note(partRow);
|
|
73
86
|
const part = parse(partRow.data);
|
|
74
87
|
if (part === null)
|
|
75
88
|
unparsed += 1;
|
|
@@ -177,10 +190,11 @@ function foldSession(db, ref, mode, builder) {
|
|
|
177
190
|
aborted = messageError?.kind === 'MessageAbortedError' ? { at: messageError.at, error: messageError } : null;
|
|
178
191
|
}
|
|
179
192
|
const children = db.prepare('SELECT id FROM session WHERE parent_id = ? ORDER BY time_created, id').all(ref.nativeSessionId);
|
|
180
|
-
for (const child of children)
|
|
193
|
+
for (const child of children) {
|
|
194
|
+
digest.update(`\n${child.id}`);
|
|
181
195
|
builder.child(child.id, null);
|
|
182
|
-
|
|
183
|
-
|
|
196
|
+
}
|
|
197
|
+
builder.source({ path: `${ref.path}#session=${ref.nativeSessionId}`, sha256: `sha256:${digest.digest('hex')}`, bytes, unparsed });
|
|
184
198
|
const last = builder.modelCalls[builder.modelCalls.length - 1];
|
|
185
199
|
if (aborted && last?.error?.kind === 'MessageAbortedError')
|
|
186
200
|
last.error = null;
|
|
@@ -189,27 +203,23 @@ function foldSession(db, ref, mode, builder) {
|
|
|
189
203
|
async function fold(ref, mode, options) {
|
|
190
204
|
options.signal?.throwIfAborted();
|
|
191
205
|
const builder = new SessionBuilder(HARNESS, FORMAT, EVIDENCE[FORMAT].servedModel, mode, ref.nativeSessionId, ref.parentNativeSessionId);
|
|
192
|
-
await
|
|
193
|
-
for (const source of sources)
|
|
194
|
-
builder.source(source);
|
|
195
|
-
foldSession(db, ref, mode, builder);
|
|
196
|
-
});
|
|
206
|
+
await withSqlite(ref.path, (db) => foldSession(db, ref, mode, builder));
|
|
197
207
|
if (ref.truncated)
|
|
198
208
|
builder.markTruncated(`${ref.path}: the store copy was cut short`);
|
|
199
209
|
return builder;
|
|
200
210
|
}
|
|
201
|
-
/** Every session in one store file,
|
|
211
|
+
/** Every session in one store file, read in place without writing it. */
|
|
202
212
|
export async function opencodeSessionsInStore(path, home, opts = {}) {
|
|
203
213
|
const mtime = await mtimeMs(path);
|
|
204
214
|
const files = [path];
|
|
205
215
|
for (const suffix of ['-wal', '-shm'])
|
|
206
216
|
if ((await mtimeMs(`${path}${suffix}`)) > 0)
|
|
207
217
|
files.push(`${path}${suffix}`);
|
|
208
|
-
return
|
|
218
|
+
return withSqlite(path, (db) => {
|
|
209
219
|
const rows = db.prepare('SELECT id, parent_id, directory, time_updated FROM session ORDER BY time_updated DESC').all();
|
|
210
220
|
return rows
|
|
211
221
|
.filter((row) => (!opts.nativeSessionId || row.id === opts.nativeSessionId)
|
|
212
|
-
&& (!opts.cwd || row.directory
|
|
222
|
+
&& (!opts.cwd || withinPath(row.directory, opts.cwd))
|
|
213
223
|
&& (!opts.sinceMs || row.time_updated >= opts.sinceMs))
|
|
214
224
|
.map((row) => ({ harness: HARNESS, format: FORMAT, nativeSessionId: row.id, path, files, home, cwd: row.directory, mtimeMs: row.time_updated || mtime, parentNativeSessionId: row.parent_id }));
|
|
215
225
|
});
|
package/dist/formats/pi.js
CHANGED
|
@@ -12,7 +12,7 @@ import { basename, join } from 'node:path';
|
|
|
12
12
|
import { SessionBuilder, count, isRecord, sessionError, str, textOf, usageOrNull } from '../builder.js';
|
|
13
13
|
import { fileRecords } from '../source.js';
|
|
14
14
|
import { runFold } from './fold.js';
|
|
15
|
-
import { EVIDENCE, STORES, mtimeMs, storeGlob, walkFiles } from '../stores.js';
|
|
15
|
+
import { EVIDENCE, STORES, mtimeMs, storeGlob, walkFiles, withinPath } from '../stores.js';
|
|
16
16
|
import { headRecord } from './head.js';
|
|
17
17
|
const FORMAT = 'pi.session-jsonl';
|
|
18
18
|
const STORE = STORES[FORMAT];
|
|
@@ -172,7 +172,7 @@ async function locate(home, opts) {
|
|
|
172
172
|
continue;
|
|
173
173
|
const header = await headRecord(path, (r) => r.type === 'session', 1);
|
|
174
174
|
const cwd = header ? str(header.cwd) : null;
|
|
175
|
-
if (opts.cwd && (cwd === null || !cwd
|
|
175
|
+
if (opts.cwd && (cwd === null || !withinPath(cwd, opts.cwd)))
|
|
176
176
|
continue;
|
|
177
177
|
const id = (header && str(header.id)) ?? idFromName(rel);
|
|
178
178
|
if (opts.nativeSessionId && id !== opts.nativeSessionId && idFromName(rel) !== opts.nativeSessionId)
|
package/dist/index.d.ts
CHANGED
|
@@ -10,7 +10,8 @@
|
|
|
10
10
|
*/
|
|
11
11
|
export type { HarnessSession, HarnessSessionReader, LocateOptions, ModelCall, NativeSessionCopy, NativeSessionStore, Part, RawEvidenceArchiveManifestV2, ReadOptions, SessionEnding, SessionError, SessionEvidenceSources, SessionFormatId, SessionIntegrity, SessionMessage, SessionRef, SessionSummary, SessionToolCall, TokenUsage, } from './schema.js';
|
|
12
12
|
export { HARNESS_SESSION_SCHEMA } from './schema.js';
|
|
13
|
-
export { EVIDENCE, SESSION_FORMAT_IDS, STORES, assertStoreRoot, storeGlob } from './catalog.js';
|
|
13
|
+
export { CODEX_HOME_RULE, EVIDENCE, SESSION_FORMAT_IDS, STORES, assertStoreRoot, isCodexHome, storeGlob, withinPath } from './catalog.js';
|
|
14
|
+
export { codexHomes } from './stores.js';
|
|
14
15
|
export { knownHarnesses, listReaders, maybeReaderFor, readerFor, readerForFormat } from './registry.js';
|
|
15
16
|
export { claudeCodeReader, claudeCodeRefForFile } from './formats/claude-code.js';
|
|
16
17
|
export { claudeRecordIsMachine, claudeTypedPromptText, type ClaudeRecordFacts } from './formats/claude-records.js';
|
package/dist/index.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
export { HARNESS_SESSION_SCHEMA } from './schema.js';
|
|
2
|
-
export { EVIDENCE, SESSION_FORMAT_IDS, STORES, assertStoreRoot, storeGlob } from './catalog.js';
|
|
2
|
+
export { CODEX_HOME_RULE, EVIDENCE, SESSION_FORMAT_IDS, STORES, assertStoreRoot, isCodexHome, storeGlob, withinPath } from './catalog.js';
|
|
3
|
+
export { codexHomes } from './stores.js';
|
|
3
4
|
export { knownHarnesses, listReaders, maybeReaderFor, readerFor, readerForFormat } from './registry.js';
|
|
4
5
|
export { claudeCodeReader, claudeCodeRefForFile } from './formats/claude-code.js';
|
|
5
6
|
export { claudeRecordIsMachine, claudeTypedPromptText } from './formats/claude-records.js';
|
package/dist/schema.d.ts
CHANGED
|
@@ -201,7 +201,7 @@ export interface SessionRef {
|
|
|
201
201
|
}
|
|
202
202
|
export interface LocateOptions {
|
|
203
203
|
nativeSessionId?: string;
|
|
204
|
-
/** Sessions whose recorded cwd
|
|
204
|
+
/** Sessions whose recorded cwd is this path or lies under it on a path boundary (`/x/agent` excludes `/x/agent-dev-container`). */
|
|
205
205
|
cwd?: string;
|
|
206
206
|
/** Sessions modified at or after this epoch ms. */
|
|
207
207
|
sinceMs?: number;
|
package/dist/source.d.ts
CHANGED
|
@@ -46,22 +46,11 @@ export declare function textRecordsSync(text: string, stats: SourceStats, option
|
|
|
46
46
|
}): Generator<Record<string, unknown>>;
|
|
47
47
|
/** Records a caller already parsed (one per line, in order). Non-objects count as unparsed. */
|
|
48
48
|
export declare function arrayRecordsSync(values: readonly unknown[], stats: SourceStats): Generator<Record<string, unknown>>;
|
|
49
|
-
/** Hash a file without parsing it. */
|
|
50
|
-
export declare function hashFile(path: string): Promise<{
|
|
51
|
-
sha256: string;
|
|
52
|
-
bytes: number;
|
|
53
|
-
}>;
|
|
54
49
|
/**
|
|
55
|
-
* Run `use` against a
|
|
56
|
-
*
|
|
57
|
-
*
|
|
58
|
-
*
|
|
50
|
+
* Run `use` against a SQLite store opened read-only in place ({@link openSqlite}). A store SQLite
|
|
51
|
+
* cannot open in place is read from a private copy instead: a write-ahead log with no `-shm` in a
|
|
52
|
+
* directory SQLite cannot write (a capture on read-only storage), or a Node whose `node:sqlite`
|
|
53
|
+
* does not accept URI file names (before 22.15).
|
|
59
54
|
*/
|
|
60
|
-
export declare function
|
|
61
|
-
path: string;
|
|
62
|
-
sha256: string;
|
|
63
|
-
bytes: number;
|
|
64
|
-
}>) => T): Promise<T>;
|
|
65
|
-
/** Remove the private store copy now instead of at process exit. */
|
|
66
|
-
export declare function releaseSqliteCopy(): void;
|
|
55
|
+
export declare function withSqlite<T>(path: string, use: (db: DatabaseSync) => T): Promise<T>;
|
|
67
56
|
export declare function isMissing(error: unknown): boolean;
|
package/dist/source.js
CHANGED
|
@@ -1,9 +1,10 @@
|
|
|
1
1
|
import { createHash } from 'node:crypto';
|
|
2
2
|
import { createReadStream, rmSync } from 'node:fs';
|
|
3
|
-
import { copyFile, mkdtemp, stat } from 'node:fs/promises';
|
|
3
|
+
import { copyFile, mkdtemp, realpath, stat } from 'node:fs/promises';
|
|
4
4
|
import { createRequire } from 'node:module';
|
|
5
5
|
import { tmpdir } from 'node:os';
|
|
6
6
|
import { join } from 'node:path';
|
|
7
|
+
import { pathToFileURL } from 'node:url';
|
|
7
8
|
const MAX_UNPARSED_LINES = 50;
|
|
8
9
|
function noteUnparsed(stats, line) {
|
|
9
10
|
stats.unparsed += 1;
|
|
@@ -157,25 +158,79 @@ export function* arrayRecordsSync(values, stats) {
|
|
|
157
158
|
yield value;
|
|
158
159
|
}
|
|
159
160
|
}
|
|
160
|
-
/** Hash a file without parsing it. */
|
|
161
|
-
export async function hashFile(path) {
|
|
162
|
-
const hash = createHash('sha256');
|
|
163
|
-
let bytes = 0;
|
|
164
|
-
for await (const chunk of createReadStream(path)) {
|
|
165
|
-
hash.update(chunk);
|
|
166
|
-
bytes += chunk.length;
|
|
167
|
-
}
|
|
168
|
-
return { sha256: `sha256:${hash.digest('hex')}`, bytes };
|
|
169
|
-
}
|
|
170
161
|
// `node:sqlite` is loaded through `createRequire`: bundlers rewrite a dynamic `import()` of a
|
|
171
162
|
// `node:` builtin under some targets, and a require obtained this way is not analyzable by them.
|
|
172
163
|
function loadSqlite() {
|
|
173
164
|
const nodeRequire = createRequire(import.meta.url);
|
|
174
165
|
return nodeRequire('node:sqlite');
|
|
175
166
|
}
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
167
|
+
async function exists(path) {
|
|
168
|
+
try {
|
|
169
|
+
await stat(path);
|
|
170
|
+
return true;
|
|
171
|
+
}
|
|
172
|
+
catch (error) {
|
|
173
|
+
if (isMissing(error))
|
|
174
|
+
return false;
|
|
175
|
+
throw error;
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
/**
|
|
179
|
+
* Open a SQLite store read-only where it lies. A store with a write-ahead log beside it is opened
|
|
180
|
+
* `mode=ro`: SQLite reads the log's committed tail and coordinates with a live writer through the
|
|
181
|
+
* store's shared-memory index (`-shm`), and never writes the store or its log or checkpoints it. A
|
|
182
|
+
* store without a log is opened `immutable=1`: no connection holds it open, the file holds every
|
|
183
|
+
* committed row, and SQLite creates no `-wal` or `-shm` beside it. Throws when SQLite cannot read
|
|
184
|
+
* the store in place.
|
|
185
|
+
*/
|
|
186
|
+
async function openSqlite(path) {
|
|
187
|
+
const mode = (await exists(`${path}-wal`)) ? 'mode=ro' : 'immutable=1';
|
|
188
|
+
const { DatabaseSync } = loadSqlite();
|
|
189
|
+
// `readOnly` as well: a Node that treated the URI as a file name would fail to open, not create it.
|
|
190
|
+
const db = new DatabaseSync(`${pathToFileURL(path).href}?${mode}`, { readOnly: true });
|
|
191
|
+
try {
|
|
192
|
+
db.exec('PRAGMA busy_timeout = 5000');
|
|
193
|
+
// The first read opens the log and its index, so a store SQLite cannot read in place fails here.
|
|
194
|
+
db.prepare('PRAGMA schema_version').get();
|
|
195
|
+
return db;
|
|
196
|
+
}
|
|
197
|
+
catch (error) {
|
|
198
|
+
db.close();
|
|
199
|
+
throw error;
|
|
200
|
+
}
|
|
201
|
+
}
|
|
202
|
+
/** SQLite's primary result code `SQLITE_CANTOPEN`, which `node:sqlite` errors carry in `errcode`. */
|
|
203
|
+
function cannotOpen(error) {
|
|
204
|
+
const code = error?.errcode;
|
|
205
|
+
return typeof code === 'number' && (code & 0xff) === 14;
|
|
206
|
+
}
|
|
207
|
+
/**
|
|
208
|
+
* Run `use` against a SQLite store opened read-only in place ({@link openSqlite}). A store SQLite
|
|
209
|
+
* cannot open in place is read from a private copy instead: a write-ahead log with no `-shm` in a
|
|
210
|
+
* directory SQLite cannot write (a capture on read-only storage), or a Node whose `node:sqlite`
|
|
211
|
+
* does not accept URI file names (before 22.15).
|
|
212
|
+
*/
|
|
213
|
+
export async function withSqlite(path, use) {
|
|
214
|
+
// SQLite follows a symlinked store to its target and keeps the log there, so the log is looked
|
|
215
|
+
// for there too. A missing store throws ENOENT here, which callers read as "no sessions".
|
|
216
|
+
const store = await realpath(path);
|
|
217
|
+
const db = await openSqlite(store).catch((error) => {
|
|
218
|
+
if (cannotOpen(error))
|
|
219
|
+
return null;
|
|
220
|
+
throw error;
|
|
221
|
+
});
|
|
222
|
+
if (db === null)
|
|
223
|
+
return withSqliteCopy(store, use);
|
|
224
|
+
try {
|
|
225
|
+
return use(db);
|
|
226
|
+
}
|
|
227
|
+
finally {
|
|
228
|
+
db.close();
|
|
229
|
+
}
|
|
230
|
+
}
|
|
231
|
+
// One private copy is kept per process: reading every session of a store SQLite cannot open in
|
|
232
|
+
// place copies it once, not once per session. It is replaced when a different store, or a store
|
|
233
|
+
// that changed, is read, and removed when the process exits.
|
|
179
234
|
let current = null;
|
|
180
235
|
function dropCopy() {
|
|
181
236
|
if (current === null)
|
|
@@ -206,47 +261,35 @@ async function storeKey(path) {
|
|
|
206
261
|
return `${path}|${parts.join('|')}`;
|
|
207
262
|
}
|
|
208
263
|
/**
|
|
209
|
-
* Run `use` against a private copy of a SQLite store
|
|
210
|
-
*
|
|
211
|
-
*
|
|
212
|
-
* replace the copy while it runs.
|
|
264
|
+
* Run `use` against a private copy of a SQLite store: the store and its `-wal` and `-shm` siblings
|
|
265
|
+
* are copied and only the copy is opened, writable, so SQLite replays the copied log into it. `use`
|
|
266
|
+
* is synchronous so no other read can replace the copy while it runs.
|
|
213
267
|
*/
|
|
214
|
-
|
|
268
|
+
async function withSqliteCopy(path, use) {
|
|
215
269
|
const key = await storeKey(path);
|
|
216
270
|
if (current?.key !== key) {
|
|
217
271
|
const { DatabaseSync } = loadSqlite();
|
|
218
272
|
const dir = await mkdtemp(join(tmpdir(), 'harness-sessions-sqlite-'));
|
|
219
273
|
try {
|
|
220
|
-
const sources = [];
|
|
221
274
|
for (const suffix of ['', '-wal', '-shm']) {
|
|
222
|
-
const copy = join(dir, `store.db${suffix}`);
|
|
223
275
|
try {
|
|
224
|
-
await copyFile(`${path}${suffix}`,
|
|
276
|
+
await copyFile(`${path}${suffix}`, join(dir, `store.db${suffix}`));
|
|
225
277
|
}
|
|
226
278
|
catch (error) {
|
|
227
279
|
if (suffix === '' || !isMissing(error))
|
|
228
280
|
throw error;
|
|
229
|
-
continue;
|
|
230
281
|
}
|
|
231
|
-
// The shared-memory index is rebuilt from the WAL; only the store and its log carry data.
|
|
232
|
-
if (suffix !== '-shm')
|
|
233
|
-
sources.push({ path: `${path}${suffix}`, ...(await hashFile(copy)) });
|
|
234
282
|
}
|
|
235
|
-
// Opened writable so SQLite replays the copied write-ahead log into the private copy.
|
|
236
283
|
const db = new DatabaseSync(join(dir, 'store.db'));
|
|
237
284
|
dropCopy();
|
|
238
|
-
current = { key, dir, db
|
|
285
|
+
current = { key, dir, db };
|
|
239
286
|
}
|
|
240
287
|
catch (error) {
|
|
241
288
|
rmSync(dir, { recursive: true, force: true });
|
|
242
289
|
throw error;
|
|
243
290
|
}
|
|
244
291
|
}
|
|
245
|
-
return use(current.db
|
|
246
|
-
}
|
|
247
|
-
/** Remove the private store copy now instead of at process exit. */
|
|
248
|
-
export function releaseSqliteCopy() {
|
|
249
|
-
dropCopy();
|
|
292
|
+
return use(current.db);
|
|
250
293
|
}
|
|
251
294
|
export function isMissing(error) {
|
|
252
295
|
const code = error?.code;
|
package/dist/stores.d.ts
CHANGED
|
@@ -1,7 +1,19 @@
|
|
|
1
1
|
import type { NativeSessionStore } from './schema.js';
|
|
2
|
-
export { assertStoreRoot, EVIDENCE, SESSION_FORMAT_IDS, STORES, storeGlob } from './catalog.js';
|
|
2
|
+
export { assertStoreRoot, CODEX_HOME_RULE, EVIDENCE, isCodexHome, SESSION_FORMAT_IDS, STORES, storeGlob, withinPath } from './catalog.js';
|
|
3
3
|
/** Every file under `dir`, as paths relative to it with `/` separators. Missing dirs yield none. */
|
|
4
4
|
export declare function walkFiles(dir: string, signal?: AbortSignal): Promise<string[]>;
|
|
5
|
+
/** Session files a locate reads at once: a store holds thousands, and each read waits on the disk. */
|
|
6
|
+
export declare const LOCATE_READS = 32;
|
|
7
|
+
/** `fn` over `items`, at most `limit` at a time, with the results in the order of `items`. */
|
|
8
|
+
export declare function mapLimit<T, R>(items: readonly T[], limit: number, fn: (item: T) => Promise<R>): Promise<R[]>;
|
|
5
9
|
/** Files under `home` that one store glob matches, as absolute paths. */
|
|
6
10
|
export declare function storeFiles(home: string, store: NativeSessionStore, glob: string, nativeSessionId?: string): Promise<string[]>;
|
|
7
11
|
export declare function mtimeMs(path: string): Promise<number>;
|
|
12
|
+
/**
|
|
13
|
+
* Every Codex home under `home`, by `CODEX_HOME_RULE`: `home` itself when it is one (a fleet seat's
|
|
14
|
+
* `codex` passed as a store root), `.codex` and each `.codex-<name>` in it, and each acct account
|
|
15
|
+
* home. `$CODEX_HOME` and `$ACCT_CONFIG` describe this process's own HOME, so they apply only when
|
|
16
|
+
* `home` is that HOME; another HOME (a store root, a capture) is read by its layout alone. Homes
|
|
17
|
+
* that resolve to one directory, such as an account home linked to `~/.codex`, count once.
|
|
18
|
+
*/
|
|
19
|
+
export declare function codexHomes(home: string): Promise<string[]>;
|
package/dist/stores.js
CHANGED
|
@@ -1,8 +1,9 @@
|
|
|
1
|
-
import { readdir, stat } from 'node:fs/promises';
|
|
2
|
-
import {
|
|
3
|
-
import {
|
|
1
|
+
import { readdir, realpath, stat } from 'node:fs/promises';
|
|
2
|
+
import { homedir } from 'node:os';
|
|
3
|
+
import { join, relative, resolve, sep } from 'node:path';
|
|
4
|
+
import { CODEX_HOME_RULE, isCodexHome, storeGlob } from './catalog.js';
|
|
4
5
|
import { isMissing } from './source.js';
|
|
5
|
-
export { assertStoreRoot, EVIDENCE, SESSION_FORMAT_IDS, STORES, storeGlob } from './catalog.js';
|
|
6
|
+
export { assertStoreRoot, CODEX_HOME_RULE, EVIDENCE, isCodexHome, SESSION_FORMAT_IDS, STORES, storeGlob, withinPath } from './catalog.js';
|
|
6
7
|
/** Every file under `dir`, as paths relative to it with `/` separators. Missing dirs yield none. */
|
|
7
8
|
export async function walkFiles(dir, signal) {
|
|
8
9
|
const out = [];
|
|
@@ -28,6 +29,22 @@ export async function walkFiles(dir, signal) {
|
|
|
28
29
|
await visit(dir);
|
|
29
30
|
return out.sort();
|
|
30
31
|
}
|
|
32
|
+
/** Session files a locate reads at once: a store holds thousands, and each read waits on the disk. */
|
|
33
|
+
export const LOCATE_READS = 32;
|
|
34
|
+
/** `fn` over `items`, at most `limit` at a time, with the results in the order of `items`. */
|
|
35
|
+
export async function mapLimit(items, limit, fn) {
|
|
36
|
+
const results = new Array(items.length);
|
|
37
|
+
let next = 0;
|
|
38
|
+
const worker = async () => {
|
|
39
|
+
while (next < items.length) {
|
|
40
|
+
const index = next;
|
|
41
|
+
next += 1;
|
|
42
|
+
results[index] = await fn(items[index]);
|
|
43
|
+
}
|
|
44
|
+
};
|
|
45
|
+
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
|
|
46
|
+
return results;
|
|
47
|
+
}
|
|
31
48
|
/** Files under `home` that one store glob matches, as absolute paths. */
|
|
32
49
|
export async function storeFiles(home, store, glob, nativeSessionId) {
|
|
33
50
|
const root = join(home, store.root);
|
|
@@ -44,3 +61,53 @@ export async function mtimeMs(path) {
|
|
|
44
61
|
throw error;
|
|
45
62
|
}
|
|
46
63
|
}
|
|
64
|
+
/** Names of the directories (or links to them) directly in `dir`. A missing dir has none. */
|
|
65
|
+
async function childDirs(dir) {
|
|
66
|
+
try {
|
|
67
|
+
return (await readdir(dir, { withFileTypes: true })).filter((e) => e.isDirectory() || e.isSymbolicLink()).map((e) => e.name).sort();
|
|
68
|
+
}
|
|
69
|
+
catch (error) {
|
|
70
|
+
if (isMissing(error))
|
|
71
|
+
return [];
|
|
72
|
+
throw error;
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
/**
|
|
76
|
+
* Every Codex home under `home`, by `CODEX_HOME_RULE`: `home` itself when it is one (a fleet seat's
|
|
77
|
+
* `codex` passed as a store root), `.codex` and each `.codex-<name>` in it, and each acct account
|
|
78
|
+
* home. `$CODEX_HOME` and `$ACCT_CONFIG` describe this process's own HOME, so they apply only when
|
|
79
|
+
* `home` is that HOME; another HOME (a store root, a capture) is read by its layout alone. Homes
|
|
80
|
+
* that resolve to one directory, such as an account home linked to `~/.codex`, count once.
|
|
81
|
+
*/
|
|
82
|
+
export async function codexHomes(home) {
|
|
83
|
+
const own = resolve(home) === resolve(process.env.HOME || homedir());
|
|
84
|
+
const candidates = isCodexHome(home) ? [home] : [];
|
|
85
|
+
for (const name of await childDirs(home))
|
|
86
|
+
if (CODEX_HOME_RULE.siblings.test(name))
|
|
87
|
+
candidates.push(join(home, name));
|
|
88
|
+
if (own && process.env.CODEX_HOME)
|
|
89
|
+
candidates.push(resolve(process.env.CODEX_HOME));
|
|
90
|
+
const accounts = own && process.env.ACCT_CONFIG ? join(resolve(process.env.ACCT_CONFIG), 'codex') : join(home, CODEX_HOME_RULE.accounts);
|
|
91
|
+
for (const name of await childDirs(accounts))
|
|
92
|
+
candidates.push(join(accounts, name));
|
|
93
|
+
const seen = new Set();
|
|
94
|
+
const homes = [];
|
|
95
|
+
for (const dir of candidates) {
|
|
96
|
+
let real;
|
|
97
|
+
try {
|
|
98
|
+
real = await realpath(dir);
|
|
99
|
+
if (!(await stat(real)).isDirectory())
|
|
100
|
+
continue;
|
|
101
|
+
}
|
|
102
|
+
catch (error) {
|
|
103
|
+
if (isMissing(error))
|
|
104
|
+
continue;
|
|
105
|
+
throw error;
|
|
106
|
+
}
|
|
107
|
+
if (seen.has(real))
|
|
108
|
+
continue;
|
|
109
|
+
seen.add(real);
|
|
110
|
+
homes.push(dir);
|
|
111
|
+
}
|
|
112
|
+
return homes;
|
|
113
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@tangle-network/harness-sessions",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.5",
|
|
4
4
|
"description": "One reader for every coding-agent harness's native session (Claude Code, Codex, OpenCode, Pi): store layouts, a normalized session with messages, tool calls, served models, usage and ending errors, and projections to chat messages, OTLP spans and turns",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"sideEffects": false,
|