worktrust 0.10.34 → 0.10.36
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +51 -0
- package/chat-exports.mjs +279 -52
- package/count-behaviour.mjs +8 -3
- package/engine-release.json +1 -1
- package/history-outbox.mjs +36 -7
- package/log-session.mjs +25 -22
- package/package-release.json +1 -1
- package/package.json +1 -1
- package/session-databases.mjs +90 -1
- package/setup-mcp.mjs +3 -1
- package/transcript-readers.mjs +14 -0
- package/worktrust.mjs +159 -20
package/README.md
CHANGED
|
@@ -1,5 +1,39 @@
|
|
|
1
1
|
# worktrust
|
|
2
2
|
|
|
3
|
+
## 0.10.36: large and cut-off ChatGPT exports; hours only with `mcp off`; a re-measure reported whole or not at all
|
|
4
|
+
|
|
5
|
+
`npx worktrust history --export` now reads a ChatGPT export of any size: a zip64 archive (an export over 4 GB), and a
|
|
6
|
+
download that stopped part way, which is read up to the last whole `conversations-NNN.json` part and said as such ("The
|
|
7
|
+
archive was cut off after conversations-NNN.json; this is not the whole export"), so you decide whether to send what was
|
|
8
|
+
read before the cut. ChatGPT's loose `conversations-NNN.json` parts, a folder of them or one part beside its siblings, are
|
|
9
|
+
read as one export in part order; a missing part is refused by its name. An empty element in an export is counted and
|
|
10
|
+
left out instead of refusing the whole export. What is sent from an export is unchanged: hours and tokens per day, never
|
|
11
|
+
text. (The reader can also count an export's file kinds, searches and model switches between model families, but nothing
|
|
12
|
+
asks for them and nothing of them is sent.)
|
|
13
|
+
|
|
14
|
+
`npx worktrust mcp off` keeps the hours and takes the WorkTrust door out of every AI app here; `mcp on` puts it back (see
|
|
15
|
+
"Hours only, or also work lines" below). A choice made for this computer in Sources › Devices is done by the computer
|
|
16
|
+
at its next session. `update` no longer takes Codex's door out on every second run at the same version.
|
|
17
|
+
|
|
18
|
+
A re-measure says "Re-measured" only when WorkTrust holds every batch of the run; otherwise it says how many it holds and
|
|
19
|
+
gives WorkTrust's own reason for a refused batch, and the session hook exits with an error. What is measured is
|
|
20
|
+
unchanged, so the measure rules stay 3 and no re-measure is offered.
|
|
21
|
+
|
|
22
|
+
## 0.10.35: Gemini CLI read; every earlier line can name its AI app
|
|
23
|
+
|
|
24
|
+
Gemini CLI's sessions are read in place, as Copilot's are: the files Gemini CLI records itself under
|
|
25
|
+
`~/.gemini/tmp/<project>/chats/session-*.jsonl` (or an older `.json`), each message's role and clock, the model and
|
|
26
|
+
the token counts, and each tool call's name and status; the messages, the thoughts and the tool calls' arguments and
|
|
27
|
+
results are parsed with the file and never kept or sent. A subagent's session lies inside its parent's and is not
|
|
28
|
+
counted. The project folder is not read, so these lines name no repository. Their lines go as `gemini-cli`.
|
|
29
|
+
|
|
30
|
+
WorkTrust now records which AI app each line came from (Copilot in VS Code, Gemini CLI, Antigravity …), so its AI apps
|
|
31
|
+
table can give them their own rows. Lines measured before carry only their product ("Other apps"). The measure rules
|
|
32
|
+
go to 3: `update` offers the re-measure once, and it sends every day again; WorkTrust replaces only the lines of
|
|
33
|
+
"Other apps" and "App not recorded" that name no app, with the same counted seconds, and keeps every other line as it is.
|
|
34
|
+
`npx worktrust history --rebuild --force-days` does the same at any time: every day of every AI app, even the days
|
|
35
|
+
WorkTrust already holds unchanged (every app at once, so no day loses another app's line).
|
|
36
|
+
|
|
3
37
|
## 0.10.34: your own commits, measured on this computer (`npx worktrust git`)
|
|
4
38
|
|
|
5
39
|
GitHub shows an App that reads commits as "Read access to code", and has no narrower permission. `npx worktrust git`
|
|
@@ -254,6 +288,9 @@ yourself run a command that sends. See [Keep your history](#keep-your-history).
|
|
|
254
288
|
(asked, answered), its model and its token counts; the messages and responses are parsed with the file and never
|
|
255
289
|
kept or sent. Neither app runs a hook WorkTrust can give it, so the WorkTrust door, which both start when they
|
|
256
290
|
open, starts the session hook in the background, at most once an hour.
|
|
291
|
+
10. **Gemini CLI, read in place (0.10.35).** Its sessions are the files Gemini CLI records under
|
|
292
|
+
`~/.gemini/tmp/<project>/chats/`: each message's role and clock, the model, the token counts and each tool
|
|
293
|
+
call's name and status, never a message, a thought or a tool's output. Subagent sessions are not counted.
|
|
257
294
|
|
|
258
295
|
```
|
|
259
296
|
npx worktrust@latest history send this computer's earlier sessions later (asks first)
|
|
@@ -269,9 +306,23 @@ npx worktrust@latest history --from-folder <dir> your own copy of an AI app's
|
|
|
269
306
|
npx worktrust codex let Codex wake the session hook (asks first)
|
|
270
307
|
npx worktrust antigravity let Antigravity wake the session hook (asks first)
|
|
271
308
|
npx worktrust@latest status what is coupled here
|
|
309
|
+
npx worktrust mcp off hours only: the WorkTrust door out of every AI app here, so nothing of WorkTrust loads into
|
|
310
|
+
your sessions; the session hook keeps measuring your hours, and update leaves the door out
|
|
311
|
+
npx worktrust mcp on the door back in: your AI may also log finished work (about 40 tokens per session); nothing
|
|
312
|
+
from the time it was off is sent
|
|
313
|
+
npx worktrust mcp which of the two this computer is in (and whether a claude.ai WorkTrust connector also loads)
|
|
272
314
|
npx worktrust@latest disconnect take it out again (shows the plan, asks first)
|
|
273
315
|
```
|
|
274
316
|
|
|
317
|
+
**Hours only, or also work lines.** Your passport's hours come from the session hook, which reads your AI apps' own
|
|
318
|
+
session files; it needs no WorkTrust door in the apps. The door adds one thing: your AI's own `log_work` line at the end
|
|
319
|
+
of a piece of work, and for that its short instruction (and, in apps that load every tool up front, its tool
|
|
320
|
+
definitions) is loaded into every session. `connect` asks which you want (Enter keeps the door, as before); an update
|
|
321
|
+
asks once on a computer coupled before. The choice lives in `~/.worktrust/door.json` (0600). The hook tells WorkTrust
|
|
322
|
+
the mode beside the CLI version, one word, so Sources › Devices can show "work lines off" instead of an unexplained gap.
|
|
323
|
+
If Claude Code has also been connected to a WorkTrust connector of your claude.ai account, the CLI says so once: keep
|
|
324
|
+
the local door (it carries this computer's key) and remove the other at claude.ai/customize/connectors.
|
|
325
|
+
|
|
275
326
|
## Keep your history
|
|
276
327
|
|
|
277
328
|
**Why.** Your AI apps decide how long your history lives, and several delete it on their own schedule:
|
package/chat-exports.mjs
CHANGED
|
@@ -49,7 +49,7 @@ const expand = (path) => String(path).replace(/^~(?=\/|$)/, homedir());
|
|
|
49
49
|
// held whole (readExport) stops at 512 MB of kept fields. A zip64 archive (one over 4 GB) is refused by name: unpack it
|
|
50
50
|
// and give conversations.json itself. No limit override.
|
|
51
51
|
export const EXPORT_LIMITS = Object.freeze({ fileBytes: 64 * 1024 ** 3, directoryBytes: 8 * 1024 ** 2, depth: 64, conversations: 2_000_000,
|
|
52
|
-
conversationBytes: 8 * 1024 ** 2, conversationTextBytes: 64 * 1024 ** 2, keptBytes: 512 * 1024 ** 2, keyBytes: 256, parts: 64, manifestBytes: 1024 ** 2 });
|
|
52
|
+
conversationBytes: 8 * 1024 ** 2, conversationTextBytes: 64 * 1024 ** 2, keptBytes: 512 * 1024 ** 2, keyBytes: 256, parts: 64, manifestBytes: 1024 ** 2, zipEntries: 1_000_000 });
|
|
53
53
|
const PROGRESS_BYTES = 64 * 1024 ** 2, CHUNK_BYTES = 1024 ** 2;
|
|
54
54
|
const limit = (condition, label) => { if (!condition) throw Error(`export limit exceeded: ${label}`); };
|
|
55
55
|
const damaged = (condition) => { if (!condition) throw Error("a damaged zip archive"); };
|
|
@@ -96,11 +96,12 @@ async function* inflated(chunks, uncompressed) {
|
|
|
96
96
|
}
|
|
97
97
|
|
|
98
98
|
/**
|
|
99
|
-
* A ZIP, READ FOR ITS CONVERSATIONS ENTRIES. No dependency: the end-of-central-directory record is found from the back,
|
|
100
|
-
*
|
|
101
|
-
*
|
|
102
|
-
* Two entries of one name
|
|
103
|
-
* bytes the directory described.
|
|
99
|
+
* A ZIP, READ FOR ITS CONVERSATIONS ENTRIES. No dependency: the end-of-central-directory record is found from the back (and,
|
|
100
|
+
* for an archive past 4 GB, the zip64 record its locator names), the central directory walked for the names `accept` takes,
|
|
101
|
+
* and only those entries' bytes read (stored or deflated, as a stream; anything else is refused by name), in part order.
|
|
102
|
+
* Nothing else in the archive is touched. Two entries of one name, or a numbered part missing from the run, are refused. The
|
|
103
|
+
* descriptor stays open until `close`, so the bytes read are the bytes the directory described. An archive with no end
|
|
104
|
+
* record that begins with an entry is a download CUT OFF (2026-10-31): its local headers are walked instead (`walkLocal`).
|
|
104
105
|
*/
|
|
105
106
|
function zipEntries(path, accept) {
|
|
106
107
|
const { size, readAt, chunksOf, close } = openExport(path, EXPORT_LIMITS.fileBytes);
|
|
@@ -109,47 +110,146 @@ function zipEntries(path, accept) {
|
|
|
109
110
|
const tail = readAt(size - tailLength, tailLength);
|
|
110
111
|
let eocd = -1;
|
|
111
112
|
for (let at = tail.length - 22; at >= 0; at -= 1) if (tail.readUInt32LE(at) === 0x06054b50 && at + 22 + tail.readUInt16LE(at + 20) === tail.length) { eocd = at; break; }
|
|
112
|
-
if (eocd < 0)
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
113
|
+
if (eocd < 0) {
|
|
114
|
+
if (size < 30 || readAt(0, 4).readUInt32LE(0) !== LOCAL) throw Error(`not a zip archive: ${path}`);
|
|
115
|
+
const { selected, cut } = walkLocal(path, size, readAt, accept);
|
|
116
|
+
return { parts: entryParts(path, selected, size, readAt, chunksOf, true), close, cut };
|
|
117
|
+
}
|
|
118
|
+
const endAt = size - tailLength + eocd;
|
|
119
|
+
let entries = tail.readUInt16LE(eocd + 10), directorySize = tail.readUInt32LE(eocd + 12), directoryAt = tail.readUInt32LE(eocd + 16), directoryEnd = endAt;
|
|
120
|
+
if (entries === 0xffff || directorySize === U32 || directoryAt === U32) {
|
|
121
|
+
// ZIP64 (an archive past 4 GB or 65,535 entries): the locator lies just before the end record and names the zip64 end record, which holds the true counts.
|
|
122
|
+
damaged(endAt >= 20);
|
|
123
|
+
const locator = readAt(endAt - 20, 20);
|
|
124
|
+
damaged(locator.readUInt32LE(0) === 0x07064b50);
|
|
125
|
+
if (locator.readUInt32LE(4) !== 0 || locator.readUInt32LE(16) !== 1) throw Error("unsupported zip: multiple disks");
|
|
126
|
+
const recordAt = big(locator, 8), record = readAt(recordAt, 56, endAt - 20);
|
|
127
|
+
damaged(record.readUInt32LE(0) === 0x06064b50 && recordAt + 12 + big(record, 4) === endAt - 20);
|
|
128
|
+
if (record.readUInt32LE(16) !== 0 || record.readUInt32LE(20) !== 0) throw Error("unsupported zip: multiple disks");
|
|
129
|
+
entries = big(record, 32); damaged(big(record, 24) === entries);
|
|
130
|
+
directorySize = big(record, 40); directoryAt = big(record, 48); directoryEnd = recordAt;
|
|
131
|
+
} else {
|
|
132
|
+
if (tail.readUInt16LE(eocd + 4) !== 0 || tail.readUInt16LE(eocd + 6) !== 0) throw Error("unsupported zip: multiple disks");
|
|
133
|
+
damaged(tail.readUInt16LE(eocd + 8) === entries);
|
|
134
|
+
}
|
|
117
135
|
limit(directorySize <= EXPORT_LIMITS.directoryBytes, "zip directory bytes");
|
|
118
|
-
damaged(directoryAt + directorySize ===
|
|
136
|
+
damaged(directoryAt + directorySize === directoryEnd);
|
|
119
137
|
const directory = readAt(directoryAt, directorySize);
|
|
120
138
|
let at = 0;
|
|
121
139
|
const selected = new Map();
|
|
122
140
|
for (let index = 0; index < entries; index += 1) {
|
|
123
141
|
damaged(at + 46 <= directory.length && directory.readUInt32LE(at) === 0x02014b50);
|
|
124
|
-
const method = directory.readUInt16LE(at + 10)
|
|
125
|
-
|
|
142
|
+
const method = directory.readUInt16LE(at + 10);
|
|
143
|
+
let compressed = directory.readUInt32LE(at + 20), uncompressed = directory.readUInt32LE(at + 24), localAt = directory.readUInt32LE(at + 42);
|
|
144
|
+
const nameLength = directory.readUInt16LE(at + 28), extraLength = directory.readUInt16LE(at + 30), commentLength = directory.readUInt16LE(at + 32);
|
|
126
145
|
const flags = directory.readUInt16LE(at + 8), disk = directory.readUInt16LE(at + 34);
|
|
127
146
|
damaged(at + 46 + nameLength + extraLength + commentLength <= directory.length);
|
|
128
|
-
const name = directory.toString("utf8", at + 46, at + 46 + nameLength);
|
|
147
|
+
const name = directory.toString("utf8", at + 46, at + 46 + nameLength), extra = directory.subarray(at + 46 + nameLength, at + 46 + nameLength + extraLength);
|
|
129
148
|
at += 46 + nameLength + extraLength + commentLength;
|
|
130
149
|
if (!accept(basename(name)) || name.endsWith("/")) continue;
|
|
131
|
-
if (compressed ===
|
|
150
|
+
if (compressed === U32 || uncompressed === U32 || localAt === U32) [uncompressed, compressed, localAt] = zip64Extra(extra, [uncompressed, compressed, localAt]);
|
|
132
151
|
if ((flags & (1 | 64)) !== 0 || disk !== 0) throw Error("unsupported zip: encrypted or multi-disk entry");
|
|
133
152
|
damaged(!selected.has(basename(name)));
|
|
134
153
|
selected.set(basename(name), { name, method, flags, compressed, uncompressed, localAt });
|
|
135
154
|
}
|
|
136
155
|
damaged(at === directory.length);
|
|
137
|
-
|
|
138
|
-
const parts = [...selected.keys()].sort(byPart).map((key) => {
|
|
139
|
-
const { name, method, flags, compressed, uncompressed, localAt } = selected.get(key);
|
|
140
|
-
const local = readAt(localAt, 30, directoryAt);
|
|
141
|
-
damaged(local.readUInt32LE(0) === 0x04034b50 && local.readUInt16LE(8) === method && local.readUInt16LE(6) === flags);
|
|
142
|
-
const nameLength = local.readUInt16LE(26), dataAt = localAt + 30 + nameLength + local.readUInt16LE(28);
|
|
143
|
-
damaged(readAt(localAt + 30, nameLength, directoryAt).toString("utf8") === name);
|
|
144
|
-
if (!(flags & 8)) damaged(local.readUInt32LE(18) === compressed && local.readUInt32LE(22) === uncompressed);
|
|
145
|
-
damaged(dataAt + compressed <= directoryAt);
|
|
146
|
-
if (method === 0) damaged(compressed === uncompressed);
|
|
147
|
-
else if (method !== 8) throw Error(`${basename(name)} in ${path} is compressed in a way this reader does not open (method ${method})`);
|
|
148
|
-
return { path, bytes: uncompressed, chunks: () => (method === 0 ? chunksOf(dataAt, compressed, directoryAt) : inflated(chunksOf(dataAt, compressed, directoryAt), uncompressed)) };
|
|
149
|
-
});
|
|
150
|
-
return { parts, close };
|
|
156
|
+
return { parts: entryParts(path, selected, directoryAt, readAt, chunksOf, false), close, cut: null };
|
|
151
157
|
} catch (error) { close(); throw error; }
|
|
152
158
|
}
|
|
159
|
+
const LOCAL = 0x04034b50, U32 = 0xffffffff;
|
|
160
|
+
/** An eight-byte length or offset, held to what a number can say exactly. */
|
|
161
|
+
const big = (buffer, at) => { const value = buffer.readBigUInt64LE(at); damaged(value <= BigInt(Number.MAX_SAFE_INTEGER)); return Number(value); };
|
|
162
|
+
/** The zip64 extended-information field (id 0x0001): an eight-byte value for each of `words` that reads 0xffffffff, in the spec's order; the others as they are. */
|
|
163
|
+
function zip64Extra(extra, words) {
|
|
164
|
+
for (let at = 0; at + 4 <= extra.length; at += 4 + extra.readUInt16LE(at + 2)) {
|
|
165
|
+
const end = at + 4 + extra.readUInt16LE(at + 2);
|
|
166
|
+
damaged(end <= extra.length);
|
|
167
|
+
if (extra.readUInt16LE(at) !== 1) continue;
|
|
168
|
+
let next = at + 4;
|
|
169
|
+
return words.map((word) => { if (word !== U32) return word; damaged(next + 8 <= end); const value = big(extra, next); next += 8; return value; });
|
|
170
|
+
}
|
|
171
|
+
return damaged(false);
|
|
172
|
+
}
|
|
173
|
+
/** The selected entries as parts in part order, each held to its local header (method, flags, name and lengths) and its bytes before `end`. */
|
|
174
|
+
function entryParts(path, selected, end, readAt, chunksOf, walked) {
|
|
175
|
+
limit(selected.size <= EXPORT_LIMITS.parts, "conversations parts");
|
|
176
|
+
contiguous([...selected.keys()], path);
|
|
177
|
+
return [...selected.keys()].sort(byPart).map((key) => {
|
|
178
|
+
const { name, method, flags, compressed, uncompressed, localAt } = selected.get(key);
|
|
179
|
+
const local = readAt(localAt, 30, end);
|
|
180
|
+
damaged(local.readUInt32LE(0) === LOCAL && local.readUInt16LE(8) === method && local.readUInt16LE(6) === flags);
|
|
181
|
+
const nameLength = local.readUInt16LE(26), extraLength = local.readUInt16LE(28), dataAt = localAt + 30 + nameLength + extraLength;
|
|
182
|
+
damaged(readAt(localAt + 30, nameLength, end).toString("utf8") === name);
|
|
183
|
+
if (!walked && !(flags & 8)) {
|
|
184
|
+
let sizes = [local.readUInt32LE(22), local.readUInt32LE(18)];
|
|
185
|
+
if (sizes.includes(U32)) sizes = zip64Extra(readAt(localAt + 30 + nameLength, extraLength, end), [U32, U32]);
|
|
186
|
+
damaged(sizes[1] === compressed && sizes[0] === uncompressed);
|
|
187
|
+
}
|
|
188
|
+
damaged(dataAt + compressed <= end);
|
|
189
|
+
if (method === 0) damaged(compressed === uncompressed);
|
|
190
|
+
else if (method !== 8) throw Error(`${basename(name)} in ${path} is compressed in a way this reader does not open (method ${method})`);
|
|
191
|
+
return { path, bytes: uncompressed, chunks: () => (method === 0 ? chunksOf(dataAt, compressed, end) : inflated(chunksOf(dataAt, compressed, end), uncompressed)) };
|
|
192
|
+
});
|
|
193
|
+
}
|
|
194
|
+
/**
|
|
195
|
+
* A ZIP CUT OFF (2026-10-31: the owner's 4 GB ChatGPT export, downloaded, stopped inside an attachment; it has no central
|
|
196
|
+
* directory and no end record). Its LOCAL headers are walked from the front, each passed by the lengths it carries, its data
|
|
197
|
+
* never read: only an entry `accept` takes is kept, and only one whose header carries its real lengths (an entry that defers
|
|
198
|
+
* them to a data descriptor cannot be passed without inflating it, so the walk stops there and says so). The walk ends at the
|
|
199
|
+
* first entry the file does not hold whole, or at the directory if the cut came later. A cut inside a conversations entry
|
|
200
|
+
* refuses the export by that entry's name; otherwise what was read is handed over WITH the cut (`cut`), never as the whole.
|
|
201
|
+
*/
|
|
202
|
+
function walkLocal(path, size, readAt, accept) {
|
|
203
|
+
const selected = new Map();
|
|
204
|
+
let at = 0, walked = 0, where = null;
|
|
205
|
+
for (;;) {
|
|
206
|
+
if (at === size) { where = "at the end of an entry"; break; }
|
|
207
|
+
if (at + 30 > size) { where = "inside an entry's header"; break; }
|
|
208
|
+
const local = readAt(at, 30), signature = local.readUInt32LE(0);
|
|
209
|
+
if (signature === 0x02014b50 || signature === 0x06064b50) { where = "inside its directory"; break; }
|
|
210
|
+
damaged(signature === LOCAL);
|
|
211
|
+
limit((walked += 1) <= EXPORT_LIMITS.zipEntries, "zip entries");
|
|
212
|
+
const flags = local.readUInt16LE(6), method = local.readUInt16LE(8), nameLength = local.readUInt16LE(26), extraLength = local.readUInt16LE(28);
|
|
213
|
+
if (at + 30 + nameLength + extraLength > size) { where = "inside an entry's header"; break; }
|
|
214
|
+
const name = readAt(at + 30, nameLength).toString("utf8"), extra = readAt(at + 30 + nameLength, extraLength);
|
|
215
|
+
let compressed = local.readUInt32LE(18), uncompressed = local.readUInt32LE(22);
|
|
216
|
+
const wide = compressed === U32 || uncompressed === U32;
|
|
217
|
+
if (wide) [uncompressed, compressed] = zip64Extra(extra, [U32, U32]);
|
|
218
|
+
const wanted = accept(basename(name)) && !name.endsWith("/"), dataAt = at + 30 + nameLength + extraLength;
|
|
219
|
+
if ((flags & 8) && compressed === 0 && uncompressed === 0) {
|
|
220
|
+
if (wanted) throw Error(`${basename(name)} in ${path} carries no length in its header and the archive has no directory: it cannot be read without one`);
|
|
221
|
+
where = "at an entry whose header carries no length"; break;
|
|
222
|
+
}
|
|
223
|
+
if (dataAt + compressed > size) {
|
|
224
|
+
if (wanted) throw Error(`${path} is cut off inside ${basename(name)} (the file ends at byte ${size}; the entry needs ${dataAt + compressed}): download the export again`);
|
|
225
|
+
where = "inside an entry that is not a conversations part"; break;
|
|
226
|
+
}
|
|
227
|
+
if (wanted) {
|
|
228
|
+
if ((flags & (1 | 64)) !== 0) throw Error("unsupported zip: encrypted or multi-disk entry");
|
|
229
|
+
damaged(!selected.has(basename(name)));
|
|
230
|
+
selected.set(basename(name), { name, method, flags, compressed, uncompressed, localAt: at });
|
|
231
|
+
}
|
|
232
|
+
at = dataAt + compressed;
|
|
233
|
+
if (flags & 8) { // the data descriptor after the data: an optional signature, the crc and two lengths (eight bytes each in zip64)
|
|
234
|
+
const length = (at + 4 <= size && readAt(at, 4).readUInt32LE(0) === 0x08074b50 ? 4 : 0) + 4 + (wide ? 16 : 8);
|
|
235
|
+
if (at + length > size) { where = "at the end of an entry"; break; }
|
|
236
|
+
at += length;
|
|
237
|
+
}
|
|
238
|
+
}
|
|
239
|
+
if (selected.size === 0) throw Error(`${path} is cut off (no directory at its end) and holds no whole conversations part before the cut: download the export again`);
|
|
240
|
+
const names = [...selected.keys()].sort(byPart);
|
|
241
|
+
return { selected, cut: { size, at, where, entries: walked, parts: names.length, first: names[0], last: names.at(-1) } };
|
|
242
|
+
}
|
|
243
|
+
/** Numbered parts run without a gap: a part missing from the middle of the run is a partial export, never read as the whole. */
|
|
244
|
+
function contiguous(names, where, fromZero = false) {
|
|
245
|
+
const numbers = names.map(partOf).filter((n) => n >= 0).sort((a, b) => a - b), missing = [];
|
|
246
|
+
if (fromZero) for (let n = 0; n < (numbers[0] ?? 0) && missing.length < 8; n += 1) missing.push(`conversations-${String(n).padStart(3, "0")}`);
|
|
247
|
+
for (let i = 1; i < numbers.length; i += 1) {
|
|
248
|
+
damaged(numbers[i] !== numbers[i - 1]);
|
|
249
|
+
for (let n = numbers[i - 1] + 1; n < numbers[i] && missing.length < 8; n += 1) missing.push(`conversations-${String(n).padStart(3, "0")}`);
|
|
250
|
+
}
|
|
251
|
+
if (missing.length > 0) throw Error(`${where} is missing conversations part${missing.length === 1 ? "" : "s"} ${missing.join(", ")}: a partial export is never read as the whole`);
|
|
252
|
+
}
|
|
153
253
|
|
|
154
254
|
/**
|
|
155
255
|
* THE CONVERSATIONS, AND ONLY THEM (owner, 2026-10-17). Inside a zip: `conversations.json`, or a part-numbered
|
|
@@ -166,9 +266,9 @@ const refuseOther = (path) => { const other = OTHER_PART.exec(basename(path)); i
|
|
|
166
266
|
/** One conversations zip: its conversations entries, or a refusal that names the zip. */
|
|
167
267
|
function zipParts(path, given) {
|
|
168
268
|
refuseOther(path);
|
|
169
|
-
const { parts, close } = zipEntries(path, (name) => CONVERSATIONS_ENTRY.test(name));
|
|
269
|
+
const { parts, close, cut } = zipEntries(path, (name) => CONVERSATIONS_ENTRY.test(name));
|
|
170
270
|
if (parts.length === 0) { close(); throw Error(`no ${FILE} in ${given}`); }
|
|
171
|
-
return { parts, close };
|
|
271
|
+
return { parts, close, cuts: cut ? [cut] : [] };
|
|
172
272
|
}
|
|
173
273
|
/** A folder's conversations zips (conversations-000.zip, -001.zip …), in part order; nothing else of the folder is read. */
|
|
174
274
|
const folderParts = (dir) => { let names = []; try { names = readdirSync(dir); } catch { return []; } return names.filter((name) => CONVERSATIONS_PART.test(name)).sort(byPart).map((name) => join(dir, name)); };
|
|
@@ -200,9 +300,29 @@ function zipsParts(zips, given) {
|
|
|
200
300
|
for (const zip of zips) opened.push(zipParts(zip, given));
|
|
201
301
|
const parts = opened.flatMap((one) => one.parts);
|
|
202
302
|
limit(parts.length <= EXPORT_LIMITS.parts && parts.reduce((sum, part) => sum + part.bytes, 0) <= EXPORT_LIMITS.fileBytes, "conversations parts");
|
|
203
|
-
return { parts, close: () => { for (const one of opened) one.close(); } };
|
|
303
|
+
return { parts, close: () => { for (const one of opened) one.close(); }, cuts: opened.flatMap((one) => one.cuts) };
|
|
204
304
|
} catch (error) { for (const one of opened) one.close(); throw error; }
|
|
205
305
|
}
|
|
306
|
+
/**
|
|
307
|
+
* CHATGPT'S OWN SPLIT, UNPACKED (2026-10-31): conversations-000.json … -050.json loose in a folder. The folder, or any one of
|
|
308
|
+
* its parts given, is ONE export: every sibling part, in part order, the digest of their bytes in that order (as the same
|
|
309
|
+
* parts inside a zip). The run starts at 000 and has no gap, or the export is refused by the missing part's name; a folder
|
|
310
|
+
* holding loose parts AND conversations zips is ambiguous and refused. Nothing else of the folder is opened.
|
|
311
|
+
*/
|
|
312
|
+
const LOOSE_PART = /^conversations[-_](\d{1,6})\.json$/i;
|
|
313
|
+
const looseOf = (dir) => { let names = []; try { names = readdirSync(dir); } catch { return []; } return names.filter((name) => LOOSE_PART.test(name)).sort(byPart); };
|
|
314
|
+
function looseParts(dir, given) {
|
|
315
|
+
if (folderParts(dir).length > 0) throw Error(`${given} holds both loose conversations parts and conversations zips: give the one or the other`);
|
|
316
|
+
const names = looseOf(dir);
|
|
317
|
+
limit(names.length <= EXPORT_LIMITS.parts, "conversations parts");
|
|
318
|
+
contiguous(names, dir, true);
|
|
319
|
+
const opened = [];
|
|
320
|
+
try {
|
|
321
|
+
for (const name of names) { const path = join(dir, name); opened.push({ path, file: openExport(path, EXPORT_LIMITS.fileBytes) }); }
|
|
322
|
+
limit(opened.reduce((sum, one) => sum + one.file.size, 0) <= EXPORT_LIMITS.fileBytes, "conversations parts");
|
|
323
|
+
return { parts: opened.map(({ path, file }) => ({ path, bytes: file.size, chunks: () => file.chunksOf(0, file.size, file.size) })), close: () => { for (const one of opened) one.file.close(); }, cuts: [] };
|
|
324
|
+
} catch (error) { for (const one of opened) one.file.close(); throw error; }
|
|
325
|
+
}
|
|
206
326
|
|
|
207
327
|
/** Every conversations part, wherever the given path keeps them (a folder, the file itself, a zip, a part, or the manifest), each a source of chunks. */
|
|
208
328
|
function exportParts(given) {
|
|
@@ -210,6 +330,7 @@ function exportParts(given) {
|
|
|
210
330
|
let stat; try { stat = statSync(path); } catch { throw Error(`export not found: ${given}`); }
|
|
211
331
|
if (stat.isDirectory()) {
|
|
212
332
|
if (existsSync(join(path, FILE))) path = join(path, FILE);
|
|
333
|
+
else if (looseOf(path).length > 0) return looseParts(path, given);
|
|
213
334
|
else {
|
|
214
335
|
const zips = folderParts(path);
|
|
215
336
|
if (zips.length === 0) throw Error(`no ${FILE} in ${given}`);
|
|
@@ -217,6 +338,7 @@ function exportParts(given) {
|
|
|
217
338
|
}
|
|
218
339
|
}
|
|
219
340
|
if (/\.zip$/i.test(path)) return zipParts(path, given);
|
|
341
|
+
if (LOOSE_PART.test(basename(path)) && stat.isFile()) return looseParts(dirname(path), given); // one loose part given: the export is it and its siblings
|
|
220
342
|
const file = openExport(path, EXPORT_LIMITS.fileBytes);
|
|
221
343
|
let manifest;
|
|
222
344
|
try {
|
|
@@ -234,9 +356,103 @@ const NONSCALAR = Object.freeze({});
|
|
|
234
356
|
/** An element whose kept fields outgrew their budget: handed over as this, counted, never built. */
|
|
235
357
|
const OVERSIZED = Symbol("oversized");
|
|
236
358
|
const S = "scalar", T = "text"; // a scalar kept as it is; a text kept as a string for the counter, as its byte length for the hook
|
|
359
|
+
const P = "presence"; // a string that is never kept: only that it was there (`true`); any other value as a scalar would be
|
|
360
|
+
/** The leaf a plan gives a string: the plan itself, or the `leaf` of a plan that also keeps an object's keys (a list of strings and objects). A function maps the string as it ends, and only what it returns is kept. */
|
|
361
|
+
const leafOf = (plan) => (plan !== null && typeof plan === "object" ? plan.leaf ?? null : plan);
|
|
237
362
|
const NODE = { keys: { parent: S, message: { keys: { author: { keys: { role: S } }, create_time: S, content: { keys: { parts: { items: T } } }, metadata: { keys: { model_slug: S, is_visually_hidden_from_conversation: S } } } } } };
|
|
238
363
|
const CONVERSATION = { keys: { title: S, name: S, conversation_id: S, id: S, uuid: S, create_time: S, created_at: S, current_node: S, mapping: { any: NODE },
|
|
239
364
|
chat_messages: { items: { keys: { sender: S, created_at: S, text: T, content: { items: { keys: { type: S, text: T } } } } } } } };
|
|
365
|
+
/**
|
|
366
|
+
* AN EXPORT'S METADATA, COUNTS ONLY (owner, 2026-10-31). Asked for (`metadata: true`, the hook's way: never with the text),
|
|
367
|
+
* the walk keeps a few more fields of a ChatGPT message, each MAPPED as it is read to a word of a closed list or to nothing, so
|
|
368
|
+
* the export's own string (a favicon URL, a file's name or type, a quoted reply, a task's title, a GPT's id) is never kept:
|
|
369
|
+
* the person's attachments by category; the answer's searches, source references, reasoning and tool icons (only the fixed
|
|
370
|
+
* names globe, code, thoughts, run-command and api_tool, and an app's icon path, which says an app was used, never which);
|
|
371
|
+
* a deep-research task, a quote reply and voice by their presence. The field names are the ones the owner's own export
|
|
372
|
+
* carries (2026-10-31, read by key name only). conversation_asset_file_names.json, ads.json and chat.html are never opened.
|
|
373
|
+
*/
|
|
374
|
+
const ICONS = Object.freeze({ globe: "web_search", code: "code", thoughts: "reasoning", "run-command": "run_command", api_tool: "connected_app" });
|
|
375
|
+
const iconOf = (name) => (Object.hasOwn(ICONS, name) ? ICONS[name] : /^\/(?:[\w-]{1,64}\/){1,8}[\w.-]{1,128}$/.test(name) ? "connected_app" : null); // a URL, a path with "//" or anything else: nothing
|
|
376
|
+
const SOURCE_REFERENCES = new Set(["grouped_webpages", "grouped_webpages_model_predicted_fallback", "webpage", "webpage_extended", "sources_footnote"]);
|
|
377
|
+
const FILE_TYPES = [["pdf", /^application\/pdf$/], ["image", /^image\//], ["audio", /^audio\//], ["video", /^video\//], ["spreadsheet", /^text\/(?:csv|tab-separated-values)$|spreadsheet|excel|\.sheet$|numbers$/], ["presentation", /presentation|powerpoint|keynote/],
|
|
378
|
+
["document", /msword|wordprocessing|opendocument\.text|rtf|epub|pages$/], ["archive", /zip|tar|gzip|7z|rar|compressed/], ["data", /json|xml|yaml|toml|x-ndjson|sqlite/], ["code", /javascript|typescript|jsx|tsx|python|x-sh|x-sql|x-c|java|ruby|php|golang|x-go|rust|swift|kotlin|css|x-shellscript/], ["text", /^text\//]];
|
|
379
|
+
const fileOf = (mime) => FILE_TYPES.find(([, test]) => test.test(String(mime).toLowerCase().slice(0, 128)))?.[0] ?? "other";
|
|
380
|
+
const META_NODE = { keys: { parent: S, message: { keys: { author: { keys: { role: S } }, create_time: S,
|
|
381
|
+
content: { keys: { content_type: (type) => (type === "thoughts" || type === "reasoning_recap" ? "reasoning" : null), parts: { items: { keys: { content_type: (type) => (/^(?:audio_transcription|audio_asset_pointer|real_time_user_audio_video_asset_pointer)$/.test(type) ? "voice" : null) }, leaf: T } } } },
|
|
382
|
+
metadata: { keys: { model_slug: S, is_visually_hidden_from_conversation: S, attachments: { items: { keys: { mime_type: fileOf, mimeType: fileOf, is_big_paste: S } } }, search_result_groups: { items: P }, content_references: { items: { keys: { type: (type) => SOURCE_REFERENCES.has(type) || null } } },
|
|
383
|
+
tool_icons: { items: iconOf }, async_task_title: P, is_async_task_result_message: S, targeted_reply_label: P, targeted_reply_source_message_id: P, bidi_voice_mode_message: S } } } } } };
|
|
384
|
+
const META_CONVERSATION = { keys: { ...CONVERSATION.keys, mapping: { any: META_NODE }, conversation_template_id: (id) => (/^g-p-/.test(id) ? "project" : /^g-/.test(id) ? "gpt" : null) } };
|
|
385
|
+
/** The closed list (owner, 2026-10-31): what a conversation-day's counts may name, and nothing else. `files` and `depth` count by category. */
|
|
386
|
+
export const EXPORT_METADATA = Object.freeze(["files", "web_search", "sources_cited", "reasoning", "model_switch", "code", "run_command", "connected_app", "custom_gpt_or_project", "deep_research_agent", "voice", "quote_reply", "depth"]);
|
|
387
|
+
/**
|
|
388
|
+
* A MODEL'S FAMILY (owner, 2026-11-03): a switch counts between families, never ChatGPT's own routing inside one. The closed table names
|
|
389
|
+
* the slugs that carry no family in their shape; every other slug is normalised by one documented rule, never guessed: lower case;
|
|
390
|
+
* `gpt-N` or `gpt-No` (omni) is the generation, whatever follows (a point version such as -2 or .6, thinking, auto, instant, mini,
|
|
391
|
+
* a codename); `oN` is that o-series generation; anything else drops its routing suffixes (thinking, auto, instant, mini, t-mini,
|
|
392
|
+
* high, medium, low, pro, mobile, a bare number) and is its own family. A slug that names no model (auto, empty) has none.
|
|
393
|
+
*/
|
|
394
|
+
const FAMILY_TABLE = Object.freeze({ "text-davinci-002-render": "gpt-3.5", "text-davinci-002-render-sha": "gpt-3.5", "text-davinci-002-render-sha-mobile": "gpt-3.5", "text-davinci-002-render-paid": "gpt-3.5", auto: null });
|
|
395
|
+
export function modelFamily(slug) {
|
|
396
|
+
const name = String(slug ?? "").trim().toLowerCase().slice(0, 64);
|
|
397
|
+
if (name === "") return null;
|
|
398
|
+
if (Object.hasOwn(FAMILY_TABLE, name)) return FAMILY_TABLE[name];
|
|
399
|
+
const gpt = /^gpt-(\d+)(o)?(?=$|[.-])/.exec(name);
|
|
400
|
+
if (gpt) return `gpt-${gpt[1]}${gpt[2] ?? ""}`;
|
|
401
|
+
const o = /^o(\d+)(?=$|-)/.exec(name);
|
|
402
|
+
if (o) return `o${o[1]}`;
|
|
403
|
+
let family = name, before;
|
|
404
|
+
do { before = family; family = family.replace(/-(?:thinking|auto|instant|t-mini|mini|high|medium|low|pro|mobile|\d+)$/, ""); } while (family !== before);
|
|
405
|
+
return family || null;
|
|
406
|
+
}
|
|
407
|
+
const DEPTH = (bytes) => (bytes < 200 ? "short" : bytes < 1000 ? "medium" : "long"); // PROPOSED bounds (bytes of the person's text as the export writes it), uncalibrated
|
|
408
|
+
const list = (value) => (Array.isArray(value) ? value : []);
|
|
409
|
+
/**
|
|
410
|
+
* One ChatGPT conversation's counts per UTC day, on its kept branch: an EXCHANGE is a person's turn and the answers to it, and
|
|
411
|
+
* each kind is counted once per exchange (a search run twice for one question is one); files, a model switch between answers,
|
|
412
|
+
* the person's prompts by depth and a day inside a custom GPT or a project are counted as they are. Integers, never a string.
|
|
413
|
+
*/
|
|
414
|
+
function chatgptMetadata(kept) {
|
|
415
|
+
const days = {}, template = kept.conversation_template_id === "gpt" || kept.conversation_template_id === "project";
|
|
416
|
+
let seen = new Set(), model = null;
|
|
417
|
+
const day = (m) => { const seconds = Number(m?.create_time ?? kept.create_time); return Number.isFinite(seconds) && seconds > 0 ? new Date(seconds * 1000).toISOString().slice(0, 10) : null; };
|
|
418
|
+
const add = (counts, key, sub = null) => { if (sub) { counts[key] ??= {}; counts[key][sub] = (counts[key][sub] ?? 0) + 1; } else counts[key] = (counts[key] ?? 0) + 1; };
|
|
419
|
+
const once = (counts, key) => { if (!seen.has(key)) { seen.add(key); add(counts, key); } };
|
|
420
|
+
for (const node of chatgptBranch(kept)) {
|
|
421
|
+
const m = node?.message, role = m?.author?.role, meta = m?.metadata && typeof m.metadata === "object" ? m.metadata : {}, at = day(m);
|
|
422
|
+
if ((role !== "user" && role !== "assistant") || meta.is_visually_hidden_from_conversation || at === null) continue;
|
|
423
|
+
const counts = (days[at] ??= {});
|
|
424
|
+
if (template) counts.custom_gpt_or_project = 1;
|
|
425
|
+
const parts = list(m.content?.parts);
|
|
426
|
+
if (role === "user") {
|
|
427
|
+
seen = new Set();
|
|
428
|
+
for (const file of list(meta.attachments)) if (file && typeof file === "object" && file !== NONSCALAR && file.is_big_paste !== true) add(counts, "files", file.mime_type ?? file.mimeType ?? "other");
|
|
429
|
+
if (meta.bidi_voice_mode_message === true || parts.some((part) => part?.content_type === "voice")) once(counts, "voice");
|
|
430
|
+
if (meta.targeted_reply_label === true || meta.targeted_reply_source_message_id === true) once(counts, "quote_reply");
|
|
431
|
+
const bytes = parts.reduce((sum, part) => sum + (typeof part === "number" ? part : 0), 0);
|
|
432
|
+
if (bytes > 0) add(counts, "depth", DEPTH(bytes));
|
|
433
|
+
continue;
|
|
434
|
+
}
|
|
435
|
+
const family = typeof meta.model_slug === "string" ? modelFamily(meta.model_slug) : null; // a switch between families, never routing inside one
|
|
436
|
+
if (family !== null) { if (model !== null && family !== model) add(counts, "model_switch"); model = family; }
|
|
437
|
+
const icons = new Set(list(meta.tool_icons).filter((icon) => typeof icon === "string"));
|
|
438
|
+
if (list(meta.search_result_groups).length > 0 || icons.has("web_search")) once(counts, "web_search");
|
|
439
|
+
if (list(meta.content_references).some((reference) => reference?.type === true)) once(counts, "sources_cited");
|
|
440
|
+
if (m.content?.content_type === "reasoning" || icons.has("reasoning")) once(counts, "reasoning");
|
|
441
|
+
for (const kind of ["code", "run_command", "connected_app"]) if (icons.has(kind)) once(counts, kind);
|
|
442
|
+
if (meta.async_task_title === true || meta.is_async_task_result_message === true) once(counts, "deep_research_agent");
|
|
443
|
+
if (meta.bidi_voice_mode_message === true) once(counts, "voice");
|
|
444
|
+
}
|
|
445
|
+
for (const [at, counts] of Object.entries(days)) if (Object.keys(counts).length === 0) delete days[at];
|
|
446
|
+
return days;
|
|
447
|
+
}
|
|
448
|
+
/** Counts added into totals, category by category. */
|
|
449
|
+
function addCounts(totals, counts) {
|
|
450
|
+
for (const [key, value] of Object.entries(counts)) {
|
|
451
|
+
if (typeof value === "number") totals[key] = (totals[key] ?? 0) + value;
|
|
452
|
+
else { totals[key] ??= {}; for (const [sub, n] of Object.entries(value)) totals[key][sub] = (totals[key][sub] ?? 0) + n; }
|
|
453
|
+
}
|
|
454
|
+
return totals;
|
|
455
|
+
}
|
|
240
456
|
const malformed = () => { throw Object.assign(Error("malformed"), { malformed: true }); };
|
|
241
457
|
const NUMBER = /^-?(?:0|[1-9]\d*)(?:\.\d+)?(?:[eE][+-]?\d+)?$/;
|
|
242
458
|
const VALUE = 0, KEY = 1, COLON = 2, AFTER = 3, STRING = 4, TOKEN = 5, DONE = 6;
|
|
@@ -272,7 +488,7 @@ function walker(plan, { text, budget, onElement }) {
|
|
|
272
488
|
vPlan = kind === "[" ? top().plan?.items ?? null : null; state = kind === "{" ? KEY : VALUE;
|
|
273
489
|
};
|
|
274
490
|
const close = (kind) => { const frame = stack.pop(); if (frame.kind !== kind) malformed(); if (frame.root) state = DONE; else store(frame.out ?? NONSCALAR, 0); };
|
|
275
|
-
const beginString = (key) => { state = STRING; sKey = key; sKeep = key ? top().out !== null :
|
|
491
|
+
const beginString = (key) => { const leaf = key ? null : leafOf(vPlan); state = STRING; sKey = key; sKeep = key ? top().out !== null : leaf === S || (leaf === T && text) || typeof leaf === "function"; sRaw = sKeep ? [] : null; sLen = 0; sEsc = 0; sEscaped = false; };
|
|
276
492
|
/** Kept bytes of the string in flight: a key past its bound stops being one; a value past the budget takes the element with it. */
|
|
277
493
|
const take = (chunk, from, to) => {
|
|
278
494
|
sLen += to - from;
|
|
@@ -290,8 +506,11 @@ function walker(plan, { text, budget, onElement }) {
|
|
|
290
506
|
else if (frame.plan?.keys && key !== null && Object.hasOwn(frame.plan.keys, key)) { frame.key = key; vPlan = frame.plan.keys[key]; }
|
|
291
507
|
else { frame.key = null; vPlan = null; }
|
|
292
508
|
state = COLON;
|
|
293
|
-
} else
|
|
294
|
-
|
|
509
|
+
} else {
|
|
510
|
+
const leaf = leafOf(vPlan);
|
|
511
|
+
if (sKeep) store(typeof leaf === "function" ? leaf(decode()) : decode(), 0); // a mapped string: only the mapper's answer is kept
|
|
512
|
+
else store(leaf === T ? sLen : leaf === P ? true : NONSCALAR, 8); // a text the hook never reads: its byte length says whether there was one
|
|
513
|
+
}
|
|
295
514
|
};
|
|
296
515
|
const finishToken = () => {
|
|
297
516
|
let value;
|
|
@@ -396,15 +615,20 @@ const transcriptLine = ({ type, timestamp, model, text }) => ({ type, timestamp,
|
|
|
396
615
|
* one without an id, marked); `report` receives the path and vendor before the first conversation, the digest, the counts
|
|
397
616
|
* and the number of parts when the stream ends. Counts only on stderr.
|
|
398
617
|
*/
|
|
399
|
-
async function* exportRecords(given, { expect = null, excluded = () => false, skipped = null, text = false, report = null } = {}) {
|
|
400
|
-
|
|
618
|
+
async function* exportRecords(given, { expect = null, excluded = () => false, skipped = null, text = false, report = null, metadata = false } = {}) {
|
|
619
|
+
if (metadata && text) throw Error("an export's metadata is counted without the text, never with it");
|
|
620
|
+
const totals = {};
|
|
621
|
+
const { parts, close, cuts = [] } = exportParts(given);
|
|
401
622
|
const path = parts[0].path, digest = createHash("sha256");
|
|
402
|
-
let vendor = null, count = 0, oversized = 0, bytes = 0, next = PROGRESS_BYTES, offset = 0;
|
|
403
|
-
if (report) { report.path = path; report.vendor = expect; report.digest = null; report.conversations = 0; report.oversized = 0; report.parts = parts.length; }
|
|
623
|
+
let vendor = null, count = 0, oversized = 0, empty = 0, bytes = 0, next = PROGRESS_BYTES, offset = 0;
|
|
624
|
+
if (report) { report.path = path; report.vendor = expect; report.digest = null; report.conversations = 0; report.oversized = 0; report.empty = 0; report.parts = parts.length; report.cut = cuts.length > 0 ? cuts : null; }
|
|
404
625
|
const one = (kept, index) => {
|
|
405
626
|
limit(count < EXPORT_LIMITS.conversations, "conversation count");
|
|
406
627
|
count += 1;
|
|
407
628
|
if (kept === OVERSIZED) { oversized += 1; return { vendor, oversized: true, lines: [] }; }
|
|
629
|
+
// AN EMPTY ELEMENT (2026-10-31: one `{}` in the owner's conversations-043.json refused the whole export): none of the fields read, so
|
|
630
|
+
// it is counted and left out and decides no vendor; an export of nothing else is refused below as no export at all.
|
|
631
|
+
if (kept && typeof kept === "object" && kept.constructor === Object && kept !== NONSCALAR && Object.keys(kept).length === 0) { empty += 1; return { vendor, empty: true, lines: [] }; }
|
|
408
632
|
const shape = kept && typeof kept === "object" && kept !== NONSCALAR ? (kept.mapping instanceof Map || kept.mapping === NONSCALAR ? "chatgpt" : Array.isArray(kept.chat_messages) ? "claude-web" : "unknown") : "unknown";
|
|
409
633
|
if (vendor === null) vendor = shape;
|
|
410
634
|
if (shape === "unknown" || shape !== vendor) throw Error(`${path} is neither a ChatGPT nor a Claude export (or mixes the two)`);
|
|
@@ -412,12 +636,14 @@ async function* exportRecords(given, { expect = null, excluded = () => false, sk
|
|
|
412
636
|
if (report) report.vendor = vendor;
|
|
413
637
|
const title = String((vendor === "chatgpt" ? kept.title : kept.name) ?? "");
|
|
414
638
|
if (excluded(title)) { skipped?.push(title.slice(0, 40)); return { vendor, excluded: true, lines: [] }; }
|
|
415
|
-
|
|
639
|
+
const record = { vendor, id: String((vendor === "chatgpt" ? kept.conversation_id ?? kept.id : kept.uuid) ?? `#${index}`), lines: vendor === "chatgpt" ? chatgptLines(kept, text) : claudeLines(kept, text) };
|
|
640
|
+
if (metadata) { record.metadata = vendor === "chatgpt" ? chatgptMetadata(kept) : null; for (const counts of Object.values(record.metadata ?? {})) addCounts(totals, counts); } // a Claude export carries no such fields: null, never zeros
|
|
641
|
+
return record;
|
|
416
642
|
};
|
|
417
643
|
try {
|
|
418
644
|
for (const part of parts) {
|
|
419
645
|
let queue = [], seen = 0;
|
|
420
|
-
const walk = walker(CONVERSATION, { text, budget: text ? EXPORT_LIMITS.conversationTextBytes : EXPORT_LIMITS.conversationBytes, onElement: (kept, index) => queue.push([kept, offset + index]) });
|
|
646
|
+
const walk = walker(metadata ? META_CONVERSATION : CONVERSATION, { text, budget: text ? EXPORT_LIMITS.conversationTextBytes : EXPORT_LIMITS.conversationBytes, onElement: (kept, index) => queue.push([kept, offset + index]) });
|
|
421
647
|
const drain = function* () { const due = queue; queue = []; for (const [kept, index] of due) { seen = index + 1 - offset; yield one(kept, index); } };
|
|
422
648
|
try {
|
|
423
649
|
for await (const chunk of part.chunks()) {
|
|
@@ -435,19 +661,20 @@ async function* exportRecords(given, { expect = null, excluded = () => false, sk
|
|
|
435
661
|
offset += seen;
|
|
436
662
|
}
|
|
437
663
|
} finally { close(); }
|
|
438
|
-
if (
|
|
664
|
+
if (vendor === null && empty > 0) throw Error(`${path} is neither a ChatGPT nor a Claude export (or mixes the two)`);
|
|
665
|
+
if (report) { report.vendor = vendor ?? expect; report.digest = digest.digest("hex"); report.conversations = count; report.oversized = oversized; report.empty = empty; if (metadata) report.metadata = vendor === "chatgpt" ? totals : null; }
|
|
439
666
|
}
|
|
440
667
|
|
|
441
668
|
/** One export, read whole but compact: its vendor by shape, one record per conversation (its kept lines; none for an excluded or oversized one), and the digest of its bytes (what `sent` is remembered by). */
|
|
442
|
-
export async function readExport(given, expect = null, { excluded = () => false, skipped = null, text = false } = {}) {
|
|
669
|
+
export async function readExport(given, expect = null, { excluded = () => false, skipped = null, text = false, metadata = false } = {}) {
|
|
443
670
|
const report = {}, conversations = [];
|
|
444
671
|
let kept = 0;
|
|
445
|
-
for await (const record of exportRecords(given, { expect, excluded, skipped, text, report })) {
|
|
446
|
-
kept += 64 + (record.id?.length ?? 0) + record.lines.reduce((sum, line) => sum + 64 + (line.model?.length ?? 0) + (line.text?.length ?? 0), 0);
|
|
672
|
+
for await (const record of exportRecords(given, { expect, excluded, skipped, text, report, metadata })) {
|
|
673
|
+
kept += 64 + (record.id?.length ?? 0) + Object.keys(record.metadata ?? {}).length * 256 + record.lines.reduce((sum, line) => sum + 64 + (line.model?.length ?? 0) + (line.text?.length ?? 0), 0);
|
|
447
674
|
limit(kept <= EXPORT_LIMITS.keptBytes, "kept bytes");
|
|
448
675
|
conversations.push(record);
|
|
449
676
|
}
|
|
450
|
-
return { vendor: report.vendor, conversations, digest: report.digest, path: report.path, parts: report.parts, oversized: report.oversized };
|
|
677
|
+
return { vendor: report.vendor, conversations, digest: report.digest, path: report.path, parts: report.parts, oversized: report.oversized, empty: report.empty, cut: report.cut, ...(metadata ? { metadata: report.metadata } : {}) };
|
|
451
678
|
}
|
|
452
679
|
|
|
453
680
|
/**
|
|
@@ -457,11 +684,11 @@ export async function readExport(given, expect = null, { excluded = () => false,
|
|
|
457
684
|
* call. `text` hands the counter the messages' text (read here, for its rubric); the hook leaves it out and gets clocks,
|
|
458
685
|
* roles and a model name. `given` is a path, or an export `readExport` already returned; `report` is filled as above.
|
|
459
686
|
*/
|
|
460
|
-
export async function* exportSessions(given, { expect = null, excluded = () => false, skipped = null, text = false, report = null } = {}) {
|
|
461
|
-
const records = typeof given === "string" ? exportRecords(given, { expect, excluded, skipped, text, report }) : given.conversations;
|
|
462
|
-
for await (const { vendor, id, lines } of records) {
|
|
687
|
+
export async function* exportSessions(given, { expect = null, excluded = () => false, skipped = null, text = false, report = null, metadata = false } = {}) {
|
|
688
|
+
const records = typeof given === "string" ? exportRecords(given, { expect, excluded, skipped, text, report, metadata }) : given.conversations;
|
|
689
|
+
for await (const { vendor, id, lines, metadata: counts = null } of records) {
|
|
463
690
|
if (lines.length === 0) continue; // excluded, left out as too large, or without a message
|
|
464
|
-
yield { key: `${vendor === "chatgpt" ? "chatgpt" : "claude.ai"}:${id}`, projectDir: vendor === "chatgpt" ? "chatgpt" : "claude-ai", basename: vendor === "chatgpt" ? "chatgpt" : "claude.ai", vendor, id, lines: lines.map(transcriptLine) };
|
|
691
|
+
yield { key: `${vendor === "chatgpt" ? "chatgpt" : "claude.ai"}:${id}`, projectDir: vendor === "chatgpt" ? "chatgpt" : "claude-ai", basename: vendor === "chatgpt" ? "chatgpt" : "claude.ai", vendor, id, lines: lines.map(transcriptLine), ...(metadata ? { metadata: counts } : {}) };
|
|
465
692
|
}
|
|
466
693
|
}
|
|
467
694
|
|
|
@@ -491,7 +718,7 @@ export async function findExports(dir = join(homedir(), "Downloads")) {
|
|
|
491
718
|
for (const name of names.sort()) {
|
|
492
719
|
const path = join(root, name);
|
|
493
720
|
let stat; try { stat = statSync(path); } catch { continue; }
|
|
494
|
-
const candidate = stat.isDirectory() ? existsSync(join(path, FILE)) || folderParts(path).length > 0 : (/\.zip$/i.test(name) && !OTHER_PART.test(name)) || name === FILE;
|
|
721
|
+
const candidate = stat.isDirectory() ? existsSync(join(path, FILE)) || folderParts(path).length > 0 || looseOf(path).length > 0 : (/\.zip$/i.test(name) && !OTHER_PART.test(name)) || name === FILE;
|
|
495
722
|
if (!candidate) continue;
|
|
496
723
|
try {
|
|
497
724
|
const { vendor, conversations, digest } = await readExport(path);
|
package/count-behaviour.mjs
CHANGED
|
@@ -638,9 +638,14 @@ const { exportSessions = null } = (await import("./chat-exports.mjs").catch(() =
|
|
|
638
638
|
// 0.10.18: the export streams, one conversation in memory at a time, with its text (`text: true`: this rubric reads it here).
|
|
639
639
|
async function* vendorExportSessions(paths, expect) {
|
|
640
640
|
if (paths.length > 0 && !exportSessions) throw Error("chat-exports.mjs is not beside this file: a data export cannot be read");
|
|
641
|
-
for (const given of paths)
|
|
642
|
-
|
|
643
|
-
|
|
641
|
+
for (const given of paths) {
|
|
642
|
+
const report = {};
|
|
643
|
+
for await (const session of exportSessions(given, { expect, excluded, skipped: skippedConversations, text: true, report })) {
|
|
644
|
+
if (expect === "chatgpt") chatgptConversations += 1; else exportConversations += 1;
|
|
645
|
+
yield session;
|
|
646
|
+
}
|
|
647
|
+
// A DOWNLOAD CUT OFF (2026-10-31): the parts before the cut are counted, and the person is told it is not the whole export.
|
|
648
|
+
for (const cut of report.cut ?? []) process.stderr.write(`export: the archive is cut off (${cut.where}) after ${cut.last}: ${cut.parts} conversations part${cut.parts === 1 ? "" : "s"} read whole, ${report.conversations} conversations; nothing past the cut is read, so this is not the whole export\n`);
|
|
644
649
|
}
|
|
645
650
|
}
|
|
646
651
|
async function* claudeExportSessions() { yield* vendorExportSessions(EXPORTS, "claude-web"); }
|
package/engine-release.json
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"keyId":"608168ccdef72562fc91f9c0f9f2c2d6876bece7f7362cb545b818f7947da6b5","payload":{"files":{"chat-exports.mjs":"
|
|
1
|
+
{"keyId":"608168ccdef72562fc91f9c0f9f2c2d6876bece7f7362cb545b818f7947da6b5","payload":{"files":{"chat-exports.mjs":"bda43095a62bab608fc9309f9e7a2232dc10b29c514fb34e1aae357daa132737","claude-mod.mjs":"02d86b0cdddf39879efc5194c0a5be6b41456faedec28963dc123a4e88dc3942","commit-rules.mjs":"e94cc32a5415f6d9e978145ab9505ac4ae0bac9431e6e9104b9b37155260b51f","config-edits.mjs":"0eba2f176ef6f7afba071d6e00de0ff19ba2b283180a801dfd957aa38c1edd94","count-behaviour.mjs":"179baf829eda488bab3f17ae6917473ec01d0d6346f47de7bc144351274f19c8","credential-store.mjs":"a4b3540a61f4ce19e535c7f1d260c0d817686cd066d559a76db2ab108144abba","credential-vault-scripts.mjs":"7e42fe1db3885c2dec6f1de87bebfbefe2292d29179a68f3da87d19eb6c7704b","credential-vault.mjs":"279baabb217e2a3f170e8d7f2b271a88484531f9031c441360ea91561a70da99","git-commits.mjs":"f9c084c54937a86df8fe30a0db2a58e04eeef0dbae69c54b8e368efc9964d6d8","history-outbox.mjs":"31609392f4e13dd37678ab3154bd2896cce1a8cca4903b78dd2d96da82b1f406","log-session.mjs":"9c78cadbc469f273367a438721fcc47ce101dc230b6c9a18fe0923960f471bc9","orchestration-contract.mjs":"ec08f303a166365b99e70b675de4cfe76c6156c2567e14b5569f6d992f0231b7","orchestration-evidence.mjs":"d0d06cecf0eb08e5c6e653c62a4f45b1eddf2119c2acb8341056b34f488d214a","orchestration-report.mjs":"d4e0a9ad8e4a01b462d22c6b1866aac291565c533ff476085d3367cb6d59bc1b","package-integrity.mjs":"a8210c9494a56bfb9a5ea3d10064c7a269f0b6330c0af7c0700e706c3c0224e3","preserve-lines.mjs":"2df64a69687aa51219bf7b86b88d175cc50990fa29574b31897e41a9ebeecf43","preserve-summary.mjs":"137f781ed7021596a3f0ea67afcf2ab8616ac293f833dbb8f176e838e0ed8321","preserve.mjs":"3e369083ba43bcda2fa0622fd526aa64fcf8ad66e7b6e8f23821752c4d47ad27","runtime-install.mjs":"aab47b52c79d726ef428a24f982c648fa3f1c2dc2b574f633611d02924864e5f","session-databases.mjs":"b9fe20d7445205bdbf4cf2eaaf9cc770dad01e23d41e644b04930ca34e10426d","setup-mcp.mjs":"dcaa912b0e15a74ab3365aa62f345ba436510b958e9f5cdceddd57ffb99ef4b7","stack-rules.mjs":"b23c56e72ec56ba02b17ed895f869172d988c468a697e8d6026b0e268f61e489","stretch-evidence.mjs":"5f5667b92adf11b1a5a6330a0f26dd6db73d56fe60823d9a5246dab71d27d380","transcript-readers.mjs":"e9f20415c2b5d62a09f35a63075002ebd1da56edb32f242d52822d58ecbd8ac3","worktrust-launch.mjs":"84801bd29d472d48188c126b9ebe3989a3132b54e13cc4fe534ff7c383a294ee","worktrust.mjs":"ff83a783504950cba6f62cb14786e504db701be6c324a44c3fefac26628f516d"},"format":"worktrust-package/1","kind":"engine","version":"0.10.36"},"signature":"1NYMJNm65pMSGf0JL43MaL9XjIWMwu_VBnL1kmdjXh61jjXeXX-RtT7Hb28yXdXM_69et0oAenpizRvOxjuiBg"}
|