sidekick-shared 0.27.1 → 0.27.2

package/CHANGELOG.md

@@ -5,6 +5,26 @@ All notable changes to sidekick-shared will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.27.2] - 2026-09-22
+
+### Fixed
+
+- `collectPromptHistory()` streams session files instead of reading each unread range into one buffer. A file whose unread part exceeded `maxFileBytes` used to be skipped on every call; now it is read in 1 MiB chunks up to `maxFileBytes` per call, stops after a complete line, keeps what it returned, and continues on the next call (cursor `backlog: true`), even if its size and mtime are unchanged. Memory stays near one chunk plus one record. Deadline, abort, `maxSessions`, `maxTotalBytes`, and the new `maxEntries` stop at the last complete line the same way instead of discarding the partly read file. The first file of a call always finishes its first record, so a record larger than the budget cannot stall. Lines split only at `\n`, so UTF-8 characters across chunk edges decode exactly; half-written final lines and bytes appended during a call wait for the next call.
+- Late answer metadata no longer depends on a replay window. `pending` now keeps `lineBytes`, `timestamp`, and `answer: { status, id?, model?, usage? }` (answer state, never text), the scan continues from the file's offset however far the answer is, and only the prompt's own line is re-read (and verified against its length and timestamp) when the prompt must be returned again. Replays that 0.27.1 dropped for exceeding `maxFileBytes` no longer happen.
+
+### Added
+
+- Bounds `maxRecordBytes` (default 16 MiB; a longer record is skipped as it streams past, without buffering, even across calls while it is still being written) and `maxEntries` (default 10 000), and bound name `'maxEntries'`.
+- `PromptHistoryResult.hasMore` (a bound left bytes or files unread) and `exclusions: PromptHistoryExclusion[]` (`{ provider, sessionId, reason: 'oversizedRecord' | 'malformedRecord', offset }`, up to 1000 per call). These are the only permanent exclusions; a prompt inside such a record is lost and consumes no ordinal.
+- Stats `sessionsWithBacklog`, `sessionsRewritten`, `recordsOverSizeLimit`, and `recordsMalformed`; types `PromptHistoryAnswerState` and `PromptHistoryExclusion`.
+- Rewrite detection: a file that shrank below its offset, whose first 4 KiB digest (`head`) changed, whose resume point no longer follows a newline, or whose pending prompt line no longer matches is read again from the start with fresh ordinals.
+
+### Changed
+
+- The cursor is `version: 3`: sessions gain optional `backlog`, `skipping`, and `head`; `pending` gains `lineBytes`, `timestamp`, and `answer`; the cursor gains `next`, the key of the file a call-wide stop left off at, where the next call starts. Version 1 and 2 cursors are accepted: each file is re-verified once, and a version 2 `pending` prompt is replayed from its line in resumable passes. Files 0.27.1 skipped for size were never in its cursor and are read in full; metadata 0.27.1 already abandoned is not recoverable. A session carried unchanged from an older cursor and not visited keeps its old shape (for example `pending` without `answer`).
+- `maxFileBytes` is now a per-file, per-call read budget instead of a skip threshold, and `stats.filesOverSizeLimit` counts files that used it up and continue on the next call.
+- The Claude first-answer fallback id for assistant records without `message.id` is the line's byte offset instead of a per-scan line index, so it stays stable across calls.
+
 ## [0.27.1] - 2026-09-22
 
 ### Fixed

package/README.md

@@ -9,6 +9,7 @@ Types, parsers, providers, readers, formatters, aggregation, search, reporting,
 
 ## Recent additions
 
+- **0.27.2: streamed prompt history** — `collectPromptHistory()` reads session logs of any size in bounded, resumable passes instead of skipping files over `maxFileBytes`: every call keeps what it read, the cursor (`version: 3`) continues where each file stopped, and a pending prompt keeps its answer state so a far-away answer still settles it. Adds `maxRecordBytes` and `maxEntries` bounds, `hasMore`, permanently skipped records in `exclusions`, and rewrite detection.
 - **0.27.1: late prompt metadata** — `collectPromptHistory()` returns a prompt again, with the same provider, sessionId, ordinal, timestamp, and text, when its first answering call is written after an earlier collection, and entries gain `metadataStatus` (`pending` / `provisional` / `final`). The cursor becomes `version: 2`, keeps a replay offset and metadata digest for the one open prompt per file (no prompt text), and still accepts 0.27.0 cursors.
 - **0.27.0: human prompt history** — `collectPromptHistory()` (Node-only) returns the prompts a person typed in Claude Code and Codex sessions under a set of workspace roots (plus git worktree siblings), with fail-closed cwd scoping, untruncated text, stable per-session ordinals, the answering model and usage, a JSON-serializable resume cursor, and bounds that stop cleanly and report themselves in `boundsHit`. `isHumanPrompt()` / `humanPromptText()` expose the same classification for any canonical transcript message, and transcript provenance now keeps Claude Code's `isCompactSummary`, `originKind`, and `promptSource`. Claude Opus 5.5 gets its own pricing and context entries.
 - **0.26.6: verified account switching** — `syncLiveAccountState()` reconciles the live Claude Code and Codex logins with saved profiles (registers unknown logins under their email, folds rotated tokens, merges duplicate Codex profiles, re-points the active pointer) and runs at startup, on account-change events, and as phase one of every switch; `getAccountHealth()` / `listAccountsWithHealth()` report secret-free `fresh | expiring | expired | unknown | missing` health; `switchAccount[Async]()` returns a `SwitchAccountResult` with verification, `runningConsumers`, `needsLogin`, and an `undoToken` for `undoLastSwitch()`; `detectRunningAccountConsumers()`, `getAccountLaunchEnv()`, `refreshInactiveAccounts()`, and `getCodexCredentialStoreMode()` are new. Claude logins run `claude auth login`, the Keychain suffix follows Claude Code 2.1.x, and the unused shell-hook helpers were removed.
@@ -310,8 +311,10 @@ const result = await collectPromptHistory({
   cursor,
   bounds: {
     maxSessions: 1000,
-    maxFileBytes: 16 << 20,
+    maxFileBytes: 16 << 20, // per file per call; the rest is read next call
     maxTotalBytes: 256 << 20,
+    maxRecordBytes: 16 << 20, // one longer record is skipped and reported
+    maxEntries: 10_000,
     deadlineMs: 30_000,
   },
   signal: AbortSignal.timeout(60_000),
@@ -321,16 +324,19 @@ for (const entry of result.entries) {
   // { provider, sessionId, ordinal, timestamp, text, cwd, gitBranch?, model?, usage?, metadataStatus }
   // Upsert by (provider, sessionId, ordinal): a prompt comes back when its answer metadata changes.
 }
-saveCursor(result.cursor); // sensitive local state; unchanged files are skipped next time
-if (result.boundsHit.length > 0) console.warn('partial pass', result.boundsHit, result.stats);
+saveCursor(result.cursor); // sensitive local state; fully read unchanged files are skipped next time
+if (result.hasMore) scheduleAnotherCall(); // a bound stopped the pass; the cursor resumes it
+for (const exclusion of result.exclusions) console.warn('record left out', exclusion); // permanent
 ```
 
 - **Scope fails closed.** A prompt is returned only when its recorded working directory (the Claude line's `cwd`; the Codex `session_meta` / `turn_context` cwd) realpath-resolves to a root or somewhere inside one. Missing or unresolvable cwds are excluded and counted in `stats.promptsOutOfScope`; Codex scope never comes from the rollout's location. `cwd` is the only path in entries. Cursor keys hold no paths, but cursor values can (Codex resume state keeps the session's cwd and branch), so store the cursor as sensitive, local-only state.
 - **Parse-only.** No redaction and no network I/O; text is untruncated.
 - **Ordinals are stable.** Every human prompt in a file consumes one ordinal, even when it is out of scope, older than `since`, or lacks a timestamp (such lines are skipped and counted, never given a synthesized time). Entries are ordered by provider, session, then ordinal.
 - **Model and usage** come from the first model call that answered the prompt (Claude: the first non-synthetic assistant message, with split records of that call replacing rather than adding usage; Codex: the first `token_count` with the model from `turn_context`). Later tool-loop calls never replace it. `inputTokens` is uncached input, and missing usage stays `undefined`, never zero.
-- **Late answers enrich the same prompt.** A prompt can be collected before its answer is written. `metadataStatus` is `pending` (no answering call yet), `provisional` (Claude call still being written, `stop_reason: null`), or `final`. While a file's last prompt is not final, the cursor keeps the byte offset of its line and a digest of what was returned (never prompt or answer text); the next call replays from that line and returns the prompt again, with the same provider, sessionId, ordinal, timestamp, and text, only when its model, usage, or status changed. A prompt closed by the next prompt without an answer comes back once as `final` with no usage. Consumers should upsert by identity. Cursors from 0.27.0 (`version: 1`) are accepted but cannot recover prompts they already returned without metadata.
-- **Bounds never throw.** Hitting `maxSessions`, `maxTotalBytes`, the deadline, or an abort stops the pass. An interrupted file contributes no entries, and the cursor covers only fully processed files, so the next call picks up where this one stopped. A file over `maxFileBytes` is different: it is skipped and counted in `stats.filesOverSizeLimit`, and calling again with the same bounds skips it again, so raise the bound to make progress on it. When only the replay of a pending prompt would exceed `maxFileBytes`, the replay is dropped and the file continues from its committed offset; that prompt is not revisited.
+- **Late answers enrich the same prompt.** A prompt can be collected before its answer is written. `metadataStatus` is `pending` (no answering call yet), `provisional` (Claude call still being written, `stop_reason: null`), or `final`. While a file's last prompt is not final, the cursor keeps the byte offset and length of its line, its timestamp, and its answer state so far (first-answer message id, model, and usage, never prompt or answer text). The next call continues from where the file stopped, however far the answer is, and returns the prompt again with the same provider, sessionId, ordinal, timestamp, and text only when its model, usage, or status changed; the text comes from re-reading that one line. A prompt closed by the next prompt without an answer comes back once as `final` with no usage. Consumers should upsert by identity.
+- **Large files are streamed and resumable.** Files are read in 1 MiB chunks and every pass stops after a complete line, so memory stays near one chunk plus one record whatever the file size. `maxFileBytes` is the most one file may read in one call: a file with more left keeps what it returned, records `backlog` in its cursor, is counted in `stats.filesOverSizeLimit`, and continues on the next call even if its size and mtime have not changed. `maxSessions`, `maxTotalBytes`, `maxEntries`, the deadline, and an abort stop the whole call the same way, and `cursor.next` makes the next call start with the file that was stopped, so a large or busy session cannot starve the others. The first file of a call always finishes its first record, so a record larger than the budget still moves forward. Bytes appended during a call and a half-written final line wait for the next call. `boundsHit` lists these resumable stops and `hasMore` says whether calling again would read more. No bound throws.
+- **Permanent exclusions are reported.** A record over `maxRecordBytes` is skipped as it streams past without being buffered, even when it is still being written, and a complete line that is not JSON is skipped. Each is counted (`stats.recordsOverSizeLimit`, `stats.recordsMalformed`) and listed in `exclusions` with its byte offset. A prompt inside such a record is lost and does not consume an ordinal. A file that shrank or whose first 4 KiB changed is treated as rewritten: it is read again from the start with fresh ordinals and counted in `stats.sessionsRewritten`.
+- **Cursor versions.** Output is `version: 3`. Cursors from 0.27.0 (`version: 1`) and 0.27.1 (`version: 2`) are accepted: each file is re-verified once, and a 0.27.1 pending prompt is replayed from its line in resumable passes, so it now settles even when its answer is far away. Files 0.27.1 skipped for size were never in its cursor and are read in full on the first call. Metadata that 0.27.1 already gave up on (a pending replay that exceeded `maxFileBytes`) was not saved and cannot be recovered, and a 0.27.0 cursor still cannot recover prompts it returned without metadata.
 
 For an already-open incremental Codex rollout, keep one `CodexRolloutParser` per stream and pass each parsed `SessionEvent[]` batch to `projectSessionTranscript()` (or append the events to an existing projection). This preserves parser state for paired tool calls, model context, and cumulative token deltas; consumers should not convert raw `response_item` rows themselves.
 

package/dist/index.d.ts

@@ -11,7 +11,7 @@ export { projectSessionTranscript } from './transcript';
 export type { TranscriptFidelity, TranscriptSourceProvenance, CanonicalTranscriptBlock, CanonicalTranscriptMessage, CanonicalToolCall, CanonicalTranscriptUsageTotals, CanonicalSessionTranscript, ProjectSessionTranscriptOptions, } from './transcript';
 export { listRecentSessions, readSessionTranscript } from './sessionTranscripts';
 export { collectPromptHistory } from './promptHistory';
-export type { CollectPromptHistoryOptions, PromptHistoryBound, PromptHistoryBounds, PromptHistoryCursor, PromptHistoryCursorSession, PromptHistoryEntry, PromptHistoryMetadataStatus, PromptHistoryPendingPrompt, PromptHistoryProvider, PromptHistoryResult, PromptHistoryResumeState, PromptHistoryStats, PromptHistoryUsage, } from './promptHistory';
+export type { CollectPromptHistoryOptions, PromptHistoryAnswerState, PromptHistoryBound, PromptHistoryBounds, PromptHistoryCursor, PromptHistoryCursorSession, PromptHistoryEntry, PromptHistoryExclusion, PromptHistoryMetadataStatus, PromptHistoryPendingPrompt, PromptHistoryProvider, PromptHistoryResult, PromptHistoryResumeState, PromptHistoryStats, PromptHistoryUsage, } from './promptHistory';
 export { HUMAN_CLAUDE_ENTRYPOINTS, HUMAN_CODEX_SOURCES, humanPromptText, isHumanPrompt, } from './humanPrompt';
 export type { HumanPromptProvider } from './humanPrompt';
 export { computeSessionFileStats, firstUserPrompt, providerContextSizeFn, readSessionFileStats, } from './sessionStats';

package/dist/node.d.ts

@@ -11,7 +11,7 @@ export { loadObservedContextWindows, recordObservedContextWindow, getObservedCon
 export type { ObservedContextWindowOptions, ObservedContextWindowStore, } from './observedContextWindows';
 export { listRecentSessions, readSessionTranscript } from './sessionTranscripts';
 export { collectPromptHistory } from './promptHistory';
-export type { CollectPromptHistoryOptions, PromptHistoryBound, PromptHistoryBounds, PromptHistoryCursor, PromptHistoryCursorSession, PromptHistoryEntry, PromptHistoryMetadataStatus, PromptHistoryPendingPrompt, PromptHistoryProvider, PromptHistoryResult, PromptHistoryResumeState, PromptHistoryStats, PromptHistoryUsage, } from './promptHistory';
+export type { CollectPromptHistoryOptions, PromptHistoryAnswerState, PromptHistoryBound, PromptHistoryBounds, PromptHistoryCursor, PromptHistoryCursorSession, PromptHistoryEntry, PromptHistoryExclusion, PromptHistoryMetadataStatus, PromptHistoryPendingPrompt, PromptHistoryProvider, PromptHistoryResult, PromptHistoryResumeState, PromptHistoryStats, PromptHistoryUsage, } from './promptHistory';
 export { HUMAN_CLAUDE_ENTRYPOINTS, HUMAN_CODEX_SOURCES, humanPromptText, isHumanPrompt, } from './humanPrompt';
 export type { HumanPromptProvider } from './humanPrompt';
 export { computeSessionFileStats, firstUserPrompt, providerContextSizeFn, readSessionFileStats, } from './sessionStats';

package/dist/promptHistory.d.ts

@@ -8,24 +8,37 @@
  * or somewhere inside one. Every bound stops cleanly and is reported in
  * `boundsHit`; nothing is silently truncated and no bound throws.
  *
+ * Files are streamed in fixed-size chunks and every pass stops at a complete
+ * line, so a large or busy log is read across several calls: the cursor holds
+ * where each file stopped and the next call continues from there. A single
+ * record over `maxRecordBytes` is skipped without being buffered and reported
+ * in `exclusions`, the only way a record is permanently left out.
+ *
  * A prompt can be written before its answer. The cursor then keeps the byte
- * offset of that prompt's line (not its text), the next call replays from it,
- * and the prompt is returned again under the same identity once its first
+ * offset of that prompt's line and its answer-tracking state (never text), and
+ * the prompt is returned again under the same identity once its first
  * answering call brings model or usage (`metadataStatus` says how settled).
  */
 import { type HumanPromptProvider } from './humanPrompt';
 export type PromptHistoryProvider = HumanPromptProvider;
-export type PromptHistoryBound = 'maxSessions' | 'maxFileBytes' | 'maxTotalBytes' | 'deadline' | 'aborted';
+export type PromptHistoryBound = 'maxSessions' | 'maxFileBytes' | 'maxTotalBytes' | 'maxEntries' | 'deadline' | 'aborted';
 export interface PromptHistoryBounds {
     /** Session files read in one call (unchanged files do not count). Default 1000. */
     maxSessions?: number;
     /**
-     * Bytes a single file may need read in this pass. Default 16 MiB. A file over
-     * it is skipped, and calling again with the same bounds skips it again.
+     * Bytes read from one file in one call. Default 16 MiB. A file with more to
+     * read stops at a line boundary and continues on the next call.
      */
     maxFileBytes?: number;
     /** Bytes read across all files in this call. Default 256 MiB. */
     maxTotalBytes?: number;
+    /**
+     * Largest single log record that is parsed. Default 16 MiB. A longer record
+     * is skipped without being buffered and reported in `exclusions`.
+     */
+    maxRecordBytes?: number;
+    /** Entries returned by one call. Default 10 000. */
+    maxEntries?: number;
     /** Wall-clock budget for the call. Default 30 000 ms. */
     deadlineMs?: number;
 }
@@ -39,18 +52,36 @@ export interface PromptHistoryResumeState {
     /** `session_meta` has been seen. */
     meta?: boolean;
 }
+/** Answer metadata gathered so far for an open prompt; never text. */
+export interface PromptHistoryAnswerState {
+    status: 'pending' | 'provisional';
+    /** Claude message id of the first answering call, once one has been seen. */
+    id?: string;
+    model?: string;
+    usage?: PromptHistoryUsage;
+}
 /**
  * The last emitted prompt of a file whose answering call is not final yet.
- * Holds a replay position and a metadata digest, never prompt or answer text.
+ * Holds a position and answer metadata, never prompt or answer text.
  */
 export interface PromptHistoryPendingPrompt {
     ordinal: number;
-    /** Byte offset where the prompt's log line starts; the next pass replays from here. */
+    /** Byte offset where the prompt's log line starts. */
     offset: number;
+    /** Byte length of the prompt's log line, without its newline. */
+    lineBytes?: number;
+    /** Timestamp of the prompt line, used to verify it when it is read again. */
+    timestamp?: string;
     /** Digest of the model, usage, and status last returned for this prompt. */
     emitted: string;
     /** Codex parse state as it was just before the prompt line. */
     state?: PromptHistoryResumeState;
+    /**
+     * Answer tracking to continue from the session offset. Absent only on a
+     * session carried unchanged from a version 2 cursor, which replays from
+     * `offset` the next time it is read.
+     */
+    answer?: PromptHistoryAnswerState;
 }
 export interface PromptHistoryCursorSession {
     provider: PromptHistoryProvider;
@@ -59,19 +90,30 @@ export interface PromptHistoryCursorSession {
     mtimeMs: number;
     /** Ordinal of the last human prompt seen in this file, or -1 when none. */
     lastOrdinal: number;
-    /** Byte offset just past the last complete line processed. */
+    /** Byte offset where the next pass starts reading. */
     offset: number;
     state?: PromptHistoryResumeState;
     pending?: PromptHistoryPendingPrompt;
+    /** A bound stopped the last pass before the end of the file. */
+    backlog?: true;
+    /** `offset` is inside a record over `maxRecordBytes`; skip to its newline. */
+    skipping?: true;
+    /** Digest of the file's first bytes, used to detect a rewritten file. */
+    head?: {
+        bytes: number;
+        hash: string;
+    };
 }
 /**
  * JSON-serializable resume point. Keys are opaque and never contain paths, but
  * values can (Codex `state.cwd`), so treat the cursor as sensitive local state.
- * Version 1 cursors from 0.27.0 are accepted as input; output is always version 2.
+ * Version 1 and 2 cursors are accepted as input; output is always version 3.
  */
 export interface PromptHistoryCursor {
-    version: 2;
+    version: 3;
     sessions: Record<string, PromptHistoryCursorSession>;
+    /** Opaque key of the file the next call starts with, after a call-wide stop. */
+    next?: string;
 }
 /**
  * How settled an entry's `model` / `usage` are:
@@ -112,14 +154,35 @@ export interface PromptHistoryEntry {
      */
     metadataStatus: PromptHistoryMetadataStatus;
 }
+/**
+ * A log record permanently left out. Records over `maxRecordBytes` and lines
+ * that are not valid JSON are never parsed, so any prompt inside is lost, and
+ * the ordinals of later prompts in that file do not count it.
+ */
+export interface PromptHistoryExclusion {
+    provider: PromptHistoryProvider;
+    sessionId: string;
+    reason: 'oversizedRecord' | 'malformedRecord';
+    /** Byte offset where the record starts. */
+    offset: number;
+}
 export interface PromptHistoryStats {
     sessionsScanned: number;
     sessionsSkippedUnchanged: number;
     /** Scanned sessions with human prompts but none inside the allowed roots. */
     sessionsOutOfScope: number;
+    /** Files whose `maxFileBytes` budget ran out; they continue on the next call. */
     filesOverSizeLimit: number;
+    /** Scanned sessions left with unread bytes by any bound; they continue next call. */
+    sessionsWithBacklog: number;
+    /** Sessions whose file was truncated or rewritten and was read again from the start. */
+    sessionsRewritten: number;
     /** Codex sessions skipped as non-interactive (subagent, fork, unknown source). */
     sessionsNotInteractive: number;
+    /** Records skipped for exceeding `maxRecordBytes` (see `exclusions`). */
+    recordsOverSizeLimit: number;
+    /** Complete lines that were not valid JSON (see `exclusions`). */
+    recordsMalformed: number;
     /** Human prompts dropped because the log line had no usable timestamp. */
     promptsMissingTimestamp: number;
     /** Human prompts dropped because their cwd was missing or outside the roots. */
@@ -143,7 +206,16 @@ export interface PromptHistoryResult {
     entries: PromptHistoryEntry[];
     cursor: PromptHistoryCursor;
     stats: PromptHistoryStats;
+    /** Resumable stops: calling again with the returned cursor continues past them. */
     boundsHit: PromptHistoryBound[];
+    /** Permanent exclusions found in this call (at most 1000; stats count all). */
+    exclusions: PromptHistoryExclusion[];
+    /** A bound left bytes or files unread; call again with the cursor to continue. */
+    hasMore: boolean;
 }
+/** Test hooks; not part of the public entry points. */
+export declare const __promptHistoryTesting: {
+    setReadChunkBytes(bytes: number | null): void;
+};
 /** Collect human-typed prompts from Claude Code and Codex logs under the given roots. */
 export declare function collectPromptHistory(options: CollectPromptHistoryOptions): Promise<PromptHistoryResult>;