cognitive-core 0.0.2 → 0.1.2

package/docs/PLAN-session-bank-implementation.md

@@ -0,0 +1,474 @@
+# Implementation Plan: Session Bank (Entire-backed)
+
+> **Design doc**: [DESIGN-session-bank.md](./DESIGN-session-bank.md)
+> **Branch**: `claude/session-bank-design-41pv3`
+
+## Overview
+
+Build a session bank that reads Entire CLI checkpoint data from the `entire/checkpoints/v1` orphan branch, exposes a generic session query API, and bridges into cognitive-core's learning pipeline via a `TrajectorySource` adapter.
+
+The session bank lives as a self-contained module at `src/session-bank/` (not a separate package — avoids monorepo overhead). It has zero imports from the rest of cognitive-core. The adapter (`EntireTrajectorySource`) lives in `src/learning/trajectory-sources/entire.ts` and bridges the two.
+
+---
+
+## Phase 1: Types + Parser
+
+**Goal**: Define the session-bank's own types and implement JSONL/metadata parsing, fully unit-tested with fixture data.
+
+### Step 1.1 — Types (`src/session-bank/types.ts`)
+
+Define all session-bank types (these are NOT cognitive-core types):
+
+- `SessionRecord` — parsed checkpoint session (main type)
+- `SessionMetadata` — per-session metadata fields
+- `TranscriptMessage` — parsed JSONL line
+- `ContentBlock` discriminated union: `TextBlock | ToolUseBlock | ToolResultBlock`
+- `SessionSummary` — AI-generated summary (from Entire)
+- `TokenUsage` — token counts with computed `total`
+- `Attribution` — line-level code attribution
+- `SessionSource` — provenance (branch, path, commit SHA)
+- `CheckpointRef` — lightweight reference for enumeration
+- `SessionQuery` — query filter options
+- `SessionQueryResult` — query response
+- `ProcessingLogEntry` — processing state tracking
+- `SessionBankOptions` — constructor options
+
+No external dependencies. Use plain TypeScript interfaces (no zod — keep standalone).
+
+### Step 1.2 — Test Fixtures (`tests/session-bank/fixtures/`)
+
+Create realistic Entire-format fixture files:
+
+- `simple-session/metadata.json` — minimal session metadata with summary
+- `simple-session/full.jsonl` — 5-line transcript (user prompt → assistant text → tool_use → tool_result → assistant text)
+- `multi-tool-session/metadata.json` — session with multiple tool calls, no summary
+- `multi-tool-session/full.jsonl` — 10+ lines with Read, Edit, Bash tool calls
+- `root-metadata.json` — checkpoint-level summary with session paths
+- `chunked-session/metadata.json` + `full.jsonl.001` + `full.jsonl.002` — chunked transcript
+
+Fixture metadata must match Entire's real Go struct format (snake_case keys):
+```json
+{
+  "cli_version": "0.3.0",
+  "checkpoint_id": "a3b2c4d5e6f7",
+  "session_id": "2026-01-08-abc123de-f456-7890-abcd-ef1234567890",
+  "strategy": "manual-commit",
+  "created_at": "2026-01-08T10:30:00Z",
+  "branch": "main",
+  "files_touched": ["src/auth.ts"],
+  "agent": "Claude Code",
+  "is_task": false,
+  "checkpoint_transcript_start": 0,
+  "token_usage": { ... },
+  "summary": { ... },
+  "initial_attribution": { ... }
+}
+```
+
+### Step 1.3 — Parser (`src/session-bank/parser.ts`)
+
+`EntireTranscriptParser` class:
+
+- `parseSession(metadataJson, transcriptJsonl, checkpointPath, sessionIndex)` → `SessionRecord`
+- `parseMetadata(metadataJson)` → parsed metadata + sessionId + checkpointId + summary + tokenUsage + attribution
+- `parseTranscript(jsonlContent)` → `TranscriptMessage[]`
+  - Handle string content (wrap in `[{ type: 'text', text: ... }]`)
+  - Handle array content (map to `ContentBlock` discriminated union)
+  - Skip malformed lines (log warning, don't throw)
+  - Track `lineIndex` for each message
+- `parseChunkedTranscript(chunks)` → concatenate then parse
+- `parseCheckpointSummary(metadataJson)` → `CheckpointRef`
+
+Key parsing details:
+- Entire uses **snake_case** JSON keys (Go convention). Parser converts to **camelCase**.
+- `created_at` is RFC3339 string → `Date`
+- `token_usage.total` is computed as sum of input + output + cache tokens
+- `message.content` can be a string OR array — normalize to always be `ContentBlock[]`
+
+### Step 1.4 — Parser Tests (`tests/session-bank/parser.test.ts`)
+
+Test cases:
+- Parse simple session metadata (with summary)
+- Parse metadata without summary
+- Parse transcript with text-only content
+- Parse transcript with tool_use + tool_result blocks
+- Parse transcript where user content is a plain string
+- Parse transcript where user content is an array of content blocks
+- Skip malformed JSONL lines gracefully
+- Parse chunked transcript (concatenation)
+- Parse root checkpoint summary
+- Token usage `total` computation
+- Attribution parsing
+
+---
+
+## Phase 2: Git Reader + SessionBank Facade
+
+**Goal**: Read from the orphan branch via git plumbing commands and expose the full query API.
+
+### Step 2.1 — Git Reader (`src/session-bank/git-reader.ts`)
+
+`EntireGitReader` class:
+
+- Constructor takes `{ repoPath: string; branch?: string }` (default branch: `'entire/checkpoints/v1'`)
+- `branchExists()` → `boolean` via `git rev-parse --verify <branch>`
+- `listCheckpoints()` → `string[]` paths like `["a3/b2c4d5e6f7", ...]`
+  - Two-level enumeration: `git ls-tree --name-only <branch>` for shard dirs, then `git ls-tree --name-only <branch> <shard>/` for checkpoint dirs
+  - Combine into full paths
+- `listSessions(checkpointPath)` → `string[]` numeric session indices
+  - `git ls-tree --name-only <branch> <path>/`
+  - Filter to numeric directories (exclude `tasks/`, `metadata.json`)
+- `readFile(path)` → `string` via `git show <branch>:<path>`
+- `readFiles(paths)` → `Map<string, string>` via batch `git cat-file --batch` (single process spawn)
+- `readFileOrNull(path)` → `string | null` (catch errors)
+- `getBranchHead()` → `string | null` (commit SHA for cache invalidation)
+
+All git operations use `child_process.execFile` (not shell=true) for safety. Errors from non-existent paths return null/empty rather than throwing.
+
+### Step 2.2 — Git Reader Tests (`tests/session-bank/git-reader.test.ts`)
+
+Use a temporary git repo with a real orphan branch:
+
+```typescript
+// beforeEach: create temp dir, git init, create orphan branch with fixture data
+// afterEach: rm temp dir
+```
+
+Test cases:
+- `branchExists()` returns true when branch exists
+- `branchExists()` returns false when branch doesn't exist
+- `listCheckpoints()` returns checkpoint paths
+- `listSessions()` returns session indices
+- `readFile()` reads file content from orphan branch
+- `readFiles()` batch-reads multiple files
+- `readFileOrNull()` returns null for missing files
+- `getBranchHead()` returns commit SHA
+
+### Step 2.3 — SessionBank Facade (`src/session-bank/session-bank.ts`)
+
+`SessionBank` class:
+
+**Constructor** takes `SessionBankOptions`:
+```typescript
+{
+  repoPath?: string;      // default: process.cwd()
+  branch?: string;        // default: 'entire/checkpoints/v1'
+  stateDir?: string;      // default: '.atlas/session-bank'
+  cache?: boolean;        // default: true
+}
+```
+
+**Lifecycle:**
+- `init()` — verify branch exists, load processing state from disk
+- `isAvailable()` — whether the orphan branch was found
+- `close()` — flush processing state
+
+**Discovery (metadata-only, fast):**
+- `listCheckpoints()` → `CheckpointRef[]`
+  - Enumerate checkpoint dirs via git reader
+  - Read root `metadata.json` for each to get session count + session IDs
+  - Return lightweight refs (no transcript loading)
+- `discover()` → `SessionRecord[]` (without transcripts populated)
+  - List all sessions, load metadata only
+- `count(query?)` → `number`
+
+**Querying (loads transcripts on demand):**
+- `query(query?)` → `SessionQueryResult`
+  - Load metadata for all sessions
+  - Apply filters: since, before, branch, agent, strategy, hasSummary, unprocessedOnly
+  - For matching sessions, load full.jsonl and parse transcript
+  - Apply `search` filter (full-text across user text blocks)
+  - Sort (newest/oldest)
+  - Apply limit
+  - Return with total count
+- `getBySessionId(sessionId)` → `SessionRecord | undefined`
+  - Search across all checkpoints (metadata scan for matching session_id)
+- `getByCheckpointId(checkpointId)` → `SessionRecord[]`
+  - Direct path lookup: `<id[:2]>/<id[2:]>/`
+- `getByCommit(commitSha)` → `SessionRecord[]`
+  - Run `git log --format=%(trailers:key=Entire-Checkpoint) <sha> -1`
+  - Parse checkpoint ID from trailer
+  - Look up via `getByCheckpointId()`
+
+**Processing tracking:**
+- `markProcessed(sessionId, metadata?)` — add to processing log, persist
+- `isProcessed(sessionId)` — check in-memory set
+- `getProcessingLog()` → `ProcessingLogEntry[]`
+- State persisted as JSON in `stateDir/processed.json`
+
+**Caching:**
+- `invalidateCache()` — clear in-memory cache
+- On query, check orphan branch HEAD SHA. If changed since last cache, invalidate.
+- Cache stores parsed `SessionRecord` keyed by `checkpointPath/sessionIndex`
+
+### Step 2.4 — SessionBank Tests (`tests/session-bank/session-bank.test.ts`)
+
+Use a temporary git repo with fixture data on an orphan branch.
+
+Test cases:
+- `init()` succeeds with valid orphan branch
+- `isAvailable()` returns false when no orphan branch
+- `listCheckpoints()` discovers checkpoints
+- `discover()` returns sessions with metadata
+- `query()` with no filters returns all sessions
+- `query({ since })` filters by date
+- `query({ branch })` filters by branch
+- `query({ agent })` filters by agent
+- `query({ unprocessedOnly: true })` skips processed sessions
+- `query({ search })` full-text search across user messages
+- `query({ limit })` respects limit
+- `query({ sort: 'oldest' })` sorts ascending
+- `getBySessionId()` finds specific session
+- `getByCheckpointId()` finds all sessions in checkpoint
+- `markProcessed()` + `isProcessed()` round-trip
+- Processing state persists across instances
+- Cache invalidation on branch HEAD change
+
+### Step 2.5 — Barrel Export (`src/session-bank/index.ts`)
+
+Export everything:
+```typescript
+export { SessionBank } from './session-bank.js';
+export { EntireGitReader } from './git-reader.js';
+export { EntireTranscriptParser } from './parser.js';
+export * from './types.js';
+```
+
+---
+
+## Phase 3: cognitive-core Adapter
+
+**Goal**: Bridge session-bank into cognitive-core's learning pipeline via `TrajectorySource`.
+
+### Step 3.1 — EntireTrajectorySource (`src/learning/trajectory-sources/entire.ts`)
+
+Implements `TrajectorySource` interface:
+
+```typescript
+class EntireTrajectorySource implements TrajectorySource {
+  readonly name = 'entire';
+  constructor(sessionBank: SessionBank, options?: EntireSynthesisOptions);
+  async load(options?: TrajectoryLoadOptions): Promise<TrajectoryWithAnalysis[]>;
+  async count(options?: TrajectoryLoadOptions): Promise<number>;
+  synthesize(session: SessionRecord): Trajectory;
+}
+```
+
+**Synthesis logic** (SessionRecord → Trajectory):
+
+- **ID**: `entire-{checkpointId}-{sessionId}`
+- **Task description**: `summary.intent` (prefer) → first user text message (fallback) → `'Unknown task'`
+- **Task domain**: configurable override → infer from `filesTouched` extensions
+- **Steps**: One `Step` per assistant tool_use block. Text blocks → `thought`. Tool results → `observation`.
+- **Outcome**: summary-based (default) — parse `summary.outcome` for failure keywords. Heuristic fallback — `filesTouched.length > 0` implies success.
+- **Token count**: `tokenUsage.total` directly
+- **Metadata**: preserve `source: 'entire'`, checkpoint/session IDs, branch, files, strategy, attribution
+
+Also export `EntireSynthesisOptions`:
+```typescript
+interface EntireSynthesisOptions {
+  domain?: string;
+  outcomeStrategy?: 'summary' | 'heuristic' | 'llm';
+}
+```
+
+### Step 3.2 — Update Trajectory Source Exports
+
+- `src/learning/trajectory-sources/index.ts` — add `export { EntireTrajectorySource } from './entire.js'`
+- `src/learning/index.ts` — add `EntireTrajectorySource` to exports
+- `src/index.ts` — add `EntireTrajectorySource` to learning exports
+
+### Step 3.3 — Config Addition (`src/types/config.ts`)
+
+Add `SessionBankConfigSchema` to `AtlasConfigSchema`:
+
+```typescript
+const SessionBankConfigSchema = z.object({
+  enabled: z.boolean().default(false),
+  branch: z.string().default('entire/checkpoints/v1'),
+  stateDir: z.string().default('session-bank'),
+  outcomeStrategy: z.enum(['summary', 'heuristic', 'llm']).default('summary'),
+  autoIngest: z.boolean().default(false),
+  maxIngestBatch: z.number().default(20),
+});
+```
+
+Add `sessionBank: SessionBankConfigSchema.default({})` to `AtlasConfigSchema`.
+Update `mergeConfig()` to include `sessionBank`.
+
+### Step 3.4 — Atlas Integration (`src/atlas.ts`)
+
+Add to Atlas class:
+
+- `private sessionBank: SessionBank | null = null`
+- In `init()`: if `config.sessionBank.enabled`, create and init SessionBank
+- `getSessionBank()` → `SessionBank | null`
+- `ingestEntireSessions(options?)` → `IngestResult`
+  - Create `EntireTrajectorySource`
+  - Query unprocessed sessions
+  - For each: `synthesize()` → `processTrajectory()` → `markProcessed()`
+  - Return ingestion stats
+- In `close()`: close session bank if active
+
+Define `IngestResult`:
+```typescript
+interface IngestResult {
+  discovered: number;
+  ingested: number;
+  failed: number;
+  errors: Array<{ sessionId: string; error: string }>;
+}
+```
+
+### Step 3.5 — Adapter Tests (`tests/learning/entire-trajectory-source.test.ts`)
+
+Test cases:
+- `synthesize()` converts SessionRecord to Trajectory with correct fields
+- Task description prefers `summary.intent`
+- Task description falls back to first user message
+- Domain inference from file extensions (.ts, .py, .go, .rs, .test.ts)
+- Domain override via options
+- Steps extracted from tool_use blocks
+- Pure reasoning steps (no tool calls) handled
+- Outcome inference from summary (success keywords)
+- Outcome inference from summary (failure keywords)
+- Heuristic fallback when no summary
+- Token usage carried through
+- Metadata preserves entire-specific fields
+- `load()` integrates with SessionBank query
+- `count()` delegates to SessionBank
+
+### Step 3.6 — Integration Test (`tests/integration/session-bank.test.ts`)
+
+End-to-end test:
+1. Create temp git repo with orphan branch containing fixture data
+2. Create SessionBank, init
+3. Create EntireTrajectorySource
+4. Load trajectories
+5. Verify synthesized Trajectory objects are valid (pass `TrajectorySchema.parse()`)
+6. Process through LearningPipeline
+7. Verify experiences created in memory
+
+---
+
+## Phase 4: Polish
+
+### Step 4.1 — Package.json Export
+
+Add session-bank as an export path in `package.json`:
+```json
+"./session-bank": {
+  "import": "./dist/session-bank/index.js",
+  "types": "./dist/session-bank/index.d.ts"
+}
+```
+
+### Step 4.2 — Commit-Linked Queries
+
+Implement `getByCommit()` in SessionBank:
+- Use `git log --format=%(trailers:key=Entire-Checkpoint,valueonly) <sha> -1` to extract trailer
+- Look up checkpoint by ID
+- Handle commits without Entire trailers (return empty array)
+
+### Step 4.3 — Full-Text Search
+
+Implement `search` filter in `SessionBank.query()`:
+- Extract text from all user `TextBlock` content blocks
+- Case-insensitive substring match against search term
+- Applied after metadata filters, before limit
+
+### Step 4.4 — Error Tolerance
+
+Ensure graceful handling of:
+- Corrupt/truncated JSONL files (skip malformed lines)
+- Missing metadata fields (use defaults)
+- Missing transcript files (return session without transcript)
+- Concurrent git operations (retry on lock errors)
+- Very large transcripts (streaming JSONL parser if needed — defer unless hit in practice)
+
+---
+
+## File Manifest
+
+### New Files
+
+| File | Purpose |
+|------|---------|
+| `src/session-bank/types.ts` | Session bank type definitions |
+| `src/session-bank/parser.ts` | JSONL + metadata parser |
+| `src/session-bank/git-reader.ts` | Git plumbing for orphan branch |
+| `src/session-bank/session-bank.ts` | SessionBank facade |
+| `src/session-bank/index.ts` | Barrel exports |
+| `src/learning/trajectory-sources/entire.ts` | TrajectorySource adapter |
+| `tests/session-bank/parser.test.ts` | Parser unit tests |
+| `tests/session-bank/git-reader.test.ts` | Git reader unit tests |
+| `tests/session-bank/session-bank.test.ts` | SessionBank facade tests |
+| `tests/session-bank/fixtures/simple-session/metadata.json` | Fixture |
+| `tests/session-bank/fixtures/simple-session/full.jsonl` | Fixture |
+| `tests/session-bank/fixtures/multi-tool-session/metadata.json` | Fixture |
+| `tests/session-bank/fixtures/multi-tool-session/full.jsonl` | Fixture |
+| `tests/session-bank/fixtures/root-metadata.json` | Fixture |
+| `tests/learning/entire-trajectory-source.test.ts` | Adapter tests |
+| `tests/integration/session-bank.test.ts` | End-to-end test |
+
+### Modified Files
+
+| File | Change |
+|------|--------|
+| `src/learning/trajectory-sources/index.ts` | Add EntireTrajectorySource export |
+| `src/learning/index.ts` | Add EntireTrajectorySource export |
+| `src/types/config.ts` | Add SessionBankConfigSchema to AtlasConfig |
+| `src/atlas.ts` | Add sessionBank field, init, ingestEntireSessions(), close() |
+| `src/index.ts` | Add session-bank + EntireTrajectorySource exports |
+| `package.json` | Add `./session-bank` export path |
+
+---
+
+## Implementation Order
+
+```
+Phase 1 (Types + Parser) ──── no dependencies
+  1.1  src/session-bank/types.ts
+  1.2  tests/session-bank/fixtures/*
+  1.3  src/session-bank/parser.ts
+  1.4  tests/session-bank/parser.test.ts
+  ── checkpoint: parser tests pass ──
+
+Phase 2 (Git Reader + Facade) ──── depends on Phase 1
+  2.1  src/session-bank/git-reader.ts
+  2.2  tests/session-bank/git-reader.test.ts
+  2.3  src/session-bank/session-bank.ts
+  2.4  tests/session-bank/session-bank.test.ts
+  2.5  src/session-bank/index.ts
+  ── checkpoint: all session-bank tests pass ──
+
+Phase 3 (cognitive-core Adapter) ──── depends on Phase 2
+  3.1  src/learning/trajectory-sources/entire.ts
+  3.2  barrel export updates
+  3.3  src/types/config.ts update
+  3.4  src/atlas.ts update
+  3.5  tests/learning/entire-trajectory-source.test.ts
+  3.6  tests/integration/session-bank.test.ts
+  ── checkpoint: all tests pass, typecheck clean ──
+
+Phase 4 (Polish) ──── depends on Phase 3
+  4.1  package.json export path
+  4.2  getByCommit() implementation
+  4.3  search filter implementation
+  4.4  error tolerance hardening
+  ── checkpoint: npm run test:run && npm run typecheck ──
+```
+
+## Key Constraints
+
+1. **Zero cognitive-core imports in session-bank** — `src/session-bank/` must not import from `src/types/`, `src/memory/`, etc. The adapter (`src/learning/trajectory-sources/entire.ts`) is the only bridge.
+
+2. **No external dependencies for session-bank** — Use only Node.js built-ins (`child_process`, `fs/promises`, `path`). No zod, no third-party libraries.
+
+3. **Snake_case → camelCase in parser** — Entire's Go structs use snake_case JSON. Parser converts all field names.
+
+4. **Never write to the orphan branch** — Session bank is read-only. Processing state goes in `.atlas/session-bank/`.
+
+5. **Lazy transcript loading** — `discover()` and `listCheckpoints()` never load JSONL files. Only `query()`, `getBySessionId()`, `getByCheckpointId()` load transcripts.
+
+6. **Existing tests must continue to pass** — No breaking changes to existing types, interfaces, or behavior.