codealmanac 0.1.4 → 0.1.6

package/guides/mini.md

@@ -144,7 +144,7 @@ wikilink-syntax
 # 2. Triage with --lead
 $ almanac show sqlite-indexer --lead
 The indexer (`src/indexer/`) builds and maintains `.almanac/index.db` — a
-SQLite database that powers all query commands (`search`, `info`, `health`,
+SQLite database that powers all query commands (`search`, `show`, `health`,
 `topics show`). It runs silently before every query command, comparing page
 file mtimes against the stored `content_hash`; only changed or new pages are
 re-parsed.
@@ -216,6 +216,32 @@ No logs at all → the hook isn't installed, or bailed before backgrounding, or
 
 ---
 
+## Staying current
+
+codealmanac checks for updates in the background (once per 24h) after each
+command. When a new version is available, you'll see a stderr banner on
+every subsequent invocation:
+
+```
+! codealmanac 0.1.6 available (you're on 0.1.5) — run: almanac update
+```
+
+The banner shows on every command until you update or dismiss it. Run:
+
+```bash
+almanac update              # upgrade to latest (foreground `npm i -g codealmanac@latest`)
+almanac update --dismiss    # skip this version; banner goes away until the next release
+almanac update --check      # check now without installing (bypasses 24h cache)
+almanac doctor              # see current update status + notifier setting
+```
+
+Auto-install is deliberately NOT the default — silent install without consent
+violates the trust contract, npm prefixes diverge across version managers, and a
+mid-invocation binary swap corrupts dynamic imports. Tier B (nag + manual
+install) is the design. See `almanac update --help` for the full flag set.
+
+---
+
 ## When in doubt
 
 - `.almanac/README.md` — repo-specific conventions + notability bar

package/guides/processing/claude-code.md

@@ -0,0 +1,152 @@
+# Processing Claude Code Sessions
+
+## Format overview
+
+Claude Code stores sessions as JSONL files (one JSON object per line) at:
+```
+~/.claude/projects/<project-hash>/<session-uuid>.jsonl
+```
+
+The project hash is a path with slashes replaced by dashes (e.g., `-Users-rohan-Desktop-Projects-myrepo`). Each session file contains the full conversation history including tool calls, tool results, thinking blocks, and metadata.
+
+Typical session sizes: 150KB (short Q&A) to 8MB+ (multi-hour coding session). Line counts range from ~75 to ~1,750.
+
+## Record types
+
+Each line is a JSON object with a `type` field at the top level:
+
+| Type | Frequency | What it contains |
+|------|-----------|-----------------|
+| `assistant` | ~40% of records, ~15% of bytes | Model responses: text, tool_use, thinking blocks. Each content item is in `message.content[]` |
+| `user` | ~35% of records, ~55% of bytes | Human messages OR tool results. Check `message.content[].type` to distinguish |
+| `attachment` | ~5% of records, ~7% of bytes | System context injected by the harness: deferred tool lists, skill listings, memory, task reminders, edited file snippets |
+| `file-history-snapshot` | ~5% of records, <1% of bytes | Checkpoint markers for undo/redo. Always tiny (~250 bytes) |
+| `permission-mode` | ~3% of records, <1% of bytes | Records when permission mode changes (e.g., `bypassPermissions`) |
+| `last-prompt` | ~3% of records, <1% of bytes | Marks turn boundaries. ~120 bytes each |
+| `system` | ~2% of records, <1% of bytes | System messages injected mid-conversation. Often empty content |
+| `ai-title` | rare | Auto-generated session title |
+| `queue-operation` | rare | Queued follow-up commands |
+
+## What to extract (signal)
+
+### 1. Human messages (highest signal density)
+- **Where:** Records with `type: "user"` where `message.content[]` contains items with `type: "text"`
+- **Also check:** `userType` field -- `"external"` means the actual human typed this
+- **What:** Intent, requirements, feedback, decisions, bug reports, design direction
+- **Example pattern:** `{"type": "user", "message": {"content": [{"type": "text", "text": "What problems did you run into?"}]}}`
+
+### 2. Assistant text responses
+- **Where:** Records with `type: "assistant"`, then `message.content[]` items with `type: "text"`
+- **What:** Explanations, decisions, summaries, architecture analysis, bug diagnoses
+- **Typical size:** 100-3000 chars per text block
+- **Example pattern:** The assistant explains a root cause, summarizes what was built, or describes a design decision
+
+### 3. Subagent results (high-value signal hidden in noise)
+- **Where:** Records with `type: "user"` that have a top-level `toolUseResult` field with `agentType` set
+- **What:** Complete results from subagents (review, critic, pair, etc.). The `content` field contains the full subagent output, often multi-thousand-character analysis
+- **Key fields:** `toolUseResult.agentType` (e.g., "general-purpose", "review", "critic", "pair"), `toolUseResult.content[].text`, `toolUseResult.prompt` (what the subagent was asked to do)
+- **Why it matters:** These are often the densest signal in a session -- complete audit reports, code reviews, architecture analyses
+
+### 4. Session metadata
+- **Where:** First few records of the file
+- **Key fields on user records:** `cwd`, `gitBranch`, `version`, `timestamp`, `sessionId`, `entrypoint` (cli vs other)
+- **Attachment records** with `type: "nested_memory"` contain the project's memory/context
+
+## What to skip (noise)
+
+### 1. Tool results (~24% of total bytes) -- SKIP
+- **Where:** `user` records where `message.content[]` has `type: "tool_result"`
+- **Why skip:** These are file contents, grep results, build output, test output. The actual files are in the repo; the output is transient
+- **The biggest offenders:** Read tool results can be 150KB+ (entire file contents dumped inline)
+
+### 2. Tool calls (~8% of total bytes) -- SKIP or SUMMARIZE
+- **Where:** `assistant` records where `message.content[]` has `type: "tool_use"`
+- **Contains:** `name` (Bash, Read, Grep, Edit, Write, Glob) and `input` (command, file path, pattern)
+- **Why skip:** The sequence of tool calls is operational, not knowledge. Exception: summarize the *pattern* of tool usage ("read 15 files in src/auth/")
+
+### 3. Wrapper overhead (~38% of total bytes in large sessions) -- SKIP
+- **Where:** Top-level fields on `user` records: `parentUuid`, `sourceToolAssistantUUID`, `toolUseResult` (when not a subagent), `slug`, `requestId`, `isMeta`, `isSidechain`
+- **Why skip:** The `toolUseResult` field on user records DUPLICATES the tool result content that already appears in `message.content[]`. This is the single biggest source of bloat. In one 8MB session, `toolUseResult` alone was 2.7MB (33%)
+- **EXCEPTION:** When `toolUseResult.agentType` is set, this is a subagent result and IS signal
+
+### 4. Empty thinking blocks (~6% in some sessions) -- SKIP
+- **Where:** `assistant` records, `message.content[]` with `type: "thinking"` but `thinking: ""`
+- **Why:** Claude Code often records thinking blocks with empty content (the actual thinking happened but was not persisted). These are pure waste
+
+### 5. Attachments (~7%) -- SKIP
+- **Where:** `type: "attachment"` records
+- **Contains:** `deferred_tools_delta` (tool availability lists), `skill_listing` (repeated skill menus), `task_reminder` (repeated TODO lists), `mcp_instructions_delta` (MCP setup)
+- **Why skip:** Harness infrastructure, not knowledge. Repeated across turns
+
+### 6. Metadata records (<1%) -- SKIP
+- `file-history-snapshot`, `permission-mode`, `last-prompt`, `queue-operation`, `ai-title`
+
+### 7. Base64 image data (~1-7% in some sessions) -- SKIP
+- **Where:** `user` records with `message.content[]` containing `type: "image"` and `source.type: "base64"`
+- **Why skip:** Screenshots pasted by the user. Can be 100KB+ of base64 per image. Not extractable as knowledge
+
+## What to summarize
+
+These patterns should be compressed rather than fully extracted or fully skipped:
+
+| Pattern | Summarize as |
+|---------|-------------|
+| 10+ consecutive tool_use/tool_result pairs reading files | "Read N files in {directory pattern}" |
+| grep/glob sequences searching for a pattern | "Searched for {pattern} across {scope}" |
+| Edit tool calls modifying files | "Modified {file}: {description from the Edit input}" |
+| Bash commands running tests | "Ran tests: {pass/fail summary from result}" |
+| Bash commands running builds | "Built {target}: {success/failure}" |
+
+## Extraction approach
+
+1. **Parse the JSONL file** line by line. Each line is one JSON object.
+2. **First pass -- extract metadata:** From the first `user` record, grab `cwd`, `gitBranch`, `version`, `timestamp`, `sessionId`.
+3. **For each record, check `type`:**
+   - `type: "user"` with `message.content[].type == "text"` -> **extract as human message**
+   - `type: "user"` with `toolUseResult.agentType` set -> **extract subagent result** from `toolUseResult.content[].text`
+   - `type: "user"` with `message.content[].type == "tool_result"` -> **skip** (or summarize tool call patterns)
+   - `type: "assistant"` with `message.content[].type == "text"` -> **extract as assistant reasoning**
+   - `type: "assistant"` with `message.content[].type == "thinking"` and non-empty `thinking` field -> **extract as internal reasoning**
+   - `type: "assistant"` with `message.content[].type == "tool_use"` -> **skip or summarize**
+   - `type: "attachment"`, `type: "system"`, metadata types -> **skip**
+4. **Second pass -- deduplicate:** The `toolUseResult` field on user records often duplicates content from `message.content[]`. Always prefer `message.content[]` and only use `toolUseResult` for subagent data.
+5. **Third pass -- compress tool sequences:** Collapse consecutive tool_use + tool_result pairs into summaries.
+
+## Example: signal extraction from a real session
+
+**Human intent (from user record):**
+> "Tell me about our ref token?"
+Signal: User wants to understand the ref token system.
+
+**Assistant reasoning (from assistant text):**
+> "Ref Token: Opaque backend-signed token that binds a locally-edited .md file to a specific page in the DB. Backend signs on download, verifies on publish. No signing secret ever touches the client."
+Signal: Architectural explanation of the ref token system.
+
+**Assistant problem diagnosis (from assistant text):**
+> "Root cause: the codealmanac CLI expects topics.yaml in the new list format but .almanac/topics.yaml is still in the old dict format"
+Signal: Bug identification with root cause.
+
+**Subagent audit (from toolUseResult with agentType):**
+> agentType: "general-purpose", prompt: "You are auditing the hosted editor..."
+> content: "Audit Report: Hosted Editor / Quill Co-Editing (Pre-Phase-5) ... 3 HIGH gaps found ... Fix malformed edit-result surfacing ... Fix Quill session isolation ..."
+Signal: Complete code audit with findings, priorities, and recommendations.
+
+**Noise skipped:** 522 tool results totaling 2MB of file contents, 466KB of empty thinking blocks, 408KB of metadata records, 2.7MB of duplicated toolUseResult data.
+
+## Gotchas
+
+1. **toolUseResult duplication is massive.** In one 8MB session, `toolUseResult` accounted for 33% of the file. It duplicates `message.content[]` tool results AND sometimes contains subagent results. Always check `agentType` before discarding.
+
+2. **Thinking blocks are always empty in recent versions.** The `thinking` field in content items of type `"thinking"` is consistently empty string in observed sessions (95 empty out of 95 in one session). The thinking content is not persisted. Do not expect signal here despite the promising field name.
+
+3. **assistant records contain message-level metadata.** The `message` object has `usage` (token counts), `model` (model name), `stop_reason` (why generation stopped). These can be useful for understanding session dynamics but are not knowledge signal.
+
+4. **user records serve double duty.** A `user` record with `userType: "external"` and text content is a real human message. A `user` record with tool_result content is just the harness returning tool output. Always check `message.content[].type`.
+
+5. **The `isSidechain` field** indicates branched conversations (user went back and tried a different approach). Sidechain records may represent abandoned approaches -- still potentially valuable as "what was tried and rejected."
+
+6. **Base64 images can be huge.** A single screenshot paste can add 100KB+ of base64 data to a user record. These look like small records until you measure them.
+
+7. **Attachment records repeat.** `skill_listing` and `task_reminder` attachments are re-injected at many turn boundaries. The same content appears 10-15 times in a long session.
+
+8. **Subagent data structure:** The `toolUseResult` for subagents includes `toolStats` (readCount, searchCount, bashCount, editFileCount, linesAdded, linesRemoved) and `usage` (input_tokens, output_tokens, cache stats). These are useful metadata about the subagent's work.

package/guides/processing/codex.md

@@ -0,0 +1,214 @@
+# Processing Codex Sessions
+
+## Format overview
+
+Codex stores sessions as JSONL files at:
+```
+~/.codex/sessions/<year>/<month>/<day>/rollout-<timestamp>-<thread-uuid>.jsonl
+```
+
+A SQLite database at `~/.codex/state_5.sqlite` provides session metadata (title, cwd, model, tokens_used, git info) in the `threads` table.
+
+Multiple rollout files for the same timestamp indicate subagent threads spawned by the parent session. The SQLite `source` column reveals this: `"vscode"` for top-level sessions, a JSON blob with `subagent.thread_spawn.parent_thread_id` for child threads.
+
+Typical session sizes: 500KB (short task) to 12MB+ (multi-turn debugging session). Line counts range from ~175 to ~2,800.
+
+## Record types
+
+Each line is a JSON object with a `type` field:
+
+| Type | % of records | % of bytes | What it contains |
+|------|-------------|------------|-----------------|
+| `response_item` | ~55% | ~36% | Model outputs: function calls, function call outputs, messages, reasoning |
+| `event_msg` | ~43% | ~29% | Harness events: command execution results, token counts, agent messages, task lifecycle |
+| `turn_context` | ~2% | ~5% | Per-turn context: model, cwd, instructions, settings. Repeated every turn |
+| `session_meta` | 1 per file | ~1-3% | Session metadata: id, cwd, model, CLI version, base instructions, skills |
+| `compacted` | rare (0-3) | 10-30% when present | Compressed conversation history from context window compaction |
+
+### response_item subtypes (in `payload.type`)
+
+| Subtype | What it contains |
+|---------|-----------------|
+| `function_call` | Tool invocations: `name` (always `exec_command`), `arguments` (JSON with cmd, workdir, yield_time_ms) |
+| `function_call_output` | Tool results: `output` (command stdout/stderr as string) |
+| `message` | Conversation messages. Check `payload.role`: `developer` (system prompts), `user` (human + context), `assistant` (model output) |
+| `reasoning` | Model reasoning. Contains `encrypted_content` (unreadable) and `summary` (always empty in observed data) |
+| `custom_tool_call` | File edit operations via `apply_patch`. Contains unified diff in `input` |
+| `custom_tool_call_output` | Patch application results: success/failure + modified file list |
+| `web_search_call` | Web search invocations (rare) |
+
+### event_msg subtypes (in `payload.type`)
+
+| Subtype | What it contains |
+|---------|-----------------|
+| `user_message` | Human input: `message` (text), `images` (base64 data URIs), `text_elements` |
+| `agent_message` | Model commentary shown to user: `message`, `phase` (always "commentary") |
+| `exec_command_end` | Command execution results (DUPLICATES `function_call_output`): stdout, stderr, aggregated_output, exit_code, duration, command |
+| `token_count` | Token usage and rate limit info per turn |
+| `task_started` | Turn lifecycle: turn_id, model_context_window, collaboration_mode |
+| `task_complete` | Turn completion: `last_agent_message` (the final text shown to user) |
+| `patch_apply_end` | File edit results: stdout, changes list, success boolean |
+| `context_compacted` | Marker that context window was compacted |
+| `turn_aborted` | Turn was cancelled |
+
+## What to extract (signal)
+
+### 1. Human messages (highest signal density)
+- **Where:** `event_msg` records with `payload.type == "user_message"`
+- **Field:** `payload.message`
+- **Also in:** `response_item` with `payload.type == "message"` and `payload.role == "user"`, `payload.content[].type == "input_text"`. The event_msg version is cleaner
+- **Watch for:** The `response_item` version also contains system/developer context injected alongside the real user message. Extract only `input_text` items where the text is NOT wrapped in XML tags like `<environment_context>`, `<permissions instructions>`, `<app-context>`, `<skills_instructions>`, `<collaboration_mode>`
+
+### 2. Agent messages (model's user-facing commentary)
+- **Where:** `event_msg` records with `payload.type == "agent_message"`
+- **Fields:** `payload.message`, `payload.phase`
+- **What:** Short status updates and reasoning the model shares with the user. These are the "thinking out loud" moments
+- **Example:** "I'm inspecting the repo for category vs topic naming drift and I'll trace it through code, routes, API shapes, and copy so the discrepancies are concrete rather than guessed."
+
+### 3. Task completion summaries
+- **Where:** `event_msg` records with `payload.type == "task_complete"`
+- **Field:** `payload.last_agent_message`
+- **What:** The final, complete response for each turn. Often the densest signal -- the model's synthesized answer after all tool use. Can be multi-thousand characters of analysis
+
+### 4. Assistant output text
+- **Where:** `response_item` with `payload.type == "message"`, `payload.role == "assistant"`, content items with `type: "output_text"`
+- **What:** Model's text responses interspersed with tool calls. Shorter than task_complete but captures incremental reasoning
+
+### 5. File edits (apply_patch)
+- **Where:** `response_item` with `payload.type == "custom_tool_call"` and `payload.name == "apply_patch"`
+- **Field:** `payload.input` contains a unified diff
+- **What:** Every code change the model made. Extract the file path and a summary of the change, not the full diff (the repo has the final state)
+
+### 6. Session metadata
+- **Where:** `session_meta` record (first line of file)
+- **Key fields:** `payload.id`, `payload.cwd`, `payload.model_provider`, `payload.cli_version`, `payload.source`, `payload.model` (in turn_context)
+- **SQLite enrichment:** Query `threads` table for `title`, `tokens_used`, `git_branch`, `first_user_message`, `source` (reveals if this is a subagent)
+
+## What to skip (noise)
+
+### 1. function_call_output records (~17% of bytes) -- SKIP
+- **Where:** `response_item` with `payload.type == "function_call_output"`
+- **Why:** Raw command output (file contents, grep results, build output). Already in the repo or transient
+
+### 2. exec_command_end records (~15% of bytes) -- SKIP
+- **Where:** `event_msg` with `payload.type == "exec_command_end"`
+- **Why:** DUPLICATES `function_call_output` with the same `call_id`. Contains stdout, stderr, aggregated_output redundantly. In one 12MB session, 399 of these consumed 1.9MB
+- **Note:** 100% overlap with function_call_output on shared call_ids
+
+### 3. Reasoning records (~6% of bytes) -- SKIP
+- **Where:** `response_item` with `payload.type == "reasoning"`
+- **Why:** Contains `encrypted_content` (base64 blob, unreadable) and `summary` (consistently empty array in all observed sessions). No extractable signal
+- **Do not confuse with:** `agent_message` records, which ARE readable model reasoning
+
+### 4. turn_context records (~5-25% of bytes) -- SKIP
+- **Where:** `type: "turn_context"`
+- **Why:** Repeated every turn. Contains model name, cwd, instructions, sandbox policy, collaboration mode. Same content each time with minor variations
+
+### 5. session_meta base_instructions (~3% of bytes) -- SKIP
+- **Where:** `session_meta` record, `payload.base_instructions.text`
+- **Why:** Codex's built-in system prompt. Same across all sessions. ~2000 chars of personality and behavior instructions
+
+### 6. Developer instructions in message records -- SKIP
+- **Where:** `response_item` messages with `payload.role == "developer"`
+- **Contains:** `<permissions instructions>`, `<app-context>`, `<collaboration_mode>`, `<apps_instructions>`, `<skills_instructions>` XML blocks
+- **Why:** Harness configuration, not user knowledge. Can be 10KB+ per occurrence
+
+### 7. token_count records (~2%) -- SKIP
+- Rate limit and token usage telemetry
+
+### 8. function_call records (~1.5%) -- SKIP or SUMMARIZE
+- **Where:** `response_item` with `payload.type == "function_call"`
+- **Contains:** `name` (always `exec_command`), `arguments` (cmd, workdir)
+- **Why:** Operational commands. Summarize the pattern, not individual calls
+
+### 9. Base64 images in user_message (~4-7% when present) -- SKIP
+- **Where:** `event_msg` with `payload.type == "user_message"`, `payload.images[]`
+- **Format:** data URIs (`data:image/png;base64,...`), 350KB-550KB each
+- **Also in:** `response_item` message content with `type: "input_image"` and `image_url`
+- **Why:** Screenshots. Not extractable as text knowledge. A single image can be 550KB
+
+### 10. Compacted records (10-30% when present) -- EXTRACT SELECTIVELY
+- **Where:** `type: "compacted"` records
+- **Contains:** `payload.replacement_history[]` -- a compressed version of earlier conversation
+- **Treatment:** These contain summarized versions of earlier turns after context compaction. The `replacement_history` items have `role` and `content[]` with `input_text`/`output_text`. Extract output_text items (model summaries) but skip input_text (already captured from the original records earlier in the file)
+
+## What to summarize
+
+| Pattern | Summarize as |
+|---------|-------------|
+| N consecutive exec_command function_call/output pairs | "Ran N commands exploring {pattern}" |
+| grep/find/sed sequences reading files | "Searched for {pattern} in {directory}" |
+| apply_patch calls | "Modified {file}: {one-line description from diff}" |
+| Multiple agent_message records saying similar things | Keep only the last one before task_complete |
+
+## Extraction approach
+
+1. **Check SQLite first** for session metadata: `SELECT id, title, cwd, model, tokens_used, git_branch, source, first_user_message FROM threads WHERE id = '<thread-id>'`. The `source` field tells you if this is a subagent session.
+
+2. **Parse the JSONL file** line by line.
+
+3. **Extract session_meta** (first record): grab `payload.id`, `payload.cwd`, `payload.source`, `payload.model_provider`.
+
+4. **For each record, route by type and subtype:**
+   - `event_msg` + `user_message` -> **extract** `payload.message` as human input. Note `payload.images` count but skip the base64 data
+   - `event_msg` + `agent_message` -> **extract** `payload.message` as model reasoning
+   - `event_msg` + `task_complete` -> **extract** `payload.last_agent_message` as turn summary
+   - `event_msg` + `exec_command_end` -> **skip** (duplicated in response_item)
+   - `event_msg` + `token_count` / `task_started` / `context_compacted` -> **skip**
+   - `response_item` + `message` + `role: "assistant"` -> **extract** output_text content
+   - `response_item` + `message` + `role: "developer"` or `role: "user"` with XML-tagged content -> **skip**
+   - `response_item` + `message` + `role: "user"` with plain text -> **extract** (but deduplicate against event_msg user_message)
+   - `response_item` + `function_call_output` -> **skip**
+   - `response_item` + `function_call` -> **summarize** command pattern
+   - `response_item` + `reasoning` -> **skip** (encrypted, empty summary)
+   - `response_item` + `custom_tool_call` -> **summarize** file path and change description
+   - `response_item` + `custom_tool_call_output` -> **skip** (just success/fail)
+   - `turn_context` -> **skip**
+   - `compacted` -> **extract** output_text from `payload.replacement_history[]`
+
+5. **Deduplicate across record types:** User messages appear in both `event_msg.user_message` AND `response_item.message` (role: user). Command output appears in both `response_item.function_call_output` AND `event_msg.exec_command_end`. Always prefer the event_msg version for user messages (cleaner), skip the duplicate command output entirely.
+
+6. **Link subagent sessions:** Check the SQLite `source` column. If it contains `subagent.thread_spawn`, this session's knowledge should be attributed to the parent thread. The `parent_thread_id` field links them.
+
+## Example: signal extraction from a real session
+
+**Human intent (from event_msg.user_message):**
+> "Look at our codebase, we have categories and topics, we want to unify everywhere to be called topics. Find all discrepancies."
+Signal: User wants a category-to-topic naming audit.
+
+**Agent reasoning (from event_msg.agent_message):**
+> "I've isolated one concrete runtime mismatch already: the page-topic footer still talks about 'categories' in UI copy while the rest of the product model is 'topics.' I'm now separating first-party mismatches from external-source fields and old design docs so the final list is usable."
+Signal: Agent's approach to categorizing the findings.
+
+**Task completion (from event_msg.task_complete):**
+> "Root Cause: This is not primarily a CSS-specificity problem. The break happens because the suggestion extension's renderHTML() emits bare <ins>/<del> tags without the CSS class..."
+Signal: Complete diagnosis with root cause and fix path.
+
+**File edit (from custom_tool_call):**
+> name: apply_patch, file: SuggestionChangesExtension.ts
+> Change: Added class attribute to renderHTML() output so CSS selectors match
+Signal: What was actually changed and why.
+
+**Noise skipped:** 442 function_call_output records (2.2MB), 399 duplicate exec_command_end records (1.9MB), 266 encrypted reasoning records (787KB), 67 repeated turn_context records (675KB), 368 token_count records (267KB).
+
+## Gotchas
+
+1. **exec_command_end duplicates function_call_output.** They share the same `call_id` and contain the same command output in different field names. 100% overlap observed. Skip exec_command_end entirely.
+
+2. **Reasoning is unreadable.** Despite having `summary` and `content` fields, reasoning records contain only `encrypted_content` (opaque base64) and consistently empty `summary: []`. There is zero extractable signal from reasoning records.
+
+3. **User messages appear twice.** Once in `event_msg.user_message` (clean, just the text + images) and again in `response_item.message` with `role: "user"` (mixed with system context). Use the event_msg version.
+
+4. **Developer messages are system prompts, not human.** `response_item.message` with `role: "developer"` contains harness instructions (permissions, app context, collaboration mode, skills). These are NOT human messages. They are wrapped in XML tags like `<permissions instructions>`, `<app-context>`, etc.
+
+5. **Images are massive.** User messages with screenshots contain base64 data URIs of 350-550KB each. A single image can make a 75-char message record balloon to 550KB. Check `payload.images` length but do not extract the base64 data.
+
+6. **Compacted records contain earlier conversation.** When the context window fills up, Codex compacts history into `compacted` records. The `replacement_history` array contains summarized earlier turns. If you are processing the file start-to-finish, you will see the original records first and then the compacted summary later -- be careful not to double-count.
+
+7. **Subagent sessions are separate files.** A parent session spawns subagent threads that are written to their own rollout files in the same date directory. The SQLite `source` column JSON identifies child threads. To get the complete picture of a multi-agent session, you must read all linked rollout files.
+
+8. **Codex uses `exec_command` for everything.** Unlike Claude Code which has specialized tools (Read, Grep, Edit, Bash), Codex wraps all operations in `exec_command` with shell commands (`sed`, `rg`, `cat`, etc.) or `apply_patch` for file edits. This means function_call records are less informative about intent -- you need to parse the `cmd` field to understand what was done.
+
+9. **task_complete contains the cleanest signal.** The `last_agent_message` in task_complete records is the final synthesized response after all tool use. If you can only extract one thing per turn, extract this.
+
+10. **The model field is in turn_context, not session_meta.** Session_meta has `model_provider` ("openai") but the actual model name (e.g., "gpt-5.4") is in `turn_context.model`.

package/guides/processing/generic.md

@@ -0,0 +1,128 @@
+# Processing Unknown Session Formats
+
+This guide is a fallback for session files that do not match known formats (Claude Code JSONL, Codex rollout JSONL). Use it when you encounter a new tool or an unrecognized file structure.
+
+## Step 1: Identify the format
+
+Check the file extension and first few lines:
+
+- **JSONL** (`.jsonl`): One JSON object per line. Parse each line independently
+- **JSON** (`.json`): Single JSON document. Could be an array of messages or a nested conversation object
+- **Markdown** (`.md`): Likely a conversation export with `## Human` / `## Assistant` headers or similar
+- **Plain text** (`.txt`, `.log`): Look for turn separators (blank lines, `---`, timestamps)
+- **SQLite** (`.sqlite`, `.db`): Database with conversation tables. List tables first, then query
+
+For JSONL, check each line for a `type`, `role`, or `kind` field. The presence of certain fields reveals the format:
+- `type: "user"` / `type: "assistant"` + `message.content` -> Claude Code format
+- `type: "response_item"` / `type: "event_msg"` -> Codex format
+- `role: "user"` / `role: "assistant"` without a wrapper type -> Raw API conversation log
+- `type: "human"` / `type: "ai"` -> LangChain/LangSmith format
+
+## Step 2: Classify each record
+
+For any conversation format, records fall into these universal categories:
+
+### Signal (extract)
+1. **Human messages** -- What the user asked, decided, or directed. These explain intent, requirements, and constraints. Look for:
+   - Records with `role: "user"` or `type: "human"` or `type: "user_message"`
+   - Text that reads like natural language instructions, questions, or feedback
+   - Short messages (under ~500 chars) are almost always signal
+
+2. **AI reasoning text** -- Explanations, analyses, decisions, and summaries the model produced. Look for:
+   - Records with `role: "assistant"` and text content (not tool calls)
+   - Fields named `text`, `content`, `message`, `output`, `response`
+   - Text that explains *why* something was done, not *what* command was run
+
+3. **Final/summary responses** -- The model's synthesized answer after a chain of tool use. Look for:
+   - The last assistant message before the next human message
+   - Fields named `final_response`, `last_message`, `summary`, `result`
+   - These are typically the densest signal per byte
+
+4. **Error messages and failures** -- What went wrong and why. Look for:
+   - Records containing `error`, `failed`, `exception`, `traceback`
+   - These often reveal important constraints, gotchas, or architectural issues
+
+### Noise (skip)
+1. **File contents returned by tools** -- Already in the repo. Look for:
+   - Records with `type: "tool_result"`, `type: "function_call_output"`, or `type: "tool_output"`
+   - Content that looks like source code (imports, function definitions, indented blocks)
+   - Long strings (>2KB) that are clearly file dumps
+   - **Size clue:** If a record is >10KB, it is almost certainly a tool result, not reasoning
+
+2. **Tool invocations** -- What commands were run. Operational, not knowledge. Look for:
+   - Records with `type: "tool_use"`, `type: "function_call"`, or `type: "tool_call"`
+   - Fields named `name`, `arguments`, `input`, `command`
+   - Exception: file edit commands may contain the *what* of a change (summarize those)
+
+3. **System prompts and instructions** -- Same across sessions. Look for:
+   - Records with `role: "system"` or `role: "developer"`
+   - Content wrapped in XML tags (`<instructions>`, `<context>`, `<rules>`)
+   - Long preambles about model behavior, tool availability, permissions
+
+4. **Metadata and telemetry** -- Session infrastructure. Look for:
+   - Token counts, usage statistics, rate limits
+   - Timestamps, UUIDs, session IDs (useful for linking but not knowledge)
+   - Permission changes, mode switches, checkpoint markers
+
+5. **Base64-encoded data** -- Images, files, binary content. Look for:
+   - Long strings matching `[A-Za-z0-9+/=]{1000,}` or data URIs
+   - Fields named `image`, `data`, `base64`, `source.data`
+
+6. **Duplicate records** -- Many formats log the same event multiple ways. Look for:
+   - Records sharing an ID field (`call_id`, `tool_use_id`, `request_id`)
+   - The same text appearing in both a streaming record and a final record
+
+### Summarize (compress)
+1. **Sequences of tool calls** -- "Read 15 files in src/lib/" is better than 15 individual read records
+2. **Repetitive status updates** -- "Still searching..." x10 -> "Searched extensively"
+3. **Build/test output** -- "Tests: 58/58 passed" not the full test runner output
+4. **File edit details** -- "Modified auth.ts: added token validation" not the full diff
+
+## Step 3: Estimate signal ratio
+
+Before processing the full file, sample it:
+
+1. Take the first 20 records, middle 20, and last 20
+2. Classify each as signal / noise / summarize
+3. Measure bytes in each category
+
+Typical ratios from known formats:
+- **Claude Code:** 3-15% signal, 85-97% noise (tool results + wrapper overhead dominate)
+- **Codex:** 10-19% signal, 50-70% noise, 20-30% ambiguous (reasoning encrypted, compacted records)
+- **Raw API logs:** 30-50% signal (no tool overhead, just conversation)
+- **Chat exports (markdown):** 60-80% signal (already cleaned by the export process)
+
+## Step 4: Extract
+
+For each signal record, extract:
+- **Who said it:** human or AI
+- **What they said:** the text content
+- **When:** timestamp if available
+- **Context:** what came before (the preceding human message gives context to an AI response)
+
+Structure the output as a sequence of turns:
+```
+[timestamp] HUMAN: <message>
+[timestamp] AI: <response>
+[timestamp] HUMAN: <follow-up>
+[timestamp] AI: <response>
+```
+
+## Step 5: Handle unknowns gracefully
+
+If you cannot classify a record:
+- If it is <1KB, include it (small records are cheap to carry)
+- If it is >10KB, skip it (large unclassified records are almost always tool output)
+- If it contains natural language prose, include it
+- If it contains code, JSON, or structured data, skip it
+
+## Privacy checks
+
+Before outputting extracted content, scan for:
+- API keys: patterns like `sk-`, `ghp_`, `Bearer `, `token: "`
+- Passwords: fields named `password`, `passwd`, `secret`
+- Personal data: email addresses, IP addresses, phone numbers
+- File paths: may reveal usernames (e.g., `/Users/johndoe/`)
+- JWT tokens: strings matching `eyJ...`
+
+Flag these but do not include them in extracted output.