elliot-stack 1.0.17 → 1.0.19

package/skills/estack-read-claude-session-history/references/recipes.md

@@ -0,0 +1,237 @@
+# Recipes
+
+Multi-step workflows. For per-mode flag reference, see `modes.md`. For schema, see `jsonl-schema.md`.
+
+In all examples, `$PY` refers to:
+```
+C:\Users\2supe\.claude\skills\read-claude-session-history\scripts\read_transcript.py
+```
+
+---
+
+## 1. Post-`/compact` recovery
+
+When `/compact` has rolled the conversation and you need what fell off the back end:
+
+```bash
+# Step 1: Recover the long version of recent assistant output
+python "$PY" --file <current-session.jsonl> --mode pre-compact
+
+# Step 2: If advisor responses were involved, grab those separately
+python "$PY" --file <current-session.jsonl> --mode advisor
+
+# Step 3: If a search would be faster than reading the whole pre-compact section
+python "$PY" --file <current-session.jsonl> --mode search --query "<keyword>"
+```
+
+The `pre-compact` window is 40 message-exchanges before the most recent compact. If multiple `/compact` events fired, only the most recent is used as the anchor.
+
+---
+
+## 2. Find-then-dump (resume work from a session you can't name)
+
+```bash
+# Step 1: Find by partial title or first prompt
+python "$PY" --mode find --title "supabase"
+#   → returns a list, including the full session UUIDs
+
+# Step 2: Get a 6-line summary to confirm it's the right one
+python "$PY" --file <picked-session>.jsonl --mode brief
+
+# Step 3: Dump the recent context to ground yourself
+python "$PY" --file <picked-session>.jsonl --mode dump -n 20
+
+# Step 4: Get the resume command for that session
+python "$PY" --mode resume-cmd --uuid <8-char-prefix>
+```
+
+---
+
+## 3. Fan-out triage (you spawned N parallel subagents and want every output)
+
+```bash
+# Option A: Get all subagent finals in one shot, separated by headers
+python "$PY" --file <parent-session>.jsonl --mode subagent-finals
+
+# Option B: Triage with a brief that includes subagent finals folded in
+python "$PY" --file <parent-session>.jsonl --mode brief --include-subagents
+
+# Option C: List first to see what types of agents ran
+python "$PY" --file <parent-session>.jsonl --mode subagent-list
+
+# Drill into one specific subagent's tools / files
+python "$PY" --mode subagent-tools --subagent <subagent-path>
+python "$PY" --mode subagent-files --subagent <subagent-path>
+```
+
+The `brief --include-subagents` output is the densest form of the standard "what did all my agents do" question and was designed for fan-out reproduction (14 parallel briefs ≡ a 14-subagent investigation).
+
+---
+
+## 4. Deletion-incident recovery (March 2026 auto-update bug, GitHub #41591)
+
+When a live `.jsonl` has been wiped but the backup is intact:
+
+```bash
+# Step 1: Find what's missing — list live vs snapshot side by side
+python "$PY" --root live        --cwd "C:\Users\2supe\Other Claude Code" --list > live.txt
+python "$PY" --root snapshot-24h --cwd "C:\Users\2supe\Other Claude Code" --list > snap.txt
+diff live.txt snap.txt
+
+# Step 2: For each missing UUID, locate it in the snapshot
+python "$PY" --root snapshot-24h --mode lookup --uuid <prefix>
+
+# Step 3: Read it directly from the snapshot path
+python "$PY" --file <snapshot-path>.jsonl --mode brief
+python "$PY" --file <snapshot-path>.jsonl --mode dump
+
+# Step 4: If the 24h snapshot is also affected, walk back further
+python "$PY" --root snapshot-1w  --mode lookup --uuid <prefix>
+python "$PY" --root snapshot-1mo --mode lookup --uuid <prefix>
+```
+
+The four backup roots (`mirror`, `snapshot-24h`, `snapshot-1w`, `snapshot-1mo`) are managed by the daily backup task documented in `reference_claude_backup_system.md`.
+
+---
+
+## 5. Week-in-review journal
+
+```bash
+# Every session in every project from the last 7 days
+python "$PY" --all-projects --mode journal --since 7d
+
+# Single project, with a hard upper bound
+python "$PY" --cwd "C:\Users\2supe\Other Claude Code" \
+  --mode journal --since 2026-05-13 --until 2026-05-20
+
+# By project name instead of path
+python "$PY" --project keel --mode journal --since 7d
+
+# Count how many sessions touched a topic
+python "$PY" --all-projects --mode count --query "linkedin"
+```
+
+The output is one 5-line block per session: `date·uuid·project` / first prompt / last assistant message / N files edited / top tools.
+
+---
+
+## 5b. Day accounting ("where did my day go?")
+
+```bash
+# Block-grouped timeline of yesterday across ALL projects, with idle gaps
+python "$PY" --mode timeline --date yesterday
+
+# How much time on one project today
+python "$PY" --mode timeline --project keel --date today
+
+# Tighter idle threshold (treat >5m quiet as a break between blocks)
+python "$PY" --mode timeline --date today --gap 5m
+
+# Multi-day window
+python "$PY" --mode timeline --since 2026-06-01 --until 2026-06-03
+```
+
+Reading the output: each block is a contiguous stretch of activity (events ≤ gap
+apart); the sessions inside it are listed with message counts; `── idle Xm ──`
+lines mark the breaks; the totals line gives summed active time vs. span.
+
+Caveat: this measures *Claude-visible* activity only — message timestamps, not
+attention or time spent away from Claude.
+
+---
+
+## 5c. Piping structured output into a next step
+
+Every mode supports `--format json`. **Run pipe chains in Bash** (the Bash tool /
+git-bash) — they work exactly as written:
+
+```bash
+# Pull the paths of yesterday's sessions for batch processing
+python "$PY" --mode list --all-projects --since yesterday --format json \
+  | python -c "import json,sys; [print(s['path']) for s in json.load(sys.stdin)]"
+
+# Machine-readable day totals
+python "$PY" --mode timeline --date yesterday --format json \
+  | python -c "import json,sys; t=json.load(sys.stdin)['totals']; print(t['active_minutes'], 'min across', t['sessions'], 'sessions')"
+```
+
+PowerShell warnings (5.1):
+- Piping between native commands injects a UTF-8 BOM and re-encodes through the
+  console codepage (can corrupt non-ASCII transcript content). If you must pipe
+  in PowerShell, read stdin as `utf-8-sig`:
+  `python -c "import io,json,sys; data=json.load(io.TextIOWrapper(sys.stdin.buffer, encoding='utf-8-sig'))"`
+- `>` redirection writes UTF-16 — read redirected files with `encoding='utf-16'`.
+
+Prefer Bash for any JSON chaining; prefer either shell for plain single commands.
+
+---
+
+## 6. Sibling-agent diff
+
+When you ran two subagents on the same task and want to see where they diverged:
+
+```bash
+# Auto-pick the first two subagents of a session
+python "$PY" --mode diff --subagents-of <parent-session>.jsonl
+
+# Explicit pairing
+python "$PY" --mode diff \
+  --file-a <parent-uuid>/subagents/agent-aaa.jsonl \
+  --file-b <parent-uuid>/subagents/agent-bbb.jsonl
+```
+
+Output is timestamp-interleaved, prefixed `A>` / `B>`. Use it to spot disagreement (e.g. one agent recommended X and the other recommended Y) without reading both transcripts in full.
+
+---
+
+## 7. "What is this session actually for?" (cold open on a stale UUID)
+
+```bash
+# Single-shot orientation in under a second
+python "$PY" --file <unknown-session>.jsonl --mode brief
+
+# If you need more than 200 chars of context per line
+python "$PY" --file <unknown-session>.jsonl --mode last -n 3
+python "$PY" --file <unknown-session>.jsonl --mode changelog | tail -30
+```
+
+`brief` is the recommended default for triaging a session you've never read.
+
+---
+
+## 8. Tool-call forensics ("when did I last `git push --force`?")
+
+```bash
+# Find every tool_use whose JSON args contain a substring, across all sessions
+python "$PY" --all-projects --mode search --query "git push --force" --in tool_use
+
+# Get full forensics on the session that matched
+python "$PY" --file <matching-session>.jsonl --mode tool-calls --tool Bash
+```
+
+---
+
+## 9. Resume previous session in the current project
+
+If you just `cd`'d into a project and want to pick up where you left off:
+
+```bash
+python "$PY" --cwd "$(pwd)" --mode resume-prev -n 15
+```
+
+Prints `--- Resuming from <uuid> (<mtime>) ---` then the last 15 exchanges.
+
+---
+
+## 10. Schema drift / silent empty results
+
+When a mode returns nothing and you don't know why:
+
+```bash
+python "$PY" --file <session>.jsonl --mode debug
+```
+
+Look for:
+- Unfamiliar `type:` values appearing in the distribution (parser might be dropping them as noise).
+- An absent `advisor_tool_result` block when you expected advisor output.
+- Missing `compact` markers in a session you know got compacted.

package/skills/estack-read-claude-session-history/scripts/lib/__init__.py

@@ -0,0 +1 @@
+"""Library modules for read_transcript."""

package/skills/estack-read-claude-session-history/scripts/lib/parser.py

@@ -0,0 +1,460 @@
+"""JSONL parsing primitives, message classification, and session summaries."""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Iterator, Literal
+
+
+NOISE_TYPES: set[str] = {
+    "permission-mode", "ai-title", "custom-title", "attachment",
+    "last-prompt", "queue-operation", "file-history-snapshot",
+    "system", "agent-name", "pr-link",
+}
+
+COMPACT_MARKER = "This session is being continued from a previous conversation"
+
+# 5 MB — beyond this, dump mode auto-degrades unless --force-dump.
+LARGE_FILE_THRESHOLD = 5 * 1024 * 1024
+
+EntryType = Literal["user", "assistant", "title", "noise", "compact"]
+
+_PARSE_CACHE: dict[Path, tuple[float, list[dict]]] = {}
+
+
+def iter_lines(path: Path) -> Iterator[dict]:
+    """Yield parsed JSON objects from a .jsonl file, streaming.
+
+    A truncated (un-newline-terminated) trailing line is dropped silently with
+    a stderr note. Malformed JSON lines are also dropped silently.
+    """
+    truncated = False
+    try:
+        with open(path, encoding="utf-8") as f:
+            for line in f:
+                stripped = line.strip()
+                if not stripped:
+                    continue
+                if not line.endswith("\n"):
+                    # Last line, no terminator — could be partial. Try to parse,
+                    # but if it fails, treat as truncation.
+                    try:
+                        yield json.loads(stripped)
+                    except json.JSONDecodeError:
+                        truncated = True
+                    continue
+                try:
+                    yield json.loads(stripped)
+                except json.JSONDecodeError:
+                    continue
+    finally:
+        if truncated:
+            print(
+                f"[note: dropped truncated trailing line in {path.name}]",
+                file=sys.stderr,
+            )
+
+
+def parse_lines(path: Path) -> list[dict]:
+    """Read all JSONL records from a file, with mtime-based caching."""
+    try:
+        mtime = path.stat().st_mtime
+    except OSError:
+        return list(iter_lines(path))
+    cached = _PARSE_CACHE.get(path)
+    if cached is not None and cached[0] == mtime:
+        return cached[1]
+    records = list(iter_lines(path))
+    _PARSE_CACHE[path] = (mtime, records)
+    return records
+
+
+def extract_text_blocks(
+    content,
+    include_thinking: bool = False,
+    include_tool_use: bool = False,
+) -> list[str]:
+    """Pull human-readable text from a content field (string or block list)."""
+    if isinstance(content, str):
+        return [content] if content.strip() else []
+    if not isinstance(content, list):
+        return []
+    texts: list[str] = []
+    for block in content:
+        if not isinstance(block, dict):
+            continue
+        t = block.get("type")
+        if t == "text" and block.get("text", "").strip():
+            texts.append(block["text"])
+        elif t == "advisor_tool_result":
+            inner = block.get("content", {})
+            if isinstance(inner, dict) and inner.get("text"):
+                texts.append(f"[ADVISOR]\n{inner['text']}")
+        elif t == "thinking" and include_thinking:
+            think = block.get("thinking", "") or block.get("text", "")
+            if think.strip():
+                texts.append(f"[THINKING]\n{think}")
+        elif t == "tool_use" and include_tool_use:
+            name = block.get("name", "?")
+            tool_input = block.get("input", {})
+            try:
+                preview = json.dumps(tool_input)[:200]
+            except (TypeError, ValueError):
+                preview = str(tool_input)[:200]
+            texts.append(f"[TOOL_USE {name}] {preview}")
+    return texts
+
+
+def is_compact_marker(text: str) -> bool:
+    return bool(text) and COMPACT_MARKER in text
+
+
+def classify_entry(obj: dict) -> EntryType:
+    """Single source of truth for entry-type classification."""
+    t = obj.get("type", "")
+    if t == "ai-title" or t == "custom-title":
+        return "title"
+    if t in NOISE_TYPES:
+        return "noise"
+    msg = obj.get("message", {})
+    if not msg:
+        return "noise"
+    role = msg.get("role")
+    if role == "user":
+        content = msg.get("content", "")
+        text = (
+            content if isinstance(content, str)
+            else " ".join(
+                b.get("text", "") for b in content
+                if isinstance(b, dict) and b.get("type") == "text"
+            )
+        )
+        if is_compact_marker(text):
+            return "compact"
+        return "user"
+    if role == "assistant":
+        return "assistant"
+    return "noise"
+
+
+def get_messages(lines: list[dict]) -> list[dict]:
+    """Filter to signal messages, returning {role, texts, line_index, is_compact, timestamp}."""
+    messages: list[dict] = []
+    for i, obj in enumerate(lines):
+        cls = classify_entry(obj)
+        if cls in ("noise", "title"):
+            continue
+        msg = obj.get("message", {})
+        if not msg:
+            continue
+        content = msg.get("content", "")
+        texts = extract_text_blocks(content)
+        timestamp = obj.get("timestamp")
+        messages.append({
+            "role": "user" if cls in ("user", "compact") else "assistant",
+            "texts": texts,
+            "line_index": i,
+            "is_compact": cls == "compact",
+            "timestamp": timestamp,
+        })
+    return messages
+
+
+def filter_by_role(
+    messages: list[dict], role: Literal["user", "assistant", "both"]
+) -> list[dict]:
+    if role == "both":
+        return messages
+    return [m for m in messages if m["role"] == role]
+
+
+# Display timezone. None → system local time. Set via set_timezone() (--tz flag).
+# JSONL timestamps are UTC; every parsed timestamp is converted to this zone so
+# all displayed times match the user's wall clock and compare cleanly against
+# parse_timespec() values (which are local).
+_TARGET_TZ: timezone | None = None
+
+_TZ_OFFSET_RE = re.compile(r"^([+-])(\d{1,2})(?::?(\d{2}))?$")
+
+
+def set_timezone(spec: str | None) -> None:
+    """Set the display timezone from a --tz spec.
+
+    Accepts:
+      - None / "local"  → system local time (default)
+      - "UTC"           → UTC
+      - fixed offsets   → "+5", "-4", "+05:30", "UTC-4"
+      - IANA names      → "America/New_York" (via zoneinfo)
+    """
+    global _TARGET_TZ
+    if not spec or spec.strip().lower() == "local":
+        _TARGET_TZ = None
+        return
+    s = spec.strip()
+    if s.upper().startswith("UTC"):
+        rest = s[3:].strip()
+        if not rest:
+            _TARGET_TZ = timezone.utc
+            return
+        s = rest  # "UTC-4" → "-4"
+    m = _TZ_OFFSET_RE.match(s)
+    if m:
+        sign = 1 if m.group(1) == "+" else -1
+        hours = int(m.group(2))
+        mins = int(m.group(3) or 0)
+        _TARGET_TZ = timezone(sign * timedelta(hours=hours, minutes=mins))
+        return
+    try:
+        from zoneinfo import ZoneInfo
+        _TARGET_TZ = ZoneInfo(spec.strip())
+    except Exception as e:
+        raise ValueError(
+            f"Unrecognized timezone: {spec!r}. "
+            "Use an IANA name (America/New_York), 'UTC', or an offset (+5, -4, +05:30)."
+        ) from e
+
+
+def to_display(dt: datetime) -> datetime:
+    """Convert an aware datetime to the display timezone, returned naive."""
+    return dt.astimezone(_TARGET_TZ).replace(tzinfo=None)
+
+
+def epoch_to_display(epoch: float) -> datetime:
+    """Convert an epoch (e.g. st_mtime) to the display timezone, returned naive."""
+    return to_display(datetime.fromtimestamp(epoch, tz=timezone.utc))
+
+
+def display_to_epoch(dt: datetime) -> float:
+    """Interpret a naive display-timezone datetime as an epoch.
+
+    Inverse of epoch_to_display. Needed because naive_dt.timestamp() assumes
+    *local* time, which is wrong under a --tz override.
+    """
+    if dt.tzinfo is None and _TARGET_TZ is not None:
+        dt = dt.replace(tzinfo=_TARGET_TZ)
+    return dt.timestamp()
+
+
+def now_display() -> datetime:
+    """Current time as a naive datetime in the display timezone."""
+    import time as _time
+    return epoch_to_display(_time.time())
+
+
+def _parse_timestamp(ts) -> datetime | None:
+    """Parse a JSONL timestamp → naive datetime in the display timezone."""
+    if not ts:
+        return None
+    if isinstance(ts, (int, float)):
+        try:
+            return epoch_to_display(float(ts))
+        except (ValueError, OSError, OverflowError):
+            return None
+    if isinstance(ts, str):
+        # ISO 8601 with possible Z
+        s = ts.replace("Z", "+00:00")
+        try:
+            dt = datetime.fromisoformat(s)
+        except ValueError:
+            return None
+        if dt.tzinfo is not None:
+            return to_display(dt)
+        return dt  # naive — assume already local
+    return None
+
+
+def filter_by_time(
+    messages: list[dict],
+    since: datetime | None,
+    until: datetime | None,
+) -> list[dict]:
+    if since is None and until is None:
+        return messages
+    out = []
+    for m in messages:
+        ts = _parse_timestamp(m.get("timestamp"))
+        if ts is None:
+            continue
+        # Strip tzinfo for naive comparison
+        if ts.tzinfo is not None:
+            ts = ts.replace(tzinfo=None)
+        if since is not None and ts < since:
+            continue
+        if until is not None and ts > until:
+            continue
+        out.append(m)
+    return out
+
+
+def _truncate(s: str, n: int) -> str:
+    if not s:
+        return ""
+    s = s.replace("\n", " ").strip()
+    return s if len(s) <= n else s[: n - 1] + "…"
+
+
+def infer_status(
+    lines: list[dict],
+    mtime: float,
+    current_session_id: str | None,
+    session_uuid: str | None,
+) -> Literal["clean", "interrupted", "pending-user", "active"]:
+    """Heuristic session status from the shape of the final entry."""
+    now = datetime.now().timestamp()
+    if (
+        current_session_id
+        and session_uuid
+        and current_session_id == session_uuid
+        and now - mtime < 300
+    ):
+        return "active"
+
+    if not lines:
+        return "clean"
+
+    # Walk backwards through non-noise entries
+    last_assistant = None
+    has_dangling_tool_use = False
+    pending_tool_use_ids: set[str] = set()
+    tool_result_ids: set[str] = set()
+    for obj in lines:
+        msg = obj.get("message", {})
+        if not isinstance(msg, dict):
+            continue
+        content = msg.get("content")
+        if not isinstance(content, list):
+            continue
+        for block in content:
+            if not isinstance(block, dict):
+                continue
+            bt = block.get("type")
+            if bt == "tool_use":
+                tid = block.get("id")
+                if tid:
+                    pending_tool_use_ids.add(tid)
+            elif bt == "tool_result":
+                tid = block.get("tool_use_id")
+                if tid:
+                    tool_result_ids.add(tid)
+
+    dangling = pending_tool_use_ids - tool_result_ids
+    if dangling:
+        has_dangling_tool_use = True
+
+    # Find the last assistant message
+    for obj in reversed(lines):
+        msg = obj.get("message", {})
+        if msg.get("role") == "assistant":
+            last_assistant = msg
+            break
+
+    if has_dangling_tool_use:
+        return "interrupted"
+
+    if last_assistant is not None:
+        content = last_assistant.get("content", "")
+        text = (
+            content if isinstance(content, str)
+            else " ".join(
+                b.get("text", "") for b in content
+                if isinstance(b, dict) and b.get("type") == "text"
+            )
+        )
+        if text.strip().endswith("?"):
+            return "pending-user"
+
+    return "clean"
+
+
+def session_summary(path: Path, current_session_id: str | None = None) -> dict:
+    """Compact per-session metrics for brief / list / journal / count modes."""
+    from .tools import extract_tool_calls, files_touched  # local import to avoid cycle
+    from .paths import decode_project_name, list_subagents
+    from .subagents import load_meta
+
+    try:
+        stat = path.stat()
+    except OSError:
+        return {
+            "path": path,
+            "uuid": path.stem,
+            "mtime": 0,
+            "size": 0,
+            "exists": False,
+        }
+
+    lines = parse_lines(path)
+    messages = get_messages(lines)
+    user_msgs = [m for m in messages if m["role"] == "user" and not m["is_compact"]]
+    assistant_msgs = [m for m in messages if m["role"] == "assistant"]
+
+    # Title
+    title = ""
+    for obj in lines:
+        if obj.get("type") in ("ai-title", "custom-title"):
+            title = obj.get("aiTitle") or obj.get("customTitle") or ""
+            if title:
+                break
+
+    first_prompt = ""
+    if user_msgs and user_msgs[0]["texts"]:
+        first_prompt = _truncate(user_msgs[0]["texts"][0], 200)
+
+    last_assistant = ""
+    if assistant_msgs and assistant_msgs[-1]["texts"]:
+        last_assistant = _truncate(assistant_msgs[-1]["texts"][-1], 200)
+
+    last_activity = epoch_to_display(stat.st_mtime).strftime("%Y-%m-%d %H:%M")
+
+    tool_calls = extract_tool_calls(lines)
+    tool_counts: dict[str, int] = {}
+    for tc in tool_calls:
+        tool_counts[tc["name"]] = tool_counts.get(tc["name"], 0) + 1
+
+    files = files_touched(lines)
+    edit_count = len(files)
+
+    subagents = list_subagents(path)
+    subagent_types: dict[str, int] = {}
+    for sa in subagents:
+        meta = load_meta(sa)
+        atype = meta.get("agentType", "unknown")
+        subagent_types[atype] = subagent_types.get(atype, 0) + 1
+
+    has_compact = any(m["is_compact"] for m in messages)
+    parent_dir_name = path.parent.name
+    decoded = decode_project_name(parent_dir_name)
+
+    status = infer_status(
+        lines, stat.st_mtime, current_session_id, path.stem
+    )
+
+    return {
+        "path": path,
+        "uuid": path.stem,
+        "mtime": stat.st_mtime,
+        "size": stat.st_size,
+        "exists": True,
+        "title": title,
+        "first_prompt": first_prompt,
+        "last_assistant": last_assistant,
+        "last_activity": last_activity,
+        "msg_count": len(messages),
+        "edit_count": edit_count,
+        "tool_counts": tool_counts,
+        "files_touched": list(files.keys()),
+        "subagent_count": len(subagents),
+        "subagent_types": subagent_types,
+        "has_compact": has_compact,
+        "has_subagents": bool(subagents),
+        "cwd": parent_dir_name,
+        "decoded_project": decoded,
+        "status": status,
+        "is_current": bool(
+            current_session_id and current_session_id == path.stem
+        ),
+    }