novahiz 0.2.2 → 0.2.5

package/bundled-skills/copilot-history-ingest/SKILL.md

@@ -1,376 +0,0 @@
----
-name: copilot-history-ingest
-description: >
-  Ingest GitHub Copilot CLI session history into an Obsidian wiki as distilled knowledge pages. Use this skill
-  when the user wants to capture their Copilot CLI sessions into a personal wiki — extracting architecture
-  decisions, debug notes, and patterns into searchable Obsidian pages. Triggers on phrases like "ingest my
-  copilot sessions into obsidian", "add my copilot history to my wiki", "pull my copilot session history into
-  the vault", "capture what I've learned from copilot into obsidian", "just the new sessions since last time",
-  or "mine patterns across my copilot sessions". Also triggers when the user mentions session-store.db,
-  ~/.copilot/session-state, or VS Code copilot-chat transcripts in the context of building a wiki or knowledge
-  base. Does NOT trigger for general copilot usage questions, searching sessions, or backing up history.
----
-
-# Copilot History Ingest — Conversation Mining
-
-You are extracting knowledge from the user's past GitHub Copilot CLI conversations and distilling it into the Obsidian wiki. Conversations are rich but messy — your job is to find the signal and compile it.
-
-This skill can be invoked directly or via the `wiki-history-ingest` router (`/wiki-history-ingest copilot`).
-
-## Before You Start
-
-**Writing profile:** Before drafting or rewriting natural-language Markdown, read and apply the `Writing Profile Resolution` section in `llm-wiki/SKILL.md`. Framework schema, provenance, safety, and operation-specific requirements take precedence.
-`WRITING.md` preferences apply only to newly drafted or rewritten natural-language Markdown; preserve source content and structured records.
-
-1. **Resolve config** — follow the Config Resolution Protocol in `llm-wiki/SKILL.md` (inline `@name` override → walk up CWD for `.env` → global config → prompt setup). This gives `OBSIDIAN_VAULT_PATH`, `COPILOT_HISTORY_PATH` (defaults to `~/.copilot/session-state`), and `COPILOT_VSCODE_STORAGE_PATH` (VS Code `workspaceStorage`; platform-specific — ask the user if absent)
-2. Read `.manifest.json` at the vault root to check what's already been ingested
-3. Read `index.md` at the vault root to know what the wiki already contains
-
-## Ingest Modes
-
-### Append Mode (default)
-
-Check `.manifest.json` for each source file (events JSONL, transcript JSONL, checkpoint, session-store DB). Only process:
-
-- Sessions not in the manifest (new sessions)
-- Sessions whose `updated_at` is newer than their `ingested_at` in the manifest
-
-This is usually what you want — the user ran a few new sessions and wants to capture the delta.
-
-### Full Mode
-
-Process everything regardless of manifest. Use after a `wiki-rebuild` or if the user explicitly asks.
-
-## GitHub Copilot Data Layout
-
-Copilot stores data in three locations. Scan **all three**.
-
-### Source 1: `~/.copilot/session-state/` (CLI sessions)
-
-```
-~/.copilot/session-state/
-├── <session-uuid>/
-│   ├── workspace.yaml           # Session metadata (id, cwd, summary_count, created_at, updated_at)
-│   ├── vscode.metadata.json     # VS Code context (workspaceFolder, repositoryProperties, customTitle)
-│   ├── events.jsonl             # Full event log — all turns, tool calls, reasoning
-│   ├── session.db               # Per-session SQLite (todos/todo_deps only — skip for ingestion)
-│   ├── index.md                 # Session summary written at session end
-│   ├── checkpoints/             # Checkpoint JSON files (mid-session summaries)
-│   │   └── <uuid>.json          # title, overview, history, work_done, technical_details,
-│   │                            #   important_files, next_steps
-│   ├── files/                   # Artifacts produced during session (plans, diagrams, etc.)
-│   └── research/                # Research artifacts
-└── ...
-```
-
-### Source 2: `~/.copilot/session-store.db` (Global SQLite)
-
-The canonical cross-session database. This is the **highest-value** source: structured, queryable, and pre-summarised.
-
-```
-sessions       — id, cwd, repository, branch, summary, created_at, updated_at, host_type
-turns          — session_id, turn_index, user_message, assistant_response, timestamp
-checkpoints    — session_id, checkpoint_number, title, overview, history, work_done,
-                 technical_details, important_files, next_steps, created_at
-session_files  — session_id, file_path, tool_name, turn_index, first_seen_at
-session_refs   — session_id, ref_type (commit/pr/issue), ref_value, turn_index, created_at
-search_index   — FTS5 virtual table (content, session_id, source_type, source_id)
-```
-
-### Source 3: VS Code Workspace Storage (`<workspaceStorage>/<hash>/GitHub.copilot-chat/`)
-
-VS Code extension data, keyed by workspace hash. The path is platform-specific and must come from `.env` or user input.
-
-```
-<hash>/GitHub.copilot-chat/
-├── transcripts/
-│   └── <session-uuid>.jsonl     # Conversation transcripts (same JSONL format as events.jsonl)
-├── memory-tool/
-│   └── memories/
-│       └── <base64-session-id>/ # Per-session saved artifacts (plan.md, etc.)
-│           └── plan.md
-└── codebase-external.sqlite     # Codebase index (skip — no conversation knowledge)
-```
-
-### Key data sources ranked by value:
-
-1. **Checkpoints** (`session-store.db` `checkpoints` table + per-session `checkpoints/*.json`) — Pre-distilled summaries with `overview`, `work_done`, `technical_details`, `important_files`, `next_steps`. Gold.
-2. **Session summaries** (`session-store.db` `sessions.summary` + `index.md`) — One-paragraph synopsis per session.
-3. **Turns** (`session-store.db` `turns` table + `events.jsonl` / transcript JSONL) — Full conversation. Rich but verbose.
-4. **Memory artifacts** (`memory-tool/memories/<id>/plan.md` etc.) — Pre-written plans and structured notes the user saved explicitly. Worth importing verbatim (or lightly summarised).
-5. **File access patterns** (`session_files` table + `tool.execution_*` events) — Which files the agent repeatedly touched — reveals high-value project files.
-6. **Session refs** (`session_refs` table) — Commits, PRs, and issues linked to sessions.
-7. **`vscode.metadata.json`** — Workspace folder path, branch, `customTitle` (user-set session label). Useful for grouping and naming.
-
-## Step 1: Survey and Compute Delta
-
-Scan all three data locations and compare against `.manifest.json`:
-
-```bash
-# --- Source 1: per-session directories ---
-# Find all session directories (each has workspace.yaml)
-ls ~/.copilot/session-state/
-
-# For each session, read workspace.yaml for id/cwd/updated_at
-# and vscode.metadata.json for customTitle / repositoryProperties
-
-# --- Source 2: global database ---
-# Query session-store.db with sqlite3 (or Python sqlite3)
-SELECT s.id, s.cwd, s.repository, s.branch, s.summary, s.updated_at,
-       COUNT(DISTINCT t.turn_index) AS turn_count,
-       COUNT(DISTINCT c.id)         AS checkpoint_count
-FROM sessions s
-LEFT JOIN turns t ON t.session_id = s.id
-LEFT JOIN checkpoints c ON c.session_id = s.id
-GROUP BY s.id
-ORDER BY s.updated_at DESC;
-
-# --- Source 3: VS Code workspace storage ---
-# For each <hash> directory under workspaceStorage, check for GitHub.copilot-chat/
-# Find transcript files
-ls <workspaceStorage>/<hash>/GitHub.copilot-chat/transcripts/
-```
-
-Build a unified inventory — one entry per session UUID — and classify:
-
-- **New** — not in manifest → needs ingesting
-- **Modified** — in manifest but `updated_at` is newer → needs re-ingesting
-- **Unchanged** — in manifest and not modified → skip in append mode
-
-Report to the user: "Found X sessions in session-state, Y in session-store.db, Z VS Code transcript files. Checkpoints: A. Delta: B new, C modified."
-
-## Step 2: Ingest Checkpoints and Summaries First
-
-Checkpoints are already distilled — process them before touching raw turns.
-
-### From `session-store.db`:
-
-```sql
-SELECT s.id, s.cwd, s.repository, s.branch, s.summary,
-       c.checkpoint_number, c.title, c.overview, c.work_done,
-       c.technical_details, c.important_files, c.next_steps,
-       c.created_at
-FROM checkpoints c
-JOIN sessions s ON c.session_id = s.id
-ORDER BY s.updated_at DESC, c.checkpoint_number ASC;
-```
-
-### From per-session `checkpoints/*.json`:
-
-Each checkpoint file has: `title`, `overview`, `history`, `work_done`, `technical_details`, `important_files`, `next_steps`.
-
-Read `index.md` (if present) as a session-level summary — it's typically written at session end and is already concise.
-
-### What to extract:
-
-- `overview` → high-level description of what the session accomplished
-- `work_done` → concrete tasks completed (good for skills / project pages)
-- `technical_details` → implementation specifics (good for concepts pages)
-- `important_files` → high-value files in the project (good for project pages)
-- `next_steps` → open threads (good for linking to ongoing project work)
-
-## Step 3: Parse Session Turns
-
-Read turns from `session-store.db` (preferred — already parsed) or from `events.jsonl` / transcript JSONL.
-
-### From `session-store.db`:
-
-```sql
-SELECT turn_index, user_message, assistant_response, timestamp
-FROM turns
-WHERE session_id = '<uuid>'
-ORDER BY turn_index ASC;
-```
-
-### From `events.jsonl` / transcript JSONL:
-
-Each file is one session. Each line is a JSON event. See `references/copilot-data-format.md` for the full schema.
-
-**Relevant event types:**
-
-| `type`                | What it is                              | Worth reading?                            |
-| --------------------- | --------------------------------------- | ----------------------------------------- |
-| `session.start`       | Session metadata (cwd, branch, version) | Yes — establishes project context         |
-| `user.message`        | User turn                               | Yes — `data.content`                      |
-| `assistant.message`   | Assistant turn                          | Yes — `data.content` (text) + `data.toolRequests` |
-| `tool.execution_start`| Tool call                               | Skim — reveals what files/commands were used |
-| `tool.execution_end`  | Tool result                             | No — usually noise                        |
-
-**Extraction strategy for `assistant.message`:**
-
-- `data.content` is the assistant's text response — extract this
-- `data.reasoningText` is internal reasoning — skip (it's the unpacked `reasoningOpaque` field)
-- `data.toolRequests` lists tool calls — skim tool names and arguments for file access patterns
-- Skip `type: "tool.execution_end"` entirely
-
-## Step 3b: Process Memory Artifacts
-
-For each session that has a `memory-tool/memories/<base64-id>/` directory in VS Code workspace storage, read any markdown files saved there (typically `plan.md`). These are documents the user explicitly saved — treat them as high-quality, user-authored content.
-
-Decode the base64 directory name to get the session UUID:
-
-```python
-import base64
-session_id = base64.b64decode(dir_name).decode('utf-8')
-```
-
-Memory artifacts map to project `skills/` or `concepts/` pages, depending on content type.
-
-## Step 3c: Extract File and Ref Patterns
-
-From `session-store.db`:
-
-```sql
--- Most-touched files per project
-SELECT repository, file_path, COUNT(*) AS touch_count
-FROM session_files
-GROUP BY repository, file_path
-ORDER BY touch_count DESC;
-
--- Linked commits/PRs/issues per session
-SELECT session_id, ref_type, ref_value, turn_index
-FROM session_refs
-ORDER BY session_id, turn_index;
-```
-
-**File access patterns** reveal which files are architecturally important — note them on project pages.
-
-**Session refs** link Copilot sessions to git history — useful for connecting wiki knowledge to concrete code changes.
-
-## Step 4: Cluster by Topic
-
-Don't create one wiki page per session. Instead:
-
-- Group extracted knowledge **by topic** across sessions
-- A single session about "debugging auth + setting up CI" → two separate topics
-- Three sessions across different days about "React performance" → one merged topic
-- `cwd` / `repository` give you a natural first-level grouping; `vscode.metadata.json`'s `customTitle` gives a human-readable session label
-
-## Step 5: Distill into Wiki Pages
-
-Each Copilot project maps to a project directory in the vault. Derive the project name from `cwd` or `repository`:
-
-```
-C:\Users\name\git\my-project   → my-project
-/Users/name/code/another-app   → another-app
-```
-
-Prefer `repository` (e.g., `owner/repo`) from `session-store.db` over raw `cwd` when available.
-
-### Project-specific vs. global knowledge
-
-| What you found                      | Where it goes               | Example                                              |
-| ----------------------------------- | --------------------------- | ---------------------------------------------------- |
-| Project architecture decisions      | `projects/<name>/concepts/` | `projects/my-project/concepts/main-architecture.md`  |
-| Project-specific debugging patterns | `projects/<name>/skills/`   | `projects/my-project/skills/api-rate-limiting.md`    |
-| General concept the user learned    | `concepts/` (global)        | `concepts/react-server-components.md`                |
-| Recurring problem across projects   | `skills/` (global)          | `skills/debugging-hydration-errors.md`               |
-| A tool/service used                 | `entities/` (global)        | `entities/vercel-functions.md`                       |
-| Patterns across many sessions       | `synthesis/` (global)       | `synthesis/common-debugging-patterns.md`             |
-
-For each project with content, create or update the project overview page at `projects/<name>/<name>.md` — **named after the project, not `_project.md`**. Obsidian's graph view uses the filename as the node label, so `_project.md` makes every project show up as `_project` in the graph. Naming it `<name>.md` gives each project a distinct, readable node name.
-
-**Important:** Distill the _knowledge_, not the conversation. Don't write "In a session on March 15, the user asked about X." Write the knowledge itself, with the session as a source attribution.
-
-**Write a `summary:` frontmatter field** on every new/updated page — 1–2 sentences, ≤200 chars, answering "what is this page about?" for a reader who hasn't opened it. `wiki-query`'s cheap retrieval path reads this field to avoid opening page bodies.
-
-**Add confidence and lifecycle fields** to every new page's frontmatter:
-```yaml
-base_confidence: 0.42
-lifecycle: draft
-lifecycle_changed: <ISO date today>
-```
-Leave `lifecycle` unchanged on update.
-
-**Mark provenance** per the convention in `llm-wiki` (Provenance Markers section):
-
-- **Checkpoints and index.md** are pre-distilled by the system — treat checkpoint-derived claims as extracted (the system wrote them from observed actions).
-- **Memory artifacts** are user-authored — treat as extracted.
-- **Conversation turn distillation** is mostly inferred. You're synthesizing a coherent claim from many turns. Apply `^[inferred]` liberally to synthesized patterns, generalizations across sessions, and "what the user really meant" interpretations.
-- Use `^[ambiguous]` when the user changed direction mid-session or when the session ended unresolved.
-- Write a `provenance:` frontmatter block on every new/updated page summarizing the rough mix.
-
-## Step 6: Update Manifest, Journal, and Special Files
-
-### Update `.manifest.json`
-
-For each session processed, add/update its entry with:
-
-- `ingested_at`, `session_id`, `updated_at`
-- `source_type`: one of `"copilot_session"`, `"copilot_checkpoint"`, `"copilot_transcript"`, `"copilot_memory_artifact"`
-- `project`: the decoded project name
-- `pages_created` and `pages_updated` lists
-
-Also update the `projects` section of the manifest:
-
-```json
-{
-  "project-name": {
-    "repository": "owner/repo",
-    "cwd": "C:\\Users\\name\\git\\project-name",
-    "vault_path": "projects/project-name",
-    "last_ingested": "TIMESTAMP",
-    "sessions_ingested": 5,
-    "sessions_total": 8,
-    "checkpoints_ingested": 12,
-    "memory_artifacts_ingested": 3
-  }
-}
-```
-
-### Create journal entry + update special files
-
-Update `index.md` and `log.md` per the standard process:
-
-```
-- [TIMESTAMP] COPILOT_HISTORY_INGEST projects=N sessions=M checkpoints=C pages_updated=X pages_created=Y mode=append|full
-```
-
-**`hot.md`** — Read `$OBSIDIAN_VAULT_PATH/hot.md` (create from the template in `wiki-ingest` if missing). Update **Recent Activity** with a one-line summary — e.g. "Ingested 5 Copilot sessions across 2 projects; surfaced patterns in API design and testing strategy." Keep the last 3 operations. Update **Active Threads** if any ongoing project is now better understood. Update `updated` timestamp.
-
-## Privacy
-
-- Distill and synthesize — don't copy raw conversation text verbatim
-- Skip anything that looks like secrets, API keys, passwords, tokens
-- `data.reasoningOpaque` / `data.reasoningText` in assistant events is internal reasoning — skip entirely, never copy to wiki
-- If you encounter personal/sensitive content, ask the user before including it
-- The user's conversations may reference other people — be thoughtful about what goes in the wiki
-
-## Reference
-
-See `references/copilot-data-format.md` for detailed data structure documentation.
-
-## QMD Refresh After Vault Writes
-
-QMD is a search index, not the source of truth. If `$QMD_WIKI_COLLECTION` is empty or unset, skip this step. Run it only after this skill has written or rewritten vault markdown. If QMD refresh fails, do not roll back the vault changes; report the QMD status separately.
-
-Use `$QMD_CLI` if set; otherwise use `qmd`.
-
-```bash
-${QMD_CLI:-qmd} update
-```
-
-If the output says vectors are needed or embeddings may be stale, run:
-
-```bash
-${QMD_CLI:-qmd} embed
-```
-
-Verify the collection with either:
-
-```bash
-${QMD_CLI:-qmd} ls "$QMD_WIKI_COLLECTION"
-```
-
-or, when a specific page path is known:
-
-```bash
-${QMD_CLI:-qmd} get "qmd://$QMD_WIKI_COLLECTION/<page>.md" -l 5
-```
-
-Record one of:
-- `QMD refreshed: update + embed + verified`
-- `QMD refreshed: update only + verified`
-- `QMD skipped: QMD_WIKI_COLLECTION unset`
-- `QMD skipped: qmd CLI unavailable`
-- `QMD failed: <short error summary>`