@aprimediet/codewalker 1.1.0 → 1.3.0

package/README.md

@@ -21,11 +21,25 @@ source files ──→ [ctags / regex] ──→ Symbol[]
                                                  index.db (SQLite + FTS5, disposable)
                                                       ↓
                                               codewalker_query (compact results)
+
+node_modules ──→ [.d.ts extract] ──→ LibSymbol[]
+                                          ↓
+                                     renderLibCard() → lib cards (version-pinned)
+                                                      ↓
+                                                 index.db (lib_symbols + FTS5)
+
+agent knowledge ──→ codewalker_enrich / codewalker_note
+                        ↓
+                   cards (enrichment, glossary, decisions)
+                        ↓
+                   index.db (notes + FTS5, unified query)
 ```
 
 - **Cards are the source of truth** — markdown in `~/.pi/projects/<id>/codewalker/entries/`.
 - **SQLite + FTS5 is a disposable index** — rebuildable from cards at any time.
 - **ctags primary, regex fallback** — ctags used when available, regex for TS/JS/Py/Go.
+- **Library layer** — extracts API surface from `node_modules` `.d.ts` files (version-pinned).
+- **Semantic + bridge layer** — agent-driven enrichment, glossary terms, and decision notes.
 - **Git-anchored** — stale index detected per query.
 
 ## Commands
@@ -35,10 +49,21 @@ source files ──→ [ctags / regex] ──→ Symbol[]
 | `/codewalker scan` | Full (re)build — walks project tree, extracts symbols, writes cards, populates DB |
 | `/codewalker sync` | Git-anchored incremental — reindexes only changed files |
 | `/codewalker query <text>` | Search the index (compact results) |
+| `/codewalker enrich <path>` | Select unenriched symbols under `<path>` and write summaries |
+| `/codewalker glossary [query]` | Search glossary terms |
+| `/codewalker decisions [query]` | Search decision notes |
+| `/codewalker libs [--dev]` | Index all direct dependencies from node_modules |
+| `/codewalker lib <pkg> [query]` | Search a specific library's API symbols |
+
+## Tools
 
-## Tool
+The model can call:
 
-The model can call `codewalker_query` directly. Returns `content` (compact text) + `details` (full rows).
+| Tool | Description |
+|------|-------------|
+| `codewalker_query` | Search code symbols, libraries, and notes with FTS5 |
+| `codewalker_enrich` | Write a one-line semantic summary back to a symbol's card + DB |
+| `codewalker_note` | Write a glossary term or decision note (bridge cards) |
 
 ## Install
 
@@ -56,7 +81,7 @@ pi -e ./node_modules/@aprimediet/codewalker/index.ts
 
 ```bash
 npm install
-npm test            # vitest — 86+ tests across 12 test files
+npm test            # vitest — 216+ tests across 19 test files
 npm run test:watch  # watch mode
 ```
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aprimediet/codewalker",
-  "version": "1.1.0",
+  "version": "1.3.0",
   "type": "module",
   "description": "Queryable, token-economical project & code index for the pi coding agent.",
   "keywords": ["pi-package"],

package/prompts/codewalker.md

@@ -1,7 +1,9 @@
 You have access to the codewalker code index for this project. Before reading or grepping
 files to find symbols (functions, consts, classes, types), use `codewalker_query` to
 look them up. The query returns compact facts — name, kind, file:line, and a one-line
-summary.
+summary. Use `source='all'` to also surface glossary terms and decision notes.
 
 - If the index is stale (shown in the result), run `/codewalker sync`.
 - For a full index, run `/codewalker scan`.
+- After reading an unfamiliar symbol, call `codewalker_enrich` to cache a summary.
+- When you discover a design decision or domain term, write it with `codewalker_note`.

package/skills/codewalker/SKILL.md

@@ -1,43 +1,133 @@
-# Codewalker — Queryable Code Index
+---
+name: codewalker
+description: >
+  Queryable code index + knowledge base for understanding codebases. Finds symbol
+  definitions, function summaries, const/class/type/interface lookups, library API
+  searches, glossary terms, and design decisions — before blindly grepping or reading
+  files. Use FIRST when exploring unfamiliar code.
+---
+
+# Codewalker — Queryable Code Index + Knowledge Base
 
 **Use this skill when:** you need to understand a codebase — find where a symbol is defined,
-understand what a function does, or check if a const/class/type exists — BEFORE editing
-files or grepping through the repo.
+understand what a function does, check if a const/class/type/interface exists, search
+library APIs, look up glossary terms or past design decisions — **before** blindly
+grepping or reading files.
 
-## Workflow
+Codewalker is a token-economical project index that surfaces compact facts instead of
+pulling whole files into your LLM context.
 
-1. **Always query first** before editing unfamiliar code:
-   ```
-   /codewalker query "<symbol-name or concept>"
-   ```
-   This returns compact facts: `name · kind · file:line · one-line summary`.
+---
 
-2. **If the query returns relevant hits**, use `file:line` to read only the span you need
-   instead of grepping the whole repo.
+## Tools (agent-facing)
 
-3. **If the query returns no hits**, the index may be stale or missing. Run:
-   ```
-   /codewalker scan
-   ```
-   (first run) or `/codewalker sync` (incremental update).
+### `codewalker_query` — Search the index
 
-4. **When results include a staleness warning** (`indexed @abc, HEAD @def`), run
-   `/codewalker sync` before trusting the results.
+Use this **first** before reading or grepping files. Returns one-line-per-hit:  
+`name · kind · file:line · one-line-summary`
 
-## Why
+Parameters:
+- `query` — symbol name or concept keywords
+- `kind` — optional filter: `function|const|class|type|method|enum|interface|glossary|decision`
+- `limit` — max hits (default 10)
+- `source` — scope: `code` (default, source symbols), `libs` (library APIs), `notes` (glossary + decisions), `all` (everything)
+
+### `codewalker_enrich` — Write a semantic summary
+
+Call this **after** reading a symbol's source span. Caches a one-line (≤120 char)
+plain-English summary so future queries surface meaning, not just names.
+
+Parameters:
+- `card` — `card_path` of the symbol (from the enrich worklist or query results)
+- `summary` — one-line description of what it does
 
-The index is built out-of-band (mechanical ctags/regex pass) so you never pay the file-scan
-cost inside the LLM context. Queries return compact, ranked facts — tens of tokens instead
-of thousands.
+### `codewalker_note` — Save domain knowledge
 
-## Commands
+Write a glossary term or design decision note. Persists as a markdown card + FTS index
+so future queries find it.
+
+Parameters:
+- `type` — `glossary` | `decision`
+- `title` — glossary term or decision title
+- `body` — definition or rationale
+- `tags` — optional comma-separated tags
+- `related` — optional comma-separated symbol names or `file:line` refs
+
+---
+
+## Commands (human-facing)
 
 | Command | Purpose |
 |---------|---------|
-| `/codewalker scan` | Full (re)build of the code index |
-| `/codewalker sync` | Git-anchored incremental update |
-| `/codewalker query <text>` | Search symbols by name or keyword |
+| `/codewalker scan` | Full (re)build of the code index from scratch |
+| `/codewalker sync` | Git-anchored incremental update (fast) |
+| `/codewalker query <text>` | Search code symbols by name or keyword |
+| `/codewalker enrich <path> [--max=N]` | List unenriched symbols under `path` for annotation |
+| `/codewalker glossary [query]` | Search glossary terms |
+| `/codewalker decisions [query]` | Search decision notes |
+| `/codewalker libs [--dev]` | Index all direct npm dependencies (--dev includes devDeps) |
+| `/codewalker lib <pkg> [query]` | Search a specific library's exported API symbols |
+| `/codewalker help` | Show this help |
+
+---
+
+## Workflow
+
+### 1. Before editing unfamiliar code
+```
+/codewalker query "<symbol-name>"
+```
+Or call `codewalker_query` directly from agent conversation.  
+If hits are relevant, use the `file:line` to read only the span you need.
+
+### 2. If the index is stale
+Results include a staleness warning like:  
+`⚠ Index is stale (3 file(s) changed since last index): indexed @abc1234, HEAD @def5678`  
+→ Run `/codewalker sync` first, then query again.
+
+### 3. First time in a project
+```
+/codewalker scan
+```
+This does a full build (ctags primary, regex fallback) and sets up the SQLite+FTS5 database.
+
+### 4. Annotating symbols for future clarity
+After reading a symbol you didn't understand, call `codewalker_enrich` with a summary.  
+This builds up the codebase knowledge map over time.
+
+### 5. Capturing domain knowledge
+When you discover a project-specific concept or learn why a decision was made:
+```
+codewalker_note(type="glossary", title="term", body="definition")
+codewalker_note(type="decision", title="why X", body="rationale")
+```
+These become searchable via `codewalker_query` with `source='notes'` or `source='all'`.
+
+### 6. Exploring library APIs
+```
+/codewalker libs            # index dependencies
+/codewalker lib express     # search express exports
+/codewalker lib lodash get  # search lodash for 'get'
+```
+
+---
+
+## Why
+
+The index is built out-of-band (mechanical ctags/regex pass) so you never pay the
+file-scan cost inside the LLM context. Queries return compact, ranked facts — tens of
+tokens instead of thousands. The note system (glossary + decisions) captures conceptual
+knowledge your future self will thank you for.
+
+---
 
-## Tool
+## Details
 
-The model can also call `codewalker_query` directly (same behavior as the command).
+- **Staleness detection**: every query result includes git-anchored staleness info
+  comparing the indexed commit against HEAD
+- **FTS5 ranking**: results use bm25 relevance scoring; `code` → `libs` → `notes` order
+  when using `source='all'`
+- **Cards as source of truth**: every symbol and note is stored as a markdown card file.
+  The DB is rebuilt from cards on `scan` — cards are the durable artifact
+- **Source filter**: use `source='libs'` to search only library APIs, `source='notes'`
+  for glossary/decisions, `source='all'` for everything at once

package/src/cards.test.ts

@@ -1,5 +1,5 @@
 import { describe, it, expect } from 'vitest';
-import { renderCard, parseCard, cardHead } from './cards.ts';
+import { renderCard, parseCard, cardHead, updateCardSummary } from './cards.ts';
 import type { Symbol } from './types.ts';
 
 function makeSymbol(overrides: Partial<Symbol> = {}): Symbol {
@@ -71,6 +71,128 @@ describe('parseCard', () => {
   });
 });
 
+describe('updateCardSummary', () => {
+  it('replaces frontmatter summary line and appends ## What it does section', () => {
+    const card = `---
+name: probeCompat
+kind: function
+signature: (cwd: string) => CompatResult
+location: compat.ts:201-243
+summary: Old summary
+---
+
+# probeCompat
+
+Some body text here.
+`;
+
+    const updated = updateCardSummary(card, 'Detect whether minion & memory are active.');
+
+    // Frontmatter summary updated
+    expect(updated).toContain('summary: Detect whether minion & memory are active.');
+    expect(updated).not.toContain('summary: Old summary');
+
+    // ## What it does section added
+    expect(updated).toContain('## What it does');
+    expect(updated).toContain('Detect whether minion & memory are active.');
+  });
+
+  it('is idempotent — second apply does not stack duplicates', () => {
+    const card = `---
+name: probeCompat
+kind: function
+location: compat.ts:201-243
+summary: Old
+---
+
+# probeCompat
+`;
+
+    const once = updateCardSummary(card, 'First summary.');
+    const twice = updateCardSummary(once, 'Second summary.');
+
+    // Has the new summary
+    expect(twice).toContain('summary: Second summary.');
+    expect(twice).not.toContain('summary: First summary.');
+
+    // Only one ## What it does section
+    const matches = twice.match(/## What it does/g);
+    expect(matches).toHaveLength(1);
+
+    // Only one summary in frontmatter
+    const summaryMatches = twice.match(/^summary:/gm);
+    expect(summaryMatches).toHaveLength(1);
+  });
+
+  it('handles empty summary in input', () => {
+    const card = `---
+name: foo
+kind: function
+location: foo.ts:1-10
+summary:
+---
+
+# foo
+`;
+
+    const updated = updateCardSummary(card, 'New summary.');
+    expect(updated).toContain('summary: New summary.');
+    expect(updated).toContain('## What it does');
+  });
+
+  it('handles card with no existing body', () => {
+    const card = `---
+name: bar
+kind: const
+location: bar.ts:5-5
+summary:
+---
+`;
+
+    const updated = updateCardSummary(card, 'Just a constant.');
+    expect(updated).toContain('summary: Just a constant.');
+    expect(updated).toContain('## What it does');
+  });
+
+  it('does not break frontmatter for cards with tags field', () => {
+    const card = `---
+name: myFunc
+kind: function
+location: a.ts:1-10
+tags: alpha, beta
+summary:
+---
+
+# myFunc
+`;
+
+    const updated = updateCardSummary(card, 'A function that does something.');
+    const parsed = parseCard(updated);
+    expect(parsed).not.toBeNull();
+    expect(parsed!.head.name).toBe('myFunc');
+    expect(parsed!.head.summary).toBe('A function that does something.');
+    expect(parsed!.head.tags).toEqual(['alpha', 'beta']);
+  });
+
+  it('round-trips through parseCard', () => {
+    const card = `---
+name: probeCompat
+kind: function
+location: compat.ts:201-243
+summary: Old
+---
+
+# probeCompat
+`;
+
+    const updated = updateCardSummary(card, 'A function to detect integrations.');
+    const parsed = parseCard(updated);
+    expect(parsed).not.toBeNull();
+    expect(parsed!.head.summary).toBe('A function to detect integrations.');
+    expect(parsed!.body).toContain('A function to detect integrations.');
+  });
+});
+
 describe('cardHead', () => {
   it('returns only the frontmatter head from a card', () => {
     const sym = makeSymbol();

package/src/cards.ts

@@ -76,6 +76,59 @@ export function parseCard(text: string): { head: CardHead; body: string } | null
   };
 }
 
+/**
+ * Pure: rewrite a card's frontmatter summary: line and upsert a ## What it does body section.
+ * Idempotent — enriching twice yields the same card (replaces the section, doesn't stack duplicates).
+ */
+export function updateCardSummary(cardText: string, summary: string): string {
+  const trimmed = cardText.trim();
+
+  // Split into frontmatter and body
+  const endOfFm = trimmed.indexOf("\n---", 3);
+  if (endOfFm === -1) return cardText; // invalid card, return as-is
+
+  const fmRaw = trimmed.slice(3, endOfFm).trim();
+  const bodyRaw = trimmed.slice(endOfFm + 4).trim();
+
+  // Rebuild frontmatter, replacing summary: line
+  const fmLines = fmRaw.split("\n");
+  const newFmLines: string[] = [];
+  let summaryReplaced = false;
+
+  for (const line of fmLines) {
+    const sep = line.indexOf(":");
+    if (sep > 0) {
+      const key = line.slice(0, sep).trim();
+      if (key === "summary") {
+        newFmLines.push(`summary: ${summary}`);
+        summaryReplaced = true;
+        continue;
+      }
+    }
+    newFmLines.push(line);
+  }
+
+  if (!summaryReplaced) {
+    newFmLines.push(`summary: ${summary}`);
+  }
+
+  const newFrontmatter = `---\n${newFmLines.join("\n")}\n---`;
+
+  // Build body — replace existing ## What it does section if present
+  let body = bodyRaw;
+  const whatItDoesRegex = /## What it does[\s\S]*?(?=\n## |$)/;
+  const whatItDoesSection = `## What it does\n\n${summary}`;
+
+  if (whatItDoesRegex.test(body)) {
+    body = body.replace(whatItDoesRegex, whatItDoesSection);
+  } else {
+    // Append after existing body (or replace empty body)
+    body = body ? `${body}\n\n${whatItDoesSection}` : whatItDoesSection;
+  }
+
+  return `${newFrontmatter}\n\n${body}\n`;
+}
+
 /**
  * Extract only the frontmatter head from a card — the compact, agent-cheap view.
  * Returns null if the card is invalid.