@aprimediet/codewalker 1.2.0 → 1.4.0

package/README.md

@@ -21,12 +21,36 @@ source files ──→ [ctags / regex] ──→ Symbol[]
                                                  index.db (SQLite + FTS5, disposable)
                                                       ↓
                                               codewalker_query (compact results)
+
+node_modules ──→ [.d.ts extract] ──→ LibSymbol[]
+                                          ↓
+                                     renderLibCard() → lib cards (version-pinned)
+                                                      ↓
+                                                 index.db (lib_symbols + FTS5)
+
+agent knowledge ──→ codewalker_enrich / codewalker_note
+                        ↓
+                   cards (enrichment, glossary, decisions, conventions)
+                        ↓
+                   index.db (notes + FTS5, unified query)
+
+coverage/*.info ──→ [coverage parser] ─┐
+source files ─────→ [debt scanner] ────┤
+agent review ─────→ codewalker_finding ─┤
+                        ↓
+                   analysis/*.md cards
+                        ↓
+                   index.db (analysis + FTS5, unified query)
 ```
 
 - **Cards are the source of truth** — markdown in `~/.pi/projects/<id>/codewalker/entries/`.
 - **SQLite + FTS5 is a disposable index** — rebuildable from cards at any time.
 - **ctags primary, regex fallback** — ctags used when available, regex for TS/JS/Py/Go.
+- **Library layer** — extracts API surface from `node_modules` `.d.ts` files (version-pinned).
+- **Semantic + bridge layer** — agent-driven enrichment, glossary terms, and decision notes.
+- **Analysis layer** — mechanical coverage and debt scanning + agent-driven best-practice review.
 - **Git-anchored** — stale index detected per query.
+- **Report, don't gate** — all analysis findings are advisory cards, never a build failure.
 
 ## Commands
 
@@ -35,10 +59,26 @@ source files ──→ [ctags / regex] ──→ Symbol[]
 | `/codewalker scan` | Full (re)build — walks project tree, extracts symbols, writes cards, populates DB |
 | `/codewalker sync` | Git-anchored incremental — reindexes only changed files |
 | `/codewalker query <text>` | Search the index (compact results) |
-
-## Tool
-
-The model can call `codewalker_query` directly. Returns `content` (compact text) + `details` (full rows).
+| `/codewalker enrich <path>` | Select unenriched symbols under `<path>` and write summaries |
+| `/codewalker analyze [path]` | Mechanical coverage + debt analysis (reads lcov.info/coverage-final.json if present) |
+| `/codewalker review <path>` | Agent-driven best-practice review against conventions/decisions (capped at 25 files) |
+| `/codewalker findings [query]` | Search analysis findings with optional `--kind=coverage|debt|practice` filter |
+| `/codewalker conventions [query]` | Search coding conventions |
+| `/codewalker glossary [query]` | Search glossary terms |
+| `/codewalker decisions [query]` | Search decision notes |
+| `/codewalker libs [--dev]` | Index all direct dependencies from node_modules |
+| `/codewalker lib <pkg> [query]` | Search a specific library's API symbols |
+
+## Tools
+
+The model can call:
+
+| Tool | Description |
+|------|-------------|
+| `codewalker_query` | Search code symbols, libraries, notes, and analysis findings with FTS5 |
+| `codewalker_enrich` | Write a one-line semantic summary back to a symbol's card + DB |
+| `codewalker_note` | Write a glossary term, decision note, or coding convention |
+| `codewalker_finding` | Write a coverage, debt, or best-practice analysis finding |
 
 ## Install
 
@@ -56,7 +96,7 @@ pi -e ./node_modules/@aprimediet/codewalker/index.ts
 
 ```bash
 npm install
-npm test            # vitest — 86+ tests across 12 test files
+npm test            # vitest — 295+ tests across 25 test files
 npm run test:watch  # watch mode
 ```
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@aprimediet/codewalker",
-  "version": "1.2.0",
+  "version": "1.4.0",
   "type": "module",
   "description": "Queryable, token-economical project & code index for the pi coding agent.",
   "keywords": ["pi-package"],

package/prompts/codewalker.md

@@ -1,7 +1,14 @@
 You have access to the codewalker code index for this project. Before reading or grepping
 files to find symbols (functions, consts, classes, types), use `codewalker_query` to
 look them up. The query returns compact facts — name, kind, file:line, and a one-line
-summary.
+summary. Use `source='all'` to also surface glossary terms, decision notes, and analysis
+findings.
 
 - If the index is stale (shown in the result), run `/codewalker sync`.
 - For a full index, run `/codewalker scan`.
+- After reading an unfamiliar symbol, call `codewalker_enrich` to cache a summary.
+- When you discover a design decision or domain term, write it with `codewalker_note`.
+- When you learn a coding convention, record it with `codewalker_note type=convention`.
+- Run `/codewalker analyze` for a health snapshot (coverage gaps, debt markers).
+- Use `/codewalker review <path>` to review files against conventions and decisions,
+  writing findings with `codewalker_finding`. Findings are advisory — report, don't gate.

package/skills/codewalker/SKILL.md

@@ -1,43 +1,180 @@
-# Codewalker — Queryable Code Index
+---
+name: codewalker
+description: >
+  Queryable code index + knowledge base for understanding codebases. Finds symbol
+  definitions, function summaries, const/class/type/interface lookups, library API
+  searches, glossary terms, and design decisions — before blindly grepping or reading
+  files. Use FIRST when exploring unfamiliar code.
+---
+
+# Codewalker — Queryable Code Index + Knowledge Base
 
 **Use this skill when:** you need to understand a codebase — find where a symbol is defined,
-understand what a function does, or check if a const/class/type exists — BEFORE editing
-files or grepping through the repo.
+understand what a function does, check if a const/class/type/interface exists, search
+library APIs, look up glossary terms or past design decisions — **before** blindly
+grepping or reading files.
 
-## Workflow
+Codewalker is a token-economical project index that surfaces compact facts instead of
+pulling whole files into your LLM context.
 
-1. **Always query first** before editing unfamiliar code:
-   ```
-   /codewalker query "<symbol-name or concept>"
-   ```
-   This returns compact facts: `name · kind · file:line · one-line summary`.
+---
 
-2. **If the query returns relevant hits**, use `file:line` to read only the span you need
-   instead of grepping the whole repo.
+## Tools (agent-facing)
 
-3. **If the query returns no hits**, the index may be stale or missing. Run:
-   ```
-   /codewalker scan
-   ```
-   (first run) or `/codewalker sync` (incremental update).
+### `codewalker_query` — Search the index
 
-4. **When results include a staleness warning** (`indexed @abc, HEAD @def`), run
-   `/codewalker sync` before trusting the results.
+Use this **first** before reading or grepping files. Returns one-line-per-hit:  
+`name · kind · file:line · one-line-summary`
 
-## Why
+Parameters:
+- `query` — symbol name or concept keywords
+- `kind` — optional filter: `function|const|class|type|method|enum|interface|glossary|decision`
+- `limit` — max hits (default 10)
+- `source` — scope: `code` (default, source symbols), `libs` (library APIs), `notes` (glossary + decisions), `all` (everything)
+
+### `codewalker_enrich` — Write a semantic summary
+
+Call this **after** reading a symbol's source span. Caches a one-line (≤120 char)
+plain-English summary so future queries surface meaning, not just names.
+
+Parameters:
+- `card` — `card_path` of the symbol (from the enrich worklist or query results)
+- `summary` — one-line description of what it does
+
+### `codewalker_note` — Save domain knowledge
+
+Write a glossary term, design decision, or coding convention. Persists as a markdown
+card + FTS index so future queries find it.
 
-The index is built out-of-band (mechanical ctags/regex pass) so you never pay the file-scan
-cost inside the LLM context. Queries return compact, ranked facts — tens of tokens instead
-of thousands.
+Parameters:
+- `type` — `glossary` | `decision` | `convention`
+- `title` — glossary term, decision title, or convention name
+- `body` — definition, rationale, or convention description
+- `tags` — optional comma-separated tags
+- `related` — optional comma-separated symbol names or `file:line` refs
 
-## Commands
+### `codewalker_finding` — Write an analysis finding
+
+Write a coverage, debt, or best-practice finding. Persists as a markdown card under
+`entries/analysis/<kind>/` + FTS index. Called by the agent during a review pass.
+
+Parameters:
+- `kind` — `coverage` | `debt` | `practice`
+- `title` — short finding label
+- `file` — optional file or `file:line` the finding is about
+- `severity` — `info` | `warn` | `high` (default `info`)
+- `body` — finding detail + why it matters, grounded in conventions/decisions
+- `metric` — optional metric string, e.g. `'42%'`, `'fn length 180'`
+- `related` — optional comma-separated symbol names or `file:line` refs
+
+---
+
+## Commands (human-facing)
 
 | Command | Purpose |
 |---------|---------|
-| `/codewalker scan` | Full (re)build of the code index |
-| `/codewalker sync` | Git-anchored incremental update |
-| `/codewalker query <text>` | Search symbols by name or keyword |
+| `/codewalker scan` | Full (re)build of the code index from scratch |
+| `/codewalker sync` | Git-anchored incremental update (fast) |
+| `/codewalker query <text>` | Search code symbols by name or keyword |
+| `/codewalker enrich <path> [--max=N]` | List unenriched symbols under `path` for annotation |
+| `/codewalker analyze [path]` | Mechanical coverage + debt analysis (reads lcov.info if present) |
+| `/codewalker review <path> [--max=N]` | Agent-driven best-practice review against conventions (capped 25 files) |
+| `/codewalker findings [query] [--kind=KIND]` | Search analysis findings |
+| `/codewalker conventions [query]` | Search coding conventions |
+| `/codewalker glossary [query]` | Search glossary terms |
+| `/codewalker decisions [query]` | Search decision notes |
+| `/codewalker libs [--dev]` | Index all direct npm dependencies (--dev includes devDeps) |
+| `/codewalker lib <pkg> [query]` | Search a specific library's exported API symbols |
+| `/codewalker help` | Show this help |
+
+---
+
+## Workflow
+
+### 1. Before editing unfamiliar code
+```
+/codewalker query "<symbol-name>"
+```
+Or call `codewalker_query` directly from agent conversation.  
+If hits are relevant, use the `file:line` to read only the span you need.
+
+### 2. If the index is stale
+Results include a staleness warning like:  
+`⚠ Index is stale (3 file(s) changed since last index): indexed @abc1234, HEAD @def5678`  
+→ Run `/codewalker sync` first, then query again.
+
+### 3. First time in a project
+```
+/codewalker scan
+```
+This does a full build (ctags primary, regex fallback) and sets up the SQLite+FTS5 database.
+
+### 4. Annotating symbols for future clarity
+After reading a symbol you didn't understand, call `codewalker_enrich` with a summary.  
+This builds up the codebase knowledge map over time.
+
+### 5. Capturing domain knowledge
+When you discover a project-specific concept or learn why a decision was made:
+```
+codewalker_note(type="glossary", title="term", body="definition")
+codewalker_note(type="decision", title="why X", body="rationale")
+```
+These become searchable via `codewalker_query` with `source='notes'` or `source='all'`.
+
+### 6. Capturing coding conventions
+When you learn a project-specific coding convention, record it so reviews can measure
+against it:
+```
+codewalker_note(type="convention", title="Use functional components",
+  body="All React components must be pure functions, not classes.")
+```
+Then search them with `/codewalker conventions [query]`.
+
+### 7. Running a health snapshot
+```
+/codewalker analyze
+```
+This parses `coverage/lcov.info` (if present) and scans source files for
+TODO/FIXME/HACK markers, `@ts-ignore`, oversized files, and long functions.
+Results are queryable via `/codewalker findings [query]` or
+`codewalker_query source='analysis'`.
+
+### 8. Reviewing a subsystem against conventions
+```
+/codewalker review src/auth
+```
+This selects files under `src/auth` (capped at 25) and produces a worklist for the
+agent to review each file against the project's conventions and decisions. The agent
+calls `codewalker_finding` to write findings back.
+
+### 9. Exploring library APIs
+```
+/codewalker libs            # index dependencies
+/codewalker lib express     # search express exports
+/codewalker lib lodash get  # search lodash for 'get'
+```
+
+---
+
+## Why
+
+The index is built out-of-band (mechanical ctags/regex pass) so you never pay the
+file-scan cost inside the LLM context. Queries return compact, ranked facts — tens of
+tokens instead of thousands. The note system (glossary + decisions) captures conceptual
+knowledge your future self will thank you for.
+
+---
 
-## Tool
+## Details
 
-The model can also call `codewalker_query` directly (same behavior as the command).
+- **Staleness detection**: every query result includes git-anchored staleness info
+  comparing the indexed commit against HEAD
+- **FTS5 ranking**: results use bm25 relevance scoring; `code` → `libs` → `notes` order
+  when using `source='all'`
+- **Cards as source of truth**: every symbol and note is stored as a markdown card file.
+  The DB is rebuilt from cards on `scan` — cards are the durable artifact
+- **Source filter**: use `source='libs'` to search only library APIs, `source='notes'`
+  for glossary/decisions, `source='analysis'` for findings, `source='all'` for everything
+  at once
+- **Report, don't gate**: analysis findings are advisory cards and FTS rows, never a CI
+  gate or build failure

package/src/analyze/analyzer.test.ts

@@ -0,0 +1,214 @@
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import * as fs from 'node:fs';
+import * as path from 'node:path';
+import * as os from 'node:os';
+import { runAnalyze, rebuildAnalysisDbFromCards } from './analyzer.ts';
+import { openDb, searchFindings } from '../db.ts';
+
+describe('analyzer.ts', () => {
+  let tmpDir: string;
+  let projectRoot: string;
+  let analysisDir: string;
+  let dbPath: string;
+
+  beforeEach(() => {
+    tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cw-analyze-'));
+    projectRoot = path.join(tmpDir, 'project');
+    analysisDir = path.join(tmpDir, 'codewalker', 'entries', 'analysis');
+    const dbDir = path.join(tmpDir, 'codewalker');
+    dbPath = path.join(dbDir, 'index.db');
+    fs.mkdirSync(analysisDir, { recursive: true });
+    fs.mkdirSync(dbDir, { recursive: true });
+  });
+
+  afterEach(() => {
+    fs.rmSync(tmpDir, { recursive: true, force: true });
+  });
+
+  it('runAnalyze with no coverage file still produces debt findings from source scan', () => {
+    // Create a source file with debt markers
+    const srcDir = path.join(projectRoot, 'src');
+    fs.mkdirSync(srcDir, { recursive: true });
+    fs.writeFileSync(
+      path.join(srcDir, 'a.ts'),
+      '// normal line\nconst x = 1;\n// TODO: fix this\nfunction y() { return x; }\n',
+      'utf-8'
+    );
+
+    runAnalyze({
+      projectRoot,
+      analysisDir,
+      dbPath,
+    });
+
+    // Should have debt findings
+    const db = openDb(dbPath);
+    const findings = searchFindings(db, '');
+    expect(findings.length).toBeGreaterThan(0);
+
+    // At least one should be a debt finding
+    const debtFindings = findings.filter(f => f.finding_kind === 'debt');
+    expect(debtFindings.length).toBeGreaterThan(0);
+
+    // Cards should be written
+    const debtDir = path.join(analysisDir, 'debt');
+    expect(fs.existsSync(debtDir)).toBe(true);
+    const cardFiles = fs.readdirSync(debtDir).filter(f => f.endsWith('.md'));
+    expect(cardFiles.length).toBeGreaterThan(0);
+
+    db.close();
+  });
+
+  it('runAnalyze parses coverage/lcov.info if present', () => {
+    const srcDir = path.join(projectRoot, 'src');
+    fs.mkdirSync(srcDir, { recursive: true });
+    fs.writeFileSync(path.join(srcDir, 'token.ts'), 'line1\nline2\nline3\n', 'utf-8');
+
+    // Write coverage file
+    const coverageDir = path.join(projectRoot, 'coverage');
+    fs.mkdirSync(coverageDir, { recursive: true });
+    fs.writeFileSync(
+      path.join(coverageDir, 'lcov.info'),
+      'SF:src/token.ts\nDA:1,1\nDA:2,0\nDA:3,1\nLF:3\nLH:2\nend_of_record\n',
+      'utf-8'
+    );
+
+    runAnalyze({
+      projectRoot,
+      analysisDir,
+      dbPath,
+    });
+
+    // Should have coverage findings
+    const db = openDb(dbPath);
+    const findings = searchFindings(db, '');
+    const coverageFindings = findings.filter(f => f.finding_kind === 'coverage');
+    expect(coverageFindings.length).toBeGreaterThan(0);
+
+    // Card files should be written
+    const coverageDir2 = path.join(analysisDir, 'coverage');
+    expect(fs.existsSync(coverageDir2)).toBe(true);
+
+    db.close();
+  });
+
+  it('runAnalyze parses coverage/coverage-final.json if lcov.info is absent', () => {
+    const srcDir = path.join(projectRoot, 'src');
+    fs.mkdirSync(srcDir, { recursive: true });
+    fs.writeFileSync(path.join(srcDir, 'token.ts'), 'line1\nline2\n', 'utf-8');
+
+    // Write coverage-final.json
+    const coverageDir = path.join(projectRoot, 'coverage');
+    fs.mkdirSync(coverageDir, { recursive: true });
+    fs.writeFileSync(
+      path.join(coverageDir, 'coverage-final.json'),
+      JSON.stringify({
+        'src/token.ts': {
+          path: 'src/token.ts',
+          statementMap: { '0': { start: { line: 1, column: 0 }, end: { line: 1, column: 5 } }, '1': { start: { line: 2, column: 0 }, end: { line: 2, column: 5 } } },
+          fnMap: {}, branchMap: {},
+          s: { '0': 1, '1': 0 },
+          f: {}, b: {},
+        },
+      }),
+      'utf-8'
+    );
+
+    runAnalyze({
+      projectRoot,
+      analysisDir,
+      dbPath,
+    });
+
+    const db = openDb(dbPath);
+    const findings = searchFindings(db, '');
+    const coverageFindings = findings.filter(f => f.finding_kind === 'coverage');
+    expect(coverageFindings.length).toBeGreaterThan(0);
+    db.close();
+  });
+
+  it('re-running analyze on same project does not duplicate findings', () => {
+    const srcDir = path.join(projectRoot, 'src');
+    fs.mkdirSync(srcDir, { recursive: true });
+    fs.writeFileSync(path.join(srcDir, 'a.ts'), '// TODO: once\n', 'utf-8');
+
+    runAnalyze({ projectRoot, analysisDir, dbPath });
+    runAnalyze({ projectRoot, analysisDir, dbPath }); // second run
+
+    const db = openDb(dbPath);
+    const findings = searchFindings(db, '');
+    // All findings should be unique — no duplicates per (finding_kind, file_path, title)
+    const titles = findings.map(f => f.name);
+    const uniqueTitles = new Set(titles);
+    expect(titles.length).toBe(uniqueTitles.size);
+    db.close();
+  });
+
+  it('rebuildAnalysisDbFromCards reconstructs the DB from card files alone', () => {
+    // First run to create cards
+    const srcDir = path.join(projectRoot, 'src');
+    fs.mkdirSync(srcDir, { recursive: true });
+    fs.writeFileSync(path.join(srcDir, 'a.ts'), '// TODO: rebuild this\nconst x = 1;\n', 'utf-8');
+
+    runAnalyze({ projectRoot, analysisDir, dbPath });
+
+    // Verify there are cards on disk
+    const debtDir = path.join(analysisDir, 'debt');
+    const cardFiles = fs.existsSync(debtDir) ? fs.readdirSync(debtDir).filter(f => f.endsWith('.md')) : [];
+    expect(cardFiles.length).toBeGreaterThan(0);
+
+    // Destroy DB and rebuild
+    const db = openDb(dbPath);
+    db.prepare('DELETE FROM analysis').run();
+    const beforeRebuild = searchFindings(db, '');
+    expect(beforeRebuild).toHaveLength(0);
+    db.close();
+
+    // Rebuild from cards
+    rebuildAnalysisDbFromCards(dbPath, analysisDir);
+
+    // Verify findings are back
+    const db2 = openDb(dbPath);
+    const afterRebuild = searchFindings(db2, '');
+    expect(afterRebuild.length).toBeGreaterThan(0);
+    expect(afterRebuild[0]!.finding_kind).toBe('debt');
+    db2.close();
+  });
+
+  it('runAnalyze reports when no coverage data is found without erroring', () => {
+    // No coverage files, no source files
+    const srcDir = path.join(projectRoot, 'src');
+    fs.mkdirSync(srcDir, { recursive: true });
+
+    // Should not throw
+    expect(() => runAnalyze({ projectRoot, analysisDir, dbPath })).not.toThrow();
+
+    // With no source files either, findings should be empty
+    const db = openDb(dbPath);
+    const findings = searchFindings(db, '');
+    expect(findings).toHaveLength(0);
+    db.close();
+  });
+
+  it('runAnalyze respects a custom path filter for debt scanning', () => {
+    const srcDir = path.join(projectRoot, 'src');
+    fs.mkdirSync(srcDir, { recursive: true });
+    fs.writeFileSync(path.join(srcDir, 'a.ts'), '// TODO: file a\n', 'utf-8');
+    fs.writeFileSync(path.join(srcDir, 'b.ts'), '// TODO: file b\n', 'utf-8');
+
+    runAnalyze({
+      projectRoot,
+      analysisDir,
+      dbPath,
+      pathFilter: 'src/a.ts',
+    });
+
+    const db = openDb(dbPath);
+    const findings = searchFindings(db, '');
+    const aFindings = findings.filter(f => f.file_path?.endsWith('a.ts'));
+    const bFindings = findings.filter(f => f.file_path?.endsWith('b.ts'));
+    expect(aFindings.length).toBeGreaterThan(0);
+    expect(bFindings).toHaveLength(0);
+    db.close();
+  });
+});