@aprimediet/codewalker 1.0.0 → 1.2.0

package/README.md

@@ -1,71 +1,65 @@
 # @aprimediet/codewalker
 
-Systematic **project intelligence** for the [pi coding agent](https://www.npmjs.com/package/@earendil-works/pi-coding-agent): analyze tech stack, goals, boundaries, status, and technical issues, then generate a **PRD** for humans and **AGENTS.md** / **CLAUDE.md** for coding agents — plus integration with `@aprimediet/minion` and `@aprimediet/memory`.
+Queryable, token-economical project & code index for the pi coding agent.
 
-Documentation is split by audience: `docs/PRD.md` holds the product *what & why* (overview, goals, users, features, metrics); `AGENTS.md` holds the engineering *how* (tech stack, structure, commands, conventions, technical boundaries, gotchas); `CLAUDE.md` is a thin pointer that imports `AGENTS.md` as the single source of truth.
+## What
 
-## Phases
+Instead of blindly scanning files (glob → grep → read) on every request — which floods the
+context window with irrelevant bytes — codewalker builds a **mechanical code index** once,
+out of band, and lets the agent query it with compact, ranked results.
 
-| Phase | What it does |
-|-------|---|
-| Phase 1 | Detect primary language, frameworks, infrastructure, and package manager from manifest files |
-| Phase 2 | Find or gather project goals and non-goals (interactively, one question at a time) |
-| Phase 3 | Document key entry points, external services, exposed interfaces, and technical constraints |
-| Phase 4 | Scan recent commit history and search for technical debt markers (TODO/FIXME/HACK/BUG) |
-| Phase 5 | Detect missing test directories, broken env files, invalid configs, and TypeScript strictness issues |
-| Phase 6 | Generate docs, split by audience: `docs/PRD.md` (product), `AGENTS.md` (engineering), `CLAUDE.md` (pointer to AGENTS.md) — each with user confirmation |
-| Phase 7 | Detect active `@aprimediet/minion` and `@aprimediet/memory` integrations via the shared `.pi/<id>.md` marker |
-| Phase 8 | Compile full project intelligence document with all sections |
-| Phase 9 | Store summary to memory (if active) or save to local file |
+**Token economy is the north star.** Pay the file scan once, outside the LLM context. Query
+cheaply, many times.
 
-## Install
+## Architecture
 
-```bash
-pi install npm:@aprimediet/codewalker
-pi list
 ```
-
-## Quick try
-
-```bash
-pi -e ./extensions/codewalker/index.ts
+source files ──→ [ctags / regex] ──→ Symbol[]
+                                          ↓
+                                     renderCard() → .md files (source of truth)
+                                                      ↓
+                                                 index.db (SQLite + FTS5, disposable)
+                                                      ↓
+                                              codewalker_query (compact results)
 ```
 
-Then run `/learn-this` in any project.
+- **Cards are the source of truth** — markdown in `~/.pi/projects/<id>/codewalker/entries/`.
+- **SQLite + FTS5 is a disposable index** — rebuildable from cards at any time.
+- **ctags primary, regex fallback** — ctags used when available, regex for TS/JS/Py/Go.
+- **Git-anchored** — stale index detected per query.
 
-## Integration
+## Commands
 
-codewalker automatically detects the presence of two companion extensions via a shared project marker:
+| Command | Description |
+|---------|-------------|
+| `/codewalker scan` | Full (re)build — walks project tree, extracts symbols, writes cards, populates DB |
+| `/codewalker sync` | Git-anchored incremental — reindexes only changed files |
+| `/codewalker query <text>` | Search the index (compact results) |
 
-**Minion Integration (read-only):**
-- Reads the `.pi/<project-id>.md` marker file from the current working directory
-- Checks `~/.pi/projects/<id>/tasks/` for open kanban cards (backlog, todo, in_progress, blocked, review)
-- Counts and reports open task count during the `/learn-this` summary
+## Tool
 
-**Memory Integration (read-only + write):**
-- Reads the same `.pi/<project-id>.md` marker file
-- Checks `~/.pi/projects/<id>/memory/` for active memory (MEMORY.md + entries/)
-- Counts memory entries and reads the index during Phase 7
-- In Phase 9, if memory is active, calls `memory_write` with scope `"project"` to store the full intelligence snapshot
+The model can call `codewalker_query` directly. Returns `content` (compact text) + `details` (full rows).
 
-Both integrations use **read-only detection** (no creation of files or directories); the marker file is created and managed by minion or memory when activated. codewalker coexists with them seamlessly in the same `~/.pi/projects/<id>/` workspace.
+## Install
 
-## Layout
+```bash
+# In the pi extensions directory:
+npm install @aprimediet/codewalker
+```
 
+Then load the extension:
+```
+pi -e ./node_modules/@aprimediet/codewalker/index.ts
 ```
-codewalker/                # @aprimediet/codewalker
-├── package.json           # pi manifest: extensions + skills
-├── index.ts               # extension factory: /learn-this command (probes + triggers the workflow)
-├── compat.ts              # minion + memory integration detection
-├── detect.ts              # phase-specific file and marker detection
-├── prd.ts                 # PRD (human) search, read, and generation
-├── agents.ts              # AGENTS.md + CLAUDE.md (agent) search and generation
-└── skills/
-    └── learn-this/
-        └── SKILL.md       # 9-phase intelligence gathering workflow + checklist
+
+## Development
+
+```bash
+npm install
+npm test            # vitest — 86+ tests across 12 test files
+npm run test:watch  # watch mode
 ```
 
-The `/learn-this` command probes integration status, shows it, then sends a user message
-that triggers the agent to invoke the `learn-this` skill and run all 9 phases.
+## License
 
-No third-party runtime deps — only the pi-core packages (peer, bundled by pi).
+MIT

package/index.ts

@@ -1,43 +1,7 @@
-import { type ExtensionAPI, type ExtensionContext } from "@earendil-works/pi-coding-agent";
-import { probeCompat } from "./compat.ts";
+/**
+ * @aprimediet/codewalker — extension entry point.
+ *
+ * Re-exports the factory from src/ so pi finds the extension at root level.
+ */
+export { default } from "./src/index.ts";
 
-export default function codewalkExtension(pi: ExtensionAPI): void {
-	pi.registerCommand("learn-this", {
-		description: "Analyze this project: tech stack, goals, status, issues, then generate PRD + AGENTS.md/CLAUDE.md.",
-		handler: async (_args, ctx: ExtensionContext) => {
-			const compat = probeCompat(ctx.cwd);
-
-			const minionLine = compat.minionActive
-				? `minion: active (project ${compat.projectId}, ${compat.openTaskCount} open tasks)`
-				: "minion: not detected";
-
-			const memoryLine = compat.memoryActive
-				? `memory: active (${compat.memoryEntries} entries)`
-				: "memory: not detected";
-
-			// Surface the probe result to the user immediately (TUI only).
-			if (ctx.hasUI) {
-				ctx.ui.notify(
-					["codewalker: starting /learn-this", `  ${minionLine}`, `  ${memoryLine}`].join("\n"),
-					"info",
-				);
-			}
-
-			// Actually kick off the workflow: send a user message that triggers an agent turn.
-			// The agent invokes the learn-this skill and runs all 9 phases. We hand it the
-			// integration facts up front so Phase 7 (and Phase 9 storage) start from real data.
-			const directive = [
-				"Run the /learn-this project intelligence workflow on the current working directory now.",
-				"Invoke the `learn-this` skill and complete every phase and checklist item in order.",
-				"Do not stop after detection — gather goals interactively (one question at a time) where they are missing, generate the docs (docs/PRD.md for humans, AGENTS.md + CLAUDE.md for coding agents, with the audience-correct content split), and finish with the summary and memory storage steps.",
-				"",
-				"Integration status detected by codewalker (use these facts in Phase 7 and Phase 9):",
-				`- ${minionLine}`,
-				`- ${memoryLine}`,
-				compat.projectId ? `- project id: ${compat.projectId}` : "- project id: none (no .pi marker found)",
-			].join("\n");
-
-			pi.sendUserMessage(directive, { deliverAs: "followUp" });
-		},
-	});
-}

package/package.json

@@ -1,47 +1,28 @@
 {
   "name": "@aprimediet/codewalker",
-  "version": "1.0.0",
+  "version": "1.2.0",
   "type": "module",
-  "description": "Project intelligence snapshot for the pi coding agent — /learn-this",
-  "keywords": [
-    "pi-package",
-    "pi-extension",
-    "codewalker",
-    "project-intelligence",
-    "prd",
-    "agents-md"
-  ],
-  "license": "MIT",
-  "author": {
-    "name": "aprimediet",
-    "url": "https://github.com/aprimediet"
-  },
-  "repository": {
-    "type": "git",
-    "url": "git+https://github.com/aprimediet/codewalker.git"
-  },
-  "bugs": {
-    "url": "https://github.com/aprimediet/codewalker/issues"
-  },
-  "homepage": "https://github.com/aprimediet/codewalker#readme",
-  "engines": {
-    "node": ">=18"
-  },
-  "publishConfig": {
-    "access": "public"
-  },
+  "description": "Queryable, token-economical project & code index for the pi coding agent.",
+  "keywords": ["pi-package"],
   "pi": {
     "extensions": ["./index.ts"],
-    "skills": ["./skills"]
+    "skills": ["./skills"],
+    "prompts": ["./prompts"]
+  },
+  "scripts": {
+    "test": "vitest run",
+    "test:watch": "vitest"
   },
-  "files": [
-    "*.ts",
-    "skills/**",
-    "README.md",
-    "LICENSE",
-    "docs/**"
-  ],
+  "files": ["*.ts", "src/**", "skills/**", "prompts/**", "README.md"],
   "peerDependencies": {
-    "@earendil-works/pi-coding-agent": "*"
-  }
+    "@earendil-works/pi-coding-agent": "*",
+    "@earendil-works/pi-agent-core": "*",
+    "@earendil-works/pi-ai": "*",
+    "@earendil-works/pi-tui": "*",
+    "typebox": "*"
+  },
+  "dependencies": { "better-sqlite3": "^11.0.0" },
+  "devDependencies": { "vitest": "^1.6.0", "@types/better-sqlite3": "^7.6.0" },
+  "engines": { "node": ">=20" },
+  "publishConfig": { "access": "public" }
 }

package/prompts/codewalker.md

@@ -0,0 +1,7 @@
+You have access to the codewalker code index for this project. Before reading or grepping
+files to find symbols (functions, consts, classes, types), use `codewalker_query` to
+look them up. The query returns compact facts — name, kind, file:line, and a one-line
+summary.
+
+- If the index is stale (shown in the result), run `/codewalker sync`.
+- For a full index, run `/codewalker scan`.

package/skills/codewalker/SKILL.md

@@ -0,0 +1,43 @@
+# Codewalker — Queryable Code Index
+
+**Use this skill when:** you need to understand a codebase — find where a symbol is defined,
+understand what a function does, or check if a const/class/type exists — BEFORE editing
+files or grepping through the repo.
+
+## Workflow
+
+1. **Always query first** before editing unfamiliar code:
+   ```
+   /codewalker query "<symbol-name or concept>"
+   ```
+   This returns compact facts: `name · kind · file:line · one-line summary`.
+
+2. **If the query returns relevant hits**, use `file:line` to read only the span you need
+   instead of grepping the whole repo.
+
+3. **If the query returns no hits**, the index may be stale or missing. Run:
+   ```
+   /codewalker scan
+   ```
+   (first run) or `/codewalker sync` (incremental update).
+
+4. **When results include a staleness warning** (`indexed @abc, HEAD @def`), run
+   `/codewalker sync` before trusting the results.
+
+## Why
+
+The index is built out-of-band (mechanical ctags/regex pass) so you never pay the file-scan
+cost inside the LLM context. Queries return compact, ranked facts — tens of tokens instead
+of thousands.
+
+## Commands
+
+| Command | Purpose |
+|---------|---------|
+| `/codewalker scan` | Full (re)build of the code index |
+| `/codewalker sync` | Git-anchored incremental update |
+| `/codewalker query <text>` | Search symbols by name or keyword |
+
+## Tool
+
+The model can also call `codewalker_query` directly (same behavior as the command).

package/src/cards.test.ts

@@ -0,0 +1,88 @@
+import { describe, it, expect } from 'vitest';
+import { renderCard, parseCard, cardHead } from './cards.ts';
+import type { Symbol } from './types.ts';
+
+function makeSymbol(overrides: Partial<Symbol> = {}): Symbol {
+  return {
+    name: 'probeCompat',
+    kind: 'function',
+    file_path: '/root/src/compat.ts',
+    line_start: 201,
+    line_end: 243,
+    signature: '(cwd: string) => CompatResult',
+    doc: 'Detect whether minion & memory are active for the project at cwd.',
+    summary: '',
+    card_path: '',
+    ...overrides,
+  };
+}
+
+describe('renderCard', () => {
+  it('renders a symbol as a markdown card with frontmatter head and body', () => {
+    const sym = makeSymbol();
+    const md = renderCard(sym);
+
+    // Head (frontmatter)
+    expect(md).toContain('---');
+    expect(md).toContain('name: probeCompat');
+    expect(md).toContain('kind: function');
+    expect(md).toContain('signature: (cwd: string) => CompatResult');
+    expect(md).toContain('location: compat.ts:201-243');
+    expect(md).toContain('summary: Detect whether minion & memory are active for the project at cwd.');
+
+    // Body
+    expect(md).toContain('# probeCompat');
+    expect(md).toContain(sym.doc);
+  });
+
+  it('uses relative path (basename only) for location', () => {
+    const sym = makeSymbol({ file_path: '/very/deep/src/util/helper.ts', line_start: 5, line_end: 10 });
+    const md = renderCard(sym);
+    expect(md).toContain('location: helper.ts:5-10');
+  });
+
+  it('handles empty doc gracefully', () => {
+    const sym = makeSymbol({ doc: '' });
+    const md = renderCard(sym);
+    expect(md).toContain('name: probeCompat');
+    expect(md).toContain('# probeCompat');
+    // Body may be empty
+  });
+});
+
+describe('parseCard', () => {
+  it('round-trips renderCard → parseCard preserving head fields', () => {
+    const sym = makeSymbol();
+    const md = renderCard(sym);
+    const parsed = parseCard(md);
+    expect(parsed).not.toBeNull();
+    expect(parsed!.head.name).toBe('probeCompat');
+    expect(parsed!.head.kind).toBe('function');
+    expect(parsed!.head.signature).toBe('(cwd: string) => CompatResult');
+    expect(parsed!.head.location).toBe('compat.ts:201-243');
+    expect(parsed!.head.summary).toContain('minion');
+    // Body contains the doc text
+    expect(parsed!.body).toContain(sym.doc);
+  });
+
+  it('returns null for invalid markdown', () => {
+    expect(parseCard('not frontmatter')).toBeNull();
+    expect(parseCard('')).toBeNull();
+  });
+});
+
+describe('cardHead', () => {
+  it('returns only the frontmatter head from a card', () => {
+    const sym = makeSymbol();
+    const md = renderCard(sym);
+    const head = cardHead(md);
+    expect(head).not.toBeNull();
+    expect(head!.name).toBe('probeCompat');
+    expect(head!.kind).toBe('function');
+    expect(head!.location).toBe('compat.ts:201-243');
+  });
+
+  it('returns null for invalid input', () => {
+    expect(cardHead('no frontmatter here')).toBeNull();
+  });
+});

package/src/cards.ts

@@ -0,0 +1,87 @@
+/**
+ * Markdown card rendering and parsing for codewalker.
+ *
+ * A card has a frontmatter head (agent-cheap, compact) and a body (human-rich, verbose).
+ * The head is what `query` returns; the full card is only expanded on demand.
+ *
+ * PURE module — no I/O.
+ */
+
+import type { Symbol, CardHead } from "./types.ts";
+import * as path from "node:path";
+
+/**
+ * Render a Symbol into a markdown card string with frontmatter head + body.
+ */
+export function renderCard(symbol: Symbol): string {
+  const location = `${path.basename(symbol.file_path)}:${symbol.line_start}-${symbol.line_end}`;
+  const name = symbol.name;
+  const summary = symbol.summary || symbol.doc.split("\n")[0]?.trim() || "";
+
+  const frontmatter = [
+    "---",
+    `name: ${name}`,
+    `kind: ${symbol.kind}`,
+    `signature: ${symbol.signature || ""}`,
+    `location: ${location}`,
+    `summary: ${summary}`,
+    "---",
+  ].join("\n");
+
+  const body = symbol.doc
+    ? [`# ${name}`, "", symbol.doc].join("\n")
+    : `# ${name}`;
+
+  return `${frontmatter}\n\n${body}\n`;
+}
+
+/**
+ * Parse a markdown card string into head + body.
+ * Returns null if the card is invalid.
+ */
+export function parseCard(text: string): { head: CardHead; body: string } | null {
+  const trimmed = text.trim();
+  if (!trimmed.startsWith("---")) return null;
+
+  // Find the closing ---
+  const endOfFm = trimmed.indexOf("\n---", 3);
+  if (endOfFm === -1) return null;
+
+  const fmRaw = trimmed.slice(3, endOfFm).trim();
+  const body = trimmed.slice(endOfFm + 4).trim();
+
+  // Parse frontmatter lines
+  const fm: Record<string, string> = {};
+  for (const line of fmRaw.split("\n")) {
+    const sep = line.indexOf(":");
+    if (sep > 0) {
+      const key = line.slice(0, sep).trim();
+      const value = line.slice(sep + 1).trim();
+      fm[key] = value;
+    }
+  }
+
+  if (!fm["name"]) return null;
+
+  return {
+    head: {
+      name: fm["name"] ?? "",
+      kind: fm["kind"] ?? "",
+      signature: fm["signature"] ?? "",
+      location: fm["location"] ?? "",
+      tags: (fm["tags"] ?? "").split(",").map((t) => t.trim()).filter(Boolean),
+      summary: fm["summary"] ?? "",
+    },
+    body,
+  };
+}
+
+/**
+ * Extract only the frontmatter head from a card — the compact, agent-cheap view.
+ * Returns null if the card is invalid.
+ */
+export function cardHead(text: string): CardHead | null {
+  const parsed = parseCard(text);
+  if (!parsed) return null;
+  return parsed.head;
+}