agent-knowledge 1.5.3 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/docs/INGEST.md ADDED
@@ -0,0 +1,189 @@
1
+ # Codebase Ingestion
2
+
3
+ The **knowledge-ingest** skill populates or updates agent-knowledge from a codebase directory. It uses tree-sitter for zero-token structural extraction, then the agent distills the results into knowledge entries and graph edges.
4
+
5
+ **Same command for both initial ingest and updates** — the skill detects whether a `.knowledge-ingest-cache.json` exists in the target directory and behaves accordingly:
6
+
7
+ | Mode | Trigger | Behavior |
8
+ | ---------------------- | ------------------- | ----------------------------------------------------------------------------------------------- |
9
+ | **Full ingest** | No cache file found | Scans all files, creates all entries from scratch |
10
+ | **Incremental update** | Cache file exists | Only reprocesses changed files (SHA256 diff), adds new files, removes entries for deleted files |
11
+
12
+ ## How It Works
13
+
14
+ ```
15
+ Codebase → tree-sitter AST → clusters → knowledge entries + graph edges
16
+ (zero tokens) (agent) (via existing MCP tools)
17
+ ↕
18
+ .knowledge-ingest-cache.json
19
+ (tracks SHA256 per file for incremental updates)
20
+ ```
21
+
22
+ **Phase 1 — Structural extraction**: A standalone Node.js script (`scripts/tree-sitter-extract.mjs`) parses source files via web-tree-sitter (WASM). Extracts classes, functions, imports, exports, call graphs, rationale comments, and SHA256 hashes. No LLM tokens consumed.
23
+
24
+ **Phase 2 — Clustering**: The agent groups files into subsystems based on the dependency graph and directory structure. Identifies god nodes (most-imported symbols), entry points, and cross-cutting concerns.
25
+
26
+ **Phase 3 — Knowledge entries**: One entry per subsystem (not per file), plus project overview, architecture decisions from rationale comments, and workflow entries from CI configs. All created via `knowledge({ action: "write" })`.
27
+
28
+ **Phase 4 — Graph edges**: Subsystem relationships mapped via `knowledge_graph({ action: "link" })` — `part_of`, `depends_on`, `builds_on`. Auto-linking adds `related_to` edges automatically.
29
+
30
+ **Phase 5 — Multi-modal**: PDFs, architecture diagrams, and URLs are processed if present.
31
+
32
+ ## Prerequisites
33
+
34
+ The tree-sitter dependencies ship with agent-knowledge (v1.6.0+):
35
+
36
+ - `web-tree-sitter` — WASM-based parser (no native compilation)
37
+ - `tree-sitter-wasms` — pre-built grammar files for 8 languages
38
+
39
+ No additional installation needed.
40
+
41
+ ## Supported Languages
42
+
43
+ | Language | Extensions |
44
+ | ---------- | ----------------------------------- |
45
+ | TypeScript | `.ts`, `.tsx` |
46
+ | JavaScript | `.js`, `.jsx`, `.mjs`, `.cjs` |
47
+ | Python | `.py` |
48
+ | Go | `.go` |
49
+ | Rust | `.rs` |
50
+ | Java | `.java` |
51
+ | C | `.c`, `.h` |
52
+ | C++ | `.cpp`, `.cc`, `.cxx`, `.hpp`, etc. |
53
+
54
+ ## Usage
55
+
56
+ ### Supported Platforms
57
+
58
+ The skill uses the [Agent Skills standard](https://agentskills.io) (`SKILL.md`) and works on Claude Code, OpenCode, Cursor, Codex CLI, and Gemini CLI. `setup.js` installs to both `~/.claude/skills/` and `~/.agents/skills/` to cover all platforms.
59
+
60
+ ```
61
+ /knowledge-ingest .
62
+ /knowledge-ingest ~/projects/my-api
63
+ /knowledge-ingest ./libs/auth
64
+ ```
65
+
66
+ Or use natural language: "ingest this codebase", "scan this project", "load this into knowledge", "update knowledge", "refresh knowledge".
67
+
68
+ ### Direct Script Usage
69
+
70
+ Run the tree-sitter extraction script directly (any platform):
71
+
72
+ ```bash
73
+ node ~/.claude/mcp-servers/agent-knowledge/scripts/tree-sitter-extract.mjs ./my-project --json
74
+ ```
75
+
76
+ Then use the JSON output to create knowledge entries via the REST API or MCP tools.
77
+
78
+ ## Extraction Script
79
+
80
+ The tree-sitter extraction script is a standalone CLI:
81
+
82
+ ```bash
83
+ node scripts/tree-sitter-extract.mjs <path> [options]
84
+ ```
85
+
86
+ ### Options
87
+
88
+ | Flag | Default | Description |
89
+ | ------------- | ------------------------------ | ------------------------------- |
90
+ | `--include` | all supported extensions | Glob patterns (comma-separated) |
91
+ | `--exclude` | node_modules, dist, .git, etc. | Directories to skip |
92
+ | `--max-files` | 2000 | Cap on files to process |
93
+ | `--json` | compact | Pretty-print JSON output |
94
+
95
+ ### Output Schema
96
+
97
+ ```json
98
+ {
99
+ "version": 1,
100
+ "timestamp": "2026-04-12T...",
101
+ "root": "/path/to/project",
102
+ "fileCount": 42,
103
+ "files": [
104
+ {
105
+ "path": "src/foo.ts",
106
+ "language": "typescript",
107
+ "sha256": "abc123...",
108
+ "size": 4200,
109
+ "symbols": [
110
+ {
111
+ "kind": "class",
112
+ "name": "FooService",
113
+ "line": 10,
114
+ "endLine": 85,
115
+ "params": null,
116
+ "docstring": "..."
117
+ }
118
+ ],
119
+ "imports": [{ "source": "./bar", "names": ["BarService"] }],
120
+ "exports": ["FooService"],
121
+ "rationale": [{ "line": 22, "tag": "WHY", "text": "Debounce due to upstream rate limit" }],
122
+ "calls": [{ "caller": "processFoo", "callee": "BarService.create" }]
123
+ }
124
+ ],
125
+ "dependencyGraph": {
126
+ "src/foo.ts": ["src/bar.ts"]
127
+ }
128
+ }
129
+ ```
130
+
131
+ ## Incremental Updates
132
+
133
+ After the first ingest, a `.knowledge-ingest-cache.json` file is written to the target directory. On subsequent runs:
134
+
135
+ - **Unchanged files** (same SHA256): skipped entirely
136
+ - **Changed files**: re-extracted, corresponding knowledge entries updated
137
+ - **Deleted files**: their knowledge entries removed
138
+ - **New files**: processed normally
139
+
140
+ ## What Gets Created
141
+
142
+ A typical 200-file project produces:
143
+
144
+ | Category | Count | Content |
145
+ | ------------ | ----- | ------------------------------------------ |
146
+ | `projects/` | 1 | Project overview, tech stack, architecture |
147
+ | `notes/` | 10-25 | Subsystem summaries |
148
+ | `decisions/` | 2-10 | From rationale comments |
149
+ | `workflows/` | 1-5 | From CI/build configs |
150
+
151
+ Plus graph edges: `part_of`, `depends_on`, `builds_on`, and auto-linked `related_to`.
152
+
153
+ ## Customization
154
+
155
+ ### Exclude patterns
156
+
157
+ ```
158
+ /knowledge-ingest ./project --exclude "test,mock,generated"
159
+ ```
160
+
161
+ Or pass to the script directly:
162
+
163
+ ```bash
164
+ node scripts/tree-sitter-extract.mjs ./project --exclude "test,mock,generated,vendor"
165
+ ```
166
+
167
+ ### Large codebases
168
+
169
+ The `--max-files` flag (default 2000) caps how many files are processed, prioritizing the most recently modified. For very large repos, the agent also caps at 30 clusters and merges smaller ones.
170
+
171
+ ## Installation
172
+
173
+ ### Via setup script
174
+
175
+ ```bash
176
+ cd /path/to/agent-knowledge
177
+ node scripts/setup.js
178
+ ```
179
+
180
+ Installs to both `~/.claude/skills/knowledge-ingest/` and `~/.agents/skills/knowledge-ingest/`, covering Claude Code, OpenCode, Cursor, Codex CLI, and Gemini CLI.
181
+
182
+ ### Manual
183
+
184
+ Copy `skills/knowledge-ingest/SKILL.md` to either location:
185
+
186
+ ```bash
187
+ ~/.claude/skills/knowledge-ingest/SKILL.md # Claude Code, OpenCode, Cursor
188
+ ~/.agents/skills/knowledge-ingest/SKILL.md # Codex CLI, Gemini CLI, universal
189
+ ```
package/docs/SETUP.md CHANGED
@@ -328,6 +328,38 @@ Add equivalent instructions to `.windsurfrules` in your project root.
328
328
 
329
329
  ---
330
330
 
331
+ ## Skills
332
+
333
+ agent-knowledge ships a **knowledge-ingest** skill for bootstrapping or updating the knowledge base from a codebase. The skill follows the [Agent Skills standard](https://agentskills.io) (`SKILL.md` format) and works across multiple platforms.
334
+
335
+ ### Automatic installation
336
+
337
+ `node scripts/setup.js` installs the skill to both `~/.claude/skills/` and `~/.agents/skills/`, covering all major platforms:
338
+
339
+ | Platform | Discovery path | Covered by setup.js |
340
+ | ----------- | ---------------------------------------- | ------------------- |
341
+ | Claude Code | `~/.claude/skills/` | Yes |
342
+ | OpenCode | `~/.claude/skills/`, `~/.agents/skills/` | Yes |
343
+ | Cursor | `~/.claude/skills/`, `~/.agents/skills/` | Yes |
344
+ | Codex CLI | `~/.agents/skills/` | Yes |
345
+ | Gemini CLI | `~/.agents/skills/` | Yes |
346
+
347
+ ### Manual installation
348
+
349
+ Copy `skills/knowledge-ingest/SKILL.md` to any of these locations:
350
+
351
+ ```bash
352
+ # Claude Code, OpenCode, Cursor
353
+ ~/.claude/skills/knowledge-ingest/SKILL.md
354
+
355
+ # Codex CLI, Gemini CLI, OpenCode (universal)
356
+ ~/.agents/skills/knowledge-ingest/SKILL.md
357
+ ```
358
+
359
+ See [Ingestion Guide](INGEST.md) for usage details, supported languages, and incremental updates.
360
+
361
+ ---
362
+
331
363
  ## Knowledge Base Setup
332
364
 
333
365
  The knowledge base is a git repository with categorized markdown files.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "agent-knowledge",
3
- "version": "1.5.3",
3
+ "version": "1.6.0",
4
4
  "description": "Cross-session memory and recall for AI agents — git-synced knowledge base, knowledge graph, confidence scoring, hybrid semantic+TF-IDF search, auto-distillation with secrets scrubbing",
5
5
  "type": "module",
6
6
  "main": "dist/index.js",
@@ -25,6 +25,7 @@
25
25
  "files": [
26
26
  "dist/",
27
27
  "scripts/",
28
+ "skills/",
28
29
  "agent-desk-plugin.json",
29
30
  "docs/*.md",
30
31
  "README.md",
@@ -79,6 +80,10 @@
79
80
  "better-sqlite3": "^12.8.0",
80
81
  "morphdom": "^2.7.8",
81
82
  "sqlite-vec": "^0.1.0",
83
+ "tree-sitter-wasms": "^0.1.13",
84
+ "vis-data": "^8.0.3",
85
+ "vis-network": "^10.0.2",
86
+ "web-tree-sitter": "^0.24.7",
82
87
  "ws": "^8.20.0"
83
88
  },
84
89
  "devDependencies": {
@@ -8,3 +8,7 @@ copyFileSync(
8
8
  join(root, 'node_modules', 'morphdom', 'dist', 'morphdom-umd.min.js'),
9
9
  join(root, 'dist', 'ui', 'morphdom.min.js'),
10
10
  );
11
+ copyFileSync(
12
+ join(root, 'node_modules', 'vis-network', 'standalone', 'umd', 'vis-network.min.js'),
13
+ join(root, 'dist', 'ui', 'vis-network.min.js'),
14
+ );
@@ -0,0 +1,238 @@
1
+ #!/usr/bin/env node
2
+
3
+ // =============================================================================
4
+ // agent-knowledge SessionStart ingest hook
5
+ //
6
+ // On session start, detects the current project and checks whether a
7
+ // knowledge-ingest cache exists. If no cache: suggests bootstrapping.
8
+ // If cache exists: runs a quick SHA256 diff against the current files
9
+ // to report how many files changed since last ingest.
10
+ //
11
+ // Zero LLM tokens — only reads files and computes hashes.
12
+ // Fail-open: errors logged to stderr, always outputs valid JSON.
13
+ // =============================================================================
14
+
15
+ import { readFileSync, readdirSync, statSync, existsSync } from 'fs';
16
+ import { join, extname, resolve, dirname } from 'path';
17
+ import { createHash } from 'crypto';
18
+ import { fileURLToPath } from 'url';
19
+
20
+ const __filename = fileURLToPath(import.meta.url);
21
+ const __dirname = dirname(__filename);
22
+
23
+ process.on('uncaughtException', (err) => {
24
+ process.stderr.write(`[session-start-ingest] fatal: ${err.message}\n`);
25
+ console.log(JSON.stringify({}));
26
+ process.exit(0);
27
+ });
28
+
29
+ const SUPPORTED_EXTENSIONS = new Set([
30
+ '.ts',
31
+ '.tsx',
32
+ '.js',
33
+ '.jsx',
34
+ '.mjs',
35
+ '.cjs',
36
+ '.py',
37
+ '.go',
38
+ '.rs',
39
+ '.java',
40
+ '.c',
41
+ '.h',
42
+ '.cpp',
43
+ '.cc',
44
+ '.cxx',
45
+ '.hpp',
46
+ '.hh',
47
+ '.hxx',
48
+ ]);
49
+
50
+ const DEFAULT_EXCLUDE = new Set([
51
+ 'node_modules',
52
+ '.git',
53
+ 'dist',
54
+ 'build',
55
+ 'target',
56
+ 'vendor',
57
+ '__pycache__',
58
+ '.venv',
59
+ 'venv',
60
+ '.tox',
61
+ '.mypy_cache',
62
+ '.next',
63
+ '.nuxt',
64
+ 'coverage',
65
+ '.pytest_cache',
66
+ '.cargo',
67
+ 'bin',
68
+ 'obj',
69
+ '.gradle',
70
+ '.idea',
71
+ '.vscode',
72
+ ]);
73
+
74
+ function sha256(content) {
75
+ return createHash('sha256').update(content).digest('hex');
76
+ }
77
+
78
+ function countFiles(dir, maxFiles = 500) {
79
+ const hashes = new Map();
80
+ let count = 0;
81
+
82
+ function walk(currentDir) {
83
+ if (count >= maxFiles) return;
84
+ let entries;
85
+ try {
86
+ entries = readdirSync(currentDir, { withFileTypes: true });
87
+ } catch {
88
+ return;
89
+ }
90
+ for (const entry of entries) {
91
+ if (count >= maxFiles) return;
92
+ if (DEFAULT_EXCLUDE.has(entry.name)) continue;
93
+ const fullPath = join(currentDir, entry.name);
94
+ if (entry.isDirectory()) {
95
+ walk(fullPath);
96
+ } else if (entry.isFile()) {
97
+ const ext = extname(entry.name).toLowerCase();
98
+ if (!SUPPORTED_EXTENSIONS.has(ext)) continue;
99
+ count++;
100
+ try {
101
+ const content = readFileSync(fullPath, 'utf-8');
102
+ const relPath = fullPath.slice(dir.length + 1).replace(/\\/g, '/');
103
+ hashes.set(relPath, sha256(content));
104
+ } catch {
105
+ // skip unreadable files
106
+ }
107
+ }
108
+ }
109
+ }
110
+
111
+ walk(dir);
112
+ return { hashes, totalFiles: count };
113
+ }
114
+
115
+ function detectProjectName(dir) {
116
+ const checks = [
117
+ { file: 'package.json', extract: (c) => JSON.parse(c).name },
118
+ { file: 'Cargo.toml', extract: (c) => c.match(/name\s*=\s*"([^"]+)"/)?.[1] },
119
+ {
120
+ file: 'go.mod',
121
+ extract: (c) =>
122
+ c
123
+ .match(/module\s+(\S+)/)?.[1]
124
+ ?.split('/')
125
+ .pop(),
126
+ },
127
+ { file: 'pyproject.toml', extract: (c) => c.match(/name\s*=\s*"([^"]+)"/)?.[1] },
128
+ ];
129
+ for (const { file, extract } of checks) {
130
+ const p = join(dir, file);
131
+ if (existsSync(p)) {
132
+ try {
133
+ return extract(readFileSync(p, 'utf-8'));
134
+ } catch {
135
+ // continue
136
+ }
137
+ }
138
+ }
139
+ return dir.split(/[\\/]/).pop();
140
+ }
141
+
142
+ async function main() {
143
+ let input = '';
144
+ for await (const chunk of process.stdin) input += chunk;
145
+
146
+ let hookData = {};
147
+ try {
148
+ hookData = JSON.parse(input);
149
+ } catch {
150
+ // no stdin data
151
+ }
152
+
153
+ const cwd = hookData?.cwd || hookData?.workspace?.current_dir || process.cwd();
154
+ const projectDir = resolve(cwd);
155
+
156
+ const cachePath = join(projectDir, '.knowledge-ingest-cache.json');
157
+
158
+ if (!existsSync(cachePath)) {
159
+ const { totalFiles } = countFiles(projectDir, 100);
160
+ if (totalFiles < 3) {
161
+ console.log(JSON.stringify({}));
162
+ return;
163
+ }
164
+
165
+ const projectName = detectProjectName(projectDir);
166
+ console.log(
167
+ JSON.stringify({
168
+ hookSpecificOutput: {
169
+ hookEventName: 'SessionStart',
170
+ additionalContext: `knowledge-ingest: Project "${projectName}" (${totalFiles}+ source files) has not been ingested yet. Run \`/knowledge-ingest ${projectDir}\` to bootstrap the knowledge base.`,
171
+ },
172
+ }),
173
+ );
174
+ return;
175
+ }
176
+
177
+ let cache;
178
+ try {
179
+ cache = JSON.parse(readFileSync(cachePath, 'utf-8'));
180
+ } catch {
181
+ console.log(JSON.stringify({}));
182
+ return;
183
+ }
184
+
185
+ const cachedFiles = cache.files || {};
186
+ const { hashes: currentHashes } = countFiles(projectDir);
187
+
188
+ let changed = 0;
189
+ let added = 0;
190
+ let removed = 0;
191
+
192
+ for (const [path, hash] of currentHashes) {
193
+ if (!cachedFiles[path]) {
194
+ added++;
195
+ } else if (cachedFiles[path].sha256 !== hash) {
196
+ changed++;
197
+ }
198
+ }
199
+
200
+ for (const path of Object.keys(cachedFiles)) {
201
+ if (!currentHashes.has(path)) {
202
+ removed++;
203
+ }
204
+ }
205
+
206
+ const total = changed + added + removed;
207
+
208
+ if (total === 0) {
209
+ console.log(
210
+ JSON.stringify({
211
+ hookSpecificOutput: {
212
+ hookEventName: 'SessionStart',
213
+ additionalContext: `knowledge-ingest: Project "${cache.project}" knowledge is up to date (${Object.keys(cachedFiles).length} files, ${(cache.entries_created || []).length} entries).`,
214
+ },
215
+ }),
216
+ );
217
+ return;
218
+ }
219
+
220
+ const parts = [];
221
+ if (changed > 0) parts.push(`${changed} changed`);
222
+ if (added > 0) parts.push(`${added} new`);
223
+ if (removed > 0) parts.push(`${removed} deleted`);
224
+
225
+ console.log(
226
+ JSON.stringify({
227
+ hookSpecificOutput: {
228
+ hookEventName: 'SessionStart',
229
+ additionalContext: `knowledge-ingest: Project "${cache.project}" has ${total} file changes since last ingest (${parts.join(', ')}). Run \`/knowledge-ingest ${projectDir}\` to update.`,
230
+ },
231
+ }),
232
+ );
233
+ }
234
+
235
+ main().catch((err) => {
236
+ process.stderr.write(`[session-start-ingest] ${err.message}\n`);
237
+ console.log(JSON.stringify({}));
238
+ });
package/scripts/setup.js CHANGED
@@ -16,7 +16,7 @@
16
16
  // Usage: node scripts/setup.js [--agent claude|generic]
17
17
  // =============================================================================
18
18
 
19
- import { readFileSync, writeFileSync, existsSync } from 'fs';
19
+ import { readFileSync, writeFileSync, existsSync, statSync, cpSync, mkdirSync } from 'fs';
20
20
  import { join, resolve, dirname } from 'path';
21
21
  import { homedir } from 'os';
22
22
  import { execSync } from 'child_process';
@@ -134,6 +134,14 @@ addUnmatchedHook(
134
134
  5,
135
135
  );
136
136
 
137
+ // SessionStart: knowledge-ingest freshness check
138
+ addUnmatchedHook(
139
+ 'SessionStart',
140
+ 'session-start-ingest.mjs',
141
+ `node "${join(hookDir, 'session-start-ingest.mjs')}"`,
142
+ 10,
143
+ );
144
+
137
145
  // PreCompact: rich summary (library) + simple text distill (heuristic)
138
146
  addUnmatchedHook(
139
147
  'PreCompact',
@@ -159,6 +167,40 @@ addUnmatchedHook(
159
167
  writeFileSync(SETTINGS_JSON, JSON.stringify(settings, null, 2));
160
168
  console.log(' Saved settings.json');
161
169
 
170
+ // ---------------------------------------------------------------------------
171
+ // Install knowledge-ingest skill
172
+ // ---------------------------------------------------------------------------
173
+
174
+ console.log('Installing knowledge-ingest skill...');
175
+
176
+ const skillSrc = join(PROJECT_DIR, 'skills', 'knowledge-ingest', 'SKILL.md');
177
+
178
+ const skillDests = [
179
+ join(HOME, '.claude', 'skills', 'knowledge-ingest', 'SKILL.md'),
180
+ join(HOME, '.agents', 'skills', 'knowledge-ingest', 'SKILL.md'),
181
+ ];
182
+
183
+ if (existsSync(skillSrc)) {
184
+ const srcMtime = statSync(skillSrc).mtimeMs;
185
+ for (const dest of skillDests) {
186
+ mkdirSync(dirname(dest), { recursive: true });
187
+ let shouldCopy = true;
188
+ if (existsSync(dest)) {
189
+ if (statSync(dest).mtimeMs >= srcMtime) {
190
+ shouldCopy = false;
191
+ }
192
+ }
193
+ if (shouldCopy) {
194
+ cpSync(skillSrc, dest);
195
+ console.log(` Installed → ${dest}`);
196
+ } else {
197
+ console.log(` Up to date → ${dest}`);
198
+ }
199
+ }
200
+ } else {
201
+ console.log(' Warning: skills/knowledge-ingest/SKILL.md not found in repo');
202
+ }
203
+
162
204
  console.log(`
163
205
  Setup complete!
164
206