carto-md 2.0.7 → 2.0.9

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/README.md +290 -26
  2. package/docs/anci/v0.1-DRAFT.md +420 -0
  3. package/docs/scale.md +129 -0
  4. package/package.json +10 -5
  5. package/scripts/postinstall.js +413 -0
  6. package/src/acp/agent.js +5 -5
  7. package/src/acp/providers/index.js +2 -2
  8. package/src/agents/leiden.js +11 -17
  9. package/src/agents/scan-structure.js +1 -1
  10. package/src/anci/consumer.js +305 -0
  11. package/src/anci/deserialize.js +160 -0
  12. package/src/anci/emit.js +85 -0
  13. package/src/anci/serialize.js +264 -0
  14. package/src/anci/yaml.js +401 -0
  15. package/src/bitmap/bitset.js +190 -0
  16. package/src/bitmap/index.js +121 -0
  17. package/src/bitmap/sidecar.js +545 -0
  18. package/src/bitmap/tools.js +310 -0
  19. package/src/cli/anci.js +237 -0
  20. package/src/cli/check.js +57 -0
  21. package/src/cli/index.js +28 -2
  22. package/src/cli/init.js +297 -65
  23. package/src/cli/inspect.js +295 -0
  24. package/src/cli/pr-impact.js +497 -0
  25. package/src/cli/serve.js +1 -1
  26. package/src/cli/watch.js +6 -0
  27. package/src/engine/worker.js +24 -4
  28. package/src/extractors/imports.js +176 -0
  29. package/src/extractors/languages/html.js +4 -1
  30. package/src/extractors/languages/javascript.js +5 -0
  31. package/src/extractors/languages/prisma.js +4 -1
  32. package/src/extractors/languages/python.js +5 -1
  33. package/src/extractors/languages/r.js +4 -1
  34. package/src/extractors/languages/typescript.js +2 -0
  35. package/src/extractors/tree-sitter-parser.js +15 -0
  36. package/src/mcp/change-plan.js +8 -8
  37. package/src/mcp/diff-parser.js +246 -0
  38. package/src/mcp/server-v2.js +489 -8
  39. package/src/mcp/validate.js +304 -0
  40. package/src/store/config-loader.js +77 -0
  41. package/src/store/sqlite-store.js +389 -4
  42. package/src/store/sync-v2.js +472 -97
  43. package/BENCHMARK_RESULTS.md +0 -34
package/README.md CHANGED
@@ -17,7 +17,9 @@ carto init
17
17
 
18
18
  That's it. Carto auto-wires itself into every AI tool you have installed. Restart your AI tool and it now knows your codebase architecturally.
19
19
 
20
- **Works with:** Cursor · Claude Code · Kiro · Claude Desktop · Windsurf · VS Code Copilot · JetBrains · Zed
20
+ **Works with:** Cursor · Claude Code · Codex · Kiro · Claude Desktop · Windsurf · VS Code Copilot · JetBrains · Zed
21
+
22
+ > Carto also publishes [**ANCI**](#anci--the-open-spec-for-codebases-describing-themselves-to-ai) — the open file format for any codebase to describe its architecture to AI tools. Carto is the reference implementation; the spec is open and any tool can consume it. [§ ANCI ↓](#anci--the-open-spec-for-codebases-describing-themselves-to-ai)
21
23
 
22
24
  ---
23
25
 
@@ -34,7 +36,7 @@ That's it. Carto auto-wires itself into every AI tool you have installed. Restar
34
36
 
35
37
  ### Claude Code (CLI)
36
38
 
37
- One command from your project root:
39
+ `carto init` writes `<project>/.mcp.json` for you when Claude Code is detected (`claude` binary on PATH or `~/.claude/` exists). Manual:
38
40
  ```bash
39
41
  claude mcp add carto -- carto serve
40
42
  ```
@@ -43,6 +45,21 @@ Or create `.mcp.json` at the project root:
43
45
  { "mcpServers": { "carto": { "command": "carto", "args": ["serve"] } } }
44
46
  ```
45
47
 
48
+ ### Codex (CLI)
49
+
50
+ `carto init` writes `~/.codex/config.toml` for you when Codex is detected (`codex` binary on PATH or `~/.codex/` exists). Manual:
51
+ ```bash
52
+ codex mcp add carto -- carto serve
53
+ ```
54
+ Or edit `~/.codex/config.toml`:
55
+ ```toml
56
+ [mcp_servers.carto]
57
+ command = "carto"
58
+ args = ["serve"]
59
+ cwd = "/your/project"
60
+ enabled = true
61
+ ```
62
+
46
63
  ### Kiro
47
64
 
48
65
  `carto init` writes this for you. Manual: `~/.kiro/settings/mcp.json`
@@ -52,21 +69,25 @@ Or create `.mcp.json` at the project root:
52
69
 
53
70
  ### Claude Desktop
54
71
 
55
- `carto init` writes this for you. Manual: `~/Library/Application Support/Claude/claude_desktop_config.json`
72
+ `carto init` writes this for you (cross-platform). Manual paths:
73
+ - macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
74
+ - Windows: `%APPDATA%\Claude\claude_desktop_config.json`
75
+ - Linux: `~/.config/Claude/claude_desktop_config.json` (community Linux builds)
76
+
56
77
  ```json
57
78
  { "mcpServers": { "carto": { "command": "carto", "args": ["serve"], "cwd": "/your/project" } } }
58
79
  ```
59
80
 
60
81
  ### VS Code Copilot
61
82
 
62
- `.vscode/mcp.json` in your project root, or Command Palette → `MCP: Add Server`
83
+ `carto init` writes `<project>/.vscode/mcp.json` for you when the `code` binary is on PATH. Manual: `.vscode/mcp.json` in your project root, or Command Palette → `MCP: Add Server`. Note: VS Code uses `servers` (not `mcpServers`) and requires `"type": "stdio"`.
63
84
  ```json
64
- { "servers": { "carto": { "command": "carto", "args": ["serve"], "cwd": "/your/project" } } }
85
+ { "servers": { "carto": { "type": "stdio", "command": "carto", "args": ["serve"] } } }
65
86
  ```
66
87
 
67
88
  ### Windsurf
68
89
 
69
- `~/.windsurf/mcp.json`
90
+ `carto init` writes this for you when Windsurf is detected. Manual: `~/.codeium/windsurf/mcp_config.json`
70
91
  ```json
71
92
  { "mcpServers": { "carto": { "command": "carto", "args": ["serve"], "cwd": "/your/project" } } }
72
93
  ```
@@ -94,6 +115,30 @@ Without carto, your AI greps text and guesses. With carto, it queries structural
94
115
 
95
116
  Your AI's response goes from "here's a guess" to "here's the right change with the architectural context."
96
117
 
118
+ ### Catching bad changes before save
119
+
120
+ The harder problem isn't finding the right file — it's stopping a confident-sounding AI from writing a refactor that breaks the rest of the repo. Carto exposes a diff-shaped query the AI can call before the user accepts a change:
121
+
122
+ > *AI proposes a 12-line patch to `packages/pg-meta/src/pg-format/index.ts`. Before showing the diff, it calls `validate_diff(patch)`.*
123
+
124
+ ```
125
+ # Diff Validation
126
+
127
+ **Risk:** 🔴 HIGH
128
+ **Files changed:** 1
129
+ **Union blast radius:** 83 transitive dependents
130
+
131
+ ## Violations (1)
132
+
133
+ | Severity | Kind | File | Detail |
134
+ |----------|-------------|----------------------------------------|-----------------------------------------------------------------|
135
+ | HIGH | high_blast | `packages/pg-meta/src/pg-format/index.ts` | Modifying this file affects 83 transitive dependents (>50). |
136
+ ```
137
+
138
+ The AI sees this *before* it proposes the diff. It revises its plan, splits the change, or asks the user. The bad refactor never makes it to the screen. Sub-millisecond on a 7,000-file repo — see the **Benchmarks** section below.
139
+
140
+ Every `validate_diff` call is also written to a local SQLite log, so a session that runs five hours later can ask `did_we_discuss_this("snake_case naming")` and get back the prior decision. The AI stops re-deciding settled questions.
141
+
97
142
  ### In the wild
98
143
 
99
144
  ![Claude Code reviewing the supabase repo through carto's MCP server](docs/screenshots/claude-code-supabase.png)
@@ -116,6 +161,9 @@ Your AI's response goes from "here's a guess" to "here's the right change with t
116
161
  | C / C++ | `.cpp` `.cc` `.cxx` `.h` `.hpp` |
117
162
  | C# | `.cs` |
118
163
  | Ruby | `.rb` |
164
+ | R | `.r` `.R` |
165
+ | Prisma schema | `.prisma` |
166
+ | HTML | `.html` (for `fetch()` discovery) |
119
167
 
120
168
  ### Route extraction (framework-aware)
121
169
 
@@ -192,7 +240,131 @@ Carto supports any LLM provider — configure in your editor:
192
240
 
193
241
  ---
194
242
 
195
- ## The 16 MCP tools
243
+ ## GitHub Action PR impact reports
244
+
245
+ Drop carto onto every PR your repo gets. Posts a sticky comment on each pull request with the diff's blast radius, cross-domain violations, affected routes, and a risk badge.
246
+
247
+ `.github/workflows/carto.yml`:
248
+ ```yaml
249
+ name: Carto Impact Report
250
+
251
+ on:
252
+ pull_request:
253
+ branches: [main]
254
+
255
+ permissions:
256
+ contents: read
257
+ pull-requests: write
258
+
259
+ jobs:
260
+ carto:
261
+ runs-on: ubuntu-latest
262
+ steps:
263
+ - uses: actions/checkout@v4
264
+ with: { fetch-depth: 0 }
265
+ - uses: theanshsonkar/carto@v2.0.9
266
+ ```
267
+
268
+ That's the whole config. The action handles `npm install`, builds (or restores from cache) the `.carto/` index, runs `carto pr-impact`, and posts the comment via `GITHUB_TOKEN`.
269
+
270
+ ### Inputs
271
+
272
+ | Input | Default | What it does |
273
+ |-------|---------|--------------|
274
+ | `carto-version` | `latest` | The `carto-md` npm version to install. Pin in production for reproducibility. |
275
+ | `base` | auto (`origin/$GITHUB_BASE_REF`) | Git ref the PR branched from. |
276
+ | `head` | auto (`$GITHUB_SHA`) | Git ref of the PR head. |
277
+ | `fail-on` | _(empty)_ | Fail the workflow when risk meets/exceeds this severity. One of `HIGH`, `MEDIUM`, `LOW`. Empty = comment-only. |
278
+ | `comment-mode` | `sticky` | `sticky` updates the existing carto comment in place. `new` posts a new comment every push. `none` skips posting (renders to stdout). |
279
+ | `node-version` | `20` | Node.js version on the runner. |
280
+
281
+ ### Outputs
282
+
283
+ | Output | Description |
284
+ |--------|-------------|
285
+ | `risk` | Rolled-up risk: `SAFE` \| `LOW` \| `MEDIUM` \| `HIGH`. Lets downstream steps gate behavior on Carto's verdict. |
286
+ | `comment-url` | URL of the posted/updated PR comment. |
287
+
288
+ ### What the comment looks like
289
+
290
+ ```markdown
291
+ ## 🗺️ Carto Impact Report
292
+
293
+ This PR touches AUTH and DATABASE domains.
294
+
295
+ | Metric | Value |
296
+ |--------|-------|
297
+ | Risk | 🔴 HIGH |
298
+ | Blast radius (union) | 23 files |
299
+ | Files changed | 6 |
300
+ | Cross-domain violations introduced | 2 |
301
+ | High-impact file changed | src/auth/session.ts (8 direct dependents) |
302
+
303
+ <details>
304
+ <summary>Affected routes (4)</summary>
305
+
306
+ - POST /auth/login — risk: HIGH
307
+ - GET /auth/me — risk: HIGH
308
+ - POST /auth/register — risk: MEDIUM
309
+ - POST /api/users — risk: LOW
310
+
311
+ </details>
312
+
313
+ <details>
314
+ <summary>Cross-domain violations (2)</summary>
315
+
316
+ - auth/login.ts now imports from payments/billing.ts (AUTH→PAYMENTS)
317
+ - database/user-repo.ts now imports from auth/jwt.ts (DATABASE→AUTH)
318
+
319
+ </details>
320
+ ```
321
+
322
+ ### Standalone CLI use
323
+
324
+ The action is a thin wrapper around `carto pr-impact`. Use it locally, in custom CI, or in pre-commit hooks:
325
+
326
+ ```bash
327
+ carto pr-impact --base origin/main --head HEAD # markdown to stdout
328
+ carto pr-impact --base origin/main --head HEAD --format json
329
+ carto pr-impact --base origin/main --head HEAD --fail-on HIGH # exit 2 on HIGH risk
330
+ ```
331
+
332
+ ---
333
+
334
+ ## ANCI — the open spec for codebases describing themselves to AI
335
+
336
+ Every AI coding tool today re-discovers a codebase's architecture from scratch on every session. Cursor builds its own embedding index. Cline builds its own. Continue builds its own. Same parsing, every tool, every session.
337
+
338
+ **ANCI** (Architecturally Normalized Code Index) is the file format that fixes this. Two files at `.carto/anci.{yaml,bin}` that describe the codebase's architecture in a form any AI tool can read without indexing it itself. OpenAPI did this for REST APIs. ANCI does it for codebases.
339
+
340
+ `carto sync` writes both files automatically. The header is grep-able YAML; the body is a compact binary import graph. Spec lives in [`docs/anci/v0.1-DRAFT.md`](docs/anci/v0.1-DRAFT.md). Carto is the reference implementation.
341
+
342
+ ```bash
343
+ carto anci publish # re-emit anci.{yaml,bin} from the index
344
+ carto anci show # human-readable summary
345
+ carto anci validate ./.carto # validate a published pair
346
+ ```
347
+
348
+ Any tool can consume an ANCI pair without depending on Carto:
349
+
350
+ ```js
351
+ const { loadAnci } = require('carto-md/src/anci/consumer');
352
+
353
+ const reader = loadAnci('./.carto');
354
+ console.log(reader.domains); // [{ name: 'AUTH', file_count: 42 }, ...]
355
+ console.log(reader.getHighImpactFiles(5)); // top 5 by transitive dependents
356
+ console.log(reader.blastRadius('src/auth/session.ts'));// { count, hops, files: [...] }
357
+ console.log(reader.simulateChangeImpact([ // multi-file change blast radius
358
+ 'src/auth/session.ts',
359
+ 'src/db/connection.ts',
360
+ ]));
361
+ ```
362
+
363
+ > **Status:** v0.1.0-DRAFT — wire format may change up to v1.0. The reference implementation lives in this repo at `src/anci/`.
364
+
365
+ ---
366
+
367
+
196
368
 
197
369
  Once carto is wired in, your AI tool can call any of these mid-task. You don't need to memorize them — your AI will pick the right ones.
198
370
 
@@ -201,6 +373,8 @@ Once carto is wired in, your AI tool can call any of these mid-task. You don't n
201
373
  | `get_architecture()` | 500-word project overview: domains, entry points, stack, key patterns. **Use this first.** |
202
374
  | `get_change_plan(intent)` | Given "add rate limiting to /api/users" → files to touch, domains affected, blast radius, similar patterns |
203
375
  | `get_blast_radius(file)` | All files affected by changing a given file, with hop distance |
376
+ | `simulate_change_impact(files)` | Union of all files transitively affected by changing **multiple** files at once. Powered by the bitmap engine — sub-millisecond on 7K-file repos. |
377
+ | `validate_diff(diff)` | Given a unified diff: violations (cross-domain imports, high-blast files), blast radius per file, risk level (SAFE/LOW/MEDIUM/HIGH), suggestions. Each call is recorded in the **episodic memory** log so other tools can ask "did we discuss this?". Sub-15ms p99. |
204
378
  | `get_context(file)` | Everything about a file: domain, blast radius, neighbors, routes, models |
205
379
  | `get_file_summary(file)` | What a file does, its role, key deps and dependents |
206
380
  | `get_similar_patterns(file)` | Files with same domain, route shape, or shared deps — find conventions before writing new code |
@@ -214,8 +388,16 @@ Once carto is wired in, your AI tool can call any of these mid-task. You don't n
214
388
  | `get_high_impact_files(n)` | Top N files by blast radius |
215
389
  | `get_env_vars(domain?)` | All env vars with domain mapping |
216
390
  | `get_domains_list()` | All detected domains with file, route, model counts |
391
+ | `get_recent_decisions(time_range, kind?)` | Recent validation decisions and architectural choices the AI has made in this project |
392
+ | `get_session_context(session_id?)` | Full context for an AI session: every decision and intervention, ordered chronologically |
393
+ | `did_we_discuss_this(topic)` | Substring search over the episodic memory log — avoid re-deciding settled questions |
394
+ | `get_intervention_history(file?)` | Past Carto-issued violations and suggestions, optionally filtered by file |
395
+
396
+ **Latency:** All bitmap-backed queries return in microseconds on real repos; cross-domain and `simulate_change_impact` settle under 2ms even on a 7.5K-file codebase like vscode. See [Benchmarks](#benchmarks) for the per-tool table.
397
+
398
+ ## Episodic Memory
217
399
 
218
- **All MCP queries:** **<5ms** on every benchmarked repo.
400
+ Carto remembers every diff it validates. The `validate_diff` tool writes one row per call into a local SQLite log (`ai_sessions`/`decisions`/`interventions` tables) — so a session that runs five hours later can still ask `did_we_discuss_this("snake_case naming")` and get back the prior decision. The log lives next to the index in `.carto/carto.db` — never sent over the network, never shared between projects.
219
401
 
220
402
  ---
221
403
 
@@ -223,6 +405,8 @@ Once carto is wired in, your AI tool can call any of these mid-task. You don't n
223
405
 
224
406
  Carto uses **Leiden+CPM graph clustering** — files that import each other heavily cluster together. Domain names are inferred from path tokens, with keyword hints for well-known patterns (AUTH, PAYMENTS, DATABASE, etc.).
225
407
 
408
+ **Adaptive strategy:** Repos under 100 files use keyword-only clustering (avoids over-fragmentation). Larger repos with dense import graphs get graph-based clustering with a gamma that scales continuously with repo size.
409
+
226
410
  Works on any repo — not just SaaS apps. vscode gets AUTH/EVENTS/DATABASE. zed (Rust) gets DATABASE/AUTH/EVENTS. A game engine would get RENDERER/PHYSICS/AUDIO.
227
411
 
228
412
  Custom domains via `carto.config.json`:
@@ -235,55 +419,111 @@ Custom domains via `carto.config.json`:
235
419
  }
236
420
  ```
237
421
 
422
+ Full schema with anchor pinning (forces files into a domain regardless of clustering):
423
+ ```json
424
+ {
425
+ "domains": {
426
+ "AUTH": {
427
+ "keywords": ["auth", "login", "session"],
428
+ "anchor": ["src/auth/session.ts", "src/auth/middleware.ts"]
429
+ }
430
+ }
431
+ }
432
+ ```
433
+
434
+ **Stability tracking:** Carto tracks domain assignments across syncs. If >5% of files change domain, `carto check` flags it as unstable. Two consecutive syncs with no code changes always produce 0% drift.
435
+
238
436
  ---
239
437
 
240
438
  ## CLI commands
241
439
 
242
440
  | Command | What it does |
243
441
  |---------|-------------|
244
- | `carto init` | Detect project, index codebase, generate AGENTS.md, auto-wire MCP into every AI tool found |
245
- | `carto sync` | Full re-index (skips unchanged files via mtime+size cache) |
246
- | `carto watch` | Incremental live re-index on every file save (<50ms) |
247
- | `carto serve` | Start MCP server (called by your AI tool usually you don't run this directly) |
442
+ | `carto init` | Detect project, index codebase, generate AGENTS.md, install git hooks (pre-commit, post-checkout, post-merge, post-rewrite), auto-wire MCP into every AI tool found |
443
+ | `carto sync` | Full re-index (skips unchanged files via mtime+size cache). Called automatically by git hooks on commit/checkout/merge/rebase. |
444
+ | `carto serve` | Start MCP server (called by your AI tool — usually you don't run this directly). On every file-aware query the server mtime+size-checks the file and re-parses inline if stale. |
445
+ | `carto watch` | **Optional.** Live re-index on every file save. Not required — git hooks + lazy MCP re-parse keep the index fresh by default. Use only for AI-heavy workflows that write 50+ files between commits. |
248
446
  | `carto agent` | Start ACP agent mode (for Zed / JetBrains / VS Code) |
249
447
  | `carto impact <file>` | Blast radius: risk level, affected files, routes at risk |
448
+ | `carto pr-impact` | Diff-shaped impact report between two git refs. Markdown (default) or JSON. Used by the [GitHub Action](#github-action--pr-impact-reports); works locally too. `--fail-on HIGH\|MEDIUM\|LOW` exits non-zero on threshold trip. |
250
449
  | `carto check` | Cross-domain violations, high-risk uncommitted changes, domain health |
450
+ | `carto inspect` | Read-only diagnostic: index paths, sizes, freshness, bitmap sidecar shape, top-impact files, schema version, sync timestamps. `--json` for piping into `jq`. Never triggers a rebuild. |
251
451
  | `carto remove` | Remove AGENTS.md and .carto/ from project |
252
452
 
253
453
  ---
254
454
 
255
455
  ## Benchmarks
256
456
 
257
- Measured on real open-source repos. Apple M-series, 8 CPUs, 8GB RAM. SHAs pinned in `~/carto-test-repos`.
457
+ Measured on real open-source repos. Apple M-series, 8 CPUs, 8GB RAM. SHAs pinned in `~/carto-test-repos`. Reproducible via `~/carto-test-repos/run-bench.sh`.
458
+
459
+ ### Indexing speed
258
460
 
259
461
  | Repo | Language | Indexed Files | First Run | Second Run | DB Size | Import Edges |
260
462
  |------|----------|---------------|-----------|------------|---------|--------------|
261
- | [prisma/prisma](https://github.com/prisma/prisma) | TypeScript | 961 | **649ms** | **110ms** | 0.7 MB | 1,387 |
262
- | [supabase/supabase](https://github.com/supabase/supabase) | TypeScript | 6,259 | **4.8s** | **751ms** | 4.0 MB | 5,321 |
263
- | [microsoft/vscode](https://github.com/microsoft/vscode) | TypeScript | 7,567 | **7.8s** | **387ms** | 6.7 MB | 13,420 |
264
- | [zed-industries/zed](https://github.com/zed-industries/zed) | Rust | 1,752 | **3.0s** | **182ms** | 4.4 MB | 2,113 |
463
+ | [prisma/prisma](https://github.com/prisma/prisma) | TypeScript | 961 | **1.0s** | **350ms** | 1.1 MB | 1,387 |
464
+ | [supabase/supabase](https://github.com/supabase/supabase) | TypeScript | 6,330 | **5.4s** | **1.2s** | 4.8 MB | 5,189 |
465
+ | [microsoft/vscode](https://github.com/microsoft/vscode) | TypeScript | 7,567 | **8.0s** | **935ms** | 14.3 MB | 13,335 |
466
+ | [zed-industries/zed](https://github.com/zed-industries/zed) | Rust | 1,752 | **2.9s** | **468ms** | 4.8 MB | 2,110 |
467
+
468
+ **Indexed Files** counts what Carto actually parses — `.ts/.js/.py/.go/.rs/...` after excluding `node_modules`, build output, and per-file test patterns (`*.test.*` / `*.spec.*` / `*.stories.*` for JS/TS, `test_*.py` / `*_test.py` for Python, `test_*` / `*_test.r` for R). The on-disk file count of the repo is larger.
469
+
470
+ **Second Run** = `carto sync` after no changes. mtime+size checked before reading content — if nothing changed, nothing is re-parsed.
471
+
472
+ ### MCP query latency (bitmap engine vs SQLite)
473
+
474
+ Carto's MCP query path is bitmap-backed on five tools, plus a sixth (`simulate_change_impact`) that's only feasible with bitmap OR-aggregation. Speedups measured against the same data, same DB, same machine — bitmap path vs the equivalent SQLite path.
475
+
476
+ | Tool | vscode (7,567 files) | Speedup vs SQLite |
477
+ |------|---------------------:|------------------:|
478
+ | `get_blast_radius` | sub-ms | **10.7×** |
479
+ | `get_cross_domain` | 2.1ms | **6.2×** |
480
+ | `get_high_impact_files` | sub-ms | **559×** |
481
+ | `get_similar_patterns` | sub-ms | **73×** |
482
+ | `simulate_change_impact` | sub-ms | **6.5×** (no SQLite equivalent at this latency) |
483
+
484
+ Median speedup across all five tools on vscode: **10.7×**. Smaller repos with denser graphs hit higher peaks — supabase `get_high_impact_files` clocks well over 100× on its tighter import graph. Reproducible via `npm run bench:bitmap -- --repo <path>`.
485
+
486
+ ### `validate_diff` latency
487
+
488
+ The new diff-shaped query that lets the AI ask "is this patch safe?" before showing it to the user. Profiled with a representative 20-line diff against 20 random mid-blast-radius files per repo, 1000 calls each.
489
+
490
+ | Repo | Files | p50 | p99 |
491
+ |------|-------|----:|----:|
492
+ | supabase | 6,259 | **0.082ms** | **0.298ms** |
493
+ | vscode | 7,567 | **0.084ms** | **0.489ms** |
494
+
495
+ Budget was p50 ≤ 5ms, p99 ≤ 15ms. Both targets are cleared by 30-60×. The bitmap engine handles every blast-radius and cross-domain query in microseconds; what's left is diff parsing + result aggregation. Reproducible via `node bench/validation-perf/index.js --repo <path>`.
496
+
497
+ ### Scale
265
498
 
266
- **Indexed Files** counts what Carto actually parses — `.ts/.js/.py/.go/.rs/...` after excluding `node_modules`, build output, and per-file `*.test.*` / `*.spec.*` / `*.stories.*` / `test_*.py` patterns. The on-disk file count of the repo is larger.
499
+ Synthetic stress sweep + real-world corpus, dense-fan-out worst case all the way to 50K files. Headline numbers (full table in [`docs/scale.md`](docs/scale.md)):
267
500
 
268
- **Second run** = only changed files re-parsed. mtime+size checked before reading contentif nothing changed, nothing is re-parsed.
501
+ - Synth 50K files: init 1.1m, `blast_radius` p50 22µs, `simulate_change_impact` p50 50µs, `high_impact_files` p50 750ns. The dense Uint32Array bitset hits 415 MB on disk and 1.35 GB peak RSS at this size the Tier-2 Roaring upgrade per PEAK §9.6 is the next move.
502
+ - Real-world (vscode, 7,567 files): `blast_radius` p50 2.7µs, `cross_domain` p50 1.23ms, `similar_patterns` p50 834ns, `simulate_change_impact` p50 19µs.
503
+
504
+ Reproducible via `npm run bench:scale -- --size <N>` (synth) and `node bench/scale-test/real-world.js --repo <path>` (any local clone, including Linux kernel or Chromium).
269
505
 
270
506
  ### Domains detected
271
507
 
272
508
  | Repo | Domains |
273
509
  |------|---------|
274
- | prisma | CORE · DATABASE · EVENTS · AUTH |
275
- | supabase | CORE · AUTH · DATABASE · PAYMENTS · EVENTS · NOTIFICATIONS · TRPC |
276
- | vscode | CORE · DATABASE · AUTH · EVENTS · NOTIFICATIONS |
277
- | zed (Rust) | CORE · DATABASE · AUTH · EVENTS · PAYMENTS · NOTIFICATIONS · TRPC |
510
+ | prisma | CORE · DATABASE · AUTH · EVENTS |
511
+ | supabase | CORE · AUTH · DATABASE · PAYMENTS · NOTIFICATIONS · EVENTS · TRPC |
512
+ | vscode | EXTENSIONS · AUTH · EVENTS · DATABASE · EXTENSION · CLI · CORE |
513
+ | zed (Rust) | CORE · DATABASE · AUTH · EVENTS · PAYMENTS · TRPC · NOTIFICATIONS |
514
+
515
+ vscode at 7,567 indexed files in around 8 seconds. Rust import graph working on zed (2,110 edges from `mod` declarations and `use crate::` paths).
516
+
517
+ ### Accuracy
278
518
 
279
- vscode at 7,567 indexed files in under 8 seconds. Rust import graph working on zed (2,113 edges from `mod` declarations and `use crate::` paths).
519
+ 12 corpus repos pass `node test/accuracy-corpus.js --samples 100` full parity between the bitmap path and the SQLite path on `blastRadius`, `crossDomain`, `highImpactFiles`, and `simulateChangeImpact`. The bitmap layer is a speedup, never a behavior change.
280
520
 
281
521
  ---
282
522
 
283
523
  ## How it works
284
524
 
285
525
  ```
286
- carto init / carto sync
526
+ carto init
287
527
 
288
528
  Discovers all files (no cap — SQLite handles the volume)
289
529
  mtime+size check → skip unchanged files
@@ -293,9 +533,22 @@ Leiden+CPM graph clustering → auto-detects domains
293
533
  Computes reverse deps → blast radius for every file
294
534
  Writes AGENTS.md + .carto/context/*.md (lazy, on-demand)
295
535
  Auto-wires MCP into every AI tool found
536
+ Installs 4 git hooks: pre-commit, post-checkout, post-merge, post-rewrite
537
+
538
+ [no daemon, no watcher, no background process]
539
+
540
+ ─── Freshness mechanism 1: git hooks (90% of cases) ───
541
+ You commit / pull / switch branches / rebase
542
+ → hook runs `carto sync` quietly in <1s
543
+ → only changed files re-parsed
296
544
 
297
- carto watch (optional)
545
+ ─── Freshness mechanism 2: lazy mtime check (the gap) ───
546
+ You edit files between commits, AI asks "blast radius of db.ts?"
547
+ → MCP server stats db.ts (mtime+size vs DB row)
548
+ → if stale, re-parses just that file inline (<50ms)
549
+ → returns fresh answer
298
550
 
551
+ ─── Optional: carto watch (AI-heavy workflows only) ───
299
552
  File saved → debounce 50ms → re-parse 1 file → SQLite write → <50ms
300
553
  ```
301
554
 
@@ -306,10 +559,21 @@ File saved → debounce 50ms → re-parse 1 file → SQLite write → <50ms
306
559
  - **Sends your code anywhere.** Local only. SQLite on disk.
307
560
  - **Writes secrets into AGENTS.md.** `.cartoignore` blocks `.env` and credential files by default.
308
561
  - **Touches your manual notes.** Writes only between `<!-- CARTO:AUTO:START -->` and `<!-- CARTO:AUTO:END -->`.
562
+ - **Forces you to install a C++ toolchain.** Prebuilt native binaries ship for macOS arm64, Linux x64 (glibc + musl/Alpine), and Windows x64. Intel Macs and other platforms transparently fall back to building from source, then to regex-only extraction if that fails.
309
563
  - **Costs money.** MIT license. Free forever.
310
564
 
311
565
  ---
312
566
 
567
+ ## Origin
568
+
569
+ I was building [Emfirge](https://www.emfirge.cloud) — a cloud security agent that maps AWS infrastructure into a graph and simulates the blast radius of every change.
570
+
571
+ To make the AI inside Emfirge understand infrastructure, I wrote a module called `cartography.py`. It mapped AWS resources, built a graph of how they connected, and wrote it into a structured map. The AI stopped hallucinating. It worked with facts, not guesses.
572
+
573
+ Carto is the same idea, applied to source code. Same insight: AI agents stop guessing once they can query the architecture.
574
+
575
+ ---
576
+
313
577
  ## License
314
578
 
315
579
  MIT. Free forever.