@maxgfr/codeindex 2.7.0

package/README.md

@@ -0,0 +1,115 @@
+# codeindex
+
+Self-contained, deterministic **repo-indexing engine**: file walking, language
+detection, symbol/import extraction (tree-sitter AST with a regex fallback),
+import resolution, a typed cross-file link-graph, and graph analytics — shipped
+as a single zero-dependency `engine.mjs` that consumer tools **vendor** (copy
+into their repo) instead of installing.
+
+Extracted from [ultraindex](https://github.com/maxgfr/ultraindex) 5.1.0's core,
+and consumed by the ultra* skill family (ultraindex, ultradoc, ultrasec,
+ultraeval, reconstruct, construct, ultra11y).
+
+## What it does
+
+- **Walk** a repo deterministically: ignore lists, binary/lockfile skips, size
+  and count caps (`capped` flag, never silent truncation), symlink-cycle guard.
+- **Scan** every file into a `FileRecord`: classification, language, symbols,
+  imports, headings, hashes — with an incremental cache fastpath.
+- **Extract symbols** via tree-sitter (10 languages, when the wasm sidecar is
+  present) or per-language regex rules (15 languages, always available).
+- **Resolve imports** across languages: tsconfig paths, package `exports`,
+  go.mod, Cargo, Java packages, PSR-4, C# namespaces.
+- **Build a typed link-graph**: `import` / `call` / `use` / `doc-link` /
+  `mention` edges at file and module level, plus Louvain communities, PageRank/
+  betweenness centrality, a tests→code map, and surprise-edge detection.
+- **Render** byte-stable `graph.json` / `symbols.json` (two builds of an
+  unchanged repo are byte-identical).
+
+## Use as a library (the vendoring model)
+
+Consumers commit `scripts/engine.mjs` + `scripts/engine.d.mts` (fetched at a
+pinned release tag) into `src/vendor/` and import from it; their bundler inlines
+the engine so they still ship a single file:
+
+```ts
+import { buildIndexArtifacts, renderGraphJson } from "./vendor/engine.mjs";
+
+const { scan, graph, symbols } = buildIndexArtifacts("/path/to/repo");
+```
+
+The AST tier is optional: without a `grammars/` directory next to the bundle
+the engine silently uses its regex tier. Only tools that want AST precision
+(e.g. ultraindex) also vendor `scripts/grammars/` (~17 MiB of wasm).
+
+## Use from npm
+
+For consumers who don't want to vendor the bundle, `@maxgfr/codeindex` also
+resolves as a regular package:
+
+```sh
+npm i @maxgfr/codeindex
+```
+
+```ts
+import { scanRepo, ENGINE_VERSION } from "@maxgfr/codeindex";
+
+const scan = scanRepo("/path/to/repo");
+```
+
+The CLI ships in the same package (`npm i -g @maxgfr/codeindex`, see below).
+Skills should still prefer vendoring: it keeps their own bundle single-file and
+pinned to an exact commit without an npm dependency.
+
+## Use as a CLI
+
+```sh
+brew install maxgfr/tap/codeindex        # or: npm i -g @maxgfr/codeindex
+
+codeindex index   --repo . --out .codeindex   # graph + symbols + incremental cache
+codeindex graph   --repo . > graph.json
+codeindex callers --repo .                    # per-symbol caller index
+codeindex grep    'pattern' --repo .
+```
+
+## Use as an MCP server
+
+`codeindex mcp` (or `node scripts/cli.mjs mcp`) serves the engine over stdio —
+tools: `scan_summary`, `graph`, `symbols`, `callers`, `workspaces`, `churn`,
+`grep`. Register it in Claude Code with:
+
+```sh
+claude mcp add codeindex -- codeindex mcp
+```
+
+`engine.mjs` is a pure side-effect-free library (safe for consumers to inline
+into their own CLIs); `cli.mjs` is the thin standalone CLI/MCP wrapper.
+
+## Versioning
+
+- `ENGINE_VERSION` — the release tag, embedded greppably in the bundle.
+- `SCHEMA_VERSION` — the `graph.json`/`symbols.json` shape (continues
+  ultraindex's lineage; currently 4). Consumers reject mismatched artifacts.
+- `EXTRACTOR_VERSION` — the extraction output shape; incremental caches keyed
+  on it are discarded wholesale when it bumps.
+
+`buildGraph`/`buildIndexArtifacts` accept `meta: { version, schemaVersion }` so
+a consumer can stamp its own identity into artifacts it persists.
+
+## Development
+
+```sh
+pnpm install
+pnpm test          # unit + fixtures + compat + no-wasm gates
+pnpm typecheck
+pnpm build         # tsup → scripts/engine.mjs + scripts/engine.d.mts
+pnpm check:build   # proves the committed bundle is byte-reproducible
+pnpm test:e2e      # opt-in: pinned real-repo builds with ratchets
+```
+
+The compat suite pins the exact bytes ultraindex 5.1.0 produced for the
+`mini-repo` fixture — the proof that extraction was lossless.
+
+## License
+
+MIT

package/docs/MIGRATION.md

@@ -0,0 +1,100 @@
+# Migrating a skill onto the codeindex engine
+
+The engine ships as two files, released together at every tag:
+
+- `scripts/engine.mjs` — self-contained zero-dependency ESM bundle (library + CLI + MCP server)
+- `scripts/engine.d.mts` — TypeScript declarations for the bundle
+
+Consumers **vendor** them (commit a copy) — never an npm dependency — so every
+skill stays standalone-installable.
+
+## Vendoring steps
+
+1. Copy `scripts/sync-engine.mjs` (reference implementation lives in the
+   ultraeval repo) into your repo and run:
+
+   ```sh
+   node scripts/sync-engine.mjs --ref v1.0.0
+   ```
+
+   It fetches `engine.mjs` + `engine.d.mts` from
+   `https://raw.githubusercontent.com/maxgfr/codeindex/<ref>/scripts/…` into
+   `src/vendor/`, greps the bundle's `ENGINE_VERSION` to assert it matches the
+   tag, and writes `src/vendor/engine.meta.json` (`{ tag, sha256, syncedAt }`).
+
+2. Import what you need from the vendored file:
+
+   ```ts
+   import { scanRepo, buildGraph, gitChurn } from "../vendor/engine.mjs";
+   ```
+
+   With `moduleResolution: "Bundler"` the sibling `.d.mts` types the import
+   with zero config. Your tsup/esbuild build inlines the engine into your own
+   single-file bundle — the user installs nothing extra.
+
+3. Exclude `src/vendor/` from your linter, and add
+   `node scripts/sync-engine.mjs --check` (offline sha256 check against
+   `engine.meta.json`) to your CI so local edits to the vendored file fail
+   loudly. Upgrades are a deliberate re-pin: run the sync with a newer tag and
+   commit the diff.
+
+The AST tier is optional: without a `grammars/` directory next to the vendored
+bundle the engine uses its regex tier (15 languages). Only vendor
+`scripts/grammars/` (~17 MiB wasm) if you need AST-exact symbols (ultraindex
+does; nobody else should).
+
+## Version constants
+
+| Constant | Meaning | On mismatch |
+|---|---|---|
+| `ENGINE_VERSION` | the release tag, greppable in the bundle | sync script refuses the fetch |
+| `SCHEMA_VERSION` | `graph.json` / `symbols.json` shape | reject the artifact, rebuild |
+| `EXTRACTOR_VERSION` | extraction output shape | discard incremental caches wholesale |
+
+`buildGraph(...)` / `buildIndexArtifacts(...)` accept
+`meta: { version, schemaVersion }` so a consumer stamps its own identity into
+artifacts it persists (ultraindex does this to keep its graph.json lineage).
+
+## Per-skill mapping (what to replace with what)
+
+| Skill | Replace | With (engine export) |
+|---|---|---|
+| **ultraeval** | `walkFiles`, `SKIP_DIRS` | `scanRepo` (gitignore on by default) |
+| | `importsOf` + `resolveImport` (JS/TS+py regex) | `FileRecord.refs` (kind `import`) + `buildResolveContext`/`resolveImport` |
+| | `gitChurn`, `changedFiles` | `gitChurn`, `changedSince` |
+| | keeps | cycle DFS, hotspot scoring, todos/maxIndent, ANALYSIS.md rendering |
+| **construct** | walker clone + language histogram + test count | `scanRepo` (`.languages`) + `isTestPath` |
+| **ultraindex** | the whole core (walk/scan/extract/resolve/modules/graph/calls/analytics/renderers) | vendored engine + shim `src/engine.ts` re-exporting it; pass `meta: { version: VERSION, schemaVersion: SCHEMA_VERSION }` to keep graph.json byte-lineage |
+| | keeps | store/manifest, encyclopedia, merge/entries, find/ask, check/verify, embeddings, orchestrate |
+| **ultradoc** | walker, `EXT_LANG` map, symbol `RULES` + `applyExportLists` | `scanRepo`, `extToLang`/`languageOf`, `extractSymbols`/`buildSymbolIndex` |
+| | `PKG_MANIFESTS` workspace probing | `detectWorkspaces` |
+| | `rgSearch` + JS fallback | `grepRepo` |
+| | keeps | issues/PRs/SO/web retrieval, citations, DOC.md |
+| **ultra11y** | walker + glob scoping | `walk`/`scanRepo` with `include`/`exclude` |
+| | tsconfig path-alias resolution | `buildResolveContext` + `resolveImport` |
+| | keeps | its JSX component graph and every WCAG check |
+| **reconstruct** | walker + gitignore parser + `categorize` | `scanRepo`, `categorize` |
+| | `EXT_LANGUAGE` | `extToLang` |
+| | workspace detection + dep graph + cycle + topo | `detectWorkspaces` (packages/dependsOn/cycle/topoOrder) |
+| | `resolveModule` | `resolveImport` |
+| | keeps | framework adapters (routes), data-model inference, PRD rendering |
+| **ultrasec** | walker + gitignore + scope + symlink guard | `scanRepo` (`scope`, `gitignore` — its own semantics, ported here) |
+| | per-language defs/imports/calls extraction | `extractCode` (`symbols`/`refs`/`calls`) |
+| | `resolveImport` | `resolveImport` |
+| | `buildGraph` (import+call edges, symbolDefs, callersBySymbol, enclosingSymbol) | `buildGraph` + `resolveCallEdges` + `buildCallerIndex` + `enclosingSymbol` |
+| | keeps | taint source→sink enumeration, external scanners, EPSS/KEV/CVSS, SARIF |
+
+## Golden-diff adjudication (every migration)
+
+Capture the skill's load-bearing artifact **before** touching code (a committed
+snapshot test), migrate, then adjudicate every diff:
+
+- **Accept + document**: file-set changes from better ignore rules (gitignore
+  honored, lockfile/binary/1 MiB skips) and strictly-more-resolved imports
+  (tsconfig paths, package `exports`, go/cargo) — list them in the commit body.
+- **Must be identical**: output schemas, rendering, scoring formulas,
+  attribution logic on identical inputs.
+- **Investigate before accepting**: anything else (a cycle, hotspot, finding or
+  score changing for an unlisted reason). If the engine is more correct, update
+  the golden citing the cause; otherwise **fix codeindex and re-release** —
+  never patch around it in the consumer.

package/package.json

@@ -0,0 +1,89 @@
+{
+  "name": "@maxgfr/codeindex",
+  "version": "2.7.0",
+  "description": "Self-contained, deterministic repo-indexing engine: walk + language detection + symbol/import extraction (tree-sitter AST with regex fallback) + import resolution + typed cross-file link-graph + analytics. Ships as a single zero-dependency engine.mjs that consumer tools vendor.",
+  "type": "module",
+  "packageManager": "pnpm@10.33.0",
+  "main": "./scripts/engine.mjs",
+  "types": "./scripts/engine.d.mts",
+  "exports": {
+    ".": {
+      "types": "./scripts/engine.d.mts",
+      "import": "./scripts/engine.mjs",
+      "default": "./scripts/engine.mjs"
+    },
+    "./package.json": "./package.json"
+  },
+  "bin": {
+    "codeindex": "scripts/cli.mjs"
+  },
+  "files": [
+    "scripts/engine.mjs",
+    "scripts/engine.d.mts",
+    "scripts/cli.mjs",
+    "scripts/grammars",
+    "src",
+    "docs/MIGRATION.md"
+  ],
+  "engines": {
+    "node": ">=18"
+  },
+  "scripts": {
+    "build": "tsup && node scripts/postbuild.mjs",
+    "grammars": "node scripts/fetch-grammars.mjs",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "test:e2e": "CODEINDEX_E2E=1 vitest run tests/e2e-real-repos.test.ts",
+    "typecheck": "tsc --noEmit",
+    "check:build": "tsup && node scripts/postbuild.mjs && git diff --exit-code -- scripts/engine.mjs scripts/engine.d.mts scripts/cli.mjs scripts/grammars",
+    "demo": "node scripts/cli.mjs graph --repo tests/fixtures/mini-repo",
+    "release": "semantic-release"
+  },
+  "keywords": [
+    "code-index",
+    "static-analysis",
+    "link-graph",
+    "symbol-extraction",
+    "tree-sitter",
+    "repo-indexing"
+  ],
+  "license": "MIT",
+  "author": "maxgfr",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/maxgfr/codeindex"
+  },
+  "devDependencies": {
+    "@semantic-release/changelog": "^6.0.3",
+    "@semantic-release/exec": "^6.0.3",
+    "@semantic-release/git": "^10.0.1",
+    "@tree-sitter-grammars/tree-sitter-lua": "0.4.1",
+    "@types/node": "^20.14.0",
+    "semantic-release": "^24.2.0",
+    "tree-sitter-bash": "0.25.1",
+    "tree-sitter-c": "^0.24.1",
+    "tree-sitter-c-sharp": "^0.23.5",
+    "tree-sitter-cpp": "^0.23.4",
+    "tree-sitter-go": "^0.25.0",
+    "tree-sitter-java": "^0.23.5",
+    "tree-sitter-javascript": "^0.25.0",
+    "tree-sitter-php": "^0.24.2",
+    "tree-sitter-python": "^0.25.0",
+    "tree-sitter-ruby": "^0.23.1",
+    "tree-sitter-rust": "^0.24.0",
+    "tree-sitter-scala": "0.24.0",
+    "tree-sitter-typescript": "^0.23.2",
+    "tsup": "^8.3.0",
+    "typescript": "^5.5.0",
+    "vitest": "^4.1.0",
+    "web-tree-sitter": "^0.26.10"
+  },
+  "pnpm": {
+    "onlyBuiltDependencies": [
+      "esbuild"
+    ]
+  },
+  "publishConfig": {
+    "access": "public"
+  }
+}

package/scripts/cli.mjs

@@ -0,0 +1,10 @@
+#!/usr/bin/env node
+
+// src/cli-entry.ts
+import { runCli } from "./engine.mjs";
+try {
+  await runCli(process.argv.slice(2));
+} catch (e) {
+  console.error(`codeindex: ${e instanceof Error ? e.message : e}`);
+  process.exit(2);
+}