codebase-index 1.6.0__py3-none-any.whl

codebase_index/skill_template/SKILL.md

@@ -0,0 +1,198 @@
+---
+name: codebase-index
+description: Use this skill before answering questions about a repository's architecture, implementation locations, symbols, references, dependencies, refactoring impact, data flow, bugs, or where something is implemented. It searches a local hybrid codebase index so Claude reads only the most relevant files instead of scanning the entire project.
+allowed-tools: Bash(python -m codebase_index *), Bash(python3 -m codebase_index *), Bash(codebase-index *), Bash(cbx *), Read, Grep, Glob
+---
+
+# Codebase Index
+
+Use this skill first for codebase questions.
+
+Never scan the entire repository before searching the index.
+
+## When to use
+
+Invoke this skill **before reading any files** when the user asks about this project's code:
+
+- "where is X implemented" / "find X" / "locate the X function"
+- "how does X work" / "explain the X flow"
+- "what breaks if I change X" / "what depends on X" (impact analysis)
+- "who calls X" / "references to X"
+- "trace the data flow of X"
+- "why is this error happening" (error/stack trace)
+- "explain the architecture" / "give me an overview"
+- Any question about symbols, files, dependencies, or refactoring scope
+
+Do **not** use it for: editing files, running the application, or non-code questions.
+
+## How to call the CLI
+
+Use the `codebase-index` CLI directly, or the bundled `cbx` wrapper:
+
+```bash
+codebase-index search "$QUERY" --json
+```
+
+Pick the subcommand by intent:
+
+| User intent | Command |
+|---|---|
+| "how does X work" / "explain X" / "walk me through" | `codebase-index explain "$QUERY" --json` |
+| overview / architecture / "map the codebase" | `codebase-index architecture --json` |
+| general / unsure | `codebase-index search "$QUERY" --json` |
+| keyword / "where is" | `codebase-index search "$QUERY" --json` |
+| a specific symbol name | `codebase-index symbol "<name>" --json` |
+| "who calls / references" | `codebase-index refs "<name>" --json` |
+| "what breaks if I change" | `codebase-index impact "<file-or-symbol>" --json` |
+| "how is X connected to Y" / dependency path | `codebase-index path "<A>" "<B>" --json` |
+| "what is X" / describe a symbol's role | `codebase-index describe "<name>" --json` |
+| visual graph / "open graph" (for the human, not for you to read) | `codebase-index graph "<file-or-symbol>" --open` |
+
+`architecture` returns the codebase map computed at index time — detected modules
+(communities), god nodes (most-connected symbols), surprising cross-module links,
+and suggested questions. Reach for it on "give me an overview" / "where do I
+start" questions instead of a broad `explain`.
+
+`path "A" "B"` returns the shortest dependency/call chain between two symbols or
+files; `describe "X"` returns a node card (definition, callers, callees,
+in/out degree, module, god-node rank). Both annotate edges with a `confidence`
+(`extracted` exact, `inferred` heuristic, `ambiguous` unresolved) — treat a path
+or callee list that leans on `inferred`/`ambiguous` edges as less certain.
+
+The `graph` command renders an HTML dependency graph for a person to look at —
+it is not a retrieval packet. Use it only when the user explicitly wants a visual
+graph; for "what depends on X" answer from `impact`/`refs` instead. In a headless
+session prefer `--output <path>` over `--open`. `--format graphml|dot|neo4j`
+exports the graph for external tools (Gephi/yEd, Graphviz, Neo4j) instead of HTML.
+
+`explain` has a higher default token budget (2200) and HOW_IT_WORKS intent weights — use it whenever the question is about understanding behavior or flow.
+
+For `search`, pick a `--mode` when the intent is clear:
+- `--mode symbol` — pure symbol lookups (faster, no FTS noise)
+- `--mode fts` — text/keyword queries where symbol names don't matter
+- `--mode hybrid` — default; best for mixed queries
+- `--mode vector` — semantic / near-synonym queries ("where do we rate-limit
+  requests" without the exact words). Requires opt-in embeddings; falls back with
+  a clear message when they are not enabled. `hybrid` already blends vectors in
+  when embeddings are on, so reach for `vector` only for pure-semantic recall.
+
+Natural-language kind words such as `method`, `function`, `class`, `interface`,
+`enum`, and `type` constrain the symbol retriever inside `search`.
+
+Use `--json` for programmatic parsing; omit for human-readable output.
+Search/read commands auto-build the index when it is missing; still check
+freshness and run `update`/`index` when responses report stale data.
+
+## Step-by-step workflow
+
+1. **Query the index** using the appropriate subcommand for `$QUERY`.
+2. **Check index freshness** in the response:
+   - `index.exists: false` → run `codebase-index index` first, then re-query.
+   - `index.stale: true`, `files_changed_since_build < 20` → run `codebase-index update`, then re-query.
+   - `index.stale: true`, `files_changed_since_build ≥ 20` → run `codebase-index index` (full rebuild).
+   - Otherwise proceed with results.
+3. **Read ONLY the `recommended_reads`** — use the Read tool with `offset`/`limit` to read the exact line ranges returned. Do not open whole files.
+4. **Answer** with file:line citations (e.g., `src/auth/token.py:88-134`).
+5. **Fallback** only if confidence is low or results are empty (see below).
+
+## Token-budgeted output interpretation
+
+The index returns a **ranked retrieval packet** with:
+
+- `rank` — result position (start with 1-3)
+- `path` — file path
+- `line_start` / `line_end` — exact line range to read
+- `symbols` — symbols found in this range
+- `score` — relevance score
+- `reason` — why this result ranked (e.g., "exact symbol match, 4 callers")
+- `snippet` — compact code excerpt (may already answer the question); `null` means budget was spent — read via `recommended_reads` instead
+- `skeletonized` — when `true`, the `snippet` is a **focus skeleton**: import/signature/class lines and the line(s) matching your query are kept, while function bodies collapse to a marker like `... 24 lines elided (read 88-134)`. Read that line range (or the result's `line_start`/`line_end`) when you need a full body.
+- `elided_lines` — how many source lines the skeleton folded away (`0` when not skeletonized).
+
+Top-level fields:
+
+- `recommended_reads` — the precise `{path, line_start, line_end}` list to open next. This is your read plan.
+- `confidence` — `high` (answer directly), `medium` (read + optionally confirm with one Grep), `low` (use fallback).
+- `fallback_suggestions` — ripgrep patterns and paths to try if the index is weak.
+- `intent` / `mode` — how the query was classified and which retrievers ran;
+  useful to sanity-check a weak result (e.g. a "how does X work" question that
+  resolved to a bare symbol lookup may need `explain` instead).
+- `pagination` — present only when more results exist than fit the page. It
+  reports `has_more` and `next_offset`. To page, re-run `search` with
+  `--offset <next_offset>` (e.g. `search "query" --limit 10 --offset 10`). Prefer
+  refining with a more specific subcommand or raising `--token-budget` first —
+  page only when the top results genuinely miss the answer.
+- `coverage` (on `refs`/`impact` only) — graph-completeness signal. Dependency
+  edges (imports/inheritance) are extracted only for fully supported languages.
+  When `coverage.partial` is `true` (the symbol/file is in a Tier-B language such
+  as Lua), an **empty or short `refs`/`impact` result is inconclusive** — it may
+  just be unanalyzed, not absent. Confirm with a Grep before concluding "nothing
+  references this". `coverage.languages` lists the affected languages.
+
+## Token efficiency rules
+
+- Trust the index. Read the **fewest** files needed — start with rank 1-3 only.
+- Read **line ranges**, not whole files. Use `line_start`/`line_end` with Read's `offset`/`limit`.
+- The `snippet` may already answer the question — re-read only if you need more context.
+- Prefer `search`/`symbol`/`refs`/`impact`/`explain` over manual Grep/Glob — those are expensive fallbacks, not step 1.
+- Don't re-run the query with trivially reworded text; refine with a different subcommand instead.
+- For broad questions (`confidence: low`, architecture, data-flow), raise the budget: `--token-budget 3000`.
+- Test files are demoted in ranking by default. Include "test" in the query to surface them.
+- Snippets are skeletonized by default to fit more results in the budget. The matched line is always preserved; pass `--raw` (CLI) or `raw: true` (MCP) on the rare occasion you need full bodies inline instead of reading the cited line range.
+
+## Fallback behavior
+
+Fall back to built-in search **only** when: results are empty, `confidence` is `low`, or the user asks for something the index clearly doesn't cover.
+
+0. If confidence is consistently low across queries, run diagnostics first:
+   ```bash
+   codebase-index stats --json    # per-language file/symbol counts + graph tier
+   codebase-index doctor          # surface config or security issues
+   ```
+   Low symbol counts for a language may mean the index needs a full rebuild: `codebase-index index`.
+   In `stats`, each language carries `graph: full|partial` (and `doctor` reports a
+   `graph_coverage` finding): `partial` (Tier-B) means `refs`/`impact` lack
+   import/inheritance edges for that language — treat empty results there as
+   inconclusive.
+
+1. Use `fallback_suggestions.ripgrep` patterns from the response via Grep.
+2. If still nothing, Glob for likely paths, then Grep within them.
+3. As a last resort, broaden the search — but tell the user the index was weak here (it may need a rebuild: `codebase-index index`).
+
+Never start with a full-repo scan when the index exists and is fresh.
+
+## Examples
+
+```bash
+# "how does the auth flow work?"
+codebase-index explain "auth flow" --json
+
+# "explain the overall architecture" / "where do I start" — modules, god nodes
+codebase-index architecture --json
+
+# "where is auth token refresh implemented?"
+codebase-index search "auth token refresh" --json
+
+# "what breaks if I change the User model?"
+codebase-index impact "User" --json
+
+# "who calls send_email?"
+codebase-index refs "send_email" --json
+
+# "find the AuthService class"
+codebase-index symbol "AuthService" --json
+
+# precise symbol search (faster, no FTS noise)
+codebase-index search "AuthService" --mode symbol --json
+
+# "how is the API layer connected to the database?"
+codebase-index path "ApiController" "Database" --json
+
+# "what is the Database class and how is it used?"
+codebase-index describe "Database" --json
+
+# generate and open an HTML graph around a file or symbol
+codebase-index graph "User" --direction both --depth 2 --open
+```
+
+Then Read only the returned line ranges and answer with citations.

codebase_index/skill_template/examples/hooks/settings.json

@@ -0,0 +1,16 @@
+{
+  "hooks": {
+    "PostToolUse": [
+      {
+        "matcher": "Edit|Write|MultiEdit",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "codebase-index update --quiet >/dev/null 2>&1 &",
+            "timeout": 5
+          }
+        ]
+      }
+    ]
+  }
+}

codebase_index/skill_template/scripts/cbx

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+# Thin, safe wrapper around the installed `codebase-index` CLI.
+# - Resolves the binary (prefers one on PATH; falls back to `python -m codebase_index`).
+# - Whitelists subcommands so the skill can never invoke destructive ones (clean/init/watch).
+set -euo pipefail
+
+ALLOWED="search explain symbol refs impact graph stats doctor update index"
+
+sub="${1:-}"
+case " $ALLOWED " in
+  *" ${sub} "*) : ;;
+  *)
+    echo "cbx: refusing subcommand '${sub}'. Allowed: ${ALLOWED}" >&2
+    exit 2
+    ;;
+esac
+
+if python -c "import codebase_index" >/dev/null 2>&1; then
+  exec python -m codebase_index "$@"
+elif command -v codebase-index >/dev/null 2>&1; then
+  exec codebase-index "$@"
+else
+  echo "cbx: codebase_index is not importable and codebase-index is not on PATH" >&2
+  exit 127
+fi

codebase_index/skill_template/scripts/cbx.ps1

@@ -0,0 +1,25 @@
+# Windows PowerShell wrapper around the installed `codebase-index` CLI.
+# Mirrors scripts/cbx: whitelists safe subcommands, falls back to `python -m codebase_index`.
+param(
+    [Parameter(Mandatory = $true, Position = 0)]
+    [string]$Subcommand,
+    [Parameter(ValueFromRemainingArguments = $true)]
+    [string[]]$Rest
+)
+
+$ErrorActionPreference = "Stop"
+$allowed = @("search", "explain", "symbol", "refs", "impact", "graph", "stats", "doctor", "update", "index")
+
+if ($allowed -notcontains $Subcommand) {
+    Write-Error "cbx: refusing subcommand '$Subcommand'. Allowed: $($allowed -join ', ')"
+    exit 2
+}
+
+& python -c "import codebase_index" 2>$null
+if ($LASTEXITCODE -eq 0) {
+    & python -m codebase_index $Subcommand @Rest
+    exit $LASTEXITCODE
+}
+$bin = Get-Command codebase-index -ErrorAction SilentlyContinue
+if ($bin) { & $bin.Source $Subcommand @Rest }
+exit $LASTEXITCODE

codebase_index/skill_update.py

@@ -0,0 +1,150 @@
+"""Skill auto-update and rollback helpers.
+
+Auto-update flow:
+  1. On any CLI invocation, compare the installed skill's .skill_version stamp
+     against the running package version.
+  2. If they differ, re-materialize the skill template silently and stamp the new
+     version.  A backup is saved first so the user can roll back.
+
+Manual commands exposed via cli.py:
+  codebase-index skill-update   -- force refresh all/one installed targets
+  codebase-index skill-rollback -- restore the last backup
+"""
+
+from __future__ import annotations
+
+import shutil
+import sys
+from pathlib import Path
+
+VERSION_FILE = ".skill_version"
+_CACHE_BACKUP_REL = ".claude/cache/codebase-index/skill-backups"
+
+
+# ---------------------------------------------------------------------------
+# version helpers
+# ---------------------------------------------------------------------------
+
+def _package_version() -> str:
+    try:
+        from importlib.metadata import version
+        return version("codebase-index")
+    except Exception:
+        return "unknown"
+
+
+def _installed_version(skill_dir: Path) -> str:
+    vf = skill_dir / VERSION_FILE
+    return vf.read_text(encoding="utf-8").strip() if vf.exists() else ""
+
+
+def _write_version(skill_dir: Path, ver: str) -> None:
+    (skill_dir / VERSION_FILE).write_text(ver + "\n", encoding="utf-8")
+
+
+def needs_update(skill_dir: Path) -> bool:
+    """True when the installed skill stamp differs from the running package version."""
+    return _installed_version(skill_dir) != _package_version()
+
+
+# ---------------------------------------------------------------------------
+# backup helpers
+# ---------------------------------------------------------------------------
+
+def _backup_dir(root: Path, target: str) -> Path:
+    """Backup lives in the cache, not next to the skill (avoids polluting skill namespaces)."""
+    return root / _CACHE_BACKUP_REL / target
+
+
+def _make_backup(root: Path, skill_dir: Path, target: str) -> bool:
+    """Copy skill_dir to the cache backup location.  Returns True if a backup was written."""
+    if not skill_dir.exists():
+        return False
+    bak = _backup_dir(root, target)
+    if bak.exists():
+        shutil.rmtree(bak)
+    bak.parent.mkdir(parents=True, exist_ok=True)
+    shutil.copytree(skill_dir, bak)
+    return True
+
+
+# ---------------------------------------------------------------------------
+# public API
+# ---------------------------------------------------------------------------
+
+def update_skill(root: Path, target: str, *, backup: bool = True) -> dict:
+    """Re-materialize the bundled skill template for *target*.
+
+    Returns a result dict:
+      - updated (bool)
+      - backed_up (bool)
+      - target (str)
+      - old_version (str)
+      - new_version (str)
+    """
+    from . import scaffold
+
+    skill_dir = root / scaffold.skill_rel_for_target(target)
+    pkg_ver = _package_version()
+    old_ver = _installed_version(skill_dir)
+
+    backed_up = _make_backup(root, skill_dir, target) if backup else False
+
+    scaffold.materialize_skill(root, force=True, target=target)
+    _write_version(skill_dir, pkg_ver)
+
+    return {
+        "target": target,
+        "old_version": old_ver,
+        "new_version": pkg_ver,
+        "backed_up": backed_up,
+        "updated": True,
+    }
+
+
+def rollback_skill(root: Path, target: str) -> dict:
+    """Restore the backed-up skill for *target*.
+
+    Returns a result dict:
+      - target (str)
+      - rolled_back (bool)
+      - reason (str, only when rolled_back=False)
+    """
+    from . import scaffold
+
+    skill_dir = root / scaffold.skill_rel_for_target(target)
+    bak = _backup_dir(root, target)
+
+    if not bak.exists():
+        return {"target": target, "rolled_back": False, "reason": "no backup found"}
+
+    if skill_dir.exists():
+        shutil.rmtree(skill_dir)
+    shutil.copytree(bak, skill_dir)
+    return {"target": target, "rolled_back": True}
+
+
+def auto_update_if_needed(root: Path, target: str) -> bool:
+    """Silently update *target* skill if the installed version is outdated.
+
+    Returns True when an update was applied.  Never raises — failures are swallowed
+    because a broken auto-update must never crash the user's real command.
+    """
+    try:
+        from . import scaffold
+
+        skill_dir = root / scaffold.skill_rel_for_target(target)
+        if not skill_dir.exists():
+            return False
+        if not needs_update(skill_dir):
+            return False
+
+        update_skill(root, target, backup=True)
+        return True
+    except Exception as exc:
+        print(
+            f"[codebase-index] skill auto-update for '{target}' failed "
+            f"({type(exc).__name__}: {exc}); run `codebase-index skill-update`.",
+            file=sys.stderr,
+        )
+        return False

codebase_index/storage/__init__.py

@@ -0,0 +1,8 @@
+"""SQLite persistence layer.
+
+db.py     : connection management, pragmas, applying schema.sql, migrations gated on
+            meta.schema_version.
+schema.sql: canonical DDL (mirrors docs/SCHEMA.md).
+repo.py   : typed accessors (upsert_file, replace_chunks, insert_symbols, insert_edges, FTS query,
+            vector query, freshness read). All SQL lives here — no raw SQL elsewhere.
+"""

codebase_index/storage/db.py

@@ -0,0 +1,116 @@
+"""SQLite connection management: pragmas, schema application, version guard."""
+
+from __future__ import annotations
+
+import sqlite3
+from importlib import resources
+from pathlib import Path
+from typing import Optional
+
+# 2: chunks gained a denormalized `symbol_names` column (FTS symbol-name boost).
+# 3: edges gained a `confidence` column (extracted/inferred/ambiguous audit trail).
+SCHEMA_VERSION = 3
+
+
+class Database:
+    """Own one SQLite connection for one CLI invocation."""
+
+    def __init__(self, path: Path | str) -> None:
+        self.path = Path(path)
+        self._conn: Optional[sqlite3.Connection] = None
+
+    def open(self) -> "Database":
+        self.path.parent.mkdir(parents=True, exist_ok=True)
+        self._conn = sqlite3.connect(self.path)
+        self._conn.row_factory = sqlite3.Row
+        self._apply_pragmas()
+        self._apply_schema()
+        self._guard_version()
+        return self
+
+    def close(self) -> None:
+        if self._conn is not None:
+            self._conn.commit()
+            self._conn.close()
+            self._conn = None
+
+    def __enter__(self) -> "Database":
+        return self.open()
+
+    def __exit__(self, *exc: object) -> None:
+        self.close()
+
+    @property
+    def conn(self) -> sqlite3.Connection:
+        if self._conn is None:
+            raise RuntimeError("Database is not open")
+        return self._conn
+
+    def get_schema_version(self) -> int:
+        row = self.conn.execute("SELECT value FROM meta WHERE key = 'schema_version'").fetchone()
+        return int(row[0]) if row else 0
+
+    def _apply_pragmas(self) -> None:
+        self.conn.execute("PRAGMA journal_mode = WAL")
+        self.conn.execute("PRAGMA synchronous = NORMAL")
+        self.conn.execute("PRAGMA foreign_keys = ON")
+        self.conn.execute("PRAGMA temp_store = MEMORY")
+
+    def _apply_schema(self) -> None:
+        ddl = resources.files("codebase_index.storage").joinpath("schema.sql").read_text(
+            encoding="utf-8"
+        )
+        self.conn.executescript(ddl)
+
+    def _guard_version(self) -> None:
+        current = self.get_schema_version()
+        if current == 0:
+            self.conn.execute(
+                "INSERT OR REPLACE INTO meta(key, value) VALUES ('schema_version', ?)",
+                (str(SCHEMA_VERSION),),
+            )
+            self.conn.commit()
+        elif current > SCHEMA_VERSION:
+            raise RuntimeError(
+                f"Index schema_version {current} is newer than supported {SCHEMA_VERSION}; "
+                "rebuild the index with an updated CLI."
+            )
+        # current < SCHEMA_VERSION is tolerated on open: queries never read the
+        # added columns, so an older index is still safely *readable*. The build
+        # commands (index/update) detect the mismatch via peek_schema_version and
+        # rebuild from scratch, since there is no in-place migration framework and
+        # schema.sql is applied with IF NOT EXISTS (old tables/triggers persist).
+
+    def enable_vectors(self) -> None:
+        """Load the sqlite-vec extension into this connection (optional extra)."""
+        try:
+            import sqlite_vec  # type: ignore[import-untyped]
+        except ImportError as exc:
+            raise RuntimeError(
+                "Vector search needs the optional extra: pip install codebase-index[embeddings]"
+            ) from exc
+        self.conn.enable_load_extension(True)
+        sqlite_vec.load(self.conn)
+        self.conn.enable_load_extension(False)
+
+
+def peek_schema_version(path: Path | str) -> int:
+    """Read meta.schema_version without applying schema or running the guard.
+
+    Returns 0 when the file, the meta table, or the key is absent/unreadable, so
+    callers can treat "0 < peek < SCHEMA_VERSION" (or a missing meta) as "rebuild".
+    """
+    p = Path(path)
+    if not p.exists():
+        return 0
+    try:
+        conn = sqlite3.connect(p)
+        try:
+            row = conn.execute(
+                "SELECT value FROM meta WHERE key = 'schema_version'"
+            ).fetchone()
+            return int(row[0]) if row else 0
+        finally:
+            conn.close()
+    except (sqlite3.Error, ValueError, OSError):
+        return 0