scip-cli 2.10.0__tar.gz → 2.12.0__tar.gz

{scip_cli-2.10.0 → scip_cli-2.12.0}/.github/workflows/ci.yml

@@ -29,7 +29,7 @@ jobs:
           python -m venv .venv
           source .venv/bin/activate
           pip install -e ".[dev]"
-          npm install -g @sourcegraph/scip-typescript
+          npm install -g github:flesler/scip-typescript#feat/incremental-tsc
           mkdir -p ~/.local/bin
           curl -sL https://github.com/scip-code/scip/releases/download/v0.9.0/scip-linux-amd64.tar.gz | tar xz -C ~/.local/bin
           echo "$VIRTUAL_ENV/bin" >> $GITHUB_PATH

{scip_cli-2.10.0 → scip_cli-2.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: scip-cli
-Version: 2.10.0
+Version: 2.12.0
 Summary: Fast code intelligence via SCIP indexes
 Project-URL: Homepage, https://github.com/flesler/scip-cli
 Author: Ariel Flesler
@@ -252,7 +252,7 @@ Other environment variables:
 |`SCIP_CLI_MAX_DEF_LINES`|Max definition lines in `code` output|
 |`SCIP_CLI_DEBUG`|Log SQL queries to stderr|
 
-**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. Language indexers (`scip-typescript`, `scip-python` via `npx`; `scip-go` via `go install @latest`) install at latest on first use. `rust-analyzer` installs via `rustup component add`.
+**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. `scip-typescript` installs via npx from the GitHub fork in `scip_cli/indexing/constants.py` when not on PATH (until upstream ships incremental flags). `scip-python` via `npx` at latest; `scip-go` via `go install @latest`. `rust-analyzer` installs via `rustup component add`.
 
 Large monorepos (>10 tsconfig projects) log per-project progress to stderr during indexing; smaller repos stay quiet aside from the final `Indexed … (size)` line.
 
@@ -266,6 +266,7 @@ scip-cli reindex --tsconfig tsconfig.app.json --tsconfig tsconfig.spec.json
 scip-cli reindex --exclude '**/*.test.ts' '**/*.spec.ts'
 scip-cli reindex --exclude          # clear persisted excludes
 scip-cli reindex --fresh            # full index; clear metadata.json
+scip-cli reindex --incremental      # reuse unchanged project shards (see SKILL for language support)
 ```
 
 `--path` limits which discovered tsconfig **directories** are indexed (prefix match, same idea as query `--path`). `--tsconfig` skips discovery and indexes those `tsconfig*.json` **files** (repeatable; globs are expanded inside the tool). File-based runs default to one `scip-typescript` process per file so each gets its own heap (`SCIP_CLI_TS_INDEX_BATCH_SIZE` still overrides). Cannot combine `--path` and `--tsconfig`. **TypeScript only.** Scope and exclude defaults are saved in `metadata.json` next to `index.db` and reused on later `reindex` runs; use `reindex --fresh` to clear them and restore a full index. `--exclude` updates the persisted exclude list; a lone bare `--exclude` clears it.

{scip_cli-2.10.0 → scip_cli-2.12.0}/README.md

@@ -231,7 +231,7 @@ Other environment variables:
 |`SCIP_CLI_MAX_DEF_LINES`|Max definition lines in `code` output|
 |`SCIP_CLI_DEBUG`|Log SQL queries to stderr|
 
-**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. Language indexers (`scip-typescript`, `scip-python` via `npx`; `scip-go` via `go install @latest`) install at latest on first use. `rust-analyzer` installs via `rustup component add`.
+**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. `scip-typescript` installs via npx from the GitHub fork in `scip_cli/indexing/constants.py` when not on PATH (until upstream ships incremental flags). `scip-python` via `npx` at latest; `scip-go` via `go install @latest`. `rust-analyzer` installs via `rustup component add`.
 
 Large monorepos (>10 tsconfig projects) log per-project progress to stderr during indexing; smaller repos stay quiet aside from the final `Indexed … (size)` line.
 
@@ -245,6 +245,7 @@ scip-cli reindex --tsconfig tsconfig.app.json --tsconfig tsconfig.spec.json
 scip-cli reindex --exclude '**/*.test.ts' '**/*.spec.ts'
 scip-cli reindex --exclude          # clear persisted excludes
 scip-cli reindex --fresh            # full index; clear metadata.json
+scip-cli reindex --incremental      # reuse unchanged project shards (see SKILL for language support)
 ```
 
 `--path` limits which discovered tsconfig **directories** are indexed (prefix match, same idea as query `--path`). `--tsconfig` skips discovery and indexes those `tsconfig*.json` **files** (repeatable; globs are expanded inside the tool). File-based runs default to one `scip-typescript` process per file so each gets its own heap (`SCIP_CLI_TS_INDEX_BATCH_SIZE` still overrides). Cannot combine `--path` and `--tsconfig`. **TypeScript only.** Scope and exclude defaults are saved in `metadata.json` next to `index.db` and reused on later `reindex` runs; use `reindex --fresh` to clear them and restore a full index. `--exclude` updates the persisted exclude list; a lone bare `--exclude` clears it.

scip_cli-2.12.0/docs/incremental-reindex-roadmap.md

@@ -0,0 +1,45 @@
+# Incremental reindex roadmap
+
+Phased plan to avoid full `scip-typescript` runs on every `reindex`. Shard reuse first; file-level and upstream TSC incremental follow.
+
+## Phase 1 — scip-cli shard reuse (shipped)
+
+Per-tsconfig shard cache with content fingerprints. **Shippable:** unit + integration tests in-repo; large monorepos can be validated manually outside the repo (not checked in).
+
+- [x] Roadmap doc
+- [x] `shards/manifest.json` in cache dir (fingerprint + part DB path per tsconfig project)
+- [x] `compute_shard_fingerprint()` — tsconfig chain + included source file content hashes + exclude globs
+- [x] `reindex --incremental` — per-project shards (batch size 1), reuse unchanged part DBs
+- [x] Persist shard part DBs under `cache/shards/*.db`
+- [x] Full `reindex` clears shard cache (avoid stale reuse after non-incremental rebuild)
+- [x] `reindex --fresh` clears shard cache
+- [x] Unit tests for fingerprint, manifest merge, and warm `index_typescript` reuse
+- [x] Reindex CLI flag tests (`--incremental`, rejects with `--fresh` / Python)
+- [x] Stderr stats: `Incremental: N shard(s) reused, M reindexed`
+- [x] Update `scip_cli/SKILL.md` and README flag list
+
+## Phase 2 — scip-typescript fork: TSC incremental program (in progress)
+
+Fork work lives outside this repo (local clone + PR to `sourcegraph/scip-typescript`).
+
+- [x] Fork `sourcegraph/scip-typescript` (local: implement + tests passing)
+- [x] `--incremental` / `--ts-build-info-dir` CLI flags
+- [x] `createIncrementalProgram` + `createIncrementalCompilerHost` in `ProjectIndexer`
+- [x] scip-cli passes `--incremental --ts-build-info-dir <cache>/tsbuildinfo` on `reindex --incremental` dirty shards
+- [x] npx fallback via `github:flesler/scip-typescript#feat/incremental-tsc` (`SCIP_TYPESCRIPT_NPX_PACKAGE`; switch back when upstream merges)
+- [ ] Upstream PR merged or fork published to npm long-term
+- [ ] Benchmark dirty shard: program build time before/after
+
+## Phase 3 — File-level incremental inside a shard
+
+- [ ] Document-level upsert in SQLite (replace rows for changed paths; merge is insert-only today)
+- [ ] `reindex --incremental` accepts optional changed-file list (content-hash manifest in cache)
+- [ ] Invalidation: changed files ∪ direct importers (from existing index `mentions`)
+- [ ] scip-typescript fork: skip unchanged files in `ProjectIndexer` loop (`--files` or internal dirty set)
+- [ ] Optional git input via env / scip-atlas (`last_index_commit`, `git diff --name-only`) — not a hard scip-cli dependency
+
+## Phase 4 — Polish
+
+- [ ] Dogfood on this repo + optional large-monorepo validation (manual, outside CI)
+- [ ] Submit scip-typescript fork as upstream PR (phase 2, then 3 indexer pieces)
+- [ ] `scripts/bench.sh` scenario for incremental reindex

{scip_cli-2.10.0 → scip_cli-2.12.0}/scip_cli/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: scip-cli
-description: Read when needing symbols, definitions, refs, members, or SQL health dashboards in TS/JS, Python, Go, or Rust
+description: Read when needing symbol lookup or SQL health dashboards in code.
 ---
 
 TypeScript/JavaScript (.ts, .tsx, .js, .jsx), Python (.py), Go (.go), and Rust (.rs) — not GraphQL, CSS, or other files.
@@ -168,3 +168,5 @@ reindex [--path DIR ...] [--tsconfig FILE_OR_GLOB ...] [--exclude [GLOB ...]] [-
 `--exclude GLOB` omits matching files from the SQLite index after conversion (repeatable; merged with `excludeGlobs` in `.scip-cli.json`). Passing `--exclude` updates the persisted exclude list; a lone bare `--exclude` (no globs on that flag) clears it — `--exclude foo --exclude` keeps `foo`. Patterns without `/` match basenames (`*.test.ts`); patterns with `/` match repo-relative paths (`tests/**`, `**/__tests__/**`). Indexers still parse excluded files when production code imports them — post-process removal is authoritative.
 
 `--fresh` ignores persisted `metadata.json` and clears it before indexing unless `--path`, `--tsconfig`, or `--exclude` are set on the same command (use `reindex --fresh` to restore a full index).
+
+`--incremental` reuses cached per-project part DBs under `shards/` in the project cache dir (`~/.cache/scip-cli/projects/<slug>/`, alongside `index.db` and `metadata.json`) when inputs are unchanged. **Currently TypeScript only** (Python reindex rejects the flag). Forces one indexer run per discovered project. On dirty shards passes TSC incremental flags; `.tsbuildinfo` under `tsbuildinfo/` in the same cache dir. When `scip-typescript` is not on PATH, npx installs from the GitHub fork (`SCIP_TYPESCRIPT_NPX_PACKAGE` in `indexing/constants.py`) until upstream ships those flags. A plain `reindex` clears the incremental cache; run `reindex --incremental` after an initial full index to benefit. Cannot combine with `--fresh`.

{scip_cli-2.10.0 → scip_cli-2.12.0}/scip_cli/__init__.py

@@ -1,3 +1,3 @@
 """scip-cli: Fast code intelligence via SCIP indexes."""
 
-__version__ = "2.10.0"
+__version__ = "2.12.0"

{scip_cli-2.10.0 → scip_cli-2.12.0}/scip_cli/__main__.py

@@ -195,6 +195,13 @@ def main() -> None:
         action="store_true",
         help="Ignore persisted metadata.json; clear it unless scope/exclude flags are set on this run",
     )
+    reindex_parser.add_argument(
+        "--incremental",
+        action="store_true",
+        help=(
+            "Reuse cached per-project index parts when inputs are unchanged (full reindex clears the incremental cache)"
+        ),
+    )
     reindex_parser.add_argument(
         "--with-external",
         action="store_true",

{scip_cli-2.10.0 → scip_cli-2.12.0}/scip_cli/commands/reindex.py

@@ -11,6 +11,7 @@ from ..cache import (
     promote_next_index,
 )
 from ..indexing import index_project, log_index_complete
+from ..indexing.shards import clear_shard_cache
 from ..metadata import UNSET, apply_metadata_updates
 from ..paths import normalize_path_scope
 from ..project import Language, find_project_root_and_language
@@ -27,6 +28,13 @@ def main(args):
     tsconfig_args = getattr(args, "tsconfig", None) or []
     exclude_groups = getattr(args, "exclude", None)
     fresh = getattr(args, "fresh", False)
+    incremental = getattr(args, "incremental", False)
+    if incremental and fresh:
+        print("Error: reindex --incremental and --fresh cannot be combined", file=sys.stderr)
+        sys.exit(1)
+    if incremental and lang is not None and lang != Language.TYPESCRIPT:
+        print("Error: reindex --incremental is only supported for TypeScript projects", file=sys.stderr)
+        sys.exit(1)
     if path_args and tsconfig_args:
         print("Error: reindex --path and --tsconfig cannot be combined", file=sys.stderr)
         sys.exit(1)
@@ -92,6 +100,8 @@ def main(args):
             exclude_globs=exclude_update,
         )
         cleanup_in_progress_index(cache_dir)
+        if fresh or not incremental:
+            clear_shard_cache(cache_dir)
         try:
             # Pass --with-external flag to indexer via environment
             if getattr(args, "with_external", False):
@@ -100,7 +110,14 @@ def main(args):
                 os.environ["SCIP_CLI_KEEP_EXTERNAL"] = "1"
 
             started = time.perf_counter()
-            _output_db, skipped, total = index_project(root, lang, cache_dir, replace=True, log=False)
+            _output_db, skipped, total = index_project(
+                root,
+                lang,
+                cache_dir,
+                replace=True,
+                log=False,
+                incremental=incremental,
+            )
             elapsed_seconds = time.perf_counter() - started
         except RuntimeError as e:
             cleanup_in_progress_index(cache_dir)

{scip_cli-2.10.0 → scip_cli-2.12.0}/scip_cli/indexing/constants.py

@@ -7,6 +7,12 @@ SCIP_INSTALL_URL = "https://github.com/scip-code/scip/releases"
 
 # Language indexers (npx / go install) use latest on first use.
 # The scip expt-convert binary is pinned separately in scip_tool.py (DB schema).
+#
+# scip-typescript npx fallback (when the binary is not on PATH). Fork adds
+# --incremental / --ts-build-info-dir for reindex --incremental. Switch back to
+# @sourcegraph/scip-typescript once upstream merges and releases; otherwise
+# publish the fork to npm long-term.
+SCIP_TYPESCRIPT_NPX_PACKAGE = "github:flesler/scip-typescript#feat/incremental-tsc"
 # scip-typescript accepts many tsconfig paths per invocation (one .scip, one convert, no merge).
 # Split only when SCIP_CLI_TS_INDEX_BATCH_SIZE is set (OOM/timeout tuning).
 DEFAULT_TS_INDEX_BATCH_SIZE = None

{scip_cli-2.10.0 → scip_cli-2.12.0}/scip_cli/indexing/core.py

@@ -97,7 +97,7 @@ def indexer_env(project_root=None):
     return env
 
 
-def index_project(root, lang, cache_dir, *, replace=False, log=True):
+def index_project(root, lang, cache_dir, *, replace=False, log=True, incremental=False):
     """Run the language-specific indexer and convert to DB."""
     from ..project import Language
 
@@ -110,7 +110,13 @@ def index_project(root, lang, cache_dir, *, replace=False, log=True):
     if lang == Language.TYPESCRIPT:
         projects = typescript_projects(root)
         output_db, _indexed, skipped, total = index_typescript(
-            root, cache_dir, projects, env, replace=replace, exclude_globs=exclude_globs
+            root,
+            cache_dir,
+            projects,
+            env,
+            replace=replace,
+            exclude_globs=exclude_globs,
+            incremental=incremental,
         )
         if log:
             log_index_complete(

scip_cli-2.12.0/scip_cli/indexing/shards.py

@@ -0,0 +1,201 @@
+"""Per-tsconfig shard fingerprints and cached part DBs for incremental reindex."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import shutil
+from pathlib import Path
+
+from ..exclude import path_matches_glob
+from ..tsconfig import resolved_exclude, resolved_include_or_files, tsconfig_for_project, walk_tsconfig_chain
+
+MANIFEST_VERSION = 1
+MANIFEST_FILENAME = "manifest.json"
+SHARDS_DIRNAME = "shards"
+TSBUILDINFO_DIRNAME = "tsbuildinfo"
+
+_SOURCE_SUFFIXES = frozenset({".ts", ".tsx", ".js", ".jsx", ".mts", ".cts", ".mjs", ".cjs"})
+
+
+def shards_dir(cache_dir: Path) -> Path:
+    return Path(cache_dir) / SHARDS_DIRNAME
+
+
+def ts_build_info_dir(cache_dir: Path) -> Path:
+    """Directory for per-tsconfig .tsbuildinfo files passed to scip-typescript."""
+    return Path(cache_dir) / TSBUILDINFO_DIRNAME
+
+
+def manifest_path(cache_dir: Path) -> Path:
+    return shards_dir(cache_dir) / MANIFEST_FILENAME
+
+
+def shard_key(project: Path) -> str:
+    """Stable manifest key for a TypeScript project path."""
+    return project.as_posix()
+
+
+def shard_db_filename(project: Path) -> str:
+    stem = shard_key(project).replace("/", "__").replace("\\", "__")
+    return f"{stem}.db"
+
+
+def shard_db_path(cache_dir: Path, project: Path) -> Path:
+    return shards_dir(cache_dir) / shard_db_filename(project)
+
+
+def clear_shard_cache(cache_dir: Path) -> None:
+    """Remove persisted shard part DBs and manifest."""
+    root = shards_dir(cache_dir)
+    if root.is_dir():
+        shutil.rmtree(root)
+
+
+def load_shard_manifest(cache_dir: Path) -> dict[str, dict[str, str]]:
+    """Return shard key → {fingerprint, part_db} from manifest.json."""
+    path = manifest_path(cache_dir)
+    if not path.is_file():
+        return {}
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return {}
+    if not isinstance(data, dict):
+        return {}
+    raw_shards = data.get("shards")
+    if not isinstance(raw_shards, dict):
+        return {}
+    out: dict[str, dict[str, str]] = {}
+    for key, entry in raw_shards.items():
+        if not isinstance(key, str) or not isinstance(entry, dict):
+            continue
+        fingerprint = entry.get("fingerprint")
+        part_db = entry.get("part_db")
+        if isinstance(fingerprint, str) and isinstance(part_db, str):
+            out[key] = {"fingerprint": fingerprint, "part_db": part_db}
+    return out
+
+
+def save_shard_manifest(cache_dir: Path, shards: dict[str, dict[str, str]]) -> None:
+    """Merge shard entries into manifest.json (full reindex clears stale entries)."""
+    root = shards_dir(cache_dir)
+    root.mkdir(parents=True, exist_ok=True)
+    merged = {**load_shard_manifest(cache_dir), **shards}
+    payload = {
+        "version": MANIFEST_VERSION,
+        "shards": {
+            key: {"fingerprint": entry["fingerprint"], "part_db": entry["part_db"]}
+            for key, entry in sorted(merged.items())
+        },
+    }
+    manifest_path(cache_dir).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+
+
+def _hash_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def _glob_source_files(base_dir: Path, patterns: list[str]) -> list[Path]:
+    found: set[Path] = set()
+    for pattern in patterns:
+        if Path(pattern).is_absolute():
+            candidate = Path(pattern)
+            if candidate.is_file() and candidate.suffix.lower() in _SOURCE_SUFFIXES:
+                found.add(candidate.resolve())
+            continue
+        for match in base_dir.glob(pattern):
+            if match.is_file() and match.suffix.lower() in _SOURCE_SUFFIXES:
+                found.add(match.resolve())
+    return sorted(found)
+
+
+def _tsconfig_excludes_path(tsconfig: Path, file_path: Path) -> bool:
+    """Return True when a file is excluded by the leaf tsconfig exclude list."""
+    exclude_patterns = resolved_exclude(tsconfig)
+    if not exclude_patterns:
+        return False
+    try:
+        relative = file_path.relative_to(tsconfig.parent).as_posix()
+    except ValueError:
+        return False
+    return any(path_matches_glob(relative, pattern) for pattern in exclude_patterns)
+
+
+def list_project_source_files(root: Path, project: Path) -> list[Path]:
+    """Source files that contribute to a shard fingerprint (tsconfig include/files)."""
+    root = Path(root).resolve()
+    tsconfig = tsconfig_for_project(root, project)
+    if tsconfig is None:
+        return []
+
+    include, files = resolved_include_or_files(tsconfig)
+    base_dir = tsconfig.parent
+    if files is not None:
+        candidates = _glob_source_files(base_dir, files)
+    elif include is not None:
+        candidates = _glob_source_files(base_dir, include)
+    else:
+        candidates = _glob_source_files(base_dir, ["**/*"])
+    return [path for path in candidates if not _tsconfig_excludes_path(tsconfig, path)]
+
+
+def compute_shard_fingerprint(
+    root: Path,
+    project: Path,
+    *,
+    exclude_globs: tuple[str, ...] = (),
+) -> str:
+    """Hash tsconfig chain + indexed sources + exclude globs for one project shard."""
+    root = Path(root).resolve()
+    tsconfig = tsconfig_for_project(root, project)
+    digest = hashlib.sha256()
+    digest.update(shard_key(project).encode())
+    digest.update(b"\0")
+
+    if tsconfig is not None:
+        for path, _data in walk_tsconfig_chain(tsconfig):
+            digest.update(path.as_posix().encode())
+            digest.update(b"\0")
+            digest.update(path.read_bytes())
+            digest.update(b"\0")
+
+    for path in list_project_source_files(root, project):
+        try:
+            relative = path.relative_to(root).as_posix()
+        except ValueError:
+            relative = path.as_posix()
+        digest.update(relative.encode())
+        digest.update(b"\0")
+        digest.update(_hash_file(path).encode())
+        digest.update(b"\0")
+
+    for glob in exclude_globs:
+        digest.update(glob.encode())
+        digest.update(b"\0")
+
+    return digest.hexdigest()
+
+
+def resolve_cached_shard_db(
+    cache_dir: Path,
+    project: Path,
+    fingerprint: str,
+    manifest: dict[str, dict[str, str]],
+) -> Path | None:
+    """Return a reusable shard part DB when fingerprint and file both match."""
+    key = shard_key(project)
+    entry = manifest.get(key)
+    if entry is None or entry.get("fingerprint") != fingerprint:
+        return None
+    candidate = shards_dir(cache_dir) / Path(entry["part_db"]).name
+    if candidate.is_file():
+        return candidate
+    fallback = shard_db_path(cache_dir, project)
+    if fallback.is_file():
+        return fallback
+    return None