scip-cli 2.11.0__tar.gz → 3.0.0__tar.gz

{scip_cli-2.11.0 → scip_cli-3.0.0}/.github/workflows/ci.yml

@@ -29,7 +29,13 @@ jobs:
           python -m venv .venv
           source .venv/bin/activate
           pip install -e ".[dev]"
-          npm install -g @sourcegraph/scip-typescript
+          # GitHub npx spec runs prepare (needs pnpm for snapshot fixtures); CI installs a built global binary instead.
+          git clone --depth 1 --branch feat/partial-files https://github.com/flesler/scip-typescript.git /tmp/scip-typescript
+          cd /tmp/scip-typescript
+          npm install --ignore-scripts
+          npm run build
+          npm install -g .
+          cd -
           mkdir -p ~/.local/bin
           curl -sL https://github.com/scip-code/scip/releases/download/v0.9.0/scip-linux-amd64.tar.gz | tar xz -C ~/.local/bin
           echo "$VIRTUAL_ENV/bin" >> $GITHUB_PATH

{scip_cli-2.11.0 → scip_cli-3.0.0}/.gitignore

@@ -49,5 +49,10 @@ dogfood.md
 
 # Local-only dev scripts (repo-specific smoke harnesses)
 scripts/deep-smoke.sh
+scripts/smoke.local.json
 tmp/
 *.db
+node_modules/
+
+# Generated when indexing the incremental-bench fixture (sources are .ts)
+tests/fixtures/incremental-bench/**/*.js

{scip_cli-2.11.0 → scip_cli-3.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: scip-cli
-Version: 2.11.0
+Version: 3.0.0
 Summary: Fast code intelligence via SCIP indexes
 Project-URL: Homepage, https://github.com/flesler/scip-cli
 Author: Ariel Flesler
@@ -252,7 +252,7 @@ Other environment variables:
 |`SCIP_CLI_MAX_DEF_LINES`|Max definition lines in `code` output|
 |`SCIP_CLI_DEBUG`|Log SQL queries to stderr|
 
-**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. Language indexers (`scip-typescript`, `scip-python` via `npx`; `scip-go` via `go install @latest`) install at latest on first use. `rust-analyzer` installs via `rustup component add`.
+**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. `scip-typescript` installs via npx from the GitHub fork in `scip_cli/indexing/constants.py` when not on PATH (until upstream ships incremental flags). `scip-python` via `npx` at latest; `scip-go` via `go install @latest`. `rust-analyzer` installs via `rustup component add`.
 
 Large monorepos (>10 tsconfig projects) log per-project progress to stderr during indexing; smaller repos stay quiet aside from the final `Indexed … (size)` line.
 

{scip_cli-2.11.0 → scip_cli-3.0.0}/README.md

@@ -231,7 +231,7 @@ Other environment variables:
 |`SCIP_CLI_MAX_DEF_LINES`|Max definition lines in `code` output|
 |`SCIP_CLI_DEBUG`|Log SQL queries to stderr|
 
-**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. Language indexers (`scip-typescript`, `scip-python` via `npx`; `scip-go` via `go install @latest`) install at latest on first use. `rust-analyzer` installs via `rustup component add`.
+**Version policy:** only the `scip` converter (`expt-convert`) is pinned to the 0.8.x release line because it defines the SQLite schema. `scip-typescript` installs via npx from the GitHub fork in `scip_cli/indexing/constants.py` when not on PATH (until upstream ships incremental flags). `scip-python` via `npx` at latest; `scip-go` via `go install @latest`. `rust-analyzer` installs via `rustup component add`.
 
 Large monorepos (>10 tsconfig projects) log per-project progress to stderr during indexing; smaller repos stay quiet aside from the final `Indexed … (size)` line.
 

scip_cli-3.0.0/docs/benchmarks.md

@@ -0,0 +1,188 @@
+# Incremental reindex benchmarks
+
+Results from benchmarking `scip-cli reindex --incremental` on ephemeral fixtures.  
+**Do not name customer repos, paths, or symbols in this file.** Large-monorepo numbers use a local tmpfs checkout (16 shard tsconfigs, ~14k source files, ~90 MB index).
+
+Related: [incremental-reindex-roadmap.md](incremental-reindex-roadmap.md) (phases and design).
+
+**Fork:** `github:flesler/scip-typescript#feat/partial-files` (`SCIP_TYPESCRIPT_NPX_PACKAGE`).  
+**Upstream:** https://github.com/sourcegraph/scip-typescript — PR to upstream **pending** (not opened).  
+Local dev: `npm link` from a clone of that branch (>= `ce6e9c9`).
+
+---
+
+## Progress log (newest first)
+
+| Date | Change | Result |
+|------|--------|--------|
+| 2026-09-25 | **Git-only incremental** (drop hashes/stat/mtime; `--incremental` requires git) | Shard skip = git delta + `tsconfig_digest`; warm git-delta scan **~0.04s** on 4-shard fixture |
+| 2026-09-25 | **Fork squashed for upstream PR** (`ce6e9c9`, `--files` only on `sourcegraph/main`) | One commit; TSC `--incremental` dropped from fork branch |
+| 2026-09-25 | **Large monorepo cold discovery** (16 shards, ~21k source paths, no node_modules) | **Git ls-files ~12.6s** vs **skip-dir glob ~26.6s**; warm incremental **~0.2s** 16/0 |
+| 2026-09-25 | **Git-aware + skip-dir discovery** (`FINGERPRINT_DISCOVERY=auto`) | Old `**/*` glob walked `node_modules` (~4s); skip-dir glob skips `SKIP_DIR_NAMES` |
+| 2026-09-25 | **Fingerprint path cache** (`file_stats` + `tsconfig_digest`, stat fast-path) | Warm **4.1s → 0.2s**; modify_large **~7s → ~3s**; glob was 4s, hash reads 0.14s |
+| 2026-09-25 | **`PARTIAL_REINDEX_MAX_RATIO=1.0`** (disable full-shard fallback) | modify_large partial ~3s; forced full (`ratio=0.002`) ~100s |
+| 2026-09-25 | **mtime rejected** (re-benchmarked) | Fingerprint-only ~2% faster than hash; warm **136s** when mtimes retouched |
+| 2026-09-25 | **Fork `a985dfc`:** map Commander `options.files` → `indexFiles` (CLI `--files` was ignored) | `scip_typescript` **8.5s → 1.7s**; modify_large **~15s → ~7s** |
+| 2026-09-25 | **Fork `22afafb`:** `--files` paths as `createProgram` rootNames (not full shard list) | Program sources **1760 → 662**; `createProgram` ~2s → ~1s (minor alone; matters with fix above) |
+| 2026-09-25 | **scip-cli:** direct-only incremental path; in-place `index.db` upsert | Warm **~4.1s** before fingerprint cache |
+| 2026-09-25 | Postprocess skip on partial part DBs (`skip_postprocess` when `index_files` set) | modify_large **~105s → ~10s** (pre-`--files` fix) |
+| Earlier | Phase 1 shard reuse | Warm **~9s** vs full **~117s** |
+
+### Fork commit (feat/partial-files)
+
+Single squashed commit for upstream PR (`ce6e9c9` on fork; rebased onto `sourcegraph/main`):
+
+| Piece | What |
+|-------|------|
+| `--files` CLI | Repeatable repo-relative paths; Commander `.files` → `.indexFiles` |
+| `resolveIndexRootNames()` | Only `--files` entries passed as `createProgram` rootNames |
+| Partial index loop | FileIndexer runs on subset; stderr logs loaded vs total project sources |
+
+Stderr when partial index works:
+
+```text
+partial index: 4 document(s), 662/1760 project sources in program
++ apps/api/tsconfig.shard-03.json (139ms)
+```
+
+---
+
+## Current architecture (settled)
+
+- **Git incremental:** `reindex --incremental` in a git repo stores `manifest.git_commit` + per-shard `tsconfig_digest`. Shard skip when git delta empty for shard. Partial: git delta → importer closure → `--files` → upsert into live `index.db`.
+- **Full reindex:** plain `reindex`, `--unversioned`, or non-git (no incremental).
+- **Env:** `SCIP_CLI_FILE_INCREMENTAL=1`. Profiling: `SCIP_CLI_INDEX_TIMING=1`.
+
+---
+
+## Current performance (large monorepo smoke, git-only + fork `ce6e9c9`)
+
+Numbers from a private local checkout — not reproducible from the public repo alone. Copy `scripts/smoke.local.example.json` to a gitignored `smoke.local.json` beside it and set `root` to your monorepo.
+
+| Scenario | Wall | Dominant phases |
+|----------|------|-----------------|
+| **Warm 16/0** | **~0.2s** | `git_delta_scan` — git diff + per-shard tsconfig digest compare |
+| **modify_large 15/1** (partial 4/1760) | **~3s** | `scip_typescript` ~1.7s + upsert ~0.1s |
+| **seed_cold** | ~125s | 16 shards full index |
+
+Example `INDEX_TIMING:summary` (modify_large):
+
+```text
+INDEX_TIMING:summary git_delta_scan=12ms parallel_index=2792ms \
+  scip_typescript=1778ms scip_convert=61ms apply_updates=101ms \
+  shards_reused=15 shards_dirty=1 postprocess_skipped=1 \
+  notes=apps/api/tsconfig.shard-03.json:partial 4/1760 dirty=1
+```
+
+### Where time goes (modify_large)
+
+| Phase | ~ms | Notes |
+|-------|-----|-------|
+| `git_delta_scan` | **~10–50** | Git subprocess + tsconfig filter per shard (no file content reads) |
+| `scip_typescript` | 1700 | Was **8500** before `--files` CLI fix |
+| `scip_convert` | 60 | Small partial `.scip` |
+| `apply_updates` | 100 | Targeted upsert into large `index.db` |
+
+## How we benchmark
+
+### Gate runner
+
+```bash
+cp scripts/smoke.local.example.json scripts/smoke.local.json
+# edit root + touch_large for your local monorepo checkout
+
+scripts/bench_incremental_gate.sh --branch smoke    # warm + modify_large
+scripts/bench_incremental_gate.sh --branch fixture  # mini fixture (no local config)
+```
+
+**Monorepo smoke:** `scripts/smoke.local.json` (gitignored) points at a local checkout with persisted scope (`metadata.json` lists shard tsconfigs).
+
+**Scenarios:** `seed_cold`, `seed_warm`, `warm` (16/0), `modify_large` (touch configured `touch_large` path).
+
+**Rules:**
+
+- **One benchmark at a time** — no parallel gate + manual runs on the smoke checkout.
+- Background runs OK; **wait for exit** before starting another or reporting numbers.
+- `SCIP_CLI_INDEX_TIMING=1` for per-phase breakdown.
+
+Gate artifacts and baselines are written under the repo's gitignored benchmark directory (see `scripts/bench_incremental_gate.py`).
+
+### Other
+
+| Tool | Purpose |
+|------|---------|
+| `scripts/bench_incremental.sh` | Smaller fixture loops |
+| `scripts/bench_fingerprint_smoke.py` | monorepo warm incremental + git-delta scan timing |
+| `SCIP_CLI_INDEX_TIMING=1` | `INDEX_TIMING:<phase>=Nms` on stderr |
+
+### Tests
+
+`tests/test_incremental_equivalence.py` — incremental fixture flows (real `scip-typescript` via npx fork).
+
+---
+
+## Historical results (superseded)
+
+<details>
+<summary>Gate branches before direct-only + fork fixes (2026-09-25 morning)</summary>
+
+### fingerprint
+
+| Scenario | hash | mtime |
+|----------|------|-------|
+| warm | 9.2s / 16/0 | 25.9s / 15/1 |
+
+### phase3
+
+| modify_large | FILE_INCREMENTAL=0 | =1 |
+|--------------|-------------------|-----|
+| | 104s | 26s* |
+
+\*Postprocess-skip + broken `--files` CLI masked true partial SCIP time.
+
+### direct vs legacy (shard DB + merge)
+
+| Scenario | legacy | direct |
+|----------|--------|--------|
+| warm | 4.88s | 4.39s |
+| modify_large | 15.15s | 15.88s |
+
+Direct won warm + disk; modify_large looked slower due to SCIP variance — actually `--files` was not applied.
+
+</details>
+
+---
+
+## Improvements (chronological)
+
+1. **Phase 1 — shard reuse** — warm ~9s vs full ~117s.
+2. **Phase 2 — TSC incremental** — no win; off by default.
+3. **Phase 3 — `--files` + upsert** — designed for partial shard reindex (CLI wiring broken until fork `a985dfc`).
+4. **Postprocess skip** — skip symbol trim on partial part DBs (`skip_postprocess` in convert).
+5. **Direct-only incremental** — manifest + in-place upsert; no shard `.db` files (~40 MB cache saved).
+6. **Fork rootNames** — smaller TypeScript program when `--files` set (example shard: 1760 → 662 sources).
+7. **Fork `--files` CLI fix** — **largest win:** indexer ran on all shard files despite `--files` flag.
+8. **Fingerprint path cache** — warm ~4s → ~0.2s (superseded by git-only incremental).
+9. **Git-only incremental** — manifest stores `git_commit` + `tsconfig_digest` only; no per-file hashes.
+
+---
+
+## Known limitations / next work
+
+1. **TS program closure** — 4 `--files` paths may still pull hundreds of sources into the TypeScript program (fork).
+2. **Importer closure** — scip-cli expands 1 dirty file to N paths via SQLite (`expand_reindex_paths`); distinct from TS program closure.
+3. **Further fork ideas** — shrink checker work on non-indexed files (hard).
+
+---
+
+## Reproduce (fixture only — no customer paths)
+
+```bash
+pip install -e ".[dev]"
+npm link   # from scip-typescript clone on feat/partial-files (commit >= ce6e9c9)
+
+pytest tests/test_incremental_equivalence.py
+scripts/bench_incremental_gate.sh --branch fixture
+```
+
+For a large monorepo, copy `scripts/smoke.local.example.json` to `scripts/smoke.local.json` and run the smoke gate locally.

scip_cli-3.0.0/docs/incremental-reindex-roadmap.md

@@ -0,0 +1,76 @@
+# Incremental reindex (settled)
+
+## Model
+
+Two modes only:
+
+| Mode | Trigger | Behaviour |
+|------|---------|-----------|
+| **Git incremental** | `reindex --incremental` in a git repo (not `--unversioned`) | Manifest stores `git_commit` + per-shard `tsconfig_digest`. Shard skip when git delta empty for shard. Partial reindex via `--files` on dirty paths ∪ importers. Upsert into live `index.db`. |
+| **Full reindex** | Plain `reindex`, `--unversioned`, or non-git | No incremental. Clears shard manifest (unless `--incremental` on same run). Full indexer per shard. |
+
+**Removed:** content hashes, stat fast-path, `SCIP_CLI_FINGERPRINT=*`, `SCIP_CLI_FINGERPRINT_DISCOVERY=*`, `SCIP_CLI_FINGERPRINT_CACHE`, manifest `file_hashes` / `fingerprint`.
+
+## Git delta (change detection)
+
+Since `manifest.git_commit`:
+
+1. `git diff <commit>..HEAD` (committed)
+2. `git diff` (unstaged)
+3. `git diff --cached` (staged)
+4. `git ls-files --others --exclude-standard` (untracked)
+
+`git mv` is treated as **delete old path + add new path** (implementation disables git rename folding so the old path lands in the removal set).
+
+Deletions flow to `remove_documents_by_paths`. Modifications/additions filtered through tsconfig include/exclude, then importer closure for partial scope.
+
+## Manifest v4
+
+```json
+{
+  "version": 4,
+  "git_commit": "<HEAD at last successful index>",
+  "shards": {
+    "packages/a/tsconfig.json": {
+      "tsconfig_digest": "<hex sha256 — see below>"
+    }
+  }
+}
+```
+
+### `tsconfig_digest`
+
+Per-shard **SHA-256 hex** of the tsconfig *chain* (not source files). Built in `tsconfig_chain_digest()`:
+
+1. Shard key (`packages/a/tsconfig.json` posix path) + `\0`
+2. For each file in `extends` chain (root tsconfig first, then parents), in order:
+   - absolute path string + `\0`
+   - raw file bytes + `\0`
+
+Any change to the shard tsconfig or an extended parent (include/exclude/files, compiler options, path) changes the digest → **full shard reindex** even when git reports no file changes.
+
+Shard skip: `tsconfig_digest` unchanged **and** no paths in git delta intersect this shard (via tsconfig include/exclude rules).
+
+## Env
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `SCIP_CLI_FILE_INCREMENTAL` | `1` | Partial `--files` reindex when git delta is small |
+
+## Benchmarks
+
+See [benchmarks.md](benchmarks.md).
+
+```bash
+scripts/bench_incremental_gate.sh --branch fixture
+scripts/bench_fingerprint_smoke.py   # requires local smoke.local.json
+SCIP_CLI_INDEX_TIMING=1 scip-cli reindex --incremental
+```
+
+## Upstream (scip-typescript)
+
+| | |
+|---|---|
+| **Upstream** | https://github.com/sourcegraph/scip-typescript |
+| **Fork** | https://github.com/flesler/scip-typescript (`feat/partial-files`, `--files` only) |
+| **Pin** | `ce6e9c9` on `feat/partial-files` |

scip_cli-3.0.0/scip_cli/SKILL.md

@@ -0,0 +1,135 @@
+---
+name: scip-cli
+description: Read when needing symbol lookup or SQL health dashboards in code.
+---
+
+TypeScript/JavaScript, Python, Go, Rust — not GraphQL, CSS, or other files. Run from project root.
+
+**Contributors:** `pip install -e ".[dev]"` so bare `scip-cli` on PATH is this checkout.
+
+## Quick guide
+
+|Question|Command|Output|
+|---|---|---|
+|Definition / body|`code X`|Snippet (default 80 lines/def; `members Class` for big types)|
+|Usages|`refs X`|`file:line` refs|
+|Symbols in file|`symbols file`|`start-end kind name` (bare filename ok)|
+|Find by name|`search pattern`|Matching symbols|
+|Who imports this file?|`rdeps file`|Importer paths|
+|What does this use?|`deps target`|Outbound deps (symbol or file)|
+|Class members|`members Class`|Methods/fields + line ranges|
+|Health / risk|`analyze`|SQL dashboard — project, dir, file, or symbol|
+
+Default `--limit` is **10** on query commands unless noted. Stdout = records; stderr = warnings/progress. Pipe with `--paths-only` / `--names-only`.
+
+## Gotchas
+
+- **Bare names** — functions, types, classes. Members need qualifiers: `code Widget.run`, `members pkg.MyClass`. Consts/vars are not indexed — use `rg`. Class methods: `members Class`, not bare `code method`.
+- **Ambiguity** — `code`/`refs` return up to `--limit` matches; `members`/`analyze` pick first with stderr warning. Narrow with dotted names or query `--path`.
+- **Stale index** — run `reindex` after substantive edits (no auto-invalidation).
+- **Query `--path` ≠ reindex scope** — query `--path` filters results only. `reindex --path` / `--tsconfig` persist TS scope and replace cache; full `reindex` restores.
+- **Monorepos** — walks `tsconfig*.json` (skips `node_modules`, `.git`). Dedupes nested projects. Extra roots: `.scip-cli.json` (README). Query scope: `--path packages/api`.
+- **allowJs** — when tsconfig `allowJs: true`, matching `.js`/`.jsx` under `include`/`files` are indexed. No root `tsconfig.json`: `--infer-tsconfig`.
+- **First run** may auto-index (parallel by default, `SCIP_CLI_INDEX_WORKERS`). >10 tsconfigs log progress. Prerequisites: Node (TS/Python), Go, or Rust toolchain; `scip` converter auto-downloads. `brew install scip` is the wrong package.
+
+## Pipelines
+
+|Goal|Pipeline|
+|---|---|
+|Blast radius|`scip-cli rdeps file.ts \|xargs -I{} scip-cli symbols {}`|
+|Health briefing|`scip-cli analyze --limit 40 --per-check-limit 5`|
+|Importer files|`scip-cli refs Foo --paths-only`|
+|Symbols in importers|`scip-cli refs Foo --paths-only \|xargs -I{} scip-cli symbols {}`|
+|Classes → members|`scip-cli search Handler --kind class --names-only \|xargs -I{} scip-cli members {}`|
+|Members → defs|`scip-cli members Widget --names-only \|xargs -I{} scip-cli code Widget.{}`|
+|Topic files|`scip-cli search Dynamo --paths-only`|
+|Outbound files|`scip-cli deps file.ts --paths-only`|
+
+`xargs` reopens the index each call — cap fan-out with `--limit` on the first command.
+
+## Commands
+
+### code
+
+```bash
+code [--kind KIND] [--limit N] [--max-lines N] [--offset N] [--full] [--path PATH] [--snippet] [--line-numbers] <symbol> [...]
+```
+
+`--kind`: `function`, `method`, `class`, `property`. Multiple symbols: each block prefixed on stdout. `--max-lines` per body (default 80; also 32k char cap unless `--full` or `--max-lines 0`). `--offset` is body-relative. `SCIP_CLI_MAX_DEF_LINES` overrides cap.
+
+### refs
+
+```bash
+refs [--limit N] [--path PATH] [--paths-only] <symbol> [...]
+```
+
+`file:line` per ref (reads source for lines). Multi-symbol groups prefixed on stdout.
+
+### search
+
+```bash
+search [--kind KIND] [--limit N] [--path PATH] [--names-only] [--paths-only] <pattern> [...]
+```
+
+`file:line kind name`. Patterns OR'd. Filters noisy symbols.
+
+### symbols
+
+```bash
+symbols [--limit N] [--path PATH] <file>
+```
+
+### rdeps
+
+```bash
+rdeps [--limit N] [--path PATH] <file>
+```
+
+Bare importer paths on stdout.
+
+### deps
+
+```bash
+deps [--limit N] [--path PATH] [--paths-only] <symbol|file>
+```
+
+Symbol: deps inside definition range. File: external symbols referenced. Format `file:line  symbol`; `--paths-only` dedupes files.
+
+### members
+
+```bash
+members [--limit N] [--path PATH] [--names-only] <symbol>
+```
+
+`start:end kind name`. Prefix match under parent; missing ranges filled from source when needed.
+
+### analyze
+
+```bash
+analyze [--limit N] [--per-check-limit N] [--path PATH] [--include-tests] [--priority LEVEL] [--check NAME] [target]
+```
+
+|Target|Scope|
+|---|---|
+|_(omit)_|Project dashboards|
+|directory|Dir dashboards + per-file sections (indexed files under dir)|
+|file|File dashboards + top symbols by external consumers|
+|symbol|Pressure, consumers, deps, affected|
+
+Priority **high → medium → low**. `--limit` = total finding rows (default 20); headers/truncation don't count. `--per-check-limit` caps each section. Agents: `--limit 40 --per-check-limit 5`. `--priority high` or `high,medium` (`1`/`2`/`3`) skips lower tiers. `--check NAME` (repeatable/comma) ANDs with priority.
+
+**Easy wins:** cycles, dead exports, dead files (production). Stale types = types with no external refs. Dead = no cross-file refs in index, not `vulture`. Empty `rdeps` on a hit is often a SCIP miss — verify with `rg -F STEM` (basename in `(...)`, same-file hits ignored).
+
+### reindex
+
+```bash
+reindex [--path DIR ...] [--tsconfig GLOB ...] [--exclude [GLOB ...]] [--fresh] [--incremental] [--unversioned] [--with-external]
+```
+
+`--path` and `--tsconfig` are TS-only and mutually exclusive. `--tsconfig` globs expanded in-tool; one heap per file by default (`SCIP_CLI_TS_INDEX_BATCH_SIZE` overrides). Scope/exclude persist in `metadata.json`.
+
+`--exclude GLOB` drops matching docs post-convert (repeatable; merges `.scip-cli.json` `excludeGlobs`). Bare `--exclude` clears list; `--exclude foo --exclude` keeps `foo`. No `/` = basename; with `/` = repo-relative.
+
+`--fresh` clears persisted metadata (unless combined with scope flags on same command).
+
+`--incremental` — git repo required; errors with `--unversioned` or non-git. Reuses `shards/manifest.json` (`git_commit` + per-shard `tsconfig_digest`); skips clean shards. Dirty: git delta ∪ importer closure → fork `--files` → upsert live `index.db`; deletions remove documents. `SCIP_CLI_FILE_INCREMENTAL=0|1` (default on). `--unversioned` forces glob/full discovery (persisted). Plain `reindex` clears manifest. No `--incremental` with `--fresh`.

{scip_cli-2.11.0 → scip_cli-3.0.0}/scip_cli/__init__.py

@@ -1,3 +1,3 @@
 """scip-cli: Fast code intelligence via SCIP indexes."""
 
-__version__ = "2.11.0"
+__version__ = "3.0.0"

{scip_cli-2.11.0 → scip_cli-3.0.0}/scip_cli/__main__.py

@@ -202,6 +202,14 @@ def main() -> None:
             "Reuse cached per-project index parts when inputs are unchanged (full reindex clears the incremental cache)"
         ),
     )
+    reindex_parser.add_argument(
+        "--unversioned",
+        action="store_true",
+        help=(
+            "Discover sources via on-disk glob instead of git (persisted in metadata.json; "
+            "for tarballs or when gitignored files must be indexed). Cleared by reindex --fresh"
+        ),
+    )
     reindex_parser.add_argument(
         "--with-external",
         action="store_true",

{scip_cli-2.11.0 → scip_cli-3.0.0}/scip_cli/commands/reindex.py

@@ -12,9 +12,11 @@ from ..cache import (
 )
 from ..indexing import index_project, log_index_complete
 from ..indexing.shards import clear_shard_cache
-from ..metadata import UNSET, apply_metadata_updates
+from ..indexing.source_discovery import is_versioned_repo
+from ..metadata import UNSET, apply_metadata_updates, index_unversioned
 from ..paths import normalize_path_scope
 from ..project import Language, find_project_root_and_language
+from ..sql import finalize_index_db
 from ..tsconfig import expand_tsconfig_patterns
 
 
@@ -29,12 +31,22 @@ def main(args):
     exclude_groups = getattr(args, "exclude", None)
     fresh = getattr(args, "fresh", False)
     incremental = getattr(args, "incremental", False)
+    unversioned = getattr(args, "unversioned", False)
     if incremental and fresh:
         print("Error: reindex --incremental and --fresh cannot be combined", file=sys.stderr)
         sys.exit(1)
     if incremental and lang is not None and lang != Language.TYPESCRIPT:
         print("Error: reindex --incremental is only supported for TypeScript projects", file=sys.stderr)
         sys.exit(1)
+    if incremental and (index_unversioned(root) or not is_versioned_repo(root)):
+        print(
+            (
+                "Error: reindex --incremental requires a git repository "
+                "(use full reindex without --incremental for --unversioned or non-git projects)"
+            ),
+            file=sys.stderr,
+        )
+        sys.exit(1)
     if path_args and tsconfig_args:
         print("Error: reindex --path and --tsconfig cannot be combined", file=sys.stderr)
         sys.exit(1)
@@ -52,14 +64,11 @@ def main(args):
             sys.exit(1)
         scope_paths = [path.as_posix() for path in tsconfig_paths]
         scope_update = scope_paths
-        print(f"Index scope: {', '.join(scope_paths)}", file=sys.stderr)
-        print(
-            (
-                "Warning: scoped reindex replaces the cache with only these tsconfig files; "
-                "run reindex --fresh to restore the full index"
-            ),
-            file=sys.stderr,
-        )
+        if len(scope_paths) == 1:
+            print(f"Index scope: {scope_paths[0]}", file=sys.stderr)
+        else:
+            patterns = ", ".join(tsconfig_args)
+            print(f"Index scope: {patterns} ({len(scope_paths)} tsconfigs)", file=sys.stderr)
     elif path_args:
         scope_paths: list[str] = []
         for path in path_args:
@@ -69,14 +78,10 @@ def main(args):
                 sys.exit(1)
             scope_paths.append(normalized)
         scope_update = scope_paths
-        print(f"Index scope: {', '.join(scope_paths)}", file=sys.stderr)
-        print(
-            (
-                "Warning: scoped reindex replaces the cache with only these projects; "
-                "run reindex --fresh to restore the full index"
-            ),
-            file=sys.stderr,
-        )
+        if len(scope_paths) == 1:
+            print(f"Index scope: {scope_paths[0]}", file=sys.stderr)
+        else:
+            print(f"Index scope: {len(scope_paths)} paths", file=sys.stderr)
 
     exclude_update: list[str] | object = UNSET
     if exclude_groups is not None:
@@ -98,6 +103,7 @@ def main(args):
             fresh=fresh,
             scope_paths=scope_update,
             exclude_globs=exclude_update,
+            unversioned=unversioned if unversioned else UNSET,
         )
         cleanup_in_progress_index(cache_dir)
         if fresh or not incremental:
@@ -110,7 +116,7 @@ def main(args):
                 os.environ["SCIP_CLI_KEEP_EXTERNAL"] = "1"
 
             started = time.perf_counter()
-            _output_db, skipped, total = index_project(
+            _output_db, skipped, total, promote = index_project(
                 root,
                 lang,
                 cache_dir,
@@ -124,13 +130,26 @@ def main(args):
             print(f"Error: {e}", file=sys.stderr)
             sys.exit(1)
 
-        next_db = index_db_path(cache_dir, replace=True)
-        if not next_db.is_file():
+        live_db = index_db_path(cache_dir, replace=False)
+        from scip_cli.indexing.performance import phase
+
+        if promote:
+            next_db = index_db_path(cache_dir, replace=True)
+            if not next_db.is_file():
+                cleanup_in_progress_index(cache_dir)
+                print("Error: No index.db found after indexing", file=sys.stderr)
+                sys.exit(1)
+            with phase("finalize"):
+                finalize_index_db(next_db)
+            with phase("promote"):
+                promote_next_index(cache_dir)
+        elif live_db.is_file():
+            with phase("finalize"):
+                finalize_index_db(live_db)
+        elif not live_db.is_file():
             cleanup_in_progress_index(cache_dir)
             print("Error: No index.db found after indexing", file=sys.stderr)
             sys.exit(1)
-
-        promote_next_index(cache_dir)
         log_index_complete(
             index_db_path(cache_dir, replace=False),
             lang.value,

{scip_cli-2.11.0 → scip_cli-3.0.0}/scip_cli/exclude.py

@@ -88,3 +88,15 @@ def path_matches_any_glob(relative_path: str, patterns: tuple[str, ...] | list[s
     if not patterns:
         return False
     return any(path_matches_glob(relative_path, pattern) for pattern in patterns)
+
+
+def filter_excluded_paths(
+    paths: tuple[str, ...] | frozenset[str] | list[str],
+    exclude_globs: tuple[str, ...],
+) -> tuple[str, ...]:
+    """Drop repo-relative paths that match persisted/config exclude globs."""
+    if not exclude_globs:
+        if isinstance(paths, tuple):
+            return paths
+        return tuple(sorted(paths))
+    return tuple(sorted(path for path in paths if not path_matches_any_glob(path, exclude_globs)))

{scip_cli-2.11.0 → scip_cli-3.0.0}/scip_cli/indexing/constants.py

@@ -1,5 +1,7 @@
 """Indexing constants."""
 
+import os
+
 INDEX_TIMEOUT = 300
 DEFAULT_MAX_HEAP_MB = 8192
 PROGRESS_LOG_MIN_PROJECTS = 10
@@ -7,6 +9,18 @@ SCIP_INSTALL_URL = "https://github.com/scip-code/scip/releases"
 
 # Language indexers (npx / go install) use latest on first use.
 # The scip expt-convert binary is pinned separately in scip_tool.py (DB schema).
+#
+# TODO: Remove github_npm.py and runners github: branch when switching back to upstream npm.
+# scip-typescript npx fallback (when the binary is not on PATH). Fork adds --files
+# for partial document indexing (see docs/benchmarks.md). Pin >= ce6e9c9.
+SCIP_TYPESCRIPT_NPX_PACKAGE = "github:flesler/scip-typescript#feat/partial-files"
+
+
+def file_incremental_enabled() -> bool:
+    """Partial reindex via fork --files (default on)."""
+    return os.environ.get("SCIP_CLI_FILE_INCREMENTAL", "1").lower() not in {"0", "false", "no"}
+
+
 # scip-typescript accepts many tsconfig paths per invocation (one .scip, one convert, no merge).
 # Split only when SCIP_CLI_TS_INDEX_BATCH_SIZE is set (OOM/timeout tuning).
 DEFAULT_TS_INDEX_BATCH_SIZE = None