@ninjaxtools/slopdex 0.6.0 → 0.9.0

package/README.md

@@ -1,172 +1,257 @@
 # slopdex
 
-Slopdex indexes named functions in TypeScript and JavaScript repositories. It uses embeddings to search code by meaning, find similar implementations, and measure whether related functions are stored near each other.
+Search functions by meaning, find duplicate-code candidates, and locate related functions spread across a codebase. Supports Python, JavaScript/JSX, TypeScript/TSX, Rust, Go, Java, and C in the same repository.
 
-Supported file types are `.ts`, `.tsx`, `.mts`, `.cts`, `.js`, `.jsx`, `.mjs`, and `.cjs`.
+## Start here
 
-## Requirements
-
-- Node.js 24 or later
-- An OpenAI or Jina AI API key
-
-## Install
+Requires **Node.js 24+** and an embedding-provider API key. Install, set your key, and run commands from the repository you want to analyze (or pass `--root /path/to/repo`):
 
 ```bash
 npm install -g @ninjaxtools/slopdex
+export OPENAI_API_KEY="your-api-key"
+slopdex search "validate an authenticated session" --format summary --limit 10
 ```
 
-## Getting Started
+The first command creates the index automatically. Later commands refresh it before searching. Add `.slopdex/` to your repository's `.gitignore`.
 
-Set your OpenAI API key:
+### Find code by purpose
 
 ```bash
-export OPENAI_API_KEY="your-api-key"
+slopdex search "keep the repository index synchronized" --format summary --limit 10
 ```
 
-Run a semantic search from the repository you want to analyze:
+Describe the behavior you need. Results show similarity, file paths, and qualified function names. To search generated descriptions of each function's role instead:
 
 ```bash
-slopdex search "validate an authenticated session" --format summary --limit 10
+slopdex use-summaries
+slopdex search-summary "keep the repository index synchronized" --format summary --limit 10
 ```
 
-## Analysis Examples
-
-### Duplicate Analysis
+`use-summaries` enables persistent, automatic summary updates. It requires `OPENAI_API_KEY` even when Jina supplies embeddings, and adds generation costs. See [summaries and scoring](#summaries-and-scoring).
 
-Compare functions in different files and include functions that span at least four lines:
+### Find duplicate-code candidates
 
 ```bash
-slopdex cross-search \
-  --cross-file-only \
-  --min-lines 4 \
-  --threshold 0.9 \
-  --limit 5
+slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9 --limit 5
 ```
 
-The default output groups related functions into clusters:
+Compare functions across files, exclude short wrappers, and group strong matches into clusters. `--limit 5` selects up to five neighbors **per source function**, not five clusters. Review the source before consolidating a match.
 
-```text
-Cluster 1 (3 functions, similarity 0.9124-0.9568)
-  src/auth/session.ts:18:1 :: validateSession
-  src/http/middleware.ts:42:1 :: authenticate
-  src/users/user-service.ts:27:3 :: UserService.authenticate
+### Review changed code or one module
+
+```bash
+slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
+slopdex cross-search --changed-since origin/main --format summary --threshold 0.9
+slopdex cross-search --source-path src/services -e '^UserService\.' --format summary --threshold 0.9
 ```
 
-A cluster contains functions connected by similarity matches. The similarity range covers the observed links in the cluster. Connected functions may be linked through another function, so review the source before deciding that code is duplicated.
+These select source functions while keeping the full eligible index available for matches. The same source filters work with `cohesion`. Combine filters to require all of them to match.
+
+### Find related code stored far apart
 
-### Cohesion Analysis
+```bash
+slopdex cohesion --threshold 0.8 --neighbors 20 --limit 50 --format summary
+```
 
-Find related functions that are separated across files and directories:
+Ranks semantically related pairs by their physical separation. Use it to review module boundaries; it is not a pass/fail architecture check.
+
+### Compare repositories
 
 ```bash
-slopdex cohesion \
-  --threshold 0.8 \
-  --neighbors 20 \
-  --limit 50 \
-  --format summary
+slopdex cross-search \
+  --target-root /path/to/other/repo \
+  --target-index /path/to/other/repo/.slopdex/index.sqlite \
+  --threshold 0.9 --format summary
 ```
 
-```text
-Cohesion: 184 functions analyzed, 37 semantic edges
-  same file 35.1%  same folder 29.7%  remote 35.2%  mean distance 1.84
+Both indexes refresh automatically and must use identical embedding profiles. The target is refreshed with the source command's embedding provider; target configuration supplies file-selection and summary settings. Use `--target-config` for a non-default target config.
 
-1. gap 0.6053  similarity 0.9400  distance 4  reciprocal
-   src/auth/session.ts:18:1 :: validateSession
-   packages/http/middleware.ts:42:1 :: authenticate
+### Inspect index health
+
+```bash
+slopdex status
+slopdex index-errors --format summary
+slopdex --version
 ```
 
-The summary divides semantic relationships into the same file, the same folder, and different folders. Mean distance increases when related functions are stored farther apart. The gap score combines semantic similarity with path distance and ranks pairs for review. `reciprocal` means both functions are among each other's nearest semantic matches.
+`status` refreshes the index and reports coverage, profiles, checkpoint, and error counts. `index-errors` reads saved failures without refreshing or requiring credentials. `--version` prints the built package version.
 
-Cohesion does not have a universal pass threshold. Compare results only when the embedding profile, threshold, neighbor count, and source scope are the same.
+## Commands
 
-## Limit Analysis Scope
+Usage: `slopdex <command> [arguments] [options]`.
 
-Restrict source functions to a file or directory:
+| Command | Purpose | Output |
+| --- | --- | --- |
+| `search <query>` | Search function code by meaning. Quote multiword queries. | JSON array; optional `summary` |
+| `use-summaries` | Generate missing purpose summaries and enable automatic updates. Repeating with unchanged inputs reuses existing summaries. | JSON statistics |
+| `search-summary <query>` | Search purpose summaries after enabling them. | JSON array; optional `summary` including summary text |
+| `cross-search` | Find neighbors for each selected function in this or another index. | `clusters` by default; optional `summary` or JSONL |
+| `cohesion` | Analyze semantic relationships versus file/folder separation. | JSON report; optional `summary` |
+| `status` | Refresh and show index metadata, counts, and profiles. | JSON object |
+| `index-errors` | Read saved file/function indexing failures. | JSON array; optional `summary` |
+| `update-git` | Explicitly refresh a Git snapshot, with current working-tree changes when targeting HEAD. | JSON update statistics |
+| `update-files <path...>` | After automatic refresh, explicitly reparse selected working-tree files. Paths are repository-relative or absolute within the root. | JSON update statistics |
+| `delete-files <path...>` | After automatic refresh, remove paths from the index; source files are not deleted. A later refresh can restore eligible files. | JSON update statistics |
+
+Manual maintenance examples:
 
 ```bash
-slopdex cross-search --source-path src/services --format summary
-slopdex cohesion --source-path src/services --format summary
+slopdex update-git
+slopdex update-files src/service.ts src/model.ts
+slopdex delete-files src/removed.ts
+slopdex update-git --target HEAD --rebuild-on-divergence
+slopdex update-git --force-reindex
 ```
 
-Analyze functions added, changed, or moved since a commit:
+## Command-line arguments
+
+Options are command-specific where indicated. Boolean flags default to off.
+
+### Location, providers, and diagnostics
+
+| Argument | Meaning / default |
+| --- | --- |
+| `--root <path>` | Repository root; current directory by default. |
+| `--config <path>` | Config file; `<root>/.slopdex/config.json` by default. |
+| `--index <path>` | Index file; `<root>/.slopdex/index.sqlite` by default. Overrides `indexPath` in config. |
+| `--provider <openai\|jina>` | Embedding provider; `openai` by default. |
+| `--model <name>` | Embedding model; `text-embedding-3-large` for OpenAI, `jina-embeddings-v4` for Jina. |
+| `--dimensions <number>` | Positive embedding dimension count; OpenAI `3072`, Jina `1024`. Must be supported by the model. |
+| `--summary-model <name>` | OpenAI summary model; `gpt-5.6-sol` initially, then the persisted model unless overridden. |
+| `--ignore-errors` | Silence warnings about saved indexing errors; records remain available. |
+| `-h`, `--help` | Show CLI usage without refreshing. |
+| `--version` | Print the package version and exit. |
+
+Explicit relative config and index paths resolve from the current directory, not `--root`. CLI settings override config settings.
+
+### Search and analysis
+
+| Argument | Applies to | Meaning / default |
+| --- | --- | --- |
+| `--limit <number>` | Both query searches, cross-search, cohesion | Positive integer. Query matches: `10`; cross-search neighbors per source: `5`; cohesion reported pairs and file rows: `50`. |
+| `--threshold <number\|min-max>` | Both query searches, cross-search, cohesion | Minimum similarity, or range with inclusive minimum and exclusive maximum. Default `-1` for query/cross-search, `0.8` for cohesion. Cohesion minimum must be at least `-1` and below `1`. |
+| `--format <json\|summary\|clusters>` | Both query searches, cross-search, cohesion, index-errors | Output format; see the commands table. `clusters` is only for cross-search. |
+| `-e <regex>`, `--regexp <regex>` | Both query searches, cross-search, cohesion | Case-sensitive JavaScript regex over qualified names. Query searches: filters results before limiting. Analysis: filters sources only. |
+| `--regex <regex>` | Cross-search, cohesion | Filter **both** source and candidate qualified names. |
+| `--min-lines <number>` | Cross-search, cohesion | Minimum source and candidate callable length; positive integer, default `2`. Use `1` to include one-line wrappers. |
+| `--source-path <path>` | Cross-search, cohesion | Select sources in a file or recursive directory, relative to the repository root (or absolute within it). |
+| `--changed-since <commit>` | Cross-search, cohesion | Select added, modified, or moved functions relative to an ancestor of the indexed Git checkpoint, including working-tree changes. Requires Git. |
+| `--uncommitted` | Cross-search, cohesion | Select functions indexed from working-tree files: staged, unstaged, or untracked changes in Git; all working-tree functions without Git. |
+| `--cross-file-only` | Cross-search | Exclude matches from the same physical file. |
+| `--include-symmetric-duplicates` | Cross-search | Allow both directions of same-index matches; otherwise each unordered pair is emitted once. |
+| `--neighbors <number>` | Cohesion | Neighbors considered per source; positive integer, default `20`. Changes the analysis graph. |
+| `--include-source` | Cohesion JSON | Include callable bodies; omitted by default. |
+| `--target-root <path>` | Cross-search | Second repository root; requires `--target-index`. |
+| `--target-index <path>` | Cross-search | Second index file; requires `--target-root`. |
+| `--target-config <path>` | Cross-search | Target config; defaults to `<target-root>/.slopdex/config.json`. Requires both target options. |
+
+Source restrictions intersect: with both Git filters, a function must have changed since the commit **and** belong to an uncommitted file. `-e`, `--source-path`, and Git filters do not restrict candidates; `--regex` and `--min-lines` do.
+
+Review adjacent similarity bands without repeating boundary matches:
 
 ```bash
-slopdex cross-search --changed-since origin/main --format summary
-slopdex cohesion --changed-since origin/main --format summary
+slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9 --limit 5
+slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9 --limit 5
 ```
 
-Analyze functions in files with uncommitted changes:
+### Refresh and recovery
 
-```bash
-slopdex cross-search --uncommitted --format summary
-slopdex cohesion --uncommitted --format summary
-```
+| Argument | Meaning |
+| --- | --- |
+| `--target <ref>` | Git snapshot for `update-git`; default `HEAD`. Non-HEAD targets exclude working-tree changes. Later commands normally refresh back to HEAD. |
+| `--rebuild-on-divergence` | Allow reconciliation when the saved checkpoint is not an ancestor of the target, such as after a rebase or branch switch. |
+| `--force-reindex` | Recreate an **incompatible** index (repository, provider, model, dimensions, strategy, or schema mismatch). A compatible index still follows normal refresh behavior. |
+| `--no-reindex` | With Git, still reconcile the committed snapshot but skip working-tree overlays. Without Git, reuse a non-empty index; missing/empty indexes are still populated. Not a general offline switch. |
+
+## Reading results
 
-These options restrict the source functions. Slopdex still compares them with the full index.
+### Similarity and duplicate clusters
 
-## Output And Filters
+Similarity is a model-dependent score, not a probability of duplication. Higher scores mean greater semantic resemblance. Query summaries show `score  path :: qualifiedName`; cross-search summaries group those lines beneath each source. Functions without matches are omitted from cross-search output.
+
+```text
+Cluster 1 (3 functions, similarity 0.9124-0.9568)
+  src/auth/session.ts:18:1 :: validateSession
+  src/http/middleware.ts:42:1 :: authenticate
+  src/users/user-service.ts:27:3 :: UserService.authenticate
+```
 
-`search` supports `json` and `summary` output. `cross-search` supports `json`, `summary`, and `clusters` output. Use `--format` to select one.
+- A cluster groups functions connected by matches. Its range covers observed links; not every pair necessarily matches directly.
+- Clusters sort by member count, then name. Cluster number is not severity.
+- Locations identify where to inspect behavior, callers, and architectural roles. Wrappers, adapters, tests, and separate interface implementations can legitimately resemble one another.
 
-Use `--threshold` with a minimum similarity or a range:
+### Cohesion
 
-```bash
-slopdex search "validate session" --threshold 0.8 --format summary
-slopdex cross-search --threshold 0.85-0.9 --format summary
+```text
+Cohesion: 184 functions analyzed, 37 semantic edges
+  same file 35.1%  same folder 29.7%  remote 35.2%  mean distance 1.84
+
+1. gap 0.6053  similarity 0.9400  distance 4  reciprocal
+   src/auth/session.ts:18:1 :: validateSession
+   packages/http/middleware.ts:42:1 :: authenticate
 ```
 
-For a range, the lower bound is included and the upper bound is excluded. Similarity values depend on the embedding model, so use them to rank results from the same index profile.
+| Field | Interpretation |
+| --- | --- |
+| Functions analyzed / semantic edges | Selected-source coverage / unique qualifying neighbor pairs, before report limiting. Not quality scores. |
+| Same file / same folder / remote | Shares of weighted semantic affinity. Higher remote affinity means more related code crosses folder boundaries. |
+| Mean distance | Weighted physical separation: `0` for the same file, `1` for different files in one folder, larger across folders. |
+| Gap / rank | A `0–1` review score combining similarity above the threshold and separation; higher gap ranks first. Same-file pairs have zero gap. |
+| Reciprocal | Both functions selected each other as neighbors. JSON `null` means the other endpoint was not evaluated under source filtering. |
+| `sourceTestPair` | A source/test relationship inferred from paths; separation may be intentional. |
+| `externalAffinityRatio` | In JSON file reports, the share of observed affinity outside that file's folder. |
 
-Run `slopdex --help` for all commands and options.
+The example suggests reviewing separated authentication responsibilities. It does not establish that they belong in one module. There is no universal cohesion pass threshold. Filtered reports describe selected sources, not the entire repository. Summary metrics cover all qualifying edges; reported pairs/files are limited, and groups are built from reported pairs.
 
-## Indexing And Data
+For automation, query searches and diagnostics return JSON arrays; cross-search returns **JSONL**, one row per matched source; cohesion returns one JSON object containing `repository`, `parameters`, `summary`, `pairs`, `files`, and `groups`. Results go to stdout; notices and warnings go to stderr.
 
-Before each analysis command, Slopdex updates its index from the current Git commit and the staged, unstaged, and untracked files in the working tree. The index is created at `.slopdex/index.sqlite` by default.
+## System behavior
 
-Add `.slopdex/` to the repository's `.gitignore` so the local index is not committed.
+### Freshness and cost
 
-Slopdex sends extracted function source to the configured embedding provider. The function metadata and embeddings are stored in the local SQLite index.
+- Indexing, search, analysis, `use-summaries`, and `status` automatically refresh. With Git, results normally reflect HEAD plus staged, unstaged, and untracked working-tree contents. `gitCheckpoint` records the committed base, not the overlay.
+- Without Git, commands warn and scan the working tree. `--no-reindex` has the limited behavior described above.
+- `index-errors`, help, and version do not refresh. Other commands require the configured embedding key, even when cached data supplies the analysis.
+- Function source and search queries go to the embedding provider. Source metadata, vectors, diagnostics, and enabled summaries stay in the local index. Summary generation also sends repository name, path, callable source, and surrounding file context to OpenAI; summary text goes to the embedding provider.
+- Unchanged embedding and summary inputs reuse cached results. Initial indexing and enabling summaries can make many API calls. Source filters reduce analysis work, not the preceding index refresh.
 
-Whole-repository cohesion analysis compares neighbors for every selected function. Use `--source-path`, `--changed-since`, or `--uncommitted` to reduce the scope in large repositories.
+### Summaries and scoring
 
-## Library
+Summaries are optional and disabled initially. `use-summaries` persists the selected summary model and keeps summaries current on later updates, including file-context and path changes. Use `slopdex use-summaries --summary-model <model-id>` to change it. `status` exposes `summariesEnabled`, `summaryCount`, and `summaryProfile`.
 
-The package also exports the index and analysis APIs:
+`search` always searches code; `search-summary` always searches purpose summaries. When all callables have enabled summaries, cross-search and cohesion automatically use **50% code similarity + 50% summary similarity**. Cross-repository analysis needs complete summaries on both sides; otherwise the entire analysis uses code-only scores. Thresholds and neighbor limits apply to the selected score.
 
-```ts
-import {
-  OpenAIEmbeddingProvider,
-  openCodeIndex,
-} from "@ninjaxtools/slopdex";
+Text output labels combined scores. JSON exposes `codeSimilarity`, `summarySimilarity`, and scoring mode/weights (`scoring` for cross-search, `parameters` for cohesion). Compare runs only with matching scoring mode, weights, embedding and summary-generator profiles, threshold, neighbor count, and source/candidate filters.
 
-const index = openCodeIndex({
-  rootDir: "/path/to/repository",
-  provider: new OpenAIEmbeddingProvider(),
-});
+### Coverage and exclusions
 
-await index.updateFromGit();
+| Language | File extensions |
+| --- | --- |
+| Python | `.py`, `.pyw` |
+| JavaScript / JSX | `.js`, `.mjs`, `.cjs`, `.jsx` |
+| TypeScript / TSX | `.ts`, `.mts`, `.cts`, `.tsx` |
+| Rust | `.rs` |
+| Go | `.go` |
+| Java | `.java` |
+| C | `.c`, `.h` |
 
-const results = await index.similaritySearch({
-  query: "validate an authenticated session",
-  limit: 10,
-});
+Indexes named functions, methods, constructors, and supported variable-bound closures with bodies, including nested callables. Anonymous callbacks and bodyless declarations are omitted. No language server or project compiler setup is needed. Macro expansion and runtime behavior are not analyzed; `.h` files are treated as C.
 
-index.close();
-```
+Root and nested `.gitignore` rules apply even to tracked files and without Git. Working-tree refreshes use current rules; committed-only snapshots use the target commit's rules. Refresh removes newly excluded files and discovers newly eligible ones. Explicit `update-files` rejects ignored files.
 
-Exports also include `crossSearch`, `analyzeCohesion`, `JinaEmbeddingProvider`, and standalone functions for index updates and searches.
+Built-in exclusions: `.git`, `.slopdex`, `node_modules`, `dist`, `build`, `coverage`, `vendor`, `generated`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, and `target`. Config `include`/`exclude` globs narrow coverage; they cannot override built-in exclusions. Ignore exceptions cannot re-include files beneath an excluded parent directory. Files over 1 MiB are skipped unless `maxFileSize` is raised.
 
-## Development
+### Failures and recovery
 
-```bash
-npm run check
-```
+Parse, extraction, read, and file-size failures are saved while healthy callables remain searchable. Inspect them with `slopdex index-errors --format summary`. JSON includes paths, locations, recoverable names, messages, available source, and snapshot provenance. `status` reports `indexingErrorCount` and `failedFileCount`; `functionCount` counts searchable callables.
 
-## Configuration
+Saved failures trigger stderr warnings, including on cached runs, help, and cross-search targets. `--ignore-errors` silences warnings without clearing records. Updates retry failed files; successful indexing, deletion, or exclusion clears their diagnostics. Version output bypasses diagnostics.
 
-Configuration is optional. Without `.slopdex/config.json`, Slopdex uses OpenAI's `text-embedding-3-large` model with 3072 dimensions and the built-in source exclusions.
+Use the recovery flag named in the error: `--rebuild-on-divergence` for Git history changes, `--force-reindex` for incompatible indexes. For provider/authentication failures, fix the reported configuration. For source-change-during-indexing errors, rerun after edits settle. Exit status is `0` on success, `2` for argument/domain errors, and `1` for other failures (or invocation without a command).
+
+## Configuration
 
-Create `.slopdex/config.json` to change these settings:
+Optional file: `<root>/.slopdex/config.json`. Example using Jina (requires `JINA_API_KEY`):
 
 ```json
 {
@@ -177,4 +262,19 @@ Create `.slopdex/config.json` to change these settings:
 }
 ```
 
-Use `OPENAI_API_KEY` for OpenAI and `JINA_API_KEY` for Jina AI. The provider, model, dimensions, and embedding strategy define the index profile. Rebuild the index with `--force-reindex` after changing the profile.
+| Property | Purpose / default |
+| --- | --- |
+| `provider`, `model`, `dimensions` | Embedding settings; defaults are listed in the CLI table. |
+| `summaryModel` | Summary model; initially `gpt-5.6-sol`. |
+| `indexPath` | Index location; `<root>/.slopdex/index.sqlite`. |
+| `include` | Repository-relative glob array; empty/unset includes all supported eligible files. |
+| `exclude` | Additional repository-relative exclusion globs. |
+| `maxFileSize` | Maximum source-file size in bytes; positive integer, default `1048576`. |
+| `embeddingBatchSize` | Embedding inputs per batch; positive integer, default `32`. |
+
+Keep keys in the environment (`OPENAI_API_KEY`, `JINA_API_KEY`). Changing the embedding profile requires rebuilding with `--force-reindex`.
+
+## Agent and developer documentation
+
+- [Agent skill](.agents/skills/slopdex/SKILL.md): task-oriented CLI guidance for coding agents.
+- [Implementation and library API](docs/implementation.md): internals, programmatic usage, build, and development checks.