@ninjaxtools/slopdex 0.8.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (9) hide show
  1. package/.agents/skills/slopdex/SKILL.md +150 -270
  2. package/README.md +159 -211
  3. package/dist/cli.js +79 -38
  4. package/dist/cli.js.map +1 -1
  5. package/dist/index.d.ts +5 -3
  6. package/dist/index.js +33 -7
  7. package/dist/index.js.map +1 -1
  8. package/docs/implementation.md +163 -0
  9. package/package.json +2 -1
package/README.md CHANGED
@@ -1,302 +1,236 @@
1
1
  # slopdex
2
2
 
3
- Slopdex uses Tree-sitter to index named functions in Python, JavaScript, JSX, TypeScript, TSX, Rust, Go, Java, and C repositories. It uses embeddings to search code by meaning, find similar implementations, and measure whether related functions are stored near each other.
3
+ Search functions by meaning, find duplicate-code candidates, and locate related functions spread across a codebase.
4
4
 
5
- ## Install
5
+ ## Start here
6
6
 
7
- ```bash
8
- npm install -g @ninjaxtools/slopdex
9
- ```
7
+ Requires an embedding-provider API key. Install, set your key, and run commands from the repository you want to analyze (or pass `--root /path/to/repo`):
10
8
 
11
- ## Getting Started
12
-
13
- Set your OpenAI API key:
9
+ Coding agents should start with the bundled [Slopdex agent skill](.agents/skills/slopdex/SKILL.md).
14
10
 
15
11
  ```bash
12
+ npm install -g @ninjaxtools/slopdex
16
13
  export OPENAI_API_KEY="your-api-key"
17
- ```
18
-
19
- Run a semantic search from the repository you want to analyze:
20
-
21
- ```bash
22
14
  slopdex search "validate an authenticated session" --format summary --limit 10
23
15
  ```
24
16
 
25
- ## Analysis Examples
17
+ The first command creates the index automatically. Later commands refresh it before searching.
26
18
 
27
- ### Search By Function Purpose
19
+ Add `.slopdex/` to your repository's `.gitignore`.
28
20
 
29
- Enable optional purpose summaries for every indexed callable:
21
+ ### Search code
30
22
 
31
23
  ```bash
32
- slopdex use-summaries
33
- slopdex search-summary "keep the repository index synchronized" --limit 10 --format summary
24
+ slopdex search "keep the repository index synchronized" --format summary --limit 10
34
25
  ```
35
26
 
36
- Summaries describe a function's responsibility and role in its codebase, using its repository name, path, source, and surrounding file context. Generation uses the OpenAI Responses API with **`gpt-5.6-sol`** by default and requires `OPENAI_API_KEY`, including when Jina supplies the embeddings.
37
-
38
- `use-summaries` stores each summary and its embedding in SQLite and enables a persistent repository-index setting. Subsequent indexing operations automatically generate summaries for new and changed files, including changes to surrounding context and file paths. Unchanged inputs reuse cached summaries and vectors; deleted functions disappear from summary search. Running `use-summaries` again is a no-op when summaries are already complete. Summary generation and embeddings are saved atomically, so a failed request does not leave partially updated callables.
39
-
40
- `search-summary` uses a separate summary embedding store with the configured embedding provider and supports the same query, limit, threshold, and output options as `search`. JSON results include the summary; `--format summary` prints it alongside each match. `status` reports `summariesEnabled`, `summaryCount`, and `summaryProfile`.
41
-
42
- To choose a different summary model, run `slopdex use-summaries --summary-model <model-id>` or set `summaryModel` in `.slopdex/config.json`. The chosen model is persisted for future updates. Existing indexes migrate automatically; summaries remain disabled until enabled explicitly.
43
-
44
- ### Combined Code And Purpose Analysis
45
-
46
- Once summaries are enabled and every indexed callable has a summary embedding, `cross-search` and `cohesion` automatically combine implementation and purpose similarity:
47
-
48
- ```text
49
- similarity = 0.5 * codeSimilarity + 0.5 * summarySimilarity
50
- ```
27
+ To search generated descriptions of each function's role instead:
51
28
 
52
29
  ```bash
53
- slopdex use-summaries
54
- slopdex cross-search --threshold 0.8 --format json
55
- slopdex cohesion --threshold 0.8 --format summary
30
+ slopdex summaries enable
31
+ slopdex search-summary "keep the repository index synchronized" --format summary --limit 10
56
32
  ```
57
33
 
58
- Both cosine scores and their average are calculated in one SQLite query. Thresholds (including half-open ranges), ranking, and neighbor/result limits apply to the combined score. Cohesion uses these combined neighbors for reciprocity, gap scores, file affinities, and groups.
59
-
60
- For cross-repository search, both indexes must have complete, enabled summaries. If either index lacks them, the entire analysis uses code-only similarity. Cohesion likewise uses code-only scoring if its summary index is incomplete or disabled. The existing embedding profiles must match across repositories; summary-generator models may differ.
34
+ This requires `OPENAI_API_KEY` even when Jina supplies embeddings, and adds generation costs. See [summaries and scoring](#summaries-and-scoring).
61
35
 
62
- Combined JSON matches and cohesion pairs expose `codeSimilarity` and `summarySimilarity` alongside `similarity`. Cross-search rows include a `scoring` object with `similarityMode`, `similarityWeights`, and both summary-generator profiles. Cohesion records the mode and weights in `parameters` and the summary-generator profile in `repository.summaryProfile`. The mode is `"code-summary-average"` with weights `{ "code": 0.5, "summary": 0.5 }`, or `"code"` with weights `{ "code": 1, "summary": 0 }`. Text output labels combined scores.
36
+ `summaries disable` turns off summary generation while retaining cached data.
63
37
 
64
- Compare analysis results only when the similarity mode and weights match, as well as the embedding profile, summary-generator profiles, thresholds, and analysis scope. Enabling summaries can change rankings and cohesion metrics. `search` continues to use only code vectors, and `search-summary` uses only summary vectors.
65
-
66
- ### Duplicate Analysis
67
-
68
- Compare functions in different files and include functions that span at least four lines:
38
+ ### Find duplicate-code candidates
69
39
 
70
40
  ```bash
71
- slopdex cross-search \
72
- --cross-file-only \
73
- --min-lines 4 \
74
- --threshold 0.9 \
75
- --limit 5
76
- ```
77
-
78
- The default output groups related functions into clusters:
79
-
80
- ```text
81
- Cluster 1 (3 functions, similarity 0.9124-0.9568)
82
- src/auth/session.ts:18:1 :: validateSession
83
- src/http/middleware.ts:42:1 :: authenticate
84
- src/users/user-service.ts:27:3 :: UserService.authenticate
41
+ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9 --limit 5
85
42
  ```
86
43
 
87
- A cluster contains functions connected by similarity matches. The similarity range covers the observed links in the cluster. Connected functions may be linked through another function, so review the source before deciding that code is duplicated.
88
-
89
- ### Cohesion Analysis
44
+ Compare functions across files, exclude short wrappers, and group strong matches into clusters. `--limit 5` selects up to five neighbors **per source function**, not five clusters.
90
45
 
91
- Find related functions that are separated across files and directories:
46
+ ### Review changed code or one module
92
47
 
93
48
  ```bash
94
- slopdex cohesion \
95
- --threshold 0.8 \
96
- --neighbors 20 \
97
- --limit 50 \
98
- --format summary
99
- ```
100
-
101
- ```text
102
- Cohesion: 184 functions analyzed, 37 semantic edges
103
- same file 35.1% same folder 29.7% remote 35.2% mean distance 1.84
104
-
105
- 1. gap 0.6053 similarity 0.9400 distance 4 reciprocal
106
- src/auth/session.ts:18:1 :: validateSession
107
- packages/http/middleware.ts:42:1 :: authenticate
49
+ slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
50
+ slopdex cross-search --changed-since origin/main --format summary --threshold 0.9
51
+ slopdex cross-search --source-path src/services -e '^UserService\.' --format summary --threshold 0.9
108
52
  ```
109
53
 
110
- The summary divides semantic relationships into the same file, the same folder, and different folders. Mean distance increases when related functions are stored farther apart. The gap score combines semantic similarity with path distance and ranks pairs for review. `reciprocal` means both functions are among each other's nearest semantic matches.
54
+ These select source functions while keeping the full eligible index available for matches. The same source filters work with `cohesion`.
111
55
 
112
- Cohesion does not have a universal pass threshold. Compare results only when the similarity mode and weights, embedding and summary-generator profiles, threshold, neighbor count, and source scope are the same.
113
-
114
- ## Limit Analysis Scope
115
-
116
- Restrict source functions to a file or directory:
56
+ ### Find related code stored far apart
117
57
 
118
58
  ```bash
119
- slopdex cross-search --source-path src/services --format summary
120
- slopdex cohesion --source-path src/services --format summary
59
+ slopdex cohesion --threshold 0.8 --neighbors 20 --limit 50 --format summary
121
60
  ```
122
61
 
123
- Analyze functions added, changed, or moved since a commit:
124
-
125
- ```bash
126
- slopdex cross-search --changed-since origin/main --format summary
127
- slopdex cohesion --changed-since origin/main --format summary
128
- ```
62
+ Ranks semantically related pairs by their physical separation.
129
63
 
130
- Analyze functions in files with uncommitted changes:
64
+ ### Compare repositories
131
65
 
132
66
  ```bash
133
- slopdex cross-search --uncommitted --format summary
134
- slopdex cohesion --uncommitted --format summary
135
- ```
136
-
137
- These options restrict the source functions. Slopdex still compares them with the full index.
138
-
139
- Filter source symbols by qualified name with `-e <regex>` (or `--regexp <regex>`):
140
-
141
- ```bash
142
- slopdex cross-search -e '^UserService\.' --format summary
143
- slopdex cohesion -e 'validate|authenticate' --source-path src/auth
67
+ slopdex cross-search \
68
+ --target-root /path/to/other/repo \
69
+ --target-index /path/to/other/repo/.slopdex/index.sqlite \
70
+ --threshold 0.9 --format summary
144
71
  ```
145
72
 
146
- The regex uses case-sensitive JavaScript syntax and matches qualified names such as `UserService.authenticate`. It filters only the source symbols for cross-search and cohesion; matching candidates are still drawn from the whole eligible index, including symbols that do not match the regex. It works with code-only and combined code/summary scoring.
73
+ Both indexes refresh automatically and must use identical embedding profiles. The target is refreshed with the source command's embedding provider; target configuration supplies file-selection and summary settings. Use `--target-config` for a non-default target config.
147
74
 
148
- Source restrictions combine by intersection:
75
+ ### Inspect index health
149
76
 
150
77
  ```bash
151
- slopdex cross-search -e 'validate' \
152
- --source-path src \
153
- --changed-since origin/main \
154
- --uncommitted \
155
- --min-lines 4 \
156
- --cross-file-only
78
+ slopdex status
79
+ slopdex index-errors --format summary
80
+ slopdex --version
157
81
  ```
158
82
 
159
- With both Git filters, a source must have changed since the specified commit and belong to an uncommitted file. Path, name, and minimum-line filters narrow the selection further. Existing target restrictions such as `--min-lines` and `--cross-file-only` continue to apply. The separate `--regex` option filters **both** analysis sources and matching candidates.
83
+ `status` refreshes the index and reports coverage, profiles, checkpoint, and error counts. `index-errors` reads saved failures without refreshing or requiring credentials. `--version` prints the built package version.
160
84
 
161
- ## Output And Filters
85
+ ## Commands
162
86
 
163
- `search` supports `json` and `summary` output. `cross-search` supports `json`, `summary`, and `clusters` output. Use `--format` to select one.
164
-
165
- For `search` and `search-summary`, `-e` restricts result symbols before the result limit is applied:
166
-
167
- ```bash
168
- slopdex search "validate session" -e '^Session\.' --limit 10
169
- slopdex search-summary "persist user data" -e 'save|persist' --limit 5
170
- ```
87
+ Usage: `slopdex <command> [arguments] [options]`.
171
88
 
172
- Use `--threshold` with a minimum similarity or a range:
89
+ | Command | Purpose | Output |
90
+ | --- | --- | --- |
91
+ | `search <query>` | Search function code by meaning. Quote multiword queries. | Summary; optional JSON array |
92
+ | `summaries <enable\|disable>` | Enable or disable automatic purpose summaries. Re-enabling with unchanged inputs reuses cached summaries. | JSON statistics |
93
+ | `search-summary <query>` | Search purpose summaries after enabling them. | Summary including summary text; optional JSON array |
94
+ | `cross-search` | Find neighbors for each selected function in this or another index. | `clusters` by default; optional `summary` or JSONL |
95
+ | `cohesion` | Analyze semantic relationships versus file/folder separation. | Summary; optional JSON report |
96
+ | `status` | Refresh and show index metadata, counts, and profiles. | JSON object |
97
+ | `index-errors` | Read saved file/function indexing failures. | Summary; optional JSON array |
98
+ | `update-git` | Explicitly refresh a Git snapshot, with current working-tree changes when targeting HEAD. | JSON update statistics |
99
+ | `update-files <path...>` | After automatic refresh, explicitly reparse selected working-tree files. Paths are repository-relative or absolute within the root. | JSON update statistics |
100
+ | `delete-files <path...>` | After automatic refresh, remove paths from the index; source files are not deleted. A later refresh can restore eligible files. | JSON update statistics |
101
+
102
+ Manual maintenance examples:
173
103
 
174
104
  ```bash
175
- slopdex search "validate session" --threshold 0.8 --format summary
176
- slopdex cross-search --threshold 0.85-0.9 --format summary
105
+ slopdex update-git
106
+ slopdex update-files src/service.ts src/model.ts
107
+ slopdex delete-files src/removed.ts
108
+ slopdex update-git --target HEAD --rebuild-on-divergence
109
+ slopdex update-git --force-reindex
177
110
  ```
178
111
 
179
- For a range, the lower bound is included and the upper bound is excluded. Similarity values depend on the embedding model, so use them to rank results from the same index profile.
180
-
181
- Run `slopdex --help` for all commands and options.
112
+ ### Location, providers, and diagnostics
182
113
 
183
- ## Indexing And Data
114
+ | Argument | Meaning / default |
115
+ | --- | --- |
116
+ | `--root <path>` | Repository root; current directory by default. |
117
+ | `--config <path>` | Config file; `<root>/.slopdex/config.json` by default. |
118
+ | `--index <path>` | Index file; `<root>/.slopdex/index.sqlite` by default. Overrides `indexPath` in config. |
119
+ | `--provider <openai\|jina>` | Embedding provider; `openai` by default. |
120
+ | `--model <name>` | Embedding model; `text-embedding-3-large` for OpenAI, `jina-embeddings-v4` for Jina. |
121
+ | `--dimensions <number>` | Positive embedding dimension count; OpenAI `3072`, Jina `1024`. Must be supported by the model. |
122
+ | `--summary-model <name>` | OpenAI summary model; `gpt-5.6-sol` initially, then the persisted model unless overridden. |
123
+ | `--ignore-errors` | Silence warnings about saved indexing errors; records remain available. |
124
+ | `-h`, `--help` | Show CLI usage without refreshing. |
125
+ | `--version` | Print the package version and exit. |
184
126
 
185
- ### Supported Languages
127
+ Explicit relative config and index paths resolve from the current directory, not `--root`.
186
128
 
187
- Languages are detected automatically from file extensions. A single index can contain multiple languages; search, purpose summaries, cross-search, and cohesion work across all of them.
129
+ ### Search and analysis
188
130
 
189
- | Language | Extensions | Extracted callables |
131
+ | Argument | Applies to | Meaning / default |
190
132
  | --- | --- | --- |
191
- | Python | `.py`, `.pyw` | Functions, async functions, class methods, constructors, generators, and bound lambdas; includes decorators in source |
192
- | JavaScript | `.js`, `.mjs`, `.cjs` | Functions, generators, methods, constructors, and named function expressions/arrows |
193
- | JSX | `.jsx` | JavaScript callables, including components returning JSX |
194
- | TypeScript | `.ts`, `.mts`, `.cts` | Typed functions, methods, constructors, and named function expressions/arrows |
195
- | TSX | `.tsx` | TypeScript callables, including generic components returning JSX |
196
- | Rust | `.rs` | Functions, `impl` methods/associated functions, trait default methods, and `let`-bound closures |
197
- | Go | `.go` | Functions, receiver methods, and function literals bound to variables or assignments |
198
- | Java | `.java` | Methods, constructors (including compact record constructors), and variable-bound lambdas |
199
- | C | `.c`, `.h` | Function definitions, including static/inline functions and functions returning pointers |
200
-
201
- Qualified names include enclosing classes, functions, and explicit modules. Go methods include their receiver type (`Store[T].Get`); Rust trait implementations include the type and trait (`<Store<T> as Read>.read`). Definitions retain their source, signature, line/column locations, and stable identity for incremental updates. Declarations without bodies and anonymous callbacks are omitted. Extraction is syntactic: Rust macros and C macros are not expanded, and C preprocessor branches are indexed as written. `.h` files use the C grammar.
202
-
203
- Tree-sitter grammars ship as package dependencies; no language server or project compiler configuration is required. Syntax errors produce warnings while recoverable callables are still indexed. Existing indexes pick up newly supported files during the next normal refresh.
204
-
205
- Default exclusions cover dependency and build directories: `.git`, `.slopdex`, `node_modules`, `dist`, `build`, `coverage`, `vendor`, `generated`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, and `target`. Optional `include` and `exclude` glob arrays in `.slopdex/config.json` further restrict the indexed files.
133
+ | `--limit <number>` | Both query searches, cross-search, cohesion | Positive integer. Query matches: `10`; cross-search neighbors per source: `5`; cohesion reported pairs and file rows: `50`. |
134
+ | `--threshold <number\|min-max>` | Both query searches, cross-search, cohesion | Minimum similarity, or range with inclusive minimum and exclusive maximum. Default `-1` for query/cross-search, `0.8` for cohesion. Cohesion minimum must be at least `-1` and below `1`. |
135
+ | `--format <json\|summary\|clusters>` | Both query searches, cross-search, cohesion, index-errors | Output format; see the commands table. `clusters` is only for cross-search. |
136
+ | `-e <regex>`, `--regexp <regex>`, `--regex <regex>` | Both query searches, cross-search, cohesion | Equivalent case-sensitive JavaScript regex options over qualified names. Query searches: filter results before limiting. Analysis: filter sources only. |
137
+ | `--min-lines <number>` | Cross-search, cohesion | Minimum source and candidate callable length; positive integer, default `2`. Use `1` to include one-line wrappers. |
138
+ | `--source-path <path>` | Cross-search, cohesion | Select sources in a file or recursive directory, relative to the repository root (or absolute within it). |
139
+ | `--changed-since <commit>` | Cross-search, cohesion | Select added, modified, or moved functions relative to an ancestor of the indexed Git checkpoint, including working-tree changes. Requires Git. |
140
+ | `--uncommitted` | Cross-search, cohesion | Select functions indexed from working-tree files: staged, unstaged, or untracked changes in Git; all working-tree functions without Git. |
141
+ | `--cross-file-only` | Cross-search | Exclude matches from the same physical file. |
142
+ | `--include-symmetric-duplicates` | Cross-search | Allow both directions of same-index matches; otherwise each unordered pair is emitted once. |
143
+ | `--neighbors <number>` | Cohesion | Neighbors considered per source; positive integer, default `20`. Changes the analysis graph. |
144
+ | `--include-source` | Cohesion JSON | Include callable bodies; omitted by default. |
145
+ | `--target-root <path>` | Cross-search | Second repository root; requires `--target-index`. |
146
+ | `--target-index <path>` | Cross-search | Second index file; requires `--target-root`. |
147
+ | `--target-config <path>` | Cross-search | Target config; defaults to `<target-root>/.slopdex/config.json`. Requires both target options. |
148
+
149
+ Review adjacent similarity bands without repeating boundary matches:
206
150
 
207
- Slopdex also respects root and nested `.gitignore` files using the [`ignore`](https://www.npmjs.com/package/ignore) package. Patterns, directory rules, anchoring, escapes, and `!` exceptions follow Git ignore semantics. A nested rule cannot re-include files beneath an excluded parent directory. These exclusions apply even to tracked files and when Git is unavailable; explicit `update-files` requests for ignored files are rejected.
208
-
209
- Working-tree indexing and HEAD overlays use the current `.gitignore` files. Historical snapshots and committed-only Git indexing use the ignore files stored in the selected commit. Normal refreshes remove previously indexed files that become ignored and discover files that become eligible again, including when only ignore rules change. Configuration includes and negated ignore rules cannot override the built-in exclusions.
210
-
211
- ### Index Updates
212
-
213
- Before each analysis command, Slopdex updates its index from the current Git commit and the staged, unstaged, and untracked files in the working tree. The index is created at `.slopdex/index.sqlite` by default.
214
-
215
- Add `.slopdex/` to the repository's `.gitignore` so the local index is not committed.
216
-
217
- Slopdex sends extracted function source to the configured embedding provider. The function metadata and embeddings are stored in the local SQLite index.
218
-
219
- Whole-repository cohesion analysis compares neighbors for every selected function. Use `--source-path`, `--changed-since`, or `--uncommitted` to reduce the scope in large repositories.
151
+ ```bash
152
+ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9 --limit 5
153
+ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9 --limit 5
154
+ ```
220
155
 
221
- ### Inspecting Indexing Errors
156
+ ### Refresh and recovery
222
157
 
223
- Slopdex persists file and function indexing diagnostics in the SQLite `indexing_errors` table. Parse errors, parser exceptions, callable-extraction failures, file-read failures, and files exceeding `maxFileSize` retain references instead of silently disappearing. Healthy functions in partially parsed files and other healthy files remain searchable; malformed callables are omitted from semantic search and listed in the diagnostics.
158
+ | Argument | Meaning |
159
+ | --- | --- |
160
+ | `--target <ref>` | Git snapshot for `update-git`; default `HEAD`. Non-HEAD targets exclude working-tree changes. Later commands normally refresh back to HEAD. |
161
+ | `--rebuild-on-divergence` | Allow reconciliation when the saved checkpoint is not an ancestor of the target, such as after a rebase or branch switch. |
162
+ | `--force-reindex` | Recreate an **incompatible** index (repository, provider, model, dimensions, strategy, or schema mismatch). A compatible index still follows normal refresh behavior. |
163
+ | `--no-reindex` | With Git, still reconcile the committed snapshot but skip working-tree overlays. Without Git, reuse a non-empty index; missing/empty indexes are still populated. Not a general offline switch. |
224
164
 
225
- ```bash
226
- slopdex index-errors
227
- slopdex index-errors --format summary
228
- slopdex index-errors --index /path/to/another/index.sqlite
229
- ```
165
+ ## Reading results
230
166
 
231
- This command reads the saved diagnostics without refreshing the index or calling an embedding/summary provider, so it works without API credentials. JSON includes the path, language, error code and message, file/function scope, recovered qualified name when available, line/column ranges, available source text, and Git/working-tree provenance. If Tree-sitter cannot identify a function, the unparsed region is still retained as a file diagnostic.
167
+ ### Similarity and duplicate clusters
232
168
 
233
- Every invocation warns on stderr while saved errors remain, including cached runs and cross-search target indexes. Suppress this warning with `--ignore-errors`:
169
+ Similarity is a model-dependent score, not a probability of duplication. Higher scores mean greater semantic resemblance. Query summaries show `score path :: qualifiedName`; cross-search summaries group those lines beneath each source. Functions without matches are omitted from cross-search output.
234
170
 
235
- ```bash
236
- slopdex cross-search --ignore-errors
237
- slopdex index-errors --format summary --ignore-errors
171
+ ```text
172
+ Cluster 1 (3 functions, similarity 0.9124-0.9568)
173
+ src/auth/session.ts:18:1 :: validateSession
174
+ src/http/middleware.ts:42:1 :: authenticate
175
+ src/users/user-service.ts:27:3 :: UserService.authenticate
238
176
  ```
239
177
 
240
- Silencing warnings does not delete diagnostics. `status` reports `indexingErrorCount` and `failedFileCount`; `functionCount` counts searchable callables. Normal updates retry failed files, including unchanged Git blobs. Errors clear when a file is successfully indexed, deleted, or excluded. Existing indexes migrate automatically and rescan once on their next full update to detect previously unreported parse failures. Diagnostics are committed atomically with the corresponding file update.
178
+ - A cluster groups functions connected by matches. Its range covers observed links; not every pair necessarily matches directly.
179
+ - Clusters sort by member count, then name. Cluster number is not severity.
180
+ - Locations identify where to inspect behavior, callers, and architectural roles. Wrappers, adapters, tests, and separate interface implementations can legitimately resemble one another.
241
181
 
242
- ## Library
182
+ ### Cohesion
243
183
 
244
- The package also exports the index and analysis APIs:
184
+ ```text
185
+ Cohesion: 184 functions analyzed, 37 semantic edges
186
+ same file 35.1% same folder 29.7% remote 35.2% mean distance 1.84
245
187
 
246
- ```ts
247
- import {
248
- OpenAIEmbeddingProvider,
249
- openCodeIndex,
250
- } from "@ninjaxtools/slopdex";
188
+ 1. gap 0.6053 similarity 0.9400 distance 4 reciprocal
189
+ src/auth/session.ts:18:1 :: validateSession
190
+ packages/http/middleware.ts:42:1 :: authenticate
191
+ ```
251
192
 
252
- const index = openCodeIndex({
253
- rootDir: "/path/to/repository",
254
- provider: new OpenAIEmbeddingProvider(),
255
- });
193
+ | Field | Interpretation |
194
+ | --- | --- |
195
+ | Functions analyzed / semantic edges | Selected-source coverage / unique qualifying neighbor pairs, before report limiting. Not quality scores. |
196
+ | Same file / same folder / remote | Shares of weighted semantic affinity. Higher remote affinity means more related code crosses folder boundaries. |
197
+ | Mean distance | Weighted physical separation: `0` for the same file, `1` for different files in one folder, larger across folders. |
198
+ | Gap / rank | A `0–1` review score combining similarity above the threshold and separation; higher gap ranks first. Same-file pairs have zero gap. |
199
+ | Reciprocal | Both functions selected each other as neighbors. JSON `null` means the other endpoint was not evaluated under source filtering. |
200
+ | `sourceTestPair` | A source/test relationship inferred from paths; separation may be intentional. |
201
+ | `externalAffinityRatio` | In JSON file reports, the share of observed affinity outside that file's folder. |
256
202
 
257
- await index.updateFromGit();
203
+ The example suggests reviewing separated authentication responsibilities. It does not establish that they belong in one module. There is no universal cohesion pass threshold. Filtered reports describe selected sources, not the entire repository. Summary metrics cover all qualifying edges; reported pairs/files are limited, and groups are built from reported pairs.
258
204
 
259
- const results = await index.similaritySearch({
260
- query: "validate an authenticated session",
261
- limit: 10,
262
- });
205
+ For automation, pass `--format json`: query searches and diagnostics return JSON arrays; cross-search returns **JSONL**, one row per matched source; cohesion returns one JSON object containing `repository`, `parameters`, `summary`, `pairs`, `files`, and `groups`. Results go to stdout; notices and warnings go to stderr.
263
206
 
264
- index.close();
265
- ```
207
+ ## System behavior
266
208
 
267
- Exports also include `crossSearch`, `analyzeCohesion`, `JinaEmbeddingProvider`, and standalone functions for index updates and searches.
209
+ ### Summaries and scoring
268
210
 
269
- Library callers can set `sourceFilter.nameRegex` on cross-search/cohesion options for source-only filtering, together with `path` and the Git filter type. A `changed-since` filter also accepts `uncommitted: true`. Query searches accept `nameRegex` in `SimilaritySearchOptions` to filter result symbols. Cohesion reports record source restrictions in `parameters.sourceFilter` and label filtered analyses as `selected-sources`.
211
+ Summaries are optional and disabled initially. `summaries enable` persists the selected summary model and keeps summaries current on later updates, including file-context and path changes. Use `slopdex summaries enable --summary-model <model-id>` to change it, or `slopdex summaries disable` to stop automatic updates and summary-based searching/scoring while retaining cached summaries. `status` exposes `summariesEnabled`, `summaryCount`, and `summaryProfile`.
270
212
 
271
- Use `index.indexErrors()` to inspect diagnostics through an open index, or the exported `readIndexErrors(indexPath)` to inspect a saved database without a provider. Error records use the exported `IndexingError` type.
213
+ `search` always searches code; `search-summary` always searches purpose summaries. When all callables have enabled summaries, cross-search and cohesion automatically use **50% code similarity + 50% summary similarity**. Cross-repository analysis needs complete summaries on both sides; otherwise the entire analysis uses code-only scores. Thresholds and neighbor limits apply to the selected score.
272
214
 
273
- For summary search, call `await index.useSummaries()` after updating the index, then `await index.searchSummary({ query: "maintain the repository index" })`. Optionally pass `summaryProvider: new OpenAISummaryProvider({ model: "gpt-5.6-sol" })` when opening an index. Custom providers implement the exported `SummaryProvider` interface. The package also exports standalone `useSummaries` and `searchSummary` helpers.
215
+ Text output labels combined scores. JSON exposes `codeSimilarity`, `summarySimilarity`, and scoring mode/weights (`scoring` for cross-search, `parameters` for cohesion). Compare runs only with matching scoring mode, weights, embedding and summary-generator profiles, threshold, neighbor count, and source/candidate filters.
274
216
 
275
- ## Development
217
+ ### Exclusions
276
218
 
277
- ```bash
278
- npm run check
279
- ```
219
+ Root and nested `.gitignore` rules apply even to tracked files and without Git. Working-tree refreshes use current rules; committed-only snapshots use the target commit's rules. Refresh removes newly excluded files and discovers newly eligible ones. Explicit `update-files` rejects ignored files.
280
220
 
281
- To cross-check the shared-language fixtures against a built sibling checkout of `treesitter-index`, run:
221
+ Built-in exclusions: `.git`, `.slopdex`, `node_modules`, `dist`, `build`, `coverage`, `vendor`, `generated`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, and `target`. Config `include`/`exclude` globs narrow coverage; they cannot override built-in exclusions. Ignore exceptions cannot re-include files beneath an excluded parent directory. Files over 1 MiB are skipped unless `maxFileSize` is raised.
282
222
 
283
- ```bash
284
- npm run check:parser-parity
285
- # Or provide another reference executable:
286
- npm run check:parser-parity -- /path/to/treesitter-index
287
- ```
223
+ ### Failures and recovery
288
224
 
289
- The default reference is `../treesitter-index/target/debug/treesitter-index`. The check covers eight shared languages; that project has no C grammar. Callable regression tests also run as part of `npm run check` without requiring the sibling checkout.
225
+ Parse, extraction, read, and file-size failures are saved while healthy callables remain searchable. Inspect them with `slopdex index-errors --format summary`. JSON includes paths, locations, recoverable names, messages, available source, and snapshot provenance. `status` reports `indexingErrorCount` and `failedFileCount`; `functionCount` counts searchable callables.
290
226
 
291
- The projects index different information: `treesitter-index` includes declarations, types, imports, and `.pyi` stubs, while Slopdex indexes callable implementations (including nested callables and bound closures). Slopdex also supports `.pyw` and C. Its native Node grammar versions are pinned for compatibility with `tree-sitter@0.21`; the reference uses newer Python and Rust grammars. In particular, Rust `unsafe extern` blocks and async closures currently produce syntax-recovery warnings in Slopdex, so those cases are not covered by the passing parity fixtures.
227
+ Saved failures trigger stderr warnings, including on cached runs, help, and cross-search targets. `--ignore-errors` silences warnings without clearing records. Updates retry failed files; successful indexing, deletion, or exclusion clears their diagnostics. Version output bypasses diagnostics.
292
228
 
293
- After upgrading parser behavior, explicitly reparse previously indexed files with `slopdex update-files <path...>` to refresh their symbols, signatures, and embeddings even when the file contents are unchanged.
229
+ Use the recovery flag named in the error: `--rebuild-on-divergence` for Git history changes, `--force-reindex` for incompatible indexes. For provider/authentication failures, fix the reported configuration. For source-change-during-indexing errors, rerun after edits settle. Exit status is `0` on success, `2` for argument/domain errors, and `1` for other failures (or invocation without a command).
294
230
 
295
231
  ## Configuration
296
232
 
297
- Configuration is optional. Without `.slopdex/config.json`, Slopdex uses OpenAI's `text-embedding-3-large` model with 3072 dimensions and the built-in source exclusions.
298
-
299
- Create `.slopdex/config.json` to change these settings:
233
+ Optional file: `<root>/.slopdex/config.json`. Example using Jina (requires `JINA_API_KEY`):
300
234
 
301
235
  ```json
302
236
  {
@@ -307,4 +241,18 @@ Create `.slopdex/config.json` to change these settings:
307
241
  }
308
242
  ```
309
243
 
310
- Use `OPENAI_API_KEY` for OpenAI and `JINA_API_KEY` for Jina AI. The provider, model, dimensions, and embedding strategy define the index profile. Rebuild the index with `--force-reindex` after changing the profile.
244
+ | Property | Purpose / default |
245
+ | --- | --- |
246
+ | `provider`, `model`, `dimensions` | Embedding settings; defaults are listed in the CLI table. |
247
+ | `summaryModel` | Summary model; initially `gpt-5.6-sol`. |
248
+ | `indexPath` | Index location; `<root>/.slopdex/index.sqlite`. |
249
+ | `include` | Repository-relative glob array; empty/unset includes all supported eligible files. |
250
+ | `exclude` | Additional repository-relative exclusion globs. |
251
+ | `maxFileSize` | Maximum source-file size in bytes; positive integer, default `1048576`. |
252
+ | `embeddingBatchSize` | Embedding inputs per batch; positive integer, default `32`. |
253
+
254
+ Keep keys in the environment (`OPENAI_API_KEY`, `JINA_API_KEY`). Changing the embedding profile requires rebuilding with `--force-reindex`.
255
+
256
+ ## Development
257
+
258
+ - [Implementation and library API](docs/implementation.md)