@ninjaxtools/slopdex 0.8.0 → 0.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (6) hide show
  1. package/.agents/skills/slopdex/SKILL.md +181 -258
  2. package/README.md +177 -207
  3. package/dist/cli.js +8 -0
  4. package/dist/cli.js.map +1 -1
  5. package/docs/implementation.md +161 -0
  6. package/package.json +2 -1
package/README.md CHANGED
@@ -1,302 +1,257 @@
1
1
  # slopdex
2
2
 
3
- Slopdex uses Tree-sitter to index named functions in Python, JavaScript, JSX, TypeScript, TSX, Rust, Go, Java, and C repositories. It uses embeddings to search code by meaning, find similar implementations, and measure whether related functions are stored near each other.
3
+ Search functions by meaning, find duplicate-code candidates, and locate related functions spread across a codebase. Supports Python, JavaScript/JSX, TypeScript/TSX, Rust, Go, Java, and C in the same repository.
4
4
 
5
- ## Install
5
+ ## Start here
6
6
 
7
- ```bash
8
- npm install -g @ninjaxtools/slopdex
9
- ```
10
-
11
- ## Getting Started
12
-
13
- Set your OpenAI API key:
7
+ Requires **Node.js 24+** and an embedding-provider API key. Install, set your key, and run commands from the repository you want to analyze (or pass `--root /path/to/repo`):
14
8
 
15
9
  ```bash
10
+ npm install -g @ninjaxtools/slopdex
16
11
  export OPENAI_API_KEY="your-api-key"
17
- ```
18
-
19
- Run a semantic search from the repository you want to analyze:
20
-
21
- ```bash
22
12
  slopdex search "validate an authenticated session" --format summary --limit 10
23
13
  ```
24
14
 
25
- ## Analysis Examples
26
-
27
- ### Search By Function Purpose
15
+ The first command creates the index automatically. Later commands refresh it before searching. Add `.slopdex/` to your repository's `.gitignore`.
28
16
 
29
- Enable optional purpose summaries for every indexed callable:
17
+ ### Find code by purpose
30
18
 
31
19
  ```bash
32
- slopdex use-summaries
33
- slopdex search-summary "keep the repository index synchronized" --limit 10 --format summary
20
+ slopdex search "keep the repository index synchronized" --format summary --limit 10
34
21
  ```
35
22
 
36
- Summaries describe a function's responsibility and role in its codebase, using its repository name, path, source, and surrounding file context. Generation uses the OpenAI Responses API with **`gpt-5.6-sol`** by default and requires `OPENAI_API_KEY`, including when Jina supplies the embeddings.
37
-
38
- `use-summaries` stores each summary and its embedding in SQLite and enables a persistent repository-index setting. Subsequent indexing operations automatically generate summaries for new and changed files, including changes to surrounding context and file paths. Unchanged inputs reuse cached summaries and vectors; deleted functions disappear from summary search. Running `use-summaries` again is a no-op when summaries are already complete. Summary generation and embeddings are saved atomically, so a failed request does not leave partially updated callables.
39
-
40
- `search-summary` uses a separate summary embedding store with the configured embedding provider and supports the same query, limit, threshold, and output options as `search`. JSON results include the summary; `--format summary` prints it alongside each match. `status` reports `summariesEnabled`, `summaryCount`, and `summaryProfile`.
41
-
42
- To choose a different summary model, run `slopdex use-summaries --summary-model <model-id>` or set `summaryModel` in `.slopdex/config.json`. The chosen model is persisted for future updates. Existing indexes migrate automatically; summaries remain disabled until enabled explicitly.
43
-
44
- ### Combined Code And Purpose Analysis
45
-
46
- Once summaries are enabled and every indexed callable has a summary embedding, `cross-search` and `cohesion` automatically combine implementation and purpose similarity:
47
-
48
- ```text
49
- similarity = 0.5 * codeSimilarity + 0.5 * summarySimilarity
50
- ```
23
+ Describe the behavior you need. Results show similarity, file paths, and qualified function names. To search generated descriptions of each function's role instead:
51
24
 
52
25
  ```bash
53
26
  slopdex use-summaries
54
- slopdex cross-search --threshold 0.8 --format json
55
- slopdex cohesion --threshold 0.8 --format summary
27
+ slopdex search-summary "keep the repository index synchronized" --format summary --limit 10
56
28
  ```
57
29
 
58
- Both cosine scores and their average are calculated in one SQLite query. Thresholds (including half-open ranges), ranking, and neighbor/result limits apply to the combined score. Cohesion uses these combined neighbors for reciprocity, gap scores, file affinities, and groups.
59
-
60
- For cross-repository search, both indexes must have complete, enabled summaries. If either index lacks them, the entire analysis uses code-only similarity. Cohesion likewise uses code-only scoring if its summary index is incomplete or disabled. The existing embedding profiles must match across repositories; summary-generator models may differ.
61
-
62
- Combined JSON matches and cohesion pairs expose `codeSimilarity` and `summarySimilarity` alongside `similarity`. Cross-search rows include a `scoring` object with `similarityMode`, `similarityWeights`, and both summary-generator profiles. Cohesion records the mode and weights in `parameters` and the summary-generator profile in `repository.summaryProfile`. The mode is `"code-summary-average"` with weights `{ "code": 0.5, "summary": 0.5 }`, or `"code"` with weights `{ "code": 1, "summary": 0 }`. Text output labels combined scores.
63
-
64
- Compare analysis results only when the similarity mode and weights match, as well as the embedding profile, summary-generator profiles, thresholds, and analysis scope. Enabling summaries can change rankings and cohesion metrics. `search` continues to use only code vectors, and `search-summary` uses only summary vectors.
30
+ `use-summaries` enables persistent, automatic summary updates. It requires `OPENAI_API_KEY` even when Jina supplies embeddings, and adds generation costs. See [summaries and scoring](#summaries-and-scoring).
65
31
 
66
- ### Duplicate Analysis
67
-
68
- Compare functions in different files and include functions that span at least four lines:
32
+ ### Find duplicate-code candidates
69
33
 
70
34
  ```bash
71
- slopdex cross-search \
72
- --cross-file-only \
73
- --min-lines 4 \
74
- --threshold 0.9 \
75
- --limit 5
76
- ```
77
-
78
- The default output groups related functions into clusters:
79
-
80
- ```text
81
- Cluster 1 (3 functions, similarity 0.9124-0.9568)
82
- src/auth/session.ts:18:1 :: validateSession
83
- src/http/middleware.ts:42:1 :: authenticate
84
- src/users/user-service.ts:27:3 :: UserService.authenticate
35
+ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9 --limit 5
85
36
  ```
86
37
 
87
- A cluster contains functions connected by similarity matches. The similarity range covers the observed links in the cluster. Connected functions may be linked through another function, so review the source before deciding that code is duplicated.
88
-
89
- ### Cohesion Analysis
38
+ Compare functions across files, exclude short wrappers, and group strong matches into clusters. `--limit 5` selects up to five neighbors **per source function**, not five clusters. Review the source before consolidating a match.
90
39
 
91
- Find related functions that are separated across files and directories:
40
+ ### Review changed code or one module
92
41
 
93
42
  ```bash
94
- slopdex cohesion \
95
- --threshold 0.8 \
96
- --neighbors 20 \
97
- --limit 50 \
98
- --format summary
43
+ slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
44
+ slopdex cross-search --changed-since origin/main --format summary --threshold 0.9
45
+ slopdex cross-search --source-path src/services -e '^UserService\.' --format summary --threshold 0.9
99
46
  ```
100
47
 
101
- ```text
102
- Cohesion: 184 functions analyzed, 37 semantic edges
103
- same file 35.1% same folder 29.7% remote 35.2% mean distance 1.84
48
+ These select source functions while keeping the full eligible index available for matches. The same source filters work with `cohesion`. Combine filters to require all of them to match.
104
49
 
105
- 1. gap 0.6053 similarity 0.9400 distance 4 reciprocal
106
- src/auth/session.ts:18:1 :: validateSession
107
- packages/http/middleware.ts:42:1 :: authenticate
108
- ```
109
-
110
- The summary divides semantic relationships into the same file, the same folder, and different folders. Mean distance increases when related functions are stored farther apart. The gap score combines semantic similarity with path distance and ranks pairs for review. `reciprocal` means both functions are among each other's nearest semantic matches.
111
-
112
- Cohesion does not have a universal pass threshold. Compare results only when the similarity mode and weights, embedding and summary-generator profiles, threshold, neighbor count, and source scope are the same.
113
-
114
- ## Limit Analysis Scope
115
-
116
- Restrict source functions to a file or directory:
50
+ ### Find related code stored far apart
117
51
 
118
52
  ```bash
119
- slopdex cross-search --source-path src/services --format summary
120
- slopdex cohesion --source-path src/services --format summary
53
+ slopdex cohesion --threshold 0.8 --neighbors 20 --limit 50 --format summary
121
54
  ```
122
55
 
123
- Analyze functions added, changed, or moved since a commit:
56
+ Ranks semantically related pairs by their physical separation. Use it to review module boundaries; it is not a pass/fail architecture check.
124
57
 
125
- ```bash
126
- slopdex cross-search --changed-since origin/main --format summary
127
- slopdex cohesion --changed-since origin/main --format summary
128
- ```
129
-
130
- Analyze functions in files with uncommitted changes:
131
-
132
- ```bash
133
- slopdex cross-search --uncommitted --format summary
134
- slopdex cohesion --uncommitted --format summary
135
- ```
136
-
137
- These options restrict the source functions. Slopdex still compares them with the full index.
138
-
139
- Filter source symbols by qualified name with `-e <regex>` (or `--regexp <regex>`):
58
+ ### Compare repositories
140
59
 
141
60
  ```bash
142
- slopdex cross-search -e '^UserService\.' --format summary
143
- slopdex cohesion -e 'validate|authenticate' --source-path src/auth
61
+ slopdex cross-search \
62
+ --target-root /path/to/other/repo \
63
+ --target-index /path/to/other/repo/.slopdex/index.sqlite \
64
+ --threshold 0.9 --format summary
144
65
  ```
145
66
 
146
- The regex uses case-sensitive JavaScript syntax and matches qualified names such as `UserService.authenticate`. It filters only the source symbols for cross-search and cohesion; matching candidates are still drawn from the whole eligible index, including symbols that do not match the regex. It works with code-only and combined code/summary scoring.
67
+ Both indexes refresh automatically and must use identical embedding profiles. The target is refreshed with the source command's embedding provider; target configuration supplies file-selection and summary settings. Use `--target-config` for a non-default target config.
147
68
 
148
- Source restrictions combine by intersection:
69
+ ### Inspect index health
149
70
 
150
71
  ```bash
151
- slopdex cross-search -e 'validate' \
152
- --source-path src \
153
- --changed-since origin/main \
154
- --uncommitted \
155
- --min-lines 4 \
156
- --cross-file-only
72
+ slopdex status
73
+ slopdex index-errors --format summary
74
+ slopdex --version
157
75
  ```
158
76
 
159
- With both Git filters, a source must have changed since the specified commit and belong to an uncommitted file. Path, name, and minimum-line filters narrow the selection further. Existing target restrictions such as `--min-lines` and `--cross-file-only` continue to apply. The separate `--regex` option filters **both** analysis sources and matching candidates.
77
+ `status` refreshes the index and reports coverage, profiles, checkpoint, and error counts. `index-errors` reads saved failures without refreshing or requiring credentials. `--version` prints the built package version.
160
78
 
161
- ## Output And Filters
79
+ ## Commands
162
80
 
163
- `search` supports `json` and `summary` output. `cross-search` supports `json`, `summary`, and `clusters` output. Use `--format` to select one.
81
+ Usage: `slopdex <command> [arguments] [options]`.
164
82
 
165
- For `search` and `search-summary`, `-e` restricts result symbols before the result limit is applied:
83
+ | Command | Purpose | Output |
84
+ | --- | --- | --- |
85
+ | `search <query>` | Search function code by meaning. Quote multiword queries. | JSON array; optional `summary` |
86
+ | `use-summaries` | Generate missing purpose summaries and enable automatic updates. Repeating with unchanged inputs reuses existing summaries. | JSON statistics |
87
+ | `search-summary <query>` | Search purpose summaries after enabling them. | JSON array; optional `summary` including summary text |
88
+ | `cross-search` | Find neighbors for each selected function in this or another index. | `clusters` by default; optional `summary` or JSONL |
89
+ | `cohesion` | Analyze semantic relationships versus file/folder separation. | JSON report; optional `summary` |
90
+ | `status` | Refresh and show index metadata, counts, and profiles. | JSON object |
91
+ | `index-errors` | Read saved file/function indexing failures. | JSON array; optional `summary` |
92
+ | `update-git` | Explicitly refresh a Git snapshot, with current working-tree changes when targeting HEAD. | JSON update statistics |
93
+ | `update-files <path...>` | After automatic refresh, explicitly reparse selected working-tree files. Paths are repository-relative or absolute within the root. | JSON update statistics |
94
+ | `delete-files <path...>` | After automatic refresh, remove paths from the index; source files are not deleted. A later refresh can restore eligible files. | JSON update statistics |
95
+
96
+ Manual maintenance examples:
166
97
 
167
98
  ```bash
168
- slopdex search "validate session" -e '^Session\.' --limit 10
169
- slopdex search-summary "persist user data" -e 'save|persist' --limit 5
99
+ slopdex update-git
100
+ slopdex update-files src/service.ts src/model.ts
101
+ slopdex delete-files src/removed.ts
102
+ slopdex update-git --target HEAD --rebuild-on-divergence
103
+ slopdex update-git --force-reindex
170
104
  ```
171
105
 
172
- Use `--threshold` with a minimum similarity or a range:
106
+ ## Command-line arguments
173
107
 
174
- ```bash
175
- slopdex search "validate session" --threshold 0.8 --format summary
176
- slopdex cross-search --threshold 0.85-0.9 --format summary
177
- ```
108
+ Options are command-specific where indicated. Boolean flags default to off.
178
109
 
179
- For a range, the lower bound is included and the upper bound is excluded. Similarity values depend on the embedding model, so use them to rank results from the same index profile.
110
+ ### Location, providers, and diagnostics
180
111
 
181
- Run `slopdex --help` for all commands and options.
112
+ | Argument | Meaning / default |
113
+ | --- | --- |
114
+ | `--root <path>` | Repository root; current directory by default. |
115
+ | `--config <path>` | Config file; `<root>/.slopdex/config.json` by default. |
116
+ | `--index <path>` | Index file; `<root>/.slopdex/index.sqlite` by default. Overrides `indexPath` in config. |
117
+ | `--provider <openai\|jina>` | Embedding provider; `openai` by default. |
118
+ | `--model <name>` | Embedding model; `text-embedding-3-large` for OpenAI, `jina-embeddings-v4` for Jina. |
119
+ | `--dimensions <number>` | Positive embedding dimension count; OpenAI `3072`, Jina `1024`. Must be supported by the model. |
120
+ | `--summary-model <name>` | OpenAI summary model; `gpt-5.6-sol` initially, then the persisted model unless overridden. |
121
+ | `--ignore-errors` | Silence warnings about saved indexing errors; records remain available. |
122
+ | `-h`, `--help` | Show CLI usage without refreshing. |
123
+ | `--version` | Print the package version and exit. |
182
124
 
183
- ## Indexing And Data
125
+ Explicit relative config and index paths resolve from the current directory, not `--root`. CLI settings override config settings.
184
126
 
185
- ### Supported Languages
127
+ ### Search and analysis
186
128
 
187
- Languages are detected automatically from file extensions. A single index can contain multiple languages; search, purpose summaries, cross-search, and cohesion work across all of them.
188
-
189
- | Language | Extensions | Extracted callables |
129
+ | Argument | Applies to | Meaning / default |
190
130
  | --- | --- | --- |
191
- | Python | `.py`, `.pyw` | Functions, async functions, class methods, constructors, generators, and bound lambdas; includes decorators in source |
192
- | JavaScript | `.js`, `.mjs`, `.cjs` | Functions, generators, methods, constructors, and named function expressions/arrows |
193
- | JSX | `.jsx` | JavaScript callables, including components returning JSX |
194
- | TypeScript | `.ts`, `.mts`, `.cts` | Typed functions, methods, constructors, and named function expressions/arrows |
195
- | TSX | `.tsx` | TypeScript callables, including generic components returning JSX |
196
- | Rust | `.rs` | Functions, `impl` methods/associated functions, trait default methods, and `let`-bound closures |
197
- | Go | `.go` | Functions, receiver methods, and function literals bound to variables or assignments |
198
- | Java | `.java` | Methods, constructors (including compact record constructors), and variable-bound lambdas |
199
- | C | `.c`, `.h` | Function definitions, including static/inline functions and functions returning pointers |
200
-
201
- Qualified names include enclosing classes, functions, and explicit modules. Go methods include their receiver type (`Store[T].Get`); Rust trait implementations include the type and trait (`<Store<T> as Read>.read`). Definitions retain their source, signature, line/column locations, and stable identity for incremental updates. Declarations without bodies and anonymous callbacks are omitted. Extraction is syntactic: Rust macros and C macros are not expanded, and C preprocessor branches are indexed as written. `.h` files use the C grammar.
202
-
203
- Tree-sitter grammars ship as package dependencies; no language server or project compiler configuration is required. Syntax errors produce warnings while recoverable callables are still indexed. Existing indexes pick up newly supported files during the next normal refresh.
204
-
205
- Default exclusions cover dependency and build directories: `.git`, `.slopdex`, `node_modules`, `dist`, `build`, `coverage`, `vendor`, `generated`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, and `target`. Optional `include` and `exclude` glob arrays in `.slopdex/config.json` further restrict the indexed files.
206
-
207
- Slopdex also respects root and nested `.gitignore` files using the [`ignore`](https://www.npmjs.com/package/ignore) package. Patterns, directory rules, anchoring, escapes, and `!` exceptions follow Git ignore semantics. A nested rule cannot re-include files beneath an excluded parent directory. These exclusions apply even to tracked files and when Git is unavailable; explicit `update-files` requests for ignored files are rejected.
208
-
209
- Working-tree indexing and HEAD overlays use the current `.gitignore` files. Historical snapshots and committed-only Git indexing use the ignore files stored in the selected commit. Normal refreshes remove previously indexed files that become ignored and discover files that become eligible again, including when only ignore rules change. Configuration includes and negated ignore rules cannot override the built-in exclusions.
131
+ | `--limit <number>` | Both query searches, cross-search, cohesion | Positive integer. Query matches: `10`; cross-search neighbors per source: `5`; cohesion reported pairs and file rows: `50`. |
132
+ | `--threshold <number\|min-max>` | Both query searches, cross-search, cohesion | Minimum similarity, or range with inclusive minimum and exclusive maximum. Default `-1` for query/cross-search, `0.8` for cohesion. Cohesion minimum must be at least `-1` and below `1`. |
133
+ | `--format <json\|summary\|clusters>` | Both query searches, cross-search, cohesion, index-errors | Output format; see the commands table. `clusters` is only for cross-search. |
134
+ | `-e <regex>`, `--regexp <regex>` | Both query searches, cross-search, cohesion | Case-sensitive JavaScript regex over qualified names. Query searches: filters results before limiting. Analysis: filters sources only. |
135
+ | `--regex <regex>` | Cross-search, cohesion | Filter **both** source and candidate qualified names. |
136
+ | `--min-lines <number>` | Cross-search, cohesion | Minimum source and candidate callable length; positive integer, default `2`. Use `1` to include one-line wrappers. |
137
+ | `--source-path <path>` | Cross-search, cohesion | Select sources in a file or recursive directory, relative to the repository root (or absolute within it). |
138
+ | `--changed-since <commit>` | Cross-search, cohesion | Select added, modified, or moved functions relative to an ancestor of the indexed Git checkpoint, including working-tree changes. Requires Git. |
139
+ | `--uncommitted` | Cross-search, cohesion | Select functions indexed from working-tree files: staged, unstaged, or untracked changes in Git; all working-tree functions without Git. |
140
+ | `--cross-file-only` | Cross-search | Exclude matches from the same physical file. |
141
+ | `--include-symmetric-duplicates` | Cross-search | Allow both directions of same-index matches; otherwise each unordered pair is emitted once. |
142
+ | `--neighbors <number>` | Cohesion | Neighbors considered per source; positive integer, default `20`. Changes the analysis graph. |
143
+ | `--include-source` | Cohesion JSON | Include callable bodies; omitted by default. |
144
+ | `--target-root <path>` | Cross-search | Second repository root; requires `--target-index`. |
145
+ | `--target-index <path>` | Cross-search | Second index file; requires `--target-root`. |
146
+ | `--target-config <path>` | Cross-search | Target config; defaults to `<target-root>/.slopdex/config.json`. Requires both target options. |
147
+
148
+ Source restrictions intersect: with both Git filters, a function must have changed since the commit **and** belong to an uncommitted file. `-e`, `--source-path`, and Git filters do not restrict candidates; `--regex` and `--min-lines` do.
149
+
150
+ Review adjacent similarity bands without repeating boundary matches:
210
151
 
211
- ### Index Updates
212
-
213
- Before each analysis command, Slopdex updates its index from the current Git commit and the staged, unstaged, and untracked files in the working tree. The index is created at `.slopdex/index.sqlite` by default.
152
+ ```bash
153
+ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9 --limit 5
154
+ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9 --limit 5
155
+ ```
214
156
 
215
- Add `.slopdex/` to the repository's `.gitignore` so the local index is not committed.
157
+ ### Refresh and recovery
216
158
 
217
- Slopdex sends extracted function source to the configured embedding provider. The function metadata and embeddings are stored in the local SQLite index.
159
+ | Argument | Meaning |
160
+ | --- | --- |
161
+ | `--target <ref>` | Git snapshot for `update-git`; default `HEAD`. Non-HEAD targets exclude working-tree changes. Later commands normally refresh back to HEAD. |
162
+ | `--rebuild-on-divergence` | Allow reconciliation when the saved checkpoint is not an ancestor of the target, such as after a rebase or branch switch. |
163
+ | `--force-reindex` | Recreate an **incompatible** index (repository, provider, model, dimensions, strategy, or schema mismatch). A compatible index still follows normal refresh behavior. |
164
+ | `--no-reindex` | With Git, still reconcile the committed snapshot but skip working-tree overlays. Without Git, reuse a non-empty index; missing/empty indexes are still populated. Not a general offline switch. |
218
165
 
219
- Whole-repository cohesion analysis compares neighbors for every selected function. Use `--source-path`, `--changed-since`, or `--uncommitted` to reduce the scope in large repositories.
166
+ ## Reading results
220
167
 
221
- ### Inspecting Indexing Errors
168
+ ### Similarity and duplicate clusters
222
169
 
223
- Slopdex persists file and function indexing diagnostics in the SQLite `indexing_errors` table. Parse errors, parser exceptions, callable-extraction failures, file-read failures, and files exceeding `maxFileSize` retain references instead of silently disappearing. Healthy functions in partially parsed files and other healthy files remain searchable; malformed callables are omitted from semantic search and listed in the diagnostics.
170
+ Similarity is a model-dependent score, not a probability of duplication. Higher scores mean greater semantic resemblance. Query summaries show `score path :: qualifiedName`; cross-search summaries group those lines beneath each source. Functions without matches are omitted from cross-search output.
224
171
 
225
- ```bash
226
- slopdex index-errors
227
- slopdex index-errors --format summary
228
- slopdex index-errors --index /path/to/another/index.sqlite
172
+ ```text
173
+ Cluster 1 (3 functions, similarity 0.9124-0.9568)
174
+ src/auth/session.ts:18:1 :: validateSession
175
+ src/http/middleware.ts:42:1 :: authenticate
176
+ src/users/user-service.ts:27:3 :: UserService.authenticate
229
177
  ```
230
178
 
231
- This command reads the saved diagnostics without refreshing the index or calling an embedding/summary provider, so it works without API credentials. JSON includes the path, language, error code and message, file/function scope, recovered qualified name when available, line/column ranges, available source text, and Git/working-tree provenance. If Tree-sitter cannot identify a function, the unparsed region is still retained as a file diagnostic.
179
+ - A cluster groups functions connected by matches. Its range covers observed links; not every pair necessarily matches directly.
180
+ - Clusters sort by member count, then name. Cluster number is not severity.
181
+ - Locations identify where to inspect behavior, callers, and architectural roles. Wrappers, adapters, tests, and separate interface implementations can legitimately resemble one another.
232
182
 
233
- Every invocation warns on stderr while saved errors remain, including cached runs and cross-search target indexes. Suppress this warning with `--ignore-errors`:
183
+ ### Cohesion
234
184
 
235
- ```bash
236
- slopdex cross-search --ignore-errors
237
- slopdex index-errors --format summary --ignore-errors
185
+ ```text
186
+ Cohesion: 184 functions analyzed, 37 semantic edges
187
+ same file 35.1% same folder 29.7% remote 35.2% mean distance 1.84
188
+
189
+ 1. gap 0.6053 similarity 0.9400 distance 4 reciprocal
190
+ src/auth/session.ts:18:1 :: validateSession
191
+ packages/http/middleware.ts:42:1 :: authenticate
238
192
  ```
239
193
 
240
- Silencing warnings does not delete diagnostics. `status` reports `indexingErrorCount` and `failedFileCount`; `functionCount` counts searchable callables. Normal updates retry failed files, including unchanged Git blobs. Errors clear when a file is successfully indexed, deleted, or excluded. Existing indexes migrate automatically and rescan once on their next full update to detect previously unreported parse failures. Diagnostics are committed atomically with the corresponding file update.
194
+ | Field | Interpretation |
195
+ | --- | --- |
196
+ | Functions analyzed / semantic edges | Selected-source coverage / unique qualifying neighbor pairs, before report limiting. Not quality scores. |
197
+ | Same file / same folder / remote | Shares of weighted semantic affinity. Higher remote affinity means more related code crosses folder boundaries. |
198
+ | Mean distance | Weighted physical separation: `0` for the same file, `1` for different files in one folder, larger across folders. |
199
+ | Gap / rank | A `0–1` review score combining similarity above the threshold and separation; higher gap ranks first. Same-file pairs have zero gap. |
200
+ | Reciprocal | Both functions selected each other as neighbors. JSON `null` means the other endpoint was not evaluated under source filtering. |
201
+ | `sourceTestPair` | A source/test relationship inferred from paths; separation may be intentional. |
202
+ | `externalAffinityRatio` | In JSON file reports, the share of observed affinity outside that file's folder. |
241
203
 
242
- ## Library
204
+ The example suggests reviewing separated authentication responsibilities. It does not establish that they belong in one module. There is no universal cohesion pass threshold. Filtered reports describe selected sources, not the entire repository. Summary metrics cover all qualifying edges; reported pairs/files are limited, and groups are built from reported pairs.
243
205
 
244
- The package also exports the index and analysis APIs:
206
+ For automation, query searches and diagnostics return JSON arrays; cross-search returns **JSONL**, one row per matched source; cohesion returns one JSON object containing `repository`, `parameters`, `summary`, `pairs`, `files`, and `groups`. Results go to stdout; notices and warnings go to stderr.
245
207
 
246
- ```ts
247
- import {
248
- OpenAIEmbeddingProvider,
249
- openCodeIndex,
250
- } from "@ninjaxtools/slopdex";
208
+ ## System behavior
251
209
 
252
- const index = openCodeIndex({
253
- rootDir: "/path/to/repository",
254
- provider: new OpenAIEmbeddingProvider(),
255
- });
210
+ ### Freshness and cost
256
211
 
257
- await index.updateFromGit();
212
+ - Indexing, search, analysis, `use-summaries`, and `status` automatically refresh. With Git, results normally reflect HEAD plus staged, unstaged, and untracked working-tree contents. `gitCheckpoint` records the committed base, not the overlay.
213
+ - Without Git, commands warn and scan the working tree. `--no-reindex` has the limited behavior described above.
214
+ - `index-errors`, help, and version do not refresh. Other commands require the configured embedding key, even when cached data supplies the analysis.
215
+ - Function source and search queries go to the embedding provider. Source metadata, vectors, diagnostics, and enabled summaries stay in the local index. Summary generation also sends repository name, path, callable source, and surrounding file context to OpenAI; summary text goes to the embedding provider.
216
+ - Unchanged embedding and summary inputs reuse cached results. Initial indexing and enabling summaries can make many API calls. Source filters reduce analysis work, not the preceding index refresh.
258
217
 
259
- const results = await index.similaritySearch({
260
- query: "validate an authenticated session",
261
- limit: 10,
262
- });
218
+ ### Summaries and scoring
263
219
 
264
- index.close();
265
- ```
220
+ Summaries are optional and disabled initially. `use-summaries` persists the selected summary model and keeps summaries current on later updates, including file-context and path changes. Use `slopdex use-summaries --summary-model <model-id>` to change it. `status` exposes `summariesEnabled`, `summaryCount`, and `summaryProfile`.
266
221
 
267
- Exports also include `crossSearch`, `analyzeCohesion`, `JinaEmbeddingProvider`, and standalone functions for index updates and searches.
222
+ `search` always searches code; `search-summary` always searches purpose summaries. When all callables have enabled summaries, cross-search and cohesion automatically use **50% code similarity + 50% summary similarity**. Cross-repository analysis needs complete summaries on both sides; otherwise the entire analysis uses code-only scores. Thresholds and neighbor limits apply to the selected score.
268
223
 
269
- Library callers can set `sourceFilter.nameRegex` on cross-search/cohesion options for source-only filtering, together with `path` and the Git filter type. A `changed-since` filter also accepts `uncommitted: true`. Query searches accept `nameRegex` in `SimilaritySearchOptions` to filter result symbols. Cohesion reports record source restrictions in `parameters.sourceFilter` and label filtered analyses as `selected-sources`.
224
+ Text output labels combined scores. JSON exposes `codeSimilarity`, `summarySimilarity`, and scoring mode/weights (`scoring` for cross-search, `parameters` for cohesion). Compare runs only with matching scoring mode, weights, embedding and summary-generator profiles, threshold, neighbor count, and source/candidate filters.
270
225
 
271
- Use `index.indexErrors()` to inspect diagnostics through an open index, or the exported `readIndexErrors(indexPath)` to inspect a saved database without a provider. Error records use the exported `IndexingError` type.
226
+ ### Coverage and exclusions
272
227
 
273
- For summary search, call `await index.useSummaries()` after updating the index, then `await index.searchSummary({ query: "maintain the repository index" })`. Optionally pass `summaryProvider: new OpenAISummaryProvider({ model: "gpt-5.6-sol" })` when opening an index. Custom providers implement the exported `SummaryProvider` interface. The package also exports standalone `useSummaries` and `searchSummary` helpers.
228
+ | Language | File extensions |
229
+ | --- | --- |
230
+ | Python | `.py`, `.pyw` |
231
+ | JavaScript / JSX | `.js`, `.mjs`, `.cjs`, `.jsx` |
232
+ | TypeScript / TSX | `.ts`, `.mts`, `.cts`, `.tsx` |
233
+ | Rust | `.rs` |
234
+ | Go | `.go` |
235
+ | Java | `.java` |
236
+ | C | `.c`, `.h` |
274
237
 
275
- ## Development
238
+ Indexes named functions, methods, constructors, and supported variable-bound closures with bodies, including nested callables. Anonymous callbacks and bodyless declarations are omitted. No language server or project compiler setup is needed. Macro expansion and runtime behavior are not analyzed; `.h` files are treated as C.
276
239
 
277
- ```bash
278
- npm run check
279
- ```
240
+ Root and nested `.gitignore` rules apply even to tracked files and without Git. Working-tree refreshes use current rules; committed-only snapshots use the target commit's rules. Refresh removes newly excluded files and discovers newly eligible ones. Explicit `update-files` rejects ignored files.
280
241
 
281
- To cross-check the shared-language fixtures against a built sibling checkout of `treesitter-index`, run:
242
+ Built-in exclusions: `.git`, `.slopdex`, `node_modules`, `dist`, `build`, `coverage`, `vendor`, `generated`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, and `target`. Config `include`/`exclude` globs narrow coverage; they cannot override built-in exclusions. Ignore exceptions cannot re-include files beneath an excluded parent directory. Files over 1 MiB are skipped unless `maxFileSize` is raised.
282
243
 
283
- ```bash
284
- npm run check:parser-parity
285
- # Or provide another reference executable:
286
- npm run check:parser-parity -- /path/to/treesitter-index
287
- ```
244
+ ### Failures and recovery
288
245
 
289
- The default reference is `../treesitter-index/target/debug/treesitter-index`. The check covers eight shared languages; that project has no C grammar. Callable regression tests also run as part of `npm run check` without requiring the sibling checkout.
246
+ Parse, extraction, read, and file-size failures are saved while healthy callables remain searchable. Inspect them with `slopdex index-errors --format summary`. JSON includes paths, locations, recoverable names, messages, available source, and snapshot provenance. `status` reports `indexingErrorCount` and `failedFileCount`; `functionCount` counts searchable callables.
290
247
 
291
- The projects index different information: `treesitter-index` includes declarations, types, imports, and `.pyi` stubs, while Slopdex indexes callable implementations (including nested callables and bound closures). Slopdex also supports `.pyw` and C. Its native Node grammar versions are pinned for compatibility with `tree-sitter@0.21`; the reference uses newer Python and Rust grammars. In particular, Rust `unsafe extern` blocks and async closures currently produce syntax-recovery warnings in Slopdex, so those cases are not covered by the passing parity fixtures.
248
+ Saved failures trigger stderr warnings, including on cached runs, help, and cross-search targets. `--ignore-errors` silences warnings without clearing records. Updates retry failed files; successful indexing, deletion, or exclusion clears their diagnostics. Version output bypasses diagnostics.
292
249
 
293
- After upgrading parser behavior, explicitly reparse previously indexed files with `slopdex update-files <path...>` to refresh their symbols, signatures, and embeddings even when the file contents are unchanged.
250
+ Use the recovery flag named in the error: `--rebuild-on-divergence` for Git history changes, `--force-reindex` for incompatible indexes. For provider/authentication failures, fix the reported configuration. For source-change-during-indexing errors, rerun after edits settle. Exit status is `0` on success, `2` for argument/domain errors, and `1` for other failures (or invocation without a command).
294
251
 
295
252
  ## Configuration
296
253
 
297
- Configuration is optional. Without `.slopdex/config.json`, Slopdex uses OpenAI's `text-embedding-3-large` model with 3072 dimensions and the built-in source exclusions.
298
-
299
- Create `.slopdex/config.json` to change these settings:
254
+ Optional file: `<root>/.slopdex/config.json`. Example using Jina (requires `JINA_API_KEY`):
300
255
 
301
256
  ```json
302
257
  {
@@ -307,4 +262,19 @@ Create `.slopdex/config.json` to change these settings:
307
262
  }
308
263
  ```
309
264
 
310
- Use `OPENAI_API_KEY` for OpenAI and `JINA_API_KEY` for Jina AI. The provider, model, dimensions, and embedding strategy define the index profile. Rebuild the index with `--force-reindex` after changing the profile.
265
+ | Property | Purpose / default |
266
+ | --- | --- |
267
+ | `provider`, `model`, `dimensions` | Embedding settings; defaults are listed in the CLI table. |
268
+ | `summaryModel` | Summary model; initially `gpt-5.6-sol`. |
269
+ | `indexPath` | Index location; `<root>/.slopdex/index.sqlite`. |
270
+ | `include` | Repository-relative glob array; empty/unset includes all supported eligible files. |
271
+ | `exclude` | Additional repository-relative exclusion globs. |
272
+ | `maxFileSize` | Maximum source-file size in bytes; positive integer, default `1048576`. |
273
+ | `embeddingBatchSize` | Embedding inputs per batch; positive integer, default `32`. |
274
+
275
+ Keep keys in the environment (`OPENAI_API_KEY`, `JINA_API_KEY`). Changing the embedding profile requires rebuilding with `--force-reindex`.
276
+
277
+ ## Agent and developer documentation
278
+
279
+ - [Agent skill](.agents/skills/slopdex/SKILL.md): task-oriented CLI guidance for coding agents.
280
+ - [Implementation and library API](docs/implementation.md): internals, programmatic usage, build, and development checks.
package/dist/cli.js CHANGED
@@ -2907,6 +2907,7 @@ var parsed = (() => {
2907
2907
  "force-reindex": { type: "boolean", default: false },
2908
2908
  "no-reindex": { type: "boolean", default: false },
2909
2909
  "ignore-errors": { type: "boolean", default: false },
2910
+ version: { type: "boolean", default: false },
2910
2911
  help: { type: "boolean", short: "h", default: false }
2911
2912
  }
2912
2913
  });
@@ -2917,6 +2918,12 @@ var parsed = (() => {
2917
2918
  }
2918
2919
  })();
2919
2920
  var [command, ...positionals] = parsed.positionals;
2921
+ if (parsed.values.version) {
2922
+ const version = true ? "0.9.0" : JSON.parse(readFileSync(new URL("../package.json", import.meta.url), "utf8")).version;
2923
+ process.stdout.write(`${version}
2924
+ `);
2925
+ process.exit(0);
2926
+ }
2920
2927
  var diagnosticIndexes = /* @__PURE__ */ new Set();
2921
2928
  process.on("exit", () => {
2922
2929
  if (parsed.values["ignore-errors"]) return;
@@ -3520,6 +3527,7 @@ Reading Analysis Output:
3520
3527
  Higher means more of a file's related affinity lies outside its folder
3521
3528
 
3522
3529
  Options:
3530
+ --version Show the package version
3523
3531
  --root <path> Repository root (default: current directory)
3524
3532
  --config <path> Config file (default: .slopdex/config.json)
3525
3533
  --index <path> SQLite index path