@ninjaxtools/slopdex 0.15.0 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +91 -224
- package/dist/chunk-5BO2LLXC.js +15 -0
- package/dist/chunk-5BO2LLXC.js.map +1 -0
- package/dist/chunk-BXWO2KMC.js +20 -0
- package/dist/chunk-BXWO2KMC.js.map +1 -0
- package/dist/chunk-DKRB2XT5.js +33 -0
- package/dist/chunk-DKRB2XT5.js.map +1 -0
- package/dist/chunk-DTI7SXGL.js +109 -0
- package/dist/chunk-DTI7SXGL.js.map +1 -0
- package/dist/chunk-GBEJHETS.js +1117 -0
- package/dist/chunk-GBEJHETS.js.map +1 -0
- package/dist/chunk-HMKVMRLF.js +2221 -0
- package/dist/chunk-HMKVMRLF.js.map +1 -0
- package/dist/chunk-IQU3YVZ3.js +85 -0
- package/dist/chunk-IQU3YVZ3.js.map +1 -0
- package/dist/chunk-JAVPHCZH.js +24 -0
- package/dist/chunk-JAVPHCZH.js.map +1 -0
- package/dist/chunk-MKUTTXZB.js +74 -0
- package/dist/chunk-MKUTTXZB.js.map +1 -0
- package/dist/chunk-OIKBO3NJ.js +133 -0
- package/dist/chunk-OIKBO3NJ.js.map +1 -0
- package/dist/chunk-P57ZJREE.js +426 -0
- package/dist/chunk-P57ZJREE.js.map +1 -0
- package/dist/chunk-S225GYCL.js +79 -0
- package/dist/chunk-S225GYCL.js.map +1 -0
- package/dist/chunk-TSURHRFF.js +240 -0
- package/dist/chunk-TSURHRFF.js.map +1 -0
- package/dist/chunk-VH5VGRCI.js +103 -0
- package/dist/chunk-VH5VGRCI.js.map +1 -0
- package/dist/cli.js +218 -3590
- package/dist/cli.js.map +1 -1
- package/dist/code-index-MWSJKV3G.js +14 -0
- package/dist/code-index-MWSJKV3G.js.map +1 -0
- package/dist/cross-search-ERBFC25T.js +10 -0
- package/dist/cross-search-ERBFC25T.js.map +1 -0
- package/dist/database-P5XWJWQL.js +14 -0
- package/dist/database-P5XWJWQL.js.map +1 -0
- package/dist/hosted-GHIUHKOU.js +13 -0
- package/dist/hosted-GHIUHKOU.js.map +1 -0
- package/dist/index.d.ts +102 -2
- package/dist/index.js +38 -3615
- package/dist/index.js.map +1 -1
- package/dist/jina-43RL7C6M.js +12 -0
- package/dist/jina-43RL7C6M.js.map +1 -0
- package/dist/openai-CYUPNBGB.js +12 -0
- package/dist/openai-CYUPNBGB.js.map +1 -0
- package/dist/openai-EC6THXKV.js +21 -0
- package/dist/openai-EC6THXKV.js.map +1 -0
- package/dist/openai-TCRXM66O.js +11 -0
- package/dist/openai-TCRXM66O.js.map +1 -0
- package/docs/implementation.md +10 -19
- package/docs/reference.md +187 -0
- package/package.json +4 -4
- package/.agents/skills/slopdex/SKILL.md +0 -239
- package/scripts/install-opencode-skill.mjs +0 -11
package/README.md
CHANGED
|
@@ -1,297 +1,164 @@
|
|
|
1
|
+
This repository employs the use of LLMs for [automatic programming](https://antirez.com/news/159).
|
|
2
|
+
|
|
3
|
+
This readme is written by a human.
|
|
4
|
+
|
|
1
5
|
# slopdex
|
|
2
6
|
|
|
3
|
-
|
|
7
|
+
<img src="sloppy-dexter.png" width="280" align="right" alt="Dexter, the sloppy slime" />
|
|
8
|
+
|
|
9
|
+
Slopdex helps with doing analysis on codebases that contain a lot of AI generated code.
|
|
4
10
|
|
|
5
|
-
|
|
11
|
+
The main supported functions are
|
|
6
12
|
|
|
7
|
-
|
|
13
|
+
- `slopdex search <query>`
|
|
14
|
+
<br/>which finds code similar to the query
|
|
15
|
+
- `slopdex cross-search`
|
|
16
|
+
<br/>which finds clusters of similar code
|
|
8
17
|
|
|
9
|
-
|
|
18
|
+
`cross-search` helps with finding duplicated code, and with `--cohesion` helps with identifying similar code that is not necessarily duplicated but spread out across the codebase, which could indicate that a refactoring could make it more cohesive.
|
|
19
|
+
|
|
20
|
+
## Getting started
|
|
21
|
+
|
|
22
|
+
An embedding-provider API key is required. You can use either OpenAI or Jina.
|
|
10
23
|
|
|
11
24
|
```bash
|
|
12
25
|
npm install -g @ninjaxtools/slopdex
|
|
26
|
+
|
|
13
27
|
export OPENAI_API_KEY="your-api-key"
|
|
14
|
-
|
|
15
|
-
|
|
28
|
+
# Or
|
|
29
|
+
# export JINA_API_KEY="your-api-key" # and pass --provider jina
|
|
16
30
|
|
|
17
|
-
|
|
31
|
+
slopdex search "validate an authenticated session"
|
|
32
|
+
```
|
|
18
33
|
|
|
19
|
-
Add `.slopdex/` to your repository's `.gitignore
|
|
34
|
+
The index tracks the current git commit and is created or updated on every command and stored in `.slopdex/index.sqlite`. Add `.slopdex/` to your repository's `.gitignore` to prevent it from being committed.
|
|
20
35
|
|
|
21
36
|
### Search code
|
|
22
37
|
|
|
23
|
-
|
|
24
|
-
slopdex search "keep the repository index synchronized" --format summary --limit 10
|
|
25
|
-
```
|
|
26
|
-
|
|
27
|
-
Optionally enable a hosted second-stage reranker for `search` and `search-description`:
|
|
38
|
+
By default only vector embeddings of code is used for search.
|
|
28
39
|
|
|
29
40
|
```bash
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
41
|
+
$ slopdex search "keep the repository index synchronized"
|
|
42
|
+
0.4284 tests/languages.test.ts :: refresh
|
|
43
|
+
0.4200 src/cli.ts :: refreshIndex
|
|
44
|
+
0.4113 src/code-index.ts :: CodeIndex.updateFromGit
|
|
45
|
+
...
|
|
34
46
|
```
|
|
35
47
|
|
|
36
|
-
|
|
48
|
+
You can also enable optional description generation which will automatically generate file and function descriptions with a configured LLM provider and include them in the search.
|
|
37
49
|
|
|
38
|
-
|
|
50
|
+
I use OpenCode Go usually with DeepSeek or Muse Spark, which are fairly good low-cost models. If you
|
|
51
|
+
sign up for OpenCode Go through [this link](https://opencode.ai/go?ref=RAR3Z744DZ), we both receive
|
|
52
|
+
$5 in credit.
|
|
39
53
|
|
|
40
54
|
```bash
|
|
55
|
+
export OPENAI_API_KEY="your-api-key"
|
|
56
|
+
# Or
|
|
57
|
+
# export OPENCODE_API_KEY="your-api-key" # for OpenCode Zen/Go descriptions
|
|
58
|
+
# Or
|
|
59
|
+
# opencode auth login # use stored credentials instead of env variables
|
|
60
|
+
|
|
41
61
|
slopdex descriptions enable
|
|
42
|
-
slopdex search-description "keep the repository index synchronized"
|
|
62
|
+
slopdex search-description "keep the repository index synchronized"
|
|
43
63
|
```
|
|
44
64
|
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
`descriptions disable` turns off description generation while retaining cached data.
|
|
48
|
-
|
|
49
|
-
To choose from OpenCode's current published models and persist description settings without creating an index:
|
|
65
|
+
You can list and configure one of OpenCode's models like this:
|
|
50
66
|
|
|
51
67
|
```bash
|
|
52
68
|
slopdex models opencode-go
|
|
53
69
|
slopdex config model opencode-go/gpt-5.6-luna
|
|
54
|
-
slopdex config descriptions enable
|
|
55
70
|
```
|
|
56
71
|
|
|
57
|
-
|
|
58
|
-
You can also pass the selection separately as `slopdex config model --description-provider opencode-go --description-model gpt-5.6-luna`.
|
|
72
|
+
### Find duplicate code
|
|
59
73
|
|
|
60
|
-
|
|
74
|
+
Compare functions across files, exclude short wrappers, and group matches into clusters:
|
|
61
75
|
|
|
62
76
|
```bash
|
|
63
|
-
slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
|
|
77
|
+
$ slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
|
|
78
|
+
Cluster 1 (3 functions, similarity 0.9124-0.9568)
|
|
79
|
+
src/auth/session.ts:18:1 :: validateSession
|
|
80
|
+
src/http/middleware.ts:42:1 :: authenticate
|
|
81
|
+
src/users/user-service.ts:27:3 :: UserService.authenticate
|
|
82
|
+
...
|
|
64
83
|
```
|
|
65
84
|
|
|
66
|
-
|
|
85
|
+
Using `--cross-file-only` is useful to exclude similar code in the same file.
|
|
67
86
|
|
|
68
|
-
|
|
87
|
+
Review adjacent bands with threshold ranges:
|
|
69
88
|
|
|
70
89
|
```bash
|
|
71
|
-
slopdex cross-search --
|
|
72
|
-
slopdex cross-search --
|
|
73
|
-
slopdex cross-search --source-path src/services -e '^UserService\.' --format summary --threshold 0.9
|
|
90
|
+
slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.9
|
|
91
|
+
slopdex cross-search --cross-file-only --min-lines 4 --threshold 0.85-0.9
|
|
74
92
|
```
|
|
75
93
|
|
|
76
|
-
|
|
94
|
+
### Restrict functions used in the cross-search
|
|
77
95
|
|
|
78
|
-
|
|
96
|
+
Only use uncommitted working-tree functions as sources:
|
|
79
97
|
|
|
80
98
|
```bash
|
|
81
|
-
slopdex cross-search --
|
|
99
|
+
slopdex cross-search --uncommitted --cross-file-only --min-lines 4 --threshold 0.9
|
|
82
100
|
```
|
|
83
101
|
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
### Compare repositories
|
|
102
|
+
Only use functions changed since origin/main as sources:
|
|
87
103
|
|
|
88
104
|
```bash
|
|
89
|
-
slopdex cross-search
|
|
90
|
-
--target-root /path/to/other/repo \
|
|
91
|
-
--target-index /path/to/other/repo/.slopdex/index.sqlite \
|
|
92
|
-
--threshold 0.9 --format summary
|
|
105
|
+
slopdex cross-search --changed-since origin/main --threshold 0.9
|
|
93
106
|
```
|
|
94
107
|
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
### Inspect index health
|
|
108
|
+
Only use matching symbols under src/services as sources:
|
|
98
109
|
|
|
99
110
|
```bash
|
|
100
|
-
slopdex
|
|
101
|
-
slopdex index-errors --format summary
|
|
102
|
-
slopdex --version
|
|
111
|
+
slopdex cross-search --source-path src/services -e '^UserService\.' --threshold 0.9
|
|
103
112
|
```
|
|
104
113
|
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
## Commands
|
|
114
|
+
### Find related code stored far apart
|
|
108
115
|
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
| Command | Purpose | Output |
|
|
112
|
-
| --- | --- | --- |
|
|
113
|
-
| `models [opencode\|opencode-go]` | Fetch valid models from the current published Zen and/or Go catalogs. | Qualified `provider/model` lines; optional JSON array |
|
|
114
|
-
| `config model <model\|provider/model>` | Validate a published OpenCode model and persist its provider/model selection without opening an index. Bare IDs auto-resolve only when unambiguous. | Updated setting summary; optional JSON |
|
|
115
|
-
| `config descriptions <enable\|disable>` | Persist whether the next index-using command should enable or disable descriptions. Does not open an index. | Updated setting summary; optional JSON |
|
|
116
|
-
| `config reranker <cohere\|jina\|openai\|disable> [model]` | Enable a hosted or OpenAI LLM query reranker, optionally selecting a model, or disable it. OpenAI accepts `--reranker-candidates <number>` from 1 to 100 and defaults to 10. Does not open an index. | Updated setting summary; optional JSON |
|
|
117
|
-
| `search <query>` | Search function code by meaning. Quote multiword queries. | Summary; optional JSON array |
|
|
118
|
-
| `descriptions <enable\|disable>` | Enable or disable automatic purpose descriptions. Re-enabling with unchanged inputs reuses cached descriptions. | JSON statistics |
|
|
119
|
-
| `search-description <query>` | Search purpose descriptions after enabling them. | Summary including description text; optional JSON array |
|
|
120
|
-
| `cross-search` | Find neighbors for each selected function in this or another index. | `clusters` by default; optional `summary` or JSONL |
|
|
121
|
-
| `status` | Refresh and show index metadata, counts, and profiles. | JSON object |
|
|
122
|
-
| `index-errors` | Read saved file/function indexing failures. | Summary; optional JSON array |
|
|
123
|
-
| `update-git` | Explicitly refresh a Git snapshot, with current working-tree changes when targeting HEAD. | JSON update statistics |
|
|
124
|
-
| `update-files <path...>` | After automatic refresh, explicitly reparse selected working-tree files. Paths are repository-relative or absolute within the root. | JSON update statistics |
|
|
125
|
-
| `reindex-files [--callables]` | Regenerate descriptions for files changed since their stored file description. By default stops after each file description; `--callables` also regenerates its callable descriptions. | JSON description statistics |
|
|
126
|
-
| `delete-files <path...>` | After automatic refresh, remove paths from the index; source files are not deleted. A later refresh can restore eligible files. | JSON update statistics |
|
|
127
|
-
|
|
128
|
-
Manual maintenance examples:
|
|
116
|
+
When code is similar but not actually duplicated, then `--cohesion` can help find similar code that exists far apart in the filesystem tree, which could potentially be refactored to make it more cohesive, by ordering matches from farthest to nearest.
|
|
129
117
|
|
|
130
118
|
```bash
|
|
131
|
-
slopdex
|
|
132
|
-
slopdex update-files src/service.ts src/model.ts
|
|
133
|
-
slopdex reindex-files
|
|
134
|
-
slopdex reindex-files --callables
|
|
135
|
-
slopdex delete-files src/removed.ts
|
|
136
|
-
slopdex update-git --target HEAD --rebuild-on-divergence
|
|
137
|
-
slopdex update-git --force-reindex
|
|
119
|
+
slopdex cross-search --cross-file-only --cohesion --threshold 0.8
|
|
138
120
|
```
|
|
121
|
+
### Use with agents
|
|
139
122
|
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
| Argument | Meaning / default |
|
|
143
|
-
| --- | --- |
|
|
144
|
-
| `--root <path>` | Repository root; current directory by default. |
|
|
145
|
-
| `--config <path>` | Config file; `<root>/.slopdex/config.json` by default. |
|
|
146
|
-
| `--index <path>` | Index file; `<root>/.slopdex/index.sqlite` by default. Overrides `indexPath` in config. |
|
|
147
|
-
| `--provider <openai\|jina>` | Embedding provider; `openai` by default. |
|
|
148
|
-
| `--model <name>` | Embedding model; `text-embedding-3-large` for OpenAI, `jina-embeddings-v4` for Jina. |
|
|
149
|
-
| `--dimensions <number>` | Positive embedding dimension count; OpenAI `3072`, Jina `1024`. Must be supported by the model. |
|
|
150
|
-
| `--description-provider <openai\|opencode\|opencode-go>` | Description provider; OpenAI by default. OpenCode values use Zen or Go with `OPENCODE_API_KEY`. |
|
|
151
|
-
| `--description-model <name>` | Description model; `gpt-5.6-sol` for OpenAI/Zen and `gpt-5.6-luna` for Go, then the persisted model unless overridden. Published OpenCode models use their documented protocol. |
|
|
152
|
-
| `--ignore-errors` | Silence warnings about saved indexing errors; records remain available. |
|
|
153
|
-
| `--verbose` | Write one stderr notice for every external model call instead of one per call kind/provider/model. |
|
|
154
|
-
| `-h`, `--help` | Show CLI usage without refreshing. |
|
|
155
|
-
| `--version` | Print the package version and exit. |
|
|
156
|
-
|
|
157
|
-
Explicit relative config and index paths resolve from the current directory, not `--root`.
|
|
158
|
-
|
|
159
|
-
### Search and analysis
|
|
160
|
-
|
|
161
|
-
| Argument | Applies to | Meaning / default |
|
|
162
|
-
| --- | --- | --- |
|
|
163
|
-
| `--limit <number>` | Both query searches, cross-search | Positive integer. Query matches: `10`; cross-search neighbors per source: `5`. |
|
|
164
|
-
| `--threshold <number\|min-max>` | Both query searches, cross-search | Minimum similarity, or range with inclusive minimum and exclusive maximum. Default `-1`. |
|
|
165
|
-
| `--format <json\|summary\|clusters>` | Both query searches, cross-search, index-errors | Output format; see the commands table. `clusters` is only for ordinary cross-search. |
|
|
166
|
-
| `-e <regex>`, `--regexp <regex>`, `--regex <regex>` | Both query searches, cross-search | Equivalent case-sensitive JavaScript regex options over qualified names. Query searches filter results before limiting; cross-search filters sources only. |
|
|
167
|
-
| `--min-lines <number>` | Cross-search | Minimum source and candidate callable length; positive integer, default `2`. Use `1` to include one-line wrappers. |
|
|
168
|
-
| `--source-path <path>` | Cross-search | Select sources in a file or recursive directory, relative to the repository root (or absolute within it). |
|
|
169
|
-
| `--changed-since <commit>` | Cross-search | Select added, modified, or moved functions relative to an ancestor of the indexed Git checkpoint, including working-tree changes. Requires Git. |
|
|
170
|
-
| `--uncommitted` | Cross-search | Select functions indexed from working-tree files: staged, unstaged, or untracked changes in Git; all working-tree functions without Git. |
|
|
171
|
-
| `--cross-file-only` | Cross-search | Exclude matches from the same physical file. |
|
|
172
|
-
| `--include-symmetric-duplicates` | Cross-search | Allow both directions of same-index matches; otherwise each unordered pair is emitted once. |
|
|
173
|
-
| `--cohesion` | Cross-search | Add `physicalDistance` and re-rank each source's matches by descending distance, with similarity as the tie-breaker. Defaults to summary output; incompatible with clusters. |
|
|
174
|
-
| `--target-root <path>` | Cross-search | Second repository root; requires `--target-index`. |
|
|
175
|
-
| `--target-index <path>` | Cross-search | Second index file; requires `--target-root`. |
|
|
176
|
-
| `--target-config <path>` | Cross-search | Target config; defaults to `<target-root>/.slopdex/config.json`. Requires both target options. |
|
|
177
|
-
|
|
178
|
-
Review adjacent similarity bands without repeating boundary matches:
|
|
123
|
+
Just put this in your `AGENTS.md` file, no skill required:
|
|
179
124
|
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
slopdex cross-search --
|
|
125
|
+
```
|
|
126
|
+
- use semantic code search to find code with: `slopdex search "..." --threshold 0.5`
|
|
127
|
+
- when reviewing uncommitted code avoid introducing duplicates by looking for related matches: `slopdex cross-search --uncommitted --threshold 0.8`
|
|
183
128
|
```
|
|
184
129
|
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
| Argument | Meaning |
|
|
188
|
-
| --- | --- |
|
|
189
|
-
| `--target <ref>` | Git snapshot for `update-git`; default `HEAD`. Non-HEAD targets exclude working-tree changes. Later commands normally refresh back to HEAD. |
|
|
190
|
-
| `--rebuild-on-divergence` | Allow reconciliation when the saved checkpoint is not an ancestor of the target, such as after a rebase or branch switch. |
|
|
191
|
-
| `--force-reindex` | Recreate an **incompatible** index (repository, provider, model, dimensions, strategy, or schema mismatch). A compatible index still follows normal refresh behavior. |
|
|
192
|
-
| `--no-reindex` | With Git, still reconcile the committed snapshot but skip working-tree overlays. Without Git, reuse a non-empty index; missing/empty indexes are still populated. Not a general offline switch. |
|
|
193
|
-
|
|
194
|
-
## Reading results
|
|
195
|
-
|
|
196
|
-
### Similarity and duplicate clusters
|
|
130
|
+
Any other use, like doing a full `cross-search` is probably better done interactively with the agent, in which case you can just ask the agent to run `slopdex --help` to get usage information.
|
|
197
131
|
|
|
198
|
-
|
|
132
|
+
### Reranking
|
|
199
133
|
|
|
200
|
-
|
|
134
|
+
Optionally a second-stage reranker can be enabled for `search` and `search-description`:
|
|
201
135
|
|
|
202
|
-
```
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
136
|
+
```bash
|
|
137
|
+
export COHERE_API_KEY="your-api-key"
|
|
138
|
+
slopdex config reranker cohere
|
|
139
|
+
# Or: export JINA_API_KEY="your-api-key" && slopdex config reranker jina
|
|
140
|
+
# Or use an LLM: export OPENAI_API_KEY="your-api-key" && slopdex config reranker openai
|
|
207
141
|
```
|
|
208
142
|
|
|
209
|
-
|
|
210
|
-
- Clusters sort by member count, then name. Cluster number is not severity.
|
|
211
|
-
- Locations identify where to inspect behavior, callers, and architectural roles. Wrappers, adapters, tests, and separate interface implementations can legitimately resemble one another.
|
|
212
|
-
|
|
213
|
-
### Physical cohesion
|
|
143
|
+
### Compare repositories
|
|
214
144
|
|
|
215
|
-
```
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
145
|
+
```bash
|
|
146
|
+
slopdex cross-search \
|
|
147
|
+
--target-root /path/to/other/repo \
|
|
148
|
+
--target-index /path/to/other/repo/.slopdex/index.sqlite \
|
|
149
|
+
--threshold 0.9
|
|
219
150
|
```
|
|
220
151
|
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
For automation, pass `--format json`: query searches and diagnostics return JSON arrays; cross-search returns **JSONL**, one row per matched source. With `--cohesion`, each match includes `physicalDistance`. Results go to stdout; notices and warnings go to stderr.
|
|
224
|
-
|
|
225
|
-
## System behavior
|
|
226
|
-
|
|
227
|
-
### Descriptions and scoring
|
|
228
|
-
|
|
229
|
-
Descriptions are optional and disabled initially. `descriptions enable` persists the selected provider and model and keeps callable descriptions current on later updates. Use `slopdex descriptions enable --description-provider opencode-go` for OpenCode Go, or combine `--description-provider` and `--description-model` to change both. `slopdex descriptions disable` stops automatic updates and description-based searching/scoring while retaining cached descriptions. `status` exposes callable/file description counts, stale file-description count, enabled state, and profile.
|
|
230
|
-
|
|
231
|
-
Descriptions are generated in source order through one conversation per file. Instructions and complete file source form a stable prefix; Slopdex asks for the overall file description first, then each callable request and answer extends that conversation. This allows supported providers to reuse their prompt cache instead of receiving a separate duplicated file context for every callable.
|
|
232
|
-
|
|
233
|
-
Ordinary updates preserve an existing file description even when its source changes, while still refreshing callable descriptions. `slopdex reindex-files` explicitly regenerates stale file descriptions and their embeddings; add `--callables` to continue through and replace every callable description in those files.
|
|
234
|
-
|
|
235
|
-
Tree-sitter extraction, generated descriptions, and document/query vectors are content-addressed in the same SQLite database. Each validated result is committed immediately, independently of the final logical index update. If indexing is interrupted or a later provider call fails, rerunning reuses every completed result whose profile, operation, input, and source context hash still match.
|
|
236
|
-
|
|
237
|
-
When Slopdex makes external vector, description, or reranking model calls, stderr identifies the call kind, provider, and model. By default each combination is reported once per process regardless of request count. Pass `--verbose`, or set `"verbose": true` in config, to report every request. Cache hits do not produce notices because they do not call a model.
|
|
238
|
-
|
|
239
|
-
With complete descriptions, `search` and cross-search average **one-third code similarity + one-third callable-description similarity + one-third file-description similarity**. `search-description` averages callable and file descriptions without code. Cross-repository analysis needs complete descriptions on both sides; otherwise the entire analysis uses code-only scores. Stale file descriptions remain searchable until explicitly reindexed. Thresholds and limits apply to the selected score.
|
|
240
|
-
|
|
241
|
-
Text output labels combined scores. JSON exposes `codeSimilarity`, `descriptionSimilarity`, `fileDescriptionSimilarity`, and cross-search scoring mode/weights. Compare runs only with matching scoring mode, weights, embedding and description-generator profiles, threshold, and source/candidate filters.
|
|
242
|
-
|
|
243
|
-
### Exclusions
|
|
244
|
-
|
|
245
|
-
Root and nested `.gitignore` rules apply even to tracked files and without Git. Working-tree refreshes use current rules; committed-only snapshots use the target commit's rules. Refresh removes newly excluded files and discovers newly eligible ones. Explicit `update-files` rejects ignored files.
|
|
246
|
-
|
|
247
|
-
Built-in exclusions: `.git`, `.slopdex`, `node_modules`, `dist`, `build`, `coverage`, `vendor`, `generated`, `.venv`, `venv`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, and `target`. Config `include`/`exclude` globs narrow coverage; they cannot override built-in exclusions. Ignore exceptions cannot re-include files beneath an excluded parent directory. Files over 1 MiB are skipped unless `maxFileSize` is raised.
|
|
248
|
-
|
|
249
|
-
### Failures and recovery
|
|
250
|
-
|
|
251
|
-
Parse, extraction, read, and file-size failures are saved while healthy callables remain searchable. Inspect them with `slopdex index-errors --format summary`. JSON includes paths, locations, recoverable names, messages, available source, and snapshot provenance. `status` reports `indexingErrorCount` and `failedFileCount`; `functionCount` counts searchable callables.
|
|
252
|
-
|
|
253
|
-
Saved failures trigger stderr warnings, including on cached runs, help, and cross-search targets. `--ignore-errors` silences warnings without clearing records. Updates retry failed files; successful indexing, deletion, or exclusion clears their diagnostics. Version output bypasses diagnostics.
|
|
254
|
-
|
|
255
|
-
Use the recovery flag named in the error: `--rebuild-on-divergence` for Git history changes, `--force-reindex` for incompatible indexes. Schema versions 6 and 7 migrate in place; earlier schemas require `--force-reindex`, with compatible rebuilds preserving reusable artifact caches. For provider/authentication failures, fix the reported configuration and rerun. For source-change-during-indexing errors, rerun after edits settle. Exit status is `0` on success, `2` for argument/domain errors, and `1` for other failures (or invocation without a command).
|
|
256
|
-
|
|
257
|
-
## Configuration
|
|
152
|
+
### Inspect index health
|
|
258
153
|
|
|
259
|
-
|
|
154
|
+
If some functions can't be indexed a warning is printed. Index errors can be investigated and fixed with the `index-errors` command to ensure the index is complete.
|
|
260
155
|
|
|
261
|
-
```
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
"dimensions": 1024,
|
|
266
|
-
"rerankingEnabled": true,
|
|
267
|
-
"rerankerProvider": "openai",
|
|
268
|
-
"rerankerModel": "gpt-5.6-luna",
|
|
269
|
-
"rerankerCandidates": 10,
|
|
270
|
-
"descriptionProvider": "opencode-go",
|
|
271
|
-
"verbose": true,
|
|
272
|
-
"exclude": ["**/fixtures/**"]
|
|
273
|
-
}
|
|
156
|
+
```bash
|
|
157
|
+
slopdex status
|
|
158
|
+
slopdex index-errors --format summary
|
|
159
|
+
slopdex --version
|
|
274
160
|
```
|
|
275
161
|
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
| `descriptionProvider` | Description provider: `openai`, `opencode` (Zen), or `opencode-go`; defaults to `openai`. |
|
|
280
|
-
| `descriptionModel` | Description model; provider default unless explicitly set. |
|
|
281
|
-
| `descriptionsEnabled` | When true or false, the next index-using command applies that enabled state during its normal refresh. Unset leaves persisted index state unchanged. |
|
|
282
|
-
| `rerankingEnabled` | Enables second-stage ranking for `search` and `search-description`; disabled/unset by default. Prefer changing it through `config reranker`. |
|
|
283
|
-
| `rerankerProvider` | Reranker: `cohere`, `jina`, or `openai`. OpenAI uses an LLM rather than a dedicated reranking endpoint. |
|
|
284
|
-
| `rerankerModel` | Provider model; defaults to Cohere `rerank-v4.0-pro`, Jina `jina-reranker-v3.5`, or OpenAI `gpt-5.6-luna`. |
|
|
285
|
-
| `rerankerCandidates` | Embedding-ranked candidates sent to the OpenAI LLM; integer from `1` to `100`, default `10`. The requested result limit takes precedence when larger, up to 100. |
|
|
286
|
-
| `indexPath` | Index location; `<root>/.slopdex/index.sqlite`. |
|
|
287
|
-
| `include` | Repository-relative glob array; empty/unset includes all supported eligible files. |
|
|
288
|
-
| `exclude` | Additional repository-relative exclusion globs. |
|
|
289
|
-
| `maxFileSize` | Maximum source-file size in bytes; positive integer, default `1048576`. |
|
|
290
|
-
| `embeddingBatchSize` | Embedding inputs per batch; positive integer, default `32`. |
|
|
291
|
-
| `verbose` | When true, report every external model request on stderr; false/unset reports each call kind/provider/model once per process. |
|
|
292
|
-
|
|
293
|
-
Keep keys in the environment (`OPENAI_API_KEY`, `JINA_API_KEY`, `COHERE_API_KEY`, `OPENCODE_API_KEY`). Reranker settings do not change the stored index and do not require a rebuild. Changing the embedding profile requires rebuilding with `--force-reindex`.
|
|
294
|
-
|
|
295
|
-
## Development
|
|
296
|
-
|
|
297
|
-
- [Implementation and library API](docs/implementation.md)
|
|
162
|
+
## Commands
|
|
163
|
+
|
|
164
|
+
See [Command reference](docs/reference.md).
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
// src/search/similarity.ts
|
|
2
|
+
var SIMILARITY_CACHE_FLOOR_ANCHOR = 0.3;
|
|
3
|
+
function similarityCacheFloor(minSimilarity) {
|
|
4
|
+
return Math.min(minSimilarity ?? -1, SIMILARITY_CACHE_FLOOR_ANCHOR);
|
|
5
|
+
}
|
|
6
|
+
function analysisSimilarity(source, target = source) {
|
|
7
|
+
const complete = (status) => status.descriptionsEnabled && status.descriptionProfile !== null && status.descriptionCount === status.functionCount && status.fileDescriptionCount === status.describableFileCount;
|
|
8
|
+
return complete(source) && complete(target) ? { similarityMode: "code-description-file-average", similarityWeights: { code: 1 / 3, description: 1 / 3, fileDescription: 1 / 3 } } : { similarityMode: "code", similarityWeights: { code: 1, description: 0, fileDescription: 0 } };
|
|
9
|
+
}
|
|
10
|
+
|
|
11
|
+
export {
|
|
12
|
+
similarityCacheFloor,
|
|
13
|
+
analysisSimilarity
|
|
14
|
+
};
|
|
15
|
+
//# sourceMappingURL=chunk-5BO2LLXC.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":["../src/search/similarity.ts"],"sourcesContent":["import type { AnalysisSimilarity, IndexStatus } from \"../types.js\";\n\n/**\n * Floor the persisted similarity cache is routinely built for. Analysis entry\n * points anchor their refresh floor here unless the caller asks for less, so\n * sweeping thresholds at or above the default shares one cache band: higher\n * floors read it for free and only a lower floor triggers an expansion\n * recompute. Matches the CLI default threshold.\n */\nexport const SIMILARITY_CACHE_FLOOR_ANCHOR = 0.3;\n\n/** Anchor a requested refresh floor so threshold sweeps share one cache band. */\nexport function similarityCacheFloor(minSimilarity?: number): number {\n return Math.min(minSimilarity ?? -1, SIMILARITY_CACHE_FLOOR_ANCHOR);\n}\n\n/** Select one scoring mode for the entire analysis, never a per-pair fallback. */\nexport function analysisSimilarity(source: IndexStatus, target: IndexStatus = source): AnalysisSimilarity {\n const complete = (status: IndexStatus): boolean => status.descriptionsEnabled\n && status.descriptionProfile !== null\n && status.descriptionCount === status.functionCount\n && status.fileDescriptionCount === status.describableFileCount;\n return complete(source) && complete(target)\n ? { similarityMode: \"code-description-file-average\", similarityWeights: { code: 1 / 3, description: 1 / 3, fileDescription: 1 / 3 } }\n : { similarityMode: \"code\", similarityWeights: { code: 1, description: 0, fileDescription: 0 } };\n}\n"],"mappings":";AASO,IAAM,gCAAgC;AAGtC,SAAS,qBAAqB,eAAgC;AACnE,SAAO,KAAK,IAAI,iBAAiB,IAAI,6BAA6B;AACpE;AAGO,SAAS,mBAAmB,QAAqB,SAAsB,QAA4B;AACxG,QAAM,WAAW,CAAC,WAAiC,OAAO,uBACrD,OAAO,uBAAuB,QAC9B,OAAO,qBAAqB,OAAO,iBACnC,OAAO,yBAAyB,OAAO;AAC5C,SAAO,SAAS,MAAM,KAAK,SAAS,MAAM,IACtC,EAAE,gBAAgB,iCAAiC,mBAAmB,EAAE,MAAM,IAAI,GAAG,aAAa,IAAI,GAAG,iBAAiB,IAAI,EAAE,EAAE,IAClI,EAAE,gBAAgB,QAAQ,mBAAmB,EAAE,MAAM,GAAG,aAAa,GAAG,iBAAiB,EAAE,EAAE;AACnG;","names":[]}
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
import {
|
|
2
|
+
writeStderr
|
|
3
|
+
} from "./chunk-S225GYCL.js";
|
|
4
|
+
|
|
5
|
+
// src/model-call-notice.ts
|
|
6
|
+
var reportedCalls = /* @__PURE__ */ new Set();
|
|
7
|
+
function reportModelCall(kind, profile, verbose = false, parallelism = 1) {
|
|
8
|
+
const key = JSON.stringify([kind, profile.provider, profile.model, parallelism]);
|
|
9
|
+
if (!verbose && reportedCalls.has(key)) return;
|
|
10
|
+
reportedCalls.add(key);
|
|
11
|
+
writeStderr(
|
|
12
|
+
`slopdex: notice: external model call: kind=${kind} provider=${JSON.stringify(profile.provider)} model=${JSON.stringify(profile.model)} parallelism=${parallelism}
|
|
13
|
+
`
|
|
14
|
+
);
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
export {
|
|
18
|
+
reportModelCall
|
|
19
|
+
};
|
|
20
|
+
//# sourceMappingURL=chunk-BXWO2KMC.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":["../src/model-call-notice.ts"],"sourcesContent":["import { writeStderr } from \"./progress.js\";\n\nconst reportedCalls = new Set<string>();\n\nexport type ModelCallKind = \"vectors\" | \"descriptions\" | \"reranking\";\n\nexport function reportModelCall(\n kind: ModelCallKind,\n profile: { provider: string; model: string },\n verbose = false,\n parallelism = 1,\n): void {\n const key = JSON.stringify([kind, profile.provider, profile.model, parallelism]);\n if (!verbose && reportedCalls.has(key)) return;\n reportedCalls.add(key);\n writeStderr(\n `slopdex: notice: external model call: kind=${kind} provider=${JSON.stringify(profile.provider)} model=${JSON.stringify(profile.model)} parallelism=${parallelism}\\n`,\n );\n}\n"],"mappings":";;;;;AAEA,IAAM,gBAAgB,oBAAI,IAAY;AAI/B,SAAS,gBACd,MACA,SACA,UAAU,OACV,cAAc,GACR;AACN,QAAM,MAAM,KAAK,UAAU,CAAC,MAAM,QAAQ,UAAU,QAAQ,OAAO,WAAW,CAAC;AAC/E,MAAI,CAAC,WAAW,cAAc,IAAI,GAAG,EAAG;AACxC,gBAAc,IAAI,GAAG;AACrB;AAAA,IACE,8CAA8C,IAAI,aAAa,KAAK,UAAU,QAAQ,QAAQ,CAAC,UAAU,KAAK,UAAU,QAAQ,KAAK,CAAC,gBAAgB,WAAW;AAAA;AAAA,EACnK;AACF;","names":[]}
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
// src/errors.ts
|
|
2
|
+
var CodeIndexError = class extends Error {
|
|
3
|
+
constructor(message, options) {
|
|
4
|
+
super(message, options);
|
|
5
|
+
this.name = "CodeIndexError";
|
|
6
|
+
}
|
|
7
|
+
};
|
|
8
|
+
var GitDivergenceError = class extends CodeIndexError {
|
|
9
|
+
constructor(checkpoint, target) {
|
|
10
|
+
super(`Stored Git checkpoint ${checkpoint} is not an ancestor of ${target}; rebuild the index.`);
|
|
11
|
+
this.name = "GitDivergenceError";
|
|
12
|
+
}
|
|
13
|
+
};
|
|
14
|
+
var GitUnavailableError = class extends CodeIndexError {
|
|
15
|
+
constructor(message, options) {
|
|
16
|
+
super(message, options);
|
|
17
|
+
this.name = "GitUnavailableError";
|
|
18
|
+
}
|
|
19
|
+
};
|
|
20
|
+
var IncompatibleIndexError = class extends CodeIndexError {
|
|
21
|
+
constructor(message) {
|
|
22
|
+
super(message);
|
|
23
|
+
this.name = "IncompatibleIndexError";
|
|
24
|
+
}
|
|
25
|
+
};
|
|
26
|
+
|
|
27
|
+
export {
|
|
28
|
+
CodeIndexError,
|
|
29
|
+
GitDivergenceError,
|
|
30
|
+
GitUnavailableError,
|
|
31
|
+
IncompatibleIndexError
|
|
32
|
+
};
|
|
33
|
+
//# sourceMappingURL=chunk-DKRB2XT5.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":["../src/errors.ts"],"sourcesContent":["export class CodeIndexError extends Error {\n public constructor(message: string, options?: ErrorOptions) {\n super(message, options);\n this.name = \"CodeIndexError\";\n }\n}\n\nexport class GitDivergenceError extends CodeIndexError {\n public constructor(checkpoint: string, target: string) {\n super(`Stored Git checkpoint ${checkpoint} is not an ancestor of ${target}; rebuild the index.`);\n this.name = \"GitDivergenceError\";\n }\n}\n\nexport class GitUnavailableError extends CodeIndexError {\n public constructor(message: string, options?: ErrorOptions) {\n super(message, options);\n this.name = \"GitUnavailableError\";\n }\n}\n\nexport class IncompatibleIndexError extends CodeIndexError {\n public constructor(message: string) {\n super(message);\n this.name = \"IncompatibleIndexError\";\n }\n}\n"],"mappings":";AAAO,IAAM,iBAAN,cAA6B,MAAM;AAAA,EACjC,YAAY,SAAiB,SAAwB;AAC1D,UAAM,SAAS,OAAO;AACtB,SAAK,OAAO;AAAA,EACd;AACF;AAEO,IAAM,qBAAN,cAAiC,eAAe;AAAA,EAC9C,YAAY,YAAoB,QAAgB;AACrD,UAAM,yBAAyB,UAAU,0BAA0B,MAAM,sBAAsB;AAC/F,SAAK,OAAO;AAAA,EACd;AACF;AAEO,IAAM,sBAAN,cAAkC,eAAe;AAAA,EAC/C,YAAY,SAAiB,SAAwB;AAC1D,UAAM,SAAS,OAAO;AACtB,SAAK,OAAO;AAAA,EACd;AACF;AAEO,IAAM,yBAAN,cAAqC,eAAe;AAAA,EAClD,YAAY,SAAiB;AAClC,UAAM,OAAO;AACb,SAAK,OAAO;AAAA,EACd;AACF;","names":[]}
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
import {
|
|
2
|
+
reportModelCall
|
|
3
|
+
} from "./chunk-BXWO2KMC.js";
|
|
4
|
+
import {
|
|
5
|
+
assertPositiveInteger,
|
|
6
|
+
throwIfAborted
|
|
7
|
+
} from "./chunk-VH5VGRCI.js";
|
|
8
|
+
import {
|
|
9
|
+
CodeIndexError
|
|
10
|
+
} from "./chunk-DKRB2XT5.js";
|
|
11
|
+
|
|
12
|
+
// src/rerankers/hosted.ts
|
|
13
|
+
var HostedReranker = class {
|
|
14
|
+
profile;
|
|
15
|
+
#apiKey;
|
|
16
|
+
#url;
|
|
17
|
+
#returnDocuments;
|
|
18
|
+
#verbose;
|
|
19
|
+
constructor(options, settings) {
|
|
20
|
+
this.#apiKey = options.apiKey ?? process.env[settings.apiKeyName] ?? "";
|
|
21
|
+
if (!this.#apiKey) throw new Error(`${settings.apiKeyName} is required.`);
|
|
22
|
+
const model = options.model ?? settings.defaultModel;
|
|
23
|
+
if (!model.trim()) throw new Error("reranker model must not be empty.");
|
|
24
|
+
this.profile = { provider: settings.provider, model };
|
|
25
|
+
this.#verbose = options.verbose ?? false;
|
|
26
|
+
const url = (options.baseUrl ?? settings.defaultUrl).replace(/\/$/, "");
|
|
27
|
+
this.#url = url.endsWith("/rerank") ? url : `${url}/rerank`;
|
|
28
|
+
this.#returnDocuments = settings.returnDocuments;
|
|
29
|
+
}
|
|
30
|
+
async rerank(query, documents, options = {}) {
|
|
31
|
+
if (documents.length === 0) return [];
|
|
32
|
+
const limit = options.limit ?? documents.length;
|
|
33
|
+
assertPositiveInteger(limit, "rerank limit");
|
|
34
|
+
throwIfAborted(options.signal);
|
|
35
|
+
reportModelCall("reranking", this.profile, this.#verbose);
|
|
36
|
+
let response;
|
|
37
|
+
try {
|
|
38
|
+
response = await fetch(this.#url, {
|
|
39
|
+
method: "POST",
|
|
40
|
+
headers: {
|
|
41
|
+
accept: "application/json",
|
|
42
|
+
authorization: `Bearer ${this.#apiKey}`,
|
|
43
|
+
"content-type": "application/json"
|
|
44
|
+
},
|
|
45
|
+
body: JSON.stringify({
|
|
46
|
+
model: this.profile.model,
|
|
47
|
+
query,
|
|
48
|
+
documents,
|
|
49
|
+
top_n: Math.min(limit, documents.length),
|
|
50
|
+
...this.#returnDocuments !== void 0 ? { return_documents: this.#returnDocuments } : {}
|
|
51
|
+
}),
|
|
52
|
+
...options.signal ? { signal: options.signal } : {}
|
|
53
|
+
});
|
|
54
|
+
} catch (error) {
|
|
55
|
+
if (options.signal?.aborted) throw error;
|
|
56
|
+
throw new CodeIndexError(`Reranking request failed: ${error instanceof Error ? error.message : String(error)}`, { cause: error });
|
|
57
|
+
}
|
|
58
|
+
if (!response.ok) {
|
|
59
|
+
throw new CodeIndexError(`Reranking request failed (${response.status}): ${(await response.text()).slice(0, 500)}`);
|
|
60
|
+
}
|
|
61
|
+
let body;
|
|
62
|
+
try {
|
|
63
|
+
body = await response.json();
|
|
64
|
+
} catch (error) {
|
|
65
|
+
throw new CodeIndexError(`${this.profile.provider} returned a malformed reranking response.`, { cause: error });
|
|
66
|
+
}
|
|
67
|
+
const results = body && typeof body === "object" && "results" in body ? body.results : void 0;
|
|
68
|
+
const expected = Math.min(limit, documents.length);
|
|
69
|
+
if (!Array.isArray(results) || results.length !== expected) {
|
|
70
|
+
throw new CodeIndexError(`${this.profile.provider} returned a malformed reranking response.`);
|
|
71
|
+
}
|
|
72
|
+
const seen = /* @__PURE__ */ new Set();
|
|
73
|
+
return results.map((result) => {
|
|
74
|
+
const value = result;
|
|
75
|
+
if (!result || typeof result !== "object" || !Number.isInteger(value.index) || value.index < 0 || value.index >= documents.length || typeof value.relevance_score !== "number" || !Number.isFinite(value.relevance_score) || seen.has(value.index)) {
|
|
76
|
+
throw new CodeIndexError(`${this.profile.provider} returned a malformed reranking response.`);
|
|
77
|
+
}
|
|
78
|
+
seen.add(value.index);
|
|
79
|
+
return { index: value.index, score: value.relevance_score };
|
|
80
|
+
});
|
|
81
|
+
}
|
|
82
|
+
};
|
|
83
|
+
var CohereReranker = class extends HostedReranker {
|
|
84
|
+
constructor(options = {}) {
|
|
85
|
+
super(options, {
|
|
86
|
+
provider: "cohere",
|
|
87
|
+
apiKeyName: "COHERE_API_KEY",
|
|
88
|
+
defaultModel: "rerank-v4.0-pro",
|
|
89
|
+
defaultUrl: "https://api.cohere.com/v2"
|
|
90
|
+
});
|
|
91
|
+
}
|
|
92
|
+
};
|
|
93
|
+
var JinaReranker = class extends HostedReranker {
|
|
94
|
+
constructor(options = {}) {
|
|
95
|
+
super(options, {
|
|
96
|
+
provider: "jina",
|
|
97
|
+
apiKeyName: "JINA_API_KEY",
|
|
98
|
+
defaultModel: "jina-reranker-v3.5",
|
|
99
|
+
defaultUrl: "https://api.jina.ai/v1",
|
|
100
|
+
returnDocuments: false
|
|
101
|
+
});
|
|
102
|
+
}
|
|
103
|
+
};
|
|
104
|
+
|
|
105
|
+
export {
|
|
106
|
+
CohereReranker,
|
|
107
|
+
JinaReranker
|
|
108
|
+
};
|
|
109
|
+
//# sourceMappingURL=chunk-DTI7SXGL.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":["../src/rerankers/hosted.ts"],"sourcesContent":["import { CodeIndexError } from \"../errors.js\";\nimport { reportModelCall } from \"../model-call-notice.js\";\nimport type { Reranker } from \"../types.js\";\nimport { assertPositiveInteger, throwIfAborted } from \"../utils.js\";\n\ninterface HostedRerankerOptions {\n apiKey?: string;\n model?: string;\n baseUrl?: string;\n verbose?: boolean;\n}\n\ninterface HostedRerankerSettings {\n provider: \"cohere\" | \"jina\";\n apiKeyName: \"COHERE_API_KEY\" | \"JINA_API_KEY\";\n defaultModel: string;\n defaultUrl: string;\n returnDocuments?: boolean;\n}\n\nabstract class HostedReranker implements Reranker {\n public readonly profile;\n readonly #apiKey: string;\n readonly #url: string;\n readonly #returnDocuments: boolean | undefined;\n readonly #verbose: boolean;\n\n protected constructor(options: HostedRerankerOptions, settings: HostedRerankerSettings) {\n this.#apiKey = options.apiKey ?? process.env[settings.apiKeyName] ?? \"\";\n if (!this.#apiKey) throw new Error(`${settings.apiKeyName} is required.`);\n const model = options.model ?? settings.defaultModel;\n if (!model.trim()) throw new Error(\"reranker model must not be empty.\");\n this.profile = { provider: settings.provider, model } as const;\n this.#verbose = options.verbose ?? false;\n const url = (options.baseUrl ?? settings.defaultUrl).replace(/\\/$/, \"\");\n this.#url = url.endsWith(\"/rerank\") ? url : `${url}/rerank`;\n this.#returnDocuments = settings.returnDocuments;\n }\n\n public async rerank(\n query: string,\n documents: readonly string[],\n options: { limit?: number; signal?: AbortSignal } = {},\n ): Promise<Array<{ index: number; score: number }>> {\n if (documents.length === 0) return [];\n const limit = options.limit ?? documents.length;\n assertPositiveInteger(limit, \"rerank limit\");\n throwIfAborted(options.signal);\n reportModelCall(\"reranking\", this.profile, this.#verbose);\n let response: Response;\n try {\n response = await fetch(this.#url, {\n method: \"POST\",\n headers: {\n accept: \"application/json\",\n authorization: `Bearer ${this.#apiKey}`,\n \"content-type\": \"application/json\",\n },\n body: JSON.stringify({\n model: this.profile.model,\n query,\n documents,\n top_n: Math.min(limit, documents.length),\n ...(this.#returnDocuments !== undefined ? { return_documents: this.#returnDocuments } : {}),\n }),\n ...(options.signal ? { signal: options.signal } : {}),\n });\n } catch (error) {\n if (options.signal?.aborted) throw error;\n throw new CodeIndexError(`Reranking request failed: ${error instanceof Error ? error.message : String(error)}`, { cause: error });\n }\n if (!response.ok) {\n throw new CodeIndexError(`Reranking request failed (${response.status}): ${(await response.text()).slice(0, 500)}`);\n }\n let body: unknown;\n try {\n body = await response.json();\n } catch (error) {\n throw new CodeIndexError(`${this.profile.provider} returned a malformed reranking response.`, { cause: error });\n }\n const results = body && typeof body === \"object\" && \"results\" in body\n ? (body as { results?: unknown }).results\n : undefined;\n const expected = Math.min(limit, documents.length);\n if (!Array.isArray(results) || results.length !== expected) {\n throw new CodeIndexError(`${this.profile.provider} returned a malformed reranking response.`);\n }\n const seen = new Set<number>();\n return results.map((result) => {\n const value = result as { index?: unknown; relevance_score?: unknown };\n if (!result || typeof result !== \"object\"\n || !Number.isInteger(value.index) || (value.index as number) < 0 || (value.index as number) >= documents.length\n || typeof value.relevance_score !== \"number\" || !Number.isFinite(value.relevance_score)\n || seen.has(value.index as number)) {\n throw new CodeIndexError(`${this.profile.provider} returned a malformed reranking response.`);\n }\n seen.add(value.index as number);\n return { index: value.index as number, score: value.relevance_score };\n });\n }\n}\n\nexport type CohereRerankerOptions = HostedRerankerOptions;\n\nexport class CohereReranker extends HostedReranker {\n public constructor(options: CohereRerankerOptions = {}) {\n super(options, {\n provider: \"cohere\",\n apiKeyName: \"COHERE_API_KEY\",\n defaultModel: \"rerank-v4.0-pro\",\n defaultUrl: \"https://api.cohere.com/v2\",\n });\n }\n}\n\nexport type JinaRerankerOptions = HostedRerankerOptions;\n\nexport class JinaReranker extends HostedReranker {\n public constructor(options: JinaRerankerOptions = {}) {\n super(options, {\n provider: \"jina\",\n apiKeyName: \"JINA_API_KEY\",\n defaultModel: \"jina-reranker-v3.5\",\n defaultUrl: \"https://api.jina.ai/v1\",\n returnDocuments: false,\n });\n }\n}\n"],"mappings":";;;;;;;;;;;;AAoBA,IAAe,iBAAf,MAAkD;AAAA,EAChC;AAAA,EACP;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EAEC,YAAY,SAAgC,UAAkC;AACtF,SAAK,UAAU,QAAQ,UAAU,QAAQ,IAAI,SAAS,UAAU,KAAK;AACrE,QAAI,CAAC,KAAK,QAAS,OAAM,IAAI,MAAM,GAAG,SAAS,UAAU,eAAe;AACxE,UAAM,QAAQ,QAAQ,SAAS,SAAS;AACxC,QAAI,CAAC,MAAM,KAAK,EAAG,OAAM,IAAI,MAAM,mCAAmC;AACtE,SAAK,UAAU,EAAE,UAAU,SAAS,UAAU,MAAM;AACpD,SAAK,WAAW,QAAQ,WAAW;AACnC,UAAM,OAAO,QAAQ,WAAW,SAAS,YAAY,QAAQ,OAAO,EAAE;AACtE,SAAK,OAAO,IAAI,SAAS,SAAS,IAAI,MAAM,GAAG,GAAG;AAClD,SAAK,mBAAmB,SAAS;AAAA,EACnC;AAAA,EAEA,MAAa,OACX,OACA,WACA,UAAoD,CAAC,GACH;AAClD,QAAI,UAAU,WAAW,EAAG,QAAO,CAAC;AACpC,UAAM,QAAQ,QAAQ,SAAS,UAAU;AACzC,0BAAsB,OAAO,cAAc;AAC3C,mBAAe,QAAQ,MAAM;AAC7B,oBAAgB,aAAa,KAAK,SAAS,KAAK,QAAQ;AACxD,QAAI;AACJ,QAAI;AACF,iBAAW,MAAM,MAAM,KAAK,MAAM;AAAA,QAChC,QAAQ;AAAA,QACR,SAAS;AAAA,UACP,QAAQ;AAAA,UACR,eAAe,UAAU,KAAK,OAAO;AAAA,UACrC,gBAAgB;AAAA,QAClB;AAAA,QACA,MAAM,KAAK,UAAU;AAAA,UACnB,OAAO,KAAK,QAAQ;AAAA,UACpB;AAAA,UACA;AAAA,UACA,OAAO,KAAK,IAAI,OAAO,UAAU,MAAM;AAAA,UACvC,GAAI,KAAK,qBAAqB,SAAY,EAAE,kBAAkB,KAAK,iBAAiB,IAAI,CAAC;AAAA,QAC3F,CAAC;AAAA,QACD,GAAI,QAAQ,SAAS,EAAE,QAAQ,QAAQ,OAAO,IAAI,CAAC;AAAA,MACrD,CAAC;AAAA,IACH,SAAS,OAAO;AACd,UAAI,QAAQ,QAAQ,QAAS,OAAM;AACnC,YAAM,IAAI,eAAe,6BAA6B,iBAAiB,QAAQ,MAAM,UAAU,OAAO,KAAK,CAAC,IAAI,EAAE,OAAO,MAAM,CAAC;AAAA,IAClI;AACA,QAAI,CAAC,SAAS,IAAI;AAChB,YAAM,IAAI,eAAe,6BAA6B,SAAS,MAAM,OAAO,MAAM,SAAS,KAAK,GAAG,MAAM,GAAG,GAAG,CAAC,EAAE;AAAA,IACpH;AACA,QAAI;AACJ,QAAI;AACF,aAAO,MAAM,SAAS,KAAK;AAAA,IAC7B,SAAS,OAAO;AACd,YAAM,IAAI,eAAe,GAAG,KAAK,QAAQ,QAAQ,6CAA6C,EAAE,OAAO,MAAM,CAAC;AAAA,IAChH;AACA,UAAM,UAAU,QAAQ,OAAO,SAAS,YAAY,aAAa,OAC5D,KAA+B,UAChC;AACJ,UAAM,WAAW,KAAK,IAAI,OAAO,UAAU,MAAM;AACjD,QAAI,CAAC,MAAM,QAAQ,OAAO,KAAK,QAAQ,WAAW,UAAU;AAC1D,YAAM,IAAI,eAAe,GAAG,KAAK,QAAQ,QAAQ,2CAA2C;AAAA,IAC9F;AACA,UAAM,OAAO,oBAAI,IAAY;AAC7B,WAAO,QAAQ,IAAI,CAAC,WAAW;AAC7B,YAAM,QAAQ;AACd,UAAI,CAAC,UAAU,OAAO,WAAW,YAC5B,CAAC,OAAO,UAAU,MAAM,KAAK,KAAM,MAAM,QAAmB,KAAM,MAAM,SAAoB,UAAU,UACtG,OAAO,MAAM,oBAAoB,YAAY,CAAC,OAAO,SAAS,MAAM,eAAe,KACnF,KAAK,IAAI,MAAM,KAAe,GAAG;AACpC,cAAM,IAAI,eAAe,GAAG,KAAK,QAAQ,QAAQ,2CAA2C;AAAA,MAC9F;AACA,WAAK,IAAI,MAAM,KAAe;AAC9B,aAAO,EAAE,OAAO,MAAM,OAAiB,OAAO,MAAM,gBAAgB;AAAA,IACtE,CAAC;AAAA,EACH;AACF;AAIO,IAAM,iBAAN,cAA6B,eAAe;AAAA,EAC1C,YAAY,UAAiC,CAAC,GAAG;AACtD,UAAM,SAAS;AAAA,MACb,UAAU;AAAA,MACV,YAAY;AAAA,MACZ,cAAc;AAAA,MACd,YAAY;AAAA,IACd,CAAC;AAAA,EACH;AACF;AAIO,IAAM,eAAN,cAA2B,eAAe;AAAA,EACxC,YAAY,UAA+B,CAAC,GAAG;AACpD,UAAM,SAAS;AAAA,MACb,UAAU;AAAA,MACV,YAAY;AAAA,MACZ,cAAc;AAAA,MACd,YAAY;AAAA,MACZ,iBAAiB;AAAA,IACnB,CAAC;AAAA,EACH;AACF;","names":[]}
|