vecscope 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- vecscope/__init__.py +6 -0
- vecscope/_knn.py +46 -0
- vecscope/cli.py +87 -0
- vecscope/diagnostics/__init__.py +20 -0
- vecscope/diagnostics/anisotropy.py +48 -0
- vecscope/diagnostics/ann_recall.py +25 -0
- vecscope/diagnostics/clusters.py +64 -0
- vecscope/diagnostics/collapse.py +32 -0
- vecscope/diagnostics/geometry.py +120 -0
- vecscope/diagnostics/hubness.py +33 -0
- vecscope/diagnostics/queries.py +65 -0
- vecscope/evidence.py +337 -0
- vecscope/io.py +58 -0
- vecscope/mcp_server.py +64 -0
- vecscope/sweep.py +207 -0
- vecscope-0.1.0.dist-info/METADATA +392 -0
- vecscope-0.1.0.dist-info/RECORD +20 -0
- vecscope-0.1.0.dist-info/WHEEL +4 -0
- vecscope-0.1.0.dist-info/entry_points.txt +3 -0
- vecscope-0.1.0.dist-info/licenses/LICENSE +21 -0
|
@@ -0,0 +1,392 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: vecscope
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Embedding space evidence for agents: hubness, anisotropy, collapse, ANN recall.
|
|
5
|
+
Project-URL: Homepage, https://github.com/076923/vecscope
|
|
6
|
+
Project-URL: Repository, https://github.com/076923/vecscope
|
|
7
|
+
Project-URL: Issues, https://github.com/076923/vecscope/issues
|
|
8
|
+
Author: Daehee Yun
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: agents,anisotropy,embeddings,hubness,mcp,rag,vector-search
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
21
|
+
Requires-Python: >=3.10
|
|
22
|
+
Requires-Dist: numpy>=1.24
|
|
23
|
+
Provides-Extra: dev
|
|
24
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
25
|
+
Requires-Dist: ruff>=0.5; extra == 'dev'
|
|
26
|
+
Provides-Extra: index
|
|
27
|
+
Requires-Dist: faiss-cpu>=1.8; extra == 'index'
|
|
28
|
+
Provides-Extra: mcp
|
|
29
|
+
Requires-Dist: mcp>=1.0; extra == 'mcp'
|
|
30
|
+
Description-Content-Type: text/markdown
|
|
31
|
+
|
|
32
|
+
<div align="center">
|
|
33
|
+
|
|
34
|
+
**English** · [한국어](docs/readme/README_ko.md)
|
|
35
|
+
|
|
36
|
+
# vecscope
|
|
37
|
+
|
|
38
|
+
### Embedding space evidence for agents.
|
|
39
|
+
|
|
40
|
+
**Before an agent trusts a vector index, let it see the space it is searching.**
|
|
41
|
+
|
|
42
|
+
[](https://pypi.org/project/vecscope/)
|
|
43
|
+
[](https://pypi.org/project/vecscope/)
|
|
44
|
+
[](https://github.com/076923/vecscope/actions/workflows/ci.yml)
|
|
45
|
+
[](LICENSE)
|
|
46
|
+
|
|
47
|
+
[Quick start](#quick-start) · [What it measures](#what-vecscope-measures) · [Queries and clusters](#queries-and-clusters) · [Index sweep](#index-sweep) · [Use with an agent](#use-with-an-agent) · [Input formats](#input-formats) · [Real embeddings](#real-embeddings) · [FAQ](#faq)
|
|
48
|
+
|
|
49
|
+
<code>pip install vecscope</code>
|
|
50
|
+
|
|
51
|
+
</div>
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
Agents build RAG pipelines, semantic search and recommendation retrieval every day, but they
|
|
56
|
+
cannot see the embedding space they are searching. When retrieval goes wrong it rarely looks
|
|
57
|
+
like a bug. It looks like "the same three chunks come back for everything", "every score is
|
|
58
|
+
0.8-something", "top-k is full of copies", or "brute force finds it and HNSW does not".
|
|
59
|
+
|
|
60
|
+
Each of those failures has a known, deterministic diagnostic in the literature. `vecscope`
|
|
61
|
+
turns them into one command and one JSON: measurements, `ok` / `warn` / `fail` verdicts, and
|
|
62
|
+
an explanation an agent can act on. It runs locally on NumPy, needs no model or network, and
|
|
63
|
+
reports the same numbers every time.
|
|
64
|
+
|
|
65
|
+
## Quick start
|
|
66
|
+
|
|
67
|
+
### 1. Install
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
pip install vecscope
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
### 2. Get a file of embeddings
|
|
74
|
+
|
|
75
|
+
`vecscope` reads a file with one embedding vector per row: the `(n, dim)` array a model's
|
|
76
|
+
`encode` returns, saved as `.npy`, `.parquet`, `.jsonl`, `.json` or `.csv`
|
|
77
|
+
([all formats](#input-formats)).
|
|
78
|
+
|
|
79
|
+
```python
|
|
80
|
+
import numpy as np
|
|
81
|
+
embeddings = model.encode(texts) # shape (n, dim), e.g. (3000, 128)
|
|
82
|
+
np.save("embeddings.npy", embeddings)
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
No embeddings at hand? Generate three synthetic matrices (healthy, anisotropic, collapsed):
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
python -c "import urllib.request as u; u.urlretrieve('https://raw.githubusercontent.com/076923/vecscope/main/examples/make_examples.py', 'make_examples.py')"
|
|
89
|
+
python make_examples.py
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
### 3. Run the audit
|
|
93
|
+
|
|
94
|
+
```bash
|
|
95
|
+
vecscope audit embeddings.npy
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
```text
|
|
99
|
+
vecscope 0.1.0 n=3000 dim=128 k=10 verdict=FAIL
|
|
100
|
+
[FAIL] hubness skewness=6.9023 (ok < 2, warn >= 2, fail >= 4)
|
|
101
|
+
k-occurrence skewness 6.90; top 1% of items take 16.3% of all neighbor slots (max N_k=356, expected 10).
|
|
102
|
+
[FAIL] antihubs antihub_ratio=0.2180 (ok < 0.05, warn >= 0.05, fail >= 0.2)
|
|
103
|
+
21.8% of items are never anyone's neighbor; they are unreachable through k-NN retrieval.
|
|
104
|
+
[FAIL] anisotropy mean_random_cosine=0.9782 (ok < 0.3, warn >= 0.3, fail >= 0.6)
|
|
105
|
+
random pairs have mean cosine 0.978 (isotropic ~0); mean unit vector has norm 0.99; centering would bring the cosine to -0.000.
|
|
106
|
+
[ok ] effective_rank effective_rank_ratio=0.9521 (ok >= 0.1, warn < 0.1, fail < 0.05)
|
|
107
|
+
effective rank 121.9 of 128 dims (95.2%); low values mean most dimensions carry nothing.
|
|
108
|
+
[ok ] collapse near_duplicate_ratio=0.0000 (ok < 0.02, warn >= 0.02, fail >= 0.1)
|
|
109
|
+
0.0% of items have a neighbor with cosine >= 0.99 (exact duplicates 0.0%).
|
|
110
|
+
[ok ] rogue_dimensions top1_dim_variance_share=0.0084 (ok < 0.05, warn >= 0.05, fail >= 0.2)
|
|
111
|
+
dimension 7 alone holds 0.8% of variance (uniform would be 0.78%); top 3 dims hold 2.5%. Rogue dimensions dominate cosine.
|
|
112
|
+
[ok ] norms norm_cv=0.0000 (ok < 0.1, warn >= 0.1, fail >= 0.3)
|
|
113
|
+
vector norms vary with CV 0.000 (median 1.000, range 1.000-1.000); 0.00% are zero vectors. Unequal norms bias inner-product search toward long vectors.
|
|
114
|
+
[ok ] metric_mismatch topk_overlap=1.0000 (ok >= 0.9, warn < 0.9, fail < 0.7)
|
|
115
|
+
cosine and inner-product top-10 share 100.0% of neighbors; 100.0% of items get identical lists. Low overlap means the index metric changes the results.
|
|
116
|
+
[ok ] distance_concentration relative_contrast=1.5260 (ok >= 1.2, warn < 1.2, fail < 1.05)
|
|
117
|
+
a random item is only 1.53x farther than the nearest neighbor (median NN distance 0.014, random 0.022). Near 1.0 means distances carry no signal.
|
|
118
|
+
[ok ] outliers outlier_ratio=0.0037 (ok < 0.01, warn >= 0.01, fail >= 0.05)
|
|
119
|
+
0.37% of items have no neighbor within cosine distance 0.018 (median 0.014); see details.sample_ids. Typically junk, empty or foreign-language content.
|
|
120
|
+
[ok ] connectivity island_ratio=0.0000 (ok < 0.01, warn >= 0.01, fail >= 0.1)
|
|
121
|
+
0.00% of items sit in 0 small island(s) of the k-NN graph (fewer than 15 items; 1 components in total). Graph-based ANN (HNSW) attaches islands to the wrong region or never reaches them.
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### 4. Read the result
|
|
125
|
+
|
|
126
|
+
- The first line gives the overall `verdict`: the worst verdict among the findings.
|
|
127
|
+
- Each finding shows the measured value, the range that would be `ok`, and a sentence saying
|
|
128
|
+
what the number means for retrieval.
|
|
129
|
+
- The exit code is `1` on `FAIL`, so the same command works as a CI gate.
|
|
130
|
+
- Add `--json` to get the same evidence as JSON for an agent or a script.
|
|
131
|
+
|
|
132
|
+
## What vecscope measures
|
|
133
|
+
|
|
134
|
+
| Finding | Metric | ok | warn | fail | What it means when it fails |
|
|
135
|
+
| --- | --- | --- | --- | --- | --- |
|
|
136
|
+
| `hubness` | skewness of the k-occurrence distribution | < 2.0 | 2.0 to 4.0 | ≥ 4.0 | a few items appear in everyone's k-NN list and will top every query |
|
|
137
|
+
| `antihubs` | share of items never retrieved as a neighbor | < 5% | 5 to 20% | ≥ 20% | part of the corpus is unreachable through k-NN |
|
|
138
|
+
| `anisotropy` | mean cosine of random pairs | < 0.30 | 0.30 to 0.60 | ≥ 0.60 | vectors crowd into a narrow cone, so every pair looks similar |
|
|
139
|
+
| `effective_rank` | participation ratio / dim | ≥ 0.10 | 0.05 to 0.10 | < 0.05 | most dimensions carry nothing |
|
|
140
|
+
| `collapse` | near-duplicate ratio at cosine ≥ 0.99 | < 2% | 2 to 10% | ≥ 10% | top-k is spent on copies of the same item |
|
|
141
|
+
| `ann_recall` | recall@k of ANN neighbors vs exact k-NN | ≥ 0.95 | 0.85 to 0.95 | < 0.85 | the index, not the model, is losing results |
|
|
142
|
+
| `rogue_dimensions` | variance share of the single largest dimension | < 5% | 5 to 20% | ≥ 20% | a few coordinates decide every cosine score |
|
|
143
|
+
| `norms` | coefficient of variation of vector norms | < 0.10 | 0.10 to 0.30 | ≥ 0.30 | inner-product search favors long vectors; zero vectors are reported too |
|
|
144
|
+
| `metric_mismatch` | top-k overlap between cosine and inner product | ≥ 0.90 | 0.70 to 0.90 | < 0.70 | the index metric changes which results come back |
|
|
145
|
+
| `distance_concentration` | random distance / nearest-neighbor distance | ≥ 1.2 | 1.05 to 1.2 | < 1.05 | the nearest neighbor is barely closer than a random item |
|
|
146
|
+
| `outliers` | share of items with an abnormally far nearest neighbor | < 1% | 1 to 5% | ≥ 5% | junk, empty or foreign items; ids are listed in `details` |
|
|
147
|
+
| `connectivity` | share of items in small k-NN graph islands | < 1% | 1 to 10% | ≥ 10% | self-contained groups too small to be clusters; HNSW attaches them badly or never reaches them |
|
|
148
|
+
| `ood_queries` | share of queries farther from the corpus than 99% of corpus items are from each other | < 5% | 5 to 20% | ≥ 20% | queries and corpus do not share a space: wrong model, missing query prefix, uncovered topics |
|
|
149
|
+
| `query_hubness` | skewness of corpus k-occurrence over query results | < 2.0 | 2.0 to 4.0 | ≥ 4.0 | the same few items answer most queries |
|
|
150
|
+
|
|
151
|
+
Every finding in the JSON carries its `healthy_range`, `warn_at` and `fail_at`, plus a `details`
|
|
152
|
+
block with the raw numbers. For `anisotropy` that block includes the cosine the same vectors
|
|
153
|
+
would have after centering, so the agent can see whether centering alone fixes the problem.
|
|
154
|
+
`ann_recall` appears only when you pass an ANN neighbor file with `--ann`; `ood_queries` and
|
|
155
|
+
`query_hubness` only when you pass query embeddings with `--queries`.
|
|
156
|
+
|
|
157
|
+
Thresholds live in `src/vecscope/evidence.py` (`THRESHOLDS`) and were calibrated on
|
|
158
|
+
[real embeddings](#real-embeddings).
|
|
159
|
+
|
|
160
|
+
## Queries and clusters
|
|
161
|
+
|
|
162
|
+
Pass the query embeddings too and `vecscope` checks whether queries live where the corpus
|
|
163
|
+
lives. Asymmetric models used without their query prefix, a different model on the query side,
|
|
164
|
+
or questions about topics the corpus does not cover all show up as `ood_queries`; the same few
|
|
165
|
+
chunks answering everything shows up as `query_hubness`.
|
|
166
|
+
|
|
167
|
+
```bash
|
|
168
|
+
vecscope audit corpus.npy --queries queries.npy
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Add `--clusters 10` to append a per-cluster map: size, tightness, near-duplicate share, how
|
|
172
|
+
much of the hub traffic each cluster takes, and sample ids the agent can look up. It carries
|
|
173
|
+
no verdicts; it tells the agent *where* a problem lives.
|
|
174
|
+
|
|
175
|
+
```text
|
|
176
|
+
clusters (3):
|
|
177
|
+
id size share cos2ctr neardup hubshare sample_ids
|
|
178
|
+
1 1014 33.8% 0.111 0.0% 34.5% [1533, 2934, 800, 1585, 824]
|
|
179
|
+
2 1001 33.4% 0.109 0.0% 33.4% [2550, 1171, 4, 1496, 668]
|
|
180
|
+
0 985 32.8% 0.111 0.0% 32.0% [1910, 2516, 2859, 124, 2392]
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
## Index sweep
|
|
184
|
+
|
|
185
|
+
`vecscope sweep` builds HNSW and IVF indexes with a small parameter grid and reports, for each
|
|
186
|
+
setting, recall@k against exact k-NN, build time and queries per second on this corpus. It then
|
|
187
|
+
recommends the fastest setting that reaches the target recall (default 0.95). It needs the
|
|
188
|
+
`faiss-cpu` extra.
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
pip install "vecscope[index]"
|
|
192
|
+
vecscope sweep embeddings.npy -k 10 --target-recall 0.95
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
```text
|
|
196
|
+
index params recall build s qps
|
|
197
|
+
*hnsw M=16, ef_construction=100, ef_search=32 0.994 0.31 141532
|
|
198
|
+
ivf nlist=76, nprobe=32 0.772 0.02 70925
|
|
199
|
+
*ivf nlist=76, nprobe=64 0.975 0.02 41427
|
|
200
|
+
recommendation: hnsw (M=32, ef_construction=100, ef_search=16) recall=0.985 qps=128425
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
Queries default to a seeded sample of the corpus; pass `--queries` to time real queries. The
|
|
204
|
+
MCP tool is `sweep_index`.
|
|
205
|
+
|
|
206
|
+
With a GPU build of faiss (`pip install faiss-gpu-cu12` on Linux) the IVF index and the exact
|
|
207
|
+
baseline run on the GPU automatically; the output header names the device. HNSW has no GPU
|
|
208
|
+
implementation in faiss and stays on the CPU. Force a device with `--device cpu|gpu`.
|
|
209
|
+
|
|
210
|
+
## Use with an agent
|
|
211
|
+
|
|
212
|
+
`vecscope` ships an MCP server with a single tool, `audit_embeddings`. Register it with any MCP
|
|
213
|
+
client (Claude Code, Cursor, Codex, ...):
|
|
214
|
+
|
|
215
|
+
```bash
|
|
216
|
+
pip install "vecscope[mcp]"
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
```json
|
|
220
|
+
{ "mcpServers": { "vecscope": { "command": "vecscope-mcp" } } }
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Then ask:
|
|
224
|
+
|
|
225
|
+
```text
|
|
226
|
+
Audit the embeddings in data/chunks.npy before we tune the retriever, and tell me
|
|
227
|
+
which finding to fix first.
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
The agent skill in [`.agents/skills/vecscope/SKILL.md`](.agents/skills/vecscope/SKILL.md) tells
|
|
231
|
+
the agent when to run the audit and what to do about each verdict: center or swap the model,
|
|
232
|
+
deduplicate, re-rank, or rebuild the index. A second tool, `sweep_index`, runs the
|
|
233
|
+
[index sweep](#index-sweep). Agents without MCP can run the CLI with `--json` and read the
|
|
234
|
+
same evidence.
|
|
235
|
+
|
|
236
|
+
## Use as a library
|
|
237
|
+
|
|
238
|
+
```python
|
|
239
|
+
import numpy as np
|
|
240
|
+
from vecscope import audit
|
|
241
|
+
|
|
242
|
+
embeddings = np.load("embeddings.npy") # any (n, dim) float array
|
|
243
|
+
|
|
244
|
+
ev = audit(embeddings, k=10)
|
|
245
|
+
ev.verdict # "ok" | "warn" | "fail"
|
|
246
|
+
ev.findings[0].value # measured value of the first finding
|
|
247
|
+
ev.to_dict() # the full evidence, same as --json
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
## CLI options
|
|
251
|
+
|
|
252
|
+
```bash
|
|
253
|
+
vecscope audit vectors.npy -k 20 # neighbors per item (default 10)
|
|
254
|
+
vecscope audit vectors.parquet --json # JSON evidence
|
|
255
|
+
vecscope audit vectors.npy --ann hnsw_neighbors.npy # score ANN recall against exact k-NN
|
|
256
|
+
vecscope audit vectors.npy --queries queries.npy # add query-side findings
|
|
257
|
+
vecscope audit vectors.npy --clusters 10 # append a per-cluster report
|
|
258
|
+
vecscope audit vectors.npy --fail-on warn # exit 1 on WARN as well
|
|
259
|
+
vecscope audit vectors.npy --near-dup 0.98 # cosine threshold for near-duplicates
|
|
260
|
+
vecscope audit vectors.npy --sample 0 # audit every row (default samples 100k)
|
|
261
|
+
vecscope sweep vectors.npy --target-recall 0.95 # HNSW/IVF parameter sweep (faiss extra)
|
|
262
|
+
vecscope sweep vectors.npy --device gpu # IVF on the GPU (faiss GPU build)
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
## Input formats
|
|
266
|
+
|
|
267
|
+
The embedding file holds one vector per row. Every format is converted to a 2-D `float32` array
|
|
268
|
+
of shape `(n, dim)` before the audit. In the examples `embeddings` is that array, for instance
|
|
269
|
+
the output of a model's `encode`:
|
|
270
|
+
|
|
271
|
+
```python
|
|
272
|
+
embeddings = model.encode(texts) # shape (n, dim), e.g. (3000, 128)
|
|
273
|
+
embeddings[0] # the vector of the first item
|
|
274
|
+
# array([ 0.0312, -0.1187, 0.0459, ...], dtype=float32)
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
| Extension | Expected layout | Example |
|
|
278
|
+
| --- | --- | --- |
|
|
279
|
+
| `.npy` | a 2-D numeric array | `np.save("vectors.npy", embeddings)` |
|
|
280
|
+
| `.npz` | the first array in the archive is used | `np.savez("vectors.npz", emb=embeddings)` |
|
|
281
|
+
| `.parquet`, `.pq` | a list-of-float column whose name contains `emb` or `vec`; other columns (ids, text) are ignored | `pd.DataFrame({"id": ids, "embedding": list(embeddings)}).to_parquet("vectors.parquet")` |
|
|
282
|
+
| `.jsonl` | one JSON object per line; the first list-valued field is the vector | `{"id": 7, "text": "...", "embedding": [0.1, -0.3, ...]}` |
|
|
283
|
+
| `.json` | a JSON array of objects (as above) or an array of lists | `[[0.1, -0.3, ...], [0.2, 0.5, ...]]` |
|
|
284
|
+
| `.csv`, `.tsv`, `.txt` | one vector per line, numbers only, no header or id column | `0.1,-0.3,0.8,...` |
|
|
285
|
+
|
|
286
|
+
The optional `--ann` file (or `ann_path` in MCP) is an integer array of shape `(n, k)`: row `i`
|
|
287
|
+
holds the ids of the `k` neighbors an ANN index returned for item `i`, in the same row order as
|
|
288
|
+
the embedding file. Any of the formats above works. Ids refer to row positions, so the file
|
|
289
|
+
must come from an index built on exactly these rows; with `--sample` in effect the ANN file is
|
|
290
|
+
ignored because the row ids no longer line up.
|
|
291
|
+
|
|
292
|
+
The optional `--queries` file (`queries_path` in MCP) holds query embeddings in any of the
|
|
293
|
+
same formats, with the same `dim` as the corpus.
|
|
294
|
+
|
|
295
|
+
## How vecscope works
|
|
296
|
+
|
|
297
|
+
`vecscope` computes exact k-NN once over the matrix and derives every diagnostic from it:
|
|
298
|
+
the k-occurrence distribution gives hubness and anti-hubs, nearest-neighbor cosines give
|
|
299
|
+
collapse, outliers and distance concentration, the k-NN graph gives connectivity, random pairs
|
|
300
|
+
and the covariance spectrum give anisotropy, effective rank and rogue dimensions, vector norms
|
|
301
|
+
give the norm spread, a second inner-product pass (skipped when norms are uniform) gives
|
|
302
|
+
metric mismatch, and the ANN neighbor file, if present, is compared against the exact
|
|
303
|
+
neighbors for recall. Query embeddings, if present, get their own exact search against the
|
|
304
|
+
corpus for the query-side findings.
|
|
305
|
+
|
|
306
|
+
Everything is deterministic: fixed seeds, exact neighbors, no learned components, no network.
|
|
307
|
+
Two runs on the same file give the same JSON. Exact k-NN is quadratic in the number of rows,
|
|
308
|
+
so inputs above 100,000 rows are audited on a seeded random sample unless you pass `--sample 0`.
|
|
309
|
+
|
|
310
|
+
## Real embeddings
|
|
311
|
+
|
|
312
|
+
Thresholds are calibrated on public, pre-computed embeddings from eight models: OpenAI
|
|
313
|
+
ada-002 and text-embedding-3-large, Cohere embed-v3, all-MiniLM-L6-v2, bge-m3,
|
|
314
|
+
mxbai-embed-large-v1 and gte-base-en-v1.5. The table shows the measured value for the first 40k
|
|
315
|
+
rows of each dataset; cells without a label are `ok`. `python examples/real_models.py`
|
|
316
|
+
reproduces it, no model download needed.
|
|
317
|
+
|
|
318
|
+
| finding | Cohere v3 (ko wiki) | Cohere v3 (en wiki) | ada-002 (dbpedia) | 3-large (dbpedia) | MiniLM-L6 (wiki) | bge-m3 (wiki) | mxbai-large (wiki) | gte-base (arxiv) |
|
|
319
|
+
|---|---|---|---|---|---|---|---|---|
|
|
320
|
+
| hubness | 1.47 | 1.16 | 1.56 | 1.21 | 1.59 | 1.85 | 1.43 | 2.28 warn |
|
|
321
|
+
| antihubs | 0.009 | 0.005 | 0.013 | 0.005 | 0.022 | 0.026 | 0.020 | 0.013 |
|
|
322
|
+
| anisotropy | 0.333 warn | 0.228 | 0.692 **fail** | 0.086 | 0.026 | 0.307 warn | 0.330 warn | 0.210 |
|
|
323
|
+
| effective_rank | 0.158 | 0.178 | 0.090 warn | 0.148 | 0.285 | 0.143 | 0.087 warn | 0.136 |
|
|
324
|
+
| collapse | 0.002 | 0.002 | 0.000 | 0.000 | 0.001 | 0.000 | 0.017 | 0.000 |
|
|
325
|
+
| rogue_dimensions | 0.003 | 0.003 | 0.002 | 0.002 | 0.004 | 0.002 | 0.002 | 0.002 |
|
|
326
|
+
| norms | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.007 |
|
|
327
|
+
| metric_mismatch | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.952 |
|
|
328
|
+
| distance_concentration | 3.00 | 3.00 | 2.53 | 1.86 | 2.93 | 2.77 | 3.99 | 2.30 |
|
|
329
|
+
| outliers | 0.001 | 0.001 | 0.000 | 0.000 | 0.001 | 0.002 | 0.003 | 0.000 |
|
|
330
|
+
| connectivity | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
|
|
331
|
+
|
|
332
|
+
ada-002's well-known anisotropy shows up as the only `fail`: its mean unit vector has norm 0.83,
|
|
333
|
+
so unrelated documents start from a cosine floor near 0.7. Its successor is isotropic on the
|
|
334
|
+
same corpus (norm 0.29). Centering brings all three models to a random-pair cosine of 0.00, which
|
|
335
|
+
is why the finding reports that number. Hubness skewness of 1.2 to 1.9 is the normal range for
|
|
336
|
+
text embeddings, so the `warn` threshold sits at 2.0; gte-base on arXiv abstracts sits just
|
|
337
|
+
above it. Three good models land in the anisotropy `warn` band (0.30 to 0.33): mild, and
|
|
338
|
+
their `details` show that centering removes it.
|
|
339
|
+
|
|
340
|
+
## FAQ
|
|
341
|
+
|
|
342
|
+
<details>
|
|
343
|
+
<summary><strong>Does it send my vectors anywhere?</strong></summary>
|
|
344
|
+
|
|
345
|
+
No. `vecscope` is NumPy only and runs entirely on your machine. It never loads a model or opens
|
|
346
|
+
a network connection.
|
|
347
|
+
|
|
348
|
+
</details>
|
|
349
|
+
|
|
350
|
+
<details>
|
|
351
|
+
<summary><strong>How large a matrix can it handle?</strong></summary>
|
|
352
|
+
|
|
353
|
+
Exact k-NN is quadratic in the number of rows. By default `vecscope` audits a seeded random
|
|
354
|
+
sample of 100,000 rows when the input is larger, and says so in the output. Use `--sample 0`
|
|
355
|
+
to audit everything.
|
|
356
|
+
|
|
357
|
+
</details>
|
|
358
|
+
|
|
359
|
+
<details>
|
|
360
|
+
<summary><strong>My production model fails the anisotropy check. Is the model broken?</strong></summary>
|
|
361
|
+
|
|
362
|
+
Not necessarily. Many popular text embedding models are strongly anisotropic by construction.
|
|
363
|
+
The finding tells you that raw cosine scores are compressed into a narrow band; centering or
|
|
364
|
+
whitening the vectors, or re-ranking, usually restores discrimination. The `details` block
|
|
365
|
+
reports the cosine after centering so you can see whether that is enough. The verdict is
|
|
366
|
+
evidence for the agent, not a grade for the model.
|
|
367
|
+
|
|
368
|
+
</details>
|
|
369
|
+
|
|
370
|
+
<details>
|
|
371
|
+
<summary><strong>Which agents can use it?</strong></summary>
|
|
372
|
+
|
|
373
|
+
Any agent that supports local MCP servers, and any agent that can run a shell command and read
|
|
374
|
+
JSON. The skill file describes both paths.
|
|
375
|
+
|
|
376
|
+
</details>
|
|
377
|
+
|
|
378
|
+
## Documentation
|
|
379
|
+
|
|
380
|
+
- [Agent skill](.agents/skills/vecscope/SKILL.md): when to run the audit and what to do with each verdict
|
|
381
|
+
- [Examples](examples/): synthetic matrices and the real-embedding calibration script
|
|
382
|
+
- [Changelog](CHANGELOG.md)
|
|
383
|
+
|
|
384
|
+
## Contributing
|
|
385
|
+
|
|
386
|
+
Issues and pull requests are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for how to run
|
|
387
|
+
the tests and what a good change looks like. Security reports go through
|
|
388
|
+
[SECURITY.md](SECURITY.md).
|
|
389
|
+
|
|
390
|
+
## License
|
|
391
|
+
|
|
392
|
+
[MIT](LICENSE)
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
vecscope/__init__.py,sha256=GgupkgVroDz7T8lUc6nwFaxjeaahz3HZIV9EGfLkEyo,190
|
|
2
|
+
vecscope/_knn.py,sha256=pMmVOtkHZjoCGeV1esb7sZwtIBermj0_IQf7Y8R6Ykk,1544
|
|
3
|
+
vecscope/cli.py,sha256=zoh7b9oA86ZNGubFGLvC1xpAJBqyjjhYF3fN51DG7og,4529
|
|
4
|
+
vecscope/evidence.py,sha256=bvlUPztvxWmdxsGbhhXvDD9hWna4Jg2qmeldIa2R8QE,13286
|
|
5
|
+
vecscope/io.py,sha256=kFY2ie_UHJWngJUOHpJQo2JVcTD4dP3znt98JCi-yJY,2295
|
|
6
|
+
vecscope/mcp_server.py,sha256=ZUF4vud6tLVKCQpb7kGuOc4C00PAVh4PYsx_dh5pLwY,2203
|
|
7
|
+
vecscope/sweep.py,sha256=O2FmzY9JFwQfzd1xVyq2xkZKbQCda3dpk4Btz45MfWI,8108
|
|
8
|
+
vecscope/diagnostics/__init__.py,sha256=QodzaukPL7R4dwW0KidLeKTCK6wfSnDZQz7voX75Ev0,693
|
|
9
|
+
vecscope/diagnostics/anisotropy.py,sha256=RVLNMnGot6OuowyWgFx8oZpYTdozZCZ9J9OPhwkQCFQ,1719
|
|
10
|
+
vecscope/diagnostics/ann_recall.py,sha256=sJWZ6WVdq8AJd36ThpHZHgos7SgjXLvZMQ7icmNr0sY,908
|
|
11
|
+
vecscope/diagnostics/clusters.py,sha256=ehrMysqIXtZ5WHSkGA4KbsUgbIZt1cMtBbJjOn5RmQ4,2218
|
|
12
|
+
vecscope/diagnostics/collapse.py,sha256=bPDXdROXlXel8XP-JWPhC-3taMzJXg7LvGlOcP03uiw,1066
|
|
13
|
+
vecscope/diagnostics/geometry.py,sha256=5ixDjywXTu9djeH7MCRgCwOOhAqx5VOLJAVwLrybNG0,4969
|
|
14
|
+
vecscope/diagnostics/hubness.py,sha256=fwYCrj1gUAZm-YJmblhs4NOwZgSmDfT6NkpEqtozlYk,1109
|
|
15
|
+
vecscope/diagnostics/queries.py,sha256=TjN2X4a040IrAtQuFE1BYjZnIjHjcZaJ0agg4ZqxGaI,2559
|
|
16
|
+
vecscope-0.1.0.dist-info/METADATA,sha256=S9kpgQJVWpuCjhPNScU9WOeaq9YYAUMuyVne3C3JYL8,20811
|
|
17
|
+
vecscope-0.1.0.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
|
|
18
|
+
vecscope-0.1.0.dist-info/entry_points.txt,sha256=K5banzv5p3798f2nQV2Fguy-Vuio-gu4CYgmNcNHVJM,87
|
|
19
|
+
vecscope-0.1.0.dist-info/licenses/LICENSE,sha256=R7zxipY_3zJ7U4bcLNBQta6AgV00lLVP4MKY9w5rAJk,1067
|
|
20
|
+
vecscope-0.1.0.dist-info/RECORD,,
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Daehee Yun
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|