squeze-cli 2.2.1 → 3.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -26,7 +26,7 @@ flowchart TB
26
26
  * **Enumeration.** Git (`ls-files --cached --others --exclude-standard`) when available, otherwise a bounded directory walk honouring nested `.gitignore` and `.reposqueeze-v2ignore`. Limits: 100,000 files, 2 GiB, 2 MiB per file, depth 60.
27
27
  * **Text pipeline.** One hook (`Repository.textFromBytes`) turns bytes into the text that is chunked, indexed, hashed and shown. It rejects binaries and replaces secret values with `[REDACTED:<rule>]` without changing the number of lines, so citations stay valid. Changing its output requires bumping `TEXT_PIPELINE_VERSION`.
28
28
  * **Chunking.** `parser.ts` uses web-tree-sitter for 20 languages. Definitions include leading doc comments and decorators and carry `signature`, `scope` and `doc`. Long definitions are split into overlapping pieces; every file also gets 64-line windows so nothing is unsearchable. Generated or minified files get bounded chunking. `chunkers.ts` supplies structural spans (headings, keys, elements, rules, statements, stages, targets) for Markdown, JSON, YAML, TOML, XML (including Maven and AEM `.content.xml`), HTML, CSS, SQL, Dockerfile, Makefile, shell, protobuf and HCL.
29
- * **Persistence.** `store.ts` keeps the cache outside the repository (`REPOSQUEEZE_CACHE_DIR`, XDG or platform cache directory, keyed by the repository's real path). Files are checksummed containers written atomically with private permissions; every loaded field is validated, and any cache failure degrades to in-memory operation. Entries are keyed by path, content hash, parser, tokenizer and text-pipeline versions. For repositories up to 5,000 files the content hash is re-verified on every run; above that, size and modification time are trusted (`REPOSQUEEZE_REFRESH_INDEX=1` overrides).
29
+ * **Persistence.** `store.ts` keeps the cache outside the repository (`REPOSQUEEZE_CACHE_DIR`, XDG or platform cache directory, keyed by the repository's real path). Files are checksummed containers written atomically with private permissions; every loaded field is validated, and any cache failure degrades to in-memory operation. Entries are keyed by path, content hash, parser, tokenizer and text-pipeline versions. For repositories up to 5,000 files the content hash is re-verified on every run; above that, size, modification time, change time and inode are trusted (`REPOSQUEEZE_REFRESH_INDEX=1` overrides). A file whose newest timestamp falls within two seconds of the scan (`RACY_WINDOW_MS`) is always re-read, because a same-size edit can keep its modification time within one timestamp tick; change time is what catches a same-size edit whose modification time was restored.
30
30
  * **Large repositories.** Above 5,000 eligible files a per-file term sketch selects 600–1,800 candidate files that are then parsed; the rest of the pipeline is unchanged.
31
31
  * **Search index.** `search-index.ts` holds per-chunk term rows and postings in typed arrays. `retrieve()` computes BM25 (k1 = 1.2, b = 0.75) plus bonuses for terms in the symbol name and path, and accepts weighted terms.
32
32
 
package/docs/GRAPH.md CHANGED
@@ -34,4 +34,4 @@ impact of binding/binding.go: 7 files via imports within 3 hops (direct 6), +17
34
34
  d1 (6): context.go, mode.go, context_test.go, deprecated.go, deprecated_test.go, mode_test.go
35
35
  ```
36
36
 
37
- Measured accuracy, token cost and build time: [benchmarks/v22/graph.md](../benchmarks/v22/graph.md). Limits: name-based references over-report importers in some repositories (precision 59% on average), Rust `use` of items is partly unresolved, and cascade-mode graphs on repositories above 5,000 files cover only the parsed candidates.
37
+ Measured accuracy, token cost and build time: [benchmarks/v22/graph.md](../benchmarks/v22/graph.md). Limits: Go imports resolve to one file instead of the whole package, so impact precision is 25% on gin and 39% on ripgrep (75% on average over six repositories), Rust `use` of items is partly unresolved, and cascade-mode graphs on repositories above 5,000 files cover only the parsed candidates.
@@ -21,6 +21,24 @@ The initial development query built the persistent 4.0 MB compressed index in 10
21
21
 
22
22
  The repository contained 50,001 eligible files during evaluation because its `.reposqueeze-v2ignore` control file was also indexed. Only 200 exact source files were parsed per query; ranking and packing still saw the complete file-level inventory.
23
23
 
24
+ ## Incremental refresh on a real repository
25
+
26
+ Django at commit `f24f38c` (5,644 eligible Python files, Git working tree), Apple Silicon workstation, one run per mode. Each probe edits the tree, takes a snapshot and checks for a unique identifier from the edit. The probe script is local and is not committed.
27
+
28
+ | Step | Default (cascade) | Full index (`REPOSQUEEZE_LARGE_THRESHOLD=10000`) |
29
+ | --- | ---: | ---: |
30
+ | Cold index: snapshot / graph | 6.2 s / 5.1 s, 200 candidates parsed | 23.9 s / 3.3 s, all 5,644 parsed |
31
+ | No-op refresh, snapshot median (min–max) | 766 ms (555–1100) | 674 ms (543–1052) |
32
+ | Single-file edit, snapshot median (min–max) | 579 ms (507–1787) | 955 ms (864–1367) |
33
+ | Single-file edit, graph rebuild median | 789 ms | 2107 ms |
34
+ | Single-file edit, total median over 5 edits | 1368 ms | 3062 ms |
35
+
36
+ - Neither mode meets a 300 ms single-file target. The graph rebuild step calls `loadGraph`, which takes its own snapshot first (`src/graph-store.ts:38-52`), so part of that step is a repeated snapshot. Removing it is a follow-up. It is not removed here because the query may affect candidate selection and needs its own test.
37
+ - Cascade mode parses 200 of 5,644 files. Its graph is marked `partial`, and every view says so. Full mode parses every file and has a 3.8× slower cold snapshot.
38
+ - Earlier scratch runs on this clone, before the probe script was fixed, measured single-file totals of 1.9 s and 2.7 s in cascade mode. Those runs did not record phase timings, so they are not in the table.
39
+ - Probes: a touch-only edit leaves the revision unchanged. A same-size identifier edit (`slugify` to `slugifx`) changes the revision and is found through change time, because the modification time cannot be restored exactly from user space. An added file is found, and deleting it reverts the revision.
40
+ - The min–max columns show run-to-run variance within a single run. Treat these as one machine's measurements, not guarantees.
41
+
24
42
  ## Reproduce
25
43
 
26
44
  ```sh
@@ -41,6 +59,6 @@ Raw evidence:
41
59
 
42
60
  This benchmark proves operation on its generated distribution, not perfect retrieval for every 50,000-file repository. Its identifiers are deterministic and the gold files contain exact term evidence. Real monorepos have generated code, duplicated APIs, cross-language names, build indirection and tasks that do not name their relevant concepts.
43
61
 
44
- The candidate stage can miss a file with no shared path or indexed term. Files with more than 8,192 unique normalized terms use deterministic term sampling. Increase `REPOSQUEEZE_MAX_CANDIDATES`, provide paths or identifiers, or use ordinary search when obligations are missing. Set `REPOSQUEEZE_REFRESH_INDEX=1` after timestamp-preserving edits.
62
+ The candidate stage can miss a file with no shared path or indexed term. Files with more than 8,192 unique normalized terms use deterministic term sampling. Increase `REPOSQUEEZE_MAX_CANDIDATES`, provide paths or identifiers, or use ordinary search when obligations are missing. Edits are detected by size, modification time, change time and inode, so a timestamp-preserving edit is re-read. `REPOSQUEEZE_REFRESH_INDEX=1` still forces a full re-read.
45
63
 
46
64
  The 100% held-out result is scale-corpus retrieval accuracy. It is not 100% answer accuracy, patch success or Copilot-credit reduction.
@@ -0,0 +1,27 @@
1
+ # Squeze standard for code structure answers
2
+
3
+ This is the bar for the code graph (`squeze graph`) and the caveman output style. A target counts as met only with a measurement in this repository. Compared with Graphify, Squeze is judged on the questions a coding agent asks and on the tokens those answers cost.
4
+
5
+ | # | Target | Measured now | Status |
6
+ | --- | --- | --- | --- |
7
+ | 1 | File dependents (`--impact`): precision ≥ 0.90 and recall ≥ 0.80 against a use-based ground truth (importers of a file's package that use a name the file defines, Go; importers, JVM/Python/JS/Rust) on six repositories | gin 1.00 / 0.84 (Go, after the fix in this release); flask, gson, thor, express, ripgrep measured before the Go fix (flask 1.00/0.87, gson 1.00/1.00, thor 1.00/1.00, express 0.88/0.60, ripgrep 0.39/0.71) | Partly met. ripgrep and express are below target. |
8
+ | 2 | Call edges ("who calls X"): precision proxy ≥ 0.80 and recall proxy ≥ 0.60 against Graphify's call edges on the same repositories | gin: 1,070 Squeze calls, 1,380 Graphify calls. 67% of Squeze's calls are also in Graphify (precision proxy); 52% of Graphify's are in Squeze (recall proxy). Before the unique-name fallback the recall proxy was 0.31. | Not met. Both are agreement with another extractor, not proof of correctness. |
9
+ | 3 | Head-to-head against Graphify on the same repositories, questions and token accounting, reported whatever the result | calls on gin (agreement above, raw in `benchmarks/v22/graph-calls-vs-graphify-gin.json`); one qualitative question (binding/binding.go): Squeze returned all 6 dependents, Graphify returned the file's own symbols | Partly met. Calls on one repository; dependents, paths and token counts on Graphify not yet compared. |
10
+ | 4 | Caveman output: at least 15% fewer tokens with no path or identifier lost, in every graph mode | impact 15.7% fewer tokens over ten gin files, 0 of 153 paths lost; path and neighbours covered by a test (shorter, all paths kept) but not yet measured | Partly met (impact only). |
11
+ | 5 | Every output within the requested token budget | tests in `tests/graph.test.ts` and `tests/mcp.test.ts` | Met. |
12
+ | 6 | Deterministic: same repository state, same bytes | cold, warm and restart runs in the index tests | Met. |
13
+ | 7 | Covers documents and configuration files, not only code | structural chunkers feed the retrieval index, not the graph | Not met. Graphify covers docs, papers and images. |
14
+ | 8 | Reproducible evidence in `benchmarks/v22/` and the README, including the wrong numbers that were found and fixed | `benchmarks/v22/graph.md`, README section 9, correction recorded | Met. |
15
+
16
+ ## What "better than Graphify" must mean
17
+
18
+ Squeze wins only where it is measured to win. Each claim needs the same repositories, the same questions and the same token count for both tools.
19
+
20
+ * **Targets that must hold before any comparison claim:** 1, 2, 3 and 4 across all six repositories.
21
+ * **Where Squeze is already designed to be stronger:** fixed token budgets with exact citations, a caveman output style that cuts tokens without losing paths, and one server that also serves retrieval (`context`, `expand`, `map`, `graph`).
22
+ * **Where Graphify is stronger today:** call edges, documents and images, and export formats (Obsidian, GraphML, Neo4j).
23
+
24
+ ## Ground truth rules
25
+
26
+ * A Go file's dependents are importers of its package that use a name the file defines. A word match on the file name is not a valid ground truth for Go (see `scripts/graph-eval.ts`).
27
+ * An import is an edge only if the importer uses something the target defines. The fallback that linked every package file was removed because it over-reported dependents (use-based precision went from 0.19 to 1.00 on gin).
@@ -0,0 +1,317 @@
1
+ :root {
2
+ color-scheme: light dark;
3
+ }
4
+
5
+ :root,
6
+ body {
7
+ --bg: var(--vscode-editor-background, #1e1e1e);
8
+ --fg: var(--vscode-foreground, #cccccc);
9
+ --muted: var(--vscode-descriptionForeground, #9d9d9d);
10
+ --panel: var(--vscode-sideBar-background, #252526);
11
+ --border: var(--vscode-panel-border, #3c3c3c);
12
+ --accent: var(--vscode-focusBorder, #007fd4);
13
+ --button: var(--vscode-button-background, #0e639c);
14
+ --button-fg: var(--vscode-button-foreground, #ffffff);
15
+ --input: var(--vscode-input-background, #3c3c3c);
16
+ --input-fg: var(--vscode-input-foreground, #cccccc);
17
+ --hover: var(--vscode-list-hoverBackground, #2a2d2e);
18
+ --selected: var(--vscode-list-activeSelectionBackground, #094771);
19
+ --selected-fg: var(--vscode-list-activeSelectionForeground, #ffffff);
20
+ --warn: var(--vscode-editorWarning-foreground, #cca700);
21
+ }
22
+
23
+ * { box-sizing: border-box; }
24
+
25
+ [hidden] { display: none !important; }
26
+
27
+ html, body {
28
+ margin: 0;
29
+ height: 100%;
30
+ background: var(--bg);
31
+ color: var(--fg);
32
+ font: 13px/1.45 var(--vscode-font-family, system-ui, -apple-system, "Segoe UI", sans-serif);
33
+ }
34
+
35
+ body {
36
+ display: flex;
37
+ flex-direction: column;
38
+ overflow: hidden;
39
+ }
40
+
41
+ .bar {
42
+ display: flex;
43
+ align-items: center;
44
+ gap: 12px;
45
+ padding: 8px 12px;
46
+ border-bottom: 1px solid var(--border);
47
+ background: var(--panel);
48
+ flex-wrap: wrap;
49
+ }
50
+
51
+ .bar h1 {
52
+ font-size: 15px;
53
+ font-weight: 600;
54
+ margin: 0;
55
+ }
56
+
57
+ .summary {
58
+ margin: 0;
59
+ color: var(--muted);
60
+ flex: 1 1 280px;
61
+ min-width: 0;
62
+ overflow: hidden;
63
+ text-overflow: ellipsis;
64
+ white-space: nowrap;
65
+ }
66
+
67
+ .actions {
68
+ display: flex;
69
+ gap: 6px;
70
+ }
71
+
72
+ button {
73
+ font: inherit;
74
+ color: var(--button-fg);
75
+ background: var(--button);
76
+ border: 1px solid transparent;
77
+ border-radius: 3px;
78
+ padding: 3px 10px;
79
+ cursor: pointer;
80
+ }
81
+
82
+ button:hover { filter: brightness(1.1); }
83
+
84
+ button.ghost {
85
+ background: transparent;
86
+ color: var(--fg);
87
+ border-color: var(--border);
88
+ }
89
+
90
+ button[aria-pressed="false"] {
91
+ background: transparent;
92
+ color: var(--muted);
93
+ border-color: var(--border);
94
+ }
95
+
96
+ :focus-visible {
97
+ outline: 2px solid var(--accent);
98
+ outline-offset: 2px;
99
+ }
100
+
101
+ .layout {
102
+ flex: 1;
103
+ min-height: 0;
104
+ display: grid;
105
+ grid-template-columns: minmax(0, 1fr) 300px;
106
+ }
107
+
108
+ .stage {
109
+ position: relative;
110
+ min-width: 0;
111
+ min-height: 0;
112
+ overflow: hidden;
113
+ }
114
+
115
+ .stage canvas {
116
+ position: absolute;
117
+ inset: 0;
118
+ width: 100%;
119
+ height: 100%;
120
+ display: block;
121
+ cursor: grab;
122
+ }
123
+
124
+ .stage canvas.dragging { cursor: grabbing; }
125
+ .stage canvas.pointer { cursor: pointer; }
126
+
127
+ .tip {
128
+ position: absolute;
129
+ left: 0;
130
+ top: 0;
131
+ max-width: 320px;
132
+ padding: 4px 8px;
133
+ border: 1px solid var(--border);
134
+ border-radius: 3px;
135
+ background: var(--panel);
136
+ color: var(--fg);
137
+ font-size: 12px;
138
+ pointer-events: none;
139
+ white-space: nowrap;
140
+ overflow: hidden;
141
+ text-overflow: ellipsis;
142
+ }
143
+
144
+ .empty {
145
+ position: absolute;
146
+ inset: 0;
147
+ display: flex;
148
+ align-items: center;
149
+ justify-content: center;
150
+ padding: 24px;
151
+ text-align: center;
152
+ color: var(--muted);
153
+ }
154
+
155
+ .side {
156
+ display: flex;
157
+ flex-direction: column;
158
+ gap: 12px;
159
+ min-height: 0;
160
+ overflow-y: auto;
161
+ padding: 12px;
162
+ border-left: 1px solid var(--border);
163
+ background: var(--panel);
164
+ }
165
+
166
+ .side section h2 {
167
+ font-size: 11px;
168
+ font-weight: 600;
169
+ text-transform: uppercase;
170
+ letter-spacing: 0.04em;
171
+ color: var(--muted);
172
+ margin: 0 0 6px;
173
+ }
174
+
175
+ .side input[type="search"] {
176
+ width: 100%;
177
+ font: inherit;
178
+ color: var(--input-fg);
179
+ background: var(--input);
180
+ border: 1px solid var(--border);
181
+ border-radius: 3px;
182
+ padding: 4px 6px;
183
+ }
184
+
185
+ .results {
186
+ list-style: none;
187
+ margin: 6px 0 0;
188
+ padding: 0;
189
+ max-height: 180px;
190
+ overflow-y: auto;
191
+ }
192
+
193
+ .results li + li { margin-top: 2px; }
194
+
195
+ .results button,
196
+ .neighbours button,
197
+ .legend button {
198
+ display: flex;
199
+ align-items: center;
200
+ gap: 6px;
201
+ width: 100%;
202
+ text-align: left;
203
+ background: transparent;
204
+ color: var(--fg);
205
+ border: 1px solid transparent;
206
+ padding: 3px 6px;
207
+ border-radius: 3px;
208
+ }
209
+
210
+ .results button:hover,
211
+ .neighbours button:hover,
212
+ .legend button:hover { background: var(--hover); }
213
+
214
+ .results button[aria-current="true"] { background: var(--selected); color: var(--selected-fg); }
215
+
216
+ .legend {
217
+ list-style: none;
218
+ margin: 0;
219
+ padding: 0;
220
+ }
221
+
222
+ .legend button[aria-pressed="false"] { color: var(--muted); }
223
+
224
+ .count {
225
+ margin-left: auto;
226
+ color: var(--muted);
227
+ font-variant-numeric: tabular-nums;
228
+ }
229
+
230
+ .swatch {
231
+ flex: none;
232
+ width: 10px;
233
+ height: 10px;
234
+ border-radius: 2px;
235
+ }
236
+
237
+ .path {
238
+ overflow: hidden;
239
+ text-overflow: ellipsis;
240
+ white-space: nowrap;
241
+ min-width: 0;
242
+ }
243
+
244
+ dl {
245
+ display: grid;
246
+ grid-template-columns: auto 1fr;
247
+ gap: 2px 10px;
248
+ margin: 0;
249
+ }
250
+
251
+ dt { color: var(--muted); }
252
+ dd { margin: 0; font-variant-numeric: tabular-nums; }
253
+
254
+ .neighbours {
255
+ list-style: none;
256
+ margin: 0;
257
+ padding: 0;
258
+ max-height: 220px;
259
+ overflow-y: auto;
260
+ }
261
+
262
+ .info-path {
263
+ font-weight: 600;
264
+ word-break: break-all;
265
+ margin: 0 0 6px;
266
+ }
267
+
268
+ .open-btn { margin-top: 8px; }
269
+
270
+ .notes {
271
+ margin: 0;
272
+ padding-left: 18px;
273
+ color: var(--warn);
274
+ }
275
+
276
+ kbd {
277
+ font: 11px var(--vscode-editor-font-family, ui-monospace, monospace);
278
+ padding: 0 4px;
279
+ border: 1px solid var(--border);
280
+ border-radius: 3px;
281
+ background: var(--bg);
282
+ }
283
+
284
+ .keys {
285
+ color: var(--muted);
286
+ font-size: 12px;
287
+ }
288
+
289
+ .keys div { margin: 2px 0; }
290
+
291
+ .sr-only {
292
+ position: absolute;
293
+ width: 1px;
294
+ height: 1px;
295
+ padding: 0;
296
+ margin: -1px;
297
+ overflow: hidden;
298
+ clip: rect(0, 0, 0, 0);
299
+ white-space: nowrap;
300
+ border: 0;
301
+ }
302
+
303
+ @media (max-width: 720px) {
304
+ body { overflow: auto; }
305
+
306
+ .layout {
307
+ flex: none;
308
+ grid-template-columns: minmax(0, 1fr);
309
+ grid-template-rows: 60vh auto;
310
+ }
311
+
312
+ .side {
313
+ border-left: 0;
314
+ border-top: 1px solid var(--border);
315
+ overflow: visible;
316
+ }
317
+ }