squeze-cli 0.0.0-stage → 2.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -0
- package/LICENSE +21 -0
- package/README.md +318 -2
- package/THIRD_PARTY_NOTICES.md +11 -0
- package/assets/libc-shim.wasm +0 -0
- package/assets/licenses/BUNDLED-DEPENDENCIES.txt +311 -0
- package/assets/licenses/tree-sitter-c-sharp.txt +24 -0
- package/assets/licenses/tree-sitter-c.txt +24 -0
- package/assets/licenses/tree-sitter-cpp.txt +24 -0
- package/assets/licenses/tree-sitter-go.txt +24 -0
- package/assets/licenses/tree-sitter-java.txt +24 -0
- package/assets/licenses/tree-sitter-javascript.txt +24 -0
- package/assets/licenses/tree-sitter-python.txt +24 -0
- package/assets/licenses/tree-sitter-ruby.txt +25 -0
- package/assets/licenses/tree-sitter-rust.txt +24 -0
- package/assets/licenses/tree-sitter-typescript.txt +24 -0
- package/assets/licenses/tree-sitter-wasms.txt +24 -0
- package/assets/licenses/web-tree-sitter.txt +21 -0
- package/assets/tree-sitter-bash.wasm +0 -0
- package/assets/tree-sitter-c.wasm +0 -0
- package/assets/tree-sitter-c_sharp.wasm +0 -0
- package/assets/tree-sitter-cpp.wasm +0 -0
- package/assets/tree-sitter-elixir.wasm +0 -0
- package/assets/tree-sitter-go.wasm +0 -0
- package/assets/tree-sitter-java.wasm +0 -0
- package/assets/tree-sitter-javascript.wasm +0 -0
- package/assets/tree-sitter-kotlin.wasm +0 -0
- package/assets/tree-sitter-objc.wasm +0 -0
- package/assets/tree-sitter-ocaml.wasm +0 -0
- package/assets/tree-sitter-php.wasm +0 -0
- package/assets/tree-sitter-python.wasm +0 -0
- package/assets/tree-sitter-ruby.wasm +0 -0
- package/assets/tree-sitter-rust.wasm +0 -0
- package/assets/tree-sitter-scala.wasm +0 -0
- package/assets/tree-sitter-solidity.wasm +0 -0
- package/assets/tree-sitter-tsx.wasm +0 -0
- package/assets/tree-sitter-typescript.wasm +0 -0
- package/assets/tree-sitter-zig.wasm +0 -0
- package/assets/tree-sitter.wasm +0 -0
- package/dist/cli.cjs +57923 -0
- package/dist/extension.cjs +17133 -0
- package/dist/index-worker.cjs +6181 -0
- package/docs/ARCHITECTURE.md +58 -0
- package/docs/GRAPH.md +37 -0
- package/docs/SCALE_EVALUATION.md +46 -0
- package/docs/V2_EVALUATION.md +106 -0
- package/docs/V2_RESEARCH.md +61 -0
- package/docs/V3_RESEARCH.md +1109 -0
- package/docs/V3_RESEARCH_SOURCES.md +215 -0
- package/package.json +282 -4
- package/skills/squeze/SKILL.md +12 -0
- package/third_party/tree-sitter-c-sharp.txt +24 -0
- package/third_party/tree-sitter-c.txt +24 -0
- package/third_party/tree-sitter-cpp.txt +24 -0
- package/third_party/tree-sitter-go.txt +24 -0
- package/third_party/tree-sitter-java.txt +24 -0
- package/third_party/tree-sitter-javascript.txt +24 -0
- package/third_party/tree-sitter-python.txt +24 -0
- package/third_party/tree-sitter-ruby.txt +25 -0
- package/third_party/tree-sitter-rust.txt +24 -0
- package/third_party/tree-sitter-typescript.txt +24 -0
- package/third_party/tree-sitter-wasms.txt +24 -0
- package/third_party/web-tree-sitter.txt +21 -0
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
# RepoSqueeze architecture (2.2)
|
|
2
|
+
|
|
3
|
+
```mermaid
|
|
4
|
+
flowchart TB
|
|
5
|
+
subgraph Index
|
|
6
|
+
A[Repository files] --> B[Enumerate: git ls-files or bounded walk]
|
|
7
|
+
B --> C[Filters: ignore rules, binary, size, symlinks, secret paths]
|
|
8
|
+
C --> D[Read + redact secret values in place]
|
|
9
|
+
D --> E[Chunk: tree-sitter (20 languages) or structural chunkers]
|
|
10
|
+
E --> F[(Cache outside the repo:<br/>per-file entries, term rows, dictionary)]
|
|
11
|
+
F --> G[Inverted index over chunks]
|
|
12
|
+
end
|
|
13
|
+
subgraph Query
|
|
14
|
+
Q[Task, error, trace] --> Q1[Condense if > 8,000 chars]
|
|
15
|
+
Q1 --> Q2[Up to 6 deterministic views]
|
|
16
|
+
Q2 --> R[BM25 per view]
|
|
17
|
+
G --> R
|
|
18
|
+
R --> S[Fusion + feedback + identifier/phrase + priors<br/>(+ optional frames, git)]
|
|
19
|
+
S --> T[Packing under the token budget]
|
|
20
|
+
end
|
|
21
|
+
T --> U[Excerpts, pointers, receipts]
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Index
|
|
25
|
+
|
|
26
|
+
* **Enumeration.** Git (`ls-files --cached --others --exclude-standard`) when available, otherwise a bounded directory walk honouring nested `.gitignore` and `.reposqueeze-v2ignore`. Limits: 100,000 files, 2 GiB, 2 MiB per file, depth 60.
|
|
27
|
+
* **Text pipeline.** One hook (`Repository.textFromBytes`) turns bytes into the text that is chunked, indexed, hashed and shown. It rejects binaries and replaces secret values with `[REDACTED:<rule>]` without changing the number of lines, so citations stay valid. Changing its output requires bumping `TEXT_PIPELINE_VERSION`.
|
|
28
|
+
* **Chunking.** `parser.ts` uses web-tree-sitter for 20 languages. Definitions include leading doc comments and decorators and carry `signature`, `scope` and `doc`. Long definitions are split into overlapping pieces; every file also gets 64-line windows so nothing is unsearchable. Generated or minified files get bounded chunking. `chunkers.ts` supplies structural spans (headings, keys, elements, rules, statements, stages, targets) for Markdown, JSON, YAML, TOML, XML (including Maven and AEM `.content.xml`), HTML, CSS, SQL, Dockerfile, Makefile, shell, protobuf and HCL.
|
|
29
|
+
* **Persistence.** `store.ts` keeps the cache outside the repository (`REPOSQUEEZE_CACHE_DIR`, XDG or platform cache directory, keyed by the repository's real path). Files are checksummed containers written atomically with private permissions; every loaded field is validated, and any cache failure degrades to in-memory operation. Entries are keyed by path, content hash, parser, tokenizer and text-pipeline versions. For repositories up to 5,000 files the content hash is re-verified on every run; above that, size and modification time are trusted (`REPOSQUEEZE_REFRESH_INDEX=1` overrides).
|
|
30
|
+
* **Large repositories.** Above 5,000 eligible files a per-file term sketch selects 600–1,800 candidate files that are then parsed; the rest of the pipeline is unchanged.
|
|
31
|
+
* **Search index.** `search-index.ts` holds per-chunk term rows and postings in typed arrays. `retrieve()` computes BM25 (k1 = 1.2, b = 0.75) plus bonuses for terms in the symbol name and path, and accepts weighted terms.
|
|
32
|
+
|
|
33
|
+
## Ranking (`v2.ts`)
|
|
34
|
+
|
|
35
|
+
1. **Views.** The query is tokenized (`tokenize.ts`: camelCase and acronym splitting, snake_case, joined identifiers, Porter stemming, hash blobs dropped). Up to six deterministic views are built: the original, diagnostic lines, identifier-like words and three leave-one-bucket-out views.
|
|
36
|
+
2. **Fusion.** Reciprocal-rank fusion over chunk and file ranks (perturbation-consensus retrieval). Stability, the share of views in which a chunk stays near the top, is a diagnostic, not a confidence.
|
|
37
|
+
3. **Signals (default on).** Pseudo-relevance feedback terms from the first-pass top chunks; definitions whose names are literally mentioned; adjacent query-word pairs; priors that down-weight tests, docs and vendored or generated paths unless the query asks for them; trace mode that prefers implementation files over the tests named in a failure.
|
|
38
|
+
4. **Signals (opt-in).** `frames`: files named by application frames of a parsed stack trace (`trace.ts`). `git`: files with uncommitted work, recent commits and co-change partners (`gitsignals.ts`).
|
|
39
|
+
5. **Obligations.** Rare query terms are listed and reported when no returned excerpt contains them.
|
|
40
|
+
|
|
41
|
+
## Packing (`pack.ts`)
|
|
42
|
+
|
|
43
|
+
Greedy selection by marginal value: a candidate's score is scaled by the share of its matched terms (IDF-weighted) not yet on the page and by a per-file crowding factor. Rendering order for a candidate: whole chunk if it fits and is moderate, otherwise the highest-scoring contiguous window (48 → 6 lines) with the enclosing declaration kept as a separate range and an explicit `…[a-b omitted]` marker. Parts of a chunk that overlap already shown lines are never re-printed. Identical chunk texts in several files are printed once and the copies named. About 12% of the budget (at most 400 tokens) is held back for "Also relevant (not shown)" pointers, and unused reservation returns to excerpts. The final text is checked against the budget with the same tokenizer that reports `outputTokens`.
|
|
44
|
+
|
|
45
|
+
Receipts are 8-hex digests of the displayed range; complete excerpts reuse their chunk id so a client can pass `known` ids to avoid re-sending text it still holds.
|
|
46
|
+
|
|
47
|
+
## Other components
|
|
48
|
+
|
|
49
|
+
* `repomap.ts`: file-level PageRank over the definition/reference graph, personalised by optional focus terms, rendered as signatures with directory-prefix compression and fitted to the budget; `skeleton.ts` renders a range as signatures plus query lines; `symbols.ts` provides name search and per-file outlines.
|
|
50
|
+
* `compact.ts` + `logs/`: log compaction with volatile-token normalisation, template mining, stack-trace folding, test-runner awareness and explicit omission markers. `structured.ts`: verified lossless JSON/TOON choice.
|
|
51
|
+
* `semantic.ts`: optional Ollama embeddings of the 300 strongest lexical candidates, cached outside the repository.
|
|
52
|
+
* `mcp.ts`, `cli.ts`, `extension.ts`: three thin front ends over the same `Engine`.
|
|
53
|
+
|
|
54
|
+
## Invariants (tested)
|
|
55
|
+
|
|
56
|
+
Output never exceeds the budget; excerpts are exact source lines of the indexed (redacted) text; identical inputs give identical bytes cold, warm and after restart; no file inside the repository is written; paths outside the root, symlinks and secret paths are refused; every elision or redaction is marked.
|
|
57
|
+
|
|
58
|
+
Rationale and the techniques that were considered or rejected, with sources: [V3_RESEARCH.md](V3_RESEARCH.md). Older 2.1 design notes: [V2_RESEARCH.md](V2_RESEARCH.md).
|
package/docs/GRAPH.md
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# Code graph
|
|
2
|
+
|
|
3
|
+
`squeze graph` builds a deterministic graph of a repository and answers structural questions in a few hundred tokens.
|
|
4
|
+
|
|
5
|
+
* **Nodes:** files and definitions. **Edges:** `imports` (resolved to repository files; marked `EXTRACTED`), `defines`, and name-based `references` (marked `INFERRED`, with generic and over-defined names down-weighted). External packages are leaf nodes and are never invented as files.
|
|
6
|
+
* **Import resolution:** JS/TS (relative paths, index files, `tsconfig` paths), Python (absolute and relative, source roots), Go (module path), Java/Kotlin/Scala (package to file), Rust (`mod`, `use crate::`), C/C++ (`#include "…"`), C#, Ruby (`require_relative`), PHP.
|
|
7
|
+
* **Analytics (all deterministic):** communities (named from their dominant directory and identifiers), hub nodes, cross-community links, shortest path, impact set, cycles.
|
|
8
|
+
* **Honesty markers:** every output states how many edges are extracted versus inferred, and says when the index is partial (large repositories parsed in cascade mode).
|
|
9
|
+
|
|
10
|
+
```sh
|
|
11
|
+
squeze graph --root REPO # subsystems, hubs, flows, cycles (default budget 1500 tokens)
|
|
12
|
+
squeze graph --root REPO --neighbors X # imports, importers, callers, callees of a file or symbol
|
|
13
|
+
squeze graph --root REPO --impact X # what depends on X
|
|
14
|
+
squeze graph --root REPO --path A B # shortest dependency chain
|
|
15
|
+
squeze graph --root REPO --mermaid [X] # Mermaid flowchart
|
|
16
|
+
squeze graph --root REPO --json # versioned JSON export
|
|
17
|
+
squeze graph --root REPO --html graph.html # self-contained HTML (no remote assets, strict CSP)
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
The same modes are available as the `squeze_graph` MCP tool (default tool set) and a VS Code language-model tool, plus the command **Squeze: Show Code Graph**, which opens the HTML view in a webview.
|
|
21
|
+
|
|
22
|
+
Real output on gin (`--budget 500`):
|
|
23
|
+
|
|
24
|
+
```text
|
|
25
|
+
code graph @e69736f · 99 files · 1123 defs · 80 imports (EXTRACTED) · 1428 refs (INFERRED, name-based) · 11 subsystems
|
|
26
|
+
S0 render · write, content, response (20f, 218d) | key: response_writer.go
|
|
27
|
+
S1 binding · context, validate, run (18f, 198d) | key: binding.go
|
|
28
|
+
...
|
|
29
|
+
god nodes: debug.go (in 6, out 1), utils.go (in 6, out 1), binding/binding.go (in 19, out 1)
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
```text
|
|
33
|
+
impact of binding/binding.go: 7 files via imports within 3 hops (direct 6), +17 only via name-based references
|
|
34
|
+
d1 (6): context.go, mode.go, context_test.go, deprecated.go, deprecated_test.go, mode_test.go
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Measured accuracy, token cost and build time: [benchmarks/v22/graph.md](../benchmarks/v22/graph.md). Limits: name-based references over-report importers in some repositories (precision 59% on average), Rust `use` of items is partly unresolved, and cascade-mode graphs on repositories above 5,000 files cover only the parsed candidates.
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# 50,000-file scale evaluation
|
|
2
|
+
|
|
3
|
+
RepoSqueeze V2.1 was tested on a deterministic synthetic repository containing 50,000 TypeScript source files, at least 1.2 million lines and 100 query/gold-file pairs. Each query has 20 files containing the same incident term but only one root-cause file containing the full identifier evidence.
|
|
4
|
+
|
|
5
|
+
The first 80 tasks were used for development. Retrieval, packing and the evaluator were then frozen at SHA-256 `2ad34e1103ac9267471a9eaf376e643e86a520b177e9e5aa675077c73a3ef0a4`. The remaining 20 tasks ran once against that fingerprint.
|
|
6
|
+
|
|
7
|
+
## Results
|
|
8
|
+
|
|
9
|
+
| Metric | Development 80 | Held-out 20 |
|
|
10
|
+
| --- | ---: | ---: |
|
|
11
|
+
| Root-cause file Any@10 | 100% | **100%** |
|
|
12
|
+
| Root-cause file Any@20 | 100% | **100%** |
|
|
13
|
+
| Root-cause file included in 3K pack | 100% | **100%** |
|
|
14
|
+
| Mean parsed candidates | 200 | 200 |
|
|
15
|
+
| Mean output tokens | 2,959 | 2,967 |
|
|
16
|
+
| Median query time | 0.75 s | 1.31 s |
|
|
17
|
+
| 95th-percentile query time | 1.14 s | 4.51 s |
|
|
18
|
+
| Peak resident memory | 541 MiB | 475 MiB |
|
|
19
|
+
|
|
20
|
+
The initial development query built the persistent 4.0 MB compressed index in 10.50 seconds. Subsequent queries reused file sketches. Timings are from one Apple Silicon workstation and should not be treated as cross-machine guarantees.
|
|
21
|
+
|
|
22
|
+
The repository contained 50,001 eligible files during evaluation because its `.reposqueeze-v2ignore` control file was also indexed. Only 200 exact source files were parsed per query; ranking and packing still saw the complete file-level inventory.
|
|
23
|
+
|
|
24
|
+
## Reproduce
|
|
25
|
+
|
|
26
|
+
```sh
|
|
27
|
+
npm run bench:scale:development
|
|
28
|
+
npm run bench:scale:test
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The first command generates `/private/tmp/reposqueeze-scale-50000`, initializes its Git index, runs 80 tasks and writes the freeze. The second command refuses to run if scale retrieval, PCR, packing or the evaluator has changed.
|
|
32
|
+
|
|
33
|
+
Raw evidence:
|
|
34
|
+
|
|
35
|
+
- [Development report](../benchmarks/scale-development.json)
|
|
36
|
+
- [Held-out report](../benchmarks/scale-test.json)
|
|
37
|
+
- [Freeze](../benchmarks/scale-freeze.json)
|
|
38
|
+
- [Evaluator and generator](../scripts/benchmark-scale.ts)
|
|
39
|
+
|
|
40
|
+
## Limits
|
|
41
|
+
|
|
42
|
+
This benchmark proves operation on its generated distribution, not perfect retrieval for every 50,000-file repository. Its identifiers are deterministic and the gold files contain exact term evidence. Real monorepos have generated code, duplicated APIs, cross-language names, build indirection and tasks that do not name their relevant concepts.
|
|
43
|
+
|
|
44
|
+
The candidate stage can miss a file with no shared path or indexed term. Files with more than 8,192 unique normalized terms use deterministic term sampling. Increase `REPOSQUEEZE_MAX_CANDIDATES`, provide paths or identifiers, or use ordinary search when obligations are missing. Set `REPOSQUEEZE_REFRESH_INDEX=1` after timestamp-preserving edits.
|
|
45
|
+
|
|
46
|
+
The 100% held-out result is scale-corpus retrieval accuracy. It is not 100% answer accuracy, patch success or Copilot-credit reduction.
|
|
@@ -0,0 +1,106 @@
|
|
|
1
|
+
# RepoSqueeze V2 evaluation
|
|
2
|
+
|
|
3
|
+
The primary result is a frozen, paired comparison of V1 retrieval and V2 perturbation-consensus retrieval on the official Agent Retrieval Bench `v2_trace2code` release. Both variants see the same prebuilt source chunks and receive the same ranking limits and 4,000-token packing budget. No embedding model is used.
|
|
4
|
+
|
|
5
|
+
The result supports a narrower claim than “100% accurate”: on this held-out retrieval set, V2 finds more labeled root-cause files, ranks them earlier, includes more labeled lines and surfaces fewer known distractors. It does not test whether a coding model writes the correct answer or patch, and it does not measure billed Copilot credits.
|
|
6
|
+
|
|
7
|
+
## Frozen protocol
|
|
8
|
+
|
|
9
|
+
- Dataset: [Agent Retrieval Bench](https://github.com/eyuansu62/agent-retrieval-bench) `v2_trace2code`, 101 samples, seven repositories and 356,074 prebuilt chunks.
|
|
10
|
+
- Archive SHA-256: `19b252e8cfff42107fedc74005dbb6972f2970af33651ce0c1571546819e41c4`.
|
|
11
|
+
- Split: sort opaque sample IDs, use the first 80 for development and the remaining 21 once for held-out testing. This is a sample-level split; repositories occur on both sides.
|
|
12
|
+
- Freeze fingerprint: `a06c988f7aa6187ceeb6b626f3463749d8e7998ea7fadd79fb0912689b9ac1e2` over PCR, V1 ranking, parsing, the evaluator and archive checksum.
|
|
13
|
+
- Query forms: original, deterministic 20% word dropout, command-field removal and line reordering.
|
|
14
|
+
- Statistics: the original query is the primary unit. Confidence intervals use 20,000 deterministic paired sample-level bootstrap resamples. Binary outcomes also use an exact two-sided McNemar test.
|
|
15
|
+
- Disclosure: the first two dataset rows were inspected while adapting the schema. Their metrics were not viewed before the freeze. The held-out command refuses to run if fingerprinted code changes.
|
|
16
|
+
|
|
17
|
+
## Held-out original queries
|
|
18
|
+
|
|
19
|
+
| Metric | V1 | V2 PCR | Paired change | Bootstrap 95% interval for change |
|
|
20
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
21
|
+
| MRR | 0.293 | **0.467** | **+0.175** | +0.099 to +0.256 |
|
|
22
|
+
| Any labeled root-cause file @5 | 47.62% | **61.90%** | **+14.29 pp** | 0.00 to +28.57 pp |
|
|
23
|
+
| Any labeled root-cause file @10 | 52.38% | **80.95%** | **+28.57 pp** | +9.52 to +47.62 pp |
|
|
24
|
+
| Any labeled root-cause file @20 | 80.95% | **90.48%** | **+9.52 pp** | 0.00 to +23.81 pp |
|
|
25
|
+
| Root-cause file recall @20 | 78.57% | **88.10%** | **+9.52 pp** | 0.00 to +23.81 pp |
|
|
26
|
+
| Known hard negative in top 5 ↓ | 66.67% | **38.10%** | **−28.57 pp** | −47.62 to −9.52 pp |
|
|
27
|
+
| Gold-line recall in 4K pack | 16.06% | **30.99%** | **+14.93 pp** | +5.05 to +27.14 pp |
|
|
28
|
+
| Gold-line precision in 4K pack | 0.93% | **1.84%** | **+0.91 pp** | +0.31 to +1.64 pp |
|
|
29
|
+
| Mean packed tokens | 3,990.0 | 3,993.6 | +3.7 | −4.6 to +18.1 |
|
|
30
|
+
|
|
31
|
+
For Any@10, six samples changed from a V1 miss to a V2 hit and none changed in the opposite direction; exact two-sided McNemar `p = 0.03125`. For hard negatives, V2 removed six V1 top-five distractor hits and introduced none; `p = 0.03125`. The other intervals are estimates from a small set and should not be presented as universal effects.
|
|
32
|
+
|
|
33
|
+
V2 used essentially the same packing budget. The improvement came from ordering and selection rather than sending a larger context.
|
|
34
|
+
|
|
35
|
+
## Query perturbations
|
|
36
|
+
|
|
37
|
+
The four forms produce 84 paired retrieval runs from 21 underlying samples. They show that the result survives the tested wording changes, but those 84 rows are correlated and are not 84 independent test samples.
|
|
38
|
+
|
|
39
|
+
| Aggregate across four forms | V1 | V2 PCR |
|
|
40
|
+
| --- | ---: | ---: |
|
|
41
|
+
| MRR | 0.291 | **0.467** |
|
|
42
|
+
| Any@10 | 52.38% | **80.95%** |
|
|
43
|
+
| Any@20 | 82.14% | **89.29%** |
|
|
44
|
+
| Recall@20 | 79.37% | **86.90%** |
|
|
45
|
+
| Hard negative in top 5 ↓ | 66.67% | **40.48%** |
|
|
46
|
+
| Gold-line recall in 4K pack | 16.19% | **30.28%** |
|
|
47
|
+
| Gold-line precision in 4K pack | 0.94% | **1.75%** |
|
|
48
|
+
|
|
49
|
+
The dropout form was the hardest V2 condition: Any@10 was 76.19%, compared with 80.95% on original wording. This is evidence of improved robustness, not invariance to arbitrary paraphrases.
|
|
50
|
+
|
|
51
|
+
## Development evidence
|
|
52
|
+
|
|
53
|
+
Before freezing, the 80-sample development partition showed the same direction. On original queries, Any@10 rose from 68.75% to 83.75%, hard-negative hit@5 fell from 57.50% to 33.75%, and line recall at 4K rose from 28.30% to 44.52%.
|
|
54
|
+
|
|
55
|
+
A separate 40-question stress suite across eight checked-out repositories tested original, dropout, telegraphic and reordered wording. At a 3,000-token budget the aggregate complete-evidence rate increased from 90.00% for V1 to 91.88% for V2. This suite influenced development and is not independent confirmation.
|
|
56
|
+
|
|
57
|
+
## Remaining failures
|
|
58
|
+
|
|
59
|
+
V2 still missed every labeled root-cause file in the top ten for four of 21 original held-out samples:
|
|
60
|
+
|
|
61
|
+
| Sample | Repository | V2 reciprocal rank | Hard negative in top 5 |
|
|
62
|
+
| --- | --- | ---: | --- |
|
|
63
|
+
| `de059c1c2e3ffc63820ce0e4` | `caddyserver/caddy` | 0.0135 | yes |
|
|
64
|
+
| `d0551ed5a2d63c99e3bbfcd5` | `gin-gonic/gin` | 0.0909 | no |
|
|
65
|
+
| `d3fe4d1559febe2ac282cced` | `gin-gonic/gin` | 0.0455 | yes |
|
|
66
|
+
| `f4fccd0dc58a48c09a195d6b` | `pallets/click` | 0.0556 | yes |
|
|
67
|
+
|
|
68
|
+
Eight of 21 samples still had a labeled hard negative in V2's top five. Absolute line precision remained low at 1.84%, meaning most packed lines were not inside the benchmark's narrow gold spans. The pack can include useful setup or dependency code outside those spans, but this evaluator does not prove that it is useful. These failures are why the tool reports missing obligations and tells callers to use expansion or ordinary search.
|
|
69
|
+
|
|
70
|
+
There were no original-query MRR regressions and no V1-only Any@10 successes in this particular held-out set. With 21 samples, that observation is not a guarantee for other repositories.
|
|
71
|
+
|
|
72
|
+
## Reproduce and audit
|
|
73
|
+
|
|
74
|
+
Install the upstream `arb` tool and download its self-contained subset:
|
|
75
|
+
|
|
76
|
+
```sh
|
|
77
|
+
arb download-benchmark --version v2_trace2code --local-dir data --force
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Point `ARB_TRACE_ROOT` at the extracted release directory containing `benchmark/`, `corpus/` and `trace.sha256`, then run:
|
|
81
|
+
|
|
82
|
+
```sh
|
|
83
|
+
npm ci
|
|
84
|
+
ARB_TRACE_ROOT=/path/to/arb-trace npm run bench:v2:development
|
|
85
|
+
ARB_TRACE_ROOT=/path/to/arb-trace npm run bench:v2:test
|
|
86
|
+
npm run bench:v2:summary
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
This blind report belongs to V2.0 commit `190ef6d934af2cc21e3f70caf1431fb1e9e079e4`. V2.1 keeps the PCR ranking algorithm but changes repository enumeration and context reporting, so its whole-file fingerprint intentionally differs. Check out that commit to reproduce the exact frozen ARB run. The V2.1 scale experiment below has its own independent freeze and held-out split; the already-seen ARB test set was not reused to tune or re-certify V2.1.
|
|
90
|
+
|
|
91
|
+
The checked-in evidence is:
|
|
92
|
+
|
|
93
|
+
- [Development raw report](../benchmarks/v2-arb-development.json)
|
|
94
|
+
- [Held-out raw report](../benchmarks/v2-arb-test.json)
|
|
95
|
+
- [Frozen fingerprint](../benchmarks/v2-freeze.json)
|
|
96
|
+
- [Deterministic statistics and failure list](../benchmarks/v2-summary.json)
|
|
97
|
+
- [40-question development stress report](../benchmarks/v2-development.json)
|
|
98
|
+
- [Evaluator](../scripts/benchmark-v2-arb.ts) and [summary script](../scripts/summarize-v2.ts)
|
|
99
|
+
|
|
100
|
+
The evaluator reads the official prebuilt chunks and does not execute code from benchmark repositories.
|
|
101
|
+
|
|
102
|
+
Repository-scale behavior is evaluated separately because the ARB release supplies prebuilt chunks and therefore does not exercise file enumeration or parsing. See the [50,000-file scale evaluation](SCALE_EVALUATION.md), including its independent 80:20 freeze, cold-index time, warm-query latency, memory use and synthetic-data limitations.
|
|
103
|
+
|
|
104
|
+
## What this does and does not prove
|
|
105
|
+
|
|
106
|
+
The paired held-out result is evidence that PCR improved retrieval on this ARB subset under an equal evidence budget. It is not 100% accuracy, a proof of worldwide algorithmic novelty, an answer-quality evaluation, an executable patch benchmark or a measurement of provider billing. Real savings depend on whether the client actually calls the tool, how often it expands, hidden prompts, cache behavior, reasoning tokens and the final answer length.
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# V2 research and gap analysis
|
|
2
|
+
|
|
3
|
+
Reviewed 31 October 2026. Primary papers, official benchmark artifacts and source repositories were used where available.
|
|
4
|
+
|
|
5
|
+
## What changed the design
|
|
6
|
+
|
|
7
|
+
The first RepoSqueeze combined lexical ranking, symbol/path boosts, lightweight identifier relations, optional embeddings, token-budget packing and exact source receipts. Its held-out set was small and showed 8/10 complete evidence for structural retrieval and 9/10 with local embeddings. The identifier-relation ablation did not improve the result. V2 does not treat that as solved.
|
|
8
|
+
|
|
9
|
+
Recent work made several obvious V2 ideas unsuitable as novelty claims:
|
|
10
|
+
|
|
11
|
+
- [Rivet](https://arxiv.org/abs/2610.34967) already grounds task anchors, follows dependencies/usages/configuration relations and shapes representations under a fixed budget. It reports executable Pass@1 improvements. It also explicitly leaves controlled robustness to wrong or missing graph edges for future study.
|
|
12
|
+
- [Agent Retrieval Bench](https://arxiv.org/abs/2607.24882) evaluates 427 workflow-derived samples, including trace-to-code and no-gold cases. It finds that retriever families have different strengths, simple rank fusion helps, raw top scores fail as natural abstention signals, and query rewriting/phrasing robustness remains future work.
|
|
13
|
+
- [ContextBench](https://arxiv.org/abs/2602.05892) supplies human-verified contexts for 1,136 tasks and shows the recall/precision cost of broad exploration. Complex agent scaffolds do not automatically improve context retrieval.
|
|
14
|
+
- [CORE-Bench](https://arxiv.org/abs/2606.11864) emphasizes repository-level requirements, similar in-repository distractors and edit localization beyond isolated semantic matching.
|
|
15
|
+
- [CodeRAG](https://arxiv.org/abs/2509.16112) uses multiple retrieval paths and learned preference reranking. V2's multiple query views are deterministic and do not reproduce its learned method.
|
|
16
|
+
- [ACON](https://github.com/microsoft/acon) optimizes compression policies for long agent trajectories. V2 focuses on initial repository evidence and explicit recovery.
|
|
17
|
+
|
|
18
|
+
## Targeted gap
|
|
19
|
+
|
|
20
|
+
Coding requests are unstable inputs. A stack trace mixes source signals with paths, test names, numbers, dependency downloads and repeated errors. Users paraphrase the same task, omit a key noun or paste only the final line. A single ranking can change sharply even when the repository evidence should not.
|
|
21
|
+
|
|
22
|
+
Agent Retrieval Bench explicitly proposes paired query rewrites as future work. Rivet reports no controlled study of graph-edge noise. ContextBench shows that adding more context can lower precision. These observations motivate a narrow question:
|
|
23
|
+
|
|
24
|
+
> Can a deterministic retriever improve worst-case evidence retrieval under small query perturbations, at the same context budget, without another hosted model?
|
|
25
|
+
|
|
26
|
+
## Perturbation-consensus retrieval
|
|
27
|
+
|
|
28
|
+
PCR creates up to six views:
|
|
29
|
+
|
|
30
|
+
1. the original request;
|
|
31
|
+
2. diagnostic lines containing failure/error/assertion signals;
|
|
32
|
+
3. identifier-heavy text;
|
|
33
|
+
4. three deterministic leave-one-term-bucket-out views.
|
|
34
|
+
|
|
35
|
+
Each view uses the same local lexical and structural ranker. Reciprocal-rank evidence is accumulated at chunk and file granularity. The original request has a modest extra weight. Stability is the fraction of views where a chunk remains near the top.
|
|
36
|
+
|
|
37
|
+
The packer also receives a small list of rare query terms found in the repository. These are **evidence obligations**, not proof requirements. They encourage coverage and are reported as covered or missing. In failure traces, visible test files are navigation anchors; same-stem source files get a bounded boost and test files get a penalty unless the request asks for tests.
|
|
38
|
+
|
|
39
|
+
This technique is deterministic, model-free and inspectable. Its main cost is running the inexpensive ranker several times. It does not compress source text semantically; selected code remains verbatim.
|
|
40
|
+
|
|
41
|
+
## Novelty wording
|
|
42
|
+
|
|
43
|
+
The repository calls the implemented technique perturbation-consensus retrieval because that precisely describes it. We found query rewriting, rank fusion, dropout, robustness testing and workflow-aware code retrieval individually in prior work. We did not find this exact deterministic combination with evidence obligations and trace-aware source inversion in the reviewed sources.
|
|
44
|
+
|
|
45
|
+
That search cannot establish worldwide novelty or patent priority. The defensible contribution is the implementation, frozen protocol, perturbation measurements, independent held-out evaluation and published negative results.
|
|
46
|
+
|
|
47
|
+
## Scaling research
|
|
48
|
+
|
|
49
|
+
Large code-search systems avoid parsing an entire corpus for every query. [Zoekt's design](https://github.com/sourcegraph/zoekt/blob/main/doc/design.md) uses a persistent positional trigram index and exact verification, targeting repositories as large as Android or Chromium. [GitHub Code Search](https://github.blog/engineering/architecture-optimization/the-technology-behind-githubs-new-code-search/) likewise explains why an n-gram inverted index is needed instead of scanning all content for every query. [ripgrep's official guide](https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md) demonstrates the value of early ignore and binary filtering for repository traversal.
|
|
50
|
+
|
|
51
|
+
RepoSqueeze V2.1 applies the same broad cascade principle to a different output: a small evidence pack. Its first stage stores compact per-file term hashes and metadata; its second stage re-reads, verifies and structurally parses a bounded candidate set. This is an engineering adaptation, not a claim to have invented indexed code search.
|
|
52
|
+
|
|
53
|
+
## Evaluation rules
|
|
54
|
+
|
|
55
|
+
- V1 and V2 receive the same repository snapshot and ranking/packing budget.
|
|
56
|
+
- Development results may be viewed before freezing. Test execution checks a SHA-256 fingerprint of V2, V1 ranking, parsing and the evaluator.
|
|
57
|
+
- The official ARB archive checksum is recorded. No upstream repository code is executed.
|
|
58
|
+
- Metrics include MRR, Any@5/10/20, Recall@20, hard-negative hits, and line recall/precision under a 4K packed-context budget.
|
|
59
|
+
- Original and perturbed query results are reported separately. Aggregate results cannot hide a regression on original wording.
|
|
60
|
+
- The first two ARB rows were inspected while adapting the file format. Their metrics were not viewed before the freeze; this limitation is disclosed rather than describing the whole release as blind.
|
|
61
|
+
- Retrieval metrics are not answer accuracy or patch success. No 100% claim is made.
|