project-librarian 0.5.7 → 0.5.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ko.md +24 -7
- package/README.md +24 -7
- package/dist/code-index/modes.js +84 -88
- package/dist/code-index/ownership.js +119 -8
- package/dist/code-index/schema.js +2 -2
- package/dist/code-index/search.js +1 -5
- package/dist/code-index.js +31 -13
- package/dist/native/darwin-arm64/project-librarian-indexer +0 -0
- package/dist/native/darwin-x64/project-librarian-indexer +0 -0
- package/dist/native/linux-arm64/project-librarian-indexer +0 -0
- package/dist/native/linux-arm64-musl/project-librarian-indexer +0 -0
- package/dist/native/linux-x64/project-librarian-indexer +0 -0
- package/dist/native/linux-x64-musl/project-librarian-indexer +0 -0
- package/dist/native/project-librarian-indexer-manifest.json +11 -11
- package/dist/native/win32-arm64/project-librarian-indexer.exe +0 -0
- package/dist/native/win32-x64/project-librarian-indexer.exe +0 -0
- package/dist/templates.js +3 -3
- package/docs/benchmarks.md +31 -11
- package/docs/ko/benchmarks.md +31 -11
- package/package.json +1 -1
package/README.ko.md
CHANGED
|
@@ -104,15 +104,32 @@ Project Librarian은 작은 첫 읽기와 더 깊은 정본으로 가는 경로
|
|
|
104
104
|
|
|
105
105
|
여기 수치는 모든 상황에 대한 약속이 아니라 관리자 릴리스 근거입니다. 모든 값은 실제 Codex JSONL 사용량과 로컬 실행 시간이며, Project Librarian이 없는 `organic` 대조군과 격리된 환경에서 비교했습니다. 위키 라우팅 트랙과 코드 그래프 트랙은 분리해 측정하며, 한 트랙의 우위가 다른 트랙의 주장을 뒷받침하지 않습니다.
|
|
106
106
|
|
|
107
|
-
위키
|
|
107
|
+
최신 clean 합성 위키 라우팅 트랙 릴리스 근거는 2026-06-29에 `gpt-5.5`, `perf/small-repo-code-evidence-safeguards` 브랜치의 `ae79390` 커밋에서 측정했습니다. 42개 시나리오, 21개 with/without 쌍을 각 3회 측정하고 1회 예열했으며, 전체 주장 게이트는 통과했고 주장 ledger는 이 보고서를 release-claimable로 분류했습니다. repair 실행은 보존된 raw JSONL에서 claimable 측정 실행 125개를 재사용하고 남은 실패 슬롯 1개를 다시 측정했습니다. 이 주장은 합성 위키 라우팅 트랙과 표기된 작업 유형에 한정되며, 코드 그래프 동작, 실제 저장소, 모든 에이전트 표면, 모든 질문 형태에 대한 주장이 아닙니다.
|
|
108
108
|
|
|
109
|
-
|
|
110
|
-
| --- | --- | --- | --- |
|
|
111
|
-
| 소형 | 14.4% 적음 | 81.0% 많음 | 22.0% 적음 |
|
|
112
|
-
| 중형 | 52.0% 적음 | 19.0% 적음 | 54.1% 적음 |
|
|
113
|
-
| 대형 | 71.1% 적음 | 29.0% 적음 | 71.8% 적음 |
|
|
109
|
+
위키 트랙 집계, 대조군 대비 변화:
|
|
114
110
|
|
|
115
|
-
|
|
111
|
+
| 규모 | 비용 가중 토큰 | 전체 토큰 | 실행 시간 | 명령 수 | 도구 출력 |
|
|
112
|
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
113
|
+
| 전체 | 51.39% 적음 | 48.67% 적음 | 19.83% 적음 | 18.40% 적음 | 85.88% 적음 |
|
|
114
|
+
| 소형 | 21.59% 적음 | 9.31% 적음 | 13.91% 적음 | 12.64% 적음 | 43.89% 적음 |
|
|
115
|
+
| 중형 | 45.95% 적음 | 37.42% 적음 | 11.88% 적음 | 4.30% 적음 | 69.12% 적음 |
|
|
116
|
+
| 대형 | 66.97% 적음 | 69.87% 적음 | 31.90% 적음 | 35.19% 적음 | 95.58% 적음 |
|
|
117
|
+
|
|
118
|
+
위키 트랙 작업 유형별 비용 가중 토큰 변화, 전체 규모 합산:
|
|
119
|
+
|
|
120
|
+
| 작업 유형 | 변화 |
|
|
121
|
+
| --- | ---: |
|
|
122
|
+
| onboarding | 65.95% 적음 |
|
|
123
|
+
| decision_lookup | 48.46% 적음 |
|
|
124
|
+
| code_impact | 55.61% 적음 |
|
|
125
|
+
| release_policy | 58.22% 적음 |
|
|
126
|
+
| change_location | 29.59% 적음 |
|
|
127
|
+
| multi_session | 52.58% 적음 |
|
|
128
|
+
| aggregation | 42.53% 적음 |
|
|
129
|
+
|
|
130
|
+
규모별 값은 서로 다릅니다. 소형/중형/대형 작업 유형별 실제 행렬과 아직 역전이 남은 셀은 [벤치마크 근거](docs/ko/benchmarks.md#위키-트랙)에 있습니다.
|
|
131
|
+
|
|
132
|
+
실행 시간과 명령 수의 한계도 있습니다. 이번 보고서에서 `code_impact`, `change_location`, `aggregation`은 토큰과 출력 바이트가 모두 개선됐지만, 실행 시간이나 명령 수에서는 일부 역전이 남아 있습니다.
|
|
116
133
|
|
|
117
134
|
코드 그래프 트랙, 대조군 대비 비용 가중 토큰:
|
|
118
135
|
|
package/README.md
CHANGED
|
@@ -104,15 +104,32 @@ See [Usage](docs/usage.md) for install scopes, runner paths, generated files, mi
|
|
|
104
104
|
|
|
105
105
|
These numbers are maintainer release evidence, not a blanket promise. Every value is real Codex JSONL usage and local wall-clock time, measured hermetically against an `organic` control with no Project Librarian. The wiki-routing track and the code-graph track are measured separately; a win on one never backs a claim about the other.
|
|
106
106
|
|
|
107
|
-
|
|
107
|
+
Latest clean synthetic wiki-routing track release evidence: 2026-06-29, `gpt-5.5`, branch `perf/small-repo-code-evidence-safeguards` at `ae79390`, 42 scenarios, 21 with/without pairs, 3 measured runs plus 1 warmup each. The overall claim gate **passed**, and the claim ledger classified the report as release-claimable. A repair run reused 125 claimable measured runs from retained raw JSONL and remeasured the remaining failed slot. This is bounded to the synthetic wiki-routing track and listed task families; it is not a claim about code-graph behavior, real repositories, every agent surface, or every question shape.
|
|
108
108
|
|
|
109
|
-
|
|
110
|
-
| --- | --- | --- | --- |
|
|
111
|
-
| Small | 14.4% less | 81.0% more | 22.0% less |
|
|
112
|
-
| Medium | 52.0% less | 19.0% less | 54.1% less |
|
|
113
|
-
| Large | 71.1% less | 29.0% less | 71.8% less |
|
|
109
|
+
Wiki track aggregate deltas vs control:
|
|
114
110
|
|
|
115
|
-
|
|
111
|
+
| Scale | Cost-weighted tokens | Total tokens | Wall time | Commands | Tool output |
|
|
112
|
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
113
|
+
| All | 51.39% less | 48.67% less | 19.83% less | 18.40% less | 85.88% less |
|
|
114
|
+
| Small | 21.59% less | 9.31% less | 13.91% less | 12.64% less | 43.89% less |
|
|
115
|
+
| Medium | 45.95% less | 37.42% less | 11.88% less | 4.30% less | 69.12% less |
|
|
116
|
+
| Large | 66.97% less | 69.87% less | 31.90% less | 35.19% less | 95.58% less |
|
|
117
|
+
|
|
118
|
+
Wiki track task-family cost-weighted token deltas vs control, all scales combined:
|
|
119
|
+
|
|
120
|
+
| Task family | Delta |
|
|
121
|
+
| --- | ---: |
|
|
122
|
+
| onboarding | 65.95% less |
|
|
123
|
+
| decision_lookup | 48.46% less |
|
|
124
|
+
| code_impact | 55.61% less |
|
|
125
|
+
| release_policy | 58.22% less |
|
|
126
|
+
| change_location | 29.59% less |
|
|
127
|
+
| multi_session | 52.58% less |
|
|
128
|
+
| aggregation | 42.53% less |
|
|
129
|
+
|
|
130
|
+
Scale-specific values vary; see [Benchmark Evidence](docs/benchmarks.md#wiki-track) for the small/medium/large task-family matrix and the cells that still regress.
|
|
131
|
+
|
|
132
|
+
Timing and command-count caveat: `code_impact`, `change_location`, and `aggregation` still had wall-time or command-count regressions in this report, even though token and output-byte metrics improved for every task family.
|
|
116
133
|
|
|
117
134
|
Code-graph track, cost-weighted tokens vs control:
|
|
118
135
|
|
package/dist/code-index/modes.js
CHANGED
|
@@ -131,6 +131,36 @@ function emitCodeIndexPhaseTimings(timings) {
|
|
|
131
131
|
return;
|
|
132
132
|
console.error(`code_index_phase_timings ${JSON.stringify(timings)}`);
|
|
133
133
|
}
|
|
134
|
+
function finishCodeEvidencePhaseTimings(started, timings) {
|
|
135
|
+
timings.total_ms = Number(elapsedMs(started).toFixed(3));
|
|
136
|
+
emitCodeIndexPhaseTimings(timings);
|
|
137
|
+
}
|
|
138
|
+
function runWithCodeEvidenceDatabase(runtime, operation, options = {}) {
|
|
139
|
+
const totalStarted = process.hrtime.bigint();
|
|
140
|
+
const phaseTimings = {};
|
|
141
|
+
let database;
|
|
142
|
+
try {
|
|
143
|
+
measurePhase(phaseTimings, "require_existing_index_ms", () => runtime.requireExistingIndex());
|
|
144
|
+
options.beforeOpen?.();
|
|
145
|
+
database = measurePhase(phaseTimings, "open_database_ms", () => runtime.openDatabase(runtime.codeEvidenceDatabasePath().absolutePath));
|
|
146
|
+
operation(database, phaseTimings);
|
|
147
|
+
}
|
|
148
|
+
finally {
|
|
149
|
+
if (database) {
|
|
150
|
+
const openedDatabase = database;
|
|
151
|
+
measurePhase(phaseTimings, "close_database_ms", () => openedDatabase.close());
|
|
152
|
+
}
|
|
153
|
+
finishCodeEvidencePhaseTimings(totalStarted, phaseTimings);
|
|
154
|
+
}
|
|
155
|
+
}
|
|
156
|
+
function requireCompatibleDatabaseTimed(database, runtime, timings) {
|
|
157
|
+
measurePhase(timings, "compatibility_ms", () => requireCompatibleDatabase(database, runtime));
|
|
158
|
+
}
|
|
159
|
+
function checkedCodeIndexStaleness(database, runtime, timings) {
|
|
160
|
+
const staleness = measurePhase(timings, "staleness_ms", () => runtime.codeIndexStaleness(database));
|
|
161
|
+
runtime.warnIfCodeIndexStale(database, staleness);
|
|
162
|
+
return staleness;
|
|
163
|
+
}
|
|
134
164
|
function configureBulkWriteConnection(database) {
|
|
135
165
|
database.exec(`
|
|
136
166
|
PRAGMA synchronous = OFF;
|
|
@@ -199,13 +229,11 @@ function runCodeIndexMode(runtime) {
|
|
|
199
229
|
if (!args_1.codeIndexIncrementalMode) {
|
|
200
230
|
try {
|
|
201
231
|
measurePhase(phaseTimings, "native_helper_ms", () => runtime.runNativeCodeIndexMode({ databasePath, discoveredFiles, parserMode, requestedEngine, scopes }));
|
|
202
|
-
|
|
203
|
-
emitCodeIndexPhaseTimings(phaseTimings);
|
|
232
|
+
finishCodeEvidencePhaseTimings(totalStarted, phaseTimings);
|
|
204
233
|
return;
|
|
205
234
|
}
|
|
206
235
|
catch (error) {
|
|
207
|
-
|
|
208
|
-
emitCodeIndexPhaseTimings(phaseTimings);
|
|
236
|
+
finishCodeEvidencePhaseTimings(totalStarted, phaseTimings);
|
|
209
237
|
throw error;
|
|
210
238
|
}
|
|
211
239
|
}
|
|
@@ -278,8 +306,7 @@ function runCodeIndexMode(runtime) {
|
|
|
278
306
|
scopes,
|
|
279
307
|
unchangedFiles,
|
|
280
308
|
}));
|
|
281
|
-
|
|
282
|
-
emitCodeIndexPhaseTimings(phaseTimings);
|
|
309
|
+
finishCodeEvidencePhaseTimings(totalStarted, phaseTimings);
|
|
283
310
|
return;
|
|
284
311
|
}
|
|
285
312
|
}
|
|
@@ -303,7 +330,6 @@ function runCodeIndexMode(runtime) {
|
|
|
303
330
|
(0, schema_1.createSecondaryIndexes)(activeDatabase);
|
|
304
331
|
activeDatabase.exec("COMMIT");
|
|
305
332
|
});
|
|
306
|
-
phaseTimings.total_ms = Number(elapsedMs(totalStarted).toFixed(3));
|
|
307
333
|
console.log("Project wiki code evidence index complete.");
|
|
308
334
|
console.log(`database: ${databasePath.relativePath}`);
|
|
309
335
|
console.log(`mode: ${incremental ? "incremental" : "full"}`);
|
|
@@ -316,7 +342,7 @@ function runCodeIndexMode(runtime) {
|
|
|
316
342
|
console.log(`reindexed_files: ${reindexedFiles.length}`);
|
|
317
343
|
console.log(`deleted_files: ${deletedPaths.length}`);
|
|
318
344
|
console.log(`unchanged_files: ${unchangedFiles}`);
|
|
319
|
-
|
|
345
|
+
finishCodeEvidencePhaseTimings(totalStarted, phaseTimings);
|
|
320
346
|
}
|
|
321
347
|
catch (error) {
|
|
322
348
|
try {
|
|
@@ -336,123 +362,93 @@ function runCodeQueryMode(runtime) {
|
|
|
336
362
|
console.error("missing SQL: use --code-query \"select ...\"");
|
|
337
363
|
process.exit(1);
|
|
338
364
|
}
|
|
339
|
-
runtime
|
|
340
|
-
if (!(0, code_index_sql_1.isReadOnlySql)(args_1.codeQuerySql)) {
|
|
341
|
-
console.error("code queries must be read-only SQL starting with SELECT or WITH");
|
|
342
|
-
process.exit(1);
|
|
343
|
-
}
|
|
344
|
-
const database = runtime.openDatabase(runtime.codeEvidenceDatabasePath().absolutePath);
|
|
345
|
-
try {
|
|
365
|
+
runWithCodeEvidenceDatabase(runtime, (database, phaseTimings) => {
|
|
346
366
|
database.exec("PRAGMA query_only = ON");
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
367
|
+
requireCompatibleDatabaseTimed(database, runtime, phaseTimings);
|
|
368
|
+
checkedCodeIndexStaleness(database, runtime, phaseTimings);
|
|
369
|
+
const rows = measurePhase(phaseTimings, "query_ms", () => database.prepare(args_1.codeQuerySql).all());
|
|
370
|
+
measurePhase(phaseTimings, "render_ms", () => printRows(rows));
|
|
371
|
+
}, {
|
|
372
|
+
beforeOpen() {
|
|
373
|
+
if (!(0, code_index_sql_1.isReadOnlySql)(args_1.codeQuerySql)) {
|
|
374
|
+
console.error("code queries must be read-only SQL starting with SELECT or WITH");
|
|
375
|
+
process.exit(1);
|
|
376
|
+
}
|
|
377
|
+
},
|
|
378
|
+
});
|
|
354
379
|
}
|
|
355
380
|
function runCodeReportMode(runtime) {
|
|
356
|
-
runtime
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
const staleness = runtime.codeIndexStaleness(database);
|
|
361
|
-
runtime.warnIfCodeIndexStale(database, staleness);
|
|
362
|
-
const report = runtime.codeReportForRequestedSection(database, args_1.codeReportSection, { staleness });
|
|
381
|
+
runWithCodeEvidenceDatabase(runtime, (database, phaseTimings) => {
|
|
382
|
+
requireCompatibleDatabaseTimed(database, runtime, phaseTimings);
|
|
383
|
+
const staleness = checkedCodeIndexStaleness(database, runtime, phaseTimings);
|
|
384
|
+
const report = measurePhase(phaseTimings, "query_ms", () => runtime.codeReportForRequestedSection(database, args_1.codeReportSection, { staleness }));
|
|
363
385
|
if (!report)
|
|
364
386
|
runtime.fail((0, reports_1.invalidCodeReportSectionMessage)(args_1.codeReportSection));
|
|
365
|
-
printJson(report);
|
|
366
|
-
}
|
|
367
|
-
finally {
|
|
368
|
-
database.close();
|
|
369
|
-
}
|
|
387
|
+
measurePhase(phaseTimings, "render_ms", () => printJson(report));
|
|
388
|
+
});
|
|
370
389
|
}
|
|
371
390
|
function runCodeStatusMode(runtime) {
|
|
372
|
-
runtime
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
requireCompatibleDatabase(database, runtime);
|
|
376
|
-
const rows = database.prepare(`
|
|
391
|
+
runWithCodeEvidenceDatabase(runtime, (database, phaseTimings) => {
|
|
392
|
+
requireCompatibleDatabaseTimed(database, runtime, phaseTimings);
|
|
393
|
+
const rows = measurePhase(phaseTimings, "query_ms", () => database.prepare(`
|
|
377
394
|
SELECT 'files' AS metric, count(*) AS value FROM files
|
|
378
395
|
UNION ALL SELECT 'symbols', count(*) FROM symbols
|
|
379
396
|
UNION ALL SELECT 'imports', count(*) FROM imports
|
|
380
397
|
UNION ALL SELECT 'routes', count(*) FROM routes
|
|
381
398
|
UNION ALL SELECT 'edges', count(*) FROM edges
|
|
382
399
|
UNION ALL SELECT 'configs', count(*) FROM configs
|
|
383
|
-
`).all();
|
|
384
|
-
const staleness = runtime.codeIndexStaleness(database);
|
|
400
|
+
`).all());
|
|
401
|
+
const staleness = measurePhase(phaseTimings, "staleness_ms", () => runtime.codeIndexStaleness(database));
|
|
385
402
|
rows.push({ metric: "stale_files", value: staleness.added + staleness.changed + staleness.deleted }, { metric: "stale_changed_files", value: staleness.changed }, { metric: "stale_added_files", value: staleness.added }, { metric: "stale_deleted_files", value: staleness.deleted });
|
|
386
|
-
printRows(rows);
|
|
387
|
-
}
|
|
388
|
-
finally {
|
|
389
|
-
database.close();
|
|
390
|
-
}
|
|
403
|
+
measurePhase(phaseTimings, "render_ms", () => printRows(rows));
|
|
404
|
+
});
|
|
391
405
|
}
|
|
392
406
|
function runCodeIndexHealthMode(runtime) {
|
|
393
407
|
printJson(runtime.codeIndexHealth());
|
|
394
408
|
}
|
|
395
409
|
function runCodeFilesMode(runtime) {
|
|
396
|
-
runtime
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
402
|
-
}
|
|
403
|
-
finally {
|
|
404
|
-
database.close();
|
|
405
|
-
}
|
|
410
|
+
runWithCodeEvidenceDatabase(runtime, (database, phaseTimings) => {
|
|
411
|
+
requireCompatibleDatabaseTimed(database, runtime, phaseTimings);
|
|
412
|
+
checkedCodeIndexStaleness(database, runtime, phaseTimings);
|
|
413
|
+
const rows = measurePhase(phaseTimings, "query_ms", () => database.prepare("SELECT path, language, profile, kind, lines, bytes FROM files ORDER BY path").all());
|
|
414
|
+
measurePhase(phaseTimings, "render_ms", () => printRows(rows));
|
|
415
|
+
});
|
|
406
416
|
}
|
|
407
417
|
function runCodeImpactMode(runtime) {
|
|
408
418
|
if (!args_1.codeImpactTarget.trim()) {
|
|
409
419
|
console.error("missing impact target: use --code-impact \"path-or-symbol-or-module\"");
|
|
410
420
|
process.exit(1);
|
|
411
421
|
}
|
|
412
|
-
runtime
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
printJson(runtime.codeImpact(database, args_1.codeImpactTarget.trim(), { staleness }));
|
|
419
|
-
}
|
|
420
|
-
finally {
|
|
421
|
-
database.close();
|
|
422
|
-
}
|
|
422
|
+
runWithCodeEvidenceDatabase(runtime, (database, phaseTimings) => {
|
|
423
|
+
requireCompatibleDatabaseTimed(database, runtime, phaseTimings);
|
|
424
|
+
const staleness = checkedCodeIndexStaleness(database, runtime, phaseTimings);
|
|
425
|
+
const impact = measurePhase(phaseTimings, "query_ms", () => runtime.codeImpact(database, args_1.codeImpactTarget.trim(), { staleness }));
|
|
426
|
+
measurePhase(phaseTimings, "render_ms", () => printJson(impact));
|
|
427
|
+
});
|
|
423
428
|
}
|
|
424
429
|
function runCodeContextPackMode(runtime) {
|
|
425
430
|
if (!args_1.codeContextPackTarget.trim()) {
|
|
426
431
|
console.error("missing context pack query: use --code-context-pack \"path-or-symbol-or-route\"");
|
|
427
432
|
process.exit(1);
|
|
428
433
|
}
|
|
429
|
-
runtime
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
console.log(runtime.codeContextPack(database, args_1.codeContextPackTarget.trim(), { staleness }));
|
|
436
|
-
}
|
|
437
|
-
finally {
|
|
438
|
-
database.close();
|
|
439
|
-
}
|
|
434
|
+
runWithCodeEvidenceDatabase(runtime, (database, phaseTimings) => {
|
|
435
|
+
requireCompatibleDatabaseTimed(database, runtime, phaseTimings);
|
|
436
|
+
const staleness = checkedCodeIndexStaleness(database, runtime, phaseTimings);
|
|
437
|
+
const pack = measurePhase(phaseTimings, "query_ms", () => runtime.codeContextPack(database, args_1.codeContextPackTarget.trim(), { staleness }));
|
|
438
|
+
measurePhase(phaseTimings, "render_ms", () => console.log(pack));
|
|
439
|
+
});
|
|
440
440
|
}
|
|
441
441
|
function runCodeSearchSymbolMode(runtime) {
|
|
442
442
|
if (!args_1.codeSearchSymbol.trim()) {
|
|
443
443
|
console.error("missing symbol search term: use --code-search-symbol \"term\"");
|
|
444
444
|
process.exit(1);
|
|
445
445
|
}
|
|
446
|
-
runtime
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
}
|
|
453
|
-
finally {
|
|
454
|
-
database.close();
|
|
455
|
-
}
|
|
446
|
+
runWithCodeEvidenceDatabase(runtime, (database, phaseTimings) => {
|
|
447
|
+
requireCompatibleDatabaseTimed(database, runtime, phaseTimings);
|
|
448
|
+
checkedCodeIndexStaleness(database, runtime, phaseTimings);
|
|
449
|
+
const rows = measurePhase(phaseTimings, "query_ms", () => (0, search_1.searchSymbols)(database, args_1.codeSearchSymbol.trim()));
|
|
450
|
+
measurePhase(phaseTimings, "render_ms", () => printRows(rows));
|
|
451
|
+
});
|
|
456
452
|
}
|
|
457
453
|
function isCodeEvidenceModeFor(flags) {
|
|
458
454
|
return Boolean(flags.codeContextPackTarget)
|
|
@@ -65,15 +65,126 @@ function readJsonObject(relativePath) {
|
|
|
65
65
|
function workspacePatternsFromRootPackage() {
|
|
66
66
|
const rootPackage = readJsonObject("package.json");
|
|
67
67
|
const workspaces = rootPackage?.workspaces;
|
|
68
|
-
if (Array.isArray(workspaces))
|
|
69
|
-
return workspaces
|
|
68
|
+
if (Array.isArray(workspaces)) {
|
|
69
|
+
return workspaces
|
|
70
|
+
.filter((value) => typeof value === "string")
|
|
71
|
+
.map((pattern) => ({ pattern, source: "package.json workspaces" }));
|
|
72
|
+
}
|
|
70
73
|
if (workspaces && typeof workspaces === "object" && !Array.isArray(workspaces)) {
|
|
71
74
|
const packages = workspaces.packages;
|
|
72
|
-
if (Array.isArray(packages))
|
|
73
|
-
return packages
|
|
75
|
+
if (Array.isArray(packages)) {
|
|
76
|
+
return packages
|
|
77
|
+
.filter((value) => typeof value === "string")
|
|
78
|
+
.map((pattern) => ({ pattern, source: "package.json workspaces" }));
|
|
79
|
+
}
|
|
74
80
|
}
|
|
75
81
|
return [];
|
|
76
82
|
}
|
|
83
|
+
function trimYamlComment(value) {
|
|
84
|
+
let quote = "";
|
|
85
|
+
for (let index = 0; index < value.length; index += 1) {
|
|
86
|
+
const character = value[index] ?? "";
|
|
87
|
+
if (quote) {
|
|
88
|
+
if (character === quote)
|
|
89
|
+
quote = "";
|
|
90
|
+
continue;
|
|
91
|
+
}
|
|
92
|
+
if (character === "\"" || character === "'") {
|
|
93
|
+
quote = character;
|
|
94
|
+
continue;
|
|
95
|
+
}
|
|
96
|
+
if (character === "#" && (index === 0 || /\s/.test(value[index - 1] ?? "")))
|
|
97
|
+
return value.slice(0, index).trimEnd();
|
|
98
|
+
}
|
|
99
|
+
return value.trimEnd();
|
|
100
|
+
}
|
|
101
|
+
function parseYamlStringScalar(value) {
|
|
102
|
+
const trimmed = trimYamlComment(value).trim();
|
|
103
|
+
if (!trimmed)
|
|
104
|
+
return null;
|
|
105
|
+
if (trimmed.startsWith("\"")) {
|
|
106
|
+
const match = trimmed.match(/^"((?:\\.|[^"\\])*)"/);
|
|
107
|
+
if (!match)
|
|
108
|
+
return null;
|
|
109
|
+
return match[1]?.replace(/\\"/g, "\"") ?? "";
|
|
110
|
+
}
|
|
111
|
+
if (trimmed.startsWith("'")) {
|
|
112
|
+
const match = trimmed.match(/^'((?:''|[^'])*)'/);
|
|
113
|
+
if (!match)
|
|
114
|
+
return null;
|
|
115
|
+
return match[1]?.replace(/''/g, "'") ?? "";
|
|
116
|
+
}
|
|
117
|
+
return trimmed;
|
|
118
|
+
}
|
|
119
|
+
function splitYamlFlowArray(value) {
|
|
120
|
+
const trimmed = trimYamlComment(value).trim();
|
|
121
|
+
if (!trimmed.startsWith("[") || !trimmed.endsWith("]"))
|
|
122
|
+
return null;
|
|
123
|
+
const entries = [];
|
|
124
|
+
let quote = "";
|
|
125
|
+
let current = "";
|
|
126
|
+
for (const character of trimmed.slice(1, -1)) {
|
|
127
|
+
if (quote) {
|
|
128
|
+
current += character;
|
|
129
|
+
if (character === quote)
|
|
130
|
+
quote = "";
|
|
131
|
+
continue;
|
|
132
|
+
}
|
|
133
|
+
if (character === "\"" || character === "'") {
|
|
134
|
+
quote = character;
|
|
135
|
+
current += character;
|
|
136
|
+
continue;
|
|
137
|
+
}
|
|
138
|
+
if (character === ",") {
|
|
139
|
+
entries.push(current);
|
|
140
|
+
current = "";
|
|
141
|
+
continue;
|
|
142
|
+
}
|
|
143
|
+
current += character;
|
|
144
|
+
}
|
|
145
|
+
entries.push(current);
|
|
146
|
+
return entries;
|
|
147
|
+
}
|
|
148
|
+
function workspacePatternsFromPnpmWorkspace() {
|
|
149
|
+
const filePath = "pnpm-workspace.yaml";
|
|
150
|
+
if (!(0, workspace_1.containedProjectFileStat)(filePath))
|
|
151
|
+
return [];
|
|
152
|
+
const lines = (0, workspace_1.read)(filePath).split(/\r?\n/);
|
|
153
|
+
const patterns = [];
|
|
154
|
+
let packagesIndent = null;
|
|
155
|
+
for (const line of lines) {
|
|
156
|
+
if (!line.trim() || line.trimStart().startsWith("#"))
|
|
157
|
+
continue;
|
|
158
|
+
const indent = line.match(/^\s*/)?.[0].length ?? 0;
|
|
159
|
+
const packagesMatch = line.match(/^(\s*)packages\s*:\s*(.*)$/);
|
|
160
|
+
if (packagesMatch) {
|
|
161
|
+
packagesIndent = packagesMatch[1]?.length ?? 0;
|
|
162
|
+
const inlineArray = splitYamlFlowArray(packagesMatch[2] ?? "");
|
|
163
|
+
if (inlineArray) {
|
|
164
|
+
for (const entry of inlineArray) {
|
|
165
|
+
const pattern = parseYamlStringScalar(entry);
|
|
166
|
+
if (pattern && !pattern.startsWith("!"))
|
|
167
|
+
patterns.push({ pattern, source: "pnpm-workspace.yaml packages" });
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
continue;
|
|
171
|
+
}
|
|
172
|
+
if (packagesIndent === null)
|
|
173
|
+
continue;
|
|
174
|
+
if (indent <= packagesIndent)
|
|
175
|
+
break;
|
|
176
|
+
const item = line.slice(indent).match(/^-\s+(.+)$/);
|
|
177
|
+
if (!item)
|
|
178
|
+
continue;
|
|
179
|
+
const pattern = parseYamlStringScalar(item[1] ?? "");
|
|
180
|
+
if (pattern && !pattern.startsWith("!"))
|
|
181
|
+
patterns.push({ pattern, source: "pnpm-workspace.yaml packages" });
|
|
182
|
+
}
|
|
183
|
+
return patterns;
|
|
184
|
+
}
|
|
185
|
+
function workspacePatterns() {
|
|
186
|
+
return [...workspacePatternsFromRootPackage(), ...workspacePatternsFromPnpmWorkspace()];
|
|
187
|
+
}
|
|
77
188
|
function workspacePatternCandidates(pattern) {
|
|
78
189
|
const normalized = (0, workspace_1.normalizePath)(pattern).replace(/^\/+/, "").replace(/\/+$/, "");
|
|
79
190
|
if (!normalized || normalized.includes(".."))
|
|
@@ -94,8 +205,8 @@ function workspacePatternCandidates(pattern) {
|
|
|
94
205
|
}
|
|
95
206
|
function workspacePackages() {
|
|
96
207
|
const packages = new Map();
|
|
97
|
-
for (const
|
|
98
|
-
for (const candidate of workspacePatternCandidates(pattern)) {
|
|
208
|
+
for (const workspacePattern of workspacePatterns()) {
|
|
209
|
+
for (const candidate of workspacePatternCandidates(workspacePattern.pattern)) {
|
|
99
210
|
const packageJsonPath = (0, workspace_1.normalizePath)(path.join(candidate, "package.json"));
|
|
100
211
|
if (!(0, workspace_1.containedProjectFileStat)(packageJsonPath))
|
|
101
212
|
continue;
|
|
@@ -104,8 +215,8 @@ function workspacePackages() {
|
|
|
104
215
|
packages.set(candidate, {
|
|
105
216
|
name: packageName,
|
|
106
217
|
root: candidate,
|
|
107
|
-
source:
|
|
108
|
-
workspace_pattern: pattern,
|
|
218
|
+
source: workspacePattern.source,
|
|
219
|
+
workspace_pattern: workspacePattern.pattern,
|
|
109
220
|
});
|
|
110
221
|
}
|
|
111
222
|
}
|
|
@@ -47,7 +47,7 @@ exports.incrementalCompatibility = incrementalCompatibility;
|
|
|
47
47
|
exports.codeIndexSnapshot = codeIndexSnapshot;
|
|
48
48
|
const crypto = __importStar(require("node:crypto"));
|
|
49
49
|
const workspace_1 = require("../workspace");
|
|
50
|
-
exports.codeIndexSchemaVersion = "
|
|
50
|
+
exports.codeIndexSchemaVersion = "6";
|
|
51
51
|
function stableFtsRowid(parts) {
|
|
52
52
|
const hash = crypto.createHash("sha256");
|
|
53
53
|
for (const part of parts) {
|
|
@@ -158,7 +158,7 @@ function createIndexStatements(database) {
|
|
|
158
158
|
insertMeta: database.prepare("INSERT OR REPLACE INTO meta (key, value) VALUES (?, ?)"),
|
|
159
159
|
insertRoute: database.prepare("INSERT INTO routes (method, route, file_path, line, handler) VALUES (?, ?, ?, ?, ?)"),
|
|
160
160
|
insertSymbol: database.prepare("INSERT INTO symbols (name, kind, file_path, line, signature) VALUES (?, ?, ?, ?, ?)"),
|
|
161
|
-
insertSymbolFts: database.prepare("INSERT INTO symbols_fts (name, kind, file_path, signature) VALUES (?, ?, ?, ?)"),
|
|
161
|
+
insertSymbolFts: database.prepare("INSERT INTO symbols_fts (rowid, name, kind, file_path, signature) VALUES (last_insert_rowid(), ?, ?, ?, ?)"),
|
|
162
162
|
};
|
|
163
163
|
}
|
|
164
164
|
function removeIndexedFile(filePath, statements) {
|
|
@@ -130,11 +130,7 @@ function searchSymbols(database, term, limit = 50) {
|
|
|
130
130
|
const ftsRows = database.prepare(`
|
|
131
131
|
SELECT symbols.name, symbols.kind, symbols.file_path, symbols.line, symbols.signature
|
|
132
132
|
FROM symbols_fts
|
|
133
|
-
JOIN symbols
|
|
134
|
-
ON symbols.name = symbols_fts.name
|
|
135
|
-
AND symbols.kind = symbols_fts.kind
|
|
136
|
-
AND symbols.file_path = symbols_fts.file_path
|
|
137
|
-
AND symbols.signature = symbols_fts.signature
|
|
133
|
+
JOIN symbols ON symbols.id = symbols_fts.rowid
|
|
138
134
|
WHERE symbols_fts MATCH ?
|
|
139
135
|
ORDER BY bm25(symbols_fts, 8.0, 1.0, 4.0, 2.0), symbols.file_path, symbols.line
|
|
140
136
|
LIMIT ?
|
package/dist/code-index.js
CHANGED
|
@@ -648,16 +648,22 @@ function codeContextScaleLine(fileCount) {
|
|
|
648
648
|
? `scale small (${fileCount} indexed files < ${code_index_file_policy_1.SMALL_REPO_FILE_THRESHOLD}); direct reads are usually cheaper for simple lookups`
|
|
649
649
|
: `scale large (${fileCount} indexed files >= ${code_index_file_policy_1.SMALL_REPO_FILE_THRESHOLD}); indexed traversal is useful for impact-style context`;
|
|
650
650
|
}
|
|
651
|
-
function
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
651
|
+
function codeContextCollectorOptions(fileCount) {
|
|
652
|
+
if (fileCount < code_index_file_policy_1.SMALL_REPO_FILE_THRESHOLD) {
|
|
653
|
+
return {
|
|
654
|
+
edgeLimit: 8,
|
|
655
|
+
fileLimit: 8,
|
|
656
|
+
includeEdgeEvidenceMatches: false,
|
|
657
|
+
includeOwnerCodeowners: false,
|
|
658
|
+
includeRouteEdges: false,
|
|
659
|
+
importLimit: 12,
|
|
660
|
+
ownerSampleLimit: 3,
|
|
661
|
+
routeEdgeLimit: 0,
|
|
662
|
+
routeLimit: 8,
|
|
663
|
+
symbolLimit: 12,
|
|
664
|
+
};
|
|
665
|
+
}
|
|
666
|
+
return {
|
|
661
667
|
edgeLimit: 30,
|
|
662
668
|
fileLimit: 12,
|
|
663
669
|
includeEdgeEvidenceMatches: true,
|
|
@@ -668,14 +674,26 @@ function codeContextPack(database, query, options = {}) {
|
|
|
668
674
|
routeEdgeLimit: 0,
|
|
669
675
|
routeLimit: 20,
|
|
670
676
|
symbolLimit: 20,
|
|
671
|
-
}
|
|
672
|
-
|
|
677
|
+
};
|
|
678
|
+
}
|
|
679
|
+
function structuralSignature(value) {
|
|
680
|
+
const signature = (0, shared_1.oneLine)(String(value ?? ""));
|
|
681
|
+
const bodyStart = signature.indexOf("{");
|
|
682
|
+
return bodyStart >= 0 ? signature.slice(0, bodyStart).trimEnd() : signature;
|
|
683
|
+
}
|
|
684
|
+
function codeContextPack(database, query, options = {}) {
|
|
685
|
+
const normalized = query.trim();
|
|
686
|
+
if (!normalized)
|
|
687
|
+
return 'Code context pack: missing query; use --code-context-pack "path-or-symbol-or-route".';
|
|
673
688
|
const coverage = (0, reports_1.evidenceCoverage)(database);
|
|
689
|
+
const indexedFileCount = Number(coverage.files ?? 0);
|
|
690
|
+
const evidence = (0, evidence_1.collectCodeEvidence)(database, normalized, codeContextCollectorOptions(indexedFileCount));
|
|
691
|
+
const staleness = options.staleness ?? codeIndexStaleness(database);
|
|
674
692
|
const staleLabel = staleness.stale
|
|
675
693
|
? `STALE ${staleness.changed} changed, ${staleness.added} added, ${staleness.deleted} deleted`
|
|
676
694
|
: "fresh";
|
|
677
695
|
const lines = [
|
|
678
|
-
`Code context pack "${normalized}": ${evidence.files.length} file matches, ${evidence.symbols.length} symbols, ${evidence.routes.length} routes, ${evidence.imports.length} imports, ${evidence.incomingEdges.length} incoming / ${evidence.outgoingEdges.length} outgoing edges; index ${staleLabel}; ${codeContextScaleLine(
|
|
696
|
+
`Code context pack "${normalized}": ${evidence.files.length} file matches, ${evidence.symbols.length} symbols, ${evidence.routes.length} routes, ${evidence.imports.length} imports, ${evidence.incomingEdges.length} incoming / ${evidence.outgoingEdges.length} outgoing edges; index ${staleLabel}; ${codeContextScaleLine(indexedFileCount)}.`,
|
|
679
697
|
"Evidence is structural only: paths, lines, signatures, routes, imports, edges, and owners; no source snippets are included.",
|
|
680
698
|
];
|
|
681
699
|
pushBudgetedSection(lines, "Files:", evidence.files, 8, (row) => ` file-match ${String(row.path)} (${String(row.language)}, ${String(row.profile)}, ${Number(row.lines ?? 0)} lines)`);
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
Binary file
|
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
"architecture": "arm64",
|
|
6
6
|
"format": "mach-o",
|
|
7
7
|
"path": "dist/native/darwin-arm64/project-librarian-indexer",
|
|
8
|
-
"sha256": "
|
|
8
|
+
"sha256": "848c29daf166e55672add5febf22fcd2ba261619a37969d9bc87cff06815787c",
|
|
9
9
|
"size": 2697520,
|
|
10
10
|
"triple": "darwin-arm64"
|
|
11
11
|
},
|
|
@@ -13,7 +13,7 @@
|
|
|
13
13
|
"architecture": "x64",
|
|
14
14
|
"format": "mach-o",
|
|
15
15
|
"path": "dist/native/darwin-x64/project-librarian-indexer",
|
|
16
|
-
"sha256": "
|
|
16
|
+
"sha256": "b419d82b7560defe1d43fc4d99d96b10222f1ead5bf6d7a6acabbae425ee7545",
|
|
17
17
|
"size": 2770592,
|
|
18
18
|
"triple": "darwin-x64"
|
|
19
19
|
},
|
|
@@ -21,7 +21,7 @@
|
|
|
21
21
|
"architecture": "arm64",
|
|
22
22
|
"format": "elf",
|
|
23
23
|
"path": "dist/native/linux-arm64/project-librarian-indexer",
|
|
24
|
-
"sha256": "
|
|
24
|
+
"sha256": "f3cc884acca4f5863fb6a6545a4a9f9a37a67872bfe780f4b14f0c4a8a667f16",
|
|
25
25
|
"size": 3065520,
|
|
26
26
|
"triple": "linux-arm64"
|
|
27
27
|
},
|
|
@@ -29,31 +29,31 @@
|
|
|
29
29
|
"architecture": "arm64",
|
|
30
30
|
"format": "elf",
|
|
31
31
|
"path": "dist/native/linux-arm64-musl/project-librarian-indexer",
|
|
32
|
-
"sha256": "
|
|
33
|
-
"size":
|
|
32
|
+
"sha256": "80f7440c4d115d05805b63b8118a172ddeb5a374b7154593e5b1d0b804a938ae",
|
|
33
|
+
"size": 3100816,
|
|
34
34
|
"triple": "linux-arm64-musl"
|
|
35
35
|
},
|
|
36
36
|
{
|
|
37
37
|
"architecture": "x64",
|
|
38
38
|
"format": "elf",
|
|
39
39
|
"path": "dist/native/linux-x64/project-librarian-indexer",
|
|
40
|
-
"sha256": "
|
|
41
|
-
"size":
|
|
40
|
+
"sha256": "3d93408db344567d7de6f477b58fdaf6aba19c943336fb7ab9720e3e9fa68c31",
|
|
41
|
+
"size": 3110832,
|
|
42
42
|
"triple": "linux-x64"
|
|
43
43
|
},
|
|
44
44
|
{
|
|
45
45
|
"architecture": "x64",
|
|
46
46
|
"format": "elf",
|
|
47
47
|
"path": "dist/native/linux-x64-musl/project-librarian-indexer",
|
|
48
|
-
"sha256": "
|
|
49
|
-
"size":
|
|
48
|
+
"sha256": "0055ec9fefa385d6d637fa1e457308c38f892e0fbdb389db3fcbdf546f0daca8",
|
|
49
|
+
"size": 3247400,
|
|
50
50
|
"triple": "linux-x64-musl"
|
|
51
51
|
},
|
|
52
52
|
{
|
|
53
53
|
"architecture": "arm64",
|
|
54
54
|
"format": "pe",
|
|
55
55
|
"path": "dist/native/win32-arm64/project-librarian-indexer.exe",
|
|
56
|
-
"sha256": "
|
|
56
|
+
"sha256": "f8ef16c98ff97c4f1dd38e2e08a034a40bb219f114f8270d165feaddd8166d8f",
|
|
57
57
|
"size": 2098688,
|
|
58
58
|
"triple": "win32-arm64"
|
|
59
59
|
},
|
|
@@ -61,7 +61,7 @@
|
|
|
61
61
|
"architecture": "x64",
|
|
62
62
|
"format": "pe",
|
|
63
63
|
"path": "dist/native/win32-x64/project-librarian-indexer.exe",
|
|
64
|
-
"sha256": "
|
|
64
|
+
"sha256": "5a81b330741f11d10ab5d296e8cbc8c60f1596c30a45c5f31e41fab49e5b2603",
|
|
65
65
|
"size": 2375680,
|
|
66
66
|
"triple": "win32-x64"
|
|
67
67
|
}
|
|
Binary file
|
|
Binary file
|
package/dist/templates.js
CHANGED
|
@@ -199,7 +199,7 @@ exports.startup = `${(0, exports.metadata)("startup-router", "short", "wiki/meta
|
|
|
199
199
|
- This project is in an initial planning state unless the canonical wiki says otherwise.
|
|
200
200
|
- Project truth lives in \`wiki/canonical/\`, future work in \`wiki/roadmaps/\` and \`wiki/plans/\`, project decisions in \`wiki/decisions/\`, and sources in \`wiki/sources/\`.
|
|
201
201
|
- Wiki operating rules and wiki operating decisions live in \`wiki/meta/\`.
|
|
202
|
-
- At session start, read only this file and \`wiki/index.md\` first;
|
|
202
|
+
- At session start, read only this file and \`wiki/index.md\` first; use the index as a route table, open matching detail files directly, and avoid broad repo/wiki search unless no route matches.
|
|
203
203
|
- Project canonical content language is not fixed by this bootstrap. The LLM should choose the language that best matches the user and project context.
|
|
204
204
|
- Completed roadmaps/plans are removed after truth/rationale/evidence capture.
|
|
205
205
|
- Update the wiki in the same turn when project-planning content changes.
|
|
@@ -232,7 +232,7 @@ exports.startup = `${(0, exports.metadata)("startup-router", "short", "wiki/meta
|
|
|
232
232
|
## Token Discipline
|
|
233
233
|
|
|
234
234
|
- Codex, Claude Code, Cursor, and Gemini CLI session-start hooks inject only this file and \`wiki/index.md\`.
|
|
235
|
-
- Detailed files are selected by
|
|
235
|
+
- Detailed files are selected by \`wiki/index.md\`; use broad wiki search only when no route matches or evidence conflicts.
|
|
236
236
|
- Long decision history is not injected wholesale; read only relevant Decision Packs or ADRs.
|
|
237
237
|
`;
|
|
238
238
|
exports.index = `${(0, exports.metadata)("wiki-router", "short", "wiki/meta/wiki-ops-v1-decisions.md", "wiki page added, moved, removed, or routing changes")}
|
|
@@ -240,7 +240,7 @@ exports.index = `${(0, exports.metadata)("wiki-router", "short", "wiki/meta/wiki
|
|
|
240
240
|
|
|
241
241
|
## How To Use This Index
|
|
242
242
|
|
|
243
|
-
This
|
|
243
|
+
This is a route table, not a page inventory. Open the matching route first; use broad wiki search or file listing only when no route matches or evidence conflicts.
|
|
244
244
|
|
|
245
245
|
## Language Policy
|
|
246
246
|
|
package/docs/benchmarks.md
CHANGED
|
@@ -17,17 +17,33 @@ Generated benchmark reports under `benchmarks/reports/llm/` are ignored by defau
|
|
|
17
17
|
|
|
18
18
|
## Wiki Track
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
|
25
|
-
|
|
|
26
|
-
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
20
|
+
Latest clean synthetic wiki-track release evidence: 2026-06-29, `gpt-5.5`, branch `perf/small-repo-code-evidence-safeguards` at `ae79390`, 42 scenarios, 21 with/without pairs, 3 measured runs plus 1 warmup each. The overall claim gate **passed**, and the claim ledger classified the report as release-claimable. A repair run reused 125 claimable measured runs from retained raw JSONL and remeasured the remaining failed slot.
|
|
21
|
+
|
|
22
|
+
Cost-weighted tokens, Project Librarian vs control, all scales combined:
|
|
23
|
+
|
|
24
|
+
| Task family | Delta |
|
|
25
|
+
| --- | ---: |
|
|
26
|
+
| onboarding | 65.95% less |
|
|
27
|
+
| decision_lookup | 48.46% less |
|
|
28
|
+
| code_impact | 55.61% less |
|
|
29
|
+
| release_policy | 58.22% less |
|
|
30
|
+
| change_location | 29.59% less |
|
|
31
|
+
| multi_session | 52.58% less |
|
|
32
|
+
| aggregation | 42.53% less |
|
|
33
|
+
|
|
34
|
+
Scale-specific cost-weighted token deltas:
|
|
35
|
+
|
|
36
|
+
| Task family | Small | Medium | Large |
|
|
37
|
+
| --- | ---: | ---: | ---: |
|
|
38
|
+
| onboarding | 60.88% less | 65.53% less | 69.17% less |
|
|
39
|
+
| decision_lookup | 19.18% less | 51.29% less | 60.12% less |
|
|
40
|
+
| code_impact | 26.61% less | 54.81% less | 68.43% less |
|
|
41
|
+
| release_policy | 30.23% less | 62.16% less | 67.26% less |
|
|
42
|
+
| change_location | 7.79% less | 2.86% more | 57.27% less |
|
|
43
|
+
| multi_session | 54.29% more | 35.32% less | 82.77% less |
|
|
44
|
+
| aggregation | 7.66% less | 51.32% less | 49.17% less |
|
|
45
|
+
|
|
46
|
+
The release claim is bounded to the synthetic wiki-routing track and the listed task families. It is not a claim about code-graph behavior, real repositories, every agent surface, or every question shape. Published boundaries remain visible: small `multi_session` and medium `change_location` cost more with the wiki in the cost-weighted token metric. `code_impact`, `change_location`, and `aggregation` also still have wall-time or command-count regressions in this report, even though token and output-byte metrics improved at the task-family aggregate level.
|
|
31
47
|
|
|
32
48
|
## Code-Graph Track
|
|
33
49
|
|
|
@@ -52,7 +68,11 @@ Repositories under test:
|
|
|
52
68
|
|
|
53
69
|
Question types:
|
|
54
70
|
|
|
71
|
+
- **onboarding** - summarize what the project is, current risks, and where to read next.
|
|
55
72
|
- **decision_lookup** - find the latest project decision and its date from the wiki.
|
|
73
|
+
- **code_impact** - identify likely impacted files or areas for a benchmark report schema change.
|
|
74
|
+
- **release_policy** - find the checks required before publishing or making benchmark claims.
|
|
75
|
+
- **change_location** - find where an agent should edit to implement the Codex LLM benchmark runner.
|
|
56
76
|
- **aggregation** - answer a question whose facts are scattered across several pages and must be synthesized.
|
|
57
77
|
- **multi_session** - a second session on the same project, measuring whether the durable wiki helps the next session, not just the first.
|
|
58
78
|
- **impact_trace** - trace the full set of direct and indirect importers for a changing module.
|
package/docs/ko/benchmarks.md
CHANGED
|
@@ -17,17 +17,33 @@ npm run benchmark:release -- --allow-codex-run
|
|
|
17
17
|
|
|
18
18
|
## 위키 트랙
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
|
25
|
-
|
|
|
26
|
-
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
20
|
+
최신 clean 합성 위키 트랙 릴리스 근거는 2026-06-29에 `gpt-5.5`, `perf/small-repo-code-evidence-safeguards` 브랜치의 `ae79390` 커밋에서 측정했습니다. 42개 시나리오, 21개 with/without 쌍을 각 3회 측정하고 1회 예열했으며, 전체 주장 게이트는 **통과**했고 주장 ledger는 이 보고서를 release-claimable로 분류했습니다. repair 실행은 보존된 raw JSONL에서 claimable 측정 실행 125개를 재사용하고 남은 실패 슬롯 1개를 다시 측정했습니다.
|
|
21
|
+
|
|
22
|
+
비용 가중 토큰, Project Librarian 대 대조군, 전체 규모 합산:
|
|
23
|
+
|
|
24
|
+
| 작업 유형 | 변화 |
|
|
25
|
+
| --- | ---: |
|
|
26
|
+
| onboarding | 65.95% 적음 |
|
|
27
|
+
| decision_lookup | 48.46% 적음 |
|
|
28
|
+
| code_impact | 55.61% 적음 |
|
|
29
|
+
| release_policy | 58.22% 적음 |
|
|
30
|
+
| change_location | 29.59% 적음 |
|
|
31
|
+
| multi_session | 52.58% 적음 |
|
|
32
|
+
| aggregation | 42.53% 적음 |
|
|
33
|
+
|
|
34
|
+
규모별 비용 가중 토큰 변화:
|
|
35
|
+
|
|
36
|
+
| 작업 유형 | 소형 | 중형 | 대형 |
|
|
37
|
+
| --- | ---: | ---: | ---: |
|
|
38
|
+
| onboarding | 60.88% 적음 | 65.53% 적음 | 69.17% 적음 |
|
|
39
|
+
| decision_lookup | 19.18% 적음 | 51.29% 적음 | 60.12% 적음 |
|
|
40
|
+
| code_impact | 26.61% 적음 | 54.81% 적음 | 68.43% 적음 |
|
|
41
|
+
| release_policy | 30.23% 적음 | 62.16% 적음 | 67.26% 적음 |
|
|
42
|
+
| change_location | 7.79% 적음 | 2.86% 많음 | 57.27% 적음 |
|
|
43
|
+
| multi_session | 54.29% 많음 | 35.32% 적음 | 82.77% 적음 |
|
|
44
|
+
| aggregation | 7.66% 적음 | 51.32% 적음 | 49.17% 적음 |
|
|
45
|
+
|
|
46
|
+
이 릴리스 주장은 합성 위키 라우팅 트랙과 표기된 작업 유형에 한정됩니다. 코드 그래프 동작, 실제 저장소, 모든 에이전트 표면, 모든 질문 형태에 대한 주장이 아닙니다. 숨기지 않는 한계도 남습니다. 비용 가중 토큰 기준으로 소형 `multi_session`과 중형 `change_location`은 위키를 켰을 때 더 비쌉니다. 또한 `code_impact`, `change_location`, `aggregation`은 작업 유형 집계에서 토큰과 출력 바이트가 개선됐더라도 이번 보고서의 실행 시간이나 명령 수에는 일부 역전이 남아 있습니다.
|
|
31
47
|
|
|
32
48
|
## 코드 그래프 트랙
|
|
33
49
|
|
|
@@ -52,7 +68,11 @@ SHA로 고정한 오픈소스 저장소 2곳에서, 손으로 작성한 정답
|
|
|
52
68
|
|
|
53
69
|
질문 유형:
|
|
54
70
|
|
|
71
|
+
- **onboarding** - 프로젝트가 무엇인지, 현재 리스크, 다음에 읽을 문서를 요약합니다.
|
|
55
72
|
- **decision_lookup** - 위키에서 가장 최근 프로젝트 결정과 그 날짜를 찾습니다.
|
|
73
|
+
- **code_impact** - 벤치마크 보고서 스키마 변경 시 영향받을 파일이나 영역을 찾습니다.
|
|
74
|
+
- **release_policy** - publish 또는 benchmark claim 전에 필요한 체크를 찾습니다.
|
|
75
|
+
- **change_location** - Codex LLM benchmark runner 구현 시 어디를 수정해야 하는지 찾습니다.
|
|
56
76
|
- **aggregation** - 여러 페이지에 흩어진 사실을 종합해야 답할 수 있는 질문입니다.
|
|
57
77
|
- **multi_session** - 같은 프로젝트의 두 번째 세션으로, 지속되는 위키가 다음 세션에도 도움이 되는지 측정합니다.
|
|
58
78
|
- **impact_trace** - 바뀌는 모듈의 직접/간접 importer 전체를 추적합니다.
|