@weftlet/toolkit 0.1.2 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. package/README.md +39 -3
  2. package/THIRD_PARTY_NOTICES.md +33 -0
  3. package/dist/application/hydrator.js +21 -6
  4. package/dist/application/overlay.js +46 -9
  5. package/dist/application/saved-evidence.d.ts +1 -0
  6. package/dist/configuration/config.d.ts +4 -0
  7. package/dist/configuration/config.js +12 -0
  8. package/dist/configuration/scip-project.d.ts +9 -0
  9. package/dist/configuration/scip-project.js +40 -0
  10. package/dist/extraction/codex-auth.js +9 -2
  11. package/dist/extraction/codex-command.d.ts +8 -0
  12. package/dist/extraction/codex-command.js +81 -0
  13. package/dist/extraction/document-references.d.ts +18 -0
  14. package/dist/extraction/document-references.js +94 -0
  15. package/dist/extraction/document.d.ts +9 -2
  16. package/dist/extraction/document.js +40 -47
  17. package/dist/extraction/module-request.d.ts +3 -0
  18. package/dist/extraction/module-request.js +25 -0
  19. package/dist/extraction/repository-nodes.d.ts +7 -0
  20. package/dist/extraction/repository-nodes.js +20 -0
  21. package/dist/extraction/routing.d.ts +14 -0
  22. package/dist/extraction/routing.js +151 -0
  23. package/dist/extraction/scip-project.d.ts +3 -0
  24. package/dist/extraction/scip-project.js +33 -0
  25. package/dist/extraction/semantic-process.js +17 -5
  26. package/dist/extraction/semantic.js +32 -5
  27. package/dist/extraction/static.d.ts +7 -3
  28. package/dist/extraction/static.js +15 -9
  29. package/dist/graph/query.js +21 -34
  30. package/dist/graph/ranking.d.ts +10 -0
  31. package/dist/graph/ranking.js +85 -0
  32. package/dist/interfaces/doctor.js +12 -3
  33. package/docs/ADR-002-saved-evidence.md +7 -1
  34. package/docs/ADR-003-draft-overlays.md +6 -0
  35. package/docs/ADR-004-rdf-query-path.md +6 -0
  36. package/docs/ADR-006-provider-boundaries.md +5 -0
  37. package/docs/ADR-010-committed-source-indexing.md +77 -0
  38. package/docs/ADR-011-explicit-document-links.md +48 -0
  39. package/docs/ADR-012-impact-ranking.md +50 -0
  40. package/docs/ADR-013-relative-module-requests.md +58 -0
  41. package/docs/ADR-014-routing-declarations.md +108 -0
  42. package/docs/ADR-015-repository-document-identity.md +57 -0
  43. package/docs/CONSUMER-TRIAL.md +166 -0
  44. package/docs/ENVIRONMENT-VALIDATION.md +9 -0
  45. package/docs/IMPACT-EVALUATION.md +363 -0
  46. package/docs/PROVIDERS.md +30 -2
  47. package/docs/RELEASE-1.0-PLAN.md +221 -0
  48. package/docs/RELEASING.md +16 -6
  49. package/docs/ROADMAP.md +53 -5
  50. package/docs/SOURCE-INVENTORY.md +10 -0
  51. package/docs/VALIDATION.md +438 -0
  52. package/docs/VIEWER.md +5 -0
  53. package/docs/WEFTLET-2.0-PLAN.md +282 -0
  54. package/docs/extraction-manifest.json +94 -17
  55. package/package.json +5 -2
@@ -0,0 +1,363 @@
1
+ # Impact retrieval evaluations
2
+
3
+ For an installed toolkit in your own project, start with the
4
+ [consumer trial](CONSUMER-TRIAL.md). It covers private log retention and human review using
5
+ existing CLI/MCP commands. The evaluator below is development tooling in this source checkout.
6
+
7
+ This is an advisory, reproducible evaluation of candidate-file retrieval on the real
8
+ repository. It is not a quality gate, a live model benchmark, or an assertion that a
9
+ proposed change is safe. The four scenarios and their expected findings were written
10
+ before the first run in `evaluations/weftlet/cases.json`. They are reviewable hypotheses;
11
+ they do not approve product intent or architecture policy.
12
+ The later dispatch evaluation below adds a separate synthetic project with new cases
13
+ declared before measurement, while keeping the implemented ranking fixed.
14
+
15
+ ## Reproduce the control
16
+
17
+ From a source checkout with the frozen dependencies installed:
18
+
19
+ ```sh
20
+ pnpm build
21
+ pnpm evaluate:impact --ref eeaaa342d62c11eeef44bfdf81fa8dedcfeb205d
22
+ ```
23
+
24
+ That exact commit must exist in local Git history. Omit `--ref` to measure the current
25
+ HEAD with the same cases. The runner rejects cases whose paths no longer exist; review
26
+ the expectations when the product changes instead of silently shrinking the denominator.
27
+ Keep the evidence SHA fixed for controlled comparisons. The earlier ordering used
28
+ commit-dependent fact IDs for ties; current ranking uses structural/citation fields first.
29
+ Freezing evidence still prevents changes in extracted facts from confounding the comparison.
30
+
31
+ The configuration uses real committed source bytes, bundled TypeScript SCIP, document
32
+ linking, RDF projection, SHACL, immutable publication and the normal saved-evidence query
33
+ service. The control provider returns **zero semantic assertions**. It permits measuring
34
+ literal document links without pretending to understand prose. No Codex login, API key,
35
+ Docker container or model call is required. The source checkout is not cleaned or modified.
36
+
37
+ Each scenario asks for 60 facts per page, up to three pages at depth 32. Reports distinguish
38
+ first-page recall, recall over the examined pages, relationships absent from the snapshot,
39
+ and relationships present but not retrieved within budget. A target counts only when a
40
+ returned fact names its exact file/document identity and is more than project membership.
41
+ Its provenance and extraction method accompany the score. This is still a generous
42
+ candidate-file metric: an import reference is weaker evidence than a typed symbol link,
43
+ and neither alone proves an end-to-end product dependency or that a test passed.
44
+
45
+ Unjudged results are not counted as false positives. Explicit distraction controls measure
46
+ noise in these tasks; their presence does not establish that the underlying graph assertion
47
+ is false. There is no claimed precision score without exhaustive human judgments.
48
+
49
+ The full local report is `.context-graph/evaluation/weftlet/report.json`. It records the
50
+ evidence commit, snapshot/content/configuration digests, exact configuration, suite and
51
+ runner-source digests, runtime, excluded working changes, coverage, witnesses and
52
+ continuations. Reports stay outside Git by default; review before sharing consumer results.
53
+ A failed refresh or scan exits nonzero and leaves the preceding report intact. Missing
54
+ coverage appears in a completed report; it is never converted into a passing quality gate.
55
+
56
+ To rescore saved evidence without rebuilding, installing a provider or reading its cache:
57
+
58
+ ```sh
59
+ pnpm evaluate:impact --snapshot <snapshot-id> --output .context-graph/evaluation/weftlet/rescored.json
60
+ ```
61
+
62
+ An absent snapshot is an error, with no refresh fallback. For a separately prepared provider
63
+ configuration, pass `--config <configuration.json> --snapshot <snapshot-id>`. Configuration
64
+ identity must match the snapshot. The runner refuses to refresh with a live provider.
65
+ Provider extraction is a separate, explicitly budgeted hydration operation; keep the same
66
+ six-document scope and review costs/authentication before running it. Source approval and
67
+ assertion review remain separate; the control configuration approves no intent sources.
68
+
69
+ ## Initial findings
70
+
71
+ The baseline uses Weftlet `eeaaa342d62c11eeef44bfdf81fa8dedcfeb205d`, before any
72
+ evaluation-specific links or indexing changes. The compact, public-safe measurement record
73
+ is [baseline.json](https://github.com/jchewitt/weftlet/blob/main/evaluations/weftlet/baseline.json);
74
+ full snapshots remain local. The evaluator is development tooling in the source repository,
75
+ not an installed `weftlet` CLI subcommand.
76
+
77
+ | Change scenario | Expected targets | First page | Three pages |
78
+ | --- | ---: | ---: | ---: |
79
+ | Atomic snapshot publication | 6 | 3 | 4 |
80
+ | Impact-query result/continuation | 6 | 2 | 3 |
81
+ | Codex executable discovery | 6 | 3 | 4 |
82
+ | Working-file capture and freshness | 6 | 5 | 5 |
83
+ | Total | 24 | 13 (54.2%) | 16 (66.7%) |
84
+
85
+ All four scans reached 1,368 of 1,369 snapshot facts; each stopped at 180 returned facts
86
+ with continuation available. The query scenario missed the MCP implementation within that
87
+ budget despite its presence in the graph. One of four distraction controls appeared within
88
+ three pages (telemetry for the atomic-publication task). These observations justify examining
89
+ ranking and overly broad relationships before treating eventual traversal recall as usefulness.
90
+
91
+ There were 849 SCIP facts from 45 paths, and 520 repository-file facts from 75 paths.
92
+ SCIP contributed **no test-file relationships**. Colocated tests appeared only through basic
93
+ import extraction; `tests/snapshot-lifecycle.test.ts` had no extracted relationship.
94
+ The core contract file had only project membership, without a useful consumer relationship.
95
+ `evidenceRoots` selects change scope; it does not itself add tests to the TypeScript project
96
+ or the repository-file extractor's roots. The committed archive also has no installed
97
+ workspace dependencies. These are configuration/indexing limitations to investigate, not
98
+ evidence that tests or core contracts are missing from the repository.
99
+
100
+ None of the six existing prose documents produced a document-link fact, and the five
101
+ distinct expected intent documents were absent from every result. The sources describe
102
+ behavior but lack explicit resolved implementation paths. This control cannot recover
103
+ conceptual prose-to-code mappings. A live provider comparison and human citation review
104
+ are needed before drawing any conclusion about semantic comprehension. Zero runtime
105
+ references and zero approved intent sources were configured intentionally.
106
+
107
+ ## Committed source-project comparison
108
+
109
+ The opt-in [source-project profile](ADR-010-committed-source-indexing.md) selects tests and
110
+ core source files explicitly and resolves `@weftlet/core` to its committed source entry.
111
+ It leaves the original profile and baseline intact. Run the comparison with:
112
+
113
+ ```sh
114
+ pnpm evaluate:impact --config evaluations/weftlet/source-project.config.json --ref eeaaa342d62c11eeef44bfdf81fa8dedcfeb205d --output .context-graph/evaluation/weftlet-source-project/report.json
115
+ ```
116
+
117
+ The commit, four cases, six documents, query budget and ranking are unchanged. The new
118
+ snapshot has 3,490 facts, including 2,970 SCIP facts and typed relationships from 33 test
119
+ files. The shared core contract and lifecycle test now appear in the results. The new
120
+ measurement is recorded in `evaluations/weftlet/source-project-baseline.json`.
121
+
122
+ | Change scenario | Original first / three pages | Source project first / three pages |
123
+ | --- | ---: | ---: |
124
+ | Atomic snapshot publication | 3 / 4 | 3 / 5 |
125
+ | Impact-query result/continuation | 2 / 3 | 4 / 4 |
126
+ | Codex executable discovery | 3 / 4 | 3 / 4 |
127
+ | Working-file capture and freshness | 5 / 5 | 5 / 5 |
128
+ | Total, out of 24 targets | 13 / 16 | 15 / 18 |
129
+
130
+ The original six test targets now have SCIP relationships as well as any simple import
131
+ facts. All four scans still reach the same broad component (3,490 facts) and stop at 180
132
+ returned facts; the MCP implementation remains outside the query budget. No labeled
133
+ distraction control appears within the examined pages in this run, which is not an
134
+ exhaustive precision result. The five expected intent documents are still unlinked.
135
+ There are still zero model assertions, approved intent sources and runtime references.
136
+
137
+ ## Explicit document-link measurement
138
+
139
+ [ADR 011](ADR-011-explicit-document-links.md) replaces loose path matching with cited committed
140
+ Markdown references. The original v2 snapshot remains queryable and unchanged. Under the new
141
+ linker, refreshing the same frozen old input adds two document references but preserves the
142
+ 15/24 first-page and 18/24 three-page scores. It cannot infer missing conceptual mappings.
143
+
144
+ The six documents now explicitly link descriptions of responsibilities to implementation and
145
+ test sources. Their evidence revision is `dcef64001d8d47cd1aeb4c1434992612158a104a`;
146
+ implementation and measurement identities are recorded in
147
+ `evaluations/weftlet/document-links-baseline.json`. Reproduce with:
148
+
149
+ ```sh
150
+ pnpm evaluate:impact --config evaluations/weftlet/source-project.config.json --ref dcef64001d8d47cd1aeb4c1434992612158a104a --output .context-graph/evaluation/weftlet-source-project/linked-documents.json
151
+ ```
152
+
153
+ | Change scenario | First page | Three pages | Labeled distraction within three pages |
154
+ | --- | ---: | ---: | --- |
155
+ | Atomic snapshot publication | 6 / 6 | 6 / 6 | None |
156
+ | Impact-query result/continuation | 5 / 6 | 6 / 6 | Codex command discovery |
157
+ | Codex executable discovery | 5 / 6 | 6 / 6 | Viewer interaction |
158
+ | Working-file capture and freshness | 6 / 6 | 6 / 6 | Codex command discovery (also on first page) |
159
+ | Total | 22 / 24 | 24 / 24 | 3 / 4 controls |
160
+
161
+ The graph contains 3,648 facts: 3,065 SCIP, 550 repository-file and 33 document-link facts.
162
+ All five expected intent-document paths now have cited links to actual file identities.
163
+ Every case still reaches the broad graph and stops at 180 returned facts with continuation.
164
+ The higher recall comes with additional noise; it is not a precision or completeness pass.
165
+
166
+ This changes the evidence input, including the preceding source-indexing implementation,
167
+ and the earlier commit-dependent tie-breaking could change ordering. The cases, query budget and ranking
168
+ implementation stay fixed, but the scores are not a controlled claim about semantic model
169
+ quality or ranking. Document links prove explicit references, not fulfillment of intent or
170
+ passing tests. There are still zero approved intent sources, model assertions, live model
171
+ calls and runtime observations. The new refresh reused six unchanged control artifacts.
172
+
173
+ ## Fixed-snapshot ranking comparison
174
+
175
+ [ADR 012](ADR-012-impact-ranking.md) changes ordering without changing extraction, reachability
176
+ or evidence authority. The version is `proximity-diversity-v1`; the measurement record is
177
+ `evaluations/weftlet/ranking-baseline.json`. Both existing snapshots were queried directly
178
+ and remained byte-for-byte unchanged. No indexer or model ran. The four cases, expected
179
+ paths, distraction labels, 60-fact page size, three-page budget and depth 32 stayed fixed.
180
+
181
+ | Saved evidence | Previous first / three pages | Ranked first / three pages | Distractions, previous / ranked within three pages |
182
+ | --- | ---: | ---: | ---: |
183
+ | Document links (3,648 facts) | 22 / 24 | 24 / 24 | 3 / 1 |
184
+ | Older source project (3,490 facts) | 15 / 18 | 19 / 19 | 0 / 1 |
185
+
186
+ Recall is out of 24 expected candidate files; distractions are out of four labeled controls.
187
+ On the document-link input all six targets in each case now appear on page one, including
188
+ the query test and setup doctor previously found later. No labeled distraction appears on
189
+ page one, down from one. On the older input, all 19 targets with extracted relationships now
190
+ appear on page one, including the MCP implementation previously outside the budget. The
191
+ five expected document links remain missing; ranking cannot supply absent evidence.
192
+
193
+ Telemetry now appears on a later page for atomic publication in both inputs. That case
194
+ previously had no labeled distraction, and the older control's noise score therefore worsens.
195
+ This is a tradeoff, not uniformly better relevance. The same broad graph remains reachable;
196
+ every scan stops at 180 facts with continuation. Four cases in one repository do not
197
+ establish generalization, precision, complete dependency discovery or change safety.
198
+
199
+ Reproduce against the saved document-link input:
200
+
201
+ ```sh
202
+ pnpm evaluate:impact --config evaluations/weftlet/source-project.config.json --snapshot dcef64001d8d47cd1aeb4c1434992612158a104a-graph-v3-links-2.0.0-42f885befb5ce881-4f68cb388720 --output .context-graph/evaluation/weftlet-source-project/ranking.json
203
+ ```
204
+
205
+ For the older control, substitute snapshot
206
+ `eeaaa342d62c11eeef44bfdf81fa8dedcfeb205d-graph-v2-42f885befb5ce881-4f68cb388720`.
207
+ These snapshots must already exist locally; the command does not rebuild missing evidence.
208
+ The report records the ranking version and query-source digests. After an ordering upgrade,
209
+ restart scans with old continuation tokens; they fail explicitly instead of silently
210
+ applying their offset to a different order.
211
+
212
+ ## Prospective dispatch evaluation
213
+
214
+ The dependency-free fictional Parcel Desk project, three cases and configuration were
215
+ committed at `fa902165a1f212a6d3245c847e1ab9b64a69715b` **before the first query**. It is
216
+ separate from Weftlet internals and the earlier pipeline fixture, but was authored after
217
+ ranking v1 existed. It is a prospective synthetic evaluation, not an independent external
218
+ benchmark. Production query/ranking bytes and both original case inputs remain unchanged.
219
+ Protocol and public-safe evidence are in `evaluations/dispatch/README.md` and
220
+ `evaluations/dispatch/baseline.json`.
221
+
222
+ Real committed TypeScript indexing, document links and RDF/SHACL produced 227 facts:
223
+ 164 SCIP, 50 repository-file and 13 document-link facts. The three draft documents generated
224
+ zero semantic assertions; no sources were approved and no runtime traces were imported.
225
+ Repeated hydration extracted zero and reused all three artifacts, with identical snapshot
226
+ identity/content. Fixture behavior checks passed separately; their execution is not a graph fact.
227
+
228
+ | Scenario | First page | Three pages | Distractions within three pages |
229
+ | --- | ---: | ---: | ---: |
230
+ | Upstream request contract | 6 / 6 | 6 / 6 | 2 / 2 |
231
+ | Asynchronous submission consumers | 5 / 6 | 5 / 6 | 2 / 2 |
232
+ | Same-name domain policy | 4 / 4 | 4 / 4 | 3 / 3 |
233
+ | Total | 15 / 16 | 15 / 16 | 7 / 7 |
234
+
235
+ No labeled distraction appears on page one. The seven labels are case-specific judgments
236
+ over three distinct files, not seven different files or exhaustive precision labels. Each
237
+ scan reaches the same 214 facts and stops at the unchanged three-page budget of 180 facts.
238
+ The source and labels were not adjusted after seeing these scores.
239
+
240
+ Two findings need separate treatment:
241
+
242
+ - The expected archival consumer has local definitions/test relationships but no extracted
243
+ route from dispatch. A supplemental scan exhausts all four pages (60/60/60/34) without
244
+ retrieving any of its facts. The generic scorer calls this `not-retrieved-within-budget`
245
+ because the file has relationships somewhere in the snapshot; here the diagnosis is a
246
+ disconnected operational route, not a lower-ranked reachable result. Matching topic text
247
+ does not establish delivery or a typed dependency.
248
+ - SCIP gives dispatch and refund `evaluatePolicy` functions distinct identities. The simpler
249
+ repository-file extractor nevertheless joins their planners through the shared literal
250
+ `contract:module:./policy.js`, even though those relative imports name different files.
251
+ A test importing both refund and audit sources extends that graph connection. This explains
252
+ a path to irrelevant candidates; it does not establish product causality. The missing
253
+ `docs/policy.md` Markdown reference produces a cited diagnostic and no guessed target.
254
+
255
+ Reproduce the frozen input from the toolkit checkout:
256
+
257
+ ```sh
258
+ pnpm evaluate:impact --suite evaluations/dispatch/cases.json --config evaluations/dispatch/context-graph.config.json --ref fa902165a1f212a6d3245c847e1ab9b64a69715b --output .context-graph/evaluation/dispatch/report.json
259
+ ```
260
+
261
+ `--suite` defaults to the original Weftlet cases. Both suite/config paths resolve from the
262
+ toolkit checkout; case paths resolve from the configured evidence root. Evaluator and evidence
263
+ commits are recorded separately, including when evidence is an independent Git repository.
264
+ Only the resolved bundled no-assertion control can refresh; other provider configurations
265
+ must use already-built evidence. Selecting a suite does not authorize a live model run.
266
+
267
+ Both prior saved snapshots were rescored: document-link retrieval remains 24/24 on page
268
+ one with one three-page distraction, and the older control remains 19/24 with one distraction.
269
+ Their cases, reachable counts and snapshot bytes are unchanged. The new measurement adds
270
+ evidence of limits; it does not claim a ranking improvement or broad semantic comprehension.
271
+
272
+ ## Fixed-input relative import correction
273
+
274
+ [ADR 013](ADR-013-relative-module-requests.md) corrects generic request identity without
275
+ changing ranking or guessing resolved implementation files. The two `./policy.js` imports
276
+ now identify `./src/dispatch/policy.js` and `./src/refund/policy.js` separately. Equivalent
277
+ relative spellings requesting the same location share canonical metadata. Bare imports keep
278
+ their existing identities; opaque requests remain importer-local.
279
+
280
+ Implementation `26c47d5b55f9390fe38b4928d1a8274eaf893583` refreshed the same committed inputs,
281
+ using the same configuration, labels, ranking and 60-fact/three-page/depth-32 budget.
282
+ The original records remain frozen; the comparison and cited witnesses are recorded in
283
+ `evaluations/dispatch/relative-imports-baseline.json`.
284
+
285
+ | Input | First-page candidates, before → after | Three-page candidates | Three-page distractions, before → after |
286
+ | --- | ---: | ---: | ---: |
287
+ | Dispatch | 15/16 → 15/16 | 15/16 → 15/16 | 7/7 → 0/7 |
288
+ | Weftlet document links | 24/24 → 24/24 | 24/24 → 24/24 | 1/4 → 1/4 |
289
+
290
+ Neither input returns a labeled distraction on page one. Dispatch reachability falls from
291
+ 214 to 183 facts for each case. Supplemental continuation exhausts 60/60/60/3 facts with no
292
+ refund/audit distraction, while the archival consumer remains disconnected. Its six local
293
+ facts are still present; correcting module identity does not establish topic-string routing.
294
+ The missing `docs/policy.md` reference still has a diagnostic. Weftlet reachability remains
295
+ 3,648 facts with 180 returned per case; telemetry remains a later-page distraction.
296
+
297
+ Hydration preserves fact counts (227 dispatch; 3,648 Weftlet) and all SCIP/document fact
298
+ fields except the containing snapshot ID. It reuses three and six semantic control artifacts
299
+ respectively, with zero extraction/model calls. A repeated dispatch refresh preserves snapshot
300
+ identity/content and reuses all three again. All three legacy saved graphs retain their byte
301
+ hashes, original scores and reachability when queried without hydration.
302
+
303
+ New graph-v4/file-v2 identities keep corrected evidence distinct from old snapshots; draft
304
+ overlay publication also binds the extractor version. Explicitly refresh to obtain the new
305
+ facts. Saved queries do not silently repair old evidence. These synthetic/self-repository
306
+ cases measure candidate retrieval and labeled distractions, not exhaustive precision,
307
+ operational delivery, approved intent or safety of the current diff.
308
+
309
+ ## Explicit routing source comparison
310
+
311
+ [ADR 014](ADR-014-routing-declarations.md) adds opt-in committed mappings with exact producer
312
+ and consumer paths. The declaration, configuration and protocol were committed at
313
+ `e5510479c0545e38a41a902b65839529a3e66e73` before measurement. The original 27 inputs, cases,
314
+ ranking and budgets remain frozen. This new source was deliberately authored; it is not a
315
+ discovered deployment configuration. See `evaluations/dispatch/routing-protocol.md` and
316
+ `evaluations/dispatch/routing-baseline.json` for the protocol and public-safe record.
317
+
318
+ Implementation `5b8d49679f48837fe2ea465940b8c852e543cf5d` compares the original configuration
319
+ with routing disabled and the new configuration with one mapping enabled, both at that new
320
+ input SHA. The mapping names the executor and archival consumer with `dispatch.completed.v1`
321
+ in `example/dispatch` scope. Its binding identity includes repository, scope and declaration
322
+ ID; channel text is not a shared join key. Both added facts are inferred/deterministic and
323
+ cite the committed mapping, without approval or observed-delivery claims.
324
+
325
+ | Scenario | First page, control → enabled | Three pages, control → enabled | Enabled distractions |
326
+ | --- | ---: | ---: | ---: |
327
+ | Upstream request contract | 6/6 → 6/6 | 6/6 → 6/6 | 0/2 |
328
+ | Asynchronous submission consumers | 5/6 → 6/6 | 5/6 → 6/6 | 0/2 |
329
+ | Same-name domain policy | 4/4 → 4/4 | 4/4 → 4/4 | 0/3 |
330
+ | Total | 15/16 → 16/16 | 15/16 → 16/16 | 0/7 |
331
+
332
+ Both configurations have zero labeled distractions at both budgets. Enabled hydration adds
333
+ two routing facts to the same 227 facts; all other fields match apart from containing snapshot
334
+ ID. Reachability increases from 183 to 198 facts per case. The normal three-page result still
335
+ stops at 180 with continuation. Supplemental full continuation exhausts 60/60/60/18 unique
336
+ facts, including all six archival implementation facts and seven archival test facts. Three
337
+ implementation facts and one test fact appear on page one. Their retrieval does not prove
338
+ the tests ran, the payloads agree, or a message was delivered.
339
+
340
+ Direct SPARQL finds the cited pair; RDF projection is field-equivalent and SHACL conforms.
341
+ Existing diagnostics, SCIP digest and semantic artifact keys remain unchanged. Both refreshes
342
+ reuse three control artifacts with zero extraction/model calls, and repeated enabled hydration
343
+ retains identity/content. Five historical snapshot byte hashes are preserved; saved-only
344
+ rescoring of the latest dispatch/Weftlet snapshots preserves their original scores/reachability.
345
+
346
+ The no-routing control still misses the archival consumer. The improvement demonstrates
347
+ extraction and cross-source use of an explicit map, not automatic operational discovery or
348
+ broader semantic quality. Review real mappings and validate actual deployment/runtime evidence
349
+ separately before treating these candidate relationships as stronger evidence.
350
+
351
+ ## Next bounded increments
352
+
353
+ 1. Evaluate actual deployment configuration or runtime routing evidence as a separate source.
354
+ Preserve the frozen no-routing control and distinguish authored declarations, configured
355
+ subscriptions and observed delivery; matching topic text alone must not bridge them.
356
+ 2. Review the explicit mappings with a consumer owner, then compare a bounded Codex extraction
357
+ against the control when ready. Preserve inferred/unreviewed assertion labels.
358
+
359
+ The six-stage public fixture continues testing extraction/linkage, queues, missing links,
360
+ ambiguous intent and inconsistent identities. The evaluator's unit tests separately check
361
+ measurement correctness, including truncation, duplicate results and stale expectation paths.
362
+ Neither set substitutes for a real semantic-quality evaluation. Browser interaction, live
363
+ authentication, fresh-cloud restoration and larger-graph performance remain separate work.
package/docs/PROVIDERS.md CHANGED
@@ -4,6 +4,12 @@ Codex CLI is the shipped model adapter. The module extension contract is tested
4
4
  providers; no additional CLI or API integration is advertised as working. An arbitrary executable
5
5
  is not compatible merely because it accepts stdin.
6
6
 
7
+ For contributors, [provider dispatch](../src/extraction/providers.ts) enforces this contract.
8
+ [Codex discovery](../src/extraction/codex-command.ts), [authentication](../src/extraction/codex-auth.ts)
9
+ and [semantic hydration](../src/extraction/semantic.ts) implement the shipped adapter;
10
+ [setup diagnostics](../src/interfaces/doctor.ts) probe its readiness. These source references
11
+ identify the implementation, not an approval of every extracted assertion or a live test result.
12
+
7
13
  `SemanticExtractor` has a stable `identity` and
8
14
  `extract(source, context, {signal, timeoutMs}) -> {parsed, rawModelOutput}`.
9
15
  The exported `SemanticOutput`, `semanticOutputSchema` and `assertSemanticOutput` define structured
@@ -114,8 +120,30 @@ HDesk subsequently reported a live end-to-end pass at commit
114
120
  `gpt-6-astra`/`medium`, using the unchanged adapter and public example. Citation/schema
115
121
  validation, real SCIP, RDF/SHACL, snapshot publication/querying and repeat artifact reuse passed.
116
122
  See [validation](VALIDATION.md#hdesk-validation-of-6710e9391a6a01345dab5f5be4ae0656e1c9d1ae)
117
- for counts and scope. Native Windows npm Codex executable discovery still needs improvement;
118
- this public-example pass does not establish broader semantic quality.
123
+ for counts and scope. The public-example pass does not establish broader semantic quality. Windows launcher coverage
124
+ is recorded separately below.
119
125
 
120
126
  Provider-dependent tests must remain opt-in, use tiny public fixtures, avoid credentials in
121
127
  untrusted CI, and record model/version/date separately from deterministic test results.
128
+
129
+ ## Windows launchers
130
+
131
+ Diagnostics, login checks and extraction share Codex discovery. On Windows, each nonempty PATH
132
+ directory is searched in order for `codex.exe`, then a standard npm `codex.cmd` launcher. Supported
133
+ npm layouts are a global prefix with `node_modules/@openai/codex` and a project-local
134
+ `node_modules/.bin`. The resolver verifies the package name, JavaScript bin entry and npm shim
135
+ reference, then runs that entry with the current Node executable. It never invokes cmd.exe or
136
+ PowerShell, so spaces and shell metacharacters stay literal. Arbitrary wrappers and alternative
137
+ package-manager layouts are not advertised as supported. An unsupported earlier launcher fails
138
+ instead of silently selecting a different installation later on PATH.
139
+
140
+ Discovery is bounded to five seconds (or the shorter requested timeout) and honors cancellation.
141
+ Extraction cancellation/timeout terminates the launcher process tree with Windows taskkill;
142
+ on POSIX the existing process-group termination remains in use. Authentication selection, secret
143
+ filtering, source isolation and semantic cache identities are unchanged. Saved-evidence queries
144
+ still do not discover or start Codex.
145
+
146
+ Deterministic tests install a public synthetic CLI with npm locally and globally, then exercise
147
+ doctor, login probing and isolated extraction without contacting a model or using real credentials.
148
+ These prove launcher plumbing, not live Codex authentication or semantic quality. Browser and live
149
+ authentication follow-up remain deferred at the maintainer's request.
@@ -0,0 +1,221 @@
1
+ # Consumer hardening and the 1.0 release plan
2
+
3
+ Status: maintainer-directed work, recorded on 2026-10-26 from a consumer integration report.
4
+ The starting release is `@weftlet/toolkit@0.2.0`. **Ship a patch for T01 first.** The remaining
5
+ items improve setup, reuse, diagnostics and document coverage; they do not block that patch
6
+ or justify expanding the consumer's migration. An official 1.0 follows this work and genuine
7
+ consumer acceptance, with its exact release scope reviewed when the evidence is available.
8
+
9
+ This is the near-term release task list. The [Weftlet 2.0 product plan](WEFTLET-2.0-PLAN.md)
10
+ remains a separate future direction for generation, approvals, a companion website and hosted
11
+ AI. None of those platform features is a dependency of the fixes here.
12
+
13
+ ## Reported evidence and boundaries
14
+
15
+ The consumer reported a cold extraction in which two source documents referred to the same
16
+ repository Markdown document. Semantic assertions gave that one repository-path identity
17
+ different node kinds, `Component` and `Document`. An isolated, unchanged two-fact graph failed
18
+ the public RDF validator's single-kind constraint. The references describe the same document;
19
+ this is an extraction identity conflict, not evidence of conflicting product intent.
20
+
21
+ The consumer stopped its larger hydration after independently proving that failure. No full-run
22
+ SHACL result or published snapshot is claimed. Several minutes spent in the larger validation
23
+ phase is a separate observation; the two-fact reproduction does not diagnose its latency.
24
+ Fresh-install checks and the package/adapter boundary were reported as successful.
25
+
26
+ Private source paths, assertions, artifacts and application data remain with the consumer.
27
+ Upstream work uses public-safe reproductions and preserves the original application's existing
28
+ integration. Consumer acceptance is reported separately from upstream deterministic tests.
29
+
30
+ ## Task list and order
31
+
32
+ | ID | Task | Priority/status | Dependencies and release boundary |
33
+ | --- | --- | --- | --- |
34
+ | **T01** | Coherent repository-document identity and kind | **Adoption blocker; patch work in progress** | First isolated patch; preserve SHACL and all assertions |
35
+ | **T02** | No-model hydration/setup preview | Planned usability work | Shares selection/cache identity with hydration; extend it with T07 traversal coverage |
36
+ | **T03** | Artifact reuse across operational configuration changes | Planned efficiency work | Preserve T01's semantic identity separation; keep snapshot/config identity distinct from reusable artifact identity |
37
+ | **T04** | Actionable, privacy-safe timeout diagnostics | Planned diagnostics work | Use stable source identifiers and run progress; document recovery with T02/T03 |
38
+ | **T05** | Early identity checks and validation progress/profiling | Planned correctness/observability work | Build on T01; profile latency independently before selecting optimizations |
39
+ | **T06** | Packaged release-documentation consistency | Planned release tooling work | Check the actual release archives; does not require a runtime architecture change |
40
+ | **T07** | Coverage-aware traversal from SOT/roadmap entry points | Planned bounded feature work | Use canonical identity from T01, preview from T02 and incremental reuse from T03; split selection/classification/extraction changes |
41
+
42
+ T02–T07 can ship in independently useful increments after the blocker patch. Do not combine
43
+ all seven into one diff or delay T01 for the optional platform work. Normal required release
44
+ checks still apply to every shipped increment.
45
+
46
+ ## T01 — Resolve repository-document identity conflicts
47
+
48
+ Define and enforce a coherent ID/kind policy for repository document references across semantic
49
+ and deterministic extraction. A prompt instruction alone is insufficient when separately valid
50
+ model outputs can conflict after assembly. Preserve both source assertions and their citations,
51
+ source authority and assertion-review state. Retain single-kind SHACL validation; do not discard
52
+ an assertion or relax the shape to conceal the conflict.
53
+
54
+ Acceptance:
55
+
56
+ - A public regression has two committed documents referring to the same Markdown document,
57
+ with one provider assertion labeling it `Component` and the other `Document` under the same
58
+ repository-path identity. Both assertions survive under the coherent policy and align with
59
+ deterministic document links; the assembled graph passes RDF/SHACL validation.
60
+ - Verify subject and object references, preserved provenance and unreviewed/inferred assertion
61
+ authority. Other incompatible identities remain diagnosable rather than silently merged away.
62
+ - Version the relevant extractor/prompt/cache identity so incompatible old artifacts cannot
63
+ satisfy a fresh extraction. Test a fresh build, subsequent artifact reuse and unchanged
64
+ snapshot identity/content on repeat input. Keep historical saved snapshots readable and unchanged.
65
+ - Run the normal deterministic/publication gate and a fresh installed-package smoke. Publish
66
+ the patch, then request a genuine consumer re-run against the recorded inputs; do not infer
67
+ consumer acceptance from the upstream fixture.
68
+
69
+ Completion evidence will link the implementation PR, tested source commit, validation record,
70
+ published patch and subsequent consumer result. No patch publication or consumer pass is claimed
71
+ by this task list yet.
72
+
73
+ ## T02 — Preview scope and work without calling a model
74
+
75
+ Add a no-model preview command, tentatively `hydrate --plan`; the CLI spelling is not implemented
76
+ by this plan. Use the same source-selection and identity logic as hydration to avoid preview drift.
77
+ Report selected sources and sizes, inclusion reasons, role/approval classification, expected
78
+ artifact hits/misses and extraction count, plus document/byte limits and any violations.
79
+
80
+ Acceptance:
81
+
82
+ - Preview calls no provider and performs no extraction, indexing, snapshot publication or active
83
+ checkout mutation. Distinguish unavailable/corrupt cache entries from known reusable artifacts.
84
+ - Identify the commit, configuration and selection assumptions; a later changed input can
85
+ invalidate the estimate. Report uncertainty explicitly instead of promising a cache hit.
86
+ - Test empty/no-document scope, cold and warm artifacts, missing sources, budget violations and
87
+ semantic-identity changes. Provide machine-readable output and a concise human summary.
88
+ - With T07, show complete inclusion/exclusion reasons, unresolved links, ownership/classification
89
+ gaps, coverage and budget effects before any model call. Do not silently trim the selection.
90
+
91
+ ## T03 — Reuse valid artifacts after operational changes
92
+
93
+ The consumer reported that changing only a timeout selected a different cache profile and made
94
+ completed extractions unavailable for reuse. Investigate both storage-location selection and
95
+ artifact key construction. Separate operational controls from semantic-output identity, while
96
+ retaining complete configuration metadata on snapshots and queries.
97
+
98
+ Acceptance:
99
+
100
+ - Changing supported operational settings such as timeout does not hide or invalidate an
101
+ otherwise reusable extraction. Explicitly classify each setting rather than assuming all
102
+ non-source changes are harmless; retain provider-specific extension behavior.
103
+ - Changing source bytes/identity, model or relevant model settings, prompt/schema/extractor/
104
+ ontology identity, or other output-affecting policy invalidates reuse as required. T01's new
105
+ identity must not accidentally reuse old conflicting artifacts.
106
+ - Preserve force-extraction semantics, concurrent-writer safety, isolation between consumers,
107
+ credentials and private cache contents. Record reuse and reasons for a miss without secrets.
108
+ - Test interrupted work followed by a changed timeout: valid completed artifacts are reused,
109
+ incomplete/invalid ones are re-extracted, and source provenance remains correct.
110
+
111
+ ## T04 — Explain timeouts and recovery
112
+
113
+ Report a privacy-safe source identifier, phase, elapsed time, configured limit and completed/
114
+ remaining counts when extraction times out. Explain which completed artifacts can be retained
115
+ and how to inspect the plan or adjust a bound before retrying. The documented two-minute default
116
+ was insufficient for the reported workload; that observation alone does not select a new default.
117
+
118
+ Acceptance:
119
+
120
+ - Use a stable opaque identifier by default, with an explicit local mapping or opt-in detail for
121
+ identifying the source. Avoid leaking source text, full prompts, credentials or provider stderr.
122
+ - Distinguish extraction timeout, cancellation, provider failure and infrastructure problems.
123
+ Preserve completed artifacts and the last valid published snapshot.
124
+ - Test timeout/cancellation during a multi-source run, accurate progress counts, safe recovery
125
+ instructions and secret-free logs. MCP protocol stdout remains clean.
126
+
127
+ ## T05 — Detect inexpensive conflicts early and explain validation work
128
+
129
+ Add inexpensive graph-assembly checks for incompatible identities before expensive validation.
130
+ Keep the full conformance requirement. Separately instrument and profile large RDF/SHACL
131
+ validation to determine where time is spent before choosing optimizations.
132
+
133
+ Acceptance:
134
+
135
+ - Early errors identify the conflicting identity/kinds and relevant evidence using the configured
136
+ privacy policy. Preserve all underlying assertions for diagnosis; early checks are not a
137
+ substitute for RDF/SHACL validation.
138
+ - Report elapsed time and honest phase/heartbeat progress for long validation. Do not display a
139
+ fabricated completion percentage when the validator exposes no meaningful denominator.
140
+ - Record reproducible public-fixture sizes, fact/node counts, runtime versions, wall time and
141
+ resource observations. Separate measured bottlenecks from hypotheses.
142
+ - Any optimized validation/projection path has equivalence tests, including negative cases;
143
+ failure must preserve the prior snapshot. Keep model-independent profiling separate from live
144
+ extraction performance and from the reported two-fact correctness defect.
145
+
146
+ ## T06 — Keep packaged documentation consistent with the release
147
+
148
+ The consumer installed 0.2.0 while its packaged README still named 0.1.2 and some archive examples
149
+ used 0.1.1. Source documentation has since been updated, but immutable published archives retain
150
+ their original text. Add a repeatable release-documentation check rather than relying on a
151
+ post-publication source edit.
152
+
153
+ Acceptance:
154
+
155
+ - Distinguish current setup/version statements and executable installation examples from valid
156
+ historical release evidence. Do not rewrite historical versions indiscriminately.
157
+ - Validate current instructions against the actual prepared core/toolkit versions and tarball
158
+ contents before publication. Test both mismatches and intentionally historical references.
159
+ - Define how versioning updates or generates current-release documentation so normal Changesets
160
+ bumps cannot silently leave it stale. Verify installation examples from fresh packed output.
161
+
162
+ ## T07 — Traverse canonical documents with explicit coverage and authority
163
+
164
+ Start from configured SOT and master-roadmap entry points and recursively follow supported
165
+ canonical Markdown links to owner/task documents. Resolve against the chosen repository commit,
166
+ with cycle detection, canonical identities and deduplication. Report missing, ambiguous, outside-
167
+ scope or unsupported links instead of inventing targets or automatically fetching external URLs.
168
+
169
+ Extract requirements, business rules, constraints and cross-feature dependencies while retaining
170
+ source citations and explicit ownership. Reaching a document through a link does not approve it.
171
+ Routine implementation checklists and inferred plans cannot become approved intent through
172
+ traversal. Classify sources/claims explicitly; flag rules without clear SOT ownership and retain
173
+ their inferred/unreviewed state. Validation documents are excluded from product-intent extraction
174
+ by default and retained through a separately classified validation-evidence path.
175
+
176
+ Acceptance:
177
+
178
+ - Split delivery into canonical selection/traversal, role/ownership classification and semantic
179
+ extraction/linkage increments. Keep the existing direct-source configuration usable.
180
+ - The T02 preview exposes entry points, traversed paths, inclusion/exclusion reasons, roles,
181
+ approval/ownership state, unresolved links and exact budget/coverage limits without model calls.
182
+ - Support cycles, shared targets, relative links/fragments and rename/delete cases without
183
+ duplicate extraction. Test cross-feature owner/task references and validation documents,
184
+ including mixed or ambiguous content that needs explicit classification.
185
+ - Compute the necessary selection before scheduling model work. If limits prevent full requested
186
+ coverage, stop or require an explicit narrower scope/budget decision; never silently truncate
187
+ dependencies to fit a document count. Persist incomplete coverage and its reason where relevant.
188
+ - Reuse unchanged extraction artifacts incrementally. Recompute reachability/classification when
189
+ links or configuration change; invalidate affected relationships and preserve removed-source
190
+ history. Partial failures retain completed artifacts and the last valid snapshot.
191
+ - Prove cross-source linkage using extracted public prose and real code relationships, with
192
+ missing links, ambiguous ownership, conflicting claims and false cross-domain matches as
193
+ negative cases. Separate deterministic fixtures from budgeted live-provider evaluation.
194
+
195
+ ## Official 1.0 acceptance
196
+
197
+ After T01's patch is available, work through T02–T07 in bounded increments and record implementation
198
+ PRs and validation evidence here or in the linked validation record. Review the exact 1.0 scope
199
+ against completed evidence; any deferral needs an explicit maintainer decision and must remain
200
+ visible rather than being treated as done. The companion website, hosted AI and broader 2.0
201
+ product plan remain separate.
202
+
203
+ Before calling the consumer migration accepted or publishing 1.0:
204
+
205
+ 1. Confirm the patched identity policy in a genuine consumer hydration, including completed full
206
+ graph validation and snapshot publication, querying and an unchanged repeat. Keep provider
207
+ details, source scope and remaining gaps explicit; the consumer performs its own application work.
208
+ 2. Verify no-model scope preview, budget/coverage reporting, operational cache reuse and recovery
209
+ diagnostics against the implemented task scope. Exercise SOT traversal and authority separation
210
+ with public fixtures and reviewed consumer evidence.
211
+ 3. Pass supported-platform build/tests, fresh packed and public-registry installation, real
212
+ TypeScript indexing, RDF/SHACL, CLI/MCP, provenance and source-inventory checks. Check packaged
213
+ release instructions before publishing the exact reviewed archives.
214
+ 4. Record independently assessed usefulness and semantic-quality limits. Preserve the distinction
215
+ between deterministic checks, live-provider tests, browser checks and private consumer results.
216
+ Existing untested environments or deferred capabilities remain explicitly labeled.
217
+ 5. Review the changelog, compatibility/migration guidance and remaining known issues, then use the
218
+ normal reviewed version PR and trusted-publishing workflow. No 1.0 publication is claimed now.
219
+
220
+ The patch's upstream pass is necessary but does not replace consumer acceptance. Track follow-up
221
+ results without auditing or refactoring the source application's business logic from this repository.