@weftlet/toolkit 0.4.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +149 -14
- package/dist/application/cache-adoption.d.ts +2 -1
- package/dist/application/cache-adoption.js +12 -5
- package/dist/application/document-coverage.d.ts +28 -0
- package/dist/application/document-coverage.js +58 -0
- package/dist/application/document-overlay.d.ts +14 -0
- package/dist/application/document-overlay.js +31 -0
- package/dist/application/extraction-identity.d.ts +2 -1
- package/dist/application/extraction-identity.js +7 -0
- package/dist/application/hydration-inputs.d.ts +15 -0
- package/dist/application/hydration-inputs.js +55 -0
- package/dist/application/hydration-plan.d.ts +7 -0
- package/dist/application/hydration-plan.js +18 -5
- package/dist/application/hydrator.d.ts +2 -0
- package/dist/application/hydrator.js +40 -18
- package/dist/application/overlay.d.ts +3 -1
- package/dist/application/overlay.js +29 -9
- package/dist/application/saved-evidence.d.ts +2 -2
- package/dist/application/semantic-run.d.ts +3 -0
- package/dist/application/semantic-scope.d.ts +1 -1
- package/dist/application/semantic-scope.js +2 -2
- package/dist/configuration/cache.js +13 -1
- package/dist/configuration/config.d.ts +2 -0
- package/dist/configuration/config.js +6 -1
- package/dist/configuration/document-traversal.d.ts +7 -0
- package/dist/configuration/document-traversal.js +13 -0
- package/dist/extraction/canonical-document-links.d.ts +28 -0
- package/dist/extraction/canonical-document-links.js +95 -0
- package/dist/extraction/document-classification.d.ts +44 -0
- package/dist/extraction/document-classification.js +161 -0
- package/dist/extraction/document-evidence.d.ts +21 -0
- package/dist/extraction/document-evidence.js +83 -0
- package/dist/extraction/document-traversal.d.ts +76 -0
- package/dist/extraction/document-traversal.js +210 -0
- package/dist/extraction/document.d.ts +3 -1
- package/dist/extraction/document.js +3 -1
- package/dist/graph/authority.js +3 -0
- package/dist/graph/coverage.d.ts +8 -0
- package/dist/graph/coverage.js +96 -0
- package/dist/graph/document-role-description.d.ts +5 -0
- package/dist/graph/document-role-description.js +18 -0
- package/dist/graph/identity-preflight.d.ts +57 -0
- package/dist/graph/identity-preflight.js +71 -0
- package/dist/graph/query.d.ts +2 -2
- package/dist/graph/query.js +22 -62
- package/dist/graph/rdf.d.ts +4 -1
- package/dist/graph/rdf.js +21 -6
- package/dist/graph/shacl-shapes.js +26 -0
- package/dist/graph/validation-progress.d.ts +14 -0
- package/dist/graph/validation-progress.js +18 -0
- package/dist/index.d.ts +7 -0
- package/dist/index.js +6 -0
- package/dist/interfaces/cli.js +29 -5
- package/dist/interfaces/document-coverage.d.ts +9 -0
- package/dist/interfaces/document-coverage.js +117 -0
- package/dist/repository/git.d.ts +11 -0
- package/dist/repository/git.js +38 -0
- package/dist/visualization/visualization.js +21 -6
- package/docs/ADR-017-shacl-validation-union.md +4 -0
- package/docs/ADR-018-operational-semantic-cache.md +2 -2
- package/docs/ADR-019-semantic-failure-diagnostics.md +2 -2
- package/docs/ADR-020-canonical-physical-file-producers.md +2 -2
- package/docs/ADR-021-identity-preflight-and-validation-timing.md +84 -0
- package/docs/ADR-022-release-documentation.md +44 -0
- package/docs/ADR-023-document-coverage-preview.md +64 -0
- package/docs/ADR-024-implementation-coverage.md +51 -0
- package/docs/ADR-025-document-classification.md +78 -0
- package/docs/ADR-026-traversal-hydration.md +94 -0
- package/docs/ADR-027-validation-closeout.md +155 -0
- package/docs/CONSUMER-TRIAL.md +1 -1
- package/docs/MIGRATING-TO-WEFTLET.md +1 -1
- package/docs/PROVIDERS.md +17 -1
- package/docs/RELEASE-1.0-PLAN.md +177 -48
- package/docs/RELEASING.md +46 -8
- package/docs/ROADMAP.md +59 -28
- package/docs/VALIDATION.md +451 -0
- package/docs/benchmarks/validation-2026-10-28/README.md +25 -0
- package/docs/benchmarks/validation-2026-10-28/large.json +1 -0
- package/docs/benchmarks/validation-2026-10-28/lifecycle.json +1 -0
- package/docs/benchmarks/validation-2026-10-28/native.json +1 -0
- package/docs/benchmarks/validation-2026-10-28/rich.json +1 -0
- package/docs/extraction-manifest.json +73 -17
- package/examples/generic-typescript/document-policy.json +9 -0
- package/package.json +7 -4
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# ADR-024: Report evidence coverage independently of display groups
|
|
2
|
+
|
|
3
|
+
Status: implemented after T07a; validation and publication are recorded separately in
|
|
4
|
+
[validation](VALIDATION.md). This is a query-reporting correction, not T07b document classification.
|
|
5
|
+
|
|
6
|
+
## Problem
|
|
7
|
+
|
|
8
|
+
The consumer reports returned code relationships with zero implementation coverage. Public
|
|
9
|
+
reproductions using the actual contract extractor and real SCIP confirm the defect: a file's
|
|
10
|
+
definition or consumption of a Contract is placed in the contracts display group. Coverage used
|
|
11
|
+
the exclusive display-group counts, so these code relationships could not count as implementation.
|
|
12
|
+
The inverse also occurred: unapproved document/model facts in the fallback implementation group
|
|
13
|
+
could suppress an implementation gap. Page-local absence was described as query-wide absence.
|
|
14
|
+
|
|
15
|
+
## Decision
|
|
16
|
+
|
|
17
|
+
Keep primary display groups, returned facts, ranking, pagination and authority unchanged. Compute
|
|
18
|
+
implementation coverage from the returned evidence independently of display placement. A production
|
|
19
|
+
code relationship can count toward both contract and implementation coverage without appearing twice
|
|
20
|
+
in the returned facts. The other coverage categories retain their existing classification.
|
|
21
|
+
|
|
22
|
+
New reports include `coverage.scope: "returned-page"` and `countsAreNonExclusive: true`.
|
|
23
|
+
Both fields are optional in core for compatibility with older Context Packs. Counts are relationship
|
|
24
|
+
counts, not percentages, totals of display sections, product completeness or proof of correct behavior.
|
|
25
|
+
Markdown and HTML explain overlap. Missing-evidence messages refer to the returned page; continuation
|
|
26
|
+
and traversal bounds can leave further evidence outside it. Existing source diagnostics remain intact.
|
|
27
|
+
|
|
28
|
+
An implementation relationship must have effective `implemented` authority, a non-document,
|
|
29
|
+
non-validation citation, and no unreviewed extraction marker. Current document/model, runtime,
|
|
30
|
+
draft and validation evidence cannot qualify by merely mentioning a code file. Supported document
|
|
31
|
+
extensions and test/spec names use the canonical core file policy. Conventional test/validation
|
|
32
|
+
directories, including `__tests__` and `__mocks__`, are excluded from production coverage.
|
|
33
|
+
|
|
34
|
+
Legacy deterministic adapters that omitted authority retain the existing authority fallback only
|
|
35
|
+
when a canonical physical Component endpoint matches the cited file. An adapter explicitly marking
|
|
36
|
+
a fact `implemented` may also use logical Component/Provider identities with a production citation.
|
|
37
|
+
Unmarked legacy provider output cannot be distinguished from adapter output when their metadata is
|
|
38
|
+
identical; no new claim of provider verification or execution is made. Saved fact authority is never
|
|
39
|
+
rewritten by this reporting classification.
|
|
40
|
+
|
|
41
|
+
## Compatibility and validation
|
|
42
|
+
|
|
43
|
+
No extractor, prompt, semantic artifact, assembly or ranking identity changes. Existing immutable
|
|
44
|
+
snapshots immediately benefit when queried by the updated toolkit; no model run or hydration is
|
|
45
|
+
needed. Historical Context Packs keep their original reports unless queried again. Existing clients
|
|
46
|
+
should inspect the new metadata rather than summing counts as a disjoint total.
|
|
47
|
+
|
|
48
|
+
Regressions use actual contract extraction, real TypeScript SCIP, strict SHACL and immutable
|
|
49
|
+
round-tripped facts. They cover overlapping counts, page continuation, API/MCP/Markdown/HTML
|
|
50
|
+
consistency, document/model/test negatives and retained source diagnostics. Installed-package smoke
|
|
51
|
+
checks query actual extracted facts rather than constructing coverage values by hand.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# ADR-025: Document roles, ownership and deterministic evidence
|
|
2
|
+
|
|
3
|
+
Status: T07b merged in PR 42; local and hosted checks and independent review passed.
|
|
4
|
+
No npm publication or consumer acceptance is claimed. See [validation](VALIDATION.md) for
|
|
5
|
+
completed checks and the [1.0 plan](RELEASE-1.0-PLAN.md) for the remaining boundary.
|
|
6
|
+
|
|
7
|
+
## Decision
|
|
8
|
+
|
|
9
|
+
Classify document sources separately from source approval, assertion review and canonical node
|
|
10
|
+
identity. The public classifier accepts explicit role/owner mappings; the standalone deterministic
|
|
11
|
+
extractor retains document presence and canonical references without semantic artifacts. T07b does
|
|
12
|
+
not register that extractor with hydration or overlays. T07c will connect selection, classification,
|
|
13
|
+
validation-document separation and artifact reuse to execution.
|
|
14
|
+
|
|
15
|
+
`documents plan --document-policy <json-file>` accepts an optional operation-time JSON sidecar,
|
|
16
|
+
resolved from the current working directory and bounded to 65,536 bytes. It is not a hydration
|
|
17
|
+
configuration field. The API equivalent is `DocumentCoverageOptions.documentPolicy`. Documents
|
|
18
|
+
come from the chosen commit; policy and existing configuration are current operation inputs.
|
|
19
|
+
The report records their digests and makes no model/provider request or graph write.
|
|
20
|
+
|
|
21
|
+
## Roles and owner mappings
|
|
22
|
+
|
|
23
|
+
Source roles are `intent`, `plan`, `validation`, `reference` and `unclassified`. Explicit role
|
|
24
|
+
mappings override configured defaults. Without a matching role mapping, evidence roots imply
|
|
25
|
+
`validation`, approved intent sources imply `intent`, and other sources remain `unclassified`.
|
|
26
|
+
Different matching explicit roles produce an ambiguity; rule order never chooses a winner.
|
|
27
|
+
Approval still comes only from `approvedIntentSources`, independently of the selected role.
|
|
28
|
+
|
|
29
|
+
Ownership requires an explicit mapping to a canonical document path. One available, approved
|
|
30
|
+
owner with an unambiguous `intent` role resolves ownership. Multiple distinct owners are
|
|
31
|
+
`ambiguous`; an unread or outside-coverage owner is `unavailable`; an unapproved owner is
|
|
32
|
+
`unapproved`; an approved owner with another or ambiguous role is `ineligible`. With no mapping,
|
|
33
|
+
ownership is `missing` for intent/plan/unclassified sources and `not-required` for validation/reference.
|
|
34
|
+
Approval does not imply self-ownership; a self-owner must also be mapped explicitly.
|
|
35
|
+
|
|
36
|
+
An owner mapping never approves its child document or an extracted assertion, and produces no
|
|
37
|
+
inferred owner edge. Missing/ambiguous roles and ownership are source-level evidence gaps, not
|
|
38
|
+
proven business-rule violations. The normalized policy digest includes role/owner mappings and
|
|
39
|
+
the approval/evidence-root policy; it does not depend on rule order or document availability.
|
|
40
|
+
|
|
41
|
+
## Evidence records and querying
|
|
42
|
+
|
|
43
|
+
Optional per-fact `sourceRole`, `sourceOwnership` and `sourcePolicyDigest` preserve classification
|
|
44
|
+
without attaching conflicting roles to a shared canonical Document node. RDF exposes the role,
|
|
45
|
+
ownership status, owner paths, mapping IDs and policy digest while retaining exact record projection.
|
|
46
|
+
Metadata-targeted SHACL constraints validate new fields; historical facts without them retain
|
|
47
|
+
their existing behavior. Document IDs/kinds and the existing assembly/cache identities stay intact.
|
|
48
|
+
|
|
49
|
+
`extractDocumentEvidence` uses the committed traversal and classifier. For each successfully read
|
|
50
|
+
document it emits a Project `contains` Document fact, including empty documents and documents
|
|
51
|
+
without outgoing links. It emits `references` facts only for resolved canonical file/document
|
|
52
|
+
links, retaining exact declaration/use citations, source digests and commit SHA. Unresolved links
|
|
53
|
+
and incomplete traversal remain explicit reports/diagnostics; no missing relationship is invented.
|
|
54
|
+
The new `document-evidence-extractor:1.0.0` identity and classification metadata enter fact IDs.
|
|
55
|
+
|
|
56
|
+
These structural facts have inferred authority and deterministic extraction review. Their source
|
|
57
|
+
approval remains separate. Presence, a reference or a validation role does not claim that a test
|
|
58
|
+
ran/passed or a product rule was approved. Explicit validation roles use validation coverage;
|
|
59
|
+
plan/reference/unclassified roles cannot establish product intent. Query and viewer details expose
|
|
60
|
+
the same role/ownership metadata; existing facts without roles retain legacy interpretation.
|
|
61
|
+
|
|
62
|
+
## Readiness and delivery boundary
|
|
63
|
+
|
|
64
|
+
The prospective plan's top-level `ready` remains traversal completeness plus budget readiness.
|
|
65
|
+
`classification.complete` independently describes role/ownership gaps and must be inspected
|
|
66
|
+
separately. Neither flag proves complete product coverage, approved assertions or hydration
|
|
67
|
+
readiness. A resolved owner can remain outside a future extraction selection; T07c must enforce
|
|
68
|
+
the execution contract rather than infer it from a preview.
|
|
69
|
+
|
|
70
|
+
Current direct-source hydration, `hydrate --plan`, configuration/cache identity and saved snapshots
|
|
71
|
+
remain unchanged. In particular, existing hydration still semantically extracts selected validation
|
|
72
|
+
documents. T07b supplies the separate evidence path needed before T07c can exclude those documents
|
|
73
|
+
from product-intent extraction without dropping their presence or citations.
|
|
74
|
+
|
|
75
|
+
Validation covers role conflicts and defaults, owner eligibility and non-transfer of authority,
|
|
76
|
+
actual committed presence/references, exact RDF/SHACL projection, query/viewer agreement and
|
|
77
|
+
legacy behavior. Exact gate and review outcomes are in the validation record; neither those
|
|
78
|
+
checks nor this decision claim npm publication or that traversal already drives hydration.
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# ADR 026: Explicit traversal hydration with separate source and assertion authority
|
|
2
|
+
|
|
3
|
+
Status: accepted and merged in PR 44 after local/hosted validation. Publication and consumer
|
|
4
|
+
acceptance remain separate; exact commits and checks are recorded in [validation](VALIDATION.md).
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
T07a follows canonical committed Markdown links; T07b classifies source roles/owners and extracts
|
|
9
|
+
cited document presence/references. Neither changed hydration. Consumers need one reviewed scope
|
|
10
|
+
for preview, extraction, cache recovery and saved queries, including validation evidence that is
|
|
11
|
+
unsuitable for product-intent extraction. A broad source list or a capped agent search must not
|
|
12
|
+
silently replace that scope.
|
|
13
|
+
|
|
14
|
+
## Decision
|
|
15
|
+
|
|
16
|
+
An optional `documentTraversal` object in `context-graph.config.json` activates traversal hydration.
|
|
17
|
+
It accepts exact `entryPoints`, optional document `roots`, bounded `maxReadDocuments`, `maxReadBytes`
|
|
18
|
+
and `maxDocumentBytes`, and an inline T07b `policy`. Unknown controls are rejected. It replaces the
|
|
19
|
+
legacy `semanticSources` list for this mode. Existing direct configurations keep their behavior and
|
|
20
|
+
identity. Combining traversal with `agentContext`, or a programmatic `semanticSources` override,
|
|
21
|
+
is rejected rather than silently narrowing or broadening scope. Requests may still filter saved
|
|
22
|
+
impact queries; they do not change the traversal hydration selection.
|
|
23
|
+
|
|
24
|
+
`selectHydrationInputs` is shared by hydration, `hydrate --plan`, explicit cache adoption and draft
|
|
25
|
+
selection. Committed Git objects provide the closure despite working changes. Every discovered
|
|
26
|
+
source is reported; canonical cycles/shared targets extract once. Missing/unsupported/ambiguous
|
|
27
|
+
links, unread coverage, ambiguous source roles and semantic budget errors stop new model work
|
|
28
|
+
before ontology readiness, indexing or extraction. Existing completed artifacts and snapshot heads
|
|
29
|
+
remain intact. The no-model plan retains the full report on failure. Ordinary hydration points to
|
|
30
|
+
that local report without printing document paths/content in its new selection error.
|
|
31
|
+
|
|
32
|
+
Validation roles default from `evidenceRoots`, with explicit role mappings taking precedence. They
|
|
33
|
+
receive deterministic presence/reference evidence, including empty reports, and no model calls.
|
|
34
|
+
Semantic budgets count the remaining eligible sources; separate traversal read limits bound all
|
|
35
|
+
read sources. The nested traversal report's comparison ceilings are the maximum supported read
|
|
36
|
+
values, not a paid-work estimate; the plan's top-level `budgets` describes actual extraction limits.
|
|
37
|
+
An explicit non-validation override is a maintainer policy choice, never model inference.
|
|
38
|
+
|
|
39
|
+
Intent, plan, reference and unclassified sources may be extracted. Unresolved/ambiguous owner
|
|
40
|
+
mappings remain visible diagnostics, not newly invented hard CI rules. A valid owner must itself
|
|
41
|
+
be reached, approved and unambiguously intent. Being linked to an owner never approves a child,
|
|
42
|
+
and source approval never approves a model assertion. Policy role describes source context, not
|
|
43
|
+
a guarantee that every sentence is a business requirement. Model assertions stay inferred and
|
|
44
|
+
unreviewed; structural facts stay deterministic and inferred. Neither a written “PASS” nor the
|
|
45
|
+
presence of a test link asserts test execution.
|
|
46
|
+
|
|
47
|
+
The canonical document extractor owns Markdown references in this mode. The legacy linker still
|
|
48
|
+
emits assertion containment and supported bare/inline-code references. Assembly attaches role,
|
|
49
|
+
source authority, ownership and policy digest without modifying immutable provider artifacts.
|
|
50
|
+
Strict identity checks and SHACL remain required; assertions and contradictory evidence are not
|
|
51
|
+
silently removed to pass validation.
|
|
52
|
+
|
|
53
|
+
## Identity, reuse and overlays
|
|
54
|
+
|
|
55
|
+
Snapshots bind the full configuration, closure/classification digest and `document-selection-v1`
|
|
56
|
+
plus `document-evidence-extractor:1.0.0`. Core adds an optional `DocumentSelectionRecord`; saved
|
|
57
|
+
JSON, Markdown and HTML expose coverage separately from ownership and assertion approval. Older
|
|
58
|
+
snapshots remain queryable and cannot satisfy current-config/extractor/commit checks.
|
|
59
|
+
|
|
60
|
+
The unchanged built-in Codex adapter receives one source without traversal/owner metadata.
|
|
61
|
+
Its source/provider/prompt/version/confidence key remains intact. Its user-cache partition ignores
|
|
62
|
+
only the validated `documentTraversal` field in addition to the five existing operational semantic
|
|
63
|
+
controls. Enabling traversal or changing role/owner/read limits can reuse unchanged artifacts;
|
|
64
|
+
source selection changes and source/model/version/approval changes retain existing conservative
|
|
65
|
+
identity behavior. Snapshot profiles always retain the entire configuration. Explicit legacy
|
|
66
|
+
cache adoption keeps conflict checks and never scans unrelated profiles.
|
|
67
|
+
|
|
68
|
+
Custom modules are trusted code with an explicit versioned provider contract. Traversal modules
|
|
69
|
+
get a separate full-config `document-semantic-v1` artifact namespace even in workspace storage.
|
|
70
|
+
Direct-mode artifacts and changed configurations cannot be silently reused there. Identical
|
|
71
|
+
configuration and unchanged source identities reuse; providers with additional mutable inputs
|
|
72
|
+
must version their declared contract or force extraction. No arbitrary context-dependent module
|
|
73
|
+
is promised automatic dependency discovery.
|
|
74
|
+
|
|
75
|
+
Draft overlays recompute the closure from the isolated base plus captured bytes. They rebuild
|
|
76
|
+
presence/references with draft authority, invalidate changed/deleted/unreachable sources and
|
|
77
|
+
assertions whose owner changed, and retain only matching source/policy/ownership evidence. An
|
|
78
|
+
incomplete closure invalidates all retained document assertions and is labeled incomplete; it can
|
|
79
|
+
still be queried as a draft. No model runs. Temporary Git timestamps are excluded from draft
|
|
80
|
+
selection identity; the base SHA and capture digest bind it instead. Committed heads are unchanged.
|
|
81
|
+
|
|
82
|
+
## Validation and limits
|
|
83
|
+
|
|
84
|
+
Public fixtures compose actual committed traversal, unchanged Codex schema/citation handling with
|
|
85
|
+
an offline prose parser, real SCIP and strict RDF/SHACL. They cover the six-stage asynchronous
|
|
86
|
+
pipeline, warm reuse, configuration/cache isolation, source authority, rename/delete/reachability,
|
|
87
|
+
owner edits, draft gaps and saved-history freshness. CLI preview uses unavailable infrastructure
|
|
88
|
+
sentinels; installed consumer smoke uses the versioned module contract and real indexing.
|
|
89
|
+
These tests do not claim live model quality, browser interaction or TradePilot acceptance.
|
|
90
|
+
|
|
91
|
+
`documents plan --document-policy` remains a separate prospective operation, including its own
|
|
92
|
+
all-document budget comparison. Use `hydrate --plan` with the execution configuration to predict
|
|
93
|
+
actual extraction and reuse. No documentation generation/backfill, ownership inference, approval
|
|
94
|
+
UI, remote source adapter or model orchestration integration is added by this decision.
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
# ADR 027: Retain full validation after bounded lifecycle and native evaluation
|
|
2
|
+
|
|
3
|
+
Status: accepted engineering decision for the 1.0 candidate, 2026-10-28. Private consumer
|
|
4
|
+
performance acceptance and official 1.0 remain separate gates. This increment changes development
|
|
5
|
+
benchmarks, tests and documentation; it does not change the runtime validator or publish a release.
|
|
6
|
+
|
|
7
|
+
## Decision
|
|
8
|
+
|
|
9
|
+
Keep complete, direct `rdf-validate-shacl` validation over the full SPO union, with the unchanged
|
|
10
|
+
shapes and supplemental citation-order check. Preserve the canonical named graph, provenance,
|
|
11
|
+
authority and prior published snapshots. Saved-snapshot queries remain independent of rebuilding.
|
|
12
|
+
|
|
13
|
+
Do not add a mandatory worker, validation receipt cache, incremental validator, Rust binary or
|
|
14
|
+
graph database for 1.0. The evaluations below support this choice; these capabilities are not
|
|
15
|
+
delivered or silently assumed. Reconsider when a measured consumer workload demonstrates a need.
|
|
16
|
+
Memory remains a material limitation, and these results do not establish a maximum supported
|
|
17
|
+
repository size or diagnose the private consumer's earlier 30-minute generation timeout.
|
|
18
|
+
|
|
19
|
+
The cold-path change in [ADR 017](ADR-017-shacl-validation-union.md) and identity/phase diagnostics
|
|
20
|
+
in [ADR 021](ADR-021-identity-preflight-and-validation-timing.md) remain the production improvements.
|
|
21
|
+
This decision closes their remaining engineering evaluation items without changing those
|
|
22
|
+
historical measurements. It does not waive genuine consumer hydration/repeat acceptance.
|
|
23
|
+
|
|
24
|
+
## Measurements and reproducibility
|
|
25
|
+
|
|
26
|
+
Raw, public, metrics-only records are in [the benchmark directory](benchmarks/validation-2026-10-28/README.md).
|
|
27
|
+
They contain versions, source/harness hashes, fixture/fact/node/quad counts, complete case results,
|
|
28
|
+
diagnostic hashes, timings and resource observations. There are no private graphs, model prompts,
|
|
29
|
+
credentials or live provider calls. Runtime code is unchanged from `174f2ae`.
|
|
30
|
+
|
|
31
|
+
Linux x64, Node 24.19.0, pnpm 10.18.2, N3 2.7.2 and rdf-validate-shacl 0.6.5 were used in a
|
|
32
|
+
32 GiB cloud container. Every fixture/mode/sample used a fresh bounded child, three samples per
|
|
33
|
+
case; garbage collection before cases was outside measurement. Repeated cases share their
|
|
34
|
+
session's imports/JIT state. The pipeline uses actual committed prose, the real Codex adapter
|
|
35
|
+
with a deterministic offline runner, real SCIP, cross-source linkage, SHACL and snapshot storage.
|
|
36
|
+
Its six eligible prose artifacts were reused on repeat; this does not test a live model.
|
|
37
|
+
|
|
38
|
+
Cold full-validation medians (union construction plus SHACL and the supplemental check):
|
|
39
|
+
|
|
40
|
+
| Input | Facts / nodes / canonical quads | Direct | Worker | Largest direct process RSS |
|
|
41
|
+
| --- | --- | ---: | ---: | ---: |
|
|
42
|
+
| Extracted six-stage pipeline | 241 / 97 / 4,929 | 167 ms | 159 ms | 308 MiB |
|
|
43
|
+
| Shared identities | 2,000 / 50 / 40,203 | 530 ms | 596 ms | 519 MiB |
|
|
44
|
+
| Unique identities | 2,000 / 4,000 / 56,003 | 620 ms | 696 ms | 554 MiB |
|
|
45
|
+
| Dense source metadata | 2,000 / 50 / 48,203 | 1,445 ms | 1,441 ms | 900 MiB |
|
|
46
|
+
| Unique identities | 10,000 / 20,000 / 280,003 | 2,860 ms | 2,830 ms | 1,676 MiB |
|
|
47
|
+
| Dense source metadata | 10,000 / 50 / 240,203 | 5,668 ms | 5,929 ms | 1,758 MiB |
|
|
48
|
+
|
|
49
|
+
RSS is the largest sampled process high-water mark across the reported lifecycle, not SHACL's
|
|
50
|
+
isolated allocation. It includes imports, fixture, canonical graph, serialization and union;
|
|
51
|
+
worker RSS includes its parent. The largest dense 10,000-fact worker case reached 3,058 MiB.
|
|
52
|
+
Dense fixtures attach source role/policy/ownership metadata to every fact, exercising the new
|
|
53
|
+
ownership disjunction. They are stress fixtures, not a claim about consumer metadata density.
|
|
54
|
+
The large cases are cold-only; the smaller fixtures include repeat and edits. There are 258
|
|
55
|
+
completed direct/worker cases and 18 paired sample groups with exact verdict/data equality.
|
|
56
|
+
|
|
57
|
+
Same-object repeat and rebuilt-identical graphs still execute full validation. Direct medians
|
|
58
|
+
for repeat/rebuild were 110/113 ms (pipeline), 471/481 ms (shared), 602/636 ms (unique), and
|
|
59
|
+
1,336/1,246 ms (dense). Insert, update, delete, rename, negative confidence, shared-identity
|
|
60
|
+
conflict and repair-by-deletion all use complete candidate graphs, retain canonical serialization,
|
|
61
|
+
and produce the expected verdict. This is correctness/performance evidence for full validation,
|
|
62
|
+
not an incremental algorithm. No timing thresholds are CI gates.
|
|
63
|
+
|
|
64
|
+
## Worker evaluation
|
|
65
|
+
|
|
66
|
+
The development worker uses the identical validator, shapes, input and full validation call.
|
|
67
|
+
All compared verdicts and serialized input hashes match direct execution, including failures.
|
|
68
|
+
The parent timer remains responsive: around 21 ms in 10,000-fact cases versus 5.4–8.2 seconds
|
|
69
|
+
for the direct harness operation (construction, serialization and validation combined). This
|
|
70
|
+
timer measures responsiveness, not percentage complete. Worker startup was approximately 0.8
|
|
71
|
+
seconds, structured cloning has a cost, and throughput/memory advantages were not consistent.
|
|
72
|
+
|
|
73
|
+
A bounded prototype terminates the worker after its actual SHACL-started boundary. In the
|
|
74
|
+
recorded runs termination took about 19–22 ms; no verdict was returned or accepted, and there is
|
|
75
|
+
no storage/publication call in this probe. This does not establish production cancellation or
|
|
76
|
+
failure recovery across a new worker boundary. Current runtime progress remains truthful phase
|
|
77
|
+
boundaries; in-phase heartbeat, interruption and worker memory limits would need separate API,
|
|
78
|
+
packaging, cancellation and head-preservation integration. There is no measured need to introduce
|
|
79
|
+
that complexity into this candidate.
|
|
80
|
+
|
|
81
|
+
## Exact successful-result reuse
|
|
82
|
+
|
|
83
|
+
The study computes an experimental key over complete named RDF serialization plus validator,
|
|
84
|
+
shape, authority/progress code hashes, workspace dependency lock and Node version. It never reads
|
|
85
|
+
a cached verdict: every case validates. Identical/rebuilt/repaired input yields the same key;
|
|
86
|
+
edits and a manifest-only timestamp change yield different keys even when conformance is unchanged.
|
|
87
|
+
Serialization plus hashing alone costs roughly 23 ms for the cold pipeline, 137 ms for 2,000
|
|
88
|
+
shared facts, 157 ms for 2,000 unique facts and 125 ms for 2,000 dense facts. Graph construction
|
|
89
|
+
is still required. A manifest timestamp therefore prevents reuse across otherwise equivalent
|
|
90
|
+
hydrations under this conservative key.
|
|
91
|
+
|
|
92
|
+
Do not ship this prototype as a cache. A production receipt needs a reviewed validation dependency
|
|
93
|
+
key, actual resolved engine/dependency identities, complete supplemental-rule inputs, coherent
|
|
94
|
+
immutable input, atomically written and checked successful receipts, and invalidation for every
|
|
95
|
+
shape/policy/semantic change. Excluding manifest fields is a separate correctness decision, not
|
|
96
|
+
an arbitrary optimization. A failed/missing/corrupt receipt must trigger full validation; no
|
|
97
|
+
receipt may satisfy current commit/scope/working-diff freshness requirements on historical data.
|
|
98
|
+
The measured seconds saved do not yet justify that state and trust surface for 1.0.
|
|
99
|
+
|
|
100
|
+
## Dependency-aware incremental evaluation
|
|
101
|
+
|
|
102
|
+
The regression demonstrates a concrete counterexample: validating a changed assertion alone
|
|
103
|
+
passes, while joining it with the unchanged producer of the same node fails `kind` maxCount.
|
|
104
|
+
Revalidating all producers of that identity reproduces the full verdict. Removing the conflicting
|
|
105
|
+
old producer repairs the graph without adding a fact. Renames require both old and new identities;
|
|
106
|
+
source-role/ownership checks also depend on all contributing assertions at a fact identity.
|
|
107
|
+
|
|
108
|
+
A safe implementation would need an index of every shape target and dependency, old/new affected
|
|
109
|
+
subjects and property values, removed targets, identities and incoming/outgoing relationships,
|
|
110
|
+
plus invalidation of prior violations. Shape/validator/policy changes and unbounded/global rules
|
|
111
|
+
require full fallback. There is no such reviewed dependency index in the toolkit. Retain full
|
|
112
|
+
validation for insertions, updates, deletions and renames; do not call a changed-facts pass a valid
|
|
113
|
+
whole-graph verdict. No previous violation is baselined away by this decision.
|
|
114
|
+
|
|
115
|
+
## Native candidate
|
|
116
|
+
|
|
117
|
+
Material memory use justified a bounded evaluation of **Rudof 0.3.35**, an Apache-2.0 Rust CLI.
|
|
118
|
+
The official Linux GNU asset was downloaded outside the repository and verified against its
|
|
119
|
+
release SHA-256. No binary, dependency or license requirement is added to the product.
|
|
120
|
+
|
|
121
|
+
The probe exports every SPO triple, passes the exact current shapes, and applies the same JS
|
|
122
|
+
citation-order supplement (which is not itself a SHACL shape). All 29 public cases agree on
|
|
123
|
+
conformance and diagnostic locations. Exact diagnostic multiplicity agrees on 27: for non-integer
|
|
124
|
+
start/end lines Rudof emits two results at the failing property where the current validator emits
|
|
125
|
+
one. Neither validator accepts those inputs. The raw report retains both diagnostics; no shape,
|
|
126
|
+
failure or assertion is suppressed. The probe exits 1 to make this exact-parity difference visible.
|
|
127
|
+
|
|
128
|
+
Each timing sample starts a fresh native process, with a 30-second whole-process-group deadline:
|
|
129
|
+
|
|
130
|
+
| Input | Native parse/startup/validation median | Node export preparation, measured once | Largest native-only RSS |
|
|
131
|
+
| --- | ---: | ---: | ---: |
|
|
132
|
+
| 2,000 unique facts | 1,049 ms | 711 ms | 488 MiB |
|
|
133
|
+
| 10,000 unique facts | 4,631 ms | 4,377 ms | 2,264 MiB |
|
|
134
|
+
| 2,000 dense facts | 968 ms | 833 ms | 552 MiB |
|
|
135
|
+
| 10,000 dense facts | 5,217 ms | 3,318 ms | 2,534 MiB |
|
|
136
|
+
|
|
137
|
+
Native time includes the Python measurement wrapper, process startup and RDF parsing, unlike the
|
|
138
|
+
JS validation-only column. Its RSS excludes the Node graph/export process; these columns are not
|
|
139
|
+
engine-only apples-to-apples comparisons. Native dense SHACL invocation is competitive, but this
|
|
140
|
+
file-transfer integration provides no demonstrated overall latency or memory win. An embedded
|
|
141
|
+
binding might behave differently; this experiment does not measure one or generalize to all Rust
|
|
142
|
+
engines. Native Windows/macOS installation, release asset delivery, supply-chain verification,
|
|
143
|
+
schema evolution and diagnostic compatibility would also need support. Do not make it mandatory
|
|
144
|
+
or add an optional backend that cannot yet be maintained and validated across supported installs.
|
|
145
|
+
No graph database was benchmarked: there is no demonstrated persistent-service need, and adding
|
|
146
|
+
one would change the toolkit's installation and operational boundary.
|
|
147
|
+
|
|
148
|
+
## Acceptance still required
|
|
149
|
+
|
|
150
|
+
Retain full upstream positive/negative/provenance/packaged-installation gates. Publish the reviewed
|
|
151
|
+
minor candidate before the consumer trial. Record genuine cold and repeated hydration, successful
|
|
152
|
+
full SHACL and snapshot publication, semantic reuse, observed resource/phase timing, saved querying
|
|
153
|
+
and the approved ten-minute provider retry in TradePilot. These public measurements cannot supply
|
|
154
|
+
that acceptance. Browser/live-auth work remains separately tentative as previously agreed; official
|
|
155
|
+
1.0 requires the consumer result and release review.
|
package/docs/CONSUMER-TRIAL.md
CHANGED
|
@@ -6,7 +6,7 @@ development workspace. Saved evidence queries work without a model login. Hydrat
|
|
|
6
6
|
prose uses the configured provider; an OAuth failure may block new extraction but does not
|
|
7
7
|
remove an already published snapshot.
|
|
8
8
|
|
|
9
|
-
Use a pinned, reviewed toolkit version and its matching documentation.
|
|
9
|
+
Use a pinned, reviewed toolkit version and its matching documentation. Release `0.2.0` introduced
|
|
10
10
|
explicit routing declarations, committed TypeScript source projects, document linkage and
|
|
11
11
|
the improved impact ranking. To evaluate a newer, unreleased commit, use both locally prepared
|
|
12
12
|
package archives from that same reviewed commit; see [release preparation](RELEASING.md).
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Migrating from Tently to Weftlet
|
|
2
2
|
|
|
3
3
|
The project is now Weftlet. LaunchEcho remains the maintainer; the GitHub repository remains
|
|
4
|
-
[jchewitt/weftlet](https://github.com/jchewitt/weftlet). Both new package names
|
|
4
|
+
[jchewitt/weftlet](https://github.com/jchewitt/weftlet). Both new package names were first published
|
|
5
5
|
at 0.1.0; see [release status](RELEASING.md) for subsequent releases.
|
|
6
6
|
|
|
7
7
|
| Previous package / command | Replacement |
|
package/docs/PROVIDERS.md
CHANGED
|
@@ -87,6 +87,13 @@ Codex attempt (or module execution), no retries, and the configured concurrency
|
|
|
87
87
|
before any model call. `maxDocuments`, `maxSourceBytes`, `timeoutMs`, and `retries` may be configured
|
|
88
88
|
within validated limits. Review scope and cost before raising them; these byte/document limits
|
|
89
89
|
are not dollar or token estimates. Ask the operator before substantial paid runs.
|
|
90
|
+
Starting with 0.5.0, `semantic.timeoutMs` accepts integer values from 1 through **600000**
|
|
91
|
+
(ten minutes); 0.4.0 and earlier reject values above 300000. The default remains 120000.
|
|
92
|
+
For the approved ten-minute retry, set `"timeoutMs": 600000` inside the existing `semantic`
|
|
93
|
+
configuration, preview reuse with `hydrate --plan`, then rerun hydration without forcing
|
|
94
|
+
re-extraction. The limit applies to each Codex attempt, or the whole module execution, rather
|
|
95
|
+
than the entire hydration. Codex authentication/discovery checks retain their separate five-second
|
|
96
|
+
bounds. This setting does not raise a CI job's overall timeout or change retry policy.
|
|
90
97
|
Only explicit transient exit status 75 is retryable, at most twice; authentication, invalid
|
|
91
98
|
output, cancellation and ambiguous failures are not retried. Ctrl-C cancels live CLI extraction.
|
|
92
99
|
Indexing cancellation is checked between phases, not inside the indexer process.
|
|
@@ -128,7 +135,7 @@ snapshot profiles. Changing `timeoutMs`, `retries`, `concurrency`, `maxDocuments
|
|
|
128
135
|
`maxSourceBytes` can reuse completed artifacts; selection budgets are still enforced. Snapshot
|
|
129
136
|
configuration digests remain complete and saved snapshots are not accepted as current merely
|
|
130
137
|
because their artifacts were reused. Workspace cache paths and custom-provider identities remain
|
|
131
|
-
unchanged. See [ADR 018](ADR-018-operational-semantic-cache.md) for exact isolation and recovery rules.
|
|
138
|
+
unchanged in direct mode. See [ADR 018](ADR-018-operational-semantic-cache.md) for exact isolation and recovery rules.
|
|
132
139
|
|
|
133
140
|
If an older user-cache run was interrupted and its configuration has already changed, retain an
|
|
134
141
|
exact copy of the previous configuration in its original directory. Its original `repositoryRoot`
|
|
@@ -197,3 +204,12 @@ Deterministic tests install a public synthetic CLI with npm locally and globally
|
|
|
197
204
|
doctor, login probing and isolated extraction without contacting a model or using real credentials.
|
|
198
205
|
These prove launcher plumbing, not live Codex authentication or semantic quality. Browser and live
|
|
199
206
|
authentication follow-up remain deferred at the maintainer's request.
|
|
207
|
+
|
|
208
|
+
|
|
209
|
+
Traversal hydration in the next minor candidate keeps the built-in Codex source/prompt contract
|
|
210
|
+
unchanged. Its validated `documentTraversal` controls are excluded from the Codex artifact pool
|
|
211
|
+
partition; classification is attached during assembly. Validation documents are excluded from
|
|
212
|
+
semantic calls and counted only against traversal read limits. Module providers use a separate
|
|
213
|
+
full-config namespace in traversal mode, including workspace storage. Modules with mutable
|
|
214
|
+
external inputs must version their declared identity or force extraction; unchanged source/config
|
|
215
|
+
reuse does not discover those external dependencies. See [ADR 026](ADR-026-traversal-hydration.md).
|