@functional-systems/srcx 0.1.0-dev.1 → 0.1.0-dev.10

package/VALIDATION.md

@@ -4,12 +4,317 @@ Date: 2026-09-25 (Asia/Seoul).
 
 ## Evidence boundary
 
-This implementation has passed local fixtures/fault injection/SDK transport checks
-and a live synthetic A/B acceptance run against the user-supplied LambdaDB
-connection. Only generated test source was uploaded. No production repository
-source or paid embeddings were used, and no Git repository/npm package was
-published. Live evidence below is limited to this small fixture; it is not a
-throughput, general concurrency, or retrieval-quality benchmark.
+The initial implementation passed local fixtures/fault injection/SDK transport checks,
+a live synthetic A/B acceptance run, and an exact-package self-index of the public
+srcx repository against the user-supplied LambdaDB connection. No private source
+or paid embeddings were used. These checks are not a throughput, general
+concurrency, or retrieval-quality benchmark. npm publication evidence is recorded
+separately below. The persistent Git branch changes have a separate synthetic
+live run recorded next; the earlier live runs do not validate this new path.
+
+## Evaluation diagnostics and transfer draft
+
+Evaluation subprocess failures now append safe command diagnostics with process
+or JSON-decode stage, exit/signal metadata and allowlisted stderr. Fixture checks
+cover no automatic retry, preservation across multiple failures, secret/source
+omission, private file permissions, malformed output, output overflow and log-write
+failure. This cannot recover stderr from old live failures. Node 24.15.0 passed
+**75 tests**, typecheck, formatting and version checks for this follow-up.
+The read-side regression invokes the actual `readHandle` for seven failure paths
+and verifies both safe-message retention and omission of contaminated stderr.
+
+A separate offline preparation imported pinned Click 8.1.8 and Cobra v1.9.1 Git
+sources into local managed-small build artifacts, without service calls or
+embedding generation. All 16 draft questions' byte hashes, source line ranges and
+inclusion were verified. The 207 files contain 544 eligible chunks with 388,771
+estimated embedding input tokens. Python/Go use the current text fallback; no new
+parser or search behavior was added. These labels are assistant-authored and not
+independently reviewed. See [draft protocol and reproduction](eval/TRANSFER-EVAL.md)
+and [diagnostic retention rules](eval/COMMAND-DIAGNOSTICS.md). No retrieval quality,
+provider cost or live operational reliability is measured by this preparation.
+
+## Retrieval mode comparison
+
+At application/harness commit `de5a23bbc2f0712d134066354b4313420d21fccd`, the
+actual CLI completed 48 searches across the same two managed corpora and fixed
+16-question suite. All **456 results/handles** and **236 reads** matched pinned
+source and version facts. Revised-label complete evidence was lexical **16/16**,
+semantic **8/16**, and hybrid **13/16**; unchanged original labels scored 14/16,
+8/16, and 11/16. These are diagnostic coverage measurements, not human answer grades.
+
+All three hybrid regressions contained the required evidence in search's top ten,
+but outside the first five reads. Keep lexical as the default pending fresh-task
+ranking evaluation. The run also exposed a query response omitting one managed
+vector; same-Tag fetch hydration now preserves non-vector payload equality and
+hit order/scores. The failed attempt remains separate from the complete fresh run.
+
+Node 22/24 passed **67 tests**, typechecking and installed-package CLI checks after
+the fix. The completed live report was revalidated without new service calls, and
+its bytes/timestamp were unchanged. See [full results](eval/RETRIEVAL-MODE-RESULTS.md)
+for per-mode metrics, inspected misses, input/usage accounting, source revisions,
+report hashes and evidence limits; [protocol](eval/RETRIEVAL-MODES.md) documents
+reproduction and effects.
+
+## Query-style supplement
+
+At harness commit `65c8e7060366f4df846c9b2bc08270e7337f0df1`, 72 searches tested
+eight new tasks with identifier, natural-language and mixed formulations using
+the same managed small model and the same two immutable publications. All
+**696 result handles and 356 reads** matched source/version facts. Complete
+first-five evidence by lexical/semantic/hybrid was **7/8, 7/8, 7/8** for identifiers,
+**0/8, 3/8, 1/8** for natural descriptions and **5/8, 7/8, 6/8** for mixed queries.
+These are eight paired tasks, not 24 independent judgments. Semantic helped on
+this diagnostic; the original keyword-heavy result must not be generalized.
+
+The frozen protocol adds verified candidate coverage at ranks 1/3/5/10, the first
+complete prefix, stdout tokens and offline token-budget prefixes. No model,
+ranking, read-selection or product default changed. The original suite/results
+and incomplete run remain intact. A completed rerun preserved report bytes and
+timestamp without service calls. Node 22.14.0 and 24.15.0 passed **69 tests**;
+typecheck, formatting and frozen-harness CI passed. See [results and cases](eval/QUERY-STYLE-RESULTS.md),
+[protocol](eval/QUERY-STYLES.md), and [embedding model research](eval/EMBEDDING-MODELS.md).
+
+## Managed small versus large comparison
+
+At frozen harness `6fa65b541dc50debdfa4f9a6297301b2cb3c2b53`, small (1536 dimensions)
+and large (3072 dimensions) completed 240 searches over identical pinned source/chunk inputs.
+All **2,304 result handles and 1,184 reads** passed source/version validation.
+Style-suite complete top-five evidence was semantic **17/24 → 16/24** and hybrid
+**14/24 → 16/24**; natural semantic improved **3/8 → 4/8**. The separate regression
+set scored semantic **8/16 → 8/16** and hybrid **13/16 → 12/16**. All 40 lexical
+candidate lists matched across models. Small coverage repeated the earlier runs.
+
+An initial large attempt stopped twice with an unknown CLI search error; its
+19 partial rows and two successful diagnostic queries remain excluded. A fresh
+isolated run reused the same publications and completed without changing the
+frozen protocol. All four completed reports revalidated without service calls
+and retained their bytes/timestamps. Keep small as the initial managed choice
+and large opt-in; these familiar diagnostic tasks do not establish a model winner.
+
+Node 22.14.0 and 24.15.0 passed **72 tests**, typecheck and installed-package CLI
+checks covering lexical/small/large. Formatting/version checks and frozen-code
+CI passed. [Full results](eval/MODEL-COMPARISON-RESULTS.md) record per-query changes,
+report hashes, original-label scores, all five roots' reserved usage, and limits;
+[protocol](eval/MODEL-COMPARISON.md) records the predeclared comparison.
+
+Review follow-up: resume and completed replay now reject saved rows whose category
+is missing or differs from the frozen question, even when the stored summary was
+recomputed to match the corruption. The subprocess regression first reproduced
+the acceptance bug, then passed with the guard; valid replay and rejected reports
+both preserve their bytes. Node 24.15.0 passed **73 tests** after this change.
+The four completed model reports have no category mismatches and remain unchanged.
+Their frozen harness remains the revision recorded above; the new harness requires
+fresh run roots rather than rewriting historical runtime fingerprints.
+
+## Managed OpenAI embedding acceptance
+
+The opt-in managed preset passed a separate synthetic live run at application and
+harness commit `177731386f76a13723a83aa8470d4e0715e29424` using Node 24.15.0 and
+the user-supplied development connection. It completed at
+`2026-09-24T20:46:46.305Z` in 111.706 seconds. Only synthetic source was uploaded.
+
+- Both commits published validated immutable Tags, with six generated 1536-number
+  vectors in each corpus. Mixed file/manifest records correctly omitted vectors.
+- Six filtered searches (lexical, semantic and hybrid for each version) returned
+  14 result handles; every full-file read matched the pinned Git blob byte for byte.
+- The second import skipped unchanged records, removed deleted content, and kept
+  the original version's handle readable. Fresh repository discovery preserved the
+  managed preset and schema.
+- Ordinary upserts submitted ten eligible chunk inputs across both imports. The
+  preflight upper bound was 167 estimated document input tokens; four searches
+  requested query embeddings. These counts are not provider-reported billed usage.
+
+The first run at `fb5b6d8` correctly remained unpublished: this deployment omitted
+vectors from list responses despite `includeVectors=true`, while fetch/query on
+that same Tag returned them. The transport now fetches those listed IDs from the
+same immutable Tag and requires exact non-vector payload equality before vector
+validation. Regression checks reject missing fetch records and changed payloads.
+The failed run and retry journal remain under `.srcx/live-managed/`; they were not
+relabelled as successful evidence. The successful run has a separate Collection
+and evidence directory, `.srcx/live-managed-verified/`, in the validation worktree.
+Both synthetic Collections are retained for inspection.
+
+The successful report SHA-256 is
+`6c96ac8af9b79c85e98c84f1f01851906f0b1b3e4d96b51d9417cb59380ac532`;
+its runtime fingerprint is
+`7d8a8006820ee523a88e80d6614327000ce305cffafe377d18fda5b50ba9c947`.
+A same-runtime rerun made no service calls and preserved the original report bytes.
+The harness rejects changed runtime/destination/Node inputs and incomplete runs
+before new requests; recovery after uncertain writes remains explicit.
+
+Node 22.14.0 and 24.15.0 each passed **61 tests**, typechecking, and installed-tarball
+CLI checks for both presets. Formatting, version validation and diff checks passed.
+Local tests cover generated-vector failures, exact payload validation, unchanged
+record reuse, uncertain-write recovery, preset/schema drift, query serialization,
+and credential-free managed previews. Live checks used the built application APIs;
+installed-package CLI checks used the loopback fault-injection store.
+These checks establish integration/source correctness, not semantic relevance gains
+or performance on real repositories. The earlier lexical evaluations remain separate.
+
+## Default CLI search/read evaluation
+
+The follow-up uses actual built CLI subprocesses with the unchanged default
+syntax/path-scope-symbol preset across pinned public `srcx` and `lambdadb-cli`
+commits. At harness commit `57b63e1cdbf7ec722b95bb0f40ac0ef5039af3e8`, both
+publications validated and all **136 search results/handles** and **76 read
+responses** matched their pinned source, version and citation. No embeddings
+were used. The run completed at `2026-09-24T20:01:58.865Z`.
+
+The frozen 16-question suite recovered complete labeled evidence for **14/16**.
+The two remaining cases reflect a one-byte blank-line boundary and an omitted
+alternative document/code answer, not demonstrated source-integrity failures.
+Mean full CLI stdout was **7,383 tokens** per query. This fixed-five-read protocol
+is distinct from the earlier full-chunk/token-budget comparison; neither measures
+general task success. Labels remain assistant-authored without independent human
+review. See [eval/CLI-RESULTS.md](https://github.com/lambdadb/srcx/blob/develop/eval/CLI-RESULTS.md)
+for per-repository scores, inspected misses and evidence boundaries, and
+[eval/CLI-WORKFLOW.md](https://github.com/lambdadb/srcx/blob/develop/eval/CLI-WORKFLOW.md)
+for reproduction and normal Collection effects.
+
+Local Node 24.15.0 passed **54 tests**, typechecking, formatting/version checks and
+the installed-package CLI contract. Node 22/24 CI passed at the harness commit.
+Live execution used built CLI subprocesses; package validation used a loopback
+fixture. A completed rerun preserved the original report bytes and timestamp.
+Raw evidence and recovery journals remain under `.srcx/cli-eval/` in the validation
+worktree; the older pilot's original report remains unchanged.
+
+## Retrieval evaluation harness
+
+The internal pilot adds five checks for unique source-range scoring, token-budget
+prefix selection, source/hit integrity, equal path-only enrichment, and rejecting
+stale runtime checkpoints before any connection. With these checks, Node.js
+24.15.0 passed **50 tests**, typechecking, version validation, and clean installed
+package checks. The default CLI preset and config hash remain unchanged.
+
+The suite fixes 16 queries, expected source byte ranges, and two full Git commits
+before retrieval. Both chunkers materialize the same included files, use separate
+config identities, and run through the existing immutable publication validation.
+Live retrieval observations and limitations are in [eval/RESULTS.md](https://github.com/lambdadb/srcx/blob/develop/eval/RESULTS.md),
+with reproduction instructions in [eval/README.md](https://github.com/lambdadb/srcx/blob/develop/eval/README.md). These diagnostic
+labels were authored from source before querying and are not independently
+human-reviewed judgments or a general retrieval-quality benchmark.
+
+Post-review, `splitEvidence` now requires complete coverage of all required ranges
+before classifying a result as split across chunks. The multi-range regression
+brings the local suite to **51 passing tests**. Offline rescoring of all 32 retained
+query/method pairs preserved every metric and category summary. The original live
+report, runtime fingerprint, and completion timestamp were not changed.
+
+## Persistent Git branch tracking
+
+Local regression coverage exercises one writer across consecutive commits,
+branch-local deletion/diff baselines, separate branches sharing canonical Tags,
+rewinds to already published commits, and imports from fresh local state. Fault
+injection verifies last-published branch resolution during partial writes, pending
+ownership protection when the local journal is missing, stale/corrupt candidate
+rejection, and retries after uncertain control writes. Branch/tag ambiguity and
+explicit full refs are covered.
+
+The CLI fixture also imports and updates a tracked branch through the real SDK
+transport and selects it for search, resolve, and read. The same fixture runs
+against the installed tarball. These checks use a loopback fault-injection model,
+not a live LambdaDB deployment.
+
+A separate **live LambdaDB run passed all 12 checkpoints** on September 25 using
+Node **24.15.0**, application source commit
+`f42a9141cef3bfc0b604e1b14a00c79607c7dd98`, and the original checkout's `.env.local`
+loaded explicitly through Node. The run took **414.195 seconds** and uploaded only
+synthetic Git source with no paid embeddings. It verified:
+
+- First-import timeout remains unpublished, followed by same-journal recovery.
+- A -> B uses one fixed Collection Branch; pending B still resolves/searches A,
+  and a fresh local state cannot overwrite that pending branch.
+- Added/modified/deleted code searches correctly after B; old A evidence still
+  reads the exact original source bytes.
+- A second Git branch shares canonical A/B Tags while maintaining its own writer
+  baseline, including B -> A -> B movement and an unchanged repeated import.
+- A SHA-only new commit uses a manual `work-*` writer.
+
+A separate invocation of the actual CLI against that live Collection passed
+branch `resolve`, branch-selected `search`, and a full-file `read` compared byte
+for byte with the pinned Git blob. It used the built CLI, not a registry-installed
+package. Local installed-package evidence remains separate above.
+
+The reproducible harness is `scripts/live-branches.mjs` (`npm run
+test:live:branches`). Its ignored evidence is `.srcx/live-branches/report.json`
+and `.srcx/live-branches/cli-report.json` in the validation worktree. The report
+records the exact source commit, harness SHA-256, per-check results, and resource
+URL. One synthetic Collection is retained with two tracked writers, one manual
+writer, the two control/checkpoint Branches, three canonical commit Tags, and seven
+candidate Tags. No application fix was needed during this live run.
+
+On Node **22.14.0** and **24.15.0**, typechecking, all **43 tests**, and installed
+package checks passed. Formatting, release-version validation, and `git diff
+--check` also passed locally. These results are not GitHub CI evidence.
+
+## Final branch review validation
+
+The final pre-PR review reproduced a read-order race: a concurrent publication
+could advance branch control beyond the version list already captured by the
+reader. Resolution now pins the control record first. The review also found that
+live reruns could skip saved checkpoints while replacing the reported source
+revision; checkpoints now require matching runtime, harness, fixture, lockfile,
+and Node fingerprints and preserve their original revision/completion time.
+Regression tests cover both cases, including rejecting stale evidence before any
+connection or report mutation.
+
+After these fixes, Node **22.14.0** and **24.15.0** each passed typechecking,
+all **45 tests**, and installed-package checks. Formatting, release-version
+validation, and diff checks passed locally.
+
+A fresh synthetic live run at application/harness commit
+`2988c7e1826a44cfed48381681d0125b28ae9086` passed all **12 checkpoints** in
+**432.836 seconds**. The built CLI also passed live branch resolution,
+search, and exact source reads. Final reports are retained separately under
+`.srcx/live-branches-review/report.json` and
+`.srcx/live-branches-review/cli-report.json`; the previous run remains intact.
+This final run retained one synthetic Collection with two tracked writers, one
+manual writer, two control/checkpoint Branches, three canonical commit Tags,
+and seven candidate Tags. CI evidence, once available on the PR, is separate
+from these local and live results.
+
+## Initial npm package and live self-index
+
+The exact `0.1.0-dev.1` tarball from merged commit
+`f94f947bf593b2f8498c07e130933b0ab444f868` passed clean installation, CLI execution,
+WASM parsing, and the loopback import/search/read contract. Its installed CLI then
+imported that same Git commit into LambdaDB: **37 files, 260 chunks**, with
+inventory, content, and query validation passing. A `validateBuild` query returned
+three results from `src/build.ts`; the full-file read matched the pinned Git blob
+byte for byte. This was a lexical-only import (`embedding=none`).
+
+The first npm publication used this tested tarball with the `dev` dist-tag. The
+registry tarball's SHA-512 matched the candidate, and a clean installation of the
+downloaded registry tarball passed the same CLI contract. Registry signatures are
+present. The local bootstrap has no GitHub Actions provenance. npm also created
+`latest=0.1.0-dev.1`; that tag does not represent a stable release.
+
+Local evidence is retained under `.srcx/releases/0.1.0-dev.1-f94f947/` (ignored),
+including `candidate.json`, `live-publication.json`, `live-verification.json`,
+`registry-version.json`, and the exact candidate and downloaded tarballs.
+The source commit's [Node 22/24 CI](https://github.com/lambdadb/srcx/actions/runs/36035063463)
+also passed. Current distribution and automation status is in
+[RELEASING.md](RELEASING.md).
+
+## First automatic development publication
+
+[CI attempt 3](https://github.com/lambdadb/srcx/actions/runs/36035063463/attempts/3)
+published `0.1.0-dev.3` using npm Trusted Publishing from the same reviewed commit.
+The version suffix is the first-parent commit count. The registry's `gitHead`
+and SLSA provenance both identify `f94f947bf593b2f8498c07e130933b0ab444f868`;
+provenance also identifies `lambdadb/srcx`, `.github/workflows/publish.yaml`, and
+the GitHub-hosted workflow run. The downloaded tarball's SHA-512 matches registry
+metadata. A clean install by package name/version passed CLI execution and the
+loopback import/search/read contract. `npm audit signatures` verified all eight
+installed registry signatures and three attestations, including srcx provenance.
+`dev=0.1.0-dev.3`; `latest=0.1.0-dev.1` remains the bootstrap prerelease.
+
+The initial automatic attempt stopped during the pre-write metadata lookup while
+npm's full package listing still returned 404. After the listing became visible,
+only the failed publishing job was rerun. No publication write was retried to
+work around propagation. Local evidence is retained in `registry-dev3.json`,
+`attestations-dev3.json`, and `provenance-dev3.json` under the release directory
+above. No stable release, GitHub Release, or Homebrew formula was created.
 
 ## Local checks
 
@@ -76,11 +381,9 @@ The test suite covers:
   reruns. Simulated registry delays/failures verify bounded post-write reads and
   no automatic publication retry; no test writes to npm.
 
-Release infrastructure is prepared in `.github/workflows/publish.yaml` for Node
-22/24. Local validation does not establish GitHub Actions or npm publication
-success. Initial npm registration, package-specific Trusted Publisher setup, and
-enabling `NPM_DEV_PUBLISH_ENABLED` remain separate release steps. See
-[RELEASING.md](RELEASING.md); no npm or Homebrew release was performed in this setup.
+Release infrastructure is in `.github/workflows/publish.yaml` for Node 22/24.
+The local checks above are distinct from the CI, npm, and live-service evidence
+recorded separately. No stable/rc or Homebrew release has been performed.
 
 The demo writes [.srcx/demo-report.json](.srcx/demo-report.json). That generated
 report identifies the retained synthetic repository, A/B build artifacts, counts,
@@ -94,19 +397,22 @@ interface verifies eligibility/cache behavior only; production provider selectio
 semantic/hybrid querying, pricing, and evaluation remain open. Public imports
 reject a vector-enabled build to prevent fixture vectors from reaching LambdaDB.
 
-Each import gets a fresh `work-*` Branch, with incremental reuse only from an exact
-validated previous writer snapshot. Existing writers are not mutated after
-publication. Persistent `git-*` tracking, automatic failed-workspace recovery,
-Alias pruning, and garbage collection are not yet implemented. Failed/candidate
+Git branch imports reuse a fixed `git-*` Branch with its last validated immutable
+baseline; SHA/tag-only imports retain frozen `work-*` writers. Automatic Git
+observation, failed-workspace recovery, branch rename/deletion handling, Alias
+pruning, and garbage collection are not yet implemented. Failed/candidate
 resources and build artifacts are retained. A single importing host is required;
-the local lock is not a distributed lease.
+the local lock is not a distributed lease. Remote pending ownership prevents
+accidental adoption from fresh local state but is not a distributed compare-and-set
+or lease protocol.
 
 Parsing uses the pinned packaged grammars. Unsupported/new syntax can fall back
 and is recorded; no claim of complete language-version coverage is made. There is
 no optimal-chunking claim. The fixed-window baseline is implemented internally;
-the planned 10–20-query retrieval comparison has not been run. The in-memory test
-model does not approximate Lucene ranking, distributed indexing, retention, or
-compaction timing.
+the first 16-query live lexical comparison is recorded in [eval/RESULTS.md](https://github.com/lambdadb/srcx/blob/develop/eval/RESULTS.md).
+It is a single-repository diagnostic, not a general optimal-chunking result. The
+in-memory test model does not approximate Lucene ranking, distributed indexing,
+retention, or compaction timing.
 
 Canonical publication records exist on main; listings require both the remote
 summary and the matching immutable Tag manifest. A candidate or a lone `ver-*`

package/dist/build.d.ts

@@ -1,4 +1,4 @@
-import { type Identity } from "./git.js";
+import { resolveCommit, type Identity } from "./git.js";
 import { CHUNKER } from "./chunk.js";
 import { type Doc } from "./common.js";
 export declare const INDEX_CONFIGS: {
@@ -26,13 +26,82 @@ export type Preset = {
     schemaVersion: 1;
     chunker: typeof CHUNKER;
     mode: "syntax" | "window";
+    /** Internal evaluation override; the CLI keeps its original enrichment. */
+    enrichment?: "path-only-v1";
     maxFileBytes: number;
     embedding: {
         model: string;
         dimensions: number;
+        managed?: true;
+        provider?: "openai";
+        similarity?: "cosine";
+        sourceField?: "embeddingText";
     } | null;
 };
 export declare const PRESET: Preset;
+/** Opt-in managed preset; PRESET stays byte-for-byte compatible with lexical corpora. */
+export declare const MANAGED_PRESET: Preset;
+export declare const MANAGED_LARGE_PRESET: Preset;
+export declare function presetFor(embedding?: string): Preset;
+export declare function supportedPreset(preset: unknown): preset is Preset;
+export declare function indexConfigs(preset?: Preset): {
+    readonly kind: {
+        readonly type: "keyword";
+    };
+    readonly path: {
+        readonly type: "keyword";
+    };
+    readonly language: {
+        readonly type: "keyword";
+    };
+    readonly fileId: {
+        readonly type: "keyword";
+    };
+    readonly symbol: {
+        readonly type: "keyword";
+    };
+    readonly searchText: {
+        readonly type: "text";
+        readonly analyzers: readonly ["standard"];
+    };
+} | {
+    embeddingText: {
+        type: string;
+        analyzers: string[];
+    };
+    embedding: {
+        type: string;
+        managedEmbedding: boolean;
+        embedding: {
+            model: string;
+            dimensions: number;
+            provider?: "openai";
+            similarity?: "cosine";
+            sourceField?: "embeddingText";
+        };
+    };
+    kind: {
+        readonly type: "keyword";
+    };
+    path: {
+        readonly type: "keyword";
+    };
+    language: {
+        readonly type: "keyword";
+    };
+    fileId: {
+        readonly type: "keyword";
+    };
+    symbol: {
+        readonly type: "keyword";
+    };
+    searchText: {
+        readonly type: "text";
+        readonly analyzers: readonly ["standard"];
+    };
+};
+/** Validate server enrichment separately; all other payload fields remain exact. */
+export declare function recordHash(doc: Doc, preset: Preset): string;
 export type InventoryItem = {
     path: string;
     pathBase64: string;
@@ -48,6 +117,8 @@ export type InventoryItem = {
     parseStatus?: string;
     tokens?: number;
     embedded?: number;
+    managed?: number;
+    managedTokens?: number;
     skipped?: number;
 };
 export type Build = {
@@ -84,6 +155,7 @@ export declare function loadBuild(directory: string): Promise<Build>;
 export declare function materialize(args: {
     identity: Identity;
     ref: string;
+    resolvedCommit?: Awaited<ReturnType<typeof resolveCommit>>;
     output: string;
     previous?: Build;
     preset?: Preset;