@functional-systems/srcx 0.1.0-dev.4 → 0.1.0-dev.40

package/DESIGN.md

@@ -1,10 +1,11 @@
 # srcx: Initial Implementation Design
 
-Status: initial CLI implemented with local tests and live synthetic A/B validation
-on LambdaDB. See [README.md](README.md) for runnable interfaces and
+Status: CLI implemented with persistent Git branch tracking, managed semantic/hybrid
+search and optional local Qwen reranking. Local, live and public-benchmark evidence
+have separate scopes. See [README.md](README.md) for runnable interfaces and
 [VALIDATION.md](VALIDATION.md) for the bounded live evidence and remaining scope.
 
-Updated: 2026-09-25.
+Updated: 2026-09-26.
 
 ## 1. Purpose, decisions, and implementation defaults
 
@@ -18,14 +19,13 @@ supporting each result. The project name is `srcx`.
 | --- | --- |
 | Project and display name | `srcx` |
 | Local directory | `srcx` |
-| Intended GitHub repository | `lambdadb/srcx` |
+| GitHub repository | `lambdadb/srcx` |
 | CLI executable | `srcx` |
-| Intended npm package | `@functional-systems/srcx` |
+| npm package | `@functional-systems/srcx` |
 | Short description | Version-aware code search on LambdaDB |
 
-The name is selected; the GitHub repository and npm package have not been created
-or published in this project. The package manifest and executable entry point are
-present in the initial CLI. Keep display/configuration naming centralized and
+The public GitHub repository and npm development channel are available. Current
+publication evidence and stable-release status are recorded in RELEASING.md. Keep display/configuration naming centralized and
 distribution identity in `package.json`. Database IDs and the existing
 `code-*` Collection naming and `purpose=code-search-v1` protocol marker remain
 independent of the product name.
@@ -40,7 +40,7 @@ Confirmed boundaries from the discussion:
 - Map each Git tag to a LambdaDB Alias targeting its published commit Tag.
   Multiple Git tags for the same commit share one commit Tag and Snapshot.
 - Keep search results and subsequent source reads pinned to the selected version.
-- Replace a changed file's entire chunk set initially; cache embeddings by input.
+- Replace a changed file's entire chunk set initially; retain unchanged remote records.
 - Keep core LambdaDB work, GitHub App/webhooks, UI, MCP, and a public SaaS outside
   the first deliverable. sbrain reference documents remain unchanged.
 - No remote repository creation, application commit, push, publication, real
@@ -53,7 +53,7 @@ Current recommendations, distinguished from user-confirmed choices:
 | Runtime | TypeScript and a supported Node.js release compatible with the existing SDK |
 | Database | LambdaDB only; no public backend selection or interchangeable database framework |
 | API key | Read `LAMBDADB_API_KEY` by default; optional environment-variable-name override |
-| Syntax | Java and TypeScript/JavaScript first; other text uses recorded fallback |
+| Syntax | Java, TypeScript/JavaScript, Python, Go, Rust, C/C++, Shell and SQL; other text uses recorded fallback |
 | Source | Exact UTF-8 source inside file documents, with local blob cache |
 | File size | Start with a 1 MiB raw-source limit and explicit exclusions |
 | Embedding | A fixed `none` preset first; with a model, embed meaningful code and prose chunks while recording policy-based skips |
@@ -72,9 +72,10 @@ This document is not a deployed-revision audit or evidence of a passing live run
 
 The initial command set below is implemented; run it as `node dist/cli.js` from
 this checkout after `npm run build`. See README for exact options and recovery.
-The first implementation uses fresh `work-*` Branches and observed-local-tag Alias
-synchronization; persistent tracked branches and authoritative pruning remain
-follow-up work. Immutable manifest `requestedRef` is normalized to the resolved
+Git branch imports now reuse persistent `git-*` Branches; explicit commit/tag
+imports use frozen `work-*` writers. Observed-local-tag Alias synchronization is
+implemented; automatic Git observation and authoritative pruning remain follow-up
+work. Immutable manifest `requestedRef` is normalized to the resolved
 commit OID; the
 original user spelling remains in the local build artifact. The broader contract
 below remains the target design.
@@ -379,18 +380,30 @@ readers on multiple machines do not require local registration state.
    changed/deleted file, compute obsolete IDs as previous file/chunk IDs minus
    replacement IDs. A deleted file has an empty replacement set. Obtain previous
    IDs from the inventory, not mutable query-by-path results.
-6. Apply the pinned per-chunk embedding policy. If embeddings are enabled, batch
-   eligible cache misses across chunks/files within the provider's token/request
-   limits. Validate successful vectors before caching.
+6. Apply the pinned per-chunk embedding policy. The managed preset materializes
+   eligible `embeddingText` inputs without calling a provider locally. LambdaDB
+   generates vectors during ordinary upsert. Skip unchanged remote records and
+   validate generated vectors in the candidate Tag before publication.
    Bound queue memory; do not load an entire repository's content into RAM.
 7. Materialize deterministic records and deletion lists as local build artifacts.
    Produce size-bounded inventory parts and a root manifest template. Check all
    serialized sizes and record-set hashes before starting uploads. The execution
    journal supplies the separate attempt ID when sealing the final manifest.
 
+Local `records.jsonl` output batches complete canonical record lines in a 64 KiB
+buffer, flushing before a line would exceed that bound. Larger individual lines
+are written directly after flushing earlier records. Hash the existing canonical
+representation, which is byte-equivalent to hashing the record object, rather
+than serializing the object again. Flush all records before writing build metadata
+and running full artifact validation. Write failures remove the partial output
+directory; previous artifacts remain untouched. This changes write-call overhead,
+not artifact bytes, order, schema or validation requirements.
+
 Dry-run executes the same inventory/chunking path but performs no database mutation
-or embedding call. For pending embedding misses, report tokens and projected vector
-size separately; do not claim a completed vector/records hash. Report every entry,
+or embedding call. Report `managed` eligible chunks and `managedTokens` (estimated
+input tokens for the complete corpus), with `embedded=0` in the local artifact.
+These are not billed usage or incremental upload counts. Record hashes bind the
+submitted payload; server-generated vectors are validated separately. Report every entry,
 exclusion reason, parse status, byte size, chunk/token counts, embedding eligibility
 and skip reasons, and change summary. Dry-run plans may show pending embedding work;
 published chunk records may not silently treat missing required vectors as skips.
@@ -418,6 +431,17 @@ connection or remote repository lookup. No `--backend` option is needed for that
 Use ordinary SDK upsert/delete for this workaround. Bulk ingestion remains later
 work; do not assume its asynchronous processing obeys the same marker ordering.
 
+For baseline comparison, prefer a local artifact only after matching repository,
+configuration, commit, build, record and inventory identities to the journal's
+remote baseline, checking its Tag/Snapshot and manifest, and fully validating the
+artifact. Reconstruct the old file/chunk and metadata hash map from that artifact.
+The local artifact is a cache, never publication evidence: missing files or a
+different build fall back to the immutable baseline's full listing; corruption
+of a matching artifact fails closed. A write-stage retry uses the same rule and
+does not require the previous local artifact. Compare inventory-part hashes too,
+so unchanged metadata parts are not uploaded again. Always write the new root
+marker separately after all required mutation ACKs.
+
 ### Replacing one file
 
 Replace a modified file's chunk set as a unit in the build plan, using the previous
@@ -436,6 +460,149 @@ record and 20 chunk records; these can share upload requests with other files.
 Original text appears once in its file record, with exact source ranges in chunks.
 Search context can repeat in chunk search text without changing the quoted source.
 
+### Experimental segmented local artifacts
+
+Status: opt-in implementation. `import --artifact-storage segments` writes local
+format 2; the default remains format 1 and a single `records.jsonl`. Presets,
+remote schema, publication ordering and full validation are unchanged. See the
+storage-only prototype and integrated-build measurements in `VALIDATION.md`.
+
+The segmented layout uses one logical segment per included file: the exact
+file record followed by all of its chunks, in ordinal order. A new artifact owns
+all objects it needs. It may share physical storage through hardlinks, but never
+depends on a previous artifact's directory, a global cache, or the current Git
+checkout after construction. This targets retained-record rewrites; it does not
+make full validation proportional to the delta.
+
+The segmented descriptor extends the local storage version only:
+
+```ts
+type RecordPart = {
+  fileId: string;
+  digest: string; // SHA-256 of exact canonical UTF-8 JSONL, including newlines.
+  byteLength: number;
+  recordCount: number;
+};
+
+// Existing logical Build fields remain unchanged.
+// `directory` is resolved at load time, never used as an external object pointer.
+type SegmentedBuild = Omit<Build, "format"> & {
+  format: 2;
+  recordParts: RecordPart[];
+};
+```
+
+Objects live at `objects/<64-lowercase-hex-digest>.jsonl` beneath the artifact.
+Ordered references preserve the builder's existing changed-then-retained record
+order, with no excluded-file segments. Full inventory entries, `recordHashes`,
+counts and aggregate hashes remain in `build.json`. This keeps logical
+identity independent of physical layout: for the same repository/commit/config,
+changing only storage layout retains `buildId`, `configHash`, `recordsHash` and
+`inventoryHash`. Object digests add storage integrity checks; they do not replace
+record/source checks.
+
+Require exactly one part per included inventory file, no duplicate file references,
+and `recordCount === 1 + chunkIds.length` for that file. Each object must begin with
+its referenced file record and contain only that file's expected chunks. Byte and
+record counts must be positive safe integers, not allocation instructions from
+untrusted metadata.
+
+The writer and full validator follow these rules:
+
+1. Fully validate the previous artifact before reusing its records or hashes.
+   Keep the existing repository/configuration and Git blob/mode reuse conditions.
+   For a format-2 previous build, find retained objects from its validated file
+   inventory and part mapping; avoid the extra whole-record copy pass.
+2. Stream added/modified file groups to unique temporary objects, with the current
+   per-document size checks and bounded write buffer. Hash actual serialized
+   bytes, close each object and install it under its digest. Do not buffer an
+   entire file's chunk set in memory or impose an arbitrary new segment-size cap.
+3. Hardlink retained objects into the new artifact, using exclusive destination
+   creation. `EXDEV`, `ENOTSUP`, `EOPNOTSUPP` and `ENOSYS` use exclusive copy
+   fallback. Other I/O, missing-object or permission failures remain failures.
+   No destination reference may point back into the previous directory.
+4. Keep new output private to its attempt. Finish all objects before installing
+   the descriptor, then run complete validation against the new artifact. Object
+   write failures remove only that attempt's partial output. Interrupted output
+   is never accepted because a descriptor or marker exists. Atomic rename alone
+   is not an fsync or power-loss durability guarantee.
+5. On every full validation, check descriptor shape and logical membership,
+   recompute each referenced object's byte digest/count, and run all existing
+   uniqueness, record-hash, exact source/position/token and embedding-input checks.
+   A valid digest or matching count cannot establish that an old file belongs to
+   the new build. Retained records are not sampled or exempted.
+6. Reject arbitrary object paths, symlink objects/directories and special files.
+   Use digest-derived paths, `O_NOFOLLOW | O_NONBLOCK` file opens and post-open
+   regular-file/size checks, plus a local regular-directory check. The feature
+   requires native no-follow support; the local measurements use macOS. The existing
+   single-host model assumes importer-owned artifact directories; hostile
+   concurrent directory replacement by the same OS user is outside that model.
+   Keep one open record handle and bounded UTF-8 line decoding, retaining the
+   existing largest-document allowance. File mode, inode identity, mtime and a
+   previous validation receipt are not integrity proofs.
+
+The shared `records()` iterator verifies object digests when fully consumed, as
+`validateBuild()` always does. Partial consumers do not establish completeness;
+publication readiness probes follow full build validation. No persistent
+validation receipt skips object/record/source checks on restart.
+
+Hardlinks reduce copying but do not enforce immutability: a write to one shared
+inode can corrupt several local artifacts. Rehashing and complete source checks
+must detect this on every use. Read-only modes can be defense in depth only.
+Remote published Tags remain the authority for immutable version reads and do
+not depend on the survival or integrity of local hardlinks.
+
+| Condition | Required behavior |
+| --- | --- |
+| Previous directory deleted after the new build | New artifact remains self-contained and validates. |
+| Artifact copied or relocated | Resolve all references beneath the new directory; no original paths needed. |
+| Missing/corrupt object in the pending build | Fail before corpus mutation; retain the journal. Never substitute the current Git tip. |
+| Previous local build entirely absent | Preserve the existing cold-build/remote-baseline recovery paths. |
+| Missing object in an otherwise present previous build | Preserve the current missing-record validation failure; no silent reuse of metadata. |
+| Previous local cache unavailable during publication retry | Validate the intact pending artifact and use its journal's immutable remote baseline, as today. |
+| Corrupt matching local baseline cache | Preserve fail-closed validation rather than trusting its declared hashes. |
+| Cross-filesystem link failure | Explicit copy fallback writes retained bytes too; no delta-write claim for that path. |
+| Partial object/descriptor write or stale local receipt | Remain unpublished; reconstruct only from the exact original build identity or restore matching objects. |
+
+No global shared-object garbage collector is needed:
+deleting one artifact unlinks its own objects without invalidating other hardlinks.
+Do not prune local artifacts associated with an active journal or claim this
+implements remote version retention/cleanup.
+
+Runtime integration and compatibility:
+
+- The storage-format reader/descriptor validator retains legacy format-1 reads
+  and resumption. Internal readers accept the already loaded Build, avoiding
+  repeated parsing of a large descriptor. Directory-based callers load its
+  `build.json`. Unknown formats and missing format-2 descriptors/objects fail;
+  they do not fall back to a stray
+  `records.jsonl` merely because it exists.
+- `--artifact-storage jsonl|segments` selects storage for a new build and defaults
+  to `jsonl`, even when the previous artifact is segmented. Initial conversion
+  from format 1 requires the full retained-record pass. Existing `--artifact`
+  imports use the recorded format and reject an explicit storage override.
+  Preview JSON adds `artifactFormat` (1 or 2); the remaining logical fields and
+  exit codes are unchanged. Storage selection is not an embedding preset change.
+- `validateBuild`, previous-record reuse, publication readiness probes and
+  changed-record uploads use the shared reader. Full candidate validation,
+  final-marker ordering and managed-vector validation/hydration remain intact.
+- Default JSONL evaluation fixtures and frozen archive byte hashes remain format
+  1. Consumers reading `records.jsonl` directly must opt into the shared reader
+  before accepting format 2; they are not automatically migrated. Installed-package
+  lexical/managed CLI tests cover segmented resumption, loss of the previous local
+  artifact, tracked imports and immutable reads.
+
+Default promotion requires end-to-end measurements including descriptor bytes, initial
+conversion, full validation, cold recovery, cross-filesystem copies and bounded
+memory/open handles. The prototype reduces record payload writes but adds many
+file operations and shows slower validation. Integrated builds also trade speed
+for materially lower writes and additional allocated storage. This justifies an
+explicit experimental choice, while format 1 remains the default. Pack-based physical
+storage or batched reads can be investigated later if per-file open/link overhead
+dominates; neither is silently assumed by this implementation. Skipping full remote
+candidate verification still requires the separate verified backend contract
+described in section 9. Dependency ingestion remains outside this work.
+
 ## 7. Code-search schema v1
 
 The Collection uses one sparse document schema with `kind` distinguishing file,
@@ -473,11 +640,12 @@ fields are omitted when absent; records of different kinds need not share them.
 | Record | Additional stored fields |
 | --- | --- |
 | File (`kind=file`) | `path`, `blobOid`, `contentHash`, `byteLength`, `sourceText`, `language`, `parseStatus`, `chunkCount` |
-| Chunk (`kind=chunk`) | `fileId`, `path`, parent `contentHash`, `language`, `chunkKind`, `startByte`, `endByte`, `startLine`, `endLine`, `ordinal`, `searchText`, `embeddingStatus`; optional `symbol`, `scope`, `signature`, `embeddingSkipReason`, `embeddingInputHash`, `embedding` |
+| Chunk (`kind=chunk`) | `fileId`, `path`, parent `contentHash`, `language`, `chunkKind`, `startByte`, `endByte`, `startLine`, `endLine`, `ordinal`, `searchText`, `embeddingStatus`; optional `symbol`, `scope`, `signature`, `embeddingSkipReason`, `embeddingInputHash`, `embeddingText`, `embedding` |
 | Corpus root (`kind=manifest`, `role=corpus`, ID `__manifest__`) | `repoId`, `indexId`, full `commitOid`, `commitTime`, `requestedRef`, full `config`, `inventoryHash`, `recordsHash`, `counts`, `buildId`, `attemptId`, `inventoryPartIds` |
 | Inventory part (`kind=manifest`, `role=inventory`) | `partOrdinal`, `partHash`, ordered `entries` with path, file/chunk IDs, blob/content identities, inclusion/exclusion reasons, parse states, sizes, and embedding coverage |
 | Repository control (`kind=manifest`, `role=repository`, main ID `__repo__`) | Full source identity, repo/index IDs, immutable preset/config, initialization state |
-| Ref control (`kind=manifest`, `role=git-ref`, main) | Full Git ref name, encoded Alias/Branch name, observed commit/time, applied version and synchronization state |
+| Git tag control (`kind=manifest`, `role=git-ref`, main) | Full Git tag ref, encoded Alias name, observed commit, applied version and synchronization state |
+| Git branch control (`kind=manifest`, `role=git-branch`, main) | Full Git branch ref, encoded Branch name, validated writer baseline, canonical published Tag and pending attempt identity |
 | Validation summary (`kind=manifest`, `role=validation`, main) | Candidate name/Snapshot, build/config/attempt identity, verifier version and checked counts/hashes; no premature claim of publication |
 
 `startByte`/`endByte` are half-open; display lines are one-based/inclusive.
@@ -486,8 +654,13 @@ content such as function, declaration, documentation, imports, or fallback; it i
 separate from the file/chunk/manifest discriminator. `counts` includes included and
 excluded files, chunks, and embedded/skipped chunks with reason totals.
 
-In a completed corpus, `embeddingStatus=embedded` requires a valid vector and
-`embeddingInputHash`, and omits the skip reason. `embeddingStatus=skipped` requires
+The managed preset uses `embeddingStatus=managed` to identify its generation policy,
+not local completion. Eligible chunks carry `embeddingText=searchText` and an
+`embeddingInputHash` over the exact input and embedding configuration; ordinary
+upserts omit `embedding`. A completed candidate must contain a nonzero, finite
+1536-number vector for every eligible chunk. Local artifacts contain no vectors.
+The internal fixture-only client embedder retains `embeddingStatus=embedded`;
+its arbitrary presets cannot be published through the CLI. `embeddingStatus=skipped` requires
 an explicit policy reason and omits the vector and input hash. The lexical-only
 preset records `embedding-disabled`; an embedding-enabled preset records its
 specific exclusion reason. Pending or failed required embedding work stays in the
@@ -543,15 +716,24 @@ is stored in the file; enriched chunk search text is never the citation authorit
 `sourceText`, role, byte/line positions, hashes, commit identity, scope/signature,
 embedding diagnostics, and inventory/config objects are stored without entries in
 `indexConfigs`. Public documentation states that fields are stored even when not
-indexed. Do not invent an `index: false` option. The `standard` analyzer is a
-baseline, not a proven code-specific ranking solution. An initial enrichment
+indexed. Do not invent an `index: false` option. The default text analyzer is
+`standard`; users can choose any nonempty combination of `english`, `japanese`,
+`korean`, and `standard`. The sorted, deduplicated list is part of the pinned
+preset/configuration hash and configures both `searchText` and managed
+`embeddingText`. Discovery requires the matching schema; a different selection
+uses a different Collection. Analyzer selection does not change embedding input
+text or the programming-language chunker. No analyzer is a proven universal
+code-search ranking solution. An initial enrichment
 candidate preserves original identifiers while adding camelCase/snake_case terms;
 pin and evaluate that transformation before adopting it as the preset.
 
-An embedding-enabled preset adds an unmanaged `embedding` vector field with a
-concrete model-selected dimension and similarity metric, fixed before Collection
-creation. An indexed vector field does not require a vector on every chunk; file
-and manifest records also omit it. The `none` preset omits the vector index entirely.
+The opt-in managed presets add indexed text `embeddingText` and a managed
+`embedding` vector field with provider `openai`: `text-embedding-3-small` uses
+1536 dimensions and `text-embedding-3-large` uses 3072, both with cosine similarity.
+The source field, dimensions and similarity are nested under the vector field's `embedding` configuration; there are no
+top-level dimension/similarity properties. Collection discovery verifies the
+complete resolved schema and pinned preset. Missing `embeddingText` on skipped
+chunks, files and manifests means no vector is generated for those records. The `none` preset omits the vector index entirely.
 Changing presets creates a separate index; do not silently mutate an existing
 serving index. Do not write placeholder or test vectors into a real corpus.
 
@@ -593,21 +775,54 @@ must include stored-only source round-trip and mixed record kinds.
 ## 8. File policy, chunking, and embedding reuse
 
 Every Git tree entry must be accounted for. Include source, tests, documents,
-configuration, build definitions, and unknown readable text by default. Maintain
-explicit versioned rules for binaries, generated output, and vendored dependencies;
-do not treat working-tree ignore rules as a complete policy. Lockfiles are included
-unless an explicit generated/size rule excludes them.
+configuration, build definitions, and unknown readable text by default, subject to
+the versioned file-purpose policy. The preset pins `filePolicy: { version: 1, rules }`
+and its ordered repository-relative path rules. File selection and embedding
+eligibility are separate: source/lexical retrieval can remain available without a
+vector. Do not treat working-tree ignore rules as a complete policy.
+
+Credential filenames/directories, key/certificate payloads, image/binary assets
+and PEM private-key headers are hard exclusions, including for tracked files.
+Credential template suffixes (`.example`, `.sample`, `.template`) remain eligible.
+This is a targeted policy, not comprehensive secret detection. Dependencies,
+virtual environments, build/cache outputs, minified assets/maps and logs are
+excluded by default. Lockfiles, legal/author files, tabular/JSONL data, snapshots,
+and generated code (common suffixes or explicit generated/do-not-edit comments in the first 8 KiB) are
+lexical-only by default. Tests, ordinary docs, manifests and build/CI config remain
+eligible. Embedded license headers are not removed from source chunks.
+
+For Java files in `src/<source-set>/java/`, package components named `build`,
+`dist`, `target`, or `coverage` do not imply build output. Those directory names
+still exclude trees above the source root; dependency/cache directories retain
+their exclusions throughout. The source policy revision is part of preset identity.
+
+`--file-policy` accepts JSON on registration and local path dry runs. Last matching
+rule wins (`exclude`, `lexical`, `semantic`); rules use case-sensitive relative
+paths, `*`, `?` and whole-segment `**`. Rules can override default purpose decisions,
+but never hard content/path/size/encoding exclusions. The resolved policy is stored
+in the preset, not reread from a mutable checkout on import. `.gitattributes` is
+not interpreted. Inventory `retrieval`/`policyReason` and chunk skip reasons expose
+the decision. Changing rules changes configHash and cannot reuse a prior baseline.
+
+Evaluation labels require explicit exclusion independently of product defaults.
+Both checkout source evaluation harnesses pin `eval/source-corpus-policy.json`
+(excluding `eval/**`) and fingerprint it. New runs use new artifact roots; retained
+results and their old corpus/runtime identities remain historical evidence.
 
 | Special entry | Initial handling |
 | --- | --- |
 | Submodule | Record gitlink OID/path; no recursive import |
 | LFS pointer | Record metadata; no payload download or pointer indexing |
+| Image asset | Exclude known image extensions case-insensitively before chunking or embedding, including text SVG and portable pixel maps; retain inventory reason `image-extension` |
 | Symlink | Record target bytes; never follow or index as source |
 | Oversized file | Exclude with actual bytes and configured limit |
 | Invalid UTF-8 content/path | Record a diagnostic and lossless path identity; do not decode lossily |
 | Empty text file | File record, zero chunks |
 | Unsupported language or parse failure | Include bounded fallback chunks and distinguish the reason |
 
+Inline SVG markup in actual source files (for example, TSX components) remains
+source. The image exclusion policy participates in the immutable preset identity.
+
 Preserve raw UTF-8 bytes, BOM, CRLF, case, Unicode, and final newlines. Byte ranges
 are authoritative `[startByte, endByte)`; display lines are one-based/inclusive.
 JavaScript UTF-16 offsets must be converted explicitly. Preserve unusual Git paths
@@ -645,6 +860,25 @@ is not an unconditional one-function-per-chunk rule.
 
 ### Initial boundary and size rules
 
+The implementation adds Python/Go/Rust/C/C++ using the existing pinned WASM
+grammars and Shell/SQL using bundled, checksum-pinned grammars (see
+`runtime/grammars/README.md`).
+Python splits top-level declarations and class bodies, keeps decorators with their
+function/class, and records nested class scopes. Go keeps functions/methods,
+extracts receiver type scopes, and separates named types in grouped declarations.
+Rust splits `impl`, `trait` and inline module bodies, retaining generic type/trait
+scopes and outer attributes/doc comments with each declaration. Macros and `cfg`
+conditions remain authored source; there is no compiler expansion or evaluation.
+C/C++ walks declarators and namespace/class bodies, preserving templates and both
+preprocessor branches without evaluating conditions. `.h` uses C; C++ header
+extensions use C++. Shell covers `.sh`/`.bash` with the Bash grammar, keeping
+functions and compound commands intact. SQL keeps each top-level statement,
+including CTEs and supported quoted bodies, and records created object/schema
+names. Unknown dialect syntax uses whole-file parse fallback.
+Functions and SQL statements larger than the token ceiling use the common bounded text splitter with
+their symbol/scope preserved; nested functions stay in the containing function.
+These rules do not change Java/TypeScript/JavaScript boundaries.
+
 Prefer functions/methods and split large nodes along internal statement boundaries.
 Cover class headers, imports, types, fields, constants, top-level code, comments,
 punctuation, and gaps. Merge small adjacent units only within one scope and budget.
@@ -724,7 +958,7 @@ structural fragments, rather than build a complex value classifier.
 | Documentation, explanatory comments/docstrings | Embed; keep attached explanations with their code where practical |
 | Meaningful type/configuration declarations | Embed initially, then evaluate |
 | Imports-only or whitespace/punctuation-only fragments | Skip with a recorded reason |
-| Repeated license headers and mechanical listings | Candidates for explicit, versioned skip rules after evaluation |
+| Lexical-only file categories or rules | Keep exact source/chunks, omit embedding inputs and record the file policy reason; embedded license comments in authored code are not stripped |
 | Parser fallback | Do not exclude solely because parsing failed; apply content policy |
 
 Skipped chunks remain in inventory and source coverage. Retain lexical search text
@@ -740,13 +974,46 @@ choice and the finer skip rules remain evaluation decisions, not measured result
 
 ### Embedding reuse
 
-Cache embeddings by the exact full input and provider/model/revision/dimensions/
-preprocessing settings. Do not include commit, line numbers, file IDs, or build IDs
-in that input. A line shift can update positions while reusing the same vector;
-changed chunk boundaries or a renamed path in the input can legitimately miss.
-Cache only successful validated outputs atomically. Query embeddings use the same
-compatible contract and have their own cache/budget accounting. Provider aliases
-may drift if no immutable model revision exists; record that limitation.
+The managed implementation reuses unchanged records from the immutable baseline:
+compare all submitted payload fields after separately validating and removing only
+the server-generated vector. Do not upload those unchanged records again. Direct
+managed vector writes are forbidden, including copies fetched from an old Tag.
+Changed files receive new chunk IDs and may require re-embedding even if some text
+is identical. Replaying an uncertain write may also incur embedding usage. There
+is no srcx-managed document or query vector cache and no promised server cache.
+The older local cache remains only for internal client-embedder fixtures.
+
+The input includes path/symbol context and source text, not commit/line/build IDs.
+The preset pins provider/model/dimensions/similarity and input policy, but the
+provider model name is not an immutable model revision. Validation proves vector
+shape and payload integrity, not deterministic numerical reproduction by OpenAI.
+
+Search remains lexical by default. Explicit `--mode semantic` uses managed
+`knn.queryText`; `--mode hybrid` combines the existing lexical query and that kNN
+query using LambdaDB RRF. Both legs enforce `kind=chunk` and the same optional
+path/language filters; kNN uses a prefilter and `k=limit`, or `k=candidates`
+when optional reranking requests a wider pool. The lexical leg keeps
+nonembedded chunks eligible. Query/fetch/list request vectors explicitly so the
+same validation applies to pinned result handles. When a deployment omits managed
+vectors from list responses, fetch the affected IDs from that same immutable Tag
+and require identical non-vector payloads before vector validation. Source reads still use the
+original file bytes, never enriched text. Bounded retrieval and reranking results
+are recorded in [VALIDATION.md](VALIDATION.md); they do not establish a universal
+winner or authorize default changes.
+
+Optional `--rerank qwen` scores complete verified chunk bytes with a pinned local
+Qwen3-Reranker-0.6B revision, preserves candidate identity and retrieval order on
+ties, and returns a separate reranker score. Setup is explicit, inference is
+offline, and failures do not silently fall back. Every CLI invocation loads its
+own model. See [README.md](README.md#optional-local-qwen-reranker) for limits and
+runtime setup; this path does not change stored Collections or embedding presets.
+
+Managed source text and semantic/hybrid query text are sent through LambdaDB to
+OpenAI and incur embedding usage plus normal LambdaDB operations/storage. Dry-run
+and lexical queries do not request query embeddings. Use explicit opt-in and a
+bounded synthetic acceptance run before evaluating private or larger corpora.
+See [managed embeddings](https://docs.lambdadb.ai/guides/collections/managed-embeddings)
+and [hybrid query](https://docs.lambdadb.ai/guides/search/hybrid) for the API contract.
 
 ## 9. Candidate validation, publication, and `versions`
 
@@ -814,10 +1081,16 @@ they do not authorize the next commit on that workspace.
    attempt identity and verifier version. Use an ID derived from the full commit
    OID and candidate Snapshot ID so attempts cannot overwrite another Snapshot's
    evidence. Confirm its readback before publication.
-5. Create the deterministic published Tag **from the validated candidate Tag**,
-   not from the mutable Branch. Verify both Tags have the same Snapshot ID and
-   matching manifest. Publication changes naming, not corpus content.
-6. Update optional branch-progress/control records and return the published version.
+5. For a first publication, create the deterministic published Tag **from the
+   validated candidate Tag**, not from the mutable Branch. Verify both Tags have
+   the same Snapshot ID and matching manifest. If the commit is already published,
+   validate and reuse its canonical Tag; retain this writer's validated candidate
+   separately when its Snapshot ID differs. Publication never overwrites a commit
+   Tag with another writer's snapshot.
+6. For a tracked Git branch, update its control record with the validated writer
+   snapshot and published commit Tag, clear pending ownership, and return the
+   published version. Branch selectors keep using the preceding published Tag
+   until this control update succeeds.
    A delayed bookkeeping write does not erase an already published immutable Tag.
 
 An empty Snapshot or incomplete candidate never becomes published. A stale Tag
@@ -827,6 +1100,33 @@ avoid repeatedly validating known stale Snapshot IDs. A payload/build defect
 requires explicit same-build recovery, not endless polling. Publication must
 finish before the same code Branch starts a later commit.
 
+### Persistent Git branch state
+
+Each full `refs/heads/...` name maps to one `git-<ref-digest>` writer and one
+`branch-<full-ref-digest>` control document on main (`role=git-branch`). The control
+record holds the last applied immutable baseline, canonical published Tag name,
+and pending attempt/build/commit identity. Record pending ownership before branch
+creation or corpus writes. A missing local journal cannot authorize a second
+import over pending remote state, even if the committed head still matches the
+old baseline. Only the same journal may resume; automatic abandoned-attempt
+recovery remains unimplemented.
+
+Compare the writer head with the recorded baseline snapshot before the next
+import, then reconcile old IDs and hashes through its immutable Tag. This works
+without the prior local artifact and supports a branch moving backward or across
+history. Every target commit still comes from one full committed Git tree.
+
+An already published commit does not skip updating a different tracked branch.
+Validate that branch's own candidate, reuse the existing canonical commit Tag,
+and retain the candidate as the writer's exact baseline when the snapshot IDs
+differ. Never overwrite the canonical commit publication with another writer's
+snapshot. This also applies when a branch returns to a previously indexed commit.
+
+Branch search/resolve/read selects the last published Tag from the control record,
+including while the next import is pending. This promises the last indexed state,
+not the current Git tip. A first pending import has no searchable branch version.
+An ambiguous tracked branch/Git tag short name requires an explicit heads/tags ref.
+
 ### `versions` reads LambdaDB
 
 1. Resolve the repository Collection through remote metadata.
@@ -853,6 +1153,23 @@ name/Snapshot/manifest before retrying. Existing valid publication is reused.
 A crash after candidate validation but before publication can be recovered by
 revalidating that candidate and publishing; a `try-*` name alone is never enough.
 
+Within one invocation, reuse the successful complete candidate check instead of
+immediately scanning it twice. Before publication, require that the candidate Tag
+still points to the Snapshot returned when it was created. This result exists
+only in memory: resuming a publication-stage journal performs full candidate
+validation again and verifies the Tag/Snapshot identity. The final Tag is still
+created from the candidate Tag, with the existing Snapshot equality check.
+
+These optimizations retain full Git enumeration, local artifact copying and
+validation, full inventory construction, and at least one complete remote
+candidate verification. Cold recovery also lists the old remote baseline.
+Achieving remote validation proportional to a delta requires a verified backend
+contract tying a candidate Snapshot to the exact validated baseline plus all
+acknowledged upserts/deletes, preserving all other records and establishing index
+and managed-embedding readiness. No such contract is assumed by this importer;
+marker visibility, matching counts and changed-document sampling cannot replace
+the complete comparison. Dependency ingestion remains outside this workflow.
+
 ## 10. Git tags, exact version selection, and recovery
 
 ### Git release Alias synchronization
@@ -971,7 +1288,7 @@ Required meaningful checks:
 - Byte coverage, Unicode/BOM/CRLF/trailing-newline/long-line spans are exact.
 - Parse failures and huge nodes stay bounded and visible in reports.
 - B contains additions/replacements/deletions while A reads remain unchanged.
-- A controlled line-shift fixture updates positions and reuses identical embeddings.
+- A controlled line-shift fixture updates positions; managed imports may re-embed changed IDs, while the internal client-cache fixture verifies input-based reuse.
 - Different paths remain distinct even with identical bytes; reuse follows input.
 - Fixed-config mismatches fail before writes; fake vectors never enter a real corpus.
 - Explicit obsolete-ID deletion preserves replacement IDs and removes surplus chunks.
@@ -999,15 +1316,18 @@ latency, and operational cost are separate measurements.
 
 ## 12. Remaining decisions and inspected evidence
 
+The initial parser/tokenizer presets, managed embedding choices and CLI retrieval
+modes are implemented and have bounded evaluation records. The following are
+future decisions or per-run requirements, not blockers awaiting initial design:
+
 | Decision | When needed |
 | --- | --- |
-| First real repositories and parser priority | Real-repository evaluation; defaults do not block fixtures |
-| Exact parser/tokenizer/preset versions, enrichment and embedding skip rules | Bootstrap and retrieval evaluation; pin before publishing a corpus |
-| Chunk size/overlap and syntax-versus-window tradeoff | Section 8 pilot after retrieval works; current values are starting points |
-| Embedding model/dimensions/provider, data destination, spending cap | Before any real source upload or paid experiment |
-| LambdaDB endpoint/project and credentials | Provisioning and live fixture run |
-| Authoritative Git-tag source and observed-ref freshness policy | Manual Alias synchronization |
-| Concurrent multi-host writers, retention/cleanup, configuration migration | Later internal operation, outside first CLI |
+| Additional parser languages and production repositories | Onboarding source beyond the currently tested coverage |
+| Quality/latency thresholds and any default changes | After actual development-task use; before stable rollout |
+| New embedding/reranking providers or chunking policies | A concrete unmet retrieval need; Jev access remains deferred |
+| Source authorization, endpoint/project, credentials and spending limits | Before every new corpus upload or paid experiment |
+| Authoritative Git-tag source and observation freshness | Automatic synchronization or pruning |
+| Multi-host writers, retention/cleanup and configuration migration | Later internal operation, outside the first CLI |
 
 Sources and evidence used during this design discussion:
 
@@ -1044,13 +1364,13 @@ Sources and evidence used during this design discussion:
 - [Tree-sitter parsing](https://tree-sitter.github.io/tree-sitter/using-parsers/2-basic-parsing.html):
   source ranges and encodings; verify the selected binding during implementation.
 
-Development checkout references identify the inspected files, not runtime
-dependencies. No
-application tests, source uploads, embeddings, or live LambdaDB validation have
-been performed as part of writing this document.
+Development checkout references identify the files inspected during the original
+design discussion, not runtime dependencies. That design-only review did not run
+application tests, source uploads, embeddings or live LambdaDB checks. Subsequent
+implementation and validation evidence is recorded in [VALIDATION.md](VALIDATION.md).
 
-Document validation: the revised lexical `indexConfigs` JSON, description, four
-reserved metadata labels, and one optional context label passed the local built
+Historical design validation (September 25): the revised lexical `indexConfigs`
+JSON, description, four reserved metadata labels, and one optional context label passed the local built
 SDK's Collection-create input validator on September 25. All three JSON examples,
 the illustrative source span, then-local links, code fences, and whitespace also passed
 document checks. These checks do not establish server acceptance, marker ordering,