@functional-systems/srcx 0.1.0-dev.3 → 0.1.0-dev.31

package/CHANGELOG.md

@@ -2,6 +2,140 @@
 
 ## Unreleased
 
+- Look up explicit generated Collection names directly instead of discovering
+  every repository. Exact Collection names take precedence over aliases; only
+  missing Collections fall back to discovery. Preserve descriptor, schema and
+  immutable-version validation and ambiguity checks for repository names/keys.
+
+- Add optional `read --implementation` for contiguous Python and TypeScript/TSX
+  overload groups. Return the pinned implementation with accurate citations;
+  preserve the selected span when linking is unsupported, ambiguous or too large.
+  Keep indexing and search defaults unchanged.
+
+- Separate file selection from embedding eligibility. Exclude credential and
+  crypto payloads, virtual environments, caches and logs; keep lockfiles, generated
+  code, snapshots, tabular data and legal/author files available lexically without
+  embedding them. Fix case-sensitive minified-asset exclusions. Add pinned
+  `--file-policy` JSON rules for registration and local previews, explicit inventory
+  reasons and evaluation corpus exclusions for `eval/**`.
+
+- Exclude image assets, including text SVG and portable pixel maps, before
+  indexing or embedding. Match image extensions case-insensitively, preserve
+  exclusion metadata, and keep inline SVG in source files searchable. Include
+  the source policy change in preset identity.
+
+- Fix Shell indexing crashes on Bash comparisons such as `[[ "$value" == 1 ]]`
+  by bundling a runtime-compatible, checksum-pinned Bash grammar with its license.
+  Include the grammar in preset identity and installed-package validation.
+
+- Bundle the srcx-search agent skill in npm and add `skills install`, `update`,
+  `remove` and `status` for Codex and Claude Code, with user/project scopes,
+  explicit setup and preservation of customized or unowned files. Removing an
+  absent skill stays read-only even when its parent directory already exists.
+
+- Explain source-language query selection in README and search help, including
+  agent translation of query intent while preserving exact identifiers and
+  answering in the user's language. Keep multilingual input accepted without
+  adding automatic translation or changing retrieval defaults.
+
+- Keep standard text analysis as the default and add `--analyzers` for any combination of
+  LambdaDB's English, Korean, Japanese and standard analyzers. Pin the canonical
+  selection in repository/build identity and verify it during discovery. This
+  pre-release preset requires an analyzer list; older presets are not accepted.
+  Make publication readiness independent of stop-word/tokenization behavior.
+  Pin existing evaluation harnesses to standard regardless of product defaults.
+
+- Add Python/Go/Rust/C/C++/Shell/SQL syntax chunking with language-specific
+  declaration, function and statement boundaries, symbol/scope metadata and
+  bounded source-preserving splits. Bundle the pinned SQL grammar with licenses.
+
+- Distinguish Qwen setup, cache, device, input-size and timeout failures with
+  actionable messages while keeping raw worker diagnostics private.
+
+- Record three developer search/read investigations with concrete change maps,
+  bounded live usage and source verification, separate from ranking benchmarks.
+
+- Refresh implemented/remaining scope and verify the published dev.17 package
+  through a fresh npm consumer and bounded live first-use workflow.
+
+- Add opt-in local Qwen search reranking with bounded candidate pools, pinned
+  source evidence, separate scores and per-command timing.
+
+- Evaluate fixed-candidate CosQA semantic reranking with local Qwen, shared
+  duplicate scores, stable ties and separate official-ID/content metrics.
+
+- Cross-check the retained CosQA lexical baseline with an independent offline
+  BM25 ranker and the same StandardAnalyzer tokenization.
+
+- Add pinned CoIR CoSQA and MTEB CodeSearchNet Python/Go retrieval baselines,
+  sharing srcx query construction and using trec_eval metrics on official labels.
+
+- Add a bounded local Qwen reranking diagnostic with frozen inputs, durable
+  per-pair reservations and offline source-selection reporting.
+
+- Prepare offline reranking inputs from verified retained Click/Cobra candidates,
+  with separate evaluator labels, original ranks and source provenance.
+
+- Add a pending human review worksheet and fixed-candidate reranking experiment plan.
+
+- Run bounded, explicitly unreviewed Click/Cobra retrieval diagnostics with managed small and unchanged source labels.
+
+- Retain allowlisted CLI evaluation failure diagnostics without raw output or credentials.
+- Prepare an offline, unreviewed Click/Cobra question set with pinned source evidence.
+- Default CLI search/read evaluation across two pinned public repositories, with
+  alternative answer sets, exact source verification, and complete stdout accounting.
+- Internal reproducible lexical chunking evaluation with pinned source labels,
+  equal path-only enrichment, isolated live indexes, and token-budget metrics.
+
+### Added
+
+- Opt-in managed `text-embedding-3-large` (3072 dimensions) in separate pinned
+  Collections, with paired small/large evaluation and unchanged small identities.
+
+- Paired identifier/natural-language/mixed-query diagnostics on eight new tasks,
+  with verified candidate-rank coverage and offline stdout-budget analysis.
+- Bounded actual-CLI lexical/semantic/hybrid evaluation on identical managed
+  corpora, with reviewed and original labels, per-question regressions, stdout
+  token counts, command timings and durable request reservations.
+
+- Opt-in LambdaDB managed OpenAI `text-embedding-3-small` Collections, offline
+  embedding input previews, and explicit semantic/RRF hybrid search modes.
+- Candidate validation of generated vectors separately from exact source payloads,
+  unchanged-record reuse, and a bounded synthetic managed acceptance harness.
+
+- Persistent one-to-one Git branch / Collection Branch mapping, branch-local
+  incremental updates, and branch-selected search, resolve, and read against the
+  last published commit Tag.
+- Remote pending import ownership and resumable in-place branch updates, including
+  shared commit Tags across branches and non-fast-forward Git branch changes.
+- `test:live:branches` for repeatable synthetic live acceptance with retained
+  checkpoint, retry, and resource evidence.
+
+### Changed
+
+- Git branch imports update their fixed `git-*` writer; SHA/tag imports retain
+  manual workspaces. Existing publications and pending manual journals remain
+  usable. See README for migration and artifact behavior.
+- Ambiguous indexed branch/Git tag names require explicit `refs/heads/...` or
+  `refs/tags/...` selectors.
+
+### Fixed
+
+- Count failed and unsupported public-benchmark queries separately, retaining
+  both in the ranking metric denominators.
+
+- Fetch omitted managed vectors from query responses through the same immutable
+  Tag, preserving hit order/scores and requiring exact non-vector payload equality.
+
+- Reject an unprepared CLI evaluation run before creating its root/lock so a
+  subsequent offline preparation can use the same path.
+- Pin branch control before enumerating versions so a concurrent publication
+  cannot advance the selector beyond the version list being read.
+- Reject live checkpoint reuse across different code/harness inputs and preserve
+  the original validation revision and completion time on same-input reruns.
+
+## [0.1.0-dev.1] - 2026-09-25
+
 ### Added
 
 - Git commit imports, syntax-aware chunks, immutable LambdaDB versions, lexical
@@ -17,4 +151,6 @@
 - Pending imports resume without the previous local build artifact.
 - Search/read `--version` selects the corpus instead of printing the CLI version.
 
-No npm release has been published yet.
+Initial npm bootstrap published to the `dev` channel from commit
+`f94f947bf593b2f8498c07e130933b0ab444f868`. This local bootstrap has no
+GitHub Actions provenance. No stable release has been published.

package/DESIGN.md

@@ -1,10 +1,11 @@
 # srcx: Initial Implementation Design
 
-Status: initial CLI implemented with local tests and live synthetic A/B validation
-on LambdaDB. See [README.md](README.md) for runnable interfaces and
+Status: CLI implemented with persistent Git branch tracking, managed semantic/hybrid
+search and optional local Qwen reranking. Local, live and public-benchmark evidence
+have separate scopes. See [README.md](README.md) for runnable interfaces and
 [VALIDATION.md](VALIDATION.md) for the bounded live evidence and remaining scope.
 
-Updated: 2026-09-25.
+Updated: 2026-09-26.
 
 ## 1. Purpose, decisions, and implementation defaults
 
@@ -18,14 +19,13 @@ supporting each result. The project name is `srcx`.
 | --- | --- |
 | Project and display name | `srcx` |
 | Local directory | `srcx` |
-| Intended GitHub repository | `lambdadb/srcx` |
+| GitHub repository | `lambdadb/srcx` |
 | CLI executable | `srcx` |
-| Intended npm package | `@functional-systems/srcx` |
+| npm package | `@functional-systems/srcx` |
 | Short description | Version-aware code search on LambdaDB |
 
-The name is selected; the GitHub repository and npm package have not been created
-or published in this project. The package manifest and executable entry point are
-present in the initial CLI. Keep display/configuration naming centralized and
+The public GitHub repository and npm development channel are available. Current
+publication evidence and stable-release status are recorded in RELEASING.md. Keep display/configuration naming centralized and
 distribution identity in `package.json`. Database IDs and the existing
 `code-*` Collection naming and `purpose=code-search-v1` protocol marker remain
 independent of the product name.
@@ -40,7 +40,7 @@ Confirmed boundaries from the discussion:
 - Map each Git tag to a LambdaDB Alias targeting its published commit Tag.
   Multiple Git tags for the same commit share one commit Tag and Snapshot.
 - Keep search results and subsequent source reads pinned to the selected version.
-- Replace a changed file's entire chunk set initially; cache embeddings by input.
+- Replace a changed file's entire chunk set initially; retain unchanged remote records.
 - Keep core LambdaDB work, GitHub App/webhooks, UI, MCP, and a public SaaS outside
   the first deliverable. sbrain reference documents remain unchanged.
 - No remote repository creation, application commit, push, publication, real
@@ -53,7 +53,7 @@ Current recommendations, distinguished from user-confirmed choices:
 | Runtime | TypeScript and a supported Node.js release compatible with the existing SDK |
 | Database | LambdaDB only; no public backend selection or interchangeable database framework |
 | API key | Read `LAMBDADB_API_KEY` by default; optional environment-variable-name override |
-| Syntax | Java and TypeScript/JavaScript first; other text uses recorded fallback |
+| Syntax | Java, TypeScript/JavaScript, Python, Go, Rust, C/C++, Shell and SQL; other text uses recorded fallback |
 | Source | Exact UTF-8 source inside file documents, with local blob cache |
 | File size | Start with a 1 MiB raw-source limit and explicit exclusions |
 | Embedding | A fixed `none` preset first; with a model, embed meaningful code and prose chunks while recording policy-based skips |
@@ -72,9 +72,10 @@ This document is not a deployed-revision audit or evidence of a passing live run
 
 The initial command set below is implemented; run it as `node dist/cli.js` from
 this checkout after `npm run build`. See README for exact options and recovery.
-The first implementation uses fresh `work-*` Branches and observed-local-tag Alias
-synchronization; persistent tracked branches and authoritative pruning remain
-follow-up work. Immutable manifest `requestedRef` is normalized to the resolved
+Git branch imports now reuse persistent `git-*` Branches; explicit commit/tag
+imports use frozen `work-*` writers. Observed-local-tag Alias synchronization is
+implemented; automatic Git observation and authoritative pruning remain follow-up
+work. Immutable manifest `requestedRef` is normalized to the resolved
 commit OID; the
 original user spelling remains in the local build artifact. The broader contract
 below remains the target design.
@@ -379,9 +380,10 @@ readers on multiple machines do not require local registration state.
    changed/deleted file, compute obsolete IDs as previous file/chunk IDs minus
    replacement IDs. A deleted file has an empty replacement set. Obtain previous
    IDs from the inventory, not mutable query-by-path results.
-6. Apply the pinned per-chunk embedding policy. If embeddings are enabled, batch
-   eligible cache misses across chunks/files within the provider's token/request
-   limits. Validate successful vectors before caching.
+6. Apply the pinned per-chunk embedding policy. The managed preset materializes
+   eligible `embeddingText` inputs without calling a provider locally. LambdaDB
+   generates vectors during ordinary upsert. Skip unchanged remote records and
+   validate generated vectors in the candidate Tag before publication.
    Bound queue memory; do not load an entire repository's content into RAM.
 7. Materialize deterministic records and deletion lists as local build artifacts.
    Produce size-bounded inventory parts and a root manifest template. Check all
@@ -389,8 +391,10 @@ readers on multiple machines do not require local registration state.
    journal supplies the separate attempt ID when sealing the final manifest.
 
 Dry-run executes the same inventory/chunking path but performs no database mutation
-or embedding call. For pending embedding misses, report tokens and projected vector
-size separately; do not claim a completed vector/records hash. Report every entry,
+or embedding call. Report `managed` eligible chunks and `managedTokens` (estimated
+input tokens for the complete corpus), with `embedded=0` in the local artifact.
+These are not billed usage or incremental upload counts. Record hashes bind the
+submitted payload; server-generated vectors are validated separately. Report every entry,
 exclusion reason, parse status, byte size, chunk/token counts, embedding eligibility
 and skip reasons, and change summary. Dry-run plans may show pending embedding work;
 published chunk records may not silently treat missing required vectors as skips.
@@ -473,11 +477,12 @@ fields are omitted when absent; records of different kinds need not share them.
 | Record | Additional stored fields |
 | --- | --- |
 | File (`kind=file`) | `path`, `blobOid`, `contentHash`, `byteLength`, `sourceText`, `language`, `parseStatus`, `chunkCount` |
-| Chunk (`kind=chunk`) | `fileId`, `path`, parent `contentHash`, `language`, `chunkKind`, `startByte`, `endByte`, `startLine`, `endLine`, `ordinal`, `searchText`, `embeddingStatus`; optional `symbol`, `scope`, `signature`, `embeddingSkipReason`, `embeddingInputHash`, `embedding` |
+| Chunk (`kind=chunk`) | `fileId`, `path`, parent `contentHash`, `language`, `chunkKind`, `startByte`, `endByte`, `startLine`, `endLine`, `ordinal`, `searchText`, `embeddingStatus`; optional `symbol`, `scope`, `signature`, `embeddingSkipReason`, `embeddingInputHash`, `embeddingText`, `embedding` |
 | Corpus root (`kind=manifest`, `role=corpus`, ID `__manifest__`) | `repoId`, `indexId`, full `commitOid`, `commitTime`, `requestedRef`, full `config`, `inventoryHash`, `recordsHash`, `counts`, `buildId`, `attemptId`, `inventoryPartIds` |
 | Inventory part (`kind=manifest`, `role=inventory`) | `partOrdinal`, `partHash`, ordered `entries` with path, file/chunk IDs, blob/content identities, inclusion/exclusion reasons, parse states, sizes, and embedding coverage |
 | Repository control (`kind=manifest`, `role=repository`, main ID `__repo__`) | Full source identity, repo/index IDs, immutable preset/config, initialization state |
-| Ref control (`kind=manifest`, `role=git-ref`, main) | Full Git ref name, encoded Alias/Branch name, observed commit/time, applied version and synchronization state |
+| Git tag control (`kind=manifest`, `role=git-ref`, main) | Full Git tag ref, encoded Alias name, observed commit, applied version and synchronization state |
+| Git branch control (`kind=manifest`, `role=git-branch`, main) | Full Git branch ref, encoded Branch name, validated writer baseline, canonical published Tag and pending attempt identity |
 | Validation summary (`kind=manifest`, `role=validation`, main) | Candidate name/Snapshot, build/config/attempt identity, verifier version and checked counts/hashes; no premature claim of publication |
 
 `startByte`/`endByte` are half-open; display lines are one-based/inclusive.
@@ -486,8 +491,13 @@ content such as function, declaration, documentation, imports, or fallback; it i
 separate from the file/chunk/manifest discriminator. `counts` includes included and
 excluded files, chunks, and embedded/skipped chunks with reason totals.
 
-In a completed corpus, `embeddingStatus=embedded` requires a valid vector and
-`embeddingInputHash`, and omits the skip reason. `embeddingStatus=skipped` requires
+The managed preset uses `embeddingStatus=managed` to identify its generation policy,
+not local completion. Eligible chunks carry `embeddingText=searchText` and an
+`embeddingInputHash` over the exact input and embedding configuration; ordinary
+upserts omit `embedding`. A completed candidate must contain a nonzero, finite
+1536-number vector for every eligible chunk. Local artifacts contain no vectors.
+The internal fixture-only client embedder retains `embeddingStatus=embedded`;
+its arbitrary presets cannot be published through the CLI. `embeddingStatus=skipped` requires
 an explicit policy reason and omits the vector and input hash. The lexical-only
 preset records `embedding-disabled`; an embedding-enabled preset records its
 specific exclusion reason. Pending or failed required embedding work stays in the
@@ -543,15 +553,24 @@ is stored in the file; enriched chunk search text is never the citation authorit
 `sourceText`, role, byte/line positions, hashes, commit identity, scope/signature,
 embedding diagnostics, and inventory/config objects are stored without entries in
 `indexConfigs`. Public documentation states that fields are stored even when not
-indexed. Do not invent an `index: false` option. The `standard` analyzer is a
-baseline, not a proven code-specific ranking solution. An initial enrichment
+indexed. Do not invent an `index: false` option. The default text analyzer is
+`standard`; users can choose any nonempty combination of `english`, `japanese`,
+`korean`, and `standard`. The sorted, deduplicated list is part of the pinned
+preset/configuration hash and configures both `searchText` and managed
+`embeddingText`. Discovery requires the matching schema; a different selection
+uses a different Collection. Analyzer selection does not change embedding input
+text or the programming-language chunker. No analyzer is a proven universal
+code-search ranking solution. An initial enrichment
 candidate preserves original identifiers while adding camelCase/snake_case terms;
 pin and evaluate that transformation before adopting it as the preset.
 
-An embedding-enabled preset adds an unmanaged `embedding` vector field with a
-concrete model-selected dimension and similarity metric, fixed before Collection
-creation. An indexed vector field does not require a vector on every chunk; file
-and manifest records also omit it. The `none` preset omits the vector index entirely.
+The opt-in managed presets add indexed text `embeddingText` and a managed
+`embedding` vector field with provider `openai`: `text-embedding-3-small` uses
+1536 dimensions and `text-embedding-3-large` uses 3072, both with cosine similarity.
+The source field, dimensions and similarity are nested under the vector field's `embedding` configuration; there are no
+top-level dimension/similarity properties. Collection discovery verifies the
+complete resolved schema and pinned preset. Missing `embeddingText` on skipped
+chunks, files and manifests means no vector is generated for those records. The `none` preset omits the vector index entirely.
 Changing presets creates a separate index; do not silently mutate an existing
 serving index. Do not write placeholder or test vectors into a real corpus.
 
@@ -593,21 +612,49 @@ must include stored-only source round-trip and mixed record kinds.
 ## 8. File policy, chunking, and embedding reuse
 
 Every Git tree entry must be accounted for. Include source, tests, documents,
-configuration, build definitions, and unknown readable text by default. Maintain
-explicit versioned rules for binaries, generated output, and vendored dependencies;
-do not treat working-tree ignore rules as a complete policy. Lockfiles are included
-unless an explicit generated/size rule excludes them.
+configuration, build definitions, and unknown readable text by default, subject to
+the versioned file-purpose policy. The preset pins `filePolicy: { version: 1, rules }`
+and its ordered repository-relative path rules. File selection and embedding
+eligibility are separate: source/lexical retrieval can remain available without a
+vector. Do not treat working-tree ignore rules as a complete policy.
+
+Credential filenames/directories, key/certificate payloads, image/binary assets
+and PEM private-key headers are hard exclusions, including for tracked files.
+Credential template suffixes (`.example`, `.sample`, `.template`) remain eligible.
+This is a targeted policy, not comprehensive secret detection. Dependencies,
+virtual environments, build/cache outputs, minified assets/maps and logs are
+excluded by default. Lockfiles, legal/author files, tabular/JSONL data, snapshots,
+and generated code (common suffixes or explicit generated/do-not-edit comments in the first 8 KiB) are
+lexical-only by default. Tests, ordinary docs, manifests and build/CI config remain
+eligible. Embedded license headers are not removed from source chunks.
+
+`--file-policy` accepts JSON on registration and local path dry runs. Last matching
+rule wins (`exclude`, `lexical`, `semantic`); rules use case-sensitive relative
+paths, `*`, `?` and whole-segment `**`. Rules can override default purpose decisions,
+but never hard content/path/size/encoding exclusions. The resolved policy is stored
+in the preset, not reread from a mutable checkout on import. `.gitattributes` is
+not interpreted. Inventory `retrieval`/`policyReason` and chunk skip reasons expose
+the decision. Changing rules changes configHash and cannot reuse a prior baseline.
+
+Evaluation labels require explicit exclusion independently of product defaults.
+Both checkout source evaluation harnesses pin `eval/source-corpus-policy.json`
+(excluding `eval/**`) and fingerprint it. New runs use new artifact roots; retained
+results and their old corpus/runtime identities remain historical evidence.
 
 | Special entry | Initial handling |
 | --- | --- |
 | Submodule | Record gitlink OID/path; no recursive import |
 | LFS pointer | Record metadata; no payload download or pointer indexing |
+| Image asset | Exclude known image extensions case-insensitively before chunking or embedding, including text SVG and portable pixel maps; retain inventory reason `image-extension` |
 | Symlink | Record target bytes; never follow or index as source |
 | Oversized file | Exclude with actual bytes and configured limit |
 | Invalid UTF-8 content/path | Record a diagnostic and lossless path identity; do not decode lossily |
 | Empty text file | File record, zero chunks |
 | Unsupported language or parse failure | Include bounded fallback chunks and distinguish the reason |
 
+Inline SVG markup in actual source files (for example, TSX components) remains
+source. The image exclusion policy participates in the immutable preset identity.
+
 Preserve raw UTF-8 bytes, BOM, CRLF, case, Unicode, and final newlines. Byte ranges
 are authoritative `[startByte, endByte)`; display lines are one-based/inclusive.
 JavaScript UTF-16 offsets must be converted explicitly. Preserve unusual Git paths
@@ -645,6 +692,25 @@ is not an unconditional one-function-per-chunk rule.
 
 ### Initial boundary and size rules
 
+The implementation adds Python/Go/Rust/C/C++ using the existing pinned WASM
+grammars and Shell/SQL using bundled, checksum-pinned grammars (see
+`runtime/grammars/README.md`).
+Python splits top-level declarations and class bodies, keeps decorators with their
+function/class, and records nested class scopes. Go keeps functions/methods,
+extracts receiver type scopes, and separates named types in grouped declarations.
+Rust splits `impl`, `trait` and inline module bodies, retaining generic type/trait
+scopes and outer attributes/doc comments with each declaration. Macros and `cfg`
+conditions remain authored source; there is no compiler expansion or evaluation.
+C/C++ walks declarators and namespace/class bodies, preserving templates and both
+preprocessor branches without evaluating conditions. `.h` uses C; C++ header
+extensions use C++. Shell covers `.sh`/`.bash` with the Bash grammar, keeping
+functions and compound commands intact. SQL keeps each top-level statement,
+including CTEs and supported quoted bodies, and records created object/schema
+names. Unknown dialect syntax uses whole-file parse fallback.
+Functions and SQL statements larger than the token ceiling use the common bounded text splitter with
+their symbol/scope preserved; nested functions stay in the containing function.
+These rules do not change Java/TypeScript/JavaScript boundaries.
+
 Prefer functions/methods and split large nodes along internal statement boundaries.
 Cover class headers, imports, types, fields, constants, top-level code, comments,
 punctuation, and gaps. Merge small adjacent units only within one scope and budget.
@@ -724,7 +790,7 @@ structural fragments, rather than build a complex value classifier.
 | Documentation, explanatory comments/docstrings | Embed; keep attached explanations with their code where practical |
 | Meaningful type/configuration declarations | Embed initially, then evaluate |
 | Imports-only or whitespace/punctuation-only fragments | Skip with a recorded reason |
-| Repeated license headers and mechanical listings | Candidates for explicit, versioned skip rules after evaluation |
+| Lexical-only file categories or rules | Keep exact source/chunks, omit embedding inputs and record the file policy reason; embedded license comments in authored code are not stripped |
 | Parser fallback | Do not exclude solely because parsing failed; apply content policy |
 
 Skipped chunks remain in inventory and source coverage. Retain lexical search text
@@ -740,13 +806,46 @@ choice and the finer skip rules remain evaluation decisions, not measured result
 
 ### Embedding reuse
 
-Cache embeddings by the exact full input and provider/model/revision/dimensions/
-preprocessing settings. Do not include commit, line numbers, file IDs, or build IDs
-in that input. A line shift can update positions while reusing the same vector;
-changed chunk boundaries or a renamed path in the input can legitimately miss.
-Cache only successful validated outputs atomically. Query embeddings use the same
-compatible contract and have their own cache/budget accounting. Provider aliases
-may drift if no immutable model revision exists; record that limitation.
+The managed implementation reuses unchanged records from the immutable baseline:
+compare all submitted payload fields after separately validating and removing only
+the server-generated vector. Do not upload those unchanged records again. Direct
+managed vector writes are forbidden, including copies fetched from an old Tag.
+Changed files receive new chunk IDs and may require re-embedding even if some text
+is identical. Replaying an uncertain write may also incur embedding usage. There
+is no srcx-managed document or query vector cache and no promised server cache.
+The older local cache remains only for internal client-embedder fixtures.
+
+The input includes path/symbol context and source text, not commit/line/build IDs.
+The preset pins provider/model/dimensions/similarity and input policy, but the
+provider model name is not an immutable model revision. Validation proves vector
+shape and payload integrity, not deterministic numerical reproduction by OpenAI.
+
+Search remains lexical by default. Explicit `--mode semantic` uses managed
+`knn.queryText`; `--mode hybrid` combines the existing lexical query and that kNN
+query using LambdaDB RRF. Both legs enforce `kind=chunk` and the same optional
+path/language filters; kNN uses a prefilter and `k=limit`, or `k=candidates`
+when optional reranking requests a wider pool. The lexical leg keeps
+nonembedded chunks eligible. Query/fetch/list request vectors explicitly so the
+same validation applies to pinned result handles. When a deployment omits managed
+vectors from list responses, fetch the affected IDs from that same immutable Tag
+and require identical non-vector payloads before vector validation. Source reads still use the
+original file bytes, never enriched text. Bounded retrieval and reranking results
+are recorded in [VALIDATION.md](VALIDATION.md); they do not establish a universal
+winner or authorize default changes.
+
+Optional `--rerank qwen` scores complete verified chunk bytes with a pinned local
+Qwen3-Reranker-0.6B revision, preserves candidate identity and retrieval order on
+ties, and returns a separate reranker score. Setup is explicit, inference is
+offline, and failures do not silently fall back. Every CLI invocation loads its
+own model. See [README.md](README.md#optional-local-qwen-reranker) for limits and
+runtime setup; this path does not change stored Collections or embedding presets.
+
+Managed source text and semantic/hybrid query text are sent through LambdaDB to
+OpenAI and incur embedding usage plus normal LambdaDB operations/storage. Dry-run
+and lexical queries do not request query embeddings. Use explicit opt-in and a
+bounded synthetic acceptance run before evaluating private or larger corpora.
+See [managed embeddings](https://docs.lambdadb.ai/guides/collections/managed-embeddings)
+and [hybrid query](https://docs.lambdadb.ai/guides/search/hybrid) for the API contract.
 
 ## 9. Candidate validation, publication, and `versions`
 
@@ -814,10 +913,16 @@ they do not authorize the next commit on that workspace.
    attempt identity and verifier version. Use an ID derived from the full commit
    OID and candidate Snapshot ID so attempts cannot overwrite another Snapshot's
    evidence. Confirm its readback before publication.
-5. Create the deterministic published Tag **from the validated candidate Tag**,
-   not from the mutable Branch. Verify both Tags have the same Snapshot ID and
-   matching manifest. Publication changes naming, not corpus content.
-6. Update optional branch-progress/control records and return the published version.
+5. For a first publication, create the deterministic published Tag **from the
+   validated candidate Tag**, not from the mutable Branch. Verify both Tags have
+   the same Snapshot ID and matching manifest. If the commit is already published,
+   validate and reuse its canonical Tag; retain this writer's validated candidate
+   separately when its Snapshot ID differs. Publication never overwrites a commit
+   Tag with another writer's snapshot.
+6. For a tracked Git branch, update its control record with the validated writer
+   snapshot and published commit Tag, clear pending ownership, and return the
+   published version. Branch selectors keep using the preceding published Tag
+   until this control update succeeds.
    A delayed bookkeeping write does not erase an already published immutable Tag.
 
 An empty Snapshot or incomplete candidate never becomes published. A stale Tag
@@ -827,6 +932,33 @@ avoid repeatedly validating known stale Snapshot IDs. A payload/build defect
 requires explicit same-build recovery, not endless polling. Publication must
 finish before the same code Branch starts a later commit.
 
+### Persistent Git branch state
+
+Each full `refs/heads/...` name maps to one `git-<ref-digest>` writer and one
+`branch-<full-ref-digest>` control document on main (`role=git-branch`). The control
+record holds the last applied immutable baseline, canonical published Tag name,
+and pending attempt/build/commit identity. Record pending ownership before branch
+creation or corpus writes. A missing local journal cannot authorize a second
+import over pending remote state, even if the committed head still matches the
+old baseline. Only the same journal may resume; automatic abandoned-attempt
+recovery remains unimplemented.
+
+Compare the writer head with the recorded baseline snapshot before the next
+import, then reconcile old IDs and hashes through its immutable Tag. This works
+without the prior local artifact and supports a branch moving backward or across
+history. Every target commit still comes from one full committed Git tree.
+
+An already published commit does not skip updating a different tracked branch.
+Validate that branch's own candidate, reuse the existing canonical commit Tag,
+and retain the candidate as the writer's exact baseline when the snapshot IDs
+differ. Never overwrite the canonical commit publication with another writer's
+snapshot. This also applies when a branch returns to a previously indexed commit.
+
+Branch search/resolve/read selects the last published Tag from the control record,
+including while the next import is pending. This promises the last indexed state,
+not the current Git tip. A first pending import has no searchable branch version.
+An ambiguous tracked branch/Git tag short name requires an explicit heads/tags ref.
+
 ### `versions` reads LambdaDB
 
 1. Resolve the repository Collection through remote metadata.
@@ -971,7 +1103,7 @@ Required meaningful checks:
 - Byte coverage, Unicode/BOM/CRLF/trailing-newline/long-line spans are exact.
 - Parse failures and huge nodes stay bounded and visible in reports.
 - B contains additions/replacements/deletions while A reads remain unchanged.
-- A controlled line-shift fixture updates positions and reuses identical embeddings.
+- A controlled line-shift fixture updates positions; managed imports may re-embed changed IDs, while the internal client-cache fixture verifies input-based reuse.
 - Different paths remain distinct even with identical bytes; reuse follows input.
 - Fixed-config mismatches fail before writes; fake vectors never enter a real corpus.
 - Explicit obsolete-ID deletion preserves replacement IDs and removes surplus chunks.
@@ -999,15 +1131,18 @@ latency, and operational cost are separate measurements.
 
 ## 12. Remaining decisions and inspected evidence
 
+The initial parser/tokenizer presets, managed embedding choices and CLI retrieval
+modes are implemented and have bounded evaluation records. The following are
+future decisions or per-run requirements, not blockers awaiting initial design:
+
 | Decision | When needed |
 | --- | --- |
-| First real repositories and parser priority | Real-repository evaluation; defaults do not block fixtures |
-| Exact parser/tokenizer/preset versions, enrichment and embedding skip rules | Bootstrap and retrieval evaluation; pin before publishing a corpus |
-| Chunk size/overlap and syntax-versus-window tradeoff | Section 8 pilot after retrieval works; current values are starting points |
-| Embedding model/dimensions/provider, data destination, spending cap | Before any real source upload or paid experiment |
-| LambdaDB endpoint/project and credentials | Provisioning and live fixture run |
-| Authoritative Git-tag source and observed-ref freshness policy | Manual Alias synchronization |
-| Concurrent multi-host writers, retention/cleanup, configuration migration | Later internal operation, outside first CLI |
+| Additional parser languages and production repositories | Onboarding source beyond the currently tested coverage |
+| Quality/latency thresholds and any default changes | After actual development-task use; before stable rollout |
+| New embedding/reranking providers or chunking policies | A concrete unmet retrieval need; Jev access remains deferred |
+| Source authorization, endpoint/project, credentials and spending limits | Before every new corpus upload or paid experiment |
+| Authoritative Git-tag source and observation freshness | Automatic synchronization or pruning |
+| Multi-host writers, retention/cleanup and configuration migration | Later internal operation, outside the first CLI |
 
 Sources and evidence used during this design discussion:
 
@@ -1044,13 +1179,13 @@ Sources and evidence used during this design discussion:
 - [Tree-sitter parsing](https://tree-sitter.github.io/tree-sitter/using-parsers/2-basic-parsing.html):
   source ranges and encodings; verify the selected binding during implementation.
 
-Development checkout references identify the inspected files, not runtime
-dependencies. No
-application tests, source uploads, embeddings, or live LambdaDB validation have
-been performed as part of writing this document.
+Development checkout references identify the files inspected during the original
+design discussion, not runtime dependencies. That design-only review did not run
+application tests, source uploads, embeddings or live LambdaDB checks. Subsequent
+implementation and validation evidence is recorded in [VALIDATION.md](VALIDATION.md).
 
-Document validation: the revised lexical `indexConfigs` JSON, description, four
-reserved metadata labels, and one optional context label passed the local built
+Historical design validation (September 25): the revised lexical `indexConfigs`
+JSON, description, four reserved metadata labels, and one optional context label passed the local built
 SDK's Collection-create input validator on September 25. All three JSON examples,
 the illustrative source span, then-local links, code fences, and whitespace also passed
 document checks. These checks do not establish server acceptance, marker ordering,