@functional-systems/srcx 0.1.0-dev.4 → 0.1.0-dev.40

package/README.md

@@ -19,10 +19,86 @@ srcx --version
 srcx --help
 ```
 
+## Install the agent skill
+
+The npm package includes the [srcx-search skill](skills/srcx-search/SKILL.md).
+After installing the CLI, explicitly select your agent:
+
+```sh
+srcx skills install --agent codex
+# Or, for Claude Code:
+srcx skills install --agent claude
+srcx skills status --agent codex
+```
+
+The default scope is `user`. Installation copies the bundled instructions into
+one of the following local discovery directories:
+
+| Agent       | User scope                              | Project scope                                   |
+| ----------- | --------------------------------------- | ----------------------------------------------- |
+| Codex       | `~/.agents/skills/srcx-search/SKILL.md` | `<project>/.agents/skills/srcx-search/SKILL.md` |
+| Claude Code | `~/.claude/skills/srcx-search/SKILL.md` | `<project>/.claude/skills/srcx-search/SKILL.md` |
+
+These are the documented local skill locations for
+[Codex](https://learn.chatgpt.com/docs/build-skills#where-codex-loads-local-skills)
+and [Claude Code](https://code.claude.com/docs/en/skills#where-skills-live).
+Cloud sessions need their own installation; a local user installation does not
+configure remote agents or account-synced skills. The installer targets these
+default locations, not custom agent configuration directories.
+
+To share a skill with a project, select an existing project directory and commit
+the installed file if appropriate for your team:
+
+```sh
+srcx skills install --agent codex --scope project --path /path/to/project
+```
+
+Project scope defaults to the current directory when `--path` is omitted; it does
+not infer the Git root. `--path` is accepted only for project scope. Prefer one
+scope per agent to avoid duplicate skill names. The installer does not change
+`AGENTS.md`, `CLAUDE.md`, agent permissions, shell profiles or credentials, and npm
+installation alone does not place skills in your environment.
+
+Make sure the agent's shell can run `srcx --version`. Configure LambdaDB and
+prepare the repository using the [connected workflow](#connected-workflow) below,
+or use an already indexed repository in the configured project. The agent process
+must inherit the API-key environment variable named by `srcx configure`; no key
+is copied into the skill. `srcx doctor` checks authentication/read access only.
+
+Start a new agent session after installation. Invoke `$srcx-search` in Codex or
+`/srcx-search` in Claude Code, and ask it to investigate an indexed repository at a
+specific branch, release or commit. Skill-aware agents may also select it from its
+description. Installation/status confirms files on disk, not that an agent loaded
+or selected them. Agent policy, disabled skills and custom discovery paths can
+still affect availability.
+
+### Update and remove
+
+```sh
+npm install -g @functional-systems/srcx@dev
+srcx skills update --agent codex
+srcx skills status --agent codex
+srcx skills remove --agent codex
+```
+
+Repeat for each installed agent/scope, supplying the same `--scope project --path`
+when applicable. `update` uses the currently running CLI's bundled skill and does
+not download anything. npm upgrades do not silently update skill copies, and npm
+uninstall does not remove them: run `skills remove` before uninstalling the CLI.
+
+A provenance comment records the package version and content hash. Repeated
+installation is a no-op when the installed copy matches. A different managed
+version requires `update`. Locally edited skills, extra files, unowned skills and
+symlinked managed paths are preserved by refusing the operation; there is no
+force-overwrite flag. Move a customized copy to a separate skill name/location
+before installing a managed copy. Interrupted operations can leave a sibling
+`.srcx-search.srcx-lock` directory: check that no installer is running and preserve
+any staged file before manually removing that lock and retrying.
+
 ## Run locally
 
-Requires Node.js 22.14+ and Git. The initial preset uses lexical search with
-`embedding=none`; no embedding account or paid API is needed.
+Requires Node.js 22.14+ and Git. The default preset uses lexical search with
+`embedding=none` and the `standard` text analyzer; no embedding account or paid API is needed.
 
 ```sh
 npm ci --ignore-scripts
@@ -63,19 +139,413 @@ and tag names that collide require `refs/heads/...` or `refs/tags/...`. No autom
 fetch, history replay, or revision expressions are supported.
 
 Each artifact contains `build.json` (inventory, configuration, counts, hashes,
-change/deletion plan) and `records.jsonl` (exact file records and retrieval chunks).
+change/deletion plan) and, by default, `records.jsonl` (exact file records and
+retrieval chunks).
+
+For an experimental alternative that reduces retained-record rewrites, use
+`--artifact-storage segments` when creating a new build:
+
+```sh
+srcx import --path /path/to/repo --ref main --dry-run \
+  --artifact-storage segments --output /tmp/srcx-segmented-a
+srcx import --path /path/to/repo --ref next --dry-run \
+  --artifact-storage segments --previous /tmp/srcx-segmented-a \
+  --output /tmp/srcx-segmented-b
+```
+
+Format 2 stores `build.json` and digest-addressed JSONL objects under `objects/`.
+Unchanged objects can share filesystem storage through hardlinks, while each
+artifact remains usable after deleting the previous directory. Full local source
+and record/hash checks still run; publication retains complete candidate
+validation. Fewer bytes are written, but more
+file operations can make builds slower; descriptor construction and full checks
+still scale with corpus size. Segmented mode requires native no-follow and
+nonblocking file-open support. Cross-filesystem/unsupported-link copies rewrite the
+retained bytes too. See [local build measurements](VALIDATION.md).
+
+The default is `--artifact-storage jsonl`; storage does not change the embedding
+preset or require a new Collection. Preview JSON includes `artifactFormat` (1 or 2).
+Use `--artifact <directory>` to import/resume either format without a storage flag;
+an explicit `--artifact-storage` cannot override an existing artifact. Older CLIs
+accept only format 1, and tools that directly read `records.jsonl` need the shared
+artifact reader to support format 2. Linked files are not tamper-proof: full checks
+detect shared-object corruption. Preserve the pending artifact and journal until
+publication completes.
+
+Local retention is manual: remove whole completed artifact directories, keeping
+any pending artifact and its original retry journal. Unchanged hardlinked objects
+remain available through surviving artifacts. Losing the latest local artifact
+requires a full local rebuild unless a surviving compatible checkpoint is
+explicitly selected with `--previous` during a dry-run build; checkpoint discovery
+is not automatic. Import that rebuilt artifact with `--artifact`.
+Local cleanup does not delete published versions or their immutable remote data.
+See [cumulative history and recovery measurements](VALIDATION.md#cumulative-segmented-history-and-recovery)
+for the storage/time tradeoff across 50 successive versions.
+
 Artifacts contain source code; keep them with the same access policy as the repo.
 Every tree entry is accounted for, including explicit exclusions for symlinks,
-submodules, LFS pointers, dependencies/build output, binary/invalid UTF-8 data,
+submodules, LFS pointers, dependencies/build output, image assets (including SVG), binary/invalid UTF-8 data,
 invalid UTF-8 paths, and files above 1 MiB. Excluded paths are preserved losslessly
 in `pathBase64`.
+Image extensions are matched case-insensitively and excluded before chunking or
+embedding, even when their payload is readable text. Source files containing
+inline images, such as a TSX component with SVG markup, remain included.
+
+File selection and embedding eligibility are separate:
+
+| Default treatment                                   | Examples                                                                                                                                                                                                                                                        |
+| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Exclude source and chunks                           | Credential files (`.env`, `.env.*`, `.npmrc`, `.netrc`, `.aws/`, `.ssh/`), key/certificate payloads, images, binaries, installed dependencies (`.venv/`, `node_modules/`, `vendor/`, `third_party/`), build/cache output, logs, minified assets and source maps |
+| Keep source and lexical chunks; skip embeddings     | Lockfiles, license/author files, CSV/TSV/JSONL data, snapshots, generated code identified by common suffixes or an explicit generated/do-not-edit comment in the first 8 KiB                                                                                    |
+| Keep source, lexical chunks and eligible embeddings | Authored code, tests, documentation, package manifests, CI/build configuration and other readable text                                                                                                                                                          |
+
+`.env.example`, `.env.sample`, `.env.template` and equivalent suffixes remain
+included as templates. Private-key PEM headers also exclude files with unrelated
+extensions. These are targeted exclusions, **not a general secret scanner**;
+review the dry-run inventory before uploading. `.gitignore` does not remove
+already tracked Git objects, and `.gitattributes` is not interpreted for selection.
+
+Java files under `src/<source-set>/java/` keep package directories named `build`,
+`dist`, `target` or `coverage`. For example, `src/main/java/example/build/Builder.java`
+is source, while `build/generated/src/main/java/example/Builder.java` is excluded.
+Dependency and cache exclusions still apply inside source roots. Other ambiguous
+layouts can use explicit file-policy rules below.
+
+Use `--file-policy policy.json` with `repo add` or `import --dry-run --path` to
+adjust the scope explicitly. The JSON contents are pinned in the repository
+preset; subsequent connected imports use that preset, not the local policy file.
+
+```json
+{
+  "version": 1,
+  "rules": [
+    { "pattern": "eval/**", "action": "exclude" },
+    { "pattern": "third_party/special/**", "action": "lexical" },
+    { "pattern": "generated/api.pb.go", "action": "semantic" }
+  ]
+}
+```
+
+Patterns match case-sensitive, repository-relative paths. `*` and `?` stay within
+one path segment; `**` must occupy a whole segment and matches any depth. Rules
+run in order; the last match wins. `semantic` permits embeddings when enabled but
+still skips structural/import-only chunks. Rules override dependency/generated/
+data defaults, but cannot override credential, crypto, image, binary, private-key
+content, size, encoding, symlink or LFS exclusions. `retrieval` and `policyReason`
+in dry-run coverage explain each included file's policy; chunk records retain
+`embeddingSkipReason`. A changed policy selects a distinct Collection/configuration.
+
+```sh
+srcx import --path . --ref develop --dry-run --embedding text-embedding-3-small --file-policy policy.json --output /tmp/srcx-preview
+srcx repo add --path . --embedding text-embedding-3-small --file-policy policy.json
+```
+
+Evaluation labels need an explicit corpus boundary. The checkout evaluation
+harnesses use [this policy](eval/source-corpus-policy.json) to exclude `eval/**`;
+labels stored elsewhere require additional exclusions before a new evaluation.
+General-purpose imports do not discard all evaluation or test directories.
+
+Java, TypeScript/TSX, JavaScript/JSX, Python, Go, Rust, C/C++, Shell and SQL use
+pinned Tree-sitter WASM grammars.
+
+| Language filter | File extensions                                    | Boundaries and metadata                                                                                     |
+| --------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
+| `python`        | `.py`, `.pyi`                                      | Decorators, functions, classes, method scopes and docstrings                                                |
+| `go`            | `.go`                                              | Named types, functions and generic/pointer/value receiver scopes                                            |
+| `rust`          | `.rs`                                              | Functions, types and `impl`/`trait`/module scopes; outer attributes and doc comments stay with declarations |
+| `c`             | `.c`, `.h`                                         | Functions, pointer declarators, types and preprocessor branches                                             |
+| `cpp`           | `.cpp`, `.cc`, `.cxx`, `.C`, `.hpp`, `.hh`, `.hxx` | Functions, templates, namespace/class scopes and qualified names                                            |
+| `shell`         | `.sh`, `.bash`                                     | Bash-compatible functions and compound commands, including heredocs                                         |
+| `sql`           | `.sql`                                             | Statements, CTEs and supported function bodies; created object names/schema scopes                          |
+
+`.h` files use the C grammar deterministically; C++ headers should use a listed
+C++ extension for syntax chunking. Shell does not infer extensionless scripts from
+shebangs or claim Zsh/Fish/PowerShell support. Shell and SQL use bundled
+[grammar artifacts](runtime/grammars/README.md); SQL dialect-specific syntax unsupported
+by that grammar falls back to text. SQL strings and supported dollar-quoted bodies
+are parsed as part of their containing statement, not split at each semicolon.
+Rust macros/`cfg` and C/C++ preprocessor conditions are not expanded or evaluated;
+search uses authored source, including both conditional branches.
+
+Markdown uses heading/paragraph/fence boundaries; configuration text uses
+section/line boundaries. Parse errors and unsupported languages use recorded text
+fallback. Chunks cover the exact source bytes, carry one-based line ranges, target
+800 tokens, and stay below 1,500 tokens including path/symbol context. The internal
+window baseline is available to the test/build API, not as a public storage backend.
+
+Added-language functions and SQL statements stay together when they fit the token
+ceiling. Larger units use bounded text splitting with the same symbol/scope
+metadata; nested functions remain inside their containing function. Syntax
+unsupported by the pinned grammars uses recorded whole-file parse fallback.
+Language labels remain available to `--language` filters even after fallback.
+
+## Choose text analyzers
+
+New repositories use `standard` by default. Choose any nonempty combination of
+LambdaDB's supported analyzers: `english`, `korean`, `japanese`, and `standard`.
+These analyze searchable text; they are separate from programming-language
+chunking and the `--language` file filter.
+
+```sh
+srcx repo add --path /path/to/repo --analyzers english,korean
+srcx repo add --path /path/to/repo --analyzers english
+
+# Preview exactly the same settings without credentials or service calls.
+srcx import --path /path/to/repo --ref main --dry-run \
+  --analyzers english,korean --output /tmp/srcx-preview
+```
 
-Java, TypeScript/TSX, and JavaScript use pinned Tree-sitter WASM grammars. Markdown
-uses heading/paragraph/fence boundaries; configuration text uses section/line
-boundaries. Parse errors and unsupported languages use recorded text fallback.
-Chunks cover the exact source bytes, carry one-based line ranges, target 800 tokens,
-and stay below 1,500 tokens including path/symbol context. The internal window
-baseline is available to the test/build API, not as a public storage backend.
+Order and duplicates do not matter: `korean,english,english` selects the same
+preset as `english,korean`. Unknown names and empty entries are rejected.
+The chosen analyzers are stored in the repository preset, included in its
+configuration hash, and checked against the Collection schema during discovery.
+Different analyzer sets select separate Collections. Connected imports and
+searches use the registered preset; `--analyzers` on import is only accepted with
+`--dry-run --path`. Inspect the selection with `srcx repo show --repo <collection>`.
+
+This pre-release preset now requires an explicit analyzer list. Older descriptors
+and artifacts without it are not accepted; existing Collections are not changed
+in place. A new registration/import uses the chosen preset.
+
+Analyzers affect lexical search and the lexical part of hybrid search. They do
+not translate queries or change the raw text sent to the embedding provider.
+The same selection configures `searchText` and managed `embeddingText` text indexes.
+See [LambdaDB text indexes](https://docs.lambdadb.ai/guides/collections/index-types#text).
+
+## Opt into managed embeddings
+
+Supported models are `text-embedding-3-small` (1536 dimensions) and
+`text-embedding-3-large` (3072 dimensions), both cosine. Use either name with
+`--embedding`; each model/analyzer combination has a separate pinned preset and
+Collection. Changing models requires a separate import.
+
+LambdaDB generates vectors for meaningful code, tests and prose. Imports-only
+and structural chunks remain lexical. The CLI uses LambdaDB credentials; it does not need an OpenAI key.
+Source text sent for embedding and semantic/hybrid queries pass through LambdaDB
+to OpenAI and incur usage charges. See [LambdaDB managed embeddings](https://docs.lambdadb.ai/guides/collections/managed-embeddings).
+
+```sh
+# Offline preview: no upload or embedding request.
+srcx import --path /path/to/repo --ref main --dry-run \
+  --embedding text-embedding-3-small --output /tmp/srcx-managed-preview
+
+# Creates a separate Collection; existing lexical Collections stay usable.
+srcx repo add --path /path/to/repo --embedding text-embedding-3-small
+# Use the exact collection value returned above, especially with multiple presets.
+srcx import --repo <collection> --ref main
+srcx search --repo <collection> --version main --query "retry failed writes" --mode hybrid
+srcx search --repo <collection> --version main --query "retry failed writes" --mode semantic
+```
+
+Connected imports, including `--dry-run --repo`, use the selected repository's
+pinned preset. `--embedding` on import is only for `--dry-run --path`.
+`search` keeps `--mode lexical` as its default; semantic and hybrid require a
+managed Collection. Hybrid uses LambdaDB RRF with identical filters on both legs.
+Existing result handles, lexical config hashes and publication journals stay valid.
+
+Local artifacts contain eligible `embeddingText` and input hashes, not vectors.
+`counts.managed` and `counts.managedTokens` estimate the complete corpus's eligible
+chunks and input tokens; `embedded=0` does not mean the server will skip them.
+Publication requires valid generated vectors in the immutable candidate Tag,
+plus the same exact payload/source checks as lexical publication. Unchanged
+records are not uploaded again. Changed file chunks and uncertain write retries
+may be embedded again; no global or query embedding cache is promised.
+
+A small synthetic live check is available after building:
+
+```sh
+npm run build
+node --env-file=.env.local scripts/live-managed.mjs
+```
+
+It uploads two synthetic commits, bounds document inputs to 10,000 estimated
+tokens and issues four managed search queries plus lexical checks. It retains a
+Collection, local fixture, artifacts, result handles and `.srcx/live-managed/report.json`.
+Completed reruns preserve the original report and make no service calls; changed
+runtime inputs require a fresh directory. Failed runs retain journals and reject
+automatic replay. These are correctness checks, not retrieval-quality measurements.
+
+## Optional local Qwen reranker
+
+`search --rerank qwen` reranks candidates from any retrieval mode with
+[Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B/blob/e61197ed45024b0ed8a2d74b80b4d909f1255473/README.md).
+It is off by default. Prepare Python 3.12+ and the model separately; npm does not
+install Python dependencies or download model weights. From a checkout:
+
+```sh
+python3.12 -m venv .srcx/qwen-venv
+.srcx/qwen-venv/bin/pip install -r runtime/requirements.txt
+.srcx/qwen-venv/bin/hf download Qwen/Qwen3-Reranker-0.6B \
+  --revision e61197ed45024b0ed8a2d74b80b4d909f1255473 \
+  config.json generation_config.json model.safetensors tokenizer.json tokenizer_config.json vocab.json merges.txt
+export SRCX_RERANK_PYTHON="$PWD/.srcx/qwen-venv/bin/python"
+
+srcx search --repo <collection> --version <commit> \
+  --query "retry failed writes" --mode semantic \
+  --rerank qwen --candidates 50 --limit 10
+```
+
+For a global npm installation, the requirements file is at
+`$(npm root -g)/@functional-systems/srcx/runtime/requirements.txt`; place the venv
+in a persistent directory of your choice and use its absolute Python path.
+`HF_HUB_CACHE` can select an existing Hugging Face cache for both download and
+search. The worker loads only the pinned revision from that cache, offline.
+`SRCX_RERANK_DEVICE` accepts `auto` (default: CUDA, then MPS, then CPU), `cuda`,
+`mps`, or `cpu`. The current worker uses float32, SDPA and batch size one.
+
+`--candidates` defaults to `max(50, limit)` and must be between `--limit` and 100.
+It requires `--rerank qwen`. Both hybrid retrieval legs use this candidate limit;
+path/language filters and the immutable version apply before reranking. The
+worker receives the query and complete, verified source bytes of each chunk,
+without a path prefix or evaluator labels. Identical source text shares a score
+within the query; every candidate remains present and ties retain retrieval order.
+It uses the code-search instruction from the CoSQA evaluation and ranks by the
+raw yes-minus-no logit margin, not a calibrated relevance probability.
+
+JSON stdout remains an array of results. `score` retains the retrieval score;
+`rerankScore`, `retrievalRank` (one-based), and `reranker` identify the new order.
+Only the returned results receive persisted evidence handles. Reranked excerpts
+use the exact chunk bytes; normal reads still expand to complete source lines.
+A JSON timing record on stderr reports verified retrieval, worker, search, and
+command-handler times. Worker time includes Python startup, model loading and
+inference. External process timing additionally includes Node startup and output.
+
+Each CLI invocation loads a fresh model; this is an opt-in local integration,
+not a warm inference service. The worker has a 120-second timeout, a 4 MiB input
+cap and an 8,192-token limit per query/chunk pair. Missing setup, oversized input,
+invalid scores and inference failures fail the command without silent truncation
+or fallback. Empty candidate pools skip model execution. Existing search defaults
+and managed embedding usage rules remain unchanged; wider pools increase source
+verification work. See the [integration validation](VALIDATION.md#local-reranker-integration).
+
+Failures include a fixed remediation message where the cause is known:
+
+| Failure                                     | Action                                                                                               |
+| ------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
+| Python executable missing or not executable | Check the path and execute permissions in `SRCX_RERANK_PYTHON`.                                      |
+| Missing or incompatible Python imports      | Install `runtime/requirements.txt` using that Python environment.                                    |
+| Missing or incomplete model cache           | Download the pinned revision and all files listed above; check `HF_HUB_CACHE`.                       |
+| Invalid or unavailable device               | Use `SRCX_RERANK_DEVICE=auto` or `cpu`, or an available `mps`/`cuda` device.                         |
+| Request exceeds 4 MiB                       | Reduce `--candidates` or shorten the query.                                                          |
+| A query/chunk pair exceeds 8,192 tokens     | Shorten the query or omit `--rerank`; reducing candidate count does not shorten an individual chunk. |
+| Worker exceeds 120 seconds                  | Reduce `--candidates` or select an available faster device.                                          |
+
+Unknown worker failures remain generic. Raw subprocess output and library
+exceptions are never included in these messages. Failed reranking does not
+silently return the original ranking; rerun without `--rerank` to opt out.
+
+For practical search → read → change-planning examples, see the
+[developer workflow pilot](https://github.com/lambdadb/srcx/blob/develop/eval/DEVELOPER-WORKFLOW-PILOT.md).
+It follows three maintenance investigations through related source and tests,
+including the limitations of previews and optional reranking.
+
+The [agent workflow diagnostic](https://github.com/lambdadb/srcx/blob/develop/eval/AGENT-WORKFLOW-PILOT.md)
+compares fresh agent sessions with Git/local tools and optional or explicitly
+invoked srcx. It found no established efficiency advantage on the small public
+corpora and records the remaining workflow hypotheses.
+
+The [semantic agent follow-up](https://github.com/lambdadb/srcx/blob/develop/eval/SEMANTIC-AGENT-RESULTS.md)
+compares local, lexical-first and semantic-first investigations on Requests. It
+finds no established agent efficiency gain and records an unresolved candidate
+retrieval discrepancy and excessive embedding of a generated SVG asset.
+
+## Query language and agent use
+
+Queries can use any language; the CLI does not translate them. For lexical search,
+match the language of the source you want to find. Prefer English search terms for
+English code and comments, preserving identifiers, API names and error messages
+exactly. For Korean documentation, use Korean terms in a Collection registered
+with the `korean` analyzer. Analyzers tokenize text; they do not translate between
+languages. The `--language` search filter selects a programming language, not the
+natural language of the query.
+
+For example, an agent answering the Korean question “데이터베이스 연결을 닫을 때
+어떻게 되나요?” can search English source with:
+
+```sh
+srcx search --repo <collection> --version <commit> \
+  --query "What happens when closing a database connection?"
+```
+
+This is a query-writing example, not a measured retrieval result. Read the returned
+source before answering. Semantic and hybrid search can help across languages,
+but relevant implementation evidence is not guaranteed to appear in the first
+results. English wording is a starting point, not an input requirement or a
+promise of better ranking for every question.
+
+Agents can use the following search guidance:
+
+> Choose query wording for the target source language, independently of the user's
+> conversation language. For English code and comments, translate the user's
+> intent into a concise English query. Preserve supplied identifiers, API names,
+> paths and error messages exactly; do not invent implementation names or assume
+> the question's premise is correct. Use the original language when seeking
+> documentation written in that language. Read results at the pinned version and
+> check that they support the answer. If evidence is incomplete, inspect relevant
+> surrounding code or try another query based on observed source. Answer in the
+> user's language and cite the source; state when evidence is missing.
+
+## Evaluate retrieval quality
+
+The [fresh agent navigation comparison](https://github.com/lambdadb/srcx/blob/develop/eval/FRESH-AGENT-NAVIGATION-RESULTS.md)
+measures complete answers, model input and wall time with local tools available in
+both conditions. It does not establish a clear efficiency advantage on a small
+local repository; retrieval quality and agent-workflow benefits remain separate.
+
+The [cross-repository version workflow](https://github.com/lambdadb/srcx/blob/develop/eval/VERSION-WORKFLOW-RESULTS.md)
+verifies exact source delivery across CLI/SDK releases from a fresh client without
+local source attachments. It is a scripted known-range diagnostic, not an agent
+benchmark. Local Git reads are much faster in this run; repeated project-wide
+repository discovery is a measured source of remote overhead.
+The subsequent [exact-Collection lookup replay](https://github.com/lambdadb/srcx/blob/develop/eval/DIRECT-LOOKUP-RESULTS.md)
+removes 22 requests per command in that project, with identical source results
+and lower observed read/search latency. It does not measure complete-agent cost.
+
+In the subsequent [agent choice comparison](https://github.com/lambdadb/srcx/blob/develop/eval/VERSION-AGENT-RESULTS.md),
+all three agents offered srcx read its installed skill but used local tools only
+to investigate the prepared CLI/SDK versions. No srcx retrieval benefit was
+established; multiple repositories and historical versions alone did not lead to
+tool adoption when their source trees were already available locally.
+
+The internal [lexical chunking pilot](https://github.com/lambdadb/srcx/blob/develop/eval/README.md) compares syntax-aware and
+window chunks on a fixed public corpus and 16 pinned investigation queries.
+`npm run eval:prepare` builds and validates the comparison offline. Live evaluation
+is an explicit separate operation and retains isolated evaluation Collections.
+
+The [default CLI workflow evaluation](https://github.com/lambdadb/srcx/blob/develop/eval/CLI-WORKFLOW.md)
+adds a second public repository and verifies real `search` → `read` results with
+the unchanged CLI preset. It uses normal repository Collections and separate local
+state; see its documented effects before opting into the live run.
+
+The [retrieval mode comparison](https://github.com/lambdadb/srcx/blob/develop/eval/RETRIEVAL-MODES.md) runs lexical, semantic and
+hybrid against the same managed corpora. It retains original and reviewed labels,
+full stdout accounting, command timings and bounded embedding request estimates.
+`npm run eval:modes:prepare` is offline; live execution is explicit and incurs
+managed embedding and LambdaDB usage.
+
+The [query-style diagnostic](https://github.com/lambdadb/srcx/blob/develop/eval/QUERY-STYLES.md)
+pairs identifiers, natural-language descriptions and mixed questions on eight new
+tasks. It adds rank-prefix and stdout-budget measurements without changing the
+model or ranking. [Embedding model candidates](https://github.com/lambdadb/srcx/blob/develop/eval/EMBEDDING-MODELS.md)
+separately describe possible follow-up comparisons and integration requirements.
+The [managed model comparison](https://github.com/lambdadb/srcx/blob/develop/eval/MODEL-COMPARISON.md)
+compares small and large on both frozen suites, including lexical controls.
+[Results](https://github.com/lambdadb/srcx/blob/develop/eval/MODEL-COMPARISON-RESULTS.md)
+show mixed gains and regressions; small remains the initial managed choice.
+
+[Transfer evaluation preparation](https://github.com/lambdadb/srcx/blob/develop/eval/TRANSFER-EVAL.md) adds source-verified,
+unreviewed Python/Go questions. The [completed development diagnostic](https://github.com/lambdadb/srcx/blob/develop/eval/TRANSFER-RESULTS.md)
+compares all three modes with managed small and text fallback; it is not an
+independently reviewed benchmark.
+The [local reranking diagnostic](https://github.com/lambdadb/srcx/blob/develop/eval/RERANK-RESULTS.md)
+compares Qwen selection on the same saved candidates, with separate offline
+coverage and compute accounting.
+The [public retrieval baseline](https://github.com/lambdadb/srcx/blob/develop/eval/PUBLIC-BENCHMARK.md)
+uses pinned CoIR CoSQA and MTEB CodeSearchNet Python/Go tasks with official
+labels and trec_eval metrics. It evaluates retrieval separately from Git import,
+chunking and CLI reads.
+New evaluation roots retain [safe command failure diagnostics](https://github.com/lambdadb/srcx/blob/develop/eval/COMMAND-DIAGNOSTICS.md)
+for explicit recovery.
 
 ## Explicit live acceptance run
 
@@ -96,39 +566,114 @@ It saves connection settings under `.srcx/live/config.json` and keeps its state
 separate under `.srcx/live/state`. Rerunning resumes the saved synthetic run and
 refuses to reuse it against a different destination.
 
+For persistent Git branch acceptance, run `npm run test:live:branches`. It uses a
+separate synthetic Collection and `.srcx/live-branches/` state/report, covering
+in-place A/B updates, last-published reads during a pending update, shared commit
+Tags across two branches, rewinds, no-op imports, and SHA-only manual imports.
+From an isolated worktree, load the original checkout's env file explicitly:
+
+```sh
+npm run build
+node --env-file=/absolute/path/to/srcx/.env.local scripts/live-branches.mjs
+```
+
+The branch harness retains resources and resumes saved checkpoints on rerun.
+It preserves the original source revision and completion time. Checkpoints from
+different runtime code, harness, fixture, lockfile, or Node version are rejected
+before connecting or overwriting evidence. To validate changed code, select a new
+`SRCX_LIVE_BRANCHES_DIR` (for example `.srcx/live-branches-next`); retain the old
+run and any pending journal. Its per-import deadline defaults to 300 seconds and accepts
+`SRCX_LIVE_TIMEOUT_MS` up to 600000. Do not delete pending journals to start over.
+
 ## Connected workflow
 
+Use the installed `srcx` binary below. For a source checkout, build first and
+replace `srcx` with `node dist/cli.js`.
+
 These commands create and write LambdaDB resources. Select an approved project
-and source scope before running them. Start live validation with the synthetic
-fixture, not a production repository.
+and source scope before running them. Start live validation with a small synthetic
+fixture. `repo add` returns a `collection` value: use that value as `<name>` below
+to disambiguate repositories with multiple presets. If isolating a first-use run,
+set `SRCX_CONFIG` and `SRCX_STATE_DIR` to fresh paths before `configure`; keep
+credentials in the shell and preserve these paths for later result reads.
 
 ```sh
 # Supply LAMBDADB_API_KEY through your shell or secret manager.
-node dist/cli.js configure --endpoint https://api.lambdadb.ai --project <project>
-node dist/cli.js doctor
-node dist/cli.js repo add --path /path/to/repo \
+srcx configure --endpoint https://api.lambdadb.ai --project <project>
+srcx doctor
+srcx repo add --path /path/to/repo \
   --description "Repository source, tests, and documentation for code search." \
   --tag team=search
-node dist/cli.js repo list
-node dist/cli.js repo show --repo <name>
+srcx repo list
+srcx repo show --repo <name>
 
 # Import the exact artifact reviewed in the credential-free preview.
-node dist/cli.js import --repo <name> --artifact /tmp/srcx-build-a
-node dist/cli.js versions --repo <name>
-node dist/cli.js search --repo <name> --version <full-commit-A> --query retry
-node dist/cli.js read --result <result-id> --context 20
-node dist/cli.js read --result <result-id> --full-file
-node dist/cli.js read --repo <name> --version <full-commit-A> \
+srcx import --repo <name> --artifact /tmp/srcx-build-a
+srcx versions --repo <name>
+
+# Keep a Git branch connected to its Collection Branch.
+srcx import --repo <name> --ref develop
+srcx search --repo <name> --version develop --query retry
+srcx resolve --repo <name> --ref refs/heads/develop
+srcx search --repo <name> --version <full-commit-A> --query retry
+srcx read --result <result-id> --context 20
+srcx read --result <result-id> --implementation
+srcx read --result <result-id> --full-file
+srcx read --repo <name> --version <full-commit-A> \
   --path src/example.ts --lines 10:30
 
-node dist/cli.js import --repo <name> --artifact /tmp/srcx-build-b
+srcx import --repo <name> --artifact /tmp/srcx-build-b
 # An old A result still reads A after B is published.
-node dist/cli.js read --result <result-id> --full-file
+srcx read --result <result-id> --full-file
 
-node dist/cli.js git sync-tags --repo <name>
-node dist/cli.js resolve --repo <name> --ref v1.0.0
+srcx git sync-tags --repo <name>
+srcx resolve --repo <name> --ref v1.0.0
 ```
 
+Prefer the exact `collection` value returned by `repo list` when the repository
+is already known. Generated Collection names (`code-<slug>-<hash>`) are looked up
+directly and take precedence over repository aliases with the same spelling.
+Repository names and keys still use discovery and reject ambiguous matches across
+presets. Only a missing Collection (HTTP 404) falls back to alias discovery;
+permission, service, descriptor and schema errors remain errors. This does not
+cache or bypass immutable-version and source-integrity checks.
+
+`lookup --symbol <name>` returns exact, case-sensitive indexed symbol matches
+and their verified `sourceText` in one call:
+
+```sh
+srcx lookup --repo <collection> --version <resolved-commit> \
+  --symbol readToFile --limit 5
+```
+
+Use a name found in the question, search results or source. For `Type.method`,
+look up `method` and inspect the returned `scope` and `path`; use `--path` or
+`--language` to narrow ambiguity. Lookup uses the existing keyword index and
+requires no managed embeddings or reindexing. Results include definitions and
+declarations, including same-name symbols and pieces of token-split definitions.
+They are source chunks, not guaranteed complete function bodies or resolved
+call targets. The limit defaults to five chunks and accepts 1–20. Results are
+bounded, not an exhaustive symbol inventory; an empty result is not proof of
+absence. `resultId` remains usable with `read` for surrounding lines or a full
+file. Supplied source text can be cited directly without another read.
+
+`read --implementation` follows a contiguous Python or TypeScript/TSX overload
+group to its unique implementation in the same file and AST scope. It reads the
+stored source at the result's immutable Tag; no reindexing or embedding is needed.
+Python supports direct `from typing`/`typing_extensions import overload` imports,
+including aliases used only as decorators. Stub files, ambiguous/interrupted
+groups, unrecognized decorators and parse failures retain the original span.
+Other languages and cross-file/type-based dispatch are not resolved. This is
+syntax navigation, not proof of runtime binding or behavior.
+
+The returned `implementation.status` is `resolved`, `not-linked`, `unsupported`,
+`parse-error` or `too-large`. Resolved reads return the body's actual lines and
+citation. Multiple overload hits may return the same body; the option does not
+deduplicate results or skip reads. Bodies exceeding 1,500 source tokens retain the
+original span with `too-large`; no silent truncation is applied. Ordinary reads
+and search rankings are unchanged. The option requires `--result` and cannot be
+combined with `--context`, `--lines` or `--full-file`.
+
 `doctor` checks authentication and Collection read access, not write permission
 or indexing/query readiness. `configure --api-key-env NAME` selects another
 credential environment variable; only its name is stored. Endpoint/project changes
@@ -142,7 +687,9 @@ identity or Collection name. Registration normalizes the selected Git remote;
 without remotes receive a persistent local source identity. Repeated registration
 preserves existing descriptions/labels and attaches the local checkout.
 
-Search is literal analyzed lexical text, restricted to `kind=chunk`. Optional
+Search defaults to literal analyzed lexical text. All retrieval modes are
+restricted to `kind=chunk`; semantic/hybrid and optional Qwen reranking are
+described above. Optional
 `--path` and `--language` filters are exact keyword filters. `--limit` is 1–100.
 Search results include a durable local `resultId`, commit, immutable Tag/Snapshot,
 path, lines, and citation. Reads verify the original content hash and refuse
@@ -152,10 +699,28 @@ are one-based and inclusive.
 ## Publication and recovery
 
 The control Branch `main` holds descriptors/version summaries. `checkpoint-empty`
-is forked before any control write. Each import uses a new `work-*` Branch,
-optionally forked from the last locally retained, remotely validated writer
-baseline. Writers are frozen after publication. A fresh local state can start a
-full import from `checkpoint-empty`; source data is never forked from a Tag.
+is forked before any control write. Importing a Git branch creates or updates its
+stable `git-*` Collection Branch, derived from the full `refs/heads/...` name.
+Git `main` also uses a `git-*` Branch; Collection `main` remains control-only.
+Each later import reconciles against that branch's last validated snapshot,
+including deletions and non-fast-forward changes. Branches sharing a commit share
+one canonical `ver-*` Tag while retaining their own writer snapshots.
+
+`search/read --version develop` and `resolve --ref develop` select the branch's
+last successfully published commit Tag. During an import or after a failed update,
+they keep serving the previous published commit; the first import is unavailable
+until publication. This is the last indexed state, not a live Git HEAD lookup.
+Nothing polls or fetches Git automatically: run import again to advance the branch.
+If an indexed branch and synchronized Git tag share a short name, use
+`refs/heads/<name>` or `refs/tags/<name>` explicitly. Known Git names take precedence
+over commit prefixes.
+
+A commit SHA or Git tag import does not infer a containing branch. An unpublished
+commit uses a fresh `work-*` Branch, optionally forked from a validated previous
+manual writer; an already published commit reuses its Tag. Manual writers remain
+frozen. Source data is never forked from a Tag. A clean tracked branch can be
+updated from fresh local state using its remote baseline. Local build reuse is
+cached separately for each Git branch and for manual imports.
 
 The importer deletes obsolete IDs, upserts changed records and inventory parts in
 bounded sequential requests, then sends `__manifest__` **alone** after every prior
@@ -167,7 +732,10 @@ an early/stale snapshot even after the marker is observed.
 
 A per-Collection local lock and durable `pending.json` journal prevent another
 commit from overwriting a partial attempt. Only one importing host is supported.
-After a timeout or unknown write result, retain the artifacts and journal:
+A tracked branch also records pending import ownership in control `main` before
+writing code. If the local journal is lost, new local state refuses to overwrite
+that pending branch even when its committed head has not advanced yet. After a
+timeout or unknown write result, retain the artifacts and journal:
 
 ```sh
 node dist/cli.js import --repo <name> --artifact /path/from/pending.json \
@@ -186,6 +754,22 @@ remove that specific lock directory. Do not delete `pending.json` to force a new
 commit over an uncertain workspace. Unexpected baselines fail closed; automatic
 recovery to a new workspace and resource cleanup are not implemented yet.
 
+Incremental publication can reuse a fully validated local baseline artifact's
+record hashes after matching its repository/configuration, commit, build and
+inventory/record hashes to the pinned remote baseline. Missing local files or a
+different cached build fall back to listing that immutable baseline; corrupted
+matching artifacts fail validation. Unchanged inventory parts also need no write.
+Every new candidate still receives full document/hash and managed-vector
+validation. A successful check is reused only within that invocation, after
+confirming the candidate Tag still names the same Snapshot. Publication-stage
+retries repeat full validation rather than trusting the journal.
+
+This reduces redundant remote reads, not import complexity to the changed-code
+size. Git tree enumeration, local full-artifact materialization/validation,
+inventory construction and at least one complete candidate read remain. Further
+reductions need a verified backend contract proving both mutation completeness
+and preservation of unchanged baseline data; the final marker alone is insufficient.
+
 Git tag synchronization observes local `refs/tags/*` only. Multiple Git tags for
 one commit create separate `rel-*` Aliases targeting the same published Tag.
 An unimported/moved target is pending; the previously pinned version remains usable
@@ -195,8 +779,17 @@ For search and resolve, a synchronized Git tag name takes precedence over a comm
 prefix, matching Git import's tag-before-OID rule. A pending tag fails explicitly
 instead of falling back to a matching commit prefix. Use `refs/tags/<name>` to
 select a Git tag explicitly.
-Explicit authoritative pruning and persistent tracked-branch synchronization are
-follow-up work; no full history import is required.
+Automatic Git observation, authoritative pruning, branch rename/deletion handling,
+and garbage collection remain follow-up work; no full history import is required.
+
+Existing commit Tags and frozen `work-*` Branches remain valid. The next import
+using a Git branch name establishes its fixed mapping, even if that commit was
+already published. A pending journal from the previous implementation resumes on
+its original `work-*` writer; import the branch again after recovery to establish
+tracking. `--artifact` uses the Git branch recorded when the artifact was built,
+with its pinned commit; it does not reread the branch's current tip. The publication
+JSON continues to describe the canonical commit Tag: its `writer` can be the
+original publishing branch when multiple Git branches share that commit.
 
 ## State and implementation map