@functional-systems/srcx 0.1.0-dev.23 → 0.1.0-dev.25

package/CHANGELOG.md

@@ -2,6 +2,27 @@
 
 ## Unreleased
 
+- Separate file selection from embedding eligibility. Exclude credential and
+  crypto payloads, virtual environments, caches and logs; keep lockfiles, generated
+  code, snapshots, tabular data and legal/author files available lexically without
+  embedding them. Fix case-sensitive minified-asset exclusions. Add pinned
+  `--file-policy` JSON rules for registration and local previews, explicit inventory
+  reasons and evaluation corpus exclusions for `eval/**`.
+
+- Exclude image assets, including text SVG and portable pixel maps, before
+  indexing or embedding. Match image extensions case-insensitively, preserve
+  exclusion metadata, and keep inline SVG in source files searchable. Include
+  the source policy change in preset identity.
+
+- Fix Shell indexing crashes on Bash comparisons such as `[[ "$value" == 1 ]]`
+  by bundling a runtime-compatible, checksum-pinned Bash grammar with its license.
+  Include the grammar in preset identity and installed-package validation.
+
+- Bundle the srcx-search agent skill in npm and add `skills install`, `update`,
+  `remove` and `status` for Codex and Claude Code, with user/project scopes,
+  explicit setup and preservation of customized or unowned files. Removing an
+  absent skill stays read-only even when its parent directory already exists.
+
 - Explain source-language query selection in README and search help, including
   agent translation of query intent while preserving exact identifiers and
   answering in the user's language. Keep multilingual input accepted without

package/DESIGN.md

@@ -612,21 +612,49 @@ must include stored-only source round-trip and mixed record kinds.
 ## 8. File policy, chunking, and embedding reuse
 
 Every Git tree entry must be accounted for. Include source, tests, documents,
-configuration, build definitions, and unknown readable text by default. Maintain
-explicit versioned rules for binaries, generated output, and vendored dependencies;
-do not treat working-tree ignore rules as a complete policy. Lockfiles are included
-unless an explicit generated/size rule excludes them.
+configuration, build definitions, and unknown readable text by default, subject to
+the versioned file-purpose policy. The preset pins `filePolicy: { version: 1, rules }`
+and its ordered repository-relative path rules. File selection and embedding
+eligibility are separate: source/lexical retrieval can remain available without a
+vector. Do not treat working-tree ignore rules as a complete policy.
+
+Credential filenames/directories, key/certificate payloads, image/binary assets
+and PEM private-key headers are hard exclusions, including for tracked files.
+Credential template suffixes (`.example`, `.sample`, `.template`) remain eligible.
+This is a targeted policy, not comprehensive secret detection. Dependencies,
+virtual environments, build/cache outputs, minified assets/maps and logs are
+excluded by default. Lockfiles, legal/author files, tabular/JSONL data, snapshots,
+and generated code (common suffixes or explicit generated/do-not-edit comments in the first 8 KiB) are
+lexical-only by default. Tests, ordinary docs, manifests and build/CI config remain
+eligible. Embedded license headers are not removed from source chunks.
+
+`--file-policy` accepts JSON on registration and local path dry runs. Last matching
+rule wins (`exclude`, `lexical`, `semantic`); rules use case-sensitive relative
+paths, `*`, `?` and whole-segment `**`. Rules can override default purpose decisions,
+but never hard content/path/size/encoding exclusions. The resolved policy is stored
+in the preset, not reread from a mutable checkout on import. `.gitattributes` is
+not interpreted. Inventory `retrieval`/`policyReason` and chunk skip reasons expose
+the decision. Changing rules changes configHash and cannot reuse a prior baseline.
+
+Evaluation labels require explicit exclusion independently of product defaults.
+Both checkout source evaluation harnesses pin `eval/source-corpus-policy.json`
+(excluding `eval/**`) and fingerprint it. New runs use new artifact roots; retained
+results and their old corpus/runtime identities remain historical evidence.
 
 | Special entry | Initial handling |
 | --- | --- |
 | Submodule | Record gitlink OID/path; no recursive import |
 | LFS pointer | Record metadata; no payload download or pointer indexing |
+| Image asset | Exclude known image extensions case-insensitively before chunking or embedding, including text SVG and portable pixel maps; retain inventory reason `image-extension` |
 | Symlink | Record target bytes; never follow or index as source |
 | Oversized file | Exclude with actual bytes and configured limit |
 | Invalid UTF-8 content/path | Record a diagnostic and lossless path identity; do not decode lossily |
 | Empty text file | File record, zero chunks |
 | Unsupported language or parse failure | Include bounded fallback chunks and distinguish the reason |
 
+Inline SVG markup in actual source files (for example, TSX components) remains
+source. The image exclusion policy participates in the immutable preset identity.
+
 Preserve raw UTF-8 bytes, BOM, CRLF, case, Unicode, and final newlines. Byte ranges
 are authoritative `[startByte, endByte)`; display lines are one-based/inclusive.
 JavaScript UTF-16 offsets must be converted explicitly. Preserve unusual Git paths
@@ -664,8 +692,8 @@ is not an unconditional one-function-per-chunk rule.
 
 ### Initial boundary and size rules
 
-The implementation adds Python/Go/Rust/C/C++/Shell using the existing pinned WASM
-grammars and SQL using a bundled, checksum-pinned grammar (see
+The implementation adds Python/Go/Rust/C/C++ using the existing pinned WASM
+grammars and Shell/SQL using bundled, checksum-pinned grammars (see
 `runtime/grammars/README.md`).
 Python splits top-level declarations and class bodies, keeps decorators with their
 function/class, and records nested class scopes. Go keeps functions/methods,
@@ -762,7 +790,7 @@ structural fragments, rather than build a complex value classifier.
 | Documentation, explanatory comments/docstrings | Embed; keep attached explanations with their code where practical |
 | Meaningful type/configuration declarations | Embed initially, then evaluate |
 | Imports-only or whitespace/punctuation-only fragments | Skip with a recorded reason |
-| Repeated license headers and mechanical listings | Candidates for explicit, versioned skip rules after evaluation |
+| Lexical-only file categories or rules | Keep exact source/chunks, omit embedding inputs and record the file policy reason; embedded license comments in authored code are not stripped |
 | Parser fallback | Do not exclude solely because parsing failed; apply content policy |
 
 Skipped chunks remain in inventory and source coverage. Retain lexical search text

package/README.md

@@ -19,6 +19,82 @@ srcx --version
 srcx --help
 ```
 
+## Install the agent skill
+
+The npm package includes the [srcx-search skill](skills/srcx-search/SKILL.md).
+After installing the CLI, explicitly select your agent:
+
+```sh
+srcx skills install --agent codex
+# Or, for Claude Code:
+srcx skills install --agent claude
+srcx skills status --agent codex
+```
+
+The default scope is `user`. Installation copies the bundled instructions into
+one of the following local discovery directories:
+
+| Agent       | User scope                              | Project scope                                   |
+| ----------- | --------------------------------------- | ----------------------------------------------- |
+| Codex       | `~/.agents/skills/srcx-search/SKILL.md` | `<project>/.agents/skills/srcx-search/SKILL.md` |
+| Claude Code | `~/.claude/skills/srcx-search/SKILL.md` | `<project>/.claude/skills/srcx-search/SKILL.md` |
+
+These are the documented local skill locations for
+[Codex](https://learn.chatgpt.com/docs/build-skills#where-codex-loads-local-skills)
+and [Claude Code](https://code.claude.com/docs/en/skills#where-skills-live).
+Cloud sessions need their own installation; a local user installation does not
+configure remote agents or account-synced skills. The installer targets these
+default locations, not custom agent configuration directories.
+
+To share a skill with a project, select an existing project directory and commit
+the installed file if appropriate for your team:
+
+```sh
+srcx skills install --agent codex --scope project --path /path/to/project
+```
+
+Project scope defaults to the current directory when `--path` is omitted; it does
+not infer the Git root. `--path` is accepted only for project scope. Prefer one
+scope per agent to avoid duplicate skill names. The installer does not change
+`AGENTS.md`, `CLAUDE.md`, agent permissions, shell profiles or credentials, and npm
+installation alone does not place skills in your environment.
+
+Make sure the agent's shell can run `srcx --version`. Configure LambdaDB and
+prepare the repository using the [connected workflow](#connected-workflow) below,
+or use an already indexed repository in the configured project. The agent process
+must inherit the API-key environment variable named by `srcx configure`; no key
+is copied into the skill. `srcx doctor` checks authentication/read access only.
+
+Start a new agent session after installation. Invoke `$srcx-search` in Codex or
+`/srcx-search` in Claude Code, and ask it to investigate an indexed repository at a
+specific branch, release or commit. Skill-aware agents may also select it from its
+description. Installation/status confirms files on disk, not that an agent loaded
+or selected them. Agent policy, disabled skills and custom discovery paths can
+still affect availability.
+
+### Update and remove
+
+```sh
+npm install -g @functional-systems/srcx@dev
+srcx skills update --agent codex
+srcx skills status --agent codex
+srcx skills remove --agent codex
+```
+
+Repeat for each installed agent/scope, supplying the same `--scope project --path`
+when applicable. `update` uses the currently running CLI's bundled skill and does
+not download anything. npm upgrades do not silently update skill copies, and npm
+uninstall does not remove them: run `skills remove` before uninstalling the CLI.
+
+A provenance comment records the package version and content hash. Repeated
+installation is a no-op when the installed copy matches. A different managed
+version requires `update`. Locally edited skills, extra files, unowned skills and
+symlinked managed paths are preserved by refusing the operation; there is no
+force-overwrite flag. Move a customized copy to a separate skill name/location
+before installing a managed copy. Interrupted operations can leave a sibling
+`.srcx-search.srcx-lock` directory: check that no installer is running and preserve
+any staged file before manually removing that lock and retrying.
+
 ## Run locally
 
 Requires Node.js 22.14+ and Git. The default preset uses lexical search with
@@ -66,9 +142,60 @@ Each artifact contains `build.json` (inventory, configuration, counts, hashes,
 change/deletion plan) and `records.jsonl` (exact file records and retrieval chunks).
 Artifacts contain source code; keep them with the same access policy as the repo.
 Every tree entry is accounted for, including explicit exclusions for symlinks,
-submodules, LFS pointers, dependencies/build output, binary/invalid UTF-8 data,
+submodules, LFS pointers, dependencies/build output, image assets (including SVG), binary/invalid UTF-8 data,
 invalid UTF-8 paths, and files above 1 MiB. Excluded paths are preserved losslessly
 in `pathBase64`.
+Image extensions are matched case-insensitively and excluded before chunking or
+embedding, even when their payload is readable text. Source files containing
+inline images, such as a TSX component with SVG markup, remain included.
+
+File selection and embedding eligibility are separate:
+
+| Default treatment                                   | Examples                                                                                                                                                                                                                                                        |
+| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Exclude source and chunks                           | Credential files (`.env`, `.env.*`, `.npmrc`, `.netrc`, `.aws/`, `.ssh/`), key/certificate payloads, images, binaries, installed dependencies (`.venv/`, `node_modules/`, `vendor/`, `third_party/`), build/cache output, logs, minified assets and source maps |
+| Keep source and lexical chunks; skip embeddings     | Lockfiles, license/author files, CSV/TSV/JSONL data, snapshots, generated code identified by common suffixes or an explicit generated/do-not-edit comment in the first 8 KiB                                                                                    |
+| Keep source, lexical chunks and eligible embeddings | Authored code, tests, documentation, package manifests, CI/build configuration and other readable text                                                                                                                                                          |
+
+`.env.example`, `.env.sample`, `.env.template` and equivalent suffixes remain
+included as templates. Private-key PEM headers also exclude files with unrelated
+extensions. These are targeted exclusions, **not a general secret scanner**;
+review the dry-run inventory before uploading. `.gitignore` does not remove
+already tracked Git objects, and `.gitattributes` is not interpreted for selection.
+
+Use `--file-policy policy.json` with `repo add` or `import --dry-run --path` to
+adjust the scope explicitly. The JSON contents are pinned in the repository
+preset; subsequent connected imports use that preset, not the local policy file.
+
+```json
+{
+  "version": 1,
+  "rules": [
+    { "pattern": "eval/**", "action": "exclude" },
+    { "pattern": "third_party/special/**", "action": "lexical" },
+    { "pattern": "generated/api.pb.go", "action": "semantic" }
+  ]
+}
+```
+
+Patterns match case-sensitive, repository-relative paths. `*` and `?` stay within
+one path segment; `**` must occupy a whole segment and matches any depth. Rules
+run in order; the last match wins. `semantic` permits embeddings when enabled but
+still skips structural/import-only chunks. Rules override dependency/generated/
+data defaults, but cannot override credential, crypto, image, binary, private-key
+content, size, encoding, symlink or LFS exclusions. `retrieval` and `policyReason`
+in dry-run coverage explain each included file's policy; chunk records retain
+`embeddingSkipReason`. A changed policy selects a distinct Collection/configuration.
+
+```sh
+srcx import --path . --ref develop --dry-run --embedding text-embedding-3-small --file-policy policy.json --output /tmp/srcx-preview
+srcx repo add --path . --embedding text-embedding-3-small --file-policy policy.json
+```
+
+Evaluation labels need an explicit corpus boundary. The checkout evaluation
+harnesses use [this policy](eval/source-corpus-policy.json) to exclude `eval/**`;
+labels stored elsewhere require additional exclusions before a new evaluation.
+General-purpose imports do not discard all evaluation or test directories.
 
 Java, TypeScript/TSX, JavaScript/JSX, Python, Go, Rust, C/C++, Shell and SQL use
 pinned Tree-sitter WASM grammars.
@@ -85,8 +212,8 @@ pinned Tree-sitter WASM grammars.
 
 `.h` files use the C grammar deterministically; C++ headers should use a listed
 C++ extension for syntax chunking. Shell does not infer extensionless scripts from
-shebangs or claim Zsh/Fish/PowerShell support. SQL uses the bundled
-[grammar artifact](runtime/grammars/README.md); dialect-specific syntax unsupported
+shebangs or claim Zsh/Fish/PowerShell support. Shell and SQL use bundled
+[grammar artifacts](runtime/grammars/README.md); SQL dialect-specific syntax unsupported
 by that grammar falls back to text. SQL strings and supported dollar-quoted bodies
 are parsed as part of their containing statement, not split at each semicolon.
 Rust macros/`cfg` and C/C++ preprocessor conditions are not expanded or evaluated;
@@ -264,6 +391,16 @@ For practical search → read → change-planning examples, see the
 It follows three maintenance investigations through related source and tests,
 including the limitations of previews and optional reranking.
 
+The [agent workflow diagnostic](https://github.com/lambdadb/srcx/blob/develop/eval/AGENT-WORKFLOW-PILOT.md)
+compares fresh agent sessions with Git/local tools and optional or explicitly
+invoked srcx. It found no established efficiency advantage on the small public
+corpora and records the remaining workflow hypotheses.
+
+The [semantic agent follow-up](https://github.com/lambdadb/srcx/blob/develop/eval/SEMANTIC-AGENT-RESULTS.md)
+compares local, lexical-first and semantic-first investigations on Requests. It
+finds no established agent efficiency gain and records an unresolved candidate
+retrieval discrepancy and excessive embedding of a generated SVG asset.
+
 ## Query language and agent use
 
 Queries can use any language; the CLI does not translate them. For lexical search,

package/VALIDATION.md

@@ -1,5 +1,139 @@
 # Validation record
 
+## File selection and embedding policy
+
+Offline validation on Node 24.15.0 passed 128 tests and seven installed-package
+checks, plus typecheck, version and formatting checks. Synthetic tracked files
+verify credential/crypto/image exclusions, case-insensitive minified assets,
+virtual environments, dependency/cache paths and logs. Lockfiles, generated code,
+snapshots and data retain source and lexical chunks without managed inputs.
+Ordinary source, tests, manifests, templates and configurable Sphinx documentation
+remain eligible. Ordered path overrides, configuration identity, baseline rejection,
+unchanged-file reuse, managed publication and CLI override rejection are covered.
+The CLI evaluator's actual offline prepare test includes an in-tree answer fixture
+and verifies it is absent from the prepared corpus.
+
+Fresh offline inventories reused the audited source commits:
+
+| Repository                                              | Result                                                                                                                                                                                                                      |
+| ------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Requests `611c6162cbc4ac2020a2f91c7cfa4f3abf9bbb60`     | Excludes 12 certificate/key payload files. Estimated managed input decreases from the image-only build's 179,548 to 159,849 tokens; legal/author files stay lexical-only and docs/conf.py stays eligible.                   |
+| srcx `e83134b44a11d78a79c5941c0e1f907383f238b3`         | Explicit `eval/**` exclusion removes 45 evaluation files: total chunk tokens decrease from 316,889 to 202,835. This is a corpus boundary, not a claim that earlier answers used leaked labels.                              |
+| lambdadb-cli `513af6e4d262edd380013c86d51a20aad16274d7` | Retains all 38 files and 200 chunks; package-lock.json (22,076 chunk tokens), LICENSE and JSONL examples have no managed input. The retained baseline was lexical-only, so this is not a measured embedding-cost reduction. |
+
+All excluded paths have zero file/chunk records and all lexical-only files have
+zero embedding inputs in these rebuilt artifacts. No service writes, provider calls
+or agent sessions were performed. Existing remote Collections and historical
+results are unchanged. The pinned policy selects a new configuration; no legacy
+preset compatibility or migration layer was added. Generated detection is limited
+to explicit markers/suffixes, `.gitattributes` is not interpreted, and credential
+rules are not a general secret scanner. Additional evaluation-label locations must
+be added to the explicit evaluation policy before running a different suite.
+
+## Image exclusion
+
+Image assets are excluded by case-insensitive extension before source records,
+chunking or embedding, including SVG and text pixel maps. Source and installed
+CLI checks cover image exclusion for lexical and managed presets; TSX containing
+inline SVG still produces managed chunks. The regression fails against the
+archived pre-fix runtime. All 125 tests, six installed-package checks, typecheck,
+version and formatting checks passed on Node 24.15.0.
+
+A credential-free rebuild of the same Requests commit used in the semantic pilot
+excluded the logo SVG: managed input fell from 806,268 to 179,548 estimated tokens
+(77.7%), and managed chunks from 1,864 to 1,081. No remote import, agent rerun or
+retrieval improvement was measured. Previous results and their runtime are retained.
+
+## Semantic agent investigation
+
+[The semantic follow-up](eval/SEMANTIC-AGENT-RESULTS.md) covers twelve fresh sessions
+on four frozen Requests questions, with the same local checkout in all conditions.
+All answers covered the required facts under assistant source review. Mean times
+were 69.9 seconds for local tools, 68.6 for lexical-first and 68.0 for semantic-first;
+these single-run differences do not establish an efficiency gain. Semantic used
+more total model input than both other conditions. All srcx sessions used one
+search followed by local reads.
+
+All 35 task previews and five separate diagnostic previews matched pinned source
+and immutable version metadata; twelve checkouts and 26 runtime/harness fingerprints
+passed verification. Payload validation does not establish nearest-neighbor recall:
+one unfiltered semantic query returned SVG chunks, while a Python-filtered query
+returned relevant methods with higher scores. The report preserves this unresolved
+retrieval discrepancy separately from the original agent comparison. No product
+search policy, embedding model or ranking default changed.
+
+## Agent workflow diagnostic
+
+[The agent workflow record](eval/AGENT-WORKFLOW-PILOT.md) covers eight fresh Codex
+sessions on three frozen questions against pinned public srcx/lambdadb-cli source.
+Optional srcx access was unused in all three paired development environments.
+A separate explicit-invocation confirmation used three searches and seven reads
+without cloning. Both it and the Git/local baseline answered correctly under
+assistant source review; srcx took 94.2 seconds versus 65.8, with more total model
+input but less uncached input. No token-cost, latency or accuracy win is claimed.
+Six successful reads and ten search previews matched pinned source/version data.
+The report separates preparation cost, guided invocation, evolving environment
+conditions and the audited wrapper reservation defect from product correctness.
+
+## Bash grammar runtime compatibility
+
+Preparing the agent workflow pilot exposed a real import failure in the public
+`lambdadb-cli` repository at `513af6e4d262edd380013c86d51a20aad16274d7`:
+`scripts/test-homebrew.sh` aborted local materialization with
+`TypeError: resolved is not a function`. A minimal `[[ "$value" == 1 ]]`
+comparison reproduces it with the previous Bash grammar/runtime combination.
+The replacement grammar is bundled with its license and pinned checksum in
+`CHUNKER.shellGrammar`; other language runtimes remain unchanged.
+
+The regression fixture fails before the replacement and passes afterward,
+preserving complete source-byte coverage and syntax spans. The same fixture is
+included in installed-package import checks. On Node 24.15.0, all 124 tests,
+six installed-package tests, formatting/version checks and typecheck passed.
+The two public pilot corpora also passed local materialization after the fix:
+139 files/928 chunks for srcx and 38 files/200 chunks for lambdadb-cli, with no
+embeddings. These checks establish indexing/packaging correctness, not an agent
+accuracy or efficiency improvement.
+
+## Agent skill installation
+
+The npm artifact includes `skills/srcx-search/SKILL.md`. The `skills` command
+installs, reports, updates and removes the bundled copy for Codex or Claude Code
+in user/project scope. No credentials or network access are needed for those
+operations. A content-hash/package-version footer distinguishes unchanged managed
+copies from custom files; modified/unowned content, extra files and symlinked
+managed paths are preserved. Scope selection does not edit unrelated agent
+settings or infer a project root.
+
+On Node 24.15.0, formatting/version checks, typecheck, all 124 tests and six packed
+CLI tests passed. Lifecycle tests cover both agents and scopes using isolated
+roots, replacement of a previous managed bundle, custom-file preservation,
+symlinks, active locks, invalid options and no-write status/absent removal. Absent
+removal also covers existing read-only parent directories for both agents/scopes
+and preserves a pre-existing lock without acquiring it. The permission regression
+tests fail before the fix and pass afterward on the local non-root POSIX runtime;
+those permission tests are skipped on Windows or when running as root. Packed
+CLI tests exercise both project destinations without config, then the existing
+loopback workflow covers skill installation, repository discovery, commit
+resolution, search and exact source reads. This is deterministic integration
+validation; a model does not choose the commands in those tests.
+
+The skill-creator validator accepted the skill. Separately, a fresh npm consumer
+installed the tarball and Codex CLI 0.157.1 `skills/list` discovered the installed
+project skill as enabled. Removing it removed it from the actual runtime catalog;
+reinstalling restored it with `forceReload`. This exercised the real discovery
+path without model requests. Claude Code's documented destination and installed
+file lifecycle are covered, but Claude runtime discovery and either agent's
+autonomous skill selection have not been exercised. Default local paths are the
+supported installer targets; custom config directories and cloud/account skill
+sync are outside this installation flow.
+
+There were no live LambdaDB changes, embedding requests, global CLI installation,
+or changes to the developer's personal skill directories. Tests and discovery
+used isolated directories. This validates packaging/discovery, not token savings,
+latency improvements or ranking quality.
+
+Date: 2026-09-26 (Asia/Seoul).
+
 ## Query-language guidance
 
 README and search help explain source-language query selection without rejecting

package/dist/build.d.ts

@@ -1,6 +1,7 @@
 import { resolveCommit, type Identity } from "./git.js";
 import { CHUNKER } from "./chunk.js";
 import { type Doc } from "./common.js";
+import { type FilePolicy } from "./file-policy.js";
 export declare const ANALYZERS: readonly ["english", "japanese", "korean", "standard"];
 export type Analyzer = (typeof ANALYZERS)[number];
 export declare function normalizeAnalyzers(value: unknown): Analyzer[];
@@ -33,6 +34,7 @@ export type Preset = {
     /** Internal evaluation override; the CLI keeps its original enrichment. */
     enrichment?: "path-only-v1";
     maxFileBytes: number;
+    filePolicy: FilePolicy;
     embedding: {
         model: string;
         dimensions: number;
@@ -46,7 +48,7 @@ export declare const PRESET: Preset;
 /** Embedding variants share the same chunking behavior. */
 export declare const MANAGED_PRESET: Preset;
 export declare const MANAGED_LARGE_PRESET: Preset;
-export declare function presetFor(embedding?: string, analyzers?: readonly string[]): Preset;
+export declare function presetFor(embedding?: string, analyzers?: readonly string[], filePolicy?: unknown): Preset;
 export declare function supportedPreset(preset: unknown): preset is Preset;
 export declare function indexConfigs(preset?: Preset): {
     searchText: {
@@ -114,6 +116,8 @@ export type InventoryItem = {
     bytes: number;
     status: "included" | "excluded";
     reason?: string;
+    retrieval?: "lexical" | "semantic";
+    policyReason?: string;
     targetBase64?: string;
     fileId?: string;
     chunkIds?: string[];

package/dist/build.js

@@ -5,6 +5,7 @@ import { join, resolve } from "node:path";
 import { Blobs, inventory, resolveCommit } from "./git.js";
 import { chunk, CHUNKER, tokens } from "./chunk.js";
 import { atomic, canonical, hash, invariant, lineAt, json, utf8, optionalJson, } from "./common.js";
+import { FILE_POLICY, normalizeFilePolicy, hardExclusion, privateKeyContent, fileDecision, } from "./file-policy.js";
 export const ANALYZERS = ["english", "japanese", "korean", "standard"];
 export function normalizeAnalyzers(value) {
     invariant(Array.isArray(value) &&
@@ -26,6 +27,7 @@ export const PRESET = {
     chunker: CHUNKER,
     mode: "syntax",
     maxFileBytes: 1024 * 1024,
+    filePolicy: FILE_POLICY,
     embedding: null,
 };
 /** Embedding variants share the same chunking behavior. */
@@ -48,7 +50,7 @@ export const MANAGED_LARGE_PRESET = {
         dimensions: 3072,
     },
 };
-export function presetFor(embedding = "none", analyzers = PRESET.analyzers) {
+export function presetFor(embedding = "none", analyzers = PRESET.analyzers, filePolicy = FILE_POLICY) {
     invariant(["none", "text-embedding-3-small", "text-embedding-3-large"].includes(embedding), "Embedding must be none, text-embedding-3-small or text-embedding-3-large.");
     const base = embedding === "none"
         ? PRESET
@@ -56,16 +58,20 @@ export function presetFor(embedding = "none", analyzers = PRESET.analyzers) {
             ? MANAGED_LARGE_PRESET
             : MANAGED_PRESET;
     const normalized = normalizeAnalyzers(analyzers);
-    return hash(normalized) === hash(base.analyzers)
-        ? base
-        : { ...base, analyzers: normalized };
+    const configured = {
+        ...base,
+        analyzers: normalized,
+        filePolicy: normalizeFilePolicy(filePolicy),
+    };
+    return hash(configured) === hash(base) ? base : configured;
 }
 export function supportedPreset(preset) {
     if (!preset || typeof preset !== "object" || Array.isArray(preset))
         return false;
     try {
         const analyzers = normalizeAnalyzers(preset.analyzers);
-        return [PRESET, MANAGED_PRESET, MANAGED_LARGE_PRESET].some((base) => hash({ ...base, analyzers }) === hash(preset));
+        const filePolicy = normalizeFilePolicy(preset.filePolicy);
+        return [PRESET, MANAGED_PRESET, MANAGED_LARGE_PRESET].some((base) => hash({ ...base, analyzers, filePolicy }) === hash(preset));
     }
     catch {
         return false;
@@ -123,18 +129,10 @@ export async function loadBuild(directory) {
     b.directory = resolve(directory);
     return b;
 }
-function exclude(path) {
-    if (/(^|\/)(node_modules|vendor|vendored|dist|build|target|coverage|\.git|\.next)(\/|$)/.test(path))
-        return "dependency-or-build-output";
-    if (/\.(png|jpe?g|gif|ico|pdf|zip|gz|jar|class|woff2?|mp[34]|exe|dll|so|dylib|wasm)$/i.test(path))
-        return "binary-extension";
-    if (/\.(min\.(js|css)|map)$/.test(path))
-        return "generated-output";
-    return undefined;
-}
 export async function materialize(args) {
     const preset = args.preset ?? PRESET, configHash = hash(preset), prev = args.previous;
-    invariant(hash(preset.analyzers) === hash(normalizeAnalyzers(preset.analyzers)) &&
+    invariant(hash(preset.filePolicy) === hash(normalizeFilePolicy(preset.filePolicy)) &&
+        hash(preset.analyzers) === hash(normalizeAnalyzers(preset.analyzers)) &&
         hash(preset.chunker) === hash(CHUNKER) &&
         ["syntax", "window"].includes(preset.mode) &&
         (preset.enrichment === undefined ||
@@ -194,7 +192,9 @@ export async function materialize(args) {
                 item.targetBase64 = (await blobs.read(e)).toString("base64");
                 continue;
             }
-            const reason = exclude(e.path);
+            const pathDecision = fileDecision(e.path, preset.filePolicy);
+            const reason = hardExclusion(e.path) ??
+                (pathDecision.action === "exclude" ? pathDecision.reason : undefined);
             if (reason) {
                 item.reason = reason;
                 continue;
@@ -224,6 +224,11 @@ export async function materialize(args) {
                 item.reason = "lfs-pointer";
                 continue;
             }
+            if (privateKeyContent(source)) {
+                item.reason = "private-key-content";
+                continue;
+            }
+            const decision = fileDecision(e.path, preset.filePolicy, source);
             const parsed = await chunk(source, e.path, preset.mode, preset.enrichment);
             const contentHash = hash(data);
             const fileId = `f-${hash([e.pathBase64, contentHash])}`;
@@ -247,6 +252,8 @@ export async function materialize(args) {
             }
             Object.assign(item, {
                 status: "included",
+                retrieval: decision.action,
+                policyReason: decision.reason,
                 contentHash,
                 fileId,
                 chunkIds: [],
@@ -273,7 +280,8 @@ export async function materialize(args) {
                         ? "structural-only"
                         : "embedding-disabled",
                 };
-                const eligible = !["imports", "structural"].includes(span.chunkKind);
+                const eligible = decision.action === "semantic" &&
+                    !["imports", "structural"].includes(span.chunkKind);
                 if (preset.embedding && eligible) {
                     const inputHash = hash([preset.embedding, span.searchText]);
                     if (preset.embedding.managed) {
@@ -309,9 +317,11 @@ export async function materialize(args) {
                 }
                 else {
                     doc.embeddingSkipReason = preset.embedding
-                        ? span.chunkKind === "imports"
-                            ? "imports-only"
-                            : "structural-only"
+                        ? decision.action === "lexical"
+                            ? decision.reason
+                            : span.chunkKind === "imports"
+                                ? "imports-only"
+                                : "structural-only"
                         : "embedding-disabled";
                     item.skipped++;
                 }
@@ -402,7 +412,10 @@ export async function validateBuild(b) {
         /^(?:[a-f0-9]{40}|[a-f0-9]{64})$/.test(b.commitOid) &&
         b.buildId ===
             hash([b.repoKey, b.commitOid, b.configHash, "srcx-builder-v1"]), "Build commit identity mismatch; rebuild from the intended Git commit.");
-    invariant(hash(b.preset.analyzers) === hash(normalizeAnalyzers(b.preset.analyzers)) &&
+    invariant(hash(b.preset.filePolicy) ===
+        hash(normalizeFilePolicy(b.preset.filePolicy)) &&
+        hash(b.preset.analyzers) ===
+            hash(normalizeAnalyzers(b.preset.analyzers)) &&
         hash(b.preset) === b.configHash &&
         hash(b.inventory) === b.inventoryHash, "Build metadata hash mismatch.");
     const seen = new Set();
@@ -411,6 +424,7 @@ export async function validateBuild(b) {
     const expectedFile = new Map(b.inventory
         .filter((e) => e.status === "included")
         .map((e) => [e.fileId, e]));
+    let activeDecision;
     const completedFiles = new Set();
     const finish = () => {
         if (activeFile) {
@@ -435,6 +449,12 @@ export async function validateBuild(b) {
             end = 0;
             count = 0;
             const entry = expectedFile.get(doc.id);
+            activeDecision = fileDecision(doc.path, b.preset.filePolicy, doc.sourceText);
+            invariant(!hardExclusion(doc.path) &&
+                !privateKeyContent(doc.sourceText) &&
+                activeDecision.action !== "exclude" &&
+                entry?.retrieval === activeDecision.action &&
+                entry?.policyReason === activeDecision.reason, "File policy or inventory decision mismatch.");
             const raw = Buffer.from(doc.sourceText);
             invariant(entry &&
                 hash(raw) === doc.contentHash &&
@@ -470,7 +490,8 @@ export async function validateBuild(b) {
                 doc.tokenCount === tokens(doc.searchText) &&
                 Number(doc.tokenCount) <= CHUNKER.maxTokens, "Invalid chunk source/position/token binding.");
             if (b.preset.embedding?.managed) {
-                const eligible = !["imports", "structural"].includes(String(doc.chunkKind));
+                const eligible = activeDecision.action === "semantic" &&
+                    !["imports", "structural"].includes(String(doc.chunkKind));
                 invariant(eligible
                     ? doc.embeddingStatus === "managed" &&
                         doc.embeddingText === doc.searchText &&
@@ -481,12 +502,16 @@ export async function validateBuild(b) {
                         doc.embeddingText === undefined &&
                         doc.embeddingInputHash === undefined &&
                         doc.embeddingSkipReason ===
-                            (doc.chunkKind === "imports"
-                                ? "imports-only"
-                                : "structural-only"), "Invalid managed embedding input or eligibility.");
+                            (activeDecision.action === "lexical"
+                                ? activeDecision.reason
+                                : doc.chunkKind === "imports"
+                                    ? "imports-only"
+                                    : "structural-only"), "Invalid managed embedding input or eligibility.");
             }
             else if (doc.embeddingStatus === "embedded")
                 invariant(b.preset.embedding &&
+                    activeDecision.action === "semantic" &&
+                    !["imports", "structural"].includes(String(doc.chunkKind)) &&
                     Array.isArray(doc.embedding) &&
                     doc.embedding.length === b.preset.embedding.dimensions &&
                     doc.embedding.every(Number.isFinite), "Invalid embedded chunk.");