sift-light 1.0.5-6 → 1.1.0

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## 1.1.0 — 2026-10-12
+
+- Add opt-in BYO HTTP JSON providers for Jev-compatible semantic judging and multimodal retrieval.
+- Keep local native text and visual workers as the default paths, with local source hashes, PDF page mapping, inspection and validation retained after remote ranking.
+- Bound remote request and response sizes, timeouts, retries, file identifiers and credentials; API keys remain environment references rather than configuration values.
+- Add configuration, protocol, MCP boundary and end-to-end local provider regression coverage.
+
+## 1.1.0-1 — 2026-10-12
+
+- Add an independent native text retrieval lane and bounded rank fusion while retaining exact source inspection, validation and both text result lanes.
+- Add opt-in MCP image search from text or an image, including image-only and mixed PDF scanned-page retrieval with original page numbers and rendered previews. Requires Apple Silicon and separately configured native workers and model files.
+- Share resident worker lifecycle management with cancellation, recovery and idle shutdown; bound native GPU cache and batching to reduce memory use.
+- Preserve default model-free exact search and explicit partial-coverage and stale-source diagnostics.
+
 ## 1.0.5-6 — 2026-10-10
 
 - Add structured `repairExamples` to invalid-regex and missing-path diagnostics. The examples preserve bounded search filters, never auto-convert regex input, and are omitted when redaction would make request data unsafe to echo.

package/README.md

@@ -34,7 +34,7 @@ Concept ranking covers every UTF-8 passage admitted by the request's documented
 
 Slow Concept and hybrid requests return within the default five-second wait window with `status: "waiting"` or `"running"`, an `operationId`, progress and an exact `nextRequest` such as `{ "mode": "await", "operationId": "..." }`. Copy that request unchanged: it resumes the same computation and never restarts the query or downgrades to a literal-only result. A final result remains available for stable re-fetch for 10 minutes, with up to 32 terminal results retained per service session, and `mode: "cancel"` stops the owned work and waits for cleanup. Each service session admits at most eight pending operations; an operation has one total deadline controlled by `SIFT_LIGHT_CONCEPT_TIMEOUT_MS` (integer milliseconds from 1000 through 3600000; default 600000) and a 120-second idle continuation lease. A real model, source or resource failure is returned as a failure with its diagnostic. Source generation is re-enumerated and re-verified before publication, so changes refresh the operation and mixed versions are never marked complete. Admission planning counts (`filesEnumerated`, `filesAdmitted`, `filesSkippedEmpty`, `filesUnavailable`, `passagesQueued`, `batchesPlanned`, `batchesCompleted`) stay visible. Empty files are a normal skip and do not mark the result partial.
 
-The first uncached Concept or hybrid search loads the local model and may take tens of seconds and more than 1 GiB of inference-worker memory, depending on the machine and search scope. Later searches reuse cached passage embeddings even when the query wording changes, although a new query still loads the model briefly to embed that query. Changed passages are recomputed; cache eviction, deletion or broad source changes can cause another cold run. Exact search modes do not load the model. These are workload-dependent observations, not a latency or memory guarantee. If the result is waiting during `model-loading`, follow its `nextRequest` to continue the same operation.
+The first uncached Concept or hybrid search loads the local model and may take tens of seconds and more than 1 GiB of inference-worker memory, depending on the machine and search scope. Later searches reuse cached passage embeddings even when the query wording changes, and an active service reuses its loaded model. The E5 worker is released after 30 seconds of inactivity; the next query then loads it again. Changed passages are recomputed; cache eviction, deletion or broad source changes can cause another cold run. Exact search modes do not load the model. These are workload-dependent observations, not a latency or memory guarantee. If the result is waiting during `model-loading`, follow its `nextRequest` to continue the same operation.
 
 ### Give several search conditions together
 
@@ -73,6 +73,30 @@ Use `mode: "outline"` with a concrete JS/TS/TSX or Python file to see bounded sy
 
 The readable result keeps the main evidence compact. Per-item ranges, counts, coverage and continuation requests remain in structured `details`, so a client can use the structured fields without requiring a second search.
 
+### Independent text retrieval on Apple Silicon
+
+Optional `nativeText` configuration makes `concept` and `hybrid` return two local retrieval lanes. Each keeps its own ranking; `details.analysis.retrievalLanes.current` and `.native` contain executable continuation requests. For `hybrid` with `ranking: "relevance"`, the default view ranks at most ten distinct files using weighted reciprocal file ranks (native 1, current 0.1). Raw scores across lanes are not comparable. Other dual-lane views interleave candidates. The default view and both lanes support `inspect` and `validate`, including source-change reporting.
+
+On Apple Silicon macOS, prepare Xcode, Metal Toolchain, a local SDK checkout and model directory, then run from this project's checkout:
+
+```sh
+bun run build:native-text -- /absolute/sdk-checkout /absolute/native-text-output
+```
+
+The builder selects a fixed revision and installs the executable and required resource bundles. The model directory must contain `model.safetensors`, `config.json`, `tokenizer.json` and `tokenizer_config.json`. Keep the complete output directory and add this to the host's active `sift-light.json`:
+
+```json
+{
+  "vectorSearchEnabled": true,
+  "nativeText": {
+    "executable": "/absolute/native-text-output/native-text",
+    "modelDirectory": "/absolute/model-directory"
+  }
+}
+```
+
+The existing E5 model is still required. The native process reads only private staged text admitted by the existing source policy. The service reuses its incremental index and removes it on shutdown; consecutive requests reuse the same loaded native model. After 60 seconds of inactivity the native worker exits and releases its memory; the next request starts a fresh worker. Cancellation, shutdown and protocol failures reap the owned worker before further work begins. `details.analysis.stats` reports `nativeModelLoads`, `nativeModelReused` and `nativeProcessId`. Existing admission controls hidden files, globs, exclusions and read budgets. Unknown text suffixes are normalized from verified UTF-8 bytes into plain text copies. Executable, model and protocol failures are explicit errors. Omit `nativeText` to retain existing behavior. This option enables text only, without image, audio or video retrieval.
+
 ## Common uses
 
 Tell your agent what you need, for example:
@@ -184,3 +208,61 @@ In hard mode, a direct search such as `grep warning report.txt` is denied. A rec
 Local searches stay on your machine. Only grant access to files your agent is allowed to read. HTTP deployments need an authenticated gateway before public exposure; see [Security](SECURITY.md).
 
 [Changelog](CHANGELOG.md) · [Contributing](CONTRIBUTING.md) · [AGPL-3.0-only license](LICENSE)
+
+### Visual search (MCP)
+
+Configure `nativeMedia` with absolute `executable` and `modelDirectory` paths, then restart the MCP host. Build the Apple Silicon worker with `bun run build:native-media <sdk-checkout> <output-directory>`. The worker uses the local model weights with vision enabled and audio disabled.
+
+The additional `sift-light-media` tool accepts `mode: "search"`, an exact local `path`, and exactly one `query` (text description) or `image` (local image path). `kind` is `all`, `image`, or `scan`. File filters `glob`, `exclude`, `hidden`, and `ignorePolicy` use the same admission policy as workspace search. Text and source navigation continue through `sift-light`.
+
+```json
+{
+  "mode": "search",
+  "path": "/absolute/project/assets",
+  "query": "a dog running on grass",
+  "kind": "image"
+}
+```
+
+```json
+{
+  "mode": "search",
+  "path": "/absolute/project/assets",
+  "image": "/absolute/query.jpg",
+  "kind": "all"
+}
+```
+
+Results retain up to ten distinct files, content hashes, original PDF page numbers, and exact `inspect`/`validate` requests for this tool. `inspect` returns actual PNG pixels at a maximum dimension of 1024; scan results render the corresponding original PDF page. Changed sources reject inspection and validation reports stale evidence. Scanned pages in mixed PDFs retain their original page numbers. Visual similarity remains candidate evidence; small print recognition and region-level text location are not guaranteed.
+
+### Bring your own provider
+
+The local model path remains available through `nativeMedia`. To use a provider you supply, add `media` to the same `sift-light.json`. The endpoint must implement the HTTP JSON contract below; it may be a third-party service or a local gateway that translates another vendor's API. A local service can omit `apiKeyEnv`; remote services should use HTTPS.
+
+```json
+{
+  "media": {
+    "provider": "http-json",
+    "endpoint": "https://your-provider.example/v1/media-search",
+    "model": "your-vision-model",
+    "apiKeyEnv": "YOUR_MEDIA_KEY",
+    "timeoutMs": 120000,
+    "maxRetries": 2,
+    "maxRequestBytes": 33554432,
+    "renderer": {
+      "executable": "/absolute/native-media",
+      "modelDirectory": "/absolute/model-directory"
+    }
+  }
+}
+```
+
+The plugin sends admitted files as bounded Base64 content with opaque `fileId` values; it never sends local absolute paths. A response must contain `version: 1`, `indexedFiles`, and `hits`. Each hit contains `fileId`, `score`, `kind`, `chunkCount`, `width`, and `height`; scanned PDF hits also contain a one-based `page`. The plugin maps IDs back to local sources and then performs hash, page, `inspect`, and `validate` checks. The remote service does not decide whether source evidence is trustworthy.
+
+Jev also accepts `semanticJudge.provider: "http-json"`. The request is `{ "version": 1, "model", "query", "candidates" }`; the response is `{ "version": 1, "model", "judgments", "usage" }`. Each judgment contains a candidate `id`, a `choice`, and a `probability` from 0 to 1. When `apiKeyEnv` is configured, the key is read only from that environment variable and is never written to configuration, logs, or result content. A keyless local provider is valid. Remote failures retain local candidates and mark coverage as partial.
+
+The remote path uses the same `sift-light-media` tool and source verification as the local path. PNG hits can be inspected directly; PDF or other image previews require a local renderer in `media.renderer`. Validation does not require a renderer.
+
+Cold indexing returns `nextRequest` with `mode: "await"`; copy it to the same tool. `mode: "cancel"` and its `operationId` stop that work. The local provider reuses one resident visual model for consecutive requests and releases it after 60 seconds idle; remote provider timeouts, retries, and request limits come from its configuration. The service owns shutdown and cancellation. Each request admits at most 512 media files, 20 MiB per file and 256 MiB total; corpus limits and enumeration limits are explicit partial coverage. Request deadline is five minutes. Results expire after ten minutes and are scoped to the originating workspace. Office documents, audio and video are outside this tool's current contract.
+
+Native text and visual workers reserve 1 GB of inference headroom above the loaded model and its working space, and return unused compute cache after each query. This is not a hard limit on total process memory. The model still uses several GB, with additional allocations during indexing and inference. Each MCP service session owns its models; multiple connections and simultaneous text/visual workers add their memory use. Ordinary exact search does not load these models. Measure memory and latency for the actual model, machine and file scope.

package/README.zh-CN.md

@@ -34,7 +34,7 @@ Concept 排名会覆盖请求所声明源码预算内接纳的全部 UTF-8 段
 
 Concept 或 hybrid 较慢时，会在默认五秒等待窗口内返回 `status: "waiting"` 或 `"running"`、`operationId`、进度和精确的 `nextRequest`，例如 `{ "mode": "await", "operationId": "..." }`。请原样复制这个请求：它会续接同一个计算，不会重启查询，也不会降级成只有字面的结果。最终结果可稳定复取十分钟；每个服务会话最多保留 32 个终态结果。`mode: "cancel"` 会停止自有任务并等待清理完成。每个服务会话最多同时接纳八个 pending operation；单个 operation 使用 `SIFT_LIGHT_CONCEPT_TIMEOUT_MS` 指定一个总执行时限（整数毫秒，1000–3600000，默认 600000），另有 120 秒无人续接租期。真实模型、来源或资源故障会以明确失败返回。发布结果前会重新枚举并校验同一来源 generation；源文件变化会刷新 operation，混合版本不会被标成 complete。接纳计划计数（`filesEnumerated`、`filesAdmitted`、`filesSkippedEmpty`、`filesUnavailable`、`passagesQueued`、`batchesPlanned`、`batchesCompleted`）会在结果里明确显示。空文件属于正常跳过，不会把结果标成 partial。
 
-首次运行尚未缓存的 Concept 或 hybrid 搜索时需要加载本地模型；耗时可能达到数十秒，推理 worker 内存也可能超过 1 GiB，具体取决于机器和搜索范围。后续搜索即使换了问题，也会复用已缓存的段落向量；新问题仍会短暂加载模型来计算问题向量。修改过的段落需要重算；缓存被清理、淘汰或大量源码变化时可能再次冷运行。精确搜索模式不会加载模型。这些都是随负载变化的观察，不是延迟或内存保证。结果在 `model-loading` 阶段等待时，请按返回的 `nextRequest` 续接同一个 operation。
+首次运行尚未缓存的 Concept 或 hybrid 搜索时需要加载本地模型；耗时可能达到数十秒，推理 worker 内存也可能超过 1 GiB，具体取决于机器和搜索范围。后续搜索即使换了问题，也会复用已缓存的段落向量；活跃服务会复用已加载的模型。E5 worker 空闲 30 秒后释放，之后的查询会再次加载。修改过的段落需要重算；缓存被清理、淘汰或大量源码变化时可能再次冷运行。精确搜索模式不会加载模型。这些都是随负载变化的观察，不是延迟或内存保证。结果在 `model-loading` 阶段等待时，请按返回的 `nextRequest` 续接同一个 operation。
 
 ### 几个条件，可以一起交代
 
@@ -73,6 +73,30 @@ Pi 和 OMP 的被动 session 状态会显示当前加载的包版本，统计已
 
 可读正文会保持精简；每项证据的范围、计数、覆盖状态和继续请求仍保留在结构化 `details` 中，客户端无需为了拿到这些字段再次搜索。
 
+### Apple Silicon 独立文本检索
+
+可选的 `nativeText` 配置让 `concept` 和 `hybrid` 同时返回两路本地结果：现有检索和原生文本检索。两路各自保留排名，`details.analysis.retrievalLanes.current` 与 `.native` 给出可直接执行的读取请求。`hybrid` 配合 `ranking: "relevance"` 时，默认结果按文件倒数排名加权，原生路权重为 1、现有路为 0.1，最多返回十个不同文件；其他双路视图交错排列，两路原始分数不作直接比较。默认结果和两路结果都可继续 `inspect` 与 `validate`，源码变化会被报告。
+
+在 Apple Silicon macOS 上，准备 Xcode、Metal Toolchain、本地 SDK checkout 和模型目录，然后在项目 checkout 中运行：
+
+```sh
+bun run build:native-text -- /absolute/sdk-checkout /absolute/native-text-output
+```
+
+构建器使用固定版本，输出程序及必需的资源 bundle。模型目录应包含 `model.safetensors`、`config.json`、`tokenizer.json` 和 `tokenizer_config.json`。将完整输出目录保留在本机，并在宿主实际读取的 `sift-light.json` 中配置：
+
+```json
+{
+  "vectorSearchEnabled": true,
+  "nativeText": {
+    "executable": "/absolute/native-text-output/native-text",
+    "modelDirectory": "/absolute/model-directory"
+  }
+}
+```
+
+现有 E5 模型仍需安装。原生程序只读取现有源码准入策略允许的私有临时文本副本，索引在服务关闭时删除；同一服务内增量重用索引，连续请求复用同一个已加载的原生模型；空闲 60 秒后退出 worker 并释放内存，之后的请求会重新启动。取消、关闭和协议失败都会等待自有 worker 退出后再继续。`details.analysis.stats` 中的 `nativeModelLoads`、`nativeModelReused` 和 `nativeProcessId` 可用于核对复用状态。隐藏文件、glob、exclude 与读取预算由现有准入层决定，未识别的文本后缀从已核验的 UTF-8 字节转换为纯文本副本。程序、模型或协议失败会明确报错。省略 `nativeText` 即使用现有行为。这一配置仅启用文本，不启用图片、音频或视频检索。
+
 ## 常见用法
 
 直接向 Agent 表达需求即可，例如：
@@ -184,3 +208,56 @@ Kimi Code 的 web 模式可能从安装目录启动插件 MCP 服务。如果相
 本地搜索在你的机器上进行。请只允许 Agent 读取已获授权的文件。HTTP 服务对外开放前需要认证网关，详见[安全说明](SECURITY.md)。
 
 [更新记录](CHANGELOG.md) · [参与贡献](CONTRIBUTING.md) · [AGPL-3.0-only 许可证](LICENSE)
+
+### 图片与扫描页检索（MCP）
+
+在配置中设置 `nativeMedia.executable` 和 `nativeMedia.modelDirectory` 的绝对路径，然后重新连接 MCP。Apple Silicon 上的构建命令为 `bun run build:native-media <sdk-checkout> <output-directory>`。模型在本地运行，启用视觉编码，不启用音频编码。
+
+新增的 `sift-light-media` 工具使用 `mode: "search"`、一个明确的本地 `path`，以及二选一的 `query`（文字描述）或 `image`（本地图片路径）。`kind` 可为 `all`、`image` 或 `scan`。`glob`、`exclude`、`hidden`、`ignorePolicy` 沿用工作区文件准入规则。文字、代码和源码导航继续使用 `sift-light`。
+
+```json
+{ "mode": "search", "path": "/absolute/project/assets", "query": "草地上奔跑的狗", "kind": "image" }
+```
+
+```json
+{
+  "mode": "search",
+  "path": "/absolute/project/assets",
+  "image": "/absolute/query.jpg",
+  "kind": "all"
+}
+```
+
+结果最多保留十个不同文件，包含原文件内容哈希、原 PDF 页码和可直接执行的 `inspect`、`validate` 请求。把这些请求继续交给本工具：`inspect` 返回最长边不超过 1024 像素的真实 PNG；扫描页返回原 PDF 中对应页面。原文件发生变化时，查看会明确拒绝，校验会报告过期。混合 PDF 中的扫描页也保留原页码。相似度只是候选依据，不保证识别所有小字，也不提供文字区域坐标。
+
+### 第三方模型（BYO）
+
+本地模型仍然使用 `nativeMedia`。如果要接入自己提供的多模态服务，在同一份 `sift-light.json` 中增加 `media`。服务必须接受下面约定的 HTTP JSON 请求；它可以是你的第三方服务，也可以是你在本机部署的转换网关。没有 API Key 的本机服务可以省略 `apiKeyEnv`；远程服务应使用 HTTPS。
+
+```json
+{
+  "media": {
+    "provider": "http-json",
+    "endpoint": "https://your-provider.example/v1/media-search",
+    "model": "your-vision-model",
+    "apiKeyEnv": "YOUR_MEDIA_KEY",
+    "timeoutMs": 120000,
+    "maxRetries": 2,
+    "maxRequestBytes": 33554432,
+    "renderer": {
+      "executable": "/absolute/native-media",
+      "modelDirectory": "/absolute/model-directory"
+    }
+  }
+}
+```
+
+插件会把准入的文件以受限大小的 Base64 内容发送，并使用不含本地绝对路径的 `fileId`；返回结果必须包含 `version: 1`、`indexedFiles` 和 `hits`，每个命中使用 `fileId`、`score`、`kind`、`chunkCount`、`width`、`height`，扫描 PDF 还要返回从 1 开始的 `page`。插件收到结果后才把 ID 映射回本地原文件，并继续做哈希、页码、`inspect` 和 `validate` 校验。远程服务不会决定源码证据是否可信。
+
+Jev 也支持 `semanticJudge.provider: "http-json"`。它接收 `{ "version": 1, "model", "query", "candidates" }`，返回 `{ "version": 1, "model", "judgments", "usage" }`；每个 judgment 包含候选 `id`、分类 `choice` 和 0 到 1 的 `probability`。如果配置了 `apiKeyEnv`，插件只从该环境变量读取密钥，不把密钥写入配置、日志或请求内容以外的地方。没有密钥的本机 provider 可以直接使用；远程 provider 失败时，本地候选会保留并标记为 partial。
+
+远程多模态结果仍沿用同一个 `sift-light-media` 工具和同一套源文件核验。PNG 命中可以直接 inspect；PDF 或其他图片格式要返回预览时，请在 `media.renderer` 中配置本地渲染 worker。只做 `validate` 不需要 renderer。
+
+首次索引较慢时，工具返回 `nextRequest` 和 `mode: "await"`；照原请求继续等待即可。使用 `mode: "cancel"` 和对应 `operationId` 可以取消。本地 provider 的连续请求复用一个视觉模型，空闲 60 秒后释放；远程 provider 的超时、重试和请求大小由配置控制。取消和关闭由服务回收自有资源。每次最多准入 512 个媒体文件，每个文件不超过 20 MiB，总量不超过 256 MiB；数量、总量和枚举限制会明确标为部分覆盖。单次任务期限为五分钟。结果保留十分钟，且只在原工作区中可用。当前入口不包含 Office、音频或视频。
+
+原生文本和视觉程序在已加载模型之外设置 1 GB 推理余量，使用计算所需的工作空间，并在每次查询后归还空闲计算缓存；这不是进程总内存的硬上限。模型本身仍会占用数 GB，索引和计算期间还有额外内存。每个 MCP 服务会话分别持有自己的模型；多个连接或文本与视觉同时工作时，占用会相加。普通精确搜索不加载这些模型。运行内存和速度必须按实际模型、机器与文件范围测量。

package/native/WorkerMemory.swift

@@ -0,0 +1,18 @@
+import Foundation
+import OmniKit
+
+/// Bounds transient inference allocations without putting the loaded model below its working floor.
+enum WorkerMemory {
+    static let headroom = 1_000_000_000
+
+    static func prepare(modelDirectory: String) throws {
+        let weights = URL(fileURLWithPath: modelDirectory).appendingPathComponent("model.safetensors")
+        let bytes = try weights.resourceValues(forKeys: [.fileSizeKey]).fileSize ?? 0
+        omniSetMemoryHeadroom(headroom, residentBytes: bytes)
+    }
+
+    static func settle() {
+        omniClearGPUCache()
+        omniSetMemoryHeadroom(headroom, residentBytes: omniGPUActiveMemory())
+    }
+}

package/native/media-search/Worker.swift

@@ -0,0 +1,147 @@
+import Foundation
+import OmniKit
+import PDFKit
+import ImageIO
+import UniformTypeIdentifiers
+import CryptoKit
+
+struct MediaRequest: Decodable {
+    let id: String
+    let query: String?
+    let image: String?
+    let expectedFiles: Int
+    let kind: String
+}
+struct MediaHit: Encodable {
+    let path: String
+    let score: Float
+    let kind: String
+    let page: Int?
+    let chunkCount: Int
+    let width: Int
+    let height: Int
+}
+struct MediaResponse: Encodable {
+    let version = 1
+    let id: String
+    let indexedFiles: Int
+    let modelLoads = 1
+    let modelReused: Bool
+    let processId = ProcessInfo.processInfo.processIdentifier
+    let hits: [MediaHit]
+}
+@main struct MediaWorker {
+    enum Failure: Error { case arguments, inventory, query, locator, render }
+    static func main() async throws {
+        let args = CommandLine.arguments
+        if args.count == 5 && args[1] == "--render" {
+            try render(path: args[2], page: Int(args[3]) ?? 0, output: args[4]); return
+        }
+        guard args.count == 5, args[4] == "--serve" else { throw Failure.arguments }
+        try WorkerMemory.prepare(modelDirectory: args[1])
+        let engine = try await OmniEngine(modelDir: URL(fileURLWithPath: args[1]), keepVision: true, keepAudio: false)
+        WorkerMemory.settle()
+        let store = try VectorStore(dbURL: URL(fileURLWithPath: args[3]))
+        defer { store.close() }
+        var completed = 0
+        var fingerprints: [String: String] = [:]
+        while let line = readLine() {
+            guard line.utf8.count <= 4096 else { throw Failure.arguments }
+            let r = try JSONDecoder().decode(MediaRequest.self, from: Data(line.utf8))
+            guard !r.id.isEmpty, r.expectedFiles > 0, ["all", "image", "scan"].contains(r.kind),
+                  (r.query == nil) != (r.image == nil) else { throw Failure.arguments }
+            let corpus = URL(fileURLWithPath: args[2])
+            let originals = try inventory(corpus)
+            let scanRoot = corpus.deletingLastPathComponent().appendingPathComponent("scan-pages")
+            let aliases = try prepareMixedScans(originals, scanRoot: scanRoot)
+            let admitted = originals.union(aliases.keys)
+            let currentFingerprints = try Dictionary(uniqueKeysWithValues: admitted.map { path in
+                (path, SHA256.hash(data: try Data(contentsOf: URL(fileURLWithPath: path))).map { String(format: "%02x", $0) }.joined())
+            })
+            var stale = Set<String>()
+            store.knownFiles().forEach { path, _ in if !admitted.contains(path) || (fingerprints[path] != nil && fingerprints[path] != currentFingerprints[path]) { stale.insert(path) } }
+            if !stale.isEmpty { store.deletePaths(stale) }
+            var settings = IndexSettings(enabledKinds: [.image, .text])
+            settings.ignore = OmniIgnore(text: "")
+            Indexer(store: store, embedder: engine).index(roots: [corpus, scanRoot], settings: settings) { _ in }
+            guard store.fileCount == r.expectedFiles + aliases.count else { throw Failure.inventory }
+            fingerprints = currentFingerprints
+            let vector: [Float]
+            if let query = r.query {
+                guard !query.isEmpty, query.count <= 256 else { throw Failure.query }
+                vector = engine.embedText(query, as: .query)
+            } else {
+                guard let image = r.image, let v = engine.embedFileQuery(URL(fileURLWithPath: image), asDocument: true) else { throw Failure.query }
+                vector = v
+            }
+            var filter = SearchFilter()
+            filter.kinds = r.kind == "all" ? ["image", "scan"] : [r.kind]
+            let hits = try store.search(vector, filter: filter, topK: 10, textQuery: r.query).map { h in
+                var page: Int? = nil
+                if h.kind == "scan" {
+                    guard h.locator.hasPrefix("Page "), let p = Int(h.locator.dropFirst(5)), p > 0, p == h.chunkIndex + 1 else { throw Failure.locator }
+                    page = p
+                }
+                if let alias = aliases[h.path] {
+                    guard let p = page, p <= alias.pages.count else { throw Failure.locator }
+                    return MediaHit(path: alias.path, score: h.score, kind: h.kind, page: alias.pages[p - 1], chunkCount: alias.total, width: h.width, height: h.height)
+                }
+                return MediaHit(path: h.path, score: h.score, kind: h.kind, page: page, chunkCount: h.chunkCount, width: h.width, height: h.height)
+            }
+            let result = MediaResponse(id: r.id, indexedFiles: originals.count, modelReused: completed > 0, hits: hits)
+            WorkerMemory.settle()
+            FileHandle.standardOutput.write(try JSONEncoder().encode(result)); FileHandle.standardOutput.write(Data([10]))
+            completed += 1
+        }
+    }
+    struct ScanAlias { let path: String; let pages: [Int]; let total: Int }
+    static func prepareMixedScans(_ paths: Set<String>, scanRoot: URL) throws -> [String: ScanAlias] {
+        let fm = FileManager.default
+        try fm.createDirectory(at: scanRoot, withIntermediateDirectories: true)
+        var aliases: [String: ScanAlias] = [:]
+        for path in paths.sorted() where URL(fileURLWithPath: path).pathExtension.lowercased() == "pdf" {
+            let url = URL(fileURLWithPath: path)
+            guard let doc = PDFDocument(url: url), !doc.isLocked else { throw Failure.inventory }
+            // Entire scans retain the same extraction and streaming path.
+            guard case .pagedText = try FileExtractor.extract(url) else { continue }
+            let scanPages = (0..<doc.pageCount).filter { (doc.page(at: $0)?.string ?? "").trimmingCharacters(in: .whitespacesAndNewlines).count < FileExtractor.minCharsPerPage }
+            guard !scanPages.isEmpty else { continue }
+            let key = SHA256.hash(data: try Data(contentsOf: url) + Data(path.utf8)).map { String(format: "%02x", $0) }.joined()
+            let output = scanRoot.appendingPathComponent(key + ".pdf")
+            if !fm.fileExists(atPath: output.path) {
+                let scans = PDFDocument()
+                for (i, pageIndex) in scanPages.enumerated() {
+                    guard let page = doc.page(at: pageIndex)?.copy() as? PDFPage else { throw Failure.render }
+                    scans.insert(page, at: i)
+                }
+                guard scans.write(to: output) else { throw Failure.render }
+            }
+            aliases[output.path] = ScanAlias(path: path, pages: scanPages.map { $0 + 1 }, total: doc.pageCount)
+        }
+        for url in try fm.contentsOfDirectory(at: scanRoot, includingPropertiesForKeys: nil) where aliases[url.path] == nil { try fm.removeItem(at: url) }
+        return aliases
+    }
+    static func inventory(_ corpus: URL) throws -> Set<String> {
+        guard let files = FileManager.default.enumerator(at: corpus, includingPropertiesForKeys: [.isRegularFileKey]) else { throw Failure.inventory }
+        var admitted = Set<String>()
+        for case let url as URL in files {
+            if try url.resourceValues(forKeys: [.isRegularFileKey]).isRegularFile == true { admitted.insert(url.path) }
+        }
+        return admitted
+    }
+    static func render(path: String, page: Int, output: String) throws {
+        let image: CGImage
+        let url = URL(fileURLWithPath: path)
+        if page > 0 {
+            guard let doc = PDFDocument(url: url), page <= doc.pageCount,
+                  let img = FileExtractor.renderPDFPage(doc, index: page - 1, maxDimension: 1024) else { throw Failure.render }
+            image = img
+        } else {
+            guard case .images(let imgs) = try FileExtractor.extract(url, maxImageDimension: 1024), let img = imgs.first else { throw Failure.render }
+            image = img
+        }
+        guard let dest = CGImageDestinationCreateWithURL(URL(fileURLWithPath: output) as CFURL, UTType.png.identifier as CFString, 1, nil) else { throw Failure.render }
+        CGImageDestinationAddImage(dest, image, nil)
+        guard CGImageDestinationFinalize(dest) else { throw Failure.render }
+    }
+}

package/native/text-search/Worker.swift

@@ -0,0 +1,129 @@
+import Foundation
+import OmniKit
+import CryptoKit
+
+struct Request: Decodable {
+    let id: String?
+    let query: String
+    let expectedFiles: Int
+}
+
+struct Hit: Encodable {
+    let path: String
+    let score: Float
+    let line: Int
+    let chunkIndex: Int
+    let locator: String
+}
+
+struct Response: Encodable {
+    let version = 2
+    let id: String
+    let modelLoads = 1
+    let modelReused: Bool
+    let processId = ProcessInfo.processInfo.processIdentifier
+    let indexedFiles: Int
+    let hits: [Hit]
+}
+
+@main struct TextSearchWorker {
+    static func main() async throws {
+        let args = CommandLine.arguments
+        guard args.count == 4 || (args.count == 5 && args[4] == "--serve") else { throw Failure.invalidArguments }
+        let serving = args.count == 5
+        try WorkerMemory.prepare(modelDirectory: args[1])
+        let engine = try await OmniEngine(modelDir: URL(fileURLWithPath: args[1]), keepVision: false, keepAudio: false)
+        WorkerMemory.settle()
+        let store = try VectorStore(dbURL: URL(fileURLWithPath: args[3]))
+        defer { store.close() }
+        var completed = 0
+        while let data = nextRequest(serving: serving, completed: completed) {
+            guard data.count <= 4096 else { throw Failure.invalidArguments }
+            let request = try JSONDecoder().decode(Request.self, from: data)
+            guard !request.query.isEmpty, request.query.count <= 256, request.expectedFiles > 0,
+                  !serving || (request.id?.isEmpty == false) else { throw Failure.invalidArguments }
+            let result = try search(request, engine: engine, store: store, path: args[2], reused: completed > 0)
+            WorkerMemory.settle()
+            FileHandle.standardOutput.write(try JSONEncoder().encode(result))
+            if serving { FileHandle.standardOutput.write(Data([10])) }
+            completed += 1
+        }
+    }
+
+    static func nextRequest(serving: Bool, completed: Int) -> Data? {
+        if serving { return readLine(strippingNewline: true).map { Data($0.utf8) } }
+        return completed == 0 ? FileHandle.standardInput.readDataToEndOfFile() : nil
+    }
+
+    static func search(_ request: Request, engine: OmniEngine, store: VectorStore, path: String, reused: Bool) throws -> Response {
+        let (corpus, normalized, aliases, settings, admitted) = try prepareCorpus(path)
+        // A private snapshot has a complete inventory; empty subroots are intentional.
+        var stale = Set<String>()
+        store.knownFiles().forEach { path, _ in
+            if !admitted.contains(path) { stale.insert(path) }
+        }
+        if !stale.isEmpty { store.deletePaths(stale) }
+        let indexer = Indexer(store: store, embedder: engine)
+        indexer.index(roots: [corpus, normalized], settings: settings) { _ in }
+        guard store.fileCount == request.expectedFiles else {
+            FileHandle.standardError.write(Data("Native index inventory: indexed=\(store.fileCount), expected=\(request.expectedFiles)\n".utf8))
+            throw Failure.incompleteIndex
+        }
+        let vector = engine.embedText(request.query, as: .query)
+        let hits = try store.search(vector, topK: 10, textQuery: request.query).map { hit in
+            guard hit.kind == "text", hit.locator.hasPrefix("Line "),
+                  let line = Int(hit.locator.dropFirst(5)), line > 0 else { throw Failure.invalidLocator }
+            return Hit(path: aliases[hit.path] ?? hit.path, score: hit.score, line: line, chunkIndex: hit.chunkIndex, locator: hit.locator)
+        }
+        return Response(id: request.id ?? "", modelReused: reused, indexedFiles: store.fileCount, hits: hits)
+    }
+
+    static func prepareCorpus(_ path: String) throws -> (URL, URL, [String: String], IndexSettings, Set<String>) {
+        let corpus = URL(fileURLWithPath: path)
+        let normalized = corpus.deletingLastPathComponent().appendingPathComponent("plain-text")
+        let fm = FileManager.default
+        try fm.createDirectory(at: normalized, withIntermediateDirectories: true)
+        var aliases: [String: String] = [:]
+        var admitted = Set<String>()
+        guard let files = fm.enumerator(at: corpus, includingPropertiesForKeys: [.isRegularFileKey]) else {
+            throw Failure.incompleteIndex
+        }
+        for case let url as URL in files {
+            guard try url.resourceValues(forKeys: [.isRegularFileKey]).isRegularFile == true else { continue }
+            if FileExtractor.textExtensions.contains(url.pathExtension.lowercased()) {
+                admitted.insert(url.path)
+                continue
+            }
+            let key = SHA256.hash(data: Data(url.path.utf8)).map { String(format: "%02x", $0) }.joined()
+            let folder = normalized.appendingPathComponent(key)
+            try fm.createDirectory(at: folder, withIntermediateDirectories: true)
+            let alias = folder.appendingPathComponent(url.lastPathComponent + ".txt")
+            let bytes = try Data(contentsOf: url)
+            if (try? Data(contentsOf: alias)) != bytes { try bytes.write(to: alias) }
+            aliases[alias.path] = url.path
+            admitted.insert(alias.path)
+        }
+        if let old = fm.enumerator(at: normalized, includingPropertiesForKeys: [.isRegularFileKey]) {
+            for case let url as URL in old where aliases[url.path] == nil {
+                if try url.resourceValues(forKeys: [.isRegularFileKey]).isRegularFile == true {
+                    try fm.removeItem(at: url)
+                }
+            }
+        }
+        var settings = IndexSettings(enabledKinds: [.text])
+        settings.ignore = OmniIgnore(text: "")
+        // Source admission is complete before this process receives its private corpus.
+        // Non-plain suffixes are normalized from already verified UTF-8 source bytes.
+        // Only normalized plain text may enter the index for these files.
+        if !aliases.isEmpty {
+            settings.ignore = OmniIgnore(text: aliases.values.map { path in
+                path.map { ch in "\\*?[]#!".contains(ch) ? "\\" + String(ch) : String(ch) }.joined()
+            }.joined(separator: "\n"))
+        }
+        return (corpus, normalized, aliases, settings, admitted)
+    }
+
+    enum Failure: Error {
+        case invalidArguments, incompleteIndex, invalidLocator
+    }
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sift-light",
-  "version": "1.0.5-6",
+  "version": "1.1.0",
   "description": "Context-efficient local search for files, documents, notes and logs across Pi, OMP and MCP clients",
   "keywords": [
     "ai-agent",
@@ -46,7 +46,13 @@
     "CHANGELOG.md",
     "SECURITY.md",
     "src/concept-worker.mjs",
-    "plugins/sift-light/**"
+    "plugins/sift-light/**",
+    "native/text-search/Worker.swift",
+    "native/WorkerMemory.swift",
+    "scripts/build-native-text.ts",
+    "native/media-search/Worker.swift",
+    "scripts/build-native-media.ts",
+    "scripts/build-native-worker.ts"
   ],
   "type": "module",
   "publishConfig": {
@@ -82,7 +88,9 @@
     "format:local:check": "bun run doc/testing/scripts/format-local.ts --check",
     "lint:local": "oxlint --no-ignore --type-aware --tsconfig doc/testing/tsconfig.json --deny-warnings --report-unused-disable-directives src scripts doc/testing/test doc/testing/scripts",
     "typecheck:local": "tsc --noEmit --project doc/testing/tsconfig.json",
-    "check:local": "bun run format:check && bun run format:local:check && bun run check:search-plugin && bun run check:concept-worker && bun run check:worker && bun run check:mcp && bun run lint:local && bun run typecheck:local && bun run test:repo && bun run test:node:repo && bun run benchmark"
+    "check:local": "bun run format:check && bun run format:local:check && bun run check:search-plugin && bun run check:concept-worker && bun run check:worker && bun run check:mcp && bun run lint:local && bun run typecheck:local && bun run test:repo && bun run test:node:repo && bun run benchmark",
+    "build:native-text": "bun run scripts/build-native-text.ts",
+    "build:native-media": "bun run scripts/build-native-media.ts"
   },
   "dependencies": {
     "@ast-grep/lang-go": "0.0.6",

package/plugins/sift-light/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "sift-light",
-  "version": "1.0.5-6",
+  "version": "1.1.0",
   "description": "Require sift-light for conventional local searches while keeping development tools available.",
   "author": {
     "name": "baoer"

package/plugins/sift-light/.codex-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "sift-light",
-  "version": "1.0.5-6",
+  "version": "1.1.0",
   "description": "Require sift-light for conventional local searches while keeping development tools available.",
   "author": {
     "name": "baoer"

package/plugins/sift-light/.mcp.json

@@ -5,7 +5,7 @@
       "args": [
         "--yes",
         "--package",
-        "sift-light-runtime@npm:sift-light@1.0.5-6",
+        "sift-light-runtime@npm:sift-light@1.1.0",
         "sift-light-mcp",
         "--stdio"
       ],

package/plugins/sift-light/kimi.plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "sift-light",
-  "version": "1.0.5-6",
+  "version": "1.1.0",
   "description": "Require sift-light for conventional local searches while keeping development tools available.",
   "author": {
     "name": "baoer"
@@ -13,7 +13,7 @@
       "args": [
         "--yes",
         "--package",
-        "sift-light-runtime@npm:sift-light@1.0.5-6",
+        "sift-light-runtime@npm:sift-light@1.1.0",
         "sift-light-mcp",
         "--stdio"
       ],