sift-rag 0.1.0__tar.gz

sift_rag-0.1.0/CHANGELOG.md

@@ -0,0 +1,26 @@
+# Changelog
+
+All notable changes are listed here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and versions follow [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-10-18
+
+First release.
+
+### Added
+
+- `sift add`, `rm`, `index`, `embed`, `search`, `vsearch`, `query`, `get`, `status` and `serve`; also available as
+  `sift-rag`, so `uvx sift-rag serve` works.
+- Incremental indexing of `.md`, `.markdown` and `.txt` files in parallel worker processes, with filterable YAML
+  front matter.
+- Embedding on NVIDIA GPUs (CUDA), Apple GPUs (MPS) or the CPU. sift caps its GPU memory below what other programs
+  leave free, calibrates the batch size by measurement, and reports progress.
+- Keyword search (SQLite FTS5, BM25), exact GPU vector search, and hybrid search with cross-encoder reranking.
+- An MCP server (streamable HTTP and stdio) whose instructions tell agents what is indexed, and whose errors tell
+  them what went wrong.
+- A start-up hint when an NVIDIA GPU can't be used, with the command that fixes it.
+
+[Unreleased]: https://github.com/tobiaswuerth/sift/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/tobiaswuerth/sift/releases/tag/v0.1.0

sift_rag-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Tobias Würth
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sift_rag-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+graft tests
+include CHANGELOG.md
+global-exclude __pycache__ *.py[cod]

sift_rag-0.1.0/PKG-INFO

@@ -0,0 +1,210 @@
+Metadata-Version: 2.4
+Name: sift-rag
+Version: 0.1.0
+Summary: Fast local search over text and Markdown files, built for AI agents.
+Author: Tobias Würth
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/tobiaswuerth/sift
+Project-URL: Documentation, https://github.com/tobiaswuerth/sift#readme
+Project-URL: Issues, https://github.com/tobiaswuerth/sift/issues
+Project-URL: Changelog, https://github.com/tobiaswuerth/sift/blob/main/CHANGELOG.md
+Keywords: search,rag,embeddings,mcp,markdown,bm25,semantic-search
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Environment :: GPU :: NVIDIA CUDA
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: MacOS
+Classifier: Operating System :: Microsoft :: Windows
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.26
+Requires-Dist: torch>=2.4
+Requires-Dist: sentence-transformers>=5.0
+Requires-Dist: mcp>=2.0
+Requires-Dist: nvidia-ml-py>=12
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-cov>=5; extra == "dev"
+Requires-Dist: ruff>=0.6; extra == "dev"
+Requires-Dist: pre-commit>=4; extra == "dev"
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=6; extra == "dev"
+Dynamic: license-file
+
+# sift
+
+[![PyPI](https://img.shields.io/pypi/v/sift-rag)](https://pypi.org/project/sift-rag/)
+[![Python](https://img.shields.io/pypi/pyversions/sift-rag)](https://pypi.org/project/sift-rag/)
+[![CI](https://github.com/tobiaswuerth/sift/actions/workflows/ci.yml/badge.svg)](https://github.com/tobiaswuerth/sift/actions/workflows/ci.yml)
+[![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)](https://github.com/tobiaswuerth/sift/actions/workflows/ci.yml)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Fast local search over your text and Markdown files, built for AI agents.
+
+sift indexes folders of `.md` and `.txt` files, embeds them on your GPU, and runs a server that agents query over MCP.
+It offers three kinds of search: exact keyword matching, semantic search, and hybrid retrieval with reranking for RAG.
+Everything runs locally. No cloud services, no API keys.
+
+## Install
+
+Requires Python 3.11 or newer. With [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv tool install --torch-backend auto sift-rag   # installs the `sift` command
+```
+
+Or run it without installing: `uvx --torch-backend auto sift-rag serve`. With pip, in a virtual environment:
+`pip install sift-rag`.
+
+On first use, sift downloads its two models from Hugging Face (about 1.5 GB).
+
+### GPU support
+
+sift uses an NVIDIA GPU (CUDA) if there is one, else an Apple GPU, else the CPU, which works but embeds slowly.
+
+- **Linux, macOS:** nothing to do. PyTorch from PyPI includes CUDA on Linux and uses the Apple GPU on macOS.
+- **Windows:** PyTorch from PyPI has no CUDA support. uv's `--torch-backend auto`, used above, picks the CUDA build
+  that matches your driver. With pip, install it afterwards:
+  `pip install --force-reinstall --no-deps torch --index-url https://download.pytorch.org/whl/cu128`.
+
+If an NVIDIA GPU sits unused, sift says so on start-up and prints the command that fixes it.
+
+## Usage
+
+```bash
+sift add ~/papers --name papers        # register a folder as a collection
+sift index                             # scan files, chunk them, build the keyword index
+sift embed                             # embed every chunk that has no vector yet
+sift search '"reward shaping"'         # exact keyword search
+sift vsearch "how do robots plan brush strokes"
+sift query "mixing pigments to match a target colour"
+sift get papers/2401.02225.md          # full document
+sift status                            # collections and counts
+sift rm papers                         # remove a collection from the index; its files stay on disk
+sift serve                             # MCP server on http://127.0.0.1:8765/mcp
+```
+
+`sift index` and `sift embed` are incremental: run them again after files change, and only new or changed files are
+processed. Deleted files are removed. An interrupted `sift embed` continues where it stopped. A running server picks up
+new vectors on its own.
+
+Search commands accept `-k` (number of results), `-c` (collection), `--path` (glob within the collection, e.g.
+`"2024/*"`), `-f KEY=VALUE` (front-matter filter, repeatable) and `--json`.
+
+### Front matter
+
+YAML front matter with flat `key: value` lines becomes filterable metadata. A list field matches when it contains the
+value:
+
+```markdown
+---
+title: "Brush stroke planning"
+categories: ["cs.RO", "cs.GR"]
+year: 2024
+---
+```
+
+```bash
+sift query "stroke order" -f categories=cs.RO -f year=2024
+```
+
+The `title` field, or else the first heading, or else the file name, becomes the document title.
+
+## MCP
+
+| Tool | Purpose |
+|---|---|
+| `search` | Exact keyword search, ranked by BM25. Supports `"phrases"`, `prefix*`, `AND`, `OR`, `NOT`. |
+| `vsearch` | Semantic search. |
+| `query` | Hybrid search for RAG: keyword and semantic candidates, reranked by a cross-encoder. |
+| `get` | Full text of a document by its id. |
+| `status` | Collections and counts. |
+
+The search tools take `query`, `k`, `collection`, `path` and `filters`. Every result carries `id`
+(`collection/path`), `title`, `lines`, `score` and the chunk `text`, so agents can cite their sources.
+
+Claude Code, with the server running (`sift serve`), shared by any number of agents:
+
+```bash
+claude mcp add --transport http sift http://127.0.0.1:8765/mcp
+```
+
+Or let Claude Code start its own server over stdio:
+
+```bash
+claude mcp add sift -- sift serve --stdio
+```
+
+## Configuration
+
+None needed: sift picks the device, the GPU memory budget, the batch size and the number of worker processes itself.
+Three optional settings exist for special cases:
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `SIFT_HOME` | `~/.sift` | Where the index lives |
+| `SIFT_EMBED_MODEL` | `Qwen/Qwen3-Embedding-0.6B` | Any Sentence Transformers embedding model |
+| `SIFT_RERANK_MODEL` | `Alibaba-NLP/gte-reranker-modernbert-base` | Any Sentence Transformers cross-encoder |
+
+After changing the embedding model, run `sift embed --reset`. Models are downloaded from Hugging Face on first use.
+
+## How it works
+
+- **Indexing:** worker processes read and parse files in parallel while the previous batch is written; small
+  updates skip the process start-up. Files are compared by modification time and size, then by content hash, so
+  unchanged files cost nothing. Markdown is split into chunks of about 3,000 characters (about 750 tokens) on line
+  boundaries, preferring headings, with a small overlap.
+- **Storage:** one SQLite database in WAL mode holds documents, chunks, metadata, an FTS5 keyword index and the
+  vectors (fp16).
+- **Embedding:** PyTorch in fp16 on the GPU (NVIDIA, else Apple), in batches sorted by length to keep padding low.
+  Each slab of 20,000 chunks is committed, so progress is never lost.
+- **GPU memory:** sift shares the GPU with other programs. On NVIDIA GPUs it caps itself at the memory that is free
+  when it starts, minus headroom (10% of the GPU, at least 1 GB), asking NVML as well as CUDA, since on Windows CUDA
+  counts memory held by other programs as free. It then measures how much memory its longest text needs at growing
+  batch sizes and uses the largest batch that fills 90% of that budget; if a batch still doesn't fit, it is halved.
+  With less than 2 GB to spare, sift uses the CPU instead.
+- **`search`:** SQLite FTS5 with BM25, titles weighted double. Input that isn't valid FTS5 syntax is treated as exact
+  phrases.
+- **`vsearch`:** exact nearest-neighbour search as one matrix multiply over all vectors on the GPU. No approximate
+  index, so nothing to tune and no recall loss; a million 1024-dimensional vectors take about 2 GB of VRAM.
+- **`query`:** the top 40 keyword and top 40 semantic candidates are merged by reciprocal rank fusion and reranked in
+  one GPU batch.
+- **Server:** `sift serve` loads models and vectors once and keeps them in memory.
+
+## Scope
+
+In scope: plain text and Markdown files; Linux, macOS and Windows; NVIDIA GPUs (CUDA) and Apple GPUs (MPS), otherwise
+CPU; one user, any number of local agents.
+
+Out of scope for now: PDF, Word and HTML parsing (convert to Markdown first); answer generation (sift retrieves, the
+agent writes); approximate nearest-neighbour indexes; live file watching; remote access and authentication.
+
+## Development
+
+```bash
+git clone https://github.com/tobiaswuerth/sift
+cd sift
+uv venv && uv pip install --torch-backend auto -e ".[dev]"   # or with pip: pip install -r requirements.txt -e ".[dev]"
+pre-commit install      # lint, format and config checks on every commit
+pytest --cov            # 100% line and branch coverage is required
+```
+
+The tests replace the embedding and reranking models with small deterministic fakes, so they need no GPU and no
+model downloads; everything else, including the MCP server, runs for real. See [CONTRIBUTING.md](CONTRIBUTING.md)
+for how changes and releases work.
+
+## License
+
+[MIT](LICENSE)

sift_rag-0.1.0/README.md

@@ -0,0 +1,166 @@
+# sift
+
+[![PyPI](https://img.shields.io/pypi/v/sift-rag)](https://pypi.org/project/sift-rag/)
+[![Python](https://img.shields.io/pypi/pyversions/sift-rag)](https://pypi.org/project/sift-rag/)
+[![CI](https://github.com/tobiaswuerth/sift/actions/workflows/ci.yml/badge.svg)](https://github.com/tobiaswuerth/sift/actions/workflows/ci.yml)
+[![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen)](https://github.com/tobiaswuerth/sift/actions/workflows/ci.yml)
+[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
+
+Fast local search over your text and Markdown files, built for AI agents.
+
+sift indexes folders of `.md` and `.txt` files, embeds them on your GPU, and runs a server that agents query over MCP.
+It offers three kinds of search: exact keyword matching, semantic search, and hybrid retrieval with reranking for RAG.
+Everything runs locally. No cloud services, no API keys.
+
+## Install
+
+Requires Python 3.11 or newer. With [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv tool install --torch-backend auto sift-rag   # installs the `sift` command
+```
+
+Or run it without installing: `uvx --torch-backend auto sift-rag serve`. With pip, in a virtual environment:
+`pip install sift-rag`.
+
+On first use, sift downloads its two models from Hugging Face (about 1.5 GB).
+
+### GPU support
+
+sift uses an NVIDIA GPU (CUDA) if there is one, else an Apple GPU, else the CPU, which works but embeds slowly.
+
+- **Linux, macOS:** nothing to do. PyTorch from PyPI includes CUDA on Linux and uses the Apple GPU on macOS.
+- **Windows:** PyTorch from PyPI has no CUDA support. uv's `--torch-backend auto`, used above, picks the CUDA build
+  that matches your driver. With pip, install it afterwards:
+  `pip install --force-reinstall --no-deps torch --index-url https://download.pytorch.org/whl/cu128`.
+
+If an NVIDIA GPU sits unused, sift says so on start-up and prints the command that fixes it.
+
+## Usage
+
+```bash
+sift add ~/papers --name papers        # register a folder as a collection
+sift index                             # scan files, chunk them, build the keyword index
+sift embed                             # embed every chunk that has no vector yet
+sift search '"reward shaping"'         # exact keyword search
+sift vsearch "how do robots plan brush strokes"
+sift query "mixing pigments to match a target colour"
+sift get papers/2401.02225.md          # full document
+sift status                            # collections and counts
+sift rm papers                         # remove a collection from the index; its files stay on disk
+sift serve                             # MCP server on http://127.0.0.1:8765/mcp
+```
+
+`sift index` and `sift embed` are incremental: run them again after files change, and only new or changed files are
+processed. Deleted files are removed. An interrupted `sift embed` continues where it stopped. A running server picks up
+new vectors on its own.
+
+Search commands accept `-k` (number of results), `-c` (collection), `--path` (glob within the collection, e.g.
+`"2024/*"`), `-f KEY=VALUE` (front-matter filter, repeatable) and `--json`.
+
+### Front matter
+
+YAML front matter with flat `key: value` lines becomes filterable metadata. A list field matches when it contains the
+value:
+
+```markdown
+---
+title: "Brush stroke planning"
+categories: ["cs.RO", "cs.GR"]
+year: 2024
+---
+```
+
+```bash
+sift query "stroke order" -f categories=cs.RO -f year=2024
+```
+
+The `title` field, or else the first heading, or else the file name, becomes the document title.
+
+## MCP
+
+| Tool | Purpose |
+|---|---|
+| `search` | Exact keyword search, ranked by BM25. Supports `"phrases"`, `prefix*`, `AND`, `OR`, `NOT`. |
+| `vsearch` | Semantic search. |
+| `query` | Hybrid search for RAG: keyword and semantic candidates, reranked by a cross-encoder. |
+| `get` | Full text of a document by its id. |
+| `status` | Collections and counts. |
+
+The search tools take `query`, `k`, `collection`, `path` and `filters`. Every result carries `id`
+(`collection/path`), `title`, `lines`, `score` and the chunk `text`, so agents can cite their sources.
+
+Claude Code, with the server running (`sift serve`), shared by any number of agents:
+
+```bash
+claude mcp add --transport http sift http://127.0.0.1:8765/mcp
+```
+
+Or let Claude Code start its own server over stdio:
+
+```bash
+claude mcp add sift -- sift serve --stdio
+```
+
+## Configuration
+
+None needed: sift picks the device, the GPU memory budget, the batch size and the number of worker processes itself.
+Three optional settings exist for special cases:
+
+| Variable | Default | Purpose |
+|---|---|---|
+| `SIFT_HOME` | `~/.sift` | Where the index lives |
+| `SIFT_EMBED_MODEL` | `Qwen/Qwen3-Embedding-0.6B` | Any Sentence Transformers embedding model |
+| `SIFT_RERANK_MODEL` | `Alibaba-NLP/gte-reranker-modernbert-base` | Any Sentence Transformers cross-encoder |
+
+After changing the embedding model, run `sift embed --reset`. Models are downloaded from Hugging Face on first use.
+
+## How it works
+
+- **Indexing:** worker processes read and parse files in parallel while the previous batch is written; small
+  updates skip the process start-up. Files are compared by modification time and size, then by content hash, so
+  unchanged files cost nothing. Markdown is split into chunks of about 3,000 characters (about 750 tokens) on line
+  boundaries, preferring headings, with a small overlap.
+- **Storage:** one SQLite database in WAL mode holds documents, chunks, metadata, an FTS5 keyword index and the
+  vectors (fp16).
+- **Embedding:** PyTorch in fp16 on the GPU (NVIDIA, else Apple), in batches sorted by length to keep padding low.
+  Each slab of 20,000 chunks is committed, so progress is never lost.
+- **GPU memory:** sift shares the GPU with other programs. On NVIDIA GPUs it caps itself at the memory that is free
+  when it starts, minus headroom (10% of the GPU, at least 1 GB), asking NVML as well as CUDA, since on Windows CUDA
+  counts memory held by other programs as free. It then measures how much memory its longest text needs at growing
+  batch sizes and uses the largest batch that fills 90% of that budget; if a batch still doesn't fit, it is halved.
+  With less than 2 GB to spare, sift uses the CPU instead.
+- **`search`:** SQLite FTS5 with BM25, titles weighted double. Input that isn't valid FTS5 syntax is treated as exact
+  phrases.
+- **`vsearch`:** exact nearest-neighbour search as one matrix multiply over all vectors on the GPU. No approximate
+  index, so nothing to tune and no recall loss; a million 1024-dimensional vectors take about 2 GB of VRAM.
+- **`query`:** the top 40 keyword and top 40 semantic candidates are merged by reciprocal rank fusion and reranked in
+  one GPU batch.
+- **Server:** `sift serve` loads models and vectors once and keeps them in memory.
+
+## Scope
+
+In scope: plain text and Markdown files; Linux, macOS and Windows; NVIDIA GPUs (CUDA) and Apple GPUs (MPS), otherwise
+CPU; one user, any number of local agents.
+
+Out of scope for now: PDF, Word and HTML parsing (convert to Markdown first); answer generation (sift retrieves, the
+agent writes); approximate nearest-neighbour indexes; live file watching; remote access and authentication.
+
+## Development
+
+```bash
+git clone https://github.com/tobiaswuerth/sift
+cd sift
+uv venv && uv pip install --torch-backend auto -e ".[dev]"   # or with pip: pip install -r requirements.txt -e ".[dev]"
+pre-commit install      # lint, format and config checks on every commit
+pytest --cov            # 100% line and branch coverage is required
+```
+
+The tests replace the embedding and reranking models with small deterministic fakes, so they need no GPU and no
+model downloads; everything else, including the MCP server, runs for real. See [CONTRIBUTING.md](CONTRIBUTING.md)
+for how changes and releases work.
+
+## License
+
+[MIT](LICENSE)

sift_rag-0.1.0/pyproject.toml

@@ -0,0 +1,84 @@
+[build-system]
+requires = ["setuptools>=77"]  # SPDX license expressions (PEP 639)
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sift-rag"
+dynamic = ["version"]
+description = "Fast local search over text and Markdown files, built for AI agents."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Tobias Würth" }]
+requires-python = ">=3.11"
+dependencies = [
+    "numpy>=1.26",
+    "torch>=2.4",
+    "sentence-transformers>=5.0",
+    "mcp>=2.0",
+    "nvidia-ml-py>=12",  # NVML: free GPU memory as nvidia-smi sees it
+]
+keywords = ["search", "rag", "embeddings", "mcp", "markdown", "bm25", "semantic-search"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: Console",
+    "Environment :: GPU :: NVIDIA CUDA",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "Operating System :: MacOS",
+    "Operating System :: Microsoft :: Windows",
+    "Operating System :: POSIX :: Linux",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Programming Language :: Python :: 3.15",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: Text Processing :: Indexing",
+]
+
+[project.urls]
+Homepage = "https://github.com/tobiaswuerth/sift"
+Documentation = "https://github.com/tobiaswuerth/sift#readme"
+Issues = "https://github.com/tobiaswuerth/sift/issues"
+Changelog = "https://github.com/tobiaswuerth/sift/blob/main/CHANGELOG.md"
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "pytest-cov>=5", "ruff>=0.6", "pre-commit>=4", "build>=1.2", "twine>=6"]
+
+[project.scripts]
+sift = "sift.cli:main"
+sift-rag = "sift.cli:main"  # so that `uvx sift-rag ...` works: uvx runs the command named like the package
+
+[tool.setuptools]
+packages = ["sift"]
+
+[tool.setuptools.dynamic]
+version = { attr = "sift.__version__" }
+
+[tool.ruff]
+line-length = 120
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "B", "UP", "SIM", "C4", "PIE", "PT", "RUF", "PERF"]
+ignore = [
+    "B905",  # zip() without strict=
+    "SIM905",  # a word list as one split string reads better than a long list literal
+]
+
+[tool.ruff.lint.flake8-pytest-style]
+parametrize-names-type = "csv"  # @pytest.mark.parametrize("a, b", ...)
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.coverage.run]
+source = ["sift"]
+branch = true
+
+[tool.coverage.report]
+fail_under = 100
+show_missing = true
+skip_covered = true

sift_rag-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

sift_rag-0.1.0/sift/__init__.py

@@ -0,0 +1,3 @@
+"""sift: fast local search over text and Markdown files, built for AI agents."""
+
+__version__ = "0.1.0"

sift_rag-0.1.0/sift/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+main()

sift_rag-0.1.0/sift/cli.py

@@ -0,0 +1,109 @@
+"""Command-line interface."""
+
+import argparse
+import json
+import sqlite3
+import sys
+import textwrap
+
+from . import __version__, store
+
+
+def _filters(pairs):
+    out = {}
+    for pair in pairs:
+        key, sep, value = pair.partition("=")
+        if not sep:
+            raise SystemExit(f"filter must be KEY=VALUE, got {pair!r}")
+        try:
+            out[key] = json.loads(value)
+        except ValueError:
+            out[key] = value
+    return out
+
+
+def _print_hits(hits, as_json):
+    if as_json:
+        print(json.dumps(hits, ensure_ascii=False, indent=2))
+        return
+    for h in hits:
+        print(f"{h['score']:>8.4f}  {h['id']}:{h['lines']}  {h['title']}")
+        print(textwrap.indent(textwrap.shorten(h["text"], 300), " " * 10))
+
+
+def main(argv=None):
+    p = argparse.ArgumentParser(prog="sift", description="Fast local search over text and Markdown files.")
+    p.add_argument("--version", action="version", version=f"sift {__version__}")
+    sub = p.add_subparsers(dest="cmd", required=True)
+
+    s = sub.add_parser("add", help="register a folder as a collection")
+    s.add_argument("path")
+    s.add_argument("--name", help="collection name (default: folder name)")
+    s = sub.add_parser("rm", help="remove a collection from the index (its files stay on disk)")
+    s.add_argument("name")
+    s = sub.add_parser("index", help="scan collections and update the keyword index")
+    s.add_argument("names", nargs="*", help="collections to scan (default: all)")
+    s = sub.add_parser("embed", help="embed all chunks that have no vector yet")
+    s.add_argument("--reset", action="store_true", help="drop all vectors first (after changing the model)")
+    for name, desc in (
+        ("search", "exact keyword search (BM25)"),
+        ("vsearch", "semantic search"),
+        ("query", "hybrid search with reranking"),
+    ):
+        s = sub.add_parser(name, help=desc)
+        s.add_argument("query")
+        s.add_argument("-k", type=int, default=10, help="number of results (default: 10)")
+        s.add_argument("-c", "--collection")
+        s.add_argument("--path", help='path glob within the collection, e.g. "2024/*"')
+        s.add_argument(
+            "-f", "--filter", action="append", default=[], metavar="KEY=VALUE", help="front-matter filter, repeatable"
+        )
+        s.add_argument("--json", action="store_true", help="print results as JSON")
+    s = sub.add_parser("get", help="print a document by its id (collection/path)")
+    s.add_argument("id")
+    sub.add_parser("status", help="show collections and index statistics")
+    s = sub.add_parser("serve", help="run the MCP server")
+    s.add_argument("--host", default="127.0.0.1")
+    s.add_argument("--port", type=int, default=8765)
+    s.add_argument("--stdio", action="store_true", help="serve over stdio instead of HTTP")
+
+    a = p.parse_args(argv)
+    if (sys.stdout.encoding or "").lower() != "utf-8":  # e.g. a Windows pipe: titles may be any script
+        sys.stdout.reconfigure(encoding="utf-8")
+    db = store.connect()
+    try:
+        if a.cmd == "add":
+            name = store.add_collection(db, a.path, a.name)
+            print(f"added collection {name!r}; run `sift index` next", file=sys.stderr)
+        elif a.cmd == "rm":
+            n = store.remove_collection(db, a.name)
+            documents = "document" if n == 1 else "documents"
+            print(f"removed collection {a.name!r} and its {n} {documents} from the index;", end=" ", file=sys.stderr)
+            print("the files on disk are untouched", file=sys.stderr)
+        elif a.cmd == "index":
+            store.index(db, a.names)
+        elif a.cmd == "embed":
+            from .embed import embed_pending
+
+            embed_pending(db, a.reset)
+        elif a.cmd in ("search", "vsearch", "query"):
+            from .search import Index
+
+            fn = getattr(Index(db), a.cmd)
+            _print_hits(fn(a.query, a.k, a.collection, a.path, _filters(a.filter)), a.json)
+        elif a.cmd == "get":
+            from .search import Index
+
+            sys.stdout.write(Index(db).get(a.id))
+        elif a.cmd == "status":
+            from .search import Index
+
+            print(json.dumps(Index(db).status(), indent=2))
+        else:  # serve
+            from .server import serve
+
+            serve(a.host, a.port, a.stdio)
+    except (LookupError, ValueError, sqlite3.Error) as e:
+        raise SystemExit(f"error: {e}") from None
+    except KeyboardInterrupt:
+        raise SystemExit(130) from None