askfaro-embedded-search 0.5.0__tar.gz

askfaro_embedded_search-0.5.0/.github/workflows/publish.yml

@@ -0,0 +1,29 @@
+name: publish
+
+# Publishes to PyPI when a GitHub Release is published. Authenticates via
+# PyPI Trusted Publishing (OIDC) — no API token is stored anywhere.
+#
+# One-time PyPI setup (project → Publishing → Add a trusted publisher):
+#   Owner:           poolside-ventures
+#   Repository:      askfaro-embedded-search
+#   Workflow name:   publish.yml
+#   Environment:     (leave blank)
+
+on:
+  release:
+    types: [published]
+  workflow_dispatch:
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write  # required for Trusted Publishing
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Build
+        run: uv build
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

askfaro_embedded_search-0.5.0/.github/workflows/test.yml

@@ -0,0 +1,38 @@
+name: test
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    # Postgres with pgvector so the Postgres backend + the cross-backend parity
+    # test run in CI on every push — not just where a local DSN happens to exist.
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    env:
+      FARO_EMBEDDED_SEARCH_TEST_DSN: postgresql+asyncpg://postgres:postgres@localhost:5432/postgres
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Run tests
+        run: |
+          uv venv --python ${{ matrix.python-version }}
+          uv pip install -e ".[dev,postgres,numpy]"
+          uv run pytest -q

askfaro_embedded_search-0.5.0/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.py[cod]
+.venv/
+dist/
+*.egg-info/
+.pytest_cache/
+*.db
+*.db-wal
+*.db-shm

askfaro_embedded_search-0.5.0/CHANGELOG.md

@@ -0,0 +1,122 @@
+# Changelog
+
+All notable changes to `askfaro-embedded-search` are documented here. The format
+follows [Keep a Changelog](https://keepachangelog.com/), and the project aims
+to follow [Semantic Versioning](https://semver.org/).
+
+## [Unreleased]
+
+### Changed
+- Embedding failures (at both index time and query time) are now logged at
+  `WARNING` on the `askfaro_embedded_search` logger with the exception, instead of
+  being swallowed silently. Behavior is otherwise unchanged (still non-fatal,
+  availability over completeness), but a misconfigured embedder (bad key, wrong
+  endpoint) now surfaces instead of quietly degrading search to lexical-only.
+
+### Docs
+- Document partition semantics (optional, defaults to `None`, exact-match filter,
+  and the two quiet-failure modes that follow) and the new embedding-failure
+  logging in the README.
+
+## [0.5.0] - 2026-06-17
+
+### Changed
+- **Package renamed** from `faro-embedded-search` to `askfaro-embedded-search`,
+  and the import name from `faro_embedded_search` to `askfaro_embedded_search`,
+  to match the AskFaro brand. Update imports and run
+  `pip install askfaro-embedded-search`. The old name ships one final `0.4.1`
+  release that re-exports this package and warns on import. No functional change.
+  Public class names (`FaroSearchError`, etc.) are unchanged, and the default
+  Postgres table name (`faro_embedded_search_index`) is preserved for backward
+  compatibility.
+
+## [0.4.0] - 2026-06-10
+
+### Added
+- **Clear setup errors** (`FaroSearchError` and subclasses) for the common
+  adopter mistakes, replacing cryptic underlying failures:
+  - `MissingDependencyError` when `PostgresBackend` / `OpenAICompatibleEmbedder`
+    are used without their install extra — names the package and the exact
+    `pip install` command. (Also subclasses `ImportError` for compatibility.)
+  - `ConfigurationError` when the pgvector extension isn't available, when an
+    embedding's dimension doesn't match its space's column, when a `SearchIndex`
+    has an embedder for a space the backend doesn't define, or when SQLite lacks
+    FTS5 — each with the concrete fix.
+- A **Troubleshooting** table in the README mapping symptom → cause → fix.
+
+[0.4.0]: https://github.com/poolside-ventures/askfaro-embedded-search/releases/tag/v0.4.0
+
+## [0.3.0] - 2026-06-10
+
+### Added
+- **Multiple named embedding spaces.** An index can hold a vector from more
+  than one model per row — e.g. a `server` space (a hosted model) and a
+  `local` space (an on-device model) — each its own column/index, queried
+  independently. Vectors from different models are never compared.
+  - `SearchIndex(backend, embedders={"server": e1, "local": e2}, default_space="server")`
+  - Backends take a `spaces` config (`PostgresBackend(..., spaces={"server": 1536, "local": 384})`,
+    `SQLiteBackend(..., spaces=("local",))`).
+  - `IndexDoc.embed_spaces` lets an object type opt out of a space — e.g.
+    emails embed server-side only, so on-device they're keyword-searchable but
+    not semantic until you query the server.
+  - `SearchIndex.search(..., space=...)` selects which space to query.
+  - `export_shard(..., spaces=("local",))` ships a device shard carrying only
+    the on-device space (smaller — no server vectors).
+
+### Changed
+- The internal sync change-row format renamed `embedding` →
+  `embeddings: {space: vector}`. The single-embedder API is unchanged
+  (`SearchIndex(backend, embedder)` → one `"default"` space).
+- The default space's column is `embedding_default` (was `embedding` in 0.2.0).
+  As this is pre-1.0 with no migration path yet, re-create/backfill a 0.2.0
+  index rather than upgrading it in place.
+
+[0.3.0]: https://github.com/poolside-ventures/askfaro-embedded-search/releases/tag/v0.3.0
+
+## [0.2.0] - 2026-06-10
+
+### Fixed
+- **Cross-backend stemming parity.** SQLite FTS5 now uses the `porter`
+  tokenizer, so morphological variants match the same way Postgres
+  `to_tsvector('english')` stems them (e.g. "groceries" matches "grocery").
+  Previously the SQLite (device) backend did not stem, so it could rank the
+  same corpus differently from the Postgres (server) backend — a violation of
+  the library's identical-semantics guarantee.
+
+### Added
+- **Generic attribute filter.** `IndexDoc.attrs` (a JSON dict) plus an
+  `attrs=` argument to `SearchIndex.search` filters results by equality /
+  containment (e.g. `attrs={"category": "finance"}`). Backed by a JSONB GIN
+  index on Postgres and `json_extract` on SQLite. Removes the need to overload
+  `partition` for non-isolation filters.
+- `py.typed` marker so downstream type checkers see the library's annotations.
+- Idempotent in-place schema upgrades: `create_schema()` (Postgres) and
+  `SQLiteBackend(...)` add the new `attrs` column to indexes created by 0.1.0.
+
+[0.2.0]: https://github.com/poolside-ventures/askfaro-embedded-search/releases/tag/v0.2.0
+
+## [0.1.0] - 2026-06-10
+
+Initial release.
+
+### Added
+- `SearchIndex` facade: incremental per-row `upsert` / `delete` (no rebuild
+  step exists) and hybrid `search` with partition, object-type, and node-kind
+  filters.
+- Rank-based **Reciprocal Rank Fusion** (RRF, K=60) in the core, so every
+  backend ranks identically for the same corpus and query regardless of its
+  native lexical/vector score scales.
+- **Postgres backend** (`[postgres]` extra): pgvector HNSW + weighted
+  `tsvector` GIN, idempotent `create_schema()`.
+- **SQLite backend** (stdlib only): FTS5 (BM25) + exact cosine scan
+  (numpy-accelerated via the `[numpy]` extra). Its schema is the documented
+  on-device shard interchange format.
+- Node-kind tiering (`leaf` / `summary` / `cluster`) in one flat pool with
+  per-object collapsing — an incremental-safe alternative to batch RAPTOR trees.
+- Per-object-type indexer registry for heterogeneous objects.
+- Pluggable embedders, including an `OpenAICompatibleEmbedder` (OpenAI,
+  LiteLLM proxies, self-hosted). Embedding failure is non-fatal.
+- Cursor-based shard replication (`replicate`, `export_shard`) with
+  tombstone deletes, for the index-server-side / retrieve-on-device pattern.
+
+[0.1.0]: https://github.com/poolside-ventures/askfaro-embedded-search/releases/tag/v0.1.0

askfaro_embedded_search-0.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Faro
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

askfaro_embedded_search-0.5.0/PKG-INFO

@@ -0,0 +1,233 @@
+Metadata-Version: 2.4
+Name: askfaro-embedded-search
+Version: 0.5.0
+Summary: Incremental hybrid retrieval (lexical + vector, RRF-fused) with identical semantics on server Postgres and on-device SQLite
+Project-URL: Homepage, https://github.com/poolside-ventures/askfaro-embedded-search
+Project-URL: Repository, https://github.com/poolside-ventures/askfaro-embedded-search
+Project-URL: Issues, https://github.com/poolside-ventures/askfaro-embedded-search/issues
+Project-URL: Changelog, https://github.com/poolside-ventures/askfaro-embedded-search/blob/main/CHANGELOG.md
+Author: Faro
+License: MIT
+License-File: LICENSE
+Keywords: hybrid-search,on-device,pgvector,rag,retrieval,search,sqlite
+Classifier: Development Status :: 3 - Alpha
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.11
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Provides-Extra: http
+Requires-Dist: httpx>=0.27; extra == 'http'
+Provides-Extra: numpy
+Requires-Dist: numpy>=1.26; extra == 'numpy'
+Provides-Extra: postgres
+Requires-Dist: asyncpg>=0.30.0; extra == 'postgres'
+Requires-Dist: sqlalchemy[asyncio]>=2.0.36; extra == 'postgres'
+Description-Content-Type: text/markdown
+
+# askfaro-embedded-search
+
+**Incremental hybrid retrieval with identical semantics on the server and on-device.**
+
+`askfaro-embedded-search` is a small, dependency-light library for searching a *continuously growing* collection of heterogeneous objects — notes, contacts, emails, tasks, tools, anything — without ever rebuilding an index. It is built for the pattern modern assistant apps need:
+
+> **Index server-side, retrieve on-device.** Embeddings are computed once, centrally. Each user's slice of the index is replicated into a local SQLite shard, and the device answers queries locally — fast, offline, and private — with *exactly* the same ranking the server would produce.
+
+Built and dogfooded by Faro; also the embedded retrieval engine of Scope.
+
+## Why another retrieval library?
+
+Most RAG tooling assumes a batch world: ingest a corpus, build a tree/graph/index, query it. Real applications have **object-level CRUD** — a contact edited, a task created, an email deleted — and small/on-device context windows that punish irrelevant results. `askfaro-embedded-search` makes two opinionated choices:
+
+1. **Incremental-first.** One upsert per object write. Postgres (pgvector HNSW + tsvector GIN) and SQLite (FTS5 + vector blobs) both take per-row inserts natively, so there is no "rebuild" anywhere in the design. Hierarchical enrichment (summary and cluster nodes) lives in the *same flat pool* as extra rows — never on the write path.
+2. **Rank-based fusion.** Lexical and semantic retrievers run in parallel and are fused with Reciprocal Rank Fusion (RRF, K=60). Because RRF consumes *ranks*, not raw scores, the incomparable scoring scales of `ts_rank_cd` (Postgres) and `bm25()` (SQLite FTS5) don't matter: the same corpus and query rank identically on both backends. That is what makes "same engine, server and device" honest rather than aspirational.
+
+## Quick start
+
+```python
+from askfaro_embedded_search import IndexDoc, SearchIndex, OpenAICompatibleEmbedder
+from askfaro_embedded_search.backends.sqlite import SQLiteBackend
+
+index = SearchIndex(
+    SQLiteBackend("search.db"),
+    OpenAICompatibleEmbedder("https://api.openai.com/v1", api_key, "text-embedding-3-small"),
+)
+
+await index.upsert(IndexDoc(
+    object_type="note", object_id="n1",
+    title="Quantum entanglement notes",
+    body="spooky action at a distance",
+    partition="user-42",                      # isolation + shard key
+))
+
+results = await index.search("entanglement", partition="user-42", k=5)
+# SearchResult(object_id='n1', match_type='hybrid', score=..., ...)
+```
+
+Deleting and updating are first-class — deletes are tombstones so they propagate through shard sync:
+
+```python
+await index.delete("note", "n1")
+```
+
+### Filtering
+
+Filter by partition (the isolation/shard key), object type, node kind, or arbitrary structured attributes:
+
+```python
+await index.upsert(IndexDoc(
+    object_type="tool", object_id="stripe/refund",
+    title="Refund a charge", body="...",
+    attrs={"category": "finance", "status": "active"},   # structured filter fields
+))
+
+# Only finance tools:
+await index.search("refund a customer", object_types=["tool"], attrs={"category": "finance"})
+```
+
+`attrs` is matched by containment (all given keys must equal), backed by a JSONB GIN index on Postgres and `json_extract` on SQLite — so it filters identically on server and device.
+
+#### Partitions
+
+`partition` is the per-tenant isolation key (and the unit of on-device shard replication). It is **optional and defaults to `None`**, and a query's `partition` filter is an **exact match**: `search(..., partition="user-42")` returns only rows upserted with `partition="user-42"`. Two consequences worth knowing up front, because both fail quietly rather than with an error:
+
+- A row indexed **without** a partition (`None`) will **not** appear in a query that filters by a partition, and vice-versa. Pick one convention per corpus — either always set a partition, or never — and don't mix.
+- `search()` with no `partition` does **not** scope to a tenant; it searches across every partition. Pass the partition on every query in a multi-tenant index.
+
+### Server-side: Postgres
+
+```python
+from askfaro_embedded_search.backends.postgres import PostgresBackend  # pip install askfaro-embedded-search[postgres]
+
+backend = PostgresBackend("postgresql+asyncpg://...", table="faro_embedded_search_index", dim=1536)
+await backend.create_schema()     # idempotent; or transcribe the DDL into your migrations
+index = SearchIndex(backend, embedder)
+```
+
+### On-device: replicate a shard, query locally
+
+```python
+from askfaro_embedded_search import export_shard, replicate
+
+# Full export of one user's partition into a SQLite file:
+shard = await export_shard(server_backend, "user-42.db", partition="user-42")
+
+# Later, incremental delta sync (inserts, updates, AND deletes):
+cursor = await replicate(server_backend, shard, partition="user-42", cursor=cursor)
+```
+
+The shard is a plain SQLite file whose schema **is** the interchange format — any runtime that can read SQLite (a future Swift/Kotlin reader, for instance) can retrieve against it. Embeddings travel with the shard; the device never needs to re-embed the corpus. Query embedding on-device can use a local model, a cached vector, or one tiny server round-trip for the query alone.
+
+## Tiering without batch trees
+
+RAPTOR-style hierarchies buy small-context windows a lot — but a recursive tree can't be rebuilt on every insert. `askfaro-embedded-search` keeps the index flat and gets the benefit through **node kinds**:
+
+- `leaf` — the object itself (default).
+- `summary` — an optional one-line abstract of an object, indexed alongside it. O(1) per object, generated on your write path or a background pass.
+- `cluster` — optional theme summaries produced by a periodic background sweep over existing embeddings.
+
+All kinds live in the same pool and are retrieved by the same top-k ("collapsed" retrieval); by default multiple hits on one object collapse into its best row, with `matched_node_kinds` telling you which handles matched:
+
+```python
+await index.upsert_many([
+    IndexDoc(object_type="note", object_id="n9", title="Meeting notes", body=transcript),
+    IndexDoc(object_type="note", object_id="n9", node_kind="summary",
+             title="Summary: hiring sync", body="decided to open two roles"),
+])
+```
+
+## Multiple embedding spaces (dual-model / on-device)
+
+A row can carry vectors from more than one model — typically a high-quality **server** model and a smaller **on-device** model — each as its own independently-queried space. Vectors from different models are never compared, so there's no mismatch.
+
+```python
+index = SearchIndex(
+    backend,                                    # configured with both spaces
+    embedders={"server": openai, "local": on_device_minilm},
+    default_space="server",
+)
+
+# A note participates in both spaces (default); an email is server-only:
+await index.upsert_many([
+    IndexDoc(object_type="note",  object_id="n1", title="...", body="..."),
+    IndexDoc(object_type="email", object_id="e1", title="...", body="...",
+             embed_spaces=["server"]),         # no on-device vector
+])
+
+# On the server, query the server space; on-device, query the local space:
+await index.search("quarterly plan", space="server")   # web/server
+await index.search("quarterly plan", space="local")    # device shard
+```
+
+The device shard carries only the on-device space (smaller — no server vectors):
+
+```python
+shard = await export_shard(server_backend, "user-42.db",
+                           partition="user-42", spaces=("local",))
+```
+
+Because a server-only object (the email above) has no local vector, on-device it's **keyword-searchable but not semantic** — its text still syncs, so FTS finds it offline; semantic search on it requires querying the server. This falls out of the per-space null-vector handling, no special-casing. Configure backends with matching spaces: `PostgresBackend(..., spaces={"server": 1536, "local": 384})`, `SQLiteBackend(..., spaces=("local",))`.
+
+## Heterogeneous objects
+
+Per-type behavior lives in code, not schema, via a tiny registry:
+
+```python
+from askfaro_embedded_search import register, docs_for
+
+@register("contact")
+def index_contact(c) -> IndexDoc:
+    return IndexDoc(object_type="contact", object_id=str(c.id),
+                    title=c.name, body=f"{c.role} at {c.company}",
+                    payload={"avatar": c.avatar_url})
+
+await index.upsert_many(docs_for("contact", some_contact))
+```
+
+`payload` is carried into results (and shards), so result lists render without joining back to your application database.
+
+## Design notes
+
+- **Embedding failure is non-fatal, but never silent.** A row written without a vector still serves lexical queries and gains semantic retrieval after a backfill — availability over completeness. Every embedding failure (at index time *and* query time) is logged at `WARNING` on the `askfaro_embedded_search` logger with the exception, so a misconfigured embedder (bad key, wrong endpoint) surfaces instead of silently degrading search to keyword-only. Wire that logger up in your app to catch it.
+- **Exact semantic scan on SQLite.** Per-user shards are small (tens of thousands of rows); an exact cosine scan (numpy-accelerated when present) costs no index maintenance and returns exact results. ANN acceleration (e.g. sqlite-vec) can be added without changing the file format.
+- **Diversity, not padding.** An optional per-group cap (`diversity_key`) drops near-duplicate siblings instead of deferring them.
+- **Stemming parity.** Postgres uses the `english` text-search config; SQLite FTS5 uses the `porter` tokenizer. Both stem morphological variants ("groceries" → "grocery") so the server and a device shard rank the same corpus identically.
+- **No framework.** Plain SQL on both backends, zero required dependencies in the core.
+
+## Requirements
+
+- **Python 3.11+.** The core (SQLite backend) needs only the standard library — SQLite's FTS5 and JSON1 extensions ship with CPython's bundled SQLite.
+- **Postgres backend** (`[postgres]` extra) needs a database with the **pgvector** extension available. `create_schema()` runs `CREATE EXTENSION IF NOT EXISTS vector` and the rest of the DDL in one idempotent call (or transcribe it into your own migrations). It is safe to re-run; it also adds new columns in place when you upgrade the library.
+
+## Troubleshooting
+
+The library raises clear, actionable errors for the common setup mistakes (all subclasses of `FaroSearchError`):
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `MissingDependencyError: PostgresBackend requires the 'postgres' extra` | Installed the core only | `pip install "askfaro-embedded-search[postgres]"` |
+| `MissingDependencyError: OpenAICompatibleEmbedder requires the 'http' extra` | Missing httpx | `pip install "askfaro-embedded-search[http]"` |
+| `ConfigurationError: pgvector extension isn't available…` | Postgres server has no `vector` extension | Enable/install [pgvector](https://github.com/pgvector/pgvector), or use `SQLiteBackend` (no extension needed) |
+| `ConfigurationError: Embedding for space 'x' has dimension N, but the index is configured for M` | Embedder output dim ≠ the space's column dim | Match the dims: configure the backend with `spaces={'x': N}` (then re-create the schema), or use an embedder that emits M-dim vectors |
+| `ConfigurationError: …embedders for space(s) [...] that the backend doesn't define` | `SearchIndex` has an embedder for a space the backend wasn't configured with | Give the backend matching spaces: `PostgresBackend(..., spaces={...})` / `SQLiteBackend(..., spaces=(...))` |
+| `ConfigurationError: This SQLite build lacks the FTS5 extension` | Rare CPython build without FTS5 | Use a Python whose bundled SQLite has FTS5 |
+| Postgres error: relation `…_index` does not exist | Forgot to set up the schema | Call `await backend.create_schema()` once (or run the equivalent migration) before indexing |
+
+Other expected behaviours (not errors): searching a space with **no embedder** returns lexical-only results, and a doc indexed without a vector still serves keyword search — both by design (availability over completeness).
+
+## Installation
+
+```bash
+pip install askfaro-embedded-search                # core (SQLite backend, stdlib only)
+pip install askfaro-embedded-search[postgres]      # + Postgres/pgvector backend
+pip install askfaro-embedded-search[http]          # + OpenAI-compatible embedder
+pip install askfaro-embedded-search[numpy]         # + fast cosine on SQLite
+```
+
+## License
+
+MIT

askfaro_embedded_search-0.5.0/README.md

@@ -0,0 +1,202 @@
+# askfaro-embedded-search
+
+**Incremental hybrid retrieval with identical semantics on the server and on-device.**
+
+`askfaro-embedded-search` is a small, dependency-light library for searching a *continuously growing* collection of heterogeneous objects — notes, contacts, emails, tasks, tools, anything — without ever rebuilding an index. It is built for the pattern modern assistant apps need:
+
+> **Index server-side, retrieve on-device.** Embeddings are computed once, centrally. Each user's slice of the index is replicated into a local SQLite shard, and the device answers queries locally — fast, offline, and private — with *exactly* the same ranking the server would produce.
+
+Built and dogfooded by Faro; also the embedded retrieval engine of Scope.
+
+## Why another retrieval library?
+
+Most RAG tooling assumes a batch world: ingest a corpus, build a tree/graph/index, query it. Real applications have **object-level CRUD** — a contact edited, a task created, an email deleted — and small/on-device context windows that punish irrelevant results. `askfaro-embedded-search` makes two opinionated choices:
+
+1. **Incremental-first.** One upsert per object write. Postgres (pgvector HNSW + tsvector GIN) and SQLite (FTS5 + vector blobs) both take per-row inserts natively, so there is no "rebuild" anywhere in the design. Hierarchical enrichment (summary and cluster nodes) lives in the *same flat pool* as extra rows — never on the write path.
+2. **Rank-based fusion.** Lexical and semantic retrievers run in parallel and are fused with Reciprocal Rank Fusion (RRF, K=60). Because RRF consumes *ranks*, not raw scores, the incomparable scoring scales of `ts_rank_cd` (Postgres) and `bm25()` (SQLite FTS5) don't matter: the same corpus and query rank identically on both backends. That is what makes "same engine, server and device" honest rather than aspirational.
+
+## Quick start
+
+```python
+from askfaro_embedded_search import IndexDoc, SearchIndex, OpenAICompatibleEmbedder
+from askfaro_embedded_search.backends.sqlite import SQLiteBackend
+
+index = SearchIndex(
+    SQLiteBackend("search.db"),
+    OpenAICompatibleEmbedder("https://api.openai.com/v1", api_key, "text-embedding-3-small"),
+)
+
+await index.upsert(IndexDoc(
+    object_type="note", object_id="n1",
+    title="Quantum entanglement notes",
+    body="spooky action at a distance",
+    partition="user-42",                      # isolation + shard key
+))
+
+results = await index.search("entanglement", partition="user-42", k=5)
+# SearchResult(object_id='n1', match_type='hybrid', score=..., ...)
+```
+
+Deleting and updating are first-class — deletes are tombstones so they propagate through shard sync:
+
+```python
+await index.delete("note", "n1")
+```
+
+### Filtering
+
+Filter by partition (the isolation/shard key), object type, node kind, or arbitrary structured attributes:
+
+```python
+await index.upsert(IndexDoc(
+    object_type="tool", object_id="stripe/refund",
+    title="Refund a charge", body="...",
+    attrs={"category": "finance", "status": "active"},   # structured filter fields
+))
+
+# Only finance tools:
+await index.search("refund a customer", object_types=["tool"], attrs={"category": "finance"})
+```
+
+`attrs` is matched by containment (all given keys must equal), backed by a JSONB GIN index on Postgres and `json_extract` on SQLite — so it filters identically on server and device.
+
+#### Partitions
+
+`partition` is the per-tenant isolation key (and the unit of on-device shard replication). It is **optional and defaults to `None`**, and a query's `partition` filter is an **exact match**: `search(..., partition="user-42")` returns only rows upserted with `partition="user-42"`. Two consequences worth knowing up front, because both fail quietly rather than with an error:
+
+- A row indexed **without** a partition (`None`) will **not** appear in a query that filters by a partition, and vice-versa. Pick one convention per corpus — either always set a partition, or never — and don't mix.
+- `search()` with no `partition` does **not** scope to a tenant; it searches across every partition. Pass the partition on every query in a multi-tenant index.
+
+### Server-side: Postgres
+
+```python
+from askfaro_embedded_search.backends.postgres import PostgresBackend  # pip install askfaro-embedded-search[postgres]
+
+backend = PostgresBackend("postgresql+asyncpg://...", table="faro_embedded_search_index", dim=1536)
+await backend.create_schema()     # idempotent; or transcribe the DDL into your migrations
+index = SearchIndex(backend, embedder)
+```
+
+### On-device: replicate a shard, query locally
+
+```python
+from askfaro_embedded_search import export_shard, replicate
+
+# Full export of one user's partition into a SQLite file:
+shard = await export_shard(server_backend, "user-42.db", partition="user-42")
+
+# Later, incremental delta sync (inserts, updates, AND deletes):
+cursor = await replicate(server_backend, shard, partition="user-42", cursor=cursor)
+```
+
+The shard is a plain SQLite file whose schema **is** the interchange format — any runtime that can read SQLite (a future Swift/Kotlin reader, for instance) can retrieve against it. Embeddings travel with the shard; the device never needs to re-embed the corpus. Query embedding on-device can use a local model, a cached vector, or one tiny server round-trip for the query alone.
+
+## Tiering without batch trees
+
+RAPTOR-style hierarchies buy small-context windows a lot — but a recursive tree can't be rebuilt on every insert. `askfaro-embedded-search` keeps the index flat and gets the benefit through **node kinds**:
+
+- `leaf` — the object itself (default).
+- `summary` — an optional one-line abstract of an object, indexed alongside it. O(1) per object, generated on your write path or a background pass.
+- `cluster` — optional theme summaries produced by a periodic background sweep over existing embeddings.
+
+All kinds live in the same pool and are retrieved by the same top-k ("collapsed" retrieval); by default multiple hits on one object collapse into its best row, with `matched_node_kinds` telling you which handles matched:
+
+```python
+await index.upsert_many([
+    IndexDoc(object_type="note", object_id="n9", title="Meeting notes", body=transcript),
+    IndexDoc(object_type="note", object_id="n9", node_kind="summary",
+             title="Summary: hiring sync", body="decided to open two roles"),
+])
+```
+
+## Multiple embedding spaces (dual-model / on-device)
+
+A row can carry vectors from more than one model — typically a high-quality **server** model and a smaller **on-device** model — each as its own independently-queried space. Vectors from different models are never compared, so there's no mismatch.
+
+```python
+index = SearchIndex(
+    backend,                                    # configured with both spaces
+    embedders={"server": openai, "local": on_device_minilm},
+    default_space="server",
+)
+
+# A note participates in both spaces (default); an email is server-only:
+await index.upsert_many([
+    IndexDoc(object_type="note",  object_id="n1", title="...", body="..."),
+    IndexDoc(object_type="email", object_id="e1", title="...", body="...",
+             embed_spaces=["server"]),         # no on-device vector
+])
+
+# On the server, query the server space; on-device, query the local space:
+await index.search("quarterly plan", space="server")   # web/server
+await index.search("quarterly plan", space="local")    # device shard
+```
+
+The device shard carries only the on-device space (smaller — no server vectors):
+
+```python
+shard = await export_shard(server_backend, "user-42.db",
+                           partition="user-42", spaces=("local",))
+```
+
+Because a server-only object (the email above) has no local vector, on-device it's **keyword-searchable but not semantic** — its text still syncs, so FTS finds it offline; semantic search on it requires querying the server. This falls out of the per-space null-vector handling, no special-casing. Configure backends with matching spaces: `PostgresBackend(..., spaces={"server": 1536, "local": 384})`, `SQLiteBackend(..., spaces=("local",))`.
+
+## Heterogeneous objects
+
+Per-type behavior lives in code, not schema, via a tiny registry:
+
+```python
+from askfaro_embedded_search import register, docs_for
+
+@register("contact")
+def index_contact(c) -> IndexDoc:
+    return IndexDoc(object_type="contact", object_id=str(c.id),
+                    title=c.name, body=f"{c.role} at {c.company}",
+                    payload={"avatar": c.avatar_url})
+
+await index.upsert_many(docs_for("contact", some_contact))
+```
+
+`payload` is carried into results (and shards), so result lists render without joining back to your application database.
+
+## Design notes
+
+- **Embedding failure is non-fatal, but never silent.** A row written without a vector still serves lexical queries and gains semantic retrieval after a backfill — availability over completeness. Every embedding failure (at index time *and* query time) is logged at `WARNING` on the `askfaro_embedded_search` logger with the exception, so a misconfigured embedder (bad key, wrong endpoint) surfaces instead of silently degrading search to keyword-only. Wire that logger up in your app to catch it.
+- **Exact semantic scan on SQLite.** Per-user shards are small (tens of thousands of rows); an exact cosine scan (numpy-accelerated when present) costs no index maintenance and returns exact results. ANN acceleration (e.g. sqlite-vec) can be added without changing the file format.
+- **Diversity, not padding.** An optional per-group cap (`diversity_key`) drops near-duplicate siblings instead of deferring them.
+- **Stemming parity.** Postgres uses the `english` text-search config; SQLite FTS5 uses the `porter` tokenizer. Both stem morphological variants ("groceries" → "grocery") so the server and a device shard rank the same corpus identically.
+- **No framework.** Plain SQL on both backends, zero required dependencies in the core.
+
+## Requirements
+
+- **Python 3.11+.** The core (SQLite backend) needs only the standard library — SQLite's FTS5 and JSON1 extensions ship with CPython's bundled SQLite.
+- **Postgres backend** (`[postgres]` extra) needs a database with the **pgvector** extension available. `create_schema()` runs `CREATE EXTENSION IF NOT EXISTS vector` and the rest of the DDL in one idempotent call (or transcribe it into your own migrations). It is safe to re-run; it also adds new columns in place when you upgrade the library.
+
+## Troubleshooting
+
+The library raises clear, actionable errors for the common setup mistakes (all subclasses of `FaroSearchError`):
+
+| Symptom | Cause | Fix |
+|---|---|---|
+| `MissingDependencyError: PostgresBackend requires the 'postgres' extra` | Installed the core only | `pip install "askfaro-embedded-search[postgres]"` |
+| `MissingDependencyError: OpenAICompatibleEmbedder requires the 'http' extra` | Missing httpx | `pip install "askfaro-embedded-search[http]"` |
+| `ConfigurationError: pgvector extension isn't available…` | Postgres server has no `vector` extension | Enable/install [pgvector](https://github.com/pgvector/pgvector), or use `SQLiteBackend` (no extension needed) |
+| `ConfigurationError: Embedding for space 'x' has dimension N, but the index is configured for M` | Embedder output dim ≠ the space's column dim | Match the dims: configure the backend with `spaces={'x': N}` (then re-create the schema), or use an embedder that emits M-dim vectors |
+| `ConfigurationError: …embedders for space(s) [...] that the backend doesn't define` | `SearchIndex` has an embedder for a space the backend wasn't configured with | Give the backend matching spaces: `PostgresBackend(..., spaces={...})` / `SQLiteBackend(..., spaces=(...))` |
+| `ConfigurationError: This SQLite build lacks the FTS5 extension` | Rare CPython build without FTS5 | Use a Python whose bundled SQLite has FTS5 |
+| Postgres error: relation `…_index` does not exist | Forgot to set up the schema | Call `await backend.create_schema()` once (or run the equivalent migration) before indexing |
+
+Other expected behaviours (not errors): searching a space with **no embedder** returns lexical-only results, and a doc indexed without a vector still serves keyword search — both by design (availability over completeness).
+
+## Installation
+
+```bash
+pip install askfaro-embedded-search                # core (SQLite backend, stdlib only)
+pip install askfaro-embedded-search[postgres]      # + Postgres/pgvector backend
+pip install askfaro-embedded-search[http]          # + OpenAI-compatible embedder
+pip install askfaro-embedded-search[numpy]         # + fast cosine on SQLite
+```
+
+## License
+
+MIT