relvector 0.1.0__tar.gz → 0.2.0__tar.gz

{relvector-0.1.0 → relvector-0.2.0}/.gitignore

@@ -38,3 +38,6 @@ audit*.jsonl
 *.rvindex-*
 relvector_index.sqlite*
 examples/relvector/_output/
+
+# Local tool state
+.claude/

relvector-0.2.0/CHANGELOG.md

@@ -0,0 +1,60 @@
+# Changelog
+
+All notable changes are listed here. The project follows semantic versioning once released.
+
+## 0.2.0 (2026-10-25)
+
+Added (all opt-in; v0.1.0 behaviour and APIs are unchanged):
+
+* **Incremental sync.** `configure(change_tracking={table: column, ...}, soft_delete=...)` and
+  `rv.sync(batch_size, lookback=..., full_fallback=False) -> SyncStats`.
+  * Changed rows are read per mapped table in bounded `(tracking column, pk)` keyset batches,
+    starting at `watermark - lookback`.
+  * Affected root documents are rebuilt, including the previous parents of reassigned
+    children, which are found through the index's provenance.
+  * Per-table watermarks are committed atomically with each batch.
+  * `index()` records watermarks captured before its scan.
+  * Missing tracking columns or stale watermarks fail clearly (`MappingError`,
+    `SyncUnavailableError`).
+* **Optional FAISS backend.** `vector_store="faiss"` / `FaissVectorStore` (`pip install
+  "relvector[faiss]"`).
+  * A CPU `IndexIVFFlat` derived from the SQLite store, with ANN candidates re-scored
+    exactly.
+  * Exact fallback for metadata filters, a requested entity in a multi-entity index, or a
+    small index.
+  * Crash-safe through a dirty marker, atomic saves and automatic rebuild from SQLite.
+* **Store contract.**
+  * `Checkpoint.state`: durable, typed, and committed with each claim.
+  * `VectorStore.delete_records`: owner-checked.
+  * `VectorStore.referencing`: an indexed reverse lookup from related-row primary keys to the
+    documents that reference them.
+  * A post-commit `_after_commit` hook for derived indexes.
+
+Compatibility: index files written by 0.1.0 open in place. The new checkpoint `state` column
+and the reference lookup table are added and backfilled on first open. This migration is
+one-way. After it, 0.1.0 can still open and search the file, but indexing with 0.1.0 fails
+with `OperationalError` (rv_checkpoints has 9 columns). There is no silent corruption. To
+go back, rebuild the index with 0.1.0 at a new path. FAISS files are derived and can be
+deleted at any time.
+
+## 0.1.0 (2026-10-25)
+
+First public release:
+
+* Read-only connections for SQLite (`mode=ro`) and PostgreSQL (read-only `REPEATABLE READ`
+  transactions).
+* Schema inspection: tables, columns, primary keys and foreign keys.
+* Entity mappings with one-hop related tables, explicit `Include(via=...)` joins,
+  sensitive-field blocking, exclusions and deterministic templates.
+* Bounded keyset extraction with per-relation `IN` queries and `ROW_NUMBER()` caps that report
+  truncation.
+* Embedding providers: local sentence-transformers and dependency-free hashing.
+* A persistent SQLite vector store with exact cosine search and metadata filters.
+* Idempotent re-indexing with content hashes, pruning of deleted rows, atomic per-batch
+  checkpoints, typed checkpoint keys (date, datetime, UUID, Decimal), resume, and single-writer
+  ownership (`RunSupersededError`).
+* Optional SQLGuard Ninja validation of extraction queries.
+* Streaming exact search: chunked cursor reads, a bounded top-k heap, SQL-side filtering and
+  deterministic tie-breaking. Search memory no longer grows with index size
+  (`LocalVectorStore(search_chunk_size=...)`).
+* PostgreSQL validation tests on a disposable database.

{relvector-0.1.0 → relvector-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: relvector
-Version: 0.1.0
+Version: 0.2.0
 Summary: Schema-aware semantic indexes over existing relational databases, with source references back to the rows.
 Project-URL: Homepage, https://github.com/Pratik653/RelVector
 Project-URL: Issues, https://github.com/Pratik653/RelVector/issues
@@ -24,11 +24,14 @@ Requires-Dist: numpy>=1.26
 Requires-Dist: sqlalchemy<3.0,>=2.0
 Provides-Extra: dev
 Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: faiss-cpu>=1.8; extra == 'dev'
 Requires-Dist: mypy>=1.10; extra == 'dev'
 Requires-Dist: psycopg[binary]>=3.1; extra == 'dev'
 Requires-Dist: pytest>=8.0; extra == 'dev'
 Requires-Dist: ruff>=0.6; extra == 'dev'
 Requires-Dist: sqlguard-ninja<0.8,>=0.7; extra == 'dev'
+Provides-Extra: faiss
+Requires-Dist: faiss-cpu>=1.8; extra == 'faiss'
 Provides-Extra: local
 Requires-Dist: sentence-transformers>=2.3; extra == 'local'
 Provides-Extra: postgres
@@ -50,7 +53,7 @@ points back to the exact source rows.
 RelVector does **not** convert a database into a vector database. The database stays the
 authoritative source and is only ever read.
 
-> Status: `0.1.0`, alpha. Tested with SQLite and PostgreSQL 16. APIs may change before 1.0.
+> Status: `0.2.0`, alpha. Tested with SQLite and PostgreSQL 16. APIs may change before 1.0.
 
 ## Install
 
@@ -60,6 +63,7 @@ Python 3.11+:
 pip install relvector                # core: SQLAlchemy + NumPy
 pip install "relvector[local]"       # + sentence-transformers for local embeddings (pulls in PyTorch)
 pip install "relvector[postgres]"    # + psycopg 3; use postgresql+psycopg://user@host/db URLs
+pip install "relvector[faiss]"       # + faiss-cpu for the optional approximate (ANN) backend
 pip install "relvector[sqlguard]"    # + optional SQLGuard Ninja query validation
 ```
 
@@ -99,6 +103,81 @@ The record for order 1002 reads:
 Runnable example: [examples/relvector/orders_shipments.py](examples/relvector/orders_shipments.py)
 (`--provider hashing` runs without any model).
 
+## Incremental sync (opt-in)
+
+If every mapped table has a column that your application reliably sets on insert *and*
+update, `sync()` re-indexes only what changed:
+
+```python
+rv.configure(
+    root_table="orders", include_tables=["shipments"],
+    text_fields=["orders.status", "shipments.delay_reason"],
+    change_tracking={"orders": "updated_at", "shipments": "updated_at"},  # every mapped table
+    soft_delete="deleted_at",            # optional: root rows with a non-NULL value are removed
+)
+rv.index()                               # a full run first; records the starting watermarks
+stats = rv.sync(lookback=timedelta(minutes=5))   # later: only changed rows, in bounded batches
+```
+
+What `sync()` guarantees:
+
+* Rows are read in `(tracking column, pk)` order from `watermark - lookback`, using `>=` so
+  equal timestamps are never skipped, up to the maximum value read when that table's scan
+  starts. The default lookback is 5 minutes for date/time columns and 0 for numeric ones.
+* Which documents are rebuilt:
+  * A changed root row rebuilds itself.
+  * A changed child rebuilds its current parent and its previous parent. The previous parent
+    comes from the index's own provenance, so a reassigned shipment leaves neither order stale.
+  * A changed many-to-one parent rebuilds every root that references it.
+  * Roots whose row is gone or soft-deleted are removed.
+* Rebuilt documents go through the same deterministic mapping and hashing as `index()`, and
+  only changed content is re-embedded.
+* Each batch's watermark is committed in the same transaction as that batch's index changes.
+  An interrupted sync can just be run again.
+
+What it does **not** guarantee:
+
+* **Numeric tracking columns get no lookback by default.** Values allocated from a sequence
+  can commit out of order (a transaction holding 41 may commit after 42 was read), so set
+  `lookback` to a number that covers the values that can be in flight at once.
+* It misses changes whose tracking value is older than `watermark - lookback`. Examples are
+  transactions that commit later than the lookback, clock skew between writers, backfills
+  with old timestamps, and code paths that don't bump the column. Rows with a NULL tracking
+  value are never seen.
+* Hard deletes are invisible to timestamps. Run a full `index()` periodically to reconcile;
+  it prunes them. A soft delete must also bump the root's tracking column.
+* Many-to-one changes are missed in three cases: a parent's primary key changes, a parent is
+  hard-deleted, or a root's FK changes without its tracking column being bumped.
+* Rows sitting inside the lookback window are re-read on every sync. They are skipped by
+  content hash, so this is cheap, but a bulk load that shares one timestamp is re-read until
+  newer changes move the watermark past it.
+
+If a mapped table has no tracking column, `sync()` raises `MappingError`. If watermarks are
+missing or stale (no full index yet, or the mapping or tracking changed), it raises
+`SyncUnavailableError`. `sync(full_fallback=True)` runs `index()` instead. This is a
+timestamp-based sync, not change-data-capture.
+
+## Optional FAISS backend (approximate search)
+
+```python
+rv = RelVector(url, vector_store="faiss", index_path="shop.rvindex")   # pip install "relvector[faiss]"
+# or tune it: RelVector(url, vector_store=FaissVectorStore("shop.rvindex", ann_min_vectors=5000))
+```
+
+* SQLite stays the source of truth. A FAISS `IndexIVFFlat` file (inner product on normalised
+  vectors) sits next to it and is rebuilt from SQLite automatically if it is missing,
+  unreadable, flagged dirty after a crash, or inconsistent with SQLite.
+* Search is **approximate**: FAISS proposes `top_k × oversample` candidates, and these are
+  re-scored exactly. Returned scores are therefore true cosine similarities, but a true
+  nearest neighbour can be missed (recall below 1). `store.last_search_mode` reports `"ann"`
+  or `"exact"`.
+* The FAISS backend falls back to exact search in three cases: metadata `filters` are given,
+  the index holds several entities and one is requested, or there are fewer than
+  `ann_min_vectors` (default 5000) vectors. Filtered results are therefore never truncated by
+  the ANN step.
+* Upserts, `sync()`, deletes and pruning keep the FAISS index current. It is saved atomically
+  at the end of each run and on close. Use one writer process per index.
+
 ## How it works
 
 | Stage | Behaviour |
@@ -153,8 +232,8 @@ a record is close to the query in embedding space. Check the source rows.
   isolation in the index.
 * Relationships: one hop from the root table, single-column joins. No self-references,
   multi-hop paths or composite foreign keys yet. Composite root primary keys are supported.
-* Re-indexing is a full re-scan. There is no change-data-capture, and no triggers or timestamps
-  are used. Edits to related rows (child shipments, a parent customer) are picked up on the
+* `index()` is a full re-scan. `sync()` is timestamp-based and has the limits listed under
+  [Incremental sync](#incremental-sync-opt-in). Neither is change-data-capture. Edits to related rows (child shipments, a parent customer) are picked up on the
   next completed run. Rows changed during a run may be read at different moments, and the next
   run corrects them. See
   [how source changes reach documents](docs/relvector/architecture.md#how-source-changes-reach-semantic-documents).
@@ -167,8 +246,9 @@ a record is close to the query in embedding space. Check the source rows.
   fail with a clear error.
 * `LocalVectorStore` search is exact and streamed: memory is bounded by
   `search_chunk_size` (default 1024 rows) and `top_k`, but every query scans all vectors that
-  pass the filters. Latency grows linearly with index size, and there is no approximate (ANN)
-  index. See [search performance](docs/relvector/architecture.md#search-performance).
+  pass the filters, so latency grows linearly with index size. The optional FAISS backend
+  is sub-linear but approximate, and filtered queries on it are still exact and linear. See
+  [search performance](docs/relvector/architecture.md#search-performance).
 * The hashing provider is lexical, not semantic.
 * SQLite and PostgreSQL only. Other databases are not supported.
 

{relvector-0.1.0 → relvector-0.2.0}/README.md

@@ -11,7 +11,7 @@ points back to the exact source rows.
 RelVector does **not** convert a database into a vector database. The database stays the
 authoritative source and is only ever read.
 
-> Status: `0.1.0`, alpha. Tested with SQLite and PostgreSQL 16. APIs may change before 1.0.
+> Status: `0.2.0`, alpha. Tested with SQLite and PostgreSQL 16. APIs may change before 1.0.
 
 ## Install
 
@@ -21,6 +21,7 @@ Python 3.11+:
 pip install relvector                # core: SQLAlchemy + NumPy
 pip install "relvector[local]"       # + sentence-transformers for local embeddings (pulls in PyTorch)
 pip install "relvector[postgres]"    # + psycopg 3; use postgresql+psycopg://user@host/db URLs
+pip install "relvector[faiss]"       # + faiss-cpu for the optional approximate (ANN) backend
 pip install "relvector[sqlguard]"    # + optional SQLGuard Ninja query validation
 ```
 
@@ -60,6 +61,81 @@ The record for order 1002 reads:
 Runnable example: [examples/relvector/orders_shipments.py](examples/relvector/orders_shipments.py)
 (`--provider hashing` runs without any model).
 
+## Incremental sync (opt-in)
+
+If every mapped table has a column that your application reliably sets on insert *and*
+update, `sync()` re-indexes only what changed:
+
+```python
+rv.configure(
+    root_table="orders", include_tables=["shipments"],
+    text_fields=["orders.status", "shipments.delay_reason"],
+    change_tracking={"orders": "updated_at", "shipments": "updated_at"},  # every mapped table
+    soft_delete="deleted_at",            # optional: root rows with a non-NULL value are removed
+)
+rv.index()                               # a full run first; records the starting watermarks
+stats = rv.sync(lookback=timedelta(minutes=5))   # later: only changed rows, in bounded batches
+```
+
+What `sync()` guarantees:
+
+* Rows are read in `(tracking column, pk)` order from `watermark - lookback`, using `>=` so
+  equal timestamps are never skipped, up to the maximum value read when that table's scan
+  starts. The default lookback is 5 minutes for date/time columns and 0 for numeric ones.
+* Which documents are rebuilt:
+  * A changed root row rebuilds itself.
+  * A changed child rebuilds its current parent and its previous parent. The previous parent
+    comes from the index's own provenance, so a reassigned shipment leaves neither order stale.
+  * A changed many-to-one parent rebuilds every root that references it.
+  * Roots whose row is gone or soft-deleted are removed.
+* Rebuilt documents go through the same deterministic mapping and hashing as `index()`, and
+  only changed content is re-embedded.
+* Each batch's watermark is committed in the same transaction as that batch's index changes.
+  An interrupted sync can just be run again.
+
+What it does **not** guarantee:
+
+* **Numeric tracking columns get no lookback by default.** Values allocated from a sequence
+  can commit out of order (a transaction holding 41 may commit after 42 was read), so set
+  `lookback` to a number that covers the values that can be in flight at once.
+* It misses changes whose tracking value is older than `watermark - lookback`. Examples are
+  transactions that commit later than the lookback, clock skew between writers, backfills
+  with old timestamps, and code paths that don't bump the column. Rows with a NULL tracking
+  value are never seen.
+* Hard deletes are invisible to timestamps. Run a full `index()` periodically to reconcile;
+  it prunes them. A soft delete must also bump the root's tracking column.
+* Many-to-one changes are missed in three cases: a parent's primary key changes, a parent is
+  hard-deleted, or a root's FK changes without its tracking column being bumped.
+* Rows sitting inside the lookback window are re-read on every sync. They are skipped by
+  content hash, so this is cheap, but a bulk load that shares one timestamp is re-read until
+  newer changes move the watermark past it.
+
+If a mapped table has no tracking column, `sync()` raises `MappingError`. If watermarks are
+missing or stale (no full index yet, or the mapping or tracking changed), it raises
+`SyncUnavailableError`. `sync(full_fallback=True)` runs `index()` instead. This is a
+timestamp-based sync, not change-data-capture.
+
+## Optional FAISS backend (approximate search)
+
+```python
+rv = RelVector(url, vector_store="faiss", index_path="shop.rvindex")   # pip install "relvector[faiss]"
+# or tune it: RelVector(url, vector_store=FaissVectorStore("shop.rvindex", ann_min_vectors=5000))
+```
+
+* SQLite stays the source of truth. A FAISS `IndexIVFFlat` file (inner product on normalised
+  vectors) sits next to it and is rebuilt from SQLite automatically if it is missing,
+  unreadable, flagged dirty after a crash, or inconsistent with SQLite.
+* Search is **approximate**: FAISS proposes `top_k × oversample` candidates, and these are
+  re-scored exactly. Returned scores are therefore true cosine similarities, but a true
+  nearest neighbour can be missed (recall below 1). `store.last_search_mode` reports `"ann"`
+  or `"exact"`.
+* The FAISS backend falls back to exact search in three cases: metadata `filters` are given,
+  the index holds several entities and one is requested, or there are fewer than
+  `ann_min_vectors` (default 5000) vectors. Filtered results are therefore never truncated by
+  the ANN step.
+* Upserts, `sync()`, deletes and pruning keep the FAISS index current. It is saved atomically
+  at the end of each run and on close. Use one writer process per index.
+
 ## How it works
 
 | Stage | Behaviour |
@@ -114,8 +190,8 @@ a record is close to the query in embedding space. Check the source rows.
   isolation in the index.
 * Relationships: one hop from the root table, single-column joins. No self-references,
   multi-hop paths or composite foreign keys yet. Composite root primary keys are supported.
-* Re-indexing is a full re-scan. There is no change-data-capture, and no triggers or timestamps
-  are used. Edits to related rows (child shipments, a parent customer) are picked up on the
+* `index()` is a full re-scan. `sync()` is timestamp-based and has the limits listed under
+  [Incremental sync](#incremental-sync-opt-in). Neither is change-data-capture. Edits to related rows (child shipments, a parent customer) are picked up on the
   next completed run. Rows changed during a run may be read at different moments, and the next
   run corrects them. See
   [how source changes reach documents](docs/relvector/architecture.md#how-source-changes-reach-semantic-documents).
@@ -128,8 +204,9 @@ a record is close to the query in embedding space. Check the source rows.
   fail with a clear error.
 * `LocalVectorStore` search is exact and streamed: memory is bounded by
   `search_chunk_size` (default 1024 rows) and `top_k`, but every query scans all vectors that
-  pass the filters. Latency grows linearly with index size, and there is no approximate (ANN)
-  index. See [search performance](docs/relvector/architecture.md#search-performance).
+  pass the filters, so latency grows linearly with index size. The optional FAISS backend
+  is sub-linear but approximate, and filtered queries on it are still exact and linear. See
+  [search performance](docs/relvector/architecture.md#search-performance).
 * The hashing provider is lexical, not semantic.
 * SQLite and PostgreSQL only. Other databases are not supported.
 

{relvector-0.1.0 → relvector-0.2.0}/docs/relvector/architecture.md

@@ -124,15 +124,74 @@ Remaining limitations:
 * **One long read.** A search holds a SQLite read snapshot for its duration. In WAL mode,
   writers are not blocked.
 
+## Store contract (v0.2)
+
+`VectorStore` (in `store.py`) is the interface every backend implements.
+
+* **Mutations.** `start_run`, `resume_run`, `write_batch` (upserts and touch-unchanged),
+  `delete_records` and `finish_run(prune=...)`. Each is owner-checked against the entity's
+  checkpoint and is atomic with the checkpoint update it carries.
+* **Checkpoint state.** `Checkpoint.state` holds typed, JSON-serialisable run and sync state,
+  such as per-table watermarks. It is committed in the same transaction as the claim that
+  writes it, so a watermark can never be durable before the index changes it describes.
+* **Reverse lookup.** `referencing(entity, table, pks)` maps related-row primary keys to the
+  documents whose stored references include them. It is backed by the `rv_refs` table,
+  maintained on every mutation and backfilled once for 0.1.0 files. Incremental sync uses it
+  to find the previous parent of a reassigned or deleted related row.
+* **Derived indexes.** `LocalVectorStore._after_commit(upserted, deleted)` runs after every
+  committed mutation, including prune. The FAISS backend uses it to stay in sync.
+* **Identity and scores.** `vector_id` is `f"{entity}:{json pk}"`. Vectors are L2-normalised
+  float32. `score` is cosine similarity, and exact ranking breaks ties by `vector_id`
+  ascending.
+
+## Measured performance (workload-specific, not guarantees)
+
+Measured on one Windows laptop (Python 3.11, NumPy 2.4, faiss-cpu 1.15.1), in-process, with
+SQLite files on local disk. Expect different numbers on other hardware and data.
+
+| Workload | Exact / full | v0.2 path | Notes |
+|---|---|---|---|
+| 20,000 orders + 29,994 shipments, 1% changed (150 order updates, 50 reassignments), `HashingEmbedding(256)`, batch 500 | full `index()` 4.39 s | `sync()` 0.367 s (about 12×) | 252 roots rebuilt, 250 embedded |
+| 50,000 clustered vectors, dim 128, 100 queries, top 10 | exact 369.8 ms/query | FAISS 3.86 ms/query (about 96×) | recall@10 1.000; nlist 894, nprobe 224, oversample 4 |
+| 50,000 random Gaussian vectors, same settings | exact 351.5 ms/query | FAISS 3.04 ms/query (about 116×) | recall@10 **0.724**; 1.000 with nprobe = nlist at 6.5 ms/query |
+
+Random high-dimensional data is close to the worst case for IVF recall. Tune `nprobe` for
+your data, and measure recall against exact search before relying on ANN results.
+
+## Optional FAISS backend (v0.2)
+
+`relvector/faiss_store.py` (`FaissVectorStore`) subclasses `LocalVectorStore`.
+* **Source of truth.** SQLite remains authoritative. `<index>.faiss` is a derived
+  `IndexIVFFlat` (inner product on normalised vectors). Each vector's FAISS id is its
+  `rv_records` rowid, which an upsert keeps.
+* **Search.** Candidates (`top_k × oversample`) are re-scored with the same float64 arithmetic
+  as exact search, so scores and tie order match; only recall is approximate.
+* **Exact fallback** applies when there are metadata filters, when one entity of a
+  multi-entity index is requested, or when the index is below `ann_min_vectors`.
+* **Crash safety.** A dirty marker is written before each SQLite commit, the change is applied
+  in memory after the commit, and the file is saved atomically at the end of each run and on
+  close.
+* **Rebuild.** On open the index is rebuilt from SQLite if the file is missing, dirty,
+  unreadable, or does not match SQLite (count, checksum, model or dimension). It is also
+  rebuilt when another connection has changed SQLite (`PRAGMA data_version`), and when it has
+  grown `retrain_growth`× past the size it was trained on.
+
+## Incremental sync (v0.2)
+
+`relvector/sync.py` (`Syncer`, reached through `RelVector.sync`). Per-table watermarks live in
+`Checkpoint.state` and are committed with each batch's index changes. The guarantees and
+limitations are listed in the README under "Incremental sync". Previous parents of
+reassigned or deleted children come from `VectorStore.referencing`, a reverse index over the
+references each record stores, maintained on every mutation.
+
 ## Roadmap (not implemented)
 
-1. **Next scalability step: sub-linear search.** Add an ANN backend (FAISS or pgvector)
-   behind `VectorStore`. Promote frequently filtered metadata fields to indexed columns so
-   filters seek instead of scanning.
-2. Multi-hop and composite-key relationships, and explicit child records as separate vectors.
-3. Incremental sync driven by `updated_at` columns or a deletion log across every mapped table.
-   CDC comes later.
-4. Parallel embedding workers behind the `Indexer` batch loop. The owner-token checkpoint
-   already gives a single committer.
+1. Metadata pre-filtering inside ANN search (per-entity or per-filter FAISS partitions), so
+   that filtered queries don't fall back to exact search.
+2. Multi-hop and composite-key relationships.
+3. A deletion feed or tombstone table for hard deletes, so that `sync()` doesn't need periodic
+   full reconciliation.
+4. Parallel embedding workers behind the batch loops; the owner-token checkpoint already
+   ensures a single committer.
 5. An opt-in remote embedding provider (`sends_data_externally = True`).
-6. A CLI (`relvector index config.yaml`) and a mapping config file format.
+6. A CLI and a mapping config file format.

{relvector-0.1.0 → relvector-0.2.0}/pyproject.toml

@@ -31,10 +31,12 @@ dependencies = [
 [project.optional-dependencies]
 # Local embedding model (pulls in PyTorch).
 local = ["sentence-transformers>=2.3"]
+# Optional approximate nearest-neighbour backend (CPU FAISS).
+faiss = ["faiss-cpu>=1.8"]
 postgres = ["psycopg[binary]>=3.1"]
 sqlguard = ["sqlguard-ninja>=0.7,<0.8"]
 dev = [
-    "relvector[postgres,sqlguard]",
+    "relvector[faiss,postgres,sqlguard]",
     "pytest>=8.0",
     "ruff>=0.6",
     "mypy>=1.10",
@@ -81,5 +83,5 @@ strict = true
 files = ["src/relvector"]
 
 [[tool.mypy.overrides]]
-module = ["sentence_transformers.*", "sqlguard_ninja.*", "pgserver.*"]
+module = ["sentence_transformers.*", "sqlguard_ninja.*", "pgserver.*", "faiss.*"]
 ignore_missing_imports = true

{relvector-0.1.0 → relvector-0.2.0}/src/relvector/__init__.py

@@ -17,14 +17,17 @@ from relvector.errors import (
     SchemaError,
     SensitiveFieldError,
 )
+from relvector.faiss_store import FaissVectorStore  # faiss itself is imported lazily
 from relvector.indexing import IndexStats
 from relvector.mapping import EntityMapping, Include, SemanticRecord
 from relvector.schema import Schema, inspect_schema
 from relvector.store import LocalVectorStore, SearchResult, VectorStore
+from relvector.sync import SyncStats, SyncUnavailableError
 
 __all__ = [
     "EmbeddingProvider",
     "EntityMapping",
+    "FaissVectorStore",
     "HashingEmbedding",
     "Include",
     "IndexMismatchError",
@@ -41,6 +44,8 @@ __all__ = [
     "SemanticRecord",
     "SensitiveFieldError",
     "SentenceTransformerEmbedding",
+    "SyncStats",
+    "SyncUnavailableError",
     "VectorStore",
     "__version__",
     "connect",

relvector-0.2.0/src/relvector/_version.py

@@ -0,0 +1 @@
+__version__ = "0.2.0"

{relvector-0.1.0 → relvector-0.2.0}/src/relvector/api.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import dataclasses
 import sqlite3
 from collections.abc import Callable, Mapping, Sequence
 from pathlib import Path
@@ -17,6 +18,7 @@ from relvector.indexing import Indexer, IndexStats
 from relvector.mapping import EntityMapping, Include, ResolvedMapping, resolve
 from relvector.schema import Schema, inspect_schema
 from relvector.store import LocalVectorStore, SearchResult, VectorStore
+from relvector.sync import Lookback, Syncer, SyncStats
 
 SUPPORTED_DIALECTS = ("sqlite", "postgresql")
 
@@ -88,9 +90,15 @@ class RelVector:
                 "service; pass allow_external_embeddings=True to opt in explicitly"
             )
         if isinstance(vector_store, str):
-            if vector_store != "local":
-                raise RelVectorError(f"unknown vector store {vector_store!r}; use 'local'")
-            self.store: VectorStore = LocalVectorStore(index_path)
+            if vector_store == "local":
+                self.store: VectorStore = LocalVectorStore(index_path)
+            elif vector_store == "faiss":  # optional extra: relvector[faiss]
+                from relvector.faiss_store import FaissVectorStore  # noqa: PLC0415
+
+                self.store = FaissVectorStore(index_path)
+            else:
+                raise RelVectorError(f"unknown vector store {vector_store!r}; use 'local' or "
+                                     "'faiss'")
         else:
             self.store = vector_store
         self.db_schema = schema
@@ -131,8 +139,23 @@ class RelVector:
         name: str | None = None,
         version: str = "1",
         max_children: int = 20,
+        change_tracking: Mapping[str, str] | None = None,
+        soft_delete: str | None = None,
     ) -> ResolvedMapping:
-        """Set the entity mapping (pass an :class:`EntityMapping` or keyword arguments)."""
+        """Set the entity mapping (pass an :class:`EntityMapping` or keyword arguments).
+
+        ``change_tracking`` (``{table: column}``, e.g. ``{"orders": "updated_at"}``) and
+        ``soft_delete`` (a root column, non-NULL = deleted) opt in to :meth:`sync`; with a
+        mapping object they override the mapping's own values. Neither changes documents.
+        """
+        if mapping is not None:
+            changes: dict[str, Any] = {}
+            if change_tracking is not None:
+                changes["change_tracking"] = dict(change_tracking)
+            if soft_delete is not None:
+                changes["soft_delete"] = soft_delete
+            if changes:
+                mapping = dataclasses.replace(mapping, **changes)
         if mapping is None:
             if root_table is None:
                 raise MappingError("root_table is required")
@@ -142,6 +165,7 @@ class RelVector:
                 exclude_fields=tuple(exclude_fields), allow_sensitive=tuple(allow_sensitive),
                 templates=dict(templates or {}), labels=dict(labels or {}), name=name,
                 version=version, max_children=max_children,
+                change_tracking=dict(change_tracking or {}), soft_delete=soft_delete,
             )
         self._mapping = resolve(mapping, self.inspect_schema())
         return self._mapping
@@ -168,6 +192,41 @@ class RelVector:
         return Indexer(self.extractor(), self.provider, self.store).run(
             batch_size, resume=resume, prune=prune, progress=progress)
 
+    def sync(
+        self,
+        batch_size: int = 500,
+        *,
+        lookback: Lookback = None,
+        full_fallback: bool = False,
+        progress: Callable[[SyncStats], None] | None = None,
+    ) -> SyncStats:
+        """Apply source changes since the last index/sync, using the change-tracking columns.
+
+        Requires ``change_tracking`` for the root and every included table and a completed
+        :meth:`index` first. ``lookback`` is the overlap re-scanned below each watermark
+        (default 5 minutes for date/time columns, 0 for numeric ones). Hard deletes are not
+        detected; see :mod:`relvector.sync` for exactly what is and is not guaranteed.
+
+        When sync cannot run safely (missing tracking, no prior index, mapping changed) it
+        raises, unless ``full_fallback=True``, which runs a full :meth:`index` instead and
+        returns stats with ``mode == "full"``.
+        """
+        syncer = Syncer(self.extractor(), self.provider, self.store)
+        if full_fallback:
+            problem = syncer.unavailable()
+            if problem is not None:
+                full = self.index(batch_size)
+                stats = SyncStats(run_id=full.run_id, mode="full", embedded=full.embedded,
+                                  unchanged=full.unchanged, deleted=full.deleted,
+                                  batches=full.batches, roots_rebuilt=full.rows,
+                                  truncated_records=full.truncated_records,
+                                  reason=str(problem), timings=dict(full.timings))
+                cp = self.store.checkpoint(self.mapping.entity)
+                if cp is not None and cp.state is not None:
+                    stats.watermarks = dict(cp.state.get("watermarks") or {})
+                return stats
+        return syncer.run(batch_size, lookback=lookback, progress=progress)
+
     def search(
         self,
         query: str,