beaver-db 2.2.0__tar.gz → 2.3.0__tar.gz

beaver_db-2.3.0/AGENTS.md

@@ -0,0 +1,143 @@
+# AGENTS.md — beaver
+
+You're an AI agent picking up **beaver** (`beaver-db` on PyPI). This file is the door.
+
+## What it is
+
+A **local-first, embedded, multi-modal database for Python**, built on a single
+SQLite file. One file, many modalities: dicts, lists, queues, logs, documents
+(FTS + fuzzy), vectors (LSH), graphs, blobs, locks, channels and sketches — all
+in the same `.db`, all through one facade.
+
+It is a **public library** (`syalia-srl/beaver`, MIT) with real downstream users,
+not an internal utility. Its biggest consumer is AInBox (`repos/ainbox`), where
+warden, magpie, superbot, peacock, help and pipelines all persist through it.
+
+Read [`design.md`](design.md) first — the living design document, with the vision
+and the guiding principles. [`STATUS.md`](STATUS.md) is the running inventory of
+what is real versus missing in the v2 cycle.
+
+## The shape of the codebase
+
+    beaver/core.py          the engine: AsyncBeaverDB + the BeaverDB sync facade,
+                            _create_all_tables, _check_version, every factory method
+    beaver/interfaces.py    IBeaver* protocols — the surface the SID consumers target
+    beaver/api.py           @expose / @local_only — how a method reaches CLI + server + client
+    beaver/queries.py       the q() filter DSL: Query.__getattr__ builds dotted paths → Filter
+    beaver/<modality>.py    dicts, lists, queues, logs, docs, vectors, graphs, blobs,
+                            locks, channels, sketches — one module each
+    beaver/migrate.py       1.x → 2.x migration
+    beaver/cli/             the typer CLI (entry point `beaver`)
+    docs/                   Quarto site: guide-*.md (users), dev-*.md (contributors)
+    issues/                 numbered issue/spec files, synced with GitHub
+    tests/{unit,integration,concurrency}
+
+**Architecture in one paragraph.** `AsyncBeaverDB` is the async core (aiosqlite,
+one connection, WAL). `BeaverDB` is the *synchronous facade*: it spins a reactor
+thread running the async loop and bridges every call to it through
+`BeaverBridge` (the Portal pattern). "Synchronous facade, asynchronous engine" —
+so write the real logic in the async class, and the sync surface follows. See
+`docs/dev-architecture.md` and `docs/dev-concurrency.md`.
+
+## Conventions
+
+- **Every table is dunder-named** — `__beaver_logs__`, `__beaver_documents__`,
+  … (`core.py:63`). The single-prefix form (`beaver_logs`) is *1.x* and its
+  presence means a legacy file; see issue #41.
+- **The `.db` stays a valid, standard SQLite file.** Anyone must be able to open
+  it with `sqlite3`. That is a promise from `design.md`, not a preference.
+- **Schema changes are additive.** `_create_all_tables()` runs on *every*
+  `connect()` with `CREATE TABLE IF NOT EXISTS`, so a new table simply appears on
+  an existing database — no migration. Adding a side table is the house pattern
+  (see `__beaver_fts_index__`, `__beaver_trigrams__`, `__beaver_lsh_index__`).
+- **`BEAVER_DB_VERSION` (`core.py:42`) is at 1 and moving it is a breaking
+  change.** `_check_version` raises `BeaverIncompatibleSchemaError` when a file's
+  `user_version` exceeds what the running beaver understands (`core.py:372`), so
+  a bump means databases written by the new release **fail to open on the old
+  one**. Prefer a design that fits additively; bump only when you have decided to
+  break file compatibility on purpose.
+- **New capability on a manager? Decorate it.** `@expose(path=…, method=…,
+  cli_name=…, cli_help=…)` makes it reachable from the CLI, the REST server and
+  the remote client at once; `@local_only("reason")` marks what cannot cross the
+  wire (infinite streams, file-level operations). A method with neither is
+  local-only *by accident* rather than by decision.
+- **Minimal dependencies is a product constraint.** Core is `aiosqlite`, `numpy`,
+  `pydantic`, `rich`, `typer`. Heavy things go behind extras (`remote`,
+  `security`). In particular, **vector indexing stays pure-numpy** — no
+  `hnswlib`, no `faiss`, no compiled deps. The LSH strategy exists precisely so
+  scaling did not require one (issue #24); HNSW (#28) stays deferred until a
+  pure-numpy ANN beats LSH at >100k vectors.
+- **Silence is the bug.** The lesson of issue #41 — beaver opened a 1.x file,
+  reported everything empty, and wrote a parallel schema beside the untouched
+  data. Nothing errored; that was the problem. When state is partial, stale or
+  unmigrated, **say so loudly or degrade to the correct-but-slower path** — never
+  return a plausible subset.
+- Ships to `main`. English for code, docs, comments and commit messages.
+
+## Working here
+
+    make sync           uv sync --extra security --extra remote
+    uv run black .      format (see the warning below)
+    make format-check   black --check — test-unit depends on this
+    make type-check     mypy
+    make test-unit      pytest tests/unit --cov=beaver     ← the default target
+    make test-all       every suite, including concurrency
+    make ci             format-check + test-unit
+
+`pytest.ini` sets `asyncio_mode = auto` (no `@pytest.mark.asyncio` needed) and a
+**30 s timeout** — a test that hangs on a lock fails rather than wedging the run.
+Markers: `unit`, `integration`, `concurrency`.
+
+> ⚠️ **`make format` and `make bugfix` commit for you**, and they do it with
+> `git commit -am` / `git commit -a`, which sweeps up **every dirty file in the
+> tree** — including work from a concurrent session that has nothing to do with
+> yours. Use **`uv run black .`** and commit the paths you touched yourself.
+> `make format-check` is safe and is what `test-unit` actually gates on.
+
+**Releases:** `NEW_VERSION=x.y.z make release` — runs `test-all`, bumps the
+version in `pyproject.toml` *and* `beaver/__init__.py`, commits, tags, pushes and
+opens the GitHub release. Don't hand-edit those two version strings.
+
+**Issues are files.** `issues/NN-slug.md` with `number / title / state / labels`
+frontmatter, round-tripped to GitHub by `make issues` (needs the `gh md-issues`
+extension). A substantial feature gets an issue file written *first* — see
+[`issues/24-…`](issues/) (LSH strategy) and
+[`issues/42-uniform-field-indexing-across-collections.md`](issues/42-uniform-field-indexing-across-collections.md)
+(field indexing) for the shape: Summary → the problem with real line references →
+Design → costs → delivery order → testing.
+
+## Gotchas paid for in advance
+
+- **FTS5 rejects punctuation in the query string** — tokenize before you hand a
+  user's words to `.search()` / `.fts()`. Measured on 2.2.0 against an indexed
+  document, three different `OperationalError`s:
+
+  | query | error |
+  |---|---|
+  | `thing` | — (1 hit) |
+  | `what's this?` | `fts5: syntax error near "'"` |
+  | `this?` | `fts5: syntax error near "?"` |
+  | `a-b` | `no such column: b` |
+
+  The hyphen case is the trap: FTS5 reads `a-b` as a **column filter**, so you
+  get a "no such column" error that says nothing about punctuation and sends you
+  hunting for a schema bug that isn't there.
+- **There is no reranking.** `docs` gives you FTS rank and trigram counts;
+  combining lexical and vector results is the caller's job (roll your own RRF).
+- **Embeddings are not part of `docs`.** A `docs()` collection and a `vectors()`
+  collection are separate things you keep in sync yourself; `docs` never embeds.
+- **`docs().where()` is a full scan today** — it compiles to `json_extract` per
+  row (`docs.py:472`), which is fast but not an index. Issue #42 is the fix.
+- **`__beaver_priority_queues__` has no declared PRIMARY KEY.** Its implicit
+  `rowid` is the only stable per-item identity. Any code assuming every table
+  declares a PK is wrong.
+- **`log()` resolves timestamp collisions by micro-incrementing and retrying**
+  (`logs.py:96`). Under many concurrent writers that loop can spin — use
+  `batched()`, which enforces monotonicity up front.
+
+## Where the planning lives
+
+Design docs and the v2 release plan live in Alex's workspace vault
+(`vault/Atlas/Architecture/2026-05-15-beaver-v2-release-plan.md`), not in this
+repo — this repo carries `design.md`, `STATUS.md` and `issues/`. If you need the
+wider roadmap and cannot see the vault, ask rather than guess.

{beaver_db-2.2.0 → beaver_db-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: beaver-db
-Version: 2.2.0
+Version: 2.3.0
 Summary: Fast, async-native, embedded, and multi-modal DB based on SQLite for AI-powered applications.
 License-File: LICENSE
 Classifier: License :: OSI Approved :: MIT License

{beaver_db-2.2.0 → beaver_db-2.3.0}/beaver/__init__.py

@@ -10,7 +10,7 @@ from .queries import q
 from .security import Secret
 from .errors import BeaverRemoteError, LocalOnlyError
 
-__version__ = "2.2.0"
+__version__ = "2.3.0"
 
 
 def connect(source: str, *, sync: bool = True, api_key: str | None = None):

{beaver_db-2.2.0 → beaver_db-2.3.0}/beaver/cli/discovery.py

@@ -171,7 +171,7 @@ def _build_command(method_name: str, meta: EndpointMeta, manager_accessor: Calla
             _run(ctx, manager_accessor, method_name, block=block, timeout=timeout)
 
     # --- shared no-arg shapes ---
-    elif method_name in ("count", "clear"):
+    elif method_name in ("count", "clear", "indexes"):
 
         def cmd(ctx: typer.Context):
             _run(ctx, manager_accessor, method_name)

{beaver_db-2.2.0 → beaver_db-2.3.0}/beaver/core.py

@@ -484,6 +484,41 @@ class AsyncBeaverDB:
             "CREATE INDEX IF NOT EXISTS idx_logs_timestamp ON __beaver_logs__ (log_name, timestamp)"
         )
 
+        # Field Index (issue #42) — uniform per-field indexing.
+        # Additive by design: new tables only, no ALTER, no version bump.
+        await c.execute(
+            """
+            CREATE TABLE IF NOT EXISTS __beaver_field_index__ (
+                kind      TEXT NOT NULL,
+                name      TEXT NOT NULL,
+                item_key  TEXT NOT NULL,
+                field     TEXT NOT NULL,
+                value     TEXT,
+                value_num REAL,
+                PRIMARY KEY (kind, name, item_key, field)
+            )
+        """
+        )
+        await c.execute(
+            "CREATE INDEX IF NOT EXISTS idx_field_lookup "
+            "ON __beaver_field_index__ (kind, name, field, value)"
+        )
+        await c.execute(
+            "CREATE INDEX IF NOT EXISTS idx_field_lookup_num "
+            "ON __beaver_field_index__ (kind, name, field, value_num)"
+        )
+        await c.execute(
+            """
+            CREATE TABLE IF NOT EXISTS __beaver_field_index_manifest__ (
+                kind     TEXT NOT NULL,
+                name     TEXT NOT NULL,
+                field    TEXT NOT NULL,
+                complete INTEGER NOT NULL DEFAULT 0,
+                PRIMARY KEY (kind, name, field)
+            )
+        """
+        )
+
         # Priority Queues
         await c.execute(
             """
@@ -685,9 +720,28 @@ class AsyncBeaverDB:
         return self.singleton(AsyncBeaverBlob, name, model=model)
 
     def log[T: BaseModel](
-        self, name: str, model: type[T] | None = None
+        self,
+        name: str,
+        model: type[T] | None = None,
+        # Quoted: inside a PEP 695 generic scope the class body is visible, so
+        # a bare `list[str]` resolves to AsyncBeaverDB.list, not the builtin.
+        indexed: "list[str] | None" = None,
     ) -> AsyncBeaverLog[T]:
-        return self.singleton(AsyncBeaverLog, name, model=model)
+        """Open a log.
+
+        `indexed` declares the fields whose values are maintained in the field
+        index. **Declare on first use:** `singleton()` keys its cache on
+        `(cls, name)` only, so after a bare `db.log("audit")` a later
+        `db.log("audit", indexed=[...])` returns the already-cached, undeclared
+        manager and the declaration is silently dropped.
+        """
+        return self.singleton(AsyncBeaverLog, name, model=model, indexed=indexed)
+
+    async def indexes(self) -> dict:
+        """Every declared field index in this database, by (kind, name)."""
+        from . import indexing
+
+        return await indexing.all_field_indexes(self.connection)
 
     def lock(
         self, name: str, timeout=None, lock_ttl=60.0, poll_interval=0.1

{beaver_db-2.2.0 → beaver_db-2.3.0}/beaver/docs.py

@@ -11,6 +11,7 @@ from typing import (
 
 from pydantic import BaseModel
 
+from . import indexing
 from .queries import Filter
 from .manager import AsyncBeaverBase, atomic, emits
 from .interfaces import Document, ScoredDocument, IDocumentQuery
@@ -468,11 +469,11 @@ class AsyncBeaverDocuments[T: BaseModel](AsyncBeaverBase[T]):
                 params.extend(q._search_fields)
 
         if q._filters:
-            for filter in q._filters:
-                where.append(
-                    f"json_extract(d.data, '$.{filter.path}') {filter.operator} ?"
-                )
-                params.append(filter.value)
+            clauses, filter_params = indexing.compile_scan_filters(
+                "data", q._filters, alias="d"
+            )
+            where.extend(clauses)
+            params.extend(filter_params)
 
         parts.append("WHERE " + " AND ".join(where))
 

beaver_db-2.3.0/beaver/indexing.py

@@ -0,0 +1,260 @@
+"""Uniform per-field indexing across collections (issue #42).
+
+Owns the whole substrate: dotted-path extraction, scalar normalization, the
+completeness manifest, index-row maintenance, and filter -> SQL compilation.
+Knows nothing about any specific collection; callers pass their `kind`, `name`
+and per-item key.
+"""
+
+from dataclasses import dataclass, field as _dc_field
+from typing import Any
+
+from pydantic import BaseModel
+
+from .queries import Filter
+
+INDEX_TABLE = "__beaver_field_index__"
+MANIFEST_TABLE = "__beaver_field_index_manifest__"
+
+_MISSING = object()
+
+
+@dataclass
+class FieldIndex:
+    """The state of one declared field.
+
+    `declared` says the write path maintains it; `complete` says the read path
+    may trust it. They are different questions and both matter.
+    """
+
+    field: str
+    declared: bool
+    complete: bool
+    rows: int
+
+
+@dataclass
+class QueryPlan:
+    """Which filters resolved by index and which fell back to a scan."""
+
+    indexed: list[str] = _dc_field(default_factory=list)
+    scanned: list[str] = _dc_field(default_factory=list)
+    estimated_rows: int = 0
+
+
+class UnindexableFieldError(TypeError):
+    """Raised when a declared field holds a non-scalar value."""
+
+
+def extract_path(item: Any, path: str) -> tuple[bool, Any]:
+    """Walk a dotted path through models and dicts.
+
+    Returns ``(found, value)``. ``found`` is False when any segment is absent
+    or the path runs through a non-traversable value — distinguishing "field
+    absent" from "field present and None".
+    """
+    current: Any = item
+    for part in path.split("."):
+        if isinstance(current, BaseModel):
+            current = getattr(current, part, _MISSING)
+        elif isinstance(current, dict):
+            current = current.get(part, _MISSING)
+        else:
+            return False, None
+        if current is _MISSING:
+            return False, None
+    return True, current
+
+
+def normalize(value: Any) -> tuple[str | None, float | None]:
+    """Split a scalar into its (text, numeric) index forms.
+
+    ``value_num`` is what makes ``duration_ms > 1000`` exclude 900; without it
+    the comparison is lexicographic and "900" > "1000".
+    """
+    if value is None:
+        return None, None
+    if isinstance(value, bool):
+        return str(value), float(value)
+    if isinstance(value, (int, float)):
+        return str(value), float(value)
+    if isinstance(value, str):
+        return value, None
+    raise UnindexableFieldError(
+        f"Cannot index a value of type {type(value).__name__}; "
+        "only str, int, float, bool and None are indexable."
+    )
+
+
+async def declare(conn, kind: str, name: str, fields: list[str]) -> None:
+    """Record fields as declared. Idempotent, and never downgrades `complete`.
+
+    Re-declaring an already-complete field must not reset it, or every process
+    restart would silently drop the collection back to scanning.
+    """
+    await conn.executemany(
+        f"INSERT OR IGNORE INTO {MANIFEST_TABLE} (kind, name, field, complete) "
+        "VALUES (?, ?, ?, 0)",
+        [(kind, name, f) for f in fields],
+    )
+
+
+async def manifest(conn, kind: str, name: str) -> dict[str, bool]:
+    """Map declared field -> whether the read path may trust its index."""
+    cursor = await conn.execute(
+        f"SELECT field, complete FROM {MANIFEST_TABLE} WHERE kind = ? AND name = ?",
+        (kind, name),
+    )
+    return {row[0]: bool(row[1]) for row in await cursor.fetchall()}
+
+
+async def mark_complete(conn, kind: str, name: str, fields: list[str]) -> None:
+    """Mark fields as fully backfilled, so queries may use the index."""
+    await conn.executemany(
+        f"UPDATE {MANIFEST_TABLE} SET complete = 1 "
+        "WHERE kind = ? AND name = ? AND field = ?",
+        [(kind, name, f) for f in fields],
+    )
+
+
+def build_rows(
+    kind: str, name: str, item_key: str, item: Any, fields: list[str]
+) -> list[tuple]:
+    """The index rows for one item. Absent fields produce no row."""
+    rows: list[tuple] = []
+    for field in fields:
+        found, raw = extract_path(item, field)
+        if not found:
+            continue
+        value, value_num = normalize(raw)
+        rows.append((kind, name, item_key, field, value, value_num))
+    return rows
+
+
+async def index_item(
+    conn, kind: str, name: str, item_key: str, item: Any, fields: list[str]
+) -> None:
+    """Replace this item's index rows. Delete-then-insert so an overwrite
+    cannot leave a stale row behind for a field that is now absent."""
+    await unindex_item(conn, kind, name, item_key)
+    rows = build_rows(kind, name, item_key, item, fields)
+    if rows:
+        await conn.executemany(
+            f"INSERT INTO {INDEX_TABLE} "
+            "(kind, name, item_key, field, value, value_num) "
+            "VALUES (?, ?, ?, ?, ?, ?)",
+            rows,
+        )
+
+
+async def unindex_item(conn, kind: str, name: str, item_key: str) -> None:
+    await conn.execute(
+        f"DELETE FROM {INDEX_TABLE} WHERE kind = ? AND name = ? AND item_key = ?",
+        (kind, name, item_key),
+    )
+
+
+async def clear_index(conn, kind: str, name: str) -> None:
+    await conn.execute(
+        f"DELETE FROM {INDEX_TABLE} WHERE kind = ? AND name = ?", (kind, name)
+    )
+
+
+async def index_rows(conn, kind: str, name: str) -> int:
+    cursor = await conn.execute(
+        f"SELECT COUNT(*) FROM {INDEX_TABLE} WHERE kind = ? AND name = ?",
+        (kind, name),
+    )
+    row = await cursor.fetchone()
+    return row[0] if row else 0
+
+
+def compile_scan_filters(
+    column: str, filters: list[Filter], alias: str = "d"
+) -> tuple[list[str], list]:
+    """Compile filters to json_extract predicates — the unindexed fallback.
+
+    Correct for any field, indexed or not, which is what lets an incomplete
+    index degrade to something slower instead of something wrong.
+    """
+    clauses: list[str] = []
+    params: list = []
+    for f in filters:
+        clauses.append(f"json_extract({alias}.{column}, '$.{f.path}') {f.operator} ?")
+        params.append(f.value)
+    return clauses, params
+
+
+_NUMERIC_OPS = {">", ">=", "<", "<="}
+
+
+def compile_indexed_filter(
+    kind: str, name: str, f: Filter, key_expr: str
+) -> tuple[str, list]:
+    """One filter as a membership test against the field index.
+
+    ``key_expr`` is how the outer table's row identity is written in SQL —
+    for logs, ``timestamp``, compared against ``CAST(item_key AS REAL)``.
+    """
+    column = "value_num" if f.operator in _NUMERIC_OPS else "value"
+    value = float(f.value) if f.operator in _NUMERIC_OPS else f.value
+    sql = (
+        f"{key_expr} IN (SELECT CAST(item_key AS REAL) FROM {INDEX_TABLE} "
+        f"WHERE kind = ? AND name = ? AND field = ? AND {column} {f.operator} ?)"
+    )
+    return sql, [kind, name, f.path, value]
+
+
+def plan_filters(
+    kind: str,
+    name: str,
+    filters: list[Filter],
+    complete: dict[str, bool],
+    key_expr: str,
+    column: str,
+    alias: str,
+) -> tuple[list[str], list, list[str], list[str]]:
+    """Split filters into indexed and scanned, and compile both.
+
+    Returns ``(clauses, params, indexed_paths, scanned_paths)``. A field is only
+    routed to the index when the manifest says it is complete — an unbackfilled
+    index would answer with a subset and say nothing about it.
+    """
+    clauses: list[str] = []
+    params: list = []
+    indexed: list[str] = []
+    scanned: list[str] = []
+    for f in filters:
+        if complete.get(f.path):
+            sql, ps = compile_indexed_filter(kind, name, f, key_expr)
+            clauses.append(sql)
+            params.extend(ps)
+            indexed.append(f.path)
+        else:
+            sql_list, ps = compile_scan_filters(column, [f], alias=alias)
+            clauses.extend(sql_list)
+            params.extend(ps)
+            scanned.append(f.path)
+    return clauses, params, indexed, scanned
+
+
+async def field_indexes(conn, kind: str, name: str) -> list[FieldIndex]:
+    declared = await manifest(conn, kind, name)
+    cursor = await conn.execute(
+        f"SELECT field, COUNT(*) FROM {INDEX_TABLE} "
+        "WHERE kind = ? AND name = ? GROUP BY field",
+        (kind, name),
+    )
+    counts = {row[0]: row[1] for row in await cursor.fetchall()}
+    return [
+        FieldIndex(field=f, declared=True, complete=c, rows=counts.get(f, 0))
+        for f, c in sorted(declared.items())
+    ]
+
+
+async def all_field_indexes(conn) -> dict[tuple[str, str], list[FieldIndex]]:
+    cursor = await conn.execute(
+        f"SELECT DISTINCT kind, name FROM {MANIFEST_TABLE} ORDER BY kind, name"
+    )
+    pairs = [(r[0], r[1]) for r in await cursor.fetchall()]
+    return {(k, n): await field_indexes(conn, k, n) for k, n in pairs}