beaver-db 0.4.0__tar.gz → 0.5.1__tar.gz

beaver_db-0.5.1/PKG-INFO

@@ -0,0 +1,171 @@
+Metadata-Version: 2.4
+Name: beaver-db
+Version: 0.5.1
+Summary: Asynchronous, embedded, modern DB based on SQLite.
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+Requires-Dist: numpy>=2.3.3
+Requires-Dist: scipy>=1.16.2
+
+# beaver 🦫
+
+A fast, single-file, multi-modal database for Python, built with the standard `sqlite3` library.
+
+`beaver` is the **B**ackend for **E**mbedded, **A**ll-in-one **V**ector, **E**ntity, and **R**elationship storage. It's a simple, local, and embedded database designed to manage complex, modern data types without requiring a database server, built on top of SQLite.
+
+## Design Philosophy
+
+`beaver` is built with a minimalistic philosophy for small, local use cases where a full-blown database server would be overkill.
+
+  - **Minimalistic & Zero-Dependency**: Uses only Python's standard libraries (`sqlite3`) and `numpy`/`scipy`.
+  - **Synchronous & Thread-Safe**: Designed for simplicity and safety in multi-threaded environments.
+  - **Built for Local Applications**: Perfect for local AI tools, RAG prototypes, chatbots, and desktop utilities that need persistent, structured data without network overhead.
+  - **Fast by Default**: It's built on SQLite, which is famously fast and reliable for local applications. The vector search is accelerated with an in-memory k-d tree.
+  - **Standard Relational Interface**: While `beaver` provides high-level features, you can always use the same SQLite file for normal relational tasks with standard SQL.
+
+## Core Features
+
+  - **Synchronous Pub/Sub**: A simple, thread-safe, Redis-like publish-subscribe system for real-time messaging.
+  - **Persistent Key-Value Store**: A simple `set`/`get` interface for storing any JSON-serializable object.
+  - **Pythonic List Management**: A fluent, Redis-like interface for managing persistent, ordered lists.
+  - **Efficient Vector Storage & Search**: Store vector embeddings and perform fast approximate nearest neighbor searches using an in-memory k-d tree.
+  - **Full-Text Search**: Automatically index and search through document metadata using SQLite's powerful FTS5 engine.
+  - **Graph Traversal**: Create relationships between documents and traverse the graph to find neighbors or perform multi-hop walks.
+  - **Single-File & Portable**: All data is stored in a single SQLite file, making it incredibly easy to move, back up, or embed in your application.
+
+## Installation
+
+```bash
+pip install beaver-db
+```
+
+## Quickstart & API Guide
+
+### Initialization
+
+All you need to do is import and instantiate the `BeaverDB` class with a file path.
+
+```python
+from beaver import BeaverDB, Document
+
+db = BeaverDB("my_application.db")
+```
+
+### Key-Value Store
+
+Use `set()` and `get()` for simple data storage. The value can be any JSON-encodable object.
+
+```python
+# Set a value
+db.set("app_config", {"theme": "dark", "user_id": 123})
+
+# Get a value
+config = db.get("app_config")
+print(f"Theme: {config.get('theme')}") # Output: Theme: dark
+```
+
+### List Management
+
+Get a list wrapper with `db.list()` and use Pythonic methods to manage it.
+
+```python
+tasks = db.list("daily_tasks")
+tasks.push("Write the project report")
+tasks.prepend("Plan the day's agenda")
+print(f"The first task is: {tasks[0]}")
+```
+
+### Vector & Text Search
+
+Store `Document` objects containing vector embeddings and metadata. When you index a document, its string fields are automatically made available for full-text search.
+
+```python
+# Get a handle to a collection
+docs = db.collection("articles")
+
+# Create and index a multi-modal document
+doc = Document(
+    id="sql-001",
+    embedding=[0.8, 0.1, 0.1],
+    content="SQLite is a powerful embedded database ideal for local apps.",
+    author="John Smith"
+)
+docs.index(doc)
+
+# 1. Perform a vector search to find semantically similar documents
+query_vector = [0.7, 0.2, 0.2]
+vector_results = docs.search(vector=query_vector, top_k=1)
+top_doc, distance = vector_results[0]
+print(f"Vector Search Result: {top_doc.content} (distance: {distance:.2f})")
+
+# 2. Perform a full-text search to find documents with specific words
+text_results = docs.match(query="database", top_k=1)
+top_doc, rank = text_results[0]
+print(f"Full-Text Search Result: {top_doc.content} (rank: {rank:.2f})")
+```
+
+### Graph Traversal
+
+Create relationships between documents and traverse them.
+
+```python
+from beaver import WalkDirection
+
+# Create documents
+alice = Document(id="alice", name="Alice")
+bob = Document(id="bob", name="Bob")
+charlie = Document(id="charlie", name="Charlie")
+
+# Index them
+social_net = db.collection("social")
+social_net.index(alice)
+social_net.index(bob)
+social_net.index(charlie)
+
+# Create edges
+social_net.connect(alice, bob, label="FOLLOWS")
+social_net.connect(bob, charlie, label="FOLLOWS")
+
+# Find direct neighbors
+following = social_net.neighbors(alice, label="FOLLOWS")
+print(f"Alice follows: {[p.id for p in following]}")
+
+# Perform a multi-hop walk to find friends of friends
+foaf = social_net.walk(
+    source=alice,
+    labels=["FOLLOWS"],
+    depth=2,
+    direction=WalkDirection.OUTGOING,
+)
+print(f"Alice's extended network: {[p.id for p in foaf]}")
+```
+
+### Synchronous Pub/Sub
+
+Publish events from one part of your app and listen in another using threads.
+
+```python
+import threading
+
+def listener():
+    for message in db.subscribe("system_events"):
+        print(f"LISTENER: Received -> {message}")
+        if message.get("event") == "shutdown":
+            break
+
+def publisher():
+    db.publish("system_events", {"event": "user_login", "user": "alice"})
+    db.publish("system_events", {"event": "shutdown"})
+
+# Run them concurrently
+listener_thread = threading.Thread(target=listener)
+publisher_thread = threading.Thread(target=publisher)
+listener_thread.start()
+publisher_thread.start()
+listener_thread.join()
+publisher_thread.join()
+```
+
+## License
+
+This project is licensed under the MIT License.

beaver_db-0.5.1/README.md

@@ -0,0 +1,162 @@
+# beaver 🦫
+
+A fast, single-file, multi-modal database for Python, built with the standard `sqlite3` library.
+
+`beaver` is the **B**ackend for **E**mbedded, **A**ll-in-one **V**ector, **E**ntity, and **R**elationship storage. It's a simple, local, and embedded database designed to manage complex, modern data types without requiring a database server, built on top of SQLite.
+
+## Design Philosophy
+
+`beaver` is built with a minimalistic philosophy for small, local use cases where a full-blown database server would be overkill.
+
+  - **Minimalistic & Zero-Dependency**: Uses only Python's standard libraries (`sqlite3`) and `numpy`/`scipy`.
+  - **Synchronous & Thread-Safe**: Designed for simplicity and safety in multi-threaded environments.
+  - **Built for Local Applications**: Perfect for local AI tools, RAG prototypes, chatbots, and desktop utilities that need persistent, structured data without network overhead.
+  - **Fast by Default**: It's built on SQLite, which is famously fast and reliable for local applications. The vector search is accelerated with an in-memory k-d tree.
+  - **Standard Relational Interface**: While `beaver` provides high-level features, you can always use the same SQLite file for normal relational tasks with standard SQL.
+
+## Core Features
+
+  - **Synchronous Pub/Sub**: A simple, thread-safe, Redis-like publish-subscribe system for real-time messaging.
+  - **Persistent Key-Value Store**: A simple `set`/`get` interface for storing any JSON-serializable object.
+  - **Pythonic List Management**: A fluent, Redis-like interface for managing persistent, ordered lists.
+  - **Efficient Vector Storage & Search**: Store vector embeddings and perform fast approximate nearest neighbor searches using an in-memory k-d tree.
+  - **Full-Text Search**: Automatically index and search through document metadata using SQLite's powerful FTS5 engine.
+  - **Graph Traversal**: Create relationships between documents and traverse the graph to find neighbors or perform multi-hop walks.
+  - **Single-File & Portable**: All data is stored in a single SQLite file, making it incredibly easy to move, back up, or embed in your application.
+
+## Installation
+
+```bash
+pip install beaver-db
+```
+
+## Quickstart & API Guide
+
+### Initialization
+
+All you need to do is import and instantiate the `BeaverDB` class with a file path.
+
+```python
+from beaver import BeaverDB, Document
+
+db = BeaverDB("my_application.db")
+```
+
+### Key-Value Store
+
+Use `set()` and `get()` for simple data storage. The value can be any JSON-encodable object.
+
+```python
+# Set a value
+db.set("app_config", {"theme": "dark", "user_id": 123})
+
+# Get a value
+config = db.get("app_config")
+print(f"Theme: {config.get('theme')}") # Output: Theme: dark
+```
+
+### List Management
+
+Get a list wrapper with `db.list()` and use Pythonic methods to manage it.
+
+```python
+tasks = db.list("daily_tasks")
+tasks.push("Write the project report")
+tasks.prepend("Plan the day's agenda")
+print(f"The first task is: {tasks[0]}")
+```
+
+### Vector & Text Search
+
+Store `Document` objects containing vector embeddings and metadata. When you index a document, its string fields are automatically made available for full-text search.
+
+```python
+# Get a handle to a collection
+docs = db.collection("articles")
+
+# Create and index a multi-modal document
+doc = Document(
+    id="sql-001",
+    embedding=[0.8, 0.1, 0.1],
+    content="SQLite is a powerful embedded database ideal for local apps.",
+    author="John Smith"
+)
+docs.index(doc)
+
+# 1. Perform a vector search to find semantically similar documents
+query_vector = [0.7, 0.2, 0.2]
+vector_results = docs.search(vector=query_vector, top_k=1)
+top_doc, distance = vector_results[0]
+print(f"Vector Search Result: {top_doc.content} (distance: {distance:.2f})")
+
+# 2. Perform a full-text search to find documents with specific words
+text_results = docs.match(query="database", top_k=1)
+top_doc, rank = text_results[0]
+print(f"Full-Text Search Result: {top_doc.content} (rank: {rank:.2f})")
+```
+
+### Graph Traversal
+
+Create relationships between documents and traverse them.
+
+```python
+from beaver import WalkDirection
+
+# Create documents
+alice = Document(id="alice", name="Alice")
+bob = Document(id="bob", name="Bob")
+charlie = Document(id="charlie", name="Charlie")
+
+# Index them
+social_net = db.collection("social")
+social_net.index(alice)
+social_net.index(bob)
+social_net.index(charlie)
+
+# Create edges
+social_net.connect(alice, bob, label="FOLLOWS")
+social_net.connect(bob, charlie, label="FOLLOWS")
+
+# Find direct neighbors
+following = social_net.neighbors(alice, label="FOLLOWS")
+print(f"Alice follows: {[p.id for p in following]}")
+
+# Perform a multi-hop walk to find friends of friends
+foaf = social_net.walk(
+    source=alice,
+    labels=["FOLLOWS"],
+    depth=2,
+    direction=WalkDirection.OUTGOING,
+)
+print(f"Alice's extended network: {[p.id for p in foaf]}")
+```
+
+### Synchronous Pub/Sub
+
+Publish events from one part of your app and listen in another using threads.
+
+```python
+import threading
+
+def listener():
+    for message in db.subscribe("system_events"):
+        print(f"LISTENER: Received -> {message}")
+        if message.get("event") == "shutdown":
+            break
+
+def publisher():
+    db.publish("system_events", {"event": "user_login", "user": "alice"})
+    db.publish("system_events", {"event": "shutdown"})
+
+# Run them concurrently
+listener_thread = threading.Thread(target=listener)
+publisher_thread = threading.Thread(target=publisher)
+listener_thread.start()
+publisher_thread.start()
+listener_thread.join()
+publisher_thread.join()
+```
+
+## License
+
+This project is licensed under the MIT License.

beaver_db-0.5.1/beaver/__init__.py

@@ -0,0 +1,2 @@
+from .core import BeaverDB
+from .collections import Document, WalkDirection

beaver_db-0.5.1/beaver/collections.py

@@ -0,0 +1,327 @@
+import json
+import sqlite3
+import uuid
+from enum import Enum
+from typing import Any, List, Literal, Set
+
+import numpy as np
+from scipy.spatial import cKDTree
+
+
+class WalkDirection(Enum):
+    OUTGOING = "outgoing"
+    INCOMING = "incoming"
+
+
+class Document:
+    """A data class representing a single item in a collection."""
+
+    def __init__(
+        self, embedding: list[float] | None = None, id: str | None = None, **metadata
+    ):
+        self.id = id or str(uuid.uuid4())
+
+        if embedding is None:
+            self.embedding = None
+        else:
+            if not isinstance(embedding, list) or not all(
+                isinstance(x, (int, float)) for x in embedding
+            ):
+                raise TypeError("Embedding must be a list of numbers.")
+            self.embedding = np.array(embedding, dtype=np.float32)
+
+        for key, value in metadata.items():
+            setattr(self, key, value)
+
+    def to_dict(self) -> dict[str, Any]:
+        """Serializes the document's metadata to a dictionary."""
+        metadata = self.__dict__.copy()
+        metadata.pop("embedding", None)
+        metadata.pop("id", None)
+        return metadata
+
+    def __repr__(self):
+        metadata_str = ", ".join(f"{k}={v!r}" for k, v in self.to_dict().items())
+        return f"Document(id='{self.id}', {metadata_str})"
+
+
+class CollectionWrapper:
+    """
+    A wrapper for multi-modal collection operations with an in-memory ANN index,
+    FTS, and graph capabilities.
+    """
+
+    def __init__(self, name: str, conn: sqlite3.Connection):
+        self._name = name
+        self._conn = conn
+        self._kdtree: cKDTree | None = None
+        self._doc_ids: List[str] = []
+        self._local_index_version = -1  # Version of the in-memory index
+
+    def _flatten_metadata(self, metadata: dict, prefix: str = "") -> dict[str, str]:
+        """Flattens a nested dictionary and filters for string values."""
+        flat_dict = {}
+        for key, value in metadata.items():
+            new_key = f"{prefix}__{key}" if prefix else key
+            if isinstance(value, dict):
+                flat_dict.update(self._flatten_metadata(value, new_key))
+            elif isinstance(value, str):
+                flat_dict[new_key] = value
+        return flat_dict
+
+    def _get_db_version(self) -> int:
+        """Gets the current version of the collection from the database."""
+        cursor = self._conn.cursor()
+        cursor.execute(
+            "SELECT version FROM beaver_collection_versions WHERE collection_name = ?",
+            (self._name,),
+        )
+        result = cursor.fetchone()
+        return result[0] if result else 0
+
+    def _is_index_stale(self) -> bool:
+        """Checks if the in-memory index is out of sync with the DB."""
+        if self._local_index_version == -1:
+            return True
+        return self._local_index_version < self._get_db_version()
+
+    def index(self, document: Document, *, fts: bool = True):
+        """Indexes a Document, performing an upsert and updating the FTS index."""
+        with self._conn:
+            if fts:
+                self._conn.execute(
+                    "DELETE FROM beaver_fts_index WHERE collection = ? AND item_id = ?",
+                    (self._name, document.id),
+                )
+                string_fields = self._flatten_metadata(document.to_dict())
+                if string_fields:
+                    fts_data = [
+                        (self._name, document.id, path, content)
+                        for path, content in string_fields.items()
+                    ]
+                    self._conn.executemany(
+                        "INSERT INTO beaver_fts_index (collection, item_id, field_path, field_content) VALUES (?, ?, ?, ?)",
+                        fts_data,
+                    )
+
+            self._conn.execute(
+                "INSERT OR REPLACE INTO beaver_collections (collection, item_id, item_vector, metadata) VALUES (?, ?, ?, ?)",
+                (
+                    self._name,
+                    document.id,
+                    (
+                        document.embedding.tobytes()
+                        if document.embedding is not None
+                        else None
+                    ),
+                    json.dumps(document.to_dict()),
+                ),
+            )
+            # Atomically increment the collection's version number
+            self._conn.execute(
+                """
+                INSERT INTO beaver_collection_versions (collection_name, version) VALUES (?, 1)
+                ON CONFLICT(collection_name) DO UPDATE SET version = version + 1
+                """,
+                (self._name,),
+            )
+
+    def drop(self, document: Document):
+        """Removes a document and all its associated data from the collection."""
+        if not isinstance(document, Document):
+            raise TypeError("Item to drop must be a Document object.")
+        with self._conn:
+            self._conn.execute(
+                "DELETE FROM beaver_collections WHERE collection = ? AND item_id = ?",
+                (self._name, document.id),
+            )
+            self._conn.execute(
+                "DELETE FROM beaver_fts_index WHERE collection = ? AND item_id = ?",
+                (self._name, document.id),
+            )
+            self._conn.execute(
+                "DELETE FROM beaver_edges WHERE collection = ? AND (source_item_id = ? OR target_item_id = ?)",
+                (self._name, document.id, document.id),
+            )
+            self._conn.execute(
+                """
+                INSERT INTO beaver_collection_versions (collection_name, version) VALUES (?, 1)
+                ON CONFLICT(collection_name) DO UPDATE SET version = version + 1
+                """,
+                (self._name,),
+            )
+
+    def refresh(self):
+        """Forces a rebuild of the in-memory ANN index from data in SQLite."""
+        cursor = self._conn.cursor()
+        cursor.execute(
+            "SELECT item_id, item_vector FROM beaver_collections WHERE collection = ? AND item_vector IS NOT NULL",
+            (self._name,),
+        )
+        vectors, self._doc_ids = [], []
+        for row in cursor.fetchall():
+            self._doc_ids.append(row["item_id"])
+            vectors.append(np.frombuffer(row["item_vector"], dtype=np.float32))
+
+        self._kdtree = cKDTree(vectors) if vectors else None
+        self._local_index_version = self._get_db_version()
+
+    def search(
+        self, vector: list[float], top_k: int = 10
+    ) -> list[tuple[Document, float]]:
+        """Performs a fast approximate nearest neighbor search."""
+        if self._is_index_stale():
+            self.refresh()
+        if not self._kdtree:
+            return []
+
+        distances, indices = self._kdtree.query(
+            np.array(vector, dtype=np.float32), k=top_k
+        )
+        if top_k == 1:
+            distances, indices = [distances], [indices]
+
+        result_ids = [self._doc_ids[i] for i in indices]
+        placeholders = ",".join("?" for _ in result_ids)
+        sql = f"SELECT item_id, item_vector, metadata FROM beaver_collections WHERE collection = ? AND item_id IN ({placeholders})"
+
+        cursor = self._conn.cursor()
+        rows = cursor.execute(sql, (self._name, *result_ids)).fetchall()
+        row_map = {row["item_id"]: row for row in rows}
+
+        results = []
+        for i, doc_id in enumerate(result_ids):
+            row = row_map.get(doc_id)
+            if row:
+                embedding = np.frombuffer(row["item_vector"], dtype=np.float32).tolist()
+                doc = Document(
+                    id=doc_id, embedding=embedding, **json.loads(row["metadata"])
+                )
+                results.append((doc, float(distances[i])))
+        return results
+
+    def match(
+        self, query: str, on_field: str | None = None, top_k: int = 10
+    ) -> list[tuple[Document, float]]:
+        """Performs a full-text search on indexed string fields."""
+        cursor = self._conn.cursor()
+        sql_query = """
+            SELECT t1.item_id, t1.item_vector, t1.metadata, fts.rank
+            FROM beaver_collections AS t1 JOIN (
+                SELECT DISTINCT item_id, rank FROM beaver_fts_index
+                WHERE beaver_fts_index MATCH ? {} ORDER BY rank LIMIT ?
+            ) AS fts ON t1.item_id = fts.item_id
+            WHERE t1.collection = ? ORDER BY fts.rank
+        """
+        params, field_filter_sql = [], ""
+        if on_field:
+            field_filter_sql = "AND field_path = ?"
+            params.extend([query, on_field])
+        else:
+            params.append(query)
+        params.extend([top_k, self._name])
+
+        rows = cursor.execute(
+            sql_query.format(field_filter_sql), tuple(params)
+        ).fetchall()
+        results = []
+        for row in rows:
+            embedding = (
+                np.frombuffer(row["item_vector"], dtype=np.float32).tolist()
+                if row["item_vector"]
+                else None
+            )
+            doc = Document(
+                id=row["item_id"], embedding=embedding, **json.loads(row["metadata"])
+            )
+            results.append((doc, row["rank"]))
+        return results
+
+    def connect(
+        self, source: Document, target: Document, label: str, metadata: dict = None
+    ):
+        """Creates a directed edge between two documents."""
+        if not isinstance(source, Document) or not isinstance(target, Document):
+            raise TypeError("Source and target must be Document objects.")
+        with self._conn:
+            self._conn.execute(
+                "INSERT OR REPLACE INTO beaver_edges (collection, source_item_id, target_item_id, label, metadata) VALUES (?, ?, ?, ?, ?)",
+                (
+                    self._name,
+                    source.id,
+                    target.id,
+                    label,
+                    json.dumps(metadata) if metadata else None,
+                ),
+            )
+
+    def neighbors(self, doc: Document, label: str | None = None) -> list[Document]:
+        """Retrieves the neighboring documents connected to a given document."""
+        sql = "SELECT t1.item_id, t1.item_vector, t1.metadata FROM beaver_collections AS t1 JOIN beaver_edges AS t2 ON t1.item_id = t2.target_item_id AND t1.collection = t2.collection WHERE t2.collection = ? AND t2.source_item_id = ?"
+        params = [self._name, doc.id]
+        if label:
+            sql += " AND t2.label = ?"
+            params.append(label)
+
+        rows = self._conn.cursor().execute(sql, tuple(params)).fetchall()
+        return [
+            Document(
+                id=row["item_id"],
+                embedding=(
+                    np.frombuffer(row["item_vector"], dtype=np.float32).tolist()
+                    if row["item_vector"]
+                    else None
+                ),
+                **json.loads(row["metadata"]),
+            )
+            for row in rows
+        ]
+
+    def walk(
+        self,
+        source: Document,
+        labels: List[str],
+        depth: int,
+        *,
+        direction: Literal[
+            WalkDirection.OUTGOING, WalkDirection.INCOMING
+        ] = WalkDirection.OUTGOING,
+    ) -> List[Document]:
+        """Performs a graph traversal (BFS) from a starting document."""
+        if not isinstance(source, Document):
+            raise TypeError("The starting point must be a Document object.")
+        if depth <= 0:
+            return []
+
+        source_col, target_col = (
+            ("source_item_id", "target_item_id")
+            if direction == WalkDirection.OUTGOING
+            else ("target_item_id", "source_item_id")
+        )
+        sql = f"""
+            WITH RECURSIVE walk_bfs(item_id, current_depth) AS (
+                SELECT ?, 0
+                UNION ALL
+                SELECT edges.{target_col}, bfs.current_depth + 1
+                FROM beaver_edges AS edges JOIN walk_bfs AS bfs ON edges.{source_col} = bfs.item_id
+                WHERE edges.collection = ? AND bfs.current_depth < ? AND edges.label IN ({','.join('?' for _ in labels)})
+            )
+            SELECT DISTINCT t1.item_id, t1.item_vector, t1.metadata
+            FROM beaver_collections AS t1 JOIN walk_bfs AS bfs ON t1.item_id = bfs.item_id
+            WHERE t1.collection = ? AND bfs.current_depth > 0
+        """
+        params = [source.id, self._name, depth] + labels + [self._name]
+
+        rows = self._conn.cursor().execute(sql, tuple(params)).fetchall()
+        return [
+            Document(
+                id=row["item_id"],
+                embedding=(
+                    np.frombuffer(row["item_vector"], dtype=np.float32).tolist()
+                    if row["item_vector"]
+                    else None
+                ),
+                **json.loads(row["metadata"]),
+            )
+            for row in rows
+        ]