beaver-db 0.7.1__tar.gz → 0.9.0__tar.gz

{beaver_db-0.7.1 → beaver_db-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: beaver-db
-Version: 0.7.1
+Version: 0.9.0
 Summary: Fast, embedded, and multi-modal DB based on SQLite for AI-powered applications.
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
@@ -9,11 +9,11 @@ Requires-Dist: numpy>=2.3.3
 Requires-Dist: scipy>=1.16.2
 Dynamic: license-file
 
-# beaver 🦫
+I've updated the README to highlight the new high-efficiency, thread-safe, and process-safe pub/sub system. I've also added an example of how you can use it to build real-time, event-driven applications.
+
+Here are the changes:
 
-![PyPI - Downloads](https://img.shields.io/pypi/dm/beaver-db)
-![PyPI](https://img.shields.io/pypi/v/beaver-db)
-![License](https://img.shields.io/github/license/apiad/beaver)
+# beaver 🦫
 
 A fast, single-file, multi-modal database for Python, built with the standard `sqlite3` library.
 
@@ -31,9 +31,10 @@ A fast, single-file, multi-modal database for Python, built with the standard `s
 
 ## Core Features
 
-  - **Synchronous Pub/Sub**: A simple, thread-safe, Redis-like publish-subscribe system for real-time messaging.
+  - **High-Efficiency Pub/Sub**: A powerful, thread and process-safe publish-subscribe system for real-time messaging with a fan-out architecture.
   - **Namespaced Key-Value Dictionaries**: A Pythonic, dictionary-like interface for storing any JSON-serializable object within separate namespaces with optional TTL for cache implementations.
   - **Pythonic List Management**: A fluent, Redis-like interface for managing persistent, ordered lists.
+  - **Persistent Priority Queue**: A high-performance, persistent queue that always returns the item with the highest priority, perfect for task management.
   - **Efficient Vector Storage & Search**: Store vector embeddings and perform fast approximate nearest neighbor searches using an in-memory k-d tree.
   - **Full-Text Search**: Automatically index and search through document metadata using SQLite's powerful FTS5 engine.
   - **Graph Traversal**: Create relationships between documents and traverse the graph to find neighbors or perform multi-hop walks.
@@ -86,7 +87,24 @@ db.close()
 
 Here are a few ideas to inspire your next project, showcasing how to combine Beaver's features to build powerful local applications.
 
-### 1. User Authentication and Profile Store
+### 1. AI Agent Task Management
+
+Use a **persistent priority queue** to manage tasks for an AI agent. This ensures the agent always works on the most important task first, even if the application restarts.
+
+```python
+tasks = db.queue("agent_tasks")
+
+# Tasks are added with a priority (lower is higher)
+tasks.put({"action": "summarize_news"}, priority=10)
+tasks.put({"action": "respond_to_user"}, priority=1)
+tasks.put({"action": "run_backup"}, priority=20)
+
+# The agent retrieves the highest-priority task
+next_task = tasks.get() # -> Returns the "respond_to_user" task
+print(f"Agent's next task: {next_task.data['action']}")
+```
+
+### 2. User Authentication and Profile Store
 
 Use a **namespaced dictionary** to create a simple and secure user store. The key can be the username, and the value can be a dictionary containing the hashed password and other profile information.
 
@@ -104,7 +122,7 @@ users["alice"] = {
 alice_profile = users.get("alice")
 ```
 
-### 2. Chatbot Conversation History
+### 3. Chatbot Conversation History
 
 A **persistent list** is perfect for storing the history of a conversation. Each time the user or the bot sends a message, just `push` it to the list. This maintains a chronological record of the entire dialogue.
 
@@ -119,7 +137,7 @@ for message in chat_history:
     print(f"{message['role']}: {message['content']}")
 ```
 
-### 3. Build a RAG (Retrieval-Augmented Generation) System
+### 4. Build a RAG (Retrieval-Augmented Generation) System
 
 Combine **vector search** and **full-text search** to build a powerful RAG pipeline for your local documents.
 
@@ -133,7 +151,7 @@ from beaver.collections import rerank
 best_context = rerank(vector_results, text_results, weights=[0.6, 0.4])
 ```
 
-### 4. Caching for Expensive API Calls
+### 5. Caching for Expensive API Calls
 
 Leverage a **dictionary with a TTL (Time-To-Live)** to cache the results of slow network requests. This can dramatically speed up your application and reduce your reliance on external services.
 
@@ -149,18 +167,36 @@ if response is None:
     api_cache.set("weather_new_york", response, ttl_seconds=3600)
 ```
 
+### 6. Real-time Event-Driven Systems
+
+Use the **high-efficiency pub/sub system** to build applications where different components react to events in real-time. This is perfect for decoupled systems, real-time UIs, or monitoring services.
+
+```python
+# In one process or thread (e.g., a monitoring service)
+system_events = db.channel("system_events")
+system_events.publish({"event": "user_login", "user_id": "alice"})
+
+# In another process or thread (e.g., a UI updater or logger)
+with db.channel("system_events").subscribe() as listener:
+    for message in listener.listen():
+        print(f"Event received: {message}")
+        # >> Event received: {'event': 'user_login', 'user_id': 'alice'}
+```
+
 ## More Examples
 
 For more in-depth examples, check out the scripts in the `examples/` directory:
 
-  - [`examples/kvstore.py`](https://www.google.com/search?q=examples/kvstore.py): A comprehensive demo of the namespaced dictionary feature.
-  - [`examples/list.py`](https://www.google.com/search?q=examples/list.py): Shows the full capabilities of the persistent list, including slicing and in-place updates.
-  - [`examples/vector.py`](https://www.google.com/search?q=examples/vector.py): Demonstrates how to index and search vector embeddings, including upserts.
-  - [`examples/fts.py`](https://www.google.com/search?q=examples/fts.py): A detailed look at full-text search, including targeted searches on specific metadata fields.
-  - [`examples/graph.py`](https://www.google.com/search?q=examples/graph.py): Shows how to create relationships between documents and perform multi-hop graph traversals.
-  - [`examples/pubsub.py`](https://www.google.com/search?q=examples/pubsub.py): A demonstration of the synchronous, thread-safe publish/subscribe system.
-  - [`examples/cache.py`](https://www.google.com/search?q=examples/cache.py): A practical example of using a dictionary with TTL as a cache for API calls.
-  - [`examples/rerank.py`](https://www.google.com/search?q=examples/rerank.py): Shows how to combine results from vector and text search for more refined results.
+  - [`examples/kvstore.py`](examples/kvstore.py): A comprehensive demo of the namespaced dictionary feature.
+  - [`examples/list.py`](examples/list.py): Shows the full capabilities of the persistent list, including slicing and in-place updates.
+  - [`examples/queue.py`](examples/queue.py): A practical example of using the persistent priority queue for task management.
+  - [`examples/vector.py`](examples/vector.py): Demonstrates how to index and search vector embeddings, including upserts.
+  - [`examples/fts.py`](examples/fts.py): A detailed look at full-text search, including targeted searches on specific metadata fields.
+  - [`examples/graph.py`](examples/graph.py): Shows how to create relationships between documents and perform multi-hop graph traversals.
+  - [`examples/pubsub.py`](examples/pubsub.py): A demonstration of the synchronous, thread-safe publish/subscribe system in a single process.
+  - [`examples/publisher.py`](examples/publisher.py) and [`examples/subscriber.py`](examples/subscriber.py): A pair of examples demonstrating inter-process message passing with the publish/subscribe system.
+  - [`examples/cache.py`](examples/cache.py): A practical example of using a dictionary with TTL as a cache for API calls.
+  - [`examples/rerank.py`](examples/rerank.py): Shows how to combine results from vector and text search for more refined results.
 
 ## Roadmap
 
@@ -168,11 +204,9 @@ These are some of the features and improvements planned for future releases:
 
   - **Fuzzy search**: Implement fuzzy matching capabilities for text search.
   - **Faster ANN**: Explore integrating more advanced ANN libraries like `faiss` for improved vector search performance.
-  - **Priority Queues**: Introduce a priority queue data structure for task management.
-  - **Improved Pub/Sub**: Fan-out implementation with a more Pythonic API.
   - **Async API**: Comprehensive async support with on-demand wrappers for all collections.
 
-Check out the [roadmap](https://www.google.com/search?q=roadmap.md) for a detailed list of upcoming features and design ideas.
+Check out the [roadmap](roadmap.md) for a detailed list of upcoming features and design ideas.
 
 ## License
 

{beaver_db-0.7.1 → beaver_db-0.9.0}/README.md

@@ -1,8 +1,8 @@
-# beaver 🦫
+I've updated the README to highlight the new high-efficiency, thread-safe, and process-safe pub/sub system. I've also added an example of how you can use it to build real-time, event-driven applications.
+
+Here are the changes:
 
-![PyPI - Downloads](https://img.shields.io/pypi/dm/beaver-db)
-![PyPI](https://img.shields.io/pypi/v/beaver-db)
-![License](https://img.shields.io/github/license/apiad/beaver)
+# beaver 🦫
 
 A fast, single-file, multi-modal database for Python, built with the standard `sqlite3` library.
 
@@ -20,9 +20,10 @@ A fast, single-file, multi-modal database for Python, built with the standard `s
 
 ## Core Features
 
-  - **Synchronous Pub/Sub**: A simple, thread-safe, Redis-like publish-subscribe system for real-time messaging.
+  - **High-Efficiency Pub/Sub**: A powerful, thread and process-safe publish-subscribe system for real-time messaging with a fan-out architecture.
   - **Namespaced Key-Value Dictionaries**: A Pythonic, dictionary-like interface for storing any JSON-serializable object within separate namespaces with optional TTL for cache implementations.
   - **Pythonic List Management**: A fluent, Redis-like interface for managing persistent, ordered lists.
+  - **Persistent Priority Queue**: A high-performance, persistent queue that always returns the item with the highest priority, perfect for task management.
   - **Efficient Vector Storage & Search**: Store vector embeddings and perform fast approximate nearest neighbor searches using an in-memory k-d tree.
   - **Full-Text Search**: Automatically index and search through document metadata using SQLite's powerful FTS5 engine.
   - **Graph Traversal**: Create relationships between documents and traverse the graph to find neighbors or perform multi-hop walks.
@@ -75,7 +76,24 @@ db.close()
 
 Here are a few ideas to inspire your next project, showcasing how to combine Beaver's features to build powerful local applications.
 
-### 1. User Authentication and Profile Store
+### 1. AI Agent Task Management
+
+Use a **persistent priority queue** to manage tasks for an AI agent. This ensures the agent always works on the most important task first, even if the application restarts.
+
+```python
+tasks = db.queue("agent_tasks")
+
+# Tasks are added with a priority (lower is higher)
+tasks.put({"action": "summarize_news"}, priority=10)
+tasks.put({"action": "respond_to_user"}, priority=1)
+tasks.put({"action": "run_backup"}, priority=20)
+
+# The agent retrieves the highest-priority task
+next_task = tasks.get() # -> Returns the "respond_to_user" task
+print(f"Agent's next task: {next_task.data['action']}")
+```
+
+### 2. User Authentication and Profile Store
 
 Use a **namespaced dictionary** to create a simple and secure user store. The key can be the username, and the value can be a dictionary containing the hashed password and other profile information.
 
@@ -93,7 +111,7 @@ users["alice"] = {
 alice_profile = users.get("alice")
 ```
 
-### 2. Chatbot Conversation History
+### 3. Chatbot Conversation History
 
 A **persistent list** is perfect for storing the history of a conversation. Each time the user or the bot sends a message, just `push` it to the list. This maintains a chronological record of the entire dialogue.
 
@@ -108,7 +126,7 @@ for message in chat_history:
     print(f"{message['role']}: {message['content']}")
 ```
 
-### 3. Build a RAG (Retrieval-Augmented Generation) System
+### 4. Build a RAG (Retrieval-Augmented Generation) System
 
 Combine **vector search** and **full-text search** to build a powerful RAG pipeline for your local documents.
 
@@ -122,7 +140,7 @@ from beaver.collections import rerank
 best_context = rerank(vector_results, text_results, weights=[0.6, 0.4])
 ```
 
-### 4. Caching for Expensive API Calls
+### 5. Caching for Expensive API Calls
 
 Leverage a **dictionary with a TTL (Time-To-Live)** to cache the results of slow network requests. This can dramatically speed up your application and reduce your reliance on external services.
 
@@ -138,18 +156,36 @@ if response is None:
     api_cache.set("weather_new_york", response, ttl_seconds=3600)
 ```
 
+### 6. Real-time Event-Driven Systems
+
+Use the **high-efficiency pub/sub system** to build applications where different components react to events in real-time. This is perfect for decoupled systems, real-time UIs, or monitoring services.
+
+```python
+# In one process or thread (e.g., a monitoring service)
+system_events = db.channel("system_events")
+system_events.publish({"event": "user_login", "user_id": "alice"})
+
+# In another process or thread (e.g., a UI updater or logger)
+with db.channel("system_events").subscribe() as listener:
+    for message in listener.listen():
+        print(f"Event received: {message}")
+        # >> Event received: {'event': 'user_login', 'user_id': 'alice'}
+```
+
 ## More Examples
 
 For more in-depth examples, check out the scripts in the `examples/` directory:
 
-  - [`examples/kvstore.py`](https://www.google.com/search?q=examples/kvstore.py): A comprehensive demo of the namespaced dictionary feature.
-  - [`examples/list.py`](https://www.google.com/search?q=examples/list.py): Shows the full capabilities of the persistent list, including slicing and in-place updates.
-  - [`examples/vector.py`](https://www.google.com/search?q=examples/vector.py): Demonstrates how to index and search vector embeddings, including upserts.
-  - [`examples/fts.py`](https://www.google.com/search?q=examples/fts.py): A detailed look at full-text search, including targeted searches on specific metadata fields.
-  - [`examples/graph.py`](https://www.google.com/search?q=examples/graph.py): Shows how to create relationships between documents and perform multi-hop graph traversals.
-  - [`examples/pubsub.py`](https://www.google.com/search?q=examples/pubsub.py): A demonstration of the synchronous, thread-safe publish/subscribe system.
-  - [`examples/cache.py`](https://www.google.com/search?q=examples/cache.py): A practical example of using a dictionary with TTL as a cache for API calls.
-  - [`examples/rerank.py`](https://www.google.com/search?q=examples/rerank.py): Shows how to combine results from vector and text search for more refined results.
+  - [`examples/kvstore.py`](examples/kvstore.py): A comprehensive demo of the namespaced dictionary feature.
+  - [`examples/list.py`](examples/list.py): Shows the full capabilities of the persistent list, including slicing and in-place updates.
+  - [`examples/queue.py`](examples/queue.py): A practical example of using the persistent priority queue for task management.
+  - [`examples/vector.py`](examples/vector.py): Demonstrates how to index and search vector embeddings, including upserts.
+  - [`examples/fts.py`](examples/fts.py): A detailed look at full-text search, including targeted searches on specific metadata fields.
+  - [`examples/graph.py`](examples/graph.py): Shows how to create relationships between documents and perform multi-hop graph traversals.
+  - [`examples/pubsub.py`](examples/pubsub.py): A demonstration of the synchronous, thread-safe publish/subscribe system in a single process.
+  - [`examples/publisher.py`](examples/publisher.py) and [`examples/subscriber.py`](examples/subscriber.py): A pair of examples demonstrating inter-process message passing with the publish/subscribe system.
+  - [`examples/cache.py`](examples/cache.py): A practical example of using a dictionary with TTL as a cache for API calls.
+  - [`examples/rerank.py`](examples/rerank.py): Shows how to combine results from vector and text search for more refined results.
 
 ## Roadmap
 
@@ -157,11 +193,9 @@ These are some of the features and improvements planned for future releases:
 
   - **Fuzzy search**: Implement fuzzy matching capabilities for text search.
   - **Faster ANN**: Explore integrating more advanced ANN libraries like `faiss` for improved vector search performance.
-  - **Priority Queues**: Introduce a priority queue data structure for task management.
-  - **Improved Pub/Sub**: Fan-out implementation with a more Pythonic API.
   - **Async API**: Comprehensive async support with on-demand wrappers for all collections.
 
-Check out the [roadmap](https://www.google.com/search?q=roadmap.md) for a detailed list of upcoming features and design ideas.
+Check out the [roadmap](roadmap.md) for a detailed list of upcoming features and design ideas.
 
 ## License
 

beaver_db-0.9.0/beaver/channels.py

@@ -0,0 +1,185 @@
+import json
+import sqlite3
+import threading
+import time
+from queue import Empty, Queue
+from typing import Any, Iterator, Set
+
+# A special message object used to signal the listener to gracefully shut down.
+_SHUTDOWN_SENTINEL = object()
+
+
+class Subscriber:
+    """
+    A thread-safe message receiver for a specific channel subscription.
+
+    This object is designed to be used as a context manager (`with` statement).
+    It holds a dedicated in-memory queue that receives messages from the
+    channel's central polling thread, ensuring that a slow listener does not
+    impact others.
+    """
+
+    def __init__(self, channel: "ChannelManager"):
+        self._channel = channel
+        self._queue: Queue = Queue()
+
+    def __enter__(self) -> "Subscriber":
+        """Registers the listener's queue with the channel to start receiving messages."""
+        self._channel._register(self._queue)
+        return self
+
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        """Unregisters the listener's queue from the channel to stop receiving messages."""
+        self._channel._unregister(self._queue)
+
+    def listen(self, timeout: float | None = None) -> Iterator[Any]:
+        """
+        Returns a blocking iterator that yields messages as they arrive.
+
+        This method pulls messages from the listener's dedicated, thread-safe
+        in-memory queue. It performs no database operations itself.
+
+        Args:
+            timeout: If provided, the iterator will raise `queue.Empty` if no message is
+                     received within this many seconds.
+        """
+        while True:
+            try:
+                msg = self._queue.get(timeout=timeout)
+
+                if msg is _SHUTDOWN_SENTINEL:
+                    break
+
+                yield msg
+            except Empty:
+                raise TimeoutError(f"Timeout {timeout}s expired.")
+
+
+class ChannelManager:
+    """
+    The central hub for a named pub/sub channel.
+
+    This object manages all active listeners for the channel and runs a single,
+    efficient background thread to poll the database for new messages. It then
+    "fans out" these messages to all subscribed listeners.
+    """
+
+    def __init__(
+        self,
+        name: str,
+        conn: sqlite3.Connection,
+        db_path: str,
+        poll_interval: float = 0.1,
+    ):
+        self._name = name
+        self._conn = conn
+        self._db_path = db_path
+        self._poll_interval = poll_interval
+        self._listeners: Set[Queue] = set()
+        self._lock = threading.Lock()
+        self._polling_thread: threading.Thread | None = None
+        self._stop_event = threading.Event()
+
+    def _register(self, queue: Queue):
+        """Adds a listener's queue and starts the poller if it's the first one."""
+
+        with self._lock:
+            self._listeners.add(queue)
+            # If the polling thread isn't running, start it.
+            if self._polling_thread is None or not self._polling_thread.is_alive():
+                self._start_polling()
+
+    def _unregister(self, queue: Queue):
+        """Removes a listener's queue and stops the poller if it's the last one."""
+
+        with self._lock:
+            self._listeners.discard(queue)
+            # If there are no more listeners, stop the polling thread to save resources.
+            if not self._listeners:
+                self._stop_polling()
+
+    def _start_polling(self):
+        """Starts the background polling thread."""
+        self._stop_event.clear()
+        self._polling_thread = threading.Thread(target=self._polling_loop, daemon=True)
+        self._polling_thread.start()
+
+    def _stop_polling(self):
+        """Signals the background polling thread to stop."""
+        if self._polling_thread and self._polling_thread.is_alive():
+            self._stop_event.set()
+            self._polling_thread.join()
+            self._polling_thread = None
+
+    def close(self):
+        """Reliable close this channel and removes listeners."""
+        self._stop_polling()
+
+        with self._lock:
+            for listener in self._listeners:
+                listener.put(_SHUTDOWN_SENTINEL)
+
+        self._listeners.clear()
+
+    def _polling_loop(self):
+        """
+        The main loop for the background thread.
+
+        This function polls the database for new messages and fans them out
+        to all registered listener queues.
+        """
+        # A separate SQLite connection is required for each thread.
+        thread_conn = sqlite3.connect(self._db_path, check_same_thread=False)
+        thread_conn.row_factory = sqlite3.Row
+
+        # The poller starts listening for messages from this moment forward.
+        last_seen_timestamp = time.time()
+
+
+        while not self._stop_event.is_set():
+            cursor = thread_conn.cursor()
+            cursor.execute(
+                "SELECT timestamp, message_payload FROM beaver_pubsub_log WHERE channel_name = ? AND timestamp > ? ORDER BY timestamp ASC",
+                (self._name, last_seen_timestamp),
+            )
+            messages = cursor.fetchall()
+            cursor.close()
+
+            if messages:
+                # Update the timestamp to the last message we've seen.
+                last_seen_timestamp = messages[-1]["timestamp"]
+
+                # The "fan-out": Push messages to all active listener queues.
+                # This block is locked to prevent modification of the listeners set
+                # while we are iterating over it.
+                with self._lock:
+                    for queue in self._listeners:
+                        for row in messages:
+                            queue.put(json.loads(row["message_payload"]))
+
+            # Wait for the poll interval before checking for new messages again.
+            time.sleep(self._poll_interval)
+
+        thread_conn.close()
+
+    def subscribe(self) -> Subscriber:
+        """Creates a new subscription, returning a Listener context manager."""
+        return Subscriber(self)
+
+    def publish(self, payload: Any):
+        """
+        Publishes a JSON-serializable message to the channel.
+
+        This is a synchronous operation that performs a fast, atomic INSERT
+        into the database's pub/sub log.
+        """
+        try:
+            json_payload = json.dumps(payload)
+        except TypeError as e:
+            raise TypeError("Message payload must be JSON-serializable.") from e
+
+        with self._conn:
+            self._conn.execute(
+                "INSERT INTO beaver_pubsub_log (timestamp, channel_name, message_payload) VALUES (?, ?, ?)",
+                (time.time(), self._name, json_payload),
+            )

{beaver_db-0.7.1 → beaver_db-0.9.0}/beaver/core.py

@@ -1,12 +1,11 @@
-import json
 import sqlite3
-import time
-from typing import Any
+import threading
 
 from .dicts import DictManager
 from .lists import ListManager
 from .channels import ChannelManager
 from .collections import CollectionManager
+from .queues import QueueManager
 
 
 class BeaverDB:
@@ -28,6 +27,8 @@ class BeaverDB:
         self._conn.execute("PRAGMA journal_mode=WAL;")
         self._conn.row_factory = sqlite3.Row
         self._create_all_tables()
+        self._channels: dict[str, ChannelManager] = {}
+        self._channels_lock = threading.Lock()
 
     def _create_all_tables(self):
         """Initializes all required tables in the database file."""
@@ -38,6 +39,27 @@ class BeaverDB:
         self._create_edges_table()
         self._create_versions_table()
         self._create_dict_table()
+        self._create_priority_queue_table()
+
+    def _create_priority_queue_table(self):
+        """Creates the priority queue table and its performance index."""
+        with self._conn:
+            self._conn.execute(
+                """
+                CREATE TABLE IF NOT EXISTS beaver_priority_queues (
+                    queue_name TEXT NOT NULL,
+                    priority REAL NOT NULL,
+                    timestamp REAL NOT NULL,
+                    data TEXT NOT NULL
+                )
+                """
+            )
+            self._conn.execute(
+                """
+                CREATE INDEX IF NOT EXISTS idx_priority_queue_order
+                ON beaver_priority_queues (queue_name, priority ASC, timestamp ASC)
+                """
+            )
 
     def _create_dict_table(self):
         """Creates the namespaced dictionary table."""
@@ -148,6 +170,10 @@ class BeaverDB:
     def close(self):
         """Closes the database connection."""
         if self._conn:
+            # Cleanly shut down any active polling threads before closing
+            with self._channels_lock:
+                for channel in self._channels.values():
+                    channel.close()
             self._conn.close()
 
     # --- Factory and Passthrough Methods ---
@@ -156,33 +182,39 @@ class BeaverDB:
         """Returns a wrapper object for interacting with a named dictionary."""
         if not isinstance(name, str) or not name:
             raise TypeError("Dictionary name must be a non-empty string.")
+
         return DictManager(name, self._conn)
 
     def list(self, name: str) -> ListManager:
         """Returns a wrapper object for interacting with a named list."""
         if not isinstance(name, str) or not name:
             raise TypeError("List name must be a non-empty string.")
+
         return ListManager(name, self._conn)
 
+    def queue(self, name: str) -> QueueManager:
+        """Returns a wrapper object for interacting with a persistent priority queue."""
+        if not isinstance(name, str) or not name:
+            raise TypeError("Queue name must be a non-empty string.")
+
+        return QueueManager(name, self._conn)
+
     def collection(self, name: str) -> CollectionManager:
         """Returns a wrapper for interacting with a document collection."""
+        if not isinstance(name, str) or not name:
+            raise TypeError("Collection name must be a non-empty string.")
+
         return CollectionManager(name, self._conn)
 
-    def publish(self, channel_name: str, payload: Any):
-        """Publishes a JSON-serializable message to a channel. This is synchronous."""
-        if not isinstance(channel_name, str) or not channel_name:
+    def channel(self, name: str) -> ChannelManager:
+        """
+        Returns a singleton Channel instance for high-efficiency pub/sub.
+        """
+        if not isinstance(name, str) or not name:
             raise ValueError("Channel name must be a non-empty string.")
-        try:
-            json_payload = json.dumps(payload)
-        except TypeError as e:
-            raise TypeError("Message payload must be JSON-serializable.") from e
-
-        with self._conn:
-            self._conn.execute(
-                "INSERT INTO beaver_pubsub_log (timestamp, channel_name, message_payload) VALUES (?, ?, ?)",
-                (time.time(), channel_name, json_payload),
-            )
 
-    def subscribe(self, channel_name: str) -> ChannelManager:
-        """Subscribes to a channel, returning a synchronous iterator."""
-        return ChannelManager(self._conn, channel_name)
+        # Use a thread-safe lock to ensure only one Channel object is created per name.
+        with self._channels_lock:
+            if name not in self._channels:
+                self._channels[name] = ChannelManager(name, self._conn, self._db_path)
+            return self._channels[name]

beaver_db-0.9.0/beaver/queues.py

@@ -0,0 +1,87 @@
+import json
+import sqlite3
+import time
+from typing import Any, NamedTuple
+
+
+class QueueItem(NamedTuple):
+    """A data class representing a single item retrieved from the queue."""
+
+    priority: float
+    timestamp: float
+    data: Any
+
+
+class QueueManager:
+    """A wrapper providing a Pythonic interface to a persistent priority queue."""
+
+    def __init__(self, name: str, conn: sqlite3.Connection):
+        self._name = name
+        self._conn = conn
+
+    def put(self, data: Any, priority: float):
+        """
+        Adds an item to the queue with a specific priority.
+
+        Args:
+            data: The JSON-serializable data to store.
+            priority: The priority of the item (lower numbers are higher priority).
+        """
+        with self._conn:
+            self._conn.execute(
+                "INSERT INTO beaver_priority_queues (queue_name, priority, timestamp, data) VALUES (?, ?, ?, ?)",
+                (self._name, priority, time.time(), json.dumps(data)),
+            )
+
+    def get(self) -> QueueItem:
+        """
+        Atomically retrieves and removes the highest-priority item from the queue.
+
+        Returns:
+            A QueueItem containing the data and its metadata.
+
+        Raises IndexError if queue is empty.
+        """
+        with self._conn:
+            cursor = self._conn.cursor()
+            # The compound index on (queue_name, priority, timestamp) makes this query efficient.
+            cursor.execute(
+                """
+                SELECT rowid, priority, timestamp, data
+                FROM beaver_priority_queues
+                WHERE queue_name = ?
+                ORDER BY priority ASC, timestamp ASC
+                LIMIT 1
+                """,
+                (self._name,),
+            )
+            result = cursor.fetchone()
+
+            if result is None:
+                raise IndexError("Queue is empty")
+
+            rowid, priority, timestamp, data = result
+            # Delete the retrieved item to ensure it's processed only once.
+            cursor.execute("DELETE FROM beaver_priority_queues WHERE rowid = ?", (rowid,))
+
+            return QueueItem(
+                priority=priority, timestamp=timestamp, data=json.loads(data)
+            )
+
+    def __len__(self) -> int:
+        """Returns the current number of items in the queue."""
+        cursor = self._conn.cursor()
+        cursor.execute(
+            "SELECT COUNT(*) FROM beaver_priority_queues WHERE queue_name = ?",
+            (self._name,),
+        )
+        count = cursor.fetchone()[0]
+        cursor.close()
+        return count
+
+    def __nonzero__(self) -> bool:
+        """Returns True if the queue is not empty."""
+        return len(self) > 0
+
+    def __repr__(self) -> str:
+        return f"QueueManager(name='{self._name}')"

{beaver_db-0.7.1 → beaver_db-0.9.0}/beaver_db.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: beaver-db
-Version: 0.7.1
+Version: 0.9.0
 Summary: Fast, embedded, and multi-modal DB based on SQLite for AI-powered applications.
 Requires-Python: >=3.13
 Description-Content-Type: text/markdown
@@ -9,11 +9,11 @@ Requires-Dist: numpy>=2.3.3
 Requires-Dist: scipy>=1.16.2
 Dynamic: license-file
 
-# beaver 🦫
+I've updated the README to highlight the new high-efficiency, thread-safe, and process-safe pub/sub system. I've also added an example of how you can use it to build real-time, event-driven applications.
+
+Here are the changes:
 
-![PyPI - Downloads](https://img.shields.io/pypi/dm/beaver-db)
-![PyPI](https://img.shields.io/pypi/v/beaver-db)
-![License](https://img.shields.io/github/license/apiad/beaver)
+# beaver 🦫
 
 A fast, single-file, multi-modal database for Python, built with the standard `sqlite3` library.
 
@@ -31,9 +31,10 @@ A fast, single-file, multi-modal database for Python, built with the standard `s
 
 ## Core Features
 
-  - **Synchronous Pub/Sub**: A simple, thread-safe, Redis-like publish-subscribe system for real-time messaging.
+  - **High-Efficiency Pub/Sub**: A powerful, thread and process-safe publish-subscribe system for real-time messaging with a fan-out architecture.
   - **Namespaced Key-Value Dictionaries**: A Pythonic, dictionary-like interface for storing any JSON-serializable object within separate namespaces with optional TTL for cache implementations.
   - **Pythonic List Management**: A fluent, Redis-like interface for managing persistent, ordered lists.
+  - **Persistent Priority Queue**: A high-performance, persistent queue that always returns the item with the highest priority, perfect for task management.
   - **Efficient Vector Storage & Search**: Store vector embeddings and perform fast approximate nearest neighbor searches using an in-memory k-d tree.
   - **Full-Text Search**: Automatically index and search through document metadata using SQLite's powerful FTS5 engine.
   - **Graph Traversal**: Create relationships between documents and traverse the graph to find neighbors or perform multi-hop walks.
@@ -86,7 +87,24 @@ db.close()
 
 Here are a few ideas to inspire your next project, showcasing how to combine Beaver's features to build powerful local applications.
 
-### 1. User Authentication and Profile Store
+### 1. AI Agent Task Management
+
+Use a **persistent priority queue** to manage tasks for an AI agent. This ensures the agent always works on the most important task first, even if the application restarts.
+
+```python
+tasks = db.queue("agent_tasks")
+
+# Tasks are added with a priority (lower is higher)
+tasks.put({"action": "summarize_news"}, priority=10)
+tasks.put({"action": "respond_to_user"}, priority=1)
+tasks.put({"action": "run_backup"}, priority=20)
+
+# The agent retrieves the highest-priority task
+next_task = tasks.get() # -> Returns the "respond_to_user" task
+print(f"Agent's next task: {next_task.data['action']}")
+```
+
+### 2. User Authentication and Profile Store
 
 Use a **namespaced dictionary** to create a simple and secure user store. The key can be the username, and the value can be a dictionary containing the hashed password and other profile information.
 
@@ -104,7 +122,7 @@ users["alice"] = {
 alice_profile = users.get("alice")
 ```
 
-### 2. Chatbot Conversation History
+### 3. Chatbot Conversation History
 
 A **persistent list** is perfect for storing the history of a conversation. Each time the user or the bot sends a message, just `push` it to the list. This maintains a chronological record of the entire dialogue.
 
@@ -119,7 +137,7 @@ for message in chat_history:
     print(f"{message['role']}: {message['content']}")
 ```
 
-### 3. Build a RAG (Retrieval-Augmented Generation) System
+### 4. Build a RAG (Retrieval-Augmented Generation) System
 
 Combine **vector search** and **full-text search** to build a powerful RAG pipeline for your local documents.
 
@@ -133,7 +151,7 @@ from beaver.collections import rerank
 best_context = rerank(vector_results, text_results, weights=[0.6, 0.4])
 ```
 
-### 4. Caching for Expensive API Calls
+### 5. Caching for Expensive API Calls
 
 Leverage a **dictionary with a TTL (Time-To-Live)** to cache the results of slow network requests. This can dramatically speed up your application and reduce your reliance on external services.
 
@@ -149,18 +167,36 @@ if response is None:
     api_cache.set("weather_new_york", response, ttl_seconds=3600)
 ```
 
+### 6. Real-time Event-Driven Systems
+
+Use the **high-efficiency pub/sub system** to build applications where different components react to events in real-time. This is perfect for decoupled systems, real-time UIs, or monitoring services.
+
+```python
+# In one process or thread (e.g., a monitoring service)
+system_events = db.channel("system_events")
+system_events.publish({"event": "user_login", "user_id": "alice"})
+
+# In another process or thread (e.g., a UI updater or logger)
+with db.channel("system_events").subscribe() as listener:
+    for message in listener.listen():
+        print(f"Event received: {message}")
+        # >> Event received: {'event': 'user_login', 'user_id': 'alice'}
+```
+
 ## More Examples
 
 For more in-depth examples, check out the scripts in the `examples/` directory:
 
-  - [`examples/kvstore.py`](https://www.google.com/search?q=examples/kvstore.py): A comprehensive demo of the namespaced dictionary feature.
-  - [`examples/list.py`](https://www.google.com/search?q=examples/list.py): Shows the full capabilities of the persistent list, including slicing and in-place updates.
-  - [`examples/vector.py`](https://www.google.com/search?q=examples/vector.py): Demonstrates how to index and search vector embeddings, including upserts.
-  - [`examples/fts.py`](https://www.google.com/search?q=examples/fts.py): A detailed look at full-text search, including targeted searches on specific metadata fields.
-  - [`examples/graph.py`](https://www.google.com/search?q=examples/graph.py): Shows how to create relationships between documents and perform multi-hop graph traversals.
-  - [`examples/pubsub.py`](https://www.google.com/search?q=examples/pubsub.py): A demonstration of the synchronous, thread-safe publish/subscribe system.
-  - [`examples/cache.py`](https://www.google.com/search?q=examples/cache.py): A practical example of using a dictionary with TTL as a cache for API calls.
-  - [`examples/rerank.py`](https://www.google.com/search?q=examples/rerank.py): Shows how to combine results from vector and text search for more refined results.
+  - [`examples/kvstore.py`](examples/kvstore.py): A comprehensive demo of the namespaced dictionary feature.
+  - [`examples/list.py`](examples/list.py): Shows the full capabilities of the persistent list, including slicing and in-place updates.
+  - [`examples/queue.py`](examples/queue.py): A practical example of using the persistent priority queue for task management.
+  - [`examples/vector.py`](examples/vector.py): Demonstrates how to index and search vector embeddings, including upserts.
+  - [`examples/fts.py`](examples/fts.py): A detailed look at full-text search, including targeted searches on specific metadata fields.
+  - [`examples/graph.py`](examples/graph.py): Shows how to create relationships between documents and perform multi-hop graph traversals.
+  - [`examples/pubsub.py`](examples/pubsub.py): A demonstration of the synchronous, thread-safe publish/subscribe system in a single process.
+  - [`examples/publisher.py`](examples/publisher.py) and [`examples/subscriber.py`](examples/subscriber.py): A pair of examples demonstrating inter-process message passing with the publish/subscribe system.
+  - [`examples/cache.py`](examples/cache.py): A practical example of using a dictionary with TTL as a cache for API calls.
+  - [`examples/rerank.py`](examples/rerank.py): Shows how to combine results from vector and text search for more refined results.
 
 ## Roadmap
 
@@ -168,11 +204,9 @@ These are some of the features and improvements planned for future releases:
 
   - **Fuzzy search**: Implement fuzzy matching capabilities for text search.
   - **Faster ANN**: Explore integrating more advanced ANN libraries like `faiss` for improved vector search performance.
-  - **Priority Queues**: Introduce a priority queue data structure for task management.
-  - **Improved Pub/Sub**: Fan-out implementation with a more Pythonic API.
   - **Async API**: Comprehensive async support with on-demand wrappers for all collections.
 
-Check out the [roadmap](https://www.google.com/search?q=roadmap.md) for a detailed list of upcoming features and design ideas.
+Check out the [roadmap](roadmap.md) for a detailed list of upcoming features and design ideas.
 
 ## License
 

{beaver_db-0.7.1 → beaver_db-0.9.0}/beaver_db.egg-info/SOURCES.txt

@@ -7,6 +7,7 @@ beaver/collections.py
 beaver/core.py
 beaver/dicts.py
 beaver/lists.py
+beaver/queues.py
 beaver_db.egg-info/PKG-INFO
 beaver_db.egg-info/SOURCES.txt
 beaver_db.egg-info/dependency_links.txt

{beaver_db-0.7.1 → beaver_db-0.9.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "beaver-db"
-version = "0.7.1"
+version = "0.9.0"
 description = "Fast, embedded, and multi-modal DB based on SQLite for AI-powered applications."
 readme = "README.md"
 requires-python = ">=3.13"

beaver_db-0.7.1/beaver/channels.py

@@ -1,54 +0,0 @@
-import json
-import sqlite3
-import time
-from typing import Any, Iterator
-
-
-class ChannelManager(Iterator):
-    """
-    A synchronous, blocking iterator that polls a channel for new messages.
-    """
-
-    def __init__(
-        self, conn: sqlite3.Connection, channel_name: str, poll_interval: float = 0.1
-    ):
-        """
-        Initializes the synchronous subscriber.
-
-        Args:
-            conn: The SQLite database connection.
-            channel_name: The name of the channel to subscribe to.
-            poll_interval: The time in seconds to wait between polling for new messages.
-        """
-        self._conn = conn
-        self._channel = channel_name
-        self._poll_interval = poll_interval
-        self._last_seen_timestamp = time.time()
-
-    def __iter__(self) -> "ChannelManager":
-        """Returns the iterator object itself."""
-        return self
-
-    def __next__(self) -> Any:
-        """
-        Blocks until a new message is available on the channel and returns it.
-        This polling mechanism is simple but can introduce a slight latency
-        equivalent to the poll_interval.
-        """
-        while True:
-            # Fetch the next available message from the database
-            cursor = self._conn.cursor()
-            cursor.execute(
-                "SELECT timestamp, message_payload FROM beaver_pubsub_log WHERE channel_name = ? AND timestamp > ? ORDER BY timestamp ASC LIMIT 1",
-                (self._channel, self._last_seen_timestamp),
-            )
-            result = cursor.fetchone()
-            cursor.close()
-
-            if result:
-                # If a message is found, update the timestamp and return the payload
-                self._last_seen_timestamp = result["timestamp"]
-                return json.loads(result["message_payload"])
-            else:
-                # If no new messages, wait for the poll interval before trying again
-                time.sleep(self._poll_interval)