citadeldb-langchain 2.0.0__tar.gz

citadeldb_langchain-2.0.0/.gitignore

@@ -0,0 +1,18 @@
+/target
+**/*.rs.bk
+*.swp
+*.swo
+*~
+.DS_Store
+site/public/
+site/static/wasm/*.wasm
+site/static/wasm/*.js
+/notes/
+__pycache__/
+*.py[cod]
+# maturin build output; the .pyd is caught by the line above only by accident.
+*.pdb
+.pytest_cache/
+.mypy_cache/
+/dist/
+packaging/*/dist/

citadeldb_langchain-2.0.0/PKG-INFO

@@ -0,0 +1,112 @@
+Metadata-Version: 2.5
+Name: citadeldb-langchain
+Version: 2.0.0
+Summary: LangChain vector store and chat history backed by Citadel: encrypted at rest, with deletes that destroy the key
+Project-URL: Homepage, https://citadeldb.dev
+Project-URL: Repository, https://github.com/yp3y5akh0v/citadel
+Author: Yuriy Peysakhov
+License-Expression: Apache-2.0
+Keywords: chat-history,encryption,langchain,memory,rag,vector-store
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Database
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: citadeldb<3,>=2.0
+Requires-Dist: langchain-core<2,>=0.3
+Provides-Extra: test
+Requires-Dist: langchain-tests>=0.3; extra == 'test'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'test'
+Requires-Dist: pytest>=8; extra == 'test'
+Description-Content-Type: text/markdown
+
+# citadeldb-langchain
+
+A [LangChain](https://github.com/langchain-ai/langchain) `VectorStore` and
+`BaseChatMessageHistory` backed by [Citadel](https://citadeldb.dev). Encrypted at rest,
+embedded in your process, and deletes that destroy the key, not just the row.
+
+```
+pip install citadeldb-langchain
+```
+
+## Vector store
+
+```python
+from langchain_openai import OpenAIEmbeddings
+from citadeldb_langchain import CitadelVectorStore
+
+store = CitadelVectorStore(OpenAIEmbeddings(), "corpus.cdl", key="your-passphrase")
+
+store.add_texts(["the deploy failed because the disk was full"], ids=["note-1"])
+store.similarity_search("why did the release break?", k=1)
+
+retriever = store.as_retriever(search_kwargs={"k": 4})
+```
+
+The width is read from your embedding model on construction, so nothing has to be
+configured to match it. Pass `dim=` to skip that probe.
+
+Adding an id that is already stored replaces it, so re-indexing a document does not
+duplicate it.
+
+### Deletes destroy the key
+
+Every document is sealed under its own key. Deleting destroys that key and then removes the
+row, so any ciphertext surviving elsewhere stays unreadable.
+
+```python
+store.delete(["note-1"])   # named ids
+store.clear()              # the whole corpus, deliberately
+```
+
+`delete()` with no ids is a no-op, matching `InMemoryVectorStore`. Emptying the store is
+`clear()`, because erasure cannot be undone.
+
+### Filters
+
+```python
+store.similarity_search("...", k=4, filter={"source": "handbook.pdf"})
+```
+
+The filter is evaluated inside the scan, so it narrows candidates before top-k rather than
+trimming results after it, and `k` is `k`: a filter matching only distant documents still
+returns them, however many others outrank them.
+
+## Chat history
+
+```python
+from citadeldb_langchain import CitadelChatMessageHistory
+
+history = CitadelChatMessageHistory("user-123", "chats.cdl", key="your-passphrase")
+history.add_user_message("remember my dog is called Mochi")
+history.messages
+```
+
+Messages round-trip through LangChain's own serialization, so tool calls, block content
+and `additional_kwargs` all survive. `clear()` destroys each message's key, so a cleared
+conversation is unreadable.
+
+Use it with `RunnableWithMessageHistory` the same way as any other history:
+
+```python
+from langchain_core.runnables.history import RunnableWithMessageHistory
+
+chain = RunnableWithMessageHistory(
+    runnable,                    # your chain
+    lambda session_id: CitadelChatMessageHistory(session_id, "chats.cdl", key="..."),
+    input_messages_key="input",
+    history_messages_key="history",
+)
+```
+
+## Notes
+
+Citadel is embedded and one process owns the file. A path already open on this thread,
+under the same passphrase, is shared, so the vector store and the chat history can sit on
+one encrypted database; construct them on the same thread.
+
+## License
+
+Apache-2.0

citadeldb_langchain-2.0.0/README.md

@@ -0,0 +1,89 @@
+# citadeldb-langchain
+
+A [LangChain](https://github.com/langchain-ai/langchain) `VectorStore` and
+`BaseChatMessageHistory` backed by [Citadel](https://citadeldb.dev). Encrypted at rest,
+embedded in your process, and deletes that destroy the key, not just the row.
+
+```
+pip install citadeldb-langchain
+```
+
+## Vector store
+
+```python
+from langchain_openai import OpenAIEmbeddings
+from citadeldb_langchain import CitadelVectorStore
+
+store = CitadelVectorStore(OpenAIEmbeddings(), "corpus.cdl", key="your-passphrase")
+
+store.add_texts(["the deploy failed because the disk was full"], ids=["note-1"])
+store.similarity_search("why did the release break?", k=1)
+
+retriever = store.as_retriever(search_kwargs={"k": 4})
+```
+
+The width is read from your embedding model on construction, so nothing has to be
+configured to match it. Pass `dim=` to skip that probe.
+
+Adding an id that is already stored replaces it, so re-indexing a document does not
+duplicate it.
+
+### Deletes destroy the key
+
+Every document is sealed under its own key. Deleting destroys that key and then removes the
+row, so any ciphertext surviving elsewhere stays unreadable.
+
+```python
+store.delete(["note-1"])   # named ids
+store.clear()              # the whole corpus, deliberately
+```
+
+`delete()` with no ids is a no-op, matching `InMemoryVectorStore`. Emptying the store is
+`clear()`, because erasure cannot be undone.
+
+### Filters
+
+```python
+store.similarity_search("...", k=4, filter={"source": "handbook.pdf"})
+```
+
+The filter is evaluated inside the scan, so it narrows candidates before top-k rather than
+trimming results after it, and `k` is `k`: a filter matching only distant documents still
+returns them, however many others outrank them.
+
+## Chat history
+
+```python
+from citadeldb_langchain import CitadelChatMessageHistory
+
+history = CitadelChatMessageHistory("user-123", "chats.cdl", key="your-passphrase")
+history.add_user_message("remember my dog is called Mochi")
+history.messages
+```
+
+Messages round-trip through LangChain's own serialization, so tool calls, block content
+and `additional_kwargs` all survive. `clear()` destroys each message's key, so a cleared
+conversation is unreadable.
+
+Use it with `RunnableWithMessageHistory` the same way as any other history:
+
+```python
+from langchain_core.runnables.history import RunnableWithMessageHistory
+
+chain = RunnableWithMessageHistory(
+    runnable,                    # your chain
+    lambda session_id: CitadelChatMessageHistory(session_id, "chats.cdl", key="..."),
+    input_messages_key="input",
+    history_messages_key="history",
+)
+```
+
+## Notes
+
+Citadel is embedded and one process owns the file. A path already open on this thread,
+under the same passphrase, is shared, so the vector store and the chat history can sit on
+one encrypted database; construct them on the same thread.
+
+## License
+
+Apache-2.0

citadeldb_langchain-2.0.0/pyproject.toml

@@ -0,0 +1,41 @@
+[build-system]
+requires = ["hatchling", "hatch-vcs"]
+build-backend = "hatchling.build"
+
+[project]
+name = "citadeldb-langchain"
+dynamic = ["version"]
+description = "LangChain vector store and chat history backed by Citadel: encrypted at rest, with deletes that destroy the key"
+readme = "README.md"
+requires-python = ">=3.10"
+license = "Apache-2.0"
+authors = [{ name = "Yuriy Peysakhov" }]
+keywords = ["langchain", "rag", "vector-store", "chat-history", "memory", "encryption"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Topic :: Database",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+# The precomputed vector needs the `embedding` field added in citadeldb 2.0.
+dependencies = ["citadeldb>=2.0,<3", "langchain-core>=0.3,<2"]
+
+[project.optional-dependencies]
+# langchain-tests carries the suite that test_conformance.py runs.
+test = ["pytest>=8", "pytest-asyncio>=0.23", "langchain-tests>=0.3"]
+
+[project.urls]
+Homepage = "https://citadeldb.dev"
+Repository = "https://github.com/yp3y5akh0v/citadel"
+
+# The version comes from the release tag, so there is nothing to bump.
+[tool.hatch.version]
+source = "vcs"
+raw-options = { root = "../..", tag_regex = '^v(?P<version>\d+\.\d+\.\d+)$' }
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/citadeldb_langchain"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"

citadeldb_langchain-2.0.0/src/citadeldb_langchain/__init__.py

@@ -0,0 +1,14 @@
+"""LangChain storage backed by Citadel, encrypted at rest."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+from .chat_history import CitadelChatMessageHistory
+from .vector_store import CitadelVectorStore
+
+__all__ = ["CitadelVectorStore", "CitadelChatMessageHistory", "__version__"]
+
+
+try:
+    __version__ = version("citadeldb-langchain")
+except PackageNotFoundError:  # running from a source tree, never installed
+    __version__ = "0+unknown"

citadeldb_langchain-2.0.0/src/citadeldb_langchain/chat_history.py

@@ -0,0 +1,136 @@
+"""LangChain chat message history over an encrypted Citadel region."""
+from __future__ import annotations
+
+import asyncio
+from typing import Any, Sequence
+
+import citadeldb
+from langchain_core.chat_history import BaseChatMessageHistory
+from langchain_core.messages import BaseMessage, message_to_dict, messages_from_dict
+
+KIND = "message"
+DEFAULT_PATH = "langchain_history.cdl"
+DEFAULT_REGION = "chat_history"
+PAGE = 10_000
+
+
+def _page(mem: Any, region: str, session_id: str) -> list[Any]:
+    """Page to the end: one fetch is bounded, and a partial clear must not look whole."""
+    out: list[Any] = []
+    after = None
+    while True:
+        got = mem.fetch(region, KIND, payload_filter={"sid": session_id}, limit=PAGE,
+                        after_id=after)
+        out.extend(got)
+        if len(got) < PAGE:
+            return out
+        after = got[-1].id
+
+
+def _messages(mem: Any, region: str, session_id: str) -> list[BaseMessage]:
+    hits = _page(mem, region, session_id)
+    return messages_from_dict([h.payload["msg"] for h in hits])
+
+
+def _add(
+    mem: Any, region: str, session_id: str, messages: Sequence[BaseMessage]
+) -> None:
+    atoms = [
+        {
+            "kind": KIND,
+            # The message content is what recall would match on.
+            "text": _searchable(m),
+            "payload": {"sid": session_id, "msg": message_to_dict(m)},
+        }
+        for m in messages
+    ]
+    if atoms:
+        # One batch draws one id range, so list order survives as id order.
+        mem.remember_batch(region, atoms)
+
+
+def _clear(mem: Any, region: str, session_id: str) -> int:
+    hits = _page(mem, region, session_id)
+    if not hits:
+        return 0
+    return mem.forget(region, [h.id for h in hits]).erased_count
+
+
+def _searchable(message: BaseMessage) -> str:
+    """Message text, or a rendering of it when the content is a block list."""
+    content = message.content
+    if isinstance(content, str) and content:
+        return content
+    if isinstance(content, list):
+        parts = [
+            b["text"]
+            for b in content
+            if isinstance(b, dict) and isinstance(b.get("text"), str) and b["text"]
+        ]
+        if parts:
+            return " ".join(parts)
+    return message.type
+
+
+class CitadelChatMessageHistory(BaseChatMessageHistory):
+    """A LangChain chat history backed by one encrypted Citadel region."""
+
+    def __init__(
+        self,
+        session_id: str,
+        path: str = DEFAULT_PATH,
+        key: str = "",
+        *,
+        region: str = DEFAULT_REGION,
+        embedder: Any | None = None,
+    ) -> None:
+        if not key:
+            raise ValueError("a passphrase is required: transcripts are the payload")
+        self.session_id = session_id
+        try:
+            self._db = citadeldb.connect(path, key=key, region_keys=True)
+        except citadeldb.OperationalError as e:
+            if "locked" not in str(e):
+                raise
+            raise RuntimeError(
+                f"{path} is open in another process. Citadel is embedded, so one "
+                f"process owns the file."
+            ) from e
+        self._mem = self._db.memory()
+        self._region = region
+        # Idempotent for a region of the same width, so a dim clash raises here.
+        self._mem.create_encrypted_region(
+            region, embedder or citadeldb.MockEmbedder(dim=64)
+        )
+
+    @property
+    def messages(self) -> list[BaseMessage]:
+        return _messages(self._mem, self._region, self.session_id)
+
+    def add_messages(self, messages: Sequence[BaseMessage]) -> None:
+        _add(self._mem, self._region, self.session_id, messages)
+
+    def clear(self) -> None:
+        _clear(self._mem, self._region, self.session_id)
+
+    # ---- async ------------------------------------------------------------
+    # The bindings are sync, so a worker thread keeps the event loop free.
+
+    async def aget_messages(self) -> list[BaseMessage]:
+        return await asyncio.to_thread(
+            _messages, self._mem, self._region, self.session_id
+        )
+
+    async def aadd_messages(self, messages: Sequence[BaseMessage]) -> None:
+        await asyncio.to_thread(
+            _add, self._mem, self._region, self.session_id, list(messages)
+        )
+
+    async def aclear(self) -> None:
+        await asyncio.to_thread(_clear, self._mem, self._region, self.session_id)
+
+    # ---- beyond the interface ---------------------------------------------
+
+    def forget(self) -> int:
+        """Destroy this session's messages, returning the number erased."""
+        return _clear(self._mem, self._region, self.session_id)