haystack-pixeltable 0.1.0__tar.gz

haystack_pixeltable-0.1.0/LICENSE

@@ -0,0 +1,112 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work.
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work, that is intentionally submitted to the Licensor for
+      inclusion in the Work by the copyright owner or by an individual or
+      Legal Entity authorized to submit on behalf of the copyright owner.
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by the Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file.
+
+   5. Submission of Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor.
+
+   7. Disclaimer of Warranty.
+
+   8. Limitation of Liability.
+
+   9. Accepting Warranty or Additional Liability.
+
+   Copyright 2024-2026 Pixeltable, Inc.
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

haystack_pixeltable-0.1.0/PKG-INFO

@@ -0,0 +1,191 @@
+Metadata-Version: 2.4
+Name: haystack-pixeltable
+Version: 0.1.0
+Summary: Haystack Document Store and Retriever backed by Pixeltable multimodal data infrastructure.
+Author-email: Pixeltable <contact@pixeltable.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/pixeltable/haystack-pixeltable
+Project-URL: Repository, https://github.com/pixeltable/haystack-pixeltable
+Project-URL: Documentation, https://docs.pixeltable.com/
+Project-URL: Issues, https://github.com/pixeltable/haystack-pixeltable/issues
+Project-URL: Discord, https://discord.gg/QPyqFYx2UN
+Keywords: haystack,pixeltable,document-store,retriever,multimodal,embeddings,rag
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: haystack-ai>=2.6.0
+Requires-Dist: pixeltable>=0.2.28
+Requires-Dist: numpy
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Dynamic: license-file
+
+# pixeltable-haystack
+
+[![PyPI](https://img.shields.io/pypi/v/pixeltable-haystack)](https://pypi.org/project/pixeltable-haystack/)
+[![CI](https://github.com/pixeltable/haystack-pixeltable/actions/workflows/ci.yml/badge.svg)](https://github.com/pixeltable/haystack-pixeltable/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+
+Haystack Document Store and Retriever backed by [Pixeltable](https://pixeltable.com/) — persistent, versioned, multimodal data infrastructure for AI applications.
+
+## Installation
+
+```bash
+pip install pixeltable-haystack
+```
+
+## Quick Start
+
+### Document Store
+
+```python
+from haystack import Document
+from haystack_pixeltable import PixeltableDocumentStore
+
+store = PixeltableDocumentStore(
+    table_name="myproject.docs",
+    embedding_dimension=1536,
+)
+
+# Write documents
+store.write_documents([
+    Document(content="Pixeltable is multimodal data infrastructure.", embedding=[...]),
+    Document(content="Haystack is a framework for building RAG pipelines.", embedding=[...]),
+])
+
+# Filter documents
+results = store.filter_documents(
+    filters={"field": "meta.category", "operator": "==", "value": "docs"}
+)
+
+# Count
+print(store.count_documents())
+```
+
+### Retriever (Similarity Search)
+
+```python
+from haystack_pixeltable import PixeltableDocumentStore, PixeltableRetriever
+
+store = PixeltableDocumentStore(
+    table_name="myproject.docs",
+    embedding_dimension=1536,
+)
+retriever = PixeltableRetriever(document_store=store, top_k=5)
+
+# Search by embedding vector
+result = retriever.run(query_embedding=[0.1, 0.2, ...])
+for doc in result["documents"]:
+    print(f"{doc.content} (score: {doc.score:.3f})")
+```
+
+### In a Haystack Pipeline
+
+```python
+from haystack import Pipeline
+from haystack.components.embedders import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
+from haystack.components.writers import DocumentWriter
+from haystack_pixeltable import PixeltableDocumentStore, PixeltableRetriever
+
+store = PixeltableDocumentStore(
+    table_name="rag.knowledge",
+    embedding_dimension=384,
+)
+
+# Indexing pipeline
+indexing = Pipeline()
+indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
+indexing.add_component("writer", DocumentWriter(document_store=store))
+indexing.connect("embedder", "writer")
+
+# Query pipeline
+query = Pipeline()
+query.add_component("embedder", SentenceTransformersTextEmbedder())
+query.add_component("retriever", PixeltableRetriever(document_store=store, top_k=5))
+query.connect("embedder.embedding", "retriever.query_embedding")
+```
+
+## Filtering
+
+The Document Store supports the [Haystack filter specification](https://docs.haystack.deepset.ai/docs/metadata-filtering):
+
+```python
+# Simple equality
+store.filter_documents(filters={"field": "meta.category", "operator": "==", "value": "science"})
+
+# Comparison operators: ==, !=, >, >=, <, <=
+store.filter_documents(filters={"field": "meta.score", "operator": ">", "value": 0.8})
+
+# Compound AND
+store.filter_documents(filters={
+    "operator": "AND",
+    "conditions": [
+        {"field": "meta.category", "operator": "==", "value": "science"},
+        {"field": "meta.score", "operator": ">", "value": 0.5},
+    ],
+})
+
+# Compound OR
+store.filter_documents(filters={
+    "operator": "OR",
+    "conditions": [
+        {"field": "meta.source", "operator": "==", "value": "arxiv"},
+        {"field": "meta.source", "operator": "==", "value": "pubmed"},
+    ],
+})
+```
+
+## Pixeltable Escape Hatch: `.table`
+
+The `.table` property gives direct access to the underlying Pixeltable table for operations beyond the Haystack interface:
+
+```python
+store = PixeltableDocumentStore(table_name="myproject.docs", embedding_dimension=1536)
+t = store.table
+
+# Add a computed column
+import pixeltable.functions.openai as openai
+t.add_computed_column(
+    summary=openai.chat_completions(
+        messages=[{"role": "user", "content": t.content}],
+        model="gpt-4o-mini",
+    )
+)
+
+# Use arbitrary Pixeltable queries
+results = t.where(t.meta["category"] == "science").select(t.content, t.summary).collect()
+
+# Version history
+print(t.count(version=-1))  # row count at previous version
+```
+
+## Why Pixeltable?
+
+| Feature | Pixeltable | Chroma | Qdrant | pgvector |
+|---------|-----------|--------|--------|----------|
+| Persistent storage | Built-in | Opt-in | Opt-in | Built-in |
+| Computed columns | Native | No | No | No |
+| Version history | Native | No | No | No |
+| Multimodal types | Image, Video, Audio, Document | Text only | Text only | Text only |
+| Metadata filtering | JSON + SQL predicates | Limited | Rich | SQL |
+| Embedding auto-compute | Via computed columns | Manual | Manual | Manual |
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+ruff check . && ruff format --check .
+```
+
+## License
+
+Apache 2.0

haystack_pixeltable-0.1.0/README.md

@@ -0,0 +1,161 @@
+# pixeltable-haystack
+
+[![PyPI](https://img.shields.io/pypi/v/pixeltable-haystack)](https://pypi.org/project/pixeltable-haystack/)
+[![CI](https://github.com/pixeltable/haystack-pixeltable/actions/workflows/ci.yml/badge.svg)](https://github.com/pixeltable/haystack-pixeltable/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+
+Haystack Document Store and Retriever backed by [Pixeltable](https://pixeltable.com/) — persistent, versioned, multimodal data infrastructure for AI applications.
+
+## Installation
+
+```bash
+pip install pixeltable-haystack
+```
+
+## Quick Start
+
+### Document Store
+
+```python
+from haystack import Document
+from haystack_pixeltable import PixeltableDocumentStore
+
+store = PixeltableDocumentStore(
+    table_name="myproject.docs",
+    embedding_dimension=1536,
+)
+
+# Write documents
+store.write_documents([
+    Document(content="Pixeltable is multimodal data infrastructure.", embedding=[...]),
+    Document(content="Haystack is a framework for building RAG pipelines.", embedding=[...]),
+])
+
+# Filter documents
+results = store.filter_documents(
+    filters={"field": "meta.category", "operator": "==", "value": "docs"}
+)
+
+# Count
+print(store.count_documents())
+```
+
+### Retriever (Similarity Search)
+
+```python
+from haystack_pixeltable import PixeltableDocumentStore, PixeltableRetriever
+
+store = PixeltableDocumentStore(
+    table_name="myproject.docs",
+    embedding_dimension=1536,
+)
+retriever = PixeltableRetriever(document_store=store, top_k=5)
+
+# Search by embedding vector
+result = retriever.run(query_embedding=[0.1, 0.2, ...])
+for doc in result["documents"]:
+    print(f"{doc.content} (score: {doc.score:.3f})")
+```
+
+### In a Haystack Pipeline
+
+```python
+from haystack import Pipeline
+from haystack.components.embedders import SentenceTransformersTextEmbedder, SentenceTransformersDocumentEmbedder
+from haystack.components.writers import DocumentWriter
+from haystack_pixeltable import PixeltableDocumentStore, PixeltableRetriever
+
+store = PixeltableDocumentStore(
+    table_name="rag.knowledge",
+    embedding_dimension=384,
+)
+
+# Indexing pipeline
+indexing = Pipeline()
+indexing.add_component("embedder", SentenceTransformersDocumentEmbedder())
+indexing.add_component("writer", DocumentWriter(document_store=store))
+indexing.connect("embedder", "writer")
+
+# Query pipeline
+query = Pipeline()
+query.add_component("embedder", SentenceTransformersTextEmbedder())
+query.add_component("retriever", PixeltableRetriever(document_store=store, top_k=5))
+query.connect("embedder.embedding", "retriever.query_embedding")
+```
+
+## Filtering
+
+The Document Store supports the [Haystack filter specification](https://docs.haystack.deepset.ai/docs/metadata-filtering):
+
+```python
+# Simple equality
+store.filter_documents(filters={"field": "meta.category", "operator": "==", "value": "science"})
+
+# Comparison operators: ==, !=, >, >=, <, <=
+store.filter_documents(filters={"field": "meta.score", "operator": ">", "value": 0.8})
+
+# Compound AND
+store.filter_documents(filters={
+    "operator": "AND",
+    "conditions": [
+        {"field": "meta.category", "operator": "==", "value": "science"},
+        {"field": "meta.score", "operator": ">", "value": 0.5},
+    ],
+})
+
+# Compound OR
+store.filter_documents(filters={
+    "operator": "OR",
+    "conditions": [
+        {"field": "meta.source", "operator": "==", "value": "arxiv"},
+        {"field": "meta.source", "operator": "==", "value": "pubmed"},
+    ],
+})
+```
+
+## Pixeltable Escape Hatch: `.table`
+
+The `.table` property gives direct access to the underlying Pixeltable table for operations beyond the Haystack interface:
+
+```python
+store = PixeltableDocumentStore(table_name="myproject.docs", embedding_dimension=1536)
+t = store.table
+
+# Add a computed column
+import pixeltable.functions.openai as openai
+t.add_computed_column(
+    summary=openai.chat_completions(
+        messages=[{"role": "user", "content": t.content}],
+        model="gpt-4o-mini",
+    )
+)
+
+# Use arbitrary Pixeltable queries
+results = t.where(t.meta["category"] == "science").select(t.content, t.summary).collect()
+
+# Version history
+print(t.count(version=-1))  # row count at previous version
+```
+
+## Why Pixeltable?
+
+| Feature | Pixeltable | Chroma | Qdrant | pgvector |
+|---------|-----------|--------|--------|----------|
+| Persistent storage | Built-in | Opt-in | Opt-in | Built-in |
+| Computed columns | Native | No | No | No |
+| Version history | Native | No | No | No |
+| Multimodal types | Image, Video, Audio, Document | Text only | Text only | Text only |
+| Metadata filtering | JSON + SQL predicates | Limited | Rich | SQL |
+| Embedding auto-compute | Via computed columns | Manual | Manual | Manual |
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+ruff check . && ruff format --check .
+```
+
+## License
+
+Apache 2.0

haystack_pixeltable-0.1.0/haystack_pixeltable/__init__.py

@@ -0,0 +1,8 @@
+"""Haystack Document Store and Retriever backed by Pixeltable."""
+
+from haystack_pixeltable.document_store import PixeltableDocumentStore
+from haystack_pixeltable.retriever import PixeltableRetriever
+
+__all__ = ["PixeltableDocumentStore", "PixeltableRetriever"]
+
+__version__ = "0.1.0"