milvus-haystack 0.0.4__tar.gz → 0.0.6__tar.gz

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/.github/workflows/test.yml

@@ -6,6 +6,7 @@ on:
       - "src/**"
       - "tests/**"
       - pyproject.toml
+      - .github/workflows/test.yml
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
@@ -17,7 +18,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.9']
+        python-version: [ '3.9' ]
     
     steps:
       - name: Checkout code
@@ -34,5 +35,17 @@ jobs:
       - name: Lint
         run: hatch run lint:all
 
-      - name: Run tests 
+      - name: Run Milvus
+        run: |
+          wget https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh
+          bash standalone_embed.sh start
+          sleep 5
+          echo "\nMilvus server started!"
+
+      - name: Run tests
         run: hatch run cov
+
+      - name: Stop and Remove Milvus
+        run: |
+          bash standalone_embed.sh stop
+          echo "\nMilvus server stopped and removed!"

milvus_haystack-0.0.6/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.3
+Name: milvus-haystack
+Version: 0.0.6
+Project-URL: Documentation, https://github.com/milvus-io/milvus-haystack#readme
+Project-URL: Issues, https://github.com/milvus-io/milvus-haystack/issues
+Project-URL: Source, https://github.com/milvus-io/milvus-haystack
+Author-email: Zilliz <support@zilliz.com>
+License-Expression: Apache-2.0
+License-File: LICENSE.txt
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Requires-Python: >=3.7
+Requires-Dist: haystack-ai
+Requires-Dist: pymilvus
+Requires-Dist: typing-extensions
+Description-Content-Type: text/markdown
+
+# Milvus Document Store for Haystack
+
+[![PyPI - Version](https://img.shields.io/pypi/v/milvus-haystack.svg)](https://pypi.org/project/milvus-haystack)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/milvus-haystack.svg)](https://pypi.org/project/milvus-haystack)
+
+## Installation
+
+```shell
+pip install --upgrade pymilvus milvus-haystack
+```
+
+## Usage
+
+By default, if you install the latest version of pymilvus, you don't need to start the milvus service manually.
+Optionally, you
+can [start the Milvus service by docker](https://milvus.io/docs/install_standalone-docker.md#Start-Milvus).
+
+Use the `MilvusDocumentStore` in a Haystack pipeline as a quick start.
+
+```python
+from haystack import Document
+from milvus_haystack import MilvusDocumentStore
+
+document_store = MilvusDocumentStore(
+    # If you have installed the latest version of pymilvus with milvus lite, you can use a local path as the uri without starting the milvus service.
+    connection_args={"uri": "./milvus.db"},
+    # Or, if you have started the milvus standalone service by docker, you can use the specified uri to connect to the service.
+    # connection_args={"uri": "http://localhost:19530"},
+    drop_old=True,
+)
+documents = [Document(
+    content="A Foo Document",
+    meta={"page": "100", "chapter": "intro"},
+    embedding=[-10.0] * 128,
+)]
+document_store.write_documents(documents)
+print(document_store.count_documents())  # 1
+```
+
+## Dive deep usage
+
+Prepare an OpenAI API key and set it as an environment variable:
+
+```shell
+export OPENAI_API_KEY=<your_api_key>
+```
+
+Here are the ways to
+
+- Create the indexing Pipeline
+- Create the retrieval pipeline
+- Create the RAG pipeline
+
+### Create the indexing Pipeline and index some documents
+
+```python
+import glob
+import os
+
+from haystack import Pipeline
+from haystack.components.converters import MarkdownToDocument
+from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
+from haystack.components.preprocessors import DocumentSplitter
+from haystack.components.writers import DocumentWriter
+
+from milvus_haystack import MilvusDocumentStore
+from milvus_haystack.milvus_embedding_retriever import MilvusEmbeddingRetriever
+
+current_file_path = os.path.abspath(__file__)
+file_paths = [current_file_path]  # You can replace it with your own file paths.
+
+document_store = MilvusDocumentStore(
+    # If you have installed the latest version of pymilvus with milvus lite, you can use a local path as the uri without starting the milvus service.
+    connection_args={"uri": "./milvus.db"},
+    # Or, if you have started the milvus standalone service by docker, you can use the specified uri to connect to the service.
+    # connection_args={"uri": "http://localhost:19530"},
+    drop_old=True,
+)
+indexing_pipeline = Pipeline()
+indexing_pipeline.add_component("converter", MarkdownToDocument())
+indexing_pipeline.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=2))
+indexing_pipeline.add_component("embedder", OpenAIDocumentEmbedder())
+indexing_pipeline.add_component("writer", DocumentWriter(document_store))
+indexing_pipeline.connect("converter", "splitter")
+indexing_pipeline.connect("splitter", "embedder")
+indexing_pipeline.connect("embedder", "writer")
+indexing_pipeline.run({"converter": {"sources": file_paths}})
+
+print("Number of documents:", document_store.count_documents())
+```
+
+### Create the retrieval pipeline and try a query
+
+```python
+question = "How to set the service uri with milvus lite?"  # You can replace it with your own question. 
+
+retrieval_pipeline = Pipeline()
+retrieval_pipeline.add_component("embedder", OpenAITextEmbedder())
+retrieval_pipeline.add_component("retriever", MilvusEmbeddingRetriever(document_store=document_store, top_k=3))
+retrieval_pipeline.connect("embedder", "retriever")
+
+retrieval_results = retrieval_pipeline.run({"embedder": {"text": question}})
+
+for doc in retrieval_results["retriever"]["documents"]:
+    print(doc.content)
+    print("-" * 10)
+```
+
+### Create the RAG pipeline and try a query
+
+```python
+from haystack.utils import Secret
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+
+prompt_template = """Answer the following query based on the provided context. If the context does
+                     not include an answer, reply with 'I don't know'.\n
+                     Query: {{query}}
+                     Documents:
+                     {% for doc in documents %}
+                        {{ doc.content }}
+                     {% endfor %}
+                     Answer: 
+                  """
+
+rag_pipeline = Pipeline()
+rag_pipeline.add_component("text_embedder", OpenAITextEmbedder())
+rag_pipeline.add_component("retriever", MilvusEmbeddingRetriever(document_store=document_store, top_k=3))
+rag_pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
+rag_pipeline.add_component("generator", OpenAIGenerator(api_key=Secret.from_token(os.getenv("OPENAI_API_KEY")),
+                                                        generation_kwargs={"temperature": 0}))
+rag_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
+rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
+rag_pipeline.connect("prompt_builder", "generator")
+
+results = rag_pipeline.run(
+    {
+        "text_embedder": {"text": question},
+        "prompt_builder": {"query": question},
+    }
+)
+print('RAG answer:', results["generator"]["replies"][0])
+
+```
+
+## License
+
+`milvus-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license.

milvus_haystack-0.0.6/README.md

@@ -0,0 +1,148 @@
+# Milvus Document Store for Haystack
+
+[![PyPI - Version](https://img.shields.io/pypi/v/milvus-haystack.svg)](https://pypi.org/project/milvus-haystack)
+[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/milvus-haystack.svg)](https://pypi.org/project/milvus-haystack)
+
+## Installation
+
+```shell
+pip install --upgrade pymilvus milvus-haystack
+```
+
+## Usage
+
+By default, if you install the latest version of pymilvus, you don't need to start the milvus service manually.
+Optionally, you
+can [start the Milvus service by docker](https://milvus.io/docs/install_standalone-docker.md#Start-Milvus).
+
+Use the `MilvusDocumentStore` in a Haystack pipeline as a quick start.
+
+```python
+from haystack import Document
+from milvus_haystack import MilvusDocumentStore
+
+document_store = MilvusDocumentStore(
+    # If you have installed the latest version of pymilvus with milvus lite, you can use a local path as the uri without starting the milvus service.
+    connection_args={"uri": "./milvus.db"},
+    # Or, if you have started the milvus standalone service by docker, you can use the specified uri to connect to the service.
+    # connection_args={"uri": "http://localhost:19530"},
+    drop_old=True,
+)
+documents = [Document(
+    content="A Foo Document",
+    meta={"page": "100", "chapter": "intro"},
+    embedding=[-10.0] * 128,
+)]
+document_store.write_documents(documents)
+print(document_store.count_documents())  # 1
+```
+
+## Dive deep usage
+
+Prepare an OpenAI API key and set it as an environment variable:
+
+```shell
+export OPENAI_API_KEY=<your_api_key>
+```
+
+Here are the ways to
+
+- Create the indexing Pipeline
+- Create the retrieval pipeline
+- Create the RAG pipeline
+
+### Create the indexing Pipeline and index some documents
+
+```python
+import glob
+import os
+
+from haystack import Pipeline
+from haystack.components.converters import MarkdownToDocument
+from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder
+from haystack.components.preprocessors import DocumentSplitter
+from haystack.components.writers import DocumentWriter
+
+from milvus_haystack import MilvusDocumentStore
+from milvus_haystack.milvus_embedding_retriever import MilvusEmbeddingRetriever
+
+current_file_path = os.path.abspath(__file__)
+file_paths = [current_file_path]  # You can replace it with your own file paths.
+
+document_store = MilvusDocumentStore(
+    # If you have installed the latest version of pymilvus with milvus lite, you can use a local path as the uri without starting the milvus service.
+    connection_args={"uri": "./milvus.db"},
+    # Or, if you have started the milvus standalone service by docker, you can use the specified uri to connect to the service.
+    # connection_args={"uri": "http://localhost:19530"},
+    drop_old=True,
+)
+indexing_pipeline = Pipeline()
+indexing_pipeline.add_component("converter", MarkdownToDocument())
+indexing_pipeline.add_component("splitter", DocumentSplitter(split_by="sentence", split_length=2))
+indexing_pipeline.add_component("embedder", OpenAIDocumentEmbedder())
+indexing_pipeline.add_component("writer", DocumentWriter(document_store))
+indexing_pipeline.connect("converter", "splitter")
+indexing_pipeline.connect("splitter", "embedder")
+indexing_pipeline.connect("embedder", "writer")
+indexing_pipeline.run({"converter": {"sources": file_paths}})
+
+print("Number of documents:", document_store.count_documents())
+```
+
+### Create the retrieval pipeline and try a query
+
+```python
+question = "How to set the service uri with milvus lite?"  # You can replace it with your own question. 
+
+retrieval_pipeline = Pipeline()
+retrieval_pipeline.add_component("embedder", OpenAITextEmbedder())
+retrieval_pipeline.add_component("retriever", MilvusEmbeddingRetriever(document_store=document_store, top_k=3))
+retrieval_pipeline.connect("embedder", "retriever")
+
+retrieval_results = retrieval_pipeline.run({"embedder": {"text": question}})
+
+for doc in retrieval_results["retriever"]["documents"]:
+    print(doc.content)
+    print("-" * 10)
+```
+
+### Create the RAG pipeline and try a query
+
+```python
+from haystack.utils import Secret
+from haystack.components.builders import PromptBuilder
+from haystack.components.generators import OpenAIGenerator
+
+prompt_template = """Answer the following query based on the provided context. If the context does
+                     not include an answer, reply with 'I don't know'.\n
+                     Query: {{query}}
+                     Documents:
+                     {% for doc in documents %}
+                        {{ doc.content }}
+                     {% endfor %}
+                     Answer: 
+                  """
+
+rag_pipeline = Pipeline()
+rag_pipeline.add_component("text_embedder", OpenAITextEmbedder())
+rag_pipeline.add_component("retriever", MilvusEmbeddingRetriever(document_store=document_store, top_k=3))
+rag_pipeline.add_component("prompt_builder", PromptBuilder(template=prompt_template))
+rag_pipeline.add_component("generator", OpenAIGenerator(api_key=Secret.from_token(os.getenv("OPENAI_API_KEY")),
+                                                        generation_kwargs={"temperature": 0}))
+rag_pipeline.connect("text_embedder.embedding", "retriever.query_embedding")
+rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
+rag_pipeline.connect("prompt_builder", "generator")
+
+results = rag_pipeline.run(
+    {
+        "text_embedder": {"text": question},
+        "prompt_builder": {"query": question},
+    }
+)
+print('RAG answer:', results["generator"]["replies"][0])
+
+```
+
+## License
+
+`milvus-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license.

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/pyproject.toml

@@ -28,7 +28,6 @@ dependencies = [
   "haystack-ai",
   "typing_extensions",
   "pymilvus",
-  "milvus",
 ]
 
 [project.urls]
@@ -177,7 +176,6 @@ module = [
   "milvus_haystack.*",
   "pymilvus.*",
   "numpy",
-  "milvus",
   "pytest.*"
 ]
 ignore_missing_imports = true

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/src/milvus_haystack/__about__.py

@@ -1,4 +1,4 @@
 # SPDX-FileCopyrightText: 2023-present Tuana Celik <tuana.celik@deepset.ai>
 #
 # SPDX-License-Identifier: Apache-2.0
-__version__ = "0.0.4"
+__version__ = "0.0.6"

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/src/milvus_haystack/document_store.py

@@ -409,9 +409,7 @@ class MilvusDocumentStore:
             elif uri.startswith("http://"):
                 given_address = uri.split("http://")[1]
             else:
-                err_msg = "Invalid Milvus URI: %s", uri
-                logger.error(err_msg)
-                raise ValueError(err_msg)
+                given_address = uri  # Milvus lite
         elif address is not None:
             given_address = address
         else:

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/src/milvus_haystack/filters.py

@@ -60,7 +60,7 @@ def _assert_comparison_filter(filters: Dict[str, Any]):
     assert "field" in filters, "field must be specified in filters"  # noqa: S101
     assert "value" in filters, "value must be specified in filters"  # noqa: S101
     assert filters["operator"] in COMPARISON_OPERATORS, FilterError(  # noqa: S101
-        "operator must be one of: %s" % LOGIC_OPERATORS
+        f"operator must be one of: {LOGIC_OPERATORS}"
     )
 
 
@@ -85,5 +85,5 @@ def _parse_logic(filters: Dict[str, Any]) -> str:
 def _assert_logic_filter(filters: Dict[str, Any]):
     assert "operator" in filters, "operator must be specified in filters"  # noqa: S101
     assert "conditions" in filters, "conditions must be specified in filters"  # noqa: S101
-    assert filters["operator"] in LOGIC_OPERATORS, "operator must be one of: %s" % LOGIC_OPERATORS  # noqa: S101
+    assert filters["operator"] in LOGIC_OPERATORS, f"operator must be one of: {LOGIC_OPERATORS}"  # noqa: S101
     assert isinstance(filters["conditions"], list), "conditions must be a list"  # noqa: S101

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/tests/test_document_store.py

@@ -12,24 +12,6 @@ logger = logging.getLogger(__name__)
 
 
 class TestDocumentStore(CountDocumentsTest, WriteDocumentsTest, DeleteDocumentsTest):
-    from milvus import MilvusServer
-
-    milvus_server = MilvusServer()
-    milvus_server.set_base_dir("test_milvus_base")
-    milvus_server.listen_port = 19530
-    try:
-        milvus_server.stop()
-    except Exception as err:
-        logger.debug("Can not stop Milvus server. %s", err)
-    try:
-        milvus_server.cleanup()
-    except Exception as err:
-        logger.debug("Can not cleanup Milvus. %s", err)
-    try:
-        milvus_server.start()
-    except Exception as err:
-        logger.debug("Can not start Milvus server. %s", err)
-
     @pytest.fixture
     def document_store(self) -> MilvusDocumentStore:
         return MilvusDocumentStore(

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/tests/test_embedding_retriever.py

@@ -10,24 +10,6 @@ logger = logging.getLogger(__name__)
 
 
 class TestMilvusEmbeddingTests:
-    from milvus import MilvusServer
-
-    milvus_server = MilvusServer()
-    milvus_server.set_base_dir("test_milvus_base")
-    milvus_server.listen_port = 19530
-    try:
-        milvus_server.stop()
-    except Exception as err:
-        logger.debug("Can not stop Milvus server. %s", err)
-    try:
-        milvus_server.cleanup()
-    except Exception as err:
-        logger.debug("Can not cleanup Milvus. %s", err)
-    try:
-        milvus_server.start()
-    except Exception as err:
-        logger.debug("Can not start Milvus server. %s", err)
-
     @pytest.fixture
     def document_store(self) -> MilvusDocumentStore:
         return MilvusDocumentStore(

{milvus_haystack-0.0.4 → milvus_haystack-0.0.6}/tests/test_filters.py

@@ -12,24 +12,6 @@ logger = logging.getLogger(__name__)
 
 
 class TestMilvusFilters(FilterDocumentsTest):
-    from milvus import MilvusServer
-
-    milvus_server = MilvusServer()
-    milvus_server.set_base_dir("test_milvus_base")
-    milvus_server.listen_port = 19530
-    try:
-        milvus_server.stop()
-    except Exception as err:
-        logger.debug("Can not stop Milvus server. %s", err)
-    try:
-        milvus_server.cleanup()
-    except Exception as err:
-        logger.debug("Can not cleanup Milvus. %s", err)
-    try:
-        milvus_server.start()
-    except Exception as err:
-        logger.debug("Can not start Milvus server. %s", err)
-
     @pytest.fixture
     def filterable_docs(self) -> List[Document]:
         documents = []

milvus_haystack-0.0.4/PKG-INFO

@@ -1,56 +0,0 @@
-Metadata-Version: 2.1
-Name: milvus-haystack
-Version: 0.0.4
-Project-URL: Documentation, https://github.com/milvus-io/milvus-haystack#readme
-Project-URL: Issues, https://github.com/milvus-io/milvus-haystack/issues
-Project-URL: Source, https://github.com/milvus-io/milvus-haystack
-Author-email: Zilliz <support@zilliz.com>
-License-Expression: Apache-2.0
-License-File: LICENSE.txt
-Classifier: Development Status :: 4 - Beta
-Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3.7
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: Implementation :: CPython
-Classifier: Programming Language :: Python :: Implementation :: PyPy
-Requires-Python: >=3.7
-Requires-Dist: haystack-ai
-Requires-Dist: milvus
-Requires-Dist: pymilvus
-Requires-Dist: typing-extensions
-Description-Content-Type: text/markdown
-
-# Milvus Document Store for Haystack
-
-
-## Installation
-
-```console
-pip install -e milvus-haystack
-```
-
-## Usage
-First, to start up a Milvus service, follow the ['Start Milvus'](https://milvus.io/docs/install_standalone-docker.md#Start-Milvus) instructions in the documentation. 
-
-Then, to use the `MilvusDocumentStore` in a Haystack pipeline"
-
-```py
-from haystack import Document
-from milvus_haystack import MilvusDocumentStore
-
-document_store = MilvusDocumentStore()
-documents = [Document(
-    content="A Foo Document",
-    meta={"page": "100", "chapter": "intro"},
-    embedding=[-10.0] * 128,
-)]
-document_store.write_documents(documents)
-document_store.count_documents()  # 1
-```
-
-## License
-
-`milvus-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license.

milvus_haystack-0.0.4/README.md

@@ -1,31 +0,0 @@
-# Milvus Document Store for Haystack
-
-
-## Installation
-
-```console
-pip install -e milvus-haystack
-```
-
-## Usage
-First, to start up a Milvus service, follow the ['Start Milvus'](https://milvus.io/docs/install_standalone-docker.md#Start-Milvus) instructions in the documentation. 
-
-Then, to use the `MilvusDocumentStore` in a Haystack pipeline"
-
-```py
-from haystack import Document
-from milvus_haystack import MilvusDocumentStore
-
-document_store = MilvusDocumentStore()
-documents = [Document(
-    content="A Foo Document",
-    meta={"page": "100", "chapter": "intro"},
-    embedding=[-10.0] * 128,
-)]
-document_store.write_documents(documents)
-document_store.count_documents()  # 1
-```
-
-## License
-
-`milvus-haystack` is distributed under the terms of the [Apache-2.0](https://spdx.org/licenses/Apache-2.0.html) license.