fse-haystack 0.1.0__tar.gz

fse_haystack-0.1.0/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog
+
+## 0.1.0 (unreleased)
+
+- Prepare the standalone fse integration. Unpublished.
+- Keep the package in the fse repository with isolated CI and a separately approved release.
+- Require fse 2.0.0 or newer for publication.
+- Add a real POTION retrieval example for end-to-end validation.

fse_haystack-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019-2026 Oliver Borchers
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fse_haystack-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+include CHANGELOG.md
+recursive-include tests *.py
+recursive-include examples *.py *.txt

fse_haystack-0.1.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: fse-haystack
+Version: 0.1.0
+Summary: fse embeddings for haystack-ai
+License-Expression: MIT
+Project-URL: Repository, https://github.com/oborchers/Fast_Sentence_Embeddings
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fse>=2.0.0
+Requires-Dist: haystack-ai>=3.4.1
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: pytest-socket>=0.7; extra == "test"
+Requires-Dist: pytest-asyncio>=0.24; extra == "test"
+Requires-Dist: numpy>=1.26.4; extra == "test"
+Dynamic: license-file
+
+# fse-haystack
+
+Prepared, unpublished Haystack integration for fse 2.0. Version 0.1.0 will be published only after fse 2.0.0 and separate maintainer approval.
+This package lives in `integrations/fse-haystack/` in the fse repository.
+
+This package depends on fse and `haystack-ai` only. Framework dependencies never enter
+fse core. This package has its own CI environment and release workflow.
+
+## Use
+
+After an fse 2.0 build and this package are installed:
+
+```python
+from haystack import Document
+from haystack_integrations.components.embedders.fse import (
+    FseDocumentEmbedder,
+    FseTextEmbedder,
+)
+
+query_embedder = FseTextEmbedder(model="minishlab/potion-base-8M", workers=1)
+query_embedder.warm_up()
+query = query_embedder.run(text="hello world")["embedding"]
+document_embedder = FseDocumentEmbedder(workers=1, batch_size=128)
+document_embedder.warm_up()
+documents = document_embedder.run(documents=[Document(content="hello world")])["documents"]
+```
+
+For an existing local model, pass `encoder=your_encoder`, built for example with
+`fse.Encoder.from_arrays`. No download occurs with an injected encoder.
+
+The `@component` classes expose `embedding` and `documents` output sockets.
+They implement `run_async` as well. Call `warm_up()` before standalone execution.
+Haystack pipelines call warm-up themselves. Document calls use batches of at most
+`batch_size` texts, preserving input order. Every document must have text content.
+The document embedder returns new documents and preserves IDs and metadata.
+It embeds content only, without metadata concatenation or implicit query prefixes.
+
+Pipeline serialization stores model and loader options, never the loaded encoder.
+An injected encoder cannot be serialized. Save it first, then use its local path
+as `model`. Serialization rejects an explicit Hub token to avoid saving credentials.
+Follow the [community integration guide](https://github.com/deepset-ai/haystack-integrations#readme)
+for a later listing. No listing is submitted in this preparation.
+
+## Framework baseline and guides
+
+The supported minimum for version 0.1.0 is `haystack-ai==3.4.1`, the current stable
+release checked on 2026-11-01 from [PyPI](https://pypi.org/project/haystack-ai/3.4.1/).
+This is our tested minimum and makes no compatibility claim for older framework releases.
+The runtime dependency allows later versions. Recheck compatibility before publication.
+
+The implementation follows the [official integration guide](https://docs.haystack.deepset.ai/docs/custom-components).
+
+## Options and behavior
+
+`load_kwargs` passes options unchanged to `Encoder.from_pretrained`.
+`load_kwargs={"normalize": False}` returns raw means for every supported model,
+including fse artifacts, Model2Vec and supported sentence-transformers models.
+`normalize="model"` (the default) preserves the model's normalization rule.
+For example, `{"revision": "<commit>", "local_files_only": True}` selects a Hub
+revision and offline loading after the files have been cached.
+Hub-only options (`revision`, `token`, `cache_dir`, `local_files_only`)
+are invalid for local paths. See the [fse loader contract](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/docs/architecture/hub.md).
+Do not pass `load_kwargs` with `encoder=`. Configure that encoder before injection.
+Use the Hub credential store rather than putting tokens in application configuration.
+
+`dtype="float32"` (default) or `"float16"` controls fse output precision.
+`workers=None` uses fse's worker policy. `workers=1` requests one worker.
+The adapter passes text and worker validation to fse without replacing its errors.
+Integer embeddings are unsupported because this interface requires float vectors.
+
+Every result converts a numpy array into Python float lists. This copies values
+and allocates Python objects, so it takes additional time and memory.
+It does not retain the array's compact float16 or float32 storage.
+Empty text and unknown tokens follow the loaded encoder's tokenizer rules.
+Async methods delegate encoding and list conversion to a thread.
+Construct models before serving async requests. Haystack also provides
+`warm_up_async()` so async pipelines can load outside the event loop.
+
+## Development
+
+Use standard Python 3.13. From this package directory in the fse checkout:
+
+```sh
+uv venv --python 3.13 .venv
+SETUPTOOLS_SCM_PRETEND_VERSION_FOR_FSE=2.0.0 \
+  uv pip install --python .venv/bin/python -e '../..' -e '.[test]'
+.venv/bin/python -m pytest -m 'not network'
+.venv/bin/python -m pytest -m network
+uv build --out-dir dist
+.venv/bin/python examples/retrieval.py
+```
+
+The version override labels the development checkout for dependency resolution.
+It does not create a tag, release or published package.
+
+The declared requirement is `fse>=2.0.0`. The version override tests the local
+unreleased checkout without weakening the published dependency.
+The network test downloads `minishlab/potion-base-8M`. Offline tests build a tiny
+local encoder with `Encoder.from_arrays` and compare vectors with fse directly.
+Tests disable Python network sockets by default and enable them for the marked
+network test. No model upload or paid service is needed.
+
+## CI and release
+
+The isolated CI job tests Python 3.11 with the minimum framework and Python 3.13
+with the latest framework. Network tests and the real-model retrieval example
+run only when requested locally. See [release setup](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/integrations/README.md#release-setup)
+for the package tag, approval environment and pending PyPI publisher.
+
+## AI assistance
+
+These package sources, tests and documentation were prepared with AI assistance
+using Codex. Validation uses offline vector parity, framework API tests and the
+real-model retrieval example. The maintainer remains responsible for review and maintenance.
+Publication and framework listing submissions require a separate maintainer decision.

fse_haystack-0.1.0/README.md

@@ -0,0 +1,115 @@
+# fse-haystack
+
+Prepared, unpublished Haystack integration for fse 2.0. Version 0.1.0 will be published only after fse 2.0.0 and separate maintainer approval.
+This package lives in `integrations/fse-haystack/` in the fse repository.
+
+This package depends on fse and `haystack-ai` only. Framework dependencies never enter
+fse core. This package has its own CI environment and release workflow.
+
+## Use
+
+After an fse 2.0 build and this package are installed:
+
+```python
+from haystack import Document
+from haystack_integrations.components.embedders.fse import (
+    FseDocumentEmbedder,
+    FseTextEmbedder,
+)
+
+query_embedder = FseTextEmbedder(model="minishlab/potion-base-8M", workers=1)
+query_embedder.warm_up()
+query = query_embedder.run(text="hello world")["embedding"]
+document_embedder = FseDocumentEmbedder(workers=1, batch_size=128)
+document_embedder.warm_up()
+documents = document_embedder.run(documents=[Document(content="hello world")])["documents"]
+```
+
+For an existing local model, pass `encoder=your_encoder`, built for example with
+`fse.Encoder.from_arrays`. No download occurs with an injected encoder.
+
+The `@component` classes expose `embedding` and `documents` output sockets.
+They implement `run_async` as well. Call `warm_up()` before standalone execution.
+Haystack pipelines call warm-up themselves. Document calls use batches of at most
+`batch_size` texts, preserving input order. Every document must have text content.
+The document embedder returns new documents and preserves IDs and metadata.
+It embeds content only, without metadata concatenation or implicit query prefixes.
+
+Pipeline serialization stores model and loader options, never the loaded encoder.
+An injected encoder cannot be serialized. Save it first, then use its local path
+as `model`. Serialization rejects an explicit Hub token to avoid saving credentials.
+Follow the [community integration guide](https://github.com/deepset-ai/haystack-integrations#readme)
+for a later listing. No listing is submitted in this preparation.
+
+## Framework baseline and guides
+
+The supported minimum for version 0.1.0 is `haystack-ai==3.4.1`, the current stable
+release checked on 2026-11-01 from [PyPI](https://pypi.org/project/haystack-ai/3.4.1/).
+This is our tested minimum and makes no compatibility claim for older framework releases.
+The runtime dependency allows later versions. Recheck compatibility before publication.
+
+The implementation follows the [official integration guide](https://docs.haystack.deepset.ai/docs/custom-components).
+
+## Options and behavior
+
+`load_kwargs` passes options unchanged to `Encoder.from_pretrained`.
+`load_kwargs={"normalize": False}` returns raw means for every supported model,
+including fse artifacts, Model2Vec and supported sentence-transformers models.
+`normalize="model"` (the default) preserves the model's normalization rule.
+For example, `{"revision": "<commit>", "local_files_only": True}` selects a Hub
+revision and offline loading after the files have been cached.
+Hub-only options (`revision`, `token`, `cache_dir`, `local_files_only`)
+are invalid for local paths. See the [fse loader contract](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/docs/architecture/hub.md).
+Do not pass `load_kwargs` with `encoder=`. Configure that encoder before injection.
+Use the Hub credential store rather than putting tokens in application configuration.
+
+`dtype="float32"` (default) or `"float16"` controls fse output precision.
+`workers=None` uses fse's worker policy. `workers=1` requests one worker.
+The adapter passes text and worker validation to fse without replacing its errors.
+Integer embeddings are unsupported because this interface requires float vectors.
+
+Every result converts a numpy array into Python float lists. This copies values
+and allocates Python objects, so it takes additional time and memory.
+It does not retain the array's compact float16 or float32 storage.
+Empty text and unknown tokens follow the loaded encoder's tokenizer rules.
+Async methods delegate encoding and list conversion to a thread.
+Construct models before serving async requests. Haystack also provides
+`warm_up_async()` so async pipelines can load outside the event loop.
+
+## Development
+
+Use standard Python 3.13. From this package directory in the fse checkout:
+
+```sh
+uv venv --python 3.13 .venv
+SETUPTOOLS_SCM_PRETEND_VERSION_FOR_FSE=2.0.0 \
+  uv pip install --python .venv/bin/python -e '../..' -e '.[test]'
+.venv/bin/python -m pytest -m 'not network'
+.venv/bin/python -m pytest -m network
+uv build --out-dir dist
+.venv/bin/python examples/retrieval.py
+```
+
+The version override labels the development checkout for dependency resolution.
+It does not create a tag, release or published package.
+
+The declared requirement is `fse>=2.0.0`. The version override tests the local
+unreleased checkout without weakening the published dependency.
+The network test downloads `minishlab/potion-base-8M`. Offline tests build a tiny
+local encoder with `Encoder.from_arrays` and compare vectors with fse directly.
+Tests disable Python network sockets by default and enable them for the marked
+network test. No model upload or paid service is needed.
+
+## CI and release
+
+The isolated CI job tests Python 3.11 with the minimum framework and Python 3.13
+with the latest framework. Network tests and the real-model retrieval example
+run only when requested locally. See [release setup](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/integrations/README.md#release-setup)
+for the package tag, approval environment and pending PyPI publisher.
+
+## AI assistance
+
+These package sources, tests and documentation were prepared with AI assistance
+using Codex. Validation uses offline vector parity, framework API tests and the
+real-model retrieval example. The maintainer remains responsible for review and maintenance.
+Publication and framework listing submissions require a separate maintainer decision.

fse_haystack-0.1.0/examples/retrieval-output.txt

@@ -0,0 +1,4 @@
+Recorded 2026-11-01; Python 3.13; local fse checkout labelled 2.0.0
+haystack-ai=3.4.1; model=minishlab/potion-base-8M
+documents=3; dimensions=256; top=Cats are small pets that purr and chase mice.
+cosine=0.796954; retrieval passed

fse_haystack-0.1.0/examples/retrieval.py

@@ -0,0 +1,33 @@
+"""Real POTION indexing and a connected Haystack retrieval pipeline."""
+
+import os
+from importlib.metadata import version
+
+from haystack import Document, Pipeline
+from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack_integrations.components.embedders.fse import FseDocumentEmbedder, FseTextEmbedder
+
+os.environ["HAYSTACK_TELEMETRY_ENABLED"] = "false"
+
+texts = [
+    "Cats are small pets that purr and chase mice.",
+    "Baking bread requires flour, water and yeast.",
+    "Planets orbit the sun in the solar system.",
+]
+embedder = FseDocumentEmbedder(model="minishlab/potion-base-8M", workers=1, batch_size=2)
+embedder.warm_up()
+documents = embedder.run([Document(content=text) for text in texts])["documents"]
+assert all(len(doc.embedding) == 256 for doc in documents)
+store = InMemoryDocumentStore(embedding_similarity_function="cosine")
+store.write_documents(documents)
+pipeline = Pipeline()
+pipeline.add_component("embed", FseTextEmbedder(model="minishlab/potion-base-8M", workers=1))
+pipeline.add_component("retrieve", InMemoryEmbeddingRetriever(document_store=store, top_k=1))
+pipeline.connect("embed.embedding", "retrieve.query_embedding")
+result = pipeline.run({"embed": {"text": "Which pets purr?"}})["retrieve"]["documents"]
+assert len(result) == 1
+assert result[0].content == texts[0]
+print(f"haystack-ai={version('haystack-ai')}; model=minishlab/potion-base-8M")
+print(f"documents={len(texts)}; dimensions=256; top={result[0].content}")
+print(f"cosine={result[0].score:.6f}; retrieval passed")

fse_haystack-0.1.0/pyproject.toml

@@ -0,0 +1,30 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "fse-haystack"
+version = "0.1.0"
+description = "fse embeddings for haystack-ai"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+dependencies = ["fse>=2.0.0", "haystack-ai>=3.4.1"]
+
+[project.urls]
+Repository = "https://github.com/oborchers/Fast_Sentence_Embeddings"
+
+[project.optional-dependencies]
+test = ["pytest>=8", "pytest-socket>=0.7", "pytest-asyncio>=0.24", "numpy>=1.26.4"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "--disable-socket --allow-unix-socket"
+asyncio_mode = "auto"
+asyncio_default_fixture_loop_scope = "function"
+filterwarnings = ["error"]
+markers = ["network: downloads a model from the Hugging Face Hub"]

fse_haystack-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

fse_haystack-0.1.0/src/fse_haystack.egg-info/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: fse-haystack
+Version: 0.1.0
+Summary: fse embeddings for haystack-ai
+License-Expression: MIT
+Project-URL: Repository, https://github.com/oborchers/Fast_Sentence_Embeddings
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fse>=2.0.0
+Requires-Dist: haystack-ai>=3.4.1
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: pytest-socket>=0.7; extra == "test"
+Requires-Dist: pytest-asyncio>=0.24; extra == "test"
+Requires-Dist: numpy>=1.26.4; extra == "test"
+Dynamic: license-file
+
+# fse-haystack
+
+Prepared, unpublished Haystack integration for fse 2.0. Version 0.1.0 will be published only after fse 2.0.0 and separate maintainer approval.
+This package lives in `integrations/fse-haystack/` in the fse repository.
+
+This package depends on fse and `haystack-ai` only. Framework dependencies never enter
+fse core. This package has its own CI environment and release workflow.
+
+## Use
+
+After an fse 2.0 build and this package are installed:
+
+```python
+from haystack import Document
+from haystack_integrations.components.embedders.fse import (
+    FseDocumentEmbedder,
+    FseTextEmbedder,
+)
+
+query_embedder = FseTextEmbedder(model="minishlab/potion-base-8M", workers=1)
+query_embedder.warm_up()
+query = query_embedder.run(text="hello world")["embedding"]
+document_embedder = FseDocumentEmbedder(workers=1, batch_size=128)
+document_embedder.warm_up()
+documents = document_embedder.run(documents=[Document(content="hello world")])["documents"]
+```
+
+For an existing local model, pass `encoder=your_encoder`, built for example with
+`fse.Encoder.from_arrays`. No download occurs with an injected encoder.
+
+The `@component` classes expose `embedding` and `documents` output sockets.
+They implement `run_async` as well. Call `warm_up()` before standalone execution.
+Haystack pipelines call warm-up themselves. Document calls use batches of at most
+`batch_size` texts, preserving input order. Every document must have text content.
+The document embedder returns new documents and preserves IDs and metadata.
+It embeds content only, without metadata concatenation or implicit query prefixes.
+
+Pipeline serialization stores model and loader options, never the loaded encoder.
+An injected encoder cannot be serialized. Save it first, then use its local path
+as `model`. Serialization rejects an explicit Hub token to avoid saving credentials.
+Follow the [community integration guide](https://github.com/deepset-ai/haystack-integrations#readme)
+for a later listing. No listing is submitted in this preparation.
+
+## Framework baseline and guides
+
+The supported minimum for version 0.1.0 is `haystack-ai==3.4.1`, the current stable
+release checked on 2026-11-01 from [PyPI](https://pypi.org/project/haystack-ai/3.4.1/).
+This is our tested minimum and makes no compatibility claim for older framework releases.
+The runtime dependency allows later versions. Recheck compatibility before publication.
+
+The implementation follows the [official integration guide](https://docs.haystack.deepset.ai/docs/custom-components).
+
+## Options and behavior
+
+`load_kwargs` passes options unchanged to `Encoder.from_pretrained`.
+`load_kwargs={"normalize": False}` returns raw means for every supported model,
+including fse artifacts, Model2Vec and supported sentence-transformers models.
+`normalize="model"` (the default) preserves the model's normalization rule.
+For example, `{"revision": "<commit>", "local_files_only": True}` selects a Hub
+revision and offline loading after the files have been cached.
+Hub-only options (`revision`, `token`, `cache_dir`, `local_files_only`)
+are invalid for local paths. See the [fse loader contract](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/docs/architecture/hub.md).
+Do not pass `load_kwargs` with `encoder=`. Configure that encoder before injection.
+Use the Hub credential store rather than putting tokens in application configuration.
+
+`dtype="float32"` (default) or `"float16"` controls fse output precision.
+`workers=None` uses fse's worker policy. `workers=1` requests one worker.
+The adapter passes text and worker validation to fse without replacing its errors.
+Integer embeddings are unsupported because this interface requires float vectors.
+
+Every result converts a numpy array into Python float lists. This copies values
+and allocates Python objects, so it takes additional time and memory.
+It does not retain the array's compact float16 or float32 storage.
+Empty text and unknown tokens follow the loaded encoder's tokenizer rules.
+Async methods delegate encoding and list conversion to a thread.
+Construct models before serving async requests. Haystack also provides
+`warm_up_async()` so async pipelines can load outside the event loop.
+
+## Development
+
+Use standard Python 3.13. From this package directory in the fse checkout:
+
+```sh
+uv venv --python 3.13 .venv
+SETUPTOOLS_SCM_PRETEND_VERSION_FOR_FSE=2.0.0 \
+  uv pip install --python .venv/bin/python -e '../..' -e '.[test]'
+.venv/bin/python -m pytest -m 'not network'
+.venv/bin/python -m pytest -m network
+uv build --out-dir dist
+.venv/bin/python examples/retrieval.py
+```
+
+The version override labels the development checkout for dependency resolution.
+It does not create a tag, release or published package.
+
+The declared requirement is `fse>=2.0.0`. The version override tests the local
+unreleased checkout without weakening the published dependency.
+The network test downloads `minishlab/potion-base-8M`. Offline tests build a tiny
+local encoder with `Encoder.from_arrays` and compare vectors with fse directly.
+Tests disable Python network sockets by default and enable them for the marked
+network test. No model upload or paid service is needed.
+
+## CI and release
+
+The isolated CI job tests Python 3.11 with the minimum framework and Python 3.13
+with the latest framework. Network tests and the real-model retrieval example
+run only when requested locally. See [release setup](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/integrations/README.md#release-setup)
+for the package tag, approval environment and pending PyPI publisher.
+
+## AI assistance
+
+These package sources, tests and documentation were prepared with AI assistance
+using Codex. Validation uses offline vector parity, framework API tests and the
+real-model retrieval example. The maintainer remains responsible for review and maintenance.
+Publication and framework listing submissions require a separate maintainer decision.

fse_haystack-0.1.0/src/fse_haystack.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+CHANGELOG.md
+LICENSE
+MANIFEST.in
+README.md
+pyproject.toml
+examples/retrieval-output.txt
+examples/retrieval.py
+src/fse_haystack.egg-info/PKG-INFO
+src/fse_haystack.egg-info/SOURCES.txt
+src/fse_haystack.egg-info/dependency_links.txt
+src/fse_haystack.egg-info/requires.txt
+src/fse_haystack.egg-info/top_level.txt
+src/haystack_integrations/components/embedders/fse/__init__.py
+tests/conftest.py
+tests/test_api.py

fse_haystack-0.1.0/src/fse_haystack.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

fse_haystack-0.1.0/src/fse_haystack.egg-info/requires.txt

@@ -0,0 +1,8 @@
+fse>=2.0.0
+haystack-ai>=3.4.1
+
+[test]
+pytest>=8
+pytest-socket>=0.7
+pytest-asyncio>=0.24
+numpy>=1.26.4

fse_haystack-0.1.0/src/fse_haystack.egg-info/top_level.txt

@@ -0,0 +1 @@
+haystack_integrations

fse_haystack-0.1.0/src/haystack_integrations/components/embedders/fse/__init__.py

@@ -0,0 +1,114 @@
+"""Haystack text and document embedders backed by fse."""
+
+from asyncio import to_thread
+from dataclasses import replace
+from pathlib import Path
+
+from haystack import Document, component, default_from_dict, default_to_dict
+
+from fse import Encoder
+from fse._validation import validate_arguments
+
+
+class _FseEmbedder:
+    @validate_arguments
+    def __init__(
+        self,
+        model: str = "minishlab/potion-base-8M",
+        *,
+        encoder: Encoder | None = None,
+        dtype: str = "float32",
+        workers: int | None = None,
+        batch_size: int = 128,
+        load_kwargs: dict | None = None,
+    ):
+        if dtype not in ("float32", "float16"):
+            raise ValueError("dtype must be float32 or float16 for float-list embeddings")
+        if type(batch_size) is not int or batch_size <= 0:
+            raise ValueError("batch_size must be a positive integer")
+        if encoder is not None and load_kwargs:
+            raise ValueError("load_kwargs cannot be used with an existing encoder")
+        self.model = model
+        self.dtype = dtype
+        self.workers = workers
+        self.batch_size = batch_size
+        self.load_kwargs = dict(load_kwargs or {})
+        self._encoder = encoder
+        self._injected = encoder is not None
+
+    def warm_up(self) -> None:
+        """Load once before running a pipeline. An injected encoder is already ready."""
+        if self._encoder is None:
+            self._encoder = Encoder.from_pretrained(self.model, **self.load_kwargs)
+
+    async def warm_up_async(self) -> None:
+        await to_thread(self.warm_up)
+
+    def _embed(self, texts: list[str]) -> list[list[float]]:
+        if self._encoder is None:
+            raise RuntimeError("Call warm_up() before run()")
+        rows = []
+        for start in range(0, len(texts), self.batch_size):
+            rows.extend(
+                self._encoder.encode(
+                    texts[start : start + self.batch_size], dtype=self.dtype, workers=self.workers
+                ).tolist()
+            )
+        return rows
+
+    def to_dict(self) -> dict:
+        """Serialize loader options. In-memory encoders must be saved separately."""
+        if self._injected:
+            raise ValueError("Save the encoder and use its local path before serialization")
+        if "token" in self.load_kwargs:
+            raise ValueError("Do not serialize Hub tokens. Use the Hub credential store")
+        return default_to_dict(
+            self,
+            model=self.model,
+            dtype=self.dtype,
+            workers=self.workers,
+            batch_size=self.batch_size,
+            load_kwargs={
+                k: str(v) if isinstance(v, Path) else v for k, v in self.load_kwargs.items()
+            },
+        )
+
+    @classmethod
+    def from_dict(cls, data: dict):
+        return default_from_dict(cls, data)
+
+
+@component
+class FseTextEmbedder(_FseEmbedder):
+    """Embed one query. Output uses Haystack's ``embedding`` socket."""
+
+    @component.output_types(embedding=list[float])
+    def run(self, text: str) -> dict:
+        return {"embedding": self._embed([text])[0]}
+
+    @component.output_types(embedding=list[float])
+    async def run_async(self, text: str) -> dict:
+        return await to_thread(self.run, text)
+
+
+@component
+class FseDocumentEmbedder(_FseEmbedder):
+    """Embed document content, preserving order and leaving inputs unchanged."""
+
+    @component.output_types(documents=list[Document])
+    def run(self, documents: list[Document]) -> dict:
+        if not isinstance(documents, list) or any(not isinstance(d, Document) for d in documents):
+            raise TypeError("documents must be a list of Haystack Documents")
+        texts = [doc.content for doc in documents]
+        if any(text is None for text in texts):
+            raise ValueError("Every document must have text content")
+        rows = self._embed(texts)
+        return {
+            "documents": [
+                replace(doc, embedding=row) for doc, row in zip(documents, rows, strict=True)
+            ]
+        }
+
+    @component.output_types(documents=list[Document])
+    async def run_async(self, documents: list[Document]) -> dict:
+        return await to_thread(self.run, documents)

fse_haystack-0.1.0/tests/conftest.py

@@ -0,0 +1,17 @@
+import os
+
+import numpy as np
+import pytest
+
+from fse import Encoder
+
+
+@pytest.fixture
+def encoder():
+    return Encoder.from_arrays(
+        np.array([[1, 0, 2], [0, 2, 1]], dtype=np.float32),
+        words=["hello", "world"],
+    )
+
+
+os.environ["HAYSTACK_TELEMETRY_ENABLED"] = "false"

fse_haystack-0.1.0/tests/test_api.py

@@ -0,0 +1,117 @@
+import runpy
+from pathlib import Path
+from threading import get_ident
+
+import pytest
+from haystack import Document, Pipeline
+from haystack_integrations.components.embedders.fse import FseDocumentEmbedder, FseTextEmbedder
+
+from fse import Encoder
+
+
+@pytest.mark.parametrize("dtype", ["float32", "float16"])
+def test_local_vectors(encoder, dtype):
+    model = FseDocumentEmbedder(encoder=encoder, dtype=dtype, workers=1, batch_size=2)
+    docs = [
+        Document(content=text, meta={"label": i})
+        for i, text in enumerate(["hello world", "", "unknown", "world", "hello"])
+    ]
+    rows = encoder.encode([doc.content for doc in docs], dtype=dtype, workers=1).tolist()
+    result = model.run(docs)["documents"]
+    assert [doc.embedding for doc in result] == rows
+    assert all(doc.embedding is None for doc in docs)
+    assert [doc.meta for doc in result] == [doc.meta for doc in docs]
+    assert [doc.id for doc in result] == [doc.id for doc in docs]
+    assert model.run([]) == {"documents": []}
+    assert FseTextEmbedder(encoder=encoder, dtype=dtype).run("hello")["embedding"] == rows[-1]
+
+
+def test_pipeline_and_serialization(encoder, monkeypatch, tmp_path):
+    calls = []
+    monkeypatch.setattr(
+        Encoder, "from_pretrained", lambda *a, **kw: calls.append((a, kw)) or encoder
+    )
+    pipeline = Pipeline()
+    pipeline.add_component(
+        "embed",
+        FseTextEmbedder("test/model", load_kwargs={"normalize": False, "cache_dir": tmp_path}),
+    )
+    restored = Pipeline.loads(pipeline.dumps())
+    assert (
+        restored.run({"embed": {"text": "hello"}})["embed"]["embedding"]
+        == encoder.encode(["hello"]).tolist()[0]
+    )
+    assert calls == [(("test/model",), {"normalize": False, "cache_dir": str(tmp_path)})]
+    restored.run({"embed": {"text": "world"}})
+    assert len(calls) == 1
+
+
+async def test_async_off_loop(encoder, monkeypatch):
+    threads = []
+    loop_thread = get_ident()
+    model = FseTextEmbedder(encoder=encoder)
+    sync = model.run
+
+    def record(text):
+        threads.append(get_ident())
+        return sync(text)
+
+    monkeypatch.setattr(model, "run", record)
+    result = await model.run_async("hello")
+    assert result["embedding"] == encoder.encode(["hello"]).tolist()[0]
+    assert threads[0] != loop_thread
+    docs = [Document(content="hello")]
+    assert (await FseDocumentEmbedder(encoder=encoder).run_async(docs))["documents"][
+        0
+    ].embedding == result["embedding"]
+
+
+def test_validation(encoder):
+    with pytest.raises(RuntimeError, match="warm_up"):
+        FseTextEmbedder().run("hello")
+    with pytest.raises(ValueError, match="Save the encoder"):
+        FseTextEmbedder(encoder=encoder).to_dict()
+    with pytest.raises(ValueError, match="tokens"):
+        FseTextEmbedder(load_kwargs={"token": "not-a-secret"}).to_dict()
+    with pytest.raises(ValueError, match="text content"):
+        FseDocumentEmbedder(encoder=encoder).run([Document()])
+    with pytest.raises(TypeError, match="Documents"):
+        FseDocumentEmbedder(encoder=encoder).run(["hello"])
+    for batch in [0, -1, True, 1.5]:
+        with pytest.raises(ValueError, match="batch_size"):
+            FseDocumentEmbedder(batch_size=batch)
+
+
+@pytest.mark.network
+@pytest.mark.enable_socket
+def test_potion_retrieval():
+    runpy.run_path(str(Path(__file__).parents[1] / "examples" / "retrieval.py"))
+
+
+def test_local_artifact(encoder, tmp_path):
+    path = tmp_path / "encoder.safetensors"
+    encoder.save(path)
+    model = FseTextEmbedder(model=str(path))
+    model.warm_up()
+    assert model.run("hello")["embedding"] == encoder.encode(["hello"]).tolist()[0]
+    pipeline = Pipeline()
+    pipeline.add_component("embed", model)
+    clone = Pipeline.loads(pipeline.dumps())
+    assert clone.run({"embed": {"text": "hello"}})["embed"] == model.run("hello")
+
+
+async def test_cold_async_pipeline_loads_off_loop(encoder, monkeypatch):
+    threads = []
+    loop_thread = get_ident()
+
+    def load(*_args, **_kwargs):
+        threads.append(get_ident())
+        return encoder
+
+    monkeypatch.setattr(Encoder, "from_pretrained", load)
+    pipeline = Pipeline()
+    pipeline.add_component("embed", FseTextEmbedder("local"))
+    result = await pipeline.run_async({"embed": {"text": "hello"}})
+    assert result["embed"]["embedding"] == encoder.encode(["hello"]).tolist()[0]
+    assert threads
+    assert all(thread != loop_thread for thread in threads)