fse-haystack 0.1.0__py3-none-any.whl

fse_haystack-0.1.0.dist-info/METADATA

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: fse-haystack
+Version: 0.1.0
+Summary: fse embeddings for haystack-ai
+License-Expression: MIT
+Project-URL: Repository, https://github.com/oborchers/Fast_Sentence_Embeddings
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fse>=2.0.0
+Requires-Dist: haystack-ai>=3.4.1
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: pytest-socket>=0.7; extra == "test"
+Requires-Dist: pytest-asyncio>=0.24; extra == "test"
+Requires-Dist: numpy>=1.26.4; extra == "test"
+Dynamic: license-file
+
+# fse-haystack
+
+Prepared, unpublished Haystack integration for fse 2.0. Version 0.1.0 will be published only after fse 2.0.0 and separate maintainer approval.
+This package lives in `integrations/fse-haystack/` in the fse repository.
+
+This package depends on fse and `haystack-ai` only. Framework dependencies never enter
+fse core. This package has its own CI environment and release workflow.
+
+## Use
+
+After an fse 2.0 build and this package are installed:
+
+```python
+from haystack import Document
+from haystack_integrations.components.embedders.fse import (
+    FseDocumentEmbedder,
+    FseTextEmbedder,
+)
+
+query_embedder = FseTextEmbedder(model="minishlab/potion-base-8M", workers=1)
+query_embedder.warm_up()
+query = query_embedder.run(text="hello world")["embedding"]
+document_embedder = FseDocumentEmbedder(workers=1, batch_size=128)
+document_embedder.warm_up()
+documents = document_embedder.run(documents=[Document(content="hello world")])["documents"]
+```
+
+For an existing local model, pass `encoder=your_encoder`, built for example with
+`fse.Encoder.from_arrays`. No download occurs with an injected encoder.
+
+The `@component` classes expose `embedding` and `documents` output sockets.
+They implement `run_async` as well. Call `warm_up()` before standalone execution.
+Haystack pipelines call warm-up themselves. Document calls use batches of at most
+`batch_size` texts, preserving input order. Every document must have text content.
+The document embedder returns new documents and preserves IDs and metadata.
+It embeds content only, without metadata concatenation or implicit query prefixes.
+
+Pipeline serialization stores model and loader options, never the loaded encoder.
+An injected encoder cannot be serialized. Save it first, then use its local path
+as `model`. Serialization rejects an explicit Hub token to avoid saving credentials.
+Follow the [community integration guide](https://github.com/deepset-ai/haystack-integrations#readme)
+for a later listing. No listing is submitted in this preparation.
+
+## Framework baseline and guides
+
+The supported minimum for version 0.1.0 is `haystack-ai==3.4.1`, the current stable
+release checked on 2026-11-01 from [PyPI](https://pypi.org/project/haystack-ai/3.4.1/).
+This is our tested minimum and makes no compatibility claim for older framework releases.
+The runtime dependency allows later versions. Recheck compatibility before publication.
+
+The implementation follows the [official integration guide](https://docs.haystack.deepset.ai/docs/custom-components).
+
+## Options and behavior
+
+`load_kwargs` passes options unchanged to `Encoder.from_pretrained`.
+`load_kwargs={"normalize": False}` returns raw means for every supported model,
+including fse artifacts, Model2Vec and supported sentence-transformers models.
+`normalize="model"` (the default) preserves the model's normalization rule.
+For example, `{"revision": "<commit>", "local_files_only": True}` selects a Hub
+revision and offline loading after the files have been cached.
+Hub-only options (`revision`, `token`, `cache_dir`, `local_files_only`)
+are invalid for local paths. See the [fse loader contract](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/docs/architecture/hub.md).
+Do not pass `load_kwargs` with `encoder=`. Configure that encoder before injection.
+Use the Hub credential store rather than putting tokens in application configuration.
+
+`dtype="float32"` (default) or `"float16"` controls fse output precision.
+`workers=None` uses fse's worker policy. `workers=1` requests one worker.
+The adapter passes text and worker validation to fse without replacing its errors.
+Integer embeddings are unsupported because this interface requires float vectors.
+
+Every result converts a numpy array into Python float lists. This copies values
+and allocates Python objects, so it takes additional time and memory.
+It does not retain the array's compact float16 or float32 storage.
+Empty text and unknown tokens follow the loaded encoder's tokenizer rules.
+Async methods delegate encoding and list conversion to a thread.
+Construct models before serving async requests. Haystack also provides
+`warm_up_async()` so async pipelines can load outside the event loop.
+
+## Development
+
+Use standard Python 3.13. From this package directory in the fse checkout:
+
+```sh
+uv venv --python 3.13 .venv
+SETUPTOOLS_SCM_PRETEND_VERSION_FOR_FSE=2.0.0 \
+  uv pip install --python .venv/bin/python -e '../..' -e '.[test]'
+.venv/bin/python -m pytest -m 'not network'
+.venv/bin/python -m pytest -m network
+uv build --out-dir dist
+.venv/bin/python examples/retrieval.py
+```
+
+The version override labels the development checkout for dependency resolution.
+It does not create a tag, release or published package.
+
+The declared requirement is `fse>=2.0.0`. The version override tests the local
+unreleased checkout without weakening the published dependency.
+The network test downloads `minishlab/potion-base-8M`. Offline tests build a tiny
+local encoder with `Encoder.from_arrays` and compare vectors with fse directly.
+Tests disable Python network sockets by default and enable them for the marked
+network test. No model upload or paid service is needed.
+
+## CI and release
+
+The isolated CI job tests Python 3.11 with the minimum framework and Python 3.13
+with the latest framework. Network tests and the real-model retrieval example
+run only when requested locally. See [release setup](https://github.com/oborchers/Fast_Sentence_Embeddings/blob/master/integrations/README.md#release-setup)
+for the package tag, approval environment and pending PyPI publisher.
+
+## AI assistance
+
+These package sources, tests and documentation were prepared with AI assistance
+using Codex. Validation uses offline vector parity, framework API tests and the
+real-model retrieval example. The maintainer remains responsible for review and maintenance.
+Publication and framework listing submissions require a separate maintainer decision.

fse_haystack-0.1.0.dist-info/RECORD

@@ -0,0 +1,6 @@
+fse_haystack-0.1.0.dist-info/licenses/LICENSE,sha256=bBiZDEcLSm07ahZaeojdtC7C1zEnIgKofEgUPekS7Bk,1077
+haystack_integrations/components/embedders/fse/__init__.py,sha256=5ONU-2B71-LGo1nk0Z-eQUzBm1TMDvVV1_vsZdLInfs,4217
+fse_haystack-0.1.0.dist-info/METADATA,sha256=BMe2rH2rLJ9dvXG8Z470A7duv9oHc5FIWMOYvKQGkWA,6513
+fse_haystack-0.1.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+fse_haystack-0.1.0.dist-info/top_level.txt,sha256=xEV0YMHti-yzzXgkLgSzsILiqsuC91-4ShNpDgsKNRI,22
+fse_haystack-0.1.0.dist-info/RECORD,,

fse_haystack-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

fse_haystack-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019-2026 Oliver Borchers
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fse_haystack-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+haystack_integrations

haystack_integrations/components/embedders/fse/__init__.py

@@ -0,0 +1,114 @@
+"""Haystack text and document embedders backed by fse."""
+
+from asyncio import to_thread
+from dataclasses import replace
+from pathlib import Path
+
+from haystack import Document, component, default_from_dict, default_to_dict
+
+from fse import Encoder
+from fse._validation import validate_arguments
+
+
+class _FseEmbedder:
+    @validate_arguments
+    def __init__(
+        self,
+        model: str = "minishlab/potion-base-8M",
+        *,
+        encoder: Encoder | None = None,
+        dtype: str = "float32",
+        workers: int | None = None,
+        batch_size: int = 128,
+        load_kwargs: dict | None = None,
+    ):
+        if dtype not in ("float32", "float16"):
+            raise ValueError("dtype must be float32 or float16 for float-list embeddings")
+        if type(batch_size) is not int or batch_size <= 0:
+            raise ValueError("batch_size must be a positive integer")
+        if encoder is not None and load_kwargs:
+            raise ValueError("load_kwargs cannot be used with an existing encoder")
+        self.model = model
+        self.dtype = dtype
+        self.workers = workers
+        self.batch_size = batch_size
+        self.load_kwargs = dict(load_kwargs or {})
+        self._encoder = encoder
+        self._injected = encoder is not None
+
+    def warm_up(self) -> None:
+        """Load once before running a pipeline. An injected encoder is already ready."""
+        if self._encoder is None:
+            self._encoder = Encoder.from_pretrained(self.model, **self.load_kwargs)
+
+    async def warm_up_async(self) -> None:
+        await to_thread(self.warm_up)
+
+    def _embed(self, texts: list[str]) -> list[list[float]]:
+        if self._encoder is None:
+            raise RuntimeError("Call warm_up() before run()")
+        rows = []
+        for start in range(0, len(texts), self.batch_size):
+            rows.extend(
+                self._encoder.encode(
+                    texts[start : start + self.batch_size], dtype=self.dtype, workers=self.workers
+                ).tolist()
+            )
+        return rows
+
+    def to_dict(self) -> dict:
+        """Serialize loader options. In-memory encoders must be saved separately."""
+        if self._injected:
+            raise ValueError("Save the encoder and use its local path before serialization")
+        if "token" in self.load_kwargs:
+            raise ValueError("Do not serialize Hub tokens. Use the Hub credential store")
+        return default_to_dict(
+            self,
+            model=self.model,
+            dtype=self.dtype,
+            workers=self.workers,
+            batch_size=self.batch_size,
+            load_kwargs={
+                k: str(v) if isinstance(v, Path) else v for k, v in self.load_kwargs.items()
+            },
+        )
+
+    @classmethod
+    def from_dict(cls, data: dict):
+        return default_from_dict(cls, data)
+
+
+@component
+class FseTextEmbedder(_FseEmbedder):
+    """Embed one query. Output uses Haystack's ``embedding`` socket."""
+
+    @component.output_types(embedding=list[float])
+    def run(self, text: str) -> dict:
+        return {"embedding": self._embed([text])[0]}
+
+    @component.output_types(embedding=list[float])
+    async def run_async(self, text: str) -> dict:
+        return await to_thread(self.run, text)
+
+
+@component
+class FseDocumentEmbedder(_FseEmbedder):
+    """Embed document content, preserving order and leaving inputs unchanged."""
+
+    @component.output_types(documents=list[Document])
+    def run(self, documents: list[Document]) -> dict:
+        if not isinstance(documents, list) or any(not isinstance(d, Document) for d in documents):
+            raise TypeError("documents must be a list of Haystack Documents")
+        texts = [doc.content for doc in documents]
+        if any(text is None for text in texts):
+            raise ValueError("Every document must have text content")
+        rows = self._embed(texts)
+        return {
+            "documents": [
+                replace(doc, embedding=row) for doc, row in zip(documents, rows, strict=True)
+            ]
+        }
+
+    @component.output_types(documents=list[Document])
+    async def run_async(self, documents: list[Document]) -> dict:
+        return await to_thread(self.run, documents)