haystack-ai 2.30.2__tar.gz → 2.31.0__tar.gz

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: haystack-ai
-Version: 2.30.2
+Version: 2.31.0
 Summary: LLM framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data.
 Project-URL: CI: GitHub, https://github.com/deepset-ai/haystack/actions
 Project-URL: Docs: RTD, https://haystack.deepset.ai/overview/intro
@@ -53,7 +53,7 @@ Description-Content-Type: text/markdown
 | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | CI/CD   | [![Tests](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml) [![types - Mypy](https://img.shields.io/badge/types-Mypy-blue.svg)](https://github.com/python/mypy) [![Coverage badge](https://raw.githubusercontent.com/deepset-ai/haystack/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/deepset-ai/haystack/blob/python-coverage-comment-action-data/htmlcov/index.html) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) |
 | Docs    | [![Website](https://img.shields.io/website?label=documentation&up_message=online&url=https%3A%2F%2Fdocs.haystack.deepset.ai)](https://docs.haystack.deepset.ai)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
-| Package | [![PyPI](https://img.shields.io/pypi/v/haystack-ai)](https://pypi.org/project/haystack-ai/) ![PyPI - Downloads](https://img.shields.io/pypi/dm/haystack-ai?color=blue&logo=pypi&logoColor=gold) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/haystack-ai?logo=python&logoColor=gold) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/haystack-ai.svg)](https://anaconda.org/conda-forge/haystack-ai) [![GitHub](https://img.shields.io/github/license/deepset-ai/haystack?color=blue)](LICENSE) [![License Compliance](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13067/badge)](https://www.bestpractices.dev/projects/13067)|
+| Package | [![PyPI](https://img.shields.io/pypi/v/haystack-ai)](https://pypi.org/project/haystack-ai/) ![PyPI - Downloads](https://img.shields.io/pypi/dm/haystack-ai?color=blue&logo=pypi&logoColor=gold) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/haystack-ai?logo=python&logoColor=gold) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/haystack-ai.svg)](https://anaconda.org/conda-forge/haystack-ai) [![GitHub](https://img.shields.io/github/license/deepset-ai/haystack?color=blue)](LICENSE) [![License Compliance](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml) [![HVTrust](https://hvtracker.net/badge/haystack.svg)](https://hvtracker.net/agents/haystack/) [![Evidence Grade](https://hvtracker.net/badge/haystack-grade.svg)](https://hvtracker.net/agents/haystack/) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13067/badge)](https://www.bestpractices.dev/projects/13067)|
 | Meta    | [![Discord](https://img.shields.io/discord/993534733298450452?logo=discord)](https://discord.com/invite/qZxjM4bAHU) [![Twitter Follow](https://img.shields.io/twitter/follow/haystack_ai)](https://twitter.com/haystack_ai)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 </div>
 

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/README.md

@@ -5,7 +5,7 @@
 | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | CI/CD   | [![Tests](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/tests.yml) [![types - Mypy](https://img.shields.io/badge/types-Mypy-blue.svg)](https://github.com/python/mypy) [![Coverage badge](https://raw.githubusercontent.com/deepset-ai/haystack/python-coverage-comment-action-data/badge.svg)](https://htmlpreview.github.io/?https://github.com/deepset-ai/haystack/blob/python-coverage-comment-action-data/htmlcov/index.html) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) |
 | Docs    | [![Website](https://img.shields.io/website?label=documentation&up_message=online&url=https%3A%2F%2Fdocs.haystack.deepset.ai)](https://docs.haystack.deepset.ai)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
-| Package | [![PyPI](https://img.shields.io/pypi/v/haystack-ai)](https://pypi.org/project/haystack-ai/) ![PyPI - Downloads](https://img.shields.io/pypi/dm/haystack-ai?color=blue&logo=pypi&logoColor=gold) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/haystack-ai?logo=python&logoColor=gold) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/haystack-ai.svg)](https://anaconda.org/conda-forge/haystack-ai) [![GitHub](https://img.shields.io/github/license/deepset-ai/haystack?color=blue)](LICENSE) [![License Compliance](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13067/badge)](https://www.bestpractices.dev/projects/13067)|
+| Package | [![PyPI](https://img.shields.io/pypi/v/haystack-ai)](https://pypi.org/project/haystack-ai/) ![PyPI - Downloads](https://img.shields.io/pypi/dm/haystack-ai?color=blue&logo=pypi&logoColor=gold) ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/haystack-ai?logo=python&logoColor=gold) [![Conda Version](https://img.shields.io/conda/vn/conda-forge/haystack-ai.svg)](https://anaconda.org/conda-forge/haystack-ai) [![GitHub](https://img.shields.io/github/license/deepset-ai/haystack?color=blue)](LICENSE) [![License Compliance](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml/badge.svg)](https://github.com/deepset-ai/haystack/actions/workflows/license_compliance.yml) [![HVTrust](https://hvtracker.net/badge/haystack.svg)](https://hvtracker.net/agents/haystack/) [![Evidence Grade](https://hvtracker.net/badge/haystack-grade.svg)](https://hvtracker.net/agents/haystack/) [![OpenSSF Best Practices](https://www.bestpractices.dev/projects/13067/badge)](https://www.bestpractices.dev/projects/13067)|
 | Meta    | [![Discord](https://img.shields.io/discord/993534733298450452?logo=discord)](https://discord.com/invite/qZxjM4bAHU) [![Twitter Follow](https://img.shields.io/twitter/follow/haystack_ai)](https://twitter.com/haystack_ai)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 </div>
 

haystack_ai-2.31.0/VERSION.txt

@@ -0,0 +1 @@
+2.31.0

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/audio/whisper_local.py

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import tempfile
+import warnings
 from pathlib import Path
 from typing import Any, Literal, get_args
 
@@ -40,12 +41,16 @@ class LocalWhisperTranscriber:
     [GitHub repository](https://github.com/openai/whisper).
 
     ### Usage example
-    <!-- test-ignore -->
+
     ```python
     from haystack.components.audio import LocalWhisperTranscriber
 
     whisper = LocalWhisperTranscriber(model="small")
     transcription = whisper.run(sources=["test/test_files/audio/answer.wav"])
+    print(transcription)
+
+    # >> {'documents': [Document(id=dae7051417caaf19304a4ef2ec845e981abe1efd4c8e6ee7ffb25867f165c411,
+    # >> content: ' Answer.', meta: {'audio_file': PosixPath('test/test_files/audio/answer.wav'), 'language': 'en'})]}
     ```
     """
 
@@ -66,6 +71,14 @@ class LocalWhisperTranscriber:
         :param device:
             The device for loading the model. If `None`, automatically selects the default device.
         """
+        warnings.warn(
+            "`LocalWhisperTranscriber` will be removed from Haystack in version 3.0, as it is moving to "
+            "the `whisper-haystack` package. To continue using it, install that package with "
+            "`pip install whisper-haystack` and update your import to "
+            "`from haystack_integrations.components.audio.whisper import LocalWhisperTranscriber`.",
+            FutureWarning,
+            stacklevel=2,
+        )
         whisper_import.check()
         if model not in get_args(WhisperLocalModel):
             raise ValueError(

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/audio/whisper_remote.py

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import io
+import warnings
 from pathlib import Path
 from typing import Any
 
@@ -82,6 +83,14 @@ class RemoteWhisperTranscriber:
             uses log probability to automatically increase the
             temperature until certain thresholds are hit.
         """
+        warnings.warn(
+            "`RemoteWhisperTranscriber` will be removed from Haystack in version 3.0, as it is moving to "
+            "the `whisper-haystack` package. To continue using it, install that package with "
+            "`pip install whisper-haystack` and update your import to "
+            "`from haystack_integrations.components.audio.whisper import RemoteWhisperTranscriber`.",
+            FutureWarning,
+            stacklevel=2,
+        )
 
         self.organization = organization
         self.model = model

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/builders/answer_builder.py

@@ -11,6 +11,9 @@ from haystack.dataclasses.chat_message import ChatMessage
 
 logger = logging.getLogger(__name__)
 
+DEFAULT_REFERENCE_PATTERN = r"\[(\d+)\]"
+EXPANDED_REFERENCE_PATTERN = r"\[(\d+(?:[,-]\d+)*)\]"
+
 
 @component
 class AnswerBuilder:
@@ -74,6 +77,7 @@ class AnswerBuilder:
         last_message_only: bool = False,
         *,
         return_only_referenced_documents: bool = True,
+        expand_reference_ranges: bool = False,
     ) -> None:
         """
         Creates an instance of the AnswerBuilder component.
@@ -104,6 +108,10 @@ class AnswerBuilder:
             If True (default value), only the documents that were actually referenced in `replies` are returned.
             If False, all documents are returned.
             If `reference_pattern` is not provided, this parameter has no effect, and all documents are returned.
+        :param expand_reference_ranges:
+            If True, reference ranges like `[6-10]` are expanded to documents 6 through 10.
+            Defaults to False for backwards compatibility.
+            When enabled with the default `reference_pattern`, a broader pattern is used automatically.
         """
         if pattern:
             AnswerBuilder._check_num_groups_in_regex(pattern)
@@ -112,6 +120,7 @@ class AnswerBuilder:
         self.reference_pattern = reference_pattern
         self.last_message_only = last_message_only
         self.return_only_referenced_documents = return_only_referenced_documents
+        self.expand_reference_ranges = expand_reference_ranges
 
     @component.output_types(answers=list[GeneratedAnswer])
     def run(
@@ -122,6 +131,7 @@ class AnswerBuilder:
         documents: list[Document] | None = None,
         pattern: str | None = None,
         reference_pattern: str | None = None,
+        expand_reference_ranges: bool | None = None,
     ) -> dict[str, Any]:
         """
         Turns the output of a Generator into `GeneratedAnswer` objects using regular expressions.
@@ -158,6 +168,9 @@ class AnswerBuilder:
             If not specified, no parsing is done, and all documents are returned.
             References need to be specified as indices of the input documents and start at [1].
             Example: `\\[(\\d+)\\]` finds "1" in a string "this is an answer[1]".
+        :param expand_reference_ranges:
+            If True, reference ranges like `[6-10]` are expanded to documents 6 through 10.
+            If not specified, the value from the component initialization is used.
 
         :returns: A dictionary with the following keys:
             - `answers`: The answers received from the output of the Generator.
@@ -172,6 +185,12 @@ class AnswerBuilder:
 
         pattern = pattern or self.pattern
         reference_pattern = reference_pattern or self.reference_pattern
+        expand_reference_ranges = (
+            self.expand_reference_ranges if expand_reference_ranges is None else expand_reference_ranges
+        )
+        reference_pattern = AnswerBuilder._resolve_reference_pattern(
+            reference_pattern=reference_pattern, expand_reference_ranges=expand_reference_ranges
+        )
 
         replies_to_iterate = replies[-1:] if self.last_message_only and replies else replies
         meta_to_iterate = meta[-1:] if self.last_message_only and meta else meta
@@ -188,7 +207,12 @@ class AnswerBuilder:
             referenced_docs = []
             if documents:
                 referenced_idxs = (
-                    AnswerBuilder._extract_reference_idxs(extracted_reply, reference_pattern)
+                    AnswerBuilder._extract_reference_idxs(
+                        extracted_reply,
+                        reference_pattern,
+                        expand_ranges=expand_reference_ranges,
+                        num_documents=len(documents),
+                    )
                     if reference_pattern
                     else set()
                 )
@@ -245,9 +269,40 @@ class AnswerBuilder:
         return ""
 
     @staticmethod
-    def _extract_reference_idxs(reply: str, reference_pattern: str) -> set[int]:
-        document_idxs = re.findall(reference_pattern, reply)
-        return {int(idx) - 1 for idx in document_idxs}
+    def _resolve_reference_pattern(reference_pattern: str | None, expand_reference_ranges: bool) -> str | None:
+        if not reference_pattern or not expand_reference_ranges:
+            return reference_pattern
+        if reference_pattern == DEFAULT_REFERENCE_PATTERN:
+            return EXPANDED_REFERENCE_PATTERN
+        return reference_pattern
+
+    @staticmethod
+    def _extract_reference_idxs(
+        reply: str, reference_pattern: str, expand_ranges: bool = False, num_documents: int | None = None
+    ) -> set[int]:
+        matches = re.findall(reference_pattern, reply)
+        idxs: set[int] = set()
+        for match in matches:
+            if expand_ranges:
+                for part in match.split(","):
+                    part = part.strip()
+                    if not part:
+                        continue
+                    if "-" in part:
+                        start_str, end_str = part.split("-", 1)
+                        start, end = int(start_str), int(end_str)
+                        if start > end:
+                            continue
+                        # Clamp the range end to the number of documents to avoid materializing a huge
+                        # set from an out-of-range citation like `[1-999999999]` in the Generator output.
+                        if num_documents is not None:
+                            end = min(end, num_documents)
+                        idxs.update(range(start - 1, end))
+                    else:
+                        idxs.add(int(part) - 1)
+            else:
+                idxs.add(int(match) - 1)
+        return idxs
 
     @staticmethod
     def _check_num_groups_in_regex(pattern: str) -> None:

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/classifiers/document_language_classifier.py

@@ -2,6 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
+import warnings
 from dataclasses import replace
 
 from haystack import Document, component, logging
@@ -68,6 +69,14 @@ class DocumentLanguageClassifier:
             See the supported languages in [`langdetect` documentation](https://github.com/Mimino666/langdetect#languages).
             If not specified, defaults to ["en"].
         """
+        warnings.warn(
+            "`DocumentLanguageClassifier` will be removed from Haystack in version 3.0, as it is moving to "
+            "the `langdetect-haystack` package. To continue using it, install that package with "
+            "`pip install langdetect-haystack` and update your import to "
+            "`from haystack_integrations.components.classifiers.langdetect import DocumentLanguageClassifier`.",
+            FutureWarning,
+            stacklevel=2,
+        )
         langdetect_import.check()
         if not languages:
             languages = ["en"]

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/classifiers/zero_shot_document_classifier.py

@@ -2,6 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
+import warnings
 from dataclasses import replace
 from typing import Any
 
@@ -112,6 +113,15 @@ class TransformersZeroShotDocumentClassifier:
             Dictionary containing keyword arguments used to initialize the
             Hugging Face pipeline for text classification.
         """
+        warnings.warn(
+            "`TransformersZeroShotDocumentClassifier` will be removed from Haystack in version 3.0, as it is moving "
+            "to the `transformers-haystack` package. To continue using it, install that package with "
+            "`pip install transformers-haystack` and update your import to "
+            "`from haystack_integrations.components.classifiers.transformers import "
+            "TransformersZeroShotDocumentClassifier`.",
+            FutureWarning,
+            stacklevel=2,
+        )
 
         torch_and_transformers_import.check()
 

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/connectors/openapi.py

@@ -2,6 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
+import warnings
 from typing import Any
 
 from haystack import component, default_from_dict, default_to_dict
@@ -61,6 +62,14 @@ class OpenAPIConnector:
         :param service_kwargs: Additional keyword arguments passed to OpenAPIClient.from_spec()
             For example, you can pass a custom config_factory or other configuration options.
         """
+        warnings.warn(
+            "`OpenAPIConnector` will be removed from Haystack in version 3.0, as it is moving to "
+            "the `openapi-haystack` package. To continue using it, install that package with "
+            "`pip install openapi-haystack` and update your import to "
+            "`from haystack_integrations.components.connectors.openapi import OpenAPIConnector`.",
+            FutureWarning,
+            stacklevel=2,
+        )
         openapi_llm_imports.check()
         self.openapi_spec = openapi_spec
         self.credentials = credentials

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/connectors/openapi_service.py

@@ -3,6 +3,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 import json
+import warnings
 from collections import defaultdict
 from copy import copy
 from typing import Any
@@ -211,6 +212,14 @@ class OpenAPIServiceConnector:
         :param ssl_verify: Decide if to use SSL verification to the requests or not,
         in case a string is passed, will be used as the CA.
         """
+        warnings.warn(
+            "`OpenAPIServiceConnector` will be removed from Haystack in version 3.0, as it is moving to "
+            "the `openapi-haystack` package. To continue using it, install that package with "
+            "`pip install openapi-haystack` and update your import to "
+            "`from haystack_integrations.components.connectors.openapi import OpenAPIServiceConnector`.",
+            FutureWarning,
+            stacklevel=2,
+        )
         openapi_imports.check()
         self.ssl_verify = ssl_verify
 

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/converters/azure.py

@@ -4,6 +4,7 @@
 
 import copy
 import os
+import warnings
 from collections import defaultdict
 from pathlib import Path
 from typing import Any, Literal, Union
@@ -98,6 +99,15 @@ class AzureOCRDocumentConverter:
             If True, the full path of the file is stored in the metadata of the document.
             If False, only the file name is stored.
         """
+        warnings.warn(
+            "`AzureOCRDocumentConverter` will be removed from Haystack in version 3.0, as it is moving to "
+            "the `azure-form-recognizer-haystack` package. To continue using it, install that package with "
+            "`pip install azure-form-recognizer-haystack` and update your import to "
+            "`from haystack_integrations.components.converters.azure_form_recognizer import "
+            "AzureOCRDocumentConverter`.",
+            FutureWarning,
+            stacklevel=2,
+        )
         azure_import.check()
         pandas_import.check()
 

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/converters/file_to_file_content.py

@@ -27,17 +27,13 @@ class FileToFileContent:
     from haystack.components.converters import FileToFileContent
 
     converter = FileToFileContent()
-
-    sources = ["document.pdf", "video.mp4"]
-
+    sources = ["test/test_files/pdf/react_paper.pdf", "test/test_files/images/haystack-logo.png"]
     file_contents = converter.run(sources=sources)["file_contents"]
-    print(file_contents)
 
-    # [FileContent(base64_data='...',
-    #              mime_type='application/pdf',
-    #              filename='document.pdf',
-    #              extra={}),
-    #  ...]
+    print(file_contents)
+    # >> [FileContent(base64_data='...', mime_type='application/pdf', filename='react_paper.pdf', extra={}),
+    # >>  FileContent(base64_data='...', mime_type='image/png', filename='haystack-logo.png', extra={})
+    # >>]
     ```
     """
 

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/converters/html.py

@@ -35,7 +35,9 @@ class HTMLToDocument:
     ```
     """
 
-    def __init__(self, extraction_kwargs: dict[str, Any] | None = None, store_full_path: bool = False) -> None:
+    def __init__(
+        self, extraction_kwargs: dict[str, Any] | None = None, store_full_path: bool = False, encoding: str = "utf-8"
+    ) -> None:
         """
         Create an HTMLToDocument component.
 
@@ -45,11 +47,15 @@ class HTMLToDocument:
         :param store_full_path:
         If True, the full path of the file is stored in the metadata of the document.
         If False, only the file name is stored.
+        :param encoding:
+            The default encoding to use when converting HTML files. If the encoding is specified in the metadata of a
+            source ByteStream, it overrides this value.
         """
         trafilatura_import.check()
 
         self.extraction_kwargs = extraction_kwargs or {}
         self.store_full_path = store_full_path
+        self.encoding = encoding
 
     def to_dict(self) -> dict[str, Any]:
         """
@@ -58,7 +64,9 @@ class HTMLToDocument:
         :returns:
             Dictionary with serialized data.
         """
-        return default_to_dict(self, extraction_kwargs=self.extraction_kwargs, store_full_path=self.store_full_path)
+        return default_to_dict(
+            self, extraction_kwargs=self.extraction_kwargs, store_full_path=self.store_full_path, encoding=self.encoding
+        )
 
     @classmethod
     def from_dict(cls, data: dict[str, Any]) -> "HTMLToDocument":
@@ -111,8 +119,13 @@ class HTMLToDocument:
                 logger.warning("Could not read {source}. Skipping it. Error: {error}", source=source, error=e)
                 continue
 
+            if not bytestream.data:
+                logger.warning("Skipping {source} because it is empty.", source=source)
+                continue
+
             try:
-                text = extract(bytestream.data.decode("utf-8"), **merged_extraction_kwargs)
+                encoding = bytestream.meta.get("encoding", self.encoding)
+                text = extract(bytestream.data.decode(encoding), **merged_extraction_kwargs)
             except Exception as conversion_e:
                 logger.warning(
                     "Failed to extract text from {source}. Skipping it. Error: {error}",

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/converters/image/document_to_image.py

@@ -34,31 +34,34 @@ class DocumentToImageContent:
     - For PDF files, a `page_number` key specifying which page to extract
 
     ### Usage example
-    <!-- test-ignore -->
+
     ```python
     from haystack import Document
     from haystack.components.converters.image.document_to_image import DocumentToImageContent
 
     converter = DocumentToImageContent(
         file_path_meta_field="file_path",
-        root_path="/data/files",
+        root_path="test/test_files",
         detail="high",
         size=(800, 600)
     )
 
     documents = [
-        Document(content="Optional description of image.jpg", meta={"file_path": "image.jpg"}),
-        Document(content="Text content of page 1 of doc.pdf", meta={"file_path": "doc.pdf", "page_number": 1})
+        Document(content="Optional description of apple.jpg", meta={"file_path": "images/apple.jpg"}),
+        Document(
+            content="Optional description of sample_pdf_1.pdf",
+            meta={"file_path": "pdf/sample_pdf_1.pdf", "page_number": 1}
+        )
     ]
 
     result = converter.run(documents)
     image_contents = result["image_contents"]
     # [ImageContent(
-    #    base64_image='/9j/4A...', mime_type='image/jpeg', detail='high', meta={'file_path': 'image.jpg'}
+    #    base64_image='/9j/4A...', mime_type='image/jpeg', detail='high', meta={'file_path': 'images/apple.jpg'}
     #  ),
     #  ImageContent(
     #    base64_image='/9j/4A...', mime_type='image/jpeg', detail='high',
-    #    meta={'page_number': 1, 'file_path': 'doc.pdf'}
+    #    meta={'file_path': 'pdf/sample_pdf_1.pdf', 'page_number': 1})
     #  )]
     ```
     """

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/converters/image/image_utils.py

@@ -248,6 +248,18 @@ def _extract_image_sources_info(
             )
 
         resolved_file_path = Path(root_path, file_path)
+
+        # When root_path is set, ensure the resolved path stays within it to block path-traversal
+        # payloads (e.g. "../../etc/passwd") coming from document metadata.
+        if root_path:
+            resolved_file_path = resolved_file_path.resolve()
+            resolved_root = Path(root_path).resolve()
+            if not resolved_file_path.is_relative_to(resolved_root):
+                raise ValueError(
+                    f"Document with ID '{doc.id}' has a file path '{file_path}' that escapes the "
+                    f"configured root '{root_path}'. Resolved path: '{resolved_file_path}'."
+                )
+
         if not resolved_file_path.is_file():
             raise ValueError(
                 f"Document with ID '{doc.id}' has an invalid file path '{resolved_file_path}'. "

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/converters/markdown.py

@@ -2,10 +2,13 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 
+import json
 import os
+import re
 from pathlib import Path
 from typing import Any
 
+import yaml
 from tqdm import tqdm
 
 from haystack import Document, component, logging
@@ -20,6 +23,8 @@ with LazyImport("Run 'pip install markdown-it-py mdit_plain'") as markdown_conve
 
 logger = logging.getLogger(__name__)
 
+_FRONTMATTER_PATTERN = re.compile(r"\A---[ \t]*\r?\n(?P<frontmatter>.*?)(?:\r?\n)---[ \t]*(?:\r?\n|$)", re.DOTALL)
+
 
 @component
 class MarkdownToDocument:
@@ -43,7 +48,13 @@ class MarkdownToDocument:
     """
 
     def __init__(
-        self, table_to_single_line: bool = False, progress_bar: bool = True, store_full_path: bool = False
+        self,
+        table_to_single_line: bool = False,
+        progress_bar: bool = True,
+        store_full_path: bool = False,
+        encoding: str = "utf-8",
+        *,
+        extract_frontmatter: bool = False,
     ) -> None:
         """
         Create a MarkdownToDocument component.
@@ -55,12 +66,20 @@ class MarkdownToDocument:
         :param store_full_path:
             If True, the full path of the file is stored in the metadata of the document.
             If False, only the file name is stored.
+        :param encoding:
+            The default encoding to use when converting Markdown files. If the encoding is specified in the metadata
+            of a source ByteStream, it overrides this value.
+        :param extract_frontmatter:
+            If True, YAML frontmatter at the beginning of the Markdown file is
+            removed from the document content and added to the document metadata.
         """
         markdown_conversion_imports.check()
 
         self.table_to_single_line = table_to_single_line
         self.progress_bar = progress_bar
         self.store_full_path = store_full_path
+        self.encoding = encoding
+        self.extract_frontmatter = extract_frontmatter
 
     @component.output_types(documents=list[Document])
     def run(
@@ -102,7 +121,9 @@ class MarkdownToDocument:
                 logger.warning("Could not read {source}. Skipping it. Error: {error}", source=source, error=e)
                 continue
             try:
-                file_content = bytestream.data.decode("utf-8")
+                encoding = bytestream.meta.get("encoding", self.encoding)
+                file_content = bytestream.data.decode(encoding)
+                file_content, frontmatter = self._extract_frontmatter(file_content, source)
                 text = parser.render(file_content)
             except Exception as conversion_e:
                 logger.warning(
@@ -112,7 +133,7 @@ class MarkdownToDocument:
                 )
                 continue
 
-            merged_metadata = {**bytestream.meta, **metadata}
+            merged_metadata = {**bytestream.meta, **frontmatter, **metadata}
 
             if not self.store_full_path and (file_path := bytestream.meta.get("file_path")):
                 merged_metadata["file_path"] = os.path.basename(file_path)
@@ -121,3 +142,39 @@ class MarkdownToDocument:
             documents.append(document)
 
         return {"documents": documents}
+
+    def _extract_frontmatter(self, file_content: str, source: str | Path | ByteStream) -> tuple[str, dict[str, Any]]:
+        if not self.extract_frontmatter:
+            return file_content, {}
+
+        match = _FRONTMATTER_PATTERN.match(file_content)
+        if not match:
+            return file_content, {}
+
+        frontmatter_text = match.group("frontmatter")
+        try:
+            frontmatter = json.loads(json.dumps(yaml.safe_load(frontmatter_text), default=str)) or {}
+        except yaml.YAMLError as error:
+            logger.warning(
+                "Could not parse YAML frontmatter in {source}. Keeping it as content. Error: {error}",
+                source=source,
+                error=error,
+            )
+            return file_content, {}
+        except (TypeError, ValueError) as error:
+            logger.warning(
+                "Could not convert YAML frontmatter in {source}. Keeping it as content. Error: {error}",
+                source=source,
+                error=error,
+            )
+            return file_content, {}
+
+        if not isinstance(frontmatter, dict):
+            logger.warning(
+                "Ignoring YAML frontmatter in {source}: expected a mapping, got {kind}.",
+                source=source,
+                kind=type(frontmatter).__name__,
+            )
+            return file_content, {}
+
+        return file_content[match.end() :], frontmatter

{haystack_ai-2.30.2 → haystack_ai-2.31.0}/haystack/components/converters/openapi_functions.py

@@ -4,6 +4,7 @@
 
 import json
 import os
+import warnings
 from pathlib import Path
 from typing import Any
 
@@ -54,6 +55,14 @@ class OpenAPIServiceToFunctions:
         """
         Create an OpenAPIServiceToFunctions component.
         """
+        warnings.warn(
+            "`OpenAPIServiceToFunctions` will be removed from Haystack in version 3.0, as it is moving to "
+            "the `openapi-haystack` package. To continue using it, install that package with "
+            "`pip install openapi-haystack` and update your import to "
+            "`from haystack_integrations.components.converters.openapi import OpenAPIServiceToFunctions`.",
+            FutureWarning,
+            stacklevel=2,
+        )
         openapi_imports.check()
 
     @component.output_types(functions=list[dict[str, Any]], openapi_specs=list[dict[str, Any]])