unique_internal_search 1.2.2__tar.gz

unique_internal_search-1.2.2/CHANGELOG.md

@@ -0,0 +1,62 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.2.2] - 2025-11-10
+- Temporarily reverting the removal of the `get_tool_call_result_for_loop_history` function, as it is still required for the Investment Research Agent. 
+
+## [1.2.1] - 2025-11-06
+- Upload and chat system reminder cleanup
+
+## [1.2.0] - 2025-11-04
+- Include system reminder for upload and chat tool about it being a forced tool in UniqueAI
+
+## [1.1.0] - 2025-10-30
+- Add support for multiple search strings in a single tool call
+- Search results from multiple queries are interleaved for better diversity
+- Add automatic deduplication of chunks by `chunk_id` when using multiple search queries
+  - Prevents duplicate content from appearing in results when multiple related queries return the same chunks
+  - Preserves first occurrence and logs number of duplicates removed
+- Add automatic parsing and cleaning of search query operators1
+  - Removes QDF (QueryDeservedFreshness) operators: `--QDF=0` to `--QDF=5` (freshness ratings)
+  - Removes boost operators: `+(term)` and `+(multi word phrase)` for query term boosting
+
+## [1.0.4] - 2025-10-28
+- Removing unused tool specific `get_tool_call_result_for_loop_history` function
+- Removing unused config `source_format_config`
+
+## [1.0.3] - 2025-10-25
+- Fix appending of metadata to chunks
+
+## [1.0.2] - 2025-10-17
+- Remove print statements originating from tool refactor
+
+## [1.0.1] - 2025-09-30
+- Fix bug in metadata filter in the search method.
+
+## [1.0.0] - 2025-09-18
+- Bump toolkit version to allow for both patch and minor updates
+
+## [0.0.7] - 2025-09-17
+- Updated to latest toolkit
+
+## [0.0.6] - 2025-09-15
+- Fix Minor bug in transforming toolResponse to toolCallResult
+
+## [0.0.5] - 2025-09-05
+- Fixed a bug around metadata-filter assignment
+
+## [0.0.4] - 2025-09-05
+- Fixed a bug around metadata-filter deep-copy
+
+## [0.0.3] - 2025-09-01
+- Migrated the `uploaded_search` into this package.
+
+## [0.0.2] - 2025-09-01
+- Migrated the `internal_search`.
+
+## [0.0.1] - 2025-08-18
+- Initial release of `internal_search`.

unique_internal_search-1.2.2/LICENSE

@@ -0,0 +1 @@
+`unique_toolkit` is covered by the [`Unique License v1`](https://github.com/Unique-AG/license/releases/tag/unique-license.v1), unless the/a header or a nested LICENSE specifies another license.

unique_internal_search-1.2.2/PKG-INFO

@@ -0,0 +1,86 @@
+Metadata-Version: 2.1
+Name: unique_internal_search
+Version: 1.2.2
+Summary: 
+License: Proprietary
+Author: Martin Fadler
+Author-email: martin.fadler@unique.ch
+Requires-Python: >=3.12,<4.0
+Classifier: License :: Other/Proprietary License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: pillow (>=10.4.0,<11.0.0)
+Requires-Dist: pydantic (>=2.8.2,<3.0.0)
+Requires-Dist: pydantic-settings (>=2.10.1,<3.0.0)
+Requires-Dist: pytest (>=8.4.1,<9.0.0)
+Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Requires-Dist: typing-extensions (>=4.9.0,<5.0.0)
+Requires-Dist: unique-sdk (>=0.10.0,<0.11.0)
+Requires-Dist: unique-toolkit (>=1.18.1,<2.0.0)
+Description-Content-Type: text/markdown
+
+# Internal Search Tool
+
+Internal Search Tool to find documents in the Knowledge Base
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), 
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.2.2] - 2025-11-10
+- Temporarily reverting the removal of the `get_tool_call_result_for_loop_history` function, as it is still required for the Investment Research Agent. 
+
+## [1.2.1] - 2025-11-06
+- Upload and chat system reminder cleanup
+
+## [1.2.0] - 2025-11-04
+- Include system reminder for upload and chat tool about it being a forced tool in UniqueAI
+
+## [1.1.0] - 2025-10-30
+- Add support for multiple search strings in a single tool call
+- Search results from multiple queries are interleaved for better diversity
+- Add automatic deduplication of chunks by `chunk_id` when using multiple search queries
+  - Prevents duplicate content from appearing in results when multiple related queries return the same chunks
+  - Preserves first occurrence and logs number of duplicates removed
+- Add automatic parsing and cleaning of search query operators1
+  - Removes QDF (QueryDeservedFreshness) operators: `--QDF=0` to `--QDF=5` (freshness ratings)
+  - Removes boost operators: `+(term)` and `+(multi word phrase)` for query term boosting
+
+## [1.0.4] - 2025-10-28
+- Removing unused tool specific `get_tool_call_result_for_loop_history` function
+- Removing unused config `source_format_config`
+
+## [1.0.3] - 2025-10-25
+- Fix appending of metadata to chunks
+
+## [1.0.2] - 2025-10-17
+- Remove print statements originating from tool refactor
+
+## [1.0.1] - 2025-09-30
+- Fix bug in metadata filter in the search method.
+
+## [1.0.0] - 2025-09-18
+- Bump toolkit version to allow for both patch and minor updates
+
+## [0.0.7] - 2025-09-17
+- Updated to latest toolkit
+
+## [0.0.6] - 2025-09-15
+- Fix Minor bug in transforming toolResponse to toolCallResult
+
+## [0.0.5] - 2025-09-05
+- Fixed a bug around metadata-filter assignment
+
+## [0.0.4] - 2025-09-05
+- Fixed a bug around metadata-filter deep-copy
+
+## [0.0.3] - 2025-09-01
+- Migrated the `uploaded_search` into this package.
+
+## [0.0.2] - 2025-09-01
+- Migrated the `internal_search`.
+
+## [0.0.1] - 2025-08-18
+- Initial release of `internal_search`.

unique_internal_search-1.2.2/README.md

@@ -0,0 +1,3 @@
+# Internal Search Tool
+
+Internal Search Tool to find documents in the Knowledge Base

unique_internal_search-1.2.2/pyproject.toml

@@ -0,0 +1,46 @@
+[tool.poetry]
+name = "unique_internal_search"
+version = "1.2.2"
+description = ""
+authors = [
+    "Martin Fadler <martin.fadler@unique.ch>",
+    "Sadique Sheik <sadique@unique.ch>",
+    "Fabian Schläpfer <fabian@unique.ch>",
+    "Pascal Hauri <pascal@unique.ch>",
+]
+readme = ["README.md", "CHANGELOG.md"]
+license = "Proprietary"
+
+[tool.poetry.dependencies]
+python = "^3.12"
+typing-extensions = "^4.9.0"
+pydantic = "^2.8.2"
+pydantic-settings = "^2.10.1"
+python-dotenv = "^1.0.1"
+pytest = "^8.4.1"
+pillow = "^10.4.0"
+unique-sdk = "^0.10.0"
+unique-toolkit = "^1.18.1"
+
+
+[tool.poetry.group.dev.dependencies]
+python = "^3.12"
+typing-extensions = "^4.9.0"
+pydantic = "^2.8.2"
+pydantic-settings = "^2.10.1"
+python-dotenv = "^1.0.1"
+pytest = "^8.4.1"
+unique-sdk = { path = "../../unique_sdk" }
+unique-toolkit = { path = "../../unique_toolkit" }
+ruff = "^0.12.10"
+
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.ruff]
+target-version = "py311"
+
+[tool.ruff.lint]
+extend-select = ["I"]

unique_internal_search-1.2.2/unique_internal_search/config.py

@@ -0,0 +1,153 @@
+from typing import Annotated, Any
+
+from pydantic import (
+    AliasChoices,
+    Field,
+)
+from pydantic.json_schema import SkipJsonSchema
+from unique_toolkit._common.chunk_relevancy_sorter.config import (
+    ChunkRelevancySortConfig,
+)
+from unique_toolkit._common.feature_flags.schema import (
+    FeatureExtendedSourceSerialization,
+)
+from unique_toolkit.agentic.evaluation.schemas import EvaluationMetricName
+from unique_toolkit.agentic.history_manager.history_manager import DeactivatedNone
+from unique_toolkit.agentic.tools.schemas import BaseToolConfig
+from unique_toolkit.content.schemas import (
+    ContentRerankerConfig,
+    ContentSearchType,
+)
+
+from unique_internal_search.prompts import (
+    DEFAULT_LANGUAGE_PARAM_DESCRIPTION,
+    DEFAULT_SEARCH_STRING_PARAM_DESCRIPTION,
+    DEFAULT_TOOL_DESCRIPTION,
+    DEFAULT_TOOL_DESCRIPTION_FOR_SYSTEM_PROMPT,
+    DEFAULT_TOOL_FORMAT_INFORMATION_FOR_SYSTEM_PROMPT,
+)
+from unique_internal_search.validators import get_string_field_with_pattern_validation
+
+
+class ExperimentalFeatures(FeatureExtendedSourceSerialization):
+    enable_multiple_search_strings_execution: bool = Field(
+        default=False,
+        description="Allow execution of multiple search strings in one call. When set to True, each string is searched individually and results are merged into a single response.",
+    )
+
+
+DEFAULT_LIMIT_CHUNK_RELEVANCY_SORT_ENABLED = 200
+DEFAULT_LIMIT_CHUNK_RELEVANCY_SORT_DISABLED = 1000
+
+
+def _search_limit_factory(data: dict[str, Any]) -> int:
+    return (
+        DEFAULT_LIMIT_CHUNK_RELEVANCY_SORT_ENABLED
+        if data["chunk_relevancy_sort_config"].enabled
+        else DEFAULT_LIMIT_CHUNK_RELEVANCY_SORT_DISABLED
+    )
+
+
+class InternalSearchConfig(BaseToolConfig):
+    search_type: ContentSearchType = Field(
+        default=ContentSearchType.COMBINED,
+        description="The type of search to perform. Two possible values: `COMBINED` or `VECTOR`.",
+    )
+    max_tokens_for_sources: SkipJsonSchema[int] = (
+        Field(  # TODO: Remove SkipJsonSchema once UI (Spaces 2.0) can be configured to not include certain fields
+            default=30_000,
+            description="The maximum number of tokens to use for the sources.",
+        )
+    )
+    percentage_of_input_tokens_for_sources: float = Field(
+        default=0.4,
+        description="The percentage of the maximum input tokens of the language model to use for the tool response.",
+        ge=0.0,
+        le=1.0,
+    )
+    language_model_max_input_tokens: SkipJsonSchema[int | None] = Field(
+        default=None,
+        description="Language model maximum input tokens",
+    )
+    scope_ids: Annotated[list[str], Field(title="Active")] | DeactivatedNone = Field(
+        default=None,
+        description="The scope ids to use for the search.",
+    )
+    scope_to_chat_on_upload: bool = Field(
+        default=False,
+        description="Whether to scope the search should be limited to files uploaded within the chat session when uploaded files are present.",
+    )
+    chunked_sources: bool = Field(
+        default=True,
+        description="Whether each chunk is added as an individual source in the final LLM prompt. If set to False, all chunks from the same document are combined into a single source.",
+    )
+    reranker_config: (
+        Annotated[ContentRerankerConfig, Field(title="Active")] | DeactivatedNone
+    ) = Field(
+        default=None,
+        description="The reranker config to use for the search.",
+    )
+    search_language: str = Field(
+        default="english",
+        validation_alias=AliasChoices("ftsSearchLanguage", "searchLanguage"),
+        description="The language to use for the search.",
+    )
+    # evaluation_config: EvaluationMetricConfig = EvaluationMetricConfig()
+    chunk_relevancy_sort_config: ChunkRelevancySortConfig = Field(
+        default_factory=ChunkRelevancySortConfig,
+        description="The chunk relevancy sort config to use for the search.",
+    )
+    limit: int = Field(
+        default_factory=_search_limit_factory,
+        description="The limit of chunks to return.",
+    )
+    chat_only: bool = Field(
+        default=False,
+        description="Whether to only chat on the upload.",
+    )
+
+    tool_description: str = get_string_field_with_pattern_validation(
+        DEFAULT_TOOL_DESCRIPTION,
+        description="Tool description.",
+    )
+    param_description_search_string: str = Field(
+        default=DEFAULT_SEARCH_STRING_PARAM_DESCRIPTION,
+        description="`search_string` parameter description.",
+    )
+    param_description_language: str = get_string_field_with_pattern_validation(
+        DEFAULT_LANGUAGE_PARAM_DESCRIPTION,
+        description="`language` parameter description.",
+    )
+    tool_description_for_system_prompt: str = get_string_field_with_pattern_validation(
+        DEFAULT_TOOL_DESCRIPTION_FOR_SYSTEM_PROMPT,
+        description="Tool description for the system prompt.",
+    )
+    tool_format_information_for_system_prompt: str = (
+        get_string_field_with_pattern_validation(
+            DEFAULT_TOOL_FORMAT_INFORMATION_FOR_SYSTEM_PROMPT,
+            description="Tool format information for the system prompt.",
+        )
+    )
+    evaluation_check_list: list[EvaluationMetricName] = Field(
+        default=[EvaluationMetricName.HALLUCINATION],
+        description="The list of evaluation metrics to check.",
+    )
+    experimental_features: ExperimentalFeatures = ExperimentalFeatures()
+
+    metadata_chunk_sections: dict[str, str] = Field(
+        default={},
+        description=(
+            "Metadata sections to be appended to each search result chunk’s text. The keys represent metadata field names (e.g., 'metadata_key'), and the values are template strings that define how the metadata should be embedded, using {} as a placeholder for the actual value (e.g., '<|metadata_key|>{}<|/metadata_key|>')."
+        ),
+    )
+
+    score_threshold: float = Field(
+        default=0.0,
+        ge=0.0,
+        le=1.0,
+        description="The score threshold to use for the search to filter chunks on relevancy.",
+    )
+    exclude_uploaded_files: bool = Field(
+        default=False,
+        description="Whether to exclude uploaded files from the search. Overrides the `chat_only` parameter as it removes the `chat_id` from the search.",
+    )

unique_internal_search-1.2.2/unique_internal_search/prompts.py

@@ -0,0 +1,64 @@
+DEFAULT_TOOL_DESCRIPTION_FOR_SYSTEM_PROMPT = (
+    "You can use the InternalSearch tool to access internal company documentations, including information on policies, procedures, benefits, groups, financial details, and specific individuals. "
+    "If this tool can help answer your question, feel free to use it to search the internal knowledge base for more information. "
+    "If possible always try to get information from the internal knowledge base with the InternalSearch tool before using other tools.\n"
+    "Use cases for the Internal Knowledge Search are:\n"
+    "- User asks to work with a document: Most likely the document is uploaded to the chat and mentioned in a message and can be loaded with this tool\n"
+    "- Policy and Procedure Verification: Use the internal search tool to find the most current company policies, procedures, or guidelines to ensure compliance and accuracy in responses.\n"
+    "- Project-Specific Information: When answering questions related to ongoing projects or initiatives, use the internal search to access project documents, reports, or meeting notes for precise details.\n"
+    "- Employee Directory and Contact Information: Utilize the internal search to locate contact details or organizational charts to facilitate communication and collaboration within the company.\n"
+    "- Confidential and Proprietary Information: When dealing with sensitive topics that require proprietary knowledge or confidential data, use the internal search to ensure the information is sourced from secure and authorized company documents.\n\n"
+    "**Instruction Query Splitting**\n"
+    'You should split the user question into multiple search strings when the user\'s question needs to be decomposed / rewritten to find different facts. Perform for each search string an individual tool call. Avoid short queries that are extremely broad and will return unrelated results. Strip the search string of any extraneous details, e.g. instructions or unnecessary context. However, you must fill in relevant context from the rest of the conversation to make the question complete. E.g. "What was their age?" => "What was Kevin\'s age?" because the preceding conversation makes it clear that the user is talking about Kevin.\n\n'
+    "Here are some examples of how to use the InternalSearch tool:\n"
+    'User: What was the GDP of France and Italy in the 1970s? => search strings: ["france gdp 1970", "italy gdp 1970"] # Splitting of the query into 2 queries and perform 2 tool calls\n'
+    'User: What does the report say about the GPT4 performance on MMLU? => search strings: ["GPT4 performance on MMLU?"] # Simplify the query'
+)
+
+DEFAULT_TOOL_FORMAT_INFORMATION_FOR_SYSTEM_PROMPT = (
+    "Whenever you use information retrieved with the InternalSearch, you must adhere to strict reference guidelines. "
+    "You must strictly reference each fact used with the `source_number` of the corresponding passage, in the following format: '[source<source_number>]'.\n\n"
+    "Example:\n"
+    "- The stock price of Apple Inc. is $150 [source0] and the company's revenue increased by 10% [source1].\n"
+    "- Moreover, the company's market capitalization is $2 trillion [source2][source3].\n"
+    "- Our internal documents tell us to invest[source4] (Internal)\n\n"
+    "A fact is preferably referenced by ONLY ONE source, e.g [sourceX], which should be the most relevant source for the fact.\n"
+    "Follow these guidelines closely and be sure to use the proper `source_number` when referencing facts.\n"
+    "Make sure that your reference follow the format [sourceX] and that the source number is correct.\n"
+    "Source is written in singular form and the number is written in digits.\n\n"
+    "IT IS VERY IMPORTANT TO FOLLOW THESE GUIDELINES!!\n"
+    "NEVER CITE A source_number THAT YOU DON'T SEE IN THE TOOL CALL RESPONSE!!!\n"
+    "The source_number in old assistant messages are no longer valid.\n"
+    "EXAMPLE: If you see [source34] and [source35] in the assistant message, you can't use [source34] again in the next assistant message, this has to be the number you find in the message with role 'tool'.\n"
+    "BE AWARE:All tool calls have been filtered to remove uncited sources. Tool calls return much more data than you see\n\n"
+    "### Internal Document Answering Protocol for Employee Questions\n"
+    "When assisting employees using internal documents, follow\n"
+    "this structured approach to ensure precise, well-grounded,\n"
+    "and context-aware responses:\n\n"
+    "#### 1. Locate and Prioritize Relevant Internal Sources\n"
+    "Give strong preference to:\n"
+    "- **Most relevant documents**, such as:\n"
+    "- **Documents authored by or involving** the employee or team in question\n"
+    "- **Cross-validated sources**, especially when multiple documents agree\n"
+    "  - Project trackers, design docs, decision logs, and OKRs\n"
+    "  - Recently updated or active files\n\n"
+    "#### 2. Source Reliability Guidelines\n"
+    "- Prioritize information that is:\n"
+    "  - **Directly written by domain experts or stakeholders**\n"
+    "  - **Part of approved or finalized documentation**\n"
+    "  - **Recently modified or reviewed**, if recency matters\n"
+    "- Be cautious with:\n"
+    "  - Outdated drafts\n"
+    "  - Undocumented opinions or partial records\n\n"
+    "#### 3. Acknowledge Limitations\n"
+    "- If no relevant information is found, or documents conflict, clearly state this\n"
+    "- Indicate where further clarification or investigation may be required"
+)
+
+DEFAULT_TOOL_DESCRIPTION = (
+    "Search in the company knowledge base for information on policies, procedures, benefits, groups, financial information or specific people. "
+    "This should be your go-to tool if no other tools are applicable."
+)
+DEFAULT_SEARCH_STRING_PARAM_DESCRIPTION = "An expanded term that is optimized for vector and full text search based on the users query it must be in english."
+
+DEFAULT_LANGUAGE_PARAM_DESCRIPTION = "The language that the user wrote the query in"

unique_internal_search-1.2.2/unique_internal_search/service.py

@@ -0,0 +1,458 @@
+from logging import Logger
+
+from pydantic import Field, create_model
+from typing_extensions import override
+from unique_toolkit._common.chunk_relevancy_sorter.exception import (
+    ChunkRelevancySorterException,
+)
+from unique_toolkit._common.chunk_relevancy_sorter.service import ChunkRelevancySorter
+from unique_toolkit.agentic.evaluation.schemas import EvaluationMetricName
+from unique_toolkit.agentic.history_manager.utils import transform_chunks_to_string
+from unique_toolkit.agentic.tools.agent_chunks_hanlder import AgentChunksHandler
+from unique_toolkit.agentic.tools.factory import ToolFactory
+from unique_toolkit.agentic.tools.schemas import ToolCallResponse
+from unique_toolkit.agentic.tools.tool import Tool
+from unique_toolkit.agentic.tools.tool_progress_reporter import ProgressState
+from unique_toolkit.app.schemas import BaseEvent, ChatEvent, Event
+from unique_toolkit.chat.service import LanguageModelToolDescription
+from unique_toolkit.content.schemas import Content, ContentChunk
+from unique_toolkit.content.service import ContentService
+from unique_toolkit.content.utils import (
+    merge_content_chunks,
+    pick_content_chunks_for_token_window,
+    sort_content_chunks,
+)
+from unique_toolkit.language_model.schemas import (
+    LanguageModelFunction,
+    LanguageModelMessage,
+    LanguageModelToolMessage,
+)
+
+from unique_internal_search.config import InternalSearchConfig
+from unique_internal_search.utils import (
+    SearchStringResult,
+    append_metadata_in_chunks,
+    clean_search_string,
+    interleave_search_results_round_robin,
+)
+
+
+class InternalSearchService:
+    def __init__(
+        self,
+        config: InternalSearchConfig,
+        content_service: ContentService,
+        chunk_relevancy_sorter: ChunkRelevancySorter,
+        chat_id: str | None,
+        logger: Logger,
+    ):
+        self.config = config
+        self.content_service = content_service
+        self.chunk_relevancy_sorter = chunk_relevancy_sorter
+        self.chat_id = chat_id
+        self.logger = logger
+        self.tool_execution_message_name = "Internal search"
+
+    async def post_progress_message(self, message: str, *args, **kwargs):
+        pass
+
+    async def get_uploaded_files(self) -> list[Content]:
+        chat_results = await self.content_service.search_contents_async(
+            where={
+                "ownerId": {
+                    "equals": self.chat_id,
+                }
+            },
+        )
+        sorted_chat_results: list[Content] = sorted(
+            chat_results,
+            key=lambda x: x.created_at,  # type: ignore
+            reverse=True,
+        )
+        return sorted_chat_results
+
+    async def is_chat_only(self, **kwargs) -> bool:
+        """Check whether the assistant should limit itself to files in chat"""
+        if self.config.chat_only:
+            return True
+        if self.config.scope_to_chat_on_upload:
+            chat_files = await self.get_uploaded_files()
+            if len(chat_files) > 0:
+                return True
+        return False
+
+    async def search(
+        self,
+        search_string: str | list[str],
+        content_ids: list[str] | None = None,
+        metadata_filter: dict | None = None,
+        **kwargs,
+    ) -> list[ContentChunk]:
+        """
+        Perform a search with one or more search strings.
+
+        Args:
+            search_string: List of search strings or single search string
+            content_ids: List of content IDs
+            metadata_filter: Metadata filter
+        """
+
+        # Convert single string to list
+        if isinstance(search_string, str):
+            search_strings = [search_string]
+        else:
+            search_strings = search_string
+
+        """
+        Perform a search in the Vector DB based on the user's message and generate a response.
+        """
+
+        # Clean search strings by removing QDF and boost operators
+        search_strings = [clean_search_string(s) for s in search_strings]
+
+        ###
+        # 2. Search for context in the Vector DB
+        ###
+        chat_only = await self.is_chat_only(**kwargs)
+
+        """
+        Handle the fact that metadata can exclude uploaded content
+        and that the search service is hardcoded to use the metadata_filter 
+        from the event if set to None
+        """
+        # Take a backup of the metadata filter
+        metadata_filter_copy = self.content_service._metadata_filter
+
+        if metadata_filter is None:
+            metadata_filter = self.content_service._metadata_filter
+        if chat_only and metadata_filter:
+            # if this is not set to none search_content_chunks_async will overwrite it inside its call thats why it needs to stay.
+            self.content_service._metadata_filter = None
+            metadata_filter = None
+
+        found_chunks_per_search_string: list[SearchStringResult] = []
+        for i, search_string in enumerate(search_strings):
+            try:
+                found_chunks: list[
+                    ContentChunk
+                ] = await self.content_service.search_content_chunks_async(
+                    search_string=search_string,  # type: ignore
+                    search_type=self.config.search_type,
+                    limit=self.config.limit,
+                    reranker_config=self.config.reranker_config,
+                    search_language=self.config.search_language,
+                    scope_ids=self.config.scope_ids,
+                    metadata_filter=metadata_filter,
+                    chat_id=self.chat_id
+                    if self.config.exclude_uploaded_files and self.chat_id
+                    else "",
+                    chat_only=chat_only,
+                    content_ids=content_ids,
+                    score_threshold=self.config.score_threshold,
+                )
+                self.logger.info(
+                    f"Found {len(found_chunks)} chunks (Query {i + 1}/{len(search_strings)})"
+                )
+            except Exception as e:
+                self.logger.error(f"Error in search_document_chunks call: {e}")
+                raise e
+
+            found_chunks_per_search_string.append(
+                SearchStringResult(
+                    query=search_string,
+                    chunks=found_chunks,
+                )
+            )
+
+        # Reset the metadata filter in case it was disabled
+        self.content_service._metadata_filter = metadata_filter_copy
+
+        # Apply chunk relevancy sorter if enabled
+        if self.config.chunk_relevancy_sort_config.enabled:
+            for i, result in enumerate(found_chunks_per_search_string):
+                await self.post_progress_message(
+                    f"{result.query} (_Resorting {len(result.chunks)} search results_ 🔄 in query {i + 1}/{len(search_strings)})",
+                    **kwargs,
+                )
+                result.chunks = await self._resort_found_chunks_if_enabled(
+                    found_chunks=result.chunks,
+                    search_string=result.query,
+                )
+
+        ###
+        # 3. Pick a subset of the search results
+        ###
+        if (
+            self.config.experimental_features.enable_multiple_search_strings_execution
+            and len(found_chunks_per_search_string) > 1
+        ):
+            found_chunks_per_search_string = interleave_search_results_round_robin(
+                found_chunks_per_search_string
+            )
+
+        await self.post_progress_message(
+            f"{', '.join(search_strings)} (_Postprocessing search results_)",
+            **kwargs,
+        )
+        found_chunks = [
+            chunk
+            for result in found_chunks_per_search_string
+            for chunk in result.chunks
+        ]
+        selected_chunks = pick_content_chunks_for_token_window(
+            found_chunks, self._get_max_tokens()
+        )
+
+        ###
+        # 4. cache them add index to search results & join them together
+        ###
+        if not self.config.chunked_sources:
+            selected_chunks = merge_content_chunks(selected_chunks)
+        else:
+            selected_chunks = sort_content_chunks(selected_chunks)
+
+        self.debug_info = {
+            "searchStrings": search_strings,
+            "metadataFilter": metadata_filter,
+            "chatOnly": chat_only,
+        }
+        return selected_chunks
+
+    async def _resort_found_chunks_if_enabled(
+        self, found_chunks: list[ContentChunk], search_string: str
+    ) -> list[ContentChunk]:
+        try:
+            total_chunks = len(found_chunks)
+            self.logger.info(f"Resorting {total_chunks} search result...")
+            chunk_relevancy_sorter_result = await self.chunk_relevancy_sorter.run(
+                input_text=search_string,
+                chunks=found_chunks,
+                config=self.config.chunk_relevancy_sort_config,
+            )
+            found_chunks = chunk_relevancy_sorter_result.content_chunks
+        except ChunkRelevancySorterException as e:
+            self.logger.warning(f"Error while sorting chunks: {e.error_message}")
+        finally:
+            return found_chunks
+
+    def _get_max_tokens(self) -> int:
+        if self.config.language_model_max_input_tokens is not None:
+            max_tokens = int(
+                self.config.language_model_max_input_tokens
+                * self.config.percentage_of_input_tokens_for_sources
+            )
+            self.logger.debug(
+                "Using %s of max tokens %s as token limit: %s",
+                self.config.percentage_of_input_tokens_for_sources,
+                self.config.language_model_max_input_tokens,
+                max_tokens,
+            )
+            return max_tokens
+        else:
+            self.logger.debug(
+                "language model input context size is not set, using default max tokens"
+            )
+            return self.config.max_tokens_for_sources
+
+
+class InternalSearchTool(Tool[InternalSearchConfig], InternalSearchService):
+    name = "InternalSearch"
+
+    def __init__(
+        self,
+        configuration: InternalSearchConfig,
+        event: BaseEvent,
+        *args,
+        **kwargs,
+    ):
+        Tool.__init__(self, configuration, event, *args, **kwargs)
+
+        content_service = ContentService.from_event(self.event)
+        chunk_relevancy_sorter = ChunkRelevancySorter.from_event(self.event)
+        # Determing chat_id if possible
+        if isinstance(self.event, (ChatEvent, Event)):
+            chat_id = self.event.payload.chat_id
+        else:
+            chat_id = None
+        InternalSearchService.__init__(
+            self,
+            config=configuration,
+            content_service=content_service,
+            chunk_relevancy_sorter=chunk_relevancy_sorter,
+            chat_id=chat_id,
+            logger=self.logger,
+        )
+
+    async def post_progress_message(
+        self, message: str, tool_call: LanguageModelFunction, **kwargs
+    ):
+        if self.tool_progress_reporter:
+            await self.tool_progress_reporter.notify_from_tool_call(
+                tool_call=tool_call,
+                name=f"**{self.tool_execution_message_name}**",
+                message=message,
+                state=ProgressState.RUNNING,
+            )
+
+    async def is_chat_only(
+        self, tool_call: LanguageModelFunction | None = None, **kwargs
+    ) -> bool:
+        if await super().is_chat_only(**kwargs):
+            return True
+        if (
+            tool_call
+            and isinstance(tool_call.arguments, dict)
+            and tool_call.arguments.get("chat_only") is True
+        ):
+            return True
+        return False
+
+    @override
+    def tool_description(self) -> LanguageModelToolDescription:
+        # Conditionally set the type based on config
+        search_string_type = (
+            list[str]
+            if self.config.experimental_features.enable_multiple_search_strings_execution
+            else str
+        )
+
+        internal_search_tool_input = create_model(
+            "InternalSearchToolInput",
+            search_string=(
+                search_string_type,
+                Field(description=self.config.param_description_search_string),
+            ),
+            language=(
+                str,
+                Field(description=self.config.param_description_language),
+            ),
+        )
+        return LanguageModelToolDescription(
+            name=self.name,
+            description=self.config.tool_description,
+            parameters=internal_search_tool_input,
+        )
+
+    def tool_description_for_system_prompt(self) -> str:
+        return self.config.tool_description_for_system_prompt
+
+    def tool_format_information_for_system_prompt(self) -> str:
+        return self.config.tool_format_information_for_system_prompt
+
+    def evaluation_check_list(self) -> list[EvaluationMetricName]:
+        return self.config.evaluation_check_list
+
+    def get_evaluation_checks_based_on_tool_response(
+        self, tool_response: ToolCallResponse
+    ) -> list[EvaluationMetricName]:
+        evaluation_check_list = self.evaluation_check_list()
+
+        # Check if the tool response is empty
+        if not tool_response.content_chunks:
+            return []
+        return evaluation_check_list
+
+    # TODO: find a solution for tracking
+    # @track(name="internal_search_tool_run")
+    async def run(self, tool_call: LanguageModelFunction) -> ToolCallResponse:
+        """
+        Perform a search in the Vector DB based on the user's message and generate a response.
+        """
+        if (
+            tool_call.arguments is None
+            or not isinstance(tool_call.arguments, dict)
+            or (
+                "search_strings" not in tool_call.arguments
+                and "search_string"
+                not in tool_call.arguments  # Backwards compatibility
+            )
+        ):
+            self.logger.error("Tool call arguments are missing or invalid")
+            return ToolCallResponse(
+                id=tool_call.id,  # type: ignore
+                name=self.name,
+                content_chunks=[],
+                debug_info={},
+            )
+
+        # Extract the search strings (handle both new and old parameter names)
+        search_strings_data = tool_call.arguments.get(
+            "search_strings", tool_call.arguments.get("search_string")
+        )
+        # Ensure it's always a list for the progress message
+        search_strings_list: list[str] = []
+        if isinstance(search_strings_data, str):
+            search_strings_list = [search_strings_data]
+        elif isinstance(search_strings_data, list):
+            search_strings_list = search_strings_data
+        else:
+            raise ValueError("Invalid search strings data")
+
+        await self.post_progress_message(f"{'; '.join(search_strings_list)}", tool_call)
+
+        selected_chunks = await self.search(
+            **tool_call.arguments,
+            tool_call=tool_call,  # Need to pass tool_call to post_progress_message
+        )
+
+        ## Modify metadata in chunks
+        selected_chunks = append_metadata_in_chunks(
+            chunks=selected_chunks,
+            metadata_chunk_sections=self.config.metadata_chunk_sections,
+        )
+
+        tool_response = ToolCallResponse(
+            id=tool_call.id,  # type: ignore
+            name=self.name,
+            content_chunks=selected_chunks,
+            debug_info=self.debug_info,
+        )
+
+        if self.tool_progress_reporter:
+            await self.tool_progress_reporter.notify_from_tool_call(
+                tool_call=tool_call,
+                name=f"**{self.tool_execution_message_name}**",
+                message=f"{'; '.join(search_strings_list)}",
+                state=ProgressState.FINISHED,
+            )
+
+        return tool_response
+
+    ## Note: This function is only used by the Investment Research Agent and Agentic Search. Once these agents are moved out of the monorepo, this function should be removed.
+    def get_tool_call_result_for_loop_history(
+        self,
+        tool_response: ToolCallResponse,
+        agent_chunks_handler: AgentChunksHandler,
+    ) -> LanguageModelMessage:
+        """
+        Process the results of the tool.
+        Args:
+            tool_response: The tool response.
+            loop_history: The loop history.
+        Returns:
+            The tool result to append to the loop history.
+        """
+        self.logger.debug(
+            f"Appending tool call result to history: {tool_response.name}"
+        )
+        # Initialize content_chunks if None
+        content_chunks = tool_response.content_chunks or []
+
+        # Get the maximum source number in the loop history
+        max_source_number = len(agent_chunks_handler.chunks)
+
+        # Transform content chunks into sources to be appended to tool result
+        sources, _ = transform_chunks_to_string(
+            content_chunks,
+            max_source_number,
+        )
+
+        # Append the result to the history
+        return LanguageModelToolMessage(
+            content=sources,
+            tool_call_id=tool_response.id,  # type: ignore
+            name=tool_response.name,
+        )
+
+
+ToolFactory.register_tool(InternalSearchTool, InternalSearchConfig)

unique_internal_search-1.2.2/unique_internal_search/uploaded_search/config.py

@@ -0,0 +1,29 @@
+from unique_internal_search.config import InternalSearchConfig
+from unique_internal_search.uploaded_search.prompts import (
+    DEFAULT_LANGUAGE_PARAM_DESCRIPTION,
+    DEFAULT_SEARCH_STRING_PARAM_DESCRIPTION,
+    DEFAULT_TOOL_DESCRIPTION_FOR_SYSTEM_PROMPT,
+    DEFAULT_TOOL_FORMAT_INFORMATION_FOR_SYSTEM_PROMPT,
+)
+from unique_internal_search.validators import get_string_field_with_pattern_validation
+
+
+class UploadedSearchConfig(InternalSearchConfig):
+    param_description_search_string: str = get_string_field_with_pattern_validation(
+        DEFAULT_SEARCH_STRING_PARAM_DESCRIPTION,
+        description="`search_string` parameter description.",
+    )
+    param_description_language: str = get_string_field_with_pattern_validation(
+        DEFAULT_LANGUAGE_PARAM_DESCRIPTION,
+        description="`language` parameter description.",
+    )
+    tool_description_for_system_prompt: str = get_string_field_with_pattern_validation(
+        DEFAULT_TOOL_DESCRIPTION_FOR_SYSTEM_PROMPT,
+        description="Tool description for the system prompt.",
+    )
+    tool_format_information_for_system_prompt: str = (
+        get_string_field_with_pattern_validation(
+            DEFAULT_TOOL_FORMAT_INFORMATION_FOR_SYSTEM_PROMPT,
+            description="Tool format information for the system prompt.",
+        )
+    )

unique_internal_search-1.2.2/unique_internal_search/uploaded_search/prompts.py

@@ -0,0 +1,57 @@
+DEFAULT_TOOL_DESCRIPTION_FOR_SYSTEM_PROMPT = (
+    "You can use the UploadedSearch tool to access and analyze documents uploaded by users during a chat. This tool is designed to handle a variety of document-related tasks, including summarization, explanation, and detailed information retrieval. "
+    "Use cases for the UploadedSearch tool include:\n"
+    "- Document Analysis: When a user uploads a document and asks for a summary, explanation, or specific details, this tool can extract and provide the requested information.\n"
+    "- Named Document Queries: If a user refers to a previously uploaded document by name (e.g., 'What does the Q2_Report.pdf say about revenue?'), this tool can locate and analyze the document to answer the query.\n"
+    "- Policy and Procedure Verification: Use the tool to find the most current company policies, procedures, or guidelines within uploaded documents.\n"
+    "- Project-Specific Information: Access project documents, reports, or meeting notes uploaded by users to provide precise details.\n"
+    "- Confidential and Proprietary Information: Ensure that sensitive topics requiring proprietary knowledge or confidential data are sourced securely from uploaded documents.\n\n"
+    "**Instruction Query Splitting**\n"
+    "You should split the user question into multiple search strings when the user's question needs to be decomposed / rewritten to find different facts. Perform an individual tool call for each search string. Avoid overly broad queries that may return unrelated results. Ensure the search string is specific and relevant to the uploaded document(s).\n\n"
+    "Examples:\n"
+    'User: "What does the Q2_Report.pdf say about revenue and expenses?" => search strings: ["Q2_Report.pdf revenue", "Q2_Report.pdf expenses"]\n'
+    'User: "Summarize the uploaded document." => search string: ["Summarize the uploaded document"]\n'
+    "**The currently uploaded documents are the following**\n\n"
+)
+
+DEFAULT_TOOL_FORMAT_INFORMATION_FOR_SYSTEM_PROMPT = (
+    "Whenever you use information retrieved with the UploadedSearch, you must adhere to strict reference guidelines. "
+    "You must strictly reference each fact used with the `source_number` of the corresponding passage, in the following format: '[source<source_number>]'.\n\n"
+    "Example:\n"
+    "- The revenue for Q2 was $5M , while expenses were $3M .\n"
+    "- The uploaded document highlights a 20% increase in productivity .\n\n"
+    "A fact is preferably referenced by ONLY ONE source, e.g [sourceX], which should be the most relevant source for the fact.\n"
+    "Follow these guidelines closely and be sure to use the proper `source_number` when referencing facts.\n"
+    "Make sure that your reference follow the format [sourceX] and that the source number is correct.\n"
+    "Source is written in singular form and the number is written in digits.\n\n"
+    "IT IS VERY IMPORTANT TO FOLLOW THESE GUIDELINES!!\n"
+    "NEVER CITE A source_number THAT YOU DON'T SEE IN THE TOOL CALL RESPONSE!!!\n"
+    "The source_number in old assistant messages are no longer valid.\n"
+    "EXAMPLE: If you see  and  in the assistant message, you can't use  again in the next assistant message, this has to be the number you find in the message with role 'tool'.\n"
+    "BE AWARE: All tool calls have been filtered to remove uncited sources. Tool calls return much more data than you see.\n\n"
+    "### Internal Document Answering Protocol for Uploaded Documents\n"
+    "When assisting users with uploaded documents, follow\n"
+    "this structured approach to ensure precise, well-grounded,\n"
+    "and context-aware responses:\n\n"
+    "#### 1. Locate and Prioritize Relevant Information\n"
+    "Focus on the **most relevant sections** of the uploaded document.\n"
+    "Prioritize documents that are:\n"
+    "- **Directly referenced by the user** (e.g., by name or context).\n"
+    "- **Recently uploaded** or actively discussed.\n\n"
+    "#### 2. Source Reliability Guidelines\n"
+    "- Prioritize information that is:\n"
+    "  - **Clearly stated in the document**.\n"
+    "  - **Part of finalized or approved sections**.\n"
+    "- Be cautious with:\n"
+    "  - Drafts or incomplete sections.\n"
+    "  - Ambiguous or conflicting information.\n\n"
+    "#### 3. Acknowledge Limitations\n"
+    "- If no relevant information is found, or the document is unclear, state this explicitly.\n"
+    "- Indicate where further clarification or investigation may be required."
+)
+
+DEFAULT_TOOL_DESCRIPTION = "Search within uploaded documents for information on policies, procedures, benefits, projects, or specific details. This tool is ideal for analyzing user-uploaded files and extracting relevant insights."
+DEFAULT_SEARCH_STRING_PARAM_DESCRIPTION = "An expanded term optimized for vector and full-text search based on the user’s query. It must be in English."
+DEFAULT_LANGUAGE_PARAM_DESCRIPTION = (
+    "The language in which the user’s query is written."
+)

unique_internal_search-1.2.2/unique_internal_search/uploaded_search/service.py

@@ -0,0 +1,126 @@
+from pydantic import Field, create_model
+from typing_extensions import override
+from unique_toolkit import ContentService
+from unique_toolkit.agentic.evaluation.schemas import EvaluationMetricName
+from unique_toolkit.agentic.tools.factory import ToolFactory
+from unique_toolkit.agentic.tools.schemas import ToolCallResponse
+from unique_toolkit.agentic.tools.tool import Tool
+from unique_toolkit.agentic.tools.tool_progress_reporter import (
+    ProgressState,
+    ToolProgressReporter,
+)
+from unique_toolkit.app.schemas import BaseEvent, ChatEvent
+from unique_toolkit.chat.service import LanguageModelToolDescription
+from unique_toolkit.language_model.schemas import (
+    LanguageModelFunction,
+)
+
+from unique_internal_search.service import InternalSearchTool
+from unique_internal_search.uploaded_search.config import UploadedSearchConfig
+
+
+class UploadedSearchTool(Tool[UploadedSearchConfig]):
+    name = "UploadedSearch"
+
+    def __init__(
+        self,
+        config: UploadedSearchConfig,
+        event: BaseEvent,
+        tool_progress_reporter: ToolProgressReporter,
+        *args,
+        **kwargs,
+    ):
+        self._tool_progress_reporter = tool_progress_reporter
+        self._content_service = ContentService.from_event(event)
+        self._config = config
+        config.chat_only = True
+        self._internal_search_tool = InternalSearchTool(
+            config, event, None, *args, **kwargs
+        )
+        if isinstance(event, ChatEvent):
+            self._user_query = event.payload.user_message.text
+        else:
+            self._user_query = None
+
+    async def post_progress_message(
+        self, message: str, tool_call: LanguageModelFunction, **kwargs
+    ):
+        if self._tool_progress_reporter:
+            await self._tool_progress_reporter.notify_from_tool_call(
+                tool_call=tool_call,
+                name="**Search Uploaded Document**",
+                message=message,
+                state=ProgressState.RUNNING,
+            )
+
+    @override
+    def tool_description(self) -> LanguageModelToolDescription:
+        internal_search_tool_input = create_model(
+            "InternalSearchToolInput",
+            search_string=(
+                str,
+                Field(description=self._config.param_description_search_string),
+            ),
+            language=(
+                str,
+                Field(description=self._config.param_description_language),
+            ),
+        )
+        return LanguageModelToolDescription(
+            name=self.name,
+            description=self._config.tool_description,
+            parameters=internal_search_tool_input,
+        )
+
+    def tool_description_for_system_prompt(self) -> str:
+        documents = self._content_service.get_documents_uploaded_to_chat()
+        list_all_documents = "".join([f"- {doc.title or doc.key}" for doc in documents])
+        return self._config.tool_description_for_system_prompt + list_all_documents
+
+    def tool_format_information_for_system_prompt(self) -> str:
+        return self._config.tool_format_information_for_system_prompt
+
+    def evaluation_check_list(self) -> list[EvaluationMetricName]:
+        return self._config.evaluation_check_list
+
+    def get_evaluation_checks_based_on_tool_response(
+        self, tool_response: ToolCallResponse
+    ) -> list[EvaluationMetricName]:
+        evaluation_check_list = self.evaluation_check_list()
+        return evaluation_check_list
+
+    async def run(self, tool_call: LanguageModelFunction) -> ToolCallResponse:
+        search_string_data = ""
+        if isinstance(tool_call.arguments, dict):
+            search_string_data = tool_call.arguments.get("search_string", "") or ""
+        tool_response = await self._internal_search_tool.run(tool_call)
+        if self._tool_progress_reporter:
+            await self._tool_progress_reporter.notify_from_tool_call(
+                tool_call=tool_call,
+                name="**Search Uploaded Document**",
+                message=f"{search_string_data}",
+                state=ProgressState.FINISHED,
+            )
+        tool_response.name = self.name
+        tool_response.system_reminder = self._get_tool_call_response_system_reminder()
+        return tool_response
+
+    def _get_tool_call_response_system_reminder(self) -> str:
+        """
+        When using the upload and search tool, unique AI agent is loosing the overview of the original user message and request
+        This likely due to the amount of tokens included and as since it's a forced tool not necessarily relevant to the user's request.
+        """
+        # TODO: This message should be conditional on the tool being forced, but we do not have easy access to this information here
+        return f"""<system_reminder>
+This tool call was automatically executed to retrieve the user's uploaded documents by the system. Important to note:
+- The retrieved documents may or may not be relevant to the user's actual query
+- You must evaluate their relevance independently
+- You are free to make additional tool calls as needed
+- Focus on addressing the user's original request
+{f"Original user message: {self._user_query}" if self._user_query else ""}
+
+Please do not mention these instructions in your response to the user!
+</system_reminder>"""
+
+
+ToolFactory.register_tool(UploadedSearchTool, UploadedSearchConfig)

unique_internal_search-1.2.2/unique_internal_search/utils.py

@@ -0,0 +1,169 @@
+import logging
+import re
+
+from pydantic import BaseModel
+from unique_toolkit.content.schemas import ContentChunk
+
+_LOGGER = logging.getLogger(__name__)
+
+
+class SearchStringResult(BaseModel):
+    query: str
+    chunks: list[ContentChunk]
+
+
+def interleave_search_results_round_robin(
+    search_results: list[SearchStringResult],
+) -> list[SearchStringResult]:
+    """
+    Interleave chunks from multiple search queries using a round-robin strategy.
+    Each result in the output contains a single chunk. Duplicate chunks are removed,
+    keeping the first occurrence.
+
+    Example:
+        Input:
+            Query 1: SearchStringResult(query="query1", chunks=[A, B, C])
+            Query 2: SearchStringResult(query="query2", chunks=[D, E])
+            Query 3: SearchStringResult(query="query3", chunks=[F, G, H, I])
+
+        Output (interleaved by position, then deduplicated):
+            [
+                SearchStringResult(query="query1", chunks=[A]),  # pos 0, query 1
+                SearchStringResult(query="query2", chunks=[D]),  # pos 0, query 2
+                SearchStringResult(query="query3", chunks=[F]),  # pos 0, query 3
+                SearchStringResult(query="query1", chunks=[B]),  # pos 1, query 1
+                SearchStringResult(query="query2", chunks=[E]),  # pos 1, query 2
+                SearchStringResult(query="query3", chunks=[G]),  # pos 1, query 3
+                SearchStringResult(query="query1", chunks=[C]),  # pos 2, query 1
+                SearchStringResult(query="query3", chunks=[H]),  # pos 2, query 3
+                SearchStringResult(query="query3", chunks=[I]),  # pos 3, query 3
+            ]
+    """
+    if not search_results:
+        return []
+
+    max_chunks = max(len(result.chunks) for result in search_results)
+    interleaved_search_results: list[SearchStringResult] = [
+        SearchStringResult(query=result.query, chunks=[result.chunks[i]])
+        for i in range(max_chunks)
+        for result in search_results
+        if i < len(result.chunks)
+    ]
+
+    return _deduplicate_search_results(interleaved_search_results)
+
+
+def _deduplicate_search_results(
+    search_results: list[SearchStringResult],
+) -> list[SearchStringResult]:
+    """
+    Remove duplicate chunks from the search results based on their `chunk_id`.
+
+    This function preserves the order of occurrences, keeping the first occurrence
+    of each unique `chunk_id`. If a chunk has no `chunk_id`, it will be ignored.
+    Duplicate chunks share the same `chunk_id`.
+
+    Args:
+        search_results (list[SearchStringResult]): A list of search results, where each
+            result contains chunks with potential duplicate `chunk_id`s.
+
+    Returns:
+        list[SearchStringResult]: A deduplicated list of search results with unique `chunk_id` chunks.
+    """
+    seen_chunk_ids: set[str] = set()
+    deduplicated_search_results: list[SearchStringResult] = []
+
+    counter_chunks = 0
+    for result in search_results:
+        for chunk in result.chunks:
+            if chunk.chunk_id and chunk.chunk_id not in seen_chunk_ids:
+                counter_chunks += 1
+                seen_chunk_ids.add(chunk.chunk_id)
+                deduplicated_search_results.append(
+                    SearchStringResult(query=result.query, chunks=[chunk])
+                )
+
+    if removed := counter_chunks - len(deduplicated_search_results):
+        _LOGGER.info(
+            f"Removed {removed} duplicate chunks ({len(deduplicated_search_results)}/{counter_chunks} unique)"
+        )
+
+    return deduplicated_search_results
+
+
+def append_metadata_in_chunks(
+    chunks: list[ContentChunk],
+    metadata_chunk_sections: dict[str, str] | None = None,
+) -> list[ContentChunk]:
+    """
+    Append metadata to chunks.
+    Args:
+        chunks: List of ContentChunk objects
+        metadata_chunk_sections: Dictionary of metadata sections to add to the chunk text
+    Returns:
+        List of ContentChunk objects with metadata appended
+    """
+    if metadata_chunk_sections is None:
+        return chunks
+    for chunk in chunks:
+        if chunk.metadata is None:
+            continue
+        chunk = _append_metadata_in_chunk(
+            chunk=chunk, metadata_chunk_sections=metadata_chunk_sections
+        )
+    return chunks
+
+
+def _append_metadata_in_chunk(
+    chunk: ContentChunk, metadata_chunk_sections: dict[str, str]
+) -> ContentChunk:
+    """
+    Format chunk text by prepending metadata according to sections config.
+    Args:
+        chunk: ContentChunk object
+        metadata_chunk_sections: Dictionary of metadata sections to add to the chunk text
+    Returns:
+        Formatted text with metadata prepended
+    """
+    meta_dict = chunk.metadata.model_dump(exclude_none=True, by_alias=True)
+
+    parts: list[str] = []
+    for key, template in metadata_chunk_sections.items():
+        if key in meta_dict:
+            formatted_section = template.format(meta_dict[key])
+            parts.append(formatted_section)
+
+    # Combine metadata parts with the main text
+    if parts:
+        chunk.text = "\n".join(parts) + "\n" + chunk.text
+
+    return chunk
+
+
+def clean_search_string(search_string: str) -> str:
+    """
+    Remove QDF (QueryDeservedFreshness) and boost operators from search string.
+
+    Examples:
+        '+(GPT4) performance on +(MMLU) benchmark --QDF=1'
+        -> 'GPT4 performance on MMLU benchmark'
+
+        'Best practices for +(security) and +(privacy) for +(cloud storage) --QDF=2'
+        -> 'Best practices for security and privacy for cloud storage'
+
+    Args:
+        search_string: Raw search string that may contain operators
+
+    Returns:
+        Cleaned search string without operators
+    """
+    # Remove --QDF=<number> operator (at the end of the string)
+    cleaned = re.sub(r"\s*--QDF=\d+\s*$", "", search_string)
+
+    # Remove +(...) boost operators - replace with just the content inside parentheses
+    cleaned = re.sub(r"\+\(([^)]+)\)", r"\1", cleaned)
+
+    # Clean up any extra whitespace
+    cleaned = " ".join(cleaned.split())
+
+    return cleaned.strip()

unique_internal_search-1.2.2/unique_internal_search/validators.py

@@ -0,0 +1,86 @@
+import enum
+import re
+from string import Template
+from typing import Any
+
+from pydantic import (
+    Field,
+)
+
+
+class PromptTemplatingEngine(enum.Enum):
+    STRING_TEMPLATE = enum.auto()
+
+
+def check_placeholder_valid(
+    placeholder: str,
+    templating_engine: PromptTemplatingEngine = PromptTemplatingEngine.STRING_TEMPLATE,
+) -> bool:
+    match templating_engine:
+        case PromptTemplatingEngine.STRING_TEMPLATE:
+            return (
+                re.fullmatch(Template.idpattern, placeholder, re.IGNORECASE) is not None
+            )
+
+
+def get_prompt_placeholder_regexp(
+    *placeholders: str,
+    templating_engine: PromptTemplatingEngine = PromptTemplatingEngine.STRING_TEMPLATE,
+) -> re.Pattern:
+    for placeholder in placeholders:
+        if not check_placeholder_valid(placeholder, templating_engine):
+            raise ValueError(f"Invalid placeholder: {placeholder}")
+
+    match templating_engine:
+        case PromptTemplatingEngine.STRING_TEMPLATE:
+            placeholder_patterns = [
+                rf"(?=.*(?:\$\{{{p}\}}|\${p}))" for p in placeholders
+            ]
+            pattern = "".join(placeholder_patterns)
+            return re.compile(pattern, re.DOTALL)
+        # We will add other templating engines here, such as Jinja2.
+
+
+def get_prompt_placeholder_regexp_from_text(
+    text: str,
+    templating_engine: PromptTemplatingEngine = PromptTemplatingEngine.STRING_TEMPLATE,
+) -> re.Pattern:
+    match templating_engine:
+        case PromptTemplatingEngine.STRING_TEMPLATE:
+            return get_prompt_placeholder_regexp(
+                *Template(text).get_identifiers(),
+                templating_engine=templating_engine,
+            )
+
+
+def get_string_field_with_pattern_validation(
+    prompt_template: str,
+    templating_engine: PromptTemplatingEngine = PromptTemplatingEngine.STRING_TEMPLATE,
+    **kwargs,
+) -> Any:
+    """Create a Pydantic Field with validation for prompt template placeholders.
+
+    Args:
+        prompt_template: The prompt template string containing placeholders.
+        templating_engine: The engine used for template processing. Defaults to STRING_TEMPLATE.
+        **kwargs: Additional keyword arguments to pass to pydantic.Field.
+                  Note that `default` will be ignored if present.
+
+    Returns:
+        pydantic.FieldInfo: A FieldInfo instance with the default value and placeholder validation pattern.
+
+    Example:
+        class ServiceConfig(BaseModel):
+            prompt: str = get_prompt_field_from_default(
+                "Hello ${name}!"
+            ) # Creates a Field with pattern validation for the "name" placeholder
+    """
+    pattern = get_prompt_placeholder_regexp_from_text(
+        prompt_template, templating_engine
+    )
+    if pattern.pattern:
+        kwargs["pattern"] = pattern
+
+    kwargs["default"] = prompt_template
+
+    return Field(**kwargs)