hamtaa-texttools 1.1.18__py3-none-any.whl → 1.1.20__py3-none-any.whl

{hamtaa_texttools-1.1.18.dist-info → hamtaa_texttools-1.1.20.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 1.1.18
+Version: 1.1.20
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
 License: MIT License
@@ -61,29 +61,59 @@ Each tool is designed to work with structured outputs (JSON / Pydantic).
 - **`summarize()`** - Text summarization
 - **`translate()`** - Text translation between languages
 - **`propositionize()`** - Convert text to atomic independence meaningful sentences 
+- **`check_fact()`** - Check a statement is relevant to source text or not 
 - **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
 
 ---
 
+## 📊 Tool Quality Tiers
+
+| Status | Meaning | Use in Production? |
+|--------|---------|-------------------|
+| **✅ Production** | Evaluated, tested, stable. | **Yes** - ready for reliable use. |
+| **🧪 Experimental** | Added to the package but **not fully evaluated**. Functional, but quality may vary. | **Use with caution** - outputs not yet validated. |
+
+### Current Status
+**Production Tools:**
+- `categorize()` (list mode)
+- `extract_keywords()`
+- `extract_entities()`
+- `is_question()`
+- `text_to_question()`
+- `merge_questions()`
+- `rewrite()`
+- `subject_to_question()`
+- `summarize()`
+- `run_custom()` (fine in most cases)
+
+**Experimental Tools:**
+- `categorize()` (tree mode)
+- `translate()`
+- `propositionize()`
+- `check_fact()`
+- `run_custom()` (not evaluated in all scenarios)
+
+---
+
 ## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator` and `priority` parameters
 
 TextTools provides several optional flags to customize LLM behavior:
 
-- **`with_analysis (bool)`** → Adds a reasoning step before generating the final output.
+- **`with_analysis: bool`** → Adds a reasoning step before generating the final output.
 **Note:** This doubles token usage per call because it triggers an additional LLM request.
 
-- **`logprobs (bool)`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
+- **`logprobs: bool`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
 **Note:** This feature works if it's supported by the model.
 
-- **`output_lang (str)`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
+- **`output_lang: str`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
 
-- **`user_prompt (str)`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
+- **`user_prompt: str`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
 
-- **`temperature (float)`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
+- **`temperature: float`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
 
-- **`validator (Callable)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a bool (True if there were no problem, False if the validation fails.) If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can specify `max_validation_retries=<N>` to change the number of retries.
+- **`validator: Callable (Experimental)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a bool (True if there were no problem, False if the validation fails.) If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can specify `max_validation_retries=<N>` to change the number of retries.
 
-- **`priority (int)`** → Task execution priority level. Higher values = higher priority. Affects processing order in queues.
+- **`priority: int (Experimental)`** → Task execution priority level. Higher values = higher priority. Affects processing order in queues.
 **Note:** This feature works if it's supported by the model and vLLM.
 
 **Note:** There might be some tools that don't support some of the parameters above.

hamtaa_texttools-1.1.20.dist-info/RECORD

@@ -0,0 +1,33 @@
+hamtaa_texttools-1.1.20.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
+texttools/__init__.py,sha256=CmCS9dEvO6061GiJ8A7gD3UAhCWHTkaID9q3Krlyq_o,311
+texttools/batch/batch_config.py,sha256=m1UgILVKjNdWE6laNbfbG4vgi4o2fEegGZbeoam6pnY,749
+texttools/batch/batch_runner.py,sha256=fVqgFDOyv8DqaNmRJQjt75wOxXTKPstisHycpt0LwcM,10026
+texttools/batch/internals/batch_manager.py,sha256=6HfsexU0PHGGBH7HKReZ-CQxaQI9DXYKAPsFXxovb_I,8740
+texttools/batch/internals/utils.py,sha256=8uNqvPHkEDFpiPp2Nyu-1nP4R-Tq8FwuSGMNSjcBogY,348
+texttools/internals/async_operator.py,sha256=rgQuvT-hl53stClVojso9FgmKhK98nm_2Cdl5WROMoc,8399
+texttools/internals/exceptions.py,sha256=h_yp_5i_5IfmqTBQ4S6ZOISrrliJBQ3HTEAjwJXrplk,495
+texttools/internals/models.py,sha256=Ro8d875_xjMHwwJIz3D_-VWxQ2WOcLLIKxsleSvqPDE,5659
+texttools/internals/operator_utils.py,sha256=jphe1DvYgdpRprZFv23ghh3bNYETvVTQ8ZZ8HPwwRVo,2759
+texttools/internals/prompt_loader.py,sha256=i4OxcVJTjHFKPSoC-DWZUM3Vf8ye_vbD7b6t3N2qB08,3972
+texttools/internals/sync_operator.py,sha256=Q0I2LlGF8MEPUCnwOOErOlxF3gjNxEvNS3oM9V6jazE,8296
+texttools/internals/text_to_chunks.py,sha256=vY3odhgCZK4E44k_SGlLoSiKkdN0ib6-lQAsPcplAHA,3843
+texttools/prompts/README.md,sha256=-5YO93CN93QLifqZpUeUnCOCBbDiOTV-cFQeJ7Gg0I4,1377
+texttools/prompts/categorize.yaml,sha256=F7VezB25B_sT5yoC25ezODBddkuDD5lUHKetSpx9FKI,2743
+texttools/prompts/check_fact.yaml,sha256=5kpBjmfZxgp81Owc8-Pd0U8-cZowFGRdYlGTFQLYQ9o,702
+texttools/prompts/extract_entities.yaml,sha256=56N3iFH1KbGLqloYBLOW-12SchVayqbHgaQ5-8JTbeY,610
+texttools/prompts/extract_keywords.yaml,sha256=Vj4Tt3vT6LtpOo_iBZPo9oWI50oVdPGXe5i8yDR8ex4,3177
+texttools/prompts/is_question.yaml,sha256=d0-vKRbXWkxvO64ikvxRjEmpAXGpCYIPGhgexvPPjws,471
+texttools/prompts/merge_questions.yaml,sha256=0J85GvTirZB4ELwH3sk8ub_WcqqpYf6PrMKr3djlZeo,1792
+texttools/prompts/propositionize.yaml,sha256=kdj-UxPOYcLSTLF7cWARDxxTxSFB0qRBaRujdThPDxw,1380
+texttools/prompts/rewrite.yaml,sha256=LO7He_IA3MZKz8a-LxH9DHJpOjpYwaYN1pbjp1Y0tFo,5392
+texttools/prompts/run_custom.yaml,sha256=6oiMYOo_WctVbOmE01wZzI1ra7nFDMJzceTTtnGdmOA,126
+texttools/prompts/subject_to_question.yaml,sha256=C7x7rNNm6U_ZG9HOn6zuzYOtvJUZ2skuWbL1-aYdd3E,1147
+texttools/prompts/summarize.yaml,sha256=o6rxGPfWtZd61Duvm8NVvCJqfq73b-wAuMSKR6UYUqY,459
+texttools/prompts/text_to_question.yaml,sha256=fnxDpUnnOEmK0yyFU5F4ItBqCnegoUCXhSTFjiTy18Y,1005
+texttools/prompts/translate.yaml,sha256=mGT2uBCei6uucWqVbs4silk-UV060v3G0jnt0P6sr50,634
+texttools/tools/async_tools.py,sha256=0ubBHyx3ii1MOwo8X_ZMfqaH43y07DaF_rk7s7DPjVA,52331
+texttools/tools/sync_tools.py,sha256=7ZY2Cs7W0y3m8D9I70Tqk9pMisIQeIzdAWwOh1A1hRc,52137
+hamtaa_texttools-1.1.20.dist-info/METADATA,sha256=s52LtRd3UQOcmNgOJ_riHjMgMC9pS2mmks2Ys0zWhXk,10610
+hamtaa_texttools-1.1.20.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+hamtaa_texttools-1.1.20.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-1.1.20.dist-info/RECORD,,

texttools/batch/batch_runner.py

@@ -2,7 +2,7 @@ import json
 import os
 import time
 from pathlib import Path
-from typing import Any, Type, TypeVar
+from typing import Type, TypeVar
 import logging
 
 from dotenv import load_dotenv
@@ -11,7 +11,7 @@ from pydantic import BaseModel
 
 from texttools.batch.internals.batch_manager import BatchManager
 from texttools.batch.batch_config import BatchConfig
-from texttools.internals.models import StrOutput
+from texttools.internals.models import Str
 from texttools.internals.exceptions import TextToolsError, ConfigurationError
 
 # Base Model type for output models
@@ -26,7 +26,7 @@ class BatchJobRunner:
     """
 
     def __init__(
-        self, config: BatchConfig = BatchConfig(), output_model: Type[T] = StrOutput
+        self, config: BatchConfig = BatchConfig(), output_model: Type[T] = Str
     ):
         try:
             self._config = config
@@ -38,7 +38,7 @@ class BatchJobRunner:
             self._output_model = output_model
             self._manager = self._init_manager()
             self._data = self._load_data()
-            self._parts: list[list[dict[str, Any]]] = []
+            self._parts: list[list[dict[str, object]]] = []
             # Map part index to job name
             self._part_idx_to_job_name: dict[int, str] = {}
             # Track retry attempts per part
@@ -130,8 +130,8 @@ class BatchJobRunner:
 
     def _save_results(
         self,
-        output_data: list[dict[str, Any]] | dict[str, Any],
-        log: list[Any],
+        output_data: list[dict[str, object]] | dict[str, object],
+        log: list[object],
         part_idx: int,
     ):
         part_suffix = f"_part_{part_idx + 1}" if len(self._parts) > 1 else ""

texttools/batch/internals/batch_manager.py

@@ -1,7 +1,7 @@
 import json
 import uuid
 from pathlib import Path
-from typing import Any, Type, TypeVar
+from typing import Type, TypeVar
 import logging
 
 from pydantic import BaseModel
@@ -31,7 +31,7 @@ class BatchManager:
         prompt_template: str,
         state_dir: Path = Path(".batch_jobs"),
         custom_json_schema_obj_str: dict | None = None,
-        **client_kwargs: Any,
+        **client_kwargs: object,
     ):
         self._client = client
         self._model = model
@@ -51,7 +51,7 @@ class BatchManager:
     def _state_file(self, job_name: str) -> Path:
         return self._state_dir / f"{job_name}.json"
 
-    def _load_state(self, job_name: str) -> list[dict[str, Any]]:
+    def _load_state(self, job_name: str) -> list[dict[str, object]]:
         """
         Loads the state (job information) from the state file for the given job name.
         Returns an empty list if the state file does not exist.
@@ -62,7 +62,7 @@ class BatchManager:
                 return json.load(f)
         return []
 
-    def _save_state(self, job_name: str, jobs: list[dict[str, Any]]) -> None:
+    def _save_state(self, job_name: str, jobs: list[dict[str, object]]) -> None:
         """
         Saves the job state to the state file for the given job name.
         """
@@ -77,11 +77,11 @@ class BatchManager:
         if path.exists():
             path.unlink()
 
-    def _build_task(self, text: str, idx: str) -> dict[str, Any]:
+    def _build_task(self, text: str, idx: str) -> dict[str, object]:
         """
         Builds a single task dictionary for the batch job, including the prompt, model, and response format configuration.
         """
-        response_format_config: dict[str, Any]
+        response_format_config: dict[str, object]
 
         if self._custom_json_schema_obj_str:
             response_format_config = {

texttools/batch/internals/utils.py

@@ -1,6 +1,3 @@
-from typing import Any
-
-
 def export_data(data) -> list[dict[str, str]]:
     """
     Produces a structure of the following form from an initial data structure:
@@ -9,7 +6,7 @@ def export_data(data) -> list[dict[str, str]]:
     return data
 
 
-def import_data(data) -> Any:
+def import_data(data) -> object:
     """
     Takes the output and adds and aggregates it to the original structure.
     """

texttools/internals/async_operator.py

@@ -1,4 +1,4 @@
-from typing import Any, TypeVar, Type
+from typing import TypeVar, Type
 from collections.abc import Callable
 import logging
 
@@ -7,7 +7,6 @@ from pydantic import BaseModel
 
 from texttools.internals.models import ToolOutput
 from texttools.internals.operator_utils import OperatorUtils
-from texttools.internals.formatters import Formatter
 from texttools.internals.prompt_loader import PromptLoader
 from texttools.internals.exceptions import (
     TextToolsError,
@@ -77,7 +76,7 @@ class AsyncOperator:
         logprobs: bool = False,
         top_logprobs: int = 3,
         priority: int | None = 0,
-    ) -> tuple[T, Any]:
+    ) -> tuple[T, object]:
         """
         Parses a chat completion using OpenAI's structured output format.
         Returns both the parsed object and the raw completion for logprobs.
@@ -124,7 +123,7 @@ class AsyncOperator:
         temperature: float,
         logprobs: bool,
         top_logprobs: int | None,
-        validator: Callable[[Any], bool] | None,
+        validator: Callable[[object], bool] | None,
         max_validation_retries: int | None,
         # Internal parameters
         prompt_file: str,
@@ -138,7 +137,6 @@ class AsyncOperator:
         """
         try:
             prompt_loader = PromptLoader()
-            formatter = Formatter()
             output = ToolOutput()
 
             # Prompt configs contain two keys: main_template and analyze template, both are string
@@ -177,7 +175,7 @@ class AsyncOperator:
                 OperatorUtils.build_user_message(prompt_configs["main_template"])
             )
 
-            messages = formatter.user_merge_format(messages)
+            messages = OperatorUtils.user_merge_format(messages)
 
             if logprobs and (not isinstance(top_logprobs, int) or top_logprobs < 2):
                 raise ValueError("top_logprobs should be an integer greater than 1")

texttools/internals/models.py

@@ -1,12 +1,12 @@
 from datetime import datetime
-from typing import Type, Any, Literal
+from typing import Type, Literal
 
 from pydantic import BaseModel, Field, create_model
 
 
 class ToolOutput(BaseModel):
-    result: Any = None
-    logprobs: list[dict[str, Any]] = []
+    result: object = None
+    logprobs: list[dict[str, object]] = []
     analysis: str = ""
     process: str | None = None
     processed_at: datetime = datetime.now()
@@ -22,23 +22,23 @@ class ToolOutput(BaseModel):
         """
 
 
-class StrOutput(BaseModel):
+class Str(BaseModel):
     result: str = Field(..., description="The output string", example="text")
 
 
-class BoolOutput(BaseModel):
+class Bool(BaseModel):
     result: bool = Field(
         ..., description="Boolean indicating the output state", example=True
     )
 
 
-class ListStrOutput(BaseModel):
+class ListStr(BaseModel):
     result: list[str] = Field(
         ..., description="The output list of strings", example=["text_1", "text_2"]
     )
 
 
-class ListDictStrStrOutput(BaseModel):
+class ListDictStrStr(BaseModel):
     result: list[dict[str, str]] = Field(
         ...,
         description="List of dictionaries containing string key-value pairs",
@@ -46,7 +46,7 @@ class ListDictStrStrOutput(BaseModel):
     )
 
 
-class ReasonListStrOutput(BaseModel):
+class ReasonListStr(BaseModel):
     reason: str = Field(..., description="Thinking process that led to the output")
     result: list[str] = Field(
         ..., description="The output list of strings", example=["text_1", "text_2"]
@@ -179,12 +179,3 @@ def create_dynamic_model(allowed_values: list[str]) -> Type[BaseModel]:
     )
 
     return CategorizerOutput
-
-
-class Entity(BaseModel):
-    text: str = Field(description="The exact text of the entity")
-    entity_type: str = Field(description="The type of the entity")
-
-
-class EntityDetectorOutput(BaseModel):
-    result: list[Entity] = Field(description="List of all extracted entities")

texttools/internals/operator_utils.py

@@ -52,3 +52,27 @@ class OperatorUtils:
         new_temp = base_temp + delta_temp
 
         return max(0.0, min(new_temp, 1.5))
+
+    @staticmethod
+    def user_merge_format(messages: list[dict[str, str]]) -> list[dict[str, str]]:
+        """
+        Merges consecutive user messages into a single message, separated by newlines.
+
+        This is useful for condensing a multi-turn user input into a single
+        message for the LLM. Assistant and system messages are left unchanged and
+        act as separators between user message groups.
+        """
+        merged = []
+
+        for message in messages:
+            role, content = message["role"], message["content"].strip()
+
+            # Merge with previous user turn
+            if merged and role == "user" and merged[-1]["role"] == "user":
+                merged[-1]["content"] += "\n" + content
+
+            # Otherwise, start a new turn
+            else:
+                merged.append({"role": role, "content": content})
+
+        return merged

texttools/internals/prompt_loader.py

@@ -44,16 +44,44 @@ class PromptLoader:
             if self.MAIN_TEMPLATE not in data:
                 raise PromptError(f"Missing 'main_template' in {prompt_file}")
 
+            if self.ANALYZE_TEMPLATE not in data:
+                raise PromptError(f"Missing 'analyze_template' in {prompt_file}")
+
             if mode and mode not in data.get(self.MAIN_TEMPLATE, {}):
                 raise PromptError(f"Mode '{mode}' not found in {prompt_file}")
 
-            return {
-                self.MAIN_TEMPLATE: data[self.MAIN_TEMPLATE][mode]
+            # Extract templates based on mode
+            main_template = (
+                data[self.MAIN_TEMPLATE][mode]
                 if mode and isinstance(data[self.MAIN_TEMPLATE], dict)
-                else data[self.MAIN_TEMPLATE],
-                self.ANALYZE_TEMPLATE: data.get(self.ANALYZE_TEMPLATE, {}).get(mode)
-                if mode and isinstance(data.get(self.ANALYZE_TEMPLATE), dict)
-                else data.get(self.ANALYZE_TEMPLATE, ""),
+                else data[self.MAIN_TEMPLATE]
+            )
+
+            analyze_template = (
+                data[self.ANALYZE_TEMPLATE][mode]
+                if mode and isinstance(data[self.ANALYZE_TEMPLATE], dict)
+                else data[self.ANALYZE_TEMPLATE]
+            )
+
+            if not main_template or not main_template.strip():
+                raise PromptError(
+                    f"Empty main_template in {prompt_file}"
+                    + (f" for mode '{mode}'" if mode else "")
+                )
+
+            if (
+                not analyze_template
+                or not analyze_template.strip()
+                or analyze_template.strip() in ["{analyze_template}", "{}"]
+            ):
+                raise PromptError(
+                    "analyze_template cannot be empty"
+                    + (f" for mode '{mode}'" if mode else "")
+                )
+
+            return {
+                self.MAIN_TEMPLATE: main_template,
+                self.ANALYZE_TEMPLATE: analyze_template,
             }
 
         except yaml.YAMLError as e:

texttools/internals/sync_operator.py

@@ -1,4 +1,4 @@
-from typing import Any, TypeVar, Type
+from typing import TypeVar, Type
 from collections.abc import Callable
 import logging
 
@@ -7,7 +7,6 @@ from pydantic import BaseModel
 
 from texttools.internals.models import ToolOutput
 from texttools.internals.operator_utils import OperatorUtils
-from texttools.internals.formatters import Formatter
 from texttools.internals.prompt_loader import PromptLoader
 from texttools.internals.exceptions import (
     TextToolsError,
@@ -77,7 +76,7 @@ class Operator:
         logprobs: bool = False,
         top_logprobs: int = 3,
         priority: int | None = 0,
-    ) -> tuple[T, Any]:
+    ) -> tuple[T, object]:
         """
         Parses a chat completion using OpenAI's structured output format.
         Returns both the parsed object and the raw completion for logprobs.
@@ -122,7 +121,7 @@ class Operator:
         temperature: float,
         logprobs: bool,
         top_logprobs: int | None,
-        validator: Callable[[Any], bool] | None,
+        validator: Callable[[object], bool] | None,
         max_validation_retries: int | None,
         # Internal parameters
         prompt_file: str,
@@ -136,7 +135,6 @@ class Operator:
         """
         try:
             prompt_loader = PromptLoader()
-            formatter = Formatter()
             output = ToolOutput()
 
             # Prompt configs contain two keys: main_template and analyze template, both are string
@@ -175,7 +173,7 @@ class Operator:
                 OperatorUtils.build_user_message(prompt_configs["main_template"])
             )
 
-            messages = formatter.user_merge_format(messages)
+            messages = OperatorUtils.user_merge_format(messages)
 
             if logprobs and (not isinstance(top_logprobs, int) or top_logprobs < 2):
                 raise ValueError("top_logprobs should be an integer greater than 1")

texttools/internals/text_to_chunks.py

@@ -0,0 +1,97 @@
+import re
+
+
+def text_to_chunks(text: str, size: int, overlap: int) -> list[str]:
+    separators = ["\n\n", "\n", " ", ""]
+    is_separator_regex = False
+    keep_separator = True  # Equivalent to 'start'
+    length_function = len
+    strip_whitespace = True
+    chunk_size = size
+    chunk_overlap = overlap
+
+    def _split_text_with_regex(
+        text: str, separator: str, keep_separator: bool
+    ) -> list[str]:
+        if not separator:
+            return [text]
+        if not keep_separator:
+            return re.split(separator, text)
+        _splits = re.split(f"({separator})", text)
+        splits = [_splits[i] + _splits[i + 1] for i in range(1, len(_splits), 2)]
+        if len(_splits) % 2 == 0:
+            splits += [_splits[-1]]
+        return [_splits[0]] + splits if _splits[0] else splits
+
+    def _join_docs(docs: list[str], separator: str) -> str | None:
+        text = separator.join(docs)
+        if strip_whitespace:
+            text = text.strip()
+        return text if text else None
+
+    def _merge_splits(splits: list[str], separator: str) -> list[str]:
+        separator_len = length_function(separator)
+        docs = []
+        current_doc = []
+        total = 0
+        for d in splits:
+            len_ = length_function(d)
+            if total + len_ + (separator_len if current_doc else 0) > chunk_size:
+                if total > chunk_size:
+                    pass
+                if current_doc:
+                    doc = _join_docs(current_doc, separator)
+                    if doc is not None:
+                        docs.append(doc)
+                    while total > chunk_overlap or (
+                        total + len_ + (separator_len if current_doc else 0)
+                        > chunk_size
+                        and total > 0
+                    ):
+                        total -= length_function(current_doc[0]) + (
+                            separator_len if len(current_doc) > 1 else 0
+                        )
+                        current_doc = current_doc[1:]
+            current_doc.append(d)
+            total += len_ + (separator_len if len(current_doc) > 1 else 0)
+        doc = _join_docs(current_doc, separator)
+        if doc is not None:
+            docs.append(doc)
+        return docs
+
+    def _split_text(text: str, separators: list[str]) -> list[str]:
+        final_chunks = []
+        separator = separators[-1]
+        new_separators = []
+        for i, _s in enumerate(separators):
+            separator_ = _s if is_separator_regex else re.escape(_s)
+            if not _s:
+                separator = _s
+                break
+            if re.search(separator_, text):
+                separator = _s
+                new_separators = separators[i + 1 :]
+                break
+        separator_ = separator if is_separator_regex else re.escape(separator)
+        splits = _split_text_with_regex(text, separator_, keep_separator)
+        _separator = "" if keep_separator else separator
+        good_splits = []
+        for s in splits:
+            if length_function(s) < chunk_size:
+                good_splits.append(s)
+            else:
+                if good_splits:
+                    merged_text = _merge_splits(good_splits, _separator)
+                    final_chunks.extend(merged_text)
+                    good_splits = []
+                if not new_separators:
+                    final_chunks.append(s)
+                else:
+                    other_info = _split_text(s, new_separators)
+                    final_chunks.extend(other_info)
+        if good_splits:
+            merged_text = _merge_splits(good_splits, _separator)
+            final_chunks.extend(merged_text)
+        return final_chunks
+
+    return _split_text(text, separators)

texttools/prompts/check_fact.yaml

@@ -0,0 +1,19 @@
+main_template: |
+  You are an expert in determining whether a statement can be concluded from the source text or not.
+  You must return a boolean value: True or False.
+  Return True if the statement can be concluded from the source, and False otherwise.
+  Respond only in JSON format (Output should be a boolean):
+  {{"result": True/False}}
+  The statement is:
+  {input}
+  The source text is:
+  {source_text}
+
+analyze_template: |
+  You should analyze a statement and a source text and provide a brief,
+  summarized analysis that could help in determining that can the statement
+  be concluded from the source or not.
+  The statement is:
+  {input}
+  The source text is:
+  {source_text}

texttools/prompts/extract_entities.yaml

@@ -1,6 +1,6 @@
 main_template: |
   You are a Named Entity Recognition (NER) extractor.
-  Identify and extract all named entities (e.g., PER, ORG, LOC, DAT, etc.) from the given text.
+  Identify and extract {entities} from the given text.
   For each entity, provide its text and a clear type.
   Respond only in JSON format:
   {{

texttools/prompts/propositionize.yaml

@@ -1,10 +1,17 @@
 main_template: |
-  You are an expert in breaking down text into atomic propositions in that language.
-  An atomic proposition is a single, self-contained fact that is concise, verifiable,
-  and does not rely on external context.
-  Each proposition must stand alone.
-  Rewrite sentences if needed to keep the context saved in each sentence.
-  Extract the atomic propositions of this text:
+  You are an expert data analyst specializing in Information Extraction.
+  Your task is to extract a list of "Atomic Propositions" from the provided text.
+
+  Definition of Atomic Proposition:
+  A single, self-contained statement of fact that is concise and verifiable.
+
+  Strict Guidelines:
+  1. Remove Meta-Data: STRICTLY EXCLUDE all citations, references, URLs, source attributions (e.g., "Source: makarem.ir"), and conversational fillers (e.g., "Based on the documents...", "In conclusion...").
+  2. Resolve Context: Replace pronouns ("it", "this", "they") with the specific nouns they refer to. Each proposition must make sense in isolation.
+  3. Preserve Logic: Keep conditions attached to their facts. Do not split a rule from its condition (e.g., "If X, then Y" should be one proposition).
+  4. No Redundancy: Do not extract summary statements that merely repeat facts already listed.
+
+  Extract the atomic propositions from the following text:
   {input}
 
 analyze_template: |

texttools/prompts/run_custom.yaml

@@ -4,4 +4,4 @@ main_template: |
   {output_model_str}
 
 analyze_template: |
-  
+  {analyze_template}

texttools/prompts/text_to_question.yaml

@@ -1,11 +1,13 @@
 main_template: |
   You are a question generator.
-  Given the following answer, generate one 
-  appropriate question that this answer would directly respond to.
+  Given the following answer, generate {number_of_questions} appropriate questions that this answer would directly respond to.
   The generated answer should be independently meaningful,
   and not mentioning any verbs like, this, that, he or she on the question.
+  There is a `reason` key, fill that up with a summerized version of your thoughts.
+  The `reason` must be less than 20 words.
+  Don't forget to fill the reason.
   Respond only in JSON format:
-  {{"result": "string"}}
+  {{"result": ["question1", "question2", ...], "reason": "string"}}
   Here is the answer:
   {input}
   
@@ -13,7 +15,7 @@ analyze_template: |
   Analyze the following answer to identify its key facts,
   main subject, and what kind of information it provides.
   Provide a brief, summarized understanding of the answer's content that will 
-  help in formulating a relevant and direct question.
+  help in formulating relevant and direct questions.
   Just mention the keypoints that was provided in the answer
   Here is the answer:
   {input}