hamtaa-texttools 1.1.19__py3-none-any.whl → 1.1.20__py3-none-any.whl

{hamtaa_texttools-1.1.19.dist-info → hamtaa_texttools-1.1.20.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 1.1.19
+Version: 1.1.20
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
 License: MIT License
@@ -99,21 +99,21 @@ Each tool is designed to work with structured outputs (JSON / Pydantic).
 
 TextTools provides several optional flags to customize LLM behavior:
 
-- **`with_analysis (bool)`** → Adds a reasoning step before generating the final output.
+- **`with_analysis: bool`** → Adds a reasoning step before generating the final output.
 **Note:** This doubles token usage per call because it triggers an additional LLM request.
 
-- **`logprobs (bool)`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
+- **`logprobs: bool`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
 **Note:** This feature works if it's supported by the model.
 
-- **`output_lang (str)`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
+- **`output_lang: str`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
 
-- **`user_prompt (str)`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
+- **`user_prompt: str`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
 
-- **`temperature (float)`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
+- **`temperature: float`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
 
-- **`validator (Callable)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a bool (True if there were no problem, False if the validation fails.) If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can specify `max_validation_retries=<N>` to change the number of retries.
+- **`validator: Callable (Experimental)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a bool (True if there were no problem, False if the validation fails.) If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can specify `max_validation_retries=<N>` to change the number of retries.
 
-- **`priority (int)`** → Task execution priority level. Higher values = higher priority. Affects processing order in queues.
+- **`priority: int (Experimental)`** → Task execution priority level. Higher values = higher priority. Affects processing order in queues.
 **Note:** This feature works if it's supported by the model and vLLM.
 
 **Note:** There might be some tools that don't support some of the parameters above.

{hamtaa_texttools-1.1.19.dist-info → hamtaa_texttools-1.1.20.dist-info}/RECORD

@@ -1,20 +1,20 @@
-hamtaa_texttools-1.1.19.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
+hamtaa_texttools-1.1.20.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
 texttools/__init__.py,sha256=CmCS9dEvO6061GiJ8A7gD3UAhCWHTkaID9q3Krlyq_o,311
 texttools/batch/batch_config.py,sha256=m1UgILVKjNdWE6laNbfbG4vgi4o2fEegGZbeoam6pnY,749
-texttools/batch/batch_runner.py,sha256=Tz-jec27UZBSZAXc0sxitc5XycDfzvOYl47Yqzq6Myw,10031
-texttools/batch/internals/batch_manager.py,sha256=UoBe76vmFG72qrSaGKDZf4HzkykFBkkkbL9TLfV8TuQ,8730
-texttools/batch/internals/utils.py,sha256=F1_7YlVFKhjUROAFX4m0SaP8KiZVZyHRMIIB87VUGQc,373
-texttools/internals/async_operator.py,sha256=_RfYSm_66RJ6nppzorJ4r3BHdhr8xr404QjeVvsvX4Q,8485
+texttools/batch/batch_runner.py,sha256=fVqgFDOyv8DqaNmRJQjt75wOxXTKPstisHycpt0LwcM,10026
+texttools/batch/internals/batch_manager.py,sha256=6HfsexU0PHGGBH7HKReZ-CQxaQI9DXYKAPsFXxovb_I,8740
+texttools/batch/internals/utils.py,sha256=8uNqvPHkEDFpiPp2Nyu-1nP4R-Tq8FwuSGMNSjcBogY,348
+texttools/internals/async_operator.py,sha256=rgQuvT-hl53stClVojso9FgmKhK98nm_2Cdl5WROMoc,8399
 texttools/internals/exceptions.py,sha256=h_yp_5i_5IfmqTBQ4S6ZOISrrliJBQ3HTEAjwJXrplk,495
-texttools/internals/formatters.py,sha256=tACNLP6PeoqaRpNudVxBaHA25zyWqWYPZQuYysIu88g,941
-texttools/internals/models.py,sha256=zmgdFhMCNyfc-5dtSE4jwulhltVgxYzITZRMDJBUF0A,5977
-texttools/internals/operator_utils.py,sha256=w1k0RJ_W_CRbVc_J2w337VuL-opHpHiCxfhEOwtyuOo,1856
+texttools/internals/models.py,sha256=Ro8d875_xjMHwwJIz3D_-VWxQ2WOcLLIKxsleSvqPDE,5659
+texttools/internals/operator_utils.py,sha256=jphe1DvYgdpRprZFv23ghh3bNYETvVTQ8ZZ8HPwwRVo,2759
 texttools/internals/prompt_loader.py,sha256=i4OxcVJTjHFKPSoC-DWZUM3Vf8ye_vbD7b6t3N2qB08,3972
-texttools/internals/sync_operator.py,sha256=7SdsNoFQxgmMrSZbUUw7SJVqyO5Xhu8dui9lm64RKsk,8382
+texttools/internals/sync_operator.py,sha256=Q0I2LlGF8MEPUCnwOOErOlxF3gjNxEvNS3oM9V6jazE,8296
+texttools/internals/text_to_chunks.py,sha256=vY3odhgCZK4E44k_SGlLoSiKkdN0ib6-lQAsPcplAHA,3843
 texttools/prompts/README.md,sha256=-5YO93CN93QLifqZpUeUnCOCBbDiOTV-cFQeJ7Gg0I4,1377
 texttools/prompts/categorize.yaml,sha256=F7VezB25B_sT5yoC25ezODBddkuDD5lUHKetSpx9FKI,2743
 texttools/prompts/check_fact.yaml,sha256=5kpBjmfZxgp81Owc8-Pd0U8-cZowFGRdYlGTFQLYQ9o,702
-texttools/prompts/extract_entities.yaml,sha256=KiKjeDpHaeh3JVtZ6q1pa3k4DYucUIU9WnEcRTCA-SE,651
+texttools/prompts/extract_entities.yaml,sha256=56N3iFH1KbGLqloYBLOW-12SchVayqbHgaQ5-8JTbeY,610
 texttools/prompts/extract_keywords.yaml,sha256=Vj4Tt3vT6LtpOo_iBZPo9oWI50oVdPGXe5i8yDR8ex4,3177
 texttools/prompts/is_question.yaml,sha256=d0-vKRbXWkxvO64ikvxRjEmpAXGpCYIPGhgexvPPjws,471
 texttools/prompts/merge_questions.yaml,sha256=0J85GvTirZB4ELwH3sk8ub_WcqqpYf6PrMKr3djlZeo,1792
@@ -23,11 +23,11 @@ texttools/prompts/rewrite.yaml,sha256=LO7He_IA3MZKz8a-LxH9DHJpOjpYwaYN1pbjp1Y0tF
 texttools/prompts/run_custom.yaml,sha256=6oiMYOo_WctVbOmE01wZzI1ra7nFDMJzceTTtnGdmOA,126
 texttools/prompts/subject_to_question.yaml,sha256=C7x7rNNm6U_ZG9HOn6zuzYOtvJUZ2skuWbL1-aYdd3E,1147
 texttools/prompts/summarize.yaml,sha256=o6rxGPfWtZd61Duvm8NVvCJqfq73b-wAuMSKR6UYUqY,459
-texttools/prompts/text_to_question.yaml,sha256=UheKYpDn6iyKI8NxunHZtFpNyfCLZZe5cvkuXpurUJY,783
+texttools/prompts/text_to_question.yaml,sha256=fnxDpUnnOEmK0yyFU5F4ItBqCnegoUCXhSTFjiTy18Y,1005
 texttools/prompts/translate.yaml,sha256=mGT2uBCei6uucWqVbs4silk-UV060v3G0jnt0P6sr50,634
-texttools/tools/async_tools.py,sha256=eNsKJqpTNL1AIM_enHvqUJYxov1Mb5ErnShZuX7oqRQ,49532
-texttools/tools/sync_tools.py,sha256=DbY7smzYnEgd1H0r-5sVW-NExJwhL23TtQ_n5ACqbBc,49344
-hamtaa_texttools-1.1.19.dist-info/METADATA,sha256=egHahU5ec3bSpRB0DK0CrT21kfmhbM6-xxJjoMp1eDU,10587
-hamtaa_texttools-1.1.19.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-hamtaa_texttools-1.1.19.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
-hamtaa_texttools-1.1.19.dist-info/RECORD,,
+texttools/tools/async_tools.py,sha256=0ubBHyx3ii1MOwo8X_ZMfqaH43y07DaF_rk7s7DPjVA,52331
+texttools/tools/sync_tools.py,sha256=7ZY2Cs7W0y3m8D9I70Tqk9pMisIQeIzdAWwOh1A1hRc,52137
+hamtaa_texttools-1.1.20.dist-info/METADATA,sha256=s52LtRd3UQOcmNgOJ_riHjMgMC9pS2mmks2Ys0zWhXk,10610
+hamtaa_texttools-1.1.20.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+hamtaa_texttools-1.1.20.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-1.1.20.dist-info/RECORD,,

texttools/batch/batch_runner.py

@@ -2,7 +2,7 @@ import json
 import os
 import time
 from pathlib import Path
-from typing import Any, Type, TypeVar
+from typing import Type, TypeVar
 import logging
 
 from dotenv import load_dotenv
@@ -11,7 +11,7 @@ from pydantic import BaseModel
 
 from texttools.batch.internals.batch_manager import BatchManager
 from texttools.batch.batch_config import BatchConfig
-from texttools.internals.models import StrOutput
+from texttools.internals.models import Str
 from texttools.internals.exceptions import TextToolsError, ConfigurationError
 
 # Base Model type for output models
@@ -26,7 +26,7 @@ class BatchJobRunner:
     """
 
     def __init__(
-        self, config: BatchConfig = BatchConfig(), output_model: Type[T] = StrOutput
+        self, config: BatchConfig = BatchConfig(), output_model: Type[T] = Str
     ):
         try:
             self._config = config
@@ -38,7 +38,7 @@ class BatchJobRunner:
             self._output_model = output_model
             self._manager = self._init_manager()
             self._data = self._load_data()
-            self._parts: list[list[dict[str, Any]]] = []
+            self._parts: list[list[dict[str, object]]] = []
             # Map part index to job name
             self._part_idx_to_job_name: dict[int, str] = {}
             # Track retry attempts per part
@@ -130,8 +130,8 @@ class BatchJobRunner:
 
     def _save_results(
         self,
-        output_data: list[dict[str, Any]] | dict[str, Any],
-        log: list[Any],
+        output_data: list[dict[str, object]] | dict[str, object],
+        log: list[object],
         part_idx: int,
     ):
         part_suffix = f"_part_{part_idx + 1}" if len(self._parts) > 1 else ""

texttools/batch/internals/batch_manager.py

@@ -1,7 +1,7 @@
 import json
 import uuid
 from pathlib import Path
-from typing import Any, Type, TypeVar
+from typing import Type, TypeVar
 import logging
 
 from pydantic import BaseModel
@@ -31,7 +31,7 @@ class BatchManager:
         prompt_template: str,
         state_dir: Path = Path(".batch_jobs"),
         custom_json_schema_obj_str: dict | None = None,
-        **client_kwargs: Any,
+        **client_kwargs: object,
     ):
         self._client = client
         self._model = model
@@ -51,7 +51,7 @@ class BatchManager:
     def _state_file(self, job_name: str) -> Path:
         return self._state_dir / f"{job_name}.json"
 
-    def _load_state(self, job_name: str) -> list[dict[str, Any]]:
+    def _load_state(self, job_name: str) -> list[dict[str, object]]:
         """
         Loads the state (job information) from the state file for the given job name.
         Returns an empty list if the state file does not exist.
@@ -62,7 +62,7 @@ class BatchManager:
                 return json.load(f)
         return []
 
-    def _save_state(self, job_name: str, jobs: list[dict[str, Any]]) -> None:
+    def _save_state(self, job_name: str, jobs: list[dict[str, object]]) -> None:
         """
         Saves the job state to the state file for the given job name.
         """
@@ -77,11 +77,11 @@ class BatchManager:
         if path.exists():
             path.unlink()
 
-    def _build_task(self, text: str, idx: str) -> dict[str, Any]:
+    def _build_task(self, text: str, idx: str) -> dict[str, object]:
         """
         Builds a single task dictionary for the batch job, including the prompt, model, and response format configuration.
         """
-        response_format_config: dict[str, Any]
+        response_format_config: dict[str, object]
 
         if self._custom_json_schema_obj_str:
             response_format_config = {

texttools/batch/internals/utils.py

@@ -1,6 +1,3 @@
-from typing import Any
-
-
 def export_data(data) -> list[dict[str, str]]:
     """
     Produces a structure of the following form from an initial data structure:
@@ -9,7 +6,7 @@ def export_data(data) -> list[dict[str, str]]:
     return data
 
 
-def import_data(data) -> Any:
+def import_data(data) -> object:
     """
     Takes the output and adds and aggregates it to the original structure.
     """

texttools/internals/async_operator.py

@@ -1,4 +1,4 @@
-from typing import Any, TypeVar, Type
+from typing import TypeVar, Type
 from collections.abc import Callable
 import logging
 
@@ -7,7 +7,6 @@ from pydantic import BaseModel
 
 from texttools.internals.models import ToolOutput
 from texttools.internals.operator_utils import OperatorUtils
-from texttools.internals.formatters import Formatter
 from texttools.internals.prompt_loader import PromptLoader
 from texttools.internals.exceptions import (
     TextToolsError,
@@ -77,7 +76,7 @@ class AsyncOperator:
         logprobs: bool = False,
         top_logprobs: int = 3,
         priority: int | None = 0,
-    ) -> tuple[T, Any]:
+    ) -> tuple[T, object]:
         """
         Parses a chat completion using OpenAI's structured output format.
         Returns both the parsed object and the raw completion for logprobs.
@@ -124,7 +123,7 @@ class AsyncOperator:
         temperature: float,
         logprobs: bool,
         top_logprobs: int | None,
-        validator: Callable[[Any], bool] | None,
+        validator: Callable[[object], bool] | None,
         max_validation_retries: int | None,
         # Internal parameters
         prompt_file: str,
@@ -138,7 +137,6 @@ class AsyncOperator:
         """
         try:
             prompt_loader = PromptLoader()
-            formatter = Formatter()
             output = ToolOutput()
 
             # Prompt configs contain two keys: main_template and analyze template, both are string
@@ -177,7 +175,7 @@ class AsyncOperator:
                 OperatorUtils.build_user_message(prompt_configs["main_template"])
             )
 
-            messages = formatter.user_merge_format(messages)
+            messages = OperatorUtils.user_merge_format(messages)
 
             if logprobs and (not isinstance(top_logprobs, int) or top_logprobs < 2):
                 raise ValueError("top_logprobs should be an integer greater than 1")

texttools/internals/models.py

@@ -1,12 +1,12 @@
 from datetime import datetime
-from typing import Type, Any, Literal
+from typing import Type, Literal
 
 from pydantic import BaseModel, Field, create_model
 
 
 class ToolOutput(BaseModel):
-    result: Any = None
-    logprobs: list[dict[str, Any]] = []
+    result: object = None
+    logprobs: list[dict[str, object]] = []
     analysis: str = ""
     process: str | None = None
     processed_at: datetime = datetime.now()
@@ -22,23 +22,23 @@ class ToolOutput(BaseModel):
         """
 
 
-class StrOutput(BaseModel):
+class Str(BaseModel):
     result: str = Field(..., description="The output string", example="text")
 
 
-class BoolOutput(BaseModel):
+class Bool(BaseModel):
     result: bool = Field(
         ..., description="Boolean indicating the output state", example=True
     )
 
 
-class ListStrOutput(BaseModel):
+class ListStr(BaseModel):
     result: list[str] = Field(
         ..., description="The output list of strings", example=["text_1", "text_2"]
     )
 
 
-class ListDictStrStrOutput(BaseModel):
+class ListDictStrStr(BaseModel):
     result: list[dict[str, str]] = Field(
         ...,
         description="List of dictionaries containing string key-value pairs",
@@ -46,7 +46,7 @@ class ListDictStrStrOutput(BaseModel):
     )
 
 
-class ReasonListStrOutput(BaseModel):
+class ReasonListStr(BaseModel):
     reason: str = Field(..., description="Thinking process that led to the output")
     result: list[str] = Field(
         ..., description="The output list of strings", example=["text_1", "text_2"]
@@ -179,12 +179,3 @@ def create_dynamic_model(allowed_values: list[str]) -> Type[BaseModel]:
     )
 
     return CategorizerOutput
-
-
-class Entity(BaseModel):
-    text: str = Field(description="The exact text of the entity")
-    entity_type: str = Field(description="The type of the entity")
-
-
-class EntityDetectorOutput(BaseModel):
-    result: list[Entity] = Field(description="List of all extracted entities")

texttools/internals/operator_utils.py

@@ -52,3 +52,27 @@ class OperatorUtils:
         new_temp = base_temp + delta_temp
 
         return max(0.0, min(new_temp, 1.5))
+
+    @staticmethod
+    def user_merge_format(messages: list[dict[str, str]]) -> list[dict[str, str]]:
+        """
+        Merges consecutive user messages into a single message, separated by newlines.
+
+        This is useful for condensing a multi-turn user input into a single
+        message for the LLM. Assistant and system messages are left unchanged and
+        act as separators between user message groups.
+        """
+        merged = []
+
+        for message in messages:
+            role, content = message["role"], message["content"].strip()
+
+            # Merge with previous user turn
+            if merged and role == "user" and merged[-1]["role"] == "user":
+                merged[-1]["content"] += "\n" + content
+
+            # Otherwise, start a new turn
+            else:
+                merged.append({"role": role, "content": content})
+
+        return merged

texttools/internals/sync_operator.py

@@ -1,4 +1,4 @@
-from typing import Any, TypeVar, Type
+from typing import TypeVar, Type
 from collections.abc import Callable
 import logging
 
@@ -7,7 +7,6 @@ from pydantic import BaseModel
 
 from texttools.internals.models import ToolOutput
 from texttools.internals.operator_utils import OperatorUtils
-from texttools.internals.formatters import Formatter
 from texttools.internals.prompt_loader import PromptLoader
 from texttools.internals.exceptions import (
     TextToolsError,
@@ -77,7 +76,7 @@ class Operator:
         logprobs: bool = False,
         top_logprobs: int = 3,
         priority: int | None = 0,
-    ) -> tuple[T, Any]:
+    ) -> tuple[T, object]:
         """
         Parses a chat completion using OpenAI's structured output format.
         Returns both the parsed object and the raw completion for logprobs.
@@ -122,7 +121,7 @@ class Operator:
         temperature: float,
         logprobs: bool,
         top_logprobs: int | None,
-        validator: Callable[[Any], bool] | None,
+        validator: Callable[[object], bool] | None,
         max_validation_retries: int | None,
         # Internal parameters
         prompt_file: str,
@@ -136,7 +135,6 @@ class Operator:
         """
         try:
             prompt_loader = PromptLoader()
-            formatter = Formatter()
             output = ToolOutput()
 
             # Prompt configs contain two keys: main_template and analyze template, both are string
@@ -175,7 +173,7 @@ class Operator:
                 OperatorUtils.build_user_message(prompt_configs["main_template"])
             )
 
-            messages = formatter.user_merge_format(messages)
+            messages = OperatorUtils.user_merge_format(messages)
 
             if logprobs and (not isinstance(top_logprobs, int) or top_logprobs < 2):
                 raise ValueError("top_logprobs should be an integer greater than 1")

texttools/internals/text_to_chunks.py

@@ -0,0 +1,97 @@
+import re
+
+
+def text_to_chunks(text: str, size: int, overlap: int) -> list[str]:
+    separators = ["\n\n", "\n", " ", ""]
+    is_separator_regex = False
+    keep_separator = True  # Equivalent to 'start'
+    length_function = len
+    strip_whitespace = True
+    chunk_size = size
+    chunk_overlap = overlap
+
+    def _split_text_with_regex(
+        text: str, separator: str, keep_separator: bool
+    ) -> list[str]:
+        if not separator:
+            return [text]
+        if not keep_separator:
+            return re.split(separator, text)
+        _splits = re.split(f"({separator})", text)
+        splits = [_splits[i] + _splits[i + 1] for i in range(1, len(_splits), 2)]
+        if len(_splits) % 2 == 0:
+            splits += [_splits[-1]]
+        return [_splits[0]] + splits if _splits[0] else splits
+
+    def _join_docs(docs: list[str], separator: str) -> str | None:
+        text = separator.join(docs)
+        if strip_whitespace:
+            text = text.strip()
+        return text if text else None
+
+    def _merge_splits(splits: list[str], separator: str) -> list[str]:
+        separator_len = length_function(separator)
+        docs = []
+        current_doc = []
+        total = 0
+        for d in splits:
+            len_ = length_function(d)
+            if total + len_ + (separator_len if current_doc else 0) > chunk_size:
+                if total > chunk_size:
+                    pass
+                if current_doc:
+                    doc = _join_docs(current_doc, separator)
+                    if doc is not None:
+                        docs.append(doc)
+                    while total > chunk_overlap or (
+                        total + len_ + (separator_len if current_doc else 0)
+                        > chunk_size
+                        and total > 0
+                    ):
+                        total -= length_function(current_doc[0]) + (
+                            separator_len if len(current_doc) > 1 else 0
+                        )
+                        current_doc = current_doc[1:]
+            current_doc.append(d)
+            total += len_ + (separator_len if len(current_doc) > 1 else 0)
+        doc = _join_docs(current_doc, separator)
+        if doc is not None:
+            docs.append(doc)
+        return docs
+
+    def _split_text(text: str, separators: list[str]) -> list[str]:
+        final_chunks = []
+        separator = separators[-1]
+        new_separators = []
+        for i, _s in enumerate(separators):
+            separator_ = _s if is_separator_regex else re.escape(_s)
+            if not _s:
+                separator = _s
+                break
+            if re.search(separator_, text):
+                separator = _s
+                new_separators = separators[i + 1 :]
+                break
+        separator_ = separator if is_separator_regex else re.escape(separator)
+        splits = _split_text_with_regex(text, separator_, keep_separator)
+        _separator = "" if keep_separator else separator
+        good_splits = []
+        for s in splits:
+            if length_function(s) < chunk_size:
+                good_splits.append(s)
+            else:
+                if good_splits:
+                    merged_text = _merge_splits(good_splits, _separator)
+                    final_chunks.extend(merged_text)
+                    good_splits = []
+                if not new_separators:
+                    final_chunks.append(s)
+                else:
+                    other_info = _split_text(s, new_separators)
+                    final_chunks.extend(other_info)
+        if good_splits:
+            merged_text = _merge_splits(good_splits, _separator)
+            final_chunks.extend(merged_text)
+        return final_chunks
+
+    return _split_text(text, separators)

texttools/prompts/extract_entities.yaml

@@ -1,6 +1,6 @@
 main_template: |
   You are a Named Entity Recognition (NER) extractor.
-  Identify and extract all named entities (e.g., PER, ORG, LOC, DAT, etc.) from the given text.
+  Identify and extract {entities} from the given text.
   For each entity, provide its text and a clear type.
   Respond only in JSON format:
   {{

texttools/prompts/text_to_question.yaml

@@ -1,11 +1,13 @@
 main_template: |
   You are a question generator.
-  Given the following answer, generate one 
-  appropriate question that this answer would directly respond to.
+  Given the following answer, generate {number_of_questions} appropriate questions that this answer would directly respond to.
   The generated answer should be independently meaningful,
   and not mentioning any verbs like, this, that, he or she on the question.
+  There is a `reason` key, fill that up with a summerized version of your thoughts.
+  The `reason` must be less than 20 words.
+  Don't forget to fill the reason.
   Respond only in JSON format:
-  {{"result": "string"}}
+  {{"result": ["question1", "question2", ...], "reason": "string"}}
   Here is the answer:
   {input}
   
@@ -13,7 +15,7 @@ analyze_template: |
   Analyze the following answer to identify its key facts,
   main subject, and what kind of information it provides.
   Provide a brief, summarized understanding of the answer's content that will 
-  help in formulating a relevant and direct question.
+  help in formulating relevant and direct questions.
   Just mention the keypoints that was provided in the answer
   Here is the answer:
   {input}