hamtaa-texttools 1.1.18__py3-none-any.whl → 1.1.19__py3-none-any.whl

{hamtaa_texttools-1.1.18.dist-info → hamtaa_texttools-1.1.19.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 1.1.18
+Version: 1.1.19
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
 License: MIT License
@@ -61,10 +61,40 @@ Each tool is designed to work with structured outputs (JSON / Pydantic).
 - **`summarize()`** - Text summarization
 - **`translate()`** - Text translation between languages
 - **`propositionize()`** - Convert text to atomic independence meaningful sentences 
+- **`check_fact()`** - Check a statement is relevant to source text or not 
 - **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
 
 ---
 
+## 📊 Tool Quality Tiers
+
+| Status | Meaning | Use in Production? |
+|--------|---------|-------------------|
+| **✅ Production** | Evaluated, tested, stable. | **Yes** - ready for reliable use. |
+| **🧪 Experimental** | Added to the package but **not fully evaluated**. Functional, but quality may vary. | **Use with caution** - outputs not yet validated. |
+
+### Current Status
+**Production Tools:**
+- `categorize()` (list mode)
+- `extract_keywords()`
+- `extract_entities()`
+- `is_question()`
+- `text_to_question()`
+- `merge_questions()`
+- `rewrite()`
+- `subject_to_question()`
+- `summarize()`
+- `run_custom()` (fine in most cases)
+
+**Experimental Tools:**
+- `categorize()` (tree mode)
+- `translate()`
+- `propositionize()`
+- `check_fact()`
+- `run_custom()` (not evaluated in all scenarios)
+
+---
+
 ## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator` and `priority` parameters
 
 TextTools provides several optional flags to customize LLM behavior:

{hamtaa_texttools-1.1.18.dist-info → hamtaa_texttools-1.1.19.dist-info}/RECORD

@@ -1,4 +1,4 @@
-hamtaa_texttools-1.1.18.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
+hamtaa_texttools-1.1.19.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
 texttools/__init__.py,sha256=CmCS9dEvO6061GiJ8A7gD3UAhCWHTkaID9q3Krlyq_o,311
 texttools/batch/batch_config.py,sha256=m1UgILVKjNdWE6laNbfbG4vgi4o2fEegGZbeoam6pnY,749
 texttools/batch/batch_runner.py,sha256=Tz-jec27UZBSZAXc0sxitc5XycDfzvOYl47Yqzq6Myw,10031
@@ -9,25 +9,25 @@ texttools/internals/exceptions.py,sha256=h_yp_5i_5IfmqTBQ4S6ZOISrrliJBQ3HTEAjwJX
 texttools/internals/formatters.py,sha256=tACNLP6PeoqaRpNudVxBaHA25zyWqWYPZQuYysIu88g,941
 texttools/internals/models.py,sha256=zmgdFhMCNyfc-5dtSE4jwulhltVgxYzITZRMDJBUF0A,5977
 texttools/internals/operator_utils.py,sha256=w1k0RJ_W_CRbVc_J2w337VuL-opHpHiCxfhEOwtyuOo,1856
-texttools/internals/prompt_loader.py,sha256=bL4F0pwVEjaBhjNRbf0kyl0YhSJ0G8NXa03Uz7XBzTE,3030
+texttools/internals/prompt_loader.py,sha256=i4OxcVJTjHFKPSoC-DWZUM3Vf8ye_vbD7b6t3N2qB08,3972
 texttools/internals/sync_operator.py,sha256=7SdsNoFQxgmMrSZbUUw7SJVqyO5Xhu8dui9lm64RKsk,8382
 texttools/prompts/README.md,sha256=-5YO93CN93QLifqZpUeUnCOCBbDiOTV-cFQeJ7Gg0I4,1377
 texttools/prompts/categorize.yaml,sha256=F7VezB25B_sT5yoC25ezODBddkuDD5lUHKetSpx9FKI,2743
-texttools/prompts/detect_entity.yaml,sha256=1rhMkJOjxSQcT4j_c5SRcIm77AUdeG-rUmeidb6VOFc,981
+texttools/prompts/check_fact.yaml,sha256=5kpBjmfZxgp81Owc8-Pd0U8-cZowFGRdYlGTFQLYQ9o,702
 texttools/prompts/extract_entities.yaml,sha256=KiKjeDpHaeh3JVtZ6q1pa3k4DYucUIU9WnEcRTCA-SE,651
 texttools/prompts/extract_keywords.yaml,sha256=Vj4Tt3vT6LtpOo_iBZPo9oWI50oVdPGXe5i8yDR8ex4,3177
 texttools/prompts/is_question.yaml,sha256=d0-vKRbXWkxvO64ikvxRjEmpAXGpCYIPGhgexvPPjws,471
 texttools/prompts/merge_questions.yaml,sha256=0J85GvTirZB4ELwH3sk8ub_WcqqpYf6PrMKr3djlZeo,1792
-texttools/prompts/propositionize.yaml,sha256=ZEFkYy8qYpFK2oCmZ-i5M7nFtlQp_q8bKQZbFSuyE3c,792
+texttools/prompts/propositionize.yaml,sha256=kdj-UxPOYcLSTLF7cWARDxxTxSFB0qRBaRujdThPDxw,1380
 texttools/prompts/rewrite.yaml,sha256=LO7He_IA3MZKz8a-LxH9DHJpOjpYwaYN1pbjp1Y0tFo,5392
-texttools/prompts/run_custom.yaml,sha256=38OkCoVITbuuS9c08UZSP1jZW4WjSmRIi8fR0RAiPu4,108
+texttools/prompts/run_custom.yaml,sha256=6oiMYOo_WctVbOmE01wZzI1ra7nFDMJzceTTtnGdmOA,126
 texttools/prompts/subject_to_question.yaml,sha256=C7x7rNNm6U_ZG9HOn6zuzYOtvJUZ2skuWbL1-aYdd3E,1147
 texttools/prompts/summarize.yaml,sha256=o6rxGPfWtZd61Duvm8NVvCJqfq73b-wAuMSKR6UYUqY,459
 texttools/prompts/text_to_question.yaml,sha256=UheKYpDn6iyKI8NxunHZtFpNyfCLZZe5cvkuXpurUJY,783
 texttools/prompts/translate.yaml,sha256=mGT2uBCei6uucWqVbs4silk-UV060v3G0jnt0P6sr50,634
-texttools/tools/async_tools.py,sha256=uNy-18XBOGKPcXkvp5mtNYfBrd16qA2HGsoxxMjTpFM,48536
-texttools/tools/sync_tools.py,sha256=UMos-6cWL9LKy8m0UwK8PBZEXqM5RVQgjjQ3k8LL4qA,48348
-hamtaa_texttools-1.1.18.dist-info/METADATA,sha256=Ro8eqilfvMDymdL8N_wkkgshnMPy9nWLJDO_IRBA9J4,9680
-hamtaa_texttools-1.1.18.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-hamtaa_texttools-1.1.18.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
-hamtaa_texttools-1.1.18.dist-info/RECORD,,
+texttools/tools/async_tools.py,sha256=eNsKJqpTNL1AIM_enHvqUJYxov1Mb5ErnShZuX7oqRQ,49532
+texttools/tools/sync_tools.py,sha256=DbY7smzYnEgd1H0r-5sVW-NExJwhL23TtQ_n5ACqbBc,49344
+hamtaa_texttools-1.1.19.dist-info/METADATA,sha256=egHahU5ec3bSpRB0DK0CrT21kfmhbM6-xxJjoMp1eDU,10587
+hamtaa_texttools-1.1.19.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+hamtaa_texttools-1.1.19.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-1.1.19.dist-info/RECORD,,

texttools/internals/prompt_loader.py

@@ -44,16 +44,44 @@ class PromptLoader:
             if self.MAIN_TEMPLATE not in data:
                 raise PromptError(f"Missing 'main_template' in {prompt_file}")
 
+            if self.ANALYZE_TEMPLATE not in data:
+                raise PromptError(f"Missing 'analyze_template' in {prompt_file}")
+
             if mode and mode not in data.get(self.MAIN_TEMPLATE, {}):
                 raise PromptError(f"Mode '{mode}' not found in {prompt_file}")
 
-            return {
-                self.MAIN_TEMPLATE: data[self.MAIN_TEMPLATE][mode]
+            # Extract templates based on mode
+            main_template = (
+                data[self.MAIN_TEMPLATE][mode]
                 if mode and isinstance(data[self.MAIN_TEMPLATE], dict)
-                else data[self.MAIN_TEMPLATE],
-                self.ANALYZE_TEMPLATE: data.get(self.ANALYZE_TEMPLATE, {}).get(mode)
-                if mode and isinstance(data.get(self.ANALYZE_TEMPLATE), dict)
-                else data.get(self.ANALYZE_TEMPLATE, ""),
+                else data[self.MAIN_TEMPLATE]
+            )
+
+            analyze_template = (
+                data[self.ANALYZE_TEMPLATE][mode]
+                if mode and isinstance(data[self.ANALYZE_TEMPLATE], dict)
+                else data[self.ANALYZE_TEMPLATE]
+            )
+
+            if not main_template or not main_template.strip():
+                raise PromptError(
+                    f"Empty main_template in {prompt_file}"
+                    + (f" for mode '{mode}'" if mode else "")
+                )
+
+            if (
+                not analyze_template
+                or not analyze_template.strip()
+                or analyze_template.strip() in ["{analyze_template}", "{}"]
+            ):
+                raise PromptError(
+                    "analyze_template cannot be empty"
+                    + (f" for mode '{mode}'" if mode else "")
+                )
+
+            return {
+                self.MAIN_TEMPLATE: main_template,
+                self.ANALYZE_TEMPLATE: analyze_template,
             }
 
         except yaml.YAMLError as e:

texttools/prompts/check_fact.yaml

@@ -0,0 +1,19 @@
+main_template: |
+  You are an expert in determining whether a statement can be concluded from the source text or not.
+  You must return a boolean value: True or False.
+  Return True if the statement can be concluded from the source, and False otherwise.
+  Respond only in JSON format (Output should be a boolean):
+  {{"result": True/False}}
+  The statement is:
+  {input}
+  The source text is:
+  {source_text}
+
+analyze_template: |
+  You should analyze a statement and a source text and provide a brief,
+  summarized analysis that could help in determining that can the statement
+  be concluded from the source or not.
+  The statement is:
+  {input}
+  The source text is:
+  {source_text}

texttools/prompts/propositionize.yaml

@@ -1,10 +1,17 @@
 main_template: |
-  You are an expert in breaking down text into atomic propositions in that language.
-  An atomic proposition is a single, self-contained fact that is concise, verifiable,
-  and does not rely on external context.
-  Each proposition must stand alone.
-  Rewrite sentences if needed to keep the context saved in each sentence.
-  Extract the atomic propositions of this text:
+  You are an expert data analyst specializing in Information Extraction.
+  Your task is to extract a list of "Atomic Propositions" from the provided text.
+
+  Definition of Atomic Proposition:
+  A single, self-contained statement of fact that is concise and verifiable.
+
+  Strict Guidelines:
+  1. Remove Meta-Data: STRICTLY EXCLUDE all citations, references, URLs, source attributions (e.g., "Source: makarem.ir"), and conversational fillers (e.g., "Based on the documents...", "In conclusion...").
+  2. Resolve Context: Replace pronouns ("it", "this", "they") with the specific nouns they refer to. Each proposition must make sense in isolation.
+  3. Preserve Logic: Keep conditions attached to their facts. Do not split a rule from its condition (e.g., "If X, then Y" should be one proposition).
+  4. No Redundancy: Do not extract summary statements that merely repeat facts already listed.
+
+  Extract the atomic propositions from the following text:
   {input}
 
 analyze_template: |

texttools/prompts/run_custom.yaml

@@ -4,4 +4,4 @@ main_template: |
   {output_model_str}
 
 analyze_template: |
-  
+  {analyze_template}

texttools/tools/async_tools.py

@@ -44,6 +44,8 @@ class AsyncTheTool:
         """
         Categorize a text into a category / category tree.
 
+        Important Note: category_tree mode is EXPERIMENTAL, you can use it but it isn't reliable.
+
         Arguments:
             text: The input text to categorize
             categories: The category / category_tree to give to LLM
@@ -822,6 +824,8 @@ class AsyncTheTool:
         """
         Translate text between languages.
 
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
+
         Arguments:
             text: The input text to translate
             target_language: The target language for translation
@@ -883,7 +887,7 @@ class AsyncTheTool:
 
         return output
 
-    async def detect_entity(
+    async def propositionize(
         self,
         text: str,
         with_analysis: bool = False,
@@ -897,7 +901,9 @@ class AsyncTheTool:
         priority: int | None = 0,
     ) -> Models.ToolOutput:
         """
-        Detects entities in a given text based on the entity_detector.yaml prompt.
+        Proposition input text to meaningful sentences.
+
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
 
         Arguments:
             text: The input text
@@ -913,7 +919,7 @@ class AsyncTheTool:
 
         Returns:
             ToolOutput: Object containing:
-                - result (list[Entity]): The entities
+                - result (list[str]): The propositions
                 - logprobs (list | None): Probability data if logprobs enabled
                 - analysis (str | None): Detailed reasoning if with_analysis enabled
                 - process (str | None): Description of the process used
@@ -938,8 +944,8 @@ class AsyncTheTool:
                 max_validation_retries=max_validation_retries,
                 priority=priority,
                 # Internal parameters
-                prompt_file="detect_entity.yaml",
-                output_model=Models.EntityDetectorOutput,
+                prompt_file="propositionize.yaml",
+                output_model=Models.ListStrOutput,
                 mode=None,
             )
             end = datetime.now()
@@ -959,9 +965,10 @@ class AsyncTheTool:
 
         return output
 
-    async def propositionize(
+    async def check_fact(
         self,
         text: str,
+        source_text: str,
         with_analysis: bool = False,
         output_lang: str | None = None,
         user_prompt: str | None = None,
@@ -973,10 +980,13 @@ class AsyncTheTool:
         priority: int | None = 0,
     ) -> Models.ToolOutput:
         """
-        Proposition input text to meaningful sentences.
+        Checks wheather a statement is relevant to the source text or not.
+
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
 
         Arguments:
             text: The input text
+            source_text: the source text that we want to check relation of text to it
             with_analysis: Whether to include detailed reasoning analysis
             output_lang: Language for the output summary
             user_prompt: Additional instructions for summarization
@@ -989,7 +999,7 @@ class AsyncTheTool:
 
         Returns:
             ToolOutput: Object containing:
-                - result (list[str]): The propositions
+                - result (bool): statement is relevant to source text or not
                 - logprobs (list | None): Probability data if logprobs enabled
                 - analysis (str | None): Detailed reasoning if with_analysis enabled
                 - process (str | None): Description of the process used
@@ -1014,14 +1024,14 @@ class AsyncTheTool:
                 max_validation_retries=max_validation_retries,
                 priority=priority,
                 # Internal parameters
-                prompt_file="propositionize.yaml",
-                output_model=Models.ListStrOutput,
+                prompt_file="check_fact.yaml",
+                output_model=Models.BoolOutput,
                 mode=None,
+                source_text=source_text,
             )
             end = datetime.now()
             output.execution_time = (end - start).total_seconds()
             return output
-
         except PromptError as e:
             output.errors.append(f"Prompt error: {e}")
         except LLMError as e:
@@ -1039,6 +1049,8 @@ class AsyncTheTool:
         self,
         prompt: str,
         output_model: Any,
+        with_analysis: bool = False,
+        analyze_template: str | None = None,
         output_lang: str | None = None,
         temperature: float | None = None,
         logprobs: bool | None = None,
@@ -1050,8 +1062,13 @@ class AsyncTheTool:
         """
         Custom tool that can do almost anything!
 
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
+
         Arguments:
-            text: The user prompt
+            prompt: The user prompt
+            output_model: Pydantic BaseModel used for structured output
+            with_analysis: Whether to include detailed reasoning analysis
+            analyze_template: The analyze template used for reasoning analysis
             output_lang: Language for the output summary
             temperature: Controls randomness (0.0 = deterministic, 1.0 = creative)
             logprobs: Whether to return token probability information
@@ -1078,6 +1095,8 @@ class AsyncTheTool:
                 # User paramaeters
                 text=prompt,
                 output_model=output_model,
+                with_analysis=with_analysis,
+                analyze_template=analyze_template,
                 output_model_str=output_model.model_json_schema(),
                 output_lang=output_lang,
                 temperature=temperature,
@@ -1089,7 +1108,6 @@ class AsyncTheTool:
                 # Internal parameters
                 prompt_file="run_custom.yaml",
                 user_prompt=None,
-                with_analysis=False,
                 mode=None,
             )
             end = datetime.now()

texttools/tools/sync_tools.py

@@ -44,6 +44,8 @@ class TheTool:
         """
         Categorize a text into a category / category tree.
 
+        Important Note: category_tree mode is EXPERIMENTAL, you can use it but it isn't reliable.
+
         Arguments:
             text: The input text to categorize
             categories: The category / category_tree to give to LLM
@@ -822,6 +824,8 @@ class TheTool:
         """
         Translate text between languages.
 
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
+
         Arguments:
             text: The input text to translate
             target_language: The target language for translation
@@ -883,7 +887,7 @@ class TheTool:
 
         return output
 
-    def detect_entity(
+    def propositionize(
         self,
         text: str,
         with_analysis: bool = False,
@@ -897,7 +901,9 @@ class TheTool:
         priority: int | None = 0,
     ) -> Models.ToolOutput:
         """
-        Detects entities in a given text based on the entity_detector.yaml prompt.
+        Proposition input text to meaningful sentences.
+
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
 
         Arguments:
             text: The input text
@@ -913,7 +919,7 @@ class TheTool:
 
         Returns:
             ToolOutput: Object containing:
-                - result (list[Entity]): The entities
+                - result (list[str]): The propositions
                 - logprobs (list | None): Probability data if logprobs enabled
                 - analysis (str | None): Detailed reasoning if with_analysis enabled
                 - process (str | None): Description of the process used
@@ -938,8 +944,8 @@ class TheTool:
                 max_validation_retries=max_validation_retries,
                 priority=priority,
                 # Internal parameters
-                prompt_file="detect_entity.yaml",
-                output_model=Models.EntityDetectorOutput,
+                prompt_file="propositionize.yaml",
+                output_model=Models.ListStrOutput,
                 mode=None,
             )
             end = datetime.now()
@@ -959,9 +965,10 @@ class TheTool:
 
         return output
 
-    def propositionize(
+    def check_fact(
         self,
         text: str,
+        source_text: str,
         with_analysis: bool = False,
         output_lang: str | None = None,
         user_prompt: str | None = None,
@@ -973,10 +980,13 @@ class TheTool:
         priority: int | None = 0,
     ) -> Models.ToolOutput:
         """
-        Proposition input text to meaningful sentences.
+        Checks wheather a statement is relevant to the source text or not.
+
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
 
         Arguments:
             text: The input text
+            source_text: the source text that we want to check relation of text to it
             with_analysis: Whether to include detailed reasoning analysis
             output_lang: Language for the output summary
             user_prompt: Additional instructions for summarization
@@ -989,7 +999,7 @@ class TheTool:
 
         Returns:
             ToolOutput: Object containing:
-                - result (list[str]): The propositions
+                - result (bool): statement is relevant to source text or not
                 - logprobs (list | None): Probability data if logprobs enabled
                 - analysis (str | None): Detailed reasoning if with_analysis enabled
                 - process (str | None): Description of the process used
@@ -998,7 +1008,6 @@ class TheTool:
                 - errors (list(str) | None): Errors occured during tool call
         """
         output = Models.ToolOutput()
-
         try:
             start = datetime.now()
             output = self._operator.run(
@@ -1014,9 +1023,10 @@ class TheTool:
                 max_validation_retries=max_validation_retries,
                 priority=priority,
                 # Internal parameters
-                prompt_file="propositionize.yaml",
-                output_model=Models.ListStrOutput,
+                prompt_file="check_fact.yaml",
+                output_model=Models.BoolOutput,
                 mode=None,
+                source_text=source_text,
             )
             end = datetime.now()
             output.execution_time = (end - start).total_seconds()
@@ -1039,6 +1049,8 @@ class TheTool:
         self,
         prompt: str,
         output_model: Any,
+        with_analysis: bool = False,
+        analyze_template: str | None = None,
         output_lang: str | None = None,
         temperature: float | None = None,
         logprobs: bool | None = None,
@@ -1050,8 +1062,13 @@ class TheTool:
         """
         Custom tool that can do almost anything!
 
+        Important Note: This tool is EXPERIMENTAL, you can use it but it isn't reliable.
+
         Arguments:
-            text: The user prompt
+            prompt: The user prompt
+            output_model: Pydantic BaseModel used for structured output
+            with_analysis: Whether to include detailed reasoning analysis
+            analyze_template: The analyze template used for reasoning analysis
             output_lang: Language for the output summary
             temperature: Controls randomness (0.0 = deterministic, 1.0 = creative)
             logprobs: Whether to return token probability information
@@ -1078,6 +1095,8 @@ class TheTool:
                 # User paramaeters
                 text=prompt,
                 output_model=output_model,
+                with_analysis=with_analysis,
+                analyze_template=analyze_template,
                 output_model_str=output_model.model_json_schema(),
                 output_lang=output_lang,
                 temperature=temperature,
@@ -1089,7 +1108,6 @@ class TheTool:
                 # Internal parameters
                 prompt_file="run_custom.yaml",
                 user_prompt=None,
-                with_analysis=False,
                 mode=None,
             )
             end = datetime.now()

texttools/prompts/detect_entity.yaml

@@ -1,22 +0,0 @@
-main_template: |
-  You are an expert Named Entity Recognition (NER) system. Extract entities from the text.
-  The output must strictly follow the provided Pydantic schema.
-  
-  Mapping Rule:
-  - Person: شخص
-  - Location: مکان
-  - Time: زمان
-  - Living Beings: موجود زنده
-  - Organization: سازمان
-  - Concept: مفهوم
-  
-  CRITICAL:
-  1. The final output structure must be a complete JSON object matching the Pydantic schema (List[Entity]).
-  2. Both the extracted text and the type must be in Persian, using the exact mapping provided above.
-  
-  Here is the text: {input}
-
-analyze_template: |
-  Analyze the following text to identify all potential named entities and their categories (Person, Location, Time, Living Beings, Organization, Concept). 
-  Provide a brief summary of the entities identified that will help the main process to extract them accurately and apply the correct Persian type label.
-  Here is the text: {input}