hamtaa-texttools 1.3.1__py3-none-any.whl → 2.0.0__py3-none-any.whl

{hamtaa_texttools-1.3.1.dist-info → hamtaa_texttools-2.0.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 1.3.1
+Version: 2.0.0
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Erfan Moosavi <erfanmoosavi84@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
 Maintainer-email: Erfan Moosavi <erfanmoosavi84@gmail.com>, Tohidi <the.mohammad.tohidi@gmail.com>
@@ -11,7 +11,7 @@ Classifier: License :: OSI Approved :: MIT License
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Text Processing
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.9
+Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: openai>=1.97.1
@@ -21,36 +21,36 @@ Dynamic: license-file
 
 # TextTools
 
+![PyPI](https://img.shields.io/pypi/v/hamtaa-texttools)
+![License](https://img.shields.io/pypi/l/hamtaa-texttools)
+
 ## 📌 Overview
 
 **TextTools** is a high-level **NLP toolkit** built on top of **LLMs**.  
 
 It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
 
-It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
-
-**Note:** Most features of `texttools` are reliable when you use `google/gemma-3n-e4b-it` model.
+It provides ready-to-use utilities for **translation, question detection, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
 
 ---
 
 ## ✨ Features
 
-TextTools provides a rich collection of high-level NLP utilities,
+TextTools provides a collection of high-level NLP utilities.
 Each tool is designed to work with structured outputs.
 
-- **`categorize()`** - Classifies text into given categories
-- **`extract_keywords()`** - Extracts keywords from the text
-- **`extract_entities()`** - Named Entity Recognition (NER) system
-- **`is_question()`** - Binary question detection
-- **`text_to_question()`** - Generates questions from text
-- **`merge_questions()`** - Merges multiple questions into one
-- **`rewrite()`** - Rewrites text in a diffrent way
-- **`subject_to_question()`** - Generates questions about a specific subject
-- **`summarize()`** - Text summarization
-- **`translate()`** - Text translation
-- **`propositionize()`** - Convert text to atomic independence meaningful sentences 
-- **`check_fact()`** - Check whether a statement is relevant to the source text
-- **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
+- **`categorize()`** - Classify text into given categories
+- **`extract_keywords()`** - Extract keywords from the text
+- **`extract_entities()`** - Perform Named Entity Recognition (NER)
+- **`is_question()`** - Detect if the input is phrased as a question
+- **`to_question()`** - Generate questions from the given text / subject
+- **`merge_questions()`** - Merge multiple questions into one
+- **`augment()`** - Rewrite text in different augmentations
+- **`summarize()`** - Summarize the given text
+- **`translate()`** - Translate text between languages
+- **`propositionize()`** - Convert a text into atomic, independent, meaningful sentences 
+- **`is_fact()`** - Check whether a statement is a fact based on the source text
+- **`run_custom()`** - Custom tool that can do almost anything
 
 ---
 
@@ -66,16 +66,14 @@ pip install -U hamtaa-texttools
 
 ## 📊 Tool Quality Tiers
 
-| Status | Meaning | Tools | Use in Production? |
+| Status | Meaning | Tools | Safe for Production? |
 |--------|---------|----------|-------------------|
-| **✅ Production** | Evaluated, tested, stable. | `categorize()` (list mode), `extract_keywords()`, `extract_entities()`, `is_question()`, `text_to_question()`, `merge_questions()`, `rewrite()`, `subject_to_question()`, `summarize()`, `run_custom()` | **Yes** - ready for reliable use. |
-| **🧪 Experimental** | Added to the package but **not fully evaluated**. Functional, but quality may vary. | `categorize()` (tree mode), `translate()`, `propositionize()`, `check_fact()` | **Use with caution** - outputs not yet validated. |
+| **✅ Production** | Evaluated and tested. | `categorize()` (list mode), `extract_keywords()`, `extract_entities()`, `is_question()`, `to_question()`, `merge_questions()`, `augment()`, `summarize()`, `run_custom()` | **Yes** - ready for reliable use. |
+| **🧪 Experimental** | Added to the package but **not fully evaluated**. | `categorize()` (tree mode), `translate()`, `propositionize()`, `is_fact()` | **Use with caution** |
 
 ---
 
-## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator`, `priority` and `timeout` parameters
-
-TextTools provides several optional flags to customize LLM behavior:
+## ⚙️ Additional Parameters
 
 - **`with_analysis: bool`** → Adds a reasoning step before generating the final output.
 **Note:** This doubles token usage per call.
@@ -85,17 +83,17 @@ TextTools provides several optional flags to customize LLM behavior:
 
 - **`output_lang: str`** → Forces the model to respond in a specific language.
 
-- **`user_prompt: str`** → Allows you to inject a custom instruction or into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
+- **`user_prompt: str`** → Allows you to inject a custom instruction into the model alongside the main template.
 
-- **`temperature: float`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
+- **`temperature: float`** → Determines how creative the model should respond. Takes a float number between `0.0` and `2.0`.
 
-- **`validator: Callable (Experimental)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a boolean. If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can also specify `max_validation_retries=<N>`.
+- **`validator: Callable (Experimental)`** → Forces the tool to validate the output result based on your validator function. Validator should return a boolean. If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can also specify `max_validation_retries=<N>`.
 
-- **`priority: int (Experimental)`** → Task execution priority level. Affects processing order in queues.
+- **`priority: int (Experimental)`** → Affects processing order in queues.  
 **Note:** This feature works if it's supported by the model and vLLM.
 
-- **`timeout: float`** → Maximum time in seconds to wait for the response before raising a timeout error
-**Note:** This feature only exists in `AsyncTheTool`.
+- **`timeout: float`** → Maximum time in seconds to wait for the response before raising a timeout error.  
+**Note:** This feature is only available in `AsyncTheTool`.
 
 
 ---
@@ -107,12 +105,14 @@ Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel wit
 - **`analysis: str`**
 - **`logprobs: list`**
 - **`errors: list[str]`**
-- **`ToolOutputMetadata`** →
+- **`ToolOutputMetadata`**  
     - **`tool_name: str`**
     - **`processed_at: datetime`**
     - **`execution_time: float`**
 
-**Note:** You can use `repr(ToolOutput)` to print your output with all the details.
+- Serialize output to JSON using the `to_json()` method.
+- Verify operation success with the `is_successful()` method.
+- Convert output to a dictionary with the `to_dict()` method.
 
 ---
 
@@ -130,13 +130,13 @@ Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel wit
 from openai import OpenAI
 from texttools import TheTool
 
-client = OpenAI(base_url = "your_url", API_KEY = "your_api_key")
+client = OpenAI(base_url="your_url", API_KEY="your_api_key")
 model = "model_name"
 
 the_tool = TheTool(client=client, model=model)
 
 detection = the_tool.is_question("Is this project open source?")
-print(repr(detection))
+print(detection.to_json())
 ```
 
 ---
@@ -154,24 +154,24 @@ async def main():
 
     async_the_tool = AsyncTheTool(client=async_client, model=model)
     
-    translation_task = async_the_tool.translate("سلام، حالت چطوره؟", target_language="English")
-    keywords_task = async_the_tool.extract_keywords("Tomorrow, we will be dead by the car crash")
+    translation_task = async_the_tool.translate("سلام، حالت چطوره؟", target_lang="English")
+    keywords_task = async_the_tool.extract_keywords("This open source project is great for processing large datasets!")
 
     (translation, keywords) = await asyncio.gather(translation_task, keywords_task)
-    print(repr(translation))
-    print(repr(keywords))
+    
+    print(translation.to_json())
+    print(keywords.to_json())
 
 asyncio.run(main())
 ```
 
 ---
 
-## 👍 Use Cases
+## ✅ Use Cases
 
 Use **TextTools** when you need to:
 
-- 🔍 **Classify** large datasets quickly without model training  
-- 🌍 **Translate** and process multilingual corpora with ease  
+- 🔍 **Classify** large datasets quickly without model training   
 - 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
 - 📊 **Analyze** large text collections using embeddings and categorization  
 
@@ -181,9 +181,3 @@ Use **TextTools** when you need to:
 
 Contributions are welcome!  
 Feel free to **open issues, suggest new features, or submit pull requests**.  
-
----
-
-## 🌿 License
-
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

hamtaa_texttools-2.0.0.dist-info/RECORD

@@ -0,0 +1,30 @@
+hamtaa_texttools-2.0.0.dist-info/licenses/LICENSE,sha256=gqxbR8wqI3utd__l3Yn6_dQ3Pou1a17W4KmydbvZGok,1084
+texttools/__init__.py,sha256=AHpTq1BbL3sWCaFiIjlSkqNfNqweq-qm2EIOSmUZRJ0,175
+texttools/models.py,sha256=CQnO1zkKHFyqeMWrYGA4IyXQ7YYLVc3Xz1WaXbXzDLw,4634
+texttools/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/core/exceptions.py,sha256=6SDjUL1rmd3ngzD3ytF4LyTRj3bQMSFR9ECrLoqXXHw,395
+texttools/core/internal_models.py,sha256=CmRtXGZRn5fZ18lVb42N8LrZXvJb6WwdjIhgiotWJdA,1952
+texttools/core/utils.py,sha256=jqXHXU1DWDKWhK0HHSjnjq4_TLg3FMcnRzrwTF1eqqc,9744
+texttools/core/operators/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/core/operators/async_operator.py,sha256=HOi9gUwIffJUtyp8WLNbMpxI8jnafNDrbtLl6vyPcUs,6221
+texttools/core/operators/sync_operator.py,sha256=yM14fsku-4Nf60lPUVePaB9Lu8HbGKb4ubwoizVWuYQ,6126
+texttools/prompts/augment.yaml,sha256=O-LMVyrihr0GQ8hp2Lx6uIR8Jh83bUDS9UZ-dvYOP7k,5453
+texttools/prompts/categorize.yaml,sha256=kN4uRPOC7q6A13bdCIox60vZZ8sgRiTtquv-kqIvTsk,1133
+texttools/prompts/extract_entities.yaml,sha256=-qe1eEvN-8nJ2_GLjeoFAPVORCPYUzsIt7UGXD485bE,648
+texttools/prompts/extract_keywords.yaml,sha256=jP74HFa4Dka01d1COStEBbdzW5onqwocwyyVsmNpECs,3276
+texttools/prompts/is_fact.yaml,sha256=kqF527DEdnlL3MG5tF1Z3ci_sRxmGv7dgNR2SuElq4Y,719
+texttools/prompts/is_question.yaml,sha256=C-ynlt0qHpUM4BAIh0oI7UJ5BxCNU9-GR9T5864jeto,496
+texttools/prompts/merge_questions.yaml,sha256=zgZs8BcwseZy1GsD_DvVGtw0yuCCc6xsK8VDmuHI2V0,1844
+texttools/prompts/propositionize.yaml,sha256=xTw3HQrxtxoMpkf8a9is0uZZ0AG4IDNfh7XE0aVlNso,1441
+texttools/prompts/run_custom.yaml,sha256=hSfR4BMJNUo9nP_AodPU7YTnhR-X_G-W7Pz0ROQzoI0,133
+texttools/prompts/summarize.yaml,sha256=0aKYFRDxODqOOEhSexi-hn3twLwkMFVmi7rtAifnCuA,464
+texttools/prompts/to_question.yaml,sha256=n8Bn28QjvSHwPHQLwRYpZ2IsaaBsq4pK9Dp_i0xk8eg,2210
+texttools/prompts/translate.yaml,sha256=omtC-TlFYMidy8WqRe7idUtKNiK4g3IhEl-iyufOwjk,649
+texttools/tools/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/tools/async_tools.py,sha256=4-SwsjFqT_mGTK7reCqV-4VXxkyYkyuY3WYJcwkRMJs,44788
+texttools/tools/sync_tools.py,sha256=trmWBf4lvF9h_cB1DelOC4MFWDaQuQ0FDcdc-gCMVxo,40534
+hamtaa_texttools-2.0.0.dist-info/METADATA,sha256=NJYJ6K5HolttdX8_2rCq_w62N6k22mqu0hqO5Qhhy8I,6968
+hamtaa_texttools-2.0.0.dist-info/WHEEL,sha256=qELbo2s1Yzl39ZmrAibXA2jjPLUYfnVhUNTlyF1rq0Y,92
+hamtaa_texttools-2.0.0.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-2.0.0.dist-info/RECORD,,

{hamtaa_texttools-1.3.1.dist-info → hamtaa_texttools-2.0.0.dist-info}/WHEEL

@@ -1,5 +1,5 @@
 Wheel-Version: 1.0
-Generator: setuptools (80.9.0)
+Generator: setuptools (80.10.1)
 Root-Is-Purelib: true
 Tag: py3-none-any
 

{hamtaa_texttools-1.3.1.dist-info → hamtaa_texttools-2.0.0.dist-info}/licenses/LICENSE

@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+SOFTWARE.

texttools/__init__.py

@@ -2,4 +2,4 @@ from .models import CategoryTree
 from .tools.async_tools import AsyncTheTool
 from .tools.sync_tools import TheTool
 
-__all__ = ["TheTool", "AsyncTheTool", "CategoryTree"]
+__all__ = ["CategoryTree", "AsyncTheTool", "TheTool"]

texttools/core/internal_models.py

@@ -1,4 +1,4 @@
-from typing import Any, Literal, Type
+from typing import Any, Literal
 
 from pydantic import BaseModel, Field, create_model
 
@@ -10,18 +10,24 @@ class OperatorOutput(BaseModel):
 
 
 class Str(BaseModel):
-    result: str = Field(..., description="The output string", example="text")
+    result: str = Field(
+        ..., description="The output string", json_schema_extra={"example": "text"}
+    )
 
 
 class Bool(BaseModel):
     result: bool = Field(
-        ..., description="Boolean indicating the output state", example=True
+        ...,
+        description="Boolean indicating the output state",
+        json_schema_extra={"example": True},
     )
 
 
 class ListStr(BaseModel):
     result: list[str] = Field(
-        ..., description="The output list of strings", example=["text_1", "text_2"]
+        ...,
+        description="The output list of strings",
+        json_schema_extra={"example": ["text_1", "text_2", "text_3"]},
     )
 
 
@@ -29,19 +35,26 @@ class ListDictStrStr(BaseModel):
     result: list[dict[str, str]] = Field(
         ...,
         description="List of dictionaries containing string key-value pairs",
-        example=[{"text": "Mohammad", "type": "PER"}, {"text": "Iran", "type": "LOC"}],
+        json_schema_extra={
+            "example": [
+                {"text": "Mohammad", "type": "PER"},
+                {"text": "Iran", "type": "LOC"},
+            ]
+        },
     )
 
 
 class ReasonListStr(BaseModel):
     reason: str = Field(..., description="Thinking process that led to the output")
     result: list[str] = Field(
-        ..., description="The output list of strings", example=["text_1", "text_2"]
+        ...,
+        description="The output list of strings",
+        json_schema_extra={"example": ["text_1", "text_2", "text_3"]},
     )
 
 
-# This function is needed to create CategorizerOutput with dynamic categories
-def create_dynamic_model(allowed_values: list[str]) -> Type[BaseModel]:
+# Create CategorizerOutput with dynamic categories
+def create_dynamic_model(allowed_values: list[str]) -> type[BaseModel]:
     literal_type = Literal[*allowed_values]
 
     CategorizerOutput = create_model(

texttools/core/operators/async_operator.py

@@ -1,15 +1,12 @@
 from collections.abc import Callable
-from typing import Any, Type, TypeVar
+from typing import Any
 
 from openai import AsyncOpenAI
 from pydantic import BaseModel
 
-from ..engine import OperatorUtils, PromptLoader
 from ..exceptions import LLMError, PromptError, TextToolsError, ValidationError
 from ..internal_models import OperatorOutput
-
-# Base Model type for output models
-T = TypeVar("T", bound=BaseModel)
+from ..utils import OperatorUtils
 
 
 class AsyncOperator:
@@ -46,15 +43,15 @@ class AsyncOperator:
     async def _parse_completion(
         self,
         main_message: list[dict[str, str]],
-        output_model: Type[T],
+        output_model: type[BaseModel],
         temperature: float,
         logprobs: bool,
         top_logprobs: int,
         priority: int | None,
-    ) -> tuple[T, Any]:
+    ) -> tuple[BaseModel, Any]:
         """
         Parses a chat completion using OpenAI's structured output format.
-        Returns both the parsed Any and the raw completion for logprobs.
+        Returns both the parsed and the completion for logprobs.
         """
         try:
             request_kwargs = {
@@ -92,7 +89,6 @@ class AsyncOperator:
 
     async def run(
         self,
-        # User parameters
         text: str,
         with_analysis: bool,
         output_lang: str | None,
@@ -103,9 +99,8 @@ class AsyncOperator:
         validator: Callable[[Any], bool] | None,
         max_validation_retries: int | None,
         priority: int | None,
-        # Internal parameters
         tool_name: str,
-        output_model: Type[T],
+        output_model: type[BaseModel],
         mode: str | None,
         **extra_kwargs,
     ) -> OperatorOutput:
@@ -113,8 +108,7 @@ class AsyncOperator:
         Execute the LLM pipeline with the given input text.
         """
         try:
-            prompt_loader = PromptLoader()
-            prompt_configs = prompt_loader.load(
+            prompt_configs = OperatorUtils.load_prompt(
                 prompt_file=tool_name + ".yaml",
                 text=text.strip(),
                 mode=mode,
@@ -129,11 +123,10 @@ class AsyncOperator:
                 )
                 analysis = await self._analyze_completion(analyze_message)
 
-            main_message = OperatorUtils.build_message(
-                OperatorUtils.build_main_prompt(
-                    prompt_configs["main_template"], analysis, output_lang, user_prompt
-                )
+            main_prompt = OperatorUtils.build_main_prompt(
+                prompt_configs["main_template"], analysis, output_lang, user_prompt
             )
+            main_message = OperatorUtils.build_message(main_prompt)
 
             parsed, completion = await self._parse_completion(
                 main_message,
@@ -144,7 +137,7 @@ class AsyncOperator:
                 priority,
             )
 
-            # Retry logic if validation fails
+            # Retry logic in case output validation fails
             if validator and not validator(parsed.result):
                 if (
                     not isinstance(max_validation_retries, int)
@@ -154,7 +147,6 @@ class AsyncOperator:
 
                 succeeded = False
                 for _ in range(max_validation_retries):
-                    # Generate a new temperature to retry
                     retry_temperature = OperatorUtils.get_retry_temp(temperature)
 
                     try:

texttools/core/operators/sync_operator.py

@@ -1,15 +1,12 @@
 from collections.abc import Callable
-from typing import Any, Type, TypeVar
+from typing import Any
 
 from openai import OpenAI
 from pydantic import BaseModel
 
-from ..engine import OperatorUtils, PromptLoader
 from ..exceptions import LLMError, PromptError, TextToolsError, ValidationError
 from ..internal_models import OperatorOutput
-
-# Base Model type for output models
-T = TypeVar("T", bound=BaseModel)
+from ..utils import OperatorUtils
 
 
 class Operator:
@@ -46,15 +43,15 @@ class Operator:
     def _parse_completion(
         self,
         main_message: list[dict[str, str]],
-        output_model: Type[T],
+        output_model: type[BaseModel],
         temperature: float,
         logprobs: bool,
         top_logprobs: int,
         priority: int | None,
-    ) -> tuple[T, Any]:
+    ) -> tuple[BaseModel, Any]:
         """
         Parses a chat completion using OpenAI's structured output format.
-        Returns both the parsed Any and the raw completion for logprobs.
+        Returns both the parsed and the completion for logprobs.
         """
         try:
             request_kwargs = {
@@ -90,7 +87,6 @@ class Operator:
 
     def run(
         self,
-        # User parameters
         text: str,
         with_analysis: bool,
         output_lang: str | None,
@@ -101,9 +97,8 @@ class Operator:
         validator: Callable[[Any], bool] | None,
         max_validation_retries: int | None,
         priority: int | None,
-        # Internal parameters
         tool_name: str,
-        output_model: Type[T],
+        output_model: type[BaseModel],
         mode: str | None,
         **extra_kwargs,
     ) -> OperatorOutput:
@@ -111,8 +106,7 @@ class Operator:
         Execute the LLM pipeline with the given input text.
         """
         try:
-            prompt_loader = PromptLoader()
-            prompt_configs = prompt_loader.load(
+            prompt_configs = OperatorUtils.load_prompt(
                 prompt_file=tool_name + ".yaml",
                 text=text.strip(),
                 mode=mode,
@@ -127,11 +121,10 @@ class Operator:
                 )
                 analysis = self._analyze_completion(analyze_message)
 
-            main_message = OperatorUtils.build_message(
-                OperatorUtils.build_main_prompt(
-                    prompt_configs["main_template"], analysis, output_lang, user_prompt
-                )
+            main_prompt = OperatorUtils.build_main_prompt(
+                prompt_configs["main_template"], analysis, output_lang, user_prompt
             )
+            main_message = OperatorUtils.build_message(main_prompt)
 
             parsed, completion = self._parse_completion(
                 main_message,
@@ -142,7 +135,7 @@ class Operator:
                 priority,
             )
 
-            # Retry logic if validation fails
+            # Retry logic in case output validation fails
             if validator and not validator(parsed.result):
                 if (
                     not isinstance(max_validation_retries, int)
@@ -152,7 +145,6 @@ class Operator:
 
                 succeeded = False
                 for _ in range(max_validation_retries):
-                    # Generate a new temperature to retry
                     retry_temperature = OperatorUtils.get_retry_temp(temperature)
 
                     try: