hamtaa-texttools 2.1.0__py3-none-any.whl → 2.3.0__py3-none-any.whl

{hamtaa_texttools-2.1.0.dist-info → hamtaa_texttools-2.3.0.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 2.1.0
+Version: 2.3.0
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Erfan Moosavi <erfanmoosavi84@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
 Maintainer-email: Erfan Moosavi <erfanmoosavi84@gmail.com>, Tohidi <the.mohammad.tohidi@gmail.com>
@@ -17,6 +17,7 @@ License-File: LICENSE
 Requires-Dist: dotenv>=0.9.9
 Requires-Dist: openai>=1.97.1
 Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pytest>=9.0.2
 Requires-Dist: pyyaml>=6.0
 Dynamic: license-file
 
@@ -29,7 +30,10 @@ Dynamic: license-file
 
 **TextTools** is a high-level **NLP toolkit** built on top of **LLMs**.  
 
-It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
+It provides three API styles for maximum flexibility:
+- Sync API (`TheTool`) - Simple, sequential operations
+- Async API (`AsyncTheTool`) - High-performance async operations
+- Batch API (`BatchTheTool`) - Process multiple texts in parallel with built-in concurrency control
 
 It provides ready-to-use utilities for **translation, question detection, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
 
@@ -76,8 +80,6 @@ pip install -U hamtaa-texttools
 
 ## ⚙️ Additional Parameters
 
-- **`raise_on_error: bool`** → (`TheTool/AsyncTheTool` parameter) Raise errors (True) or return them in output (False). Default is True.
-
 - **`with_analysis: bool`** → Adds a reasoning step before generating the final output.
 **Note:** This doubles token usage per call.
 
@@ -98,32 +100,49 @@ pip install -U hamtaa-texttools
 - **`timeout: float`** → Maximum time in seconds to wait for the response before raising a timeout error.  
 **Note:** This feature is only available in `AsyncTheTool`.
 
+- **`raise_on_error: bool`** → (`TheTool/AsyncTheTool`) Raise errors (True) or return them in output (False). Default is True.
+
+- **`max_concurrency: int`** → (`BatchTheTool` only) Maximum number of concurrent API calls. Default is 5.
 
 ---
 
 ## 🧩 ToolOutput
 
 Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
+
 - **`result: Any`**
 - **`analysis: str`**
 - **`logprobs: list`**
 - **`errors: list[str]`**
-- **`ToolOutputMetadata`**  
+- **`ToolOutputMetadata`**
     - **`tool_name: str`**
+    - **`processed_by: str`**
     - **`processed_at: datetime`**
     - **`execution_time: float`**
+    - **`token_usage: TokenUsage`**
+        - **`completion_usage: CompletionUsage`**
+            - **`prompt_tokens: int`**
+            - **`completion_tokens: int`**
+            - **`total_tokens: int`**
+        - **`analyze_usage: AnalyzeUsage`**
+            - **`prompt_tokens: int`**
+            - **`completion_tokens: int`**
+            - **`total_tokens: int`**
 
 - Serialize output to JSON using the `to_json()` method.
 - Verify operation success with the `is_successful()` method.
 - Convert output to a dictionary with the `to_dict()` method.
 
+**Note:** For BatchTheTool: Each method returns a `list[ToolOutput]` containing results for all input texts.
+
 ---
 
-## 🧨 Sync vs Async
-| Tool         | Style   | Use case                                    |
-|--------------|---------|---------------------------------------------|
-| `TheTool`    | Sync    | Simple scripts, sequential workflows        |
-| `AsyncTheTool` | Async | High-throughput apps, APIs, concurrent tasks |
+## 🧨 Sync vs Async vs Batch
+| Tool | Style | Use Case | Best For |
+|------|-------|----------|----------|
+| `TheTool` | **Sync** | Simple scripts, sequential workflows | • Quick prototyping<br>• Simple scripts<br>• Sequential processing<br>• Debugging |
+| `AsyncTheTool` | **Async** | High-throughput applications, APIs, concurrent tasks | • Web APIs<br>• Concurrent operations<br>• High-performance apps<br>• Real-time processing |
+| `BatchTheTool` | **Batch** | Process multiple texts efficiently with controlled concurrency | • Bulk processing<br>• Large datasets<br>• Parallel execution<br>• Resource optimization |
 
 ---
 
@@ -168,6 +187,35 @@ async def main():
 asyncio.run(main())
 ```
 
+## ⚡ Quick Start (Batch)
+
+```python
+import asyncio
+from openai import AsyncOpenAI
+from texttools import BatchTheTool
+
+async def main():
+    async_client = AsyncOpenAI(base_url="your_url", api_key="your_api_key")
+    model = "model_name"
+    
+    batch_the_tool = BatchTheTool(client=async_client, model=model, max_concurrency=3)
+    
+    categories = await batch_tool.categorize(
+        texts=[
+            "Climate change impacts on agriculture",
+            "Artificial intelligence in healthcare",
+            "Economic effects of remote work",
+            "Advancements in quantum computing",
+        ],
+        categories=["Science", "Technology", "Economics", "Environment"],
+    )
+    
+    for i, result in enumerate(categories):
+        print(f"Text {i+1}: {result.result}")
+
+asyncio.run(main())
+```
+
 ---
 
 ## ✅ Use Cases
@@ -176,4 +224,20 @@ Use **TextTools** when you need to:
 
 - 🔍 **Classify** large datasets quickly without model training   
 - 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
-- 📊 **Analyze** large text collections using embeddings and categorization  
+- 📊 **Analyze** large text collections using embeddings and categorization
+
+---
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+---
+
+## 🤝 Contributing
+
+We welcome contributions from the community! - see the [CONTRIBUTING](CONTRIBUTING.md) file for details.
+
+## 📚 Documentation
+
+For detailed documentation, architecture overview, and implementation details, please visit the [docs](docs) directory.

hamtaa_texttools-2.3.0.dist-info/RECORD

@@ -0,0 +1,31 @@
+hamtaa_texttools-2.3.0.dist-info/licenses/LICENSE,sha256=gqxbR8wqI3utd__l3Yn6_dQ3Pou1a17W4KmydbvZGok,1084
+texttools/__init__.py,sha256=c7L4bv0vxwpkZW2XHAnhlK_aVdM6CVZLAA0rmkSfIao,163
+texttools/models.py,sha256=_CLKOij2XvKPzQK2wyfNoBaZo0FEkF584Hdv2BHznLU,4746
+texttools/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/core/__init__.py,sha256=9ymBg2lGljm6wriYt4_-T5HOK8-2hHUwsC_bGdlOCsk,720
+texttools/core/exceptions.py,sha256=6SDjUL1rmd3ngzD3ytF4LyTRj3bQMSFR9ECrLoqXXHw,395
+texttools/core/internal_models.py,sha256=ZauSJWWILnpDEPtAPLvrq49TMKs1BS2znZHD3FeKsRM,3774
+texttools/core/utils.py,sha256=sQTOJmW_6EISEaUI-NWLRymAP1fvA7pZ-RzJRyQveu8,11169
+texttools/core/operators/__init__.py,sha256=PFYEVwqT1mgVc5wok9d_jj_A-3EyOg3-Ssyi7kwkuPk,123
+texttools/core/operators/async_operator.py,sha256=kBiQgu8TEG46AhjlEPysG7EBm4F3xanbBpYIyZqtdC8,6501
+texttools/core/operators/sync_operator.py,sha256=5puFQrBErk4biZ2sG2ivq2AYCqP4tSdppTJUEKEOwUo,6366
+texttools/prompts/augment.yaml,sha256=uJnnP-uEafiATdBx74LiOQWX6spvwcC0J-yfhySfoAM,5423
+texttools/prompts/categorize.yaml,sha256=kN4uRPOC7q6A13bdCIox60vZZ8sgRiTtquv-kqIvTsk,1133
+texttools/prompts/extract_entities.yaml,sha256=-qe1eEvN-8nJ2_GLjeoFAPVORCPYUzsIt7UGXD485bE,648
+texttools/prompts/extract_keywords.yaml,sha256=jP74HFa4Dka01d1COStEBbdzW5onqwocwyyVsmNpECs,3276
+texttools/prompts/is_fact.yaml,sha256=kqF527DEdnlL3MG5tF1Z3ci_sRxmGv7dgNR2SuElq4Y,719
+texttools/prompts/is_question.yaml,sha256=C-ynlt0qHpUM4BAIh0oI7UJ5BxCNU9-GR9T5864jeto,496
+texttools/prompts/merge_questions.yaml,sha256=zgZs8BcwseZy1GsD_DvVGtw0yuCCc6xsK8VDmuHI2V0,1844
+texttools/prompts/propositionize.yaml,sha256=xTw3HQrxtxoMpkf8a9is0uZZ0AG4IDNfh7XE0aVlNso,1441
+texttools/prompts/run_custom.yaml,sha256=hSfR4BMJNUo9nP_AodPU7YTnhR-X_G-W7Pz0ROQzoI0,133
+texttools/prompts/summarize.yaml,sha256=0aKYFRDxODqOOEhSexi-hn3twLwkMFVmi7rtAifnCuA,464
+texttools/prompts/to_question.yaml,sha256=lv4YouHE6yv594-RbO8fysH9aAz_mSms3PpbHzYzQmc,2118
+texttools/prompts/translate.yaml,sha256=FVYhx9GamYyCf7QGDmmhRi_N-H0SwIma8jx_yT6TKNY,659
+texttools/tools/__init__.py,sha256=xsFiKJq66HrgPebssYm5fxV2AyEjbz5urfAuNUsz8oE,168
+texttools/tools/async_tools.py,sha256=2xKlzs--V01JguNNYY9GDcuc43oCJHUNYXTkuEOzo1M,47941
+texttools/tools/batch_tools.py,sha256=hwWutcSWc2k79vZX5Urft1arTgHpDnnxztHZba54xtg,29899
+texttools/tools/sync_tools.py,sha256=tyD-aq7x1yHvfLDnClNUiwqT_ZxHHeq2SNnfv5zVoBo,43688
+hamtaa_texttools-2.3.0.dist-info/METADATA,sha256=ghPfIzci1KJpjKnOcTNQTKo5YvmU65a0bTLHXAhHmYw,9370
+hamtaa_texttools-2.3.0.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+hamtaa_texttools-2.3.0.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-2.3.0.dist-info/RECORD,,

texttools/__init__.py

@@ -1,5 +1,4 @@
 from .models import CategoryTree
-from .tools.async_tools import AsyncTheTool
-from .tools.sync_tools import TheTool
+from .tools import AsyncTheTool, BatchTheTool, TheTool
 
-__all__ = ["CategoryTree", "AsyncTheTool", "TheTool"]
+__all__ = ["CategoryTree", "AsyncTheTool", "BatchTheTool", "TheTool"]

texttools/core/__init__.py

@@ -0,0 +1,34 @@
+from .exceptions import LLMError, PromptError, TextToolsError, ValidationError
+from .internal_models import (
+    Bool,
+    ListDictStrStr,
+    ListStr,
+    ReasonListStr,
+    Str,
+    TokenUsage,
+    create_dynamic_model,
+)
+from .operators import AsyncOperator, Operator
+from .utils import OperatorUtils, TheToolUtils
+
+__all__ = [
+    # Exceptions
+    "LLMError",
+    "PromptError",
+    "TextToolsError",
+    "ValidationError",
+    # Internal models
+    "Bool",
+    "ListDictStrStr",
+    "ListStr",
+    "ReasonListStr",
+    "Str",
+    "TokenUsage",
+    "create_dynamic_model",
+    # Operators
+    "AsyncOperator",
+    "Operator",
+    # Utils
+    "OperatorUtils",
+    "TheToolUtils",
+]

texttools/core/internal_models.py

@@ -1,12 +1,64 @@
+from __future__ import annotations
+
 from typing import Any, Literal
 
 from pydantic import BaseModel, Field, create_model
 
 
+class CompletionUsage(BaseModel):
+    prompt_tokens: int = 0
+    completion_tokens: int = 0
+    total_tokens: int = 0
+
+
+class AnalyzeUsage(BaseModel):
+    prompt_tokens: int = 0
+    completion_tokens: int = 0
+    total_tokens: int = 0
+
+
+class TokenUsage(BaseModel):
+    completion_usage: CompletionUsage = CompletionUsage()
+    analyze_usage: AnalyzeUsage = AnalyzeUsage()
+    total_tokens: int = 0
+
+    def __add__(self, other: TokenUsage) -> TokenUsage:
+        new_completion_usage = CompletionUsage(
+            prompt_tokens=self.completion_usage.prompt_tokens
+            + other.completion_usage.prompt_tokens,
+            completion_tokens=self.completion_usage.completion_tokens
+            + other.completion_usage.completion_tokens,
+            total_tokens=self.completion_usage.total_tokens
+            + other.completion_usage.total_tokens,
+        )
+        new_analyze_usage = AnalyzeUsage(
+            prompt_tokens=self.analyze_usage.prompt_tokens
+            + other.analyze_usage.prompt_tokens,
+            completion_tokens=self.analyze_usage.completion_tokens
+            + other.analyze_usage.completion_tokens,
+            total_tokens=self.analyze_usage.total_tokens
+            + other.analyze_usage.total_tokens,
+        )
+        total_tokens = (
+            new_completion_usage.total_tokens + new_analyze_usage.total_tokens
+        )
+
+        return TokenUsage(
+            completion_usage=new_completion_usage,
+            analyze_usage=new_analyze_usage,
+            total_tokens=total_tokens,
+        )
+
+
 class OperatorOutput(BaseModel):
     result: Any
     analysis: str | None
     logprobs: list[dict[str, Any]] | None
+    token_usage: TokenUsage | None = None
+    prompt_tokens: int | None = None
+    completion_tokens: int | None = None
+    analysis_tokens: int | None = None
+    total_tokens: int | None = None
 
 
 class Str(BaseModel):

texttools/core/operators/__init__.py

@@ -0,0 +1,4 @@
+from .async_operator import AsyncOperator
+from .sync_operator import Operator
+
+__all__ = ["AsyncOperator", "Operator"]

texttools/core/operators/async_operator.py

@@ -18,7 +18,9 @@ class AsyncOperator:
         self._client = client
         self._model = model
 
-    async def _analyze_completion(self, analyze_message: list[dict[str, str]]) -> str:
+    async def _analyze_completion(
+        self, analyze_message: list[dict[str, str]]
+    ) -> tuple[str, Any]:
         try:
             completion = await self._client.chat.completions.create(
                 model=self._model,
@@ -33,7 +35,7 @@ class AsyncOperator:
             if not analysis:
                 raise LLMError("Empty analysis response")
 
-            return analysis
+            return analysis, completion
 
         except Exception as e:
             if isinstance(e, (PromptError, LLMError)):
@@ -116,12 +118,15 @@ class AsyncOperator:
             )
 
             analysis: str | None = None
+            analyze_completion: Any = None
 
             if with_analysis:
                 analyze_message = OperatorUtils.build_message(
                     prompt_configs["analyze_template"]
                 )
-                analysis = await self._analyze_completion(analyze_message)
+                analysis, analyze_completion = await self._analyze_completion(
+                    analyze_message
+                )
 
             main_prompt = OperatorUtils.build_main_prompt(
                 prompt_configs["main_template"], analysis, output_lang, user_prompt
@@ -176,6 +181,9 @@ class AsyncOperator:
                 logprobs=OperatorUtils.extract_logprobs(completion)
                 if logprobs
                 else None,
+                token_usage=OperatorUtils.extract_token_usage(
+                    completion, analyze_completion
+                ),
             )
 
             return operator_output

texttools/core/operators/sync_operator.py

@@ -18,7 +18,9 @@ class Operator:
         self._client = client
         self._model = model
 
-    def _analyze_completion(self, analyze_message: list[dict[str, str]]) -> str:
+    def _analyze_completion(
+        self, analyze_message: list[dict[str, str]]
+    ) -> tuple[str, Any]:
         try:
             completion = self._client.chat.completions.create(
                 model=self._model,
@@ -33,7 +35,7 @@ class Operator:
             if not analysis:
                 raise LLMError("Empty analysis response")
 
-            return analysis
+            return analysis, completion
 
         except Exception as e:
             if isinstance(e, (PromptError, LLMError)):
@@ -114,12 +116,13 @@ class Operator:
             )
 
             analysis: str | None = None
+            analyze_completion: Any = None
 
             if with_analysis:
                 analyze_message = OperatorUtils.build_message(
                     prompt_configs["analyze_template"]
                 )
-                analysis = self._analyze_completion(analyze_message)
+                analysis, analyze_completion = self._analyze_completion(analyze_message)
 
             main_prompt = OperatorUtils.build_main_prompt(
                 prompt_configs["main_template"], analysis, output_lang, user_prompt
@@ -174,6 +177,9 @@ class Operator:
                 logprobs=OperatorUtils.extract_logprobs(completion)
                 if logprobs
                 else None,
+                token_usage=OperatorUtils.extract_token_usage(
+                    completion, analyze_completion
+                ),
             )
 
             return operator_output

texttools/core/utils.py

@@ -9,6 +9,7 @@ from typing import Any
 import yaml
 
 from .exceptions import PromptError
+from .internal_models import AnalyzeUsage, CompletionUsage, TokenUsage
 
 
 class OperatorUtils:
@@ -148,6 +149,38 @@ class OperatorUtils:
         new_temp = base_temp + random.choice([-1, 1]) * random.uniform(0.1, 0.9)
         return max(0.0, min(new_temp, 1.5))
 
+    @staticmethod
+    def extract_token_usage(completion: Any, analyze_completion: Any) -> TokenUsage:
+        completion_usage = completion.usage
+        analyze_usage = analyze_completion.usage if analyze_completion else None
+
+        completion_usage_model = CompletionUsage(
+            prompt_tokens=getattr(completion_usage, "prompt_tokens", 00),
+            completion_tokens=getattr(completion_usage, "completion_tokens", 00),
+            total_tokens=getattr(completion_usage, "total_tokens", 00),
+        )
+        analyze_usage_model = AnalyzeUsage(
+            prompt_tokens=getattr(analyze_usage, "prompt_tokens", 0),
+            completion_tokens=getattr(analyze_usage, "completion_tokens", 0),
+            total_tokens=getattr(analyze_usage, "total_tokens", 0),
+        )
+        total_analyze_tokens = (
+            analyze_usage_model.prompt_tokens + analyze_usage_model.completion_tokens
+            if analyze_completion
+            else 0
+        )
+        total_tokens = (
+            completion_usage_model.prompt_tokens
+            + completion_usage_model.completion_tokens
+            + total_analyze_tokens
+        )
+
+        return TokenUsage(
+            completion_usage=completion_usage_model,
+            analyze_usage=analyze_usage_model,
+            total_tokens=total_tokens,
+        )
+
 
 class TheToolUtils:
     """

texttools/models.py

@@ -5,11 +5,15 @@ from typing import Any
 
 from pydantic import BaseModel, Field
 
+from .core import TokenUsage
+
 
 class ToolOutputMetadata(BaseModel):
     tool_name: str
+    processed_by: str | None = None
     processed_at: datetime = Field(default_factory=datetime.now)
     execution_time: float | None = None
+    token_usage: TokenUsage | None = None
 
 
 class ToolOutput(BaseModel):

texttools/prompts/augment.yaml

@@ -38,25 +38,25 @@ main_template:
     "{text}"
 
   hard_negative: |
-      You are an AI assistant designed to generate high-quality training data for semantic text embedding models.
-      Your task is to create a hard-negative sample for a given "Anchor" text.
+    You are an AI assistant designed to generate high-quality training data for semantic text embedding models.
+    Your task is to create a hard-negative sample for a given "Anchor" text.
 
-      A high-quality hard-negative sample is a sentence that is topically related but semantically distinct from the Anchor.
-      It should share some context (e.g., same domain, same entities) but differ in a crucial piece of information, action, conclusion, or specific detail.
+    A high-quality hard-negative sample is a sentence that is topically related but semantically distinct from the Anchor.
+    It should share some context (e.g., same domain, same entities) but differ in a crucial piece of information, action, conclusion, or specific detail.
 
-      Instructions:
-      - Stay in General Domain: Remain in the same broad domain (e.g., religious topics), but choose a completely different subject matter.
-      - Maintain Topical Overlap: Keep the same domain, subject, or entities (e.g., people, products, concepts) as the Anchor.
-      - Alter a Key Semantic Element: Reverse a key word or condition or place or proper name that completely reverses the meaning of the sentence.
-      - Avoid Being a Paraphrase: The sentence must NOT be semantically equivalent. The core factual claim or intent must be different.
-      - Make it Challenging: The difference should be subtle enough that it requires a deep understanding of the text to identify, not just a simple keyword mismatch.
-      - Maintain Similar Length: The generated sentence should be of roughly the same length and level of detail as the Anchor.
+    Instructions:
+    - Stay in General Domain: Remain in the same broad domain (e.g., religious topics), but choose a completely different subject matter.
+    - Maintain Topical Overlap: Keep the same domain, subject, or entities (e.g., people, products, concepts) as the Anchor.
+    - Alter a Key Semantic Element: Reverse a key word or condition or place or proper name that completely reverses the meaning of the sentence.
+    - Avoid Being a Paraphrase: The sentence must NOT be semantically equivalent. The core factual claim or intent must be different.
+    - Make it Challenging: The difference should be subtle enough that it requires a deep understanding of the text to identify, not just a simple keyword mismatch.
+    - Maintain Similar Length: The generated sentence should be of roughly the same length and level of detail as the Anchor.
 
-      Respond only in JSON format:
-      {{"result": "rewriteen_text"}}
+    Respond only in JSON format:
+    {{"result": "rewriteen_text"}}
 
-      Anchor Text:
-      "{text}"
+    Anchor Text:
+    "{text}"
 
 
 analyze_template:

texttools/prompts/to_question.yaml

@@ -7,7 +7,6 @@ main_template:
     and must not mention any verbs like this, that, he or she in the question.
     
     There is a `reason` key, fill that up with a summerized version of your thoughts.
-    The `reason` must be less than 20 words.
     Don't forget to fill the reason.
 
     Respond only in JSON format:
@@ -23,7 +22,6 @@ main_template:
     and must not mention any verbs like this, that, he or she in the question.
     
     There is a `reason` key, fill that up with a summerized version of your thoughts.
-    The `reason` must be less than 20 words.
     Don't forget to fill the reason.
 
     Respond only in JSON format:

texttools/prompts/translate.yaml

@@ -3,9 +3,9 @@ main_template: |
   Output only the translated text.
 
   Respond only in JSON format:
-  {{"result": "string"}}
+  {{"result": "translated_text"}}
 
-  Don't translate proper name, only transliterate them to {target_lang}
+  Don't translate proper names, only transliterate them to {target_lang}
   
   Translate the following text to {target_lang}:
   {text}

texttools/tools/__init__.py

@@ -0,0 +1,5 @@
+from .async_tools import AsyncTheTool
+from .batch_tools import BatchTheTool
+from .sync_tools import TheTool
+
+__all__ = ["AsyncTheTool", "BatchTheTool", "TheTool"]