hamtaa-texttools 1.0.5__py3-none-any.whl → 1.1.16__py3-none-any.whl

hamtaa_texttools-1.1.16.dist-info/METADATA

@@ -0,0 +1,255 @@
+Metadata-Version: 2.4
+Name: hamtaa-texttools
+Version: 1.1.16
+Summary: A high-level NLP toolkit built on top of modern LLMs.
+Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>
+License: MIT License
+        
+        Copyright (c) 2025 Hamtaa
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+Keywords: nlp,llm,text-processing,openai
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai==1.97.1
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Dynamic: license-file
+
+# TextTools
+
+## 📌 Overview
+
+**TextTools** is a high-level **NLP toolkit** built on top of modern **LLMs**.  
+
+It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
+
+It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
+
+---
+
+## ✨ Features
+
+TextTools provides a rich collection of high-level NLP utilities,
+Each tool is designed to work with structured outputs (JSON / Pydantic).
+
+- **`categorize()`** - Classifies text into given categories (You have to create a category tree)
+- **`extract_keywords()`** - Extracts keywords from text
+- **`extract_entities()`** - Named Entity Recognition (NER) system
+- **`is_question()`** - Binary detection of whether input is a question
+- **`text_to_question()`** - Generates questions from text
+- **`merge_questions()`** - Merges multiple questions with different modes
+- **`rewrite()`** - Rewrites text with different wording/meaning
+- **`subject_to_question()`** - Generates questions about a specific subject
+- **`summarize()`** - Text summarization
+- **`translate()`** - Text translation between languages
+- **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
+
+---
+
+## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator` and `priority` parameters
+
+TextTools provides several optional flags to customize LLM behavior:
+
+- **`with_analysis (bool)`** → Adds a reasoning step before generating the final output.
+**Note:** This doubles token usage per call because it triggers an additional LLM request.
+
+- **`logprobs (bool)`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
+**Note:** This feature works if it's supported by the model.
+
+- **`output_lang (str)`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
+
+- **`user_prompt (str)`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
+
+- **`temperature (float)`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
+
+- **`validator (Callable)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a bool (True if there were no problem, False if the validation fails.) If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can specify `max_validation_retries=<N>` to change the number of retries.
+
+- **`priority (int)`** → Task execution priority level. Higher values = higher priority. Affects processing order in queues.
+**Note:** This feature works if it's supported by the model and vLLM.
+
+**Note:** There might be some tools that don't support some of the parameters above.
+
+---
+
+## 🧩 ToolOutput
+
+Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
+- **`result: Any`** → The output of LLM
+- **`analysis: str`** → The reasoning step before generating the final output
+- **`logprobs: list`** → Token-level probabilities for the generated output 
+- **`process: str`** → The tool name which processed the input
+- **`processed_at: datetime`** → The process time
+- **`execution_time: float`** → The execution time (seconds)
+- **`errors: list[str]`** → Any error that have occured during calling LLM
+
+**Note:** You can use `repr(ToolOutput)` to see details of your ToolOutput.
+
+---
+
+## 🚀 Installation
+
+Install the latest release via PyPI:
+
+```bash
+pip install -U hamtaa-texttools
+```
+
+---
+
+## 🧨 Sync vs Async
+| Tool         | Style   | Use case                                    |
+|--------------|---------|---------------------------------------------|
+| `TheTool`    | Sync    | Simple scripts, sequential workflows        |
+| `AsyncTheTool` | Async | High-throughput apps, APIs, concurrent tasks |
+
+---
+
+## ⚡ Quick Start (Sync)
+
+```python
+from openai import OpenAI
+from texttools import TheTool
+
+# Create your OpenAI client
+client = OpenAI(base_url = "your_url", API_KEY = "your_api_key")
+
+# Specify the model
+model = "gpt-4o-mini"
+
+# Create an instance of TheTool
+the_tool = TheTool(client=client, model=model)
+
+# Example: Question Detection
+detection = the_tool.is_question("Is this project open source?", logprobs=True, top_logprobs=2)
+print(detection.result)
+print(detection.logprobs)
+# Output: True + logprobs
+
+# Example: Translation
+translation = the_tool.translate("سلام، حالت چطوره؟" target_language="English", with_analysis=True)
+print(translation.result)
+print(translation.analysis)
+# Output: "Hi! How are you?"  + analysis
+```
+
+---
+
+## ⚡ Quick Start (Async)
+
+```python
+import asyncio
+from openai import AsyncOpenAI
+from texttools import AsyncTheTool
+
+async def main():
+    # Create your AsyncOpenAI client
+    async_client = AsyncOpenAI(base_url="your_url", api_key="your_api_key")
+
+    # Specify the model
+    model = "gpt-4o-mini"
+
+    # Create an instance of AsyncTheTool
+    async_the_tool = AsyncTheTool(client=async_client, model=model)
+    
+    # Example: Async Translation and Keyword Extraction
+    translation_task = async_the_tool.translate("سلام، حالت چطوره؟", target_language="English")
+    keywords_task = async_the_tool.extract_keywords("Tomorrow, we will be dead by the car crash")
+
+    (translation, keywords) = await asyncio.gather(translation_task, keywords_task)
+    print(translation.result)
+    print(keywords.result)
+
+asyncio.run(main())
+```
+
+---
+
+## 👍 Use Cases
+
+Use **TextTools** when you need to:
+
+- 🔍 **Classify** large datasets quickly without model training  
+- 🌍 **Translate** and process multilingual corpora with ease  
+- 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
+- 📊 **Analyze** large text collections using embeddings and categorization  
+
+---
+
+## 🔍 Logging
+
+TextTools uses Python's standard `logging` module. The library's default logger level is `WARNING`, so if you want to modify it, follow instructions:
+
+
+```python
+import logging
+
+# Default: warnings and errors only
+logging.basicConfig(level=logging.WARNING)
+
+# Debug everything (verbose)
+logging.basicConfig(level=logging.DEBUG)
+
+# Complete silence
+logging.basicConfig(level=logging.CRITICAL)
+```
+
+---
+
+## 📚 Batch Processing
+
+Process large datasets efficiently using OpenAI's batch API.
+
+## ⚡ Quick Start (Batch)
+
+```python
+from pydantic import BaseModel
+from texttools import BatchJobRunner, BatchConfig
+
+# Configure your batch job
+config = BatchConfig(
+    system_prompt="Extract entities from the text",
+    job_name="entity_extraction",
+    input_data_path="data.json",
+    output_data_filename="results.json",
+    model="gpt-4o-mini"
+)
+
+# Define your output schema
+class Output(BaseModel):
+    entities: list[str]
+
+# Run the batch job
+runner = BatchJobRunner(config, output_model=Output)
+runner.run()
+```
+
+---
+
+## 🤝 Contributing
+
+Contributions are welcome!  
+Feel free to **open issues, suggest new features, or submit pull requests**.  
+
+---
+
+## 🌿 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

hamtaa_texttools-1.1.16.dist-info/RECORD

@@ -0,0 +1,31 @@
+hamtaa_texttools-1.1.16.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
+texttools/__init__.py,sha256=dc81lXGWP29k7oVvq2BMoMotz6lgiwX4PO2jHHBe2S8,317
+texttools/batch/batch_config.py,sha256=m1UgILVKjNdWE6laNbfbG4vgi4o2fEegGZbeoam6pnY,749
+texttools/batch/batch_runner.py,sha256=9e4SPLlvLHHs3U7bHkuuMVw8TFNwsGUzRjkAMKN4_ik,9378
+texttools/batch/internals/batch_manager.py,sha256=UoBe76vmFG72qrSaGKDZf4HzkykFBkkkbL9TLfV8TuQ,8730
+texttools/batch/internals/utils.py,sha256=F1_7YlVFKhjUROAFX4m0SaP8KiZVZyHRMIIB87VUGQc,373
+texttools/prompts/README.md,sha256=-5YO93CN93QLifqZpUeUnCOCBbDiOTV-cFQeJ7Gg0I4,1377
+texttools/prompts/categorize.yaml,sha256=F7VezB25B_sT5yoC25ezODBddkuDD5lUHKetSpx9FKI,2743
+texttools/prompts/detect_entity.yaml,sha256=1rhMkJOjxSQcT4j_c5SRcIm77AUdeG-rUmeidb6VOFc,981
+texttools/prompts/extract_entities.yaml,sha256=KiKjeDpHaeh3JVtZ6q1pa3k4DYucUIU9WnEcRTCA-SE,651
+texttools/prompts/extract_keywords.yaml,sha256=Vj4Tt3vT6LtpOo_iBZPo9oWI50oVdPGXe5i8yDR8ex4,3177
+texttools/prompts/is_question.yaml,sha256=d0-vKRbXWkxvO64ikvxRjEmpAXGpCYIPGhgexvPPjws,471
+texttools/prompts/merge_questions.yaml,sha256=0J85GvTirZB4ELwH3sk8ub_WcqqpYf6PrMKr3djlZeo,1792
+texttools/prompts/rewrite.yaml,sha256=LO7He_IA3MZKz8a-LxH9DHJpOjpYwaYN1pbjp1Y0tFo,5392
+texttools/prompts/run_custom.yaml,sha256=38OkCoVITbuuS9c08UZSP1jZW4WjSmRIi8fR0RAiPu4,108
+texttools/prompts/subject_to_question.yaml,sha256=C7x7rNNm6U_ZG9HOn6zuzYOtvJUZ2skuWbL1-aYdd3E,1147
+texttools/prompts/summarize.yaml,sha256=o6rxGPfWtZd61Duvm8NVvCJqfq73b-wAuMSKR6UYUqY,459
+texttools/prompts/text_to_question.yaml,sha256=UheKYpDn6iyKI8NxunHZtFpNyfCLZZe5cvkuXpurUJY,783
+texttools/prompts/translate.yaml,sha256=mGT2uBCei6uucWqVbs4silk-UV060v3G0jnt0P6sr50,634
+texttools/tools/async_tools.py,sha256=vNAg0gxwUZPsMS4q8JCv7RlYymS8l_5FsFI5adEYT7w,34376
+texttools/tools/sync_tools.py,sha256=hFifFa9YatvSeGif2E_bIG006eMdIBr6SV9HsZ_dAlg,34187
+texttools/tools/internals/async_operator.py,sha256=1TMr8e1qbE9GSz8jl0q3MKdM8lIYE-1ZuSxHjYPqKHI,7198
+texttools/tools/internals/formatters.py,sha256=tACNLP6PeoqaRpNudVxBaHA25zyWqWYPZQuYysIu88g,941
+texttools/tools/internals/models.py,sha256=2QnvMiijuSqOqpCl026848rJy_pHNbRoDESlQvcdHlk,5839
+texttools/tools/internals/operator_utils.py,sha256=w1k0RJ_W_CRbVc_J2w337VuL-opHpHiCxfhEOwtyuOo,1856
+texttools/tools/internals/prompt_loader.py,sha256=4g6-U8kqrGN7VpNaRcrBcnF-h03PXjUDBP0lL0_4EZY,1953
+texttools/tools/internals/sync_operator.py,sha256=4-V__o55Q8w29lWxkhG4St-exZLZTfBbiW76knOXbc0,7106
+hamtaa_texttools-1.1.16.dist-info/METADATA,sha256=DL-cjlGMv7bft8QVd-pn5E_tNDuPgQHkTKGl4YTosGw,9555
+hamtaa_texttools-1.1.16.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+hamtaa_texttools-1.1.16.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-1.1.16.dist-info/RECORD,,

texttools/__init__.py

@@ -1,9 +1,7 @@
-from .batch import BatchJobRunner, SimpleBatchManager
-from .tools import AsyncTheTool, TheTool
+from .batch.batch_runner import BatchJobRunner
+from .batch.batch_config import BatchConfig
+from .tools.sync_tools import TheTool
+from .tools.async_tools import AsyncTheTool
+from .tools.internals.models import CategoryTree
 
-__all__ = [
-    "TheTool",
-    "AsyncTheTool",
-    "SimpleBatchManager",
-    "BatchJobRunner",
-]
+__all__ = ["TheTool", "AsyncTheTool", "BatchJobRunner", "BatchConfig", "CategoryTree"]

texttools/batch/batch_config.py

@@ -0,0 +1,26 @@
+from dataclasses import dataclass
+from collections.abc import Callable
+
+from texttools.batch.internals.utils import import_data, export_data
+
+
+@dataclass
+class BatchConfig:
+    """
+    Configuration for batch job runner.
+    """
+
+    system_prompt: str = ""
+    job_name: str = ""
+    input_data_path: str = ""
+    output_data_filename: str = ""
+    model: str = "gpt-4.1-mini"
+    MAX_BATCH_SIZE: int = 100
+    MAX_TOTAL_TOKENS: int = 2_000_000
+    CHARS_PER_TOKEN: float = 2.7
+    PROMPT_TOKEN_MULTIPLIER: int = 1_000
+    BASE_OUTPUT_DIR: str = "Data/batch_entity_result"
+    import_function: Callable = import_data
+    export_function: Callable = export_data
+    poll_interval_seconds: int = 30
+    max_retries: int = 3