hamtaa-texttools 1.1.16__py3-none-any.whl → 1.2.0__py3-none-any.whl

hamtaa_texttools-1.2.0.dist-info/METADATA

@@ -0,0 +1,212 @@
+Metadata-Version: 2.4
+Name: hamtaa-texttools
+Version: 1.2.0
+Summary: A high-level NLP toolkit built on top of modern LLMs.
+Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Erfan Moosavi <erfanmoosavi84@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
+Maintainer-email: Erfan Moosavi <erfanmoosavi84@gmail.com>, Tohidi <the.mohammad.tohidi@gmail.com>
+License: MIT
+Keywords: nlp,llm,text-processing,openai
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.97.1
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Dynamic: license-file
+
+# TextTools
+
+## 📌 Overview
+
+**TextTools** is a high-level **NLP toolkit** built on top of **LLMs**.  
+
+It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
+
+It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
+
+**Note:** Most features of `texttools` are reliable when you use `google/gemma-3n-e4b-it` model.
+
+---
+
+## ✨ Features
+
+TextTools provides a rich collection of high-level NLP utilities,
+Each tool is designed to work with structured outputs.
+
+- **`categorize()`** - Classifies text into given categories
+- **`extract_keywords()`** - Extracts keywords from the text
+- **`extract_entities()`** - Named Entity Recognition (NER) system
+- **`is_question()`** - Binary question detection
+- **`text_to_question()`** - Generates questions from text
+- **`merge_questions()`** - Merges multiple questions into one
+- **`rewrite()`** - Rewrites text in a diffrent way
+- **`subject_to_question()`** - Generates questions about a specific subject
+- **`summarize()`** - Text summarization
+- **`translate()`** - Text translation
+- **`propositionize()`** - Convert text to atomic independence meaningful sentences 
+- **`check_fact()`** - Check whether a statement is relevant to the source text
+- **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
+
+---
+
+## 🚀 Installation
+
+Install the latest release via PyPI:
+
+```bash
+pip install -U hamtaa-texttools
+```
+
+---
+
+## 📊 Tool Quality Tiers
+
+| Status | Meaning | Tools | Use in Production? |
+|--------|---------|----------|-------------------|
+| **✅ Production** | Evaluated, tested, stable. | `categorize()` (list mode), `extract_keywords()`, `extract_entities()`, `is_question()`, `text_to_question()`, `merge_questions()`, `rewrite()`, `subject_to_question()`, `summarize()`, `run_custom()` | **Yes** - ready for reliable use. |
+| **🧪 Experimental** | Added to the package but **not fully evaluated**. Functional, but quality may vary. | `categorize()` (tree mode), `translate()`, `propositionize()`, `check_fact()` | **Use with caution** - outputs not yet validated. |
+
+---
+
+## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator` and `priority` parameters
+
+TextTools provides several optional flags to customize LLM behavior:
+
+- **`with_analysis: bool`** → Adds a reasoning step before generating the final output.
+**Note:** This doubles token usage per call.
+
+- **`logprobs: bool`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
+**Note:** This feature works if it's supported by the model.
+
+- **`output_lang: str`** → Forces the model to respond in a specific language.
+
+- **`user_prompt: str`** → Allows you to inject a custom instruction or into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
+
+- **`temperature: float`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
+
+- **`validator: Callable (Experimental)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a boolean. If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can also specify `max_validation_retries=<N>`.
+
+- **`priority: int (Experimental)`** → Task execution priority level. Affects processing order in queues.
+**Note:** This feature works if it's supported by the model and vLLM.
+
+---
+
+## 🧩 ToolOutput
+
+Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
+- **`result: Any`**
+- **`analysis: str`**
+- **`logprobs: list`**
+- **`errors: list[str]`**
+- **`ToolOutputMetadata`** →
+    - **`tool_name: str`**
+    - **`processed_at: datetime`**
+    - **`execution_time: float`**
+
+**Note:** You can use `repr(ToolOutput)` to print your output with all the details.
+
+---
+
+## 🧨 Sync vs Async
+| Tool         | Style   | Use case                                    |
+|--------------|---------|---------------------------------------------|
+| `TheTool`    | Sync    | Simple scripts, sequential workflows        |
+| `AsyncTheTool` | Async | High-throughput apps, APIs, concurrent tasks |
+
+---
+
+## ⚡ Quick Start (Sync)
+
+```python
+from openai import OpenAI
+from texttools import TheTool
+
+client = OpenAI(base_url = "your_url", API_KEY = "your_api_key")
+model = "model_name"
+
+the_tool = TheTool(client=client, model=model)
+
+detection = the_tool.is_question("Is this project open source?")
+print(repr(detection))
+```
+
+---
+
+## ⚡ Quick Start (Async)
+
+```python
+import asyncio
+from openai import AsyncOpenAI
+from texttools import AsyncTheTool
+
+async def main():
+    async_client = AsyncOpenAI(base_url="your_url", api_key="your_api_key")
+    model = "model_name"
+
+    async_the_tool = AsyncTheTool(client=async_client, model=model)
+    
+    translation_task = async_the_tool.translate("سلام، حالت چطوره؟", target_language="English")
+    keywords_task = async_the_tool.extract_keywords("Tomorrow, we will be dead by the car crash")
+
+    (translation, keywords) = await asyncio.gather(translation_task, keywords_task)
+    print(repr(translation))
+    print(repr(keywords))
+
+asyncio.run(main())
+```
+
+---
+
+## 👍 Use Cases
+
+Use **TextTools** when you need to:
+
+- 🔍 **Classify** large datasets quickly without model training  
+- 🌍 **Translate** and process multilingual corpora with ease  
+- 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
+- 📊 **Analyze** large text collections using embeddings and categorization  
+
+---
+
+## 📚 Batch Processing
+
+Process large datasets efficiently using OpenAI's batch API.
+
+## ⚡ Quick Start (Batch Runner)
+
+```python
+from pydantic import BaseModel
+from texttools import BatchRunner, BatchConfig
+
+config = BatchConfig(
+    system_prompt="Extract entities from the text",
+    job_name="entity_extraction",
+    input_data_path="data.json",
+    output_data_filename="results.json",
+    model="gpt-4o-mini"
+)
+
+class Output(BaseModel):
+    entities: list[str]
+
+runner = BatchRunner(config, output_model=Output)
+runner.run()
+```
+
+---
+
+## 🤝 Contributing
+
+Contributions are welcome!  
+Feel free to **open issues, suggest new features, or submit pull requests**.  
+
+---
+
+## 🌿 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

hamtaa_texttools-1.2.0.dist-info/RECORD

@@ -0,0 +1,34 @@
+hamtaa_texttools-1.2.0.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
+texttools/__init__.py,sha256=4z7wInlrgbGSlWlXHQNeZMCGQH1sN2xtARsbgLHOLd8,283
+texttools/models.py,sha256=5eT2cSrFq8Xa38kANznV7gbi7lwB2PoDxciLKTpsd6c,2516
+texttools/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/batch/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/batch/config.py,sha256=GDDXuhRZ_bOGVwSIlU4tWP247tx1_A7qzLJn7VqDyLU,1050
+texttools/batch/manager.py,sha256=XZtf8UkdClfQlnRKne4nWEcFvdSKE67EamEePKy7jwI,8730
+texttools/batch/runner.py,sha256=9qxXIMfYRXW5SXDqqKtRr61rnQdYZkbCGqKImhSrY6I,9923
+texttools/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/core/engine.py,sha256=WhEtxZjYEbCnD0gExGRR4eSmAY4J05E9csovt2Qqlm8,9281
+texttools/core/exceptions.py,sha256=6SDjUL1rmd3ngzD3ytF4LyTRj3bQMSFR9ECrLoqXXHw,395
+texttools/core/internal_models.py,sha256=aExdLvhXhSev8NY1kuAJckeXdFBEisQtKZPxybd3rW8,1703
+texttools/core/operators/async_operator.py,sha256=wFs7eZ9QJrL0jBOu00YffgfPnIrCSavNjecSorXh-mE,6452
+texttools/core/operators/sync_operator.py,sha256=NaUS-aLh3y0QNMiKut4qtcSZKYXbuPbw0o2jvPsYKdY,6357
+texttools/prompts/categorize.yaml,sha256=42Rp3SgVHaDLKrJ27_uK788LiQud0pOXJthz4r0a40Y,1214
+texttools/prompts/check_fact.yaml,sha256=zWFQDRhEE1ij9wSeeenS9YSTM-bY5zzUaG390zUgmcs,714
+texttools/prompts/extract_entities.yaml,sha256=_zYKHNJDIzVDI_-TnwFCKyMs-XLM5igvmWhvSTc3INQ,637
+texttools/prompts/extract_keywords.yaml,sha256=1o4u3uwzapNtB1BUpNIRL5qtrwjW0Yhvyq0TZJiafdg,3272
+texttools/prompts/is_question.yaml,sha256=jnPARd2ZiulLzHW_r4WAsz3sOryfz6Gy5-yYXp-2hd0,496
+texttools/prompts/merge_questions.yaml,sha256=l9Q2OEjPp3SDkxbq3zZCj2ZmXacWSnmYMpUr3l6r5yE,1816
+texttools/prompts/propositionize.yaml,sha256=nbGAfbm1-2Hoc0JLtqZi-S7VHQfnMmuTKI7dZeBxQW0,1403
+texttools/prompts/rewrite.yaml,sha256=klEm8MqXK-Bo8RsS5R9KLMT0zlD-BKo_G6tz9lpAcEY,5420
+texttools/prompts/run_custom.yaml,sha256=IETY9H0wPGWIIzcnupfbwwKQblwZrbYAxB754W9MhgU,125
+texttools/prompts/subject_to_question.yaml,sha256=AK16pZW9HUppIF8JBSEenbUNOU3aqeVV781_WUXnLqk,1160
+texttools/prompts/summarize.yaml,sha256=rPh060Bx_yI1W2JNg-nr83LUk9itatYLKM8ciH2pOvg,486
+texttools/prompts/text_to_question.yaml,sha256=pUwPgK9l5f8S4E5fCht9JY7PFVK2aY1InPfASr7R5o4,1017
+texttools/prompts/translate.yaml,sha256=Dd5bs3O8SI-FlVSwHMYGeEjMmdOWeRlcfBHkhixCx7c,665
+texttools/tools/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/tools/async_tools.py,sha256=DonASaaOPbWp7Gh1UY4RlP3yPoYTuhJtVmLns8KYupE,42949
+texttools/tools/sync_tools.py,sha256=y4nMlabgvRapb-YFoiGA5-5HflKrRHttiWSHpkg9tug,42742
+hamtaa_texttools-1.2.0.dist-info/METADATA,sha256=vN4XmIWdH6mdAGfgSkjRdQLEoFNzbhhH32jOyEv9H6w,7846
+hamtaa_texttools-1.2.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+hamtaa_texttools-1.2.0.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-1.2.0.dist-info/RECORD,,

texttools/__init__.py

@@ -1,7 +1,7 @@
-from .batch.batch_runner import BatchJobRunner
-from .batch.batch_config import BatchConfig
-from .tools.sync_tools import TheTool
+from .batch.config import BatchConfig
+from .batch.runner import BatchRunner
+from .models import CategoryTree
 from .tools.async_tools import AsyncTheTool
-from .tools.internals.models import CategoryTree
+from .tools.sync_tools import TheTool
 
-__all__ = ["TheTool", "AsyncTheTool", "BatchJobRunner", "BatchConfig", "CategoryTree"]
+__all__ = ["TheTool", "AsyncTheTool", "CategoryTree", "BatchRunner", "BatchConfig"]

texttools/batch/{batch_config.py → config.py}

@@ -1,7 +1,21 @@
-from dataclasses import dataclass
 from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any
+
 
-from texttools.batch.internals.utils import import_data, export_data
+def export_data(data) -> list[dict[str, str]]:
+    """
+    Produces a structure of the following form from an initial data structure:
+    [{"id": str, "text": str},...]
+    """
+    return data
+
+
+def import_data(data) -> Any:
+    """
+    Takes the output and adds and aggregates it to the original structure.
+    """
+    return data
 
 
 @dataclass

texttools/batch/{internals/batch_manager.py → manager.py}

@@ -1,12 +1,12 @@
 import json
+import logging
 import uuid
 from pathlib import Path
 from typing import Any, Type, TypeVar
-import logging
 
-from pydantic import BaseModel
 from openai import OpenAI
 from openai.lib._pydantic import to_strict_json_schema
+from pydantic import BaseModel
 
 # Base Model type for output models
 T = TypeVar("T", bound=BaseModel)

texttools/batch/{batch_runner.py → runner.py}

@@ -1,17 +1,18 @@
 import json
+import logging
 import os
 import time
 from pathlib import Path
 from typing import Any, Type, TypeVar
-import logging
 
 from dotenv import load_dotenv
 from openai import OpenAI
 from pydantic import BaseModel
 
-from texttools.batch.internals.batch_manager import BatchManager
-from texttools.batch.batch_config import BatchConfig
-from texttools.tools.internals.models import StrOutput
+from ..core.exceptions import TextToolsError
+from ..core.internal_models import Str
+from .config import BatchConfig
+from .manager import BatchManager
 
 # Base Model type for output models
 T = TypeVar("T", bound=BaseModel)
@@ -19,30 +20,34 @@ T = TypeVar("T", bound=BaseModel)
 logger = logging.getLogger("texttools.batch_runner")
 
 
-class BatchJobRunner:
+class BatchRunner:
     """
     Handles running batch jobs using a batch manager and configuration.
     """
 
     def __init__(
-        self, config: BatchConfig = BatchConfig(), output_model: Type[T] = StrOutput
+        self, config: BatchConfig = BatchConfig(), output_model: Type[T] = Str
     ):
-        self._config = config
-        self._system_prompt = config.system_prompt
-        self._job_name = config.job_name
-        self._input_data_path = config.input_data_path
-        self._output_data_filename = config.output_data_filename
-        self._model = config.model
-        self._output_model = output_model
-        self._manager = self._init_manager()
-        self._data = self._load_data()
-        self._parts: list[list[dict[str, Any]]] = []
-        # Map part index to job name
-        self._part_idx_to_job_name: dict[int, str] = {}
-        # Track retry attempts per part
-        self._part_attempts: dict[int, int] = {}
-        self._partition_data()
-        Path(self._config.BASE_OUTPUT_DIR).mkdir(parents=True, exist_ok=True)
+        try:
+            self._config = config
+            self._system_prompt = config.system_prompt
+            self._job_name = config.job_name
+            self._input_data_path = config.input_data_path
+            self._output_data_filename = config.output_data_filename
+            self._model = config.model
+            self._output_model = output_model
+            self._manager = self._init_manager()
+            self._data = self._load_data()
+            self._parts: list[list[dict[str, Any]]] = []
+            # Map part index to job name
+            self._part_idx_to_job_name: dict[int, str] = {}
+            # Track retry attempts per part
+            self._part_attempts: dict[int, int] = {}
+            self._partition_data()
+            Path(self._config.BASE_OUTPUT_DIR).mkdir(parents=True, exist_ok=True)
+
+        except Exception as e:
+            raise TextToolsError(f"Batch runner initialization failed: {e}")
 
     def _init_manager(self) -> BatchManager:
         load_dotenv()
@@ -162,56 +167,62 @@ class BatchJobRunner:
 
         Submits jobs, monitors progress, handles retries, and saves results.
         """
-        # Submit all jobs up-front for concurrent execution
-        self._submit_all_jobs()
-        pending_parts: set[int] = set(self._part_idx_to_job_name.keys())
-        logger.info(f"Pending parts: {sorted(pending_parts)}")
-        # Polling loop
-        while pending_parts:
-            finished_this_round: list[int] = []
-            for part_idx in list(pending_parts):
-                job_name = self._part_idx_to_job_name[part_idx]
-                status = self._manager.check_status(job_name=job_name)
-                logger.info(f"Status for {job_name}: {status}")
-                if status == "completed":
-                    logger.info(
-                        f"Job completed. Fetching results for part {part_idx + 1}..."
-                    )
-                    output_data, log = self._manager.fetch_results(
-                        job_name=job_name, remove_cache=False
-                    )
-                    output_data = self._config.import_function(output_data)
-                    self._save_results(output_data, log, part_idx)
-                    logger.info(f"Fetched and saved results for part {part_idx + 1}.")
-                    finished_this_round.append(part_idx)
-                elif status == "failed":
-                    attempt = self._part_attempts.get(part_idx, 0) + 1
-                    self._part_attempts[part_idx] = attempt
-                    if attempt <= self._config.max_retries:
+        try:
+            # Submit all jobs up-front for concurrent execution
+            self._submit_all_jobs()
+            pending_parts: set[int] = set(self._part_idx_to_job_name.keys())
+            logger.info(f"Pending parts: {sorted(pending_parts)}")
+            # Polling loop
+            while pending_parts:
+                finished_this_round: list[int] = []
+                for part_idx in list(pending_parts):
+                    job_name = self._part_idx_to_job_name[part_idx]
+                    status = self._manager.check_status(job_name=job_name)
+                    logger.info(f"Status for {job_name}: {status}")
+                    if status == "completed":
                         logger.info(
-                            f"Job {job_name} failed (attempt {attempt}). Retrying after short backoff..."
+                            f"Job completed. Fetching results for part {part_idx + 1}..."
                         )
-                        self._manager._clear_state(job_name)
-                        time.sleep(10)
-                        payload = self._to_manager_payload(self._parts[part_idx])
-                        new_job_name = (
-                            f"{self._job_name}_part_{part_idx + 1}_retry_{attempt}"
+                        output_data, log = self._manager.fetch_results(
+                            job_name=job_name, remove_cache=False
                         )
-                        self._manager.start(payload, job_name=new_job_name)
-                        self._part_idx_to_job_name[part_idx] = new_job_name
-                    else:
+                        output_data = self._config.import_function(output_data)
+                        self._save_results(output_data, log, part_idx)
                         logger.info(
-                            f"Job {job_name} failed after {attempt - 1} retries. Marking as failed."
+                            f"Fetched and saved results for part {part_idx + 1}."
                         )
                         finished_this_round.append(part_idx)
-                else:
-                    # Still running or queued
-                    continue
-            # Remove finished parts
-            for part_idx in finished_this_round:
-                pending_parts.discard(part_idx)
-            if pending_parts:
-                logger.info(
-                    f"Waiting {self._config.poll_interval_seconds}s before next status check for parts: {sorted(pending_parts)}"
-                )
-                time.sleep(self._config.poll_interval_seconds)
+                    elif status == "failed":
+                        attempt = self._part_attempts.get(part_idx, 0) + 1
+                        self._part_attempts[part_idx] = attempt
+                        if attempt <= self._config.max_retries:
+                            logger.info(
+                                f"Job {job_name} failed (attempt {attempt}). Retrying after short backoff..."
+                            )
+                            self._manager._clear_state(job_name)
+                            time.sleep(10)
+                            payload = self._to_manager_payload(self._parts[part_idx])
+                            new_job_name = (
+                                f"{self._job_name}_part_{part_idx + 1}_retry_{attempt}"
+                            )
+                            self._manager.start(payload, job_name=new_job_name)
+                            self._part_idx_to_job_name[part_idx] = new_job_name
+                        else:
+                            logger.info(
+                                f"Job {job_name} failed after {attempt - 1} retries. Marking as failed."
+                            )
+                            finished_this_round.append(part_idx)
+                    else:
+                        # Still running or queued
+                        continue
+                # Remove finished parts
+                for part_idx in finished_this_round:
+                    pending_parts.discard(part_idx)
+                if pending_parts:
+                    logger.info(
+                        f"Waiting {self._config.poll_interval_seconds}s before next status check for parts: {sorted(pending_parts)}"
+                    )
+                    time.sleep(self._config.poll_interval_seconds)
+
+        except Exception as e:
+            raise TextToolsError(f"Batch job execution failed: {e}")