hamtaa-texttools 1.1.1__py3-none-any.whl → 1.2.0__py3-none-any.whl

hamtaa_texttools-1.2.0.dist-info/METADATA

@@ -0,0 +1,212 @@
+Metadata-Version: 2.4
+Name: hamtaa-texttools
+Version: 1.2.0
+Summary: A high-level NLP toolkit built on top of modern LLMs.
+Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Erfan Moosavi <erfanmoosavi84@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
+Maintainer-email: Erfan Moosavi <erfanmoosavi84@gmail.com>, Tohidi <the.mohammad.tohidi@gmail.com>
+License: MIT
+Keywords: nlp,llm,text-processing,openai
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.97.1
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pyyaml>=6.0
+Dynamic: license-file
+
+# TextTools
+
+## 📌 Overview
+
+**TextTools** is a high-level **NLP toolkit** built on top of **LLMs**.  
+
+It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
+
+It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
+
+**Note:** Most features of `texttools` are reliable when you use `google/gemma-3n-e4b-it` model.
+
+---
+
+## ✨ Features
+
+TextTools provides a rich collection of high-level NLP utilities,
+Each tool is designed to work with structured outputs.
+
+- **`categorize()`** - Classifies text into given categories
+- **`extract_keywords()`** - Extracts keywords from the text
+- **`extract_entities()`** - Named Entity Recognition (NER) system
+- **`is_question()`** - Binary question detection
+- **`text_to_question()`** - Generates questions from text
+- **`merge_questions()`** - Merges multiple questions into one
+- **`rewrite()`** - Rewrites text in a diffrent way
+- **`subject_to_question()`** - Generates questions about a specific subject
+- **`summarize()`** - Text summarization
+- **`translate()`** - Text translation
+- **`propositionize()`** - Convert text to atomic independence meaningful sentences 
+- **`check_fact()`** - Check whether a statement is relevant to the source text
+- **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
+
+---
+
+## 🚀 Installation
+
+Install the latest release via PyPI:
+
+```bash
+pip install -U hamtaa-texttools
+```
+
+---
+
+## 📊 Tool Quality Tiers
+
+| Status | Meaning | Tools | Use in Production? |
+|--------|---------|----------|-------------------|
+| **✅ Production** | Evaluated, tested, stable. | `categorize()` (list mode), `extract_keywords()`, `extract_entities()`, `is_question()`, `text_to_question()`, `merge_questions()`, `rewrite()`, `subject_to_question()`, `summarize()`, `run_custom()` | **Yes** - ready for reliable use. |
+| **🧪 Experimental** | Added to the package but **not fully evaluated**. Functional, but quality may vary. | `categorize()` (tree mode), `translate()`, `propositionize()`, `check_fact()` | **Use with caution** - outputs not yet validated. |
+
+---
+
+## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator` and `priority` parameters
+
+TextTools provides several optional flags to customize LLM behavior:
+
+- **`with_analysis: bool`** → Adds a reasoning step before generating the final output.
+**Note:** This doubles token usage per call.
+
+- **`logprobs: bool`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
+**Note:** This feature works if it's supported by the model.
+
+- **`output_lang: str`** → Forces the model to respond in a specific language.
+
+- **`user_prompt: str`** → Allows you to inject a custom instruction or into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
+
+- **`temperature: float`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
+
+- **`validator: Callable (Experimental)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a boolean. If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can also specify `max_validation_retries=<N>`.
+
+- **`priority: int (Experimental)`** → Task execution priority level. Affects processing order in queues.
+**Note:** This feature works if it's supported by the model and vLLM.
+
+---
+
+## 🧩 ToolOutput
+
+Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
+- **`result: Any`**
+- **`analysis: str`**
+- **`logprobs: list`**
+- **`errors: list[str]`**
+- **`ToolOutputMetadata`** →
+    - **`tool_name: str`**
+    - **`processed_at: datetime`**
+    - **`execution_time: float`**
+
+**Note:** You can use `repr(ToolOutput)` to print your output with all the details.
+
+---
+
+## 🧨 Sync vs Async
+| Tool         | Style   | Use case                                    |
+|--------------|---------|---------------------------------------------|
+| `TheTool`    | Sync    | Simple scripts, sequential workflows        |
+| `AsyncTheTool` | Async | High-throughput apps, APIs, concurrent tasks |
+
+---
+
+## ⚡ Quick Start (Sync)
+
+```python
+from openai import OpenAI
+from texttools import TheTool
+
+client = OpenAI(base_url = "your_url", API_KEY = "your_api_key")
+model = "model_name"
+
+the_tool = TheTool(client=client, model=model)
+
+detection = the_tool.is_question("Is this project open source?")
+print(repr(detection))
+```
+
+---
+
+## ⚡ Quick Start (Async)
+
+```python
+import asyncio
+from openai import AsyncOpenAI
+from texttools import AsyncTheTool
+
+async def main():
+    async_client = AsyncOpenAI(base_url="your_url", api_key="your_api_key")
+    model = "model_name"
+
+    async_the_tool = AsyncTheTool(client=async_client, model=model)
+    
+    translation_task = async_the_tool.translate("سلام، حالت چطوره؟", target_language="English")
+    keywords_task = async_the_tool.extract_keywords("Tomorrow, we will be dead by the car crash")
+
+    (translation, keywords) = await asyncio.gather(translation_task, keywords_task)
+    print(repr(translation))
+    print(repr(keywords))
+
+asyncio.run(main())
+```
+
+---
+
+## 👍 Use Cases
+
+Use **TextTools** when you need to:
+
+- 🔍 **Classify** large datasets quickly without model training  
+- 🌍 **Translate** and process multilingual corpora with ease  
+- 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
+- 📊 **Analyze** large text collections using embeddings and categorization  
+
+---
+
+## 📚 Batch Processing
+
+Process large datasets efficiently using OpenAI's batch API.
+
+## ⚡ Quick Start (Batch Runner)
+
+```python
+from pydantic import BaseModel
+from texttools import BatchRunner, BatchConfig
+
+config = BatchConfig(
+    system_prompt="Extract entities from the text",
+    job_name="entity_extraction",
+    input_data_path="data.json",
+    output_data_filename="results.json",
+    model="gpt-4o-mini"
+)
+
+class Output(BaseModel):
+    entities: list[str]
+
+runner = BatchRunner(config, output_model=Output)
+runner.run()
+```
+
+---
+
+## 🤝 Contributing
+
+Contributions are welcome!  
+Feel free to **open issues, suggest new features, or submit pull requests**.  
+
+---
+
+## 🌿 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

hamtaa_texttools-1.2.0.dist-info/RECORD

@@ -0,0 +1,34 @@
+hamtaa_texttools-1.2.0.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
+texttools/__init__.py,sha256=4z7wInlrgbGSlWlXHQNeZMCGQH1sN2xtARsbgLHOLd8,283
+texttools/models.py,sha256=5eT2cSrFq8Xa38kANznV7gbi7lwB2PoDxciLKTpsd6c,2516
+texttools/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/batch/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/batch/config.py,sha256=GDDXuhRZ_bOGVwSIlU4tWP247tx1_A7qzLJn7VqDyLU,1050
+texttools/batch/manager.py,sha256=XZtf8UkdClfQlnRKne4nWEcFvdSKE67EamEePKy7jwI,8730
+texttools/batch/runner.py,sha256=9qxXIMfYRXW5SXDqqKtRr61rnQdYZkbCGqKImhSrY6I,9923
+texttools/core/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/core/engine.py,sha256=WhEtxZjYEbCnD0gExGRR4eSmAY4J05E9csovt2Qqlm8,9281
+texttools/core/exceptions.py,sha256=6SDjUL1rmd3ngzD3ytF4LyTRj3bQMSFR9ECrLoqXXHw,395
+texttools/core/internal_models.py,sha256=aExdLvhXhSev8NY1kuAJckeXdFBEisQtKZPxybd3rW8,1703
+texttools/core/operators/async_operator.py,sha256=wFs7eZ9QJrL0jBOu00YffgfPnIrCSavNjecSorXh-mE,6452
+texttools/core/operators/sync_operator.py,sha256=NaUS-aLh3y0QNMiKut4qtcSZKYXbuPbw0o2jvPsYKdY,6357
+texttools/prompts/categorize.yaml,sha256=42Rp3SgVHaDLKrJ27_uK788LiQud0pOXJthz4r0a40Y,1214
+texttools/prompts/check_fact.yaml,sha256=zWFQDRhEE1ij9wSeeenS9YSTM-bY5zzUaG390zUgmcs,714
+texttools/prompts/extract_entities.yaml,sha256=_zYKHNJDIzVDI_-TnwFCKyMs-XLM5igvmWhvSTc3INQ,637
+texttools/prompts/extract_keywords.yaml,sha256=1o4u3uwzapNtB1BUpNIRL5qtrwjW0Yhvyq0TZJiafdg,3272
+texttools/prompts/is_question.yaml,sha256=jnPARd2ZiulLzHW_r4WAsz3sOryfz6Gy5-yYXp-2hd0,496
+texttools/prompts/merge_questions.yaml,sha256=l9Q2OEjPp3SDkxbq3zZCj2ZmXacWSnmYMpUr3l6r5yE,1816
+texttools/prompts/propositionize.yaml,sha256=nbGAfbm1-2Hoc0JLtqZi-S7VHQfnMmuTKI7dZeBxQW0,1403
+texttools/prompts/rewrite.yaml,sha256=klEm8MqXK-Bo8RsS5R9KLMT0zlD-BKo_G6tz9lpAcEY,5420
+texttools/prompts/run_custom.yaml,sha256=IETY9H0wPGWIIzcnupfbwwKQblwZrbYAxB754W9MhgU,125
+texttools/prompts/subject_to_question.yaml,sha256=AK16pZW9HUppIF8JBSEenbUNOU3aqeVV781_WUXnLqk,1160
+texttools/prompts/summarize.yaml,sha256=rPh060Bx_yI1W2JNg-nr83LUk9itatYLKM8ciH2pOvg,486
+texttools/prompts/text_to_question.yaml,sha256=pUwPgK9l5f8S4E5fCht9JY7PFVK2aY1InPfASr7R5o4,1017
+texttools/prompts/translate.yaml,sha256=Dd5bs3O8SI-FlVSwHMYGeEjMmdOWeRlcfBHkhixCx7c,665
+texttools/tools/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+texttools/tools/async_tools.py,sha256=DonASaaOPbWp7Gh1UY4RlP3yPoYTuhJtVmLns8KYupE,42949
+texttools/tools/sync_tools.py,sha256=y4nMlabgvRapb-YFoiGA5-5HflKrRHttiWSHpkg9tug,42742
+hamtaa_texttools-1.2.0.dist-info/METADATA,sha256=vN4XmIWdH6mdAGfgSkjRdQLEoFNzbhhH32jOyEv9H6w,7846
+hamtaa_texttools-1.2.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+hamtaa_texttools-1.2.0.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-1.2.0.dist-info/RECORD,,

texttools/__init__.py

@@ -1,9 +1,7 @@
-from .batch import BatchJobRunner, SimpleBatchManager
-from .tools import AsyncTheTool, TheTool
+from .batch.config import BatchConfig
+from .batch.runner import BatchRunner
+from .models import CategoryTree
+from .tools.async_tools import AsyncTheTool
+from .tools.sync_tools import TheTool
 
-__all__ = [
-    "TheTool",
-    "AsyncTheTool",
-    "SimpleBatchManager",
-    "BatchJobRunner",
-]
+__all__ = ["TheTool", "AsyncTheTool", "CategoryTree", "BatchRunner", "BatchConfig"]

texttools/batch/__init__.py

@@ -1,4 +0,0 @@
-from .batch_manager import SimpleBatchManager
-from .batch_runner import BatchJobRunner
-
-__all__ = ["SimpleBatchManager", "BatchJobRunner"]

texttools/batch/config.py

@@ -0,0 +1,40 @@
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any
+
+
+def export_data(data) -> list[dict[str, str]]:
+    """
+    Produces a structure of the following form from an initial data structure:
+    [{"id": str, "text": str},...]
+    """
+    return data
+
+
+def import_data(data) -> Any:
+    """
+    Takes the output and adds and aggregates it to the original structure.
+    """
+    return data
+
+
+@dataclass
+class BatchConfig:
+    """
+    Configuration for batch job runner.
+    """
+
+    system_prompt: str = ""
+    job_name: str = ""
+    input_data_path: str = ""
+    output_data_filename: str = ""
+    model: str = "gpt-4.1-mini"
+    MAX_BATCH_SIZE: int = 100
+    MAX_TOTAL_TOKENS: int = 2_000_000
+    CHARS_PER_TOKEN: float = 2.7
+    PROMPT_TOKEN_MULTIPLIER: int = 1_000
+    BASE_OUTPUT_DIR: str = "Data/batch_entity_result"
+    import_function: Callable = import_data
+    export_function: Callable = export_data
+    poll_interval_seconds: int = 30
+    max_retries: int = 3

texttools/batch/{batch_manager.py → manager.py}

@@ -1,19 +1,20 @@
 import json
+import logging
 import uuid
 from pathlib import Path
-from typing import Any, Type
-import logging
+from typing import Any, Type, TypeVar
 
-from pydantic import BaseModel
 from openai import OpenAI
 from openai.lib._pydantic import to_strict_json_schema
+from pydantic import BaseModel
+
+# Base Model type for output models
+T = TypeVar("T", bound=BaseModel)
 
-# Configure logger
-logger = logging.getLogger("batch_runner")
-logger.setLevel(logging.INFO)
+logger = logging.getLogger("texttools.batch_manager")
 
 
-class SimpleBatchManager:
+class BatchManager:
     """
     Manages batch processing jobs for OpenAI's chat completions with structured outputs.
 
@@ -26,30 +27,29 @@ class SimpleBatchManager:
         self,
         client: OpenAI,
         model: str,
-        output_model: Type[BaseModel],
+        output_model: Type[T],
         prompt_template: str,
-        handlers: list[Any] | None = None,
         state_dir: Path = Path(".batch_jobs"),
         custom_json_schema_obj_str: dict | None = None,
         **client_kwargs: Any,
     ):
-        self.client = client
-        self.model = model
-        self.output_model = output_model
-        self.prompt_template = prompt_template
-        self.handlers = handlers or []
-        self.state_dir = state_dir
-        self.state_dir.mkdir(parents=True, exist_ok=True)
-        self.custom_json_schema_obj_str = custom_json_schema_obj_str
-        self.client_kwargs = client_kwargs
-        self.dict_input = False
-
-        if self.custom_json_schema_obj_str:
-            if self.custom_json_schema_obj_str is not dict:
-                raise ValueError("schema should be a dict")
+        self._client = client
+        self._model = model
+        self._output_model = output_model
+        self._prompt_template = prompt_template
+        self._state_dir = state_dir
+        self._custom_json_schema_obj_str = custom_json_schema_obj_str
+        self._client_kwargs = client_kwargs
+        self._dict_input = False
+        self._state_dir.mkdir(parents=True, exist_ok=True)
+
+        if custom_json_schema_obj_str and not isinstance(
+            custom_json_schema_obj_str, dict
+        ):
+            raise ValueError("Schema should be a dict")
 
     def _state_file(self, job_name: str) -> Path:
-        return self.state_dir / f"{job_name}.json"
+        return self._state_dir / f"{job_name}.json"
 
     def _load_state(self, job_name: str) -> list[dict[str, Any]]:
         """
@@ -83,17 +83,17 @@ class SimpleBatchManager:
         """
         response_format_config: dict[str, Any]
 
-        if self.custom_json_schema_obj_str:
+        if self._custom_json_schema_obj_str:
             response_format_config = {
                 "type": "json_schema",
-                "json_schema": self.custom_json_schema_obj_str,
+                "json_schema": self._custom_json_schema_obj_str,
             }
         else:
-            raw_schema = to_strict_json_schema(self.output_model)
+            raw_schema = to_strict_json_schema(self._output_model)
             response_format_config = {
                 "type": "json_schema",
                 "json_schema": {
-                    "name": self.output_model.__name__,
+                    "name": self._output_model.__name__,
                     "schema": raw_schema,
                 },
             }
@@ -105,11 +105,11 @@ class SimpleBatchManager:
             "body": {
                 "model": self.model,
                 "messages": [
-                    {"role": "system", "content": self.prompt_template},
+                    {"role": "system", "content": self._prompt_template},
                     {"role": "user", "content": text},
                 ],
                 "response_format": response_format_config,
-                **self.client_kwargs,
+                **self._client_kwargs,
             },
         }
 
@@ -127,10 +127,10 @@ class SimpleBatchManager:
 
         else:
             raise TypeError(
-                "The input must be either a list of texts or a dictionary in the form {'id': str, 'text': str}."
+                "The input must be either a list of texts or a dictionary in the form {'id': str, 'text': str}"
             )
 
-        file_path = self.state_dir / f"batch_{uuid.uuid4().hex}.jsonl"
+        file_path = self._state_dir / f"batch_{uuid.uuid4().hex}.jsonl"
         with open(file_path, "w", encoding="utf-8") as f:
             for task in tasks:
                 f.write(json.dumps(task) + "\n")
@@ -143,9 +143,10 @@ class SimpleBatchManager:
         """
         if self._load_state(job_name):
             return
+
         path = self._prepare_file(payload)
-        upload = self.client.files.create(file=open(path, "rb"), purpose="batch")
-        job = self.client.batches.create(
+        upload = self._client.files.create(file=open(path, "rb"), purpose="batch")
+        job = self._client.batches.create(
             input_file_id=upload.id,
             endpoint="/v1/chat/completions",
             completion_window="24h",
@@ -161,7 +162,7 @@ class SimpleBatchManager:
         if not job:
             return "completed"
 
-        info = self.client.batches.retrieve(job["id"])
+        info = self._client.batches.retrieve(job["id"])
         job = info.to_dict()
         self._save_state(job_name, [job])
         logger.info("Batch job status: %s", job)
@@ -179,18 +180,18 @@ class SimpleBatchManager:
             return {}
         batch_id = job["id"]
 
-        info = self.client.batches.retrieve(batch_id)
+        info = self._client.batches.retrieve(batch_id)
         out_file_id = info.output_file_id
         if not out_file_id:
             error_file_id = info.error_file_id
             if error_file_id:
                 err_content = (
-                    self.client.files.content(error_file_id).read().decode("utf-8")
+                    self._client.files.content(error_file_id).read().decode("utf-8")
                 )
-                logger.info("Error file content:", err_content)
+                logger.error("Error file content:", err_content)
             return {}
 
-        content = self.client.files.content(out_file_id).read().decode("utf-8")
+        content = self._client.files.content(out_file_id).read().decode("utf-8")
         lines = content.splitlines()
         results = {}
         log = []
@@ -201,7 +202,7 @@ class SimpleBatchManager:
                 content = result["response"]["body"]["choices"][0]["message"]["content"]
                 try:
                     parsed_content = json.loads(content)
-                    model_instance = self.output_model(**parsed_content)
+                    model_instance = self._output_model(**parsed_content)
                     results[custom_id] = model_instance.model_dump(mode="json")
                 except json.JSONDecodeError:
                     results[custom_id] = {"error": "Failed to parse content as JSON"}
@@ -221,8 +222,6 @@ class SimpleBatchManager:
                 error_d = {custom_id: results[custom_id]}
                 log.append(error_d)
 
-        for handler in self.handlers:
-            handler.handle(results)
         if remove_cache:
             self._clear_state(job_name)