hamtaa-texttools 1.1.1__py3-none-any.whl → 1.2.0__py3-none-any.whl

hamtaa_texttools-1.1.1.dist-info/METADATA

@@ -1,183 +0,0 @@
-Metadata-Version: 2.4
-Name: hamtaa-texttools
-Version: 1.1.1
-Summary: A high-level NLP toolkit built on top of modern LLMs.
-Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>
-License: MIT License
-        
-        Copyright (c) 2025 Hamtaa
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy
-        of this software and associated documentation files (the "Software"), to deal
-        in the Software without restriction, including without limitation the rights
-        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-        copies of the Software, and to permit persons to whom the Software is
-        furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all
-        copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-        SOFTWARE.
-Keywords: nlp,llm,text-processing,openai
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: openai==1.97.1
-Requires-Dist: pyyaml>=6.0
-Dynamic: license-file
-
-# TextTools
-
-## 📌 Overview
-
-**TextTools** is a high-level **NLP toolkit** built on top of modern **LLMs**.  
-
-It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
-
-It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extractor, and more** — designed to help you integrate AI-powered text processing into your applications with minimal effort.
-
----
-
-## ✨ Features
-
-TextTools provides a rich collection of high-level NLP utilities built on top of LLMs.  
-Each tool is designed to work out-of-the-box with structured outputs (JSON / Pydantic).
-
-- **`categorize()`** - Classifies text into Islamic studies categories 
-- **`is_question()`** - Binary detection of whether input is a question
-- **`extract_keywords()`** - Extracts keywords from text
-- **`extract_entities()`** - Named Entity Recognition (NER) system
-- **`summarize()`** - Text summarization
-- **`text_to_question()`** - Generates questions from text
-- **`merge_questions()`** - Merges multiple questions with different modes
-- **`rewrite()`** - Rewrites text with different wording/meaning
-- **`subject_to_question()`** - Generates questions about a specific subject
-- **`translate()`** - Text translation between languages
-- **`run_custom()`** - Allows users to define a custom tool with arbitrary BaseModel
-
----
-
-## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt` and `temperature` parameters
-
-TextTools provides several optional flags to customize LLM behavior:
-
-- **`with_analysis=True`** → Adds a reasoning step before generating the final output. Useful for debugging, improving prompts, or understanding model behavior.  
-Note: This doubles token usage per call because it triggers an additional LLM request.
-
-- **`logprobs=True`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
-
-- **`output_lang="en"`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
-
-- **`user_prompt="..."`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
-
-- **`temperature=0.0`** → Determines how creative the model should respond. Takes a float number from `0.0` to `1.0`.
-
-All these parameters can be used individually or together to tailor the behavior of any tool in **TextTools**.
-
-**Note:** There might be some tools that don't support some of the parameters above.
-
----
-
-## 🚀 Installation
-
-Install the latest release via PyPI:
-
-```bash
-pip install -U hamtaa-texttools
-```
-
----
-
-## Sync vs Async
-| Tool         | Style   | Use case                                    |
-|--------------|---------|---------------------------------------------|
-| `TheTool`    | Sync    | Simple scripts, sequential workflows        |
-| `AsyncTheTool` | Async | High-throughput apps, APIs, concurrent tasks |
-
----
-
-## ⚡ Quick Start (Sync)
-
-```python
-from openai import OpenAI
-from texttools import TheTool
-
-# Create your OpenAI client
-client = OpenAI(base_url = "your_url", API_KEY = "your_api_key")
-
-# Specify the model
-model = "gpt-4o-mini"
-
-# Create an instance of TheTool
-the_tool = TheTool(client=client, model=model)
-
-# Example: Question Detection
-detection = the_tool.is_question("Is this project open source?", logprobs=True, top_logprobs=2)
-print(detection.result)
-print(detection.logprobs)
-# Output: True \n --logprobs
-
-# Example: Translation
-translation = the_tool.translate("سلام، حالت چطوره؟" target_language="English", with_analysis=True)
-print(translation.result)
-print(translation.analysis)
-# Output: "Hi! How are you?" \n --analysis
-```
-
----
-
-## ⚡ Quick Start (Async)
-
-```python
-import asyncio
-from openai import AsyncOpenAI
-from texttools import AsyncTheTool
-
-async def main():
-    # Create your AsyncOpenAI client
-    async_client = AsyncOpenAI(base_url="your_url", api_key="your_api_key")
-
-    # Specify the model
-    model = "gpt-4o-mini"
-
-    # Create an instance of AsyncTheTool
-    the_tool = AsyncTheTool(client=async_client, model=model)
-
-    # Example: Async Translation
-    translation = await the_tool.translate("سلام، حالت چطوره؟", target_language="English")
-    print(translation.result)
-    # Output: "Hi! How are you?"
-
-asyncio.run(main())
-```
-
----
-
-## 📚 Use Cases
-
-Use **TextTools** when you need to:
-
-- 🔍 **Classify** large datasets quickly without model training  
-- 🌍 **Translate** and process multilingual corpora with ease  
-- 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
-- 📊 **Analyze** large text collections using embeddings and categorization  
-- 👍 **Automate** common text-processing tasks without reinventing the wheel  
-
----
-
-## 🤝 Contributing
-
-Contributions are welcome!  
-Feel free to **open issues, suggest new features, or submit pull requests**.  
-
----
-
-## License
-
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

hamtaa_texttools-1.1.1.dist-info/RECORD

@@ -1,30 +0,0 @@
-hamtaa_texttools-1.1.1.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
-texttools/__init__.py,sha256=v3tQCH_Cjj47fCpuhK6sKSVAqEjNkc-cZbY4OJa4IZw,202
-texttools/batch/__init__.py,sha256=q50JsQsmQGp_8RW0KNasYeYWVV0R4FUNZ-ujXwEJemY,143
-texttools/batch/batch_manager.py,sha256=leVIFkR-3HpDkQi_MK3TgFNnHYsCN-wbS4mTWoPmO3c,8828
-texttools/batch/batch_runner.py,sha256=cgiCYLIBQQC0dBWM8_lVP9c5QLJoAmS2ijMtp0p3U2o,10313
-texttools/prompts/README.md,sha256=rclMaCV1N8gT1KcpZu0-ka0dKGNg2f1CEcRMdQkgQOc,1379
-texttools/prompts/categorizer.yaml,sha256=GMqIIzQFhgnlpkgU1qi3FAD3mD4A2jiWD5TilQ2XnnE,1204
-texttools/prompts/extract_entities.yaml,sha256=KiKjeDpHaeh3JVtZ6q1pa3k4DYucUIU9WnEcRTCA-SE,651
-texttools/prompts/extract_keywords.yaml,sha256=0O7ypL_OsEOxtvlQ2CZjnsv9637DJwAKprZsf9Vo2_s,769
-texttools/prompts/is_question.yaml,sha256=d0-vKRbXWkxvO64ikvxRjEmpAXGpCYIPGhgexvPPjws,471
-texttools/prompts/merge_questions.yaml,sha256=0J85GvTirZB4ELwH3sk8ub_WcqqpYf6PrMKr3djlZeo,1792
-texttools/prompts/rewrite.yaml,sha256=LO7He_IA3MZKz8a-LxH9DHJpOjpYwaYN1pbjp1Y0tFo,5392
-texttools/prompts/run_custom.yaml,sha256=38OkCoVITbuuS9c08UZSP1jZW4WjSmRIi8fR0RAiPu4,108
-texttools/prompts/subject_to_question.yaml,sha256=C7x7rNNm6U_ZG9HOn6zuzYOtvJUZ2skuWbL1-aYdd3E,1147
-texttools/prompts/summarize.yaml,sha256=o6rxGPfWtZd61Duvm8NVvCJqfq73b-wAuMSKR6UYUqY,459
-texttools/prompts/text_to_question.yaml,sha256=UheKYpDn6iyKI8NxunHZtFpNyfCLZZe5cvkuXpurUJY,783
-texttools/prompts/translate.yaml,sha256=mGT2uBCei6uucWqVbs4silk-UV060v3G0jnt0P6sr50,634
-texttools/tools/__init__.py,sha256=hG1I28Q7BJ1Dbs95x6QMKXdsAlC5Eh_tqC-EbAibwiU,114
-texttools/tools/async_the_tool.py,sha256=h6-Zkedet-eRUrkV5fANNoh4WmoqhXU5wJEHpd8nyNU,14377
-texttools/tools/the_tool.py,sha256=lKy3_CKcWo2cBLQ7dDgvh7-oos7UOx1NYM26tcMhwaI,14143
-texttools/tools/internals/async_operator.py,sha256=Kj-DLBKcKbZPCJYn4lVo4Iiei11M04pwgWpIl8L69aM,6169
-texttools/tools/internals/base_operator.py,sha256=OWJe8ybA6qmmoc7ysYeB8ccHPneDlEtmFGH1jLWQCeY,3135
-texttools/tools/internals/formatters.py,sha256=tACNLP6PeoqaRpNudVxBaHA25zyWqWYPZQuYysIu88g,941
-texttools/tools/internals/operator.py,sha256=g1E1WkgnKRDgOs6fEFu0-gPCw1Bniwb4VI9Er3Op_gk,6063
-texttools/tools/internals/output_models.py,sha256=gbVbzBWeyHUVNsCBuawdgz9ZEzsC7wfygGgZJsAaexY,1662
-texttools/tools/internals/prompt_loader.py,sha256=rbitJD3e8vAdcooP1Yx6KnSI83g28ho-FegfZ1cJ4j4,1979
-hamtaa_texttools-1.1.1.dist-info/METADATA,sha256=Cc1Rq94QyXgJ8SNhsBgyUfhho3oywzGpx6y16s50b-Q,7144
-hamtaa_texttools-1.1.1.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-hamtaa_texttools-1.1.1.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
-hamtaa_texttools-1.1.1.dist-info/RECORD,,

texttools/batch/batch_runner.py

@@ -1,263 +0,0 @@
-import json
-import os
-import time
-from dataclasses import dataclass
-from pathlib import Path
-from typing import Any, Callable
-import logging
-
-from dotenv import load_dotenv
-from openai import OpenAI
-from pydantic import BaseModel
-
-from texttools.batch import SimpleBatchManager
-
-# Configure logger
-logger = logging.getLogger("batch_runner")
-logger.setLevel(logging.INFO)
-
-
-class OutputModel(BaseModel):
-    desired_output: str
-
-
-def export_data(data):
-    """
-    Produces a structure of the following form from an initial data structure:
-    [{"id": str, "text": str},...]
-    """
-    return data
-
-
-def import_data(data):
-    """
-    Takes the output and adds and aggregates it to the original structure.
-    """
-    return data
-
-
-@dataclass
-class BatchConfig:
-    """
-    Configuration for batch job runner.
-    """
-
-    system_prompt: str = ""
-    job_name: str = ""
-    input_data_path: str = ""
-    output_data_filename: str = ""
-    model: str = "gpt-4.1-mini"
-    MAX_BATCH_SIZE: int = 100
-    MAX_TOTAL_TOKENS: int = 2000000
-    CHARS_PER_TOKEN: float = 2.7
-    PROMPT_TOKEN_MULTIPLIER: int = 1000
-    BASE_OUTPUT_DIR: str = "Data/batch_entity_result"
-    import_function: Callable = import_data
-    export_function: Callable = export_data
-    poll_interval_seconds: int = 30
-    max_retries: int = 3
-
-
-class BatchJobRunner:
-    """
-    Handles running batch jobs using a batch manager and configuration.
-    """
-
-    def __init__(
-        self, config: BatchConfig = BatchConfig(), output_model: type = OutputModel
-    ):
-        self.config = config
-        self.system_prompt = config.system_prompt
-        self.job_name = config.job_name
-        self.input_data_path = config.input_data_path
-        self.output_data_filename = config.output_data_filename
-        self.model = config.model
-        self.output_model = output_model
-        self.manager = self._init_manager()
-        self.data = self._load_data()
-        self.parts: list[list[dict[str, Any]]] = []
-        self._partition_data()
-        Path(self.config.BASE_OUTPUT_DIR).mkdir(parents=True, exist_ok=True)
-        # Map part index to job name
-        self.part_idx_to_job_name: dict[int, str] = {}
-        # Track retry attempts per part
-        self.part_attempts: dict[int, int] = {}
-
-    def _init_manager(self) -> SimpleBatchManager:
-        load_dotenv()
-        api_key = os.getenv("OPENAI_API_KEY")
-        client = OpenAI(api_key=api_key)
-        return SimpleBatchManager(
-            client=client,
-            model=self.model,
-            prompt_template=self.system_prompt,
-            output_model=self.output_model,
-        )
-
-    def _load_data(self):
-        with open(self.input_data_path, "r", encoding="utf-8") as f:
-            data = json.load(f)
-        data = self.config.export_function(data)
-
-        # Ensure data is a list of dicts with 'id' and 'content' as strings
-        if not isinstance(data, list):
-            raise ValueError(
-                'Exported data must be a list in this form:  [ {"id": str, "content": str},...]'
-            )
-        for item in data:
-            if not (isinstance(item, dict) and "id" in item and "content" in item):
-                raise ValueError(
-                    "Each item must be a dict with 'id' and 'content' keys."
-                )
-            if not (isinstance(item["id"], str) and isinstance(item["content"], str)):
-                raise ValueError("'id' and 'content' must be strings.")
-        return data
-
-    def _partition_data(self):
-        total_length = sum(len(item["content"]) for item in self.data)
-        prompt_length = len(self.system_prompt)
-        total = total_length + (prompt_length * len(self.data))
-        calculation = total / self.config.CHARS_PER_TOKEN
-        logger.info(
-            f"Total chars: {total_length}, Prompt chars: {prompt_length}, Total: {total}, Tokens: {calculation}"
-        )
-        if calculation < self.config.MAX_TOTAL_TOKENS:
-            self.parts = [self.data]
-        else:
-            # Partition into chunks of MAX_BATCH_SIZE
-            self.parts = [
-                self.data[i : i + self.config.MAX_BATCH_SIZE]
-                for i in range(0, len(self.data), self.config.MAX_BATCH_SIZE)
-            ]
-        logger.info(f"Data split into {len(self.parts)} part(s)")
-
-    def _submit_all_jobs(self) -> None:
-        for idx, part in enumerate(self.parts):
-            if self._result_exists(idx):
-                logger.info(f"Skipping part {idx + 1}: result already exists.")
-                continue
-            part_job_name = (
-                f"{self.job_name}_part_{idx + 1}"
-                if len(self.parts) > 1
-                else self.job_name
-            )
-            # If a job with this name already exists, register and skip submitting
-            existing_job = self.manager._load_state(part_job_name)
-            if existing_job:
-                logger.info(
-                    f"Skipping part {idx + 1}: job already exists ({part_job_name})."
-                )
-                self.part_idx_to_job_name[idx] = part_job_name
-                self.part_attempts.setdefault(idx, 0)
-                continue
-
-            payload = part
-            logger.info(
-                f"Submitting job for part {idx + 1}/{len(self.parts)}: {part_job_name}"
-            )
-            self.manager.start(payload, job_name=part_job_name)
-            self.part_idx_to_job_name[idx] = part_job_name
-            self.part_attempts.setdefault(idx, 0)
-            # This is added for letting file get uploaded, before starting the next part.
-            logger.info("Uploading...")
-            time.sleep(30)
-
-    def run(self):
-        # Submit all jobs up-front for concurrent execution
-        self._submit_all_jobs()
-        pending_parts: set[int] = set(self.part_idx_to_job_name.keys())
-        logger.info(f"Pending parts: {sorted(pending_parts)}")
-        # Polling loop
-        while pending_parts:
-            finished_this_round: list[int] = []
-            for part_idx in list(pending_parts):
-                job_name = self.part_idx_to_job_name[part_idx]
-                status = self.manager.check_status(job_name=job_name)
-                logger.info(f"Status for {job_name}: {status}")
-                if status == "completed":
-                    logger.info(
-                        f"Job completed. Fetching results for part {part_idx + 1}..."
-                    )
-                    output_data, log = self.manager.fetch_results(
-                        job_name=job_name, remove_cache=False
-                    )
-                    output_data = self.config.import_function(output_data)
-                    self._save_results(output_data, log, part_idx)
-                    logger.info(f"Fetched and saved results for part {part_idx + 1}.")
-                    finished_this_round.append(part_idx)
-                elif status == "failed":
-                    attempt = self.part_attempts.get(part_idx, 0) + 1
-                    self.part_attempts[part_idx] = attempt
-                    if attempt <= self.config.max_retries:
-                        logger.info(
-                            f"Job {job_name} failed (attempt {attempt}). Retrying after short backoff..."
-                        )
-                        self.manager._clear_state(job_name)
-                        time.sleep(10)
-                        payload = self._to_manager_payload(self.parts[part_idx])
-                        new_job_name = (
-                            f"{self.job_name}_part_{part_idx + 1}_retry_{attempt}"
-                        )
-                        self.manager.start(payload, job_name=new_job_name)
-                        self.part_idx_to_job_name[part_idx] = new_job_name
-                    else:
-                        logger.info(
-                            f"Job {job_name} failed after {attempt - 1} retries. Marking as failed."
-                        )
-                        finished_this_round.append(part_idx)
-                else:
-                    # Still running or queued
-                    continue
-            # Remove finished parts
-            for part_idx in finished_this_round:
-                pending_parts.discard(part_idx)
-            if pending_parts:
-                logger.info(
-                    f"Waiting {self.config.poll_interval_seconds}s before next status check for parts: {sorted(pending_parts)}"
-                )
-                time.sleep(self.config.poll_interval_seconds)
-
-    def _save_results(
-        self,
-        output_data: list[dict[str, Any]] | dict[str, Any],
-        log: list[Any],
-        part_idx: int,
-    ):
-        part_suffix = f"_part_{part_idx + 1}" if len(self.parts) > 1 else ""
-        result_path = (
-            Path(self.config.BASE_OUTPUT_DIR)
-            / f"{Path(self.output_data_filename).stem}{part_suffix}.json"
-        )
-        if not output_data:
-            logger.info("No output data to save. Skipping this part.")
-            return
-        else:
-            with open(result_path, "w", encoding="utf-8") as f:
-                json.dump(output_data, f, ensure_ascii=False, indent=4)
-        if log:
-            log_path = (
-                Path(self.config.BASE_OUTPUT_DIR)
-                / f"{Path(self.output_data_filename).stem}{part_suffix}_log.json"
-            )
-            with open(log_path, "w", encoding="utf-8") as f:
-                json.dump(log, f, ensure_ascii=False, indent=4)
-
-    def _result_exists(self, part_idx: int) -> bool:
-        part_suffix = f"_part_{part_idx + 1}" if len(self.parts) > 1 else ""
-        result_path = (
-            Path(self.config.BASE_OUTPUT_DIR)
-            / f"{Path(self.output_data_filename).stem}{part_suffix}.json"
-        )
-        return result_path.exists()
-
-
-if __name__ == "__main__":
-    logger.info("=== Batch Job Runner ===")
-    config = BatchConfig(
-        system_prompt="",
-        job_name="job_name",
-        input_data_path="Data.json",
-        output_data_filename="output",
-    )
-    runner = BatchJobRunner(config)
-    runner.run()

texttools/prompts/README.md

@@ -1,35 +0,0 @@
-# Prompts
-
-## Overview
-This folder contains YAML files for all prompts used in the project. Each file represents a separate prompt template, which can be loaded by tools or scripts that require structured prompts for AI models.
-
----
-
-## Structure
-- **prompt_file.yaml**: Each YAML file represents a single prompt template.
-- **main_template**: The main instruction template for the model.
-- **analyze_template** (optional): A secondary reasoning template used before generating the final response.
-- **Modes** (optional): Some prompts may have multiple modes (e.g., `default`, `reason`) to allow different behaviors.
-
-### Example YAML Structure
-```yaml
-main_template:
-  default: |
-    Your main instructions here with placeholders like {input}.
-  reason: |
-    Optional reasoning instructions here.
-
-analyze_template:
-  default: |
-    Analyze and summarize the input.
-  reason: |
-    Optional detailed analysis template.
-```
-
----
-
-## Guidelines
-1. **Naming**: Use descriptive names for each YAML file corresponding to the tool or task it serves.
-2. **Placeholders**: Use `{input}` or other relevant placeholders to dynamically inject data.
-3. **Modes**: If using modes, ensure both `main_template` and `analyze_template` contain the corresponding keys.
-4. **Consistency**: Keep formatting consistent across files for easier parsing by scripts.

texttools/prompts/categorizer.yaml

@@ -1,28 +0,0 @@
-main_template: |
-  تو یک متخصص علوم دینی هستی
-  من یک متن به تو میدهم و تو باید
-  آن متن را در یکی از دسته بندی های زیر طبقه بندی کنی
-  دسته بندی ها:
-  "باورهای دینی",
-  "اخلاق اسلامی",
-  "احکام و فقه",
-  "تاریخ اسلام و شخصیت ها",
-  "منابع دینی",
-  "دین و جامعه/سیاست",
-  "عرفان و معنویت",
-  "هیچکدام",
-  فقط با این فرمت json پاسخ بده:
-  {{
-  	  "reason": "<دلیل انتخابت رو به صورت خلاصه بگو>",
-      "result": "<یکی از دسته بندی ها>"
-  }}
-  متنی که باید طبقه بندی کنی:
-  {input}
-
-analyze_template: |
-  ما میخواهیم متنی که داده می شود را طبقه بندی کنیم.
-  برای بهبود طبقه بندی، نیاز به آنالیز متن داریم.
-  متنی که داده می شود را آنالیز کن و ایده اصلی و آنالیزی کوتاه از آن را بنویس.
-  آنالیز باید بسیار خلاصه باشد
-  نهایتا 20 کلمه
-  {input}