hamtaa-texttools 1.1.16__py3-none-any.whl → 1.2.0__py3-none-any.whl

hamtaa_texttools-1.1.16.dist-info/METADATA

@@ -1,255 +0,0 @@
-Metadata-Version: 2.4
-Name: hamtaa-texttools
-Version: 1.1.16
-Summary: A high-level NLP toolkit built on top of modern LLMs.
-Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>
-License: MIT License
-        
-        Copyright (c) 2025 Hamtaa
-        
-        Permission is hereby granted, free of charge, to any person obtaining a copy
-        of this software and associated documentation files (the "Software"), to deal
-        in the Software without restriction, including without limitation the rights
-        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-        copies of the Software, and to permit persons to whom the Software is
-        furnished to do so, subject to the following conditions:
-        
-        The above copyright notice and this permission notice shall be included in all
-        copies or substantial portions of the Software.
-        
-        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-        SOFTWARE.
-Keywords: nlp,llm,text-processing,openai
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: openai==1.97.1
-Requires-Dist: pydantic>=2.0.0
-Requires-Dist: pyyaml>=6.0
-Dynamic: license-file
-
-# TextTools
-
-## 📌 Overview
-
-**TextTools** is a high-level **NLP toolkit** built on top of modern **LLMs**.  
-
-It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
-
-It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
-
----
-
-## ✨ Features
-
-TextTools provides a rich collection of high-level NLP utilities,
-Each tool is designed to work with structured outputs (JSON / Pydantic).
-
-- **`categorize()`** - Classifies text into given categories (You have to create a category tree)
-- **`extract_keywords()`** - Extracts keywords from text
-- **`extract_entities()`** - Named Entity Recognition (NER) system
-- **`is_question()`** - Binary detection of whether input is a question
-- **`text_to_question()`** - Generates questions from text
-- **`merge_questions()`** - Merges multiple questions with different modes
-- **`rewrite()`** - Rewrites text with different wording/meaning
-- **`subject_to_question()`** - Generates questions about a specific subject
-- **`summarize()`** - Text summarization
-- **`translate()`** - Text translation between languages
-- **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
-
----
-
-## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator` and `priority` parameters
-
-TextTools provides several optional flags to customize LLM behavior:
-
-- **`with_analysis (bool)`** → Adds a reasoning step before generating the final output.
-**Note:** This doubles token usage per call because it triggers an additional LLM request.
-
-- **`logprobs (bool)`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
-**Note:** This feature works if it's supported by the model.
-
-- **`output_lang (str)`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
-
-- **`user_prompt (str)`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
-
-- **`temperature (float)`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
-
-- **`validator (Callable)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a bool (True if there were no problem, False if the validation fails.) If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can specify `max_validation_retries=<N>` to change the number of retries.
-
-- **`priority (int)`** → Task execution priority level. Higher values = higher priority. Affects processing order in queues.
-**Note:** This feature works if it's supported by the model and vLLM.
-
-**Note:** There might be some tools that don't support some of the parameters above.
-
----
-
-## 🧩 ToolOutput
-
-Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
-- **`result: Any`** → The output of LLM
-- **`analysis: str`** → The reasoning step before generating the final output
-- **`logprobs: list`** → Token-level probabilities for the generated output 
-- **`process: str`** → The tool name which processed the input
-- **`processed_at: datetime`** → The process time
-- **`execution_time: float`** → The execution time (seconds)
-- **`errors: list[str]`** → Any error that have occured during calling LLM
-
-**Note:** You can use `repr(ToolOutput)` to see details of your ToolOutput.
-
----
-
-## 🚀 Installation
-
-Install the latest release via PyPI:
-
-```bash
-pip install -U hamtaa-texttools
-```
-
----
-
-## 🧨 Sync vs Async
-| Tool         | Style   | Use case                                    |
-|--------------|---------|---------------------------------------------|
-| `TheTool`    | Sync    | Simple scripts, sequential workflows        |
-| `AsyncTheTool` | Async | High-throughput apps, APIs, concurrent tasks |
-
----
-
-## ⚡ Quick Start (Sync)
-
-```python
-from openai import OpenAI
-from texttools import TheTool
-
-# Create your OpenAI client
-client = OpenAI(base_url = "your_url", API_KEY = "your_api_key")
-
-# Specify the model
-model = "gpt-4o-mini"
-
-# Create an instance of TheTool
-the_tool = TheTool(client=client, model=model)
-
-# Example: Question Detection
-detection = the_tool.is_question("Is this project open source?", logprobs=True, top_logprobs=2)
-print(detection.result)
-print(detection.logprobs)
-# Output: True + logprobs
-
-# Example: Translation
-translation = the_tool.translate("سلام، حالت چطوره؟" target_language="English", with_analysis=True)
-print(translation.result)
-print(translation.analysis)
-# Output: "Hi! How are you?"  + analysis
-```
-
----
-
-## ⚡ Quick Start (Async)
-
-```python
-import asyncio
-from openai import AsyncOpenAI
-from texttools import AsyncTheTool
-
-async def main():
-    # Create your AsyncOpenAI client
-    async_client = AsyncOpenAI(base_url="your_url", api_key="your_api_key")
-
-    # Specify the model
-    model = "gpt-4o-mini"
-
-    # Create an instance of AsyncTheTool
-    async_the_tool = AsyncTheTool(client=async_client, model=model)
-    
-    # Example: Async Translation and Keyword Extraction
-    translation_task = async_the_tool.translate("سلام، حالت چطوره؟", target_language="English")
-    keywords_task = async_the_tool.extract_keywords("Tomorrow, we will be dead by the car crash")
-
-    (translation, keywords) = await asyncio.gather(translation_task, keywords_task)
-    print(translation.result)
-    print(keywords.result)
-
-asyncio.run(main())
-```
-
----
-
-## 👍 Use Cases
-
-Use **TextTools** when you need to:
-
-- 🔍 **Classify** large datasets quickly without model training  
-- 🌍 **Translate** and process multilingual corpora with ease  
-- 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
-- 📊 **Analyze** large text collections using embeddings and categorization  
-
----
-
-## 🔍 Logging
-
-TextTools uses Python's standard `logging` module. The library's default logger level is `WARNING`, so if you want to modify it, follow instructions:
-
-
-```python
-import logging
-
-# Default: warnings and errors only
-logging.basicConfig(level=logging.WARNING)
-
-# Debug everything (verbose)
-logging.basicConfig(level=logging.DEBUG)
-
-# Complete silence
-logging.basicConfig(level=logging.CRITICAL)
-```
-
----
-
-## 📚 Batch Processing
-
-Process large datasets efficiently using OpenAI's batch API.
-
-## ⚡ Quick Start (Batch)
-
-```python
-from pydantic import BaseModel
-from texttools import BatchJobRunner, BatchConfig
-
-# Configure your batch job
-config = BatchConfig(
-    system_prompt="Extract entities from the text",
-    job_name="entity_extraction",
-    input_data_path="data.json",
-    output_data_filename="results.json",
-    model="gpt-4o-mini"
-)
-
-# Define your output schema
-class Output(BaseModel):
-    entities: list[str]
-
-# Run the batch job
-runner = BatchJobRunner(config, output_model=Output)
-runner.run()
-```
-
----
-
-## 🤝 Contributing
-
-Contributions are welcome!  
-Feel free to **open issues, suggest new features, or submit pull requests**.  
-
----
-
-## 🌿 License
-
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

hamtaa_texttools-1.1.16.dist-info/RECORD

@@ -1,31 +0,0 @@
-hamtaa_texttools-1.1.16.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
-texttools/__init__.py,sha256=dc81lXGWP29k7oVvq2BMoMotz6lgiwX4PO2jHHBe2S8,317
-texttools/batch/batch_config.py,sha256=m1UgILVKjNdWE6laNbfbG4vgi4o2fEegGZbeoam6pnY,749
-texttools/batch/batch_runner.py,sha256=9e4SPLlvLHHs3U7bHkuuMVw8TFNwsGUzRjkAMKN4_ik,9378
-texttools/batch/internals/batch_manager.py,sha256=UoBe76vmFG72qrSaGKDZf4HzkykFBkkkbL9TLfV8TuQ,8730
-texttools/batch/internals/utils.py,sha256=F1_7YlVFKhjUROAFX4m0SaP8KiZVZyHRMIIB87VUGQc,373
-texttools/prompts/README.md,sha256=-5YO93CN93QLifqZpUeUnCOCBbDiOTV-cFQeJ7Gg0I4,1377
-texttools/prompts/categorize.yaml,sha256=F7VezB25B_sT5yoC25ezODBddkuDD5lUHKetSpx9FKI,2743
-texttools/prompts/detect_entity.yaml,sha256=1rhMkJOjxSQcT4j_c5SRcIm77AUdeG-rUmeidb6VOFc,981
-texttools/prompts/extract_entities.yaml,sha256=KiKjeDpHaeh3JVtZ6q1pa3k4DYucUIU9WnEcRTCA-SE,651
-texttools/prompts/extract_keywords.yaml,sha256=Vj4Tt3vT6LtpOo_iBZPo9oWI50oVdPGXe5i8yDR8ex4,3177
-texttools/prompts/is_question.yaml,sha256=d0-vKRbXWkxvO64ikvxRjEmpAXGpCYIPGhgexvPPjws,471
-texttools/prompts/merge_questions.yaml,sha256=0J85GvTirZB4ELwH3sk8ub_WcqqpYf6PrMKr3djlZeo,1792
-texttools/prompts/rewrite.yaml,sha256=LO7He_IA3MZKz8a-LxH9DHJpOjpYwaYN1pbjp1Y0tFo,5392
-texttools/prompts/run_custom.yaml,sha256=38OkCoVITbuuS9c08UZSP1jZW4WjSmRIi8fR0RAiPu4,108
-texttools/prompts/subject_to_question.yaml,sha256=C7x7rNNm6U_ZG9HOn6zuzYOtvJUZ2skuWbL1-aYdd3E,1147
-texttools/prompts/summarize.yaml,sha256=o6rxGPfWtZd61Duvm8NVvCJqfq73b-wAuMSKR6UYUqY,459
-texttools/prompts/text_to_question.yaml,sha256=UheKYpDn6iyKI8NxunHZtFpNyfCLZZe5cvkuXpurUJY,783
-texttools/prompts/translate.yaml,sha256=mGT2uBCei6uucWqVbs4silk-UV060v3G0jnt0P6sr50,634
-texttools/tools/async_tools.py,sha256=vNAg0gxwUZPsMS4q8JCv7RlYymS8l_5FsFI5adEYT7w,34376
-texttools/tools/sync_tools.py,sha256=hFifFa9YatvSeGif2E_bIG006eMdIBr6SV9HsZ_dAlg,34187
-texttools/tools/internals/async_operator.py,sha256=1TMr8e1qbE9GSz8jl0q3MKdM8lIYE-1ZuSxHjYPqKHI,7198
-texttools/tools/internals/formatters.py,sha256=tACNLP6PeoqaRpNudVxBaHA25zyWqWYPZQuYysIu88g,941
-texttools/tools/internals/models.py,sha256=2QnvMiijuSqOqpCl026848rJy_pHNbRoDESlQvcdHlk,5839
-texttools/tools/internals/operator_utils.py,sha256=w1k0RJ_W_CRbVc_J2w337VuL-opHpHiCxfhEOwtyuOo,1856
-texttools/tools/internals/prompt_loader.py,sha256=4g6-U8kqrGN7VpNaRcrBcnF-h03PXjUDBP0lL0_4EZY,1953
-texttools/tools/internals/sync_operator.py,sha256=4-V__o55Q8w29lWxkhG4St-exZLZTfBbiW76knOXbc0,7106
-hamtaa_texttools-1.1.16.dist-info/METADATA,sha256=DL-cjlGMv7bft8QVd-pn5E_tNDuPgQHkTKGl4YTosGw,9555
-hamtaa_texttools-1.1.16.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
-hamtaa_texttools-1.1.16.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
-hamtaa_texttools-1.1.16.dist-info/RECORD,,

texttools/batch/internals/utils.py

@@ -1,16 +0,0 @@
-from typing import Any
-
-
-def export_data(data) -> list[dict[str, str]]:
-    """
-    Produces a structure of the following form from an initial data structure:
-    [{"id": str, "text": str},...]
-    """
-    return data
-
-
-def import_data(data) -> Any:
-    """
-    Takes the output and adds and aggregates it to the original structure.
-    """
-    return data

texttools/prompts/README.md

@@ -1,35 +0,0 @@
-# Prompts
-
-## Overview
-This folder contains YAML files for all prompts used in the project. Each file represents a separate prompt template, which can be loaded by tools or scripts that require structured prompts for AI models.
-
----
-
-## Structure
-- **prompt_file.yaml**: Each YAML file represents a single prompt template.
-- **main_template**: The main instruction template for the model.
-- **analyze_template** (optional): A secondary reasoning template used before generating the final response.
-- **Modes** (optional): Some prompts may have multiple modes (e.g., `default`, `reason`) to allow different behaviors.
-
-### Example YAML Structure
-```yaml
-main_template:
-  mode_1: |
-    Your main instructions here with placeholders like {input}.
-  mode_2: |
-    Optional reasoning instructions here.
-
-analyze_template:
-  mode_1: |
-    Analyze and summarize the input.
-  mode_2: |
-    Optional detailed analysis template.
-```
-
----
-
-## Guidelines
-1. **Naming**: Use descriptive names for each YAML file corresponding to the tool or task it serves.
-2. **Placeholders**: Use `{input}` or other relevant placeholders to dynamically inject data.
-3. **Modes**: If using modes, ensure both `main_template` and `analyze_template` contain the corresponding keys.
-4. **Consistency**: Keep formatting consistent across files for easier parsing by scripts.

texttools/prompts/detect_entity.yaml

@@ -1,22 +0,0 @@
-main_template: |
-  You are an expert Named Entity Recognition (NER) system. Extract entities from the text.
-  The output must strictly follow the provided Pydantic schema.
-  
-  Mapping Rule:
-  - Person: شخص
-  - Location: مکان
-  - Time: زمان
-  - Living Beings: موجود زنده
-  - Organization: سازمان
-  - Concept: مفهوم
-  
-  CRITICAL:
-  1. The final output structure must be a complete JSON object matching the Pydantic schema (List[Entity]).
-  2. Both the extracted text and the type must be in Persian, using the exact mapping provided above.
-  
-  Here is the text: {input}
-
-analyze_template: |
-  Analyze the following text to identify all potential named entities and their categories (Person, Location, Time, Living Beings, Organization, Concept). 
-  Provide a brief summary of the entities identified that will help the main process to extract them accurately and apply the correct Persian type label.
-  Here is the text: {input}

texttools/tools/internals/async_operator.py

@@ -1,200 +0,0 @@
-from typing import Any, TypeVar, Type
-from collections.abc import Callable
-import logging
-
-from openai import AsyncOpenAI
-from pydantic import BaseModel
-
-from texttools.tools.internals.models import ToolOutput
-from texttools.tools.internals.operator_utils import OperatorUtils
-from texttools.tools.internals.formatters import Formatter
-from texttools.tools.internals.prompt_loader import PromptLoader
-
-# Base Model type for output models
-T = TypeVar("T", bound=BaseModel)
-
-logger = logging.getLogger("texttools.async_operator")
-
-
-class AsyncOperator:
-    """
-    Core engine for running text-processing operations with an LLM (Async).
-
-    It wires together:
-    - `PromptLoader` → loads YAML prompt templates.
-    - `UserMergeFormatter` → applies formatting to messages (e.g., merging).
-    - AsyncOpenAI client → executes completions/parsed completions.
-    """
-
-    def __init__(self, client: AsyncOpenAI, model: str):
-        self._client = client
-        self._model = model
-
-    async def _analyze(self, prompt_configs: dict[str, str], temperature: float) -> str:
-        """
-        Calls OpenAI API for analysis using the configured prompt template.
-        Returns the analyzed content as a string.
-        """
-        analyze_prompt = prompt_configs["analyze_template"]
-        analyze_message = [OperatorUtils.build_user_message(analyze_prompt)]
-        completion = await self._client.chat.completions.create(
-            model=self._model,
-            messages=analyze_message,
-            temperature=temperature,
-        )
-        analysis = completion.choices[0].message.content.strip()
-        return analysis
-
-    async def _parse_completion(
-        self,
-        message: list[dict[str, str]],
-        output_model: Type[T],
-        temperature: float,
-        logprobs: bool = False,
-        top_logprobs: int = 3,
-        priority: int | None = 0,
-    ) -> tuple[T, Any]:
-        """
-        Parses a chat completion using OpenAI's structured output format.
-        Returns both the parsed object and the raw completion for logprobs.
-        """
-        request_kwargs = {
-            "model": self._model,
-            "messages": message,
-            "response_format": output_model,
-            "temperature": temperature,
-        }
-
-        if logprobs:
-            request_kwargs["logprobs"] = True
-            request_kwargs["top_logprobs"] = top_logprobs
-        if priority:
-            request_kwargs["extra_body"] = {"priority": priority}
-        completion = await self._client.beta.chat.completions.parse(**request_kwargs)
-        parsed = completion.choices[0].message.parsed
-        return parsed, completion
-
-    async def run(
-        self,
-        # User parameters
-        text: str,
-        with_analysis: bool,
-        output_lang: str | None,
-        user_prompt: str | None,
-        temperature: float,
-        logprobs: bool,
-        top_logprobs: int | None,
-        validator: Callable[[Any], bool] | None,
-        max_validation_retries: int | None,
-        # Internal parameters
-        prompt_file: str,
-        output_model: Type[T],
-        mode: str | None,
-        priority: int | None = 0,
-        **extra_kwargs,
-    ) -> ToolOutput:
-        """
-        Execute the async LLM pipeline with the given input text. (Async)
-        """
-        prompt_loader = PromptLoader()
-        formatter = Formatter()
-        output = ToolOutput()
-
-        try:
-            # Prompt configs contain two keys: main_template and analyze template, both are string
-            prompt_configs = prompt_loader.load(
-                prompt_file=prompt_file,
-                text=text.strip(),
-                mode=mode,
-                **extra_kwargs,
-            )
-
-            messages = []
-
-            if with_analysis:
-                analysis = await self._analyze(prompt_configs, temperature)
-                messages.append(
-                    OperatorUtils.build_user_message(
-                        f"Based on this analysis: {analysis}"
-                    )
-                )
-
-            if output_lang:
-                messages.append(
-                    OperatorUtils.build_user_message(
-                        f"Respond only in the {output_lang} language."
-                    )
-                )
-
-            if user_prompt:
-                messages.append(
-                    OperatorUtils.build_user_message(
-                        f"Consider this instruction {user_prompt}"
-                    )
-                )
-
-            messages.append(
-                OperatorUtils.build_user_message(prompt_configs["main_template"])
-            )
-
-            messages = formatter.user_merge_format(messages)
-
-            parsed, completion = await self._parse_completion(
-                messages, output_model, temperature, logprobs, top_logprobs, priority
-            )
-
-            output.result = parsed.result
-
-            # Retry logic if validation fails
-            if validator and not validator(output.result):
-                for attempt in range(max_validation_retries):
-                    logger.warning(
-                        f"Validation failed, retrying for the {attempt + 1} time."
-                    )
-
-                    # Generate new temperature for retry
-                    retry_temperature = OperatorUtils.get_retry_temp(temperature)
-                    try:
-                        parsed, completion = await self._parse_completion(
-                            messages,
-                            output_model,
-                            retry_temperature,
-                            logprobs,
-                            top_logprobs,
-                        )
-
-                        output.result = parsed.result
-
-                        # Check if retry was successful
-                        if validator(output.result):
-                            logger.info(
-                                f"Validation passed on retry attempt {attempt + 1}"
-                            )
-                            break
-                        else:
-                            logger.warning(
-                                f"Validation still failing after retry attempt {attempt + 1}"
-                            )
-
-                    except Exception as e:
-                        logger.error(f"Retry attempt {attempt + 1} failed: {e}")
-                        # Continue to next retry attempt if this one fails
-
-            # Final check after all retries
-            if validator and not validator(output.result):
-                output.errors.append("Validation failed after all retry attempts")
-
-            if logprobs:
-                output.logprobs = OperatorUtils.extract_logprobs(completion)
-
-            if with_analysis:
-                output.analysis = analysis
-
-            output.process = prompt_file[:-5]
-
-            return output
-
-        except Exception as e:
-            logger.error(f"AsyncTheTool failed: {e}")
-            output.errors.append(str(e))
-            return output

texttools/tools/internals/formatters.py

@@ -1,24 +0,0 @@
-class Formatter:
-    @staticmethod
-    def user_merge_format(messages: list[dict[str, str]]) -> list[dict[str, str]]:
-        """
-        Merges consecutive user messages into a single message, separated by newlines.
-
-        This is useful for condensing a multi-turn user input into a single
-        message for the LLM. Assistant and system messages are left unchanged and
-        act as separators between user message groups.
-        """
-        merged: list[dict[str, str]] = []
-
-        for message in messages:
-            role, content = message["role"], message["content"].strip()
-
-            # Merge with previous user turn
-            if merged and role == "user" and merged[-1]["role"] == "user":
-                merged[-1]["content"] += "\n" + content
-
-            # Otherwise, start a new turn
-            else:
-                merged.append({"role": role, "content": content})
-
-        return merged