hamtaa-texttools 1.1.20__tar.gz → 1.1.21__tar.gz

{hamtaa_texttools-1.1.20/hamtaa_texttools.egg-info → hamtaa_texttools-1.1.21}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 1.1.20
+Version: 1.1.21
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
 License: MIT License
@@ -50,7 +50,7 @@ It provides ready-to-use utilities for **translation, question detection, keywor
 TextTools provides a rich collection of high-level NLP utilities,
 Each tool is designed to work with structured outputs (JSON / Pydantic).
 
-- **`categorize()`** - Classifies text into given categories (You have to create a category tree)
+- **`categorize()`** - Classifies text into given categories
 - **`extract_keywords()`** - Extracts keywords from text
 - **`extract_entities()`** - Named Entity Recognition (NER) system
 - **`is_question()`** - Binary detection of whether input is a question
@@ -61,7 +61,7 @@ Each tool is designed to work with structured outputs (JSON / Pydantic).
 - **`summarize()`** - Text summarization
 - **`translate()`** - Text translation between languages
 - **`propositionize()`** - Convert text to atomic independence meaningful sentences 
-- **`check_fact()`** - Check a statement is relevant to source text or not 
+- **`check_fact()`** - Check whether a statement is relevant to the source text
 - **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
 
 ---
@@ -125,11 +125,12 @@ TextTools provides several optional flags to customize LLM behavior:
 Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
 - **`result: Any`** → The output of LLM
 - **`analysis: str`** → The reasoning step before generating the final output
-- **`logprobs: list`** → Token-level probabilities for the generated output 
-- **`process: str`** → The tool name which processed the input
-- **`processed_at: datetime`** → The process time
-- **`execution_time: float`** → The execution time (seconds)
+- **`logprobs: list`** → Token-level probabilities for the generated output
 - **`errors: list[str]`** → Any error that have occured during calling LLM
+- **`ToolOutputMetadata`** →
+    - **`tool_name: str`** → The tool name which processed the input
+    - **`processed_at: datetime`** → The process time
+    - **`execution_time: float`** → The execution time (seconds)
 
 **Note:** You can use `repr(ToolOutput)` to see details of your ToolOutput.
 
@@ -224,26 +225,6 @@ Use **TextTools** when you need to:
 
 ---
 
-## 🔍 Logging
-
-TextTools uses Python's standard `logging` module. The library's default logger level is `WARNING`, so if you want to modify it, follow instructions:
-
-
-```python
-import logging
-
-# Default: warnings and errors only
-logging.basicConfig(level=logging.WARNING)
-
-# Debug everything (verbose)
-logging.basicConfig(level=logging.DEBUG)
-
-# Complete silence
-logging.basicConfig(level=logging.CRITICAL)
-```
-
----
-
 ## 📚 Batch Processing
 
 Process large datasets efficiently using OpenAI's batch API.

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21}/README.md

@@ -15,7 +15,7 @@ It provides ready-to-use utilities for **translation, question detection, keywor
 TextTools provides a rich collection of high-level NLP utilities,
 Each tool is designed to work with structured outputs (JSON / Pydantic).
 
-- **`categorize()`** - Classifies text into given categories (You have to create a category tree)
+- **`categorize()`** - Classifies text into given categories
 - **`extract_keywords()`** - Extracts keywords from text
 - **`extract_entities()`** - Named Entity Recognition (NER) system
 - **`is_question()`** - Binary detection of whether input is a question
@@ -26,7 +26,7 @@ Each tool is designed to work with structured outputs (JSON / Pydantic).
 - **`summarize()`** - Text summarization
 - **`translate()`** - Text translation between languages
 - **`propositionize()`** - Convert text to atomic independence meaningful sentences 
-- **`check_fact()`** - Check a statement is relevant to source text or not 
+- **`check_fact()`** - Check whether a statement is relevant to the source text
 - **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
 
 ---
@@ -90,11 +90,12 @@ TextTools provides several optional flags to customize LLM behavior:
 Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
 - **`result: Any`** → The output of LLM
 - **`analysis: str`** → The reasoning step before generating the final output
-- **`logprobs: list`** → Token-level probabilities for the generated output 
-- **`process: str`** → The tool name which processed the input
-- **`processed_at: datetime`** → The process time
-- **`execution_time: float`** → The execution time (seconds)
+- **`logprobs: list`** → Token-level probabilities for the generated output
 - **`errors: list[str]`** → Any error that have occured during calling LLM
+- **`ToolOutputMetadata`** →
+    - **`tool_name: str`** → The tool name which processed the input
+    - **`processed_at: datetime`** → The process time
+    - **`execution_time: float`** → The execution time (seconds)
 
 **Note:** You can use `repr(ToolOutput)` to see details of your ToolOutput.
 
@@ -189,26 +190,6 @@ Use **TextTools** when you need to:
 
 ---
 
-## 🔍 Logging
-
-TextTools uses Python's standard `logging` module. The library's default logger level is `WARNING`, so if you want to modify it, follow instructions:
-
-
-```python
-import logging
-
-# Default: warnings and errors only
-logging.basicConfig(level=logging.WARNING)
-
-# Debug everything (verbose)
-logging.basicConfig(level=logging.DEBUG)
-
-# Complete silence
-logging.basicConfig(level=logging.CRITICAL)
-```
-
----
-
 ## 📚 Batch Processing
 
 Process large datasets efficiently using OpenAI's batch API.

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21/hamtaa_texttools.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 1.1.20
+Version: 1.1.21
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>, Zareshahi <a.zareshahi1377@gmail.com>
 License: MIT License
@@ -50,7 +50,7 @@ It provides ready-to-use utilities for **translation, question detection, keywor
 TextTools provides a rich collection of high-level NLP utilities,
 Each tool is designed to work with structured outputs (JSON / Pydantic).
 
-- **`categorize()`** - Classifies text into given categories (You have to create a category tree)
+- **`categorize()`** - Classifies text into given categories
 - **`extract_keywords()`** - Extracts keywords from text
 - **`extract_entities()`** - Named Entity Recognition (NER) system
 - **`is_question()`** - Binary detection of whether input is a question
@@ -61,7 +61,7 @@ Each tool is designed to work with structured outputs (JSON / Pydantic).
 - **`summarize()`** - Text summarization
 - **`translate()`** - Text translation between languages
 - **`propositionize()`** - Convert text to atomic independence meaningful sentences 
-- **`check_fact()`** - Check a statement is relevant to source text or not 
+- **`check_fact()`** - Check whether a statement is relevant to the source text
 - **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
 
 ---
@@ -125,11 +125,12 @@ TextTools provides several optional flags to customize LLM behavior:
 Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
 - **`result: Any`** → The output of LLM
 - **`analysis: str`** → The reasoning step before generating the final output
-- **`logprobs: list`** → Token-level probabilities for the generated output 
-- **`process: str`** → The tool name which processed the input
-- **`processed_at: datetime`** → The process time
-- **`execution_time: float`** → The execution time (seconds)
+- **`logprobs: list`** → Token-level probabilities for the generated output
 - **`errors: list[str]`** → Any error that have occured during calling LLM
+- **`ToolOutputMetadata`** →
+    - **`tool_name: str`** → The tool name which processed the input
+    - **`processed_at: datetime`** → The process time
+    - **`execution_time: float`** → The execution time (seconds)
 
 **Note:** You can use `repr(ToolOutput)` to see details of your ToolOutput.
 
@@ -224,26 +225,6 @@ Use **TextTools** when you need to:
 
 ---
 
-## 🔍 Logging
-
-TextTools uses Python's standard `logging` module. The library's default logger level is `WARNING`, so if you want to modify it, follow instructions:
-
-
-```python
-import logging
-
-# Default: warnings and errors only
-logging.basicConfig(level=logging.WARNING)
-
-# Debug everything (verbose)
-logging.basicConfig(level=logging.DEBUG)
-
-# Complete silence
-logging.basicConfig(level=logging.CRITICAL)
-```
-
----
-
 ## 📚 Batch Processing
 
 Process large datasets efficiently using OpenAI's batch API.

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21}/hamtaa_texttools.egg-info/SOURCES.txt

@@ -12,9 +12,8 @@ tests/test_all_tools.py
 tests/test_output_validation.py
 texttools/__init__.py
 texttools/batch/batch_config.py
+texttools/batch/batch_manager.py
 texttools/batch/batch_runner.py
-texttools/batch/internals/batch_manager.py
-texttools/batch/internals/utils.py
 texttools/internals/async_operator.py
 texttools/internals/exceptions.py
 texttools/internals/models.py

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "hamtaa-texttools"
-version = "1.1.20"
+version = "1.1.21"
 authors = [
   { name = "Tohidi", email = "the.mohammad.tohidi@gmail.com" },
   { name = "Montazer", email = "montazerh82@gmail.com" },

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21}/tests/test_all_tools.py

@@ -28,19 +28,20 @@ category = t.categorize(
 print(repr(category))
 
 # Categorizer: tree mode
-tree = CategoryTree("category_test_tree")
-tree.add_node("اخلاق")
-tree.add_node("معرفت شناسی")
-tree.add_node("متافیزیک", description="اراده قدرت در حیطه متافیزیک است")
-tree.add_node("فلسفه ذهن", description="فلسفه ذهن به چگونگی درک ما از جهان می پردازد")
-tree.add_node("آگاهی", "فلسفه ذهن", description="آگاهی خیلی مهم است")
+tree = CategoryTree()
+tree.add_node("اخلاق", "root")
+tree.add_node("معرفت شناسی", "root")
+tree.add_node("متافیزیک", "root", description="اراده قدرت در حیطه متافیزیک است")
+tree.add_node(
+    "فلسفه ذهن", "root", description="فلسفه ذهن به چگونگی درک ما از جهان می پردازد"
+)
+tree.add_node("آگاهی", "فلسفه ذهن")
 tree.add_node("ذهن و بدن", "فلسفه ذهن")
 tree.add_node("امکان و ضرورت", "متافیزیک")
 
 categories = t.categorize(
     "اراده قدرت مفهومی مهم در مابعد الطبیعه است که توسط نیچه مطرح شده",
     tree,
-    mode="category_tree",
 )
 print(repr(categories))
 
@@ -51,7 +52,12 @@ keywords = t.extract_keywords(
 print(repr(keywords))
 
 # NER Extractor
-entities = t.extract_entities("Ali will be dead by the car crash", entities=["EVENT"])
+entities = t.extract_entities(
+    "Ali will be dead by the car crash",
+    entities=["EVENT"],
+    with_analysis=True,
+    logprobs=True,
+)
 print(repr(entities))
 
 

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21}/texttools/batch/batch_config.py

@@ -1,7 +1,20 @@
 from dataclasses import dataclass
 from collections.abc import Callable
 
-from texttools.batch.internals.utils import import_data, export_data
+
+def export_data(data) -> list[dict[str, str]]:
+    """
+    Produces a structure of the following form from an initial data structure:
+    [{"id": str, "text": str},...]
+    """
+    return data
+
+
+def import_data(data) -> object:
+    """
+    Takes the output and adds and aggregates it to the original structure.
+    """
+    return data
 
 
 @dataclass

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21}/texttools/batch/batch_runner.py

@@ -9,7 +9,7 @@ from dotenv import load_dotenv
 from openai import OpenAI
 from pydantic import BaseModel
 
-from texttools.batch.internals.batch_manager import BatchManager
+from texttools.batch.batch_manager import BatchManager
 from texttools.batch.batch_config import BatchConfig
 from texttools.internals.models import Str
 from texttools.internals.exceptions import TextToolsError, ConfigurationError

{hamtaa_texttools-1.1.20 → hamtaa_texttools-1.1.21}/texttools/internals/async_operator.py

@@ -1,11 +1,10 @@
 from typing import TypeVar, Type
 from collections.abc import Callable
-import logging
 
 from openai import AsyncOpenAI
 from pydantic import BaseModel
 
-from texttools.internals.models import ToolOutput
+from texttools.internals.models import OperatorOutput
 from texttools.internals.operator_utils import OperatorUtils
 from texttools.internals.prompt_loader import PromptLoader
 from texttools.internals.exceptions import (
@@ -18,35 +17,23 @@ from texttools.internals.exceptions import (
 # Base Model type for output models
 T = TypeVar("T", bound=BaseModel)
 
-logger = logging.getLogger("texttools.async_operator")
-
 
 class AsyncOperator:
     """
-    Core engine for running text-processing operations with an LLM (Async).
-
-    It wires together:
-    - `PromptLoader` → loads YAML prompt templates.
-    - `UserMergeFormatter` → applies formatting to messages (e.g., merging).
-    - AsyncOpenAI client → executes completions/parsed completions.
+    Core engine for running text-processing operations with an LLM.
     """
 
     def __init__(self, client: AsyncOpenAI, model: str):
         self._client = client
         self._model = model
 
-    async def _analyze(self, prompt_configs: dict[str, str], temperature: float) -> str:
-        """
-        Calls OpenAI API for analysis using the configured prompt template.
-        Returns the analyzed content as a string.
-        """
+    async def _analyze_completion(self, analyze_prompt: str, temperature: float) -> str:
         try:
-            analyze_prompt = prompt_configs["analyze_template"]
-
             if not analyze_prompt:
                 raise PromptError("Analyze template is empty")
 
-            analyze_message = [OperatorUtils.build_user_message(analyze_prompt)]
+            analyze_message = OperatorUtils.build_user_message(analyze_prompt)
+
             completion = await self._client.chat.completions.create(
                 model=self._model,
                 messages=analyze_message,
@@ -61,7 +48,7 @@ class AsyncOperator:
             if not analysis:
                 raise LLMError("Empty analysis response")
 
-            return analysis.strip()
+            return analysis
 
         except Exception as e:
             if isinstance(e, (PromptError, LLMError)):
@@ -70,21 +57,23 @@ class AsyncOperator:
 
     async def _parse_completion(
         self,
-        message: list[dict[str, str]],
+        main_prompt: str,
         output_model: Type[T],
         temperature: float,
-        logprobs: bool = False,
-        top_logprobs: int = 3,
-        priority: int | None = 0,
+        logprobs: bool,
+        top_logprobs: int,
+        priority: int,
     ) -> tuple[T, object]:
         """
         Parses a chat completion using OpenAI's structured output format.
         Returns both the parsed object and the raw completion for logprobs.
         """
         try:
+            main_message = OperatorUtils.build_user_message(main_prompt)
+
             request_kwargs = {
                 "model": self._model,
-                "messages": message,
+                "messages": main_message,
                 "response_format": output_model,
                 "temperature": temperature,
             }
@@ -92,8 +81,10 @@ class AsyncOperator:
             if logprobs:
                 request_kwargs["logprobs"] = True
                 request_kwargs["top_logprobs"] = top_logprobs
+
             if priority:
                 request_kwargs["extra_body"] = {"priority": priority}
+
             completion = await self._client.beta.chat.completions.parse(
                 **request_kwargs
             )
@@ -122,24 +113,22 @@ class AsyncOperator:
         user_prompt: str | None,
         temperature: float,
         logprobs: bool,
-        top_logprobs: int | None,
+        top_logprobs: int,
         validator: Callable[[object], bool] | None,
         max_validation_retries: int | None,
+        priority: int,
         # Internal parameters
         prompt_file: str,
         output_model: Type[T],
         mode: str | None,
-        priority: int | None = 0,
         **extra_kwargs,
-    ) -> ToolOutput:
+    ) -> OperatorOutput:
         """
-        Execute the LLM pipeline with the given input text. (Async)
+        Execute the LLM pipeline with the given input text. (Sync)
         """
         try:
             prompt_loader = PromptLoader()
-            output = ToolOutput()
 
-            # Prompt configs contain two keys: main_template and analyze template, both are string
             prompt_configs = prompt_loader.load(
                 prompt_file=prompt_file,
                 text=text.strip(),
@@ -147,47 +136,32 @@ class AsyncOperator:
                 **extra_kwargs,
             )
 
-            messages = []
+            main_prompt = ""
+            analysis = ""
 
             if with_analysis:
-                analysis = await self._analyze(prompt_configs, temperature)
-                messages.append(
-                    OperatorUtils.build_user_message(
-                        f"Based on this analysis: {analysis}"
-                    )
+                analysis = await self._analyze_completion(
+                    prompt_configs["analyze_template"], temperature
                 )
+                main_prompt += f"Based on this analysis:\n{analysis}\n"
 
             if output_lang:
-                messages.append(
-                    OperatorUtils.build_user_message(
-                        f"Respond only in the {output_lang} language."
-                    )
-                )
+                main_prompt += f"Respond only in the {output_lang} language.\n"
 
             if user_prompt:
-                messages.append(
-                    OperatorUtils.build_user_message(
-                        f"Consider this instruction {user_prompt}"
-                    )
-                )
-
-            messages.append(
-                OperatorUtils.build_user_message(prompt_configs["main_template"])
-            )
+                main_prompt += f"Consider this instruction {user_prompt}\n"
 
-            messages = OperatorUtils.user_merge_format(messages)
+            main_prompt += prompt_configs["main_template"]
 
             if logprobs and (not isinstance(top_logprobs, int) or top_logprobs < 2):
                 raise ValueError("top_logprobs should be an integer greater than 1")
 
             parsed, completion = await self._parse_completion(
-                messages, output_model, temperature, logprobs, top_logprobs, priority
+                main_prompt, output_model, temperature, logprobs, top_logprobs, priority
             )
 
-            output.result = parsed.result
-
             # Retry logic if validation fails
-            if validator and not validator(output.result):
+            if validator and not validator(parsed.result):
                 if (
                     not isinstance(max_validation_retries, int)
                     or max_validation_retries < 1
@@ -197,17 +171,13 @@ class AsyncOperator:
                     )
 
                 succeeded = False
-                for attempt in range(max_validation_retries):
-                    logger.warning(
-                        f"Validation failed, retrying for the {attempt + 1} time."
-                    )
-
-                    # Generate new temperature for retry
+                for _ in range(max_validation_retries):
+                    # Generate a new temperature to retry
                     retry_temperature = OperatorUtils.get_retry_temp(temperature)
 
                     try:
                         parsed, completion = await self._parse_completion(
-                            messages,
+                            main_prompt,
                             output_model,
                             retry_temperature,
                             logprobs,
@@ -215,30 +185,26 @@ class AsyncOperator:
                             priority=priority,
                         )
 
-                        output.result = parsed.result
-
                         # Check if retry was successful
-                        if validator(output.result):
+                        if validator(parsed.result):
                             succeeded = True
                             break
 
-                    except LLMError as e:
-                        logger.error(f"Retry attempt {attempt + 1} failed: {e}")
+                    except LLMError:
+                        pass
 
                 if not succeeded:
-                    raise ValidationError(
-                        f"Validation failed after {max_validation_retries} retries"
-                    )
-
-            if logprobs:
-                output.logprobs = OperatorUtils.extract_logprobs(completion)
-
-            if with_analysis:
-                output.analysis = analysis
-
-            output.process = prompt_file[:-5]
+                    raise ValidationError("Validation failed after all retries")
+
+            operator_output = OperatorOutput(
+                result=parsed.result,
+                analysis=analysis if with_analysis else None,
+                logprobs=OperatorUtils.extract_logprobs(completion)
+                if logprobs
+                else None,
+            )
 
-            return output
+            return operator_output
 
         except (PromptError, LLMError, ValidationError):
             raise