hamtaa-texttools 1.1.1__py3-none-any.whl → 1.1.16__py3-none-any.whl

{hamtaa_texttools-1.1.1.dist-info → hamtaa_texttools-1.1.16.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hamtaa-texttools
-Version: 1.1.1
+Version: 1.1.16
 Summary: A high-level NLP toolkit built on top of modern LLMs.
 Author-email: Tohidi <the.mohammad.tohidi@gmail.com>, Montazer <montazerh82@gmail.com>, Givechi <mohamad.m.givechi@gmail.com>, MoosaviNejad <erfanmoosavi84@gmail.com>
 License: MIT License
@@ -29,6 +29,7 @@ Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: openai==1.97.1
+Requires-Dist: pydantic>=2.0.0
 Requires-Dist: pyyaml>=6.0
 Dynamic: license-file
 
@@ -40,50 +41,69 @@ Dynamic: license-file
 
 It provides both **sync (`TheTool`)** and **async (`AsyncTheTool`)** APIs for maximum flexibility.
 
-It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extractor, and more** — designed to help you integrate AI-powered text processing into your applications with minimal effort.
+It provides ready-to-use utilities for **translation, question detection, keyword extraction, categorization, NER extraction, and more** - designed to help you integrate AI-powered text processing into your applications with minimal effort.
 
 ---
 
 ## ✨ Features
 
-TextTools provides a rich collection of high-level NLP utilities built on top of LLMs.  
-Each tool is designed to work out-of-the-box with structured outputs (JSON / Pydantic).
+TextTools provides a rich collection of high-level NLP utilities,
+Each tool is designed to work with structured outputs (JSON / Pydantic).
 
-- **`categorize()`** - Classifies text into Islamic studies categories 
-- **`is_question()`** - Binary detection of whether input is a question
+- **`categorize()`** - Classifies text into given categories (You have to create a category tree)
 - **`extract_keywords()`** - Extracts keywords from text
 - **`extract_entities()`** - Named Entity Recognition (NER) system
-- **`summarize()`** - Text summarization
+- **`is_question()`** - Binary detection of whether input is a question
 - **`text_to_question()`** - Generates questions from text
 - **`merge_questions()`** - Merges multiple questions with different modes
 - **`rewrite()`** - Rewrites text with different wording/meaning
 - **`subject_to_question()`** - Generates questions about a specific subject
+- **`summarize()`** - Text summarization
 - **`translate()`** - Text translation between languages
-- **`run_custom()`** - Allows users to define a custom tool with arbitrary BaseModel
+- **`run_custom()`** - Allows users to define a custom tool with an arbitrary BaseModel
 
 ---
 
-## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt` and `temperature` parameters
+## ⚙️ `with_analysis`, `logprobs`, `output_lang`, `user_prompt`, `temperature`, `validator` and `priority` parameters
 
 TextTools provides several optional flags to customize LLM behavior:
 
-- **`with_analysis=True`** → Adds a reasoning step before generating the final output. Useful for debugging, improving prompts, or understanding model behavior.  
-Note: This doubles token usage per call because it triggers an additional LLM request.
+- **`with_analysis (bool)`** → Adds a reasoning step before generating the final output.
+**Note:** This doubles token usage per call because it triggers an additional LLM request.
 
-- **`logprobs=True`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
+- **`logprobs (bool)`** → Returns token-level probabilities for the generated output. You can also specify `top_logprobs=<N>` to get the top N alternative tokens and their probabilities.  
+**Note:** This feature works if it's supported by the model.
 
-- **`output_lang="en"`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
+- **`output_lang (str)`** → Forces the model to respond in a specific language. The model will ignore other instructions about language and respond strictly in the requested language.
 
-- **`user_prompt="..."`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
+- **`user_prompt (str)`** → Allows you to inject a custom instruction or prompt into the model alongside the main template. This gives you fine-grained control over how the model interprets or modifies the input text.
 
-- **`temperature=0.0`** → Determines how creative the model should respond. Takes a float number from `0.0` to `1.0`.
+- **`temperature (float)`** → Determines how creative the model should respond. Takes a float number from `0.0` to `2.0`.
 
-All these parameters can be used individually or together to tailor the behavior of any tool in **TextTools**.
+- **`validator (Callable)`** → Forces TheTool to validate the output result based on your custom validator. Validator should return a bool (True if there were no problem, False if the validation fails.) If the validator fails, TheTool will retry to get another output by modifying `temperature`. You can specify `max_validation_retries=<N>` to change the number of retries.
+
+- **`priority (int)`** → Task execution priority level. Higher values = higher priority. Affects processing order in queues.
+**Note:** This feature works if it's supported by the model and vLLM.
 
 **Note:** There might be some tools that don't support some of the parameters above.
 
 ---
 
+## 🧩 ToolOutput
+
+Every tool of `TextTools` returns a `ToolOutput` object which is a BaseModel with attributes:
+- **`result: Any`** → The output of LLM
+- **`analysis: str`** → The reasoning step before generating the final output
+- **`logprobs: list`** → Token-level probabilities for the generated output 
+- **`process: str`** → The tool name which processed the input
+- **`processed_at: datetime`** → The process time
+- **`execution_time: float`** → The execution time (seconds)
+- **`errors: list[str]`** → Any error that have occured during calling LLM
+
+**Note:** You can use `repr(ToolOutput)` to see details of your ToolOutput.
+
+---
+
 ## 🚀 Installation
 
 Install the latest release via PyPI:
@@ -94,7 +114,7 @@ pip install -U hamtaa-texttools
 
 ---
 
-## Sync vs Async
+## 🧨 Sync vs Async
 | Tool         | Style   | Use case                                    |
 |--------------|---------|---------------------------------------------|
 | `TheTool`    | Sync    | Simple scripts, sequential workflows        |
@@ -121,13 +141,13 @@ the_tool = TheTool(client=client, model=model)
 detection = the_tool.is_question("Is this project open source?", logprobs=True, top_logprobs=2)
 print(detection.result)
 print(detection.logprobs)
-# Output: True \n --logprobs
+# Output: True + logprobs
 
 # Example: Translation
 translation = the_tool.translate("سلام، حالت چطوره؟" target_language="English", with_analysis=True)
 print(translation.result)
 print(translation.analysis)
-# Output: "Hi! How are you?" \n --analysis
+# Output: "Hi! How are you?"  + analysis
 ```
 
 ---
@@ -147,19 +167,22 @@ async def main():
     model = "gpt-4o-mini"
 
     # Create an instance of AsyncTheTool
-    the_tool = AsyncTheTool(client=async_client, model=model)
+    async_the_tool = AsyncTheTool(client=async_client, model=model)
+    
+    # Example: Async Translation and Keyword Extraction
+    translation_task = async_the_tool.translate("سلام، حالت چطوره؟", target_language="English")
+    keywords_task = async_the_tool.extract_keywords("Tomorrow, we will be dead by the car crash")
 
-    # Example: Async Translation
-    translation = await the_tool.translate("سلام، حالت چطوره؟", target_language="English")
+    (translation, keywords) = await asyncio.gather(translation_task, keywords_task)
     print(translation.result)
-    # Output: "Hi! How are you?"
+    print(keywords.result)
 
 asyncio.run(main())
 ```
 
 ---
 
-## 📚 Use Cases
+## 👍 Use Cases
 
 Use **TextTools** when you need to:
 
@@ -167,7 +190,56 @@ Use **TextTools** when you need to:
 - 🌍 **Translate** and process multilingual corpora with ease  
 - 🧩 **Integrate** LLMs into production pipelines (structured outputs)  
 - 📊 **Analyze** large text collections using embeddings and categorization  
-- 👍 **Automate** common text-processing tasks without reinventing the wheel  
+
+---
+
+## 🔍 Logging
+
+TextTools uses Python's standard `logging` module. The library's default logger level is `WARNING`, so if you want to modify it, follow instructions:
+
+
+```python
+import logging
+
+# Default: warnings and errors only
+logging.basicConfig(level=logging.WARNING)
+
+# Debug everything (verbose)
+logging.basicConfig(level=logging.DEBUG)
+
+# Complete silence
+logging.basicConfig(level=logging.CRITICAL)
+```
+
+---
+
+## 📚 Batch Processing
+
+Process large datasets efficiently using OpenAI's batch API.
+
+## ⚡ Quick Start (Batch)
+
+```python
+from pydantic import BaseModel
+from texttools import BatchJobRunner, BatchConfig
+
+# Configure your batch job
+config = BatchConfig(
+    system_prompt="Extract entities from the text",
+    job_name="entity_extraction",
+    input_data_path="data.json",
+    output_data_filename="results.json",
+    model="gpt-4o-mini"
+)
+
+# Define your output schema
+class Output(BaseModel):
+    entities: list[str]
+
+# Run the batch job
+runner = BatchJobRunner(config, output_model=Output)
+runner.run()
+```
 
 ---
 
@@ -178,6 +250,6 @@ Feel free to **open issues, suggest new features, or submit pull requests**.
 
 ---
 
-## License
+## 🌿 License
 
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

hamtaa_texttools-1.1.16.dist-info/RECORD

@@ -0,0 +1,31 @@
+hamtaa_texttools-1.1.16.dist-info/licenses/LICENSE,sha256=Hb2YOBKy2MJQLnyLrX37B4ZVuac8eaIcE71SvVIMOLg,1082
+texttools/__init__.py,sha256=dc81lXGWP29k7oVvq2BMoMotz6lgiwX4PO2jHHBe2S8,317
+texttools/batch/batch_config.py,sha256=m1UgILVKjNdWE6laNbfbG4vgi4o2fEegGZbeoam6pnY,749
+texttools/batch/batch_runner.py,sha256=9e4SPLlvLHHs3U7bHkuuMVw8TFNwsGUzRjkAMKN4_ik,9378
+texttools/batch/internals/batch_manager.py,sha256=UoBe76vmFG72qrSaGKDZf4HzkykFBkkkbL9TLfV8TuQ,8730
+texttools/batch/internals/utils.py,sha256=F1_7YlVFKhjUROAFX4m0SaP8KiZVZyHRMIIB87VUGQc,373
+texttools/prompts/README.md,sha256=-5YO93CN93QLifqZpUeUnCOCBbDiOTV-cFQeJ7Gg0I4,1377
+texttools/prompts/categorize.yaml,sha256=F7VezB25B_sT5yoC25ezODBddkuDD5lUHKetSpx9FKI,2743
+texttools/prompts/detect_entity.yaml,sha256=1rhMkJOjxSQcT4j_c5SRcIm77AUdeG-rUmeidb6VOFc,981
+texttools/prompts/extract_entities.yaml,sha256=KiKjeDpHaeh3JVtZ6q1pa3k4DYucUIU9WnEcRTCA-SE,651
+texttools/prompts/extract_keywords.yaml,sha256=Vj4Tt3vT6LtpOo_iBZPo9oWI50oVdPGXe5i8yDR8ex4,3177
+texttools/prompts/is_question.yaml,sha256=d0-vKRbXWkxvO64ikvxRjEmpAXGpCYIPGhgexvPPjws,471
+texttools/prompts/merge_questions.yaml,sha256=0J85GvTirZB4ELwH3sk8ub_WcqqpYf6PrMKr3djlZeo,1792
+texttools/prompts/rewrite.yaml,sha256=LO7He_IA3MZKz8a-LxH9DHJpOjpYwaYN1pbjp1Y0tFo,5392
+texttools/prompts/run_custom.yaml,sha256=38OkCoVITbuuS9c08UZSP1jZW4WjSmRIi8fR0RAiPu4,108
+texttools/prompts/subject_to_question.yaml,sha256=C7x7rNNm6U_ZG9HOn6zuzYOtvJUZ2skuWbL1-aYdd3E,1147
+texttools/prompts/summarize.yaml,sha256=o6rxGPfWtZd61Duvm8NVvCJqfq73b-wAuMSKR6UYUqY,459
+texttools/prompts/text_to_question.yaml,sha256=UheKYpDn6iyKI8NxunHZtFpNyfCLZZe5cvkuXpurUJY,783
+texttools/prompts/translate.yaml,sha256=mGT2uBCei6uucWqVbs4silk-UV060v3G0jnt0P6sr50,634
+texttools/tools/async_tools.py,sha256=vNAg0gxwUZPsMS4q8JCv7RlYymS8l_5FsFI5adEYT7w,34376
+texttools/tools/sync_tools.py,sha256=hFifFa9YatvSeGif2E_bIG006eMdIBr6SV9HsZ_dAlg,34187
+texttools/tools/internals/async_operator.py,sha256=1TMr8e1qbE9GSz8jl0q3MKdM8lIYE-1ZuSxHjYPqKHI,7198
+texttools/tools/internals/formatters.py,sha256=tACNLP6PeoqaRpNudVxBaHA25zyWqWYPZQuYysIu88g,941
+texttools/tools/internals/models.py,sha256=2QnvMiijuSqOqpCl026848rJy_pHNbRoDESlQvcdHlk,5839
+texttools/tools/internals/operator_utils.py,sha256=w1k0RJ_W_CRbVc_J2w337VuL-opHpHiCxfhEOwtyuOo,1856
+texttools/tools/internals/prompt_loader.py,sha256=4g6-U8kqrGN7VpNaRcrBcnF-h03PXjUDBP0lL0_4EZY,1953
+texttools/tools/internals/sync_operator.py,sha256=4-V__o55Q8w29lWxkhG4St-exZLZTfBbiW76knOXbc0,7106
+hamtaa_texttools-1.1.16.dist-info/METADATA,sha256=DL-cjlGMv7bft8QVd-pn5E_tNDuPgQHkTKGl4YTosGw,9555
+hamtaa_texttools-1.1.16.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
+hamtaa_texttools-1.1.16.dist-info/top_level.txt,sha256=5Mh0jIxxZ5rOXHGJ6Mp-JPKviywwN0MYuH0xk5bEWqE,10
+hamtaa_texttools-1.1.16.dist-info/RECORD,,

texttools/__init__.py

@@ -1,9 +1,7 @@
-from .batch import BatchJobRunner, SimpleBatchManager
-from .tools import AsyncTheTool, TheTool
+from .batch.batch_runner import BatchJobRunner
+from .batch.batch_config import BatchConfig
+from .tools.sync_tools import TheTool
+from .tools.async_tools import AsyncTheTool
+from .tools.internals.models import CategoryTree
 
-__all__ = [
-    "TheTool",
-    "AsyncTheTool",
-    "SimpleBatchManager",
-    "BatchJobRunner",
-]
+__all__ = ["TheTool", "AsyncTheTool", "BatchJobRunner", "BatchConfig", "CategoryTree"]

texttools/batch/batch_config.py

@@ -0,0 +1,26 @@
+from dataclasses import dataclass
+from collections.abc import Callable
+
+from texttools.batch.internals.utils import import_data, export_data
+
+
+@dataclass
+class BatchConfig:
+    """
+    Configuration for batch job runner.
+    """
+
+    system_prompt: str = ""
+    job_name: str = ""
+    input_data_path: str = ""
+    output_data_filename: str = ""
+    model: str = "gpt-4.1-mini"
+    MAX_BATCH_SIZE: int = 100
+    MAX_TOTAL_TOKENS: int = 2_000_000
+    CHARS_PER_TOKEN: float = 2.7
+    PROMPT_TOKEN_MULTIPLIER: int = 1_000
+    BASE_OUTPUT_DIR: str = "Data/batch_entity_result"
+    import_function: Callable = import_data
+    export_function: Callable = export_data
+    poll_interval_seconds: int = 30
+    max_retries: int = 3

texttools/batch/batch_runner.py

@@ -1,61 +1,22 @@
 import json
 import os
 import time
-from dataclasses import dataclass
 from pathlib import Path
-from typing import Any, Callable
+from typing import Any, Type, TypeVar
 import logging
 
 from dotenv import load_dotenv
 from openai import OpenAI
 from pydantic import BaseModel
 
-from texttools.batch import SimpleBatchManager
+from texttools.batch.internals.batch_manager import BatchManager
+from texttools.batch.batch_config import BatchConfig
+from texttools.tools.internals.models import StrOutput
 
-# Configure logger
-logger = logging.getLogger("batch_runner")
-logger.setLevel(logging.INFO)
+# Base Model type for output models
+T = TypeVar("T", bound=BaseModel)
 
-
-class OutputModel(BaseModel):
-    desired_output: str
-
-
-def export_data(data):
-    """
-    Produces a structure of the following form from an initial data structure:
-    [{"id": str, "text": str},...]
-    """
-    return data
-
-
-def import_data(data):
-    """
-    Takes the output and adds and aggregates it to the original structure.
-    """
-    return data
-
-
-@dataclass
-class BatchConfig:
-    """
-    Configuration for batch job runner.
-    """
-
-    system_prompt: str = ""
-    job_name: str = ""
-    input_data_path: str = ""
-    output_data_filename: str = ""
-    model: str = "gpt-4.1-mini"
-    MAX_BATCH_SIZE: int = 100
-    MAX_TOTAL_TOKENS: int = 2000000
-    CHARS_PER_TOKEN: float = 2.7
-    PROMPT_TOKEN_MULTIPLIER: int = 1000
-    BASE_OUTPUT_DIR: str = "Data/batch_entity_result"
-    import_function: Callable = import_data
-    export_function: Callable = export_data
-    poll_interval_seconds: int = 30
-    max_retries: int = 3
+logger = logging.getLogger("texttools.batch_runner")
 
 
 class BatchJobRunner:
@@ -64,142 +25,180 @@ class BatchJobRunner:
     """
 
     def __init__(
-        self, config: BatchConfig = BatchConfig(), output_model: type = OutputModel
+        self, config: BatchConfig = BatchConfig(), output_model: Type[T] = StrOutput
     ):
-        self.config = config
-        self.system_prompt = config.system_prompt
-        self.job_name = config.job_name
-        self.input_data_path = config.input_data_path
-        self.output_data_filename = config.output_data_filename
-        self.model = config.model
-        self.output_model = output_model
-        self.manager = self._init_manager()
-        self.data = self._load_data()
-        self.parts: list[list[dict[str, Any]]] = []
-        self._partition_data()
-        Path(self.config.BASE_OUTPUT_DIR).mkdir(parents=True, exist_ok=True)
+        self._config = config
+        self._system_prompt = config.system_prompt
+        self._job_name = config.job_name
+        self._input_data_path = config.input_data_path
+        self._output_data_filename = config.output_data_filename
+        self._model = config.model
+        self._output_model = output_model
+        self._manager = self._init_manager()
+        self._data = self._load_data()
+        self._parts: list[list[dict[str, Any]]] = []
         # Map part index to job name
-        self.part_idx_to_job_name: dict[int, str] = {}
+        self._part_idx_to_job_name: dict[int, str] = {}
         # Track retry attempts per part
-        self.part_attempts: dict[int, int] = {}
+        self._part_attempts: dict[int, int] = {}
+        self._partition_data()
+        Path(self._config.BASE_OUTPUT_DIR).mkdir(parents=True, exist_ok=True)
 
-    def _init_manager(self) -> SimpleBatchManager:
+    def _init_manager(self) -> BatchManager:
         load_dotenv()
         api_key = os.getenv("OPENAI_API_KEY")
         client = OpenAI(api_key=api_key)
-        return SimpleBatchManager(
+        return BatchManager(
             client=client,
-            model=self.model,
-            prompt_template=self.system_prompt,
-            output_model=self.output_model,
+            model=self._model,
+            prompt_template=self._system_prompt,
+            output_model=self._output_model,
         )
 
     def _load_data(self):
-        with open(self.input_data_path, "r", encoding="utf-8") as f:
+        with open(self._input_data_path, "r", encoding="utf-8") as f:
             data = json.load(f)
-        data = self.config.export_function(data)
+        data = self._config.export_function(data)
 
         # Ensure data is a list of dicts with 'id' and 'content' as strings
         if not isinstance(data, list):
             raise ValueError(
-                'Exported data must be a list in this form:  [ {"id": str, "content": str},...]'
+                "Exported data must be a list of dicts with 'id' and 'content' keys"
             )
         for item in data:
             if not (isinstance(item, dict) and "id" in item and "content" in item):
                 raise ValueError(
-                    "Each item must be a dict with 'id' and 'content' keys."
+                    f"Item must be a dict with 'id' and 'content' keys. Got: {type(item)}"
                 )
             if not (isinstance(item["id"], str) and isinstance(item["content"], str)):
                 raise ValueError("'id' and 'content' must be strings.")
         return data
 
     def _partition_data(self):
-        total_length = sum(len(item["content"]) for item in self.data)
-        prompt_length = len(self.system_prompt)
-        total = total_length + (prompt_length * len(self.data))
-        calculation = total / self.config.CHARS_PER_TOKEN
+        total_length = sum(len(item["content"]) for item in self._data)
+        prompt_length = len(self._system_prompt)
+        total = total_length + (prompt_length * len(self._data))
+        calculation = total / self._config.CHARS_PER_TOKEN
         logger.info(
             f"Total chars: {total_length}, Prompt chars: {prompt_length}, Total: {total}, Tokens: {calculation}"
         )
-        if calculation < self.config.MAX_TOTAL_TOKENS:
-            self.parts = [self.data]
+        if calculation < self._config.MAX_TOTAL_TOKENS:
+            self._parts = [self._data]
         else:
             # Partition into chunks of MAX_BATCH_SIZE
-            self.parts = [
-                self.data[i : i + self.config.MAX_BATCH_SIZE]
-                for i in range(0, len(self.data), self.config.MAX_BATCH_SIZE)
+            self._parts = [
+                self._data[i : i + self._config.MAX_BATCH_SIZE]
+                for i in range(0, len(self._data), self._config.MAX_BATCH_SIZE)
             ]
-        logger.info(f"Data split into {len(self.parts)} part(s)")
+        logger.info(f"Data split into {len(self._parts)} part(s)")
 
     def _submit_all_jobs(self) -> None:
-        for idx, part in enumerate(self.parts):
+        for idx, part in enumerate(self._parts):
             if self._result_exists(idx):
                 logger.info(f"Skipping part {idx + 1}: result already exists.")
                 continue
             part_job_name = (
-                f"{self.job_name}_part_{idx + 1}"
-                if len(self.parts) > 1
-                else self.job_name
+                f"{self._job_name}_part_{idx + 1}"
+                if len(self._parts) > 1
+                else self._job_name
             )
             # If a job with this name already exists, register and skip submitting
-            existing_job = self.manager._load_state(part_job_name)
+            existing_job = self._manager._load_state(part_job_name)
             if existing_job:
                 logger.info(
                     f"Skipping part {idx + 1}: job already exists ({part_job_name})."
                 )
-                self.part_idx_to_job_name[idx] = part_job_name
-                self.part_attempts.setdefault(idx, 0)
+                self._part_idx_to_job_name[idx] = part_job_name
+                self._part_attempts.setdefault(idx, 0)
                 continue
 
             payload = part
             logger.info(
-                f"Submitting job for part {idx + 1}/{len(self.parts)}: {part_job_name}"
+                f"Submitting job for part {idx + 1}/{len(self._parts)}: {part_job_name}"
             )
-            self.manager.start(payload, job_name=part_job_name)
-            self.part_idx_to_job_name[idx] = part_job_name
-            self.part_attempts.setdefault(idx, 0)
+            self._manager.start(payload, job_name=part_job_name)
+            self._part_idx_to_job_name[idx] = part_job_name
+            self._part_attempts.setdefault(idx, 0)
             # This is added for letting file get uploaded, before starting the next part.
             logger.info("Uploading...")
             time.sleep(30)
 
+    def _save_results(
+        self,
+        output_data: list[dict[str, Any]] | dict[str, Any],
+        log: list[Any],
+        part_idx: int,
+    ):
+        part_suffix = f"_part_{part_idx + 1}" if len(self._parts) > 1 else ""
+        result_path = (
+            Path(self._config.BASE_OUTPUT_DIR)
+            / f"{Path(self._output_data_filename).stem}{part_suffix}.json"
+        )
+        if not output_data:
+            logger.info("No output data to save. Skipping this part.")
+            return
+        else:
+            with open(result_path, "w", encoding="utf-8") as f:
+                json.dump(output_data, f, ensure_ascii=False, indent=4)
+        if log:
+            log_path = (
+                Path(self._config.BASE_OUTPUT_DIR)
+                / f"{Path(self._output_data_filename).stem}{part_suffix}_log.json"
+            )
+            with open(log_path, "w", encoding="utf-8") as f:
+                json.dump(log, f, ensure_ascii=False, indent=4)
+
+    def _result_exists(self, part_idx: int) -> bool:
+        part_suffix = f"_part_{part_idx + 1}" if len(self._parts) > 1 else ""
+        result_path = (
+            Path(self._config.BASE_OUTPUT_DIR)
+            / f"{Path(self._output_data_filename).stem}{part_suffix}.json"
+        )
+        return result_path.exists()
+
     def run(self):
+        """
+        Execute the batch job processing pipeline.
+
+        Submits jobs, monitors progress, handles retries, and saves results.
+        """
         # Submit all jobs up-front for concurrent execution
         self._submit_all_jobs()
-        pending_parts: set[int] = set(self.part_idx_to_job_name.keys())
+        pending_parts: set[int] = set(self._part_idx_to_job_name.keys())
         logger.info(f"Pending parts: {sorted(pending_parts)}")
         # Polling loop
         while pending_parts:
             finished_this_round: list[int] = []
             for part_idx in list(pending_parts):
-                job_name = self.part_idx_to_job_name[part_idx]
-                status = self.manager.check_status(job_name=job_name)
+                job_name = self._part_idx_to_job_name[part_idx]
+                status = self._manager.check_status(job_name=job_name)
                 logger.info(f"Status for {job_name}: {status}")
                 if status == "completed":
                     logger.info(
                         f"Job completed. Fetching results for part {part_idx + 1}..."
                     )
-                    output_data, log = self.manager.fetch_results(
+                    output_data, log = self._manager.fetch_results(
                         job_name=job_name, remove_cache=False
                     )
-                    output_data = self.config.import_function(output_data)
+                    output_data = self._config.import_function(output_data)
                     self._save_results(output_data, log, part_idx)
                     logger.info(f"Fetched and saved results for part {part_idx + 1}.")
                     finished_this_round.append(part_idx)
                 elif status == "failed":
-                    attempt = self.part_attempts.get(part_idx, 0) + 1
-                    self.part_attempts[part_idx] = attempt
-                    if attempt <= self.config.max_retries:
+                    attempt = self._part_attempts.get(part_idx, 0) + 1
+                    self._part_attempts[part_idx] = attempt
+                    if attempt <= self._config.max_retries:
                         logger.info(
                             f"Job {job_name} failed (attempt {attempt}). Retrying after short backoff..."
                         )
-                        self.manager._clear_state(job_name)
+                        self._manager._clear_state(job_name)
                         time.sleep(10)
-                        payload = self._to_manager_payload(self.parts[part_idx])
+                        payload = self._to_manager_payload(self._parts[part_idx])
                         new_job_name = (
-                            f"{self.job_name}_part_{part_idx + 1}_retry_{attempt}"
+                            f"{self._job_name}_part_{part_idx + 1}_retry_{attempt}"
                         )
-                        self.manager.start(payload, job_name=new_job_name)
-                        self.part_idx_to_job_name[part_idx] = new_job_name
+                        self._manager.start(payload, job_name=new_job_name)
+                        self._part_idx_to_job_name[part_idx] = new_job_name
                     else:
                         logger.info(
                             f"Job {job_name} failed after {attempt - 1} retries. Marking as failed."
@@ -213,51 +212,6 @@ class BatchJobRunner:
                 pending_parts.discard(part_idx)
             if pending_parts:
                 logger.info(
-                    f"Waiting {self.config.poll_interval_seconds}s before next status check for parts: {sorted(pending_parts)}"
+                    f"Waiting {self._config.poll_interval_seconds}s before next status check for parts: {sorted(pending_parts)}"
                 )
-                time.sleep(self.config.poll_interval_seconds)
-
-    def _save_results(
-        self,
-        output_data: list[dict[str, Any]] | dict[str, Any],
-        log: list[Any],
-        part_idx: int,
-    ):
-        part_suffix = f"_part_{part_idx + 1}" if len(self.parts) > 1 else ""
-        result_path = (
-            Path(self.config.BASE_OUTPUT_DIR)
-            / f"{Path(self.output_data_filename).stem}{part_suffix}.json"
-        )
-        if not output_data:
-            logger.info("No output data to save. Skipping this part.")
-            return
-        else:
-            with open(result_path, "w", encoding="utf-8") as f:
-                json.dump(output_data, f, ensure_ascii=False, indent=4)
-        if log:
-            log_path = (
-                Path(self.config.BASE_OUTPUT_DIR)
-                / f"{Path(self.output_data_filename).stem}{part_suffix}_log.json"
-            )
-            with open(log_path, "w", encoding="utf-8") as f:
-                json.dump(log, f, ensure_ascii=False, indent=4)
-
-    def _result_exists(self, part_idx: int) -> bool:
-        part_suffix = f"_part_{part_idx + 1}" if len(self.parts) > 1 else ""
-        result_path = (
-            Path(self.config.BASE_OUTPUT_DIR)
-            / f"{Path(self.output_data_filename).stem}{part_suffix}.json"
-        )
-        return result_path.exists()
-
-
-if __name__ == "__main__":
-    logger.info("=== Batch Job Runner ===")
-    config = BatchConfig(
-        system_prompt="",
-        job_name="job_name",
-        input_data_path="Data.json",
-        output_data_filename="output",
-    )
-    runner = BatchJobRunner(config)
-    runner.run()
+                time.sleep(self._config.poll_interval_seconds)