hamtaa-texttools 0.1.43__tar.gz

hamtaa_texttools-0.1.43/PKG-INFO

@@ -0,0 +1,60 @@
+Metadata-Version: 2.4
+Name: hamtaa-texttools
+Version: 0.1.43
+Summary: A set of high-level NLP tools
+Author: Tohidi, Montazer, Givechi, Mousavinezhad
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: openai==1.97.1
+Requires-Dist: numpy==1.26.4
+
+# Text Tools
+
+<p align="center">
+  <img src="https://img.shields.io/badge/TextTools-Python%20Text%20Processing-black?style=for-the-badge&logo=python&logoColor=white">
+</p>
+
+
+<p align="center">
+  <img src="docs/logo.png" alt="Preview" width="300" height="300">
+</p>
+
+
+### How to Install
+
+Install the package using:
+
+```bash
+pip install -U hamta-texttools
+```
+
+
+---
+
+## What This Library Is *Not*
+
+This is **not** a collection of low-level utilities.
+
+To clarify: this library **does not** include things like:
+- An standard `regex`
+- Word normalization utilities
+
+---
+
+## What This Library *Provides*
+
+This is a set of **high-level natural language processing (NLP)** tools.
+
+Some of the features include:
+- `question_detector`: Detecting if an incoming text is a question or not
+- `categorizer`: No finetuning need, categorizer
+- ... (Tell me what you want!)
+
+---
+
+## When to Use This Library
+
+Use `texttools` when:
+- You need to **process large volumes of data using OpenAI’s GPT models** via the BATCH API.
+- You want to treat an **LLM as a function** in Python that outputs structured JSON or Pydantic models.
+- You need to **categorize large datasets** using vector embeddings, efficiently and at scale.

hamtaa_texttools-0.1.43/README.md

@@ -0,0 +1,50 @@
+# Text Tools
+
+<p align="center">
+  <img src="https://img.shields.io/badge/TextTools-Python%20Text%20Processing-black?style=for-the-badge&logo=python&logoColor=white">
+</p>
+
+
+<p align="center">
+  <img src="docs/logo.png" alt="Preview" width="300" height="300">
+</p>
+
+
+### How to Install
+
+Install the package using:
+
+```bash
+pip install -U hamta-texttools
+```
+
+
+---
+
+## What This Library Is *Not*
+
+This is **not** a collection of low-level utilities.
+
+To clarify: this library **does not** include things like:
+- An standard `regex`
+- Word normalization utilities
+
+---
+
+## What This Library *Provides*
+
+This is a set of **high-level natural language processing (NLP)** tools.
+
+Some of the features include:
+- `question_detector`: Detecting if an incoming text is a question or not
+- `categorizer`: No finetuning need, categorizer
+- ... (Tell me what you want!)
+
+---
+
+## When to Use This Library
+
+Use `texttools` when:
+- You need to **process large volumes of data using OpenAI’s GPT models** via the BATCH API.
+- You want to treat an **LLM as a function** in Python that outputs structured JSON or Pydantic models.
+- You need to **categorize large datasets** using vector embeddings, efficiently and at scale.

hamtaa_texttools-0.1.43/hamtaa_texttools.egg-info/PKG-INFO

@@ -0,0 +1,60 @@
+Metadata-Version: 2.4
+Name: hamtaa-texttools
+Version: 0.1.43
+Summary: A set of high-level NLP tools
+Author: Tohidi, Montazer, Givechi, Mousavinezhad
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: openai==1.97.1
+Requires-Dist: numpy==1.26.4
+
+# Text Tools
+
+<p align="center">
+  <img src="https://img.shields.io/badge/TextTools-Python%20Text%20Processing-black?style=for-the-badge&logo=python&logoColor=white">
+</p>
+
+
+<p align="center">
+  <img src="docs/logo.png" alt="Preview" width="300" height="300">
+</p>
+
+
+### How to Install
+
+Install the package using:
+
+```bash
+pip install -U hamta-texttools
+```
+
+
+---
+
+## What This Library Is *Not*
+
+This is **not** a collection of low-level utilities.
+
+To clarify: this library **does not** include things like:
+- An standard `regex`
+- Word normalization utilities
+
+---
+
+## What This Library *Provides*
+
+This is a set of **high-level natural language processing (NLP)** tools.
+
+Some of the features include:
+- `question_detector`: Detecting if an incoming text is a question or not
+- `categorizer`: No finetuning need, categorizer
+- ... (Tell me what you want!)
+
+---
+
+## When to Use This Library
+
+Use `texttools` when:
+- You need to **process large volumes of data using OpenAI’s GPT models** via the BATCH API.
+- You want to treat an **LLM as a function** in Python that outputs structured JSON or Pydantic models.
+- You need to **categorize large datasets** using vector embeddings, efficiently and at scale.

hamtaa_texttools-0.1.43/hamtaa_texttools.egg-info/SOURCES.txt

@@ -0,0 +1,65 @@
+README.md
+pyproject.toml
+setup.py
+hamtaa_texttools.egg-info/PKG-INFO
+hamtaa_texttools.egg-info/SOURCES.txt
+hamtaa_texttools.egg-info/dependency_links.txt
+hamtaa_texttools.egg-info/requires.txt
+hamtaa_texttools.egg-info/top_level.txt
+tests/test_vllm_output.py
+texttools/__init__.py
+texttools/base/__init__.py
+texttools/base/base_categorizer.py
+texttools/base/base_keyword_extractor.py
+texttools/base/base_ner_extractor.py
+texttools/base/base_question_detector.py
+texttools/base/base_question_generator.py
+texttools/base/base_question_merger.py
+texttools/base/base_question_rewriter.py
+texttools/base/base_router.py
+texttools/base/base_summarizer.py
+texttools/base/base_task_performer.py
+texttools/base/base_translator.py
+texttools/batch_manager/__init__.py
+texttools/batch_manager/batch_manager.py
+texttools/batch_manager/batch_runner.py
+texttools/formatter/__init__.py
+texttools/formatter/base.py
+texttools/formatter/gemma3_formatter.py
+texttools/handlers/__init__.py
+texttools/handlers/handlers.py
+texttools/handlers/categorizer/__init__.py
+texttools/handlers/categorizer/categorizer.py
+texttools/tools/__init__.py
+texttools/tools/categorizer/__init__.py
+texttools/tools/categorizer/encoder_model/__init__.py
+texttools/tools/categorizer/encoder_model/encoder_vectorizer.py
+texttools/tools/categorizer/llm/__init__.py
+texttools/tools/categorizer/llm/gemma_categorizer.py
+texttools/tools/categorizer/llm/openai_categorizer.py
+texttools/tools/keyword_extractor/__init__.py
+texttools/tools/keyword_extractor/gemma_extractor.py
+texttools/tools/merger/__init__.py
+texttools/tools/merger/gemma_question_merger.py
+texttools/tools/ner/__init__.py
+texttools/tools/ner/gemma_ner_extractor.py
+texttools/tools/question_detector/__init__.py
+texttools/tools/question_detector/gemma_detector.py
+texttools/tools/question_detector/llm_detector.py
+texttools/tools/question_generator/__init__.py
+texttools/tools/question_generator/gemma_question_generator.py
+texttools/tools/reranker/__init__.py
+texttools/tools/reranker/reranker.py
+texttools/tools/reranker/scorer.py
+texttools/tools/reranker/sorter.py
+texttools/tools/rewriter/__init__.py
+texttools/tools/rewriter/gemma_question_rewriter.py
+texttools/tools/router/__init__.py
+texttools/tools/router/gemma_router.py
+texttools/tools/subject_to_question/__init__.py
+texttools/tools/subject_to_question/gemma_question_generator.py
+texttools/tools/summarizer/__init__.py
+texttools/tools/summarizer/gemma_summarizer.py
+texttools/tools/summarizer/llm_summerizer.py
+texttools/tools/translator/__init__.py
+texttools/tools/translator/gemma_translator.py

hamtaa_texttools-0.1.43/hamtaa_texttools.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

hamtaa_texttools-0.1.43/hamtaa_texttools.egg-info/requires.txt

@@ -0,0 +1,2 @@
+openai==1.97.1
+numpy==1.26.4

hamtaa_texttools-0.1.43/hamtaa_texttools.egg-info/top_level.txt

@@ -0,0 +1 @@
+texttools

hamtaa_texttools-0.1.43/pyproject.toml

@@ -0,0 +1,24 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "hamtaa-texttools"
+version = "0.1.43"
+description = "A set of high-level NLP tools"
+readme = "README.md"
+requires-python = ">=3.8"
+authors = [
+  { name = "Tohidi" },
+  { name = "Montazer" },
+  { name = "Givechi" },
+  { name = "Mousavinezhad" }
+]
+dependencies = [
+  "openai==1.97.1",
+  "numpy==1.26.4",
+]
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["texttools*"]

hamtaa_texttools-0.1.43/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

hamtaa_texttools-0.1.43/setup.py

@@ -0,0 +1,4 @@
+# setup.py
+from setuptools import setup
+
+setup()

hamtaa_texttools-0.1.43/tests/test_vllm_output.py

@@ -0,0 +1,79 @@
+import json
+from typing import Literal
+
+from dotenv import load_dotenv
+from openai import OpenAI
+from pydantic import BaseModel
+
+load_dotenv()
+
+client = OpenAI()
+
+
+class Output(BaseModel):
+    reason: str
+    tag: Literal["Positive", "Negative"]
+
+
+messages = [
+    {
+        "role": "user",
+        "content": """
+        هدف ما طبقه بندی متن هست
+        متن رو بخون و ایده اصلی و آنالیزی کوتاه از اون رو ارائه بده
+
+        بسیار خلاصه باشه خروجی تو
+        نهایتا 20 کلمه
+
+        در نهایت یکی از تگ هارو انتخاب کن
+ 
+        متن:
+        
+        امروز میخواهم به خونه برگردم!!
+        """,
+    }
+]
+
+
+def run_parse():
+    return client.beta.chat.completions.parse(
+        model="gemma-3",
+        messages=messages,
+        response_format=Output,
+        extra_body=dict(guided_decoding_backend="auto"),
+    )
+
+
+def run_json_schema():
+    return client.chat.completions.create(
+        model="gemma-3",
+        messages=messages,
+        response_format={
+            "type": "json_schema",
+            "json_schema": {
+                "name": "output-schema",
+                "schema": Output.model_json_schema(),
+            },
+        },
+        extra_body=dict(guided_decoding_backend="auto"),
+    )
+
+
+def main():
+    # Run parse() and print JSON dict
+    parsed_response = run_parse()
+
+    parsed_response = parsed_response.choices[0].message
+    parsed_response = parsed_response.parsed
+
+    print(parsed_response)
+
+    # Run json_schema and parse + print JSON dict
+    json_schema_response = run_json_schema()
+    raw_content = json_schema_response.choices[0].message.content
+    content_json = json.loads(raw_content)
+    print(json.dumps(content_json, ensure_ascii=False, indent=2))
+
+
+if __name__ == "__main__":
+    main()

hamtaa_texttools-0.1.43/texttools/__init__.py

@@ -0,0 +1,26 @@
+from texttools.batch_manager import BatchJobRunner, SimpleBatchManager
+from texttools.handlers import (
+    NoOpResultHandler,
+    PrintResultHandler,
+    ResultHandler,
+    SaveToFileResultHandler,
+)
+from texttools.tools.categorizer.encoder_model.encoder_vectorizer import (
+    EmbeddingCategorizer,
+)
+from texttools.tools.categorizer.llm.openai_categorizer import LLMCategorizer
+from texttools.tools.question_detector.llm_detector import LLMQuestionDetector
+from texttools.tools.summarizer import LLMSummarizer
+
+__all__ = [
+    "LLMQuestionDetector",
+    "NoOpResultHandler",
+    "PrintResultHandler",
+    "ResultHandler",
+    "SaveToFileResultHandler",
+    "EmbeddingCategorizer",
+    "LLMCategorizer",
+    "SimpleBatchManager",
+    "BatchJobRunner",
+    "LLMSummarizer",
+]

hamtaa_texttools-0.1.43/texttools/base/__init__.py

@@ -0,0 +1,3 @@
+from texttools.base.base_categorizer import BaseCategorizer
+from texttools.base.base_question_detector import BaseQuestionDetector
+from texttools.base.base_summarizer import BaseSummarizer

hamtaa_texttools-0.1.43/texttools/base/base_categorizer.py

@@ -0,0 +1,40 @@
+import logging
+from abc import ABC, abstractmethod
+from enum import Enum
+from typing import Optional
+
+from texttools.handlers import NoOpResultHandler, ResultHandler
+
+
+class BaseCategorizer(ABC):
+    def __init__(
+        self,
+        handlers: Optional[list[ResultHandler]] = None,
+    ):
+        """
+        handlers: List of ResultHandler objects that will process results after categorization.
+        """
+        self.handlers = handlers or [NoOpResultHandler()]
+
+    @abstractmethod
+    def categorize(self, text: str) -> Enum:
+        """
+        Categorize the input text.
+        Must return one of the Enum members defined in self.categories.
+        """
+        pass
+
+    def preprocess(self, text: str) -> str:
+        """
+        Optional: Preprocess text before categorization.
+        """
+        return text
+
+    def _dispatch(self, results: dict) -> None:
+        for handler in self.handlers:
+            try:
+                handler.handle(results)
+            except Exception:
+                logging.error(
+                    f"Handler {handler.__class__.__name__} failed", exc_info=True
+                )

hamtaa_texttools-0.1.43/texttools/base/base_keyword_extractor.py

@@ -0,0 +1,35 @@
+from abc import ABC, abstractmethod
+from typing import Any, Optional
+
+
+class BaseKeywordExtractor(ABC):
+    """
+    Base class for all detectors that output a list of keywords.
+    """
+
+    def __init__(
+        self,
+        handlers: Optional[list[Any]] = None,
+    ):
+        self.handlers = handlers or []
+
+    @abstractmethod
+    def extract_keywords(self, text: str) -> list[str]:
+        """
+        Extract keywords from the input text.
+        Should return a list of strings, where each string is a keyword.
+        """
+        pass
+
+    def preprocess(self, text: str) -> str:
+        """
+        Optional text preprocessing step.
+        """
+        return text.strip()
+
+    def _dispatch(self, result: dict) -> None:
+        """
+        Dispatch the result to handlers.
+        """
+        for handler in self.handlers:
+            handler.handle(result)

hamtaa_texttools-0.1.43/texttools/base/base_ner_extractor.py

@@ -0,0 +1,61 @@
+import logging
+from abc import ABC, abstractmethod
+from typing import Any, Optional
+
+
+class BaseNERExtractor(ABC):
+    """
+    Base class for all Named Entity Recognition (NER) systems.
+    """
+
+    def __init__(self, handlers: Optional[list[Any]] = None):
+        """
+        Initializes the BaseNERExtractor with optional result handlers.
+
+        :param handlers: Optional list of handlers to process the NER results.
+        """
+        self.handlers = handlers or []
+
+    @abstractmethod
+    def extract_entities(self, text: str) -> list[dict[str, str]]:
+        """
+        Extracts named entities from the input text.
+
+        :param text: The text from which to extract entities.
+        :return: A list of dictionaries, where each dictionary represents an entity
+                 and typically includes 'text' and 'type' keys (e.g.,
+                 [{"text": "John Doe", "type": "PERSON"}, ...]).
+        """
+        pass
+
+    def preprocess(self, text: str) -> str:
+        """
+        Optional: Preprocess the input text before entity extraction.
+
+        :param text: Raw input text.
+        :return: Preprocessed text.
+        """
+        return text.strip()
+
+    def _dispatch(
+        self, entities: list[dict[str, str]], original_text: Optional[str] = None
+    ) -> None:
+        """
+        Sends the extracted entities to any registered result handlers.
+
+        :param entities: The list of extracted entities.
+        :param original_text: Optionally pass the original text.
+        """
+        result_data = {
+            "entities": entities,
+        }
+        if original_text is not None:
+            result_data["original_text"] = original_text
+
+        for handler in self.handlers:
+            try:
+                handler.handle(result_data)
+            except Exception:
+                logging.error(
+                    f"Handler {handler.__class__.__name__} failed", exc_info=True
+                )

hamtaa_texttools-0.1.43/texttools/base/base_question_detector.py

@@ -0,0 +1,35 @@
+from abc import ABC, abstractmethod
+from typing import Any, Optional
+
+
+class BaseQuestionDetector(ABC):
+    """
+    Base class for all detectors that output a boolean (True/False).
+    """
+
+    def __init__(
+        self,
+        handlers: Optional[list[Any]] = None,
+    ):
+        self.handlers = handlers or []
+
+    @abstractmethod
+    def detect(self, text: str) -> bool:
+        """
+        Detect if the input text meets the condition.
+        Should return True or False.
+        """
+        pass
+
+    def preprocess(self, text: str) -> str:
+        """
+        Optional text preprocessing step.
+        """
+        return text.strip()
+
+    def _dispatch(self, result: dict) -> None:
+        """
+        Dispatch the result to handlers.
+        """
+        for handler in self.handlers:
+            handler.handle(result)

hamtaa_texttools-0.1.43/texttools/base/base_question_generator.py

@@ -0,0 +1,99 @@
+import logging
+from abc import ABC, abstractmethod
+from typing import Any, Optional
+
+
+class BaseQuestionGenerator(ABC):
+    """
+    Base class for all systems that generate a question from a given answer.
+    """
+
+    def __init__(self, handlers: Optional[list[Any]] = None):
+        """
+        Initializes the BaseQuestionGenerator with optional result handlers.
+
+        :param handlers: Optional list of handlers to process the generation results.
+        """
+        self.handlers = handlers or []
+
+    @abstractmethod
+    def generate_question(self, answer: str) -> str:
+        """
+        Generates an appropriate question for the provided answer.
+
+        :param answer: The answer string for which a question needs to be generated.
+        :return: The generated question string.
+        """
+        pass
+
+    def preprocess(self, text: str) -> str:
+        """
+        Optional: Preprocess the input answer text before question generation.
+
+        :param text: Raw input answer text.
+        :return: Preprocessed text.
+        """
+        return text.strip()
+
+    def _dispatch(self, result_data: dict) -> None:
+        """
+        Sends the generated question and original answer to any registered result handlers.
+
+        :param result_data: A dictionary containing the results (e.g., {"original_answer": ..., "generated_question": ...}).
+        """
+        for handler in self.handlers:
+            try:
+                handler.handle(result_data)
+            except Exception:
+                logging.error(
+                    f"Handler {handler.__class__.__name__} failed", exc_info=True
+                )
+
+
+class BaseQuestionGeneratorFromSubject(ABC):
+    """
+    Base class for all systems that generate a question from a given subject
+    it will curate some number of questions
+
+    """
+
+    def __init__(self, handlers: Optional[list[Any]] = None):
+        """
+        Initializes the BaseQuestionGeneratorFromSubject with optional result handlers.
+
+        :param handlers: Optional list of handlers to process the generation results.
+        """
+        self.handlers = handlers or []
+
+    @abstractmethod
+    def generate_question(self, subject: str) -> str:
+        """
+        Generates an appropriate question for the provided answer.
+
+        :param answer: The answer string for which a question needs to be generated.
+        :return: The generated question string.
+        """
+        pass
+
+    def preprocess(self, text: str) -> str:
+        """
+        Optional: Preprocess the input answer text before question generation.
+
+        :param text: Raw input answer text.
+        :return: Preprocessed text.
+        """
+        return text.strip()
+
+    def _dispatch(self, result_data: dict) -> None:
+        """
+        Sends the generated question and original answer to any registered result handlers.
+
+        :param result_data: A dictionary containing the results (e.g., {"original_answer": ..., "generated_question": ...}).
+        """
+        for handler in self.handlers:
+            try:
+                handler.handle(result_data)
+            except Exception:
+                logging.error(
+                    f"Handler {handler.__class__.__name__} failed", exc_info=True
+                )