txt2phrases 1.0.1__tar.gz

txt2phrases-1.0.1/PKG-INFO

@@ -0,0 +1,223 @@
+Metadata-Version: 2.4
+Name: txt2phrases
+Version: 1.0.1
+Summary: A comprehensive library for text processing, keyword extraction, and classification from PDF and HTML documents
+Home-page: https://github.com/semanticClimate/encyclopedia/tree/main/txt2phrases
+Author: Udita Agarwal
+Author-email: udita20agarwal@gmail.com
+Keywords: text-processing,keyword-extraction,pdf,html,nlp,tf-idf,classification
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Text Processing
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.7
+Description-Content-Type: text/markdown
+Requires-Dist: beautifulsoup4>=4.9.0
+Requires-Dist: transformers>=4.0.0
+Requires-Dist: pandas>=1.0.0
+Requires-Dist: tqdm>=4.50.0
+Requires-Dist: PyPDF2>=2.0.0
+Requires-Dist: scikit-learn>=1.0.0
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: keywords
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# txt2phrases
+
+`txt2phrases` is a Python library and CLI tool designed for processing and analyzing text data.  
+It provides a streamlined pipeline for converting documents (HTML, PDF) into plain text, extracting keywords using AI models, and classifying keywords into specific and general categories using TF-IDF.
+
+---
+
+## ✨ Features
+
+### 1. **PDF to Text Conversion**
+- Extract plain text from PDF files for further processing.
+
+### 2. **HTML to Text Conversion**
+- Convert HTML documents into clean, plain text.
+
+### 3. **AI-Powered Keyword Extraction**
+- Use advanced NLP models (e.g., Hugging Face Transformers) to extract and rank the most important keywords from text files.
+
+### 4. **Automated Pipeline**
+- Run the entire pipeline (PDF/HTML → TXT → Keywords) with a single command.
+
+### 5. **Batch Processing**
+- Process single files or entire directories efficiently.
+
+### 6. **Configurable Parameters**
+- Customize thresholds, batch sizes, and output formats to suit your needs.
+
+---
+
+## 🧩 Installation
+
+Install `txt2phrases` directly from PyPI:
+
+```bash
+pip install txt2phrases
+```
+
+---
+
+## 🚀 Quick Start
+
+```bash
+# Convert PDF to text
+txt2phrases pdf2txt -i document.pdf -o output_folder
+
+# Convert HTML to text
+txt2phrases html2txt -i webpage.html -o output_folder
+
+# Extract keywords from text files
+txt2phrases keyphrases -i text_files/ -o keywords/ -n 500
+
+# Run complete pipeline
+txt2phrases auto -i pygetpapers_output/ -o results/ -n 100
+```
+
+---
+
+## 🐍 Python API
+
+```python
+from txt2phrases import (
+    convert_pdf_to_text,
+    convert_html_to_text, 
+    KeywordExtraction,
+    classify_keywords_split_files
+)
+
+# Convert PDF to text
+txt_path = convert_pdf_to_text("document.pdf", "output_folder")
+
+# Extract keywords
+extractor = KeywordExtraction(
+    input_path="text_files/",
+    output_folder="keywords/",
+    top_n=1000
+)
+extractor.extract()
+```
+---
+
+## 🧠 CLI Commands
+
+### 📄 `pdf2txt`
+Convert PDF files to text format.
+
+```bash
+txt2phrases pdf2txt -i input.pdf -o output_folder
+txt2phrases pdf2txt -i pdfs_directory/ -o text_output/
+```
+
+---
+
+### 🌐 `html2txt`
+Convert HTML files to clean text format.
+
+```bash
+txt2phrases html2txt -i webpage.html -o output_folder
+txt2phrases html2txt -i html_directory/ -o text_output/
+```
+
+---
+
+### 🔑 `keyphrases`
+Extract keyphrases from text files using advanced NLP models.
+
+```bash
+txt2phrases keyphrases -i text.txt -o keywords/ -n 500
+txt2phrases keyphrases -i text_directory/ -o keywords/ -n 1000
+```
+
+---
+
+### ⚙️ `auto`
+Complete processing pipeline for PyGetPapers output or PDF directories.
+
+```bash
+txt2phrases auto -i pygetpapers_output/ -o results/ -n 200
+txt2phrases auto -i pdf_collection/ -o results/ -n 100
+```
+
+---
+
+## 🔍 Advanced Features
+---
+
+### 2. **Complete Research Pipeline**
+
+```bash
+# Download papers with PyGetPapers
+pygetpapers -q "machine learning" -o papers/ -k 100
+
+# Process and analyze  
+txt2phrases auto -i papers/ -o analysis/ -n 200
+
+# Classify results
+python -c "
+from txt2phrases import classify_keywords_split_files
+classify_keywords_split_files('analysis/', 'classified/', threshold=0.7)
+"
+```
+
+---
+
+## 📦 Output Formats
+
+- **Text Conversion:** `.txt` files with extracted text  
+- **Keyword Extraction:** `.csv` files containing keyword and count columns  
+
+---
+
+## 🧱 Requirements
+
+To use `txt2phrases`, ensure you have the following installed:
+
+- **Python 3.8+**
+- **Dependencies:**
+  - `argparse`: For CLI argument parsing  
+  - `beautifulsoup4`: For HTML parsing  
+  - `pandas`: For data manipulation and CSV export  
+  - `tqdm`: For progress bars during batch processing  
+  - `transformers`: For AI-powered keyword extraction  
+  - `scikit-learn`: For TF-IDF-based keyword classification  
+  - `torch`: For running NLP models  
+
+Install dependencies with:
+
+```bash
+pip install -r requirements.txt
+```
+
+---
+
+## 📚 Documentation
+
+For full documentation and examples, visit the [GitHub repository](https://github.com/semanticClimate/encyclopedia/tree/main/txt2phrases).
+
+---
+
+## 📄 License
+
+This project is licensed under the **MIT License** — see the `LICENSE` file for details.
+

txt2phrases-1.0.1/README.md

@@ -0,0 +1,182 @@
+# txt2phrases
+
+`txt2phrases` is a Python library and CLI tool designed for processing and analyzing text data.  
+It provides a streamlined pipeline for converting documents (HTML, PDF) into plain text, extracting keywords using AI models, and classifying keywords into specific and general categories using TF-IDF.
+
+---
+
+## ✨ Features
+
+### 1. **PDF to Text Conversion**
+- Extract plain text from PDF files for further processing.
+
+### 2. **HTML to Text Conversion**
+- Convert HTML documents into clean, plain text.
+
+### 3. **AI-Powered Keyword Extraction**
+- Use advanced NLP models (e.g., Hugging Face Transformers) to extract and rank the most important keywords from text files.
+
+### 4. **Automated Pipeline**
+- Run the entire pipeline (PDF/HTML → TXT → Keywords) with a single command.
+
+### 5. **Batch Processing**
+- Process single files or entire directories efficiently.
+
+### 6. **Configurable Parameters**
+- Customize thresholds, batch sizes, and output formats to suit your needs.
+
+---
+
+## 🧩 Installation
+
+Install `txt2phrases` directly from PyPI:
+
+```bash
+pip install txt2phrases
+```
+
+---
+
+## 🚀 Quick Start
+
+```bash
+# Convert PDF to text
+txt2phrases pdf2txt -i document.pdf -o output_folder
+
+# Convert HTML to text
+txt2phrases html2txt -i webpage.html -o output_folder
+
+# Extract keywords from text files
+txt2phrases keyphrases -i text_files/ -o keywords/ -n 500
+
+# Run complete pipeline
+txt2phrases auto -i pygetpapers_output/ -o results/ -n 100
+```
+
+---
+
+## 🐍 Python API
+
+```python
+from txt2phrases import (
+    convert_pdf_to_text,
+    convert_html_to_text, 
+    KeywordExtraction,
+    classify_keywords_split_files
+)
+
+# Convert PDF to text
+txt_path = convert_pdf_to_text("document.pdf", "output_folder")
+
+# Extract keywords
+extractor = KeywordExtraction(
+    input_path="text_files/",
+    output_folder="keywords/",
+    top_n=1000
+)
+extractor.extract()
+```
+---
+
+## 🧠 CLI Commands
+
+### 📄 `pdf2txt`
+Convert PDF files to text format.
+
+```bash
+txt2phrases pdf2txt -i input.pdf -o output_folder
+txt2phrases pdf2txt -i pdfs_directory/ -o text_output/
+```
+
+---
+
+### 🌐 `html2txt`
+Convert HTML files to clean text format.
+
+```bash
+txt2phrases html2txt -i webpage.html -o output_folder
+txt2phrases html2txt -i html_directory/ -o text_output/
+```
+
+---
+
+### 🔑 `keyphrases`
+Extract keyphrases from text files using advanced NLP models.
+
+```bash
+txt2phrases keyphrases -i text.txt -o keywords/ -n 500
+txt2phrases keyphrases -i text_directory/ -o keywords/ -n 1000
+```
+
+---
+
+### ⚙️ `auto`
+Complete processing pipeline for PyGetPapers output or PDF directories.
+
+```bash
+txt2phrases auto -i pygetpapers_output/ -o results/ -n 200
+txt2phrases auto -i pdf_collection/ -o results/ -n 100
+```
+
+---
+
+## 🔍 Advanced Features
+---
+
+### 2. **Complete Research Pipeline**
+
+```bash
+# Download papers with PyGetPapers
+pygetpapers -q "machine learning" -o papers/ -k 100
+
+# Process and analyze  
+txt2phrases auto -i papers/ -o analysis/ -n 200
+
+# Classify results
+python -c "
+from txt2phrases import classify_keywords_split_files
+classify_keywords_split_files('analysis/', 'classified/', threshold=0.7)
+"
+```
+
+---
+
+## 📦 Output Formats
+
+- **Text Conversion:** `.txt` files with extracted text  
+- **Keyword Extraction:** `.csv` files containing keyword and count columns  
+
+---
+
+## 🧱 Requirements
+
+To use `txt2phrases`, ensure you have the following installed:
+
+- **Python 3.8+**
+- **Dependencies:**
+  - `argparse`: For CLI argument parsing  
+  - `beautifulsoup4`: For HTML parsing  
+  - `pandas`: For data manipulation and CSV export  
+  - `tqdm`: For progress bars during batch processing  
+  - `transformers`: For AI-powered keyword extraction  
+  - `scikit-learn`: For TF-IDF-based keyword classification  
+  - `torch`: For running NLP models  
+
+Install dependencies with:
+
+```bash
+pip install -r requirements.txt
+```
+
+---
+
+## 📚 Documentation
+
+For full documentation and examples, visit the [GitHub repository](https://github.com/semanticClimate/encyclopedia/tree/main/txt2phrases).
+
+---
+
+## 📄 License
+
+This project is licensed under the **MIT License** — see the `LICENSE` file for details.
+

txt2phrases-1.0.1/pyproject.toml

@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel", "setuptools_scm"]
+build-backend = "setuptools.build_meta"

txt2phrases-1.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

txt2phrases-1.0.1/setup.py

@@ -0,0 +1,47 @@
+from setuptools import setup, find_packages
+
+with open("README.md", "r", encoding="utf-8") as fh:
+    long_description = fh.read()
+
+setup(
+    name="txt2phrases",
+    version="1.0.1",
+    author="Udita Agarwal",
+    author_email="udita20agarwal@gmail.com",
+    description="A comprehensive library for text processing, keyword extraction, and classification from PDF and HTML documents",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/semanticClimate/encyclopedia/tree/main/txt2phrases",
+    packages=find_packages(),
+    classifiers=[
+        "Development Status :: 4 - Beta",
+        "Intended Audience :: Science/Research",
+        "Intended Audience :: Developers",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
+        "Topic :: Text Processing",
+        "Topic :: Scientific/Engineering :: Information Analysis",
+        "Topic :: Software Development :: Libraries :: Python Modules",
+    ],
+    python_requires=">=3.7",
+    install_requires=[
+        "beautifulsoup4>=4.9.0",
+        "transformers>=4.0.0",
+        "pandas>=1.0.0",
+        "tqdm>=4.50.0",
+        "PyPDF2>=2.0.0",
+        "scikit-learn>=1.0.0",
+    ],
+    entry_points={
+        "console_scripts": [
+            "txt2phrases=txt2phrases.cli:main",
+        ],
+    },
+    keywords="text-processing, keyword-extraction, pdf, html, nlp, tf-idf, classification",
+)

txt2phrases-1.0.1/txt2phrases/__init__.py

@@ -0,0 +1,32 @@
+# __init__.py
+"""
+txt2phrases: A library for text processing, keyword extraction, and classification.
+"""
+
+__version__ = "1.0.0"
+__author__ = "Your Name"
+
+try:
+    from .html2txt import convert_html_to_text
+    from .pdf2txt import convert_pdf_to_text
+    from .keyword import KeywordExtraction
+    from .classify_specific import classify_keywords_split_files
+    from .pygetpaper import main as pygetpaper_main
+except ImportError:
+    # Handle the case when running as standalone scripts
+    from html2txt import convert_html_to_text
+    from pdf2txt import convert_pdf_to_text
+    from keyword import KeywordExtraction
+    from classify_specific import classify_keywords_split_files
+    from pygetpaper import main as pygetpaper_main
+
+__all__ = [
+    "convert_html_to_text",
+    "convert_pdf_to_text",
+    "KeywordExtraction",
+    "classify_keywords_split_files",
+    "pygetpaper_main"
+]
+
+
+

txt2phrases-1.0.1/txt2phrases/classify_specific.py

@@ -0,0 +1,143 @@
+import os
+import pandas as pd
+from collections import defaultdict
+from argparse import ArgumentParser
+from sklearn.feature_extraction.text import TfidfTransformer
+
+
+def classify_keywords_split_files(input_dir, output_dir, threshold=0.6, min_freq=5):
+    """
+    For each chapter, create a separate CSV listing keywords that are 'specific' to it
+    according to TF-IDF >= threshold. Also, create a single CSV with general/specific chapters.
+
+    Input: multiple chapter CSVs in `input_dir`, each with columns: keyword, count
+    Output:
+        - <chapter>_specific_keywords.csv in `output_dir`
+        - general_specific_keywords.csv in `output_dir`
+    """
+    os.makedirs(output_dir, exist_ok=True)
+
+    # --- Load per-chapter frequencies (with min_freq filtering) ---
+    chapter_files = [f for f in os.listdir(input_dir) if f.endswith(".csv")]
+    if not chapter_files:
+        print(f"No CSV files found in {input_dir}")
+        return
+
+    keyword_chapter_freq = defaultdict(dict)  # {keyword: {chapter: count}}
+    chapters = []
+
+    for file in chapter_files:
+        chapter = os.path.splitext(file)[0]
+        chapters.append(chapter)
+        df = pd.read_csv(os.path.join(input_dir, file))
+        if not {"keyword", "count"}.issubset(df.columns):
+            raise ValueError(f"{file} must contain 'keyword' and 'count' columns")
+        df = df[df["count"] >= min_freq]
+        for _, row in df.iterrows():
+            keyword_chapter_freq[str(row["keyword"])][chapter] = int(row["count"])
+
+    chapters = sorted(chapters)
+    keywords = list(keyword_chapter_freq.keys())
+
+    if not keywords:
+        # create empty files if no keywords pass min_freq
+        for chapter in chapters:
+            pd.DataFrame(columns=["keyword", "tfidf", "count"]).to_csv(
+                os.path.join(output_dir, f"{chapter}_specific_keywords.csv"), index=False
+            )
+        pd.DataFrame(columns=["keyword", "General", "Specific"]).to_csv(
+            os.path.join(output_dir, "general_specific_keywords.csv"), index=False
+        )
+        print("No keywords passed min_freq. Empty files created.")
+        return
+
+    # --- Build keyword x chapter matrix ---
+    data = []
+    for kw in keywords:
+        row = [keyword_chapter_freq[kw].get(ch, 0) for ch in chapters]
+        data.append(row)
+    df_matrix = pd.DataFrame(data, index=keywords, columns=chapters)
+
+    # --- TF-IDF over (keywords x chapters) ---
+    transformer = TfidfTransformer()
+    tfidf_matrix = transformer.fit_transform(df_matrix)  # sparse
+
+    # --- For each chapter, collect specific keywords ---
+    for j, chapter in enumerate(chapters):
+        rows = []
+        for i, keyword in enumerate(df_matrix.index):
+            score = tfidf_matrix[i, j]
+            freq_in_chapter = int(df_matrix.iloc[i, j])
+            if score >= threshold and freq_in_chapter > 0:
+                rows.append((keyword, float(score), freq_in_chapter))
+        rows.sort(key=lambda x: (x[1], x[2]), reverse=True)
+        pd.DataFrame(rows, columns=["keyword", "tfidf", "count"]).to_csv(
+            os.path.join(output_dir, f"{chapter}_specific_keywords.csv"), index=False
+        )
+        print(f"Saved {chapter}_specific_keywords.csv ({len(rows)} keywords)")
+
+    # --- Generate general/specific keywords file ---
+    general_specific_rows = []
+    for i, keyword in enumerate(df_matrix.index):
+        specific_chapters = []
+        general_chapters = []
+        for j, chapter in enumerate(chapters):
+            score = tfidf_matrix[i, j]
+            if score >= threshold:
+                specific_chapters.append(chapter)
+            else:
+                general_chapters.append(chapter)
+        general_specific_rows.append((
+            keyword,
+            " ".join(general_chapters),
+            " ".join(specific_chapters)
+        ))
+
+    out_df = pd.DataFrame(general_specific_rows, columns=["keyword", "General", "Specific"])
+    out_path = os.path.join(output_dir, "general_specific_keywords.csv")
+    out_df.to_csv(out_path, index=False)
+    print(f"Saved general_specific_keywords.csv ({len(out_df)} keywords)")
+
+
+def main():
+    """
+    Command-line interface for generating per-chapter specific keywords
+    and a general/specific keywords CSV.
+    """
+    import argparse
+    from .classify_specific import classify_keywords_split_files
+    import os
+    parser = argparse.ArgumentParser(
+        description="Create per-chapter files of TF-IDF-specific keywords "
+                    "and a general/specific keywords CSV"
+    )
+    parser.add_argument(
+        "-i", "--input_dir", required=True,
+        help="Path to folder with chapter CSVs (keyword,count)"
+    )
+    parser.add_argument(
+        "-o", "--output_dir", required=True,
+        help="Path to save per-chapter specific and general/specific keyword files"
+    )
+    parser.add_argument(
+        "-t", "--threshold", type=float, default=0.6,
+        help="TF-IDF threshold for specificity (default: 0.6)"
+    )
+    parser.add_argument(
+        "-f", "--min_freq", type=int, default=5,
+        help="Minimum frequency of keywords to consider (default: 5)"
+    )
+
+    args = parser.parse_args()
+
+    classify_keywords_split_files(
+        input_dir=args.input_dir,
+        output_dir=args.output_dir,
+        threshold=args.threshold,
+        min_freq=args.min_freq
+    )
+
+if __name__ == "__main__":
+    main()
+
+