suur-data 1.0.0__tar.gz

suur_data-1.0.0/PKG-INFO

@@ -0,0 +1,209 @@
+Metadata-Version: 2.4
+Name: suur_data
+Version: 1.0.0
+Summary: Intelligent data ingestion and tokenization pipeline
+Author: Your Name
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Requires-Dist: scikit-learn
+Requires-Dist: numpy
+Requires-Dist: click
+Requires-Dist: chardet
+Provides-Extra: pdf
+Requires-Dist: pdfminer.six; extra == "pdf"
+Provides-Extra: docx
+Requires-Dist: python-docx; extra == "docx"
+Provides-Extra: epub
+Requires-Dist: ebooklib; extra == "epub"
+Provides-Extra: hf
+Requires-Dist: transformers; extra == "hf"
+Requires-Dist: tokenizers; extra == "hf"
+Provides-Extra: all
+Requires-Dist: pdfminer.six; extra == "all"
+Requires-Dist: python-docx; extra == "all"
+Requires-Dist: ebooklib; extra == "all"
+Requires-Dist: transformers; extra == "all"
+Requires-Dist: tokenizers; extra == "all"
+Dynamic: author
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# Suur Data
+
+**Intelligent data ingestion and tokenization pipeline.**
+
+Suur Data fetches text from any source, filters it by topic using a neural relevance scorer, then tokenizes it using either a pretrained HuggingFace tokenizer or a custom-trained BPE tokenizer.
+
+---
+
+## Installation
+
+```bash
+# Core (URLs, .txt, .csv, .json, .html)
+pip install -e .
+
+# With all optional formats + HuggingFace tokenizers
+pip install -e ".[all]"
+```
+
+---
+
+## Python API
+
+```python
+from suur_data import suur_data
+
+# Minimal — fetches URL, no filter, GPT-2 tokenizer
+tokens = suur_data("https://en.wikipedia.org/wiki/Neuroscience")
+
+# Filter by topic, custom BPE tokenizer
+tokens = suur_data(
+    "research_paper.pdf",
+    topic="quantum computing",
+    tokenizer="custom",
+    vocab_size=4000,
+    save_dir="./my_tokenizer",
+)
+
+# Local file, pretrained BERT tokenizer, strict filter
+tokens = suur_data(
+    "~/corpus/biology.txt",
+    topic="cell biology",
+    tokenizer="pretrained",
+    model="bert",
+    threshold=0.10,
+)
+
+# Skip the filter entirely
+tokens = suur_data("data.csv", no_filter=True)
+
+print(tokens[:20])   # list of integer token IDs
+print(len(tokens))   # total token count
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `data_location` | str | — | URL or local file path |
+| `topic` | str | `""` | Subject for relevance filtering (empty = skip filter) |
+| `tokenizer` | str | `"pretrained"` | `"pretrained"` or `"custom"` |
+| `model` | str | `"gpt2"` | HuggingFace model shortcut or full ID |
+| `vocab_size` | int | `8000` | BPE vocab size for custom tokenizer |
+| `threshold` | float | `0.05`` | Cosine similarity cutoff (0.0–1.0) |
+| `save_dir` | str | `None` | Path to save tokenizer files |
+| `no_filter` | bool | `False` | Skip the relevance filter |
+| `verbose` | bool | `True` | Show progress output |
+
+### Returns
+`List[int]` — flat list of integer token IDs.
+
+---
+
+## CLI
+
+```bash
+# Basic URL fetch
+suur_data fetch https://example.com/article --topic "machine learning"
+
+# PDF with custom BPE tokenizer
+suur_data fetch paper.pdf --topic "protein folding" --tokenizer custom --vocab-size 6000
+
+# Local file, pretrained BERT, save tokenizer
+suur_data fetch corpus.txt --tokenizer pretrained --model bert --save-dir ./bert_tok
+
+# Skip filter, save tokens to file
+suur_data fetch data.json --no-filter --output tokens.json
+
+# See supported models
+suur_data models
+
+# See supported file formats
+suur_data formats
+```
+
+---
+
+## Supported Input Formats
+
+| Format | Notes |
+|--------|-------|
+| `.txt`, `.md`, `.rst` | Plain text |
+| `.pdf` | Requires `pdfminer.six` |
+| `.docx` | Requires `python-docx` |
+| `.csv`, `.tsv` | All cells joined as text |
+| `.json` | Recursively flattened key-value pairs |
+| `.html`, `.htm` | Scripts/styles stripped (requires `beautifulsoup4`) |
+| `.epub` | E-books (requires `ebooklib` + `beautifulsoup4`) |
+| HTTP/HTTPS URL | Auto-downloaded, then parsed by extension |
+
+---
+
+## Pretrained Model Shortcuts
+
+| Shortcut | Model |
+|----------|-------|
+| `gpt2` | GPT-2 (OpenAI) |
+| `bert` | BERT base uncased |
+| `roberta` | RoBERTa base |
+| `distilbert` | DistilBERT base uncased |
+| `t5` | T5 small |
+
+You can also pass any HuggingFace Hub model ID directly:
+```
+--model "facebook/opt-125m"
+```
+
+---
+
+## Architecture
+
+```
+Source (URL / file)
+        │
+        ▼
+  Stage 1: Ingest
+  Handles 8 file types + HTTP download
+        │
+        ▼
+  Stage 2: Neural Filter
+  Splits into paragraph chunks
+  Scores each chunk against topic via TF-IDF cosine similarity
+  Drops chunks below threshold
+        │
+        ▼
+  Stage 3: Tokenize
+  ┌─────────────────────┐  ┌────────────────────────────┐
+  │  Pretrained mode    │  │  Custom mode               │
+  │  HuggingFace        │  │  BPE trainer (HF library   │
+  │  AutoTokenizer      │  │  or pure-Python fallback)  │
+  └─────────────────────┘  └────────────────────────────┘
+        │
+        ▼
+  List[int]  ←  token IDs
+```
+
+---
+
+## Dependency Matrix
+
+| Feature | Required packages |
+|---------|------------------|
+| Core pipeline | `requests`, `beautifulsoup4`, `scikit-learn`, `numpy`, `click`, `chardet` |
+| PDF support | `pdfminer.six` |
+| .docx support | `python-docx` |
+| .epub support | `ebooklib` |
+| Pretrained tokenizers | `transformers` |
+| Fast BPE training | `tokenizers` |
+
+All optional — the tool degrades gracefully with built-in fallbacks when optional packages are missing.
+

suur_data-1.0.0/README.md

@@ -0,0 +1,170 @@
+# Suur Data
+
+**Intelligent data ingestion and tokenization pipeline.**
+
+Suur Data fetches text from any source, filters it by topic using a neural relevance scorer, then tokenizes it using either a pretrained HuggingFace tokenizer or a custom-trained BPE tokenizer.
+
+---
+
+## Installation
+
+```bash
+# Core (URLs, .txt, .csv, .json, .html)
+pip install -e .
+
+# With all optional formats + HuggingFace tokenizers
+pip install -e ".[all]"
+```
+
+---
+
+## Python API
+
+```python
+from suur_data import suur_data
+
+# Minimal — fetches URL, no filter, GPT-2 tokenizer
+tokens = suur_data("https://en.wikipedia.org/wiki/Neuroscience")
+
+# Filter by topic, custom BPE tokenizer
+tokens = suur_data(
+    "research_paper.pdf",
+    topic="quantum computing",
+    tokenizer="custom",
+    vocab_size=4000,
+    save_dir="./my_tokenizer",
+)
+
+# Local file, pretrained BERT tokenizer, strict filter
+tokens = suur_data(
+    "~/corpus/biology.txt",
+    topic="cell biology",
+    tokenizer="pretrained",
+    model="bert",
+    threshold=0.10,
+)
+
+# Skip the filter entirely
+tokens = suur_data("data.csv", no_filter=True)
+
+print(tokens[:20])   # list of integer token IDs
+print(len(tokens))   # total token count
+```
+
+### Parameters
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `data_location` | str | — | URL or local file path |
+| `topic` | str | `""` | Subject for relevance filtering (empty = skip filter) |
+| `tokenizer` | str | `"pretrained"` | `"pretrained"` or `"custom"` |
+| `model` | str | `"gpt2"` | HuggingFace model shortcut or full ID |
+| `vocab_size` | int | `8000` | BPE vocab size for custom tokenizer |
+| `threshold` | float | `0.05`` | Cosine similarity cutoff (0.0–1.0) |
+| `save_dir` | str | `None` | Path to save tokenizer files |
+| `no_filter` | bool | `False` | Skip the relevance filter |
+| `verbose` | bool | `True` | Show progress output |
+
+### Returns
+`List[int]` — flat list of integer token IDs.
+
+---
+
+## CLI
+
+```bash
+# Basic URL fetch
+suur_data fetch https://example.com/article --topic "machine learning"
+
+# PDF with custom BPE tokenizer
+suur_data fetch paper.pdf --topic "protein folding" --tokenizer custom --vocab-size 6000
+
+# Local file, pretrained BERT, save tokenizer
+suur_data fetch corpus.txt --tokenizer pretrained --model bert --save-dir ./bert_tok
+
+# Skip filter, save tokens to file
+suur_data fetch data.json --no-filter --output tokens.json
+
+# See supported models
+suur_data models
+
+# See supported file formats
+suur_data formats
+```
+
+---
+
+## Supported Input Formats
+
+| Format | Notes |
+|--------|-------|
+| `.txt`, `.md`, `.rst` | Plain text |
+| `.pdf` | Requires `pdfminer.six` |
+| `.docx` | Requires `python-docx` |
+| `.csv`, `.tsv` | All cells joined as text |
+| `.json` | Recursively flattened key-value pairs |
+| `.html`, `.htm` | Scripts/styles stripped (requires `beautifulsoup4`) |
+| `.epub` | E-books (requires `ebooklib` + `beautifulsoup4`) |
+| HTTP/HTTPS URL | Auto-downloaded, then parsed by extension |
+
+---
+
+## Pretrained Model Shortcuts
+
+| Shortcut | Model |
+|----------|-------|
+| `gpt2` | GPT-2 (OpenAI) |
+| `bert` | BERT base uncased |
+| `roberta` | RoBERTa base |
+| `distilbert` | DistilBERT base uncased |
+| `t5` | T5 small |
+
+You can also pass any HuggingFace Hub model ID directly:
+```
+--model "facebook/opt-125m"
+```
+
+---
+
+## Architecture
+
+```
+Source (URL / file)
+        │
+        ▼
+  Stage 1: Ingest
+  Handles 8 file types + HTTP download
+        │
+        ▼
+  Stage 2: Neural Filter
+  Splits into paragraph chunks
+  Scores each chunk against topic via TF-IDF cosine similarity
+  Drops chunks below threshold
+        │
+        ▼
+  Stage 3: Tokenize
+  ┌─────────────────────┐  ┌────────────────────────────┐
+  │  Pretrained mode    │  │  Custom mode               │
+  │  HuggingFace        │  │  BPE trainer (HF library   │
+  │  AutoTokenizer      │  │  or pure-Python fallback)  │
+  └─────────────────────┘  └────────────────────────────┘
+        │
+        ▼
+  List[int]  ←  token IDs
+```
+
+---
+
+## Dependency Matrix
+
+| Feature | Required packages |
+|---------|------------------|
+| Core pipeline | `requests`, `beautifulsoup4`, `scikit-learn`, `numpy`, `click`, `chardet` |
+| PDF support | `pdfminer.six` |
+| .docx support | `python-docx` |
+| .epub support | `ebooklib` |
+| Pretrained tokenizers | `transformers` |
+| Fast BPE training | `tokenizers` |
+
+All optional — the tool degrades gracefully with built-in fallbacks when optional packages are missing.
+

suur_data-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

suur_data-1.0.0/setup.py

@@ -0,0 +1,43 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="suur_data",
+    version="1.0.0",
+    description="Intelligent data ingestion and tokenization pipeline",
+    long_description=open("README.md").read(),
+    long_description_content_type="text/markdown",
+    author="Your Name",
+    python_requires=">=3.10",
+    packages=find_packages(),
+    install_requires=[
+        "requests",
+        "beautifulsoup4",
+        "scikit-learn",
+        "numpy",
+        "click",
+        "chardet",
+    ],
+    extras_require={
+        "pdf":   ["pdfminer.six"],
+        "docx":  ["python-docx"],
+        "epub":  ["ebooklib"],
+        "hf":    ["transformers", "tokenizers"],  # pretrained + HF BPE
+        "all":   [
+            "pdfminer.six",
+            "python-docx",
+            "ebooklib",
+            "transformers",
+            "tokenizers",
+        ],
+    },
+    entry_points={
+        "console_scripts": [
+            "suur-data=suur_data:main",
+        ],
+    },
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "Topic :: Text Processing :: Linguistic",
+    ],
+)
+

suur_data-1.0.0/suur_data/__init__.py

@@ -0,0 +1,233 @@
+"""
+suur_data
+========
+A smart data ingestion and tokenization pipeline.
+
+Python API usage:
+    from suur_data import suur_data
+
+    tokens = suur_data("https://example.com/article.html")
+    tokens = suur_data("my_corpus.pdf", topic="quantum computing")
+    tokens = suur_data("data.txt", tokenizer="custom", vocab_size=4000)
+
+CLI usage:
+    suur_data fetch <source> [OPTIONS]
+
+    Options:
+      --topic TEXT            Topic/subject to filter content by
+      --tokenizer [pretrained|custom]
+      --model TEXT            Pretrained model name (default: gpt2)
+      --vocab-size INT        BPE vocab size for custom mode (default: 8000)
+      --threshold FLOAT       Relevance threshold 0.0–1.0 (default: 0.05)
+      --save-dir PATH         Where to save tokenizer artifacts
+      --no-filter             Skip the relevance filter
+      --verbose / --quiet
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+import json
+import tempfile
+from typing import List, Optional
+
+import click
+
+from .ingest import ingest
+from .filter import filter_chunks
+from .tokenizer import tokenize_pretrained, tokenize_custom
+
+
+# ---------------------------------------------------------------------------
+# Python API
+# ---------------------------------------------------------------------------
+
+def suur_data(
+    data_location: str,
+    topic: str = "",
+    tokenizer: str = "pretrained",
+    model: str = "gpt2",
+    vocab_size: int = 8000,
+    threshold: float = 0.05,
+    save_dir: Optional[str] = None,
+    no_filter: bool = False,
+    verbose: bool = True,
+) -> List[int]:
+    """
+    End-to-end pipeline: ingest → filter → tokenize.
+
+    Parameters
+    ----------
+    data_location : str
+        URL or local file path. Supports .txt, .pdf, .docx, .csv,
+        .json, .html, .htm, .epub, .md, .rst
+    topic : str
+        Subject/keyword for the neural relevance filter.
+        Leave empty to skip filtering.
+    tokenizer : str
+        "pretrained" (default) or "custom".
+    model : str
+        HuggingFace model name for pretrained mode (default: "gpt2").
+    vocab_size : int
+        BPE vocabulary size for custom mode (default: 8000).
+    threshold : float
+        Cosine similarity cutoff for relevance filter (default: 0.05).
+    save_dir : str | None
+        Directory to save tokenizer files. None = don't save.
+    no_filter : bool
+        If True, skip the relevance filter and tokenize everything.
+    verbose : bool
+        Print progress information.
+
+    Returns
+    -------
+    List[int]
+        List of integer token IDs.
+    """
+    tmp = tempfile.mkdtemp(prefix="suur_data_")
+
+    # --- Stage 1: Ingest ---
+    if verbose:
+        print(f"\n[suur_data] Stage 1 — Ingesting: {data_location}")
+    raw_text = ingest(data_location, tmp_dir=tmp)
+
+    # --- Stage 2: Filter ---
+    if no_filter or not topic.strip():
+        if verbose:
+            if no_filter:
+                print("[suur_data] Stage 2 — Filter: SKIPPED (--no-filter)")
+            else:
+                print("[suur_data] Stage 2 — Filter: SKIPPED (no topic given)")
+        filtered_text = raw_text
+    else:
+        if verbose:
+            print(f"[suur_data] Stage 2 — Neural Filter (topic: '{topic}')")
+        filtered_text, _ = filter_chunks(
+            raw_text,
+            topic=topic,
+            threshold=threshold,
+            verbose=verbose,
+        )
+
+    # --- Stage 3: Tokenize ---
+    if verbose:
+        print(f"[suur_data] Stage 3 — Tokenizing ({tokenizer} mode)")
+
+    if tokenizer == "custom":
+        tokens = tokenize_custom(filtered_text, vocab_size=vocab_size, save_dir=save_dir)
+    else:
+        tokens = tokenize_pretrained(filtered_text, model_name=model, save_dir=save_dir)
+
+    if verbose:
+        print(f"\n[suur_data] Done. Total tokens: {len(tokens):,}\n")
+
+    return tokens
+
+
+# ---------------------------------------------------------------------------
+# CLI
+# ---------------------------------------------------------------------------
+
+@click.group()
+@click.version_option("1.0.0", prog_name="suur-data")
+def cli():
+    """
+    \b
+    Suur Data — intelligent data ingestion and tokenization pipeline.
+
+    Fetch any text source, filter it by topic using TF-IDF relevance
+    scoring, then tokenize with a pretrained or custom BPE tokenizer.
+    """
+
+
+@cli.command("fetch")
+@click.argument("source")
+@click.option("--topic",      default="",           show_default=True,  help="Topic for relevance filtering.")
+@click.option("--tokenizer",  default="pretrained", show_default=True,  type=click.Choice(["pretrained", "custom"]), help="Tokenizer mode.")
+@click.option("--model",      default="gpt2",       show_default=True,  help="Pretrained model name (HuggingFace).")
+@click.option("--vocab-size", default=8000,         show_default=True,  help="BPE vocab size for custom tokenizer.")
+@click.option("--threshold",  default=0.05,         show_default=True,  help="Relevance filter threshold (0.0–1.0).")
+@click.option("--save-dir",   default=None,                             help="Directory to save tokenizer artifacts.")
+@click.option("--no-filter",  is_flag=True,         default=False,      help="Skip the relevance filter.")
+@click.option("--output",     default=None,                             help="Save token IDs to a JSON file.")
+@click.option("--quiet",      is_flag=True,         default=False,      help="Suppress progress output.")
+def fetch_cmd(source, topic, tokenizer, model, vocab_size, threshold,
+              save_dir, no_filter, output, quiet):
+    """
+    Ingest SOURCE (URL or file path), filter by topic, and tokenize.
+
+    \b
+    Examples:
+      suur_data fetch https://en.wikipedia.org/wiki/Neuroscience --topic "brain"
+      suur_data fetch corpus.pdf --topic "machine learning" --tokenizer custom
+      suur_data fetch data.txt --no-filter --tokenizer pretrained --model bert
+    """
+    tokens = suur_data(
+        data_location=source,
+        topic=topic,
+        tokenizer=tokenizer,
+        model=model,
+        vocab_size=vocab_size,
+        threshold=threshold,
+        save_dir=save_dir,
+        no_filter=no_filter,
+        verbose=not quiet,
+    )
+
+    if output:
+        with open(output, "w") as f:
+            json.dump(tokens, f)
+        click.echo(f"Tokens saved to {output}")
+    else:
+        click.echo(f"Token count: {len(tokens):,}")
+        click.echo(f"First 50 tokens: {tokens[:50]}")
+
+
+@cli.command("models")
+def models_cmd():
+    """List supported pretrained model shortcuts."""
+    rows = [
+        ("gpt2",       "GPT-2 (OpenAI)"),
+        ("bert",       "BERT base uncased"),
+        ("roberta",    "RoBERTa base"),
+        ("distilbert", "DistilBERT base uncased"),
+        ("t5",         "T5 small"),
+    ]
+    click.echo("\nSupported pretrained model shortcuts:\n")
+    for key, name in rows:
+        click.echo(f"  {key:<14} {name}")
+    click.echo("\nYou can also pass any HuggingFace model ID directly.")
+    click.echo('Example: --model "facebook/opt-125m"\n')
+
+
+@cli.command("formats")
+def formats_cmd():
+    """List supported input file formats."""
+    fmts = [
+        (".txt / .md / .rst", "Plain text, Markdown, reStructuredText"),
+        (".pdf",              "PDF documents (requires pdfminer.six)"),
+        (".docx",             "Word documents (requires python-docx)"),
+        (".csv / .tsv",       "Comma/tab-separated values"),
+        (".json",             "JSON — recursively flattens key-value pairs"),
+        (".html / .htm",      "HTML pages (requires beautifulsoup4)"),
+        (".epub",             "E-books (requires ebooklib + beautifulsoup4)"),
+        ("URL",               "Any HTTP/HTTPS URL — auto-downloaded"),
+    ]
+    click.echo("\nSupported input formats:\n")
+    for ext, desc in fmts:
+        click.echo(f"  {ext:<22} {desc}")
+    click.echo()
+
+
+# ---------------------------------------------------------------------------
+# Allow running as: python -m suur_data
+# ---------------------------------------------------------------------------
+
+def main():
+    cli()
+
+
+if __name__ == "__main__":
+    main()
+