dataset-preprocessing-agent 0.1.0__tar.gz

dataset_preprocessing_agent-0.1.0/LICENCE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Ton Nom
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dataset_preprocessing_agent-0.1.0/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.4
+Name: dataset-preprocessing-agent
+Version: 0.1.0
+Summary: LLM-based automated dataset standardization and evaluation framework
+License: MIT License
+        
+        Copyright (c) 2025 Ton Nom
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENCE
+Requires-Dist: datasets>=3.6.0
+Requires-Dist: unitxt
+Requires-Dist: transformers
+Requires-Dist: torch
+Requires-Dist: accelerate
+Requires-Dist: sentence-transformers
+Requires-Dist: openai
+Requires-Dist: pandas
+Requires-Dist: tasksource
+Provides-Extra: notebook
+Requires-Dist: seaborn; extra == "notebook"
+Requires-Dist: matplotlib; extra == "notebook"
+Requires-Dist: ipywidgets; extra == "notebook"
+Dynamic: license-file
+
+# dataset-preprocessing-agent
+
+**Automated dataset standardization using LLM agents.**
+
+Every HuggingFace dataset has a unique schema (`tweet_text`, `review_body`, `sentence1`, …), making it hard to reuse models across tasks without manual mapping work. This library automates that step: given a raw dataset, an LLM inspects a small sample and produces a JSON mapping of raw column names to a canonical schema, evaluated against [Unitxt](https://github.com/IBM/unitxt) and [tasksource](https://github.com/sileod/tasksource) ground truths.
+
+---
+
+## Installation
+
+```bash
+pip install dataset-preprocessing-agent
+```
+
+For notebook visualization support:
+
+```bash
+pip install "dataset-preprocessing-agent[notebook]"
+```
+
+**Python 3.10+ required.**
+
+For the API backend, export your OpenRouter key:
+
+```bash
+export OPENROUTER_API_KEY="your_key_here"
+```
+
+---
+
+## Quick Start
+
+```python
+from dataset_preprocessing_agent.standardize_api import load_standardized_dataset
+
+result = load_standardized_dataset("glue", config="sst2")
+print(result["mapping"])
+# {"task": "classification", "text": "sentence", "label": "label"}
+```
+
+### Evaluate against Unitxt ground truth
+
+```python
+from dataset_preprocessing_agent.eval import evaluate
+
+result = evaluate(hf_name="glue", hf_config="sst2", card_id="sst2")
+print(result["score"])        
+print(result["gt_cols"])      # e.g. ['label', 'sentence']
+print(result["pred_cols"])    # e.g. ['label', 'sentence']
+```
+
+### Evaluate against tasksource ground truth
+
+```python
+from dataset_preprocessing_agent.eval_ts import evaluate_ts
+
+result = evaluate_ts("glue", "rte")
+print(result["score"])
+print(result["ts_gt"])        # GT mapping from tasksource preprocessing
+```
+
+---
+
+## Architecture
+
+The pipeline runs in three stages:
+
+1. **Standardization** — an LLM inspects 5–10 raw samples and outputs a JSON mapping of raw column names to canonical fields (`task`, `text` / `text_a` + `text_b`, `label`).
+2. **Mapping application** — `apply_llm_mapping` renames columns and converts integer labels to class name strings.
+3. **Evaluation** — the predicted raw column set is compared to the ground-truth column set using Jaccard similarity on raw HuggingFace column names.
+
+### Backends
+
+| Module | Backend |
+|--------|---------|
+| `standardize_api` | Cloud LLM via OpenRouter API |
+| `standardize_local` | Local HuggingFace model |
+
+### Baselines
+
+| Baseline | Method |
+|----------|--------|
+| `baseline_keyword_match` | Synonym dictionary matching |
+| `baseline_embedding_match` | Cosine similarity via `all-MiniLM-L6-v2` |
+
+### Evaluation backends
+
+| Module | Ground truth |
+|--------|-------------|
+| `eval` | Unitxt task cards |
+| `eval_ts` | tasksource preprocessing objects |
+
+---
+
+## Dependencies
+
+| Package | Purpose |
+|---------|---------|
+| `unitxt` | Ground-truth task cards |
+| `tasksource` | Ground-truth preprocessing objects |
+| `datasets` | HuggingFace dataset loading |
+| `transformers` + `torch` + `accelerate` | Local model inference |
+| `openai` | OpenRouter API client |
+| `sentence-transformers` | Embedding baseline |
+| `pandas` | Result DataFrames |

dataset_preprocessing_agent-0.1.0/README.md

@@ -0,0 +1,105 @@
+# dataset-preprocessing-agent
+
+**Automated dataset standardization using LLM agents.**
+
+Every HuggingFace dataset has a unique schema (`tweet_text`, `review_body`, `sentence1`, …), making it hard to reuse models across tasks without manual mapping work. This library automates that step: given a raw dataset, an LLM inspects a small sample and produces a JSON mapping of raw column names to a canonical schema, evaluated against [Unitxt](https://github.com/IBM/unitxt) and [tasksource](https://github.com/sileod/tasksource) ground truths.
+
+---
+
+## Installation
+
+```bash
+pip install dataset-preprocessing-agent
+```
+
+For notebook visualization support:
+
+```bash
+pip install "dataset-preprocessing-agent[notebook]"
+```
+
+**Python 3.10+ required.**
+
+For the API backend, export your OpenRouter key:
+
+```bash
+export OPENROUTER_API_KEY="your_key_here"
+```
+
+---
+
+## Quick Start
+
+```python
+from dataset_preprocessing_agent.standardize_api import load_standardized_dataset
+
+result = load_standardized_dataset("glue", config="sst2")
+print(result["mapping"])
+# {"task": "classification", "text": "sentence", "label": "label"}
+```
+
+### Evaluate against Unitxt ground truth
+
+```python
+from dataset_preprocessing_agent.eval import evaluate
+
+result = evaluate(hf_name="glue", hf_config="sst2", card_id="sst2")
+print(result["score"])        
+print(result["gt_cols"])      # e.g. ['label', 'sentence']
+print(result["pred_cols"])    # e.g. ['label', 'sentence']
+```
+
+### Evaluate against tasksource ground truth
+
+```python
+from dataset_preprocessing_agent.eval_ts import evaluate_ts
+
+result = evaluate_ts("glue", "rte")
+print(result["score"])
+print(result["ts_gt"])        # GT mapping from tasksource preprocessing
+```
+
+---
+
+## Architecture
+
+The pipeline runs in three stages:
+
+1. **Standardization** — an LLM inspects 5–10 raw samples and outputs a JSON mapping of raw column names to canonical fields (`task`, `text` / `text_a` + `text_b`, `label`).
+2. **Mapping application** — `apply_llm_mapping` renames columns and converts integer labels to class name strings.
+3. **Evaluation** — the predicted raw column set is compared to the ground-truth column set using Jaccard similarity on raw HuggingFace column names.
+
+### Backends
+
+| Module | Backend |
+|--------|---------|
+| `standardize_api` | Cloud LLM via OpenRouter API |
+| `standardize_local` | Local HuggingFace model |
+
+### Baselines
+
+| Baseline | Method |
+|----------|--------|
+| `baseline_keyword_match` | Synonym dictionary matching |
+| `baseline_embedding_match` | Cosine similarity via `all-MiniLM-L6-v2` |
+
+### Evaluation backends
+
+| Module | Ground truth |
+|--------|-------------|
+| `eval` | Unitxt task cards |
+| `eval_ts` | tasksource preprocessing objects |
+
+---
+
+## Dependencies
+
+| Package | Purpose |
+|---------|---------|
+| `unitxt` | Ground-truth task cards |
+| `tasksource` | Ground-truth preprocessing objects |
+| `datasets` | HuggingFace dataset loading |
+| `transformers` + `torch` + `accelerate` | Local model inference |
+| `openai` | OpenRouter API client |
+| `sentence-transformers` | Embedding baseline |
+| `pandas` | Result DataFrames |

dataset_preprocessing_agent-0.1.0/pyproject.toml

@@ -0,0 +1,41 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "dataset-preprocessing-agent"
+version = "0.1.0"
+description = "LLM-based automated dataset standardization and evaluation framework"
+readme = "README.md"
+license = { file = "LICENCE" }
+requires-python = ">=3.10"
+dependencies = [
+    "datasets>=3.6.0",
+    "unitxt",
+    "transformers",
+    "torch",
+    "accelerate",
+    "sentence-transformers",
+    "openai",
+    "pandas",
+    "tasksource",
+]
+
+[project.optional-dependencies]
+notebook = [
+    "seaborn",
+    "matplotlib",
+    "ipywidgets",
+]
+
+[tool.setuptools]
+include-package-data = false
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["dataset_preprocessing_agent*"]
+exclude = ["notebooks*", "results*", "tests*"]
+
+[tool.pyright]
+pythonVersion = "3.10"
+extraPaths = ["src"]

dataset_preprocessing_agent-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

dataset_preprocessing_agent-0.1.0/src/dataset_preprocessing_agent/__init__.py

@@ -0,0 +1,14 @@
+from .standardize_api import load_standardized_dataset
+from .standardize_local import load_standardized_dataset_local
+from .eval import evaluate
+from .eval_ts import evaluate_ts
+from .baselines import baseline_keyword_match, baseline_embedding_match
+
+__all__ = [
+    "load_standardized_dataset",
+    "load_standardized_dataset_local",
+    "evaluate",
+    "evaluate_ts",
+    "baseline_keyword_match",
+    "baseline_embedding_match",
+]

dataset_preprocessing_agent-0.1.0/src/dataset_preprocessing_agent/baselines.py

@@ -0,0 +1,174 @@
+"""
+Baseline methods for dataset column mapping.
+Compares against LLM-based standardization approach.
+"""
+import re
+from datasets import load_dataset
+from sentence_transformers import SentenceTransformer, util
+from .utils import generate_code
+
+_FALLBACK_SPLITS = ["train", "test", "validation"]
+
+
+def _load_split(name: str, config: str | None) -> object:
+    """Load a HuggingFace dataset trying train → test → validation.
+
+    If config is None and the dataset requires one, the first suggested
+    config is extracted from the error message and retried automatically.
+    """
+    last_err = None
+    for split in _FALLBACK_SPLITS:
+        try:
+            return load_dataset(name, config, split=split, streaming=True) if config else \
+                   load_dataset(name, split=split, streaming=True)
+        except Exception as e:
+            err_str = str(e)
+            if config is None and "Config name is missing" in err_str:
+                candidates = [c for c in re.findall(r"'([^']+)'", err_str) if c != name]
+                if candidates:
+                    try:
+                        return load_dataset(name, candidates[0], split=split, streaming=True)
+                    except Exception as e2:
+                        last_err = e2
+                else:
+                    last_err = e
+                    break
+            else:
+                last_err = e
+    raise ValueError(f"No accessible split for {name}/{config}: {last_err}")
+
+
+FIELD_SYNONYMS = {
+    "text": ["text", "sentence", "review", "body", "content", "tweet", "document"],
+    "label": ["label", "target", "score", "class", "category", "sentiment"],
+    "text_a": ["text_a", "premise", "sentence1", "context", "source","question", "query", "prompt"],
+    "text_b": ["text_b", "hypothesis", "sentence2", "response", "answer", "response", "output"],
+}
+
+STANDARD_FIELDS = ["text", "label", "text_a", "text_b", "question", "answer", "input", "output"]
+
+_embedding_model = None
+
+
+def _get_embedding_model():
+    """
+    Lazy load sentence transformer model.
+
+    Returns:
+        SentenceTransformer model instance.
+    """
+    global _embedding_model
+    if _embedding_model is None:
+        _embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
+    return _embedding_model
+
+
+def _infer_task(mapping: dict) -> str:
+    """
+    Infer task type from mapping keys.
+
+    Args:
+        mapping: Dictionary mapping standard field names to source column names.
+
+    Returns:
+        Inferred task type as string (nli, qa, generation, or classification).
+    """
+    keys = set(mapping.keys())
+    if keys & {"text_a", "text_b", "premise", "hypothesis"}:
+        return "nli"
+    if keys & {"question", "answer"}:
+        return "qa"
+    if keys & {"input", "output"}:
+        return "generation"
+    return "classification"
+
+
+def baseline_keyword_match(dataset, config: str = None) -> dict:
+    """
+    Rule-based keyword matching baseline.
+
+    Maps columns to Unitxt fields using synonym dictionaries.
+
+    Args:
+        dataset: Dataset name (str) or dataset object.
+        config: Optional dataset configuration name.
+
+    Returns:
+        Dictionary containing mapping, code, score, and dataset.
+    """
+    if isinstance(dataset, str):
+        ds = _load_split(dataset, config)
+    else:
+        ds = dataset
+
+    features = ds.features
+    if features is None:
+        sample = next(iter(ds))
+        features = {k: type(v).__name__ for k, v in sample.items()}
+    columns = set(features.keys())
+    mapping = {}
+
+    for field, synonyms in FIELD_SYNONYMS.items():
+        for col in columns:
+            if col.lower() in synonyms:
+                mapping[field] = col
+                break
+    
+    mapping["task"] = _infer_task(mapping)
+    
+    return {
+        "mapping": mapping,
+        "code": generate_code(mapping),
+        "score": 0.0,
+        "dataset": ds,
+    }
+
+
+def baseline_embedding_match(dataset, config: str = None, threshold: float = 0.6) -> dict:
+    """
+    Semantic similarity baseline using sentence-transformers.
+
+    Matches columns to Unitxt fields via cosine similarity.
+
+    Args:
+        dataset: Dataset name (str) or dataset object.
+        config: Optional dataset configuration name.
+        threshold: Minimum cosine similarity threshold for matching.
+
+    Returns:
+        Dictionary containing mapping, code, score, and dataset.
+    """
+    if isinstance(dataset, str):
+        ds = _load_split(dataset, config)
+    else:
+        ds = dataset
+
+    features = ds.features
+    if features is None:
+        sample = next(iter(ds))
+        features = {k: type(v).__name__ for k, v in sample.items()}
+    columns = list(features.keys())
+    model = _get_embedding_model()
+
+    col_embeddings = model.encode(columns, convert_to_tensor=True)
+    field_embeddings = model.encode(STANDARD_FIELDS, convert_to_tensor=True)
+
+    similarities = util.cos_sim(field_embeddings, col_embeddings)
+    
+    mapping = {}
+    for i, field in enumerate(STANDARD_FIELDS):
+        best_idx = similarities[i].argmax().item()
+        best_score = similarities[i][best_idx].item()
+        if best_score >= threshold:
+            mapping[field] = columns[best_idx]
+    
+    mapping["task"] = _infer_task(mapping)
+    
+    return {
+        "mapping": mapping,
+        "code": generate_code(mapping),
+        "score": 0.0,
+        "dataset": ds,
+    }
+
+