datasetdna 0.1.2__tar.gz

datasetdna-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kavya Rajput
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datasetdna-0.1.2/PKG-INFO

@@ -0,0 +1,176 @@
+Metadata-Version: 2.4
+Name: datasetdna
+Version: 0.1.2
+Summary: One-command dataset profiling and health analysis
+Author: Kavya Rajput
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/KAVYA-29-ai/DatasetDNA
+Project-URL: Repository, https://github.com/KAVYA-29-ai/DatasetDNA
+Project-URL: Issues, https://github.com/KAVYA-29-ai/DatasetDNA/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: rich
+Requires-Dist: typer
+Requires-Dist: plotly
+Dynamic: license-file
+
+# 🧬 DatasetDNA
+
+**One command. Know if your dataset is safe to train on.**
+
+DatasetDNA profiles any CSV and produces a clear, actionable health report — catching real data-quality problems (missing values, duplicates, invalid ranges) while separating them from statistical signals that need human judgment, not automatic penalties (correlation, skew, cardinality).
+
+```bash
+pip install datasetdna
+datasetdna your_data.csv
+```
+
+That's it. No config, no notebook, no cloud upload — your data never leaves your machine.
+
+---
+
+## The problem with most profiling tools
+
+Most auto-EDA tools treat every statistical pattern as a "problem" and dock your data a few points for it. That sounds fine — until you run it on the Iris dataset, one of the most famous, cleanest datasets in all of ML, and it tells you `petal_length ↔ petal_width` correlation of `0.96` is a health issue.
+
+It isn't. That correlation is **expected botany**, not a data-quality defect. A tool that can't tell the difference between "this is broken" and "this is just how the world works" isn't trustworthy enough to run before training a real model.
+
+DatasetDNA draws a hard line between the two:
+
+| 🧬 Data Quality | 📊 Statistical Signals |
+|---|---|
+| **Affects your Health Score.** Objectively bad, in any context. | **Never affects your score.** Informational — could be fine, could be worth a look. |
+| Missing values | Strong correlations |
+| Duplicate rows | Skewness |
+| Target class imbalance | High cardinality |
+| Impossible values (age = 150, age = -5) | Outliers |
+
+This distinction is the core design decision behind the whole tool — not an afterthought.
+
+---
+
+## What it checks
+
+```
+CSV
+ ↓
+Loader + Cleaning        → handles $1,200 / "1,234" / N/A / null / "?" / "-" as one CSV, cleanly
+ ↓
+Semantic Type Detection  → tells an ID column from a real feature, a date from a category
+ ↓
+┌─────────────────────────────┐
+│ Overview        Numerical   │
+│ Schema          Categorical │
+│ Missingness     Outliers    │
+│ Duplicates      Correlations│
+│ Cardinality     Target      │
+└─────────────────────────────┘
+ ↓
+Health Scoring   → 0–100, driven only by real quality issues
+ ↓
+Console Report   → color-coded, readable, no PhD required
+```
+
+**Semantic Type Detection** is what separates this from a plain `df.describe()` script — it automatically:
+- Cleans numbers written as text (`"$55,000"`, `"1,200"`) into real numeric columns
+- Recognizes disguised missing values (`N/A`, `null`, `?`, `-`, `unknown`) as actual missing data, not valid strings
+- Flags ID-like columns (customer IDs, emails) and excludes them from correlation/association analysis, where they'd otherwise create meaningless "100% correlated" noise
+- Validates domain-plausible ranges (e.g., an `age` column with values of `-5` or `150` gets flagged as a real error, not just an "outlier")
+
+---
+
+## Example output
+
+```bash
+$ datasetdna examples/real.csv
+
+╭────────────── 🧬 Dataset Health ──────────────╮
+│ 🟠 Health Score                               │
+│                                               │
+│ 70 / 100                                      │
+│ Fair                                          │
+│                                               │
+│ 6 quality issue(s) · 14 statistical signal(s) │
+╰───────────────────────────────────────────────╯
+
+🧬 DATA QUALITY
+ MEDIUM  Column 'age' contains 10.0% missing values.
+ MEDIUM  Dataset contains 5.0% duplicate rows.
+ HIGH    Age-like column 'age' contains values outside 0-120. (range: -5.0 to 150.0)
+
+📊 STATISTICAL SIGNALS
+ 🔎 HIGH    Strong correlation detected between age and salary. (Pearson: 0.949)
+ 🔎 MEDIUM  Column 'customer_id' has very high cardinality. (100.0% unique)
+```
+
+Full breakdown tables for schema, missingness, cardinality, numerical stats, outliers, correlations, and categorical associations follow below the summary.
+
+---
+
+## Installation
+
+```bash
+pip install datasetdna
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/KAVYA-29-ai/DatasetDNA
+cd DatasetDNA
+pip install -e .
+```
+
+Requires Python 3.10+.
+
+## Usage
+
+```bash
+# Basic profile
+datasetdna data.csv
+
+# Include target-column analysis (class imbalance, target relationships)
+datasetdna data.csv --target churn
+```
+
+---
+
+## Why this exists
+
+Most ML failures aren't model failures — they're data failures nobody caught before training started: a leaked ID column, silent duplicates, a numeric column secretly stored as `"$1,200"` strings, a target with 95/5 imbalance nobody noticed until the model just predicted the majority class every time.
+
+DatasetDNA is meant to be the first command you run against any new CSV, before you write a single line of `train_test_split`.
+
+---
+
+## Roadmap
+
+- [x] **Phase 1 — Core profiler**: semantic types, missingness, duplicates, cardinality, numerical/categorical analysis, outliers, correlations, target analysis, two-tier health scoring
+- [x] **Phase 2 — Better UX**: actionable recommendations per issue, expanded validation, deeper test coverage
+- [x] **Phase 3 — HTML dashboard**: interactive Plotly charts, exportable shareable reports
+- [ ] **Phase 4 — PyPI**: stable public release
+
+---
+
+## Contributing
+
+Issues and PRs welcome. If you've found a dataset that breaks DatasetDNA (crashes, wrong classification, missed issue) — that's exactly the kind of bug report that helps most. Open an issue with the CSV shape (not necessarily the data itself) and what you expected to see.
+
+## License
+
+MIT © Kavya Rajput

datasetdna-0.1.2/README.md

@@ -0,0 +1,145 @@
+# 🧬 DatasetDNA
+
+**One command. Know if your dataset is safe to train on.**
+
+DatasetDNA profiles any CSV and produces a clear, actionable health report — catching real data-quality problems (missing values, duplicates, invalid ranges) while separating them from statistical signals that need human judgment, not automatic penalties (correlation, skew, cardinality).
+
+```bash
+pip install datasetdna
+datasetdna your_data.csv
+```
+
+That's it. No config, no notebook, no cloud upload — your data never leaves your machine.
+
+---
+
+## The problem with most profiling tools
+
+Most auto-EDA tools treat every statistical pattern as a "problem" and dock your data a few points for it. That sounds fine — until you run it on the Iris dataset, one of the most famous, cleanest datasets in all of ML, and it tells you `petal_length ↔ petal_width` correlation of `0.96` is a health issue.
+
+It isn't. That correlation is **expected botany**, not a data-quality defect. A tool that can't tell the difference between "this is broken" and "this is just how the world works" isn't trustworthy enough to run before training a real model.
+
+DatasetDNA draws a hard line between the two:
+
+| 🧬 Data Quality | 📊 Statistical Signals |
+|---|---|
+| **Affects your Health Score.** Objectively bad, in any context. | **Never affects your score.** Informational — could be fine, could be worth a look. |
+| Missing values | Strong correlations |
+| Duplicate rows | Skewness |
+| Target class imbalance | High cardinality |
+| Impossible values (age = 150, age = -5) | Outliers |
+
+This distinction is the core design decision behind the whole tool — not an afterthought.
+
+---
+
+## What it checks
+
+```
+CSV
+ ↓
+Loader + Cleaning        → handles $1,200 / "1,234" / N/A / null / "?" / "-" as one CSV, cleanly
+ ↓
+Semantic Type Detection  → tells an ID column from a real feature, a date from a category
+ ↓
+┌─────────────────────────────┐
+│ Overview        Numerical   │
+│ Schema          Categorical │
+│ Missingness     Outliers    │
+│ Duplicates      Correlations│
+│ Cardinality     Target      │
+└─────────────────────────────┘
+ ↓
+Health Scoring   → 0–100, driven only by real quality issues
+ ↓
+Console Report   → color-coded, readable, no PhD required
+```
+
+**Semantic Type Detection** is what separates this from a plain `df.describe()` script — it automatically:
+- Cleans numbers written as text (`"$55,000"`, `"1,200"`) into real numeric columns
+- Recognizes disguised missing values (`N/A`, `null`, `?`, `-`, `unknown`) as actual missing data, not valid strings
+- Flags ID-like columns (customer IDs, emails) and excludes them from correlation/association analysis, where they'd otherwise create meaningless "100% correlated" noise
+- Validates domain-plausible ranges (e.g., an `age` column with values of `-5` or `150` gets flagged as a real error, not just an "outlier")
+
+---
+
+## Example output
+
+```bash
+$ datasetdna examples/real.csv
+
+╭────────────── 🧬 Dataset Health ──────────────╮
+│ 🟠 Health Score                               │
+│                                               │
+│ 70 / 100                                      │
+│ Fair                                          │
+│                                               │
+│ 6 quality issue(s) · 14 statistical signal(s) │
+╰───────────────────────────────────────────────╯
+
+🧬 DATA QUALITY
+ MEDIUM  Column 'age' contains 10.0% missing values.
+ MEDIUM  Dataset contains 5.0% duplicate rows.
+ HIGH    Age-like column 'age' contains values outside 0-120. (range: -5.0 to 150.0)
+
+📊 STATISTICAL SIGNALS
+ 🔎 HIGH    Strong correlation detected between age and salary. (Pearson: 0.949)
+ 🔎 MEDIUM  Column 'customer_id' has very high cardinality. (100.0% unique)
+```
+
+Full breakdown tables for schema, missingness, cardinality, numerical stats, outliers, correlations, and categorical associations follow below the summary.
+
+---
+
+## Installation
+
+```bash
+pip install datasetdna
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/KAVYA-29-ai/DatasetDNA
+cd DatasetDNA
+pip install -e .
+```
+
+Requires Python 3.10+.
+
+## Usage
+
+```bash
+# Basic profile
+datasetdna data.csv
+
+# Include target-column analysis (class imbalance, target relationships)
+datasetdna data.csv --target churn
+```
+
+---
+
+## Why this exists
+
+Most ML failures aren't model failures — they're data failures nobody caught before training started: a leaked ID column, silent duplicates, a numeric column secretly stored as `"$1,200"` strings, a target with 95/5 imbalance nobody noticed until the model just predicted the majority class every time.
+
+DatasetDNA is meant to be the first command you run against any new CSV, before you write a single line of `train_test_split`.
+
+---
+
+## Roadmap
+
+- [x] **Phase 1 — Core profiler**: semantic types, missingness, duplicates, cardinality, numerical/categorical analysis, outliers, correlations, target analysis, two-tier health scoring
+- [x] **Phase 2 — Better UX**: actionable recommendations per issue, expanded validation, deeper test coverage
+- [x] **Phase 3 — HTML dashboard**: interactive Plotly charts, exportable shareable reports
+- [ ] **Phase 4 — PyPI**: stable public release
+
+---
+
+## Contributing
+
+Issues and PRs welcome. If you've found a dataset that breaks DatasetDNA (crashes, wrong classification, missed issue) — that's exactly the kind of bug report that helps most. Open an issue with the CSV shape (not necessarily the data itself) and what you expected to see.
+
+## License
+
+MIT © Kavya Rajput

datasetdna-0.1.2/datasetdna/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

datasetdna-0.1.2/datasetdna/cli.py

@@ -0,0 +1,307 @@
+from __future__ import annotations
+
+import os
+
+import typer
+
+from datasetdna.profiler.overview import check_overview
+from datasetdna.profiler.schema import check_schema
+from datasetdna.profiler.missing import check_missing
+from datasetdna.profiler.duplicates import check_duplicates
+from datasetdna.profiler.cardinality import check_cardinality
+from datasetdna.profiler.numerical import check_numerical
+from datasetdna.profiler.categorical import check_categorical
+from datasetdna.profiler.outliers import check_outliers
+from datasetdna.profiler.correlations import check_correlations
+from datasetdna.profiler.target import check_target
+
+from datasetdna.scoring.health_score import (
+    calculate_health_score,
+)
+
+from datasetdna.recommendations.recommendations import (
+    generate_recommendations,
+)
+
+from datasetdna.reporting.console import (
+    console,
+    render_report,
+)
+
+from datasetdna.reporting.html import (
+    render_html_report,
+)
+
+from datasetdna.utils.helpers import (
+    LARGE_FILE_SIZE_BYTES,
+    LARGE_FILE_SAMPLE_SIZE,
+    load_dataset,
+)
+
+
+app = typer.Typer(
+    help="DatasetDNA - Automated Dataset Health Profiler",
+    add_completion=False,
+)
+
+
+# =============================================================
+# TARGET INFERENCE
+# =============================================================
+
+TARGET_COLUMN_CANDIDATES = (
+    "target",
+    "label",
+    "churn",
+)
+
+
+def infer_target(
+    df,
+) -> tuple[str | None, bool]:
+    """
+    Infer a target column when the user does not provide one.
+
+    Priority:
+        target -> label -> churn
+
+    Returns:
+        (column_name, inferred)
+    """
+
+    normalized_columns = {
+        column.strip().lower(): column
+        for column in df.columns
+    }
+
+    for candidate in TARGET_COLUMN_CANDIDATES:
+
+        if candidate in normalized_columns:
+            return (
+                normalized_columns[candidate],
+                True,
+            )
+
+    return None, False
+
+
+# =============================================================
+# LARGE FILE WARNING
+# =============================================================
+
+def warn_if_large_file(
+    path: str,
+) -> None:
+    """
+    Warn the user when DatasetDNA will analyze a sample
+    instead of the complete dataset.
+    """
+
+    if not os.path.exists(path):
+        return
+
+    if os.path.getsize(path) > LARGE_FILE_SIZE_BYTES:
+        typer.echo(
+            f"Large file detected — analyzing a "
+            f"{LARGE_FILE_SAMPLE_SIZE:,}-row sample."
+        )
+
+
+# =============================================================
+# CLI COMMAND
+# =============================================================
+
+@app.command()
+def profile(
+    file: str = typer.Argument(
+        ...,
+        help="Path to the CSV file.",
+    ),
+    target: str | None = typer.Option(
+        None,
+        "--target",
+        "-t",
+        help="Optional target column.",
+    ),
+    html: bool = typer.Option(
+        False,
+        "--html",
+        help="Generate an HTML report.",
+    ),
+    output: str = typer.Option(
+        "datasetdna_report.html",
+        "--output",
+        "-o",
+        help="HTML output file path.",
+    ),
+):
+    """
+    Profile a CSV dataset and generate a health report.
+    """
+
+    try:
+
+        # ====================================================
+        # LARGE FILE WARNING
+        # ====================================================
+
+        warn_if_large_file(file)
+
+        # ====================================================
+        # LOAD DATASET
+        # ====================================================
+
+        df = load_dataset(file)
+
+        # ====================================================
+        # TARGET INFERENCE
+        # ====================================================
+
+        target_was_inferred = False
+
+        if target is None:
+
+            target, target_was_inferred = infer_target(
+                df
+            )
+
+        # ====================================================
+        # RUN PROFILERS
+        # ====================================================
+
+        overview = check_overview(df)
+
+        schema = check_schema(df)
+
+        missing = check_missing(df)
+
+        duplicates = check_duplicates(df)
+
+        cardinality = check_cardinality(df)
+
+        numerical = check_numerical(df)
+
+        categorical = check_categorical(df)
+
+        outliers = check_outliers(df)
+
+        correlations = check_correlations(df)
+
+        target_result = check_target(
+            df,
+            target,
+        )
+
+        # ====================================================
+        # MARK INFERRED TARGET
+        # ====================================================
+
+        if target_was_inferred:
+            target_result["inferred"] = True
+
+            console.print(
+                f"[yellow]ℹ Auto-inferred '{target}' as the primary target variable.[/yellow]\n"
+            )
+
+        # ====================================================
+        # COLLECT RESULTS
+        # ====================================================
+
+        results = {
+            "overview": overview,
+            "schema": schema,
+            "missing": missing,
+            "duplicates": duplicates,
+            "cardinality": cardinality,
+            "numerical": numerical,
+            "categorical": categorical,
+            "outliers": outliers,
+            "correlations": correlations,
+            "target": target_result,
+        }
+
+        # ====================================================
+        # HEALTH SCORE
+        # ====================================================
+
+        health = calculate_health_score(
+            results
+        )
+
+        results["health"] = health
+
+        # ====================================================
+        # RECOMMENDATIONS
+        # ====================================================
+
+        recommendations = generate_recommendations(
+            results
+        )
+
+        results["recommendations"] = recommendations
+
+        # ====================================================
+        # CONSOLE REPORT
+        # ====================================================
+
+        render_report(
+            results,
+            health,
+            recommendations,
+        )
+
+        # ====================================================
+        # HTML REPORT
+        # ====================================================
+
+        if html:
+
+            output_path = render_html_report(
+                results,
+                output_path=output,
+            )
+
+            typer.echo(
+                f"\nHTML report generated: {output_path}"
+            )
+
+    except FileNotFoundError as error:
+
+        typer.echo(
+            f"Error: {error}",
+            err=True,
+        )
+
+        raise typer.Exit(
+            code=1
+        )
+
+    except ValueError as error:
+
+        typer.echo(
+            f"Error: {error}",
+            err=True,
+        )
+
+        raise typer.Exit(
+            code=1
+        )
+
+    except Exception as error:
+
+        typer.echo(
+            f"Unexpected error: {error}",
+            err=True,
+        )
+
+        raise typer.Exit(
+            code=1
+        )
+
+
+# =============================================================
+# ENTRY POINT
+# =============================================================
+
+if __name__ == "__main__":
+    app()