data-detective-toolkit 0.3.1__tar.gz

data_detective_toolkit-0.3.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Mursal Gorchuyev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data_detective_toolkit-0.3.1/PKG-INFO

@@ -0,0 +1,271 @@
+Metadata-Version: 2.4
+Name: data-detective-toolkit
+Version: 0.3.1
+Summary: Automated data-quality profiling: CLI, web app, and REST API for CSV intelligence reports.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/murs5l/data-detective
+Project-URL: Issues, https://github.com/murs5l/data-detective/issues
+Keywords: data-quality,profiling,csv,data-profiling,eda,outlier-detection
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas<4.0,>=2.0
+Requires-Dist: numpy<3.0,>=1.24
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Provides-Extra: api
+Requires-Dist: fastapi>=0.110; extra == "api"
+Requires-Dist: uvicorn[standard]>=0.29; extra == "api"
+Requires-Dist: python-multipart>=0.0.9; extra == "api"
+Dynamic: license-file
+
+# 🕵️ Data Detective
+
+[![Tests](https://github.com/murs5l/data-detective/actions/workflows/tests.yml/badge.svg)](https://github.com/murs5l/data-detective/actions/workflows/tests.yml)
+
+**Automated data quality profiling in seconds.** Point Data Detective at a CSV and get back an intelligent report: shape, data types, missing values, duplicates, outliers, correlations, distribution skew, ID-like columns, and actionable insights. Choose your interface: fast CLI for scripts and CI, browser-based web app for exploration, or REST API for integration.
+
+### Why Data Detective?
+
+- **Fast**: profiles 1000+ rows in under a second
+- **Thorough**: detects 20+ data quality issues (outliers, duplicates, high cardinality, mixed types, skew, etc.)
+- **Private**: your data never leaves your machine; everything runs locally
+- **Flexible**: use as a CLI tool, web app, Python library, or REST API
+- **Lightweight**: no external ML services or cloud dependencies
+
+### Two ways to use it
+
+1. **CLI**: `data-detective analyze myfile.csv --html` for scripts and CI pipelines.
+2. **Web app**: `docker compose up` for interactive exploration with live charts.
+
+Both interfaces use the same profiling engine and produce identical reports.
+
+---
+
+## Quick start
+
+### Web app (interactive, no command line needed)
+
+The easiest way to explore a CSV:
+
+```bash
+git clone https://github.com/murs5l/data-detective.git
+cd data-detective
+docker compose up --build
+```
+
+Then open **http://localhost:8000**, drag and drop a CSV, and explore. The report includes:
+- Data shape and types
+- Missing value heatmap
+- Column Explorer: drill down into any numeric column to see histogram, boxplot, and 5-number summary
+- Outlier detection (IQR and MAD methods)
+- Correlation heatmap for numeric column pairs
+- Insights highlighted in plain English
+
+The browser-based UI is designed for non-technical and technical users alike: actionable insights appear first, and granular technical details are tucked behind a "Full technical report" toggle.
+
+**Running without Docker:** if you have Python and FastAPI installed:
+```bash
+pip install -e ".[api]"
+uvicorn backend.app.main:app --reload
+```
+Then visit `http://localhost:8000`.
+
+### CLI (scripting and CI/CD)
+
+Install locally for use in scripts and automated pipelines:
+
+```bash
+pip install data-detective-toolkit
+data-detective analyze myfile.csv
+```
+
+> The PyPI package is `data-detective-toolkit`; the installed command is `data-detective`.
+
+See [CLI docs](#cli) below for all flags and output formats.
+
+## CLI reference
+
+### Installation
+
+```bash
+pip install data-detective-toolkit
+```
+
+### Basic usage
+
+```bash
+# Print a text summary to stdout
+data-detective analyze myfile.csv
+
+# Generate an interactive HTML report
+data-detective analyze myfile.csv --html
+
+# Output JSON for programmatic use
+data-detective analyze myfile.csv --json | jq '.insights'
+
+# Write to specific files
+data-detective analyze myfile.csv --output-html reports/q1_data.html
+```
+
+### Options
+
+| Flag               | Description                                      | Example |
+|--------------------|-------------------------------------------------|---------|
+| `--json`           | Print full report as JSON to stdout              | `data-detective analyze data.csv --json` |
+| `--html`           | Generate `report.html` in current directory      | `data-detective analyze data.csv --html` |
+| `--output-json`    | Write JSON report to a specific file path         | `--output-json reports/profile.json` |
+| `--output-html`    | Write HTML report to a specific file path         | `--output-html reports/profile.html` |
+| `--outlier-method` | Choose outlier detection: `iqr` or `mad` (default) | `--outlier-method iqr` |
+| `--quiet`          | Suppress non-error progress messages              | `--quiet` |
+
+### Examples
+
+**CI/CD pipeline**: fail if data quality issues exceed a threshold
+```bash
+data-detective analyze incoming.csv --json | jq '.outliers_mad | length' | xargs -I {} bash -c 'exit {}'
+```
+
+**Batch processing**: profile all CSVs in a directory
+```bash
+for f in data/*.csv; do
+  data-detective analyze "$f" --output-html "reports/$(basename $f .csv).html"
+done
+```
+
+**Data validation**: check for specific issues before ETL
+```bash
+data-detective analyze input.csv --quiet --json | jq '.high_cardinality_columns'
+```
+
+## REST API (for integration)
+
+When running the web app or backend API server, you can integrate Data Detective into your own applications.
+
+### Endpoints
+
+**POST /api/analyze** – Analyze a CSV file, return JSON report
+```bash
+curl -F "file=@data.csv" "http://localhost:8000/api/analyze?outlier_method=mad"
+```
+
+Response: JSON object with all profiling data (see example below).
+
+**POST /api/analyze/html** – Analyze a CSV file, return standalone HTML report
+```bash
+curl -F "file=@data.csv" "http://localhost:8000/api/analyze/html" > report.html
+```
+
+Response: A self-contained HTML file (no external dependencies).
+
+**GET /api/health** – Health check
+```bash
+curl http://localhost:8000/api/health
+```
+
+Response: `{"status": "ok", "version": "0.3.0"}`
+
+### Python example
+
+```python
+import requests
+
+with open("mydata.csv", "rb") as f:
+    response = requests.post(
+        "http://localhost:8000/api/analyze",
+        files={"file": f},
+        params={"outlier_method": "mad"}
+    )
+
+report = response.json()
+print(f"Shape: {report['shape']}")
+print(f"Insights: {report['insights']}")
+print(f"Outliers detected: {report['outliers_mad']}")
+```
+
+### JavaScript example
+
+```javascript
+const formData = new FormData();
+formData.append("file", csvFile); // File object from input[type=file]
+
+const response = await fetch("/api/analyze?outlier_method=mad", {
+  method: "POST",
+  body: formData
+});
+
+const report = await response.json();
+console.log(report.insights);
+```
+
+Full API documentation is available at `/docs` (Swagger UI) when the backend is running.
+
+## What it detects
+
+Data Detective automatically flags 20+ data quality issues:
+
+**Structure & Completeness**
+- Shape (rows × columns) and column data types
+- Missing values (count and percentage per column)
+- Duplicate rows
+- Duplicate columns (identical data, different names)
+- Constant columns (no variation; e.g., all 1s)
+
+**Statistical Issues**
+- Outliers: IQR method (classic box-and-whisker) or MAD (robust for skewed data)
+- Highly correlated numeric pairs
+- Skewed distributions (heavy left/right tails)
+- Kurtosis (fat tails or sharp peaks)
+
+**Data Type & Value Issues**
+- High-cardinality columns (likely IDs or keys)
+- Mixed-type columns (e.g., numbers and strings in the same column)
+- Unexpected negative values (e.g., negative `age`, `price`, `quantity`)
+- Date-like values not parsed as datetime
+
+**Text Columns**
+- Length statistics (min, max, average)
+- Blank or whitespace-only values
+
+**Aggregates**
+- Full numeric correlation matrix
+- Histograms with bin edges and counts for numeric columns
+- Five-number summary (min, Q1, median, Q3, max) per numeric column
+- Per-column memory footprint (KB)
+
+**Output**
+- Plain-English actionable insights highlighting the most important issues
+
+## Development
+
+```bash
+# core engine + CLI
+pip install -e ".[dev]"
+pytest tests
+
+# backend API
+pip install -e ".[api]"
+pip install -r backend/requirements-dev.txt
+pytest backend/tests
+
+# fastscan (Go speed layer)
+cd tools/fastscan && go test ./...
+```
+
+CI runs all three (`.github/workflows/tests.yml`) on every push and PR.
+
+## Roadmap
+
+- GitHub Action to run Data Detective as a data-quality gate in CI
+- Support for additional sources (Parquet, JSON, database connections)
+- Scheduled/hosted profiling for pipelines beyond ad-hoc uploads
+
+## License
+
+See [LICENSE](LICENSE).

data_detective_toolkit-0.3.1/README.md

@@ -0,0 +1,244 @@
+# 🕵️ Data Detective
+
+[![Tests](https://github.com/murs5l/data-detective/actions/workflows/tests.yml/badge.svg)](https://github.com/murs5l/data-detective/actions/workflows/tests.yml)
+
+**Automated data quality profiling in seconds.** Point Data Detective at a CSV and get back an intelligent report: shape, data types, missing values, duplicates, outliers, correlations, distribution skew, ID-like columns, and actionable insights. Choose your interface: fast CLI for scripts and CI, browser-based web app for exploration, or REST API for integration.
+
+### Why Data Detective?
+
+- **Fast**: profiles 1000+ rows in under a second
+- **Thorough**: detects 20+ data quality issues (outliers, duplicates, high cardinality, mixed types, skew, etc.)
+- **Private**: your data never leaves your machine; everything runs locally
+- **Flexible**: use as a CLI tool, web app, Python library, or REST API
+- **Lightweight**: no external ML services or cloud dependencies
+
+### Two ways to use it
+
+1. **CLI**: `data-detective analyze myfile.csv --html` for scripts and CI pipelines.
+2. **Web app**: `docker compose up` for interactive exploration with live charts.
+
+Both interfaces use the same profiling engine and produce identical reports.
+
+---
+
+## Quick start
+
+### Web app (interactive, no command line needed)
+
+The easiest way to explore a CSV:
+
+```bash
+git clone https://github.com/murs5l/data-detective.git
+cd data-detective
+docker compose up --build
+```
+
+Then open **http://localhost:8000**, drag and drop a CSV, and explore. The report includes:
+- Data shape and types
+- Missing value heatmap
+- Column Explorer: drill down into any numeric column to see histogram, boxplot, and 5-number summary
+- Outlier detection (IQR and MAD methods)
+- Correlation heatmap for numeric column pairs
+- Insights highlighted in plain English
+
+The browser-based UI is designed for non-technical and technical users alike: actionable insights appear first, and granular technical details are tucked behind a "Full technical report" toggle.
+
+**Running without Docker:** if you have Python and FastAPI installed:
+```bash
+pip install -e ".[api]"
+uvicorn backend.app.main:app --reload
+```
+Then visit `http://localhost:8000`.
+
+### CLI (scripting and CI/CD)
+
+Install locally for use in scripts and automated pipelines:
+
+```bash
+pip install data-detective-toolkit
+data-detective analyze myfile.csv
+```
+
+> The PyPI package is `data-detective-toolkit`; the installed command is `data-detective`.
+
+See [CLI docs](#cli) below for all flags and output formats.
+
+## CLI reference
+
+### Installation
+
+```bash
+pip install data-detective-toolkit
+```
+
+### Basic usage
+
+```bash
+# Print a text summary to stdout
+data-detective analyze myfile.csv
+
+# Generate an interactive HTML report
+data-detective analyze myfile.csv --html
+
+# Output JSON for programmatic use
+data-detective analyze myfile.csv --json | jq '.insights'
+
+# Write to specific files
+data-detective analyze myfile.csv --output-html reports/q1_data.html
+```
+
+### Options
+
+| Flag               | Description                                      | Example |
+|--------------------|-------------------------------------------------|---------|
+| `--json`           | Print full report as JSON to stdout              | `data-detective analyze data.csv --json` |
+| `--html`           | Generate `report.html` in current directory      | `data-detective analyze data.csv --html` |
+| `--output-json`    | Write JSON report to a specific file path         | `--output-json reports/profile.json` |
+| `--output-html`    | Write HTML report to a specific file path         | `--output-html reports/profile.html` |
+| `--outlier-method` | Choose outlier detection: `iqr` or `mad` (default) | `--outlier-method iqr` |
+| `--quiet`          | Suppress non-error progress messages              | `--quiet` |
+
+### Examples
+
+**CI/CD pipeline**: fail if data quality issues exceed a threshold
+```bash
+data-detective analyze incoming.csv --json | jq '.outliers_mad | length' | xargs -I {} bash -c 'exit {}'
+```
+
+**Batch processing**: profile all CSVs in a directory
+```bash
+for f in data/*.csv; do
+  data-detective analyze "$f" --output-html "reports/$(basename $f .csv).html"
+done
+```
+
+**Data validation**: check for specific issues before ETL
+```bash
+data-detective analyze input.csv --quiet --json | jq '.high_cardinality_columns'
+```
+
+## REST API (for integration)
+
+When running the web app or backend API server, you can integrate Data Detective into your own applications.
+
+### Endpoints
+
+**POST /api/analyze** – Analyze a CSV file, return JSON report
+```bash
+curl -F "file=@data.csv" "http://localhost:8000/api/analyze?outlier_method=mad"
+```
+
+Response: JSON object with all profiling data (see example below).
+
+**POST /api/analyze/html** – Analyze a CSV file, return standalone HTML report
+```bash
+curl -F "file=@data.csv" "http://localhost:8000/api/analyze/html" > report.html
+```
+
+Response: A self-contained HTML file (no external dependencies).
+
+**GET /api/health** – Health check
+```bash
+curl http://localhost:8000/api/health
+```
+
+Response: `{"status": "ok", "version": "0.3.0"}`
+
+### Python example
+
+```python
+import requests
+
+with open("mydata.csv", "rb") as f:
+    response = requests.post(
+        "http://localhost:8000/api/analyze",
+        files={"file": f},
+        params={"outlier_method": "mad"}
+    )
+
+report = response.json()
+print(f"Shape: {report['shape']}")
+print(f"Insights: {report['insights']}")
+print(f"Outliers detected: {report['outliers_mad']}")
+```
+
+### JavaScript example
+
+```javascript
+const formData = new FormData();
+formData.append("file", csvFile); // File object from input[type=file]
+
+const response = await fetch("/api/analyze?outlier_method=mad", {
+  method: "POST",
+  body: formData
+});
+
+const report = await response.json();
+console.log(report.insights);
+```
+
+Full API documentation is available at `/docs` (Swagger UI) when the backend is running.
+
+## What it detects
+
+Data Detective automatically flags 20+ data quality issues:
+
+**Structure & Completeness**
+- Shape (rows × columns) and column data types
+- Missing values (count and percentage per column)
+- Duplicate rows
+- Duplicate columns (identical data, different names)
+- Constant columns (no variation; e.g., all 1s)
+
+**Statistical Issues**
+- Outliers: IQR method (classic box-and-whisker) or MAD (robust for skewed data)
+- Highly correlated numeric pairs
+- Skewed distributions (heavy left/right tails)
+- Kurtosis (fat tails or sharp peaks)
+
+**Data Type & Value Issues**
+- High-cardinality columns (likely IDs or keys)
+- Mixed-type columns (e.g., numbers and strings in the same column)
+- Unexpected negative values (e.g., negative `age`, `price`, `quantity`)
+- Date-like values not parsed as datetime
+
+**Text Columns**
+- Length statistics (min, max, average)
+- Blank or whitespace-only values
+
+**Aggregates**
+- Full numeric correlation matrix
+- Histograms with bin edges and counts for numeric columns
+- Five-number summary (min, Q1, median, Q3, max) per numeric column
+- Per-column memory footprint (KB)
+
+**Output**
+- Plain-English actionable insights highlighting the most important issues
+
+## Development
+
+```bash
+# core engine + CLI
+pip install -e ".[dev]"
+pytest tests
+
+# backend API
+pip install -e ".[api]"
+pip install -r backend/requirements-dev.txt
+pytest backend/tests
+
+# fastscan (Go speed layer)
+cd tools/fastscan && go test ./...
+```
+
+CI runs all three (`.github/workflows/tests.yml`) on every push and PR.
+
+## Roadmap
+
+- GitHub Action to run Data Detective as a data-quality gate in CI
+- Support for additional sources (Parquet, JSON, database connections)
+- Scheduled/hosted profiling for pipelines beyond ad-hoc uploads
+
+## License
+
+See [LICENSE](LICENSE).

data_detective_toolkit-0.3.1/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=77", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "data-detective-toolkit"
+version = "0.3.1"
+description = "Automated data-quality profiling: CLI, web app, and REST API for CSV intelligence reports."
+readme = "README.md"
+license = "MIT"
+requires-python = ">=3.9"
+keywords = ["data-quality", "profiling", "csv", "data-profiling", "eda", "outlier-detection"]
+classifiers = [
+	"Development Status :: 4 - Beta",
+	"Intended Audience :: Developers",
+	"Intended Audience :: Science/Research",
+	"Programming Language :: Python :: 3",
+	"Topic :: Scientific/Engineering :: Information Analysis",
+	"Topic :: Utilities",
+]
+dependencies = [
+	# Upper bounds are intentional: CI has verified compatibility up to
+	# pandas 3.x (Python 3.11/3.12 already resolve to it), but the next
+	# major bump should be a deliberate version-bump PR, not a silent
+	# `pip install` picking up unverified breaking changes.
+	"pandas>=2.0,<4.0",
+	"numpy>=1.24,<3.0",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0"]
+api = [
+	"fastapi>=0.110",
+	"uvicorn[standard]>=0.29",
+	"python-multipart>=0.0.9",
+]
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["data_detective*"]
+
+[project.scripts]
+data-detective = "data_detective.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/murs5l/data-detective"
+Issues = "https://github.com/murs5l/data-detective/issues"

data_detective_toolkit-0.3.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

data_detective_toolkit-0.3.1/src/data_detective/cli.py

@@ -0,0 +1,108 @@
+import argparse
+import json
+import sys
+from pathlib import Path
+
+from data_detective.exceptions import DataLoadError
+from data_detective.html_report import generate_html_report
+from data_detective.loader import load_csv
+from data_detective.profiler import DataProfiler
+from data_detective.report import print_report
+
+
+# -----------------------------
+# COMMAND HANDLER
+# -----------------------------
+def run_analyze(args):
+    if not args.quiet:
+        print("🔥 Data Detective starting...\n")
+
+    df = load_csv(args.file)
+    profiler = DataProfiler(df)
+    report = profiler.run_full_profile(outlier_method=args.outlier_method)
+
+    emit_html = args.html or bool(args.output_html)
+    emit_json = args.json or bool(args.output_json)
+
+    if emit_html:
+        output_html_path = args.output_html or "report.html"
+        generate_html_report(report, output_path=output_html_path)
+
+    if emit_json:
+        payload = json.dumps(report, indent=2)
+        if args.output_json:
+            Path(args.output_json).write_text(payload + "\n", encoding="utf-8")
+        elif args.json:
+            print(payload)
+
+    if emit_html or emit_json:
+        return
+
+    if not args.json and not args.html:
+        print_report(report)
+
+    return 0
+
+
+# -----------------------------
+# CLI SETUP
+# -----------------------------
+def build_parser():
+    parser = argparse.ArgumentParser(
+        prog="data-detective",
+        description="🕵️ Data Detective - Smart Data Profiling Tool"
+    )
+
+    subparsers = parser.add_subparsers(dest="command")
+
+    analyze_parser = subparsers.add_parser(
+        "analyze",
+        help="Analyze a CSV file"
+    )
+
+    analyze_parser.add_argument("file", help="Path to CSV file")
+    analyze_parser.add_argument("--json", action="store_true", help="Output JSON report")
+    analyze_parser.add_argument("--html", action="store_true", help="Generate HTML report")
+    analyze_parser.add_argument("--output-json", help="Write JSON report to the given file path")
+    analyze_parser.add_argument("--output-html", help="Write HTML report to the given file path")
+    analyze_parser.add_argument(
+        "--outlier-method",
+        choices=["iqr", "mad"],
+        default="mad",
+        help="Outlier detection method used for insights (default: mad)",
+    )
+    analyze_parser.add_argument(
+        "--quiet",
+        action="store_true",
+        help="Suppress non-error progress messages",
+    )
+
+    analyze_parser.set_defaults(func=run_analyze)
+
+    return parser
+
+
+# -----------------------------
+# MAIN ENTRY POINT
+# -----------------------------
+def main():
+    parser = build_parser()
+    args = parser.parse_args()
+
+    if not hasattr(args, "func"):
+        parser.print_help()
+        return 1
+
+    try:
+        return args.func(args)
+    except DataLoadError as e:
+        print(f"❌ {e}", file=sys.stderr)
+        return 1
+    except Exception as e:
+        print(f"❌ Unexpected error: {e}", file=sys.stderr)
+        return 1
+
+
+# REQUIRED ENTRY POINT
+if __name__ == "__main__":
+    sys.exit(main())