datascope-dq 2.1.0__tar.gz

datascope_dq-2.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Shawn (Lailara LLC)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

datascope_dq-2.1.0/PKG-INFO

@@ -0,0 +1,185 @@
+Metadata-Version: 2.4
+Name: datascope-dq
+Version: 2.1.0
+Summary: Data quality diagnostics for tabular datasets — surfaces hidden problems in plain English
+Author-email: Shawn <msshawnp@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/MsShawnP/datascope
+Project-URL: Repository, https://github.com/MsShawnP/datascope
+Project-URL: Issues, https://github.com/MsShawnP/datascope/issues
+Keywords: data-quality,diagnostics,csv,excel,type-detection,data-validation
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: reportlab>=4.0.0
+Requires-Dist: defusedxml>=0.7.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Dynamic: license-file
+
+# datascope
+
+Data created upstream — by manufacturing teams entering UPCs, inventory staff assigning product codes, offshore developers choosing column types — silently breaks systems downstream. A product code with letters where EDI expects numbers. Fifteen "N/A" strings buried in 500 numeric rows that pandas silently drops, skewing every calculation by 3%.
+
+datascope finds these problems, explains what's wrong in plain English, and tells you what to fix. It reads each cell's actual type (not what pandas infers), detects hidden quality issues, classifies their severity by downstream impact, and generates a professional diagnostic report.
+
+---
+
+## What It Finds
+
+| Detection | Example | Severity |
+|---|---|---|
+| **Mixed types** | 485 numbers + 15 strings in a "numeric" column | Critical |
+| **Sentinel values** | "N/A", "TBD", "pending" hiding in numeric data | Critical |
+| **Leading-zero inconsistency** | "00123" alongside "456" — keys that won't match | Warning |
+| **Mixed date formats** | "01/15/2026" and "2026-01-15" in the same column | Warning |
+| **Suspected duplicate IDs** | 98% unique in an ID column — the other 2% will fan out joins | Warning |
+| **Near-constant columns** | 1 distinct value across 10,000 rows | Info |
+
+Each finding is expressed as **assumption vs. reality**: what the data *appears* to be vs. what it *actually contains*. Every finding includes a downstream impact explanation, a fix recommendation, and a prevention rule.
+
+---
+
+## Installation
+
+```bash
+pip install datascope-dq
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/MsShawnP/datascope.git
+cd datascope
+pip install -e .
+```
+
+---
+
+## Usage
+
+```bash
+# Analyze an Excel file
+datascope data.xlsx
+
+# Analyze a CSV
+datascope sales_export.csv
+
+# Specify a sheet and output directory
+datascope data.xlsx --sheet Revenue --output-dir ./client_reports
+```
+
+The tool produces a PDF diagnostic report and prints a summary to stdout:
+
+```
+datascope: Analyzing sample_mixed_types.xlsx...
+  200 rows x 6 columns
+
+Found 4 findings:
+  2 Critical  ########
+  1 Warning   ####
+  1 Info      ####
+
+Top critical findings:
+  * revenue_mixed: 15 non-numeric values hiding in an otherwise numeric column
+  * status: Sentinel values 'N/A' and 'TBD' in numeric data
+
+Report saved: reports/sample_mixed_types_diagnostic.pdf
+```
+
+---
+
+## The Report
+
+The PDF report is structured for non-technical readers — no jargon, no composite scores, no unexplained metrics.
+
+**Executive Summary** — overall health assessment, finding counts by severity, top critical issues highlighted.
+
+**Findings by Severity** — each finding presented as a card:
+- **Assumption**: what the data appears to be
+- **Reality**: what it actually contains
+- **Impact**: what breaks downstream
+- **Recommended Fix**: what to do now
+- **Prevention Rule**: what right looks like going forward
+
+**Field Inventory** — summary table of all columns with their detected issue types and severity.
+
+Findings are color-coded (red/amber/blue) and grouped by severity so readers know what to fix first.
+
+---
+
+## How It Works
+
+Most tools let pandas (or the SQL driver, or Excel) decide column types. A column with 485 numbers and 15 strings becomes `float64` — the strings become `NaN`, the type problem disappears, and every downstream calculation is quietly wrong.
+
+datascope reads each cell's actual Python type via openpyxl (for Excel) or raw-string inference (for CSV). This cell-level type detection is always on — there's no flag to enable it because skipping it defeats the purpose.
+
+The analysis pipeline:
+
+1. **Load** — read with cell-level type preservation (no silent coercion)
+2. **Detect** — five analyzers scan for type inconsistencies, sentinels, format issues, and cardinality anomalies
+3. **Classify** — severity assigned by downstream impact (critical = silent data loss, warning = likely misinterpretation, info = worth noting)
+4. **Compose** — plain-English narrative generated for each finding
+5. **Report** — professional PDF rendered with reportlab
+
+---
+
+## Severity Model
+
+| Level | Meaning | Examples |
+|---|---|---|
+| **Critical** | Silent data loss or incorrect calculations will occur | Mixed types in numeric columns; sentinel values pandas drops without warning |
+| **Warning** | Key mismatches or misinterpretation likely | Leading-zero stripping; ambiguous date formats; duplicate IDs |
+| **Info** | Worth noting, no direct downstream breakage | Near-constant columns; unusual cardinality |
+
+---
+
+## Project Structure
+
+```
+datascope/
+├── loaders/          # Excel and CSV with cell-level type tracking
+│   ├── excel.py      # openpyxl-based, preserves per-cell Python types
+│   ├── csv_loader.py # Raw string inference (None → int → float → bool → datetime → str)
+│   └── base.py       # Extension-based dispatch
+├── analyzers/        # Five detectors, each returns list[Finding]
+│   ├── type_consistency.py
+│   ├── sentinel.py
+│   ├── format_check.py
+│   └── cardinality.py
+├── findings/         # Severity classifier + NL template engine
+│   ├── severity.py   # Impact-based classification rules
+│   ├── templates.py  # Plain-English templates per finding sub-type
+│   ├── composer.py   # Template dispatch
+│   └── pipeline.py   # classify → compose → sort
+├── reports/
+│   └── pdf.py        # Professional PDF with reportlab
+└── cli.py            # argparse CLI, pipeline orchestration
+```
+
+---
+
+## Requirements
+
+- Python 3.10+
+- pandas >= 2.0
+- openpyxl >= 3.1
+- reportlab >= 4.0
+- defusedxml >= 0.7
+
+---
+
+## License
+
+MIT

datascope_dq-2.1.0/README.md

@@ -0,0 +1,155 @@
+# datascope
+
+Data created upstream — by manufacturing teams entering UPCs, inventory staff assigning product codes, offshore developers choosing column types — silently breaks systems downstream. A product code with letters where EDI expects numbers. Fifteen "N/A" strings buried in 500 numeric rows that pandas silently drops, skewing every calculation by 3%.
+
+datascope finds these problems, explains what's wrong in plain English, and tells you what to fix. It reads each cell's actual type (not what pandas infers), detects hidden quality issues, classifies their severity by downstream impact, and generates a professional diagnostic report.
+
+---
+
+## What It Finds
+
+| Detection | Example | Severity |
+|---|---|---|
+| **Mixed types** | 485 numbers + 15 strings in a "numeric" column | Critical |
+| **Sentinel values** | "N/A", "TBD", "pending" hiding in numeric data | Critical |
+| **Leading-zero inconsistency** | "00123" alongside "456" — keys that won't match | Warning |
+| **Mixed date formats** | "01/15/2026" and "2026-01-15" in the same column | Warning |
+| **Suspected duplicate IDs** | 98% unique in an ID column — the other 2% will fan out joins | Warning |
+| **Near-constant columns** | 1 distinct value across 10,000 rows | Info |
+
+Each finding is expressed as **assumption vs. reality**: what the data *appears* to be vs. what it *actually contains*. Every finding includes a downstream impact explanation, a fix recommendation, and a prevention rule.
+
+---
+
+## Installation
+
+```bash
+pip install datascope-dq
+```
+
+Or install from source:
+
+```bash
+git clone https://github.com/MsShawnP/datascope.git
+cd datascope
+pip install -e .
+```
+
+---
+
+## Usage
+
+```bash
+# Analyze an Excel file
+datascope data.xlsx
+
+# Analyze a CSV
+datascope sales_export.csv
+
+# Specify a sheet and output directory
+datascope data.xlsx --sheet Revenue --output-dir ./client_reports
+```
+
+The tool produces a PDF diagnostic report and prints a summary to stdout:
+
+```
+datascope: Analyzing sample_mixed_types.xlsx...
+  200 rows x 6 columns
+
+Found 4 findings:
+  2 Critical  ########
+  1 Warning   ####
+  1 Info      ####
+
+Top critical findings:
+  * revenue_mixed: 15 non-numeric values hiding in an otherwise numeric column
+  * status: Sentinel values 'N/A' and 'TBD' in numeric data
+
+Report saved: reports/sample_mixed_types_diagnostic.pdf
+```
+
+---
+
+## The Report
+
+The PDF report is structured for non-technical readers — no jargon, no composite scores, no unexplained metrics.
+
+**Executive Summary** — overall health assessment, finding counts by severity, top critical issues highlighted.
+
+**Findings by Severity** — each finding presented as a card:
+- **Assumption**: what the data appears to be
+- **Reality**: what it actually contains
+- **Impact**: what breaks downstream
+- **Recommended Fix**: what to do now
+- **Prevention Rule**: what right looks like going forward
+
+**Field Inventory** — summary table of all columns with their detected issue types and severity.
+
+Findings are color-coded (red/amber/blue) and grouped by severity so readers know what to fix first.
+
+---
+
+## How It Works
+
+Most tools let pandas (or the SQL driver, or Excel) decide column types. A column with 485 numbers and 15 strings becomes `float64` — the strings become `NaN`, the type problem disappears, and every downstream calculation is quietly wrong.
+
+datascope reads each cell's actual Python type via openpyxl (for Excel) or raw-string inference (for CSV). This cell-level type detection is always on — there's no flag to enable it because skipping it defeats the purpose.
+
+The analysis pipeline:
+
+1. **Load** — read with cell-level type preservation (no silent coercion)
+2. **Detect** — five analyzers scan for type inconsistencies, sentinels, format issues, and cardinality anomalies
+3. **Classify** — severity assigned by downstream impact (critical = silent data loss, warning = likely misinterpretation, info = worth noting)
+4. **Compose** — plain-English narrative generated for each finding
+5. **Report** — professional PDF rendered with reportlab
+
+---
+
+## Severity Model
+
+| Level | Meaning | Examples |
+|---|---|---|
+| **Critical** | Silent data loss or incorrect calculations will occur | Mixed types in numeric columns; sentinel values pandas drops without warning |
+| **Warning** | Key mismatches or misinterpretation likely | Leading-zero stripping; ambiguous date formats; duplicate IDs |
+| **Info** | Worth noting, no direct downstream breakage | Near-constant columns; unusual cardinality |
+
+---
+
+## Project Structure
+
+```
+datascope/
+├── loaders/          # Excel and CSV with cell-level type tracking
+│   ├── excel.py      # openpyxl-based, preserves per-cell Python types
+│   ├── csv_loader.py # Raw string inference (None → int → float → bool → datetime → str)
+│   └── base.py       # Extension-based dispatch
+├── analyzers/        # Five detectors, each returns list[Finding]
+│   ├── type_consistency.py
+│   ├── sentinel.py
+│   ├── format_check.py
+│   └── cardinality.py
+├── findings/         # Severity classifier + NL template engine
+│   ├── severity.py   # Impact-based classification rules
+│   ├── templates.py  # Plain-English templates per finding sub-type
+│   ├── composer.py   # Template dispatch
+│   └── pipeline.py   # classify → compose → sort
+├── reports/
+│   └── pdf.py        # Professional PDF with reportlab
+└── cli.py            # argparse CLI, pipeline orchestration
+```
+
+---
+
+## Requirements
+
+- Python 3.10+
+- pandas >= 2.0
+- openpyxl >= 3.1
+- reportlab >= 4.0
+- defusedxml >= 0.7
+
+---
+
+## License
+
+MIT

datascope_dq-2.1.0/datascope/__init__.py

@@ -0,0 +1,5 @@
+"""datascope -- data quality diagnostics for tabular datasets."""
+
+from __future__ import annotations
+
+__version__ = "2.1.0"

datascope_dq-2.1.0/datascope/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point for ``python -m datascope``."""
+
+from datascope.cli import main
+
+if __name__ == "__main__":
+    main()

datascope_dq-2.1.0/datascope/analyzers/__init__.py

@@ -0,0 +1,16 @@
+"""datascope.analyzers -- detectors that produce Finding objects."""
+
+from __future__ import annotations
+
+from datascope.analyzers.type_consistency import analyze_type_consistency
+from datascope.analyzers.sentinel import analyze_sentinels
+from datascope.analyzers.format_check import analyze_leading_zeros, analyze_mixed_dates
+from datascope.analyzers.cardinality import analyze_cardinality
+
+__all__ = [
+    "analyze_type_consistency",
+    "analyze_sentinels",
+    "analyze_leading_zeros",
+    "analyze_mixed_dates",
+    "analyze_cardinality",
+]

datascope_dq-2.1.0/datascope/analyzers/base.py

@@ -0,0 +1,19 @@
+"""Analyzer contract and registry.
+
+There is no abstract base class.  An analyzer is any callable matching::
+
+    Callable[[LoaderResult], list[Finding]]
+
+It receives a :class:`~datascope.models.LoaderResult` and returns zero or
+more :class:`~datascope.models.Finding` instances.  The pipeline runner
+(U9) will call each registered analyzer and merge the results.
+"""
+
+from __future__ import annotations
+
+from typing import Callable
+
+from datascope.models import Finding, LoaderResult
+
+#: Type alias documenting the analyzer contract.
+Analyzer = Callable[[LoaderResult], list[Finding]]

datascope_dq-2.1.0/datascope/analyzers/cardinality.py

@@ -0,0 +1,112 @@
+"""Cardinality-anomaly detector.
+
+Flags columns that are near-constant (very low uniqueness) or that look
+like ID columns with unexpected duplicates (very high but not 100%
+uniqueness).
+
+Produces :class:`~datascope.models.Finding` instances with
+:attr:`~datascope.models.FindingType.NEAR_CONSTANT` or
+:attr:`~datascope.models.FindingType.DUPLICATE_IDS`.
+
+Severity is *not* assigned here -- that is the severity classifier's job (U7).
+"""
+
+from __future__ import annotations
+
+from collections import Counter
+
+from datascope.models import Finding, FindingType, LoaderResult
+
+
+# ---------------------------------------------------------------------------
+# Thresholds
+# ---------------------------------------------------------------------------
+
+_MIN_ROWS = 10             # Skip columns with fewer non-null rows
+_NEAR_CONSTANT_MAX = 0.01  # uniqueness_ratio < this => near-constant
+_SUSPECTED_ID_MIN = 0.95   # uniqueness_ratio > this AND < 1.0 => suspected-ID dups
+
+
+# ---------------------------------------------------------------------------
+# Detector
+# ---------------------------------------------------------------------------
+
+def analyze_cardinality(result: LoaderResult) -> list[Finding]:
+    """Detect cardinality anomalies in each column.
+
+    Two patterns are flagged:
+
+    * **Near-constant**: fewer than 1% of values are unique.
+    * **Suspected duplicate IDs**: more than 95% unique but less than
+      100%, suggesting an ID column with unexpected duplicates.
+
+    Columns with fewer than 10 non-null rows are skipped because
+    cardinality ratios are unreliable for tiny samples.
+
+    Parameters
+    ----------
+    result:
+        A :class:`~datascope.models.LoaderResult`.
+
+    Returns
+    -------
+    list[Finding]
+        One finding per column that exhibits a cardinality anomaly.
+    """
+    findings: list[Finding] = []
+
+    for col_name in result.dataframe.columns:
+        series = result.dataframe[col_name]
+        filled = series.dropna()
+        total_count = len(filled)
+
+        if total_count < _MIN_ROWS:
+            continue
+
+        unique_count = filled.nunique()
+        uniqueness_ratio = round(unique_count / total_count, 4)
+
+        if uniqueness_ratio < _NEAR_CONSTANT_MAX:
+            # Near-constant column
+            value_counts = Counter(filled)
+            top_values = [
+                {"value": str(val), "count": cnt}
+                for val, cnt in value_counts.most_common(5)
+            ]
+
+            evidence = {
+                "unique_count": unique_count,
+                "total_count": total_count,
+                "uniqueness_ratio": uniqueness_ratio,
+                "top_values": top_values,
+            }
+
+            findings.append(Finding(
+                field_name=col_name,
+                finding_type=FindingType.NEAR_CONSTANT,
+                evidence=evidence,
+            ))
+
+        elif _SUSPECTED_ID_MIN < uniqueness_ratio < 1.0:
+            # Suspected duplicate IDs
+            value_counts = Counter(filled)
+            duplicate_values = [
+                str(val)
+                for val, cnt in value_counts.most_common()
+                if cnt > 1
+            ][:10]
+
+            evidence = {
+                "unique_count": unique_count,
+                "total_count": total_count,
+                "uniqueness_ratio": uniqueness_ratio,
+                "duplicate_values": duplicate_values,
+            }
+
+            findings.append(Finding(
+                field_name=col_name,
+                finding_type=FindingType.DUPLICATE_IDS,
+                evidence=evidence,
+            ))
+
+    return findings