dqal 0.1.0__tar.gz

dqal-0.1.0/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to DQAL will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] — 2026-09-09
+
+### Added
+
+- **Quality Scorer** with three sub-signals: Missingness, Drift (PSI/JS divergence), and Outlier (Isolation Forest / Mahalanobis).
+- **3-tier gating state machine** (`SERVE` / `FLAG` / `ABSTAIN`) with hysteresis dead-band and consecutive confirmation to prevent flapping.
+- **Model-agnostic adapter** wrapping scikit-learn estimators, PyTorch `nn.Module`, and custom callables via `DQAL.predict()`.
+- **Privacy-preserving SQLite telemetry logger** — logs statistical metrics only; raw feature logging disabled by default.
+- **Retraining orchestrator** with minimum data volume, label verification, and validation-gated model promotion with rollback.
+- **Learned Quality Scorer** for training a meta-model on historical quality signals.
+- **Terminal telemetry reporting & SQLite logger** for real-time batch metrics, gating decisions, and selective retraining history.
+- **60-batch degradation benchmark** (`benchmarks/run_benchmark.py`) validating correlation, latency, early warning, and label safety.
+- **CI pipeline** via GitHub Actions across Python 3.10 / 3.11 / 3.12.
+- **YAML-based configuration** (`configs/default_config.yaml`) for weights, thresholds, hysteresis, logging, and retrain policy.

dqal-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DQAL Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dqal-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+recursive-include configs *.yaml
+recursive-include dqal *.py

dqal-0.1.0/PKG-INFO

@@ -0,0 +1,269 @@
+Metadata-Version: 2.4
+Name: dqal
+Version: 0.1.0
+Summary: DQAL — Data-Quality-Aware Learning: A real-time, model-agnostic quality assurance and selective retraining framework for ML pipelines.
+Author-email: Pratham Khatri <prathamdkhatri@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/Pratham1227-beep/dqal
+Project-URL: Repository, https://github.com/Pratham1227-beep/dqal
+Project-URL: Bug Tracker, https://github.com/Pratham1227-beep/dqal/issues
+Project-URL: Changelog, https://github.com/Pratham1227-beep/dqal/blob/main/CHANGELOG.md
+Keywords: machine-learning,data-quality,mlops,drift-detection,model-monitoring
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.23.0
+Requires-Dist: pandas>=1.5.0
+Requires-Dist: scipy>=1.9.0
+Requires-Dist: scikit-learn>=1.1.0
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: torch
+Requires-Dist: torch>=2.0.0; extra == "torch"
+Provides-Extra: viz
+Requires-Dist: matplotlib>=3.6.0; extra == "viz"
+Requires-Dist: seaborn>=0.12.0; extra == "viz"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Provides-Extra: all
+Requires-Dist: torch>=2.0.0; extra == "all"
+Requires-Dist: matplotlib>=3.6.0; extra == "all"
+Requires-Dist: seaborn>=0.12.0; extra == "all"
+Requires-Dist: pytest>=7.0.0; extra == "all"
+Dynamic: license-file
+
+# 🛡️ DQAL — Data-Quality-Aware Learning
+
+[![DQAL CI](https://github.com/Pratham1227-beep/dqal/actions/workflows/ci.yml/badge.svg)](https://github.com/Pratham1227-beep/dqal)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+> **Real-time, model-agnostic inference quality scoring, 3-tier gating (`SERVE` / `FLAG` / `ABSTAIN`), and selective retraining orchestration.**
+
+---
+
+## 1. Problem Statement
+
+Deployed machine learning models fail silently. A model trained on clean, stationary data continues generating overconfident predictions when:
+- **Upstream features drop out** (missing fields, API breaking changes, sensor timeouts)
+- **Input distributions drift** (seasonal shifts, demographic changes, covariate shifts)
+- **Extreme noise/outliers slip through** (hardware faults, corrupted data joins)
+
+Traditional monitoring detects these weeks after downstream accuracy degrades. **DQAL** provides a lightweight, inline guardrail layer that inspects each prediction batch in real time, computes an interpretable Quality Score $Q \in [0, 1]$, gates execution, and safely triggers retraining.
+
+---
+
+## 2. System Architecture
+
+```
+                    ┌─────────────────────────────────────────┐
+   Incoming batch → │           1. QUALITY SCORER             │
+   of data          │   • Missingness (Feature null rates)    │
+                    │   • Drift (Continuous PSI / Cat JS)     │
+                    │   • Outlier (Isolation Forest / Mahala) │
+                    └────────────────────┬────────────────────┘
+                                         │ Q ∈ [0, 1], Sub-signals
+                                         ▼
+                    ┌─────────────────────────────────────────┐
+                    │            2. TRIGGER LOGIC             │
+                    │   • Q > 0.80       →  SERVE             │
+                    │   • 0.50 < Q ≤ 0.80 →  FLAG              │
+                    │   • Q ≤ 0.50       →  ABSTAIN           │
+                    │   • Hysteresis dead-band & confirmation │
+                    └────────────────────┬────────────────────┘
+                                         │ Decision
+                    ┌────────────────────┴────────────────────┐
+                    ▼                                         ▼
+          ┌───────────────────┐                     ┌───────────────────┐
+          │  3. WRAPPED MODEL │                     │ 4. SQLITE LOGGER  │
+          │ (Sklearn/PyTorch) │                     │ (Privacy-safe: no │
+          │  Serve/Abstain    │                     │  raw PII by def)  │
+          └───────────────────┘                     └─────────┬─────────┘
+                                                              │
+                                                              ▼
+                                                    ┌───────────────────┐
+                                                    │ 5. ORCHESTRATOR   │
+                                                    │ • Volume checks   │
+                                                    │ • Label gating    │
+                                                    │ • Val. promotion  │
+                                                    └─────────┬─────────┘
+                                                              │
+                                                              ▼
+                                                    ┌───────────────────┐
+                                                    │ 6. TERMINAL LOGS  │
+                                                    │ (Real-time stdout │
+                                                    │  telemetry & Q)   │
+                                                    └───────────────────┘
+```
+
+---
+
+## 3. Key Highlights & Features
+
+- **⚡ Model-Agnostic Adapter**: Wraps any `scikit-learn` estimator, `PyTorch nn.Module`, or custom callable without modifying underlying model code.
+- **📊 Interpretable Quality Score $Q$**: Combines Missingness, Population Stability Index (PSI), and Isolation Forest anomaly scores into a single metric $Q \in [0, 1]$.
+- **🚦 3-Tier State Machine with Hysteresis**: Prevents state flapping near threshold boundaries with dead-bands and consecutive confirmation counts.
+- **🔒 Privacy-Preserving Telemetry**: Logs statistical metrics to SQLite; disables raw feature logging by default to prevent PII exposure.
+- **🔄 Validated Retraining & Rollback**: Enforces minimum data volume and verified label availability before retraining. Only promotes candidate models if validation accuracy beats active models.
+- **💻 Terminal-First Telemetry & Inspection**: Real-time command-line logging of $Q$, gating decisions (`SERVE`/`FLAG`/`ABSTAIN`), sub-signal diagnostics, and model transition events directly in terminal output.
+
+---
+
+## 4. Installation & Setup
+
+```bash
+# Clone repository
+git clone https://github.com/Pratham1227-beep/dqal.git
+cd dqal
+
+# Install in editable mode with all optional dependencies (visualization, PyTorch, dev)
+pip install -e ".[all]"
+```
+
+---
+
+## 5. Quickstart Example
+
+### Wrapping a Scikit-Learn Model
+
+```python
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.datasets import load_breast_cancer
+from sklearn.model_selection import train_test_split
+from dqal import DQAL, DQALConfig
+
+# 1. Train your baseline model
+data = load_breast_cancer(as_frame=True)
+X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.3, random_state=42)
+
+model = RandomForestClassifier(n_estimators=100, random_state=42).fit(X_train, y_train)
+
+# 2. Wrap with DQAL and fit baseline distributions
+# CRITICAL: Baseline distributions must be captured from clean training data!
+dqal = DQAL(model=model, model_version="v1.0.0")
+dqal.fit_baseline(X_train)
+
+# 3. Predict on incoming batches
+result = dqal.predict(X_test.iloc[:50])
+
+print(f"Quality Score Q: {result.Q:.3f}")
+print(f"Gating Decision: {result.decision}")  # SERVE, FLAG, or ABSTAIN
+print(f"Sub-signals:     {result.signals}")
+print(f"Predictions:     {result.predictions[:5]}")
+```
+
+### Wrapping a PyTorch Model
+
+```python
+import torch
+import torch.nn as nn
+from dqal import DQAL
+
+torch_model = nn.Sequential(
+    nn.Linear(30, 16),
+    nn.ReLU(),
+    nn.Linear(16, 2)
+)
+
+dqal = DQAL(model=torch_model, model_version="v1.0.0-torch")
+dqal.fit_baseline(X_train)
+
+result = dqal.predict(X_test.iloc[:20])
+```
+
+---
+
+## 6. Configuration (`configs/default_config.yaml`)
+
+```yaml
+weights:
+  missingness: 0.30
+  drift: 0.40
+  outlier: 0.30
+
+thresholds:
+  abstain_below: 0.50
+  flag_below: 0.80
+
+hysteresis:
+  deadband: 0.03
+  consecutive_batches: 2
+
+logging:
+  db_path: "dqal_telemetry.db"
+  log_raw_features: false  # Opt-in only
+
+retrain:
+  min_flagged_batches: 5
+  min_samples: 100
+  require_labels: true
+  min_improvement_delta: 0.01
+```
+
+---
+
+## 7. Empirical Validation & Benchmarks
+
+We evaluated DQAL across a progressive 60-batch degradation simulation (`benchmarks/run_benchmark.py`):
+1. **Pristine Baseline** (Batches 0–11)
+2. **Mild Covariate Shift** (Batches 12–23)
+3. **Missing Fields Spike** (Batches 24–35)
+4. **Severe Noise & Outliers** (Batches 36–47)
+5. **Catastrophic Drift** (Batches 48–59)
+
+### Benchmark Results
+
+| Metric | Target | DQAL Result | Status |
+| :--- | :--- | :--- | :--- |
+| **Pearson Correlation ($r$)** | $r < -0.70$ or $> 0.70$ | **$r = 0.8502$** ($p = 8.45 \times 10^{-18}$) | Passed |
+| **Inference Overhead** | $< 50$ ms / batch | **$49.79$ ms / batch** | Passed |
+| **Early Warning Lead Time** | $> 0$ batches | **$9$ batches lead time** | Passed |
+| **Label Safety Enforced** | $100\%$ | **Blocked unlabelled retraining** | Passed |
+
+![DQAL Benchmark Results](benchmark_results.png)
+
+---
+
+## 8. Inspecting Telemetry in Terminal
+
+DQAL logs structured, privacy-safe metrics per batch to SQLite (`dqal_telemetry.db`) and displays real-time gating decisions and sub-signals in your terminal:
+
+```python
+from dqal.logger import TelemetryLogger
+
+# Inspect logged telemetry in terminal
+logger = TelemetryLogger("dqal_telemetry.db")
+df = logger.get_telemetry_df()
+print(df[["batch_id", "Q", "decision", "missing_signal", "drift_signal", "outlier_signal"]].tail(10))
+```
+
+Or query directly using the SQLite CLI:
+
+```bash
+sqlite3 dqal_telemetry.db "SELECT batch_id, Q, decision, model_version FROM predictions_telemetry ORDER BY id DESC LIMIT 10;"
+```
+
+---
+
+## 9. Running Tests
+
+```bash
+pytest -v
+```
+
+---
+
+## 10. Known Limitations
+
+- **Row vs. Batch Granularity**: Scoring single isolated rows has higher variance due to sample statistics. Batch mode ($\ge 30$ rows) provides robust distribution estimation.
+- **Concept Drift vs. Anomaly**: DQAL detects statistical covariate and quality shift. Distinguishing permanent regime shift from temporary noise is surfaced to operators via the `FLAG` state rather than automated blind retraining.

dqal-0.1.0/README.md

@@ -0,0 +1,225 @@
+# 🛡️ DQAL — Data-Quality-Aware Learning
+
+[![DQAL CI](https://github.com/Pratham1227-beep/dqal/actions/workflows/ci.yml/badge.svg)](https://github.com/Pratham1227-beep/dqal)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+> **Real-time, model-agnostic inference quality scoring, 3-tier gating (`SERVE` / `FLAG` / `ABSTAIN`), and selective retraining orchestration.**
+
+---
+
+## 1. Problem Statement
+
+Deployed machine learning models fail silently. A model trained on clean, stationary data continues generating overconfident predictions when:
+- **Upstream features drop out** (missing fields, API breaking changes, sensor timeouts)
+- **Input distributions drift** (seasonal shifts, demographic changes, covariate shifts)
+- **Extreme noise/outliers slip through** (hardware faults, corrupted data joins)
+
+Traditional monitoring detects these weeks after downstream accuracy degrades. **DQAL** provides a lightweight, inline guardrail layer that inspects each prediction batch in real time, computes an interpretable Quality Score $Q \in [0, 1]$, gates execution, and safely triggers retraining.
+
+---
+
+## 2. System Architecture
+
+```
+                    ┌─────────────────────────────────────────┐
+   Incoming batch → │           1. QUALITY SCORER             │
+   of data          │   • Missingness (Feature null rates)    │
+                    │   • Drift (Continuous PSI / Cat JS)     │
+                    │   • Outlier (Isolation Forest / Mahala) │
+                    └────────────────────┬────────────────────┘
+                                         │ Q ∈ [0, 1], Sub-signals
+                                         ▼
+                    ┌─────────────────────────────────────────┐
+                    │            2. TRIGGER LOGIC             │
+                    │   • Q > 0.80       →  SERVE             │
+                    │   • 0.50 < Q ≤ 0.80 →  FLAG              │
+                    │   • Q ≤ 0.50       →  ABSTAIN           │
+                    │   • Hysteresis dead-band & confirmation │
+                    └────────────────────┬────────────────────┘
+                                         │ Decision
+                    ┌────────────────────┴────────────────────┐
+                    ▼                                         ▼
+          ┌───────────────────┐                     ┌───────────────────┐
+          │  3. WRAPPED MODEL │                     │ 4. SQLITE LOGGER  │
+          │ (Sklearn/PyTorch) │                     │ (Privacy-safe: no │
+          │  Serve/Abstain    │                     │  raw PII by def)  │
+          └───────────────────┘                     └─────────┬─────────┘
+                                                              │
+                                                              ▼
+                                                    ┌───────────────────┐
+                                                    │ 5. ORCHESTRATOR   │
+                                                    │ • Volume checks   │
+                                                    │ • Label gating    │
+                                                    │ • Val. promotion  │
+                                                    └─────────┬─────────┘
+                                                              │
+                                                              ▼
+                                                    ┌───────────────────┐
+                                                    │ 6. TERMINAL LOGS  │
+                                                    │ (Real-time stdout │
+                                                    │  telemetry & Q)   │
+                                                    └───────────────────┘
+```
+
+---
+
+## 3. Key Highlights & Features
+
+- **⚡ Model-Agnostic Adapter**: Wraps any `scikit-learn` estimator, `PyTorch nn.Module`, or custom callable without modifying underlying model code.
+- **📊 Interpretable Quality Score $Q$**: Combines Missingness, Population Stability Index (PSI), and Isolation Forest anomaly scores into a single metric $Q \in [0, 1]$.
+- **🚦 3-Tier State Machine with Hysteresis**: Prevents state flapping near threshold boundaries with dead-bands and consecutive confirmation counts.
+- **🔒 Privacy-Preserving Telemetry**: Logs statistical metrics to SQLite; disables raw feature logging by default to prevent PII exposure.
+- **🔄 Validated Retraining & Rollback**: Enforces minimum data volume and verified label availability before retraining. Only promotes candidate models if validation accuracy beats active models.
+- **💻 Terminal-First Telemetry & Inspection**: Real-time command-line logging of $Q$, gating decisions (`SERVE`/`FLAG`/`ABSTAIN`), sub-signal diagnostics, and model transition events directly in terminal output.
+
+---
+
+## 4. Installation & Setup
+
+```bash
+# Clone repository
+git clone https://github.com/Pratham1227-beep/dqal.git
+cd dqal
+
+# Install in editable mode with all optional dependencies (visualization, PyTorch, dev)
+pip install -e ".[all]"
+```
+
+---
+
+## 5. Quickstart Example
+
+### Wrapping a Scikit-Learn Model
+
+```python
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.datasets import load_breast_cancer
+from sklearn.model_selection import train_test_split
+from dqal import DQAL, DQALConfig
+
+# 1. Train your baseline model
+data = load_breast_cancer(as_frame=True)
+X_train, X_test, y_train, y_test = train_test_split(data.data, data.target, test_size=0.3, random_state=42)
+
+model = RandomForestClassifier(n_estimators=100, random_state=42).fit(X_train, y_train)
+
+# 2. Wrap with DQAL and fit baseline distributions
+# CRITICAL: Baseline distributions must be captured from clean training data!
+dqal = DQAL(model=model, model_version="v1.0.0")
+dqal.fit_baseline(X_train)
+
+# 3. Predict on incoming batches
+result = dqal.predict(X_test.iloc[:50])
+
+print(f"Quality Score Q: {result.Q:.3f}")
+print(f"Gating Decision: {result.decision}")  # SERVE, FLAG, or ABSTAIN
+print(f"Sub-signals:     {result.signals}")
+print(f"Predictions:     {result.predictions[:5]}")
+```
+
+### Wrapping a PyTorch Model
+
+```python
+import torch
+import torch.nn as nn
+from dqal import DQAL
+
+torch_model = nn.Sequential(
+    nn.Linear(30, 16),
+    nn.ReLU(),
+    nn.Linear(16, 2)
+)
+
+dqal = DQAL(model=torch_model, model_version="v1.0.0-torch")
+dqal.fit_baseline(X_train)
+
+result = dqal.predict(X_test.iloc[:20])
+```
+
+---
+
+## 6. Configuration (`configs/default_config.yaml`)
+
+```yaml
+weights:
+  missingness: 0.30
+  drift: 0.40
+  outlier: 0.30
+
+thresholds:
+  abstain_below: 0.50
+  flag_below: 0.80
+
+hysteresis:
+  deadband: 0.03
+  consecutive_batches: 2
+
+logging:
+  db_path: "dqal_telemetry.db"
+  log_raw_features: false  # Opt-in only
+
+retrain:
+  min_flagged_batches: 5
+  min_samples: 100
+  require_labels: true
+  min_improvement_delta: 0.01
+```
+
+---
+
+## 7. Empirical Validation & Benchmarks
+
+We evaluated DQAL across a progressive 60-batch degradation simulation (`benchmarks/run_benchmark.py`):
+1. **Pristine Baseline** (Batches 0–11)
+2. **Mild Covariate Shift** (Batches 12–23)
+3. **Missing Fields Spike** (Batches 24–35)
+4. **Severe Noise & Outliers** (Batches 36–47)
+5. **Catastrophic Drift** (Batches 48–59)
+
+### Benchmark Results
+
+| Metric | Target | DQAL Result | Status |
+| :--- | :--- | :--- | :--- |
+| **Pearson Correlation ($r$)** | $r < -0.70$ or $> 0.70$ | **$r = 0.8502$** ($p = 8.45 \times 10^{-18}$) | Passed |
+| **Inference Overhead** | $< 50$ ms / batch | **$49.79$ ms / batch** | Passed |
+| **Early Warning Lead Time** | $> 0$ batches | **$9$ batches lead time** | Passed |
+| **Label Safety Enforced** | $100\%$ | **Blocked unlabelled retraining** | Passed |
+
+![DQAL Benchmark Results](benchmark_results.png)
+
+---
+
+## 8. Inspecting Telemetry in Terminal
+
+DQAL logs structured, privacy-safe metrics per batch to SQLite (`dqal_telemetry.db`) and displays real-time gating decisions and sub-signals in your terminal:
+
+```python
+from dqal.logger import TelemetryLogger
+
+# Inspect logged telemetry in terminal
+logger = TelemetryLogger("dqal_telemetry.db")
+df = logger.get_telemetry_df()
+print(df[["batch_id", "Q", "decision", "missing_signal", "drift_signal", "outlier_signal"]].tail(10))
+```
+
+Or query directly using the SQLite CLI:
+
+```bash
+sqlite3 dqal_telemetry.db "SELECT batch_id, Q, decision, model_version FROM predictions_telemetry ORDER BY id DESC LIMIT 10;"
+```
+
+---
+
+## 9. Running Tests
+
+```bash
+pytest -v
+```
+
+---
+
+## 10. Known Limitations
+
+- **Row vs. Batch Granularity**: Scoring single isolated rows has higher variance due to sample statistics. Batch mode ($\ge 30$ rows) provides robust distribution estimation.
+- **Concept Drift vs. Anomaly**: DQAL detects statistical covariate and quality shift. Distinguishing permanent regime shift from temporary noise is surfaced to operators via the `FLAG` state rather than automated blind retraining.

dqal-0.1.0/benchmarks/__init__.py

@@ -0,0 +1,7 @@
+"""
+Benchmark module for DQAL.
+"""
+from benchmarks.simulator import DegradationSimulator, DegradationPhase
+from benchmarks.run_benchmark import run_full_benchmark
+
+__all__ = ["DegradationSimulator", "DegradationPhase", "run_full_benchmark"]