modelshift 0.1.0__tar.gz

modelshift-0.1.0/PKG-INFO

@@ -0,0 +1,129 @@
+Metadata-Version: 2.4
+Name: modelshift
+Version: 0.1.0
+Summary: A lightweight machine learning drift monitoring and alerting engine.
+Author: Krishna
+Author-email: ryomensukuna2530@gmail.com
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: pandas
+Requires-Dist: numpy
+Requires-Dist: scipy
+Requires-Dist: requests
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# 🚦 ModelShift-Lite  
+### Label-Free Monitoring for Deployed Machine Learning Models
+
+> A lightweight, behavior-centric system to detect **silent reliability degradation** in deployed machine learning models — without requiring ground-truth labels.
+
+---
+
+## 📌 Why ModelShift-Lite?
+
+Machine learning models rarely fail loudly after deployment.  
+Instead, they **silently degrade** as real-world data changes — while true labels are unavailable for continuous evaluation.
+
+**ModelShift-Lite addresses this blind spot.**
+
+---
+
+## 🧩 Problem Statement
+
+Deployed machine learning models often degrade silently over time due to changing data distributions, while ground-truth labels are unavailable for continuous performance evaluation.
+
+---
+
+## 🎯 Project Objective
+
+Design a **label-free, post-deployment monitoring system** that tracks:
+
+- Data distribution shifts  
+- Prediction behavior instability  
+- Model reliability trends  
+
+to provide **early warning signals** of degradation **without modifying the deployed model**.
+
+---
+
+## 🚫 What This Project Does *Not* Do
+
+To maintain clarity of scope, ModelShift-Lite explicitly does **not**:
+
+- ❌ Retrain models  
+- ❌ Correct predictions  
+- ❌ Compute accuracy on production data  
+
+It focuses solely on **monitoring and interpretability**.
+
+---
+
+## 🧠 Core Idea (In Simple Terms)
+
+> *If we cannot measure correctness, we can still monitor behavior.*
+
+ModelShift-Lite observes how a model **reacts** to changing data and identifies signs of instability before failures become obvious.
+
+---
+
+## 🛠️ Key Components
+
+- **Reference Baseline Handling**  
+  Captures normal model behavior from historical or validation data
+
+- **Live Inference Monitoring**  
+  Tracks incoming production data and predictions
+
+- **Feature Drift Detection**  
+  Identifies changes in input distributions
+
+- **Prediction Behavior Analysis**  
+  Monitors confidence, stability, and output distribution shifts
+
+- **Model Health Scoring**  
+  Aggregates drift signals into an interpretable reliability indicator
+
+- **Visualization Dashboard**  
+  Displays trends, drift severity, and degradation warnings
+
+---
+Reference Data →
+→ Drift Detection → Health Scoring → Monitoring Dashboard
+Live Inference →
+
+
+*(Detailed architecture diagrams are provided in `/docs`)*
+
+---
+
+## 💻 Technology Stack
+
+- **Language:** Python  
+- **Data Processing:** NumPy, Pandas  
+- **Statistical Analysis:** SciPy  
+- **Visualization:** Streamlit, Matplotlib  
+- **Storage:** SQLite (local, replaceable)  
+
+---
+
+## 📂 Repository Structure
+
+```text
+modelshift-lite/
+├── modelshift/        # Core monitoring logic
+├── dashboard/         # Streamlit visualization app
+├── experiments/       # Drift simulation & analysis
+├── data/              # Reference & live data
+├── docs/              # Architecture and design docs
+└── README.md
+## 🏗️ High-Level Architecture
+

modelshift-0.1.0/README.md

@@ -0,0 +1,106 @@
+# 🚦 ModelShift-Lite  
+### Label-Free Monitoring for Deployed Machine Learning Models
+
+> A lightweight, behavior-centric system to detect **silent reliability degradation** in deployed machine learning models — without requiring ground-truth labels.
+
+---
+
+## 📌 Why ModelShift-Lite?
+
+Machine learning models rarely fail loudly after deployment.  
+Instead, they **silently degrade** as real-world data changes — while true labels are unavailable for continuous evaluation.
+
+**ModelShift-Lite addresses this blind spot.**
+
+---
+
+## 🧩 Problem Statement
+
+Deployed machine learning models often degrade silently over time due to changing data distributions, while ground-truth labels are unavailable for continuous performance evaluation.
+
+---
+
+## 🎯 Project Objective
+
+Design a **label-free, post-deployment monitoring system** that tracks:
+
+- Data distribution shifts  
+- Prediction behavior instability  
+- Model reliability trends  
+
+to provide **early warning signals** of degradation **without modifying the deployed model**.
+
+---
+
+## 🚫 What This Project Does *Not* Do
+
+To maintain clarity of scope, ModelShift-Lite explicitly does **not**:
+
+- ❌ Retrain models  
+- ❌ Correct predictions  
+- ❌ Compute accuracy on production data  
+
+It focuses solely on **monitoring and interpretability**.
+
+---
+
+## 🧠 Core Idea (In Simple Terms)
+
+> *If we cannot measure correctness, we can still monitor behavior.*
+
+ModelShift-Lite observes how a model **reacts** to changing data and identifies signs of instability before failures become obvious.
+
+---
+
+## 🛠️ Key Components
+
+- **Reference Baseline Handling**  
+  Captures normal model behavior from historical or validation data
+
+- **Live Inference Monitoring**  
+  Tracks incoming production data and predictions
+
+- **Feature Drift Detection**  
+  Identifies changes in input distributions
+
+- **Prediction Behavior Analysis**  
+  Monitors confidence, stability, and output distribution shifts
+
+- **Model Health Scoring**  
+  Aggregates drift signals into an interpretable reliability indicator
+
+- **Visualization Dashboard**  
+  Displays trends, drift severity, and degradation warnings
+
+---
+Reference Data →
+→ Drift Detection → Health Scoring → Monitoring Dashboard
+Live Inference →
+
+
+*(Detailed architecture diagrams are provided in `/docs`)*
+
+---
+
+## 💻 Technology Stack
+
+- **Language:** Python  
+- **Data Processing:** NumPy, Pandas  
+- **Statistical Analysis:** SciPy  
+- **Visualization:** Streamlit, Matplotlib  
+- **Storage:** SQLite (local, replaceable)  
+
+---
+
+## 📂 Repository Structure
+
+```text
+modelshift-lite/
+├── modelshift/        # Core monitoring logic
+├── dashboard/         # Streamlit visualization app
+├── experiments/       # Drift simulation & analysis
+├── data/              # Reference & live data
+├── docs/              # Architecture and design docs
+└── README.md
+## 🏗️ High-Level Architecture
+

modelshift-0.1.0/modelshift/__init__.py

@@ -0,0 +1,3 @@
+"""
+ModelShift-Lite package initialization.
+"""

modelshift-0.1.0/modelshift/baseline.py

@@ -0,0 +1,37 @@
+import pandas as pd
+
+
+class BaselineWindow:
+    """
+    Stores and manages reference baseline data
+    representing normal model behavior.
+    """
+
+    def __init__(self, data: pd.DataFrame):
+        self._validate(data)
+        self.data = data.copy()
+        self.feature_names = list(data.columns)
+        self.num_samples = len(data)
+
+    def _validate(self, data):
+        if not isinstance(data, pd.DataFrame):
+            raise TypeError("Baseline data must be a pandas DataFrame")
+
+        if data.empty:
+            raise ValueError("Baseline data cannot be empty")
+
+    def get_data(self) -> pd.DataFrame:
+        """
+        Returns a copy of baseline data.
+        """
+        return self.data.copy()
+
+    def summary(self) -> dict:
+        """
+        Returns basic metadata about the baseline window.
+        """
+        return {
+            "num_samples": self.num_samples,
+            "num_features": len(self.feature_names),
+            "feature_names": self.feature_names,
+        }

modelshift-0.1.0/modelshift/drift/__init__.py

@@ -0,0 +1,3 @@
+"""
+Drift detection modules.
+"""

modelshift-0.1.0/modelshift/drift/feature_drift.py

@@ -0,0 +1,50 @@
+import pandas as pd
+from scipy.stats import ks_2samp
+
+
+def compute_feature_drift(
+    baseline_data: pd.DataFrame,
+    live_data: pd.DataFrame
+) -> dict:
+    """
+    Compute feature-level drift using the Kolmogorov–Smirnov test.
+
+    Returns a dictionary:
+    {
+        feature_name: {
+            "ks_statistic": float,
+            "p_value": float
+        }
+    }
+    """
+
+    _validate_inputs(baseline_data, live_data)
+
+    drift_results = {}
+
+    for feature in baseline_data.columns:
+        baseline_values = baseline_data[feature].dropna()
+        live_values = live_data[feature].dropna()
+
+        ks_stat, p_value = ks_2samp(baseline_values, live_values)
+
+        drift_results[feature] = {
+            "ks_statistic": float(ks_stat),
+            "p_value": float(p_value),
+        }
+
+    return drift_results
+
+
+def _validate_inputs(baseline_data, live_data):
+    if not isinstance(baseline_data, pd.DataFrame):
+        raise TypeError("Baseline data must be a pandas DataFrame")
+
+    if not isinstance(live_data, pd.DataFrame):
+        raise TypeError("Live data must be a pandas DataFrame")
+
+    if baseline_data.empty or live_data.empty:
+        raise ValueError("Baseline and live data cannot be empty")
+
+    if list(baseline_data.columns) != list(live_data.columns):
+        raise ValueError("Baseline and live data must have identical features")

modelshift-0.1.0/modelshift/drift/prediction_drift.py

@@ -0,0 +1,111 @@
+from __future__ import annotations
+
+import numpy as np
+from scipy.stats import ks_2samp
+
+
+def compute_prediction_drift(
+    baseline_predictions: np.ndarray,
+    live_predictions: np.ndarray
+) -> dict:
+    """
+    Compute prediction behavior drift using:
+      1) KS-test on prediction probability distributions
+      2) Binary entropy change (mean entropy of predicted probabilities)
+
+    Notes:
+    - Expects 1D probability arrays (values in [0, 1]).
+    - Uses full binary entropy: -(p*log(p) + (1-p)*log(1-p))
+    """
+
+    baseline = _prepare_predictions("baseline", baseline_predictions)
+    live = _prepare_predictions("live", live_predictions)
+
+    # KS-test on prediction distributions
+    ks_stat, p_value = ks_2samp(baseline, live)
+
+    # Entropy analysis (full binary entropy)
+    baseline_entropy = _binary_entropy_mean(baseline)
+    live_entropy = _binary_entropy_mean(live)
+    entropy_change = live_entropy - baseline_entropy
+
+    # Lightweight shape/center diagnostics (useful for dashboards/reports)
+    baseline_mean = float(np.mean(baseline))
+    live_mean = float(np.mean(live))
+    baseline_std = float(np.std(baseline))
+    live_std = float(np.std(live))
+    baseline_median = float(np.median(baseline))
+    live_median = float(np.median(live))
+
+    return {
+        "ks_statistic": float(ks_stat),
+        "p_value": float(p_value),
+
+        "baseline_entropy": round(float(baseline_entropy), 6),
+        "live_entropy": round(float(live_entropy), 6),
+        "entropy_change": round(float(entropy_change), 6),
+        "abs_entropy_change": round(float(abs(entropy_change)), 6),
+
+        "baseline_mean_prob": round(baseline_mean, 6),
+        "live_mean_prob": round(live_mean, 6),
+        "mean_prob_shift": round(float(live_mean - baseline_mean), 6),
+
+        "baseline_median_prob": round(baseline_median, 6),
+        "live_median_prob": round(live_median, 6),
+        "median_prob_shift": round(float(live_median - baseline_median), 6),
+
+        "baseline_std_prob": round(baseline_std, 6),
+        "live_std_prob": round(live_std, 6),
+        "std_prob_shift": round(float(live_std - baseline_std), 6),
+
+        "n_baseline": int(baseline.size),
+        "n_live": int(live.size),
+    }
+
+
+def _binary_entropy_mean(preds: np.ndarray) -> float:
+    """
+    Mean binary entropy for probability predictions:
+      H(p) = -(p*log(p) + (1-p)*log(1-p))
+
+    Uses natural log (nats).
+    """
+    eps = 1e-9
+    p = np.clip(preds.astype(float), eps, 1.0 - eps)
+    entropy = -(p * np.log(p) + (1.0 - p) * np.log(1.0 - p))
+    return float(np.mean(entropy))
+
+
+def _prepare_predictions(name: str, arr) -> np.ndarray:
+    """
+    Validate and normalize prediction arrays to a clean 1D float numpy array.
+    """
+    if arr is None:
+        raise ValueError(f"{name.capitalize()} predictions cannot be None")
+
+    if not isinstance(arr, np.ndarray):
+        # Allow lists/Series while staying user-friendly
+        try:
+            arr = np.asarray(arr, dtype=float)
+        except Exception as exc:
+            raise TypeError(
+                f"{name.capitalize()} predictions must be a numpy array or array-like of numeric values"
+            ) from exc
+    else:
+        arr = arr.astype(float, copy=False)
+
+    arr = np.ravel(arr)
+
+    if arr.size == 0:
+        raise ValueError(f"{name.capitalize()} prediction array cannot be empty")
+
+    if not np.all(np.isfinite(arr)):
+        raise ValueError(f"{name.capitalize()} predictions contain NaN/Inf values")
+
+    # We treat these as probability predictions for entropy-based drift
+    if np.min(arr) < 0.0 or np.max(arr) > 1.0:
+        raise ValueError(
+            f"{name.capitalize()} predictions must be probability values in [0, 1]"
+        )
+
+    return arr

modelshift-0.1.0/modelshift/drift/severity.py

@@ -0,0 +1,239 @@
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+
+# ----------------------------
+# Basic per-signal thresholds
+# ----------------------------
+FEATURE_KS_LOW = 0.10
+FEATURE_KS_MEDIUM = 0.20
+FEATURE_KS_HIGH = 0.35
+
+PRED_KS_WARNING = 0.10
+PRED_KS_CRITICAL = 0.15
+
+ENTROPY_DELTA_WARNING = 0.01
+ENTROPY_DELTA_CRITICAL = 0.02
+
+
+def classify_severity(ks_statistic: float) -> str:
+    """
+    Classify severity from a KS-like drift signal.
+
+    Returns one of:
+      LOW / MEDIUM / HIGH / CRITICAL
+    """
+    ks = _safe_float(ks_statistic, default=0.0)
+    if ks < FEATURE_KS_LOW:
+        return "LOW"
+    if ks < FEATURE_KS_MEDIUM:
+        return "MEDIUM"
+    if ks < FEATURE_KS_HIGH:
+        return "HIGH"
+    return "CRITICAL"
+
+
+def compute_health_score(feature_drift_results: dict) -> float:
+    """
+    Compute overall model health score (0–100) from feature drift.
+    Higher score = healthier model.
+
+    Uses average feature KS:
+      health = max(0, 100 * (1 - avg_ks))
+    """
+    summary = summarize_feature_drift(feature_drift_results)
+    if summary["feature_count"] == 0:
+        raise ValueError("Feature drift results cannot be empty")
+
+    avg_ks = summary["avg_ks"]
+    health_score = max(0.0, 100.0 * (1.0 - avg_ks))
+    return round(float(health_score), 2)
+
+
+def summarize_feature_drift(feature_drift_results: Optional[dict]) -> Dict[str, Any]:
+    """
+    Extract feature drift summary stats from feature_drift_results.
+    Safe against missing/malformed values.
+    """
+    if not isinstance(feature_drift_results, dict):
+        return {
+            "feature_count": 0,
+            "avg_ks": 0.0,
+            "max_ks": 0.0,
+            "max_feature": None,
+            "ks_values": [],
+        }
+
+    ks_pairs: List[tuple[str, float]] = []
+    for feature, values in feature_drift_results.items():
+        if not isinstance(values, dict):
+            continue
+        ks = _safe_float(values.get("ks_statistic"), default=None)
+        if ks is None:
+            continue
+        ks_pairs.append((str(feature), ks))
+
+    if not ks_pairs:
+        return {
+            "feature_count": 0,
+            "avg_ks": 0.0,
+            "max_ks": 0.0,
+            "max_feature": None,
+            "ks_values": [],
+        }
+
+    ks_values = [ks for _, ks in ks_pairs]
+    max_feature, max_ks = max(ks_pairs, key=lambda x: x[1])
+
+    return {
+        "feature_count": len(ks_values),
+        "avg_ks": round(sum(ks_values) / len(ks_values), 6),
+        "max_ks": round(float(max_ks), 6),
+        "max_feature": max_feature,
+        "ks_values": [round(float(v), 6) for v in ks_values],
+    }
+
+
+def classify_drift_taxonomy(
+    feature_drift_results: Optional[dict] = None,
+    prediction_drift_results: Optional[dict] = None,
+) -> str:
+    """
+    Agreement/disagreement taxonomy between feature drift and prediction drift.
+
+    Returns one of:
+      STABLE
+      ROBUST_SHIFT              (feature drift high, prediction drift low)
+      SILENT_BEHAVIOR_DRIFT     (feature drift low, prediction drift high)
+      DEGRADING_DRIFT           (both high)
+    """
+    f_summary = summarize_feature_drift(feature_drift_results)
+    max_feature_ks = _safe_float(f_summary.get("max_ks"), default=0.0)
+
+    pred_ks = 0.0
+    if isinstance(prediction_drift_results, dict):
+        pred_ks = _safe_float(prediction_drift_results.get("ks_statistic"), default=0.0)
+
+    feature_high = max_feature_ks >= FEATURE_KS_MEDIUM
+    pred_high = pred_ks >= PRED_KS_WARNING
+
+    if not feature_high and not pred_high:
+        return "STABLE"
+    if feature_high and not pred_high:
+        return "ROBUST_SHIFT"
+    if (not feature_high) and pred_high:
+        return "SILENT_BEHAVIOR_DRIFT"
+    return "DEGRADING_DRIFT"
+
+
+def evaluate_drift_state(
+    feature_drift_results: Optional[dict] = None,
+    prediction_drift_results: Optional[dict] = None,
+) -> Dict[str, Any]:
+    """
+    Composite severity + status decision engine.
+
+    Combines:
+    - avg feature KS
+    - max feature KS
+    - prediction KS
+    - entropy delta
+
+    Returns a normalized decision payload with:
+      severity, status, taxonomy, health_score, signals, thresholds
+    """
+    f_summary = summarize_feature_drift(feature_drift_results)
+
+    avg_feature_ks = _safe_float(f_summary.get("avg_ks"), default=0.0)
+    max_feature_ks = _safe_float(f_summary.get("max_ks"), default=0.0)
+    pred_ks = 0.0
+    entropy_change = 0.0
+
+    if isinstance(prediction_drift_results, dict):
+        pred_ks = _safe_float(prediction_drift_results.get("ks_statistic"), default=0.0)
+        entropy_change = _safe_float(prediction_drift_results.get("entropy_change"), default=0.0)
+
+    # Normalize signals into 0..1 severity components
+    # (Threshold denominators chosen to align with your current observed ranges.)
+    avg_comp = min(1.0, avg_feature_ks / FEATURE_KS_MEDIUM)           # avg drift
+    max_comp = min(1.0, max_feature_ks / FEATURE_KS_HIGH)             # worst feature
+    pred_comp = min(1.0, pred_ks / PRED_KS_CRITICAL)                  # behavior drift
+    ent_comp = min(1.0, abs(entropy_change) / ENTROPY_DELTA_CRITICAL) # confidence shift
+
+    # Weighted composite score [0,1]
+    composite_score = (
+        0.30 * avg_comp +
+        0.25 * max_comp +
+        0.35 * pred_comp +
+        0.10 * ent_comp
+    )
+
+    severity = _classify_composite_severity(composite_score)
+
+    # Status favors prediction drift a bit more (behavior-centric monitoring)
+    if pred_ks >= PRED_KS_CRITICAL or max_feature_ks >= FEATURE_KS_HIGH:
+        status = "CRITICAL_DRIFT"
+    elif pred_ks >= PRED_KS_WARNING or max_feature_ks >= FEATURE_KS_MEDIUM or avg_feature_ks >= FEATURE_KS_LOW:
+        status = "WARNING_DRIFT"
+    else:
+        status = "STABLE"
+
+    taxonomy = classify_drift_taxonomy(feature_drift_results, prediction_drift_results)
+
+    # Health score is only meaningful if feature drift exists
+    health_score = None
+    if f_summary["feature_count"] > 0:
+        health_score = compute_health_score(feature_drift_results)
+
+    return {
+        "severity": severity,
+        "status": status,
+        "taxonomy": taxonomy,
+        "health_score": health_score,
+        "signals": {
+            "avg_feature_ks": round(avg_feature_ks, 6),
+            "max_feature_ks": round(max_feature_ks, 6),
+            "max_feature_name": f_summary.get("max_feature"),
+            "prediction_ks": round(pred_ks, 6),
+            "entropy_change": round(entropy_change, 6),
+            "composite_score": round(float(composite_score), 6),
+            "feature_count": int(f_summary.get("feature_count", 0)),
+        },
+        "thresholds": {
+            "feature_ks_low": FEATURE_KS_LOW,
+            "feature_ks_medium": FEATURE_KS_MEDIUM,
+            "feature_ks_high": FEATURE_KS_HIGH,
+            "pred_ks_warning": PRED_KS_WARNING,
+            "pred_ks_critical": PRED_KS_CRITICAL,
+            "entropy_delta_warning": ENTROPY_DELTA_WARNING,
+            "entropy_delta_critical": ENTROPY_DELTA_CRITICAL,
+        },
+    }
+
+
+# ----------------------------
+# Internal helpers
+# ----------------------------
+def _classify_composite_severity(score: float) -> str:
+    """
+    Composite score is already normalized 0..1.
+    """
+    s = max(0.0, min(1.0, _safe_float(score, default=0.0)))
+
+    if s < 0.20:
+        return "LOW"
+    if s < 0.45:
+        return "MEDIUM"
+    if s < 0.70:
+        return "HIGH"
+    return "CRITICAL"
+
+
+def _safe_float(value: Any, default: Optional[float] = 0.0) -> Optional[float]:
+    try:
+        if value is None:
+            return default
+        return float(value)
+    except (TypeError, ValueError):
+        return default