checkdrift 1.0.0__tar.gz

checkdrift-1.0.0/.github/workflows/publish.yml

@@ -0,0 +1,27 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write  # Required for trusted publishing
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.14"
+
+      - name: Install build tools
+        run: pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

checkdrift-1.0.0/.gitignore

@@ -0,0 +1,33 @@
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+*.egg
+dist/
+build/
+.eggs/
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Local config
+*.local
+.env

checkdrift-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Valentyn Danylchuk
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

checkdrift-1.0.0/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.4
+Name: checkdrift
+Version: 1.0.0
+Summary: One-line drift detection for ML APIs. Like Pydantic or a rate limiter, but for data drift.
+Project-URL: Homepage, https://github.com/valdanylchuk/driftdetect
+Project-URL: Repository, https://github.com/valdanylchuk/driftdetect
+Project-URL: Issues, https://github.com/valdanylchuk/driftdetect/issues
+Author-email: Valentyn Danylchuk <val@danylchuk.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: drift-detection,fastapi,machine-learning,mlops,monitoring
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: FastAPI
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.21.0
+Requires-Dist: scipy>=1.7.0
+Provides-Extra: dev
+Requires-Dist: fastapi>=0.100.0; extra == 'dev'
+Requires-Dist: httpx>=0.24.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# checkdrift
+
+One-line drift detection for ML APIs. Like Pydantic or a rate limiter, but for data drift.
+
+Just add @check_drift decorator to your FastAPI endpoint:
+
+```python
+@app.post("/predict")
+@check_drift(baseline="baseline.json")
+async def predict(application: LoanApplication):
+    return model.predict(application)
+```
+
+## Installation
+
+```bash
+pip install checkdrift
+```
+
+## What It Does
+
+- Maintains a sliding window of recent requests
+- Computes drift metrics every N requests (default: 50)
+- Logs warnings when drift is detected
+- Minimal impact on your endpoint response (about 1ms in my tests)
+
+Uses PSI (Population Stability Index) and KS test - industry standards from banking.
+
+## Baseline Format
+
+```json
+{"distributions": {"feature1": [1.0, 2.0, ...], "feature2": [...]}}
+```
+
+See [examples/lendingclub](examples/lendingclub) for a complete example with sample data.
+
+## Options
+
+```python
+@check_drift(
+    baseline="baseline.json",
+    window_size=100,       # Sliding window size
+    check_interval=50,     # Check every N requests
+    on_drift=my_callback,  # Optional callback
+)
+```
+
+## License
+
+MIT

checkdrift-1.0.0/README.md

@@ -0,0 +1,50 @@
+# checkdrift
+
+One-line drift detection for ML APIs. Like Pydantic or a rate limiter, but for data drift.
+
+Just add @check_drift decorator to your FastAPI endpoint:
+
+```python
+@app.post("/predict")
+@check_drift(baseline="baseline.json")
+async def predict(application: LoanApplication):
+    return model.predict(application)
+```
+
+## Installation
+
+```bash
+pip install checkdrift
+```
+
+## What It Does
+
+- Maintains a sliding window of recent requests
+- Computes drift metrics every N requests (default: 50)
+- Logs warnings when drift is detected
+- Minimal impact on your endpoint response (about 1ms in my tests)
+
+Uses PSI (Population Stability Index) and KS test - industry standards from banking.
+
+## Baseline Format
+
+```json
+{"distributions": {"feature1": [1.0, 2.0, ...], "feature2": [...]}}
+```
+
+See [examples/lendingclub](examples/lendingclub) for a complete example with sample data.
+
+## Options
+
+```python
+@check_drift(
+    baseline="baseline.json",
+    window_size=100,       # Sliding window size
+    check_interval=50,     # Check every N requests
+    on_drift=my_callback,  # Optional callback
+)
+```
+
+## License
+
+MIT

checkdrift-1.0.0/checkdrift/__init__.py

@@ -0,0 +1,25 @@
+"""
+checkdrift - One-line drift detection for ML APIs.
+
+"Like Pydantic or a rate limiter, but for data drift."
+"""
+
+from .decorators import (
+    check_drift,
+    DriftMonitor,
+    DriftCallback,
+    DriftCheckResult,
+    DriftSeverity,
+    FeatureDriftResult,
+)
+
+__version__ = "1.0.0"
+
+__all__ = [
+    "check_drift",
+    "DriftMonitor",
+    "DriftCallback",
+    "DriftCheckResult",
+    "DriftSeverity",
+    "FeatureDriftResult",
+]

checkdrift-1.0.0/checkdrift/decorators.py

@@ -0,0 +1,280 @@
+"""
+Drift detection decorator for FastAPI endpoints.
+"""
+
+import functools
+import json
+import logging
+from collections import deque
+from dataclasses import dataclass, field
+from enum import Enum
+from pathlib import Path
+from typing import Callable
+
+import numpy as np
+from scipy import stats
+
+logger = logging.getLogger("checkdrift")
+
+
+class DriftSeverity(Enum):
+    """Drift severity levels."""
+    OK = "ok"
+    WARNING = "warning"  # Moderate drift detected
+    ALARM = "alarm"      # Significant drift detected
+
+
+@dataclass
+class FeatureDriftResult:
+    """Drift detection result for a single feature."""
+    feature: str
+    psi: float
+    ks_stat: float
+    ks_pvalue: float
+    wasserstein: float
+    severity: DriftSeverity
+
+    @property
+    def is_drifted(self) -> bool:
+        return self.severity != DriftSeverity.OK
+
+
+@dataclass
+class DriftCheckResult:
+    """Complete drift check result passed to callback."""
+    request_count: int
+    features: dict[str, FeatureDriftResult] = field(default_factory=dict)
+
+    @property
+    def severity(self) -> DriftSeverity:
+        """Overall severity (worst across all features)."""
+        if any(f.severity == DriftSeverity.ALARM for f in self.features.values()):
+            return DriftSeverity.ALARM
+        if any(f.severity == DriftSeverity.WARNING for f in self.features.values()):
+            return DriftSeverity.WARNING
+        return DriftSeverity.OK
+
+    @property
+    def drifted_features(self) -> list[str]:
+        return [name for name, f in self.features.items() if f.is_drifted]
+
+
+# Type alias for callback
+DriftCallback = Callable[[DriftCheckResult], None]
+
+
+def compute_psi(reference: np.ndarray, current: np.ndarray, bins: int = 10) -> float:
+    """
+    Compute Population Stability Index between two distributions.
+    """
+    _, bin_edges = np.histogram(reference, bins=bins)
+
+    ref_counts, _ = np.histogram(reference, bins=bin_edges)
+    curr_counts, _ = np.histogram(current, bins=bin_edges)
+
+    # Convert to proportions, avoid division by zero
+    ref_pct = (ref_counts + 1) / (len(reference) + bins)
+    curr_pct = (curr_counts + 1) / (len(current) + bins)
+
+    # PSI formula: sum((curr - ref) * ln(curr / ref))
+    psi = np.sum((curr_pct - ref_pct) * np.log(curr_pct / ref_pct))
+    return float(psi)
+
+
+class DriftMonitor:
+    """
+    Monitors feature distributions for drift against a baseline.
+
+    Maintains a sliding window of recent observations and periodically
+    checks for drift using PSI, KS test, and Wasserstein distance.
+
+    Thresholds (banking industry standard):
+    - PSI > 0.2: Significant drift
+    - PSI > 0.1: Moderate drift
+    - KS p-value < 0.05: Statistically significant shift
+    """
+
+    PSI_WARNING = 0.1
+    PSI_ALARM = 0.2
+    KS_PVALUE_THRESHOLD = 0.05
+
+    def __init__(
+        self,
+        baseline_path: str | Path,
+        window_size: int = 100,
+        check_interval: int = 50,
+        psi_threshold: float = 0.2,
+        ks_pvalue_threshold: float = 0.05,
+        callback: DriftCallback | None = None,
+    ):
+        self.baseline_path = Path(baseline_path)
+        self.window_size = window_size
+        self.check_interval = check_interval
+        self.psi_threshold = psi_threshold
+        self.ks_pvalue_threshold = ks_pvalue_threshold
+        self.callback = callback
+
+        # Load baseline distributions
+        with open(self.baseline_path) as f:
+            self.baseline = json.load(f)
+
+        self.features = list(self.baseline["distributions"].keys())
+
+        # Sliding window per feature
+        self.windows: dict[str, deque] = {
+            feat: deque(maxlen=window_size) for feat in self.features
+        }
+        self.request_count = 0
+        self.last_check_results: DriftCheckResult | None = None
+
+    def push(self, values: dict[str, float]) -> None:
+        """Add observation to sliding windows."""
+        for feat in self.features:
+            if feat in values:
+                self.windows[feat].append(values[feat])
+        self.request_count += 1
+
+        if self.request_count % self.check_interval == 0:
+            self._check_drift()
+
+    def _determine_severity(self, psi: float, ks_pvalue: float) -> DriftSeverity:
+        """Determine drift severity combining PSI and KS test."""
+        if psi > self.PSI_ALARM:
+            return DriftSeverity.ALARM
+        if psi > self.PSI_WARNING and ks_pvalue < self.ks_pvalue_threshold:
+            return DriftSeverity.ALARM
+        if psi > self.PSI_WARNING or ks_pvalue < self.ks_pvalue_threshold:
+            return DriftSeverity.WARNING
+        return DriftSeverity.OK
+
+    def _check_drift(self) -> None:
+        """Run drift detection on current windows."""
+        result = DriftCheckResult(request_count=self.request_count)
+
+        for feat in self.features:
+            if len(self.windows[feat]) < self.check_interval:
+                continue
+
+            current = np.array(self.windows[feat])
+            reference = np.array(self.baseline["distributions"][feat])
+
+            psi = compute_psi(reference, current)
+            ks_stat, ks_pvalue = stats.ks_2samp(reference, current)
+            wasserstein = stats.wasserstein_distance(reference, current)
+
+            severity = self._determine_severity(psi, ks_pvalue)
+
+            result.features[feat] = FeatureDriftResult(
+                feature=feat,
+                psi=round(psi, 4),
+                ks_stat=round(ks_stat, 4),
+                ks_pvalue=round(ks_pvalue, 4),
+                wasserstein=round(wasserstein, 4),
+                severity=severity,
+            )
+
+        self.last_check_results = result
+
+        if result.severity != DriftSeverity.OK:
+            self._handle_drift(result)
+
+    def _handle_drift(self, result: DriftCheckResult) -> None:
+        """Log drift or invoke callback."""
+        if self.callback:
+            try:
+                self.callback(result)
+            except Exception as e:
+                logger.error(f"[checkdrift] Callback error: {e}")
+        else:
+            drifted = result.drifted_features
+            severity = result.severity.value.upper()
+
+            summary = {
+                feat: {
+                    "psi": r.psi,
+                    "ks_p": r.ks_pvalue,
+                    "wasserstein": r.wasserstein,
+                }
+                for feat, r in result.features.items()
+                if r.is_drifted
+            }
+
+            log_msg = (
+                f"Drift {severity} in {drifted} "
+                f"after {result.request_count} requests: {summary}"
+            )
+
+            if result.severity == DriftSeverity.ALARM:
+                logger.warning(log_msg)
+            else:
+                logger.info(log_msg)
+
+
+# Global registry of monitors
+_monitors: dict[str, DriftMonitor] = {}
+
+
+def check_drift(
+    baseline: str | Path,
+    window_size: int = 100,
+    check_interval: int = 50,
+    psi_threshold: float = 0.2,
+    ks_pvalue_threshold: float = 0.05,
+    on_drift: DriftCallback | None = None,
+):
+    """
+    Decorator for drift detection on FastAPI endpoints.
+
+    Monitors incoming request distributions against a baseline using
+    industry-standard metrics:
+    - PSI (Population Stability Index): >0.2 = significant, >0.1 = moderate
+    - KS test p-value: <0.05 = statistically significant shift
+    - Wasserstein distance: magnitude of distribution shift
+
+    Args:
+        baseline: Path to baseline JSON file with reference distributions
+        window_size: Number of recent requests to keep in sliding window
+        check_interval: Check drift every N requests
+        psi_threshold: PSI threshold for drift alert (default 0.2)
+        ks_pvalue_threshold: KS test p-value threshold (default 0.05)
+        on_drift: Callback invoked when drift is detected
+
+    Example:
+        @app.post("/predict")
+        @check_drift(baseline="baseline.json")
+        async def predict(application: LoanApplication):
+            return model.predict(application)
+    """
+    def decorator(func: Callable) -> Callable:
+        endpoint_name = func.__name__
+
+        if endpoint_name not in _monitors:
+            _monitors[endpoint_name] = DriftMonitor(
+                baseline_path=baseline,
+                window_size=window_size,
+                check_interval=check_interval,
+                psi_threshold=psi_threshold,
+                ks_pvalue_threshold=ks_pvalue_threshold,
+                callback=on_drift,
+            )
+
+        monitor = _monitors[endpoint_name]
+
+        @functools.wraps(func)
+        async def wrapper(*args, **kwargs):
+            for arg in kwargs.values():
+                if hasattr(arg, "model_dump"):  # Pydantic v2
+                    data = arg.model_dump()
+                    features = {
+                        k: float(v) for k, v in data.items()
+                        if isinstance(v, (int, float)) and k in monitor.features
+                    }
+                    monitor.push(features)
+                    break
+
+            return await func(*args, **kwargs)
+
+        wrapper.drift_monitor = monitor
+        return wrapper
+
+    return decorator

checkdrift-1.0.0/examples/__init__.py

@@ -0,0 +1 @@
+"""checkdrift examples."""

checkdrift-1.0.0/examples/lendingclub/README.md

@@ -0,0 +1,64 @@
+# LendingClub Example
+
+Demonstrates drift detection on a loan triage API using LendingClub data.
+
+## Files
+
+- `api.py` - FastAPI app with `@check_drift` decorator
+- `models.py` - Pydantic models and mock ML model
+- `test_run.py` - Demo script showing drift detection
+- `generate_baseline.py` - Script to create baseline.json from CSV
+- `data/lendingclub_2015_sample.csv.gz` - Reference data (1% sample, ~4k rows)
+- `data/lendingclub_2017_sample.csv.gz` - Test data with drift (1% sample)
+- `baseline.json` - Pre-generated baseline from 2015 data
+
+## Quick Start
+
+```bash
+# Install dependencies
+pip install checkdrift fastapi uvicorn
+
+# Run the API
+uvicorn examples.lendingclub.api:app --port 8000
+
+# Test endpoint
+curl -X POST http://localhost:8000/triage \
+  -H "Content-Type: application/json" \
+  -d '{"annual_inc": 75000, "dti": 18.5, "loan_amnt": 15000, "int_rate": 11.99}'
+```
+
+## Demo: Detecting Drift
+
+Run the test script to send requests to the API and see drift detection logs:
+
+```bash
+python examples/lendingclub/test_run.py
+```
+
+Output:
+
+```
+checkdrift - WARNING - Drift ALARM in ['annual_inc', 'dti', 'loan_amnt', 'int_rate'] after 50 requests: {'annual_inc': {'psi': 0.5026, ...}, 'dti': {'psi': 0.1764, ...}, 'loan_amnt': {'psi': 0.1239, ...}, 'int_rate': {'psi': 0.3146, ...}}
+checkdrift - WARNING - Drift ALARM in ['annual_inc', 'dti', 'loan_amnt', 'int_rate'] after 100 requests: ...
+checkdrift - WARNING - Drift ALARM in ['annual_inc', 'dti', 'loan_amnt', 'int_rate'] after 150 requests: ...
+Sending 200 requests to /triage...
+Drift logs will appear below:
+
+Done.
+```
+
+The 2017 data triggers **ALARM** on multiple features - the distribution shifted between 2015 (baseline) and 2017.
+
+## Regenerate Baseline
+
+```bash
+python generate_baseline.py data/lendingclub_2015_sample.csv.gz baseline.json
+```
+
+## Data
+
+The sample datasets are 1% random samples of [LendingClub loan data](https://www.kaggle.com/datasets/wordsforthewise/lending-club) for two years:
+- **2015**: Reference distribution (baseline)
+- **2017**: Shows drift over 2 years
+
+Features: `annual_inc`, `dti`, `loan_amnt`, `int_rate`

checkdrift-1.0.0/examples/lendingclub/__init__.py

@@ -0,0 +1 @@
+"""LendingClub example for checkdrift."""

checkdrift-1.0.0/examples/lendingclub/api.py

@@ -0,0 +1,63 @@
+"""
+Loan Triage API - Example with Drift Detection
+
+Demo service showing how to add drift detection to a FastAPI endpoint.
+
+Usage:
+    uvicorn examples.lendingclub.api:app --port 8000
+
+Endpoints:
+    POST /triage - Score a loan application
+    GET  /health - Service health check
+"""
+
+from contextlib import asynccontextmanager
+from pathlib import Path
+
+from fastapi import FastAPI, HTTPException
+
+from checkdrift import check_drift
+
+from .models import LoanApplication, TriageResult, LoanTriageModel
+
+
+BASELINE_PATH = Path(__file__).parent / "baseline.json"
+
+model: LoanTriageModel | None = None
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    """Load model on startup."""
+    global model
+    model = LoanTriageModel(seed=42)
+    yield
+    model = None
+
+
+app = FastAPI(
+    title="Loan Triage API",
+    description="Demo API with drift detection using checkdrift.",
+    version="1.0.0",
+    lifespan=lifespan,
+)
+
+
+@app.get("/health")
+async def health_check() -> dict:
+    """Service health check."""
+    return {"status": "healthy", "model_loaded": model is not None}
+
+
+@app.post("/triage", response_model=TriageResult)
+@check_drift(baseline=BASELINE_PATH)
+async def triage_application(application: LoanApplication) -> TriageResult:
+    """
+    Score a loan application and assign to review queue.
+
+    Returns risk score and recommended review tier.
+    """
+    if model is None:
+        raise HTTPException(status_code=503, detail="Model not loaded")
+
+    return model.predict(application)