driftguard-ai-sdk 1.0.0__tar.gz

driftguard_ai_sdk-1.0.0/PKG-INFO

@@ -0,0 +1,39 @@
+Metadata-Version: 2.4
+Name: driftguard-ai-sdk
+Version: 1.0.0
+Summary: DriftGuard — Autonomous Model Health Platform
+Author: DriftGuard Team
+Requires-Python: >=3.9
+Requires-Dist: numpy>=1.24
+Requires-Dist: httpx>=0.24
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: river>=0.21.2
+Requires-Dist: scikit-learn>=1.3
+Provides-Extra: server
+Requires-Dist: fastapi==0.111.0; extra == "server"
+Requires-Dist: uvicorn==0.28.1; extra == "server"
+Requires-Dist: pydantic==1.10.13; extra == "server"
+Requires-Dist: prometheus-client==0.20.0; extra == "server"
+Requires-Dist: pandas==2.2.2; extra == "server"
+Requires-Dist: redis==5.0.4; extra == "server"
+Requires-Dist: psycopg2-binary==2.9.9; extra == "server"
+Requires-Dist: sqlalchemy==2.0.30; extra == "server"
+Provides-Extra: validation
+Requires-Dist: great-expectations==0.18.15; extra == "validation"
+Requires-Dist: sqlalchemy==1.4.41; extra == "validation"
+Provides-Extra: evidently
+Requires-Dist: evidently==0.4.30; extra == "evidently"
+Provides-Extra: serving
+Requires-Dist: bentoml==1.2.0; extra == "serving"
+Requires-Dist: ray[serve]==2.10.0; extra == "serving"
+Provides-Extra: pipeline
+Requires-Dist: prefect==2.19.0; extra == "pipeline"
+Requires-Dist: zenml==0.57.0; extra == "pipeline"
+Provides-Extra: test
+Requires-Dist: pytest==8.2.0; extra == "test"
+Requires-Dist: pytest-asyncio==0.23.6; extra == "test"
+Dynamic: author
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary

driftguard_ai_sdk-1.0.0/README.md

@@ -0,0 +1,197 @@
+# 🛡️ DriftGuard — Autonomous Model Health Platform
+
+DriftGuard is a production-grade, self-healing MLOps platform designed to detect data drift, concept drift, and model degradation in real time, automatically trigger validating retraining pipelines, and progressively deploy champion models via progressive canary routers.
+
+---
+
+## 🏗️ Architecture Design
+
+```
+                     +---------------------------------------+
+                     |          Client Application           |
+                     +-------------------+-------------------+
+                                         |
+                                (Predict Telemetry)
+                                         v
+                     +-------------------+-------------------+
+                     |          DriftGuard SDK               |
+                     |  - Wrapper pattern intercept          |
+                     |  - River ADWIN concept drift checks   |
+                     +-------------------+-------------------+
+                                         |
+                                 (HTTP Telemetry)
+                                         v
+                     +-------------------+-------------------+
+                     |       DriftGuard FastAPI Core API     | <---+ NextJS Dashboard (:3000)
+                     |       - /register, /predict, /drift   | <---+ Grafana (:3001)
+                     |       - Prom metrics /metrics (:8000) |
+                     +-------------------+-------------------+
+                                         |
+                        (SLA Drift Breach Trigger)
+                                         v
+                     +-------------------+-------------------+
+                     |      Prefect Orchestration Server     |
+                     |      - drift_detection_flow (:4200)   |
+                     +-------------------+-------------------+
+                                         |
+                                 (Runs steps)
+                                         v
+                     +-------------------+-------------------+
+                     |      ZenML Step Training Pipelines    |
+                     |  - Step 1: Great Expectations Validate|
+                     |  - Step 2: Feast Feature Store Check  |
+                     |  - Step 3: Train & Track (MLflow/W&B) |
+                     |  - Step 4: Validate (>1% boost check) |
+                     |  - Step 5: Canary Progressive Deploy  |
+                     |  - Step 6: Immutable JSON Ledger & PDF|
+                     +-------------------+-------------------+
+                                         |
+                            (Progressive Split Promotes)
+                                         v
+                     +-------------------+-------------------+
+                     |       BentoML & Ray Serve Fleet       |
+                     |       - canary_router: 10%->100%      |
+                     |       - SLA Monitoring & Rollbacks    |
+                     +---------------------------------------+
+```
+
+---
+
+## ⚡ Prerequisites
+
+To run and configure DriftGuard, ensure the following are installed:
+- **Python 3.11** only (Ray and BentoML have incomplete 3.12 support).
+- **Docker & Docker Compose** (for multi-service orchestration).
+- **kubectl & Helm** (optional, for Kubernetes deployments).
+- **HashiCorp Terraform** (optional, for AWS cloud provisioning).
+
+---
+
+## 🚀 Quick Start in 5 Lines
+
+Wrap any scikit-learn, PyTorch, or HuggingFace model with DriftGuard SDK to track predictions, compute concept drift, and initiate auto-healing:
+
+```python
+from driftguard import DriftGuard
+
+# 1. Initialize DriftGuard
+dg = DriftGuard(model_id="fraud-detector-v1", api_url="http://localhost:8000", drift_threshold=0.15, auto_retrain=True)
+
+# 2. Wrap model seamlessly
+model = dg.wrap(trained_sklearn_model)
+
+# 3. Predict normally - DriftGuard tracks inputs, outputs, and triggers retrain on drift!
+prediction = model.predict(features)
+```
+
+---
+
+## 📦 Installation & Setup
+
+### 1. Local Package Installation
+Clone the repository and install the DriftGuard package locally:
+```bash
+git clone https://github.com/your-repo/DriftGuard.git
+cd DriftGuard
+pip install -e .
+```
+
+To install validation pipelines dependencies (Great Expectations + SQLAlchemy 1.4 pin) separately:
+```bash
+pip install -e ".[validation]"
+```
+
+### 2. Launch Platform Services
+Spin up the entire 8-service DriftGuard stack (FastAPI, NextJS dashboard, MLflow, Prefect, Postgres, Redis, Prometheus, Grafana) instantly:
+```bash
+docker-compose -f infra/docker-compose.yml up --build -d
+```
+
+---
+
+## ⚙️ SDK Configuration Parameters
+
+The `DriftGuard` class accepts the following parameters:
+
+| Parameter | Type | Default | Description |
+|---|---|---|---|
+| `model_id` | `str` | *Required* | Unique name identifier of the model. |
+| `api_url` | `str` | `http://localhost:8000` | Gateway endpoint of the DriftGuard API. |
+| `drift_threshold` | `float` | `0.15` | Limit before concept drift alert and retraining triggers. |
+| `auto_retrain` | `bool` | `True` | Automatically triggers retraining flow on threshold breach. |
+
+---
+
+## 📡 API Gateway Documentation
+
+DriftGuard Core API runs on port `8000`. Key REST endpoints include:
+
+### `POST /register`
+Registers a model for monitoring.
+- **Request Body:**
+  ```json
+  {
+    "model_id": "fraud-detector-v1",
+    "drift_threshold": 0.15,
+    "reference_data_path": "./data/baseline.parquet",
+    "features": ["amount", "location_score", "velocity"]
+  }
+  ```
+- **Response:** `{"status": "registered", "model_id": "fraud-detector-v1"}`
+
+### `POST /predict/{model_id}`
+SDK telemetry gateway recording prediction details and updating gauges.
+- **Request Body:**
+  ```json
+  {
+    "features": [1.2, 0.4, 9.8],
+    "prediction": [1.0],
+    "drift_score": 0.08
+  }
+  ```
+
+### `GET /drift/{model_id}`
+Fetches the last 100 historical drift scores for Recharts charts rendering.
+
+### `POST /retrain/{model_id}`
+Manually triggers the background retraining pipeline flow.
+
+### `GET /metrics`
+Exposes system health gauges for Prometheus scrapers in OpenMetrics format.
+
+---
+
+## 🖥️ Dashboards & Observability Portals
+
+Once the docker services are online:
+1. **NextJS UI Dashboard:** Navigate to [http://localhost:3000](http://localhost:3000) to review models list, drift histories, vertical retraining timelines, and searchable audit logs.
+2. **MLflow Registry:** Access [http://localhost:5000](http://localhost:5000) to review runs parameters, artifacts (confusion matrix plots), and staging/production champions.
+3. **Prefect Dashboard:** Access [http://localhost:4200](http://localhost:4200) to inspect flows execution history.
+4. **Grafana Dashboards:** Open [http://localhost:3001](http://localhost:3001) (User: `admin` | Pass: `admin`) to view pre-provisioned telemetry panels scraping predictions, drift rates, accuracy levels, and quantiles latency.
+
+---
+
+## 🧪 Running Unit Tests
+
+Run all unit tests verifying ADWIN concept detectors, Great Expectations validators, canary routing splits, emergency rollbacks, and cryptographic audit log chains:
+```bash
+pytest tests/ -v
+```
+
+---
+
+## ☁️ Deploying to AWS Cloud (Terraform)
+
+Deploy DriftGuard core infrastructure to Amazon Web Services:
+```bash
+cd infra/terraform
+terraform init
+terraform plan
+terraform apply -var="db_password=SecurePasswordPass22!"
+```
+This provisions:
+- Amazon EKS cluster for progressive Kubernetes canary Rollouts.
+- Amazon RDS PostgreSQL database for MLflow, Prefect, and platform metadata.
+- Amazon ElastiCache Redis for online real-time Feast features access.
+- Amazon S3 bucket for artifacts.
+- Amazon ECR for Docker images.

driftguard_ai_sdk-1.0.0/driftguard/__init__.py

@@ -0,0 +1,16 @@
+"""
+DriftGuard — public package entry point.
+
+Users import from here:
+    from driftguard import DriftGuard
+"""
+from .tracker import DriftGuard, DriftGuardModelWrapper
+from .callback_runner import RetrainerCallbackRunner
+from .config import settings
+
+__all__ = [
+    "DriftGuard",
+    "DriftGuardModelWrapper",
+    "RetrainerCallbackRunner",
+    "settings",
+]

driftguard_ai_sdk-1.0.0/driftguard/alert.py

@@ -0,0 +1,89 @@
+"""
+DriftGuard Alerting & Notification Client.
+Handles distribution of system alert updates to console logs and webhook endpoints like Slack.
+"""
+import httpx
+import logging
+from typing import Dict, Any, Optional
+
+from driftguard.config import settings
+
+logger = logging.getLogger("DriftGuard.Alert")
+
+def send_alert(
+    event_type: str,
+    message: str,
+    details: Optional[Dict[str, Any]] = None,
+    webhook_url: Optional[str] = None
+) -> bool:
+    """
+    Distributes alert notification updates to platform targets.
+    
+    Args:
+        event_type: Category of the alert (e.g., 'drift_detected', 'retrain_triggered', 'validation_failed', 'model_promoted').
+        message: Readable summary message string.
+        details: Optional JSON data properties containing details about the event.
+        webhook_url: Optional override of settings Slack Webhook URL.
+        
+    Returns:
+        True if successfully sent (or logged when offline), False on HTTP failure.
+    """
+    payload_details = details or {}
+    url = webhook_url or settings.SLACK_WEBHOOK_URL
+    
+    # 1. Standard Logger Print
+    log_msg = f"[ALERT - {event_type.upper()}] {message} | Details: {payload_details}"
+    if event_type in ["drift_detected", "validation_failed", "rollback"]:
+        logger.error(log_msg)
+    else:
+        logger.info(log_msg)
+
+    # 2. Slack Webhook Dispatch
+    if not url or "mock_webhook_url" in url:
+        logger.debug("No active Slack webhook configured. Skipping network alert.")
+        return True
+
+    try:
+        # Construct premium formatted Slack block
+        blocks = [
+            {
+                "type": "header",
+                "text": {
+                    "type": "plain_text",
+                    "text": f"🛡️ DriftGuard Alert: {event_type.replace('_', ' ').title()}",
+                    "emoji": True
+                }
+            },
+            {
+                "type": "section",
+                "text": {
+                    "type": "mrkdwn",
+                    "text": f"*Message:*\n{message}"
+                }
+            }
+        ]
+
+        if payload_details:
+            details_str = "\n".join([f"• *{k}:* {v}" for k, v in payload_details.items()])
+            blocks.append({
+                "type": "section",
+                "text": {
+                    "type": "mrkdwn",
+                    "text": f"*Metadata:*\n{details_str}"
+                }
+            })
+
+        slack_payload = {"blocks": blocks}
+        
+        with httpx.Client(timeout=3.0) as client:
+            resp = client.post(url, json=slack_payload)
+            if resp.status_code in [200, 201]:
+                logger.info("Successfully posted alert to Slack channel.")
+                return True
+            else:
+                logger.error(f"Slack webhook endpoint returned HTTP {resp.status_code}: {resp.text}")
+                return False
+                
+    except Exception as e:
+        logger.error(f"Failed to transmit Slack webhook: {e}")
+        return False

driftguard_ai_sdk-1.0.0/driftguard/callback_runner.py

@@ -0,0 +1,342 @@
+"""
+DriftGuard Retrainer Callback Runner.
+
+Executes user-registered ``@dg.retrainer`` callbacks entirely inside the
+SDK process — where the user's data, credentials, and environment live.
+
+Flow
+----
+1. Notify API: POST /retrain/{model_id} with source="sdk_callback"
+   → Server creates a DBRetrainingEvent record and returns event_id.
+   → Server does NOT spawn its own background pipeline.
+2. Invoke the user's retrainer function → get challenger model.
+3. Validate challenger vs champion using dg.set_validation_data().
+4. Report outcome: POST /retrain/{model_id}/complete.
+   → Server updates model version, accuracy, audit log, Prometheus, Slack.
+5. Update tracker._champion_model to the new champion.
+6. Reset tracker.retraining_triggered so future drift events can fire.
+
+NOTE: Production telemetry stored in dg_predictions is never touched.
+      Training data comes exclusively from the user's registered callback.
+"""
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING, Any, Optional, Tuple
+
+import httpx
+
+if TYPE_CHECKING:
+    from driftguard.tracker import DriftGuard
+
+logger = logging.getLogger("DriftGuard.CallbackRunner")
+
+
+class RetrainerCallbackRunner:
+    """
+    Orchestrates the full local retraining pipeline for a registered callback.
+
+    Parameters
+    ----------
+    tracker:
+        The ``DriftGuard`` instance that owns the callback.
+    """
+
+    def __init__(self, tracker: "DriftGuard") -> None:
+        self.tracker = tracker
+        self.api_url = tracker.api_url
+        self.model_id = tracker.model_id
+
+    # ------------------------------------------------------------------
+    # Public entry point
+    # ------------------------------------------------------------------
+
+    def run(self, drift_score: float) -> bool:
+        """
+        Execute the full callback-based retraining pipeline.
+
+        Returns
+        -------
+        bool
+            ``True`` if the challenger was promoted; ``False`` otherwise.
+        """
+        logger.info(
+            f"[{self.model_id}] Callback retraining pipeline started "
+            f"(drift_score={drift_score:.4f})"
+        )
+
+        # Step 1 — Record event on the server (no background task spawned)
+        current_version = self._get_current_version()
+        event_id = self._notify_retrain_start(drift_score)
+
+        try:
+            # Step 2 — Invoke user-registered callback
+            challenger_model = self._invoke_callback()
+
+            # Step 3 — Validate challenger against champion
+            validation_passed, champ_score, chall_score = self._validate(challenger_model)
+
+            print("\n===== VALIDATION RESULTS =====")
+            print("Champion:", champ_score)
+            print("Challenger:", chall_score)
+            print("Passed:", validation_passed)
+            print("==============================\n")
+
+            if not validation_passed:
+                reason = (
+                    f"Challenger accuracy {chall_score:.4f} did not beat "
+                    f"champion {champ_score:.4f} by ≥1%."
+                )
+                logger.warning(f"[{self.model_id}] {reason}")
+                self._report_failure(event_id=event_id, reason=reason, chall_score=chall_score)
+                return False
+
+            # Step 4 — Promote challenger
+            print("PROMOTION STAGE STARTED")
+            new_version = self._bump_version(current_version)
+            print(f"NEW VERSION = {new_version}")
+
+            # Persist challenger model before promotion
+            if self.tracker.project_id:
+                try:
+                    import joblib
+                    import os
+                    from driftguard.config import settings as _settings
+                    dir_path = os.path.join(
+                        _settings.ARTIFACT_ROOT,
+                        str(self.tracker.project_id),
+                        self.model_id
+                    )
+                    os.makedirs(dir_path, exist_ok=True)
+                    file_path = os.path.join(dir_path, f"version_{new_version}.pkl")
+                    joblib.dump(challenger_model, file_path)
+                    print(f"PERSISTED CHALLENGER MODEL TO {file_path}")
+                    logger.info(f"[{self.model_id}] Persisted challenger model before promotion to {file_path}")
+                except Exception as e:
+                    print(f"FAILED TO PERSIST CHALLENGER MODEL: {e}")
+                    logger.warning(f"[{self.model_id}] Failed to persist challenger model: {e}")
+
+            print("POSTING COMPLETION EVENT")
+            self._report_success(
+                event_id=event_id,
+                new_version=new_version,
+                new_accuracy=chall_score,
+                old_accuracy=champ_score,
+            )
+            print("COMPLETION EVENT POSTED")
+
+            # Step 5 — Update local champion reference so next comparison is correct
+            self.tracker._champion_model = challenger_model
+            print("CHAMPION UPDATED")
+
+            logger.info(
+                f"[{self.model_id}] Challenger promoted: "
+                f"{current_version} → {new_version}  "
+                f"(accuracy {champ_score:.4f} → {chall_score:.4f})"
+            )
+            return True
+
+        except Exception as exc:
+            import traceback
+            traceback.print_exc()
+            logger.error(
+                f"[{self.model_id}] Callback retraining pipeline failed: {exc}",
+                exc_info=True,
+            )
+            self._report_failure(event_id=event_id, reason=str(exc), chall_score=0.0)
+            return False
+
+        finally:
+            # Always reset so future drift events can trigger a new run
+            self.tracker.retraining_triggered = False
+
+    # ------------------------------------------------------------------
+    # Step implementations
+    # ------------------------------------------------------------------
+
+    def _invoke_callback(self) -> Any:
+        """
+        Call the user's ``@dg.retrainer`` function and return the trained model.
+
+        Raises
+        ------
+        RuntimeError
+            If the callback raises or returns ``None``.
+        """
+        fn = self.tracker._retrainer_fn
+        logger.info(f"[{self.model_id}] Invoking retrainer callback: {fn.__name__}()")
+        try:
+            model = fn()
+        except Exception as exc:
+            raise RuntimeError(
+                f"Retrainer callback '{fn.__name__}' raised an exception: {exc}"
+            ) from exc
+
+        if model is None:
+            raise ValueError(
+                f"Retrainer callback '{fn.__name__}' returned None. "
+                "The function must return a trained model object."
+            )
+        if not (hasattr(model, "predict") or callable(model)):
+            raise TypeError(
+                f"Retrainer callback '{fn.__name__}' returned an invalid model type '{type(model).__name__}'. "
+                "The model must have a 'predict' method or be callable."
+            )
+        return model
+
+    def _validate(self, challenger_model: Any) -> Tuple[bool, float, float]:
+        """
+        Compare challenger against the current champion.
+
+        If ``dg.set_champion()`` has not been called, the challenger is
+        promoted directly as the first known-good version.
+
+        If ``dg.set_validation_data()`` has not been called, validation is
+        skipped and the challenger is promoted with a warning.
+
+        Returns
+        -------
+        (validation_passed, champion_score, challenger_score)
+        """
+        champion_model = self.tracker._champion_model
+
+        if champion_model is None:
+            logger.info(
+                f"[{self.model_id}] No champion registered via dg.set_champion(). "
+                "Promoting challenger as first champion."
+            )
+            return True, 0.0, 1.0
+
+        val_features = self.tracker._validation_features
+        val_labels = self.tracker._validation_labels
+
+        if val_features is None or val_labels is None:
+            raise ValueError(
+                f"Validation data is missing for model '{self.model_id}'. "
+                "Validation datasets are required when retraining triggers."
+            )
+
+        from driftguard.validation import validate_challenger_vs_champion
+
+        return validate_challenger_vs_champion(
+            champion_model=champion_model,
+            challenger_model=challenger_model,
+            val_features=val_features,
+            val_labels=val_labels,
+            threshold_pct=0.01,  # challenger must beat champion by ≥1%
+        )
+
+    # ------------------------------------------------------------------
+    # API notification helpers
+    # ------------------------------------------------------------------
+
+    def _get_current_version(self) -> str:
+        """Fetch the model's current version string from the API."""
+        try:
+            headers = {"X-API-Key": self.tracker.api_key} if self.tracker.api_key else {}
+            with httpx.Client(timeout=5.0) as client:
+                resp = client.get(f"{self.api_url}/models/{self.model_id}", headers=headers)
+                if resp.status_code == 200:
+                    return resp.json().get("version", "1.0.0")
+        except Exception as exc:
+            logger.debug(f"[{self.model_id}] Could not fetch current version: {exc}")
+        return "1.0.0"
+
+    def _bump_version(self, current_version: str) -> str:
+        """Increment the patch segment of a semantic version string."""
+        try:
+            parts = current_version.split(".")
+            parts[-1] = str(int(parts[-1]) + 1)
+            return ".".join(parts)
+        except Exception:
+            return "1.0.1"
+
+    def _notify_retrain_start(self, drift_score: float) -> Optional[int]:
+        """
+        POST to ``/retrain/{model_id}`` with ``source="sdk_callback"``.
+
+        The server records the event and returns an ``event_id`` but does
+        NOT spawn its own background retraining task.
+        """
+        try:
+            headers = {"X-API-Key": self.tracker.api_key} if self.tracker.api_key else {}
+            with httpx.Client(timeout=5.0) as client:
+                resp = client.post(
+                    f"{self.api_url}/retrain/{self.model_id}",
+                    json={
+                        "drift_score": drift_score,
+                        "triggered_by": "automatic",
+                        "source": "sdk_callback",
+                    },
+                    headers=headers,
+                )
+                if resp.status_code == 200:
+                    data = resp.json()
+                    event_id = data.get("event_id")
+                    logger.debug(
+                        f"[{self.model_id}] Retrain event created, event_id={event_id}"
+                    )
+                    return event_id
+                logger.warning(
+                    f"[{self.model_id}] /retrain returned HTTP {resp.status_code}"
+                )
+        except Exception as exc:
+            logger.warning(
+                f"[{self.model_id}] Could not notify API of retrain start: {exc}"
+            )
+        return None
+
+    def _report_success(
+        self,
+        event_id: Optional[int],
+        new_version: str,
+        new_accuracy: float,
+        old_accuracy: float,
+    ) -> None:
+        """POST callback pipeline results to ``/retrain/{model_id}/complete``."""
+        try:
+            headers = {"X-API-Key": self.tracker.api_key} if self.tracker.api_key else {}
+            with httpx.Client(timeout=10.0) as client:
+                client.post(
+                    f"{self.api_url}/retrain/{self.model_id}/complete",
+                    json={
+                        "event_id": event_id,
+                        "validation_passed": True,
+                        "new_version": new_version,
+                        "new_accuracy": new_accuracy,
+                        "old_accuracy": old_accuracy,
+                        "error": None,
+                    },
+                    headers=headers,
+                )
+        except Exception as exc:
+            logger.warning(
+                f"[{self.model_id}] Could not report retrain success to API: {exc}"
+            )
+
+    def _report_failure(
+        self,
+        event_id: Optional[int],
+        reason: str,
+        chall_score: float,
+    ) -> None:
+        """Report a failed callback pipeline to ``/retrain/{model_id}/complete``."""
+        try:
+            headers = {"X-API-Key": self.tracker.api_key} if self.tracker.api_key else {}
+            with httpx.Client(timeout=10.0) as client:
+                client.post(
+                    f"{self.api_url}/retrain/{self.model_id}/complete",
+                    json={
+                        "event_id": event_id,
+                        "validation_passed": False,
+                        "new_version": None,
+                        "new_accuracy": chall_score,
+                        "old_accuracy": None,
+                        "error": reason,
+                    },
+                    headers=headers,
+                )
+        except Exception as exc:
+            logger.warning(
+                f"[{self.model_id}] Could not report retrain failure to API: {exc}"
+            )