explainiverse 0.8.3__tar.gz → 0.8.5__tar.gz

{explainiverse-0.8.3 → explainiverse-0.8.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: explainiverse
-Version: 0.8.3
+Version: 0.8.5
 Summary: Unified, extensible explainability framework supporting 18 XAI methods including LIME, SHAP, LRP, TCAV, GradCAM, and more
 Home-page: https://github.com/jemsbhai/explainiverse
 License: MIT
@@ -44,7 +44,7 @@ Description-Content-Type: text/markdown
 | Feature | Description |
 |---------|-------------|
 | **18 Explainers** | LIME, KernelSHAP, TreeSHAP, Integrated Gradients, DeepLIFT, DeepSHAP, SmoothGrad, Saliency Maps, GradCAM/GradCAM++, LRP, TCAV, Anchors, Counterfactual, Permutation Importance, PDP, ALE, SAGE, ProtoDash |
-| **8 Evaluation Metrics** | Faithfulness (PGI, PGU, Comprehensiveness, Sufficiency, Correlation) and Stability (RIS, ROS, Lipschitz) |
+| **14 Evaluation Metrics** | Faithfulness (PGI, PGU, Comprehensiveness, Sufficiency, Correlation, Faithfulness Estimate, Monotonicity, Monotonicity-Nguyen, Pixel Flipping, Region Perturbation) and Stability (RIS, ROS, Lipschitz) |
 | **Unified API** | Consistent `BaseExplainer` interface with standardized `Explanation` output |
 | **Plugin Registry** | Filter explainers by scope, model type, data type; automatic recommendations |
 | **Framework Support** | Adapters for scikit-learn and PyTorch (with gradient computation) |
@@ -96,6 +96,11 @@ Explainiverse includes a comprehensive suite of evaluation metrics based on the
 | **Comprehensiveness** | Drop when removing top-k features | [DeYoung et al., 2020](https://arxiv.org/abs/1911.03429) |
 | **Sufficiency** | Prediction using only top-k features | [DeYoung et al., 2020](https://arxiv.org/abs/1911.03429) |
 | **Faithfulness Correlation** | Correlation between attribution and impact | [Bhatt et al., 2020](https://arxiv.org/abs/2005.00631) |
+| **Faithfulness Estimate** | Correlation of attributions with single-feature perturbation impact | [Alvarez-Melis & Jaakkola, 2018](https://arxiv.org/abs/1806.08049) |
+| **Monotonicity** | Sequential feature addition shows monotonic prediction increase | [Arya et al., 2019](https://arxiv.org/abs/1909.03012) |
+| **Monotonicity-Nguyen** | Spearman correlation between attributions and feature removal impact | [Nguyen & Martinez, 2020](https://arxiv.org/abs/2010.07455) |
+| **Pixel Flipping** | AUC of prediction degradation when removing features by importance | [Bach et al., 2015](https://doi.org/10.1371/journal.pone.0130140) |
+| **Region Perturbation** | AUC of prediction degradation when perturbing feature regions by importance | [Samek et al., 2015](https://arxiv.org/abs/1509.06321) |
 
 ### Stability Metrics
 
@@ -715,6 +720,16 @@ poetry run pytest tests/test_lrp.py::TestLRPConv2d -v
 - [x] Evaluation: Stability metrics (RIS, ROS, Lipschitz)
 - [x] PyTorch adapter with gradient support
 
+### In Progress 🔄
+- [ ] **Evaluation metrics expansion** - Adding 42 more metrics across 7 categories to exceed Quantus (37 metrics)
+  - Phase 1: Faithfulness (+9 metrics) - 4/12 complete
+  - Phase 2: Robustness (+7 metrics)
+  - Phase 3: Localisation (+8 metrics)
+  - Phase 4: Complexity (+4 metrics)
+  - Phase 5: Randomisation (+5 metrics)
+  - Phase 6: Axiomatic (+4 metrics)
+  - Phase 7: Fairness (+4 metrics)
+
 ### Planned 📋
 - [ ] Attention-based explanations (for Transformers)
 - [ ] TensorFlow/Keras adapter
@@ -734,7 +749,7 @@ If you use Explainiverse in your research, please cite:
   author = {Syed, Muntaser},
   year = {2025},
   url = {https://github.com/jemsbhai/explainiverse},
-  version = {0.8.0}
+  version = {0.8.4}
 }
 ```
 

{explainiverse-0.8.3 → explainiverse-0.8.5}/README.md

@@ -13,7 +13,7 @@
 | Feature | Description |
 |---------|-------------|
 | **18 Explainers** | LIME, KernelSHAP, TreeSHAP, Integrated Gradients, DeepLIFT, DeepSHAP, SmoothGrad, Saliency Maps, GradCAM/GradCAM++, LRP, TCAV, Anchors, Counterfactual, Permutation Importance, PDP, ALE, SAGE, ProtoDash |
-| **8 Evaluation Metrics** | Faithfulness (PGI, PGU, Comprehensiveness, Sufficiency, Correlation) and Stability (RIS, ROS, Lipschitz) |
+| **14 Evaluation Metrics** | Faithfulness (PGI, PGU, Comprehensiveness, Sufficiency, Correlation, Faithfulness Estimate, Monotonicity, Monotonicity-Nguyen, Pixel Flipping, Region Perturbation) and Stability (RIS, ROS, Lipschitz) |
 | **Unified API** | Consistent `BaseExplainer` interface with standardized `Explanation` output |
 | **Plugin Registry** | Filter explainers by scope, model type, data type; automatic recommendations |
 | **Framework Support** | Adapters for scikit-learn and PyTorch (with gradient computation) |
@@ -65,6 +65,11 @@ Explainiverse includes a comprehensive suite of evaluation metrics based on the
 | **Comprehensiveness** | Drop when removing top-k features | [DeYoung et al., 2020](https://arxiv.org/abs/1911.03429) |
 | **Sufficiency** | Prediction using only top-k features | [DeYoung et al., 2020](https://arxiv.org/abs/1911.03429) |
 | **Faithfulness Correlation** | Correlation between attribution and impact | [Bhatt et al., 2020](https://arxiv.org/abs/2005.00631) |
+| **Faithfulness Estimate** | Correlation of attributions with single-feature perturbation impact | [Alvarez-Melis & Jaakkola, 2018](https://arxiv.org/abs/1806.08049) |
+| **Monotonicity** | Sequential feature addition shows monotonic prediction increase | [Arya et al., 2019](https://arxiv.org/abs/1909.03012) |
+| **Monotonicity-Nguyen** | Spearman correlation between attributions and feature removal impact | [Nguyen & Martinez, 2020](https://arxiv.org/abs/2010.07455) |
+| **Pixel Flipping** | AUC of prediction degradation when removing features by importance | [Bach et al., 2015](https://doi.org/10.1371/journal.pone.0130140) |
+| **Region Perturbation** | AUC of prediction degradation when perturbing feature regions by importance | [Samek et al., 2015](https://arxiv.org/abs/1509.06321) |
 
 ### Stability Metrics
 
@@ -684,6 +689,16 @@ poetry run pytest tests/test_lrp.py::TestLRPConv2d -v
 - [x] Evaluation: Stability metrics (RIS, ROS, Lipschitz)
 - [x] PyTorch adapter with gradient support
 
+### In Progress 🔄
+- [ ] **Evaluation metrics expansion** - Adding 42 more metrics across 7 categories to exceed Quantus (37 metrics)
+  - Phase 1: Faithfulness (+9 metrics) - 4/12 complete
+  - Phase 2: Robustness (+7 metrics)
+  - Phase 3: Localisation (+8 metrics)
+  - Phase 4: Complexity (+4 metrics)
+  - Phase 5: Randomisation (+5 metrics)
+  - Phase 6: Axiomatic (+4 metrics)
+  - Phase 7: Fairness (+4 metrics)
+
 ### Planned 📋
 - [ ] Attention-based explanations (for Transformers)
 - [ ] TensorFlow/Keras adapter
@@ -703,7 +718,7 @@ If you use Explainiverse in your research, please cite:
   author = {Syed, Muntaser},
   year = {2025},
   url = {https://github.com/jemsbhai/explainiverse},
-  version = {0.8.0}
+  version = {0.8.4}
 }
 ```
 

{explainiverse-0.8.3 → explainiverse-0.8.5}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "explainiverse"
-version = "0.8.3"
+version = "0.8.5"
 description = "Unified, extensible explainability framework supporting 18 XAI methods including LIME, SHAP, LRP, TCAV, GradCAM, and more"
 authors = ["Muntaser Syed <jemsbhai@gmail.com>"]
 license = "MIT"

{explainiverse-0.8.3 → explainiverse-0.8.5}/src/explainiverse/__init__.py

@@ -34,7 +34,7 @@ from explainiverse.adapters.sklearn_adapter import SklearnAdapter
 from explainiverse.adapters import TORCH_AVAILABLE
 from explainiverse.engine.suite import ExplanationSuite
 
-__version__ = "0.8.3"
+__version__ = "0.8.5"
 
 __all__ = [
     # Core

{explainiverse-0.8.3 → explainiverse-0.8.5}/src/explainiverse/evaluation/__init__.py

@@ -43,6 +43,10 @@ from explainiverse.evaluation.faithfulness_extended import (
     compute_batch_monotonicity,
     compute_monotonicity_nguyen,
     compute_batch_monotonicity_nguyen,
+    compute_pixel_flipping,
+    compute_batch_pixel_flipping,
+    compute_region_perturbation,
+    compute_batch_region_perturbation,
 )
 
 __all__ = [
@@ -74,4 +78,8 @@ __all__ = [
     "compute_batch_monotonicity",
     "compute_monotonicity_nguyen",
     "compute_batch_monotonicity_nguyen",
+    "compute_pixel_flipping",
+    "compute_batch_pixel_flipping",
+    "compute_region_perturbation",
+    "compute_batch_region_perturbation",
 ]

{explainiverse-0.8.3 → explainiverse-0.8.5}/src/explainiverse/evaluation/faithfulness_extended.py

@@ -256,6 +256,396 @@ def compute_batch_faithfulness_estimate(
     }
 
 
+# =============================================================================
+# Metric 5: Region Perturbation (Samek et al., 2015)
+# =============================================================================
+
+def compute_region_perturbation(
+    model,
+    instance: np.ndarray,
+    explanation: Explanation,
+    baseline: Union[str, float, np.ndarray, Callable] = "mean",
+    background_data: np.ndarray = None,
+    target_class: int = None,
+    region_size: int = None,
+    use_absolute: bool = True,
+    return_curve: bool = False,
+) -> Union[float, Dict[str, Union[float, np.ndarray]]]:
+    """
+    Compute Region Perturbation score (Samek et al., 2015).
+    
+    Similar to Pixel Flipping, but operates on regions (groups) of features
+    rather than individual features. Features are divided into non-overlapping
+    regions, and regions are perturbed in order of their cumulative importance
+    (sum of attributions within the region).
+    
+    This metric is particularly relevant for image data where local spatial
+    correlations exist, but is also applicable to tabular data with groups
+    of related features.
+    
+    The score is the Area Under the perturbation Curve (AUC), normalized
+    to [0, 1]. Lower AUC indicates better faithfulness (faster degradation
+    when important regions are removed first).
+    
+    Args:
+        model: Model adapter with predict/predict_proba method
+        instance: Input instance (1D array)
+        explanation: Explanation object with feature_attributions
+        baseline: Baseline for feature removal ("mean", "median", scalar, array, callable)
+        background_data: Reference data for computing baseline (required for "mean"/"median")
+        target_class: Target class index for probability (default: predicted class)
+        region_size: Number of features per region. If None, defaults to max(1, n_features // 4)
+            For image-like data, this would correspond to patch size.
+        use_absolute: If True, sort regions by absolute attribution sum (default: True)
+        return_curve: If True, return full degradation curve and details
+        
+    Returns:
+        If return_curve=False: AUC score (float, 0 to 1, lower is better)
+        If return_curve=True: Dictionary with:
+            - 'auc': float - Area under the perturbation curve
+            - 'curve': np.ndarray - Normalized prediction values at each step
+            - 'predictions': np.ndarray - Raw prediction values
+            - 'region_order': list - Order in which regions were perturbed
+            - 'regions': list - List of feature indices in each region
+            - 'n_regions': int - Number of regions
+            - 'region_size': int - Size of each region
+        
+    References:
+        Samek, W., Binder, A., Montavon, G., Lapuschkin, S., & Müller, K. R. (2015).
+        Evaluating the Visualization of What a Deep Neural Network has Learned.
+        arXiv preprint arXiv:1509.06321.
+    """
+    instance = np.asarray(instance).flatten()
+    n_features = len(instance)
+    
+    # Get baseline values
+    baseline_values = compute_baseline_values(
+        baseline, background_data, n_features
+    )
+    
+    # Extract attributions as array
+    attr_array = _extract_attribution_array(explanation, n_features)
+    
+    # Determine region size
+    if region_size is None:
+        # Default: divide features into ~4 regions
+        region_size = max(1, n_features // 4)
+    region_size = max(1, min(region_size, n_features))  # Clamp to valid range
+    
+    # Create non-overlapping regions
+    regions = []
+    for start_idx in range(0, n_features, region_size):
+        end_idx = min(start_idx + region_size, n_features)
+        regions.append(list(range(start_idx, end_idx)))
+    
+    n_regions = len(regions)
+    
+    # Compute region importance (sum of attributions in each region)
+    region_importance = []
+    for region in regions:
+        if use_absolute:
+            importance = np.sum(np.abs(attr_array[region]))
+        else:
+            importance = np.sum(attr_array[region])
+        region_importance.append(importance)
+    
+    # Sort regions by importance (descending - most important first)
+    sorted_region_indices = np.argsort(-np.array(region_importance))
+    
+    # Determine target class
+    if target_class is None:
+        pred = get_prediction_value(model, instance.reshape(1, -1))
+        if isinstance(pred, np.ndarray) and pred.ndim > 0:
+            target_class = int(np.argmax(pred))
+        else:
+            target_class = 0
+    
+    # Get original prediction for the target class
+    original_pred = get_prediction_value(model, instance.reshape(1, -1))
+    if isinstance(original_pred, np.ndarray) and original_pred.ndim > 0 and len(original_pred) > target_class:
+        original_value = original_pred[target_class]
+    else:
+        original_value = float(original_pred)
+    
+    # Start with original instance
+    current = instance.copy()
+    
+    # Track predictions as regions are perturbed
+    predictions = [original_value]
+    
+    # Perturb regions one by one (most important first)
+    for region_idx in sorted_region_indices:
+        region = regions[region_idx]
+        
+        # Replace all features in this region with baseline
+        for feat_idx in region:
+            current[feat_idx] = baseline_values[feat_idx]
+        
+        # Get prediction
+        pred = get_prediction_value(model, current.reshape(1, -1))
+        if isinstance(pred, np.ndarray) and pred.ndim > 0 and len(pred) > target_class:
+            predictions.append(pred[target_class])
+        else:
+            predictions.append(float(pred))
+    
+    predictions = np.array(predictions)
+    
+    # Normalize predictions to [0, 1] relative to original
+    # curve[i] = prediction after perturbing i regions / original prediction
+    if abs(original_value) > 1e-10:
+        curve = predictions / original_value
+    else:
+        # Handle zero original prediction
+        curve = predictions
+    
+    # Compute AUC using trapezoidal rule
+    # x-axis: fraction of regions perturbed (0 to 1)
+    # y-axis: relative prediction value
+    x = np.linspace(0, 1, len(predictions))
+    auc = np.trapz(curve, x)
+    
+    if return_curve:
+        return {
+            "auc": float(auc),
+            "curve": curve,
+            "predictions": predictions,
+            "region_order": sorted_region_indices.tolist(),
+            "regions": regions,
+            "n_regions": n_regions,
+            "region_size": region_size,
+        }
+    
+    return float(auc)
+
+
+def compute_batch_region_perturbation(
+    model,
+    X: np.ndarray,
+    explanations: List[Explanation],
+    baseline: Union[str, float, np.ndarray, Callable] = "mean",
+    max_samples: int = None,
+    region_size: int = None,
+    use_absolute: bool = True,
+) -> Dict[str, float]:
+    """
+    Compute average Region Perturbation score over a batch of instances.
+    
+    Args:
+        model: Model adapter
+        X: Input data (2D array)
+        explanations: List of Explanation objects (one per instance)
+        baseline: Baseline for feature removal
+        max_samples: Maximum number of samples to evaluate
+        region_size: Number of features per region (default: n_features // 4)
+        use_absolute: If True, sort regions by absolute attribution sum
+        
+    Returns:
+        Dictionary with mean, std, min, max, and count of valid scores
+    """
+    n_samples = len(explanations)
+    if max_samples:
+        n_samples = min(n_samples, max_samples)
+    
+    scores = []
+    
+    for i in range(n_samples):
+        try:
+            score = compute_region_perturbation(
+                model, X[i], explanations[i],
+                baseline=baseline, background_data=X,
+                region_size=region_size,
+                use_absolute=use_absolute
+            )
+            if not np.isnan(score):
+                scores.append(score)
+        except Exception:
+            continue
+    
+    if not scores:
+        return {"mean": 0.0, "std": 0.0, "min": 0.0, "max": 0.0, "n_samples": 0}
+    
+    return {
+        "mean": float(np.mean(scores)),
+        "std": float(np.std(scores)),
+        "min": float(np.min(scores)),
+        "max": float(np.max(scores)),
+        "n_samples": len(scores),
+    }
+
+
+# =============================================================================
+# Metric 4: Pixel Flipping (Bach et al., 2015)
+# =============================================================================
+
+def compute_pixel_flipping(
+    model,
+    instance: np.ndarray,
+    explanation: Explanation,
+    baseline: Union[str, float, np.ndarray, Callable] = "mean",
+    background_data: np.ndarray = None,
+    target_class: int = None,
+    use_absolute: bool = True,
+    return_curve: bool = False,
+) -> Union[float, Dict[str, Union[float, np.ndarray]]]:
+    """
+    Compute Pixel Flipping score (Bach et al., 2015).
+    
+    Sequentially removes features in order of attributed importance (most
+    important first) and measures the cumulative prediction degradation.
+    A faithful explanation should cause rapid prediction drop when the
+    most important features are removed first.
+    
+    The score is the Area Under the perturbation Curve (AUC), normalized
+    to [0, 1]. Lower AUC indicates better faithfulness (faster degradation).
+    
+    Args:
+        model: Model adapter with predict/predict_proba method
+        instance: Input instance (1D array)
+        explanation: Explanation object with feature_attributions
+        baseline: Baseline for feature removal ("mean", "median", scalar, array, callable)
+        background_data: Reference data for computing baseline (required for "mean"/"median")
+        target_class: Target class index for probability (default: predicted class)
+        use_absolute: If True, sort features by absolute attribution value
+        return_curve: If True, return full degradation curve and predictions
+        
+    Returns:
+        If return_curve=False: AUC score (float, 0 to 1, lower is better)
+        If return_curve=True: Dictionary with 'auc', 'curve', 'predictions', 'feature_order'
+        
+    References:
+        Bach, S., et al. (2015). On Pixel-Wise Explanations for Non-Linear 
+        Classifier Decisions by Layer-Wise Relevance Propagation. PLOS ONE.
+    """
+    instance = np.asarray(instance).flatten()
+    n_features = len(instance)
+    
+    # Get baseline values
+    baseline_values = compute_baseline_values(
+        baseline, background_data, n_features
+    )
+    
+    # Extract attributions as array
+    attr_array = _extract_attribution_array(explanation, n_features)
+    
+    # Sort features by attribution (descending - most important first)
+    if use_absolute:
+        sorted_indices = np.argsort(-np.abs(attr_array))
+    else:
+        sorted_indices = np.argsort(-attr_array)
+    
+    # Determine target class
+    if target_class is None:
+        pred = get_prediction_value(model, instance.reshape(1, -1))
+        if isinstance(pred, np.ndarray) and pred.ndim > 0:
+            target_class = int(np.argmax(pred))
+        else:
+            target_class = 0
+    
+    # Get original prediction for the target class
+    original_pred = get_prediction_value(model, instance.reshape(1, -1))
+    if isinstance(original_pred, np.ndarray) and original_pred.ndim > 0 and len(original_pred) > target_class:
+        original_value = original_pred[target_class]
+    else:
+        original_value = float(original_pred)
+    
+    # Start with original instance
+    current = instance.copy()
+    
+    # Track predictions as features are removed
+    predictions = [original_value]
+    
+    # Remove features one by one (most important first)
+    for idx in sorted_indices:
+        # Remove this feature (replace with baseline)
+        current[idx] = baseline_values[idx]
+        
+        # Get prediction
+        pred = get_prediction_value(model, current.reshape(1, -1))
+        if isinstance(pred, np.ndarray) and pred.ndim > 0 and len(pred) > target_class:
+            predictions.append(pred[target_class])
+        else:
+            predictions.append(float(pred))
+    
+    predictions = np.array(predictions)
+    
+    # Normalize predictions to [0, 1] relative to original
+    # curve[i] = prediction after removing i features / original prediction
+    if abs(original_value) > 1e-10:
+        curve = predictions / original_value
+    else:
+        # Handle zero original prediction
+        curve = predictions
+    
+    # Compute AUC using trapezoidal rule
+    # x-axis: fraction of features removed (0 to 1)
+    # y-axis: relative prediction value
+    x = np.linspace(0, 1, len(predictions))
+    auc = np.trapz(curve, x)
+    
+    if return_curve:
+        return {
+            "auc": float(auc),
+            "curve": curve,
+            "predictions": predictions,
+            "feature_order": sorted_indices,
+            "n_features": n_features,
+        }
+    
+    return float(auc)
+
+
+def compute_batch_pixel_flipping(
+    model,
+    X: np.ndarray,
+    explanations: List[Explanation],
+    baseline: Union[str, float, np.ndarray, Callable] = "mean",
+    max_samples: int = None,
+    use_absolute: bool = True,
+) -> Dict[str, float]:
+    """
+    Compute average Pixel Flipping score over a batch of instances.
+    
+    Args:
+        model: Model adapter
+        X: Input data (2D array)
+        explanations: List of Explanation objects (one per instance)
+        baseline: Baseline for feature removal
+        max_samples: Maximum number of samples to evaluate
+        use_absolute: If True, sort features by absolute attribution value
+        
+    Returns:
+        Dictionary with mean, std, min, max, and count of valid scores
+    """
+    n_samples = len(explanations)
+    if max_samples:
+        n_samples = min(n_samples, max_samples)
+    
+    scores = []
+    
+    for i in range(n_samples):
+        try:
+            score = compute_pixel_flipping(
+                model, X[i], explanations[i],
+                baseline=baseline, background_data=X,
+                use_absolute=use_absolute
+            )
+            if not np.isnan(score):
+                scores.append(score)
+        except Exception:
+            continue
+    
+    if not scores:
+        return {"mean": 0.0, "std": 0.0, "min": 0.0, "max": 0.0, "n_samples": 0}
+    
+    return {
+        "mean": float(np.mean(scores)),
+        "std": float(np.std(scores)),
+        "min": float(np.min(scores)),
+        "max": float(np.max(scores)),
+        "n_samples": len(scores),
+    }
+
+
 # =============================================================================
 # Metric 3: Monotonicity-Nguyen (Nguyen et al., 2020)
 # =============================================================================