ai-critic 0.2.1__tar.gz → 0.2.3__tar.gz

ai_critic-0.2.3/PKG-INFO

@@ -0,0 +1,76 @@
+Metadata-Version: 2.4
+Name: ai-critic
+Version: 0.2.3
+Summary: Fast AI evaluator for scikit-learn models
+Author-email: Luiz Seabra <filipedemarco@yahoo.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: scikit-learn
+
+Performance under noise
+
+> Visualizations are optional and do not affect the decision logic.
+
+---
+
+## ⚙️ Main API
+
+### `AICritic(model, X, y)`
+
+* `model`: scikit-learn compatible estimator
+* `X`: feature matrix
+* `y`: target vector
+
+### `evaluate(view="all", plot=False)`
+
+* `view`: `"executive"`, `"technical"`, `"details"`, `"all"` or custom list
+* `plot`: generates graphs when `True`
+
+---
+
+## 🧠 What ai-critic Detects
+
+| Category | Risks |
+
+| ------------ | ---------------------------------------- |
+
+| 🔍 Data | Target Leakage, NaNs, Imbalance |
+
+| 🧱 Structure | Excessive Complexity, Overfitting |
+
+| 📈 Validation | Perfect or Statistically Suspicious CV |
+
+| 🧪 Robustness | Stable, Fragile, or Misleading |
+
+---
+
+## 🛡️ Best Practices
+
+* **CI/CD:** Use executive output as a *quality gate*
+* **Iteration:** Use technical output during tuning
+* **Governance:** Log detailed output
+* **Skepticism:** Never blindly trust a perfect CV
+
+---
+
+## 🧭 Use Cases
+
+* Pre-deployment Audit
+* ML Governance
+* CI/CD Pipelines
+* Risk Communication for Non-Technical Users
+
+---
+
+## 📄 License
+
+Distributed under the **MIT License**.
+
+---
+
+## 🧠 Final Note
+
+**ai-critic** is not a *benchmarking* tool. It's a **decision-making tool**.
+
+If a model fails here, it doesn't mean it's bad—it means it **shouldn't be trusted yet**.

ai_critic-0.2.3/README.md

@@ -0,0 +1,66 @@
+Performance under noise
+
+> Visualizations are optional and do not affect the decision logic.
+
+---
+
+## ⚙️ Main API
+
+### `AICritic(model, X, y)`
+
+* `model`: scikit-learn compatible estimator
+* `X`: feature matrix
+* `y`: target vector
+
+### `evaluate(view="all", plot=False)`
+
+* `view`: `"executive"`, `"technical"`, `"details"`, `"all"` or custom list
+* `plot`: generates graphs when `True`
+
+---
+
+## 🧠 What ai-critic Detects
+
+| Category | Risks |
+
+| ------------ | ---------------------------------------- |
+
+| 🔍 Data | Target Leakage, NaNs, Imbalance |
+
+| 🧱 Structure | Excessive Complexity, Overfitting |
+
+| 📈 Validation | Perfect or Statistically Suspicious CV |
+
+| 🧪 Robustness | Stable, Fragile, or Misleading |
+
+---
+
+## 🛡️ Best Practices
+
+* **CI/CD:** Use executive output as a *quality gate*
+* **Iteration:** Use technical output during tuning
+* **Governance:** Log detailed output
+* **Skepticism:** Never blindly trust a perfect CV
+
+---
+
+## 🧭 Use Cases
+
+* Pre-deployment Audit
+* ML Governance
+* CI/CD Pipelines
+* Risk Communication for Non-Technical Users
+
+---
+
+## 📄 License
+
+Distributed under the **MIT License**.
+
+---
+
+## 🧠 Final Note
+
+**ai-critic** is not a *benchmarking* tool. It's a **decision-making tool**.
+
+If a model fails here, it doesn't mean it's bad—it means it **shouldn't be trusted yet**.

ai_critic-0.2.3/ai_critic/critic.py

@@ -0,0 +1,132 @@
+from ai_critic.evaluators import (
+    robustness,
+    config,
+    data,
+    performance
+)
+from ai_critic.evaluators.summary import HumanSummary
+
+
+class AICritic:
+    """
+    Automated reviewer for scikit-learn models.
+
+    Produces a multi-layered risk assessment including:
+    - Data integrity analysis
+    - Model configuration sanity checks
+    - Performance evaluation (CV + learning curves)
+    - Robustness & leakage heuristics
+    - Human-readable executive and technical summaries
+    """
+
+    def __init__(self, model, X, y, random_state=None):
+        """
+        Parameters
+        ----------
+        model : sklearn-compatible estimator
+        X : np.ndarray
+            Feature matrix
+        y : np.ndarray
+            Target vector
+        random_state : int or None
+            Global seed for reproducibility (optional)
+        """
+        self.model = model
+        self.X = X
+        self.y = y
+        self.random_state = random_state
+
+    def evaluate(self, view="all", plot=False):
+        """
+        Evaluate the model.
+
+        Parameters
+        ----------
+        view : str or list
+            - "all" : full payload (default)
+            - "executive" : executive summary only
+            - "technical" : technical summary only
+            - "details" : low-level evaluator outputs
+            - list : subset of views (e.g. ["executive", "details"])
+        plot : bool
+            - True : generate plots (learning curve, heatmap, robustness)
+            - False : no plots
+
+        Returns
+        -------
+        dict
+            Evaluation payload according to selected view
+        """
+
+        # =========================
+        # Low-level evaluator outputs
+        # =========================
+        details = {}
+
+        # -------------------------
+        # Data analysis
+        # -------------------------
+        data_report = data.evaluate(
+            self.X,
+            self.y,
+            plot=plot
+        )
+        details["data"] = data_report
+
+        # -------------------------
+        # Model configuration sanity
+        # -------------------------
+        details["config"] = config.evaluate(
+            self.model,
+            n_samples=data_report["n_samples"],
+            n_features=data_report["n_features"]
+        )
+
+        # -------------------------
+        # Performance evaluation
+        # (CV strategy inferred automatically)
+        # -------------------------
+        details["performance"] = performance.evaluate(
+            self.model,
+            self.X,
+            self.y,
+            plot=plot
+        )
+
+        # -------------------------
+        # Robustness & leakage analysis
+        # -------------------------
+        details["robustness"] = robustness.evaluate(
+            self.model,
+            self.X,
+            self.y,
+            leakage_suspected=data_report["data_leakage"]["suspected"],
+            plot=plot
+        )
+
+        # =========================
+        # Human-centered summaries
+        # =========================
+        human_summary = HumanSummary().generate(details)
+
+        # =========================
+        # Full payload (PUBLIC API)
+        # =========================
+        payload = {
+            "executive": human_summary["executive_summary"],
+            "technical": human_summary["technical_summary"],
+            "details": details,
+            # Convenience shortcut (prevents KeyError in user code)
+            "performance": details["performance"]
+        }
+
+        # =========================
+        # View selector
+        # =========================
+        if view == "all":
+            return payload
+
+        if isinstance(view, list):
+            return {k: payload[k] for k in view if k in payload}
+
+        return payload.get(view)

ai_critic-0.2.3/ai_critic/evaluators/__init__.py

@@ -0,0 +1,11 @@
+from . import data
+from . import performance
+from . import robustness
+from . import config
+
+__all__ = [
+    "data",
+    "performance",
+    "robustness",
+    "config",
+]

ai_critic-0.2.3/ai_critic/evaluators/performance.py

@@ -0,0 +1,72 @@
+from sklearn.model_selection import cross_val_score, learning_curve
+import matplotlib.pyplot as plt
+import numpy as np
+
+from .validation import make_cv
+
+
+def evaluate(model, X, y, plot=False):
+    """
+    Avalia a performance do modelo usando validação cruzada
+    automaticamente adequada (StratifiedKFold ou KFold).
+    """
+
+    # =========================
+    # Cross-validation adaptativa
+    # =========================
+    cv = make_cv(y)
+
+    scores = cross_val_score(model, X, y, cv=cv)
+    mean = float(scores.mean())
+    std = float(scores.std())
+    suspicious = mean > 0.995
+
+    result = {
+        "cv_mean_score": mean,
+        "cv_std": std,
+        "suspiciously_perfect": suspicious,
+        "validation_strategy": type(cv).__name__,
+        "message": (
+            "Perfect CV score detected — possible data leakage."
+            if suspicious
+            else "CV performance within expected range."
+        )
+    }
+
+    # =========================
+    # Learning curve
+    # =========================
+    if plot:
+        train_sizes, train_scores, test_scores = learning_curve(
+            model,
+            X,
+            y,
+            cv=cv,  # <- MESMA estratégia de validação
+            train_sizes=np.linspace(0.1, 1.0, 5)
+        )
+
+        plt.figure(figsize=(6, 4))
+        plt.plot(
+            train_sizes,
+            np.mean(train_scores, axis=1),
+            label="Treino"
+        )
+        plt.plot(
+            train_sizes,
+            np.mean(test_scores, axis=1),
+            label="Validação"
+        )
+        plt.fill_between(
+            train_sizes,
+            np.mean(test_scores, axis=1) - np.std(test_scores, axis=1),
+            np.mean(test_scores, axis=1) + np.std(test_scores, axis=1),
+            alpha=0.2
+        )
+        plt.xlabel("Amostra de treino")
+        plt.ylabel("Score")
+        plt.title("Learning Curve")
+        plt.legend()
+        plt.tight_layout()
+        plt.show()
+
+    return result

{ai_critic-0.2.1 → ai_critic-0.2.3}/ai_critic/evaluators/robustness.py

@@ -12,8 +12,13 @@ def evaluate(model, X, y, leakage_suspected=False, plot=False):
     model_clean = clone(model)
     model_noisy = clone(model)
 
-    score_clean = cross_val_score(model_clean, X, y, cv=3, n_jobs=1).mean()
-    score_noisy = cross_val_score(model_noisy, X_noisy, y, cv=3, n_jobs=1).mean()
+    from .validation import make_cv
+
+    cv = make_cv(y)
+
+    score_clean = cross_val_score(model_clean, X, y, cv=cv, n_jobs=1).mean()
+    score_noisy = cross_val_score(model_noisy, X_noisy, y, cv=cv, n_jobs=1).mean()
+
     drop = score_clean - score_noisy
 
     # =========================

{ai_critic-0.2.1 → ai_critic-0.2.3}/ai_critic/evaluators/summary.py

@@ -78,6 +78,13 @@ class HumanSummary:
             recommendations.append(
                 "Fix baseline performance issues before trusting robustness metrics."
             )
+        elif robustness_verdict == "fragile":
+            key_risks.append(
+                "Model is fragile under noise perturbations."
+            )
+            recommendations.append(
+                "Consider regularization or simpler model architecture."
+            )
 
         technical_summary = {
             "key_risks": key_risks or ["No significant risks detected."],

ai_critic-0.2.3/ai_critic/evaluators/validation.py

@@ -0,0 +1,41 @@
+# validation.py
+import numpy as np
+from sklearn.model_selection import KFold, StratifiedKFold
+
+def infer_problem_type(y):
+    """
+    Infer whether the task is classification or regression.
+    """
+    y = np.asarray(y)
+
+    unique_values = np.unique(y)
+    n_unique = len(unique_values)
+
+    # Heurística conservadora
+    if (
+        np.issubdtype(y.dtype, np.integer)
+        or n_unique <= 20
+    ):
+        return "classification"
+
+    return "regression"
+
+
+def make_cv(y, n_splits=3, random_state=42):
+    """
+    Automatically selects the correct CV strategy.
+    """
+    problem_type = infer_problem_type(y)
+
+    if problem_type == "classification":
+        return StratifiedKFold(
+            n_splits=n_splits,
+            shuffle=True,
+            random_state=random_state
+        )
+
+    return KFold(
+        n_splits=n_splits,
+        shuffle=True,
+        random_state=random_state
+    )

ai_critic-0.2.3/ai_critic.egg-info/PKG-INFO

@@ -0,0 +1,76 @@
+Metadata-Version: 2.4
+Name: ai-critic
+Version: 0.2.3
+Summary: Fast AI evaluator for scikit-learn models
+Author-email: Luiz Seabra <filipedemarco@yahoo.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: scikit-learn
+
+Performance under noise
+
+> Visualizations are optional and do not affect the decision logic.
+
+---
+
+## ⚙️ Main API
+
+### `AICritic(model, X, y)`
+
+* `model`: scikit-learn compatible estimator
+* `X`: feature matrix
+* `y`: target vector
+
+### `evaluate(view="all", plot=False)`
+
+* `view`: `"executive"`, `"technical"`, `"details"`, `"all"` or custom list
+* `plot`: generates graphs when `True`
+
+---
+
+## 🧠 What ai-critic Detects
+
+| Category | Risks |
+
+| ------------ | ---------------------------------------- |
+
+| 🔍 Data | Target Leakage, NaNs, Imbalance |
+
+| 🧱 Structure | Excessive Complexity, Overfitting |
+
+| 📈 Validation | Perfect or Statistically Suspicious CV |
+
+| 🧪 Robustness | Stable, Fragile, or Misleading |
+
+---
+
+## 🛡️ Best Practices
+
+* **CI/CD:** Use executive output as a *quality gate*
+* **Iteration:** Use technical output during tuning
+* **Governance:** Log detailed output
+* **Skepticism:** Never blindly trust a perfect CV
+
+---
+
+## 🧭 Use Cases
+
+* Pre-deployment Audit
+* ML Governance
+* CI/CD Pipelines
+* Risk Communication for Non-Technical Users
+
+---
+
+## 📄 License
+
+Distributed under the **MIT License**.
+
+---
+
+## 🧠 Final Note
+
+**ai-critic** is not a *benchmarking* tool. It's a **decision-making tool**.
+
+If a model fails here, it doesn't mean it's bad—it means it **shouldn't be trusted yet**.

{ai_critic-0.2.1 → ai_critic-0.2.3}/ai_critic.egg-info/SOURCES.txt

@@ -13,5 +13,6 @@ ai_critic/evaluators/data.py
 ai_critic/evaluators/performance.py
 ai_critic/evaluators/robustness.py
 ai_critic/evaluators/summary.py
+ai_critic/evaluators/validation.py
 test/test_in_ia.py
 test/test_model.py

{ai_critic-0.2.1 → ai_critic-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "ai-critic"
-version = "0.2.1"
+version = "0.2.3"
 description = "Fast AI evaluator for scikit-learn models"
 readme = "README.md"
 authors = [

ai_critic-0.2.3/test/test_in_ia.py

@@ -0,0 +1,24 @@
+from ai_critic import AICritic
+from sklearn.ensemble import RandomForestClassifier
+import numpy as np
+
+# Dataset propositalmente desbalanceado
+X = np.random.rand(200, 10)
+y = np.array([0] * 180 + [1] * 20)
+
+model = RandomForestClassifier(
+    max_depth=12,
+    random_state=42
+)
+
+critic = AICritic(model, X, y)
+report = critic.evaluate(plot=False)
+
+print("\n=== EXECUTIVE SUMMARY ===")
+print(report["executive"])
+
+print("\n=== TECHNICAL SUMMARY ===")
+print(report["technical"])
+
+print("\n=== PERFORMANCE DETAILS ===")
+print(report["performance"])