ai-critic 1.1.0__py3-none-any.whl → 2.0.0__py3-none-any.whl

ai_critic-2.0.0.dist-info/METADATA

@@ -0,0 +1,390 @@
+Metadata-Version: 2.4
+Name: ai-critic
+Version: 2.0.0
+Summary: Fast AI evaluator for scikit-learn models
+Author-email: Luiz Seabra <filipedemarco@yahoo.com>
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: scikit-learn
+
+# ai-critic 🧠
+
+## The Quality Gate for Machine Learning Models
+
+**ai-critic** is a specialized **decision-making system** designed to evaluate whether a machine learning model is **safe, reliable, and trustworthy enough** to be deployed in real-world environments.
+
+Unlike traditional ML evaluation tools that focus almost exclusively on *performance metrics*, **ai-critic** operates as a **Quality Gate** — a final checkpoint that actively probes models to uncover **hidden risks** that frequently cause silent failures in production.
+
+> **ai-critic does not ask *“How accurate is this model?”***
+> It asks ***“Can this model be trusted in the real world?”***
+
+---
+
+## 🎯 What Problem Does ai-critic Solve?
+
+In production, most ML failures are **not accuracy problems**.
+
+They are caused by:
+
+* Data leakage hidden inside features
+* Overfitting disguised as strong validation scores
+* Models that collapse under small noise
+* Models that rely on a single fragile signal
+* Configuration choices that look fine — but are structurally unsafe
+
+These failures usually appear **after deployment**, when it is already expensive or dangerous to fix them.
+
+**ai-critic exists to catch these failures *before* deployment.**
+
+---
+
+## 🚀 Getting Started (The Basics)
+
+This section is intentionally designed for **beginners**, **students**, and **engineers under time pressure**.
+
+If you only want a **fast, conservative verdict**, this is all you need.
+
+---
+
+### Installation
+
+Install directly from PyPI:
+
+```bash
+pip install ai-critic
+```
+
+Python ≥ 3.8 is recommended.
+
+---
+
+### The Quick Verdict
+
+With just a few lines of code, you can obtain:
+
+* An **executive-level verdict**
+* A **risk classification**
+* A **deployment recommendation**
+
+```python
+from ai_critic import AICritic
+from sklearn.ensemble import RandomForestClassifier
+from sklearn.datasets import make_classification
+
+# 1. Prepare data and model
+X, y = make_classification(
+    n_samples=1000,
+    n_features=20,
+    random_state=42
+)
+
+model = RandomForestClassifier(
+    max_depth=5,
+    random_state=42
+)
+
+# 2. Initialize the Critic
+critic = AICritic(model, X, y)
+
+# 3. Run the audit
+report = critic.evaluate(view="executive")
+
+print(f"Verdict: {report['verdict']}")
+print(f"Risk Level: {report['risk_level']}")
+print(f"Deploy Recommended: {report['deploy_recommended']}")
+print(f"Main Reason: {report['main_reason']}")
+```
+
+**Example Output:**
+
+```text
+Verdict: ⚠️ Risky
+Risk Level: medium
+Deploy Recommended: False
+Main Reason: Structural, robustness, or dependency-related risks detected.
+```
+
+This verdict is intentionally **conservative by design**.
+
+> If **ai-critic approves deployment**, it means **no meaningful risks were detected** by multiple independent heuristics.
+
+---
+
+## 🧭 How to Read the Verdict
+
+| Field                | Meaning                 |
+| -------------------- | ----------------------- |
+| `verdict`            | Human-readable summary  |
+| `risk_level`         | low / medium / high     |
+| `deploy_recommended` | Final gate decision     |
+| `main_reason`        | Primary blocking factor |
+
+The goal is clarity, not ambiguity.
+
+---
+
+## 💡 Understanding the Critique (Intermediate Level)
+
+This section is for **data scientists**, **ML engineers**, and **students** who want to understand *why* the model was flagged — and how to improve it.
+
+---
+
+### The Four Pillars of the Audit
+
+**ai-critic** evaluates models across **four independent risk dimensions**.
+
+| Pillar             | What It Detects                  | Why It Matters       |
+| ------------------ | -------------------------------- | -------------------- |
+| 📊 Data Integrity  | Leakage, correlations, shortcuts | Inflated performance |
+| 🧠 Model Structure | Over-complexity, unsafe configs  | Poor generalization  |
+| 📈 Performance     | Suspicious CV behavior           | False confidence     |
+| 🧪 Robustness      | Noise sensitivity                | Production collapse  |
+
+Each pillar produces **signals**, not binary judgments.
+
+Those signals are later aggregated by the **deployment gate**.
+
+---
+
+## 📊 Data Integrity Analysis
+
+This pillar focuses on **the relationship between features and the target**.
+
+It answers questions like:
+
+* Are some features *too predictive*?
+* Are there suspicious correlations?
+* Does performance collapse when a single feature is disturbed?
+
+These are classic symptoms of **data leakage** and **shortcut learning**.
+
+---
+
+## 🧠 Model Structure Analysis
+
+A model can be accurate and still be unsafe.
+
+Structural analysis looks for:
+
+* Excessive depth
+* Over-parameterization
+* Configuration choices that amplify variance
+* Inconsistent bias–variance tradeoffs
+
+This is especially important for:
+
+* Decision trees
+* Boosting models
+* Neural networks with limited data
+
+---
+
+## 📈 Performance Sanity Checks
+
+Rather than optimizing metrics, **ai-critic questions them**.
+
+It checks:
+
+* Cross-validation stability
+* Variance across folds
+* Learning curve consistency
+* Performance under perturbations
+
+A strong score that behaves strangely is treated as **a warning, not a success**.
+
+---
+
+## 🧪 Robustness Testing (Noise Injection)
+
+Production data is **never clean**.
+
+This test injects controlled noise into inputs and measures degradation.
+
+```python
+robustness = report["details"]["robustness"]
+
+print(f"Original CV Score: {robustness['cv_score_original']}")
+print(f"Noisy CV Score: {robustness['cv_score_noisy']}")
+print(f"Performance Drop: {robustness['performance_drop']}")
+print(f"Verdict: {robustness['verdict']}")
+```
+
+Possible outcomes:
+
+* `stable` → acceptable degradation
+* `fragile` → high sensitivity
+* `misleading` → performance likely inflated
+
+---
+
+## 🔍 Explainability & Feature Sensitivity
+
+Accuracy alone hides *why* a model works.
+
+The explainability module performs **feature sensitivity analysis** to detect:
+
+* Feature-level leakage
+* Over-reliance on a single signal
+* Structural shortcuts
+
+---
+
+### How Explainability Works
+
+For each feature:
+
+1. The feature is randomly permuted.
+2. The model is re-evaluated.
+3. Performance drop is measured.
+
+Large drops indicate **critical dependency**.
+
+This approach is:
+
+* Model-agnostic
+* Lightweight
+* Framework-independent
+* Interpretable by humans
+
+---
+
+### Explainability Verdicts
+
+| Verdict                | Meaning                  |
+| ---------------------- | ------------------------ |
+| `stable`               | Balanced feature usage   |
+| `feature_dependency`   | Few features dominate    |
+| `feature_leakage_risk` | Single feature dominates |
+
+These verdicts **directly affect**:
+
+* Deployment decision
+* Confidence score
+* Recommendations
+
+---
+
+## 🧠 Recommendations Engine (New)
+
+**ai-critic does not stop at “deploy or not”.**
+
+It generates **actionable recommendations**, such as:
+
+* “Reduce `max_depth`”
+* “Increase regularization”
+* “Likely feature leakage detected”
+* “Model shows structural overfitting”
+* “High noise sensitivity — retrain with augmentation”
+
+These recommendations are **rule-based + data-driven**, not LLM hallucinations.
+
+---
+
+## ⚙️ Deployment Gate
+
+The final decision is produced by `deploy_decision()`.
+
+```python
+decision = critic.deploy_decision()
+
+print(decision["deploy"])
+print(decision["risk_level"])
+print(decision["confidence"])
+print(decision["blocking_issues"])
+```
+
+Conceptually:
+
+* **Hard blockers** → deployment denied
+* **Soft blockers** → deployment discouraged
+* **Confidence score (0–1)** → heuristic trust
+
+---
+
+## 🔄 Feedback Loop & Learning Critic
+
+**ai-critic improves over time**.
+
+Each evaluation can be stored as feedback:
+
+* Model config
+* Signals
+* Final outcome
+* Human override (optional)
+
+This enables:
+
+* Meta-learning
+* Better future recommendations
+* Context-aware criticism
+
+---
+
+## 🧪 Session Tracking & Comparison
+
+You can compare models over time:
+
+```python
+critic_v1 = AICritic(model, X, y, session="v1")
+critic_v1.evaluate()
+
+critic_v2 = AICritic(model, X, y, session="v2")
+critic_v2.evaluate()
+
+critic_v2.compare_with("v1")
+```
+
+Use cases:
+
+* Regression detection
+* Risk drift
+* Governance audits
+
+---
+
+## ⚙️ Multi-Framework Support
+
+The same API works for:
+
+* scikit-learn
+* PyTorch
+* TensorFlow
+
+Adapters handle training, evaluation, and probing internally.
+
+---
+
+## 🧩 Design Philosophy
+
+**ai-critic is intentionally skeptical.**
+
+It assumes:
+
+* Metrics can lie
+* Data is imperfect
+* Models fail silently
+* Confidence must be earned
+
+This makes it ideal as a **final gate**, not a tuning toy.
+
+---
+
+## 🛡️ What ai-critic Is NOT
+
+* ❌ A hyperparameter optimizer
+* ❌ A leaderboard benchmark tool
+* ❌ A replacement for domain expertise
+* ❌ A magic “approve all” system
+
+---
+
+## 🧠 Final Note
+
+> **ai-critic is not here to make models look good.**
+> It exists to **prevent bad models from looking good enough to deploy**.
+
+A failed audit does **not** mean your model is bad.
+It means your model is **not yet safe to trust**.
+
+That distinction is everything.

ai_critic-2.0.0.dist-info/RECORD

@@ -0,0 +1,37 @@
+ai_critic/__init__.py,sha256=H6DlPMmbcFUamhsNULPLk9vHx81XCiXuKKf63EJ8eM0,53
+ai_critic/cli.py,sha256=4rf9g-CjtYaS9jjLPYWI56Z_6JLHhKh4KSbELYWwsX8,4133
+ai_critic/critic.py,sha256=4rf9g-CjtYaS9jjLPYWI56Z_6JLHhKh4KSbELYWwsX8,4133
+ai_critic/ai_suggestions/predictor.py,sha256=pn20sG1MjXuzXSquu0IqAcSzM2Y3gROHTq8zRSle0RM,155
+ai_critic/ai_suggestions/rules.py,sha256=ZbCtPc5OZgPGzqgYNXf43C2aVdGWKsjhQ-_zdsmtftw,132
+ai_critic/evaluators/__init__.py,sha256=ri6InmL8_LIcO-JZpU_gEFKLO4URdqo3z6rh7fV6M8Y,169
+ai_critic/evaluators/adapters.py,sha256=8Xw9Ccg1iGVNwVQDGVIqhWj5-Sg6evqCZhg21u8EP20,3068
+ai_critic/evaluators/config.py,sha256=eeHOqyU-GOMFdQlhoyJnsaEZYMF1XewyoFKYP7d0o-w,1911
+ai_critic/evaluators/data.py,sha256=YAK5NkwCeJOny_UueZ5ALwvEcRDIbEck404eV2oqWnc,1871
+ai_critic/evaluators/explainability.py,sha256=UWbcb5uVI78d1ljfdrWd2DrjlwEz1y9CeVtkukefEfA,1759
+ai_critic/evaluators/performance.py,sha256=1CQx5DueK0XkelYyJnAGRJ3AjQtjsKeW8_1JQZqKVOI,1973
+ai_critic/evaluators/robustness.py,sha256=mfVQ67Z6t6aRvtIq-XQEQYbwvyf8UefM1myeOGVrnAE,1869
+ai_critic/evaluators/scoring.py,sha256=9rgkCXKKm9G1Lfwn5i9HcsJTN5OUjxMycOUzhWkp_2g,1576
+ai_critic/evaluators/summary.py,sha256=H9rU9tXAXqyQ34L6bOOOHrdIapSq71gcjjc8jfyJMq4,5003
+ai_critic/evaluators/validation.py,sha256=rnzRwD78Cugey33gl9geE8JoBURsKEEnqrIOhBZv0LY,904
+ai_critic/feedback/__init__.py,sha256=JWzTxV8ycpoQPrHuwWzwmuDlpVAL-vzUx2dXy1_7z9c,62
+ai_critic/feedback/store.py,sha256=BpWAM9byj-zfFG6cjxNOvVDoEPP5TUe5DSiAgVQ_8Rg,606
+ai_critic/learning/__init__.py,sha256=umxvyyh8bKdyZP5tRidJUJ-mqeRlTkNcV0HE7_0qHTU,318
+ai_critic/learning/critic_model.py,sha256=KqTOYjk4DsdHjcps-IZO-Udx0KPq6fHVWD9w8OViWyo,708
+ai_critic/learning/features.py,sha256=T5thN96ZBv63gSKxHARc87al2UJP5AIF4_4SAPt1fuE,663
+ai_critic/learning/policy.py,sha256=c2bAtvUu6DK1KDtxTScK-C2xMcQC-HvXmPxpLwggygg,511
+ai_critic/learning/recommender.py,sha256=DxL_C-oqahJJ_u69486JyxBNszYibbru1YBV-TyWgRQ,715
+ai_critic/learning/trainer.py,sha256=UsSS_1QMYQNf4UCGBvbVtWr3n60qC6ZUMP9AmBilKz8,497
+ai_critic/ml/suggester.py,sha256=DKq5NjT7oLBXVw2jNtDLeSjzamV7un2QPqxMHYRIh1Q,1757
+ai_critic/sessions/__init__.py,sha256=Yp7mphSPJwt8a4cJgcQNErqwqHVuP_xAJODrs0y0Abw,72
+ai_critic/sessions/store.py,sha256=65m9WXFVFWv4pPzvXV4l8zLHoHWMfCGe6eHh4X-8agY,947
+ai_critic/telemetry/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+ai_critic/telemetry/anonymizer.py,sha256=Mg40ZZ1U8vbUEar_n3ZtrDwTfLvJB-qnRlmqXEDAsnk,332
+ai_critic/telemetry/client.py,sha256=CuEibfDkI5Y_Y1yV8BT0aXEjFgZurcL5p5Pk7yfpTz4,130
+ai_critic/telemetry/event.py,sha256=9n2tGmrrAyFhx26Yb3wUImO72ANJc06ZM8yPs4Q48Uc,697
+ai_critic/telemetry/local_store.py,sha256=zZ3CgvHYMUhwgpa_VuITI4rlPHCV4yJKxxYjIGoijfA,225
+ai_critic/telemetry/schema.py,sha256=1fbZqmEmgvmM4FNmhI7O2sQfUaGEZty5SzqLYVo3y0g,200
+ai_critic/telemetry/sender.py,sha256=rsxKbmbT2UvoT_oFK8mQxdcqZRfHLRgrkozHJqbsDps,228
+ai_critic-2.0.0.dist-info/METADATA,sha256=MUuBILCf0XNrD0D8kjdOa7HX-ENYPqW4AGfNsBlpajQ,9293
+ai_critic-2.0.0.dist-info/WHEEL,sha256=wUyA8OaulRlbfwMtmQsvNngGrxQHAvkKcvRmdizlJi0,92
+ai_critic-2.0.0.dist-info/top_level.txt,sha256=TRyZkm1vyLLcFDg_80yeg5cHvPis_oW1Ti170417jkw,10
+ai_critic-2.0.0.dist-info/RECORD,,

ai_critic-1.1.0.dist-info/METADATA

@@ -1,289 +0,0 @@
-Metadata-Version: 2.4
-Name: ai-critic
-Version: 1.1.0
-Summary: Fast AI evaluator for scikit-learn models
-Author-email: Luiz Seabra <filipedemarco@yahoo.com>
-Requires-Python: >=3.9
-Description-Content-Type: text/markdown
-Requires-Dist: numpy
-Requires-Dist: scikit-learn
-
-# ai-critic 🧠: The Quality Gate for Machine Learning Models
-
-**ai-critic** is a specialized **decision-making** tool designed to audit the reliability and readiness for deployment of **scikit-learn**, **PyTorch**, and **TensorFlow** models.
-
-Instead of merely measuring performance (accuracy, F1 score), **ai-critic** acts as a **Quality Gate**, actively probing the model to uncover *hidden risks* that commonly cause production failures — such as **data leakage**, **structural overfitting**, and **fragility under noise**.
-
-> **ai-critic does not ask “How good is this model?”**
-> It asks **“Can this model be trusted?”**
-
----
-
-## 🚀 Getting Started (The Basics)
-
-This section is ideal for beginners who need a **fast and reliable verdict** on a trained model.
-
-### Installation
-
-Install directly from PyPI:
-
-```bash
-pip install ai-critic
-```
-
----
-
-### The Quick Verdict
-
-With just a few lines of code, you obtain an **executive-level assessment** and a **deployment recommendation**.
-
-```python
-from ai_critic import AICritic
-from sklearn.ensemble import RandomForestClassifier
-from sklearn.datasets import make_classification
-
-# 1. Prepare data and model
-X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
-model = RandomForestClassifier(max_depth=5, random_state=42)
-
-# 2. Initialize the Critic
-critic = AICritic(model, X, y)
-
-# 3. Run the audit (executive mode)
-report = critic.evaluate(view="executive")
-
-print(f"Verdict: {report['verdict']}")
-print(f"Risk Level: {report['risk_level']}")
-print(f"Main Reason: {report['main_reason']}")
-```
-
-**Expected Output (example):**
-
-```text
-Verdict: ⚠️ Risky
-Risk Level: medium
-Main Reason: Structural or robustness-related risks detected.
-```
-
-This output is intentionally **conservative**.
-If **ai-critic** recommends deployment, it means meaningful risks were *not* detected.
-
----
-
-## 💡 Understanding the Critique (The Intermediary)
-
-For data scientists who want to understand **why** the model received a given verdict and **how to improve it**.
-
----
-
-### The Four Pillars of the Audit
-
-**ai-critic** evaluates models across four independent risk dimensions:
-
-| Pillar                 | Main Risk Detected                     | Internal Module          |
-| ---------------------- | -------------------------------------- | ------------------------ |
-| 📊 **Data Integrity**  | Target Leakage & Correlation Artifacts | `evaluators.data`        |
-| 🧠 **Model Structure** | Over-complexity & Misconfiguration     | `evaluators.config`      |
-| 📈 **Performance**     | Suspicious CV or Learning Curves       | `evaluators.performance` |
-| 🧪 **Robustness**      | Sensitivity to Noise                   | `evaluators.robustness`  |
-
-Each pillar contributes signals used later in the **deployment gate**.
-
----
-
-### Full Technical & Visual Analysis
-
-To access **all internal diagnostics**, including plots and recommendations, use `view="all"`.
-
-```python
-full_report = critic.evaluate(view="all", plot=True)
-
-technical_summary = full_report["technical"]
-
-print("\n--- Key Risks Detected ---")
-for i, risk in enumerate(technical_summary["key_risks"], start=1):
-    print(f"{i}. {risk}")
-
-print("\n--- Recommendations ---")
-for rec in technical_summary["recommendations"]:
-    print(f"- {rec}")
-```
-
-Generated plots may include:
-
-* Feature correlation heatmaps
-* Learning curves
-* Robustness degradation charts
-
----
-
-### Robustness Test (Noise Injection)
-
-A model that collapses under small perturbations is **not production-safe**.
-
-```python
-robustness = full_report["details"]["robustness"]
-
-print("\n--- Robustness Analysis ---")
-print(f"Original CV Score: {robustness['cv_score_original']:.4f}")
-print(f"Noisy CV Score: {robustness['cv_score_noisy']:.4f}")
-print(f"Performance Drop: {robustness['performance_drop']:.4f}")
-print(f"Verdict: {robustness['verdict']}")
-```
-
-**Possible Verdicts:**
-
-* `stable` → acceptable degradation
-* `fragile` → high sensitivity to noise
-* `misleading` → performance likely inflated by leakage
-
----
-
-## ⚙️ Integration and Governance (The Advanced)
-
-This section targets **MLOps engineers**, **architects**, and teams operating automated pipelines.
-
----
-
-### Multi-Framework Support
-
-**ai-critic 1.0+** supports models from multiple frameworks with the **same API**:
-
-```python
-# PyTorch Example
-import torch
-import torch.nn as nn
-from ai_critic import AICritic
-
-X = torch.randn(1000, 20)
-y = torch.randint(0, 2, (1000,))
-
-model = nn.Sequential(
-    nn.Linear(20, 32),
-    nn.ReLU(),
-    nn.Linear(32, 2)
-)
-
-critic = AICritic(model, X, y, framework="torch", adapter_kwargs={"epochs":5, "batch_size":64})
-report = critic.evaluate(view="executive")
-print(report)
-
-# TensorFlow Example
-import tensorflow as tf
-
-model = tf.keras.Sequential([
-    tf.keras.layers.Dense(32, activation="relu", input_shape=(20,)),
-    tf.keras.layers.Dense(2)
-])
-critic = AICritic(model, X.numpy(), y.numpy(), framework="tensorflow", adapter_kwargs={"epochs":5})
-report = critic.evaluate(view="executive")
-print(report)
-```
-
-> No need to rewrite evaluation code — **one Critic API works for sklearn, PyTorch, or TensorFlow**.
-
----
-
-### The Deployment Gate (`deploy_decision`)
-
-The `deploy_decision()` method aggregates *all detected risks* and produces a final gate decision.
-
-```python
-decision = critic.deploy_decision()
-
-if decision["deploy"]:
-    print("✅ Deployment Approved")
-else:
-    print("❌ Deployment Blocked")
-
-print(f"Risk Level: {decision['risk_level']}")
-print(f"Confidence Score: {decision['confidence']:.2f}")
-
-print("\nBlocking Issues:")
-for issue in decision["blocking_issues"]:
-    print(f"- {issue}")
-```
-
-**Conceptual model:**
-
-* **Hard Blockers** → deployment denied
-* **Soft Blockers** → deployment discouraged
-* **Confidence Score (0–1)** → heuristic trust level
-
----
-
-### Modes & Views (API Design)
-
-The `evaluate()` method supports **multiple modes** via the `view` parameter:
-
-| View          | Description                        |
-| ------------- | ---------------------------------- |
-| `"executive"` | High-level verdict (non-technical) |
-| `"technical"` | Risks & recommendations            |
-| `"details"`   | Raw evaluator outputs              |
-| `"all"`       | Complete payload                   |
-
-Example:
-
-```python
-critic.evaluate(view="technical")
-critic.evaluate(view=["executive", "performance"])
-```
-
----
-
-### Session Tracking & Model Comparison
-
-You can persist evaluations and compare model versions over time.
-
-```python
-critic_v1 = AICritic(model, X, y, session="v1")
-critic_v1.evaluate()
-
-critic_v2 = AICritic(model, X, y, session="v2")
-critic_v2.evaluate()
-
-comparison = critic_v2.compare_with("v1")
-print(comparison["score_diff"])
-```
-
-This enables:
-
-* Regression tracking
-* Risk drift detection
-* Governance & audit trails
-
----
-
-### Best Practices & Use Cases
-
-| Scenario                | Recommended Usage                      |
-| ----------------------- | -------------------------------------- |
-| **CI/CD**               | Block merges using `deploy_decision()` |
-| **Model Tuning**        | Use technical view for guidance        |
-| **Governance**          | Persist session outputs                |
-| **Stakeholder Reports** | Share executive summaries              |
-
----
-
-## 🔒 API Stability
-
-Starting from version **1.0.0**, the public API of **ai-critic** follows semantic versioning.
-Breaking changes will only occur in major releases.
-
----
-
-## 📄 License
-
-Distributed under the **MIT License**.
-
----
-
-## 🧠 Final Note
-
-> **ai-critic is not a benchmarking tool.**
-> It is a *decision-making system*.
-
-A failed audit does **not** mean the model is bad — it means the model **is not ready to be trusted**.
-
-The purpose of **ai-critic** is to introduce *structured skepticism* into machine learning workflows — exactly where it belongs.