ml-reliability-toolkit 0.1.0__tar.gz

ml_reliability_toolkit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Meharsai26
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ml_reliability_toolkit-0.1.0/PKG-INFO

@@ -0,0 +1,359 @@
+Metadata-Version: 2.4
+Name: ml-reliability-toolkit
+Version: 0.1.0
+Summary: Pre-deployment reliability checks for machine-learning systems.
+Author: Meharsai26
+Author-email: Meharsai26 <meharsai514@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Dist: numpy>=2.0
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: scikit-learn>=1.7.2 ; extra == 'demo'
+Requires-Python: >=3.10
+Provides-Extra: demo
+Description-Content-Type: text/markdown
+
+# ML Reliability Toolkit
+
+A Python toolkit for assessing machine-learning reliability beyond basic accuracy. It combines data-quality validation, model diagnostics, distribution analysis, and controlled stress testing to identify potential issues before deployment.
+
+## Features
+
+### 1. Data Quality Checks
+- Missing-value detection
+- Duplicate-row detection
+- Dataset schema validation
+- Feature-cardinality and identifier detection
+- Target-distribution analysis
+
+### 2. Model Evaluation
+- Classification performance using accuracy
+- Regression performance using R-squared
+- Generalization-gap analysis
+- Classification metrics (precision, recall, and F1-score)
+- Regression metrics (MAE and RMSE)
+- Classification calibration analysis
+- Slice-based performance analysis
+- Train/test distribution-shift analysis
+
+### 3. Stress Testing
+
+Evaluate model behavior under controlled input perturbations:
+
+| Stress test | Purpose |
+|---|---|
+| `missing_values` | Inject missing values into test features |
+| `feature_noise` | Add noise to numerical features |
+| `feature_dropout` | Mask selected feature values |
+| `categorical_corruption` | Corrupt selected categorical values |
+| `distribution_perturbation` | Shift and scale numerical features |
+
+Stress tests require a compatible model, test features, test targets, and a recognized task type.
+
+### 4. Reporting
+
+Reporting is available in four formats:
+
+- Console output
+- JSON
+- Markdown
+- HTML
+
+JSON, Markdown, and HTML reports can be exported to files. Reports include the overall assessment, individual check results, recommendations, and stress-test outcomes.
+
+## Requirements
+
+- Python 3.10 or newer
+- NumPy 2.0 or newer
+- pandas 2.3.3 or newer
+- scikit-learn 1.7.2 or newer (required for the included demos)
+
+## Installation
+
+### Install from source
+
+Clone the repository and navigate to the project directory.
+
+Install the package and development dependencies:
+
+```bash
+uv sync
+```
+
+Install the optional dependencies required for the demos:
+
+```bash
+uv sync --extra demo
+```
+
+### Install a locally built wheel
+
+```bash
+uv pip install ./dist/ml_reliability_toolkit-0.1.0-py3-none-any.whl
+```
+
+### PyPI publication
+
+The package has not yet been published to PyPI. Once published, it can be installed using:
+
+```bash
+pip install ml-reliability-toolkit
+```
+
+## Basic Usage
+
+The following example runs applicable data-quality and target-distribution checks.
+
+```python
+import pandas as pd
+from ml_reliability import ReliabilitySuite
+
+X_train = pd.DataFrame(
+    {
+        "age": [22, 25, 30, 35, 40],
+        "income": [25000, 32000, 45000, 52000, 65000],
+    }
+)
+
+y_train = pd.Series([0, 0, 1, 1, 1])
+
+report = ReliabilitySuite().run(
+    X_train=X_train,
+    y_train=y_train,
+)
+
+print("Reliability score:", report.score)
+print("Grade:", report.grade)
+
+for check in report.checks:
+    print(check.title, check.passed, check.message)
+```
+
+Only applicable checks are executed. Model-performance checks require a model and test data.
+
+## Model Evaluation
+
+The toolkit supports classification and regression workflows.
+
+Set the task type explicitly when automatic target inference is unsuitable.
+
+```python
+suite = ReliabilitySuite(task_type="classification")
+
+report = suite.run(
+    X_train=X_train,
+    y_train=y_train,
+    X_test=X_test,
+    y_test=y_test,
+    model=trained_model,
+)
+```
+
+Supported basic performance metrics:
+
+- Classification: Accuracy
+- Regression: R-squared
+
+Additional checks cover classification metrics, regression metrics, calibration, generalization gaps, and slice-based analysis.
+
+A successfully calculated metric does not automatically mean the model is reliable. Review the metric values and check results.
+
+## Running the Demos
+
+The project includes two executable examples demonstrating classification and regression workflows.
+
+Install the optional demo dependency first:
+
+```bash
+uv sync --extra demo
+```
+
+### Classification Demo
+
+Uses the Breast Cancer Wisconsin dataset and a Logistic Regression model.
+
+Run:
+
+```bash
+uv run python demo/demo_classification.py
+```
+
+The demo:
+- Loads and splits the dataset.
+- Trains a Logistic Regression pipeline.
+- Runs the reliability checks.
+- Executes stress tests.
+- Displays individual check results and evidence.
+- Generates JSON, Markdown, and HTML reports.
+
+### Regression Demo
+
+Uses the California Housing dataset and a Random Forest Regressor.
+
+Run:
+
+```bash
+uv run python demo/demo_regression.py
+```
+
+The demo:
+- Loads and splits the dataset.
+- Trains a Random Forest regression pipeline.
+- Runs the reliability checks.
+- Executes stress tests.
+- Displays individual check results and evidence.
+- Generates JSON, Markdown, and HTML reports.
+
+**Note:** Both demos create a `demo_outputs/` directory containing their generated reports. The first execution of the regression demo may require an internet connection to download the California Housing dataset.
+
+## Stress Testing
+
+Stress tests can be configured through the `stress_tests` argument.
+
+```python
+report = suite.run(
+    X_train=X_train,
+    y_train=y_train,
+    X_test=X_test,
+    y_test=y_test,
+    model=trained_model,
+    stress_tests={
+        "missing_values": {
+            "rate": 0.1,
+            "random_state": 42,
+        },
+        "feature_noise": {
+            "noise_level": 0.1,
+            "random_state": 42,
+        },
+        "feature_dropout": {
+            "features": ["age"],
+        },
+        "categorical_corruption": {
+            "columns": ["city"],
+            "rate": 0.1,
+            "random_state": 42,
+        },
+        "distribution_perturbation": {
+            "shift": 0.2,
+            "scale": 1.1,
+            "random_state": 42,
+        },
+    },
+)
+```
+
+All five configurations are optional. Use only those relevant to your dataset and model.
+
+Stress-test results are reported separately from the overall reliability score.
+
+A stress test's `passed` field indicates whether its evaluation completed successfully. It does not indicate that the model met a robustness threshold.
+
+## Reporting and Export
+
+### Console
+
+```python
+from ml_reliability.reporting import format_console_report
+
+print(format_console_report(report))
+```
+
+### JSON
+
+```python
+report.to_json("reliability_report.json")
+```
+
+JSON output contains report details, check results, and stress-test outcomes. Non-finite numeric values are converted to `null` to produce standards-compliant JSON.
+
+### Markdown
+
+```python
+report.to_markdown("reliability_report.md")
+```
+
+### HTML
+
+```python
+report.to_html("reliability_report.html")
+```
+
+HTML reports can be opened in a web browser.
+
+## Understanding the Reliability Score
+
+The overall score is calculated as the arithmetic mean of valid scored checks.
+
+A check score must be numeric, finite, non-boolean, and between 0 and 100 to contribute to the overall score.
+
+| Grade | Score |
+|---|---:|
+| A | 90-100 |
+| B | 75-89.99 |
+| C | 60-74.99 |
+| D | 40-59.99 |
+| F | Below 40 |
+| N/A | No valid scored checks |
+
+Important considerations:
+
+- Checks without a score do not contribute to the average.
+- Invalid check scores are excluded and identified in the report.
+- Critical failures are highlighted separately; they do not receive an additional severity-based scoring penalty.
+- Stress-test outcomes are excluded from the overall score.
+- Regression MAE and RMSE are calculated, but do not contribute to the overall score when no scoring thresholds are configured.
+- A high score does not guarantee that a model is accurate, fair, robust, or production-ready.
+
+The score represents the applicable checks performed with the supplied inputs, not a universal measure of model quality.
+
+## Known Limitations
+
+- Cardinality detection is heuristic. High-cardinality numerical features are not necessarily identifiers.
+- Distribution perturbation applies synthetic transformations and may not represent realistic production data changes.
+- Stress-test outcomes depend on the selected dataset, model, perturbation settings, and random seed.
+- Regression metrics require suitable thresholds to contribute to the overall reliability score.
+- Reliability results should be interpreted alongside domain-specific validation and appropriate evaluation datasets.
+
+## Development and Testing
+
+Run the complete test suite:
+
+```bash
+uv run pytest -q
+```
+
+Run Python compilation checks:
+
+```bash
+uv run python -m compileall -q src demo
+```
+
+## Project Structure
+
+```text
+ml-reliability-toolkit/
+|-- demo/
+|   |-- demo_classification.py
+|   `-- demo_regression.py
+|-- src/
+|   `-- ml_reliability/
+|       |-- checks/
+|       |   `-- stress/
+|       |-- core/
+|       |-- reporting.py
+|       `-- suite.py
+|-- tests/
+|-- pyproject.toml
+|-- README.md
+`-- uv.lock
+```
+
+## Current Status
+
+- Package: `ml-reliability-toolkit`
+- Version: `0.1.0`
+- Python: `>=3.10`
+- Latest verified full test run: 479 tests passed.
+
+The toolkit is intended to support pre-deployment assessment. Its results should be interpreted alongside domain-specific validation, appropriate evaluation datasets, and human review. It is not a guarantee of production reliability.

ml_reliability_toolkit-0.1.0/README.md

@@ -0,0 +1,344 @@
+# ML Reliability Toolkit
+
+A Python toolkit for assessing machine-learning reliability beyond basic accuracy. It combines data-quality validation, model diagnostics, distribution analysis, and controlled stress testing to identify potential issues before deployment.
+
+## Features
+
+### 1. Data Quality Checks
+- Missing-value detection
+- Duplicate-row detection
+- Dataset schema validation
+- Feature-cardinality and identifier detection
+- Target-distribution analysis
+
+### 2. Model Evaluation
+- Classification performance using accuracy
+- Regression performance using R-squared
+- Generalization-gap analysis
+- Classification metrics (precision, recall, and F1-score)
+- Regression metrics (MAE and RMSE)
+- Classification calibration analysis
+- Slice-based performance analysis
+- Train/test distribution-shift analysis
+
+### 3. Stress Testing
+
+Evaluate model behavior under controlled input perturbations:
+
+| Stress test | Purpose |
+|---|---|
+| `missing_values` | Inject missing values into test features |
+| `feature_noise` | Add noise to numerical features |
+| `feature_dropout` | Mask selected feature values |
+| `categorical_corruption` | Corrupt selected categorical values |
+| `distribution_perturbation` | Shift and scale numerical features |
+
+Stress tests require a compatible model, test features, test targets, and a recognized task type.
+
+### 4. Reporting
+
+Reporting is available in four formats:
+
+- Console output
+- JSON
+- Markdown
+- HTML
+
+JSON, Markdown, and HTML reports can be exported to files. Reports include the overall assessment, individual check results, recommendations, and stress-test outcomes.
+
+## Requirements
+
+- Python 3.10 or newer
+- NumPy 2.0 or newer
+- pandas 2.3.3 or newer
+- scikit-learn 1.7.2 or newer (required for the included demos)
+
+## Installation
+
+### Install from source
+
+Clone the repository and navigate to the project directory.
+
+Install the package and development dependencies:
+
+```bash
+uv sync
+```
+
+Install the optional dependencies required for the demos:
+
+```bash
+uv sync --extra demo
+```
+
+### Install a locally built wheel
+
+```bash
+uv pip install ./dist/ml_reliability_toolkit-0.1.0-py3-none-any.whl
+```
+
+### PyPI publication
+
+The package has not yet been published to PyPI. Once published, it can be installed using:
+
+```bash
+pip install ml-reliability-toolkit
+```
+
+## Basic Usage
+
+The following example runs applicable data-quality and target-distribution checks.
+
+```python
+import pandas as pd
+from ml_reliability import ReliabilitySuite
+
+X_train = pd.DataFrame(
+    {
+        "age": [22, 25, 30, 35, 40],
+        "income": [25000, 32000, 45000, 52000, 65000],
+    }
+)
+
+y_train = pd.Series([0, 0, 1, 1, 1])
+
+report = ReliabilitySuite().run(
+    X_train=X_train,
+    y_train=y_train,
+)
+
+print("Reliability score:", report.score)
+print("Grade:", report.grade)
+
+for check in report.checks:
+    print(check.title, check.passed, check.message)
+```
+
+Only applicable checks are executed. Model-performance checks require a model and test data.
+
+## Model Evaluation
+
+The toolkit supports classification and regression workflows.
+
+Set the task type explicitly when automatic target inference is unsuitable.
+
+```python
+suite = ReliabilitySuite(task_type="classification")
+
+report = suite.run(
+    X_train=X_train,
+    y_train=y_train,
+    X_test=X_test,
+    y_test=y_test,
+    model=trained_model,
+)
+```
+
+Supported basic performance metrics:
+
+- Classification: Accuracy
+- Regression: R-squared
+
+Additional checks cover classification metrics, regression metrics, calibration, generalization gaps, and slice-based analysis.
+
+A successfully calculated metric does not automatically mean the model is reliable. Review the metric values and check results.
+
+## Running the Demos
+
+The project includes two executable examples demonstrating classification and regression workflows.
+
+Install the optional demo dependency first:
+
+```bash
+uv sync --extra demo
+```
+
+### Classification Demo
+
+Uses the Breast Cancer Wisconsin dataset and a Logistic Regression model.
+
+Run:
+
+```bash
+uv run python demo/demo_classification.py
+```
+
+The demo:
+- Loads and splits the dataset.
+- Trains a Logistic Regression pipeline.
+- Runs the reliability checks.
+- Executes stress tests.
+- Displays individual check results and evidence.
+- Generates JSON, Markdown, and HTML reports.
+
+### Regression Demo
+
+Uses the California Housing dataset and a Random Forest Regressor.
+
+Run:
+
+```bash
+uv run python demo/demo_regression.py
+```
+
+The demo:
+- Loads and splits the dataset.
+- Trains a Random Forest regression pipeline.
+- Runs the reliability checks.
+- Executes stress tests.
+- Displays individual check results and evidence.
+- Generates JSON, Markdown, and HTML reports.
+
+**Note:** Both demos create a `demo_outputs/` directory containing their generated reports. The first execution of the regression demo may require an internet connection to download the California Housing dataset.
+
+## Stress Testing
+
+Stress tests can be configured through the `stress_tests` argument.
+
+```python
+report = suite.run(
+    X_train=X_train,
+    y_train=y_train,
+    X_test=X_test,
+    y_test=y_test,
+    model=trained_model,
+    stress_tests={
+        "missing_values": {
+            "rate": 0.1,
+            "random_state": 42,
+        },
+        "feature_noise": {
+            "noise_level": 0.1,
+            "random_state": 42,
+        },
+        "feature_dropout": {
+            "features": ["age"],
+        },
+        "categorical_corruption": {
+            "columns": ["city"],
+            "rate": 0.1,
+            "random_state": 42,
+        },
+        "distribution_perturbation": {
+            "shift": 0.2,
+            "scale": 1.1,
+            "random_state": 42,
+        },
+    },
+)
+```
+
+All five configurations are optional. Use only those relevant to your dataset and model.
+
+Stress-test results are reported separately from the overall reliability score.
+
+A stress test's `passed` field indicates whether its evaluation completed successfully. It does not indicate that the model met a robustness threshold.
+
+## Reporting and Export
+
+### Console
+
+```python
+from ml_reliability.reporting import format_console_report
+
+print(format_console_report(report))
+```
+
+### JSON
+
+```python
+report.to_json("reliability_report.json")
+```
+
+JSON output contains report details, check results, and stress-test outcomes. Non-finite numeric values are converted to `null` to produce standards-compliant JSON.
+
+### Markdown
+
+```python
+report.to_markdown("reliability_report.md")
+```
+
+### HTML
+
+```python
+report.to_html("reliability_report.html")
+```
+
+HTML reports can be opened in a web browser.
+
+## Understanding the Reliability Score
+
+The overall score is calculated as the arithmetic mean of valid scored checks.
+
+A check score must be numeric, finite, non-boolean, and between 0 and 100 to contribute to the overall score.
+
+| Grade | Score |
+|---|---:|
+| A | 90-100 |
+| B | 75-89.99 |
+| C | 60-74.99 |
+| D | 40-59.99 |
+| F | Below 40 |
+| N/A | No valid scored checks |
+
+Important considerations:
+
+- Checks without a score do not contribute to the average.
+- Invalid check scores are excluded and identified in the report.
+- Critical failures are highlighted separately; they do not receive an additional severity-based scoring penalty.
+- Stress-test outcomes are excluded from the overall score.
+- Regression MAE and RMSE are calculated, but do not contribute to the overall score when no scoring thresholds are configured.
+- A high score does not guarantee that a model is accurate, fair, robust, or production-ready.
+
+The score represents the applicable checks performed with the supplied inputs, not a universal measure of model quality.
+
+## Known Limitations
+
+- Cardinality detection is heuristic. High-cardinality numerical features are not necessarily identifiers.
+- Distribution perturbation applies synthetic transformations and may not represent realistic production data changes.
+- Stress-test outcomes depend on the selected dataset, model, perturbation settings, and random seed.
+- Regression metrics require suitable thresholds to contribute to the overall reliability score.
+- Reliability results should be interpreted alongside domain-specific validation and appropriate evaluation datasets.
+
+## Development and Testing
+
+Run the complete test suite:
+
+```bash
+uv run pytest -q
+```
+
+Run Python compilation checks:
+
+```bash
+uv run python -m compileall -q src demo
+```
+
+## Project Structure
+
+```text
+ml-reliability-toolkit/
+|-- demo/
+|   |-- demo_classification.py
+|   `-- demo_regression.py
+|-- src/
+|   `-- ml_reliability/
+|       |-- checks/
+|       |   `-- stress/
+|       |-- core/
+|       |-- reporting.py
+|       `-- suite.py
+|-- tests/
+|-- pyproject.toml
+|-- README.md
+`-- uv.lock
+```
+
+## Current Status
+
+- Package: `ml-reliability-toolkit`
+- Version: `0.1.0`
+- Python: `>=3.10`
+- Latest verified full test run: 479 tests passed.
+
+The toolkit is intended to support pre-deployment assessment. Its results should be interpreted alongside domain-specific validation, appropriate evaluation datasets, and human review. It is not a guarantee of production reliability.