automr 0.1.0__tar.gz

automr-0.1.0/PKG-INFO

@@ -0,0 +1,263 @@
+Metadata-Version: 2.4
+Name: automr
+Version: 0.1.0
+Summary: Metamorphic Testing Framework for Regression-based Autonomous Driving ML/AI Models
+Author: Charith Manujaya, Raveesha Peiris, Thurunu Pabasara, Tharika Akurana
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/CharithManaujayaMUTEC/AutoMR-Framework
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: tqdm
+Requires-Dist: opencv-python
+Requires-Dist: tensorflow
+Requires-Dist: python-dotenv
+
+<div align="center">
+  <img src="automrlogo.png" width="400"/>
+  <br/><br/>
+  <img src="https://img.shields.io/badge/Python-3.8+-3776AB?style=flat-square&logo=python&logoColor=white"/>
+  <img src="https://img.shields.io/badge/License-MIT-22c55e?style=flat-square"/>
+  <img src="https://img.shields.io/badge/Status-Active-22c55e?style=flat-square"/>
+  <img src="https://img.shields.io/badge/Domain-Autonomous%20Driving-7c6fcd?style=flat-square"/>
+  <br/><br/>
+  <p>Model agnostic + Input agnostic + Output agnostic Metamorphic testing framework for regressional autonomous driving AI/ML models.</p>
+</div>
+
+---
+
+## Overview
+
+AutoMR evaluates ML models by verifying **metamorphic relations (MRs)** — expected behavioral properties that must hold under controlled input transformations. Instead of checking exact outputs against ground truth, AutoMR checks whether the model behaves _consistently_ when inputs are perturbed in predictable ways.
+
+| Problem                  | What AutoMR does                                         |
+| ------------------------ | -------------------------------------------------------- |
+| No labeled data          | Tests models without any ground-truth labels             |
+| Real-world perturbations | Measures robustness under realistic noise and conditions |
+| Silent failures          | Pinpoints when and how models begin to fail              |
+
+---
+
+## Key Features
+
+- **Model-agnostic** — works with TensorFlow, PyTorch, sklearn, or any custom model
+- **Input-agnostic** — supports images, text, tabular, and sequential data
+- **Output-agnostic** — handles regression and classification outputs
+- **Built-in MR pipeline** — end-to-end execution with zero boilerplate
+- **Parametric testing** — sweep transformation parameters across configurable ranges
+- **Automated analysis** — failure rate, severity scores, and worst-case detection
+- **Automatic CSV export** — all results persisted without manual intervention
+- **Optional progress tracking** — live progress bars for long-running tests
+
+---
+
+## Installation
+
+```bash
+git clone https://github.com/CharithManaujayaMUTEC/AutoMR-Framework.git
+cd AutoMR-Framework
+
+python -m venv venv
+venv\Scripts\activate
+
+pip install -r requirements.txt
+```
+
+---
+
+## Quick Start
+
+```python
+from automr.api import AutoMR
+
+automr = AutoMR(model)
+
+df, results = automr.run_full_test(
+    dataset,
+    max_samples=2000,
+    samples_per_mr=5,
+    show_progress=True
+)
+```
+
+---
+
+## Execution Flow
+
+```
+1. Load dataset       →  user-defined input source
+2. Load model         →  any model exposing predict(x)
+3. Apply transforms   →  brightness, rotation, noise, fog, ...
+4. Generate outputs   →  original vs. transformed predictions
+5. Validate MRs       →  check expected behavioral properties
+6. Analyze results    →  failure rate, severity, worst cases
+7. Export             →  CSV files written to /results
+```
+
+---
+
+## Output Files
+
+All results are saved automatically to the `/results` directory.
+
+| File                   | Description                                |
+| ---------------------- | ------------------------------------------ |
+| `automr_results.csv`   | Full per-sample test log                   |
+| `failure_summary.csv`  | Failure rate per metamorphic relation      |
+| `severity_summary.csv` | Average output deviation per MR            |
+| `worst_cases.csv`      | Samples with highest deviation             |
+| `failure_regions.txt`  | Parametric boundaries where failures occur |
+
+### Output columns
+
+| Column              | Description                       |
+| ------------------- | --------------------------------- |
+| `mr`                | Metamorphic relation identifier   |
+| `param`             | Transformation parameter value    |
+| `original`          | Original model prediction         |
+| `transformed`       | Prediction after transformation   |
+| `difference`        | Raw output difference             |
+| `percent_change`    | Percentage change between outputs |
+| `status`            | `PASS` / `FAIL`                   |
+| `expected_behavior` | Expected MR rule                  |
+| `actual_behavior`   | `Consistent` / `Violation`        |
+| `sample_id`         | Input sample index                |
+
+---
+
+## Metamorphic Relations
+
+| Relation              | Description                          | Type       |
+| --------------------- | ------------------------------------ | ---------- |
+| `BrightnessRelation`  | Output invariant to lighting changes | Invariance |
+| `RotationRelation`    | Stable under small rotations         | Invariance |
+| `TranslationRelation` | Stable under image shifts            | Invariance |
+| `NoiseRelation`       | Robust to random noise               | Robustness |
+| `FogRelation`         | Robust to visibility degradation     | Robustness |
+| `TemporalSmoothness`  | Consistent outputs across frames     | Monotonic  |
+
+---
+
+## Transformations
+
+| Transform   | Description               |
+| ----------- | ------------------------- |
+| Brightness  | Adjust pixel intensity    |
+| Rotation    | Rotate image by angle     |
+| Translation | Shift image spatially     |
+| Noise       | Add random Gaussian noise |
+| Fog / Rain  | Simulate adverse weather  |
+| Blur        | Apply smoothing filter    |
+
+---
+
+## Design Principles
+
+### Model-agnostic
+
+Works with any model that implements a `predict(x)` interface:
+
+```python
+# TensorFlow, PyTorch, sklearn, or fully custom — all compatible
+output = model.predict(input)
+```
+
+### Input-agnostic
+
+Accepts any input type — images, sequences, tabular data, or custom formats. Transformations are applied modularly and do not depend on input structure.
+
+### Modular architecture
+
+| Component   | Role                                   |
+| ----------- | -------------------------------------- |
+| `Model`     | Generates predictions                  |
+| `Transform` | Modifies input samples                 |
+| `Relation`  | Defines expected behavioral properties |
+| `Analyzer`  | Computes failure metrics and summaries |
+
+---
+
+## Project Structure
+
+```
+AutoMR-Framework/
+│
+├── automr/
+│   ├── api.py
+│   ├── comparator.py
+│   │
+│   ├── core/
+│   │   ├── range_tester.py
+│   │   └── failure_analysis.py
+│   │
+│   ├── relations/
+│   ├── transforms/
+│   └── analysis/
+│
+├── run_test_example.py
+├── requirements.txt
+├── .gitignore
+└── automrlogo.png
+```
+
+---
+
+## Example Run
+
+```
+Running AutoMR: ██████████████ 100%
+
+=== AutoMR Results ===
+Failure Summary:
+  BrightnessRelation   →  12.4% failure rate  |  avg deviation: 0.031
+  RotationRelation     →   8.7% failure rate  |  avg deviation: 0.019
+  NoiseRelation        →  21.1% failure rate  |  avg deviation: 0.074
+  FogRelation          →  34.2% failure rate  |  avg deviation: 0.112
+
+DONE: Results saved in /results
+```
+
+---
+
+## Built-in Analysis
+
+AutoMR automatically computes the following after each test run:
+
+- **Failure rate** per metamorphic relation
+- **Severity** — average output deviation across failures
+- **Worst-case failures** — samples with the largest deviations
+- **Failure regions** — parameter ranges where the model is most unstable
+
+---
+
+## Limitations
+
+- Current transformation suite is image-focused
+- Comparator thresholds require manual tuning per task
+- End-to-end performance depends on model inference speed
+
+---
+
+## Future Work
+
+- NLP and tabular transformation extensions
+- Classification-specific comparators
+- Streamlit dashboard for interactive analysis
+- Cross-model MR testing
+- Automated result visualizations (plots and charts)
+
+---
+
+## Authors
+
+**CharithManaujayaMUTEC** — [github.com/CharithManaujayaMUTEC](https://github.com/CharithManaujayaMUTEC)  
+**RaveeshaPeiris** — [github.com/RaveeshaPeiris](https://github.com/RaveeshaPeiris)
+
+> Final Year Project — Metamorphic Testing Framework for Regressional Based Autonomous Driving AI/ML Models
+
+---
+
+## License
+
+Released under the [MIT License](LICENSE).

automr-0.1.0/README.md

@@ -0,0 +1,247 @@
+<div align="center">
+  <img src="automrlogo.png" width="400"/>
+  <br/><br/>
+  <img src="https://img.shields.io/badge/Python-3.8+-3776AB?style=flat-square&logo=python&logoColor=white"/>
+  <img src="https://img.shields.io/badge/License-MIT-22c55e?style=flat-square"/>
+  <img src="https://img.shields.io/badge/Status-Active-22c55e?style=flat-square"/>
+  <img src="https://img.shields.io/badge/Domain-Autonomous%20Driving-7c6fcd?style=flat-square"/>
+  <br/><br/>
+  <p>Model agnostic + Input agnostic + Output agnostic Metamorphic testing framework for regressional autonomous driving AI/ML models.</p>
+</div>
+
+---
+
+## Overview
+
+AutoMR evaluates ML models by verifying **metamorphic relations (MRs)** — expected behavioral properties that must hold under controlled input transformations. Instead of checking exact outputs against ground truth, AutoMR checks whether the model behaves _consistently_ when inputs are perturbed in predictable ways.
+
+| Problem                  | What AutoMR does                                         |
+| ------------------------ | -------------------------------------------------------- |
+| No labeled data          | Tests models without any ground-truth labels             |
+| Real-world perturbations | Measures robustness under realistic noise and conditions |
+| Silent failures          | Pinpoints when and how models begin to fail              |
+
+---
+
+## Key Features
+
+- **Model-agnostic** — works with TensorFlow, PyTorch, sklearn, or any custom model
+- **Input-agnostic** — supports images, text, tabular, and sequential data
+- **Output-agnostic** — handles regression and classification outputs
+- **Built-in MR pipeline** — end-to-end execution with zero boilerplate
+- **Parametric testing** — sweep transformation parameters across configurable ranges
+- **Automated analysis** — failure rate, severity scores, and worst-case detection
+- **Automatic CSV export** — all results persisted without manual intervention
+- **Optional progress tracking** — live progress bars for long-running tests
+
+---
+
+## Installation
+
+```bash
+git clone https://github.com/CharithManaujayaMUTEC/AutoMR-Framework.git
+cd AutoMR-Framework
+
+python -m venv venv
+venv\Scripts\activate
+
+pip install -r requirements.txt
+```
+
+---
+
+## Quick Start
+
+```python
+from automr.api import AutoMR
+
+automr = AutoMR(model)
+
+df, results = automr.run_full_test(
+    dataset,
+    max_samples=2000,
+    samples_per_mr=5,
+    show_progress=True
+)
+```
+
+---
+
+## Execution Flow
+
+```
+1. Load dataset       →  user-defined input source
+2. Load model         →  any model exposing predict(x)
+3. Apply transforms   →  brightness, rotation, noise, fog, ...
+4. Generate outputs   →  original vs. transformed predictions
+5. Validate MRs       →  check expected behavioral properties
+6. Analyze results    →  failure rate, severity, worst cases
+7. Export             →  CSV files written to /results
+```
+
+---
+
+## Output Files
+
+All results are saved automatically to the `/results` directory.
+
+| File                   | Description                                |
+| ---------------------- | ------------------------------------------ |
+| `automr_results.csv`   | Full per-sample test log                   |
+| `failure_summary.csv`  | Failure rate per metamorphic relation      |
+| `severity_summary.csv` | Average output deviation per MR            |
+| `worst_cases.csv`      | Samples with highest deviation             |
+| `failure_regions.txt`  | Parametric boundaries where failures occur |
+
+### Output columns
+
+| Column              | Description                       |
+| ------------------- | --------------------------------- |
+| `mr`                | Metamorphic relation identifier   |
+| `param`             | Transformation parameter value    |
+| `original`          | Original model prediction         |
+| `transformed`       | Prediction after transformation   |
+| `difference`        | Raw output difference             |
+| `percent_change`    | Percentage change between outputs |
+| `status`            | `PASS` / `FAIL`                   |
+| `expected_behavior` | Expected MR rule                  |
+| `actual_behavior`   | `Consistent` / `Violation`        |
+| `sample_id`         | Input sample index                |
+
+---
+
+## Metamorphic Relations
+
+| Relation              | Description                          | Type       |
+| --------------------- | ------------------------------------ | ---------- |
+| `BrightnessRelation`  | Output invariant to lighting changes | Invariance |
+| `RotationRelation`    | Stable under small rotations         | Invariance |
+| `TranslationRelation` | Stable under image shifts            | Invariance |
+| `NoiseRelation`       | Robust to random noise               | Robustness |
+| `FogRelation`         | Robust to visibility degradation     | Robustness |
+| `TemporalSmoothness`  | Consistent outputs across frames     | Monotonic  |
+
+---
+
+## Transformations
+
+| Transform   | Description               |
+| ----------- | ------------------------- |
+| Brightness  | Adjust pixel intensity    |
+| Rotation    | Rotate image by angle     |
+| Translation | Shift image spatially     |
+| Noise       | Add random Gaussian noise |
+| Fog / Rain  | Simulate adverse weather  |
+| Blur        | Apply smoothing filter    |
+
+---
+
+## Design Principles
+
+### Model-agnostic
+
+Works with any model that implements a `predict(x)` interface:
+
+```python
+# TensorFlow, PyTorch, sklearn, or fully custom — all compatible
+output = model.predict(input)
+```
+
+### Input-agnostic
+
+Accepts any input type — images, sequences, tabular data, or custom formats. Transformations are applied modularly and do not depend on input structure.
+
+### Modular architecture
+
+| Component   | Role                                   |
+| ----------- | -------------------------------------- |
+| `Model`     | Generates predictions                  |
+| `Transform` | Modifies input samples                 |
+| `Relation`  | Defines expected behavioral properties |
+| `Analyzer`  | Computes failure metrics and summaries |
+
+---
+
+## Project Structure
+
+```
+AutoMR-Framework/
+│
+├── automr/
+│   ├── api.py
+│   ├── comparator.py
+│   │
+│   ├── core/
+│   │   ├── range_tester.py
+│   │   └── failure_analysis.py
+│   │
+│   ├── relations/
+│   ├── transforms/
+│   └── analysis/
+│
+├── run_test_example.py
+├── requirements.txt
+├── .gitignore
+└── automrlogo.png
+```
+
+---
+
+## Example Run
+
+```
+Running AutoMR: ██████████████ 100%
+
+=== AutoMR Results ===
+Failure Summary:
+  BrightnessRelation   →  12.4% failure rate  |  avg deviation: 0.031
+  RotationRelation     →   8.7% failure rate  |  avg deviation: 0.019
+  NoiseRelation        →  21.1% failure rate  |  avg deviation: 0.074
+  FogRelation          →  34.2% failure rate  |  avg deviation: 0.112
+
+DONE: Results saved in /results
+```
+
+---
+
+## Built-in Analysis
+
+AutoMR automatically computes the following after each test run:
+
+- **Failure rate** per metamorphic relation
+- **Severity** — average output deviation across failures
+- **Worst-case failures** — samples with the largest deviations
+- **Failure regions** — parameter ranges where the model is most unstable
+
+---
+
+## Limitations
+
+- Current transformation suite is image-focused
+- Comparator thresholds require manual tuning per task
+- End-to-end performance depends on model inference speed
+
+---
+
+## Future Work
+
+- NLP and tabular transformation extensions
+- Classification-specific comparators
+- Streamlit dashboard for interactive analysis
+- Cross-model MR testing
+- Automated result visualizations (plots and charts)
+
+---
+
+## Authors
+
+**CharithManaujayaMUTEC** — [github.com/CharithManaujayaMUTEC](https://github.com/CharithManaujayaMUTEC)  
+**RaveeshaPeiris** — [github.com/RaveeshaPeiris](https://github.com/RaveeshaPeiris)
+
+> Final Year Project — Metamorphic Testing Framework for Regressional Based Autonomous Driving AI/ML Models
+
+---
+
+## License
+
+Released under the [MIT License](LICENSE).

automr-0.1.0/automr/__init__.py

@@ -0,0 +1 @@
+from .api import AutoMR

automr-0.1.0/automr/analysis.py

@@ -0,0 +1,31 @@
+import pandas as pd
+class Analyzer:
+
+    def to_dataframe(self, results):
+        if not results:
+            return pd.DataFrame()
+
+        df = pd.DataFrame(results)
+        df["status"] = df["passed"].apply(lambda x: "PASS" if x else "FAIL")
+        return df
+
+    def summary(self, df):
+
+        if df.empty:
+            return {
+                "total": 0,
+                "passed": 0,
+                "failed": 0,
+                "pass_rate": 0.0
+            }
+
+        total = len(df)
+        passed = df["passed"].sum()
+        failed = total - passed
+
+        return {
+            "total": total,
+            "passed": int(passed),
+            "failed": int(failed),
+            "pass_rate": float(passed / total * 100)
+        }