auditml 0.1.0__tar.gz

auditml-0.1.0/PKG-INFO

@@ -0,0 +1,267 @@
+Metadata-Version: 2.4
+Name: auditml
+Version: 0.1.0
+Summary: A privacy auditing toolkit for PyTorch machine learning models.
+Author-email: Eeman Asghar <2022bba124@student.uet.edu.pk>
+License: MIT
+Project-URL: Homepage, https://github.com/EemanAsghar/AuditML-Privacy-Toolkit
+Project-URL: Documentation, https://eemanasghar.github.io/AuditML-Privacy-Toolkit/
+Project-URL: Repository, https://github.com/EemanAsghar/AuditML-Privacy-Toolkit
+Project-URL: Bug Tracker, https://github.com/EemanAsghar/AuditML-Privacy-Toolkit/issues
+Keywords: privacy,machine-learning,pytorch,membership-inference,differential-privacy
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: torch>=2.1.0
+Requires-Dist: torchvision>=0.16.0
+Requires-Dist: numpy>=1.26.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: scikit-learn>=1.3.0
+Requires-Dist: opacus>=1.4.0
+Requires-Dist: matplotlib>=3.8.0
+Requires-Dist: seaborn>=0.12.0
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: tqdm>=4.66.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.7.0; extra == "dev"
+Requires-Dist: mkdocs-material>=9.0.0; extra == "dev"
+Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "dev"
+
+# AuditML
+
+**Privacy Auditing Toolkit for PyTorch Models**
+
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.x-orange)](https://pytorch.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green)](LICENSE)
+[![Docs](https://img.shields.io/badge/docs-mkdocs-blueviolet)](https://eemanasghar.github.io/AuditML-Privacy-Toolkit/)
+
+AuditML lets you measure how much private information leaks from a trained PyTorch model.
+One function call audits your model for membership inference, model inversion, and attribute
+inference attacks — with an interactive HTML report that opens automatically in your browser.
+
+---
+
+## Features
+
+| | |
+|---|---|
+| **Threshold MIA** | Exploit loss/confidence/entropy gaps between members and non-members |
+| **Shadow Model MIA** | Train surrogate models to build a membership classifier |
+| **Model Inversion** | Reconstruct per-class images via gradient ascent |
+| **Attribute Inference** | Predict sensitive attributes from model outputs |
+| **DP Training** | Opacus DP-SGD with automatic (ε, δ) accounting |
+| **HTML Reports** | Interactive browser report — charts, ROC curves, risk level — auto-opens after audit |
+| **Rust acceleration** | 11× faster threshold scanning, 3× faster SSIM (optional) |
+
+---
+
+## Quick Start
+
+```bash
+pip install auditml
+```
+
+```python
+import auditml
+
+# Split your training set into members / non-members
+member_loader, nonmember_loader = auditml.split_loaders(train_dataset)
+
+# Audit your model — works with any nn.Module
+results = auditml.audit(model, member_loader, nonmember_loader)
+
+print(results.summary())
+# ⚠  Leakage detected — highest AUC: 0.641 (mia_threshold)
+
+# Open an interactive HTML report in your browser
+results.report("./report", open_browser=True)
+
+# Save results to reload later
+results.save("audit_results.json")
+results2 = auditml.AuditResults.load("audit_results.json")
+```
+
+---
+
+## Installation
+
+```bash
+pip install auditml
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/EemanAsghar/AuditML-Privacy-Toolkit.git
+cd AuditML-Privacy-Toolkit
+pip install -e ".[dev]"
+```
+
+### Optional: Rust extension (~11× speedup)
+
+```bash
+pip install maturin
+cd rust && maturin build --release --out ../dist
+pip install ../dist/auditml_rust-*.whl --force-reinstall
+```
+
+---
+
+## Python API
+
+```python
+import auditml
+from torch.utils.data import DataLoader
+
+# 1. Split dataset into members / non-members
+member_loader, nonmember_loader = auditml.split_loaders(
+    train_dataset,
+    member_ratio=0.5,   # 50/50 split
+    batch_size=64,
+    seed=42,
+)
+
+# 2. Run all attacks (or pick specific ones)
+results = auditml.audit(
+    model,
+    member_loader,
+    nonmember_loader,
+    attacks=["mia_threshold", "model_inversion"],  # omit for all 4
+    device="auto",   # auto | cpu | cuda | mps
+)
+
+# 3. Inspect results
+print(results.summary())
+results["mia_threshold"].auc_roc   # → 0.641
+results.most_vulnerable()          # → AttackSummary(mia_threshold: ...)
+results.is_vulnerable()            # → True
+
+# 4. HTML report (auto-opens in browser)
+results.report("./my_report", open_browser=True)
+
+# 5. Save / reload without re-running
+results.save("results.json")
+results2 = auditml.AuditResults.load("results.json")
+```
+
+### Shadow MIA with a custom architecture
+
+```python
+results = auditml.audit(
+    model, member_loader, nonmember_loader,
+    attacks=["mia_shadow"],
+    shadow_model_fn=lambda: MyCNN(num_classes=10),  # optional — MLP fallback used if omitted
+)
+```
+
+---
+
+## CLI
+
+```bash
+# Train a model
+auditml train --config configs/mnist_baseline.yaml
+
+# Run a full privacy audit (opens HTML report automatically)
+auditml audit --config configs/mnist_baseline.yaml
+
+# Run specific attacks
+auditml audit --config configs/mnist_baseline.yaml --attack mia_threshold model_inversion
+
+# Print resolved config as JSON
+auditml show-config --config configs/mnist_baseline.yaml
+```
+
+---
+
+## Config format
+
+```yaml
+experiment_name: mnist_baseline
+
+data:
+  dataset: mnist          # mnist | cifar10 | cifar100
+  train_ratio: 0.5
+
+model:
+  arch: cnn               # cnn | resnet
+
+training:
+  epochs: 30
+  batch_size: 64
+  learning_rate: 0.001
+  device: auto            # auto | cpu | cuda | mps
+
+attacks:
+  - mia_threshold
+
+dp:
+  enabled: false
+  epsilon: 5.0
+  delta: 1.0e-5
+  max_grad_norm: 1.0
+
+reporting:
+  output_dir: ./outputs
+```
+
+---
+
+## Project structure
+
+```
+AuditML/
+├── src/auditml/
+│   ├── attacks/           # MIA, shadow, model inversion, attribute inference
+│   ├── config/            # YAML schema → typed dataclasses
+│   ├── data/              # Dataset loaders (MNIST, CIFAR-10, CIFAR-100)
+│   ├── models/            # CNN + ResNet architectures
+│   ├── training/          # Standard trainer + Opacus DP trainer
+│   ├── reporting/         # Report generator, HTML report, comparison modules
+│   └── utils/             # Device detection, Rust acceleration, logging
+├── rust/                  # Rust/PyO3 extension (optional)
+├── configs/               # Example YAML configs
+├── scripts/               # Experiment runners
+├── benchmarks/            # Rust vs NumPy benchmark
+├── tests/                 # pytest suite — 380 tests
+└── docs/                  # MkDocs documentation
+```
+
+---
+
+## Benchmark: Rust acceleration
+
+```
+find_best_threshold  (N=10,000)
+  NumPy   : 160.00 ms
+  Rust    :  14.00 ms
+  Speedup : 11.4x  ✅
+
+compute_ssim  (pixels=784)
+  NumPy   :  21.0 µs
+  Rust    :   7.0 µs
+  Speedup :  3.0x  ✅
+```
+
+---
+
+## Documentation
+
+Full documentation at **[eemanasghar.github.io/AuditML-Privacy-Toolkit](https://eemanasghar.github.io/AuditML-Privacy-Toolkit/)**
+
+---
+
+## License
+
+MIT © Eeman Asghar, NUML Faisalabad, 2025

auditml-0.1.0/README.md

@@ -0,0 +1,228 @@
+# AuditML
+
+**Privacy Auditing Toolkit for PyTorch Models**
+
+[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
+[![PyTorch](https://img.shields.io/badge/PyTorch-2.x-orange)](https://pytorch.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green)](LICENSE)
+[![Docs](https://img.shields.io/badge/docs-mkdocs-blueviolet)](https://eemanasghar.github.io/AuditML-Privacy-Toolkit/)
+
+AuditML lets you measure how much private information leaks from a trained PyTorch model.
+One function call audits your model for membership inference, model inversion, and attribute
+inference attacks — with an interactive HTML report that opens automatically in your browser.
+
+---
+
+## Features
+
+| | |
+|---|---|
+| **Threshold MIA** | Exploit loss/confidence/entropy gaps between members and non-members |
+| **Shadow Model MIA** | Train surrogate models to build a membership classifier |
+| **Model Inversion** | Reconstruct per-class images via gradient ascent |
+| **Attribute Inference** | Predict sensitive attributes from model outputs |
+| **DP Training** | Opacus DP-SGD with automatic (ε, δ) accounting |
+| **HTML Reports** | Interactive browser report — charts, ROC curves, risk level — auto-opens after audit |
+| **Rust acceleration** | 11× faster threshold scanning, 3× faster SSIM (optional) |
+
+---
+
+## Quick Start
+
+```bash
+pip install auditml
+```
+
+```python
+import auditml
+
+# Split your training set into members / non-members
+member_loader, nonmember_loader = auditml.split_loaders(train_dataset)
+
+# Audit your model — works with any nn.Module
+results = auditml.audit(model, member_loader, nonmember_loader)
+
+print(results.summary())
+# ⚠  Leakage detected — highest AUC: 0.641 (mia_threshold)
+
+# Open an interactive HTML report in your browser
+results.report("./report", open_browser=True)
+
+# Save results to reload later
+results.save("audit_results.json")
+results2 = auditml.AuditResults.load("audit_results.json")
+```
+
+---
+
+## Installation
+
+```bash
+pip install auditml
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/EemanAsghar/AuditML-Privacy-Toolkit.git
+cd AuditML-Privacy-Toolkit
+pip install -e ".[dev]"
+```
+
+### Optional: Rust extension (~11× speedup)
+
+```bash
+pip install maturin
+cd rust && maturin build --release --out ../dist
+pip install ../dist/auditml_rust-*.whl --force-reinstall
+```
+
+---
+
+## Python API
+
+```python
+import auditml
+from torch.utils.data import DataLoader
+
+# 1. Split dataset into members / non-members
+member_loader, nonmember_loader = auditml.split_loaders(
+    train_dataset,
+    member_ratio=0.5,   # 50/50 split
+    batch_size=64,
+    seed=42,
+)
+
+# 2. Run all attacks (or pick specific ones)
+results = auditml.audit(
+    model,
+    member_loader,
+    nonmember_loader,
+    attacks=["mia_threshold", "model_inversion"],  # omit for all 4
+    device="auto",   # auto | cpu | cuda | mps
+)
+
+# 3. Inspect results
+print(results.summary())
+results["mia_threshold"].auc_roc   # → 0.641
+results.most_vulnerable()          # → AttackSummary(mia_threshold: ...)
+results.is_vulnerable()            # → True
+
+# 4. HTML report (auto-opens in browser)
+results.report("./my_report", open_browser=True)
+
+# 5. Save / reload without re-running
+results.save("results.json")
+results2 = auditml.AuditResults.load("results.json")
+```
+
+### Shadow MIA with a custom architecture
+
+```python
+results = auditml.audit(
+    model, member_loader, nonmember_loader,
+    attacks=["mia_shadow"],
+    shadow_model_fn=lambda: MyCNN(num_classes=10),  # optional — MLP fallback used if omitted
+)
+```
+
+---
+
+## CLI
+
+```bash
+# Train a model
+auditml train --config configs/mnist_baseline.yaml
+
+# Run a full privacy audit (opens HTML report automatically)
+auditml audit --config configs/mnist_baseline.yaml
+
+# Run specific attacks
+auditml audit --config configs/mnist_baseline.yaml --attack mia_threshold model_inversion
+
+# Print resolved config as JSON
+auditml show-config --config configs/mnist_baseline.yaml
+```
+
+---
+
+## Config format
+
+```yaml
+experiment_name: mnist_baseline
+
+data:
+  dataset: mnist          # mnist | cifar10 | cifar100
+  train_ratio: 0.5
+
+model:
+  arch: cnn               # cnn | resnet
+
+training:
+  epochs: 30
+  batch_size: 64
+  learning_rate: 0.001
+  device: auto            # auto | cpu | cuda | mps
+
+attacks:
+  - mia_threshold
+
+dp:
+  enabled: false
+  epsilon: 5.0
+  delta: 1.0e-5
+  max_grad_norm: 1.0
+
+reporting:
+  output_dir: ./outputs
+```
+
+---
+
+## Project structure
+
+```
+AuditML/
+├── src/auditml/
+│   ├── attacks/           # MIA, shadow, model inversion, attribute inference
+│   ├── config/            # YAML schema → typed dataclasses
+│   ├── data/              # Dataset loaders (MNIST, CIFAR-10, CIFAR-100)
+│   ├── models/            # CNN + ResNet architectures
+│   ├── training/          # Standard trainer + Opacus DP trainer
+│   ├── reporting/         # Report generator, HTML report, comparison modules
+│   └── utils/             # Device detection, Rust acceleration, logging
+├── rust/                  # Rust/PyO3 extension (optional)
+├── configs/               # Example YAML configs
+├── scripts/               # Experiment runners
+├── benchmarks/            # Rust vs NumPy benchmark
+├── tests/                 # pytest suite — 380 tests
+└── docs/                  # MkDocs documentation
+```
+
+---
+
+## Benchmark: Rust acceleration
+
+```
+find_best_threshold  (N=10,000)
+  NumPy   : 160.00 ms
+  Rust    :  14.00 ms
+  Speedup : 11.4x  ✅
+
+compute_ssim  (pixels=784)
+  NumPy   :  21.0 µs
+  Rust    :   7.0 µs
+  Speedup :  3.0x  ✅
+```
+
+---
+
+## Documentation
+
+Full documentation at **[eemanasghar.github.io/AuditML-Privacy-Toolkit](https://eemanasghar.github.io/AuditML-Privacy-Toolkit/)**
+
+---
+
+## License
+
+MIT © Eeman Asghar, NUML Faisalabad, 2025

auditml-0.1.0/pyproject.toml

@@ -0,0 +1,87 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel>=0.41.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "auditml"
+version = "0.1.0"
+description = "A privacy auditing toolkit for PyTorch machine learning models."
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.10"
+authors = [
+    {name = "Eeman Asghar", email = "2022bba124@student.uet.edu.pk"},
+]
+keywords = ["privacy", "machine-learning", "pytorch", "membership-inference", "differential-privacy"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+
+dependencies = [
+    "torch>=2.1.0",
+    "torchvision>=0.16.0",
+    "numpy>=1.26.0",
+    "pandas>=2.0.0",
+    "scikit-learn>=1.3.0",
+    "opacus>=1.4.0",
+    "matplotlib>=3.8.0",
+    "seaborn>=0.12.0",
+    "pyyaml>=6.0.1",
+    "tqdm>=4.66.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/EemanAsghar/AuditML-Privacy-Toolkit"
+Documentation = "https://eemanasghar.github.io/AuditML-Privacy-Toolkit/"
+Repository = "https://github.com/EemanAsghar/AuditML-Privacy-Toolkit"
+"Bug Tracker" = "https://github.com/EemanAsghar/AuditML-Privacy-Toolkit/issues"
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.4.0",
+    "pytest-cov>=4.1.0",
+    "ruff>=0.1.0",
+    "mypy>=1.7.0",
+    "mkdocs-material>=9.0.0",
+    "mkdocstrings[python]>=0.24.0",
+]
+
+[project.scripts]
+auditml = "auditml.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+auditml = ["py.typed"]
+
+[tool.ruff]
+target-version = "py310"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "W", "UP"]
+ignore = [
+    "N812",   # torch.nn.functional as F is standard PyTorch convention
+    "E402",   # sys.path manipulation before project imports is intentional in scripts
+]
+
+[tool.ruff.lint.per-file-ignores]
+"src/auditml/reporting/html_report.py" = ["E501"]  # HTML/CSS template strings
+
+[tool.mypy]
+python_version = "3.10"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = true
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-v"

auditml-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

auditml-0.1.0/src/auditml/__init__.py

@@ -0,0 +1,20 @@
+"""AuditML — A privacy auditing toolkit for PyTorch models.
+
+Quick start
+-----------
+>>> import auditml
+>>> results = auditml.audit(model, train_loader, test_loader)
+>>> print(results.summary())
+"""
+
+from auditml.auditor import AttackSummary, AuditResults, audit, split_loaders
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "audit",
+    "split_loaders",
+    "AuditResults",
+    "AttackSummary",
+    "__version__",
+]

auditml-0.1.0/src/auditml/attacks/__init__.py

@@ -0,0 +1,87 @@
+"""AuditML privacy attack implementations.
+
+All attacks inherit from ``BaseAttack`` and return ``AttackResult``.
+Use ``get_attack()`` to instantiate an attack by name.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import torch.nn as nn
+
+from auditml.attacks.base import BaseAttack
+from auditml.attacks.results import AttackResult
+from auditml.config.schema import AttackType, AuditMLConfig
+
+if TYPE_CHECKING:
+    pass
+
+# Registry mapping AttackType → concrete class.
+# Entries are added as each attack is implemented in Tasks 2.2–2.12.
+# Using strings for lazy imports avoids circular-import issues and
+# means we don't crash if an attack file has an uninstalled dependency.
+_ATTACK_REGISTRY: dict[AttackType, str] = {
+    AttackType.MIA_THRESHOLD: "auditml.attacks.mia_threshold.ThresholdMIA",
+    AttackType.MIA_SHADOW: "auditml.attacks.mia_shadow.ShadowMIA",
+    AttackType.MODEL_INVERSION: "auditml.attacks.model_inversion.ModelInversion",
+    AttackType.ATTRIBUTE_INFERENCE: "auditml.attacks.attribute_inference.AttributeInference",
+}
+
+
+def get_attack(
+    attack_type: AttackType | str,
+    target_model: nn.Module,
+    config: AuditMLConfig,
+    device: str = "cpu",
+    **kwargs,
+) -> BaseAttack:
+    """Instantiate a concrete attack by type.
+
+    Parameters
+    ----------
+    attack_type:
+        Which attack to create — an ``AttackType`` enum value or its
+        string form (e.g. ``"mia_threshold"``).
+    target_model:
+        The trained model to attack.
+    config:
+        Full AuditML configuration.
+    device:
+        Torch device string.
+
+    Returns
+    -------
+    BaseAttack
+        A ready-to-run attack instance.
+
+    Raises
+    ------
+    ValueError
+        If *attack_type* is not recognised or not yet implemented.
+    """
+    if isinstance(attack_type, str):
+        attack_type = AttackType(attack_type)
+
+    if attack_type not in _ATTACK_REGISTRY:
+        implemented = [k.value for k in _ATTACK_REGISTRY]
+        raise ValueError(
+            f"Attack {attack_type.value!r} is not yet implemented. "
+            f"Available: {implemented or 'none yet — coming in Tasks 2.2-2.12'}"
+        )
+
+    # Lazy import: "auditml.attacks.mia_threshold.ThresholdMIA" → class
+    dotted_path = _ATTACK_REGISTRY[attack_type]
+    module_path, class_name = dotted_path.rsplit(".", 1)
+    import importlib
+    module = importlib.import_module(module_path)
+    cls = getattr(module, class_name)
+
+    return cls(target_model=target_model, config=config, device=device, **kwargs)
+
+
+__all__ = [
+    "AttackResult",
+    "BaseAttack",
+    "get_attack",
+]