patra-toolkit 0.1.1__tar.gz

patra_toolkit-0.1.1/LICENSE

@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, Indiana University Board of Trustees.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

patra_toolkit-0.1.1/PKG-INFO

@@ -0,0 +1,15 @@
+Metadata-Version: 2.1
+Name: patra-toolkit
+Version: 0.1.1
+Summary: Toolkit for semi-automated modelcard creation for AI/ML models.
+Home-page: https://github.com/Data-to-Insight-Center/patra-toolkit.git
+Author: Data to Insight Center
+Author-email: d2i@iu.edu
+License: BSD-3-Clause
+License-File: LICENSE
+Requires-Dist: jsonschema>4.18.5
+Requires-Dist: fairlearn~=0.11.0
+Requires-Dist: shap~=0.46.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: numpy>2.0.0
+Requires-Dist: requests>2.32.2

patra_toolkit-0.1.1/README.md

@@ -0,0 +1,129 @@
+# Patra Model Card Toolkit
+
+[![Documentation Status](https://img.shields.io/badge/docs-latest-blue.svg)](https://patra-toolkit.readthedocs.io/en/latest/)
+[![Build Status](https://github.com/Data-to-Insight-Center/patra-toolkit/actions/workflows/ci.yml/badge.svg)](https://github.com/Data-to-Insight-Center/patra-toolkit/actions)
+[![PyPI version](https://badge.fury.io/py/patra-model-card.svg)](https://pypi.org/project/patra-model-card/)
+
+The Patra Toolkit is a component of the Patra ModelCards framework designed to simplify the process of creating and documenting AI/ML models. It provides a structured schema that guides users in providing essential information about their models, including details about the model's purpose, development process, and performance. The toolkit also includes features for semi-automating the capture of key information, such as fairness and explainability metrics, through integrated analysis tools. By reducing the manual effort involved in creating model cards, the Patra Toolkit encourages researchers and developers to adopt best practices for documenting their models, ultimately contributing to greater transparency and accountability in AI/ML development.
+
+## Features 
+
+- **Structured Schema:** The Patra Toolkit offers a structured schema to guide users in providing crucial model information. This includes details such as the model's intended use, development process, and performance metrics.
+  
+- **Semi-Automated Information Capture:** The toolkit supports semi-automatic capture of certain descriptive fields. It achieves this by running a variety of automated scanners, with the results incorporated into the Model Card.  These include,
+    - **Fairness Scanner** evaluates the model's fairness by examining its predictions across different groups. 
+    - **Explainability Scanner** generates explainability metrics to help understand the model's decision-making process.
+    - **Model Requirements Scanner** captures the Python packages and versions required to run the model.
+
+- **Validation and JSON Generation:** Once a Model Card is created using the Toolkit, it validates the data against the defined schema to ensure completeness and accuracy. It then generates the Model Card as a JSON file, ready for integration into the Patra Knowledge Base.
+  
+- **Integration with Patra Knowledge Base:** The Model Cards created using the Patra Toolkit are designed to be added to the [Patra Knowledge Base](https://github.com/Data-to-Insight-Center/patra-kg), which is a graph database that stores and manages these cards.
+
+The Patra Toolkit plays a crucial role in promoting transparency and accountability in AI/ML development by making it easier for developers to create comprehensive and informative Model Cards. By automating certain aspects of the documentation process and providing a structured schema, the Toolkit reduces the barriers to entry for creating high-quality model documentation.
+
+For more information, please refer to the [Patra ModelCards paper](https://ieeexplore.ieee.org/document/10678710).
+
+## Getting Started
+
+#### (Optional) Create a virtual environment for using Patra Model Card Toolkit
+We recommend creating a new virtual environment using venv before installing patra-model-card. To do so, please follow instructions [here](https://docs.python.org/3/library/venv.html).
+
+#### Installing Patra Model Card
+The latest version can be installed from PyPI:
+```shell
+pip install patra-toolkit
+```
+
+For local installation, clone the repository and install using:
+```shell
+pip install -e <local_git_dir>/patra_toolkit
+```
+
+## Usage
+### Create a Model Card
+Find the descriptions of the Model Card parameters in the [schema descriptions document](https://github.com/Data-to-Insight-Center/patra-toolkit/blob/main/Schema_Descriptions.md).
+
+```python
+from patra_toolkit import ModelCard
+
+mc = ModelCard(
+  name="UCI Adult Data Analysis model using Tensorflow",
+  version="0.1",
+  short_description="UCI Adult Data analysis using Tensorflow for demonstration of Patra Model Cards.",
+  full_description="We have trained a ML model using the tensorflow framework to predict income for the UCI Adult Dataset. We leverage this data to run the Patra model cards to capture metadata about the model as well as fairness and explainability metrics.",
+  keywords="uci adult, tensorflow, explainability, fairness, patra",
+  author="Sachith Withana",
+  input_type="Tabular",
+  category="classification",
+  foundational_model="None"
+)
+
+# Add Model Metadata
+mc.input_data = 'https://archive.ics.uci.edu/dataset/2/adult'
+mc.output_data = 'https://github.iu.edu/swithana/mcwork/rf_sklearn/adult_model.pkl'
+```
+
+### Initialize an AI/ML Model
+
+```python
+from patra_toolkit import AIModel
+
+ai_model = AIModel(
+  name="UCI Adult Random Forest model",
+  version="0.1",
+  description="Census classification problem using Random Forest",
+  owner="Sachith Withana",
+  location="https://github.iu.edu/swithana/mcwork/randomforest/adult_model.pkl",
+  license="BSD-3 Clause",
+  framework="sklearn",
+  model_type="random_forest",
+  test_accuracy=accuracy
+)
+
+# Populate Model Structure
+ai_model.populate_model_structure(trained_model)
+mc.ai_model = ai_model
+
+# Add Custom Metrics
+ai_model.add_metric("Test loss", loss)
+ai_model.add_metric("Epochs", 100)
+ai_model.add_metric("Batch Size", 32)
+ai_model.add_metric("Optimizer", "Adam")
+ai_model.add_metric("Learning Rate", 0.0001)
+ai_model.add_metric("Input Shape", "(26048, 100)")
+```
+
+### Run Fairness and Explainability Scanners
+```python
+# To assess fairness, provide the sensitive feature, test data, labels, and predictions
+mc.populate_bias(X_test, y_test, predictions, "gender", X_test['sex'], clf)
+
+# To generate explainability metrics, specify the dataset, column names, model, and number of features
+mc.populate_xai(X_test, x_columns, model, top_n=10)
+```
+
+### Validate and Save the Model Card
+```python
+# Verify the model card content against the schema
+mc.validate()
+mc.save(<file_path>)
+
+# Capture Python package dependencies and versions
+mc.populate_requirements()
+
+# Upload the model card to the Patra server
+mc.submit(<patra_server_url>)
+```
+
+---
+## Examples
+Explore the following example notebooks and model cards to learn more about how to use the Patra Model Card Toolkit:
+[Notebook Examples](./examples/notebooks), [Model Card Examples](./examples/model_cards)
+
+---
+
+## License
+The Patra Model Card toolkit is developed by Indiana University and distributed under the BSD 3-Clause License. See `LICENSE.txt` for more details.
+
+## Reference
+S. Withana and B. Plale, "Patra ModelCards: AI/ML Accountability in the Edge-Cloud Continuum," 2024 IEEE 20th International Conference on e-Science (e-Science), Osaka, Japan, 2024, pp. 1-10, doi: 10.1109/e-Science62913.2024.10678710. Keywords: Analytical models, Vectors, Edge-cloud continuum, Model cards, AI/ML accountability, Provenance

patra_toolkit-0.1.1/patra_toolkit/__init__.py

@@ -0,0 +1,3 @@
+from .patra_model_card import *
+from .fairlearn_bias import *
+from .shap_xai import *

patra_toolkit-0.1.1/patra_toolkit/fairlearn_bias.py

@@ -0,0 +1,32 @@
+from fairlearn.metrics import demographic_parity_difference, equalized_odds_difference
+
+
+class BiasAnalyzer:
+    """
+    Provides automated capture of fairness metrics in a given AI workflow.
+    """
+    def __init__(self, dataset, true_labels, predicted_labels, sensitive_feature_name, sensitive_feature_data, model):
+        self.dataset = dataset
+        self.true_labels = true_labels
+        self.predicted_labels = predicted_labels
+        self.sensitive_feature = sensitive_feature_name
+        self.sensitive_data = sensitive_feature_data
+        self.model = model
+
+    def calculate_bias_metrics(self):
+        """
+        Calculate bias metrics.
+
+        Args:
+            None
+
+        Returns:
+            dict: A dictionary of bias
+        """
+        demographic_parity = demographic_parity_difference(self.true_labels, self.predicted_labels,
+                                                           sensitive_features=self.sensitive_data)
+        equal_odds_diff = equalized_odds_difference(self.true_labels, self.predicted_labels,
+                                                    sensitive_features=self.sensitive_data)
+
+        return {"demographic_parity_diff": demographic_parity,
+                "equal_odds_difference": equal_odds_diff}

patra_toolkit-0.1.1/patra_toolkit/patra_model_card.py

@@ -0,0 +1,376 @@
+import hashlib
+import json
+import os.path
+from dataclasses import dataclass, field
+from json import JSONEncoder
+from typing import List, Optional, Dict
+
+import jsonschema
+import pkg_resources
+import requests
+
+from .fairlearn_bias import BiasAnalyzer
+from .shap_xai import ExplainabilityAnalyser
+
+SCHEMA_JSON = os.path.join(os.path.dirname(__file__), 'schema', 'schema.json')
+
+
+@dataclass
+class Metric:
+    """
+    Data class for storing metric key-value pairs.
+
+    Args:
+        key (str): The name of the metric.
+        value (str): The value of the metric.
+    """
+    key: str
+    value: str
+
+@dataclass
+class AIModel:
+    """
+    Represents and stores AI model metadata and its performance metrics.
+
+    Args:
+        name (str): The name of the model.
+        version (str): The version identifier of the model.
+        description (str): A detailed description of the model.
+        owner (str): The owner of the model.
+        location (str): The file path or URL where the model is stored.
+        license (str): The license under which the model is distributed.
+        framework (str): The framework used to build the model (e.g., TensorFlow, PyTorch).
+        model_type (str): The type of model (e.g., classifier, regressor).
+        test_accuracy (str): The accuracy of the model on a test dataset.
+        model_structure (str): The structure of the model as a dictionary (optional).
+        metrics (str): A dictionary storing performance metrics for the model.
+
+    Example:
+        .. code-block:: python
+
+            ai_model = AIModel(
+                name="Model Name",
+                version="1.0",
+                description="Model description",
+                owner="Model owner",
+                location="Model location",
+                license="Model license",
+                framework="Model framework",
+                model_type="Model type",
+                test_accuracy=0.95,
+                model_structure={},
+                metrics={"accuracy": "0.95"}
+            )
+    """
+    name: str
+    version: str
+    description: str
+    owner: str
+    location: str
+    license: str
+    framework: str
+    model_type: str
+    test_accuracy: float
+    model_structure: Optional[object] = field(default_factory=dict)
+    metrics: Dict[str, str] = field(default_factory=dict)
+
+    def add_metric(self, key: str, value: str) -> None:
+        """
+        Adds a performance metric to the model's metrics.
+
+        Args:
+            key (str): The name of the metric.
+            value (str): The value of the metric.
+
+        Returns:
+            None
+        """
+        self.metrics[key] = value
+
+    def remove_nulls(self, model_structure):
+        """
+        Recursively removes null values from the model structure.
+
+        Args:
+            model_structure (object): The model structure as a dictionary or list.
+
+        Returns:
+            object: Model structure with null values removed.
+        """
+        if isinstance(model_structure, dict):
+            return {k: self.remove_nulls(v) for k, v in model_structure.items() if v is not None}
+        elif isinstance(model_structure, list):
+            return [self.remove_nulls(v) for v in model_structure if v is not None]
+        return model_structure
+
+    def populate_model_structure(self, trained_model):
+        """
+        Populates the `model_structure` attribute from a trained model object.
+
+        Args:
+            trained_model (object): A trained machine learning model object.
+
+        Returns:
+            None
+        """
+        if self.framework == 'tensorflow':
+            json_structure = json.loads(trained_model.to_json())
+            self.model_structure = self.remove_nulls(json_structure)
+        else:
+            self.model_structure = {}
+
+
+@dataclass
+class BiasAnalysis:
+    """
+    Class to store results from bias analysis.
+
+    Args:
+        demographic_parity_difference (float): The difference in demographic parity between groups.
+        equal_odds_difference (float): The difference in equal odds between groups.
+    """
+    demographic_parity_difference: float
+    equal_odds_difference: float
+
+
+@dataclass
+class ExplainabilityAnalysis:
+    """
+    Class to store explainability metrics.
+
+    Args:
+        name (str): Name of the explainability method used.
+        metrics (List[Metric]): List of metrics related to explainability analysis.
+    """
+    name: str
+    metrics: List[Metric] = field(default_factory=list)
+
+
+@dataclass
+class ModelCard:
+    """
+    Represents an AI model card to document model metadata, analyses, and requirements.
+
+    Args:
+        name (str): The name of the model.
+        version (str): The model's version.
+        short_description (str): A brief description of the model.
+        full_description (str): A detailed description of the model.
+        keywords (str): Comma-separated keywords for searchability.
+        author (str): The model's creator or owner.
+        input_type (str): Type of input data (e.g., "Image", "Text").
+        category (str): The category of the model (e.g., "Classification", "Regression").
+        input_data (Optional[str]): Description of the model's input data.
+        output_data (Optional[str]): Description of the model's output data.
+        foundational_model (Optional[str]): Reference to any foundational model used.
+        ai_model (Optional[AIModel]): An instance of `AIModel` containing model details.
+        bias_analysis (Optional[BiasAnalysis]): Instance of `BiasAnalysis` containing bias metrics.
+        xai_analysis (Optional[ExplainabilityAnalysis]): Instance of `ExplainabilityAnalysis` with interpretability metrics.
+        model_requirements (Optional[List[str]]): List of required packages and dependencies.
+        id (Optional[str]): Unique identifier for the model card, generated upon submission.
+
+    Example:
+        .. code-block:: python
+
+            model_card = ModelCard(
+                name="Model Name",
+                version="1.0",
+                short_description="A brief description",
+                full_description="A detailed description of the model's purpose and usage.",
+                keywords="classification, AI, image processing",
+                author="Author Name",
+                input_type="Image",
+                category="Classification",
+                input_data="Images of size 28x28.",
+                output_data="Prediction probabilities for classes.",
+                foundational_model="Base Model Reference",
+                ai_model=AIModel(
+                    name="Model Name",
+                    version="1.0",
+                    description="Detailed model description",
+                    owner="Model owner",
+                    location="Storage location",
+                    license="MIT",
+                    framework="TensorFlow",
+                    model_type="Classifier",
+                    test_accuracy=0.95,
+                    model_structure={},
+                    metrics={"accuracy": "0.95"}
+                ),
+                bias_analysis=BiasAnalysis(
+                    demographic_parity_difference=0.05,
+                    equal_odds_difference=0.1
+                ),
+                xai_analysis=ExplainabilityAnalysis(
+                    name="SHAP",
+                    metrics=[Metric(key="Feature A", value="0.1")]
+                ),
+                model_requirements=["numpy>=1.19.2", "tensorflow>=2.4.1"]
+            )
+    """
+    name: str
+    version: str
+    short_description: str
+    full_description: str
+    keywords: str
+    author: str
+    input_type: str
+    category: str
+    input_data: Optional[str] = ""
+    output_data: Optional[str] = ""
+    foundational_model: Optional[str] = ""
+    ai_model: Optional[AIModel] = None
+    bias_analysis: Optional[BiasAnalysis] = None
+    xai_analysis: Optional[ExplainabilityAnalysis] = None
+    model_requirements: Optional[List] = None
+    id: Optional[str] = field(init=False, default=None)
+
+    def __str__(self):
+        """
+        Returns a JSON string representation of the model card.
+
+        Returns:
+            str: A JSON-formatted string representing the model card.
+        """
+        return json.dumps(self.__dict__, cls=ModelCardJSONEncoder, indent=4, separators=(',', ': '))
+
+    def populate_bias(self, dataset, true_labels, predicted_labels, sensitive_feature_name, sensitive_feature_data, model):
+        """
+        Calculates and stores fairness metrics.
+
+        Args:
+            dataset (object): The dataset used for bias analysis.
+            true_labels (list): The ground truth labels.
+            predicted_labels (list): Model's predictions.
+            sensitive_feature_name (str): The name of the sensitive attribute.
+            sensitive_feature_data (list): Values for the sensitive feature.
+            model (object): The model being analyzed.
+
+        Returns:
+            None
+        """
+        bias_analyzer = BiasAnalyzer(dataset, true_labels, predicted_labels, sensitive_feature_name,
+                                     sensitive_feature_data, model)
+        self.bias_analysis = bias_analyzer.calculate_bias_metrics()
+
+    def populate_xai(self, train_dataset, column_names, model, n_features=10):
+        """
+        Computes and stores feature importance metrics.
+
+        Args:
+            train_dataset (object): Training dataset used in the analysis.
+            column_names (list): Names of the features.
+            model (object): The model being explained.
+            n_features (int, optional): Number of features to analyze. Default is 10.
+
+        Returns:
+            None
+        """
+        xai_analyzer = ExplainabilityAnalyser(train_dataset, column_names, model)
+        self.xai_analysis = xai_analyzer.calculate_xai_features(n_features)
+
+    def populate_requirements(self):
+        """
+        Collects package requirements for the model card, excluding specific dependencies.
+
+        Returns:
+            None
+        """
+        exclude_packages = {"shap", "fairlearn"}
+        installed_packages = pkg_resources.working_set
+        packages_list = sorted(["%s==%s" % (i.key, i.version) for i in installed_packages])
+        self.model_requirements = [pkg for pkg in packages_list if pkg.split("==")[0] not in exclude_packages]
+
+    def validate(self):
+        """
+        Validates the model card against a predefined JSON schema.
+
+        Returns:
+            bool: True if the model card is valid according to the schema, False otherwise.
+        """
+        mc_json = self.__str__()
+        try:
+            with open(SCHEMA_JSON, 'r') as schema_file:
+                schema = json.load(schema_file)
+            jsonschema.validate(instance=json.loads(mc_json), schema=schema)
+            return True
+        except jsonschema.ValidationError as e:
+            print(e.message)
+            return False
+        except Exception as e:
+            print(f"An unexpected error occurred: {e}")
+            return False
+
+    def submit(self, patra_server_url):
+        """
+        Validates and submits the model card to the specified Patra server.
+
+        Args:
+            patra_server_url (str): The Patra server URL where the model card should be submitted.
+
+        Returns:
+            dict: The server's response as a JSON object.
+        """
+        if self.validate():
+            try:
+                self.id = self._get_hash_id(patra_server_url)
+                patra_submit_url = f"{patra_server_url}/upload_mc"
+                headers = {'Content-Type': 'application/json'}
+                response = requests.post(patra_submit_url, json=json.loads(str(self)), headers=headers)
+                response.raise_for_status()
+                return response.json()
+            except requests.exceptions.RequestException as e:
+                print("The Patra Server cannot be reached. Please try again.")
+                return None
+        return {"An error occurred: valid patra_server_url not provided. Unable to upload."}
+
+    def _get_hash_id(self, patra_server_url):
+        """
+        Generates a unique identifier for the model card based on its metadata.
+
+        Args:
+            patra_server_url (str): The Patra server URL used to generate the ID.
+
+        Returns:
+            str: A unique hash identifier for the model card.
+        """
+        combined_string = f"{self.name}:{self.version}:{self.author}"
+        try:
+            if patra_server_url:
+                patra_hash_url = f"{patra_server_url}/get_hash_id"
+                headers = {'Content-Type': 'application/json'}
+                response = requests.get(patra_hash_url, params={"combined_string": combined_string}, headers=headers)
+                response.raise_for_status()
+                return response.json()
+            else:
+                return hashlib.sha256(combined_string.encode()).hexdigest()
+        except requests.exceptions.RequestException as e:
+            print("Could not connect to the Patra Server, generating the ID locally")
+            return hashlib.sha256(combined_string.encode()).hexdigest()
+
+    def save(self, file_location):
+        """
+        Saves the model card as a JSON file to the specified location.
+
+        Args:
+            file_location (str): The path where the model card JSON file will be saved.
+
+        Returns:
+            None
+        """
+        with open(file_location, 'w') as json_file:
+            json_file.write(str(self))
+
+
+class ModelCardJSONEncoder(JSONEncoder):
+    """
+    Custom JSON Encoder for ModelCard to handle complex objects.
+
+    Methods:
+        default: Serializes non-serializable fields.
+    """
+
+    def default(self, obj):
+        if isinstance(obj, (ModelCard, Metric, AIModel, ExplainabilityAnalysis, BiasAnalysis)):
+            return obj.__dict__
+        return super().default(obj)