validmind 2.5.6__py3-none-any.whl → 2.5.15__py3-none-any.whl

validmind/tests/model_validation/statsmodels/GINITable.py

@@ -2,34 +2,37 @@
 # See the LICENSE file in the root of this repository for details.
 # SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial
 
-from dataclasses import dataclass
-
 import numpy as np
 import pandas as pd
 from sklearn.metrics import roc_auc_score, roc_curve
 
-from validmind.vm_models import Metric, ResultSummary, ResultTable, ResultTableMetadata
+from validmind import tags, tasks
 
 
-@dataclass
-class GINITable(Metric):
+@tags("model_performance")
+@tasks("classification")
+def GINITable(dataset, model):
     """
     Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets.
 
-    **Purpose**: The 'GINITable' metric is designed to evaluate the performance of a classification model by
-    emphasizing its discriminatory power. Specifically, it calculates and presents three important metrics - the Area
-    under the ROC Curve (AUC), the GINI coefficient, and the Kolmogov-Smirnov (KS) statistic - for both training and
-    test datasets.
+    ### Purpose
+
+    The 'GINITable' metric is designed to evaluate the performance of a classification model by emphasizing its
+    discriminatory power. Specifically, it calculates and presents three important metrics - the Area under the ROC
+    Curve (AUC), the GINI coefficient, and the Kolmogorov-Smirnov (KS) statistic - for both training and test datasets.
+
+    ### Test Mechanism
 
-    **Test Mechanism**: Using a dictionary for storing performance metrics for both the training and test datasets, the
-    'GINITable' metric calculates each of these metrics sequentially. The Area under the ROC Curve (AUC) is calculated
-    via the `roc_auc_score` function from the Scikit-Learn library. The GINI coefficient, a measure of statistical
-    dispersion, is then computed by doubling the AUC and subtracting 1. Finally, the Kolmogov-Smirnov (KS) statistic is
+    Using a dictionary for storing performance metrics for both the training and test datasets, the 'GINITable' metric
+    calculates each of these metrics sequentially. The Area under the ROC Curve (AUC) is calculated via the
+    `roc_auc_score` function from the Scikit-Learn library. The GINI coefficient, a measure of statistical dispersion,
+    is then computed by doubling the AUC and subtracting 1. Finally, the Kolmogorov-Smirnov (KS) statistic is
     calculated via the `roc_curve` function from Scikit-Learn, with the False Positive Rate (FPR) subtracted from the
     True Positive Rate (TPR) and the maximum value taken from the resulting data. These metrics are then stored in a
     pandas DataFrame for convenient visualization.
 
-    **Signs of High Risk**:
+    ### Signs of High Risk
+
     - Low values for performance metrics may suggest a reduction in model performance, particularly a low AUC which
     indicates poor classification performance, or a low GINI coefficient, which could suggest a decreased ability to
     discriminate different classes.
@@ -38,7 +41,8 @@ class GINITable(Metric):
     - Significant discrepancies between the performance on the training dataset and the test dataset may present
     another signal of high risk.
 
-    **Strengths**:
+    ### Strengths
+
     - Offers three key performance metrics (AUC, GINI, and KS) in one test, providing a more comprehensive evaluation
     of the model.
     - Provides a direct comparison between the model's performance on training and testing datasets, which aids in
@@ -47,7 +51,8 @@ class GINITable(Metric):
     performance even when dealing with imbalanced datasets.
     - Presents the metrics in a user-friendly table format for easy comprehension and analysis.
 
-    **Limitations**:
+    ### Limitations
+
     - The GINI coefficient and KS statistic are both dependent on the AUC value. Therefore, any errors in the
     calculation of the latter will adversely impact the former metrics too.
     - Mainly suited for binary classification models and may require modifications for effective application in
@@ -57,64 +62,26 @@ class GINITable(Metric):
     lead to inaccuracies in the metrics if the data is not appropriately preprocessed.
     """
 
-    name = "gini_table"
-    required_inputs = ["model", "datasets"]
-    tasks = ["classification"]
-    tags = ["visualization", "model_performance"]
-
-    def run(self):
-
-        summary_metrics = self.compute_metrics()
-
-        return self.cache_results(
-            {
-                "metrics_summary": summary_metrics.to_dict(orient="records"),
-            }
-        )
-
-    def compute_metrics(self):
-        """Computes AUC, GINI, and KS for an arbitrary number of datasets."""
-        # Initialize the dictionary to store results
-        metrics_dict = {"Dataset": [], "AUC": [], "GINI": [], "KS": []}
-
-        # Iterate over each dataset in the inputs
-        for _, dataset in enumerate(self.inputs.datasets):
-            dataset_label = (
-                dataset.input_id
-            )  # Use input_id as the label for each dataset
-            metrics_dict["Dataset"].append(dataset_label)
-
-            # Retrieve y_true and y_pred for the current dataset
-            y_true = np.ravel(dataset.y)  # Flatten y_true to make it one-dimensional
-            y_prob = dataset.y_prob(self.inputs.model)
-
-            # Compute metrics
-            y_true = np.array(y_true, dtype=float)
-            y_prob = np.array(y_prob, dtype=float)
-
-            fpr, tpr, _ = roc_curve(y_true, y_prob)
-            ks = max(tpr - fpr)
-            auc = roc_auc_score(y_true, y_prob)
-            gini = 2 * auc - 1
-
-            # Add the metrics to the dictionary
-            metrics_dict["AUC"].append(auc)
-            metrics_dict["GINI"].append(gini)
-            metrics_dict["KS"].append(ks)
-
-        # Create a DataFrame to store and return the results
-        metrics_df = pd.DataFrame(metrics_dict)
-        return metrics_df
-
-    def summary(self, metric_value):
-        summary_metrics_table = metric_value["metrics_summary"]
-        return ResultSummary(
-            results=[
-                ResultTable(
-                    data=summary_metrics_table,
-                    metadata=ResultTableMetadata(
-                        title="AUC, GINI and KS for train and test datasets"
-                    ),
-                )
-            ]
-        )
+    metrics_dict = {"AUC": [], "GINI": [], "KS": []}
+
+    # Retrieve y_true and y_pred for the current dataset
+    y_true = np.ravel(dataset.y)  # Flatten y_true to make it one-dimensional
+    y_prob = dataset.y_prob(model)
+
+    # Compute metrics
+    y_true = np.array(y_true, dtype=float)
+    y_prob = np.array(y_prob, dtype=float)
+
+    fpr, tpr, _ = roc_curve(y_true, y_prob)
+    ks = max(tpr - fpr)
+    auc = roc_auc_score(y_true, y_prob)
+    gini = 2 * auc - 1
+
+    # Add the metrics to the dictionary
+    metrics_dict["AUC"].append(auc)
+    metrics_dict["GINI"].append(gini)
+    metrics_dict["KS"].append(ks)
+
+    # Create a DataFrame to store and return the results
+    metrics_df = pd.DataFrame(metrics_dict)
+    return metrics_df

validmind/tests/model_validation/statsmodels/JarqueBera.py

@@ -11,36 +11,41 @@ class JarqueBera(Metric):
     """
     Assesses normality of dataset features in an ML model using the Jarque-Bera test.
 
-    **Purpose**: The purpose of the Jarque-Bera test as implemented in this metric is to determine if the features in
-    the dataset of a given Machine Learning model follows a normal distribution. This is crucial for understanding the
-    distribution and behavior of the model's features, as numerous statistical methods assume normal distribution of
-    the data.
-
-    **Test Mechanism**: The test mechanism involves computing the Jarque-Bera statistic, p-value, skew, and kurtosis
-    for each feature in the dataset. It utilizes the 'jarque_bera' function from the 'statsmodels' library in Python,
-    storing the results in a dictionary. The test evaluates the skewness and kurtosis to ascertain whether the dataset
-    follows a normal distribution. A significant p-value (typically less than 0.05) implies that the data does not
-    possess normal distribution.
-
-    **Signs of High Risk**:
-    - A high Jarque-Bera statistic and a low p-value (usually less than 0.05) indicates high-risk conditions.
+    ### Purpose
+
+    The purpose of the Jarque-Bera test as implemented in this metric is to determine if the features in the dataset of
+    a given Machine Learning model follow a normal distribution. This is crucial for understanding the distribution and
+    behavior of the model's features, as numerous statistical methods assume normal distribution of the data.
+
+    ### Test Mechanism
+
+    The test mechanism involves computing the Jarque-Bera statistic, p-value, skew, and kurtosis for each feature in
+    the dataset. It utilizes the 'jarque_bera' function from the 'statsmodels' library in Python, storing the results
+    in a dictionary. The test evaluates the skewness and kurtosis to ascertain whether the dataset follows a normal
+    distribution. A significant p-value (typically less than 0.05) implies that the data does not possess normal
+    distribution.
+
+    ### Signs of High Risk
+
+    - A high Jarque-Bera statistic and a low p-value (usually less than 0.05) indicate high-risk conditions.
     - Such results suggest the data significantly deviates from a normal distribution. If a machine learning model
     expects feature data to be normally distributed, these findings imply that it may not function as intended.
 
-    **Strengths**:
-    - This test provides insights into the shape of the data distribution, helping determine whether a given set of
-    data follows a normal distribution.
-    - This is particularly useful for risk assessment for models that assume a normal distribution of data.
+    ### Strengths
+
+    - Provides insights into the shape of the data distribution, helping determine whether a given set of data follows
+    a normal distribution.
+    - Particularly useful for risk assessment for models that assume a normal distribution of data.
     - By measuring skewness and kurtosis, it provides additional insights into the nature and magnitude of a
     distribution's deviation.
 
-    **Limitations**:
-    - The Jarque-Bera test only checks for normality in the data distribution. It cannot provide insights into other
-    types of distributions.
+    ### Limitations
+
+    - Only checks for normality in the data distribution. It cannot provide insights into other types of distributions.
     - Datasets that aren't normally distributed but follow some other distribution might lead to inaccurate risk
     assessments.
-    - The test is highly sensitive to large sample sizes, often rejecting the null hypothesis (that data is normally
-    distributed) even for minor deviations in larger datasets.
+    - Highly sensitive to large sample sizes, often rejecting the null hypothesis (that data is normally distributed)
+    even for minor deviations in larger datasets.
     """
 
     name = "jarque_bera"

validmind/tests/model_validation/statsmodels/KolmogorovSmirnov.py

@@ -13,40 +13,39 @@ from validmind.vm_models import Metric, ResultSummary, ResultTable, ResultTableM
 @dataclass
 class KolmogorovSmirnov(Metric):
     """
-    Executes a feature-wise Kolmogorov-Smirnov test to evaluate alignment with normal distribution in datasets.
-
-    **Purpose**: This metric employs the Kolmogorov-Smirnov (KS) test to evaluate the distribution of a dataset's
-    features. It specifically gauges whether the data from each feature aligns with a normal distribution, a common
-    presumption in many statistical methods and machine learning models.
-
-    **Test Mechanism**: This KS test calculates the KS statistic and the corresponding p-value for each column in a
-    dataset. It achieves this by contrasting the cumulative distribution function of the dataset's feature with an
-    ideal normal distribution. Subsequently, a feature-by-feature KS statistic and p-value are stored in a dictionary.
-    The specific threshold p-value (the value below which we reject the hypothesis that the data is drawn from a normal
-    distribution) is not firmly set within this implementation, allowing for definitional flexibility depending on the
-    specific application.
-
-    **Signs of High Risk**:
-    - Elevated KS statistic for a feature combined with a low p-value. This suggests a significant divergence between
-    the feature's distribution and a normal one.
-    - Features with notable deviations. These could create problems if the applicable model makes assumptions about
-    normal data distribution, thereby representing a risk.
-
-    **Strengths**:
-    - The KS test is acutely sensitive to differences in the location and shape of the empirical cumulative
-    distribution functions of two samples.
-    - It is non-parametric and does not presuppose any specific data distribution, making it adaptable to a range of
-    datasets.
-    - With its focus on individual features, it offers detailed insights into data distribution.
-
-    **Limitations**:
-    - The sensitivity of the KS test to disparities in data distribution tails can be excessive. Such sensitivity might
-    prompt false alarms about non-normal distributions, particularly in situations where these tail tendencies are
-    irrelevant to the model.
-    - It could become less effective when applied to multivariate distributions, considering that it's primarily
-    configured for univariate distributions.
-    - As a goodness-of-fit test, the KS test does not identify specific types of non-normality, such as skewness or
-    kurtosis, that could directly impact model fitting.
+    Assesses whether each feature in the dataset aligns with a normal distribution using the Kolmogorov-Smirnov test.
+
+    ### Purpose
+
+    The Kolmogorov-Smirnov (KS) test evaluates the distribution of features in a dataset to determine their alignment
+    with a normal distribution. This is important because many statistical methods and machine learning models assume
+    normality in the data distribution.
+
+    ### Test Mechanism
+
+    This test calculates the KS statistic and corresponding p-value for each feature in the dataset. It does so by
+    comparing the cumulative distribution function of the feature with an ideal normal distribution. The KS statistic
+    and p-value for each feature are then stored in a dictionary. The p-value threshold to reject the normal
+    distribution hypothesis is not preset, providing flexibility for different applications.
+
+    ### Signs of High Risk
+
+    - Elevated KS statistic for a feature combined with a low p-value, indicating a significant divergence from a
+    normal distribution.
+    - Features with notable deviations that could create problems if the model assumes normality in data distribution.
+
+    ### Strengths
+
+    - The KS test is sensitive to differences in the location and shape of empirical cumulative distribution functions.
+    - It is non-parametric and adaptable to various datasets, as it does not assume any specific data distribution.
+    - Provides detailed insights into the distribution of individual features.
+
+    ### Limitations
+
+    - The test's sensitivity to disparities in the tails of data distribution might cause false alarms about
+    non-normality.
+    - Less effective for multivariate distributions, as it is designed for univariate distributions.
+    - Does not identify specific types of non-normality, such as skewness or kurtosis, which could impact model fitting.
     """
 
     name = "kolmogorov_smirnov"

validmind/tests/model_validation/statsmodels/LJungBox.py

@@ -11,36 +11,40 @@ class LJungBox(Metric):
     """
     Assesses autocorrelations in dataset features by performing a Ljung-Box test on each feature.
 
-    **Purpose**: The Ljung-Box test is a type of statistical test utilized to ascertain whether there are
-    autocorrelations within a given dataset that differ significantly from zero. In the context of a machine learning
-    model, this test is primarily used to evaluate data utilized in regression tasks, especially those involving time
-    series and forecasting.
-
-    **Test Mechanism**: The test operates by iterating over each feature within the training dataset and applying the
-    `acorr_ljungbox` function from the `statsmodels.stats.diagnostic` library. This function calculates the Ljung-Box
-    statistic and p-value for each feature. These results are then stored in a dictionary where the keys are the
-    feature names and the values are dictionaries containing the statistic and p-value respectively. Generally, a lower
-    p-value indicates a higher likelihood of significant autocorrelations within the feature.
-
-    **Signs of High Risk**:
-    - A high risk or failure in the model's performance relating to this test might be indicated by high Ljung-Box
-    statistic values or low p-values.
-    - These outcomes suggest the presence of significant autocorrelations in the respective features. If not properly
-    consider or handle in the machine learning model, these can negatively affect model performance or bias.
-
-    **Strengths**:
-    - The Ljung-Box test is a powerful tool for detecting autocorrelations within datasets, especially in time series
-    data.
-    - It provides quantitative measures (statistic and p-value) that allow for precise evaluation of autocorrelation.
-    - This test can be instrumental in avoiding issues related to autoregressive residuals and other challenges in
-    regression models.
-
-    **Limitations**:
-    - The Ljung-Box test cannot detect all types of non-linearity or complex interrelationships among variables.
+    ### Purpose
+
+    The Ljung-Box test is a type of statistical test utilized to ascertain whether there are autocorrelations within a
+    given dataset that differ significantly from zero. In the context of a machine learning model, this test is
+    primarily used to evaluate data utilized in regression tasks, especially those involving time series and
+    forecasting.
+
+    ### Test Mechanism
+
+    The test operates by iterating over each feature within the training dataset and applying the `acorr_ljungbox`
+    function from the `statsmodels.stats.diagnostic` library. This function calculates the Ljung-Box statistic and
+    p-value for each feature. These results are then stored in a dictionary where the keys are the feature names and
+    the values are dictionaries containing the statistic and p-value respectively. Generally, a lower p-value indicates
+    a higher likelihood of significant autocorrelations within the feature.
+
+    ### Signs of High Risk
+
+    - High Ljung-Box statistic values or low p-values.
+    - Presence of significant autocorrelations in the respective features.
+    - Potential for negative impact on model performance or bias if autocorrelations are not properly handled.
+
+    ### Strengths
+
+    - Powerful tool for detecting autocorrelations within datasets, especially in time series data.
+    - Provides quantitative measures (statistic and p-value) for precise evaluation.
+    - Helps avoid issues related to autoregressive residuals and other challenges in regression models.
+
+    ### Limitations
+
+    - Cannot detect all types of non-linearity or complex interrelationships among variables.
     - Testing individual features may not fully encapsulate the dynamics of the data if features interact with each
     other.
-    - It is designed more for traditional statistical models and may not be fully compatible with certain types of
-    complex machine learning models.
+    - Designed more for traditional statistical models and may not be fully compatible with certain types of complex
+    machine learning models.
     """
 
     name = "ljung_box"

validmind/tests/model_validation/statsmodels/Lilliefors.py

@@ -14,44 +14,47 @@ class Lilliefors(Metric):
     """
     Assesses the normality of feature distributions in an ML model's training dataset using the Lilliefors test.
 
-    **Purpose**: The purpose of this metric is to utilize the Lilliefors test, named in honor of the Swedish
-    statistician Hubert Lilliefors, in order to assess whether the features of the machine learning model's training
-    dataset conform to a normal distribution. This is done because the assumption of normal distribution plays a vital
-    role in numerous statistical procedures as well as numerous machine learning models. Should the features fail to
-    follow a normal distribution, some model types may not operate at optimal efficiency. This can potentially lead to
-    inaccurate predictions.
-
-    **Test Mechanism**: The application of this test happens across all feature columns within the training dataset.
-    For each feature, the Lilliefors test returns a test statistic and p-value. The test statistic quantifies how far
-    the feature's distribution is from an ideal normal distribution, whereas the p-value aids in determining the
-    statistical relevance of this deviation. The final results are stored within a dictionary, the keys of which
-    correspond to the name of the feature column, and the values being another dictionary which houses the test
-    statistic and p-value.
-
-    **Signs of High Risk**:
+    ### Purpose
+
+    The purpose of this metric is to utilize the Lilliefors test, named in honor of the Swedish statistician Hubert
+    Lilliefors, in order to assess whether the features of the machine learning model's training dataset conform to a
+    normal distribution. This is done because the assumption of normal distribution plays a vital role in numerous
+    statistical procedures as well as numerous machine learning models. Should the features fail to follow a normal
+    distribution, some model types may not operate at optimal efficiency. This can potentially lead to inaccurate
+    predictions.
+
+    ### Test Mechanism
+
+    The application of this test happens across all feature columns within the training dataset. For each feature, the
+    Lilliefors test returns a test statistic and p-value. The test statistic quantifies how far the feature's
+    distribution is from an ideal normal distribution, whereas the p-value aids in determining the statistical
+    relevance of this deviation. The final results are stored within a dictionary, the keys of which correspond to the
+    name of the feature column, and the values being another dictionary which houses the test statistic and p-value.
+
+    ### Signs of High Risk
 
     - If the p-value corresponding to a specific feature sinks below a pre-established significance level, generally
     set at 0.05, then it can be deduced that the distribution of that feature significantly deviates from a normal
     distribution. This can present a high risk for models that assume normality, as these models may perform
     inaccurately or inefficiently in the presence of such a feature.
 
-    **Strengths**:
+    ### Strengths
 
     - One advantage of the Lilliefors test is its utility irrespective of whether the mean and variance of the normal
     distribution are known in advance. This makes it a more robust option in real-world situations where these values
     might not be known.
-    - Second, the test has the ability to screen every feature column, offering a holistic view of the dataset.
+    - The test has the ability to screen every feature column, offering a holistic view of the dataset.
 
-    **Limitations**:
+    ### Limitations
 
     - Despite the practical applications of the Lilliefors test in validating normality, it does come with some
     limitations.
-    - Firstly, it is only capable of testing unidimensional data, thus rendering it ineffective for datasets with
-    interactions between features or multi-dimensional phenomena.
-    - Additionally, the test might not be as sensitive as some other tests (like the Anderson-Darling test) in
-    detecting deviations from a normal distribution.
-    - Lastly, like any other statistical test, Lilliefors test may also produce false positives or negatives. Hence,
-    banking solely on this test, without considering other characteristics of the data, may give rise to risks.
+    - It is only capable of testing unidimensional data, thus rendering it ineffective for datasets with interactions
+    between features or multi-dimensional phenomena.
+    - The test might not be as sensitive as some other tests (like the Anderson-Darling test) in detecting deviations
+    from a normal distribution.
+    - Like any other statistical test, Lilliefors test may also produce false positives or negatives. Hence, banking
+    solely on this test, without considering other characteristics of the data, may give rise to risks.
     """
 
     name = "lilliefors_test"