validmind 2.5.8__py3-none-any.whl → 2.5.18__py3-none-any.whl

validmind/tests/model_validation/sklearn/HomogeneityScore.py

@@ -15,29 +15,36 @@ class HomogeneityScore(ClusterPerformance):
     Assesses clustering homogeneity by comparing true and predicted labels, scoring from 0 (heterogeneous) to 1
     (homogeneous).
 
-    **Purpose**: The Homogeneity Score encapsulated in this performance test is used to measure the homogeneity of the
-    clusters formed by a machine learning model. In simple terms, a clustering result satisfies homogeneity if all of
-    its clusters contain only points which are members of a single class.
+    ### Purpose
 
-    **Test Mechanism**: This test uses the `homogeneity_score` function from the `sklearn.metrics` library to compare
-    the ground truth class labels of the training and testing sets with the labels predicted by the given model. The
-    returned score is a metric of the clustering accuracy, and ranges from 0.0 to 1.0, with 1.0 denoting the highest
-    possible degree of homogeneity.
+    The Homogeneity Score encapsulated in this performance test is used to measure the homogeneity of the clusters
+    formed by a machine learning model. In simple terms, a clustering result satisfies homogeneity if all of its
+    clusters contain only points which are members of a single class.
+
+    ### Test Mechanism
+
+    This test uses the `homogeneity_score` function from the `sklearn.metrics` library to compare the ground truth
+    class labels of the training and testing sets with the labels predicted by the given model. The returned score is a
+    metric of the clustering accuracy, and ranges from 0.0 to 1.0, with 1.0 denoting the highest possible degree of
+    homogeneity.
+
+    ### Signs of High Risk
 
-    **Signs of High Risk**:
     - A score close to 0: This denotes that clusters are highly heterogenous and points within the same cluster might
     not belong to the same class.
     - A significantly lower score for testing data compared to the score for training data: This can indicate
     overfitting, where the model has learned to perfectly match the training data but fails to perform well on unseen
     data.
 
-    **Strengths**:
+    ### Strengths
+
     - It provides a simple quantitative measure of the degree to which clusters contain points from only one class.
-    - Useful for validating clustering solutions where the ground truth - class membership of points - is known.
+    - Useful for validating clustering solutions where the ground truth — class membership of points — is known.
     - It's agnostic to the absolute labels, and cares only that the points within the same cluster have the same class
     label.
 
-    **Limitations**:
+    ### Limitations
+
     - The Homogeneity Score is not useful for clustering solutions where the ground truth labels are not known.
     - It doesn’t work well with differently sized clusters since it gives predominance to larger clusters.
     - The score does not address the actual number of clusters formed, or the evenness of cluster sizes. It only checks
@@ -45,7 +52,7 @@ class HomogeneityScore(ClusterPerformance):
     """
 
     name = "homogeneity_score"
-    required_inputs = ["model", "datasets"]
+    required_inputs = ["model", "dataset"]
     tasks = ["clustering"]
     tags = [
         "sklearn",

validmind/tests/model_validation/sklearn/HyperParametersTuning.py

@@ -16,37 +16,42 @@ class HyperParametersTuning(Metric):
     """
     Exerts exhaustive grid search to identify optimal hyperparameters for the model, improving performance.
 
-    **Purpose:** The "HyperParametersTuning" metric being used here is intended to find the optimal set of
-    hyperparameters for a given model. The test essentially aims to enhance the performance of the model under scrutiny
-    by determining the best configuration of hyperparameters. The parameters that are being optimized are defined by
-    the parameter grid that is passed to the metric.
-
-    **Test Mechanism:** The HyperParametersTuning test employs a grid search mechanism using the function GridSearchCV
-    from the scikit-learn library. The grid search algorithm is exhaustive: it systematically works through multiple
-    combinations of the parameter tunes, cross-validated to determine which tune gives the best model performance. The
-    chosen model and the parameters grid that are to be passed for tuning are the required inputs. Once the grid search
-    is complete, the test caches and returns the details of the best model and its associated parameters.
-
-    **Signs of High Risk:**
-    - The test raises a SkipTestError if the param_grid is not supplied. This suggests that there are no specific
-    parameters to optimize, which is a risk in certain model types that rely heavily on parameter tuning.
-    - Poorly chosen scoring metrics that don't align well with the specific model or problem at hand might also reflect
-    as a potential risk or failure in achieving the best performance.
-
-    **Strengths:**
-    - The test is a comprehensive exploratory mechanism that figures out the best set of hyperparameters for the
-    supplied model, thereby helping improve its performance.
-    - The implementation of GridSearchCV simplifies and automates the time-consuming task of hyperparameter tuning.
-
-    **Limitations:**
-    - The grid search algorithm can be computationally expensive, particularly with a large dataset or complex models.
-    This grid search approach can be time-consuming as it tries out all possible combinations within the specified
-    parameter grid.
-    - The suitability of the tuning heavily relies on the quality of the data and it only accepts datasets with
+    ### Purpose:
+
+    The "HyperParametersTuning" metric aims to find the optimal set of hyperparameters for a given model. The test is
+    designed to enhance the performance of the model by determining the best configuration of hyperparameters. The
+    parameters that are being optimized are defined by the parameter grid provided to the metric.
+
+    ### Test Mechanism:
+
+    The HyperParametersTuning test employs a grid search mechanism using the GridSearchCV function from the
+    scikit-learn library. The grid search algorithm systematically works through multiple combinations of parameter
+    values, cross-validating to determine which combination gives the best model performance. The chosen model and the
+    parameter grid passed for tuning are necessary inputs. Once the grid search is complete, the test caches and
+    returns details of the best model and its associated parameters.
+
+    ### Signs of High Risk:
+
+    - The test raises a SkipTestError if the param_grid is not supplied, indicating a lack of specific parameters to
+    optimize, which can be risky for certain model types reliant on parameter tuning.
+    - Poorly chosen scoring metrics that do not align well with the specific model or problem at hand could reflect
+    potential risks or failures in achieving optimal performance.
+
+    ### Strengths:
+
+    - Provides a comprehensive exploration mechanism to identify the best set of hyperparameters for the supplied
+    model, thereby enhancing its performance.
+    - Implements GridSearchCV, simplifying and automating the time-consuming task of hyperparameter tuning.
+
+    ### Limitations:
+
+    - The grid search algorithm can be computationally expensive, especially with large datasets or complex models, and
+    can be time-consuming as it tests all possible combinations within the specified parameter grid.
+    - The effectiveness of the tuning is heavily dependent on the quality of data and only accepts datasets with
     numerical or ordered categories.
-    - The functionality assumes that the same set of hyperparameters is optimal for all problem sets, which may not
-    hold true in every scenario.
-    - There is a potential risk of overfitting the model if the training set is not representative of the data the
+    - Assumes that the same set of hyperparameters is optimal for all problem sets, which may not be true in every
+    scenario.
+    - There's a potential risk of overfitting the model if the training set is not representative of the data that the
     model will be applied to.
     """
 

validmind/tests/model_validation/sklearn/KMeansClustersOptimization.py

@@ -19,13 +19,15 @@ class KMeansClustersOptimization(Metric):
     """
     Optimizes the number of clusters in K-means models using Elbow and Silhouette methods.
 
-    **Purpose:**
+    ### Purpose
+
     This metric is used to optimize the number of clusters used in K-means clustering models. It intends to measure and
     evaluate the optimal number of clusters by leveraging two methodologies, namely the Elbow method and the Silhouette
     method. This is crucial as an inappropriate number of clusters can either overly simplify or overcomplicate the
     structure of the data, thereby undermining the effectiveness of the model.
 
-    **Test Mechanism:**
+    ### Test Mechanism
+
     The test mechanism involves iterating over a predefined range of cluster numbers and applying both the Elbow method
     and the Silhouette method. The Elbow method computes the sum of the minimum euclidean distances between data points
     and their respective cluster centers (distortion). This value decreases as the number of clusters increases; the
@@ -35,20 +37,23 @@ class KMeansClustersOptimization(Metric):
     of clusters under this method is the one that maximizes the average silhouette score. The results of both methods
     are plotted for visual inspection.
 
-    **Signs of High Risk:**
+    ### Signs of High Risk
+
     - A high distortion value or a low silhouette average score for the optimal number of clusters.
     - No clear 'elbow' point or plateau observed in the distortion plot, or a uniformly low silhouette average score
     across different numbers of clusters, suggesting the data is not amenable to clustering.
     - An optimal cluster number that is unreasonably high or low, suggestive of overfitting or underfitting,
     respectively.
 
-    **Strengths:**
+    ### Strengths
+
     - Provides both a visual and quantitative method to determine the optimal number of clusters.
     - Leverages two different methods (Elbow and Silhouette), thereby affording robustness and versatility in assessing
     the data's clusterability.
     - Facilitates improved model performance by allowing for an informed selection of the number of clusters.
 
-    **Limitations:**
+    ### Limitations
+
     - Assumes that a suitable number of clusters exists in the data, which may not always be true, especially for
     complex or noisy data.
     - Both methods may fail to provide definitive answers when the data lacks clear cluster structures.

validmind/tests/model_validation/sklearn/MinimumAccuracy.py

@@ -22,38 +22,38 @@ class MinimumAccuracy(ThresholdTest):
     """
     Checks if the model's prediction accuracy meets or surpasses a specified threshold.
 
-    **Purpose**: The Minimum Accuracy test’s objective is to verify whether the model's prediction accuracy on a
-    specific dataset meets or surpasses a predetermined minimum threshold. Accuracy, which is simply the ratio of right
-    predictions to total predictions, is a key metric for evaluating the model's performance. Considering binary as
-    well as multiclass classifications, accurate labeling becomes indispensable.
-
-    **Test Mechanism**: The test mechanism involves contrasting the model's accuracy score with a pre-set minimum
-    threshold value, default value being 0.7. The accuracy score is computed utilizing sklearn’s `accuracy_score`
-    method, where the true label `y_true` and predicted label `class_pred` are compared. If the accuracy score is above
-    the threshold, the test gets a passing mark. The test returns the result along with the accuracy score and
-    threshold used for the test.
-
-    **Signs of High Risk**:
-    - The risk level for this test surges considerably when the model is unable to achieve or surpass the predefined
-    score threshold.
-    - When the model persistently scores below the threshold, it suggests a high risk of inaccurate predictions, which
-    in turn affects the model’s efficiency and reliability.
-
-    **Strengths**:
-    - One of the key strengths of this test is its simplicity, presenting a straightforward measure of the holistic
-    model performance across all classes.
-    - This test is particularly advantageous when classes are balanced.
-    - Another advantage of this test is its versatility as it can be implemented on both binary and multiclass
-    classification tasks.
-
-    **Limitations**:
-    - When analyzing imbalanced datasets, certain limitations of this test emerge. The accuracy score can be misleading
-    when classes in the dataset are skewed considerably.
-    - This can result in favoritism towards the majority class, consequently giving an inaccurate perception of the
-    model performance.
-    - Another limitation is its inability to measure the model's precision, recall, or capacity to manage false
-    positives or false negatives.
-    - The test majorly focuses on overall correctness and may not be sufficient for all types of model analytics.
+    ### Purpose
+
+    The Minimum Accuracy test’s objective is to verify whether the model's prediction accuracy on a specific dataset
+    meets or surpasses a predetermined minimum threshold. Accuracy, which is simply the ratio of correct predictions to
+    total predictions, is a key metric for evaluating the model's performance. Considering binary as well as multiclass
+    classifications, accurate labeling becomes indispensable.
+
+    ### Test Mechanism
+
+    The test mechanism involves contrasting the model's accuracy score with a preset minimum threshold value, with the
+    default being 0.7. The accuracy score is computed utilizing sklearn’s `accuracy_score` method, where the true
+    labels `y_true` and predicted labels `class_pred` are compared. If the accuracy score is above the threshold, the
+    test receives a passing mark. The test returns the result along with the accuracy score and threshold used for the
+    test.
+
+    ### Signs of High Risk
+
+    - Model fails to achieve or surpass the predefined score threshold.
+    - Persistent scores below the threshold, indicating a high risk of inaccurate predictions.
+
+    ### Strengths
+
+    - Simplicity, presenting a straightforward measure of holistic model performance across all classes.
+    - Particularly advantageous when classes are balanced.
+    - Versatile, as it can be implemented on both binary and multiclass classification tasks.
+
+    ### Limitations
+
+    - Misleading accuracy scores when classes in the dataset are highly imbalanced.
+    - Favoritism towards the majority class, giving an inaccurate perception of model performance.
+    - Inability to measure the model's precision, recall, or capacity to manage false positives or false negatives.
+    - Focused on overall correctness and may not be sufficient for all types of model analytics.
     """
 
     name = "accuracy_score"

validmind/tests/model_validation/sklearn/MinimumF1Score.py

@@ -21,42 +21,42 @@ from validmind.vm_models import (
 @dataclass
 class MinimumF1Score(ThresholdTest):
     """
-    Evaluates if the model's F1 score on the validation set meets a predefined minimum threshold.
+    Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced
+    performance between precision and recall.
+
+    ### Purpose
 
-    **Purpose:**
     The main objective of this test is to ensure that the F1 score, a balanced measure of precision and recall, of the
     model meets or surpasses a predefined threshold on the validation dataset. The F1 score is highly useful for
     gauging model performance in classification tasks, especially in cases where the distribution of positive and
     negative classes is skewed.
 
-    **Test Mechanism:**
-    The F1 score for the validation dataset is computed through the scikit-learn's metrics in Python. The scoring
-    mechanism differs based on the classification problem: for multi-class problems, macro averaging is used (metrics
-    are calculated separately and their unweighted mean is found), and for binary classification, the built-in f1_score
-    calculation is used. The obtained F1 score is then assessed against the predefined minimum F1 score that is
-    expected from the model.
+    ### Test Mechanism
+
+    The F1 score for the validation dataset is computed through scikit-learn's metrics in Python. The scoring mechanism
+    differs based on the classification problem: for multi-class problems, macro averaging is used, and for binary
+    classification, the built-in `f1_score` calculation is used. The obtained F1 score is then assessed against the
+    predefined minimum F1 score that is expected from the model.
 
-    **Signs of High Risk:**
+    ### Signs of High Risk
 
     - If a model returns an F1 score that is less than the established threshold, it is regarded as high risk.
-    - A low F1 score might suggest that the model is not finding an optimal balance between precision and recall, see:
-    it isn't successfully identifying positive classes while minimizing false positives.
+    - A low F1 score might suggest that the model is not finding an optimal balance between precision and recall,
+    failing to effectively identify positive classes while minimizing false positives.
 
-    **Strengths:**
+    ### Strengths
 
-    - This metric gives a balanced measure of a model's performance by accounting for both false positives and false
-    negatives.
-    - It has a particular advantage in scenarios with imbalanced class distribution, where an accuracy measure can be
-    misleading.
-    - The flexibility of setting the threshold value allows for tailoring the minimum acceptable performance.
+    - Provides a balanced measure of a model's performance by accounting for both false positives and false negatives.
+    - Particularly advantageous in scenarios with imbalanced class distribution, where accuracy can be misleading.
+    - Flexibility in setting the threshold value allows tailored minimum acceptable performance standards.
 
-    **Limitations:**
+    ### Limitations
 
-    - The testing method may not be suitable for all types of models and machine learning tasks.
-    - Although the F1 score gives a balanced view of a model's performance, it presupposes an equal cost for false
-    positives and false negatives, which may not always be true in certain real-world scenarios. As a consequence,
-    practitioners might have to rely on other metrics such as precision, recall, or the ROC-AUC score that align more
-    closely with their specific requirements.
+    - May not be suitable for all types of models and machine learning tasks.
+    - The F1 score assumes an equal cost for false positives and false negatives, which may not be true in some
+    real-world scenarios.
+    - Practitioners might need to rely on other metrics such as precision, recall, or the ROC-AUC score that align more
+    closely with specific requirements.
     """
 
     name = "f1_score"

validmind/tests/model_validation/sklearn/MinimumROCAUCScore.py

@@ -23,32 +23,37 @@ class MinimumROCAUCScore(ThresholdTest):
     """
     Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.
 
-    **Purpose**:
-    This test metric, Minimum ROC AUC Score, is used to determine the model's performance by ensuring that the Receiver
-    Operating Characteristic Area Under the Curve (ROC AUC) score on the validation dataset meets or exceeds a
-    predefined threshold. The ROC AUC score is an indicator of how well the model is capable of distinguishing between
-    different classes, making it a crucial measure in binary and multiclass classification tasks.
+    ### Purpose
+
+    The Minimum ROC AUC Score test is used to determine the model's performance by ensuring that the Receiver Operating
+    Characteristic Area Under the Curve (ROC AUC) score on the validation dataset meets or exceeds a predefined
+    threshold. The ROC AUC score indicates how well the model can distinguish between different classes, making it a
+    crucial measure in binary and multiclass classification tasks.
+
+    ### Test Mechanism
 
-    **Test Mechanism**:
     This test implementation calculates the multiclass ROC AUC score on the true target values and the model's
-    prediction. The test converts the multi-class target variables into binary format using `LabelBinarizer` before
+    predictions. The test converts the multi-class target variables into binary format using `LabelBinarizer` before
     computing the score. If this ROC AUC score is higher than the predefined threshold (defaulted to 0.5), the test
     passes; otherwise, it fails. The results, including the ROC AUC score, the threshold, and whether the test passed
     or failed, are then stored in a `ThresholdTestResult` object.
 
-    **Signs of High Risk**:
+    ### Signs of High Risk
+
     - A high risk or failure in the model's performance as related to this metric would be represented by a low ROC AUC
     score, specifically any score lower than the predefined minimum threshold. This suggests that the model is
     struggling to distinguish between different classes effectively.
 
-    **Strengths**:
+    ### Strengths
+
     - The test considers both the true positive rate and false positive rate, providing a comprehensive performance
     measure.
     - ROC AUC score is threshold-independent meaning it measures the model's quality across various classification
     thresholds.
     - Works robustly with binary as well as multi-class classification problems.
 
-    **Limitations**:
+    ### Limitations
+
     - ROC AUC may not be useful if the class distribution is highly imbalanced; it could perform well in terms of AUC
     but still fail to predict the minority class.
     - The test does not provide insight into what specific aspects of the model are causing poor performance if the ROC

validmind/tests/model_validation/sklearn/ModelsPerformanceComparison.py

@@ -19,33 +19,40 @@ class ModelsPerformanceComparison(ClassifierPerformance):
     Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy,
     precision, recall, and F1 score.
 
-    **Purpose**: This metric test aims to evaluate and compare the performance of various Machine Learning models using
-    test data. It employs multiple metrics such as accuracy, precision, recall, and the F1 score, among others, to
-    assess model performance and assist in selecting the most effective model for the designated task.
-
-    **Test Mechanism**: The test employs Scikit-learn’s performance metrics to evaluate each model's performance for
-    both binary and multiclass classification tasks. To compare performances, the test runs each model against the test
-    dataset, then produces a comprehensive classification report. This report includes metrics such as accuracy,
-    precision, recall, and the F1 score. Based on whether the task at hand is binary or multiclass classification, it
-    calculates metrics all the classes and their weighted averages, macro averages,
-    and per class metrics. The test will be skipped if no models are supplied.
-
-    **Signs of High Risk**:
+    ### Purpose
+
+    The Models Performance Comparison test aims to evaluate and compare the performance of various Machine Learning
+    models using test data. It employs multiple metrics such as accuracy, precision, recall, and the F1 score, among
+    others, to assess model performance and assist in selecting the most effective model for the designated task.
+
+    ### Test Mechanism
+
+    The test employs Scikit-learn’s performance metrics to evaluate each model's performance for both binary and
+    multiclass classification tasks. To compare performances, the test runs each model against the test dataset, then
+    produces a comprehensive classification report. This report includes metrics such as accuracy, precision, recall,
+    and the F1 score. Based on whether the task at hand is binary or multiclass classification, it calculates metrics
+    for all the classes and their weighted averages, macro averages, and per-class metrics. The test will be skipped if
+    no models are supplied.
+
+    ### Signs of High Risk
+
     - Low scores in accuracy, precision, recall, and F1 metrics indicate a potentially high risk.
     - A low area under the Receiver Operating Characteristic (ROC) curve (roc_auc score) is another possible indicator
     of high risk.
     - If the metrics scores are significantly lower than alternative models, this might suggest a high risk of failure.
 
-    **Strengths**:
-    - The test provides a simple way to compare the performance of multiple models, accommodating both binary and
-    multiclass classification tasks.
-    - It provides a holistic view of model performance through a comprehensive report of key performance metrics.
+    ### Strengths
+
+    - Provides a simple way to compare the performance of multiple models, accommodating both binary and multiclass
+    classification tasks.
+    - Offers a holistic view of model performance through a comprehensive report of key performance metrics.
     - The inclusion of the ROC AUC score is advantageous, as this robust performance metric can effectively handle
     class imbalance issues.
 
-    **Limitations**:
-    - This test may not be suitable for more complex performance evaluations that consider factors such as prediction
-    speed, computational cost, or business-specific constraints.
+    ### Limitations
+
+    - May not be suitable for more complex performance evaluations that consider factors such as prediction speed,
+    computational cost, or business-specific constraints.
     - The test's reliability depends on the provided test dataset; hence, the selected models' performance could vary
     with unseen data or changes in the data distribution.
     - The ROC AUC score might not be as meaningful or easily interpretable for multilabel/multiclass tasks.

validmind/tests/model_validation/sklearn/OverfitDiagnosis.py

@@ -187,6 +187,8 @@ def overfit_diagnosis(  # noqa: C901
     feature columns. It calculates the difference between the training and test performance
     for each group and identifies regions where the difference exceeds a specified threshold.
 
+    ## Test Methodology
+
     This test works for both classification and regression models and with a variety of
     performance metrics. By default, it uses the AUC metric for classification models and
     the MSE metric for regression models. The threshold for identifying overfit regions
@@ -308,28 +310,46 @@ def overfit_diagnosis(  # noqa: C901
 
 @dataclass
 class OverfitDiagnosis(ThresholdTest):
-    """Identify overfit regions in a model's predictions.
+    """
+    Assesses potential overfitting in a model's predictions, identifying regions where performance between training and
+    testing sets deviates significantly.
 
-    This test compares the model's performance on training versus test data, grouped by
-    feature columns. It calculates the difference between the training and test performance
-    for each group and identifies regions where the difference exceeds a specified threshold.
+    ### Purpose
 
-    This test works for both classification and regression models and with a variety of
-    performance metrics. By default, it uses the AUC metric for classification models and
-    the MSE metric for regression models. The threshold for identifying overfit regions
-    defaults to 0.04 but should be adjusted based on the specific use case.
+    The Overfit Diagnosis test aims to identify areas in a model's predictions where there is a significant difference
+    in performance between the training and testing sets. This test helps to pinpoint specific regions or feature
+    segments where the model may be overfitting.
 
-    ## Inputs
-    - `model` (VMModel): The ValidMind model object to evaluate.
-    - `datasets` (List[VMDataset]): A list of two VMDataset objects where the first dataset
-        is the training data and the second dataset is the test data.
+    ### Test Mechanism
 
-    ## Parameters
-    - `metric` (str, optional): The performance metric to use for evaluation. Choose from:
-        'accuracy', 'auc', 'f1', 'precision', 'recall', 'mse', 'mae', 'r2', 'mape'.
-        Defaults to 'auc' for classification models and 'mse' for regression models.
-    - `cut_off_threshold` (float, optional): The threshold for identifying overfit regions.
-        Defaults to 0.04.
+    This test compares the model's performance on training versus test data, grouped by feature columns. It calculates
+    the difference between the training and test performance for each group and identifies regions where this
+    difference exceeds a specified threshold:
+
+    - The test works for both classification and regression models.
+    - It defaults to using the AUC metric for classification models and the MSE metric for regression models.
+    - The threshold for identifying overfitting regions is set to 0.04 by default.
+    - The test calculates the performance metrics for each feature segment and plots regions where the performance gap
+    exceeds the threshold.
+
+    ### Signs of High Risk
+
+    - Significant gaps between training and test performance metrics for specific feature segments.
+    - Multiple regions with performance gaps exceeding the defined threshold.
+    - Higher than expected differences in predicted versus actual values in the test set compared to the training set.
+
+    ### Strengths
+
+    - Identifies specific areas where overfitting occurs.
+    - Supports multiple performance metrics, providing flexibility.
+    - Applicable to both classification and regression models.
+    - Visualization of overfitting segments aids in better understanding and debugging.
+
+    ### Limitations
+
+    - The default threshold may not be suitable for all use cases and requires tuning.
+    - May not capture more subtle forms of overfitting that do not exceed the threshold.
+    - Assumes that the binning of features adequately represents the data segments.
     """
 
     required_inputs = ["model", "datasets"]

validmind/tests/model_validation/sklearn/PermutationFeatureImportance.py

@@ -20,34 +20,40 @@ class PermutationFeatureImportance(Metric):
     Assesses the significance of each feature in a model by evaluating the impact on model performance when feature
     values are randomly rearranged.
 
-    **Purpose**: The purpose of the Permutation Feature Importance (PFI) metric is to assess the importance of each
-    feature used by the Machine Learning model. The significance is measured by evaluating the decrease in the model's
-    performance when the feature's values are randomly arranged.
+    ### Purpose
 
-    **Test Mechanism**: PFI is calculated via the `permutation_importance` method from the `sklearn.inspection` module.
-    This method shuffles the columns of the feature dataset and measures the impact on the model's performance. A
-    significant decrease in performance after permutating a feature's values deems the feature as important. On the
-    other hand, if performance remains the same, the feature is likely not important. The output of the PFI metric is a
-    figure illustrating the importance of each feature.
+    The Permutation Feature Importance (PFI) metric aims to assess the importance of each feature used by the Machine
+    Learning model. The significance is measured by evaluating the decrease in the model's performance when the
+    feature's values are randomly arranged.
+
+    ### Test Mechanism
+
+    PFI is calculated via the `permutation_importance` method from the `sklearn.inspection` module. This method
+    shuffles the columns of the feature dataset and measures the impact on the model's performance. A significant
+    decrease in performance after permutating a feature's values deems the feature as important. On the other hand, if
+    performance remains the same, the feature is likely not important. The output of the PFI metric is a figure
+    illustrating the importance of each feature.
+
+    ### Signs of High Risk
 
-    **Signs of High Risk**:
     - The model heavily relies on a feature with highly variable or easily permutable values, indicating instability.
-    - A feature, deemed unimportant by the model but based on domain knowledge should have a significant effect on the
-    outcome, is not influencing the model's predictions.
-
-    **Strengths**:
-    - PFI provides insights into the importance of different features and may reveal underlying data structure.
-    - It can indicate overfitting if a particular feature or set of features overly impacts the model's predictions.
-    - The metric is model-agnostic and can be used with any classifier that provides a measure of prediction accuracy
-    before and after feature permutation.
-
-    **Limitations**:
-    - The feature importance calculated does not imply causality, it only presents the amount of information that a
-    feature provides for the prediction task.
-    - The metric does not account for interactions between features. If features are correlated, the permutation
-    importance may allocate importance to one and not the other.
-    - PFI cannot interact with certain libraries like statsmodels, pytorch, catboost, etc, thus limiting its
-    applicability.
+    - A feature deemed unimportant by the model but expected to have a significant effect on the outcome based on
+    domain knowledge is not influencing the model's predictions.
+
+    ### Strengths
+
+    - Provides insights into the importance of different features and may reveal underlying data structure.
+    - Can indicate overfitting if a particular feature or set of features overly impacts the model's predictions.
+    - Model-agnostic and can be used with any classifier that provides a measure of prediction accuracy before and
+    after feature permutation.
+
+    ### Limitations
+
+    - Does not imply causality; it only presents the amount of information that a feature provides for the prediction
+    task.
+    - Does not account for interactions between features. If features are correlated, the permutation importance may
+    allocate importance to one and not the other.
+    - Cannot interact with certain libraries like statsmodels, pytorch, catboost, etc., thus limiting its applicability.
     """
 
     name = "pfi"
@@ -103,7 +109,7 @@ class PermutationFeatureImportance(Metric):
             )
         )
         fig.update_layout(
-            title_text="Permutation Importances (train set)",
+            title_text="Permutation Importances",
             yaxis=dict(
                 tickmode="linear",  # set tick mode to linear
                 dtick=1,  # set interval between ticks