validmind 2.5.8__py3-none-any.whl → 2.5.15__py3-none-any.whl

validmind/tests/model_validation/statsmodels/RegressionFeatureSignificance.py

@@ -19,31 +19,36 @@ class RegressionFeatureSignificance(Metric):
     """
     Assesses and visualizes the statistical significance of features in a set of regression models.
 
-    **Purpose**:
+    ### Purpose
+
     The Regression Feature Significance metric assesses the significance of each feature in a given set of regression
     models. It creates a visualization displaying p-values for every feature of each model, assisting model developers
     in understanding which features are most influential in their models.
 
-    **Test Mechanism**:
+    ### Test Mechanism
+
     The test mechanism involves going through each fitted regression model in a given list, extracting the model
     coefficients and p-values for each feature, and then plotting these values. The x-axis on the plot contains the
     p-values while the y-axis denotes the coefficients of each feature. A vertical red line is drawn at the threshold
     for p-value significance, which is 0.05 by default. Any features with p-values to the left of this line are
     considered statistically significant at the chosen level.
 
-    **Signs of High Risk**:
+    ### Signs of High Risk
+
     - Any feature with a high p-value (greater than the threshold) is considered a potential high risk, as it suggests
     the feature is not statistically significant and may not be reliably contributing to the model's predictions.
     - A high number of such features may indicate problems with the model validation, variable selection, and overall
     reliability of the model predictions.
 
-    **Strengths**:
+    ### Strengths
+
     - Helps identify the features that significantly contribute to a model's prediction, providing insights into the
     feature importance.
     - Provides tangible, easy-to-understand visualizations to interpret the feature significance.
     - Facilitates comparison of feature importance across multiple models.
 
-    **Limitations**:
+    ### Limitations
+
     - This metric assumes model features are independent, which may not always be the case. Multicollinearity (high
     correlation amongst predictors) can cause high variance and unreliable statistical tests of significance.
     - The p-value strategy for feature selection doesn't take into account the magnitude of the effect, focusing solely
@@ -54,7 +59,7 @@ class RegressionFeatureSignificance(Metric):
     """
 
     name = "regression_feature_significance"
-    required_inputs = ["models"]
+    required_inputs = ["model"]
 
     default_params = {"fontsize": 10, "p_threshold": 0.05}
     tasks = ["regression"]
@@ -70,10 +75,10 @@ class RegressionFeatureSignificance(Metric):
         p_threshold = self.params["p_threshold"]
 
         # Check models list is not empty
-        if not self.inputs.models:
-            raise ValueError("List of models must be provided in the models parameter")
+        if not self.inputs.model:
+            raise ValueError("Model must be provided in the models parameter")
 
-        figures = self._plot_pvalues(self.inputs.models, fontsize, p_threshold)
+        figures = self._plot_pvalues(self.inputs.model, fontsize, p_threshold)
 
         return self.cache_results(figures=figures)
 

validmind/tests/model_validation/statsmodels/RegressionModelForecastPlot.py

@@ -19,26 +19,30 @@ class RegressionModelForecastPlot(Metric):
     Generates plots to visually compare the forecasted outcomes of one or more regression models against actual
     observed values over a specified date range.
 
-    **Purpose:** The "regression_forecast_plot" is intended to visually depict the performance of one or more
-    regression models by comparing the model's forecasted outcomes against actual observed values within a specified
-    date range. This metric is especially useful in time-series models or any model where the outcome changes over
-    time, allowing direct comparison of predicted vs actual values.
+    ### Purpose
 
-    **Test Mechanism:** This test generates a plot for each fitted model in the list. The x-axis represents the date
-    ranging from the specified "start_date" to the "end_date", while the y-axis shows the value of the outcome
-    variable. Two lines are plotted: one representing the forecasted values and the other representing the observed
-    values. The "start_date" and "end_date" can be parameters of this test; if these parameters are not provided, they
-    are set to the minimum and maximum date available in the dataset. The test verifies that the provided date range is
-    within the limits of the available data.
+    The "regression_forecast_plot" is intended to visually depict the performance of one or more regression models by
+    comparing the model's forecasted outcomes against actual observed values within a specified date range. This metric
+    is especially useful in time-series models or any model where the outcome changes over time, allowing direct
+    comparison of predicted vs actual values.
 
-    **Signs of High Risk:**
+    ### Test Mechanism
+
+    This test generates a plot for each fitted model in the list. The x-axis represents the date ranging from the
+    specified "start_date" to the "end_date", while the y-axis shows the value of the outcome variable. Two lines are
+    plotted: one representing the forecasted values and the other representing the observed values. The "start_date"
+    and "end_date" can be parameters of this test; if these parameters are not provided, they are set to the minimum
+    and maximum date available in the dataset. The test verifies that the provided date range is within the limits of
+    the available data.
+
+    ### Signs of High Risk
 
     - High risk or failure signs could be deduced visually from the plots if the forecasted line significantly deviates
     from the observed line, indicating the model's predicted values are not matching actual outcomes.
     - A model that struggles to handle the edge conditions like maximum and minimum data points could also be
     considered a sign of risk.
 
-    **Strengths:**
+    ### Strengths
 
     - Visualization: The plot provides an intuitive and clear illustration of how well the forecast matches the actual
     values, making it straightforward even for non-technical stakeholders to interpret.
@@ -46,7 +50,7 @@ class RegressionModelForecastPlot(Metric):
     - Model Evaluation: It can be useful in identifying overfitting or underfitting situations, as these will manifest
     as discrepancies between the forecasted and observed values.
 
-    **Limitations:**
+    ### Limitations
 
     - Interpretation Bias: Interpretation of the plot is subjective and can lead to different conclusions by different
     evaluators.

validmind/tests/model_validation/statsmodels/RegressionModelForecastPlotLevels.py

@@ -14,49 +14,52 @@ from validmind.vm_models import Figure, Metric
 @dataclass
 class RegressionModelForecastPlotLevels(Metric):
     """
-    Compares and visualizes forecasted and actual values of regression models on both raw and transformed datasets.
-
-    **Purpose:**
-    The `RegressionModelForecastPlotLevels` metric is designed to visually assess a series of regression models'
-    performance. It achieves this by contrasting the models' forecasts with the observed data from the respective
-    training and test datasets. The gauge of accuracy here involves determining the extent of closeness between
-    forecasted and actual values. Accordingly, if any transformations are specified, the metric will handle
-    transforming the data before making this comparison.
-
-    **Test Mechanism:**
-    The `RegressionModelForecastPlotLevels` class in Python initiates with a `transformation` parameter, which default
-    aggregates to None. Initially, the class checks for the presence of model objects and raises a `ValueError` if none
-    are found. Each model is then processed, creating predictive forecasts for both its training and testing datasets.
-    These forecasts are then contrasted with the actual values and plotted. In situations where a specified
-    transformation, like "integrate," is specified, the class navigates the transformation steps (performing cumulative
-    sums to generate a novel series, for instance). Finally, plots are produced that compare observed and forecasted
-    values for both the raw and transformed datasets.
-
-    **Signs of High Risk:**
-    Indications of high risk or failure in the model's performance can be derived from checking the generated plots.
-    When the forecasted values dramatically deviate from the observed values in either the training or testing
-    datasets, it suggests a high risk situation. A significant deviation could be a symptom of either overfitting or
-    underfitting, both scenarios are worrying. Such discrepancies could inhibit the model's ability to create precise,
-    generalized results.
-
-    **Strengths:**
-
-    - Visual Evaluations: The metric provides a visual and comparative way of assessing multiple regression models at
-    once. This allows easier interpretation and evaluation of their forecasting accuracy.
-    - Transformation Handling: This metric can handle transformations like "integrate," enhancing its breadth and
-    flexibility in evaluating different models.
-    - Detailed Perspective: By looking at the performance on both datasets (training and testing), the metric may give
-    a detailed overview of the model.
-
-    **Limitations:**
-
-    - Subjectivity: Relying heavily on visual interpretations; assessments may differ from person to person.
-    - Limited Transformation Capability: Currently, only the "integrate" transformation is supported, implying complex
-    transformations might go unchecked or unhandled.
-    - Overhead: The plotting mechanism may become computationally costly when applying to extensive datasets,
-    increasing runtime.
-    - Numerical Measurement: Although visualization is instrumental, a corresponding numerical measure would further
-    reinforce the observations. However, this metric does not provide numerical measures.
+    Assesses the alignment between forecasted and observed values in regression models through visual plots, including
+    handling data transformations.
+
+    ### Purpose
+
+    The `RegressionModelForecastPlotLevels` test aims to visually assess the performance of a series of regression
+    models by comparing their forecasted values against the actual observed values in both training and test datasets.
+    This test helps determine the accuracy of the models and can handle specific data transformations before making the
+    comparison, providing a comprehensive evaluation of model performance.
+
+    ### Test Mechanism
+
+    The test mechanism involves initializing the `RegressionModelForecastPlotLevels` class with an optional
+    `transformation` parameter. The class then:
+
+    - Checks for the presence of model objects and raises a `ValueError` if none are found.
+    - Processes each model to generate predictive forecasts for both training and testing datasets.
+    - Contrasts these forecasts with the actual observed values.
+    - Produces plots to visually compare forecasted and observed values for both raw and transformed datasets.
+    - Handles specified transformations (e.g., "integrate") by performing cumulative sums to create a new series before
+    plotting.
+
+    ### Signs of High Risk
+
+    - Significant deviation between forecasted and observed values in training or testing datasets.
+    - Patterns suggesting overfitting or underfitting.
+    - Large discrepancies in the plotted forecasts, indicating potential issues with model generalizability and
+    precision.
+
+    ### Strengths
+
+    - **Visual Evaluations**: Provides an intuitive, visual way to assess multiple regression models, aiding in easier
+    interpretation and evaluation of forecast accuracy.
+    - **Transformation Handling**: Can process specified data transformations such as "integrate," enhancing
+    flexibility.
+    - **Detailed Perspective**: Assesses performance on both training and testing datasets, offering a comprehensive
+    view of model behavior.
+
+    ### Limitations
+
+    - **Subjectivity**: Relies heavily on visual interpretation, which may vary between individuals.
+    - **Limited Transformation Capability**: Supports only the "integrate" transformation; other complex
+    transformations might not be handled.
+    - **Overhead**: Plotting can be computationally intensive for large datasets, increasing runtime.
+    - **Numerical Measurement**: Does not provide a numerical metric to quantify forecast accuracy, relying solely on
+    visual assessment.
     """
 
     name = "regression_forecast_plot_levels"

validmind/tests/model_validation/statsmodels/RegressionModelSensitivityPlot.py

@@ -16,44 +16,46 @@ logger = get_logger(__name__)
 @dataclass
 class RegressionModelSensitivityPlot(Metric):
     """
-    Tests the sensitivity of a regression model to variations in independent variables by applying shocks and
-    visualizing the effects.
-
-    **Purpose**: The Regression Sensitivity Plot metric is designed to perform sensitivity analysis on regression
-    models. This metric aims to measure the impact of slight changes (shocks) applied to individual variables on the
-    system's outcome while keeping all other variables constant. By doing so, it analyzes the effects of each
-    independent variable on the dependent variable within the regression model and helps identify significant risk
-    factors that could substantially influence the model's output.
-
-    **Test Mechanism**: This metric operates by initially applying shocks of varying magnitudes, defined by specific
-    parameters, to each of the model's features, one at a time. With all other variables held constant, a new
-    prediction is made for each dataset subjected to shocks. Any changes in the model's predictions are directly
-    attributed to the shocks applied. In the event that the transformation parameter is set to "integrate", initial
-    predictions and target values undergo transformation via an integration function before being plotted. Lastly, a
-    plot demonstrating observed values against predicted values for each model is generated, showcasing a distinct line
-    graph illustrating predictions for each shock.
-
-    **Signs of High Risk**:
-    - If the plot exhibits drastic alterations in model predictions consequent to minor shocks to an individual
-    variable, it may indicate high risk. This underscores potentially high model sensitivity to changes in that
-    variable, suggesting over-dependence on that variable for predictions.
-    - Unusually high or unpredictable shifts in response to shocks may also denote potential model instability or
+    Assesses the sensitivity of a regression model to changes in independent variables by applying shocks and
+    visualizing the impact.
+
+    ### Purpose
+
+    The Regression Sensitivity Plot test is designed to perform sensitivity analysis on regression models. This test
+    aims to measure the impact of slight changes (shocks) applied to individual variables on the system's outcome while
+    keeping all other variables constant. By doing so, it analyzes the effects of each independent variable on the
+    dependent variable within the regression model, helping identify significant risk factors that could substantially
+    influence the model's output.
+
+    ### Test Mechanism
+
+    This test operates by initially applying shocks of varying magnitudes, defined by specific parameters, to each of
+    the model's features, one at a time. With all other variables held constant, a new prediction is made for each
+    dataset subjected to shocks. Any changes in the model's predictions are directly attributed to the shocks applied.
+    If the transformation parameter is set to "integrate," initial predictions and target values undergo transformation
+    via an integration function before being plotted. Finally, a plot demonstrating observed values against predicted
+    values for each model is generated, showcasing a distinct line graph illustrating predictions for each shock.
+
+    ### Signs of High Risk
+
+    - Drastic alterations in model predictions due to minor shocks to an individual variable, indicating high
+    sensitivity and potential over-dependence on that variable.
+    - Unusually high or unpredictable shifts in response to shocks, suggesting potential model instability or
     overfitting.
 
-    **Strengths**:
-    - The metric allows identification of variables strongly influencing the model outcomes, paving the way for
-    understanding feature importance.
-    - It generates visual plots which make the results easily interpretable even to non-technical stakeholders.
-    - Beneficial in identifying overfitting and detecting unstable models that over-react to minor changes in variables.
-
-    **Limitations**:
-    - The metric operates on the assumption that all other variables remain unchanged during the application of a
-    shock. However, real-world situations where variables may possess intricate interdependencies may not always
-    reflect this.
-    - It is best compatible with linear models and may not effectively evaluate the sensitivity of non-linear model
-    configurations.
-    - The metric does not provide a numerical risk measure. It offers only a visual representation, which may invite
-    subjectivity in interpretation.
+    ### Strengths
+
+    - Helps identify variables that strongly influence model outcomes, aiding in understanding feature importance.
+    - Generates visual plots, making results easily interpretable even to non-technical stakeholders.
+    - Useful in identifying overfitting and detecting unstable models that react excessively to minor variable changes.
+
+    ### Limitations
+
+    - Operates on the assumption that all other variables remain unchanged during the application of a shock, which may
+    not reflect real-world interdependencies.
+    - Best compatible with linear models and may not effectively evaluate the sensitivity of non-linear models.
+    - Provides a visual representation without a numerical risk measure, potentially introducing subjectivity in
+    interpretation.
     """
 
     name = "regression_sensitivity_plot"

validmind/tests/model_validation/statsmodels/RegressionModelSummary.py

@@ -17,36 +17,38 @@ class RegressionModelSummary(Metric):
     """
     Evaluates regression model performance using metrics including R-Squared, Adjusted R-Squared, MSE, and RMSE.
 
-    **Purpose**: This metric test evaluates the performance of regression models by measuring their predictive ability
-    with regards to dependent variables given changes in the independent variables. Its measurement tools include
-    conventional regression metrics such as R-Squared, Adjusted R-Squared, Mean Squared Error (MSE), and Root Mean
-    Squared Error (RMSE).
-
-    **Test Mechanism**: This test employs the 'train_ds' attribute of the model to gather and analyze the training
-    data. Initially, it fetches the independent variables and uses the model to make predictions on these given
-    features. Subsequently, it calculates several standard regression performance metrics including R-Square, Adjusted
-    R-Squared, Mean Squared Error (MSE), and Root Mean Squared Error (RMSE), which quantify the approximation of the
-    predicted responses to the actual responses.
-
-    **Signs of High Risk**:
-    - Low R-Squared and Adjusted R-Squared values. A poor fit between the model predictions and the true responses is
-    indicated by low values of these metrics, suggesting the model explains a small fraction of the variance in the
-    target variable.
-    - High MSE and RMSE values represent a high prediction error and point to poor model performance.
-
-    **Strengths**:
+    ### Purpose
+
+    The Regression Model Summary test evaluates the performance of regression models by measuring their predictive
+    ability regarding dependent variables given changes in the independent variables. It uses conventional regression
+    metrics such as R-Squared, Adjusted R-Squared, Mean Squared Error (MSE), and Root Mean Squared Error (RMSE) to
+    assess the model's accuracy and fit.
+
+    ### Test Mechanism
+
+    This test employs the 'train_ds' attribute of the model to gather and analyze the training data. Initially, it
+    fetches the independent variables and uses the model to make predictions on these given features. Subsequently, it
+    calculates several standard regression performance metrics including R-Squared, Adjusted R-Squared, Mean Squared
+    Error (MSE), and Root Mean Squared Error (RMSE), which quantify the approximation of the predicted responses to the
+    actual responses.
+
+    ### Signs of High Risk
+
+    - Low R-Squared and Adjusted R-Squared values.
+    - High MSE and RMSE values.
+
+    ### Strengths
+
     - Offers an extensive evaluation of regression models by combining four key measures of model accuracy and fit.
     - Provides a comprehensive view of the model's performance.
-    - Both the R-Squared and Adjusted R-Squared measures are readily interpretable. They represent the proportion of
-    total variation in the dependent variable that can be explained by the independent variables.
-
-    **Limitations**:
-    - Applicable exclusively to regression models. It is not suited for evaluating binary classification models or time
-    series models, thus limiting its scope.
-    - Although RMSE and MSE are sound measures of prediction error, they might be sensitive to outliers, potentially
-    leading to an overestimation of the model's prediction error.
-    - A high R-squared or adjusted R-squared may not necessarily indicate a good model, especially in cases where the
-    model is possibly overfitting the data.
+    - Both the R-Squared and Adjusted R-Squared measures are readily interpretable.
+
+    ### Limitations
+
+    - Applicable exclusively to regression models.
+    - RMSE and MSE might be sensitive to outliers.
+    - A high R-Squared or Adjusted R-Squared may not necessarily indicate a good model, especially in cases of
+    overfitting.
     """
 
     name = "regression_model_summary"

validmind/tests/model_validation/statsmodels/RegressionPermutationFeatureImportance.py

@@ -21,28 +21,35 @@ logger = get_logger(__name__)
 class RegressionPermutationFeatureImportance(Metric):
     """
     Assesses the significance of each feature in a model by evaluating the impact on model performance when feature
-    values are randomly rearranged. Specifically designed for use with statsmodels, this metric offers insight into the
-    importance of features based on the decrease in model's predictive accuracy, typically R².
+    values are randomly rearranged.
 
-    **Purpose**: The primary purpose of this metric is to determine which features significantly impact the performance
-    of a regression model developed using statsmodels. The metric measures how much the prediction accuracy deteriorates
+    ### Purpose
+
+    The primary purpose of this metric is to determine which features significantly impact the performance of a
+    regression model developed using statsmodels. The metric measures how much the prediction accuracy deteriorates
     when each feature's values are permuted.
 
-    **Test Mechanism**: This metric shuffles the values of each feature one at a time in the dataset, computes the model's
-    performance after each permutation, and compares it to the baseline performance. A significant decrease in performance
+    ### Test Mechanism
+
+    This metric shuffles the values of each feature one at a time in the dataset, computes the model's performance
+    after each permutation, and compares it to the baseline performance. A significant decrease in performance
     indicates the importance of the feature.
 
-    **Signs of High Risk**:
-    - Significant reliance on a feature that when permuted leads to a substantial decrease in performance, suggesting
+    ### Signs of High Risk
+
+    - Significant reliance on a feature that, when permuted, leads to a substantial decrease in performance, suggesting
     overfitting or high model dependency on that feature.
     - Features identified as unimportant despite known impacts from domain knowledge, suggesting potential issues in
     model training or data preprocessing.
 
-    **Strengths**:
-    - Directly assesses the impact of each feature on model performance, providing clear insights into model dependencies.
+    ### Strengths
+
+    - Directly assesses the impact of each feature on model performance, providing clear insights into model
+    dependencies.
     - Model-agnostic within the scope of statsmodels, applicable to any regression model that outputs predictions.
 
-    **Limitations**:
+    ### Limitations
+
     - The metric is specific to statsmodels and cannot be used with other types of models without adaptation.
     - It does not capture interactions between features, which can lead to underestimating the importance of correlated
     features.

validmind/tests/model_validation/statsmodels/RunsTest.py

@@ -11,41 +11,45 @@ class RunsTest(Metric):
     """
     Executes Runs Test on ML model to detect non-random patterns in output data sequence.
 
-    **Purpose**: The Runs Test is a statistical procedure used to determine whether the sequence of data extracted from
-    the ML model behaves randomly or not. Specifically, it analyzes runs, sequences of consecutive positives or
-    negatives, in the data to check if there are more or fewer runs than expected under the assumption of randomness.
-    This can be an indication of some pattern, trend, or cycle in the model's output which may need attention.
-
-    **Test Mechanism**: The testing mechanism applies the Runs Test from the statsmodels module on each column of the
-    training dataset. For every feature in the dataset, a Runs Test is executed, whose output includes a Runs Statistic
-    and P-value. A low P-value suggests that data arrangement in the feature is not likely to be random. The results
-    are stored in a dictionary where the keys are the feature names, and the values are another dictionary storing the
-    test statistic and the P-value for each feature.
-
-    **Signs of High Risk**:
+    ### Purpose
+
+    The Runs Test is a statistical procedure used to determine whether the sequence of data extracted from the ML model
+    behaves randomly or not. Specifically, it analyzes runs, sequences of consecutive positives or negatives, in the
+    data to check if there are more or fewer runs than expected under the assumption of randomness. This can be an
+    indication of some pattern, trend, or cycle in the model's output which may need attention.
+
+    ### Test Mechanism
+
+    The testing mechanism applies the Runs Test from the statsmodels module on each column of the training dataset. For
+    every feature in the dataset, a Runs Test is executed, whose output includes a Runs Statistic and P-value. A low
+    P-value suggests that data arrangement in the feature is not likely to be random. The results are stored in a
+    dictionary where the keys are the feature names, and the values are another dictionary storing the test statistic
+    and the P-value for each feature.
+
+    ### Signs of High Risk
+
     - High risk is indicated when the P-value is close to zero.
-    - If the p-value is less than a predefined significance level (like 0.05), it suggests that the runs (series of
+    - If the P-value is less than a predefined significance level (like 0.05), it suggests that the runs (series of
     positive or negative values) in the model's output are not random and are longer or shorter than what is expected
     under a random scenario.
     - This would mean there's a high risk of non-random distribution of errors or model outcomes, suggesting potential
     issues with the model.
 
-    **Strengths**:
-    - The strength of the Runs Test is that it's straightforward and fast for detecting non-random patterns in data
-    sequence.
-    - It can validate assumptions of randomness, which is particularly valuable for checking error distributions in
-    regression models, trendless time series data, and making sure a classifier doesn't favour one class over another.
-    - Moreover, it can be applied to both classification and regression tasks, making it versatile.
-
-    **Limitations**:
-    - The test assumes that the data is independently and identically distributed (i.i.d.), which might not be the case
-    for many real-world datasets.
-    - The conclusion drawn from the low p-value indicating non-randomness does not provide information about the type
+    ### Strengths
+
+    - Straightforward and fast for detecting non-random patterns in data sequence.
+    - Validates assumptions of randomness, which is valuable for checking error distributions in regression models,
+    trendless time series data, and ensuring a classifier doesn't favor one class over another.
+    - Can be applied to both classification and regression tasks, making it versatile.
+
+    ### Limitations
+
+    - Assumes that the data is independently and identically distributed (i.i.d.), which might not be the case for many
+    real-world datasets.
+    - The conclusion drawn from the low P-value indicating non-randomness does not provide information about the type
     or the source of the detected pattern.
-    - Also, it is sensitive to extreme values (outliers), and overly large or small run sequences can influence the
-    results.
-    - Furthermore, this test does not provide model performance evaluation; it is used to detect patterns in the
-    sequence of outputs only.
+    - Sensitive to extreme values (outliers), and overly large or small run sequences can influence the results.
+    - Does not provide model performance evaluation; it is used to detect patterns in the sequence of outputs only.
     """
 
     name = "runs_test"