gitlabds 2.4.0__tar.gz → 2.4.2__tar.gz

{gitlabds-2.4.0 → gitlabds-2.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gitlabds
-Version: 2.4.0
+Version: 2.4.2
 Summary: GitLab Data Science Tools
 Author-email: Kevin Dietz <kdietz@gitlab.com>
 License-Expression: MIT
@@ -376,9 +376,9 @@ test_df_transformed = gitlabds.apply_dummy(
 <details><summary> Remove Low Variation </summary>
 
 #### Description
-Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value.
+Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value. Columns that are entirely null are always dropped.
 
-`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True):`
+`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True, dropna=False):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -386,6 +386,7 @@ Remove columns from a dataset that do not meet the variation threshold. That is,
 - **_columns_** : Will examine at all columns by default. To limit to just a subset of columns, pass a list of column names. 
 - **_threshold_**: The maximum percentage one value in a column can represent. columns that exceed this threshold will be dropped. For example, the default value of `0.98` will drop any column where one value is present in more than 98% of rows.
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `False` (default) to count missing values as their own value. Set to `True` to measure variation over non-null values only, so a column that is one value plus NaN is dropped. Use `True` when missing values are left in place for a model that handles them natively (e.g. XGBoost, LightGBM).
 
 #### Returns
 - DataFrame with low variation columns dropped.
@@ -399,6 +400,10 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', columns='all',
 # Drop any of the selected columns where one value is present in more than 99% of rows.
 new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB', 'colC'], threshold=.99)
 ```
+```python
+# Ignore missing values: drop columns where one value makes up more than 98% of the non-null rows.
+new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', dropna=True)
+```
 </details>
 
 <details><summary> Correlation Reduction </summary>
@@ -406,7 +411,7 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB
 #### Description
 Reduce the number of columns on a dataframe by dropping columns that are highly correlated with other columns. Note: only one of the two highly correlated columns will be dropped.
 
-`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True):`
+`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -414,6 +419,8 @@ Reduce the number of columns on a dataframe by dropping columns that are highly
 - **_threshold_**: The threshold above which columns will be dropped. If two variables exceed this threshold, one will be dropped from the dataframe. For example, the default value of `0.90` will identify columns that have correlations greater than 90% to each other and drop one of those columns.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each pair of columns (and each column with the outcome) using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both columns of a pair are non-null for the pair to be evaluated. Pairs below it are never treated as correlated. Guards against spurious high correlations between sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with redundant correlated columns dropped.
@@ -434,7 +441,7 @@ new_df = gitlabds.correlation_reduction(df=my_df, dv=None, threshold=0.90, metho
 #### Description
 Remove columns that are highly correlated with the outcome (target) column.
 
-`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True):`
+`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -442,6 +449,8 @@ Remove columns that are highly correlated with the outcome (target) column.
 - _**threshold**_ : The correlation value to the outcome above which columns will be dropped. For example, the default value of `0.80` will identify and drop columns that have correlations greater than 80% to the outcome.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each column to the outcome using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both the column and the outcome are non-null for the column to be evaluated. Columns below it are never dropped. Guards against spurious high correlations from sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with outcome proxy columns dropped.
@@ -455,6 +464,10 @@ new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.7
 # Drop columns with correlations to the outcome greater than 80% using Spearman correlation
 new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.8, method="spearman")        
 ```
+```python
+# Data with missing values: only evaluate columns with at least 100 non-null rows alongside the outcome
+new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', method="spearman", min_periods=100)
+```
 </details>
 
 <details><summary> Drop Categorical </summary>
@@ -901,7 +914,7 @@ shap_df = gitlabds.per_record_shap(
 #### Description
 "Change X -> +Y" counterfactual search, direction chosen empirically. For each actionable lever SHAP surfaced for a record, evaluates candidate values drawn from peer-group quantiles, re-scores the model at each, and keeps the single move that maximizes `gain` — the improvement in the model score, signed by `score_goal` so that bigger is always better. No monotonicity assumption is made.
 
-`gitlabds.per_record_counterfactuals(model, df, shap_df, unique_id="record_id", peer_group=None, candidate_quantiles=[0.1, 0.25, 0.5, 0.75, 0.9], direction=None, score_goal="maximize", actionable_values=["Yes", "Partial"], min_gain=0.005, impact_medium=0.01, impact_high=0.03, levels={"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}, n_bins=5):`
+`gitlabds.per_record_counterfactuals(model, df, shap_df, unique_id="record_id", peer_group=None, candidate_quantiles=[0.1, 0.25, 0.5, 0.75, 0.9], direction=None, score_goal="maximize", actionable_values=["Yes", "Partial"], min_gain=0.005, impact_medium=0.01, impact_high=0.03, levels={"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}, n_bins=5, prefer_nearest_within=None):`
 
 #### Parameters:
 - _**model**_ : trained model (as in `per_record_shap`). How probabilities are scored is chosen from the model object itself — `.predict_proba()` where available, otherwise a raw `xgboost.Booster` or `lightgbm.Booster` is scored natively. An unsupported type raises a `TypeError` naming it
@@ -913,10 +926,11 @@ shap_df = gitlabds.per_record_shap(
 - _**direction**_ : optional per-feature constraint mapping feature -> `'increasing'` | `'decreasing'` | `'both'` | `'none'` (dict or Series). `'increasing'`/`'decreasing'` restrict candidates to moves in that direction (an achievability constraint); `'both'` allows either; `'none'` excludes the feature from lever evaluation entirely, for features with no meaningful good direction (categoricals, identifiers, context-only numerics). Features absent from the mapping, or `direction=None`, default to `'both'`. A blank/NaN raises — normalize it to `'none'` where the feature dictionary is read
 - _**score_goal**_ : `"maximize"` if a higher predicted probability of class 1 is the good outcome (e.g. class 1 = uptier), `"minimize"` if it is the bad one (e.g. class 1 = churn). Sets the sign of `gain`, which drives best-move selection, the `min_gain` filter, `impact_bucket`, and the output ordering. This is the outcome's polarity only — `direction` above is a separate, feature-level concept and is unaffected. Default is `"maximize"`
 - _**actionable_values**_ : values in `shap_df["actionable"]` treated as actionable levers
-- _**min_gain**_ : drop levers whose best achievable gain is below this. Note this is a threshold on `gain`, not on raw `delta_p`: under `score_goal="minimize"` a good move has a negative `delta_p` but a positive `gain`, so this stays positive for both polarities. Negative values (or `-inf`) are allowed, and keep moves that make the outcome worse — useful for inspecting the full candidate distribution while tuning thresholds
+- _**min_gain**_ : drop levers whose best achievable gain is below this. Note this is a threshold on `gain`, not on raw `delta_p`: under `score_goal="minimize"` a good move has a negative `delta_p` but a positive `gain`, so this stays positive for both polarities. Negative values (or `-inf`) are allowed, and keep moves that make the outcome worse — useful for inspecting the full candidate distribution while tuning thresholds. With `prefer_nearest_within` set, selection runs first, so this threshold applies to the gain of the chosen (nearer, lower-gain) candidate rather than the lever's best — a lever whose best gain clears `min_gain` can still be dropped on the nearer move it settled for
 - _**impact_medium, impact_high**_ : `gain` thresholds bucketing each lever into Low/Medium/High `impact_bucket`. Only meaningful for `gain > 0` — under a negative `min_gain` the retained harmful moves stay `"Low"` (the level-crossing floor doesn't apply to them), so read `gain`'s sign rather than the bucket alone
 - _**levels**_ : dict mapping label -> upper bound on P(class 1) for that level (e.g., default `{"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}` means `p <= 0.03` -> "poor fit", `0.03 < p <= 0.10` -> "good fit", `p > 0.10` -> "best fit"). Key order doesn't matter — levels are sorted by cutoff and labels follow that order — but the top level's cutoff must be `float("inf")`. Cutoffs are always in P(class 1) space regardless of `score_goal`, and `crosses_level` is a plain inequality, so name the labels for your own outcome: a churn model would pass something like `{"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")}`. Used to detect whether a move crosses a level boundary (any crossing move is floored at "Medium" impact)
 - _**n_bins**_ : quantile bins for numeric `peer_group` columns
+- _**prefer_nearest_within**_ : `None` (default) keeps the highest-gain candidate per (record, feature). A float in `(0, 1]` instead keeps the *nearest* candidate that still captures that fraction of the best gain: among candidates with `gain >= prefer_nearest_within * best_gain`, the one with the smallest `abs(intervention_value - current_value)` wins, ties broken on higher gain. 
 
 #### Returns
 - Long DataFrame, one row per (record, lever): `current_value`, `intervention_value`, `intervention_quantile`, `p_before`, `p_after`, `delta_p`, `gain`, `impact_bucket`, `level_before`, `level_after`, `crosses_level`
@@ -947,6 +961,16 @@ counterfactuals_df = gitlabds.per_record_counterfactuals(
     direction=dict(zip(feature_dict["field"], feature_dict["good_direction"])),
     levels={"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")},
 )
+
+# Prefer a nearer, more sellable move: the closest candidate that still captures
+# at least 70% of the best available gain.
+counterfactuals_df = gitlabds.per_record_counterfactuals(
+    model=trained_model,
+    df=model_features,
+    shap_df=shap_df,
+    peer_group=peer_group_df,
+    prefer_nearest_within=0.7,
+)
 ```
 
 </details>

{gitlabds-2.4.0 → gitlabds-2.4.2}/README.md

@@ -340,9 +340,9 @@ test_df_transformed = gitlabds.apply_dummy(
 <details><summary> Remove Low Variation </summary>
 
 #### Description
-Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value.
+Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value. Columns that are entirely null are always dropped.
 
-`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True):`
+`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True, dropna=False):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -350,6 +350,7 @@ Remove columns from a dataset that do not meet the variation threshold. That is,
 - **_columns_** : Will examine at all columns by default. To limit to just a subset of columns, pass a list of column names. 
 - **_threshold_**: The maximum percentage one value in a column can represent. columns that exceed this threshold will be dropped. For example, the default value of `0.98` will drop any column where one value is present in more than 98% of rows.
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `False` (default) to count missing values as their own value. Set to `True` to measure variation over non-null values only, so a column that is one value plus NaN is dropped. Use `True` when missing values are left in place for a model that handles them natively (e.g. XGBoost, LightGBM).
 
 #### Returns
 - DataFrame with low variation columns dropped.
@@ -363,6 +364,10 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', columns='all',
 # Drop any of the selected columns where one value is present in more than 99% of rows.
 new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB', 'colC'], threshold=.99)
 ```
+```python
+# Ignore missing values: drop columns where one value makes up more than 98% of the non-null rows.
+new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', dropna=True)
+```
 </details>
 
 <details><summary> Correlation Reduction </summary>
@@ -370,7 +375,7 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB
 #### Description
 Reduce the number of columns on a dataframe by dropping columns that are highly correlated with other columns. Note: only one of the two highly correlated columns will be dropped.
 
-`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True):`
+`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -378,6 +383,8 @@ Reduce the number of columns on a dataframe by dropping columns that are highly
 - **_threshold_**: The threshold above which columns will be dropped. If two variables exceed this threshold, one will be dropped from the dataframe. For example, the default value of `0.90` will identify columns that have correlations greater than 90% to each other and drop one of those columns.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each pair of columns (and each column with the outcome) using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both columns of a pair are non-null for the pair to be evaluated. Pairs below it are never treated as correlated. Guards against spurious high correlations between sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with redundant correlated columns dropped.
@@ -398,7 +405,7 @@ new_df = gitlabds.correlation_reduction(df=my_df, dv=None, threshold=0.90, metho
 #### Description
 Remove columns that are highly correlated with the outcome (target) column.
 
-`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True):`
+`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -406,6 +413,8 @@ Remove columns that are highly correlated with the outcome (target) column.
 - _**threshold**_ : The correlation value to the outcome above which columns will be dropped. For example, the default value of `0.80` will identify and drop columns that have correlations greater than 80% to the outcome.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each column to the outcome using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both the column and the outcome are non-null for the column to be evaluated. Columns below it are never dropped. Guards against spurious high correlations from sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with outcome proxy columns dropped.
@@ -419,6 +428,10 @@ new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.7
 # Drop columns with correlations to the outcome greater than 80% using Spearman correlation
 new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.8, method="spearman")        
 ```
+```python
+# Data with missing values: only evaluate columns with at least 100 non-null rows alongside the outcome
+new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', method="spearman", min_periods=100)
+```
 </details>
 
 <details><summary> Drop Categorical </summary>
@@ -865,7 +878,7 @@ shap_df = gitlabds.per_record_shap(
 #### Description
 "Change X -> +Y" counterfactual search, direction chosen empirically. For each actionable lever SHAP surfaced for a record, evaluates candidate values drawn from peer-group quantiles, re-scores the model at each, and keeps the single move that maximizes `gain` — the improvement in the model score, signed by `score_goal` so that bigger is always better. No monotonicity assumption is made.
 
-`gitlabds.per_record_counterfactuals(model, df, shap_df, unique_id="record_id", peer_group=None, candidate_quantiles=[0.1, 0.25, 0.5, 0.75, 0.9], direction=None, score_goal="maximize", actionable_values=["Yes", "Partial"], min_gain=0.005, impact_medium=0.01, impact_high=0.03, levels={"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}, n_bins=5):`
+`gitlabds.per_record_counterfactuals(model, df, shap_df, unique_id="record_id", peer_group=None, candidate_quantiles=[0.1, 0.25, 0.5, 0.75, 0.9], direction=None, score_goal="maximize", actionable_values=["Yes", "Partial"], min_gain=0.005, impact_medium=0.01, impact_high=0.03, levels={"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}, n_bins=5, prefer_nearest_within=None):`
 
 #### Parameters:
 - _**model**_ : trained model (as in `per_record_shap`). How probabilities are scored is chosen from the model object itself — `.predict_proba()` where available, otherwise a raw `xgboost.Booster` or `lightgbm.Booster` is scored natively. An unsupported type raises a `TypeError` naming it
@@ -877,10 +890,11 @@ shap_df = gitlabds.per_record_shap(
 - _**direction**_ : optional per-feature constraint mapping feature -> `'increasing'` | `'decreasing'` | `'both'` | `'none'` (dict or Series). `'increasing'`/`'decreasing'` restrict candidates to moves in that direction (an achievability constraint); `'both'` allows either; `'none'` excludes the feature from lever evaluation entirely, for features with no meaningful good direction (categoricals, identifiers, context-only numerics). Features absent from the mapping, or `direction=None`, default to `'both'`. A blank/NaN raises — normalize it to `'none'` where the feature dictionary is read
 - _**score_goal**_ : `"maximize"` if a higher predicted probability of class 1 is the good outcome (e.g. class 1 = uptier), `"minimize"` if it is the bad one (e.g. class 1 = churn). Sets the sign of `gain`, which drives best-move selection, the `min_gain` filter, `impact_bucket`, and the output ordering. This is the outcome's polarity only — `direction` above is a separate, feature-level concept and is unaffected. Default is `"maximize"`
 - _**actionable_values**_ : values in `shap_df["actionable"]` treated as actionable levers
-- _**min_gain**_ : drop levers whose best achievable gain is below this. Note this is a threshold on `gain`, not on raw `delta_p`: under `score_goal="minimize"` a good move has a negative `delta_p` but a positive `gain`, so this stays positive for both polarities. Negative values (or `-inf`) are allowed, and keep moves that make the outcome worse — useful for inspecting the full candidate distribution while tuning thresholds
+- _**min_gain**_ : drop levers whose best achievable gain is below this. Note this is a threshold on `gain`, not on raw `delta_p`: under `score_goal="minimize"` a good move has a negative `delta_p` but a positive `gain`, so this stays positive for both polarities. Negative values (or `-inf`) are allowed, and keep moves that make the outcome worse — useful for inspecting the full candidate distribution while tuning thresholds. With `prefer_nearest_within` set, selection runs first, so this threshold applies to the gain of the chosen (nearer, lower-gain) candidate rather than the lever's best — a lever whose best gain clears `min_gain` can still be dropped on the nearer move it settled for
 - _**impact_medium, impact_high**_ : `gain` thresholds bucketing each lever into Low/Medium/High `impact_bucket`. Only meaningful for `gain > 0` — under a negative `min_gain` the retained harmful moves stay `"Low"` (the level-crossing floor doesn't apply to them), so read `gain`'s sign rather than the bucket alone
 - _**levels**_ : dict mapping label -> upper bound on P(class 1) for that level (e.g., default `{"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}` means `p <= 0.03` -> "poor fit", `0.03 < p <= 0.10` -> "good fit", `p > 0.10` -> "best fit"). Key order doesn't matter — levels are sorted by cutoff and labels follow that order — but the top level's cutoff must be `float("inf")`. Cutoffs are always in P(class 1) space regardless of `score_goal`, and `crosses_level` is a plain inequality, so name the labels for your own outcome: a churn model would pass something like `{"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")}`. Used to detect whether a move crosses a level boundary (any crossing move is floored at "Medium" impact)
 - _**n_bins**_ : quantile bins for numeric `peer_group` columns
+- _**prefer_nearest_within**_ : `None` (default) keeps the highest-gain candidate per (record, feature). A float in `(0, 1]` instead keeps the *nearest* candidate that still captures that fraction of the best gain: among candidates with `gain >= prefer_nearest_within * best_gain`, the one with the smallest `abs(intervention_value - current_value)` wins, ties broken on higher gain. 
 
 #### Returns
 - Long DataFrame, one row per (record, lever): `current_value`, `intervention_value`, `intervention_quantile`, `p_before`, `p_after`, `delta_p`, `gain`, `impact_bucket`, `level_before`, `level_after`, `crosses_level`
@@ -911,6 +925,16 @@ counterfactuals_df = gitlabds.per_record_counterfactuals(
     direction=dict(zip(feature_dict["field"], feature_dict["good_direction"])),
     levels={"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")},
 )
+
+# Prefer a nearer, more sellable move: the closest candidate that still captures
+# at least 70% of the best available gain.
+counterfactuals_df = gitlabds.per_record_counterfactuals(
+    model=trained_model,
+    df=model_features,
+    shap_df=shap_df,
+    peer_group=peer_group_df,
+    prefer_nearest_within=0.7,
+)
 ```
 
 </details>