gitlabds 2.3.1__tar.gz → 2.4.1__tar.gz

{gitlabds-2.3.1 → gitlabds-2.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gitlabds
-Version: 2.3.1
+Version: 2.4.1
 Summary: GitLab Data Science Tools
 Author-email: Kevin Dietz <kdietz@gitlab.com>
 License-Expression: MIT
@@ -376,9 +376,9 @@ test_df_transformed = gitlabds.apply_dummy(
 <details><summary> Remove Low Variation </summary>
 
 #### Description
-Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value.
+Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value. Columns that are entirely null are always dropped.
 
-`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True):`
+`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True, dropna=False):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -386,6 +386,7 @@ Remove columns from a dataset that do not meet the variation threshold. That is,
 - **_columns_** : Will examine at all columns by default. To limit to just a subset of columns, pass a list of column names. 
 - **_threshold_**: The maximum percentage one value in a column can represent. columns that exceed this threshold will be dropped. For example, the default value of `0.98` will drop any column where one value is present in more than 98% of rows.
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `False` (default) to count missing values as their own value. Set to `True` to measure variation over non-null values only, so a column that is one value plus NaN is dropped. Use `True` when missing values are left in place for a model that handles them natively (e.g. XGBoost, LightGBM).
 
 #### Returns
 - DataFrame with low variation columns dropped.
@@ -399,6 +400,10 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', columns='all',
 # Drop any of the selected columns where one value is present in more than 99% of rows.
 new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB', 'colC'], threshold=.99)
 ```
+```python
+# Ignore missing values: drop columns where one value makes up more than 98% of the non-null rows.
+new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', dropna=True)
+```
 </details>
 
 <details><summary> Correlation Reduction </summary>
@@ -406,7 +411,7 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB
 #### Description
 Reduce the number of columns on a dataframe by dropping columns that are highly correlated with other columns. Note: only one of the two highly correlated columns will be dropped.
 
-`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True):`
+`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -414,6 +419,8 @@ Reduce the number of columns on a dataframe by dropping columns that are highly
 - **_threshold_**: The threshold above which columns will be dropped. If two variables exceed this threshold, one will be dropped from the dataframe. For example, the default value of `0.90` will identify columns that have correlations greater than 90% to each other and drop one of those columns.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each pair of columns (and each column with the outcome) using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both columns of a pair are non-null for the pair to be evaluated. Pairs below it are never treated as correlated. Guards against spurious high correlations between sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with redundant correlated columns dropped.
@@ -434,7 +441,7 @@ new_df = gitlabds.correlation_reduction(df=my_df, dv=None, threshold=0.90, metho
 #### Description
 Remove columns that are highly correlated with the outcome (target) column.
 
-`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True):`
+`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -442,6 +449,8 @@ Remove columns that are highly correlated with the outcome (target) column.
 - _**threshold**_ : The correlation value to the outcome above which columns will be dropped. For example, the default value of `0.80` will identify and drop columns that have correlations greater than 80% to the outcome.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each column to the outcome using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both the column and the outcome are non-null for the column to be evaluated. Columns below it are never dropped. Guards against spurious high correlations from sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with outcome proxy columns dropped.
@@ -455,6 +464,10 @@ new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.7
 # Drop columns with correlations to the outcome greater than 80% using Spearman correlation
 new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.8, method="spearman")        
 ```
+```python
+# Data with missing values: only evaluate columns with at least 100 non-null rows alongside the outcome
+new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', method="spearman", min_periods=100)
+```
 </details>
 
 <details><summary> Drop Categorical </summary>
@@ -1135,7 +1148,7 @@ feature_drift = results['feature_drift']
 
 #### Snowflake Connectors
 
-Thin wrappers over the native Snowflake SDKs that resolve GitLab's two credential paths — local (dbt profile + SSO) and GitLab CI (env vars + password or key pair) — and then hand off to the native objects. Migrated from `gitlabdata.orchestration_utils`; see the migration notes at the end of this section.
+Thin wrappers over the native Snowflake SDKs that resolve GitLab's two credential paths — local (dbt profile + SSO) and GitLab CI (env vars + key-pair auth) — and then hand off to the native objects. Migrated from `gitlabdata.orchestration_utils`; see the migration notes at the end of this section.
 
 All of these require `pip install gitlabds[snowflake]`.
 
@@ -1144,7 +1157,7 @@ All of these require `pip install gitlabds[snowflake]`.
 #### Description
 Open a Snowflake connection, resolving credentials from your dbt profile locally or from GitLab CI environment variables in CI.
 
-`gitlabds.snowflake_engine_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="dev", schema="", auth_method="password", profiles_path=None, **connect_kwargs):`
+`gitlabds.snowflake_engine_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="dev", schema="", auth_method="keypair", profiles_path=None, **connect_kwargs):`
 
 #### Parameters:
 - **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
@@ -1152,10 +1165,12 @@ Open a Snowflake connection, resolving credentials from your dbt profile locally
 - **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
 - **_profile_target_** : dbt profile output name. Only used on the local path.
 - **_schema_** : Schema to connect to. Falls back to the dbt profile's schema locally; when it resolves empty the connection is opened with no default schema.
-- **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path. Key pair secrets are not yet provisioned in CI.
+- **_auth_method_** : CI authentication method. `"keypair"` is the only supported value and the default, so passing it explicitly is the same as omitting it. Retained so callers that pass `auth_method="keypair"` keep working. `"password"` is accepted but ignored, with a logged warning naming the fix; any other value raises `ValueError`.
 - **_profiles_path_** : Override the dbt `profiles.yml` location (defaults to `~/.dbt/profiles.yml`). Only used on the local path.
 - **_\*\*connect_kwargs_** : Passed through to `snowflake.connector.connect()`, e.g. `autocommit=False`, `client_session_keep_alive=True`.
 
+> **CI authentication** uses key-pair auth exclusively (password-based CI auth was removed ahead of Snowflake's 2026-09-22 service-user password deprecation). In CI, reads `SNOWFLAKE_DATA_SCIENCE_LOAD_PRIVATE_KEY` (raw PEM or base64-encoded PEM — base64 is required for GitLab CI/CD variables since multi-line values cannot be masked) and optionally `SNOWFLAKE_DATA_SCIENCE_LOAD_PRIVATE_KEY_PASSPHRASE` (or the `_PASSWORD` variant) for encrypted keys.
+
 #### Returns
 - An open `snowflake.connector.SnowflakeConnection`. The caller owns its lifecycle.
 
@@ -1300,17 +1315,19 @@ gitlabds.snowflake_stage_load_copy_remove(
 #### Description
 Create a Snowflake Feature Store connection (Snowpark Session + native `FeatureStore`), resolving credentials the same way as `snowflake_engine_factory`.
 
-`gitlabds.feature_store_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="feature_store", auth_method="password", creation_mode="CreationMode.FAIL_IF_NOT_EXIST", profiles_path=None):`
+`gitlabds.feature_store_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="feature_store", auth_method="keypair", creation_mode="CreationMode.FAIL_IF_NOT_EXIST", profiles_path=None):`
 
 #### Parameters:
 - **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
 - **_role_** : Snowflake role for CI credentials. The feature store's database and schema come from `SNOWFLAKE_FEATURE_STORE_DATABASE` / `SNOWFLAKE_FEATURE_STORE_SCHEMA` on the CI path, not from the role's usual database/schema.
 - **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
 - **_profile_target_** : dbt profile output name. Only used on the local path.
-- **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path.
+- **_auth_method_** : CI authentication method. `"keypair"` is the only supported value and the default, so passing it explicitly is the same as omitting it. Retained so callers that pass `auth_method="keypair"` keep working. `"password"` is accepted but ignored, with a logged warning naming the fix; any other value raises `ValueError`.
 - **_creation_mode_** : Either a `"CreationMode.X"` string or a `snowflake.ml.feature_store.CreationMode` member.
 - **_profiles_path_** : Override the dbt `profiles.yml` location. Only used on the local path.
 
+> **CI authentication** uses key-pair auth exclusively — see `snowflake_engine_factory` for the env var names and base64 encoding details.
+
 #### Returns
 - A `snowflake.ml.feature_store.FeatureStore`. The underlying Snowpark session is reachable as `fs._session` and is not closed by this function.
 

{gitlabds-2.3.1 → gitlabds-2.4.1}/README.md

@@ -340,9 +340,9 @@ test_df_transformed = gitlabds.apply_dummy(
 <details><summary> Remove Low Variation </summary>
 
 #### Description
-Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value.
+Remove columns from a dataset that do not meet the variation threshold. That is, columns will be dropped that contain a high percentage of one value. Columns that are entirely null are always dropped.
 
-`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True):`
+`gitlabds.remove_low_variation(df=None, dv=None, columns='all', threshold=.98, verbose=True, dropna=False):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -350,6 +350,7 @@ Remove columns from a dataset that do not meet the variation threshold. That is,
 - **_columns_** : Will examine at all columns by default. To limit to just a subset of columns, pass a list of column names. 
 - **_threshold_**: The maximum percentage one value in a column can represent. columns that exceed this threshold will be dropped. For example, the default value of `0.98` will drop any column where one value is present in more than 98% of rows.
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `False` (default) to count missing values as their own value. Set to `True` to measure variation over non-null values only, so a column that is one value plus NaN is dropped. Use `True` when missing values are left in place for a model that handles them natively (e.g. XGBoost, LightGBM).
 
 #### Returns
 - DataFrame with low variation columns dropped.
@@ -363,6 +364,10 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', columns='all',
 # Drop any of the selected columns where one value is present in more than 99% of rows.
 new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB', 'colC'], threshold=.99)
 ```
+```python
+# Ignore missing values: drop columns where one value makes up more than 98% of the non-null rows.
+new_df = gitlabds.remove_low_variation(df=my_df, dv='my_outcome', dropna=True)
+```
 </details>
 
 <details><summary> Correlation Reduction </summary>
@@ -370,7 +375,7 @@ new_df = gitlabds.remove_low_variation(df=my_df, dv=None, columns=['colA', 'colB
 #### Description
 Reduce the number of columns on a dataframe by dropping columns that are highly correlated with other columns. Note: only one of the two highly correlated columns will be dropped.
 
-`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True):`
+`gitlabds.correlation_reduction(df=None, dv=None, threshold=0.9, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -378,6 +383,8 @@ Reduce the number of columns on a dataframe by dropping columns that are highly
 - **_threshold_**: The threshold above which columns will be dropped. If two variables exceed this threshold, one will be dropped from the dataframe. For example, the default value of `0.90` will identify columns that have correlations greater than 90% to each other and drop one of those columns.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each pair of columns (and each column with the outcome) using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both columns of a pair are non-null for the pair to be evaluated. Pairs below it are never treated as correlated. Guards against spurious high correlations between sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with redundant correlated columns dropped.
@@ -398,7 +405,7 @@ new_df = gitlabds.correlation_reduction(df=my_df, dv=None, threshold=0.90, metho
 #### Description
 Remove columns that are highly correlated with the outcome (target) column.
 
-`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True):`
+`gitlabds.remove_outcome_proxies(df, dv, threshold=.8, method="pearson", verbose=True, dropna=True, min_periods=1):`
 
 #### Parameters:
 - _**df**_ : your pandas dataframe
@@ -406,6 +413,8 @@ Remove columns that are highly correlated with the outcome (target) column.
 - _**threshold**_ : The correlation value to the outcome above which columns will be dropped. For example, the default value of `0.80` will identify and drop columns that have correlations greater than 80% to the outcome.
 - **_method_**: The correlation method to use. Options are "pearson" (linear relationships), "spearman" (monotonic relationships), or "mutual_info" (any statistical dependency).
 - **_verbose_** : Set to `True` to print outputs of columns being dropped. Set to `False` to suppress.
+- **_dropna_** : Set to `True` (default) to compare each column to the outcome using only rows where both are non-null. Pearson and Spearman always work this way; for "mutual_info", `False` raises an error if any missing values are present.
+- **_min_periods_** : Minimum number of rows where both the column and the outcome are non-null for the column to be evaluated. Columns below it are never dropped. Guards against spurious high correlations from sparsely populated columns. "mutual_info" always requires at least 4 such rows.
 
 #### Returns
 - DataFrame with outcome proxy columns dropped.
@@ -419,6 +428,10 @@ new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.7
 # Drop columns with correlations to the outcome greater than 80% using Spearman correlation
 new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', threshold=.8, method="spearman")        
 ```
+```python
+# Data with missing values: only evaluate columns with at least 100 non-null rows alongside the outcome
+new_df = gitlabds.remove_outcome_proxies(df=my_df, dv='my_outcome', method="spearman", min_periods=100)
+```
 </details>
 
 <details><summary> Drop Categorical </summary>
@@ -1099,7 +1112,7 @@ feature_drift = results['feature_drift']
 
 #### Snowflake Connectors
 
-Thin wrappers over the native Snowflake SDKs that resolve GitLab's two credential paths — local (dbt profile + SSO) and GitLab CI (env vars + password or key pair) — and then hand off to the native objects. Migrated from `gitlabdata.orchestration_utils`; see the migration notes at the end of this section.
+Thin wrappers over the native Snowflake SDKs that resolve GitLab's two credential paths — local (dbt profile + SSO) and GitLab CI (env vars + key-pair auth) — and then hand off to the native objects. Migrated from `gitlabdata.orchestration_utils`; see the migration notes at the end of this section.
 
 All of these require `pip install gitlabds[snowflake]`.
 
@@ -1108,7 +1121,7 @@ All of these require `pip install gitlabds[snowflake]`.
 #### Description
 Open a Snowflake connection, resolving credentials from your dbt profile locally or from GitLab CI environment variables in CI.
 
-`gitlabds.snowflake_engine_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="dev", schema="", auth_method="password", profiles_path=None, **connect_kwargs):`
+`gitlabds.snowflake_engine_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="dev", schema="", auth_method="keypair", profiles_path=None, **connect_kwargs):`
 
 #### Parameters:
 - **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
@@ -1116,10 +1129,12 @@ Open a Snowflake connection, resolving credentials from your dbt profile locally
 - **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
 - **_profile_target_** : dbt profile output name. Only used on the local path.
 - **_schema_** : Schema to connect to. Falls back to the dbt profile's schema locally; when it resolves empty the connection is opened with no default schema.
-- **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path. Key pair secrets are not yet provisioned in CI.
+- **_auth_method_** : CI authentication method. `"keypair"` is the only supported value and the default, so passing it explicitly is the same as omitting it. Retained so callers that pass `auth_method="keypair"` keep working. `"password"` is accepted but ignored, with a logged warning naming the fix; any other value raises `ValueError`.
 - **_profiles_path_** : Override the dbt `profiles.yml` location (defaults to `~/.dbt/profiles.yml`). Only used on the local path.
 - **_\*\*connect_kwargs_** : Passed through to `snowflake.connector.connect()`, e.g. `autocommit=False`, `client_session_keep_alive=True`.
 
+> **CI authentication** uses key-pair auth exclusively (password-based CI auth was removed ahead of Snowflake's 2026-09-22 service-user password deprecation). In CI, reads `SNOWFLAKE_DATA_SCIENCE_LOAD_PRIVATE_KEY` (raw PEM or base64-encoded PEM — base64 is required for GitLab CI/CD variables since multi-line values cannot be masked) and optionally `SNOWFLAKE_DATA_SCIENCE_LOAD_PRIVATE_KEY_PASSPHRASE` (or the `_PASSWORD` variant) for encrypted keys.
+
 #### Returns
 - An open `snowflake.connector.SnowflakeConnection`. The caller owns its lifecycle.
 
@@ -1264,17 +1279,19 @@ gitlabds.snowflake_stage_load_copy_remove(
 #### Description
 Create a Snowflake Feature Store connection (Snowpark Session + native `FeatureStore`), resolving credentials the same way as `snowflake_engine_factory`.
 
-`gitlabds.feature_store_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="feature_store", auth_method="password", creation_mode="CreationMode.FAIL_IF_NOT_EXIST", profiles_path=None):`
+`gitlabds.feature_store_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="feature_store", auth_method="keypair", creation_mode="CreationMode.FAIL_IF_NOT_EXIST", profiles_path=None):`
 
 #### Parameters:
 - **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
 - **_role_** : Snowflake role for CI credentials. The feature store's database and schema come from `SNOWFLAKE_FEATURE_STORE_DATABASE` / `SNOWFLAKE_FEATURE_STORE_SCHEMA` on the CI path, not from the role's usual database/schema.
 - **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
 - **_profile_target_** : dbt profile output name. Only used on the local path.
-- **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path.
+- **_auth_method_** : CI authentication method. `"keypair"` is the only supported value and the default, so passing it explicitly is the same as omitting it. Retained so callers that pass `auth_method="keypair"` keep working. `"password"` is accepted but ignored, with a logged warning naming the fix; any other value raises `ValueError`.
 - **_creation_mode_** : Either a `"CreationMode.X"` string or a `snowflake.ml.feature_store.CreationMode` member.
 - **_profiles_path_** : Override the dbt `profiles.yml` location. Only used on the local path.
 
+> **CI authentication** uses key-pair auth exclusively — see `snowflake_engine_factory` for the env var names and base64 encoding details.
+
 #### Returns
 - A `snowflake.ml.feature_store.FeatureStore`. The underlying Snowpark session is reachable as `fs._session` and is not closed by this function.