gitlabds 2.1.8__tar.gz → 2.1.10__tar.gz

{gitlabds-2.1.8 → gitlabds-2.1.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gitlabds
-Version: 2.1.8
+Version: 2.1.10
 Summary: GitLab Data Science Tools
 Author-email: Kevin Dietz <kdietz@gitlab.com>
 License-Expression: MIT
@@ -31,7 +31,7 @@ Requires-Dist: snowflake-ml-python; extra == "feature-store"
 Dynamic: license-file
 
 [![PyPI version](https://badge.fury.io/py/gitlabds.svg)](https://badge.fury.io/py/gitlabds)
-[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 # gitlabds
@@ -148,7 +148,7 @@ test_df_transformed = gitlabds.apply_outliers(df=test_data, outlier_limits=outli
 #### Description
 Detect and optionally fill missing values in a DataFrame, with support for various filling methods and detailed reporting.
 
-`gitlabds.missing_values(df, threshold=0.0, method=None, columns="all", constant_value=None, verbose=True, operation="both")`
+`gitlabds.missing_values(df, threshold=0.0, method=None, columns="all", constant_value=None, groupby=None, verbose=True, operation="both")`
 
 #### Parameters:
 - **_df_** : Your pandas dataframe
@@ -163,6 +163,7 @@ Detect and optionally fill missing values in a DataFrame, with support for vario
   - "drop_row": Remove rows with any missing values in specified columns
 - **_columns_** : Columns to check and/or fill. If "all", processes all columns with missing values.
 - **_constant_value_** : Value to use when method="constant" or when specified columns use the constant method.
+- **_groupby_** : Column name or list of column names to group by when computing fill values. When provided, missing values are filled per-group rather than from the full-column statistic/distribution. A global fallback (computed across the full column) is used for any group value not seen during training. Only applies to `mean`, `median`, and `random` methods; ignored for other methods.
 - **_verbose_** : Whether to print detailed information about missing values and filling operations.
 - **_operation_** : Operation mode:
   - "check": Only check for missing values, don't fill
@@ -173,7 +174,7 @@ Detect and optionally fill missing values in a DataFrame, with support for vario
 - If operation="check": List of column names with missing values (or None)
 - If operation="fill" or "both": Tuple containing:
   - DataFrame with missing values handled
-  - Dictionary with missing value information that can be used with apply_missing_fill()
+  - Dictionary with missing value information that can be used with apply_missing_values()
     
 #### Examples:
 ```python
@@ -190,6 +191,20 @@ df_filled, missing_info = gitlabds.missing_values(
     constant_value="Unknown",
     verbose=True
 )
+
+# Fill using per-segment statistics (groupby) — works for mean, median, and random.
+# A global fallback (across the full column) is used for any group value not seen
+# during training.
+df_filled, missing_info = gitlabds.missing_values(
+    df,
+    columns=["lam", "employee_count"],
+    method="median",
+    groupby="sales_segment",
+    operation="fill"
+)
+
+# Group medians from training are reused when applied to new data via apply_missing_values()
+test_df_filled = gitlabds.apply_missing_values(test_df, missing_info)
 ```
 </details>
 
@@ -508,7 +523,7 @@ df_optimized = gitlabds.memory_optimization(
 #### Description
 This function splits your data into train and test datasets, separating the outcome from the rest of the file. It supports stratified sampling, balanced upsampling for imbalanced datasets, and provides model weights for compensating sampling adjustments.
 
-`gitlabds.split_data(df, train_pct=0.7, dv=None, dv_threshold=0.0, random_state=5435, stratify=True, sampling_strategy=None, shuffle=True, verbose=True):`
+`gitlabds.split_data(df, train_pct=0.7, dv=None, dv_threshold=0.0, random_state=5435, stratify=True, groupby=None, sampling_strategy=None, shuffle=True, verbose=True):`
 
 #### Parameters:
 - **_df_** : your pandas dataframe
@@ -516,7 +531,8 @@ This function splits your data into train and test datasets, separating the outc
 - **_dv_** : The column name of your outcome. If None, the function will return the entire dataframe split without separating features and target.
 - **_dv_threshold_** : The minimum percentage of rows that must contain a positive instance (i.e. > 0) of the outcome. SMOTE/SMOTE-NC will be used to upsample positive instances until this threshold is reached. Can be disabled by setting to 0. Only accepts values 0 to 0.5.
 - **_random_state_** : Random seed to use for splitting dataframe and for up-sampling (if needed).
-- **_stratify_** : Controls stratified sampling. If True and dv is provided, stratifies by the outcome variable. If a list of column names, stratifies by those columns. If False, does not use stratified sampling.
+- **_stratify_** : Controls stratified sampling. If True and dv is provided, stratifies by the outcome variable. If a list of column names, stratifies by those columns. If False, does not use stratified sampling. Note: a list is not supported when `groupby` is provided.
+- **_groupby_** : Column name OR index level name to use for group-aware splitting. When provided, all rows belonging to the same group are kept together in either train or test (never split across both) — useful when the dataset has repeated observations per entity (e.g., the same account at multiple snapshot dates) and naive row-level splits would leak entities across train/test. When `stratify=True`, stratification is computed at the group level using the max of `dv` per group ("ever-positive" vs. "never-positive" for binary targets; best-effort and ordinal-like for multi-class). Groups whose `dv` is entirely NaN are dropped with a warning. When `groupby` refers to a regular column (not an index level), that column is NOT dropped from the returned `x_train`/`x_test` — drop it manually if you don't want it as a feature.
 - **_sampling_strategy_** : Sampling strategy for imbalanced data. If None, will use dv_threshold. See imblearn documentation for more details on acceptable values.
 - **_shuffle_** : Whether to shuffle the data before splitting.
 - **_verbose_** : Whether to print information about the splitting process.
@@ -558,6 +574,17 @@ x_train, y_train, x_test, y_test, model_weights = gitlabds.split_data(
 )
 ```
 
+```python
+# Group-aware split: keep all snapshots of an account together in train OR test,
+# with stratification computed at the group level (ever-positive vs never-positive)
+x_train, y_train, x_test, y_test, model_weights = gitlabds.split_data(
+    df=my_df,
+    dv='my_outcome',
+    groupby='dim_crm_account_id',
+    stratify=True
+)
+```
+
 ```python
 # Split entire dataframe without separating target
 train_df, _, test_df, _, _ = gitlabds.split_data(
@@ -640,7 +667,7 @@ A comprehensive framework for evaluating machine learning models, supporting bot
 - _**x_oot**_ : Optional out-of-time validation features.
 - _**y_oot**_ : Optional out-of-time validation labels.
 - _**classification**_ : Whether this is a classification model. If False, regression metrics will be used.
-- _**algo**_ : Algorithm type for feature importance calculation. Options: 'xgb', 'rf', 'mars'. For other algorithms, use `None`
+- _**algo**_ : Algorithm type for feature importance calculation. Options: 'xgb', 'lgb', 'rf', 'mars'. For other algorithms, use `None`
 - _**f1_threshold**_ : Threshold for binary classification.
 - _**decile_n**_ : Number of n-tiles for lift calculation. Defaults to 10 for deciles
 - _**top_features_n**_ : Number of top features to display in visualizations.
@@ -943,7 +970,7 @@ feature_drift = results['feature_drift']
 #### Description
 Serve features from Snowflake Feature Store with built-in distributed locking and retry logic. This function orchestrates feature retrieval from multiple feature views with support for point-in-time lookups and flexible parameter customization.
 
-`gitlabds.serve_features(session, feature_store, feature_views_dict, spine_df=None, feature_date=None, spine_timestamp_col=None, include_feature_view_timestamp_col=False, lookback_window_value=None, lookback_window_unit=None, lock_timeout_seconds=300):`
+`gitlabds.serve_features(session, feature_store, feature_views_dict, spine_df=None, feature_date=None, spine_timestamp_col=None, include_feature_view_timestamp_col=False, lookback_window_value=None, lookback_window_unit=None, lock_timeout_seconds=300, max_fetch_retries=3, fetch_retry_backoff_seconds=30):`
 
 #### Parameters:
 - **_session_** : Active Snowflake Session object
@@ -959,6 +986,8 @@ Serve features from Snowflake Feature Store with built-in distributed locking an
 - **_lookback_window_value_** : Either a global integer value (e.g., 6) or a dict with feature-view-specific values
 - **_lookback_window_unit_** : Either a global string value (e.g., 'months') or a dict with feature-view-specific values
 - **_lock_timeout_seconds_** : Timeout in seconds for acquiring the Snowflake lock. Default is 300.
+- **_max_fetch_retries_** : Maximum number of attempts for the result-fetch step (`retrieve_feature_values` + `to_arrow().to_pandas()`) when a transient network/streaming error occurs (e.g., `ChunkedEncodingError`, "Response ended prematurely"). Set to 1 to disable retries. Default is 3.
+- **_fetch_retry_backoff_seconds_** : Base sleep duration between fetch retries. Sleep grows linearly with attempt number (`attempt * backoff`). With defaults (3 retries, 30s base), sleeps are 30s before attempt 2 and 60s before attempt 3. Default is 30.
 
 #### Returns
 - DataFrame with combined features from all feature views

{gitlabds-2.1.8 → gitlabds-2.1.10}/README.md

@@ -1,5 +1,5 @@
 [![PyPI version](https://badge.fury.io/py/gitlabds.svg)](https://badge.fury.io/py/gitlabds)
-[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
 # gitlabds
@@ -116,7 +116,7 @@ test_df_transformed = gitlabds.apply_outliers(df=test_data, outlier_limits=outli
 #### Description
 Detect and optionally fill missing values in a DataFrame, with support for various filling methods and detailed reporting.
 
-`gitlabds.missing_values(df, threshold=0.0, method=None, columns="all", constant_value=None, verbose=True, operation="both")`
+`gitlabds.missing_values(df, threshold=0.0, method=None, columns="all", constant_value=None, groupby=None, verbose=True, operation="both")`
 
 #### Parameters:
 - **_df_** : Your pandas dataframe
@@ -131,6 +131,7 @@ Detect and optionally fill missing values in a DataFrame, with support for vario
   - "drop_row": Remove rows with any missing values in specified columns
 - **_columns_** : Columns to check and/or fill. If "all", processes all columns with missing values.
 - **_constant_value_** : Value to use when method="constant" or when specified columns use the constant method.
+- **_groupby_** : Column name or list of column names to group by when computing fill values. When provided, missing values are filled per-group rather than from the full-column statistic/distribution. A global fallback (computed across the full column) is used for any group value not seen during training. Only applies to `mean`, `median`, and `random` methods; ignored for other methods.
 - **_verbose_** : Whether to print detailed information about missing values and filling operations.
 - **_operation_** : Operation mode:
   - "check": Only check for missing values, don't fill
@@ -141,7 +142,7 @@ Detect and optionally fill missing values in a DataFrame, with support for vario
 - If operation="check": List of column names with missing values (or None)
 - If operation="fill" or "both": Tuple containing:
   - DataFrame with missing values handled
-  - Dictionary with missing value information that can be used with apply_missing_fill()
+  - Dictionary with missing value information that can be used with apply_missing_values()
     
 #### Examples:
 ```python
@@ -158,6 +159,20 @@ df_filled, missing_info = gitlabds.missing_values(
     constant_value="Unknown",
     verbose=True
 )
+
+# Fill using per-segment statistics (groupby) — works for mean, median, and random.
+# A global fallback (across the full column) is used for any group value not seen
+# during training.
+df_filled, missing_info = gitlabds.missing_values(
+    df,
+    columns=["lam", "employee_count"],
+    method="median",
+    groupby="sales_segment",
+    operation="fill"
+)
+
+# Group medians from training are reused when applied to new data via apply_missing_values()
+test_df_filled = gitlabds.apply_missing_values(test_df, missing_info)
 ```
 </details>
 
@@ -476,7 +491,7 @@ df_optimized = gitlabds.memory_optimization(
 #### Description
 This function splits your data into train and test datasets, separating the outcome from the rest of the file. It supports stratified sampling, balanced upsampling for imbalanced datasets, and provides model weights for compensating sampling adjustments.
 
-`gitlabds.split_data(df, train_pct=0.7, dv=None, dv_threshold=0.0, random_state=5435, stratify=True, sampling_strategy=None, shuffle=True, verbose=True):`
+`gitlabds.split_data(df, train_pct=0.7, dv=None, dv_threshold=0.0, random_state=5435, stratify=True, groupby=None, sampling_strategy=None, shuffle=True, verbose=True):`
 
 #### Parameters:
 - **_df_** : your pandas dataframe
@@ -484,7 +499,8 @@ This function splits your data into train and test datasets, separating the outc
 - **_dv_** : The column name of your outcome. If None, the function will return the entire dataframe split without separating features and target.
 - **_dv_threshold_** : The minimum percentage of rows that must contain a positive instance (i.e. > 0) of the outcome. SMOTE/SMOTE-NC will be used to upsample positive instances until this threshold is reached. Can be disabled by setting to 0. Only accepts values 0 to 0.5.
 - **_random_state_** : Random seed to use for splitting dataframe and for up-sampling (if needed).
-- **_stratify_** : Controls stratified sampling. If True and dv is provided, stratifies by the outcome variable. If a list of column names, stratifies by those columns. If False, does not use stratified sampling.
+- **_stratify_** : Controls stratified sampling. If True and dv is provided, stratifies by the outcome variable. If a list of column names, stratifies by those columns. If False, does not use stratified sampling. Note: a list is not supported when `groupby` is provided.
+- **_groupby_** : Column name OR index level name to use for group-aware splitting. When provided, all rows belonging to the same group are kept together in either train or test (never split across both) — useful when the dataset has repeated observations per entity (e.g., the same account at multiple snapshot dates) and naive row-level splits would leak entities across train/test. When `stratify=True`, stratification is computed at the group level using the max of `dv` per group ("ever-positive" vs. "never-positive" for binary targets; best-effort and ordinal-like for multi-class). Groups whose `dv` is entirely NaN are dropped with a warning. When `groupby` refers to a regular column (not an index level), that column is NOT dropped from the returned `x_train`/`x_test` — drop it manually if you don't want it as a feature.
 - **_sampling_strategy_** : Sampling strategy for imbalanced data. If None, will use dv_threshold. See imblearn documentation for more details on acceptable values.
 - **_shuffle_** : Whether to shuffle the data before splitting.
 - **_verbose_** : Whether to print information about the splitting process.
@@ -526,6 +542,17 @@ x_train, y_train, x_test, y_test, model_weights = gitlabds.split_data(
 )
 ```
 
+```python
+# Group-aware split: keep all snapshots of an account together in train OR test,
+# with stratification computed at the group level (ever-positive vs never-positive)
+x_train, y_train, x_test, y_test, model_weights = gitlabds.split_data(
+    df=my_df,
+    dv='my_outcome',
+    groupby='dim_crm_account_id',
+    stratify=True
+)
+```
+
 ```python
 # Split entire dataframe without separating target
 train_df, _, test_df, _, _ = gitlabds.split_data(
@@ -608,7 +635,7 @@ A comprehensive framework for evaluating machine learning models, supporting bot
 - _**x_oot**_ : Optional out-of-time validation features.
 - _**y_oot**_ : Optional out-of-time validation labels.
 - _**classification**_ : Whether this is a classification model. If False, regression metrics will be used.
-- _**algo**_ : Algorithm type for feature importance calculation. Options: 'xgb', 'rf', 'mars'. For other algorithms, use `None`
+- _**algo**_ : Algorithm type for feature importance calculation. Options: 'xgb', 'lgb', 'rf', 'mars'. For other algorithms, use `None`
 - _**f1_threshold**_ : Threshold for binary classification.
 - _**decile_n**_ : Number of n-tiles for lift calculation. Defaults to 10 for deciles
 - _**top_features_n**_ : Number of top features to display in visualizations.
@@ -911,7 +938,7 @@ feature_drift = results['feature_drift']
 #### Description
 Serve features from Snowflake Feature Store with built-in distributed locking and retry logic. This function orchestrates feature retrieval from multiple feature views with support for point-in-time lookups and flexible parameter customization.
 
-`gitlabds.serve_features(session, feature_store, feature_views_dict, spine_df=None, feature_date=None, spine_timestamp_col=None, include_feature_view_timestamp_col=False, lookback_window_value=None, lookback_window_unit=None, lock_timeout_seconds=300):`
+`gitlabds.serve_features(session, feature_store, feature_views_dict, spine_df=None, feature_date=None, spine_timestamp_col=None, include_feature_view_timestamp_col=False, lookback_window_value=None, lookback_window_unit=None, lock_timeout_seconds=300, max_fetch_retries=3, fetch_retry_backoff_seconds=30):`
 
 #### Parameters:
 - **_session_** : Active Snowflake Session object
@@ -927,6 +954,8 @@ Serve features from Snowflake Feature Store with built-in distributed locking an
 - **_lookback_window_value_** : Either a global integer value (e.g., 6) or a dict with feature-view-specific values
 - **_lookback_window_unit_** : Either a global string value (e.g., 'months') or a dict with feature-view-specific values
 - **_lock_timeout_seconds_** : Timeout in seconds for acquiring the Snowflake lock. Default is 300.
+- **_max_fetch_retries_** : Maximum number of attempts for the result-fetch step (`retrieve_feature_values` + `to_arrow().to_pandas()`) when a transient network/streaming error occurs (e.g., `ChunkedEncodingError`, "Response ended prematurely"). Set to 1 to disable retries. Default is 3.
+- **_fetch_retry_backoff_seconds_** : Base sleep duration between fetch retries. Sleep grows linearly with attempt number (`attempt * backoff`). With defaults (3 retries, 30s base), sleeps are 30s before attempt 2 and 60s before attempt 3. Default is 30.
 
 #### Returns
 - DataFrame with combined features from all feature views

{gitlabds-2.1.8 → gitlabds-2.1.10}/gitlabds/memory_optimization.py

@@ -809,11 +809,9 @@ def memory_optimization(
                     if sparsity > sparse_threshold:
                         try:
                             if isinstance(df_result, pd.Series):
-                                df_result = pd.Series.sparse.from_dense(
-                                    df_result, fill_value=fill_value
-                                )
-                            else:
-                                df_result[col] = pd.Series.sparse.from_dense(
+
+                                df_result = pd.arrays.SparseArray(df_result, fill_value=fill_value)
+                                df_result[col] = pd.arrays.SparseArray(
                                     df_result[col], fill_value=fill_value
                                 )