gitlabds 2.2.2__tar.gz → 2.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. {gitlabds-2.2.2/gitlabds.egg-info → gitlabds-2.3.0}/PKG-INFO +271 -22
  2. gitlabds-2.2.2/PKG-INFO → gitlabds-2.3.0/README.md +264 -50
  3. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/__init__.py +17 -7
  4. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/feature_descriptions.py +3 -22
  5. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/record_benchmarking.py +234 -53
  6. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/serving_features.py +4 -26
  7. gitlabds-2.3.0/gitlabds/snowflake_connectors.py +450 -0
  8. gitlabds-2.2.2/README.md → gitlabds-2.3.0/gitlabds.egg-info/PKG-INFO +299 -18
  9. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds.egg-info/SOURCES.txt +3 -1
  10. gitlabds-2.3.0/gitlabds.egg-info/requires.txt +16 -0
  11. {gitlabds-2.2.2 → gitlabds-2.3.0}/pyproject.toml +6 -3
  12. gitlabds-2.3.0/tests/test_snowflake_connectors.py +769 -0
  13. gitlabds-2.2.2/gitlabds.egg-info/requires.txt +0 -13
  14. {gitlabds-2.2.2 → gitlabds-2.3.0}/LICENSE +0 -0
  15. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/baselines.py +0 -0
  16. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/config_generator.py +0 -0
  17. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/dummy.py +0 -0
  18. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/feature_reduction.py +0 -0
  19. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/insights.py +0 -0
  20. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/memory_optimization.py +0 -0
  21. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/missing.py +0 -0
  22. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/model_evaluator.py +0 -0
  23. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/monitoring_metrics.py +0 -0
  24. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/outliers.py +0 -0
  25. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/split_data.py +0 -0
  26. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/trends.py +0 -0
  27. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds.egg-info/dependency_links.txt +0 -0
  28. {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds.egg-info/top_level.txt +0 -0
  29. {gitlabds-2.2.2 → gitlabds-2.3.0}/setup.cfg +0 -0
  30. {gitlabds-2.2.2 → gitlabds-2.3.0}/setup.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: gitlabds
3
- Version: 2.2.2
3
+ Version: 2.3.0
4
4
  Summary: GitLab Data Science Tools
5
5
  Author-email: Kevin Dietz <kdietz@gitlab.com>
6
6
  License-Expression: MIT
@@ -25,9 +25,12 @@ Requires-Dist: seaborn>=0.13.2
25
25
  Requires-Dist: shap>=0.50.0
26
26
  Requires-Dist: tqdm>=4.66.2
27
27
  Requires-Dist: xgboost>=1.6.1
28
- Provides-Extra: feature-store
29
- Requires-Dist: snowflake-snowpark-python; extra == "feature-store"
30
- Requires-Dist: snowflake-ml-python; extra == "feature-store"
28
+ Requires-Dist: pyyaml>=6.0.0
29
+ Provides-Extra: snowflake
30
+ Requires-Dist: cryptography>=41.0.0; extra == "snowflake"
31
+ Requires-Dist: snowflake-connector-python[pandas]>=3.0.0; extra == "snowflake"
32
+ Requires-Dist: snowflake-snowpark-python>=1.37.0; extra == "snowflake"
33
+ Requires-Dist: snowflake-ml-python>=1.11.0; extra == "snowflake"
31
34
  Dynamic: license-file
32
35
 
33
36
  [![PyPI version](https://badge.fury.io/py/gitlabds.svg)](https://badge.fury.io/py/gitlabds)
@@ -61,19 +64,23 @@ pip install gitlabds
61
64
 
62
65
  ### Optional Dependencies
63
66
 
64
- #### Snowflake Feature Store
67
+ #### Snowflake
65
68
 
66
- The `serve_features`, `clear_feature_serving_locks`, and `get_feature_descriptions` functions require additional Snowflake dependencies.
69
+ The Snowflake connectors (`snowflake_engine_factory`, `ds_query_dataframe`, `query_executor`, `snowflake_stage_load_copy_remove`, `feature_store_factory`) and the feature serving functions (`serve_features`, `clear_feature_serving_locks`, `get_feature_descriptions`) require additional dependencies.
67
70
 
68
71
  To use these functions, install with:
69
72
 
70
73
  ```bash
71
- pip install gitlabds[feature-store]
74
+ pip install gitlabds[snowflake]
72
75
  ```
73
76
 
74
77
  This will install:
78
+ - `snowflake-connector-python[pandas]`
75
79
  - `snowflake-snowpark-python`
76
80
  - `snowflake-ml-python`
81
+ - `cryptography`
82
+
83
+ These are optional rather than core because `snowflake-ml-python` constrains the numpy/scikit-learn/xgboost stack that the core package also depends on.
77
84
 
78
85
  If you try to use these functions without installing the optional dependencies, you'll get an error message with installation instructions.
79
86
 
@@ -827,10 +834,10 @@ Status label for a value vs. its peer median (+/- tolerance), not percentile ran
827
834
  #### Parameters:
828
835
  - _**value**_ : value(s) to evaluate, scalar or array-like
829
836
  - _**peer_median**_ : peer group median(s) to compare against
830
- - _**good_direction**_ : `"decreasing"` flips which side of the median counts as "above" (e.g., for a metric where lower is better). Anything else is treated as increasing/default.
837
+ - _**good_direction**_ : one of `"increasing"`, `"decreasing"`, `"both"`, `"none"` (scalar or array-like). `"decreasing"` flips which side of the median counts as "above" (e.g., for a metric where lower is better). `"both"` and `"none"` mean the feature has no single good direction, so the output is positional only: no flip, and the `ceiling_label` upgrade is suppressed. `None` (the default) is treated as `"none"`. Any other value raises — a blank/NaN in a feature dictionary must be normalized to `"none"` first rather than silently defaulting to increasing.
831
838
  - _**tolerance**_ : relative distance from the peer median (as a fraction of the median) required to be classified above/below average rather than "average". Default is 0.15 (a value must be >15% above/below its peer median to leave the "average" band)
832
839
  - _**no_activity_value**_ : value that should always map to `labels[3]` ("No activity") regardless of tolerance
833
- - _**peer_percentile**_ : optional 0-1 percentile rank within the peer group. When provided, an "above average" reading that also sits in the top `ceiling_pct` toward `good_direction` is upgraded to `ceiling_label`
840
+ - _**peer_percentile**_ : optional 0-1 percentile rank within the peer group. When provided, an "above average" reading that also sits in the top `ceiling_pct` toward `good_direction` is upgraded to `ceiling_label`. Only applies when `good_direction` is `"increasing"` or `"decreasing"`
834
841
  - _**ceiling_pct**_ : percentile threshold used with `peer_percentile` to detect a "tapped out" strength. Default is 0.95
835
842
  - _**ceiling_label**_ : label applied when the ceiling condition is met. Default is "near ceiling"
836
843
  - _**labels**_ : tuple of (below average, average, above average, no activity) labels
@@ -859,7 +866,7 @@ Per-record SHAP attribution (log-odds / margin space) for a tree model. Uses tre
859
866
  `gitlabds.per_record_shap(model, df, feature_dict, unique_id="record_id", top_n=25, min_actionable=3, actionable_values=["Yes", "Partial"]):`
860
867
 
861
868
  #### Parameters:
862
- - _**model**_ : trained tree-based model (e.g., LightGBM, XGBoost)
869
+ - _**model**_ : trained tree-based model (e.g., LightGBM, XGBoost). A raw `xgboost.Booster` (from `xgb.train`, or loaded back from a saved model file) works as well as the sklearn wrapper
863
870
  - _**df**_ : model-ready feature matrix, indexed by `unique_id`
864
871
  - _**feature_dict**_ : DataFrame or path to CSV with `field` and `actionable` columns
865
872
  - _**unique_id**_ : name to use for the record identifier column in the output. Default is "record_id"
@@ -870,6 +877,8 @@ Per-record SHAP attribution (log-odds / margin space) for a tree model. Uses tre
870
877
  #### Returns
871
878
  - Long DataFrame, one row per (record, feature): `unique_id`, `rank`, `feature`, `contribution`, `direction`, `feature_value`, `actionable`
872
879
 
880
+ `direction` ("increases"/"decreases") describes the effect on the model's predicted probability of class 1, not on the business outcome. Where class 1 is a bad outcome (e.g. churn), "increases" means worse — any narrative or scorecard layer reading this column has to apply that polarity itself.
881
+
873
882
  #### Requirements
874
883
  This function requires the `shap` package (included in core dependencies).
875
884
 
@@ -889,27 +898,29 @@ shap_df = gitlabds.per_record_shap(
889
898
  <details><summary> Per-Record Counterfactuals </summary>
890
899
 
891
900
  #### Description
892
- "Change X -> +Y" counterfactual search, direction chosen empirically. For each actionable lever SHAP surfaced for a record, evaluates candidate values drawn from peer-group quantiles, re-scores the model at each, and keeps the single move that maximizes the gain in predicted probability. No monotonicity assumption is made.
901
+ "Change X -> +Y" counterfactual search, direction chosen empirically. For each actionable lever SHAP surfaced for a record, evaluates candidate values drawn from peer-group quantiles, re-scores the model at each, and keeps the single move that maximizes `gain` — the improvement in the model score, signed by `score_goal` so that bigger is always better. No monotonicity assumption is made.
893
902
 
894
- `gitlabds.per_record_counterfactuals(model, df, shap_df, model_algo="lgb", unique_id="record_id", peer_group=None, candidate_quantiles=[0.1, 0.25, 0.5, 0.75, 0.9], direction=None, actionable_values=["Yes", "Partial"], min_delta_p=0.005, impact_medium=0.01, impact_high=0.03, levels={"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}, n_bins=5):`
903
+ `gitlabds.per_record_counterfactuals(model, df, shap_df, unique_id="record_id", peer_group=None, candidate_quantiles=[0.1, 0.25, 0.5, 0.75, 0.9], direction=None, score_goal="maximize", actionable_values=["Yes", "Partial"], min_gain=0.005, impact_medium=0.01, impact_high=0.03, levels={"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}, n_bins=5):`
895
904
 
896
905
  #### Parameters:
897
- - _**model**_ : trained model (as in `per_record_shap`)
906
+ - _**model**_ : trained model (as in `per_record_shap`). How probabilities are scored is chosen from the model object itself — `.predict_proba()` where available, otherwise a raw `xgboost.Booster` or `lightgbm.Booster` is scored natively. An unsupported type raises a `TypeError` naming it
898
907
  - _**df**_ : model-ready feature matrix, indexed by `unique_id`
899
- - _**shap_df**_ : output of `per_record_shap`; scopes which (record, feature) levers to test
900
- - _**model_algo**_ : `"lgb"` or `"xgb"`/`"xgboost"`; controls how probabilities are scored
908
+ - _**shap_df**_ : output of `per_record_shap`; scopes which (record, feature) levers to test. Only numeric non-bool levers are evaluated — candidate values are peer-group quantiles, which need an ordered numeric scale. Categoricals, bools, and strings are skipped with a `UserWarning`. Marking them `'none'` in `direction` is the explicit way to express the same thing, and suppresses the warning
901
909
  - _**unique_id**_ : name of the record identifier column. Default is "record_id"
902
910
  - _**peer_group**_ : Series/DataFrame indexed by `unique_id`; numeric columns are auto-binned into `n_bins` quantile bands. `None` uses one global group
903
911
  - _**candidate_quantiles**_ : peer-group quantiles evaluated as candidate values
904
- - _**direction**_ : optional per-feature achievability constraint mapping feature -> `'increasing'` | `'decreasing'` | `'both'` (dict or Series)
912
+ - _**direction**_ : optional per-feature constraint mapping feature -> `'increasing'` | `'decreasing'` | `'both'` | `'none'` (dict or Series). `'increasing'`/`'decreasing'` restrict candidates to moves in that direction (an achievability constraint); `'both'` allows either; `'none'` excludes the feature from lever evaluation entirely, for features with no meaningful good direction (categoricals, identifiers, context-only numerics). Features absent from the mapping, or `direction=None`, default to `'both'`. A blank/NaN raises — normalize it to `'none'` where the feature dictionary is read
913
+ - _**score_goal**_ : `"maximize"` if a higher predicted probability of class 1 is the good outcome (e.g. class 1 = uptier), `"minimize"` if it is the bad one (e.g. class 1 = churn). Sets the sign of `gain`, which drives best-move selection, the `min_gain` filter, `impact_bucket`, and the output ordering. This is the outcome's polarity only — `direction` above is a separate, feature-level concept and is unaffected. Default is `"maximize"`
905
914
  - _**actionable_values**_ : values in `shap_df["actionable"]` treated as actionable levers
906
- - _**min_delta_p**_ : drop levers whose best achievable gain is below this
907
- - _**impact_medium, impact_high**_ : delta_p thresholds bucketing each lever into Low/Medium/High `impact_bucket`
908
- - _**levels**_ : dict mapping label -> upper bound on `p_before`/`p_after` for that level, ordered worst to best (e.g., default `{"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}` means `p <= 0.03` -> "poor fit", `0.03 < p <= 0.10` -> "good fit", `p > 0.10` -> "best fit"). Key order doesn't matter (sorted by cutoff), but the top level's cutoff must be `float("inf")`. Used to detect whether a move crosses a level boundary (any crossing move is floored at "Medium" impact). Pass your own dict (e.g., `{"low": 0.03, "medium": 0.10, "high": float("inf")}`) to match your own pipeline
915
+ - _**min_gain**_ : drop levers whose best achievable gain is below this. Note this is a threshold on `gain`, not on raw `delta_p`: under `score_goal="minimize"` a good move has a negative `delta_p` but a positive `gain`, so this stays positive for both polarities. Negative values (or `-inf`) are allowed, and keep moves that make the outcome worse — useful for inspecting the full candidate distribution while tuning thresholds
916
+ - _**impact_medium, impact_high**_ : `gain` thresholds bucketing each lever into Low/Medium/High `impact_bucket`. Only meaningful for `gain > 0` — under a negative `min_gain` the retained harmful moves stay `"Low"` (the level-crossing floor doesn't apply to them), so read `gain`'s sign rather than the bucket alone
917
+ - _**levels**_ : dict mapping label -> upper bound on P(class 1) for that level (e.g., default `{"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}` means `p <= 0.03` -> "poor fit", `0.03 < p <= 0.10` -> "good fit", `p > 0.10` -> "best fit"). Key order doesn't matter — levels are sorted by cutoff and labels follow that order — but the top level's cutoff must be `float("inf")`. Cutoffs are always in P(class 1) space regardless of `score_goal`, and `crosses_level` is a plain inequality, so name the labels for your own outcome: a churn model would pass something like `{"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")}`. Used to detect whether a move crosses a level boundary (any crossing move is floored at "Medium" impact)
909
918
  - _**n_bins**_ : quantile bins for numeric `peer_group` columns
910
919
 
911
920
  #### Returns
912
- - Long DataFrame, one row per (record, lever): `current_value`, `intervention_value`, `intervention_quantile`, `p_before`, `p_after`, `delta_p`, `impact_bucket`, `level_before`, `level_after`, `crosses_level`
921
+ - Long DataFrame, one row per (record, lever): `current_value`, `intervention_value`, `intervention_quantile`, `p_before`, `p_after`, `delta_p`, `gain`, `impact_bucket`, `level_before`, `level_after`, `crosses_level`
922
+
923
+ `delta_p` is the raw signed change in P(class 1); `gain` is that change signed by `score_goal` so bigger is always better. For a `score_goal="minimize"` model, `delta_p` will be negative and `gain` positive.
913
924
 
914
925
  #### Requirements
915
926
  For XGBoost models without `predict_proba`, requires the `xgboost` package (included in core dependencies).
@@ -924,6 +935,17 @@ counterfactuals_df = gitlabds.per_record_counterfactuals(
924
935
  shap_df=shap_df,
925
936
  peer_group=peer_group_df,
926
937
  )
938
+
939
+ # Churn model: class 1 = churn, so a lower score is the good outcome.
940
+ counterfactuals_df = gitlabds.per_record_counterfactuals(
941
+ model=churn_model,
942
+ df=model_features,
943
+ shap_df=shap_df,
944
+ peer_group=peer_group_df,
945
+ score_goal="minimize",
946
+ direction=dict(zip(feature_dict["field"], feature_dict["good_direction"])),
947
+ levels={"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")},
948
+ )
927
949
  ```
928
950
 
929
951
  </details>
@@ -944,6 +966,8 @@ Peer benchmarking. Each record's percentile for each feature within its peer gro
944
966
  - _**features**_ : optional explicit list of features to benchmark (used as-is, no actionable filter). If `None` (default), computed for every actionable feature in `df`
945
967
  - _**n_bins**_ : quantile bins for numeric `peer_group` columns
946
968
 
969
+ Either way, only numeric non-bool columns are benchmarked — percentile ranks and quantiles need an ordered numeric scale. Categoricals, bools, and strings are skipped with a `UserWarning` naming them.
970
+
947
971
  #### Returns
948
972
  - Long DataFrame, one row per (record, feature): `unique_id`, `feature`, `record_value`, `peer_group`, `percentile`, `peer_p25`, `peer_p50`, `peer_p75`
949
973
 
@@ -1108,6 +1132,231 @@ feature_drift = results['feature_drift']
1108
1132
 
1109
1133
  ### Deployment
1110
1134
 
1135
+ #### Snowflake Connectors
1136
+
1137
+ Thin wrappers over the native Snowflake SDKs that resolve GitLab's two credential paths — local (dbt profile + SSO) and GitLab CI (env vars + password or key pair) — and then hand off to the native objects. Migrated from `gitlabdata.orchestration_utils`; see the migration notes at the end of this section.
1138
+
1139
+ All of these require `pip install gitlabds[snowflake]`.
1140
+
1141
+ <details><summary> Snowflake Engine Factory </summary>
1142
+
1143
+ #### Description
1144
+ Open a Snowflake connection, resolving credentials from your dbt profile locally or from GitLab CI environment variables in CI.
1145
+
1146
+ `gitlabds.snowflake_engine_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="dev", schema="", auth_method="password", profiles_path=None, **connect_kwargs):`
1147
+
1148
+ #### Parameters:
1149
+ - **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
1150
+ - **_role_** : CI credential set to use. Currently only supports `DATA_SCIENCE_LOADER`
1151
+ - **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
1152
+ - **_profile_target_** : dbt profile output name. Only used on the local path.
1153
+ - **_schema_** : Schema to connect to. Falls back to the dbt profile's schema locally; when it resolves empty the connection is opened with no default schema.
1154
+ - **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path. Key pair secrets are not yet provisioned in CI.
1155
+ - **_profiles_path_** : Override the dbt `profiles.yml` location (defaults to `~/.dbt/profiles.yml`). Only used on the local path.
1156
+ - **_\*\*connect_kwargs_** : Passed through to `snowflake.connector.connect()`, e.g. `autocommit=False`, `client_session_keep_alive=True`.
1157
+
1158
+ #### Returns
1159
+ - An open `snowflake.connector.SnowflakeConnection`. The caller owns its lifecycle.
1160
+
1161
+ #### Requirements
1162
+ ```bash
1163
+ pip install gitlabds[snowflake]
1164
+ ```
1165
+
1166
+ #### Examples:
1167
+
1168
+ ```python
1169
+ import gitlabds
1170
+
1171
+ # Local: reads the "dev" target from ~/.dbt/profiles.yml and authenticates via SSO
1172
+ conn = gitlabds.snowflake_engine_factory()
1173
+ try:
1174
+ df = gitlabds.ds_query_dataframe(conn, "select * from prod.data_science.my_table")
1175
+ finally:
1176
+ conn.close()
1177
+ ```
1178
+ ```python
1179
+ # CI: credentials come from os.environ; run_target is inferred from GITLAB_CI
1180
+ conn = gitlabds.snowflake_engine_factory(role="DATA_SCIENCE_LOADER", schema="data_science")
1181
+
1182
+ # Opt out of autocommit to group statements into one transaction
1183
+ conn = gitlabds.snowflake_engine_factory(autocommit=False)
1184
+ ```
1185
+ </details>
1186
+
1187
+ <details><summary> DS Query Dataframe </summary>
1188
+
1189
+ #### Description
1190
+ Run a SELECT and return a pandas DataFrame, using Snowflake's Arrow-based fetch. Prefer this over `query_executor` for large reads such as pulling training data.
1191
+
1192
+ `gitlabds.ds_query_dataframe(conn, query, optimize_memory=False, verbose=False):`
1193
+
1194
+ #### Parameters:
1195
+ - **_conn_** : An open connection, e.g. from `snowflake_engine_factory()`
1196
+ - **_query_** : SELECT query to run
1197
+ - **_optimize_memory_** : Run the result through `reduce_memory_usage()` (dtype downcasting) before returning. Off by default because it is a lossy transform (e.g. float64 → float32).
1198
+ - **_verbose_** : Print the memory-optimization report and progress bar. Only applies when `optimize_memory=True`.
1199
+
1200
+ #### Returns
1201
+ - A pandas DataFrame
1202
+
1203
+ #### Requirements
1204
+ ```bash
1205
+ pip install gitlabds[snowflake]
1206
+ ```
1207
+
1208
+ #### Examples:
1209
+
1210
+ ```python
1211
+ df = gitlabds.ds_query_dataframe(conn, "select * from my_table")
1212
+
1213
+ # Large pull: downcast dtypes on the way out
1214
+ df = gitlabds.ds_query_dataframe(conn, training_query, optimize_memory=True, verbose=True)
1215
+ ```
1216
+ </details>
1217
+
1218
+ <details><summary> Query Executor </summary>
1219
+
1220
+ #### Description
1221
+ Execute a SQL statement and return its rows as a list of dicts keyed by column name, as Snowflake returns them. Use for DML/DDL/transaction-control statements, or reads where a DataFrame isn't needed.
1222
+
1223
+ `gitlabds.query_executor(conn, query):`
1224
+
1225
+ #### Parameters:
1226
+ - **_conn_** : An open connection, e.g. from `snowflake_engine_factory()`
1227
+ - **_query_** : SQL statement to run
1228
+
1229
+ #### Returns
1230
+ - `List[Dict[str, Any]]` — rows as `{column: value}`. DML and DDL statements return Snowflake's own status row (e.g. `{"number of rows inserted": 12}`) rather than an empty list.
1231
+
1232
+ #### Requirements
1233
+ ```bash
1234
+ pip install gitlabds[snowflake]
1235
+ ```
1236
+
1237
+ #### Examples:
1238
+
1239
+ ```python
1240
+ rows = gitlabds.query_executor(conn, "select count(*) as n from my_table")
1241
+ print(rows[0]["N"])
1242
+ ```
1243
+ ```python
1244
+ # Connections autocommit by default, so group statements explicitly
1245
+ gitlabds.query_executor(conn, "begin;")
1246
+ gitlabds.query_executor(conn, "delete from my_table where run_date = '2026-01-01';")
1247
+ gitlabds.query_executor(conn, "commit;")
1248
+ ```
1249
+ </details>
1250
+
1251
+ <details><summary> Snowflake Stage Load Copy Remove </summary>
1252
+
1253
+ #### Description
1254
+ Load a local file into a Snowflake table via an internal stage: remove any leftover staged file, PUT the file, COPY INTO the target table, then REMOVE the staged file.
1255
+
1256
+ `gitlabds.snowflake_stage_load_copy_remove(file, stage, table_path, conn, type="json", on_error="abort_statement", file_format_options="", col_names=""):`
1257
+
1258
+ #### Parameters:
1259
+ - **_conn_** : An open connection, e.g. from `snowflake_engine_factory()`
1260
+ - **_file_** : Path to the local file to upload
1261
+ - **_stage_** : Internal stage name **without** a leading `@` — the function adds it (e.g. `"my_stage"` or `"db.schema.my_stage"`)
1262
+ - **_table_path_** : Fully-qualified target table
1263
+ - **_type_** : File format type (e.g. `"json"`, `"csv"`)
1264
+ - **_on_error_** : `COPY INTO` `ON_ERROR` behavior
1265
+ - **_file_format_options_** : Extra `FILE_FORMAT` options appended after `TYPE`, e.g. `'FIELD_DELIMITER="," SKIP_HEADER=1'`
1266
+ - **_col_names_** : Explicit column list for `COPY INTO`, e.g. `"(first_name, last_name)"`. Defaults to `"(jsontext)"` for json, empty otherwise.
1267
+
1268
+ #### Returns
1269
+ - `None`. Progress is emitted at INFO level through the `logging` module; call `logging.basicConfig()` in your entry point to see it.
1270
+
1271
+ #### Requirements
1272
+ ```bash
1273
+ pip install gitlabds[snowflake]
1274
+ ```
1275
+
1276
+ #### Examples:
1277
+
1278
+ ```python
1279
+ gitlabds.snowflake_stage_load_copy_remove(
1280
+ file="scores.json", stage="my_stage", table_path="prod.data_science.scores", conn=conn
1281
+ )
1282
+ ```
1283
+ ```python
1284
+ # CSV with a header row and an explicit column list
1285
+ gitlabds.snowflake_stage_load_copy_remove(
1286
+ file="scores.csv",
1287
+ stage="my_stage",
1288
+ table_path="prod.data_science.scores",
1289
+ conn=conn,
1290
+ type="csv",
1291
+ file_format_options='FIELD_DELIMITER="," SKIP_HEADER=1',
1292
+ col_names="(account_id, score)",
1293
+ )
1294
+ ```
1295
+ </details>
1296
+
1297
+ <details><summary> Feature Store Factory </summary>
1298
+
1299
+ #### Description
1300
+ Create a Snowflake Feature Store connection (Snowpark Session + native `FeatureStore`), resolving credentials the same way as `snowflake_engine_factory`.
1301
+
1302
+ `gitlabds.feature_store_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="feature_store", auth_method="password", creation_mode="CreationMode.FAIL_IF_NOT_EXIST", profiles_path=None):`
1303
+
1304
+ #### Parameters:
1305
+ - **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
1306
+ - **_role_** : Snowflake role for CI credentials. The feature store's database and schema come from `SNOWFLAKE_FEATURE_STORE_DATABASE` / `SNOWFLAKE_FEATURE_STORE_SCHEMA` on the CI path, not from the role's usual database/schema.
1307
+ - **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
1308
+ - **_profile_target_** : dbt profile output name. Only used on the local path.
1309
+ - **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path.
1310
+ - **_creation_mode_** : Either a `"CreationMode.X"` string or a `snowflake.ml.feature_store.CreationMode` member.
1311
+ - **_profiles_path_** : Override the dbt `profiles.yml` location. Only used on the local path.
1312
+
1313
+ #### Returns
1314
+ - A `snowflake.ml.feature_store.FeatureStore`. The underlying Snowpark session is reachable as `fs._session` and is not closed by this function.
1315
+
1316
+ #### Requirements
1317
+ ```bash
1318
+ pip install gitlabds[snowflake]
1319
+ ```
1320
+
1321
+ #### Examples:
1322
+
1323
+ ```python
1324
+ import gitlabds
1325
+
1326
+ fs = gitlabds.feature_store_factory()
1327
+
1328
+ df = gitlabds.serve_features(
1329
+ session=fs._session,
1330
+ feature_store=fs,
1331
+ feature_views_dict={"account_ci_usage": "1.0"},
1332
+ spine_df="select account_id from prod.data_science.my_spine",
1333
+ )
1334
+ ```
1335
+ </details>
1336
+
1337
+ <details><summary> Migrating from gitlabdata.orchestration_utils </summary>
1338
+
1339
+ These functions replace their `gitlabdata.orchestration_utils` equivalents, but the signatures intentionally differ — this is not a drop-in import swap. Changes that fail at runtime rather than at import:
1340
+
1341
+ | Was | Now |
1342
+ |---|---|
1343
+ | `snowflake_engine_factory(...)` returns a SQLAlchemy `Engine` | returns a native `SnowflakeConnection` |
1344
+ | `data_science_engine_factory(...)` | folded into `snowflake_engine_factory(...)`, parameterized by `role` / `profile_target` |
1345
+ | `pd.read_sql(query, engine)` | `gitlabds.ds_query_dataframe(conn, query)` |
1346
+ | `df.to_sql(..., engine)` | `snowflake.connector.pandas_tools.write_pandas(conn, df, ...)` |
1347
+ | `engine.connect()` / `engine.dispose()` | `conn` is already open; `conn.close()` |
1348
+ | `query_executor(engine, query)` returns tuples — `row[0]` | returns dicts — `row["COLUMN_NAME"]` |
1349
+ | `query_executor(..., dispose_engine=..., connection=...)` | dropped; the caller owns the connection and passes it in |
1350
+ | `snowflake_stage_load_copy_remove(..., engine, type=...)` | unchanged, except `engine` is now a connection rather than a SQLAlchemy Engine |
1351
+ | `in_docker=True` | `profiles_path="/usr/local/snowflake_profile/profiles.yml"` |
1352
+ | `run_target="local"` default | `run_target=None`, inferred from `GITLAB_CI` |
1353
+
1354
+ Also worth knowing:
1355
+ - `file_format_options` is now honored for `json` too; the original silently dropped it for that type.
1356
+ - `stage` must be passed **without** a leading `@` (the function adds it). The original behaved the same way but did not document or enforce it.
1357
+ - These functions log through `logging.getLogger(__name__)` and no longer call `logging.basicConfig()` for you.
1358
+ </details>
1359
+
1111
1360
  #### Feature Serving
1112
1361
 
1113
1362
  <details><summary> Serve Features </summary>
@@ -1140,7 +1389,7 @@ Serve features from Snowflake Feature Store with built-in distributed locking an
1140
1389
  #### Requirements
1141
1390
  This function requires Snowflake packages. Install with:
1142
1391
  ```bash
1143
- pip install snowflake-snowpark-python snowflake-ml-python
1392
+ pip install gitlabds[snowflake]
1144
1393
  ```
1145
1394
 
1146
1395
  #### Examples:
@@ -1235,7 +1484,7 @@ Look up each feature's source feature view and description from the feature stor
1235
1484
  #### Requirements
1236
1485
  This function requires Snowflake packages. Install with:
1237
1486
  ```bash
1238
- pip install snowflake-snowpark-python snowflake-ml-python
1487
+ pip install gitlabds[snowflake]
1239
1488
  ```
1240
1489
 
1241
1490
  #### Examples: