gitlabds 2.2.2__tar.gz → 2.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {gitlabds-2.2.2/gitlabds.egg-info → gitlabds-2.3.0}/PKG-INFO +271 -22
- gitlabds-2.2.2/PKG-INFO → gitlabds-2.3.0/README.md +264 -50
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/__init__.py +17 -7
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/feature_descriptions.py +3 -22
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/record_benchmarking.py +234 -53
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/serving_features.py +4 -26
- gitlabds-2.3.0/gitlabds/snowflake_connectors.py +450 -0
- gitlabds-2.2.2/README.md → gitlabds-2.3.0/gitlabds.egg-info/PKG-INFO +299 -18
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds.egg-info/SOURCES.txt +3 -1
- gitlabds-2.3.0/gitlabds.egg-info/requires.txt +16 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/pyproject.toml +6 -3
- gitlabds-2.3.0/tests/test_snowflake_connectors.py +769 -0
- gitlabds-2.2.2/gitlabds.egg-info/requires.txt +0 -13
- {gitlabds-2.2.2 → gitlabds-2.3.0}/LICENSE +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/baselines.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/config_generator.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/dummy.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/feature_reduction.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/insights.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/memory_optimization.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/missing.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/model_evaluator.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/monitoring_metrics.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/outliers.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/split_data.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds/trends.py +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds.egg-info/dependency_links.txt +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/gitlabds.egg-info/top_level.txt +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/setup.cfg +0 -0
- {gitlabds-2.2.2 → gitlabds-2.3.0}/setup.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: gitlabds
|
|
3
|
-
Version: 2.
|
|
3
|
+
Version: 2.3.0
|
|
4
4
|
Summary: GitLab Data Science Tools
|
|
5
5
|
Author-email: Kevin Dietz <kdietz@gitlab.com>
|
|
6
6
|
License-Expression: MIT
|
|
@@ -25,9 +25,12 @@ Requires-Dist: seaborn>=0.13.2
|
|
|
25
25
|
Requires-Dist: shap>=0.50.0
|
|
26
26
|
Requires-Dist: tqdm>=4.66.2
|
|
27
27
|
Requires-Dist: xgboost>=1.6.1
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
Requires-Dist:
|
|
28
|
+
Requires-Dist: pyyaml>=6.0.0
|
|
29
|
+
Provides-Extra: snowflake
|
|
30
|
+
Requires-Dist: cryptography>=41.0.0; extra == "snowflake"
|
|
31
|
+
Requires-Dist: snowflake-connector-python[pandas]>=3.0.0; extra == "snowflake"
|
|
32
|
+
Requires-Dist: snowflake-snowpark-python>=1.37.0; extra == "snowflake"
|
|
33
|
+
Requires-Dist: snowflake-ml-python>=1.11.0; extra == "snowflake"
|
|
31
34
|
Dynamic: license-file
|
|
32
35
|
|
|
33
36
|
[](https://badge.fury.io/py/gitlabds)
|
|
@@ -61,19 +64,23 @@ pip install gitlabds
|
|
|
61
64
|
|
|
62
65
|
### Optional Dependencies
|
|
63
66
|
|
|
64
|
-
#### Snowflake
|
|
67
|
+
#### Snowflake
|
|
65
68
|
|
|
66
|
-
The `
|
|
69
|
+
The Snowflake connectors (`snowflake_engine_factory`, `ds_query_dataframe`, `query_executor`, `snowflake_stage_load_copy_remove`, `feature_store_factory`) and the feature serving functions (`serve_features`, `clear_feature_serving_locks`, `get_feature_descriptions`) require additional dependencies.
|
|
67
70
|
|
|
68
71
|
To use these functions, install with:
|
|
69
72
|
|
|
70
73
|
```bash
|
|
71
|
-
pip install gitlabds[
|
|
74
|
+
pip install gitlabds[snowflake]
|
|
72
75
|
```
|
|
73
76
|
|
|
74
77
|
This will install:
|
|
78
|
+
- `snowflake-connector-python[pandas]`
|
|
75
79
|
- `snowflake-snowpark-python`
|
|
76
80
|
- `snowflake-ml-python`
|
|
81
|
+
- `cryptography`
|
|
82
|
+
|
|
83
|
+
These are optional rather than core because `snowflake-ml-python` constrains the numpy/scikit-learn/xgboost stack that the core package also depends on.
|
|
77
84
|
|
|
78
85
|
If you try to use these functions without installing the optional dependencies, you'll get an error message with installation instructions.
|
|
79
86
|
|
|
@@ -827,10 +834,10 @@ Status label for a value vs. its peer median (+/- tolerance), not percentile ran
|
|
|
827
834
|
#### Parameters:
|
|
828
835
|
- _**value**_ : value(s) to evaluate, scalar or array-like
|
|
829
836
|
- _**peer_median**_ : peer group median(s) to compare against
|
|
830
|
-
- _**good_direction**_ : `"decreasing"` flips which side of the median counts as "above" (e.g., for a metric where lower is better).
|
|
837
|
+
- _**good_direction**_ : one of `"increasing"`, `"decreasing"`, `"both"`, `"none"` (scalar or array-like). `"decreasing"` flips which side of the median counts as "above" (e.g., for a metric where lower is better). `"both"` and `"none"` mean the feature has no single good direction, so the output is positional only: no flip, and the `ceiling_label` upgrade is suppressed. `None` (the default) is treated as `"none"`. Any other value raises — a blank/NaN in a feature dictionary must be normalized to `"none"` first rather than silently defaulting to increasing.
|
|
831
838
|
- _**tolerance**_ : relative distance from the peer median (as a fraction of the median) required to be classified above/below average rather than "average". Default is 0.15 (a value must be >15% above/below its peer median to leave the "average" band)
|
|
832
839
|
- _**no_activity_value**_ : value that should always map to `labels[3]` ("No activity") regardless of tolerance
|
|
833
|
-
- _**peer_percentile**_ : optional 0-1 percentile rank within the peer group. When provided, an "above average" reading that also sits in the top `ceiling_pct` toward `good_direction` is upgraded to `ceiling_label`
|
|
840
|
+
- _**peer_percentile**_ : optional 0-1 percentile rank within the peer group. When provided, an "above average" reading that also sits in the top `ceiling_pct` toward `good_direction` is upgraded to `ceiling_label`. Only applies when `good_direction` is `"increasing"` or `"decreasing"`
|
|
834
841
|
- _**ceiling_pct**_ : percentile threshold used with `peer_percentile` to detect a "tapped out" strength. Default is 0.95
|
|
835
842
|
- _**ceiling_label**_ : label applied when the ceiling condition is met. Default is "near ceiling"
|
|
836
843
|
- _**labels**_ : tuple of (below average, average, above average, no activity) labels
|
|
@@ -859,7 +866,7 @@ Per-record SHAP attribution (log-odds / margin space) for a tree model. Uses tre
|
|
|
859
866
|
`gitlabds.per_record_shap(model, df, feature_dict, unique_id="record_id", top_n=25, min_actionable=3, actionable_values=["Yes", "Partial"]):`
|
|
860
867
|
|
|
861
868
|
#### Parameters:
|
|
862
|
-
- _**model**_ : trained tree-based model (e.g., LightGBM, XGBoost)
|
|
869
|
+
- _**model**_ : trained tree-based model (e.g., LightGBM, XGBoost). A raw `xgboost.Booster` (from `xgb.train`, or loaded back from a saved model file) works as well as the sklearn wrapper
|
|
863
870
|
- _**df**_ : model-ready feature matrix, indexed by `unique_id`
|
|
864
871
|
- _**feature_dict**_ : DataFrame or path to CSV with `field` and `actionable` columns
|
|
865
872
|
- _**unique_id**_ : name to use for the record identifier column in the output. Default is "record_id"
|
|
@@ -870,6 +877,8 @@ Per-record SHAP attribution (log-odds / margin space) for a tree model. Uses tre
|
|
|
870
877
|
#### Returns
|
|
871
878
|
- Long DataFrame, one row per (record, feature): `unique_id`, `rank`, `feature`, `contribution`, `direction`, `feature_value`, `actionable`
|
|
872
879
|
|
|
880
|
+
`direction` ("increases"/"decreases") describes the effect on the model's predicted probability of class 1, not on the business outcome. Where class 1 is a bad outcome (e.g. churn), "increases" means worse — any narrative or scorecard layer reading this column has to apply that polarity itself.
|
|
881
|
+
|
|
873
882
|
#### Requirements
|
|
874
883
|
This function requires the `shap` package (included in core dependencies).
|
|
875
884
|
|
|
@@ -889,27 +898,29 @@ shap_df = gitlabds.per_record_shap(
|
|
|
889
898
|
<details><summary> Per-Record Counterfactuals </summary>
|
|
890
899
|
|
|
891
900
|
#### Description
|
|
892
|
-
"Change X -> +Y" counterfactual search, direction chosen empirically. For each actionable lever SHAP surfaced for a record, evaluates candidate values drawn from peer-group quantiles, re-scores the model at each, and keeps the single move that maximizes the
|
|
901
|
+
"Change X -> +Y" counterfactual search, direction chosen empirically. For each actionable lever SHAP surfaced for a record, evaluates candidate values drawn from peer-group quantiles, re-scores the model at each, and keeps the single move that maximizes `gain` — the improvement in the model score, signed by `score_goal` so that bigger is always better. No monotonicity assumption is made.
|
|
893
902
|
|
|
894
|
-
`gitlabds.per_record_counterfactuals(model, df, shap_df,
|
|
903
|
+
`gitlabds.per_record_counterfactuals(model, df, shap_df, unique_id="record_id", peer_group=None, candidate_quantiles=[0.1, 0.25, 0.5, 0.75, 0.9], direction=None, score_goal="maximize", actionable_values=["Yes", "Partial"], min_gain=0.005, impact_medium=0.01, impact_high=0.03, levels={"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}, n_bins=5):`
|
|
895
904
|
|
|
896
905
|
#### Parameters:
|
|
897
|
-
- _**model**_ : trained model (as in `per_record_shap`)
|
|
906
|
+
- _**model**_ : trained model (as in `per_record_shap`). How probabilities are scored is chosen from the model object itself — `.predict_proba()` where available, otherwise a raw `xgboost.Booster` or `lightgbm.Booster` is scored natively. An unsupported type raises a `TypeError` naming it
|
|
898
907
|
- _**df**_ : model-ready feature matrix, indexed by `unique_id`
|
|
899
|
-
- _**shap_df**_ : output of `per_record_shap`; scopes which (record, feature) levers to test
|
|
900
|
-
- _**model_algo**_ : `"lgb"` or `"xgb"`/`"xgboost"`; controls how probabilities are scored
|
|
908
|
+
- _**shap_df**_ : output of `per_record_shap`; scopes which (record, feature) levers to test. Only numeric non-bool levers are evaluated — candidate values are peer-group quantiles, which need an ordered numeric scale. Categoricals, bools, and strings are skipped with a `UserWarning`. Marking them `'none'` in `direction` is the explicit way to express the same thing, and suppresses the warning
|
|
901
909
|
- _**unique_id**_ : name of the record identifier column. Default is "record_id"
|
|
902
910
|
- _**peer_group**_ : Series/DataFrame indexed by `unique_id`; numeric columns are auto-binned into `n_bins` quantile bands. `None` uses one global group
|
|
903
911
|
- _**candidate_quantiles**_ : peer-group quantiles evaluated as candidate values
|
|
904
|
-
- _**direction**_ : optional per-feature
|
|
912
|
+
- _**direction**_ : optional per-feature constraint mapping feature -> `'increasing'` | `'decreasing'` | `'both'` | `'none'` (dict or Series). `'increasing'`/`'decreasing'` restrict candidates to moves in that direction (an achievability constraint); `'both'` allows either; `'none'` excludes the feature from lever evaluation entirely, for features with no meaningful good direction (categoricals, identifiers, context-only numerics). Features absent from the mapping, or `direction=None`, default to `'both'`. A blank/NaN raises — normalize it to `'none'` where the feature dictionary is read
|
|
913
|
+
- _**score_goal**_ : `"maximize"` if a higher predicted probability of class 1 is the good outcome (e.g. class 1 = uptier), `"minimize"` if it is the bad one (e.g. class 1 = churn). Sets the sign of `gain`, which drives best-move selection, the `min_gain` filter, `impact_bucket`, and the output ordering. This is the outcome's polarity only — `direction` above is a separate, feature-level concept and is unaffected. Default is `"maximize"`
|
|
905
914
|
- _**actionable_values**_ : values in `shap_df["actionable"]` treated as actionable levers
|
|
906
|
-
- _**
|
|
907
|
-
- _**impact_medium, impact_high**_ :
|
|
908
|
-
- _**levels**_ : dict mapping label -> upper bound on
|
|
915
|
+
- _**min_gain**_ : drop levers whose best achievable gain is below this. Note this is a threshold on `gain`, not on raw `delta_p`: under `score_goal="minimize"` a good move has a negative `delta_p` but a positive `gain`, so this stays positive for both polarities. Negative values (or `-inf`) are allowed, and keep moves that make the outcome worse — useful for inspecting the full candidate distribution while tuning thresholds
|
|
916
|
+
- _**impact_medium, impact_high**_ : `gain` thresholds bucketing each lever into Low/Medium/High `impact_bucket`. Only meaningful for `gain > 0` — under a negative `min_gain` the retained harmful moves stay `"Low"` (the level-crossing floor doesn't apply to them), so read `gain`'s sign rather than the bucket alone
|
|
917
|
+
- _**levels**_ : dict mapping label -> upper bound on P(class 1) for that level (e.g., default `{"poor fit": 0.03, "good fit": 0.10, "best fit": float("inf")}` means `p <= 0.03` -> "poor fit", `0.03 < p <= 0.10` -> "good fit", `p > 0.10` -> "best fit"). Key order doesn't matter — levels are sorted by cutoff and labels follow that order — but the top level's cutoff must be `float("inf")`. Cutoffs are always in P(class 1) space regardless of `score_goal`, and `crosses_level` is a plain inequality, so name the labels for your own outcome: a churn model would pass something like `{"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")}`. Used to detect whether a move crosses a level boundary (any crossing move is floored at "Medium" impact)
|
|
909
918
|
- _**n_bins**_ : quantile bins for numeric `peer_group` columns
|
|
910
919
|
|
|
911
920
|
#### Returns
|
|
912
|
-
- Long DataFrame, one row per (record, lever): `current_value`, `intervention_value`, `intervention_quantile`, `p_before`, `p_after`, `delta_p`, `impact_bucket`, `level_before`, `level_after`, `crosses_level`
|
|
921
|
+
- Long DataFrame, one row per (record, lever): `current_value`, `intervention_value`, `intervention_quantile`, `p_before`, `p_after`, `delta_p`, `gain`, `impact_bucket`, `level_before`, `level_after`, `crosses_level`
|
|
922
|
+
|
|
923
|
+
`delta_p` is the raw signed change in P(class 1); `gain` is that change signed by `score_goal` so bigger is always better. For a `score_goal="minimize"` model, `delta_p` will be negative and `gain` positive.
|
|
913
924
|
|
|
914
925
|
#### Requirements
|
|
915
926
|
For XGBoost models without `predict_proba`, requires the `xgboost` package (included in core dependencies).
|
|
@@ -924,6 +935,17 @@ counterfactuals_df = gitlabds.per_record_counterfactuals(
|
|
|
924
935
|
shap_df=shap_df,
|
|
925
936
|
peer_group=peer_group_df,
|
|
926
937
|
)
|
|
938
|
+
|
|
939
|
+
# Churn model: class 1 = churn, so a lower score is the good outcome.
|
|
940
|
+
counterfactuals_df = gitlabds.per_record_counterfactuals(
|
|
941
|
+
model=churn_model,
|
|
942
|
+
df=model_features,
|
|
943
|
+
shap_df=shap_df,
|
|
944
|
+
peer_group=peer_group_df,
|
|
945
|
+
score_goal="minimize",
|
|
946
|
+
direction=dict(zip(feature_dict["field"], feature_dict["good_direction"])),
|
|
947
|
+
levels={"low risk": 0.03, "medium risk": 0.10, "high risk": float("inf")},
|
|
948
|
+
)
|
|
927
949
|
```
|
|
928
950
|
|
|
929
951
|
</details>
|
|
@@ -944,6 +966,8 @@ Peer benchmarking. Each record's percentile for each feature within its peer gro
|
|
|
944
966
|
- _**features**_ : optional explicit list of features to benchmark (used as-is, no actionable filter). If `None` (default), computed for every actionable feature in `df`
|
|
945
967
|
- _**n_bins**_ : quantile bins for numeric `peer_group` columns
|
|
946
968
|
|
|
969
|
+
Either way, only numeric non-bool columns are benchmarked — percentile ranks and quantiles need an ordered numeric scale. Categoricals, bools, and strings are skipped with a `UserWarning` naming them.
|
|
970
|
+
|
|
947
971
|
#### Returns
|
|
948
972
|
- Long DataFrame, one row per (record, feature): `unique_id`, `feature`, `record_value`, `peer_group`, `percentile`, `peer_p25`, `peer_p50`, `peer_p75`
|
|
949
973
|
|
|
@@ -1108,6 +1132,231 @@ feature_drift = results['feature_drift']
|
|
|
1108
1132
|
|
|
1109
1133
|
### Deployment
|
|
1110
1134
|
|
|
1135
|
+
#### Snowflake Connectors
|
|
1136
|
+
|
|
1137
|
+
Thin wrappers over the native Snowflake SDKs that resolve GitLab's two credential paths — local (dbt profile + SSO) and GitLab CI (env vars + password or key pair) — and then hand off to the native objects. Migrated from `gitlabdata.orchestration_utils`; see the migration notes at the end of this section.
|
|
1138
|
+
|
|
1139
|
+
All of these require `pip install gitlabds[snowflake]`.
|
|
1140
|
+
|
|
1141
|
+
<details><summary> Snowflake Engine Factory </summary>
|
|
1142
|
+
|
|
1143
|
+
#### Description
|
|
1144
|
+
Open a Snowflake connection, resolving credentials from your dbt profile locally or from GitLab CI environment variables in CI.
|
|
1145
|
+
|
|
1146
|
+
`gitlabds.snowflake_engine_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="dev", schema="", auth_method="password", profiles_path=None, **connect_kwargs):`
|
|
1147
|
+
|
|
1148
|
+
#### Parameters:
|
|
1149
|
+
- **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
|
|
1150
|
+
- **_role_** : CI credential set to use. Currently only supports `DATA_SCIENCE_LOADER`
|
|
1151
|
+
- **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
|
|
1152
|
+
- **_profile_target_** : dbt profile output name. Only used on the local path.
|
|
1153
|
+
- **_schema_** : Schema to connect to. Falls back to the dbt profile's schema locally; when it resolves empty the connection is opened with no default schema.
|
|
1154
|
+
- **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path. Key pair secrets are not yet provisioned in CI.
|
|
1155
|
+
- **_profiles_path_** : Override the dbt `profiles.yml` location (defaults to `~/.dbt/profiles.yml`). Only used on the local path.
|
|
1156
|
+
- **_\*\*connect_kwargs_** : Passed through to `snowflake.connector.connect()`, e.g. `autocommit=False`, `client_session_keep_alive=True`.
|
|
1157
|
+
|
|
1158
|
+
#### Returns
|
|
1159
|
+
- An open `snowflake.connector.SnowflakeConnection`. The caller owns its lifecycle.
|
|
1160
|
+
|
|
1161
|
+
#### Requirements
|
|
1162
|
+
```bash
|
|
1163
|
+
pip install gitlabds[snowflake]
|
|
1164
|
+
```
|
|
1165
|
+
|
|
1166
|
+
#### Examples:
|
|
1167
|
+
|
|
1168
|
+
```python
|
|
1169
|
+
import gitlabds
|
|
1170
|
+
|
|
1171
|
+
# Local: reads the "dev" target from ~/.dbt/profiles.yml and authenticates via SSO
|
|
1172
|
+
conn = gitlabds.snowflake_engine_factory()
|
|
1173
|
+
try:
|
|
1174
|
+
df = gitlabds.ds_query_dataframe(conn, "select * from prod.data_science.my_table")
|
|
1175
|
+
finally:
|
|
1176
|
+
conn.close()
|
|
1177
|
+
```
|
|
1178
|
+
```python
|
|
1179
|
+
# CI: credentials come from os.environ; run_target is inferred from GITLAB_CI
|
|
1180
|
+
conn = gitlabds.snowflake_engine_factory(role="DATA_SCIENCE_LOADER", schema="data_science")
|
|
1181
|
+
|
|
1182
|
+
# Opt out of autocommit to group statements into one transaction
|
|
1183
|
+
conn = gitlabds.snowflake_engine_factory(autocommit=False)
|
|
1184
|
+
```
|
|
1185
|
+
</details>
|
|
1186
|
+
|
|
1187
|
+
<details><summary> DS Query Dataframe </summary>
|
|
1188
|
+
|
|
1189
|
+
#### Description
|
|
1190
|
+
Run a SELECT and return a pandas DataFrame, using Snowflake's Arrow-based fetch. Prefer this over `query_executor` for large reads such as pulling training data.
|
|
1191
|
+
|
|
1192
|
+
`gitlabds.ds_query_dataframe(conn, query, optimize_memory=False, verbose=False):`
|
|
1193
|
+
|
|
1194
|
+
#### Parameters:
|
|
1195
|
+
- **_conn_** : An open connection, e.g. from `snowflake_engine_factory()`
|
|
1196
|
+
- **_query_** : SELECT query to run
|
|
1197
|
+
- **_optimize_memory_** : Run the result through `reduce_memory_usage()` (dtype downcasting) before returning. Off by default because it is a lossy transform (e.g. float64 → float32).
|
|
1198
|
+
- **_verbose_** : Print the memory-optimization report and progress bar. Only applies when `optimize_memory=True`.
|
|
1199
|
+
|
|
1200
|
+
#### Returns
|
|
1201
|
+
- A pandas DataFrame
|
|
1202
|
+
|
|
1203
|
+
#### Requirements
|
|
1204
|
+
```bash
|
|
1205
|
+
pip install gitlabds[snowflake]
|
|
1206
|
+
```
|
|
1207
|
+
|
|
1208
|
+
#### Examples:
|
|
1209
|
+
|
|
1210
|
+
```python
|
|
1211
|
+
df = gitlabds.ds_query_dataframe(conn, "select * from my_table")
|
|
1212
|
+
|
|
1213
|
+
# Large pull: downcast dtypes on the way out
|
|
1214
|
+
df = gitlabds.ds_query_dataframe(conn, training_query, optimize_memory=True, verbose=True)
|
|
1215
|
+
```
|
|
1216
|
+
</details>
|
|
1217
|
+
|
|
1218
|
+
<details><summary> Query Executor </summary>
|
|
1219
|
+
|
|
1220
|
+
#### Description
|
|
1221
|
+
Execute a SQL statement and return its rows as a list of dicts keyed by column name, as Snowflake returns them. Use for DML/DDL/transaction-control statements, or reads where a DataFrame isn't needed.
|
|
1222
|
+
|
|
1223
|
+
`gitlabds.query_executor(conn, query):`
|
|
1224
|
+
|
|
1225
|
+
#### Parameters:
|
|
1226
|
+
- **_conn_** : An open connection, e.g. from `snowflake_engine_factory()`
|
|
1227
|
+
- **_query_** : SQL statement to run
|
|
1228
|
+
|
|
1229
|
+
#### Returns
|
|
1230
|
+
- `List[Dict[str, Any]]` — rows as `{column: value}`. DML and DDL statements return Snowflake's own status row (e.g. `{"number of rows inserted": 12}`) rather than an empty list.
|
|
1231
|
+
|
|
1232
|
+
#### Requirements
|
|
1233
|
+
```bash
|
|
1234
|
+
pip install gitlabds[snowflake]
|
|
1235
|
+
```
|
|
1236
|
+
|
|
1237
|
+
#### Examples:
|
|
1238
|
+
|
|
1239
|
+
```python
|
|
1240
|
+
rows = gitlabds.query_executor(conn, "select count(*) as n from my_table")
|
|
1241
|
+
print(rows[0]["N"])
|
|
1242
|
+
```
|
|
1243
|
+
```python
|
|
1244
|
+
# Connections autocommit by default, so group statements explicitly
|
|
1245
|
+
gitlabds.query_executor(conn, "begin;")
|
|
1246
|
+
gitlabds.query_executor(conn, "delete from my_table where run_date = '2026-01-01';")
|
|
1247
|
+
gitlabds.query_executor(conn, "commit;")
|
|
1248
|
+
```
|
|
1249
|
+
</details>
|
|
1250
|
+
|
|
1251
|
+
<details><summary> Snowflake Stage Load Copy Remove </summary>
|
|
1252
|
+
|
|
1253
|
+
#### Description
|
|
1254
|
+
Load a local file into a Snowflake table via an internal stage: remove any leftover staged file, PUT the file, COPY INTO the target table, then REMOVE the staged file.
|
|
1255
|
+
|
|
1256
|
+
`gitlabds.snowflake_stage_load_copy_remove(file, stage, table_path, conn, type="json", on_error="abort_statement", file_format_options="", col_names=""):`
|
|
1257
|
+
|
|
1258
|
+
#### Parameters:
|
|
1259
|
+
- **_conn_** : An open connection, e.g. from `snowflake_engine_factory()`
|
|
1260
|
+
- **_file_** : Path to the local file to upload
|
|
1261
|
+
- **_stage_** : Internal stage name **without** a leading `@` — the function adds it (e.g. `"my_stage"` or `"db.schema.my_stage"`)
|
|
1262
|
+
- **_table_path_** : Fully-qualified target table
|
|
1263
|
+
- **_type_** : File format type (e.g. `"json"`, `"csv"`)
|
|
1264
|
+
- **_on_error_** : `COPY INTO` `ON_ERROR` behavior
|
|
1265
|
+
- **_file_format_options_** : Extra `FILE_FORMAT` options appended after `TYPE`, e.g. `'FIELD_DELIMITER="," SKIP_HEADER=1'`
|
|
1266
|
+
- **_col_names_** : Explicit column list for `COPY INTO`, e.g. `"(first_name, last_name)"`. Defaults to `"(jsontext)"` for json, empty otherwise.
|
|
1267
|
+
|
|
1268
|
+
#### Returns
|
|
1269
|
+
- `None`. Progress is emitted at INFO level through the `logging` module; call `logging.basicConfig()` in your entry point to see it.
|
|
1270
|
+
|
|
1271
|
+
#### Requirements
|
|
1272
|
+
```bash
|
|
1273
|
+
pip install gitlabds[snowflake]
|
|
1274
|
+
```
|
|
1275
|
+
|
|
1276
|
+
#### Examples:
|
|
1277
|
+
|
|
1278
|
+
```python
|
|
1279
|
+
gitlabds.snowflake_stage_load_copy_remove(
|
|
1280
|
+
file="scores.json", stage="my_stage", table_path="prod.data_science.scores", conn=conn
|
|
1281
|
+
)
|
|
1282
|
+
```
|
|
1283
|
+
```python
|
|
1284
|
+
# CSV with a header row and an explicit column list
|
|
1285
|
+
gitlabds.snowflake_stage_load_copy_remove(
|
|
1286
|
+
file="scores.csv",
|
|
1287
|
+
stage="my_stage",
|
|
1288
|
+
table_path="prod.data_science.scores",
|
|
1289
|
+
conn=conn,
|
|
1290
|
+
type="csv",
|
|
1291
|
+
file_format_options='FIELD_DELIMITER="," SKIP_HEADER=1',
|
|
1292
|
+
col_names="(account_id, score)",
|
|
1293
|
+
)
|
|
1294
|
+
```
|
|
1295
|
+
</details>
|
|
1296
|
+
|
|
1297
|
+
<details><summary> Feature Store Factory </summary>
|
|
1298
|
+
|
|
1299
|
+
#### Description
|
|
1300
|
+
Create a Snowflake Feature Store connection (Snowpark Session + native `FeatureStore`), resolving credentials the same way as `snowflake_engine_factory`.
|
|
1301
|
+
|
|
1302
|
+
`gitlabds.feature_store_factory(args=None, role="DATA_SCIENCE_LOADER", run_target=None, profile_target="feature_store", auth_method="password", creation_mode="CreationMode.FAIL_IF_NOT_EXIST", profiles_path=None):`
|
|
1303
|
+
|
|
1304
|
+
#### Parameters:
|
|
1305
|
+
- **_args_** : Environment mapping to read CI credentials from. Defaults to `os.environ`. Ignored on the local path.
|
|
1306
|
+
- **_role_** : Snowflake role for CI credentials. The feature store's database and schema come from `SNOWFLAKE_FEATURE_STORE_DATABASE` / `SNOWFLAKE_FEATURE_STORE_SCHEMA` on the CI path, not from the role's usual database/schema.
|
|
1307
|
+
- **_run_target_** : `"local"` or `"ci"`. Inferred from the `GITLAB_CI` environment variable when not given.
|
|
1308
|
+
- **_profile_target_** : dbt profile output name. Only used on the local path.
|
|
1309
|
+
- **_auth_method_** : `"password"` or `"keypair"`. Only used on the CI path.
|
|
1310
|
+
- **_creation_mode_** : Either a `"CreationMode.X"` string or a `snowflake.ml.feature_store.CreationMode` member.
|
|
1311
|
+
- **_profiles_path_** : Override the dbt `profiles.yml` location. Only used on the local path.
|
|
1312
|
+
|
|
1313
|
+
#### Returns
|
|
1314
|
+
- A `snowflake.ml.feature_store.FeatureStore`. The underlying Snowpark session is reachable as `fs._session` and is not closed by this function.
|
|
1315
|
+
|
|
1316
|
+
#### Requirements
|
|
1317
|
+
```bash
|
|
1318
|
+
pip install gitlabds[snowflake]
|
|
1319
|
+
```
|
|
1320
|
+
|
|
1321
|
+
#### Examples:
|
|
1322
|
+
|
|
1323
|
+
```python
|
|
1324
|
+
import gitlabds
|
|
1325
|
+
|
|
1326
|
+
fs = gitlabds.feature_store_factory()
|
|
1327
|
+
|
|
1328
|
+
df = gitlabds.serve_features(
|
|
1329
|
+
session=fs._session,
|
|
1330
|
+
feature_store=fs,
|
|
1331
|
+
feature_views_dict={"account_ci_usage": "1.0"},
|
|
1332
|
+
spine_df="select account_id from prod.data_science.my_spine",
|
|
1333
|
+
)
|
|
1334
|
+
```
|
|
1335
|
+
</details>
|
|
1336
|
+
|
|
1337
|
+
<details><summary> Migrating from gitlabdata.orchestration_utils </summary>
|
|
1338
|
+
|
|
1339
|
+
These functions replace their `gitlabdata.orchestration_utils` equivalents, but the signatures intentionally differ — this is not a drop-in import swap. Changes that fail at runtime rather than at import:
|
|
1340
|
+
|
|
1341
|
+
| Was | Now |
|
|
1342
|
+
|---|---|
|
|
1343
|
+
| `snowflake_engine_factory(...)` returns a SQLAlchemy `Engine` | returns a native `SnowflakeConnection` |
|
|
1344
|
+
| `data_science_engine_factory(...)` | folded into `snowflake_engine_factory(...)`, parameterized by `role` / `profile_target` |
|
|
1345
|
+
| `pd.read_sql(query, engine)` | `gitlabds.ds_query_dataframe(conn, query)` |
|
|
1346
|
+
| `df.to_sql(..., engine)` | `snowflake.connector.pandas_tools.write_pandas(conn, df, ...)` |
|
|
1347
|
+
| `engine.connect()` / `engine.dispose()` | `conn` is already open; `conn.close()` |
|
|
1348
|
+
| `query_executor(engine, query)` returns tuples — `row[0]` | returns dicts — `row["COLUMN_NAME"]` |
|
|
1349
|
+
| `query_executor(..., dispose_engine=..., connection=...)` | dropped; the caller owns the connection and passes it in |
|
|
1350
|
+
| `snowflake_stage_load_copy_remove(..., engine, type=...)` | unchanged, except `engine` is now a connection rather than a SQLAlchemy Engine |
|
|
1351
|
+
| `in_docker=True` | `profiles_path="/usr/local/snowflake_profile/profiles.yml"` |
|
|
1352
|
+
| `run_target="local"` default | `run_target=None`, inferred from `GITLAB_CI` |
|
|
1353
|
+
|
|
1354
|
+
Also worth knowing:
|
|
1355
|
+
- `file_format_options` is now honored for `json` too; the original silently dropped it for that type.
|
|
1356
|
+
- `stage` must be passed **without** a leading `@` (the function adds it). The original behaved the same way but did not document or enforce it.
|
|
1357
|
+
- These functions log through `logging.getLogger(__name__)` and no longer call `logging.basicConfig()` for you.
|
|
1358
|
+
</details>
|
|
1359
|
+
|
|
1111
1360
|
#### Feature Serving
|
|
1112
1361
|
|
|
1113
1362
|
<details><summary> Serve Features </summary>
|
|
@@ -1140,7 +1389,7 @@ Serve features from Snowflake Feature Store with built-in distributed locking an
|
|
|
1140
1389
|
#### Requirements
|
|
1141
1390
|
This function requires Snowflake packages. Install with:
|
|
1142
1391
|
```bash
|
|
1143
|
-
pip install snowflake
|
|
1392
|
+
pip install gitlabds[snowflake]
|
|
1144
1393
|
```
|
|
1145
1394
|
|
|
1146
1395
|
#### Examples:
|
|
@@ -1235,7 +1484,7 @@ Look up each feature's source feature view and description from the feature stor
|
|
|
1235
1484
|
#### Requirements
|
|
1236
1485
|
This function requires Snowflake packages. Install with:
|
|
1237
1486
|
```bash
|
|
1238
|
-
pip install snowflake
|
|
1487
|
+
pip install gitlabds[snowflake]
|
|
1239
1488
|
```
|
|
1240
1489
|
|
|
1241
1490
|
#### Examples:
|