bigquery-data-science 0.7.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- bigquery_data_science/__init__.py +58 -0
- bigquery_data_science/_skills/PLANS.md +76 -0
- bigquery_data_science/_skills/bigquery-ai/SKILL.md +66 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_agg.md +528 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_classify.md +450 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_count_tokens.md +180 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_detect_anomalies.md +322 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_embed.md +425 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_evaluate.md +368 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_forecast.md +437 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_generate.md +693 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_generate_bool.md +248 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_generate_double.md +254 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_generate_embedding.md +402 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_generate_int.md +252 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_generate_table.md +447 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_generate_text.md +356 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_if.md +373 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_key_drivers.md +331 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_parse_document.md +416 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_predict.md +559 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_score.md +310 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_search.md +465 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ai_similarity.md +371 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/catalog_search.md +951 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/content_analysis.md +284 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/content_moderation.md +369 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/data_enrichment.md +426 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/document_intelligence.md +470 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/document_rag.md +453 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/image_deduplication.md +770 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/log_analysis.md +668 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/metric_diagnostics.md +568 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ml_annotate_image.md +680 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ml_generate_embedding.md +217 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ml_generate_text.md +223 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ml_process_document.md +392 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ml_transcribe.md +815 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ml_translate.md +600 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/ml_understand_text.md +673 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/multimodal_analysis.md +266 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/rag_pipeline.md +593 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/semantic_search.md +510 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/tabular_prediction.md +485 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/time_series_intelligence.md +489 -0
- bigquery_data_science/_skills/bigquery-ai/narrative/vector_search.md +672 -0
- bigquery_data_science/_skills/bigquery-ai/reference/augmented-analytics.md +77 -0
- bigquery_data_science/_skills/bigquery-ai/reference/cloud-ai-service-models.md +88 -0
- bigquery_data_science/_skills/bigquery-ai/reference/document-processing.md +95 -0
- bigquery_data_science/_skills/bigquery-ai/reference/embeddings-and-search.md +166 -0
- bigquery_data_science/_skills/bigquery-ai/reference/general-purpose-generation.md +87 -0
- bigquery_data_science/_skills/bigquery-ai/reference/managed-functions.md +72 -0
- bigquery_data_science/_skills/bigquery-ai/reference/predictive-ai.md +113 -0
- bigquery_data_science/_skills/bigquery-ai/reference/workflows.md +56 -0
- bigquery_data_science/_skills/bigquery-ai/skill.manifest.json +109 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/INDEX.md +71 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aiagg.md +67 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aicausal_effect-covered-in-bigquery-ml.md +17 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aiclassify.md +72 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aicount_tokens.md +53 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aidetect_anomalies.md +63 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aiembed.md +85 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aievaluate.md +114 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aiforecast.md +123 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aigenerate.md +69 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aigenerate_bool.md +53 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aigenerate_double.md +45 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aigenerate_embedding.md +108 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aigenerate_int.md +45 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aigenerate_table.md +64 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aigenerate_text.md +167 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aiif.md +56 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aikey_drivers.md +76 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aiparse_document.md +91 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aipredict.md +74 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aiscore.md +52 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aisearch.md +66 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/aisimilarity.md +79 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/choosing-between-these-and-the-ai-functions.md +11 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/hybrid-search-capability.md +165 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/mlannotate_image.md +24 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/mlgenerate_embedding.md +34 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/mlgenerate_text.md +65 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/mlprocess_document.md +125 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/mltranscribe.md +26 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/mltranslate.md +27 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/mlunderstand_text.md +32 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/multimodal-input-patterns-by-function.md +128 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/object-tables.md +109 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/objectref-and-objectrefruntime-schema-reference.md +50 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/objfetch_metadata.md +49 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/objget_access_url.md +66 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/objget_read_url.md +40 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/objlist.md +34 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/objmake_ref.md +58 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/per-row-failures-and-the-retry-pattern.md +19 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/setup-apis-connection-roles.md +39 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/two-input-shapes.md +22 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/vector_search.md +78 -0
- bigquery_data_science/_skills/bigquery-ai/syntax/what-the-five-have-in-common.md +27 -0
- bigquery_data_science/_skills/bigquery-data-science/SKILL.md +69 -0
- bigquery_data_science/_skills/bigquery-data-science/reference/namespace-matrix.md +112 -0
- bigquery_data_science/_skills/bigquery-data-science/skill.manifest.json +13 -0
- bigquery_data_science/_skills/bigquery-ml/SKILL.md +70 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/airflow_with_kfp.md +379 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/anomaly_fraud_detection.md +195 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/arima_plus.md +635 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/arima_plus_xreg.md +453 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/autoencoder.md +470 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/automl_classifier.md +291 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/automl_regressor.md +274 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/boosted_tree_classifier.md +449 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/boosted_tree_regressor.md +400 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/bucketizing.md +201 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/causal_effect.md +773 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/churn_retention.md +271 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/cloud_workflows.md +370 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/composer_airflow.md +475 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/contribution_analysis.md +346 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/cross_validation.md +185 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/customer_segmentation.md +233 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/dataform.md +372 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/dbt.md +283 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/difference_in_differences.md +280 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/distance.md +195 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/dnn_classifier.md +326 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/dnn_regressor.md +308 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/embeddings_classification.md +426 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/encoding.md +238 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/ensembling.md +278 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/evaluation.md +445 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/exploration.md +779 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/export.md +321 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/feature_engineering.md +280 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/feature_store.md +652 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/feature_store_workflow.md +426 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/ga4_churn_prediction.md +285 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/hierarchical_forecasting.md +441 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/image.md +791 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/imported.md +426 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/kmeans.md +348 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/linear_regression.md +372 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/logistic_regression.md +390 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/matrix_factorization.md +404 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/model_monitoring.md +609 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/pca.md +357 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/price_elasticity_dml.md +286 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/propensity_score_matching.md +350 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/random_forest_classifier.md +445 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/random_forest_regressor.md +409 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/recommendation.md +281 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/regression_based_forecasting.md +491 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/remote.md +428 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/scalers.md +325 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/scheduled_queries.md +284 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/sql_scripting.md +243 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/survival_analysis.md +363 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/synthetic_control.md +221 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/text.md +215 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/time_series.md +669 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/transform_only.md +273 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/uplift_cate.md +241 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/vertex_kfp.md +360 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/wide_and_deep_classifier.md +316 -0
- bigquery_data_science/_skills/bigquery-ml/narrative/wide_and_deep_regressor.md +298 -0
- bigquery_data_science/_skills/bigquery-ml/reference/classification.md +80 -0
- bigquery_data_science/_skills/bigquery-ml/reference/model-management.md +90 -0
- bigquery_data_science/_skills/bigquery-ml/reference/preprocessing-functions.md +234 -0
- bigquery_data_science/_skills/bigquery-ml/reference/regression.md +79 -0
- bigquery_data_science/_skills/bigquery-ml/reference/unsupervised-and-specialized.md +101 -0
- bigquery_data_science/_skills/bigquery-ml/reference/workflows-and-pipelines.md +111 -0
- bigquery_data_science/_skills/bigquery-ml/skill.manifest.json +177 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/1-model_type-catalog.md +28 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/2-task-mlevaluate-metric-columns.md +26 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/3-capability-matrix-per-model-type.md +21 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/4-explainability-weights-functions.md +14 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/5-connection-matrix.md +11 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/6-which-model-free-function-no-create-model-no-connection.md +15 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/7-point-in-time-feature-retrieval-which-of-the-two.md +11 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/INDEX.md +111 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/aicausal_effect.md +85 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/arima_plus-forecasting-lifecycle-functions.md +14 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/arima_plus-univariate-time-series-forecasting.md +132 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/arima_plus_xreg-multivariate-time-series-with-external-regressors.md +103 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/autoencoder.md +102 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/automl_classifier-automl_regressor.md +88 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/boosted_tree_classifier-boosted_tree_regressor.md +117 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/contribution_analysis.md +113 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/dnn_classifier-dnn_regressor.md +107 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/dnn_linear_combined_classifier-dnn_linear_combined_regressor-wide-and-deep.md +98 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/explainability-functions-mlexplain_predict-mlglobal_explain-mlfeature_importance.md +7 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/exploratory-data-analysis-mldescribe_data-mlcorrelation.md +11 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/export-model.md +92 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/feature-engineering-standalone-preprocessing-functions.md +13 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/image-preprocessing-functions-mldecode_image-mlresize_image-mlconvert_image_type-mlconvert_color_space.md +63 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/imported-models-tensorflow-tensorflow_lite-onnx-xgboost.md +36 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/kmeans.md +87 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/linear_reg-logistic_reg-generalized-linear-models.md +101 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/matrix_factorization.md +162 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mladvanced_weights.md +76 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlarima_coefficients.md +30 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlarima_evaluate.md +49 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlbag_of_words.md +47 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlbucketize.md +47 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlcentroids.md +46 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlconfusion_matrix.md +56 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlconvert_color_space.md +53 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlconvert_image_type.md +39 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlcorrelation.md +78 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mldecode_image.md +44 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mldescribe_data.md +74 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mldetect_anomalies.md +86 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mldistance.md +48 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlentity_features_at_time.md +56 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlevaluate.md +72 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlexplain_forecast.md +50 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlexplain_predict.md +66 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlfeature_cross.md +46 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlfeature_importance.md +47 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlfeature_info.md +50 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlfeatures_at_time.md +51 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlforecast.md +76 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlgenerate_embedding-from-a-bqml-pca-autoencoder-matrix_factorization-model.md +64 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlglobal_explain.md +63 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlhash_bucketize.md +47 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlholiday_info.md +46 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlimputer.md +46 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mllp_norm.md +47 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlmetrics.md +80 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlngrams.md +37 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlone_hot_encoder-mlmulti_hot_encoder-mllabel_encoder.md +67 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlpolynomial_expand.md +47 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlpredict.md +67 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlprincipal_component_info.md +42 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlprincipal_components.md +40 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlquantile_bucketize.md +43 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlrecommend.md +60 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlreconstruction_loss.md +48 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlresize_image.md +45 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlroc_curve.md +63 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mltf_idf.md +37 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mltfdv_describe.md +34 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mltfdv_validate.md +48 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mltraining_info.md +78 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mltransform-applying-a-saved-transform.md +52 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mltranspose-not-a-function-pointer.md +32 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mltrial_info.md +60 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlvalidate_data_drift.md +99 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlvalidate_data_skew.md +50 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/mlweights.md +84 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/model-free-time-series-decomposition-mltrend-mlseasonality-mldetect_change_points.md +100 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/model-introspection-mlfeature_info-mltraining_info-mltrial_info.md +7 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/model-monitoring-data-validation.md +23 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/numerical-scalers-mlstandard_scaler-mlmin_max_scaler-mlmax_abs_scaler-mlrobust_scaler-mlnormalizer.md +77 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/onnx.md +55 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/pca.md +91 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/point-in-time-feature-retrieval-mlfeatures_at_time-mlentity_features_at_time.md +25 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/quick-comparison.md +14 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/random_forest_classifier-random_forest_regressor.md +123 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/remote-remote-model-over-a-vertex-ai-endpoint-the-mechanism.md +87 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/tensorflow.md +48 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/tensorflow_lite.md +29 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/text-preprocessing-functions-mlngrams-mltf_idf-mlbag_of_words.md +18 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/timesfm-built-in-foundation-forecaster-see-bigquery-ai.md +28 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/transform_only.md +91 -0
- bigquery_data_science/_skills/bigquery-ml/syntax/xgboost.md +72 -0
- bigquery_data_science/_skills/pinned-models.json +28 -0
- bigquery_data_science/catalog.py +1107 -0
- bigquery_data_science/cli.py +786 -0
- bigquery_data_science/installer.py +282 -0
- bigquery_data_science/mcp_server.py +317 -0
- bigquery_data_science/tooling/__init__.py +15 -0
- bigquery_data_science/tooling/convert_notebook.py +176 -0
- bigquery_data_science/tooling/link_policy.py +247 -0
- bigquery_data_science/tooling/manifest.py +152 -0
- bigquery_data_science/tooling/markdown_anchors.py +84 -0
- bigquery_data_science/tooling/narrative_drift.py +163 -0
- bigquery_data_science/tooling/pinned_models.py +216 -0
- bigquery_data_science/tooling/reference_structure.py +85 -0
- bigquery_data_science/tooling/sync_syntax.py +407 -0
- bigquery_data_science/tooling/validate.py +208 -0
- bigquery_data_science-0.7.0.dist-info/METADATA +306 -0
- bigquery_data_science-0.7.0.dist-info/RECORD +286 -0
- bigquery_data_science-0.7.0.dist-info/WHEEL +4 -0
- bigquery_data_science-0.7.0.dist-info/entry_points.txt +5 -0
- bigquery_data_science-0.7.0.dist-info/licenses/LICENSE +201 -0
|
@@ -0,0 +1,273 @@
|
|
|
1
|
+
> **Source (`statmike/bigquery-data-science`):** **Notebook:** [`bigquery-ml/models/transform_only/transform_only.ipynb`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/models/transform_only/transform_only.ipynb) · **SQL:** [`bigquery-ml/models/transform_only/transform_only.sql`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/models/transform_only/transform_only.sql)
|
|
2
|
+
|
|
3
|
+
# Transform-Only Models — BigQuery ML
|
|
4
|
+
|
|
5
|
+
Train a model that contains **only a `TRANSFORM` clause and no learning algorithm** with `CREATE MODEL` (model_type = `TRANSFORM_ONLY`) — a reusable, exportable preprocessing pipeline, decoupled from any estimator. There's no label, no training/evaluation of a predictor: the "output" of this model is the preprocessed data itself, produced by `ML.TRANSFORM`.
|
|
6
|
+
|
|
7
|
+
**Lifecycle:** `CREATE MODEL` (with `TRANSFORM`, no estimator) → `ML.TRANSFORM` → `ML.FEATURE_INFO` → `EXPORT MODEL` → feed the pipeline's output into a downstream `CREATE MODEL`
|
|
8
|
+
|
|
9
|
+
**Why this is its own model category:** every other model type in this project embeds an *optional* `TRANSFORM` clause directly on a predictive model (see `models/pca` (PCA) Step 10, or `models/kmeans` (K-Means)) — that embedded transform auto-applies at `ML.PREDICT`/`ML.EVALUATE` time, but it isn't reusable outside that one model. A `TRANSFORM_ONLY` model flips that: the pipeline is a first-class, reusable object with no estimator attached, at the cost of having to explicitly re-apply it (`ML.TRANSFORM`) before feeding new data into whatever downstream model consumes it.
|
|
10
|
+
|
|
11
|
+
**When to use a transform-only model:**
|
|
12
|
+
- Decouple feature engineering from model training so the same preprocessing (with its frozen train-time statistics — means, stddevs, vocabularies) can be reused across many downstream models.
|
|
13
|
+
- Build modular, feature-store-style transforms and chain them with CTEs or a view.
|
|
14
|
+
- Guarantee training/serving consistency without embedding the same `TRANSFORM` clause repeatedly across model definitions.
|
|
15
|
+
|
|
16
|
+
**A gotcha this notebook demonstrates directly, not just describes:** a downstream model trained on a `TRANSFORM_ONLY` pipeline's output has **no embedded `TRANSFORM` of its own**. If you call `ML.PREDICT` on that downstream model with *raw*, untransformed data, BigQuery ML does **not** error — it silently predicts using values on the wrong scale, giving confidently wrong answers. You must re-wrap new data in `ML.TRANSFORM` (using the same pipeline model) before every `ML.PREDICT`/`ML.EVALUATE` call.
|
|
17
|
+
|
|
18
|
+
**Data:** [`bigquery-public-data.ml_datasets.penguins`](https://console.cloud.google.com/marketplace/product/bigquery-public-datasets) — same dataset as `models/kmeans` (K-Means), `models/pca` (PCA), and `models/autoencoder` (Autoencoder). The pipeline imputes missing `body_mass_g`, standard-scales two measurements, robust-scales a third, and one-hot encodes `sex`; a downstream `LOGISTIC_REG` model then predicts `species` from the pipeline's output.
|
|
19
|
+
|
|
20
|
+
**Bridges into:** Phase 6 (model-free `ML.*` functions) documents each preprocessing function (`ML.IMPUTER`, the scalers, the encoders) individually and in standalone SQL — this notebook shows several of them **composed into one reusable pipeline object**, which is the more realistic way they get used in practice.
|
|
21
|
+
|
|
22
|
+
**References:** `RESOURCES.md` (Full reference) | [CREATE MODEL (transform-only) docs](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-transform) | `setup` (Setup guide)
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
## Setup
|
|
26
|
+
|
|
27
|
+
Set your project and location, authenticate, and create a shared dataset for the model.
|
|
28
|
+
|
|
29
|
+
> A transform-only model trains on data already in BigQuery — no connection or remote model required. See the `setup` (Setup Reference) for details.
|
|
30
|
+
|
|
31
|
+
```python
|
|
32
|
+
PROJECT_ID = 'statmike-mlops-349915' # <-- Replace with your project ID
|
|
33
|
+
LOCATION = 'US' # BigQuery dataset location
|
|
34
|
+
DATASET_ID = 'bq_ml' # Shared dataset across all bigquery-ml notebooks
|
|
35
|
+
BUCKET = 'statmike-mlops-349915' # <-- Replace with your GCS bucket (same location as DATASET_ID) -- used in Step 4 to export the pipeline
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
### Environment
|
|
39
|
+
|
|
40
|
+
> **Already set up the project environment?** The cell below is a no-op — packages are already in your kernel. See the `setup` (Setup Reference) for details.
|
|
41
|
+
>
|
|
42
|
+
> **Running standalone** (Colab, Colab Enterprise, Vertex AI Workbench)? The cell below installs required packages into your current kernel.
|
|
43
|
+
|
|
44
|
+
```python
|
|
45
|
+
from google.cloud import bigquery
|
|
46
|
+
import pandas as pd
|
|
47
|
+
|
|
48
|
+
client = bigquery.Client(project=PROJECT_ID)
|
|
49
|
+
pd.set_option('display.max_colwidth', None)
|
|
50
|
+
|
|
51
|
+
# Create the shared dataset (idempotent)
|
|
52
|
+
dataset_ref = bigquery.DatasetReference(PROJECT_ID, DATASET_ID)
|
|
53
|
+
dataset = bigquery.Dataset(dataset_ref)
|
|
54
|
+
dataset.location = LOCATION
|
|
55
|
+
client.create_dataset(dataset, exists_ok=True)
|
|
56
|
+
print(f'Dataset {PROJECT_ID}.{DATASET_ID} ready')
|
|
57
|
+
|
|
58
|
+
# Register %%bigquery cell magic (auto-loaded in Colab, needed elsewhere)
|
|
59
|
+
%load_ext bigquery_magics
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
## Step 1 — Create the pipeline with `CREATE MODEL`
|
|
64
|
+
|
|
65
|
+
`model_type = 'TRANSFORM_ONLY'` is the only option that applies here — no `input_label_cols`, no learning options (any would error, since there's no estimator to configure). Every transformed column needs an explicit alias; `species` and `island` pass through untransformed.
|
|
66
|
+
|
|
67
|
+
- `ML.IMPUTER(body_mass_g, 'mean')` fills the 2 rows with a missing `body_mass_g`.
|
|
68
|
+
- `ML.STANDARD_SCALER` z-scores `culmen_length_mm`/`culmen_depth_mm`.
|
|
69
|
+
- `ML.ROBUST_SCALER` scales `flipper_length_mm` by median/IQR instead (outlier-robust).
|
|
70
|
+
- `ML.ONE_HOT_ENCODER(sex)` turns the 3-category (`MALE`/`FEMALE`/`NULL`) string into a sparse vector.
|
|
71
|
+
|
|
72
|
+
All four are **analytic** functions and require the empty `OVER()` — `ML.IMPUTER` additionally requires a `strategy` argument (`'mean'`/`'median'`/`'most_frequent'`), unlike the scalers, which take only the column.
|
|
73
|
+
|
|
74
|
+
```python
|
|
75
|
+
query = f"""
|
|
76
|
+
CREATE OR REPLACE MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_penguins`
|
|
77
|
+
TRANSFORM(
|
|
78
|
+
species,
|
|
79
|
+
island,
|
|
80
|
+
ML.IMPUTER(body_mass_g, 'mean') OVER() AS body_mass_g,
|
|
81
|
+
ML.STANDARD_SCALER(culmen_length_mm) OVER() AS culmen_length_mm,
|
|
82
|
+
ML.STANDARD_SCALER(culmen_depth_mm) OVER() AS culmen_depth_mm,
|
|
83
|
+
ML.ROBUST_SCALER(flipper_length_mm) OVER() AS flipper_length_mm,
|
|
84
|
+
ML.ONE_HOT_ENCODER(sex) OVER() AS sex_encoded
|
|
85
|
+
)
|
|
86
|
+
OPTIONS(
|
|
87
|
+
model_type = 'TRANSFORM_ONLY'
|
|
88
|
+
) AS
|
|
89
|
+
SELECT species, island, sex, body_mass_g, culmen_length_mm, culmen_depth_mm, flipper_length_mm
|
|
90
|
+
FROM `bigquery-public-data.ml_datasets.penguins`
|
|
91
|
+
"""
|
|
92
|
+
client.query(query).result()
|
|
93
|
+
print('Model transform_only_penguins created')
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
## Step 2 — Apply the pipeline with `ML.TRANSFORM`
|
|
98
|
+
|
|
99
|
+
Returns the columns the `TRANSFORM` clause produces for the inputs it consumes.
|
|
100
|
+
|
|
101
|
+
> **GOTCHA (verified):** any input column whose name isn't reused as an output alias in the `TRANSFORM` clause passes straight through **untouched**, appended after the transform outputs — here that's raw `sex`, which the pipeline consumes (`ML.ONE_HOT_ENCODER(sex) OVER() AS sex_encoded`) but never re-emits under its own name, so it comes through a second time as-is right alongside `sex_encoded`. Useful for carrying an id/label column through without re-listing it in the pipeline, but easy to mistake for the pipeline itself re-emitting a raw column — it's really just unused-by-name input passed through.
|
|
102
|
+
|
|
103
|
+
```python
|
|
104
|
+
query = f"""
|
|
105
|
+
SELECT *
|
|
106
|
+
FROM ML.TRANSFORM(
|
|
107
|
+
MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_penguins`,
|
|
108
|
+
(SELECT species, island, sex, body_mass_g, culmen_length_mm, culmen_depth_mm, flipper_length_mm
|
|
109
|
+
FROM `bigquery-public-data.ml_datasets.penguins`
|
|
110
|
+
LIMIT 5)
|
|
111
|
+
)
|
|
112
|
+
"""
|
|
113
|
+
client.query(query).to_dataframe()
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
---
|
|
117
|
+
## Step 3 — Introspect the raw inputs with `ML.FEATURE_INFO`
|
|
118
|
+
|
|
119
|
+
Reports statistics on the **raw** input columns (not the transformed outputs) — e.g. `sex` shows `null_count=10`, confirming why `ML.ONE_HOT_ENCODER` needs to handle nulls, and `body_mass_g` shows `null_count=2`, confirming why `ML.IMPUTER` is in the pipeline at all.
|
|
120
|
+
|
|
121
|
+
```python
|
|
122
|
+
query = f"""
|
|
123
|
+
SELECT *
|
|
124
|
+
FROM ML.FEATURE_INFO(MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_penguins`)
|
|
125
|
+
"""
|
|
126
|
+
client.query(query).to_dataframe()
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
---
|
|
130
|
+
## Step 4 — Export the pipeline with `EXPORT MODEL`
|
|
131
|
+
|
|
132
|
+
A transform-only export contains no predictive weights — there's no estimator — just the preprocessing graph and the frozen statistics (means, stddevs, one-hot vocabulary) computed at creation time. It exports as a `transform/saved_model.pb` (plus `assets`/`variables`), a distinct layout from a trained estimator's export.
|
|
133
|
+
|
|
134
|
+
```python
|
|
135
|
+
query = f"""
|
|
136
|
+
EXPORT MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_penguins`
|
|
137
|
+
OPTIONS (URI = 'gs://{BUCKET}/bq_ml/transform_only/model')
|
|
138
|
+
"""
|
|
139
|
+
client.query(query).result()
|
|
140
|
+
print('Model exported')
|
|
141
|
+
|
|
142
|
+
from google.cloud import storage
|
|
143
|
+
storage_client = storage.Client(project=PROJECT_ID)
|
|
144
|
+
bucket = storage_client.bucket(BUCKET)
|
|
145
|
+
for blob in bucket.list_blobs(prefix='bq_ml/transform_only/model'):
|
|
146
|
+
print(blob.name)
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
---
|
|
150
|
+
## Step 5 — Feed the pipeline's output into a downstream `CREATE MODEL`
|
|
151
|
+
|
|
152
|
+
The downstream model has **no embedded `TRANSFORM` of its own** — it trains directly on the already-transformed columns coming out of `ML.TRANSFORM`. This is the feature-store-style pattern: one shared pipeline, reused across as many downstream models as you like, each choosing which transformed columns to consume.
|
|
153
|
+
|
|
154
|
+
```python
|
|
155
|
+
query = f"""
|
|
156
|
+
CREATE OR REPLACE MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_downstream`
|
|
157
|
+
OPTIONS(
|
|
158
|
+
model_type = 'LOGISTIC_REG',
|
|
159
|
+
input_label_cols = ['species']
|
|
160
|
+
) AS
|
|
161
|
+
SELECT species, island, body_mass_g, culmen_length_mm, culmen_depth_mm, flipper_length_mm
|
|
162
|
+
FROM ML.TRANSFORM(
|
|
163
|
+
MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_penguins`,
|
|
164
|
+
(SELECT species, island, sex, body_mass_g, culmen_length_mm, culmen_depth_mm, flipper_length_mm
|
|
165
|
+
FROM `bigquery-public-data.ml_datasets.penguins`
|
|
166
|
+
WHERE body_mass_g IS NOT NULL)
|
|
167
|
+
)
|
|
168
|
+
"""
|
|
169
|
+
client.query(query).result()
|
|
170
|
+
print('Model transform_only_downstream created')
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
---
|
|
174
|
+
## Step 6 — Evaluate the downstream model with `ML.EVALUATE`
|
|
175
|
+
|
|
176
|
+
```python
|
|
177
|
+
query = f"""
|
|
178
|
+
SELECT *
|
|
179
|
+
FROM ML.EVALUATE(MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_downstream`)
|
|
180
|
+
"""
|
|
181
|
+
client.query(query).to_dataframe()
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
---
|
|
185
|
+
## Step 7 — The gotcha: predicting on raw data silently gives garbage
|
|
186
|
+
|
|
187
|
+
Because the downstream model has no embedded `TRANSFORM`, `ML.PREDICT` does **not** know to re-apply the pipeline. Feeding it raw (untransformed) feature values doesn't error — BigQuery ML happily predicts using values on the wrong scale, it just gives **wrong** answers.
|
|
188
|
+
|
|
189
|
+
> **Verified live:** every row below predicts `Gentoo penguin` (the largest species) regardless of its true species, because raw `body_mass_g`/`culmen_*` values are far outside the z-score-scaled range the model was actually trained on.
|
|
190
|
+
|
|
191
|
+
```python
|
|
192
|
+
query = f"""
|
|
193
|
+
SELECT species, predicted_species
|
|
194
|
+
FROM ML.PREDICT(
|
|
195
|
+
MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_downstream`,
|
|
196
|
+
(SELECT species, island, body_mass_g, culmen_length_mm, culmen_depth_mm, flipper_length_mm
|
|
197
|
+
FROM `bigquery-public-data.ml_datasets.penguins`
|
|
198
|
+
WHERE body_mass_g IS NOT NULL LIMIT 5)
|
|
199
|
+
)
|
|
200
|
+
"""
|
|
201
|
+
client.query(query).to_dataframe()
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
## Step 8 — The fix: re-apply `ML.TRANSFORM` before `ML.PREDICT`
|
|
206
|
+
|
|
207
|
+
Wrapping the same raw rows in `ML.TRANSFORM` first reproduces the exact preprocessing used at training time, and predictions become correct again. This re-application is the price of a transform-only model's reusability — contrast with an **embedded** `TRANSFORM` clause on a predictive model (e.g. `models/pca` (PCA) Step 10), which auto-applies at predict time with no re-application needed, but isn't reusable across other models the way a standalone `TRANSFORM_ONLY` model is.
|
|
208
|
+
|
|
209
|
+
```python
|
|
210
|
+
query = f"""
|
|
211
|
+
SELECT species, predicted_species
|
|
212
|
+
FROM ML.PREDICT(
|
|
213
|
+
MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_downstream`,
|
|
214
|
+
(SELECT species, island, body_mass_g, culmen_length_mm, culmen_depth_mm, flipper_length_mm
|
|
215
|
+
FROM ML.TRANSFORM(
|
|
216
|
+
MODEL `{PROJECT_ID}.{DATASET_ID}.transform_only_penguins`,
|
|
217
|
+
(SELECT species, island, sex, body_mass_g, culmen_length_mm, culmen_depth_mm, flipper_length_mm
|
|
218
|
+
FROM `bigquery-public-data.ml_datasets.penguins`
|
|
219
|
+
WHERE body_mass_g IS NOT NULL LIMIT 5)
|
|
220
|
+
)
|
|
221
|
+
)
|
|
222
|
+
)
|
|
223
|
+
"""
|
|
224
|
+
client.query(query).to_dataframe()
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
## Examples — `%%bigquery` Magics
|
|
229
|
+
|
|
230
|
+
The same operations using IPython magic commands — write SQL directly in cells without Python string wrapping.
|
|
231
|
+
|
|
232
|
+
- `%%bigquery` — run SQL, display results inline
|
|
233
|
+
- `%%bigquery df` — run SQL, capture results into a pandas DataFrame
|
|
234
|
+
|
|
235
|
+
### `ML.FEATURE_INFO` with `%%bigquery`
|
|
236
|
+
|
|
237
|
+
```sql
|
|
238
|
+
%%bigquery --project {PROJECT_ID}
|
|
239
|
+
|
|
240
|
+
SELECT *
|
|
241
|
+
FROM ML.FEATURE_INFO(MODEL `statmike-mlops-349915.bq_ml.transform_only_penguins`)
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
## Examples — BigFrames
|
|
246
|
+
|
|
247
|
+
There is **no single direct `TRANSFORM_ONLY` class** in BigFrames — the equivalent functionality is the `bigframes.ml.preprocessing` transformers (`StandardScaler`, `MaxAbsScaler`, `OneHotEncoder`, etc.) composed with `bigframes.ml.pipeline.Pipeline`, which compile down to the same BQML preprocessing functions under the hood. This shows the closest equivalent: a two-step `Pipeline` (impute, then scale) — but note it does not produce a queryable `TRANSFORM_ONLY` model object the way the SQL path does; it's a Python-side pipeline that gets applied at `.fit()`/`.predict()` time.
|
|
248
|
+
|
|
249
|
+
```python
|
|
250
|
+
import bigframes.pandas as bpd
|
|
251
|
+
|
|
252
|
+
bpd.close_session() # Reset session to apply project/location settings
|
|
253
|
+
bpd.options.bigquery.project = PROJECT_ID
|
|
254
|
+
bpd.options.bigquery.location = LOCATION
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
```python
|
|
258
|
+
from bigframes.ml.preprocessing import StandardScaler
|
|
259
|
+
from bigframes.ml.impute import SimpleImputer
|
|
260
|
+
from bigframes.ml.compose import ColumnTransformer
|
|
261
|
+
|
|
262
|
+
df = bpd.read_gbq('bigquery-public-data.ml_datasets.penguins')
|
|
263
|
+
df = df[df['body_mass_g'].notnull()]
|
|
264
|
+
|
|
265
|
+
# Impute then scale, composed as a ColumnTransformer (closest BigFrames equivalent
|
|
266
|
+
# to a reusable TRANSFORM_ONLY pipeline)
|
|
267
|
+
preprocessor = ColumnTransformer([
|
|
268
|
+
('scale_length', StandardScaler(), 'culmen_length_mm'),
|
|
269
|
+
('scale_depth', StandardScaler(), 'culmen_depth_mm'),
|
|
270
|
+
])
|
|
271
|
+
preprocessor.fit(df[['culmen_length_mm', 'culmen_depth_mm']])
|
|
272
|
+
preprocessor.transform(df[['culmen_length_mm', 'culmen_depth_mm']]).peek()
|
|
273
|
+
```
|
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
> **Source (`statmike/bigquery-data-science`):** **Notebook:** [`bigquery-ml/workflows/uplift_cate/uplift_cate.ipynb`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/uplift_cate/uplift_cate.ipynb) · **SQL:** [`bigquery-ml/workflows/uplift_cate/uplift_cate.sql`](https://github.com/statmike/bigquery-data-science/blob/main/bigquery-ml/workflows/uplift_cate/uplift_cate.sql)
|
|
2
|
+
|
|
3
|
+
# Uplift Modeling / CATE (T-Learner) — BigQuery ML
|
|
4
|
+
|
|
5
|
+
**An average effect can hide the only number that matters for targeting.** `workflows/propensity_score_matching` (`propensity_score_matching`) estimated *one* treatment effect for an entire population. Real targeting decisions need more: *which* users actually respond to a treatment? Uplift modeling estimates the **Conditional Average Treatment Effect (CATE)** — a per-individual (or per-segment) treatment effect — so you can target the people the treatment actually moves, not everyone.
|
|
6
|
+
|
|
7
|
+
**Models used:** `BOOSTED_TREE_CLASSIFIER` (a T-learner: two independent models)
|
|
8
|
+
**Functions used:** `ML.PREDICT`
|
|
9
|
+
**Data:** [`bigquery-public-data.google_analytics_sample`](https://console.cloud.google.com/marketplace/product/bigquery-public-datasets) — the Google Merchandise Store's real Universal Analytics (GA360) export, 2016-08-01 to 2017-08-01.
|
|
10
|
+
**References:** `RESOURCES.md` (Full reference) | [CREATE MODEL (Boosted Tree) docs](https://cloud.google.com/bigquery/docs/reference/standard-sql/bigqueryml-syntax-create-boosted-tree) | `setup` (Setup guide)
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Why this, beyond a simple average effect
|
|
15
|
+
|
|
16
|
+
Uplift modeling (estimating CATE) is genuinely current production practice — Uber's `causalml` and Microsoft's `EconML` libraries exist specifically for this, and ad-tech/growth teams use it to decide *who* to target with a limited marketing budget, not just *whether* a campaign works on average. The simplest real implementation is a **T-learner**:
|
|
17
|
+
|
|
18
|
+
1. Split the population into treated and control groups (here: sessions from paid marketing vs. organic search).
|
|
19
|
+
2. Fit **two separate models**, each predicting the outcome (did this session convert?) — one trained *only* on treated sessions, one trained *only* on control sessions.
|
|
20
|
+
3. Score **every** session with **both** models, regardless of which group it actually belonged to.
|
|
21
|
+
4. `CATE = P(convert | treated model) − P(convert | control model)` — the model's estimate of how much *this specific session* would have benefited from the treatment.
|
|
22
|
+
|
|
23
|
+
This is fully native to BigQuery ML: two `BOOSTED_TREE_CLASSIFIER` models and two `ML.PREDICT` calls.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
## Setup
|
|
27
|
+
|
|
28
|
+
Set your project and location, authenticate, and create a shared dataset.
|
|
29
|
+
|
|
30
|
+
```python
|
|
31
|
+
PROJECT_ID = 'statmike-mlops-349915' # <-- Replace with your project ID
|
|
32
|
+
LOCATION = 'US' # BigQuery dataset location
|
|
33
|
+
DATASET_ID = 'bq_ml' # Shared dataset across all bigquery-ml notebooks
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
### Environment
|
|
37
|
+
|
|
38
|
+
> **Already set up the project environment?** The cell below is a no-op — packages are already in your kernel. See the `setup` (Setup Reference) for details.
|
|
39
|
+
>
|
|
40
|
+
> **Running standalone** (Colab, Colab Enterprise, Vertex AI Workbench)? The cell below installs required packages into your current kernel.
|
|
41
|
+
|
|
42
|
+
```python
|
|
43
|
+
from google.cloud import bigquery
|
|
44
|
+
import pandas as pd
|
|
45
|
+
import matplotlib.pyplot as plt
|
|
46
|
+
|
|
47
|
+
client = bigquery.Client(project=PROJECT_ID)
|
|
48
|
+
pd.set_option('display.max_colwidth', None)
|
|
49
|
+
|
|
50
|
+
# Create the shared dataset (idempotent)
|
|
51
|
+
dataset_ref = bigquery.DatasetReference(PROJECT_ID, DATASET_ID)
|
|
52
|
+
dataset = bigquery.Dataset(dataset_ref)
|
|
53
|
+
dataset.location = LOCATION
|
|
54
|
+
client.create_dataset(dataset, exists_ok=True)
|
|
55
|
+
print(f'Dataset {PROJECT_ID}.{DATASET_ID} ready')
|
|
56
|
+
|
|
57
|
+
# Register %%bigquery cell magic (auto-loaded in Colab, needed elsewhere)
|
|
58
|
+
%load_ext bigquery_magics
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
## Step 1 — Build the session cohort: a real, observational treatment
|
|
63
|
+
|
|
64
|
+
**Treatment:** did the session arrive via paid marketing (`Paid Search` or `Display`) = 1, vs. organic search = 0 (control) — real, observational (not a randomized experiment; matches this project's `propensity_score_matching` precedent for treating observational data honestly). **Outcome:** did the session convert (`totals.transactions >= 1`). **Covariates:** device category, new-vs-returning visitor, visit number, operating system, country — all pre-treatment attributes usable for both modeling and later segmentation. A synthetic `row_id` is added since `fullVisitorId`+`visitId` isn't a perfectly unique session key in this public sample (verified live: 903,653 rows vs. 902,755 distinct combinations).
|
|
65
|
+
|
|
66
|
+
```python
|
|
67
|
+
query = f"""
|
|
68
|
+
CREATE OR REPLACE TABLE `{PROJECT_ID}.{DATASET_ID}.uplift_cohort` AS
|
|
69
|
+
SELECT
|
|
70
|
+
ROW_NUMBER() OVER() AS row_id,
|
|
71
|
+
treatment, device_category, visitor_type, visit_number, os, country, purchased
|
|
72
|
+
FROM (
|
|
73
|
+
SELECT
|
|
74
|
+
CASE WHEN channelGrouping IN ('Paid Search', 'Display') THEN 1
|
|
75
|
+
WHEN channelGrouping = 'Organic Search' THEN 0
|
|
76
|
+
ELSE NULL END AS treatment,
|
|
77
|
+
device.deviceCategory AS device_category,
|
|
78
|
+
IF(IFNULL(totals.newVisits, 0) = 1, 'new_visitor', 'returning_visitor') AS visitor_type,
|
|
79
|
+
IFNULL(visitNumber, 1) AS visit_number,
|
|
80
|
+
device.operatingSystem AS os,
|
|
81
|
+
geoNetwork.country AS country,
|
|
82
|
+
IF(totals.transactions >= 1, 1, 0) AS purchased
|
|
83
|
+
FROM `bigquery-public-data.google_analytics_sample.ga_sessions_*`
|
|
84
|
+
WHERE _TABLE_SUFFIX BETWEEN '20160801' AND '20170801'
|
|
85
|
+
)
|
|
86
|
+
"""
|
|
87
|
+
client.query(query).result()
|
|
88
|
+
|
|
89
|
+
query = f"""
|
|
90
|
+
SELECT treatment, COUNT(*) AS n, ROUND(AVG(purchased) * 100, 3) AS conv_pct
|
|
91
|
+
FROM `{PROJECT_ID}.{DATASET_ID}.uplift_cohort`
|
|
92
|
+
WHERE treatment IS NOT NULL
|
|
93
|
+
GROUP BY treatment
|
|
94
|
+
ORDER BY treatment
|
|
95
|
+
"""
|
|
96
|
+
client.query(query).to_dataframe()
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
**Verified finding:** 381,561 organic (control) sessions convert at 0.902%; 31,588 paid (treated) sessions convert at 1.937% — a real ~2.1x average lift. **This is the number a normal targeting model would stop at.**
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
## Step 2 — Naive average effect: real, but hides everything a targeting decision needs
|
|
103
|
+
|
|
104
|
+
The 2.1x lift above is real, but it answers "does paid marketing work on average," not "who should I actually target." Segment it by device and visitor type to see whether that average is hiding real variation.
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
query = f"""
|
|
108
|
+
SELECT device_category, treatment, COUNT(*) AS n, ROUND(AVG(purchased) * 100, 3) AS conv_pct
|
|
109
|
+
FROM `{PROJECT_ID}.{DATASET_ID}.uplift_cohort`
|
|
110
|
+
WHERE treatment IS NOT NULL
|
|
111
|
+
GROUP BY device_category, treatment
|
|
112
|
+
ORDER BY device_category, treatment
|
|
113
|
+
"""
|
|
114
|
+
by_device = client.query(query).to_dataframe()
|
|
115
|
+
by_device
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
**Verified finding — real heterogeneity, not a flat effect:** desktop's absolute uplift (control 1.149% → treated 2.860%, a +1.71pp gain) is roughly **10x tablet's** (control 0.589% → treated 0.751%, +0.16pp). A campaign spend decision based only on the pooled 2.1x average would badly misallocate budget across these segments. This is exactly the gap a CATE model is built to quantify per-session, not just per pre-defined segment.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
## Step 3 — T-learner: two models, one per treatment arm
|
|
122
|
+
|
|
123
|
+
`Model_treated` is fit **only** on treated (paid) sessions; `Model_control` is fit **only** on control (organic) sessions — both predicting `purchased` from the same covariates. Every session then gets scored by **both** models (`ML.PREDICT` over the full cohort, not just each model's own training arm) to estimate what *would* have happened under either condition.
|
|
124
|
+
|
|
125
|
+
```python
|
|
126
|
+
query = f"""
|
|
127
|
+
CREATE OR REPLACE MODEL `{PROJECT_ID}.{DATASET_ID}.uplift_model_treated`
|
|
128
|
+
OPTIONS(model_type = 'BOOSTED_TREE_CLASSIFIER', input_label_cols = ['purchased'], data_split_method = 'NO_SPLIT') AS
|
|
129
|
+
SELECT device_category, visitor_type, visit_number, os, country, purchased
|
|
130
|
+
FROM `{PROJECT_ID}.{DATASET_ID}.uplift_cohort`
|
|
131
|
+
WHERE treatment = 1
|
|
132
|
+
"""
|
|
133
|
+
client.query(query).result()
|
|
134
|
+
|
|
135
|
+
query = f"""
|
|
136
|
+
CREATE OR REPLACE MODEL `{PROJECT_ID}.{DATASET_ID}.uplift_model_control`
|
|
137
|
+
OPTIONS(model_type = 'BOOSTED_TREE_CLASSIFIER', input_label_cols = ['purchased'], data_split_method = 'NO_SPLIT') AS
|
|
138
|
+
SELECT device_category, visitor_type, visit_number, os, country, purchased
|
|
139
|
+
FROM `{PROJECT_ID}.{DATASET_ID}.uplift_cohort`
|
|
140
|
+
WHERE treatment = 0
|
|
141
|
+
"""
|
|
142
|
+
client.query(query).result()
|
|
143
|
+
print('Both T-learner arms trained')
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
```python
|
|
147
|
+
query = f"""
|
|
148
|
+
CREATE OR REPLACE TABLE `{PROJECT_ID}.{DATASET_ID}.uplift_scored` AS
|
|
149
|
+
SELECT
|
|
150
|
+
c.row_id, c.treatment, c.device_category, c.visitor_type, c.purchased,
|
|
151
|
+
(SELECT prob FROM UNNEST(pt.predicted_purchased_probs) WHERE label = 1) AS p_treated,
|
|
152
|
+
(SELECT prob FROM UNNEST(pc.predicted_purchased_probs) WHERE label = 1) AS p_control
|
|
153
|
+
FROM `{PROJECT_ID}.{DATASET_ID}.uplift_cohort` c
|
|
154
|
+
JOIN ML.PREDICT(MODEL `{PROJECT_ID}.{DATASET_ID}.uplift_model_treated`, TABLE `{PROJECT_ID}.{DATASET_ID}.uplift_cohort`) pt USING (row_id)
|
|
155
|
+
JOIN ML.PREDICT(MODEL `{PROJECT_ID}.{DATASET_ID}.uplift_model_control`, TABLE `{PROJECT_ID}.{DATASET_ID}.uplift_cohort`) pc USING (row_id)
|
|
156
|
+
WHERE c.treatment IS NOT NULL
|
|
157
|
+
"""
|
|
158
|
+
client.query(query).result()
|
|
159
|
+
|
|
160
|
+
query = f"SELECT COUNT(*) AS n FROM `{PROJECT_ID}.{DATASET_ID}.uplift_scored`"
|
|
161
|
+
client.query(query).to_dataframe()
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
**Gotcha found live:** `predicted_purchased_probs` is not guaranteed sorted by ascending label value — verified this model's output orders it `[label='1', label='0']`, not `[label='0', label='1']`. A positional `[OFFSET(0)]` would silently grab the wrong class's probability if that ordering ever differs run to run. Filtering explicitly with `WHERE label = 1` (the same safe pattern already used in `workflows/propensity_score_matching` (`propensity_score_matching`)) avoids the risk entirely.
|
|
165
|
+
|
|
166
|
+
413,149 sessions, every one now scored by both arms. `CATE = p_treated - p_control` per session.
|
|
167
|
+
|
|
168
|
+
---
|
|
169
|
+
## Step 4 — Show the heterogeneity visually: the Qini curve
|
|
170
|
+
|
|
171
|
+
The standard uplift-modeling evaluation visual: rank every session by predicted `CATE` descending, then plot the **cumulative actual incremental conversions captured** as you move down that ranking, against a random-targeting baseline. A model with real discriminative power pulls above the diagonal — if you could only afford to target the top 10%, this tells you whether picking by predicted CATE actually beats picking at random.
|
|
172
|
+
|
|
173
|
+
```python
|
|
174
|
+
query = f"""
|
|
175
|
+
SELECT treatment, purchased, p_treated - p_control AS cate
|
|
176
|
+
FROM `{PROJECT_ID}.{DATASET_ID}.uplift_scored`
|
|
177
|
+
"""
|
|
178
|
+
scored_df = client.query(query).to_dataframe()
|
|
179
|
+
scored_df = scored_df.sort_values('cate', ascending=False).reset_index(drop=True)
|
|
180
|
+
|
|
181
|
+
n_treated_total = (scored_df['treatment'] == 1).sum()
|
|
182
|
+
n_control_total = (scored_df['treatment'] == 0).sum()
|
|
183
|
+
|
|
184
|
+
scored_df['cum_treated_purchases'] = (scored_df['treatment'] * scored_df['purchased']).cumsum()
|
|
185
|
+
scored_df['cum_control_purchases'] = ((1 - scored_df['treatment']) * scored_df['purchased']).cumsum()
|
|
186
|
+
scored_df['cum_n_treated'] = scored_df['treatment'].cumsum()
|
|
187
|
+
scored_df['cum_n_control'] = (1 - scored_df['treatment']).cumsum()
|
|
188
|
+
|
|
189
|
+
# Qini: incremental conversions captured, control scaled up to the treated group's size
|
|
190
|
+
scaling = scored_df['cum_n_treated'] / scored_df['cum_n_control'].replace(0, pd.NA)
|
|
191
|
+
scored_df['qini'] = scored_df['cum_treated_purchases'] - scored_df['cum_control_purchases'] * scaling
|
|
192
|
+
scored_df['pct_targeted'] = (scored_df.index + 1) / len(scored_df)
|
|
193
|
+
|
|
194
|
+
total_incremental = scored_df['qini'].iloc[-1]
|
|
195
|
+
random_line = scored_df['pct_targeted'] * total_incremental
|
|
196
|
+
|
|
197
|
+
fig, ax = plt.subplots(figsize=(8, 5))
|
|
198
|
+
ax.plot(scored_df['pct_targeted'], scored_df['qini'], label='Targeting by predicted CATE', color='darkorange')
|
|
199
|
+
ax.plot(scored_df['pct_targeted'], random_line, label='Random targeting', color='gray', linestyle='--')
|
|
200
|
+
ax.set_xlabel('% of population targeted (ranked by predicted CATE, descending)')
|
|
201
|
+
ax.set_ylabel('Cumulative incremental conversions captured')
|
|
202
|
+
ax.set_title('Qini curve: does targeting by predicted CATE beat random?')
|
|
203
|
+
ax.legend()
|
|
204
|
+
plt.show()
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
**Verified finding — real, front-loaded separation, not a textbook-perfect curve:** the CATE-ranked curve pulls clearly above the random-targeting diagonal across roughly the top 30% of the population — exactly where it matters most for a real budget-constrained targeting decision — then tracks close to the diagonal through the middle before both lines meet at 100% (as they must, by construction). Shown honestly rather than smoothed into a uniformly-above-diagonal shape it doesn't have. Segment-level check (from Step 2's device/visitor-type table, computed directly rather than from the model): the top predicted-CATE decile shows a +1.59pp actual uplift gap (above the population average of +1.04pp), while the bottom decile shows a **negative** gap — some sessions are predicted to respond worse to paid marketing than to organic, a real and actionable signal a single average effect could never surface.
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
## Step 5 — Honest finding
|
|
211
|
+
|
|
212
|
+
**Real, actionable heterogeneity exists**: desktop and new-visitor sessions respond far more to paid marketing than tablet and returning-visitor sessions, and the T-learner's per-session CATE ranking (visualized above via the Qini curve) captures this well enough to genuinely outperform random targeting — targeting the top predicted-CATE segment captures disproportionately more of the real incremental lift than targeting everyone or targeting randomly.
|
|
213
|
+
|
|
214
|
+
**A real limitation of the T-learner specifically, stated plainly:** fitting two *entirely separate* models (rather than one model that directly estimates the treatment-effect contrast) is known to be more bias-prone than more advanced meta-learners — particularly the **X-learner** and doubly-robust **R-learner** — especially when, as here, the treatment and control groups are very different sizes (31,588 vs. 381,561, a ~12:1 imbalance). Libraries like `causalml` (Uber) and `EconML` (Microsoft) implement these more robust estimators. This notebook doesn't need them to make its point: the T-learner's own diagnostic — does its ranking actually separate high- from low-responders, verified via the Qini curve above — is what matters for demonstrating that uplift modeling works, even though a production targeting system built on very imbalanced arms would likely reach for a more robust meta-learner.
|
|
215
|
+
|
|
216
|
+
---
|
|
217
|
+
## Related content
|
|
218
|
+
|
|
219
|
+
- `workflows/propensity_score_matching` (`workflows/propensity_score_matching/`) — estimates one *average* treatment effect from observational data; this workflow's natural prerequisite and its complement (average effect vs. heterogeneous/individual effect).
|
|
220
|
+
- `models/boosted_tree_classifier` (`models/boosted_tree_classifier/`) — `BOOSTED_TREE_CLASSIFIER` mechanics in depth (used here twice, once per treatment arm, rather than in its usual single-model predictive role).
|
|
221
|
+
- `workflows/ga4_churn_prediction` (`workflows/ga4_churn_prediction/`) — a different Google Analytics-based workflow (the newer GA4 export, not this notebook's Universal Analytics sample) using first-week activity as its dominant signal, the same kind of behavioral covariate used here.
|
|
222
|
+
|
|
223
|
+
---
|
|
224
|
+
## Examples — `%%bigquery` Magics
|
|
225
|
+
|
|
226
|
+
The same treatment/outcome summary using IPython magic commands.
|
|
227
|
+
|
|
228
|
+
```sql
|
|
229
|
+
%%bigquery --project {PROJECT_ID}
|
|
230
|
+
|
|
231
|
+
SELECT treatment, COUNT(*) AS n, ROUND(AVG(purchased) * 100, 3) AS conv_pct
|
|
232
|
+
FROM `statmike-mlops-349915.bq_ml.uplift_cohort`
|
|
233
|
+
WHERE treatment IS NOT NULL
|
|
234
|
+
GROUP BY treatment
|
|
235
|
+
ORDER BY treatment
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
## Examples — BigFrames
|
|
240
|
+
|
|
241
|
+
`bigframes.ml.ensemble.XGBClassifier` is a valid drop-in for both T-learner arms (same pattern demonstrated in `models/boosted_tree_classifier` (`models/boosted_tree_classifier/`)) — the CATE subtraction and Qini-curve computation would still be plain pandas either way, same as this project's other multi-technique workflows (`workflows/propensity_score_matching` (`propensity_score_matching`), `workflows/survival_analysis` (`survival_analysis`)).
|