snowflake-ml-python 1.0.1__py3-none-any.whl → 1.0.3__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- snowflake/ml/_internal/env_utils.py +2 -1
- snowflake/ml/_internal/file_utils.py +35 -40
- snowflake/ml/_internal/telemetry.py +5 -8
- snowflake/ml/_internal/utils/identifier.py +74 -7
- snowflake/ml/_internal/utils/uri.py +7 -2
- snowflake/ml/model/_core_requirements.py +1 -1
- snowflake/ml/model/_deploy_client/image_builds/base_image_builder.py +15 -0
- snowflake/ml/model/_deploy_client/image_builds/client_image_builder.py +259 -0
- snowflake/ml/model/_deploy_client/image_builds/docker_context.py +89 -0
- snowflake/ml/model/_deploy_client/image_builds/gunicorn_run.sh +24 -0
- snowflake/ml/model/_deploy_client/image_builds/inference_server/main.py +118 -0
- snowflake/ml/model/_deploy_client/image_builds/templates/dockerfile_template +40 -0
- snowflake/ml/model/_deploy_client/snowservice/deploy.py +199 -0
- snowflake/ml/model/_deploy_client/snowservice/deploy_options.py +88 -0
- snowflake/ml/model/_deploy_client/snowservice/templates/service_spec_template +24 -0
- snowflake/ml/model/_deploy_client/utils/constants.py +47 -0
- snowflake/ml/model/_deploy_client/utils/snowservice_client.py +178 -0
- snowflake/ml/model/_deploy_client/warehouse/deploy.py +25 -28
- snowflake/ml/model/_deploy_client/warehouse/infer_template.py +7 -4
- snowflake/ml/model/_deployer.py +14 -27
- snowflake/ml/model/_env.py +4 -4
- snowflake/ml/model/_handlers/_base.py +3 -1
- snowflake/ml/model/_handlers/custom.py +14 -2
- snowflake/ml/model/_handlers/pytorch.py +186 -0
- snowflake/ml/model/_handlers/sklearn.py +14 -8
- snowflake/ml/model/_handlers/snowmlmodel.py +14 -9
- snowflake/ml/model/_handlers/torchscript.py +180 -0
- snowflake/ml/model/_handlers/xgboost.py +19 -9
- snowflake/ml/model/_model.py +27 -21
- snowflake/ml/model/_model_meta.py +33 -19
- snowflake/ml/model/model_signature.py +446 -66
- snowflake/ml/model/type_hints.py +28 -15
- snowflake/ml/modeling/calibration/calibrated_classifier_cv.py +79 -43
- snowflake/ml/modeling/cluster/affinity_propagation.py +79 -43
- snowflake/ml/modeling/cluster/agglomerative_clustering.py +79 -43
- snowflake/ml/modeling/cluster/birch.py +79 -43
- snowflake/ml/modeling/cluster/bisecting_k_means.py +79 -43
- snowflake/ml/modeling/cluster/dbscan.py +79 -43
- snowflake/ml/modeling/cluster/feature_agglomeration.py +79 -43
- snowflake/ml/modeling/cluster/k_means.py +79 -43
- snowflake/ml/modeling/cluster/mean_shift.py +79 -43
- snowflake/ml/modeling/cluster/mini_batch_k_means.py +79 -43
- snowflake/ml/modeling/cluster/optics.py +79 -43
- snowflake/ml/modeling/cluster/spectral_biclustering.py +79 -43
- snowflake/ml/modeling/cluster/spectral_clustering.py +79 -43
- snowflake/ml/modeling/cluster/spectral_coclustering.py +79 -43
- snowflake/ml/modeling/compose/column_transformer.py +79 -43
- snowflake/ml/modeling/compose/transformed_target_regressor.py +79 -43
- snowflake/ml/modeling/covariance/elliptic_envelope.py +79 -43
- snowflake/ml/modeling/covariance/empirical_covariance.py +79 -43
- snowflake/ml/modeling/covariance/graphical_lasso.py +79 -43
- snowflake/ml/modeling/covariance/graphical_lasso_cv.py +79 -43
- snowflake/ml/modeling/covariance/ledoit_wolf.py +79 -43
- snowflake/ml/modeling/covariance/min_cov_det.py +79 -43
- snowflake/ml/modeling/covariance/oas.py +79 -43
- snowflake/ml/modeling/covariance/shrunk_covariance.py +79 -43
- snowflake/ml/modeling/decomposition/dictionary_learning.py +79 -43
- snowflake/ml/modeling/decomposition/factor_analysis.py +79 -43
- snowflake/ml/modeling/decomposition/fast_ica.py +79 -43
- snowflake/ml/modeling/decomposition/incremental_pca.py +79 -43
- snowflake/ml/modeling/decomposition/kernel_pca.py +79 -43
- snowflake/ml/modeling/decomposition/mini_batch_dictionary_learning.py +79 -43
- snowflake/ml/modeling/decomposition/mini_batch_sparse_pca.py +79 -43
- snowflake/ml/modeling/decomposition/pca.py +79 -43
- snowflake/ml/modeling/decomposition/sparse_pca.py +79 -43
- snowflake/ml/modeling/decomposition/truncated_svd.py +79 -43
- snowflake/ml/modeling/discriminant_analysis/linear_discriminant_analysis.py +79 -43
- snowflake/ml/modeling/discriminant_analysis/quadratic_discriminant_analysis.py +79 -43
- snowflake/ml/modeling/ensemble/ada_boost_classifier.py +79 -43
- snowflake/ml/modeling/ensemble/ada_boost_regressor.py +79 -43
- snowflake/ml/modeling/ensemble/bagging_classifier.py +79 -43
- snowflake/ml/modeling/ensemble/bagging_regressor.py +79 -43
- snowflake/ml/modeling/ensemble/extra_trees_classifier.py +79 -43
- snowflake/ml/modeling/ensemble/extra_trees_regressor.py +79 -43
- snowflake/ml/modeling/ensemble/gradient_boosting_classifier.py +79 -43
- snowflake/ml/modeling/ensemble/gradient_boosting_regressor.py +79 -43
- snowflake/ml/modeling/ensemble/hist_gradient_boosting_classifier.py +79 -43
- snowflake/ml/modeling/ensemble/hist_gradient_boosting_regressor.py +79 -43
- snowflake/ml/modeling/ensemble/isolation_forest.py +79 -43
- snowflake/ml/modeling/ensemble/random_forest_classifier.py +79 -43
- snowflake/ml/modeling/ensemble/random_forest_regressor.py +79 -43
- snowflake/ml/modeling/ensemble/stacking_regressor.py +79 -43
- snowflake/ml/modeling/ensemble/voting_classifier.py +79 -43
- snowflake/ml/modeling/ensemble/voting_regressor.py +79 -43
- snowflake/ml/modeling/feature_selection/generic_univariate_select.py +79 -43
- snowflake/ml/modeling/feature_selection/select_fdr.py +79 -43
- snowflake/ml/modeling/feature_selection/select_fpr.py +79 -43
- snowflake/ml/modeling/feature_selection/select_fwe.py +79 -43
- snowflake/ml/modeling/feature_selection/select_k_best.py +79 -43
- snowflake/ml/modeling/feature_selection/select_percentile.py +79 -43
- snowflake/ml/modeling/feature_selection/sequential_feature_selector.py +79 -43
- snowflake/ml/modeling/feature_selection/variance_threshold.py +79 -43
- snowflake/ml/modeling/gaussian_process/gaussian_process_classifier.py +79 -43
- snowflake/ml/modeling/gaussian_process/gaussian_process_regressor.py +79 -43
- snowflake/ml/modeling/impute/iterative_imputer.py +79 -43
- snowflake/ml/modeling/impute/knn_imputer.py +79 -43
- snowflake/ml/modeling/impute/missing_indicator.py +79 -43
- snowflake/ml/modeling/kernel_approximation/additive_chi2_sampler.py +79 -43
- snowflake/ml/modeling/kernel_approximation/nystroem.py +79 -43
- snowflake/ml/modeling/kernel_approximation/polynomial_count_sketch.py +79 -43
- snowflake/ml/modeling/kernel_approximation/rbf_sampler.py +79 -43
- snowflake/ml/modeling/kernel_approximation/skewed_chi2_sampler.py +79 -43
- snowflake/ml/modeling/kernel_ridge/kernel_ridge.py +79 -43
- snowflake/ml/modeling/lightgbm/lgbm_classifier.py +79 -43
- snowflake/ml/modeling/lightgbm/lgbm_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/ard_regression.py +79 -43
- snowflake/ml/modeling/linear_model/bayesian_ridge.py +79 -43
- snowflake/ml/modeling/linear_model/elastic_net.py +79 -43
- snowflake/ml/modeling/linear_model/elastic_net_cv.py +79 -43
- snowflake/ml/modeling/linear_model/gamma_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/huber_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/lars.py +79 -43
- snowflake/ml/modeling/linear_model/lars_cv.py +79 -43
- snowflake/ml/modeling/linear_model/lasso.py +79 -43
- snowflake/ml/modeling/linear_model/lasso_cv.py +79 -43
- snowflake/ml/modeling/linear_model/lasso_lars.py +79 -43
- snowflake/ml/modeling/linear_model/lasso_lars_cv.py +79 -43
- snowflake/ml/modeling/linear_model/lasso_lars_ic.py +79 -43
- snowflake/ml/modeling/linear_model/linear_regression.py +79 -43
- snowflake/ml/modeling/linear_model/logistic_regression.py +79 -43
- snowflake/ml/modeling/linear_model/logistic_regression_cv.py +79 -43
- snowflake/ml/modeling/linear_model/multi_task_elastic_net.py +79 -43
- snowflake/ml/modeling/linear_model/multi_task_elastic_net_cv.py +79 -43
- snowflake/ml/modeling/linear_model/multi_task_lasso.py +79 -43
- snowflake/ml/modeling/linear_model/multi_task_lasso_cv.py +79 -43
- snowflake/ml/modeling/linear_model/orthogonal_matching_pursuit.py +79 -43
- snowflake/ml/modeling/linear_model/passive_aggressive_classifier.py +79 -43
- snowflake/ml/modeling/linear_model/passive_aggressive_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/perceptron.py +79 -43
- snowflake/ml/modeling/linear_model/poisson_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/ransac_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/ridge.py +79 -43
- snowflake/ml/modeling/linear_model/ridge_classifier.py +79 -43
- snowflake/ml/modeling/linear_model/ridge_classifier_cv.py +79 -43
- snowflake/ml/modeling/linear_model/ridge_cv.py +79 -43
- snowflake/ml/modeling/linear_model/sgd_classifier.py +79 -43
- snowflake/ml/modeling/linear_model/sgd_one_class_svm.py +79 -43
- snowflake/ml/modeling/linear_model/sgd_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/theil_sen_regressor.py +79 -43
- snowflake/ml/modeling/linear_model/tweedie_regressor.py +79 -43
- snowflake/ml/modeling/manifold/isomap.py +79 -43
- snowflake/ml/modeling/manifold/mds.py +79 -43
- snowflake/ml/modeling/manifold/spectral_embedding.py +79 -43
- snowflake/ml/modeling/manifold/tsne.py +79 -43
- snowflake/ml/modeling/metrics/classification.py +6 -1
- snowflake/ml/modeling/metrics/regression.py +517 -9
- snowflake/ml/modeling/mixture/bayesian_gaussian_mixture.py +79 -43
- snowflake/ml/modeling/mixture/gaussian_mixture.py +79 -43
- snowflake/ml/modeling/model_selection/grid_search_cv.py +79 -43
- snowflake/ml/modeling/model_selection/randomized_search_cv.py +79 -43
- snowflake/ml/modeling/multiclass/one_vs_one_classifier.py +79 -43
- snowflake/ml/modeling/multiclass/one_vs_rest_classifier.py +79 -43
- snowflake/ml/modeling/multiclass/output_code_classifier.py +79 -43
- snowflake/ml/modeling/naive_bayes/bernoulli_nb.py +79 -43
- snowflake/ml/modeling/naive_bayes/categorical_nb.py +79 -43
- snowflake/ml/modeling/naive_bayes/complement_nb.py +79 -43
- snowflake/ml/modeling/naive_bayes/gaussian_nb.py +79 -43
- snowflake/ml/modeling/naive_bayes/multinomial_nb.py +79 -43
- snowflake/ml/modeling/neighbors/k_neighbors_classifier.py +79 -43
- snowflake/ml/modeling/neighbors/k_neighbors_regressor.py +79 -43
- snowflake/ml/modeling/neighbors/kernel_density.py +79 -43
- snowflake/ml/modeling/neighbors/local_outlier_factor.py +79 -43
- snowflake/ml/modeling/neighbors/nearest_centroid.py +79 -43
- snowflake/ml/modeling/neighbors/nearest_neighbors.py +79 -43
- snowflake/ml/modeling/neighbors/neighborhood_components_analysis.py +79 -43
- snowflake/ml/modeling/neighbors/radius_neighbors_classifier.py +79 -43
- snowflake/ml/modeling/neighbors/radius_neighbors_regressor.py +79 -43
- snowflake/ml/modeling/neural_network/bernoulli_rbm.py +79 -43
- snowflake/ml/modeling/neural_network/mlp_classifier.py +79 -43
- snowflake/ml/modeling/neural_network/mlp_regressor.py +79 -43
- snowflake/ml/modeling/pipeline/pipeline.py +24 -0
- snowflake/ml/modeling/preprocessing/one_hot_encoder.py +18 -19
- snowflake/ml/modeling/preprocessing/ordinal_encoder.py +2 -0
- snowflake/ml/modeling/preprocessing/polynomial_features.py +79 -43
- snowflake/ml/modeling/semi_supervised/label_propagation.py +79 -43
- snowflake/ml/modeling/semi_supervised/label_spreading.py +79 -43
- snowflake/ml/modeling/svm/linear_svc.py +79 -43
- snowflake/ml/modeling/svm/linear_svr.py +79 -43
- snowflake/ml/modeling/svm/nu_svc.py +79 -43
- snowflake/ml/modeling/svm/nu_svr.py +79 -43
- snowflake/ml/modeling/svm/svc.py +79 -43
- snowflake/ml/modeling/svm/svr.py +79 -43
- snowflake/ml/modeling/tree/decision_tree_classifier.py +79 -43
- snowflake/ml/modeling/tree/decision_tree_regressor.py +79 -43
- snowflake/ml/modeling/tree/extra_tree_classifier.py +79 -43
- snowflake/ml/modeling/tree/extra_tree_regressor.py +79 -43
- snowflake/ml/modeling/xgboost/xgb_classifier.py +79 -43
- snowflake/ml/modeling/xgboost/xgb_regressor.py +79 -43
- snowflake/ml/modeling/xgboost/xgbrf_classifier.py +79 -43
- snowflake/ml/modeling/xgboost/xgbrf_regressor.py +79 -43
- snowflake/ml/registry/model_registry.py +123 -121
- snowflake/ml/version.py +1 -1
- {snowflake_ml_python-1.0.1.dist-info → snowflake_ml_python-1.0.3.dist-info}/METADATA +50 -8
- snowflake_ml_python-1.0.3.dist-info/RECORD +259 -0
- snowflake_ml_python-1.0.1.dist-info/RECORD +0 -246
- {snowflake_ml_python-1.0.1.dist-info → snowflake_ml_python-1.0.3.dist-info}/WHEEL +0 -0
@@ -1,6 +1,7 @@
|
|
1
1
|
import inspect
|
2
2
|
import json
|
3
3
|
import os
|
4
|
+
import posixpath
|
4
5
|
import sys
|
5
6
|
import tempfile
|
6
7
|
import types
|
@@ -8,7 +9,6 @@ import zipfile
|
|
8
9
|
from typing import TYPE_CHECKING, Any, Dict, List, Optional, cast
|
9
10
|
from uuid import uuid1
|
10
11
|
|
11
|
-
import cloudpickle as cp
|
12
12
|
from absl import logging
|
13
13
|
|
14
14
|
from snowflake import connector, snowpark
|
@@ -35,7 +35,7 @@ _DEFAULT_REGISTRY_NAME: str = "_SYSTEM_MODEL_REGISTRY"
|
|
35
35
|
_DEFAULT_SCHEMA_NAME: str = "_SYSTEM_MODEL_REGISTRY_SCHEMA"
|
36
36
|
_MODELS_TABLE_NAME: str = "_SYSTEM_REGISTRY_MODELS"
|
37
37
|
_METADATA_TABLE_NAME: str = "_SYSTEM_REGISTRY_METADATA"
|
38
|
-
_DEPLOYMENT_TABLE_NAME: str = "
|
38
|
+
_DEPLOYMENT_TABLE_NAME: str = "_SYSTEM_REGISTRY_DEPLOYMENTS"
|
39
39
|
|
40
40
|
# Metadata operation types.
|
41
41
|
_SET_METADATA_OPERATION: str = "SET"
|
@@ -84,9 +84,11 @@ def create_model_registry(
|
|
84
84
|
"""
|
85
85
|
|
86
86
|
# These might be exposed as parameters in the future.
|
87
|
-
|
88
|
-
|
89
|
-
|
87
|
+
database_name = identifier.get_inferred_name(database_name)
|
88
|
+
schema_name = identifier.get_inferred_name(schema_name)
|
89
|
+
registry_table_name = identifier.get_inferred_name(_MODELS_TABLE_NAME)
|
90
|
+
metadata_table_name = identifier.get_inferred_name(_METADATA_TABLE_NAME)
|
91
|
+
deployment_table_name = identifier.get_inferred_name(_DEPLOYMENT_TABLE_NAME)
|
90
92
|
statement_params = telemetry.get_function_usage_statement_params(
|
91
93
|
project=_TELEMETRY_PROJECT,
|
92
94
|
subproject=_TELEMETRY_SUBPROJECT,
|
@@ -130,14 +132,14 @@ def _create_registry_database(
|
|
130
132
|
database_name: Desired name of the model registry database.
|
131
133
|
statement_params: Function usage statement parameters used in sql query executions.
|
132
134
|
"""
|
133
|
-
registry_databases = session.sql(f"SHOW DATABASES LIKE '{database_name}'").collect(
|
135
|
+
registry_databases = session.sql(f"SHOW DATABASES LIKE '{identifier.get_unescaped_names(database_name)}'").collect(
|
134
136
|
statement_params=statement_params
|
135
137
|
)
|
136
138
|
if len(registry_databases) > 0:
|
137
139
|
logging.warning(f"The database {database_name} already exists. Skipping creation.")
|
138
140
|
return
|
139
141
|
|
140
|
-
session.sql(f
|
142
|
+
session.sql(f"CREATE DATABASE {database_name}").collect(statement_params=statement_params)
|
141
143
|
|
142
144
|
|
143
145
|
def _create_registry_schema(
|
@@ -154,31 +156,31 @@ def _create_registry_schema(
|
|
154
156
|
statement_params: Function usage statement parameters used in sql query executions.
|
155
157
|
"""
|
156
158
|
# The default PUBLIC schema is created by default so it might already exist even in a new database.
|
157
|
-
registry_schemas = session.sql(
|
158
|
-
|
159
|
-
)
|
159
|
+
registry_schemas = session.sql(
|
160
|
+
f"SHOW SCHEMAS LIKE '{identifier.get_unescaped_names(schema_name)}' IN DATABASE {database_name}"
|
161
|
+
).collect(statement_params=statement_params)
|
160
162
|
|
161
163
|
if len(registry_schemas) > 0:
|
162
|
-
logging.warning(
|
164
|
+
logging.warning(
|
165
|
+
f"The schema {_get_fully_qualified_schema_name(database_name, schema_name)}already exists. "
|
166
|
+
+ "Skipping creation."
|
167
|
+
)
|
163
168
|
return
|
164
169
|
|
165
|
-
session.sql(f
|
170
|
+
session.sql(f"CREATE SCHEMA {_get_fully_qualified_schema_name(database_name, schema_name)}").collect(
|
171
|
+
statement_params=statement_params
|
172
|
+
)
|
166
173
|
|
167
174
|
|
168
175
|
def _get_fully_qualified_schema_name(database_name: str, schema_name: str) -> str:
|
169
|
-
return ".".join(
|
170
|
-
[
|
171
|
-
f"{identifier.quote_name_without_upper_casing(database_name)}",
|
172
|
-
f"{identifier.quote_name_without_upper_casing(schema_name)}",
|
173
|
-
]
|
174
|
-
)
|
176
|
+
return ".".join([database_name, schema_name])
|
175
177
|
|
176
178
|
|
177
179
|
def _get_fully_qualified_table_name(database_name: str, schema_name: str, table_name: str) -> str:
|
178
180
|
return ".".join(
|
179
181
|
[
|
180
182
|
_get_fully_qualified_schema_name(database_name, schema_name),
|
181
|
-
|
183
|
+
table_name,
|
182
184
|
]
|
183
185
|
)
|
184
186
|
|
@@ -292,10 +294,10 @@ def _create_registry_views(
|
|
292
294
|
)
|
293
295
|
|
294
296
|
# Create views on most recent metadata items.
|
295
|
-
metadata_view_name_prefix = metadata_table_name
|
297
|
+
metadata_view_name_prefix = identifier.concat_names([metadata_table_name, "_LAST_"])
|
296
298
|
metadata_view_template = formatting.unwrap(
|
297
|
-
"""CREATE OR REPLACE VIEW
|
298
|
-
SELECT DISTINCT MODEL_ID, {select_expression} AS {final_attribute_name} FROM
|
299
|
+
"""CREATE OR REPLACE VIEW {database}.{schema}.{attribute_view} COPY GRANTS AS
|
300
|
+
SELECT DISTINCT MODEL_ID, {select_expression} AS {final_attribute_name} FROM {metadata_table}
|
299
301
|
WHERE ATTRIBUTE_NAME = '{attribute_name}'"""
|
300
302
|
)
|
301
303
|
|
@@ -303,7 +305,7 @@ def _create_registry_views(
|
|
303
305
|
metadata_view_names = []
|
304
306
|
metadata_select_fields = []
|
305
307
|
for attribute_name in _LIST_METADATA_ATTRIBUTE:
|
306
|
-
view_name =
|
308
|
+
view_name = identifier.concat_names([metadata_view_name_prefix, attribute_name])
|
307
309
|
select_expression = f"(LAST_VALUE(VALUE) OVER (PARTITION BY MODEL_ID ORDER BY SEQUENCE_ID))['{attribute_name}']"
|
308
310
|
sql = metadata_view_template.format(
|
309
311
|
database=database_name,
|
@@ -316,14 +318,12 @@ def _create_registry_views(
|
|
316
318
|
)
|
317
319
|
session.sql(sql).collect(statement_params=statement_params)
|
318
320
|
metadata_view_names.append(view_name)
|
319
|
-
metadata_select_fields.append(
|
320
|
-
f"{identifier.quote_name_without_upper_casing(view_name)}.{attribute_name} AS {attribute_name}"
|
321
|
-
)
|
321
|
+
metadata_select_fields.append(f"{view_name}.{attribute_name} AS {attribute_name}")
|
322
322
|
|
323
323
|
# Create a special view for the registration timestamp.
|
324
324
|
attribute_name = _METADATA_ATTRIBUTE_REGISTRATION
|
325
|
-
final_attribute_name = attribute_name
|
326
|
-
view_name =
|
325
|
+
final_attribute_name = identifier.concat_names([attribute_name, "_TIMESTAMP"])
|
326
|
+
view_name = identifier.concat_names([metadata_view_name_prefix, attribute_name])
|
327
327
|
create_registration_view_sql = metadata_view_template.format(
|
328
328
|
database=database_name,
|
329
329
|
schema=schema_name,
|
@@ -335,13 +335,11 @@ def _create_registry_views(
|
|
335
335
|
)
|
336
336
|
session.sql(create_registration_view_sql).collect(statement_params=statement_params)
|
337
337
|
metadata_view_names.append(view_name)
|
338
|
-
metadata_select_fields.append(
|
339
|
-
f"{identifier.quote_name_without_upper_casing(view_name)}.{final_attribute_name} AS {final_attribute_name}"
|
340
|
-
)
|
338
|
+
metadata_select_fields.append(f"{view_name}.{final_attribute_name} AS {final_attribute_name}")
|
341
339
|
|
342
340
|
metadata_views_join = " ".join(
|
343
341
|
[
|
344
|
-
|
342
|
+
"LEFT JOIN {view} ON ({view}.MODEL_ID = {registry_table}.ID)".format(
|
345
343
|
view=view, registry_table=registry_table_name
|
346
344
|
)
|
347
345
|
for view in metadata_view_names
|
@@ -349,12 +347,12 @@ def _create_registry_views(
|
|
349
347
|
)
|
350
348
|
|
351
349
|
# Create view to combine all attributes.
|
352
|
-
registry_view_name = registry_table_name
|
350
|
+
registry_view_name = identifier.concat_names([registry_table_name, "_VIEW"])
|
353
351
|
metadata_select_fields_formatted = ",".join(metadata_select_fields)
|
354
352
|
session.sql(
|
355
|
-
f"""CREATE OR REPLACE VIEW {fully_qualified_schema_name}.
|
356
|
-
SELECT
|
357
|
-
FROM
|
353
|
+
f"""CREATE OR REPLACE VIEW {fully_qualified_schema_name}.{registry_view_name} COPY GRANTS AS
|
354
|
+
SELECT {registry_table_name}.*, {metadata_select_fields_formatted}
|
355
|
+
FROM {registry_table_name} {metadata_views_join}"""
|
358
356
|
).collect(statement_params=statement_params)
|
359
357
|
|
360
358
|
|
@@ -377,8 +375,9 @@ def _create_active_permanent_deployment_view(
|
|
377
375
|
|
378
376
|
# Create a view on active permanent deployments
|
379
377
|
# Active deployments are those whose last operation is not DROP.
|
378
|
+
active_deployments_view_name = identifier.concat_names([deployment_table_name, "_VIEW"])
|
380
379
|
active_deployments_view_expr = f"""
|
381
|
-
CREATE OR REPLACE VIEW {fully_qualified_schema_name}.
|
380
|
+
CREATE OR REPLACE VIEW {fully_qualified_schema_name}.{active_deployments_view_name}
|
382
381
|
COPY GRANTS AS
|
383
382
|
SELECT
|
384
383
|
DEPLOYMENT_NAME,
|
@@ -417,14 +416,14 @@ class ModelRegistry:
|
|
417
416
|
database_name: Desired name of the model registry database.
|
418
417
|
schema_name: Desired name of the schema used by this model registry inside the database.
|
419
418
|
"""
|
420
|
-
self._name = database_name
|
421
|
-
self._schema = schema_name
|
422
|
-
self._registry_table = _MODELS_TABLE_NAME
|
423
|
-
self._registry_table_view = self._registry_table
|
424
|
-
self._metadata_table = _METADATA_TABLE_NAME
|
425
|
-
self._deployment_table = _DEPLOYMENT_TABLE_NAME
|
426
|
-
self._permanent_deployment_view = self._deployment_table
|
427
|
-
self._permanent_deployment_stage = self._deployment_table
|
419
|
+
self._name = identifier.get_inferred_name(database_name)
|
420
|
+
self._schema = identifier.get_inferred_name(schema_name)
|
421
|
+
self._registry_table = identifier.get_inferred_name(_MODELS_TABLE_NAME)
|
422
|
+
self._registry_table_view = identifier.concat_names([self._registry_table, "_VIEW"])
|
423
|
+
self._metadata_table = identifier.get_inferred_name(_METADATA_TABLE_NAME)
|
424
|
+
self._deployment_table = identifier.get_inferred_name(_DEPLOYMENT_TABLE_NAME)
|
425
|
+
self._permanent_deployment_view = identifier.concat_names([self._deployment_table, "_VIEW"])
|
426
|
+
self._permanent_deployment_stage = identifier.concat_names([self._deployment_table, "_STAGE"])
|
428
427
|
|
429
428
|
self._session = session
|
430
429
|
|
@@ -441,23 +440,39 @@ class ModelRegistry:
|
|
441
440
|
# Check that the required tables exist and are accessible by the current role.
|
442
441
|
|
443
442
|
query_result_checker.SqlResultValidator(
|
444
|
-
self._session, query=f"SHOW DATABASES LIKE '{self._name}'"
|
443
|
+
self._session, query=f"SHOW DATABASES LIKE '{identifier.get_unescaped_names(self._name)}'"
|
445
444
|
).has_dimensions(expected_rows=1).validate()
|
446
445
|
|
447
446
|
query_result_checker.SqlResultValidator(
|
448
|
-
self._session,
|
447
|
+
self._session,
|
448
|
+
query=f"SHOW SCHEMAS LIKE '{identifier.get_unescaped_names(self._schema)}' IN DATABASE {self._name}",
|
449
449
|
).has_dimensions(expected_rows=1).validate()
|
450
450
|
|
451
451
|
query_result_checker.SqlResultValidator(
|
452
|
-
self._session,
|
452
|
+
self._session,
|
453
|
+
query=formatting.unwrap(
|
454
|
+
f"""
|
455
|
+
SHOW TABLES LIKE '{identifier.get_unescaped_names(self._registry_table)}'
|
456
|
+
IN {self._fully_qualified_schema_name()}"""
|
457
|
+
),
|
453
458
|
).has_dimensions(expected_rows=1).validate()
|
454
459
|
|
455
460
|
query_result_checker.SqlResultValidator(
|
456
|
-
self._session,
|
461
|
+
self._session,
|
462
|
+
query=formatting.unwrap(
|
463
|
+
f"""
|
464
|
+
SHOW TABLES LIKE '{identifier.get_unescaped_names(self._metadata_table)}'
|
465
|
+
IN {self._fully_qualified_schema_name()}"""
|
466
|
+
),
|
457
467
|
).has_dimensions(expected_rows=1).validate()
|
458
468
|
|
459
469
|
query_result_checker.SqlResultValidator(
|
460
|
-
self._session,
|
470
|
+
self._session,
|
471
|
+
query=formatting.unwrap(
|
472
|
+
f"""
|
473
|
+
SHOW TABLES LIKE '{identifier.get_unescaped_names(self._deployment_table)}'
|
474
|
+
IN {self._fully_qualified_schema_name()}"""
|
475
|
+
),
|
461
476
|
).has_dimensions(expected_rows=1).validate()
|
462
477
|
|
463
478
|
# TODO(zzhu): Also check validity of views. Consider checking schema as well.
|
@@ -825,7 +840,7 @@ class ModelRegistry:
|
|
825
840
|
before setting the metadata attribute. False by default meaning that by default we will check.
|
826
841
|
|
827
842
|
Raises:
|
828
|
-
DataError: Failed to set the
|
843
|
+
DataError: Failed to set the metadata attribute.
|
829
844
|
KeyError: The target model doesn't exist
|
830
845
|
"""
|
831
846
|
selected_models = self._list_selected_models(id=id, model_name=model_name, model_version=model_version)
|
@@ -955,13 +970,13 @@ class ModelRegistry:
|
|
955
970
|
# Check if directory or file and adapt accordingly.
|
956
971
|
# TODO: Unify and explicit about compression for both file and directory.
|
957
972
|
if os.path.isfile(path):
|
958
|
-
self._session.file.put(path,
|
973
|
+
self._session.file.put(path, posixpath.join(fully_qualified_model_stage_name, "data"))
|
959
974
|
elif os.path.isdir(path):
|
960
975
|
with file_utils.zip_file_or_directory_to_stream(path, path) as input_stream:
|
961
976
|
self._session._conn.upload_stream(
|
962
977
|
input_stream=input_stream,
|
963
978
|
stage_location=fully_qualified_model_stage_name,
|
964
|
-
dest_filename=f"{
|
979
|
+
dest_filename=f"{posixpath.basename(path)}.zip",
|
965
980
|
dest_prefix="",
|
966
981
|
source_compression="DEFLATE",
|
967
982
|
compress_data=False,
|
@@ -1067,7 +1082,7 @@ class ModelRegistry:
|
|
1067
1082
|
"""
|
1068
1083
|
# Explicitly not calling collect.
|
1069
1084
|
return self._session.sql(
|
1070
|
-
|
1085
|
+
"SELECT * FROM {database}.{schema}.{view}".format(
|
1071
1086
|
database=self._name, schema=self._schema, view=self._registry_table_view
|
1072
1087
|
)
|
1073
1088
|
)
|
@@ -1124,7 +1139,9 @@ class ModelRegistry:
|
|
1124
1139
|
try:
|
1125
1140
|
del model_tags[tag_name]
|
1126
1141
|
except KeyError:
|
1127
|
-
raise connector.DataError(
|
1142
|
+
raise connector.DataError(
|
1143
|
+
f"Model {model_name}/{model_version} has no tag named {tag_name}. Full list of tags: {model_tags}"
|
1144
|
+
)
|
1128
1145
|
|
1129
1146
|
self._set_metadata_attribute(
|
1130
1147
|
_METADATA_ATTRIBUTE_TAGS, model_tags, model_name=model_name, model_version=model_version
|
@@ -1227,7 +1244,7 @@ class ModelRegistry:
|
|
1227
1244
|
result = self._get_metadata_attribute(
|
1228
1245
|
_METADATA_ATTRIBUTE_DESCRIPTION, model_name=model_name, model_version=model_version
|
1229
1246
|
)
|
1230
|
-
return None if result is None else
|
1247
|
+
return None if result is None else json.loads(result)
|
1231
1248
|
|
1232
1249
|
@telemetry.send_api_usage_telemetry(
|
1233
1250
|
project=_TELEMETRY_PROJECT,
|
@@ -1453,6 +1470,8 @@ class ModelRegistry:
|
|
1453
1470
|
pip_requirements: Optional[List[str]] = None,
|
1454
1471
|
signatures: Optional[Dict[str, model_signature.ModelSignature]] = None,
|
1455
1472
|
sample_input_data: Optional[Any] = None,
|
1473
|
+
code_paths: Optional[List[str]] = None,
|
1474
|
+
options: Optional[model_types.ModelSaveOption] = None,
|
1456
1475
|
) -> str:
|
1457
1476
|
"""Uploads and register a model to the Model Registry.
|
1458
1477
|
|
@@ -1472,6 +1491,8 @@ class ModelRegistry:
|
|
1472
1491
|
signatures: Signatures of the model, which is a mapping from target method name to signatures of input and
|
1473
1492
|
output, which could be inferred by calling `infer_signature` method with sample input data.
|
1474
1493
|
sample_input_data: Sample of the input data for the model.
|
1494
|
+
code_paths: Directory of code to import when loading and deploying the model.
|
1495
|
+
options: Additional options when saving the model.
|
1475
1496
|
|
1476
1497
|
Raises:
|
1477
1498
|
TypeError: Raised when both signatures and sample_input_data is not presented. Will be captured locally.
|
@@ -1490,60 +1511,50 @@ class ModelRegistry:
|
|
1490
1511
|
raise connector.DataError(f"Model {model_name}/{model_version} already exists. Unable to log the model.")
|
1491
1512
|
with tempfile.TemporaryDirectory() as tmpdir:
|
1492
1513
|
model = cast(model_types.SupportedModelType, model)
|
1493
|
-
|
1494
|
-
|
1495
|
-
|
1496
|
-
|
1497
|
-
|
1498
|
-
|
1499
|
-
|
1500
|
-
|
1501
|
-
|
1502
|
-
|
1503
|
-
|
1504
|
-
elif sample_input_data is not None:
|
1505
|
-
model_api.save_model(
|
1506
|
-
name=model_name,
|
1507
|
-
model_dir_path=tmpdir,
|
1508
|
-
model=model,
|
1509
|
-
metadata=tags,
|
1510
|
-
conda_dependencies=conda_dependencies,
|
1511
|
-
pip_requirements=pip_requirements,
|
1512
|
-
sample_input=sample_input_data,
|
1513
|
-
)
|
1514
|
-
elif isinstance(model, base.BaseEstimator):
|
1515
|
-
model_api.save_model(
|
1516
|
-
name=model_name,
|
1517
|
-
model_dir_path=tmpdir,
|
1518
|
-
model=model,
|
1519
|
-
metadata=tags,
|
1520
|
-
conda_dependencies=conda_dependencies,
|
1521
|
-
pip_requirements=pip_requirements,
|
1522
|
-
)
|
1523
|
-
else:
|
1524
|
-
raise TypeError("Either signature or sample input data should exist for native model packaging.")
|
1525
|
-
return self._log_model_path(
|
1526
|
-
model_name=model_name,
|
1527
|
-
model_version=model_version,
|
1528
|
-
path=tmpdir,
|
1529
|
-
type="snowflake_native",
|
1530
|
-
description=description,
|
1531
|
-
tags=tags, # TODO: Inherent model type enum.
|
1514
|
+
if signatures:
|
1515
|
+
model_metadata = model_api.save_model(
|
1516
|
+
name=model_name,
|
1517
|
+
model_dir_path=tmpdir,
|
1518
|
+
model=model,
|
1519
|
+
signatures=signatures,
|
1520
|
+
metadata=tags,
|
1521
|
+
conda_dependencies=conda_dependencies,
|
1522
|
+
pip_requirements=pip_requirements,
|
1523
|
+
code_paths=code_paths,
|
1524
|
+
options=options,
|
1532
1525
|
)
|
1533
|
-
|
1534
|
-
|
1535
|
-
|
1536
|
-
|
1537
|
-
|
1538
|
-
|
1539
|
-
|
1526
|
+
elif sample_input_data is not None:
|
1527
|
+
model_metadata = model_api.save_model(
|
1528
|
+
name=model_name,
|
1529
|
+
model_dir_path=tmpdir,
|
1530
|
+
model=model,
|
1531
|
+
metadata=tags,
|
1532
|
+
conda_dependencies=conda_dependencies,
|
1533
|
+
pip_requirements=pip_requirements,
|
1534
|
+
sample_input=sample_input_data,
|
1535
|
+
code_paths=code_paths,
|
1536
|
+
options=options,
|
1537
|
+
)
|
1538
|
+
elif isinstance(model, base.BaseEstimator):
|
1539
|
+
model_metadata = model_api.save_model(
|
1540
|
+
name=model_name,
|
1541
|
+
model_dir_path=tmpdir,
|
1542
|
+
model=model,
|
1543
|
+
metadata=tags,
|
1544
|
+
conda_dependencies=conda_dependencies,
|
1545
|
+
pip_requirements=pip_requirements,
|
1546
|
+
code_paths=code_paths,
|
1547
|
+
options=options,
|
1548
|
+
)
|
1549
|
+
else:
|
1550
|
+
raise TypeError("Either signature or sample input data should exist for native model packaging.")
|
1540
1551
|
return self._log_model_path(
|
1541
1552
|
model_name=model_name,
|
1542
1553
|
model_version=model_version,
|
1543
|
-
path=
|
1544
|
-
type=
|
1554
|
+
path=tmpdir,
|
1555
|
+
type=model_metadata.model_type,
|
1545
1556
|
description=description,
|
1546
|
-
tags=tags,
|
1557
|
+
tags=tags, # TODO: Inherent model type enum.
|
1547
1558
|
)
|
1548
1559
|
|
1549
1560
|
@telemetry.send_api_usage_telemetry(
|
@@ -1565,22 +1576,13 @@ class ModelRegistry:
|
|
1565
1576
|
restored_model = None
|
1566
1577
|
with tempfile.TemporaryDirectory() as local_model_directory:
|
1567
1578
|
self._session.file.get(remote_model_path, local_model_directory)
|
1568
|
-
|
1569
|
-
|
1570
|
-
|
1571
|
-
|
1572
|
-
|
1573
|
-
|
1574
|
-
|
1575
|
-
myzip.extractall(extracted_dir)
|
1576
|
-
restored_model, _meta = model_api.load_model(model_dir_path=extracted_dir)
|
1577
|
-
is_native_model_format = True
|
1578
|
-
except TypeError:
|
1579
|
-
pass
|
1580
|
-
if not is_native_model_format:
|
1581
|
-
file_path = os.path.join(local_model_directory, os.path.basename(os.path.basename(remote_model_path)))
|
1582
|
-
with open(file_path, mode="r+b") as model_file:
|
1583
|
-
restored_model = cp.load(model_file)
|
1579
|
+
local_path = os.path.join(local_model_directory, posixpath.basename(remote_model_path))
|
1580
|
+
if zipfile.is_zipfile(local_path):
|
1581
|
+
extracted_dir = os.path.join(local_model_directory, "extracted")
|
1582
|
+
with zipfile.ZipFile(local_path, "r") as myzip:
|
1583
|
+
if len(myzip.namelist()) > 1:
|
1584
|
+
myzip.extractall(extracted_dir)
|
1585
|
+
restored_model, _ = model_api.load_model(model_dir_path=extracted_dir)
|
1584
1586
|
|
1585
1587
|
return restored_model
|
1586
1588
|
|
snowflake/ml/version.py
CHANGED
@@ -1 +1 @@
|
|
1
|
-
VERSION="1.0.
|
1
|
+
VERSION="1.0.3"
|
@@ -1,11 +1,15 @@
|
|
1
1
|
Metadata-Version: 2.1
|
2
2
|
Name: snowflake-ml-python
|
3
|
-
Version: 1.0.1
|
4
|
-
Description-Content-Type: text/markdown
|
5
3
|
Author: Snowflake, Inc
|
6
4
|
Author-email: support@snowflake.com
|
7
5
|
Home-page: https://github.com/snowflakedb/snowflake-ml-python
|
8
6
|
License: Apache License, Version 2.0
|
7
|
+
Description-Content-Type: text/markdown
|
8
|
+
Summary: The machine learning client library that is used for interacting with Snowflake to build machine learning solutions.
|
9
|
+
Project-URL: Changelog, https://github.com/snowflakedb/snowflake-ml-python/blob/main/CHANGELOG.md
|
10
|
+
Project-URL: Documentation, https://docs.snowflake.com/developer-guide/snowpark-ml
|
11
|
+
Project-URL: Issues, https://github.com/snowflakedb/snowflake-ml-python/issues
|
12
|
+
Project-URL: Source, https://github.com/snowflakedb/snowflake-ml-python
|
9
13
|
Classifier: Development Status :: 3 - Alpha
|
10
14
|
Classifier: Environment :: Console
|
11
15
|
Classifier: Environment :: Other Environment
|
@@ -25,17 +29,15 @@ Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
|
|
25
29
|
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
26
30
|
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
27
31
|
Requires-Python: >=3.8,<4
|
28
|
-
Summary: The machine learning client library that is used for interacting with Snowflake to build machine learning solutions.
|
29
32
|
Requires-Dist: absl-py>=0.15,<2
|
30
33
|
Requires-Dist: anyio>=3.5.0,<4
|
31
34
|
Requires-Dist: cloudpickle
|
32
|
-
Requires-Dist: cryptography>=3.1.0,<39.0.0
|
33
35
|
Requires-Dist: fsspec[http]>=2022.11,<=2023.1
|
34
36
|
Requires-Dist: numpy>=1.23,<2
|
35
37
|
Requires-Dist: packaging>=20.9,<24
|
36
38
|
Requires-Dist: pandas>=1.0.0,<2
|
37
39
|
Requires-Dist: pyyaml>=6.0,<7
|
38
|
-
Requires-Dist: scikit-learn>=1.2.1,<
|
40
|
+
Requires-Dist: scikit-learn>=1.2.1,<1.3
|
39
41
|
Requires-Dist: scipy>=1.9,<2
|
40
42
|
Requires-Dist: snowflake-connector-python[pandas]>=3.0.3,<4
|
41
43
|
Requires-Dist: snowflake-snowpark-python>=1.4.0,<2
|
@@ -52,6 +54,7 @@ Provides-Extra: tensorflow
|
|
52
54
|
Requires-Dist: tensorflow>=2.9,<3; extra == 'tensorflow'
|
53
55
|
Provides-Extra: torch
|
54
56
|
Requires-Dist: torchdata>=0.4,<1; extra == 'torch'
|
57
|
+
Version: 1.0.3
|
55
58
|
|
56
59
|
# Snowpark ML
|
57
60
|
|
@@ -77,9 +80,7 @@ Snowpark MLOps complements the Snowpark ML Development API, and provides model m
|
|
77
80
|
|
78
81
|
During PrPr, we are iterating on API without backward compatibility guarantees. It is better to recreate your registry everytime you update the package. This means, at this time, you cannot use the registry for production use.
|
79
82
|
|
80
|
-
- [Documentation](
|
81
|
-
- [Issues](https://github.com/snowflakedb/snowflake-ml-python/issues)
|
82
|
-
- [Source](https://github.com/snowflakedb/snowflake-ml-python/)
|
83
|
+
- [Documentation](https://docs.snowflake.com/developer-guide/snowpark-ml)
|
83
84
|
|
84
85
|
## Getting started
|
85
86
|
### Have your Snowflake account ready
|
@@ -96,6 +97,47 @@ pip install snowflake-ml-python
|
|
96
97
|
```
|
97
98
|
# Release History
|
98
99
|
|
100
|
+
## 1.0.3 (2023-07-14)
|
101
|
+
|
102
|
+
### Behavior Changes
|
103
|
+
- Model Registry: When predicting a model whose output is a list of NumPy ndarray, the output would not be flattened, instead, every ndarray will act as a feature(column) in the output.
|
104
|
+
|
105
|
+
### New Features
|
106
|
+
- Model Registry: Added support save/load/deploy PyTorch models (`torch.nn.Module` and `torch.jit.ScriptModule`).
|
107
|
+
|
108
|
+
### Bug Fixes
|
109
|
+
|
110
|
+
- Model Registry: Fix an issue that when database or schema name provided to `create_model_registry` contains special characters, the model registry cannot be created.
|
111
|
+
- Model Registry: Fix an issue that `get_model_description` returns with additional quotes.
|
112
|
+
- Model Registry: Fix incorrect error message when attempting to remove a unset tag of a model.
|
113
|
+
- Model Registry: Fix a typo in the default deployment table name.
|
114
|
+
- Model Registry: Snowpark dataframe for sample input or input for `predict` method that contains a column with Snowflake `NUMBER(precision, scale)` data type where `scale = 0` will not lead to error, and will now correctly recognized as `INT64` data type in model signature.
|
115
|
+
- Model Registry: Fix an issue that prevent model logged in the system whose default encoding is not UTF-8 compatible from deploying.
|
116
|
+
- Model Registry: Added earlier and better error message when any file name in the model or the file name of model itself contains characters that are unable to be encoded using ASCII. It is currently not supported to deploy such a model.
|
117
|
+
|
118
|
+
## 1.0.2 (2023-06-22)
|
119
|
+
|
120
|
+
### Behavior Changes
|
121
|
+
- Model Registry: Prohibit non-snowflake-native models from being logged.
|
122
|
+
- Model Registry: `_use_local_snowml` parameter in options of `deploy()` has been removed.
|
123
|
+
- Model Registry: A default `False` `embed_local_ml_library` parameter has been added to the options of `log_model()`. With this set to `False` (default), the version of the local snowflake-ml-python library will be recorded and used when deploying the model. With this set to `True`, local snowflake-ml-python library will be embedded into the logged model, and will be used when you load or deploy the model.
|
124
|
+
|
125
|
+
### New Features
|
126
|
+
- Model Registry: A new optional argument named `code_paths` has been added to the arguments of `log_model()` for users to specify additional code paths to be imported when loading and deploying the model.
|
127
|
+
- Model Registry: A new optional argument named `options` has been added to the arguments of `log_model()` to specify any additional options when saving the model.
|
128
|
+
- Model Development: Added metrics:
|
129
|
+
- d2_absolute_error_score
|
130
|
+
- d2_pinball_score
|
131
|
+
- explained_variance_score
|
132
|
+
- mean_absolute_error
|
133
|
+
- mean_absolute_percentage_error
|
134
|
+
- mean_squared_error
|
135
|
+
|
136
|
+
### Bug Fixes
|
137
|
+
|
138
|
+
- Model Development: `accuracy_score()` now works when given label column names are lists of a single value.
|
139
|
+
|
140
|
+
|
99
141
|
## 1.0.1 (2023-06-16)
|
100
142
|
### Behavior Changes
|
101
143
|
|