snowflake-ml-python 1.19.0__tar.gz → 1.21.0__tar.gz

{snowflake_ml_python-1.19.0 → snowflake_ml_python-1.21.0}/CHANGELOG.md

@@ -1,10 +1,97 @@
 # Release History
 
-## 1.19.0
+## 1.21.0
+
+### New Features
+
+* Inference Autocapture (Preview): The `create_service` API will now accept `autocapture` as a new argument to indicate
+  whether inference data will be captured.
+* DataConnector: Added `to_huggingface_dataset()` method for converting Snowflake data to HuggingFace datasets.
+  Supports both in-memory `Dataset` (streaming=False) and streaming `IterableDataset` (streaming=True) modes.
+* Introducing `snowflake.ml.model.models.huggingface.TransformersPipeline` model which will replace object
+`snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel`. This is a wrapper class to
+  `transformers.Pipeline` model. Currently, following tasks
+  are supported to log without manually specifying model signatures:
+  * "fill-mask"
+  * "question-answering"
+  * "summarization"
+  * "table-question-answering"
+  * "text2text-generation"
+  * "text-classification" (alias "sentiment-analysis" available)
+  * "text-generation"
+  * "token-classification" (alias "ner" available)
+  * "translation"
+  * "translation_xx_to_yy"
+  * "zero-shot-classification"
+  which provides utility to log the model without loading the model in memory.
 
 ### Bug Fixes
 
-* Experiment Tracking (PrPr): No longer throw an exception in `list_artifacts` when run does not have artifacts.
+* Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
+  instead of raising an exception.
+* Registry: `ModelVersion.run()` will now raise a `ValueError` if the model does not support running on warehouse
+  (e.g. SPCS-only models) and `service_name` is not provided.
+
+### Behavior Changes
+
+* ML Job: The `additional_payloads` (Preview API) behavior is changing.
+  Use the `imports` argument to declare additional dependencies, such as zip files and Python modules.
+  Local directories and Python files are automatically compressed, and their internal layout is determined by the
+  specified import path. The import path applies only to local directories, Python files and staged python files;
+  it has no effect on other import types. When referencing files in a stage, only individual files
+  are supported—not directories.
+* Online Inference (PuPr): `list_services()` now shows internal endpoint that does not need EAI to call from another
+  SPCS node or Notebook.
+* Inference Autocapture (Preview): `list_services()` now shows if model service has autocapture enabled.
+* Experiment Tracking (PuPr): `ExperimentTracking` is now a singleton class.
+
+### Deprecations
+
+* Deprecating `snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel` and will be removed in later
+ version with a notice.
+
+## 1.20.0
+
+### Bug Fixes
+
+* Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
+  instead of raising an exception.
+
+### New Features
+
+* Registry (PrPr): Introducing vLLM as a backend inference engine. The `create_service` API will now
+ accept `inference_engine_options` as an argument.
+
+```python
+from snowflake.ml.model.inference_engine import InferenceEngine
+
+mv = snowflake_registry.log_model(
+    model=generator,
+    model_name=...,
+    ...,
+    # Specifying OPENAI_CHAT_SIGNATURE is necessary to use vLLM inference engine
+    signatures=openai_signatures.OPENAI_CHAT_SIGNATURE,
+)
+
+mv.create_service(
+    service_name=my_serv,
+    service_compute_pool=...,
+    ...,
+    inference_engine_options={
+        "engine": InferenceEngine.VLLM,
+        "engine_args_override": [
+            "--max-model-len=2048",
+            "--gpu-memory-utilization=0.9"
+        ]
+    }
+)
+```
+
+## 1.19.0 (11-13-2025)
+
+### Bug Fixes
+
+* Experiment Tracking (PuPr): No longer throw an exception in `list_artifacts` when run does not have artifacts.
 * Registry: Fix `get_version_by_alias`: now requires an exact match of snowflake identifier.
 
 ### Behavior Changes
@@ -106,8 +193,6 @@ options = {
 * Registry: Dropping support for deprecated `conversational` task type for Huggingface models.
   To read more <https://github.com/huggingface/transformers/pull/31165>
 
-### New Features
-
 ## 1.14.0 (09-18-2025)
 
 ### Bug Fixes

{snowflake_ml_python-1.19.0/snowflake_ml_python.egg-info → snowflake_ml_python-1.21.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: snowflake-ml-python
-Version: 1.19.0
+Version: 1.21.0
 Summary: The machine learning client library that is used for interacting with Snowflake to build machine learning solutions.
 Author-email: "Snowflake, Inc" <support@snowflake.com>
 License: 
@@ -230,7 +230,7 @@ Classifier: Topic :: Software Development :: Libraries
 Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
-Requires-Python: <3.13,>=3.9
+Requires-Python: <3.14,>=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 Requires-Dist: anyio<5,>=3.5.0
@@ -240,7 +240,7 @@ Requires-Dist: cryptography
 Requires-Dist: fsspec[http]<2026,>=2024.6.1
 Requires-Dist: importlib_resources<7,>=6.1.1
 Requires-Dist: numpy<3,>=1.23
-Requires-Dist: packaging<25,>=20.9
+Requires-Dist: packaging<26,>=20.9
 Requires-Dist: pandas<3,>=2.1.4
 Requires-Dist: platformdirs<5
 Requires-Dist: pyarrow<19.0.0
@@ -307,7 +307,9 @@ Requires-Dist: torchdata<1,>=0.4; extra == "torch"
 Provides-Extra: transformers
 Requires-Dist: sentence-transformers<4,>=2.7.0; extra == "transformers"
 Requires-Dist: sentencepiece<0.2.0,>=0.1.95; extra == "transformers"
+Requires-Dist: tokenizers<1,>=0.15.1; extra == "transformers"
 Requires-Dist: torch<3,>=2.0.1; extra == "transformers"
+Requires-Dist: transformers!=4.51.3,<5,>=4.39.3; extra == "transformers"
 Dynamic: license-file
 
 # Snowflake ML Python
@@ -415,11 +417,98 @@ NOTE: Version 1.7.0 is used as example here. Please choose the the latest versio
 
 # Release History
 
-## 1.19.0
+## 1.21.0
+
+### New Features
+
+* Inference Autocapture (Preview): The `create_service` API will now accept `autocapture` as a new argument to indicate
+  whether inference data will be captured.
+* DataConnector: Added `to_huggingface_dataset()` method for converting Snowflake data to HuggingFace datasets.
+  Supports both in-memory `Dataset` (streaming=False) and streaming `IterableDataset` (streaming=True) modes.
+* Introducing `snowflake.ml.model.models.huggingface.TransformersPipeline` model which will replace object
+`snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel`. This is a wrapper class to
+  `transformers.Pipeline` model. Currently, following tasks
+  are supported to log without manually specifying model signatures:
+  * "fill-mask"
+  * "question-answering"
+  * "summarization"
+  * "table-question-answering"
+  * "text2text-generation"
+  * "text-classification" (alias "sentiment-analysis" available)
+  * "text-generation"
+  * "token-classification" (alias "ner" available)
+  * "translation"
+  * "translation_xx_to_yy"
+  * "zero-shot-classification"
+  which provides utility to log the model without loading the model in memory.
 
 ### Bug Fixes
 
-* Experiment Tracking (PrPr): No longer throw an exception in `list_artifacts` when run does not have artifacts.
+* Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
+  instead of raising an exception.
+* Registry: `ModelVersion.run()` will now raise a `ValueError` if the model does not support running on warehouse
+  (e.g. SPCS-only models) and `service_name` is not provided.
+
+### Behavior Changes
+
+* ML Job: The `additional_payloads` (Preview API) behavior is changing.
+  Use the `imports` argument to declare additional dependencies, such as zip files and Python modules.
+  Local directories and Python files are automatically compressed, and their internal layout is determined by the
+  specified import path. The import path applies only to local directories, Python files and staged python files;
+  it has no effect on other import types. When referencing files in a stage, only individual files
+  are supported—not directories.
+* Online Inference (PuPr): `list_services()` now shows internal endpoint that does not need EAI to call from another
+  SPCS node or Notebook.
+* Inference Autocapture (Preview): `list_services()` now shows if model service has autocapture enabled.
+* Experiment Tracking (PuPr): `ExperimentTracking` is now a singleton class.
+
+### Deprecations
+
+* Deprecating `snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel` and will be removed in later
+ version with a notice.
+
+## 1.20.0
+
+### Bug Fixes
+
+* Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
+  instead of raising an exception.
+
+### New Features
+
+* Registry (PrPr): Introducing vLLM as a backend inference engine. The `create_service` API will now
+ accept `inference_engine_options` as an argument.
+
+```python
+from snowflake.ml.model.inference_engine import InferenceEngine
+
+mv = snowflake_registry.log_model(
+    model=generator,
+    model_name=...,
+    ...,
+    # Specifying OPENAI_CHAT_SIGNATURE is necessary to use vLLM inference engine
+    signatures=openai_signatures.OPENAI_CHAT_SIGNATURE,
+)
+
+mv.create_service(
+    service_name=my_serv,
+    service_compute_pool=...,
+    ...,
+    inference_engine_options={
+        "engine": InferenceEngine.VLLM,
+        "engine_args_override": [
+            "--max-model-len=2048",
+            "--gpu-memory-utilization=0.9"
+        ]
+    }
+)
+```
+
+## 1.19.0 (11-13-2025)
+
+### Bug Fixes
+
+* Experiment Tracking (PuPr): No longer throw an exception in `list_artifacts` when run does not have artifacts.
 * Registry: Fix `get_version_by_alias`: now requires an exact match of snowflake identifier.
 
 ### Behavior Changes
@@ -521,8 +610,6 @@ options = {
 * Registry: Dropping support for deprecated `conversational` task type for Huggingface models.
   To read more <https://github.com/huggingface/transformers/pull/31165>
 
-### New Features
-
 ## 1.14.0 (09-18-2025)
 
 ### Bug Fixes

{snowflake_ml_python-1.19.0 → snowflake_ml_python-1.21.0}/pyproject.toml

@@ -6,9 +6,9 @@ build-backend = "setuptools.build_meta"
 name = "snowflake-ml-python"
 description = "The machine learning client library that is used for interacting with Snowflake to build machine learning solutions."
 classifiers = [ "Development Status :: 5 - Production/Stable", "Environment :: Console", "Environment :: Other Environment", "Intended Audience :: Developers", "Intended Audience :: Education", "Intended Audience :: Information Technology", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Topic :: Database", "Topic :: Software Development", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Application Frameworks", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: Scientific/Engineering :: Information Analysis",]
-requires-python = ">=3.9, <3.13"
+requires-python = ">=3.9, <3.14"
 dynamic = [ "version", "readme",]
-dependencies = [ "anyio>=3.5.0,<5", "cachetools>=3.1.1,<6", "cloudpickle>=2.0.0", "cryptography", "fsspec[http]>=2024.6.1,<2026", "importlib_resources>=6.1.1, <7", "numpy>=1.23,<3", "packaging>=20.9,<25", "pandas>=2.1.4,<3", "platformdirs<5", "pyarrow<19.0.0", "pydantic>=2.8.2, <3", "pyjwt>=2.0.0, <3", "pytimeparse>=1.1.8,<2", "pyyaml>=6.0,<7", "retrying>=1.3.3,<2", "s3fs>=2024.6.1,<2026", "scikit-learn<1.8", "scipy>=1.9,<2", "shap>=0.46.0,<1", "snowflake-connector-python[pandas]>=3.17.3,<4", "snowflake-snowpark-python>=1.17.0,<2,!=1.26.0", "snowflake.core>=1.0.2,<2", "sqlparse>=0.4,<1", "tqdm<5", "typing-extensions>=4.1.0,<5", "xgboost<4",]
+dependencies = [ "anyio>=3.5.0,<5", "cachetools>=3.1.1,<6", "cloudpickle>=2.0.0", "cryptography", "fsspec[http]>=2024.6.1,<2026", "importlib_resources>=6.1.1, <7", "numpy>=1.23,<3", "packaging>=20.9,<26", "pandas>=2.1.4,<3", "platformdirs<5", "pyarrow<19.0.0", "pydantic>=2.8.2, <3", "pyjwt>=2.0.0, <3", "pytimeparse>=1.1.8,<2", "pyyaml>=6.0,<7", "retrying>=1.3.3,<2", "s3fs>=2024.6.1,<2026", "scikit-learn<1.8", "scipy>=1.9,<2", "shap>=0.46.0,<1", "snowflake-connector-python[pandas]>=3.17.3,<4", "snowflake-snowpark-python>=1.17.0,<2,!=1.26.0", "snowflake.core>=1.0.2,<2", "sqlparse>=0.4,<1", "tqdm<5", "typing-extensions>=4.1.0,<5", "xgboost<4",]
 [[project.authors]]
 name = "Snowflake, Inc"
 email = "support@snowflake.com"
@@ -35,7 +35,7 @@ prophet = [ "prophet>=1.1.0, <2",]
 streamlit = [ "streamlit>=1.30.0,<2",]
 tensorflow = [ "tensorflow>=2.17.0,<3",]
 torch = [ "torch>=2.0.1,<3", "torchdata>=0.4,<1",]
-transformers = [ "sentence-transformers>=2.7.0,<4", "sentencepiece>=0.1.95,<0.2.0", "torch>=2.0.1,<3",]
+transformers = [ "sentence-transformers>=2.7.0,<4", "sentencepiece>=0.1.95,<0.2.0", "tokenizers>=0.15.1,<1", "torch>=2.0.1,<3", "transformers>=4.39.3,<5,!=4.51.3",]
 
 [tool.setuptools.package-data]
 "*" = [ "*",]

{snowflake_ml_python-1.19.0 → snowflake_ml_python-1.21.0}/snowflake/ml/_internal/env_utils.py

@@ -16,6 +16,7 @@ from snowflake.ml import version as snowml_version
 from snowflake.ml._internal import env as snowml_env, relax_version_strategy
 from snowflake.ml._internal.utils import query_result_checker
 from snowflake.snowpark import context, exceptions, session
+from snowflake.snowpark._internal import utils as snowpark_utils
 
 
 class CONDA_OS(Enum):
@@ -38,6 +39,21 @@ SNOWPARK_ML_PKG_NAME = "snowflake-ml-python"
 SNOWFLAKE_CONDA_CHANNEL_URL = "https://repo.anaconda.com/pkgs/snowflake"
 
 
+def get_execution_context() -> str:
+    """Detect execution context: EXTERNAL, SPCS, or SPROC.
+
+    Returns:
+        str: The execution context - "SPROC" if running in a stored procedure,
+             "SPCS" if running in SPCS ML runtime, "EXTERNAL" otherwise.
+    """
+    if snowpark_utils.is_in_stored_procedure():  # type: ignore[no-untyped-call]
+        return "SPROC"
+    elif snowml_env.IN_ML_RUNTIME:
+        return "SPCS"
+    else:
+        return "EXTERNAL"
+
+
 def _validate_pip_requirement_string(req_str: str) -> requirements.Requirement:
     """Validate the input pip requirement string according to PEP 508.
 

{snowflake_ml_python-1.19.0 → snowflake_ml_python-1.21.0}/snowflake/ml/_internal/platform_capabilities.py

@@ -18,6 +18,8 @@ logger = logging.getLogger(__name__)
 LIVE_COMMIT_PARAMETER = "ENABLE_LIVE_VERSION_IN_SDK"
 INLINE_DEPLOYMENT_SPEC_PARAMETER = "ENABLE_INLINE_DEPLOYMENT_SPEC_FROM_CLIENT_VERSION"
 SET_MODULE_FUNCTIONS_VOLATILITY_FROM_MANIFEST = "SET_MODULE_FUNCTIONS_VOLATILITY_FROM_MANIFEST"
+ENABLE_MODEL_METHOD_SIGNATURE_PARAMETERS = "ENABLE_MODEL_METHOD_SIGNATURE_PARAMETERS"
+FEATURE_MODEL_INFERENCE_AUTOCAPTURE = "FEATURE_MODEL_INFERENCE_AUTOCAPTURE"
 
 
 class PlatformCapabilities:
@@ -80,6 +82,12 @@ class PlatformCapabilities:
     def is_live_commit_enabled(self) -> bool:
         return self._get_bool_feature(LIVE_COMMIT_PARAMETER, False)
 
+    def is_model_method_signature_parameters_enabled(self) -> bool:
+        return self._get_bool_feature(ENABLE_MODEL_METHOD_SIGNATURE_PARAMETERS, False)
+
+    def is_inference_autocapture_enabled(self) -> bool:
+        return self._is_feature_enabled(FEATURE_MODEL_INFERENCE_AUTOCAPTURE)
+
     @staticmethod
     def _get_features(session: snowpark_session.Session) -> dict[str, Any]:
         try:
@@ -182,3 +190,31 @@ class PlatformCapabilities:
             f"current={current_version}, feature={feature_version}, enabled={result}"
         )
         return result
+
+    def _is_feature_enabled(self, feature_name: str) -> bool:
+        """Check if the feature parameter value belongs to enabled values.
+
+        Args:
+            feature_name: The name of the feature to retrieve.
+
+        Returns:
+            bool: True if the value is "ENABLED" or "ENABLED_PUBLIC_PREVIEW",
+                  False if the value is "DISABLED", "DISABLED_PRIVATE_PREVIEW", or not set.
+
+        Raises:
+            ValueError: If the feature value is set but not one of the recognized values.
+        """
+        value = self.features.get(feature_name)
+        if value is None:
+            logger.debug(f"Feature {feature_name} not found.")
+            return False
+
+        if isinstance(value, str):
+            value_str = str(value)
+            if value_str.upper() in ["ENABLED", "ENABLED_PUBLIC_PREVIEW"]:
+                return True
+            elif value_str.upper() in ["DISABLED", "DISABLED_PRIVATE_PREVIEW"]:
+                return False
+            else:
+                raise ValueError(f"Invalid feature parameter value: {value} for feature {feature_name}")
+        raise ValueError(f"Invalid feature parameter string value: {value} for feature {feature_name}")

{snowflake_ml_python-1.19.0 → snowflake_ml_python-1.21.0}/snowflake/ml/_internal/telemetry.py

@@ -16,7 +16,7 @@ from typing_extensions import ParamSpec
 from snowflake import connector
 from snowflake.connector import connect, telemetry as connector_telemetry, time_util
 from snowflake.ml import version as snowml_version
-from snowflake.ml._internal import env
+from snowflake.ml._internal import env, env_utils
 from snowflake.ml._internal.exceptions import (
     error_codes,
     exceptions as snowml_exceptions,
@@ -37,6 +37,22 @@ _CONNECTION_TYPES = {
 _Args = ParamSpec("_Args")
 _ReturnValue = TypeVar("_ReturnValue")
 
+_conn: Optional[connector.SnowflakeConnection] = None
+
+
+def clear_cached_conn() -> None:
+    """Clear the cached Snowflake connection. Primarily for testing purposes."""
+    global _conn
+    if _conn is not None and _conn.is_valid():
+        _conn.close()
+    _conn = None
+
+
+def get_cached_conn() -> Optional[connector.SnowflakeConnection]:
+    """Get the cached Snowflake connection. Primarily for testing purposes."""
+    global _conn
+    return _conn
+
 
 def _get_login_token() -> Union[str, bytes]:
     with open("/snowflake/session/token") as f:
@@ -44,7 +60,11 @@ def _get_login_token() -> Union[str, bytes]:
 
 
 def _get_snowflake_connection() -> Optional[connector.SnowflakeConnection]:
-    conn = None
+    global _conn
+    if _conn is not None and _conn.is_valid():
+        return _conn
+
+    conn: Optional[connector.SnowflakeConnection] = None
     if os.getenv("SNOWFLAKE_HOST") is not None and os.getenv("SNOWFLAKE_ACCOUNT") is not None:
         try:
             conn = connect(
@@ -66,6 +86,13 @@ def _get_snowflake_connection() -> Optional[connector.SnowflakeConnection]:
             # Failed to get an active session. No connection available.
             pass
 
+    # cache the connection if it's a SnowflakeConnection. there is a behavior at runtime where it could be a
+    # StoredProcConnection perhaps incorrect type hinting somewhere
+    if isinstance(conn, connector.SnowflakeConnection):
+        # if _conn was expired, we need to copy telemetry data to new connection
+        if _conn is not None and conn is not None:
+            conn._telemetry._log_batch.extend(_conn._telemetry._log_batch)
+        _conn = conn
     return conn
 
 
@@ -113,6 +140,13 @@ class TelemetryField(enum.Enum):
     FUNC_CAT_USAGE = "usage"
 
 
+@enum.unique
+class CustomTagKey(enum.Enum):
+    """Keys for custom tags in telemetry."""
+
+    EXECUTION_CONTEXT = "execution_context"
+
+
 class _TelemetrySourceType(enum.Enum):
     # Automatically inferred telemetry/statement parameters
     AUTO_TELEMETRY = "SNOWML_AUTO_TELEMETRY"
@@ -441,6 +475,7 @@ def send_api_usage_telemetry(
     sfqids_extractor: Optional[Callable[..., list[str]]] = None,
     subproject_extractor: Optional[Callable[[Any], str]] = None,
     custom_tags: Optional[dict[str, Union[bool, int, str, float]]] = None,
+    log_execution_context: bool = True,
 ) -> Callable[[Callable[_Args, _ReturnValue]], Callable[_Args, _ReturnValue]]:
     """
     Decorator that sends API usage telemetry and adds function usage statement parameters to the dataframe returned by
@@ -455,6 +490,8 @@ def send_api_usage_telemetry(
         sfqids_extractor: Extract sfqids from `self`.
         subproject_extractor: Extract subproject at runtime from `self`.
         custom_tags: Custom tags.
+        log_execution_context: If True, automatically detect and log execution context
+            (EXTERNAL, SPCS, or SPROC) in custom_tags.
 
     Returns:
         Decorator that sends function usage telemetry for any call to the decorated function.
@@ -495,6 +532,11 @@ def send_api_usage_telemetry(
             if subproject_extractor is not None:
                 subproject_name = subproject_extractor(args[0])
 
+            # Add execution context if enabled
+            final_custom_tags = {**custom_tags} if custom_tags is not None else {}
+            if log_execution_context:
+                final_custom_tags[CustomTagKey.EXECUTION_CONTEXT.value] = env_utils.get_execution_context()
+
             statement_params = get_function_usage_statement_params(
                 project=project,
                 subproject=subproject_name,
@@ -502,7 +544,7 @@ def send_api_usage_telemetry(
                 function_name=_get_full_func_name(func),
                 function_parameters=params,
                 api_calls=api_calls,
-                custom_tags=custom_tags,
+                custom_tags=final_custom_tags,
             )
 
             def update_stmt_params_if_snowpark_df(obj: _ReturnValue, statement_params: dict[str, Any]) -> _ReturnValue:
@@ -538,7 +580,10 @@ def send_api_usage_telemetry(
             if conn_attr_name:
                 # raise AttributeError if conn attribute does not exist in `self`
                 conn = operator.attrgetter(conn_attr_name)(args[0])
-                if not isinstance(conn, _CONNECTION_TYPES.get(type(conn).__name__, connector.SnowflakeConnection)):
+                if not isinstance(
+                    conn,
+                    _CONNECTION_TYPES.get(type(conn).__name__, connector.SnowflakeConnection),
+                ):
                     raise TypeError(
                         f"Expected a conn object of type {' or '.join(_CONNECTION_TYPES.keys())} but got {type(conn)}"
                     )
@@ -560,7 +605,7 @@ def send_api_usage_telemetry(
                 func_params=params,
                 api_calls=api_calls,
                 sfqids=sfqids,
-                custom_tags=custom_tags,
+                custom_tags=final_custom_tags,
             )
             try:
                 return ctx.run(execute_func_with_statement_params)
@@ -571,7 +616,8 @@ def send_api_usage_telemetry(
                         raise
                     if isinstance(e, snowpark_exceptions.SnowparkClientException):
                         me = snowml_exceptions.SnowflakeMLException(
-                            error_code=error_codes.INTERNAL_SNOWPARK_ERROR, original_exception=e
+                            error_code=error_codes.INTERNAL_SNOWPARK_ERROR,
+                            original_exception=e,
                         )
                     else:
                         me = snowml_exceptions.SnowflakeMLException(
@@ -627,7 +673,10 @@ def _get_full_func_name(func: Callable[..., Any]) -> str:
 
 
 def _get_func_params(
-    func: Callable[..., Any], func_params_to_log: Optional[Iterable[str]], args: Any, kwargs: Any
+    func: Callable[..., Any],
+    func_params_to_log: Optional[Iterable[str]],
+    args: Any,
+    kwargs: Any,
 ) -> dict[str, Any]:
     """
     Get function parameters.

{snowflake_ml_python-1.19.0 → snowflake_ml_python-1.21.0}/snowflake/ml/data/_internal/arrow_ingestor.py

@@ -1,6 +1,8 @@
+import base64
 import collections
 import logging
 import os
+import re
 import time
 from typing import TYPE_CHECKING, Any, Deque, Iterator, Optional, Sequence, Union
 
@@ -165,8 +167,71 @@ class ArrowIngestor(data_ingestor.DataIngestor, mixins.SerializableSessionMixin)
         # Re-shuffle input files on each iteration start
         if shuffle:
             np.random.shuffle(sources)
-        pa_dataset: pds.Dataset = pds.dataset(sources, format=format, **self._kwargs)
-        return pa_dataset
+        try:
+            pa_dataset: pds.Dataset = pds.dataset(sources, format=format, **self._kwargs)
+            return pa_dataset
+        except Exception as e:
+            self._tmp_debug_parquet_invalid(e, sources)
+
+    def _tmp_debug_parquet_invalid(self, e: Exception, sources: list[Any]) -> None:
+        # Attach rich debug info to help diagnose intermittent Parquet footer/magic byte errors
+        debug_parts: list[str] = []
+        debug_parts.append("SNOWML DEBUG: Failed to construct Arrow Dataset")
+        debug_parts.append(
+            "SNOWML DEBUG: " f"data_sources_count={len(self._data_sources)} " f"resolved_sources_count={len(sources)}"
+        )
+        # Try to include the exact file path mentioned by pyarrow, if present
+        error_text = str(e)
+        snow_paths: list[str] = []
+        try:
+            # Extract snow://... tokens possibly wrapped in quotes
+            for match in re.finditer(r'(snow://[^\s\'"]+)', error_text):
+                token = match.group(1).rstrip(").,;]")
+                snow_paths.append(token)
+        except Exception:
+            pass
+        fs = self._kwargs.get("filesystem")
+        if fs is not None:
+            # Always include a directory listing with sizes for context
+            try:
+                debug_parts.append("SNOWML DEBUG: Listing resolved sources with sizes:")
+                for s in sources:
+                    try:
+                        info = fs.info(s)
+                        size = info.get("size", None)
+                        md5 = info.get("md5", None)
+                        debug_parts.append(f"  - {s} size={size} md5={md5}")
+                    except Exception as le:
+                        debug_parts.append(f"  - {s} info_failed={le}")
+            except Exception as le:
+                debug_parts.append(f"SNOWML DEBUG: listing sources failed: {le}")
+            # If pyarrow referenced a specific file, dump its full contents (base64) for inspection
+            for path in snow_paths[:1]:  # usually only one path appears in the message
+                try:
+                    info = fs.info(path)
+                    size = info.get("size", None)
+                    debug_parts.append(f"SNOWML DEBUG: Inspecting referenced file: {path} size={size}")
+                    with fs.open(path, "rb") as f:
+                        content = f.read()
+                    magic_head = content[:4]
+                    magic_tail = content[-4:] if content else b""
+                    looks_like_parquet = (magic_head == b"PAR1") and (magic_tail == b"PAR1")
+                    debug_parts.append(
+                        "SNOWML DEBUG: "
+                        f"file_magic_head={magic_head!r} "
+                        f"file_magic_tail={magic_tail!r} "
+                        f"parquet_magic_detected={looks_like_parquet}"
+                    )
+                    b64 = base64.b64encode(content).decode("ascii")
+                    debug_parts.append("SNOWML DEBUG: file_content_base64 (entire file):")
+                    debug_parts.append(b64)
+                except Exception as fe:
+                    debug_parts.append(f"SNOWML DEBUG: failed to read referenced file {path}: {fe}")
+        else:
+            debug_parts.append("SNOWML DEBUG: No filesystem available; cannot inspect files")
+        debug_message = "\n".join(debug_parts)
+        # Re-raise with augmented message to surface in stacktrace
+        raise RuntimeError(f"{e}\n{debug_message}") from e
 
     def _get_batches_from_buffer(self, batch_size: int) -> dict[str, npt.NDArray[Any]]:
         """Generate new batches from the existing record batch buffer."""