snowflake-ml-python 1.20.0__tar.gz → 1.22.0__tar.gz

{snowflake_ml_python-1.20.0 → snowflake_ml_python-1.22.0}/CHANGELOG.md

@@ -1,14 +1,90 @@
 # Release History
 
-## 1.20.0
+## 1.22.0
+
+### New Features
+
+* Registry: Introducing remotely logging a transformer pipeline model using SPCS job.
+
+```python
+# create reference of the model
+model = huggingface.TransformersPipeline(
+    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+    task="text-generation",
+)
+
+# Remotely log the model, a SPCS job will run async and log the model
+mv = registry.log_model(
+    model=model,
+    model_name="tinyllama_remote_log",
+    target_platforms=["SNOWPARK_CONTAINER_SERVICES"],
+    signatures=openai_signatures.OPENAI_CHAT_SIGNATURE,
+)
+```
+
+### Bug Fixes
+
+### Behavior Changes
+
+### Deprecations
+
+## 1.21.0
+
+### New Features
+
+* Inference Autocapture (Preview): The `create_service` API will now accept `autocapture` as a new argument to indicate
+  whether inference data will be captured.
+* DataConnector: Added `to_huggingface_dataset()` method for converting Snowflake data to HuggingFace datasets.
+  Supports both in-memory `Dataset` (streaming=False) and streaming `IterableDataset` (streaming=True) modes.
+* Introducing `snowflake.ml.model.models.huggingface.TransformersPipeline` model which will replace object
+`snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel`. This is a wrapper class to
+  `transformers.Pipeline` model. Currently, following tasks
+  are supported to log without manually specifying model signatures:
+  * "fill-mask"
+  * "question-answering"
+  * "summarization"
+  * "table-question-answering"
+  * "text2text-generation"
+  * "text-classification" (alias "sentiment-analysis" available)
+  * "text-generation"
+  * "token-classification" (alias "ner" available)
+  * "translation"
+  * "translation_xx_to_yy"
+  * "zero-shot-classification"
+  which provides utility to log the model without loading the model in memory.
 
 ### Bug Fixes
 
 * Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
   instead of raising an exception.
+* Registry: `ModelVersion.run()` will now raise a `ValueError` if the model does not support running on warehouse
+  (e.g. SPCS-only models) and `service_name` is not provided.
 
 ### Behavior Changes
 
+* ML Job: The `additional_payloads` (Preview API) behavior is changing.
+  Use the `imports` argument to declare additional dependencies, such as zip files and Python modules.
+  Local directories and Python files are automatically compressed, and their internal layout is determined by the
+  specified import path. The import path applies only to local directories, Python files and staged python files;
+  it has no effect on other import types. When referencing files in a stage, only individual files
+  are supported—not directories.
+* Online Inference (PuPr): `list_services()` now shows internal endpoint that does not need EAI to call from another
+  SPCS node or Notebook.
+* Inference Autocapture (Preview): `list_services()` now shows if model service has autocapture enabled.
+* Experiment Tracking (PuPr): `ExperimentTracking` is now a singleton class.
+
+### Deprecations
+
+* Deprecating `snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel` and will be removed in later
+ version with a notice.
+
+## 1.20.0
+
+### Bug Fixes
+
+* Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
+  instead of raising an exception.
+
 ### New Features
 
 * Registry (PrPr): Introducing vLLM as a backend inference engine. The `create_service` API will now
@@ -39,8 +115,6 @@ mv.create_service(
 )
 ```
 
-### Deprecations
-
 ## 1.19.0 (11-13-2025)
 
 ### Bug Fixes
@@ -147,8 +221,6 @@ options = {
 * Registry: Dropping support for deprecated `conversational` task type for Huggingface models.
   To read more <https://github.com/huggingface/transformers/pull/31165>
 
-### New Features
-
 ## 1.14.0 (09-18-2025)
 
 ### Bug Fixes

{snowflake_ml_python-1.20.0/snowflake_ml_python.egg-info → snowflake_ml_python-1.22.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: snowflake-ml-python
-Version: 1.20.0
+Version: 1.22.0
 Summary: The machine learning client library that is used for interacting with Snowflake to build machine learning solutions.
 Author-email: "Snowflake, Inc" <support@snowflake.com>
 License: 
@@ -230,7 +230,7 @@ Classifier: Topic :: Software Development :: Libraries
 Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
-Requires-Python: <3.13,>=3.9
+Requires-Python: <3.14,>=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
 Requires-Dist: anyio<5,>=3.5.0
@@ -307,7 +307,9 @@ Requires-Dist: torchdata<1,>=0.4; extra == "torch"
 Provides-Extra: transformers
 Requires-Dist: sentence-transformers<4,>=2.7.0; extra == "transformers"
 Requires-Dist: sentencepiece<0.2.0,>=0.1.95; extra == "transformers"
+Requires-Dist: tokenizers<1,>=0.15.1; extra == "transformers"
 Requires-Dist: torch<3,>=2.0.1; extra == "transformers"
+Requires-Dist: transformers!=4.51.3,<5,>=4.39.3; extra == "transformers"
 Dynamic: license-file
 
 # Snowflake ML Python
@@ -415,15 +417,91 @@ NOTE: Version 1.7.0 is used as example here. Please choose the the latest versio
 
 # Release History
 
-## 1.20.0
+## 1.22.0
+
+### New Features
+
+* Registry: Introducing remotely logging a transformer pipeline model using SPCS job.
+
+```python
+# create reference of the model
+model = huggingface.TransformersPipeline(
+    model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
+    task="text-generation",
+)
+
+# Remotely log the model, a SPCS job will run async and log the model
+mv = registry.log_model(
+    model=model,
+    model_name="tinyllama_remote_log",
+    target_platforms=["SNOWPARK_CONTAINER_SERVICES"],
+    signatures=openai_signatures.OPENAI_CHAT_SIGNATURE,
+)
+```
+
+### Bug Fixes
+
+### Behavior Changes
+
+### Deprecations
+
+## 1.21.0
+
+### New Features
+
+* Inference Autocapture (Preview): The `create_service` API will now accept `autocapture` as a new argument to indicate
+  whether inference data will be captured.
+* DataConnector: Added `to_huggingface_dataset()` method for converting Snowflake data to HuggingFace datasets.
+  Supports both in-memory `Dataset` (streaming=False) and streaming `IterableDataset` (streaming=True) modes.
+* Introducing `snowflake.ml.model.models.huggingface.TransformersPipeline` model which will replace object
+`snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel`. This is a wrapper class to
+  `transformers.Pipeline` model. Currently, following tasks
+  are supported to log without manually specifying model signatures:
+  * "fill-mask"
+  * "question-answering"
+  * "summarization"
+  * "table-question-answering"
+  * "text2text-generation"
+  * "text-classification" (alias "sentiment-analysis" available)
+  * "text-generation"
+  * "token-classification" (alias "ner" available)
+  * "translation"
+  * "translation_xx_to_yy"
+  * "zero-shot-classification"
+  which provides utility to log the model without loading the model in memory.
 
 ### Bug Fixes
 
 * Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
   instead of raising an exception.
+* Registry: `ModelVersion.run()` will now raise a `ValueError` if the model does not support running on warehouse
+  (e.g. SPCS-only models) and `service_name` is not provided.
 
 ### Behavior Changes
 
+* ML Job: The `additional_payloads` (Preview API) behavior is changing.
+  Use the `imports` argument to declare additional dependencies, such as zip files and Python modules.
+  Local directories and Python files are automatically compressed, and their internal layout is determined by the
+  specified import path. The import path applies only to local directories, Python files and staged python files;
+  it has no effect on other import types. When referencing files in a stage, only individual files
+  are supported—not directories.
+* Online Inference (PuPr): `list_services()` now shows internal endpoint that does not need EAI to call from another
+  SPCS node or Notebook.
+* Inference Autocapture (Preview): `list_services()` now shows if model service has autocapture enabled.
+* Experiment Tracking (PuPr): `ExperimentTracking` is now a singleton class.
+
+### Deprecations
+
+* Deprecating `snowflake.ml.model.models.huggingface_pipeline.HuggingfacePipelineModel` and will be removed in later
+ version with a notice.
+
+## 1.20.0
+
+### Bug Fixes
+
+* Experiment Tracking (PuPr): Reaching the run metadata size limit in `log_metrics` or `log_params` will warn the user
+  instead of raising an exception.
+
 ### New Features
 
 * Registry (PrPr): Introducing vLLM as a backend inference engine. The `create_service` API will now
@@ -454,8 +532,6 @@ mv.create_service(
 )
 ```
 
-### Deprecations
-
 ## 1.19.0 (11-13-2025)
 
 ### Bug Fixes
@@ -562,8 +638,6 @@ options = {
 * Registry: Dropping support for deprecated `conversational` task type for Huggingface models.
   To read more <https://github.com/huggingface/transformers/pull/31165>
 
-### New Features
-
 ## 1.14.0 (09-18-2025)
 
 ### Bug Fixes

{snowflake_ml_python-1.20.0 → snowflake_ml_python-1.22.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 name = "snowflake-ml-python"
 description = "The machine learning client library that is used for interacting with Snowflake to build machine learning solutions."
 classifiers = [ "Development Status :: 5 - Production/Stable", "Environment :: Console", "Environment :: Other Environment", "Intended Audience :: Developers", "Intended Audience :: Education", "Intended Audience :: Information Technology", "Intended Audience :: System Administrators", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Topic :: Database", "Topic :: Software Development", "Topic :: Software Development :: Libraries", "Topic :: Software Development :: Libraries :: Application Frameworks", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: Scientific/Engineering :: Information Analysis",]
-requires-python = ">=3.9, <3.13"
+requires-python = ">=3.9, <3.14"
 dynamic = [ "version", "readme",]
 dependencies = [ "anyio>=3.5.0,<5", "cachetools>=3.1.1,<6", "cloudpickle>=2.0.0", "cryptography", "fsspec[http]>=2024.6.1,<2026", "importlib_resources>=6.1.1, <7", "numpy>=1.23,<3", "packaging>=20.9,<26", "pandas>=2.1.4,<3", "platformdirs<5", "pyarrow<19.0.0", "pydantic>=2.8.2, <3", "pyjwt>=2.0.0, <3", "pytimeparse>=1.1.8,<2", "pyyaml>=6.0,<7", "retrying>=1.3.3,<2", "s3fs>=2024.6.1,<2026", "scikit-learn<1.8", "scipy>=1.9,<2", "shap>=0.46.0,<1", "snowflake-connector-python[pandas]>=3.17.3,<4", "snowflake-snowpark-python>=1.17.0,<2,!=1.26.0", "snowflake.core>=1.0.2,<2", "sqlparse>=0.4,<1", "tqdm<5", "typing-extensions>=4.1.0,<5", "xgboost<4",]
 [[project.authors]]
@@ -35,7 +35,7 @@ prophet = [ "prophet>=1.1.0, <2",]
 streamlit = [ "streamlit>=1.30.0,<2",]
 tensorflow = [ "tensorflow>=2.17.0,<3",]
 torch = [ "torch>=2.0.1,<3", "torchdata>=0.4,<1",]
-transformers = [ "sentence-transformers>=2.7.0,<4", "sentencepiece>=0.1.95,<0.2.0", "torch>=2.0.1,<3",]
+transformers = [ "sentence-transformers>=2.7.0,<4", "sentencepiece>=0.1.95,<0.2.0", "tokenizers>=0.15.1,<1", "torch>=2.0.1,<3", "transformers>=4.39.3,<5,!=4.51.3",]
 
 [tool.setuptools.package-data]
 "*" = [ "*",]

{snowflake_ml_python-1.20.0 → snowflake_ml_python-1.22.0}/snowflake/ml/_internal/platform_capabilities.py

@@ -18,6 +18,8 @@ logger = logging.getLogger(__name__)
 LIVE_COMMIT_PARAMETER = "ENABLE_LIVE_VERSION_IN_SDK"
 INLINE_DEPLOYMENT_SPEC_PARAMETER = "ENABLE_INLINE_DEPLOYMENT_SPEC_FROM_CLIENT_VERSION"
 SET_MODULE_FUNCTIONS_VOLATILITY_FROM_MANIFEST = "SET_MODULE_FUNCTIONS_VOLATILITY_FROM_MANIFEST"
+ENABLE_MODEL_METHOD_SIGNATURE_PARAMETERS = "ENABLE_MODEL_METHOD_SIGNATURE_PARAMETERS"
+FEATURE_MODEL_INFERENCE_AUTOCAPTURE = "FEATURE_MODEL_INFERENCE_AUTOCAPTURE"
 
 
 class PlatformCapabilities:
@@ -80,6 +82,12 @@ class PlatformCapabilities:
     def is_live_commit_enabled(self) -> bool:
         return self._get_bool_feature(LIVE_COMMIT_PARAMETER, False)
 
+    def is_model_method_signature_parameters_enabled(self) -> bool:
+        return self._get_bool_feature(ENABLE_MODEL_METHOD_SIGNATURE_PARAMETERS, False)
+
+    def is_inference_autocapture_enabled(self) -> bool:
+        return self._is_feature_enabled(FEATURE_MODEL_INFERENCE_AUTOCAPTURE)
+
     @staticmethod
     def _get_features(session: snowpark_session.Session) -> dict[str, Any]:
         try:
@@ -182,3 +190,31 @@ class PlatformCapabilities:
             f"current={current_version}, feature={feature_version}, enabled={result}"
         )
         return result
+
+    def _is_feature_enabled(self, feature_name: str) -> bool:
+        """Check if the feature parameter value belongs to enabled values.
+
+        Args:
+            feature_name: The name of the feature to retrieve.
+
+        Returns:
+            bool: True if the value is "ENABLED" or "ENABLED_PUBLIC_PREVIEW",
+                  False if the value is "DISABLED", "DISABLED_PRIVATE_PREVIEW", or not set.
+
+        Raises:
+            ValueError: If the feature value is set but not one of the recognized values.
+        """
+        value = self.features.get(feature_name)
+        if value is None:
+            logger.debug(f"Feature {feature_name} not found.")
+            return False
+
+        if isinstance(value, str):
+            value_str = str(value)
+            if value_str.upper() in ["ENABLED", "ENABLED_PUBLIC_PREVIEW"]:
+                return True
+            elif value_str.upper() in ["DISABLED", "DISABLED_PRIVATE_PREVIEW"]:
+                return False
+            else:
+                raise ValueError(f"Invalid feature parameter value: {value} for feature {feature_name}")
+        raise ValueError(f"Invalid feature parameter string value: {value} for feature {feature_name}")

snowflake_ml_python-1.22.0/snowflake/ml/_internal/utils/url.py

@@ -0,0 +1,42 @@
+from urllib.parse import urlunparse
+
+from snowflake import snowpark as snowpark
+
+JOB_URL_PREFIX = "#/compute/job/"
+SERVICE_URL_PREFIX = "#/compute/service/"
+
+
+def get_snowflake_url(
+    session: snowpark.Session,
+    url_path: str,
+    params: str = "",
+    query: str = "",
+    fragment: str = "",
+) -> str:
+    """Construct a Snowflake URL from session connection details and URL components.
+
+    Args:
+        session: The Snowpark session containing connection details.
+        url_path: The path component of the URL (e.g., "/compute/job/123").
+        params: Optional parameters for the URL (RFC 1808). Defaults to "".
+        query: Optional query string for the URL. Defaults to "".
+        fragment: Optional fragment identifier for the URL (e.g., "#section"). Defaults to "".
+
+    Returns:
+        A fully constructed Snowflake URL string with scheme, host, and specified components.
+    """
+    scheme = "https"
+    if hasattr(session.connection, "scheme"):
+        scheme = session.connection.scheme
+    host = session.connection.host
+
+    return urlunparse(
+        (
+            scheme,
+            host,
+            url_path,
+            params,
+            query,
+            fragment,
+        )
+    )

{snowflake_ml_python-1.20.0 → snowflake_ml_python-1.22.0}/snowflake/ml/data/_internal/arrow_ingestor.py

@@ -1,6 +1,8 @@
+import base64
 import collections
 import logging
 import os
+import re
 import time
 from typing import TYPE_CHECKING, Any, Deque, Iterator, Optional, Sequence, Union
 
@@ -165,8 +167,71 @@ class ArrowIngestor(data_ingestor.DataIngestor, mixins.SerializableSessionMixin)
         # Re-shuffle input files on each iteration start
         if shuffle:
             np.random.shuffle(sources)
-        pa_dataset: pds.Dataset = pds.dataset(sources, format=format, **self._kwargs)
-        return pa_dataset
+        try:
+            pa_dataset: pds.Dataset = pds.dataset(sources, format=format, **self._kwargs)
+            return pa_dataset
+        except Exception as e:
+            self._tmp_debug_parquet_invalid(e, sources)
+
+    def _tmp_debug_parquet_invalid(self, e: Exception, sources: list[Any]) -> None:
+        # Attach rich debug info to help diagnose intermittent Parquet footer/magic byte errors
+        debug_parts: list[str] = []
+        debug_parts.append("SNOWML DEBUG: Failed to construct Arrow Dataset")
+        debug_parts.append(
+            "SNOWML DEBUG: " f"data_sources_count={len(self._data_sources)} " f"resolved_sources_count={len(sources)}"
+        )
+        # Try to include the exact file path mentioned by pyarrow, if present
+        error_text = str(e)
+        snow_paths: list[str] = []
+        try:
+            # Extract snow://... tokens possibly wrapped in quotes
+            for match in re.finditer(r'(snow://[^\s\'"]+)', error_text):
+                token = match.group(1).rstrip(").,;]")
+                snow_paths.append(token)
+        except Exception:
+            pass
+        fs = self._kwargs.get("filesystem")
+        if fs is not None:
+            # Always include a directory listing with sizes for context
+            try:
+                debug_parts.append("SNOWML DEBUG: Listing resolved sources with sizes:")
+                for s in sources:
+                    try:
+                        info = fs.info(s)
+                        size = info.get("size", None)
+                        md5 = info.get("md5", None)
+                        debug_parts.append(f"  - {s} size={size} md5={md5}")
+                    except Exception as le:
+                        debug_parts.append(f"  - {s} info_failed={le}")
+            except Exception as le:
+                debug_parts.append(f"SNOWML DEBUG: listing sources failed: {le}")
+            # If pyarrow referenced a specific file, dump its full contents (base64) for inspection
+            for path in snow_paths[:1]:  # usually only one path appears in the message
+                try:
+                    info = fs.info(path)
+                    size = info.get("size", None)
+                    debug_parts.append(f"SNOWML DEBUG: Inspecting referenced file: {path} size={size}")
+                    with fs.open(path, "rb") as f:
+                        content = f.read()
+                    magic_head = content[:4]
+                    magic_tail = content[-4:] if content else b""
+                    looks_like_parquet = (magic_head == b"PAR1") and (magic_tail == b"PAR1")
+                    debug_parts.append(
+                        "SNOWML DEBUG: "
+                        f"file_magic_head={magic_head!r} "
+                        f"file_magic_tail={magic_tail!r} "
+                        f"parquet_magic_detected={looks_like_parquet}"
+                    )
+                    b64 = base64.b64encode(content).decode("ascii")
+                    debug_parts.append("SNOWML DEBUG: file_content_base64 (entire file):")
+                    debug_parts.append(b64)
+                except Exception as fe:
+                    debug_parts.append(f"SNOWML DEBUG: failed to read referenced file {path}: {fe}")
+        else:
+            debug_parts.append("SNOWML DEBUG: No filesystem available; cannot inspect files")
+        debug_message = "\n".join(debug_parts)
+        # Re-raise with augmented message to surface in stacktrace
+        raise RuntimeError(f"{e}\n{debug_message}") from e
 
     def _get_batches_from_buffer(self, batch_size: int) -> dict[str, npt.NDArray[Any]]:
         """Generate new batches from the existing record batch buffer."""

{snowflake_ml_python-1.20.0 → snowflake_ml_python-1.22.0}/snowflake/ml/data/data_connector.py

@@ -1,5 +1,15 @@
 import os
-from typing import TYPE_CHECKING, Any, Generator, Optional, Sequence, TypeVar
+from typing import (
+    TYPE_CHECKING,
+    Any,
+    Generator,
+    Literal,
+    Optional,
+    Sequence,
+    TypeVar,
+    Union,
+    overload,
+)
 
 import numpy.typing as npt
 from typing_extensions import deprecated
@@ -11,6 +21,7 @@ from snowflake.ml.data._internal.arrow_ingestor import ArrowIngestor
 from snowflake.snowpark import context as sp_context
 
 if TYPE_CHECKING:
+    import datasets as hf_datasets
     import pandas as pd
     import ray
     import tensorflow as tf
@@ -257,6 +268,97 @@ class DataConnector:
         except ImportError as e:
             raise ImportError("Ray is not installed, please install ray in your local environment.") from e
 
+    @overload
+    def to_huggingface_dataset(
+        self,
+        *,
+        streaming: Literal[False] = ...,
+        limit: Optional[int] = ...,
+    ) -> "hf_datasets.Dataset":
+        ...
+
+    @overload
+    def to_huggingface_dataset(
+        self,
+        *,
+        streaming: Literal[True],
+        limit: Optional[int] = ...,
+        batch_size: int = ...,
+        shuffle: bool = ...,
+        drop_last_batch: bool = ...,
+    ) -> "hf_datasets.IterableDataset":
+        ...
+
+    @telemetry.send_api_usage_telemetry(
+        project=_PROJECT,
+        subproject_extractor=lambda self: type(self).__name__,
+        func_params_to_log=["streaming", "limit", "batch_size", "shuffle", "drop_last_batch"],
+    )
+    def to_huggingface_dataset(
+        self,
+        *,
+        streaming: bool = False,
+        limit: Optional[int] = None,
+        batch_size: int = 1024,
+        shuffle: bool = False,
+        drop_last_batch: bool = False,
+    ) -> "Union[hf_datasets.Dataset, hf_datasets.IterableDataset]":
+        """Retrieve the Snowflake data as a HuggingFace Dataset.
+
+        Args:
+            streaming: If True, returns an IterableDataset that streams data in batches.
+                If False (default), returns an in-memory Dataset.
+            limit: Maximum number of rows to load. If None, loads all rows.
+            batch_size: Size of batches for internal data retrieval.
+            shuffle: Whether to shuffle the data. If True, files will be shuffled and rows
+                in each file will also be shuffled.
+            drop_last_batch: Whether to drop the last batch if it's smaller than batch_size.
+
+        Returns:
+            A HuggingFace Dataset (in-memory) or IterableDataset (streaming).
+        """
+        import datasets as hf_datasets
+
+        if streaming:
+            return self._to_huggingface_iterable_dataset(
+                limit=limit,
+                batch_size=batch_size,
+                shuffle=shuffle,
+                drop_last_batch=drop_last_batch,
+            )
+        else:
+            return hf_datasets.Dataset.from_pandas(self._ingestor.to_pandas(limit))
+
+    def _to_huggingface_iterable_dataset(
+        self,
+        *,
+        limit: Optional[int],
+        batch_size: int,
+        shuffle: bool,
+        drop_last_batch: bool,
+    ) -> "hf_datasets.IterableDataset":
+        """Create a HuggingFace IterableDataset that streams data in batches."""
+        import datasets as hf_datasets
+
+        def generator() -> Generator[dict[str, Any], None, None]:
+            rows_yielded = 0
+            for batch in self._ingestor.to_batches(batch_size, shuffle, drop_last_batch):
+                # Yield individual rows from each batch
+                num_rows = len(next(iter(batch.values())))
+                for i in range(num_rows):
+                    if limit is not None and rows_yielded >= limit:
+                        return
+                    yield {k: v[i].item() if hasattr(v[i], "item") else v[i] for k, v in batch.items()}
+                    rows_yielded += 1
+
+        result = hf_datasets.IterableDataset.from_generator(generator)
+        # In datasets >= 3.x, from_generator returns IterableDatasetDict
+        # We need to extract the IterableDataset from the dict
+        if hasattr(hf_datasets, "IterableDatasetDict") and isinstance(result, hf_datasets.IterableDatasetDict):
+            # Get the first (and only) dataset from the dict
+            result = next(iter(result.values()))
+        return result
+
 
 # Switch to use Runtime's Data Ingester if running in ML runtime
 # Fail silently if the data ingester is not found

{snowflake_ml_python-1.20.0 → snowflake_ml_python-1.22.0}/snowflake/ml/experiment/_client/experiment_tracking_sql_client.py

@@ -41,8 +41,14 @@ class ExperimentTrackingSQLClient(_base._BaseSQLClient):
         ).has_dimensions(expected_rows=1, expected_cols=1).validate()
 
     @telemetry.send_api_usage_telemetry(project=telemetry.TelemetryProject.EXPERIMENT_TRACKING.value)
-    def drop_experiment(self, *, experiment_name: sql_identifier.SqlIdentifier) -> None:
-        experiment_fqn = self.fully_qualified_object_name(self._database_name, self._schema_name, experiment_name)
+    def drop_experiment(
+        self,
+        *,
+        database_name: sql_identifier.SqlIdentifier,
+        schema_name: sql_identifier.SqlIdentifier,
+        experiment_name: sql_identifier.SqlIdentifier,
+    ) -> None:
+        experiment_fqn = self.fully_qualified_object_name(database_name, schema_name, experiment_name)
         query_result_checker.SqlResultValidator(self._session, f"DROP EXPERIMENT {experiment_fqn}").has_dimensions(
             expected_rows=1, expected_cols=1
         ).validate()

{snowflake_ml_python-1.20.0 → snowflake_ml_python-1.22.0}/snowflake/ml/experiment/callback/keras.py

@@ -12,6 +12,8 @@ if TYPE_CHECKING:
 
 
 class SnowflakeKerasCallback(keras.callbacks.Callback):
+    """Keras callback for automatically logging to a Snowflake ML Experiment."""
+
     def __init__(
         self,
         experiment_tracking: "ExperimentTracking",
@@ -23,12 +25,33 @@ class SnowflakeKerasCallback(keras.callbacks.Callback):
         version_name: Optional[str] = None,
         model_signature: Optional["ModelSignature"] = None,
     ) -> None:
+        """
+        Creates a new Keras callback.
+
+        Args:
+            experiment_tracking (snowflake.ml.experiment.ExperimentTracking): The Experiment Tracking instance
+                to use for logging.
+            log_model (bool): Whether to log the model at the end of training. Default is True.
+            log_metrics (bool): Whether to log metrics during training. Default is True.
+            log_params (bool): Whether to log model parameters at the start of training. Default is True.
+            log_every_n_epochs (int): Frequency with which to log metrics. Must be positive.
+                Default is 1, logging after every epoch.
+            model_name (Optional[str]): The model name to use when logging the model.
+                If None, the model name will be derived from the experiment name.
+            version_name (Optional[str]): The model version name to use when logging the model.
+                If None, the version name will be randomly generated.
+            model_signature (Optional[snowflake.ml.model.model_signature.ModelSignature]): The model signature to use
+                when logging the model. This is required if ``log_model`` is set to True.
+
+        Raises:
+            ValueError: When ``log_every_n_epochs`` is not a positive integer.
+        """
         self._experiment_tracking = experiment_tracking
         self.log_model = log_model
         self.log_metrics = log_metrics
         self.log_params = log_params
-        if log_every_n_epochs < 1:
-            raise ValueError("`log_every_n_epochs` must be positive.")
+        if not (utils.is_integer(log_every_n_epochs) and log_every_n_epochs > 0):
+            raise ValueError("`log_every_n_epochs` must be a positive integer.")
         self.log_every_n_epochs = log_every_n_epochs
         self.model_name = model_name
         self.version_name = version_name