snowpark-checkpoints-collectors 0.1.0rc1__tar.gz → 0.2.0rc1__tar.gz

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: snowpark-checkpoints-collectors
-Version: 0.1.0rc1
+Version: 0.2.0rc1
 Summary: Snowpark column and table statistics collection
 Project-URL: Bug Tracker, https://github.com/snowflakedb/snowpark-checkpoints/issues
 Project-URL: Source code, https://github.com/snowflakedb/snowpark-checkpoints/
@@ -243,7 +243,7 @@ Requires-Dist: setuptools>=70.0.0; extra == 'development'
 Requires-Dist: twine==5.1.1; extra == 'development'
 Description-Content-Type: text/markdown
 
-# Data Collection from Spark Pipelines
+# snowpark-checkpoints-collectors
 
 ---
 **NOTE**
@@ -252,25 +252,96 @@ This package is on Private Preview.
 
 ---
 
-The `snowpark-checkpoints-collector` package can collect
-schema and check information from a spark pipeline and
-record those results into a set of JSON files corresponding to different intermediate dataframes. These files can be inspected manually
-and handed over to teams implementing the snowpark pipeline. The `snowpark-checkpoints-collector` package is designed to have minimal
-dependencies and the generated files are meant to be inspected by security
-teams.
+**snowpark-checkpoints-collector** package offers a function for extracting information from PySpark dataframes. We can then use that data to validate against the converted Snowpark dataframes to ensure that behavioral equivalence has been achieved.
+## Features
 
-On the snowpark side the `snowpark-checkpoints` package can use these files to perform schema and data validation checks against snowpark dataframes at the same, intermediate logical "checkpoints".
+- Schema inference collected data mode (Schema): This is the default mode, which leverages Pandera schema inference to obtain the metadata and checks that will be evaluated for the specified dataframe. This mode also collects custom data from columns of the DataFrame based on the PySpark type.
+- DataFrame collected data mode (DataFrame): This mode collects the data of the PySpark dataframe. In this case, the mechanism saves all data of the given dataframe in parquet format. Using the default user Snowflake connection, it tries to upload the parquet files into the Snowflake temporal stage and create a table based on the information in the stage. The name of the file and the table is the same as the checkpoint.
 
-## collect_dataframe_schema
 
+
+## Functionalities
+
+### Collect DataFrame Checkpoint
+
+
+
+```python
+from pyspark.sql import DataFrame as SparkDataFrame
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+from typing import Optional
+
+# Signature of the function
+def collect_dataframe_checkpoint(
+    df: SparkDataFrame,
+    checkpoint_name: str,
+    sample: Optional[float] = None,
+    mode: Optional[CheckpointMode] = None,
+    output_path: Optional[str] = None,
+) -> None:
+    ...
+```
+
+- `df`: The input Spark dataframe to collect.
+- `checkpoint_name`: Name of the checkpoint schema file or dataframe.
+- `sample`: Fraction of DataFrame to sample for schema inference, defaults to 1.0.
+- `mode`: The mode to execution the collection (Schema or Dataframe), defaults to CheckpointMode.Schema.
+- `output_path`: The output path to save the checkpoint, defaults to current working directory.
+
+
+## Usage Example
+
+### Schema mode
+
+```python
+from pyspark.sql import SparkSession
+from snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+
+spark_session = SparkSession.builder.getOrCreate()
+sample_size = 1.0
+
+pyspark_df = spark_session.createDataFrame(
+    [("apple", 21), ("lemon", 34), ("banana", 50)], schema="fruit string, age integer"
+)
+
+collect_dataframe_checkpoint(
+    pyspark_df,
+    checkpoint_name="collect_checkpoint_mode_1",
+    sample=sample_size,
+    mode=CheckpointMode.SCHEMA,
+)
 ```
-from snowflake.snowpark_checkpoints_collector import collect_dataframe_schema;
-collect_dataframe_schema(df:SparkDataFrame,
-                              checkpoint_name,
-                              sample=0.1)
+
+
+### Dataframe mode
+
+```python
+from pyspark.sql import SparkSession
+from snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+from pyspark.sql.types import StructType, StructField, ByteType, StringType, IntegerType 
+
+spark_schema = StructType(
+    [
+        StructField("BYTE", ByteType(), True),
+        StructField("STRING", StringType(), True),
+        StructField("INTEGER", IntegerType(), True)
+    ]
+)
+
+data = [(1, "apple", 21), (2, "lemon", 34), (3, "banana", 50)]
+
+spark_session = SparkSession.builder.getOrCreate()
+pyspark_df = spark_session.createDataFrame(data, schema=spark_schema).orderBy(
+    "INTEGER"
+)
+
+collect_dataframe_checkpoint(
+    pyspark_df,
+    checkpoint_name="collect_checkpoint_mode_2",
+    mode=CheckpointMode.DATAFRAME,
+)
 ```
 
-- df - the spark data frame to collect the schema from
-- checkpoint_name - the name of the "checkpoint". Generated JSON files
-  will have the name "snowpark-[checkpoint_name]-schema.json"
-- sample - sample size of the spark data frame to use to generate the schema
+------

snowpark_checkpoints_collectors-0.2.0rc1/README.md

@@ -0,0 +1,102 @@
+# snowpark-checkpoints-collectors
+
+---
+**NOTE**
+
+This package is on Private Preview.
+
+---
+
+**snowpark-checkpoints-collector** package offers a function for extracting information from PySpark dataframes. We can then use that data to validate against the converted Snowpark dataframes to ensure that behavioral equivalence has been achieved.
+## Features
+
+- Schema inference collected data mode (Schema): This is the default mode, which leverages Pandera schema inference to obtain the metadata and checks that will be evaluated for the specified dataframe. This mode also collects custom data from columns of the DataFrame based on the PySpark type.
+- DataFrame collected data mode (DataFrame): This mode collects the data of the PySpark dataframe. In this case, the mechanism saves all data of the given dataframe in parquet format. Using the default user Snowflake connection, it tries to upload the parquet files into the Snowflake temporal stage and create a table based on the information in the stage. The name of the file and the table is the same as the checkpoint.
+
+
+
+## Functionalities
+
+### Collect DataFrame Checkpoint
+
+
+
+```python
+from pyspark.sql import DataFrame as SparkDataFrame
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+from typing import Optional
+
+# Signature of the function
+def collect_dataframe_checkpoint(
+    df: SparkDataFrame,
+    checkpoint_name: str,
+    sample: Optional[float] = None,
+    mode: Optional[CheckpointMode] = None,
+    output_path: Optional[str] = None,
+) -> None:
+    ...
+```
+
+- `df`: The input Spark dataframe to collect.
+- `checkpoint_name`: Name of the checkpoint schema file or dataframe.
+- `sample`: Fraction of DataFrame to sample for schema inference, defaults to 1.0.
+- `mode`: The mode to execution the collection (Schema or Dataframe), defaults to CheckpointMode.Schema.
+- `output_path`: The output path to save the checkpoint, defaults to current working directory.
+
+
+## Usage Example
+
+### Schema mode
+
+```python
+from pyspark.sql import SparkSession
+from snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+
+spark_session = SparkSession.builder.getOrCreate()
+sample_size = 1.0
+
+pyspark_df = spark_session.createDataFrame(
+    [("apple", 21), ("lemon", 34), ("banana", 50)], schema="fruit string, age integer"
+)
+
+collect_dataframe_checkpoint(
+    pyspark_df,
+    checkpoint_name="collect_checkpoint_mode_1",
+    sample=sample_size,
+    mode=CheckpointMode.SCHEMA,
+)
+```
+
+
+### Dataframe mode
+
+```python
+from pyspark.sql import SparkSession
+from snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+from pyspark.sql.types import StructType, StructField, ByteType, StringType, IntegerType 
+
+spark_schema = StructType(
+    [
+        StructField("BYTE", ByteType(), True),
+        StructField("STRING", StringType(), True),
+        StructField("INTEGER", IntegerType(), True)
+    ]
+)
+
+data = [(1, "apple", 21), (2, "lemon", 34), (3, "banana", 50)]
+
+spark_session = SparkSession.builder.getOrCreate()
+pyspark_df = spark_session.createDataFrame(data, schema=spark_schema).orderBy(
+    "INTEGER"
+)
+
+collect_dataframe_checkpoint(
+    pyspark_df,
+    checkpoint_name="collect_checkpoint_mode_2",
+    mode=CheckpointMode.DATAFRAME,
+)
+```
+
+------

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/src/snowflake/snowpark_checkpoints_collector/__init__.py

@@ -2,9 +2,10 @@
 # Copyright (c) 2012-2024 Snowflake Computing Inc. All rights reserved.
 #
 
-__all__ = ["collect_dataframe_checkpoint", "Singleton"]
+__all__ = ["collect_dataframe_checkpoint", "CheckpointMode"]
 
-from snowflake.snowpark_checkpoints_collector.singleton import Singleton
 from snowflake.snowpark_checkpoints_collector.summary_stats_collector import (
     collect_dataframe_checkpoint,
 )
+
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/src/snowflake/snowpark_checkpoints_collector/collection_common.py

@@ -8,8 +8,13 @@ from enum import IntEnum
 
 
 class CheckpointMode(IntEnum):
+
+    """Enum class representing the collection mode."""
+
     SCHEMA = 1
+    """Collect automatic schema inference"""
     DATAFRAME = 2
+    """Export DataFrame as Parquet file to Snowflake"""
 
 
 # CONSTANTS

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/src/snowflake/snowpark_checkpoints_collector/collection_result/model/collection_point_result_manager.py

@@ -5,10 +5,10 @@ import json
 
 from typing import Optional
 
-from snowflake.snowpark_checkpoints_collector import Singleton
 from snowflake.snowpark_checkpoints_collector.collection_result.model import (
     CollectionPointResult,
 )
+from snowflake.snowpark_checkpoints_collector.singleton import Singleton
 from snowflake.snowpark_checkpoints_collector.utils import file_utils
 
 

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/src/snowflake/snowpark_checkpoints_collector/column_collection/model/timestamp_column_collector.py

@@ -14,7 +14,7 @@ from snowflake.snowpark_checkpoints_collector.column_collection.model.column_col
 )
 
 
-FORMAT = "%Y-%m-%dH:%M:%S"
+FORMAT = "%Y-%m-%dT%H:%M:%S%z"
 
 
 class TimestampColumnCollector(ColumnCollectorBase):

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/src/snowflake/snowpark_checkpoints_collector/column_collection/model/timestamp_ntz_column_collector.py

@@ -14,7 +14,7 @@ from snowflake.snowpark_checkpoints_collector.column_collection.model.column_col
 )
 
 
-FORMAT = "%Y-%m-%dT%H:%M:%S%z"
+FORMAT = "%Y-%m-%dH:%M:%S"
 
 
 class TimestampNTZColumnCollector(ColumnCollectorBase):

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/src/snowflake/snowpark_checkpoints_collector/column_pandera_checks/pandera_column_checks_manager.py

@@ -1,6 +1,7 @@
 #
 # Copyright (c) 2012-2024 Snowflake Computing Inc. All rights reserved.
 #
+import pandas as pd
 
 from pandas import DataFrame as PandasDataFrame
 from pandera import Check, Column
@@ -62,23 +63,6 @@ def column_register(*args):
     return wrapper
 
 
-def _datetime_like_type_checks(
-    clm_name: str, pandas_df: PandasDataFrame, pandera_column: Column
-) -> None:
-    column_values = pandas_df[clm_name].dropna()
-    min_value = str(column_values.min())
-    max_value = str(column_values.max())
-    pandera_column.checks.append(
-        Check.between(
-            min_value=min_value,
-            max_value=max_value,
-            include_max=True,
-            include_min=True,
-            title=BETWEEN_CHECK_ERROR_MESSAGE_FORMAT.format(min_value, max_value),
-        )
-    )
-
-
 @collector_register
 class PanderaColumnChecksManager:
 
@@ -117,13 +101,24 @@ class PanderaColumnChecksManager:
     def _add_date_type_checks(
         self, clm_name: str, pandas_df: PandasDataFrame, pandera_column: Column
     ) -> None:
-        _datetime_like_type_checks(clm_name, pandas_df, pandera_column)
+        pass
 
     @column_register(DAYTIMEINTERVAL_COLUMN_TYPE)
     def _add_daytimeinterval_type_checks(
         self, clm_name: str, pandas_df: PandasDataFrame, pandera_column: Column
     ) -> None:
-        _datetime_like_type_checks(clm_name, pandas_df, pandera_column)
+        column_values = pandas_df[clm_name].dropna()
+        min_value = pd.to_timedelta(column_values.min())
+        max_value = pd.to_timedelta(column_values.max())
+        pandera_column.checks.append(
+            Check.between(
+                min_value=min_value,
+                max_value=max_value,
+                include_max=True,
+                include_min=True,
+                title=BETWEEN_CHECK_ERROR_MESSAGE_FORMAT.format(min_value, max_value),
+            )
+        )
 
     @column_register(
         BYTE_COLUMN_TYPE,
@@ -153,16 +148,42 @@ class PanderaColumnChecksManager:
     def _add_string_type_checks(
         self, clm_name: str, pandas_df: PandasDataFrame, pandera_column: Column
     ) -> None:
-        pass
+        column_values = pandas_df[clm_name].dropna()
+        colum_str_length = column_values.str.len()
+        min_length = colum_str_length.min().item()
+        max_length = colum_str_length.max().item()
+        pandera_column.checks.append(Check.str_length(min_length, max_length))
 
     @column_register(TIMESTAMP_COLUMN_TYPE)
     def _add_timestamp_type_checks(
         self, clm_name: str, pandas_df: PandasDataFrame, pandera_column: Column
     ) -> None:
-        _datetime_like_type_checks(clm_name, pandas_df, pandera_column)
+        column_values = pandas_df[clm_name].dropna()
+        min_value = pd.Timestamp(column_values.min())
+        max_value = pd.Timestamp(column_values.max())
+        pandera_column.checks.append(
+            Check.between(
+                min_value=min_value,
+                max_value=max_value,
+                include_max=True,
+                include_min=True,
+                title=BETWEEN_CHECK_ERROR_MESSAGE_FORMAT.format(min_value, max_value),
+            )
+        )
 
     @column_register(TIMESTAMP_NTZ_COLUMN_TYPE)
     def _add_timestamp_ntz_type_checks(
         self, clm_name: str, pandas_df: PandasDataFrame, pandera_column: Column
     ) -> None:
-        _datetime_like_type_checks(clm_name, pandas_df, pandera_column)
+        column_values = pandas_df[clm_name].dropna()
+        min_value = pd.Timestamp(column_values.min())
+        max_value = pd.Timestamp(column_values.max())
+        pandera_column.checks.append(
+            Check.between(
+                min_value=min_value,
+                max_value=max_value,
+                include_max=True,
+                include_min=True,
+                title=BETWEEN_CHECK_ERROR_MESSAGE_FORMAT.format(min_value, max_value),
+            )
+        )

{snowpark_checkpoints_collectors-0.1.0rc1 → snowpark_checkpoints_collectors-0.2.0rc1}/src/snowflake/snowpark_checkpoints_collector/summary_stats_collector.py

@@ -41,7 +41,10 @@ from snowflake.snowpark_checkpoints_collector.column_pandera_checks import (
 from snowflake.snowpark_checkpoints_collector.snow_connection_model import (
     SnowConnection,
 )
-from snowflake.snowpark_checkpoints_collector.utils import file_utils
+from snowflake.snowpark_checkpoints_collector.utils import (
+    checkpoint_name_utils,
+    file_utils,
+)
 from snowflake.snowpark_checkpoints_collector.utils.extra_config import (
     get_checkpoint_mode,
     get_checkpoint_sample,
@@ -71,52 +74,83 @@ def collect_dataframe_checkpoint(
 
     Raises:
         Exception: Invalid mode value.
+        Exception: Invalid checkpoint name. Checkpoint names must only contain alphanumeric characters and underscores.
 
     """
-    collection_point_file_path = file_utils.get_collection_point_source_file_path()
-    collection_point_line_of_code = file_utils.get_collection_point_line_of_code()
-    collection_point_result = CollectionPointResult(
-        collection_point_file_path, collection_point_line_of_code, checkpoint_name
-    )
-
     try:
-        if is_checkpoint_enabled(checkpoint_name):
+        normalized_checkpoint_name = checkpoint_name_utils.normalize_checkpoint_name(
+            checkpoint_name
+        )
+        is_valid_checkpoint_name = checkpoint_name_utils.is_valid_checkpoint_name(
+            normalized_checkpoint_name
+        )
+        if not is_valid_checkpoint_name:
+            raise Exception(
+                f"Invalid checkpoint name: {checkpoint_name}. Checkpoint names must only contain alphanumeric "
+                f"characters and underscores."
+            )
 
-            _sample = get_checkpoint_sample(checkpoint_name, sample)
+        if is_checkpoint_enabled(normalized_checkpoint_name):
 
-            if _is_empty_dataframe_without_schema(df):
-                raise Exception(
-                    "It is not possible to collect an empty DataFrame without schema"
-                )
+            collection_point_file_path = (
+                file_utils.get_collection_point_source_file_path()
+            )
+            collection_point_line_of_code = (
+                file_utils.get_collection_point_line_of_code()
+            )
+            collection_point_result = CollectionPointResult(
+                collection_point_file_path,
+                collection_point_line_of_code,
+                normalized_checkpoint_name,
+            )
 
-            _mode = get_checkpoint_mode(checkpoint_name, mode)
+            try:
 
-            if _mode == CheckpointMode.SCHEMA:
-                column_type_dict = _get_spark_column_types(df)
-                _collect_dataframe_checkpoint_mode_schema(
-                    checkpoint_name, df, _sample, column_type_dict, output_path
-                )
+                _sample = get_checkpoint_sample(normalized_checkpoint_name, sample)
 
-            elif _mode == CheckpointMode.DATAFRAME:
-                snow_connection = SnowConnection()
-                _collect_dataframe_checkpoint_mode_dataframe(
-                    checkpoint_name, df, snow_connection, output_path
-                )
+                if _is_empty_dataframe_without_schema(df):
+                    raise Exception(
+                        "It is not possible to collect an empty DataFrame without schema"
+                    )
+
+                _mode = get_checkpoint_mode(normalized_checkpoint_name, mode)
+
+                if _mode == CheckpointMode.SCHEMA:
+                    column_type_dict = _get_spark_column_types(df)
+                    _collect_dataframe_checkpoint_mode_schema(
+                        normalized_checkpoint_name,
+                        df,
+                        _sample,
+                        column_type_dict,
+                        output_path,
+                    )
+
+                elif _mode == CheckpointMode.DATAFRAME:
+                    snow_connection = SnowConnection()
+                    _collect_dataframe_checkpoint_mode_dataframe(
+                        normalized_checkpoint_name, df, snow_connection, output_path
+                    )
 
-            else:
-                raise Exception("Invalid mode value.")
+                else:
+                    raise Exception("Invalid mode value.")
 
-            collection_point_result.result = CollectionResult.PASS
+                collection_point_result.result = CollectionResult.PASS
+
+            except Exception as err:
+                collection_point_result.result = CollectionResult.FAIL
+                error_message = str(err)
+                raise Exception(error_message) from err
+
+            finally:
+                collection_point_result_manager = CollectionPointResultManager(
+                    output_path
+                )
+                collection_point_result_manager.add_result(collection_point_result)
 
     except Exception as err:
-        collection_point_result.result = CollectionResult.FAIL
         error_message = str(err)
         raise Exception(error_message) from err
 
-    finally:
-        collection_point_result_manager = CollectionPointResultManager(output_path)
-        collection_point_result_manager.add_result(collection_point_result)
-
 
 @report_telemetry(params_list=["column_type_dict"])
 def _collect_dataframe_checkpoint_mode_schema(
@@ -239,7 +273,6 @@ def _get_pandera_infer_schema_as_dict(
 def _generate_json_checkpoint_file(
     checkpoint_name, dataframe_schema_contract, output_path: Optional[str] = None
 ) -> None:
-
     checkpoint_file_name = CHECKPOINT_JSON_OUTPUT_FILE_NAME_FORMAT.format(
         checkpoint_name
     )

snowpark_checkpoints_collectors-0.2.0rc1/src/snowflake/snowpark_checkpoints_collector/utils/checkpoint_name_utils.py

@@ -0,0 +1,49 @@
+import re as regx
+
+
+CHECKPOINT_NAME_REGEX_PATTERN = r"^[a-zA-Z_][a-zA-Z0-9_]+"
+WHITESPACE_TOKEN = " "
+HYPHEN_TOKEN = "-"
+UNDERSCORE_TOKEN = "_"
+TOKEN_TO_REPLACE_COLLECTION = [WHITESPACE_TOKEN, HYPHEN_TOKEN]
+
+
+def normalize_checkpoint_name(checkpoint_name: str) -> str:
+    """Normalize the provided checkpoint name by replacing: the whitespace and hyphen tokens by underscore token.
+
+    Args:
+        checkpoint_name (str): The checkpoint name to normalize.
+
+    Returns:
+        str: the checkpoint name normalized.
+
+    """
+    normalized_checkpoint_name = checkpoint_name
+    for token in TOKEN_TO_REPLACE_COLLECTION:
+        normalized_checkpoint_name = normalized_checkpoint_name.replace(
+            token, UNDERSCORE_TOKEN
+        )
+
+    return normalized_checkpoint_name
+
+
+def is_valid_checkpoint_name(checkpoint_name: str) -> bool:
+    """Check if the provided checkpoint name is valid.
+
+    A valid checkpoint name must:
+    - Start with a letter (a-z, A-Z) or an underscore (_)
+    - Be followed by any combination of letters, digits (0-9) and underscores (_).
+
+    Args:
+        checkpoint_name (str): The checkpoint name to validate.
+
+    Returns:
+        bool: True if the checkpoint name is valid; otherwise, False.
+
+    """
+    matched = regx.findall(CHECKPOINT_NAME_REGEX_PATTERN, checkpoint_name)
+    if len(matched) == 0 or len(matched) > 1:
+        return False
+
+    is_valid = matched[0] == checkpoint_name
+    return is_valid

snowpark_checkpoints_collectors-0.2.0rc1/test/integ/test_checkpoint_name.py

@@ -0,0 +1,51 @@
+import os
+import tempfile
+
+import pytest
+from pyspark.sql import SparkSession
+from pyspark.sql.types import StructType
+
+from snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint
+from snowflake.snowpark_checkpoints_collector.singleton import Singleton
+from pathlib import Path
+from snowflake.snowpark_checkpoints_collector.utils.telemetry import (
+    get_telemetry_manager,
+)
+
+telemetry_folder = "telemetry"
+
+
+@pytest.fixture
+def spark_session():
+    return SparkSession.builder.getOrCreate()
+
+
+@pytest.fixture
+def singleton():
+    Singleton._instances = {}
+
+
+@pytest.fixture(scope="function")
+def output_path():
+    folder = os.urandom(8).hex()
+    directory = Path(tempfile.gettempdir()).resolve() / folder
+    os.makedirs(directory)
+    return str(directory)
+
+
+def test_collect_dataframe_with_invalid_checkpoint_name(
+    spark_session, singleton, output_path
+):
+    checkpoint_name = "6*invalid"
+    data = []
+    columns = StructType()
+    pyspark_df = spark_session.createDataFrame(data=data, schema=columns)
+
+    with pytest.raises(Exception) as ex_info:
+        collect_dataframe_checkpoint(
+            pyspark_df, checkpoint_name=checkpoint_name, output_path=output_path
+        )
+    assert (
+        f"Invalid checkpoint name: {checkpoint_name}. Checkpoint names must only contain alphanumeric characters "
+        f"and underscores."
+    ) == str(ex_info.value)