snowpark-checkpoints-collectors 0.1.0rc1__py3-none-any.whl → 0.1.0rc2__py3-none-any.whl

{snowpark_checkpoints_collectors-0.1.0rc1.dist-info → snowpark_checkpoints_collectors-0.1.0rc2.dist-info}/METADATA

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: snowpark-checkpoints-collectors
-Version: 0.1.0rc1
+Version: 0.1.0rc2
 Summary: Snowpark column and table statistics collection
 Project-URL: Bug Tracker, https://github.com/snowflakedb/snowpark-checkpoints/issues
 Project-URL: Source code, https://github.com/snowflakedb/snowpark-checkpoints/
@@ -243,7 +243,7 @@ Requires-Dist: setuptools>=70.0.0; extra == 'development'
 Requires-Dist: twine==5.1.1; extra == 'development'
 Description-Content-Type: text/markdown
 
-# Data Collection from Spark Pipelines
+# snowpark-checkpoints-collectors
 
 ---
 **NOTE**
@@ -252,25 +252,96 @@ This package is on Private Preview.
 
 ---
 
-The `snowpark-checkpoints-collector` package can collect
-schema and check information from a spark pipeline and
-record those results into a set of JSON files corresponding to different intermediate dataframes. These files can be inspected manually
-and handed over to teams implementing the snowpark pipeline. The `snowpark-checkpoints-collector` package is designed to have minimal
-dependencies and the generated files are meant to be inspected by security
-teams.
+**snowpark-checkpoints-collector** package offers a function for extracting information from PySpark dataframes. We can then use that data to validate against the converted Snowpark dataframes to ensure that behavioral equivalence has been achieved.
+## Features
 
-On the snowpark side the `snowpark-checkpoints` package can use these files to perform schema and data validation checks against snowpark dataframes at the same, intermediate logical "checkpoints".
+- Schema inference collected data mode (Schema): This is the default mode, which leverages Pandera schema inference to obtain the metadata and checks that will be evaluated for the specified dataframe. This mode also collects custom data from columns of the DataFrame based on the PySpark type.
+- DataFrame collected data mode (DataFrame): This mode collects the data of the PySpark dataframe. In this case, the mechanism saves all data of the given dataframe in parquet format. Using the default user Snowflake connection, it tries to upload the parquet files into the Snowflake temporal stage and create a table based on the information in the stage. The name of the file and the table is the same as the checkpoint.
 
-## collect_dataframe_schema
 
+
+## Functionalities
+
+### Collect DataFrame Checkpoint
+
+
+
+```python
+from pyspark.sql import DataFrame as SparkDataFrame
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+from typing import Optional
+
+# Signature of the function
+def collect_dataframe_checkpoint(
+    df: SparkDataFrame,
+    checkpoint_name: str,
+    sample: Optional[float] = None,
+    mode: Optional[CheckpointMode] = None,
+    output_path: Optional[str] = None,
+) -> None:
+    ...
+```
+
+- `df`: The input Spark dataframe to collect.
+- `checkpoint_name`: Name of the checkpoint schema file or dataframe.
+- `sample`: Fraction of DataFrame to sample for schema inference, defaults to 1.0.
+- `mode`: The mode to execution the collection (Schema or Dataframe), defaults to CheckpointMode.Schema.
+- `output_path`: The output path to save the checkpoint, defaults to current working directory.
+
+
+## Usage Example
+
+### Schema mode
+
+```python
+from pyspark.sql import SparkSession
+from snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+
+spark_session = SparkSession.builder.getOrCreate()
+sample_size = 1.0
+
+pyspark_df = spark_session.createDataFrame(
+    [("apple", 21), ("lemon", 34), ("banana", 50)], schema="fruit string, age integer"
+)
+
+collect_dataframe_checkpoint(
+    pyspark_df,
+    checkpoint_name="collect_checkpoint_mode_1",
+    sample=sample_size,
+    mode=CheckpointMode.SCHEMA,
+)
 ```
-from snowflake.snowpark_checkpoints_collector import collect_dataframe_schema;
-collect_dataframe_schema(df:SparkDataFrame,
-                              checkpoint_name,
-                              sample=0.1)
+
+
+### Dataframe mode
+
+```python
+from pyspark.sql import SparkSession
+from snowflake.snowpark_checkpoints_collector import collect_dataframe_checkpoint
+from snowflake.snowpark_checkpoints_collector.collection_common import CheckpointMode
+from pyspark.sql.types import StructType, StructField, ByteType, StringType, IntegerType 
+
+spark_schema = StructType(
+    [
+        StructField("BYTE", ByteType(), True),
+        StructField("STRING", StringType(), True),
+        StructField("INTEGER", IntegerType(), True)
+    ]
+)
+
+data = [(1, "apple", 21), (2, "lemon", 34), (3, "banana", 50)]
+
+spark_session = SparkSession.builder.getOrCreate()
+pyspark_df = spark_session.createDataFrame(data, schema=spark_schema).orderBy(
+    "INTEGER"
+)
+
+collect_dataframe_checkpoint(
+    pyspark_df,
+    checkpoint_name="collect_checkpoint_mode_2",
+    mode=CheckpointMode.DATAFRAME,
+)
 ```
 
-- df - the spark data frame to collect the schema from
-- checkpoint_name - the name of the "checkpoint". Generated JSON files
-  will have the name "snowpark-[checkpoint_name]-schema.json"
-- sample - sample size of the spark data frame to use to generate the schema
+------

snowpark_checkpoints_collectors-0.1.0rc2.dist-info/RECORD

@@ -0,0 +1,4 @@
+snowpark_checkpoints_collectors-0.1.0rc2.dist-info/METADATA,sha256=8Ep1fq-1C-mR2_uhK11XBnnIOJth9ZMvACaVDKof0nw,18424
+snowpark_checkpoints_collectors-0.1.0rc2.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
+snowpark_checkpoints_collectors-0.1.0rc2.dist-info/licenses/LICENSE,sha256=pmjhbh6uVhV5MBXOlou_UZgFP7CYVQITkCCdvfcS5lY,11340
+snowpark_checkpoints_collectors-0.1.0rc2.dist-info/RECORD,,

snowpark_checkpoints_collectors-0.1.0rc1.dist-info/RECORD

@@ -1,4 +0,0 @@
-snowpark_checkpoints_collectors-0.1.0rc1.dist-info/METADATA,sha256=0-TZAWO6EULBbXeqMNXqEVpb-q0KnZy87pJvTkjVM6g,16247
-snowpark_checkpoints_collectors-0.1.0rc1.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
-snowpark_checkpoints_collectors-0.1.0rc1.dist-info/licenses/LICENSE,sha256=pmjhbh6uVhV5MBXOlou_UZgFP7CYVQITkCCdvfcS5lY,11340
-snowpark_checkpoints_collectors-0.1.0rc1.dist-info/RECORD,,