pyspark-testframework 2.2.3__tar.gz → 2.3.0__tar.gz

{pyspark_testframework-2.2.3 → pyspark_testframework-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: pyspark-testframework
-Version: 2.2.3
+Version: 2.3.0
 Summary: Testframework for PySpark DataFrames
 Author-email: Woonstad Rotterdam <info@woonstadrotterdam.nl>, Tomer Gabay <tomer.gabay@woonstadrotterdam.nl>, Vincent van der Meij <vincent.van.der.meij@woonstadrotterdam.nl>, Tiddo Loos <tiddo.loos@woonstadrotterdam.nl>, Ben Verhees <ben.verhees@woonstadrotterdam.nl>
 License: MIT License
@@ -54,18 +54,6 @@ Requires-Dist: ruff==0.4.8; extra == "dev"
 
 # pyspark-testframework
 
-⏳ **Work in progress**
-
-![](https://progress-bar.dev/100/?title=RegexTest&width=120)  
-![](https://progress-bar.dev/100/?title=IsIntegerString&width=83)  
-![](https://progress-bar.dev/100/?title=ValidNumericRange&width=72)  
-![](https://progress-bar.dev/100/?title=ValidCategory&width=95)  
-![](https://progress-bar.dev/100/?title=CorrectValue&width=102)  
-![](https://progress-bar.dev/100/?title=ValidNumericStringRange&width=36)  
-![](https://progress-bar.dev/50/?title=ValidEmail&width=113)  
-![](https://progress-bar.dev/0/?title=ContainsValue&width=95)  
-![](<https://progress-bar.dev/0/?title=(...)&width=145>)
-
 The goal of the `pyspark-testframework` is to provide a simple way to create tests for PySpark DataFrames. The test results are returned in DataFrame format as well.
 
 # Tutorial
@@ -87,7 +75,7 @@ spark = SparkSession.builder.appName("PySparkTestFrameworkTutorial").getOrCreate
 # Define the schema
 schema = StructType(
     [
-        StructField("primary_key", IntegerType(), True),
+        StructField("id", IntegerType(), True),
         StructField("street", StringType(), True),
         StructField("house_number", IntegerType(), True),
     ]
@@ -107,15 +95,15 @@ df = spark.createDataFrame(data, schema)
 df.show(truncate=False)
 ```
 
-    +-----------+--------------------+------------+
-    |primary_key|street              |house_number|
-    +-----------+--------------------+------------+
-    |1          |Rochussenstraat     |27          |
-    |2          |Coolsingel          |31          |
-    |3          |%Witte de Withstraat|27          |
-    |4          |Lijnbaan            |-3          |
-    |5          |null                |13          |
-    +-----------+--------------------+------------+
+    +---+--------------------+------------+
+    |id |street              |house_number|
+    +---+--------------------+------------+
+    |1  |Rochussenstraat     |27          |
+    |2  |Coolsingel          |31          |
+    |3  |%Witte de Withstraat|27          |
+    |4  |Lijnbaan            |-3          |
+    |5  |null                |13          |
+    +---+--------------------+------------+
 
 **Import and initialize the `DataFrameTester`**
 
@@ -126,7 +114,7 @@ from testframework.dataquality import DataFrameTester
 ```python
 df_tester = DataFrameTester(
     df=df,
-    primary_key="primary_key",
+    primary_key="id",
     spark=spark,
 )
 ```
@@ -140,32 +128,32 @@ from testframework.dataquality.tests import ValidNumericRange, RegexTest
 **Initialize the `RegexTest` to test for valid street names**
 
 ```python
-valid_street_name = RegexTest(
-    name="ValidStreetName",
+valid_street_format = RegexTest(
+    name="ValidStreetFormat",
     pattern=r"^[A-Z][a-zéèáàëï]*([ -][A-Z]?[a-zéèáàëï]*)*$",
 )
 ```
 
-**Run `valid_street_name` on the _street_ column using the `.test()` method of `DataFrameTester`.**
+**Run `valid_street_format` on the _street_ column using the `.test()` method of `DataFrameTester`.**
 
 ```python
 df_tester.test(
     col="street",
-    test=valid_street_name,
-    nullable=False,  # nullable, hence null values are converted to True
-    description="street contains valid Dutch street name.",
+    test=valid_street_format,
+    nullable=False,  # nullable is False, hence null values are converted to False
+    description="Street is in valid Dutch street format.",
 ).show(truncate=False)
 ```
 
-    +-----------+--------------------+-----------------------+
-    |primary_key|street              |street__ValidStreetName|
-    +-----------+--------------------+-----------------------+
-    |1          |Rochussenstraat     |true                   |
-    |2          |Coolsingel          |true                   |
-    |3          |%Witte de Withstraat|false                  |
-    |4          |Lijnbaan            |true                   |
-    |5          |null                |false                  |
-    +-----------+--------------------+-----------------------+
+    +---+--------------------+-------------------------+
+    |id |street              |street__ValidStreetFormat|
+    +---+--------------------+-------------------------+
+    |1  |Rochussenstraat     |true                     |
+    |2  |Coolsingel          |true                     |
+    |3  |%Witte de Withstraat|false                    |
+    |4  |Lijnbaan            |true                     |
+    |5  |null                |false                    |
+    +---+--------------------+-------------------------+
 
 **Run the `IntegerString` test on the _number_ column**
 
@@ -175,19 +163,19 @@ By setting the `return_failed_rows` parameter to `True`, we can get only the row
 df_tester.test(
     col="house_number",
     test=ValidNumericRange(
-        min_value=0,
+        min_value=1,
     ),
-    nullable=True,  # nullable, hence null values are converted to True
-    # description is optional, let's not define it for illustration purposes
+    nullable=False,
+    # description="House number is in a valid format" # optional, let's not define it for illustration purposes
     return_failed_rows=True,  # only return the failed rows
 ).show()
 ```
 
-    +-----------+------------+-------------------------------+
-    |primary_key|house_number|house_number__ValidNumericRange|
-    +-----------+------------+-------------------------------+
-    |          4|          -3|                          false|
-    +-----------+------------+-------------------------------+
+    +---+------------+-------------------------------+
+    | id|house_number|house_number__ValidNumericRange|
+    +---+------------+-------------------------------+
+    |  4|          -3|                          false|
+    +---+------------+-------------------------------+
 
 **Let's take a look at the test results of the DataFrame using the `.results` attribute.**
 
@@ -195,15 +183,15 @@ df_tester.test(
 df_tester.results.show(truncate=False)
 ```
 
-    +-----------+-----------------------+-------------------------------+
-    |primary_key|street__ValidStreetName|house_number__ValidNumericRange|
-    +-----------+-----------------------+-------------------------------+
-    |1          |true                   |true                           |
-    |2          |true                   |true                           |
-    |3          |false                  |true                           |
-    |4          |true                   |false                          |
-    |5          |false                  |true                           |
-    +-----------+-----------------------+-------------------------------+
+    +---+-------------------------+-------------------------------+
+    |id |street__ValidStreetFormat|house_number__ValidNumericRange|
+    +---+-------------------------+-------------------------------+
+    |1  |true                     |true                           |
+    |2  |true                     |true                           |
+    |3  |false                    |true                           |
+    |4  |true                     |false                          |
+    |5  |false                    |true                           |
+    +---+-------------------------+-------------------------------+
 
 **We can use `.descriptions` or `.descriptions_df` to get the descriptions of the tests.**
 
@@ -215,8 +203,8 @@ For example to create reports for the business with more detailed information th
 df_tester.descriptions
 ```
 
-    {'street__ValidStreetName': 'street contains valid Dutch street name.',
-     'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=0.0, max_value=inf)'}
+    {'street__ValidStreetFormat': 'Street is in valid Dutch street format.',
+     'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=1.0, max_value=inf)'}
 
 ```python
 df_tester.description_df.show(truncate=False)
@@ -225,8 +213,8 @@ df_tester.description_df.show(truncate=False)
     +-------------------------------+-------------------------------------------------------------+
     |test                           |description                                                  |
     +-------------------------------+-------------------------------------------------------------+
-    |street__ValidStreetName        |street contains valid Dutch street name.                     |
-    |house_number__ValidNumericRange|house_number__ValidNumericRange(min_value=0.0, max_value=inf)|
+    |street__ValidStreetFormat      |Street is in valid Dutch street format.                      |
+    |house_number__ValidNumericRange|house_number__ValidNumericRange(min_value=1.0, max_value=inf)|
     +-------------------------------+-------------------------------------------------------------+
 
 ### Custom tests
@@ -237,18 +225,18 @@ Let's do this using a custom test which should tests that every house has a bath
 
 ```python
 rooms = [
-    (1, "living room"),
-    (1, "bath room"),
-    (1, "kitchen"),
-    (1, "bed room"),
-    (2, "living room"),
-    (2, "bed room"),
-    (2, "kitchen"),
+    (1,1, "living room"),
+    (2,1, "bathroom"),
+    (3,1, "kitchen"),
+    (4,1, "bed room"),
+    (5,2, "living room"),
+    (6,2, "bed room"),
+    (7,2, "kitchen"),
 ]
 
 schema_rooms = StructType(
-    [
-        StructField("primary_key", IntegerType(), True),
+    [   StructField("id", IntegerType(), True),
+        StructField("house_id", IntegerType(), True),
         StructField("room", StringType(), True),
     ]
 )
@@ -258,38 +246,38 @@ room_df = spark.createDataFrame(rooms, schema=schema_rooms)
 room_df.show(truncate=False)
 ```
 
-    +-----------+-----------+
-    |primary_key|room       |
-    +-----------+-----------+
-    |1          |living room|
-    |1          |bath room  |
-    |1          |kitchen    |
-    |1          |bed room   |
-    |2          |living room|
-    |2          |bed room   |
-    |2          |kitchen    |
-    +-----------+-----------+
+    +---+--------+-----------+
+    |id |house_id|room       |
+    +---+--------+-----------+
+    |1  |1       |living room|
+    |2  |1       |bathroom   |
+    |3  |1       |kitchen    |
+    |4  |1       |bed room   |
+    |5  |2       |living room|
+    |6  |2       |bed room   |
+    |7  |2       |kitchen    |
+    +---+--------+-----------+
 
 To create a custom test, we should create a pyspark DataFrame which contains the same primary_key column as the DataFrame to be tested using the `DataFrameTester`.
 
 Let's create a boolean column that indicates whether the house has a bath room or not.
 
 ```python
-house_has_bath_room = room_df.groupBy("primary_key").agg(
-    F.max(F.when(F.col("room") == "bath room", True).otherwise(False)).alias(
-        "has_bath_room"
+house_has_bathroom = room_df.groupBy("house_id").agg(
+    F.max(F.when(F.col("room") == "bathroom", True).otherwise(False)).alias(
+        "has_bathroom"
     )
 )
 
-house_has_bath_room.show(truncate=False)
+house_has_bathroom.show(truncate=False)
 ```
 
-    +-----------+-------------+
-    |primary_key|has_bath_room|
-    +-----------+-------------+
-    |1          |true         |
-    |2          |false        |
-    +-----------+-------------+
+    +--------+------------+
+    |house_id|has_bathroom|
+    +--------+------------+
+    |1       |true        |
+    |2       |false       |
+    +--------+------------+
 
 **We can add this 'custom test' to the `DataFrameTester` using `add_custom_test_result`.**
 
@@ -297,22 +285,22 @@ In the background, all kinds of data validation checks are done by `DataFrameTes
 
 ```python
 df_tester.add_custom_test_result(
-    result=house_has_bath_room,
-    name="has_bath_room",
-    description="House has a bath room",
+    result=house_has_bathroom.withColumnRenamed("house_id", "id"),
+    name="has_bathroom",
+    description="House has a bathroom",
     # fillna_value=0, # optional; by default null.
 ).show(truncate=False)
 ```
 
-    +-----------+-------------+
-    |primary_key|has_bath_room|
-    +-----------+-------------+
-    |1          |true         |
-    |2          |false        |
-    |3          |null         |
-    |4          |null         |
-    |5          |null         |
-    +-----------+-------------+
+    +---+------------+
+    |id |has_bathroom|
+    +---+------------+
+    |1  |true        |
+    |2  |false       |
+    |3  |null        |
+    |4  |null        |
+    |5  |null        |
+    +---+------------+
 
 **Despite that the data whether a house has a bath room is not available in the house DataFrame; we can still add the custom test to the `DataFrameTester` object.**
 
@@ -320,23 +308,23 @@ df_tester.add_custom_test_result(
 df_tester.results.show(truncate=False)
 ```
 
-    +-----------+-----------------------+-------------------------------+-------------+
-    |primary_key|street__ValidStreetName|house_number__ValidNumericRange|has_bath_room|
-    +-----------+-----------------------+-------------------------------+-------------+
-    |1          |true                   |true                           |true         |
-    |2          |true                   |true                           |false        |
-    |3          |false                  |true                           |null         |
-    |4          |true                   |false                          |null         |
-    |5          |false                  |true                           |null         |
-    +-----------+-----------------------+-------------------------------+-------------+
+    +---+-------------------------+-------------------------------+------------+
+    |id |street__ValidStreetFormat|house_number__ValidNumericRange|has_bathroom|
+    +---+-------------------------+-------------------------------+------------+
+    |1  |true                     |true                           |true        |
+    |2  |true                     |true                           |false       |
+    |3  |false                    |true                           |null        |
+    |4  |true                     |false                          |null        |
+    |5  |false                    |true                           |null        |
+    +---+-------------------------+-------------------------------+------------+
 
 ```python
 df_tester.descriptions
 ```
 
-    {'street__ValidStreetName': 'street contains valid Dutch street name.',
-     'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=0.0, max_value=inf)',
-     'has_bath_room': 'House has a bath room'}
+    {'street__ValidStreetFormat': 'Street is in valid Dutch street format.',
+     'house_number__ValidNumericRange': 'house_number__ValidNumericRange(min_value=1.0, max_value=inf)',
+     'has_bathroom': 'House has a bathroom'}
 
 **We can also get a summary of the test results using the `.summary` attribute.**
 
@@ -347,9 +335,9 @@ df_tester.summary.show(truncate=False)
     +-------------------------------+-------------------------------------------------------------+-------+--------+-----------------+--------+-----------------+
     |test                           |description                                                  |n_tests|n_passed|percentage_passed|n_failed|percentage_failed|
     +-------------------------------+-------------------------------------------------------------+-------+--------+-----------------+--------+-----------------+
-    |street__ValidStreetName        |street contains valid Dutch street name.                     |5      |3       |60.0             |2       |40.0             |
-    |house_number__ValidNumericRange|house_number__ValidNumericRange(min_value=0.0, max_value=inf)|5      |4       |80.0             |1       |20.0             |
-    |has_bath_room                  |House has a bath room                                        |2      |1       |50.0             |1       |50.0             |
+    |street__ValidStreetFormat      |Street is in valid Dutch street format.                      |5      |3       |60.0             |2       |40.0             |
+    |house_number__ValidNumericRange|house_number__ValidNumericRange(min_value=1.0, max_value=inf)|5      |4       |80.0             |1       |20.0             |
+    |has_bathroom                   |House has a bathroom                                         |2      |1       |50.0             |1       |50.0             |
     +-------------------------------+-------------------------------------------------------------+-------+--------+-----------------+--------+-----------------+
 
 **If you want to see all rows that failed any of the tests, you can use the `.failed_tests` attribute.**
@@ -358,14 +346,14 @@ df_tester.summary.show(truncate=False)
 df_tester.failed_tests.show(truncate=False)
 ```
 
-    +-----------+-----------------------+-------------------------------+-------------+
-    |primary_key|street__ValidStreetName|house_number__ValidNumericRange|has_bath_room|
-    +-----------+-----------------------+-------------------------------+-------------+
-    |2          |true                   |true                           |false        |
-    |3          |false                  |true                           |null         |
-    |4          |true                   |false                          |null         |
-    |5          |false                  |true                           |null         |
-    +-----------+-----------------------+-------------------------------+-------------+
+    +---+-------------------------+-------------------------------+------------+
+    |id |street__ValidStreetFormat|house_number__ValidNumericRange|has_bathroom|
+    +---+-------------------------+-------------------------------+------------+
+    |2  |true                     |true                           |false       |
+    |3  |false                    |true                           |null        |
+    |4  |true                     |false                          |null        |
+    |5  |false                    |true                           |null        |
+    +---+-------------------------+-------------------------------+------------+
 
 **Of course, you can also see all rows that passed all tests using the `.passed_tests` attribute.**
 
@@ -373,8 +361,8 @@ df_tester.failed_tests.show(truncate=False)
 df_tester.passed_tests.show(truncate=False)
 ```
 
-    +-----------+-----------------------+-------------------------------+-------------+
-    |primary_key|street__ValidStreetName|house_number__ValidNumericRange|has_bath_room|
-    +-----------+-----------------------+-------------------------------+-------------+
-    |1          |true                   |true                           |true         |
-    +-----------+-----------------------+-------------------------------+-------------+
+    +---+-------------------------+-------------------------------+------------+
+    |id |street__ValidStreetFormat|house_number__ValidNumericRange|has_bathroom|
+    +---+-------------------------+-------------------------------+------------+
+    |1  |true                     |true                           |true        |
+    +---+-------------------------+-------------------------------+------------+