pointblank 0.9.5__tar.gz → 0.9.6__tar.gz

{pointblank-0.9.5 → pointblank-0.9.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pointblank
-Version: 0.9.5
+Version: 0.9.6
 Summary: Find out if your data is what you think it is.
 Author-email: Richard Iannone <riannone@me.com>
 License: MIT License
@@ -55,6 +55,8 @@ Requires-Dist: chatlas>=0.3.0; extra == "generate"
 Requires-Dist: anthropic[bedrock]>=0.45.2; extra == "generate"
 Requires-Dist: openai>=1.63.0; extra == "generate"
 Requires-Dist: shiny>=1.3.0; extra == "generate"
+Provides-Extra: bigquery
+Requires-Dist: ibis-framework[bigquery]>=9.5.0; extra == "bigquery"
 Provides-Extra: databricks
 Requires-Dist: ibis-framework[databricks]>=9.5.0; extra == "databricks"
 Provides-Extra: duckdb

{pointblank-0.9.5 → pointblank-0.9.6}/docs/_quarto.yml

@@ -178,6 +178,7 @@ quartodoc:
         - name: Validate.all_passed
         - name: Validate.assert_passing
         - name: Validate.assert_below_threshold
+        - name: Validate.above_threshold
         - name: Validate.n
         - name: Validate.n_passed
         - name: Validate.n_failed

pointblank-0.9.6/docs/blog/overhauled-user-guide/index.qmd

@@ -0,0 +1,154 @@
+---
+jupyter: python3
+html-table-processing: none
+title: Overhauling Pointblank's User Guide
+author: Rich Iannone and Michael Chow
+date: 2025-05-20
+freeze: true
+---
+
+```{python}
+#| echo: false
+#| output: false
+import pointblank as pb
+pb.config(report_incl_footer=False)
+```
+
+The Pointblank documentation just got a major upgrade! We've completely overhauled our
+[**User Guide**](../../user-guide/index.qmd). Our goal was to enable readers to start fast on
+validation and work through the many pieces needed in realistic situations.
+
+We realized that at the core of Pointblank is the validation plan. Its made up of rules, results,
+and steps.
+
+![](validation-three-things.png){width=80%}
+
+For example, the first row is a step that checks whether values in column 'a' are less than `10`.
+The `COLUMNS` and `VALUES` column contain the rules used to configure the step. The `PASS` column
+for the first row indicates that all 13 values in the column passed.
+
+In this post, we'll cover:
+
+- using a spiral sequence to balance introductory breadth with subsequent depth
+- improving examples with direct explanatory text
+- rounding out API coverage for cross-linking
+- surfacing advanced topics
+
+Let us walk you through the key improvements in our refreshed [**User Guide**](../../user-guide/index.qmd)!
+
+## Introduction: Embracing the Spiral Sequence
+
+We chose to use a spiral sequence for our [Introduction](../../user-guide/index.qmd) and
+*Validation Plan* section. The [Introduction](../../user-guide/index.qmd) quickly covers parts of
+validation plan, while each article of the *Validation Plan* section dives
+deeper into different aspects of defining validation rules.
+
+<div style="text-align: center;">
+
+![](breadth-and-depth.png){width=50%}
+
+</div>
+
+The introduction does a broad pass on the validation plan table diagram, identifying the core pieces
+of the output and then giving a quick overview of the code that produced it. This sets people up for
+the *Validation Plan* section in the guide, where each concept is discussed in depth.
+
+## Improving Examples
+
+Examples are everywhere in the [**User Guide**](../../user-guide/index.qmd). We've tightened up our
+approach to examples by:
+
+- presenting example code, output, or both, early in each section
+- showing the actual output you'll see in your environment
+- following up with explanatory text that guides attention to specific places in the output
+
+This approach makes learning more intuitive. Here's an excerpt that shows this in practice.
+
+![](explanation-of-visual.png){width=80%}
+
+The blue arrow marks the flow of reading and the red arrows map where we anticipate people will look
+from the text to the output. Focusing explicitly on where we think attention will go forces us to
+think carefully about exactly what readers will get from the output. The hope is that readers get to
+work more quickly on new concepts.
+
+## Rounding Out API Coverage
+
+Documentation has to balance jobs between a user guide and an API Reference:
+
+- User Guide: explains concepts that cut across functions (like common arguments across
+[validation methods](/user-guide/validation-methods.qmd))
+- API Reference: explains each individual function
+
+Importantly, user guides often link to the API reference so, as part of this work, we made sure that
+all individual API entries are well-documented and linked to from the guide.
+Here's an excerpt from the [**User Guide**](../../user-guide/index.qmd) that shows links marked:
+
+![](links-in-guide.png){width=80%}
+
+## Surfacing Advanced Topics
+
+There's a lot of potential slicing and dicing involved in validation, as well as work after
+validation (post interrogation) to make sense of the results. We added pages to the
+[**User Guide**](../../user-guide/index.qmd) for some core situations.
+In this section, I'll highlight two advanced topics we added pages for:
+
+- segmentation: splitting a column into groups, and validating each group
+- step reports: view failing cases (e.g., view rows of data that failed validation)
+
+These are marked in the [**User Guide**](../../user-guide/index.qmd) sidebar screenshot below:
+
+<div style="text-align: center;">
+
+![](advanced-in-sidebar.png){width=35%}
+
+</div>
+
+### Segmentation
+
+Here's a screenshot of a validation report with two validation steps, one for each segment (`"low"`
+and `"high"`) in the `f` column of the `small_table` dataset.
+
+![](segments.png){width=80%}
+
+Notice that segments split columns into groups and apply the same validation to each of the groups.
+Each group is given its own step.
+
+Each of the 20+ validation methods accept a `segment=` argument. The value of the
+[Segmentation article in the User Guide](../../user-guide/segmentation.qmd) is to describe this
+cross-cutting behavior in a single place.
+
+Compare the `segments=` parameter in the API Reference (e.g., look at
+[`col_vals_gt()`](../../reference/Validate.col_vals_gt.qmd)) and the
+[Segmentation article](../../user-guide/segmentation.qmd) to get a feel for how each location
+documents the segments feature.
+
+### Step Report
+
+Step reports display failing cases (e.g., rows) for a validation step, so you can dig deeper into
+validation failures. Here's a screenshot of a step report for some validation step 2:
+
+![](step-report.png){width=80%}
+
+Notice the arrow pointing to 'Step 2' in the title. Failing values are highlighted in red. Once we
+know we have failures, it's important to take action and discover *why* data is failing. Looking at
+failing cases in step reports often uncovers obvious causes behind failures.
+
+The `get_step_report()` entry is one of 50 in the API Reference. Here it is listed the API
+Reference, in the [Interrogation and Reporting section](../../reference/index.qmd#interrogation-and-reporting).
+Critically, it's only one of 20 entries in the [User Guide](../../user-guide/index.qmd), which
+emphasizes its importance in validation workflows.
+
+## Looking Forward
+
+The refreshed **User Guide** is just the beginning of our documentation improvements. We're
+committed to continuously enhancing our documentation to support your data validation needs.
+
+Michael Chow gave feedback on this **User Guide** in preparation for his
+[upcoming talk at SciPy 2025](https://cfp.scipy.org/scipy2025/talk/NRMNDX/).
+
+We'd love to hear your feedback on the new User Guide! Feel free to open an issue on our
+[GitHub repository](https://github.com/posit-dev/pointblank/issues) with suggestions, corrections,
+or requests for additional topics you'd like to see covered. You can also join our community
+discussions in the dedicated `#Documentation` channel on our
+[Discord server](https://discord.com/invite/YH7CybCNCQ), where you can share ideas, ask questions,
+and get help directly from the Pointblank team and other users.

{pointblank-0.9.5 → pointblank-0.9.6}/docs/user-guide/installation.qmd

@@ -80,6 +80,7 @@ pip install pointblank[duckdb]      # DuckDB
 pip install pointblank[postgres]    # PostgreSQL
 pip install pointblank[mysql]       # MySQL
 pip install pointblank[mssql]       # Microsoft SQL Server
+pip install pointblank[bigquery]    # BigQuery
 pip install pointblank[pyspark]     # Apache Spark
 pip install pointblank[databricks]  # Databricks
 pip install pointblank[snowflake]   # Snowflake
@@ -96,6 +97,7 @@ uv pip install pointblank[duckdb]      # DuckDB
 uv pip install pointblank[postgres]    # PostgreSQL
 uv pip install pointblank[mysql]       # MySQL
 uv pip install pointblank[mssql]       # Microsoft SQL Server
+uv pip install pointblank[bigquery]    # BigQuery
 uv pip install pointblank[pyspark]     # Apache Spark
 uv pip install pointblank[databricks]  # Databricks
 uv pip install pointblank[snowflake]   # Snowflake
@@ -112,6 +114,7 @@ conda install -c conda-forge pointblank-duckdb      # DuckDB
 conda install -c conda-forge pointblank-postgres    # PostgreSQL
 conda install -c conda-forge pointblank-mysql       # MySQL
 conda install -c conda-forge pointblank-mssql       # Microsoft SQL Server
+conda install -c conda-forge pointblank-bigquery    # BigQuery
 conda install -c conda-forge pointblank-pyspark     # Apache Spark
 conda install -c conda-forge pointblank-databricks  # Databricks
 conda install -c conda-forge pointblank-snowflake   # Snowflake

{pointblank-0.9.5 → pointblank-0.9.6}/pointblank/_constants.py

@@ -109,6 +109,7 @@ ROW_BASED_VALIDATION_TYPES = [
 ]
 
 IBIS_BACKENDS = [
+    "bigquery",
     "databricks",
     "duckdb",
     "memtable",
@@ -165,6 +166,7 @@ TABLE_TYPE_STYLES = {
     "parquet": {"background": "#3F9FF9", "text": "#FFFFFF", "label": "Parquet"},
     "memtable": {"background": "#2C3E50", "text": "#FFFFFF", "label": "Ibis memtable"},
     "mssql": {"background": "#E2E2E2", "text": "#222222", "label": "MSSQL"},
+    "bigquery": {"background": "#4285F4", "text": "#FFFFFF", "label": "BigQuery"},
     "pyspark": {"background": "#E66F21", "text": "#FFFFFF", "label": "Spark DataFrame"},
     "databricks": {"background": "#FF3621", "text": "#FFFFFF", "label": "Databricks"},
 }

{pointblank-0.9.5 → pointblank-0.9.6}/pointblank/_utils.py

@@ -514,6 +514,8 @@ def _get_api_text() -> str:
         "Validate.get_data_extracts",
         "Validate.all_passed",
         "Validate.assert_passing",
+        "Validate.assert_below_threshold",
+        "Validate.above_threshold",
         "Validate.n",
         "Validate.n_passed",
         "Validate.n_failed",

{pointblank-0.9.5 → pointblank-0.9.6}/pointblank/actions.py

@@ -216,7 +216,7 @@ def send_slack_notification(
             thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),
             actions=pb.Actions(critical=notify_slack),
         )
-        .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
+        .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}[0-9]{3}")
         .col_vals_gt(columns="item_revenue", value=0.05)
         .col_vals_gt(columns="session_duration", value=15)
         .interrogate()
@@ -248,7 +248,7 @@ def send_slack_notification(
             thresholds=pb.Thresholds(warning=0.05, error=0.10, critical=0.15),
             final_actions=pb.FinalActions(notify_slack),
         )
-        .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
+        .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}[0-9]{3}")
         .col_vals_gt(columns="item_revenue", value=0.05)
         .col_vals_gt(columns="session_duration", value=15)
         .interrogate()
@@ -316,7 +316,7 @@ def send_slack_notification(
             actions=pb.Actions(default=notify_slack),
             final_actions=pb.FinalActions(notify_slack),
         )
-        .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}\d{3}")
+        .col_vals_regex(columns="player_id", pattern=r"[A-Z]{12}[0-9]{3}")
         .col_vals_gt(columns="item_revenue", value=0.05)
         .col_vals_gt(columns="session_duration", value=15)
         .interrogate()

{pointblank-0.9.5 → pointblank-0.9.6}/pointblank/column.py

@@ -1007,7 +1007,7 @@ def matches(pattern: str, case_sensitive: bool = False) -> Matches:
     `[rev_01, rev_02, profit_01, profit_02, age]`
 
     and you want to validate columns that have two digits at the end of the name, you can use
-    `columns=matches(r"\d{2}$")`. This will select the `rev_01`, `rev_02`, `profit_01`, and
+    `columns=matches(r"[0-9]{2}$")`. This will select the `rev_01`, `rev_02`, `profit_01`, and
     `profit_02` columns.
 
     There will be a validation step created for every resolved column. Note that if there aren't any
@@ -1061,7 +1061,7 @@ def matches(pattern: str, case_sensitive: bool = False) -> Matches:
     [`col()`](`pointblank.col`) function, like this:
 
     ```python
-    col(matches(r"^\d{5}") & ends_with("_id"))
+    col(matches(r"^[0-9]{5}") & ends_with("_id"))
     ```
 
     There are four operators that can be used to compose column selectors:
@@ -1107,7 +1107,7 @@ def matches(pattern: str, case_sensitive: bool = False) -> Matches:
 
     validation = (
         pb.Validate(data=tbl)
-        .col_vals_regex(columns=pb.matches("id|identifier"), pattern=r"ID\d{4}")
+        .col_vals_regex(columns=pb.matches("id|identifier"), pattern=r"ID[0-9]{4}")
         .interrogate()
     )
 
@@ -1115,7 +1115,7 @@ def matches(pattern: str, case_sensitive: bool = False) -> Matches:
     ```
 
     From the results of the validation table we get two validation steps, one for `id_old` and one
-    for `new_identifier`. The values in both columns all match the pattern `"ID\d{4}"`.
+    for `new_identifier`. The values in both columns all match the pattern `"ID[0-9]{4}"`.
 
     We can also use the `matches()` function in combination with other column selectors (within
     [`col()`](`pointblank.col`)) to create more complex column selection criteria (i.e., to select