databuck-spark-sdk 0.5.1__tar.gz

databuck_spark_sdk-0.5.1/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md
+include RELEASING.md
+global-exclude *.jar

databuck_spark_sdk-0.5.1/PKG-INFO

@@ -0,0 +1,160 @@
+Metadata-Version: 2.4
+Name: databuck-spark-sdk
+Version: 0.5.1
+Summary: DataBuck data quality SDK for PySpark and Databricks
+Author: DataBuck
+Project-URL: Repository, https://github.com/FirstEigen-Labs/databuck_sdk
+Keywords: data quality,databricks,pyspark,data profiling
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Database
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: google-genai>=1.0.0
+
+# databuck-spark-sdk
+
+Python wrapper for the DataBuck Spark SDK, used from PySpark and Databricks.
+
+## Install
+
+```bash
+python -m pip install databuck-spark-sdk
+python -c "import databuck"
+```
+
+Importing `databuck` downloads `databuck-spark-sdk.jar` from the DataBuck S3
+URL to `databuck/jars/` in the installed Python package, with progress output.
+The downloaded path is set in `DATABUCK_SPARK_SDK_JAR` for the current Python
+process. A later import reuses the existing JAR.
+The JAR is not included in the wheel because it exceeds PyPI's
+default per-file upload limit. An internet connection and write access to the
+installed package directory are required for the first download.
+
+`python -m databuck` is an explicit download command that also prints the
+local path. Set `DATABUCK_SPARK_SDK_AUTO_DOWNLOAD=0` before import to defer the
+download when offline. `DataBuck.jar_path()` returns the path after download.
+Downloading the JAR does not install it as a Databricks compute library.
+
+To use an existing JAR or another writable destination, set
+`DATABUCK_SPARK_SDK_JAR` to that file path. If the download URL changes, set
+`DATABUCK_SPARK_SDK_JAR_URL` to the new HTTPS URL before importing or running
+`python -m databuck`. The download is checked against
+the published JAR's SHA-256 hash. If the JAR content changes, publish a new
+Python package version with its new hash, or set
+`DATABUCK_SPARK_SDK_JAR_SHA256` to the expected hash.
+
+For a Databricks notebook, install the Python distribution as a library and
+make the JAR available on the cluster. If the package directory is read-only,
+configure `DATABUCK_SPARK_SDK_JAR` to a writable driver path before importing.
+
+## Usage
+
+```python
+from pyspark.sql import SparkSession
+import os
+
+from databuck import DataBuck
+
+spark = (
+    SparkSession.builder
+    .config("spark.jars", DataBuck.jar_path())
+    .getOrCreate()
+)
+
+df = spark.read.csv("customers.csv", header=True)
+
+print(DataBuck.count(df))
+```
+
+## Discover and export rules for Databricks
+
+```python
+json_path = DataBuck.discover_and_export(
+    df,
+    "/Volumes/catalog/schema/volume/expectations.json",
+    context={
+        "business_context": (
+            "Missing customer email should be monitored. "
+            "Orders without an order ID must be discarded. "
+            "An unknown currency must stop publication."
+        ),
+        "gemini_api_key": dbutils.secrets.get(scope="databuck", key="gemini-api-key"),
+    },
+)
+print(json_path)
+```
+
+The single call runs profile discovery and context-aware BuckGPT discovery,
+then writes both sets of row expectations into one JSON file. Context-aware
+expectations are named `BuckGPT_Rule_001`, `BuckGPT_Rule_002`, etc. Their
+invalid-row SQL queries are converted to Lakeflow pass conditions. Only
+audited, passed BuckGPT rules in the required
+`SELECT * FROM {{DATAFRAME}} WHERE <invalid-row condition>` shape can be
+exported; an unsupported query fails the export.
+
+To export only automatic profiling rules, omit `context`:
+
+```python
+json_path = DataBuck.discover_and_export(
+    df, "/Volumes/catalog/schema/volume/expectations.json"
+)
+```
+
+Without context, BuckGPT is not called and all exported rules use `warn`.
+
+The JSON contains three dictionaries: `warn`, `drop`, and `fail`. Gemini
+classifies every exported expectation using the business context, DataFrame
+schema, and up to five sample rows. This sends those inputs to Gemini. If an
+LLM response is incomplete or invalid, export fails instead of assigning a
+destructive action.
+In a Lakeflow pipeline, use the dictionaries with `dp.expect_all`,
+`dp.expect_all_or_drop`, and `dp.expect_all_or_fail`, respectively.
+
+For example, in the Lakeflow pipeline source file:
+
+```python
+import json
+from pyspark import pipelines as dp
+
+with open("/Volumes/catalog/schema/volume/expectations.json", encoding="utf-8") as stream:
+    expectations = json.load(stream)
+
+@dp.table
+@dp.expect_all(expectations["warn"])
+@dp.expect_all_or_drop(expectations["drop"])
+@dp.expect_all_or_fail(expectations["fail"])
+def customers_checked():
+    return spark.read.table("catalog.schema.customers_source")
+```
+
+The pipeline must read a DataFrame with the columns used by the exported
+expectations. Regenerate the JSON when the source schema or business policy
+changes, then refresh the pipeline.
+
+```json
+{
+  "warn": {"not_null_customer_email": "`customer_email` IS NOT NULL"},
+  "drop": {"not_null_order_id": "`order_id` IS NOT NULL"},
+  "fail": {"valid_pattern_currency_code": "`currency_code` IS NULL OR CAST(`currency_code` AS STRING) RLIKE '(?:^[A-Z][A-Z][A-Z]$)'"}
+}
+```
+
+The example shows the file shape; the actual actions come from the supplied
+business context. `DataBuck.discover_rules(df)` and
+`DataBuck.discover(df, context)` are still available separately. The new
+`DataBuck.discover_and_export(...)` call combines them.
+
+Each action dictionary maps expectation names to Spark SQL pass conditions,
+such as `{"not_null_subscriber_id": "`subscriber_id` IS NOT NULL"}`. Null, pattern,
+and length rules are converted from their profiling metadata, including rules
+from older SDK JARs whose `expression` field is empty. Dataset-level rules
+and catalog rules without a row condition remain in `rules` but are omitted
+from the file. Null rules are exported only when their threshold is 0%.
+The export does not encode aggregate failure thresholds. Profiling pattern
+`A` maps to `[A-Z]`, and `#` maps to `[0-9]`.

databuck_spark_sdk-0.5.1/README.md

@@ -0,0 +1,141 @@
+# databuck-spark-sdk
+
+Python wrapper for the DataBuck Spark SDK, used from PySpark and Databricks.
+
+## Install
+
+```bash
+python -m pip install databuck-spark-sdk
+python -c "import databuck"
+```
+
+Importing `databuck` downloads `databuck-spark-sdk.jar` from the DataBuck S3
+URL to `databuck/jars/` in the installed Python package, with progress output.
+The downloaded path is set in `DATABUCK_SPARK_SDK_JAR` for the current Python
+process. A later import reuses the existing JAR.
+The JAR is not included in the wheel because it exceeds PyPI's
+default per-file upload limit. An internet connection and write access to the
+installed package directory are required for the first download.
+
+`python -m databuck` is an explicit download command that also prints the
+local path. Set `DATABUCK_SPARK_SDK_AUTO_DOWNLOAD=0` before import to defer the
+download when offline. `DataBuck.jar_path()` returns the path after download.
+Downloading the JAR does not install it as a Databricks compute library.
+
+To use an existing JAR or another writable destination, set
+`DATABUCK_SPARK_SDK_JAR` to that file path. If the download URL changes, set
+`DATABUCK_SPARK_SDK_JAR_URL` to the new HTTPS URL before importing or running
+`python -m databuck`. The download is checked against
+the published JAR's SHA-256 hash. If the JAR content changes, publish a new
+Python package version with its new hash, or set
+`DATABUCK_SPARK_SDK_JAR_SHA256` to the expected hash.
+
+For a Databricks notebook, install the Python distribution as a library and
+make the JAR available on the cluster. If the package directory is read-only,
+configure `DATABUCK_SPARK_SDK_JAR` to a writable driver path before importing.
+
+## Usage
+
+```python
+from pyspark.sql import SparkSession
+import os
+
+from databuck import DataBuck
+
+spark = (
+    SparkSession.builder
+    .config("spark.jars", DataBuck.jar_path())
+    .getOrCreate()
+)
+
+df = spark.read.csv("customers.csv", header=True)
+
+print(DataBuck.count(df))
+```
+
+## Discover and export rules for Databricks
+
+```python
+json_path = DataBuck.discover_and_export(
+    df,
+    "/Volumes/catalog/schema/volume/expectations.json",
+    context={
+        "business_context": (
+            "Missing customer email should be monitored. "
+            "Orders without an order ID must be discarded. "
+            "An unknown currency must stop publication."
+        ),
+        "gemini_api_key": dbutils.secrets.get(scope="databuck", key="gemini-api-key"),
+    },
+)
+print(json_path)
+```
+
+The single call runs profile discovery and context-aware BuckGPT discovery,
+then writes both sets of row expectations into one JSON file. Context-aware
+expectations are named `BuckGPT_Rule_001`, `BuckGPT_Rule_002`, etc. Their
+invalid-row SQL queries are converted to Lakeflow pass conditions. Only
+audited, passed BuckGPT rules in the required
+`SELECT * FROM {{DATAFRAME}} WHERE <invalid-row condition>` shape can be
+exported; an unsupported query fails the export.
+
+To export only automatic profiling rules, omit `context`:
+
+```python
+json_path = DataBuck.discover_and_export(
+    df, "/Volumes/catalog/schema/volume/expectations.json"
+)
+```
+
+Without context, BuckGPT is not called and all exported rules use `warn`.
+
+The JSON contains three dictionaries: `warn`, `drop`, and `fail`. Gemini
+classifies every exported expectation using the business context, DataFrame
+schema, and up to five sample rows. This sends those inputs to Gemini. If an
+LLM response is incomplete or invalid, export fails instead of assigning a
+destructive action.
+In a Lakeflow pipeline, use the dictionaries with `dp.expect_all`,
+`dp.expect_all_or_drop`, and `dp.expect_all_or_fail`, respectively.
+
+For example, in the Lakeflow pipeline source file:
+
+```python
+import json
+from pyspark import pipelines as dp
+
+with open("/Volumes/catalog/schema/volume/expectations.json", encoding="utf-8") as stream:
+    expectations = json.load(stream)
+
+@dp.table
+@dp.expect_all(expectations["warn"])
+@dp.expect_all_or_drop(expectations["drop"])
+@dp.expect_all_or_fail(expectations["fail"])
+def customers_checked():
+    return spark.read.table("catalog.schema.customers_source")
+```
+
+The pipeline must read a DataFrame with the columns used by the exported
+expectations. Regenerate the JSON when the source schema or business policy
+changes, then refresh the pipeline.
+
+```json
+{
+  "warn": {"not_null_customer_email": "`customer_email` IS NOT NULL"},
+  "drop": {"not_null_order_id": "`order_id` IS NOT NULL"},
+  "fail": {"valid_pattern_currency_code": "`currency_code` IS NULL OR CAST(`currency_code` AS STRING) RLIKE '(?:^[A-Z][A-Z][A-Z]$)'"}
+}
+```
+
+The example shows the file shape; the actual actions come from the supplied
+business context. `DataBuck.discover_rules(df)` and
+`DataBuck.discover(df, context)` are still available separately. The new
+`DataBuck.discover_and_export(...)` call combines them.
+
+Each action dictionary maps expectation names to Spark SQL pass conditions,
+such as `{"not_null_subscriber_id": "`subscriber_id` IS NOT NULL"}`. Null, pattern,
+and length rules are converted from their profiling metadata, including rules
+from older SDK JARs whose `expression` field is empty. Dataset-level rules
+and catalog rules without a row condition remain in `rules` but are omitted
+from the file. Null rules are exported only when their threshold is 0%.
+The export does not encode aggregate failure thresholds. Profiling pattern
+`A` maps to `[A-Z]`, and `#` maps to `[0-9]`.

databuck_spark_sdk-0.5.1/RELEASING.md

@@ -0,0 +1,42 @@
+# Publish with GitHub Actions trusted publishing
+
+The publishing workflow is `.github/workflows/publish.yml` at the repository
+root. It runs only when manually started on `main`. It tests and builds the
+Python package from `databuck-python/databuck-spark/`, checks the distributions,
+and publishes them to PyPI without a manually created API token.
+
+For the first release, add a pending trusted publisher on PyPI with:
+
+- PyPI project name: `databuck-spark-sdk`
+- Owner: `FirstEigen-Labs`
+- Repository name: `databuck_sdk`
+- Workflow name: `publish.yml`
+- Environment name: leave blank
+
+Commit and push the workflow to `main` before running it. Then in GitHub, open
+**Actions → Publish databuck-spark-sdk to PyPI → Run workflow**, select `main`,
+and start the run. Check its result and the new release on PyPI.
+
+Each PyPI release needs a new version in `pyproject.toml`; release files for an
+existing version cannot be overwritten. Review the version and source changes
+before starting the workflow.
+
+## Optional local build check
+
+Run these commands from `databuck-spark/` after setting the intended version
+in `pyproject.toml`:
+
+```bash
+python -m pip install --upgrade build twine
+python -m build --outdir dist-release-NEW_VERSION
+python -m twine check dist-release-NEW_VERSION/*
+```
+
+Inspect both generated files (`.whl` and `.tar.gz`) before upload. The wheel
+must contain the `databuck` package and its metadata. The JAR is downloaded
+when `databuck` is imported or by `python -m databuck` after installation. Check that its
+public S3 URL works without credentials before publishing. Do not stage the
+large JAR in the Python distribution.
+
+Use a fresh output directory for each build. On Windows, rebuilding into a
+directory that already contains the same archive can fail with `WinError 5`.

databuck_spark_sdk-0.5.1/databuck/__init__.py

@@ -0,0 +1,8 @@
+import os
+
+from .sdk import DataBuck
+
+if os.environ.get("DATABUCK_SPARK_SDK_AUTO_DOWNLOAD", "1").lower() not in ("0", "false", "no"):
+    os.environ["DATABUCK_SPARK_SDK_JAR"] = DataBuck.download_jar()
+
+__all__ = ["DataBuck"]

databuck_spark_sdk-0.5.1/databuck/__main__.py

@@ -0,0 +1,7 @@
+"""Download the DataBuck Spark SDK JAR after installing the Python package."""
+
+from .sdk import DataBuck
+
+
+if __name__ == "__main__":
+    print(DataBuck.download_jar())