dataowl 0.1.0__tar.gz → 0.2.0__tar.gz

dataowl-0.2.0/CHANGELOG.md

@@ -0,0 +1,62 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+## [Unreleased]
+
+## [0.2.0] - 2026-10-08
+
+Column analysis (`analyze`) and an excerpt of the Delta history in `inspect`.
+
+### Added
+- `dataowl.analyze(table, *, key=None, timestamp=None, compare=None, days=30, spark=None)`
+  returning a `ColumnAnalysis` with `to_dict()` and `show(show_days=False)`. Only the
+  columns given by the user are analyzed, and at least one of `key`, `timestamp` and
+  `compare` is required.
+- Key analysis (`key`): rows, rows with null in the key, distinct keys, keys occurring more
+  than once and their rows, median and maximum rows per key, and the number of keys with 1,
+  2–10, 11–100 and more than 100 rows. Single columns, composite keys and lists of both.
+- Declared primary key from `information_schema.table_constraints` and
+  `information_schema.key_column_usage`, collected when `key` is given. Shown as
+  informational and not enforced.
+- Time column analysis (`timestamp`) for `TIMESTAMP`, `TIMESTAMP_NTZ` and `DATE`: min, max,
+  nulls and their share, values after the current time, distinct dates, gaps in whole days
+  between distinct dates, and rows per day for the last `days` whole days without today.
+- Comparison of two time columns (`compare`): rows where the second column is after, equal
+  to or before the first, and rows where either is null.
+- `show(show_days=True)` prints one line per day in the rows-per-day window.
+- History excerpt in `inspect` from `DESCRIBE HISTORY`: new `Overview` fields
+  `history_first_commit`, `history_last_commit`, `history_num_commits` and
+  `history_operations`. `WRITE` with mode `Overwrite` is counted as `WRITE (overwrite)`.
+  Not run for views.
+
+### Changed
+- The `inspect` output has a HISTORY block between the summary and the schema, with the
+  observation window of the history.
+
+## [0.1.0] - 2026-10-08
+
+First release: overview facts for a table (`inspect`).
+
+### Added
+- `dataowl.inspect(table, *, spark=None, count_views=False)` returning an `Overview` with
+  `to_dict()` and `show()`.
+- Facts from `information_schema.tables`: object type (with the raw `table_type`), format,
+  owner, comment and created time.
+- Facts from `DESCRIBE DETAIL`: size, number of files, average file size, last data
+  modification, partition and clustering columns. Not run for views.
+- Schema from `information_schema.columns`, and the field count including nested fields
+  from the Spark schema.
+- Exact row count with `COUNT(*)`. Skipped for views, materialized views, foreign tables and
+  objects whose type cannot be read, unless `count_views=True`.
+- Change Data Feed, log retention and deleted file retention from `SHOW TBLPROPERTIES`. Not
+  run for views.
+- Every fact carries its source (`metadata`, `exact`, `derived`). Facts that cannot be
+  collected are marked unavailable with a reason; only a missing table raises
+  (`TableNotFoundError`).
+- Terminal rendering of the overview.
+
+### Known deviations
+- `information_schema.columns.ordinal_position` is observed 0-based in Databricks, although
+  the documentation says it is numbered from 1. `ColumnInfo.position` keeps the raw value;
+  the `#` column in the output is a running number from 1.

dataowl-0.2.0/PKG-INFO

@@ -0,0 +1,345 @@
+Metadata-Version: 2.5
+Name: dataowl
+Version: 0.2.0
+Summary: Facts about Databricks data products for building dbt staging models.
+Project-URL: Homepage, https://github.com/alexD1990/dataowl
+Project-URL: Repository, https://github.com/alexD1990/dataowl
+Project-URL: Changelog, https://github.com/alexD1990/dataowl/blob/master/CHANGELOG.md
+Project-URL: Issues, https://github.com/alexD1990/dataowl/issues
+Author: Alexandro Dronnen
+License-Expression: MIT
+License-File: LICENSE
+Keywords: data-engineering,databricks,dbt,delta-lake,unity-catalog
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: mypy==2.3.1; extra == 'dev'
+Requires-Dist: pyspark>=3.4; extra == 'dev'
+Requires-Dist: pytest; extra == 'dev'
+Requires-Dist: ruff==0.16.9; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# dataowl
+
+dataowl reports facts about a table in Databricks (Unity Catalog / Delta Lake): size, files,
+rows, schema, table properties and an excerpt of the Delta history, and facts about the
+columns you choose: grain and uniqueness of keys, time span and cadence of time columns, and
+how two time columns compare. It is meant for data engineers who build a dbt staging model
+and want the facts without writing exploratory SQL. It reports what exists and never gives
+recommendations.
+
+## Requirements
+
+- Unity Catalog.
+- Databricks Runtime 14.3 LTS or above, or serverless compute.
+  Verified on serverless (Spark Connect).
+- PySpark is provided by Databricks. dataowl has no other runtime dependencies.
+
+## Installation
+
+In a Databricks notebook:
+
+```python
+%pip install dataowl==0.2.0
+```
+
+## Usage
+
+dataowl works in two steps: `inspect` gives an overview of the table, and `analyze` gives
+facts about columns you choose.
+
+### inspect
+
+```python
+import dataowl
+
+dataowl.inspect("catalog.schema.table").show()
+```
+
+As a JSON-serializable dict, where every fact has a value, a source and a reason when it
+is unavailable:
+
+```python
+overview = dataowl.inspect("catalog.schema.table")
+overview.to_dict()
+```
+
+Row counts are skipped for views, materialized views and foreign tables unless you ask for
+them:
+
+```python
+dataowl.inspect("catalog.schema.some_view", count_views=True).show()
+```
+
+Names with characters other than letters, digits and underscore are quoted with backticks:
+`` dataowl.inspect("`my-catalog`.schema.table") ``.
+
+The overview includes an excerpt of the Delta history from `DESCRIBE HISTORY`: the oldest and
+newest commit, the number of commits and the number of commits per operation. The header of
+the HISTORY block always shows the observation window (oldest – newest commit), because an
+operation that is missing is only missing for the period the history covers:
+
+- The history is limited by `delta.logRetentionDuration` (see **Log retention**). Older
+  commits are not included.
+- `WRITE (overwrite)` is a `WRITE` commit with mode `Overwrite`. It also includes partial
+  overwrites, for example with `replaceWhere`.
+- `CREATE OR REPLACE TABLE AS SELECT`, which dbt uses for the `table` materialization, is
+  shown under its own operation name, not as `WRITE (overwrite)`.
+
+### analyze
+
+```python
+analysis = dataowl.analyze(
+    "catalog.schema.table",
+    key=["customer_id", ("customer_id", "order_id")],
+    timestamp="updated_at",
+    compare=("created_at", "updated_at"),
+)
+analysis.show()
+```
+
+Full signature:
+
+```python
+analyze(
+    table: str,
+    *,
+    key: str | tuple[str, ...] | list[str | tuple[str, ...]] | None = None,
+    timestamp: str | list[str] | None = None,
+    compare: tuple[str, str] | None = None,
+    days: int = 30,
+    spark: SparkSession | None = None,
+) -> ColumnAnalysis
+```
+
+At least one of `key`, `timestamp` and `compare` must be given. Only the columns you give are
+analyzed; dataowl never chooses or suggests columns. Each key, each timestamp column and the
+comparison run their own queries against the table's data, so every column or combination
+you add costs compute. Column names are matched case insensitively.
+
+- **key** analyzes grain and uniqueness. A string is one column, a tuple is one composite
+  key, and a list holds several keys of either kind, each analyzed separately:
+  `key=["customer_id", ("customer_id", "order_id")]` analyzes `customer_id` on its own and
+  the combination of `customer_id` and `order_id`. MAP columns cannot be used.
+- **Primary key.** When `key` is given, dataowl also reads the declared primary key from
+  `information_schema`. A primary key in Databricks is informational and not enforced. In
+  `ColumnAnalysis.primary_key`, `None` means it was not collected (no `key`), a fact with
+  value `None` means no primary key is declared, and an unavailable fact has a reason.
+- **timestamp** analyzes one column or a list of columns of type `TIMESTAMP`,
+  `TIMESTAMP_NTZ` or `DATE`, each separately: min, max, nulls, values after the current
+  time, distinct dates, the gaps in whole days between distinct dates, and rows per day.
+  Other types, including dates stored as `STRING`, raise `ValueError`.
+- **days** (default 30) sets the window for rows per day: the last `days` whole days, not
+  including today, in the session time zone. Days without rows count as 0.
+- **compare** counts the rows where the second column is after, equal to or before the
+  first, and the rows where at least one of them is null. Columns of different types are
+  compared with Spark's implicit type conversion: a `DATE` is compared as that date at
+  00:00:00, so a `DATE` on the same day as a `TIMESTAMP` later than midnight counts as
+  before it.
+
+```python
+analysis.show(show_days=True)  # also prints one line per day in the window
+analysis.to_dict()  # JSON-serializable, like Overview.to_dict()
+```
+
+`analyze` raises:
+
+- `ValueError` for an invalid table name, when `key`, `timestamp` and `compare` are all
+  `None`, for invalid `days`, and for an unknown or unsupported column. All input is
+  validated before any query against the table's data.
+- `RuntimeError` when no SparkSession is available, and when the schema cannot be read from
+  `information_schema.columns`, since the columns cannot then be validated.
+- `TableNotFoundError` when the table does not exist or is not accessible.
+
+Every other failure makes the affected facts `n/a (<reason>)`.
+
+## Example output
+
+Output for Databricks' public sample data (`samples.nyctaxi.trips`).
+
+```python
+dataowl.inspect("samples.nyctaxi.trips").show()
+```
+
+```text
+samples.nyctaxi.trips   (MANAGED, DELTA)
+
+Size:                    354.2 KB
+Files:                   1  (avg 354.2 KB)
+Rows:                    21 932
+Columns:                 6  (6 incl. nested)
+Partitioned by:          –
+Clustered by:            –
+Change Data Feed:        true
+Log retention:           not set (default)
+Deleted file retention:  not set (default)
+Created:                 2025-09-30 11:28
+Last modified:           2026-09-14 15:07
+Owner:                   System user
+Comment:                 –
+
+HISTORY  (2025-09-09 15:05 – 2026-09-14 15:07, 223 commits)
+  CREATE OR REPLACE TABLE AS SELECT:  212
+  SET TBLPROPERTIES:                  11
+
+SCHEMA
+  #  name                   type       nullable  comment
+  1  tpep_pickup_datetime   timestamp  yes
+  2  tpep_dropoff_datetime  timestamp  yes
+  3  trip_distance          double     yes
+  4  fare_amount            double     yes
+  5  pickup_zip             int        yes
+  6  dropoff_zip            int        yes
+```
+
+```python
+dataowl.analyze(
+    "samples.nyctaxi.trips",
+    key=["pickup_zip", ("pickup_zip", "dropoff_zip")],
+    timestamp="tpep_pickup_datetime",
+    compare=("tpep_pickup_datetime", "tpep_dropoff_datetime"),
+).show()
+```
+
+```text
+samples.nyctaxi.trips
+
+PRIMARY KEY  none declared
+
+KEY  pickup_zip
+  Rows:                    21 932
+  Rows with null in key:   0
+  Distinct keys:           128
+  Keys occurring >1 time:  100
+  Rows in those keys:      21 904
+  Rows per key:            median 19 · max 1 227
+    1 row:                 28
+    2–10 rows:             27
+    11–100 rows:           31
+    >100 rows:             42
+
+KEY  pickup_zip + dropoff_zip
+  Rows:                    21 932
+  Rows with null in key:   0
+  Distinct keys:           3 369
+  Keys occurring >1 time:  2 060
+  Rows in those keys:      20 623
+  Rows per key:            median 2 · max 143
+    1 row:                 1 309
+    2–10 rows:             1 533
+    11–100 rows:           518
+    >100 rows:             9
+
+TIMESTAMP  tpep_pickup_datetime
+  Min:             2016-01-01 00:04
+  Max:             2016-02-29 23:51
+  Nulls:           0 (0.00 %)
+  Future values:   0
+  Distinct dates:  60
+  Days between distinct dates:
+    min 1 · median 1 · mean 1 · max 1
+  Rows per day, last 30 complete days (2026-09-08 – 2026-10-07):
+    median 0 · mean 0 · min 0 · max 0
+  Note: gaps are measured in whole days; cadence below one day is not visible.
+  Note: counts reflect current values of the column, not historical changes.
+
+COMPARE  tpep_pickup_datetime → tpep_dropoff_datetime
+  tpep_dropoff_datetime > tpep_pickup_datetime:  21 931
+  tpep_dropoff_datetime = tpep_pickup_datetime:  1
+  tpep_dropoff_datetime < tpep_pickup_datetime:  0
+  Either is null:                                0
+```
+
+The data in `samples.nyctaxi.trips` is from January and February 2016, so every day in the
+rows-per-day window has 0 rows.
+
+## Principles
+
+- **Read-only.** Only `SELECT`, `DESCRIBE` and `SHOW`. Nothing is written, optimized or
+  analyzed.
+- **Facts only.** No recommendations or assessments.
+- **No row values leave Spark.** Only aggregates and metadata are collected. Min and max of
+  time columns are the only exception.
+- **You choose the columns.** `analyze` only analyzes the columns you give.
+- **Unavailable facts are reported with a reason.** Missing permissions or an unsupported
+  object type make the affected facts `n/a (<reason>)`; the rest of the report is still
+  produced. Only a table that does not exist or is not accessible raises
+  `TableNotFoundError`.
+
+## Facts per object type
+
+| Facts | Source | MANAGED, EXTERNAL | STREAMING_TABLE, MATERIALIZED_VIEW | FOREIGN | VIEW | UNKNOWN or n/a type |
+|---|---|---|---|---|---|---|
+| Object type, format, owner, comment, created | `information_schema.tables` | yes | yes | yes | yes | yes (n/a if the query fails) |
+| Size, files, avg file size, last modified, partitioning, clustering | `DESCRIBE DETAIL` | yes | attempted¹ | attempted¹ | no | attempted¹ |
+| Columns and schema | `information_schema.columns` | yes | yes | yes | yes | yes |
+| Field count incl. nested | Spark schema | yes | yes | yes | yes | yes |
+| Change Data Feed, log and deleted file retention | `SHOW TBLPROPERTIES` | yes | attempted¹ | attempted¹ | no | attempted¹ |
+| History excerpt | `DESCRIBE HISTORY` | yes | attempted¹ | attempted¹ | no | attempted¹ |
+| Rows | `COUNT(*)` | yes | MATERIALIZED_VIEW: with `count_views=True`; STREAMING_TABLE: yes | with `count_views=True` | with `count_views=True` | UNKNOWN: yes; n/a type: with `count_views=True` |
+
+¹ The query runs. If Databricks does not support it for the object, the facts are shown as
+`n/a (<reason>)`.
+
+"UNKNOWN" is a `table_type` dataowl does not recognize, such as `MANAGED_SHALLOW_CLONE`. The
+original value is shown in the header. "n/a type" means the object type could not be read.
+
+## Fields
+
+### inspect
+
+- **Last modified** is the last data change, from `DESCRIBE DETAIL`. **Created** is from
+  `information_schema.tables`.
+- **Rows** is an exact `COUNT(*)`. On large tables this costs compute.
+- **Files (avg ...)** is the size divided by the number of files.
+- **Columns (... incl. nested)** counts every field, including fields inside structs, array
+  elements and map keys and values.
+- **not set (default)** means the table property is not set, so Databricks uses its default.
+  dataowl does not report what the default is.
+- **#** in the schema table is a running number from 1 in column order.
+- **HISTORY** shows the oldest and newest commit in the available history, the number of
+  commits, and the number of commits per operation.
+
+### analyze
+
+- **PRIMARY KEY** is the declared primary key, in declared order. It is informational and
+  not enforced by Databricks. The line is left out when `key` is not given.
+- **KEY ... Rows** is every row in the table, including rows with null in a key column.
+- **Rows with null in key** counts rows where at least one key column is null. These rows
+  are left out of every other line in the KEY block.
+- **Distinct keys** is the number of distinct key values without null.
+- **Keys occurring >1 time** is the number of key values that occur in more than one row.
+  **Rows in those keys** is the number of rows that have one of those key values.
+- **Rows per key** is the median and the maximum number of rows per key value, followed by
+  the number of key values with 1, 2–10, 11–100 and more than 100 rows.
+- **TIMESTAMP ... Nulls** is the number of rows with null, and their share of all rows.
+- **Future values** counts values after the current time of the session. A `DATE` is
+  compared as that date at 00:00:00.
+- **Days between distinct dates** is measured in whole days between consecutive distinct
+  dates. Cadence below one day is not visible.
+- **Rows per day** covers the last `days` whole days without today, in the session time
+  zone, and counts current values of the column, not historical changes.
+- **COMPARE first → second** counts rows where the second column is after (`>`), equal to
+  (`=`) or before (`<`) the first. Rows with null in either column are counted only in
+  **Either is null**, so the four numbers add up to all rows.
+- Medians and means are rounded to one decimal and shown without decimals when the rounded
+  value is whole.
+
+## Known deviations
+
+- `information_schema.columns.ordinal_position` is documented as numbered from 1, but has
+  been observed 0-based in Databricks (serverless, October 2026). dataowl keeps the raw
+  value in `ColumnInfo.position` and shows a running number in the `#` column.
+
+## License
+
+MIT. Copyright (c) 2026 Alexandro Dronnen. See [LICENSE](LICENSE).