dataplat 0.2.2__tar.gz → 0.2.3__tar.gz

{dataplat-0.2.2 → dataplat-0.2.3}/CHANGELOG.md

@@ -1,5 +1,42 @@
 # Changelog
 
+## 0.2.3
+
+Redshift-only fixes. Nothing changes for PostgreSQL targets.
+
+### Fixed
+
+- `dp db role show` no longer claims every Redshift user has no password. The
+  attribute query reported `password_set=False` unconditionally, but
+  `pg_user.passwd` is masked to `'********'` there just as `pg_roles.rolpassword`
+  is on PostgreSQL — so it asserted "this login has no password" for every user,
+  the same falsehood 0.2.2 fixed on the PostgreSQL side. It now reports
+  `unknown`, with the reason. A Redshift *group* still reports `no`, because a
+  group has no password to hold.
+
+- `dp db describe <schema>` now reports `USAGE` grants on Redshift. The query
+  read `information_schema.usage_privileges` filtered to `object_type = 'SCHEMA'`,
+  which the SQL standard defines over domains, collations and sequences — never
+  schemas — so it returned nothing on every server. It now scans
+  `has_schema_privilege`, mirroring how the same query has always reported
+  `CREATE` on that path. As with `CREATE`, a privilege scan cannot report a
+  grantor or a grant option, so both stay empty; the PostgreSQL path reads the
+  ACL and does better on both counts.
+
+  Both fixes rest on documented behaviour and internal precedent rather than a
+  live cluster — Redshift cannot be containerized, so CI cannot cover it. See
+  below.
+
+### Added
+
+- A Redshift conformance harness (`tests/integration/redshift/`) for anyone who
+  runs dataplat against a real cluster. The read-only tier is safe to point at a
+  warehouse in use — a guard refuses anything that is not plainly a read before
+  it reaches the server — and it interrogates the assumptions the two fixes above
+  depend on, printing what your cluster answered. `CONTRIBUTING.md` documents
+  both tiers and the evidence rules for changing SQL that runs on a dialect CI
+  cannot reach.
+
 ## 0.2.2
 
 Closes the six defects 0.2.1's integration suite found and pinned as expected

dataplat-0.2.3/CONTRIBUTING.md

@@ -0,0 +1,177 @@
+# Contributing to dataplat
+
+## Setup
+
+```bash
+git clone https://github.com/hanslemm/dataplat
+cd dataplat
+uv sync --group dev --all-extras
+```
+
+## Checks
+
+The four gates CI runs, across Python 3.12 and 3.13:
+
+```bash
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+uv run mypy dataplat
+```
+
+`uv run pytest` is green without Docker: the database-backed tests skip. To run
+them, start a server and point the suite at it:
+
+```bash
+docker run -d --name dp-pg-test \
+  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=dataplat_test \
+  -p 55432:5432 postgres:16 -c shared_preload_libraries=pg_stat_statements
+docker exec dp-pg-test psql -U postgres -d dataplat_test \
+  -c 'CREATE EXTENSION IF NOT EXISTS pg_stat_statements'
+
+DP_TEST_PG_REQUIRED=1 uv run pytest              # everything
+uv run pytest -m "not integration"               # skip the database half
+docker rm -f -v dp-pg-test                       # -v, or the volume dangles
+```
+
+`DP_TEST_PG_REQUIRED=1` turns an unreachable server into an error instead of a
+skip. CI sets it; without it a broken database would make the whole suite skip
+and still report success.
+
+Commits follow [Conventional Commits](https://www.conventionalcommits.org/).
+
+## Testing against a real Redshift cluster
+
+Redshift is a managed service, so there is no container and CI cannot cover it.
+If you have a cluster, you can. The suite is in `tests/integration/redshift/` and
+is split into two tiers, because they need different permission to run:
+
+| Marker | Mutates? | Needs |
+| --- | --- | --- |
+| `redshift` | no — read-only, safe against a warehouse in use | a reachable cluster |
+| `redshift_ddl` | **yes** | a cluster you can throw away |
+
+Credentials come from an ordinary dataplat target, so they stay in your own
+`.envrc` and never reach the repo:
+
+```bash
+export DP_TARGETS=warehouse WAREHOUSE_ENGINE=redshift \
+    WAREHOUSE_HOST=... WAREHOUSE_USER=... WAREHOUSE_DATABASE=... \
+    WAREHOUSE_PASSWORD=...
+export DP_TEST_RS_TARGET=warehouse      # or DP_TEST_RS_DSN=... as an escape hatch
+
+DP_TEST_RS_REQUIRED=1 uv run pytest -m redshift        # read-only tier
+```
+
+| Variable | Effect |
+| --- | --- |
+| `DP_TEST_RS_TARGET` | a dataplat target name, resolved by the tool's own config |
+| `DP_TEST_RS_DSN` | a raw libpq URL, if you would rather not declare a target |
+| `DP_TEST_RS_REQUIRED` | an unreachable cluster becomes an error instead of a skip |
+| `DP_TEST_RS_DISPOSABLE` | **required** before any `redshift_ddl` test will run |
+| `DP_TEST_RS_SCHEMA` | a schema the read-only tier may inspect (otherwise discovered) |
+
+A plain `uv run pytest` is unaffected: with nothing configured, both tiers skip.
+
+### Why there is a client-side read-only guard
+
+`rs_cursor` refuses anything that is not plainly a read *before it is sent*, on
+top of the server-side `READ ONLY` transaction. Two layers, because the cluster
+may be production: Redshift roles are cluster-wide, and its transactional-DDL
+semantics differ from PostgreSQL's, so the rollback-per-test isolation the
+PostgreSQL harness relies on cannot be assumed to clean up a mistake. A
+server-side check would refuse the statement too — but only after it crossed the
+network to a warehouse someone depends on.
+
+It denies by default: only `SELECT`, `WITH … SELECT`, `EXPLAIN` and `SHOW` pass.
+It is not fooled by a leading comment, case, a stray semicolon, a second
+statement smuggled after a `SELECT`, a data-modifying CTE, `SELECT … INTO`, or a
+side-effecting builtin such as `pg_terminate_backend`. The statement splitter is
+hand-written rather than regex-based because a regex that ignores quoting can
+*hide* a statement — naive comment stripping turns `SELECT '--' ; DROP TABLE t`
+into a harmless-looking fragment plus a `DROP` the server will happily run. The
+one hole it cannot close is an unlisted side-effecting UDF; that is what the
+server-side layer is for, and `assert_read_only`'s docstring says so.
+
+### What a run does and does not prove
+
+A green read-only run proves dataplat's `SELECT`s are **valid Redshift SQL
+against a real server, returning results that unpack** — which is precisely the
+class of defect the PostgreSQL suite found repeatedly in these same functions
+(an empty `pg_partition_tree`, a masked column, a view that does not cover
+schemas). It cannot tell you anything about `GRANT`, `DROP`, `RENAME` or session
+termination; those need the DDL tier and a disposable cluster.
+
+The run also prints a conformance table of what the cluster answered, because
+the point is learning what the engine does — a green run that recorded nothing
+has taught nobody anything.
+
+## Dialect changes: what counts as evidence
+
+`dataplat/services/db` targets PostgreSQL and Redshift. PostgreSQL has a real
+integration suite behind it. Redshift has none and cannot get one cheaply — it
+is a managed service, so there is no container to run in CI.
+
+For a while the rule was simply "don't touch SQL that runs on Redshift, because
+you can't test it." That is a good instinct and a bad rule. Applied literally it
+blocked seven known defects, and when they were finally looked at one at a time,
+six were fixable and only one genuinely needed Redshift-specific SQL. Five did
+not touch Redshift SQL at all, and the sixth turned out to use a construct the
+codebase was already shipping to Redshift elsewhere.
+
+So the question is not "can I test this?" but **"what evidence do I have?"** A
+change affecting the Redshift path needs at least one of the following, in
+descending order of strength:
+
+0. **A conformance run confirmed it against a real cluster.** Strongest, and the
+   only one that is evidence rather than inference — see the section above. A fix
+   currently resting on class 2 should be upgraded to class 0 when someone runs
+   the suite, and revisited if the run *refutes* it. `test_conformance.py` names
+   the assumptions each shipped fix depends on for exactly this reason.
+
+1. **It changes no Redshift SQL.** The fix is pure Python, or touches only a
+   `_*_SQL_POSTGRES` constant. Dialect risk is zero by construction — verify
+   that claim honestly, then go ahead. Aggregation bugs, error handling, and
+   return-value shaping usually land here.
+
+2. **The construct is already in production on the Redshift path.** Cite the
+   file and line. `ESCAPE '\'` was safe to add to `orphans.py` because
+   `top_tables.py` had always sent it to both engines;
+   `has_schema_privilege(...)` is safe in `describe.py`'s Redshift branch
+   because that branch already calls it. Internal precedent beats
+   documentation: it is the same server, the same driver, and code someone is
+   already running.
+
+3. **It withdraws a claim rather than making one.** Replacing a confidently
+   wrong value with "unknown" cannot be more wrong than what it replaced. See
+   the standing rule below.
+
+4. **Documented Redshift behaviour, cited, plus a fake-cursor test** pinning the
+   SQL the Redshift branch emits. Weakest of the four, because documentation and
+   deployed reality drift. Use it when the change is worth the residual risk,
+   and say so in the commit.
+
+If none of the four applies, **do not guess.** Leave the defect, and record it
+in a comment next to the code it affects — not in a tracker nobody reads. The
+comment is what lets the next person re-evaluate instead of rediscovering.
+
+### Requirements either way
+
+- **Keep the engine constants split.** Never edit a `_*_SQL_REDSHIFT` constant
+  to fix a PostgreSQL bug. If a shared statement needs to diverge, split it and
+  leave the Redshift half byte-for-byte as it was.
+- **Add a fake-cursor test** asserting what the Redshift branch emits. It is the
+  only mechanism that covers that path at all, and it catches the common
+  accident of "fixed both branches when I meant one".
+- **Record the evidence in the code**, not just the commit message. A future
+  reader deciding whether they may touch the line needs to see why it is the way
+  it is.
+
+### Standing rule: prefer "unknown" to a confident falsehood
+
+`dp db role show` used to print `Password set: yes` for every role, including
+passwordless ones, because the column it read is masked to `'********'` and can
+never be NULL. A report that states something false is worse than one that
+admits a gap — especially a report someone is using for an audit. When the
+server will not tell you, say so, and say why in the same breath: a bare
+"unknown" reads as a tool defect.

{dataplat-0.2.2 → dataplat-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataplat
-Version: 0.2.2
+Version: 0.2.3
 Summary: One command to manage any shape of data platform: databases, ingestion, BI, cloud, and CI.
 Project-URL: Homepage, https://github.com/hanslemm/dataplat
 Project-URL: Repository, https://github.com/hanslemm/dataplat
@@ -324,6 +324,9 @@ dp ingest airbyte connections set-cursor -c <connection-id> --xmin 0 --yes
 
 ## Development
 
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the integration suite and the rules
+for changing SQL that runs on Redshift.
+
 ```bash
 git clone https://github.com/hanslemm/dataplat
 cd dataplat
@@ -337,6 +340,13 @@ uv run mypy dataplat
 CI runs those four across Python 3.12 and 3.13 — the floor the wheel
 advertises as well as the pinned dev version.
 
+Redshift cannot be containerized, so CI cannot cover it. If you run dataplat
+against a Redshift cluster, you can verify your own deployment: point
+`DP_TEST_RS_TARGET` at one of your targets and run the read-only tier
+(`uv run pytest -m redshift`). It only issues `SELECT`s — a guard refuses
+anything else before it reaches the server — and prints what your cluster
+answered. See [CONTRIBUTING.md](CONTRIBUTING.md#testing-against-a-real-redshift-cluster).
+
 ### Integration tests against a real PostgreSQL
 
 Most of the suite drives a fake database cursor. That proves a code path

{dataplat-0.2.2 → dataplat-0.2.3}/README.md

@@ -279,6 +279,9 @@ dp ingest airbyte connections set-cursor -c <connection-id> --xmin 0 --yes
 
 ## Development
 
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the integration suite and the rules
+for changing SQL that runs on Redshift.
+
 ```bash
 git clone https://github.com/hanslemm/dataplat
 cd dataplat
@@ -292,6 +295,13 @@ uv run mypy dataplat
 CI runs those four across Python 3.12 and 3.13 — the floor the wheel
 advertises as well as the pinned dev version.
 
+Redshift cannot be containerized, so CI cannot cover it. If you run dataplat
+against a Redshift cluster, you can verify your own deployment: point
+`DP_TEST_RS_TARGET` at one of your targets and run the read-only tier
+(`uv run pytest -m redshift`). It only issues `SELECT`s — a guard refuses
+anything else before it reaches the server — and prints what your cluster
+answered. See [CONTRIBUTING.md](CONTRIBUTING.md#testing-against-a-real-redshift-cluster).
+
 ### Integration tests against a real PostgreSQL
 
 Most of the suite drives a fake database cursor. That proves a code path

{dataplat-0.2.2 → dataplat-0.2.3}/dataplat/cli/db/role.py

@@ -72,16 +72,23 @@ def _more_line(hidden: int) -> str:
     return f"  [dim italic]… and {hidden} more (raise --limit to see all).[/dim italic]"
 
 
-def _password_set_value(password_set: bool | None) -> str:
+def _password_set_value(password_set: bool | None, engine: SqlEngine) -> str:
     """Render the tri-state ``password_set`` as markup.
 
-    ``None`` means the server refused to say (pg_authid is superuser-only), and
-    the reason travels with the word: a bare "unknown" in a security report
-    reads as a tool defect, and rendering it as "no" would be a false negative
-    on exactly the field an auditor came for.
+    ``None`` means the server would not say, and the reason travels with the
+    word: a bare "unknown" in a security report reads as a tool defect, and
+    rendering it as "no" would be a false negative on exactly the field an
+    auditor came for. The reason differs by engine, and naming pg_authid on
+    Redshift — which has no such relation — would send the reader somewhere
+    that does not exist.
     """
     if password_set is None:
-        return "unknown [dim](needs superuser to read pg_authid)[/dim]"
+        reason = (
+            "no readable password catalog on Redshift"
+            if engine == SqlEngine.redshift
+            else "needs superuser to read pg_authid"
+        )
+        return f"unknown [dim]({reason})[/dim]"
     return "yes" if password_set else "no"
 
 
@@ -120,7 +127,10 @@ def _attributes_metadata(attrs: RoleAttributes) -> list[tuple[str, str]]:
 
 
 def _render_attributes(
-    console: Console, counter: _SectionCounter, attrs: RoleAttributes
+    console: Console,
+    counter: _SectionCounter,
+    attrs: RoleAttributes,
+    engine: SqlEngine,
 ) -> None:
     _print_section_heading(
         console, counter, "Attributes", "Role flags, login, and limits."
@@ -139,7 +149,7 @@ def _render_attributes(
         "Connection limit",
         "unlimited" if attrs.connection_limit < 0 else str(attrs.connection_limit),
     )
-    table.add_row("Password set", _password_set_value(attrs.password_set))
+    table.add_row("Password set", _password_set_value(attrs.password_set, engine))
     table.add_row("Valid until", cell(attrs.valid_until or "—"))
     console.print(_indent(table))
 
@@ -345,7 +355,7 @@ def render_role_description(
     console.print()
 
     counter = _SectionCounter()
-    _render_attributes(console, counter, desc.attributes)
+    _render_attributes(console, counter, desc.attributes, engine)
     parent_caption = (
         "Roles this role is a member of (single-level; Redshift groups)."
         if engine == SqlEngine.redshift

{dataplat-0.2.2 → dataplat-0.2.3}/dataplat/services/db/describe.py

@@ -787,14 +787,29 @@ ORDER BY grantee, privilege_type
 """
 
 
-# Unchanged from when this query was shared by both engines, and deliberately
-# NOT switched to the aclexplode form above: Redshift has no aclexplode() and
-# no cluster is available to test a replacement against. Its USAGE reporting is
-# therefore still unverified and probably as empty as PostgreSQL's was.
+# Not the aclexplode form above: Redshift has no aclexplode(), and no cluster is
+# available to test a replacement against.
+#
+# The USAGE half used to read information_schema.usage_privileges filtered to
+# object_type = 'SCHEMA', which the SQL standard defines over domains,
+# collations and sequences — never schemas. It returned zero rows on every
+# server, so a GRANT USAGE ON SCHEMA was invisible here exactly as it was on
+# PostgreSQL. It is replaced by a has_schema_privilege() scan mirroring the
+# CREATE half directly below, which this same query has always run against
+# Redshift: same function, same catalog, same shape, only the privilege string
+# differs (CONTRIBUTING, evidence 2 — internal precedent on this very path).
+#
+# Known limitation, shared with the CREATE half and unchanged: a privilege scan
+# cannot report a grantor or a grant option, so both are reported empty, and
+# roles holding the privilege only through membership appear as their own rows.
+# The PostgreSQL path reads the ACL and does better on both counts.
 _SCHEMA_PRIVILEGES_SQL_REDSHIFT = """
-SELECT grantee, privilege_type, is_grantable = 'YES', grantor
-FROM information_schema.usage_privileges
-WHERE object_schema = %s AND object_type = 'SCHEMA'
+SELECT grantee, 'USAGE', false, ''
+FROM (
+    SELECT r.rolname AS grantee
+    FROM pg_roles r
+    WHERE has_schema_privilege(r.rolname, %s, 'USAGE')
+) u
 UNION ALL
 SELECT grantee, 'CREATE', false, ''
 FROM (

{dataplat-0.2.2 → dataplat-0.2.3}/dataplat/services/db/role.py

@@ -166,7 +166,9 @@ def fetch_attributes(cursor: Any, name: str, engine: SqlEngine) -> RoleAttribute
         cursor.execute(_ATTRS_SQL_REDSHIFT, (name,))
         row = cursor.fetchone()
         if row is None:
-            # Group — no login, no attributes table.
+            # Group — no login, no attributes table. password_set=False is a
+            # real answer here rather than a guess: a Redshift group has no
+            # password to hold.
             return RoleAttributes(
                 can_login=False,
                 superuser=False,
@@ -189,7 +191,13 @@ def fetch_attributes(cursor: Any, name: str, engine: SqlEngine) -> RoleAttribute
             replication=False,
             bypass_rls=False,
             connection_limit=-1,
-            password_set=False,
+            # Unknown, not False. This is a Redshift *user*, and pg_user.passwd
+            # is masked to '********' exactly as pg_roles.rolpassword is on
+            # PostgreSQL — so False asserted "this login has no password" for
+            # every user, which is the same falsehood that bug fixed there.
+            # Reporting unknown withdraws the claim without inventing Redshift
+            # SQL nobody here can execute (see CONTRIBUTING, evidence 1 and 3).
+            password_set=None,
             valid_until=str(valid_until) if valid_until is not None else None,
         )
     authid_readable = _pg_authid_readable(cursor)

{dataplat-0.2.2 → dataplat-0.2.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "dataplat"
-version = "0.2.2"
+version = "0.2.3"
 description = "One command to manage any shape of data platform: databases, ingestion, BI, cloud, and CI."
 readme = "README.md"
 license = "MIT"
@@ -75,6 +75,13 @@ markers = [
     # PostgreSQL (DP_TEST_PG_DSN). Registered so `-m 'not integration'`
     # selects against a known marker instead of warning about an unknown one.
     "integration: requires a live PostgreSQL server",
+    # Redshift cannot be containerized, so these two run only when someone
+    # points them at a cluster. Read-only and mutating are separate markers
+    # because they need different permission to run: `redshift` is safe against
+    # a warehouse in use, `redshift_ddl` is not, and keeping them selectable
+    # apart is what lets a production cluster be used for the first tier only.
+    "redshift: requires a live Redshift cluster; read-only",
+    "redshift_ddl: requires a DISPOSABLE Redshift cluster; mutates it",
 ]
 
 [tool.ruff]

dataplat-0.2.3/tests/integration/redshift/__init__.py

@@ -0,0 +1,19 @@
+"""Integration tests that run against a live Amazon Redshift cluster.
+
+Two tiers, two markers, because the two have very different risk profiles:
+
+``redshift``
+    Needs a reachable cluster and is **read-only**. Every statement passes a
+    client-side guard before it reaches the server, and the session asks the
+    server for a READ ONLY transaction on top of that, so this tier is safe to
+    point at a warehouse somebody depends on.
+
+``redshift_ddl``
+    Needs a **disposable** cluster and mutates it. Refuses to run unless
+    ``DP_TEST_RS_DISPOSABLE`` says out loud that the cluster is expendable.
+
+See ``tests/integration/redshift/conftest.py`` for the fixture contract and the
+``DP_TEST_RS_*`` environment variables. Everything here is also marked
+``integration`` by ``tests/integration/conftest.py``, so ``pytest -m 'not
+integration'`` excludes it.
+"""