dataplat 0.2.2__tar.gz → 0.3.0__tar.gz

{dataplat-0.2.2 → dataplat-0.3.0}/CHANGELOG.md

@@ -1,5 +1,128 @@
 # Changelog
 
+## 0.3.0
+
+### Changed
+
+- **Exit codes now say what went wrong.** Every failure used to be `1`, so a
+  wrapper script could not tell "your config is wrong" from "the warehouse is
+  down" — and only one of those is worth retrying. Typed failures now carry
+  their own code: `2` invalid input, `3` configuration, `4` authentication,
+  `5` external service. `0`, `1` and `2` keep their conventional meanings, and
+  `2` is deliberately shared with Click's own usage error, because
+  `--format nope` and `-t nosuchtarget` are one condition to the caller.
+
+  An unreachable warehouse exits `5`, since that is the retryable case. A bad
+  statement against a reachable server stays `1`: retrying a syntax error would
+  fail identically forever. Untyped failures and a declined confirmation also
+  stay `1`. The full table is in the README.
+
+  **Scripts that branch on a non-zero exit will see new numbers.** Anything
+  testing `== 1` for a config or auth problem needs updating.
+
+- **`dp db dbt-orphans` is more aggressive.** Its "which models are live" query
+  interpolated the dbt project name into a `LIKE` pattern without escaping it,
+  and dbt project names are snake_case — so the `_` in `my_project` matched any
+  character and a *sibling* project's models (`my2project`) sharing the
+  `dbt_artifacts` schema were counted as live. The same applied to
+  `DP_DBT_INVOCATION_COMMAND`, where `_` is ordinary in a dbt selector.
+
+  The direction is the point: an over-large live set makes **fewer** objects look
+  orphaned. Escaping it shrinks the live set, so dbt-orphans will now rename —
+  and after the grace period drop — objects it previously left alone. Run
+  `dp db dbt-orphans` (dry-run is the default) and read the plan before you
+  `--no-dry-run` the first time after upgrading.
+
+- **`dp status` runs its checks concurrently**: 40.7s to 10.4s with five
+  targets, one unreachable. Sections run in parallel and each database target is
+  probed in parallel within them, which is where the time actually went — a 10s
+  connect timeout per target was paid serially. Key order and section order are
+  unchanged, because both pools iterate the declared mapping rather than
+  completion order. The AWS section stays serial and last: it may hand the
+  terminal to an interactive `aws sso login`.
+
+### Added
+
+- **`--verbose` (or `DP_VERBOSE=1`) shows what dataplat actually sent** — SQL
+  statements, HTTP requests with status and duration, and AWS service calls.
+  It writes to stderr only, so `--json` and `--format csv` stay pipeable, and
+  everything passes through a redactor: passwords, secret values, bearer tokens
+  and API keys are never traced. Parameter values and response bodies are not
+  traced either — those are the data, not the request.
+
+- **Third-party command areas.** `dp` discovers areas declared in the
+  `dataplat.areas` entry-point group, so a package can add a command area
+  without a change here. Discovery reads only the entry-point metadata, never
+  imports the plugin, so `dp --version` and `dp --help` stay import-free and
+  fast. A plugin that fails to import warns on stderr and leaves the built-in
+  areas working; it cannot shadow a built-in area.
+
+- `dp config doctor` warns when a loaded `.envrc` value still contains an
+  unexpanded `$VAR`. dataplat does not run a shell, so
+  `export PGHOST=$DB_HOST` loads the literal text — which then surfaces as a
+  baffling connection failure rather than as the configuration mistake it is.
+
+- Shell completion is documented (`dp --install-completion`). It always worked;
+  the README never said so.
+
+### Fixed
+
+- `dp db query --format json|csv` could emit output that would not parse. The
+  progress spinner painted to stdout, which was invisible while Rich only did
+  that for a real terminal — the frames are erased — but `FORCE_COLOR` makes
+  Rich treat a pipe as a terminal too, and then the escape sequences ended up in
+  the redirected file. The spinner now follows the same sink as the notices.
+
+- `dp db describe` reported the owner's grant option two different ways
+  depending on whether you asked about a schema or a relation. PostgreSQL grants
+  an owner every grant option implicitly and never records it in the ACL, so
+  reading the ACL reported "cannot delegate" about a role that demonstrably can.
+  Schema privileges now agree with relation privileges.
+
+- The `Operating System :: OS Independent` classifier was an overclaim and has
+  been narrowed. Four things break on Windows: `dp config init` creates a
+  symlink, the dependency auto-install re-execs through `os.execvp`, the runner
+  commands shell out to `docker` with a POSIX default workdir, and the
+  credentials file is written with a `0o600` mode Windows ignores — after which
+  dataplat reports its own file as insecurely permissioned. CI tests Linux only.
+
+## 0.2.3
+
+Redshift-only fixes. Nothing changes for PostgreSQL targets.
+
+### Fixed
+
+- `dp db role show` no longer claims every Redshift user has no password. The
+  attribute query reported `password_set=False` unconditionally, but
+  `pg_user.passwd` is masked to `'********'` there just as `pg_roles.rolpassword`
+  is on PostgreSQL — so it asserted "this login has no password" for every user,
+  the same falsehood 0.2.2 fixed on the PostgreSQL side. It now reports
+  `unknown`, with the reason. A Redshift *group* still reports `no`, because a
+  group has no password to hold.
+
+- `dp db describe <schema>` now reports `USAGE` grants on Redshift. The query
+  read `information_schema.usage_privileges` filtered to `object_type = 'SCHEMA'`,
+  which the SQL standard defines over domains, collations and sequences — never
+  schemas — so it returned nothing on every server. It now scans
+  `has_schema_privilege`, mirroring how the same query has always reported
+  `CREATE` on that path. As with `CREATE`, a privilege scan cannot report a
+  grantor or a grant option, so both stay empty; the PostgreSQL path reads the
+  ACL and does better on both counts.
+
+  Both fixes rest on documented behaviour and internal precedent rather than a
+  live cluster — Redshift cannot be containerized, so CI cannot cover it. See
+  below.
+
+### Added
+
+- A Redshift conformance harness (`tests/integration/redshift/`) for anyone who
+  runs dataplat against a real cluster. The read-only tier is safe to point at a
+  warehouse in use — a guard refuses anything that is not plainly a read before
+  it reaches the server — and it interrogates the assumptions the two fixes above
+  depend on, printing what your cluster answered. `CONTRIBUTING.md` documents
+  both tiers and the evidence rules for changing SQL that runs on a dialect CI
+  cannot reach.
+
 ## 0.2.2
 
 Closes the six defects 0.2.1's integration suite found and pinned as expected

dataplat-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,177 @@
+# Contributing to dataplat
+
+## Setup
+
+```bash
+git clone https://github.com/hanslemm/dataplat
+cd dataplat
+uv sync --group dev --all-extras
+```
+
+## Checks
+
+The four gates CI runs, across Python 3.12 and 3.13:
+
+```bash
+uv run pytest
+uv run ruff check .
+uv run ruff format --check .
+uv run mypy dataplat
+```
+
+`uv run pytest` is green without Docker: the database-backed tests skip. To run
+them, start a server and point the suite at it:
+
+```bash
+docker run -d --name dp-pg-test \
+  -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=dataplat_test \
+  -p 55432:5432 postgres:16 -c shared_preload_libraries=pg_stat_statements
+docker exec dp-pg-test psql -U postgres -d dataplat_test \
+  -c 'CREATE EXTENSION IF NOT EXISTS pg_stat_statements'
+
+DP_TEST_PG_REQUIRED=1 uv run pytest              # everything
+uv run pytest -m "not integration"               # skip the database half
+docker rm -f -v dp-pg-test                       # -v, or the volume dangles
+```
+
+`DP_TEST_PG_REQUIRED=1` turns an unreachable server into an error instead of a
+skip. CI sets it; without it a broken database would make the whole suite skip
+and still report success.
+
+Commits follow [Conventional Commits](https://www.conventionalcommits.org/).
+
+## Testing against a real Redshift cluster
+
+Redshift is a managed service, so there is no container and CI cannot cover it.
+If you have a cluster, you can. The suite is in `tests/integration/redshift/` and
+is split into two tiers, because they need different permission to run:
+
+| Marker | Mutates? | Needs |
+| --- | --- | --- |
+| `redshift` | no — read-only, safe against a warehouse in use | a reachable cluster |
+| `redshift_ddl` | **yes** | a cluster you can throw away |
+
+Credentials come from an ordinary dataplat target, so they stay in your own
+`.envrc` and never reach the repo:
+
+```bash
+export DP_TARGETS=warehouse WAREHOUSE_ENGINE=redshift \
+    WAREHOUSE_HOST=... WAREHOUSE_USER=... WAREHOUSE_DATABASE=... \
+    WAREHOUSE_PASSWORD=...
+export DP_TEST_RS_TARGET=warehouse      # or DP_TEST_RS_DSN=... as an escape hatch
+
+DP_TEST_RS_REQUIRED=1 uv run pytest -m redshift        # read-only tier
+```
+
+| Variable | Effect |
+| --- | --- |
+| `DP_TEST_RS_TARGET` | a dataplat target name, resolved by the tool's own config |
+| `DP_TEST_RS_DSN` | a raw libpq URL, if you would rather not declare a target |
+| `DP_TEST_RS_REQUIRED` | an unreachable cluster becomes an error instead of a skip |
+| `DP_TEST_RS_DISPOSABLE` | **required** before any `redshift_ddl` test will run |
+| `DP_TEST_RS_SCHEMA` | a schema the read-only tier may inspect (otherwise discovered) |
+
+A plain `uv run pytest` is unaffected: with nothing configured, both tiers skip.
+
+### Why there is a client-side read-only guard
+
+`rs_cursor` refuses anything that is not plainly a read *before it is sent*, on
+top of the server-side `READ ONLY` transaction. Two layers, because the cluster
+may be production: Redshift roles are cluster-wide, and its transactional-DDL
+semantics differ from PostgreSQL's, so the rollback-per-test isolation the
+PostgreSQL harness relies on cannot be assumed to clean up a mistake. A
+server-side check would refuse the statement too — but only after it crossed the
+network to a warehouse someone depends on.
+
+It denies by default: only `SELECT`, `WITH … SELECT`, `EXPLAIN` and `SHOW` pass.
+It is not fooled by a leading comment, case, a stray semicolon, a second
+statement smuggled after a `SELECT`, a data-modifying CTE, `SELECT … INTO`, or a
+side-effecting builtin such as `pg_terminate_backend`. The statement splitter is
+hand-written rather than regex-based because a regex that ignores quoting can
+*hide* a statement — naive comment stripping turns `SELECT '--' ; DROP TABLE t`
+into a harmless-looking fragment plus a `DROP` the server will happily run. The
+one hole it cannot close is an unlisted side-effecting UDF; that is what the
+server-side layer is for, and `assert_read_only`'s docstring says so.
+
+### What a run does and does not prove
+
+A green read-only run proves dataplat's `SELECT`s are **valid Redshift SQL
+against a real server, returning results that unpack** — which is precisely the
+class of defect the PostgreSQL suite found repeatedly in these same functions
+(an empty `pg_partition_tree`, a masked column, a view that does not cover
+schemas). It cannot tell you anything about `GRANT`, `DROP`, `RENAME` or session
+termination; those need the DDL tier and a disposable cluster.
+
+The run also prints a conformance table of what the cluster answered, because
+the point is learning what the engine does — a green run that recorded nothing
+has taught nobody anything.
+
+## Dialect changes: what counts as evidence
+
+`dataplat/services/db` targets PostgreSQL and Redshift. PostgreSQL has a real
+integration suite behind it. Redshift has none and cannot get one cheaply — it
+is a managed service, so there is no container to run in CI.
+
+For a while the rule was simply "don't touch SQL that runs on Redshift, because
+you can't test it." That is a good instinct and a bad rule. Applied literally it
+blocked seven known defects, and when they were finally looked at one at a time,
+six were fixable and only one genuinely needed Redshift-specific SQL. Five did
+not touch Redshift SQL at all, and the sixth turned out to use a construct the
+codebase was already shipping to Redshift elsewhere.
+
+So the question is not "can I test this?" but **"what evidence do I have?"** A
+change affecting the Redshift path needs at least one of the following, in
+descending order of strength:
+
+0. **A conformance run confirmed it against a real cluster.** Strongest, and the
+   only one that is evidence rather than inference — see the section above. A fix
+   currently resting on class 2 should be upgraded to class 0 when someone runs
+   the suite, and revisited if the run *refutes* it. `test_conformance.py` names
+   the assumptions each shipped fix depends on for exactly this reason.
+
+1. **It changes no Redshift SQL.** The fix is pure Python, or touches only a
+   `_*_SQL_POSTGRES` constant. Dialect risk is zero by construction — verify
+   that claim honestly, then go ahead. Aggregation bugs, error handling, and
+   return-value shaping usually land here.
+
+2. **The construct is already in production on the Redshift path.** Cite the
+   file and line. `ESCAPE '\'` was safe to add to `orphans.py` because
+   `top_tables.py` had always sent it to both engines;
+   `has_schema_privilege(...)` is safe in `describe.py`'s Redshift branch
+   because that branch already calls it. Internal precedent beats
+   documentation: it is the same server, the same driver, and code someone is
+   already running.
+
+3. **It withdraws a claim rather than making one.** Replacing a confidently
+   wrong value with "unknown" cannot be more wrong than what it replaced. See
+   the standing rule below.
+
+4. **Documented Redshift behaviour, cited, plus a fake-cursor test** pinning the
+   SQL the Redshift branch emits. Weakest of the four, because documentation and
+   deployed reality drift. Use it when the change is worth the residual risk,
+   and say so in the commit.
+
+If none of the four applies, **do not guess.** Leave the defect, and record it
+in a comment next to the code it affects — not in a tracker nobody reads. The
+comment is what lets the next person re-evaluate instead of rediscovering.
+
+### Requirements either way
+
+- **Keep the engine constants split.** Never edit a `_*_SQL_REDSHIFT` constant
+  to fix a PostgreSQL bug. If a shared statement needs to diverge, split it and
+  leave the Redshift half byte-for-byte as it was.
+- **Add a fake-cursor test** asserting what the Redshift branch emits. It is the
+  only mechanism that covers that path at all, and it catches the common
+  accident of "fixed both branches when I meant one".
+- **Record the evidence in the code**, not just the commit message. A future
+  reader deciding whether they may touch the line needs to see why it is the way
+  it is.
+
+### Standing rule: prefer "unknown" to a confident falsehood
+
+`dp db role show` used to print `Password set: yes` for every role, including
+passwordless ones, because the column it read is masked to `'********'` and can
+never be NULL. A report that states something false is worse than one that
+admits a gap — especially a report someone is using for an audit. When the
+server will not tell you, say so, and say why in the same breath: a bare
+"unknown" reads as a tool defect.

{dataplat-0.2.2 → dataplat-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataplat
-Version: 0.2.2
+Version: 0.3.0
 Summary: One command to manage any shape of data platform: databases, ingestion, BI, cloud, and CI.
 Project-URL: Homepage, https://github.com/hanslemm/dataplat
 Project-URL: Repository, https://github.com/hanslemm/dataplat
@@ -14,7 +14,8 @@ Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: System Administrators
-Classifier: Operating System :: OS Independent
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Operating System :: POSIX :: Linux
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Database
@@ -94,7 +95,9 @@ dp
 
 ## Installation
 
-Requires Python 3.12 or newer.
+Requires Python 3.12 or newer, on Linux or macOS. Windows is untested and parts
+of it will not work: `dp config init` creates a symlink, the dependency
+self-install re-execs the process, and `dp ci github runner` drives `docker`.
 
 ```bash
 uv tool install "dataplat[all]"     # recommended: everything
@@ -141,6 +144,23 @@ Either path only ever *adds* to your install: the command is pinned to the
 tool underneath you, and it carries your existing extras along, so adding
 `db` cannot drop an `ingest` you already had.
 
+## Shell completion
+
+```bash
+dp --install-completion    # detect the shell, write the script, hook it up
+dp --show-completion       # print it instead, and install it yourself
+```
+
+Completion takes effect in the next shell. bash, zsh and fish are supported;
+`--show-completion` is the escape hatch when your rc file is managed by
+something else (Nix, chezmoi, a dotfiles repo) or when shell detection fails.
+
+`dp <TAB>` is answered from the area names alone and imports nothing.
+Completing *inside* an area has to import it, because the subcommands it owes
+the shell are that area's own — so the first `dp db <TAB>` pays for psycopg, and
+an area whose extra is not installed completes to nothing rather than offering
+to install it mid-keystroke.
+
 ## Quick start
 
 1. Declare your database targets — any names you like:
@@ -203,6 +223,7 @@ rely only on the global link you chose.
 | --- | --- |
 | `DP_ENVRC_PATH` | Explicit `.envrc` to load, ahead of every other candidate. |
 | `DP_ENVRC_ALLOW_CWD` | Set to `0` to stop picking up `.envrc` from the current directory. |
+| `DP_VERBOSE` | Set to `1` to trace every statement and request to stderr, for a whole session — same switch as `--verbose`. |
 | `DP_TARGETS` | Comma-separated DB target names (e.g. `warehouse,lake`). |
 | `DP_DEFAULT_TARGET` | Target used when `--target` is omitted (default: first of `DP_TARGETS`). |
 | `<NAME>_ENGINE` | `postgresql` (default) or `redshift`, per target. |
@@ -231,6 +252,89 @@ rely only on the global link you chose.
 - **`--limit/-n`** — row caps share one spelling everywhere.
 - **Secrets stay off argv** — prefer `--value-stdin` / hidden prompts; values
   are never echoed back.
+- **`--verbose`** — a root flag: show what the tool actually sent, on stderr.
+
+### Exit codes
+
+Exit codes are a contract, not an implementation detail — a wrapper script
+branches on them long after it has stopped reading our output:
+
+| Code | Meaning | Retry? |
+| --- | --- | --- |
+| `0` | Success. | — |
+| `1` | Unexpected or not-yet-classified failure. Also a declined confirmation: "no" is not an error, but it is not "done" either. | No — you don't know what happened. |
+| `2` | Invalid input: an unknown flag or target, a value that cannot be parsed, a combination of arguments that cannot work. | No — the command itself is wrong. |
+| `3` | Configuration problem: missing connection settings, an unknown engine, an unset `AIRBYTE_BASE_URL` or `DP_DBT_PROJECT`. | No — a human has to fix the config. |
+| `4` | Authentication failure: credentials rejected, a login endpoint that would not authenticate, `aws sso login` failed. | No — a new credential is needed. |
+| `5` | External service failure: a call to Airbyte, Superset or AWS failed, timed out or returned something unusable; a warehouse that refused the operation. | **Yes** — the only class where a retry can help. |
+
+`0`, `1` and `2` keep their conventional meanings. `2` is Click's own code for a
+usage error, which is why invalid input shares it: `dp db query --format nope`
+(Click's complaint) and `-t nosuchtarget` (ours) are one condition to the
+caller — "you passed something I cannot use" — and splitting them by who noticed
+would be a distinction with no use.
+
+The point of the codes above `2` is that `5` is the one worth retrying, and `3`
+and `4` are the ones you must never retry: no amount of sleeping and trying
+again creates a missing config file or repairs a rejected password. Cap the
+retries anyway — `5` means "the other end failed", which covers a warehouse
+that was restarting *and* a `DROP` the server refused because something still
+depends on it, and only the first of those gets better on its own.
+
+```bash
+dp db long-queries -t warehouse --json > queries.json
+case $? in
+  0) ;;
+  5) echo "service unavailable; will retry" >&2; exit 75 ;;   # EX_TEMPFAIL
+  *) echo "not retryable; fix and re-run" >&2; exit 1 ;;
+esac
+```
+
+Treat `1` as "unknown", never as "retryable": it is the code for a failure
+dataplat has not classified, so retrying it is a guess.
+
+### Verbose tracing
+
+`--verbose` (or `DP_VERBOSE=1`) answers the one question logs cannot: what did
+`dp` actually send?
+
+```bash
+dp --verbose db query 'SELECT 1'              # root flag, before the subcommand
+DP_VERBOSE=1 dp db long-queries 2> trace.log  # or for a whole session
+dp --verbose db describe public 2>&1 >/dev/null | grep '\[dp:sql\]'
+```
+
+Every line is prefixed with its category — `[dp:sql]` or `[dp:http]` — and
+collapsed onto one line, so the output greps cleanly:
+
+```text
+[dp:sql] connect me@db.example.com:5432/analytics engine=postgresql
+[dp:sql] SELECT 1 FROM pg_namespace WHERE nspname = %s | 1 params bound
+[dp:http] GET https://api.airbyte.com/v1/jobs?limit=20
+[dp:http] GET https://api.airbyte.com/v1/jobs?limit=20 -> 200 143.8ms
+```
+
+SQL is traced *before* the statement runs, which is the point: the trace you
+need is the one for the query that never came back, and a line written afterwards
+would never be written at all. That is also why there is no duration on it — use
+`dp db long-queries` for how long. HTTP gets two lines for the same reason, one
+on the way out and one on the response; a line with no `-> status` partner *is*
+the signal that a request hung, was refused, or never connected.
+
+**It writes to stderr and never to stdout**, so `--json` and `--format csv` stay
+machine-readable with tracing on. Piping into `jq` is still valid, and
+`2>/dev/null` drops the trace without touching the data:
+
+```bash
+dp --verbose db query --format json 'SELECT 1' 2>/dev/null | jq
+```
+
+**Secrets are never traced.** Every message is redacted on the way out —
+passwords (including the SQL `PASSWORD '…'` literal that role creation sends),
+tokens, API keys, `Authorization` headers and credentials embedded in a URL all
+become `***`. Parameter values, result rows and response bodies are not traced
+at all: they are your warehouse's data, and a trace that scrolls the answer past
+you has hidden the request it exists to show.
 
 ## Examples
 
@@ -324,6 +428,9 @@ dp ingest airbyte connections set-cursor -c <connection-id> --xmin 0 --yes
 
 ## Development
 
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the integration suite and the rules
+for changing SQL that runs on Redshift.
+
 ```bash
 git clone https://github.com/hanslemm/dataplat
 cd dataplat
@@ -337,6 +444,13 @@ uv run mypy dataplat
 CI runs those four across Python 3.12 and 3.13 — the floor the wheel
 advertises as well as the pinned dev version.
 
+Redshift cannot be containerized, so CI cannot cover it. If you run dataplat
+against a Redshift cluster, you can verify your own deployment: point
+`DP_TEST_RS_TARGET` at one of your targets and run the read-only tier
+(`uv run pytest -m redshift`). It only issues `SELECT`s — a guard refuses
+anything else before it reaches the server — and prints what your cluster
+answered. See [CONTRIBUTING.md](CONTRIBUTING.md#testing-against-a-real-redshift-cluster).
+
 ### Integration tests against a real PostgreSQL
 
 Most of the suite drives a fake database cursor. That proves a code path

{dataplat-0.2.2 → dataplat-0.3.0}/README.md

@@ -49,7 +49,9 @@ dp
 
 ## Installation
 
-Requires Python 3.12 or newer.
+Requires Python 3.12 or newer, on Linux or macOS. Windows is untested and parts
+of it will not work: `dp config init` creates a symlink, the dependency
+self-install re-execs the process, and `dp ci github runner` drives `docker`.
 
 ```bash
 uv tool install "dataplat[all]"     # recommended: everything
@@ -96,6 +98,23 @@ Either path only ever *adds* to your install: the command is pinned to the
 tool underneath you, and it carries your existing extras along, so adding
 `db` cannot drop an `ingest` you already had.
 
+## Shell completion
+
+```bash
+dp --install-completion    # detect the shell, write the script, hook it up
+dp --show-completion       # print it instead, and install it yourself
+```
+
+Completion takes effect in the next shell. bash, zsh and fish are supported;
+`--show-completion` is the escape hatch when your rc file is managed by
+something else (Nix, chezmoi, a dotfiles repo) or when shell detection fails.
+
+`dp <TAB>` is answered from the area names alone and imports nothing.
+Completing *inside* an area has to import it, because the subcommands it owes
+the shell are that area's own — so the first `dp db <TAB>` pays for psycopg, and
+an area whose extra is not installed completes to nothing rather than offering
+to install it mid-keystroke.
+
 ## Quick start
 
 1. Declare your database targets — any names you like:
@@ -158,6 +177,7 @@ rely only on the global link you chose.
 | --- | --- |
 | `DP_ENVRC_PATH` | Explicit `.envrc` to load, ahead of every other candidate. |
 | `DP_ENVRC_ALLOW_CWD` | Set to `0` to stop picking up `.envrc` from the current directory. |
+| `DP_VERBOSE` | Set to `1` to trace every statement and request to stderr, for a whole session — same switch as `--verbose`. |
 | `DP_TARGETS` | Comma-separated DB target names (e.g. `warehouse,lake`). |
 | `DP_DEFAULT_TARGET` | Target used when `--target` is omitted (default: first of `DP_TARGETS`). |
 | `<NAME>_ENGINE` | `postgresql` (default) or `redshift`, per target. |
@@ -186,6 +206,89 @@ rely only on the global link you chose.
 - **`--limit/-n`** — row caps share one spelling everywhere.
 - **Secrets stay off argv** — prefer `--value-stdin` / hidden prompts; values
   are never echoed back.
+- **`--verbose`** — a root flag: show what the tool actually sent, on stderr.
+
+### Exit codes
+
+Exit codes are a contract, not an implementation detail — a wrapper script
+branches on them long after it has stopped reading our output:
+
+| Code | Meaning | Retry? |
+| --- | --- | --- |
+| `0` | Success. | — |
+| `1` | Unexpected or not-yet-classified failure. Also a declined confirmation: "no" is not an error, but it is not "done" either. | No — you don't know what happened. |
+| `2` | Invalid input: an unknown flag or target, a value that cannot be parsed, a combination of arguments that cannot work. | No — the command itself is wrong. |
+| `3` | Configuration problem: missing connection settings, an unknown engine, an unset `AIRBYTE_BASE_URL` or `DP_DBT_PROJECT`. | No — a human has to fix the config. |
+| `4` | Authentication failure: credentials rejected, a login endpoint that would not authenticate, `aws sso login` failed. | No — a new credential is needed. |
+| `5` | External service failure: a call to Airbyte, Superset or AWS failed, timed out or returned something unusable; a warehouse that refused the operation. | **Yes** — the only class where a retry can help. |
+
+`0`, `1` and `2` keep their conventional meanings. `2` is Click's own code for a
+usage error, which is why invalid input shares it: `dp db query --format nope`
+(Click's complaint) and `-t nosuchtarget` (ours) are one condition to the
+caller — "you passed something I cannot use" — and splitting them by who noticed
+would be a distinction with no use.
+
+The point of the codes above `2` is that `5` is the one worth retrying, and `3`
+and `4` are the ones you must never retry: no amount of sleeping and trying
+again creates a missing config file or repairs a rejected password. Cap the
+retries anyway — `5` means "the other end failed", which covers a warehouse
+that was restarting *and* a `DROP` the server refused because something still
+depends on it, and only the first of those gets better on its own.
+
+```bash
+dp db long-queries -t warehouse --json > queries.json
+case $? in
+  0) ;;
+  5) echo "service unavailable; will retry" >&2; exit 75 ;;   # EX_TEMPFAIL
+  *) echo "not retryable; fix and re-run" >&2; exit 1 ;;
+esac
+```
+
+Treat `1` as "unknown", never as "retryable": it is the code for a failure
+dataplat has not classified, so retrying it is a guess.
+
+### Verbose tracing
+
+`--verbose` (or `DP_VERBOSE=1`) answers the one question logs cannot: what did
+`dp` actually send?
+
+```bash
+dp --verbose db query 'SELECT 1'              # root flag, before the subcommand
+DP_VERBOSE=1 dp db long-queries 2> trace.log  # or for a whole session
+dp --verbose db describe public 2>&1 >/dev/null | grep '\[dp:sql\]'
+```
+
+Every line is prefixed with its category — `[dp:sql]` or `[dp:http]` — and
+collapsed onto one line, so the output greps cleanly:
+
+```text
+[dp:sql] connect me@db.example.com:5432/analytics engine=postgresql
+[dp:sql] SELECT 1 FROM pg_namespace WHERE nspname = %s | 1 params bound
+[dp:http] GET https://api.airbyte.com/v1/jobs?limit=20
+[dp:http] GET https://api.airbyte.com/v1/jobs?limit=20 -> 200 143.8ms
+```
+
+SQL is traced *before* the statement runs, which is the point: the trace you
+need is the one for the query that never came back, and a line written afterwards
+would never be written at all. That is also why there is no duration on it — use
+`dp db long-queries` for how long. HTTP gets two lines for the same reason, one
+on the way out and one on the response; a line with no `-> status` partner *is*
+the signal that a request hung, was refused, or never connected.
+
+**It writes to stderr and never to stdout**, so `--json` and `--format csv` stay
+machine-readable with tracing on. Piping into `jq` is still valid, and
+`2>/dev/null` drops the trace without touching the data:
+
+```bash
+dp --verbose db query --format json 'SELECT 1' 2>/dev/null | jq
+```
+
+**Secrets are never traced.** Every message is redacted on the way out —
+passwords (including the SQL `PASSWORD '…'` literal that role creation sends),
+tokens, API keys, `Authorization` headers and credentials embedded in a URL all
+become `***`. Parameter values, result rows and response bodies are not traced
+at all: they are your warehouse's data, and a trace that scrolls the answer past
+you has hidden the request it exists to show.
 
 ## Examples
 
@@ -279,6 +382,9 @@ dp ingest airbyte connections set-cursor -c <connection-id> --xmin 0 --yes
 
 ## Development
 
+See [CONTRIBUTING.md](CONTRIBUTING.md) for the integration suite and the rules
+for changing SQL that runs on Redshift.
+
 ```bash
 git clone https://github.com/hanslemm/dataplat
 cd dataplat
@@ -292,6 +398,13 @@ uv run mypy dataplat
 CI runs those four across Python 3.12 and 3.13 — the floor the wheel
 advertises as well as the pinned dev version.
 
+Redshift cannot be containerized, so CI cannot cover it. If you run dataplat
+against a Redshift cluster, you can verify your own deployment: point
+`DP_TEST_RS_TARGET` at one of your targets and run the read-only tier
+(`uv run pytest -m redshift`). It only issues `SELECT`s — a guard refuses
+anything else before it reaches the server — and prints what your cluster
+answered. See [CONTRIBUTING.md](CONTRIBUTING.md#testing-against-a-real-redshift-cluster).
+
 ### Integration tests against a real PostgreSQL
 
 Most of the suite drives a fake database cursor. That proves a code path