dataplat 0.2.3__tar.gz → 0.4.0__tar.gz

{dataplat-0.2.3 → dataplat-0.4.0}/.github/workflows/ci.yml

@@ -32,6 +32,10 @@ jobs:
       - name: Setup Python
         run: uv python install ${{ matrix.python-version }}
 
+      # --all-extras is what installs psycopg (the "db" extra) and duckdb (the
+      # "duckdb" extra). The DuckDB step below needs the latter and has no skip
+      # path, so dropping this flag turns into a hard failure rather than a
+      # quietly shrinking test run.
       - name: Install dependencies
         run: uv sync --group dev --all-extras
 
@@ -53,6 +57,23 @@ jobs:
       - name: Pytest (unit)
         run: uv run pytest -m "not integration"
 
+      # The third engine's real-SQL tier, and the only one that belongs in this
+      # job: DuckDB is a library, so it needs no container, no credentials and no
+      # wait loop -- just the extra `uv sync` already installed. Running it in
+      # both matrix legs is the point, since an embedded engine is the one place
+      # an interpreter difference could plausibly show up in a database.
+      #
+      # Selected by path rather than by marker. tests/integration/duckdb/ sits
+      # under tests/integration/, so its conftest chain stamps every test with
+      # the `integration` marker the step above deselects; a `duckdb` marker
+      # would have to be registered in pyproject.toml, and one line here is
+      # cheaper than a marker that then has to be kept in sync with a directory.
+      # These tests are consequently also collected by the `integration` job
+      # below -- harmless, since they run in under two seconds and touch no
+      # server, but *this* is where DuckDB is proved, on both interpreters.
+      - name: Pytest (DuckDB integration)
+        run: uv run pytest tests/integration/duckdb
+
   # A separate job, not extra steps in the matrix above, for two reasons: the
   # unit legs are the fast signal and must not queue behind a container boot,
   # and a database per Python version would buy nothing -- the SQL these tests

{dataplat-0.2.3 → dataplat-0.4.0}/.github/workflows/release.yml

@@ -50,6 +50,9 @@ jobs:
           print(f"Tag '{tag}' matches [project].version.")
           PY
 
+      # --all-extras installs psycopg ("db") and duckdb ("duckdb"). The wheel
+      # advertises both as extras, so both are installed before the tag is
+      # allowed through.
       - name: Install dependencies
         run: uv sync --group dev --all-extras
 
@@ -67,6 +70,16 @@ jobs:
       - name: Pytest (unit)
         run: uv run pytest -m "not integration"
 
+      # Also mirrors ci.yml, and it belongs in the tag gate for the same reason
+      # the `integration` job does: a version can be yanked but never replaced,
+      # and `verify`'s fake cursors cannot tell whether a `duckdb_*()` query
+      # parses. This one costs no container and runs on both interpreters, so
+      # there is no argument for leaving the third engine's SQL unexecuted at
+      # publish time. Path-selected because the tier inherits the `integration`
+      # marker the step above deselects -- see the note in ci.yml.
+      - name: Pytest (DuckDB integration)
+        run: uv run pytest tests/integration/duckdb
+
   # This gate exists because the alternative is shipping SQL no server has ever
   # parsed. dataplat's db services are thousands of lines of generated SQL, and
   # the fake-cursor tests in `verify` prove only that execute() was called -- an

{dataplat-0.2.3 → dataplat-0.4.0}/CHANGELOG.md

@@ -1,5 +1,134 @@
 # Changelog
 
+## 0.4.0
+
+### Added
+
+- **DuckDB is a supported engine.** Set `<NAME>_ENGINE=duckdb` and
+  `<NAME>_PATH=/path/to.duckdb` (or `:memory:`), and install
+  `dataplat[duckdb]`. `<NAME>_DATABASE` works as a fallback for the path, and
+  `<NAME>_READ_ONLY=1` opens the file read-only.
+
+  DuckDB is not a smaller PostgreSQL — it runs inside the `dp` process, backed by
+  a file, with a single implicit user — so only the commands that mean something
+  there are available:
+
+  | command | DuckDB | why |
+  | --- | --- | --- |
+  | `db query` | works | |
+  | `db describe` | works | sizes and view definitions come from DuckDB's own catalog |
+  | `db top-tables` | works | sizes are DuckDB estimates, not comparable to PostgreSQL's |
+  | `db role *` | refused | there are no users or roles to describe |
+  | `db long-queries`, `db kill` | refused | in-process: there are no other sessions |
+  | `db dbt-orphans` | refused | it works by renaming, and DuckDB refuses to rename a relation a view depends on |
+
+  A refused command exits 2 and says which engine and why. These are properties
+  of the database, not gaps — the messages say so, because a user deserves to
+  know a thing can never work rather than assume it is coming.
+
+  `db describe` reports what DuckDB has no concept of — materialized views,
+  partitions, triggers, row-level security, privileges — as not applicable, with
+  the reason, rather than as empty sections that would read as "you have none".
+
+  Unlike PostgreSQL (needs a container) and Redshift (needs a cluster), the
+  DuckDB test tier needs nothing at all, so it never skips: this is the first
+  dialect with unconditional executing coverage in CI, and the second real-SQL
+  target the shared queries have ever been checked against.
+
+### Fixed
+
+- `dp config doctor` reported failures for a valid DuckDB target: it demanded
+  `<NAME>_HOST`, `<NAME>_USER` and `<NAME>_PASSWORD`, then failed the connection
+  check by dialing a server that does not exist. It now asks for the variables
+  the engine actually uses, and probes a DuckDB target by opening its file —
+  which is the health check.
+
+## 0.3.0
+
+### Changed
+
+- **Exit codes now say what went wrong.** Every failure used to be `1`, so a
+  wrapper script could not tell "your config is wrong" from "the warehouse is
+  down" — and only one of those is worth retrying. Typed failures now carry
+  their own code: `2` invalid input, `3` configuration, `4` authentication,
+  `5` external service. `0`, `1` and `2` keep their conventional meanings, and
+  `2` is deliberately shared with Click's own usage error, because
+  `--format nope` and `-t nosuchtarget` are one condition to the caller.
+
+  An unreachable warehouse exits `5`, since that is the retryable case. A bad
+  statement against a reachable server stays `1`: retrying a syntax error would
+  fail identically forever. Untyped failures and a declined confirmation also
+  stay `1`. The full table is in the README.
+
+  **Scripts that branch on a non-zero exit will see new numbers.** Anything
+  testing `== 1` for a config or auth problem needs updating.
+
+- **`dp db dbt-orphans` is more aggressive.** Its "which models are live" query
+  interpolated the dbt project name into a `LIKE` pattern without escaping it,
+  and dbt project names are snake_case — so the `_` in `my_project` matched any
+  character and a *sibling* project's models (`my2project`) sharing the
+  `dbt_artifacts` schema were counted as live. The same applied to
+  `DP_DBT_INVOCATION_COMMAND`, where `_` is ordinary in a dbt selector.
+
+  The direction is the point: an over-large live set makes **fewer** objects look
+  orphaned. Escaping it shrinks the live set, so dbt-orphans will now rename —
+  and after the grace period drop — objects it previously left alone. Run
+  `dp db dbt-orphans` (dry-run is the default) and read the plan before you
+  `--no-dry-run` the first time after upgrading.
+
+- **`dp status` runs its checks concurrently**: 40.7s to 10.4s with five
+  targets, one unreachable. Sections run in parallel and each database target is
+  probed in parallel within them, which is where the time actually went — a 10s
+  connect timeout per target was paid serially. Key order and section order are
+  unchanged, because both pools iterate the declared mapping rather than
+  completion order. The AWS section stays serial and last: it may hand the
+  terminal to an interactive `aws sso login`.
+
+### Added
+
+- **`--verbose` (or `DP_VERBOSE=1`) shows what dataplat actually sent** — SQL
+  statements, HTTP requests with status and duration, and AWS service calls.
+  It writes to stderr only, so `--json` and `--format csv` stay pipeable, and
+  everything passes through a redactor: passwords, secret values, bearer tokens
+  and API keys are never traced. Parameter values and response bodies are not
+  traced either — those are the data, not the request.
+
+- **Third-party command areas.** `dp` discovers areas declared in the
+  `dataplat.areas` entry-point group, so a package can add a command area
+  without a change here. Discovery reads only the entry-point metadata, never
+  imports the plugin, so `dp --version` and `dp --help` stay import-free and
+  fast. A plugin that fails to import warns on stderr and leaves the built-in
+  areas working; it cannot shadow a built-in area.
+
+- `dp config doctor` warns when a loaded `.envrc` value still contains an
+  unexpanded `$VAR`. dataplat does not run a shell, so
+  `export PGHOST=$DB_HOST` loads the literal text — which then surfaces as a
+  baffling connection failure rather than as the configuration mistake it is.
+
+- Shell completion is documented (`dp --install-completion`). It always worked;
+  the README never said so.
+
+### Fixed
+
+- `dp db query --format json|csv` could emit output that would not parse. The
+  progress spinner painted to stdout, which was invisible while Rich only did
+  that for a real terminal — the frames are erased — but `FORCE_COLOR` makes
+  Rich treat a pipe as a terminal too, and then the escape sequences ended up in
+  the redirected file. The spinner now follows the same sink as the notices.
+
+- `dp db describe` reported the owner's grant option two different ways
+  depending on whether you asked about a schema or a relation. PostgreSQL grants
+  an owner every grant option implicitly and never records it in the ACL, so
+  reading the ACL reported "cannot delegate" about a role that demonstrably can.
+  Schema privileges now agree with relation privileges.
+
+- The `Operating System :: OS Independent` classifier was an overclaim and has
+  been narrowed. Four things break on Windows: `dp config init` creates a
+  symlink, the dependency auto-install re-execs through `os.execvp`, the runner
+  commands shell out to `docker` with a POSIX default workdir, and the
+  credentials file is written with a `0o600` mode Windows ignores — after which
+  dataplat reports its own file as insecurely permissioned. CI tests Linux only.
+
 ## 0.2.3
 
 Redshift-only fixes. Nothing changes for PostgreSQL targets.

{dataplat-0.2.3 → dataplat-0.4.0}/CONTRIBUTING.md

@@ -19,8 +19,10 @@ uv run ruff format --check .
 uv run mypy dataplat
 ```
 
-`uv run pytest` is green without Docker: the database-backed tests skip. To run
-them, start a server and point the suite at it:
+`uv run pytest` is green without Docker: the PostgreSQL-backed tests skip. The
+DuckDB ones do not — there is no server for them to miss, so they always run
+(see below). To exercise the PostgreSQL half, start a server and point the suite
+at it:
 
 ```bash
 docker run -d --name dp-pg-test \
@@ -30,7 +32,7 @@ docker exec dp-pg-test psql -U postgres -d dataplat_test \
   -c 'CREATE EXTENSION IF NOT EXISTS pg_stat_statements'
 
 DP_TEST_PG_REQUIRED=1 uv run pytest              # everything
-uv run pytest -m "not integration"               # skip the database half
+uv run pytest -m "not integration"               # deselect the real-database tiers
 docker rm -f -v dp-pg-test                       # -v, or the volume dangles
 ```
 
@@ -40,6 +42,40 @@ and still report success.
 
 Commits follow [Conventional Commits](https://www.conventionalcommits.org/).
 
+## Testing against DuckDB
+
+There is nothing to provision. DuckDB is a library that opens a file inside the
+test process, so the fixtures in `tests/integration/duckdb/` build a throwaway
+database in pytest's `tmp_path`:
+
+```bash
+uv run pytest tests/integration/duckdb        # ~1s, no server, no environment
+```
+
+This tier deliberately has **none** of the availability machinery the other two
+have: no DSN variable, no `_REQUIRED` switch, no disposability gate, no
+read-only guard, and no skip path. Its `conftest.py` argues each omission at
+length; the short version is that `DP_TEST_*_REQUIRED` exists because a
+silently-skipping tier reports green having executed none of the SQL it was
+written to validate — and here there is nothing that *can* be unavailable. The
+one thing that can be missing is the `duckdb` package, which is a broken dev
+environment rather than absent infrastructure (`uv sync --group dev
+--all-extras`), so it is a hard collection error carrying the install hint,
+never a skip.
+
+It carries the `integration` marker, because everything under
+`tests/integration/` does — that is the safety net which keeps
+`-m 'not integration'` airtight. Select it by path, as above, when you want it
+without the PostgreSQL container.
+
+Isolation is a fresh database file per test rather than the PostgreSQL harness's
+BEGIN/ROLLBACK. DuckDB *does* roll back DDL, so the trick would work
+mechanically — but a table created inside an open transaction reports
+`estimated_size = 0`, and `pragma_database_size()` reports `0 bytes` until a
+`CHECKPOINT` that cannot run in a writing transaction, so every row-estimate and
+size assertion would have passed against zeros. Both facts are pinned by tests,
+so the decision can be re-checked instead of re-derived.
+
 ## Testing against a real Redshift cluster
 
 Redshift is a managed service, so there is no container and CI cannot cover it.
@@ -108,9 +144,58 @@ has taught nobody anything.
 
 ## Dialect changes: what counts as evidence
 
-`dataplat/services/db` targets PostgreSQL and Redshift. PostgreSQL has a real
-integration suite behind it. Redshift has none and cannot get one cheaply — it
-is a managed service, so there is no container to run in CI.
+`dataplat/services/db` targets three engines, and they are not equally knowable:
+
+| Engine | SQL really executed? | What it costs to find out |
+| --- | --- | --- |
+| PostgreSQL | yes, in CI | a container |
+| DuckDB | yes, in CI — and its tier cannot skip | nothing: a library and a temp file |
+| Redshift | **no** — generated and asserted, never run | a cluster you own; CI cannot have one |
+
+That asymmetry is the whole reason this section exists, and it is now lopsided in
+one direction only. PostgreSQL has a real integration suite behind it. So does
+DuckDB, and DuckDB is the cheap case: there is no server to be unreachable and
+nothing to provision, so a claim about it can always be checked *before* it is
+made. Redshift has none and cannot get one cheaply — it is a managed service, so
+there is no container to run in CI.
+
+Which *commands* an engine refuses, and why, is declared in one place:
+`dataplat/services/db/capabilities.py` — one entry per engine with no defaults,
+so adding an engine and forgetting what it can do is a construction error there
+rather than a wrong answer somewhere downstream. Change what an engine can be
+asked there and nowhere else, and cite the code or the probe the entry rests on,
+exactly as the existing ones do.
+
+A command that still runs but has to leave a *section* out is the other case, and
+it declares that next to the section (`services/db/describe.py`'s
+`NotApplicable`) and prints it — `dp db describe` on DuckDB ends with the list of
+sections the engine has no concept of, with a reason each. Either way the reason
+is written once, next to the fact, and a refusal never says "not implemented"
+about something the engine simply is not.
+
+### DuckDB: the rules apply, and the excuse does not
+
+The evidence classes below apply to DuckDB unchanged in principle — same
+ordering, same requirement to record in the code *why* a line is the way it is.
+In practice you will never reach past the first one, because the strongest class
+(0: confirmed against a real engine) is always available: `uv run pytest
+tests/integration/duckdb` runs in about a second, needs no server, no
+credentials and no container, and cannot skip.
+
+So, plainly: **"I could not test it" is never an acceptable reason for a DuckDB
+change.** If a DuckDB behaviour is in doubt, open a connection and find out —
+then turn the probe into a test in `tests/integration/duckdb/`, so the next
+person inherits the answer instead of the doubt. A comment recording an
+*unverified* DuckDB assumption is not a legitimate outcome here the way it is for
+Redshift; it is a probe someone declined to run.
+
+The corollary applies to the capability matrix too. Every DuckDB `False` in
+`capabilities.py` names something the engine does not have, and
+`tests/integration/duckdb/test_duckdb_services.py` asserts the declaration
+*alongside the missing catalog* — so a future DuckDB that grows `pg_roles` fails
+the suite instead of quietly going on being refused.
+
+### Redshift: weighing evidence without a server
 
 For a while the rule was simply "don't touch SQL that runs on Redshift, because
 you can't test it." That is a good instinct and a bad rule. Applied literally it
@@ -159,10 +244,20 @@ comment is what lets the next person re-evaluate instead of rediscovering.
 
 - **Keep the engine constants split.** Never edit a `_*_SQL_REDSHIFT` constant
   to fix a PostgreSQL bug. If a shared statement needs to diverge, split it and
-  leave the Redshift half byte-for-byte as it was.
+  leave the Redshift half byte-for-byte as it was. The DuckDB statements are
+  split the same way (`_*_SQL_DUCKDB`, `_DUCKDB_*_SQL`), with one difference: you
+  can run them, so a DuckDB constant you touched comes back through
+  `tests/integration/duckdb/`, not through inspection.
 - **Add a fake-cursor test** asserting what the Redshift branch emits. It is the
   only mechanism that covers that path at all, and it catches the common
-  accident of "fixed both branches when I meant one".
+  accident of "fixed both branches when I meant one". DuckDB's equivalent is an
+  *executing* test — a fake cursor there proves less than the real engine does,
+  for the same effort.
+- **Do not translate placeholders between engines.** psycopg binds `%s`; DuckDB
+  binds `?`. Rewriting one into the other means deciding which `%` in a
+  statement is a literal, and getting that wrong changes the SQL — so
+  `DuckDbCursor` passes statements through untouched, and DuckDB SQL is written
+  for DuckDB.
 - **Record the evidence in the code**, not just the commit message. A future
   reader deciding whether they may touch the line needs to see why it is the way
   it is.
@@ -175,3 +270,16 @@ never be NULL. A report that states something false is worse than one that
 admits a gap — especially a report someone is using for an audit. When the
 server will not tell you, say so, and say why in the same breath: a bare
 "unknown" reads as a tool defect.
+
+`dp db top-tables` on DuckDB is the same rule applied to a whole column, and it
+shows where the rule ends. DuckDB has no per-relation size function and no
+catalog column carrying bytes, so every `size_bytes` there is `None` — never `0`,
+because a table whose size cannot be read is not an empty table, and `--json`
+consumers have to be able to tell those apart.
+
+The table rendering then goes one step further and *drops* the Size and
+"% of disk" columns instead of filling them with `—`, printing the basis for the
+numbers it does show. Withdrawing a claim is not enough on its own when the gap
+is an entire column: a report of dashes under a `0 B (0.0% of disk)` footer reads
+as a broken tool, not as the engine's answer. Say what you cannot know, then
+show what you can.