thunderduck-sqlalchemy 0.1.0__tar.gz

thunderduck_sqlalchemy-0.1.0/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.3
+Name: thunderduck-sqlalchemy
+Version: 0.1.0
+Summary: SQLAlchemy dialect and DBAPI for thunderduck.io — query your lakehouse from Superset, Airflow, pandas and friends.
+Keywords: sqlalchemy,duckdb,iceberg,lakehouse,thunderduck,superset
+Author: YILDIRIM ADIGUZEL
+Author-email: YILDIRIM ADIGUZEL <yadiguzel@gmail.com>
+License: Apache-2.0
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Database :: Front-Ends
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: sqlalchemy>=1.4
+Requires-Python: >=3.10
+Project-URL: Homepage, https://thunderduck.io
+Project-URL: Source, https://github.com/212data/thunderduck
+Description-Content-Type: text/markdown
+
+# thunderduck-sqlalchemy
+
+A SQLAlchemy dialect and PEP 249 DBAPI for [thunderduck](https://thunderduck.io).
+Query your lakehouse from anything that speaks SQLAlchemy — Superset, Airflow,
+Dagster, Redash, pandas, Jupyter, Streamlit.
+
+```bash
+pip install thunderduck-sqlalchemy
+```
+
+Works with SQLAlchemy 1.4 and 2.x, on Python 3.10+ — so it installs cleanly
+into Apache Superset (which pins SQLAlchemy 1.4) as well as modern Airflow,
+Dagster and pandas environments.
+
+## Connect
+
+Create an API token in the thunderduck console (**Settings → API Tokens**; it
+starts with `tdk_` and is shown once), then:
+
+```python
+from sqlalchemy import create_engine, text
+from sqlalchemy.pool import NullPool
+
+engine = create_engine(
+    "thunderduck://:tdk_your_token@api.thunderduck.io/",
+    poolclass=NullPool,
+)
+
+with engine.connect() as conn:
+    for row in conn.execute(text('SELECT * FROM "lego"."public"."lego_sets" LIMIT 10')):
+        print(row)
+```
+
+pandas works directly:
+
+```python
+import pandas as pd
+
+df = pd.read_sql('SELECT * FROM "thunder_duck_demo"."iris"', engine)
+```
+
+### URL options
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| password / `?token=` | — | Required. Your `tdk_…` API token. |
+| `?ssl=false` | `true` | Use plain HTTP. For in-cluster use against the console-api Service. |
+| `?poll_ms=` | `1000` | How often to poll for completion. |
+| `?timeout_ms=` | `300000` | How long to wait for a query. |
+| `?arraysize=` | `1000` | Rows fetched per page. |
+
+## Table names
+
+thunderduck names are `catalog.schema.table` (flat-file catalogs) or
+`catalog.table` (Iceberg/Nessie catalogs). SQLAlchemy models two levels, so
+**the schema is the whole dotted prefix**:
+
+```python
+from sqlalchemy import MetaData, Table
+
+md = MetaData()
+sets = Table("lego_sets", md, schema="lego.public", autoload_with=engine)  # 3-level
+iris = Table("iris", md, schema="thunder_duck_demo", autoload_with=engine)  # 2-level
+```
+
+Reflection (`inspect(engine).get_schema_names()` etc.) reads thunderduck's
+catalog API rather than issuing SQL, so browsing schemas is fast and free.
+
+## Apache Superset
+
+Installing the package is all that is needed — it registers a Superset DB
+engine spec automatically. Add a database of type **Other** with:
+
+```
+thunderduck://:tdk_your_token@api.thunderduck.io/
+```
+
+Superset's stop button genuinely cancels the running query.
+
+## How it works, and what follows from that
+
+Every statement is submitted to thunderduck's REST API, which runs it as an
+isolated Kubernetes Job; results are then paged back over HTTP. That model has
+consequences worth knowing up front:
+
+- **Latency.** Expect seconds, not milliseconds, before the first row. Built
+  for analytics and BI, not for tight interactive loops.
+- **Read-only.** No DML, no DDL, no transactions. `commit()` and `rollback()`
+  are no-ops.
+- **Use `NullPool`.** A connection holds no server-side state, so pooling buys
+  nothing. The driver also never issues a `SELECT 1` health check, because
+  that would start a whole Job.
+- **Parameters are rendered client-side.** thunderduck's API accepts SQL only,
+  so this driver renders bound parameters into the statement, escaping them
+  strictly and refusing types it does not recognise. Always pass untrusted
+  values as parameters — never build SQL by string concatenation. The
+  paramstyle is `pyformat`, so — as with psycopg2 — a literal percent sign in
+  hand-written SQL must be escaped as `%%` when you pass parameters.
+- **Row ceiling.** The server caps how many rows one page returns and how many
+  a result set stores. Very large result sets are truncated; aggregate in SQL
+  rather than pulling raw rows.
+
+## Also available
+
+For JVM tools (DBeaver, DataGrip) there is a JDBC driver — see the
+`jdbc-driver/` directory of the thunderduck repository.
+
+## Development
+
+```bash
+uv sync --dev
+uv run pytest
+uv run ruff check . && uv run ruff format --check .
+```
+
+One gotcha: the committed `uv.lock` resolves SQLAlchemy 2.x, so a plain
+`uv run pytest` only exercises **one** of the two supported majors. Apache
+Superset pins `sqlalchemy<2`, so check that combination too before releasing:
+
+```bash
+uv venv --python 3.10 .sa14
+uv pip install --python .sa14 -e . "sqlalchemy<2" pytest
+.sa14/bin/python -c "import sqlalchemy; print('SQLALCHEMY', sqlalchemy.__version__)"
+.sa14/bin/pytest -q
+```
+
+CI runs both legs on every push.
+
+## License
+
+Apache-2.0

thunderduck_sqlalchemy-0.1.0/README.md

@@ -0,0 +1,131 @@
+# thunderduck-sqlalchemy
+
+A SQLAlchemy dialect and PEP 249 DBAPI for [thunderduck](https://thunderduck.io).
+Query your lakehouse from anything that speaks SQLAlchemy — Superset, Airflow,
+Dagster, Redash, pandas, Jupyter, Streamlit.
+
+```bash
+pip install thunderduck-sqlalchemy
+```
+
+Works with SQLAlchemy 1.4 and 2.x, on Python 3.10+ — so it installs cleanly
+into Apache Superset (which pins SQLAlchemy 1.4) as well as modern Airflow,
+Dagster and pandas environments.
+
+## Connect
+
+Create an API token in the thunderduck console (**Settings → API Tokens**; it
+starts with `tdk_` and is shown once), then:
+
+```python
+from sqlalchemy import create_engine, text
+from sqlalchemy.pool import NullPool
+
+engine = create_engine(
+    "thunderduck://:tdk_your_token@api.thunderduck.io/",
+    poolclass=NullPool,
+)
+
+with engine.connect() as conn:
+    for row in conn.execute(text('SELECT * FROM "lego"."public"."lego_sets" LIMIT 10')):
+        print(row)
+```
+
+pandas works directly:
+
+```python
+import pandas as pd
+
+df = pd.read_sql('SELECT * FROM "thunder_duck_demo"."iris"', engine)
+```
+
+### URL options
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| password / `?token=` | — | Required. Your `tdk_…` API token. |
+| `?ssl=false` | `true` | Use plain HTTP. For in-cluster use against the console-api Service. |
+| `?poll_ms=` | `1000` | How often to poll for completion. |
+| `?timeout_ms=` | `300000` | How long to wait for a query. |
+| `?arraysize=` | `1000` | Rows fetched per page. |
+
+## Table names
+
+thunderduck names are `catalog.schema.table` (flat-file catalogs) or
+`catalog.table` (Iceberg/Nessie catalogs). SQLAlchemy models two levels, so
+**the schema is the whole dotted prefix**:
+
+```python
+from sqlalchemy import MetaData, Table
+
+md = MetaData()
+sets = Table("lego_sets", md, schema="lego.public", autoload_with=engine)  # 3-level
+iris = Table("iris", md, schema="thunder_duck_demo", autoload_with=engine)  # 2-level
+```
+
+Reflection (`inspect(engine).get_schema_names()` etc.) reads thunderduck's
+catalog API rather than issuing SQL, so browsing schemas is fast and free.
+
+## Apache Superset
+
+Installing the package is all that is needed — it registers a Superset DB
+engine spec automatically. Add a database of type **Other** with:
+
+```
+thunderduck://:tdk_your_token@api.thunderduck.io/
+```
+
+Superset's stop button genuinely cancels the running query.
+
+## How it works, and what follows from that
+
+Every statement is submitted to thunderduck's REST API, which runs it as an
+isolated Kubernetes Job; results are then paged back over HTTP. That model has
+consequences worth knowing up front:
+
+- **Latency.** Expect seconds, not milliseconds, before the first row. Built
+  for analytics and BI, not for tight interactive loops.
+- **Read-only.** No DML, no DDL, no transactions. `commit()` and `rollback()`
+  are no-ops.
+- **Use `NullPool`.** A connection holds no server-side state, so pooling buys
+  nothing. The driver also never issues a `SELECT 1` health check, because
+  that would start a whole Job.
+- **Parameters are rendered client-side.** thunderduck's API accepts SQL only,
+  so this driver renders bound parameters into the statement, escaping them
+  strictly and refusing types it does not recognise. Always pass untrusted
+  values as parameters — never build SQL by string concatenation. The
+  paramstyle is `pyformat`, so — as with psycopg2 — a literal percent sign in
+  hand-written SQL must be escaped as `%%` when you pass parameters.
+- **Row ceiling.** The server caps how many rows one page returns and how many
+  a result set stores. Very large result sets are truncated; aggregate in SQL
+  rather than pulling raw rows.
+
+## Also available
+
+For JVM tools (DBeaver, DataGrip) there is a JDBC driver — see the
+`jdbc-driver/` directory of the thunderduck repository.
+
+## Development
+
+```bash
+uv sync --dev
+uv run pytest
+uv run ruff check . && uv run ruff format --check .
+```
+
+One gotcha: the committed `uv.lock` resolves SQLAlchemy 2.x, so a plain
+`uv run pytest` only exercises **one** of the two supported majors. Apache
+Superset pins `sqlalchemy<2`, so check that combination too before releasing:
+
+```bash
+uv venv --python 3.10 .sa14
+uv pip install --python .sa14 -e . "sqlalchemy<2" pytest
+.sa14/bin/python -c "import sqlalchemy; print('SQLALCHEMY', sqlalchemy.__version__)"
+.sa14/bin/pytest -q
+```
+
+CI runs both legs on every push.
+
+## License
+
+Apache-2.0

thunderduck_sqlalchemy-0.1.0/pyproject.toml

@@ -0,0 +1,70 @@
+[project]
+name = "thunderduck-sqlalchemy"
+version = "0.1.0"
+description = "SQLAlchemy dialect and DBAPI for thunderduck.io — query your lakehouse from Superset, Airflow, pandas and friends."
+readme = "README.md"
+requires-python = ">=3.10"
+keywords = [
+    "sqlalchemy",
+    "duckdb",
+    "iceberg",
+    "lakehouse",
+    "thunderduck",
+    "superset",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Database",
+    "Topic :: Database :: Front-Ends",
+]
+dependencies = [
+    "httpx>=0.27.0",
+    "sqlalchemy>=1.4",
+]
+
+[[project.authors]]
+name = "YILDIRIM ADIGUZEL"
+email = "yadiguzel@gmail.com"
+
+[project.license]
+text = "Apache-2.0"
+
+[project.urls]
+Homepage = "https://thunderduck.io"
+Source = "https://github.com/212data/thunderduck"
+
+[project.entry-points."sqlalchemy.dialects"]
+thunderduck = "thunderduck_sqlalchemy.dialect:ThunderduckDialect"
+"thunderduck.rest" = "thunderduck_sqlalchemy.dialect:ThunderduckDialect"
+
+[project.entry-points."superset.db_engine_specs"]
+thunderduck = "thunderduck_sqlalchemy.superset_spec:ThunderduckEngineSpec"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0.0",
+    "ruff>=0.8.0",
+]
+
+[build-system]
+requires = ["uv_build>=0.12.1,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = [
+    "E",
+    "F",
+    "I",
+    "UP",
+]

thunderduck_sqlalchemy-0.1.0/pyproject.toml.orig

@@ -0,0 +1,62 @@
+[project]
+name = "thunderduck-sqlalchemy"
+version = "0.1.0"
+description = "SQLAlchemy dialect and DBAPI for thunderduck.io — query your lakehouse from Superset, Airflow, pandas and friends."
+readme = "README.md"
+authors = [
+    { name = "YILDIRIM ADIGUZEL", email = "yadiguzel@gmail.com" }
+]
+requires-python = ">=3.10"
+license = { text = "Apache-2.0" }
+keywords = ["sqlalchemy", "duckdb", "iceberg", "lakehouse", "thunderduck", "superset"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Database",
+    "Topic :: Database :: Front-Ends",
+]
+# sqlalchemy>=1.4, NOT >=2.0: apache-superset 6.1.0 pins sqlalchemy<2, so a 2.0
+# floor would make this package uninstallable in the Superset image. See the
+# plan's Global Constraints.
+dependencies = [
+    "httpx>=0.27.0",
+    "sqlalchemy>=1.4",
+]
+
+# Deliberately NO [project.optional-dependencies]: an `apache-superset` extra
+# cannot be resolved alongside this package (its sqlalchemy<2 pin conflicts),
+# and Superset supplies itself at runtime anyway.
+
+[project.urls]
+Homepage = "https://thunderduck.io"
+Source = "https://github.com/212data/thunderduck"
+
+[project.entry-points."sqlalchemy.dialects"]
+thunderduck = "thunderduck_sqlalchemy.dialect:ThunderduckDialect"
+"thunderduck.rest" = "thunderduck_sqlalchemy.dialect:ThunderduckDialect"
+
+[project.entry-points."superset.db_engine_specs"]
+thunderduck = "thunderduck_sqlalchemy.superset_spec:ThunderduckEngineSpec"
+
+[dependency-groups]
+dev = [
+    "pytest>=8.0.0",
+    "ruff>=0.8.0",
+]
+
+[build-system]
+requires = ["uv_build>=0.12.1,<0.13.0"]
+build-backend = "uv_build"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP"]

thunderduck_sqlalchemy-0.1.0/src/thunderduck_sqlalchemy/__init__.py

@@ -0,0 +1,13 @@
+"""SQLAlchemy dialect and DBAPI for thunderduck.io.
+
+Read-only by design: every statement is submitted to console-api, which runs
+it as a Kubernetes Job, and results are paged back over HTTP. See README.md
+for the connection URL format and the caveats that follow from that model.
+"""
+
+from .dbapi import connect
+from .dialect import ThunderduckDialect
+
+__version__ = "0.1.0"
+
+__all__ = ["__version__", "connect", "ThunderduckDialect"]

thunderduck_sqlalchemy-0.1.0/src/thunderduck_sqlalchemy/catalog.py

@@ -0,0 +1,76 @@
+"""Flatten console-api's `/catalog/schema` tree into `(schema, table)` pairs.
+
+## Two catalog shapes, one rule
+
+thunderduck names are not uniformly three-part. Nessie catalogs are 2-level
+(`catalog.table`) and put their tables directly under the catalog; flat-file
+catalogs are 3-level (`catalog.schema.table`) and put them under named
+namespaces. SQLAlchemy and Superset both model exactly two levels.
+
+The rule that reconciles them: **the composite `schema` is the whole dotted
+prefix and the `table` is the leaf.** So `iris` in the nessie catalog
+`thunder_duck_demo` has schema `"thunder_duck_demo"`, while `lego_sets` has
+schema `"lego.public"`. The dialect's identifier preparer later splits that
+prefix back apart and quotes each segment, so both compile correctly.
+
+A catalog carrying an `error` (console-api could not introspect it) is skipped
+rather than raised on: one unreachable catalog must not break reflection of
+every other one.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+
+@dataclass(frozen=True)
+class TableRef:
+    schema: str
+    table: str
+    kind: str = "table"
+    columns: list[dict[str, str]] = field(default_factory=list)
+
+
+def flatten(tree: dict[str, Any] | None) -> list[TableRef]:
+    refs: list[TableRef] = []
+    for catalog in (tree or {}).get("catalogs") or []:
+        if catalog.get("error"):
+            continue
+        catalog_name = catalog.get("name")
+        if not catalog_name:
+            continue
+        for table in catalog.get("tables") or []:
+            ref = _make_ref(catalog_name, table)
+            if ref:
+                refs.append(ref)
+        for namespace in catalog.get("namespaces") or []:
+            namespace_name = namespace.get("name")
+            if not namespace_name:
+                continue
+            schema = f"{catalog_name}.{namespace_name}"
+            for table in namespace.get("tables") or []:
+                ref = _make_ref(schema, table)
+                if ref:
+                    refs.append(ref)
+    return refs
+
+
+def _make_ref(schema: str, table: dict[str, Any]) -> TableRef | None:
+    name = table.get("name")
+    if not name:
+        return None
+    kind = str(table.get("kind") or "table").lower()
+    columns = [
+        {"name": str(c.get("name")), "type": str(c.get("type") or "")}
+        for c in (table.get("columns") or [])
+        if c.get("name")
+    ]
+    return TableRef(schema=schema, table=str(name), kind=kind, columns=columns)
+
+
+def split_schema(schema: str) -> list[str]:
+    """Split a composite schema into its segments (`"lego.public"` -> 2)."""
+    if not schema:
+        return []
+    return schema.split(".")