hotdata-ibis 0.1.1__py3-none-any.whl

hotdata_ibis-0.1.1.dist-info/METADATA

@@ -0,0 +1,176 @@
+Metadata-Version: 2.4
+Name: hotdata-ibis
+Version: 0.1.1
+Summary: Ibis backend for Hotdata federated SQL API (depends on the hotdata SDK only; not hotdata-runtime)
+Author: Hotdata Ibis contributors
+License: Apache-2.0
+Project-URL: Documentation, https://www.hotdata.dev/docs/api-reference
+Project-URL: Ibis, https://ibis-project.org/
+Keywords: ibis,hotdata,sql,dataframe
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: ibis-framework<11,>=10.0
+Requires-Dist: hotdata>=0.2.0
+Requires-Dist: pyarrow>=15
+Requires-Dist: pyarrow-hotfix>=0.6
+Requires-Dist: pandas>=2
+Requires-Dist: sqlglot>=24
+
+# hotdata-ibis
+
+Experimental [Ibis](https://ibis-project.org/) backend for [Hotdata](https://www.hotdata.dev/docs/api-reference): compile expressions with Ibis, run federated SQL over the Hotdata API. REST calls use the official **[hotdata](https://github.com/hotdata-dev/sdk-python)** Python SDK. Repo examples use **httpx** (listed under the **dev** dependency group).
+
+**Requirements:** Python 3.10+, **ibis-framework** 10.x, **hotdata** ≥0.2.
+
+## Install
+
+```bash
+uv pip install hotdata-ibis
+# or: python -m pip install hotdata-ibis
+```
+
+## Features
+
+- **Ibis connection API** — connect with `ibis.hotdata.connect(...)` or `ibis.connect("hotdata://...")`.
+- **Hotdata catalog mapping** — expose Hotdata connections, schemas, and tables through Ibis catalogs, databases, and tables.
+- **SQL-backed expression execution** — compile Ibis expressions with the Postgres SQLGlot compiler and execute them through Hotdata query APIs.
+- **Typed table discovery** — load schema metadata from Hotdata information schema and map SQL types into Ibis types.
+- **Arrow and pandas results** — materialize expressions as pandas DataFrames, PyArrow tables, or local Arrow record batches.
+- **Raw SQL escape hatch** — use `con.sql(..., dialect="postgres")` when Hotdata-specific federated SQL is clearer than modeled Ibis expressions.
+- **Managed database writes** — create managed connections with `create_database`, load local pandas or PyArrow data through `create_table`, and clean up with `drop_table` / `drop_database`.
+
+## Connect
+
+Programmatic API:
+
+```python
+import ibis
+
+con = ibis.hotdata.connect(
+    api_url="https://api.hotdata.dev",
+    token="YOUR_API_TOKEN",
+    workspace_id="ws_…",
+    session_id=None,       # optional: X-Session-Id (sandbox)
+    verify_ssl=True,
+    timeout=120.0,
+    default_connection=None,  # Hotdata connection id → Ibis catalog
+    default_schema=None,      # remote schema → Ibis database
+    poll_interval_s=0.25,
+    poll_timeout_s=600.0,
+)
+```
+
+URL style (token may live in the query string or the URL “password” segment):
+
+```python
+con = ibis.connect(
+    "hotdata://api.hotdata.dev/?token=…&workspace_id=ws_…&verify_ssl=true"
+)
+```
+
+**Mapping:** Ibis **catalog** = Hotdata connection id; **database** = remote schema; **table** = table name. SQL references look like `connection.schema.table`. With a single connection and schema, defaults are inferred; otherwise set `default_connection` / `default_schema` or qualify `con.table(..., database=(conn_id, schema))`.
+
+**Execution:** SQL is compiled with Ibis’s **Postgres** SQLGlot compiler. The client submits queries asynchronously with `POST /v1/query`, polls `GET /v1/query-runs/{id}`, then downloads ready results as Arrow IPC from `GET /v1/results/{id}`. Tuning: `poll_interval_s`, `poll_timeout_s` on `connect()`.
+
+**Types:** Typed tables come from Hotdata’s information schema. `con.sql(...)` types are inferred from a small preview query and Arrow schema; see [Hotdata SQL](https://www.hotdata.dev/docs/sql) for server behavior.
+
+## Ibis Support Overview
+
+`hotdata-ibis` is a read-oriented SQL backend. It is useful for exploring Hotdata workspaces with Ibis expressions, running federated SQL, and materializing results locally, but it is not a full mutable database backend.
+
+Supported today:
+
+- **Connection setup:** `ibis.hotdata.connect(...)` and `ibis.connect("hotdata://...")` with token, workspace, optional sandbox session, TLS, timeout, and polling settings.
+- **Catalog discovery:** `list_catalogs`, `list_databases`, `list_tables`, `current_catalog`, and `current_database` map Hotdata connections and remote schemas into Ibis' catalog/database/table hierarchy.
+- **Table schemas:** `con.table(...)` uses Hotdata information schema column metadata and maps SQL types through Ibis' Postgres type parser.
+- **SQL-backed expressions:** Ibis expressions compile with the Postgres SQLGlot compiler and execute through Hotdata. Common `SELECT` workloads such as projection, filtering, joins, grouping, aggregation, ordering, limits, scalar expressions, and `con.sql(...)` work when the generated SQL is accepted by Hotdata.
+- **Result materialization:** `.execute()` returns pandas objects. `.to_pyarrow()` and `.to_pyarrow_batches()` use the Arrow IPC result data exposed by Hotdata without converting through JSON rows; batches are split locally after the result is downloaded.
+- **Raw SQL escape hatch:** `con.sql("SELECT ...", dialect="postgres")` is the most reliable way to use Hotdata-specific federated table names or SQL that Ibis does not model directly.
+- **Managed database lifecycle:** `create_database("sales", schema="public", tables=["orders"])` registers a managed connection (Ibis catalog). `create_table("orders", pandas_df, database=("sales", "public"))` uploads Parquet and loads it with replace mode. Query as `sales.public.orders` in SQL. `drop_table` clears a managed table; `drop_database` deletes the connection.
+- **Parquet uploads:** `create_table` accepts pandas DataFrames, PyArrow tables, or schema-only empty tables. Tables must live in a managed connection — declare them with `create_database(..., tables=[...])` first. Loads always use replace mode; pass `overwrite=True` to replace an existing synced table (the default `overwrite=False` raises if the table already exists).
+
+Not supported as full Ibis backend features:
+
+- **General DDL and mutations:** Arbitrary remote DDL, inserts, updates, deletes, and schema-altering operations on external connections are not implemented. Managed-database writes are limited to `create_database`, `create_table`, `drop_table`, and `drop_database` as described above.
+- **Temporary tables and in-memory registration:** `supports_temporary_tables` is false, and in-memory tables are not uploaded automatically for joins.
+- **Python UDFs:** `supports_python_udfs` is false.
+- **Transactions and sessions as database state:** Hotdata sandbox sessions can be passed as `session_id`, but the backend does not expose transaction APIs.
+- **Backend-native SQL dialect:** Compilation uses Ibis' Postgres dialect as the closest fit. Hotdata SQL and federation rules are authoritative, so not every Ibis expression that compiles is guaranteed to execute remotely.
+- **Complete Ibis compliance:** The backend is experimental and has focused test coverage for connection, discovery, schema mapping, execution, uploads, and Arrow results. It has not yet been validated against the full Ibis backend test suite.
+- **Hotdata platform APIs beyond SQL and managed databases:** embeddings, indexes, query history management, sandbox lifecycle management, and other Hotdata-specific APIs are outside the Ibis backend surface.
+
+## Development
+
+```bash
+uv sync   # installs dev group by default (pytest, ruff, httpx for examples)
+uv run pytest
+uv run ruff check src tests examples
+```
+
+Lockfile CI: `uv sync --locked && uv run pytest`.
+
+## TPC-H for the examples
+
+Examples assume something like **`tpch.tpch_sf1.customer`**. Provision TPC-H in your workspace (commonly a **DuckDB** connection, then DuckDB’s `tpch` extension and `CALL dbgen(sf = 1)` — see [DuckDB TPC-H](https://www.duckdb.org/docs/current/core_extensions/tpch.html) and [Hotdata Quick Start](https://www.hotdata.dev/docs/quick-start)). If your data lives under `main` instead, pass `--default-schema` / `--default-connection` or set `HOTDATA_DEFAULT_*` (see `examples/_helpers.py`).
+
+## Examples
+
+Needs `HOTDATA_API_KEY` and `HOTDATA_WORKSPACE`.
+
+```bash
+uv sync
+export HOTDATA_API_KEY=…
+export HOTDATA_WORKSPACE=…
+uv run python examples/01_catalog_introspection.py
+uv run python examples/02_execute_sql.py 'SELECT COUNT(*) AS n FROM tpch.tpch_sf1.customer'
+uv run python examples/03_connect_via_url.py
+uv run python examples/04_ibis_table_workflows.py
+```
+
+### Ibis tables → pandas DataFrames
+
+Calling **`.execute()`** on a table expression runs the compiled SQL on Hotdata and returns a **pandas** `DataFrame` (Ibis’s default for this backend).
+
+Hotdata’s SQL often uses a **federated prefix** (for example `tpch.tpch_sf1`) that may not match the Ibis **catalog** string (the connection id). A reliable pattern is to start from **`con.sql("SELECT * FROM tpch.tpch_sf1.mytable", dialect="postgres")`**, then chain filters and aggregates—see **`examples/04_ibis_table_workflows.py`**.
+
+When **`con.table("mytable")`** is enough (single connection/schema and names align with compiled SQL), the same operations apply:
+
+```python
+t = con.table("customer")  # or con.table("customer", database=(conn_id, "tpch_sf1"))
+
+df = (
+    t.filter(t.c_mktsegment == "AUTOMOBILE")
+    .select("c_custkey", "c_name")
+    .limit(100)
+    .execute()
+)
+
+by_seg = t.group_by(t.c_mktsegment).agg(n=t.count()).execute()
+
+o = con.table("orders")
+orders_with_names = (
+    t.join(o, t.c_custkey == o.o_custkey)
+    .select(t.c_name, o.o_totalprice)
+    .limit(50)
+    .execute()
+)
+
+total = t.c_acctbal.sum().execute()
+```
+
+Other useful paths: **`.to_pyarrow()`** / **`.to_pyarrow_batches()`** for Arrow; **`con.sql("SELECT …", dialect="postgres")`** then chain the returned table expression.
+
+## References
+
+- [Hotdata Python SDK](https://github.com/hotdata-dev/sdk-python)
+- [Hotdata API](https://www.hotdata.dev/docs/api-reference) · [Hotdata SQL](https://www.hotdata.dev/docs/sql)
+- [Ibis](https://ibis-project.org/) · [Ibis backend hierarchy](https://ibis-project.org/concepts/backend-table-hierarchy.qmd)

hotdata_ibis-0.1.1.dist-info/RECORD

@@ -0,0 +1,10 @@
+ibis_hotdata/__init__.py,sha256=oFalV5QK1XhHut0zCNaADMTR7eioubYVTJPpVqhFrng,252
+ibis_hotdata/backend.py,sha256=DlzQpE76iBJElDpKIJliytk6iqw9eB8k5g2Fi8_VyVw,24394
+ibis_hotdata/http.py,sha256=W2UuMq56RFV3GA13oNOSuJNHeczV3MlenqMt3sYoksI,10251
+ibis_hotdata/managed.py,sha256=B289vMWtPdrEZ5U4JCGi7qb953SSOBmz4fErtVYzbhk,428
+ibis_hotdata/types.py,sha256=_vuVB2gooJ9BCbvxjswp-PqMCraAYVji744EP_wb2qI,636
+hotdata_ibis-0.1.1.dist-info/METADATA,sha256=m3hn7g9ectyDTsT1z88dT333bTtPXMR3E4BKaelyXbE,10538
+hotdata_ibis-0.1.1.dist-info/WHEEL,sha256=aeYiig01lYGDzBgS8HxWXOg3uV61G9ijOsup-k9o1sk,91
+hotdata_ibis-0.1.1.dist-info/entry_points.txt,sha256=tPFpKl3RYEJj2XvxtUMMcvL_Bq1f-HlTn_drbKFECpg,39
+hotdata_ibis-0.1.1.dist-info/top_level.txt,sha256=zV4cow300tmsMSssOOfS3nrlS35lhG_ZX3aNsFJ5hvY,13
+hotdata_ibis-0.1.1.dist-info/RECORD,,

hotdata_ibis-0.1.1.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (82.0.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

hotdata_ibis-0.1.1.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[ibis.backends]
+hotdata = ibis_hotdata

hotdata_ibis-0.1.1.dist-info/top_level.txt

@@ -0,0 +1 @@
+ibis_hotdata

ibis_hotdata/__init__.py

@@ -0,0 +1,10 @@
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("hotdata-ibis")
+except PackageNotFoundError:
+    __version__ = "0.0.0+unknown"
+
+from ibis_hotdata.backend import Backend
+
+__all__ = ["Backend", "__version__"]

ibis_hotdata/backend.py

@@ -0,0 +1,673 @@
+"""Ibis backend for Hotdata (federated SQL over HTTP).
+
+Identifier mapping:
+
+* **Ibis catalog** ↔ Hotdata ``connection_id`` returned by ``GET /v1/connections`` /
+  ``connection`` on ``GET /v1/information_schema`` rows.
+* **Ibis database** ↔ remote **schema name** surfaced by Hotdata (`schema` column).
+* **Ibis table name** ↔ remote table name (`table` column).
+
+Use fully qualified SQL in Hotdata exactly as documented for federated Postgres-style
+references (typically ``connection.schema.table``, quoting if needed).
+
+See the README for dialect and typing limitations.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import io
+from collections.abc import Iterable, Mapping, Sequence
+from functools import cached_property
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as pkg_version
+from typing import TYPE_CHECKING, Any
+from urllib.parse import ParseResult, parse_qsl, unquote_plus
+
+import ibis.backends.sql.compilers as sc
+import ibis.common.exceptions as com
+import ibis.expr.datatypes as dt
+import ibis.expr.operations as ops
+import ibis.expr.schema as sch
+import ibis.expr.types as ir
+import sqlglot as sg
+import sqlglot.expressions as sge
+from ibis.backends import (
+    CanCreateDatabase,
+    CanListCatalog,
+    CanListDatabase,
+    HasCurrentCatalog,
+    HasCurrentDatabase,
+    NoExampleLoader,
+)
+from ibis.backends.sql import SQLBackend
+
+from ibis_hotdata.http import HotdataAPIError, HotdataClient
+from ibis_hotdata.managed import DEFAULT_SCHEMA, MANAGED_SOURCE_TYPE
+from ibis_hotdata.types import dtype_from_hotdata_sql_type
+
+_INFORMATION_SCHEMA_PAGE_SIZE = 500
+
+
+def _ibis_err_from_hotdata(exc: HotdataAPIError) -> com.IbisError:
+    return com.IbisError(str(exc))
+
+
+if TYPE_CHECKING:
+    from collections.abc import Iterator
+
+    import pandas as pd
+
+
+class Backend(
+    SQLBackend,
+    CanCreateDatabase,
+    CanListCatalog,
+    CanListDatabase,
+    HasCurrentCatalog,
+    HasCurrentDatabase,
+    NoExampleLoader,
+):
+    """Hotdata-backed Ibis client."""
+
+    name = "hotdata"
+    compiler = sc.postgres.compiler
+    supports_python_udfs = False
+    supports_temporary_tables = False
+
+    _http: HotdataClient
+    _default_connection: str | None
+    _default_schema: str | None
+
+    def _from_url(self, url: ParseResult, **kwarg_overrides: Any):
+        """Connect using ``hotdata://host`` or ``hotdata://host/path`` URLs.
+
+        * Base URL defaults to ``https://{host}`` plus optional leading ``path``.
+        * Query string may include ``token``, ``workspace_id``, ``session_id``,
+          ``timeout``, ``verify_ssl`` (``true`` / ``false``), ``default_connection``,
+          ``default_schema``, ``poll_interval_s``, ``poll_timeout_s``.
+        * If ``token`` is omitted, ``urlparse`` password (`user:TOKEN@`) is accepted.
+        """
+        q = dict(parse_qsl(url.query, keep_blank_values=True))
+        q.update(kwarg_overrides)
+
+        netloc = url.netloc
+        path_prefix = url.path.rstrip("/")
+        if not netloc:
+            raise com.IbisError(
+                "hotdata:// URL requires a network location, e.g. hotdata://api.hotdata.dev/"
+            )
+
+        verify = q.pop("verify_ssl", None)
+        if verify is None:
+            verify_ssl: bool | str = True
+        elif isinstance(verify, str):
+            verify_ssl = verify.lower() in ("true", "1", "yes")
+        else:
+            verify_ssl = bool(verify)
+
+        timeout = float(q.pop("timeout", "120"))
+        api_url = q.pop("api_url", None) or ("https://" + netloc + path_prefix)
+
+        token = q.pop("token", None) or (unquote_plus(url.password) if url.password else None)
+        workspace_id = q.pop("workspace_id", None)
+
+        kwargs = dict(
+            api_url=api_url,
+            token=token,
+            workspace_id=workspace_id,
+            session_id=q.pop("session_id", None),
+            timeout=timeout,
+            verify_ssl=verify_ssl,
+            default_connection=q.pop("default_connection", None),
+            default_schema=q.pop("default_schema", None),
+            poll_interval_s=float(q.pop("poll_interval_s", "0.25")),
+            poll_timeout_s=float(q.pop("poll_timeout_s", "600")),
+        )
+
+        self._convert_kwargs(kwargs)
+        if not kwargs.get("token"):
+            raise com.IbisError("Hotdata URL missing token (query parameter or password segment)")
+        if not kwargs.get("workspace_id"):
+            raise com.IbisError("Hotdata URL missing workspace_id query parameter")
+
+        return self.connect(**kwargs)
+
+    def do_connect(
+        self,
+        *,
+        api_url: str,
+        token: str,
+        workspace_id: str,
+        session_id: str | None = None,
+        timeout: float = 120.0,
+        verify_ssl: bool | str = True,
+        default_connection: str | None = None,
+        default_schema: str | None = None,
+        poll_interval_s: float = 0.25,
+        poll_timeout_s: float = 600.0,
+    ) -> None:
+        """Create an Ibis client for a Hotdata workspace.
+
+        Query execution always uses Hotdata's async path and downloads ready
+        results as Arrow IPC from ``GET /v1/results/{id}``.
+
+        Parameters
+        ----------
+        api_url
+            Hotdata API base URL (e.g. ``https://api.hotdata.dev``).
+        token
+            API bearer token (``Authorization`` header).
+        workspace_id
+            Workspace public id (``X-Workspace-Id`` header).
+        session_id
+            Optional sandbox id (``X-Session-Id`` header).
+        timeout
+            HTTP timeout in seconds (per request).
+        verify_ssl
+            Passed through to the Hotdata SDK configuration (boolean or path to a CA bundle).
+        default_connection
+            Optional default **catalog** (Hotdata connection id). If omitted and the
+            workspace exposes exactly one connection, it is chosen automatically;
+            otherwise you must set this (or use fully qualified ``database=``).
+        default_schema
+            Optional default **database** (remote schema name). If omitted and only
+            one schema exists for the default connection, it is chosen automatically.
+        poll_interval_s
+            Sleep between ``GET /v1/query-runs/{id}`` polls.
+        poll_timeout_s
+            Max wall-clock time spent waiting on async submissions.
+        """
+        self.disconnect()
+        self._default_connection = default_connection
+        self._default_schema = default_schema
+        self._poll_interval_s = poll_interval_s
+        self._poll_timeout_s = poll_timeout_s
+
+        self._http = HotdataClient(
+            api_url=api_url,
+            token=token,
+            workspace_id=workspace_id,
+            session_id=session_id,
+            timeout=timeout,
+            verify_ssl=verify_ssl,
+        )
+
+    def disconnect(self) -> None:
+        if getattr(self, "_http", None) is not None:
+            self._http.close()
+
+    # --- hierarchy ---------------------------------------------------------
+
+    def _infer_default_connection(self) -> str:
+        ids = self._connection_ids()
+        if self._default_connection is not None:
+            return self._default_connection
+        if len(ids) == 1:
+            self._default_connection = ids[0]
+            return self._default_connection
+        raise com.IbisInputError(
+            "Multiple Hotdata connections in this workspace — pass default_connection="
+            "... when connecting or qualify tables with database=('conn_id','schema')."
+        )
+
+    def _infer_default_schema(self, connection_id: str) -> str:
+        schemas = sorted(
+            {
+                row["schema"]
+                for row in self._iterate_information_schema(
+                    {"connection_id": connection_id}, include_columns=False
+                )
+            }
+        )
+        if self._default_schema is not None:
+            if self._default_schema not in schemas:
+                raise com.IbisInputError(
+                    f"Unknown schema {self._default_schema!r} for connection {connection_id!r}"
+                )
+            return self._default_schema
+        if len(schemas) == 1:
+            self._default_schema = schemas[0]
+            return self._default_schema
+        raise com.IbisInputError(
+            "Could not infer default schema — pass default_schema=... when connecting"
+            " or qualify with database=('connection_id','schema')."
+        )
+
+    @property
+    def current_catalog(self) -> str:
+        return self._infer_default_connection()
+
+    @property
+    def current_database(self) -> str:
+        return self._infer_default_schema(self.current_catalog)
+
+    # --- catalogs / databases ----------------------------------------------
+
+    def _to_catalog_db_tuple(self, table_loc: sge.Table):
+        """Use the compiler SQL dialect when stringifying qualifiers (backend name is not a dialect)."""
+
+        dialect = self.dialect
+        if (sg_cat := table_loc.args["catalog"]) is not None:
+            sg_cat.args["quoted"] = False
+            sg_cat = sg_cat.sql(dialect=dialect)
+        if (sg_db := table_loc.args["db"]) is not None:
+            sg_db.args["quoted"] = False
+            sg_db = sg_db.sql(dialect=dialect)
+
+        return sg_cat, sg_db
+
+    def _connection_ids(self) -> list[str]:
+        data = self._http.list_connections()
+        return [c["id"] for c in data["connections"]]
+
+    def list_catalogs(self, *, like: str | None = None) -> list[str]:
+        names = self._connection_ids()
+        return self._filter_with_like(names, like)
+
+    def list_databases(
+        self,
+        *,
+        like: str | None = None,
+        catalog: str | None = None,
+    ) -> list[str]:
+        conn = catalog or self.current_catalog
+        schemas = sorted(
+            {
+                row["schema"]
+                for row in self._iterate_information_schema(
+                    {"connection_id": conn}, include_columns=False
+                )
+            }
+        )
+        return self._filter_with_like(list(schemas), like)
+
+    def list_tables(
+        self,
+        *,
+        like: str | None = None,
+        database: tuple[str, str] | str | None = None,
+    ) -> list[str]:
+        loc = database if database is not None else (self.current_catalog, self.current_database)
+        table_loc = self._to_sqlglot_table(loc)
+        catalog_part, schema_part = self._to_catalog_db_tuple(table_loc)
+        params: dict[str, Any] = {}
+        if catalog_part:
+            params["connection_id"] = catalog_part
+        if schema_part:
+            params["schema"] = schema_part
+
+        tables = sorted(
+            {
+                row["table"]
+                for row in self._iterate_information_schema(params, include_columns=False)
+            }
+        )
+        return self._filter_with_like(tables, like)
+
+    def _iterate_information_schema(
+        self, filters: Mapping[str, Any], *, include_columns: bool
+    ) -> Iterable[dict[str, Any]]:
+        cursor: str | None = None
+        while True:
+            params: dict[str, Any] = dict(filters)
+            params["limit"] = _INFORMATION_SCHEMA_PAGE_SIZE
+            params["include_columns"] = include_columns
+            if cursor:
+                params["cursor"] = cursor
+            chunk = self._http.get_information_schema(params)
+            yield from chunk["tables"]
+            if not chunk.get("has_more"):
+                break
+            cursor = chunk.get("next_cursor")
+            if cursor is None:
+                break
+
+    def _resolve_connection(self, name_or_id: str) -> dict[str, Any]:
+        data = self._http.list_connections()
+        for conn in data["connections"]:
+            if conn["id"] == name_or_id or conn.get("name") == name_or_id:
+                return conn
+        raise com.IbisError(f"Unknown Hotdata connection {name_or_id!r}")
+
+    def _resolve_managed_connection(self, name_or_id: str) -> dict[str, Any]:
+        conn = self._resolve_connection(name_or_id)
+        if conn.get("source_type") != MANAGED_SOURCE_TYPE:
+            raise com.IbisInputError(
+                f"{name_or_id!r} is not a managed database "
+                f"(source_type={conn.get('source_type')!r})"
+            )
+        return conn
+
+    def _managed_table_synced(
+        self,
+        connection_id: str,
+        schema_name: str,
+        table_name: str,
+    ) -> bool:
+        """Return True only if the table exists and its last load has completed.
+
+        A table whose ``synced`` flag is False is still being loaded; we treat
+        it as writable (returns False) so that an in-progress load can be
+        retried without requiring ``overwrite=True``. Tables not present in the
+        information schema also return False (not yet created).
+        """
+        for row in self._iterate_information_schema(
+            {"connection_id": connection_id, "schema": schema_name, "table": table_name},
+            include_columns=False,
+        ):
+            if row["table"] == table_name and row["schema"] == schema_name:
+                return bool(row.get("synced", True))
+        return False
+
+    def _table_location(
+        self,
+        database: tuple[str, str] | str | None,
+    ) -> tuple[str, str]:
+        if database is None:
+            if self._default_connection is None or self._default_schema is None:
+                raise com.IbisInputError(
+                    "Requires database=(catalog, schema) or default_connection and default_schema"
+                )
+            conn = self._default_connection
+            schema = self._default_schema
+        elif isinstance(database, tuple):
+            conn, schema = database
+        else:
+            conn = self._default_connection or self.current_catalog
+            schema = database
+            if conn is None:
+                raise com.IbisInputError(
+                    "create_table with database=schema requires default_connection or current catalog"
+                )
+        conn_record = self._resolve_managed_connection(conn)
+        return conn_record["id"], schema
+
+    # --- schema / sql execution --------------------------------------------
+
+    def get_schema(
+        self,
+        table_name: str,
+        *,
+        catalog: str | None = None,
+        database: str | None = None,
+    ) -> sch.Schema:
+        conn = catalog or self.current_catalog
+        schema_name = database or self.current_database
+        matches: list[dict[str, Any]] = []
+        for row in self._iterate_information_schema(
+            {"connection_id": conn, "schema": schema_name, "table": table_name},
+            include_columns=True,
+        ):
+            if row["table"] == table_name and row["schema"] == schema_name:
+                matches.append(row)
+        if not matches:
+            fqn = sg.table(table_name, db=schema_name, catalog=conn).sql(self.compiler.dialect)
+            raise com.TableNotFound(fqn)
+
+        columns = matches[0].get("columns") or []
+        mapping: dict[str, dt.DataType] = {}
+        for col in columns:
+            name = col["name"]
+            sql_t = col.get("data_type")
+            null = bool(col.get("nullable", True))
+            mapping[name] = dtype_from_hotdata_sql_type(sql_t, nullable=null)
+        return sch.Schema(mapping)
+
+    def _get_schema_using_query(self, query: str) -> sch.Schema:
+        stripped = query.strip().rstrip(";")
+        preview_sql = f"SELECT * FROM ({stripped}) AS ibis_hotdata_preview LIMIT 1"
+        try:
+            data = self._http.execute_query(
+                preview_sql,
+                poll_interval_s=self._poll_interval_s,
+                poll_timeout_s=self._poll_timeout_s,
+            )
+        except HotdataAPIError as exc:
+            raise _ibis_err_from_hotdata(exc) from exc
+
+        from ibis.formats.pyarrow import PyArrowSchema
+
+        return PyArrowSchema.to_ibis(data["pa_table"].schema)
+
+    @contextlib.contextmanager
+    def _safe_raw_sql(
+        self,
+        query: str | sge.Expression,
+    ) -> Iterator[Any]:
+        if not isinstance(query, str):
+            query = query.sql(dialect=self.compiler.dialect, pretty=True)
+
+        try:
+            payload = self._http.execute_query(
+                query,
+                poll_interval_s=self._poll_interval_s,
+                poll_timeout_s=self._poll_timeout_s,
+            )
+        except HotdataAPIError as exc:
+            raise _ibis_err_from_hotdata(exc) from exc
+
+        yield payload["pa_table"]
+
+    def _fetch_from_cursor(self, cursor, schema: sch.Schema) -> pd.DataFrame:
+        from ibis.formats.pandas import PandasData
+
+        df = cursor.to_pandas()
+        return PandasData.convert_table(df, schema)
+
+    def to_pyarrow(
+        self,
+        expr: ir.Expr,
+        /,
+        *,
+        params: Mapping[ir.Scalar, Any] | None = None,
+        limit: int | str | None = None,
+        **kwargs: Any,
+    ):
+        self._run_pre_execute_hooks(expr)
+        table_expr = expr.as_table()
+        sql = self.compile(table_expr, params=params, limit=limit, **kwargs)
+        try:
+            payload = self._http.execute_query(
+                sql,
+                poll_interval_s=self._poll_interval_s,
+                poll_timeout_s=self._poll_timeout_s,
+            )
+        except HotdataAPIError as exc:
+            raise _ibis_err_from_hotdata(exc) from exc
+        table = payload["pa_table"]
+        arrow_schema = table_expr.schema().to_pyarrow()
+        table = table.rename_columns(list(table_expr.columns)).cast(arrow_schema)
+        return expr.__pyarrow_result__(table)
+
+    def to_pyarrow_batches(
+        self,
+        expr: ir.Expr,
+        /,
+        *,
+        params: Mapping[ir.Scalar, Any] | None = None,
+        limit: int | str | None = None,
+        chunk_size: int = 1_000_000,
+        **kwargs: Any,
+    ):
+        """Execute to Arrow and expose local record batches.
+
+        Hotdata currently returns one Arrow IPC result for the full query. This
+        method downloads that result first, then splits it into local batches.
+        """
+        import pyarrow as pa
+
+        table = self.to_pyarrow(expr.as_table(), params=params, limit=limit, **kwargs)
+        return pa.ipc.RecordBatchReader.from_batches(
+            table.schema,
+            table.to_batches(max_chunksize=chunk_size),
+        )
+
+    def upload_file(self, data: bytes, *, content_type: str | None = None) -> dict[str, Any]:
+        """POST ``/v1/files``; returns the upload record (use ``id`` with managed table loads)."""
+        try:
+            return self._http.upload_file(data, content_type=content_type)
+        except HotdataAPIError as exc:
+            raise _ibis_err_from_hotdata(exc) from exc
+
+    def create_database(
+        self,
+        name: str,
+        /,
+        *,
+        catalog: str | None = None,
+        schema: str = DEFAULT_SCHEMA,
+        tables: Sequence[str] | None = None,
+        force: bool = False,
+    ) -> None:
+        """Create a managed Hotdata connection (Ibis catalog) with optional declared tables."""
+        if catalog is not None:
+            raise com.UnsupportedOperationError(
+                "Hotdata create_database creates a managed connection (catalog); catalog= is not supported"
+            )
+        try:
+            existing = self._resolve_connection(name)
+        except com.IbisError:
+            existing = None
+        if existing is not None:
+            if not force:
+                raise com.IbisInputError(f"Managed database {name!r} already exists")
+            if existing.get("source_type") != MANAGED_SOURCE_TYPE:
+                raise com.IbisInputError(
+                    f"{name!r} is not a managed database "
+                    f"(source_type={existing.get('source_type')!r})"
+                )
+            return
+        try:
+            self._http.create_managed_database(name, schema=schema, tables=list(tables or ()))
+        except HotdataAPIError as exc:
+            raise _ibis_err_from_hotdata(exc) from exc
+
+    def drop_database(
+        self,
+        name: str,
+        /,
+        *,
+        catalog: str | None = None,
+        force: bool = False,
+    ) -> None:
+        """Delete a managed Hotdata connection (Ibis catalog)."""
+        if catalog is not None:
+            raise com.UnsupportedOperationError(
+                "Hotdata drop_database deletes a managed connection (catalog); catalog= is not supported"
+            )
+        try:
+            conn = self._resolve_managed_connection(name)
+        except com.IbisInputError:
+            raise
+        except com.IbisError:
+            if force:
+                return
+            raise
+        try:
+            self._http.delete_connection(conn["id"])
+        except HotdataAPIError as exc:
+            if force and exc.status_code == 404:
+                return
+            raise _ibis_err_from_hotdata(exc) from exc
+
+    def _local_table_to_parquet(self, obj: Any, schema: sch.Schema | None):
+        import pandas as pd
+        import pyarrow as pa
+        import pyarrow.parquet as pq
+
+        if obj is not None and schema is not None:
+            raise com.IbisInputError("create_table accepts only one of obj or schema")
+
+        if obj is None:
+            if schema is None:
+                raise com.IbisInputError("create_table requires a pandas/pyarrow object or schema")
+            arrow_schema = schema.to_pyarrow()
+            table = pa.Table.from_arrays(
+                [pa.array([], type=field.type) for field in arrow_schema],
+                schema=arrow_schema,
+            )
+        elif isinstance(obj, pa.Table):
+            table = obj
+        elif isinstance(obj, pd.DataFrame):
+            table = pa.Table.from_pandas(obj, preserve_index=False)
+        else:
+            raise com.IbisInputError(
+                "create_table currently accepts pandas.DataFrame or pyarrow.Table"
+            )
+
+        sink = io.BytesIO()
+        pq.write_table(table, sink)
+        return sink.getvalue()
+
+    def create_table(
+        self,
+        name: str,
+        /,
+        obj: Any = None,
+        *,
+        schema: sch.Schema | None = None,
+        database: tuple[str, str] | str | None = None,
+        temp: bool = False,
+        overwrite: bool = False,
+    ) -> ir.Table:
+        """Upload local data into a declared managed table.
+
+        Hotdata loads always use ``replace`` mode (the only API option). When
+        ``overwrite=False`` (the Ibis default), an existing synced table raises
+        :class:`~ibis.common.exceptions.IbisInputError` instead of replacing it.
+        """
+        if temp:
+            raise NotImplementedError("Hotdata does not support temporary tables.")
+
+        data = self._local_table_to_parquet(obj, schema)
+        connection_id, schema_name = self._table_location(database)
+        if not overwrite and self._managed_table_synced(connection_id, schema_name, name):
+            raise com.IbisInputError(
+                f"Table {name!r} already exists; pass overwrite=True to replace"
+            )
+        upload = self.upload_file(data, content_type="application/parquet")
+        try:
+            self._http.load_managed_table(
+                connection_id,
+                schema_name,
+                name,
+                upload_id=upload["id"],
+            )
+        except HotdataAPIError as exc:
+            raise _ibis_err_from_hotdata(exc) from exc
+        return self.table(name, database=(connection_id, schema_name))
+
+    def drop_table(
+        self,
+        name: str,
+        /,
+        *,
+        database: tuple[str, str] | str | None = None,
+        force: bool = False,
+    ) -> None:
+        try:
+            connection_id, schema_name = self._table_location(database)
+        except com.IbisInputError:
+            raise
+        except com.IbisError:
+            if force:
+                return
+            raise
+        try:
+            self._http.delete_managed_table(connection_id, schema_name, name)
+        except HotdataAPIError as exc:
+            if force and exc.status_code == 404:
+                return
+            raise _ibis_err_from_hotdata(exc) from exc
+
+    def _register_in_memory_table(self, _op: ops.InMemoryTable) -> None:
+        return
+
+    @cached_property
+    def version(self) -> str:
+        try:
+            v = pkg_version("hotdata-ibis")
+        except PackageNotFoundError:
+            v = "0.0.0"
+        return f"hotdata-ibis {v} (Hotdata /v1/query)"

ibis_hotdata/http.py

@@ -0,0 +1,290 @@
+"""HTTP access to Hotdata via the official ``hotdata`` Python SDK (OpenAPI client)."""
+
+from __future__ import annotations
+
+import io
+import json
+import time
+from collections.abc import Callable, Mapping, Sequence
+from typing import Any, TypeVar
+
+import pyarrow as pa
+import pyarrow_hotfix  # noqa: F401
+import pyarrow.ipc as pa_ipc
+
+from hotdata import ApiClient, Configuration
+from hotdata.api import (
+    ConnectionsApi,
+    InformationSchemaApi,
+    QueryApi,
+    QueryRunsApi,
+    ResultsApi,
+    UploadsApi,
+)
+from hotdata.exceptions import ApiException
+from hotdata.models import CreateConnectionRequest, QueryRequest
+from hotdata.models.async_query_response import AsyncQueryResponse
+from hotdata.models.load_managed_table_request import LoadManagedTableRequest
+
+from ibis_hotdata.managed import DEFAULT_SCHEMA, MANAGED_SOURCE_TYPE, build_managed_config
+
+T = TypeVar("T")
+
+# Matches Hotdata / runtimedb ``GET /v1/results/{{id}}`` Arrow responses.
+APPLICATION_ARROW_STREAM = "application/vnd.apache.arrow.stream"
+
+
+def _sleep_until(deadline: float, interval: float) -> None:
+    """Sleep up to ``interval`` s but never past ``deadline`` (cleaner timeout behavior)."""
+    remaining = deadline - time.monotonic()
+    if remaining > 0:
+        time.sleep(min(interval, remaining))
+
+
+class HotdataAPIError(Exception):
+    def __init__(self, message: str, *, status_code: int | None = None, body: Any = None):
+        super().__init__(message)
+        self.status_code = status_code
+        self.body = body
+
+
+def _from_api_exception(exc: ApiException) -> HotdataAPIError:
+    body = exc.body
+    if isinstance(body, (bytes, bytearray)):
+        body = body.decode("utf-8", errors="replace")
+    msg = f"Hotdata API error: {exc.reason}"
+    if body:
+        msg = f"{msg} {body}"
+    return HotdataAPIError(msg.strip(), status_code=exc.status, body=exc.body)
+
+
+def _ipc_stream_bytes_to_table(data: bytes) -> pa.Table:
+    with pa_ipc.open_stream(io.BytesIO(data)) as reader:
+        return reader.read_all()
+
+
+def _json_utf8(obj: bytes) -> Any:
+    return json.loads(obj.decode("utf-8"))
+
+
+class HotdataClient:
+    """Thin wrapper around the SDK used by the Ibis backend."""
+
+    def __init__(
+        self,
+        *,
+        api_url: str,
+        token: str,
+        workspace_id: str,
+        session_id: str | None = None,
+        timeout: float = 120.0,
+        verify_ssl: bool | str = True,
+    ) -> None:
+        host = api_url.rstrip("/")
+        conf = Configuration(
+            host=host, api_key=token, workspace_id=workspace_id, session_id=session_id
+        )
+        if verify_ssl is False:
+            conf.verify_ssl = False
+        elif isinstance(verify_ssl, str):
+            conf.ssl_ca_cert = verify_ssl
+        self._timeout = timeout
+        self._client = ApiClient(conf)
+        self._query = QueryApi(self._client)
+        self._query_runs = QueryRunsApi(self._client)
+        self._results = ResultsApi(self._client)
+        self._connections = ConnectionsApi(self._client)
+        self._information_schema = InformationSchemaApi(self._client)
+        self._uploads = UploadsApi(self._client)
+
+    def close(self) -> None:
+        client = self._client
+        close_fn = getattr(client, "close", None)
+        if callable(close_fn):
+            close_fn()
+            return
+        pool = getattr(getattr(client, "rest_client", None), "pool_manager", None)
+        if pool is not None:
+            pool.clear()
+
+    def _safe_call(self, fn: Callable[..., T], /, *args: Any, **kwargs: Any) -> T:
+        try:
+            return fn(*args, _request_timeout=self._timeout, **kwargs)
+        except ApiException as exc:
+            raise _from_api_exception(exc) from exc
+
+    def list_connections(self) -> dict[str, Any]:
+        """GET ``/v1/connections``."""
+        out = self._safe_call(self._connections.list_connections)
+        return out.model_dump(by_alias=True, mode="json")
+
+    def get_information_schema(self, params: Mapping[str, Any]) -> dict[str, Any]:
+        """GET ``/v1/information_schema`` — ``params`` uses REST names (``schema`` not ``var_schema``)."""
+        out = self._safe_call(
+            self._information_schema.information_schema,
+            connection_id=params.get("connection_id"),
+            var_schema=params.get("schema"),
+            table=params.get("table"),
+            include_columns=params.get("include_columns"),
+            limit=params.get("limit"),
+            cursor=params.get("cursor"),
+        )
+        return out.model_dump(by_alias=True, mode="json")
+
+    def execute_query(
+        self,
+        sql: str,
+        *,
+        async_after_ms: int | None = None,
+        poll_interval_s: float = 0.25,
+        poll_timeout_s: float = 600.0,
+    ) -> dict[str, Any]:
+        req = QueryRequest(sql=sql, var_async=True, async_after_ms=async_after_ms)
+        out = self._safe_call(self._query.query, req)
+        if isinstance(out, AsyncQueryResponse):
+            query_run_id = out.query_run_id
+            deadline = time.monotonic() + poll_timeout_s
+            while time.monotonic() < deadline:
+                qr = self._safe_call(self._query_runs.get_query_run, query_run_id)
+                status = qr.status
+                if status == "failed":
+                    raise HotdataAPIError(qr.error_message or "Query run failed")
+                if status == "succeeded":
+                    result_id = qr.result_id
+                    if result_id is None:
+                        raise HotdataAPIError("succeeded query run missing result_id")
+                    return self._poll_result_arrow(
+                        result_id,
+                        deadline=deadline,
+                        poll_interval_s=poll_interval_s,
+                    )
+                _sleep_until(deadline, poll_interval_s)
+            raise HotdataAPIError("Timeout waiting for asynchronous query")
+        raise HotdataAPIError("Unexpected query response type")
+
+    def upload_file(self, data: bytes, *, content_type: str | None = None) -> dict[str, Any]:
+        kwargs: dict[str, Any] = {}
+        if content_type is not None:
+            kwargs["_content_type"] = content_type
+        resp = self._safe_call(self._uploads.upload_file, data, **kwargs)
+        return resp.model_dump(by_alias=True, mode="json")
+
+    def create_managed_database(
+        self,
+        name: str,
+        *,
+        schema: str = DEFAULT_SCHEMA,
+        tables: Sequence[str] = (),
+    ) -> dict[str, Any]:
+        req = CreateConnectionRequest(
+            name=name,
+            source_type=MANAGED_SOURCE_TYPE,
+            config=build_managed_config(schema, list(tables)),
+            skip_discovery=True,
+        )
+        resp = self._safe_call(self._connections.create_connection, req)
+        return resp.model_dump(by_alias=True, mode="json")
+
+    def delete_connection(self, connection_id: str) -> None:
+        self._safe_call(self._connections.delete_connection, connection_id)
+
+    def load_managed_table(
+        self,
+        connection_id: str,
+        schema: str,
+        table: str,
+        *,
+        upload_id: str,
+    ) -> dict[str, Any]:
+        req = LoadManagedTableRequest(mode="replace", upload_id=upload_id)
+        resp = self._safe_call(
+            self._connections.load_managed_table,
+            connection_id,
+            schema,
+            table,
+            req,
+        )
+        return resp.model_dump(by_alias=True, mode="json")
+
+    def delete_managed_table(self, connection_id: str, schema: str, table: str) -> None:
+        self._safe_call(
+            self._connections.delete_managed_table,
+            connection_id,
+            schema,
+            table,
+        )
+
+    def _poll_result_arrow(
+        self,
+        result_id: str,
+        *,
+        deadline: float,
+        poll_interval_s: float,
+    ) -> dict[str, Any]:
+        """Poll ``GET /v1/results/{{id}}`` with ``Accept: application/vnd.apache.arrow.stream``."""
+        while time.monotonic() < deadline:
+            try:
+                raw = self._results.get_result_without_preload_content(
+                    result_id,
+                    _headers={"Accept": APPLICATION_ARROW_STREAM},
+                    _request_timeout=self._timeout,
+                )
+            except ApiException as exc:
+                raise _from_api_exception(exc) from exc
+            body = raw.read()
+            status = raw.status
+            ctype = (raw.headers.get("Content-Type") or "").split(";")[0].strip().lower()
+
+            if status == 200 and ctype == APPLICATION_ARROW_STREAM.lower():
+                table = _ipc_stream_bytes_to_table(body)
+                return self._arrow_payload_from_table(table, result_id=result_id)
+
+            if status == 202:
+                _sleep_until(deadline, poll_interval_s)
+                continue
+
+            if status == 409:
+                d = _json_utf8(body) if body else {}
+                raise HotdataAPIError(
+                    d.get("error_message") or "Result failed",
+                    status_code=409,
+                    body=d,
+                )
+
+            if status == 404:
+                d = _json_utf8(body) if body else {}
+                raise HotdataAPIError(
+                    d.get("detail") or f"Result {result_id!r} not found",
+                    status_code=404,
+                    body=d,
+                )
+
+            raise HotdataAPIError(
+                f"Unexpected GET /v1/results/{result_id} status {status}",
+                status_code=status,
+                body=body,
+            )
+
+        raise HotdataAPIError("Timeout waiting for Arrow query result")
+
+    def _arrow_payload_from_table(
+        self,
+        table: pa.Table,
+        *,
+        result_id: str,
+    ) -> dict[str, Any]:
+        sch = table.schema
+        columns = sch.names
+        nullable = [sch.field(i).nullable for i in range(len(columns))]
+        return {
+            "format": "arrow",
+            "pa_table": table,
+            "columns": columns,
+            "nullable": nullable,
+            "rows": [],
+            "result_id": result_id,
+            "row_count": table.num_rows,
+            "execution_time_ms": None,
+            "query_run_id": None,
+            "warning": None,
+        }

ibis_hotdata/managed.py

@@ -0,0 +1,19 @@
+"""Helpers for Hotdata managed database connections."""
+
+from __future__ import annotations
+
+from typing import Any
+
+MANAGED_SOURCE_TYPE = "managed"
+DEFAULT_SCHEMA = "public"
+
+
+def build_managed_config(schema: str, tables: list[str]) -> dict[str, Any]:
+    return {
+        "schemas": [
+            {
+                "name": schema,
+                "tables": [{"name": table} for table in tables],
+            }
+        ]
+    }

ibis_hotdata/types.py

@@ -0,0 +1,16 @@
+"""Map Hotdata metadata to Ibis dtypes."""
+
+from __future__ import annotations
+
+import ibis.expr.datatypes as dt
+from ibis.backends.sql.datatypes import PostgresType
+
+
+def dtype_from_hotdata_sql_type(sql_type: str | None, *, nullable: bool) -> dt.DataType:
+    """Best-effort mapping from Hotdata `/information_schema` column `data_type` strings."""
+    if not sql_type:
+        return dt.String(nullable=nullable)
+    try:
+        return PostgresType.from_string(sql_type.strip(), nullable=nullable)
+    except Exception:  # ibis/sqlglot raise a variety of parse errors; fall back to String
+        return dt.String(nullable=nullable)