hotdata-framework 0.12.1__tar.gz → 0.14.0__tar.gz

{hotdata_framework-0.12.1 → hotdata_framework-0.14.0}/CHANGELOG.md

@@ -8,6 +8,119 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 
+## [0.14.0] - 2026-09-01
+
+### Fixed
+
+- fix(managed): wait on the query run instead of downloading the result to check it.
+
+  Reading a managed table made three calls and used one. `POST /v1/query` returned
+  an inline preview of the rows, `GET /v1/results/{id}` was polled until the
+  result was `ready`, and the result was then fetched as Arrow. Only the Arrow
+  copy was used.
+
+  The readiness poll was the expensive one. `limit` on that endpoint defaults to
+  unbounded, so polling a ready result downloads the entire result body to read
+  one status field. It is also the wrong endpoint to lean on as a table grows:
+  a JSON body over the instance's per-fetch memory budget is refused with 413,
+  and one that would fit alone but not alongside concurrent JSON fetches with
+  429 — so the readiness check starts failing on exactly the largest tables.
+
+  The query is now submitted with `async`, so the server returns a run id rather
+  than a preview, and readiness comes from `GET /v1/query-runs/{id}`, which
+  carries no rows at any size. `result_id` is read off the run rather than off
+  the query reply, because a run can succeed having saved nothing and the run is
+  what reports that — and that case now raises rather than reading as an empty
+  table. `fetch_table` answered `None` for it, which `fetch_table_rows` turns
+  into `[]`, the same answer both give for a table that is not synced. A
+  read-modify-write load would have read no existing rows and written only its
+  new batch, dropping every row already there. A reply shape this client does not
+  recognise raises for the same reason, as `HotdataClient` already did — so a
+  `None` from `fetch_table` now means one thing only: the table is not synced.
+  Arrow stays the only path the data travels, so column types come from the
+  server's schema rather than being inferred from JSON.
+
+  Costs one extra round trip on a query that would have answered synchronously,
+  in exchange for not transferring the result twice.
+
+  The Arrow fetch now also waits out a result that reports itself not ready, in
+  case that ordering ever stops holding. It should be unreachable, and it is
+  cheap to keep: that endpoint answers a result which is not ready with a small
+  refusal rather than with data, which is exactly what made waiting on the JSON
+  result body expensive and waiting here not.
+
+- fix(managed): recognise `interrupted`, and drop a run status the API never sends.
+
+  Both `ManagedDatabaseClient` and `HotdataClient` treated `failed` and
+  `cancelled` as the terminal run failures. `cancelled` is not a status this API
+  returns. `interrupted` is — a run whose server was replaced before it finished
+  — and it matched neither, so an interrupted run was polled for the full
+  five-minute timeout and then raised `TimeoutError`: a retryable condition
+  hidden behind a long wait and an error naming the wrong problem.
+
+  On `ManagedDatabaseClient` an interrupted run is now raised as transient, so
+  the surrounding retry re-submits the query. That needed `classify_sdk_error` to
+  pass an already-classified error through unchanged rather than demoting a
+  caller-raised transient error to terminal. `HotdataClient.execute_sql` now
+  fails fast on it with the run's own message.
+
+  Both polls keep enumerating the statuses that mean *finished*, and an
+  unrecognised status still waits. Calling an unknown status terminal would make
+  the omission easier to diagnose and much worse to live with: one status added
+  upstream would fail every read at once, where waiting costs a single slow call.
+  What made `interrupted` expensive was not the waiting — it was that the
+  timeout never said which status it had been waiting on. Both timeouts now name
+  it.
+
+## [0.13.0] - 2026-08-27
+
+### Fixed
+
+- fix(load): retry an `append` load instead of running it at most once.
+
+  `append` was excluded from retries on the grounds that it is not idempotent:
+  if the server commits but the response is lost, a retry would duplicate rows.
+  That is not how the server behaves. It keys a receipt on `upload_id`, and a
+  re-POST of the same id replays the committed result instead of applying the
+  load again — so what makes a retry safe is re-sending the same upload, not
+  the mode. This client stages once, in `upload_parquet`, outside the retried
+  operation, so the invariant holds for every mode.
+
+  The exclusion cost real availability. The destination serialises writes per
+  table and refuses rather than queues, so concurrent writers to one table get
+  `409 RESOURCE_LOCKED` — and an append had no budget to wait it out, whatever
+  `max_retries` the caller had configured.
+
+  `HotdataClient.load_managed_table(file=...)` uploads inside the call and so
+  does not hold the invariant. It is unwrapped and unaffected.
+
+- fix(errors): classify a 409 by its `error.code` rather than by the status alone.
+
+  `CONFLICT` is now terminal: it means the request cannot succeed as posted, so
+  the previous behaviour spent the entire retry budget arriving at the same
+  answer. `RESOURCE_LOCKED` stays transient. A 409 with no error envelope — a
+  failed query result, say — is classified as before.
+
+- fix(retry): honour `Retry-After`, and jitter the backoff.
+
+  `Retry-After` is taken as a floor on the ramp, capped like the ramp so a bad
+  header cannot park an attempt for an hour. Jitter of up to +50% is added on
+  top and never subtracted, so a stated `Retry-After` is not undercut. Without
+  it, writers that collided on one table retry in lockstep and collide again.
+
+  This lengthens a 20-attempt budget from 285s to roughly 316-405s.
+
+- docs: scope the "a load is not idempotent" claim in the README and in
+  `test_retry_policy` to the transport layer, which is where it is still true
+  and where those two were always talking about. Left unscoped they read as
+  repo-wide and contradict the call-layer retry above.
+
+### Added
+
+- `HotdataError` carries `status_code`, `code` and `retry_after_seconds`. The
+  message is flattened and truncated for readability, so it could not serve as
+  a discriminator; these can.
+
 ## [0.12.1] - 2026-08-18
 
 ### Fixed

{hotdata_framework-0.12.1 → hotdata_framework-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: hotdata-framework
-Version: 0.12.1
+Version: 0.14.0
 Summary: Python framework for building Hotdata integrations: workspace runtime, query execution, and managed databases
 Project-URL: Homepage, https://www.hotdata.dev
 Project-URL: Documentation, https://www.hotdata.dev/docs
@@ -38,7 +38,7 @@ Runtime boundary and guarantees are defined in `CONTRACT.md`.
 
 - **Environment-driven client setup** — create clients from `HOTDATA_API_KEY`, optional `HOTDATA_API_URL`, and `HOTDATA_WORKSPACE`.
 - **Workspace resolution** — choose an explicit workspace from env, otherwise discover workspaces and select the active workspace or first available workspace.
-- **HTTP resilience** — retry SQL execution on stale pooled sockets. Transport-level retries are the SDK's own default, which this package leaves in place so a non-idempotent request is never replayed on a response status.
+- **HTTP resilience** — retry SQL execution on stale pooled sockets. Transport-level retries are the SDK's own default, which this package leaves in place so a request is never blindly replayed on a response status. That is a claim about the transport, which cannot know what it would be replaying. `ManagedDatabaseClient` retries at the call layer, which can: a managed load is safe to re-send because it carries the same `upload_id` and the API replays its receipt for that id rather than applying the load twice.
 - **SQL execution helper** — run SQL through `POST /v1/query`, poll async query runs when needed, and return a `QueryResult`.
 - **Result utilities** — convert query results to records, pandas DataFrames, or metadata dictionaries for adapter display layers.
 - **History helpers** — list recent results and query run history with normalized dataclasses.

{hotdata_framework-0.12.1 → hotdata_framework-0.14.0}/README.md

@@ -10,7 +10,7 @@ Runtime boundary and guarantees are defined in `CONTRACT.md`.
 
 - **Environment-driven client setup** — create clients from `HOTDATA_API_KEY`, optional `HOTDATA_API_URL`, and `HOTDATA_WORKSPACE`.
 - **Workspace resolution** — choose an explicit workspace from env, otherwise discover workspaces and select the active workspace or first available workspace.
-- **HTTP resilience** — retry SQL execution on stale pooled sockets. Transport-level retries are the SDK's own default, which this package leaves in place so a non-idempotent request is never replayed on a response status.
+- **HTTP resilience** — retry SQL execution on stale pooled sockets. Transport-level retries are the SDK's own default, which this package leaves in place so a request is never blindly replayed on a response status. That is a claim about the transport, which cannot know what it would be replaying. `ManagedDatabaseClient` retries at the call layer, which can: a managed load is safe to re-send because it carries the same `upload_id` and the API replays its receipt for that id rather than applying the load twice.
 - **SQL execution helper** — run SQL through `POST /v1/query`, poll async query runs when needed, and return a `QueryResult`.
 - **Result utilities** — convert query results to records, pandas DataFrames, or metadata dictionaries for adapter display layers.
 - **History helpers** — list recent results and query run history with normalized dataclasses.

{hotdata_framework-0.12.1 → hotdata_framework-0.14.0}/hotdata_framework/client.py

@@ -77,8 +77,17 @@ VectorMetric = Literal["l2", "cosine", "dot"]
 _INDEX_TYPES = frozenset(get_args(IndexType))
 _VECTOR_METRICS = frozenset(get_args(VectorMetric))
 
-_TERMINAL = frozenset({"succeeded", "failed", "cancelled"})
-_RESULT_FAILURE = frozenset({"failed", "cancelled"})
+# Query-run statuses that mean the run is over. `interrupted` belongs here --
+# omitting it is what made an interrupted run wait out the full timeout -- and
+# `cancelled`, listed here for a long time, is not a status this API sends.
+#
+# Enumerating the terminal side rather than the in-flight side is deliberate. An
+# unrecognised status then keeps polling and costs one slow call, where treating
+# it as terminal would fail every query the moment a status is added upstream.
+# The timeout names the status it last saw, so a missing one is diagnosable
+# without being dangerous.
+_RUN_TERMINAL = frozenset({"succeeded", "failed", "interrupted"})
+_RESULT_FAILURE = frozenset({"failed"})
 # Jobs have no "cancelled" state; "partially_succeeded" carries an error_message.
 _JOB_TERMINAL = frozenset({"succeeded", "partially_succeeded", "failed"})
 
@@ -918,7 +927,7 @@ class HotdataClient:
         last = None
         while time.monotonic() < deadline:
             last = runs.get_query_run(query_run_id)
-            if last.status in _TERMINAL:
+            if last.status in _RUN_TERMINAL:
                 return last
             time.sleep(interval_s)
         raise TimeoutError(
@@ -985,9 +994,11 @@ class HotdataClient:
         durable state rather than from a connection that has to stay alive. That
         also gives a caller a handle: the job id is returned on
         `LoadManagedTableResult`, so "did it land?" is answerable after a lost
-        response. `append` stays non-retryable -- knowing the id makes the question
-        answerable, it does not make a blind re-submission safe, and that call is
-        the caller's to make.
+        response. That answer is a convenience rather than a precondition for
+        retrying: re-sending the same upload_id replays the server's receipt
+        instead of applying the load a second time, which is what makes a retry
+        safe in every mode. It stops being safe for a caller that re-stages the
+        upload, because a fresh upload id has no receipt to replay.
 
         `partially_succeeded` is terminal and carries a message, so it is raised
         rather than returned -- a caller asked for a table's contents to be

hotdata_framework-0.14.0/hotdata_framework/errors.py

@@ -0,0 +1,140 @@
+from __future__ import annotations
+
+import json
+from collections.abc import Mapping
+
+from hotdata.rest import ApiException
+
+# The API explains a 409 with a machine-readable code, and the two it sends
+# mean opposite things to a retry policy. RESOURCE_LOCKED is a refusal taken
+# before any work: the insert that would have created the unit of work lost a
+# unique-constraint race, so nothing was claimed and nothing was written.
+# CONFLICT is the opposite — the request cannot succeed as posted, so retrying
+# spends the whole budget arriving at the same answer.
+_TERMINAL_CONFLICT_CODE = "CONFLICT"
+
+
+class HotdataError(RuntimeError):
+    """An API failure, carrying what a retry policy needs to decide.
+
+    The message cannot be the discriminator: it is flattened and truncated for
+    readability, so keying on it means substring-matching prose. ``status_code``
+    and ``code`` are the machine-readable form of the same answer, and
+    ``retry_after_seconds`` is the server's own estimate of how long the
+    condition it just refused will last.
+    """
+
+    def __init__(
+        self,
+        message: str,
+        *,
+        status_code: int | None = None,
+        code: str | None = None,
+        retry_after_seconds: float | None = None,
+    ) -> None:
+        super().__init__(message)
+        self.status_code = status_code
+        self.code = code
+        self.retry_after_seconds = retry_after_seconds
+
+
+class HotdataTransientError(HotdataError):
+    pass
+
+
+class HotdataTerminalError(HotdataError):
+    pass
+
+
+def _error_code(body: object) -> str | None:
+    """The ``error.code`` an API error envelope carries, if this body is one.
+
+    Not every 409 comes from an endpoint that speaks the envelope — a failed
+    query result is reported as one and carries a result document instead — so
+    a missing code is ordinary, and callers fall back to the status.
+    """
+    if not isinstance(body, (str, bytes, bytearray)):
+        return None
+    try:
+        parsed: object = json.loads(body)
+    except ValueError:
+        return None
+    if not isinstance(parsed, Mapping):
+        return None
+    error: object = parsed.get("error")
+    if not isinstance(error, Mapping):
+        return None
+    code: object = error.get("code")
+    return code if isinstance(code, str) else None
+
+
+def _retry_after_seconds(headers: object) -> float | None:
+    """``Retry-After`` as a number of seconds, when the response states one.
+
+    Only the delta-seconds form is read. That is what the API sends, and the
+    HTTP-date form would need a comparison against a server clock we do not
+    have to be worth anything.
+    """
+    if not isinstance(headers, Mapping):
+        return None
+    raw: object = headers.get("Retry-After")
+    if raw is None:
+        # The SDK hands us urllib3's case-insensitive mapping and the API sends
+        # the header lower-cased, so the direct hit is what normally answers.
+        # Fall back for any plain dict that reaches us instead — a missed
+        # header is silent, and silence here reads as "the server asked for
+        # nothing".
+        raw = next((v for k, v in headers.items() if str(k).lower() == "retry-after"), None)
+    if raw is None:
+        return None
+    try:
+        seconds = float(str(raw).strip())
+    except ValueError:
+        return None
+    return seconds if seconds >= 0 else None
+
+
+def _error_class(status_code: int, code: str | None) -> type[HotdataError]:
+    if status_code == 409 and code == _TERMINAL_CONFLICT_CODE:
+        # The request cannot succeed as posted — an upload already consumed
+        # with nothing to replay, a receipt naming a different target, an
+        # incompatible column type. Every retry reaches the same 409.
+        return HotdataTerminalError
+    if status_code in (408, 409, 425, 429):
+        return HotdataTransientError
+    if status_code == 501:
+        # Not Implemented is a permanent capability gap (e.g. the storage
+        # backend cannot issue presigned URLs) — retrying cannot succeed.
+        return HotdataTerminalError
+    if 500 <= status_code <= 599:
+        return HotdataTransientError
+    return HotdataTerminalError
+
+
+def classify_sdk_error(error: Exception) -> HotdataError:
+    if isinstance(error, HotdataError):
+        # Already classified. A caller that read transience off a typed status
+        # -- an interrupted query run, say -- knows more than this function can
+        # recover from the exception, and the fallback below would demote it to
+        # terminal and cost the retry.
+        return error
+    if isinstance(error, TimeoutError):
+        return HotdataTransientError(str(error))
+    if isinstance(error, ConnectionError):
+        return HotdataTransientError(str(error))
+    if isinstance(error, ApiException):
+        status_code = int(error.status or 0)
+        message = f"{status_code}: {error.reason or 'unknown error'}"
+        # The response body is where the API explains itself (e.g. which
+        # header is missing) — without it "400: Bad Request" is undebuggable.
+        body: object = getattr(error, "body", None)
+        if body:
+            message = f"{message} — {' '.join(str(body).split())[:500]}"
+        code = _error_code(body)
+        return _error_class(status_code, code)(
+            message,
+            status_code=status_code,
+            code=code,
+            retry_after_seconds=_retry_after_seconds(getattr(error, "headers", None)),
+        )
+    return HotdataTerminalError(str(error))

hotdata_framework-0.14.0/hotdata_framework/managed_client.py

@@ -0,0 +1,313 @@
+"""Retry-wrapped managed-database client shared by Hotdata adapter packages.
+
+Both hotdata-airflow and hotdata-dlt-destination import this module so that
+the higher-level client logic (retries, Arrow queries, table management) lives
+in one place rather than being duplicated per adapter.
+"""
+
+from __future__ import annotations
+
+import random
+import time
+from collections.abc import Callable
+from typing import Any, TypeVar
+
+import pyarrow as pa
+from hotdata.api.query_api import QueryApi
+from hotdata.api.query_runs_api import QueryRunsApi
+from hotdata.arrow import ResultNotReadyError
+from hotdata.arrow import ResultsApi as ArrowResultsApi
+from hotdata.models.async_query_response import AsyncQueryResponse
+from hotdata.models.query_request import QueryRequest
+from hotdata.models.query_response import QueryResponse
+
+from hotdata_framework.client import HotdataClient as RuntimeClient
+from hotdata_framework.client import ManagedLoadMode
+from hotdata_framework.databases import LoadManagedTableResult, ManagedDatabase
+from hotdata_framework.errors import (
+    HotdataTransientError,
+    classify_sdk_error,
+)
+
+T = TypeVar("T")
+
+
+class ManagedDatabaseClient:
+    """Managed-database client with bounded retries over hotdata-framework.
+
+    This is the shared client used by Hotdata adapter packages (Airflow,
+    dlt, etc.).  It wraps the lower-level RuntimeClient with retry logic,
+    Arrow-based result fetching, and convenience helpers for the managed
+    database lifecycle.
+    """
+
+    _QUERY_TIMEOUT_SECONDS = 300.0
+    _POLL_INTERVAL_SECONDS = 0.4
+    _MAX_BACKOFF_SECONDS = 30.0
+    # Spread as a fraction of the wait, added on top of it. Half an interval is
+    # enough to decorrelate writers that started together without materially
+    # changing how long the budget lasts.
+    _RETRY_JITTER_FRACTION = 0.5
+
+    def __init__(
+        self,
+        *,
+        api_key: str,
+        workspace_id: str,
+        api_base_url: str,
+        max_retries: int,
+        retry_backoff_seconds: float,
+        request_timeout: float | tuple[float, float] | None = None,
+    ) -> None:
+        self._max_retries = max_retries
+        self._retry_backoff_seconds = retry_backoff_seconds
+        self._runtime = RuntimeClient(
+            api_key,
+            workspace_id,
+            host=api_base_url.rstrip("/"),
+            request_timeout=request_timeout,
+        )
+
+    def close(self) -> None:
+        self._runtime.close()
+
+    def ensure_managed_database(
+        self,
+        name: str,
+        *,
+        schema: str,
+        tables: list[str],
+        create_if_missing: bool,
+    ) -> ManagedDatabase:
+        def operation() -> ManagedDatabase:
+            try:
+                return self._runtime.resolve_managed_database(name)
+            except KeyError:
+                if not create_if_missing:
+                    raise
+                return self._runtime.create_managed_database(
+                    description=name,
+                    schema=schema,
+                    tables=sorted(set(tables)),
+                )
+
+        return self._request_with_retry(operation)
+
+    def table_is_synced(self, database: str, table: str, *, schema: str) -> bool:
+        for managed_table in self._runtime.list_managed_tables(database, schema=schema):
+            if managed_table.table == table:
+                return managed_table.synced
+        return False
+
+    def fetch_table(self, *, database: str, schema: str, table: str) -> pa.Table | None:
+        def operation() -> pa.Table | None:
+            if not self.table_is_synced(database, table, schema=schema):
+                return None
+            db = self._runtime.resolve_managed_database(database)
+            sql = f'SELECT * FROM "default"."{schema}"."{table}"'
+            result_id = self._query_database_scoped(sql, database_id=db.id)
+            if result_id is None:
+                return None
+            return self._fetch_result_arrow(result_id, database_id=db.id)
+
+        return self._request_with_retry(operation)
+
+    def _fetch_result_arrow(self, result_id: str, *, database_id: str) -> pa.Table:
+        """Fetch a ready result as Arrow, carrying the database scope.
+
+        Results of database-scoped queries are themselves database-scoped —
+        the results endpoints reject requests without the scope. The hotdata
+        0.6.0 SDK exposes (and requires) ``x_database_id`` on the Arrow
+        helper directly.
+        """
+        arrow = ArrowResultsApi(self._runtime.api)
+        deadline = time.monotonic() + self._QUERY_TIMEOUT_SECONDS
+        while True:
+            try:
+                return arrow.get_result_arrow(result_id, x_database_id=database_id)
+            except ResultNotReadyError:
+                # Waiting on the run should already have made this unreachable:
+                # a run reports `succeeded` only once its result is saved and
+                # ready. Tolerating it anyway costs nothing and removes the need
+                # to take that ordering on trust. The Arrow endpoint answers a
+                # result that is not ready with a small refusal rather than with
+                # data, so waiting here is cheap in the way waiting on the JSON
+                # result body -- which is what this change removed -- is not.
+                if time.monotonic() >= deadline:
+                    raise
+                time.sleep(self._POLL_INTERVAL_SECONDS)
+
+    def _query_database_scoped(self, sql: str, *, database_id: str) -> str | None:
+        raw = QueryApi(self._runtime.api).query(
+            # Asked asynchronously because this caller wants a result id, not
+            # rows. A synchronous submit always builds an inline preview of the
+            # result and sends it -- megabytes, on a path that then reads the
+            # whole result as Arrow anyway and never looks at the preview. The
+            # async reply carries a run id and nothing else, and there is no way
+            # to suppress the preview on a synchronous one.
+            #
+            # It also settles the types: the preview is JSON, which has no Arrow
+            # schema and renders non-finite floats as null, so it could not have
+            # substituted for the Arrow fetch even when it holds every row.
+            #
+            # `var_async` is the generated SDK's spelling of the wire field
+            # `async`, which is a Python keyword and so cannot be the attribute
+            # name.
+            QueryRequest(sql=sql, var_async=True),
+            x_database_id=database_id,
+        )
+        # Both reply shapes carry `query_run_id`, and the run is the readiness
+        # signal for either -- a synchronous reply (which `async_after_ms` can
+        # still produce) returns rows inline but goes on saving the full result
+        # in the background, so it is not the finish line either.
+        if isinstance(raw, (QueryResponse, AsyncQueryResponse)):
+            return self._await_query_run(raw.query_run_id, database_id=database_id)
+        # Returning nothing here would read as an empty table: `fetch_table`
+        # answers `None`, `fetch_table_rows` turns that into `[]`, and a
+        # read-modify-write load would write only its new batch over rows it
+        # believed were not there. A reply shape this client does not know is a
+        # reason to stop, not to report emptiness. `HotdataClient` raises on the
+        # same condition.
+        raise RuntimeError(f"Unexpected query response type: {type(raw)!r}")
+
+    def _await_query_run(self, query_run_id: str, *, database_id: str) -> str | None:
+        """Wait for a query run to finish; return the result id it produced.
+
+        The run is the whole wait. A run turns `succeeded` only after its result
+        has been saved and is `ready`, so `succeeded` needs no second check
+        against the result -- and asking the result endpoint instead would mean
+        downloading the entire result to read one field, which the server
+        refuses outright (413/429) once the result is large enough.
+
+        `result_id` comes off the run rather than off the query reply because a
+        `succeeded` run reports none when every row came back inline but the
+        result could not be saved for later retrieval.
+        """
+        runs = QueryRunsApi(self._runtime.api)
+        deadline = time.monotonic() + self._QUERY_TIMEOUT_SECONDS
+        last_status: str | None = None
+        while time.monotonic() < deadline:
+            # Runs (like results) of database-scoped queries are database-scoped.
+            run = runs.get_query_run(query_run_id, x_database_id=database_id)
+            last_status = run.status
+            if run.status == "succeeded":
+                if run.result_id is None:
+                    # A run succeeds with no result id when its rows were
+                    # returned inline but the result could not be saved for
+                    # later retrieval. Returning nothing here would surface as
+                    # an empty table -- `fetch_table` answers `None`, and
+                    # `fetch_table_rows` turns that into `[]`, which is the same
+                    # answer it gives for a table that does not exist. A
+                    # read-modify-write load would then read no existing rows
+                    # and write only its new batch, dropping what was there.
+                    # Terminal rather than transient: re-running the query
+                    # cannot save a result that was already discarded.
+                    # `getattr` because this runs while building an error: if
+                    # the field ever goes away, losing the explanation is a far
+                    # better outcome than an AttributeError replacing the raise.
+                    warning = getattr(run, "warning_message", None)
+                    raise RuntimeError(
+                        f"Query run {query_run_id} succeeded but its result was not "
+                        f"saved, so the table cannot be read"
+                        + (f": {warning}" if warning else "")
+                    )
+                return run.result_id
+            if run.status == "interrupted":
+                # Terminal, but the server lost the run rather than rejecting
+                # the query, so it is the one failure here worth re-running.
+                # Raised pre-classified: `classify_sdk_error` cannot tell this
+                # apart from an ordinary RuntimeError and would call it terminal.
+                raise HotdataTransientError(
+                    run.error_message or f"Query run {query_run_id} was interrupted"
+                )
+            if run.status == "failed":
+                raise RuntimeError(run.error_message or f"Query run {query_run_id} failed")
+            # Any other status keeps polling, including one this client has never
+            # seen. Treating an unrecognised status as terminal is the cheaper
+            # failure to diagnose and by far the more expensive one to suffer: a
+            # single status added upstream would then fail every query at once,
+            # where waiting costs one slow call. What made `interrupted`
+            # expensive was not the waiting, it was that the timeout never said
+            # which status it had waited on -- so the message now carries it.
+            time.sleep(self._POLL_INTERVAL_SECONDS)
+        raise TimeoutError(
+            f"Query run {query_run_id} did not finish within "
+            f"{self._QUERY_TIMEOUT_SECONDS}s (last status: {last_status})"
+        )
+
+    def fetch_table_rows(self, *, database: str, schema: str, table: str) -> list[dict[str, Any]]:
+        result = self.fetch_table(database=database, schema=schema, table=table)
+        return result.to_pylist() if result is not None else []
+
+    def upload_parquet(self, path: str) -> str:
+        return self._request_with_retry(lambda: self._runtime.upload_parquet(path))
+
+    def load_managed_table(
+        self,
+        database: str,
+        table: str,
+        *,
+        schema: str,
+        upload_id: str,
+        mode: ManagedLoadMode = "replace",
+        key: list[str] | None = None,
+    ) -> LoadManagedTableResult:
+        # Retryable in every mode, append included. A retry re-sends the SAME
+        # upload_id, and the server keys a receipt on it: a replay returns the
+        # committed result rather than applying the load a second time. So the
+        # invariant that makes this safe is the upload id, not the mode — a
+        # caller that re-stages the upload between attempts mints a new id,
+        # loses the receipt, and a retried append would then duplicate rows.
+        # This client stages once, in upload_parquet, outside the operation
+        # retried here. `HotdataClient.load_managed_table(file=...)` uploads
+        # inside the call and so does not hold the invariant; it is unwrapped,
+        # and retrying an append through it is the caller's to justify.
+        #
+        # `key` is the merge key for delete/update/upsert loads: when set it is
+        # matched per-load instead of a key declared at table creation. Omit it
+        # to use the table's declared key. Ignored for replace/append.
+        return self._request_with_retry(
+            lambda: self._runtime.load_managed_table(
+                database,
+                table,
+                schema=schema,
+                upload_id=upload_id,
+                mode=mode,
+                key=key,
+            )
+        )
+
+    def _request_with_retry(self, operation: Callable[[], T]) -> T:
+        max_attempts = self._max_retries
+        for attempt in range(1, max_attempts + 1):
+            try:
+                return operation()
+            except Exception as error:
+                mapped_error = classify_sdk_error(error.__cause__ or error)
+                if isinstance(mapped_error, HotdataTransientError) and attempt < max_attempts:
+                    time.sleep(self._retry_delay(attempt, mapped_error.retry_after_seconds))
+                    continue
+                raise mapped_error from error
+        raise RuntimeError("No retry attempts configured")
+
+    def _retry_delay(self, attempt: int, retry_after_seconds: float | None) -> float:
+        """A linear ramp, floored by the server's Retry-After and spread by jitter.
+
+        Retry-After is a floor rather than a replacement: it says how long the
+        condition just refused typically lasts, while the ramp is what gives up
+        eventually, and taking the larger of the two honours both. It is capped
+        like the ramp so a hostile or mistaken header cannot park an attempt for
+        an hour.
+
+        Jitter is added on top and never subtracted, so a stated Retry-After is
+        not undercut. It matters because the callers that collide are the ones
+        that started together: writers refused by one table's lock would retry
+        in lockstep on an identical ramp and re-collide every time.
+        _MAX_BACKOFF_SECONDS caps the ramp, deliberately not the jitter above
+        it — clamping the total would flatten every late attempt onto the same
+        value and re-correlate exactly the waits that most need spreading.
+        """
+        base = min(self._retry_backoff_seconds * attempt, self._MAX_BACKOFF_SECONDS)
+        if retry_after_seconds is not None:
+            base = max(base, min(retry_after_seconds, self._MAX_BACKOFF_SECONDS))
+        return base * (1.0 + random.random() * self._RETRY_JITTER_FRACTION)

{hotdata_framework-0.12.1 → hotdata_framework-0.14.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hotdata-framework"
-version = "0.12.1"
+version = "0.14.0"
 description = "Python framework for building Hotdata integrations: workspace runtime, query execution, and managed databases"
 readme = "README.md"
 requires-python = ">=3.10"