renovara-sql 0.2.0__py3-none-any.whl

renovara_sql-0.2.0.dist-info/METADATA

@@ -0,0 +1,119 @@
+Metadata-Version: 2.5
+Name: renovara-sql
+Version: 0.2.0
+Summary: Python client for the Renovara SQL API: DuckDB SQL over Australian NEM and east coast gas market data from AEMO
+Project-URL: Homepage, https://renovara.co/sql-api
+Project-URL: Documentation, https://renovara.co/how-renovara-sql-works
+Project-URL: API description, https://sql.api.renovara.co/v1/openapi.json
+Author-email: "Scoptico Pty Ltd (Renovara)" <info@renovara.co>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: aemo,australia,duckdb,electricity,energy,gas,nem,sql
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Provides-Extra: pandas
+Requires-Dist: pandas; extra == 'pandas'
+Requires-Dist: pyarrow; extra == 'pandas'
+Description-Content-Type: text/markdown
+
+# Renovara SQL API — Python client
+
+Read-only SQL over Australian National Electricity Market (NEM) and east
+coast gas market data from AEMO, from your own code. The SQL is DuckDB's.
+
+```bash
+pip install renovara-sql
+```
+
+## See what the data is — no credentials needed
+
+```python
+from renovara_sql import Client
+
+public = Client("https://sql.api.renovara.co/v1")
+print(public.guide()[:2000])       # which table answers what, and the traps
+for t in public.tables():          # every table, one line each
+    print(t["table"], "-", t["description"][:80])
+print(public.table("silver_dispatchis_reports_dispatch_price")["columns"][:5])
+```
+
+## Credentials
+
+Your Renovara credentials page gives you a file:
+
+```json
+{
+  "api_url": "https://sql.api.renovara.co/v1/",
+  "token_url": "https://auth.api.renovara.co/oauth2/token",
+  "client_id": "…",
+  "client_secret": "…",
+  "scope": "renovara-sql/query"
+}
+```
+
+Keep it private: anyone holding it can query as your organisation. It is an
+OAuth client-credentials pair, exchanged for an access token that lasts an
+hour; the client keeps the token for the hour and renews it for you.
+
+## Run queries
+
+```python
+from renovara_sql import Client
+
+client = Client.from_file("~/.config/renovara-sql/acme.json")
+result = client.query("""
+    SELECT REGIONID, round(avg(RRP), 2) AS avg_price
+    FROM silver_dispatchis_reports_dispatch_price
+    WHERE SETTLEMENTDATE >= current_timestamp + INTERVAL 10 HOUR - INTERVAL 7 DAY
+      AND INTERVENTION = 0
+    GROUP BY 1 ORDER BY 1
+""")
+print(result.column_names, result.rows)
+print(result.usage)                 # your plan, query time used, statements today
+frame = result.to_pandas()          # needs pandas; pyarrow for exact types
+result.download("prices.parquet")   # the full result, any size
+```
+
+`python quickstart.py <credentials file>` runs a short tour.
+
+## What to know
+
+- **DuckDB SQL**, one `SELECT` per request (or `EXPLAIN SELECT …` for the
+  plan). `client.guide()` lists the differences from Databricks SQL. Tables
+  answer to their bare names or to `external_data.nemweb.<table>`.
+- **Times**: `SETTLEMENTDATE` and the other NEM timestamps are AEST market
+  time (UTC+10, no daylight saving) already. Don't shift them. "Now" in market
+  time is `current_timestamp + INTERVAL 10 HOUR`.
+- **Long queries**: `query()` returns when the query finishes, polling if the
+  server needs more than ~20 s. A statement stops at 300 s. For control, use
+  `submit(sql)`, `status(statement_id)` and `cancel(statement_id)`.
+- **Results**: up to 10,000 rows come back inline as `result.rows`; every
+  result is also a Parquet file at `result.parquet_url` for an hour. Over
+  5,000,000 rows a statement is refused: aggregate in SQL. Statements and
+  results last 24 hours.
+- **Retries are safe**: `query()` sends an idempotency key, so a retry never
+  runs a statement twice.
+- **Errors** raise `QueryFailed` (a statement that ran and failed: `code` is
+  `QUERY_REJECTED`, `SQL_ERROR`, `QUERY_TIMEOUT`, `RESULT_TOO_LARGE` or
+  `CANCELLED`) or `RenovaraError` (a refused request, e.g. `DAILY_LIMIT`,
+  `ALLOWANCE_USED`). Each has a `docs_url`.
+
+## Without the client
+
+```bash
+TOKEN=$(curl -s -u "$CLIENT_ID:$CLIENT_SECRET" -d grant_type=client_credentials \
+  -d scope=renovara-sql/query "$TOKEN_URL" | jq -r .access_token)
+# Keep $TOKEN for its hour: each one issued is billed, so not one per query.
+
+curl -s -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
+  -H "Idempotency-Key: $(uuidgen)" \
+  -d '{"sql": "SELECT count(*) FROM silver_dispatchis_reports_dispatch_price", "wait_seconds": 20}' \
+  "${API_URL}statements"
+# -> {"statement_id": "…", "state": "SUCCEEDED", "result": {"columns": …, "rows": …}}
+#    or "state": "RUNNING": then GET ${API_URL}statements/<statement_id> until it isn't
+```
+
+The API describes itself: `GET ${API_URL}` and `GET ${API_URL}openapi.json`.

renovara_sql-0.2.0.dist-info/RECORD

@@ -0,0 +1,5 @@
+renovara_sql.py,sha256=sZ3BKDzGRwQGcH9I6V6AsSmG8lSUt5FeEwSm_77YVl8,13248
+renovara_sql-0.2.0.dist-info/METADATA,sha256=LePR-Ta4edL0LfT-wRMb6USCeh3fjb7HGUQA3ijNOcM,4803
+renovara_sql-0.2.0.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
+renovara_sql-0.2.0.dist-info/licenses/LICENSE,sha256=yYZwktV_ENrfr7x1YCg_UG9Jot3rBXmctGS4j4v5e8w,1095
+renovara_sql-0.2.0.dist-info/RECORD,,

renovara_sql-0.2.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.33.0
+Root-Is-Purelib: true
+Tag: py3-none-any

renovara_sql-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Scoptico Pty Ltd (trading as Renovara)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

renovara_sql.py

@@ -0,0 +1,284 @@
+"""Python client for the Renovara SQL API.
+
+One file, standard library only. pandas (and pyarrow, for exact column types)
+are needed only for Result.to_pandas().
+
+    from renovara_sql import Client
+
+    client = Client.from_file("~/.config/renovara-sql/acme.json")
+    result = client.query("SELECT REGIONID, avg(RRP) FROM silver_dispatchis_reports_dispatch_price "
+                          "WHERE SETTLEMENTDATE >= current_timestamp + INTERVAL 10 HOUR - INTERVAL 7 DAY "
+                          "GROUP BY 1")
+    print(result.rows)
+
+Queries are DuckDB SQL, not Databricks SQL: read client.guide() first, which
+lists the differences. NEM timestamps (SETTLEMENTDATE and friends) are AEST
+market time already; the server's session zone is UTC so they are shown and
+compared unshifted.
+
+The guide and the tables' descriptions need no credentials:
+
+    Client("https://sql.api.renovara.co/v1").table("silver_dispatchis_reports_dispatch_price")
+
+How a query runs: query() submits it and the server waits up to 20 s. Most
+queries finish inside that and come back in one call. A longer one comes back
+with a statement ID, and query() polls until it finishes. Use submit(),
+status() and cancel() to do that yourself.
+"""
+from __future__ import annotations
+
+import base64
+import io
+import json
+import os
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+import uuid
+from dataclasses import dataclass, field
+from pathlib import Path
+
+__version__ = "0.2.0"
+
+TERMINAL = {"SUCCEEDED", "FAILED"}
+# Retry-After longer than this is a limit to report, not a pause to sit out.
+MAX_RETRY_WAIT_S = 60
+
+
+class RenovaraError(Exception):
+    """A request the API refused, with its error code, message and docs link."""
+
+    def __init__(self, status: int, code: str, message: str, docs_url: str | None = None,
+                 retry_after: float | None = None):
+        super().__init__(f"{code}: {message}" if code else message)
+        self.status, self.code, self.message = status, code, message
+        self.docs_url, self.retry_after = docs_url, retry_after
+
+
+class QueryFailed(RenovaraError):
+    """The query ran and failed: rejected, a SQL error, timed out or cancelled."""
+
+    def __init__(self, statement_id: str, code: str, message: str, docs_url: str | None = None):
+        super().__init__(200, code, message, docs_url)
+        self.statement_id = statement_id
+
+
+@dataclass
+class Result:
+    statement_id: str
+    columns: list[dict]
+    row_count: int
+    parquet_url: str
+    rows: list[list] | None = None
+    query_ms: int | None = None
+    usage: dict | None = None
+    raw: dict = field(default_factory=dict, repr=False)
+
+    @property
+    def column_names(self) -> list[str]:
+        return [c["name"] for c in self.columns]
+
+    def download(self, path: str | os.PathLike) -> Path:
+        """Save the full result as Parquet. The link works for an hour."""
+        path = Path(path)
+        with urllib.request.urlopen(self.parquet_url, timeout=300) as response:
+            path.write_bytes(response.read())
+        return path
+
+    def to_pandas(self):
+        """The result as a DataFrame.
+
+        With pyarrow installed this reads the Parquet file, keeping exact
+        types. Without it, it builds the frame from the inline rows and
+        parses timestamp columns.
+        """
+        import pandas as pd
+
+        try:
+            import pyarrow  # noqa: F401
+        except ImportError:
+            if self.rows is None:
+                raise RuntimeError(
+                    f"This result has {self.row_count:,} rows, too many to return inline. "
+                    "Install pyarrow to read its Parquet file.") from None
+            frame = pd.DataFrame(self.rows, columns=self.column_names)
+            for column in self.columns:
+                if column["type"].startswith(("TIMESTAMP", "DATE")):
+                    frame[column["name"]] = pd.to_datetime(frame[column["name"]])
+            return frame
+        with urllib.request.urlopen(self.parquet_url, timeout=300) as response:
+            return pd.read_parquet(io.BytesIO(response.read()))
+
+
+class Client:
+    def __init__(self, api_url: str, token_url: str | None = None, client_id: str | None = None,
+                 client_secret: str | None = None, scope: str = "renovara-sql/query",
+                 timeout: float = 30):
+        self.api_url = api_url.rstrip("/") + "/"
+        self.token_url = token_url
+        self.client_id = client_id
+        self._client_secret = client_secret
+        self.scope = scope
+        self.timeout = timeout
+        self._token: str | None = None
+        self._token_expires = 0.0
+
+    @classmethod
+    def from_file(cls, path: str | os.PathLike) -> "Client":
+        """Read the credentials file from your Renovara credentials page."""
+        data = json.loads(Path(path).expanduser().read_text())
+        return cls(data["api_url"], data["token_url"], data["client_id"],
+                   data["client_secret"], data.get("scope", "renovara-sql/query"))
+
+    @classmethod
+    def from_env(cls) -> "Client":
+        """RENOVARA_SQL_API_URL, _TOKEN_URL, _CLIENT_ID and _CLIENT_SECRET."""
+        e = os.environ
+        return cls(e["RENOVARA_SQL_API_URL"], e["RENOVARA_SQL_TOKEN_URL"],
+                   e["RENOVARA_SQL_CLIENT_ID"], e["RENOVARA_SQL_CLIENT_SECRET"])
+
+    # -- auth ------------------------------------------------------------
+
+    def _access_token(self) -> str:
+        """An OAuth client-credentials token, kept for its hour.
+
+        Renewed a minute before it expires, not per request: each token
+        Renovara's sign-in service issues is billed, so a fresh one per query
+        would cost more than the query.
+        """
+        if self._token and time.time() < self._token_expires - 60:
+            return self._token
+        if not (self.token_url and self.client_id and self._client_secret):
+            raise RenovaraError(401, "UNAUTHORIZED",
+                                "Running queries needs credentials: Client.from_file(...) with "
+                                "the file from your Renovara credentials page.")
+        basic = base64.b64encode(f"{self.client_id}:{self._client_secret}".encode()).decode()
+        request = urllib.request.Request(
+            self.token_url, method="POST",
+            data=urllib.parse.urlencode({"grant_type": "client_credentials",
+                                         "scope": self.scope}).encode(),
+            headers={"Authorization": f"Basic {basic}",
+                     "Content-Type": "application/x-www-form-urlencoded"})
+        try:
+            with urllib.request.urlopen(request, timeout=self.timeout) as response:
+                body = json.load(response)
+        except urllib.error.HTTPError as e:
+            raise RenovaraError(e.code, "AUTH_FAILED",
+                                f"Could not get an access token: {e.read().decode()[:300]}") from None
+        self._token = body["access_token"]
+        self._token_expires = time.time() + int(body.get("expires_in", 3600))
+        return self._token
+
+    # -- HTTP ------------------------------------------------------------
+
+    def _call(self, method: str, path: str, body: dict | None = None, timeout: float | None = None,
+              headers: dict | None = None, auth: bool = True, text: bool = False):
+        for attempt in range(5):
+            sent = {"User-Agent": f"renovara-sql-python/{__version__}", **(headers or {})}
+            if body is not None:
+                sent["Content-Type"] = "application/json"
+            if auth:
+                sent["Authorization"] = f"Bearer {self._access_token()}"
+            request = urllib.request.Request(
+                self.api_url + path, method=method,
+                data=json.dumps(body).encode() if body is not None else None, headers=sent)
+            try:
+                with urllib.request.urlopen(request, timeout=timeout or self.timeout) as response:
+                    return response.read().decode() if text else json.load(response)
+            except urllib.error.HTTPError as e:
+                raw = e.read().decode(errors="replace")
+                try:
+                    error = json.loads(raw).get("error") or {}
+                except ValueError:
+                    error = {}
+                error = error if isinstance(error, dict) else {}
+                retry_after = float(e.headers.get("Retry-After") or 0) or None
+                if e.code == 401 and auth and attempt == 0:
+                    self._token = None  # expired or revoked mid-flight: renew once
+                    continue
+                retryable = e.code in (502, 503, 504) or (
+                    e.code == 429 and (retry_after or 1) <= MAX_RETRY_WAIT_S)
+                if retryable and attempt < 4:
+                    time.sleep(retry_after or 2 ** attempt)
+                    continue
+                raise RenovaraError(e.code, error.get("code", "HTTP_ERROR"),
+                                    error.get("message", raw[:300] or e.reason),
+                                    error.get("docs_url"), retry_after) from None
+        raise RenovaraError(429, "THROTTLED", "Too many requests; try again shortly.")
+
+    # -- what the data is (no credentials needed) ----------------------------
+
+    def guide(self, domain: str = "nem") -> str:
+        """The analyst guide (markdown): which table answers what, the traps,
+        worked SQL, and how DuckDB differs from the Databricks SQL it quotes.
+        domain is "nem" (electricity) or "gas"."""
+        return self._call("GET", f"guide?domain={urllib.parse.quote(domain)}", auth=False, text=True)
+
+    def tables(self) -> list[dict]:
+        """Every table you can query, each with a one-line description."""
+        return self._call("GET", "tables", auth=False)["tables"]
+
+    def table(self, name: str) -> dict:
+        """A table's columns with their meaning, primary key and measured coverage."""
+        return self._call("GET", f"tables/{urllib.parse.quote(name)}", auth=False)
+
+    def reference(self, path: str, domain: str = "nem") -> dict:
+        """A file the guide points to, by its references/ path."""
+        query = urllib.parse.urlencode({"path": path, "domain": domain})
+        return self._call("GET", f"reference?{query}", auth=False)
+
+    # -- running SQL -----------------------------------------------------
+
+    def submit(self, sql: str, wait_seconds: float = 0, idempotency_key: str | None = None) -> dict:
+        """Start a query. With wait_seconds > 0 (max 25) the server waits for it first.
+
+        With an idempotency_key, sending the same key again returns the first
+        statement instead of running the query twice.
+        """
+        headers = {"Idempotency-Key": idempotency_key} if idempotency_key else None
+        return self._call("POST", "statements", {"sql": sql, "wait_seconds": wait_seconds},
+                          timeout=max(self.timeout, wait_seconds + 10), headers=headers)
+
+    def status(self, statement_id: str) -> dict:
+        """A statement's state, and its result once it has SUCCEEDED."""
+        return self._call("GET", f"statements/{statement_id}")
+
+    def cancel(self, statement_id: str) -> dict:
+        """Stop a pending or running statement. Its state becomes FAILED, CANCELLED."""
+        return self._call("POST", f"statements/{statement_id}/cancel", {})
+
+    def query(self, sql: str, timeout: float = 900, on_progress=None) -> Result:
+        """Run a query and return its result, polling if it takes a while.
+
+        Safe to retry: the submit carries an idempotency key, so a retry after
+        a dropped connection returns the same statement. on_progress(statement)
+        is called after each poll, if given.
+        """
+        statement = self.submit(sql, wait_seconds=20, idempotency_key=uuid.uuid4().hex)
+        usage = statement.get("usage")
+        deadline = time.monotonic() + timeout
+        pause = 1.0
+        while statement["state"] not in TERMINAL:
+            if time.monotonic() > deadline:
+                raise TimeoutError(f"Statement {statement['statement_id']} still "
+                                   f"{statement['state']} after {timeout:.0f} s; "
+                                   "poll it later with status(), or cancel() it.")
+            time.sleep(pause)
+            pause = min(pause * 1.5, 5.0)
+            statement = self.status(statement["statement_id"])
+            if on_progress:
+                on_progress(statement)
+        return self._result(statement, usage)
+
+    @staticmethod
+    def _result(statement: dict, usage: dict | None = None) -> Result:
+        if statement["state"] == "FAILED":
+            error = statement.get("error") or {}
+            raise QueryFailed(statement["statement_id"], error.get("code", "FAILED"),
+                              error.get("message", "The query failed."), error.get("docs_url"))
+        result = statement["result"]
+        return Result(statement_id=statement["statement_id"], columns=result["columns"],
+                      row_count=result["row_count"], parquet_url=result["parquet_url"],
+                      rows=result.get("rows"), query_ms=statement.get("query_ms"),
+                      usage=usage or statement.get("usage"), raw=statement)