periplus-python-sdk 0.6.1__py3-none-any.whl → 0.8.0__py3-none-any.whl

periplus_python_sdk-0.8.0.dist-info/METADATA

@@ -0,0 +1,70 @@
+Metadata-Version: 2.4
+Name: periplus-python-sdk
+Version: 0.8.0
+Summary: Read-only Python client for the public Periplus query API
+License-Expression: Apache-2.0
+Project-URL: Repository, https://github.com/elei-io/periplus
+Project-URL: Issues, https://github.com/elei-io/periplus/issues
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: httpx>=0.28
+Requires-Dist: pydantic<3,>=2.12
+Requires-Dist: sqlalchemy<3,>=2.0
+Provides-Extra: notebook
+Requires-Dist: marimo[sql]>=0.24.1; extra == "notebook"
+Dynamic: license-file
+
+# Periplus Python SDK
+
+Read-only access to the public ClickHouse corpus through the Periplus HTTP API.
+Install the SDK from the same checkout as your deployment:
+
+```sh
+python -m pip install ./packages/periplus-python-sdk
+```
+
+```python
+from periplus_sdk import Client
+
+with Client("http://localhost:8080") as client:
+    result = client.execute(
+        "SELECT capture_id, url FROM public_v1.captures LIMIT ?", [10]
+    )
+    print(result.columns, result.rows)
+```
+
+Use `PERIPLUS_PUBLIC_URL` to omit the URL argument. No database credentials or
+service token are needed for the public gateway. `AsyncClient` provides async
+methods. `prepare` validates/explains SQL; `execute` returns typed columns, rows,
+truncation and query metadata; `helpers` describes the installed public views.
+
+The public schema is `public_v1`. HTML joins use `document_id` plus node index;
+`document_id` identifies retained bytes and their HTML interpretation. Raw-byte hashes stay internal.
+There is one query endpoint, with no experimental fallback. Client errors preserve
+server categories and do not automatically retry executed queries.
+
+For notebook/SQLAlchemy integration:
+
+```python
+from periplus_sdk import sql_api
+from sqlalchemy import text
+
+engine = sql_api.create_engine(base_url="http://localhost:8080")
+with engine.connect() as connection:
+    print(connection.execute(text("SELECT url FROM public_v1.captures LIMIT 5")).all())
+engine.dispose()
+```
+
+Marimo discovers the five public views through the helper catalogue. Column
+reflection uses `SELECT * ... LIMIT 0` to retrieve native types without reading
+corpus rows; no `SHOW`, `DESCRIBE`, or system-table access is required.
+
+The DB-API connection advertises the ClickHouse dialect and converts native
+nullable integer, decimal, date and datetime types. Nested types retain JSON wire
+values. It is read-only: there are no client transactions or writable sessions.
+Streaming cursors expose incomplete/truncated results explicitly; configure
+`allow_partial` only when partial results suit the application.
+
+See [the public schema](../../docs/SCHEMA.md) and [query boundary](../../docs/QUERY.md).

periplus_python_sdk-0.8.0.dist-info/RECORD

@@ -0,0 +1,16 @@
+periplus_python_sdk-0.8.0.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
+periplus_python_sdk-0.8.0.dist-info/licenses/NOTICE,sha256=bhbYSqcUB3U_P1-XzloiT81JGniqoYaRLxNkQ1Pm9MQ,52
+periplus_sdk/__init__.py,sha256=WimXYlPB6tCimBO4VSwhcp00dwSL87jMmMuQ4-kINfM,546
+periplus_sdk/client.py,sha256=trcsOL4hnpDByMr8wZ9iz2vzxQWCW60vSYTmpqCj1Gs,7745
+periplus_sdk/dbapi.py,sha256=yDrL2CV7tI3Pawp9hN2czFYqrbXF8ret0Jk4VHRHoYg,12014
+periplus_sdk/errors.py,sha256=rB1n-v8Hc2tu2dtHivz-MlqsCoRC5pTcWogTfM7SMLw,855
+periplus_sdk/py.typed,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
+periplus_sdk/sql_api.py,sha256=vRQXzBIcJOUg_eZzzRs7cpE3xygxK5Jo7YfYWN4NxBg,1868
+periplus_sdk/sqlalchemy.py,sha256=e9j4nI_JQ5EDwMOK6WawX1W_QrOnaY3Wa8LchIIJJio,5567
+periplus_sdk/stream.py,sha256=7gOYOMpfeB7NcxQgWjJ6f5NvymYTWhYbPkVdCijSlP8,5218
+periplus_sdk/types.py,sha256=yh_xHwn6TwC45r8eWhxtlrz0w86M5lvhmPOS3QJbXiU,1253
+periplus_python_sdk-0.8.0.dist-info/METADATA,sha256=7A_Jpwq8NJpskMpwQORGPhx-xbMHn_nehsqslBb_6c8,2730
+periplus_python_sdk-0.8.0.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
+periplus_python_sdk-0.8.0.dist-info/entry_points.txt,sha256=Pr14L_7AhLinq-4qDxB4awFVubrvR1BEfkaFVRpdCbU,73
+periplus_python_sdk-0.8.0.dist-info/top_level.txt,sha256=o41t5TzwgoxzSmbKoP6olWW1FyEAGWVYTjeoKadBK40,13
+periplus_python_sdk-0.8.0.dist-info/RECORD,,

periplus_sdk/client.py

@@ -79,10 +79,11 @@ def _payload(sql: str, parameters: Sequence[JsonValue] | None, schema_version: s
 class Client:
     """Reusable synchronous public query client. Close it or use a with block."""
 
-    def __init__(self, base_url: str | None = None, *, timeout: float = 140, mode: Literal["stable", "experimental"] = "stable"):
-        if mode not in {"stable", "experimental"}:
-            raise ConfigurationError("mode must be stable or experimental.")
-        self._query_path = "api/query/experimental/" if mode == "experimental" else "api/query/"
+    def __init__(self, base_url: str | None = None, *, timeout: float = 620, mode: Literal["stable"] = "stable"):
+        if mode not in {"stable"}:
+            raise ConfigurationError("Only the stable public catalogue is available.")
+        self.schema_version = "public_v1"
+        self._query_path = "api/query/"
         self._http = httpx.Client(**_options(base_url, timeout))
 
     def __enter__(self) -> Client:
@@ -101,11 +102,30 @@ class Client:
             raise TransportError("Could not complete the public query request.") from None
         return _decode(response, model)
 
-    def prepare(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str = "public_v1") -> PreparedQuery:
-        return self._request("POST", "prep", PreparedQuery, json=_payload(sql, parameters, schema_version))
+    def prepare(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str | None = None) -> PreparedQuery:
+        return self._request("POST", "prep", PreparedQuery, json=_payload(sql, parameters, schema_version if schema_version is not None else self.schema_version))
 
-    def execute(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str = "public_v1") -> QueryResult:
-        return self._request("POST", "exec", QueryResult, json=_payload(sql, parameters, schema_version))
+    def execute(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str | None = None) -> QueryResult:
+        return self._request("POST", "exec", QueryResult, json=_payload(sql, parameters, schema_version if schema_version is not None else self.schema_version))
+
+    def stream(self, sql: str, parameters: Sequence[JsonValue] | None = None, *,
+               schema_version: str | None = None, allow_partial: bool = False):
+        """Stream batches from one snapshot; use as a context manager for early exit."""
+        from .stream import MEDIA_TYPE, QueryStream
+        try:
+            request = self._http.build_request("POST", self._query_path + "exec",
+                headers={"accept": MEDIA_TYPE}, json=_payload(sql, parameters, schema_version if schema_version is not None else self.schema_version))
+            response = self._http.send(request, stream=True)
+            try:
+                if not response.is_success:
+                    response.read()
+                    _decode(response, QueryResult)
+                return QueryStream(response, allow_partial=allow_partial)
+            except BaseException:
+                response.close()
+                raise
+        except httpx.RequestError:
+            raise TransportError("Could not open the public query stream.") from None
 
     def helpers(self) -> QueryHelpers:
         return self._request("GET", "helpers", QueryHelpers)
@@ -114,10 +134,11 @@ class Client:
 class AsyncClient:
     """Reusable asynchronous public query client. Use an async with block."""
 
-    def __init__(self, base_url: str | None = None, *, timeout: float = 140, mode: Literal["stable", "experimental"] = "stable"):
-        if mode not in {"stable", "experimental"}:
-            raise ConfigurationError("mode must be stable or experimental.")
-        self._query_path = "api/query/experimental/" if mode == "experimental" else "api/query/"
+    def __init__(self, base_url: str | None = None, *, timeout: float = 620, mode: Literal["stable"] = "stable"):
+        if mode not in {"stable"}:
+            raise ConfigurationError("Only the stable public catalogue is available.")
+        self.schema_version = "public_v1"
+        self._query_path = "api/query/"
         self._http = httpx.AsyncClient(**_options(base_url, timeout))
 
     async def __aenter__(self) -> AsyncClient:
@@ -136,11 +157,11 @@ class AsyncClient:
             raise TransportError("Could not complete the public query request.") from None
         return _decode(response, model)
 
-    async def prepare(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str = "public_v1") -> PreparedQuery:
-        return await self._request("POST", "prep", PreparedQuery, json=_payload(sql, parameters, schema_version))
+    async def prepare(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str | None = None) -> PreparedQuery:
+        return await self._request("POST", "prep", PreparedQuery, json=_payload(sql, parameters, schema_version if schema_version is not None else self.schema_version))
 
-    async def execute(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str = "public_v1") -> QueryResult:
-        return await self._request("POST", "exec", QueryResult, json=_payload(sql, parameters, schema_version))
+    async def execute(self, sql: str, parameters: Sequence[JsonValue] | None = None, *, schema_version: str | None = None) -> QueryResult:
+        return await self._request("POST", "exec", QueryResult, json=_payload(sql, parameters, schema_version if schema_version is not None else self.schema_version))
 
     async def helpers(self) -> QueryHelpers:
         return await self._request("GET", "helpers", QueryHelpers)

periplus_sdk/dbapi.py

@@ -1,7 +1,7 @@
 """Read-only DB-API 2.0 connection over the public query API.
 
-Each execute is an independent server snapshot. Fetching consumes a bounded local
-result, never a remote cursor. Connections and cursors must not be shared by threads.
+Each execute is an independent server snapshot. Fetching consumes a bounded stream.
+Connections and cursors must not be shared by threads.
 """
 from __future__ import annotations
 
@@ -11,11 +11,10 @@ from collections.abc import Sequence
 from datetime import date, datetime, time
 from decimal import Decimal
 from typing import Any, Literal
-import warnings
 
 from .client import Client
 from .errors import ApiError, ConfigurationError, PeriplusError, ResponseError, TransportError
-from .types import QueryResult
+from .stream import StreamResult
 
 apilevel = "2.0"
 threadsafety = 1
@@ -26,10 +25,6 @@ class Warning(builtins.Warning):
     """DB-API warning."""
 
 
-class TruncationWarning(Warning):
-    """The server returned only part of the query result."""
-
-
 class Error(PeriplusError):
     """Base DB-API error; API failures preserve their safe error attributes."""
 
@@ -91,54 +86,47 @@ def TimestampFromTicks(ticks: float) -> datetime:
     return datetime.fromtimestamp(ticks)
 
 
-_INTEGER_TYPES = {"TINYINT", "SMALLINT", "INTEGER", "BIGINT", "HUGEINT", "UTINYINT",
-                  "USMALLINT", "UINTEGER", "UBIGINT", "UHUGEINT", "BIGNUM"}
-_FLOAT_TYPES = {"FLOAT", "DOUBLE", "REAL"}
-_TIME_TYPES = {"TIME", "TIME WITH TIME ZONE", "TIMETZ"}
-_TIMESTAMP_TYPES = {"TIMESTAMP", "TIMESTAMP_S", "TIMESTAMP_MS", "TIMESTAMP_NS",
-                    "TIMESTAMP WITH TIME ZONE", "TIMESTAMPTZ"}
+_INTEGER_TYPES = {f'{prefix}Int{bits}' for prefix in ('', 'U') for bits in (8,16,32,64,128,256)}
+_FLOAT_TYPES = {'Float32', 'Float64'}
+
+
+def _base_type(value: str) -> str:
+    while value.startswith(('Nullable(', 'LowCardinality(')):
+        value = value[value.index('(')+1:-1]
+    return value
 
 
 class _TypeCategory:
-    def __init__(self, names: set[str], prefix: str = ""):
-        self.names, self.prefix = names, prefix
+    def __init__(self, names: set[str], prefixes: tuple[str, ...] = ()):
+        self.names, self.prefixes = names, prefixes
 
     def __eq__(self, other: object) -> bool:
-        return isinstance(other, str) and (other in self.names or bool(self.prefix and other.startswith(self.prefix)))
+        if not isinstance(other, str):return False
+        value = _base_type(other)
+        return value in self.names or value.startswith(self.prefixes)
 
 
-STRING = _TypeCategory({"VARCHAR", "UUID", "JSON", "ENUM"})
-BINARY = _TypeCategory({"BLOB"})
-NUMBER = _TypeCategory(_INTEGER_TYPES | _FLOAT_TYPES | {"BOOLEAN"}, "DECIMAL(")
-DATETIME = _TypeCategory({"DATE"} | _TIME_TYPES | _TIMESTAMP_TYPES)
+STRING = _TypeCategory({'String','UUID','JSON'}, ('FixedString(', 'Enum'))
+BINARY = _TypeCategory(set())
+NUMBER = _TypeCategory(_INTEGER_TYPES | _FLOAT_TYPES | {'Bool'}, ('Decimal',))
+DATETIME = _TypeCategory({'Date','Date32'}, ('DateTime','Time'))
 ROWID = _TypeCategory(set())
 
 
 def _value(value: Any, sql_type: str) -> Any:
-    if value is None:
-        return None
-    if sql_type in _INTEGER_TYPES:
-        return int(value)
-    if sql_type in _FLOAT_TYPES:
-        return float(value)
-    if sql_type.startswith("DECIMAL("):
-        return Decimal(str(value))
-    # Preserve infinities and out-of-range dates rather than clipping them.
-    if sql_type == "DATE":
-        try:
-            return date.fromisoformat(value)
-        except ValueError:
-            return value
-    if sql_type in _TIME_TYPES:
-        return time.fromisoformat(value)
-    if sql_type in _TIMESTAMP_TYPES:
-        try:
-            return datetime.fromisoformat(value)
-        except ValueError:
-            return value
-    if sql_type == "BLOB":
-        return base64.b64decode(value, validate=True)
-    # UUIDs remain strings, and nested/other types retain their JSON wire values.
+    if value is None:return None
+    sql_type = _base_type(sql_type)
+    if sql_type in _INTEGER_TYPES:return int(value)
+    if sql_type in _FLOAT_TYPES:return float(value)
+    if sql_type.startswith('Decimal'):return Decimal(str(value))
+    if sql_type == 'Bool':return value in (True, 1, '1', 'true')
+    if sql_type in ('Date','Date32'):
+        try:return date.fromisoformat(value)
+        except ValueError:return value
+    if sql_type.startswith('DateTime'):
+        try:return datetime.fromisoformat(value)
+        except ValueError:return value
+    if sql_type.startswith('Time'):return time.fromisoformat(value)
     return value
 
 
@@ -147,24 +135,30 @@ def _parameter(value: Any) -> Any:
         return value
     if isinstance(value, (Decimal, date, time)):
         return str(value) if isinstance(value, Decimal) else value.isoformat()
-    raise ProgrammingError("Parameters must be scalar JSON values, Decimal, date, time or datetime; use explicit SQL casts for typed strings.")
+    if isinstance(value, (list, tuple)):
+        if any(isinstance(item, (list, tuple, dict)) for item in value):
+            raise ProgrammingError("Collection parameters must be one-dimensional lists of scalar values.")
+        return [_parameter(item) for item in value]
+    raise ProgrammingError("Parameters must be scalar values or scalar lists; use explicit SQL casts for typed values.")
 
 
 class Connection:
     """Marimo-discoverable, read-only connection; commit is a no-op."""
 
-    dialect = "duckdb"
+    dialect = "clickhouse"
 
-    def __init__(self, base_url: str | None = None, *, timeout: float = 140,
-                 mode: Literal["stable", "experimental"] = "stable",
-                 schema_version: str = "public_v1"):
+    def __init__(self, base_url: str | None = None, *, timeout: float = 620,
+                 mode: Literal["stable"] = "stable",
+                 schema_version: str | None = None, allow_partial: bool = False):
         try:
             self._client = Client(base_url, timeout=timeout, mode=mode)
         except ConfigurationError as exc:
             raise InterfaceError(str(exc)) from exc
-        self.schema_version = schema_version
+        self.schema_version = schema_version if schema_version is not None else ("public_v1")
+        self.allow_partial = allow_partial
+        self._cursors = set()
         self.closed = False
-        self.last_result: QueryResult | None = None
+        self.last_result: StreamResult | None = None
 
     def _check(self) -> None:
         if self.closed:
@@ -172,7 +166,9 @@ class Connection:
 
     def cursor(self) -> Cursor:
         self._check()
-        return Cursor(self)
+        cursor = Cursor(self)
+        self._cursors.add(cursor)
+        return cursor
 
     def execute(self, operation: str, parameters: Sequence[Any] | None = None) -> Cursor:
         cursor = self.cursor()
@@ -192,6 +188,8 @@ class Connection:
 
     def close(self) -> None:
         if not self.closed:
+            for cursor in list(self._cursors):
+                cursor.close()
             self._client.close()
             self.closed = True
             self.last_result = None
@@ -204,25 +202,26 @@ class Connection:
         self.close()
 
 
-def connect(base_url: str | None = None, *, timeout: float = 140,
-            mode: Literal["stable", "experimental"] = "stable",
-            schema_version: str = "public_v1") -> Connection:
-    return Connection(base_url, timeout=timeout, mode=mode, schema_version=schema_version)
+def connect(base_url: str | None = None, *, timeout: float = 620,
+            mode: Literal["stable"] = "stable",
+            schema_version: str | None = None, allow_partial: bool = False) -> Connection:
+    return Connection(base_url, timeout=timeout, mode=mode, schema_version=schema_version, allow_partial=allow_partial)
 
 
 class Cursor:
-    """A buffered result. Metadata stays on result after rows are consumed."""
+    """Incremental cursor. Metadata stays available without retaining consumed rows."""
 
     arraysize = 1
 
     def __init__(self, connection: Connection):
         self.connection = connection
         self.closed = False
-        self.result: QueryResult | None = None
+        self.result: StreamResult | None = None
         self.description: list[tuple[Any, ...]] | None = None
         self.rowcount = -1
         self._rows: list[tuple[Any, ...]] = []
         self._position = 0
+        self._stream = None
 
     def _check(self, *, result: bool = False) -> None:
         self.connection._check()
@@ -233,6 +232,9 @@ class Cursor:
 
     def execute(self, operation: str, parameters: Sequence[Any] | None = None) -> Cursor:
         self._check()
+        if self._stream is not None:
+            self._stream.close()
+        self._stream = None
         self.result, self.description, self.rowcount = None, None, -1
         self._rows, self._position = [], 0
         self.connection.last_result = None
@@ -242,7 +244,9 @@ class Cursor:
             raise ProgrammingError("Use a positional parameter sequence with ? placeholders.")
         values = [_parameter(v) for v in parameters] if parameters is not None else []
         try:
-            result = self.connection._client.execute(operation, values, schema_version=self.connection.schema_version)
+            self._stream = self.connection._client.stream(operation, values, schema_version=self.connection.schema_version,
+                                                          allow_partial=self.connection.allow_partial)
+            result = self._stream.result
         except ApiError as exc:
             error = ProgrammingError if exc.code == "sql_invalid" else OperationalError
             raise error(str(exc), status_code=exc.status_code, code=exc.code,
@@ -251,26 +255,37 @@ class Cursor:
             raise OperationalError(str(exc)) from exc
         except ResponseError as exc:
             raise InterfaceError(str(exc)) from exc
-        if len(result.columns) != len(result.types) or any(len(r) != len(result.columns) for r in result.rows):
-            raise InterfaceError("Query columns, types and rows have inconsistent widths.")
-        try:
-            rows = [tuple(_value(v, t) for v, t in zip(row, result.types, strict=True)) for row in result.rows]
-        except (ValueError, TypeError, ArithmeticError) as exc:
-            raise DataError("Query value does not match its SQL type.") from exc
         self.result = self.connection.last_result = result
         self.description = [(name, kind, None, None, None, None, None)
                             for name, kind in zip(result.columns, result.types, strict=True)]
-        self._rows = rows
-        self.rowcount = -1 if result.truncated else len(rows)
-        if result.truncated:
-            warnings.warn(f"Periplus returned a truncated result ({len(rows)} rows); inspect connection.last_result or cursor.result. Fetching does not retrieve additional rows.",
-                          TruncationWarning, stacklevel=2)
         return self
 
+    def _batch(self):
+        try:
+            batch = next(self._stream)
+            self._rows = [tuple(_value(v, t) for v, t in zip(row, self.result.types, strict=True)) for row in batch]
+            self._position = 0
+            return True
+        except StopIteration:
+            self.rowcount = -1 if self.result.truncated else self.result.row_count
+            self._rows, self._position = [], 0
+            return False
+        except ApiError as exc:
+            raise OperationalError(str(exc), status_code=exc.status_code, code=exc.code,
+                                   retry_after_seconds=exc.retry_after_seconds) from exc
+        except TransportError as exc:
+            raise OperationalError(str(exc)) from exc
+        except ResponseError as exc:
+            raise InterfaceError(str(exc)) from exc
+        except (ValueError, TypeError, ArithmeticError) as exc:
+            self._stream.close()
+            raise DataError("Query value does not match its SQL type.") from exc
+
     def fetchone(self) -> tuple[Any, ...] | None:
         self._check(result=True)
-        if self._position == len(self._rows):
-            return None
+        while self._position == len(self._rows):
+            if not self._batch():
+                return None
         row = self._rows[self._position]
         self._position += 1
         return row
@@ -280,14 +295,17 @@ class Cursor:
         size = self.arraysize if size is None else size
         if not isinstance(size, int) or size < 0:
             raise ProgrammingError("Fetch size must be a non-negative integer.")
-        end = min(self._position + size, len(self._rows))
-        rows = self._rows[self._position:end]
-        self._position = end
+        rows = []
+        for _ in range(size):
+            row = self.fetchone()
+            if row is None:
+                break
+            rows.append(row)
         return rows
 
     def fetchall(self) -> list[tuple[Any, ...]]:
         self._check(result=True)
-        return self.fetchmany(len(self._rows) - self._position)
+        return list(self)
 
     def executemany(self, operation: str, seq_of_parameters: Any) -> None:
         self._check()
@@ -300,6 +318,9 @@ class Cursor:
         self._check()
 
     def close(self) -> None:
+        if self._stream is not None:
+            self._stream.close()
+        self.connection._cursors.discard(self)
         self.closed = True
         self._rows = []
         self.result = None

periplus_sdk/sql_api.py

@@ -3,23 +3,45 @@ from __future__ import annotations
 
 from typing import Literal
 
-from sqlalchemy import create_engine as _create_engine
+from sqlalchemy import create_engine as _create_engine, event
 from sqlalchemy.engine import Engine, URL
 
 
 def create_engine(
     base_url: str | None = None,
     *,
-    mode: Literal["stable", "experimental"] = "stable",
-    timeout: float = 140,
-    schema_version: str = "public_v1",
+    mode: Literal["stable"] = "stable",
+    timeout: float = 620,
+    schema_version: str | None = None,
+    allow_partial: bool = False,
 ) -> Engine:
     """Create a SQLAlchemy engine recognized by marimo and other SQL tools.
 
     The public URL defaults to PERIPLUS_PUBLIC_URL. Connections are opened lazily;
     dispose the engine when finished. Each query uses an independent server snapshot.
     """
-    return _create_engine(
+    engine = _create_engine(
         URL.create("periplus", database="periplus"),
-        connect_args={"base_url": base_url, "mode": mode, "timeout": timeout, "schema_version": schema_version},
+        connect_args={"base_url": base_url, "mode": mode, "timeout": timeout, "schema_version": schema_version, "allow_partial": allow_partial},
     )
+
+    event.listen(engine, "before_execute", _parameters, retval=True)
+    return engine
+
+
+def _parameters(connection, clauseelement, multiparams, params, execution_options):
+    bindings = execution_options.get("periplus_parameters", {})
+    if bindings and not multiparams:
+        names = clauseelement.compile().params
+        params = {**{name: value for name, value in bindings.items() if name in names}, **params}
+    return clauseelement, multiparams, params
+
+
+def bind(engine: Engine, **parameters) -> Engine:
+    """Create a marimo-discoverable engine with named SQL parameters for this cell.
+
+    Uses the original engine's pool. No query, upload, or server state is created.
+    Write :name placeholders in SQL cells; lists can be CAST(:ids AS VARCHAR[]).
+    """
+    from .dbapi import _parameter
+    return engine.execution_options(periplus_parameters={name: _parameter(value) for name, value in parameters.items()})

periplus_sdk/sqlalchemy.py

@@ -9,10 +9,11 @@ from sqlalchemy.engine.reflection import cache
 from sqlalchemy.sql.compiler import IdentifierPreparer
 
 from . import dbapi
+from .errors import ApiError, ResponseError, TransportError
 
 
 class SQLType(types.UserDefinedType):
-    """Preserve DuckDB type names, including nested types, during reflection."""
+    """Preserve ClickHouse type names, including nested types, during reflection."""
 
     cache_ok = True
 
@@ -24,28 +25,21 @@ class SQLType(types.UserDefinedType):
 
     @property
     def python_type(self) -> type:
-        if self.name in dbapi._INTEGER_TYPES:
-            return int
-        if self.name in dbapi._FLOAT_TYPES:
-            return float
-        if self.name == "BOOLEAN":
-            return bool
-        if self.name.startswith("DECIMAL("):
-            return dbapi.Decimal
-        if self.name == "DATE":
-            return dbapi.date
-        if self.name in dbapi._TIMESTAMP_TYPES:
-            return dbapi.datetime
-        if self.name in dbapi._TIME_TYPES:
-            return dbapi.time
-        if self.name == "BLOB":
-            return bytes
+        name = dbapi._base_type(self.name)
+        if name in dbapi._INTEGER_TYPES:return int
+        if name in dbapi._FLOAT_TYPES:return float
+        if name == 'Bool':return bool
+        if name.startswith('Decimal'):return dbapi.Decimal
+        if name in ('Date','Date32'):return dbapi.date
+        if name.startswith('DateTime'):return dbapi.datetime
+        if name.startswith('Time'):return dbapi.time
         return str
 
 
+
 class PeriplusDialect(default.DefaultDialect):
-    # The server speaks DuckDB SQL; this enables the correct notebook SQL dialect.
-    name = "duckdb"
+    # The server speaks ClickHouse SQL; this enables the correct notebook SQL dialect.
+    name = "clickhouse"
     driver = "periplus"
     supports_statement_cache = False
     supports_sane_rowcount = False
@@ -91,19 +85,26 @@ class PeriplusDialect(default.DefaultDialect):
 
     def _complete(self, result):
         raw = result.cursor.result
-        if raw.truncated:
-            result.close()
-            raise exc.InvalidRequestError("Catalogue discovery was truncated by public query limits; refusing an incomplete schema.")
         try:
-            return result.fetchall()
+            rows = result.fetchall()
+            if raw.truncated:
+                raise exc.InvalidRequestError("Catalogue discovery was truncated by public query limits; refusing an incomplete schema.")
+            return rows
         finally:
             result.close()
 
     @cache
     def get_view_names(self, connection, schema=None, **kw):
         schema = self._schema(connection, schema)
-        result = connection.exec_driver_sql(f"SHOW TABLES FROM {self.identifier_preparer.quote_identifier(schema)}")
-        return [row[0] for row in self._complete(result)]
+        try:
+            catalogue = connection.connection.dbapi_connection._client.helpers()
+        except (ApiError, ResponseError, TransportError) as error:
+            raise exc.InvalidRequestError(f"Public catalogue discovery failed: {error}") from error
+        if catalogue.schema_version != schema:
+            raise exc.InvalidRequestError("Public catalogue does not match the configured schema.")
+        prefix = f"{schema}."
+        return [relation.name.removeprefix(prefix) for relation in catalogue.relations
+                if relation.kind == "view" and relation.name.startswith(prefix)]
 
     @cache
     def get_table_names(self, connection, schema=None, **kw):
@@ -119,9 +120,15 @@ class PeriplusDialect(default.DefaultDialect):
     def get_columns(self, connection, table_name, schema=None, **kw):
         schema = self._schema(connection, schema)
         quote = self.identifier_preparer.quote_identifier
-        result = connection.exec_driver_sql(f"DESCRIBE {quote(schema)}.{quote(table_name)}")
-        return [{"name": row[0], "type": SQLType(row[1]), "nullable": row[2] != "NO",
-                 "default": row[4]} for row in self._complete(result)]
+        # A zero-row SELECT obtains native types through the supported public
+        # query contract, without scanning the view or requiring system access.
+        result = connection.exec_driver_sql(f"SELECT * FROM {quote(schema)}.{quote(table_name)} LIMIT 0")
+        metadata = result.cursor.result
+        self._complete(result)
+        return [{"name": name, "type": SQLType(kind),
+                 "nullable": kind.startswith("Nullable(") or kind.startswith("LowCardinality(Nullable("),
+                 "default": None}
+                for name, kind in zip(metadata.columns, metadata.types, strict=True)]
 
     @cache
     def get_pk_constraint(self, connection, table_name, schema=None, **kw):

periplus_sdk/stream.py

@@ -0,0 +1,134 @@
+"""Incremental query frames. EOF is never a successful completion marker."""
+from __future__ import annotations
+
+import json
+from typing import Literal
+
+import httpx
+from pydantic import BaseModel, Field, JsonValue, ValidationError
+
+from .errors import ApiError, ResponseError, TransportError
+from .types import PreparedQuery
+
+MEDIA_TYPE = "application/x-ndjson"
+
+
+class StreamResult(PreparedQuery):
+    columns: list[str]
+    types: list[str]
+    source_snapshot: int | None = Field(default=None, ge=0)
+    limits: dict[str, int]
+    complete: bool = False
+    truncated: bool = False
+    truncation_reason: Literal["max_rows", "max_result_bytes"] | None = None
+    row_count: int = 0
+    result_bytes: int = 0
+    elapsed_ms: float = 0
+
+
+class Rows(BaseModel):
+    type: Literal["rows"]
+    rows: list[list[JsonValue]]
+
+
+class Completion(BaseModel):
+    type: Literal["complete"]
+    row_count: int = Field(ge=0)
+    result_bytes: int = Field(ge=0)
+    truncated: bool
+    truncation_reason: Literal["max_rows", "max_result_bytes"] | None
+    elapsed_ms: float = Field(ge=0)
+
+
+class QueryStream:
+    """One response, consumed a batch at a time. Close early to cancel delivery."""
+
+    def __init__(self, response: httpx.Response, *, allow_partial: bool = False):
+        self.response = response
+        self.allow_partial = allow_partial
+        self.closed = False
+        self._count = 0
+        self._lines = response.iter_lines()
+        try:
+            if response.headers.get("content-type", "").split(";")[0] != MEDIA_TYPE:
+                raise ResponseError("Expected a streaming query response.")
+            frame = self._frame()
+            if frame.get("type") != "metadata":
+                raise ResponseError("Query stream is missing metadata.")
+            self.result = StreamResult.model_validate(frame)
+            if len(self.result.columns) != len(self.result.types):
+                raise ResponseError("Query columns and types have inconsistent widths.")
+        except ValidationError:
+            self.close()
+            raise ResponseError("Invalid query stream metadata.") from None
+        except BaseException:
+            self.close()
+            raise
+
+    def _frame(self):
+        try:
+            for line in self._lines:
+                if not line.strip():
+                    continue
+                frame = json.loads(line)
+                if not isinstance(frame, dict):
+                    raise ResponseError("Invalid query stream frame.")
+                if frame.get("type") == "error":
+                    raise ApiError(frame.get("detail", "Query stream failed."),
+                                   status_code=frame.get("status", 500), code=frame.get("code"))
+                return frame
+        except httpx.RequestError:
+            raise TransportError("Query stream interrupted; the result is incomplete.") from None
+        except ValueError:
+            raise ResponseError("Invalid query stream frame.") from None
+        raise ResponseError("Query stream ended without completion; the result is incomplete.")
+
+    def __iter__(self):
+        return self
+
+    def __next__(self) -> list[list[JsonValue]]:
+        if self.closed:
+            if not self.result.complete:
+                raise ResponseError("Query stream closed before completion; the result is incomplete.")
+            if self.result.truncated and not self.allow_partial:
+                raise ResponseError("Query result exceeded its budget and is incomplete.")
+            raise StopIteration
+        try:
+            frame = self._frame()
+            if frame.get("type") == "rows":
+                rows = Rows.model_validate(frame).rows
+                if any(len(row) != len(self.result.columns) for row in rows):
+                    raise ResponseError("Query row has an inconsistent width.")
+                self._count += len(rows)
+                return rows
+            completion = Completion.model_validate(frame)
+            if completion.row_count != self._count:
+                raise ResponseError("Query completion row count does not match delivered rows.")
+            for name, value in completion.model_dump(exclude={"type"}).items():
+                setattr(self.result, name, value)
+            self.result.complete = True
+            self.close()
+            if completion.truncated and not self.allow_partial:
+                budget = completion.truncation_reason
+                limit = self.result.limits.get(budget, "unknown")
+                raise ApiError(f"Query result is incomplete: {budget} ({limit}) reached after {self._count} rows. "
+                               "Narrow the query or ask the administrator to increase this budget.",
+                               status_code=422, code="result_limit")
+            raise StopIteration
+        except ValidationError:
+            self.close()
+            raise ResponseError("Invalid query stream frame.") from None
+        except BaseException:
+            self.close()
+            raise
+
+    def close(self):
+        if not self.closed:
+            self.closed = True
+            self.response.close()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *args):
+        self.close()

periplus_sdk/types.py

@@ -11,7 +11,7 @@ class Diagnostic(BaseModel):
 
 
 class PreparedQuery(BaseModel):
-    query_mode: Literal["stable", "experimental"]
+    query_mode: Literal["stable"]
     compiler_version: str
     optimizations: list[str]
     schema_version: str
@@ -28,7 +28,10 @@ class QueryResult(PreparedQuery):
     rows: list[list[JsonValue]]
     truncated: bool
     elapsed_ms: float
-    source_snapshot: int = Field(ge=0)
+    source_snapshot: int | None = Field(default=None, ge=0)
+    row_count: int = Field(ge=0)
+    result_bytes: int = Field(ge=0)
+    truncation_reason: Literal["max_rows", "max_result_bytes"] | None = None
 
 
 class HelperField(BaseModel):
@@ -48,4 +51,6 @@ class QueryHelper(BaseModel):
 
 class QueryHelpers(BaseModel):
     catalogue_version: str
+    schema_version: str
+    relations: list[QueryHelper]
     helpers: list[QueryHelper]

periplus_python_sdk-0.6.1.dist-info/METADATA

@@ -1,234 +0,0 @@
-Metadata-Version: 2.4
-Name: periplus-python-sdk
-Version: 0.6.1
-Summary: Read-only Python client for the public Periplus query API
-License-Expression: Apache-2.0
-Project-URL: Repository, https://github.com/elei-io/periplus
-Project-URL: Issues, https://github.com/elei-io/periplus/issues
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-License-File: NOTICE
-Requires-Dist: httpx>=0.28
-Requires-Dist: pydantic<3,>=2.12
-Requires-Dist: sqlalchemy<3,>=2.0
-Provides-Extra: notebook
-Requires-Dist: marimo[sql]>=0.24.1; extra == "notebook"
-Dynamic: license-file
-
-# Periplus Python SDK
-
-A read-only client for the public Periplus query API. Python 3.11 or later.
-Configure the **public web application URL**, not the internal query or control service.
-No API token, DuckDB installation or lake credentials are needed.
-
-```python
-from periplus_sdk import Client
-
-with Client("http://localhost:8080") as client:
-    result = client.execute(
-        "SELECT capture_id FROM public_v1.capture LIMIT ?", [10]
-    )
-    print(result.columns, result.types)
-    print(result.rows)
-    print(result.source_snapshot, result.truncated)
-```
-
-For a hosted deployment, replace the URL with its public HTTPS origin. Alternatively set
-`PERIPLUS_PUBLIC_URL` and use `Client()`. An optional URL path prefix is preserved.
-The client reuses HTTP connections; close it with a context manager or `close()`.
-
-## Marimo SQL cells and schema browser
-
-Install the notebook integration from PyPI:
-
-```sh
-uv add "periplus-python-sdk[notebook]>=0.6.1"
-```
-
-In a Python setup cell, create a SQLAlchemy engine:
-
-```python
-from periplus_sdk import sql_api
-
-pp = sql_api.create_engine("https://periplus.dev", mode="stable")
-```
-
-Add a SQL cell, select **pp** in its connection dropdown, and enter:
-
-```sql
-SELECT capture_id, requested_url
-FROM public_v1.capture
-LIMIT 10
-```
-
-Marimo displays the result as a table. Expand **pp → periplus → public_v1** in Data Sources
-to discover views and expand a view to load its columns for SQL completion.
-Discovery uses bounded `SHOW TABLES` and `DESCRIBE` through the same public API;
-no internal catalogue or storage credentials are used. Truncated discovery fails
-explicitly rather than displaying a silently incomplete schema. To eagerly load
-schemas and views, enable their discovery in marimo's Packages & Data settings.
-Column discovery is on demand by default, to avoid many public API requests.
-
-The Python equivalent of a SQL cell is:
-
-```python
-import marimo as mo
-
-captures = mo.sql(
-    "SELECT capture_id FROM public_v1.capture LIMIT 10",
-    engine=pp,
-)
-```
-
-Set `mode="experimental"` for the experimental service. Omit the URL to use
-`PERIPLUS_PUBLIC_URL`. Optional `timeout=140` and `schema_version="public_v1"`
-arguments configure the client deadline and public schema. Run `pp.dispose()` when finished. This is a read-only
-SQLAlchemy dialect for textual SQL and reflection, not a writable ORM backend.
-Each statement has its own server snapshot; SQLAlchemy transaction blocks do not
-provide a shared snapshot or rollback. The adapter makes no transaction requests.
-
-A complete notebook is in `examples/notebook.py`. The integration is tested with
-marimo 0.24.1 and SQLAlchemy 2.x. SQLAlchemy is included in the standard SDK install; the `notebook` extra adds
-marimo. Existing marimo environments only need `uv add "periplus-python-sdk>=0.6.1"`.
-The returned object is a standard SQLAlchemy Engine, also usable with pandas and
-ordinary Python scripts. Engine creation is lazy; the first query opens a connection.
-
-## DB-API connection
-
-For SQL cells without schema browsing, or standard cursor-based Python code:
-
-```python
-from periplus_sdk import connect
-
-with connect("https://periplus.dev", mode="stable") as connection:
-    with connection.cursor() as cursor:
-        cursor.execute("SELECT capture_id FROM public_v1.capture LIMIT ?", [10])
-        print(cursor.description)
-        print(cursor.fetchall())
-        print(cursor.result.source_snapshot)
-```
-
-Connections expose `cursor`, `execute`, `close`, and context managers. Cursors
-support `execute`, `fetchone`, `fetchmany`, `fetchall`, iteration, and close.
-Use positional `?` parameters. Decimal and temporal parameters are sent as
-strings; use explicit SQL casts. Binary and nested parameters are not supported
-by this adapter. Fetching only consumes the bounded result already received;
-it never issues pagination or retries. Connections/cursors are not thread-shared.
-`commit()` is a no-op; `rollback()` and `executemany()` are unsupported.
-
-`cursor.result` preserves the original query response. `connection.last_result`
-also retains it after marimo closes a cursor; a new execution clears it first.
-Truncation emits `periplus_sdk.dbapi.TruncationWarning` and sets `rowcount` to -1.
-DB-API failures use the standard exception hierarchy in `periplus_sdk.dbapi`;
-HTTP errors retain `status_code`, `code`, and `retry_after_seconds`.
-
-Scalar integer, floating-point, decimal, date, time, timestamp and BLOB results
-are decoded to Python values. UUIDs remain strings. Nested/other SQL types keep
-their JSON wire representation; out-of-range dates/timestamps remain strings.
-Temporal precision is limited to what the server JSON transport preserves.
-The cursor preserves duplicate column names, but dataframe libraries/marimo may
-not: use unique SQL aliases. Dataframe inference can lose types for empty or
-all-null results; `cursor.description` retains the SQL type names.
-
-## Stable and experimental APIs
-
-Both clients accept `mode="stable"` (the default) or `mode="experimental"` at initialization:
-
-```python
-with Client("https://periplus.dev", mode="experimental") as client:
-    result = client.execute("SELECT capture_id FROM public_v1.capture LIMIT 1")
-    print(result.query_mode, result.compiler_version, result.optimizations)
-```
-
-The selected mode applies to preparation, execution, and helper discovery. Experimental
-requests use the public application's `/api/query/experimental/` routes. There is no
-automatic fallback to stable if the experimental service is unavailable.
-`AsyncClient` accepts the same option. Invalid modes raise `ConfigurationError`.
-
-## Preparation and helpers
-
-```python
-with Client("http://localhost:8080") as client:
-    prepared = client.prepare("SELECT capture_id FROM public_v1.capture LIMIT ?", [10])
-    print(prepared.diagnostics, prepared.plan)
-    result = client.execute(prepared.sql, prepared.parameters)
-    helpers = client.helpers()
-    print(helpers.catalogue_version, helpers.helpers)
-```
-
-Preparation validates and explains without executing the analytical query. Execution independently
-validates and prepares; a prior preparation never authorizes SQL. Linting, diagnostics and future
-SQL optimizations belong to the server. The SDK sends SQL unchanged.
-
-## Async use
-
-```python
-from periplus_sdk import AsyncClient
-
-async def observations():
-    async with AsyncClient("http://localhost:8080") as client:
-        return await client.execute("SELECT capture_id FROM public_v1.capture LIMIT 10")
-```
-
-Use `aclose()` when managing an async client's lifetime explicitly.
-
-## Permissions, results and errors
-
-- The same public SQL feature switch, shared rate budget, namespace validation and read-only
-  execution apply as in the public web workspace. The SDK provides no writes, crawling,
-  administrative controls or direct lake attachment.
-- Results retain `query_id`, SQL, parameters, diagnostics, plan, columns, SQL types, JSON rows,
-  elapsed milliseconds, `source_snapshot` and `truncated`. Decimals and large integers remain
-  strings exactly as returned by the server. Duplicate column names are preserved.
-- Operator-configured execution limits default to 1,000 rows, an 8 MiB result budget and a
-  20-second server deadline. Always inspect `truncated`. The SDK does not silently fetch more rows or retry.
-- `ApiError` exposes `status_code`, safe `code`, and `retry_after_seconds` when supplied.
-  `TransportError` means HTTP failed; `ResponseError` means a malformed successful response.
-  The client timeout defaults to 140 seconds and can be set with `timeout=`. A timeout or local
-  cancellation does not guarantee server cancellation. Redirects are not followed automatically.
-- Preparation and execution are attributed to `sdk` in the existing private query history.
-  Original SQL and parameters are retained for 30 days; result rows are not stored. Recording is
-  best-effort and can be lost during outages or backpressure. This label is not a user identity.
-
-## Installation and verification
-
-Install the public-v1 client from PyPI:
-
-```sh
-python -m pip install "periplus-python-sdk>=0.6.1"
-```
-
-Version 0.6.1 supports the current public-v1 contract. For production, configure
-`PERIPLUS_PUBLIC_URL=https://periplus.dev`; no API token is required.
-Run the installed package against an available public app:
-
-```sh
-PERIPLUS_PUBLIC_URL=http://localhost:8080 python packages/periplus-python-sdk/examples/smoke.py
-```
-
-## Releasing
-
-Repository CI publishes immutable releases from tags named
-`periplus-python-sdk-v<version>`. The tag must exactly match the static version
-in `pyproject.toml`; for example, version `0.6.1` is released with:
-
-```sh
-git tag periplus-python-sdk-v0.6.1
-git push origin periplus-python-sdk-v0.6.1
-```
-
-PyPI publishing uses Trusted Publishing rather than a stored API token. The
-PyPI publisher must be configured for GitHub owner `elei-io`, repository
-`periplus`, workflow `python-sdk-release.yml`, and environment `pypi`. Protect
-that GitHub environment with required reviewers before the first release.
-
-## Public v1
-
-Install the updated SDK from PyPI with `python -m pip install "periplus-python-sdk>=0.6.1"`. The previously published 0.2.0 release predates this contract. `prepare` and `execute` accept keyword-only `schema_version="public_v1"` (the default); responses preserve `schema_version` separately from `source_snapshot`. Unavailable versions are rejected by the server.
-
-## License
-
-Copyright (c) 2026 Ekku Leivonen (elei.io). Licensed under [Apache-2.0](LICENSE);
-see [NOTICE](NOTICE). The server and other repository packages have separate
-licensing described in the root LICENSING.md.

periplus_python_sdk-0.6.1.dist-info/RECORD

@@ -1,15 +0,0 @@
-periplus_python_sdk-0.6.1.dist-info/licenses/LICENSE,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358
-periplus_python_sdk-0.6.1.dist-info/licenses/NOTICE,sha256=bhbYSqcUB3U_P1-XzloiT81JGniqoYaRLxNkQ1Pm9MQ,52
-periplus_sdk/__init__.py,sha256=WimXYlPB6tCimBO4VSwhcp00dwSL87jMmMuQ4-kINfM,546
-periplus_sdk/client.py,sha256=ZwMmVJKNF-FrGh_qwiQ5myhu8XJodb_96ohkqK47yDA,6557
-periplus_sdk/dbapi.py,sha256=lK9ZROaMKXm26wvC7DCKywm3qwSvqeHa7zkktnJVP80,11244
-periplus_sdk/errors.py,sha256=rB1n-v8Hc2tu2dtHivz-MlqsCoRC5pTcWogTfM7SMLw,855
-periplus_sdk/py.typed,sha256=AbpHGcgLb-kRsJGnwFEktk7uzpZOCcBY74-YBdrKVGs,1
-periplus_sdk/sql_api.py,sha256=ym50tZNVDKzUk9lBXlrT0K79E9b1Cn0btOqw4xshuM8,880
-periplus_sdk/sqlalchemy.py,sha256=AE68tzCrx-OZGOPkSsCZAlXaJn190yH2ez2ZuxI5VrY,4831
-periplus_sdk/types.py,sha256=PTdJO6BYTY97dBd3HMEZ52k6P9s0cMvidjpnc8zUpy4,1045
-periplus_python_sdk-0.6.1.dist-info/METADATA,sha256=K76t7I8jbUYVeWt02bAvRylw8yCJVy7TIoczCK-eQgY,10212
-periplus_python_sdk-0.6.1.dist-info/WHEEL,sha256=YVMoNqKzERt-wjUZwJ33xBGAwnFl-4cqbYkTtWa4itE,91
-periplus_python_sdk-0.6.1.dist-info/entry_points.txt,sha256=Pr14L_7AhLinq-4qDxB4awFVubrvR1BEfkaFVRpdCbU,73
-periplus_python_sdk-0.6.1.dist-info/top_level.txt,sha256=o41t5TzwgoxzSmbKoP6olWW1FyEAGWVYTjeoKadBK40,13
-periplus_python_sdk-0.6.1.dist-info/RECORD,,