interloper-sql 0.2.0rc1__tar.gz → 0.94.2__tar.gz

interloper_sql-0.94.2/PKG-INFO

@@ -0,0 +1,90 @@
+Metadata-Version: 2.3
+Name: interloper-sql
+Version: 0.94.2
+Summary: Interloper SQL integration: a SQLAlchemy-backed destination for PostgreSQL and other SQL databases
+Author: Guillaume Onfroy
+Author-email: Guillaume Onfroy <guillaume@digitlcloud.com>
+Requires-Dist: interloper-core
+Requires-Dist: psycopg[binary]>=3.1
+Requires-Dist: sqlalchemy>=2.0
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# interloper-sql
+
+A SQL database destination for interloper: `SQLDestination`, a
+`DatabaseDestination` over SQLAlchemy Core, and the `SQLConnection` that holds
+the database URL.
+
+One backend serves every SQLAlchemy dialect whose driver is installed.
+PostgreSQL works out of the box (the package depends on `psycopg[binary]`);
+MySQL, SQLite, Redshift, SQL Server and the rest need their own driver.
+
+## Setup
+
+The connection is a single SQLAlchemy URL:
+
+| Database | URL | Driver to install |
+|----------|-----|-------------------|
+| PostgreSQL | `postgresql+psycopg://user:password@host:5432/db` | none, ships with the package |
+| MySQL / MariaDB | `mysql+pymysql://user:password@host:3306/db` | `pymysql` |
+| SQL Server | `mssql+pyodbc://user:password@host/db?driver=ODBC+Driver+18+for+SQL+Server` | `pyodbc` (and the ODBC driver) |
+| Redshift | `redshift+redshift_connector://user:password@host:5439/db` | `sqlalchemy-redshift`, `redshift_connector` |
+| SQLite | `sqlite:///path/to/file.db` | none, in the standard library |
+
+Install the driver next to interloper (`uv pip install pymysql`, or a custom
+image built from the slim variant). Special characters in the password must be
+URL-encoded.
+
+The URL carries the password, so it is a secret field: encrypted at rest in a
+deployed instance and masked in the UI. It also loads from the environment
+(`SQL_URL`), so `SQLConnection()` works with no arguments.
+
+The connection's check runs `SELECT 1`.
+
+## Usage
+
+```python
+import interloper as il
+from interloper_sql import SQLConnection, SQLDestination
+
+warehouse = SQLDestination(
+    connection=SQLConnection(url="postgresql+psycopg://user:password@host:5432/analytics"),
+    default_dataset="marketing",
+)
+```
+
+In a deployed instance you configure this through the UI instead: add a SQL
+database connection, then a SQL database destination using it.
+
+## Datasets are schemas
+
+An asset's `dataset` is the SQL schema its table lives in. Without one, the
+destination's `default_dataset` applies; without either, the table goes in
+the connection's default schema (`public` on PostgreSQL, the database itself on
+MySQL). A named schema is created on first write when the database supports
+schemas and it does not exist yet.
+
+## Tables
+
+A table is created on first write from the asset's effective schema (declared,
+or inferred during conform): integers are `BIGINT`, floats `DOUBLE`, decimals
+`NUMERIC(38, 9)`, datetimes timezone-aware timestamps, strings `TEXT`, and
+nested or repeated fields `JSON`. An existing table is never altered: columns
+it lacks are dropped from the write with a warning, and NaN or infinite floats
+are written as `NULL`.
+
+## Partitions
+
+Writes replace, inside one transaction:
+
+- unpartitioned: every row is deleted, then the data inserted;
+- a time partition: the rows inside its half-open bounds
+  (`column >= start AND column < end`) are deleted, then the data inserted;
+- any other partition: the rows equal to its id are deleted;
+- a window: each partition's rows are deleted, then the whole batch is inserted
+  once.
+
+A failed insert rolls the delete back, so a partition is never left empty.
+Inserts go in batches of 1000 rows. Reads return rows as `list[dict]`; reading
+a table that was never written raises `DataNotFoundError`.

interloper_sql-0.94.2/README.md

@@ -0,0 +1,78 @@
+# interloper-sql
+
+A SQL database destination for interloper: `SQLDestination`, a
+`DatabaseDestination` over SQLAlchemy Core, and the `SQLConnection` that holds
+the database URL.
+
+One backend serves every SQLAlchemy dialect whose driver is installed.
+PostgreSQL works out of the box (the package depends on `psycopg[binary]`);
+MySQL, SQLite, Redshift, SQL Server and the rest need their own driver.
+
+## Setup
+
+The connection is a single SQLAlchemy URL:
+
+| Database | URL | Driver to install |
+|----------|-----|-------------------|
+| PostgreSQL | `postgresql+psycopg://user:password@host:5432/db` | none, ships with the package |
+| MySQL / MariaDB | `mysql+pymysql://user:password@host:3306/db` | `pymysql` |
+| SQL Server | `mssql+pyodbc://user:password@host/db?driver=ODBC+Driver+18+for+SQL+Server` | `pyodbc` (and the ODBC driver) |
+| Redshift | `redshift+redshift_connector://user:password@host:5439/db` | `sqlalchemy-redshift`, `redshift_connector` |
+| SQLite | `sqlite:///path/to/file.db` | none, in the standard library |
+
+Install the driver next to interloper (`uv pip install pymysql`, or a custom
+image built from the slim variant). Special characters in the password must be
+URL-encoded.
+
+The URL carries the password, so it is a secret field: encrypted at rest in a
+deployed instance and masked in the UI. It also loads from the environment
+(`SQL_URL`), so `SQLConnection()` works with no arguments.
+
+The connection's check runs `SELECT 1`.
+
+## Usage
+
+```python
+import interloper as il
+from interloper_sql import SQLConnection, SQLDestination
+
+warehouse = SQLDestination(
+    connection=SQLConnection(url="postgresql+psycopg://user:password@host:5432/analytics"),
+    default_dataset="marketing",
+)
+```
+
+In a deployed instance you configure this through the UI instead: add a SQL
+database connection, then a SQL database destination using it.
+
+## Datasets are schemas
+
+An asset's `dataset` is the SQL schema its table lives in. Without one, the
+destination's `default_dataset` applies; without either, the table goes in
+the connection's default schema (`public` on PostgreSQL, the database itself on
+MySQL). A named schema is created on first write when the database supports
+schemas and it does not exist yet.
+
+## Tables
+
+A table is created on first write from the asset's effective schema (declared,
+or inferred during conform): integers are `BIGINT`, floats `DOUBLE`, decimals
+`NUMERIC(38, 9)`, datetimes timezone-aware timestamps, strings `TEXT`, and
+nested or repeated fields `JSON`. An existing table is never altered: columns
+it lacks are dropped from the write with a warning, and NaN or infinite floats
+are written as `NULL`.
+
+## Partitions
+
+Writes replace, inside one transaction:
+
+- unpartitioned: every row is deleted, then the data inserted;
+- a time partition: the rows inside its half-open bounds
+  (`column >= start AND column < end`) are deleted, then the data inserted;
+- any other partition: the rows equal to its id are deleted;
+- a window: each partition's rows are deleted, then the whole batch is inserted
+  once.
+
+A failed insert rolls the delete back, so a partition is never left empty.
+Inserts go in batches of 1000 rows. Reads return rows as `list[dict]`; reading
+a table that was never written raises `DataNotFoundError`.

interloper_sql-0.94.2/pyproject.toml

@@ -0,0 +1,60 @@
+[project]
+name = "interloper-sql"
+version = "0.94.2"
+description = "Interloper SQL integration: a SQLAlchemy-backed destination for PostgreSQL and other SQL databases"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "interloper-core",
+    "psycopg[binary]>=3.1",
+    "sqlalchemy>=2.0",
+]
+
+[[project.authors]]
+name = "Guillaume Onfroy"
+email = "guillaume@digitlcloud.com"
+
+[project.entry-points."interloper.components"]
+sql = "interloper_sql"
+
+[build-system]
+requires = ["uv_build>=0.12.9,<0.13"]
+build-backend = "uv_build"
+
+[tool.uv.sources.interloper-core]
+workspace = true
+
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+preview = true
+extend-select = [
+    "E",
+    "I",
+    "UP",
+    "ANN001",
+    "ANN201",
+    "ANN202",
+    "DOC",
+    "D",
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = [
+    "F401",
+    "F403",
+]
+"tests/**" = [
+    "ANN",
+    "F811",
+    "D101",
+    "D102",
+    "D103",
+    "D104",
+    "RUF069",
+    "PLW0108",
+]

interloper_sql-0.2.0rc1/pyproject.toml → interloper_sql-0.94.2/pyproject.toml.orig

@@ -3,22 +3,22 @@
 # ###############
 [project]
 name = "interloper-sql"
-version = "0.2.0-rc.1"
-description = "Interloper SQLAlchemy IO managers"
+version = "0.94.2"
+description = "Interloper SQL integration: a SQLAlchemy-backed destination for PostgreSQL and other SQL databases"
 readme = "README.md"
 authors = [{ name = "Guillaume Onfroy", email = "guillaume@digitlcloud.com" }]
 requires-python = ">=3.10"
 dependencies = [
-    "sqlalchemy>=2.0",
     "interloper-core",
+    "psycopg[binary]>=3.1",
+    "sqlalchemy>=2.0",
 ]
 
-[project.optional-dependencies]
-postgres = ["psycopg2-binary>=2.9"]
-mysql = ["pymysql>=1.1"]
+[project.entry-points."interloper.components"]
+sql = "interloper_sql"
 
 [build-system]
-requires = ["uv_build>=0.9.5,<0.10.0"]
+requires = ["uv_build>=0.12.9,<0.13"]
 build-backend = "uv_build"
 
 [tool.uv.sources]
@@ -31,17 +31,12 @@ interloper-core = { workspace = true }
 line-length = 120
 
 [tool.ruff.lint]
-extend-select = ["E", "I", "UP", "ANN001", "ANN201", "ANN202"]
+preview = true
+extend-select = ["E", "I", "UP", "ANN001", "ANN201", "ANN202", "DOC", "D"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
 
 [tool.ruff.lint.per-file-ignores]
 "__init__.py" = ["F401", "F403"]
-"tests/**" = ["ANN", "F811"]
-
-# ###############
-# PYRIGHT
-# ###############
-[tool.pyright]
-include = ["src"]
-typeCheckingMode = "basic"
-reportMissingParameterType = true
-ignore = ["libs/**", "tests/**", "scripts/**"]
+"tests/**" = ["ANN", "F811", "D101", "D102", "D103", "D104", "RUF069", "PLW0108"]

interloper_sql-0.94.2/src/interloper_sql/__init__.py

@@ -0,0 +1,9 @@
+"""Interloper SQL integration: a SQLAlchemy-backed destination and its connection."""
+
+from interloper_sql.connection import SQLConnection
+from interloper_sql.destination import SQLDestination
+
+__all__ = [
+    "SQLConnection",
+    "SQLDestination",
+]

interloper_sql-0.94.2/src/interloper_sql/connection.py

@@ -0,0 +1,74 @@
+"""SQL connection resource holding a SQLAlchemy database URL."""
+
+from __future__ import annotations
+
+from functools import cached_property
+
+import sqlalchemy
+from interloper.connection import Connection, connection
+from interloper.resource.fields import SecretField, fetch_field_provider
+from pydantic_settings import SettingsConfigDict
+from sqlalchemy.engine import Engine
+
+
+@connection(
+    key="sql_connection",
+    name="SQL database",
+    icon="icon:sql",
+    tags=["Database"],
+)
+class SQLConnection(Connection):
+    """Connection resource holding the URL of a SQL database.
+
+    Any SQLAlchemy dialect whose driver is installed works; PostgreSQL through
+    psycopg ships with the package.
+    """
+
+    model_config = SettingsConfigDict(env_prefix="sql_")
+
+    url: str = SecretField(
+        label="Database URL",
+        description="SQLAlchemy URL, e.g. postgresql+psycopg://user:password@host:5432/db",
+        info=(
+            "The URL carries the password, so it is stored as a secret. PostgreSQL works out of the box "
+            "(postgresql+psycopg://); other databases need their SQLAlchemy driver installed, e.g. "
+            "mysql+pymysql:// or mssql+pyodbc://."
+        ),
+    )
+
+    @cached_property
+    def engine(self) -> Engine:
+        """The engine every statement goes through.
+
+        ``pool_pre_ping`` tests a pooled connection before handing it out, so
+        a connection the server dropped between runs is replaced instead of
+        failing the next write.
+
+        Returns:
+            The engine, cached per connection instance.
+        """
+        return sqlalchemy.create_engine(self.url, pool_pre_ping=True)
+
+    @fetch_field_provider
+    def schemas(self) -> list[dict[str, str]]:
+        """List the schemas of the database.
+
+        Not wired to a field yet: the destination's ``default_dataset`` is
+        optional, and a ``FetchField`` is a required pick. It is kept for a
+        future schema picker.
+
+        Returns:
+            Schema options with ``name``, sorted case-insensitively.
+        """
+        names = sqlalchemy.inspect(self.engine).get_schema_names()
+        return sorted(({"name": name} for name in names), key=lambda s: s["name"].lower())
+
+    def check(self) -> bool:
+        """Prove the URL works by running ``SELECT 1``.
+
+        Returns:
+            True; an unreachable database or a rejected login raises instead.
+        """
+        with self.engine.connect() as conn:
+            conn.execute(sqlalchemy.text("SELECT 1"))
+        return True

interloper_sql-0.94.2/src/interloper_sql/destination.py

@@ -0,0 +1,351 @@
+"""SQL database destination implementation over SQLAlchemy Core."""
+
+from __future__ import annotations
+
+import datetime
+import math
+import threading
+import warnings
+from collections.abc import Iterator, Sequence
+from contextlib import contextmanager
+from typing import Any
+
+from interloper.destination import IOContext, destination
+from interloper.destination.database import DatabaseDestination, PartitionFilter
+from interloper.errors import DataNotFoundError
+from interloper.representation import Representation
+from interloper.resource.fields import InputField
+from interloper.schema import FieldSpec
+from pydantic import PrivateAttr
+from sqlalchemy import Column, MetaData, String, Table, and_, cast, func, inspect, select
+from sqlalchemy.engine import Connection as SAConnection
+from sqlalchemy.schema import CreateSchema
+from sqlalchemy.sql.elements import ColumnElement
+
+from interloper_sql.connection import SQLConnection
+from interloper_sql.types import column_type
+
+_BATCH_SIZE = 1000
+
+
+@destination(
+    key="sql_destination",
+    name="SQL database",
+    icon="icon:sql",
+    tags=["Database"],
+)
+class SQLDestination(DatabaseDestination):
+    """SQL database destination.
+
+    Each asset is a table; its dataset is the SQL schema holding it. A table
+    is created from the effective schema on first write and never altered
+    afterwards.
+    """
+
+    connection: SQLConnection
+
+    default_dataset: str | None = InputField(default=None, description="Default schema for assets without a dataset")
+
+    # Keyed by thread because one instance serves every asset of a run, and
+    # each write runs whole on its own worker thread: a single slot would let
+    # concurrent writes swap each other's transaction. A plain dict, unlike
+    # threading.local, still deep-copies and pickles.
+    _held: dict[int, SAConnection] = PrivateAttr(default_factory=dict)
+
+    # -- Helpers ---------------------------------------------------------------
+
+    def _resolve_dataset(self, dataset: str | None) -> str | None:
+        """Return the schema to use.
+
+        Args:
+            dataset: The asset's dataset, or ``None`` to fall back to ``default_dataset``.
+
+        Returns:
+            The schema name, or ``None`` for the connection's default schema,
+            which every SQL database has.
+        """
+        return dataset or self.default_dataset
+
+    @contextmanager
+    def _begin(self) -> Iterator[SAConnection]:
+        """Yield the connection held by :meth:`transaction`, or a fresh one in its own transaction.
+
+        Yields:
+            A connection inside a transaction.
+        """
+        held = self._held.get(threading.get_ident())
+        if held is not None:
+            yield held
+            return
+        with self.connection.engine.begin() as conn:
+            yield conn
+
+    @staticmethod
+    def _table(table: str, dataset: str | None, conn: SAConnection) -> Table | None:
+        """Reflect an existing table.
+
+        Args:
+            table: Table name.
+            dataset: The resolved schema, or ``None`` for the default schema.
+            conn: The connection to reflect through.
+
+        Returns:
+            The reflected table, or ``None`` if it does not exist.
+        """
+        if not inspect(conn).has_table(table, schema=dataset):
+            return None
+        return Table(table, MetaData(), schema=dataset, autoload_with=conn)
+
+    @staticmethod
+    def _needs_schema(dataset: str | None, conn: SAConnection) -> bool:
+        """Whether the schema must be created before a table can go in it.
+
+        Args:
+            dataset: The resolved schema, or ``None`` for the default schema.
+            conn: The connection to inspect through.
+
+        Returns:
+            True when a schema is named, the dialect has schemas, and it does not exist yet.
+        """
+        if dataset is None or not getattr(conn.dialect, "supports_schemas", False):
+            return False
+        return not inspect(conn).has_schema(dataset)
+
+    def _create_table(self, table: str, dataset: str | None, specs: Sequence[FieldSpec], conn: SAConnection) -> Table:
+        """Create a table from field specs, creating its schema first when missing.
+
+        Args:
+            table: Table name.
+            dataset: The resolved schema, or ``None`` for the default schema.
+            specs: The field specs the columns are built from.
+            conn: The connection to create through.
+
+        Returns:
+            The created table.
+        """
+        if dataset is not None and self._needs_schema(dataset, conn):
+            conn.execute(CreateSchema(dataset, if_not_exists=True))
+        columns = [Column(spec.name, column_type(spec), nullable=True) for spec in specs]
+        sql_table = Table(table, MetaData(), *columns, schema=dataset)
+        sql_table.create(conn, checkfirst=True)
+        return sql_table
+
+    @staticmethod
+    def _ref(table: str, dataset: str | None) -> str:
+        """Name a table for messages.
+
+        Args:
+            table: Table name.
+            dataset: The resolved schema, or ``None`` for the default schema.
+
+        Returns:
+            ``schema.table``, or ``table`` alone in the default schema.
+        """
+        return f"{dataset}.{table}" if dataset else table
+
+    def _not_found(self, table: str, dataset: str | None) -> str:
+        """Word the error a read raises on a table that does not exist.
+
+        Args:
+            table: Table name.
+            dataset: The resolved schema, or ``None`` for the default schema.
+
+        Returns:
+            The error message.
+        """
+        return f"Table '{self._ref(table, dataset)}' does not exist. Has the asset been materialized?"
+
+    # -- DatabaseDestination hooks ---------------------------------------------
+
+    @contextmanager
+    def transaction(self) -> Iterator[None]:
+        """Run a delete and its insert in one database transaction.
+
+        Yields:
+            ``None``; the hooks called inside the block share its connection.
+        """
+        with self.connection.engine.begin() as conn:
+            self._held[threading.get_ident()] = conn
+            try:
+                yield
+            finally:
+                del self._held[threading.get_ident()]
+
+    def insert(self, table: str, dataset: str | None, data: Any, context: IOContext) -> None:
+        """Insert rows, creating the table from the effective schema on first write.
+
+        Without a schema on the context, one is inferred from the data so the
+        table is still typed. Columns the table does not have are dropped with
+        a warning, and non-finite floats are written as ``NULL``.
+
+        Args:
+            table: Target table name.
+            dataset: The asset's schema, or ``None`` for ``default_dataset``.
+            data: The data in its native representation.
+            context: IO context carrying the effective schema.
+        """
+        dataset = self._resolve_dataset(dataset)
+        view = Representation.of(data)
+        with self._begin() as conn:
+            sql_table = self._table(table, dataset, conn)
+            if sql_table is None:
+                schema = context.schema if context.schema is not None else view.infer()
+                sql_table = self._create_table(table, dataset, schema.field_specs(), conn)
+            rows = _align(view.records, [c.name for c in sql_table.columns], self._ref(table, dataset))
+            for start in range(0, len(rows), _BATCH_SIZE):
+                conn.execute(sql_table.insert(), rows[start : start + _BATCH_SIZE])
+
+    def delete(self, table: str, dataset: str | None, where: PartitionFilter | None) -> None:
+        """Delete the rows a filter selects, or every row.
+
+        A table that does not exist has nothing to delete.
+
+        Args:
+            table: Target table name.
+            dataset: The asset's schema, or ``None`` for ``default_dataset``.
+            where: The rows to delete; ``None`` for the whole table.
+        """
+        dataset = self._resolve_dataset(dataset)
+        with self._begin() as conn:
+            sql_table = self._table(table, dataset, conn)
+            if sql_table is None:
+                return
+            statement = sql_table.delete()
+            if where is not None:
+                statement = statement.where(_predicate(sql_table, where))
+            conn.execute(statement)
+
+    def select(self, table: str, dataset: str | None, where: PartitionFilter | None) -> list[dict[str, Any]]:
+        """Select the rows a filter selects, or every row, as records.
+
+        Args:
+            table: Target table name.
+            dataset: The asset's schema, or ``None`` for ``default_dataset``.
+            where: The rows to select; ``None`` for the whole table.
+
+        Returns:
+            The selected rows.
+
+        Raises:
+            DataNotFoundError: If the table does not exist yet.
+        """
+        dataset = self._resolve_dataset(dataset)
+        with self._begin() as conn:
+            sql_table = self._table(table, dataset, conn)
+            if sql_table is None:
+                raise DataNotFoundError(self._not_found(table, dataset))
+            statement = select(sql_table)
+            if where is not None:
+                statement = statement.where(_predicate(sql_table, where))
+            return [dict(row._mapping) for row in conn.execute(statement)]
+
+    def count(self, table: str, dataset: str | None, column: str) -> dict[str, int]:
+        """Return row counts grouped by a column's values, cast to strings.
+
+        Args:
+            table: Target table name.
+            dataset: The asset's schema, or ``None`` for ``default_dataset``.
+            column: Column to group by.
+
+        Returns:
+            Mapping from partition value (as string) to row count.
+
+        Raises:
+            DataNotFoundError: If the table does not exist.
+        """
+        dataset = self._resolve_dataset(dataset)
+        with self._begin() as conn:
+            sql_table = self._table(table, dataset, conn)
+            if sql_table is None:
+                raise DataNotFoundError(self._not_found(table, dataset))
+            value = cast(sql_table.c[column], String).label("partition_value")
+            statement = select(value, func.count().label("cnt")).group_by(value)
+            return {row.partition_value: row.cnt for row in conn.execute(statement)}
+
+
+# -- Utility functions ---------------------------------------------------------
+
+
+def _align(records: list[dict[str, Any]], columns: list[str], ref: str) -> list[dict[str, Any]]:
+    """Shape records to the table's columns for one executemany.
+
+    Every row gets the same keys, the table columns any record carries, so
+    a column no record mentions keeps its server default.
+
+    Args:
+        records: The rows to write.
+        columns: The table's column names, in table order.
+        ref: The table's name, for the warning.
+
+    Returns:
+        The aligned rows, non-finite floats replaced by ``None``.
+    """
+    present = dict.fromkeys(key for record in records for key in record)
+    extras = [str(key) for key in present if key not in columns]
+    if extras:
+        warnings.warn(
+            f"Columns {extras} are not in the schema for '{ref}' and will not be written.",
+            UserWarning,
+            stacklevel=3,
+        )
+    keys = [name for name in columns if name in present]
+    return [{key: _finite(record.get(key)) for key in keys} for record in records]
+
+
+def _finite(value: Any) -> Any:
+    """Replace a NaN or infinite float with ``None``, which no SQL column type rejects.
+
+    Args:
+        value: A cell value.
+
+    Returns:
+        ``None`` for a non-finite float, else the value unchanged.
+    """
+    if isinstance(value, float) and not math.isfinite(value):
+        return None
+    return value
+
+
+def _predicate(table: Table, where: PartitionFilter) -> ColumnElement[bool]:
+    """Render a partition filter against a reflected table.
+
+    Args:
+        table: The reflected table.
+        where: The filter: equality on ``value``, or half-open ``bounds``.
+
+    Returns:
+        The ``WHERE`` clause.
+    """
+    column = table.c[where.column]
+    if where.bounds is not None:
+        start, end = where.bounds
+        return and_(column >= _coerce(column, start), column < _coerce(column, end))
+    return column == _coerce(column, where.value)
+
+
+def _coerce(column: Column[Any], value: Any) -> Any:
+    """Coerce a filter value to the Python type a date or timestamp column binds.
+
+    A partition id arrives as a string (``"2024-01-01"``) and a day's bounds
+    as dates; bound as-is against a ``DATE`` or ``TIMESTAMP`` column, they
+    would be sent as ``VARCHAR`` or rejected by the dialect's type.
+
+    Args:
+        column: The column the value is compared with.
+        value: The filter value.
+
+    Returns:
+        The value as a ``date`` or ``datetime`` when the column holds one, else unchanged.
+    """
+    try:
+        python_type = column.type.python_type
+    except NotImplementedError:
+        return value
+    if python_type is datetime.datetime:
+        if isinstance(value, str):
+            return datetime.datetime.fromisoformat(value)
+        if isinstance(value, datetime.date) and not isinstance(value, datetime.datetime):
+            return datetime.datetime.combine(value, datetime.time())
+    elif python_type is datetime.date and isinstance(value, str):
+        return datetime.date.fromisoformat(value)
+    return value

interloper_sql-0.94.2/src/interloper_sql/types.py

@@ -0,0 +1,43 @@
+"""SQLAlchemy's view of interloper's field types: one table, read in order."""
+
+from __future__ import annotations
+
+import datetime
+from decimal import Decimal
+
+from interloper.schema import FieldSpec
+from sqlalchemy import JSON, BigInteger, Boolean, Date, DateTime, Double, LargeBinary, Numeric, Text
+from sqlalchemy.types import TypeEngine
+
+# Ordered: the first base class that matches wins, so bool (a subclass of int)
+# and datetime (a subclass of date) must come before their parents.
+_PYTHON_TO_SQL: dict[type, TypeEngine] = {
+    bool: Boolean(),
+    int: BigInteger(),
+    float: Double(),
+    Decimal: Numeric(38, 9),
+    datetime.datetime: DateTime(timezone=True),
+    datetime.date: Date(),
+    bytes: LargeBinary(),
+    str: Text(),
+}
+
+
+def column_type(spec: FieldSpec) -> TypeEngine:
+    """Return the SQLAlchemy column type for a field spec.
+
+    Args:
+        spec: The field spec to map, from :meth:`Schema.field_specs`. A nested
+            model or a repeated field is ``JSON``; a type that is not a class
+            (``typing.Any``) or matches no row of the table is ``Text``.
+
+    Returns:
+        The column type, rendered by each dialect in its own vocabulary.
+    """
+    if spec.fields is not None or spec.repeated:
+        return JSON()
+    if isinstance(spec.type, type):
+        for base, sql_type in _PYTHON_TO_SQL.items():
+            if issubclass(spec.type, base):
+                return sql_type
+    return Text()

interloper_sql-0.2.0rc1/PKG-INFO

@@ -1,18 +0,0 @@
-Metadata-Version: 2.3
-Name: interloper-sql
-Version: 0.2.0rc1
-Summary: Interloper SQLAlchemy IO managers
-Author: Guillaume Onfroy
-Author-email: Guillaume Onfroy <guillaume@digitlcloud.com>
-Requires-Dist: sqlalchemy>=2.0
-Requires-Dist: interloper-core
-Requires-Dist: pymysql>=1.1 ; extra == 'mysql'
-Requires-Dist: psycopg2-binary>=2.9 ; extra == 'postgres'
-Requires-Python: >=3.10
-Provides-Extra: mysql
-Provides-Extra: postgres
-Description-Content-Type: text/markdown
-
-# interloper-sql
-
-SQLAlchemy IO managers for the Interloper framework.

interloper_sql-0.2.0rc1/README.md

@@ -1,3 +0,0 @@
-# interloper-sql
-
-SQLAlchemy IO managers for the Interloper framework.

interloper_sql-0.2.0rc1/src/interloper_sql/__init__.py

@@ -1,10 +0,0 @@
-"""Interloper SQL integration for relational database IO via SQLAlchemy."""
-
-from interloper_sql.io import MySQLIO, PostgresIO, SqlIO, SqliteIO
-
-__all__ = [
-    "MySQLIO",
-    "PostgresIO",
-    "SqlIO",
-    "SqliteIO",
-]

interloper_sql-0.2.0rc1/src/interloper_sql/io/__init__.py

@@ -1,13 +0,0 @@
-"""SQL IO managers for reading and writing to databases via SQLAlchemy."""
-
-from interloper_sql.io.base import SqlIO
-from interloper_sql.io.mysql import MySQLIO
-from interloper_sql.io.postgres import PostgresIO
-from interloper_sql.io.sqlite import SqliteIO
-
-__all__ = [
-    "MySQLIO",
-    "PostgresIO",
-    "SqlIO",
-    "SqliteIO",
-]

interloper_sql-0.2.0rc1/src/interloper_sql/io/base.py

@@ -1,281 +0,0 @@
-"""SQLAlchemy IO implementation."""
-
-from __future__ import annotations
-
-from collections.abc import Iterator
-from contextlib import contextmanager
-from typing import TYPE_CHECKING, Any
-
-from interloper.errors import TableNotFoundError
-from interloper.io.database import DatabaseIO, WriteDisposition
-from sqlalchemy import Column, MetaData, Table, create_engine
-from sqlalchemy import inspect as sa_inspect
-
-if TYPE_CHECKING:
-    from interloper.io.adapter import DataAdapter
-    from sqlalchemy.engine import URL, Connection, Engine
-
-
-def _infer_sa_type(value: Any) -> Any:
-    """Infer a SQLAlchemy column type from a Python value.
-
-    Args:
-        value: A sample Python value used to determine the column type.
-
-    Returns:
-        A SQLAlchemy type instance.
-    """
-    import datetime
-    from decimal import Decimal
-
-    from sqlalchemy import BigInteger, Boolean, Date, DateTime, Float, LargeBinary, Numeric, Text
-
-    if isinstance(value, bool):
-        return Boolean()
-    if isinstance(value, int):
-        return BigInteger()
-    if isinstance(value, float):
-        return Float()
-    if isinstance(value, Decimal):
-        return Numeric()
-    if isinstance(value, datetime.datetime):
-        return DateTime()
-    if isinstance(value, datetime.date):
-        return Date()
-    if isinstance(value, bytes):
-        return LargeBinary()
-    return Text()
-
-
-class SqlIO(DatabaseIO):
-    """Base IO implementation for SQL databases via SQLAlchemy.
-
-    Provides connection management, transactional writes, table reflection, and
-    automatic table creation.  Not intended for direct instantiation — use a
-    dialect subclass (:class:`PostgresIO`, :class:`MySQLIO`, :class:`SqliteIO`)
-    which accepts explicit connection parameters and implements ``to_spec``.
-
-    The IO is fully stateless with respect to table identity — the table name
-    and schema are passed through from the asset context on every call, so a
-    single instance can safely serve multiple assets.
-
-    Args:
-        url: SQLAlchemy connection URL or :class:`~sqlalchemy.engine.URL` object
-            (constructed by dialect subclasses).
-        write_disposition: Controls whether existing rows are deleted before
-            writing.  Defaults to :attr:`WriteDisposition.REPLACE`.
-        chunk_size: Number of rows per insert batch
-        adapter: Optional data adapter for type conversion
-    """
-
-    def __init__(
-        self,
-        url: str | URL,
-        write_disposition: WriteDisposition = WriteDisposition.REPLACE,
-        chunk_size: int = 1000,
-        adapter: DataAdapter | str | None = None,
-    ) -> None:
-        super().__init__(write_disposition, chunk_size, adapter)
-        self._engine: Engine = create_engine(url)
-        self._table_cache: dict[tuple[str, str | None], Table] = {}
-        self._conn: Connection | None = None
-
-    # ------------------------------------------------------------------
-    # Table helpers
-    # ------------------------------------------------------------------
-
-    def _resolve_table(self, table: str, schema: str | None) -> Table | None:
-        """Reflect and cache the SQLAlchemy Table, or return None if it doesn't exist."""
-        key = (table, schema)
-        if key not in self._table_cache:
-            if sa_inspect(self._engine).has_table(table, schema=schema):
-                metadata = MetaData(schema=schema)
-                self._table_cache[key] = Table(table, metadata, autoload_with=self._engine)
-        return self._table_cache.get(key)
-
-    def _require_table(self, table: str, schema: str | None) -> Table:
-        """Resolve the table, raising if it does not exist."""
-        sa_table = self._resolve_table(table, schema)
-        if sa_table is None:
-            qualified = f"{schema}.{table}" if schema else table
-            raise TableNotFoundError(f"Table '{qualified}' does not exist. Has the asset been materialized?")
-        return sa_table
-
-    def _create_table(self, table: str, schema: str | None, rows: list[dict[str, Any]]) -> Table:
-        """Create a new table from the structure of the first row.
-
-        Column types are inferred from the Python values in the sample row
-        using :func:`_infer_sa_type`.
-
-        Args:
-            table: Target table name
-            schema: Database schema
-            rows: Row data (at least one row required for schema inference).
-
-        Returns:
-            The newly created :class:`~sqlalchemy.schema.Table`.
-        """
-        assert self._conn is not None
-        sample = rows[0]
-        columns = [Column(name, _infer_sa_type(value)) for name, value in sample.items()]
-        sa_metadata = MetaData(schema=schema)
-        sa_table = Table(table, sa_metadata, *columns)
-        sa_metadata.create_all(self._conn)
-        self._table_cache[(table, schema)] = sa_table
-        return sa_table
-
-    # ------------------------------------------------------------------
-    # Transaction management
-    # ------------------------------------------------------------------
-
-    @contextmanager
-    def _transaction(self) -> Iterator[None]:
-        """Open a SQLAlchemy transactional connection for write operations.
-
-        Sets ``self._conn`` for the duration of the block.  The connection is
-        committed on success and rolled back on exception (``engine.begin()``
-        semantics).
-
-        Yields:
-            None
-        """
-        with self._engine.begin() as conn:
-            self._conn = conn
-            try:
-                yield
-            finally:
-                self._conn = None
-
-    # ------------------------------------------------------------------
-    # DatabaseIO hooks
-    # ------------------------------------------------------------------
-
-    def _insert(self, table: str, schema: str | None, rows: list[dict[str, Any]]) -> None:
-        """Insert rows in chunks using the active transaction connection.
-
-        If the table does not exist yet, it is created from the row data
-        before inserting.
-
-        Args:
-            table: Target table name
-            schema: Database schema
-            rows: Row data as list of dicts
-        """
-        assert self._conn is not None
-        sa_table = self._resolve_table(table, schema)
-        if sa_table is None:
-            sa_table = self._create_table(table, schema, rows)
-        for i in range(0, len(rows), self.chunk_size):
-            self._conn.execute(sa_table.insert(), rows[i : i + self.chunk_size])
-
-    def _delete_all(self, table: str, schema: str | None) -> None:
-        """Delete all rows from the table using the active transaction connection.
-
-        No-op when the table does not exist yet.
-
-        Args:
-            table: Target table name
-            schema: Database schema
-        """
-        assert self._conn is not None
-        sa_table = self._resolve_table(table, schema)
-        if sa_table is None:
-            return
-        self._conn.execute(sa_table.delete())
-
-    def _delete_partition(self, table: str, schema: str | None, column: str, value: Any) -> None:
-        """Delete rows matching a partition value using the active transaction connection.
-
-        No-op when the table does not exist yet.
-
-        Args:
-            table: Target table name
-            schema: Database schema
-            column: Partition column name
-            value: Partition value to match
-        """
-        assert self._conn is not None
-        sa_table = self._resolve_table(table, schema)
-        if sa_table is None:
-            return
-        self._conn.execute(sa_table.delete().where(sa_table.c[column] == value))
-
-    def _select_all(self, table: str, schema: str | None) -> list[dict[str, Any]]:
-        """Select all rows from the table.
-
-        Opens a dedicated read connection (not part of the write transaction).
-
-        Args:
-            table: Target table name
-            schema: Database schema
-
-        Returns:
-            All rows as list of dicts
-
-        Raises:
-            ValueError: If the table does not exist
-        """
-        sa_table = self._require_table(table, schema)
-        with self._engine.connect() as conn:
-            result = conn.execute(sa_table.select())
-            return [dict(row._mapping) for row in result]
-
-    def _select_partition(self, table: str, schema: str | None, column: str, value: Any) -> list[dict[str, Any]]:
-        """Select rows matching a partition value.
-
-        Opens a dedicated read connection (not part of the write transaction).
-
-        Args:
-            table: Target table name
-            schema: Database schema
-            column: Partition column name
-            value: Partition value to match
-
-        Returns:
-            Matching rows as list of dicts
-
-        Raises:
-            ValueError: If the table does not exist
-        """
-        sa_table = self._require_table(table, schema)
-        with self._engine.connect() as conn:
-            result = conn.execute(sa_table.select().where(sa_table.c[column] == value))
-            return [dict(row._mapping) for row in result]
-
-    # ------------------------------------------------------------------
-    # Introspection
-    # ------------------------------------------------------------------
-
-    def _count_by_partition(
-        self, table: str, schema: str | None, column: str,
-    ) -> dict[str, int]:
-        """Return row counts grouped by partition column via SQL ``GROUP BY``.
-
-        Args:
-            table: Target table name
-            schema: Database schema
-            column: Column to group by
-
-        Returns:
-            Mapping from partition value (as string) to row count.
-
-        Raises:
-            TableNotFoundError: If the table does not exist.
-        """
-        from sqlalchemy import func
-
-        sa_table = self._require_table(table, schema)
-        col = sa_table.c[column]
-        stmt = sa_table.select().with_only_columns(col, func.count()).group_by(col)
-        with self._engine.connect() as conn:
-            result = conn.execute(stmt)
-            return {str(row[0]): row[1] for row in result}
-
-    # ------------------------------------------------------------------
-    # Lifecycle
-    # ------------------------------------------------------------------
-
-    def dispose(self) -> None:
-        """Dispose the SQLAlchemy engine and clear the table cache."""
-        self._engine.dispose()
-        self._table_cache.clear()

interloper_sql-0.2.0rc1/src/interloper_sql/io/mysql.py

@@ -1,70 +0,0 @@
-"""MySQL IO implementation."""
-
-from __future__ import annotations
-
-from typing import TYPE_CHECKING
-
-from interloper.io.database import WriteDisposition
-from interloper.serialization.io import IOSpec
-from sqlalchemy.engine import URL
-
-from interloper_sql.io.base import SqlIO
-
-if TYPE_CHECKING:
-    from interloper.io.adapter import DataAdapter
-
-
-class MySQLIO(SqlIO):
-    """MySQL-specific IO manager.
-
-    Extends :class:`SqlIO` for MySQL connections. Uses standard ``DELETE`` for
-    row removal because MySQL's ``TRUNCATE`` causes an implicit commit and
-    cannot participate in a transaction.
-
-    Args:
-        host: Database server hostname
-        database: Database name
-        port: Database server port
-        user: Database user
-        password: Database password
-        driver: SQLAlchemy driver (e.g. ``pymysql``, ``mysqlconnector``)
-        write_disposition: Controls whether existing rows are deleted before writing
-        chunk_size: Number of rows per insert batch
-        adapter: Optional data adapter for type conversion
-    """
-
-    def __init__(
-        self,
-        host: str,
-        database: str,
-        port: int = 3306,
-        user: str = "root",
-        password: str | None = None,
-        driver: str | None = None,
-        write_disposition: WriteDisposition = WriteDisposition.REPLACE,
-        chunk_size: int = 1000,
-        adapter: DataAdapter | str | None = None,
-    ) -> None:
-        self.host = host
-        self.port = port
-        self.database = database
-        self.user = user
-        self.password = password
-        self.driver = driver
-
-        drivername = f"mysql+{driver}" if driver else "mysql"
-        url = URL.create(drivername, user, password, host, port, database)
-        super().__init__(url, write_disposition, chunk_size, adapter)
-
-    def to_spec(self) -> IOSpec:
-        """Convert to serializable spec."""
-        init = self._base_init_kwargs()
-        init["host"] = self.host
-        init["port"] = self.port
-        init["database"] = self.database
-        init["user"] = self.user
-        if self.password is not None:
-            init["password"] = self.password
-        if self.driver is not None:
-            init["driver"] = self.driver
-        return IOSpec(path=self.path, init=init)

interloper_sql-0.2.0rc1/src/interloper_sql/io/postgres.py

@@ -1,87 +0,0 @@
-"""PostgreSQL IO implementation."""
-
-from __future__ import annotations
-
-from typing import TYPE_CHECKING
-
-from interloper.io.database import WriteDisposition
-from interloper.serialization.io import IOSpec
-from sqlalchemy import text
-from sqlalchemy.engine import URL
-
-from interloper_sql.io.base import SqlIO
-
-if TYPE_CHECKING:
-    from interloper.io.adapter import DataAdapter
-
-
-class PostgresIO(SqlIO):
-    """PostgreSQL-specific IO manager.
-
-    Extends :class:`SqlIO` with PostgreSQL optimisations:
-
-    * Uses ``TRUNCATE`` (transactional in Postgres) instead of ``DELETE`` for
-      full-table replacements, which is significantly faster on large tables.
-
-    Args:
-        host: Database server hostname
-        port: Database server port
-        database: Database name
-        user: Database user
-        password: Database password
-        driver: SQLAlchemy driver (e.g. ``psycopg2``, ``asyncpg``)
-        write_disposition: Controls whether existing rows are deleted before writing
-        chunk_size: Number of rows per insert batch
-        adapter: Optional data adapter for type conversion
-    """
-
-    def __init__(
-        self,
-        host: str,
-        port: int = 5432,
-        database: str = "postgres",
-        user: str = "postgres",
-        password: str | None = None,
-        driver: str | None = None,
-        write_disposition: WriteDisposition = WriteDisposition.REPLACE,
-        chunk_size: int = 1000,
-        adapter: DataAdapter | str | None = None,
-    ) -> None:
-        self.host = host
-        self.port = port
-        self.database = database
-        self.user = user
-        self.password = password
-        self.driver = driver
-
-        drivername = f"postgresql+{driver}" if driver else "postgresql"
-        url = URL.create(drivername, user, password, host, port, database)
-        super().__init__(url, write_disposition, chunk_size, adapter)
-
-    def _delete_all(self, table: str, schema: str | None) -> None:
-        """Use TRUNCATE for full-table deletes (transactional in PostgreSQL).
-
-        No-op when the table does not exist yet.
-
-        Args:
-            table: Target table name
-            schema: Database schema
-        """
-        assert self._conn is not None
-        sa_table = self._resolve_table(table, schema)
-        if sa_table is None:
-            return
-        self._conn.execute(text(f"TRUNCATE TABLE {sa_table.fullname}"))
-
-    def to_spec(self) -> IOSpec:
-        """Convert to serializable spec."""
-        init = self._base_init_kwargs()
-        init["host"] = self.host
-        init["port"] = self.port
-        init["database"] = self.database
-        init["user"] = self.user
-        if self.password is not None:
-            init["password"] = self.password
-        if self.driver is not None:
-            init["driver"] = self.driver
-        return IOSpec(path=self.path, init=init)

interloper_sql-0.2.0rc1/src/interloper_sql/io/sqlite.py

@@ -1,45 +0,0 @@
-"""SQLite IO implementation."""
-
-from __future__ import annotations
-
-from typing import TYPE_CHECKING
-
-from interloper.io.database import WriteDisposition
-from interloper.serialization.io import IOSpec
-
-from interloper_sql.io.base import SqlIO
-
-if TYPE_CHECKING:
-    from interloper.io.adapter import DataAdapter
-
-
-class SqliteIO(SqlIO):
-    """SQLite-specific IO manager.
-
-    Extends :class:`SqlIO` for SQLite connections. Useful for local development
-    and testing without requiring an external database server.
-
-    Args:
-        database: Path to the SQLite database file, or ``":memory:"`` for an
-            in-memory database.
-        write_disposition: Controls whether existing rows are deleted before writing
-        chunk_size: Number of rows per insert batch
-        adapter: Optional data adapter for type conversion
-    """
-
-    def __init__(
-        self,
-        database: str = ":memory:",
-        write_disposition: WriteDisposition = WriteDisposition.REPLACE,
-        chunk_size: int = 1000,
-        adapter: DataAdapter | str | None = None,
-    ) -> None:
-        self.database = database
-        url = f"sqlite:///{database}"
-        super().__init__(url, write_disposition, chunk_size, adapter)
-
-    def to_spec(self) -> IOSpec:
-        """Convert to serializable spec."""
-        init = self._base_init_kwargs()
-        init["database"] = self.database
-        return IOSpec(path=self.path, init=init)