interloper-snowflake 0.94.0__tar.gz

interloper_snowflake-0.94.0/PKG-INFO

@@ -0,0 +1,148 @@
+Metadata-Version: 2.3
+Name: interloper-snowflake
+Version: 0.94.0
+Summary: Interloper Snowflake integration: connection and destination
+Author: Guillaume Onfroy
+Author-email: Guillaume Onfroy <guillaume@digitlcloud.com>
+Requires-Dist: interloper-core
+Requires-Dist: interloper-pandas
+Requires-Dist: snowflake-connector-python[pandas]>=3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# interloper-snowflake
+
+Snowflake integration for interloper: a `SnowflakeDestination` that stores
+assets as Snowflake tables, and the `SnowflakeConnection` that holds the
+credentials.
+
+The destination is a `DatabaseDestination`: it writes the Snowflake dialect
+and nothing else. Partition replacement, windows and reads by partition come
+from core, exactly as for BigQuery.
+
+## Setup
+
+The connection needs an account identifier, a user and a password, and
+optionally a role (the user's default role otherwise).
+
+The account identifier is the part of your Snowflake URL before
+`.snowflakecomputing.com`, in either format:
+
+- `myorg-myaccount`: organisation and account name (preferred)
+- `xy12345.eu-central-1`: the legacy account locator with its region, and
+  its cloud where your URL carries one (e.g. `xy12345.us-east-2.aws`)
+
+The role the connection runs as needs:
+
+- `USAGE` on the warehouse the destination loads through
+- `USAGE` on the database
+- `CREATE SCHEMA` on the database, or `USAGE` on every schema the assets
+  write to if you create them yourself
+- `CREATE TABLE` and `CREATE STAGE` on those schemas (the load goes through a
+  temporary stage)
+- `INSERT`, `DELETE`, `SELECT` on the tables, and `TRUNCATE` for
+  unpartitioned assets (the table owner has all of them)
+
+A sketch for a dedicated loading role:
+
+```sql
+CREATE ROLE interloper;
+GRANT USAGE ON WAREHOUSE load_wh TO ROLE interloper;
+GRANT USAGE, CREATE SCHEMA ON DATABASE analytics TO ROLE interloper;
+GRANT ROLE interloper TO USER interloper_loader;
+```
+
+Schemas and tables the role creates are owned by it, so the remaining grants
+follow.
+
+## Usage
+
+```python
+import interloper as il
+from interloper_snowflake import SnowflakeConnection, SnowflakeDestination
+
+destination = SnowflakeDestination(
+    connection=SnowflakeConnection(account="myorg-myaccount", user="LOADER", password="..."),
+    database="ANALYTICS",
+    warehouse="LOAD_WH",
+    default_dataset="raw",
+)
+```
+
+The credentials also load from the environment (`SNOWFLAKE_ACCOUNT`,
+`SNOWFLAKE_USER`, `SNOWFLAKE_PASSWORD`, `SNOWFLAKE_ROLE`), so
+`SnowflakeConnection()` works with no arguments.
+
+In a deployed instance you configure this through the UI instead: add a
+Snowflake connection, then a Snowflake destination, picking the database and
+warehouse from the lists the connection can see.
+
+## Datasets are schemas
+
+An asset's `dataset` is the Snowflake schema its table lives in, inside the
+destination's `database`. An asset without a dataset falls back to
+`default_dataset`; with neither, the write fails with a `ConfigError`. A
+missing schema is created on the first write, and a missing table is created
+typed from the asset's schema (or one inferred from the data):
+
+| Field type | Snowflake type |
+|------------|----------------|
+| `bool` | `BOOLEAN` |
+| `int` | `NUMBER(38,0)` |
+| `float` | `FLOAT` |
+| `Decimal` | `NUMBER(38,9)` |
+| `datetime` | `TIMESTAMP_NTZ` |
+| `date` | `DATE` |
+| `bytes` | `BINARY` |
+| `str`, `Any` | `VARCHAR` |
+| nested models, lists, dicts | `VARIANT` |
+
+An existing table is never altered: a column the data carries but the table
+does not is dropped with a warning.
+
+## Quoted identifiers
+
+Every database, schema, table and column name is double-quoted, so Snowflake
+keeps it exactly as the asset spells it. Snowflake folds *unquoted*
+identifiers to upper case, so a lower-case asset must be queried with quotes:
+
+```sql
+SELECT "cost" FROM "ANALYTICS"."marts"."ads_stats";
+-- SELECT cost FROM analytics.marts.ads_stats looks for "COST" in "ADS_STATS" and fails
+```
+
+## Partitions
+
+A partitioned write replaces the partition's rows: it deletes them, then loads
+the data, inside one `BEGIN ... COMMIT`, rolled back on failure. A time
+partition deletes by half-open bounds (`"day" >= %s AND "day" < %s`), so a
+monthly partition whose rows hold daily dates is replaced whole; any other
+partition deletes by equality on its id. A window deletes each partition it
+covers and loads the whole batch once. An unpartitioned asset truncates its
+table and reloads it.
+
+Each write first does everything that is DDL or file transfer: it creates the
+schema and table if missing, creates a temporary stage in the schema (once per
+session), and uploads the data as one Parquet file with `PUT`, under a prefix of
+its own. Only then does it open the transaction, which holds nothing but the
+`DELETE` (or `TRUNCATE`) and a `COPY INTO` projecting the file's columns by name
+(`$1:"cost"`). Snowflake commits an open transaction whenever it runs DDL, so
+keeping DDL out of the block is what makes a replace atomic: a failed load
+rolls back the delete and the partition keeps its old rows. Reads go through
+`fetch_pandas_all`, so both directions stay columnar.
+
+## Notes
+
+Each destination opens its own session through the connection, on its own
+warehouse and database, so destinations sharing a connection never switch
+each other's warehouse. That session is shared by every asset the destination
+writes, and a Snowflake transaction belongs to the session rather than to a
+cursor, so the destination serialises its writes.
+
+## Docker images
+
+The published interloper images do not ship this package. They are built on
+Alpine, and `snowflake-connector-python` publishes no musllinux wheels, so
+installing it there means compiling its C++ extension from source. Run it from
+a glibc-based image (for example a `python:3.12-slim` base with
+`pip install interloper-snowflake`), or anywhere outside the images.

interloper_snowflake-0.94.0/README.md

@@ -0,0 +1,136 @@
+# interloper-snowflake
+
+Snowflake integration for interloper: a `SnowflakeDestination` that stores
+assets as Snowflake tables, and the `SnowflakeConnection` that holds the
+credentials.
+
+The destination is a `DatabaseDestination`: it writes the Snowflake dialect
+and nothing else. Partition replacement, windows and reads by partition come
+from core, exactly as for BigQuery.
+
+## Setup
+
+The connection needs an account identifier, a user and a password, and
+optionally a role (the user's default role otherwise).
+
+The account identifier is the part of your Snowflake URL before
+`.snowflakecomputing.com`, in either format:
+
+- `myorg-myaccount`: organisation and account name (preferred)
+- `xy12345.eu-central-1`: the legacy account locator with its region, and
+  its cloud where your URL carries one (e.g. `xy12345.us-east-2.aws`)
+
+The role the connection runs as needs:
+
+- `USAGE` on the warehouse the destination loads through
+- `USAGE` on the database
+- `CREATE SCHEMA` on the database, or `USAGE` on every schema the assets
+  write to if you create them yourself
+- `CREATE TABLE` and `CREATE STAGE` on those schemas (the load goes through a
+  temporary stage)
+- `INSERT`, `DELETE`, `SELECT` on the tables, and `TRUNCATE` for
+  unpartitioned assets (the table owner has all of them)
+
+A sketch for a dedicated loading role:
+
+```sql
+CREATE ROLE interloper;
+GRANT USAGE ON WAREHOUSE load_wh TO ROLE interloper;
+GRANT USAGE, CREATE SCHEMA ON DATABASE analytics TO ROLE interloper;
+GRANT ROLE interloper TO USER interloper_loader;
+```
+
+Schemas and tables the role creates are owned by it, so the remaining grants
+follow.
+
+## Usage
+
+```python
+import interloper as il
+from interloper_snowflake import SnowflakeConnection, SnowflakeDestination
+
+destination = SnowflakeDestination(
+    connection=SnowflakeConnection(account="myorg-myaccount", user="LOADER", password="..."),
+    database="ANALYTICS",
+    warehouse="LOAD_WH",
+    default_dataset="raw",
+)
+```
+
+The credentials also load from the environment (`SNOWFLAKE_ACCOUNT`,
+`SNOWFLAKE_USER`, `SNOWFLAKE_PASSWORD`, `SNOWFLAKE_ROLE`), so
+`SnowflakeConnection()` works with no arguments.
+
+In a deployed instance you configure this through the UI instead: add a
+Snowflake connection, then a Snowflake destination, picking the database and
+warehouse from the lists the connection can see.
+
+## Datasets are schemas
+
+An asset's `dataset` is the Snowflake schema its table lives in, inside the
+destination's `database`. An asset without a dataset falls back to
+`default_dataset`; with neither, the write fails with a `ConfigError`. A
+missing schema is created on the first write, and a missing table is created
+typed from the asset's schema (or one inferred from the data):
+
+| Field type | Snowflake type |
+|------------|----------------|
+| `bool` | `BOOLEAN` |
+| `int` | `NUMBER(38,0)` |
+| `float` | `FLOAT` |
+| `Decimal` | `NUMBER(38,9)` |
+| `datetime` | `TIMESTAMP_NTZ` |
+| `date` | `DATE` |
+| `bytes` | `BINARY` |
+| `str`, `Any` | `VARCHAR` |
+| nested models, lists, dicts | `VARIANT` |
+
+An existing table is never altered: a column the data carries but the table
+does not is dropped with a warning.
+
+## Quoted identifiers
+
+Every database, schema, table and column name is double-quoted, so Snowflake
+keeps it exactly as the asset spells it. Snowflake folds *unquoted*
+identifiers to upper case, so a lower-case asset must be queried with quotes:
+
+```sql
+SELECT "cost" FROM "ANALYTICS"."marts"."ads_stats";
+-- SELECT cost FROM analytics.marts.ads_stats looks for "COST" in "ADS_STATS" and fails
+```
+
+## Partitions
+
+A partitioned write replaces the partition's rows: it deletes them, then loads
+the data, inside one `BEGIN ... COMMIT`, rolled back on failure. A time
+partition deletes by half-open bounds (`"day" >= %s AND "day" < %s`), so a
+monthly partition whose rows hold daily dates is replaced whole; any other
+partition deletes by equality on its id. A window deletes each partition it
+covers and loads the whole batch once. An unpartitioned asset truncates its
+table and reloads it.
+
+Each write first does everything that is DDL or file transfer: it creates the
+schema and table if missing, creates a temporary stage in the schema (once per
+session), and uploads the data as one Parquet file with `PUT`, under a prefix of
+its own. Only then does it open the transaction, which holds nothing but the
+`DELETE` (or `TRUNCATE`) and a `COPY INTO` projecting the file's columns by name
+(`$1:"cost"`). Snowflake commits an open transaction whenever it runs DDL, so
+keeping DDL out of the block is what makes a replace atomic: a failed load
+rolls back the delete and the partition keeps its old rows. Reads go through
+`fetch_pandas_all`, so both directions stay columnar.
+
+## Notes
+
+Each destination opens its own session through the connection, on its own
+warehouse and database, so destinations sharing a connection never switch
+each other's warehouse. That session is shared by every asset the destination
+writes, and a Snowflake transaction belongs to the session rather than to a
+cursor, so the destination serialises its writes.
+
+## Docker images
+
+The published interloper images do not ship this package. They are built on
+Alpine, and `snowflake-connector-python` publishes no musllinux wheels, so
+installing it there means compiling its C++ extension from source. Run it from
+a glibc-based image (for example a `python:3.12-slim` base with
+`pip install interloper-snowflake`), or anywhere outside the images.

interloper_snowflake-0.94.0/pyproject.toml

@@ -0,0 +1,63 @@
+[project]
+name = "interloper-snowflake"
+version = "0.94.0"
+description = "Interloper Snowflake integration: connection and destination"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "interloper-core",
+    "interloper-pandas",
+    "snowflake-connector-python[pandas]>=3.12",
+]
+
+[[project.authors]]
+name = "Guillaume Onfroy"
+email = "guillaume@digitlcloud.com"
+
+[project.entry-points."interloper.components"]
+snowflake = "interloper_snowflake"
+
+[build-system]
+requires = ["uv_build>=0.12.9,<0.13"]
+build-backend = "uv_build"
+
+[tool.uv.sources.interloper-core]
+workspace = true
+
+[tool.uv.sources.interloper-pandas]
+workspace = true
+
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+preview = true
+extend-select = [
+    "E",
+    "I",
+    "UP",
+    "ANN001",
+    "ANN201",
+    "ANN202",
+    "DOC",
+    "D",
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = [
+    "F401",
+    "F403",
+]
+"tests/**" = [
+    "ANN",
+    "F811",
+    "D101",
+    "D102",
+    "D103",
+    "D104",
+    "RUF069",
+    "PLW0108",
+]

interloper_snowflake-0.94.0/pyproject.toml.orig

@@ -0,0 +1,43 @@
+# ###############
+# PROJECT / UV
+# ###############
+[project]
+name = "interloper-snowflake"
+version = "0.94.0"
+description = "Interloper Snowflake integration: connection and destination"
+readme = "README.md"
+authors = [{ name = "Guillaume Onfroy", email = "guillaume@digitlcloud.com" }]
+requires-python = ">=3.10"
+dependencies = [
+    "interloper-core",
+    "interloper-pandas",
+    "snowflake-connector-python[pandas]>=3.12",
+]
+
+[project.entry-points."interloper.components"]
+snowflake = "interloper_snowflake"
+
+[build-system]
+requires = ["uv_build>=0.12.9,<0.13"]
+build-backend = "uv_build"
+
+[tool.uv.sources]
+interloper-core = { workspace = true }
+interloper-pandas = { workspace = true }
+
+# ###############
+# RUFF
+# ###############
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+preview = true
+extend-select = ["E", "I", "UP", "ANN001", "ANN201", "ANN202", "DOC", "D"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = ["F401", "F403"]
+"tests/**" = ["ANN", "F811", "D101", "D102", "D103", "D104", "RUF069", "PLW0108"]

interloper_snowflake-0.94.0/src/interloper_snowflake/__init__.py

@@ -0,0 +1,9 @@
+"""Interloper Snowflake integration: connection and destination."""
+
+from interloper_snowflake.connection import SnowflakeConnection
+from interloper_snowflake.destination import SnowflakeDestination
+
+__all__ = [
+    "SnowflakeConnection",
+    "SnowflakeDestination",
+]

interloper_snowflake-0.94.0/src/interloper_snowflake/connection.py

@@ -0,0 +1,126 @@
+"""Snowflake connection resource holding user and password credentials."""
+
+from __future__ import annotations
+
+from functools import cached_property
+from typing import Any
+
+import snowflake.connector
+from interloper.connection import Connection, connection
+from interloper.resource.fields import InputField, SecretField, fetch_field_provider
+from pydantic_settings import SettingsConfigDict
+from snowflake.connector import SnowflakeConnection as Session
+
+
+@connection(
+    key="snowflake_connection",
+    name="Snowflake",
+    icon="icon:snowflake",
+    tags=["Cloud"],
+)
+class SnowflakeConnection(Connection):
+    """Connection resource holding Snowflake credentials.
+
+    The connection holds the credentials; each destination bound to it opens
+    its own session on its own database and warehouse.
+    """
+
+    model_config = SettingsConfigDict(env_prefix="snowflake_")
+
+    account: str = InputField(label="Account identifier", description="e.g. xy12345.eu-central-1 or myorg-myaccount")
+    user: str = InputField(description="Snowflake user name")
+    password: str = SecretField(description="Snowflake user password")
+    role: str | None = InputField(default=None, description="Role to assume; the user's default role when empty")
+
+    def connect(self, **session: Any) -> Session:
+        """Open a new Snowflake session with this connection's credentials.
+
+        Autocommit stays on so a lone statement commits by itself; a caller
+        that needs atomicity opens an explicit ``BEGIN``.
+
+        Args:
+            **session: Session settings passed to the connector, such as
+                ``warehouse`` and ``database``.
+
+        Returns:
+            The new connector session.
+        """
+        return snowflake.connector.connect(
+            account=self.account,
+            user=self.user,
+            password=self.password,
+            role=self.role,
+            autocommit=True,
+            **session,
+        )
+
+    @cached_property
+    def client(self) -> Session:
+        """The session the connection's own check and pickers run on.
+
+        A destination opens its own session through :meth:`connect` instead,
+        so its warehouse and transactions never touch a session another
+        component shares.
+
+        Returns:
+            The connector session, cached per connection instance.
+        """
+        return self.connect()
+
+    def _names(self, sql: str) -> list[dict[str, str]]:
+        """Run a ``SHOW`` statement and return its ``name`` column as options.
+
+        ``SHOW`` output carries many columns whose order is not part of its
+        contract, so the column is found by name through the cursor's
+        description.
+
+        Args:
+            sql: The ``SHOW`` statement to run.
+
+        Returns:
+            Options with ``name``, sorted case-insensitively.
+        """
+        cursor = self.client.cursor()
+        try:
+            cursor.execute(sql)
+            rows: list[Any] = cursor.fetchall()
+            index = [column[0] for column in cursor.description].index("name")
+        finally:
+            cursor.close()
+        return sorted(({"name": row[index]} for row in rows), key=lambda option: option["name"].lower())
+
+    @fetch_field_provider
+    def databases(self) -> list[dict[str, str]]:
+        """List the databases this connection's role can see.
+
+        Backs the destination's ``database`` ``FetchField``.
+
+        Returns:
+            Database options with ``name``.
+        """
+        return self._names("SHOW DATABASES")
+
+    @fetch_field_provider
+    def warehouses(self) -> list[dict[str, str]]:
+        """List the virtual warehouses this connection's role can see.
+
+        Backs the destination's ``warehouse`` ``FetchField``.
+
+        Returns:
+            Warehouse options with ``name``.
+        """
+        return self._names("SHOW WAREHOUSES")
+
+    def check(self) -> bool:
+        """Prove the credentials work by opening a session and running ``SELECT 1``.
+
+        Returns:
+            True; a login or network failure raises out of the connector.
+        """
+        cursor = self.client.cursor()
+        try:
+            cursor.execute("SELECT 1")
+            cursor.fetchall()
+        finally:
+            cursor.close()
+        return True

interloper_snowflake-0.94.0/src/interloper_snowflake/destination.py

@@ -0,0 +1,448 @@
+"""Snowflake destination implementation."""
+
+from __future__ import annotations
+
+import tempfile
+import threading
+import uuid
+import warnings
+from collections.abc import Iterator, Sequence
+from contextlib import contextmanager
+from dataclasses import dataclass
+from functools import cached_property
+from pathlib import Path
+from typing import Any
+
+import pandas as pd
+from interloper.destination import IOContext, destination
+from interloper.destination.database import DatabaseDestination, PartitionFilter
+from interloper.errors import ConfigError, DataNotFoundError
+from interloper.representation import Representation
+from interloper.resource.fields import FetchField, InputField
+from interloper.schema import FieldSpec
+from interloper.utils.data import is_empty
+from pydantic import PrivateAttr
+from snowflake.connector import SnowflakeConnection as Session
+from snowflake.connector.cursor import SnowflakeCursor
+
+from interloper_snowflake.connection import SnowflakeConnection
+from interloper_snowflake.types import column_type
+
+_STAGE = "interloper_load"
+
+
+def _quote(identifier: str) -> str:
+    """Quote an identifier so Snowflake keeps it exactly as written.
+
+    Args:
+        identifier: A database, schema, table or column name.
+
+    Returns:
+        The identifier in double quotes, embedded quotes doubled.
+    """
+    return '"' + identifier.replace('"', '""') + '"'
+
+
+@dataclass(frozen=True)
+class _Staged:
+    """Data uploaded to a stage, ready to be copied into its table.
+
+    Attributes:
+        table: The target table name.
+        schema: The resolved schema name.
+        location: The stage path holding the file, ``@stage/prefix``.
+        columns: The file's columns, all of them the table's.
+    """
+
+    table: str
+    schema: str
+    location: str
+    columns: tuple[str, ...]
+
+
+@destination(
+    key="snowflake_destination",
+    name="Snowflake",
+    icon="icon:snowflake",
+    tags=["Cloud"],
+)
+class SnowflakeDestination(DatabaseDestination):
+    """Snowflake destination.
+
+    A dataset is a Snowflake schema inside the destination's database. Every
+    identifier is quoted, so tables and columns keep the case the asset gives
+    them.
+
+    The destination opens its own session on its warehouse and database. That
+    session is still shared by every asset written through the destination,
+    and a Snowflake transaction belongs to the session, so writes are
+    serialised: two concurrent writes would otherwise commit or roll back each
+    other's statements.
+    """
+
+    connection: SnowflakeConnection
+
+    database: str = FetchField(
+        provider="connection.databases",
+        label_key="name",
+        value_key="name",
+        description="Snowflake database",
+        discriminator=True,
+    )
+    warehouse: str = FetchField(
+        provider="connection.warehouses",
+        label_key="name",
+        value_key="name",
+        description="Virtual warehouse running the loads and queries",
+    )
+    default_dataset: str | None = InputField(default=None, description="Default schema for assets without a dataset")
+
+    _stages: set[str] = PrivateAttr(default_factory=set)
+    _lock: Any = PrivateAttr(default_factory=threading.RLock)
+    _local: threading.local = PrivateAttr(default_factory=threading.local)
+
+    @cached_property
+    def client(self) -> Session:
+        """The destination's own session, on its warehouse and database.
+
+        Returns:
+            The connector session, cached per destination instance.
+        """
+        return self.connection.connect(warehouse=self.warehouse, database=self.database)
+
+    # -- Session -----------------------------------------------------------------
+
+    def _execute(self, sql: str, params: Sequence[Any] | None = None) -> SnowflakeCursor:
+        """Run one statement, on the open transaction's cursor or on a fresh one.
+
+        Args:
+            sql: The statement, with ``%s`` placeholders for *params*.
+            params: The statement's parameters; defaults to none.
+
+        Returns:
+            The cursor the statement ran on, ready to fetch from.
+        """
+        cursor = getattr(self._local, "cursor", None)
+        if cursor is None:
+            cursor = self.client.cursor()
+        cursor.execute(sql, params)
+        return cursor
+
+    @contextmanager
+    def transaction(self) -> Iterator[None]:
+        """Run one write, a delete followed by an insert, as ``BEGIN ... COMMIT``.
+
+        Yields:
+            ``None``; the write runs inside the block, rolled back and
+            re-raised if it raises.
+        """
+        with self._lock:
+            cursor = self.client.cursor()
+            cursor.execute("BEGIN")
+            self._local.cursor = cursor
+            try:
+                yield
+            except BaseException:
+                cursor.execute("ROLLBACK")
+                raise
+            else:
+                cursor.execute("COMMIT")
+            finally:
+                self._local.cursor = None
+                cursor.close()
+
+    # -- Destination interface -----------------------------------------------------
+
+    def write(self, context: IOContext, data: Any) -> None:
+        """Stage the data, then let the base replace its partitions.
+
+        Snowflake commits an open transaction whenever it runs DDL, and the
+        base calls :meth:`insert` inside :meth:`transaction`. So everything
+        that is DDL or file transfer (creating the schema, the table and the
+        stage, and the ``PUT``) runs here, before the base opens ``BEGIN``;
+        inside it, only the ``DELETE`` and the ``COPY INTO`` run. Both of the
+        base's paths, a single partition and a window, pass through here.
+
+        The whole write holds the destination's lock: the session is shared,
+        so another write's DDL would otherwise commit this one's open
+        transaction.
+
+        Args:
+            context: IO context carrying the target asset, the partition or window,
+                and the effective schema.
+            data: The data to write, in its native representation.
+        """
+        if is_empty(data):
+            return
+        table, dataset = self._target(context)
+        with self._lock:
+            self._local.staged = self._stage(table, dataset, data, context)
+            try:
+                super().write(context, data)
+            finally:
+                self._local.staged = None
+
+    # -- Naming --------------------------------------------------------------------
+
+    def _resolve_dataset(self, dataset: str | None) -> str:
+        """Return the Snowflake schema to use.
+
+        Args:
+            dataset: The asset's dataset, or ``None`` to fall back to the destination's default.
+
+        Returns:
+            The resolved schema name.
+
+        Raises:
+            ConfigError: If neither the asset nor the destination names a dataset.
+        """
+        schema = dataset or self.default_dataset
+        if schema is None:
+            raise ConfigError(
+                "SnowflakeDestination requires a dataset. Either set 'dataset' on the asset "
+                "or provide 'default_dataset' on the destination."
+            )
+        return schema
+
+    def _table_ref(self, table: str, schema: str) -> str:
+        """Build a fully-qualified, quoted table reference.
+
+        Args:
+            table: Table name.
+            schema: The resolved schema name.
+
+        Returns:
+            ``"database"."schema"."table"``.
+        """
+        return f"{_quote(self.database)}.{_quote(schema)}.{_quote(table)}"
+
+    def _table_exists(self, table: str, schema: str) -> bool:
+        """Check whether a table exists, through the database's information schema.
+
+        Args:
+            table: Table name.
+            schema: The resolved schema name.
+
+        Returns:
+            ``True`` if the table exists, ``False`` otherwise.
+        """
+        cursor = self._execute(
+            f"SELECT 1 FROM {_quote(self.database)}.information_schema.tables "
+            "WHERE table_schema = %s AND table_name = %s",
+            (schema, table),
+        )
+        return bool(cursor.fetchall())
+
+    # -- Staging -------------------------------------------------------------------
+
+    def _ensure_table(self, table: str, schema: str, specs: Sequence[FieldSpec]) -> None:
+        """Create the schema and a typed table when the table does not exist.
+
+        Args:
+            table: Table name.
+            schema: The resolved schema name.
+            specs: The table's field specs.
+        """
+        if self._table_exists(table, schema):
+            return
+        self._execute(f"CREATE SCHEMA IF NOT EXISTS {_quote(self.database)}.{_quote(schema)}")
+        self._execute(f"CREATE TABLE IF NOT EXISTS {self._table_ref(table, schema)} ({_columns_ddl(specs)})")
+
+    def _ensure_stage(self, schema: str) -> str:
+        """Create the session's temporary stage in a schema, once per session.
+
+        Args:
+            schema: The resolved schema name.
+
+        Returns:
+            The stage's fully-qualified, quoted name.
+        """
+        stage = f"{_quote(self.database)}.{_quote(schema)}.{_quote(_STAGE)}"
+        if schema not in self._stages:
+            self._execute(f"CREATE TEMPORARY STAGE IF NOT EXISTS {stage}")
+            self._stages.add(schema)
+        return stage
+
+    def _stage(self, table: str, dataset: str | None, data: Any, context: IOContext) -> _Staged:
+        """Create what the load needs and upload the data as one Parquet file.
+
+        A new table is typed from the effective schema (declared on the asset,
+        or inferred during conform), or from a schema inferred from the data
+        when the context carries none. The frame is aligned to the table's
+        columns: an extra column is dropped with a warning, since an existing
+        table is never altered. The file goes under a prefix of its own, so
+        concurrent writes never load each other's files.
+
+        Args:
+            table: Target table name.
+            dataset: The Snowflake schema, or ``None`` for the destination's default.
+            data: The data in its native representation.
+            context: IO context carrying the asset and effective schema.
+
+        Returns:
+            Where the file was staged and the columns it carries.
+        """
+        schema = self._resolve_dataset(dataset)
+        specs = (context.schema or Representation.of(data).infer()).field_specs()
+        self._ensure_table(table, schema, specs)
+        stage = self._ensure_stage(schema)
+
+        frame = Representation.of(data).to("dataframe")
+        names = [spec.name for spec in specs]
+        extras = [str(c) for c in frame.columns if str(c) not in names]
+        if extras:
+            warnings.warn(
+                f"Columns {extras} are not in the schema for '{self._table_ref(table, schema)}' "
+                "and will not be written.",
+                UserWarning,
+                stacklevel=3,
+            )
+        columns = tuple(c for c in names if c in frame.columns)
+
+        location = f"@{stage}/{uuid.uuid4().hex}"
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory) / "data.parquet"
+            frame[list(columns)].to_parquet(path, index=False)
+            uri = path.as_posix().replace("'", "\\'")
+            self._execute(f"PUT 'file://{uri}' {location} OVERWRITE=TRUE AUTO_COMPRESS=FALSE")
+        return _Staged(table=table, schema=schema, location=location, columns=columns)
+
+    # -- DatabaseDestination hooks ---------------------------------------------------
+
+    def insert(self, table: str, dataset: str | None, data: Any, context: IOContext) -> None:
+        """Copy the staged data into the table.
+
+        :meth:`write` has already staged the data outside the transaction;
+        called any other way, the data is staged here first.
+
+        Args:
+            table: Target table name.
+            dataset: The Snowflake schema, or ``None`` for the destination's default.
+            data: The data in its native representation.
+            context: IO context carrying the asset and effective schema.
+        """
+        staged = getattr(self._local, "staged", None)
+        if staged is None or staged.table != table or staged.schema != self._resolve_dataset(dataset):
+            staged = self._stage(table, dataset, data, context)
+        self._copy(staged)
+
+    def _copy(self, staged: _Staged) -> None:
+        """Load a staged Parquet file with ``COPY INTO``, projecting its columns by name.
+
+        Parquet data is one ``$1`` object per row, so each column is read as
+        ``$1:"name"`` and the file's column order does not matter. Binary
+        columns stay binary, and logical types (dates, timestamps, decimals)
+        are honoured. The file is purged once loaded.
+
+        Args:
+            staged: The staged file and its columns.
+        """
+        targets = ", ".join(_quote(c) for c in staged.columns)
+        projection = ", ".join(f"$1:{_quote(c)}" for c in staged.columns)
+        self._execute(
+            f"COPY INTO {self._table_ref(staged.table, staged.schema)} ({targets}) "
+            f"FROM (SELECT {projection} FROM {staged.location}) "
+            "FILE_FORMAT=(TYPE=PARQUET USE_LOGICAL_TYPE=TRUE BINARY_AS_TEXT=FALSE) PURGE=TRUE"
+        )
+
+    def delete(self, table: str, dataset: str | None, where: PartitionFilter | None) -> None:
+        """Delete the rows a filter selects, or truncate the table.
+
+        A table that does not exist has nothing to delete.
+
+        Args:
+            table: Target table name.
+            dataset: The Snowflake schema, or ``None`` for the destination's default.
+            where: The rows to delete; ``None`` for the whole table.
+        """
+        schema = self._resolve_dataset(dataset)
+        if not self._table_exists(table, schema):
+            return
+        ref = self._table_ref(table, schema)
+        if where is None:
+            self._execute(f"TRUNCATE TABLE {ref}")
+            return
+        predicate, params = _predicate(where)
+        self._execute(f"DELETE FROM {ref} WHERE {predicate}", params)
+
+    def select(self, table: str, dataset: str | None, where: PartitionFilter | None) -> pd.DataFrame:
+        """Select the rows a filter selects, or every row, as a DataFrame.
+
+        The connector builds the frame from the result's Arrow batches, so
+        column types survive the read without a pass through Python records.
+
+        Args:
+            table: Target table name.
+            dataset: The Snowflake schema, or ``None`` for the destination's default.
+            where: The rows to select; ``None`` for the whole table.
+
+        Returns:
+            The selected rows.
+
+        Raises:
+            DataNotFoundError: If the table does not exist yet.
+        """
+        schema = self._resolve_dataset(dataset)
+        ref = self._table_ref(table, schema)
+        if not self._table_exists(table, schema):
+            raise DataNotFoundError(f"Table '{ref}' does not exist. Has the asset been materialized?")
+        if where is None:
+            return self._execute(f"SELECT * FROM {ref}").fetch_pandas_all()
+        predicate, params = _predicate(where)
+        return self._execute(f"SELECT * FROM {ref} WHERE {predicate}", params).fetch_pandas_all()
+
+    def count(self, table: str, dataset: str | None, column: str) -> dict[str, int]:
+        """Return row counts grouped by a column.
+
+        Args:
+            table: Target table name.
+            dataset: The Snowflake schema, or ``None`` for the destination's default.
+            column: Column to group by.
+
+        Returns:
+            Mapping from the column's value (as string) to row count.
+
+        Raises:
+            DataNotFoundError: If the table does not exist yet.
+        """
+        schema = self._resolve_dataset(dataset)
+        ref = self._table_ref(table, schema)
+        if not self._table_exists(table, schema):
+            raise DataNotFoundError(f"Table '{ref}' does not exist. Has the asset been materialized?")
+        cursor = self._execute(
+            f"SELECT TO_VARCHAR({_quote(column)}) AS partition_value, COUNT(*) AS cnt FROM {ref} GROUP BY 1"
+        )
+        return {value: count for value, count in cursor.fetchall()}
+
+
+# -- Utility functions ---------------------------------------------------------------
+
+
+def _columns_ddl(specs: Sequence[FieldSpec]) -> str:
+    """Render field specs as the column list of a ``CREATE TABLE``.
+
+    Args:
+        specs: The table's field specs.
+
+    Returns:
+        Comma-separated quoted column definitions. Every column is nullable:
+        conform already enforces the schema's nullability, and a constraint
+        here would only turn a schema change into a failed load.
+    """
+    return ", ".join(f"{_quote(spec.name)} {column_type(spec)}" for spec in specs)
+
+
+def _predicate(where: PartitionFilter) -> tuple[str, tuple[Any, ...]]:
+    """Render a partition filter as a parameterised SQL predicate.
+
+    Args:
+        where: The filter to render.
+
+    Returns:
+        The predicate text and the parameters its ``%s`` placeholders name.
+    """
+    column = _quote(where.column)
+    if where.bounds is None:
+        return f"{column} = %s", (where.value,)
+    start, end = where.bounds
+    return f"{column} >= %s AND {column} < %s", (start, end)

interloper_snowflake-0.94.0/src/interloper_snowflake/types.py

@@ -0,0 +1,43 @@
+"""Snowflake's view of interloper's field types."""
+
+from __future__ import annotations
+
+import datetime
+from decimal import Decimal
+
+from interloper.schema import FieldSpec
+
+# Ordered: the first base class that matches wins, so bool (a subclass of int)
+# and datetime (a subclass of date) must come before their parents.
+_PYTHON_TO_SNOWFLAKE: dict[type, str] = {
+    bool: "BOOLEAN",
+    int: "NUMBER(38,0)",
+    float: "FLOAT",
+    Decimal: "NUMBER(38,9)",
+    datetime.datetime: "TIMESTAMP_NTZ",
+    datetime.date: "DATE",
+    bytes: "BINARY",
+    str: "VARCHAR",
+    dict: "VARIANT",
+    list: "VARIANT",
+}
+
+
+def column_type(spec: FieldSpec) -> str:
+    """Return the Snowflake column type for a field spec.
+
+    Args:
+        spec: The field spec, from :meth:`Schema.field_specs` or an inferred schema.
+
+    Returns:
+        ``VARIANT`` for a nested or repeated field, the type the spec's Python
+        type maps to otherwise, and ``VARCHAR`` for anything unmapped
+        (``typing.Any``).
+    """
+    if spec.fields is not None or spec.repeated:
+        return "VARIANT"
+    if isinstance(spec.type, type):
+        for base, name in _PYTHON_TO_SNOWFLAKE.items():
+            if issubclass(spec.type, base):
+                return name
+    return "VARCHAR"