interloper-azure 0.96.0__tar.gz

interloper_azure-0.96.0/PKG-INFO

@@ -0,0 +1,194 @@
+Metadata-Version: 2.3
+Name: interloper-azure
+Version: 0.96.0
+Summary: Interloper Microsoft Azure integration: Entra connection and Microsoft Fabric Warehouse destination
+Author: Guillaume Onfroy
+Author-email: Guillaume Onfroy <guillaume@digitlcloud.com>
+Requires-Dist: interloper-core
+Requires-Dist: azure-identity>=1.19
+Requires-Dist: mssql-python>=1.15
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# interloper-azure
+
+Microsoft Azure integration for interloper: the `AzureConnection` that holds a
+Microsoft Entra service principal, and a `FabricWarehouseDestination` that
+stores assets as tables in a Microsoft Fabric Warehouse.
+
+The destination is a `DatabaseDestination`: it writes Fabric's T-SQL dialect
+and nothing else. Partition replacement, windows and reads by partition come
+from core, exactly as for BigQuery or Snowflake.
+
+It targets a Fabric **Warehouse**. A Lakehouse's SQL analytics endpoint speaks
+the same protocol but is read-only: creating tables and inserting or deleting
+rows is only supported in a Warehouse.
+
+## Setup
+
+### 1. Create a service principal
+
+In the [Microsoft Entra admin center](https://entra.microsoft.com), under
+*App registrations*, register an application (no redirect URI needed), then on
+its *Certificates & secrets* page create a client secret. Note three values:
+
+- the *Directory (tenant) ID*
+- the *Application (client) ID*
+- the secret's *Value* (shown once, not its *Secret ID*)
+
+### 2. Let service principals use Fabric
+
+A Fabric administrator must enable **Service principals can use Fabric APIs**
+in the Fabric admin portal (*Tenant settings*, *Developer settings*), either for
+the whole organisation or for a security group the principal belongs to. The
+same setting governs both the REST API (used by the connection check and the
+workspace picker) and SQL connections to a warehouse.
+
+### 3. Give the principal access to the workspace
+
+In the workspace, open *Manage access* and add the app with the
+**Contributor** role. Contributor, like Admin and Member, grants `CONTROL` on
+every warehouse of the workspace, which covers creating schemas and tables and
+writing rows; `CREATE SCHEMA` in particular requires one of those three roles.
+Viewer only reads, and sharing a single warehouse with no extra permissions
+only lets the principal connect.
+
+### 4. Find the SQL connection string
+
+Open the warehouse's *Settings* and its *SQL connection string* page; the
+string looks like `xxxxxxxx-xxxx.datawarehouse.fabric.microsoft.com`. The string
+belongs to the workspace: every warehouse in it shares the same one, and the
+warehouse's name selects the database. In the UI you do not need to copy it:
+the destination's *Workspace* picker lists the workspaces the principal can see
+that hold a warehouse, and stores that workspace's connection string.
+
+## Usage
+
+```python
+from interloper_azure import AzureConnection, FabricWarehouseDestination
+
+destination = FabricWarehouseDestination(
+    connection=AzureConnection(tenant_id="...", client_id="...", client_secret="..."),
+    server="xxxxxxxx-xxxx.datawarehouse.fabric.microsoft.com",
+    warehouse="Analytics",
+    default_dataset="raw",
+)
+```
+
+The credentials also load from the environment under the standard
+azure-identity names (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`,
+`AZURE_CLIENT_SECRET`), so `AzureConnection()` works with no arguments.
+
+In a deployed instance you configure this through the UI: add a Microsoft
+Azure connection, then a Microsoft Fabric Warehouse destination, pick the
+workspace and type the warehouse name.
+
+## The driver and its system libraries
+
+Statements go through [`mssql-python`](https://github.com/microsoft/mssql-python),
+Microsoft's DB-API driver, which signs in with an Entra access token obtained
+from the connection's credential (scope `https://database.windows.net/.default`).
+It is pip-installable and bundles Microsoft's ODBC driver, but that driver
+loads a few system libraries on Linux. On Debian or Ubuntu:
+
+```sh
+apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2
+```
+
+The published scheduler and core images, the ones that run assets, ship these
+libraries. The api and mcp images only describe the destination and never open
+a session, so they leave them out.
+
+Without them the package imports fine (the catalog, the connection check and
+the workspace picker all work), but opening a warehouse session fails with
+`Failed to load the driver`. `mssql-python` also offers an alternate, Rust-based
+native provider that needs none of these libraries: install `mssql-python-rs`
+and set `MSSQL_PYTHON_NATIVE_PROVIDER=mssql-odbc`. Microsoft labels that
+provider alpha.
+
+## Datasets are schemas
+
+An asset's `dataset` is the warehouse schema its table lives in. An asset
+without a dataset falls back to `default_dataset`, then to `dbo`. A missing
+schema is created on the first write, and a missing table is created typed
+from the asset's schema (or one inferred from the data):
+
+| Field type | Fabric Warehouse type |
+|------------|-----------------------|
+| `bool` | `bit` |
+| `int` | `bigint` |
+| `float` | `float` |
+| `Decimal` | `decimal(38,9)` |
+| `datetime` | `datetime2(6)` |
+| `date` | `date` |
+| `bytes` | `varbinary(max)` |
+| `str`, `Any` | `varchar(max)` |
+| nested models, lists, dicts | `varchar(max)` holding JSON |
+
+An existing table is never altered: a column the data carries but the table
+does not is dropped with a warning. Reads return records; nested values come
+back as the JSON text they are stored as. Identifiers are bracketed
+(`[marts].[ads_stats]`), and the warehouse's default collation is
+case-sensitive, so a table is queried with the exact case the asset gives it.
+
+## How Fabric's T-SQL shaped the design
+
+Fabric Warehouse speaks T-SQL over the SQL Server protocol, but it is not SQL
+Server. The differences that matter here:
+
+- **Types.** Tables accept a subset of SQL Server's types: `datetime2` and
+  `time` keep at most six fractional digits, and there is no `datetime`,
+  `datetimeoffset`, `nvarchar`, `nchar`, `tinyint`, `money` or `json`. Hence
+  `varchar` (UTF-8) for text and JSON, and `datetime2(6)` for timestamps;
+  timezone-aware values are written as UTC. `varchar(max)` and
+  `varbinary(max)` exist but hold at most 16 MB per value. Microsoft
+  recommends the shortest fitting `varchar(n)` for query performance;
+  `varchar(max)` is used because the destination cannot know a column's
+  longest value up front.
+- **Schemas.** There is no `CREATE SCHEMA IF NOT EXISTS`, and `CREATE SCHEMA`
+  must be alone in its batch, so the destination checks `sys.schemas` first
+  and creates the schema in a statement of its own. Table existence is read
+  from `sys.tables` and `sys.columns`; the warehouse refuses a query mixing
+  system and user tables, so the catalog check never touches the data.
+- **Transactions.** Explicit `BEGIN TRANSACTION ... COMMIT TRANSACTION` is
+  supported, always under snapshot isolation, and DDL is allowed inside one.
+  DDL in a transaction holds locks on the catalog views until commit, though,
+  blocking every other write's existence checks, so a new schema and table
+  are created and committed before the write's transaction opens; inside it
+  run only the `DELETE` and the `INSERT`s.
+- **Whole-table replaces delete rather than truncate.** `TRUNCATE TABLE`
+  takes a schema-modification lock that blocks readers until commit; a
+  `DELETE` does not.
+- **Write-write conflicts are per table.** Two transactions deleting from the
+  same table conflict even when they touch different rows: the first to commit
+  wins and the other fails with error 24556 or 24706. Writes to different
+  tables never conflict. Concurrent writes of different partitions of one
+  asset (overlapping backfills, for instance) should be avoided, or covered by
+  a retry policy.
+
+## Partitions
+
+A partitioned write replaces the partition's rows: it deletes them, then
+inserts the data, inside one transaction, rolled back on failure. A time
+partition deletes by half-open bounds (`[day] >= ? AND [day] < ?`), so a
+monthly partition whose rows hold daily dates is replaced whole; any other
+partition deletes by equality on its id. A window deletes each partition it
+covers and inserts the whole batch once. An unpartitioned asset deletes every
+row and reloads the table. Readers keep seeing the previous rows until the
+commit.
+
+## The load path and its limits
+
+Rows are written with parameterised multi-row `INSERT ... VALUES` statements.
+Each statement carries as many rows as fit under SQL Server's 2100-parameter
+cap (2099 parameters, one per value) and 1000 rows, whichever is smaller: a
+5-column table loads 419 rows per statement, a 40-column table 52. Every
+statement is a round trip and writes new Parquet files under the table, which
+the warehouse compacts in the background.
+
+This needs no infrastructure beyond the warehouse and suits incremental loads:
+thousands to low hundreds of thousands of rows per write. Microsoft recommends
+against frequent small inserts, and for bulk volumes (millions of rows per
+write, or a full reload of a large table) the documented high-throughput path
+is `COPY INTO` from files staged in OneLake or Azure Data Lake Storage, which
+this destination does not implement.

interloper_azure-0.96.0/README.md

@@ -0,0 +1,182 @@
+# interloper-azure
+
+Microsoft Azure integration for interloper: the `AzureConnection` that holds a
+Microsoft Entra service principal, and a `FabricWarehouseDestination` that
+stores assets as tables in a Microsoft Fabric Warehouse.
+
+The destination is a `DatabaseDestination`: it writes Fabric's T-SQL dialect
+and nothing else. Partition replacement, windows and reads by partition come
+from core, exactly as for BigQuery or Snowflake.
+
+It targets a Fabric **Warehouse**. A Lakehouse's SQL analytics endpoint speaks
+the same protocol but is read-only: creating tables and inserting or deleting
+rows is only supported in a Warehouse.
+
+## Setup
+
+### 1. Create a service principal
+
+In the [Microsoft Entra admin center](https://entra.microsoft.com), under
+*App registrations*, register an application (no redirect URI needed), then on
+its *Certificates & secrets* page create a client secret. Note three values:
+
+- the *Directory (tenant) ID*
+- the *Application (client) ID*
+- the secret's *Value* (shown once, not its *Secret ID*)
+
+### 2. Let service principals use Fabric
+
+A Fabric administrator must enable **Service principals can use Fabric APIs**
+in the Fabric admin portal (*Tenant settings*, *Developer settings*), either for
+the whole organisation or for a security group the principal belongs to. The
+same setting governs both the REST API (used by the connection check and the
+workspace picker) and SQL connections to a warehouse.
+
+### 3. Give the principal access to the workspace
+
+In the workspace, open *Manage access* and add the app with the
+**Contributor** role. Contributor, like Admin and Member, grants `CONTROL` on
+every warehouse of the workspace, which covers creating schemas and tables and
+writing rows; `CREATE SCHEMA` in particular requires one of those three roles.
+Viewer only reads, and sharing a single warehouse with no extra permissions
+only lets the principal connect.
+
+### 4. Find the SQL connection string
+
+Open the warehouse's *Settings* and its *SQL connection string* page; the
+string looks like `xxxxxxxx-xxxx.datawarehouse.fabric.microsoft.com`. The string
+belongs to the workspace: every warehouse in it shares the same one, and the
+warehouse's name selects the database. In the UI you do not need to copy it:
+the destination's *Workspace* picker lists the workspaces the principal can see
+that hold a warehouse, and stores that workspace's connection string.
+
+## Usage
+
+```python
+from interloper_azure import AzureConnection, FabricWarehouseDestination
+
+destination = FabricWarehouseDestination(
+    connection=AzureConnection(tenant_id="...", client_id="...", client_secret="..."),
+    server="xxxxxxxx-xxxx.datawarehouse.fabric.microsoft.com",
+    warehouse="Analytics",
+    default_dataset="raw",
+)
+```
+
+The credentials also load from the environment under the standard
+azure-identity names (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`,
+`AZURE_CLIENT_SECRET`), so `AzureConnection()` works with no arguments.
+
+In a deployed instance you configure this through the UI: add a Microsoft
+Azure connection, then a Microsoft Fabric Warehouse destination, pick the
+workspace and type the warehouse name.
+
+## The driver and its system libraries
+
+Statements go through [`mssql-python`](https://github.com/microsoft/mssql-python),
+Microsoft's DB-API driver, which signs in with an Entra access token obtained
+from the connection's credential (scope `https://database.windows.net/.default`).
+It is pip-installable and bundles Microsoft's ODBC driver, but that driver
+loads a few system libraries on Linux. On Debian or Ubuntu:
+
+```sh
+apt-get install -y libltdl7 libkrb5-3 libgssapi-krb5-2
+```
+
+The published scheduler and core images, the ones that run assets, ship these
+libraries. The api and mcp images only describe the destination and never open
+a session, so they leave them out.
+
+Without them the package imports fine (the catalog, the connection check and
+the workspace picker all work), but opening a warehouse session fails with
+`Failed to load the driver`. `mssql-python` also offers an alternate, Rust-based
+native provider that needs none of these libraries: install `mssql-python-rs`
+and set `MSSQL_PYTHON_NATIVE_PROVIDER=mssql-odbc`. Microsoft labels that
+provider alpha.
+
+## Datasets are schemas
+
+An asset's `dataset` is the warehouse schema its table lives in. An asset
+without a dataset falls back to `default_dataset`, then to `dbo`. A missing
+schema is created on the first write, and a missing table is created typed
+from the asset's schema (or one inferred from the data):
+
+| Field type | Fabric Warehouse type |
+|------------|-----------------------|
+| `bool` | `bit` |
+| `int` | `bigint` |
+| `float` | `float` |
+| `Decimal` | `decimal(38,9)` |
+| `datetime` | `datetime2(6)` |
+| `date` | `date` |
+| `bytes` | `varbinary(max)` |
+| `str`, `Any` | `varchar(max)` |
+| nested models, lists, dicts | `varchar(max)` holding JSON |
+
+An existing table is never altered: a column the data carries but the table
+does not is dropped with a warning. Reads return records; nested values come
+back as the JSON text they are stored as. Identifiers are bracketed
+(`[marts].[ads_stats]`), and the warehouse's default collation is
+case-sensitive, so a table is queried with the exact case the asset gives it.
+
+## How Fabric's T-SQL shaped the design
+
+Fabric Warehouse speaks T-SQL over the SQL Server protocol, but it is not SQL
+Server. The differences that matter here:
+
+- **Types.** Tables accept a subset of SQL Server's types: `datetime2` and
+  `time` keep at most six fractional digits, and there is no `datetime`,
+  `datetimeoffset`, `nvarchar`, `nchar`, `tinyint`, `money` or `json`. Hence
+  `varchar` (UTF-8) for text and JSON, and `datetime2(6)` for timestamps;
+  timezone-aware values are written as UTC. `varchar(max)` and
+  `varbinary(max)` exist but hold at most 16 MB per value. Microsoft
+  recommends the shortest fitting `varchar(n)` for query performance;
+  `varchar(max)` is used because the destination cannot know a column's
+  longest value up front.
+- **Schemas.** There is no `CREATE SCHEMA IF NOT EXISTS`, and `CREATE SCHEMA`
+  must be alone in its batch, so the destination checks `sys.schemas` first
+  and creates the schema in a statement of its own. Table existence is read
+  from `sys.tables` and `sys.columns`; the warehouse refuses a query mixing
+  system and user tables, so the catalog check never touches the data.
+- **Transactions.** Explicit `BEGIN TRANSACTION ... COMMIT TRANSACTION` is
+  supported, always under snapshot isolation, and DDL is allowed inside one.
+  DDL in a transaction holds locks on the catalog views until commit, though,
+  blocking every other write's existence checks, so a new schema and table
+  are created and committed before the write's transaction opens; inside it
+  run only the `DELETE` and the `INSERT`s.
+- **Whole-table replaces delete rather than truncate.** `TRUNCATE TABLE`
+  takes a schema-modification lock that blocks readers until commit; a
+  `DELETE` does not.
+- **Write-write conflicts are per table.** Two transactions deleting from the
+  same table conflict even when they touch different rows: the first to commit
+  wins and the other fails with error 24556 or 24706. Writes to different
+  tables never conflict. Concurrent writes of different partitions of one
+  asset (overlapping backfills, for instance) should be avoided, or covered by
+  a retry policy.
+
+## Partitions
+
+A partitioned write replaces the partition's rows: it deletes them, then
+inserts the data, inside one transaction, rolled back on failure. A time
+partition deletes by half-open bounds (`[day] >= ? AND [day] < ?`), so a
+monthly partition whose rows hold daily dates is replaced whole; any other
+partition deletes by equality on its id. A window deletes each partition it
+covers and inserts the whole batch once. An unpartitioned asset deletes every
+row and reloads the table. Readers keep seeing the previous rows until the
+commit.
+
+## The load path and its limits
+
+Rows are written with parameterised multi-row `INSERT ... VALUES` statements.
+Each statement carries as many rows as fit under SQL Server's 2100-parameter
+cap (2099 parameters, one per value) and 1000 rows, whichever is smaller: a
+5-column table loads 419 rows per statement, a 40-column table 52. Every
+statement is a round trip and writes new Parquet files under the table, which
+the warehouse compacts in the background.
+
+This needs no infrastructure beyond the warehouse and suits incremental loads:
+thousands to low hundreds of thousands of rows per write. Microsoft recommends
+against frequent small inserts, and for bulk volumes (millions of rows per
+write, or a full reload of a large table) the documented high-throughput path
+is `COPY INTO` from files staged in OneLake or Azure Data Lake Storage, which
+this destination does not implement.

interloper_azure-0.96.0/pyproject.toml

@@ -0,0 +1,60 @@
+[project]
+name = "interloper-azure"
+version = "0.96.0"
+description = "Interloper Microsoft Azure integration: Entra connection and Microsoft Fabric Warehouse destination"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "interloper-core",
+    "azure-identity>=1.19",
+    "mssql-python>=1.15",
+]
+
+[[project.authors]]
+name = "Guillaume Onfroy"
+email = "guillaume@digitlcloud.com"
+
+[project.entry-points."interloper.components"]
+azure = "interloper_azure"
+
+[build-system]
+requires = ["uv_build>=0.12.9,<0.13"]
+build-backend = "uv_build"
+
+[tool.uv.sources.interloper-core]
+workspace = true
+
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+preview = true
+extend-select = [
+    "E",
+    "I",
+    "UP",
+    "ANN001",
+    "ANN201",
+    "ANN202",
+    "DOC",
+    "D",
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = [
+    "F401",
+    "F403",
+]
+"tests/**" = [
+    "ANN",
+    "F811",
+    "D101",
+    "D102",
+    "D103",
+    "D104",
+    "RUF069",
+    "PLW0108",
+]

interloper_azure-0.96.0/pyproject.toml.orig

@@ -0,0 +1,42 @@
+# ###############
+# PROJECT / UV
+# ###############
+[project]
+name = "interloper-azure"
+version = "0.96.0"
+description = "Interloper Microsoft Azure integration: Entra connection and Microsoft Fabric Warehouse destination"
+readme = "README.md"
+authors = [{ name = "Guillaume Onfroy", email = "guillaume@digitlcloud.com" }]
+requires-python = ">=3.10"
+dependencies = [
+    "interloper-core",
+    "azure-identity>=1.19",
+    "mssql-python>=1.15",
+]
+
+[project.entry-points."interloper.components"]
+azure = "interloper_azure"
+
+[build-system]
+requires = ["uv_build>=0.12.9,<0.13"]
+build-backend = "uv_build"
+
+[tool.uv.sources]
+interloper-core = { workspace = true }
+
+# ###############
+# RUFF
+# ###############
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+preview = true
+extend-select = ["E", "I", "UP", "ANN001", "ANN201", "ANN202", "DOC", "D"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = ["F401", "F403"]
+"tests/**" = ["ANN", "F811", "D101", "D102", "D103", "D104", "RUF069", "PLW0108"]

interloper_azure-0.96.0/src/interloper_azure/__init__.py

@@ -0,0 +1,9 @@
+"""Interloper Microsoft Azure integration: Entra connection and Microsoft Fabric Warehouse destination."""
+
+from interloper_azure.connection import AzureConnection
+from interloper_azure.fabric import FabricWarehouseDestination
+
+__all__ = [
+    "AzureConnection",
+    "FabricWarehouseDestination",
+]

interloper_azure-0.96.0/src/interloper_azure/connection.py

@@ -0,0 +1,161 @@
+"""Microsoft Azure connection resource holding a Microsoft Entra service principal."""
+
+from __future__ import annotations
+
+from collections.abc import Generator
+from functools import cached_property
+from typing import Any
+
+import httpx2
+from azure.identity import ClientSecretCredential
+from interloper.connection import Connection, connection
+from interloper.resource.fields import InputField, SecretField, fetch_field_provider
+from interloper.rest import JSONCursorPaginator, RESTClient
+from pydantic_settings import SettingsConfigDict
+
+FABRIC_API = "https://api.fabric.microsoft.com/v1"
+FABRIC_SCOPE = "https://api.fabric.microsoft.com/.default"
+
+# Every REST call here serves an operator waiting on a form or a check.
+_TIMEOUT = 30.0
+
+
+class _CredentialAuth(httpx2.Auth):
+    """Bearer auth drawing the token from the connection on every request.
+
+    The credential caches the token and renews it before it expires, so a
+    long-lived client never sends a stale one.
+    """
+
+    def __init__(self, owner: AzureConnection, scope: str) -> None:
+        """Bind the auth to the connection whose credential signs the requests.
+
+        Args:
+            owner: The connection holding the credential.
+            scope: The scope the token is requested for.
+        """
+        self._owner = owner
+        self._scope = scope
+
+    def auth_flow(self, request: httpx2.Request) -> Generator[httpx2.Request, httpx2.Response, None]:
+        """Authenticate the request with a bearer token for the scope.
+
+        Args:
+            request: The request to authenticate.
+
+        Yields:
+            The authenticated request.
+        """
+        request.headers["Authorization"] = f"Bearer {self._owner.token(self._scope)}"
+        yield request
+
+
+@connection(
+    key="azure_connection",
+    name="Microsoft Azure",
+    icon="icon:azure",
+    tags=["Cloud"],
+)
+class AzureConnection(Connection):
+    """Connection resource holding a Microsoft Entra service principal.
+
+    The principal is an app registration's client ID and secret in a tenant.
+    ``azure-identity`` only obtains the tokens; every REST call goes through
+    httpx2.
+    """
+
+    model_config = SettingsConfigDict(env_prefix="azure_")
+
+    tenant_id: str = InputField(label="Tenant ID", description="Directory (tenant) ID of the Microsoft Entra tenant")
+    client_id: str = InputField(label="Client ID", description="Application (client) ID of the service principal")
+    client_secret: str = SecretField(
+        label="Client secret",
+        description="Client secret of the service principal",
+        info="From the app registration's Certificates & secrets page: the secret's Value, not its ID.",
+    )
+
+    @cached_property
+    def credential(self) -> ClientSecretCredential:
+        """The credential every token is obtained through.
+
+        Returns:
+            The client-secret credential, cached per connection instance so
+            its token cache is shared by every caller.
+        """
+        return ClientSecretCredential(self.tenant_id, self.client_id, self.client_secret)
+
+    def token(self, scope: str) -> str:
+        """Obtain an access token for a scope.
+
+        Args:
+            scope: The scope requested, e.g. ``https://api.fabric.microsoft.com/.default``.
+
+        Returns:
+            The bearer token.
+        """
+        return self.credential.get_token(scope).token
+
+    @cached_property
+    def client(self) -> RESTClient:
+        """The Fabric REST API client the check and the picker share.
+
+        Returns:
+            The client, authenticated for the Fabric API and cached per connection instance.
+        """
+        return RESTClient(FABRIC_API, auth=_CredentialAuth(self, FABRIC_SCOPE), timeout=_TIMEOUT)
+
+    def _list(self, path: str) -> list[dict[str, Any]]:
+        """List every item of a Fabric collection, following continuation tokens.
+
+        Args:
+            path: The collection path, relative to the API root.
+
+        Returns:
+            The items of every page.
+        """
+        pages = self.client.paginate(
+            path,
+            JSONCursorPaginator(cursor_path="continuationToken", cursor_param="continuationToken"),
+            data_selector="value",
+        )
+        return [item for page in pages for item in page]
+
+    @fetch_field_provider
+    def workspaces(self) -> list[dict[str, str]]:
+        """List the workspaces holding a warehouse, with the SQL endpoint each serves.
+
+        Backs the warehouse destination's ``server`` ``FetchField``. Every
+        warehouse of a workspace is served by the workspace's one SQL
+        connection string, so the first warehouse listed names it, and a
+        workspace without a warehouse has nothing to write to and is left out.
+
+        Returns:
+            Workspace options with ``id``, ``name`` and ``server``, sorted
+            case-insensitively by name.
+        """
+        options: list[dict[str, str]] = []
+        for workspace in self._list("/workspaces"):
+            response = self.client.get(f"/workspaces/{workspace['id']}/warehouses")
+            response.raise_for_status()
+            warehouses = response.json().get("value", [])
+            if not warehouses:
+                continue
+            options.append(
+                {
+                    "id": workspace["id"],
+                    "name": workspace["displayName"],
+                    "server": warehouses[0]["properties"]["connectionString"],
+                }
+            )
+        return sorted(options, key=lambda option: option["name"].lower())
+
+    def check(self) -> bool:
+        """Prove the principal can authenticate and reach Fabric by listing one page of workspaces.
+
+        Returns:
+            True; a rejected secret raises out of the credential, and a principal
+            Fabric refuses (the tenant does not let service principals use
+            Fabric APIs) raises an HTTP 401 or 403.
+        """
+        self.client.get("/workspaces").raise_for_status()
+        return True

interloper_azure-0.96.0/src/interloper_azure/fabric/__init__.py

@@ -0,0 +1,7 @@
+"""Microsoft Fabric destination implementations."""
+
+from interloper_azure.fabric.destination import FabricWarehouseDestination
+
+__all__ = [
+    "FabricWarehouseDestination",
+]

interloper_azure-0.96.0/src/interloper_azure/fabric/destination.py

@@ -0,0 +1,554 @@
+"""Microsoft Fabric Warehouse destination over the warehouse's SQL (TDS) endpoint."""
+
+from __future__ import annotations
+
+import datetime
+import json
+import math
+import threading
+import warnings
+from collections.abc import Iterator, Sequence
+from contextlib import contextmanager
+from dataclasses import dataclass
+from typing import Any
+
+import mssql_python
+from interloper.destination import IOContext, destination
+from interloper.destination.database import DatabaseDestination, PartitionFilter
+from interloper.errors import DataNotFoundError
+from interloper.representation import Representation
+from interloper.resource.fields import FetchField, InputField
+from interloper.schema import FieldSpec
+from interloper.utils.json import json_default, replace_non_finite
+from pydantic import PrivateAttr
+
+from interloper_azure.connection import AzureConnection
+from interloper_azure.fabric.types import column_type
+
+DEFAULT_SCHEMA = "dbo"
+
+# SQL Server caps a request at 2100 parameters; stay one under it, as
+# SQLAlchemy's mssql dialect does.
+MAX_PARAMETERS = 2099
+
+# SQL Server's row cap for an INSERT ... VALUES list; Fabric documents none, so keep it.
+MAX_ROWS = 1000
+
+# Two writes creating the same schema, or partitions of one asset creating its
+# table, would both pass the catalog check and the second CREATE would fail.
+_DDL_LOCK = threading.Lock()
+
+
+@dataclass
+class _Transaction:
+    """One write's transaction: the session it runs on, and whether it has begun.
+
+    Attributes:
+        session: The driver connection every statement of the write runs on.
+        begun: Whether ``BEGIN TRANSACTION`` has been issued on it.
+    """
+
+    session: mssql_python.Connection
+    begun: bool = False
+
+
+@destination(
+    key="fabric_warehouse_destination",
+    name="Microsoft Fabric Warehouse",
+    icon="icon:fabric",
+    tags=["Cloud"],
+)
+class FabricWarehouseDestination(DatabaseDestination):
+    """Microsoft Fabric Warehouse destination.
+
+    A dataset is a schema of the warehouse: the asset's dataset, else
+    ``default_dataset``, else ``dbo``. Tables are created on first write with
+    typed columns and are never altered afterwards. Only a Warehouse accepts
+    writes; a Lakehouse's SQL analytics endpoint is read-only.
+
+    Statements go through Microsoft's ``mssql-python`` driver, signed in as
+    the connection's service principal with a Microsoft Entra access token.
+    """
+
+    connection: AzureConnection
+
+    server: str = FetchField(
+        provider="connection.workspaces",
+        label_key="name",
+        value_key="server",
+        label="Workspace",
+        description="Workspace holding the warehouse",
+        info=(
+            "Stores the workspace's SQL connection string (…datawarehouse.fabric.microsoft.com), the one "
+            "shown in the warehouse's settings; every warehouse of a workspace shares it."
+        ),
+    )
+    warehouse: str = InputField(description="Warehouse name, as shown in the workspace", discriminator=True)
+    default_dataset: str | None = InputField(
+        default=None, description="Default schema for assets without a dataset; dbo when empty"
+    )
+
+    _transactions: dict[int, _Transaction] = PrivateAttr(default_factory=dict)
+
+    # -- Session ---------------------------------------------------------------
+
+    def connect(self) -> mssql_python.Connection:
+        """Open a session on the warehouse, signed in as the connection's service principal.
+
+        Autocommit stays on so a lone statement commits by itself; a write
+        opens an explicit ``BEGIN TRANSACTION`` (see :meth:`transaction`). The
+        driver takes its token from the connection's credential, which caches
+        and renews it, so every new session signs in with a valid one.
+
+        Returns:
+            The new driver connection.
+        """
+        return mssql_python.connect(
+            f"Server={_odbc(self.server)};Database={_odbc(self.warehouse)};Encrypt=yes;TrustServerCertificate=no",
+            autocommit=True,
+            token_provider=self.connection.credential,
+        )
+
+    @contextmanager
+    def _session(self) -> Iterator[mssql_python.Connection]:
+        """Yield the session an operation runs on.
+
+        Inside :meth:`transaction` that is the thread's transaction session,
+        so the delete and the insert commit together; otherwise a fresh
+        session, closed after use.
+
+        Yields:
+            The driver connection.
+        """
+        held = self._transactions.get(threading.get_ident())
+        if held is not None:
+            yield held.session
+            return
+        session = self.connect()
+        try:
+            yield session
+        finally:
+            session.close()
+
+    def _begin(self) -> None:
+        """Begin the calling thread's transaction, if it holds one that has not begun.
+
+        Called right before the first data change rather than on entering
+        :meth:`transaction`, so a table that write creates is created, and
+        committed, before the transaction opens. DDL inside a transaction is
+        allowed by the warehouse but holds locks on its system catalog views
+        until commit, blocking every other write's existence checks.
+        """
+        held = self._transactions.get(threading.get_ident())
+        if held is not None and not held.begun:
+            _run(held.session, "BEGIN TRANSACTION")
+            held.begun = True
+
+    # -- Naming ----------------------------------------------------------------
+
+    def _schema(self, dataset: str | None) -> str:
+        """Return the schema a dataset resolves to.
+
+        Args:
+            dataset: The asset's dataset, or ``None`` to fall back to the destination's default.
+
+        Returns:
+            The schema name.
+        """
+        return dataset or self.default_dataset or DEFAULT_SCHEMA
+
+    @staticmethod
+    def _ref(table: str, schema: str) -> str:
+        """Build the quoted, schema-qualified table reference.
+
+        Args:
+            table: Table name.
+            schema: The resolved schema.
+
+        Returns:
+            ``[schema].[table]``.
+        """
+        return f"{_quote(schema)}.{_quote(table)}"
+
+    @staticmethod
+    def _not_found(table: str, schema: str) -> str:
+        """Word the error a read raises on a table that does not exist.
+
+        Args:
+            table: Table name.
+            schema: The resolved schema.
+
+        Returns:
+            The error message.
+        """
+        return f"Table '{schema}.{table}' does not exist. Has the asset been materialized?"
+
+    # -- Catalog ---------------------------------------------------------------
+
+    @staticmethod
+    def _columns(session: mssql_python.Connection, table: str, schema: str) -> list[str]:
+        """Read a table's column names from the catalog views.
+
+        The query reads catalog views only: the warehouse refuses a query
+        that mixes system and user tables.
+
+        Args:
+            session: The session to query through.
+            table: Table name.
+            schema: The resolved schema.
+
+        Returns:
+            The column names in table order, empty when the table does not exist.
+        """
+        _, rows = _fetch(
+            session,
+            "SELECT c.name FROM sys.columns AS c "
+            "JOIN sys.tables AS t ON c.object_id = t.object_id "
+            "JOIN sys.schemas AS s ON t.schema_id = s.schema_id "
+            "WHERE s.name = ? AND t.name = ? ORDER BY c.column_id",
+            [schema, table],
+        )
+        return [row[0] for row in rows]
+
+    def _create_table(
+        self, session: mssql_python.Connection, table: str, schema: str, specs: Sequence[FieldSpec]
+    ) -> None:
+        """Create the schema and the table, unless they already exist.
+
+        T-SQL has no ``CREATE SCHEMA IF NOT EXISTS``, so each creation is
+        guarded by a catalog check, one creation at a time in the process.
+        ``CREATE SCHEMA`` must be alone in its batch, so it runs as its own
+        statement. Runs before the write's transaction begins, so each
+        statement commits by itself.
+
+        Args:
+            session: The session to run the statements on, outside a transaction.
+            table: Table name.
+            schema: The resolved schema.
+            specs: The table's field specs.
+        """
+        with _DDL_LOCK:
+            if self._columns(session, table, schema):
+                return
+            _, found = _fetch(session, "SELECT 1 FROM sys.schemas WHERE name = ?", [schema])
+            if not found:
+                _run(session, f"CREATE SCHEMA {_quote(schema)}")
+            _run(session, f"CREATE TABLE {self._ref(table, schema)} ({_columns_ddl(specs)})")
+
+    # -- DatabaseDestination hooks ---------------------------------------------
+
+    @contextmanager
+    def transaction(self) -> Iterator[None]:
+        """Run one write, a delete followed by an insert, as ``BEGIN ... COMMIT``.
+
+        The transaction lives on one session held for the calling thread,
+        which the hooks pick up through :meth:`_session`: one instance serves
+        every asset of a run, each write running whole on its own worker
+        thread, so concurrent writes each hold their own. It begins at the
+        first data change (see :meth:`_begin`). The warehouse runs every
+        transaction under snapshot isolation, so readers keep seeing the old
+        rows until the commit.
+
+        Yields:
+            ``None``; the write runs inside the block, committed on success and
+            rolled back on any exception.
+        """
+        ident = threading.get_ident()
+        held = _Transaction(self.connect())
+        self._transactions[ident] = held
+        try:
+            yield
+        except BaseException:
+            if held.begun:
+                _run(held.session, "ROLLBACK TRANSACTION")
+            raise
+        else:
+            if held.begun:
+                _run(held.session, "COMMIT TRANSACTION")
+        finally:
+            del self._transactions[ident]
+            held.session.close()
+
+    def insert(self, table: str, dataset: str | None, data: Any, context: IOContext) -> None:
+        """Insert rows in multi-row ``INSERT ... VALUES`` batches, creating the table on first write.
+
+        A missing table is created from the effective schema (declared on the
+        asset, or inferred during conform), else from a schema inferred from
+        the data here, so the table is always typed. Columns the table does
+        not have are dropped with a warning. Each batch carries as many rows
+        as the parameter and row caps allow (see :func:`batch_size`).
+
+        Args:
+            table: Target table name.
+            dataset: The schema, or ``None`` for the destination's default.
+            data: The data in its native representation.
+            context: IO context carrying the asset and effective schema.
+        """
+        schema = self._schema(dataset)
+        view = Representation.of(data)
+        with self._session() as session:
+            columns = self._columns(session, table, schema)
+            if not columns:
+                specs = (context.schema or view.infer()).field_specs()
+                self._create_table(session, table, schema, specs)
+                columns = self._columns(session, table, schema)
+            keys, rows = _align(view.records, columns, f"{schema}.{table}")
+            if not keys or not rows:
+                return
+            self._begin()
+            names = ", ".join(_quote(key) for key in keys)
+            row = "(" + ", ".join("?" for _ in keys) + ")"
+            size = batch_size(len(keys))
+            for start in range(0, len(rows), size):
+                batch = rows[start : start + size]
+                _run(
+                    session,
+                    f"INSERT INTO {self._ref(table, schema)} ({names}) VALUES {', '.join(row for _ in batch)}",
+                    [value for values in batch for value in values],
+                )
+
+    def delete(self, table: str, dataset: str | None, where: PartitionFilter | None) -> None:
+        """Delete the rows a filter selects, or every row.
+
+        A whole-table replace deletes rather than truncates: ``TRUNCATE``
+        takes a schema-modification lock that blocks readers until the
+        commit, a ``DELETE`` does not. A table that does not exist has nothing
+        to delete.
+
+        Args:
+            table: Target table name.
+            dataset: The schema, or ``None`` for the destination's default.
+            where: The rows to delete; ``None`` for the whole table.
+        """
+        schema = self._schema(dataset)
+        with self._session() as session:
+            if not self._columns(session, table, schema):
+                return
+            self._begin()
+            ref = self._ref(table, schema)
+            if where is None:
+                _run(session, f"DELETE FROM {ref}")
+                return
+            predicate, parameters = _predicate(where)
+            _run(session, f"DELETE FROM {ref} WHERE {predicate}", parameters)
+
+    def select(self, table: str, dataset: str | None, where: PartitionFilter | None) -> list[dict[str, Any]]:
+        """Select the rows a filter selects, or every row, as records.
+
+        Nested and repeated values come back as the JSON text they are stored as.
+
+        Args:
+            table: Target table name.
+            dataset: The schema, or ``None`` for the destination's default.
+            where: The rows to select; ``None`` for the whole table.
+
+        Returns:
+            The selected rows.
+
+        Raises:
+            DataNotFoundError: If the table does not exist yet.
+        """
+        schema = self._schema(dataset)
+        with self._session() as session:
+            if not self._columns(session, table, schema):
+                raise DataNotFoundError(self._not_found(table, schema))
+            ref = self._ref(table, schema)
+            if where is None:
+                names, rows = _fetch(session, f"SELECT * FROM {ref}")
+            else:
+                predicate, parameters = _predicate(where)
+                names, rows = _fetch(session, f"SELECT * FROM {ref} WHERE {predicate}", parameters)
+        return [dict(zip(names, row, strict=True)) for row in rows]
+
+    def count(self, table: str, dataset: str | None, column: str) -> dict[str, int]:
+        """Return row counts grouped by a column's values, cast to text.
+
+        Args:
+            table: Target table name.
+            dataset: The schema, or ``None`` for the destination's default.
+            column: Column to group by.
+
+        Returns:
+            Mapping from the column's value (as a string) to its row count.
+
+        Raises:
+            DataNotFoundError: If the table does not exist.
+        """
+        schema = self._schema(dataset)
+        value = f"CAST({_quote(column)} AS varchar(max))"
+        with self._session() as session:
+            if not self._columns(session, table, schema):
+                raise DataNotFoundError(self._not_found(table, schema))
+            _, rows = _fetch(
+                session,
+                f"SELECT {value} AS partition_value, COUNT(*) AS cnt FROM {self._ref(table, schema)} GROUP BY {value}",
+            )
+        return {row[0]: row[1] for row in rows}
+
+
+# -- Utility functions ---------------------------------------------------------
+
+
+def batch_size(columns: int) -> int:
+    """Return how many rows one ``INSERT ... VALUES`` statement carries.
+
+    Every value is one parameter, so a batch is bounded by
+    :data:`MAX_PARAMETERS` divided by the column count, and by
+    :data:`MAX_ROWS` for narrow tables.
+
+    Args:
+        columns: The number of columns each row binds.
+
+    Returns:
+        The rows per statement, at least one.
+    """
+    return max(1, min(MAX_ROWS, MAX_PARAMETERS // max(columns, 1)))
+
+
+def _quote(identifier: str) -> str:
+    """Quote an identifier in T-SQL brackets, escaping embedded closing brackets.
+
+    Args:
+        identifier: A schema, table or column name.
+
+    Returns:
+        The bracketed identifier.
+    """
+    return "[" + identifier.replace("]", "]]") + "]"
+
+
+def _odbc(value: str) -> str:
+    """Brace a connection string value so separators inside it are taken literally.
+
+    Args:
+        value: A server or database name.
+
+    Returns:
+        The value in braces, embedded closing braces doubled.
+    """
+    return "{" + value.replace("}", "}}") + "}"
+
+
+def _columns_ddl(specs: Sequence[FieldSpec]) -> str:
+    """Render field specs as the column list of a ``CREATE TABLE``.
+
+    Args:
+        specs: The table's field specs.
+
+    Returns:
+        Comma-separated bracketed column definitions. Every column is
+        nullable: conform already enforces the schema's nullability.
+    """
+    return ", ".join(f"{_quote(spec.name)} {column_type(spec)} NULL" for spec in specs)
+
+
+def _run(session: mssql_python.Connection, sql: str, parameters: Sequence[Any] | None = None) -> None:
+    """Run one statement that returns no rows.
+
+    Args:
+        session: The session to run it on.
+        sql: The statement, with ``?`` placeholders for *parameters*.
+        parameters: The statement's parameters; defaults to none.
+    """
+    cursor = session.cursor()
+    try:
+        if parameters is None:
+            cursor.execute(sql)
+        else:
+            cursor.execute(sql, list(parameters))
+    finally:
+        cursor.close()
+
+
+def _fetch(
+    session: mssql_python.Connection, sql: str, parameters: Sequence[Any] | None = None
+) -> tuple[list[str], list[tuple[Any, ...]]]:
+    """Run one query and fetch its rows.
+
+    Args:
+        session: The session to run it on.
+        sql: The query, with ``?`` placeholders for *parameters*.
+        parameters: The query's parameters; defaults to none.
+
+    Returns:
+        The result's column names and its rows, as plain tuples.
+    """
+    cursor = session.cursor()
+    try:
+        if parameters is None:
+            cursor.execute(sql)
+        else:
+            cursor.execute(sql, list(parameters))
+        names = [column[0] for column in cursor.description or []]
+        return names, [tuple(row) for row in cursor.fetchall()]
+    finally:
+        cursor.close()
+
+
+def _predicate(where: PartitionFilter) -> tuple[str, list[Any]]:
+    """Render a partition filter as a parameterised predicate.
+
+    No cast is needed: T-SQL converts a parameter to the column's type when
+    that type ranks higher, so a ``'2024-01-01'`` partition id compares as a
+    ``date``.
+
+    Args:
+        where: The filter to render.
+
+    Returns:
+        The predicate text and the parameters its ``?`` placeholders name.
+    """
+    column = _quote(where.column)
+    if where.bounds is None:
+        return f"{column} = ?", [_bind(where.value)]
+    start, end = where.bounds
+    return f"{column} >= ? AND {column} < ?", [_bind(start), _bind(end)]
+
+
+def _align(records: list[dict[str, Any]], columns: list[str], ref: str) -> tuple[list[str], list[tuple[Any, ...]]]:
+    """Shape records to the table's columns for one batch of inserts.
+
+    Every row binds the same columns, the table columns any record carries,
+    so a column no record mentions is left ``NULL``.
+
+    Args:
+        records: The rows to write.
+        columns: The table's column names, in table order.
+        ref: The table's name, for the warning.
+
+    Returns:
+        The columns written, and each row's values in that order, bound for the driver.
+    """
+    present = dict.fromkeys(key for record in records for key in record)
+    extras = [str(key) for key in present if key not in columns]
+    if extras:
+        warnings.warn(
+            f"Columns {extras} are not in the schema for '{ref}' and will not be written.",
+            UserWarning,
+            stacklevel=3,
+        )
+    keys = [name for name in columns if name in present]
+    return keys, [tuple(_bind(record.get(key)) for key in keys) for record in records]
+
+
+def _bind(value: Any) -> Any:
+    """Turn a conformed value into what the driver binds for its column.
+
+    A nested or repeated value becomes JSON text; a non-finite float becomes
+    ``NULL``, which no column rejects; a timezone-aware datetime becomes naive
+    UTC, since ``datetime2`` holds no offset and the driver would otherwise
+    send a ``datetimeoffset`` that the conversion truncates to local time.
+
+    Args:
+        value: A cell value.
+
+    Returns:
+        The value to bind.
+    """
+    if isinstance(value, float) and not math.isfinite(value):
+        return None
+    if isinstance(value, (dict, list)):
+        return json.dumps(replace_non_finite(value), default=json_default)
+    if isinstance(value, datetime.datetime) and value.tzinfo is not None:
+        return value.astimezone(datetime.timezone.utc).replace(tzinfo=None)
+    return value

interloper_azure-0.96.0/src/interloper_azure/fabric/types.py

@@ -0,0 +1,46 @@
+"""Fabric Warehouse's view of interloper's field types: one table, read in order."""
+
+from __future__ import annotations
+
+import datetime
+from decimal import Decimal
+
+from interloper.schema import FieldSpec
+
+# The warehouse has no json column type and recommends varchar instead.
+JSON = "varchar(max)"
+
+# Ordered: the first base class that matches wins, so bool (a subclass of int)
+# and datetime (a subclass of date) must come before their parents.
+_PYTHON_TO_FABRIC: dict[type, str] = {
+    bool: "bit",
+    int: "bigint",
+    float: "float",
+    Decimal: "decimal(38,9)",
+    datetime.datetime: "datetime2(6)",
+    datetime.date: "date",
+    bytes: "varbinary(max)",
+    str: "varchar(max)",
+    dict: JSON,
+    list: JSON,
+}
+
+
+def column_type(spec: FieldSpec) -> str:
+    """Return the Fabric Warehouse column type for a field spec.
+
+    Args:
+        spec: The field spec, from :meth:`Schema.field_specs` or an inferred schema.
+
+    Returns:
+        :data:`JSON` for a nested or repeated field, the type the spec's Python
+        type maps to otherwise, and ``varchar(max)`` for anything unmapped
+        (``typing.Any``).
+    """
+    if spec.fields is not None or spec.repeated:
+        return JSON
+    if isinstance(spec.type, type):
+        for base, name in _PYTHON_TO_FABRIC.items():
+            if issubclass(spec.type, base):
+                return name
+    return "varchar(max)"