interloper-aws 0.94.0__tar.gz

interloper_aws-0.94.0/PKG-INFO

@@ -0,0 +1,77 @@
+Metadata-Version: 2.3
+Name: interloper-aws
+Version: 0.94.0
+Summary: Interloper AWS integration: Amazon S3 destination and connection
+Author: Guillaume Onfroy
+Author-email: Guillaume Onfroy <guillaume@digitlcloud.com>
+Requires-Dist: interloper-core
+Requires-Dist: boto3>=1.34
+Requires-Dist: pyarrow>=14
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# interloper-aws
+
+AWS integration for interloper: an `S3Destination` that stores asset data in
+an Amazon S3 bucket, and the `AWSConnection` that holds the credentials.
+
+`S3Destination` is an `ObjectStoreDestination`, the same base as the Cloud
+Storage destination in `interloper-google-cloud`, so the two behave
+identically: one object per partition in a hive-partitioned layout, Parquet
+(the default), JSONL or CSV, the partition column in the path only, and each
+object's row count stored as `row_count` user metadata.
+
+```
+s3://{bucket}/{prefix}/{dataset}/{table}/data.{ext}
+s3://{bucket}/{prefix}/{dataset}/{table}/{column}={partition}/data.{ext}
+```
+
+## Usage
+
+```python
+import interloper as il
+from interloper_aws import AWSConnection, S3Destination
+
+destination = S3Destination(
+    connection=AWSConnection(region="eu-central-1"),
+    bucket="acme-lake",
+    prefix="raw",
+    format="parquet",
+)
+
+source = Shop(destinations=destination)
+```
+
+## Credentials
+
+The connection takes an access key pair (`access_key_id`,
+`secret_access_key`, plus `session_token` for temporary credentials) and a
+`region`, defaulting to `eu-central-1`. Each field also loads from the
+standard environment variables: `AWS_ACCESS_KEY_ID`,
+`AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` and `AWS_REGION`.
+
+Leave the key pair empty and boto3's default credential chain applies:
+environment, shared config files, then the instance or pod role (an EC2
+instance profile, EKS IRSA or Pod Identity). In a cluster, that means no
+stored secret at all: grant the workload's role access to the bucket.
+
+The connection's check calls STS `GetCallerIdentity`, which needs no IAM
+permission, so a failing check means the credentials themselves are wrong.
+The bucket picker calls `ListBuckets`.
+
+## Permissions
+
+The destination needs, on the bucket and its objects:
+
+- `s3:PutObject` to write
+- `s3:GetObject` to read, and to count objects written without `row_count`
+  metadata
+- `s3:ListBucket` for partition row counts
+- `s3:ListAllMyBuckets` for the bucket picker only
+
+## Notes
+
+`ListObjectsV2` returns no user metadata, so partition row counts issue one
+`HeadObject` per listed object. That is still no download: the count comes
+from the `row_count` metadata stamped at write time. Objects written by other
+tools, without that metadata, are downloaded and counted.

interloper_aws-0.94.0/README.md

@@ -0,0 +1,65 @@
+# interloper-aws
+
+AWS integration for interloper: an `S3Destination` that stores asset data in
+an Amazon S3 bucket, and the `AWSConnection` that holds the credentials.
+
+`S3Destination` is an `ObjectStoreDestination`, the same base as the Cloud
+Storage destination in `interloper-google-cloud`, so the two behave
+identically: one object per partition in a hive-partitioned layout, Parquet
+(the default), JSONL or CSV, the partition column in the path only, and each
+object's row count stored as `row_count` user metadata.
+
+```
+s3://{bucket}/{prefix}/{dataset}/{table}/data.{ext}
+s3://{bucket}/{prefix}/{dataset}/{table}/{column}={partition}/data.{ext}
+```
+
+## Usage
+
+```python
+import interloper as il
+from interloper_aws import AWSConnection, S3Destination
+
+destination = S3Destination(
+    connection=AWSConnection(region="eu-central-1"),
+    bucket="acme-lake",
+    prefix="raw",
+    format="parquet",
+)
+
+source = Shop(destinations=destination)
+```
+
+## Credentials
+
+The connection takes an access key pair (`access_key_id`,
+`secret_access_key`, plus `session_token` for temporary credentials) and a
+`region`, defaulting to `eu-central-1`. Each field also loads from the
+standard environment variables: `AWS_ACCESS_KEY_ID`,
+`AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` and `AWS_REGION`.
+
+Leave the key pair empty and boto3's default credential chain applies:
+environment, shared config files, then the instance or pod role (an EC2
+instance profile, EKS IRSA or Pod Identity). In a cluster, that means no
+stored secret at all: grant the workload's role access to the bucket.
+
+The connection's check calls STS `GetCallerIdentity`, which needs no IAM
+permission, so a failing check means the credentials themselves are wrong.
+The bucket picker calls `ListBuckets`.
+
+## Permissions
+
+The destination needs, on the bucket and its objects:
+
+- `s3:PutObject` to write
+- `s3:GetObject` to read, and to count objects written without `row_count`
+  metadata
+- `s3:ListBucket` for partition row counts
+- `s3:ListAllMyBuckets` for the bucket picker only
+
+## Notes
+
+`ListObjectsV2` returns no user metadata, so partition row counts issue one
+`HeadObject` per listed object. That is still no download: the count comes
+from the `row_count` metadata stamped at write time. Objects written by other
+tools, without that metadata, are downloaded and counted.

interloper_aws-0.94.0/pyproject.toml

@@ -0,0 +1,60 @@
+[project]
+name = "interloper-aws"
+version = "0.94.0"
+description = "Interloper AWS integration: Amazon S3 destination and connection"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "interloper-core",
+    "boto3>=1.34",
+    "pyarrow>=14",
+]
+
+[[project.authors]]
+name = "Guillaume Onfroy"
+email = "guillaume@digitlcloud.com"
+
+[project.entry-points."interloper.components"]
+aws = "interloper_aws"
+
+[build-system]
+requires = ["uv_build>=0.12.9,<0.13"]
+build-backend = "uv_build"
+
+[tool.uv.sources.interloper-core]
+workspace = true
+
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+preview = true
+extend-select = [
+    "E",
+    "I",
+    "UP",
+    "ANN001",
+    "ANN201",
+    "ANN202",
+    "DOC",
+    "D",
+]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = [
+    "F401",
+    "F403",
+]
+"tests/**" = [
+    "ANN",
+    "F811",
+    "D101",
+    "D102",
+    "D103",
+    "D104",
+    "RUF069",
+    "PLW0108",
+]

interloper_aws-0.94.0/pyproject.toml.orig

@@ -0,0 +1,42 @@
+# ###############
+# PROJECT / UV
+# ###############
+[project]
+name = "interloper-aws"
+version = "0.94.0"
+description = "Interloper AWS integration: Amazon S3 destination and connection"
+readme = "README.md"
+authors = [{ name = "Guillaume Onfroy", email = "guillaume@digitlcloud.com" }]
+requires-python = ">=3.10"
+dependencies = [
+    "interloper-core",
+    "boto3>=1.34",
+    "pyarrow>=14",
+]
+
+[project.entry-points."interloper.components"]
+aws = "interloper_aws"
+
+[build-system]
+requires = ["uv_build>=0.12.9,<0.13"]
+build-backend = "uv_build"
+
+[tool.uv.sources]
+interloper-core = { workspace = true }
+
+# ###############
+# RUFF
+# ###############
+[tool.ruff]
+line-length = 120
+
+[tool.ruff.lint]
+preview = true
+extend-select = ["E", "I", "UP", "ANN001", "ANN201", "ANN202", "DOC", "D"]
+
+[tool.ruff.lint.pydocstyle]
+convention = "google"
+
+[tool.ruff.lint.per-file-ignores]
+"__init__.py" = ["F401", "F403"]
+"tests/**" = ["ANN", "F811", "D101", "D102", "D103", "D104", "RUF069", "PLW0108"]

interloper_aws-0.94.0/src/interloper_aws/__init__.py

@@ -0,0 +1,9 @@
+"""Interloper AWS integration: Amazon S3 destination and connection."""
+
+from interloper_aws.connection import AWSConnection
+from interloper_aws.s3 import S3Destination
+
+__all__ = [
+    "AWSConnection",
+    "S3Destination",
+]

interloper_aws-0.94.0/src/interloper_aws/connection.py

@@ -0,0 +1,102 @@
+"""AWS connection resource holding IAM credentials."""
+
+from __future__ import annotations
+
+from functools import cached_property
+from typing import Any
+
+import boto3
+from interloper.connection import Connection, connection
+from interloper.resource.fields import InputField, SecretField, fetch_field_provider
+from pydantic_settings import SettingsConfigDict
+
+
+@connection(
+    key="aws_connection",
+    name="AWS",
+    icon="icon:aws",
+    tags=["Cloud"],
+)
+class AWSConnection(Connection):
+    """Connection resource holding AWS credentials.
+
+    With an access key pair, every client signs with it (plus the session
+    token, for temporary credentials). Without one, boto3's default
+    credential chain applies: environment, shared config, then the instance
+    or pod role (EC2 instance profile, EKS IRSA or Pod Identity), so an
+    in-cluster deployment needs no stored secret at all. Every field also
+    loads from the standard environment variables (``AWS_ACCESS_KEY_ID``,
+    ``AWS_SECRET_ACCESS_KEY``, ``AWS_SESSION_TOKEN``, ``AWS_REGION``).
+    """
+
+    model_config = SettingsConfigDict(env_prefix="aws_")
+
+    access_key_id: str | None = InputField(
+        default=None,
+        label="Access key ID",
+        description="IAM access key ID; leave empty to use the ambient role",
+    )
+    secret_access_key: str | None = SecretField(
+        default=None,
+        label="Secret access key",
+        description="IAM secret access key",
+    )
+    session_token: str | None = SecretField(
+        default=None,
+        label="Session token",
+        description="Session token, for temporary credentials only",
+    )
+    region: str = InputField(default="eu-central-1", description="AWS region clients are built for")
+
+    @cached_property
+    def session(self) -> boto3.session.Session:
+        """The boto3 session every client is built from.
+
+        Unset credentials are passed as ``None``, which leaves boto3 to
+        resolve them through its default chain.
+
+        Returns:
+            The session, cached per connection instance.
+        """
+        return boto3.session.Session(
+            aws_access_key_id=self.access_key_id,
+            aws_secret_access_key=self.secret_access_key,
+            aws_session_token=self.session_token,
+            region_name=self.region,
+        )
+
+    def client(self, service: str) -> Any:
+        """Build a client for one AWS service from the session.
+
+        Args:
+            service: The boto3 service name, such as ``"s3"`` or ``"sts"``.
+
+        Returns:
+            The service client.
+        """
+        return self.session.client(service)
+
+    @fetch_field_provider
+    def buckets(self) -> list[dict[str, str]]:
+        """List the S3 buckets the credentials can see.
+
+        Backs the S3 destination's ``bucket`` ``FetchField``.
+
+        Returns:
+            Bucket options with ``name``, sorted case-insensitively.
+        """
+        response = self.client("s3").list_buckets()
+        results = [{"name": bucket["Name"]} for bucket in response.get("Buckets", [])]
+        return sorted(results, key=lambda b: b["name"].lower())
+
+    def check(self) -> bool:
+        """Prove the credentials work via STS ``GetCallerIdentity``.
+
+        The call needs no IAM permission, so it isolates a bad credential
+        from a missing grant.
+
+        Returns:
+            True; invalid credentials raise out of the call instead.
+        """
+        self.client("sts").get_caller_identity()
+        return True

interloper_aws-0.94.0/src/interloper_aws/s3/__init__.py

@@ -0,0 +1,7 @@
+"""Amazon S3 destination implementations."""
+
+from interloper_aws.s3.destination import S3Destination
+
+__all__ = [
+    "S3Destination",
+]

interloper_aws-0.94.0/src/interloper_aws/s3/destination.py

@@ -0,0 +1,118 @@
+"""Amazon S3 destination implementation."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from functools import cached_property
+from typing import Any
+
+from botocore.exceptions import ClientError
+from interloper.destination import ObjectStoreDestination, StoredObject, destination
+from interloper.resource.fields import FetchField
+
+from interloper_aws.connection import AWSConnection
+
+
+@destination(
+    key="s3_destination",
+    name="Amazon S3",
+    icon="icon:s3",
+    tags=["Cloud"],
+)
+class S3Destination(ObjectStoreDestination):
+    """Amazon S3 destination.
+
+    An :class:`~interloper.destination.ObjectStoreDestination` over S3: the
+    base owns the hive layout, the formats and the row-count metadata, so
+    objects land at::
+
+        s3://{bucket}/{prefix}/{dataset}/{table}/data.{ext}
+        s3://{bucket}/{prefix}/{dataset}/{table}/{column}={partition}/data.{ext}
+    """
+
+    connection: AWSConnection
+
+    bucket: str = FetchField(
+        provider="connection.buckets",
+        label_key="name",
+        value_key="name",
+        description="S3 bucket",
+        discriminator=True,
+    )
+
+    @cached_property
+    def client(self) -> Any:
+        """The S3 client every read and write goes through.
+
+        Returns:
+            The client, cached per destination instance.
+        """
+        return self.connection.client("s3")
+
+    # -- Object hooks ----------------------------------------------------------
+
+    def put_object(self, name: str, payload: bytes, content_type: str, metadata: dict[str, str]) -> None:
+        """Upload an object, overwriting any object of the same key.
+
+        Args:
+            name: The object key.
+            payload: The object's bytes.
+            content_type: The payload's media type.
+            metadata: User metadata to store with the object.
+        """
+        self.client.put_object(
+            Bucket=self.bucket,
+            Key=name,
+            Body=payload,
+            ContentType=content_type,
+            Metadata=metadata,
+        )
+
+    def get_object(self, name: str) -> bytes | None:
+        """Download an object.
+
+        Args:
+            name: The object key.
+
+        Returns:
+            The object's bytes, or ``None`` when no object has that key.
+
+        Raises:
+            ClientError: For any S3 error other than a missing key.
+        """
+        try:
+            response = self.client.get_object(Bucket=self.bucket, Key=name)
+        except ClientError as error:
+            if error.response.get("Error", {}).get("Code") == "NoSuchKey":
+                return None
+            raise
+        return response["Body"].read()
+
+    def list_objects(self, prefix: str) -> Iterator[StoredObject]:
+        """List the objects under a key prefix, with their user metadata.
+
+        ``ListObjectsV2`` carries no user metadata, so each listed key is
+        followed by a ``HeadObject``.
+
+        Args:
+            prefix: The key prefix, ending with ``/``.
+
+        Yields:
+            One stored object per key.
+        """
+        paginator = self.client.get_paginator("list_objects_v2")
+        for page in paginator.paginate(Bucket=self.bucket, Prefix=prefix):
+            for listed in page.get("Contents", []):
+                head = self.client.head_object(Bucket=self.bucket, Key=listed["Key"])
+                yield StoredObject(name=listed["Key"], metadata=dict(head.get("Metadata", {})))
+
+    def object_uri(self, name: str) -> str:
+        """Return the ``s3://`` URI of an object.
+
+        Args:
+            name: The object key.
+
+        Returns:
+            ``s3://{bucket}/{name}``.
+        """
+        return f"s3://{self.bucket}/{name}"