muscles-data-s3 0.1.0__tar.gz

muscles_data_s3-0.1.0/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: muscles-data-s3
+Version: 0.1.0
+Summary: S3-compatible ObjectStorePort adapter for muscles-data
+Author-email: "Denis B." <denis@butko.info>
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: muscles-data<1.0.0,>=0.1.0
+Requires-Dist: boto3<2.0,>=1.34
+Provides-Extra: dev
+Requires-Dist: pytest<9.0,>=8.0; extra == "dev"
+
+# muscles-data-s3
+
+S3-compatible adapter package for `muscles-data`.
+
+This package is intentionally separate from `muscles-data`: the core package
+owns typed ports and runtime, while this package owns the boto3-backed
+`ObjectStorePort` adapter.
+
+## Related packages
+
+- Core runtime and port contracts:
+  [`muscles-data`](https://github.com/butkoden/muscles-data)
+- Elasticsearch search adapter:
+  [`muscles-data-elasticsearch`](https://github.com/butkoden/muscles-data-elasticsearch)
+- OpenSearch search adapter:
+  [`muscles-data-opensearch`](https://github.com/butkoden/muscles-data-opensearch)
+- Redis key-value/lock/stream adapter:
+  [`muscles-data-redis`](https://github.com/butkoden/muscles-data-redis)
+- Qdrant vector adapter:
+  [`muscles-data-qdrant`](https://github.com/butkoden/muscles-data-qdrant)
+- MongoDB document-store adapter:
+  [`muscles-data-mongodb`](https://github.com/butkoden/muscles-data-mongodb)
+- SQLAlchemy direct SQL resource adapter:
+  [`muscles-data-sqlalchemy`](https://github.com/butkoden/muscles-data-sqlalchemy)
+- Executable example:
+  [`example_data_s3_1`](https://github.com/butkoden/muscular-example/tree/master/example_data_s3_1)
+
+## Install
+
+```bash
+python -m pip install muscles-data-s3
+```
+
+For local framework development:
+
+```bash
+PYTHONPATH=../muscles-data/src:src python3 -m pytest -q
+```
+
+## Configuration
+
+```yaml
+data:
+  resources:
+    objects.docs:
+      type: s3
+      endpoint_url: ${S3_ENDPOINT}
+      bucket: documents
+      region_name: us-east-1
+      prefix: raw
+      max_keys: 100
+```
+
+The adapter also supports standard boto3 credential options:
+
+- `aws_access_key_id`;
+- `aws_secret_access_key`;
+- `aws_session_token`;
+- `profile_name`;
+- the default boto3 environment/provider chain.
+
+## Usage
+
+Register the external factory in the project composition root:
+
+```python
+from muscles_data.catalog import DataAdapterCatalog
+from muscles_data.config import DataConfig
+from muscles_data.ports import ObjectStorePort
+from muscles_data.runtime import DataRuntime
+from muscles_data_s3 import S3ObjectStoreFactory
+
+catalog = DataAdapterCatalog.with_defaults()
+catalog.register(S3ObjectStoreFactory())
+
+runtime = DataRuntime(config=DataConfig.from_raw(config), catalog=catalog)
+objects = runtime.require_port("objects.docs", ObjectStorePort)
+```
+
+Then use the narrow port:
+
+```python
+objects.put_object("docs/readme.txt", b"hello", content_type="text/plain")
+blob = objects.get_object("docs/readme.txt")
+items = objects.list_objects(prefix="docs", limit=20)
+objects.delete_object("docs/readme.txt")
+```
+
+Backend-specific request options are available without exposing the boto3
+client: `get_object()` accepts `range`, conditional headers and
+version/checksum options; `list_objects()` accepts `delimiter` and continuation
+helpers; and `delete_object()` accepts version and retention options.
+
+If `prefix: raw` is configured, the public key `docs/readme.txt` is stored as
+`raw/docs/readme.txt` in S3 and returned to port consumers without the configured
+prefix. This keeps application code independent from bucket layout details.
+
+## Capabilities
+
+`S3ObjectStoreFactory` provides:
+
+- `object_store`;
+- `healthcheck`;
+- `native_client` only when the resource declares `native_client: true`.
+
+Native access is an advanced project escape hatch:
+
+```python
+from muscles_data import DataCapability
+
+client = runtime.require_resource("objects.docs", DataCapability.NATIVE_CLIENT).native_client()
+```
+
+Use the native client for backend-specific operations such as presigned URLs,
+multipart upload configuration, bucket policies or lifecycle rules. Keep normal
+blob reads and writes on `ObjectStorePort`.
+
+## Boundaries
+
+This package owns:
+
+- lazy boto3 client creation;
+- S3-compatible endpoint and bucket binding;
+- object put/get/list/delete;
+- key normalization and optional prefix mapping;
+- safe `inspect()` and `doctor()`.
+
+It does not own:
+
+- document parsing;
+- application storage schemas;
+- lifecycle policies;
+- multipart/streaming abstractions in the MVP;
+- cross-resource transactions.
+
+`data.doctor` performs `head_bucket`. `data.resources.list` and package
+initialization do not open an S3 connection.

muscles_data_s3-0.1.0/README.md

@@ -0,0 +1,137 @@
+# muscles-data-s3
+
+S3-compatible adapter package for `muscles-data`.
+
+This package is intentionally separate from `muscles-data`: the core package
+owns typed ports and runtime, while this package owns the boto3-backed
+`ObjectStorePort` adapter.
+
+## Related packages
+
+- Core runtime and port contracts:
+  [`muscles-data`](https://github.com/butkoden/muscles-data)
+- Elasticsearch search adapter:
+  [`muscles-data-elasticsearch`](https://github.com/butkoden/muscles-data-elasticsearch)
+- OpenSearch search adapter:
+  [`muscles-data-opensearch`](https://github.com/butkoden/muscles-data-opensearch)
+- Redis key-value/lock/stream adapter:
+  [`muscles-data-redis`](https://github.com/butkoden/muscles-data-redis)
+- Qdrant vector adapter:
+  [`muscles-data-qdrant`](https://github.com/butkoden/muscles-data-qdrant)
+- MongoDB document-store adapter:
+  [`muscles-data-mongodb`](https://github.com/butkoden/muscles-data-mongodb)
+- SQLAlchemy direct SQL resource adapter:
+  [`muscles-data-sqlalchemy`](https://github.com/butkoden/muscles-data-sqlalchemy)
+- Executable example:
+  [`example_data_s3_1`](https://github.com/butkoden/muscular-example/tree/master/example_data_s3_1)
+
+## Install
+
+```bash
+python -m pip install muscles-data-s3
+```
+
+For local framework development:
+
+```bash
+PYTHONPATH=../muscles-data/src:src python3 -m pytest -q
+```
+
+## Configuration
+
+```yaml
+data:
+  resources:
+    objects.docs:
+      type: s3
+      endpoint_url: ${S3_ENDPOINT}
+      bucket: documents
+      region_name: us-east-1
+      prefix: raw
+      max_keys: 100
+```
+
+The adapter also supports standard boto3 credential options:
+
+- `aws_access_key_id`;
+- `aws_secret_access_key`;
+- `aws_session_token`;
+- `profile_name`;
+- the default boto3 environment/provider chain.
+
+## Usage
+
+Register the external factory in the project composition root:
+
+```python
+from muscles_data.catalog import DataAdapterCatalog
+from muscles_data.config import DataConfig
+from muscles_data.ports import ObjectStorePort
+from muscles_data.runtime import DataRuntime
+from muscles_data_s3 import S3ObjectStoreFactory
+
+catalog = DataAdapterCatalog.with_defaults()
+catalog.register(S3ObjectStoreFactory())
+
+runtime = DataRuntime(config=DataConfig.from_raw(config), catalog=catalog)
+objects = runtime.require_port("objects.docs", ObjectStorePort)
+```
+
+Then use the narrow port:
+
+```python
+objects.put_object("docs/readme.txt", b"hello", content_type="text/plain")
+blob = objects.get_object("docs/readme.txt")
+items = objects.list_objects(prefix="docs", limit=20)
+objects.delete_object("docs/readme.txt")
+```
+
+Backend-specific request options are available without exposing the boto3
+client: `get_object()` accepts `range`, conditional headers and
+version/checksum options; `list_objects()` accepts `delimiter` and continuation
+helpers; and `delete_object()` accepts version and retention options.
+
+If `prefix: raw` is configured, the public key `docs/readme.txt` is stored as
+`raw/docs/readme.txt` in S3 and returned to port consumers without the configured
+prefix. This keeps application code independent from bucket layout details.
+
+## Capabilities
+
+`S3ObjectStoreFactory` provides:
+
+- `object_store`;
+- `healthcheck`;
+- `native_client` only when the resource declares `native_client: true`.
+
+Native access is an advanced project escape hatch:
+
+```python
+from muscles_data import DataCapability
+
+client = runtime.require_resource("objects.docs", DataCapability.NATIVE_CLIENT).native_client()
+```
+
+Use the native client for backend-specific operations such as presigned URLs,
+multipart upload configuration, bucket policies or lifecycle rules. Keep normal
+blob reads and writes on `ObjectStorePort`.
+
+## Boundaries
+
+This package owns:
+
+- lazy boto3 client creation;
+- S3-compatible endpoint and bucket binding;
+- object put/get/list/delete;
+- key normalization and optional prefix mapping;
+- safe `inspect()` and `doctor()`.
+
+It does not own:
+
+- document parsing;
+- application storage schemas;
+- lifecycle policies;
+- multipart/streaming abstractions in the MVP;
+- cross-resource transactions.
+
+`data.doctor` performs `head_bucket`. `data.resources.list` and package
+initialization do not open an S3 connection.

muscles_data_s3-0.1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "muscles-data-s3"
+version = "0.1.0"
+description = "S3-compatible ObjectStorePort adapter for muscles-data"
+readme = "README.md"
+requires-python = ">=3.10"
+authors = [{ name = "Denis B.", email = "denis@butko.info" }]
+dependencies = [
+  "muscles-data>=0.1.0,<1.0.0",
+  "boto3>=1.34,<2.0",
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=8.0,<9.0",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+markers = [
+  "integration: tests that require the local backend integration stack",
+]
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]

muscles_data_s3-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

muscles_data_s3-0.1.0/src/muscles_data_s3/__init__.py

@@ -0,0 +1,20 @@
+from __future__ import annotations
+
+from .adapter import (
+    S3AdapterError,
+    S3ClientMissingError,
+    S3ConfigError,
+    S3ConnectionError,
+    S3ObjectStoreAdapter,
+    S3ObjectStoreFactory,
+)
+
+
+__all__ = [
+    "S3AdapterError",
+    "S3ClientMissingError",
+    "S3ConfigError",
+    "S3ConnectionError",
+    "S3ObjectStoreAdapter",
+    "S3ObjectStoreFactory",
+]

muscles_data_s3-0.1.0/src/muscles_data_s3/adapter.py

@@ -0,0 +1,424 @@
+from __future__ import annotations
+
+import importlib
+import threading
+from collections.abc import Callable
+from dataclasses import asdict
+from pathlib import PurePosixPath
+from typing import Any, Mapping
+from urllib.parse import urlsplit
+
+from muscles_data.config import DataResourceConfig
+from muscles_data.errors import DataError
+from muscles_data.models import DataCapability, HealthResult, InspectResult, ObjectBlob, ObjectInfo, WriteResult
+
+
+_CLIENT_UNSET = object()
+_ALLOWED_OPTIONS = {
+    "endpoint_url",
+    "bucket",
+    "region_name",
+    "aws_access_key_id",
+    "aws_secret_access_key",
+    "aws_session_token",
+    "profile_name",
+    "prefix",
+    "max_keys",
+    "native_client",
+    "timeout",
+    "connect_timeout",
+    "read_timeout",
+    "addressing_style",
+    "verify",
+    "use_ssl",
+}
+_SECRET_MARKERS = ("password", "passwd", "secret", "token", "api_key", "apikey", "credential", "access_key")
+
+
+class S3AdapterError(DataError):
+    """Base error for S3 object-store adapter failures."""
+
+
+class S3ConfigError(ValueError, S3AdapterError):
+    """Raised when an S3 resource config cannot be mapped safely."""
+
+
+class S3ClientMissingError(S3AdapterError):
+    """Raised when S3 adapter is used without an available client."""
+
+
+class S3ConnectionError(S3AdapterError):
+    """Raised when an S3 operation cannot reach or use the backend."""
+
+
+class S3ObjectStoreAdapter:
+    resource_type = "s3"
+
+    def __init__(
+        self,
+        config: DataResourceConfig,
+        *,
+        client_factory: Callable[[DataResourceConfig], Any] | None = None,
+    ) -> None:
+        self.config = config
+        self._client_factory = client_factory
+        self._client: Any = _CLIENT_UNSET
+        self._lock = threading.RLock()
+        self.closed = False
+
+    def put_object(
+        self,
+        key: str,
+        content: bytes,
+        content_type: str | None = None,
+        metadata: Mapping[str, Any] | None = None,
+        options: Mapping[str, Any] | None = None,
+    ) -> WriteResult:
+        options = dict(options or {})
+        kwargs: dict[str, Any] = {
+            "Bucket": self.bucket_name(),
+            "Key": self._storage_key(key),
+            "Body": bytes(content),
+        }
+        if content_type:
+            kwargs["ContentType"] = content_type
+        normalized_metadata = _normalize_metadata(metadata)
+        if normalized_metadata:
+            kwargs["Metadata"] = normalized_metadata
+        for option_key, target_key in {
+            "cache_control": "CacheControl",
+            "content_disposition": "ContentDisposition",
+            "content_encoding": "ContentEncoding",
+        }.items():
+            if option_key in options:
+                kwargs[target_key] = options[option_key]
+
+        try:
+            self._client_instance().put_object(**kwargs)
+        except (S3ClientMissingError, S3ConfigError):
+            raise
+        except Exception as exc:
+            raise S3ConnectionError(self._safe_error(exc)) from exc
+        return WriteResult(written=1, matched=1)
+
+    def get_object(self, key: str, options: Mapping[str, Any] | None = None) -> ObjectBlob:
+        normalized = _normalize_object_key(key)
+        request: dict[str, Any] = {
+            "Bucket": self.bucket_name(),
+            "Key": self._storage_key(normalized),
+        }
+        for option_key, request_key in {
+            "version_id": "VersionId",
+            "range": "Range",
+            "if_match": "IfMatch",
+            "if_none_match": "IfNoneMatch",
+            "checksum_mode": "ChecksumMode",
+        }.items():
+            if option_key in (options or {}):
+                request[request_key] = (options or {})[option_key]
+        try:
+            response = self._client_instance().get_object(**request)
+            body = response.get("Body")
+            content = body.read() if hasattr(body, "read") else body
+        except (S3ClientMissingError, S3ConfigError):
+            raise
+        except Exception as exc:
+            raise S3ConnectionError(self._safe_error(exc)) from exc
+        return ObjectBlob(
+            key=normalized,
+            content=bytes(content or b""),
+            content_type=response.get("ContentType"),
+            metadata=dict(response.get("Metadata") or {}),
+        )
+
+    def list_objects(
+        self,
+        prefix: str | None = None,
+        limit: int = 100,
+        cursor: str | None = None,
+        options: Mapping[str, Any] | None = None,
+    ) -> list[ObjectInfo]:
+        bounded_limit = self._bounded_limit(limit)
+        kwargs: dict[str, Any] = {
+            "Bucket": self.bucket_name(),
+            "MaxKeys": bounded_limit,
+        }
+        storage_prefix = self._storage_prefix(prefix)
+        if storage_prefix:
+            kwargs["Prefix"] = storage_prefix
+        if cursor:
+            kwargs["ContinuationToken"] = str(cursor)
+        for option_key, request_key in {
+            "delimiter": "Delimiter",
+            "start_after": "StartAfter",
+            "fetch_owner": "FetchOwner",
+            "request_payer": "RequestPayer",
+        }.items():
+            if option_key in (options or {}):
+                kwargs[request_key] = (options or {})[option_key]
+        try:
+            response = self._client_instance().list_objects_v2(**kwargs)
+        except (S3ClientMissingError, S3ConfigError):
+            raise
+        except Exception as exc:
+            raise S3ConnectionError(self._safe_error(exc)) from exc
+
+        output: list[ObjectInfo] = []
+        for item in response.get("Contents", []) or []:
+            key = self._public_key(str(item.get("Key", "")))
+            output.append(
+                ObjectInfo(
+                    key=key,
+                    size=int(item.get("Size", 0) or 0),
+                    metadata=_object_info_metadata(item),
+                )
+            )
+        return output
+
+    def delete_object(self, key: str, options: Mapping[str, Any] | None = None) -> WriteResult:
+        request: dict[str, Any] = {
+            "Bucket": self.bucket_name(),
+            "Key": self._storage_key(key),
+        }
+        for option_key, request_key in {
+            "version_id": "VersionId",
+            "mfa": "MFA",
+            "bypass_governance_retention": "BypassGovernanceRetention",
+            "request_payer": "RequestPayer",
+        }.items():
+            if option_key in (options or {}):
+                request[request_key] = (options or {})[option_key]
+        try:
+            self._client_instance().delete_object(**request)
+        except (S3ClientMissingError, S3ConfigError):
+            raise
+        except Exception as exc:
+            raise S3ConnectionError(self._safe_error(exc)) from exc
+        return WriteResult(deleted=1, matched=1)
+
+    def inspect(self) -> dict[str, Any]:
+        return asdict(
+            InspectResult(
+                name=self.config.name,
+                type=self.config.type,
+                capabilities=[],
+                initialized=self._client is not _CLIENT_UNSET,
+                status="ok",
+                options=self.config.safe_options(),
+                details={
+                    "backend": "s3",
+                    "bucket": self.bucket_name(),
+                    "prefix": self.prefix(),
+                    "max_keys": self.max_keys(),
+                },
+            )
+        )
+
+    def health(self) -> HealthResult:
+        try:
+            bucket = self.bucket_name()
+            self._client_instance().head_bucket(Bucket=bucket)
+        except Exception as exc:
+            return HealthResult(status="failed", message=self._safe_error(exc), details=self._safe_details())
+        return HealthResult(status="ok", message="S3 bucket is available", details=self._safe_details())
+
+    def close(self) -> None:
+        if self._client is _CLIENT_UNSET:
+            self.closed = True
+            return
+        close = getattr(self._client, "close", None)
+        if callable(close):
+            close()
+        self.closed = True
+
+    def native_client(self):
+        return self._client_instance()
+
+    def bucket_name(self) -> str:
+        bucket = str(self.config.options.get("bucket", "")).strip()
+        if not bucket:
+            raise S3ConfigError("S3 resource requires bucket")
+        return bucket
+
+    def prefix(self) -> str | None:
+        return _normalize_object_prefix(self.config.options.get("prefix"))
+
+    def max_keys(self) -> int:
+        value = int(self.config.options.get("max_keys", 1000))
+        if value <= 0:
+            raise S3ConfigError("S3 max_keys must be positive")
+        return value
+
+    def _bounded_limit(self, limit: int) -> int:
+        return min(max(0, int(limit)), self.max_keys())
+
+    def _storage_key(self, key: str) -> str:
+        normalized = _normalize_object_key(key)
+        prefix = self.prefix()
+        return f"{prefix}{normalized}" if prefix else normalized
+
+    def _storage_prefix(self, prefix: str | None) -> str | None:
+        configured = self.prefix() or ""
+        requested = _normalize_object_prefix(prefix)
+        if requested is None:
+            return configured or None
+        return f"{configured}{requested}"
+
+    def _public_key(self, storage_key: str) -> str:
+        prefix = self.prefix()
+        if prefix and storage_key.startswith(prefix):
+            return storage_key[len(prefix) :]
+        return storage_key
+
+    def _client_instance(self):
+        if self._client is _CLIENT_UNSET:
+            with self._lock:
+                if self._client is _CLIENT_UNSET:
+                    self._validate_options()
+                    client = self._client_factory(self.config) if self._client_factory else _default_s3_client(self.config)
+                    if client is None:
+                        raise S3ClientMissingError("S3 client is not available")
+                    self._client = client
+        return self._client
+
+    def _validate_options(self) -> None:
+        unknown = sorted(set(self.config.options) - _ALLOWED_OPTIONS)
+        if unknown:
+            names = ", ".join(unknown)
+            raise S3ConfigError(f"Unsupported S3 resource options: {names}")
+        self.bucket_name()
+        self.prefix()
+        self.max_keys()
+
+    def _safe_details(self) -> dict[str, Any]:
+        details: dict[str, Any] = {"backend": "s3"}
+        try:
+            details["bucket"] = self.bucket_name()
+        except S3ConfigError:
+            pass
+        try:
+            details["prefix"] = self.prefix()
+        except S3ConfigError:
+            pass
+        return details
+
+    def _safe_error(self, exc: Exception) -> str:
+        message = str(exc)
+        sensitive_values = set()
+        for key, value in self.config.options.items():
+            if any(marker in str(key).lower() for marker in _SECRET_MARKERS) and value:
+                sensitive_values.add(str(value))
+        endpoint_url = str(self.config.options.get("endpoint_url", ""))
+        if endpoint_url:
+            sensitive_values.add(endpoint_url)
+        try:
+            parsed = urlsplit(endpoint_url)
+            if parsed.username:
+                sensitive_values.add(parsed.username)
+            if parsed.password:
+                sensitive_values.add(parsed.password)
+        except Exception:  # pragma: no cover
+            pass
+        for value in sorted((item for item in sensitive_values if item), key=len, reverse=True):
+            message = message.replace(value, "***")
+        return message
+
+
+class S3ObjectStoreFactory:
+    resource_type = "s3"
+
+    def __init__(self, *, client_factory: Callable[[DataResourceConfig], Any] | None = None) -> None:
+        self._client_factory = client_factory
+
+    def capabilities(self, config: DataResourceConfig) -> set[DataCapability]:
+        native = {DataCapability.NATIVE_CLIENT} if bool(config.options.get("native_client")) else set()
+        return {DataCapability.OBJECT_STORE, DataCapability.HEALTHCHECK} | native
+
+    def create(self, config: DataResourceConfig) -> S3ObjectStoreAdapter:
+        return S3ObjectStoreAdapter(config, client_factory=self._client_factory)
+
+
+def _default_s3_client(config: DataResourceConfig):
+    try:
+        boto3 = importlib.import_module("boto3")
+    except ImportError as exc:
+        raise S3ClientMissingError("boto3 package is not installed") from exc
+
+    kwargs: dict[str, Any] = {}
+    for option_key in (
+        "endpoint_url",
+        "region_name",
+        "aws_access_key_id",
+        "aws_secret_access_key",
+        "aws_session_token",
+        "verify",
+        "use_ssl",
+    ):
+        if option_key in config.options:
+            kwargs[option_key] = config.options[option_key]
+
+    client_config = _botocore_config(config)
+    if client_config is not None:
+        kwargs["config"] = client_config
+
+    profile_name = config.options.get("profile_name")
+    if profile_name:
+        session_kwargs: dict[str, Any] = {"profile_name": profile_name}
+        if "region_name" in kwargs:
+            session_kwargs["region_name"] = kwargs.pop("region_name")
+        session = boto3.session.Session(**session_kwargs)
+        return session.client("s3", **kwargs)
+    return boto3.client("s3", **kwargs)
+
+
+def _botocore_config(config: DataResourceConfig):
+    config_kwargs: dict[str, Any] = {}
+    if "timeout" in config.options:
+        timeout = float(config.options["timeout"])
+        config_kwargs["connect_timeout"] = timeout
+        config_kwargs["read_timeout"] = timeout
+    if "connect_timeout" in config.options:
+        config_kwargs["connect_timeout"] = float(config.options["connect_timeout"])
+    if "read_timeout" in config.options:
+        config_kwargs["read_timeout"] = float(config.options["read_timeout"])
+    if "addressing_style" in config.options:
+        config_kwargs["s3"] = {"addressing_style": str(config.options["addressing_style"])}
+    if not config_kwargs:
+        return None
+    try:
+        config_module = importlib.import_module("botocore.config")
+    except ImportError as exc:
+        raise S3ClientMissingError("botocore package is not installed") from exc
+    return config_module.Config(**config_kwargs)
+
+
+def _normalize_object_key(key: str) -> str:
+    path = PurePosixPath(str(key))
+    if path.is_absolute() or any(part in {"", ".", ".."} for part in path.parts):
+        raise S3ConfigError(f"unsafe object key: {key}")
+    return path.as_posix()
+
+
+def _normalize_object_prefix(prefix: Any) -> str | None:
+    if prefix is None:
+        return None
+    cleaned = str(prefix).strip("/")
+    if not cleaned:
+        return None
+    return _normalize_object_key(cleaned) + "/"
+
+
+def _normalize_metadata(metadata: Mapping[str, Any] | None) -> dict[str, str]:
+    return {str(key): str(value) for key, value in dict(metadata or {}).items()}
+
+
+def _object_info_metadata(item: Mapping[str, Any]) -> dict[str, Any]:
+    metadata: dict[str, Any] = {}
+    for source_key, target_key in {
+        "ETag": "etag",
+        "LastModified": "last_modified",
+        "StorageClass": "storage_class",
+    }.items():
+        if source_key in item:
+            metadata[target_key] = item[source_key]
+    return metadata

muscles_data_s3-0.1.0/src/muscles_data_s3.egg-info/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: muscles-data-s3
+Version: 0.1.0
+Summary: S3-compatible ObjectStorePort adapter for muscles-data
+Author-email: "Denis B." <denis@butko.info>
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: muscles-data<1.0.0,>=0.1.0
+Requires-Dist: boto3<2.0,>=1.34
+Provides-Extra: dev
+Requires-Dist: pytest<9.0,>=8.0; extra == "dev"
+
+# muscles-data-s3
+
+S3-compatible adapter package for `muscles-data`.
+
+This package is intentionally separate from `muscles-data`: the core package
+owns typed ports and runtime, while this package owns the boto3-backed
+`ObjectStorePort` adapter.
+
+## Related packages
+
+- Core runtime and port contracts:
+  [`muscles-data`](https://github.com/butkoden/muscles-data)
+- Elasticsearch search adapter:
+  [`muscles-data-elasticsearch`](https://github.com/butkoden/muscles-data-elasticsearch)
+- OpenSearch search adapter:
+  [`muscles-data-opensearch`](https://github.com/butkoden/muscles-data-opensearch)
+- Redis key-value/lock/stream adapter:
+  [`muscles-data-redis`](https://github.com/butkoden/muscles-data-redis)
+- Qdrant vector adapter:
+  [`muscles-data-qdrant`](https://github.com/butkoden/muscles-data-qdrant)
+- MongoDB document-store adapter:
+  [`muscles-data-mongodb`](https://github.com/butkoden/muscles-data-mongodb)
+- SQLAlchemy direct SQL resource adapter:
+  [`muscles-data-sqlalchemy`](https://github.com/butkoden/muscles-data-sqlalchemy)
+- Executable example:
+  [`example_data_s3_1`](https://github.com/butkoden/muscular-example/tree/master/example_data_s3_1)
+
+## Install
+
+```bash
+python -m pip install muscles-data-s3
+```
+
+For local framework development:
+
+```bash
+PYTHONPATH=../muscles-data/src:src python3 -m pytest -q
+```
+
+## Configuration
+
+```yaml
+data:
+  resources:
+    objects.docs:
+      type: s3
+      endpoint_url: ${S3_ENDPOINT}
+      bucket: documents
+      region_name: us-east-1
+      prefix: raw
+      max_keys: 100
+```
+
+The adapter also supports standard boto3 credential options:
+
+- `aws_access_key_id`;
+- `aws_secret_access_key`;
+- `aws_session_token`;
+- `profile_name`;
+- the default boto3 environment/provider chain.
+
+## Usage
+
+Register the external factory in the project composition root:
+
+```python
+from muscles_data.catalog import DataAdapterCatalog
+from muscles_data.config import DataConfig
+from muscles_data.ports import ObjectStorePort
+from muscles_data.runtime import DataRuntime
+from muscles_data_s3 import S3ObjectStoreFactory
+
+catalog = DataAdapterCatalog.with_defaults()
+catalog.register(S3ObjectStoreFactory())
+
+runtime = DataRuntime(config=DataConfig.from_raw(config), catalog=catalog)
+objects = runtime.require_port("objects.docs", ObjectStorePort)
+```
+
+Then use the narrow port:
+
+```python
+objects.put_object("docs/readme.txt", b"hello", content_type="text/plain")
+blob = objects.get_object("docs/readme.txt")
+items = objects.list_objects(prefix="docs", limit=20)
+objects.delete_object("docs/readme.txt")
+```
+
+Backend-specific request options are available without exposing the boto3
+client: `get_object()` accepts `range`, conditional headers and
+version/checksum options; `list_objects()` accepts `delimiter` and continuation
+helpers; and `delete_object()` accepts version and retention options.
+
+If `prefix: raw` is configured, the public key `docs/readme.txt` is stored as
+`raw/docs/readme.txt` in S3 and returned to port consumers without the configured
+prefix. This keeps application code independent from bucket layout details.
+
+## Capabilities
+
+`S3ObjectStoreFactory` provides:
+
+- `object_store`;
+- `healthcheck`;
+- `native_client` only when the resource declares `native_client: true`.
+
+Native access is an advanced project escape hatch:
+
+```python
+from muscles_data import DataCapability
+
+client = runtime.require_resource("objects.docs", DataCapability.NATIVE_CLIENT).native_client()
+```
+
+Use the native client for backend-specific operations such as presigned URLs,
+multipart upload configuration, bucket policies or lifecycle rules. Keep normal
+blob reads and writes on `ObjectStorePort`.
+
+## Boundaries
+
+This package owns:
+
+- lazy boto3 client creation;
+- S3-compatible endpoint and bucket binding;
+- object put/get/list/delete;
+- key normalization and optional prefix mapping;
+- safe `inspect()` and `doctor()`.
+
+It does not own:
+
+- document parsing;
+- application storage schemas;
+- lifecycle policies;
+- multipart/streaming abstractions in the MVP;
+- cross-resource transactions.
+
+`data.doctor` performs `head_bucket`. `data.resources.list` and package
+initialization do not open an S3 connection.

muscles_data_s3-0.1.0/src/muscles_data_s3.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+README.md
+pyproject.toml
+src/muscles_data_s3/__init__.py
+src/muscles_data_s3/adapter.py
+src/muscles_data_s3.egg-info/PKG-INFO
+src/muscles_data_s3.egg-info/SOURCES.txt
+src/muscles_data_s3.egg-info/dependency_links.txt
+src/muscles_data_s3.egg-info/requires.txt
+src/muscles_data_s3.egg-info/top_level.txt
+tests/test_integration.py
+tests/test_s3_adapter.py

muscles_data_s3-0.1.0/src/muscles_data_s3.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

muscles_data_s3-0.1.0/src/muscles_data_s3.egg-info/requires.txt

@@ -0,0 +1,5 @@
+muscles-data<1.0.0,>=0.1.0
+boto3<2.0,>=1.34
+
+[dev]
+pytest<9.0,>=8.0

muscles_data_s3-0.1.0/src/muscles_data_s3.egg-info/top_level.txt

@@ -0,0 +1 @@
+muscles_data_s3

muscles_data_s3-0.1.0/tests/test_integration.py

@@ -0,0 +1,64 @@
+from __future__ import annotations
+
+import os
+from uuid import uuid4
+
+import pytest
+from muscles_data.catalog import DataAdapterCatalog
+from muscles_data.config import DataConfig
+from muscles_data.models import DataCapability
+from muscles_data.ports import ObjectStorePort
+from muscles_data.runtime import DataRuntime
+
+from muscles_data_s3 import S3ObjectStoreFactory
+
+
+pytestmark = [
+    pytest.mark.integration,
+    pytest.mark.skipif(not os.getenv("MUSCLES_DATA_INTEGRATION"), reason="backend integration is disabled"),
+]
+
+
+def test_s3_real_object_lifecycle_against_minio():
+    bucket = f"muscles-data-it-{uuid4().hex[:12]}"
+    config = DataConfig.from_raw(
+        {
+            "data": {
+                "resources": {
+                    "objects.s3": {
+                        "type": "s3",
+                        "endpoint_url": os.environ["S3_ENDPOINT_URL"],
+                        "bucket": bucket,
+                        "region_name": "us-east-1",
+                        "aws_access_key_id": "minioadmin",
+                        "aws_secret_access_key": "minioadmin",
+                        "addressing_style": "path",
+                        "native_client": True,
+                    }
+                }
+            }
+        }
+    )
+    catalog = DataAdapterCatalog.with_defaults()
+    catalog.register(S3ObjectStoreFactory())
+    runtime = DataRuntime(config=config, catalog=catalog)
+
+    client = None
+    try:
+        client = runtime.require_resource("objects.s3", DataCapability.NATIVE_CLIENT).native_client()
+        client.create_bucket(Bucket=bucket)
+        store = runtime.require_port("objects.s3", ObjectStorePort)
+        contracts = pytest.importorskip("muscles_data.contracts")
+        contract = getattr(contracts, "assert_object_store_contract", None)
+        if contract is not None:
+            contract(lambda: store)
+        assert runtime.doctor()["status"] == "ok"
+    finally:
+        try:
+            if client is not None:
+                objects = client.list_objects_v2(Bucket=bucket).get("Contents", [])
+                for item in objects:
+                    client.delete_object(Bucket=bucket, Key=item["Key"])
+                client.delete_bucket(Bucket=bucket)
+        finally:
+            runtime.close()

muscles_data_s3-0.1.0/tests/test_s3_adapter.py

@@ -0,0 +1,227 @@
+from __future__ import annotations
+
+from types import SimpleNamespace
+from typing import Any
+
+import pytest
+from muscles_data.catalog import DataAdapterCatalog
+from muscles_data.config import DataConfig
+from muscles_data.models import DataCapability
+from muscles_data.ports import ObjectStorePort
+from muscles_data.runtime import DataRuntime
+
+from muscles_data_s3 import (
+    S3ClientMissingError,
+    S3ConfigError,
+    S3ConnectionError,
+    S3ObjectStoreFactory,
+)
+
+
+class FakeBody:
+    def __init__(self, content: bytes) -> None:
+        self.content = content
+
+    def read(self) -> bytes:
+        return self.content
+
+
+class FakeS3Client:
+    def __init__(self, *, fail_head: bool = False) -> None:
+        self.fail_head = fail_head
+        self.objects: dict[tuple[str, str], dict[str, Any]] = {}
+        self.put_calls: list[dict[str, Any]] = []
+        self.get_calls: list[dict[str, Any]] = []
+        self.list_calls: list[dict[str, Any]] = []
+        self.delete_calls: list[dict[str, Any]] = []
+        self.head_bucket_calls: list[dict[str, Any]] = []
+
+    def put_object(self, **kwargs):
+        self.put_calls.append(dict(kwargs))
+        self.objects[(kwargs["Bucket"], kwargs["Key"])] = {
+            "Body": bytes(kwargs["Body"]),
+            "ContentType": kwargs.get("ContentType"),
+            "Metadata": dict(kwargs.get("Metadata") or {}),
+        }
+        return {"ETag": '"etag"'}
+
+    def get_object(self, **kwargs):
+        self.get_calls.append(dict(kwargs))
+        item = self.objects[(kwargs["Bucket"], kwargs["Key"])]
+        return {
+            "Body": FakeBody(item["Body"]),
+            "ContentLength": len(item["Body"]),
+            "ContentType": item.get("ContentType"),
+            "Metadata": dict(item.get("Metadata") or {}),
+        }
+
+    def list_objects_v2(self, **kwargs):
+        self.list_calls.append(dict(kwargs))
+        bucket = kwargs["Bucket"]
+        prefix = kwargs.get("Prefix", "")
+        max_keys = int(kwargs.get("MaxKeys", 100))
+        contents = []
+        for (stored_bucket, key), item in sorted(self.objects.items()):
+            if stored_bucket != bucket or not key.startswith(prefix):
+                continue
+            contents.append({"Key": key, "Size": len(item["Body"]), "ETag": '"etag"'})
+        return {"Contents": contents[:max_keys], "IsTruncated": len(contents) > max_keys}
+
+    def delete_object(self, **kwargs):
+        self.delete_calls.append(dict(kwargs))
+        self.objects.pop((kwargs["Bucket"], kwargs["Key"]), None)
+        return {"DeleteMarker": True}
+
+    def head_bucket(self, **kwargs):
+        self.head_bucket_calls.append(dict(kwargs))
+        if self.fail_head:
+            raise TimeoutError("https://user:s3-secret@s3.example timed out")
+        return {"ResponseMetadata": {"HTTPStatusCode": 200}}
+
+
+def _s3_config(endpoint_url: str = "https://user:s3-secret@s3.example") -> dict[str, Any]:
+    return {
+        "data": {
+            "resources": {
+                "objects.docs": {
+                    "type": "s3",
+                    "endpoint_url": endpoint_url,
+                    "bucket": "documents",
+                    "region_name": "us-east-1",
+                    "aws_access_key_id": "access-secret",
+                    "aws_secret_access_key": "secret-key",
+                    "prefix": "raw",
+                    "max_keys": 2,
+                    "native_client": True,
+                }
+            }
+        }
+    }
+
+
+def _catalog(factory: S3ObjectStoreFactory) -> DataAdapterCatalog:
+    catalog = DataAdapterCatalog.with_defaults()
+    catalog.register(factory)
+    return catalog
+
+
+def _runtime(client: FakeS3Client | None):
+    return DataRuntime(
+        config=DataConfig.from_raw(_s3_config()),
+        catalog=_catalog(S3ObjectStoreFactory(client_factory=lambda _config: client)),
+    )
+
+
+def test_s3_object_store_is_registered_lazy_and_maps_operations():
+    client = FakeS3Client()
+    runtime = _runtime(client)
+
+    listed = runtime.list_resources()
+    resource = next(item for item in listed if item["name"] == "objects.docs")
+    inspected_before = runtime.inspect_resource("objects.docs")
+
+    assert resource["type"] == "s3"
+    assert "object_store" in resource["capabilities"]
+    assert "native_client" not in resource["capabilities"]
+    assert resource["initialized"] is False
+    assert inspected_before["initialized"] is False
+    assert inspected_before["options"]["endpoint_url"] == "***"
+    assert inspected_before["options"]["aws_secret_access_key"] == "***"
+
+    store = runtime.require_port("objects.docs", ObjectStorePort)
+    write = store.put_object(
+        "docs/readme.txt",
+        b"hello",
+        content_type="text/plain",
+        metadata={"owner": "denis", "version": 1},
+    )
+    blob = store.get_object("docs/readme.txt")
+    ranged = store.get_object("docs/readme.txt", options={"range": "bytes=0-2", "checksum_mode": "ENABLED"})
+    store.put_object("docs/guide.txt", b"guide")
+    store.put_object("images/logo.png", b"png")
+    listed_objects = store.list_objects(prefix="docs", limit=10)
+    deleted = store.delete_object("docs/guide.txt", options={"version_id": "v1"})
+
+    assert write.written == 1
+    assert client.put_calls[0]["Bucket"] == "documents"
+    assert client.put_calls[0]["Key"] == "raw/docs/readme.txt"
+    assert client.put_calls[0]["Body"] == b"hello"
+    assert client.put_calls[0]["ContentType"] == "text/plain"
+    assert client.put_calls[0]["Metadata"] == {"owner": "denis", "version": "1"}
+    assert blob.key == "docs/readme.txt"
+    assert blob.content == b"hello"
+    assert blob.content_type == "text/plain"
+    assert blob.metadata == {"owner": "denis", "version": "1"}
+    assert ranged.content == b"hello"
+    assert client.get_calls[-1]["Range"] == "bytes=0-2"
+    assert client.get_calls[-1]["ChecksumMode"] == "ENABLED"
+    assert [item.key for item in listed_objects] == ["docs/guide.txt", "docs/readme.txt"]
+    assert client.list_calls[-1]["Prefix"] == "raw/docs/"
+    assert client.list_calls[-1]["MaxKeys"] == 2
+    assert deleted.deleted == 1
+    assert client.delete_calls[-1] == {"Bucket": "documents", "Key": "raw/docs/guide.txt", "VersionId": "v1"}
+
+    native = runtime.require_resource("objects.docs", DataCapability.NATIVE_CLIENT).native_client()
+    assert native is client
+    assert client.head_bucket_calls == []
+
+
+def test_s3_inspect_doctor_and_safe_failures():
+    client = FakeS3Client()
+    runtime = _runtime(client)
+
+    doctor = runtime.doctor()
+    checks = [check for check in doctor["checks"] if check["resource"] == "objects.docs"]
+
+    assert checks[0]["status"] == "ok"
+    assert client.head_bucket_calls == [{"Bucket": "documents"}]
+    assert "s3-secret" not in repr(doctor)
+    assert "secret-key" not in repr(runtime.inspect_resource("objects.docs"))
+
+    missing_client_runtime = _runtime(None)
+    with pytest.raises(S3ClientMissingError):
+        missing_client_runtime.require_port("objects.docs", ObjectStorePort).get_object("docs/readme.txt")
+
+    failing_runtime = DataRuntime(
+        config=DataConfig.from_raw(_s3_config()),
+        catalog=_catalog(S3ObjectStoreFactory(client_factory=lambda _config: FakeS3Client(fail_head=True))),
+    )
+    failing_doctor = failing_runtime.doctor()
+    assert failing_doctor["status"] == "failed"
+    assert [check for check in failing_doctor["checks"] if check["resource"] == "objects.docs"][0]["status"] == "failed"
+    assert "s3-secret" not in repr(failing_doctor)
+
+    bad_client = FakeS3Client()
+    bad_client.get_object = lambda **_kwargs: (_ for _ in ()).throw(RuntimeError("secret-key leaked"))  # type: ignore[method-assign]
+    bad_runtime = DataRuntime(
+        config=DataConfig.from_raw(_s3_config()),
+        catalog=_catalog(S3ObjectStoreFactory(client_factory=lambda _config: bad_client)),
+    )
+    with pytest.raises(S3ConnectionError) as exc_info:
+        bad_runtime.require_port("objects.docs", ObjectStorePort).get_object("docs/readme.txt")
+    assert "secret-key" not in str(exc_info.value)
+
+
+def test_s3_resource_requires_bucket_and_rejects_unsafe_keys():
+    assert DataConfig.from_raw(_s3_config()).resources["objects.docs"].type == "s3"
+
+    adapter = S3ObjectStoreFactory(client_factory=lambda _config: FakeS3Client()).create(
+        DataConfig.from_raw(
+            {
+                "data": {
+                    "resources": {
+                        "objects.docs": {
+                            "type": "s3",
+                            "endpoint_url": "https://s3.example",
+                        }
+                    }
+                }
+            }
+        ).resources["objects.docs"]
+    )
+    with pytest.raises(S3ConfigError, match="requires bucket"):
+        adapter.put_object("docs/readme.txt", b"hello")
+
+    unsafe = _runtime(FakeS3Client()).require_port("objects.docs", ObjectStorePort)
+    with pytest.raises(S3ConfigError, match="unsafe object key"):
+        unsafe.put_object("../secret.txt", b"nope")