10x-cloud-storage-manager 0.2.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,248 @@
1
+ Metadata-Version: 2.4
2
+ Name: 10x-cloud-storage-manager
3
+ Version: 0.2.0
4
+ Summary: A unified async interface for AWS S3, Google Cloud Storage and Azure Blob Storage
5
+ Author-email: MUTHARASU A <mutharasu39@gmail.com>
6
+ License: MIT License
7
+
8
+ Copyright (c) 2024 MUTHARASU a
9
+
10
+ Permission is hereby granted, free of charge, to any person obtaining a copy
11
+ of this software and associated documentation files (the “Software”), to deal
12
+ in the Software without restriction, including without limitation the rights
13
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
+ copies of the Software, and to permit persons to whom the Software is furnished
15
+ to do so, subject to the following conditions:
16
+
17
+ The above copyright notice and this permission notice shall be included in all
18
+ copies or substantial portions of the Software.
19
+
20
+ THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
26
+ THE SOFTWARE.
27
+
28
+ Project-URL: Homepage, https://github.com/10xHub/cloud-storage-manager
29
+ Project-URL: Bug Tracker, https://github.com/10xHub/cloud-storage-manager/issues
30
+ Keywords: aws,s3,gcp,azure,blob,cloud storage,file upload,async
31
+ Classifier: Development Status :: 3 - Alpha
32
+ Classifier: Intended Audience :: Developers
33
+ Classifier: License :: OSI Approved :: MIT License
34
+ Classifier: Programming Language :: Python :: 3
35
+ Classifier: Programming Language :: Python :: 3.10
36
+ Classifier: Programming Language :: Python :: 3.11
37
+ Classifier: Programming Language :: Python :: 3.12
38
+ Classifier: Programming Language :: Python :: 3.13
39
+ Classifier: Framework :: AsyncIO
40
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
41
+ Requires-Python: >=3.10
42
+ Description-Content-Type: text/markdown
43
+ License-File: LICENSE
44
+ Requires-Dist: aioboto3>=11.0.0
45
+ Requires-Dist: google-cloud-storage>=2.0.0
46
+ Requires-Dist: pydantic>=2.0.0
47
+ Provides-Extra: aws
48
+ Requires-Dist: aioboto3>=11.0.0; extra == "aws"
49
+ Provides-Extra: gcp
50
+ Requires-Dist: google-cloud-storage>=2.0.0; extra == "gcp"
51
+ Provides-Extra: azure
52
+ Requires-Dist: azure-storage-blob>=12.19.0; extra == "azure"
53
+ Requires-Dist: aiohttp>=3.8.0; extra == "azure"
54
+ Provides-Extra: all
55
+ Requires-Dist: 10x-cloud-storage-manager[aws,azure,gcp]; extra == "all"
56
+ Provides-Extra: dev
57
+ Requires-Dist: 10x-cloud-storage-manager[all]; extra == "dev"
58
+ Requires-Dist: pytest>=7.0.0; extra == "dev"
59
+ Requires-Dist: pytest-asyncio>=0.23.0; extra == "dev"
60
+ Requires-Dist: packaging>=23.0; extra == "dev"
61
+ Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
62
+ Requires-Dist: ruff>=0.5.0; extra == "dev"
63
+ Dynamic: license-file
64
+
65
+ # Cloud Storage Manager
66
+
67
+ One async interface over AWS S3, Google Cloud Storage and Azure Blob Storage. Application code
68
+ calls the same methods everywhere; switching cloud is a configuration change.
69
+
70
+ ## Features
71
+
72
+ - One interface for AWS S3, Google Cloud Storage and Azure Blob Storage
73
+ - Async operations
74
+ - Upload from a file or from bytes, download, list, delete, signed read URLs
75
+ - `upload_bytes` stores a content type, so a signed video URL plays in a browser
76
+ - Type-safe configuration using Pydantic
77
+ - A drop-in upgrade from 0.1.x (see below)
78
+
79
+ ## Installation
80
+
81
+ The base install covers AWS S3 and Google Cloud Storage, as 0.1.x did. Azure is an extra:
82
+
83
+ ```bash
84
+ pip install 10x-cloud-storage-manager # AWS S3 + Google Cloud Storage
85
+ pip install "10x-cloud-storage-manager[azure]" # adds Azure Blob Storage
86
+ pip install "10x-cloud-storage-manager[all]" # all three
87
+ ```
88
+
89
+ The import name is still `cloud_storage_manager`. From 0.2.0 the package is published as
90
+ `10x-cloud-storage-manager`; `cloud-storage-manager` on PyPI stops at 0.1.2.
91
+
92
+ Requires Python 3.10+.
93
+
94
+ ## Quick start
95
+
96
+ ```python
97
+ import asyncio
98
+ from cloud_storage_manager import CloudStorageFactory, StorageConfig
99
+
100
+
101
+ async def main(config: StorageConfig):
102
+ storage = CloudStorageFactory.from_config(config)
103
+
104
+ await storage.upload("local/clip.mp4", "recordings/clip.mp4")
105
+ await storage.upload_bytes(b"# Notes", "notes/a.md", content_type="text/markdown")
106
+
107
+ data = await storage.download_bytes("notes/a.md")
108
+ names = await storage.list_objects("recordings/")
109
+ sizes = await storage.list_object_sizes("recordings/") # {name: bytes}
110
+
111
+ url = await storage.get_public_url("recordings/clip.mp4", expiration=3600)
112
+
113
+ deleted = await storage.delete_if_exists("notes/a.md") # False if it was already gone
114
+ ```
115
+
116
+ `CloudStorageFactory.from_config` picks the provider from the config.
117
+ `CloudStorageFactory.get_storage(StorageProvider.GCP, config)` still works.
118
+
119
+ ## Configuration
120
+
121
+ `CloudStorageFactory.get_storage(provider, config)` uses the provider you name, so a config may
122
+ hold more than one. `config.provider` and `from_config` infer it, and raise `ValueError` unless
123
+ exactly one is set.
124
+
125
+ ### AWS S3
126
+
127
+ ```python
128
+ from cloud_storage_manager import AwsConfig, StorageConfig
129
+
130
+ config = StorageConfig(
131
+ aws=AwsConfig(
132
+ bucket_name="my-bucket",
133
+ access_key_id="...",
134
+ secret_access_key="...",
135
+ region="us-east-1", # stored but not used; boto resolves the region, as in 0.1.x
136
+ )
137
+ )
138
+ ```
139
+
140
+ ### Google Cloud Storage
141
+
142
+ ```python
143
+ from cloud_storage_manager import GcpConfig, StorageConfig
144
+
145
+ config = StorageConfig(
146
+ gcp=GcpConfig(
147
+ bucket_name="my-bucket",
148
+ project_id="my-project",
149
+ credentials={...}, # service account JSON as a dict
150
+ )
151
+ )
152
+ ```
153
+
154
+ ### Azure Blob Storage
155
+
156
+ Give exactly one of `account_key` or `connection_string`.
157
+
158
+ ```python
159
+ from cloud_storage_manager import AzureConfig, StorageConfig
160
+
161
+ config = StorageConfig(
162
+ azure=AzureConfig(
163
+ account_name="mystorageaccount",
164
+ container_name="my-container",
165
+ account_key="...",
166
+ # or: connection_string="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;"
167
+ )
168
+ )
169
+ ```
170
+
171
+ `get_public_url` on Azure returns a read-only blob SAS URL, signed with the account key. A
172
+ connection string that carries a SAS instead of an `AccountKey` can read and write but cannot
173
+ sign URLs.
174
+
175
+ ## Interface
176
+
177
+ Every provider implements `BaseCloudStorage`:
178
+
179
+ | Method | Returns | Notes |
180
+ | --- | --- | --- |
181
+ | `upload(file_path, cloud_path)` | `str` | GCP and Azure type the object from its file name; S3 does not (0.1.x behaviour) |
182
+ | `upload_bytes(data, cloud_path, content_type=None)` | `str` | Content type guessed from `cloud_path` when omitted |
183
+ | `download_bytes(cloud_path)` | `bytes` | `NotFoundError` if missing |
184
+ | `list_objects(prefix)` | `list[str]` | Every page |
185
+ | `list_object_sizes(prefix)` | `dict[str, int]` | Name to size in bytes |
186
+ | `delete(cloud_path)` | `bool` | 0.1.x behaviour: a missing object raises `DeleteError` on GCP and Azure, returns `True` on S3 |
187
+ | `delete_if_exists(cloud_path)` | `bool` | `False` if it did not exist, on every cloud |
188
+ | `get_public_url(cloud_path, expiration=3600)` | `str` | Signed, read-only, expires after `expiration` seconds |
189
+
190
+ ## Error handling
191
+
192
+ Every error is a `CloudStorageError`: `UploadError`, `DownloadError`, `NotFoundError`,
193
+ `ListError`, `DeleteError`. The SDK's own exception is kept as `__cause__`.
194
+
195
+ ```python
196
+ from cloud_storage_manager import NotFoundError
197
+
198
+ try:
199
+ data = await storage.download_bytes("missing.txt")
200
+ except NotFoundError:
201
+ data = None
202
+ ```
203
+
204
+ ## Upgrading from 0.1.x
205
+
206
+ Change only the requirement: swap `cloud-storage-manager` for `10x-cloud-storage-manager`
207
+ (uninstall the old package first, since both ship the `cloud_storage_manager` module). Your code
208
+ does not change: every 0.1.x call behaves as before.
209
+
210
+ ```bash
211
+ pip uninstall -y cloud-storage-manager
212
+ pip install 10x-cloud-storage-manager
213
+ ```
214
+
215
+ - The base install pulls the same packages as 0.1.x; Azure is the `[azure]` extra.
216
+ - `StorageConfig` accepts every combination 0.1.x accepted.
217
+ - `upload`, `delete` and `get_public_url` behave as in 0.1.x on S3 and GCP; the new
218
+ `delete_if_exists` is the one to use for the same answer on every cloud.
219
+ - Only those three methods are abstract, so a subclass written for 0.1.x still instantiates.
220
+ - `AWSClientStorage` and `GcpClientStorage` are still importable from
221
+ `cloud_storage_manager.storage_provider`.
222
+ - Python 3.10+ (0.1.x already used 3.10 syntax and did not import on 3.8 or 3.9).
223
+
224
+ ## Development
225
+
226
+ ```bash
227
+ python -m venv .venv
228
+ source .venv/bin/activate
229
+ pip install -e ".[dev]"
230
+ pytest --cov
231
+ ruff check . && ruff format --check .
232
+ ```
233
+
234
+ Tests mock every cloud SDK and never call a real cloud.
235
+
236
+ ### Branches and releases
237
+
238
+ - `dev` is the default branch. Open feature PRs against it.
239
+ - `prod` is the release branch. **Every merge into `prod` publishes to PyPI** and creates a
240
+ `v<version>` GitHub release (`.github/workflows/release.yml`).
241
+ - Bump `[project].version` in `pyproject.toml` in the PR to `prod`. CI fails the PR if PyPI
242
+ already has that version.
243
+ - Publishing uses the `PYPI_API_TOKEN` secret on the `pypi` GitHub environment, which only
244
+ the `prod` branch can deploy to. Never upload to PyPI by hand.
245
+
246
+ ## License
247
+
248
+ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
@@ -0,0 +1,14 @@
1
+ 10x_cloud_storage_manager-0.2.0.dist-info/licenses/LICENSE,sha256=Xg5xEpG9i8MQHOAPZJ-GDeHZAf3bgHFhmH473ao_HLk,1107
2
+ cloud_storage_manager/__init__.py,sha256=6oe09IdqmjiBJvTtzpl_TTuPdtMuYxDU9il_PlPOI4Q,767
3
+ cloud_storage_manager/_distribution.py,sha256=KfllOA7XManYfmIIbP41lTIppW5k895gDD_eBelWhZg,280
4
+ cloud_storage_manager/aws_s3_storage.py,sha256=ecuhZK1SjmcXWUwg0Nb_lE5P4jF9VZ6HQ2MLfQWC_-U,8045
5
+ cloud_storage_manager/azure_storage.py,sha256=N0FFL70AQm9jLlOqxdSZzUwDLvEfguDeHOYPdfjO7kI,8106
6
+ cloud_storage_manager/base_storage.py,sha256=eGt_bab8CWIwBOiHPc4Y4AG7Hi_kQNqHUvdWV7AskRE,5101
7
+ cloud_storage_manager/exceptions.py,sha256=bm-0yM86SXLFjzXahdgd7hj8G7Ori1SDOoZbycWloH0,597
8
+ cloud_storage_manager/gcp_storage.py,sha256=nPO6IPX9yf4cyDzKtX2b_Tu_tbYRsyN-bS-MXJd5rto,9019
9
+ cloud_storage_manager/storage_provider.py,sha256=g6u5ppbrTqBY5t2o8VNwES3wPkdaVDd-WbmfXoL0vbc,2856
10
+ cloud_storage_manager/storage_schema.py,sha256=--cMVQpYswsSnnBWYxOWGnY_GZ_X5AxFLCatV2oktNE,2859
11
+ 10x_cloud_storage_manager-0.2.0.dist-info/METADATA,sha256=q4e4TCuTOW8wBAg20UAK3GUc9mZ9bVspxLG9FTbzIBQ,9520
12
+ 10x_cloud_storage_manager-0.2.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
13
+ 10x_cloud_storage_manager-0.2.0.dist-info/top_level.txt,sha256=NulGei6qRmnP61OgHE8wIilOcm2XMPiRJuZ35W_lQd0,22
14
+ 10x_cloud_storage_manager-0.2.0.dist-info/RECORD,,
@@ -0,0 +1,5 @@
1
+ Wheel-Version: 1.0
2
+ Generator: setuptools (84.2.1)
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
5
+
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2024 MUTHARASU a
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the “Software”), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is furnished
10
+ to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
@@ -0,0 +1 @@
1
+ cloud_storage_manager
@@ -0,0 +1,38 @@
1
+ """
2
+ Cloud Storage Manager Package
3
+ """
4
+
5
+ import logging
6
+
7
+ from .base_storage import BaseCloudStorage
8
+ from .exceptions import (
9
+ AuthorizationError,
10
+ CloudStorageError,
11
+ DeleteError,
12
+ DownloadError,
13
+ ListError,
14
+ NotFoundError,
15
+ UploadError,
16
+ )
17
+ from .storage_provider import CloudStorageFactory
18
+ from .storage_schema import AwsConfig, AzureConfig, GcpConfig, StorageConfig, StorageProvider
19
+
20
+ logger = logging.getLogger(__name__)
21
+
22
+ __all__ = [
23
+ "AuthorizationError",
24
+ "AwsConfig",
25
+ "AzureConfig",
26
+ "BaseCloudStorage",
27
+ "CloudStorageError",
28
+ "CloudStorageFactory",
29
+ "DeleteError",
30
+ "DownloadError",
31
+ "GcpConfig",
32
+ "ListError",
33
+ "NotFoundError",
34
+ "StorageConfig",
35
+ "StorageProvider",
36
+ "UploadError",
37
+ "logger",
38
+ ]
@@ -0,0 +1,4 @@
1
+ # The PyPI name differs from the import name: 0.1.x shipped as `cloud-storage-manager` from an
2
+ # account that can no longer publish, so 0.2.0 onwards ships under this name. Keep it in sync
3
+ # with `[project].name` in pyproject.toml.
4
+ DISTRIBUTION_NAME = "10x-cloud-storage-manager"
@@ -0,0 +1,168 @@
1
+ from pathlib import Path
2
+
3
+ import aioboto3
4
+ from botocore.exceptions import ClientError
5
+
6
+ from . import logger
7
+ from .base_storage import BaseCloudStorage, guess_content_type
8
+ from .exceptions import DeleteError, DownloadError, ListError, NotFoundError, UploadError
9
+ from .storage_schema import StorageConfig
10
+
11
+ _MISSING_CODES = {"404", "NoSuchKey", "NotFound"}
12
+
13
+
14
+ def _is_missing(error: ClientError) -> bool:
15
+ return error.response.get("Error", {}).get("Code") in _MISSING_CODES
16
+
17
+
18
+ class AWSClientStorage(BaseCloudStorage):
19
+ """
20
+ AWSClientStorage is a class for interacting with AWS S3 storage.
21
+ It implements every BaseCloudStorage operation against one S3 bucket.
22
+ Attributes:
23
+ session (aioboto3.Session): The boto3 session for AWS S3 interactions.
24
+ bucket (str): The name of the S3 bucket.
25
+ """
26
+
27
+ def __init__(self, config: StorageConfig):
28
+ """Initialize AWS S3 storage client.
29
+
30
+ Args:
31
+ config (StorageConfig): Configuration containing AWS credentials and settings
32
+ Must include aws.access_key_id, aws.secret_access_key, and aws.bucket_name
33
+
34
+ Raises:
35
+ ValueError: If AWS configuration is missing
36
+ """
37
+ if not config.aws:
38
+ raise ValueError("AWS configuration is required")
39
+ super().__init__(config)
40
+ # `region` is deliberately not passed: 0.1.x let boto resolve the region from the
41
+ # environment, and forcing the field's us-east-1 default would re-sign callers' URLs.
42
+ self.session = aioboto3.Session(
43
+ aws_access_key_id=config.aws.access_key_id,
44
+ aws_secret_access_key=config.aws.secret_access_key,
45
+ )
46
+ self.bucket = config.aws.bucket_name
47
+ logger.info(f"AWSClientStorage initialized with bucket: {self.bucket}")
48
+
49
+ async def upload(self, file_path: Path | str, cloud_path: str) -> str:
50
+ """Upload a local file to the S3 bucket; returns the S3 object path."""
51
+ logger.info(f"Starting upload of {file_path} to s3://{self.bucket}/{cloud_path}")
52
+ return await self.upload_file(file_path, cloud_path)
53
+
54
+ async def upload_bytes(
55
+ self, data: bytes, cloud_path: str, content_type: str | None = None
56
+ ) -> str:
57
+ """Upload in-memory bytes to the S3 bucket; returns the S3 object path."""
58
+ logger.info(f"Uploading {len(data)} bytes to s3://{self.bucket}/{cloud_path}")
59
+ content_type = content_type or guess_content_type(cloud_path)
60
+ extra = {"ContentType": content_type} if content_type else {}
61
+ async with self.session.client("s3") as s3:
62
+ try:
63
+ await s3.put_object(Bucket=self.bucket, Key=cloud_path, Body=data, **extra)
64
+ return cloud_path
65
+ except Exception as e:
66
+ logger.error(f"Upload failed: {e}")
67
+ raise UploadError(f"Failed to upload bytes to S3: {e}") from e
68
+
69
+ async def download_bytes(self, cloud_path: str) -> bytes:
70
+ """Download the S3 object's content."""
71
+ logger.info(f"Downloading s3://{self.bucket}/{cloud_path}")
72
+ async with self.session.client("s3") as s3:
73
+ try:
74
+ response = await s3.get_object(Bucket=self.bucket, Key=cloud_path)
75
+ async with response["Body"] as body:
76
+ return await body.read()
77
+ except ClientError as e:
78
+ if _is_missing(e):
79
+ raise NotFoundError(f"S3 object not found: {cloud_path}") from e
80
+ logger.error(f"Download failed: {e}")
81
+ raise DownloadError(f"Failed to download file from S3: {e}") from e
82
+ except Exception as e:
83
+ logger.error(f"Download failed: {e}")
84
+ raise DownloadError(f"Failed to download file from S3: {e}") from e
85
+
86
+ async def list_objects(self, prefix: str) -> list[str]:
87
+ """List S3 object keys under `prefix`."""
88
+ return list(await self.list_object_sizes(prefix))
89
+
90
+ async def list_object_sizes(self, prefix: str) -> dict[str, int]:
91
+ """S3 object key -> size in bytes under `prefix`."""
92
+ logger.info(f"Listing s3://{self.bucket}/{prefix}")
93
+ async with self.session.client("s3") as s3:
94
+ try:
95
+ sizes: dict[str, int] = {}
96
+ paginator = s3.get_paginator("list_objects_v2")
97
+ async for page in paginator.paginate(Bucket=self.bucket, Prefix=prefix):
98
+ for item in page.get("Contents", []):
99
+ sizes[item["Key"]] = item.get("Size", 0)
100
+ return sizes
101
+ except Exception as e:
102
+ logger.error(f"Listing failed: {e}")
103
+ raise ListError(f"Failed to list objects in S3: {e}") from e
104
+
105
+ async def delete(self, cloud_path: str) -> bool:
106
+ """Delete a file from the S3 bucket; True even for a missing key, as S3 answers."""
107
+ logger.info(f"Deleting s3://{self.bucket}/{cloud_path}")
108
+ return await self.delete_file(cloud_path)
109
+
110
+ async def delete_if_exists(self, cloud_path: str) -> bool:
111
+ """Delete an S3 object; False when it did not exist."""
112
+ logger.info(f"Deleting s3://{self.bucket}/{cloud_path} if it exists")
113
+ async with self.session.client("s3") as s3:
114
+ try:
115
+ # S3 deletes a missing key without complaint, so existence is checked first.
116
+ await s3.head_object(Bucket=self.bucket, Key=cloud_path)
117
+ except ClientError as e:
118
+ if _is_missing(e):
119
+ return False
120
+ logger.error(f"Deletion failed: {e}")
121
+ raise DeleteError(f"Failed to delete file from S3: {e}") from e
122
+ except Exception as e:
123
+ logger.error(f"Deletion failed: {e}")
124
+ raise DeleteError(f"Failed to delete file from S3: {e}") from e
125
+ return await self.delete_file(cloud_path)
126
+
127
+ async def get_public_url(self, cloud_path: str, expiration: int = 3600) -> str:
128
+ """Generate a pre-signed GET URL for a private S3 object."""
129
+ logger.info(f"Generating pre-signed URL for s3://{self.bucket}/{cloud_path}")
130
+ async with self.session.client("s3") as s3:
131
+ try:
132
+ url = await s3.generate_presigned_url(
133
+ "get_object",
134
+ Params={"Bucket": self.bucket, "Key": cloud_path},
135
+ ExpiresIn=expiration,
136
+ )
137
+ logger.debug(f"Generated pre-signed URL (expires in {expiration}s)")
138
+ return url
139
+ except Exception as e:
140
+ logger.error(f"Failed to generate pre-signed URL: {e}")
141
+ raise DownloadError(f"Failed to generate pre-signed URL: {e}") from e
142
+
143
+ # Helper methods
144
+ async def upload_file(self, file_path: Path | str, cloud_path: str) -> str:
145
+ """Internal method to handle file upload to S3.
146
+
147
+ No ContentType is sent, as in 0.1.x, so objects keep the type S3 assigned them before.
148
+ """
149
+ async with self.session.client("s3") as s3:
150
+ try:
151
+ with Path(file_path).open("rb") as file_obj:
152
+ await s3.upload_fileobj(file_obj, self.bucket, cloud_path)
153
+ logger.debug(f"Successfully uploaded to s3://{self.bucket}/{cloud_path}")
154
+ return cloud_path
155
+ except Exception as e:
156
+ logger.error(f"Upload failed: {e}")
157
+ raise UploadError(f"Failed to upload file to S3: {e}") from e
158
+
159
+ async def delete_file(self, cloud_path: str) -> bool:
160
+ """Internal method to handle file deletion from S3."""
161
+ async with self.session.client("s3") as s3:
162
+ try:
163
+ await s3.delete_object(Bucket=self.bucket, Key=cloud_path)
164
+ logger.debug(f"Successfully deleted s3://{self.bucket}/{cloud_path}")
165
+ return True
166
+ except Exception as e:
167
+ logger.error(f"Deletion failed: {e}")
168
+ raise DeleteError(f"Failed to delete file from S3: {e}") from e
@@ -0,0 +1,169 @@
1
+ from datetime import datetime, timedelta, timezone
2
+ from pathlib import Path
3
+
4
+ from azure.core.exceptions import ResourceNotFoundError
5
+ from azure.storage.blob import BlobSasPermissions, ContentSettings, generate_blob_sas
6
+ from azure.storage.blob.aio import BlobServiceClient
7
+
8
+ from . import logger
9
+ from .base_storage import BaseCloudStorage, guess_content_type
10
+ from .exceptions import DeleteError, DownloadError, ListError, NotFoundError, UploadError
11
+ from .storage_schema import StorageConfig
12
+
13
+
14
+ def _connection_string_key(connection_string: str) -> str | None:
15
+ """The AccountKey inside a connection string; None for a SAS-only one."""
16
+ parts = dict(part.split("=", 1) for part in connection_string.split(";") if "=" in part)
17
+ return {k.lower(): v for k, v in parts.items()}.get("accountkey")
18
+
19
+
20
+ def _content_settings(content_type: str | None) -> dict:
21
+ # Untyped blobs are served as octet-stream, which browsers download instead of playing.
22
+ return {"content_settings": ContentSettings(content_type=content_type)} if content_type else {}
23
+
24
+
25
+ class AzureBlobStorage(BaseCloudStorage):
26
+ """Azure Blob Storage client implementation.
27
+
28
+ Implements every BaseCloudStorage operation against one blob container, using the
29
+ async SDK. Signed URLs are read-only blob SAS tokens signed with the account key.
30
+
31
+ Attributes:
32
+ account_name (str): Storage account name.
33
+ container_name (str): Blob container name.
34
+ """
35
+
36
+ def __init__(self, config: StorageConfig):
37
+ """Initialize Azure Blob storage client.
38
+
39
+ Args:
40
+ config: Configuration object containing azure.account_name,
41
+ azure.container_name and one of azure.account_key / azure.connection_string.
42
+
43
+ Raises:
44
+ ValueError: If Azure configuration is missing from config.
45
+ """
46
+ if not config.azure:
47
+ raise ValueError("Azure configuration is required")
48
+ super().__init__(config)
49
+ self.account_name = config.azure.account_name
50
+ self.container_name = config.azure.container_name
51
+ self._connection_string = config.azure.connection_string
52
+ self._account_key = config.azure.account_key or _connection_string_key(
53
+ config.azure.connection_string or ""
54
+ )
55
+ logger.info(f"AzureBlobStorage initialized with container: {self.container_name}")
56
+
57
+ def _service_client(self) -> BlobServiceClient:
58
+ if self._connection_string:
59
+ return BlobServiceClient.from_connection_string(self._connection_string)
60
+ return BlobServiceClient(
61
+ account_url=f"https://{self.account_name}.blob.core.windows.net",
62
+ credential=self._account_key,
63
+ )
64
+
65
+ async def upload(self, file_path: Path | str, cloud_path: str) -> str:
66
+ """Upload a local file to the container; returns the blob path."""
67
+ logger.info(f"Starting upload of {file_path} to azure://{self.container_name}/{cloud_path}")
68
+ settings = _content_settings(guess_content_type(file_path))
69
+ try:
70
+ async with self._service_client() as client:
71
+ blob = client.get_blob_client(self.container_name, cloud_path)
72
+ with Path(file_path).open("rb") as file_obj:
73
+ await blob.upload_blob(file_obj, overwrite=True, **settings)
74
+ return cloud_path
75
+ except Exception as e:
76
+ logger.error(f"Upload failed: {e}")
77
+ raise UploadError(f"Failed to upload file to Azure: {e}") from e
78
+
79
+ async def upload_bytes(
80
+ self, data: bytes, cloud_path: str, content_type: str | None = None
81
+ ) -> str:
82
+ """Upload in-memory bytes to the container; returns the blob path."""
83
+ logger.info(f"Uploading {len(data)} bytes to azure://{self.container_name}/{cloud_path}")
84
+ settings = _content_settings(content_type or guess_content_type(cloud_path))
85
+ try:
86
+ async with self._service_client() as client:
87
+ blob = client.get_blob_client(self.container_name, cloud_path)
88
+ await blob.upload_blob(data, overwrite=True, **settings)
89
+ return cloud_path
90
+ except Exception as e:
91
+ logger.error(f"Upload failed: {e}")
92
+ raise UploadError(f"Failed to upload bytes to Azure: {e}") from e
93
+
94
+ async def download_bytes(self, cloud_path: str) -> bytes:
95
+ """Download the blob's content."""
96
+ logger.info(f"Downloading azure://{self.container_name}/{cloud_path}")
97
+ try:
98
+ async with self._service_client() as client:
99
+ blob = client.get_blob_client(self.container_name, cloud_path)
100
+ downloader = await blob.download_blob()
101
+ return await downloader.readall()
102
+ except ResourceNotFoundError as e:
103
+ raise NotFoundError(f"Azure blob not found: {cloud_path}") from e
104
+ except Exception as e:
105
+ logger.error(f"Download failed: {e}")
106
+ raise DownloadError(f"Failed to download blob from Azure: {e}") from e
107
+
108
+ async def list_objects(self, prefix: str) -> list[str]:
109
+ """List blob names under `prefix`."""
110
+ return list(await self.list_object_sizes(prefix))
111
+
112
+ async def list_object_sizes(self, prefix: str) -> dict[str, int]:
113
+ """Blob name -> size in bytes under `prefix`."""
114
+ logger.info(f"Listing azure://{self.container_name}/{prefix}")
115
+ try:
116
+ async with self._service_client() as client:
117
+ container = client.get_container_client(self.container_name)
118
+ return {
119
+ b.name: b.size or 0
120
+ async for b in container.list_blobs(name_starts_with=prefix or None)
121
+ }
122
+ except Exception as e:
123
+ logger.error(f"Listing failed: {e}")
124
+ raise ListError(f"Failed to list blobs in Azure: {e}") from e
125
+
126
+ async def delete(self, cloud_path: str) -> bool:
127
+ """Delete a blob; a missing blob raises DeleteError, as GCP's delete does."""
128
+ if not await self.delete_if_exists(cloud_path):
129
+ raise DeleteError(f"Failed to delete blob from Azure: {cloud_path} does not exist")
130
+ return True
131
+
132
+ async def delete_if_exists(self, cloud_path: str) -> bool:
133
+ """Delete a blob; False when it did not exist."""
134
+ logger.info(f"Deleting azure://{self.container_name}/{cloud_path} if it exists")
135
+ try:
136
+ async with self._service_client() as client:
137
+ await client.get_blob_client(self.container_name, cloud_path).delete_blob()
138
+ return True
139
+ except ResourceNotFoundError:
140
+ return False
141
+ except Exception as e:
142
+ logger.error(f"Deletion failed: {e}")
143
+ raise DeleteError(f"Failed to delete blob from Azure: {e}") from e
144
+
145
+ async def get_public_url(self, cloud_path: str, expiration: int = 3600) -> str:
146
+ """Generate a read-only SAS URL for the blob, valid for `expiration` seconds.
147
+
148
+ The SAS does not override the response Content-Type, so the browser receives the
149
+ type stored at upload.
150
+ """
151
+ logger.info(f"Generating SAS URL for azure://{self.container_name}/{cloud_path}")
152
+ if not self._account_key:
153
+ raise DownloadError("Azure SAS signing needs an account key in the configuration")
154
+ try:
155
+ sas = generate_blob_sas(
156
+ account_name=self.account_name,
157
+ container_name=self.container_name,
158
+ blob_name=cloud_path,
159
+ account_key=self._account_key,
160
+ permission=BlobSasPermissions(read=True),
161
+ expiry=datetime.now(timezone.utc) + timedelta(seconds=expiration),
162
+ )
163
+ async with self._service_client() as client:
164
+ blob_url = client.get_blob_client(self.container_name, cloud_path).url
165
+ logger.debug(f"Generated SAS URL (expires in {expiration}s)")
166
+ return f"{blob_url}?{sas}"
167
+ except Exception as e:
168
+ logger.error(f"Failed to generate SAS URL: {e}")
169
+ raise DownloadError(f"Failed to generate SAS URL: {e}") from e
@@ -0,0 +1,156 @@
1
+ import mimetypes
2
+ from abc import ABC, abstractmethod
3
+ from pathlib import Path
4
+
5
+ from .storage_schema import StorageConfig
6
+
7
+
8
+ def guess_content_type(path: Path | str) -> str | None:
9
+ """Content type from a file name's extension, or None when it is unknown."""
10
+ return mimetypes.guess_type(str(path))[0]
11
+
12
+
13
+ class BaseCloudStorage(ABC):
14
+ """Abstract base class for cloud storage operations.
15
+
16
+ Only the three 0.1.x operations (upload, delete, get_public_url) are abstract, so a
17
+ subclass written against 0.1.x still instantiates. The operations added in 0.2.0 raise
18
+ NotImplementedError until a subclass provides them; every built-in provider does.
19
+
20
+ Attributes:
21
+ config (StorageConfig): Configuration object for the cloud storage.
22
+ """
23
+
24
+ def __init__(self, config: StorageConfig):
25
+ """Initialize the cloud storage interface.
26
+
27
+ Args:
28
+ config (StorageConfig): Configuration object for the cloud storage.
29
+ """
30
+ self.config = config
31
+
32
+ def _unsupported(self, operation: str) -> NotImplementedError:
33
+ return NotImplementedError(f"{type(self).__name__} does not implement {operation}")
34
+
35
+ @abstractmethod
36
+ async def upload(self, file_path: Path | str, cloud_path: str) -> str:
37
+ """Upload a local file to cloud storage.
38
+
39
+ Args:
40
+ file_path (Path | str): Local path to the file to be uploaded.
41
+ cloud_path (str): Destination path in cloud storage.
42
+
43
+ Returns:
44
+ str: The path of the uploaded file in cloud storage.
45
+
46
+ Raises:
47
+ UploadError: If the upload fails.
48
+ """
49
+
50
+ async def upload_bytes(
51
+ self, data: bytes, cloud_path: str, content_type: str | None = None
52
+ ) -> str:
53
+ """Upload in-memory bytes to cloud storage.
54
+
55
+ Args:
56
+ data (bytes): The object's content.
57
+ cloud_path (str): Destination path in cloud storage.
58
+ content_type (str | None): Stored content type; guessed from cloud_path when None.
59
+
60
+ Returns:
61
+ str: The path of the uploaded object in cloud storage.
62
+
63
+ Raises:
64
+ UploadError: If the upload fails.
65
+ """
66
+ raise self._unsupported("upload_bytes")
67
+
68
+ async def download_bytes(self, cloud_path: str) -> bytes:
69
+ """Download an object's content.
70
+
71
+ Args:
72
+ cloud_path (str): Path to the object in cloud storage.
73
+
74
+ Returns:
75
+ bytes: The object's content.
76
+
77
+ Raises:
78
+ NotFoundError: If the object does not exist.
79
+ DownloadError: If the download fails for any other reason.
80
+ """
81
+ raise self._unsupported("download_bytes")
82
+
83
+ async def list_objects(self, prefix: str) -> list[str]:
84
+ """List the names of every object under a prefix.
85
+
86
+ Args:
87
+ prefix (str): Path prefix; an empty string lists the whole bucket.
88
+
89
+ Returns:
90
+ list[str]: Object names, across every page of the listing.
91
+
92
+ Raises:
93
+ ListError: If the listing fails.
94
+ """
95
+ raise self._unsupported("list_objects")
96
+
97
+ async def list_object_sizes(self, prefix: str) -> dict[str, int]:
98
+ """Map every object under a prefix to its size in bytes.
99
+
100
+ Args:
101
+ prefix (str): Path prefix; an empty string lists the whole bucket.
102
+
103
+ Returns:
104
+ dict[str, int]: Object name to size in bytes.
105
+
106
+ Raises:
107
+ ListError: If the listing fails.
108
+ """
109
+ raise self._unsupported("list_object_sizes")
110
+
111
+ @abstractmethod
112
+ async def delete(self, cloud_path: str) -> bool:
113
+ """Delete a file from cloud storage, as 0.1.x did.
114
+
115
+ A missing object gets each cloud's own answer, unchanged since 0.1.x: GCP and Azure
116
+ raise DeleteError, S3 deletes nothing and returns True. delete_if_exists gives the
117
+ same answer on every cloud.
118
+
119
+ Args:
120
+ cloud_path (str): Path to the file in cloud storage to be deleted.
121
+
122
+ Returns:
123
+ bool: True once the delete request succeeded.
124
+
125
+ Raises:
126
+ DeleteError: If the deletion fails.
127
+ """
128
+
129
+ async def delete_if_exists(self, cloud_path: str) -> bool:
130
+ """Delete an object, reporting whether it was there, the same way on every cloud.
131
+
132
+ Args:
133
+ cloud_path (str): Path to the object in cloud storage.
134
+
135
+ Returns:
136
+ bool: True if the object was deleted, False if it did not exist.
137
+
138
+ Raises:
139
+ DeleteError: If the deletion fails for any other reason.
140
+ """
141
+ raise self._unsupported("delete_if_exists")
142
+
143
+ @abstractmethod
144
+ async def get_public_url(self, cloud_path: str, expiration: int = 3600) -> str:
145
+ """Generate a time-limited, read-only signed URL for the file.
146
+
147
+ Args:
148
+ cloud_path (str): Path to the file in cloud storage.
149
+ expiration (int, optional): URL expiration time in seconds. Defaults to 3600.
150
+
151
+ Returns:
152
+ str: Signed URL for reading the file.
153
+
154
+ Raises:
155
+ DownloadError: If URL generation fails.
156
+ """
@@ -0,0 +1,26 @@
1
+ class CloudStorageError(Exception):
2
+ """Base exception for cloud storage operations"""
3
+
4
+
5
+ class UploadError(CloudStorageError):
6
+ """Error occurred during file upload"""
7
+
8
+
9
+ class DeleteError(CloudStorageError):
10
+ """Error occurred during file deletion"""
11
+
12
+
13
+ class DownloadError(CloudStorageError):
14
+ """Error occurred during file download"""
15
+
16
+
17
+ class NotFoundError(CloudStorageError):
18
+ """Resource not found"""
19
+
20
+
21
+ class AuthorizationError(CloudStorageError):
22
+ """Authorization/Authentication failed"""
23
+
24
+
25
+ class ListError(CloudStorageError):
26
+ """Error occurred while listing objects"""
@@ -0,0 +1,226 @@
1
+ import asyncio
2
+ from pathlib import Path
3
+
4
+ from google.cloud import storage
5
+ from google.cloud.exceptions import NotFound
6
+ from google.oauth2 import service_account
7
+
8
+ from . import logger
9
+ from .base_storage import BaseCloudStorage, guess_content_type
10
+ from .exceptions import DeleteError, DownloadError, ListError, NotFoundError, UploadError
11
+ from .storage_schema import StorageConfig
12
+
13
+
14
+ class GcpClientStorage(BaseCloudStorage):
15
+ """Google Cloud Storage client implementation.
16
+
17
+ This class implements the BaseCloudStorage interface for Google Cloud Storage.
18
+ It provides methods for uploading, deleting files and generating signed URLs
19
+ for temporary access to private objects.
20
+
21
+ Attributes:
22
+ credentials: The GCP service account credentials.
23
+ bucket_name (str): Name of the GCP storage bucket.
24
+ client: The GCP storage client instance.
25
+ bucket: The GCP bucket instance.
26
+
27
+ Example:
28
+ config = StorageConfig(gcp={
29
+ 'credentials': {...},
30
+ 'bucket_name': 'my-bucket'
31
+ })
32
+ storage = GcpClientStorage(config)
33
+ await storage.upload('path.txt', 'remote/path.txt')
34
+ """
35
+
36
+ def __init__(self, config: StorageConfig):
37
+ """Initialize GCP storage client.
38
+
39
+ Args:
40
+ config: Configuration object containing GCP settings.
41
+ Must include gcp.credentials (dict) and gcp.bucket_name (str).
42
+
43
+ Raises:
44
+ ValueError: If GCP configuration is missing from config.
45
+ """
46
+ if not config.gcp:
47
+ raise ValueError("GCP configuration is required")
48
+ super().__init__(config)
49
+ self.credentials = service_account.Credentials.from_service_account_info(
50
+ config.gcp.credentials
51
+ )
52
+ self.bucket_name = config.gcp.bucket_name
53
+ self.client = storage.Client(credentials=self.credentials)
54
+ self.bucket = self.client.get_bucket(self.bucket_name)
55
+ logger.info(f"GcpClientStorage initialized with bucket: {self.bucket_name}")
56
+
57
+ async def upload(self, file_path: Path | str, cloud_path: str | None = None) -> str:
58
+ """Upload a file to Google Cloud Storage bucket.
59
+
60
+ Args:
61
+ file_path: Local filesystem path of the file to upload.
62
+ Can be string or Path object.
63
+ cloud_path: Target path in GCP bucket. If None, uses the original filename.
64
+
65
+ Returns:
66
+ The GCP object path where the file was uploaded.
67
+
68
+ Raises:
69
+ UploadError: If the upload operation fails for any reason.
70
+ """
71
+ if cloud_path is None:
72
+ cloud_path = Path(file_path).name
73
+
74
+ logger.info(f"Starting upload of {file_path} to gs://{self.bucket_name}/{cloud_path}")
75
+ try:
76
+ return await asyncio.to_thread(self._upload_file, file_path, cloud_path)
77
+ except Exception as e:
78
+ logger.error(f"Upload failed: {e}")
79
+ raise UploadError(f"Failed to upload file to GCP: {e}") from e
80
+
81
+ async def upload_bytes(
82
+ self, data: bytes, cloud_path: str, content_type: str | None = None
83
+ ) -> str:
84
+ """Upload in-memory bytes to the GCP bucket; returns the object path."""
85
+ logger.info(f"Uploading {len(data)} bytes to gs://{self.bucket_name}/{cloud_path}")
86
+ content_type = content_type or guess_content_type(cloud_path)
87
+ try:
88
+ await asyncio.to_thread(
89
+ self.bucket.blob(cloud_path).upload_from_string, data, content_type=content_type
90
+ )
91
+ return cloud_path
92
+ except Exception as e:
93
+ logger.error(f"Upload failed: {e}")
94
+ raise UploadError(f"Failed to upload bytes to GCP: {e}") from e
95
+
96
+ async def download_bytes(self, cloud_path: str) -> bytes:
97
+ """Download the GCP object's content."""
98
+ logger.info(f"Downloading gs://{self.bucket_name}/{cloud_path}")
99
+ try:
100
+ return await asyncio.to_thread(self.bucket.blob(cloud_path).download_as_bytes)
101
+ except NotFound as e:
102
+ raise NotFoundError(f"GCP object not found: {cloud_path}") from e
103
+ except Exception as e:
104
+ logger.error(f"Download failed: {e}")
105
+ raise DownloadError(f"Failed to download file from GCP: {e}") from e
106
+
107
+ async def list_objects(self, prefix: str) -> list[str]:
108
+ """List GCP object names under `prefix`."""
109
+ return list(await self.list_object_sizes(prefix))
110
+
111
+ async def list_object_sizes(self, prefix: str) -> dict[str, int]:
112
+ """GCP object name -> size in bytes under `prefix`."""
113
+ logger.info(f"Listing gs://{self.bucket_name}/{prefix}")
114
+ try:
115
+ return await asyncio.to_thread(self._list_sizes, prefix)
116
+ except Exception as e:
117
+ logger.error(f"Listing failed: {e}")
118
+ raise ListError(f"Failed to list objects in GCP: {e}") from e
119
+
120
+ async def delete(self, cloud_path: str) -> bool:
121
+ """Delete a file from Google Cloud Storage bucket.
122
+
123
+ Args:
124
+ cloud_path: The object path in GCP bucket to delete.
125
+
126
+ Returns:
127
+ True if deletion was successful.
128
+
129
+ Raises:
130
+ DeleteError: If the deletion operation fails, including a missing object.
131
+ """
132
+ logger.info(f"Deleting gs://{self.bucket_name}/{cloud_path}")
133
+ try:
134
+ return await asyncio.to_thread(self._delete_file, cloud_path)
135
+ except Exception as e:
136
+ logger.error(f"Deletion failed: {e}")
137
+ raise DeleteError(f"Failed to delete file from GCP: {e}") from e
138
+
139
+ async def delete_if_exists(self, cloud_path: str) -> bool:
140
+ """Delete a GCP object; False when it did not exist."""
141
+ logger.info(f"Deleting gs://{self.bucket_name}/{cloud_path} if it exists")
142
+ try:
143
+ return await asyncio.to_thread(self._delete_file, cloud_path)
144
+ except NotFound:
145
+ return False
146
+ except Exception as e:
147
+ logger.error(f"Deletion failed: {e}")
148
+ raise DeleteError(f"Failed to delete file from GCP: {e}") from e
149
+
150
+ async def get_public_url(self, cloud_path: str, expiration: int = 3600) -> str:
151
+ """Generate a temporary signed URL for accessing a private GCP object.
152
+
153
+ Args:
154
+ cloud_path: The object path in GCP bucket.
155
+ expiration: URL validity duration in seconds. Defaults to 1 hour (3600 seconds).
156
+
157
+ Returns:
158
+ A signed URL that provides temporary access to the private object.
159
+
160
+ Raises:
161
+ DownloadError: If URL generation fails.
162
+ """
163
+ logger.info(f"Generating signed URL for gs://{self.bucket_name}/{cloud_path}")
164
+ try:
165
+ return await asyncio.to_thread(self._generate_signed_url, cloud_path, expiration)
166
+ except Exception as e:
167
+ logger.error(f"Failed to generate signed URL: {e}")
168
+ raise DownloadError(f"Failed to generate signed URL: {e}") from e
169
+
170
+ def _upload_file(self, file_path: str | Path, cloud_path: str) -> str:
171
+ """Perform synchronous file upload to GCP bucket.
172
+
173
+ Internal method used by the async upload method.
174
+
175
+ Args:
176
+ file_path: Local file path to upload.
177
+ cloud_path: Target path in GCP bucket.
178
+
179
+ Returns:
180
+ The GCP object path where file was uploaded.
181
+ """
182
+ blob = self.bucket.blob(cloud_path)
183
+ blob.upload_from_filename(str(file_path))
184
+ logger.debug(f"Successfully uploaded to gs://{self.bucket_name}/{cloud_path}")
185
+ return cloud_path
186
+
187
+ def _delete_file(self, cloud_path: str) -> bool:
188
+ """Perform synchronous file deletion from GCP bucket.
189
+
190
+ Internal method used by the async delete method.
191
+
192
+ Args:
193
+ cloud_path: Object path in GCP bucket to delete.
194
+
195
+ Returns:
196
+ True if deletion was successful.
197
+ """
198
+ blob = self.bucket.blob(cloud_path)
199
+ blob.delete()
200
+ logger.debug(f"Successfully deleted gs://{self.bucket_name}/{cloud_path}")
201
+ return True
202
+
203
+ def _list_sizes(self, prefix: str) -> dict[str, int]:
204
+ """Name -> size for every object under `prefix`.
205
+
206
+ `list_blobs` pages lazily over HTTP as it is iterated, so it is drained here on the
207
+ worker thread rather than back on the event loop.
208
+ """
209
+ return {b.name: b.size or 0 for b in self.bucket.list_blobs(prefix=prefix)}
210
+
211
+ def _generate_signed_url(self, cloud_path: str, expiration: int) -> str:
212
+ """Generate a signed URL for temporary object access.
213
+
214
+ Internal method used by the async get_public_url method.
215
+
216
+ Args:
217
+ cloud_path: Object path in GCP bucket.
218
+ expiration: URL validity duration in seconds.
219
+
220
+ Returns:
221
+ A signed URL for temporary object access.
222
+ """
223
+ blob = self.bucket.blob(cloud_path)
224
+ url = blob.generate_signed_url(expiration=expiration, method="GET", version="v4")
225
+ logger.debug(f"Generated signed URL (expires in {expiration}s)")
226
+ return url
@@ -0,0 +1,72 @@
1
+ from importlib import import_module
2
+
3
+ from ._distribution import DISTRIBUTION_NAME
4
+ from .base_storage import BaseCloudStorage
5
+ from .storage_schema import StorageConfig, StorageProvider
6
+
7
+ __all__ = ["CloudStorageFactory", "StorageProvider"]
8
+
9
+ # 0.1.x imported these two classes here eagerly; they stay importable from this module, but load
10
+ # on first access so `import cloud_storage_manager` pulls in no cloud SDK.
11
+ _LEGACY_CLASSES = {
12
+ "AWSClientStorage": ".aws_s3_storage",
13
+ "GcpClientStorage": ".gcp_storage",
14
+ }
15
+
16
+
17
+ def __getattr__(name: str):
18
+ if name in _LEGACY_CLASSES:
19
+ return getattr(import_module(_LEGACY_CLASSES[name], __package__), name)
20
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
21
+
22
+
23
+ class CloudStorageFactory:
24
+ """Factory class for creating cloud storage client instances.
25
+
26
+ Each provider module is imported only when it is asked for, so installing a
27
+ single cloud's extra (``10x-cloud-storage-manager[gcp]``) is enough to use it.
28
+
29
+ Example:
30
+ >>> config = StorageConfig(gcp=GcpConfig(...))
31
+ >>> storage = CloudStorageFactory.from_config(config)
32
+ >>> await storage.upload_bytes(b"...", "remote/path.txt", "text/plain")
33
+ """
34
+
35
+ _providers: dict[StorageProvider, tuple[str, str]] = {
36
+ StorageProvider.AWS: (".aws_s3_storage", "AWSClientStorage"),
37
+ StorageProvider.GCP: (".gcp_storage", "GcpClientStorage"),
38
+ StorageProvider.AZURE: (".azure_storage", "AzureBlobStorage"),
39
+ }
40
+
41
+ @classmethod
42
+ def get_storage(cls, provider: StorageProvider, config: StorageConfig) -> BaseCloudStorage:
43
+ """Creates and returns a cloud storage instance for the specified provider.
44
+
45
+ Args:
46
+ provider: The cloud storage provider to use.
47
+ config: Configuration parameters for initializing the storage client.
48
+
49
+ Returns:
50
+ A concrete implementation of BaseCloudStorage for the specified provider.
51
+
52
+ Raises:
53
+ ValueError: If the specified provider is not supported.
54
+ ImportError: If the provider's extra is not installed.
55
+ """
56
+ target = cls._providers.get(provider)
57
+ if not target:
58
+ raise ValueError(f"Unsupported storage provider: {provider}")
59
+ module_name, class_name = target
60
+ try:
61
+ module = import_module(module_name, __package__)
62
+ except ImportError as e:
63
+ raise ImportError(
64
+ f"{provider.value} storage needs its extra: "
65
+ f"pip install '{DISTRIBUTION_NAME}[{provider.value}]'"
66
+ ) from e
67
+ return getattr(module, class_name)(config)
68
+
69
+ @classmethod
70
+ def from_config(cls, config: StorageConfig) -> BaseCloudStorage:
71
+ """Creates the storage client for whichever provider the config sets."""
72
+ return cls.get_storage(config.provider, config)
@@ -0,0 +1,73 @@
1
+ from enum import Enum
2
+
3
+ from pydantic import BaseModel, Field, model_validator
4
+
5
+
6
+ class StorageProvider(Enum):
7
+ """Enumeration of supported cloud storage providers."""
8
+
9
+ AWS = "aws"
10
+ GCP = "gcp"
11
+ AZURE = "azure"
12
+
13
+
14
+ class AwsConfig(BaseModel):
15
+ bucket_name: str = Field(..., description="AWS S3 bucket name")
16
+ access_key_id: str = Field(..., description="AWS access key ID")
17
+ secret_access_key: str = Field(..., description="AWS secret access key")
18
+ region: str = Field(default="us-east-1", description="AWS region")
19
+
20
+
21
+ class GcpConfig(BaseModel):
22
+ bucket_name: str = Field(..., description="GCP bucket name")
23
+ credentials: dict = Field(..., description="GCP service account credentials")
24
+ project_id: str = Field(..., description="GCP project ID")
25
+
26
+
27
+ class AzureConfig(BaseModel):
28
+ account_name: str = Field(..., description="Azure storage account name")
29
+ container_name: str = Field(..., description="Azure Blob container name")
30
+ account_key: str | None = Field(None, description="Storage account shared key")
31
+ connection_string: str | None = Field(None, description="Storage account connection string")
32
+
33
+ @model_validator(mode="after")
34
+ def _exactly_one_credential(self) -> "AzureConfig":
35
+ if (self.account_key is None) == (self.connection_string is None):
36
+ raise ValueError("Set exactly one of account_key or connection_string")
37
+ return self
38
+
39
+
40
+ class StorageConfig(BaseModel):
41
+ """Holds one config per cloud.
42
+
43
+ Construction accepts any combination, as 0.1.x did: a caller picks the provider explicitly
44
+ with CloudStorageFactory.get_storage(provider, config). Only `provider` and
45
+ `CloudStorageFactory.from_config`, which have to infer it, need exactly one.
46
+ """
47
+
48
+ aws: AwsConfig | None = Field(None, description="AWS configuration")
49
+ gcp: GcpConfig | None = Field(None, description="GCP configuration")
50
+ azure: AzureConfig | None = Field(None, description="Azure Blob configuration")
51
+
52
+ def _configured(self) -> list[StorageProvider]:
53
+ return [p for p in StorageProvider if getattr(self, p.value) is not None]
54
+
55
+ @property
56
+ def provider(self) -> StorageProvider:
57
+ """The one provider this config sets.
58
+
59
+ Raises:
60
+ ValueError: If no provider, or more than one, is configured.
61
+ """
62
+ configured = self._configured()
63
+ if len(configured) != 1:
64
+ names = ", ".join(p.value for p in configured) or "none"
65
+ raise ValueError(f"Configure exactly one storage provider (got: {names})")
66
+ return configured[0]
67
+
68
+ def has_valid_provider(self) -> bool:
69
+ """True when at least one provider is configured, as in 0.1.x."""
70
+ return bool(self._configured())
71
+
72
+ def validate_provider(self) -> None:
73
+ """Kept for 0.1.x callers, where it never raised; it still does not."""