PyS3Uploader 0.0.0a0__py3-none-any.whl → 0.1.1__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of PyS3Uploader might be problematic. Click here for more details.

@@ -0,0 +1,187 @@
1
+ Metadata-Version: 2.2
2
+ Name: PyS3Uploader
3
+ Version: 0.1.1
4
+ Summary: Python module to upload objects to an S3 bucket.
5
+ Author-email: Vignesh Rao <svignesh1793@gmail.com>
6
+ License: MIT License
7
+
8
+ Copyright (c) 2025 Vignesh Rao
9
+
10
+ Permission is hereby granted, free of charge, to any person obtaining a copy
11
+ of this software and associated documentation files (the "Software"), to deal
12
+ in the Software without restriction, including without limitation the rights
13
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
+ copies of the Software, and to permit persons to whom the Software is
15
+ furnished to do so, subject to the following conditions:
16
+
17
+ The above copyright notice and this permission notice shall be included in all
18
+ copies or substantial portions of the Software.
19
+
20
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
26
+ SOFTWARE.
27
+
28
+ Project-URL: Homepage, https://github.com/thevickypedia/PyS3Uploader
29
+ Project-URL: Docs, https://thevickypedia.github.io/PyS3Uploader/
30
+ Project-URL: Source, https://github.com/thevickypedia/PyS3Uploader
31
+ Project-URL: Bug Tracker, https://github.com/thevickypedia/PyS3Uploader/issues
32
+ Keywords: s3
33
+ Classifier: Development Status :: 1 - Planning
34
+ Classifier: Intended Audience :: Information Technology
35
+ Classifier: Operating System :: OS Independent
36
+ Classifier: License :: OSI Approved :: MIT License
37
+ Classifier: Programming Language :: Python :: 3.11
38
+ Classifier: Topic :: Internet :: File Transfer Protocol (FTP)
39
+ Requires-Python: >=3.11
40
+ Description-Content-Type: text/markdown
41
+ License-File: LICENSE
42
+ Requires-Dist: boto3==1.40.*
43
+ Requires-Dist: tqdm==4.67.*
44
+ Provides-Extra: dev
45
+ Requires-Dist: sphinx==5.1.1; extra == "dev"
46
+ Requires-Dist: pre-commit; extra == "dev"
47
+ Requires-Dist: recommonmark; extra == "dev"
48
+
49
+ **Versions Supported**
50
+
51
+ ![Python](https://img.shields.io/badge/python-3.11-blue)
52
+
53
+ **Language Stats**
54
+
55
+ ![Language count](https://img.shields.io/github/languages/count/thevickypedia/PyS3Uploader)
56
+ ![Code coverage](https://img.shields.io/github/languages/top/thevickypedia/PyS3Uploader)
57
+
58
+ **Repo Stats**
59
+
60
+ [![GitHub](https://img.shields.io/github/license/thevickypedia/PyS3Uploader)][license]
61
+ [![GitHub repo size](https://img.shields.io/github/repo-size/thevickypedia/PyS3Uploader)][repo]
62
+ [![GitHub code size](https://img.shields.io/github/languages/code-size/thevickypedia/PyS3Uploader)][repo]
63
+
64
+ **Activity**
65
+
66
+ [![GitHub Repo created](https://img.shields.io/date/1618966420)][repo]
67
+ [![GitHub commit activity](https://img.shields.io/github/commit-activity/y/thevickypedia/PyS3Uploader)][repo]
68
+ [![GitHub last commit](https://img.shields.io/github/last-commit/thevickypedia/PyS3Uploader)][repo]
69
+
70
+ **Build Status**
71
+
72
+ [![pypi-publish][gha-pypi-badge]][gha-pypi]
73
+ [![pages-build-deployment][gha-pages-badge]][gha-pages]
74
+
75
+ # PyS3Uploader
76
+ Python module to upload an entire directory to an S3 bucket.
77
+
78
+ ### Installation
79
+ ```shell
80
+ pip install PyS3Uploader
81
+ ```
82
+
83
+ ### Usage
84
+
85
+ ##### Upload objects in parallel
86
+ ```python
87
+ import s3
88
+
89
+ if __name__ == '__main__':
90
+ wrapper = s3.Uploader(
91
+ bucket_name="BUCKET_NAME",
92
+ upload_dir="FULL_PATH_TO_UPLOAD",
93
+ exclude_path="PART_OF_UPLOAD_DIR_TO_EXCLUDE"
94
+ )
95
+ wrapper.run_in_parallel()
96
+ ```
97
+
98
+ ##### Upload objects in sequence
99
+ ```python
100
+ import s3
101
+
102
+ if __name__ == '__main__':
103
+ wrapper = s3.Uploader(
104
+ bucket_name="BUCKET_NAME",
105
+ upload_dir="FULL_PATH_TO_UPLOAD",
106
+ exclude_path="PART_OF_UPLOAD_DIR_TO_EXCLUDE"
107
+ )
108
+ wrapper.run()
109
+ ```
110
+
111
+ #### Mandatory arg
112
+ - **bucket_name** - Name of the s3 bucket.
113
+ - **upload_dir** - Directory to upload.
114
+
115
+ #### Optional kwargs
116
+ - **s3_prefix** - S3 object prefix for each file. Defaults to ``None``
117
+ - **exclude_path** - Path in ``upload_dir`` that has to be excluded in object keys. Defaults to `None`
118
+ - **logger** - Bring your own custom pre-configured logger. Defaults to on-screen logging.
119
+ <br><br>
120
+ - **region_name** - AWS region name. Defaults to the env var `AWS_DEFAULT_REGION`
121
+ - **profile_name** - AWS profile name. Defaults to the env var `PROFILE_NAME`
122
+ - **aws_access_key_id** - AWS access key ID. Defaults to the env var `AWS_ACCESS_KEY_ID`
123
+ - **aws_secret_access_key** - AWS secret access key. Defaults to the env var `AWS_SECRET_ACCESS_KEY`
124
+ > AWS values are loaded from env vars or the default config at `~/.aws/config` / `~/.aws/credentials`
125
+
126
+ ### Coding Standards
127
+ Docstring format: [`Google`](https://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings) <br>
128
+ Styling conventions: [`PEP 8`](https://www.python.org/dev/peps/pep-0008/) <br>
129
+ Clean code with pre-commit hooks: [`flake8`](https://flake8.pycqa.org/en/latest/) and
130
+ [`isort`](https://pycqa.github.io/isort/)
131
+
132
+ ## [Release Notes][release-notes]
133
+ **Requirement**
134
+ ```shell
135
+ python -m pip install gitverse
136
+ ```
137
+
138
+ **Usage**
139
+ ```shell
140
+ gitverse-release reverse -f release_notes.rst -t 'Release Notes'
141
+ ```
142
+
143
+ ## Linting
144
+ `pre-commit` will ensure linting, run pytest, generate runbook & release notes, and validate hyperlinks in ALL
145
+ markdown files (including Wiki pages)
146
+
147
+ **Requirement**
148
+ ```shell
149
+ pip install sphinx==5.1.1 pre-commit recommonmark
150
+ ```
151
+
152
+ **Usage**
153
+ ```shell
154
+ pre-commit run --all-files
155
+ ```
156
+
157
+ ## Pypi Package
158
+ [![pypi-module][label-pypi-package]][pypi-repo]
159
+
160
+ [https://pypi.org/project/PyS3Uploader/][pypi]
161
+
162
+ ## Runbook
163
+ [![made-with-sphinx-doc][label-sphinx-doc]][sphinx]
164
+
165
+ [https://thevickypedia.github.io/PyS3Uploader/][runbook]
166
+
167
+ ## License & copyright
168
+
169
+ &copy; Vignesh Rao
170
+
171
+ Licensed under the [MIT License][license]
172
+
173
+ [license]: https://github.com/thevickypedia/PyS3Uploader/blob/main/LICENSE
174
+ [release-notes]: https://github.com/thevickypedia/PyS3Uploader/blob/main/release_notes.rst
175
+ [pypi]: https://pypi.org/project/PyS3Uploader/
176
+ [pypi-tutorials]: https://packaging.python.org/tutorials/packaging-projects/
177
+ [pypi-logo]: https://img.shields.io/badge/Software%20Repository-pypi-1f425f.svg
178
+ [repo]: https://api.github.com/repos/thevickypedia/PyS3Uploader
179
+ [gha-pages-badge]: https://github.com/thevickypedia/PyS3Uploader/actions/workflows/pages/pages-build-deployment/badge.svg
180
+ [gha-pypi-badge]: https://github.com/thevickypedia/PyS3Uploader/actions/workflows/python-publish.yml/badge.svg
181
+ [gha-pages]: https://github.com/thevickypedia/PyS3Uploader/actions/workflows/pages/pages-build-deployment
182
+ [gha-pypi]: https://github.com/thevickypedia/PyS3Uploader/actions/workflows/python-publish.yml
183
+ [sphinx]: https://www.sphinx-doc.org/en/master/man/sphinx-autogen.html
184
+ [label-sphinx-doc]: https://img.shields.io/badge/Made%20with-Sphinx-blue?style=for-the-badge&logo=Sphinx
185
+ [runbook]: https://thevickypedia.github.io/PyS3Uploader/
186
+ [label-pypi-package]: https://img.shields.io/badge/Pypi%20Package-PyS3Uploader-blue?style=for-the-badge&logo=Python
187
+ [pypi-repo]: https://packaging.python.org/tutorials/packaging-projects/
@@ -0,0 +1,11 @@
1
+ s3/__init__.py,sha256=XgYHKbn7gc5_nzydIKmKVjigeMtOBLqRHKHb8GJi5M4,66
2
+ s3/exceptions.py,sha256=hH3jlMOe8yjBatQK9EdndWZz4QESU74KSY_iDhQ37SY,2585
3
+ s3/logger.py,sha256=oH540oq8jY723jA4lDWlgfFPLbNgGXTkDwFpB7TLO_o,1196
4
+ s3/tree.py,sha256=DiQ2ekMMaj2m_P3-iKkEqSuJCJZ_UZxcAwHtAoPVa5c,1824
5
+ s3/uploader.py,sha256=tQaelL7grZSWFydZOekQgVz4Fipm0PHzbt2J17ddYHs,8563
6
+ s3/utils.py,sha256=pKVT2GbDGQKpFaHOmVrCfiQhvgr1vuSsITt_0gHguAA,687
7
+ pys3uploader-0.1.1.dist-info/LICENSE,sha256=8k-hEraOzyum0GvmmK65YxNRTFXK7eIFHJ0OshJXeTk,1068
8
+ pys3uploader-0.1.1.dist-info/METADATA,sha256=sW_fsQxpoZ8f8ivI0Vb4oUXt1RSlFuHJDmpP9h_CXVU,7286
9
+ pys3uploader-0.1.1.dist-info/WHEEL,sha256=beeZ86-EfXScwlR_HKu4SllMC9wUEj_8Z_4FJ3egI2w,91
10
+ pys3uploader-0.1.1.dist-info/top_level.txt,sha256=iQp4y1P58Q633gj8M08kHE4mqqT0hixuDWcniDk_RJ4,3
11
+ pys3uploader-0.1.1.dist-info/RECORD,,
s3/__init__.py CHANGED
@@ -1 +1,3 @@
1
- version = "0.0.0a0"
1
+ from s3.uploader import Uploader # noqa: F401
2
+
3
+ version = "0.1.1"
s3/exceptions.py ADDED
@@ -0,0 +1,82 @@
1
+ """Module to store all the custom exceptions and formatters.
2
+
3
+ >>> S3Error
4
+
5
+ """
6
+
7
+ from typing import Dict, Set
8
+
9
+
10
+ class S3Error(Exception):
11
+ """Custom error for base exception to the PyS3Uploader module."""
12
+
13
+
14
+ class BucketNotFound(S3Error):
15
+ """Custom error for bucket not found."""
16
+
17
+
18
+ class NoObjectFound(S3Error):
19
+ """Custom error for no objects found."""
20
+
21
+
22
+ def convert_to_folder_structure(sequence: Set[str]) -> str:
23
+ """Convert objects in a s3 buckets into a folder like representation.
24
+
25
+ Args:
26
+ sequence: Takes either a mutable or immutable sequence as an argument.
27
+
28
+ Returns:
29
+ str:
30
+ String representation of the architecture.
31
+ """
32
+ folder_structure = {}
33
+ for item in sequence:
34
+ parts = item.split("/")
35
+ current_level = folder_structure
36
+ for part in parts:
37
+ current_level = current_level.setdefault(part, {})
38
+
39
+ def generate_folder_structure(structure: Dict[str, dict], indent: str = "") -> str:
40
+ """Generates the folder like structure.
41
+
42
+ Args:
43
+ structure: Structure of folder objects as key-value pairs.
44
+ indent: Required indentation for the ASCII.
45
+ """
46
+ result = ""
47
+ for i, (key, value) in enumerate(structure.items()):
48
+ if i == len(structure) - 1:
49
+ result += indent + "└── " + key + "\n"
50
+ sub_indent = indent + " "
51
+ else:
52
+ result += indent + "├── " + key + "\n"
53
+ sub_indent = indent + "│ "
54
+ if value:
55
+ result += generate_folder_structure(value, sub_indent)
56
+ return result
57
+
58
+ return generate_folder_structure(folder_structure)
59
+
60
+
61
+ class InvalidPrefix(S3Error):
62
+ """Custom exception for invalid prefix value."""
63
+
64
+ def __init__(self, prefix: str, bucket_name: str, available: Set[str]):
65
+ """Initialize an instance of ``InvalidPrefix`` object inherited from ``S3Error``
66
+
67
+ Args:
68
+ prefix: Prefix to limit the objects.
69
+ bucket_name: Name of the S3 bucket.
70
+ available: Available objects in the s3.
71
+ """
72
+ self.prefix = prefix
73
+ self.bucket_name = bucket_name
74
+ self.available = available
75
+ super().__init__(self.format_error_message())
76
+
77
+ def format_error_message(self):
78
+ """Returns the formatter error message as a string."""
79
+ return (
80
+ f"\n\n\t{self.prefix!r} was not found in {self.bucket_name}.\n\t"
81
+ f"Available: {self.available}\n\n{convert_to_folder_structure(self.available)}"
82
+ )
s3/logger.py ADDED
@@ -0,0 +1,45 @@
1
+ """Loads a default logger with StreamHandler set to DEBUG mode.
2
+
3
+ >>> logging.Logger
4
+
5
+ """
6
+
7
+ import logging
8
+
9
+
10
+ def default_handler() -> logging.StreamHandler:
11
+ """Creates a ``StreamHandler`` and assigns a default format to it.
12
+
13
+ Returns:
14
+ logging.StreamHandler:
15
+ Returns an instance of the ``StreamHandler`` object.
16
+ """
17
+ handler = logging.StreamHandler()
18
+ handler.setFormatter(fmt=default_format())
19
+ return handler
20
+
21
+
22
+ def default_format() -> logging.Formatter:
23
+ """Creates a logging ``Formatter`` with a custom message and datetime format.
24
+
25
+ Returns:
26
+ logging.Formatter:
27
+ Returns an instance of the ``Formatter`` object.
28
+ """
29
+ return logging.Formatter(
30
+ fmt="%(asctime)s - %(levelname)s - [%(module)s:%(lineno)d] - %(funcName)s - %(message)s",
31
+ datefmt="%b-%d-%Y %I:%M:%S %p",
32
+ )
33
+
34
+
35
+ def default_logger() -> logging.Logger:
36
+ """Creates a default logger with debug mode enabled.
37
+
38
+ Returns:
39
+ logging.Logger:
40
+ Returns an instance of the ``Logger`` object.
41
+ """
42
+ logger = logging.getLogger(__name__)
43
+ logger.addHandler(hdlr=default_handler())
44
+ logger.setLevel(level=logging.DEBUG)
45
+ return logger
s3/tree.py ADDED
@@ -0,0 +1,53 @@
1
+ import pathlib
2
+ from typing import List
3
+
4
+
5
+ class Tree:
6
+ """Root tree formatter for a particular directory location.
7
+
8
+ This class allows the creation of a visual representation of the file
9
+ system hierarchy of a specified directory. It can optionally skip files
10
+ that start with a dot (hidden files).
11
+
12
+ >>> Tree
13
+
14
+ """
15
+
16
+ def __init__(self, skip_dot_files: bool):
17
+ """Instantiates the tree object.
18
+
19
+ Args:
20
+ skip_dot_files (bool): If True, skips files with a dot prefix (hidden files).
21
+ """
22
+ self.tree_text = []
23
+ self.skip_dot_files = skip_dot_files
24
+
25
+ def scan(self, path: pathlib.Path, last: bool = True, header: str = "") -> List[str]:
26
+ """Returns contents for a folder as a root tree.
27
+
28
+ Args:
29
+ path: Directory path for which the root tree is to be extracted.
30
+ last: Indicates if the current item is the last in the directory.
31
+ header: The prefix for the current level in the tree structure.
32
+
33
+ Returns:
34
+ List[str]:
35
+ A list of strings representing the directory structure.
36
+ """
37
+ elbow = "└──"
38
+ pipe = "│ "
39
+ tee = "├──"
40
+ blank = " "
41
+ self.tree_text.append(header + (elbow if last else tee) + path.name)
42
+ if path.is_dir():
43
+ children = list(path.iterdir())
44
+ for idx, child in enumerate(children):
45
+ # Skip child file/directory when dot files are supposed to be hidden
46
+ if self.skip_dot_files and child.name.startswith("."):
47
+ continue
48
+ self.scan(
49
+ child,
50
+ header=header + (blank if last else pipe),
51
+ last=idx == len(children) - 1,
52
+ )
53
+ return self.tree_text
s3/uploader.py ADDED
@@ -0,0 +1,202 @@
1
+ import logging
2
+ import os
3
+ import time
4
+ from concurrent.futures import ThreadPoolExecutor, as_completed
5
+ from typing import Dict
6
+
7
+ import boto3.resources.factory
8
+ from botocore.config import Config
9
+ from botocore.exceptions import ClientError
10
+ from tqdm import tqdm
11
+
12
+ from s3.exceptions import BucketNotFound
13
+ from s3.logger import default_logger
14
+ from s3.utils import UploadResults, getenv, urljoin
15
+
16
+
17
+ class Uploader:
18
+ """Initiates Uploader object to upload entire directory to S3.
19
+
20
+ >>> Uploader
21
+
22
+ """
23
+
24
+ RETRY_CONFIG: Config = Config(retries={"max_attempts": 10, "mode": "standard"})
25
+
26
+ def __init__(
27
+ self,
28
+ bucket_name: str,
29
+ upload_dir: str,
30
+ s3_prefix: str = None,
31
+ exclude_path: str = None,
32
+ region_name: str = None,
33
+ profile_name: str = None,
34
+ aws_access_key_id: str = None,
35
+ aws_secret_access_key: str = None,
36
+ logger: logging.Logger = None,
37
+ ):
38
+ """Initiates all the necessary args and creates a boto3 session with retry logic.
39
+
40
+ Args:
41
+ bucket_name: Name of the bucket.
42
+ upload_dir: Full path of the directory to be uploaded.
43
+ s3_prefix: Particular bucket prefix within which the upload should happen.
44
+ exclude_path: Full directory path to exclude from S3 object prefix.
45
+ region_name: Name of the AWS region.
46
+ profile_name: AWS profile name.
47
+ aws_access_key_id: AWS access key ID.
48
+ aws_secret_access_key: AWS secret access key.
49
+ logger: Bring your own logger.
50
+
51
+ See Also:
52
+ exclude_path:
53
+ When upload directory is "/home/ubuntu/Desktop/S3Upload", each file will naturally have the full prefix.
54
+ However, this behavior can be avoided by specifying the ``exclude_path`` parameter.
55
+
56
+ If exclude_path is set to: ``/home/ubuntu/Desktop``, then the file path
57
+ ``/home/ubuntu/Desktop/S3Upload/sub-dir/photo.jpg`` will be uploaded as ``S3Upload/sub-dir/photo.jpg``
58
+
59
+ s3_prefix:
60
+ If provided, ``s3_prefix`` will always be attached to each object.
61
+
62
+ If ``s3_prefix`` is set to: ``2025``, then the file path
63
+ ``/home/ubuntu/Desktop/S3Upload/sub/photo.jpg`` will be uploaded as ``2025/S3Upload/sub/photo.jpg``
64
+ """
65
+ self.session = boto3.Session(
66
+ profile_name=profile_name or getenv("PROFILE_NAME"),
67
+ region_name=region_name or getenv("AWS_DEFAULT_REGION"),
68
+ aws_access_key_id=aws_access_key_id or getenv("AWS_ACCESS_KEY_ID"),
69
+ aws_secret_access_key=aws_secret_access_key or getenv("AWS_SECRET_ACCESS_KEY"),
70
+ )
71
+ self.s3 = self.session.resource(service_name="s3", config=self.RETRY_CONFIG)
72
+ self.logger = logger or default_logger()
73
+ self.upload_dir = upload_dir or getenv("UPLOAD_DIR", "UPLOAD_SOURCE")
74
+ self.s3_prefix = s3_prefix
75
+ self.exclude_path = exclude_path
76
+ self.bucket_name = bucket_name
77
+ # noinspection PyUnresolvedReferences
78
+ self.bucket: boto3.resources.factory.s3.Bucket = None
79
+ self.results = UploadResults()
80
+ self.start = time.time()
81
+
82
+ def init(self) -> None:
83
+ """Instantiates the bucket instance.
84
+
85
+ Raises:
86
+ ValueError: If no bucket name was passed.
87
+ BucketNotFound: If bucket name was not found.
88
+ """
89
+ self.start = time.time()
90
+ if self.exclude_path and self.exclude_path not in self.upload_dir:
91
+ raise ValueError(
92
+ f"\n\n\tStart folder {self.exclude_path!r} is not a part of upload directory {self.upload_dir!r}"
93
+ )
94
+ if not self.upload_dir:
95
+ raise ValueError("\n\n\tCannot proceed without an upload directory.")
96
+ try:
97
+ assert os.path.exists(self.upload_dir)
98
+ except AssertionError:
99
+ raise ValueError(f"\n\n\tPath not found: {self.upload_dir}")
100
+ buckets = [bucket.name for bucket in self.s3.buckets.all()]
101
+ if not self.bucket_name:
102
+ raise ValueError(f"\n\n\tCannot proceed without a bucket name.\n\tAvailable: {buckets}")
103
+ _account_id, _alias = self.session.resource(service_name="iam").CurrentUser().arn.split("/")
104
+ if self.bucket_name not in buckets:
105
+ raise BucketNotFound(f"\n\n\t{self.bucket_name} was not found in {_alias} account.\n\tAvailable: {buckets}")
106
+ self.upload_dir = os.path.abspath(self.upload_dir)
107
+ # noinspection PyUnresolvedReferences
108
+ self.bucket: boto3.resources.factory.s3.Bucket = self.s3.Bucket(self.bucket_name)
109
+
110
+ def exit(self) -> None:
111
+ """Exits after printing results, and run time."""
112
+ total = self.results.success + self.results.failed
113
+ self.logger.info(
114
+ "Total number of uploads: %d, success: %d, failed: %d", total, self.results.success, self.results.failed
115
+ )
116
+ self.logger.info("Run Time: %.2fs", time.time() - self.start)
117
+
118
+ def _uploader(self, objectpath: str, filepath: str) -> None:
119
+ """Uploads the filepath to the specified S3 bucket.
120
+
121
+ Args:
122
+ objectpath: Object path ref in S3.
123
+ filepath: Filepath to upload.
124
+ """
125
+ self.bucket.upload_file(filepath, objectpath)
126
+
127
+ def _get_files(self) -> Dict[str, str]:
128
+ """Get a mapping for all the file path and object paths in upload directory.
129
+
130
+ Returns:
131
+ Dict[str, str]:
132
+ Returns a dictionary object path and filepath.
133
+ """
134
+ files_to_upload = {}
135
+ for __path, __directory, __files in os.walk(self.upload_dir):
136
+ for file_ in __files:
137
+ file_path = os.path.join(__path, file_)
138
+ if self.exclude_path:
139
+ file_path = file_path.replace(self.exclude_path, "")
140
+ # Lists in python are ordered, so s3 prefix will get loaded first when provided
141
+ url_parts = []
142
+ if self.s3_prefix:
143
+ url_parts.extend(
144
+ self.s3_prefix.split(os.sep) if os.sep in self.s3_prefix else self.s3_prefix.split("/")
145
+ )
146
+ # Add rest of the file path to parts before normalizing as an S3 object URL
147
+ url_parts.extend(file_path.split(os.sep))
148
+ # Remove falsy values using filter - "None", "bool", "len" or "lambda item: item"
149
+ object_path = urljoin(*filter(None, url_parts))
150
+ files_to_upload[object_path] = file_path
151
+ return files_to_upload
152
+
153
+ def run(self) -> None:
154
+ """Initiates object upload in a traditional loop."""
155
+ self.init()
156
+ keys = self._get_files()
157
+ self.logger.debug(keys)
158
+ self.logger.info("%d files from '%s' will be uploaded to '%s'", len(keys), self.upload_dir, self.bucket_name)
159
+ self.logger.info("Initiating upload process.")
160
+ for objectpath, filepath in tqdm(
161
+ keys.items(), total=len(keys), unit="file", leave=True, desc=f"Uploading files from {self.upload_dir}"
162
+ ):
163
+ try:
164
+ self._uploader(objectpath=objectpath, filepath=filepath)
165
+ self.results.success += 1
166
+ except ClientError as error:
167
+ self.logger.error(error)
168
+ self.results.failed += 1
169
+ self.exit()
170
+
171
+ def run_in_parallel(self, max_workers: int = 5) -> None:
172
+ """Initiates upload in multi-threading.
173
+
174
+ Args:
175
+ max_workers: Number of maximum threads to use.
176
+ """
177
+ self.init()
178
+ keys = self._get_files()
179
+ self.logger.debug(keys)
180
+ self.logger.info(
181
+ "%d files from '%s' will be uploaded to '%s' with maximum concurrency of: %d",
182
+ len(keys),
183
+ self.upload_dir,
184
+ self.bucket_name,
185
+ max_workers,
186
+ )
187
+ with ThreadPoolExecutor(max_workers=max_workers) as executor:
188
+ futures = [executor.submit(self._uploader, *kv) for kv in keys.items()]
189
+ for future in tqdm(
190
+ iterable=as_completed(futures),
191
+ total=len(futures),
192
+ desc=f"Uploading files to {self.bucket_name}",
193
+ unit="files",
194
+ leave=True,
195
+ ):
196
+ try:
197
+ future.result()
198
+ self.results.success += 1
199
+ except ClientError as error:
200
+ self.logger.error(f"Upload failed: {error}")
201
+ self.results.failed += 1
202
+ self.exit()
s3/utils.py ADDED
@@ -0,0 +1,30 @@
1
+ import os
2
+
3
+
4
+ class UploadResults(dict):
5
+ """Object to store results of S3 upload.
6
+
7
+ >>> UploadResults
8
+
9
+ """
10
+
11
+ success: int = 0
12
+ failed: int = 0
13
+
14
+
15
+ def getenv(*args, default: str = None) -> str:
16
+ """Returns the key-ed environment variable or the default value."""
17
+ for key in args:
18
+ if value := os.environ.get(key.upper()) or os.environ.get(key.lower()):
19
+ return value
20
+ return default
21
+
22
+
23
+ def urljoin(*args) -> str:
24
+ """Joins given arguments into a url. Trailing but not leading slashes are stripped for each argument.
25
+
26
+ Returns:
27
+ str:
28
+ Joined url.
29
+ """
30
+ return "/".join(map(lambda x: str(x).rstrip("/").lstrip("/"), args))
@@ -1,49 +0,0 @@
1
- Metadata-Version: 2.2
2
- Name: PyS3Uploader
3
- Version: 0.0.0a0
4
- Summary: Python module to upload objects to an S3 bucket.
5
- Author-email: Vignesh Rao <svignesh1793@gmail.com>
6
- License: MIT License
7
-
8
- Copyright (c) 2025 Vignesh Rao
9
-
10
- Permission is hereby granted, free of charge, to any person obtaining a copy
11
- of this software and associated documentation files (the "Software"), to deal
12
- in the Software without restriction, including without limitation the rights
13
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
- copies of the Software, and to permit persons to whom the Software is
15
- furnished to do so, subject to the following conditions:
16
-
17
- The above copyright notice and this permission notice shall be included in all
18
- copies or substantial portions of the Software.
19
-
20
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
26
- SOFTWARE.
27
-
28
- Project-URL: Homepage, https://github.com/thevickypedia/s3-uploader
29
- Project-URL: Docs, https://thevickypedia.github.io/s3-uploader/
30
- Project-URL: Source, https://github.com/thevickypedia/s3-uploader
31
- Project-URL: Bug Tracker, https://github.com/thevickypedia/s3-uploader/issues
32
- Keywords: s3
33
- Classifier: Development Status :: 1 - Planning
34
- Classifier: Intended Audience :: Information Technology
35
- Classifier: Operating System :: OS Independent
36
- Classifier: Programming Language :: Python :: 3.9
37
- Classifier: Topic :: Internet :: File Transfer Protocol (FTP)
38
- Requires-Python: >=3.8
39
- Description-Content-Type: text/markdown
40
- License-File: LICENSE
41
- Requires-Dist: boto3
42
- Requires-Dist: tqdm
43
- Provides-Extra: dev
44
- Requires-Dist: sphinx==5.1.1; extra == "dev"
45
- Requires-Dist: pre-commit; extra == "dev"
46
- Requires-Dist: recommonmark; extra == "dev"
47
-
48
- # s3-uploader
49
- Upload objects to S3
@@ -1,6 +0,0 @@
1
- s3/__init__.py,sha256=wq6ADf1YC8sccYtXrSjPGhw7W-AHqruT82iKIaMcttM,20
2
- pys3uploader-0.0.0a0.dist-info/LICENSE,sha256=8k-hEraOzyum0GvmmK65YxNRTFXK7eIFHJ0OshJXeTk,1068
3
- pys3uploader-0.0.0a0.dist-info/METADATA,sha256=nWmw-uD4ok-IyS1eXHgjINPF6DKO3cJ4xSqgGzg_f_8,2277
4
- pys3uploader-0.0.0a0.dist-info/WHEEL,sha256=beeZ86-EfXScwlR_HKu4SllMC9wUEj_8Z_4FJ3egI2w,91
5
- pys3uploader-0.0.0a0.dist-info/top_level.txt,sha256=iQp4y1P58Q633gj8M08kHE4mqqT0hixuDWcniDk_RJ4,3
6
- pys3uploader-0.0.0a0.dist-info/RECORD,,