yes3 0.0.1__tar.gz

yes3-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Erik Schomburg
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

yes3-0.0.1/PKG-INFO

@@ -0,0 +1,145 @@
+Metadata-Version: 2.4
+Name: yes3
+Version: 0.0.1
+Summary: A simple, intuitive, flexible interface for Amazon S3
+Author-email: Erik Schomburg <eschomburg@gmail.com>
+Project-URL: Homepage, https://github.com/eschombu/yes3
+Project-URL: Issues, https://github.com/eschombu/yes3/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: boto3
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: tqdm
+Provides-Extra: dev
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: moto; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# `yes3`
+
+## A library for intuitive reading, writing, listing, and caching with AWS S3 (Simple Storage Service).
+
+This library wraps the `boto3` S3 API boilerplate with a simple and intuitive interface, path flexibility, and powerful
+utilities for easily listing, reading, and writing data on/from/to S3.
+
+## Installation
+
+Using a virtual environment is recommended. The simplest way to install is with pip: `pip install yes3`. To install the
+latest version, you can install from github: `pip install git+https://github.com/eschombu/yes3.git`.
+
+To run tests and test scripts, and/or contribute to `yes3`, clone this repository from
+https://github.com/eschombu/yes3.git, and install the dev requirements:
+```
+git clone https://github.com/eschombu/yes3.git
+cd yes3
+# Optionally create a virtual environment:
+# python3.1x -m venv .venv/yes3
+# source .venv/yes3/bin/activate
+pip install -e .[dev]
+pytest
+```
+
+## TODO
+
+1. Documentation
+2. Replace message printing with loggers
+
+## Usage
+
+### S3 Locations and Paths
+
+The `boto3` APIs for S3 typically consider the 'bucket' and 'key' of an S3 object:
+```
+import boto3
+s3_client = boto3.client('s3')
+s3_client.download_file('my-bucket', 'key/to/object', 'path/to/local/file')
+```
+
+The awscli uses urls:
+```
+aws s3 cp s3://my-bucket/key/to/object path/to/local/file
+```
+
+In `yes3`, we accept either, attempting to flexibly interpret input arguments as S3 locations and local paths,
+converting S3 locations into `S3Location` objects:
+```
+from yes3 import s3, S3Location
+
+# The following download calls are equivalent
+s3.download('s3://my-bucket/key/to/object', 'path/to/local/file')
+s3.download('my-bucket', 'key/to/object', 'path/to/local/file')
+
+s3_loc = S3Location('s3://my-bucket/key/to/object')
+print(s3_loc.bucket)  # 'my-bucket'
+print(s3_loc.key)  # 'key/to/object'
+print(s3_loc.exists())  # True
+print(s3_loc.is_bucket())  # False
+print(s3_loc.is_dir())  # False
+print(s3_loc.is_object())  # True
+s3.download(s3_loc, 'path/to/local/file')
+```
+
+If the local path is to a directory, the object will be downloaded with the filename inferred from the S3 path.
+Recursive downloads are also supported.
+```
+s3_dir = S3Location('s3://my-bucket/path/to/dir')
+print(s3_dir.is_dir())  # True
+print(s3_dir.is_object())  # False
+s3.download(s3_dir, 'local_dir/')  # raises ValueError because s3_dir is not a single S3 object
+s3.download(s3_dir, 'local_dir/', recursive=True)  # downloads all objects to the `local_dir` directory (which is created if it does not already exist)
+```
+
+Direct read/write functions are also available: `s3.read`, `s3.write_to_s3` (which actually creates a local temp file,
+which is removed afterwards), and `s3.touch`.
+
+Convenient object and directory listing methods are available:
+* `s3.list_objects`: list all objects with the specified prefix
+* `s3.list_dir`: List objects and directories only up to the specified depth (default: 1). S3 does not actually have a
+directory structure, but this function works as if it does.
+
+### Easy key-based caching utilities, for local, S3, and multi-location caches
+
+To quickly and easily cache data, and allow for such a cache to be synced across devices, this package includes `Cache`
+classes, which include `LocalDiskCache` and `S3Cache` subclasses, as well as a `MultiCache` which can utilize multiple
+cache locations. Caching is key-value based, with customizable serializers that can store objects with `pickle` or
+alternative data/file formats.
+
+A helper function, `setup_cache`, provides a simple interface to create a `Cache` object with the default
+`PickleSerializer` serializer:
+
+```
+from yes3.caching import setup_cache
+
+local_cache = setup_cache('path/to/cache/dir')
+s3_cache = setup_cache('s3://my-bucket/cache/dir/prefix')
+
+if 'data' in s3_cache:
+    data = s3_cache['data']
+else:
+    data = expensive_data_processing(args)
+if 'data' not in local_cache:
+    local_cache['data'] = data
+
+multi_cache = MultiCache([local_cache, s3_cache])
+multi_cache.sync_now()  # Add any data missing found in either cache to the one in which it is missing
+multi_cache.sync_always()  # Keep the caches synced moving forward
+
+new_data = get_more_data()
+multi_cache.put('new_data', new_data)
+print('new_data' in local_cache)  # True
+print('new_data' in s3_cache)  # True
+
+from yes3 import s3
+for loc in s3.list_objects(s3_cache.path):
+    print(loc.s3_uri)
+# 's3://my-bucket/cache/dir/prefix/data.meta'
+# 's3://my-bucket/cache/dir/prefix/data.pkl'
+# 's3://my-bucket/cache/dir/prefix/new_data.meta'
+# 's3://my-bucket/cache/dir/prefix/new_data.pkl'
+```

yes3-0.0.1/README.md

@@ -0,0 +1,122 @@
+# `yes3`
+
+## A library for intuitive reading, writing, listing, and caching with AWS S3 (Simple Storage Service).
+
+This library wraps the `boto3` S3 API boilerplate with a simple and intuitive interface, path flexibility, and powerful
+utilities for easily listing, reading, and writing data on/from/to S3.
+
+## Installation
+
+Using a virtual environment is recommended. The simplest way to install is with pip: `pip install yes3`. To install the
+latest version, you can install from github: `pip install git+https://github.com/eschombu/yes3.git`.
+
+To run tests and test scripts, and/or contribute to `yes3`, clone this repository from
+https://github.com/eschombu/yes3.git, and install the dev requirements:
+```
+git clone https://github.com/eschombu/yes3.git
+cd yes3
+# Optionally create a virtual environment:
+# python3.1x -m venv .venv/yes3
+# source .venv/yes3/bin/activate
+pip install -e .[dev]
+pytest
+```
+
+## TODO
+
+1. Documentation
+2. Replace message printing with loggers
+
+## Usage
+
+### S3 Locations and Paths
+
+The `boto3` APIs for S3 typically consider the 'bucket' and 'key' of an S3 object:
+```
+import boto3
+s3_client = boto3.client('s3')
+s3_client.download_file('my-bucket', 'key/to/object', 'path/to/local/file')
+```
+
+The awscli uses urls:
+```
+aws s3 cp s3://my-bucket/key/to/object path/to/local/file
+```
+
+In `yes3`, we accept either, attempting to flexibly interpret input arguments as S3 locations and local paths,
+converting S3 locations into `S3Location` objects:
+```
+from yes3 import s3, S3Location
+
+# The following download calls are equivalent
+s3.download('s3://my-bucket/key/to/object', 'path/to/local/file')
+s3.download('my-bucket', 'key/to/object', 'path/to/local/file')
+
+s3_loc = S3Location('s3://my-bucket/key/to/object')
+print(s3_loc.bucket)  # 'my-bucket'
+print(s3_loc.key)  # 'key/to/object'
+print(s3_loc.exists())  # True
+print(s3_loc.is_bucket())  # False
+print(s3_loc.is_dir())  # False
+print(s3_loc.is_object())  # True
+s3.download(s3_loc, 'path/to/local/file')
+```
+
+If the local path is to a directory, the object will be downloaded with the filename inferred from the S3 path.
+Recursive downloads are also supported.
+```
+s3_dir = S3Location('s3://my-bucket/path/to/dir')
+print(s3_dir.is_dir())  # True
+print(s3_dir.is_object())  # False
+s3.download(s3_dir, 'local_dir/')  # raises ValueError because s3_dir is not a single S3 object
+s3.download(s3_dir, 'local_dir/', recursive=True)  # downloads all objects to the `local_dir` directory (which is created if it does not already exist)
+```
+
+Direct read/write functions are also available: `s3.read`, `s3.write_to_s3` (which actually creates a local temp file,
+which is removed afterwards), and `s3.touch`.
+
+Convenient object and directory listing methods are available:
+* `s3.list_objects`: list all objects with the specified prefix
+* `s3.list_dir`: List objects and directories only up to the specified depth (default: 1). S3 does not actually have a
+directory structure, but this function works as if it does.
+
+### Easy key-based caching utilities, for local, S3, and multi-location caches
+
+To quickly and easily cache data, and allow for such a cache to be synced across devices, this package includes `Cache`
+classes, which include `LocalDiskCache` and `S3Cache` subclasses, as well as a `MultiCache` which can utilize multiple
+cache locations. Caching is key-value based, with customizable serializers that can store objects with `pickle` or
+alternative data/file formats.
+
+A helper function, `setup_cache`, provides a simple interface to create a `Cache` object with the default
+`PickleSerializer` serializer:
+
+```
+from yes3.caching import setup_cache
+
+local_cache = setup_cache('path/to/cache/dir')
+s3_cache = setup_cache('s3://my-bucket/cache/dir/prefix')
+
+if 'data' in s3_cache:
+    data = s3_cache['data']
+else:
+    data = expensive_data_processing(args)
+if 'data' not in local_cache:
+    local_cache['data'] = data
+
+multi_cache = MultiCache([local_cache, s3_cache])
+multi_cache.sync_now()  # Add any data missing found in either cache to the one in which it is missing
+multi_cache.sync_always()  # Keep the caches synced moving forward
+
+new_data = get_more_data()
+multi_cache.put('new_data', new_data)
+print('new_data' in local_cache)  # True
+print('new_data' in s3_cache)  # True
+
+from yes3 import s3
+for loc in s3.list_objects(s3_cache.path):
+    print(loc.s3_uri)
+# 's3://my-bucket/cache/dir/prefix/data.meta'
+# 's3://my-bucket/cache/dir/prefix/data.pkl'
+# 's3://my-bucket/cache/dir/prefix/new_data.meta'
+# 's3://my-bucket/cache/dir/prefix/new_data.pkl'
+```

yes3-0.0.1/pyproject.toml

@@ -0,0 +1,41 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "yes3"
+version = "0.0.1"
+authors = [
+  { name="Erik Schomburg", email="eschomburg@gmail.com" },
+]
+description = "A simple, intuitive, flexible interface for Amazon S3"
+readme = "README.md"
+requires-python = ">=3.8"
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+  "boto3",
+  "numpy",
+  "pandas",
+  "tqdm",
+]
+
+[project.optional-dependencies]
+dev = [
+    "flake8",
+    "moto",
+    "pytest",
+]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests"]
+
+[project.urls]
+Homepage = "https://github.com/eschombu/yes3"
+Issues = "https://github.com/eschombu/yes3/issues"
+
+

yes3-0.0.1/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

yes3-0.0.1/src/yes3/__init__.py

@@ -0,0 +1 @@
+from .s3 import S3Location, is_s3_url

yes3-0.0.1/src/yes3/caching/__init__.py

@@ -0,0 +1,6 @@
+from .base import Cache, CacheCore, CachedItemMeta, Serializer, check_meta_mismatches
+from .local_cache import LocalDiskCache
+from .memory_cache import MemoryCache
+from .multi_cache import MultiCache
+from .s3_cache import S3Cache
+from .setup_helpers import setup_cache

yes3-0.0.1/src/yes3/caching/base.py

@@ -0,0 +1,302 @@
+from abc import ABCMeta, abstractmethod
+from collections.abc import Callable
+from dataclasses import dataclass
+from datetime import datetime, UTC
+from typing import Iterable, Iterator, Optional, Self
+
+_NotSpecified = object()
+
+
+def raise_not_found(key) -> KeyError:
+    raise KeyError(f"key '{key}' not found in cache")
+
+
+@dataclass
+class CachedItemMeta:
+    key: str
+    path: Optional[str]
+    size: Optional[int]
+    timestamp: Optional[datetime]
+
+    _ts_format = '%Y-%m-%d %H:%M:%S.%f %z'
+
+    def __post_init__(self):
+        if isinstance(self.timestamp, float):
+            self.timestamp = datetime.fromtimestamp(self.timestamp, UTC)
+        if isinstance(self.timestamp, str):
+            self.timestamp = datetime.strptime(self.timestamp, self._ts_format)
+
+    def to_dict(self) -> dict:
+        return {
+            'key': self.key,
+            'path': self.path,
+            'size': self.size,
+            'timestamp': self.timestamp.strftime(self._ts_format) if self.timestamp else None,
+        }
+
+
+class CacheCore(metaclass=ABCMeta):
+    def __init__(self, active=True, read_only=False):
+        self._read_only = read_only
+        self._active = active
+
+    @abstractmethod
+    def __contains__(self, key):
+        pass
+
+    @abstractmethod
+    def get(self, key, default=_NotSpecified):
+        pass
+
+    @abstractmethod
+    def get_meta(self, key) -> CachedItemMeta:
+        pass
+
+    @abstractmethod
+    def put(self, key, obj, update=False, meta: Optional[CachedItemMeta] = None):
+        pass
+
+    @abstractmethod
+    def remove(self, key):
+        pass
+
+    @abstractmethod
+    def keys(self):
+        pass
+
+    def __getitem__(self, key: str):
+        return self.get(key)
+
+    def __setitem__(self, key: str, obj) -> None:
+        self.put(key, obj)
+
+    def __delitem__(self, key: str) -> None:
+        self.remove(key)
+
+    def is_active(self) -> bool:
+        return self._active
+
+    def activate(self):
+        self._active = True
+        return self
+
+    def deactivate(self):
+        self._active = False
+        return self
+
+    def is_read_only(self) -> bool:
+        return self._read_only
+
+    def set_read_only(self, value: bool) -> Self:
+        self._read_only = value
+        return self
+
+    def update(self, key: str, obj):
+        if key not in self:
+            raise_not_found(key)
+        self.put(key, obj, update=True)
+
+    def pop(self, key: str, default=_NotSpecified):
+        obj = self.get(key, default=default)
+        self.remove(key)
+        return obj
+
+    def list(self) -> dict[str, CachedItemMeta]:
+        items_meta = {}
+        for key in self.keys():
+            items_meta[key] = self.get_meta(key)
+        return items_meta
+
+    def subcache(self, *args, **kwargs) -> Self:
+        raise NotImplementedError(f"`subcache` method is not defined for class {type(self).__name__}")
+
+
+class CacheReaderWriter(metaclass=ABCMeta):
+    @abstractmethod
+    def read(self, key: str):
+        pass
+
+    @abstractmethod
+    def get_meta(self, key: str) -> CachedItemMeta:
+        pass
+
+    @abstractmethod
+    def write(self, key: str, obj, meta=None) -> CachedItemMeta:
+        pass
+
+    @abstractmethod
+    def delete(self, key: str, meta_only=False):
+        pass
+
+
+class CacheCatalog(metaclass=ABCMeta):
+    @abstractmethod
+    def contains(self, key: str):
+        pass
+
+    @abstractmethod
+    def add(self, key: str, info: CachedItemMeta):
+        pass
+
+    @abstractmethod
+    def get(self, key: str) -> CachedItemMeta:
+        pass
+
+    @abstractmethod
+    def remove(self, key: str):
+        pass
+
+    @abstractmethod
+    def keys(self):
+        pass
+
+    @abstractmethod
+    def items(self):
+        pass
+
+
+_CatalogT = dict[str, CachedItemMeta]
+_CatalogBuilderT = Callable[[], _CatalogT]
+
+
+class CacheDictCatalog(CacheCatalog):
+    def __init__(
+            self,
+            catalog: Optional[dict[str, CachedItemMeta]] = None,
+            catalog_builder: Optional[_CatalogBuilderT] = None,
+    ):
+        self._catalog = catalog
+        if catalog_builder is None:
+            catalog_builder = dict
+        self._build_catalog = catalog_builder
+        if self._catalog is None:
+            self.rebuild()
+
+    def rebuild(self):
+        self._catalog = self._build_catalog().copy()
+
+    def contains(self, key: str):
+        return str(key) in self._catalog
+
+    def add(self, key: str, meta: CachedItemMeta):
+        self._catalog[str(key)] = meta
+
+    def get(self, key: str) -> CachedItemMeta:
+        return self._catalog[str(key)]
+
+    def remove(self, key: str):
+        self._catalog.pop(str(key))
+
+    def keys(self):
+        return list(self._catalog.keys())
+
+    def items(self) -> Iterator[tuple[str, CachedItemMeta]]:
+        return iter(self._catalog.items())
+
+
+class Cache(CacheCore, metaclass=ABCMeta):
+    def __init__(self, catalog: CacheCatalog, reader_writer: CacheReaderWriter, active=True, read_only=False):
+        super().__init__(active=active, read_only=read_only)
+        self._catalog = catalog
+        self._reader_writer = reader_writer
+
+    @classmethod
+    @abstractmethod
+    def create(cls, *args, **kwargs):
+        pass
+
+    def __contains__(self, key: str) -> bool:
+        if not self.is_active():
+            return False
+        return self._catalog.contains(key)
+
+    def get(self, key: str, default=_NotSpecified):
+        if not self.is_active() or key not in self:
+            if default is _NotSpecified:
+                raise_not_found(key)
+            else:
+                return default
+        return self._reader_writer.read(key)
+
+    def get_meta(self, key: str) -> CachedItemMeta:
+        if not self.is_active() or key not in self:
+            raise_not_found(key)
+        return self._catalog.get(key)
+
+    def put(self, key: str, obj, *, update=False, meta: Optional[CachedItemMeta] = None) -> Self:
+        if self.is_read_only():
+            raise TypeError('Cache is in read only mode')
+        if self.is_active():
+            if key in self and not update:
+                raise ValueError(f"key '{key}' already exists in cache; use 'update' to overwrite")
+            meta = self._reader_writer.write(key, obj, meta=meta)
+            self._catalog.add(key, meta)
+        else:
+            print(f'WARNING: {type(self).__name__} is not active')
+        return self
+
+    def remove(self, key: str, meta_only=False) -> Self:
+        if self.is_active() and key in self:
+            if self.is_read_only():
+                raise TypeError('Cache is in read only mode')
+            self._catalog.remove(key)
+            self._reader_writer.delete(key, meta_only=meta_only)
+        return self
+
+    def remove_meta(self, key: str) -> Self:
+        return self.remove(key, meta_only=True)
+
+    def keys(self) -> list[str]:
+        if not self.is_active():
+            return []
+        else:
+            return list(self._catalog.keys())
+
+    def _repr_params(self) -> list[str]:
+        params = [f'{len(self.keys())} items']
+        if not self.is_active():
+            params.append('NOT ACTIVE')
+        if self.is_read_only():
+            params.append('READ ONLY')
+        return params
+
+    def __repr__(self):
+        return f"{type(self).__name__}({', '.join(self._repr_params())})"
+
+
+def check_meta_mismatches(caches: Iterable[CacheCore], key=None) -> dict[str, tuple[CachedItemMeta, ...]]:
+    if key is not None and not isinstance(key, str):
+        raise TypeError('key is not a string')
+    for cache in caches:
+        if not isinstance(cache, CacheCore):
+            raise TypeError('caches must be an iterable containing Cache instances')
+    mismatches = {}
+    if key is None:
+        keys = set(k for cache in caches for k in cache.keys())
+    else:
+        keys = [key]
+    for k in keys:
+        metas = [cache.get_meta(k) for cache in caches if k in cache]
+        if len(metas) > 1:
+            first_meta = metas[0]
+            if any(meta != first_meta for meta in metas[1:]):
+                mismatches[k] = tuple(metas)
+    return mismatches
+
+
+class Serializer(metaclass=ABCMeta):
+    default_ext = None
+
+    def __init__(self, ext=None):
+        if ext is None:
+            self.ext = self.default_ext
+        else:
+            self.ext = ext
+
+    @abstractmethod
+    def read(self, path):
+        pass
+
+    @abstractmethod
+    def write(self, path, obj):
+        pass