asanypath 2.2.0__tar.gz → 2.3.1__tar.gz

{asanypath-2.2.0 → asanypath-2.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: asanypath
-Version: 2.2.0
+Version: 2.3.1
 Summary: Async pathlib for local and cloud storage
 License-Expression: MIT
 License-File: LICENSE
@@ -19,12 +19,11 @@ Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.10
 Requires-Dist: anyio>=4
-Requires-Dist: asanypath-native~=2.2.0
+Requires-Dist: asanypath-native~=2.3.0
 Requires-Dist: msgspec>=0.22.0
 Requires-Dist: yarl>=1.2
 Provides-Extra: all
 Requires-Dist: aioftp>=0.21; extra == 'all'
-Requires-Dist: diskcache>=5.6; extra == 'all'
 Requires-Dist: keyring>=24; extra == 'all'
 Provides-Extra: bench
 Requires-Dist: adlfs; extra == 'bench'
@@ -32,21 +31,19 @@ Requires-Dist: aiobotocore; extra == 'bench'
 Requires-Dist: aiofiles; extra == 'bench'
 Requires-Dist: asyncssh>=2.14; extra == 'bench'
 Requires-Dist: azure-storage-blob; extra == 'bench'
+Requires-Dist: boto3; extra == 'bench'
 Requires-Dist: cloudpathlib[all]; extra == 'bench'
 Requires-Dist: fsspec; extra == 'bench'
 Requires-Dist: gcloud-aio-storage; extra == 'bench'
 Requires-Dist: gcsfs; extra == 'bench'
 Requires-Dist: requests; extra == 'bench'
 Requires-Dist: s3fs; extra == 'bench'
-Provides-Extra: cache
-Requires-Dist: diskcache>=5.6; extra == 'cache'
 Provides-Extra: cli
 Requires-Dist: rich-click>=1.9.7; extra == 'cli'
 Provides-Extra: credstore
 Requires-Dist: keyring>=24; extra == 'credstore'
 Provides-Extra: dev
 Requires-Dist: aioftp>=0.21; extra == 'dev'
-Requires-Dist: diskcache>=5.6; extra == 'dev'
 Requires-Dist: keyring>=24; extra == 'dev'
 Requires-Dist: pre-commit>=4; extra == 'dev'
 Requires-Dist: pytest-asyncio>=1.3; extra == 'dev'
@@ -149,6 +146,52 @@ await src.copy(dst, recursive=True)
 await path.rmdir(recursive=True)
 ```
 
+Copies automatically use spare transfer concurrency for large ranged reads.
+The default target is 8 streams, with an 8 MiB minimum range size and at most
+8 range streams per file. Files smaller than 16 MiB stay single-stream. When
+enough files are active to meet the target, automatic copies use one request
+per file instead of adding ranges. The target is best-effort, not a minimum
+number of requests that must always be launched.
+
+S3, GCS, Azure, Artifactory, SSH, and range-capable HTTP/HTTPS sources use the
+shared reader for local downloads and source reads in cross-backend copies.
+FTP/FTPS, sources without size metadata, and HTTP servers without advertised
+range support retain single-stream downloads. Local-only copies and native
+server-side copy/rename operations retain their existing implementations.
+Large S3/Azure uploads also use concurrent parts within this shared budget.
+GCS resumable sessions remain sequential because the protocol requires ordered
+offsets.
+
+Concurrent library copies share a default budget within the event loop. To
+configure a batch explicitly:
+
+```python
+import asyncio
+from asanypath import TransferBudget
+
+with TransferBudget(floor=8, ceiling=32, min_range_size=8 * 1024 * 1024, per_file_cap=8):
+    await asyncio.gather(*(src.copy(dst) for src, dst in pairs))
+```
+
+The ceiling bounds concurrent range reads and upload parts in this budget,
+not metadata requests, single-request writes, or other processes. Parallel
+buffers are bounded by active streams times their chunk sizes. Cross-backend
+copies spool the downloaded source to a temporary file before using the
+destination's upload API. S3/Azure then upload concurrent parts; destinations
+without a streaming upload implementation can still buffer the complete write.
+Same-backend server-side copies are unchanged.
+`floor=1` disables automatic range splitting and makes upload parts sequential.
+Explicit `chunk_size` requests retain sequential streaming when splitting
+would be inappropriate, and `chunk_size=0` disables streaming to local paths.
+
+S3/Azure ETags, GCS generations, and HTTP conditional requests pin the source
+version when available. Other sources are checked against fresh metadata
+after the download, which cannot guarantee a snapshot if the backend does
+not expose a reliable change token. Use `atomic=True` to keep an existing
+local destination intact on failure. Moves remove the source only after a
+successful copy. The updated Python package requires the matching native
+extension for conditional cloud reads.
+
 For synchronous code (no `await` needed):
 
 ```python
@@ -182,14 +225,24 @@ async with AsAnyPath("s3://bucket/huge.bin").open("wb") as f:
 await AsAnyPath("/data/movie.mp4").copy("s3://bucket/movie.mp4")
 ```
 
-Peak memory per upload is one part — **8 MiB by default**. Objects smaller than
-one part upload in a single request. (`write_bytes(data)` / `write_text(...)`
+S3 multipart and Azure block uploads send **up to 8 parts concurrently by
+default**, filling freed slots without waiting for a whole batch. Concurrent
+files share the budget rather than each multiplying the target concurrency.
+At the default 8 MiB part size, a single large upload buffers up to **64 MiB of
+part payload**, plus file spooling, transport and runtime overhead. Native part
+requests borrow immutable Python bytes instead of copying each buffer again.
+Objects smaller than one part upload in a single request.
+(`write_bytes(data)` / `write_text(...)`
 take the whole payload in memory by definition — use `open("wb")` or `copy()`
-for large data.)
+for large data; direct `write_bytes()` remains a single write.)
 
-**Tuning the memory footprint.** The part size *is* the max RAM held per
-in-flight chunk (parts upload sequentially, so there is no concurrency
-multiplier). Set it globally via env, or per call:
+**Tuning throughput and memory.** Buffered part payload scales with part size
+times concurrent upload slots. A slot is reserved before reading its part,
+so queued uploads do not allocate unbounded buffers. Tune concurrency with
+`TransferBudget(floor=..., ceiling=..., per_file_cap=...)`, or CLI
+`--concurrency-floor` (also used by `cp`, `mv`, and `sync`). To limit a batch
+to one in-flight part/range, use `TransferBudget(floor=1, ceiling=1)`.
+Set part size globally via env, or per call:
 
 ```bash
 export ASANYPATH_UPLOAD_CHUNK_SIZE=33554432   # 32 MiB parts (bytes)
@@ -208,6 +261,11 @@ Bigger parts mean fewer requests and a larger single-object ceiling (S3 allows
 applied automatically: **S3** clamps to ≥ 5 MiB parts, **GCS** rounds down to a
 256 KiB multiple.
 
+On S3 failure or cancellation, in-flight workers are cancelled and joined
+before aborting the multipart upload; completion lists remain in source order.
+Azure only commits the ordered block list after all stages succeed; abandoned
+uncommitted blocks are left for Azure's automatic expiry.
+
 Unknown URI schemes now construct as `UnsupportedProtocolPath` placeholders.
 Pure path operations (e.g. `.name`, `.parent`, joins) still work, while backend
 operations (e.g. `.exists()`, `.read_bytes()`, `.open()`) raise
@@ -219,9 +277,8 @@ Reading the same object or URL twice always re-downloads it — there is **no
 cache by default**, and the read path is unchanged unless you opt in, so
 non-caching callers pay zero overhead.
 
-Enable it with the optional `cache` extra (`pip install asanypath[cache]`,
-backed by [`diskcache`](https://pypi.org/project/diskcache/) — a pure-Python,
-zero-dependency store). Opt in per read with `use_cache=True`:
+Caching is included in the native extension with no extra dependencies or
+installation extras. Opt in per read with `use_cache=True`:
 
 ```python
 from asanypath import AsAnyPath
@@ -240,7 +297,7 @@ from asanypath import configure_cache
 
 configure_cache(
     directory="/var/cache/asanypath",  # default: ~/.cache/asanypath
-    size_limit=2 * 1024**3,            # total on-disk cap in bytes (default 1 GiB, LRU)
+    size_limit=2 * 1024**3,            # stored key + payload budget (default 1 GiB, LRU)
     ttl=3600,                          # per-entry expiry in seconds (default: none)
     enabled=True,                      # make use_cache default to True process-wide
 )
@@ -255,9 +312,15 @@ Behaviour and trade-offs:
   default flips process-wide; an explicit `use_cache=False` always bypasses.
 - **Works for every remote backend** (S3, GCS, Azure, Artifactory, HTTP, SSH,
   FTP). Local paths are never cached.
-- **Total-size bounded, LRU-evicted;** disk I/O runs in a worker thread so the
+- **Size-bounded, LRU-evicted;** disk I/O runs in a worker thread so the
   event loop stays responsive. Concurrent reads of the same key collapse to a
-  single fetch.
+  single fetch within a cache instance. The byte budget counts UTF-8 keys and
+  payloads; SQLite metadata and transaction journals add filesystem overhead.
+  Objects larger than the budget are returned without being cached.
+- **Byte-only native storage:** SQLite stores raw bytes, never Python pickles.
+  The database lives under `native-v1/` in the configured cache directory.
+  Legacy DiskCache files are ignored, not migrated or deleted. Multiple
+  processes can share the new store; use the same size limit across them.
 - **Staleness is on you.** Entries are keyed by URL with no revalidation — a
   cached object that changes upstream is served stale until it expires (`ttl`)
   or is evicted. Prefer caching immutable/versioned objects, or set a `ttl`.
@@ -356,6 +419,12 @@ asanypath cp /local/file.txt s3://bucket/file.txt
 # Copy cloud → local
 asanypath cp s3://bucket/file.txt /local/file.txt
 
+# Allow 32 concurrent files; fill spare capacity toward 8 streams for large files
+asanypath cp -r s3://bucket/data/ ./data/ -j 32 --concurrency-floor 8
+
+# Disable automatic range splitting (also supported by mv and sync)
+asanypath cp s3://bucket/file.txt ./file.txt --concurrency-floor 1
+
 # Delete a file
 asanypath rm s3://bucket/file.txt --force
 
@@ -463,6 +532,22 @@ uv run --extra bench python scripts/bench.py --backend all --rounds 3 --objects
 uv run --extra bench python scripts/bench.py --backend art --rounds 3 --concurrency-levels 1,10,50
 ```
 
+For multipart upload throughput, use `scripts/bench_upload.py` with real shard
+files and an explicitly selected S3-compatible endpoint:
+
+```bash
+uv run --extra bench python scripts/bench_upload.py \
+  --destination s3://my-bucket/bench --endpoint-url https://my-storage-endpoint \
+  --part-mib 8 --streams 1 8 --rounds 3 /data/shard-001 /data/shard-002
+```
+
+This compares asanypath and boto3 using one shared request budget per client.
+It warms up each case, reports JSON timings and MiB/s, verifies uploaded contents
+outside the timed interval, and deletes only the uniquely named benchmark
+objects it created. Both clients use normal AWS credentials (or `--profile`).
+Run with one large shard and again with several representative shards; results
+from a local test server are not evidence of production endpoint throughput.
+
 ### S3 — live S3-compatible service (median of 20 rounds, 10 objects, c=1)
 
 | Operation | asanypath | s3fs | aiobotocore | cloudpathlib |

{asanypath-2.2.0 → asanypath-2.3.1}/README.md

@@ -90,6 +90,52 @@ await src.copy(dst, recursive=True)
 await path.rmdir(recursive=True)
 ```
 
+Copies automatically use spare transfer concurrency for large ranged reads.
+The default target is 8 streams, with an 8 MiB minimum range size and at most
+8 range streams per file. Files smaller than 16 MiB stay single-stream. When
+enough files are active to meet the target, automatic copies use one request
+per file instead of adding ranges. The target is best-effort, not a minimum
+number of requests that must always be launched.
+
+S3, GCS, Azure, Artifactory, SSH, and range-capable HTTP/HTTPS sources use the
+shared reader for local downloads and source reads in cross-backend copies.
+FTP/FTPS, sources without size metadata, and HTTP servers without advertised
+range support retain single-stream downloads. Local-only copies and native
+server-side copy/rename operations retain their existing implementations.
+Large S3/Azure uploads also use concurrent parts within this shared budget.
+GCS resumable sessions remain sequential because the protocol requires ordered
+offsets.
+
+Concurrent library copies share a default budget within the event loop. To
+configure a batch explicitly:
+
+```python
+import asyncio
+from asanypath import TransferBudget
+
+with TransferBudget(floor=8, ceiling=32, min_range_size=8 * 1024 * 1024, per_file_cap=8):
+    await asyncio.gather(*(src.copy(dst) for src, dst in pairs))
+```
+
+The ceiling bounds concurrent range reads and upload parts in this budget,
+not metadata requests, single-request writes, or other processes. Parallel
+buffers are bounded by active streams times their chunk sizes. Cross-backend
+copies spool the downloaded source to a temporary file before using the
+destination's upload API. S3/Azure then upload concurrent parts; destinations
+without a streaming upload implementation can still buffer the complete write.
+Same-backend server-side copies are unchanged.
+`floor=1` disables automatic range splitting and makes upload parts sequential.
+Explicit `chunk_size` requests retain sequential streaming when splitting
+would be inappropriate, and `chunk_size=0` disables streaming to local paths.
+
+S3/Azure ETags, GCS generations, and HTTP conditional requests pin the source
+version when available. Other sources are checked against fresh metadata
+after the download, which cannot guarantee a snapshot if the backend does
+not expose a reliable change token. Use `atomic=True` to keep an existing
+local destination intact on failure. Moves remove the source only after a
+successful copy. The updated Python package requires the matching native
+extension for conditional cloud reads.
+
 For synchronous code (no `await` needed):
 
 ```python
@@ -123,14 +169,24 @@ async with AsAnyPath("s3://bucket/huge.bin").open("wb") as f:
 await AsAnyPath("/data/movie.mp4").copy("s3://bucket/movie.mp4")
 ```
 
-Peak memory per upload is one part — **8 MiB by default**. Objects smaller than
-one part upload in a single request. (`write_bytes(data)` / `write_text(...)`
+S3 multipart and Azure block uploads send **up to 8 parts concurrently by
+default**, filling freed slots without waiting for a whole batch. Concurrent
+files share the budget rather than each multiplying the target concurrency.
+At the default 8 MiB part size, a single large upload buffers up to **64 MiB of
+part payload**, plus file spooling, transport and runtime overhead. Native part
+requests borrow immutable Python bytes instead of copying each buffer again.
+Objects smaller than one part upload in a single request.
+(`write_bytes(data)` / `write_text(...)`
 take the whole payload in memory by definition — use `open("wb")` or `copy()`
-for large data.)
+for large data; direct `write_bytes()` remains a single write.)
 
-**Tuning the memory footprint.** The part size *is* the max RAM held per
-in-flight chunk (parts upload sequentially, so there is no concurrency
-multiplier). Set it globally via env, or per call:
+**Tuning throughput and memory.** Buffered part payload scales with part size
+times concurrent upload slots. A slot is reserved before reading its part,
+so queued uploads do not allocate unbounded buffers. Tune concurrency with
+`TransferBudget(floor=..., ceiling=..., per_file_cap=...)`, or CLI
+`--concurrency-floor` (also used by `cp`, `mv`, and `sync`). To limit a batch
+to one in-flight part/range, use `TransferBudget(floor=1, ceiling=1)`.
+Set part size globally via env, or per call:
 
 ```bash
 export ASANYPATH_UPLOAD_CHUNK_SIZE=33554432   # 32 MiB parts (bytes)
@@ -149,6 +205,11 @@ Bigger parts mean fewer requests and a larger single-object ceiling (S3 allows
 applied automatically: **S3** clamps to ≥ 5 MiB parts, **GCS** rounds down to a
 256 KiB multiple.
 
+On S3 failure or cancellation, in-flight workers are cancelled and joined
+before aborting the multipart upload; completion lists remain in source order.
+Azure only commits the ordered block list after all stages succeed; abandoned
+uncommitted blocks are left for Azure's automatic expiry.
+
 Unknown URI schemes now construct as `UnsupportedProtocolPath` placeholders.
 Pure path operations (e.g. `.name`, `.parent`, joins) still work, while backend
 operations (e.g. `.exists()`, `.read_bytes()`, `.open()`) raise
@@ -160,9 +221,8 @@ Reading the same object or URL twice always re-downloads it — there is **no
 cache by default**, and the read path is unchanged unless you opt in, so
 non-caching callers pay zero overhead.
 
-Enable it with the optional `cache` extra (`pip install asanypath[cache]`,
-backed by [`diskcache`](https://pypi.org/project/diskcache/) — a pure-Python,
-zero-dependency store). Opt in per read with `use_cache=True`:
+Caching is included in the native extension with no extra dependencies or
+installation extras. Opt in per read with `use_cache=True`:
 
 ```python
 from asanypath import AsAnyPath
@@ -181,7 +241,7 @@ from asanypath import configure_cache
 
 configure_cache(
     directory="/var/cache/asanypath",  # default: ~/.cache/asanypath
-    size_limit=2 * 1024**3,            # total on-disk cap in bytes (default 1 GiB, LRU)
+    size_limit=2 * 1024**3,            # stored key + payload budget (default 1 GiB, LRU)
     ttl=3600,                          # per-entry expiry in seconds (default: none)
     enabled=True,                      # make use_cache default to True process-wide
 )
@@ -196,9 +256,15 @@ Behaviour and trade-offs:
   default flips process-wide; an explicit `use_cache=False` always bypasses.
 - **Works for every remote backend** (S3, GCS, Azure, Artifactory, HTTP, SSH,
   FTP). Local paths are never cached.
-- **Total-size bounded, LRU-evicted;** disk I/O runs in a worker thread so the
+- **Size-bounded, LRU-evicted;** disk I/O runs in a worker thread so the
   event loop stays responsive. Concurrent reads of the same key collapse to a
-  single fetch.
+  single fetch within a cache instance. The byte budget counts UTF-8 keys and
+  payloads; SQLite metadata and transaction journals add filesystem overhead.
+  Objects larger than the budget are returned without being cached.
+- **Byte-only native storage:** SQLite stores raw bytes, never Python pickles.
+  The database lives under `native-v1/` in the configured cache directory.
+  Legacy DiskCache files are ignored, not migrated or deleted. Multiple
+  processes can share the new store; use the same size limit across them.
 - **Staleness is on you.** Entries are keyed by URL with no revalidation — a
   cached object that changes upstream is served stale until it expires (`ttl`)
   or is evicted. Prefer caching immutable/versioned objects, or set a `ttl`.
@@ -297,6 +363,12 @@ asanypath cp /local/file.txt s3://bucket/file.txt
 # Copy cloud → local
 asanypath cp s3://bucket/file.txt /local/file.txt
 
+# Allow 32 concurrent files; fill spare capacity toward 8 streams for large files
+asanypath cp -r s3://bucket/data/ ./data/ -j 32 --concurrency-floor 8
+
+# Disable automatic range splitting (also supported by mv and sync)
+asanypath cp s3://bucket/file.txt ./file.txt --concurrency-floor 1
+
 # Delete a file
 asanypath rm s3://bucket/file.txt --force
 
@@ -404,6 +476,22 @@ uv run --extra bench python scripts/bench.py --backend all --rounds 3 --objects
 uv run --extra bench python scripts/bench.py --backend art --rounds 3 --concurrency-levels 1,10,50
 ```
 
+For multipart upload throughput, use `scripts/bench_upload.py` with real shard
+files and an explicitly selected S3-compatible endpoint:
+
+```bash
+uv run --extra bench python scripts/bench_upload.py \
+  --destination s3://my-bucket/bench --endpoint-url https://my-storage-endpoint \
+  --part-mib 8 --streams 1 8 --rounds 3 /data/shard-001 /data/shard-002
+```
+
+This compares asanypath and boto3 using one shared request budget per client.
+It warms up each case, reports JSON timings and MiB/s, verifies uploaded contents
+outside the timed interval, and deletes only the uniquely named benchmark
+objects it created. Both clients use normal AWS credentials (or `--profile`).
+Run with one large shard and again with several representative shards; results
+from a local test server are not evidence of production endpoint throughput.
+
 ### S3 — live S3-compatible service (median of 20 rounds, 10 objects, c=1)
 
 | Operation | asanypath | s3fs | aiobotocore | cloudpathlib |

{asanypath-2.2.0 → asanypath-2.3.1}/pyproject.toml

@@ -58,8 +58,8 @@ dev = [
 cli = ["rich-click>=1.9.7"]
 ftp = ["aioftp>=0.21"]
 credstore = ["keyring>=24"]
-cache = ["diskcache>=5.6"]
 bench = [
+    "boto3",
     "asyncssh>=2.14",
     "fsspec",
     "s3fs",
@@ -72,7 +72,7 @@ bench = [
     "aiofiles",
     "requests",
 ]
-all = ["asanypath[credstore,ftp,cache]"]
+all = ["asanypath[credstore,ftp]"]
 [project.scripts]
 asanypath = "asanypath.cli:main"
 
@@ -151,10 +151,6 @@ disable_error_code = ["no-any-return", "misc", "union-attr", "type-var", "attr-d
 module = "asanypath.batcher"
 disallow_untyped_defs = false
 
-[[tool.mypy.overrides]]
-module = "diskcache.*"
-ignore_missing_imports = true
-
 [tool.pytest.ini_options]
 asyncio_mode = "auto"
 testpaths = ["tests"]

asanypath-2.3.1/scripts/bench_upload.py

@@ -0,0 +1,181 @@
+# Copyright (C) 2026 Axis Communications AB, Lund, Sweden
+#
+# Use of this source code is governed by an MIT-style
+# license that can be found in the LICENSE file or at
+# https://opensource.org/licenses/MIT.
+
+"""Compare bounded multipart uploads of real files to an explicitly chosen S3 prefix.
+
+Uses normal AWS credential resolution; never reads repository credential files.
+Each case warms up once, uploads to unique keys, verifies contents outside the
+timed interval, and deletes only its own objects. Timings include public copy API
+overhead for asanypath and upload_file API overhead for boto3.
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import hashlib
+import json
+import statistics
+import time
+from collections.abc import Awaitable, Iterable
+from pathlib import Path
+from uuid import uuid4
+
+from asanypath import AsAnyPath, TransferBudget
+
+
+async def finish_uploads(uploads: Iterable[Awaitable[object]]) -> None:
+    pending = asyncio.gather(*uploads, return_exceptions=True)
+    try:
+        results = await asyncio.shield(pending)
+    except asyncio.CancelledError:
+        await pending
+        raise
+    for result in results:
+        if isinstance(result, BaseException):
+            raise result
+
+
+def positive(value: str) -> int:
+    result = int(value)
+    if result < 1:
+        raise argparse.ArgumentTypeError("must be positive")
+    return result
+
+
+def digest(path: Path) -> bytes:
+    checksum = hashlib.sha256()
+    with path.open("rb") as file:
+        while chunk := file.read(8 * 1024 * 1024):
+            checksum.update(chunk)
+    return checksum.digest()
+
+
+async def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("files", nargs="+", type=Path, help="Representative local shard files")
+    parser.add_argument("--destination", required=True, help="s3://bucket/benchmark-prefix")
+    parser.add_argument("--endpoint-url", help="The storage endpoint used by both clients")
+    parser.add_argument("--profile", help="AWS profile used by both clients")
+    parser.add_argument("--streams", nargs="+", type=positive, default=[1, 8])
+    parser.add_argument("--part-mib", type=positive, default=8)
+    parser.add_argument("--rounds", type=positive, default=3)
+    parser.add_argument(
+        "--implementations",
+        nargs="+",
+        choices=["asanypath", "boto3"],
+        default=["asanypath", "boto3"],
+    )
+    args = parser.parse_args()
+    if not args.destination.startswith("s3://") or not args.destination[5:].split("/")[0]:
+        parser.error("--destination must be an s3://bucket/prefix URI")
+    if args.part_mib < 5:
+        parser.error("--part-mib must be at least 5 for S3")
+    for file in args.files:
+        if not file.is_file():
+            parser.error(f"Not a file: {file}")
+    kwargs = {}
+    if args.endpoint_url:
+        kwargs["endpoint_url"] = args.endpoint_url
+    if args.profile:
+        kwargs["profile"] = args.profile
+    part_size = args.part_mib * 1024 * 1024
+    total = sum(file.stat().st_size for file in args.files)
+    expected = [digest(file) for file in args.files]
+    base = args.destination.rstrip("/") + "/asanypath-upload-bench-" + uuid4().hex
+    bucket = base[5:].split("/", 1)[0]
+
+    for streams in args.streams:
+        for implementation in args.implementations:
+            client = transfer = None
+            if implementation == "boto3":
+                import boto3
+                from boto3.s3.transfer import S3Transfer, TransferConfig
+                from botocore.config import Config
+
+                client = boto3.Session(profile_name=args.profile).client(
+                    "s3",
+                    endpoint_url=args.endpoint_url,
+                    config=Config(max_pool_connections=max(10, streams)),
+                )
+                # One manager shares its request budget across all files.
+                transfer = S3Transfer(
+                    client,
+                    TransferConfig(
+                        multipart_threshold=part_size,
+                        multipart_chunksize=part_size,
+                        max_concurrency=streams,
+                        preferred_transfer_client="classic",
+                    ),
+                )
+            samples = []
+            try:
+                for round_number in range(args.rounds + 1):
+                    uris = [
+                        f"{base}/{implementation}/{streams}/{round_number}/{index}"
+                        for index in range(len(args.files))
+                    ]
+                    destinations = [AsAnyPath(uri, **kwargs) for uri in uris]
+                    try:
+                        start = time.perf_counter()
+                        if implementation == "asanypath":
+                            with TransferBudget(
+                                floor=streams, ceiling=streams, per_file_cap=streams
+                            ):
+                                await finish_uploads(
+                                    (
+                                        AsAnyPath(file).copy(dst, chunk_size=part_size)
+                                        for file, dst in zip(args.files, destinations, strict=True)
+                                    )
+                                )
+                        else:
+                            await finish_uploads(
+                                (
+                                    asyncio.to_thread(
+                                        transfer.upload_file,
+                                        str(file),
+                                        bucket,
+                                        uri[5:].split("/", 1)[1],
+                                    )
+                                    for file, uri in zip(args.files, uris, strict=True)
+                                )
+                            )
+                        elapsed = time.perf_counter() - start
+                        for dst, checksum in zip(destinations, expected, strict=True):
+                            actual = hashlib.sha256()
+                            async for chunk in dst.iter_bytes(part_size):
+                                actual.update(chunk)
+                            if actual.digest() != checksum:
+                                raise OSError(f"Uploaded content verification failed: {dst}")
+                        if round_number:
+                            samples.append(elapsed)
+                    finally:
+                        await asyncio.gather(*(dst.unlink(missing_ok=True) for dst in destinations))
+                median = statistics.median(samples)
+                print(
+                    json.dumps(
+                        {
+                            "implementation": implementation,
+                            "streams": streams,
+                            "files": len(args.files),
+                            "bytes": total,
+                            "part_bytes": part_size,
+                            "seconds": samples,
+                            "median_seconds": median,
+                            "median_mib_per_second": total / (1024 * 1024) / median,
+                        }
+                    ),
+                    flush=True,
+                )
+            finally:
+                if transfer is not None:
+                    await asyncio.to_thread(transfer.__exit__, None, None, None)
+                if client is not None:
+                    client.close()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

{asanypath-2.2.0 → asanypath-2.3.1}/src/asanypath/__init__.py

@@ -4,6 +4,7 @@
 # license that can be found in the LICENSE file or at
 # https://opensource.org/licenses/MIT.
 
+from asanypath._transfer import TransferBudget
 from asanypath.artifactory import ArtifactoryPath
 from asanypath.asanypath import (
     PROTOCOL_MAP,
@@ -53,6 +54,7 @@ __all__ = [
     "AsAnyPath",
     "AsyncPath",
     "SyncPath",
+    "TransferBudget",
     "BackendOptions",
     "AccessGrant",
     "AccessPolicy",

{asanypath-2.2.0 → asanypath-2.3.1}/src/asanypath/__init__.pyi

@@ -6,6 +6,7 @@
 
 from __future__ import annotations
 
+from ._transfer import TransferBudget as TransferBudget
 from ._version import __version__ as __version__
 from .artifactory import ArtifactoryPath as ArtifactoryPath
 from .asanypath import (