riffsdk 0.4.2__tar.gz → 0.5.0__tar.gz

{riffsdk-0.4.2 → riffsdk-0.5.0}/.github/workflows/release.yml

@@ -86,3 +86,19 @@ jobs:
           path: dist/
       - name: Publish to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
+      - name: Verify release landed on PyPI
+        env:
+          TAG: ${{ needs.release.outputs.tag }}
+        run: |
+          VER="${TAG#v}"
+          echo "Verifying riffsdk ${VER} is available on PyPI..."
+          for i in $(seq 1 12); do
+            if curl -sf "https://pypi.org/pypi/riffsdk/${VER}/json" >/dev/null; then
+              echo "riffsdk ${VER} is live on PyPI"
+              exit 0
+            fi
+            echo "not yet on PyPI, retry ${i}/12..."
+            sleep 10
+          done
+          echo "::error::riffsdk ${VER} never appeared on PyPI despite a successful upload step (likely a silent edge 200 — see https://upload.pypi.org/legacy/ response in the publish step)"
+          exit 1

{riffsdk-0.4.2 → riffsdk-0.5.0}/CHANGELOG.md

@@ -2,6 +2,48 @@
 
 <!-- version list -->
 
+## v0.5.0 (2026-06-11)
+
+### Bug Fixes
+
+- Bugfixes for large file streaming ([#13](https://github.com/databutton/riff-sdk-python/pull/13),
+  [`619fd57`](https://github.com/databutton/riff-sdk-python/commit/619fd575ca353d955de23bb5e455ff6c6dc005d0))
+
+### Features
+
+- Rebuild staged-part state on streaming resume
+  ([#13](https://github.com/databutton/riff-sdk-python/pull/13),
+  [`619fd57`](https://github.com/databutton/riff-sdk-python/commit/619fd575ca353d955de23bb5e455ff6c6dc005d0))
+
+- Support uploading streams of unknown size
+  ([#13](https://github.com/databutton/riff-sdk-python/pull/13),
+  [`619fd57`](https://github.com/databutton/riff-sdk-python/commit/619fd575ca353d955de23bb5e455ff6c6dc005d0))
+
+
+## v0.4.3 (2026-06-11)
+
+### Bug Fixes
+
+- Add project homepage URL ([#12](https://github.com/databutton/riff-sdk-python/pull/12),
+  [`7333ba9`](https://github.com/databutton/riff-sdk-python/commit/7333ba9badb64eef3b111b74ee060fb5a6538c5b))
+
+- Declare MIT license and package author
+  ([#12](https://github.com/databutton/riff-sdk-python/pull/12),
+  [`7333ba9`](https://github.com/databutton/riff-sdk-python/commit/7333ba9badb64eef3b111b74ee060fb5a6538c5b))
+
+- Include README as package long description
+  ([#12](https://github.com/databutton/riff-sdk-python/pull/12),
+  [`7333ba9`](https://github.com/databutton/riff-sdk-python/commit/7333ba9badb64eef3b111b74ee060fb5a6538c5b))
+
+- Populate pypi release details and check publishing in CI job
+  ([#12](https://github.com/databutton/riff-sdk-python/pull/12),
+  [`7333ba9`](https://github.com/databutton/riff-sdk-python/commit/7333ba9badb64eef3b111b74ee060fb5a6538c5b))
+
+- Verify PyPI publish actually landed before passing release
+  ([#12](https://github.com/databutton/riff-sdk-python/pull/12),
+  [`7333ba9`](https://github.com/databutton/riff-sdk-python/commit/7333ba9badb64eef3b111b74ee060fb5a6538c5b))
+
+
 ## v0.4.2 (2026-06-11)
 
 ### Bug Fixes

riffsdk-0.5.0/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.4
+Name: riffsdk
+Version: 0.5.0
+Summary: Python SDK for the Riff Storage API
+Project-URL: Homepage, https://riff.ai
+Author-email: Martin Sandve Alnæs <msa@databutton.io>
+License-Expression: MIT
+License-File: LICENCE
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.6
+Requires-Dist: tenacity>=8.3
+Description-Content-Type: text/markdown
+
+# riffsdk
+
+Python SDK for the Riff Storage API. Provides sync and async clients for storing, retrieving, and managing objects.
+
+## Installation
+
+```bash
+uv add riffsdk
+# or
+pip install riffsdk
+```
+
+Or install from a branch (for pre-release testing):
+
+```bash
+uv add git+https://github.com/databutton/riff-sdk-python.git@main
+# or
+pip install git+https://github.com/databutton/riff-sdk-python.git@main
+```
+
+## Quick start
+
+```python
+from riffsdk.storage import StorageClient
+
+client = StorageClient()
+
+# Upload
+meta = client.put("hello.txt", "Hello, world!")
+
+# Download
+data = client.get("hello.txt")
+
+# List
+for obj in client.list("hello"):
+    print(f"{obj.key} ({obj.size} bytes)")
+
+# Delete
+client.delete("hello.txt")
+
+client.close()
+```
+
+### Async
+
+```python
+from riffsdk.storage import AsyncStorageClient
+
+async with AsyncStorageClient() as client:
+    await client.put("key", b"data", content_type="application/octet-stream")
+    data = await client.get("key")
+```
+
+## Authentication
+
+Set the `RIFF_TOKEN` environment variable. The SDK picks it up automatically.
+
+## API
+
+### Clients
+
+- `StorageClient` -- sync client
+- `AsyncStorageClient` -- async client
+
+Both support: `put`, `get`, `stat`, `exists`, `list`, `delete`, `close`, and context manager usage.
+
+### Models
+
+- `ObjectMeta` -- metadata for a stored object (key, version, size, content_type, timestamps)
+- `UploadResult`, `DownloadResult` -- operation results
+- `ListPage` -- paginated listing
+- `Scope` -- access scope (use `account_scope()`, `project_scope()`, `session_scope()`)
+
+### Uploads
+
+- `ResumableUpload` / `AsyncResumableUpload` -- multipart resumable uploads for large files
+- `StorageReader` / `StorageWriter` -- streaming read/write
+
+### Exceptions
+
+All exceptions inherit from `StorageError`:
+
+- `AuthorisationError`
+- `ObjectNotFoundError`
+- `VersionConflictError`
+- `AlreadyExistsError`
+- `LeaseConflictError`
+- `UploadNotFoundError`
+- `QuotaExceededError`
+- `PartMismatchError`
+- `StorageTransportError`
+
+## Examples
+
+See the `examples/` directory for complete working examples:
+
+- `basic_crud.py` -- put, get, list, delete
+- `async_client.py` -- async usage with asyncio
+- `file_upload_download.py` -- file uploads with progress
+- `optimistic_concurrency.py` -- version-based conflict handling
+
+## Development
+
+Requires Python 3.11+ and [uv](https://docs.astral.sh/uv/).
+
+```bash
+uv sync --dev        # Install dependencies
+mise run test        # Run tests
+mise run lint        # Lint
+mise run format      # Format code
+```
+
+See `AGENTS.md` for full development workflow details.

{riffsdk-0.4.2 → riffsdk-0.5.0}/pyproject.toml

@@ -4,11 +4,18 @@ build-backend = "hatchling.build"
 
 [project]
 name = "riffsdk"
-version = "0.4.2"
+version = "0.5.0"
 description = "Python SDK for the Riff Storage API"
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENCE"]
+authors = [{ name = "Martin Sandve Alnæs", email = "msa@databutton.io" }]
 requires-python = ">=3.11"
 dependencies = ["httpx>=0.27", "pydantic>=2.6", "tenacity>=8.3"]
 
+[project.urls]
+Homepage = "https://riff.ai"
+
 [project.scripts]
 riffsdk = "riffsdk._cli:main"
 

{riffsdk-0.4.2 → riffsdk-0.5.0}/src/riffsdk/_cli.py

@@ -3,8 +3,9 @@ from __future__ import annotations
 import argparse
 import mimetypes
 import os
+import stat
 import sys
-import tempfile
+from typing import BinaryIO
 
 from riffsdk.storage._client import StorageClient
 from riffsdk.storage._models import Scope
@@ -39,6 +40,84 @@ def _format_size(size: int) -> str:
     return f"{size:.1f} PB"
 
 
+_SIZE_UNITS = {
+    "B": 1,
+    "K": 1024,
+    "KB": 1024,
+    "M": 1024**2,
+    "MB": 1024**2,
+    "G": 1024**3,
+    "GB": 1024**3,
+    "T": 1024**4,
+    "TB": 1024**4,
+    "P": 1024**5,
+    "PB": 1024**5,
+}
+
+
+def _parse_size(text: str) -> int:
+    """Parse a human-readable size like '10G', '500MB', '1024' (bytes)."""
+    s = text.strip().upper()
+    unit = "B"
+    num = s
+    for u in ("PB", "TB", "GB", "MB", "KB", "P", "T", "G", "M", "K", "B"):
+        if s.endswith(u) and s[: -len(u)].strip():
+            num, unit = s[: -len(u)].strip(), u
+            break
+    try:
+        value = float(num)
+    except ValueError:
+        _die(f"invalid size: {text!r}")
+    if value < 0:
+        _die(f"invalid size: {text!r} (must not be negative)")
+    return int(value * _SIZE_UNITS[unit])
+
+
+class _PrefixedStream:
+    """Read-only stream that returns a pre-read prefix, then the rest.
+
+    Lets the CLI peek the first block of stdin (to detect an empty pipe)
+    without buffering or losing those bytes for the actual upload.
+    """
+
+    def __init__(self, prefix: bytes, rest: BinaryIO) -> None:
+        self._prefix = prefix
+        self._rest = rest
+
+    def read(self, size: int = -1) -> bytes:
+        if not self._prefix:
+            return self._rest.read(size)
+        if size is None or size < 0:
+            data = self._prefix + self._rest.read()
+            self._prefix = b""
+            return data
+        if size <= len(self._prefix):
+            data, self._prefix = self._prefix[:size], self._prefix[size:]
+            return data
+        data = self._prefix + self._rest.read(size - len(self._prefix))
+        self._prefix = b""
+        return data
+
+
+def _resolve_stream_size(args: argparse.Namespace) -> int | None:
+    """Total size of a stdin upload, or None if it can't be known without reading.
+
+    Uses an explicit --size, else the size of stdin when it is a regular file
+    (e.g. `riffsdk upload key -s < big.iso`). Returns None for non-seekable
+    pipes, where the size is genuinely unknown until the stream is consumed.
+    """
+    if getattr(args, "size", None):
+        return _parse_size(args.size)
+    try:
+        st = os.fstat(sys.stdin.fileno())
+    except (OSError, ValueError, TypeError):
+        # No real fd (closed, replaced, or a non-file object): size is unknown.
+        return None
+    if stat.S_ISREG(st.st_mode) and st.st_size > 0:
+        return st.st_size
+    return None
+
+
 def _make_progress(label: str) -> callable:
     def callback(done: int, total: int) -> None:
         if not sys.stderr.isatty():
@@ -92,20 +171,33 @@ def cmd_upload(client: StorageClient, args: argparse.Namespace) -> None:
     if args.stream:
         if args.input:
             _die("--input and --stream are mutually exclusive")
-        with tempfile.SpooledTemporaryFile(max_size=10 * 1024 * 1024) as tmp:
-            while True:
-                chunk = sys.stdin.buffer.read(65536)
-                if not chunk:
-                    break
-                tmp.write(chunk)
-            total_size = tmp.tell()
-            if total_size == 0:
+        content_type = ct or _guess_content_type(key)
+        chunk_size = _parse_size(args.chunk_size) if args.chunk_size else None
+        total_size = _resolve_stream_size(args)
+
+        if total_size is None:
+            # Unknown size: stream stdin straight through (the server signs
+            # part URLs on demand). Peek one block so an empty stdin is
+            # reported clearly without consuming the whole input. No progress
+            # bar — the total is unknown.
+            first = sys.stdin.buffer.read(65536)
+            if not first:
                 _die("no data received on stdin")
-            tmp.seek(0)
-            content_type = ct or _guess_content_type(key)
-            result = client.upload_stream(
-                key, tmp, total_size=total_size, content_type=content_type, scope=scope
-            )
+            source: object = _PrefixedStream(first, sys.stdin.buffer)
+            progress = None
+        else:
+            source = sys.stdin.buffer
+            progress = _make_progress(key)
+
+        result = client.upload_stream(
+            key,
+            source,
+            total_size=total_size,
+            content_type=content_type,
+            chunk_size=chunk_size,
+            on_progress=progress,
+            scope=scope,
+        )
         if args.verbose:
             sys.stderr.write(
                 f"Uploaded {result.key} ({_format_size(result.size)}, "
@@ -116,7 +208,12 @@ def cmd_upload(client: StorageClient, args: argparse.Namespace) -> None:
             _die(f"file not found: {args.input}")
         progress = _make_progress(os.path.basename(args.input))
         result = client.upload_file(
-            key, args.input, content_type=ct, on_progress=progress, scope=scope
+            key,
+            args.input,
+            content_type=ct,
+            chunk_size=_parse_size(args.chunk_size) if args.chunk_size else None,
+            on_progress=progress,
+            scope=scope,
         )
         if args.verbose:
             sys.stderr.write(
@@ -251,6 +348,18 @@ def _build_parser() -> argparse.ArgumentParser:
     p.add_argument(
         "-s", "--stream", action="store_true", help="read from stdin for piping"
     )
+    p.add_argument(
+        "--size",
+        default=None,
+        help="total size of the stdin stream (e.g. 10G); streams without "
+        "buffering. Auto-detected when stdin is a regular file.",
+    )
+    p.add_argument(
+        "--chunk-size",
+        default=None,
+        help="upload part size (e.g. 64M); larger chunks mean fewer parts "
+        "for very large uploads",
+    )
     p.add_argument("-t", "--content-type", default=None, help="MIME content type")
     p.set_defaults(func=cmd_upload)
 

{riffsdk-0.4.2 → riffsdk-0.5.0}/src/riffsdk/storage/__init__.py

@@ -6,6 +6,7 @@ from riffsdk.storage._models import (
     ListPage,
     ObjectMeta,
     Scope,
+    UploadedPart,
     UploadPart,
     UploadResult,
     account_scope,
@@ -39,6 +40,7 @@ __all__ = [
     "StorageClient",
     "ObjectMeta",
     "UploadPart",
+    "UploadedPart",
     "ListPage",
     "UploadResult",
     "StorageReader",

{riffsdk-0.4.2 → riffsdk-0.5.0}/src/riffsdk/storage/_async_client.py

@@ -17,6 +17,7 @@ from riffsdk.storage._models import (
     ListPage,
     ObjectMeta,
     Scope,
+    UploadedPart,
     UploadPart,
     UploadResult,
 )
@@ -293,13 +294,19 @@ class AsyncStorageClient:
         stream: BinaryIO,
         *,
         content_type: str = "application/octet-stream",
-        total_size: int,
+        total_size: int | None = None,
         chunk_size: int | None = None,
         if_version_matches: str | None = None,
         if_not_exists: bool = False,
         on_progress: Callable[[int, int], None] | None = None,
         scope: Scope = None,
     ) -> UploadResult:
+        """Upload a binary stream.
+
+        Pass ``total_size`` when the size is known to use pre-signed part URLs.
+        Leave it ``None`` to stream an input of unknown size: the source is read
+        in part-size chunks until EOF, signing part URLs on demand.
+        """
         cs = chunk_size or self._chunk_size
 
         upload = await self.begin_upload(
@@ -313,14 +320,25 @@ class AsyncStorageClient:
         )
 
         bytes_sent = 0
-        for part_num in range(1, upload.part_count + 1):
-            chunk = stream.read(upload.part_size)
-            if not chunk:
-                break
-            await upload.send_chunk(chunk, part_number=part_num)
-            bytes_sent += len(chunk)
-            if on_progress:
-                on_progress(bytes_sent, total_size)
+        if total_size is None:
+            # Streaming: read until EOF, letting send_chunk auto-number parts.
+            while True:
+                chunk = stream.read(upload.part_size)
+                if not chunk:
+                    break
+                await upload.send_chunk(chunk)
+                bytes_sent += len(chunk)
+                if on_progress:
+                    on_progress(bytes_sent, bytes_sent)
+        else:
+            for part_num in range(1, upload.part_count + 1):
+                chunk = stream.read(upload.part_size)
+                if not chunk:
+                    break
+                await upload.send_chunk(chunk, part_number=part_num)
+                bytes_sent += len(chunk)
+                if on_progress:
+                    on_progress(bytes_sent, total_size)
 
         return await upload.complete()
 
@@ -328,7 +346,7 @@ class AsyncStorageClient:
         self,
         key: str,
         *,
-        total_size: int,
+        total_size: int | None = None,
         content_type: str = "application/octet-stream",
         chunk_size: int | None = None,
         if_version_matches: str | None = None,
@@ -337,9 +355,12 @@ class AsyncStorageClient:
     ) -> AsyncResumableUpload:
         body: dict = {
             "key": key,
-            "size": total_size,
             "content_type": content_type,
         }
+        # Omit size to request a streaming upload (server returns part_count 0
+        # and signs part URLs on demand via refresh).
+        if total_size is not None:
+            body["size"] = total_size
         cs = chunk_size or self._chunk_size
         body["part_size"] = cs
         if if_version_matches is not None:
@@ -363,7 +384,7 @@ class AsyncStorageClient:
             sessions=self._sessions,
             upload_id=data["upload_id"],
             key=data["key"],
-            total_size=total_size,
+            total_size=total_size if total_size is not None else 0,
             part_count=data["part_count"],
             part_size=data["part_size"],
             expires_at=datetime.fromisoformat(data["expires_at"]),
@@ -385,6 +406,9 @@ class AsyncStorageClient:
         )
         data = resp.json()
         parts = [UploadPart.model_validate(p) for p in data["parts"]]
+        uploaded_parts = [
+            UploadedPart.model_validate(p) for p in data.get("uploaded_parts", [])
+        ]
 
         return AsyncResumableUpload(
             transport=self._transport,
@@ -401,6 +425,7 @@ class AsyncStorageClient:
             session_expires_at=datetime.fromisoformat(data["session_expires_at"]),
             parts=parts,
             content_type=data.get("content_type", "application/octet-stream"),
+            uploaded_parts=uploaded_parts,
         )
 
     # --- Download ---

{riffsdk-0.4.2 → riffsdk-0.5.0}/src/riffsdk/storage/_client.py

@@ -17,6 +17,7 @@ from riffsdk.storage._models import (
     ListPage,
     ObjectMeta,
     Scope,
+    UploadedPart,
     UploadPart,
     UploadResult,
 )
@@ -293,13 +294,19 @@ class StorageClient:
         stream: BinaryIO,
         *,
         content_type: str = "application/octet-stream",
-        total_size: int,
+        total_size: int | None = None,
         chunk_size: int | None = None,
         if_version_matches: str | None = None,
         if_not_exists: bool = False,
         on_progress: Callable[[int, int], None] | None = None,
         scope: Scope = None,
     ) -> UploadResult:
+        """Upload a binary stream.
+
+        Pass ``total_size`` when the size is known to use pre-signed part URLs.
+        Leave it ``None`` to stream an input of unknown size: the source is read
+        in part-size chunks until EOF, signing part URLs on demand.
+        """
         cs = chunk_size or self._chunk_size
 
         upload = self.begin_upload(
@@ -313,14 +320,25 @@ class StorageClient:
         )
 
         bytes_sent = 0
-        for part_num in range(1, upload.part_count + 1):
-            chunk = stream.read(upload.part_size)
-            if not chunk:
-                break
-            upload.send_chunk(chunk, part_number=part_num)
-            bytes_sent += len(chunk)
-            if on_progress:
-                on_progress(bytes_sent, total_size)
+        if total_size is None:
+            # Streaming: read until EOF, letting send_chunk auto-number parts.
+            while True:
+                chunk = stream.read(upload.part_size)
+                if not chunk:
+                    break
+                upload.send_chunk(chunk)
+                bytes_sent += len(chunk)
+                if on_progress:
+                    on_progress(bytes_sent, bytes_sent)
+        else:
+            for part_num in range(1, upload.part_count + 1):
+                chunk = stream.read(upload.part_size)
+                if not chunk:
+                    break
+                upload.send_chunk(chunk, part_number=part_num)
+                bytes_sent += len(chunk)
+                if on_progress:
+                    on_progress(bytes_sent, total_size)
 
         return upload.complete()
 
@@ -328,7 +346,7 @@ class StorageClient:
         self,
         key: str,
         *,
-        total_size: int,
+        total_size: int | None = None,
         content_type: str = "application/octet-stream",
         chunk_size: int | None = None,
         if_version_matches: str | None = None,
@@ -337,9 +355,12 @@ class StorageClient:
     ) -> ResumableUpload:
         body: dict = {
             "key": key,
-            "size": total_size,
             "content_type": content_type,
         }
+        # Omit size to request a streaming upload (server returns part_count 0
+        # and signs part URLs on demand via refresh).
+        if total_size is not None:
+            body["size"] = total_size
         cs = chunk_size or self._chunk_size
         body["part_size"] = cs
         if if_version_matches is not None:
@@ -363,7 +384,7 @@ class StorageClient:
             sessions=self._sessions,
             upload_id=data["upload_id"],
             key=data["key"],
-            total_size=total_size,
+            total_size=total_size if total_size is not None else 0,
             part_count=data["part_count"],
             part_size=data["part_size"],
             expires_at=datetime.fromisoformat(data["expires_at"]),
@@ -383,6 +404,9 @@ class StorageClient:
         )
         data = resp.json()
         parts = [UploadPart.model_validate(p) for p in data["parts"]]
+        uploaded_parts = [
+            UploadedPart.model_validate(p) for p in data.get("uploaded_parts", [])
+        ]
 
         return ResumableUpload(
             transport=self._transport,
@@ -399,6 +423,7 @@ class StorageClient:
             session_expires_at=datetime.fromisoformat(data["session_expires_at"]),
             parts=parts,
             content_type=data.get("content_type", "application/octet-stream"),
+            uploaded_parts=uploaded_parts,
         )
 
     # --- Download ---

{riffsdk-0.4.2 → riffsdk-0.5.0}/src/riffsdk/storage/_models.py

@@ -41,6 +41,22 @@ class UploadPart(BaseModel):
     url_expires_at: datetime
 
 
+class UploadedPart(BaseModel):
+    """A part already present in staging, reported by refresh/resume.
+
+    Enumerated live from the staging directory by the server. ``etag`` is the
+    GCS object etag (currently informational — complete() composes by part
+    number); ``size`` is the staged object size, useful for spotting a
+    truncated final part on resume.
+    """
+
+    model_config = ConfigDict(frozen=True)
+
+    part_number: int
+    size: int = 0
+    etag: str = ""
+
+
 class ListPage(BaseModel):
     model_config = ConfigDict(frozen=True)