hyperstudy 0.2.1__tar.gz → 0.2.3__tar.gz

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: hyperstudy
-Version: 0.2.1
+Version: 0.2.3
 Summary: Python SDK for the HyperStudy experiment platform API
 Project-URL: Homepage, https://hyperstudy.io
 Project-URL: Documentation, https://docs.hyperstudy.io/developers/python-sdk

hyperstudy-0.2.3/docs/superpowers/specs/2026-04-10-recording-downloads-design.md

@@ -0,0 +1,128 @@
+# Recording Downloads via Python SDK
+
+## Problem
+
+The Python SDK's `get_recordings()` returns metadata only. Users need the actual audio/video files for offline analysis (ML models, manual review, archival). Currently they must manually extract `downloadUrl` from each record and fetch files themselves.
+
+## Decision: SDK-only, no backend changes
+
+The V3 API already returns signed GCS download URLs (7-day expiry) in the recording metadata. The SDK will fetch metadata and download files in the same call, so URL expiry is not a practical concern. This matches how the frontend downloads recordings.
+
+## API Surface
+
+### `download_recordings()` — Bulk download
+
+```python
+df = hs.download_recordings(
+    "exp_abc123",
+    output_dir="./data/recordings",
+    scope="experiment",           # "experiment" | "room" | "participant"
+    deployment_id=None,           # optional filter
+    room_id=None,                 # optional filter
+    recording_type=None,          # "audio" | "video" | None (both)
+    progress=True,                # tqdm progress bar
+    skip_existing=True,           # skip files already on disk with matching size
+)
+```
+
+**Returns**: `pandas.DataFrame` with all recording metadata columns plus:
+- `local_path` — absolute path to the downloaded file on disk
+- `download_status` — `"downloaded"`, `"skipped"`, or `"failed"`
+
+**Side effects**:
+- Writes media files to `output_dir`
+- Writes `recordings_metadata.csv` to `output_dir`
+
+### `download_recording()` — Single recording
+
+```python
+path = hs.download_recording(
+    recording,                    # dict from get_recordings(output="dict")
+    output_dir="./data/recordings",
+)
+```
+
+**Returns**: `pathlib.Path` to downloaded file.
+
+## Directory Structure
+
+```
+output_dir/
+  recordings_metadata.csv
+  user1_video_EG_abc123.mp4
+  user1_audio_EG_def456.webm
+  user2_video_EG_ghi789.mp4
+```
+
+**Filename pattern**: `{participantName}_{recordingType}_{recordingId}.{ext}`
+
+- `participantName`: from recording metadata, sanitized for filesystem safety
+- `recordingType`: `"video"` or `"audio"` from `metadata.type`
+- `recordingId`: egressId or recordingId
+- `ext`: from `format` field, falling back to `mp4` (video) or `webm` (audio)
+
+## Internal Design
+
+### Download flow (`download_recordings`)
+
+1. Call `self.get_recordings(scope_id, scope=scope, output="dict")` to get metadata
+2. Filter by `recording_type` if specified (via `metadata.type`)
+3. Create `output_dir` via `os.makedirs(exist_ok=True)`
+4. For each recording:
+   - Build filename using pattern above
+   - If `skip_existing=True` and file exists with size matching `fileSize` metadata, mark as `"skipped"`
+   - Otherwise, fetch from `downloadUrl` (fallback: `url`) using streaming HTTP GET
+   - Write to disk in 8KB chunks
+   - Mark as `"downloaded"` or `"failed"` (with warning logged)
+5. Build DataFrame from metadata, add `local_path` and `download_status` columns
+6. Write `recordings_metadata.csv` to `output_dir`
+7. Return DataFrame
+
+### Streaming downloads
+
+Use `requests.get(url, stream=True)` with chunked iteration to avoid loading large video files into memory. The SDK's existing `HttpTransport` handles JSON responses only, so file downloads use a standalone `requests.get()` — the signed GCS URLs don't need API key auth.
+
+### Error handling
+
+- Per-file failure tolerance: if one recording fails (404, timeout, network error), log a warning, set `download_status="failed"`, continue with remaining files
+- If the metadata API call itself fails, raise normally (same as `get_recordings()`)
+- Invalid/missing `downloadUrl`: set `download_status="failed"`, log warning
+
+### Skip-existing logic
+
+Compare `os.path.getsize(local_path)` against `fileSize` from metadata. If `fileSize` is `None` (metadata missing), fall back to checking file existence only (any existing file is considered complete).
+
+## File Layout
+
+| File | Change |
+|------|--------|
+| `src/hyperstudy/_downloads.py` | **New.** `build_filename()`, `download_file()` streaming helper |
+| `src/hyperstudy/client.py` | Add `download_recordings()` and `download_recording()` methods |
+| `tests/test_downloads.py` | **New.** Unit tests for filename building, skip logic, status tracking |
+| `tests/test_client.py` | Integration test: mock API + GCS, verify files + DataFrame |
+| `tests/fixtures/sparse_ratings_response.json` | Already exists (from prior work) |
+
+## Testing
+
+### Unit tests (`tests/test_downloads.py`)
+- `test_build_filename` — video, audio, missing fields, filesystem-unsafe characters
+- `test_build_filename_dedup` — duplicate names get numeric suffix
+- `test_skip_existing_matching_size` — file with correct size is skipped
+- `test_skip_existing_wrong_size` — file with wrong size is re-downloaded
+
+### Integration tests (`tests/test_client.py`)
+- `test_download_recordings` — mock API + GCS fetch, verify files on disk, CSV sidecar, DataFrame with `local_path` + `download_status`
+- `test_download_recordings_filter_type` — `recording_type="audio"` only downloads audio
+- `test_download_recording_single` — single recording download
+
+### Mocking strategy
+- V3 API: `responses` library (existing pattern)
+- GCS signed URL: also `responses` (it's just an HTTP GET to a URL)
+- File I/O: real writes to `pytest` `tmp_path`
+
+## No Backend Changes Required
+
+The existing V3 API endpoints return all necessary data:
+- `GET /api/v3/data/recordings/{scope}/{scopeId}` returns metadata with `downloadUrl`
+- Signed GCS URLs are valid for 7 days
+- SDK downloads immediately after fetching metadata, so expiry is not an issue

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "hyperstudy"
-version = "0.2.1"
+version = "0.2.3"
 description = "Python SDK for the HyperStudy experiment platform API"
 readme = "README.md"
 license = "MIT"

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/src/hyperstudy/__init__.py

@@ -19,7 +19,7 @@ from .exceptions import (
     ValidationError,
 )
 
-__version__ = "0.2.0"
+__version__ = "0.2.3"
 
 __all__ = [
     "HyperStudy",

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/src/hyperstudy/_dataframe.py

@@ -6,6 +6,51 @@ from typing import Any
 
 import pandas as pd
 
+# Nested dict fields to flatten into top-level columns.
+# Mapping of {field_name: prefix} — sub-keys become ``{prefix}_{sub_key}``.
+FLATTEN_FIELDS: dict[str, str] = {
+    "sparseRatingData": "sparseRatingData",
+    "metadata": "metadata",
+}
+
+
+def _flatten_nested_dicts(
+    data: list[dict[str, Any]],
+    fields: dict[str, str] | None = None,
+) -> list[dict[str, Any]]:
+    """Promote sub-keys of nested dict fields to top-level keys.
+
+    For each *field* present in a record whose value is a ``dict``, every
+    sub-key is copied to ``{prefix}_{sub_key}``.  The original nested dict
+    is preserved for backward compatibility.
+
+    Records where the target field is ``None`` or missing are left
+    untouched — downstream DataFrame construction fills those columns
+    with ``NaN`` / ``null``.
+    """
+    if not data:
+        return data
+
+    fields = fields if fields is not None else FLATTEN_FIELDS
+
+    # Quick check on first record — skip work when no target fields exist.
+    sample = data[0]
+    targets = [f for f in fields if f in sample and isinstance(sample[f], dict)]
+    if not targets:
+        return data
+
+    out: list[dict[str, Any]] = []
+    for record in data:
+        record = dict(record)  # shallow copy to avoid mutating caller's data
+        for field in targets:
+            nested = record.get(field)
+            if isinstance(nested, dict):
+                prefix = fields[field]
+                for sub_key, sub_val in nested.items():
+                    record[f"{prefix}_{sub_key}"] = sub_val
+        out.append(record)
+    return out
+
 
 def _post_process(df: pd.DataFrame) -> pd.DataFrame:
     """Shared post-processing for pandas DataFrames.
@@ -32,6 +77,7 @@ def to_pandas(data: list[dict[str, Any]]) -> pd.DataFrame:
     """Convert API response data to a pandas DataFrame with post-processing."""
     if not data:
         return pd.DataFrame()
+    data = _flatten_nested_dicts(data)
     df = pd.DataFrame(data)
     return _post_process(df)
 
@@ -51,6 +97,7 @@ def to_polars(data: list[dict[str, Any]]):
     if not data:
         return pl.DataFrame()
 
+    data = _flatten_nested_dicts(data)
     df = pl.DataFrame(data)
 
     # Parse timestamps

hyperstudy-0.2.3/src/hyperstudy/_downloads.py

@@ -0,0 +1,76 @@
+"""Helpers for downloading recording files from signed URLs."""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+from typing import Any
+
+import requests
+
+_CHUNK_SIZE = 65536  # 64 KB — good balance for large video files
+_UNSAFE_RE = re.compile(r"[^\w\-]")
+
+
+def get_download_url(recording: dict[str, Any]) -> str | None:
+    """Return the best download URL from a recording dict, or ``None``."""
+    return recording.get("downloadUrl") or recording.get("url") or None
+
+
+def build_filename(recording: dict[str, Any]) -> str:
+    """Build a filesystem-safe filename from recording metadata.
+
+    Pattern: ``{participantName}_{type}_{recordingId}.{ext}``
+    """
+    name = recording.get("participantName") or recording.get("participantId") or "unknown"
+    name = _UNSAFE_RE.sub("_", name)
+
+    meta = recording.get("metadata") or {}
+    rec_type = meta.get("type") or "recording"
+
+    rec_id = recording.get("recordingId") or recording.get("egressId") or "unknown"
+
+    fmt = recording.get("format")
+    if not fmt:
+        fmt = "webm" if rec_type == "audio" else "mp4"
+
+    return f"{name}_{rec_type}_{rec_id}.{fmt}"
+
+
+def download_file(url: str, dest: Path, timeout: int = 300) -> int:
+    """Stream-download *url* to *dest* and return bytes written.
+
+    Verifies ``Content-Length`` (when the server provides it) against bytes
+    actually written. On any error — HTTP, mid-stream disconnect, or length
+    mismatch — the partial file on disk is removed so ``skip_existing``
+    logic won't mistake it for a complete download on retry.
+    """
+    resp = requests.get(url, stream=True, timeout=timeout)
+    resp.raise_for_status()
+
+    expected: int | None
+    if "Content-Length" in resp.headers:
+        try:
+            expected = int(resp.headers["Content-Length"])
+        except (TypeError, ValueError):
+            expected = None
+    else:
+        expected = None
+
+    written = 0
+    # BaseException so Ctrl-C also cleans up, not just exceptions.
+    try:
+        with open(dest, "wb") as fh:
+            for chunk in resp.iter_content(chunk_size=_CHUNK_SIZE):
+                fh.write(chunk)
+                written += len(chunk)
+
+        if expected is not None and written != expected:
+            raise IOError(
+                f"Truncated download: wrote {written} of {expected} bytes for {dest.name}"
+            )
+    except BaseException:
+        dest.unlink(missing_ok=True)
+        raise
+
+    return written

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/src/hyperstudy/_http.py

@@ -63,8 +63,14 @@ class HttpTransport:
     # Public helpers
     # ------------------------------------------------------------------
 
-    def get(self, path: str, params: dict[str, Any] | None = None) -> dict:
-        return self._request("GET", path, params=params)
+    def get(
+        self,
+        path: str,
+        params: dict[str, Any] | None = None,
+        *,
+        timeout: int | float | None = None,
+    ) -> dict:
+        return self._request("GET", path, params=params, timeout=timeout)
 
     def post(self, path: str, json: dict[str, Any] | None = None) -> dict:
         return self._request("POST", path, json=json)
@@ -81,7 +87,8 @@ class HttpTransport:
 
     def _request(self, method: str, path: str, **kwargs) -> dict:
         url = f"{self.base_url}/{path.lstrip('/')}"
-        kwargs.setdefault("timeout", self.timeout)
+        if kwargs.get("timeout") is None:
+            kwargs["timeout"] = self.timeout
 
         resp = self._session.request(method, url, **kwargs)
         return self._handle_response(resp)

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/src/hyperstudy/_pagination.py

@@ -16,6 +16,7 @@ def fetch_all_pages(
     *,
     page_size: int = 1000,
     progress: bool = True,
+    timeout: int | float | None = None,
 ) -> tuple[list[dict], dict]:
     """Fetch every page of a paginated endpoint.
 
@@ -28,7 +29,7 @@ def fetch_all_pages(
     params.setdefault("offset", 0)
 
     # First request to learn total
-    body = transport.get(path, params=params)
+    body = transport.get(path, params=params, timeout=timeout)
     metadata = body.get("metadata", {})
     pagination = metadata.get("pagination", {})
     total = pagination.get("total", 0)
@@ -43,7 +44,7 @@ def fetch_all_pages(
     while has_more:
         params["offset"] = pagination.get("nextOffset", params["offset"] + page_size)
 
-        body = transport.get(path, params=params)
+        body = transport.get(path, params=params, timeout=timeout)
         metadata = body.get("metadata", {})
         pagination = metadata.get("pagination", {})
 

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/src/hyperstudy/client.py

@@ -2,12 +2,18 @@
 
 from __future__ import annotations
 
+import warnings
+from pathlib import Path
 from typing import Any
 
+from tqdm.auto import tqdm
+
 from ._dataframe import to_pandas, to_polars
+from ._downloads import build_filename, download_file, get_download_url
 from ._http import HttpTransport
 from ._pagination import fetch_all_pages
 from ._types import Scope
+from .exceptions import HyperStudyError
 from .experiments import ExperimentMixin
 
 
@@ -466,6 +472,145 @@ class HyperStudy(ExperimentMixin):
             "consent": self.get_consent(participant_id, **common),
         }
 
+    # ------------------------------------------------------------------
+    # Recording downloads
+    # ------------------------------------------------------------------
+
+    def download_recording(
+        self,
+        recording: dict[str, Any],
+        output_dir: str = ".",
+    ) -> Path:
+        """Download a single recording file to disk.
+
+        Args:
+            recording: A recording dict (from ``get_recordings(output="dict")``).
+            output_dir: Directory to save the file.
+
+        Returns:
+            Path to the downloaded file.
+        """
+        url = get_download_url(recording)
+        if not url:
+            raise ValueError("Recording has no downloadUrl or url field")
+
+        dest_dir = Path(output_dir)
+        dest_dir.mkdir(parents=True, exist_ok=True)
+
+        filename = build_filename(recording)
+        dest = dest_dir / filename
+        download_file(url, dest)
+        return dest
+
+    def download_recordings(
+        self,
+        scope_id: str,
+        *,
+        output_dir: str,
+        scope: str = "experiment",
+        deployment_id: str | None = None,
+        room_id: str | None = None,
+        recording_type: str | None = None,
+        progress: bool = True,
+        skip_existing: bool = True,
+    ):
+        """Download recording files to disk.
+
+        Fetches recording metadata, downloads each file from its signed
+        URL, writes a ``recordings_metadata.csv`` sidecar, and returns a
+        DataFrame with a ``local_path`` column.
+
+        Args:
+            scope_id: Experiment, room, or participant ID.
+            output_dir: Directory to save files.
+            scope: ``"experiment"``, ``"room"``, or ``"participant"``.
+            deployment_id: Filter by deployment (experiment scope only).
+            room_id: Filter by room.
+            recording_type: ``"audio"``, ``"video"``, or ``None`` (both).
+            progress: Show progress bar.
+            skip_existing: Skip files already on disk with matching size.
+
+        Returns:
+            pandas DataFrame with recording metadata plus ``local_path``
+            and ``download_status`` columns.
+        """
+        try:
+            # Signing per-recording GCS URLs scales with count; 30s default is too short.
+            recordings = self._fetch_data(
+                "recordings",
+                scope_id,
+                scope=scope,
+                deployment_id=deployment_id,
+                room_id=room_id,
+                output="dict",
+                timeout=300,
+            )
+        except Exception as exc:
+            raise HyperStudyError(
+                f"Failed to list recordings for {scope}={scope_id!r}: {exc}",
+                code=getattr(exc, "code", "LIST_RECORDINGS_FAILED"),
+                status_code=getattr(exc, "status_code", None),
+                details=getattr(exc, "details", None),
+            ) from exc
+
+        if recording_type:
+            recordings = [
+                r for r in recordings
+                if (r.get("metadata") or {}).get("type") == recording_type
+            ]
+
+        dest_dir = Path(output_dir)
+        dest_dir.mkdir(parents=True, exist_ok=True)
+
+        local_paths: list[str | None] = []
+        statuses: list[str] = []
+
+        for rec in tqdm(recordings, desc="Downloading recordings", disable=not progress):
+            filename = build_filename(rec)
+            dest = dest_dir / filename
+
+            url = get_download_url(rec)
+            if not url:
+                local_paths.append(None)
+                statuses.append("failed")
+                warnings.warn(f"Recording {rec.get('recordingId')} has no download URL")
+                continue
+
+            if skip_existing and dest.exists():
+                expected_size = rec.get("fileSize")
+                # Require positive size match; missing size → re-download to
+                # avoid trusting a truncated file from a prior failed run.
+                if expected_size is not None and dest.stat().st_size == expected_size:
+                    local_paths.append(str(dest.resolve()))
+                    statuses.append("skipped")
+                    continue
+
+            try:
+                download_file(url, dest)
+                local_paths.append(str(dest.resolve()))
+                statuses.append("downloaded")
+            except Exception as exc:
+                local_paths.append(None)
+                statuses.append("failed")
+                warnings.warn(
+                    f"Failed to download recording {rec.get('recordingId')}: {exc}"
+                )
+
+        failed_count = sum(1 for s in statuses if s == "failed")
+        if failed_count:
+            warnings.warn(
+                f"{failed_count}/{len(recordings)} recordings failed to download; "
+                "see the 'download_status' column for per-file results."
+            )
+
+        df = to_pandas(recordings)
+        if not df.empty:
+            df["local_path"] = local_paths
+            df["download_status"] = statuses
+            df.to_csv(dest_dir / "recordings_metadata.csv", index=False)
+
+        return df
+
     # ------------------------------------------------------------------
     # Internal helpers
     # ------------------------------------------------------------------
@@ -525,6 +670,7 @@ class HyperStudy(ExperimentMixin):
         offset: int = 0,
         output: str = "pandas",
         progress: bool = True,
+        timeout: int | float | None = None,
         **extra_params,
     ):
         """Generic data-fetching logic shared by all ``get_*`` methods."""
@@ -551,11 +697,15 @@ class HyperStudy(ExperimentMixin):
         # Single page or all pages
         if limit is not None:
             params["limit"] = limit
-            body = self._transport.get(path, params=params)
+            body = self._transport.get(path, params=params, timeout=timeout)
             data = body.get("data", [])
         else:
             data, _ = fetch_all_pages(
-                self._transport, path, params=params, progress=progress
+                self._transport,
+                path,
+                params=params,
+                progress=progress,
+                timeout=timeout,
             )
 
         return self._convert_output(data, output)

{hyperstudy-0.2.1 → hyperstudy-0.2.3}/tests/conftest.py

@@ -71,6 +71,16 @@ def deployment_sessions_response():
     return load_fixture("deployment_sessions_response.json")
 
 
+@pytest.fixture
+def sparse_ratings_response():
+    return load_fixture("sparse_ratings_response.json")
+
+
+@pytest.fixture
+def recordings_response():
+    return load_fixture("recordings_response.json")
+
+
 @pytest.fixture
 def warnings_response():
     return load_fixture("warnings_response.json")

hyperstudy-0.2.3/tests/fixtures/recordings_response.json

@@ -0,0 +1,71 @@
+{
+  "status": "success",
+  "metadata": {
+    "dataType": "recordings",
+    "scope": "experiment",
+    "scopeId": "exp_abc123",
+    "timestamp": "2024-06-15T10:00:00.000Z",
+    "query": {
+      "limit": 1000,
+      "offset": 0,
+      "sort": "startTime",
+      "order": "asc"
+    },
+    "pagination": {
+      "total": 2,
+      "returned": 2,
+      "hasMore": false,
+      "limit": 1000,
+      "offset": 0
+    },
+    "processing": {
+      "processingTimeMs": 35,
+      "enriched": true,
+      "version": "3.0.0"
+    }
+  },
+  "data": [
+    {
+      "recordingId": "EG_video_001",
+      "egressId": "EG_video_001",
+      "participantId": "user_1",
+      "participantName": "Alice",
+      "startTime": "2024-06-15T10:00:05.000Z",
+      "endTime": "2024-06-15T10:05:05.000Z",
+      "duration": 300000,
+      "videoOffset": 500,
+      "url": "https://storage.googleapis.com/bucket/recordings/video1.mp4",
+      "downloadUrl": "https://storage.googleapis.com/bucket/recordings/video1.mp4?X-Goog-Signature=abc",
+      "fileSize": 1024,
+      "format": "mp4",
+      "status": "complete",
+      "metadata": {
+        "type": "video",
+        "recordingType": "individual",
+        "roomName": "room_1",
+        "experimentId": "exp_abc123"
+      }
+    },
+    {
+      "recordingId": "EG_audio_002",
+      "egressId": "EG_audio_002",
+      "participantId": "user_1",
+      "participantName": "Alice",
+      "startTime": "2024-06-15T10:00:05.000Z",
+      "endTime": "2024-06-15T10:05:05.000Z",
+      "duration": 300000,
+      "videoOffset": 500,
+      "url": "https://storage.googleapis.com/bucket/recordings/audio1.webm",
+      "downloadUrl": "https://storage.googleapis.com/bucket/recordings/audio1.webm?X-Goog-Signature=def",
+      "fileSize": 512,
+      "format": "webm",
+      "status": "complete",
+      "metadata": {
+        "type": "audio",
+        "recordingType": "audio",
+        "roomName": "room_1",
+        "experimentId": "exp_abc123"
+      }
+    }
+  ]
+}