radiens-data-catalog 0.1.0__tar.gz

radiens_data_catalog-0.1.0/.gitignore

@@ -0,0 +1,25 @@
+__pycache__/
+dist/
+build/
+*.egg-info/
+.coverage
+htmlcov/
+coverage.xml
+.vscode/
+.idea/
+*.swp
+*.swo
+.env
+.venv
+venv/
+env/
+.DS_Store
+*.log
+.mypy_cache
+.pytest_cache
+.ruff_cache
+site/
+.secrets/
+catalog.json
+
+.claude/settings.local.json

radiens_data_catalog-0.1.0/CHANGELOG.md

@@ -0,0 +1,361 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+_no unreleased changes_
+
+## [0.1.0] - 2026-10-12
+
+### Changed
+
+- **Breaking:** the package is now `radiens-data-catalog`, imported as `radiens_data_catalog`. Install `radiens-data-catalog` in place of `radiens-drive-catalog` and update imports. Releases up to 0.0.19 remain on PyPI as `radiens-drive-catalog`.
+- **Breaking:** the config environment variable is now `RADIENS_DATA_CATALOG_CONFIG`. `RADIENS_DRIVE_CATALOG_CONFIG` is no longer read.
+- The documentation moved to <https://neuronexus.github.io/radiens-data-catalog/latest/>.
+
+## [0.0.19] - 2026-10-12
+
+### Added
+
+- A `"regex"` match mode (`MatchMode`) for catalog filters: the pattern is searched anywhere in the field (`re.search` semantics; anchor with `^`/`$` for a full match). It works wherever the other modes do: `list_recordings()`, `list_items()`, `prefetch()` and `match_series()`.
+- `Catalog.get_meta(drive_path, base_name)` returns one recording's parsed `.xdat.json` metadata, and `Catalog.load_meta()` returns every matching recording's, keyed by `(drive_path, base_name)`, without downloading recording data. A downloaded recording's own file is read in place; any other is fetched alone (~100 KB) into `{catalog dir}/meta_cache/`, keyed by Drive file ID so a re-upload is never served stale, with missing files fetched concurrently. A file that fails to fetch or parse is left out with one summary warning, and a file Drive refuses permanently (403/404) is cached as empty so it isn't refetched on every call.
+- `recordings_df` gains `last_used` (when `get_recording_path()` last returned the recording on this machine, falling back to the data file's download time) and `downloaded` (whether every xdat file is actually on disk, since `local_path` is only a hint reconciled by `scan()`). Uses are stamped in `usage.json` beside the catalog, not in `catalog.json`, so a lost update under concurrent writers costs only a timestamp, never download state.
+- `Catalog.prefetch(keys=...)` restricts the download to an exact set of `(drive_path, base_name)` recordings, on top of the other filters, for callers that select recordings by something the filters can't express (e.g. metadata).
+- `Catalog.orphaned_downloads(drive_path=None, min_age_s=600)` lists the temp files that hard-killed recording downloads left behind, keyed by `(drive_path, base_name)`. Temp files written in the last `min_age_s` seconds are left out, since a download may still be writing them.
+
+### Fixed
+
+- Drive shortcuts are no longer catalogued or downloaded. A shortcut has no content of its own, so it became a recording with no size that Drive refused to download ("Only files with binary content can be downloaded"), and downloading a folder containing one failed. The files they point at are usually catalogued from their real location anyway.
+- A download interrupted with Ctrl-C (`KeyboardInterrupt`) left its `<file>.<uuid>.tmp` temp file behind; the cleanup now runs for any exception.
+- `Catalog.release_recording()` now also deletes the recording's orphaned download temp files, even when it was never fully downloaded. Before, nothing removed them.
+
+## [0.0.18] - 2026-08-13
+
+### Fixed
+
+- The docs site's Quick Start (`docs/index.md`) and Usage Guide (`docs/usage.md`), and the repo's `example.py`, documented an API generation removed in 0.0.8–0.0.14 (`list_datasets()`, `download_dataset()`, `get_dataset_path()`, `list_assets()`, `download_asset()`, `get_asset_path()`, `catalog.df`/`assets_df`, `asset_name`/`asset_type`, `drive_path_prefix`/`drive_path_contains`, and the old `local_data_dir/assets/{drive_path}/{name}` item path). Updated to the current `Catalog` API (`list_recordings()`/`list_items()` with `drive_path_mode`/`base_name_mode`, `recordings_df`/`items_df`, `download_recording()`/`download_item()`, `get_recording_path()`/`get_item_path()`), and added coverage of `prefetch()`, `release_recording()`/`release_item()`, `local_footprint()`, `file_tree()`, and `summary()`, which previously weren't documented anywhere on the site.
+
+## [0.0.17] - 2026-07-20
+
+- documentation updates
+
+## [0.0.16] - 2026-07-20
+
+### Added
+
+- `Catalog.prefetch()` now supports concurrent downloading via `max_workers` (default `4`). Downloads run in parallel across a thread pool with thread-safe progress tracking and worker slot indicators. Passing `max_workers=1` forces sequential execution.
+- `Catalog.release_recording(drive_path, base_name)` and `Catalog.release_item(drive_path, name)` delete a recording's or item's local files and clear its `local_path`, returning the number of bytes freed. Releasing a recording removes only its own xdat files (never the shared `drive_path` directory or sibling entries); releasing a folder item removes its full local subtree. Both are idempotent no-ops when the entry is not downloaded.
+- `Catalog.local_footprint()` returns the total on-disk size, in bytes, of all downloaded recordings and items. Together with `release_*`, this lets a caller cap local storage by releasing data until the footprint drops below a chosen limit — enabling batch processing of a corpus larger than local disk.
+
+### Fixed
+
+- Local size totals (the `summary()` "local" row, and now `local_footprint()`) no longer over-count downloaded folder items. Because the scanner catalogs every nested file and subfolder as its own item, downloading a folder marked its children local too, and each nested entry's subtree was summed again on top of the enclosing folder's — inflating the reported local size (e.g. 800 B on disk reported as 1900 B). Nested entries are now counted once, as part of their enclosing downloaded folder. Per-entry Drive `size` values and the Drive-size total were already correct and are unchanged.
+
+## [0.0.15] - 2026-07-18
+
+### Added
+
+- `match_series()`, exported from the package root, for filtering an already-fetched `recordings_df` / `items_df` (or any `pd.Series[str]`) with the same exact/prefix/contains semantics `Catalog.list_recordings()` and `Catalog.list_items()` use internally.
+
+### Changed
+
+- `Catalog`'s `drive_path`-taking methods (`list_recordings()`, `list_items()`, `get_recording()`, `download_recording()`, `get_recording_path()`, `download_item()`, `get_item_path()`, `prefetch()`) now accept `str` or `Path` for `drive_path` — previously `str`-only. `Path` values are converted internally, so a `Path` built via ordinary path arithmetic (e.g. `some_path.parent`) can be passed directly instead of failing a silent `Path != str` comparison.
+
+## [0.0.14] - 2026-07-18
+
+### Added
+
+- `base_name` / `base_name_mode` filtering on `Catalog.list_recordings()` and `Catalog.prefetch()` — recordings can now be queried and bulk-downloaded by filename stem (exact match, prefix, or substring), not just by `drive_path`.
+- `MatchMode` type (`Literal["exact", "prefix", "contains"]`), exported from the package root, describing how a filter value is matched.
+- `Catalog.get_recording()` accepts an optional `drive_path` argument to disambiguate recordings that share a `base_name` at different Drive locations.
+
+### Changed
+
+- **Breaking:** `Catalog.list_recordings()`, `Catalog.list_items()`, and `Catalog.prefetch()` replace the `drive_path_prefix` / `drive_path_contains` parameters with a single `drive_path_mode: MatchMode` parameter (default `"exact"`), paired with `drive_path`.
+- **Breaking:** All filter parameters on `list_recordings()`, `list_items()`, and `prefetch()` are now keyword-only.
+
+## [0.0.13] - 2026-07-16
+
+### Added
+
+- `size: int | None` field on `RecordingEntry` and `DriveItemEntry`, populated from Drive's file size metadata during `scan()` (no download required). For recordings, it's the combined size of the constituent xdat files that reported a size. `None` for folders and for files Drive doesn't report a size for (e.g. Google Workspace-native Docs/Sheets/Slides). Older catalog files without this field continue to load fine (`size` defaults to `None`).
+- `Catalog.summary(rescan=False, verbose=False)` — a human-readable report of the whole catalog. The default output is headline-only: recording/item counts and local-download progress (as a table), the upload-date range, and Drive/local storage size (broken out separately for recordings and items, with any unsized entries called out rather than silently dropped from the total, plus average recording size and an incomplete-recordings count broken down by missing file type). Pass `rescan=True` to refresh the catalog first. Pass `verbose=True` for everything that scales with catalog size: an item-type breakdown, the largest entries by size, the itemized incomplete-recordings list, a per-top-level-folder breakdown, and a full per-entry listing with size and local status.
+
+### Changed
+
+- `Config.from_file()` auto-discovery of `.secrets/config.json` / `config.json` now searches upward from the current working directory through every parent directory (closest match wins), instead of checking only the current working directory. This mirrors how git locates `.git`, so discovery works the same whether you run from the repo root or a nested subdirectory (e.g. a notebook in `notebooks/`).
+
+### Fixed
+
+- The `~/.config/radiens-drive/config.json` fallback location now actually works. Previously it was silently unreachable: the discovery code joined it onto the current working directory without expanding `~` first, so it could never match a real path.
+
+## [0.0.12] - 2026-07-16
+
+### Fixed
+
+- `Config.from_file()` now resolves relative paths (`credentials_path`, `local_data_dir`, `catalog_path`) against the config file's own directory instead of the process's current working directory. Previously, loading a config with relative paths from a different working directory (e.g. a Jupyter notebook in a subdirectory) would resolve those paths incorrectly, forcing absolute paths as a workaround. Direct `Config(...)` construction is unaffected and still resolves relative paths against the current working directory.
+
+## [0.0.11] - 2026-06-08
+
+### Added
+
+- `RecordingEntry` TypedDict — replaces `DatasetEntry` (fields unchanged).
+- `DriveItemEntry` TypedDict — replaces `AssetEntry`; `asset_name`/`asset_type` fields replaced by `name: str` and `is_folder: bool`.
+- `AmbiguousRecordingError` exception (subclass of `CatalogError`) — raised by `get_recording()` when more than one recording matches a given `base_name`.
+- `Catalog.get_recording(base_name)` — look up a recording by base name, requiring it to be unique across all drive paths. Raises `EntryNotFoundError` (zero matches) or `AmbiguousRecordingError` (multiple matches).
+- `Catalog.file_tree()` — returns a multi-line string rendering of the Drive folder hierarchy, annotating each entry as `[recording]`, `[folder]`, or `[file]` and `[local]` or `[not local]`.
+- `xdat_recording_exists_locally()` utility — replaces `xdat_dataset_exists_locally()`.
+
+### Changed
+
+- **Breaking:** `DatasetEntry` renamed to `RecordingEntry`; `AssetEntry` renamed to `DriveItemEntry`.
+- **Breaking:** `Catalog.recordings_df` replaces `Catalog.df`; `Catalog.items_df` replaces `Catalog.assets_df`.
+- **Breaking:** `Catalog.list_recordings()` replaces `Catalog.list_datasets()`; `Catalog.list_items()` replaces `Catalog.list_assets()`. The `asset_type: str | None` filter on `list_items()` is replaced by `is_folder: bool | None`.
+- **Breaking:** `Catalog.download_recording()` replaces `Catalog.download_dataset()`; `Catalog.get_recording_path()` replaces `Catalog.get_dataset_path()`.
+- **Breaking:** `Catalog.download_item()` replaces `Catalog.download_asset()`; `Catalog.get_item_path()` replaces `Catalog.get_asset_path()`.
+- **Breaking:** `ScanResult` fields renamed: `datasets_new/existing/removed` → `recordings_new/existing/removed`; `assets_new/existing/removed` → `items_new/existing/removed`.
+- **Breaking:** `PrefetchResult` fields renamed: `datasets_downloaded/skipped` → `recordings_downloaded/skipped`; `assets_downloaded/skipped` → `items_downloaded/skipped`.
+- **Breaking:** `prefetch()` parameter `asset_type` replaced by `is_folder: bool | None`.
+- **Breaking:** Item local paths simplified from `{local_data_dir}/assets/{drive_path}/{name}` to `{local_data_dir}/{drive_path}/{name}`, matching the same Drive-mirroring convention used for recordings.
+- `scan_drive()` and `scan_drive_flat()` now catalog **all** non-xdat files and **all** non-root folders as `DriveItemEntry` regardless of depth. The previous depth-based `_classify_item` heuristic is removed entirely — consumer code owns the semantic classification of items.
+- `_load_catalog` reads both new (`"recordings"`/`"items"`) and old (`"datasets"`/`"assets"`) JSON keys, and handles both `name`/`is_folder` and legacy `asset_name`/`asset_type` field names, so existing catalog files continue to load without error.
+
+### Removed
+
+- `DatasetEntry`, `AssetEntry` — replaced by `RecordingEntry` and `DriveItemEntry` respectively.
+- `xdat_dataset_exists_locally()` — replaced by `xdat_recording_exists_locally()`.
+- Depth-based `_classify_item()` internal function — catalog no longer applies structural heuristics to classify items.
+
+## [0.0.10] - 2026-06-04
+
+### Added
+
+- `xdat_dataset_exists_locally()` utility to verify all three canonical xdat files (`_data.xdat`, `.xdat.json`, `_timestamp.xdat`) exist locally.
+- `Catalog.scan()` now warns when a dataset on Drive is missing one or more required xdat files.
+- `Catalog.get_dataset_path()` and `Catalog.get_asset_path()` now automatically trigger a `scan()` if the requested entry is not found in the local catalog, improving resilience to out-of-date catalogs.
+
+### Changed
+
+- Catalog JSON writes are now atomic (using a temporary file and `replace()`) to prevent data loss during interruptions.
+- File downloads are now atomic (using a `.tmp` suffix) to ensure interrupted downloads do not leave partial or corrupted files on disk.
+- `Catalog.scan()` now reconciles local datasets by verifying the presence of all three xdat files, preventing a dataset from being marked "local" if it is incomplete.
+- Refactored `Catalog` internal path restoration logic into `_restore_dataset_local_paths()` and `_restore_asset_local_paths()`.
+
+### Fixed
+
+- Improved validation of asset entries during catalog load; `drive_path` must now be a non-empty string.
+- Streamlined local path construction for assets in `download_asset()`.
+
+## [0.0.9] - 2026-05-20
+
+### Added
+
+- `upload_time` column in `Catalog.df` and `Catalog.assets_df`: a UTC-aware
+  `datetime64` column populated from the Drive API's `createdTime` field during
+  `scan()`. Rows without a recorded date are `NaT`. Existing catalogs without
+  this field load cleanly — missing values default to `None` / `NaT`.
+
+### Changed
+
+- `Catalog.prefetch()` progress bar now covers only items being downloaded
+  (not skipped ones), so the total count and ETA reflect real work. The bar is
+  suppressed entirely when all matching items are already local.
+- Skipped items in `prefetch()` are logged at `DEBUG` level instead of being
+  silently counted.
+- Per-item download log messages in `Catalog.download_dataset()` and
+  `Catalog.download_asset()` lowered from `INFO` to `DEBUG`. The `prefetch()`
+  progress bar is the intended progress signal for bulk operations; direct
+  single-item calls remain observable via debug logging.
+
+## [0.0.8] - 2026-04-21
+
+### Added
+
+- `Catalog.scan()` now returns a `ScanResult` dataclass with counts of
+  `datasets_new`, `datasets_existing`, `datasets_removed`, `assets_new`,
+  `assets_existing`, and `assets_removed` — making it easy to inspect
+  what changed after a Drive rescan. `ScanResult.__str__` renders as a
+  compact table.
+- `ScanResult` dataclass exported from the top-level package.
+- Progress bar during Drive file traversal in `scan()` (respects the
+  `quiet=` flag passed to `Catalog`).
+- Progress bar in `prefetch()` showing overall item count and the name
+  of the entry currently being downloaded (also respects `quiet=`).
+- `Catalog.prefetch(drive_path, drive_path_prefix, drive_path_contains,
+  datasets=True, assets=True, asset_type=None, force=False)` — bulk,
+  idempotent downloader that skips entries already present on disk.
+  Pass `force=True` to re-download every matching entry regardless of
+  local state. Returns a `PrefetchResult` with per-category download
+  and skip counts.
+- `PrefetchResult` dataclass exported from the top-level package.
+- `CatalogError` and `EntryNotFoundError` exceptions exported from the
+  top-level package. `EntryNotFoundError` (subclass of `CatalogError`)
+  is raised when a dataset or asset lookup fails; `CatalogError` is
+  raised for malformed catalog JSON.
+- `Catalog.list_datasets(drive_path, drive_path_prefix,
+  drive_path_contains)` — convenience method returning a filtered
+  dataset DataFrame.
+- `Catalog.list_assets(drive_path, drive_path_prefix,
+  drive_path_contains, asset_type)` — convenience method returning a
+  filtered asset DataFrame. Supports an additional `asset_type="folder"`
+  or `"file"` filter.
+- `quiet` keyword argument on `Catalog(...)` suppresses tqdm progress
+  bars during downloads.
+- `DatasetEntry` TypedDict exported from the top-level package
+  (previously only `AssetEntry` was exported).
+
+### Changed
+
+- `Catalog.scan()` return type changed from `None` to `ScanResult`.
+- Drive scan warnings (e.g. incomplete search results) are now emitted
+  via `logging.getLogger` rather than `warnings.warn`.
+- Dataset identity is now `(drive_path, base_name)` instead of
+  `base_name` alone, allowing distinct recordings with the same base
+  name in different folders.
+- Dataset downloads now mirror Drive hierarchy under local storage:
+  `local_data_dir/{drive_path}/...`.
+- `Catalog.df` and `Catalog.assets_df` now use a path-first schema and
+  no longer include legacy `date_folder` / `experiment` columns.
+- **Breaking:** `Catalog.download()` renamed to
+  `Catalog.download_dataset()` for symmetry with `download_asset()`.
+- **Breaking:** `Catalog.get_path()` renamed to
+  `Catalog.get_dataset_path()` for symmetry with `get_asset_path()`.
+- `Catalog.download_dataset()` and `Catalog.get_dataset_path()` now
+  require both `drive_path` and `base_name` to identify a dataset.
+- Catalog JSON parse errors now raise `CatalogError` instead of
+  `ValueError`. "Not found" errors on single-item download / get_path
+  methods now raise `EntryNotFoundError` instead of `ValueError`.
+
+### Removed
+
+- **Breaking:** `Catalog(config_path=...)` constructor shortcut removed.
+  Use `Catalog(Config.from_file("path"))` or `Catalog()` (which calls
+  `Config.from_file()` with auto-discovery).
+- Removed `Catalog.list_experiment_paths()`, `Catalog.list_by_path()`,
+  and `Catalog.status()` convenience methods. Use
+  `Catalog.list_datasets()` / `Catalog.list_assets()` or operate on the
+  raw DataFrames directly.
+
+### Fixed
+
+- `Catalog.scan()` now preserves `local_path` for datasets by
+  `(drive_path, base_name)` during refresh, preventing path collisions
+  when multiple folders contain the same `base_name`.
+- `py.typed` marker (PEP 561) — the package now declares itself as typed,
+  allowing mypy to use its annotations in downstream consumers.
+- `Catalog.scan()` now reconciles `local_path` against disk rather than
+  trusting the previous catalog blindly. If the catalog is deleted,
+  entries whose data already exists at the canonical local path
+  (`local_data_dir/{drive_path}` for datasets,
+  `local_data_dir/assets/{drive_path}/{asset_name}` for assets) are
+  recovered automatically instead of triggering unnecessary re-downloads.
+  Stale catalog paths that no longer exist on disk are also cleared.
+
+## [v0.0.7] - 2026-04-08
+
+### Changed
+
+- `Catalog.scan()` now defaults to the flat scanner (`flat=True`), which is
+  faster and works even when the root folder is inaccessible. Pass
+  `flat=False` to use the recursive traversal.
+- Removed automatic fallback logic from `scan()`. The caller now chooses the
+  scan strategy via the `flat` keyword argument.
+
+## [v0.0.6] - 2026-04-07
+
+### Added
+
+- `Catalog.list_experiment_paths()` and `Catalog.list_by_path()` to help
+  discover and filter datasets by their full Drive path.
+
+### Fixed
+
+- Depth-based asset cataloging: nested subfolders inside experiments (depth 3+)
+  are now correctly cataloged as folder assets. Previously only direct
+  subfolders (depth 2) were included.
+
+## [v0.0.5] - 2026-04-06
+
+### Added
+
+- Flat-scan fallback: when the root folder is inaccessible (HTTP 403/404) or
+  returns an empty listing, `Catalog.scan()` automatically falls back to a
+  flat `files.list()` of all files visible to the service account and
+  reconstructs the folder hierarchy from `parents` metadata. A
+  `UserWarning` is emitted when the fallback activates.
+- `scan_drive_flat()` internal function for flat-listing all Drive files with
+  paginated `pageSize=1000` requests and parent-chain path reconstruction.
+- `Catalog` can now be created without a pre-built `Config` object:
+  `Catalog(config_path="/path/to/config.json")` and `Catalog()` are both
+  supported.
+
+### Changed
+
+- `Config.from_file()` discovery now also checks
+  `~/.config/radiens-drive/config.json` and `/etc/radiens-drive/config.json`
+  after the existing environment variable and local path checks.
+
+## [0.0.4] - 2026-04-02
+
+### Fixed
+
+## [0.0.3] - 2026-04-02
+
+### Changed
+
+- `Config.from_file()` now auto-discovers config files when called with no
+  arguments: checks `RADIENS_DRIVE_CATALOG_CONFIG` env var, then
+  `.secrets/config.json`, then `config.json` in the current working directory.
+  Raises `FileNotFoundError` (previously `ValueError`) when no config is found.
+
+## [0.0.2] - 2026-04-02
+
+### Added
+
+- `AssetEntry` TypedDict for non-xdat Drive content (folders like `logs/`,
+  PowerPoints, writeups, etc.).
+- `scan_drive()` now returns `(datasets, assets)` tuple. Assets are
+  auto-discovered during scan: non-xdat files inside date or experiment
+  folders become file assets; subfolders of experiment folders become folder
+  assets and are still recursed for xdat datasets.
+- `download_asset()` in `drive.py` for downloading file or folder assets;
+  folder assets are downloaded recursively, mirroring the Drive subtree.
+- `Catalog.assets_df` property — full asset catalog as a pandas DataFrame.
+- `Catalog.list_assets()` — query assets with optional `date_folder`,
+  `experiment`, and `asset_type` filters.
+- `Catalog.download_asset(drive_path, asset_name)` — download an asset to
+  `local_data_dir/assets/{drive_path}/{asset_name}`.
+- `Catalog.get_asset_path(drive_path, asset_name)` — return local path,
+  downloading automatically if needed.
+- Catalog JSON format changed from a bare list to
+  `{"datasets": [...], "assets": [...]}`. Old flat-list catalogs are
+  migrated automatically on the next `scan()`.
+- `AssetEntry` exported from the top-level package.
+
+### Changed
+
+- `Catalog.list()` and `Catalog.list_assets()`: renamed the `date` parameter
+  to `date_folder` to match the catalog column name and make clear that an
+  exact folder name (e.g. `"2026-02-15_batch"`) is required, not a date
+  prefix.
+
+## [0.0.1] - 2026-04-02
+
+### Added
+
+- `Config` dataclass with `from_file()` classmethod; supports `~` and `$ENV_VAR`
+  expansion in path fields and `RADIENS_DRIVE_CATALOG_CONFIG` env var fallback.
+- `build_drive_service()` for authenticating with a Google service account.
+- `scan_drive()` for recursive Drive scanning; returns `DatasetEntry` records
+  with `date_folder`, `experiment`, `drive_path`, and `drive_file_ids`.
+- `download_dataset()` for chunked download of xdat filesets to a local directory.
+- `Catalog` class with `scan()`, `df`, `list()`, `download()`, `get_path()`,
+  and `status()`.
+- MkDocs documentation site with Material theme, auto-generated API reference,
+  and versioned deployment via `mike`.
+- CI/CD workflows: linting, type checking, tests, docs deployment, and PyPI
+  publishing via GitHub Actions.

radiens_data_catalog-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 NeuroNexus
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

radiens_data_catalog-0.1.0/PKG-INFO

@@ -0,0 +1,156 @@
+Metadata-Version: 2.5
+Name: radiens-data-catalog
+Version: 0.1.0
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: google-api-python-client>=2.188.0
+Requires-Dist: google-auth-oauthlib>=1.2.4
+Requires-Dist: google-auth>=2.47.0
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: tqdm>=4.67.3
+Description-Content-Type: text/markdown
+
+# radiens-data-catalog
+
+A Python package for programmatically managing large neural datasets stored on Google Drive. It handles Drive scanning, local cataloging, and selective dataset download. Analysis is done locally — this package is purely about data management.
+
+Documentation: <https://neuronexus.github.io/radiens-data-catalog/latest/>
+
+## Overview
+
+Neural data is stored as xdat filesets (NeuroNexus format) on a shared Google Drive. Each dataset consists of 3 files sharing a common `base_name`:
+
+```
+{base_name}_data.xdat
+{base_name}.xdat.json
+{base_name}_timestamp.xdat
+```
+
+`radiens-data-catalog` scans the Drive hierarchy, builds a local catalog indexed by `base_name`, and lets you query and download datasets selectively. Non-xdat content found alongside recordings — logs directories, PowerPoints, writeups — is also discovered and tracked as **Drive items**.
+
+## Usage
+
+### Recordings
+
+```python
+from radiens_data_catalog import Catalog, Config
+
+config = Config.from_file("config.json")
+catalog = Catalog(config)
+
+# Scan Drive and build the catalog (discovers recordings and Drive items)
+result = catalog.scan()      # returns ScanResult with new/existing/removed counts
+result = catalog.scan(flat=False)  # recursive traversal instead of flat file scan
+
+# Query recordings using pandas directly
+catalog.recordings_df
+catalog.list_recordings()                                                                    # everything
+catalog.list_recordings(drive_path="2026-02-15_batch/reaching")                             # exact folder
+catalog.list_recordings(drive_path="2026-02-15_batch", drive_path_mode="prefix")            # full date subtree
+catalog.list_recordings(drive_path="reaching", drive_path_mode="contains")                   # any depth
+catalog.list_recordings(base_name="rat01_session3")                                          # exact base_name
+catalog.list_recordings(base_name="rat01", base_name_mode="prefix")                           # base_name prefix
+catalog.list_recordings(base_name="session3", base_name_mode="contains")                     # base_name substring
+
+# Filters are ANDed together, e.g. all "rat01" recordings under "2026-02-15_batch"
+catalog.list_recordings(drive_path="2026-02-15_batch", drive_path_mode="prefix", base_name="rat01", base_name_mode="prefix")
+
+# Look up a recording by base_name (raises AmbiguousRecordingError if not globally
+# unique — pass drive_path to disambiguate)
+entry = catalog.get_recording("rat01_session3")
+entry = catalog.get_recording("rat01_session3", drive_path="2026-02-15_batch/reaching")
+
+# Download a recording (3 xdat files)
+catalog.download_recording("2026-02-15_batch/reaching", "rat01_session3")
+
+# Get the local path, downloading automatically if needed
+path = catalog.get_recording_path("2026-02-15_batch/reaching", "rat01_session3")
+```
+
+### Drive items (non-xdat content)
+
+Non-xdat files and folders (e.g. `logs/`, PowerPoints, writeups) are automatically cataloged as Drive items during `scan()`.
+
+```python
+# Query items using pandas directly
+catalog.items_df
+catalog.items_df[catalog.items_df["drive_path"].str.startswith("2026-02-15_batch")]
+catalog.items_df[catalog.items_df["is_folder"] == True]
+
+# Download an item (drive_path is the slash-joined path to the item's parent folder)
+catalog.download_item("2026-02-15_batch/reaching", "logs")
+
+# Get the local path, downloading automatically if needed
+path = catalog.get_item_path("2026-02-15_batch/reaching", "logs")
+
+# Render an annotated tree of the full Drive hierarchy
+print(catalog.file_tree())
+
+# Print a headline summary report: counts, download progress, date
+# range, and Drive/local storage totals
+print(catalog.summary())
+
+# verbose=True adds an item-type breakdown, largest entries, itemized
+# incomplete recordings, a per-folder breakdown, and a full per-entry listing
+print(catalog.summary(verbose=True))
+```
+
+Items land under `local_data_dir/{drive_path}/{name}`, using the same Drive-mirroring convention as recordings.
+
+## Configuration
+
+Create a `config.json` (outside your repo — do not commit it):
+
+```json
+{
+    "credentials_path": "/path/to/service_account.json",
+    "root_folder_id": "your-drive-folder-id",
+    "local_data_dir": "/path/to/local/data",
+    "catalog_path": "/path/to/local/data/catalog.json"
+}
+```
+
+`Config.from_file()` locates the config file using this resolution order:
+
+1. Explicit `path` argument.
+2. `RADIENS_DATA_CATALOG_CONFIG` environment variable.
+3. `.secrets/config.json`, then `config.json`, searched starting in the current working directory and then each parent directory up to the filesystem root (closest directory wins) — the same convention git uses to locate `.git`, so this works the same whether you run from the repo root or a nested subdirectory (e.g. a notebook in `notebooks/`).
+4. `~/.config/radiens-drive/config.json`.
+5. `/etc/radiens-drive/config.json`.
+
+```python
+# Automatic discovery (env var or well-known paths)
+config = Config.from_file()
+
+# Explicit path
+config = Config.from_file("/path/to/config.json")
+```
+
+The `root_folder_id` is the alphanumeric string in the Drive URL when you're inside the root data folder.
+
+## Authentication
+
+This package uses a Google service account for shared access among collaborators. To set it up:
+
+1. Create a project in [Google Cloud Console](https://console.cloud.google.com)
+2. Enable the Google Drive API
+3. Create a service account and download its JSON credentials file
+4. Share your root Drive data folder with the service account's email address (Viewer access is sufficient)
+5. Point `credentials_path` in your config at the downloaded JSON file
+
+Distribute the credentials file to collaborators securely — treat it like a password.
+
+## Installation
+
+```bash
+pip install radiens-data-catalog
+```
+
+or, with [uv](https://github.com/astral-sh/uv):
+
+```bash
+uv add radiens-data-catalog
+```
+
+Requires Python 3.12+ and a Google service account with read access to the root
+Drive folder (see [Authentication](#authentication) above).