containerio 2.4.0__tar.gz → 2.5.0__tar.gz

{containerio-2.4.0 → containerio-2.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: containerio
-Version: 2.4.0
+Version: 2.5.0
 Summary: Azure Blob Storage handler with unified local/cloud interface
 Author: Alliance SwissPass
 Requires-Dist: polars>=1.30.0
@@ -20,7 +20,7 @@ Provides a unified interface for reading and writing data to both local filesyst
 ## Features
 
 - **Unified API**: Same code works for local files and Azure Blob Storage
-- **Flexible configuration**: Load Azure config from a dotenv file, from existing environment variables, or directly from strings (e.g. values from a TOML file)
+- **Flexible configuration**: Load Azure config from a dotenv file, environment variables, strings, or a `container-config.toml` with direnv-populated secrets (see [TOML + direnv configuration](#toml--direnv-configuration))
 - **[Polars](https://pola.rs/) integration**: Native support for Polars DataFrames and LazyFrames
 - **Multiple auth methods**: Device code flow (default), client credentials (CI/CD), or SAS tokens
 - **CLI tools**: Unified `io` command for listing, downloading, uploading, deleting blobs and generating SAS tokens
@@ -81,7 +81,9 @@ containerio = { git = "https://codefloe.com/Alliance-SwissPass/py-containerio.gi
 When installed as a dependency, containerio integrates seamlessly with your project:
 
 - **CLI tools are available** - Run `uv run io` directly from your project
-- **Dotenv is the default convention** - The CLI best-effort-loads `.env.<name>` matching `--env <name>` from your current working directory. Other formats are supported via a small wrapper script — see [Providing config to the CLI](#providing-config-to-the-cli).
+- **Dotenv is the default convention** - The CLI best-effort-loads `.env.<name>` matching `--env <name>` from your current working directory.
+- **Native TOML + direnv support** - Pass `--container-config` to resolve `--container` as a logical key from a `container-config.toml` with direnv-populated secrets — see [TOML + direnv configuration](#toml--direnv-configuration).
+- **Other formats via a small wrapper script** - See [Providing config to the CLI](#providing-config-to-the-cli).
 - **No configuration in containerio needed** - Just add the dependency and configure in your project
 
 Example workflow in your project:
@@ -218,7 +220,109 @@ AZURE_CLIENT_SECRET_CONTAINER=<container_app_client_secret>
 AZURE_STORAGE_SAS_TOKEN_RAW_DATA_STG=<sas_token_for_staging_container>
 ```
 
-> **Note:** There is no registry of containers — you simply pass `--container <name>` (or `container="..."` in code) and the call is forwarded to Azure. Whether the container exists and what you can do with it is decided by Azure RBAC on your identity.
+> **Note:** There is no registry of containers — you simply pass `--container <name>` (or `container="..."` in code) and the call is forwarded to Azure.
+> Whether the container exists and what you can do with it is decided by Azure RBAC on your identity.
+
+### TOML + direnv configuration
+
+For projects with several logical containers across environments, py-containerio supports a combined TOML + direnv approach (the Python-native equivalent of r-containerio's `containerio_config()`):
+
+- Non-sensitive fields (storage account, container names, and identity IDs) live in a project-local `container-config.toml`, which is safe to commit.
+- Sensitive values (client secrets) never appear in the TOML: `client_secret_env` names the environment variable that holds the secret, resolved from `os.environ` at load time.
+- `tenant_id`, `client_id`, and `device_code_client_id` are resolved environment-first: their `*_env` keys name environment variables that win when set at runtime, and the TOML value is only the fallback.
+- The environment is populated through direnv: a commit-safe `.envrc` that pulls values at runtime, e.g. from Azure Key Vault via the authenticated `az` CLI.
+  direnv only fills gaps (`${VAR:-...}`), so a value injected by CI (Crow `from_secret` becomes a plain environment variable) or an existing shell export stays authoritative.
+
+Copy `container-config_example.toml` from this repository to `container-config.toml` and adapt it.
+Each container maps an environment (`staging`, `prod`) to a concrete container name and an Azure auth configuration. Auth can be referenced in two ways:
+
+- **Tag indirection** (shared auth across containers): `azure_auth.<env>` is a string referencing a shared `[azure_auth.<tag>]` section.
+- **Inline nested auth** (per-container): `azure_auth.<env>` is a nested table defining the auth fields directly under `[containers.<key>.azure_auth.<env>]`.
+
+If the resolved auth spec defines no `storage_account`, the full configuration is read from environment variables via `AzureStorageConfig.from_env()` instead.
+
+#### Example `.envrc` (direnv)
+
+The `.envrc` is committed; the secrets it pulls are not. Secrets are fetched at shell-entry time with the already-authenticated `az` CLI:
+
+```bash
+# Load the base tool environment (optional, e.g. uv/mise).
+use flake
+
+# Fetch a Key Vault secret into an env var at direnv load time.
+# Requires a logged-in 'az' CLI with access to the vault.
+azure_kv_secret() {
+  az keyvault secret show \
+    --vault-name "$1" \
+    --name "$2" \
+    --query value \
+    --output tsv
+}
+
+# Identity values live in a distinct vault from the application secrets.
+# Env-first: only fetch when the variable is not already set, so CI
+# (Crow from_secret -> plain env var) and existing exports win.
+export AZURE_TENANT_ID="${AZURE_TENANT_ID:-$(azure_kv_secret <identity-vault-name> <tenant-id-secret-name>)}"
+export AZURE_CLIENT_ID_CONTAINER="${AZURE_CLIENT_ID_CONTAINER:-$(azure_kv_secret <identity-vault-name> <client-id-secret-name>)}"
+export AZURE_CLIENT_ID="${AZURE_CLIENT_ID:-$(azure_kv_secret <identity-vault-name> <device-code-client-id-secret-name>)}"
+
+export AZURE_CLIENT_SECRET_CONTAINER_RAW_STAGING="${AZURE_CLIENT_SECRET_CONTAINER_RAW_STAGING:-$(azure_kv_secret <app-vault-name> <secret-name-staging>)}"
+export AZURE_CLIENT_SECRET_CONTAINER_RAW_PROD="${AZURE_CLIENT_SECRET_CONTAINER_RAW_PROD:-$(azure_kv_secret <app-vault-name> <secret-name-prod>)}"
+
+# SAS tokens for containers are read by the handler from env vars named
+# AZURE_STORAGE_SAS_TOKEN_<CONTAINER> (dashes -> underscores) and can be
+# populated the same way:
+export AZURE_STORAGE_SAS_TOKEN_RAW_DATA="${AZURE_STORAGE_SAS_TOKEN_RAW_DATA:-$(azure_kv_secret <app-vault-name> <sas-secret-name>)}"
+
+# ARM_SUBSCRIPTION_ID is not used by py-containerio; never hardcode it here.
+```
+
+Allow the file once with `direnv allow`.
+See `examples/toml-direnv-demo/envrc.example` for a full template that fetches several values in parallel, classifies failures, and skips variables that are already set.
+
+#### Library usage
+
+```python
+from containerio import resolve_container_config, StorageHandler
+
+# Resolve a logical key for an environment.
+# Returns (container_name, AzureStorageConfig); both None for env="local".
+container, config = resolve_container_config(
+    container_key="input1",
+    env="staging",
+)
+
+handler = StorageHandler(container=container, config=config)
+```
+
+Errors mirror the R wrapper: unknown container key, missing auth for the selected env, unknown auth tag, missing container name, and a malformed `name`/`azure_auth` table all raise `ValueError`.
+A missing TOML file raises `FileNotFoundError`.
+An unset `client_secret_env` does not raise at resolve time: the secret resolves to `None` and only client-credentials auth reports it, when that method is actually used, so device-code login still works.
+
+#### CLI usage
+
+The `io` command consumes the combined TOML + direnv config directly.
+Pass `--container-config` (optionally with a path, default `container-config.toml` in the current directory) and select the container by its logical key.
+`--env` is required and selects the `name.<env>` / `azure_auth.<env>` entries:
+
+```bash
+# List blobs in the container mapped from logical key 'input1' for staging
+uv run io ls --env staging --container input1 --container-config
+
+# Explicit TOML path
+uv run io ls --env prod --container input1 --container-config path/to/container-config.toml
+
+# All other commands work the same way, e.g.
+uv run io download data/file.csv --env prod --container output --container-config
+uv run io auth --env staging --container input1 --container-config
+```
+
+With `--container-config`, the `--container` value is always interpreted as a logical key under `[containers]`.
+The `io session` command stores both the logical key and the TOML path, so a saved session keeps resolving through the TOML on later runs.
+
+> **Note:** When using `--container-config`, pass `--env` explicitly.
+>
+> The dotenv preload of the default CLI flow is unaffected and keeps working; secrets reach the process through direnv or plain exports.
 
 ### Authentication
 
@@ -272,14 +376,18 @@ The chosen auth method is saved to the session. Subsequent commands (`ls`, `down
 
 ## CLI Tools
 
-The package provides a unified `io` command with subcommands for all storage operations. All commands accept `--env prod|staging` (selects the env file) and `--container <name>` (selects the Azure container).
+The package provides a unified `io` command with subcommands for all storage operations.
+All commands accept `--env prod|staging` (selects the env file) and `--container <name>` (selects the Azure container).
+`--container-config` resolves `--container` as a logical key from a `container-config.toml` (see [TOML + direnv configuration](#toml--direnv-configuration)).
 
 | Command | Usage | Description |
 |---------|-------|-------------|
 | `auth` | `uv run io auth [login\|status\|clear] [--method M] [--force]` | Manage authentication |
 | `ls` | `uv run io ls --env <env> --container <name>` | List blobs in a container |
 | `download` | `uv run io download <blob_name> --env <env> --container <name>` | Download a blob to local `data/` directory |
+| `download-dir` | `uv run io download-dir [<prefix>] [--local-dir <path>] --env <env> --container <name>` | Recursively download all blobs under a prefix to a local directory |
 | `upload` | `uv run io upload <blob_name> <file_path> --env <env> --container <name>` | Upload a local file to a container |
+| `upload-dir` | `uv run io upload-dir <local_dir> [--blob-prefix <prefix>] --env <env> --container <name>` | Upload all files from a local directory to storage |
 | `rm` | `uv run io rm <path> --env <env> --container <name>` | Delete a blob or folder from a container |
 | `mv` | `uv run io mv <source> <dest> [--overwrite] --env <env> --container <name>` | Move (rename) a blob within a container |
 | `generate-sas` | `uv run io generate-sas --env <env> --container <name> [--days N] [--update-env] [--write]` | Generate an Azure Storage SAS token for a container (read+list by default; add `--write` for write+delete) |
@@ -369,23 +477,29 @@ The session is also available programmatically via the `Session` class:
 from containerio.session import Session
 
 session = Session()
-print(session.auth_method)  # e.g. 'client-credentials'
-print(session.env)          # e.g. 'staging'
-print(session.container)    # e.g. 'raw-data'
-print(session.is_active)    # True if any session data exists
+print(session.auth_method)      # e.g. 'client-credentials'
+print(session.env)              # e.g. 'staging'
+print(session.container)        # e.g. 'raw-data'
+print(session.container_config) # e.g. 'container-config.toml', or None
+print(session.is_active)        # True if any session data exists
 ```
 
 CLI options:
 
-- `--env <name>`: selects the environment. The CLI best-effort-preloads `.env.<name>` (no-op if missing) and uses the name as a label in prints, sessions, and SAS file write-back. Any name works (`prod`, `staging`, `dev`, `eu-prod`, …); `local` means the local filesystem.
-- `--container`: the Azure container name to target
+- `--env <name>`: selects the environment. The CLI best-effort-preloads `.env.<name>` (no-op if missing) and uses the name as a label in prints, sessions, and SAS file write-back.
+  Any name works (`prod`, `staging`, `dev`, `eu-prod`, …); `local` means the local filesystem.
+- `--container`: the Azure container name to target. With `--container-config`, this is a logical key under `[containers]` in the TOML file.
+- `--container-config [path]`: resolve `--container` as a logical key from a `container-config.toml` (default: `container-config.toml` in the current directory).
+  Secrets come from the environment (e.g. via direnv). See [TOML + direnv configuration](#toml--direnv-configuration).
 - `--days`: SAS token validity in days, max 7 (default: 1, `generate-sas` only)
 - `--update-env`: write SAS token to env file instead of printing (`generate-sas` only)
 - `--write`: opt in to write+delete permissions (`generate-sas` only). Default is read+list.
 
 **Note:** SAS tokens are generated using a *user delegation key* which has a maximum validity of **7 days** (Azure limitation). See [Microsoft documentation](https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key#request-body) for details.
 
-The token's effective permissions are the **intersection** of the permissions requested here and the Azure RBAC permissions held by your identity (the user-delegation-key requester) at generation time — so requesting `--write` only produces a write-capable token if your RBAC role allows it. See [Microsoft documentation on user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#permissions-for-a-user-delegation-sas) for details.
+The token's effective permissions are the **intersection** of the permissions requested here and the Azure RBAC permissions held by your identity (the user-delegation-key requester) at generation time.
+This means requesting `--write` only produces a write-capable token if your RBAC role allows it.
+See [Microsoft documentation on user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#permissions-for-a-user-delegation-sas) for details.
 
 ## API Reference
 
@@ -445,9 +559,9 @@ StorageHandler(
 ### AzureStorageConfig
 
 Configuration dataclass for Azure storage settings.
-Fields: `storage_account`, `tenant_id` (optional), `dotenv_path` (optional — the file the config was loaded from; useful in logs and `repr`), `client_id` (optional), `client_secret` (optional) and `device_code_client_id` (optional).
+Fields: `storage_account`, `tenant_id` (optional), `dotenv_path` (optional), `client_id` (optional), `client_secret` (optional), `device_code_client_id` (optional).
+`dotenv_path` is the file the config was loaded from; it is useful in logs and `repr`.
 `storage_endpoint` is a derived property — `https://<storage_account>.blob.core.windows.net`.
-
 The config is the single carrier of account and identity data: the handler reads these fields and never reads environment variables for them itself.
 `from_env()` is the one seam that maps env vars onto the config (`AZURE_CLIENT_ID_CONTAINER` → `client_id`, `AZURE_CLIENT_SECRET_CONTAINER` → `client_secret`, `AZURE_CLIENT_ID` → `device_code_client_id`).
 The lone exception is the SAS token, which is container-scoped and therefore resolved by the handler from `AZURE_STORAGE_SAS_TOKEN_<NAME>` at use time.
@@ -509,8 +623,8 @@ Errors:
 
 ### Providing config to the CLI
 
-The CLI reads required Azure variables from `os.environ` — it does not parse any specific config format itself.
-This keeps the CLI format-agnostic and lets each project choose how to manage config.
+The CLI reads required Azure variables from `os.environ` — by default it does not parse any specific config format itself, keeping it format-agnostic so each project can choose how to manage config.
+For the native combined TOML + direnv approach, pass `--container-config` (see [TOML + direnv configuration](#toml--direnv-configuration)); the rest of this section covers the default dotenv / env-var flow.
 
 **Dotenv is the default.**
 If a `.env.<name>` file matching `--env <name>` exists, the CLI auto-loads it via `python-dotenv` before reading `os.environ`.
@@ -623,6 +737,7 @@ The `.gitsv/config.yaml` file defines how versions are bumped based on commit me
 4. **Release**: The CI pipeline automatically handles tagging and publishing
 
 The CI pipeline will automatically:
+
 - Bump the version based on commit history
 - Generate and pin the changelog as a Git issue
 - Build and publish the package to Forgejo
@@ -641,6 +756,7 @@ The repository includes a Crow CI pipeline that automatically builds, tests, and
 1. `git sv tag`: generate tag with version and push it to remote in one go, based on git commit parsing.
 
 The pipeline will:
+
 - Run linting and tests
 - Update the version in `pyproject.toml` automatically
 - Generate changelog from git history using `git-sv`

{containerio-2.4.0 → containerio-2.5.0}/README.md

@@ -6,7 +6,7 @@ Provides a unified interface for reading and writing data to both local filesyst
 ## Features
 
 - **Unified API**: Same code works for local files and Azure Blob Storage
-- **Flexible configuration**: Load Azure config from a dotenv file, from existing environment variables, or directly from strings (e.g. values from a TOML file)
+- **Flexible configuration**: Load Azure config from a dotenv file, environment variables, strings, or a `container-config.toml` with direnv-populated secrets (see [TOML + direnv configuration](#toml--direnv-configuration))
 - **[Polars](https://pola.rs/) integration**: Native support for Polars DataFrames and LazyFrames
 - **Multiple auth methods**: Device code flow (default), client credentials (CI/CD), or SAS tokens
 - **CLI tools**: Unified `io` command for listing, downloading, uploading, deleting blobs and generating SAS tokens
@@ -67,7 +67,9 @@ containerio = { git = "https://codefloe.com/Alliance-SwissPass/py-containerio.gi
 When installed as a dependency, containerio integrates seamlessly with your project:
 
 - **CLI tools are available** - Run `uv run io` directly from your project
-- **Dotenv is the default convention** - The CLI best-effort-loads `.env.<name>` matching `--env <name>` from your current working directory. Other formats are supported via a small wrapper script — see [Providing config to the CLI](#providing-config-to-the-cli).
+- **Dotenv is the default convention** - The CLI best-effort-loads `.env.<name>` matching `--env <name>` from your current working directory.
+- **Native TOML + direnv support** - Pass `--container-config` to resolve `--container` as a logical key from a `container-config.toml` with direnv-populated secrets — see [TOML + direnv configuration](#toml--direnv-configuration).
+- **Other formats via a small wrapper script** - See [Providing config to the CLI](#providing-config-to-the-cli).
 - **No configuration in containerio needed** - Just add the dependency and configure in your project
 
 Example workflow in your project:
@@ -204,7 +206,109 @@ AZURE_CLIENT_SECRET_CONTAINER=<container_app_client_secret>
 AZURE_STORAGE_SAS_TOKEN_RAW_DATA_STG=<sas_token_for_staging_container>
 ```
 
-> **Note:** There is no registry of containers — you simply pass `--container <name>` (or `container="..."` in code) and the call is forwarded to Azure. Whether the container exists and what you can do with it is decided by Azure RBAC on your identity.
+> **Note:** There is no registry of containers — you simply pass `--container <name>` (or `container="..."` in code) and the call is forwarded to Azure.
+> Whether the container exists and what you can do with it is decided by Azure RBAC on your identity.
+
+### TOML + direnv configuration
+
+For projects with several logical containers across environments, py-containerio supports a combined TOML + direnv approach (the Python-native equivalent of r-containerio's `containerio_config()`):
+
+- Non-sensitive fields (storage account, container names, and identity IDs) live in a project-local `container-config.toml`, which is safe to commit.
+- Sensitive values (client secrets) never appear in the TOML: `client_secret_env` names the environment variable that holds the secret, resolved from `os.environ` at load time.
+- `tenant_id`, `client_id`, and `device_code_client_id` are resolved environment-first: their `*_env` keys name environment variables that win when set at runtime, and the TOML value is only the fallback.
+- The environment is populated through direnv: a commit-safe `.envrc` that pulls values at runtime, e.g. from Azure Key Vault via the authenticated `az` CLI.
+  direnv only fills gaps (`${VAR:-...}`), so a value injected by CI (Crow `from_secret` becomes a plain environment variable) or an existing shell export stays authoritative.
+
+Copy `container-config_example.toml` from this repository to `container-config.toml` and adapt it.
+Each container maps an environment (`staging`, `prod`) to a concrete container name and an Azure auth configuration. Auth can be referenced in two ways:
+
+- **Tag indirection** (shared auth across containers): `azure_auth.<env>` is a string referencing a shared `[azure_auth.<tag>]` section.
+- **Inline nested auth** (per-container): `azure_auth.<env>` is a nested table defining the auth fields directly under `[containers.<key>.azure_auth.<env>]`.
+
+If the resolved auth spec defines no `storage_account`, the full configuration is read from environment variables via `AzureStorageConfig.from_env()` instead.
+
+#### Example `.envrc` (direnv)
+
+The `.envrc` is committed; the secrets it pulls are not. Secrets are fetched at shell-entry time with the already-authenticated `az` CLI:
+
+```bash
+# Load the base tool environment (optional, e.g. uv/mise).
+use flake
+
+# Fetch a Key Vault secret into an env var at direnv load time.
+# Requires a logged-in 'az' CLI with access to the vault.
+azure_kv_secret() {
+  az keyvault secret show \
+    --vault-name "$1" \
+    --name "$2" \
+    --query value \
+    --output tsv
+}
+
+# Identity values live in a distinct vault from the application secrets.
+# Env-first: only fetch when the variable is not already set, so CI
+# (Crow from_secret -> plain env var) and existing exports win.
+export AZURE_TENANT_ID="${AZURE_TENANT_ID:-$(azure_kv_secret <identity-vault-name> <tenant-id-secret-name>)}"
+export AZURE_CLIENT_ID_CONTAINER="${AZURE_CLIENT_ID_CONTAINER:-$(azure_kv_secret <identity-vault-name> <client-id-secret-name>)}"
+export AZURE_CLIENT_ID="${AZURE_CLIENT_ID:-$(azure_kv_secret <identity-vault-name> <device-code-client-id-secret-name>)}"
+
+export AZURE_CLIENT_SECRET_CONTAINER_RAW_STAGING="${AZURE_CLIENT_SECRET_CONTAINER_RAW_STAGING:-$(azure_kv_secret <app-vault-name> <secret-name-staging>)}"
+export AZURE_CLIENT_SECRET_CONTAINER_RAW_PROD="${AZURE_CLIENT_SECRET_CONTAINER_RAW_PROD:-$(azure_kv_secret <app-vault-name> <secret-name-prod>)}"
+
+# SAS tokens for containers are read by the handler from env vars named
+# AZURE_STORAGE_SAS_TOKEN_<CONTAINER> (dashes -> underscores) and can be
+# populated the same way:
+export AZURE_STORAGE_SAS_TOKEN_RAW_DATA="${AZURE_STORAGE_SAS_TOKEN_RAW_DATA:-$(azure_kv_secret <app-vault-name> <sas-secret-name>)}"
+
+# ARM_SUBSCRIPTION_ID is not used by py-containerio; never hardcode it here.
+```
+
+Allow the file once with `direnv allow`.
+See `examples/toml-direnv-demo/envrc.example` for a full template that fetches several values in parallel, classifies failures, and skips variables that are already set.
+
+#### Library usage
+
+```python
+from containerio import resolve_container_config, StorageHandler
+
+# Resolve a logical key for an environment.
+# Returns (container_name, AzureStorageConfig); both None for env="local".
+container, config = resolve_container_config(
+    container_key="input1",
+    env="staging",
+)
+
+handler = StorageHandler(container=container, config=config)
+```
+
+Errors mirror the R wrapper: unknown container key, missing auth for the selected env, unknown auth tag, missing container name, and a malformed `name`/`azure_auth` table all raise `ValueError`.
+A missing TOML file raises `FileNotFoundError`.
+An unset `client_secret_env` does not raise at resolve time: the secret resolves to `None` and only client-credentials auth reports it, when that method is actually used, so device-code login still works.
+
+#### CLI usage
+
+The `io` command consumes the combined TOML + direnv config directly.
+Pass `--container-config` (optionally with a path, default `container-config.toml` in the current directory) and select the container by its logical key.
+`--env` is required and selects the `name.<env>` / `azure_auth.<env>` entries:
+
+```bash
+# List blobs in the container mapped from logical key 'input1' for staging
+uv run io ls --env staging --container input1 --container-config
+
+# Explicit TOML path
+uv run io ls --env prod --container input1 --container-config path/to/container-config.toml
+
+# All other commands work the same way, e.g.
+uv run io download data/file.csv --env prod --container output --container-config
+uv run io auth --env staging --container input1 --container-config
+```
+
+With `--container-config`, the `--container` value is always interpreted as a logical key under `[containers]`.
+The `io session` command stores both the logical key and the TOML path, so a saved session keeps resolving through the TOML on later runs.
+
+> **Note:** When using `--container-config`, pass `--env` explicitly.
+>
+> The dotenv preload of the default CLI flow is unaffected and keeps working; secrets reach the process through direnv or plain exports.
 
 ### Authentication
 
@@ -258,14 +362,18 @@ The chosen auth method is saved to the session. Subsequent commands (`ls`, `down
 
 ## CLI Tools
 
-The package provides a unified `io` command with subcommands for all storage operations. All commands accept `--env prod|staging` (selects the env file) and `--container <name>` (selects the Azure container).
+The package provides a unified `io` command with subcommands for all storage operations.
+All commands accept `--env prod|staging` (selects the env file) and `--container <name>` (selects the Azure container).
+`--container-config` resolves `--container` as a logical key from a `container-config.toml` (see [TOML + direnv configuration](#toml--direnv-configuration)).
 
 | Command | Usage | Description |
 |---------|-------|-------------|
 | `auth` | `uv run io auth [login\|status\|clear] [--method M] [--force]` | Manage authentication |
 | `ls` | `uv run io ls --env <env> --container <name>` | List blobs in a container |
 | `download` | `uv run io download <blob_name> --env <env> --container <name>` | Download a blob to local `data/` directory |
+| `download-dir` | `uv run io download-dir [<prefix>] [--local-dir <path>] --env <env> --container <name>` | Recursively download all blobs under a prefix to a local directory |
 | `upload` | `uv run io upload <blob_name> <file_path> --env <env> --container <name>` | Upload a local file to a container |
+| `upload-dir` | `uv run io upload-dir <local_dir> [--blob-prefix <prefix>] --env <env> --container <name>` | Upload all files from a local directory to storage |
 | `rm` | `uv run io rm <path> --env <env> --container <name>` | Delete a blob or folder from a container |
 | `mv` | `uv run io mv <source> <dest> [--overwrite] --env <env> --container <name>` | Move (rename) a blob within a container |
 | `generate-sas` | `uv run io generate-sas --env <env> --container <name> [--days N] [--update-env] [--write]` | Generate an Azure Storage SAS token for a container (read+list by default; add `--write` for write+delete) |
@@ -355,23 +463,29 @@ The session is also available programmatically via the `Session` class:
 from containerio.session import Session
 
 session = Session()
-print(session.auth_method)  # e.g. 'client-credentials'
-print(session.env)          # e.g. 'staging'
-print(session.container)    # e.g. 'raw-data'
-print(session.is_active)    # True if any session data exists
+print(session.auth_method)      # e.g. 'client-credentials'
+print(session.env)              # e.g. 'staging'
+print(session.container)        # e.g. 'raw-data'
+print(session.container_config) # e.g. 'container-config.toml', or None
+print(session.is_active)        # True if any session data exists
 ```
 
 CLI options:
 
-- `--env <name>`: selects the environment. The CLI best-effort-preloads `.env.<name>` (no-op if missing) and uses the name as a label in prints, sessions, and SAS file write-back. Any name works (`prod`, `staging`, `dev`, `eu-prod`, …); `local` means the local filesystem.
-- `--container`: the Azure container name to target
+- `--env <name>`: selects the environment. The CLI best-effort-preloads `.env.<name>` (no-op if missing) and uses the name as a label in prints, sessions, and SAS file write-back.
+  Any name works (`prod`, `staging`, `dev`, `eu-prod`, …); `local` means the local filesystem.
+- `--container`: the Azure container name to target. With `--container-config`, this is a logical key under `[containers]` in the TOML file.
+- `--container-config [path]`: resolve `--container` as a logical key from a `container-config.toml` (default: `container-config.toml` in the current directory).
+  Secrets come from the environment (e.g. via direnv). See [TOML + direnv configuration](#toml--direnv-configuration).
 - `--days`: SAS token validity in days, max 7 (default: 1, `generate-sas` only)
 - `--update-env`: write SAS token to env file instead of printing (`generate-sas` only)
 - `--write`: opt in to write+delete permissions (`generate-sas` only). Default is read+list.
 
 **Note:** SAS tokens are generated using a *user delegation key* which has a maximum validity of **7 days** (Azure limitation). See [Microsoft documentation](https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key#request-body) for details.
 
-The token's effective permissions are the **intersection** of the permissions requested here and the Azure RBAC permissions held by your identity (the user-delegation-key requester) at generation time — so requesting `--write` only produces a write-capable token if your RBAC role allows it. See [Microsoft documentation on user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#permissions-for-a-user-delegation-sas) for details.
+The token's effective permissions are the **intersection** of the permissions requested here and the Azure RBAC permissions held by your identity (the user-delegation-key requester) at generation time.
+This means requesting `--write` only produces a write-capable token if your RBAC role allows it.
+See [Microsoft documentation on user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas#permissions-for-a-user-delegation-sas) for details.
 
 ## API Reference
 
@@ -431,9 +545,9 @@ StorageHandler(
 ### AzureStorageConfig
 
 Configuration dataclass for Azure storage settings.
-Fields: `storage_account`, `tenant_id` (optional), `dotenv_path` (optional — the file the config was loaded from; useful in logs and `repr`), `client_id` (optional), `client_secret` (optional) and `device_code_client_id` (optional).
+Fields: `storage_account`, `tenant_id` (optional), `dotenv_path` (optional), `client_id` (optional), `client_secret` (optional), `device_code_client_id` (optional).
+`dotenv_path` is the file the config was loaded from; it is useful in logs and `repr`.
 `storage_endpoint` is a derived property — `https://<storage_account>.blob.core.windows.net`.
-
 The config is the single carrier of account and identity data: the handler reads these fields and never reads environment variables for them itself.
 `from_env()` is the one seam that maps env vars onto the config (`AZURE_CLIENT_ID_CONTAINER` → `client_id`, `AZURE_CLIENT_SECRET_CONTAINER` → `client_secret`, `AZURE_CLIENT_ID` → `device_code_client_id`).
 The lone exception is the SAS token, which is container-scoped and therefore resolved by the handler from `AZURE_STORAGE_SAS_TOKEN_<NAME>` at use time.
@@ -495,8 +609,8 @@ Errors:
 
 ### Providing config to the CLI
 
-The CLI reads required Azure variables from `os.environ` — it does not parse any specific config format itself.
-This keeps the CLI format-agnostic and lets each project choose how to manage config.
+The CLI reads required Azure variables from `os.environ` — by default it does not parse any specific config format itself, keeping it format-agnostic so each project can choose how to manage config.
+For the native combined TOML + direnv approach, pass `--container-config` (see [TOML + direnv configuration](#toml--direnv-configuration)); the rest of this section covers the default dotenv / env-var flow.
 
 **Dotenv is the default.**
 If a `.env.<name>` file matching `--env <name>` exists, the CLI auto-loads it via `python-dotenv` before reading `os.environ`.
@@ -609,6 +723,7 @@ The `.gitsv/config.yaml` file defines how versions are bumped based on commit me
 4. **Release**: The CI pipeline automatically handles tagging and publishing
 
 The CI pipeline will automatically:
+
 - Bump the version based on commit history
 - Generate and pin the changelog as a Git issue
 - Build and publish the package to Forgejo
@@ -627,6 +742,7 @@ The repository includes a Crow CI pipeline that automatically builds, tests, and
 1. `git sv tag`: generate tag with version and push it to remote in one go, based on git commit parsing.
 
 The pipeline will:
+
 - Run linting and tests
 - Update the version in `pyproject.toml` automatically
 - Generate changelog from git history using `git-sv`

{containerio-2.4.0 → containerio-2.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "containerio"
-version = "2.4.0"
+version = "2.5.0"
 description = "Azure Blob Storage handler with unified local/cloud interface"
 readme = "README.md"
 requires-python = ">=3.11"

{containerio-2.4.0 → containerio-2.5.0}/pyproject.toml.orig

@@ -1,6 +1,6 @@
 [project]
 name = "containerio"
-version = "2.4.0"
+version = "2.5.0"
 description = "Azure Blob Storage handler with unified local/cloud interface"
 readme = "README.md"
 requires-python = ">=3.11"

{containerio-2.4.0 → containerio-2.5.0}/src/containerio/__init__.py

@@ -1,4 +1,5 @@
 from containerio.azure_config import AzureStorageConfig
+from containerio.config import resolve_container_config
 from containerio.handler import AzureAuthType, BlobInfo, StorageHandler
 
 __all__ = [
@@ -6,4 +7,5 @@ __all__ = [
     "AzureStorageConfig",
     "BlobInfo",
     "StorageHandler",
+    "resolve_container_config",
 ]

containerio-2.5.0/src/containerio/config.py

@@ -0,0 +1,209 @@
+"""Resolve container configuration from a TOML file + environment.
+
+This module implements the Python-native equivalent of r-containerio's
+`containerio_config()`: a project-local `container-config.toml` maps
+logical container keys and environments (`staging`, `prod`) to concrete
+Azure container names plus auth specs.
+
+Non-sensitive fields (storage account, container names) come from the
+TOML. Identity values (`tenant_id`, `client_id`,
+`device_code_client_id`) and the client secret are resolved
+environment-first: their `*_env` keys name an environment variable that
+wins when it is set at runtime, and the TOML value is only the
+fallback. That way a CI-injected value (Crow `from_secret` becomes a
+plain environment variable) or an existing shell export is never
+overwritten by the local direnv fetcher. The client secret has no TOML
+fallback: it is resolved from `os.environ` by name only. Combined with
+direnv (a commit-safe `.envrc` that pulls values at runtime, e.g. from
+Azure Key Vault via the authenticated `az` CLI), the TOML file is safe
+to commit.
+
+If the resolved auth spec defines no `storage_account`, the module
+falls back to `AzureStorageConfig.from_env()`, which reads the full
+configuration from environment variables.
+"""
+
+import os
+import tomllib
+from pathlib import Path
+from typing import Optional, Tuple, Union
+
+from containerio.azure_config import AzureStorageConfig, is_local
+
+DEFAULT_CONTAINER_CONFIG_FILE = "container-config.toml"
+
+
+def _resolve_env_or_direct(
+    auth: dict,
+    value_key: str,
+    env_key: str,
+) -> Optional[str]:
+    """Resolve an auth value environment-first, else TOML.
+
+    The environment variable named by `env_key` wins when it is set at
+    runtime; otherwise the direct `value_key` from the TOML is used.
+    This lets a CI-injected or pre-exported value take precedence over
+    the local direnv fallback.
+
+    Args:
+        auth (dict): The resolved auth table.
+        value_key (str): TOML key holding the direct fallback value.
+        env_key (str): TOML key naming the environment variable.
+
+    Returns:
+        Optional[str]: The resolved value, or `None` when neither the
+            named environment variable nor the direct value is set.
+    """
+    env_var = auth.get(env_key)
+
+    if env_var:
+        value = os.getenv(env_var)
+
+        if value:
+            return value
+
+    return auth.get(value_key)
+
+
+def resolve_container_config(
+    container_key: str,
+    env: str,
+    toml_file: Union[str, Path] = DEFAULT_CONTAINER_CONFIG_FILE,
+) -> Tuple[Optional[str], Optional[AzureStorageConfig]]:
+    """Resolve a logical container key to a name + Azure config.
+
+    The config is resolved from a TOML file plus the environment.
+
+    Looks up the container definition under `[containers]` in the TOML
+    file and maps it to an Azure auth configuration for the selected
+    `env`. Auth can be referenced in two ways:
+
+    - **Tag indirection** (shared auth across containers):
+      `azure_auth.<env>` is a string referencing a
+      `[azure_auth.<tag>]` section.
+    - **Inline nested auth** (per-container): `azure_auth.<env>` is a
+      nested table defining the auth fields directly under
+      `[containers.<key>.azure_auth.<env>]`.
+
+    `tenant_id_env`, `client_id_env`, and `device_code_client_id_env`
+    name the environment variables that hold those identity values. An
+    environment variable that is set at runtime wins; otherwise the
+    direct `tenant_id`, `client_id`, or `device_code_client_id` value
+    from the TOML is used. This keeps CI-provided or pre-exported
+    values authoritative over the local direnv fallback.
+
+    `client_secret_env` is the name of the environment variable that
+    holds the client secret; it is resolved from `os.environ` at load
+    time so no secret is stored in the TOML. An unset variable yields
+    `None` rather than an error: only client-credentials auth needs the
+    secret, and the storage handler reports the missing value when that
+    method is actually used. This keeps device-code login working when
+    only `device_code_client_id` is available. SAS tokens are not part
+    of this file: the storage handler resolves them from the
+    environment via `sas_token_env_var()` when connecting.
+
+    If the resolved auth spec defines no `storage_account`, the full
+    configuration is read from environment variables via
+    `AzureStorageConfig.from_env()` instead.
+
+    Args:
+        container_key (str): Logical container key under `[containers]`.
+        env (str): Environment name (e.g. 'staging', 'prod'). 'local'
+            short-circuits to the local filesystem.
+        toml_file (Union[str, Path]): Path to the TOML config file.
+            Defaults to `container-config.toml` in the current working
+            directory.
+
+    Returns:
+        Tuple[Optional[str], Optional[AzureStorageConfig]]: The resolved
+            concrete container name and an `AzureStorageConfig` suitable
+            for `StorageHandler`. Both are `None` for the local
+            filesystem (`env='local'`).
+
+    Raises:
+        FileNotFoundError: If `toml_file` does not exist.
+        ValueError: If the container key is unknown, the TOML is
+            missing the auth or name for the selected env, `name` or
+            `azure_auth` is not a table keyed by environment, or the
+            referenced auth tag does not exist.
+    """
+    if is_local(env):
+        return None, None
+
+    path = Path(toml_file)
+
+    if not path.is_file():
+        raise FileNotFoundError(f"container config file not found: {path}")
+
+    with path.open("rb") as handle:
+        cfg = tomllib.load(handle)
+
+    containers = cfg.get("containers")
+    container = containers.get(container_key) if isinstance(containers, dict) else None
+
+    if container is None:
+        raise ValueError(f"Unknown container: {container_key}")
+
+    name_map = container.get("name") or {}
+
+    if not isinstance(name_map, dict):
+        raise ValueError(
+            f"Invalid 'name' for container '{container_key}': expected a "
+            f'table keyed by environment (e.g. name.{env} = "..."), '
+            f"got {type(name_map).__name__}."
+        )
+
+    container_name = name_map.get(env)
+
+    if not container_name:
+        raise ValueError(
+            f"No container name defined for env '{env}' in container '{container_key}'"
+        )
+
+    auth_map = container.get("azure_auth") or {}
+
+    if not isinstance(auth_map, dict):
+        raise ValueError(
+            f"Invalid 'azure_auth' for container '{container_key}': expected "
+            f"a table keyed by environment (e.g. azure_auth.{env} = ...), "
+            f"got {type(auth_map).__name__}."
+        )
+
+    auth_spec = auth_map.get(env)
+
+    if auth_spec is None:
+        raise ValueError(
+            f"Container '{container_key}' has no azure_auth defined for env '{env}'."
+        )
+
+    # `azure_auth.<env>` may be either:
+    #  - a string referencing a shared `[azure_auth.<tag>]` section, or
+    #  - an inline (nested) table defining the auth fields directly
+    #    under `[containers.<key>.azure_auth.<env>]`.
+    if isinstance(auth_spec, dict):
+        auth = auth_spec
+    else:
+        tag = auth_spec
+        shared = cfg.get("azure_auth") or {}
+        auth = shared.get(tag) if isinstance(shared, dict) else None
+
+        if auth is None:
+            raise ValueError(f"Unknown azure_auth: {tag}")
+
+    if auth.get("storage_account"):
+        client_secret_env = auth.get("client_secret_env")
+        client_secret = os.getenv(client_secret_env) if client_secret_env else None
+
+        config = AzureStorageConfig(
+            storage_account=auth["storage_account"],
+            tenant_id=_resolve_env_or_direct(auth, "tenant_id", "tenant_id_env"),
+            client_id=_resolve_env_or_direct(auth, "client_id", "client_id_env"),
+            client_secret=client_secret,
+            device_code_client_id=_resolve_env_or_direct(
+                auth, "device_code_client_id", "device_code_client_id_env"
+            ),
+        )
+    else:
+        config = AzureStorageConfig.from_env()
+
+    return container_name, config

{containerio-2.4.0 → containerio-2.5.0}/src/containerio/session.py

@@ -33,6 +33,15 @@ class Session:
         """The Azure container name (e.g. 'raw-data')."""
         return self._data.get("container")
 
+    @property
+    def container_config(self) -> Optional[str]:
+        """The container-config TOML path saved with the session.
+
+        When set, `container` holds a logical key that is resolved
+        through this TOML file on later runs.
+        """
+        return self._data.get("container_config")
+
     @property
     def is_active(self) -> bool:
         """Whether a session has any data."""
@@ -43,6 +52,8 @@ class Session:
         auth_method: Optional[str] = None,
         env: Optional[str] = None,
         container: Optional[str] = None,
+        container_config: Optional[str] = None,
+        clear_container_config: bool = False,
     ) -> None:
         """Update session fields and persist to disk.
 
@@ -50,6 +61,11 @@ class Session:
             auth_method (Optional[str]): Azure auth method.
             env (Optional[str]): Storage environment.
             container (Optional[str]): Azure container name.
+            container_config (Optional[str]): Path to the TOML file used
+                to resolve `container` as a logical key.
+            clear_container_config (bool): Remove any stored
+                container-config path. Takes precedence over
+                `container_config`.
         """
         if auth_method is not None:
             self._data["auth_method"] = auth_method
@@ -60,6 +76,11 @@ class Session:
         if container is not None:
             self._data["container"] = container
 
+        if clear_container_config:
+            self._data.pop("container_config", None)
+        elif container_config is not None:
+            self._data["container_config"] = container_config
+
         self._save()
 
     def clear(self) -> None:

{containerio-2.4.0 → containerio-2.5.0}/src/containerio/tools/cli.py

@@ -29,6 +29,10 @@ from containerio import (
     StorageHandler,
 )
 from containerio.azure_config import env_file, is_local, sas_token_env_var
+from containerio.config import (
+    DEFAULT_CONTAINER_CONFIG_FILE,
+    resolve_container_config,
+)
 from containerio.session import Session
 
 MAX_SAS_DAYS_VALID = 7
@@ -86,6 +90,54 @@ def _require_container(args: argparse.Namespace) -> None:
         )
 
 
+def _apply_container_config(args: argparse.Namespace) -> None:
+    """Resolve --container-config selection onto args, in place.
+
+    When `--container-config` is set and a cloud env is selected,
+    `--container` is interpreted as a logical key under `[containers]`
+    in the TOML file. It is replaced by the resolved concrete container
+    name and the resolved `AzureStorageConfig` is stashed on the
+    namespace as `_toml_config` for `_build_config` to pick up.
+
+    The 'session' command is skipped so logical keys — not concrete
+    names — persist in the session. `_cmd_session` additionally stores
+    the TOML path, and `_apply_session_defaults` restores it on later
+    runs so the key keeps resolving.
+
+    Args:
+        args (argparse.Namespace): Parsed CLI arguments.
+    """
+    toml_file = getattr(args, "container_config", None)
+
+    if not toml_file or args.command == "session":
+        return
+
+    if not args.env:
+        _fail(
+            "Error: --env <name> is required when --container-config is used. "
+            "Pass a cloud env (e.g. --env staging) or drop --container-config."
+        )
+
+    if is_local(args.env):
+        return
+
+    if not args.container:
+        _fail(
+            "Error: --container <logical-key> is required when "
+            "--container-config is used. Pass a key defined under "
+            "[containers] in the TOML file."
+        )
+
+    container_name, config = resolve_container_config(
+        container_key=args.container,
+        env=args.env,
+        toml_file=toml_file,
+    )
+
+    args.container = container_name
+    args._toml_config = config
+
+
 def _build_config(args: argparse.Namespace) -> Optional[AzureStorageConfig]:
     """Build an AzureStorageConfig from CLI args, or None for local.
 
@@ -106,6 +158,12 @@ def _build_config(args: argparse.Namespace) -> Optional[AzureStorageConfig]:
     """
     if not args.env or is_local(args.env):
         return None
+
+    toml_config = getattr(args, "_toml_config", None)
+
+    if toml_config is not None:
+        return toml_config
+
     return AzureStorageConfig.from_env()
 
 
@@ -127,6 +185,38 @@ def _require_config(args: argparse.Namespace) -> AzureStorageConfig:
     return config
 
 
+def _apply_session_defaults(
+    args: argparse.Namespace,
+    session: Session,
+) -> None:
+    """Fill missing env/container/container-config from a session.
+
+    Explicit flags always win. When the container is taken from the
+    session and that session was started with `--container-config`, the
+    stored TOML path is restored too, so the logical key keeps
+    resolving on later runs instead of being treated as a concrete
+    container name.
+
+    Args:
+        args (argparse.Namespace): Parsed CLI arguments, updated in
+            place.
+        session (Session): The active session.
+    """
+    # The 'session' command reads the session directly and must see the
+    # user's explicit flags, so it never inherits defaults.
+    if getattr(args, "command", None) == "session":
+        return
+
+    if not args.env and session.env:
+        args.env = session.env
+
+    if getattr(args, "container", None) is None and session.container:
+        args.container = session.container
+
+        if session.container_config and not getattr(args, "container_config", None):
+            args.container_config = session.container_config
+
+
 def _get_handler(args: argparse.Namespace) -> StorageHandler:
     """Validate --container and construct a StorageHandler.
 
@@ -153,7 +243,13 @@ def _cmd_session(args: argparse.Namespace) -> None:
         return
 
     if args.env and args.container:
-        session.update(env=args.env, container=args.container)
+        container_config = getattr(args, "container_config", None)
+        session.update(
+            env=args.env,
+            container=args.container,
+            container_config=container_config,
+            clear_container_config=not container_config,
+        )
         print(f"Session started: env={args.env}, container={args.container}")
         return
 
@@ -856,7 +952,20 @@ def main() -> None:
     parent.add_argument(
         "--container",
         default=None,
-        help="Azure container name (required for Azure operations)",
+        help="Azure container name (required for Azure operations). "
+        "With --container-config, this is a logical key under "
+        "[containers] in the TOML file.",
+    )
+    parent.add_argument(
+        "--container-config",
+        nargs="?",
+        const=DEFAULT_CONTAINER_CONFIG_FILE,
+        default=None,
+        help="Resolve --container as a logical key from a "
+        "container-config.toml file (default name: "
+        "container-config.toml in the current directory). Secrets "
+        "are resolved from the environment (e.g. via direnv). "
+        "Requires --env.",
     )
 
     parser = argparse.ArgumentParser(
@@ -1043,20 +1152,20 @@ def main() -> None:
         sys.exit(1)
 
     session = Session()
-
-    if not args.env and session.env:
-        args.env = session.env
-    if getattr(args, "container", None) is None and session.container:
-        args.container = session.container
+    _apply_session_defaults(args=args, session=session)
 
     if args.env:
         os.environ["STORAGE_ENV"] = args.env
         if not is_local(args.env):
-            load_dotenv(dotenv_path=env_file(args.env), override=True)
+            load_dotenv(
+                dotenv_path=env_file(args.env),
+                override=not getattr(args, "container_config", None),
+            )
 
     try:
+        _apply_container_config(args=args)
         args.func(args)
-    except ValueError as error:
+    except (ValueError, FileNotFoundError) as error:
         _fail(f"Error: {error}")