containerio 1.0.0__tar.gz

containerio-1.0.0/PKG-INFO

@@ -0,0 +1,524 @@
+Metadata-Version: 2.3
+Name: containerio
+Version: 1.0.0
+Summary: Azure Blob Storage handler with unified local/cloud interface
+Author: Alliance SwissPass
+Requires-Dist: polars>=1.30.0
+Requires-Dist: azure-storage-blob>=12.26.0
+Requires-Dist: azure-identity>=1.25.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: rich>=13.0.0
+Requires-Python: >=3.11
+Project-URL: Repository, https://codefloe.com/Alliance-SwissPass/py-containerio
+Description-Content-Type: text/markdown
+
+# py-containerio
+
+Shared Azure Blob Storage handler for internal and collaboration projects of [Alliance SwissPass](https://www.allianceswisspass.ch/).
+Provides a unified interface for reading and writing data to both local filesystem and Azure Blob Storage, allowing seamless switching between local development and cloud environments.
+
+## Features
+
+- **Unified API**: Same code works for local files and Azure Blob Storage
+- **Environment-based switching**: Use `STORAGE_ENV` to switch between local/staging/prod
+- **[Polars](https://pola.rs/) integration**: Native support for Polars DataFrames and LazyFrames
+- **Multiple auth methods**: Device code flow (default), client credentials (CI/CD), or SAS tokens
+- **CLI tools**: Unified `io` command for listing, downloading, uploading, deleting blobs and generating SAS tokens
+- **Memory-efficient**: Streaming support with `scan_*` and `sink_*` methods
+
+## Install containerio
+
+### From PyPI
+
+The package is available on PyPI: [https://pypi.org/project/containerio/](https://pypi.org/project/containerio/)
+
+```bash
+pip install containerio
+```
+
+Or with uv:
+
+```bash
+uv add containerio
+uv sync
+```
+
+## Use containerio as a package dependency
+
+### From PyPI (recommended)
+
+Add to your project's `pyproject.toml`:
+
+```toml
+[project]
+dependencies = [
+    "containerio",
+]
+```
+
+Then run:
+
+```bash
+uv sync
+```
+
+### From Forgejo Package Registry
+
+If you need the Forgejo version, configure the custom index:
+
+```toml
+[project]
+dependencies = [
+    "containerio",
+]
+
+[tool.uv.sources]
+containerio = { index = "forgejo" }
+
+[[tool.uv.index]]
+name = "forgejo"
+url = "https://codefloe.com/api/packages/Alliance-SwissPass/pypi/simple"
+```
+
+Then run:
+
+```bash
+uv sync
+```
+
+### From Git
+
+For development or testing, install directly from the repository:
+
+```toml
+[project]
+dependencies = [
+    "containerio",
+]
+
+[tool.uv.sources]
+containerio = { git = "https://codefloe.com/Alliance-SwissPass/py-containerio.git", tag = "v1.0.0" }
+```
+
+## Usage in Other Projects
+
+When installed as a dependency, containerio integrates seamlessly with your project:
+
+- **CLI tools are available** - Run `uv run io` directly from your project
+- **`.env` files live in your project** - The storage module reads `.env` and `.env.staging` from your current working directory, not from the containerio package
+- **No configuration in containerio needed** - Just add the dependency and configure in your project
+
+Example workflow in your project:
+
+```bash
+# Your project directory
+cd my-project
+
+# Create your .env.staging file here
+cp /path/to/.env.example .env.staging
+# Edit .env.staging with your credentials
+
+# List blobs in both containers
+uv run io ls --env staging
+
+# List blobs in local mode (./data/ directory)
+uv run io ls
+
+# Generate SAS tokens (updates .env.staging)
+uv run io generate-sas --env staging --update-env
+```
+
+## Example Usage
+
+```python
+import os
+from containerio import StorageHandler
+
+# Set environment (defaults to "local" which uses ./data/ directory)
+os.environ["STORAGE_ENV"] = "staging"
+# or "prod"
+
+# Read operations (uses read container)
+handler = StorageHandler(container="ro")
+
+# Read files
+df = handler.read_csv_blob("path/to/file.csv")
+df = handler.read_excel_blob("path/to/file.xlsx")
+df, metadata = handler.read_parquet_blob("path/to/file.parquet")
+
+# Lazy scan (memory-efficient for large files)
+lf, metadata = handler.scan_parquet_blob("large_file.parquet")
+lf = handler.scan_csv_blob("large_file.csv")
+
+# Write operations (uses write container)
+handler_rw = StorageHandler(container="rw")
+
+handler_rw.write_csv_blob("output.csv", df)
+handler_rw.write_parquet_blob("output.parquet", df)
+handler_rw.write_excel_blob("output.xlsx", df)
+
+# Stream LazyFrame directly to storage
+handler_rw.sink_parquet_blob("output.parquet", lf)
+
+# Upload any file (JSON, YAML, images, etc.)
+handler_rw.upload_blob("config.json", "./local/config.json")
+
+# List blobs
+blobs = handler.list_blobs()    # Returns List[BlobInfo]
+handler.download_blob("file")   # Download to local ./data/
+handler_rw.delete_blob("file")  # Delete a blob or folder
+handler_rw.move_blob("old.csv", "archive/old.csv")  # Move/rename
+```
+
+## Configuration
+
+### Environment Variables
+
+Set `STORAGE_ENV` to control the storage backend:
+
+| Value | Behavior |
+|-------|----------|
+| `local` (default) | Uses local filesystem (`./data/` directory) |
+| `staging` | Uses Azure staging storage (reads from `.env.staging`) |
+| `prod` | Uses Azure production storage (reads from `.env`) |
+
+### Azure Configuration
+
+For Azure storage, create a `.env` file (for prod) or `.env.staging` (for staging).
+
+**Important:** Restrict permissions on your `.env` file so only you can access it:
+
+```bash
+chmod 600 .env
+```
+
+#### Example `.env` (production)
+
+```bash
+AZURE_STORAGE_ACCOUNT=<azure_storage_account>
+AZURE_CONTAINER_NAME_READ_PROD=<azure_container_name_read_prod>
+AZURE_CONTAINER_NAME_WRITE_PROD=<azure_container_name_write_prod>
+AZURE_TENANT_ID=<azure_tenant_id>
+AZURE_CLIENT_ID=<azure_client_id>
+
+# SAS tokens (optional, if not using device code auth)
+AZURE_STORAGE_SAS_TOKEN_READ=<azure_storage_sas_token_read>
+AZURE_STORAGE_SAS_TOKEN_WRITE=<azure_storage_sas_token_write>
+```
+
+#### Example `.env.staging`
+
+```bash
+AZURE_STORAGE_ACCOUNT=<azure_storage_account>
+AZURE_CONTAINER_NAME_READ_STAGING=<azure_container_name_read_staging>
+AZURE_CONTAINER_NAME_WRITE_STAGING=<azure_container_name_write_staging>
+AZURE_TENANT_ID=<azure_tenant_id>
+AZURE_CLIENT_ID=<azure_client_id>
+
+# SAS tokens (optional, if not using device code auth)
+AZURE_STORAGE_SAS_TOKEN_READ=<azure_storage_sas_token_read>
+AZURE_STORAGE_SAS_TOKEN_WRITE=<azure_storage_sas_token_write>
+```
+
+### Authentication
+
+Use `io auth` to manage authentication explicitly.
+Device-code login and status auto-discover the environment from your `.env` / `.env.staging` file, so `--env` is optional.
+SAS token and client credentials login require explicit `--env`:
+
+```bash
+# Device code login (default) — auto-discovers env from .env file
+uv run io auth
+
+# Device code login with explicit env
+uv run io auth --env staging
+
+# Check auth status across all methods
+uv run io auth status
+
+# SAS token login — requires --env (wrong env = wrong token)
+uv run io auth login --method sas-token --env staging
+
+# Client credentials — requires --env (for CI/CD pipelines)
+uv run io auth login --method client-credentials --env prod
+
+# Clear cached device code tokens
+uv run io auth clear
+
+# Force re-authentication (clears cache first)
+uv run io auth login --force
+```
+
+The chosen auth method is saved to the session. Subsequent commands (`ls`, `download`, etc.) automatically use it.
+
+### Authentication Methods
+
+1. **Device Code Flow** (recommended for development)
+   - Interactive browser-based authentication
+   - Tokens are cached locally for 90 days
+   - No credentials stored in files
+
+2. **Client Credentials**
+   - For CI/CD pipelines
+   - Use `io auth login --method client-credentials` to authenticate
+   - Requires:
+     - `AZURE_TENANT_ID`: Service principal's tenant ID
+     - `AZURE_CLIENT_ID_CONTAINER`: Service principal's client ID
+     - `AZURE_CLIENT_SECRET_CONTAINER_PROD`: Service principal's client secret for production
+     - `AZURE_CLIENT_SECRET_CONTAINER_STAGING`: Service principal's client secret for staging
+
+3. **SAS Token**
+   - Use pre-generated tokens with limited validity
+   - Good for sharing temporary access
+
+## CLI Tools
+
+The package provides a unified `io` command with subcommands for all storage operations. All commands accept `--env prod|staging` to override `STORAGE_ENV`.
+
+| Command | Usage | Description |
+|---------|-------|-------------|
+| `auth` | `uv run io auth [login\|status\|clear] [--method M] [--force]` | Manage authentication |
+| `ls` | `uv run io ls [--container ro\|rw\|ro-rw]` | List blobs in storage |
+| `download` | `uv run io download <blob_name> [--container ro\|rw]` | Download a blob to local `data/` directory |
+| `upload` | `uv run io upload <blob_name> <file_path>` | Upload a local file to the write container |
+| `rm` | `uv run io rm <path>` | Delete a blob or folder from the write container |
+| `mv` | `uv run io mv <source> <dest> [--overwrite]` | Move (rename) a blob within the write container |
+| `generate-sas` | `uv run io generate-sas [--container ro\|rw\|ro-rw] [--days N] [--update-env]` | Generate Azure Storage SAS tokens |
+
+### Examples
+
+```bash
+# Authenticate with device code flow (auto-discovers env)
+uv run io auth
+
+# Check authentication status (auto-discovers env)
+uv run io auth status
+
+# List blobs in local mode (./data/)
+uv run io ls
+
+# List both read and write containers (default --container ro-rw)
+uv run io ls --env staging
+
+# Download a blob to the local data/ directory
+uv run io download path/to/file.csv --env staging
+
+# Upload a local file to the write container
+uv run io upload path/to/blob.csv ./local/file.csv --env staging
+
+# Delete a blob or folder from the write container
+uv run io rm path/to/blob.csv --env staging
+uv run io rm old-folder/ --env staging
+
+# Move a blob within the write container
+uv run io mv data/old.csv archive/old.csv --env staging
+
+# Overwrite an existing destination
+uv run io mv data/old.csv data/existing.csv --env staging --overwrite
+
+# Generate SAS tokens and update .env.staging file
+uv run io generate-sas --env staging --update-env
+
+# Generate only read container token, valid for 3 days
+uv run io generate-sas --env prod --container ro --days 3
+```
+
+### Session
+
+When doing multiple operations on the same container, use `session` to save defaults so you don't have to repeat `--env` and `--container` on every command.
+Both `--env` and `--container` are required to start a session.
+
+```bash
+# Start a session — saves env and container
+uv run io session --env staging --container rw
+
+# Show active session
+uv run io session
+
+# Now these use the saved defaults
+uv run io ls
+uv run io rm old-folder/
+uv run io mv data/old.csv archive/old.csv
+
+# Explicit flags still override the session
+uv run io ls --container ro
+
+# End the session
+uv run io session clear
+```
+
+The session is stored in `.containerio/session.json` in the current directory. It is project-local and git-ignored.
+
+The session is also available programmatically via the `Session` class:
+
+```python
+from containerio.session import Session
+
+session = Session()
+print(session.auth_method)  # e.g. 'client-credentials'
+print(session.env)          # e.g. 'staging'
+print(session.is_active)    # True if any session data exists
+```
+
+Options:
+
+- `--env`: `prod` or `staging` (default: staging for sas; uses `STORAGE_ENV` for other commands)
+- `--container`: `ro`, `rw`, or `ro-rw` (default: ro-rw)
+- `--days`: Token validity in days, max 7 (default: 1)
+- `--update-env`: Write tokens to .env file instead of printing
+
+**Note:** SAS tokens are generated using a *user delegation key* which has a maximum validity of **7 days** (Azure limitation). See [Microsoft documentation](https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key#request-body) for details.
+
+## API Reference
+
+### StorageHandler
+
+Main class for storage operations.
+
+```python
+StorageHandler(
+    container: Literal["ro", "rw"] = "ro",
+    auth_type: Optional[str] = None,
+    session: Optional[Session] = None,
+)
+```
+
+- `auth_type=None`: Falls back to session auth method, then device code flow
+- `auth_type="device-code"`: Device code flow (interactive)
+- `auth_type="sas-token"`: SAS token authentication
+- `auth_type="client-credentials"`: Client credentials (service principal)
+- `session=None`: Creates a new `Session` (skipped in local mode). Pass an existing `Session` instance for dependency injection.
+- `container="ro"`: Uses the read container (`AZURE_CONTAINER_NAME_READ_{ENV}`)
+- `container="rw"`: Uses the write container (`AZURE_CONTAINER_NAME_WRITE_{ENV}`)
+
+#### Read Methods
+
+| Method | Description |
+|--------|-------------|
+| `read_parquet_blob(blob_name)` | Read parquet file, returns `(DataFrame, metadata)` |
+| `read_csv_blob(blob_name, **kwargs)` | Read CSV file into DataFrame |
+| `read_excel_blob(blob_name, **kwargs)` | Read Excel file into DataFrame |
+| `scan_parquet_blob(blob_name)` | Lazy scan parquet, returns `(LazyFrame, metadata)` |
+| `scan_csv_blob(blob_name, **kwargs)` | Lazy scan CSV file |
+
+#### Write Methods
+
+| Method | Description |
+|--------|-------------|
+| `write_parquet_blob(blob_name, df)` | Write DataFrame to parquet |
+| `write_csv_blob(blob_name, df, **kwargs)` | Write DataFrame to CSV |
+| `write_excel_blob(blob_name, df, **kwargs)` | Write DataFrame to Excel |
+| `sink_parquet_blob(blob_name, lf)` | Stream LazyFrame to parquet |
+
+#### Utility Methods
+
+| Method | Description |
+|--------|-------------|
+| `upload_blob(blob_name, file_path, progress_hook)` | Upload a local file to storage |
+| `delete_blob(blob_name)` | Delete a blob, file, or folder recursively. Returns `List[str]` of deleted paths |
+| `move_blob(source, destination, overwrite)` | Move (rename) a blob within the same container |
+| `download_blob(blob_name, progress_hook)` | Download blob to local `data/` directory. Returns `Optional[Path]` |
+| `list_blobs()` | List all blobs, returns `List[BlobInfo]` |
+
+### AzureStorageConfig
+
+Configuration dataclass for Azure storage settings.
+
+```python
+from containerio import AzureStorageConfig
+
+# Load from environment
+config = AzureStorageConfig.from_environment()  # Uses STORAGE_ENV
+config = AzureStorageConfig.from_environment(environment="staging")
+
+# Access configuration
+print(config.storage_account)
+print(config.container_name_read)
+```
+
+### Environment
+
+Enum for storage environments.
+
+```python
+from containerio import Environment
+
+# Parse from string
+env = Environment.from_string("prod")  # Environment.PROD
+
+# Use enum directly
+if env == Environment.LOCAL:
+    print("Using local storage")
+```
+
+## Development
+
+```bash
+# Clone and install
+git clone ssh://git@codefloe.com/Alliance-SwissPass/py-containerio.git
+cd py-containerio
+uv sync --group dev
+
+# Run tests
+uv run pytest
+
+# Lint and format
+uv run ruff check
+uv run ruff format
+```
+
+## Version Management with git-sv
+
+This project uses [git-sv](https://github.com/thegeeklab/git-sv) for semantic versioning and changelog generation.
+
+The `.gitsv/config.yaml` file defines how versions are bumped based on commit messages.
+
+1. **Commit messages**: Follow [conventional commits](https://www.conventionalcommits.org/en/v1.0.0/) format
+2. **Version bump**: The CI pipeline automatically runs `git-sv bump` based on commit history
+3. **Changelog**: The CI pipeline generates the changelog using `git-sv changelog` and pins it as a Git issue
+4. **Release**: The CI pipeline automatically handles tagging and publishing
+
+The CI pipeline will automatically:
+- Bump the version based on commit history
+- Generate and pin the changelog as a Git issue
+- Build and publish the package to Forgejo
+- Create a release with the changelog notes
+
+## Publishing
+
+### Via CI Pipeline (recommended)
+
+The repository includes a Crow CI pipeline that automatically builds, tests, and publishes the package when a git tag is pushed:
+
+2. Commit and push your changes
+3. Create and push a git tag (e.g., `git tag v1.0.0 && git push --tags`)
+
+The pipeline will:
+- Run linting and tests
+- Update the version in `pyproject.toml` automatically
+- Generate changelog from git history using `git-sv`
+- Build and publish the package to Forgejo and PyPI
+- Create a release with changelog notes
+- Push version updates back to the repository
+
+**Note:** For CI publishing, ensure the `UV_PUBLISH_TOKEN` secret is configured in your Crow CI pipeline settings.
+
+## Security Best Practices
+
+1. **Never commit credentials**: Always add `.env*` to your `.gitignore` file
+2. **Restrict file permissions**: Use `chmod 600 .env` to limit access to your credentials
+3. **Use short-lived tokens**: SAS tokens have a maximum validity of 7 days
+4. **Rotate secrets regularly**: Especially for production environments and CI/CD pipelines
+5. **Use device code flow**: For development to avoid storing credentials in files
+6. **Environment isolation**: Use separate credentials for staging and production
+7. **Limit token scope**: When generating SAS tokens, specify the minimum required permissions
+
+## References
+
+- [Delegate access with shared access signature](https://learn.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature)
+- [Create user delegation SAS](https://learn.microsoft.com/en-us/rest/api/storageservices/create-user-delegation-sas)
+- [Get user delegation key](https://learn.microsoft.com/en-us/rest/api/storageservices/get-user-delegation-key)
+
+## License
+
+Alliance SwissPass <https://allianceswisspass.ch>