waldur-site-agent-cscs-dwdi 0.7.0__tar.gz → 0.7.3__tar.gz

waldur_site_agent_cscs_dwdi-0.7.3/PKG-INFO

@@ -0,0 +1,284 @@
+Metadata-Version: 2.4
+Name: waldur-site-agent-cscs-dwdi
+Version: 0.7.3
+Summary: CSCS-DWDI reporting plugin for Waldur Site Agent
+Author-email: OpenNode Team <info@opennodecloud.com>
+Requires-Python: <4,>=3.9
+Requires-Dist: httpx>=0.25.0
+Requires-Dist: waldur-site-agent>=0.7.0
+Description-Content-Type: text/markdown
+
+# CSCS-DWDI Plugin for Waldur Site Agent
+
+This plugin provides integration with the CSCS Data Warehouse Data Intelligence (DWDI) system to report both
+computational and storage usage data to Waldur.
+
+## Overview
+
+The plugin implements two separate backends to handle different types of accounting data:
+
+- **Compute Backend** (`cscs-dwdi-compute`): Reports CPU and node hour usage from HPC clusters
+- **Storage Backend** (`cscs-dwdi-storage`): Reports storage space and inode usage from filesystems
+
+## Backend Types
+
+### Compute Backend
+
+The compute backend queries the DWDI API for computational resource usage and reports:
+
+- Node hours consumed by accounts and users
+- CPU hours consumed by accounts and users
+- Account-level and user-level usage aggregation
+
+**API Endpoints Used:**
+
+- `/api/v1/compute/usage-month/account` - Monthly usage data
+- `/api/v1/compute/usage-day/account` - Daily usage data
+
+### Storage Backend
+
+The storage backend queries the DWDI API for storage resource usage and reports:
+
+- Storage space used (converted from bytes to configured units)
+- Inode (file count) usage
+- Path-based resource identification
+
+**API Endpoints Used:**
+
+- `/api/v1/storage/usage-month/filesystem_name/data_type` - Monthly storage usage
+- `/api/v1/storage/usage-day/filesystem_name/data_type` - Daily storage usage
+
+## Configuration
+
+### Compute Backend Configuration
+
+```yaml
+backend_type: "cscs-dwdi-compute"
+
+backend_settings:
+  cscs_dwdi_api_url: "https://dwdi.cscs.ch"
+  cscs_dwdi_client_id: "your_oidc_client_id"
+  cscs_dwdi_client_secret: "your_oidc_client_secret"
+  cscs_dwdi_oidc_token_url: "https://auth.cscs.ch/realms/cscs/protocol/openid-connect/token"
+  cscs_dwdi_oidc_scope: "openid"  # Optional
+
+backend_components:
+  nodeHours:
+    measured_unit: "node-hours"
+    unit_factor: 1
+    accounting_type: "usage"
+    label: "Node Hours"
+
+  cpuHours:
+    measured_unit: "cpu-hours"
+    unit_factor: 1
+    accounting_type: "usage"
+    label: "CPU Hours"
+```
+
+### Storage Backend Configuration
+
+```yaml
+backend_type: "cscs-dwdi-storage"
+
+backend_settings:
+  cscs_dwdi_api_url: "https://dwdi.cscs.ch"
+  cscs_dwdi_client_id: "your_oidc_client_id"
+  cscs_dwdi_client_secret: "your_oidc_client_secret"
+  cscs_dwdi_oidc_token_url: "https://auth.cscs.ch/realms/cscs/protocol/openid-connect/token"
+
+  # Storage-specific settings
+  storage_filesystem: "lustre"
+  storage_data_type: "projects"
+  storage_tenant: "cscs"  # Optional
+
+  # Map Waldur resource IDs to storage paths
+  storage_path_mapping:
+    "project_123": "/store/projects/proj123"
+    "project_456": "/store/projects/proj456"
+
+backend_components:
+  storage_space:
+    measured_unit: "GB"
+    unit_factor: 0.000000001  # Convert bytes to GB
+    accounting_type: "usage"
+    label: "Storage Space (GB)"
+
+  storage_inodes:
+    measured_unit: "count"
+    unit_factor: 1
+    accounting_type: "usage"
+    label: "File Count"
+```
+
+## Authentication
+
+Both backends use OIDC client credentials flow for authentication with the DWDI API. You need:
+
+- `cscs_dwdi_client_id`: OIDC client identifier
+- `cscs_dwdi_client_secret`: OIDC client secret
+- `cscs_dwdi_oidc_token_url`: OIDC token endpoint URL
+- `cscs_dwdi_oidc_scope`: OIDC scope (optional, defaults to "openid")
+
+## SOCKS Proxy Support
+
+Both backends support SOCKS proxy for network connectivity. This is useful when the DWDI API is only accessible
+through a proxy or jump host.
+
+### SOCKS Proxy Configuration
+
+Add the SOCKS proxy setting to your backend configuration:
+
+```yaml
+backend_settings:
+  # ... other settings ...
+  socks_proxy: "socks5://localhost:12345"  # SOCKS5 proxy URL
+```
+
+### Supported Proxy Types
+
+- **SOCKS5**: `socks5://hostname:port`
+- **SOCKS4**: `socks4://hostname:port`
+- **HTTP**: `http://hostname:port`
+
+### Usage Examples
+
+**SSH Tunnel with SOCKS5:**
+
+```bash
+# Create SSH tunnel to jump host
+ssh -D 12345 -N user@jumphost.cscs.ch
+
+# Configure backend to use tunnel
+backend_settings:
+  socks_proxy: "socks5://localhost:12345"
+```
+
+**HTTP Proxy:**
+
+```yaml
+backend_settings:
+  socks_proxy: "http://proxy.cscs.ch:8080"
+```
+
+## Resource Identification
+
+### Compute Resources
+
+For compute resources, the system uses account names as returned by the DWDI API. The Waldur resource
+`backend_id` should match the account name in the cluster accounting system.
+
+### Storage Resources
+
+For storage resources, there are two options:
+
+1. **Direct Path Usage**: Set the Waldur resource `backend_id` to the actual filesystem path
+2. **Path Mapping**: Use the `storage_path_mapping` setting to map resource IDs to paths
+
+## Usage Reporting
+
+Both backends are read-only and designed for usage reporting. They implement the `_get_usage_report()` method
+but do not support:
+
+- Account creation/deletion
+- Resource management
+- User management
+- Limit setting
+
+## Example Configurations
+
+See the `examples/` directory for complete configuration examples:
+
+- `cscs-dwdi-compute-config.yaml` - Compute backend only
+- `cscs-dwdi-storage-config.yaml` - Storage backend only
+- `cscs-dwdi-combined-config.yaml` - Both backends in one configuration
+
+## Installation
+
+The plugin is automatically discovered when the waldur-site-agent-cscs-dwdi package is installed alongside waldur-site-agent.
+
+```bash
+# Install all workspace packages including cscs-dwdi plugin
+uv sync --all-packages
+```
+
+## Testing
+
+Run the test suite:
+
+```bash
+uv run pytest plugins/cscs-dwdi/tests/
+```
+
+## API Compatibility
+
+This plugin is compatible with DWDI API version 1 (`/api/v1/`). It requires the following API endpoints to be available:
+
+**Compute API:**
+
+- `/api/v1/compute/usage-month/account`
+- `/api/v1/compute/usage-day/account`
+
+**Storage API:**
+
+- `/api/v1/storage/usage-month/filesystem_name/data_type`
+- `/api/v1/storage/usage-day/filesystem_name/data_type`
+
+## Troubleshooting
+
+### Authentication Issues
+
+- Verify OIDC client credentials are correct
+- Check that the token endpoint URL is accessible
+- Ensure the client has appropriate scopes
+
+### Storage Backend Issues
+
+- Verify `storage_filesystem` and `storage_data_type` match available values in DWDI
+- Check `storage_path_mapping` if using custom resource IDs
+- Ensure storage paths exist in the DWDI system
+
+### Connection Issues
+
+- Use the `ping()` method to test API connectivity
+- Check network connectivity to the DWDI API endpoint
+- Verify SSL/TLS configuration
+- If behind a firewall, configure SOCKS proxy (`socks_proxy` setting)
+
+### Proxy Issues
+
+- Verify proxy server is running and accessible
+- Check proxy authentication if required
+- Test proxy connectivity manually: `curl --proxy socks5://localhost:12345 https://dwdi.cscs.ch`
+- Ensure proxy supports the required protocol (SOCKS4/5, HTTP)
+
+## Development
+
+### Project Structure
+
+```text
+plugins/cscs-dwdi/
+├── pyproject.toml                           # Plugin configuration
+├── README.md                               # This documentation
+├── examples/                               # Configuration examples
+├── waldur_site_agent_cscs_dwdi/
+│   ├── __init__.py                         # Package init
+│   ├── backend.py                          # Backend implementations
+│   └── client.py                          # CSCS-DWDI API client
+└── tests/
+    └── test_cscs_dwdi.py                  # Plugin tests
+```
+
+### Key Classes
+
+- **`CSCSDWDIComputeBackend`**: Compute usage reporting backend
+- **`CSCSDWDIStorageBackend`**: Storage usage reporting backend
+- **`CSCSDWDIClient`**: HTTP client for CSCS-DWDI API communication
+
+### Extension Points
+
+To extend the plugin:
+
+1. **Additional Endpoints**: Modify `CSCSDWDIClient` to support more API endpoints
+2. **Authentication Methods**: Update authentication logic in `client.py`
+3. **Data Processing**: Enhance response processing methods for additional data formats

waldur_site_agent_cscs_dwdi-0.7.3/README.md

@@ -0,0 +1,274 @@
+# CSCS-DWDI Plugin for Waldur Site Agent
+
+This plugin provides integration with the CSCS Data Warehouse Data Intelligence (DWDI) system to report both
+computational and storage usage data to Waldur.
+
+## Overview
+
+The plugin implements two separate backends to handle different types of accounting data:
+
+- **Compute Backend** (`cscs-dwdi-compute`): Reports CPU and node hour usage from HPC clusters
+- **Storage Backend** (`cscs-dwdi-storage`): Reports storage space and inode usage from filesystems
+
+## Backend Types
+
+### Compute Backend
+
+The compute backend queries the DWDI API for computational resource usage and reports:
+
+- Node hours consumed by accounts and users
+- CPU hours consumed by accounts and users
+- Account-level and user-level usage aggregation
+
+**API Endpoints Used:**
+
+- `/api/v1/compute/usage-month/account` - Monthly usage data
+- `/api/v1/compute/usage-day/account` - Daily usage data
+
+### Storage Backend
+
+The storage backend queries the DWDI API for storage resource usage and reports:
+
+- Storage space used (converted from bytes to configured units)
+- Inode (file count) usage
+- Path-based resource identification
+
+**API Endpoints Used:**
+
+- `/api/v1/storage/usage-month/filesystem_name/data_type` - Monthly storage usage
+- `/api/v1/storage/usage-day/filesystem_name/data_type` - Daily storage usage
+
+## Configuration
+
+### Compute Backend Configuration
+
+```yaml
+backend_type: "cscs-dwdi-compute"
+
+backend_settings:
+  cscs_dwdi_api_url: "https://dwdi.cscs.ch"
+  cscs_dwdi_client_id: "your_oidc_client_id"
+  cscs_dwdi_client_secret: "your_oidc_client_secret"
+  cscs_dwdi_oidc_token_url: "https://auth.cscs.ch/realms/cscs/protocol/openid-connect/token"
+  cscs_dwdi_oidc_scope: "openid"  # Optional
+
+backend_components:
+  nodeHours:
+    measured_unit: "node-hours"
+    unit_factor: 1
+    accounting_type: "usage"
+    label: "Node Hours"
+
+  cpuHours:
+    measured_unit: "cpu-hours"
+    unit_factor: 1
+    accounting_type: "usage"
+    label: "CPU Hours"
+```
+
+### Storage Backend Configuration
+
+```yaml
+backend_type: "cscs-dwdi-storage"
+
+backend_settings:
+  cscs_dwdi_api_url: "https://dwdi.cscs.ch"
+  cscs_dwdi_client_id: "your_oidc_client_id"
+  cscs_dwdi_client_secret: "your_oidc_client_secret"
+  cscs_dwdi_oidc_token_url: "https://auth.cscs.ch/realms/cscs/protocol/openid-connect/token"
+
+  # Storage-specific settings
+  storage_filesystem: "lustre"
+  storage_data_type: "projects"
+  storage_tenant: "cscs"  # Optional
+
+  # Map Waldur resource IDs to storage paths
+  storage_path_mapping:
+    "project_123": "/store/projects/proj123"
+    "project_456": "/store/projects/proj456"
+
+backend_components:
+  storage_space:
+    measured_unit: "GB"
+    unit_factor: 0.000000001  # Convert bytes to GB
+    accounting_type: "usage"
+    label: "Storage Space (GB)"
+
+  storage_inodes:
+    measured_unit: "count"
+    unit_factor: 1
+    accounting_type: "usage"
+    label: "File Count"
+```
+
+## Authentication
+
+Both backends use OIDC client credentials flow for authentication with the DWDI API. You need:
+
+- `cscs_dwdi_client_id`: OIDC client identifier
+- `cscs_dwdi_client_secret`: OIDC client secret
+- `cscs_dwdi_oidc_token_url`: OIDC token endpoint URL
+- `cscs_dwdi_oidc_scope`: OIDC scope (optional, defaults to "openid")
+
+## SOCKS Proxy Support
+
+Both backends support SOCKS proxy for network connectivity. This is useful when the DWDI API is only accessible
+through a proxy or jump host.
+
+### SOCKS Proxy Configuration
+
+Add the SOCKS proxy setting to your backend configuration:
+
+```yaml
+backend_settings:
+  # ... other settings ...
+  socks_proxy: "socks5://localhost:12345"  # SOCKS5 proxy URL
+```
+
+### Supported Proxy Types
+
+- **SOCKS5**: `socks5://hostname:port`
+- **SOCKS4**: `socks4://hostname:port`
+- **HTTP**: `http://hostname:port`
+
+### Usage Examples
+
+**SSH Tunnel with SOCKS5:**
+
+```bash
+# Create SSH tunnel to jump host
+ssh -D 12345 -N user@jumphost.cscs.ch
+
+# Configure backend to use tunnel
+backend_settings:
+  socks_proxy: "socks5://localhost:12345"
+```
+
+**HTTP Proxy:**
+
+```yaml
+backend_settings:
+  socks_proxy: "http://proxy.cscs.ch:8080"
+```
+
+## Resource Identification
+
+### Compute Resources
+
+For compute resources, the system uses account names as returned by the DWDI API. The Waldur resource
+`backend_id` should match the account name in the cluster accounting system.
+
+### Storage Resources
+
+For storage resources, there are two options:
+
+1. **Direct Path Usage**: Set the Waldur resource `backend_id` to the actual filesystem path
+2. **Path Mapping**: Use the `storage_path_mapping` setting to map resource IDs to paths
+
+## Usage Reporting
+
+Both backends are read-only and designed for usage reporting. They implement the `_get_usage_report()` method
+but do not support:
+
+- Account creation/deletion
+- Resource management
+- User management
+- Limit setting
+
+## Example Configurations
+
+See the `examples/` directory for complete configuration examples:
+
+- `cscs-dwdi-compute-config.yaml` - Compute backend only
+- `cscs-dwdi-storage-config.yaml` - Storage backend only
+- `cscs-dwdi-combined-config.yaml` - Both backends in one configuration
+
+## Installation
+
+The plugin is automatically discovered when the waldur-site-agent-cscs-dwdi package is installed alongside waldur-site-agent.
+
+```bash
+# Install all workspace packages including cscs-dwdi plugin
+uv sync --all-packages
+```
+
+## Testing
+
+Run the test suite:
+
+```bash
+uv run pytest plugins/cscs-dwdi/tests/
+```
+
+## API Compatibility
+
+This plugin is compatible with DWDI API version 1 (`/api/v1/`). It requires the following API endpoints to be available:
+
+**Compute API:**
+
+- `/api/v1/compute/usage-month/account`
+- `/api/v1/compute/usage-day/account`
+
+**Storage API:**
+
+- `/api/v1/storage/usage-month/filesystem_name/data_type`
+- `/api/v1/storage/usage-day/filesystem_name/data_type`
+
+## Troubleshooting
+
+### Authentication Issues
+
+- Verify OIDC client credentials are correct
+- Check that the token endpoint URL is accessible
+- Ensure the client has appropriate scopes
+
+### Storage Backend Issues
+
+- Verify `storage_filesystem` and `storage_data_type` match available values in DWDI
+- Check `storage_path_mapping` if using custom resource IDs
+- Ensure storage paths exist in the DWDI system
+
+### Connection Issues
+
+- Use the `ping()` method to test API connectivity
+- Check network connectivity to the DWDI API endpoint
+- Verify SSL/TLS configuration
+- If behind a firewall, configure SOCKS proxy (`socks_proxy` setting)
+
+### Proxy Issues
+
+- Verify proxy server is running and accessible
+- Check proxy authentication if required
+- Test proxy connectivity manually: `curl --proxy socks5://localhost:12345 https://dwdi.cscs.ch`
+- Ensure proxy supports the required protocol (SOCKS4/5, HTTP)
+
+## Development
+
+### Project Structure
+
+```text
+plugins/cscs-dwdi/
+├── pyproject.toml                           # Plugin configuration
+├── README.md                               # This documentation
+├── examples/                               # Configuration examples
+├── waldur_site_agent_cscs_dwdi/
+│   ├── __init__.py                         # Package init
+│   ├── backend.py                          # Backend implementations
+│   └── client.py                          # CSCS-DWDI API client
+└── tests/
+    └── test_cscs_dwdi.py                  # Plugin tests
+```
+
+### Key Classes
+
+- **`CSCSDWDIComputeBackend`**: Compute usage reporting backend
+- **`CSCSDWDIStorageBackend`**: Storage usage reporting backend
+- **`CSCSDWDIClient`**: HTTP client for CSCS-DWDI API communication
+
+### Extension Points
+
+To extend the plugin:
+
+1. **Additional Endpoints**: Modify `CSCSDWDIClient` to support more API endpoints
+2. **Authentication Methods**: Update authentication logic in `client.py`
+3. **Data Processing**: Enhance response processing methods for additional data formats

{waldur_site_agent_cscs_dwdi-0.7.0 → waldur_site_agent_cscs_dwdi-0.7.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "waldur-site-agent-cscs-dwdi"
-version = "0.7.0"
+version = "0.7.3"
 description = "CSCS-DWDI reporting plugin for Waldur Site Agent"
 readme = "README.md"
 authors = [{ name = "OpenNode Team", email = "info@opennodecloud.com" }]
@@ -19,4 +19,5 @@ waldur-site-agent = { workspace = true }
 
 # Entry points for exporting backends
 [project.entry-points."waldur_site_agent.backends"]
-cscs-dwdi = "waldur_site_agent_cscs_dwdi.backend:CSCSDWDIBackend"
+cscs-dwdi-compute = "waldur_site_agent_cscs_dwdi.backend:CSCSDWDIComputeBackend"
+cscs-dwdi-storage = "waldur_site_agent_cscs_dwdi.backend:CSCSDWDIStorageBackend"

{waldur_site_agent_cscs_dwdi-0.7.0 → waldur_site_agent_cscs_dwdi-0.7.3}/tests/test_cscs_dwdi.py

@@ -5,7 +5,7 @@ from typing import Any
 from unittest.mock import MagicMock, patch
 
 import pytest
-from waldur_site_agent_cscs_dwdi.backend import CSCSDWDIBackend
+from waldur_site_agent_cscs_dwdi.backend import CSCSDWDIComputeBackend, CSCSDWDIStorageBackend
 from waldur_site_agent_cscs_dwdi.client import CSCSDWDIClient
 
 
@@ -25,6 +25,18 @@ class TestCSCSDWDIClient:
         assert client.client_secret == "test_secret"
         assert client.oidc_token_url is None
         assert client.oidc_scope == "openid"
+        assert client.socks_proxy is None
+
+    def test_client_initialization_with_proxy(self) -> None:
+        """Test client initializes with SOCKS proxy."""
+        client = CSCSDWDIClient(
+            api_url="https://api.example.com",
+            client_id="test_client",
+            client_secret="test_secret",
+            socks_proxy="socks5://localhost:12345",
+        )
+
+        assert client.socks_proxy == "socks5://localhost:12345"
 
     def test_client_strips_trailing_slash(self) -> None:
         """Test client strips trailing slash from API URL."""
@@ -230,8 +242,8 @@ class TestCSCSDWDIClient:
         assert token1 == token2
 
 
-class TestCSCSDWDIBackend:
-    """Tests for CSCS-DWDI backend."""
+class TestCSCSDWDIComputeBackend:
+    """Tests for CSCS-DWDI compute backend."""
 
     def test_backend_initialization(self) -> None:
         """Test backend initializes with correct configuration."""
@@ -250,12 +262,36 @@ class TestCSCSDWDIBackend:
             }
         }
 
-        backend = CSCSDWDIBackend(backend_settings, backend_components)
+        backend = CSCSDWDIComputeBackend(backend_settings, backend_components)
 
         assert backend.api_url == "https://api.example.com"
         assert backend.client_id == "test_client"
         assert backend.client_secret == "test_secret"
         assert isinstance(backend.cscs_client, CSCSDWDIClient)
+        assert backend.socks_proxy is None
+
+    def test_backend_initialization_with_proxy(self) -> None:
+        """Test backend initializes with SOCKS proxy configuration."""
+        backend_settings = {
+            "cscs_dwdi_api_url": "https://api.example.com",
+            "cscs_dwdi_client_id": "test_client",
+            "cscs_dwdi_client_secret": "test_secret",
+            "cscs_dwdi_oidc_token_url": "https://oidc.example.com/token",
+            "socks_proxy": "socks5://localhost:12345",
+        }
+        backend_components = {
+            "nodeHours": {
+                "measured_unit": "node-hours",
+                "unit_factor": 1,
+                "accounting_type": "usage",
+                "label": "Node Hours",
+            }
+        }
+
+        backend = CSCSDWDIComputeBackend(backend_settings, backend_components)
+
+        assert backend.socks_proxy == "socks5://localhost:12345"
+        assert backend.cscs_client.socks_proxy == "socks5://localhost:12345"
 
     def test_backend_initialization_missing_config(self) -> None:
         """Test backend raises error when configuration is missing."""
@@ -266,7 +302,7 @@ class TestCSCSDWDIBackend:
         backend_components: dict[str, dict] = {}
 
         with pytest.raises(ValueError) as exc_info:
-            CSCSDWDIBackend(backend_settings, backend_components)
+            CSCSDWDIComputeBackend(backend_settings, backend_components)
 
         assert "cscs_dwdi_oidc_token_url" in str(exc_info.value)
 
@@ -286,7 +322,7 @@ class TestCSCSDWDIBackend:
                 "label": "Node Hours",
             }
         }
-        backend = CSCSDWDIBackend(backend_settings, backend_components)
+        backend = CSCSDWDIComputeBackend(backend_settings, backend_components)
 
         # Sample API response
         api_response = {
@@ -347,7 +383,7 @@ class TestCSCSDWDIBackend:
                 "label": "Node Hours",
             }
         }
-        backend = CSCSDWDIBackend(backend_settings, backend_components)
+        backend = CSCSDWDIComputeBackend(backend_settings, backend_components)
 
         # API response with same user appearing multiple times
         api_response = {
@@ -416,7 +452,7 @@ class TestCSCSDWDIBackend:
                 "label": "Node Hours",
             }
         }
-        backend = CSCSDWDIBackend(backend_settings, backend_components)
+        backend = CSCSDWDIComputeBackend(backend_settings, backend_components)
 
         # Test the method
         result = backend._get_usage_report(["account1", "account2"])
@@ -448,7 +484,7 @@ class TestCSCSDWDIBackend:
                 "label": "Node Hours",
             }
         }
-        backend = CSCSDWDIBackend(backend_settings, backend_components)
+        backend = CSCSDWDIComputeBackend(backend_settings, backend_components)
 
         # Mock the client method
         with patch.object(backend.cscs_client, "get_usage_for_month") as mock_get:
@@ -496,7 +532,7 @@ class TestCSCSDWDIBackend:
                 "label": "Node Hours",
             }
         }
-        backend = CSCSDWDIBackend(backend_settings, backend_components)
+        backend = CSCSDWDIComputeBackend(backend_settings, backend_components)
 
         with pytest.raises(NotImplementedError):
             backend.create_account({})