elaunira-airflow-provider-r2index 0.3.0__tar.gz → 0.3.2__tar.gz

{elaunira_airflow_provider_r2index-0.3.0 → elaunira_airflow_provider_r2index-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: elaunira-airflow-provider-r2index
-Version: 0.3.0
+Version: 0.3.2
 Summary: Airflow provider for Elaunira R2Index connections
 Project-URL: Repository, https://github.com/elaunira/elaunira-airflow-provider-r2index
 License-Expression: MIT

elaunira_airflow_provider_r2index-0.3.2/README.md

@@ -0,0 +1,169 @@
+# Elaunira Airflow Provider for R2Index
+
+Airflow provider package for R2Index, providing connection type, operators, and TaskFlow decorators.
+
+## Installation
+
+```bash
+pip install elaunira-airflow-provider-r2index
+```
+
+## Features
+
+- **R2Index** connection type in Airflow UI
+- `R2IndexUploadOperator` and `R2IndexDownloadOperator` for file transfers
+- `@task.r2index_upload` and `@task.r2index_download` TaskFlow decorators
+
+## Connection Configuration
+
+After installation, the **R2Index** connection type will be available in Airflow's connection UI.
+
+### Vault/OpenBao Mode (Recommended)
+
+Fetches credentials dynamically from HashiCorp Vault or OpenBao:
+
+| Field | Description |
+|-------|-------------|
+| Vault Connection ID | Airflow Vault connection ID (e.g., `openbao-elaunira`) |
+| Vault Namespace | OpenBao namespace (e.g., `elaunira/production`) |
+| Vault Secrets Mapping | JSON mapping of config keys to secret paths |
+
+Example Vault Secrets Mapping:
+```json
+{
+    "r2index_api_url": "cloudflare/r2index#api-url",
+    "r2index_api_token": "cloudflare/r2index#api-token",
+    "r2_access_key_id": "cloudflare/r2/e2e-tests#access-key-id",
+    "r2_secret_access_key": "cloudflare/r2/e2e-tests#secret-access-key",
+    "r2_endpoint_url": "cloudflare/r2/e2e-tests#endpoint-url"
+}
+```
+
+Secret path format: `path#key` (e.g., `cloudflare/r2index#api-url`)
+
+### Direct Mode
+
+Store credentials directly in the connection:
+
+| Field | Description |
+|-------|-------------|
+| R2Index API URL | API endpoint URL |
+| R2Index API Token | API authentication token |
+| R2 Access Key ID | Cloudflare R2 access key |
+| R2 Secret Access Key | Cloudflare R2 secret key |
+| R2 Endpoint URL | Cloudflare R2 endpoint |
+
+### Environment Variables Fallback
+
+If no connection is configured, the hook falls back to environment variables:
+
+- `R2INDEX_API_URL`
+- `R2INDEX_API_TOKEN`
+- `R2_ACCESS_KEY_ID`
+- `R2_SECRET_ACCESS_KEY`
+- `R2_ENDPOINT_URL`
+
+## Operators
+
+### R2IndexUploadOperator
+
+Upload files to R2Index:
+
+```python
+from elaunira.airflow.provider.r2index.operators import R2IndexUploadOperator, UploadItem
+
+upload = R2IndexUploadOperator(
+    task_id="upload_file",
+    bucket="my-bucket",
+    r2index_conn_id="my_r2index_connection",
+    items=UploadItem(
+        source="/tmp/data.csv",
+        category="example",
+        entity="sample-data",
+        extension="csv",
+        media_type="text/csv",
+        destination_path="example/data",
+        destination_filename="data.csv",
+        destination_version="{{ ds }}",
+    ),
+)
+```
+
+### R2IndexDownloadOperator
+
+Download files from R2Index:
+
+```python
+from elaunira.airflow.provider.r2index.operators import R2IndexDownloadOperator, DownloadItem
+
+download = R2IndexDownloadOperator(
+    task_id="download_file",
+    bucket="my-bucket",
+    r2index_conn_id="my_r2index_connection",
+    items=DownloadItem(
+        source_path="example/data",
+        source_filename="data.csv",
+        source_version="{{ ds }}",
+        destination="/tmp/downloaded.csv",
+    ),
+)
+```
+
+## TaskFlow Decorators
+
+### @task.r2index_upload
+
+```python
+from airflow.sdk import dag, task
+from elaunira.airflow.provider.r2index.operators import UploadItem
+
+@dag(schedule=None)
+def my_dag():
+    @task.r2index_upload(bucket="my-bucket", r2index_conn_id="my_connection")
+    def prepare_upload() -> UploadItem:
+        return UploadItem(
+            source="/tmp/data.csv",
+            category="example",
+            entity="sample-data",
+            extension="csv",
+            media_type="text/csv",
+            destination_path="example/data",
+            destination_filename="data.csv",
+            destination_version="2024-01-01",
+        )
+
+    prepare_upload()
+```
+
+### @task.r2index_download
+
+```python
+from airflow.sdk import dag, task
+from elaunira.airflow.provider.r2index.operators import DownloadItem
+
+@dag(schedule=None)
+def my_dag():
+    @task.r2index_download(bucket="my-bucket", r2index_conn_id="my_connection")
+    def prepare_download() -> DownloadItem:
+        return DownloadItem(
+            source_path="example/data",
+            source_filename="data.csv",
+            source_version="2024-01-01",
+            destination="/tmp/downloaded.csv",
+        )
+
+    prepare_download()
+```
+
+## Hook Usage
+
+```python
+from elaunira.airflow.provider.r2index.hooks import R2IndexHook
+
+hook = R2IndexHook(r2index_conn_id="my_r2index_connection")
+client = hook.get_conn()
+```
+
+## License
+
+MIT

{elaunira_airflow_provider_r2index-0.3.0 → elaunira_airflow_provider_r2index-0.3.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "elaunira-airflow-provider-r2index"
-version = "0.3.0"
+version = "0.3.2"
 description = "Airflow provider for Elaunira R2Index connections"
 requires-python = ">=3.12"
 license = "MIT"

{elaunira_airflow_provider_r2index-0.3.0 → elaunira_airflow_provider_r2index-0.3.2}/src/elaunira/airflow/provider/r2index/hooks/r2index.py

@@ -191,14 +191,20 @@ class R2IndexHook(BaseHook):
 
                 return secret_cache[path].get(key)
 
-            return {
+            config = {
                 "index_api_url": get_secret_value("r2index_api_url"),
                 "index_api_token": get_secret_value("r2index_api_token"),
                 "r2_access_key_id": get_secret_value("r2_access_key_id"),
                 "r2_secret_access_key": get_secret_value("r2_secret_access_key"),
                 "r2_endpoint_url": get_secret_value("r2_endpoint_url"),
             }
-        except Exception:
+            # Log which values are missing
+            missing = [k for k, v in config.items() if v is None]
+            if missing:
+                self.log.warning("Missing Vault secrets: %s", missing)
+            return config
+        except Exception as e:
+            self.log.error("Failed to get config from Vault: %s", e)
             return None
 
     def _get_config_from_connection(self) -> dict[str, str | None] | None:
@@ -233,7 +239,8 @@ class R2IndexHook(BaseHook):
                 "r2_secret_access_key": extra.get("r2_secret_access_key"),
                 "r2_endpoint_url": extra.get("r2_endpoint_url"),
             }
-        except Exception:
+        except Exception as e:
+            self.log.error("Failed to get config from connection: %s", e)
             return None
 
     def get_conn(self) -> R2IndexClient:

{elaunira_airflow_provider_r2index-0.3.0 → elaunira_airflow_provider_r2index-0.3.2}/src/elaunira/airflow/provider/r2index/links/r2index.py

@@ -29,7 +29,7 @@ class R2IndexFileLink(BaseOperatorLink):
         """Get the link to the R2Index file."""
         from airflow.models import XCom
 
-        result = XCom.get_value(ti_key=ti_key)
+        result = XCom.get_value(ti_key=ti_key, key="return_value")
         if result and isinstance(result, dict):
             file_id = result.get("id") or result.get("file_record", {}).get("id")
             if file_id:

elaunira_airflow_provider_r2index-0.3.0/README.md

@@ -1,75 +0,0 @@
-# Elaunira Airflow Provider for R2Index
-
-Airflow provider package that adds the **Elaunira R2Index** connection type for managing R2Index credentials.
-
-## Installation
-
-```bash
-pip install elaunira-airflow-provider-r2index
-```
-
-## Usage
-
-After installation, a new connection type **Elaunira R2Index** will be available in Airflow's connection UI.
-
-### Connection Configuration
-
-The connection supports two modes:
-
-#### 1. Vault/OpenBao Mode (Recommended)
-
-Fetches credentials dynamically from HashiCorp Vault or OpenBao:
-
-| Field | Description |
-|-------|-------------|
-| Vault Connection ID | Airflow Vault connection ID (e.g., `openbao-ipregistry`) |
-| Vault Namespace | OpenBao namespace (e.g., `ipregistry/production`) |
-| Vault Secrets (JSON) | JSON mapping of config keys to secret paths |
-
-Example Vault Secrets JSON:
-```json
-{
-    "r2index_api_url": "cloudflare/r2index#api-url",
-    "r2index_api_token": "cloudflare/r2index#api-token",
-    "r2_access_key_id": "cloudflare/r2/airflow#access-key-id",
-    "r2_secret_access_key": "cloudflare/r2/airflow#secret-access-key",
-    "r2_endpoint_url": "cloudflare/r2/airflow#endpoint-url"
-}
-```
-
-Secret path format: `path#key` (e.g., `cloudflare/r2index#api-url`)
-
-#### 2. Direct Mode
-
-Store credentials directly in the connection:
-
-| Field | Description |
-|-------|-------------|
-| R2Index API URL | API endpoint URL |
-| R2Index API Token | API authentication token |
-| R2 Access Key ID | Cloudflare R2 access key |
-| R2 Secret Access Key | Cloudflare R2 secret key |
-| R2 Endpoint URL | Cloudflare R2 endpoint |
-
-### Environment Variables Fallback
-
-If no connection is configured, the hook falls back to environment variables:
-
-- `R2INDEX_API_URL`
-- `R2INDEX_API_TOKEN`
-- `R2_ACCESS_KEY_ID`
-- `R2_SECRET_ACCESS_KEY`
-- `R2_ENDPOINT_URL`
-
-## Hook Usage
-
-```python
-from elaunira.airflow.provider.r2index.hooks import R2IndexHook
-
-hook = R2IndexHook(r2index_conn_id="my_r2index_connection")
-client = hook.get_conn()
-```
-
-## License
-
-MIT