@aifabrix/builder 2.33.0 → 2.33.3

package/templates/applications/dataplane/README.md

@@ -0,0 +1,205 @@
+# Dataplane Builder
+
+Build, run, and deploy Dataplane using `@aifabrix/builder`.
+
+---
+
+## Quick Start
+
+### 1. Install
+
+```bash
+npm install -g @aifabrix/builder
+```
+
+### 2. First Time Setup
+
+```bash
+# Check your environment
+aifabrix doctor
+
+# Login to controller (change your own port)
+aifabrix login --method device --environment dev --controller http://localhost:3100
+
+# Register your application (gets you credentials automatically)
+aifabrix app register dataplane
+```
+
+### 3. Build & Run Locally
+
+```bash
+# Build the Docker image
+aifabrix build dataplane
+
+# Run locally
+aifabrix run dataplane
+```
+
+**Access your app:** <http://localhost:3111> (host port from `build.localPort`; container uses 3001)
+
+---
+
+## Testing dataplane (use DATAPLANE_TEST_GUIDE)
+
+**Use the builder's Dataplane Test Guide** for auth, health, wizard, external systems, and pipeline checks:
+
+- **In aifabrix-builder:** `integration/hubspot/DATAPLANE_TEST_GUIDE.md`
+- **Dataplane base URL:** `http://localhost:3111`
+- **Controller:** `http://localhost:3110` (login, token)
+
+The guide defines: token setup, `/health`, wizard API, external systems API, pipeline API, and quick checks. 
+Keep `build.localPort` in `variables.yaml` at **3111** so it matches that guide.
+
+**View logs:**
+
+```bash
+docker logs aifabrix-dataplane -f
+```
+
+**Stop:**
+
+```bash
+docker stop aifabrix-dataplane
+```
+
+### 4. Deploy to Azure
+
+```bash
+# Build with version tag
+aifabrix build dataplane --tag v1.0.0
+
+# Push to registry
+aifabrix push dataplane --registry myacr.azurecr.io --tag "v1.0.0,latest"
+
+# Deploy to miso-controller
+aifabrix deploy dataplane
+```
+
+---
+
+## Using miso-client
+
+> [miso-client](https://github.com/esystemsdev/aifabrix-miso-client)
+
+After registering your app, you automatically get credentials in your secret file. Use miso-client for login, RBAC, audit logs, etc.
+
+**Rotate credentials if needed:**
+
+```bash
+aifabrix app rotate-secret dataplane
+```
+
+---
+
+## Reference
+
+### Common Commands
+
+```bash
+# Development
+aifabrix build dataplane                        # Build app
+aifabrix run dataplane                          # Run locally
+aifabrix dockerfile dataplane --force           # Generate Dockerfile
+aifabrix resolve dataplane                      # Generate .env file
+
+# Deployment
+aifabrix json dataplane                         # Preview deployment JSON
+aifabrix genkey dataplane                       # Generate deployment key
+aifabrix push dataplane --registry myacr.azurecr.io # Push to ACR
+aifabrix deploy dataplane --controller <url>    # Deploy to Azure
+
+# Management
+aifabrix app register dataplane
+aifabrix app list
+aifabrix app rotate-secret dataplane
+
+# Utilities
+aifabrix doctor                                   # Check environment
+aifabrix login --method device  # Login
+aifabrix --help                                   # Get help
+```
+
+### Build Options
+
+```bash
+aifabrix build dataplane --tag v1.0.0           # Custom tag
+aifabrix build dataplane --force-template       # Force template regeneration
+aifabrix build dataplane --language typescript  # Override language detection
+```
+
+### Run Options
+
+```bash
+aifabrix run dataplane --port 3000          # Custom port
+aifabrix run dataplane --debug                  # Debug output
+```
+
+### Push Options
+
+```bash
+aifabrix push dataplane --registry myacr.azurecr.io --tag v1.0.0
+aifabrix push dataplane --registry myacr.azurecr.io --tag "v1.0.0,latest,stable"
+```
+
+### Deploy Options
+
+```bash
+aifabrix deploy dataplane
+aifabrix deploy dataplane --no-poll
+```
+
+### Login Methods
+
+```bash
+# Device code flow
+aifabrix login --method device --environment dev
+
+# Credentials (reads from secrets.local.yaml)
+aifabrix login --method credentials --app dataplane --environment dev
+
+# Explicit credentials
+aifabrix login --method credentials --app dataplane --client-id $CLIENT_ID --client-secret $CLIENT_SECRET --environment dev
+```
+
+### Environment Variables
+
+```bash
+export AIFABRIX_HOME=/custom/path
+export AIFABRIX_SECRETS=/path/to/secrets.yaml
+```
+
+---
+
+## Troubleshooting
+
+- **"Docker not running"** → Start Docker Desktop
+- **"Not logged in"** → Run `aifabrix login` first
+- **"Port already in use"** → Use `--port` flag or change `build.localPort` in `variables.yaml` (default: 3111, must match DATAPLANE_TEST_GUIDE)
+- **"Authentication failed"** → Run `aifabrix login` again
+- **"Build fails"** → Check Docker is running and `variables.yaml` → `build.secrets` path is correct
+- **"Can't connect"** → Verify infrastructure is running and PostgreSQL is accessible
+
+**Regenerate files:**
+
+```bash
+aifabrix resolve dataplane --force
+aifabrix json dataplane
+aifabrix genkey dataplane
+```
+
+---
+
+## Prerequisites
+
+- `@aifabrix/builder` installed globally
+- Docker Desktop running
+- Azure CLI installed (for push command)
+- Authenticated with controller (for deploy command)
+- PostgreSQL database (ensure infrastructure is running)
+- Redis (ensure infrastructure is running)
+- File storage configured
+- Authentication/RBAC configured
+
+---
+
+**Application**: dataplane | **Port**: 3111 (local) / 3001 (container) | **Registry**: myacr.azurecr.io | **Image**: aifabrix/dataplane:latest

package/templates/applications/dataplane/env.template

@@ -0,0 +1,143 @@
+# Environment Variables Template
+# Use kv:// references for secrets (resolved from .aifabrix/secrets.yaml)
+# Use ${VAR} for environment-specific values
+
+# =============================================================================
+# APPLICATION ENVIRONMENT
+# =============================================================================
+
+PORT=3001
+ENVIRONMENT=development
+DEBUG=false
+LOG_LEVEL=INFO
+LOG_FORMAT=json
+LOG_FILE_PATH=/mnt/data/logs/app.log
+LOCAL_MODE=false
+
+# When API_KEY is set, a matching Bearer token bypasses OAuth2 validation
+API_KEY=kv://miso-controller-api-key-secretKeyVault
+
+# API Configuration
+API_V1_STR=/api/v1
+VERSION=1.6.0
+
+# CORS Configuration
+ALLOWED_ORIGINS=http://localhost:*
+IDE_CORS_ORIGINS=
+
+# Encryption Configuration
+ENCRYPTION_KEY=kv://secrets-encryptionKeyVault
+
+# =============================================================================
+# DATABASE CONFIGURATION
+# =============================================================================
+
+DATABASE_URL=kv://databases-dataplane-0-urlKeyVault
+DB_0_PASSWORD=kv://databases-dataplane-0-passwordKeyVault
+
+# Vector and document store DB: chunks, embeddings, vector indexes (pgvector).
+# Binaries path: config.processing.fileStoragePath or /data/documents.
+VECTOR_DATABASE_URL=kv://databases-dataplane-1-urlKeyVault
+DB_1_PASSWORD=kv://databases-dataplane-1-passwordKeyVault
+
+# Logs Database Configuration (for execution, audit, ABAC traces)
+LOGS_DATABASE_URL=kv://databases-dataplane-2-urlKeyVault
+DB_2_PASSWORD=kv://databases-dataplane-2-passwordKeyVault
+
+# Records Database Configuration (for external records storage)
+RECORDS_DATABASE_URL=kv://databases-dataplane-3-urlKeyVault
+DB_3_PASSWORD=kv://databases-dataplane-3-passwordKeyVault
+
+# =============================================================================
+# REDIS CONFIGURATION
+# =============================================================================
+# Connects to external redis from aifabrix-setup
+
+REDIS_URL=kv://redis-url
+
+# =============================================================================
+# CACHE CONFIGURATION
+# =============================================================================
+
+CACHE_ENABLED=true
+CACHE_CIP_EXECUTION_TTL=1800
+CACHE_METADATA_FILTER_TTL=3600
+
+# =============================================================================
+# AUTHENTICATION CONFIGURATION
+# =============================================================================
+
+# MISO Application Client Credentials (per application)
+MISO_CLIENTID=kv://dataplane-client-idKeyVault
+MISO_CLIENTSECRET=kv://dataplane-client-secretKeyVault
+
+# Keycloak Configuration (for OAuth2 endpoints)
+KEYCLOAK_SERVER_URL=kv://keycloak-server-urlKeyVault
+KEYCLOAK_REALM=aifabrix
+
+# MISO Controller URL
+MISO_CONTROLLER_URL=http://${MISO_HOST}:${MISO_PORT}
+
+# =============================================================================
+# AI/LLM CONFIGURATION
+# =============================================================================
+
+# OpenAI Configuration
+OPENAI_API_KEY=kv://secrets-openaiApiKeyVault
+
+# Azure OpenAI Configuration
+AZURE_OPENAI_ENDPOINT=
+AZURE_OPENAI_API_KEY=kv://secrets-azureOpenaiApiKeyVault
+AZURE_OPENAI_API_VERSION=2024-02-15-preview
+AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o
+
+# =============================================================================
+# AUDIT CONFIGURATION
+# =============================================================================
+
+# General Audit Settings
+AUDIT_ENABLED=true
+MISO_LOG_FORWARD_ERRORS=true
+AUTH_AUDIT_ENABLED=true
+
+# ABAC Audit Configuration
+ABAC_AUDIT_ENABLED=true
+ABAC_AUDIT_DETAIL_LEVEL=summary
+ABAC_EXPLAIN_MODE_ENABLED=false
+ABAC_PERFORMANCE_THRESHOLD_MS=1000
+
+# RBAC Audit Configuration
+RBAC_AUDIT_ENABLED=true
+RBAC_AUDIT_DETAIL_LEVEL=summary
+RBAC_EXPLAIN_MODE_ENABLED=false
+
+# =============================================================================
+# OBSERVABILITY CONFIGURATION
+# =============================================================================
+
+# OpenTelemetry Configuration
+OPENTELEMETRY_ENABLED=false
+OPENTELEMETRY_ENDPOINT=
+
+# =============================================================================
+# CIP EXECUTION CONFIGURATION
+# =============================================================================
+# These control CIP (Composable Integration Pipeline) execution behavior
+
+CIP_EXECUTION_MAX_RESPONSE_SIZE_MB=100.0
+CIP_EXECUTION_MAX_RECORDS=100000
+CIP_EXECUTION_OPERATION_TIMEOUT=300.0
+CIP_EXECUTION_HTTP_TIMEOUT=30.0
+CIP_EXECUTION_MAX_RETRIES=3
+CIP_EXECUTION_RETRY_BACKOFF_FACTOR=2.0
+CIP_EXECUTION_RETRY_INITIAL_DELAY=1.0
+
+# Circuit Breaker Configuration
+CIP_EXECUTION_CIRCUIT_BREAKER_FAILURE_THRESHOLD=5
+CIP_EXECUTION_CIRCUIT_BREAKER_TIME_WINDOW=60
+CIP_EXECUTION_CIRCUIT_BREAKER_SUCCESS_THRESHOLD=2
+CIP_EXECUTION_CIRCUIT_BREAKER_HALF_OPEN_TIMEOUT=30
+
+# Rate Limiting Configuration
+CIP_EXECUTION_RATE_LIMIT_REQUESTS_PER_SECOND=10.0
+CIP_EXECUTION_RATE_LIMIT_BURST_SIZE=20

package/templates/applications/dataplane/rbac.yaml

@@ -0,0 +1,283 @@
+# AI Fabrix Dataplane - RBAC Configuration
+# Roles and permissions for this application
+#
+# Note: Environment access is managed by MisoClient, not by per-environment roles.
+# MisoClient validates environment access automatically based on user/application permissions.
+
+roles:
+  - name: "AI Fabrix Platform Admin"
+    value: "aifabrix-platform-admin"
+    description: "Full platform infrastructure management and enterprise controller access"
+    groups: ["AI-Fabrix-Platform-Admins"]
+
+  - name: "AI Fabrix Security Admin"
+    value: "aifabrix-security-admin"
+    description: "Security and compliance management for enterprise controller"
+    groups: ["AI-Fabrix-Security-Admins"]
+
+  - name: "AI Fabrix Deployment Admin"
+    value: "aifabrix-deployment-admin"
+    description: "Application deployment orchestration and environment management"
+    groups: ["AI-Fabrix-Deployment-Admins"]
+
+  - name: "AI Fabrix Compliance Admin"
+    value: "aifabrix-compliance-admin"
+    description: "ISO 27001 compliance monitoring and audit management"
+    groups: ["AI-Fabrix-Compliance-Admins"]
+
+  - name: "AI Fabrix Developer"
+    value: "aifabrix-developer"
+    description: "Developer access to deploy applications via GitHub Actions"
+    groups: ["AI-Fabrix-Developers"]
+
+  - name: "AI Fabrix Observer"
+    value: "aifabrix-observer"
+    description: "Read-only access to monitoring, logs, and compliance reports"
+    groups: ["AI-Fabrix-Observers"]
+
+permissions:
+  # Credential management
+  - name: "credential:create"
+    roles: ["aifabrix-platform-admin"]
+    description: "Create credentials"
+  
+  - name: "credential:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-compliance-admin", "aifabrix-observer"]
+    description: "Read credentials"
+  
+  - name: "credential:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin"]
+    description: "Update credentials"
+  
+  - name: "credential:delete"
+    roles: ["aifabrix-platform-admin"]
+    description: "Delete credentials"
+  
+  # External data source management
+  - name: "external-data-source:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create external data sources"
+  
+  - name: "external-data-source:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read external data sources"
+  
+  - name: "external-data-source:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Update external data sources"
+  
+  - name: "external-data-source:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete external data sources"
+  
+  - name: "external-data-source:sync"
+    roles: ["aifabrix-platform-admin", "aifabrix-deployment-admin", "aifabrix-developer"]
+    description: "Sync external data sources"
+  
+  # External system management
+  - name: "external-system:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create external systems"
+  
+  - name: "external-system:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read external systems"
+  
+  - name: "external-system:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Update external systems"
+  
+  - name: "external-system:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete external systems"
+  
+  # Document storage management
+  - name: "document-storage:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create document storage"
+  
+  - name: "document-storage:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read document storage"
+  
+  - name: "document-storage:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Update document storage"
+  
+  - name: "document-storage:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete document storage"
+  
+  - name: "document-storage:process"
+    roles: ["aifabrix-platform-admin", "aifabrix-deployment-admin", "aifabrix-developer"]
+    description: "Process documents"
+  
+  # Integration template management
+  - name: "integration-template:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create integration templates"
+  
+  - name: "integration-template:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read integration templates"
+  
+  - name: "integration-template:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-deployment-admin", "aifabrix-developer"]
+    description: "Update integration templates"
+  
+  - name: "integration-template:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete integration templates"
+  
+  # Generic dataplane operations
+  - name: "dataplane:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read dataplane data"
+  
+  - name: "dataplane:write"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Write dataplane data"
+  
+  - name: "dataplane:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete dataplane data"
+  
+  - name: "dataplane:process"
+    roles: ["aifabrix-platform-admin", "aifabrix-deployment-admin", "aifabrix-developer"]
+    description: "Process dataplane operations"
+  
+  # External data source publishing
+  - name: "external-data-source:publish"
+    roles: ["aifabrix-platform-admin", "aifabrix-deployment-admin", "aifabrix-developer"]
+    description: "Publish external data sources"
+  
+  # External system publishing
+  - name: "external-system:publish"
+    roles: ["aifabrix-platform-admin", "aifabrix-deployment-admin", "aifabrix-developer"]
+    description: "Publish external systems"
+  
+  # Document record management
+  - name: "document-record:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create document records"
+  
+  - name: "document-record:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read document records"
+  
+  - name: "document-record:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Update document records"
+  
+  - name: "document-record:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete document records"
+  
+  - name: "document-record:write"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Write document records"
+  
+  - name: "document-record:validate"
+    roles: ["aifabrix-platform-admin", "aifabrix-deployment-admin", "aifabrix-developer"]
+    description: "Validate document records"
+  
+  - name: "document-record:approve"
+    roles: ["aifabrix-platform-admin", "aifabrix-compliance-admin"]
+    description: "Approve document records"
+  
+  # External record management
+  - name: "external-record:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create external records"
+  
+  - name: "external-record:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read external records"
+  
+  - name: "external-record:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Update external records"
+  
+  - name: "external-record:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete external records"
+  
+  # External data access grant management
+  - name: "external-data-access-grant:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create external data access grants"
+  
+  - name: "external-data-access-grant:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read external data access grants"
+  
+  - name: "external-data-access-grant:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Update external data access grants"
+  
+  - name: "external-data-access-grant:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete external data access grants"
+  
+  # User and group management
+  - name: "user:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-compliance-admin", "aifabrix-observer"]
+    description: "Read user information"
+  
+  - name: "group:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-compliance-admin", "aifabrix-observer"]
+    description: "Read group information"
+  
+  # OpenAPI file management
+  - name: "openapi-file:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read OpenAPI files"
+  
+  - name: "openapi-file:update"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Update OpenAPI files"
+  
+  - name: "openapi-file:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete OpenAPI files"
+  
+  # External data source write operations
+  - name: "external-data-source:write"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Write external data source data"
+  
+  # Record relation management
+  - name: "record-relation:create"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Create record relations"
+  
+  - name: "record-relation:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Read record relations"
+  
+  - name: "record-relation:delete"
+    roles: ["aifabrix-platform-admin", "aifabrix-developer"]
+    description: "Delete record relations"
+  
+  # Audit operations
+  - name: "audit:read"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-compliance-admin", "aifabrix-observer"]
+    description: "Read audit logs and execution history"
+  
+  # Search operations
+  - name: "document:search"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Search documents"
+  
+  - name: "record:search"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-deployment-admin", "aifabrix-compliance-admin", "aifabrix-developer", "aifabrix-observer"]
+    description: "Search records"
+  
+  # IDE simulation operations
+  - name: "dataplane:abac-simulate"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-developer"]
+    description: "Simulate ABAC policy evaluation in IDE"
+  
+  - name: "dataplane:rbac-simulate"
+    roles: ["aifabrix-platform-admin", "aifabrix-security-admin", "aifabrix-developer"]
+    description: "Simulate RBAC policy evaluation in IDE"