workspace-architect 1.3.0

package/assets/instructions/power-bi-devops-alm-best-practices.instructions.md

@@ -0,0 +1,623 @@
+---
+description: 'Comprehensive guide for Power BI DevOps, Application Lifecycle Management (ALM), CI/CD pipelines, deployment automation, and version control best practices.'
+applyTo: '**/*.{yml,yaml,ps1,json,pbix,pbir}'
+---
+
+# Power BI DevOps and Application Lifecycle Management Best Practices
+
+## Overview
+This document provides comprehensive instructions for implementing DevOps practices, CI/CD pipelines, and Application Lifecycle Management (ALM) for Power BI solutions, based on Microsoft's recommended patterns and best practices.
+
+## Power BI Project Structure and Version Control
+
+### 1. PBIP (Power BI Project) Structure
+```markdown
+// Power BI project file organization
+├── Model/
+│   ├── model.tmdl
+│   ├── tables/
+│   │   ├── FactSales.tmdl
+│   │   └── DimProduct.tmdl
+│   ├── relationships/
+│   │   └── relationships.tmdl
+│   └── measures/
+│       └── measures.tmdl
+├── Report/
+│   ├── report.json
+│   ├── pages/
+│   │   ├── ReportSection1/
+│   │   │   ├── page.json
+│   │   │   └── visuals/
+│   │   └── pages.json
+│   └── bookmarks/
+└── .git/
+```
+
+### 2. Git Integration Best Practices
+```powershell
+# Initialize Power BI project with Git
+git init
+git add .
+git commit -m "Initial Power BI project structure"
+
+# Create feature branch for development
+git checkout -b feature/new-dashboard
+git add Model/tables/NewTable.tmdl
+git commit -m "Add new dimension table"
+
+# Merge and deploy workflow
+git checkout main
+git merge feature/new-dashboard
+git tag -a v1.2.0 -m "Release version 1.2.0"
+```
+
+## Deployment Pipelines and Automation
+
+### 1. Power BI Deployment Pipelines API
+```powershell
+# Automated deployment using Power BI REST API
+$url = "pipelines/{0}/Deploy" -f "Insert your pipeline ID here"
+$body = @{ 
+    sourceStageOrder = 0 # Development (0), Test (1)
+    datasets = @(
+        @{sourceId = "Insert your dataset ID here" }
+    )      
+    reports = @(
+        @{sourceId = "Insert your report ID here" }
+    )            
+    dashboards = @(
+        @{sourceId = "Insert your dashboard ID here" }
+    )
+
+    options = @{
+        # Allows creating new item if needed on the Test stage workspace
+        allowCreateArtifact = $TRUE
+
+        # Allows overwriting existing item if needed on the Test stage workspace
+        allowOverwriteArtifact = $TRUE
+    }
+} | ConvertTo-Json
+
+$deployResult = Invoke-PowerBIRestMethod -Url $url -Method Post -Body $body | ConvertFrom-Json
+
+# Poll deployment status
+$url = "pipelines/{0}/Operations/{1}" -f "Insert your pipeline ID here",$deployResult.id
+$operation = Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json    
+while($operation.Status -eq "NotStarted" -or $operation.Status -eq "Executing")
+{
+    # Sleep for 5 seconds
+    Start-Sleep -s 5
+    $operation = Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json
+}
+```
+
+### 2. Azure DevOps Integration
+```yaml
+# Azure DevOps pipeline for Power BI deployment
+trigger:
+- main
+
+pool:
+  vmImage: windows-latest
+
+steps:
+- task: CopyFiles@2
+  inputs:
+    Contents: '**'
+    TargetFolder: '$(Build.ArtifactStagingDirectory)'
+    CleanTargetFolder: true
+    ignoreMakeDirErrors: true
+  displayName: 'Copy files from Repo'
+
+- task: PowerPlatformToolInstaller@2
+  inputs:
+    DefaultVersion: true
+
+- task: PowerPlatformExportData@2
+  inputs:
+    authenticationType: 'PowerPlatformSPN'
+    PowerPlatformSPN: 'PowerBIServiceConnection'
+    Environment: '$(BuildTools.EnvironmentUrl)'
+    SchemaFile: '$(Build.ArtifactStagingDirectory)\source\schema.xml'
+    DataFile: 'data.zip'
+  displayName: 'Export Power BI metadata'
+
+- task: PowerShell@2  
+  inputs:
+    targetType: 'inline'
+    script: |
+      # Deploy Power BI project using FabricPS-PBIP
+      $workspaceName = "$(WorkspaceName)"
+      $pbipSemanticModelPath = "$(Build.ArtifactStagingDirectory)\$(ProjectName).SemanticModel"
+      $pbipReportPath = "$(Build.ArtifactStagingDirectory)\$(ProjectName).Report"
+      
+      # Download and install FabricPS-PBIP module
+      New-Item -ItemType Directory -Path ".\modules" -ErrorAction SilentlyContinue | Out-Null
+      @("https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psm1",
+        "https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psd1") |% {
+          Invoke-WebRequest -Uri $_ -OutFile ".\modules\$(Split-Path $_ -Leaf)"
+      }
+      
+      Import-Module ".\modules\FabricPS-PBIP" -Force
+      
+      # Authenticate and deploy
+      Set-FabricAuthToken -reset
+      $workspaceId = New-FabricWorkspace -name $workspaceName -skipErrorIfExists
+      $semanticModelImport = Import-FabricItem -workspaceId $workspaceId -path $pbipSemanticModelPath
+      $reportImport = Import-FabricItem -workspaceId $workspaceId -path $pbipReportPath -itemProperties @{"semanticModelId" = $semanticModelImport.Id}
+  displayName: 'Deploy to Power BI Service'
+```
+
+### 3. Fabric REST API Deployment
+```powershell
+# Complete PowerShell deployment script
+# Parameters 
+$workspaceName = "[Workspace Name]"
+$pbipSemanticModelPath = "[PBIP Path]\[Item Name].SemanticModel"
+$pbipReportPath = "[PBIP Path]\[Item Name].Report"
+$currentPath = (Split-Path $MyInvocation.MyCommand.Definition -Parent)
+Set-Location $currentPath
+
+# Download modules and install
+New-Item -ItemType Directory -Path ".\modules" -ErrorAction SilentlyContinue | Out-Null
+@("https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psm1",
+  "https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psd1") |% {
+    Invoke-WebRequest -Uri $_ -OutFile ".\modules\$(Split-Path $_ -Leaf)"
+}
+
+if(-not (Get-Module Az.Accounts -ListAvailable)) { 
+    Install-Module Az.Accounts -Scope CurrentUser -Force
+}
+Import-Module ".\modules\FabricPS-PBIP" -Force
+
+# Authenticate
+Set-FabricAuthToken -reset
+
+# Ensure workspace exists
+$workspaceId = New-FabricWorkspace -name $workspaceName -skipErrorIfExists
+
+# Import the semantic model and save the item id
+$semanticModelImport = Import-FabricItem -workspaceId $workspaceId -path $pbipSemanticModelPath
+
+# Import the report and ensure its bound to the previous imported semantic model
+$reportImport = Import-FabricItem -workspaceId $workspaceId -path $pbipReportPath -itemProperties @{"semanticModelId" = $semanticModelImport.Id}
+```
+
+## Environment Management
+
+### 1. Multi-Environment Strategy
+```json
+{
+  "environments": {
+    "development": {
+      "workspaceId": "dev-workspace-id",
+      "dataSourceUrl": "dev-database.database.windows.net",
+      "refreshSchedule": "manual",
+      "sensitivityLabel": "Internal"
+    },
+    "test": {
+      "workspaceId": "test-workspace-id",
+      "dataSourceUrl": "test-database.database.windows.net",
+      "refreshSchedule": "daily",
+      "sensitivityLabel": "Internal"
+    },
+    "production": {
+      "workspaceId": "prod-workspace-id", 
+      "dataSourceUrl": "prod-database.database.windows.net",
+      "refreshSchedule": "hourly",
+      "sensitivityLabel": "Confidential"
+    }
+  }
+}
+```
+
+### 2. Parameter-Driven Deployment
+```powershell
+# Environment-specific parameter management
+param(
+    [Parameter(Mandatory=$true)]
+    [ValidateSet("dev", "test", "prod")]
+    [string]$Environment,
+    
+    [Parameter(Mandatory=$true)]
+    [string]$WorkspaceName,
+    
+    [Parameter(Mandatory=$false)]
+    [string]$DataSourceServer
+)
+
+# Load environment-specific configuration
+$configPath = ".\config\$Environment.json"
+$config = Get-Content $configPath | ConvertFrom-Json
+
+# Update connection strings based on environment
+$connectionString = "Data Source=$($config.dataSourceUrl);Initial Catalog=$($config.database);Integrated Security=SSPI;"
+
+# Deploy with environment-specific settings
+Write-Host "Deploying to $Environment environment..."
+Write-Host "Workspace: $($config.workspaceId)"
+Write-Host "Data Source: $($config.dataSourceUrl)"
+```
+
+## Automated Testing Framework
+
+### 1. Data Quality Tests
+```powershell
+# Automated data quality validation
+function Test-PowerBIDataQuality {
+    param(
+        [string]$WorkspaceId,
+        [string]$DatasetId
+    )
+    
+    # Test 1: Row count validation
+    $rowCountQuery = @"
+        EVALUATE
+        ADDCOLUMNS(
+            SUMMARIZE(Sales, Sales[Year]),
+            "RowCount", COUNTROWS(Sales),
+            "ExpectedMin", 1000,
+            "Test", IF(COUNTROWS(Sales) >= 1000, "PASS", "FAIL")
+        )
+"@
+    
+    # Test 2: Data freshness validation  
+    $freshnessQuery = @"
+        EVALUATE
+        ADDCOLUMNS(
+            ROW("LastRefresh", MAX(Sales[Date])),
+            "DaysOld", DATEDIFF(MAX(Sales[Date]), TODAY(), DAY),
+            "Test", IF(DATEDIFF(MAX(Sales[Date]), TODAY(), DAY) <= 1, "PASS", "FAIL")
+        )
+"@
+    
+    # Execute tests
+    $rowCountResult = Invoke-PowerBIRestMethod -Url "groups/$WorkspaceId/datasets/$DatasetId/executeQueries" -Method Post -Body (@{queries=@(@{query=$rowCountQuery})} | ConvertTo-Json)
+    $freshnessResult = Invoke-PowerBIRestMethod -Url "groups/$WorkspaceId/datasets/$DatasetId/executeQueries" -Method Post -Body (@{queries=@(@{query=$freshnessQuery})} | ConvertTo-Json)
+    
+    return @{
+        RowCountTest = $rowCountResult
+        FreshnessTest = $freshnessResult
+    }
+}
+```
+
+### 2. Performance Regression Tests
+```powershell
+# Performance benchmark testing
+function Test-PowerBIPerformance {
+    param(
+        [string]$WorkspaceId,
+        [string]$ReportId
+    )
+    
+    $performanceTests = @(
+        @{
+            Name = "Dashboard Load Time"
+            Query = "EVALUATE TOPN(1000, Sales)"
+            MaxDurationMs = 5000
+        },
+        @{
+            Name = "Complex Calculation"
+            Query = "EVALUATE ADDCOLUMNS(Sales, 'ComplexCalc', [Sales] * [Profit Margin %])"
+            MaxDurationMs = 10000
+        }
+    )
+    
+    $results = @()
+    foreach ($test in $performanceTests) {
+        $startTime = Get-Date
+        $result = Invoke-PowerBIRestMethod -Url "groups/$WorkspaceId/datasets/$DatasetId/executeQueries" -Method Post -Body (@{queries=@(@{query=$test.Query})} | ConvertTo-Json)
+        $endTime = Get-Date
+        $duration = ($endTime - $startTime).TotalMilliseconds
+        
+        $results += @{
+            TestName = $test.Name
+            Duration = $duration
+            Passed = $duration -le $test.MaxDurationMs
+            Threshold = $test.MaxDurationMs
+        }
+    }
+    
+    return $results
+}
+```
+
+## Configuration Management
+
+### 1. Infrastructure as Code
+```json
+{
+  "workspace": {
+    "name": "Production Analytics",
+    "description": "Production Power BI workspace for sales analytics",
+    "capacityId": "A1-capacity-id",
+    "users": [
+      {
+        "emailAddress": "admin@contoso.com",
+        "accessRight": "Admin"
+      },
+      {
+        "emailAddress": "powerbi-service-principal@contoso.com", 
+        "accessRight": "Member",
+        "principalType": "App"
+      }
+    ],
+    "settings": {
+      "datasetDefaultStorageFormat": "Large",
+      "blockResourceKeyAuthentication": true
+    }
+  },
+  "datasets": [
+    {
+      "name": "Sales Analytics",
+      "refreshSchedule": {
+        "enabled": true,
+        "times": ["06:00", "12:00", "18:00"],
+        "days": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
+        "timeZone": "UTC"
+      },
+      "datasourceCredentials": {
+        "credentialType": "OAuth2",
+        "encryptedConnection": "Encrypted"
+      }
+    }
+  ]
+}
+```
+
+### 2. Secret Management
+```powershell
+# Azure Key Vault integration for secrets
+function Get-PowerBICredentials {
+    param(
+        [string]$KeyVaultName,
+        [string]$Environment
+    )
+    
+    # Retrieve secrets from Key Vault
+    $servicePrincipalId = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "PowerBI-ServicePrincipal-Id-$Environment" -AsPlainText
+    $servicePrincipalSecret = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "PowerBI-ServicePrincipal-Secret-$Environment" -AsPlainText
+    $tenantId = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "PowerBI-TenantId-$Environment" -AsPlainText
+    
+    return @{
+        ServicePrincipalId = $servicePrincipalId
+        ServicePrincipalSecret = $servicePrincipalSecret
+        TenantId = $tenantId
+    }
+}
+
+# Authenticate using retrieved credentials
+$credentials = Get-PowerBICredentials -KeyVaultName "PowerBI-KeyVault" -Environment "Production"
+$securePassword = ConvertTo-SecureString $credentials.ServicePrincipalSecret -AsPlainText -Force
+$credential = New-Object System.Management.Automation.PSCredential($credentials.ServicePrincipalId, $securePassword)
+Connect-PowerBIServiceAccount -ServicePrincipal -Credential $credential -TenantId $credentials.TenantId
+```
+
+## Release Management
+
+### 1. Release Pipeline
+```yaml
+# Multi-stage release pipeline
+stages:
+- stage: Build
+  displayName: 'Build Stage'
+  jobs:
+  - job: Build
+    steps:
+    - task: PowerShell@2
+      displayName: 'Validate Power BI Project'
+      inputs:
+        targetType: 'inline'
+        script: |
+          # Validate PBIP structure
+          if (-not (Test-Path "Model\model.tmdl")) {
+            throw "Missing model.tmdl file"
+          }
+          
+          # Validate required files
+          $requiredFiles = @("Report\report.json", "Model\tables")
+          foreach ($file in $requiredFiles) {
+            if (-not (Test-Path $file)) {
+              throw "Missing required file: $file"
+            }
+          }
+          
+          Write-Host "✅ Project validation passed"
+
+- stage: DeployTest
+  displayName: 'Deploy to Test'
+  dependsOn: Build
+  condition: succeeded()
+  jobs:
+  - deployment: DeployTest
+    environment: 'PowerBI-Test'
+    strategy:
+      runOnce:
+        deploy:
+          steps:
+          - template: deploy-powerbi.yml
+            parameters:
+              environment: 'test'
+              workspaceName: '$(TestWorkspaceName)'
+
+- stage: DeployProd
+  displayName: 'Deploy to Production'
+  dependsOn: DeployTest
+  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
+  jobs:
+  - deployment: DeployProd
+    environment: 'PowerBI-Production'
+    strategy:
+      runOnce:
+        deploy:
+          steps:
+          - template: deploy-powerbi.yml
+            parameters:
+              environment: 'prod'
+              workspaceName: '$(ProdWorkspaceName)'
+```
+
+### 2. Rollback Strategy
+```powershell
+# Automated rollback mechanism
+function Invoke-PowerBIRollback {
+    param(
+        [string]$WorkspaceId,
+        [string]$BackupVersion,
+        [string]$BackupLocation
+    )
+    
+    Write-Host "Initiating rollback to version: $BackupVersion"
+    
+    # Step 1: Export current state as emergency backup
+    $emergencyBackup = "emergency-backup-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
+    Export-PowerBIReport -WorkspaceId $WorkspaceId -BackupName $emergencyBackup
+    
+    # Step 2: Restore from backup
+    $backupPath = Join-Path $BackupLocation "$BackupVersion.pbix"
+    if (Test-Path $backupPath) {
+        Import-PowerBIReport -WorkspaceId $WorkspaceId -FilePath $backupPath -ConflictAction "Overwrite"
+        Write-Host "✅ Rollback completed successfully"
+    } else {
+        throw "Backup file not found: $backupPath"
+    }
+    
+    # Step 3: Validate rollback
+    Test-PowerBIDataQuality -WorkspaceId $WorkspaceId
+}
+```
+
+## Monitoring and Alerting
+
+### 1. Deployment Health Checks
+```powershell
+# Post-deployment validation
+function Test-DeploymentHealth {
+    param(
+        [string]$WorkspaceId,
+        [array]$ExpectedReports,
+        [array]$ExpectedDatasets
+    )
+    
+    $healthCheck = @{
+        Status = "Healthy"
+        Issues = @()
+        Timestamp = Get-Date
+    }
+    
+    # Check reports
+    $reports = Get-PowerBIReport -WorkspaceId $WorkspaceId
+    foreach ($expectedReport in $ExpectedReports) {
+        if (-not ($reports.Name -contains $expectedReport)) {
+            $healthCheck.Issues += "Missing report: $expectedReport"
+            $healthCheck.Status = "Unhealthy"
+        }
+    }
+    
+    # Check datasets
+    $datasets = Get-PowerBIDataset -WorkspaceId $WorkspaceId
+    foreach ($expectedDataset in $ExpectedDatasets) {
+        $dataset = $datasets | Where-Object { $_.Name -eq $expectedDataset }
+        if (-not $dataset) {
+            $healthCheck.Issues += "Missing dataset: $expectedDataset"
+            $healthCheck.Status = "Unhealthy"
+        } elseif ($dataset.RefreshState -eq "Failed") {
+            $healthCheck.Issues += "Dataset refresh failed: $expectedDataset"
+            $healthCheck.Status = "Degraded"
+        }
+    }
+    
+    return $healthCheck
+}
+```
+
+### 2. Automated Alerting
+```powershell
+# Teams notification for deployment status
+function Send-DeploymentNotification {
+    param(
+        [string]$TeamsWebhookUrl,
+        [object]$DeploymentResult,
+        [string]$Environment
+    )
+    
+    $color = switch ($DeploymentResult.Status) {
+        "Success" { "28A745" }
+        "Warning" { "FFC107" }
+        "Failed" { "DC3545" }
+    }
+    
+    $teamsMessage = @{
+        "@type" = "MessageCard"
+        "@context" = "https://schema.org/extensions"
+        "summary" = "Power BI Deployment $($DeploymentResult.Status)"
+        "themeColor" = $color
+        "sections" = @(
+            @{
+                "activityTitle" = "Power BI Deployment to $Environment"
+                "activitySubtitle" = "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
+                "facts" = @(
+                    @{
+                        "name" = "Status"
+                        "value" = $DeploymentResult.Status
+                    },
+                    @{
+                        "name" = "Duration"
+                        "value" = "$($DeploymentResult.Duration) minutes"
+                    },
+                    @{
+                        "name" = "Reports Deployed"
+                        "value" = $DeploymentResult.ReportsCount
+                    }
+                )
+            }
+        )
+    }
+    
+    Invoke-RestMethod -Uri $TeamsWebhookUrl -Method Post -Body ($teamsMessage | ConvertTo-Json -Depth 10) -ContentType 'application/json'
+}
+```
+
+## Best Practices Summary
+
+### ✅ DevOps Best Practices
+
+1. **Version Control Everything**
+   - Use PBIP format for source control
+   - Include model, reports, and configuration
+   - Implement branching strategies (GitFlow)
+
+2. **Automated Testing**
+   - Data quality validation
+   - Performance regression tests
+   - Security compliance checks
+
+3. **Environment Isolation**
+   - Separate dev/test/prod environments
+   - Environment-specific configurations
+   - Automated promotion pipelines
+
+4. **Security Integration**
+   - Service principal authentication
+   - Secret management with Key Vault
+   - Role-based access controls
+
+### ❌ Anti-Patterns to Avoid
+
+1. **Manual Deployments**
+   - Direct publishing from Desktop
+   - Manual configuration changes
+   - No rollback strategy
+
+2. **Environment Coupling**
+   - Hardcoded connection strings
+   - Environment-specific reports
+   - Manual testing only
+
+3. **Poor Change Management**
+   - No version control
+   - Direct production changes
+   - Missing audit trails
+
+Remember: DevOps for Power BI requires a combination of proper tooling, automated processes, and organizational discipline. Start with basic CI/CD and gradually mature your practices based on organizational needs and compliance requirements.