workspace-architect 1.3.0 → 1.5.0

package/assets/instructions/power-bi-devops-alm-best-practices.instructions.md

@@ -1,623 +0,0 @@
----
-description: 'Comprehensive guide for Power BI DevOps, Application Lifecycle Management (ALM), CI/CD pipelines, deployment automation, and version control best practices.'
-applyTo: '**/*.{yml,yaml,ps1,json,pbix,pbir}'
----
-
-# Power BI DevOps and Application Lifecycle Management Best Practices
-
-## Overview
-This document provides comprehensive instructions for implementing DevOps practices, CI/CD pipelines, and Application Lifecycle Management (ALM) for Power BI solutions, based on Microsoft's recommended patterns and best practices.
-
-## Power BI Project Structure and Version Control
-
-### 1. PBIP (Power BI Project) Structure
-```markdown
-// Power BI project file organization
-├── Model/
-│   ├── model.tmdl
-│   ├── tables/
-│   │   ├── FactSales.tmdl
-│   │   └── DimProduct.tmdl
-│   ├── relationships/
-│   │   └── relationships.tmdl
-│   └── measures/
-│       └── measures.tmdl
-├── Report/
-│   ├── report.json
-│   ├── pages/
-│   │   ├── ReportSection1/
-│   │   │   ├── page.json
-│   │   │   └── visuals/
-│   │   └── pages.json
-│   └── bookmarks/
-└── .git/
-```
-
-### 2. Git Integration Best Practices
-```powershell
-# Initialize Power BI project with Git
-git init
-git add .
-git commit -m "Initial Power BI project structure"
-
-# Create feature branch for development
-git checkout -b feature/new-dashboard
-git add Model/tables/NewTable.tmdl
-git commit -m "Add new dimension table"
-
-# Merge and deploy workflow
-git checkout main
-git merge feature/new-dashboard
-git tag -a v1.2.0 -m "Release version 1.2.0"
-```
-
-## Deployment Pipelines and Automation
-
-### 1. Power BI Deployment Pipelines API
-```powershell
-# Automated deployment using Power BI REST API
-$url = "pipelines/{0}/Deploy" -f "Insert your pipeline ID here"
-$body = @{ 
-    sourceStageOrder = 0 # Development (0), Test (1)
-    datasets = @(
-        @{sourceId = "Insert your dataset ID here" }
-    )      
-    reports = @(
-        @{sourceId = "Insert your report ID here" }
-    )            
-    dashboards = @(
-        @{sourceId = "Insert your dashboard ID here" }
-    )
-
-    options = @{
-        # Allows creating new item if needed on the Test stage workspace
-        allowCreateArtifact = $TRUE
-
-        # Allows overwriting existing item if needed on the Test stage workspace
-        allowOverwriteArtifact = $TRUE
-    }
-} | ConvertTo-Json
-
-$deployResult = Invoke-PowerBIRestMethod -Url $url -Method Post -Body $body | ConvertFrom-Json
-
-# Poll deployment status
-$url = "pipelines/{0}/Operations/{1}" -f "Insert your pipeline ID here",$deployResult.id
-$operation = Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json    
-while($operation.Status -eq "NotStarted" -or $operation.Status -eq "Executing")
-{
-    # Sleep for 5 seconds
-    Start-Sleep -s 5
-    $operation = Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json
-}
-```
-
-### 2. Azure DevOps Integration
-```yaml
-# Azure DevOps pipeline for Power BI deployment
-trigger:
-- main
-
-pool:
-  vmImage: windows-latest
-
-steps:
-- task: CopyFiles@2
-  inputs:
-    Contents: '**'
-    TargetFolder: '$(Build.ArtifactStagingDirectory)'
-    CleanTargetFolder: true
-    ignoreMakeDirErrors: true
-  displayName: 'Copy files from Repo'
-
-- task: PowerPlatformToolInstaller@2
-  inputs:
-    DefaultVersion: true
-
-- task: PowerPlatformExportData@2
-  inputs:
-    authenticationType: 'PowerPlatformSPN'
-    PowerPlatformSPN: 'PowerBIServiceConnection'
-    Environment: '$(BuildTools.EnvironmentUrl)'
-    SchemaFile: '$(Build.ArtifactStagingDirectory)\source\schema.xml'
-    DataFile: 'data.zip'
-  displayName: 'Export Power BI metadata'
-
-- task: PowerShell@2  
-  inputs:
-    targetType: 'inline'
-    script: |
-      # Deploy Power BI project using FabricPS-PBIP
-      $workspaceName = "$(WorkspaceName)"
-      $pbipSemanticModelPath = "$(Build.ArtifactStagingDirectory)\$(ProjectName).SemanticModel"
-      $pbipReportPath = "$(Build.ArtifactStagingDirectory)\$(ProjectName).Report"
-      
-      # Download and install FabricPS-PBIP module
-      New-Item -ItemType Directory -Path ".\modules" -ErrorAction SilentlyContinue | Out-Null
-      @("https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psm1",
-        "https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psd1") |% {
-          Invoke-WebRequest -Uri $_ -OutFile ".\modules\$(Split-Path $_ -Leaf)"
-      }
-      
-      Import-Module ".\modules\FabricPS-PBIP" -Force
-      
-      # Authenticate and deploy
-      Set-FabricAuthToken -reset
-      $workspaceId = New-FabricWorkspace -name $workspaceName -skipErrorIfExists
-      $semanticModelImport = Import-FabricItem -workspaceId $workspaceId -path $pbipSemanticModelPath
-      $reportImport = Import-FabricItem -workspaceId $workspaceId -path $pbipReportPath -itemProperties @{"semanticModelId" = $semanticModelImport.Id}
-  displayName: 'Deploy to Power BI Service'
-```
-
-### 3. Fabric REST API Deployment
-```powershell
-# Complete PowerShell deployment script
-# Parameters 
-$workspaceName = "[Workspace Name]"
-$pbipSemanticModelPath = "[PBIP Path]\[Item Name].SemanticModel"
-$pbipReportPath = "[PBIP Path]\[Item Name].Report"
-$currentPath = (Split-Path $MyInvocation.MyCommand.Definition -Parent)
-Set-Location $currentPath
-
-# Download modules and install
-New-Item -ItemType Directory -Path ".\modules" -ErrorAction SilentlyContinue | Out-Null
-@("https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psm1",
-  "https://raw.githubusercontent.com/microsoft/Analysis-Services/master/pbidevmode/fabricps-pbip/FabricPS-PBIP.psd1") |% {
-    Invoke-WebRequest -Uri $_ -OutFile ".\modules\$(Split-Path $_ -Leaf)"
-}
-
-if(-not (Get-Module Az.Accounts -ListAvailable)) { 
-    Install-Module Az.Accounts -Scope CurrentUser -Force
-}
-Import-Module ".\modules\FabricPS-PBIP" -Force
-
-# Authenticate
-Set-FabricAuthToken -reset
-
-# Ensure workspace exists
-$workspaceId = New-FabricWorkspace -name $workspaceName -skipErrorIfExists
-
-# Import the semantic model and save the item id
-$semanticModelImport = Import-FabricItem -workspaceId $workspaceId -path $pbipSemanticModelPath
-
-# Import the report and ensure its bound to the previous imported semantic model
-$reportImport = Import-FabricItem -workspaceId $workspaceId -path $pbipReportPath -itemProperties @{"semanticModelId" = $semanticModelImport.Id}
-```
-
-## Environment Management
-
-### 1. Multi-Environment Strategy
-```json
-{
-  "environments": {
-    "development": {
-      "workspaceId": "dev-workspace-id",
-      "dataSourceUrl": "dev-database.database.windows.net",
-      "refreshSchedule": "manual",
-      "sensitivityLabel": "Internal"
-    },
-    "test": {
-      "workspaceId": "test-workspace-id",
-      "dataSourceUrl": "test-database.database.windows.net",
-      "refreshSchedule": "daily",
-      "sensitivityLabel": "Internal"
-    },
-    "production": {
-      "workspaceId": "prod-workspace-id", 
-      "dataSourceUrl": "prod-database.database.windows.net",
-      "refreshSchedule": "hourly",
-      "sensitivityLabel": "Confidential"
-    }
-  }
-}
-```
-
-### 2. Parameter-Driven Deployment
-```powershell
-# Environment-specific parameter management
-param(
-    [Parameter(Mandatory=$true)]
-    [ValidateSet("dev", "test", "prod")]
-    [string]$Environment,
-    
-    [Parameter(Mandatory=$true)]
-    [string]$WorkspaceName,
-    
-    [Parameter(Mandatory=$false)]
-    [string]$DataSourceServer
-)
-
-# Load environment-specific configuration
-$configPath = ".\config\$Environment.json"
-$config = Get-Content $configPath | ConvertFrom-Json
-
-# Update connection strings based on environment
-$connectionString = "Data Source=$($config.dataSourceUrl);Initial Catalog=$($config.database);Integrated Security=SSPI;"
-
-# Deploy with environment-specific settings
-Write-Host "Deploying to $Environment environment..."
-Write-Host "Workspace: $($config.workspaceId)"
-Write-Host "Data Source: $($config.dataSourceUrl)"
-```
-
-## Automated Testing Framework
-
-### 1. Data Quality Tests
-```powershell
-# Automated data quality validation
-function Test-PowerBIDataQuality {
-    param(
-        [string]$WorkspaceId,
-        [string]$DatasetId
-    )
-    
-    # Test 1: Row count validation
-    $rowCountQuery = @"
-        EVALUATE
-        ADDCOLUMNS(
-            SUMMARIZE(Sales, Sales[Year]),
-            "RowCount", COUNTROWS(Sales),
-            "ExpectedMin", 1000,
-            "Test", IF(COUNTROWS(Sales) >= 1000, "PASS", "FAIL")
-        )
-"@
-    
-    # Test 2: Data freshness validation  
-    $freshnessQuery = @"
-        EVALUATE
-        ADDCOLUMNS(
-            ROW("LastRefresh", MAX(Sales[Date])),
-            "DaysOld", DATEDIFF(MAX(Sales[Date]), TODAY(), DAY),
-            "Test", IF(DATEDIFF(MAX(Sales[Date]), TODAY(), DAY) <= 1, "PASS", "FAIL")
-        )
-"@
-    
-    # Execute tests
-    $rowCountResult = Invoke-PowerBIRestMethod -Url "groups/$WorkspaceId/datasets/$DatasetId/executeQueries" -Method Post -Body (@{queries=@(@{query=$rowCountQuery})} | ConvertTo-Json)
-    $freshnessResult = Invoke-PowerBIRestMethod -Url "groups/$WorkspaceId/datasets/$DatasetId/executeQueries" -Method Post -Body (@{queries=@(@{query=$freshnessQuery})} | ConvertTo-Json)
-    
-    return @{
-        RowCountTest = $rowCountResult
-        FreshnessTest = $freshnessResult
-    }
-}
-```
-
-### 2. Performance Regression Tests
-```powershell
-# Performance benchmark testing
-function Test-PowerBIPerformance {
-    param(
-        [string]$WorkspaceId,
-        [string]$ReportId
-    )
-    
-    $performanceTests = @(
-        @{
-            Name = "Dashboard Load Time"
-            Query = "EVALUATE TOPN(1000, Sales)"
-            MaxDurationMs = 5000
-        },
-        @{
-            Name = "Complex Calculation"
-            Query = "EVALUATE ADDCOLUMNS(Sales, 'ComplexCalc', [Sales] * [Profit Margin %])"
-            MaxDurationMs = 10000
-        }
-    )
-    
-    $results = @()
-    foreach ($test in $performanceTests) {
-        $startTime = Get-Date
-        $result = Invoke-PowerBIRestMethod -Url "groups/$WorkspaceId/datasets/$DatasetId/executeQueries" -Method Post -Body (@{queries=@(@{query=$test.Query})} | ConvertTo-Json)
-        $endTime = Get-Date
-        $duration = ($endTime - $startTime).TotalMilliseconds
-        
-        $results += @{
-            TestName = $test.Name
-            Duration = $duration
-            Passed = $duration -le $test.MaxDurationMs
-            Threshold = $test.MaxDurationMs
-        }
-    }
-    
-    return $results
-}
-```
-
-## Configuration Management
-
-### 1. Infrastructure as Code
-```json
-{
-  "workspace": {
-    "name": "Production Analytics",
-    "description": "Production Power BI workspace for sales analytics",
-    "capacityId": "A1-capacity-id",
-    "users": [
-      {
-        "emailAddress": "admin@contoso.com",
-        "accessRight": "Admin"
-      },
-      {
-        "emailAddress": "powerbi-service-principal@contoso.com", 
-        "accessRight": "Member",
-        "principalType": "App"
-      }
-    ],
-    "settings": {
-      "datasetDefaultStorageFormat": "Large",
-      "blockResourceKeyAuthentication": true
-    }
-  },
-  "datasets": [
-    {
-      "name": "Sales Analytics",
-      "refreshSchedule": {
-        "enabled": true,
-        "times": ["06:00", "12:00", "18:00"],
-        "days": ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"],
-        "timeZone": "UTC"
-      },
-      "datasourceCredentials": {
-        "credentialType": "OAuth2",
-        "encryptedConnection": "Encrypted"
-      }
-    }
-  ]
-}
-```
-
-### 2. Secret Management
-```powershell
-# Azure Key Vault integration for secrets
-function Get-PowerBICredentials {
-    param(
-        [string]$KeyVaultName,
-        [string]$Environment
-    )
-    
-    # Retrieve secrets from Key Vault
-    $servicePrincipalId = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "PowerBI-ServicePrincipal-Id-$Environment" -AsPlainText
-    $servicePrincipalSecret = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "PowerBI-ServicePrincipal-Secret-$Environment" -AsPlainText
-    $tenantId = Get-AzKeyVaultSecret -VaultName $KeyVaultName -Name "PowerBI-TenantId-$Environment" -AsPlainText
-    
-    return @{
-        ServicePrincipalId = $servicePrincipalId
-        ServicePrincipalSecret = $servicePrincipalSecret
-        TenantId = $tenantId
-    }
-}
-
-# Authenticate using retrieved credentials
-$credentials = Get-PowerBICredentials -KeyVaultName "PowerBI-KeyVault" -Environment "Production"
-$securePassword = ConvertTo-SecureString $credentials.ServicePrincipalSecret -AsPlainText -Force
-$credential = New-Object System.Management.Automation.PSCredential($credentials.ServicePrincipalId, $securePassword)
-Connect-PowerBIServiceAccount -ServicePrincipal -Credential $credential -TenantId $credentials.TenantId
-```
-
-## Release Management
-
-### 1. Release Pipeline
-```yaml
-# Multi-stage release pipeline
-stages:
-- stage: Build
-  displayName: 'Build Stage'
-  jobs:
-  - job: Build
-    steps:
-    - task: PowerShell@2
-      displayName: 'Validate Power BI Project'
-      inputs:
-        targetType: 'inline'
-        script: |
-          # Validate PBIP structure
-          if (-not (Test-Path "Model\model.tmdl")) {
-            throw "Missing model.tmdl file"
-          }
-          
-          # Validate required files
-          $requiredFiles = @("Report\report.json", "Model\tables")
-          foreach ($file in $requiredFiles) {
-            if (-not (Test-Path $file)) {
-              throw "Missing required file: $file"
-            }
-          }
-          
-          Write-Host "✅ Project validation passed"
-
-- stage: DeployTest
-  displayName: 'Deploy to Test'
-  dependsOn: Build
-  condition: succeeded()
-  jobs:
-  - deployment: DeployTest
-    environment: 'PowerBI-Test'
-    strategy:
-      runOnce:
-        deploy:
-          steps:
-          - template: deploy-powerbi.yml
-            parameters:
-              environment: 'test'
-              workspaceName: '$(TestWorkspaceName)'
-
-- stage: DeployProd
-  displayName: 'Deploy to Production'
-  dependsOn: DeployTest
-  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
-  jobs:
-  - deployment: DeployProd
-    environment: 'PowerBI-Production'
-    strategy:
-      runOnce:
-        deploy:
-          steps:
-          - template: deploy-powerbi.yml
-            parameters:
-              environment: 'prod'
-              workspaceName: '$(ProdWorkspaceName)'
-```
-
-### 2. Rollback Strategy
-```powershell
-# Automated rollback mechanism
-function Invoke-PowerBIRollback {
-    param(
-        [string]$WorkspaceId,
-        [string]$BackupVersion,
-        [string]$BackupLocation
-    )
-    
-    Write-Host "Initiating rollback to version: $BackupVersion"
-    
-    # Step 1: Export current state as emergency backup
-    $emergencyBackup = "emergency-backup-$(Get-Date -Format 'yyyyMMdd-HHmmss')"
-    Export-PowerBIReport -WorkspaceId $WorkspaceId -BackupName $emergencyBackup
-    
-    # Step 2: Restore from backup
-    $backupPath = Join-Path $BackupLocation "$BackupVersion.pbix"
-    if (Test-Path $backupPath) {
-        Import-PowerBIReport -WorkspaceId $WorkspaceId -FilePath $backupPath -ConflictAction "Overwrite"
-        Write-Host "✅ Rollback completed successfully"
-    } else {
-        throw "Backup file not found: $backupPath"
-    }
-    
-    # Step 3: Validate rollback
-    Test-PowerBIDataQuality -WorkspaceId $WorkspaceId
-}
-```
-
-## Monitoring and Alerting
-
-### 1. Deployment Health Checks
-```powershell
-# Post-deployment validation
-function Test-DeploymentHealth {
-    param(
-        [string]$WorkspaceId,
-        [array]$ExpectedReports,
-        [array]$ExpectedDatasets
-    )
-    
-    $healthCheck = @{
-        Status = "Healthy"
-        Issues = @()
-        Timestamp = Get-Date
-    }
-    
-    # Check reports
-    $reports = Get-PowerBIReport -WorkspaceId $WorkspaceId
-    foreach ($expectedReport in $ExpectedReports) {
-        if (-not ($reports.Name -contains $expectedReport)) {
-            $healthCheck.Issues += "Missing report: $expectedReport"
-            $healthCheck.Status = "Unhealthy"
-        }
-    }
-    
-    # Check datasets
-    $datasets = Get-PowerBIDataset -WorkspaceId $WorkspaceId
-    foreach ($expectedDataset in $ExpectedDatasets) {
-        $dataset = $datasets | Where-Object { $_.Name -eq $expectedDataset }
-        if (-not $dataset) {
-            $healthCheck.Issues += "Missing dataset: $expectedDataset"
-            $healthCheck.Status = "Unhealthy"
-        } elseif ($dataset.RefreshState -eq "Failed") {
-            $healthCheck.Issues += "Dataset refresh failed: $expectedDataset"
-            $healthCheck.Status = "Degraded"
-        }
-    }
-    
-    return $healthCheck
-}
-```
-
-### 2. Automated Alerting
-```powershell
-# Teams notification for deployment status
-function Send-DeploymentNotification {
-    param(
-        [string]$TeamsWebhookUrl,
-        [object]$DeploymentResult,
-        [string]$Environment
-    )
-    
-    $color = switch ($DeploymentResult.Status) {
-        "Success" { "28A745" }
-        "Warning" { "FFC107" }
-        "Failed" { "DC3545" }
-    }
-    
-    $teamsMessage = @{
-        "@type" = "MessageCard"
-        "@context" = "https://schema.org/extensions"
-        "summary" = "Power BI Deployment $($DeploymentResult.Status)"
-        "themeColor" = $color
-        "sections" = @(
-            @{
-                "activityTitle" = "Power BI Deployment to $Environment"
-                "activitySubtitle" = "$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"
-                "facts" = @(
-                    @{
-                        "name" = "Status"
-                        "value" = $DeploymentResult.Status
-                    },
-                    @{
-                        "name" = "Duration"
-                        "value" = "$($DeploymentResult.Duration) minutes"
-                    },
-                    @{
-                        "name" = "Reports Deployed"
-                        "value" = $DeploymentResult.ReportsCount
-                    }
-                )
-            }
-        )
-    }
-    
-    Invoke-RestMethod -Uri $TeamsWebhookUrl -Method Post -Body ($teamsMessage | ConvertTo-Json -Depth 10) -ContentType 'application/json'
-}
-```
-
-## Best Practices Summary
-
-### ✅ DevOps Best Practices
-
-1. **Version Control Everything**
-   - Use PBIP format for source control
-   - Include model, reports, and configuration
-   - Implement branching strategies (GitFlow)
-
-2. **Automated Testing**
-   - Data quality validation
-   - Performance regression tests
-   - Security compliance checks
-
-3. **Environment Isolation**
-   - Separate dev/test/prod environments
-   - Environment-specific configurations
-   - Automated promotion pipelines
-
-4. **Security Integration**
-   - Service principal authentication
-   - Secret management with Key Vault
-   - Role-based access controls
-
-### ❌ Anti-Patterns to Avoid
-
-1. **Manual Deployments**
-   - Direct publishing from Desktop
-   - Manual configuration changes
-   - No rollback strategy
-
-2. **Environment Coupling**
-   - Hardcoded connection strings
-   - Environment-specific reports
-   - Manual testing only
-
-3. **Poor Change Management**
-   - No version control
-   - Direct production changes
-   - Missing audit trails
-
-Remember: DevOps for Power BI requires a combination of proper tooling, automated processes, and organizational discipline. Start with basic CI/CD and gradually mature your practices based on organizational needs and compliance requirements.