@luquimbo/bi-superpowers 1.0.0

package/library/snippets/power-query/performance.md

@@ -0,0 +1,506 @@
+# Performance Optimization in Power Query
+
+Best practices and patterns for optimizing Power Query performance, with emphasis on query folding.
+
+---
+
+## Query Folding Basics
+
+### What is Query Folding?
+
+Query folding translates Power Query (M) operations into native database queries (SQL). This pushes processing to the data source, reducing data transfer and improving performance.
+
+### Check if Step Folds
+
+Right-click any step > "View Native Query"
+- If available: Step folds
+- If grayed out: Step breaks folding
+
+### Operations That Typically Fold
+
+| Operation | SQL Equivalent |
+|-----------|---------------|
+| `Table.SelectRows` | WHERE |
+| `Table.SelectColumns` | SELECT |
+| `Table.Sort` | ORDER BY |
+| `Table.Group` | GROUP BY |
+| `Table.NestedJoin` | JOIN |
+| `Table.Distinct` | DISTINCT |
+| `Table.FirstN` | TOP / LIMIT |
+| `Table.Skip` | OFFSET |
+
+### Operations That Break Folding
+
+| Operation | Why |
+|-----------|-----|
+| `Table.AddColumn` with custom function | Custom M logic |
+| `Table.Buffer` | Explicit materialization |
+| `Table.FromList` | Memory operations |
+| `List.Generate` | Iterative logic |
+| `Text.Proper`, `Text.Trim` (sometimes) | Source-specific |
+| Accessing other queries | Cross-query dependency |
+
+---
+
+## Optimize Filter Order
+
+### Push Filters Early
+
+```m
+let
+    // GOOD: Filter first, then transform
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    // Step 1: Filter (folds to WHERE)
+    FilteredRows = Table.SelectRows(Sales, each [Year] = 2024),
+
+    // Step 2: Select columns (folds to SELECT)
+    SelectColumns = Table.SelectColumns(FilteredRows, {"Date", "Amount", "CustomerID"}),
+
+    // Step 3: Transform (may break folding - but on smaller dataset)
+    Transformed = Table.TransformColumns(SelectColumns, {{"Amount", each _ * 1.1}})
+in
+    Transformed
+```
+
+### Anti-Pattern: Late Filtering
+
+```m
+let
+    // BAD: Transform first, filter later
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    // This might break folding
+    Transformed = Table.TransformColumns(Sales, {{"Amount", each _ * 1.1}}),
+
+    // Now filtering happens in memory (slow!)
+    FilteredRows = Table.SelectRows(Transformed, each [Year] = 2024)
+in
+    FilteredRows
+```
+
+---
+
+## Column Selection Optimization
+
+### Select Only Needed Columns
+
+```m
+let
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    // GOOD: Select columns early
+    SelectedColumns = Table.SelectColumns(Sales, {
+        "OrderID", "CustomerID", "Amount", "OrderDate"
+    }),
+
+    // Remaining operations on fewer columns
+    Filtered = Table.SelectRows(SelectedColumns, each [Amount] > 100)
+in
+    Filtered
+```
+
+### Remove Unnecessary Columns
+
+```m
+let
+    Source = YourSource,
+
+    // Remove columns you don't need
+    RemovedColumns = Table.RemoveColumns(Source, {
+        "InternalID", "CreatedBy", "ModifiedBy", "Notes"
+    })
+in
+    RemovedColumns
+```
+
+---
+
+## Avoid Row-by-Row Operations
+
+### Use Native Functions Instead
+
+```m
+let
+    Source = YourSource,
+
+    // BAD: Row-by-row with List operations
+    BadPattern = Table.AddColumn(Source, "Total",
+        each List.Sum({[Price1], [Price2], [Price3]})
+    ),
+
+    // GOOD: Direct column reference
+    GoodPattern = Table.AddColumn(Source, "Total",
+        each [Price1] + [Price2] + [Price3], type number
+    )
+in
+    GoodPattern
+```
+
+### Bulk Transformations
+
+```m
+let
+    Source = YourSource,
+
+    // BAD: Multiple Table.AddColumn calls
+    Step1 = Table.AddColumn(Source, "Year", each Date.Year([Date])),
+    Step2 = Table.AddColumn(Step1, "Month", each Date.Month([Date])),
+    Step3 = Table.AddColumn(Step2, "Day", each Date.Day([Date])),
+
+    // BETTER: Transform existing columns where possible
+    BetterPattern = Table.TransformColumns(Source, {
+        {"DateColumn", Date.Year, Int64.Type}  // Replaces column with year
+    })
+in
+    Step3  // or BetterPattern depending on need
+```
+
+---
+
+## Table.Buffer Usage
+
+### When to Buffer
+
+```m
+let
+    Source = YourSource,
+
+    // Buffer when table is used multiple times
+    BufferedSource = Table.Buffer(Source),
+
+    // Multiple references to buffered table
+    MaxValue = List.Max(BufferedSource[Amount]),
+    MinValue = List.Min(BufferedSource[Amount]),
+
+    Result = Table.AddColumn(BufferedSource, "Normalized",
+        each ([Amount] - MinValue) / (MaxValue - MinValue)
+    )
+in
+    Result
+```
+
+### When NOT to Buffer
+
+```m
+let
+    // DON'T buffer if query folds
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    // This would break folding and load all data
+    // AVOID: Buffered = Table.Buffer(Sales)
+
+    // Let the filter fold instead
+    Filtered = Table.SelectRows(Sales, each [Year] = 2024)
+in
+    Filtered
+```
+
+---
+
+## Partitioning Strategies
+
+### Process Large Data in Chunks
+
+```m
+let
+    // Get distinct years for partitioning
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    Years = List.Distinct(Table.Column(Sales, "Year")),
+
+    // Process each year separately (folded)
+    ProcessYear = (year as number) =>
+        let
+            Filtered = Table.SelectRows(Sales, each [Year] = year),
+            Transformed = Table.TransformColumns(Filtered, {
+                {"Amount", each _ * 1.1}
+            })
+        in
+            Transformed,
+
+    // Combine results
+    AllYears = Table.Combine(List.Transform(Years, ProcessYear))
+in
+    AllYears
+```
+
+### Incremental Refresh Pattern
+
+```m
+let
+    // Parameters for incremental refresh
+    RangeStart = #datetime(2024, 1, 1, 0, 0, 0),
+    RangeEnd = #datetime(2024, 12, 31, 23, 59, 59),
+
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    // Date filter - required for incremental refresh
+    FilteredByDate = Table.SelectRows(Sales, each
+        [OrderDate] >= RangeStart and [OrderDate] < RangeEnd
+    )
+in
+    FilteredByDate
+```
+
+---
+
+## Data Type Optimization
+
+### Set Types Early
+
+```m
+let
+    Source = YourSource,
+
+    // Set types immediately after source
+    TypedTable = Table.TransformColumnTypes(Source, {
+        {"OrderID", Int64.Type},
+        {"CustomerID", Int64.Type},
+        {"Amount", Currency.Type},
+        {"OrderDate", type date},
+        {"CustomerName", type text}
+    })
+in
+    TypedTable
+```
+
+### Avoid Implicit Type Conversion
+
+```m
+let
+    Source = YourSource,
+
+    // BAD: Implicit conversion in comparison
+    BadFilter = Table.SelectRows(Source, each [Year] = "2024"),  // Text comparison
+
+    // GOOD: Explicit types
+    GoodFilter = Table.SelectRows(Source, each [Year] = 2024)  // Number comparison
+in
+    GoodFilter
+```
+
+---
+
+## Reduce Query Dependencies
+
+### Avoid Circular References
+
+```m
+// Separate staging and transformation queries
+
+// Query: Sales_Staging (disable load)
+let
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+    Typed = Table.TransformColumnTypes(Sales, {
+        {"Amount", type number},
+        {"Date", type date}
+    })
+in
+    Typed
+
+// Query: Sales_Fact (enable load)
+let
+    Source = Sales_Staging,  // Reference staging query
+    Filtered = Table.SelectRows(Source, each [Year] = 2024)
+in
+    Filtered
+```
+
+### Use Parameters Instead of Queries
+
+```m
+let
+    // Reference parameter instead of query
+    CurrentYear = CurrentYearParameter,  // Defined as parameter
+
+    Source = Sql.Database("server", "database"),
+    Sales = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    Filtered = Table.SelectRows(Sales, each [Year] = CurrentYear)
+in
+    Filtered
+```
+
+---
+
+## Native SQL When Needed
+
+### Use Value.NativeQuery
+
+```m
+let
+    Source = Sql.Database("server", "database"),
+
+    // Complex query that's hard to fold
+    NativeQuery = Value.NativeQuery(
+        Source,
+        "SELECT
+            CustomerID,
+            SUM(Amount) as TotalAmount,
+            COUNT(*) as OrderCount
+         FROM Sales
+         WHERE Year = 2024
+         GROUP BY CustomerID
+         HAVING SUM(Amount) > 10000"
+    )
+in
+    NativeQuery
+```
+
+### Parameterized Native Query
+
+```m
+let
+    Source = Sql.Database("server", "database"),
+
+    // Using parameter to prevent SQL injection
+    YearValue = 2024,
+
+    Query = Value.NativeQuery(
+        Source,
+        "SELECT * FROM Sales WHERE Year = @year",
+        [year = YearValue]
+    )
+in
+    Query
+```
+
+---
+
+## Memory Management
+
+### Avoid Large Lists in Memory
+
+```m
+let
+    Source = YourSource,
+
+    // BAD: Creates large list in memory
+    BadPattern = List.Buffer(Table.Column(Source, "ID")),
+
+    // GOOD: Process without full materialization
+    GoodPattern = Table.SelectRows(Source, each [Status] = "Active")
+in
+    GoodPattern
+```
+
+### Stream Processing
+
+```m
+let
+    // For very large files, enable streaming
+    Source = Csv.Document(
+        File.Contents("C:\large_file.csv"),
+        [Delimiter=",", QuoteStyle=QuoteStyle.None]
+    ),
+
+    // First filter - reduces data in memory
+    Filtered = Table.SelectRows(Source, each [Status] = "Active"),
+
+    // Then type
+    Typed = Table.TransformColumnTypes(Filtered, {{"Amount", type number}})
+in
+    Typed
+```
+
+---
+
+## Query Diagnostics
+
+### Enable Query Diagnostics
+
+1. Tools > Options > Diagnostics
+2. Enable "Query Diagnostics"
+3. Tools > Start Diagnostics
+4. Refresh query
+5. Tools > Stop Diagnostics
+6. Analyze the diagnostic tables
+
+### Key Metrics to Monitor
+
+| Metric | Meaning |
+|--------|---------|
+| Duration | Total time for step |
+| Row Count | Data volume |
+| Data Source Query | Native query sent |
+| Exclusive Duration | Time without dependencies |
+| Resource | Which data source |
+
+---
+
+## Performance Checklist
+
+### Before Deploying
+
+- [ ] Filter early in the query
+- [ ] Select only needed columns
+- [ ] Verify query folding (right-click > View Native Query)
+- [ ] Set data types explicitly
+- [ ] Remove or disable staging queries from load
+- [ ] Check for unnecessary Table.Buffer calls
+- [ ] Review query dependencies
+- [ ] Test with production data volumes
+- [ ] Enable query diagnostics for complex queries
+
+### Common Issues
+
+| Symptom | Likely Cause | Fix |
+|---------|--------------|-----|
+| Slow refresh | Broken folding | Move filters up, simplify transforms |
+| Memory errors | Large data in memory | Use Table.Buffer sparingly, filter early |
+| Timeout | Complex query | Use native SQL, partition data |
+| Slow step | Row-by-row processing | Use bulk operations |
+
+---
+
+## Folding Quick Reference
+
+### Functions That Typically Fold
+
+```
+Table.SelectRows
+Table.SelectColumns
+Table.RemoveColumns
+Table.Sort
+Table.Group
+Table.NestedJoin
+Table.Distinct
+Table.FirstN
+Table.Skip
+Table.TransformColumnTypes (basic types)
+```
+
+### Functions That Typically Break Folding
+
+```
+Table.AddColumn (custom functions)
+Table.Buffer
+Table.FromList
+Table.TransformColumns (complex transforms)
+List.* functions in row context
+Custom M functions
+References to other queries
+```
+
+### Verify Folding
+
+```m
+// Add this to see the native query
+let
+    Source = Sql.Database("server", "database"),
+    Query = Source{[Schema="dbo", Item="Sales"]}[Data],
+    Filtered = Table.SelectRows(Query, each [Year] = 2024),
+
+    // This will show the SQL if folding works
+    NativeQuery = Value.NativeQuery(Source, "", [EnableFolding=true])
+in
+    Filtered
+```