@luquimbo/bi-superpowers 1.0.0

package/library/snippets/power-query/README.md

@@ -0,0 +1,389 @@
+# Power Query Snippets
+
+Production-ready M code patterns for Power BI and Excel data transformation.
+
+---
+
+## Available Snippets
+
+| File | Description | Level |
+|------|-------------|-------|
+| [data-cleaning.md](./data-cleaning.md) | Null handling, text cleaning, duplicates, type fixes | Beginner |
+| [transformations.md](./transformations.md) | Unpivot, pivot, split, group, date operations | Intermediate |
+| [connections.md](./connections.md) | SQL, Excel, CSV, API, folder connections | Beginner |
+| [parameters.md](./parameters.md) | Dynamic parameters, environment switching | Intermediate |
+| [performance.md](./performance.md) | Query folding, optimization, memory management | Advanced |
+| [error-handling.md](./error-handling.md) | Try-otherwise, validation, debugging patterns | Intermediate |
+| [api-integration.md](./api-integration.md) | REST APIs, pagination, authentication, error handling | Advanced |
+
+---
+
+## Quick Reference
+
+### Filtering
+```m
+// Remove nulls
+Table.SelectRows(Source, each [Column] <> null)
+
+// Multiple conditions
+Table.SelectRows(Source, each [Status] = "Active" and [Amount] > 100)
+
+// Using list
+Table.SelectRows(Source, each List.Contains({"A", "B", "C"}, [Category]))
+```
+
+### Transformations
+```m
+// Unpivot columns
+Table.UnpivotOtherColumns(Source, {"KeepCol1", "KeepCol2"}, "Attribute", "Value")
+
+// Pivot
+Table.Pivot(Source, List.Distinct(Source[Category]), "Category", "Amount", List.Sum)
+
+// Group and aggregate
+Table.Group(Source, {"GroupCol"}, {
+    {"Sum", each List.Sum([Amount]), type number},
+    {"Count", each Table.RowCount(_), Int64.Type}
+})
+```
+
+### Data Types
+```m
+// Set column types
+Table.TransformColumnTypes(Source, {
+    {"Amount", type number},
+    {"Date", type date},
+    {"Name", type text}
+})
+
+// Safe type conversion
+Table.TransformColumns(Source, {{"Column", each try Number.From(_) otherwise null}})
+```
+
+### Adding Columns
+```m
+// Conditional column
+Table.AddColumn(Source, "Category", each if [Value] > 100 then "High" else "Low")
+
+// Calculated column
+Table.AddColumn(Source, "Total", each [Quantity] * [Price], type number)
+
+// From another table (lookup)
+Table.NestedJoin(Source, {"Key"}, LookupTable, {"Key"}, "Lookup", JoinKind.LeftOuter)
+```
+
+### Text Operations
+```m
+// Trim and clean
+Table.TransformColumns(Source, {{"Name", each Text.Trim(Text.Clean(_)), type text}})
+
+// Split column
+Table.SplitColumn(Source, "FullName", Splitter.SplitTextByDelimiter(" "), {"First", "Last"})
+
+// Combine columns
+Table.CombineColumns(Source, {"First", "Last"}, Combiner.CombineTextByDelimiter(" "), "FullName")
+```
+
+---
+
+## Pattern Categories
+
+### By Function Area
+
+| Category | Key Operations | Use Case |
+|----------|---------------|----------|
+| **Filtering** | SelectRows, RemoveRows | Data subsetting |
+| **Columns** | AddColumn, SelectColumns, RenameColumns | Schema changes |
+| **Reshaping** | Pivot, Unpivot, Transpose | Structure changes |
+| **Aggregation** | Group, Distinct | Summarization |
+| **Joining** | NestedJoin, Merge, Combine | Combining tables |
+| **Types** | TransformColumnTypes, TransformColumns | Data quality |
+
+### By Difficulty
+
+| Level | Patterns |
+|-------|----------|
+| **Beginner** | Basic filtering, type conversion, column selection |
+| **Intermediate** | Joins, grouping, conditional logic, parameters |
+| **Advanced** | Query folding optimization, custom functions, error handling |
+
+---
+
+## Query Folding
+
+Query folding pushes operations to the data source (SQL) for better performance.
+
+### Operations That Fold
+
+| Operation | SQL Equivalent |
+|-----------|---------------|
+| Table.SelectRows | WHERE |
+| Table.SelectColumns | SELECT |
+| Table.Sort | ORDER BY |
+| Table.Group | GROUP BY |
+| Table.NestedJoin | JOIN |
+| Table.Distinct | DISTINCT |
+| Table.FirstN | TOP / LIMIT |
+
+### Operations That Break Folding
+
+| Operation | Why |
+|-----------|-----|
+| Table.AddColumn (custom function) | M-specific logic |
+| Table.Buffer | Forces materialization |
+| List.* in row context | Iterative operations |
+| Table.TransformColumns (complex) | Custom transformations |
+
+### Check Folding Status
+
+Right-click any step > "View Native Query"
+- If available: Step folds
+- If grayed out: Folding is broken
+
+### Optimization Pattern
+
+```m
+let
+    // 1. Connect
+    Source = Sql.Database("server", "database"),
+    Table = Source{[Schema="dbo", Item="Sales"]}[Data],
+
+    // 2. Filter EARLY (folds to WHERE)
+    Filtered = Table.SelectRows(Table, each [Year] = 2024),
+
+    // 3. Select columns EARLY (folds to SELECT)
+    Selected = Table.SelectColumns(Filtered, {"Date", "Amount", "CustomerID"}),
+
+    // 4. Transform LAST (may break folding, but on smaller data)
+    Transformed = Table.AddColumn(Selected, "AmountUSD", each [Amount] * 1.1)
+in
+    Transformed
+```
+
+---
+
+## Best Practices
+
+### 1. Filter Early, Transform Late
+
+```m
+// GOOD: Filter first
+let
+    Source = MySource,
+    Filtered = Table.SelectRows(Source, each [Status] = "Active"),  // Early
+    Transformed = Table.AddColumn(Filtered, "NewCol", each [A] + [B])  // Late
+in
+    Transformed
+```
+
+### 2. Select Only Needed Columns
+
+```m
+// Remove unused columns as soon as possible
+SelectedColumns = Table.SelectColumns(Source, {"ID", "Name", "Amount"})
+```
+
+### 3. Use Descriptive Step Names
+
+```m
+// GOOD: Meaningful names
+let
+    Source = Excel.Workbook(File.Contents(FilePath)),
+    Sales_Sheet = Source{[Item="Sales", Kind="Sheet"]}[Data],
+    Promoted_Headers = Table.PromoteHeaders(Sales_Sheet),
+    Removed_Null_Rows = Table.SelectRows(Promoted_Headers, each [ID] <> null),
+    Set_Column_Types = Table.TransformColumnTypes(Removed_Null_Rows, {
+        {"ID", Int64.Type},
+        {"Amount", Currency.Type}
+    })
+in
+    Set_Column_Types
+```
+
+### 4. Use Parameters for Paths
+
+```m
+// Define parameter
+FilePath = "C:\Data\Sales.xlsx"  // Easy to change
+
+// Use in query
+Source = Excel.Workbook(File.Contents(FilePath))
+```
+
+### 5. Handle Errors Gracefully
+
+```m
+// Safe conversion
+Table.TransformColumns(Source, {
+    {"Amount", each try Number.From(_) otherwise null, type number}
+})
+```
+
+### 6. Document Complex Logic
+
+```m
+// Add comments for clarity
+let
+    Source = MySource,
+
+    // Remove weekend transactions per business rule BR-123
+    FilteredRows = Table.SelectRows(Source, each
+        Date.DayOfWeek([Date]) <> Day.Saturday and
+        Date.DayOfWeek([Date]) <> Day.Sunday
+    )
+in
+    FilteredRows
+```
+
+---
+
+## Performance Tips
+
+| Tip | Benefit |
+|-----|---------|
+| Filter early | Less data to process |
+| Select columns early | Less memory |
+| Use Table.Buffer sparingly | Avoid unnecessary materialization |
+| Avoid row-by-row operations | Better engine optimization |
+| Check query folding | Push work to source |
+| Use parameters | Easier maintenance |
+| Disable load for staging queries | Faster refresh |
+
+### Memory Management
+
+```m
+// DON'T buffer everything
+Bad = Table.Buffer(LargeTable)  // Loads all to memory
+
+// DO buffer only when needed for multiple references
+let
+    BufferedTable = Table.Buffer(SmallTable),
+    Max = List.Max(BufferedTable[Amount]),
+    Min = List.Min(BufferedTable[Amount])
+in
+    // Both use buffered table
+    Max - Min
+```
+
+---
+
+## Common Patterns
+
+### COALESCE (First Non-Null)
+
+```m
+Table.AddColumn(Source, "Email",
+    each if [PrimaryEmail] <> null then [PrimaryEmail]
+         else if [SecondaryEmail] <> null then [SecondaryEmail]
+         else "unknown@example.com"
+)
+```
+
+### Lookup from Another Table
+
+```m
+let
+    Merged = Table.NestedJoin(
+        Source, {"ProductID"},
+        ProductTable, {"ProductID"},
+        "Product", JoinKind.LeftOuter
+    ),
+    Expanded = Table.ExpandTableColumn(Merged, "Product", {"ProductName", "Category"})
+in
+    Expanded
+```
+
+### Running Total
+
+```m
+let
+    Sorted = Table.Sort(Source, {{"Date", Order.Ascending}}),
+    Indexed = Table.AddIndexColumn(Sorted, "Index", 1, 1),
+    Running = Table.AddColumn(Indexed, "RunningTotal",
+        each List.Sum(List.FirstN(Sorted[Amount], [Index]))
+    )
+in
+    Running
+```
+
+### Date Table Generation
+
+```m
+let
+    StartDate = #date(2020, 1, 1),
+    EndDate = #date(2025, 12, 31),
+    DateList = List.Dates(StartDate, Duration.Days(EndDate - StartDate) + 1, #duration(1, 0, 0, 0)),
+    DateTable = Table.FromList(DateList, Splitter.SplitByNothing(), {"Date"}),
+    Typed = Table.TransformColumnTypes(DateTable, {{"Date", type date}})
+in
+    Typed
+```
+
+---
+
+## Debugging
+
+### Check Row Count at Each Step
+
+```m
+// Temporarily add to see counts
+let
+    Source = MySource,
+    _ = Text.From(Table.RowCount(Source)),  // Check in formula bar
+    Filtered = Table.SelectRows(Source, each [Status] = "Active"),
+    __ = Text.From(Table.RowCount(Filtered))  // Check after filter
+in
+    Filtered
+```
+
+### Enable Query Diagnostics
+
+1. Tools > Options > Diagnostics
+2. Enable "Query Diagnostics"
+3. Tools > Start Diagnostics
+4. Refresh query
+5. Tools > Stop Diagnostics
+6. Analyze diagnostic tables
+
+---
+
+## Contributing
+
+Add new snippets as `.md` files with:
+
+1. **Clear title** - Descriptive name
+2. **Pattern category** - Cleaning, Transform, etc.
+3. **Working M code** - Tested examples
+4. **Parameter explanations** - What to customize
+5. **Usage notes** - When to use
+6. **Performance notes** - Folding impact if relevant
+
+### File Template
+
+```markdown
+# Pattern Name
+
+Description of what this pattern does.
+
+---
+
+## Basic Pattern
+
+\`\`\`m
+let
+    Source = YourSource,
+    Result = Table.TransformColumns(Source, ...)
+in
+    Result
+\`\`\`
+
+## Variations
+
+### Variation 1
+...
+
+## Usage Notes
+
+- Note 1
+- Note 2
+```