@luquimbo/bi-superpowers 3.1.0 → 3.2.0

package/library/snippets/power-query/README.md

@@ -1,389 +0,0 @@
-# Power Query Snippets
-
-Production-ready M code patterns for Power BI and Excel data transformation.
-
----
-
-## Available Snippets
-
-| File | Description | Level |
-|------|-------------|-------|
-| [data-cleaning.md](./data-cleaning.md) | Null handling, text cleaning, duplicates, type fixes | Beginner |
-| [transformations.md](./transformations.md) | Unpivot, pivot, split, group, date operations | Intermediate |
-| [connections.md](./connections.md) | SQL, Excel, CSV, API, folder connections | Beginner |
-| [parameters.md](./parameters.md) | Dynamic parameters, environment switching | Intermediate |
-| [performance.md](./performance.md) | Query folding, optimization, memory management | Advanced |
-| [error-handling.md](./error-handling.md) | Try-otherwise, validation, debugging patterns | Intermediate |
-| [api-integration.md](./api-integration.md) | REST APIs, pagination, authentication, error handling | Advanced |
-
----
-
-## Quick Reference
-
-### Filtering
-```m
-// Remove nulls
-Table.SelectRows(Source, each [Column] <> null)
-
-// Multiple conditions
-Table.SelectRows(Source, each [Status] = "Active" and [Amount] > 100)
-
-// Using list
-Table.SelectRows(Source, each List.Contains({"A", "B", "C"}, [Category]))
-```
-
-### Transformations
-```m
-// Unpivot columns
-Table.UnpivotOtherColumns(Source, {"KeepCol1", "KeepCol2"}, "Attribute", "Value")
-
-// Pivot
-Table.Pivot(Source, List.Distinct(Source[Category]), "Category", "Amount", List.Sum)
-
-// Group and aggregate
-Table.Group(Source, {"GroupCol"}, {
-    {"Sum", each List.Sum([Amount]), type number},
-    {"Count", each Table.RowCount(_), Int64.Type}
-})
-```
-
-### Data Types
-```m
-// Set column types
-Table.TransformColumnTypes(Source, {
-    {"Amount", type number},
-    {"Date", type date},
-    {"Name", type text}
-})
-
-// Safe type conversion
-Table.TransformColumns(Source, {{"Column", each try Number.From(_) otherwise null}})
-```
-
-### Adding Columns
-```m
-// Conditional column
-Table.AddColumn(Source, "Category", each if [Value] > 100 then "High" else "Low")
-
-// Calculated column
-Table.AddColumn(Source, "Total", each [Quantity] * [Price], type number)
-
-// From another table (lookup)
-Table.NestedJoin(Source, {"Key"}, LookupTable, {"Key"}, "Lookup", JoinKind.LeftOuter)
-```
-
-### Text Operations
-```m
-// Trim and clean
-Table.TransformColumns(Source, {{"Name", each Text.Trim(Text.Clean(_)), type text}})
-
-// Split column
-Table.SplitColumn(Source, "FullName", Splitter.SplitTextByDelimiter(" "), {"First", "Last"})
-
-// Combine columns
-Table.CombineColumns(Source, {"First", "Last"}, Combiner.CombineTextByDelimiter(" "), "FullName")
-```
-
----
-
-## Pattern Categories
-
-### By Function Area
-
-| Category | Key Operations | Use Case |
-|----------|---------------|----------|
-| **Filtering** | SelectRows, RemoveRows | Data subsetting |
-| **Columns** | AddColumn, SelectColumns, RenameColumns | Schema changes |
-| **Reshaping** | Pivot, Unpivot, Transpose | Structure changes |
-| **Aggregation** | Group, Distinct | Summarization |
-| **Joining** | NestedJoin, Merge, Combine | Combining tables |
-| **Types** | TransformColumnTypes, TransformColumns | Data quality |
-
-### By Difficulty
-
-| Level | Patterns |
-|-------|----------|
-| **Beginner** | Basic filtering, type conversion, column selection |
-| **Intermediate** | Joins, grouping, conditional logic, parameters |
-| **Advanced** | Query folding optimization, custom functions, error handling |
-
----
-
-## Query Folding
-
-Query folding pushes operations to the data source (SQL) for better performance.
-
-### Operations That Fold
-
-| Operation | SQL Equivalent |
-|-----------|---------------|
-| Table.SelectRows | WHERE |
-| Table.SelectColumns | SELECT |
-| Table.Sort | ORDER BY |
-| Table.Group | GROUP BY |
-| Table.NestedJoin | JOIN |
-| Table.Distinct | DISTINCT |
-| Table.FirstN | TOP / LIMIT |
-
-### Operations That Break Folding
-
-| Operation | Why |
-|-----------|-----|
-| Table.AddColumn (custom function) | M-specific logic |
-| Table.Buffer | Forces materialization |
-| List.* in row context | Iterative operations |
-| Table.TransformColumns (complex) | Custom transformations |
-
-### Check Folding Status
-
-Right-click any step > "View Native Query"
-- If available: Step folds
-- If grayed out: Folding is broken
-
-### Optimization Pattern
-
-```m
-let
-    // 1. Connect
-    Source = Sql.Database("server", "database"),
-    Table = Source{[Schema="dbo", Item="Sales"]}[Data],
-
-    // 2. Filter EARLY (folds to WHERE)
-    Filtered = Table.SelectRows(Table, each [Year] = 2024),
-
-    // 3. Select columns EARLY (folds to SELECT)
-    Selected = Table.SelectColumns(Filtered, {"Date", "Amount", "CustomerID"}),
-
-    // 4. Transform LAST (may break folding, but on smaller data)
-    Transformed = Table.AddColumn(Selected, "AmountUSD", each [Amount] * 1.1)
-in
-    Transformed
-```
-
----
-
-## Best Practices
-
-### 1. Filter Early, Transform Late
-
-```m
-// GOOD: Filter first
-let
-    Source = MySource,
-    Filtered = Table.SelectRows(Source, each [Status] = "Active"),  // Early
-    Transformed = Table.AddColumn(Filtered, "NewCol", each [A] + [B])  // Late
-in
-    Transformed
-```
-
-### 2. Select Only Needed Columns
-
-```m
-// Remove unused columns as soon as possible
-SelectedColumns = Table.SelectColumns(Source, {"ID", "Name", "Amount"})
-```
-
-### 3. Use Descriptive Step Names
-
-```m
-// GOOD: Meaningful names
-let
-    Source = Excel.Workbook(File.Contents(FilePath)),
-    Sales_Sheet = Source{[Item="Sales", Kind="Sheet"]}[Data],
-    Promoted_Headers = Table.PromoteHeaders(Sales_Sheet),
-    Removed_Null_Rows = Table.SelectRows(Promoted_Headers, each [ID] <> null),
-    Set_Column_Types = Table.TransformColumnTypes(Removed_Null_Rows, {
-        {"ID", Int64.Type},
-        {"Amount", Currency.Type}
-    })
-in
-    Set_Column_Types
-```
-
-### 4. Use Parameters for Paths
-
-```m
-// Define parameter
-FilePath = "C:\Data\Sales.xlsx"  // Easy to change
-
-// Use in query
-Source = Excel.Workbook(File.Contents(FilePath))
-```
-
-### 5. Handle Errors Gracefully
-
-```m
-// Safe conversion
-Table.TransformColumns(Source, {
-    {"Amount", each try Number.From(_) otherwise null, type number}
-})
-```
-
-### 6. Document Complex Logic
-
-```m
-// Add comments for clarity
-let
-    Source = MySource,
-
-    // Remove weekend transactions per business rule BR-123
-    FilteredRows = Table.SelectRows(Source, each
-        Date.DayOfWeek([Date]) <> Day.Saturday and
-        Date.DayOfWeek([Date]) <> Day.Sunday
-    )
-in
-    FilteredRows
-```
-
----
-
-## Performance Tips
-
-| Tip | Benefit |
-|-----|---------|
-| Filter early | Less data to process |
-| Select columns early | Less memory |
-| Use Table.Buffer sparingly | Avoid unnecessary materialization |
-| Avoid row-by-row operations | Better engine optimization |
-| Check query folding | Push work to source |
-| Use parameters | Easier maintenance |
-| Disable load for staging queries | Faster refresh |
-
-### Memory Management
-
-```m
-// DON'T buffer everything
-Bad = Table.Buffer(LargeTable)  // Loads all to memory
-
-// DO buffer only when needed for multiple references
-let
-    BufferedTable = Table.Buffer(SmallTable),
-    Max = List.Max(BufferedTable[Amount]),
-    Min = List.Min(BufferedTable[Amount])
-in
-    // Both use buffered table
-    Max - Min
-```
-
----
-
-## Common Patterns
-
-### COALESCE (First Non-Null)
-
-```m
-Table.AddColumn(Source, "Email",
-    each if [PrimaryEmail] <> null then [PrimaryEmail]
-         else if [SecondaryEmail] <> null then [SecondaryEmail]
-         else "unknown@example.com"
-)
-```
-
-### Lookup from Another Table
-
-```m
-let
-    Merged = Table.NestedJoin(
-        Source, {"ProductID"},
-        ProductTable, {"ProductID"},
-        "Product", JoinKind.LeftOuter
-    ),
-    Expanded = Table.ExpandTableColumn(Merged, "Product", {"ProductName", "Category"})
-in
-    Expanded
-```
-
-### Running Total
-
-```m
-let
-    Sorted = Table.Sort(Source, {{"Date", Order.Ascending}}),
-    Indexed = Table.AddIndexColumn(Sorted, "Index", 1, 1),
-    Running = Table.AddColumn(Indexed, "RunningTotal",
-        each List.Sum(List.FirstN(Sorted[Amount], [Index]))
-    )
-in
-    Running
-```
-
-### Date Table Generation
-
-```m
-let
-    StartDate = #date(2020, 1, 1),
-    EndDate = #date(2025, 12, 31),
-    DateList = List.Dates(StartDate, Duration.Days(EndDate - StartDate) + 1, #duration(1, 0, 0, 0)),
-    DateTable = Table.FromList(DateList, Splitter.SplitByNothing(), {"Date"}),
-    Typed = Table.TransformColumnTypes(DateTable, {{"Date", type date}})
-in
-    Typed
-```
-
----
-
-## Debugging
-
-### Check Row Count at Each Step
-
-```m
-// Temporarily add to see counts
-let
-    Source = MySource,
-    _ = Text.From(Table.RowCount(Source)),  // Check in formula bar
-    Filtered = Table.SelectRows(Source, each [Status] = "Active"),
-    __ = Text.From(Table.RowCount(Filtered))  // Check after filter
-in
-    Filtered
-```
-
-### Enable Query Diagnostics
-
-1. Tools > Options > Diagnostics
-2. Enable "Query Diagnostics"
-3. Tools > Start Diagnostics
-4. Refresh query
-5. Tools > Stop Diagnostics
-6. Analyze diagnostic tables
-
----
-
-## Contributing
-
-Add new snippets as `.md` files with:
-
-1. **Clear title** - Descriptive name
-2. **Pattern category** - Cleaning, Transform, etc.
-3. **Working M code** - Tested examples
-4. **Parameter explanations** - What to customize
-5. **Usage notes** - When to use
-6. **Performance notes** - Folding impact if relevant
-
-### File Template
-
-```markdown
-# Pattern Name
-
-Description of what this pattern does.
-
----
-
-## Basic Pattern
-
-\`\`\`m
-let
-    Source = YourSource,
-    Result = Table.TransformColumns(Source, ...)
-in
-    Result
-\`\`\`
-
-## Variations
-
-### Variation 1
-...
-
-## Usage Notes
-
-- Note 1
-- Note 2
-```