@hailer/mcp 1.3.14 → 1.3.23

package/.claude/skills/sdk-insight-calculations/SKILL.md

@@ -0,0 +1,364 @@
+---
+name: sdk-insight-calculations
+description: Named calculation recipes for Hailer insights — weighted sums, CASE bucketing, conditional counts, percentage-of-total, conversion rates, date truncation, age-since, top-N, cross-join for zero-activity rows, and Finnish DST offset. Load when designing insights that do more than plain SUM/COUNT/GROUP BY. Use alongside sdk-insight-queries (SQL mechanics) and insight-join-patterns (cross-workflow JOINs).
+---
+
+# Insight Calculation Recipes
+
+A library of named calculation patterns you can drop into insight SQL. Each recipe includes when to use it, a working query, and common mistakes to avoid.
+
+**Always validate** with `mcp__hailer__run_insight` before committing. Many recipes depend on function field materialization or SQL dialect quirks only visible with real rows.
+
+---
+
+## Recipe 1 — Weighted Sum by Category
+
+**When to use:** Forecast a pipeline by phase probability, or any metric where each row's contribution depends on a category column.
+
+```sql
+SELECT
+  COALESCE(SUM(d.value * CASE d.phaseId
+    WHEN '${Deals_PhaseIds.new_abc}'          THEN 0.10
+    WHEN '${Deals_PhaseIds.qualified_def}'    THEN 0.25
+    WHEN '${Deals_PhaseIds.proposal_ghi}'     THEN 0.50
+    WHEN '${Deals_PhaseIds.negotiation_jkl}'  THEN 0.75
+    WHEN '${Deals_PhaseIds.won_mno}'          THEN 1.00
+    ELSE 0
+  END), 0) AS "Ennustearvo",
+  COUNT(*) AS "Kauppoja"
+FROM deals d
+WHERE d.phaseId != '${Deals_PhaseIds.lost_pqr}'
+```
+
+**Common mistakes:**
+- Forgetting `COALESCE` on outer SUM → null if no rows
+- Matching on `phaseName` instead of `phaseId` → breaks when someone renames a phase
+- Missing `ELSE 0` → unknown phases contribute NULL and poison the SUM
+
+---
+
+## Recipe 2 — CASE Bucketing
+
+**When to use:** Group a continuous numeric column into labeled bins (deal size tiers, revenue brackets, response-time buckets).
+
+```sql
+SELECT
+  CASE
+    WHEN d.value < 1000   THEN '1_small'
+    WHEN d.value < 10000  THEN '2_mid'
+    WHEN d.value < 100000 THEN '3_large'
+    ELSE                       '4_enterprise'
+  END AS bucket,
+  COUNT(*) AS "Kauppoja",
+  COALESCE(SUM(d.value), 0) AS "Yhteensä"
+FROM deals d
+GROUP BY bucket
+ORDER BY bucket
+```
+
+**Common mistakes:**
+- Unsorted bucket labels → add sort prefix (`1_small`, `2_mid`) so ORDER BY bucket gives natural order
+- Bucketing a NULL-able field — add `WHEN d.value IS NULL THEN '0_unknown'` as first branch
+
+---
+
+## Recipe 3 — Conditional Count
+
+**When to use:** Count rows matching a condition **within** a broader group — e.g., "won deals per owner, alongside total deals per owner."
+
+```sql
+SELECT
+  COALESCE(u.name, 'Unassigned') AS "Omistaja",
+  COUNT(*) AS "Kauppoja yhteensä",
+  COUNT(CASE WHEN d.phaseId = '${Deals_PhaseIds.won_mno}' THEN 1 END) AS "Voitettu",
+  COUNT(CASE WHEN d.phaseId = '${Deals_PhaseIds.lost_pqr}' THEN 1 END) AS "Hävitty"
+FROM deals d
+LEFT JOIN user u ON u._id = d.owner
+GROUP BY "Omistaja"
+ORDER BY "Kauppoja yhteensä" DESC
+```
+
+**Key insight:** `COUNT(CASE WHEN ... THEN 1 END)` counts only matched rows (NULL from non-matches is ignored by COUNT).
+
+**Common mistake:** `COUNT(CASE WHEN ... THEN 1 ELSE 0 END)` — ELSE 0 makes it count all rows. Drop the ELSE branch.
+
+---
+
+## Recipe 4 — Percentage of Total
+
+**When to use:** Show each row's share of the group-wide sum.
+
+```sql
+SELECT
+  d.phaseName AS "Vaihe",
+  COALESCE(SUM(d.value), 0) AS "Arvo",
+  ROUND(
+    100.0 * SUM(d.value) / NULLIF((SELECT SUM(value) FROM deals), 0),
+    1
+  ) AS "% kokonaisarvosta"
+FROM deals d
+GROUP BY d.phaseName
+ORDER BY "Arvo" DESC
+```
+
+**Key parts:**
+- `100.0 *` forces float division (integer division truncates to 0%)
+- `NULLIF(denominator, 0)` prevents division-by-zero
+- Correlated subquery runs once for all rows
+
+---
+
+## Recipe 5 — Conversion Rate / Win Rate
+
+**When to use:** Ratio of "ever reached phase X" count to "ever reached phase Y" count.
+
+**Use `meta: 'phaseLastMove'` — one declared source field per phase.**
+
+```javascript
+// Source declaration
+sources: [{
+  name: 'offers',
+  workflowId: WorkflowIds.offers,
+  fields: [
+    { name: 'id',          meta: '_id' },
+    { name: 'owner',       fieldId: Offers_FieldIds.owner_abc },
+    { name: 'plm_sent',    meta: 'phaseLastMove', phaseId: Offers_PhaseIds.sent_def },
+    { name: 'plm_accepted',meta: 'phaseLastMove', phaseId: Offers_PhaseIds.accepted_ghi },
+  ]
+}]
+```
+
+```sql
+SELECT
+  COALESCE(u.name, 'Unassigned') AS "Omistaja",
+  COUNT(CASE WHEN plm_sent     IS NOT NULL THEN 1 END) AS "Lähetetty",
+  COUNT(CASE WHEN plm_accepted IS NOT NULL THEN 1 END) AS "Hyväksytty",
+  ROUND(
+    100.0 * COUNT(CASE WHEN plm_accepted IS NOT NULL THEN 1 END)
+          / NULLIF(COUNT(CASE WHEN plm_sent IS NOT NULL THEN 1 END), 0),
+    1
+  ) AS "Hyväksymisprosentti"
+FROM offers
+LEFT JOIN user u ON u._id = offers.owner
+GROUP BY "Omistaja"
+ORDER BY "Hyväksymisprosentti" DESC NULLS LAST
+```
+
+**Why `plm_X IS NOT NULL`, not `phaseId = X`:** An offer that went Sent → Accepted → Archived has `phaseId = Archived` now, but `plm_accepted` is non-null. `phaseId = Accepted` only catches activities currently parked there — it misses every accepted offer that moved on.
+
+**Common mistakes:**
+- Using `CASE WHEN phaseId = X` for conversion — misses activities that passed through X
+- Trying `movedTo<PhaseName>` — that column doesn't exist; declare a `phaseLastMove` source field per phase instead
+
+---
+
+## Recipe 6 — Date Truncation for Time Series
+
+**When to use:** Group rows by month/week/quarter/year to produce trend charts.
+
+**For a millisecond timestamp** (activity meta like `d.created`):
+
+```sql
+SELECT
+  strftime('%Y-%m', d.created / 1000, 'unixepoch') AS month,
+  COUNT(*) AS deal_count,
+  COALESCE(SUM(d.value), 0) AS revenue
+FROM deals d
+WHERE d.phaseId = '${Deals_PhaseIds.won_mno}'
+GROUP BY month
+ORDER BY month
+```
+
+**For a seconds timestamp** (most custom date fields):
+
+```sql
+strftime('%Y-%m', d.closeDate, 'unixepoch') AS month
+```
+
+**Truncation format codes:**
+- `'%Y'` → year (e.g. `2026`)
+- `'%Y-%m'` → month (e.g. `2026-04`)
+- `'%Y-%W'` → ISO week
+- `'%Y-%m-%d'` → day
+
+**Common mistakes:**
+- Forgetting to divide by 1000 when field is ms → year 57259
+- Forgetting the `'unixepoch'` modifier → garbage output
+- Forgetting ORDER BY — time-series charts need sorted output
+
+---
+
+## Recipe 7 — Age / Days-Since
+
+**When to use:** "Stale deals" reports without relying on a function field.
+
+**"Now" in SQLite is `strftime('%s', 'now')`** — returns seconds. `NOW()` and `UNIX_TIMESTAMP()` are not available.
+
+**If comparing to a millisecond field** (e.g. `meta.created`):
+
+```sql
+SELECT
+  d.name AS deal,
+  CAST((strftime('%s', 'now') * 1000 - d.created) / 86400000 AS INTEGER) AS days_old
+FROM deals d
+WHERE d.phaseId NOT IN ('${Deals_PhaseIds.won_mno}', '${Deals_PhaseIds.lost_pqr}')
+  AND (strftime('%s', 'now') * 1000 - d.created) > (30 * 86400000)
+ORDER BY days_old DESC
+```
+
+**If comparing to a seconds field** (custom date field):
+
+```sql
+CAST((strftime('%s', 'now') - d.closeDate) / 86400 AS INTEGER) AS days_until_close
+```
+
+**strftime('%s','now') returns TEXT — affects MIN/MAX and comparisons.** Always wrap in `CAST(... AS INTEGER)` before mixing into numeric MIN/MAX/comparison.
+
+```sql
+-- Dangerous — string ordering, wrong result
+MIN(strftime('%s','now'), 9999999999)
+
+-- Safe
+MIN(CAST(strftime('%s','now') AS INTEGER), 9999999999)
+```
+
+---
+
+## Recipe 8 — Top-N per Group
+
+**When to use:** "Top 3 deals per owner", "Largest 5 accounts per industry."
+
+```sql
+SELECT owner, deal, value
+FROM (
+  SELECT
+    COALESCE(u.name, 'Unassigned') AS owner,
+    d.name AS deal,
+    d.value,
+    ROW_NUMBER() OVER (PARTITION BY d.owner ORDER BY d.value DESC) AS rn
+  FROM deals d
+  LEFT JOIN user u ON u._id = d.owner
+) ranked
+WHERE rn <= 3
+ORDER BY owner, value DESC
+```
+
+**Fallback — correlated subquery if no window functions:**
+
+```sql
+SELECT COALESCE(u.name, 'Unassigned') AS owner, d.name AS deal, d.value
+FROM deals d
+LEFT JOIN user u ON u._id = d.owner
+WHERE (
+  SELECT COUNT(*) FROM deals d2 WHERE d2.owner = d.owner AND d2.value > d.value
+) < 3
+ORDER BY owner, d.value DESC
+```
+
+---
+
+## Recipe 9 — Function Fields in WHERE/SELECT
+
+Function field values are materialized on each activity and queryable in all SQL positions.
+
+| Usage | Works? |
+|---|---|
+| `SELECT d.my_ff` | Yes — returns stored FF value (NULL if FF returned null) |
+| `WHERE d.my_ff > 30` | Yes |
+| `ORDER BY d.my_ff` | Yes |
+| `SUM(d.my_ff)` / `AVG(d.my_ff)` | Yes |
+
+**Type note:** FFs declared numeric arrive as `real` (float). If a value is `0.0`, the run_insight text renderer may display it as blank — use `CAST(d.my_ff AS TEXT)` to confirm.
+
+**Inline re-computation is still recommended when:**
+- You don't trust the FF compute to be current (e.g. after bulk phase moves, FFs may be stale until next write)
+- The calculation is simple enough to express inline
+
+---
+
+## Recipe 10 — Helsinki / Finnish DST Offset
+
+**When to use:** Any insight that formats timestamps to local Finnish time.
+
+**Offsets:** EEST (UTC+3, late March → late October), EET (UTC+2, the rest of the year).
+
+**Critical — wrap all `strftime` DST comparisons in `CAST(... AS INTEGER)`** — `strftime('%s', ...)` returns TEXT; comparing a numeric unix column against it inside CASE falls back to string ordering.
+
+```sql
+WITH dst_bounds AS (
+  SELECT
+    CAST(strftime('%s',
+      date(strftime('%Y', ts, 'unixepoch') || '-03-31', 'weekday 0', '-7 days')
+      || ' 03:00:00'
+    ) AS INTEGER) AS summer_start,
+    CAST(strftime('%s',
+      date(strftime('%Y', ts, 'unixepoch') || '-10-31', 'weekday 0', '-7 days')
+      || ' 04:00:00'
+    ) AS INTEGER) AS summer_end
+  FROM src LIMIT 1
+)
+SELECT
+  strftime(
+    '%d.%m.%Y %H:%M', ts, 'unixepoch',
+    CASE
+      WHEN CAST(ts AS INTEGER) >= (SELECT summer_start FROM dst_bounds)
+       AND CAST(ts AS INTEGER) <  (SELECT summer_end   FROM dst_bounds)
+      THEN '+3 hours'
+      ELSE '+2 hours'
+    END
+  ) AS local_time
+FROM src
+```
+
+**For millisecond-unit timestamps:** divide by 1000 before all `strftime` calls.
+
+---
+
+## Recipe 11 — Show All Entities Even with No Activity (Cross-Join Pattern)
+
+**When to use:** You want a complete grid — every entity × every time period — even when some combinations have zero activity. Without this, GROUP BY silently omits zero-count rows.
+
+```sql
+WITH entities AS (
+  SELECT _id AS entity_id, activityName AS entity_name
+  FROM entity_workflow
+),
+periods AS (
+  SELECT DISTINCT strftime('%Y-%m', booking_start, 'unixepoch') AS period
+  FROM bookings
+),
+aggregated AS (
+  SELECT entity_link AS entity_id, strftime('%Y-%m', booking_start, 'unixepoch') AS period, COUNT(*) AS booking_count
+  FROM bookings
+  GROUP BY entity_link, period
+)
+SELECT
+  e.entity_id, e.entity_name, p.period,
+  COALESCE(a.booking_count, 0) AS booking_count
+FROM entities e
+CROSS JOIN periods p
+LEFT JOIN aggregated a ON a.entity_id = e.entity_id AND a.period = p.period
+ORDER BY e.entity_name, p.period
+```
+
+**Key mechanics:**
+- `CROSS JOIN` produces every entity × period combination
+- `LEFT JOIN aggregated` brings in actual data; unmatched combinations stay as NULL
+- `COALESCE(..., 0)` converts NULL to 0
+
+**Diagnostic reminder:** if row count is lower than expected and data "feels like it should be there", check whether you're using INNER JOIN or plain GROUP BY — both silently drop zero-activity entities.
+
+---
+
+## Debugging Checklist (Silent-Wrong-Answer Mode)
+
+Before trusting an insight's numbers:
+
+1. **Row count check** — `SELECT COUNT(*) FROM wf` with no filters matches expected workspace count
+2. **Spot-check a known row** — pick one activity you know the value of, `WHERE _id = 'known'`, assert returned value matches
+3. **Denominator exists** — any `X / Y` has `NULLIF(Y, 0)` on Y
+4. **Float division** — any percentage multiplies by `100.0` not `100`
+5. **NULL handling** — outer aggregates wrapped in `COALESCE(..., 0)` or similar default
+6. **Phase IDs not names** — all phase filters use `phaseId` (rename-safe)
+7. **Preview with real data** — never ship an insight whose `run_insight` output you haven't read row by row
+8. **Row count lower than expected?** — check for INNER JOIN or plain GROUP BY silently dropping zero-activity entities; use CROSS JOIN + LEFT JOIN (Recipe 11)