@ketecode/loom 0.0.0-stage → 0.2.0

package/skills/marketing-campaign/SKILL.md

@@ -0,0 +1,111 @@
+---
+name: marketing-campaign
+description: Builds a go-to-market or promotional campaign plan with one measurable objective, a defined audience, positioning and core message, a channel mix, a week-by-week calendar, a budget split and the metrics that decide success. Use when a founder or team is planning a product launch, a seasonal promotion, a user-acquisition push or any time-boxed marketing campaign and needs a plan they can execute and measure.
+license: MIT
+compatibility: Any Agent Skills host. No tools required; reads product docs or analytics exports if the user provides them.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: business
+  kete-loom/risk: read-only
+  kete-loom/owner: "Kete-org"
+---
+
+# Marketing campaign plan
+
+Produce a plan someone can run on Monday: one objective, one audience, one message, few channels, a calendar, a budget and the numbers that say whether it worked.
+
+## 0. Brief: collect before planning
+
+Ask for anything missing. Do not invent figures the plan depends on.
+
+1. **Product or offer**: what is being promoted and what is new about it.
+2. **Objective**: one primary outcome with a number and a date ("400 paid sign-ups by 30 June"). If the user gives several, ask which one wins when they conflict.
+3. **Audience**: who buys, who uses, where they already spend attention.
+4. **Budget and people**: total spend, and who will create content and run channels.
+5. **Constraints**: dates, regulated claims (finance, health), markets and languages, brand rules.
+6. **History**: past campaigns, current conversion rates, cost per acquisition, list sizes. If there is none, say the first campaign is also a measurement exercise.
+
+**Stop and ask** if there is no measurable objective or no budget range. A plan without them cannot be judged.
+
+## 1. Audience
+
+Write one primary segment (and at most one secondary) as:
+
+- Who they are (role, context, market).
+- The problem or trigger that makes them look for a solution now.
+- What they use today instead, and why it falls short.
+- Where they get information (communities, platforms, publications, people they trust).
+
+If the user has research or customer interviews, use their words. Otherwise mark the profile as an assumption to test.
+
+## 2. Positioning and message
+
+- **Positioning statement**: For <audience> who <need>, <product> is the <category> that <key benefit>, unlike <alternative>, because <proof>.
+- **Core message**: one sentence a customer could repeat.
+- **Three proof points**: facts, numbers, customer quotes or demos. No unprovable superlatives.
+- **Call to action**: one action per touchpoint (sign up, book a demo, buy).
+
+If the brand has a voice guide, apply it; the `brand-voice` skill covers that.
+
+## 3. Channels
+
+Pick two to four channels the audience actually uses and the team can sustain. For each, record its role in the funnel:
+
+| Channel | Role | Owned / earned / paid | Content needed | Owner |
+|---|---|---|---|---|
+
+Decision rules:
+
+- Owned first (email list, existing users, website, social accounts): cheapest, fastest feedback.
+- Paid only with tracking in place (UTM parameters on every link, conversion events firing) and a test budget before the main spend.
+- Earned (press, partners, communities, creators) needs lead time; start outreach before launch week.
+- Drop a channel nobody on the team can run every week.
+
+## 4. Calendar
+
+Lay out the campaign in phases with dates:
+
+1. **Prepare** (assets, landing page, tracking, audiences, outreach).
+2. **Warm-up** (teasers, waitlist, partner briefings).
+3. **Launch** (the main push; concentrated activity across channels).
+4. **Sustain** (follow-ups, retargeting, case studies, reminders).
+5. **Review** (results, learnings, decision on what continues).
+
+Give each item a date, channel, asset, owner and status. A week-by-week table is enough.
+
+## 5. Budget split
+
+- Allocate by role: content production, paid media (spend a small test slice first and scale only the channels that hit the cost-per-result ceiling), tools, partners or creators, contingency.
+- State the assumed cost per result for each paid channel and where the assumption came from (history, the user, or a guess to validate).
+- Show the arithmetic from budget to expected results. If the expected results miss the objective, say so and propose either a smaller objective or more budget. Do not pad the numbers to make it fit.
+
+## 6. Success metrics
+
+- One **primary metric** tied to the objective.
+- Two to four **leading indicators** per stage (reach, click-through, sign-up rate, activation) with a target.
+- **Guardrails**: cost per acquisition ceiling, unsubscribe or complaint rate, refund rate.
+- How each metric is measured (tool, event name, UTM convention) and when it is reviewed.
+- A decision rule: "If cost per sign-up exceeds X after the test budget, pause the channel and reallocate."
+
+## Rules
+
+- Never fabricate benchmarks, conversion rates or audience sizes. Use the user's data or label a figure as an assumption.
+- Never recommend buying email lists, spamming communities, fake reviews or undisclosed paid endorsements. Follow consent rules for email and SMS in the target markets, and ask the user to confirm them.
+- Keep it executable by the stated team. Fewer channels done well beats many done badly.
+
+## Output format
+
+```
+# Campaign plan: <name>
+Objective: <metric, target, date>
+Audience: <primary segment>
+Core message: <one sentence>  | CTA: <action>
+
+## Positioning and proof points
+## Channels (table)
+## Calendar (week-by-week table)
+## Budget (table with arithmetic to expected results)
+## Metrics, guardrails and decision rules
+## Assumptions to validate
+## Risks and open questions
+```

package/skills/marketing-campaign/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Plan a launch campaign for our invoicing app, we have $5,000 and six weeks",
+    "We're running a Black Friday promotion for the online store. Help me put together the channels, calendar and budget.",
+    "I need a go-to-market campaign to get our first 500 paying users in Accra. Who do we target and how?",
+    "Make a campaign plan with success metrics for relaunching our newsletter product",
+    "How should I split a small marketing budget between Instagram ads, email and influencers for our product launch?"
+  ],
+  "should_not_trigger": [
+    "Run a technical SEO check on our marketing site and list the indexing problems",
+    "Define our brand's tone of voice and give do and don't examples for copywriters",
+    "Research our three main competitors and estimate the size of the payroll software market in Kenya",
+    "Write the blog post announcing our new feature, around 1200 words",
+    "Draft the monthly investor update with our MRR and burn"
+  ]
+}

package/skills/mcp-server-patterns/SKILL.md

@@ -0,0 +1,127 @@
+---
+name: mcp-server-patterns
+description: Model Context Protocol server construction, from naming tools and writing their JSON Schema inputs to returning isError results, structured output, tool annotations, OAuth-protected Streamable HTTP or local stdio transports, and testing with the MCP Inspector. Use when creating an MCP server, exposing an API or database to agents as MCP tools, or debugging why a client can't connect to, list or call a server's tools.
+license: MIT
+compatibility: Any Agent Skills host. Testing uses the MCP Inspector via npx (its README lists Node 22.19 or later). Rules are SDK-agnostic; use the official SDK for your language.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: ai
+  kete-loom/risk: runs-commands
+  kete-loom/requires: "node, npx"
+  kete-loom/owner: "Kete-org"
+---
+
+# MCP server patterns
+
+An MCP server is an API whose caller is a language model. Design every tool so a model that has only read its name, description and schema calls it correctly the first time, and so a wrong call does no harm.
+
+Protocol details move fast. The current revision when this skill was written is `2026-07-28`; `references/protocol-notes.md` covers what changed from the `2025-xx` revisions. Check modelcontextprotocol.io/specification for the revision your SDK targets before relying on a wire detail.
+
+## 0. Before you start
+
+1. Ask, or find out, who runs the server: **local** (launched by the user's client as a subprocess → stdio) or **remote** (a hosted service many users connect to → Streamable HTTP with OAuth). **Stop and ask the user** if it isn't clear; the auth model depends on it.
+2. Pick the official SDK for the project's language (TypeScript, Python, C#, Go, Rust, Ruby, Java and others are listed at modelcontextprotocol.io/docs/sdk). Do not hand-roll JSON-RPC.
+3. List the user jobs the server must support (e.g. "find a customer and see their open invoices"), not the endpoints of the API you're wrapping.
+
+## 1. Design the tool set
+
+- One tool per user job, not one per REST endpoint. `find_customer` that searches by name, email or id beats `get_customer_by_id`, `get_customer_by_email`, `list_customers`.
+- Keep the set small (aim for under about 20 tools per server). Every tool's description costs context in every conversation.
+- Separate reads from writes. Never hide a write inside a tool that sounds like a read.
+- Destructive actions (delete, send, pay, deploy) get their own tool with a clear verb, and should support a dry-run or preview argument where the backend allows.
+
+### Naming
+
+- Names: 1 to 128 characters from `A-Z a-z 0-9 _ - .`, case-sensitive, unique within the server. Use `verb_noun` in one consistent style: `search_issues`, `create_issue`, `close_issue`.
+- Clients that aggregate servers may prefix names; don't rely on your name being globally unique, and don't encode the server name in every tool.
+- Set `title` for a human-friendly display name.
+
+### Descriptions
+
+Write each description for the model, in this order: what it does, when to use it (and when to use a sibling tool instead), what it returns, side effects and limits.
+
+```
+Search issues by text, label or assignee. Use before create_issue to avoid duplicates.
+Returns up to `limit` issues (id, title, state, url), newest first. Read-only.
+```
+
+## 2. Input schemas
+
+- `inputSchema` is JSON Schema (2020-12 unless `$schema` says otherwise) and must be an object schema. For a tool with no inputs use `{"type": "object", "additionalProperties": false}`.
+- Give every property a `description`, and use `enum`, `minimum`/`maximum`, `pattern`, `format` and `default` instead of explaining constraints in prose.
+- Prefer flat, primitive arguments. Deeply nested objects and free-form JSON strings are where models make mistakes.
+- Use the identifiers the model will have (names, emails, URLs) or return ids from a search tool that it can pass on. Never require an internal id the model cannot discover.
+- Paginate or cap list results with `limit` (with a sane default and maximum) and a `cursor`.
+- **Validate every input on the server** regardless of the schema. The schema helps the model; it is not a security boundary.
+
+## 3. Results
+
+- Return concise text the model can act on: the fields it needs, not the raw upstream payload. Large blobs blow the context window; truncate and say so ("showing 20 of 312; pass cursor=...").
+- When a tool returns structured data, declare `outputSchema`, return conforming `structuredContent`, and also return the serialized JSON in a text content block for older clients.
+- Use `resource_link` content to point at large items (files, reports) instead of inlining them.
+- Set annotations honestly: `readOnlyHint`, `destructiveHint`, `idempotentHint`, `openWorldHint`. Defaults assume the worst (not read-only, destructive, open world). Clients treat annotations from untrusted servers as hints only, so annotations never replace server-side checks.
+
+## 4. Errors
+
+Two kinds, and the choice matters:
+
+| Situation | Return |
+|---|---|
+| Unknown tool, malformed request, server crash | JSON-RPC error (e.g. `-32602` invalid params, `-32603` internal error) |
+| Bad argument values, upstream API failure, not found, business rule, expired handle | A normal result with `isError: true` and a message that tells the model how to recover |
+
+Phrase each tool error as an instruction: `"No customer matches 'acme'. Try search by email, or call list_customers with a prefix."` Never return a stack trace, an internal hostname or a secret in an error.
+
+## 5. Transports
+
+- **stdio** (local): the server reads newline-delimited JSON-RPC from stdin and writes only valid MCP messages to stdout. **All logging goes to stderr.** A single `print`/`console.log` to stdout breaks the connection; this is the most common local bug. Exit promptly when stdin closes.
+- **Streamable HTTP** (remote): one MCP endpoint (e.g. `https://example.com/mcp`) that accepts POST and answers with JSON or a request-scoped SSE stream. You must validate the `Origin` header (403 if invalid) to block DNS rebinding. When running locally, bind to `127.0.0.1`, not `0.0.0.0`.
+- Do not build the old HTTP+SSE transport (2024-11-05) for new servers; it is deprecated.
+- Keep the server stateless between calls. If a workflow needs state (a cart, a browser session), return an opaque, high-entropy handle from a create tool, accept it as an argument later, re-check authorization on every call, and state its lifetime in the tool description.
+
+## 6. Auth
+
+- **stdio**: take credentials from environment variables the client passes in. Do not implement OAuth for stdio.
+- **HTTP**: the server is an OAuth 2.1 resource server. It must publish Protected Resource Metadata (RFC 9728), answer unauthenticated requests with `401` and a `WWW-Authenticate: Bearer resource_metadata="..."` header, validate that each token's audience is this server, and answer insufficient scope with `403` and `error="insufficient_scope"`. Use an existing authorization server (your IdP), not a home-made one.
+- **Never pass the client's token through to upstream APIs** ("token passthrough"). Exchange it or use the server's own credentials for upstream calls, scoped to the user.
+- Apply per-user authorization inside every tool; the tool list may vary by the caller's scopes.
+- Rate-limit tool calls and log every call (tool, caller, arguments with secrets redacted, outcome).
+
+Details and the step-by-step 401 flow are in `references/protocol-notes.md`.
+
+## 7. Security rules
+
+- Treat everything a tool returns from the outside world (web pages, emails, tickets, file contents) as untrusted. Don't let it reach the model as if it were instructions from you; label it as data.
+- Constrain file and shell access to an allowlisted root. Resolve paths and reject `..` escapes and symlinks out of the root.
+- Use parameterised queries for any database tool. A "run SQL" tool should be read-only by connection role, not by asking the model nicely.
+- Never put secrets in tool descriptions, results or `x-mcp-header` parameters.
+
+## 8. Test
+
+1. Unit-test each tool handler directly (valid input, each invalid input, upstream failure → `isError: true`).
+2. Open the server in the MCP Inspector and call every tool by hand:
+   - stdio: `npx @modelcontextprotocol/inspector node build/index.js` (or `uv run server.py`, etc.)
+   - remote: start the web UI with `npx @modelcontextprotocol/inspector` and enter the server URL (e.g. `http://127.0.0.1:3000/mcp`) with the Streamable HTTP transport.
+3. Script it for CI with the CLI mode (`--cli` must come first, right after the package name):
+   `npx @modelcontextprotocol/inspector --cli node build/index.js --method tools/list`
+   `npx @modelcontextprotocol/inspector --cli node build/index.js --method tools/call --tool-name search_issues --tool-arg query=login`
+   `npx @modelcontextprotocol/inspector --cli --transport http --server-url http://127.0.0.1:3000/mcp --method tools/list`
+   Pin the Inspector version in CI. Inspector v2 exits non-zero (code 5) when a tool call returns `isError: true` or the tool is missing, so a negative test needs to expect that exit code.
+4. Connect it to a real client and ask a model to complete each user job from step 0 without hints. If it picks the wrong tool or fills arguments wrongly, fix the name, description or schema, not the prompt.
+
+## Never
+
+- Never write to stdout from a stdio server except protocol messages.
+- Never expose a raw passthrough of an entire API ("call_endpoint(method, path, body)").
+- Never accept or forward tokens that were not issued for this server.
+- Never rely on the schema or annotations for security.
+- Never ship a destructive tool without a test that proves it refuses unauthorized callers.
+
+## Report format
+
+```
+Server: <name>  Transport: <stdio | streamable HTTP>  Auth: <env | OAuth via ...>
+Tools: <name> (read-only | writes | destructive), ...
+Inspector: tools/list OK, <n>/<n> tools called successfully
+Open issues: <anything not tested, spec revision assumptions>
+```

package/skills/mcp-server-patterns/evals/triggers.json

@@ -0,0 +1,18 @@
+{
+  "should_trigger": [
+    "Build an MCP server that exposes our Jira instance as tools for Claude and Cursor",
+    "My stdio MCP server disconnects right after starting, the client says invalid JSON",
+    "How should I name and describe the tools on my Model Context Protocol server?",
+    "Add OAuth to our remote MCP server running over streamable HTTP",
+    "Turn our internal billing REST API into an MCP server agents can call",
+    "Test my MCP server's tools with the inspector before publishing it"
+  ],
+  "should_not_trigger": [
+    "Design the overall tool set and sandbox for our autonomous coding agent",
+    "Add vector search and reranking over our product manuals for the support bot",
+    "Design a clean REST API for our orders service with versioning and pagination",
+    "Write prompts that make the model call functions more reliably",
+    "Set up OAuth login with Google for our web app users",
+    "Review our retrieval-augmented generation code for chunking problems"
+  ]
+}

package/skills/mcp-server-patterns/references/protocol-notes.md

@@ -0,0 +1,68 @@
+# MCP protocol notes
+
+Read this when a wire-level detail matters: version negotiation, HTTP headers, auth flow, or interop with older clients. The source of truth is modelcontextprotocol.io/specification; these notes follow revision `2026-07-28`.
+
+## Revisions and eras
+
+- `2024-11-05`: original revision. HTTP+SSE transport (separate SSE and POST endpoints). Now deprecated.
+- `2025-03-26` to `2025-11-25`: Streamable HTTP with optional sessions (`Mcp-Session-Id`), standalone GET stream for server messages, resumable streams, `initialize` handshake.
+- `2026-07-28`: no `initialize` handshake and no protocol-level sessions. Every request carries its protocol version and client capabilities in `params._meta` (`io.modelcontextprotocol/protocolVersion`, `io.modelcontextprotocol/clientCapabilities`, `io.modelcontextprotocol/clientInfo`). Servers no longer send JSON-RPC requests to clients; sampling, elicitation and roots come back as an `InputRequiredResult` the client answers by retrying the call. Change notifications arrive on a `subscriptions/listen` stream.
+
+Your SDK handles most of this. What matters to a server author:
+
+- Do not keep per-connection state; the protocol has none. Use explicit handles (see SKILL.md section 5).
+- If you must support older clients, check your SDK's backward-compatibility support rather than implementing both eras yourself.
+
+## Streamable HTTP (2026-07-28)
+
+- Single endpoint, POST only. GET or DELETE from old clients: `405 Method Not Allowed`. Ignore any `Mcp-Session-Id` and `Last-Event-ID` headers.
+- Request headers the client must send: `MCP-Protocol-Version`, `Mcp-Method`, and `Mcp-Name` for `tools/call`, `resources/read`, `prompts/get`. The server must reject a mismatch between header and body with `400` and JSON-RPC error `-32020` (HeaderMismatch).
+- Unsupported protocol version: `400` with an `UnsupportedProtocolVersionError` listing supported versions.
+- Unknown method: `404` with JSON-RPC `-32601`.
+- Notifications POSTed by the client: `202 Accepted`, no body.
+- Responses: `application/json` or `text/event-stream`. Set `X-Accel-Buffering: no` on SSE responses behind nginx-style proxies.
+- Cancellation on HTTP is the client closing the response stream; stop work when that happens.
+- `x-mcp-header` in a property's schema mirrors that argument into an `Mcp-Param-<Name>` header for routing. Only primitive string, integer or boolean properties reachable through `properties` keys. Never on secrets or PII.
+
+## stdio
+
+- Newline-delimited JSON-RPC, no embedded newlines, UTF-8.
+- stdout carries only MCP messages; stderr is free for logs.
+- Cancellation is a `notifications/cancelled` notification from the client.
+- Exit when stdin reaches end-of-file.
+
+## Authorization flow (HTTP servers)
+
+1. Client calls the MCP endpoint without a token.
+2. Server returns `401` with `WWW-Authenticate: Bearer resource_metadata="https://mcp.example.com/.well-known/oauth-protected-resource", scope="files:read"`.
+3. Client fetches the Protected Resource Metadata (RFC 9728), which lists `authorization_servers` and `scopes_supported`.
+4. Client discovers the authorization server (RFC 8414 or OpenID Connect Discovery), registers (Client ID Metadata Documents preferred; Dynamic Client Registration is deprecated but allowed), and runs authorization code + PKCE with a `resource` parameter (RFC 8707) naming the MCP server's canonical URI.
+5. Client sends `Authorization: Bearer <token>` on every request. Never in the query string.
+6. Server validates the token per OAuth 2.1 and checks its audience is this server. Invalid or expired: `401`. Insufficient scope: `403` with `WWW-Authenticate: Bearer error="insufficient_scope", scope="...", resource_metadata="..."`, listing all scopes the operation needs in one challenge.
+7. Server must not accept or forward tokens issued for anything else.
+
+Keep `offline_access` out of the scopes the server advertises; refresh tokens are a client concern.
+
+## Error codes
+
+| Code | Meaning |
+|---|---|
+| -32700 | Parse error |
+| -32600 | Invalid request |
+| -32601 | Method not found |
+| -32602 | Invalid params (also: unknown tool) |
+| -32603 | Internal error |
+| -32020 | Header mismatch (HTTP) |
+| -32021 | Missing required client capability |
+| -32022 | Unsupported protocol version |
+
+Tool execution failures are not JSON-RPC errors: return a result with `isError: true`.
+
+## Tool annotation defaults
+
+| Hint | Default | Meaning when true |
+|---|---|---|
+| `readOnlyHint` | false | Does not modify its environment |
+| `destructiveHint` | true | May delete or overwrite (only meaningful if not read-only) |
+| `idempotentHint` | false | Repeating with same args has no extra effect (only meaningful if not read-only) |
+| `openWorldHint` | true | Talks to external systems (web, third-party APIs) |

package/skills/metrics-dashboards/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: metrics-dashboards
+description: Designs business and product metric dashboards that people act on, starting from the decisions the audience makes, writing each metric as a precise definition with formula, grain, filters, time zone and owner, picking a chart type per question, laying out headline numbers then trends then breakdowns, showing data freshness and quality status on the page, and cutting vanity metrics that never change a decision. Use when planning a KPI or operations dashboard, a dashboard nobody trusts or opens, two reports disagree on the same number, or choosing which metrics and charts belong on a Looker, Metabase, Superset, Tableau, Power BI or Grafana board.
+license: MIT
+compatibility: Any Agent Skills host. Tool-neutral; works with any BI or dashboard tool. Querying data needs the project's warehouse or database client.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: data
+  kete-loom/risk: read-only
+  kete-loom/owner: "Kete-org"
+---
+
+# Metrics dashboards
+
+A dashboard earns its place when a named person looks at it on a schedule and it changes what they do. Design for that person, define every number so two people compute it the same way, and show when the data cannot be trusted.
+
+## 1. Start from decisions, not data
+
+Ask the user (and **stop until you have answers**):
+
+1. **Who** looks at it (role, not "everyone") and **how often** (live, daily stand-up, weekly review, monthly board)?
+2. **What decisions** do they make with it? ("Add support staff", "pause a campaign", "page on-call".)
+3. **What would make them act?** A threshold, a target, or a change versus last period.
+
+Write one line per decision: *"<Audience> uses <metric> to decide <action> when <condition>."* Any metric that fits no line is a candidate to cut.
+
+## 2. Define every metric precisely
+
+Fill this card for each metric before building any chart. Put the cards in the repo or the BI tool's metric layer (dbt semantic layer, LookML, Cube, the tool's own) so they are defined once.
+
+```
+Name:          Weekly active customers
+Question:      How many paying customers used the product this week?
+Formula:       count(distinct customer_id) where event in (core actions) and plan != 'free'
+Grain:         one value per ISO week (Mon-Sun)
+Time zone:     UTC  (state it; local-time reports will differ)
+Filters:       excludes internal and test accounts (is_internal = false)
+Source:        analytics.fct_events, analytics.dim_customers
+Owner:         <person or team who answers questions about it>
+Target/alert:  <target or threshold, if any>
+Caveats:       events before 2025-03-01 missing mobile app
+```
+
+Rules:
+
+- **Ratios state numerator and denominator** and whether the result is an average of ratios or a ratio of sums.
+- **Counts say what is counted once** (distinct users, sessions, orders) and over what window.
+- **Money states currency, gross or net, and tax and refund treatment.**
+- If two reports disagree, compare their cards line by line; the difference is almost always filters, time zone, or a different window. Do not average them or pick one silently: report the cause to the user.
+
+## 3. Cut vanity metrics
+
+Remove or demote a metric when:
+
+- it only ever goes up (cumulative signups, total page views all time);
+- no one can name an action they would take if it moved;
+- it is easy to game without improving the outcome (raw clicks, time on page alone);
+- it duplicates another metric on the board.
+
+Prefer rates and per-unit measures (conversion rate, revenue per active customer, p95 latency) and cohort or period-over-period views over all-time totals. Pair a metric that can be gamed with a counter-metric (tickets closed with reopen rate; signups with activation rate).
+
+## 4. Choose the chart for the question
+
+| Question | Use | Avoid |
+|---|---|---|
+| Where are we now versus target? | Single number with comparison (vs last period, vs target) | Gauges and dials |
+| How is it trending? | Line chart, time on x-axis, consistent intervals | Pie, 3D, dual y-axes |
+| How do categories compare? | Horizontal bar, sorted by value | Pie with more than 3 to 4 slices |
+| What makes up the total? | Stacked bar (few parts) or a table | Donut with many slices |
+| How is it distributed? | Histogram or box plot; show percentiles | Averages alone for skewed data like latency or revenue |
+| Which items need attention? | Sorted table with conditional formatting | Dense heatmap without labels |
+| Did the funnel leak? | Funnel or step bar with conversion between steps | |
+
+Chart rules: start bar axes at zero; label units and time zone; mark incomplete periods (the current day or week) so a partial bar is not read as a drop; annotate known events (launch, outage, pricing change); use color for meaning (bad, good, highlighted series), not decoration, and do not rely on color alone.
+
+## 5. Layout
+
+1. **Top row**: 3 to 6 headline numbers tied to the decisions in step 1, each with its comparison.
+2. **Middle**: trends for those same metrics.
+3. **Bottom**: breakdowns and drill-down tables (by segment, region, channel).
+4. Global filters (date range, segment) at the top, applied to every tile; say so if a tile ignores a filter.
+5. One dashboard per audience and cadence. If it needs scrolling through more than two screens or more than about 12 tiles, split it.
+6. Each tile has a title that is a question or a plain name, plus a short description or link to its metric card.
+
+## 6. Freshness and data quality on the page
+
+- Show **"Data as of <timestamp, time zone>"** from the actual `max(loaded_at)` of the source, not the page render time.
+- Show a status when an upstream check failed or the data is late ("Orders data delayed: last load 09:10 UTC, expected 06:00"). Pull this from the pipeline's freshness and test results rather than typing it by hand.
+- Match refresh cadence to the decision cadence; a weekly review does not need a 1-minute refresh.
+- Note known gaps and definition changes on the dashboard with the date they took effect.
+
+## 7. Build and verify
+
+1. Write the query for each metric from its card; reuse the metric layer or a shared model rather than copying SQL per tile.
+2. Check each headline number against an independent count (a direct query, the source system's own report) for one recent period. Report any gap above 1% to the user with its cause.
+3. Check edge cases: empty segments, the current partial period, time-zone boundaries, and a filter combination with no data.
+4. Ask the intended audience to read it once and say what they would do; adjust or cut tiles they skip.
+5. Set a review date (for example quarterly) to delete tiles no one uses; most BI tools report view counts.
+
+## Never
+
+- Never ship a metric without a written definition and owner.
+- Never present a partial period as a complete one.
+- Never change a metric's definition without a dated note on the dashboard and telling its owner.
+- Never hide a failed data check by showing the last good number with no warning.
+
+## Report back
+
+- Audience, cadence and the decision lines.
+- Metric cards (one per metric) and the metrics cut, with reasons.
+- Layout: tiles in order with chart type.
+- Verification results: each headline number versus its independent check.
+- Freshness and quality indicators and where they read from.

package/skills/metrics-dashboards/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Help me design a KPI dashboard for our support team's weekly review",
+    "Finance and product report different monthly active user numbers, figure out why and fix the definition",
+    "Nobody uses our Metabase dashboard, how should we redesign it?",
+    "Which chart should I use to show conversion by marketing channel on the growth dashboard?",
+    "Add a data freshness indicator to our Looker operations board"
+  ],
+  "should_not_trigger": [
+    "Build the dbt mart that feeds the revenue dashboard",
+    "Write the nightly job that loads Stripe charges into the warehouse",
+    "Pick the five metrics investors care about for our seed pitch deck",
+    "Set up Prometheus alerts for API p99 latency above 500ms",
+    "Run a statistical test on last month's A/B experiment results in a notebook"
+  ]
+}

package/skills/ml-experiment-workflow/SKILL.md

@@ -0,0 +1,125 @@
+---
+name: ml-experiment-workflow
+description: Plans and runs machine learning experiments so results can be trusted, from a dumb baseline and leakage-free train, validation and test splits, through tracked runs, metrics chosen to match the business goal, and ablations, to turning a notebook into a repeatable pipeline. Use when starting a new model, comparing approaches, deciding whether an improvement is real, or productionising notebook experiments.
+license: MIT
+compatibility: Any Agent Skills host. Framework-agnostic (scikit-learn, PyTorch, XGBoost and others); needs Python to run experiments.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: ai
+  kete-loom/risk: runs-commands
+  kete-loom/requires: "python, git"
+  kete-loom/owner: "Kete-org"
+---
+
+# ML experiment workflow
+
+An experiment is only useful if someone else could rerun it and get the same answer, and if the number it reports means what the user thinks it means. Follow these steps in order. For PyTorch training-loop mechanics use the `pytorch-patterns` skill; for evaluating LLM outputs use `llm-evals`.
+
+## 1. Pin down the goal
+
+Write these down with the user before touching data:
+
+1. **Decision.** What will be done differently if the model is good? (Auto-approve claims, rank search results, flag churn risk.)
+2. **Metric that matches it.** Pick from the cost of errors, not habit:
+   - Imbalanced classes: precision/recall at an operating threshold, PR-AUC, or cost-weighted error. Accuracy is misleading when one class is 95% of the data.
+   - Ranking: NDCG@k, recall@k, MRR at the k the product shows.
+   - Probabilities used downstream: log loss and calibration (Brier score, reliability plot).
+   - Regression: MAE when all errors cost the same, RMSE when large errors are much worse, MAPE only when targets are never near zero.
+3. **Bar to beat.** The current system's number, or a minimum the decision needs.
+4. **Unit of prediction.** Per user, per transaction, per day? This drives the split.
+
+**Stop and ask** if the user cannot name the decision or the metric. Don't pick one silently.
+
+## 2. Split without leakage
+
+Decide the split before any exploration, then never look at the test set until the end.
+
+- **Time-ordered data** (forecasting, anything predicted about the future): split by time. Train on the past, validate and test on later periods. Use `TimeSeriesSplit` or a rolling-origin scheme, not random K-fold. When the label looks ahead (for example "churns in the next 30 days"), leave a gap of at least that horizon between train and validation (`TimeSeriesSplit(gap=...)`) so training labels do not overlap the validation period.
+- **Grouped data** (several rows per patient, user, device, document): split by group so no group appears on both sides. Use `GroupKFold` / `GroupShuffleSplit`.
+- **Imbalanced labels**: stratify (`StratifiedKFold`, `StratifiedGroupKFold`).
+- **Duplicates**: dedupe, or group near-duplicates, before splitting.
+
+Leakage checklist (any yes means fix before continuing):
+
+- Is any preprocessing (scaling, imputation, target encoding, feature selection, vocabulary, PCA) fitted on data that includes validation or test rows? Fit it inside the training fold only; in scikit-learn, put it in a `Pipeline` passed to cross-validation.
+- Does any feature use information not available at prediction time (a later timestamp, a status set after the outcome, an aggregate over the whole dataset)?
+- Was the test set used to choose features, hyperparameters, thresholds or the stopping epoch?
+- Is a feature suspiciously predictive on its own? Check top-importance features one by one.
+
+## 3. Baselines first
+
+Before any real model, record:
+
+1. A trivial baseline: majority class / mean prediction / "same as last period" / popularity ranking.
+2. A simple model: logistic or linear regression, or gradient-boosted trees for tabular data, with default settings.
+3. The current production system, if there is one.
+
+If a complex model does not clearly beat the simple one, report that; it is a result, not a failure.
+
+## 4. Track every run
+
+Each run records, automatically:
+
+- git commit (and whether the tree was dirty), config/hyperparameters, seed
+- dataset version: file hash, snapshot id or query plus date
+- metrics on validation (and test only for the final run), with the split definition
+- environment: Python and key library versions, hardware
+- artifacts: model file, predictions on the validation set
+
+Use whatever the project already has (MLflow, Weights & Biases, DVC, a CSV log). If there's nothing, write one JSON line per run to a log file and commit the code that writes it. Don't add a tracking service the user did not ask for.
+
+## 5. Decide whether a gain is real
+
+- Run the leading candidates with at least 3 seeds (or use cross-validation folds) and report mean and spread, not the best run.
+- A gain smaller than the seed-to-seed spread is noise. Say so.
+- For a single held-out set, use a paired bootstrap over test examples to get an interval on the difference.
+- Compare models on the same split, the same preprocessing and the same eval code.
+- Look at the errors: slice the metric by important segments (region, device, class, new vs returning users). An average gain that hides a regression on a key segment must be reported.
+
+## 6. Ablations
+
+When a model has several changes stacked together, remove one at a time from the final configuration and rerun. Report a table:
+
+```
+| Variant                  | Val metric (mean ± std, n seeds) | Δ vs full |
+|--------------------------|----------------------------------|-----------|
+| Full                     | 0.812 ± 0.004                    | —         |
+| − feature group X        | 0.809 ± 0.005                    | −0.003    |
+| − augmentation           | 0.781 ± 0.006                    | −0.031    |
+```
+
+Drop any component whose removal does not hurt beyond noise; it is complexity with no evidence behind it.
+
+## 7. From notebook to pipeline
+
+Do this once an approach is worth keeping:
+
+1. Move code out of cells into modules with functions: `load_data`, `build_features`, `split`, `train`, `evaluate`. The notebook then only imports and calls them.
+2. Put every constant (paths, dates, hyperparameters, seed) in one config file.
+3. Provide one command that runs the whole thing end to end, for example `python -m project.train --config configs/baseline.yaml`.
+4. Add tests for the feature code and the split: a test that train and test share no group ids or overlapping time ranges is cheap and catches the worst bugs.
+5. Rerun from a clean checkout and confirm the metric matches the notebook within seed noise. If it does not, find out why before anything else.
+
+## 8. Final evaluation
+
+Only after model, features and threshold are frozen: evaluate once on the test set. If the test result is then used to change anything, that set is no longer a test set; say so and hold out new data.
+
+## Never
+
+- Never report a test-set number that was used for tuning.
+- Never report the best of many seeds as the result.
+- Never fit preprocessing on the full dataset before splitting.
+- Never claim an improvement without the baseline beside it.
+
+## Report
+
+```
+Goal: <decision> | Metric: <metric @ threshold/k> | Bar: <number>
+Split: <method, sizes, date ranges or group key>
+Results (validation, n seeds):
+  baseline: ... | simple model: ... | candidate: ...
+Test (final, once): ...
+Leakage checks: <passed / issues found>
+Caveats: <segments that regressed, what was not checked>
+Reproduce: <exact command and commit>
+```

package/skills/ml-experiment-workflow/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "We want to predict customer churn next quarter. How should I set up the experiment and which metric should we use?",
+    "My new model is 0.4% better on validation than the old one. Is that a real improvement?",
+    "Turn my exploratory Jupyter notebook into a reproducible training pipeline with tracked runs",
+    "Plan an ablation study for the three changes we stacked into the fraud model",
+    "How do I split patient records into train and test so the same patient never shows up in both?"
+  ],
+  "should_not_trigger": [
+    "My torch DataLoader is slow and the GPU sits at 20% utilisation",
+    "Check this pull request's feature engineering code for target leakage and broadcasting bugs",
+    "Write a SQL query that counts weekly active users by country",
+    "Score our RAG assistant's answers for faithfulness against the source documents",
+    "Benchmark this sorting function against the old version and tell me if it's faster"
+  ]
+}