@malloy-publisher/skills 0.1.0

package/skills/malloy-html-data-app-runtime/SKILL.md

@@ -0,0 +1,158 @@
+---
+name: malloy-html-data-app-runtime
+description: Write the JavaScript that drives an in-package HTML data app, calling Publisher.query, building queries from filter state, and handling results and errors. Read before writing the page's data code.
+---
+
+# HTML Data App Runtime
+
+> `Publisher.query(modelPath, malloy)` returns an array of plain row objects. Build the Malloy string, let the model do the work, and render the rows with whatever front-end code you like.
+
+The runtime loads from the root-relative `<script src="/sdk/publisher.js">` and adds one global, `window.Publisher`.
+
+## The query contract
+
+| Call | Returns | Use for |
+|---|---|---|
+| `Publisher.query(modelPath, malloy, opts?)` | `Promise<Array>` of rows | driving your own charts and tables |
+| `Publisher.queryFull(modelPath, malloy, opts?)` | `Promise<MalloyResult>` | handing to `<malloy-render>` |
+
+- `modelPath` is the model FILE path within the package, with `/` separators (`"subscriptions.malloy"`, `"models/events.malloy"`). It is not the source name.
+- `malloy` is any query string, written in standard Malloy. This skill covers only the JavaScript glue, not Malloy syntax.
+- `opts` (all optional): `sourceName`, `queryName`, `givens` (a `{ name: value }` map bound to the model's Malloy `given:` runtime parameters for this query; safe parameterization, values are bound by the runtime, not string-interpolated), `filterParams` (values for the model's legacy `#(filter)` source filters), `bypassFilters`, and `environment` / `package` (only if the page is served from outside `/environments/<env>/packages/<pkg>/`). `givens` and `filterParams` compose (both apply).
+
+## Structure the app as modules, not one inline script
+
+Past a single tile, an inline `<script>` becomes unmaintainable and untestable. Split the work, and load it without a build step: put your shared libraries first as plain globals, then one ES-module entry point that `import`s your own files.
+
+```html
+<!-- Globals first: the runtime, then any vendored chart library. -->
+<script src="/sdk/publisher.js"></script>
+<script src="./vendor/chart.umd.js"></script>
+<!-- One module entry; it imports the rest. ES modules resolve with no bundler. -->
+<script type="module" src="./app.js"></script>
+```
+
+A separation that keeps each piece testable and changeable on its own:
+
+- **`format.js`**. Pure functions only: number/date formatting, a series-align-by-month helper, status thresholds. No DOM, no globals. This is the file `node --test` can cover directly.
+- **`charts.js`**. Turns a prepared data object into a drawn chart; the only file that touches the chart library.
+- **`tiles.js`**. Your tiles as *data*: for each, its model/source/view (and target source, if any), plus a pure `build(rows)` that shapes query rows for the chart. This is the single source of truth for what each tile queries.
+- **`app.js`**. The thin entry point: reads `tiles.js`, runs the queries, wires results to the DOM. Adding a tile means adding a `tiles.js` entry, not editing `app.js`.
+
+Declare each tile's source and view names once, in `tiles.js`, and have everything else (render code, any agent prompt, tests) read from there. A second copy of those names in another file is the classic drift bug, and a *derived* name (`okr_4_4_2_targets` invented from a tile code) is simply wrong: a target source may have an irregular name or not exist at all. Read the model; don't compute names.
+
+## Patterns that work
+
+These run against the example `html-data-app` package (source `subscriptions`; views `plan_mix`, `mrr_by_industry`, `kpis`). Swap in your own model and view names.
+
+Run a named view:
+
+```js
+const rows = await Publisher.query("subscriptions.malloy", "run: subscriptions -> plan_mix");
+```
+
+Refine a view from UI state by appending a `where:`. Restrict the values to ones you control (for example a dropdown populated from the model's own distinct values) and escape each interpolated value with a backslash before quotes and backslashes (Malloy rejects the SQL-style `''` doubling). An unescaped apostrophe in a value breaks out of the literal:
+
+```js
+function whereClause(state) {
+  const q = (s) => s.replace(/\\/g, "\\\\").replace(/'/g, "\\'"); // backslash-escape for Malloy
+  const parts = [];
+  if (state.plan) parts.push(`plan = '${q(state.plan)}'`);
+  if (state.industry) parts.push(`industry = '${q(state.industry)}'`);
+  return parts.length ? `where: ${parts.join(", ")}` : "";
+}
+const rows = await Publisher.query(
+  "subscriptions.malloy",
+  `run: subscriptions -> plan_mix + { ${whereClause(state)} }`,
+);
+```
+
+Do not interpolate free-text or otherwise untrusted input into the query string. Route parameterized input through `opts.givens` (or the legacy `opts.filterParams`) instead: those values are bound by the runtime as typed parameters, not string-interpolated, so they can't inject query syntax. (One nuance: a `filter<T>`-typed given takes Malloy filter syntax as its value, so validate it against a known set like any other input; scalar givens carry no syntax at all.) `opts.givens` is safe *parameterization*, not an authorization boundary: a client-supplied given is client-trusted unless a server upstream (a trusted gateway, or an operator's per-package config) strips or finalizes it. Where you must build query text from input, constrain it to a known set and escape it, or keep the filtering in model-defined views.
+
+KPI or single-row view. Destructure element zero:
+
+```js
+const [kpis] = await Publisher.query("subscriptions.malloy", "run: subscriptions -> kpis");
+el.textContent = kpis.active_mrr;   // the result is an array; kpis.active_mrr, not rows.active_mrr
+```
+
+Refresh a dashboard. Fire the tiles together:
+
+```js
+const [planMix, byIndustry, kpisRows] = await Promise.all([
+  Publisher.query("subscriptions.malloy", "run: subscriptions -> plan_mix"),
+  Publisher.query("subscriptions.malloy", "run: subscriptions -> mrr_by_industry"),
+  Publisher.query("subscriptions.malloy", "run: subscriptions -> kpis"),
+]);
+```
+
+Prefer defining the views in the model (one per tile, pre-aggregated and sorted) over building long query strings in JS.
+
+Get the numbers right. The fastest way to ship a wrong-but-convincing dashboard is to paper over missing data:
+
+- **Missing is not zero.** When you join two series (actuals to a separately-keyed target) and a key is absent, leave it `null` so the chart skips it. Do not `|| 0`, which plots a real-looking zero and reads as "we hit nothing that month." Align on a normalized key (`"YYYY-MM"`), and let the renderer omit null points:
+
+  ```js
+  // monthKey/monthLabel are your own format.js helpers: monthKey normalizes a
+  // date to a "YYYY-MM" string; monthLabel formats it for display.
+  // target may not cover every actual month; an absent month stays null, never 0.
+  const target = new Map(planRows.map((r) => [monthKey(r.plan_month), Number(r.target_revenue)]));
+  const data = actualRows.map((r) => ({
+    label: monthLabel(r.order_month),
+    actual: Number(r.revenue),
+    target: target.has(monthKey(r.order_month)) ? target.get(monthKey(r.order_month)) : null,
+  }));
+  ```
+
+- **"Current" means latest non-null.** For a KPI scorecard, scan back to the last month that actually has data rather than reading the final row, which may be an incomplete current month.
+- **Guard division in Malloy, not after.** `avg(paid / nullif(active, 0))`. A `nullif` in the query beats catching `Infinity`/`NaN` in JS.
+- **Convert units explicitly.** If the model stores a 0 to 1 fraction and you show a percent, multiply once in `build()` and comment it. Mismatched units are a silent off-by-100.
+
+Loading, empty, and error states. Handle all three; a bare `.then()` that assumes rows leaves the page blank when the query is slow or fails:
+
+```js
+const el = document.getElementById("out");
+el.textContent = "Loading...";
+Publisher.query("subscriptions.malloy", "run: subscriptions -> plan_mix")
+  .then((rows) => {
+    if (!rows.length) { el.textContent = "No data."; return; }
+    render(rows);
+  })
+  .catch((err) => {
+    el.textContent = `Query failed (${err.status ?? ""}): ${err.response?.message ?? err.message}`;
+  });
+```
+
+Render through `<malloy-render>`. `queryFull` returns the full Malloy result envelope (the JSON form of the server's result, not a live result object) to hand to the component:
+
+```js
+const el = document.querySelector("malloy-render");
+el.result = await Publisher.queryFull("subscriptions.malloy", "run: subscriptions -> plan_mix");
+```
+
+Publisher does not serve or bundle `<malloy-render>`; you must obtain a built component bundle matched to your model's Malloy version and vendor it into `public/` yourself, then confirm it accepts the envelope as-is. The shipped example renders rows with a plain chart library instead, so this path is not exercised there. A view tagged in the model (for example `# bar_chart`) drives how it draws.
+
+Validate every query before wiring it into render code, using whatever query tool your environment provides, or by POSTing the query to a running Publisher at `/api/v0/environments/<env>/packages/<pkg>/models/<modelPath>/query` with body `{"compactJson":true,"query":"..."}`, or by running `Publisher.query` once and logging the rows. Malloy names result columns after the `group_by` / `aggregate` field names (`group_by: plan` gives a `plan` column; `aggregate: account_count` gives an `account_count` column), so confirm those names against real output before you read them.
+
+If you validate the rendered page in a headless browser (Playwright or Puppeteer), do not wait for network idle: `publisher.js` holds the live-reload SSE stream open, so the page never reaches it. Wait on `domcontentloaded` or `load` plus a content selector instead.
+
+## Context, auth, live reload (all automatic)
+
+- Context. A page served under `/environments/<env>/packages/<pkg>/...` infers its environment and package, so `query` needs no env or package args. Serving from elsewhere? Pass `opts.environment` and `opts.package`.
+- Auth. By default the runtime sends cookies (`credentials: include`), so a signed-in user is authenticated with no code. For a bearer token, call `Publisher.setToken(token)` first; `Publisher.setToken(null)` reverts to cookies.
+- Live reload. Under `--watch-env`, the page reloads on package changes by itself. Nothing to wire.
+
+## When the app fails
+
+| Symptom | Likely cause and fix |
+|---|---|
+| 404 or "model not found" | `modelPath` wrong. It is the file path (`"subscriptions.malloy"`), with `/` separators, not the source name. |
+| "source/view not defined" | View or source name guessed. Read the model (your environment's context tool, or open the `.malloy` file) and use the real names. |
+| Promise rejects, message starts `Publisher.query:` | Read `error.status` and `error.response` for the server's reason (compile error, missing required parameter, permission). |
+| Empty array when you expect rows | Filter value mismatch (case, spelling, type, or a non-ASCII character like `≤` or an en-dash in the literal). Copy the literal verbatim from the model, do not retype the user's paraphrase, and confirm it with a distinct-values query (`run: src -> { group_by: the_dimension }`). Quote strings, use `@` for dates. |
+| 400 on a given (ungated source) | An unknown given name (check spelling; names are case-sensitive), a required given left unset, or a value that doesn't fit the declared type. Malloy rejects it when preparing the query; supply declared givens via `opts.givens` with the right shape (see the givens type table). |
+| 403 on a query that should be allowed, when passing givens to a gated source | On a source with `#(authorize)`, a bad given (unknown name or wrong-typed value) fails closed in the authorize check, so it looks like access denied rather than validation. Check the given names and values against the model. |
+| KPI shows `undefined` | The result is an array. Read `rows[0].field` (or destructure `const [k] = ...`), not `rows.field`. |
+| Page loads in dev but is not listed or not served | The file is not under the package's `public/` directory. Publisher serves only `public/`; a page written anywhere else (for example `/tmp`) is never reachable at `/environments/<env>/packages/<pkg>/<file>`. |
+| Queries fail only when embedded cross-origin | Cookies are not sent cross-site. Serve same-origin, or pass a bearer token. |
+| No live reload | Watch mode is off. Start with `--watch-env <env>`; without it the events stream reports `mode: disabled` and never reloads. |

package/skills/malloy-html-data-apps/SKILL.md

@@ -0,0 +1,165 @@
+---
+name: malloy-html-data-apps
+description: Build or modify an in-package HTML data app for a Malloy Publisher package (a public/ directory the package serves). Use when the user wants a hand-authored HTML dashboard or web page backed by a package's Malloy models, with no build step.
+---
+
+# In-Package HTML Data Apps
+
+> A package becomes a web app by adding a `public/` directory. Publisher serves those files and gives the page `Publisher.query(...)` to run Malloy against the package's models. No build step, no npm, no framework.
+
+## When this is the right tool
+
+| The user wants | Use |
+|---|---|
+| A hand-authored HTML/JS dashboard, no toolchain | this skill (an HTML data app) |
+| A React app with managed components | the Publisher React SDK (out of scope here) |
+| An analyst notebook with charts | a Malloy notebook (`.malloynb`) |
+| Point-and-click exploration, no code | the Publisher Explorer |
+
+Pick an HTML data app when the user wants full control of the markup and only plain web files.
+
+## Package anatomy
+
+```
+my-package/
+  publisher.json        # name, version, description
+  subscriptions.malloy  # the model(s), stays private
+  subscriptions.parquet # data, stays private
+  public/               # ONLY this directory is web-served
+    index.html
+    app.js
+    vendor/             # chart library, vendored rather than loaded from a CDN
+      chart.umd.js
+```
+
+Only `public/` is reachable over the web, at `/environments/<env>/packages/<pkg>/<file>`. Models, data, and `publisher.json` are private and reached only through the query API, which still applies the model's filters, access modifiers, and authorize rules. There is no flag to set: a `public/` directory is what makes a package an app.
+
+## Build sequence
+
+The agent orchestrates these. Each query and chart step hands off to a focused skill.
+
+1. READ THE MODEL FIRST. Get the model's real source and view names, through your environment's context tool if it has one, or by opening the `.malloy` file directly. Never guess field or view names.
+2. SCAFFOLD the package (template below).
+3. WRITE THE QUERIES with `skill:malloy-html-data-app-runtime`. Validate each before pasting it into the page, using whatever query tool your environment provides or a running Publisher (see `skill:malloy-html-data-app-runtime`). Malloy syntax questions go to `skill:malloy-queries`.
+4. CHOOSE CHARTS with `skill:malloy-charts` when rendering through `<malloy-render>`; otherwise it is your own chart library drawing the returned rows. Vendor any chart library into `public/` and load it locally, not from a CDN. Two reasons: embedded author JavaScript runs with the viewing user's data authority, and a blocked CDN (agent sandboxes and many corporate networks block them) is easy to miss, because the script never runs and the charts come up empty. The `html-data-app` and `storefront` examples both ship their chart library in `public/vendor/` and load it from `public/index.html` as `./vendor/chart.umd.js`. Copy that, but resolve the path against the page's own directory: a page in a subdirectory (`public/reports/index.html`) needs `../vendor/chart.umd.js`. A wrong relative path 404s and leaves the charts blank, which is the failure you are trying to avoid.
+5. EMBED (optional) with `skill:malloy-html-data-app-embedding`.
+6. PREVIEW with the local authoring loop (below).
+7. VERIFY before you call it done (see "What 'done' means" below). This step is not optional.
+
+The scaffold in step 2 only proves the wiring. It is the start, not the deliverable. What you ship is a production app that meets the recipe below.
+
+## What "done" means (production recipe)
+
+A data app you can defend has all of these. Build to this list, not to the scaffold.
+
+- **Real names, never guessed.** Every source, view, and field name comes from the model you read in step 1. A name you derived or assumed is a bug waiting to surface as an empty tile.
+- **DOM-only - never `innerHTML` with interpolated values.** Build every element with `createElement` + `textContent`; do not assign `innerHTML` (or `insertAdjacentHTML`, `document.write`) with any string that contains a model value. Query results render any markup they contain - an XSS vector, and blocked outright under a Trusted-Types CSP. This is a hard build rule, not a lint suggestion: an app that interpolates a model value into `innerHTML` is not done.
+- **Modular, not one inline blob.** Split the page into modules per `skill:malloy-html-data-app-runtime` (pure formatting helpers, a chart layer, your tile/query definitions as data, a thin entry point). One source of truth for each tile's model/source/view, no parallel maps that drift.
+- **Every tile handles loading, empty, and error on its own.** One failing query must not blank the page. (`skill:malloy-html-data-app-runtime`.)
+- **Defensible numbers.** Missing ≠ zero (omit the point, don't plot a fake 0); show the latest non-null value for "current"; guard division with `nullif`; convert units explicitly. (`skill:malloy-html-data-app-runtime`.)
+- **Visible assumptions.** When you assume something (two sources joined by month, an in-month proxy that differs from a certified definition) or a metric is incomplete, say so *in the app*: a caption, a footnote, a placeholder card with the reason. The non-technical user cannot see your reasoning; bury a caveat and you have misled them. Don't silently drop a metric you couldn't model. Show a placeholder that names what's missing and why.
+- **Looks decent.** Give it real layout, type, and color: a styled card grid with a clear hierarchy, not raw unstyled tables.
+- **Vendored libraries.** Chart and helper libraries live in `public/`, loaded locally (step 4).
+- **Lazy-load below the fold (once there are many tiles).** Don't fire every tile's query on load. `reference/lazy-load.md` is the recipe: `IntersectionObserver` (rootMargin ~240px) + a small concurrency cap + reserve each tile's height so lazy tiles don't reflow. Includes the verification trap - on a short/tall-default viewport all tiles intersect at once and you get a false "everything deferred" pass, so test on a deliberately small viewport.
+
+### Verify before you call it done
+
+You are building for someone who cannot tell a correct dashboard from a broken one. Verification is your job, not theirs.
+
+- **Validate every query against the model before wiring it in** (step 3): confirm it compiles and that the column names match what your render code reads.
+- **Load the finished page and confirm every tile shows real numbers**: not stuck on "Loading…", not an error, not an empty state you didn't intend. In a headless browser, wait on `load` plus a content selector, not network idle (`publisher.js` holds an SSE stream open; see `skill:malloy-html-data-app-runtime`). Don't hand-roll this each time - `reference/verification-harness.md` is a copy-adaptable recipe: a mock `sdk/publisher.js` returning canned rows keyed by `(model, query)`, a `python3 -m http.server` webroot, and Playwright assertions (KPIs non-null, no `.is-error`, no stuck `.kit-skeleton`, a chart/table present). It also documents the false-"stuck skeleton" trap (assert after the mock's async delay, never on `networkidle`).
+- **Unit-test any non-trivial pure logic** (a month-join, a de-cumulation, a unit conversion). Keep that logic in DOM-free helpers so `node --test` can cover it, and run it.
+
+## Minimal scaffold
+
+`publisher.json` at the package root:
+
+```json
+{ "name": "my-package", "version": "0.0.1", "description": "..." }
+```
+
+`public/index.html` is a NEW file you create (make the `public/` directory if it does not exist). Load the runtime root-relative, then query. The examples below use the shipped `html-data-app` package (source `subscriptions`); swap in your own model and a view it defines.
+
+Start with the smallest page that proves the wiring, dumping the rows:
+
+```html
+<!doctype html>
+<title>My dashboard</title>
+<pre id="out"></pre>
+<script src="/sdk/publisher.js"></script>
+<script>
+  Publisher.query("subscriptions.malloy", "run: subscriptions -> plan_mix").then((rows) => {
+    document.getElementById("out").textContent = JSON.stringify(rows, null, 2);
+  });
+</script>
+```
+
+Then render the rows. This page builds a table from whatever columns the view returns, so it does not depend on the exact field names:
+
+```html
+<!doctype html>
+<title>Account mix by plan</title>
+<table id="t"><thead></thead><tbody></tbody></table>
+<script src="/sdk/publisher.js"></script>
+<script>
+  Publisher.query("subscriptions.malloy", "run: subscriptions -> plan_mix").then((rows) => {
+    const t = document.getElementById("t");
+    if (!rows.length) { t.textContent = "No rows."; return; }
+    const cols = Object.keys(rows[0]);
+    const headRow = t.tHead.insertRow();
+    for (const c of cols) {
+      const th = document.createElement("th");
+      th.textContent = c;
+      headRow.appendChild(th);
+    }
+    for (const r of rows) {
+      const tr = t.tBodies[0].insertRow();
+      for (const c of cols) tr.insertCell().textContent = r[c];
+    }
+  });
+</script>
+```
+
+Build row content with `textContent`, not `innerHTML` with model values: an `innerHTML` table renders any markup a value contains. This is the HTML-output side of the don't-trust-interpolated-values rule that `skill:malloy-html-data-app-runtime` applies to Malloy query strings.
+
+Two invariants break a page most often:
+
+- **The file must live under `public/`.** Publisher serves only `public/`, so a page written anywhere else (for example `/tmp`) is never reachable at `/environments/<env>/packages/<pkg>/<file>`.
+- **The script src must be the root-relative `/sdk/publisher.js`**, not a relative path.
+
+A third gotcha: the first argument to `Publisher.query` is the model FILE path (`"subscriptions.malloy"`), not the source name.
+
+## Authoring loop and publishing
+
+Authoring happens locally, then you publish. These are two stages.
+
+### Author locally (with live reload)
+
+Run a local Publisher from the directory that holds your `publisher.config.json` and package folder(s):
+
+```sh
+npx @malloy-publisher/server --server_root . --port 4000 --watch-env <env>
+```
+
+`--watch-env <env>` (or `PUBLISHER_WATCH=<env>`) mounts that environment's local-dir packages in place (a symlink, not a copy) and watches them: editing a `.malloy` recompiles the package, and editing a `public/` file live-reloads any open page over an SSE stream. Nothing to wire in the page. The app is served at `http://localhost:4000/environments/<env>/packages/<pkg>/index.html`.
+
+`publisher.config.json` (at `--server_root`) declares the environment, its packages, and its connections:
+
+```json
+{
+  "frozenConfig": false,
+  "environments": [
+    {
+      "name": "<env>",
+      "packages": [{ "name": "<pkg>", "location": "./<pkg>" }],
+      "connections": []
+    }
+  ]
+}
+```
+
+A local package uses a filesystem `location` (`"./<pkg>"`, relative to the directory holding `publisher.config.json`); a remote one uses a GitHub `tree` URL. If one model in the package fails to compile, the **whole package** fails to load, so a stray notebook/model error blanks every tile. (Common one: a `.malloynb` whose cells each `import "x.malloy"`, the notebook compiles as one batch, so the repeated import errors `Cannot redefine 'x'`. Import once in the first cell.)
+
+### Publishing
+
+Publishing an app is publishing its package: get the package into publishable shape and hand it to your host's publishing workflow. A deployed package serves its `public/` app the same way a local one does, at `/environments/<env>/packages/<pkg>/<file>`. A deployed environment has no `--watch-env` live reload, so the loop there is author, publish, then view.

package/skills/malloy-html-data-apps/reference/lazy-load.md

@@ -0,0 +1,84 @@
+# Lazy-Loading Tiles Below the Fold
+
+> Defer off-screen tile queries until they scroll into view, so a dashboard with many tiles doesn't fire every query on load. Referenced from `SKILL.md`. Add this once a page has enough tiles that loading them all at once is wasteful or slow.
+
+Three parts, all required — drop any one and you get a subtle bug rather than an obvious one:
+
+1. **`IntersectionObserver`** with a `rootMargin` (~240px) so a tile starts loading just *before* it scrolls into view, not the instant it appears.
+2. **A small concurrency cap** so a fast scroll to the bottom doesn't fire twenty queries at once.
+3. **Reserve each tile's height** while its skeleton shows, so lazy-loaded tiles don't reflow the page and retrigger the observer.
+
+```js
+// lazy.js — observe tiles, run each tile's query once, capped concurrency.
+const MARGIN = "240px";  // start loading before the tile is visible
+const MAX_INFLIGHT = 3;  // cap concurrent queries
+
+let inflight = 0;
+const queue = [];
+
+function pump() {
+  while (inflight < MAX_INFLIGHT && queue.length) {
+    const run = queue.shift();
+    inflight++;
+    run().finally(() => { inflight--; pump(); });
+  }
+}
+
+// loadTile(el) runs the tile's query (from tiles.js) and renders it; returns a Promise.
+export function lazyLoad(tileEls, loadTile) {
+  const io = new IntersectionObserver((entries, obs) => {
+    for (const e of entries) {
+      if (!e.isIntersecting) continue;
+      const el = e.target;
+      obs.unobserve(el);                 // load once; never re-fire
+      queue.push(() => loadTile(el));
+      pump();
+    }
+  }, { rootMargin: MARGIN });
+
+  for (const el of tileEls) {
+    reserveHeight(el);                   // see below — prevents reflow loops
+    io.observe(el);
+  }
+}
+
+// Reserve space before the query resolves. Without this, an empty tile has ~0
+// height, so many tiles intersect at once (false "all visible"), and when each
+// resolves the page reflows and the observer re-fires. Give the skeleton a real
+// min-height matched to the rendered tile.
+function reserveHeight(el) {
+  if (!el.style.minHeight) el.style.minHeight = "220px"; // match your tile height
+}
+```
+
+## Verification gotcha — test on a *small* viewport
+
+On a short page (or a tall default headless viewport like 1280×720+), every tile intersects the `rootMargin` box at once, so *nothing* actually defers — and your test reports a false "everything lazy-loaded correctly" pass while the feature does nothing. **Verify with a deliberately small viewport** so tiles genuinely start off-screen:
+
+```js
+// In the Playwright harness (see reference/verification-harness.md):
+await page.setViewportSize({ width: 480, height: 520 }); // force real off-screen tiles
+
+const total = (await page.$$(".tile")).length;
+const loaded = () => page.$$eval(".tile .value", (e) => e.length);
+
+// Let the top tiles settle, THEN measure. Don't sample right after the first
+// value: with MAX_INFLIGHT capped, an eager (broken) page and a correct page
+// both show only ~cap tiles at that instant — the guard wouldn't fire. Wait for
+// the loaded count to go quiet, then require that NOT ALL tiles loaded.
+await page.waitForFunction(() => document.querySelector(".tile .value"));
+let prev = -1, settled = await loaded();
+while (settled !== prev) { prev = settled; await page.waitForTimeout(150); settled = await loaded(); }
+if (settled >= total) throw new Error("nothing deferred — viewport too tall to test lazy-load");
+
+// Scroll INCREMENTALLY, one viewport at a time. A single jump to the bottom
+// skips past mid-page tiles — IntersectionObserver never fires for them, their
+// queries never run, and the final wait times out even though lazy-load works.
+const vh = 520;
+const pageH = await page.evaluate(() => document.body.scrollHeight);
+for (let y = 0; y <= pageH; y += vh) {
+  await page.evaluate((yy) => window.scrollTo(0, yy), y);
+  await page.waitForTimeout(150); // let observers fire + queries resolve
+}
+await page.waitForFunction((n) => document.querySelectorAll(".tile .value").length >= n, total);
+```

package/skills/malloy-html-data-apps/reference/verification-harness.md

@@ -0,0 +1,121 @@
+# Headless Verification Harness
+
+> Scaffolding to verify a finished data app end-to-end without a live warehouse. Referenced from `SKILL.md` ("Verify before you call it done"). You are building for someone who cannot tell a correct dashboard from a broken one — this harness is how you check, so they don't have to.
+
+The app talks to the world through exactly one seam: `window.Publisher.query(modelPath, malloy)`. Mock that seam and you can load the real page in a real browser with canned data, then assert on what actually rendered.
+
+## 1. Mock `sdk/publisher.js`
+
+Serve a stand-in at the same root-relative path the page loads (`/sdk/publisher.js`). It returns canned rows keyed by `(modelPath, query)`, and — this is the part that bites — it reproduces the **async delay** of the real runtime, so your assertions exercise the loading→loaded transition instead of racing a synchronous stub.
+
+```js
+// mock/sdk/publisher.js — served at /sdk/publisher.js during verification
+(function () {
+  // Key canned data by "modelPath::query" (exact strings from tiles.js).
+  const FIXTURES = {
+    "carriers.malloy::run: carriers -> kpis": [{ total: 1234, active: 1180 }],
+    "carriers.malloy::run: carriers -> by_letter": [
+      { letter: "A", n: 12 }, { letter: "B", n: 7 },
+    ],
+    // ...one entry per (model, query) your tiles.js declares
+  };
+  const DELAY_MS = 40; // > 0 on purpose: mimic the real async round-trip
+
+  function resolve(modelPath, malloy) {
+    const rows = FIXTURES[`${modelPath}::${malloy}`];
+    // Unknown key = test bug (query string drifted). Fail loudly, don't return [].
+    if (!rows) return Promise.reject(new Error(`No fixture for ${modelPath}::${malloy}`));
+    return new Promise((r) => setTimeout(() => r(rows.map((x) => ({ ...x }))), DELAY_MS));
+  }
+  window.Publisher = {
+    query: resolve,
+    // Placeholder shape ONLY. The real queryFull returns a Malloy result
+    // *envelope* handed to `<malloy-render>` el.result (see
+    // skill:malloy-html-data-app-runtime), NOT { data: rows }. If any tile renders
+    // via <malloy-render>, make this fixture a real envelope or that tile breaks.
+    queryFull: (m, q) => resolve(m, q).then((rows) => ({ data: rows })),
+    setToken() {},
+  };
+})();
+```
+
+Point the harness at the mock by serving it *over* the real path. Copy `public/` and the mock into a webroot so `/sdk/publisher.js` resolves to the mock:
+
+```sh
+webroot=$(mktemp -d)
+cp -r public/* "$webroot"/
+mkdir -p "$webroot/sdk" && cp mock/sdk/publisher.js "$webroot/sdk/publisher.js"
+python3 -m http.server 4173 --directory "$webroot" &
+server=$!
+trap 'kill "$server" 2>/dev/null; rm -rf "$webroot"' EXIT   # always tear the server down
+```
+
+## 2. Drive it with Playwright and assert on the rendered DOM
+
+```js
+// verify.mjs — node verify.mjs   (assumes the server above is on :4173)
+import { chromium } from "playwright";
+
+const browser = await chromium.launch();
+const page = await browser.newPage();
+const errors = [];
+page.on("pageerror", (e) => errors.push(e.message));
+page.on("console", (m) => m.type() === "error" && errors.push(m.text()));
+
+await page.goto("http://localhost:4173/index.html", { waitUntil: "load" });
+// publisher.js holds an SSE stream open (even with watch off), so networkidle NEVER fires.
+// Wait on CONTENT — and for ALL tiles to RESOLVE, not just the first to appear.
+// Asserting after only the first .value renders races the others and yields a
+// false "stuck skeleton" / "empty value" FAIL. "Resolved" = every tile has a
+// value and no skeleton remains. This single wait subsumes the DELAY_MS delay
+// AND the stuck-skeleton check; if it times out, a tile really is stuck.
+// NOTE: assumes a NON-lazy page — every tile loads on open. For a lazy-loaded
+// page (reference/lazy-load.md) below-fold tiles legitimately keep their
+// skeletons until scrolled in, so this wait would (correctly) time out. Test a
+// lazy page with the scroll-and-assert loop in lazy-load.md instead.
+await page.waitForFunction(() => {
+  const tiles = [...document.querySelectorAll(".tile")];
+  // Assert PER TILE, not global counts. `.tile .value` count >= tile count
+  // false-FAILs a chart/table tile (it has no .value) and false-PASSes when one
+  // multi-.value KPI tile inflates the total enough to mask a silently-empty
+  // sibling. "Resolved" = each tile shows its OWN content: a value, a chart, a
+  // table row, or its error state.
+  return tiles.length > 0 &&
+    document.querySelectorAll(".kit-skeleton").length === 0 &&
+    tiles.every((t) => t.querySelector(".value, canvas, table tbody tr, .is-error"));
+}, null, { timeout: 5000 });
+
+// Per tile: flag any tile that is content-empty (nothing rendered at all) or
+// whose KPI values are blank/"NaN". A global .value sweep would let a valueless
+// tile vanish; walking tiles keeps every one accountable.
+const tileReport = await page.$$eval(".tile", (tiles) =>
+  tiles.map((t, i) => {
+    const vals = [...t.querySelectorAll(".value")].map((e) => e.textContent.trim());
+    return {
+      i,
+      empty: !t.querySelector(".value, canvas, table tbody tr, .is-error"),
+      errored: !!t.querySelector(".is-error"),
+      badVals: vals.filter((v) => !v || /^(loading|nan|undefined|null)$/i.test(v)),
+    };
+  }));
+
+const problems = [];
+const emptyTiles = tileReport.filter((t) => t.empty);
+const badValueTiles = tileReport.filter((t) => t.badVals.length);
+const errorTiles = tileReport.filter((t) => t.errored);
+if (emptyTiles.length) problems.push(`empty tiles (no content): ${emptyTiles.map((t) => t.i)}`);
+if (badValueTiles.length) problems.push(`blank/NaN values: ${JSON.stringify(badValueTiles)}`);
+if (errorTiles.length) problems.push(`${errorTiles.length} tile(s) in error state: ${errorTiles.map((t) => t.i)}`);
+if (errors.length) problems.push(`console/page errors: ${errors.join(" | ")}`);
+
+await browser.close();
+if (problems.length) { console.error("FAIL:\n- " + problems.join("\n- ")); process.exit(1); }
+console.log(`OK — all ${tileReport.length} tiles rendered content`);
+```
+
+## Gotchas (each cost a real debugging cycle)
+
+- **Wait out the mock's async delay before asserting.** Asserting immediately after `load` reads the skeleton, not the resolved tile, and reports a false "stuck skeleton." Wait until every tile shows its own resolved content (the per-tile `waitForFunction` above), never on `networkidle` — `publisher.js` keeps the live-reload SSE stream open, so the page never reaches network idle.
+- **A missing fixture is a test bug, not empty data.** Reject on an unknown `(model, query)` key so a drifted query string fails loudly, instead of returning `[]` and masquerading as a real empty state.
+- **Assert on the rendered DOM, not on `Publisher.query` return values.** The bugs live in the render path (NaN formatting, wrong column read, `|| 0` faking a zero). Reading the query result back proves nothing the model didn't already prove.
+- Match the selectors (`.tile .value`, `.is-error`, `.kit-skeleton`, `canvas`) to whatever your app actually emits.

package/skills/malloy-lookml-review/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: malloy-lookml-review
+description: Analyze LookML files as prior art for Malloy modeling. Used during Step 1 (DISCOVER) when .lkml files are present. Coordinates reference files that extract business logic, relationships, and curation decisions. Works with or without a database connection.
+---
+
+# LookML Review
+
+> **Purpose:** Evaluate a LookML project as prior art for building a Malloy semantic model. This skill coordinates the LookML adapter. The implementation lives in reference files under `reference/`.
+
+> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.
+
+> **This is NOT a blind conversion.** Each LookML pattern is evaluated for quality and relevance to Malloy. Bad practices, Looker-specific UI patterns, and performance-only constructs are identified and skipped.
+
+## When to Use
+
+- **Auto-detected:** The agent finds `.lkml` files during Step 1 (DISCOVER) and the user confirms they should be used as prior art.
+- **Explicitly requested:** The user says "model from LookML", "convert LookML", or provides a path to LookML files.
+
+## Two Modes
+
+| Mode | When | Behavior |
+|------|------|----------|
+| **LookML + live data** | A connection is configured and you can query the data | LookML provides prior art; the data validates it. Full data-driven proposals. |
+| **LookML only** | No connection, or queries return nothing | LookML provides all context. Proposals flagged as **unvalidated**. |
+
+If in LookML-only mode, warn the user: "No database connection found. I'll use LookML as the sole source of context, but proposals cannot be validated against live data."
+
+## Numeric Parity Validation (preflight before you trust the Looker path)
+
+To prove the Malloy numbers match Looker, there are two channels, and the "obvious" one fails silently more often than you'd expect.
+
+**Preflight the Looker-API path before attempting it.** Running the original explore through the Looker API only works if the API service account **satisfies that explore's `required_access_grants`**. A service account that doesn't (e.g. its `org_id` user attribute is empty/`NULL`, or an `*_user_id` attribute the grant keys on is unset) gets a **404 on every restricted explore**, indistinguishable at a glance from "explore not found", and cannot self-provision without `administer`/`sudo`. So before you build a parity harness on the Looker API:
+
+1. Identify the target explore's `required_access_grants` and the user attributes they key on.
+2. Verify the service account actually has non-empty values for those attributes. If it doesn't, the Looker path is a dead end: don't spend time discovering that through 404s.
+
+**SQL-level parity against the same warehouse is a first-class fallback, not a consolation prize.** When the Looker path is blocked (or just as the primary method), validate by running equivalent SQL directly against the **same warehouse** the LookML explore reads and comparing to the Malloy result (`execute_query`). This is what actually validates the numbers in practice: reach for it first if access grants are in doubt.
+
+## Reference Files
+
+Each reference file is loaded by the workflow phase that needs it (via dispatch tables in each phase skill). You do not need to read them all at once.
+
+| Reference File | Phase | What It Does |
+|------------|-------|-------------|
+| `reference/discover.md` | Step 1 (DISCOVER) | Inventory .lkml files, extract source candidates, capture prior-art notes |
+| `reference/propose-fields.md` | Step 4 (DEFINE) | Extract field proposals from .lkml views |
+| `reference/build-derived-tables.md` | Step 5 (BUILD) | Classify and convert LookML derived tables |
+| `reference/build-unnest.md` | Step 5 (BUILD) | Convert UNNEST joins and struct field access |
+| `reference/curate-visibility.md` | Step 8 (CURATE) | Map LookML visibility mechanisms to Malloy access modifiers |
+| `reference/document.md` | Step 9 (DOCUMENT) | Extract LookML descriptions as `#(doc)` tag seeds |
+| `reference/review-coverage.md` | Step 7 (REVIEW) | Compare Malloy model against LookML: source, field, and join coverage with rationale for gaps |
+
+### Shared Reference
+
+`reference/_concepts.md` is the LookML to Malloy concept mapping table. Referenced by `propose-fields.md` and `build-derived-tables.md` for type mapping and syntax translation.
+
+## What LookML Provides
+
+- Field names, descriptions, and business logic (accelerates Step 4)
+- Join relationships and cardinality (accelerates Step 3)
+- Field visibility decisions: `hidden: yes`, `fields` exclusions, `required_access_grants` (accelerates Step 8)
+- Organizational structure via `group_label` and `view_label` (informs source design)
+- Derived table intent: NDTs to computed sources, PDTs to evaluate
+
+## What to Skip
+
+- Looker UI patterns (`link:`, `drill_fields:`, `html:`, `action:`)
+- Liquid templating (`{% %}`, `{{ }}`): strip and note intent
+- `parameter:` definitions: note the business intent, don't replicate
+- PDT optimization (`partition_keys:`, `datagroup_trigger:`, `increment_key:`)
+- Dashboard files (`.dashboard.lookml`)
+- `sql_always_where:`: document as context, don't bake into Malloy
+
+## What to Flag for User Decision
+
+- Complex SQL dimensions (50+ lines): default is simplify or push upstream
+- Derived tables: classify as performance-only, transformation, or aggregation
+- Refinement structure (`+view`): consolidate vs. preserve layering via `extend`
+- Synthetic primary keys: ask about actual grain