@zestl/zvolv-widgets 0.0.1 → 0.0.3

package/CHANGELOG.md

@@ -0,0 +1,62 @@
+# Changelog
+
+## 0.0.3
+
+These are all client-side enhancements that run after the query; none changes
+what is queried. Existing configs give byte-identical queries, rows and default
+markup, which `test/compat.mjs` checks against snapshots captured from 0.0.2.
+There is nothing for ZAI-Dashboard or the server to mirror.
+
+### Added
+- **KPI card layout.**
+  - `config.layout: "card"` (or `cardConfig.layout`) renders every item inside one card, through a `"KPI_CARD:card"` template override or the default `KpiCard`.
+  - `KpiWidget` takes `renderCard` (or `render`), which receives `{ vms, byKey, roles, state, tokens, config, onItemClick }`. Both are used only in card mode, so a generic `render` a host passes to every widget leaves KPIs unchanged.
+  - `KpiContentConfig.role` (`"main" | "sub" | "chip" | "series"`) places an item in the card; `resolveKpiRoles` does this sorting.
+  - `KpiSparkline` gains `width`, `height`, `area` and `scale: "minmax" | "zero"`.
+- **`unpivot`.** On `useWidgetData` and every data widget. It turns row 0's columns into one `{ label, value }` row per column, in the given order. A missing column gives 0. `extra` copies more columns onto the output rows. It runs after postOperations. With no declared series, x and y become `labelField` and `valueField`. Pure helper: `unpivotRows`.
+- **`ranking.order: "none"`.** `RankingBarWidget` keeps the incoming row order.
+- **`FunnelWidget` (type `FUNNEL`).** Each stage resolves to `{ label, value, ratioToFirst, stepPct, isFirst, index, raw }`, with `onItemClick(row)` for drill-down. `render` gets `{ rows, first, tokens, config, toLabel, onItemClick }`. Pure helper: `computeFunnelRows`.
+- **`categoryGap`.** On `useWidgetData` and every data widget. It gives one row per category in the given order, or in the form's dropdown order with `"fromForm"`. Missing buckets get 0 for every series valueField, and values that are not listed are kept at the end. It runs after postOperations and widget filters, and before local search, column filters and sort. Pure helpers: `fillCategoryGaps`, `elementOptionValues`.
+- **`FilterPanelWidget` options from a data source.**
+  - `filters[].optionsSource = { dataSource, labelField, valueField, sort?, allLabel? }` fetches the options in one request: a distinct-values aggregation over the whole index, with the source's filters applied server-side (`optionsDataSource`).
+  - Values restored from URL params match numeric options (`matchFilterOption`).
+  - `onFiltersChange(WidgetFilter[])` emits `{ field, operator: "IN", value: [v] }`, or nothing for "All".
+  - The `renderFilter` slot receives `{ control, options, value, setValue, loading, tokens }`.
+  - Pure helpers: `filterPanelToWidgetFilters`, `rowsToFilterOptions`.
+- **New tests:** `kpiCard`, `unpivot`, `funnel`, `categoryGap`, `filterOptions`. `compat` now also snapshots rows and default markup. A missing snapshot now fails the test instead of being regenerated.
+
+## 0.0.2
+
+Every new feature is opt-in. Configs that existed before this release produce
+byte-identical queries (`test/compat.mjs`). The query-generator changes that
+ZAI-Dashboard and the server must mirror are listed in `MIRROR_NOTES.md`.
+
+### Fixed
+- **A pivot output name is no longer used as the ES field name.** Pivot rows, columns and summaries now group and aggregate on `masterField`. `title` only names the output key. Before, `groupBy(f, "alias")` queried a field called `alias` and returned no buckets. Configs where `title === masterField` are unchanged.
+  - This also fixes the library's own `groupBy(taskField("Status"))`, which used to query `Status.keyword` instead of `status.keyword`.
+- **`fillDateGaps` no longer overwrites rows in the same period.** Day-level rows shown weekly or monthly are now summed per series `valueField`. The first row's other fields are kept. `merge: "last"` restores the old behaviour.
+- **Changing `dateGap` now refetches.** Before, `useWidgetData` kept rows normalised with the previous `dateGap`.
+
+### Added
+- **Calendar-period grouping.** `groupBy(f, title, { interval, timeZone })` groups by `"day" | "week" | "month" | "quarter" | "year"` using an ES composite `date_histogram` source. Keys are epoch ms at the start of each period.
+- **More `dateGap` units.** `DateGapUnit` adds `"quarter"` and `"year"`. Weeks now start on Monday (ISO), consistent with ES `1w`.
+- **Running totals.** `dateGap.cumulative` (`true` or a list of fields) with a `baseline` value, or with `baselineSource` to have the hook fetch the baseline in one extra request. New pure helper: `applyCumulative`.
+- **Time between two dates.** `dateDiff(start, end | "TODAY", unit)` for `aggregate(..., "avg" | "min" | "max" | "sum" | "percentiles")`. It uses a painless script metric, and documents missing either date are excluded.
+- **Percentiles.** `"percentiles"` is now an `AggregationType`, with `{ percents }`. Rows get `<name>_p50`-style keys (`flattenPercentiles`, `percentileKey`).
+- **Several conditions per metric.** `aggregate(f, type, name, { where: [[field, where(...)], ...] })` filters one metric by AND-ed conditions. All filtered metrics still run in one request. `PivotFilter.combine: "AND"` opts stored configs into AND-ing.
+- **Self-fetching KPI sparklines.** `KpiContentConfig.seriesSource = { mode: "timeseries", dataSource, xField, valueField, dateGap }` makes `KpiWidget` fetch the sparkline itself. A new `externalSeriesByIndex` prop is also accepted.
+- **New exports:**
+  - functions: `dateDiff`, `startOfPeriod`, `addPeriod`
+  - types: `DateGapOptions`, `DateInterval`, `DateDiffUnit`, `DateDiffSpec`, `DateDiffField`, `AggregateOptions`, `KpiTimeseriesSource`, `FillDateGapsArgs`
+- **New tests:** `test/compat.mjs`, `test/timeseries.mjs`, `test/datediff.mjs`, `test/summaryFilters.mjs`.
+
+### Fixed during review
+- **Time zones.** `dateGap` now computes periods in the `groupBy(..., { timeZone })` zone, or in `dateGap.timeZone`. Before, it used the browser's zone, so ES buckets landed in the wrong period whenever the two zones differed. `startOfPeriod` and `addPeriod` take an optional zone, and there is a new `parseDateValue` helper.
+- **Running totals without `end`.** `cumulative` now drops rows before `start` even when `end` is missing. Before, they were counted twice: once in the rows and once in the baseline.
+- **Sort order.** AND-filtered metrics keep the summary `bucket_sort`.
+- **KPI sparklines.** A self-fetched sparkline is cleared when its item loses `seriesSource.dataSource` or the items are reordered.
+- **`dateDiff` end values.** `end` now accepts any date token and literal dates (`"2026-03-31"`, ISO date-times). An unknown `{@[…]}` token logs a warning instead of silently being queried as a field.
+
+### Changed
+- **Week periods in `fillDateGaps` start on Monday** instead of following the dayjs locale, which defaulted to Sunday.
+- **Version bumped to 0.0.2.** `package.json` said `0.0.0`.

package/MIRROR_NOTES.md

@@ -0,0 +1,156 @@
+# Mirror notes: what ZAI-Dashboard and the server must copy
+
+`src/runtime/queryGenerator.ts` is a verbatim port of ZAI-Dashboard
+`src/utils/queryGenerator.ts`. Every change in this repo is in a block marked
+`// [zvolv-widgets extension]`. Search for that marker to find each one.
+
+**Compatibility guarantee.** For every config shape that existed before 0.0.2,
+the generated query is byte-identical. `test/compat.mjs` checks 12 such shapes
+against snapshots captured from the old code.
+
+**The one intended exception (item 1).** A pivot entry whose `title` differs
+from its `masterField` now queries `masterField`. Before, it queried a field
+named after the title, which normally does not exist. Saved ZAI-Dashboard
+configs observed so far have `title === masterField`. Before rolling out,
+check task widgets that might have stored `title: "Status"` with
+`masterField: "status"`.
+
+---
+
+## A. ZAI-Dashboard `queryGenerator.ts`: changes to copy
+
+### A1. ES field comes from `masterField`, not `title` (bug fix)
+
+In each function below, `title` stays the aggregation or source **name**, which
+is the output key. Only the **field** changes.
+
+| Function | Before | After |
+| --- | --- | --- |
+| `getCompositeAggregations` | `terms.field` built from `row.title` in all four branches: `resolveFormSearchViewField(row.title)`, `` `${row.title}.mediaKey.keyword` ``, `` `${row.title}.keyword` ``, `row.title` | the same, built from `esField = row.masterField \|\| row.title` |
+| `getCompositeAggregations` | `terms: scriptFields[row.title]` | unchanged (script fields are keyed by header) |
+| `getColumnAggregations` | `terms: { field: col.title }` | `terms: { field: col.masterField \|\| col.title }` |
+| `getSummaryAggregations` | `aggField = sumary.title` (and `.keyword`, `.mediaKey.keyword`, `resolveFormSearchViewField(...)` variants) | the same, built from `esField = sumary.masterField \|\| sumary.title` |
+| `getSummaryAggregations` | `usesRawField = aggField === sumary.title`, `addBucketSort(sumary.title)` | `usesRawField = aggField === esField`, `addBucketSort(esField)` |
+
+### A2. `date_histogram` grouping (`PivotRow.interval`)
+
+In `getCompositeAggregations`, before the existing `FORM_SEARCH` / `FILE_UPLOAD_V2` /
+`.keyword` / raw branches:
+
+```ts
+if (row.interval && (dateTypes.includes(mfType) || isTaskDateField)) {
+  sourceObj[row.title] = { date_histogram: {
+    field: esField,
+    calendar_interval: { day: '1d', week: '1w', month: '1M', quarter: '1q', year: '1y' }[row.interval],
+    time_zone: row.timeZone || timezone,          // module-level Intl zone
+    ...(sortOrder ? { order: sortOrder } : {}),
+  } };
+}
+```
+
+- A task date field is one in `taskDate`, where the row or its field metadata has `isTaskField` or `taskfield` set.
+- When `interval` is set on a non-date field, the generator calls `console.warn` and falls back to the existing branches.
+- Helpers: `getDateHistogramSource`, `HISTOGRAM_INTERVALS`.
+
+### A3. Per-summary extensions: `dateDiff`, `percentiles`, AND-ed filters
+
+**New module state.** `summaryFilterLists: Map<string, any[]>`, which is
+`pivot.filters` grouped by `field` with every entry kept. The original
+`filterMap` keeps only the last entry per field.
+- `generateQuery` resets it next to `currentMappings`.
+- `generateQuery` fills it with `groupPivotFilters(dataSource.pivot.filters)` right after building `filterMap2`.
+
+**Hook.** At the top of the `pivotBody.summary.forEach` loop in
+`getSummaryAggregations`, after `aggDataPoints.summary.push`, add:
+
+```ts
+if (buildExtendedSummary(sumary, summaryAggs, loggedInUser)) return;
+```
+
+`buildExtendedSummary` returns `false` and changes nothing unless one of these holds:
+
+1. **`sumary.dateDiff = { start, end, unit }` is set.** The metric becomes `{ [type]: { script } }`, where `script` is:
+   - with two date fields: `{ source: "if(doc['S'].size()==0 || doc['E'].size()==0) { return null; } return (doc['E'].value.millis - doc['S'].value.millis) / <div>;", lang: 'painless' }`
+   - with a fixed `end`: `{ source: "if(doc['S'].size()==0) { return null; } return (params.now - doc['S'].value.millis) / <div>;", lang: 'painless', params: { now } }`. `end` is fixed when it is either:
+     - a `dateFormulas` token: `now = dayjs(getDate(end)).valueOf()`
+     - a literal date matching `/^\d{4}-\d{2}-\d{2}([T ][\d:.]+(Z|±hh:mm)?)?$/`: `now = dayjs(end).valueOf()`, where a date-only value means local midnight
+   - Any other `end` is a field name. An unknown `{@[…]}` token logs a `console.warn`.
+   - `<div>` is `86400000.0` for days, `3600000.0` for hours, `60000.0` for minutes.
+   - It is always wrapped in a filter: `must` gets `{ exists: { field: S } }`, plus `{ exists: { field: E } }` unless `end` is fixed.
+2. **`sumary.type === 'percentiles'` and `sumary.percents` is set.** The metric gets `percents` (`{ percentiles: { field | script, percents } }`).
+3. **The summary has two or more `pivot.filters` entries, at least one with `combine: 'AND'`.** Every entry is AND-ed:
+   - Each entry is converted with the original `getFilters(fb, 'PIVOT')` call and the same `fb` shape as the original branch.
+   - Operators in `mustNotOp` go into `bool.must_not`; all others go into `bool.must`.
+   - `_ALL` and `MULTIPLE` are skipped, because they only make sense as global filters.
+
+The output nesting is the same as the original per-summary filter branch:
+
+```json
+"<name>": { "filter": { "bool": { "must": [...], "must_not": [...] } }, "aggs": { "<name>": { "<type>": { ... } } } }
+```
+
+or the bare `"<name>": { "<type>": { ... } }` when there are no conditions. A
+summary-only query with a filtered summary therefore still takes the existing
+`dummy_row` composite path. Field resolution for plain-field percentiles and
+AND-filtered metrics (`.keyword`, `FORM_SEARCH` `.view`, `FILE_UPLOAD_V2`
+`.mediaKey.keyword`, text → `value_count`) copies the original loop.
+
+**Sort.** AND-filtered single-value metrics on a raw field get the same
+`<name>_sort` `bucket_sort` as the original `addBucketSort`, so
+`buildExtendedSummary` receives `sortKeys`. Percentiles and `dateDiff` metrics
+get no sort, because they have no single-value field to sort on.
+
+Helpers: `buildExtendedSummary`, `finish`, `getDateDiffScript`, `getDateDiffFixedEnd`,
+`getSummaryFilterClause`, `groupPivotFilters`, `DATE_DIFF_DIVISOR`.
+
+### A4. Optional: `fillDateGaps`, if the host has its own copy
+
+If the host has its own `fillDateGaps`, mirror these changes from `src/transforms/dateGaps.ts`:
+- Rows in the same period are summed, not overwritten.
+- Weeks start on Monday (ISO), which matches ES `1w`.
+- `quarter` and `year` units are supported.
+- Periods are keyed by their start instant.
+- An optional `timeZone` computes periods on that zone's calendar. It must match the `date_histogram` `time_zone`; otherwise bucket keys land in the previous or next period.
+
+---
+
+## B. Server: aggregation response flattener
+
+This repo cannot see the flattener that turns ES `aggregations` into
+`elements` rows, driven by `aggsDataPoints.rows` / `cols` / `summary`. Each
+item below is what the client expects. Verify each one against a live index.
+
+| # | ES response shape | What the client expects in each row | Likely server change |
+| --- | --- | --- | --- |
+| B1 | `date_histogram` composite source: `bucket.key = { "Month": 1767205800000 }` (epoch ms, a number) | `row["Month"] = 1767205800000` | **Probably none.** Composite `terms` keys on date fields are already epoch-ms numbers. Confirm the flattener copies `bucket.key[row]` without assuming the source is `terms`. |
+| B2 | Filter-wrapped script metric: `bucket["Avg Days"] = { doc_count, "Avg Days": { value: 4.5 } }` | `row["Avg Days"] = 4.5` | **None** if filtered summaries already work: the nesting is identical. A bucket where no document has both dates returns `value: null`. Keep that as `null`; the client coerces it to 0 for charts. |
+| B3 | `percentiles`: `{ values: { "50.0": 3, "95.0": 9 } }`, either at the top level (summary only) or under the filter wrapper | Either `row["Days"] = { values: {...} }` or `{ "50.0": 3, ... }`, or flat `row["Days_p50"]` / `row["Days_p95"]` | **Likely needed.** If the flattener reads only `.value`, percentiles come back `undefined`. Pass the `values` object through, or emit `<name>_p<percent>` keys (`99.9` → `_p99_9`). The client accepts all three shapes. |
+| B4 | Summary-only `dateDiff` metric under `dummy_row` | `row["Avg Days to Offer"]` | None. Same path as existing filtered KPI summaries. |
+| B5 | Script metrics in general | n/a | `doc[...].value.millis` is the idiom `getFormula`'s DAYS_DIFF already uses, so it works on the current cluster. On ES 8+ it would need `.toInstant().toEpochMilli()`. |
+
+---
+
+## C. Not query-generator changes, for reference
+
+- **`useWidgetData`:**
+  - New `dateGap.merge`, `cumulative`, `baseline` and `baselineSource` options. `baselineSource` makes one extra request with `dateField < start` and no dashboard date range.
+  - Percentile-row flattening.
+  - `dateGap` content changes now trigger a refetch. Before, the effect ignored them and showed stale rows.
+- **`KpiWidget`:**
+  - Self-fetches `seriesSource: { mode: "timeseries", dataSource, ... }`, with one request per content item.
+  - New `externalSeriesByIndex` prop.
+- **`dataSource.ts`:**
+  - `groupBy(f, title, { interval, timeZone })`.
+  - `dateDiff()`.
+  - `aggregate(f, type, name, { percents, where })`.
+  - `pivot()`, `formSource()` and `taskSource()` move `where` conditions into `pivot.filters` with `combine: "AND"`.
+
+---
+
+## 0.0.3: nothing to mirror
+
+`unpivot`, `categoryGap`, the KPI card layout, `FunnelWidget` and filter
+`optionsSource` all run in the browser after the query. They add no query
+shapes and need no server change. `categoryGap: "fromForm"` reads the form
+through the same `getSingleForm` call the widget's access check already makes.
+

package/README.md

@@ -343,7 +343,7 @@ queried. The query generator reads `dataSource`, so a widget with a perfect
 config and no data source shows "No data source configured".
 
 ```tsx
-import { formSource, taskSource, field, taskField, groupBy, aggregate, derive, where, pivot } from "@zestl/zvolv-widgets";
+import { formSource, taskSource, field, taskField, groupBy, aggregate, dateDiff, derive, where, pivot } from "@zestl/zvolv-widgets";
 
 const region  = field("Region",  "DROP_DOWN");
 const revenue = field("Revenue", "INPUT_CURRENCY");
@@ -394,6 +394,268 @@ field("Created", "INPUT_DATE", { filter: where("DATE_IS", "{@[TODAY]}") })
 defaultWidgetFilters={[{ field: "Status", operator: "IN", value: ["Open"] }]}
 ```
 
+### Output names vs field names
+
+`groupBy(f, title)` and `aggregate(f, type, outputName)` take an output name,
+which is the key the rows come back under. The query always groups and
+aggregates on the field's `masterfield`. So `groupBy(requestDate, "day")`
+queries `Request Date` and returns rows keyed `day`. (Before 0.0.2, the title
+was used as the field name, so an alias returned no buckets.)
+
+### Grouping by calendar period
+
+Pass `interval` to group a date field by calendar period
+(`date_histogram` in Elasticsearch) instead of by exact value:
+
+```tsx
+const requestDate = field("Request Date", "DATE_PICKER");
+const requests    = field("Request ID", "INPUT_TEXT");
+
+const monthly = formSource(1042, [requestDate, requests], {
+  pivot: pivot({
+    rows:    [groupBy(requestDate, "Month", { interval: "month" })], // day | week | month | quarter | year
+    summary: [aggregate(requests, "value_count", "Requests")],
+  }),
+});
+
+<LineChartWidget
+  dataSource={monthly}
+  configurations={{ series: [{ argumentField: "Month", valueField: "Requests" }] }}
+  dateGap={{ unit: "month", start: "2026-01-01", end: "2026-12-31" }}
+/>
+```
+
+- Row keys are epoch ms at the start of each period.
+- Weeks start on Monday (ISO).
+- Period boundaries use the browser's time zone. Pass `{ interval, timeZone: "Asia/Kolkata" }` to fix one. `dateGap` then fills and merges periods in that same zone.
+- `interval` on a non-date field is ignored, with a console warning.
+
+`dateGap` fills empty periods with zero rows. Rows that fall in the same
+period are **summed** (`merge: "sum"`, the default), so day-level data shown
+weekly or monthly no longer loses rows. `merge: "last"` restores the old
+keep-the-last-row behaviour.
+
+### Running totals (level over time)
+
+```tsx
+<LineChartWidget
+  dataSource={monthly}
+  configurations={{ series: [{ argumentField: "Month", valueField: "Requests" }] }}
+  dateGap={{
+    unit: "month", start: "2026-01-01", end: "2026-12-31",
+    cumulative: true,                 // or ["Requests"]
+    baseline: { Requests: 1250 },     // the total before `start`
+  }}
+/>
+```
+
+`cumulative` runs after gap filling, so each point is the level at the end of
+its period. Rows before `start` are dropped even when there is no `end`,
+because the baseline already counts them. You can let the hook fetch the baseline instead. This costs one
+extra request:
+
+```tsx
+dateGap={{
+  unit: "month", start: "2026-01-01", end: "2026-12-31", cumulative: true,
+  baselineSource: {
+    dataSource: formSource(1042, [requestDate, requests], {
+      pivot: pivot({ summary: [aggregate(requests, "value_count", "Requests")] }), // same output name
+    }),
+    dateField: "Request Date",   // the hook adds Request Date < 2026-01-01
+  },
+}}
+```
+
+The baseline query applies the widget's dashboard and widget filters, but not
+the dashboard date range, because that range would exclude everything before
+`start`. An explicit `baseline` wins over the fetched value, field by field.
+
+### KPI sparklines without custom code
+
+A KPI item can fetch its own sparkline:
+
+```tsx
+<KpiWidget
+  dataSource={formSource(1042, [requests], { pivot: pivot({ summary: [aggregate(requests, "value_count", "Total")] }) })}
+  configurations={{ contents: [{
+    title: "Requests", dataPoint: "Total", template: "trend",
+    seriesSource: {
+      mode: "timeseries",
+      dataSource: monthly,            // grouped by a date row with `interval`
+      xField: "Month", valueField: "Requests",
+      dateGap: { unit: "month", start: "2026-01-01", end: "2026-06-30", cumulative: true },
+    },
+  }] }}
+/>
+```
+
+Each such item makes one request through `useWidgetData`, with the widget's
+dashboard filters applied. `externalSeriesByIndex={{ 0: [..] }}` still works
+and takes precedence.
+
+### Time between two dates (avg / min / max / sum / percentiles)
+
+```tsx
+const offerDate = field("Actual Offer Date", "DATE_PICKER");
+const partner   = field("Lead Partner", "DROP_DOWN");
+
+// Overall: one number
+formSource(1042, [requestDate, offerDate], { pivot: pivot({
+  summary: [aggregate(dateDiff(requestDate, offerDate, "days"), "avg", "Avg Days to Offer")],
+}) });
+
+// Per partner, or per month with groupBy(requestDate, "Month", { interval: "month" })
+formSource(1042, [requestDate, offerDate, partner], { pivot: pivot({
+  rows:    [groupBy(partner)],
+  summary: [
+    aggregate(dateDiff(requestDate, offerDate), "avg", "Avg Days to Offer"),
+    aggregate(dateDiff(requestDate, offerDate), "percentiles", "Days", { percents: [50, 90] }),
+    aggregate(dateDiff(requestDate, "TODAY"), "max", "Oldest Open (days)"),
+  ],
+}) });
+```
+
+- The value is `end − start`. The unit is `"days"`, `"hours"` or `"minutes"`, and the result is fractional.
+- Documents missing either date are skipped.
+- `end` can be a fixed end instead of a field: `"TODAY"`, a date token such as `{@[30DAYSAGO]}`, or a literal date (`"2026-03-31"` means local midnight; an ISO date-time is also accepted). Only the start date is then required. Any other string is treated as a field name, and an unknown `{@[…]}` token logs a warning.
+- Percentiles come back as `<name>_p50`, `<name>_p90` (`99.9` becomes `_p99_9`). `<name>` itself holds the median when 50 was requested.
+
+### Filtered metrics: many KPIs, one request
+
+```tsx
+const amount = field("Amount", "INPUT_CURRENCY");
+const status = field("Status", "DROP_DOWN");
+const invoiceDate = field("Invoice Date", "DATE_PICKER");
+
+formSource(1042, [amount, status, invoiceDate], { pivot: pivot({
+  summary: [
+    aggregate(amount, "sum", "Billed", { where: [
+      [status, where("IN", "Invoiced", "Paid")],
+      [invoiceDate, where("BETWEEN", "2026-04-01", "2026-06-30")],  // AND-ed
+    ] }),
+    aggregate(amount, "sum", "Pipeline", { where: [[status, where("IN", "Quoted")]] }),
+    aggregate(amount, "sum", "Total"),
+  ],
+}) });
+```
+
+Each metric is filtered on its own, and all of them run in **one** request. A
+KPI card with 20 numbers on one form is one call. `pivot()`, `formSource()` and
+`taskSource()` move the `where` conditions into `pivot.filters`, marked
+`combine: "AND"`. Stored `pivot.filters` entries without that marker keep the
+original rule: the last entry per metric wins.
+
+### Reshaping rows after the query (0.0.3)
+
+These run in the browser after the query returns. They never change what is
+queried.
+
+**One wide row → one row per category (`unpivot`).** A multi-number source
+returns one row such as `{ Longlist: 4, Contacted: 4, … }`. Turn it into one
+row per column, in your order:
+
+```tsx
+const STAGES = ["Longlist", "Contacted", "Interviewed", "Offered", "Placed"];
+
+<RankingBarWidget
+  dataSource={FUNNEL_ONE}
+  unpivot={{ columns: STAGES, extra: { "Contacted %": "step" } }}
+  configurations={{ ranking: { order: "none" } }}   // keep stage order; ratio = value / max
+/>
+```
+
+- Output rows are `{ label, value }`. Rename the keys with `labelField` / `valueField`.
+- A missing column gives 0.
+- `unpivot` runs after `postOperations`, so a `derive()` column can be a stage.
+- With no declared `series`, x and y become `labelField` and `valueField`.
+- `extra` copies a source column onto the stage row whose name it starts with. `"Contacted %"` lands on the "Contacted" row as `step`. A source column that matches no stage is copied onto every row.
+
+**All categories, in dropdown order (`categoryGap`).** ES returns a bucket only
+for values that have data. `categoryGap` adds a zero row for every missing
+category and orders the rows:
+
+```tsx
+<BarChartWidget
+  dataSource={MINI_FUNNEL}
+  categoryGap={{ field: "Pipeline Stage", values: "fromForm" }}   // or an explicit list
+  widgetFilters={[{ field: "Assignment", operator: "IN", value: [title] }]}
+  render={({ rows }) => <MiniBars rows={rows} />}
+/>
+```
+
+- `field` is the chart's argumentField.
+- `"fromForm"` reads that field's dropdown options from the form schema, once per form; it reuses the widget's form lookup.
+- Every series `valueField` is 0 in the added rows.
+- Values that are not in the list are kept, after the listed ones.
+- It runs after `postOperations` and widget filters, and before local search, column filters and sort.
+
+### Funnel
+
+```tsx
+<FunnelWidget
+  dataSource={FUNNEL_ONE}
+  unpivot={{ columns: STAGES }}
+  onItemClick={(row) => setStage(row.label)}
+/>
+```
+
+Each stage resolves to `{ label, value, ratioToFirst, stepPct, isFirst, index, raw }`:
+- `stepPct` is value ÷ previous stage × 100. It is `null` for the first stage, and after a stage of 0.
+- `ratioToFirst` is value ÷ first stage. It is 0 when the first stage is 0.
+
+Stages are never re-sorted. `render` gets `{ rows, first, tokens, config, toLabel, onItemClick }`.
+
+### One KPI card from several numbers
+
+```tsx
+<KpiWidget dataSource={cardSource} configurations={{ layout: "card", contents: [
+  { dataPoint: "value",  role: "main", title: "Open mandates", template: "chart", seriesPoints: P1toP12 },
+  { dataPoint: "total",  role: "sub",  title: "total" },
+  { dataPoint: "atRisk", role: "sub",  title: "at risk" },
+  { dataPoint: "P12",    role: "chip", template: "trend", deltaSource: "datapoint", deltaPoint: "P8",
+    deltaMode: "absolute", subtitle: "vs 4 wk" },
+]}} />
+```
+
+The default card puts the items in slots by `role`:
+- `main` is the big value. Without a `main` role, it is the first item with no role.
+- `sub` items are small inline values under it.
+- `chip` is the change badge (its `delta`).
+- `series` supplies the sparkline. Without it, the main item's series is used.
+
+Card mode turns on only with `layout: "card"`, set either at the config root
+or in `cardConfig`, where platform KPI payloads keep their items. To build the
+card yourself, pass `renderCard={({ vms, byKey, roles, state, tokens }) => …}`;
+`render` also works, but likewise only in card mode, so a `render` a host
+passes to every widget never changes a KPI. `byKey` maps each `dataPoint` to
+its resolved item; the first item wins when two share one. You can also
+register a `"KPI_CARD:card"` template. `KpiSparkline` accepts
+`width`, `height`, `area` (fill under the line) and `scale: "minmax" | "zero"`.
+
+### Filter options from another form
+
+```tsx
+<FilterPanelWidget
+  configurations={{ filters: [{
+    key: "scope", field: "Assignment", type: "select",
+    optionsSource: {
+      dataSource: OPEN_MANDATES,              // e.g. a formSource on Assignment with Status IN Open
+      labelField: "Assignment Title", valueField: "Assignment Title",
+      sort: "asc", allLabel: "Firm-wide",
+    },
+  }] }}
+  onFiltersChange={(filters) => setScopeFilters(filters)}   // → other widgets' widgetFilters
+  renderFilter={({ options, value, setValue, loading }) => <MySelect … />}
+/>
+```
+
+- **Fetch:** the options come from one request: a distinct-values (composite terms) aggregation over the **whole index**, with the source's own filters applied by Elasticsearch. It is not built from a page of rows. The cap is the composite size (50,000 distinct values, or `displayRowLimit`). Declare `labelField` and `valueField` in the source's `fields` so text fields aggregate on `.keyword`. A source that already has a `pivot` is used as is. The panel refetches only when `optionsSource.dataSource` changes.
+- **Where the filter runs:** the emitted filter is applied in the target widget's ES query, so over the whole index. Only automation-row and legacy widget-id sources, which have no query, filter their loaded rows.
+- **URL values:** values restored from URL params (strings) match numeric options, and the emitted filter carries the option's own typed value.
+- **Output:** choosing a value emits `[{ field, operator: "IN", value: [v] }]`; "All" emits `[]`.
+- **Lookup targets:** when `field` is a FORM_SEARCH lookup on the target form, the filter matches the lookup's `.view` value. So `valueField` must be the field named as the lookup's **ViewLabel**.
+- **Without a slot:** the panel renders a plain `<select>` for these controls.
+
 ---
 
 ## 5. Controlling the UI
@@ -449,9 +711,10 @@ Never re-derive values from `vm.raw`.
 
 | Family | Components | Type strings | `render` receives |
 | --- | --- | --- | --- |
-| KPI | `KpiWidget` | `KPI` `KPI_CARD` `KPI_CARD_V2` | templates get `{ vm, index }` |
+| KPI | `KpiWidget` | `KPI` `KPI_CARD` `KPI_CARD_V2` | templates get `{ vm, index }`; `render` / `"KPI_CARD:card"` get `{ vms, byKey, roles, state, tokens, config }` |
 | Charts | `BarChartWidget` `LineChartWidget` `AreaChartWidget` `PieChartWidget` `DoughnutChartWidget` `ScatterChartWidget` | `BAR(_CHART)` `LINE(_CHART)` `AREA_CHART` `PIE(_CHART)` `DOUGHNUT(_CHART)` `SCATTER_CHART` | `{ rows, xKey, series, colorFor, toLabel, template, config, tokens, meta }` |
-| Ranking | `RankingBarWidget` | `RANKING_BAR` | `{ rows: {rank,label,value,ratio,raw}[], max, … }` |
+| Ranking | `RankingBarWidget` | `RANKING_BAR` | `{ rows: {rank,label,value,ratio,raw}[], max, … }` (`ranking.order: "none"` keeps row order) |
+| Funnel | `FunnelWidget` | `FUNNEL` | `{ rows: {label,value,ratioToFirst,stepPct,isFirst,index,raw}[], first, tokens, config, toLabel, onItemClick }` |
 | Gauge | `GaugeMeterWidget` | `GAUGE_METER` | `{ value, min, max, ratio, color, band, thresholds, rows }` |
 | Heat map | `HeatMapWidget` | `HEAT_MAP` | `{ cells: {zone,count,color,band}[], grid }` |
 | Tables (columns from `config.columns[]` as `{ label, dataPoint, type }`, the stored shape) | `DashboardTableWidget` `DatabaseTableWidget` `TaskTrackerTableWidget` `ChildTableWidget` `PivotGridWidget` | `TABLE` `DATABASE_TABLE` `TASK_TRACKER_TABLE` `CHILD_TABLE` `CHILD_DATABASE_TABLE` `PIVOTGRID` | `{ rows, columns, renderCell, toLabel, interactions, headerControls, getColumnOptions, enableSort, showColumnFilter, mode }` (+ `rowFields`, `columnFields`, `valueFields` for pivot) |
@@ -470,9 +733,10 @@ and HTML sanitization stay in the host app. The package never calls
 `dangerouslySetInnerHTML`.
 
 **`FilterPanelWidget` vs widget filters.** `FilterPanelWidget` is a
-*dashboard-level* control. It is controlled (`value` / `onChange`) and issues no
-queries. Feed its values into other widgets' `dashFilters`. A widget filter
-(§3) narrows only its own widget.
+*dashboard-level* control. It is controlled (`value` / `onChange`), and it
+issues no queries except one per `optionsSource`. Feed its values into other
+widgets' `dashFilters`, or pass `onFiltersChange` output to their
+`widgetFilters`. A widget filter (§3) narrows only its own widget.
 
 ---
 
@@ -650,6 +914,7 @@ writes widget code. Without it, Claude tends to write its own data fetching.
 - **Tables fetch up to `rowLimit` rows (default 1000).** There is no server pagination yet. Server-side search, filters and sort run over the full dataset, but only the first `rowLimit` matches are shown.
 - **Column-filter options** come from the rows loaded so far. They accumulate across fetches, so options do not vanish once one is ticked. They are not a full distinct-values query.
 - **Widget filters are AND-ed.** For OR logic, pass a host ZFilters payload through `localFilters`.
+- **0.0.2 query features need server and host support.** `interval` grouping, `dateDiff` metrics, percentiles and AND-ed per-metric filters produce ES shapes that the server's aggregation flattener and ZAI-Dashboard must also handle. See `MIRROR_NOTES.md`.
 - **Calendar / MeetingScheduler are simplified.** There is no drag-to-resize, month-grid layout or virtual scrolling. For full parity, wrap FullCalendar in `render`.
 
 ---

package/SKILL.md

@@ -127,7 +127,7 @@ Stored table columns are `{ label, dataPoint, type }` and are read as-is.
 ## Authoring a data source
 
 ```tsx
-import { formSource, taskSource, field, taskField, groupBy, aggregate, derive, where, pivot } from "@zestl/zvolv-widgets";
+import { formSource, taskSource, field, taskField, groupBy, aggregate, dateDiff, derive, where, pivot } from "@zestl/zvolv-widgets";
 
 const region  = field("Region",  "DROP_DOWN");       // use the REAL element type
 const revenue = field("Revenue", "INPUT_CURRENCY");
@@ -147,6 +147,21 @@ field("Status", "DROP_DOWN", { filter: where("IN", "Open") });  // baked-in auth
 
 Element type → ES type: `INPUT_NUMBER`/`INPUT_CURRENCY`/`INPUT_PERCENT` → FLOAT · `FORM_SEARCH`/`INPUT_TAG` → ARRAY · `CHECK_BOX` → BOOLEAN · else STRING.
 
+Time series, date differences and filtered metrics (0.0.2). Never post-process rows for these:
+
+```tsx
+groupBy(requestDate, "Month", { interval: "month" })      // date_histogram; day|week(Mon)|month|quarter|year; keys = epoch ms
+aggregate(dateDiff(requestDate, offerDate, "days"), "avg", "Avg Days")   // end − start; docs missing a date skipped
+aggregate(dateDiff(requestDate, "TODAY"), "max", "Age")
+aggregate(dateDiff(a, b), "percentiles", "Days", { percents: [50, 90] }) // rows get Days_p50, Days_p90
+aggregate(amount, "sum", "Billed", { where: [[status, where("IN", "Paid")], [date, where("BETWEEN", from, to)]] }) // AND-ed; many metrics = 1 request
+```
+
+- `title` / `outputName` only name the output key. The query uses the field's `masterfield`.
+- `dateGap={{ unit, start, end }}` sums rows in the same period (`merge: "last"` gives the old behaviour).
+- Add `cumulative: true | [fields]` with `baseline: { field: n }`, or `baselineSource: { dataSource, dateField }`, for running totals.
+- KPI sparkline: `contents[i].seriesSource = { mode: "timeseries", dataSource, xField, valueField, dateGap }`. The widget fetches it.
+
 ## Writing configuration
 
 ```tsx
@@ -176,9 +191,10 @@ KPI templates: `standard` `trend` `comparison` `progress` `financial` `status` `
 
 | Family | Components | `render` args |
 | --- | --- | --- |
-| KPI | `KpiWidget` | templates get `{ vm, index }`. `vm` is fully resolved; never read `vm.raw` |
+| KPI | `KpiWidget` | templates get `{ vm, index }`. `vm` is fully resolved; never read `vm.raw`. Card: `layout: "card"` (root or `cardConfig`) + `contents[].role` (`main`/`sub`/`chip`/`series`); custom card via `renderCard` (only in card mode) |
 | Charts | `BarChartWidget` `LineChartWidget` `AreaChartWidget` `PieChartWidget` `DoughnutChartWidget` `ScatterChartWidget` | `{ rows, xKey, series, colorFor, toLabel, template, config, tokens, meta }` |
-| Ranking | `RankingBarWidget` | `{ rows: {rank,label,value,ratio}[], max }` |
+| Ranking | `RankingBarWidget` | `{ rows: {rank,label,value,ratio}[], max }`; `ranking.order: "none"` keeps row order |
+| Funnel | `FunnelWidget` (`FUNNEL`) | `{ rows: {label,value,ratioToFirst,stepPct,isFirst}[], first, tokens }`; `onItemClick(row)` |
 | Gauge | `GaugeMeterWidget` | `{ value, min, max, ratio, color, band }` |
 | Heat map | `HeatMapWidget` | `{ cells: {zone,count,color,band}[], grid }` |
 | Tables | `DashboardTableWidget` `DatabaseTableWidget` `TaskTrackerTableWidget` `ChildTableWidget` `PivotGridWidget` | `{ rows, columns, renderCell, toLabel, interactions, headerControls(col), getColumnOptions(key), enableSort, showColumnFilter, mode }` |
@@ -188,6 +204,17 @@ KPI templates: `standard` `trend` `comparison` `progress` `financial` `status` `
 | Display | `HtmlWidget` `RawHtmlWidget` `IframeWidget` `GoogleMapWidget` `TreeWidget` `ActionCardWidget` | see types |
 | Containers | `TabWidget` `FormWidget` `FilterPanelWidget` `ZaiChatWidget` | see types |
 
+Reshaping after the query (0.0.3). Use these instead of re-shaping rows in React:
+
+```tsx
+unpivot={{ columns: STAGES, extra: { "Contacted %": "step" } }}   // one wide row → { label, value } per column, in order; missing → 0
+categoryGap={{ field: "Pipeline Stage", values: "fromForm" }}     // every category in dropdown order, 0 where no bucket
+<FunnelWidget dataSource={ONE_ROW} unpivot={{ columns: STAGES }} onItemClick={(r) => …} />
+<FilterPanelWidget configurations={{ filters: [{ key, field: "Assignment", type: "select",
+  optionsSource: { dataSource, labelField, valueField, allLabel: "Firm-wide" } }] }}
+  onFiltersChange={(f) => setWidgetFilters(f)} />   // valueField = the lookup's ViewLabel field for FORM_SEARCH targets
+```
+
 **`render` is REQUIRED** for `ZaiChatWidget`, `GoogleMapWidget` and `HtmlWidget` markup. `FormWidget` renders `<ZvolvForm>` by default (see Forms). The package never calls `dangerouslySetInnerHTML`.
 
 In chart `render`, always use `xKey` and `series[].valueField` from the args, pass axis and user values through `toLabel`, and colour with `colorFor(i, s)`.