trellum 0.2.0__tar.gz

trellum-0.2.0/AGENTS.md

@@ -0,0 +1,671 @@
+# trellum -- Agent Workflow Guide
+
+This is the **workflow guide** for AI assistants (Claude Code, Cursor, Codex)
+building reports with this framework. It is the only document you need resident.
+
+`README.md` is written for people, and it is long. **Do not load it wholesale.**
+Ask for the part you need instead:
+
+```
+trellum                    what the framework is, in ~30 lines
+trellum guide <topic>      one section of depth, on demand
+trellum checks             every validator check id and its level
+trellum validate <dir>     the current validation state of a built report
+```
+
+`trellum guide` topics: `queries`, `format`, `generator`, `components`,
+`filters`, `rawhtml`, `validation`, `report-yaml`, `themes`, `review`.
+
+**Why this shape.** Your whole context is re-read on every API round-trip, so a
+large always-loaded reference is paid for dozens of times per task. A command is
+paid for once, when it is actually needed. Reach for `trellum guide` freely —
+it is cheaper than it looks, and far cheaper than guessing.
+
+## Where to look, by question
+
+| Question | Answer |
+|---|---|
+| What components exist? | `trellum` — the bare command lists all of them |
+| What does this validator check id mean? | `trellum checks`, then `trellum guide validation` |
+| How do I filter a chart? | `trellum guide filters` |
+| Why is my chart empty / not reacting? | `trellum guide format` — it is almost always wide-vs-long |
+| Where does aggregation belong? | `trellum guide queries` — pandas, not SQL |
+| What goes in report.yaml? | `trellum guide report-yaml` |
+| Everything else | `README.md`, read the relevant section only |
+
+**Before writing a generator, read `trellum guide queries` and
+`trellum guide format`.** Those two rules are not guessable from the API, and
+getting them wrong produces a report that builds cleanly and displays nothing
+useful.
+
+## When to use the framework
+
+| Goal | Where it goes |
+|---|---|
+| Recurring, interactive report — served standalone, on a schedule, or embedded in another application | `reports/{slug}/` (the framework) |
+| One-off data exploration, "what was X yesterday", quick prototyping | A plain script or notebook — the framework's structure buys nothing for throwaway analysis |
+
+Scaffold a new report: `python3 -m trellum.new my-report --studio my-studio --category Revenue`. Build it: `python3 -m trellum.run reports/my-report --no-serve`.
+
+**Always pass `--no-serve` unless you have been asked to open the report in a
+browser.** Without it the command starts a preview server on :8050 and never
+exits — the build itself finishes in under a second, but the process stays in
+the foreground until it is killed. Build first, then serve as a separate step
+if a human is going to look at it.
+
+**Read files with your file-reading tool, not with `cat`, `ls` or `grep` in a
+shell.** Every shell command starts a process; your native read, glob and search
+tools do not. On the machine these runs were measured on a bare `ls` cost about
+twelve seconds and an identical `Read` cost none — that gap is environment-
+specific, but its direction never is.
+
+The trap is that batching looks cheaper: `cat a.py; cat b.py; cat c.py` is one
+command where three reads are three calls. Under a per-command cost that
+reasoning is right, and here it is exactly backwards — three native reads are
+free and the one shell command is not. Use the shell for things that genuinely
+need it: running the build, the validator, git.
+
+When you do serve, two things are not guessable and both have cost real time:
+
+- **A single report is served at `/`, not at `/<slug>`.** Only `--all --serve`
+  puts an index at `/` with reports beneath it. `/<slug>` on a single-report
+  server returns an error page.
+- **Wait for the `Serving at ...` line before opening the URL.** `--serve`
+  rebuilds the report and binds the port last, so until that line appears the
+  URL may still be answered by a previous server — showing a different report
+  rather than failing, which is far more confusing than a refused connection.
+
+<!-- topic: queries -->
+## How to write queries (MANDATORY — read before touching `queries.py`)
+
+**Exploring the data first?** `python -m trellum query "SELECT ..."` runs
+ad-hoc SQL against a configured source by NAME — never hunt for the database
+file or hardcode its path, both of which break the day the source is a
+remote warehouse. `--param day=2026-06-15` binds `:day`; sqlite sources are
+opened read-only.
+
+**The rule for report queries:** SQL pulls raw rows with a date filter.
+Python does the aggregation. Never both in the same query.
+
+```sql
+-- ✓ RIGHT — minimal SQL: date filter + column projection, no aggregation
+SELECT event_date, dim_a, dim_b, metric_col
+FROM <schema>.<fact_table>
+WHERE event_date BETWEEN DATE :start_date AND DATE :end_date
+```
+
+```python
+# ✓ RIGHT — aggregate in the generator with pandas
+df_raw = query_df(conn, queries.MY_QUERY, params={...})
+daily = df_raw.groupby("event_date")["metric_col"].sum().reset_index()
+split = df_raw.groupby(["event_date", "dim_a"])["metric_col"].sum().reset_index()
+```
+
+```sql
+-- ✗ WRONG — SQL-side GROUP BY on a fact table
+SELECT event_date, dim_a, SUM(metric_col)
+FROM <schema>.<fact_table>
+WHERE event_date BETWEEN ... GROUP BY 1, 2
+```
+
+Two reasons this rule is non-negotiable:
+
+1. **Memory footprint.** A `GROUP BY + SUM` on a wide fact table makes
+   the database pre-allocate several GB for the hash aggregate, and
+   local-dev user accounts frequently run under tight resource-pool
+   caps. Raw-row retrieval with a date filter stays small — a typical
+   30-day window of a fact table fits in tens of MB of pandas.
+2. **Dynamic filtering is the framework's whole point.** The
+   client-side `FilterBar` re-aggregates every time the user changes
+   a filter. Pre-aggregating in SQL collapses dimensions and breaks
+   that interactivity — the chart freezes at whatever grain the SQL
+   produced. Raw-row `DataSource`s let the client slice any dimension
+   you included.
+
+Code smells — triggers to rewrite the query:
+
+- `SUM()` / `AVG()` / `COUNT(DISTINCT)` in the SELECT that collapses a
+  dimension the user might want to filter by.
+- `GROUP BY` on only a subset of the filterable dimensions. Either
+  drop the GROUP BY or group by every dim the client might filter on
+  (denormalized grain).
+- `HAVING` — do it in pandas.
+- `JOIN` that explodes row count to attach a dim — better as a pandas
+  `.merge()` after both sides load.
+
+Acceptable exceptions (rare):
+
+- Querying an already-aggregated table (one whose name signals a
+  pre-rolled grain like `*_agg_*` / `*_daily_*`): pull columns
+  directly, no further SQL aggregation.
+- A `GROUP BY` on the union of every filterable dimension. Preserves
+  filterability but compresses duplicates. Use only if raw-row
+  retrieval genuinely returns too much data.
+
+Project-specific table names, memory caps, and schema particulars live
+in `project_context/chat_rules/` — the chat feature loads them at
+runtime; human readers can look there for the concrete tables to query.
+
+<!-- topic: format -->
+## Long format vs wide format (MANDATORY)
+
+**Rule:** dimensions stay in **rows**, never in **column names**. One
+row per (date × every breakdown dim), one column per metric.
+
+```python
+# ✓ RIGHT — long format
+event_date  | platform | spender_tier | dau   | iap_revenue | ad_revenue
+2026-04-01  | ios      | Whale        | 12000 | 9800        | 250
+2026-04-01  | android  | Whale        | 8500  | 5400        | 180
+```
+
+```python
+# ✗ WRONG — platform baked into column names ("wide format")
+event_date  | spender_tier | dau_ios | dau_android | iap_revenue_ios | iap_revenue_android | ...
+2026-04-01  | Whale        | 12000   | 8500        | 9800            | 5400                | ...
+```
+
+Why long format is the default:
+
+1. **Filter coverage.** The client filter engine matches on column
+   *values*, not column *names*. A filter on `platform = "ios"` only
+   works when there's a `platform` column carrying `"ios"` as a value.
+   Wide format makes the filter inert.
+2. **Smaller payload.** Dictionary encoding via `_serialize_columnar`
+   compresses high-repetition columns (e.g. `platform` with 8 unique
+   values across 50 k rows) to integer indices — typically 60–70 %
+   smaller than the wide equivalent with one column per platform.
+3. **Auto-discovery.** Adding a new platform just adds rows to the
+   data; charts using `stack_by="platform"` pick it up. Wide format
+   forces editing every chart's `y=[col_a, col_b, ...]` list.
+4. **Simpler SQL.** `GROUP BY event_date, platform, ...` instead of
+   one `SUM(CASE WHEN platform='ios' THEN dau END) AS dau_ios` per
+   metric per platform.
+
+Render breakdowns with `stack_by`:
+
+```python
+# Long format → one line per platform via stack_by
+LineChart(df=df, x="event_date", y="iap_revenue",
+          stack_by="platform", dataset_id=ds, y_format="currency",
+          title="IAP Revenue by Platform")
+
+# Long format + ratio → one line per platform, computing iap/dau per (date, platform)
+LineChart(df=df, x="event_date", dataset_id=ds,
+          ratios=[{"numerator": "iap_revenue", "denominator": "dau",
+                   "label": "IAP ARPDAU"}],
+          stack_by="platform", y_format="currency",
+          title="ARPDAU (IAP) by Platform")
+```
+
+The `chart-filter-coverage` validator catches accidental wide-pivots —
+when a chart's DataSource cannot react to a FilterBar filter because
+the dimension was collapsed into column names. Long format is the
+fix; suppression is reserved for charts that are intentionally a
+fixed rollup (e.g. period-over-period reference rollups).
+
+<!-- topic: generator -->
+## The 5-step process to write generator.py
+
+1. Subclass `BaseReport` and implement `generate(self, ctx)`.
+2. `conn = ctx.get_connection("primary_warehouse")` (or whatever name the project uses in `data-sources/config.yaml`); query with `query_df(conn, queries.X, params={...})`.
+3. Wrap each DataFrame in a `DataSource` + a `FilterBar` placed together in an **untitled section**: `ctx.add_section("", [DataSource(...), FilterBar(...)])`. Untitled is required for sticky positioning.
+4. Add content sections: `ctx.add_section(title, [...])`.
+5. **Every chart, KPI, and table component MUST carry `dataset_id="..."`** pointing to its DataSource. Without it the component renders statically and silently ignores filters — this is the most common mistake.
+
+To copy from, list what this project actually has (`ls reports/`) and open one.
+Report names are not named here on purpose: this file ships with the framework
+and travels into every project, so any slug written down is a report somebody
+else has. A measured run followed three such names and got three "file does not
+exist" errors before it thought to look.
+
+<!-- topic: report-yaml -->
+## Write the report.yaml description for discoverability (MANDATORY)
+
+A search or assistant layer routes user questions to reports using
+the `description` and `tags` in `report.yaml`, plus the dataset columns it
+indexes from the built output. A vague description ("Revenue report") makes
+the report invisible to it. The description MUST state:
+
+1. **Which business questions the report answers** ("how many payers churn
+   per week and why"), not just its topic.
+2. **The key metrics and dimensions** it carries (churn rate, revenue at
+   risk; split by platform/tier/country).
+3. **Grain and freshness** (weekly cohorts; daily; refreshes every 5 min).
+
+The `report-description-weak` validator check WARNs on short descriptions.
+Descriptive column names in DataSources matter for the same reason — the
+assistant reads them from data.json to decide which dataset answers a
+question.
+
+<!-- topic: components -->
+## Component Reuse Policy
+
+**Before writing any custom HTML, CSS, or JS, name the framework component you ruled out and why.** Most needs are already solved.
+
+| Need | Required component |
+|---|---|
+| Filters / dropdowns / sticky filter bar | `DataSource` + `FilterBar` |
+| Section-local filter on same data (no propagate_to wiring) | `ScopedDataSource(id, parent=...)` + section-scoped `FilterBar` |
+| Cascading dropdowns (parent → child option narrowing) | Add `depends_on: "parent_col"` to a dropdown filter spec |
+| KPI cards | `KpiRow(dataset_id=...)` (`agg`: `sum` / `ratio` / `count` / `abssum`) |
+| Time series | `LineChart` |
+| Bar / stacked bar | `BarChart` / `StackedBar` |
+| Area / stacked area | `AreaChart` (`stacked=True`) |
+| Doughnut / pie | `DoughnutChart` |
+| Heatmap / day×hour / cohort grid | `HeatmapChart` |
+| Funnel | `FunnelChart` |
+| Treemap | `TreemapChart` |
+| Scatter / correlation / bubble | `ScatterChart` |
+| Dual-axis bar + line | `ComboChart` |
+| Ratio metrics (ARPDAU, ARPPU, retention, conversion) | `LineChart(ratios=[{numerator, denominator, label}])` — never hand-roll the division in JS |
+| Ratio by category (CPD per comfort_zone, ARPPU by country) | `BarChart(ratios=[...], horizontal=True, sort='desc')` — categorical ratio bars |
+| Tables | `DataTable`, `ComparisonTable`, `PivotTable` |
+| Layout | `Grid`, `Panel`, `SplitPane`, `TabGroup` |
+| Card-panel grid | `Grid(card=True)` |
+| Cross-grain filter sync | `FilterBar(propagate_to={target_ds: {src_col: tgt_col}})` |
+| A/B test report | `ABCompare.from_users(df, variant_col=..., control=..., test=..., metrics=[Metric(...)])` — the front door: a per-user frame in YOUR column names, out comes the finished component with an SRM badge and raw / winsor / CUPED modes (a `Metric` that declares `pre_col` opts into CUPED). Groups fetched separately (one query per arm)? `ABCompare.from_groups({"Control": df_a, "Test": df_b}, metrics=[...])`. Self-contained — no DataSource/FilterBar needed. The raw constructor `ABCompare(rows=..., modes=...)` plus `trellum.stats.ab.{winsorize_user_df, cuped_user_df, bootstrap_ab_cis}` remains for custom row sets. See `trellum/README.md` "A/B Testing". |
+
+`RawHTML` is the **exception, not the default**. It is permitted only when **both**:
+
+1. The visualization is genuinely novel (force-directed graph, Sankey, etc.) and no combination of framework components can express it.
+2. You have explicitly named the components you considered and why each was insufficient.
+
+For complex reports prefer the **hybrid pattern**: framework components for filters/standard charts/KPIs; `RawHTML` only for the genuinely novel section. The custom JS must subscribe to the framework filter engine — `window._fwFilterEngine.subscribe(dsId, id, fn)` — instead of managing its own filter state. See `trellum/README.md` "Custom Dashboards" for the full pattern and the `fw.*` API.
+
+### Anti-patterns to avoid
+
+- **Hardcoded hex colors anywhere in RawHTML JS/HTML** — the `rawhtml-hardcoded-hex` validator **FAILS** the report on any `#RGB`/`#RRGGBB` literal except `#fff` and `#000`. Use `fw.getThemeColors().chart_colors[i]` for chart palettes and CSS vars (`var(--accent-red)`, `var(--text-main)`, `var(--bg-card)`) for HTML/CSS.
+- **Hardcoded `rgba?()`/`rgb()` literals in RawHTML JS** — `rawhtml-hardcoded-rgba` (WARN) flags numeric color literals that don't update on theme switch. Use `fw.getThemeColors().grid_color` for grid lines, `.tick_color` for axis ticks / legend labels, `.chart_colors[i]` for dataset colors. `rgba(0,0,0,0)` (transparent) is exempt. Suppress for intentional fixed-color semantic annotations.
+- **CSS variable strings in Chart.js color properties** — `rawhtml-css-var-in-chartjs` (WARN) flags patterns like `color: 'var(--text-main)'`. Chart.js has no CSS resolver; the string is used as-is (invalid color). Use `fw.getThemeColors().tick_color` / `.grid_color`, or resolve with `getComputedStyle(document.documentElement).getPropertyValue('--name').trim()`.
+- Re-implementing Slim Select dropdowns (use `FilterBar`; `ScopedDataSource` for section-local; `depends_on` for cascading)
+- Writing custom `getFilteredRows()` / filter state management (use the public `fw.filterEngine.*` API)
+- Polling with `setTimeout(init, 100)` to wait for `_fwFilterEngine` (use `fw.filterEngine.onReady(dsId, fn)`)
+- Referencing `window._fwFilterEngine` directly in RawHTML (it's private; use `fw.filterEngine`)
+- Duplicating a DataFrame to get a separate DataSource for section-local filtering (use `ScopedDataSource(id, parent=...)`)
+- Duplicating KPI rendering when `KpiRow(dataset_id=...)` covers it
+- Building custom layout grids when `Grid` / `Panel` / `SplitPane` suffice
+- Going 100% `RawHTML` when only one or two sections need custom logic
+- LEFT JOINing coarser-grain data (monthly MAU, install cohort) into a finer-grain DataFrame to avoid creating a second DataSource — duplicates rows and silently breaks ratio aggregations under filtering
+
+### KpiRow aggregation semantics (exact, from the runtime)
+
+Every `agg` recomputes over the **filtered rows**, so pass raw columns and let the runtime do the arithmetic — a pre-divided or pre-summed column cannot re-aggregate. The complete set:
+
+| `agg` | Computes | Notes |
+|---|---|---|
+| `sum` | `sum(column)` — or summed across a `columns` list | the default |
+| `count` | number of filtered rows | no `column` needed; add a literal `df["x_n"] = 1` column when you also need a ratio denominator |
+| `abssum` | `sum(abs(column))` | for signed ledgers |
+| `ratio` | `sum(numerator) / abs(sum(denominator)) × 100` | **always ×100 — only correct with `format: "percent"`.** A currency or plain-number ratio (AOV, ARPDAU, revenue per order) does NOT belong in a KpiRow: put it on a chart via `ratios=[{numerator, denominator, label}]`, which divides without the ×100 and formats per axis |
+| `avg_by_date` | `sum(column) / count(distinct date_col)` | per-day average; `date_col` defaults to `event_date` |
+| `purchase_pct` | share of `source_col` volume where `type_col` is in `match_values`, ×100 | percent-format only, like `ratio` |
+
+The ×100 in `ratio` is the most re-derived fact in measured sessions — one agent spent 15 tool calls reading `kpis.py`, `validation.py` and the JS runtime to establish it. It is stated here so the next one does not have to.
+
+<!-- topic: rawhtml -->
+## RawHTML chart lifecycle (mandatory when the chart lives inside a `Visible`)
+
+If your `RawHTML` creates a Chart.js instance and the `RawHTML` is anywhere inside a toggle-driven `Visible`, the JS **must** patch BOTH `_initToggleVis` and `renderAll`, AND defer the render with `requestAnimationFrame`:
+
+```js
+function _refresh() {
+    if (fw.filterEngine.isReady(dsId)) _render(canvasId, dsId, fw.filterEngine.getFiltered(dsId));
+}
+
+fw.filterEngine.subscribe(dsId, 'mychart', _render);
+window.addEventListener('fw-theme-change', _refresh);
+
+// _initToggleVis runs on initial page load (renderAll does NOT — the framework
+// calls _renderComps() directly on first load). Without this hook the chart is
+// blank on first load and on URL-preloaded state.
+window._initToggleVis = (function (prev) {
+    return function () { if (prev) prev(); requestAnimationFrame(_refresh); };
+})(window._initToggleVis);
+
+// renderAll catches scope switches, theme changes, auto-refresh.
+window.renderAll = (function (prev) {
+    return function () { if (prev) prev(); requestAnimationFrame(_refresh); };
+})(window.renderAll);
+```
+
+Why each piece:
+
+| Hook | Catches | Without it |
+|---|---|---|
+| `fw.filterEngine.subscribe(dsId, ...)` | filter changes | chart never updates when user filters |
+| `fw-theme-change` listener | theme switches | chart keeps old palette |
+| `window._initToggleVis` patch | **initial page load + URL preload** | chart blank on first load — only renders after user clicks a toggle |
+| `window.renderAll` patch | scope switches, auto-refresh, theme switches | chart stale after re-renders |
+| `requestAnimationFrame` defer | layout reflow after `_updateToggleVis` flips `display` | Chart.js measures canvas at 0×0 → blank chart even though parent is visible |
+
+For multi-canvas patterns (one canvas per per-feature DataSource), use a `_ensureSubs()` that finds canvases by attribute (e.g. `[data-split-ds]`) and lazily subscribes once each, then `requestAnimationFrame(_refreshAll)` from both lifecycle hooks.
+
+<!-- topic: filters -->
+## DataSource + FilterBar rules
+
+- Every report has at least one `DataSource` + `FilterBar` pair (exceptions: `ABCompare`-only reports; single-day dashboards may omit the date_range filter but should still use `DataSource` for any filterable dimension).
+- `date_range` filter first when there's a time-series dimension; one filter per useful categorical dimension.
+- **One grain = one DataSource.** Different grain (monthly MAU vs daily revenue) gets its own `DataSource`. Use `propagate_to` to sync shared dimensions.
+- `static=True` only when data is fundamentally incompatible with the report's filters (point-in-time snapshot, external system data with different date semantics) — rare.
+
+### Two layout patterns: main vs section FilterBar
+
+The framework supports two FilterBar placements, and they compose:
+
+**Main FilterBar** (the default for most reports). Placed in an
+**untitled section** at the top. Use it for filters that apply across
+the whole report — typically `date_range`, `audience_segment`,
+`platform`, etc. The main FilterBar reaches its primary DataSource
+plus any DataSources listed in `propagate_to`. Sticks to the viewport
+top.
+
+```python
+ctx.add_section("", [                # untitled section — sticky top
+    DataSource("daily", df_daily, chunk_by="month"),
+    DataSource("monthly", df_monthly),
+    FilterBar("daily", df_daily, filters=[
+        {"column": "event_date", "type": "date_range"},
+        {"column": "platform"},
+    ], propagate_to={
+        "monthly": {"event_date": "event_date"},
+    }),
+])
+```
+
+**Section-scoped FilterBar** (drill-downs). Placed inside a titled
+section right after that section's DataSource(s). Use it when a
+section has a dimension that doesn't make sense for the rest of the
+report — e.g. `chest_level` only applies to chest charts. Section
+FilterBars compose with the main FilterBar: a chart inside a section
+is filtered by the AND of both. Sticks below the main FilterBar while
+its section is in view.
+
+Canonical layout for a section-scoped pair (validator recognizes this
+exact order):
+
+```python
+ctx.add_section("Chest Daily Trends", [
+    DataSource("chest_daily", df_chest),
+    FilterBar("chest_daily", df_chest, filters=[
+        {"column": "chest_level"},
+        {"column": "difficulty_tier"},
+    ]),
+    LineChart(..., dataset_id="chest_daily"),
+    LineChart(..., dataset_id="chest_daily"),
+])
+```
+
+### Section-local filters on the SAME data — use `ScopedDataSource`
+
+When a section needs **another local filter** on data that's already
+loaded via the main DataSource (e.g. "Price Tier" filter that only
+affects one chart, not the whole report), use a `ScopedDataSource`
+instead of a fresh DataSource. It:
+
+- inherits all of the parent's filters automatically — no `propagate_to`
+  wiring needed
+- ships no extra data — rows are derived from the parent at runtime
+- composes with the section's own FilterBar (parent filters AND child
+  filters)
+
+```python
+# Untitled top section: ONE base DataSource + main FilterBar
+ctx.add_section("", [
+    DataSource("cpd", df),
+    FilterBar("cpd", df, filters=[
+        {"column": "event_date", "type": "date_range"},
+        {"column": "package_group"},
+    ]),
+])
+
+# Drill-down section with its OWN local filter
+ctx.add_section("CPD by Price Point", [
+    ScopedDataSource("cpd_pp", parent="cpd"),
+    FilterBar("cpd_pp", df, filters=[
+        {"column": "price_tier"},      # only affects this section
+    ]),
+    LineChart(df=df, x="event_date", dataset_id="cpd_pp",
+              ratios=[{"numerator": "chips", "denominator": "revenue",
+                       "label": "CPD"}],
+              stack_by="price_point_display",
+              title="CPD by Price Point"),
+])
+```
+
+**Cascading dropdowns** (e.g. *Package Group* → *Package Name*): add
+`depends_on` to the child filter spec. The child's option list narrows
+automatically when the parent selection changes:
+
+```python
+FilterBar("cpd_pkg", df, filters=[
+    {"column": "package_group"},
+    {"column": "package_name", "depends_on": "package_group"},
+])
+```
+
+### Custom RawHTML lifecycle
+
+When you do need custom JS (genuinely novel visualization), always use
+the public `fw.*` API:
+
+```js
+fw.filterEngine.onReady('cpd', function() {
+    // engine is ready — wire your chart now
+    fw.filterEngine.subscribe('cpd', 'my-chart', _render);
+    _render(fw.filterEngine.getFiltered('cpd'));
+});
+```
+
+**Never** poll with `setTimeout(init, 100)`, **never** reference
+`window._fwFilterEngine` directly, **never** instantiate `new SlimSelect(...)`
+in RawHTML. Each of those is a validator WARN now. Use FilterBar +
+ScopedDataSource for the dropdown UX; use `fw.filterEngine.onReady` for
+ready-detection; use `fw.filterEngine.*` for everything filter-related.
+
+Validator constraints:
+
+- At most one main FilterBar per report (untitled top section).
+- At most one section-scoped FilterBar per titled section.
+- Section-scoped pair must be laid out as `[DataSource(s) or ScopedDataSource, FilterBar, ...content]` at the start of the section.
+- A `ScopedDataSource`'s `parent` must reference a base `DataSource` (not another scoped child).
+- A filter with `depends_on` must reference another filter on the same FilterBar with a real column in the underlying DataFrame.
+
+Use `ScopedDataSource` when a section filters the *same* rows more narrowly. When the grain genuinely differs, give each section its own `DataSource` instead — different data is not a scope of the same data.
+
+<!-- topic: review -->
+## The live review loop
+
+Review mode turns a served report into a feedback surface: the user clicks
+elements in the browser, types change requests, queues them with one optional
+chat message, and hits **Send** — you receive the batch in the terminal with
+each item's section title, component kind and title, DOM id, and selector,
+which map straight to lines of `generator.py`.
+
+The loop, from your seat:
+
+1. `python -m trellum review start reports/<slug>` — builds if output is
+   missing (`--rebuild` to force), starts or reuses the background server,
+   enables review mode, and opens the browser. If a server predating review
+   mode holds the port, rerun with `--restart-server`.
+2. `python -m trellum review poll` — **BLOCKS** until the user sends
+   feedback. From an agent harness, run it with a generous timeout or as a
+   tracked background task; if it gets killed, just rerun it — queued
+   feedback is never lost. Exit codes: `0` feedback arrived, `2` your
+   `--timeout` elapsed, `3` the user ended the session.
+3. Edit `reports/<slug>/generator.py` and rebuild with
+   `python -m trellum.run reports/<slug> --no-serve`. The browser notices
+   the rebuild and reloads itself within ~2 seconds — never hand-edit
+   `output/` HTML, and never restart the server to "refresh".
+4. `python -m trellum review poll --reply "what you changed"` — the reply
+   appears in the browser's chat and unlocks the user's Send button, then
+   the command waits for the next round.
+5. Repeat until poll exits `3` (the user pressed End, or Send & End — that
+   final batch still arrives first). `python -m trellum review end` ends
+   it from your side when the user asks you to wrap up in conversation.
+
+**Several reports, several agents, one project:** one server serves the
+whole `output/` tree and one review session spans it — every report page
+gets the overlay, and batches are tagged with their report's slug. When more
+than one agent works the same project, each MUST poll with
+`--report <slug>` (and reply with `--report <slug>`): a slug-scoped poll
+drains only that report's batches, so agents never steal each other's
+feedback. A slug-less poll drains everything — fine only when you are the
+only agent. Different projects are automatically separate: each output
+directory gets its own server on its own port (discovered via
+`/_fw/server.json` identity on ports 8050–8069), with fully independent
+review state.
+
+The user can also send element-free chat messages; they arrive as a batch
+whose `items` list is empty and whose message is the `note`. And on Chart.js
+charts they can point at the data itself — Alt-click picks the nearest data
+point, dragging marks an x-range — which arrives as a `data` field on the
+item (`chart-point`: series/x/value; `chart-range`: x_from/x_to plus
+per-series n/min/max and the points). That is a data investigation, not a
+styling request: answer it with `trellum query` against the exact dates
+before touching any code. Everything is served from the normal `trellum
+serve` server — review endpoints live under `/_fw/review/`, loopback-only,
+and the on-disk HTML is never modified (the overlay is injected into the
+served copy only).
+
+<!-- topic: validation -->
+## Validator response protocol (mandatory)
+
+Every report run executes `trellum/validation/` post-generation. Output goes to console + `output/<slug>/_validation.json`.
+
+A report task is **not done** until you have:
+
+1. Read the validator output.
+2. Surfaced every **FAIL** and **WARN** to the user with check ID, affected component/section, and a concrete fix.
+3. **Fixed every FAIL.** WARNs should be fixed unless intentionally accepted (then suppress in `report.yaml` under `validation.suppress` with a one-line reason for why).
+4. If the report has zero FAILs and zero WARNs, said so explicitly.
+5. **Handed the user a clickable link.** The last action before reporting done is `python -m trellum serve --background` — it starts (or reuses) a detached server, prints the URLs, and returns immediately — and your final message to the user must contain the printed report URL. A finished build the user cannot click is not finished. (Keep using `--no-serve` for the build/iterate loop itself; the link is the closing move, not a per-build cost.)
+
+Severity:
+
+| Level | Action |
+|---|---|
+| **FAIL** | Must fix. Structural bug → broken rendering, missing interactivity, wrong data. Common: missing `dataset_id`, orphan FilterBar, missing DataFrame columns. |
+| **WARN** | Fix unless accepted + suppressed. Common: non-date x-axis with annotations enabled (silently skipped), hardcoded colors (breaks on theme switch), missing `renderAll` in custom JS. |
+| **INFO** | No action required. Just confirm intent. |
+
+Full check list, suppression syntax, and how to add new checks: `trellum/README.md` "Validation".
+
+### Filter coverage check
+
+The validator emits **`chart-filter-coverage`** (WARN) for every chart whose `dataset_id` is not reached by one or more FilterBar filters — the most common silent bug, where users see filter dropdowns that change nothing on certain charts. The fix is almost always to extend `FilterBar.propagate_to` with the missing `{src_col: tgt_col}` entry.
+
+A sibling check, **`chart-value-grain-mismatch`** (WARN), catches a subtler failure mode: the chart's value column is constant within each x-axis value (e.g. a daily total merged on `event_date` only into a date × audience × spender DataSource). Filters reach the DataSource structurally, but the chart's plotted value never changes — ratios stay constant, absolutes inflate by the count of replicated rows. Fix by extracting the column to its own DataSource at native grain (date-only) and filtering by `event_date` only there.
+
+When a filter genuinely cannot apply to a DataSource (its SQL doesn't carry that dimension and adding it would require fictional attribution — e.g. attaching a `template_type` to a purchase event), accept the gap **per (DataSource, filter_column)** in `report.yaml`:
+
+```yaml
+validation:
+  accept_inactive_filters:
+    tc_rev_gop3:
+      - template_type   # purchases don't carry a template_type — see queries.py:156
+    tc_boost_gop3:
+      - template_type   # boosters live in economy_balance, no template dim
+      - chest_tier
+```
+
+This is the **preferred** form because future dead filters on the same DataSource still fire — only the explicitly-listed columns are silenced. The broader `validation.suppress_per_dataset: {ds: [chart-filter-coverage]}` swallows ALL future inactive filters on that DS and should be reserved for DataSources that are fundamentally non-reactive (tiny static rollups feeding one chart).
+
+Full per-chart, per-filter matrix lives at `output/<slug>/_details.json` (also embedded in `_meta.json` under `details.filter_matrix`) and can be rendered as a coverage table. `details.totals` carries dataset count, chart count, total row count, total serialized data size — so you can see at a glance how heavy a report is. There is also a `data_source: "real" | "mock"` flag and an amber banner when the run was a mock/test build, so synthetic row counts don't get read as production reality.
+
+With the health view enabled (`?health=1` in the URL, or an embedding application that reports the viewer as an admin), the rendered report itself shows a small **filter-health badge** above each chart whose dimensions are not all reactive — a red `⚠ N/M filters dead` chip that pops a styled per-filter detail panel on hover. Hidden by default for non-admins so end-users see a clean report.
+
+
+### Suppression hierarchy
+
+Three layers, narrowest → broadest:
+
+```yaml
+validation:
+  # Granular, PREFERRED — accept specific (DataSource, filter_column) pairs
+  # as inactive. Any OTHER inactive filter on the same DS still fires.
+  accept_inactive_filters:
+    pf_country_gop3: [platform]
+    pf_cz_gop3: [platform, spender_tier]
+
+  # Per-DataSource — silences the named check ID for ALL filters on that DS.
+  # Use only when the DataSource is fundamentally non-reactive across every
+  # dimension (tiny static rollup feeding one chart).
+  suppress_per_dataset:
+    economy_test_net: [ds-no-filterbar]
+
+  # Global — silences every instance of the named check ID across the report.
+  # Last resort; use for systemic exemptions (e.g. annotations-non-date-x-axis
+  # on a report that is intentionally month-bucketed).
+  suppress:
+    - some-systemic-check
+```
+
+Suppressed checks are NOT removed from the result — they're flagged with `suppressed: true` and grouped under **Suppressed** with rationale text. The matrix renders suppressed inactive cells as `✗ⓢ` so a reviewer can audit later. Always pair a suppression with a `# rationale:` YAML comment naming the specific reason.
+
+## After framework changes (only if explicitly asked)
+
+If the task explicitly requires modifying `trellum/*`:
+
+1. **Add or update validation checks in `trellum/validation/`** for the new behavior — non-negotiable. Skipping silently breaks future reports.
+2. Run unit tests:
+   - `python3 -m pytest trellum/testing/test_components.py -v` — component changes
+   - `python3 -m pytest trellum/testing/test_js_runtime.py -v` — JS runtime changes
+   - `python3 -m pytest trellum/testing/test_interactions.py -v` — theme / rendering changes
+3. Rebuild a couple of this project's existing reports to verify nothing broke —
+   pick ones that between them cover charts, KPIs, filters and any RawHTML:
+   ```bash
+   python3 -m trellum.run reports/<slug> --no-serve
+   ```
+4. For broader coverage, `python3 -m trellum.run --all --no-serve`.
+
+## This repo is a submodule — how changes reach reports
+
+The framework is mounted as a git submodule at `trellum/` inside a consumer
+repository — a report project, or an application that serves several of them.
+Consumers pin a **release tag**, so a merged framework change does not reach
+reports until a release is cut and the consumer bumps its submodule pointer.
+
+Two consequences when working inside `trellum/`:
+
+- **Commit on a branch, push to this repository.** Submodules check out
+  detached HEAD; commits made there belong to `trellum`, not the
+  consumer repo. `git checkout -b feature/x origin/develop` first.
+- **Do not commit the moved submodule pointer on the consumer's main branch.**
+  That pointer is only bumped when moving to a released tag.
+
+Releases are published as **GitHub Releases** (a tag alone is not a release).
+There are no Actions runners for this repo, so it is a manual three-step
+sequence — push the tag, `gh release create --generate-notes --latest`, then
+fast-forward the `release` branch. `__version__` must match the tag.
+
+Branch model, versioning rules, the release procedure, and the consumer bump
+commands are all in [`RELEASING.md`](RELEASING.md). Read it before tagging a
+release or changing `__version__`.
+
+## What NOT to do
+
+- **Do not edit `trellum/*` while building a report.** Reports only touch `reports/{slug}/`. If a task seems to require framework changes, stop and discuss scope with the user — framework changes affect every report.
+- **Do not run `--all` to verify a single change.** Build only the affected report (`python3 -m trellum.run reports/<slug> --no-serve`).
+- **Do not start the preview server to check whether a build worked.** The build prints its own result and the validator writes `output/<slug>/_validation.json`. A bare `trellum.run` blocks until killed; `--no-serve` exits.
+- **Do not write custom Chart.js code for any chart type listed above.** Use the framework component.
+- **Do not hardcode colors.** CSS variables for HTML/CSS, `getThemeColors()` for Chart.js.
+- **Do not import Chart.js or other JS libraries.** Bundled in `trellum/static/vendor/`.
+- **Do not write connection / credential code.** Use `ctx.get_connection()`.
+- **Do not use Chart.js native `legend: { display: true }` on non-doughnut charts.** The `fwLegend` plugin handles it.
+- **Do not open `output/<slug>/index.html` as `file://`.** `data.json` is fetched via XHR and requires HTTP — the runner serves automatically on :8050.
+
+## Licensing (applies to every change)
+
+This repository is MIT licensed.
+
+- **Do not add per-file licence headers.** There are none, deliberately;
+  `LICENSE` governs the tree. A half-headered repository is worse than either
+  consistent choice.
+- **Adding a dependency means updating `THIRD-PARTY.md` in the same commit.**
+  `docs/LICENSING.md` has the audit command. Anything permissive is fine;
+  anything copyleft or source-available needs a conversation first, because it
+  changes what a downstream user may do with a build of this repository.
+- **Bundling is the stricter case than depending.** Code added under
+  `static/vendor/` ships inside every copy of this repository, so its notice
+  travels with it — record it in `static/vendor/MANIFEST.json`.