@antelopejs/dms-ai 0.8.0 → 0.9.0

package/skills/dms-external-database/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: dms-external-database
-description: Connects a DMS project to a client's existing database, runs the Builder's analysis of it, settles what the analysis could not decide from the catalog and aggregates, generates TableViews through the Builder, and adds stats pages over them. Use when the user wants pages over a database the project does not own, PostgreSQL, MongoDB or RethinkDB ("plug in our client's Postgres", "map their Mongo database", "generate the admin for this database", "finish the mapping the Builder proposed").
+description: Connects a DMS project to a client's existing database, runs the Builder's analysis of it, settles what the analysis could not decide from the catalog and aggregates, generates TableViews through the Builder, names them and gives each the table features its data supports, and always builds an analytics dashboard first in the menu. Use when the user wants pages over a database the project does not own, PostgreSQL, MongoDB or RethinkDB ("plug in our client's Postgres", "map their Mongo database", "generate the admin for this database", "finish the mapping the Builder proposed").
 icon: i-ph-database
 category: Builder
 tags: [Database, Builder, Pages, DMS]
@@ -10,8 +10,8 @@ tags: [Database, Builder, Pages, DMS]
 
 The Builder analyses the database by itself: from the catalog and the profiles, fixed rules
 propose a mapping and name every choice's rule. Your work is what those rules cannot do: answer
-the questions they left open, revisit the defaults the names and comments contradict, and make the
-stats pages. You never start from scratch.
+the questions they left open, revisit the defaults the names and comments contradict, and plan the
+analytics pages from the data. You never start from scratch.
 
 You work through the `Database*` and `Builder*` tools only. You never read rows of the client's
 database: the catalog and aggregates are what you have, and they are enough. You never see or
@@ -91,17 +91,48 @@ profile of the whole database again: the Builder did it.
   neither is an array of ids as a relation.
 - **Editable.** Every TableView is proposed read-only. Make one editable only where the integrator
   wants it; show `writes` beside it, so a read-only role in this environment is visible.
-- **Stats pages,** which the analysis does not make, one per area:
-  - KPIs: row counts, and sums of the `measure` columns that add up (a total, a quantity; never
-    a unit price or a rate);
-  - one series over the main time axis by month, in the project's time zone;
-  - top lists over `enum-like` columns.
+- **Pages people can work with.** Load the `dms-page-design` skill. For every TableView, plan
+  from its columns, profiles and categories, as *Name every page* and *Tables* there say: a
+  title, an icon and a description; the tabs, saved views, display modes, footer totals and
+  KPI cards its data supports. Decide each from the data, and leave out what the data does
+  not support.
+- **The analytics dashboard,** which the analysis does not make. Every run ends with one,
+  first in the menu, even when the integrator only asked for the tables. Plan it from the data:
+  1. **Inspect.** For each relation, read from the proposal its rows, its `time-axis` columns
+     with their `range`, its `measure` columns, and its `enum-like` columns with their
+     `categories`. Profile a column the proposal leaves without them, such as the main date
+     column's range. Write each range down: the period rule needs the last date.
+  2. **Ask.** Write the 4 to 8 questions the owner of this data asks every week: how revenue
+     or volume trends, what sells, who buys, where, and how it compares with last year. Each
+     question names the relation, the measure (a sum of a `measure` column or a row count)
+     and the grouping (the time axis or a column of the same relation).
+  3. **Plan pages.** The relation with a time axis, a measure and the most rows is the main
+     fact relation. It gets the dashboard, named for its area ("Sales", or "Overview" when it
+     also carries the other relations' headline figures), holding every question about it.
+     Another relation gets its own analytics page only when it has its own time axis and at
+     least 100 rows. When no relation has a time axis, the dashboard still exists: KPI cards
+     of counts and sums, and breakdowns by the category columns, with no period selector.
+  4. **Choose blocks** for each question with `dms-page-design`, and its default period from
+     the main time axis's real range.
+- **The minimum.** Your plan, and the pages you write, meet all of these:
+  - one analytics dashboard, first in the connection's menu (`order: 0`), opened by a toolbar
+    with a period selector when the data has dates;
+  - every page with a title in the owner's words, an icon and a description, never the raw
+    relation name;
+  - every table given the tabs, views, display modes, footer totals and KPI cards its data
+    supports, as `dms-page-design` *Tables* decides;
+  - at least 3 different chart kinds on it when the data has the shapes for them;
+  - a default period computed from the data's real dates, never assumed;
+  - every page built with one `BuilderComposePage` call, previewed first, and every block whose
+    preview is empty, zero or failed dropped or changed;
+  - every page opened with `NavigateToPage` and its block report read before you report.
 - **Not possible yet:** breakdowns by a related entity's name, such as revenue per customer.
   Templates read one relation, so they would show ids. List them under *Not possible yet*,
   never as pages.
 
 Write your plan (`ExitPlanMode`) in the format of REFERENCE.md *Plan*: what you change in the
-proposal and why, the stats pages, the open items you leave, and what is not possible yet.
+proposal and why, the questions and the analytics pages that answer them, the open items you
+leave, and what is not possible yet.
 
 ## 4. Validate
 
@@ -110,18 +141,29 @@ again. Write nothing before Run.
 
 ## 5. Apply
 
-1. One `BuilderMapResources` with every relation of the settled proposal, TableViews and
-   stats-only relations alike, and the relations decided against in `excluded`
-   (REFERENCE.md *Mapping*).
-   - Send the proposal's choices as they stand where you changed nothing.
+Work in this order, so the most visible result lands first: map the tables, build the
+dashboard and check it, then add KPI rows above tables, then any second analytics page.
+
+1. `BuilderMapResources` in batches of 3 or 4 relations per call, TableViews and stats-only
+   relations alike; the relations decided against go in `excluded` on the first call
+   (REFERENCE.md *Mapping*). One call with every relation of a large database can abort.
+   - Send the proposal's choices as they stand where you changed nothing, plus per resource
+     the `title`, `icon`, `description` and `tableOptions` you planned.
    - On `invalid_config`, fix the choice it names and call again: a refused call writes nothing.
+     When the same choice is refused twice, drop it (send the relation without that display or
+     option) and say so in your report, rather than trying more variants.
    - It answers each relation's `ref`, which queries name it by, `category`, the connection's
      category ref (`pages.<connection>`), and `leftOut`: carry those into your report.
-2. Per stats page, `BuilderCreatePage` with `module` set to the generated module and `category`
-   set to the `category` MapResources answered. Then `BuilderAddQuery` per query, and
-   `BuilderAddBlock` per block with the query's route as its `fetchUrl`, as in the
-   `dms-builder-safe` skill's *Queries*.
+2. Per analytics page, one `BuilderComposePage` call with `create` (`name`, `displayName`,
+   `module` set to the generated module, `category` set to the `category` MapResources
+   answered, an icon, a description, and `order: 0` for the dashboard, 1, 2… for the next
+   ones) and every block in the order `dms-page-design` lays out, each data block with its
+   query inline. Preview first, then write, as `dms-page-design` *Build a page in one call*
+   says.
    - Queries read the mapped resources by their refs (`<connection>/<schema>/<relation>`).
+3. Per table page that gets KPI cards above its table, one `BuilderComposePage` call with
+   `placement: { "before": "table" }`, previewed first. A table option MapResources refused
+   comes back as a warning: fix it with `BuilderConfigureTableView`.
 
 To change a relation later, read its current choices with `BuilderMapping`, then call
 `BuilderMapResources` with that relation alone, its included columns as they are and only your
@@ -137,11 +179,15 @@ change made: a mapped relation is rewritten from the choices sent, and its page
     investigate further.
 - `QueryLogs` for registration warnings (lossy fields, unserved indexes) and for the errors of
   the routes the pages called.
+- On each analytics page, fix or remove every block `NavigateToPage` reports as empty or
+  failed, and navigate again, as in `dms-page-design` *Check every block*.
 
 Then report:
 
 - what the Builder mapped by its rules, and what you changed;
-- the pages made;
+- the pages made, with each analytics page's default period and the date range it came from,
+  and per table the features you gave it and why;
+- every block that still shows no data, and why;
 - what was left out and why;
 - the open items left, which the integrator finds in the Builder's *Database* workspace;
 - the variable every deployment must set.

package/skills/dms-page-design/SKILL.md

@@ -0,0 +1,192 @@
+---
+name: dms-page-design
+description: Designs DMS pages that show data, through the Builder tools. Use when building or changing an analytics, stats or dashboard page or a table page, choosing a chart or a KPI for some data, picking a page's period, naming a page and its icon, or deciding how a table displays (badges, prices, relation labels, status tabs, saved views, kanban, cards, footer totals, KPI cards above the table).
+icon: i-ph-chart-pie-slice
+category: Builder
+tags: [Pages, Charts, Dashboard, Tables, DMS]
+---
+
+# Designing data pages
+
+A data page answers the questions its owner asks about the data. Each block answers one
+question, and its title says which ("Revenue per month", "Top 10 countries by revenue").
+Build every page from the questions, the shapes below and the checks at the end, in that
+order, so that two runs over the same data produce the same page.
+
+This skill decides **what** to build. The tools and their arguments are in `dms-builder-safe`
+(safe mode) and `dms-pages` (code); `dms-best-practices` still holds.
+
+## Pick the block from the data's shape
+
+| The data | Block | Query |
+| --- | --- | --- |
+| A measure or a row count over time | `ChartCard` around `ChartArea`, full width | `series`, `groupBy` the time axis, `bucket`, `response: "card"` |
+| A second series over time (orders beside revenue) | `ChartCard` around `ChartColumn` or `ChartLine` | as above |
+| A column with 2 to 6 values (status, segment, media type) | `ChartCard` around `ChartDonut`, with `showSeriesLegend: true` | `series`, `groupBy` the column, `response: "card"` |
+| A column with 7 or more values, a top N | A bare `ChartBar`, or a `TopListCard` with `showBar` | `series`, `orderBy: "measure"`, `direction: "desc"`, `limit: 10`; `response: "card"` for the chart, `"items"` for the list |
+| One figure with its trend (revenue, orders, customers) | `KpiCard` with `showDelta`, `showSparkline` | `series` by the time axis (never `count` or `aggregate`, which answer no previous period and no sparkline), `response: "value"`, `compare: true` |
+| Numbers used as categories (a rating 1 to 5, a year) | `ChartCard` around `ChartColumn` with `xaxisType: "category"` | `series`, `groupBy` the column |
+| A status column on a table | Tabs on the `TableView`, plus a donut on the analytics page | see *Tables* |
+
+- Put a chart in a `ChartCard`: it adds the headline figure. In safe mode its `chart` is a
+  nested block value, `{ "$block": { "type": "ChartArea", "config": {} } }`. A horizontal
+  `ChartBar` stays bare: inside a card its category labels read `NaN`.
+- The `response` must match the block: `card` for a `ChartCard` or a bare chart, `value` for a
+  `KpiCard`, `items` for a `TopListCard`. Any other pairing draws nothing.
+- Compare periods on KPI cards and top lists (`compare: true`), never on a chart over time:
+  the previous period's buckets are other categories, so they draw beside the period instead
+  of under it.
+- Sum a measure that adds up (a total, a quantity). Count rows for everything else. Never sum
+  a unit price or a rate.
+- Format money with `valueFormat: "currency"` and a `currencyCode` only when the data names its
+  currency (a currency column, a comment). Otherwise use `valueFormat: "number"`: the default
+  currency is the euro, and a wrong symbol is worse than none.
+- Grouping by a foreign key shows ids. Group by the relation's own columns only; a breakdown by
+  a related entity's name goes under *Not possible yet* in your report.
+
+## Lay out an analytics page
+
+Add the blocks in this order. Every analytics page has items 1 to 4.
+
+1. A `PageToolbar` holding a `PeriodSelector`, first on the page (see *Period*).
+2. One `GridRow` of 3 or 4 `KpiCard`s, each bound to the period.
+3. The main series over time, a full-width `ChartCard`.
+4. A `GridRow` of two breakdowns: a bare `ChartBar` for the largest category column, and a
+   `ChartDonut` or a `ChartColumn` for another column or a second measure over time.
+5. A `TopListCard` for a second category column, when one exists.
+
+Use at least 3 different chart kinds on the page when the data has the shapes for them. The
+chart kinds are `ChartArea`, `ChartLine`, `ChartColumn`, `ChartBar` and `ChartDonut`; KPI cards
+and top lists are not chart kinds. Use a second kind before you repeat one. A page of KPI cards
+alone is never done.
+
+Every card binds the page's period: `periodScope: "page"` on the block, and on its query
+`where` filters on the time axis bound to the request,
+`{ "field": "<time axis>", "op": "ge", "value": { "$param": { "name": "from" } } }` and
+`"le"` with `"to"`. A card over a relation with no time axis stays unbound.
+
+## Period
+
+The default period is computed from the data's real dates, never assumed. Read the time
+axis's `range` from the analysis, or profile that column. Then take the first rule that holds,
+counting from today:
+
+1. The data reaches into the last 30 days and spans at least 90 days: `last-90-days`,
+   compared with `previous-period`.
+2. The current year holds at least 3 months of data: `ytd`, compared with `previous-year`.
+3. Last calendar year holds data: `last-year`, compared with `previous-year` when the year
+   before it holds data too, else `none`.
+4. Otherwise bind nothing to a period, leave the selector out, and say so in your report.
+
+Set it on the selector: `defaultPreset`, `defaultComparison`, `presets` (the default, any
+other preset whose window holds data, and `custom`) and `comparisons`. Leave `all` out of
+`presets`: it sends no `from` or `to`, and every bound query fails without them.
+
+Pick the bucket so the default window holds 8 to 31 points: `day` up to 31 days, `week` up to about 7 months (a 90-day window is 13 weeks), `month` beyond. A time chart with fewer than 8 points reads as a curve with no story. An axis
+chart (area, line, column, bar) over time takes `xaxisType: "category"`, unless a preview's
+`x` values are epoch milliseconds, which take `xaxisType: "datetime"`. A donut or pie takes
+no axis option.
+
+## Name every page
+
+Every page you create or map gets a title, an icon and a description; a raw relation name is
+never a title.
+
+- **Title**: what the owner calls the content, in their words and plural for lists ("Support
+  tickets", not "support_ticket"; "Sales", not "orders stats"). Drop technical prefixes and
+  suffixes (`tbl_`, `_v2`). An analytics page is named for the area it covers ("Sales",
+  "Customers", "Overview"). Two pages never share a title: when a table page is already
+  "Customers", its analytics page is "Customer insights".
+- **Icon**: a Phosphor icon, `i-ph-<name>`, that a reader links to the content at a glance
+  (`i-ph-users` for people, `i-ph-package` for goods, `i-ph-receipt` for invoices,
+  `i-ph-chart-line-up` for an analytics page). Two pages in one menu never share an icon.
+- **Description**: one sentence on what the page holds or answers, from the data
+  ("Every order with its payment and shipping status").
+- **Order**: the main analytics page first (`order: 0`), then the most used lists. Keep a
+  table page's order at 100 or more unless it must sit among the dashboards.
+
+Set them with `BuilderCreatePage` on new pages, per resource in `BuilderMapResources`
+(`title`, `icon`, `description`, `order`), or with `BuilderConfigurePage` afterwards.
+
+## Tables
+
+A table page is a working tool: the reader finds a row, sees its state, and acts. Read the
+relation's columns with `BuilderResourceStructure` (and the proposal's profiles when there
+is one), then give the table every feature its data supports, and none it does not. Each
+rule below names the shape that earns the feature; decide from the columns and their values,
+never from the relation's name.
+
+**Columns.** A primary key gets no display and is never a row's label. A boolean keeps its
+default display (no badge). A column that a view filters on, a tab splits on, or a footer
+`where` reads must be `filterable` in the mapping; one that a view sorts on, `sortable`.
+Keep the Builder's proposed displays unless the column's meaning says otherwise:
+a column with few distinct values (a status, a type, a channel, a priority, under about 12
+values) shows coloured badges, one tone per value (green for done or good, amber for waiting,
+red for failed or lost, grey for closed or neutral); free text (a subject, a note, a name)
+never shows badges, even when the sample repeats. A money column shows a price. A relation
+column (a `customer_id`, an `order_id`) is titled by its target ("Customer") and shows the
+target's label, never the raw number: map the target relation with a label first, then send
+the relation column as a relation. Raw id columns stay off the list. Send each field's `display` and the relation's `tabs` in `BuilderMapResources`.
+
+**Tabs.** A single-choice column with 2 to 6 values that splits the rows into states people
+work through (a status, a stage) gives one tab per value after "All". One tab column per table.
+
+**Saved views** (`views`). Add 2 to 4 views when a recurring question is a fixed filter on the
+table: rows waiting for action (a status that means "to do"), recent rows (a date
+`greater_than` `{{now-7d}}` or `{{now-30d}}`), rows assigned to the reader (an owner column
+equal to `{{user.id}}`), outliers worth a look (a stock or a score under a threshold read from
+the profile, a high amount). Each view has a short label that states the question
+("Awaiting shipment", "This week", "Low stock"), a sort that puts the first row to handle on
+top, and `count: true` when the number itself matters. Keep views and tabs from repeating each
+other: views for questions that cross columns or dates, tabs for the plain split by state.
+
+**Display modes.** Offer another display when the data reads better that way, and make it the
+default only when it is the main way to work the table:
+
+- **Kanban** (`kanban.groupBy`) when a single-choice status or stage column describes a flow
+  rows move through, with 3 to 7 values and rows that are handled one by one (tickets, tasks,
+  deliveries, applications). Not for a column that only classifies (a country, a category).
+  Kanban works on a read-only table: it shows the flow without moving cards. Leave `draggable`
+  off unless the table is editable.
+- **Cards** (`cards.fields`) when each row is an entity people recognise by a few fields
+  rather than compare across many columns (a product, a person, a place): 3 to 5 fields, the
+  label first, then the badge and the key figure.
+- **Grouped** (`grouped`) when rows are read by day or week (a log, bookings, events): group by
+  the main date column with `by: "day"` or `"week"`.
+
+**Footer totals** (`footer.summary`). When the table has an additive money or quantity column,
+show its `sum`, and a `count` of rows; add a `count` with a `where` on a status value when that
+number is what the reader checks ("Refunded"). Never sum a unit price, a rate or a score.
+
+**KPI cards above the table.** When the relation has a measure or a state worth tracking (an
+amount, rows waiting, a share of a status), put a `GridRow` of 3 or 4 `KpiCard`s above the
+`TableView`, each answering one question the table's reader asks ("Revenue this month",
+"Awaiting shipment", "Refund rate"), with the series and `compare` rules above. A lookup list
+(a reference table, tags, settings) gets none.
+
+Set displays, views and footer per resource as `tableOptions` in `BuilderMapResources`, or on
+an existing page with `BuilderConfigureTableView`. A refused choice comes back as
+`invalid_config` with the setting it names: fix that one and send again.
+
+## Build a page in one call, and check it first
+
+Build every page, or every set of blocks added to a page, with one `BuilderComposePage` call:
+the blocks as an ordered tree (containers carry `children`), each data block with its `query`
+inline (same fields as `BuilderAddQuery`). It writes everything at once and typechecks once.
+Use the single-block tools only for a small fix afterwards.
+
+1. Call it first with `"preview": true` and the default period's `from` and `to` as `args`.
+   It writes nothing and answers each block's status. Change or drop every block marked
+   `empty`, `zero` or `error` (another measure, the whole history for a card that should not
+   follow the period), then preview again. Never write a block over an empty answer.
+2. Call it again without `preview` to write. A refusal lists every refused block by pointer:
+   fix those and call again; nothing was written.
+3. KPI cards above a table: `pageRef` of the table page, `placement: { "before": "table" }`,
+   and the `GridRow` of `KpiCard`s as `blocks`. A later remap keeps them.
+4. After the last page, `NavigateToPage` once per page and fix any block it reports as empty
+   or failed.
+5. Your report names every block that still shows no data, and why.
+
+The page is done when every block has data in the default period, the page meets the
+layout above, it has a title, an icon and a description, and each table uses every feature
+its data supports.

package/skills/dms-page-design/cases.json

@@ -0,0 +1,57 @@
+{
+  "version": 1,
+  "cases": [
+    {
+      "id": "dms-page-design-dashboard",
+      "area": "dms-ai",
+      "style": "clean",
+      "prompt": "Build a sales dashboard over the orders table with the Builder: KPIs, a trend over time and a breakdown by status.",
+      "expect": {
+        "skills": ["dms-page-design"],
+        "acceptable": ["dms-builder-safe", "dms-best-practices", "dms-pages"]
+      }
+    },
+    {
+      "id": "dms-page-design-chart-choice",
+      "area": "dms-ai",
+      "style": "casual",
+      "prompt": "which chart should i use for orders by country, theres like 24 countries",
+      "expect": {
+        "skills": ["dms-page-design"],
+        "acceptable": ["dms-builder-safe", "dms-pages"]
+      }
+    },
+    {
+      "id": "dms-page-design-period",
+      "area": "dms-ai",
+      "style": "clean",
+      "prompt": "The charts on the stats page are empty because the period defaults to the last 30 days and the data stops last year. Fix the default period.",
+      "expect": {
+        "skills": ["dms-page-design"],
+        "acceptable": ["dms-builder-safe", "dms-pages", "dms-best-practices"],
+        "mustMention": ["PeriodSelector"]
+      }
+    },
+    {
+      "id": "dms-page-design-table-badges",
+      "area": "dms-ai",
+      "style": "clean",
+      "prompt": "On the orders table, show the status as coloured badges, the total as a price, and add a tab per status.",
+      "expect": {
+        "skills": ["dms-page-design"],
+        "acceptable": ["dms-builder-safe", "dms-external-database", "dms-pages"]
+      }
+    },
+    {
+      "id": "dms-page-design-not-safe-mode-approval",
+      "area": "dms-ai",
+      "style": "clean",
+      "prompt": "In safe mode the Builder refuses to add a block to the sales page until I approve it. Where do I approve a pending Builder change?",
+      "expect": {
+        "skills": ["dms-builder-safe"],
+        "acceptable": ["dms-builder", "dms-pages"],
+        "avoid": ["dms-page-design"]
+      }
+    }
+  ]
+}