domma-cms 1.6.60 → 1.7.0

package/docs/superpowers/specs/2026-08-19-collection-context-sort-export-design.md

@@ -0,0 +1,330 @@
+# Collection context menu - sort, group, export & copy - design
+
+Extends the public-site right-click menu (`public/js/collection-context.js`) from filter-only
+to filter + sort + group + export + copy + print, with schema-level export permissions
+configurable from the collection editor.
+
+## Goal
+
+The right-click menu on `[collection]` / `[view]` displays currently offers filters and free-text
+search. Add sorting, grouping, export, clipboard copy, print, and a shareable-view link - without
+regressing the one-click speed of the filter path that makes the current menu good.
+
+Secondary goal, surfaced during brainstorming: make export **coherent**. Two export surfaces
+already exist and disagree with each other (see Prior art). This design unifies the dialect.
+
+## Prior art found during brainstorming
+
+Three discoveries reshaped the design. Recorded here because none are visible from the feature's
+own vantage point.
+
+1. **`exportable="true"` already exists.** `server/services/markdown.js:474` (tables) and
+   `:1122`/`:1231` (Collection Browser) already accept a per-placement export attribute, passed
+   through to `T.create` by `public/js/site.js:764`. A schema-level gate must not regress it.
+
+2. **`exportEntries()` already exists.** `server/services/collections.js:475` has served the
+   admin's CSV/JSON export since the collections feature landed. It quotes every cell and escapes
+   `"` as `""` - and nothing else. Object-valued fields export as the literal string
+   `[object Object]`; this is a live bug affecting e.g. `blog-posts.seo`.
+
+3. **Seven admin view files are committed mangled.** `collection-editor.js`, `form-editor.js`,
+   `block-editor.js`, `action-editor.js`, `page-editor.js`, `view-editor.js` and `navigation.js`
+   were readable at `35f0bdd` (2026-03-10) and mangled by `d492e38` (2026-03-12) - identifiers
+   reduced to single letters, JSDoc stripped. The other 81 files under `admin/js` are unaffected,
+   and `scripts/build.js` writes only to `_publish/`, so the current build is not the cause.
+   Restoration is out of scope here (see Deliberately deferred).
+
+## Decisions (from brainstorming)
+
+1. **Export permission lives in the collection schema**, as an `export` block alongside `api`.
+2. **Default on, but role-gated.** Absent config means signed-in users get export and anonymous
+   visitors do not. Public export is an explicit opt-in.
+3. **`exportable="true"` is grandfathered as an explicit public opt-in**, because that is exactly
+   what it means today. No existing page loses its CSV button on update.
+4. **Scope is the wide tier**: sort, group, export, copy, print, copy-link-to-view.
+5. **Menu structure is drill-in panels**, not hover flyouts or tabs.
+6. **The export gate is client-side and soft**, and is documented as such.
+7. **One shared CSV dialect** across the public export and the server's `exportEntries()`,
+   in one shared `.mjs` module imported by both. Fixes the `[object Object]` bug as a side effect.
+
+## Schema
+
+New optional block in `content/collections/<slug>/schema.json`, parallel to `api`:
+
+```json
+"export": {
+  "enabled": true,
+  "access":  "subscriber",
+  "formats": ["csv", "json"]
+}
+```
+
+- `enabled` - default `true`.
+- `access` - `"public"` or a role name. Default: the least-privileged role in
+  `userTypes.getRoleHierarchy()` (the highest level number), so the default is effectively
+  "any authenticated user". Never `"token"`; API tokens are not a browser concept.
+- `formats` - subset of `["csv", "json"]`. Default: both.
+
+### Resolution order
+
+Evaluated per display, first match wins:
+
+1. `export.enabled === false` → no export. `exportable="true"` is ignored.
+2. `exportable="true"` on the shortcode → export offered to everyone (grandfathering).
+3. `export.access === "public"` → offered to everyone.
+4. Otherwise → offered only when `GET /api/auth/me` resolves a user whose role level is at or
+   above `export.access`.
+
+`schema.export` answers *may this collection be exported, and by whom*. `exportable="true"`
+answers *does this placement render a visible button*. They are not redundant; both survive.
+
+### Why the gate is client-side
+
+`server/routes/public.js:138-149` deliberately skips `jwtVerify()` when a page's visibility is
+effectively public, and keys the response cache `role:anon` so all anonymous traffic shares one
+entry. Server-rendering a per-visitor export flag would force JWT verification on every public
+page request and fragment that cache.
+
+This costs nothing, because the gate was never a security boundary: the full dataset already
+ships inside the base64 `data-ctx` attribute. The gate stops a casual visitor being handed a
+download button; it does not stop a determined one reading DevTools. **This must be stated in
+`docs/configuration.md`** so no operator mistakes it for enforcement. A hard gate would require a
+server export endpoint re-deriving rows from posted filter state - deferred.
+
+## Delivery to the client
+
+Two routes, because tables already differ:
+
+- **`data-ctx` displays** - `buildCtxAttr(fields, entries, opts)` gains a third parameter
+  carrying `{slug, export: {allowed, access, formats}}`. Threaded explicitly through all 8 call
+  sites (`markdown.js:289, 508, 535, 566, 604, 666, 759`). **Not** module-level state:
+  `markdown.js` renders shortcodes recursively, so a shared variable is clobbered by a
+  `[collection]` nested inside a `[card]`.
+- **`[data-collection-table]`** - rides the existing base64 `payload` object that
+  `site.js:758` already decodes.
+
+## Menu structure
+
+Drill-in panels: one panel whose content swaps, with a `‹ Back` row. Chosen over hover flyouts
+because it reuses the existing viewport clamp (`collection-context.js:393-395`) unchanged, needs
+no hover-intent timing, survives long-press on touch, and is keyboard-navigable for free.
+
+Root panel, split by scope - `resolveEntry()` already distinguishes the two:
+
+```
+ENTRY-SCOPED (heading = clicked field label)
+  Filter  = "value"
+  Filter  ≠ "value"
+  Filter  > / <  "value"        (numeric/date values only, as today)
+  Copy value
+DISPLAY-SCOPED
+  Sort…
+  Group by…                     (DOM-toggle display types only - see Grouping)
+  Copy…
+  Export…                       (only when resolution order permits)
+  Print
+  Copy link to this view
+  Clear all filters (n)         (only when filters active, as today)
+[ Search… ]
+```
+
+The free-text search input stays pinned at the bottom of the root panel and keeps autofocus.
+Filter remains one click. Sort becomes two.
+
+## Sort
+
+State: `wrapper._dmCtxSort = {field, label, dir}` or `null`. Applied in `apply()` directly after
+the existing filter step:
+
+```js
+let allowedEntries = model.entries.filter(e => filters.every(f => matches(e, f)));
+if (sort) allowedEntries = allowedEntries.slice().sort(comparator(sort));
+```
+
+`.slice()` is required - `model.entries` is the pristine source order and clearing the sort must
+return to it.
+
+Three of the four existing strategies inherit sort with no further work, because each takes its
+order from the array it is handed: table (`_dmTable.render`), timeline (`instance.setItems`),
+carousel (rebuild from pristine slides). Only the DOM-toggle path needs new code: reorder the
+`[data-entry-id]` siblings with `appendChild` in sorted order. Moving nodes rather than cloning
+preserves event listeners and Domma instance references.
+
+**Comparator.** Extract the type-detection ladder currently inlined in `matches()`
+(`collection-context.js:92-99`) into a shared `compareValues(a, b)`: try `parseFloat`, then
+`Date.parse`, then string compare. Both `gt`/`lt` filtering and sorting then share one definition
+of "which value is larger", which otherwise drifts.
+
+**Direction labels follow the detected type** - `A → Z` for strings, `newest → oldest` for dates,
+`low → high` for numbers.
+
+## Grouping
+
+Available **only on display types that use the DOM-toggle strategy**: cards, list, listgroup,
+accordion, block. These render entries as siblings, so a heading element can be inserted between
+groups.
+
+The other three cannot take it, for three distinct reasons:
+
+- **table** - `T.create` has no grouping API; injected heading rows would fight its pagination.
+- **timeline** - `Progression.setItems()` accepts items, not section headers.
+- **carousel** - slides are a sequence; grouped slides are not meaningful.
+
+The `Group by…` row is **hidden** on those types, not disabled - a greyed row invites "why?" on
+every open.
+
+Implementation: group field becomes the primary sort; a `.dm-ctx-group-heading` div is inserted
+before the first entry of each distinct value. An active sort becomes the secondary sort within
+each group. Clearing the group leaves the sort intact.
+
+## Export, copy, print
+
+One serialiser, several sinks:
+
+```js
+serialise(entries, fields, format)   // format: 'csv' | 'json' | 'markdown' → string
+```
+
+| Menu row                   | Format   | Sink            |
+|----------------------------|----------|-----------------|
+| Export → CSV               | csv      | Blob download   |
+| Export → JSON              | json     | Blob download   |
+| Copy → this value          | -        | clipboard (raw) |
+| Copy → this row            | json     | clipboard       |
+| Copy → visible as table    | markdown | clipboard       |
+
+**Row set:** the filtered-and-sorted entries - exactly what is on screen. No "all vs visible"
+choice; the chips above the display already state what has been filtered out. Grouping is
+presentational and does not affect the row set.
+
+**Column set:** `model.fields`, which is already the rendered-field allowlist produced by
+`buildCtxAttr`. Export cannot emit a column the page did not render.
+
+**Filename:** `<slug>-<YYYY-MM-DD>.<ext>`. Requires adding `slug` to the ctx payload.
+
+**Print:** stamp `data-dm-print` on the clicked wrapper, apply a print stylesheet that hides
+everything outside that subtree, call `window.print()`, then remove the attribute. Accordions
+must be force-expanded in print CSS or the output is a stack of closed headers.
+
+**Copy link to this view:** call `syncHash()`, then copy `location.href`. Confirmation via
+`E.toast` when Domma is present, falling back to a transient inline note.
+
+## The shared CSV dialect
+
+Both `serialise(..., 'csv')` (browser) and `exportEntries(slug, 'csv')` (server) must produce
+byte-identical output for the same values. **They share one module** - verified during planning:
+root `package.json` is `"type": "module"`, and although `public/js/package.json` pins
+`"type": "commonjs"`, a `.mjs` file is ESM unconditionally and imports cleanly from server code.
+`tests/services/menu-decor-client.test.js:7` is the existing precedent for the test side; a
+`node -e` import from the server side was confirmed working.
+
+So `public/js/collection-export.mjs` is the single source of truth, imported by
+`server/services/collections.js` *and* served to the browser. No duplication, so no parity
+fixture is needed - the dialect simply cannot diverge.
+
+The **dialect** is shared. The **column set** legitimately differs: the admin export includes
+`id` and `createdAt` and every schema field; the public export emits only the rendered allowlist.
+
+Rules, in order:
+
+1. **Coercion.**
+   - `null` / `undefined` → `''`
+   - `Array` → elements coerced individually, joined with `'; '`
+   - `Date` → ISO 8601 string
+   - non-null non-array `object` → `JSON.stringify(v)`
+   - otherwise → `String(v)`
+2. **Formula-injection sanitising.** If the coerced string matches `/^[=+\-@\t\r]/` **and is not
+   a finite number**, prefix a single apostrophe. The numeric exemption is essential - without it
+   every negative number exports as `'-5`. Applied to `csv` only; `json` and `markdown` are
+   unaffected.
+3. **Quoting.** Every cell is wrapped in double quotes, with `"` escaped as `""`. Matches the
+   existing server behaviour; quoting unconditionally avoids a needs-quoting predicate.
+4. **Line ending:** `\n`. Deviates from RFC 4180's CRLF, but matches what `exportEntries()`
+   already emits - changing it risks breaking operators' existing importers for no user-visible
+   gain. Deliberate.
+5. **Byte order mark:** output is prefixed with `\uFEFF` (U+FEFF) so Excel reads UTF-8 correctly.
+
+Rule 1 fixes the `[object Object]` bug. Rules 2 and 5 are new to both surfaces.
+
+## URL state
+
+The `dmf=` hash payload currently maps display index → filter array. It becomes display index →
+`{f, s, g}` (filters, sort, group). Back-compat is three lines in `readHash()`: if the per-index
+value is an `Array`, treat it as a legacy filter list. Existing shared links keep working.
+
+## Admin UI
+
+The collection editor (`admin/js/templates/collection-editor.html:13-18`) has tabs
+Settings / Fields / API Access / Storage. Export joins the **API Access** panel as a fifth row
+below the four verbs, with a Formats sub-row:
+
+```
+  Export    ☑ Allow export                [ subscriber ▾ ]
+    Formats      ☑ CSV   ☑ JSON
+```
+
+The existing **Read fields** sub-row (`collection-editor.js:236-247`) is the precedent for a row
+carrying an extra control, so Formats introduces no new pattern.
+
+Wiring mirrors the existing shape: `readApiRows()` (`:297-305`) reads `api-<verb>-enabled` /
+`api-<verb>-access` by DOM id; export reads `export-enabled`, `export-access`, and two format
+checkboxes. The three save paths (`:432`, `:463`, `:480`) currently send `api: b` and gain
+`export: <obj>` alongside.
+
+The tab is renamed **"API & Export"** - a user hunting for an export setting will not open a tab
+labelled "API Access".
+
+New code in this file uses **readable identifiers** despite the surrounding mangled style. A file
+that is 5% legible is better than one that is 0%.
+
+## File layout
+
+| File | Change |
+|---|---|
+| `public/js/collection-export.mjs` | **new** - `serialise()`, CSV dialect, download + clipboard sinks. `.mjs` so both the browser and `collections.js` import it |
+| `public/js/collection-context.js` | drill-in panels, sort state, grouping, comparator extraction, menu rows |
+| `public/js/site.js` | pass export config from table payload to the wrapper |
+| `public/css/site.css` | group headings, print stylesheet, panel transition |
+| `server/services/markdown.js` | `buildCtxAttr()` third parameter, threaded through 8 call sites |
+| `server/services/collections.js` | `exportEntries()` imports and uses the shared dialect |
+| `admin/js/templates/collection-editor.html` | tab rename, export rows |
+| `admin/js/views/collection-editor.js` | render + read + save the export block |
+| `docs/configuration.md` | `export` schema block; explicit note that the gate is soft |
+| `docs/markdown-shortcodes.md` | menu capabilities per display type |
+
+`collection-context.js` is 421 lines today and would approach 700 with all of this. Splitting the
+serialiser into `collection-export.mjs` keeps each file focused: one owns menu and display state,
+the other owns turning entries into text.
+
+**Load-order consequence:** `page.html:120` currently loads `collection-context.js` as a classic
+`defer` script, which cannot `import`. It becomes `type="module"` - matching `site.js` (`:119`)
+and `forms.js` (`:128`), which already do exactly this. Module scripts defer by default, so the
+`defer` attribute is dropped. The file is already an IIFE under `'use strict'`, so no other
+semantics change. The `?v=` cache-buster string must be bumped on every touched public asset.
+
+## Testing
+
+Existing suites live under `tests/` by area (`node --test`, concurrency 1).
+
+- **`tests/services/csv-dialect.test.js`** - unit tests for the single shared dialect: ~9
+  awkward values covering formula-leading string, negative number, embedded double quote,
+  embedded comma, embedded newline, array, nested object, accented text, empty.
+- **`tests/services/collections-export.test.js`** - extend for the `[object Object]` regression.
+- **`tests/markdown/collection-ctx.test.js`** - `buildCtxAttr` emits export config; respects
+  `enabled: false`; grandfathers `exportable="true"`; never emits an unrendered field.
+- **Manual, in a real browser** - sort across all 8 display types; grouping hidden on
+  table/timeline/carousel; drill-in on long-press; legacy `dmf=` link still restores; print
+  output; export gate visible signed-in and absent signed-out.
+
+Per project convention, UI behaviour is not claimed as verified without a real browser load.
+
+## Deliberately deferred
+
+- **Restoring the seven mangled admin view files.** Readable source exists at `35f0bdd`, but five
+  months of changes have landed on the mangled lineage, so it is a rename-against-reference job,
+  not a revert. Its own task - it should not ride along with a feature.
+- **A hard server-side export gate.** Would need `/api/collections/:slug/export` re-deriving rows
+  from posted filter state. Only worth it if an operator needs genuine enforcement.
+- **Column visibility toggles** for table displays.
+- **Multi-field sort.** Single field plus an optional group is enough.
+- **Export of grouped structure.** Groups are presentational; exports stay flat.