archaeopteryx 3.9.0 → 3.11.0

package/README.md

@@ -33,6 +33,7 @@ config key live and shows the exact config JSON to copy into your own
 * [Caliciviridae (186 strains)](https://cmzmasek.github.io/archaeopteryx-js/demo.html?tree=caliciviridae_500)
 * [Adenoviridae (321 strains)](https://cmzmasek.github.io/archaeopteryx-js/demo.html?tree=adenoviridae)
 * [Apaf-1 gene family (domain architectures)](https://cmzmasek.github.io/archaeopteryx-js/demo.html?tree=apaf)
+* [Clustergram (heat map + column clustering)](https://cmzmasek.github.io/archaeopteryx-js/demo.html?tree=clustergram)
 * [Nucleotide alignment (600 columns)](https://cmzmasek.github.io/archaeopteryx-js/demo.html?tree=alignment_nt)
 * [Genome alignment (150 × 30,000 columns)](https://cmzmasek.github.io/archaeopteryx-js/demo.html?tree=genome_alignment)
 * [Sequence alignment](https://cmzmasek.github.io/archaeopteryx-js/demo.html?tree=alignment)
@@ -220,7 +221,7 @@ Subtree** opens everything below a node; the tool row's uncollapse-all button
 (the desktop's glyph, lit only while something is collapsed) opens the whole
 tree, and so does **Esc**. The wedge has its apex at the node, one edge
 reaching the clade's nearest tip and the other its farthest, so the shape
-shows how uneven the clade's branch lengths are, as iTOL draws it (one depth
+shows how uneven the clade's branch lengths are (one depth
 step in a cladogram). Its label stands where a tip's would: on the label
 column in the aligned phylogram and on the outer ring in circular, with the
 same guide line; it is filled in the colour most of its tips wear under the current
@@ -370,8 +371,9 @@ Date" comes back as `Collection Date` in every menu; a header that already
 reads as `namespace:name` is kept as it is). From there nothing is special:
 the columns are offered for **Color-by** and **Shape** by the same rules as
 any property, with the same legends; they are **search** fields, typed
-numeric when every filled cell is a number; they appear in the **node data**;
-and they are written into a phyloXML export, so a saved tree keeps them.
+numeric when every filled cell is a number; the numeric ones become **heat
+map** columns; they appear in the **node data**; and they are written into a
+phyloXML export, so a saved tree keeps them.
 Tip names are matched exactly, then case-insensitively; empty cells add
 nothing; a column the tree already carries under the same ref is replaced by
 the table's values. Quoted cells, `#` comment lines and Windows line ends are
@@ -533,6 +535,88 @@ elements, or in a **Nexus** file whose characters matrix accompanies its tree.
 The **Nexus** entry in the Download menu writes the current tree *and* its
 alignment back into one Nexus file (Taxa, Characters and Trees blocks).
 
+## Heat maps
+
+A tree whose tips carry numeric fields shows them as a **heat map** beside the
+tree: one row per tip, one column per field, each cell coloured by its value
+(the columns stand on the tips' common edge; in the **circular** layout that
+edge is a ring, so each column becomes a concentric ring past the labels — the
+unrooted layout has no such edge, and so no heat map). Turn
+it on with the **Heat Map** checkbox under
+Display Data; it is offered whenever the tree has two or more numeric per-tip
+fields.
+
+Every column is painted on **one shared scale**, so a colour means the same
+number wherever it appears — that is what makes a block of related columns
+readable as a block, and it is the point of a heat map rather than a row of
+independent stripes. The scale spans the whole tree, so entering a subtree
+narrows the rows and leaves the colours where they were. The columns that
+appear are exactly the numeric fields the **Color by** menu offers, so the two
+agree about what the tree holds; by default their left-to-right order follows
+the order the file lists them in, which keeps a producer's grouping (core genes,
+then resistance, then prophages) intact even where some tips are missing a
+field — see **Order columns** below for the alternatives.
+
+A cell **nobody filled in** is drawn as an outlined empty box, never as the
+scale's low end: on a presence/absence matrix, reading a missing field as zero
+states the opposite of what the file says. The key beside the scale names it.
+
+**Hover any cell** for its tip, the column and its value — or `not assessed` —
+and the scale it was coloured against. The column names stand under the matrix,
+turned; a matrix with more columns than will fit shows a window and says so
+(`Columns 1–240 of 400`), which the mouse wheel over the matrix scrolls.
+
+**Order columns** (in the Heat Map section of the panel) decides where the
+columns go. The two **clustered** orders put columns that behave alike side by
+side and draw the clustering itself as a **dendrogram above the matrix** — a
+clustergram — and one of them is what a heat map opens on, because reading block
+structure is what a matrix beside a tree is for. *As in the input* is there when
+you want the producer's own grouping back — and, because phyloXML gives every
+node its own property list, its tooltip says whether that really is the input's
+order or only the order most tips agree on. Both clustered orders are
+complete-linkage hierarchical clustering, written to give the same answer as
+R's `hclust(dist(t(m)), method = "complete")`; they differ in what "alike" means:
+
+* **Clustered (co-occurrence)** uses Euclidean distance, the default of R's
+  `pheatmap`, `heatmap.2` and `ComplexHeatmap`.
+* **Clustered (ignoring shared absence)** uses the **Bray–Curtis** dissimilarity
+  (R `vegan`'s `vegdist` default). Euclidean distance has the *double-zero
+  problem*: two genes both **absent** from the same strains are counted as
+  agreeing there, so on a sparse pan-genome the rare genes cluster together
+  merely for being rare. Bray–Curtis drops a tip where both columns are 0
+  instead of scoring it as agreement. On 0/1 data it is exactly the
+  Sørensen–Dice dissimilarity.
+
+Which of the two a tree opens on is decided by its values: **Bray–Curtis where
+the matrix has zeros to ignore and nothing negative, Euclidean otherwise**. A
+zero is precisely the precondition for a double zero to exist, and Bray–Curtis is
+meant for values 0 or more — on a matrix of years or coordinates it reads
+magnitude instead of pattern, and negative values leave some pairs with no
+distance at all. Set `heatmapColumnOrder` to override; an explicit choice is
+never re-derived.
+
+*Alphabetical* and *Frequency* (highest mean value first, over the tips that
+have a value) are there too, and **Reorder columns…** opens a list you can drag
+(or move with the arrow keys) into any order you like. Doing so sets **Manual**,
+and nothing re-sorts a manual order — a sorting mode that quietly undid your
+arrangement would make the arrangement pointless. *Automatic* in that dialog
+hands the order back to the data. A manual order rides in a shared view, so a
+link reproduces the figure exactly; a column it does not name — one added by an
+edit, or a table joined since — follows the ones it does.
+
+Whatever the mode, a blank is never read as 0, and
+the dendrogram is drawn only when it describes the columns actually on screen —
+never over a scrolled window, never over an order that did not come from a
+clustering, and never over the rings, where it would have to bend.
+
+Sørensen, T. (1948) *Biol. Skr.* 5, 1–34 · Bray, J.R., Curtis, J.T. (1957)
+*Ecol. Monogr.* 27, 325–349 · Eisen, M.B. *et al.* (1998) *PNAS* 95, 14863–8.
+
+Heat-map data arrives with the tree, as phyloXML `<property>` elements on the
+tips (`<property ref="meta:recA" datatype="xsd:integer" applies_to="node">2
+</property>`), or from a **metadata table** joined to it on the open page — a
+table's numeric columns become heat-map columns like any others.
+
 ## Time trees
 
 A tree whose nodes carry phyloXML `<date>` elements is drawn against time.
@@ -957,6 +1041,9 @@ copy-pastable JSON.
 | `layout` | `'rectangular'` | The starting layout: `'rectangular'`, `'circular'`, or `'unrooted'`. |
 | `ladderizeTree` | `true` | Ladderize the tree on load: at each node, the larger clade first (any number of children, so a polytomy sorts too). |
 | `showMsa` | tree-derived | Open with the alignment track shown. Default: on when the tree carries an aligned `mol_seq`, off otherwise — an explicit `true`/`false` overrides that. |
+| `showHeatmap` | `false` | Open with the heat map shown. Offered whenever the tree carries two or more numeric per-tip fields, but off unless asked for: almost any annotated tree has such fields, so turning it on by itself would be an opinion about the tree rather than a service. |
+| `heatmapColumnOrder` | tree-derived | How the heat map's columns are ordered: `'document'` (as the file lists them), `'clustered'` (Euclidean), `'clustered-presence'` (Bray–Curtis), `'alphabetical'`, `'frequency'`. The clustered modes also draw the dendrogram. Default: a **clustered** order, with the distance chosen from the values — Bray–Curtis where the matrix has zeros to ignore and nothing negative, Euclidean otherwise. An explicit value always wins and is never re-derived. |
+| `heatmapManualOrder` | `null` | The heat map's columns in your own order, as an array of property refs (`['meta:recA', 'meta:gyrA', …]`). Only read while `heatmapColumnOrder` is `'manual'`. A ref the tree has not got is ignored, and a column the list does not name follows the ones it does. |
 | `showDomainArchitectures` | tree-derived | Open with the domain tracks shown. Default: on when any tip carries a `<domain_architecture>`, off otherwise — an explicit `true`/`false` overrides that. |
 | `domainLabels` | `'domains'` | Where domain names go: `'domains'` (on the boxes), `'legend'` (a card), or `'none'`. |
 | `domainGlow` | `false` | Open with the glow around each domain box on. |
@@ -1505,7 +1592,7 @@ Legend fieldset (Show / Dir / four arrows / R) is gone; so is the shift- or
 alt-click placement it documented. `visualizationsLegendXpos` and
 `visualizationsLegendYpos` still set where they start out.
 
-## The layouts, alignment track and time axes (developer spec)
+## The layouts, tracks and time axes (developer spec)
 
 The 2026 additions beyond the visualization system, specified tightly enough
 to rebuild. All pure logic lives in forester.js under `npm test`; the viewer
@@ -1632,6 +1719,195 @@ tip's row, so a row reads back to its sequence without counting.
 The conservation bar, consensus row and column ruler are a **floating strip**
 (see the time axes below); the residue rows stay with their tips.
 
+### The heat map
+
+Model (`forester.heatmapColumns(tree)` → `{refs:[{ref,label}], min, max}`,
+pure, in `test/heatmap_test.js`): a column is any candidate that
+`forester.visualizationCandidates` already calls a **numeric property** — so
+candidacy is not re-invented here and the refusal rules are inherited — carried
+by at least one **tip**; an internal node has no row, so a ref only internal
+nodes hold is not a column and its values never reach the scale. `min`/`max`
+span every drawn cell of the whole tree. `forester.heatmapValue(node, ref)`
+returns the number or **null**: null for absent, blank and non-numeric alike,
+and for a ref a node carries twice, the first.
+
+Column order is a **consensus** (`forester.heatmapConsensusOrder`), because
+phyloXML gives every node its own property list: tips can list the same columns
+in different orders, and on `docs/data/influenza.xml` 2 of 6 do. Each tip votes
+on the pairs it lists next to each other, the majority direction of each pair
+becomes an edge, and the columns are read off a topological sort of that graph.
+
+Two rules this replaced, both wrong in ways that showed:
+
+* **First appearance** put a core gene at column 40 of 40 — 7 of 100 tips in the
+  pan-genome demo carry no `dnaK` and the first tip is one of them. (Worse, the
+  traversal reaches a flat tree's tips *last*-first, so "first" meant the file's
+  last tip.)
+* **Mean position** fixed that, and then interleaved blocks no tip carries
+  together: tips holding `A,B` and tips holding `X,Y` put A and X both at
+  position 0, giving `X A Y B` instead of `A B X Y`. No tie-break can mend it —
+  the average is what is wrong, because it compares positions measured on tips
+  that share no frame of reference.
+
+Only **adjacent** pairs vote, which keeps this linear in the data; every pair
+would be quadratic in the column count, which is exactly where a wide matrix
+hurts. Transitivity comes from the sort instead. The cost is that adjacency can
+manufacture a cycle where all pairs would not (tips `A,B,C` and `C,A` give
+A→B→C→A), so cycles are broken at the column fewest others wait on rather than
+assumed away — a column is never dropped because the votes disagreed. Where the
+graph is silent the old rule still decides: mean position, then the
+better-attested column, then the ref. None of it depends on the order the tips
+were reached in, which matters because that order follows the tree's current
+child arrangement and `ladderizeTree` rewrites it — measured, one file once gave
+`B A` ladderized and `A B` not.
+
+`'manual'` is the reader's own order (`forester.heatmapManualOrder`), and the
+only mode that is **not** normalised first — being left alone is the whole point
+of it. It is set by the **Reorder columns…** dialog, which drags or arrow-keys a
+list of the columns as drawn; applying it writes `heatmapManualOrder` and sets
+the mode, exactly as the desktop's Annotation Fields arrows switch a tab to
+Manual. (Theirs also chooses which fields are columns and of what type; ours
+takes every numeric field automatically, so the dialog is about order alone.)
+Choosing Manual from the menu with nothing arranged yet freezes the order on
+screen — an empty list would mean "no opinion" and silently re-derive the very
+order the reader asked to keep. A manual order offers no dendrogram, because
+nothing clustered it.
+
+The mode a tree opens on, when the caller did not say, is
+`forester.heatmapDefaultOrder`: `'clustered-presence'` when some value is 0 and
+none is negative, `'clustered'` otherwise, `'document'` when there are no values
+at all. Measured, which is why the rule is not simply "always Bray–Curtis": on a
+latitude/longitude matrix 2 of 6 pairs come out `+Infinity` (no distance, so
+they join last arbitrarily), and a year column among small ones sits at 0.9995
+from every one of them — maximally distant for being large rather than for any
+pattern, while *within* a block of comparable magnitude Bray–Curtis is exactly
+right. The desktop fixes its default at Euclidean instead, which is consistent
+with its columns being hand-picked where ours are every numeric field the tree
+carries. The resolved choice is cached on the model, never written back into the
+state, so the next tree in a multi-tree file does not inherit this one's answer.
+
+Gate (`heatmapShown`): `showHeatmap` state (default **false**) AND **not** the
+unrooted layout AND at least `HEATMAP_MIN_COLUMNS` (2) columns with values —
+one column is a stripe, not a matrix. Rectangular draws the track, circular the
+rings (`heatmapCircular`).
+
+Geometry: the matrix reserves `HEATMAP_TRACK_GAP(8) + band` from `_w`, where
+`band = min(columns × 14 px, clamp(viewportWidth × 0.45, 60 px, whatever leaves
+the tree ≥ 220 px))`. It is budgeted **before** the alignment, and the
+alignment's own band then yields to it: the matrix wants a fixed, finite width
+while the alignment's band is a window that scrolls and so loses nothing by
+giving way. (Budgeted the other way round, an 18-column matrix got a 60 px
+sliver.) It sits between the domain tracks and the alignment, its right edge at
+`displayWidth − rootOffset − msaReserve`. Rows tile the cluster height by the
+same once-derived midpoints the alignment uses. Column width is
+`clamp(band / columns, 3, 14)`; past `band / 3` columns the matrix shows a
+window, scrolled by the wheel and stated in the caption.
+
+Colour: `d3.scaleLinear().range(VIS_COLOR_RAMP).domain([min, mid, max])` — the
+same 3-stop viridis the numeric visualizations use — built once and kept with
+the model, so every redraw and every view paints the same colours. A degenerate
+range (one value everywhere) takes the ramp's middle stop rather than mapping a
+zero-width domain. Same-colour runs merge into single rects.
+
+A blank cell is the background **with an outline**, and a run of them stays a
+run of cells wherever a column is at least 7 px wide. Leaving a blank bare is
+not enough: measured on the sparse demo, a bare blank stands at a contrast
+ratio of 13.4 against the scale's low end in the light theme but **1.08** in
+the dark one — that is to say, in the dark theme "not assessed" and "zero" were
+the same picture.
+
+**Column order** (`forester.heatmapOrder(tree, columns, mode)` →
+`{columns, dendrogram}`, pure, in `test/clustering_test.js`): the desktop's
+*View → Order Matrix Columns*, minus its Manual mode, which means "the order you
+dragged the rows into" and there is no drag-to-reorder dialog here. Every
+data-driven mode first normalises to **document order**
+(`forester.heatmapInDocumentOrder`, the same consensus over exactly the columns
+given, with columns no tip carries at the end) so a result cannot depend on the
+order the columns happened to be in. Measured: without it, the 6×6 linkage
+fixture fed in reverse clustered to `h6,h5,h3,h4,h2,h1` instead of R's
+`h6,h3,h5,h4,h1,h2` — the index order drives the tie-break and the leaf order,
+so the incoming order really does leak through. The normaliser deliberately does
+**not** go through `heatmapColumns`: that applies candidacy rules (a constant
+column is refused), and ordering must not move with them.
+
+Distances are `forester.heatmapEuclideanDistances` (R's `dist` convention for
+missing values: pairwise deletion, the squared sum scaled up by `tips / used`;
+a pair sharing no assessed tip is `+Infinity`, where R returns `NA` and its
+`hclust` then refuses) and `forester.heatmapBrayCurtisDistances`
+(`sum|x−y| / sum(x+y)` over the jointly assessed tips — being a ratio it needs
+no scale-up; three cases `vegdist` cannot answer are decided so clustering never
+sees a `NaN`: no shared tip → `+Infinity`, 0 at every shared tip → 0, a
+non-positive total with columns that differ → `+Infinity`).
+`forester.heatmapCompleteLinkage` returns R's `hclust` object — `$merge` node
+ids (negative = singleton, positive = the cluster formed at that stage), the
+`$height` each merge happened at, and `$order` — reproducing R's tie-break
+(first pair the `(i, j)` scan finds) and `hcass2` leaf order.
+
+**The expectations are R's own output**, R 4.5.3 and vegan 2.7-2, carried over
+from the desktop's `MatrixColumnOrderTest`, which generated them. Pinning to R
+pins the JS and the desktop to each other, which is the point: the two must
+cluster a matrix identically.
+
+The dendrogram is drawn above the grid in the tree's own group, so it zooms,
+pans and exports with the cells. Heights are **linear in the merge distance**,
+so a block that joins low really is drawn tighter than one that joins high;
+spacing merges evenly by rank would read more clearly and would say something
+false. A merge with no finite height goes to the top of the band, above every
+measurable merge, which is where it belongs: it joined last. It is drawn only
+when the dendrogram's leaves ARE the columns on screen — never over a scrolled
+**window**, whose merges would reach columns the reader cannot see.
+
+Its band is reserved **inside** the layout (the rows are laid out in a span
+shorter by the band and every node moves down by it), not taken off the canvas
+and made up for by the fit: a fit happens once, and switching the order is a
+redraw, so the fit route left the dendrogram 34 px above the top of the window.
+
+**Circular** (`drawHeatmapRings`) is the same matrix in polar form: each column
+a concentric **ring** starting at `maxRad + tipLabelSpace + 7`, each cell an arc
+over that tip's own angular slice, with the slice boundaries derived once
+between neighbours exactly as the rows are. Ring thickness is
+`clamp(maxRad × 1.2 / columns, 1, 14)` — capped against the tree's own radius,
+so a wide matrix cannot turn the tree into a dot at the middle of a dartboard —
+and `fitRadialExtent` allows for the stack. Same-colour neighbours merge into
+one sector, which is what keeps a big fan drawable (281 paths for 18 × 50
+cells on the sparse demo).
+
+Each ring carries its name **tangentially at the fan's seam**, the gap the
+layout already leaves between the last tip and the first: the one place a label
+can sit without covering a cell. Tangentially means rotated by the angle
+itself — `polarXY` puts angle *a* at screen direction *a* − 90°, so rotating by
+*a* − 90 instead lays every name out along the seam, where the rings are one
+ring-width apart and the names several times that. Measured before the fix: 44
+overlapping pairs out of 18 names.
+
+The scale becomes a **corner card** (`drawHeatmapRingLegend`, bottom-left,
+screen coordinates so it neither turns nor scales with the fan), because a ring
+has no "under" to hang a strip from. There is no dendrogram: it would have to
+bend, and the desktop makes the same call. **Unrooted gets no heat map at all** —
+every tip there ends at its own radius, so a column has no ring to be. Hover is
+the same two questions in polar form: which ring, and which slice, the slice
+compared modulo a full turn because the fan rotates.
+
+**Where the readout lands** is one function, `placeHoverReadout(el, event)`,
+shared by the heat map, the alignment track and the node tooltip. It sets the
+text, then measures the element — `.aptx-tip` is `width:max-content`, so its
+size is whatever it is currently saying — and offers it at `pageX + 14`,
+`pageY + 14`, flipping to the other side of the pointer on either axis only
+where that would leave the window (clamped to the edge if neither side fits).
+The heat map and the alignment both used to guess instead, shifting left by a
+hard-coded 280 px against a readout 181 px wide: since both tracks hug the right
+edge, every cell flipped and every readout floated ~95 px clear of the cell it
+described. `test_trees/heatmap_tip.html` pins it, in both layouts, and fails
+unless all four placements were exercised.
+
+In the rectangular layout the turned column names, the gradient, its two
+numbers, the blank key and the caption are a **floating strip**; its reserve adds `MSA_NAV_RESERVE` while the
+alignment is shown, or the alignment's navigation bar (fixed at the viewport
+bottom) covers the scale.
+
+Everything is plain rects, lines, text and one `<linearGradient>`, so the SVG,
+PDF and PNG exports match the screen.
+
 ### The time axes
 
 Data model: per-node `date = {unit, desc, value, minimum, maximum}` —

package/archaeopteryx.d.ts

@@ -83,6 +83,20 @@ export interface ArchaeopteryxConfig {
     rootOffset?: number;
     searchAinitialValue?: string | null;
     searchBinitialValue?: string | null;
+    /** Open with the heat map shown. Default false: it is offered whenever
+     * the tree has two or more numeric per-tip fields, which is most annotated
+     * trees, so it waits to be asked for. */
+    /** How the heat map's columns are ordered. The clustered modes also draw
+     * the dendrogram above the matrix. Default: a clustered order with the
+     * distance chosen from the values — Bray–Curtis where the matrix has zeros
+     * to ignore and nothing negative, Euclidean otherwise. An explicit value
+     * always wins and is never re-derived. */
+    heatmapColumnOrder?: 'document' | 'clustered' | 'clustered-presence' | 'alphabetical' | 'frequency' | 'manual';
+    /** The heat map's columns in your own order, as property refs. Only read
+     * while heatmapColumnOrder is 'manual'. A ref the tree has not got is
+     * ignored; a column this does not name follows the ones it does. */
+    heatmapManualOrder?: string[] | null;
+    showHeatmap?: boolean;
     showMsa?: boolean;
     showSupportDots?: boolean;
     showTimeAxis?: boolean;
@@ -136,6 +150,9 @@ export interface ViewState {
     rotation?: number;
     horizontalLabels?: boolean;
     msa?: boolean;
+    heatmap?: boolean;
+    heatmapOrder?: 'document' | 'clustered' | 'clustered-presence' | 'alphabetical' | 'frequency' | 'manual';
+    heatmapManual?: string[];
     domains?: boolean;
     domainLabels?: 'none' | 'domains' | 'legend';
     domainGlow?: boolean;