openflo 2.6.0__tar.gz → 2.6.2__tar.gz

{openflo-2.6.0 → openflo-2.6.2}/CHANGELOG.md

@@ -6,6 +6,315 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 
 ## [Unreleased]
 
+## [2.6.2] - 2026-10-09
+
+### Changed
+- **Example markers and cutoffs are generic too.** The example panel,
+  synthetic data, CLI defaults, docs and tests now use the markers `CD901b`,
+  `CD902` and `CD903` (they still parse as CD markers) and round example
+  cutoffs (0.5 / 0.4 / 0.45). The privacy gate blocks the real markers.
+- **Examples, fixtures and docs use generic detector names.** Channels in
+  the example panel, synthetic data, CLI defaults, README, tests and golden
+  fixtures are now `FL1-A`, `FL2-A`, ... instead of real fluorochromes, and
+  the comp-control matcher tests use generic dyes (`FLB`, `FLB-Cy7`, ...)
+  with the same tandem structure. The privacy gate now blocks the three
+  most common real ones, except in a functional fluor catalog
+  (`src/openflo/fluor_catalog.py`), which names dyes without revealing a
+  panel. Note: a session or template saved against the old example channel
+  names will not match the renamed example panel.
+- **Repo tidy for the beta.** Removed four empty files at the repo root
+  (`UMAP`, `data`, `events`, `i'`) left by stray shell redirects, and
+  CONTRIBUTING.md now points to `src/openflo/pipeline.py` for
+  `OpenFloError` instead of the nonexistent `flow_pipeline.py`.
+- **The privacy gate now runs on every PR and before every PyPI upload.**
+  It used to run only when the public mirror was refreshed, and the sdist
+  is built from this repo, so nothing scrubbed what PyPI would get. That
+  gap was real: the sdist shipped `scripts/publish_public.py`, whose
+  pattern list names the private tokens the gate exists to block.
+  MANIFEST.in now excludes it, `publish_public.py` gains `--check` and
+  `--check-dist DIR`, and CI's new `privacy gate` job and the release
+  workflow run them.
+- **The side panel is compact, and the plot's controls fit the default
+  window.** At 1500×840 the Samples & Gates panel took 747px, half the
+  window, and the plot's control bar was cut off ("Color" showed one
+  letter).
+  - The panel is now 372px and the plot pane 1116px.
+  - The action buttons use tighter padding and sit in three columns.
+    Undo and Redo share one cell.
+  - The Pipeline button reads "Pipeline (beta)".
+  - A "Layout (to be added)" button marks the place of a FlowJo-style
+    layout editor, which is not built yet. Clicking it says so and offers
+    File → Figure layout…, which builds multi-panel figures from preset
+    arrangements.
+  - Showing the log/console no longer widens the panel.
+  - The tree's "All" heading is no longer clipped.
+- **Acquisition QC on load is off by default.** It is a new Preferences
+  option: Loading → "Run acquisition QC on load". QC removes events from
+  time bins where the signal drifts or the flow rate jumps, and events piled
+  at a channel's ceiling, before any gate. FlowJo has no such step, so with
+  QC on, the same gates count fewer events in OpenFlo than in FlowJo.
+  - Turning the option on applies to files loaded afterwards.
+  - A sample keeps the choice it was loaded with when it is re-compensated,
+    reopened from a session, moved to another window or reloaded by the
+    Workspace.
+  - Sessions saved before this change reload their samples with QC, as they
+    were loaded.
+  - `openflo-run` is unchanged.
+- **The type check gives the same answer locally and in CI** (development
+  only; nothing a user runs changes).
+  - CI and the pre-push hook now run the same `scripts/typecheck.py`, with
+    the same packages: CI also installs the optional backends
+    (`requirements-typecheck.txt`), which it used to leave unresolved.
+  - The script refuses to run (exit 2) when pyright would read a different
+    environment than CI: wrong package versions, a different pyright, or
+    pyright resolving imports from another interpreter.
+  - In a git worktree it uses the main checkout's `.venv`; no junction needed.
+  - A release tag is built only after the same type check passes.
+
+### Fixed
+- **Reference spectra of dim dyes on autofluorescent cells.** The positive
+  population of a dim single-stain control was picked by total brightness.
+  It then included bright-AF cells, which mixed autofluorescence into the
+  dye's spectrum. Example: a dye at 400 on cells where 30% have 10x AF read
+  cosine 0.53–0.98 to its true spectrum, and unmixing error was 0.38–0.72.
+  - Positives are now picked by the dye's own excess in its primary
+    detector. Each event's autofluorescence is subtracted using its nearest
+    unstained neighbours in the detectors where AF dominates. The same case
+    now reads 0.9999 and unmixing error 0.044–0.061, which matches the
+    true-spectrum floor.
+  - Guardrail: the old estimate is kept unless the two estimates disagree
+    (cosine < 0.99) or the control is poorly separated. Over a 576-cell
+    simulation grid, no cell got worse than before. Every bright dye
+    (≥10k) and every control without AF variation gives a bit-identical
+    result.
+  - Each spectrum now says how it was built. Controls too dim to trust are
+    named with a recommendation (beads or a brighter fluor), in the CLI
+    report, the editor status bar and the Spectral QC window.
+  - New `method='auto'|'total'|'matched'` and `beads=` options, and the CLI
+    flags `--unmix-spectra` and `--unmix-beads`.
+- **Selecting a sample or gate on large files is fast.** At 20M events,
+  clicking a sample row took 1.4 s and clicking a gate row 13 s. Every replot
+  shuffled all events to pick the drawn subsample, re-scanned them for
+  missing values, and recounted each gate's chain twice.
+  - Now 0.2 s for either once a sample has been prepared. The first view of
+    a sample takes 0.4 s, and 1.8 s for a gate row.
+  - Filter mode went from 8.1 s to 0.18 s, Highlight from 3.0 s to 0.27 s,
+    and Histogram from 1.1 s to 0.17 s.
+  - After a change, the displayed sample is drawn first. The other loaded
+    samples are prepared in the background, at about 1.7 s per 20M sample,
+    without blocking the window.
+  - No count, percentage or histogram value changes. The drawn subsample is
+    still fixed per sample, axes and cap, and now also across restarts.
+  - Caches are dropped whenever a sample's data or a gate on its chain
+    changes, and are bounded at 192 MB.
+- **Gate bounds are shown as intensities, not stored logicle values.** A cut
+  at an intensity of about 1,000 read `CD3-A >= 0.453` and now reads
+  `CD3-A >= 987`, in the same style as the axis ticks.
+  - Applies to the gate list, unnamed population names in Statistics,
+    Frequencies and their CSVs, the histogram slider, the auto-threshold
+    status, the backgate legend, the Expression, FMO and Workspace windows,
+    and the `openflo-run` log.
+  - Open sides are said as open (`CD3-A < 2.5K`) instead of printing the
+    ±1e12 sentinel. A channel of unknown scale keeps its stored value,
+    marked "(stored, scale unknown)".
+  - Colour bars and the CLI scatter PNGs are ticked at round intensities
+    (`0, 100, 10^3, 10^4`) instead of `0.2 … 1.0`.
+  - Populations are still matched across samples by gate structure, so
+    Frequencies is unaffected by the new names.
+- **Frequencies: a population missing from a sample is missing, not 0.**
+  - Populations were matched across samples by their displayed path. An
+    unnamed gate's path includes its coordinates, so a cut nudged in one
+    sample, or a marker on another detector, split one population into
+    several.
+  - Differential abundance then counted each gap as 0 events. A gate drawn
+    only in one group's samples read as a population that had vanished from
+    the other group.
+  - Populations now match by gate name, or by kind of gate, markers and
+    position among siblings.
+  - A sample that still lacks a population shows it as missing in the
+    table, the tests, the overview and the exports. Differential abundance
+    fits it on the samples that have it and reports `n_missing`.
+  - %Parent of an empty parent is NaN, not 0%.
+- **Compensation is applied once, with each sample's own matrix.** Reported
+  as "either ignored or applied twice"; both happened.
+  - **Applied twice.** A second compensation call (`auto_compensate` then
+    `compensate_from_wsp`, the same call twice, or `FMOGater.prepare` run
+    twice) compensated values that were already compensated: a channel with
+    0.30 spillover moved by about 15,000 units. A new matrix now replaces the
+    one applied before. Compensating a channel that is already transformed
+    raises `CompensationError` instead of mixing logicle values.
+  - **Wrong matrix or none.** `compensate_from_wsp` applied the workspace's
+    first matrix to every file, and raised on a workspace with no matrix.
+    That affected Compare workspace, `FMOGater` and `Experiment.run_all`. It
+    now uses the matrix inside the file's own `<Sample>`, or the file's
+    `$SPILL` (FlowJo's Acquisition-defined) when the workspace gives it none.
+    Several same-named matrices are no longer collapsed into the last one.
+  - **Editor.** Opening a `.wsp` applied each FCS's `$SPILL` instead of the
+    workspace's matrix for that sample. A reopened session, a sample moved
+    to another window and a workspace re-run silently went back to `$SPILL`,
+    discarding a matrix the user had applied. Each now uses the matrix the
+    sample was compensated with. A sample restored from a processed CSV keeps
+    its matrix and the link to its FCS.
+- **FlowJo gate coordinates are converted to and from OpenFlo's display
+  scale.** A `.wsp` holds gates in linear intensity, while the editor and
+  `openflo-run` keep gates in the stored (e.g. logicle) coordinates. Nothing
+  converted between them:
+  - an imported `FL1-A > 1000` selected 0 events where FlowJo counts 7,653;
+  - an exported logicle threshold of 0.454 reached FlowJo as an intensity of
+    0.454, so FlowJo counted 14,087 events where OpenFlo counts 7,653.
+
+  Import (workspace and template) and export (editor and `--export-wsp`) now
+  convert per channel. Bounds convert exactly. Polygon edges are subdivided,
+  and ellipses are traced as polygons. A channel of unknown scale is left
+  unconverted, with a warning. Importing also keeps sibling gates on one
+  channel: one workspace's 5 gates arrived as 3, with a child moved under
+  the wrong parent.
+- **GeoMean and rCV follow FlowJo's definitions.** These change the
+  statistics table and its CSV.
+  - rCV is now 100 × ½ (P84.13 − P15.87) / median. The MAD form used before
+    read 74% where FlowJo reports 89% on a skewed population.
+  - GeoMean on a transformed channel is the mean in display space,
+    back-transformed. It used to drop every non-positive event, so a dim
+    population read 212 where FlowJo reports 84.
+- **Differential abundance is calibrated.** The negative-binomial test gave
+  a false positive in 24% of no-difference 3-vs-3 comparisons at the 5%
+  level, and in 64–84% for nested gates. It solved the Poisson score and
+  used one dispersion for every population, and nested parent gates pulled
+  that dispersion to almost zero. Each population now has its own NB fit
+  and dispersion, with a likelihood-ratio test on an F(1, df) reference.
+  Simulations give 5–6% false positives, and rows gain a `df` column.
+  One sample per group now gives no p-value.
+- **One fold-change direction: log2(B / A) everywhere.** `differential_test`
+  alone gave A over B, so one comparison showed opposite signs in two
+  windows. A group mean of 0 now gives ±inf, not ±29.2 from a 1e-9
+  pseudo-count.
+- **Absolute counts use the bead and sample volumes and the dilution.**
+  50 µL of beads at 1,000/µL in 100 µL of blood read 2,000 cells/µL where
+  1,000 is right. With no volumes entered, the result is unchanged.
+- **`compare_conditions` compares populations, not cluster numbers.**
+  Samples clustered one by one number their clusters independently, so
+  cluster 0 could be CD4 cells in one sample and CD8 cells in another: a
+  CD4 split of 80% vs 20% read 80% vs 80%. The compared samples are now
+  clustered together first. `openflo-run` already did this.
+- **Compensation matrix files.**
+  - A `$SPILL` that names its channels by parameter number now compensates.
+    Before, nothing matched and the sample stayed uncompensated.
+  - A matrix in percent (diagonal ≈100) is converted, with a warning, from
+    any source. Before, it left every channel at 0.01×.
+  - CSV rows are matched to the header by their labels.
+  - Header-only CSVs from FlowJo now read.
+  - A `.wsp` with no matrix returns `(None, None)`, as documented.
+- **Optimize compensation is unbiased.** It now uses the difference of the
+  positive and negative medians and drops saturated events. On bead peaks
+  the estimate was 0.158 for a true 0.20, and 0.63 with 3% of events
+  saturated; it is now 0.200.
+- **Compensation QC** ranks spillover by magnitude relative to each source's
+  own signal. A −0.35 entry ranked below +0.04.
+- **Spectral unmixing.**
+  - The editor unmixes what the detectors measured, i.e. the values before
+    compensation and transform. It used to read compensated logicle values
+    (median abundance error 36–54%).
+  - The spillover spread matrix uses the standard formula, net of the
+    negative population. It overstated spread 4–6× when the control was
+    mostly negative, and understated it at 0.58× with non-negative
+    unmixing.
+  - `run_qc` drops the same events from `raw`, so `--unmix` no longer
+    unmixes events that QC removed.
+- **Values reported on the linear scale.** Several outputs carried logicle
+  coordinates as intensities:
+  - CLI `<sample>_stats.csv` and `compare_shared_clusters.csv` medians read
+    0.43 / 0.75 where 796 / 20,104 is right;
+  - the trajectory trends export;
+  - the AnnData export, which keeps X and adds a `linear` layer and each
+    marker's transform;
+  - the FMO and auto-gate status lines, which now give the linear cut;
+  - the heatmaps, which now say they are on the display scale.
+- **GPU logicle** no longer clamps values above ~7.4×10⁵ (error 0.16 →
+  2×10⁻⁸).
+- **Scatter, Time and instrument channels are recognised in any case**
+  (`TIME`, `fsc-a`, Beckman `FS INT`/`SS INT`, Sony `BSC-A`, CyTOF
+  `Event_length`…). They were logicle-transformed and clustered as markers.
+- **Switching a channel to 'log'** now asks first and names how many events
+  at or below zero it will empty.
+- **Gating.**
+  - A threshold counts `x >= value`, as Gating-ML and FlowJo do. On integer
+    data events exactly on the cut were lost: 50 vs FlowJo's 60. This
+    includes FMO `_pos` flags.
+  - A gate drawn to the end of an axis is open there, so the negative
+    events piled on the axis floor are inside. A rectangle from the corner
+    counted 271 of 2,543 events.
+  - Axis ticks and the range dialog are in intensity, not stored logicle
+    units.
+  - A `.wsp` template applies one sample's gate tree, not every sample's.
+- **Compare workspace.**
+  - A gate on a bare (uncompensated) parameter is counted on uncompensated
+    values, as FlowJo does: 5,216 → 8,413.
+  - A population under a gate OpenFlo cannot read is reported as an error.
+    It used to be re-rooted and counted.
+- **Cancel stops a Workspace run's whole process tree on Linux and macOS.**
+  Before, it stopped only the job process, so PhenoGraph's worker pool and
+  the Louvain binaries kept running. Windows already killed the tree.
+- **Events removed by the editor's load QC are reported** in the load status
+  and the `.wsp` export warning. FlowJo has no such step, so its counts
+  include them.
+- **Ellipses drawn in FlowJo are imported.** FlowJo stores such an ellipse
+  as foci and edge points in its display space. The import (workspace,
+  template, Compare workspace) now converts it with the sample's own axes:
+  exactly on linear axes, and as a 64-vertex polygon on logicle and biex axes
+  (FlowJo's biex lookup table). Before, it was dropped and each population
+  under it counted every event the ellipse excluded.
+  - An ellipse on any other axis (log, arcsinh, a linear gain other than 1)
+    becomes a new gate kind, `flowjo_ellipse`. It counts no events, its
+    populations stay under it, the gate list and status bar flag it, and an
+    export to FlowJo writes it back as FlowJo drew it, with its axes.
+  - **Session files:** the `.flowsession` keys and schema version are
+    unchanged (v1), and older sessions load as before. A session can now
+    contain a `flowjo_ellipse` gate. An older OpenFlo does not know the kind
+    and lets every event through it, so open such sessions in this version
+    or later.
+- **Single-stain auto-detect** (Optimize compensation). A marker name in the
+  channel no longer decides the match (`CD4 FL3-A` took `FL5.fcs`), `-A` and
+  `-H` channels of one fluor share its control, and joined names match
+  (`AF647`, `CD3FLA`, `eF450`, `compFLA`). A file that two fluors fit
+  equally is reported, not assigned.
+
+## [2.6.1] - 2026-10-02
+
+### Fixed
+- **FlowJo round trip, measured against FlowJo 10.10.2.** FlowJo opened,
+  computed and saved synthetic probe workspaces and OpenFlo exports with no
+  manual steps; each FlowJo behaviour below comes from those saved files.
+  - **Ellipse gates.** A workspace with an ellipse gate lost every sample that
+    carried it, with no message. Ellipses are now exported as 64-vertex
+    polygons.
+  - **Quadrants.** A quadrant was exported as a QuadrantGate, which FlowJo
+    counts as the whole parent. It is now four rectangle gates in FlowJo's own
+    form, with each open side left unbounded, and a child gate stays under its
+    own sector. Importing FlowJo's own quadrants also failed: their sectors
+    have one bound per side, and the reader skipped every such rectangle.
+  - **Fluorescence gates were applied to uncompensated data in FlowJo.** They
+    were exported under the raw channel name, which FlowJo reads as the raw
+    parameter. They are now written as `Comp-<channel>` for each channel the
+    sample's own matrix compensates.
+  - **Per-sample `<Keywords>`.** Without the FCS TEXT in each sample, FlowJo
+    keeps the sample but computes none of its populations.
+  - **Time gates were 100× off** across the `.wsp` boundary. FlowJo works in
+    seconds: ticks × the FCS file's own `$TIMESTEP`, or ($ETIM − $BTIM) / last
+    tick without one, even when the workspace says otherwise. Export and
+    import now convert the same way.
+  - **Compensation.** `read_compensation_matrix` found no spillover in any FCS
+    file, because flowio lower-cases keywords. It now reads SPILL, and prefers
+    it over `$SPILLOVER`, as FlowJo does. Each exported sample carries its own
+    matrix, so a run that mixes files with different spillover no longer gets
+    one file's matrix applied to all.
+  - **Import of a lower edge near the bottom of an axis.** FlowJo treats such
+    an edge as open. Imported gates now do the same, so an imported gate no
+    longer undercounts by the events below the axis. The writer warns about
+    each exported edge FlowJo will open.
+  - Polygon counts still differ: by about 2% for scatter polygons and 3-7%
+    for exported ellipses. See "Where FlowJo and OpenFlo count differently" in
+    the README.
+
 ## [2.6.0] - 2026-09-10
 
 ### Added
@@ -503,7 +812,7 @@ filename, and the loader looks for both.
   40-population workspace arrived fully unnamed while its geometry survived
   intact, which is why a round trip looked like it worked. A compensation
   matrix whose *channel name contained a comma* (a `$PnS` description like
-  "CD4 PE-Cy7, clone SK3" is ordinary) wrote a file this same module then
+  "CD4 FL3, clone SK3" is ordinary) wrote a file this same module then
   refused to read. Standards-conformant Gating-ML `-INF` bounds reached the
   session file as `-Infinity`, which *RFC 8259 forbids* — valid to Python,
   rejected by `JSON.parse` and `jq`. An *unusable reference spectrum* (a
@@ -736,7 +1045,7 @@ filename, and the loader looks for both.
 - **The golden baseline's compensation metric never ran compensation.** It
   wrote the generator's own planted spillover matrix to a CSV, read it straight
   back, and asserted the value it had just written — a metric labelled
-  "Compensation APC -> APC-Fire spill" that could not observe compensation
+  "Compensation FL2 -> FL8 spill" that could not observe compensation
   being broken, and which therefore stayed green through the 2.2.1 transpose
   bug that corrupted every real compensated analysis. It now applies
   compensation to data with a known spillover baked in and reports two numbers
@@ -744,7 +1053,7 @@ filename, and the loader looks for both.
   error against the true signal, relative to signal scale) and
   `signal_retained` (which a "zero everything" implementation would fail while
   flattering the residual). Verified to catch a transposed matrix, a zeroing
-  implementation, and a no-op. **Baseline keys changed**: `compensation.apc_leak`
+  implementation, and a no-op. **Baseline keys changed**: `compensation.fl2_leak`
   is replaced by `compensation.residual_err` + `compensation.signal_retained`.
 - **`--update` could bless a bug as the baseline.** Re-pinning rewrites every
   golden value from the current run, so running it while a bug was live
@@ -1051,7 +1360,7 @@ filename, and the loader looks for both.
   sample. Malformed or incomplete gate dicts are reported rather than discarded
   in silence.
 - **CLI `--gates` help documented a form the parser rejects.** Copying the
-  documented `{"BV421-A":0.5}` example ran the analysis **ungated**; the help now
+  documented `{"FL1-A":0.5}` example ran the analysis **ungated**; the help now
   shows the accepted JSON-list form.
 - **`--export-wsp` resolved zero samples** for a comma-string or per-sample-dict
   `samples` spec (it iterated the string or dict). It now normalises groups
@@ -1190,7 +1499,7 @@ filename, and the loader looks for both.
   - Session restore: a corrupt per-channel display range is skipped instead of
     aborting the whole load; auto-saved sessions are schema-migrated on resume
     (same as a manual open).
-  - FlowJo import: gates on natively-compensated channels (FlowJo's `<PE-A>`
+  - FlowJo import: gates on natively-compensated channels (FlowJo's `<FL5-A>`
     bracket form) now match the data instead of selecting everything; the
     region/CLI gate filter evaluates all gate kinds (ellipsoid / cluster /
     category / boolean), not just lines and polygons.
@@ -1819,10 +2128,10 @@ filename, and the loader looks for both.
   cells along a differentiation trajectory: a symmetric kNN graph over the
   enabled samples' shared fluor channels (concatenated, so a day-series becomes
   one continuous trajectory), with pseudotime = geodesic distance from a root
-  cell chosen at the extreme of a marker (e.g. CD34-high progenitors as t=0).
+  cell chosen at the extreme of a marker (e.g. CD902-high progenitors as t=0).
   It writes a ``pseudotime`` column to every sample (selectable as a plot
   colour) and draws each marker's mean expression along pseudotime — the
-  CD34-down / CD11b-up maturation curve — with CSV / Prism XY / figure export.
+  CD902-down / CD901b-up maturation curve — with CSV / Prism XY / figure export.
   Backend (`openflo.trajectory`: `compute_pseudotime`, `robust_root`,
   `pseudotime_trends`) is pure (numpy/scipy/sklearn), subsamples large data for
   the graph and propagates by nearest neighbour, and is exported.
@@ -1883,7 +2192,7 @@ filename, and the loader looks for both.
   the current plot into a publication-style small-multiples figure: one panel
   per sample (current channels), one panel per channel pair (samples overlaid),
   a samples × pairs grid, or a single panel. Channel pairs accept marker labels
-  or channel names (e.g. `CD34/CD11b, CD11b/CD45`). Each panel reuses the live
+  or channel names (e.g. `CD902/CD901b, CD901b/CD903`). Each panel reuses the live
   rendering pipeline (mode, density/colour, axis scales, gates) via an
   axes-swap (`_render_into`), so panels match the on-screen plot exactly. A
   preview window saves to PNG / PDF / SVG / TIFF at 300 dpi, with a
@@ -1918,7 +2227,7 @@ filename, and the loader looks for both.
   reads a CD↔fluorophore sheet and maps each fluorophore to its detector
   channel, merged with `--labels`. New `read_staining_panel` / `find_panel_xlsx`.
 - **Per-group marker-pair scatters (CLI).** Every group now emits, for each
-  pair in `--pairs` (default CD34/CD11b, CD11b/CD45, CD34/CD45), an *overlay*
+  pair in `--pairs` (default CD902/CD901b, CD901b/CD903, CD902/CD903), an *overlay*
   (all samples on one axes, coloured by sample) and a *grid* (one density panel
   per sample, shared limits). New `save_group_pair_scatters`.
 - **Adjustable plot point cap.** A **Max points** control (presets + free
@@ -2188,7 +2497,7 @@ First public release.
   stamps each gate's channel with its antibody label; applying that
   template to a sample where the marker sits on a *different* detector
   retargets the gate to that sample's detector (`relabel_gate_for_sample`).
-  So a CD11b gate applies wherever CD11b lives in each sample, across
+  So a CD901b gate applies wherever CD901b lives in each sample, across
   panels — compensation is unaffected (only which column the gate
   reads changes).
 - **Per-sample FMO override in the config GUI.** A group's samples
@@ -2206,7 +2515,7 @@ First public release.
   the common label set, renaming each sample's fluors to labels). The
   statistics table now names per-channel columns by **each sample's own**
   label, so a marker on different fluors merges into one column
-  (`Median CD11b`) across samples. The editor flags a non-common fluor
+  (`Median CD901b`) across samples. The editor flags a non-common fluor
   panel on sample load and in the Statistics window; non-common labels
   are simply blank where absent.
 - **Population statistics table (FlowJo-style).** A "Statistics…" window
@@ -2214,7 +2523,7 @@ First public release.
   as the cumulative gate chain): Count, %Parent, %Total, and per-channel
   Median / Mean / CV. Columns are modular (checkbox toggles); the table
   exports to analysis-ready CSV. Populations show a FlowJo-style path
-  (`Cells/Singlets/CD11b+`). Computed on full sample data, not the plot
+  (`Cells/Singlets/CD901b+`). Computed on full sample data, not the plot
   downsample.
 - **Full FlowJo gate parity — ellipsoid + quadrant.** `WspReader`
   parses `EllipsoidGate` (mean + covariance + distanceSquare) and
@@ -2345,7 +2654,7 @@ First public release.
   for single-stain control files now uses an ordered candidate
   tokenizer (joined form first, then each dash-separated part
   longest-first) instead of the naïve "first dash-separated token"
-  heuristic. `PE-Cy7-A` now produces tokens `['pecy7', 'cy7']`
+  heuristic. `FL3-A` now produces tokens `['fl3', 'cy7']`
   instead of just `['pe']`. Ambiguous and unmatched channels surface
   in the status bar.
 

{openflo-2.6.0 → openflo-2.6.2}/MANIFEST.in

@@ -7,3 +7,6 @@ recursive-include templates *.json *.md
 recursive-include src/openflo/template_library *.json *.md
 include CHANGELOG.md
 include src/openflo/_golden.json
+# The public-mirror publisher lists the private tokens its scrub gate blocks,
+# so it must never ship (same reason it is export-ignore in .gitattributes).
+exclude scripts/publish_public.py

{openflo-2.6.0/src/openflo.egg-info → openflo-2.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: openflo
-Version: 2.6.0
+Version: 2.6.2
 Summary: Flow cytometry analysis pipeline + Tk GUI with FlowJo .wsp round-trip
 Author: Skyler Niedzielski
 License-Expression: MIT
@@ -69,8 +69,8 @@ FlowJo `.wsp` that round-trips back to the desktop tool. A themed (light/dark)
 editor with resizable, pop-out panels drives it all interactively.
 
 > Panel-agnostic — point it at any FCS set and pass `--channels`. The
-> examples throughout use a generic 3-colour panel (BV421=CD11b, APC=CD34,
-> PE-Cy7=CD45); substitute your own markers and controls.
+> examples throughout use a generic 3-colour panel (FL1=CD901b, FL2=CD902,
+> FL3=CD903); substitute your own markers and controls.
 
 > ### 📣 Free to use — the one ask is a citation
 > OpenFlo is free (MIT). If it contributes to research you **publish or
@@ -266,7 +266,7 @@ openflo-run \
   --trials trials/day1,trials/day2 \
   --groups '[{"name":"Day1","samples":["m1","m2"],"fmo_set":"standard"},
              {"name":"Day2","samples":["m1","m2"],"fmo_set":"standard"}]' \
-  --fmo-sets '{"standard":{"BV421-A":"bv421-fmo","APC-A":"apc-fmo","PE-Cy7-A":"pecy7-fmo"}}' \
+  --fmo-sets '{"standard":{"FL1-A":"fl1-fmo","FL2-A":"fl2-fmo","FL3-A":"fl3-fmo"}}' \
   --k 30 \
   --workers 4 \
   --seed 42 \
@@ -278,6 +278,75 @@ The result: `outputs/Day1/`, `outputs/Day2/`, each with cluster
 heatmaps + UMAP + per-sample stats; plus a single `analysis.wsp` you
 can open in FlowJo to validate the gates.
 
+#### What `openflo-run` writes
+
+With several `--trials` folders in the default `independent` mode, each trial
+gets its own sub-folder holding all of the below.
+
+- **Per sample**, in `<out>/<group>/`: `<name>_stats.csv` and the sample's
+  plots (`<name>_heatmap.png`, `_scatter.png`, `_fsc_ssc.png`).
+  `<name>_stats.csv` has one row per cluster of that sample's **own**
+  clustering: `cluster`, `population`, `count`, `pct_total` and a
+  `median_<marker>` per marker. For every threshold gate applied to the
+  sample (its FMO set, or a `--gates` threshold) it also has a
+  `pct_pos_<marker>` column: the % of the cluster's events above the cut.
+- **Per group**, in `<out>/`: `<group>_<trial>_umap.png` (`<group>_umap.png`
+  with `--batch-mode concatenate`) and the marker-pair scatters
+  (`..._<X>_<Y>_overlay.png`, `_grid.png`).
+- **Per pair of groups**, in `<out>/`: `compare_<A>_vs_<B>.csv` and `.png`,
+  each cluster's `mean_pct`, `sd_pct` and `n_samples` per group, and a
+  `cluster_ids` column saying where the ids come from (below).
+- **`compare_shared_clusters.csv`**, in `<out>/`: what each comparison cluster
+  is. One row per sample and cluster: `group`, `sample`, `cluster`,
+  `population`, `count`, `pct_total`, `median_<marker>`.
+
+**A comparison's cluster ids are not the `<name>_stats.csv` ids.**
+PhenoGraph, Leiden and FlowSOM number clusters arbitrarily, so one id names
+different populations in two separately clustered samples. The comparisons
+therefore come from ONE clustering of every sample of an analysis together
+(one per trial folder in `independent` mode). Read a comparison id's phenotype
+in `compare_shared_clusters.csv`, never in a `<name>_stats.csv`. That shared
+clustering:
+
+- clusters at most `--max-events` events **IN TOTAL**, an equal share from
+  each sample (a smaller sample gives all it has). With no cap it clusters as
+  many events as the largest sample holds, or Leiden's 200,000 / FlowSOM's
+  50,000 when that is smaller. Every other event takes the label of its
+  nearest clustered event; with FlowSOM, its best-matching SOM node's.
+- matches channels across samples by antibody label ($PnS, `--labels`,
+  `--panel`), so a marker that moved to another fluor on one day still lines
+  up. A channel with no label is matched by detector. The run names every
+  channel it leaves out or matches across different detectors.
+- is written together with every comparison, or not at all. If any sample of
+  any group is missing (it failed, its file was not found, a trial folder of
+  a `concatenate` run lacks its file, or `--skip-existing` kept it but its
+  file is gone or cannot be reloaded), no comparison file is written, the
+  run names each missing sample and why, files from an earlier run are
+  left as they were, and the run exits with status 1. The same holds when
+  the shared clustering itself fails. A single pair whose comparison fails or has no
+  rows gets no file, and its file from an earlier run is removed (the run
+  says so), so no old file sits beside the new set. The files are written
+  to a new `.compare-staging-*` folder and moved into place together once
+  all exist, so a run killed while computing them leaves the earlier set
+  untouched. A run killed while the files are moved can leave part of the
+  earlier set, part of the new one, or none; the next run replaces it, as
+  every run recomputes the comparisons, and removes the staging folder.
+  If a comparison file is in use (open in Excel), nothing is
+  moved: the earlier set stays, the run names the file and the staging
+  folder holding the new set, goes on with any other trial folder, and
+  exits with status 1.
+- is recomputed on every run. `--skip-existing` skips only the per-sample
+  work: the kept samples' events are reloaded (not re-clustered) and the
+  comparisons made again, so they always match the current groups, files,
+  settings and OpenFlo version. The comparison files of an earlier run
+  (`compare_shared_clusters.csv` and every `compare_<A>_vs_<B>.csv` /
+  `.png`) are replaced or removed; other files, such as an
+  `openflo-compare` report, are left alone.
+
+**GUI, Pipeline Workspace → Batch folder…**: each file's `<label>_clusters.csv`
+and `<label>_events.csv` come from at most the run bar's *max ev* events (a
+random subsample, seeded by the run seed), as a **Run** does.
+
 ### Compare OpenFlo's gating against FlowJo
 
 `openflo-compare` reads a FlowJo workspace, re-applies every Population
@@ -294,6 +363,35 @@ The HTML report colour-codes |Δ| > 5% rows for review. Use it to
 validate a new OpenFlo version against an established FlowJo
 analysis before switching the analysis pipeline over.
 
+### Where FlowJo and OpenFlo count differently
+
+Measured against FlowJo 10.10.2, which opened, computed and saved
+OpenFlo-exported workspaces and synthetic probe workspaces:
+
+- **Rectangle gates agree exactly.** That covers 1-D ranges, 2-D rectangles
+  and quadrant sectors.
+- **Polygon gates do not.** FlowJo decides polygon membership on a display
+  grid whose size `gateResolution` sets (default 256). It is not an exact
+  point-in-polygon test, and its exact rule is not known. On real data
+  FlowJo's polygon populations, and everything gated below them, usually come
+  out higher than OpenFlo's: by about 2% for scatter polygons, and by 3-7% for
+  exported ellipses. The display axis matters too. With no axis settings in
+  the workspace, FlowJo uses each FCS file's `P<n>DISPLAY` keyword (LIN or
+  LOG), so the same polygon can count differently on two acquisitions.
+  OpenFlo keeps the exact test.
+- **The bottom of an axis.** FlowJo treats a rectangle's lower edge at or near
+  the bottom of an axis as open, and also counts every event below the axis.
+  On linear scatter and Time axes, "near" means within about 20 units of the
+  axis minimum: a Time gate that starts at 10 s starts at 0 s in FlowJo. A
+  polygon whose bottom reaches the axis minimum is opened the same way. On
+  import (`openflo-compare`, the gate editor, templates) OpenFlo applies the
+  rule to rectangles. On export it cannot change FlowJo, so it lists each
+  affected gate as a warning.
+- **Ellipse gates** are exported as 64-vertex polygons, with a warning.
+  FlowJo silently drops a whole sample that carries a Gating-ML ellipse.
+- **Time gates** cross the `.wsp` boundary in seconds (ticks × `$TIMESTEP`),
+  the unit FlowJo uses.
+
 ---
 
 ## Synthetic data & self-test