phenotopo 0.5.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. phenotopo-0.5.0/.github/workflows/tests.yml +21 -0
  2. phenotopo-0.5.0/.gitignore +23 -0
  3. phenotopo-0.5.0/CHANGELOG.md +99 -0
  4. phenotopo-0.5.0/CITATION.cff +21 -0
  5. phenotopo-0.5.0/LICENSE +21 -0
  6. phenotopo-0.5.0/PKG-INFO +416 -0
  7. phenotopo-0.5.0/README.md +356 -0
  8. phenotopo-0.5.0/examples/figures/connectivity.png +0 -0
  9. phenotopo-0.5.0/examples/figures/connectivity_heatmap.png +0 -0
  10. phenotopo-0.5.0/examples/figures/density.png +0 -0
  11. phenotopo-0.5.0/examples/figures/explain.png +0 -0
  12. phenotopo-0.5.0/examples/figures/hyperbolic.png +0 -0
  13. phenotopo-0.5.0/examples/figures/mapper.png +0 -0
  14. phenotopo-0.5.0/examples/figures/qc.png +0 -0
  15. phenotopo-0.5.0/examples/figures/small_multiples.png +0 -0
  16. phenotopo-0.5.0/examples/quickstart.py +124 -0
  17. phenotopo-0.5.0/pyproject.toml +49 -0
  18. phenotopo-0.5.0/src/phenotopo/__init__.py +61 -0
  19. phenotopo-0.5.0/src/phenotopo/cli.py +155 -0
  20. phenotopo-0.5.0/src/phenotopo/cohort.py +409 -0
  21. phenotopo-0.5.0/src/phenotopo/data.py +142 -0
  22. phenotopo-0.5.0/src/phenotopo/explain.py +231 -0
  23. phenotopo-0.5.0/src/phenotopo/graph.py +95 -0
  24. phenotopo-0.5.0/src/phenotopo/hyperbolic.py +132 -0
  25. phenotopo-0.5.0/src/phenotopo/layout.py +246 -0
  26. phenotopo-0.5.0/src/phenotopo/mapper.py +72 -0
  27. phenotopo-0.5.0/src/phenotopo/outliers.py +133 -0
  28. phenotopo-0.5.0/src/phenotopo/qc.py +254 -0
  29. phenotopo-0.5.0/src/phenotopo/report.py +198 -0
  30. phenotopo-0.5.0/src/phenotopo/robustness.py +167 -0
  31. phenotopo-0.5.0/src/phenotopo/stats.py +143 -0
  32. phenotopo-0.5.0/tests/test_cli.py +69 -0
  33. phenotopo-0.5.0/tests/test_cohort.py +101 -0
  34. phenotopo-0.5.0/tests/test_explain.py +98 -0
  35. phenotopo-0.5.0/tests/test_graph.py +49 -0
  36. phenotopo-0.5.0/tests/test_hyperbolic.py +41 -0
  37. phenotopo-0.5.0/tests/test_layout.py +15 -0
  38. phenotopo-0.5.0/tests/test_mapper.py +18 -0
  39. phenotopo-0.5.0/tests/test_outliers.py +56 -0
  40. phenotopo-0.5.0/tests/test_qc.py +58 -0
  41. phenotopo-0.5.0/tests/test_report.py +29 -0
  42. phenotopo-0.5.0/tests/test_robustness.py +38 -0
  43. phenotopo-0.5.0/tests/test_stats.py +52 -0
@@ -0,0 +1,21 @@
1
+ name: tests
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ pull_request:
7
+
8
+ jobs:
9
+ test:
10
+ runs-on: ubuntu-latest
11
+ strategy:
12
+ matrix:
13
+ python-version: ["3.10", "3.12"]
14
+ steps:
15
+ - uses: actions/checkout@v4
16
+ - uses: actions/setup-python@v5
17
+ with:
18
+ python-version: ${{ matrix.python-version }}
19
+ - run: python -m pip install --upgrade pip
20
+ - run: pip install -e ".[dev]"
21
+ - run: pytest -q
@@ -0,0 +1,23 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ .pytest_cache/
4
+ .ipynb_checkpoints/
5
+ build/
6
+ dist/
7
+ *.egg-info/
8
+ .venv/
9
+ venv/
10
+ .DS_Store
11
+ .vscode/
12
+ .idea/
13
+
14
+ # never commit cohort data, ontology dumps or analysis output
15
+ *.obo
16
+ *.hpoa
17
+ *.xlsx
18
+ *.csv
19
+ *.png
20
+ *.pdf
21
+ *.svg
22
+ # ... except the README figures produced by the synthetic example
23
+ !examples/figures/*.png
@@ -0,0 +1,99 @@
1
+ # Changelog
2
+
3
+ Format follows [Keep a Changelog](https://keepachangelog.com/); versioning is [SemVer](https://semver.org/).
4
+
5
+ ## [0.5.0] — 2026-09-05
6
+
7
+ ### Changed
8
+ - **`Cohort.distance()` now defaults to SimGIC**, the IC-weighted ontology-aware
9
+ measure, with cosine kept as a sensitivity analysis; `Cohort.distances()` returns
10
+ both, ready for the robustness protocol. (Behaviour change: an explicit
11
+ `distance("cosine")` reproduces the old default.)
12
+ - QC flags are relative and descriptive — `LOW_ANNOTATION_DEPTH`,
13
+ `LOW_SPECIFICITY_RELATIVE_TO_COHORT`, `HIGH_REDUNDANCY`, resolving to
14
+ `review recommended` — replacing the absolute `under-phenotyped` / `non-specific`.
15
+ A short, well-chosen phenotype list is not a poor one. Summary keys renamed
16
+ accordingly (`pct_low_annotation_depth`, `pct_low_specificity`,
17
+ `pct_review_recommended`).
18
+ - `annotation_bias` reports `confound_risk` (LOW / MODERATE / HIGH) with an explicit
19
+ note that annotation counts predicting a group does **not** establish that the
20
+ separation is caused by them, replacing the causal verdict string.
21
+
22
+ ### Added
23
+ - `distance(..., negatives="use")`: excluded phenotypes (propagated downwards, since
24
+ excluding a parent excludes its children) enter the similarity, so a shared
25
+ ruled-out phenotype counts. Refuses to run where absence was never recorded;
26
+ `onset=` is declared and raises rather than silently ignoring.
27
+ - `Ontology.descendants`, `Cohort.propagated_excluded`, `Cohort.has_negatives`,
28
+ `Cohort.excluded_terms`, `Cohort.distances`.
29
+ - `explain_groups` reports `excluded_a` / `excluded_b` — the fraction in whom a
30
+ phenotype was looked for and ruled out — alongside the test, never inside it.
31
+ - Command line (`phenotopo`): `demo` (synthetic cohort → report in a browser),
32
+ `report` (Phenopackets or a CSV/XLSX table → the same HTML, no Python needed),
33
+ `ontology install` / `path` (explicit local ontology cache; the only place the
34
+ package touches the network).
35
+
36
+ ## [0.4.0] — 2026-09-05
37
+
38
+ ### Added
39
+ - `cohort.Cohort` / `cohort.Ontology` with readers `from_hpo_table` and
40
+ `from_phenopackets` (GA4GH Phenopackets v2: excluded phenotypes, onset, disease and
41
+ gene), ancestor propagation, information content, and cosine / SimGIC distances —
42
+ so an analysis no longer starts with the user building a distance matrix.
43
+ - `qc.phenotype_qc`: per-patient annotation quality (terms, specificity, ontology
44
+ depth, redundant ancestors, recorded absence and onset) with flags, plus
45
+ `qc.plot_qc`.
46
+ - `qc.annotation_bias`: Kruskal–Wallis on annotation depth across groups and a
47
+ cross-validated k-NN comparison of group recovery from phenotype versus from
48
+ annotation counts alone — how much of the apparent structure is bookkeeping.
49
+ - `outliers.patient_outliers` (isolation and neighbourhood discordance reported
50
+ separately) and `outliers.explain_outlier` (terms that make a patient unusual, and
51
+ the terms its neighbourhood has that it lacks).
52
+ - `explain.explain_groups`: ontology-aware group comparison with a Westfall–Young
53
+ max-T permutation (FWER control that respects parent–child dependence), Newcombe
54
+ confidence intervals, an effect-size threshold and pruning of redundant ancestor
55
+ terms; `explain.plot_explain`.
56
+ - `report.cohort_report`: one self-contained local HTML file with QC, annotation
57
+ bias, outliers, robustness verdicts and group comparisons, figures embedded.
58
+ - `data.synthetic_hpo_cohort`: an annotated cohort with designed faults (a thinly
59
+ phenotyped site, discordant cases, redundant ancestor terms).
60
+
61
+ ### Changed
62
+ - `phenotype_qc` returns its patient table in cohort order, so it lines up with
63
+ labels and distance matrices; the flagged subset is the sorted view.
64
+
65
+ ## [0.3.1] — 2026-09-03
66
+
67
+ ### Fixed
68
+ - Robustness tests assumed group sizes that the synthetic cohort does not have; simplified
69
+ pair-label formatting in `plot_forest`.
70
+
71
+ ## [0.3.0] — 2026-09-03
72
+
73
+ ### Added
74
+ - `robustness.connectivity_robustness`: the multi-configuration protocol (distances × k,
75
+ minimum group size, effect-size thresholds, all-configurations verdicts) and
76
+ `robustness.plot_forest`.
77
+ - Continuous integration (pytest on push / pull request), `CITATION.cff`.
78
+
79
+ ## [0.2.0] — 2026-09-03
80
+
81
+ ### Added
82
+ - `stats.permutation_test`: label-permutation null for every connectivity ratio
83
+ (p-values, BH q-values, z-scores, 95 % null interval), `stats.bootstrap_ratio`
84
+ (subsampling percentile CI), significance overlay on the connectivity heatmap.
85
+ - `plot_connectivity_heatmap`; optional adjustText label repulsion.
86
+
87
+ ## [0.1.0] — 2026-09-03
88
+
89
+ ### Added
90
+ - `knn_graph`, `group_connectivity`: PAGA-style connectivity between labelled groups
91
+ (observed k-NN edges relative to a configuration-model expectation).
92
+ - `plot_connectivity`: abstracted group graph drawn at group centroids.
93
+ - `plot_density`, `plot_small_multiples`: overlap-honest alternatives to a single
94
+ many-colour scatter.
95
+ - `hyperbolic`: Poincaré embedding of a term hierarchy, Einstein-midpoint placement
96
+ of patients in the disk, radial specificity axis.
97
+ - `mapper`: KeplerMapper wrapper producing a `networkx` graph coloured by any
98
+ per-patient value.
99
+ - `data.synthetic_cohort` / `data.synthetic_hierarchy` for examples and tests.
@@ -0,0 +1,21 @@
1
+ cff-version: 1.2.0
2
+ message: "If you use phenotopo, please cite it as below."
3
+ title: "phenotopo: a QC and explainability toolkit for phenotype cohorts"
4
+ type: software
5
+ version: 0.5.0
6
+ date-released: "2026-09-05"
7
+ license: MIT
8
+ repository-code: "https://github.com/MargoSolo/phenotopo"
9
+ authors:
10
+ - family-names: Soloshenko
11
+ given-names: Margarita
12
+ email: margosoloshenko@mail.ru
13
+ keywords:
14
+ - HPO
15
+ - Phenopacket
16
+ - phenotype
17
+ - rare disease
18
+ - quality control
19
+ - outlier detection
20
+ - visualization
21
+ - connectivity
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Margarita Soloshenko
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,416 @@
1
+ Metadata-Version: 2.5
2
+ Name: phenotopo
3
+ Version: 0.5.0
4
+ Summary: Phenotype QC, outlier detection and robust cohort structure for HPO / Phenopacket rare-disease cohorts.
5
+ Project-URL: Homepage, https://github.com/MargoSolo/phenotopo
6
+ Project-URL: Issues, https://github.com/MargoSolo/phenotopo/issues
7
+ Author-email: Margarita Soloshenko <margosoloshenko@mail.ru>
8
+ License: MIT License
9
+
10
+ Copyright (c) 2026 Margarita Soloshenko
11
+
12
+ Permission is hereby granted, free of charge, to any person obtaining a copy
13
+ of this software and associated documentation files (the "Software"), to deal
14
+ in the Software without restriction, including without limitation the rights
15
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
16
+ copies of the Software, and to permit persons to whom the Software is
17
+ furnished to do so, subject to the following conditions:
18
+
19
+ The above copyright notice and this permission notice shall be included in all
20
+ copies or substantial portions of the Software.
21
+
22
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
23
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
24
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
25
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
26
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
27
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
28
+ SOFTWARE.
29
+ License-File: LICENSE
30
+ Keywords: HPO,PAGA,Phenopacket,genotype-phenotype,outlier detection,phenotype,quality control,rare disease,visualization
31
+ Classifier: Development Status :: 4 - Beta
32
+ Classifier: Intended Audience :: Science/Research
33
+ Classifier: License :: OSI Approved :: MIT License
34
+ Classifier: Operating System :: OS Independent
35
+ Classifier: Programming Language :: Python :: 3
36
+ Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
37
+ Classifier: Topic :: Scientific/Engineering :: Visualization
38
+ Requires-Python: >=3.10
39
+ Requires-Dist: matplotlib>=3.7
40
+ Requires-Dist: networkx>=3.0
41
+ Requires-Dist: numpy>=1.24
42
+ Requires-Dist: pandas>=2.0
43
+ Requires-Dist: scikit-learn>=1.3
44
+ Requires-Dist: scipy>=1.10
45
+ Provides-Extra: all
46
+ Requires-Dist: adjusttext>=1.0; extra == 'all'
47
+ Requires-Dist: gensim>=4.3; extra == 'all'
48
+ Requires-Dist: kmapper>=2.0; extra == 'all'
49
+ Requires-Dist: umap-learn>=0.5; extra == 'all'
50
+ Provides-Extra: dev
51
+ Requires-Dist: adjusttext>=1.0; extra == 'dev'
52
+ Requires-Dist: gensim>=4.3; extra == 'dev'
53
+ Requires-Dist: kmapper>=2.0; extra == 'dev'
54
+ Requires-Dist: pytest>=7.0; extra == 'dev'
55
+ Provides-Extra: hyperbolic
56
+ Requires-Dist: gensim>=4.3; extra == 'hyperbolic'
57
+ Provides-Extra: tda
58
+ Requires-Dist: kmapper>=2.0; extra == 'tda'
59
+ Description-Content-Type: text/markdown
60
+
61
+ # phenotopo
62
+
63
+ **A QC and explainability toolkit for phenotype cohorts.**
64
+
65
+ [![tests](https://github.com/MargoSolo/phenotopo/actions/workflows/tests.yml/badge.svg)](https://github.com/MargoSolo/phenotopo/actions/workflows/tests.yml)
66
+ [![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
67
+ [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
68
+ [![PyPI](https://img.shields.io/pypi/v/phenotopo.svg)](https://pypi.org/project/phenotopo/)
69
+ [![version](https://img.shields.io/badge/version-0.5.0-informational.svg)](CHANGELOG.md)
70
+
71
+ Two things go wrong when a rare-disease cohort is analysed by HPO terms.
72
+
73
+ The first is **bookkeeping mistaken for biology**: some patients were phenotyped in
74
+ two vague terms and others in twenty specific ones, the difference tracks a site or
75
+ a clinician, and every map then shows structure that is really annotation depth.
76
+ The second is **clusters that are not there**: a cohort described by phenotype is
77
+ usually a continuum with local structure, and one UMAP with twenty colours invites
78
+ everyone to read islands into it.
79
+
80
+ `phenotopo` is built around those two failures. It scores how well each patient was
81
+ phenotyped and flags when the group signal might be bookkeeping; it finds the
82
+ patients who sit away from their own group; it says which phenotypes actually
83
+ separate two groups, tested in a way that respects the ontology; and it reports
84
+ cohort structure only where the answer survives a change of distance, of *k* and of
85
+ resampling. Treating a cohort as a continuum rather than a set of clusters is the
86
+ methodology underneath; QC and explainability are what it is for.
87
+
88
+ Everything runs locally. No patient data leaves the machine, and the library makes
89
+ no network calls — the single exception is the explicit `phenotopo ontology install`
90
+ command, which prints the URL it fetches.
91
+
92
+ ```bash
93
+ pip install phenotopo
94
+ phenotopo demo # a report from a synthetic cohort, in a browser, in 30 seconds
95
+ ```
96
+
97
+ ---
98
+
99
+ ## Installation
100
+
101
+ ```bash
102
+ pip install phenotopo # core: cohorts, QC, outliers, comparison, connectivity
103
+ pip install "phenotopo[hyperbolic]" # + Poincaré disk (gensim)
104
+ pip install "phenotopo[tda]" # + Mapper graph (kmapper)
105
+ pip install "phenotopo[all]" # everything, incl. umap-learn
106
+ ```
107
+
108
+ For development, clone the repository and `pip install -e ".[dev]"`.
109
+
110
+ If `adjustText` is installed, overlapping labels on the connectivity graph and the
111
+ Poincaré disk are pushed apart automatically; without it the plots still work.
112
+
113
+ ## Without writing Python
114
+
115
+ ```bash
116
+ phenotopo demo # see what a report looks like
117
+ phenotopo ontology install hp # cache the HPO release once
118
+ phenotopo report patients/ --labels diagnosis --compare GENE_A GENE_B -o report.html
119
+ ```
120
+
121
+ `report` takes a directory of Phenopackets, a single `.json` packet, or a CSV/XLSX
122
+ table of HPO terms, and writes the same self-contained HTML file the Python API
123
+ produces. This is not a substitute for a real interface — it is what makes the tool
124
+ usable by someone who will not open a notebook.
125
+
126
+ ## Quick start
127
+
128
+ ```python
129
+ import phenotopo as pt
130
+
131
+ cohort = pt.from_phenopackets("patients/", ontology="hp.obo") # or from_hpo_table(df, ...)
132
+ D = cohort.distance() # SimGIC, the ontology-aware default
133
+ qc = pt.phenotype_qc(cohort) # how was each patient annotated?
134
+ bias = pt.annotation_bias(qc, cohort.labels("site"), # could the group signal be bookkeeping?
135
+ distance=D)
136
+ out = pt.patient_outliers(D, cohort.labels("diagnosis"), # who sits away from their group?
137
+ ids=cohort.ids)
138
+ diff = pt.explain_groups(cohort, cohort.labels("gene"), # which phenotypes separate them?
139
+ "GENE_A", "GENE_B")
140
+ pt.cohort_report(cohort, labels="diagnosis", distance=D, # one local HTML file
141
+ comparisons=diff, path="cohort_report.html")
142
+ ```
143
+
144
+ A `Cohort` is optional throughout: every analysis function also takes a plain
145
+ distance matrix and an array of labels, so `phenotopo` sits on top of a pipeline
146
+ you already have rather than replacing it.
147
+
148
+ ### Similarity, and which fields are actually used
149
+
150
+ `cohort.distance()` is **SimGIC** — the IC-weighted Jaccard of the propagated term
151
+ sets (Pesquita et al. 2008), the ontology-aware measure the HPO literature uses.
152
+ `distance("cosine")` (cosine between IC-weighted propagated vectors) is kept as a
153
+ sensitivity analysis, and `cohort.distances()` returns both, ready to hand to the
154
+ robustness protocol. Both are hierarchy-aware: two patients sharing no term but
155
+ sharing an ancestor are still similar.
156
+
157
+ `from_phenopackets` reads GA4GH Phenopackets v2 and keeps what a bare list of term
158
+ IDs throws away. What each field currently affects — stated plainly, because storing
159
+ a field is not the same as using it:
160
+
161
+ | Field | Read | Used by |
162
+ |---|---|---|
163
+ | present phenotypes | yes | everything |
164
+ | **excluded** phenotypes (looked for and absent) | yes | `phenotype_qc`; `distance(..., negatives="use")`, where a shared ruled-out phenotype makes two patients more similar; reported as a separate column by `explain_groups` and never mixed into the test |
165
+ | **onset** per observation | yes | `phenotype_qc` completeness only — `distance(..., onset=...)` is declared and raises, rather than pretending |
166
+ | disease, gene, sex | yes | metadata, so `labels("gene")` and `explain_groups` work straight after reading |
167
+
168
+ `not recorded` and `looked for and absent` are different observations, and only the
169
+ second is evidence of absence — so negatives are opt-in and refuse to run on a cohort
170
+ where absence was never recorded.
171
+
172
+ ---
173
+
174
+ The figures below come from [`examples/quickstart.py`](examples/quickstart.py), run on
175
+ synthetic cohorts with *designed* faults and structure, so each can be checked against
176
+ ground truth.
177
+
178
+ ## 1 · Phenotyping quality
179
+
180
+ ```python
181
+ qc = pt.phenotype_qc(cohort)
182
+ qc["summary"] # median terms, % low depth, % low specificity, % redundant, ...
183
+ qc["flagged"] # worst-annotated patients first, with reasons
184
+ pt.plot_qc(qc, cohort.labels("site"))
185
+ ```
186
+
187
+ <p align="center"><img src="examples/figures/qc.png" width="760"></p>
188
+
189
+ Per patient: terms asserted and after propagation, explicitly excluded phenotypes,
190
+ observations with an onset, mean and total information content, mean ontology depth,
191
+ **redundant ancestor terms** (a term recorded next to its own child — bookkeeping, not
192
+ information), the specificity percentile, and flags.
193
+
194
+ The flags are deliberately **relative to this cohort and descriptive**:
195
+ `LOW_ANNOTATION_DEPTH`, `LOW_SPECIFICITY_RELATIVE_TO_COHORT`, `HIGH_REDUNDANCY`,
196
+ resolving to `review recommended`. Six well-chosen terms can describe a skeletal
197
+ dysplasia completely while fifteen vague ones describe a neurodevelopmental case
198
+ badly, so the tool never declares a patient "under-phenotyped" in absolute terms.
199
+
200
+ Then the question that decides whether any of the rest can be believed:
201
+
202
+ ```python
203
+ bias = pt.annotation_bias(qc, labels, distance=D)
204
+ bias["kruskal"] # do the groups differ in annotation depth? H, p, epsilon-squared
205
+ bias["recovery"] # {'phenotype': 0.92, 'annotation_only': 1.00, 'confound_risk': 'HIGH', ...}
206
+ ```
207
+
208
+ Group labels are predicted twice by cross-validated k-NN — from the phenotype
209
+ distance, and from **how much was written down alone** (`n_terms`, `n_propagated`).
210
+ Specificity measures are deliberately excluded from that second model: they depend on
211
+ *which* terms a patient has, so including them would report real biology as bias.
212
+
213
+ The result is an **annotation-confound risk** (LOW / MODERATE / HIGH), not a verdict.
214
+ That annotation counts predict the group does *not* establish that the separation is
215
+ caused by them: a group that genuinely differs in phenotype severity is usually also
216
+ annotated more thoroughly, and the chain runs group → severity → annotation depth.
217
+ HIGH means the analysis must address the confound — by matching, stratification or a
218
+ depth-controlled comparison — not that the finding is an artefact.
219
+
220
+ ## 2 · Outliers — who sits away from their own group
221
+
222
+ ```python
223
+ out = pt.patient_outliers(D, labels, ids=cohort.ids, k=15)
224
+ pt.explain_outlier(cohort, index, D, labels)
225
+ ```
226
+
227
+ Two things get called an outlier, and mixing them produces nonsense, so they are
228
+ reported separately: **isolation** (far from everybody — a sparse or unusual
229
+ phenotype) and **discordance** (plenty of close neighbours, and they belong to a
230
+ different group). `explain_outlier` names the terms that make a patient unusual for
231
+ its neighbourhood, and — often more useful — the terms its neighbourhood has that it
232
+ lacks, which is as likely to be a phenotyping gap as biology.
233
+
234
+ The output is never a claim that a diagnosis is wrong. It is *phenotypically
235
+ discordant with the assigned group*: a candidate for review in a diagnostic cohort,
236
+ a genotype–phenotype study or a reclassification project.
237
+
238
+ ## 3 · What separates two groups
239
+
240
+ ```python
241
+ diff = pt.explain_groups(cohort, labels, "GENE_A", "GENE_B", min_effect=0.15)
242
+ diff["top"] # term, name, prevalence in each group, effect in pp with CI, adjusted p
243
+ pt.plot_explain(diff)
244
+ ```
245
+
246
+ <p align="center"><img src="examples/figures/explain.png" width="700"></p>
247
+
248
+ One test per HPO term, sorted by p-value, answers this badly, for two reasons
249
+ specific to ontologies:
250
+
251
+ - **The terms are not independent.** After propagation a patient annotated *Status
252
+ epilepticus* is also annotated *Seizure* and everything up to the root, so one
253
+ finding lights up a whole ancestor chain and Benjamini–Hochberg assumes far more
254
+ independence than exists. `explain_groups` therefore uses a **Westfall–Young max-T
255
+ permutation**: labels are shuffled and the *largest* prevalence difference over all
256
+ terms is recorded each time, so the null already contains the ontology's
257
+ correlation structure and the adjusted p-values control the family-wise error rate.
258
+ - **Significance is not the answer.** With a few thousand patients a 2-point
259
+ difference is significant and clinically empty. Terms are reported only above an
260
+ explicit **effect-size threshold** in percentage points, with a Newcombe confidence
261
+ interval, and redundant ancestors of an already-reported term are pruned — ties go
262
+ to the deeper term, so *Seizure* is dropped for *Status epilepticus* and never the
263
+ other way round.
264
+
265
+ ## 4 · Cohort structure, only where it is robust
266
+
267
+ ```python
268
+ res = pt.connectivity_robustness({"cosine": D_cos, "simgic": D_gic}, labels,
269
+ ks=(10, 15, 30), min_size=100)
270
+ res["summary"] # per pair: ratio, CI, range across configurations, verdict
271
+ pt.plot_forest(res)
272
+ ```
273
+
274
+ The connectivity ratio compares the k-NN edges crossing between two groups with the
275
+ number expected under a degree-preserving null (the PAGA abstraction, Wolf et al.
276
+ 2019): `1` is random mixing, `> 1` blending, `< 1` separation, and the diagonal is
277
+ within-group cohesion. A ratio from one distance, one *k* and one sample is a single
278
+ draw, so the protocol (i) drops groups below `min_size`, whose expected counts are
279
+ tiny and whose ratios explode; (ii) recomputes every ratio for every distance × *k*
280
+ with a permutation *q* and a bootstrap CI; (iii) applies an **effect-size threshold**,
281
+ not just significance; and (iv) issues a verdict only if the criterion holds in
282
+ **every** configuration. What survives is a sentence you can write down.
283
+
284
+ The underlying views, when you want to look rather than test:
285
+
286
+ ```python
287
+ conn = pt.group_connectivity(pt.knn_graph(distance=D, k=15), labels)
288
+ pt.plot_connectivity(conn, embedding=emb, labels=labels) # node-link, few groups
289
+ pt.plot_connectivity_heatmap(conn, min_size=30, # every pair, many groups
290
+ significance=pt.permutation_test(pt.knn_graph(distance=D), labels))
291
+ ```
292
+
293
+ <p align="center">
294
+ <img src="examples/figures/connectivity.png" width="420">
295
+ <img src="examples/figures/connectivity_heatmap.png" width="380">
296
+ </p>
297
+
298
+ Nodes sit at each group's median position, node area is group size, edge width ∝ log
299
+ ratio. Past about eight groups a node–link drawing stops being readable and the
300
+ heatmap is the honest form: every pair, the separated ones (`ratio < 1`) included,
301
+ `*` where the permutation q survives correction.
302
+
303
+ ## 5 · Looking at the continuum
304
+
305
+ ```python
306
+ pt.plot_density(emb, labels, levels=(0.5, 0.85)) # where groups overlap, drawn not hidden
307
+ pt.plot_small_multiples(emb, labels, ncols=5) # one panel per group, no 20-colour legend
308
+ ```
309
+
310
+ <p align="center">
311
+ <img src="examples/figures/density.png" width="420">
312
+ <img src="examples/figures/small_multiples.png" width="420">
313
+ </p>
314
+
315
+ Two further views, for hierarchy and shape:
316
+
317
+ ```python
318
+ from phenotopo.hyperbolic import poincare_terms, place_patients, plot_disk
319
+ from phenotopo.mapper import mapper_graph, node_values, plot_mapper
320
+ ```
321
+
322
+ <p align="center">
323
+ <img src="examples/figures/hyperbolic.png" width="380">
324
+ <img src="examples/figures/mapper.png" width="440">
325
+ </p>
326
+
327
+ The **Poincaré disk** fits a hierarchy into two dimensions with the root at the centre
328
+ and specific leaves at the rim; patients sit at the Einstein midpoint of their terms,
329
+ so phenotyping specificity becomes a radial axis — the annotation-depth confound made
330
+ geometric instead of hidden. The **Mapper graph** covers the space with overlapping
331
+ bins and clusters locally inside each, showing branches, bridges and flares without
332
+ forcing every patient into a cluster; colour it by any per-patient outcome (here a
333
+ synthetic diagnostic yield) to see where that outcome concentrates.
334
+
335
+ ## 6 · One file to hand over
336
+
337
+ ```python
338
+ pt.cohort_report(cohort, labels="diagnosis", distance=D,
339
+ robustness=res, comparisons=[diff], path="cohort_report.html")
340
+ ```
341
+
342
+ Cohort overview, QC and the annotation-bias verdict, the outlier list, the robustness
343
+ table with its forest plot and every group comparison — in a single self-contained
344
+ HTML file, figures embedded, no server and no network. It opens in a browser and
345
+ survives being emailed.
346
+
347
+ ---
348
+
349
+ ## Reading the connectivity ratio
350
+
351
+ | `ratio` | Meaning |
352
+ |---|---|
353
+ | ≫ 1 | groups blend — many more cross edges than chance |
354
+ | ≈ 1 | as connected as random mixing |
355
+ | ≪ 1 | separated — a real boundary in phenotype space |
356
+
357
+ The null preserves every group's total degree (the configuration model behind
358
+ modularity), so large groups are not rewarded merely for being large.
359
+
360
+ ## API
361
+
362
+ | Module | Contents |
363
+ |---|---|
364
+ | `phenotopo.cohort` | `Cohort`, `Ontology`, `from_hpo_table`, `from_phenopackets` |
365
+ | `phenotopo.qc` | `phenotype_qc`, `annotation_bias`, `plot_qc` |
366
+ | `phenotopo.outliers` | `patient_outliers`, `explain_outlier` |
367
+ | `phenotopo.explain` | `explain_groups`, `plot_explain` |
368
+ | `phenotopo.report` | `cohort_report` |
369
+ | `phenotopo.graph` | `knn_graph`, `group_connectivity`, `group_centroids` |
370
+ | `phenotopo.stats` | `permutation_test`, `bootstrap_ratio`, `benjamini_hochberg` |
371
+ | `phenotopo.robustness` | `connectivity_robustness`, `plot_forest` |
372
+ | `phenotopo.layout` | `plot_connectivity`, `plot_connectivity_heatmap`, `plot_density`, `plot_small_multiples`, `default_palette` |
373
+ | `phenotopo.hyperbolic` | `poincare_terms`, `einstein_midpoint`, `place_patients`, `radial_specificity`, `plot_disk` |
374
+ | `phenotopo.mapper` | `mapper_graph`, `node_values`, `plot_mapper` |
375
+ | `phenotopo.data` | `synthetic_cohort`, `synthetic_hpo_cohort`, `synthetic_hierarchy`, `synthetic_term_lists` |
376
+ | `phenotopo.cli` | `phenotopo demo`, `phenotopo report`, `phenotopo ontology install/path` |
377
+
378
+ ## Scope
379
+
380
+ `phenotopo` deliberately does **not** prioritise genes or diagnoses for a single
381
+ patient — Exomiser and LIRICAL do that — and does not capture phenotypes, pedigrees
382
+ or variants, which is PhenoTips' job. It sits after capture and beside
383
+ prioritisation: quality control, cohort structure, outliers and group comparison,
384
+ reading Phenopackets so it fits between the two.
385
+
386
+ ## Citation
387
+
388
+ See [`CITATION.cff`](CITATION.cff).
389
+
390
+ ## Tests
391
+
392
+ ```bash
393
+ pytest -q
394
+ ```
395
+
396
+ Runs on synthetic data only, by design: **no patient data belongs in this
397
+ repository**, and `.gitignore` blocks ontology dumps, cohort files and analysis
398
+ output. The figures in this README come from synthetic cohorts with designed
399
+ structure, which makes them checkable but not evidence about real cohorts; a
400
+ case study on public, published cases (Phenopacket Store) is the next release.
401
+
402
+ ## References
403
+
404
+ - Wolf F.A. et al. *PAGA: graph abstraction reconciles clustering with trajectory inference through a topology preserving map of single cells.* Genome Biology, 2019.
405
+ - Westfall P.H., Young S.S. *Resampling-based multiple testing.* Wiley, 1993 — max-T procedure.
406
+ - Newcombe R.G. *Interval estimation for the difference between independent proportions.* Statistics in Medicine, 1998.
407
+ - Pesquita C. et al. *Metrics for GO based protein semantic similarity: a systematic evaluation.* BMC Bioinformatics, 2008 — SimGIC.
408
+ - Jacobsen J.O.B. et al. *The GA4GH Phenopacket schema defines a computable representation of clinical data.* Nature Biotechnology, 2022.
409
+ - Nickel M., Kiela D. *Poincaré embeddings for learning hierarchical representations.* NeurIPS, 2017.
410
+ - Ungar A.A. *Analytic hyperbolic geometry.* World Scientific, 2005 — Einstein midpoint.
411
+ - Singh G., Mémoli F., Carlsson G. *Topological methods for the analysis of high dimensional data sets.* Eurographics, 2007 — Mapper.
412
+ - van Veen H.J. et al. *Kepler Mapper.* Journal of Open Source Software, 2019.
413
+
414
+ ## License
415
+
416
+ MIT — see [LICENSE](LICENSE).