@pikaa-ai/pikaa 0.2.4 → 0.3.0

package/skills/waypoint-bio/SKILL.md

@@ -0,0 +1,273 @@
+---
+name: waypoint-bio
+description: Use when working with Outpost Bio's open microbiome foundation models - the Waypoint checkpoints (Waypoint-6m, Waypoint-45m, Waypoint-170m), the Atlas pretraining corpus, the Compass eight-task benchmark, or the `waypoint` CLI from the `waypoint-bio` package. Covers embedding microbiome samples, fine-tuning on taxonomic abundance data, benchmarking a checkpoint on Compass, pretraining a GPT-2 model on taxonomic abundance profiles, and converting MetaPhlAn, Kraken2, QIIME 2, or MGnify abundance tables into waypoint format.
+license: MIT
+compatibility: Requires Python 3.10+ with `waypoint-bio` (pulls torch, transformers, datasets, peft, scikit-learn). Needs network access and a Hugging Face token with access granted to the gated outpost-bio repos. A GPU is strongly recommended for pretraining and benchmarking.
+metadata:
+  version: "1.0"
+  skill-author: K-Dense Inc.
+  upstream-version: "waypoint-bio 1.0.2 (PyPI); GitHub main 1.0.4"
+  last-reviewed: "2026-08-17"
+  openclaw:
+    primaryEnv: HF_TOKEN
+    envVars:
+      - name: HF_TOKEN
+        required: true
+        description: Hugging Face read token with access to the gated outpost-bio/Waypoint-*, outpost-bio/Atlas, and outpost-bio/Compass repos.
+---
+
+# Waypoint: Outpost Bio's Open Microbiome Foundation Models
+
+## Overview
+
+Outpost Bio open-sourced three artefacts under Apache 2.0, described in
+[Treloar et al., bioRxiv 2026.05.02.722381](https://www.biorxiv.org/content/10.64898/2026.05.02.722381v2):
+
+| Artefact | What it is | Hugging Face |
+| --- | --- | --- |
+| **Waypoint** | GPT-2-style causal LMs over taxonomic tokens, 6M–170M params | `outpost-bio/Waypoint-6m`, `-45m`, `-170m` |
+| **Atlas** | 539,308 microbiome samples scraped from MGnify (485,377 pretrain / 53,931 benchmark) | `outpost-bio/Atlas` |
+| **Compass** | Eight downstream tasks over four studies | `outpost-bio/Compass` |
+
+The unifying idea: a microbiome sample is a *sentence*. Each taxon is one token, tokens are ordered
+by descending abundance z-score, and the model is trained with next-token prediction. A pretrained
+checkpoint then supplies sample-level embeddings or a fine-tuning backbone for prediction tasks.
+
+All of it is driven by one CLI, `waypoint`, with five subcommands: `prepare-dataset`, `embed`,
+`finetune`, `benchmark`, `pretrain`.
+
+## When to use
+
+- Embedding 16S/shotgun taxonomic profiles into fixed-size vectors for clustering, visualisation, or
+  a downstream classifier.
+- Fine-tuning a Waypoint checkpoint to predict a phenotype, treatment, or continuous readout from
+  community composition.
+- Scoring your own microbiome model against Compass so the number is comparable to the paper.
+- Pretraining a taxonomic language model on Atlas or on your own corpus.
+- Converting profiler output (MetaPhlAn, Kraken2/Bracken, QIIME 2, MGnify TSVs) into the input format
+  these tools expect.
+
+**Do not reach for this** when you have fewer than ~1,000 labelled samples — see
+[Scientific caveats](#scientific-caveats). A random forest on relative abundances is the better tool
+there, and the paper says so.
+
+## Setup
+
+```bash
+pip install waypoint-bio       # installs the `waypoint` command
+```
+
+Atlas, Compass, and every Waypoint checkpoint are **gated**. Access is auto-approved, but you must
+click through once per repo and then authenticate:
+
+1. Request access on each repo page you need: [Waypoint-6m](https://huggingface.co/outpost-bio/Waypoint-6m),
+   [Waypoint-45m](https://huggingface.co/outpost-bio/Waypoint-45m),
+   [Waypoint-170m](https://huggingface.co/outpost-bio/Waypoint-170m),
+   [Atlas](https://huggingface.co/datasets/outpost-bio/Atlas),
+   [Compass](https://huggingface.co/datasets/outpost-bio/Compass).
+2. Authenticate locally:
+
+   ```bash
+   hf auth login          # or: export HF_TOKEN=hf_...
+   ```
+
+A 401/403 from any subcommand almost always means access was never requested on that specific repo —
+a token alone is not enough. Use a read-scoped token. The tokenizer loads via
+`trust_remote_code=True`, so pin a `revision` if you need the remote code fixed across runs.
+
+## The waypoint data format
+
+Everything except `prepare-dataset` consumes **waypoint format**: a `.parquet` / `.csv` / `.tsv`
+whose rows are samples, with two aligned list-columns plus any label columns you need.
+
+| Column | Type | Notes |
+| --- | --- | --- |
+| `Taxa` | `list[str]` | Full lineage strings, `;`-separated: `k__Bacteria; p__Firmicutes; ...; g__Lactobacillus` |
+| `Relative Abundances` | `list[float]` | Same length as `Taxa`, same order |
+| *(any)* | scalar | Targets, covariates, or a `Split` column |
+
+Prefer parquet. CSV/TSV stores the lists as `repr` strings and round-trips through `ast.literal_eval`.
+
+**Give full lineages, not bare names.** The tokenizer extracts the genus segment (`g__`) from each
+lineage and falls back to the most specific higher rank when genus is missing. Bare names disable
+that fallback entirely.
+
+## Workflow
+
+### 1. Get your data into waypoint format
+
+If you already have a sample × taxa (or taxa × sample) abundance matrix with lineage labels:
+
+```bash
+waypoint prepare-dataset \
+    --input abundance_matrix.tsv \
+    --metadata sample_labels.csv \
+    --output dataset.parquet
+```
+
+Orientation is auto-detected from the first column header (`taxonomy`, `lineage`, `taxon`, `otu`,
+`#otu id` ⇒ taxa-as-rows); override with `--orientation`. Rows are normalised to sum to 1 unless you
+pass `--no_normalize`, and zeros are dropped unless you pass `--keep_zeros`.
+
+`prepare-dataset` cannot read profiler output directly — MetaPhlAn uses `|` separators, Kraken2
+reports encode the hierarchy as indentation, and QIIME 2/SILVA prefixes the domain `d__` instead of
+`k__` (which the tokenizer silently ignores). Use the bundled converter for those:
+
+```bash
+python scripts/profiler_to_waypoint.py \
+    --input merged_metaphlan.tsv --format metaphlan \
+    --output dataset.parquet
+
+python scripts/profiler_to_waypoint.py \
+    --input reports/*.kreport --format kraken \
+    --output dataset.parquet
+
+python scripts/profiler_to_waypoint.py \
+    --input feature-table.tsv --format qiime2 \
+    --output dataset.parquet
+```
+
+See `references/data-preparation.md` for every input layout, rank handling, and the `d__`/`|` gotchas.
+
+### 2. Check vocabulary coverage before anything else
+
+Waypoint's vocabulary is fixed at pretraining time from Atlas. Taxa absent from it become `<unk>` and
+are **silently dropped** by `waypoint embed`; the paper names this as the models' main limitation. A
+sample whose taxa are all out-of-vocabulary yields a degenerate `[BOS][EOS]` embedding.
+
+```bash
+python scripts/vocab_coverage.py --model outpost-bio/Waypoint-6m --data dataset.parquet
+```
+
+It reports per-sample and abundance-weighted coverage and flags samples below a threshold. Treat
+median abundance-weighted coverage under ~0.8 as a reason to re-examine your taxonomy labels before
+trusting any downstream number.
+
+### 3. Embed samples
+
+```bash
+waypoint embed \
+    --model outpost-bio/Waypoint-6m \
+    --data dataset.parquet \
+    --output embeddings.parquet
+```
+
+Output is indexed by sample ID with columns `dim_0 … dim_{H-1}` (`H` = 256 for 6m, 512 for 45m,
+768 for 170m). Defaults: `--pooling last_token`, `--batch_size 32`, `--max_length 512`, device
+auto-detected (`cuda` → `mps` → `cpu`).
+
+Keep `--pooling last_token` unless you have a reason to change it: it matches how the checkpoints
+were pretrained and how `benchmark` and `finetune` pool. `mean` is a reasonable alternative for
+unsupervised use; `first_token`/`cls_token` return the BOS position and carry little signal in a
+causal LM.
+
+### 4. Fine-tune on your labels
+
+```bash
+# classification
+waypoint finetune \
+    --model outpost-bio/Waypoint-45m \
+    --data dataset.parquet \
+    --output_dir outputs/ft_disease \
+    --task_type classification \
+    --target "Disease Status" \
+    --config configs/finetune_classification.yaml
+
+# regression, with a categorical covariate one-hot appended to the pooled embedding
+waypoint finetune \
+    --model outpost-bio/Waypoint-45m \
+    --data dataset.parquet \
+    --output_dir outputs/ft_degradation \
+    --task_type regression \
+    --target "Degradation Rate" \
+    --covariate_column Drug \
+    --config configs/finetune_regression.yaml
+```
+
+Config paths resolve against the bundled `waypoint_bio/configs/` tree, so `configs/...` works from
+any directory without cloning.
+
+Defaults worth overriding for small datasets: `warmup_steps: 1000` (drop to ~50 so warmup finishes
+before early stopping), `num_epochs: 1` in the shipped configs (raise it — early stopping on
+validation loss is what actually terminates training), and `use_lora: true` when VRAM is tight
+(~1% of parameters trained; adapters are merged back before saving, so the checkpoint stays a plain
+`AutoModel`).
+
+Splits default to a random 80/10/10. **Set `split_column` to a `Split` column whenever samples are
+correlated** — repeated measures, one donor sampled over time, technical replicates — or a random
+split leaks and the test score is meaningless.
+
+Outputs land in `--output_dir`: `best_model/` (loadable by `embed`/`benchmark`),
+`test_metrics.json`, `training_log.csv` + `.html`, and `finetune_results.json`.
+
+### 5. Benchmark on Compass
+
+```bash
+waypoint benchmark --model outpost-bio/Waypoint-6m --output_dir outputs/benchmark
+waypoint benchmark --model outputs/pretrain/best_model --tasks 1 6 --output_dir outputs/smoke
+```
+
+Fine-tunes a fresh head per task and writes `benchmark_results.json`. Classification tasks score
+macro-F1; the one regression task scores R² clamped to [0, 1]; `final_score` is the unweighted mean
+across tasks. Full task table, metric keys, and result-file schema: `references/compass-benchmark.md`.
+
+### 6. Pretrain
+
+```bash
+waypoint pretrain \
+    --model_config configs/models/gpt2-45m.yaml \
+    --pretrain_config configs/pretraining.yaml \
+    --output_dir outputs/pretrain_45m
+```
+
+Downloads Atlas, builds a taxonomic tokenizer from the corpus, computes per-token abundance
+mean/std for z-score ordering, then trains with next-token prediction and early stopping. Add
+`--data my_corpus.parquet` to pretrain on your own waypoint-format corpus instead, and
+`--max_samples N` for a smoke test.
+
+Nine architectures ship, from `gpt2-6m.yaml` (8 layers, 256 hidden) to `gpt2-170m.yaml` (24 layers,
+768 hidden); per-head dimension is fixed at 64 throughout. `references/cli-reference.md` has the
+full table and every config key.
+
+## Scientific caveats
+
+These are load-bearing. Ignoring them produces numbers that look fine and mean nothing.
+
+- **Below ~1,000 labelled examples, Waypoint underperforms a random forest on raw abundances.** The
+  paper's crossover against the RF baseline sits near **10,000** training examples. Fit the baseline
+  first; only adopt the transformer if it wins on your data.
+- **Out-of-vocabulary taxa are dropped, not flagged.** Every Compass dataset carries some. Run
+  `scripts/vocab_coverage.py` and report the coverage alongside your results.
+- **45M, not 170M, was the best benchmark model.** Pretraining loss keeps falling with scale, but
+  downstream Compass score does not — start at 6m or 45m and only scale up if it demonstrably helps.
+- **Genus-level tokenisation is the default**, so species-level distinctions are collapsed. Changing
+  `taxon_rank` requires re-pretraining, not just re-tokenising.
+- **Compositional data.** Relative abundances are constrained to sum to 1; differences in one taxon
+  induce apparent changes in others. This affects interpretation of any per-taxon attribution.
+- **Batch and study effects dominate microbiome data.** Atlas spans MGnify pipelines v1.0–v5.0 and
+  four sequencing modalities. Never let a study or run boundary coincide with your label boundary.
+- **Not a clinical or diagnostic tool.** The model cards state this explicitly.
+
+## References
+
+- `references/cli-reference.md` — every subcommand flag, every config key, the model-size table.
+- `references/compass-benchmark.md` — the eight tasks, filters, metrics, `benchmark_results.json` schema.
+- `references/data-preparation.md` — waypoint format, profiler conversions, taxonomy string rules.
+- `references/python-api.md` — using the tokenizer, datasets, heads, and checkpoints from Python.
+
+## Scripts
+
+- `scripts/profiler_to_waypoint.py` — MetaPhlAn / Kraken2 / QIIME 2 / generic lineage tables → waypoint format.
+- `scripts/vocab_coverage.py` — tokenizer coverage report for a waypoint-format file.
+
+## Upstream
+
+Code [github.com/Outpost-Bio/waypoint](https://github.com/Outpost-Bio/waypoint) ·
+package `waypoint-bio` ·
+paper [bioRxiv 2026.05.02.722381](https://www.biorxiv.org/content/10.64898/2026.05.02.722381v2) ·
+community [Waypoint Slack](https://join.slack.com/t/outpostbio-waypoint/shared_invite/zt-3w6ivgtba-WJOCkdxiISxQpwVq9ZZxTA) ·
+contact `waypoint@outpost.bio`.
+
+Cite Treloar, N. J., Ur-Rehman, S., Yang, J., & Outpost Bio (2026). *Learning the Language of the
+Microbiome with Transformers.* bioRxiv. Per-artefact DOIs are listed at
+[outpost.bio/citations](https://www.outpost.bio/citations).

package/skills/waypoint-bio/references/cli-reference.md

@@ -0,0 +1,210 @@
+# `waypoint` CLI reference
+
+Targets `waypoint-bio` 1.0.2 (PyPI) / 1.0.4 (GitHub main, commit `f45eee6`, 2026-07-16).
+
+```
+waypoint {pretrain,benchmark,finetune,embed,prepare-dataset} ...
+```
+
+Config paths are resolved first against the working directory, then against the bundled
+`waypoint_bio/configs/` tree inside the installed wheel. So `--config configs/benchmark.yaml`
+works from anywhere without cloning the repo. The same fallback applies to the bundled example
+data (`examples/abundance_matrix.tsv`, `examples/finetune_classification.parquet`, …).
+
+---
+
+## `waypoint prepare-dataset`
+
+Converts a sample × taxa abundance matrix into waypoint format.
+
+| Flag | Default | Notes |
+| --- | --- | --- |
+| `--input` | *required* | `.csv` / `.tsv` abundance matrix. |
+| `--output` | *required* | `.parquet` recommended; `.csv` supported. |
+| `--orientation` | `auto` | `auto`, `samples_as_rows`, `taxa_as_rows`. |
+| `--taxonomy_format` | `full` | `full` for lineage strings; a rank name (`genus`, `species`, …) to prefix bare names. |
+| `--no_normalize` | off | Skip row-normalisation to relative abundances. |
+| `--keep_zeros` | off | Keep zero-abundance entries in each sample's lists. |
+| `--metadata` | none | CSV/TSV/parquet of per-sample metadata, indexed by sample ID, merged in as extra columns. |
+
+`auto` treats the file as taxa-as-rows when the first column header is `taxonomy`, `lineage`,
+`taxon`, `otu`, or `#otu id` (case-insensitive); otherwise samples-as-rows with the first column
+as the sample ID.
+
+`--taxonomy_format genus` prefixes bare column names with `g__`. It disables higher-rank fallback,
+because a bare name carries no lineage to fall back to — prefer real lineage strings.
+
+---
+
+## `waypoint embed`
+
+One fixed-size vector per sample from a pretrained checkpoint. No fine-tuning, no labels needed.
+
+| Flag | Default | Notes |
+| --- | --- | --- |
+| `--model` | `outpost-bio/Waypoint-6m` | Hub id or local checkpoint directory. |
+| `--data` | *required* | Waypoint-format `.parquet` / `.csv` / `.tsv`. |
+| `--output` | *required* | `.parquet`, or `.csv` if the path ends in `.csv`. |
+| `--pooling` | `last_token` | `last_token`, `mean`, `first_token`, `cls_token`. |
+| `--batch_size` | `32` | |
+| `--max_length` | `512` | Truncates after ordering, so the least informative taxa are lost first. |
+| `--device` | auto | `cuda`, `mps`, or `cpu`; auto-detects in that order. |
+
+Output columns are `dim_0 … dim_{H-1}`, indexed by sample ID. Hidden size `H` is 256 (6m),
+512 (45m), 768 (170m).
+
+**Behaviour worth knowing:** tokens that map to `<unk>` are *dropped* before ordering, not encoded.
+A row with no in-vocabulary taxa still produces an output row, but its sequence is `[BOS][EOS]` and
+the embedding is meaningless. Run `scripts/vocab_coverage.py` first.
+
+Ordering: by descending abundance z-score when `token_std_means.parquet` is present (it ships with
+every published checkpoint and with `waypoint pretrain` output), otherwise by descending raw
+relative abundance.
+
+---
+
+## `waypoint finetune`
+
+Fine-tunes a checkpoint on your own labelled waypoint-format data.
+
+| Flag | Default | Notes |
+| --- | --- | --- |
+| `--model` | *required* | Hub id or local checkpoint. |
+| `--data` | *required* | Waypoint-format file containing `--target`. |
+| `--output_dir` | *required* | |
+| `--task_type` | *required* | `classification` or `regression`. |
+| `--target` | *required* | Target column name. |
+| `--covariate_column` | none | Categorical column, one-hot encoded and concatenated to the pooled embedding before the head. |
+| `--config` | task default | Flat YAML; defaults to the bundled classification/regression config. |
+
+### Fine-tuning config keys
+
+```yaml
+split_column: null        # column holding train/validation/test; null = random split
+val_fraction: 0.1
+test_fraction: 0.1
+
+max_length: 512           # must match the checkpoint's pretraining context
+pooling_strategy: last_token
+filter_unk_taxa: true     # drop out-of-vocabulary taxa rather than feed <unk>
+
+seed: 42
+learning_rate: 0.00003
+num_epochs: 1             # raise this; early stopping is what should terminate training
+batch_size: 64
+warmup_steps: 1000        # lower to ~50 for small datasets
+weight_decay: 0.001
+eval_strategy: steps
+eval_steps: 400
+logging_steps: 5
+patience: 5               # eval steps without improvement before early stopping
+save_total_limit: 1
+
+use_lora: false
+lora_r: 8
+lora_alpha: 16            # convention: 2 * r
+lora_dropout: 0.05
+lora_target_modules: [c_attn, c_proj]   # GPT-2 fused QKV and output projection
+lora_bias: none
+lora_fan_in_fan_out: true               # required for GPT-2 Conv1D layouts
+```
+
+`num_epochs: 1` in the shipped configs is tuned for the large Compass tasks. On a few-thousand-row
+dataset one epoch is a handful of optimizer steps and the model barely moves — raise `num_epochs`
+and let `patience` stop it. Likewise `eval_steps: 400` may never fire; lower it so early stopping
+and best-checkpoint selection can actually work.
+
+LoRA adapters are merged back into the base transformer before saving, so `best_model/` loads with
+a plain `AutoModel.from_pretrained` and works with `waypoint embed` and `waypoint benchmark`.
+
+### Outputs
+
+| Path | Contents |
+| --- | --- |
+| `best_model/` | Fine-tuned base transformer in standard HF format, plus tokenizer and `token_std_means.parquet`. |
+| `best_model/finetuned_model_state.pt` | Full torch state dict: transformer + head + covariate embedding. |
+| `validation_metrics.json`, `test_metrics.json` | Per-split scores, benchmark-equivalent. |
+| `training_log.csv`, `training_log.html` | Every row of `trainer.state.log_history`; the HTML is an interactive plotly line plot. |
+| `finetune_results.json` | Run config, label maps, covariate map, val/test scores. |
+
+---
+
+## `waypoint benchmark`
+
+| Flag | Default | Notes |
+| --- | --- | --- |
+| `--model` | `outpost-bio/Waypoint-6m` | Hub id or local checkpoint. |
+| `--config` | bundled `configs/benchmark.yaml` | Shared by all eight tasks. |
+| `--output_dir` | `outputs/benchmark` | |
+| `--tasks` | all 8 | Space-separated task numbers, e.g. `--tasks 1 6`. |
+| `--seed` | `42` | |
+| `--max_samples` | none | Caps each split; use for smoke tests only, never for a reported score. |
+
+`configs/benchmark.yaml` is the fine-tuning config applied identically to every task:
+`learning_rate: 3e-5`, `num_epochs: 1`, `batch_size: 64`, `warmup_steps: 1000`,
+`weight_decay: 0.001`, `patience: 5`, `pooling_strategy: last_token`, `eval_steps: 400`,
+`filter_unk_taxa: true`, `seed: 42`. Change it and your score is no longer comparable to the paper.
+
+The paper reports means over three independent runs. A single run is noisy; vary `--seed` and
+report the spread.
+
+---
+
+## `waypoint pretrain`
+
+| Flag | Default | Notes |
+| --- | --- | --- |
+| `--model_config` | `configs/models/gpt2-6m.yaml` | Architecture YAML. |
+| `--pretrain_config` | `configs/pretraining.yaml` | Hyperparameter YAML. |
+| `--output_dir` | `outputs/pretrain` | Best checkpoint written to `<output_dir>/best_model/`. |
+| `--max_samples` | none | Limit training samples for a quick test. |
+| `--data` | none | Local waypoint-format corpus instead of downloading Atlas. |
+
+Steps: download the Atlas `pretrain` split → build a taxonomic tokenizer from the corpus →
+compute per-token abundance mean/std for z-score ordering → train GPT-2 with next-token prediction
+and early stopping → save `best_model/`.
+
+### `configs/pretraining.yaml`
+
+```yaml
+training_type: next_token_prediction
+taxon_rank: genus              # tokenization rank; changing it means re-pretraining
+fallback_to_higher_rank: true  # use the most specific higher rank when genus is absent
+max_length: 512
+learning_rate: 0.001
+warmup_steps: 1000
+weight_decay: 0.001
+batch_size: 32
+num_epochs: 100
+patience: 10
+eval_steps: 3261
+save_steps: 3261
+logging_steps: 100
+val_split: 0.1
+seed: 42
+```
+
+### Architectures
+
+All share `model_type: gpt2`, `n_positions: 512`, and a fixed per-head dimension of 64.
+
+| Config | Layers | Hidden | Heads | ~Params |
+| --- | --- | --- | --- | --- |
+| `gpt2-6m.yaml` | 8 | 256 | 4 | 6M |
+| `gpt2-6m-mgm.yaml` | 8 | 256 | 8 | 6M — matches the MGM baseline architecture |
+| `gpt2-10m.yaml` | 8 | 320 | 5 | 10M |
+| `gpt2-18m.yaml` | 10 | 384 | 6 | 18M |
+| `gpt2-29m.yaml` | 12 | 448 | 7 | 29M |
+| `gpt2-45m.yaml` | 14 | 512 | 8 | 45M |
+| `gpt2-79m.yaml` | 16 | 640 | 10 | 79M |
+| `gpt2-85m-gpt-small.yaml` | 12 | 768 | 12 | 85M — GPT-2 small geometry |
+| `gpt2-170m.yaml` | 24 | 768 | 12 | 170M |
+
+Only 6m, 45m, and 170m are published as checkpoints. The rest exist so the paper's scaling study is
+reproducible; `gpt2-6m-mgm` isolates the effect of head count against the MGM baseline.
+
+Parameter counts exclude token and positional embeddings, so the Hub's reported sizes are larger
+(the 6m checkpoint reports ~10.1M, the 45m ~51.8M).
+
+Pretraining Atlas end to end is a multi-GPU-day job. Validate the pipeline with
+`--max_samples 5000` before committing to a full run.

package/skills/waypoint-bio/references/compass-benchmark.md

@@ -0,0 +1,124 @@
+# Compass: the eight-task microbiome benchmark
+
+`outpost-bio/Compass` on the Hugging Face Hub — gated, Apache 2.0, ~605 MB, ~62.8k rows across four
+Hub configurations. Eight tasks are derived from those four configurations by filtering and by
+choosing different target columns.
+
+Every configuration exposes `train` / `validation` / `test` splits and carries a `Split` column
+recording the same assignment.
+
+```python
+from datasets import load_dataset
+ds = load_dataset("outpost-bio/Compass", "mgnify-biomes")   # requires access + HF_TOKEN
+```
+
+## The four source datasets
+
+| Config | Source | Rows (train/val/test) | Extra columns |
+| --- | --- | --- | --- |
+| `mgnify-biomes` | MGnify metagenomic profiles across gut, skin, oral, marine, freshwater, soil, engineered systems | 33,121 / 4,139 / 4,139 | `Biome 1`–`Biome 5`, `Run Accession`, `Data Type`, `Sequencing Method`, `Pipeline Version`, `Study Accession` |
+| `handuo` | Han, Duo et al. — 16S amplicon study of drug–microbiome interactions in stool-derived communities | 3,168 / 396 / 396 | `SIC Name`, `Control`, `ATC Class`, `Sample ID` |
+| `mastrorilli` | Mastrorilli et al. — drug degradation by gut communities | 9,282 / 3,084 / 3,053 | `Degradation Rate`, `Drug`, `Sample ID` |
+| `roswall` | Roswall et al. — longitudinal infant gut cohort | 2,031 total | `Timepoint`, `Delivery Mode`, `Sample ID` |
+
+All configs carry `Taxa` and `Relative Abundances` as aligned list columns.
+
+## The eight tasks
+
+As defined in `waypoint_bio/benchmark.py`:
+
+| # | Internal id | Config | Targets | Type | Pre-filter |
+| --- | --- | --- | --- | --- | --- |
+| 1 | `1_biome` | `mgnify-biomes` | `Biome 1`–`Biome 5` | classification (5 outputs) | none |
+| 2 | `2_biome_gut` | `mgnify-biomes` | `Biome 4`, `Biome 5` | classification (2 outputs) | `Biome 3 == "Digestive system"` |
+| 3 | `3_sic` | `handuo` | `SIC Name` | classification | `SIC Name` starts with `SIC`, excludes `control` and `seed` |
+| 4 | `4_drug_non_drug` | `handuo` | `Control` | binary classification | none |
+| 5 | `5_drug_class` | `handuo` | `ATC Class` | classification | `ATC Class` not null |
+| 6 | `6_drug_degradation` | `mastrorilli` | `Degradation Rate` | regression | none; `Drug` used as covariate |
+| 7 | `7_infant_age` | `roswall` | `Timepoint` | classification | none |
+| 8 | `8_birth_mode` | `roswall` | `Delivery Mode` | binary classification | none |
+
+What each asks, in plain terms:
+
+1. **Biome classification** — predict all five levels of the MGnify biome ontology at once
+   (e.g. `root → Host-associated → Human → Digestive system → Large intestine`).
+2. **Gut biome classification** — same, restricted to digestive-system samples, predicting only the
+   two finest levels. Harder: the easy environmental separations are gone.
+3. **SIC classification** — identify which stool-derived in-vitro community a drug-perturbed sample
+   came from.
+4. **Drug vs. control** — did this community receive a drug?
+5. **Drug class** — recover the ATC class of the applied drug from the resulting composition.
+6. **Drug degradation** — regress the degradation rate from composition plus drug identity. The
+   `Drug` covariate is one-hot encoded and concatenated to the pooled embedding.
+7. **Infant age** — predict the sampling timepoint from an infant gut sample.
+8. **Birth mode** — vaginal vs. caesarean delivery.
+
+## Scoring
+
+- **Classification:** macro-averaged F1 — F1 per class, averaged with equal weight. Chosen so the
+  metric is not dominated by majority classes. Where a task has several target columns (1 and 2),
+  the per-target macro-F1s are averaged.
+- **Regression (task 6):** R², clamped to `[0, 1]` so it shares a scale with the F1 scores. A
+  negative R² therefore reads as `0.0`, not as "worse than the mean".
+- **Final score:** unweighted arithmetic mean of the eight task scores.
+
+Supplementary metrics are computed and stored but do not enter the score: one-vs-one macro ROC-AUC,
+macro PR-AUC (pairwise average precision over the same OVO pairs), balanced accuracy, plain
+accuracy; and MSE, Pearson, Spearman for regression.
+
+## `benchmark_results.json`
+
+```
+benchmark_results.json
+├── model          string — the value passed to --model
+├── final_score    number — mean of every results[].score
+└── results        array, one object per task
+    ├── task       string — "1_biome", "6_drug_degradation", ...
+    ├── task_type  "classification" | "regression"
+    ├── score      number — macro F1, or R² clamped to [0,1]
+    └── metrics    object — keys depend on task_type
+```
+
+`metrics` keys are suffixed with the target column name:
+
+| Task type | Keys |
+| --- | --- |
+| `classification` | `accuracy_<target>`, `balanced_accuracy_<target>`, `f1_macro_<target>`; with probabilities, binary `roc_auc_<target>` / `pr_auc_<target>` or multiclass `roc_auc_macro_ovo_<target>` / `pr_auc_macro_ovo_<target>`. Means: `f1_macro_mean`, optionally `roc_auc_mean`, `pr_auc_mean`. |
+| `regression` | `mse_<target>`, `r2_<target>`, usually `pearson_<target>` and `spearman_<target>`. Mean: `r2_mean`. |
+
+Example:
+
+```json
+{
+  "model": "outpost-bio/Waypoint-6m",
+  "final_score": 0.71,
+  "results": [
+    {"task": "1_biome", "task_type": "classification", "score": 0.65,
+     "metrics": {"f1_macro_mean": 0.65, "roc_auc_mean": 0.81, "pr_auc_mean": 0.74}},
+    {"task": "6_drug_degradation", "task_type": "regression", "score": 0.42,
+     "metrics": {"mse_Degradation Rate": 0.019, "r2_Degradation Rate": 0.44, "r2_mean": 0.44}}
+  ]
+}
+```
+
+The numbers above are the illustrative values from the upstream README, not measured results.
+
+## Interpreting a benchmark run
+
+**Baselines matter more than the absolute score.** The paper compares Waypoint against classical
+baselines (random forest and logistic regression on relative abundances) and against MGM, the prior
+microbiome foundation model. Two findings shape how a Compass number should be read:
+
+- Waypoint beats the random-forest baseline from roughly **10,000 training examples upward**, and
+  *loses* to it below about 1,000. Report the training-set size next to any score.
+- Baselines can use every taxon; the transformer sees only its fixed vocabulary. The paper's fair
+  comparison is the `(no unk)` baseline, with out-of-vocabulary taxa stripped from the baseline's
+  input too. Compare against that, not against a baseline given the full table.
+
+**Scale does not monotonically help.** Pretraining loss falls all the way to 170M, but the best
+Compass score in the paper came from the **45M** model. Non-pretrained transformers get *worse* as
+they grow — the gain from scale is a property of pretraining, not of capacity.
+
+**Reproducibility.** Use the bundled `configs/benchmark.yaml` unchanged, do not pass `--max_samples`,
+and run at least three seeds. Comparing a run that changed the learning rate or capped splits against
+published numbers is not a comparison.