bruise-data 1.2.0__tar.gz

bruise_data-1.2.0/PKG-INFO

@@ -0,0 +1,607 @@
+Metadata-Version: 2.4
+Name: bruise-data
+Version: 1.2.0
+Summary: PyTorch loaders for the BRUISE_DATA collection (NIJ, Fenwick, ALS, mimics).
+Keywords: pytorch,dataset,dataloader,bruise,segmentation,medical-imaging
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: torch>=1.12
+Requires-Dist: numpy
+Requires-Dist: pandas>=1.3
+Requires-Dist: Pillow>=9
+Provides-Extra: augment
+Requires-Dist: albumentations; extra == "augment"
+
+# BRUISE_DATA
+
+All bruise images (NIJ and Fenwick) and bruise look-alike images (mimics) in one consistent layout, plus a PyTorch loader that gives you `train`, `val` and `test` DataLoaders for any mix of them in one line. New datasets plug in by adding a folder with a `manifest.csv` ([section 11.1](#111-adding-a-new-dataset-plug-and-play)):
+
+```python
+from bruise_data import get_dataloaders
+train_dl, val_dl, test_dl = get_dataloaders(["nij", "fenwick", "mimics"])
+```
+
+| Dataset | What it is | Images (train / val / test) | Masks | Skin tone |
+|---|---|---|---|---|
+| `nij` | NIJ white-light bruise photos, Labelbox full-consensus subset | 664 / 167 / 185 (subjects: 92 / 23 / 28) | yes (Paul; majority vote on test) | L\*a\*b\* + ITA per visit |
+| `nij_als` | NIJ ALS photos (415–535 nm), same subjects and folds as `nij` ([3.5](#35-als-nij_als-fenwick_als)) | ≈ 6,868 train+val / 1,459 test (July 2026 export; built on the server) | yes (Paul; majority vote on test) | as `nij` |
+| `fenwick` | Fenwick white-light bruise photos | 1,536 / 385 / 127 (subjects: 85 / 20 / 15) | yes (mzehra2; majority vote on test) | L\*a\*b\* + ITA per lesion |
+| `fenwick_als` | Fenwick ALS photos (410/455 nm), same subjects and folds as `fenwick` | counted when downloaded | yes (mzehra2; majority vote on test) | as `fenwick` |
+| `mimics` | Open-source images of conditions that look like bruises, in 9 taxonomy classes | 4,360 / 543 / 524 | no bruise (empty mask) | Fitzpatrick / Monk where the source gives it |
+| `extra_bruise` | Web bruise photos (no masks, classification only) | 68 / 12 / 8 | none | none |
+
+Hard negatives (healthy skin, generic rashes, moles, chronic ulcers) are left out on purpose.
+
+---
+
+## Contents
+
+1. [Quick start](#1-quick-start)
+2. [Folder layout](#2-folder-layout)
+3. [The datasets](#3-the-datasets)
+4. [Naming rules](#4-naming-rules)
+5. [manifest.csv columns](#5-manifestcsv-columns)
+6. [Using the dataloader](#6-using-the-dataloader)
+7. [Recipes](#7-recipes)
+8. [Skin tone (L\*a\*b\*)](#8-skin-tone-lab)
+9. [Command-line tools](#9-command-line-tools)
+10. [Running on the GPU server](#10-running-on-the-gpu-server)
+11. [Rebuilding or adding data](#11-rebuilding-or-adding-data) (incl. [11.1 adding a new dataset](#111-adding-a-new-dataset-plug-and-play), [11.2 ALS data](#112-adding-the-als-data-nij_als-fenwick_als))
+12. [Licences and data rules](#12-licences-and-data-rules)
+13. [Troubleshooting](#13-troubleshooting)
+
+---
+
+## 1. Quick start
+
+**Requirements:** Python 3.9+, `torch`, `numpy`, `pandas`, `Pillow`. Nothing else; `albumentations` is optional.
+
+```bash
+cd BRUISE_DATA
+pip install -r requirements.txt          # skip if torch/pandas/pillow are already installed
+
+python -m bruise_data check              # 1. confirm files are there and splits don't leak
+python -m bruise_data summary            # 2. see how many images each split has
+python examples/quick_start.py           # 3. load one batch and print what is inside
+```
+
+In your own script:
+
+```python
+import sys; sys.path.insert(0, "/path/to/BRUISE_DATA")    # not needed if you run from inside BRUISE_DATA
+from bruise_data import get_dataloaders
+
+if __name__ == "__main__":                                  # required on Windows when num_workers > 0
+    train_dl, val_dl, test_dl = get_dataloaders(["nij", "fenwick", "mimics"], image_size=512, batch_size=8)
+    for batch in train_dl:
+        images = batch["image"]     # float  [8, 3, 512, 512]  (ImageNet-normalised)
+        masks  = batch["mask"]      # float  [8, 1, 512, 512]  1 = bruise; all zeros for mimics
+        labels = batch["label"]     # int    [8]               0 = bruise, 1.. = mimic classes
+        break
+```
+
+---
+
+## 2. Folder layout
+
+```
+BRUISE_DATA/
+├── README.md                 this file
+├── requirements.txt
+├── bruise_data/              the Python loader package (import bruise_data)
+├── examples/                 quick_start.py, train_segmentation.py
+├── tools/build_bruise_data.py   rebuilds every folder below from the original sources
+├── tools/labelbox_als/       Labelbox ALS downloader for Hopper (notebook, Slurm job, README)
+│
+├── nij/
+│   ├── train/   images/  masks/            primary annotator (paul)
+│   ├── val/     images/  masks/            primary annotator, fold 0 of the subject 5-fold split
+│   ├── test/    images/  masks/            majority vote of paul + gbarimah + erik
+│   │            masks_by_annotator/{paul,gbarimah,erik}/
+│   ├── alt_annotators/{gbarimah,erik}/{images,masks}/   other annotators' train pools
+│   ├── metadata/skin_tone_lab.csv          L*a*b* per subject / visit / arm
+│   ├── metadata/skin_tone_lab_subject_median.csv
+│   ├── metadata/original/                  the untouched source CSVs
+│   ├── reports/build_logs/                 how the dataset was built (consensus filter, leakage check)
+│   └── manifest.csv                        one row per image (and per annotator)
+│
+├── fenwick/                                same layout; primary = mzehra2, alt = hliu36, nmousta5
+│
+├── nij_als/, fenwick_als/                  same layout (train/ val/ test/ + masks_by_annotator/), no alt_annotators;
+│                                           server only (≈ 55 GB), see section 11.2
+│
+├── mimics/
+│   ├── <class>/<source_dataset>/<image>    e.g. purpura/scin/..., burn/sunburn_roboflow/...
+│   ├── manifest.csv
+│   ├── mimic_class_map.csv                 every original diagnosis -> taxonomy class
+│   └── README.md                           per-class counts and sources
+│
+└── extra_bruise/
+    ├── <source_dataset>/<image>
+    └── manifest.csv
+```
+
+Every image and mask is a hard link to the original file, so this tree takes almost no extra disk space on the laptop. The zip archives hold real copies.
+
+---
+
+## 3. The datasets
+
+### 3.1 NIJ (`nij/`)
+
+- **What it is:** white-light (`_N`) photos of experimentally induced bruises on the upper and lower arm, from GMU and partner sites. Only images where all three Labelbox raters said "bruise visible" (full consensus) are included.
+- **Annotators:**
+  - `paul` is the primary annotator; his masks are used for `train`/`val`.
+  - `gbarimah` (862 images) and `erik` (763 images) have their own train pools under `alt_annotators/`.
+- **Test set:** 185 images from 28 subjects. Nobody's train pool contains these subjects.
+  - Masks: `test/masks/` holds the majority vote (a pixel is bruise if at least 2 of the 3 annotators marked it).
+  - `test/masks_by_annotator/` holds each annotator's own mask.
+- **Validation:** fold 0 of a subject-level 5-fold split, so a subject is never in both train and val. Any other fold can be used instead (`val_fold=`).
+- **Image names:** `AB012_V03_UA_N` means subject AB012, visit 3, upper arm (`UA`; `LA` = lower arm), white light.
+
+### 3.2 Fenwick (`fenwick/`)
+
+- **What it is:** white-light photos of bruises from the Fenwick study.
+- **Annotators:** the top-3 labelers from the labeler-selection study.
+  - `mzehra2` is the primary annotator.
+  - `hliu36` (1,124 images) and `nmousta5` (507 images) are under `alt_annotators/`.
+- **Test set:** 127 images from 15 subjects.
+  - Masks: majority vote of the three labelers in `test/masks/`, individual masks in `test/masks_by_annotator/`.
+- **Validation:** fold 0 of the subject 5-fold split, same as NIJ.
+- **Image names:** `123_F_2-4_WL_Bruise_...` means participant 123, lesion F (example IDs).
+
+### 3.3 Mimics (`mimics/`)
+
+Classes follow the bruise taxonomy (`BRUISE_LOOKALIKE/TAXONOMY/Bruise Taxonomy.pdf`): its non-bruise leaf classes that can be mistaken for a bruise. Each class folder has one sub-folder per source dataset.
+
+| Class | Taxonomy branch | Contains | Total images |
+|---|---|---|---|
+| `purpura` | Ecchymosis → Purpura / large ecchymosis | purpura, vasculitis, pigmented purpuric eruption, hemosiderin staining, erythema elevatum diutinum | 771 |
+| `petechiae` | Ecchymosis → Petechiae (<3 mm) | petechiae | 10 (merged into `purpura` by default) |
+| `hemangioma` | Vascular mimic → no blanching | hemangioma, port wine stain, pyogenic granuloma, Kaposi sarcoma, lymphangioma | 1,071 |
+| `erythema` | Vascular mimic → blanches | cellulitis, erythema multiforme / nodosum / migrans, annular erythemas, urticaria, phytophotodermatitis | 1,197 |
+| `pigmentary_lesion` | Pigmentary lesion (melanin, endogenous / exogenous) | melasma, hyperpigmentation, PIH, acanthosis nigricans, arsenical pigmentation, fixed drug eruption, dark circles | 1,521 |
+| `burn` | Wound → Burn | thermal burns, sunburn, erythema ab igne | 589 |
+| `abrasion` | Wound → Abrasion | scrapes, grazes | 138 |
+| `laceration` | Wound → Laceration | blunt-force tears | 61 |
+| `incised_wound` | Wound → Incised wound | cuts, stab wounds | 69 |
+
+- **Splits:** train/val/test are stored in `manifest.csv`, not in folders, so the folders stay `class/source`. The split is about 80/10/10. Images from the same patient (SCIN case, PAD-UFES patient) and near-duplicate photos always share a split.
+- **Filters applied:** short side ≥ 200 px, no cut-outs on white backgrounds, one image per near-duplicate group.
+- **Exact mapping:** `mimic_class_map.csv` lists every original diagnosis name and its class. `mimics/README.md` gives counts per class, source and split.
+- **Not filled yet:** `ecchymosis` (non-traumatic) and `surface_mark` (ink, paint, tattoo) have no open source yet, so they have no folder.
+- **Mixed source to watch:** in `pigmentary_lesion`, 740 of 1,521 images are arsenicosis photos (`skin_diseases_normal_arsenic`), often palms with keratoses. Use `exclude_mimic_sources=["skin_diseases_normal_arsenic"]` if you want a cleaner class.
+
+### 3.4 Extra bruise (`extra_bruise/`)
+
+88 unique web photos of bruises from the Hugging Face "Clinical Wound Dataset" (`SaniaSalah/wound`) and the Kaggle wound dataset (`yasinpratomo`), with duplicates removed. They have no masks, so the loader only accepts them with `task="classification"`.
+
+### 3.5 ALS (`nij_als/`, `fenwick_als/`)
+
+- **What it is:** the same studies photographed under an alternate light source (narrow-band light + barrier filter), downloaded from Labelbox with `tools/labelbox_als/` (section 11.2).
+  - NIJ: wavelengths 415, 450, 475, 495, 515, 535 nm. The filter letter (`Y`/`O`/`R`) is in `light_filter`. UV photos are left out.
+  - Fenwick: `ALS_410_455`.
+- **Selection:** the same rule as white light.
+  - NIJ: Paul's mask, and every Labelbox rater said "bruise visible".
+  - Fenwick: mzehra2 drew a bruise mask.
+  - Only the primary annotator is in train/val (no `alt_annotators/`).
+- **Same subjects, same folds:** every subject keeps its white-light fold, and the white-light test subjects are the only ALS test subjects. So `get_dataloaders(["nij", "nij_als"])` is leak-free, and `check` verifies it across the pair. Subjects that only have ALS photos are added to the lightest folds.
+- **Test set:** majority vote of the three annotators (paul / gbarimah / erik, or mzehra2 / hliu36 / nmousta5), as for white light. Individual masks are in `test/masks_by_annotator/`.
+- **Image names:** `GM059_V11_LA_415_Y` = subject GM059, visit 11, lower arm, 415 nm, yellow filter. `182_K_2-6_ALS_410_455_Bruise_...` = Fenwick participant 182, lesion K.
+- **Where:** built on the GPU server only. The NIJ part is about 55 GB, too large for the laptop or a zip.
+
+---
+
+## 4. Naming rules
+
+These are the same in every folder:
+
+- **Splits** are always `train`, `val`, `test`. No "fixed", "holdout" or "minus".
+- **Images** are in `images/`, **masks** in `masks/`. A mask has the image's stem plus `.png`. Mask pixels are 0 = background, 255 = bruise; the loader turns them into 0/1.
+- **Folder and file names** are lower-case `snake_case`.
+- **Dataset names** are the folder names: `nij`, `fenwick`, `mimics`, `extra_bruise`, plus any folder you add with a `manifest.csv`.
+- **`image_id`** is `<dataset>__<original stem>`, unique across everything.
+- **Skin tone** always lives in `metadata/skin_tone_lab.csv`.
+- **Manifests:** each dataset has exactly one `manifest.csv`, and all manifests share the same columns (next section).
+- **Paths** inside manifests are relative to `BRUISE_DATA/`, so the folder can be moved anywhere.
+
+---
+
+## 5. `manifest.csv` columns
+
+| Column | Meaning |
+|---|---|
+| `image_id` | Unique ID: `<dataset>__<stem>` |
+| `dataset` | `nij`, `fenwick`, `mimics`, `extra_bruise` |
+| `split` | `train` / `val` / `test` (for NIJ/Fenwick, `val` = fold 0; the loader can re-cut with `val_fold`) |
+| `fold` | Subject-level fold 0–4 for NIJ/Fenwick train/val rows; `-1` otherwise |
+| `class_name` | `bruise` or a mimic class |
+| `is_bruise` | True for NIJ, Fenwick, extra_bruise |
+| `source_dataset` | Where the image came from (`nij`, `fenwick`, `scin`, `dermnet`, ...) |
+| `source_class` | Original label from that source (e.g. "Pigmented purpuric eruption") |
+| `annotator` | Whose mask `mask_path` points to: `paul`, `gbarimah`, `erik`, `mzehra2`, `hliu36`, `nmousta5`, `majority` (test only), `none` (mimics) |
+| `is_primary_annotator` | True for the default rows (primary annotator for train/val, majority vote for test) |
+| `image_path`, `mask_path` | Relative to `BRUISE_DATA/`. `mask_path` is empty for mimics and extra_bruise |
+| `subject_id` | Participant / patient / case ID (for leak-free splits) |
+| `lesion_id` | Fenwick lesion letter |
+| `visit` | NIJ visit number |
+| `near_dup_group` | Mimics: images whose perceptual hash matches |
+| `width`, `height` | Original image size in pixels |
+| `mask_positive_pixels` | Bruise pixels in the mask (5 NIJ train images have empty masks) |
+| `skin_L`, `skin_a`, `skin_b`, `skin_ita` | Colorimeter CIE L\*a\*b\* of normal skin next to the bruise, and ITA° |
+| `lesion_L`, `lesion_a`, `lesion_b` | Colorimeter L\*a\*b\* on the bruise itself |
+| `ita_category` | Very Light / Light / Intermediate / Tan / Brown / Dark |
+| `fitzpatrick`, `monk_tone` | Mimics, where the source provides it |
+| `body_site` | NIJ `upper_arm` / `lower_arm`; mimics where known |
+| `modality`, `lighting` | Photo type; `lighting` is `white_light`, or `als` for `nij_als` / `fenwick_als` |
+| `licence`, `commercial_use` | See [section 12](#12-licences-and-data-rules) |
+| `original_file` | File name or path in the original source |
+| `wavelength`, `light_filter` | ALS only: NIJ `415`…`535` + filter `Y`/`O`/`R`; Fenwick `ALS_410_455`. Empty for white light |
+
+NIJ and Fenwick manifests have one row per image **per annotator**, so an image's masks from different annotators sit side by side. The loader always picks one annotator per dataset, so you never get the same image twice.
+
+---
+
+## 6. Using the dataloader
+
+### 6.1 `get_dataloaders(...)` → `(train_dl, val_dl, test_dl)`
+
+```python
+get_dataloaders(
+    datasets=("nij", "fenwick"),     # any dataset folder: "nij", "fenwick", "mimics", "extra_bruise", ... (list_datasets())
+    task="segmentation",             # or "classification"
+    # --- which mimics -------------------------------------------------------------
+    mimic_classes="all",             # or a list, e.g. ["purpura", "hemangioma"]
+    mimic_sources=None,              # keep only these sources, e.g. ["scin", "fitzpatrick17k"]
+    exclude_mimic_sources=None,      # drop these sources, e.g. ["dermnet"]
+    merge=None,                      # class renames; default {"petechiae": "purpura"}; {} = no merging
+    min_class_images=50,             # with mimic_classes="all", skip classes smaller than this
+    binary=False,                    # True: labels are 0 = non_bruise (all mimic classes in one), 1 = bruise
+    commercial_only=False,           # True = only images whose licence allows commercial use
+    # --- which masks and folds -----------------------------------------------------
+    annotator="primary",             # train/val masks: "primary", an annotator name, or a dict
+    test_annotator="majority",       # test masks: "majority", "primary", a name, or a dict
+    val_fold=0,                      # which subject fold (0-4) is val for NIJ/Fenwick
+    # --- images ------------------------------------------------------------------
+    image_size=512,                  # output is image_size x image_size
+    resize="pad",                    # "pad" keeps aspect ratio (black borders), "stretch" squashes
+    augment=True,                    # light train augmentation (crop, flip, ±15° rotation, brightness/contrast/saturation)
+    train_transform=None,            # your own transform (e.g. albumentations) replaces the built-in one
+    eval_transform=None,
+    # --- loader ------------------------------------------------------------------
+    batch_size=8, eval_batch_size=None, num_workers=4,
+    balance="dataset",               # None, "dataset", "class" or "source" (see below)
+    seed=42,
+    root=None,                       # BRUISE_DATA folder; default = the folder holding bruise_data/
+    verbose=True,                    # prints a count table
+)
+```
+
+**What the loader does, step by step:**
+
+1. Reads `manifest.csv` of each dataset you asked for.
+2. Filters rows. Mimics are filtered by class, source and licence. NIJ/Fenwick rows are reduced to one annotator (`annotator` for train/val, `test_annotator` for test).
+3. Assigns splits. For NIJ/Fenwick, `fold == val_fold` becomes `val`, the other folds `train`, and the test set never changes. Mimic splits come straight from the manifest.
+4. Gives every class an integer label: `0 = bruise`, then the mimic classes in alphabetical order. The list is in `train_dl.dataset.class_names`. With `binary=True` there are only two, and bruise is the positive class: `0 = non_bruise`, `1 = bruise`.
+5. For each image:
+   1. Loads it. Large JPEGs are decoded at reduced resolution, which is much faster for NIJ's 24 MP photos.
+   2. Loads the mask, or uses an all-zero mask for mimics.
+   3. Applies the transform and returns a dict.
+6. Draws training batches with `WeightedRandomSampler` when `balance` is set:
+   - `"dataset"`: NIJ, Fenwick and mimics are drawn equally often. This stops 5,000 mimics from drowning 664 NIJ images.
+   - `"class"`: every class (bruise and each mimic class) is drawn equally often.
+   - `"source"`: every source dataset is drawn equally often.
+   - `None`: plain shuffling.
+   Val and test are never re-weighted or shuffled.
+
+### 6.2 What one batch contains
+
+| Key | Type / shape | Notes |
+|---|---|---|
+| `image` | float `[B, 3, S, S]` | ImageNet-normalised; `bruise_data.denormalize()` undoes it for plotting |
+| `mask` | float `[B, 1, S, S]` | 1 = bruise. Segmentation only. All zeros for mimics |
+| `label` | int `[B]` | index into `class_names` |
+| `class_name`, `dataset`, `source_dataset`, `subject_id`, `image_id` | list of str | for analysis / per-group metrics |
+| `is_bruise` | bool `[B]` | |
+| `skin_L`, `skin_a`, `skin_b`, `skin_ita`, `lesion_L`, `lesion_a`, `lesion_b` | float `[B]` | NaN when unknown (mimics) |
+| `fitzpatrick`, `monk_tone` | float `[B]` | NaN when unknown |
+
+### 6.3 Other functions
+
+| Function | Use |
+|---|---|
+| `get_test_loaders(datasets, ...)` | `{"nij": dl, "fenwick": dl, "mimics": dl}`, one test loader per dataset, to report each one separately |
+| `get_datasets(datasets, ...)` | Returns `(train_ds, val_ds, test_ds)` PyTorch Datasets, if you want to build DataLoaders yourself |
+| `load_index(datasets, ...)` | The filtered table as a pandas DataFrame: no images loaded, good for counting or custom splits |
+| `summary(df)` | Count table per dataset / class / split |
+| `check_splits()` | Runs all leakage and file checks; returns a list of problems |
+| `BruiseTransform(image_size, train=...)` | The built-in transform, if you want to tweak it (`vflip=0.5`, `rotate=30`, `jitter=0`, `normalize=False`, ...) |
+| `read_manifest("nij")` | Raw manifest as a DataFrame (missing optional columns filled with defaults) |
+| `list_datasets()` | Every dataset found: each folder of `BRUISE_DATA/` that has a `manifest.csv` |
+| `dataset_info("nij")` | What a manifest holds: annotators, primary, masks, majority-vote test, folds |
+
+---
+
+## 7. Recipes
+
+**Bruise segmentation on NIJ only, primary annotator (the setup of earlier experiments):**
+```python
+train_dl, val_dl, test_dl = get_dataloaders(["nij"], balance=None)
+```
+
+**NIJ + Fenwick + all mimics as negatives (segmentation):**
+```python
+train_dl, val_dl, test_dl = get_dataloaders(["nij", "fenwick", "mimics"], balance="dataset")
+```
+Mimic images come with empty masks, so the model learns not to fire on purpura, burns, erythema and the rest.
+
+**Only the strongest mimics, without DermNet:**
+```python
+get_dataloaders(["fenwick", "mimics"], mimic_classes=["purpura", "hemangioma", "pigmentary_lesion"],
+                exclude_mimic_sources=["dermnet"])
+```
+
+**Classification (bruise vs. each mimic class), including the extra web bruises:**
+```python
+train_dl, val_dl, test_dl = get_dataloaders(["nij", "fenwick", "mimics", "extra_bruise"],
+                                            task="classification", image_size=224, balance="class")
+print(train_dl.dataset.class_names)
+# ['bruise', 'abrasion', 'burn', 'erythema', 'hemangioma', 'incised_wound', 'laceration', 'pigmentary_lesion', 'purpura']
+```
+
+**Keep petechiae as its own class:**
+```python
+get_dataloaders(["mimics"], task="classification", merge={}, min_class_images=0)
+```
+
+**Train on a different annotator; test against one annotator instead of the majority:**
+```python
+get_dataloaders(["nij"], annotator="gbarimah", test_annotator="paul")
+get_dataloaders(["nij", "fenwick"], annotator={"nij": "erik", "fenwick": "hliu36"})
+```
+
+**5-fold cross-validation:**
+```python
+for k in range(5):
+    train_dl, val_dl, test_dl = get_dataloaders(["nij", "fenwick"], val_fold=k, verbose=False)
+    ...
+```
+
+**Your own albumentations pipeline** (the built-in transform is replaced; add Normalize + ToTensorV2 yourself):
+```python
+import albumentations as A
+from albumentations.pytorch import ToTensorV2
+train_tf = A.Compose([A.LongestMaxSize(640), A.PadIfNeeded(640, 640), A.RandomCrop(512, 512),
+                      A.HorizontalFlip(), A.Normalize(), ToTensorV2()])
+eval_tf  = A.Compose([A.LongestMaxSize(512), A.PadIfNeeded(512, 512), A.Normalize(), ToTensorV2()])
+get_dataloaders(["nij", "mimics"], train_transform=train_tf, eval_transform=eval_tf)
+```
+With a custom transform, images are decoded at full resolution, so your transform should do the resizing.
+
+**Per-dataset test scores:**
+```python
+for name, dl in get_test_loaders(["nij", "fenwick", "mimics"], image_size=512).items():
+    print(name, evaluate(model, dl))     # mimics: count false-positive pixels / images
+```
+
+**Results by skin tone:**
+```python
+for batch in test_dl:
+    pred = model(batch["image"])
+    # batch["skin_ita"] / batch["skin_L"] / batch["subject_id"] let you group the metric
+```
+
+**ALS (server only, section 3.5):**
+```python
+get_dataloaders(["nij", "nij_als", "fenwick", "fenwick_als"])     # white light + ALS, leak-free (shared folds)
+get_dataloaders(["nij_als", "fenwick_als"])                       # ALS only
+tests = get_test_loaders(["nij", "nij_als"])                      # report white light and ALS separately
+
+# one wavelength only (e.g. 415 nm): filter the index, then build the Datasets yourself
+from bruise_data import load_index, BruiseDataset, BruiseTransform
+df = load_index(["nij_als"])
+df = df[df.wavelength.astype(str) == "415"]
+parts = {s: df[df.split == s].copy() for s in ("train", "val", "test")}
+for part in parts.values():
+    part.attrs = dict(df.attrs)                                   # keeps class names and root
+train_ds = BruiseDataset(parts["train"], transform=BruiseTransform(512, train=True), fast_decode=1024)
+val_ds = BruiseDataset(parts["val"], transform=BruiseTransform(512, train=False), fast_decode=1024)
+test_ds = BruiseDataset(parts["test"], transform=BruiseTransform(512, train=False), fast_decode=1024)
+```
+
+**Only the table (no images):**
+```python
+from bruise_data import load_index
+df = load_index(["nij", "fenwick", "mimics"])
+df.groupby(["split", "ita_category"]).size()
+```
+
+A full, runnable training loop is in `examples/train_segmentation.py`.
+
+---
+
+## 8. Skin tone (L\*a\*b\*)
+
+Both bruise datasets carry colorimeter readings measured during the study (CIE L\*a\*b\*). ITA° = arctan((L\* − 50) / b\*) × 180/π, and `ita_category` uses the standard bands: >55 Very Light, 41–55 Light, 28–41 Intermediate, 10–28 Tan, −30–10 Brown, < −30 Dark.
+
+| | NIJ | Fenwick |
+|---|---|---|
+| File | `nij/metadata/skin_tone_lab.csv` | `fenwick/metadata/skin_tone_lab.csv` |
+| Key | `subject_id, visit, arm` (UA/LA) | `subject_id, lesion_id` |
+| Skin (`skin_*`) | "around" reading next to the bruise | `cie_skin_l/a/b` |
+| Lesion (`lesion_*`) | "bruise" reading on the bruise | `cie_lesion_l/a/b` |
+| Source | `final_upperarm_dataset.csv`, `final_lowerarm_dataset.csv` | REDCap export `IAL_Fenwick_10_12_2026.csv` |
+| Fallback | subject median ("around", same arm) when the visit has no reading | none needed |
+| Coverage | 99.7% of images (skin), 97.9% (lesion) | 100% |
+
+The untouched source files are in each dataset's `metadata/original/`. Skin-tone values are already joined into every manifest row and returned in every batch.
+
+---
+
+## 9. Command-line tools
+
+Run from inside `BRUISE_DATA/`:
+
+```bash
+python -m bruise_data datasets                      # every dataset found, its annotators / masks / folds
+python -m bruise_data check                         # every dataset: files exist + no subject/near-duplicate leakage
+python -m bruise_data check --deep 50               # also open 50 images per dataset, compare mask size
+python -m bruise_data summary --datasets nij fenwick mimics
+python -m bruise_data summary --datasets nij fenwick mimics extra_bruise --task classification
+python -m bruise_data classes                       # mimic counts per class x source x split
+python -m bruise_data preview --datasets nij mimics --out preview.jpg   # 16 training images, bruise mask in red
+```
+
+---
+
+## 10. Running on the GPU server
+
+1. Copy the data: unzip all four archives into the **same** folder, e.g. `/home/data/bruise/BRUISE_DATA/`. Every path in the manifests is relative, so nothing has to be edited.
+   | Archive | Contains | Size |
+   |---|---|---|
+   | `bruise_data_code.zip` | `README.md`, `requirements.txt`, `bruise_data/`, `examples/`, `tools/` | < 1 MB |
+   | `nij.zip` | `nij/` | 6.6 GB (4 GB of it is `alt_annotators/`) |
+   | `fenwick.zip` | `fenwick/` | 0.6 GB |
+   | `mimics.zip` | `mimics/` and `extra_bruise/` | 2.6 GB |
+   ```bash
+   # on the laptop (Git Bash): send the zips (~10.4 GB)
+   scp C:/BRUISE_ORGANIZED/KIYARASH_DATASET/BRUISE_DATA_zips/*.zip tbommawa@mlidl:/home/data/bruise/
+
+   # on the server: unzip (needs ~11 GB more free space), make it readable for everyone, remove the zips
+   df -h /home/data
+   mkdir -p /home/data/bruise/BRUISE_DATA && cd /home/data/bruise/BRUISE_DATA
+   for z in bruise_data_code nij fenwick mimics; do unzip -q /home/data/bruise/$z.zip; done
+   chmod -R a+rX /home/data/bruise/BRUISE_DATA
+   rm /home/data/bruise/bruise_data_code.zip /home/data/bruise/nij.zip /home/data/bruise/fenwick.zip /home/data/bruise/mimics.zip
+   ```
+2. Activate your environment and check:
+   ```bash
+   conda activate /home/tbommawa/conda_envs/bruise   # or any env with torch, pandas, pillow
+   cd /home/data/bruise/BRUISE_DATA
+   python -m bruise_data check
+   python -m bruise_data summary --datasets nij fenwick mimics
+   python examples/quick_start.py
+   ```
+3. Use it from a training script anywhere:
+   ```bash
+   export BRUISE_DATA_ROOT=/home/data/bruise/BRUISE_DATA
+   export PYTHONPATH=$BRUISE_DATA_ROOT:$PYTHONPATH
+   python my_train.py
+   ```
+   or in Python: `get_dataloaders(..., root="/home/data/bruise/BRUISE_DATA")`.
+4. Example training run:
+   ```bash
+   python examples/train_segmentation.py --datasets nij fenwick mimics --epochs 2 --image-size 512 --batch-size 8
+   ```
+
+Use `num_workers=8` or more on the server. Decoding the large NIJ JPEGs is the slowest step.
+
+The ALS datasets (`nij_als`, `fenwick_als`) are not in these zips. They are built directly on the server: see [11.2](#112-adding-the-als-data-nij_als-fenwick_als).
+
+---
+
+## 11. Rebuilding or adding data
+
+`tools/build_bruise_data.py` rebuilds every folder and manifest from the original sources in `C:/BRUISE_ORGANIZED/KIYARASH_DATASET`. It is safe to re-run: existing links are kept and manifests are rewritten.
+
+```bash
+python tools/build_bruise_data.py                 # everything
+python tools/build_bruise_data.py --only mimics   # one part
+python -m bruise_data check                       # always re-check afterwards
+```
+
+- **Change a mimic class mapping:** edit `CLASS_MAP` at the top of the builder, re-run `--only mimics`.
+- **Add Roboflow "Basic Wound Classify"** (needs a Roboflow API key): export it in *folder* format into `BRUISE_LOOKALIKE/downloads/basic_wound_classify/` (`<split>/<class>/*.jpg`), then re-run `--only mimics`. Abrasion, burn and laceration go to mimics, bruise to `extra_bruise`, and duplicates of existing images are removed automatically.
+- **Splits are fixed.** NIJ/Fenwick folds come from the original subject split. Mimic splits come from the look-alike package. New web images get a split from a hash of their near-duplicate group, so re-running never moves an image to another split.
+
+### 11.1 Adding a new dataset (plug-and-play)
+
+The loader does not hard-code any dataset. **Every sub-folder of `BRUISE_DATA/` that holds a `manifest.csv` is a dataset**, and the loader treats it according to what its manifest contains. To add one:
+
+1. Put the files in `BRUISE_DATA/<name>/`. Any layout works; `train|val|test/images|masks/` matches the other datasets.
+2. Write `BRUISE_DATA/<name>/manifest.csv`.
+3. Run `python -m bruise_data datasets` (it should appear in the list) and `python -m bruise_data check` (everything should PASS).
+4. Use it: `get_dataloaders(["nij", "fenwick", "<name>"])`. No code changes are needed, including in the SegFormer script (`--datasets nij fenwick <name>`).
+
+**Required columns** (only these five): `image_id`, `split` (`train`/`val`/`test`), `class_name`, `is_bruise`, `image_path` (relative to `BRUISE_DATA/`). Every other column in [section 5](#5-manifestcsv-columns) is optional; missing ones get defaults:
+
+| Column | Default |
+|---|---|
+| `mask_path` | `""` (no mask) |
+| `annotator` | `none` |
+| `is_primary_annotator` | `True` |
+| `fold` | `-1` |
+| `subject_id` | `""` |
+| `source_dataset` | the folder name |
+| skin-tone columns | NaN |
+
+**How the manifest decides the behaviour** (`dataset_info("<name>")` shows the result):
+
+| The manifest has… | It is treated like | What the loader does |
+|---|---|---|
+| bruise rows with real annotator names (one flagged `is_primary_annotator=True`) | `nij`, `fenwick` | Trains on the chosen annotator's masks (`annotator=`, default the primary). If train/val rows have `fold` 0–4, val is re-cut with `val_fold`; otherwise the manifest's split is kept. Test uses rows with `annotator == "majority"` if present, otherwise the primary (with a warning). |
+| bruise rows, no annotators | `extra_bruise` | Uses the rows as listed. Segmentation needs a `mask_path`, otherwise it is classification only. |
+| no bruise rows (`is_bruise` all False) | `mimics` | Each image is a negative with an empty mask. `mimic_classes`, `mimic_sources`, `exclude_mimic_sources`, `merge` and `min_class_images` apply to all negative datasets together. |
+
+**Rules that keep it leak-free:**
+- For annotated bruise datasets, give every row a `subject_id`, and keep each subject in one fold and out of test. `check` verifies this.
+- For negatives, set `near_dup_group` (and `subject_id` if known) so that duplicates stay in one split.
+- Use `class_name="bruise"` for bruise images, and the taxonomy names for mimics so that classes merge across sources.
+
+Minimal example: a new negative set with only the required columns.
+```python
+import pandas as pd
+rows = [dict(image_id="newneg__001", split="train", class_name="purpura", is_bruise=False,
+             image_path="newneg/purpura/001.jpg")]          # ... one dict per image
+pd.DataFrame(rows).to_csv("BRUISE_DATA/newneg/manifest.csv", index=False)
+```
+An annotated bruise set adds `mask_path`, `annotator`, `is_primary_annotator`, `fold` and `subject_id`. It has one row per image per annotator, plus `annotator="majority"` rows on test if you have a consensus mask. For a dataset that is rebuilt from raw sources, add a `build_<name>()` function to `tools/build_bruise_data.py` (copy `build_fenwick`), so the manifest can be regenerated.
+
+### 11.2 Adding the ALS data (`nij_als`, `fenwick_als`)
+
+The ALS photos are downloaded from Labelbox on the Hopper cluster and built into BRUISE_DATA on mlidl. They are never on the laptop, which has too little space. Full steps with every command are in `tools/labelbox_als/README.md`. In short:
+
+```bash
+# Hopper (/scratch/tbommawa/als_download, after the one-time setup in that README)
+sbatch --export=ALL,STUDY=nij     run_als_download.slurm
+sbatch --export=ALL,STUDY=fenwick run_als_download.slurm
+# mlidl: pull the finished download, then build and check
+mkdir -p /home/data/bruise/als_download && cd /home/data/bruise/als_download
+ssh tbommawa@hopper.orc.gmu.edu 'cd /scratch/tbommawa/als_download && tar cf - labelbox_*_als/train_*_als_minus_test_subjects labelbox_*_als/fixed_consensus_test_als labelbox_*_als/parsed_tables' | tar xf -
+cd /home/data/bruise/BRUISE_DATA
+python tools/build_bruise_data.py --only nij_als fenwick_als --als-src /home/data/bruise/als_download
+python -m bruise_data check
+```
+
+The builder needs `nij/` and `fenwick/` in the same BRUISE_DATA: it reads the subject folds, test subjects and skin tone from them. It does not rebuild them.
+
+---
+
+## 12. Licences and data rules
+
+- **NIJ and Fenwick** are private study data. Never upload them to GitHub, a public bucket or a model hub.
+- **Mimics** come from open datasets with different licences (`licence` and `commercial_use` columns). Non-commercial: DermNet, Fitzpatrick17k, SCIN, arsenic set. Commercial use allowed: Skin Burn (CC0), the Roboflow sets and MEMI-DS (CC BY 4.0), Clinical Wound (Apache-2.0). The Kaggle wound set (`wound_yasinpratomo`) has no stated licence. Use `commercial_only=True` to restrict.
+- **Citations:** cite the source datasets in papers. Their URLs are in `BRUISE_LOOKALIKE/bruise_lookalike_negatives/sources.csv`.
+
+---
+
+## 13. Troubleshooting
+
+| Problem | Fix |
+|---|---|
+| `FileNotFoundError: .../manifest.csv` | `root` is wrong. Pass `root="/path/to/BRUISE_DATA"` or set `BRUISE_DATA_ROOT` |
+| `ModuleNotFoundError: bruise_data` | Run from inside `BRUISE_DATA/` or add it to `PYTHONPATH` / `sys.path` |
+| Windows: `RuntimeError ... freeze_support` or the script restarts | Put your code under `if __name__ == "__main__":`, or use `num_workers=0` |
+| Training is slow | Raise `num_workers`; keep the built-in transform (it decodes big JPEGs at reduced scale); use a smaller `image_size` |
+| `'extra_bruise' has no masks` | Use `task="classification"`, or drop `extra_bruise` (same for any bruise dataset without `mask_path`) |
+| `unknown dataset(s)` | The name must be a folder of `BRUISE_DATA/` with a `manifest.csv`; `python -m bruise_data datasets` lists them |
+| `missing required column(s)` | A new manifest needs `image_id, split, class_name, is_bruise, image_path` |
+| `annotators [...] but none has is_primary_annotator=True` | Flag exactly one annotator's train/val rows as primary |
+| Warning "mimic classes with < 50 images left out" | Expected for small classes. Name them in `mimic_classes`, or set `min_class_images=0` |
+| `unknown mimic class` | Run `python -m bruise_data classes` to see valid names |
+| A batch has NaN `skin_L` | Normal for mimics and the 0.3% of NIJ images without a colorimeter reading |