dslighting 1.3.9__py3-none-any.whl → 1.7.1__py3-none-any.whl

dslighting/registry/rsna-breast-cancer-detection/description.md

@@ -0,0 +1,190 @@
+# Overview
+
+## Overview
+
+### Description
+
+#### Goal of the Competition
+
+The goal of this competition is to identify breast cancer. You'll train your model with screening mammograms obtained from regular screening.
+
+Your work improving the automation of detection in screening mammography may enable radiologists to be more accurate and efficient, improving the quality and safety of patient care. It could also help reduce costs and unnecessary medical procedures.
+
+#### Context
+
+According to the WHO, breast cancer is the most commonly occurring cancer worldwide. In 2020 alone, there were 2.3 million new breast cancer diagnoses and 685,000 deaths. Yet breast cancer mortality in high-income countries has dropped by 40% since the 1980s when health authorities implemented regular mammography screening in age groups considered at risk. Early detection and treatment are critical to reducing cancer fatalities, and your machine learning skills could help streamline the process radiologists use to evaluate screening mammograms.
+
+Currently, early detection of breast cancer requires the expertise of highly-trained human observers, making screening mammography programs expensive to conduct. A looming shortage of radiologists in several countries will likely worsen this problem. Mammography screening also leads to a high incidence of false positive results. This can result in unnecessary anxiety, inconvenient follow-up care, extra imaging tests, and sometimes a need for tissue sampling (often a needle biopsy).
+
+The competition host, the Radiological Society of North America (RSNA) is a non-profit organization that represents 31 radiologic subspecialties from 145 countries around the world. RSNA promotes excellence in patient care and health care delivery through education, research, and technological innovation.
+
+Your efforts in this competition could help extend the benefits of early detection to a broader population. Greater access could further reduce breast cancer mortality worldwide.
+
+> This is a Code Competition. Refer to [Code Requirements](https://www.kaggle.com/c/rsna-breast-cancer-detection/overview/code-requirements) for details.
+
+### Evaluation
+
+Submissions are evaluated using the [probabilistic F1 score](https://aclanthology.org/2020.eval4nlp-1.9.pdf) (pF1). This extension of the traditional F score accepts probabilities instead of binary classifications. You can find [a Python implementation here](https://www.kaggle.com/code/sohier/probabilistic-f-score).
+
+With pX as the probabilistic version of X:
+
+$$
+pF_1 = 2 \frac{pPrecision \cdot pRecall}{pPrecision + pRecall}
+$$
+
+where:
+
+$$
+pPrecision = \frac{pTP}{pTP + pFP}
+$$
+
+$$
+pRecall = \frac{pTP}{TP + FN}
+$$
+
+#### Submission Format
+
+For each `prediction_id`, you should predict the likelihood of cancer in the corresponding `cancer` column. The submission file should have the following format:
+
+prediction_id,cancer\
+0-L,0\
+0-R,0.5\
+1-L,1\
+...
+
+### Timeline
+
+- **November 28, 2022** - Start Date.
+- **February 20, 2023** - Entry Deadline. You must accept the competition rules before this date in order to compete.
+- **February 20, 2023** - Team Merger Deadline. This is the last day participants may join or merge teams.
+- **February 27, 2023** - Final Submission Deadline.
+- **April 14, 2023** - Winners' Requirements Deadline. This is the deadline for winners to submit to the host/Kaggle their training code, video, method description.
+
+All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
+
+### Prizes
+
+- 1st Place - \$ 10,000
+- 2nd Place - \$ 8,000
+- 3rd Place - \$ 7,000
+- 4th - 8th Place(s) - \$ 5,000
+
+Because this competition is being hosted in coordination with the Radiological Society of North America (RSNA®) Annual Meeting, winners will be invited and strongly encouraged to attend the conference with waived fees, contingent on review of solution and fulfillment of winners' obligations.
+
+Note that, per the [competition rules](https://www.kaggle.com/c/rsna-2022-cervical-spine-fracture-detection/rules), in addition to the standard Kaggle Winners' Obligations (open-source licensing requirements, [solution packaging/delivery](https://www.kaggle.com/WinningModelDocumentationGuidelines), presentation to host), the host team also asks that you:
+
+(i) create a short video presenting your approach and solution, and
+
+(ii) publish a link to your open sourced code on the competition forum
+
+(iii) (strongly suggested) make some version of your model publicly available for more hands-on testing purposes only. As an example of a hosted algorithm, please see <http://demos.md.ai/#/bone-age>.
+
+### Code Requirements
+
+![Kerneler-white-desc2_transparent.png](https://storage.googleapis.com/kaggle-media/competitions/general/Kerneler-white-desc2_transparent.png)
+
+### This is a Code Competition
+
+Submissions to this competition must be made through Notebooks. In order for the "Submit" button to be active after a commit, the following conditions must be met:
+
+- CPU Notebook `<= 9` hours run-time
+- GPU Notebook `<= 9` hours run-time
+- Internet access disabled
+- Freely & publicly available external data is allowed, including pre-trained models
+- Submission file must be named `submission.csv`
+
+Please see the [Code Competition FAQ](https://www.kaggle.com/docs/competitions#notebooks-only-FAQ) for more information on how to submit. And review the [code debugging doc](https://www.kaggle.com/code-competition-debugging) if you are encountering submission errors.
+
+### Acknowledgements
+
+#### Screening Mammography Breast Cancer Detection Challenge Acknowledgements
+
+![RSNA Logo](https://storage.googleapis.com/kaggle-media/competitions/RSNA-2021/RSNA%20Logo.svg)         
+
+RSNA would like to thank the following individuals and organizations whose contributions made possible the 2023 RSNA Screening Mammography Breast Cancer Detection AI Challenge:
+
+#### Challenge Organizing Team
+
+- Katherine P. Andriole, PhD - MGH & BWH Center for Clinical Data Science
+- Robyn Ball, PhD - The Jackson Laboratory
+- Yan Chen, PhD - University of Nottingham
+- Helen Frazer, MBBS - BreastScreen Victoria
+- Tatiana Kelil, MD - University of California - San Francisco
+- Felipe Kitamura, MD - Universidade Federal de São Paulo
+- Jackson Kwok - BreastScreen Victoria
+- Matthew Lungren, MD - Stanford University
+- Ritse Mann, MD, PhD - Radboud University Medical Center
+- John Mongan, MD, PhD - University of California - San Francisco
+- Linda Moy, MD - New York University Grossman School of Medicine
+- George Partridge - University of Nottingham
+- Hari Trivedi, MD - Emory University
+- Xin Wang, PhD - Radboud University Medical Center
+- Luyan Yao, PhD - University of Nottingham
+- Tianyu Zhang - Radboud University Medical Center
+
+#### Data Contributors
+
+Two research institutions provided large volumes of de-identified mammography studies and related clinical information that were assembled to create the challenge dataset.
+
+![BreastScreen Victoria logo](https://storage.googleapis.com/kaggle-media/competitions/RSNA-2021/BreastScreen%20Victoria%20logo_full-colour_Primary_Positive_CMYK%5B7%5D.jpg)         
+
+*BreastScreen Victoria, Australia*
+
+-   Helen Frazer, MBBS FRANZCR M.Epi
+-   Michael S. Elliott, BEng (Hons)
+
+![Hiti Lab logo](https://storage.googleapis.com/kaggle-media/competitions/RSNA-2021/emory_hitti.png)         
+
+*Emory University | Health Innovation and Translational Informatics (HITI) Lab - Hari Trivedi, MD*
+
+-   Judy Gichoya, MD
+-   Minjae Woo, PhD
+-   Imon Banerjee, PhD
+-   Jason Jeong, PhD
+-   Beatrice Brown-Mulry, BS
+
+![MD.ai logo](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F603584%2Fb54f8ef1c415f02a1376f7e880257814%2Fmdai-logo.png?generation=1568657775088597&alt=media)
+
+Special thanks to *MD.ai* for providing tooling for the data annotation process.
+
+![Kaggle logo](https://www.kaggle.com/static/images/logos/kaggle-logo-transparent-300.png)         
+
+Special thanks to *Kaggle* for providing the competition platform, as well as design and technical support.
+
+### Citation
+
+Chris Carr, FelipeKitamura, MD, PhD, George Partridge, inversion, Jayashree Kalpathy-Cramer, John Mongan, Katherine Andriole, Lavender, Maryam Vazirabad, Michelle Riopel, Robyn Ball, Sohier Dane, Yan Chen. (2022). RSNA Screening Mammography Breast Cancer Detection. Kaggle. https://kaggle.com/competitions/rsna-breast-cancer-detection
+
+# Data
+
+## Dataset Description
+
+**Note: The dataset for this challenge contains radiographic breast images of female subjects.**
+
+The goal of this competition is to identify cases of breast cancer in mammograms from screening exams. It is important to identify cases of cancer for obvious reasons, but false positives also have downsides for patients. As millions of women get mammograms each year, a useful machine learning tool could help a great many people.
+
+This competition uses a hidden test. When your submitted notebook is scored the actual test data (including a full length sample submission) will be made available to your notebook.
+
+### Files
+
+**[train/test]_images/[patient_id]/[image_id].dcm** The mammograms, in dicom format. You can expect roughly 8,000 patients in the hidden test set. There are usually but not always 4 images per patient. Note that many of the images use the jpeg 2000 format which may you may need special libraries to load.
+
+**sample_submission.csv** A valid sample submission. Only the first few rows are available for download.
+
+**[train/test].csv** Metadata for each patient and image. Only the first few rows of the test set are available for download.
+
+- `site_id` - ID code for the source hospital.
+- `patient_id` - ID code for the patient.
+- `image_id` - ID code for the image.
+- `laterality` - Whether the image is of the left or right breast.
+- `view` - The orientation of the image. The default for a screening exam is to capture two views per breast.
+- `age` - The patient's age in years.
+- `implant` - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.
+- `density` - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult. Only provided for train.
+- `machine_id` - An ID code for the imaging device.
+- `cancer` - Whether or not the breast was positive for malignant cancer. The target value. Only provided for train.
+- `biopsy` - Whether or not a follow-up biopsy was performed on the breast. Only provided for train.
+- `invasive` - If the breast is positive for cancer, whether or not the cancer proved to be invasive. Only provided for train.
+- `BIRADS` - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. Only provided for train.
+- `prediction_id` - The ID for the matching submission row. Multiple images will share the same prediction ID. Test only.
+- `difficult_negative_case` - True if the case was unusually difficult. Only provided for train.

dslighting/registry/rsna-breast-cancer-detection/description_obfuscated.md

@@ -0,0 +1,58 @@
+# Task
+
+Detect breast cancer in mammograms.
+
+# Metric
+
+[Probabilistic F1 score](https://aclanthology.org/2020.eval4nlp-1.9.pdf) (pF1). This extension of the traditional F score accepts probabilities instead of binary classifications. 
+
+With pX as the probabilistic version of X:
+
+$$
+pF_1 = 2 \frac{pPrecision \cdot pRecall}{pPrecision + pRecall}
+$$
+
+where:
+
+$$
+pPrecision = \frac{pTP}{pTP + pFP}
+$$
+
+$$
+pRecall = \frac{pTP}{TP + FN}
+$$
+
+# Submission Format
+
+For each `prediction_id`, you should predict the likelihood of cancer in the corresponding `cancer` column. The submission file should have the following format:
+
+```
+prediction_id,cancer
+0-L,0
+0-R,0.5
+0-R,0.5
+1-L,1
+...
+# Dataset
+
+**[train/test]_images/[patient_id]/[image_id].dcm** The mammograms, in dicom format. You can expect roughly 8,000 patients in the hidden test set. There are usually but not always 4 images per patient. Note that many of the images use the jpeg 2000 format which may you may need special libraries to load.
+
+**sample_submission.csv** A valid sample submission.
+
+**[train/test].csv** Metadata for each patient and image. Only the first few rows of the test set are available for download.
+
+- `site_id` - ID code for the source hospital.
+- `patient_id` - ID code for the patient.
+- `image_id` - ID code for the image.
+- `laterality` - Whether the image is of the left or right breast.
+- `view` - The orientation of the image. The default for a screening exam is to capture two views per breast.
+- `age` - The patient's age in years.
+- `implant` - Whether or not the patient had breast implants. Site 1 only provides breast implant information at the patient level, not at the breast level.
+- `density` - A rating for how dense the breast tissue is, with A being the least dense and D being the most dense. Extremely dense tissue can make diagnosis more difficult. Only provided for train.
+- `machine_id` - An ID code for the imaging device.
+- `cancer` - Whether or not the breast was positive for malignant cancer. The target value. Only provided for train.
+- `biopsy` - Whether or not a follow-up biopsy was performed on the breast. Only provided for train.
+- `invasive` - If the breast is positive for cancer, whether or not the cancer proved to be invasive. Only provided for train.
+- `BIRADS` - 0 if the breast required follow-up, 1 if the breast was rated as negative for cancer, and 2 if the breast was rated as normal. Only provided for train.
+- `prediction_id` - The ID for the matching submission row. Multiple images will share the same prediction ID. Test only.
+- `difficult_negative_case` - True if the case was unusually difficult. Only provided for train.

dslighting/registry/rsna-breast-cancer-detection/grade.py

@@ -0,0 +1,65 @@
+import pandas as pd
+
+from mlebench.grade_helpers import InvalidSubmissionError
+
+
+def pfbeta(labels, predictions, beta):
+    """
+    Adapted from https://www.kaggle.com/code/sohier/probabilistic-f-score
+    adaptations: code formatting, this docstring.
+    """
+    y_true_count = 0
+    ctp = 0
+    cfp = 0
+
+    for idx in range(len(labels)):
+        prediction = min(max(predictions[idx], 0), 1)
+        if labels[idx]:
+            y_true_count += 1
+            ctp += prediction
+        else:
+            cfp += prediction
+
+    beta_squared = beta * beta
+    c_precision = ctp / (ctp + cfp)
+    c_recall = ctp / y_true_count
+    if c_precision > 0 and c_recall > 0:
+        result = (
+            (1 + beta_squared) * (c_precision * c_recall) / (beta_squared * c_precision + c_recall)
+        )
+        return result
+    else:
+        return 0
+
+
+def prepare_for_metric(submission: pd.DataFrame, answers: pd.DataFrame) -> dict:
+    id_col = "prediction_id"
+    target_col = "cancer"
+
+    assert id_col in answers.columns, f"{id_col} not in answers"
+    assert target_col in answers.columns, f"{target_col} not in answers"
+
+    if id_col not in submission.columns:
+        raise InvalidSubmissionError(f"{id_col} not in submission")
+    if target_col not in submission.columns:
+        raise InvalidSubmissionError(f"{target_col} not in submission")
+    if len(submission) != len(answers):
+        raise InvalidSubmissionError("submission and answers have different lengths")
+
+    submission = submission.sort_values(by=id_col)
+    answers = answers.sort_values(by=id_col)
+
+    if (submission[id_col].values != answers[id_col].values).any():
+        raise InvalidSubmissionError(f"{id_col} not aligned")
+
+    labels = answers[target_col].to_numpy()
+    predictions = submission[target_col].to_numpy()
+
+    return {"labels": labels, "predictions": predictions}
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    pf1_inputs = prepare_for_metric(submission, answers)
+    # F1 score, so beta=1; https://www.kaggle.com/competitions/rsna-breast-cancer-detection/discussion/370123
+    score = pfbeta(**pf1_inputs, beta=1)
+    return score

dslighting/registry/rsna-breast-cancer-detection/prepare.py

@@ -0,0 +1,141 @@
+import shutil
+from pathlib import Path
+
+from sklearn.model_selection import train_test_split
+from tqdm.auto import tqdm
+
+from mlebench.utils import read_csv
+
+# columns in train.csv
+train_columns = [
+    "site_id",
+    "patient_id",
+    "image_id",
+    "laterality",
+    "view",
+    "age",
+    "cancer",
+    "biopsy",
+    "invasive",
+    "BIRADS",
+    "implant",
+    "density",
+    "machine_id",
+    "difficult_negative_case",
+]
+# columns in test.csv
+test_columns = [
+    "site_id",
+    "patient_id",
+    "image_id",
+    "laterality",
+    "view",
+    "age",
+    "implant",
+    "machine_id",
+    "prediction_id",
+]
+
+# columns in answers/submission
+submission_columns = ["prediction_id", "cancer"]
+
+
+DEV = False
+
+
+def prepare(raw: Path, public: Path, private: Path):
+
+    old_train = read_csv(raw / "train.csv")
+    # work on 5k samples for now, instead of 54k
+    if DEV:
+        old_train = old_train.sample(5000, random_state=42)
+
+    # "You can expect roughly 8,000 patients" in the test set
+    # so, split on patients. There are 11913 patients in train set
+    patient_ids = old_train["patient_id"].unique()
+    # Original ratio is 8000/ (8000 + 11913) ~ 0.4
+    # We use 0.1 to avoid taking too many samples out of train
+    train_patients, test_patients = train_test_split(patient_ids, test_size=0.1, random_state=42)
+    train_patients = set(train_patients)
+    test_patients = set(test_patients)
+
+    new_train = old_train[old_train["patient_id"].isin(train_patients)].copy()
+    # dont index the `prediction_id` (last col) since its not in train and we need to build it
+    # index also the `cancer` column, which we'll drop later for the without_labels version
+    answers = old_train[old_train["patient_id"].isin(test_patients)][
+        test_columns[:-1] + ["cancer"]
+    ].copy()
+
+    new_train.to_csv(public / "train.csv", index=False)
+
+    answers["prediction_id"] = answers["patient_id"].astype(str) + "_" + answers["laterality"]
+    new_test_without_labels = answers.drop(columns=["cancer"])
+    new_test_without_labels.to_csv(public / "test.csv", index=False)
+
+    # merge multiple prediction_ids for the same patient into one for sample_submission and private test
+    answers = answers[submission_columns].copy()
+    # just take the first label for each prediction id -- the rest will be identical duplicates
+    answers = answers.groupby("prediction_id").first().reset_index()
+    answers.to_csv(private / "answers.csv", index=False)
+
+    sample_submission = answers.copy()
+    sample_submission["cancer"] = new_train.cancer.mean()  # mean cancer rate in train set
+    sample_submission.to_csv(public / "sample_submission.csv", index=False)
+
+    assert len(sample_submission) == len(
+        answers
+    ), "sample_submission and test.csv should have the same number of rows"
+    assert len(new_test_without_labels) + len(new_train) == len(
+        old_train
+    ), "The sum of the rows in new_test_without_labels and new_train should be equal to the number of rows in old_train"
+    # because of the merging
+    assert len(answers) != len(
+        new_test_without_labels
+    ), "new_test and new_test_without_labels should have different number of rows"
+
+    assert (
+        answers.columns.tolist() == submission_columns
+    ), f"answers should have columns {submission_columns}"
+    assert (
+        sample_submission.columns.tolist() == submission_columns
+    ), f"sample_submission should have columns {submission_columns}"
+
+    assert (
+        new_train.columns.tolist() == old_train.columns.tolist()
+    ), f"new_train should have columns {old_train.columns.tolist()}, got {new_train.columns.tolist()}"
+    assert (
+        new_test_without_labels.columns.tolist() == test_columns
+    ), f"new_test_without_labels should have columns {test_columns}, got {new_test_without_labels.columns.tolist()}"
+
+    assert set(new_test_without_labels["patient_id"]).isdisjoint(
+        set(new_train["patient_id"])
+    ), "new_test_without_labels and new_train should have disjoint patient_ids"
+
+    # finally, split the images
+    (public / "train_images").mkdir(exist_ok=True)
+    for patient_id in tqdm(train_patients, total=len(train_patients)):
+        patient_id_str = str(patient_id)
+        patient_dir = public / "train_images" / patient_id_str
+        patient_dir.mkdir(exist_ok=True)
+        image_ids = new_train[new_train["patient_id"] == patient_id]["image_id"].to_list()
+        for image_id in image_ids:
+            shutil.copy(raw / "train_images" / patient_id_str / f"{image_id}.dcm", patient_dir)
+
+    (public / "test_images").mkdir(exist_ok=True)
+    for patient_id in tqdm(test_patients, total=len(test_patients)):
+        patient_id_str = str(patient_id)
+        patient_dir = public / "test_images" / patient_id_str
+        patient_dir.mkdir(exist_ok=True)
+        image_ids = new_test_without_labels[new_test_without_labels["patient_id"] == patient_id][
+            "image_id"
+        ].to_list()
+        for image_id in image_ids:
+            shutil.copy(raw / "train_images" / patient_id_str / f"{image_id}.dcm", patient_dir)
+
+    # final checks
+    assert len(list((public / "train_images").rglob("*.dcm"))) == len(
+        new_train
+    ), "Number of images in train_images should be equal to the number of rows in new_train"
+    assert len(list((public / "test_images").rglob("*.dcm"))) == len(
+        new_test_without_labels
+    ), "Number of images in test_images should be equal to the number of rows in new_test_without_labels"

dslighting/registry/rsna-breast-cancer-detection/prepare_val.py

@@ -0,0 +1,201 @@
+import shutil
+from pathlib import Path
+
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from tqdm.auto import tqdm
+
+from mlebench.utils import read_csv
+
+# columns in train.csv
+train_columns = [
+    "site_id",
+    "patient_id",
+    "image_id",
+    "laterality",
+    "view",
+    "age",
+    "cancer",
+    "biopsy",
+    "invasive",
+    "BIRADS",
+    "implant",
+    "density",
+    "machine_id",
+    "difficult_negative_case",
+]
+# columns in test.csv
+test_columns = [
+    "site_id",
+    "patient_id",
+    "image_id",
+    "laterality",
+    "view",
+    "age",
+    "implant",
+    "machine_id",
+    "prediction_id",
+]
+
+# columns in answers/submission
+submission_columns = ["prediction_id", "cancer"]
+
+
+DEV = False
+
+
+def _process_split(
+    source_df: pd.DataFrame,
+    train_pids: set,
+    test_pids: set,
+    public_path: Path,
+    private_path: Path,
+    raw_images_path: Path,
+):
+    """
+    Processes a data split, creating all required files and directories.
+
+    Args:
+        source_df: The DataFrame to split (e.g., the full dataset or a training subset).
+        train_pids: A set of patient IDs for the training set.
+        test_pids: A set of patient IDs for the test set.
+        public_path: The public output directory.
+        private_path: The private output directory.
+        raw_images_path: Path to the original raw images.
+    """
+    public_path.mkdir(exist_ok=True, parents=True)
+    private_path.mkdir(exist_ok=True, parents=True)
+
+    # Create train and test dataframes from the source data
+    new_train = source_df[source_df["patient_id"].isin(train_pids)].copy()
+    # dont index the `prediction_id` (last col) since its not in train and we need to build it
+    # index also the `cancer` column, which we'll drop later for the without_labels version
+    answers = source_df[source_df["patient_id"].isin(test_pids)][
+        test_columns[:-1] + ["cancer"]
+    ].copy()
+
+    new_train.to_csv(public_path / "train.csv", index=False)
+
+    answers["prediction_id"] = answers["patient_id"].astype(str) + "_" + answers["laterality"]
+    new_test_without_labels = answers.drop(columns=["cancer"])
+    new_test_without_labels.to_csv(public_path / "test.csv", index=False)
+
+    # merge multiple prediction_ids for the same patient into one for sample_submission and private test
+    answers = answers[submission_columns].copy()
+    # just take the first label for each prediction id -- the rest will be identical duplicates
+    answers = answers.groupby("prediction_id").first().reset_index()
+    answers.to_csv(private_path / "answers.csv", index=False)
+
+    sample_submission = answers.copy()
+    sample_submission["cancer"] = new_train.cancer.mean()  # mean cancer rate in train set
+    sample_submission.to_csv(public_path / "sample_submission.csv", index=False)
+
+    assert len(sample_submission) == len(
+        answers
+    ), "sample_submission and test.csv should have the same number of rows"
+    assert len(new_test_without_labels) + len(new_train) == len(
+        source_df
+    ), "The sum of the rows in new_test_without_labels and new_train should be equal to the number of rows in the source_df"
+    # because of the merging
+    assert len(answers) != len(
+        new_test_without_labels
+    ), "new_test and new_test_without_labels should have different number of rows"
+
+    assert (
+        answers.columns.tolist() == submission_columns
+    ), f"answers should have columns {submission_columns}"
+    assert (
+        sample_submission.columns.tolist() == submission_columns
+    ), f"sample_submission should have columns {submission_columns}"
+
+    assert (
+        new_train.columns.tolist() == source_df.columns.tolist()
+    ), f"new_train should have columns {source_df.columns.tolist()}, got {new_train.columns.tolist()}"
+    assert (
+        new_test_without_labels.columns.tolist() == test_columns
+    ), f"new_test_without_labels should have columns {test_columns}, got {new_test_without_labels.columns.tolist()}"
+
+    assert set(new_test_without_labels["patient_id"]).isdisjoint(
+        set(new_train["patient_id"])
+    ), "new_test_without_labels and new_train should have disjoint patient_ids"
+
+    # finally, split the images
+    (public_path / "train_images").mkdir(exist_ok=True)
+    for patient_id in tqdm(train_pids, desc=f"Copying train images to {public_path.name}"):
+        patient_id_str = str(patient_id)
+        patient_dir = public_path / "train_images" / patient_id_str
+        patient_dir.mkdir(exist_ok=True)
+        image_ids = new_train[new_train["patient_id"] == patient_id]["image_id"].to_list()
+        for image_id in image_ids:
+            shutil.copy(raw_images_path / patient_id_str / f"{image_id}.dcm", patient_dir)
+
+    (public_path / "test_images").mkdir(exist_ok=True)
+    for patient_id in tqdm(test_pids, desc=f"Copying test images to {public_path.name}"):
+        patient_id_str = str(patient_id)
+        patient_dir = public_path / "test_images" / patient_id_str
+        patient_dir.mkdir(exist_ok=True)
+        image_ids = new_test_without_labels[new_test_without_labels["patient_id"] == patient_id][
+            "image_id"
+        ].to_list()
+        for image_id in image_ids:
+            shutil.copy(raw_images_path / patient_id_str / f"{image_id}.dcm", patient_dir)
+
+    # final checks
+    assert len(list((public_path / "train_images").rglob("*.dcm"))) == len(
+        new_train
+    ), "Number of images in train_images should be equal to the number of rows in new_train"
+    assert len(list((public_path / "test_images").rglob("*.dcm"))) == len(
+        new_test_without_labels
+    ), "Number of images in test_images should be equal to the number of rows in new_test_without_labels"
+
+
+def prepare(raw: Path, public: Path, private: Path):
+
+    old_train = read_csv(raw / "train.csv")
+    # work on 5k samples for now, instead of 54k
+    if DEV:
+        old_train = old_train.sample(5000, random_state=42)
+
+    # "You can expect roughly 8,000 patients" in the test set
+    # so, split on patients. There are 11913 patients in train set
+    # Original ratio is 8000/ (8000 + 11913) ~ 0.4
+    # We use 0.1 to avoid taking too many samples out of train
+    all_patient_ids = old_train["patient_id"].unique()
+    train_patients_orig, test_patients_orig = train_test_split(
+        all_patient_ids, test_size=0.1, random_state=42
+    )
+
+    # --- 1. Create the original public/private split ---
+    # This call generates the primary competition data. Its outputs must not be changed.
+    _process_split(
+        source_df=old_train,
+        train_pids=set(train_patients_orig),
+        test_pids=set(test_patients_orig),
+        public_path=public,
+        private_path=private,
+        raw_images_path=raw / "train_images",
+    )
+
+    # --- 2. Create the new validation split ---
+    # Define new parallel directories for the validation set.
+    public_val = public.parent / "public_val"
+    private_val = private.parent / "private_val"
+
+    # Split the *original training patients* again to create a new train/validation set.
+    # We use the same test_size and random_state to replicate the splitting methodology.
+    train_val_patients, test_val_patients = train_test_split(
+        train_patients_orig, test_size=0.1, random_state=42
+    )
+
+    # The source data for this second split is the original training data.
+    original_train_df = old_train[old_train["patient_id"].isin(train_patients_orig)].copy()
+
+    # This call generates the new validation data in public_val/private_val.
+    _process_split(
+        source_df=original_train_df,
+        train_pids=set(train_val_patients),
+        test_pids=set(test_val_patients),
+        public_path=public_val,
+        private_path=private_val,
+        raw_images_path=raw / "train_images",
+    )

dslighting/registry/rsna-miccai-brain-tumor-radiogenomic-classification/checksums.yaml

@@ -0,0 +1,6 @@
+private:
+  test.csv: 9069c7813ae83cd1dc962efed1875074
+public:
+  sample_submission.csv: 893556127819a5b9428102dbbc0e0ed3
+  train_labels.csv: 891e1c30bfe288e25c931759a746f254
+zip: 148f363879c193b0c510b840832833e3