dslighting 1.3.9__py3-none-any.whl → 1.7.1__py3-none-any.whl

dslighting/registry/rsna-miccai-brain-tumor-radiogenomic-classification/config.yaml

@@ -0,0 +1,32 @@
+id: rsna-miccai-brain-tumor-radiogenomic-classification
+name: RSNA-MICCAI Brain Tumor Radiogenomic Classification
+competition_type: code
+awards_medals: true
+prizes:
+  - position: 1
+    value: 6000
+  - position: 2
+    value: 5000
+  - position: 3
+    value: 4000
+  - position: 4
+    value: 3000
+  - position: 5
+    value: 3000
+  - position: 6
+    value: 3000
+  - position: 7
+    value: 3000
+  - position: 8
+    value: 3000
+description: mlebench/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/description.md
+
+dataset:
+  answers: rsna-miccai-brain-tumor-radiogenomic-classification/prepared/private/test.csv
+  sample_submission: rsna-miccai-brain-tumor-radiogenomic-classification/prepared/public/sample_submission.csv
+
+grader:
+  name: auc-roc
+  grade_fn: mlebench.competitions.rsna-miccai-brain-tumor-radiogenomic-classification.grade:grade
+
+preparer: mlebench.competitions.rsna-miccai-brain-tumor-radiogenomic-classification.prepare:prepare

dslighting/registry/rsna-miccai-brain-tumor-radiogenomic-classification/description.md

@@ -0,0 +1,202 @@
+# RSNA-MICCAI Brain Tumor Radiogenomic Classification
+
+## Description
+
+A malignant tumor in the brain is a life-threatening condition. Known as glioblastoma, it's both the most common form of brain cancer in adults and the one with the worst prognosis, with median survival being less than a year. The presence of a specific genetic sequence in the tumor known as MGMT promoter methylation has been shown to be a favorable prognostic factor and a strong predictor of responsiveness to chemotherapy.
+
+![](https://storage.googleapis.com/kaggle-media/competitions/RSNA-2021/image2.png)
+
+Currently, genetic analysis of cancer requires surgery to extract a tissue sample. Then it can take several weeks to determine the genetic characterization of the tumor. Depending upon the results and type of initial therapy chosen, a subsequent surgery may be necessary. If an accurate method to predict the genetics of the cancer through imaging (i.e., radiogenomics) alone could be developed, this would potentially minimize the number of surgeries and refine the type of therapy required.
+
+The Radiological Society of North America (RSNA) has teamed up with the Medical Image Computing and Computer Assisted Intervention Society (the MICCAI Society) to improve diagnosis and treatment planning for patients with glioblastoma. In this competition you will predict the genetic subtype of glioblastoma using MRI (magnetic resonance imaging) scans to train and test your model to detect for the presence of MGMT promoter methylation.
+
+If successful, you'll help brain cancer patients receive less invasive diagnoses and treatments. The introduction of new and customized treatment strategies before surgery has the potential to improve the management, survival, and prospects of patients with brain cancer.
+
+### Acknowledgments
+
+**The Radiological Society of North America (RSNA®)** is a non-profit organization that represents 31 radiologic subspecialties from 145 countries around the world. RSNA promotes excellence in patient care and health care delivery through education, research and technological innovation.
+
+RSNA provides high-quality educational resources, publishes five top peer-reviewed journals, hosts the world’s largest radiology conference and is dedicated to building the future of the profession through the RSNA Research & Education (R&E) Foundation, which has funded $66 million in grants since its inception. RSNA also supports and facilitates artificial intelligence (AI) research in medical imaging by sponsoring an ongoing series of AI challenge competitions.
+
+**The Medical Image Computing and Computer Assisted Intervention Society (the MICCAI Society)** is dedicated to the promotion, preservation and facilitation of research, education and practice in the field of medical image computing and computer assisted medical interventions including biomedical imaging and medical robotics. The Society achieves this aim through the organization and operation of annual high quality international conferences, workshops, tutorials and publications that promote and foster the exchange and dissemination of advanced knowledge, expertise and experience in the field produced by leading institutions and outstanding scientists, physicians and educators around the world.
+
+[A full set of acknowledgments can be found on this page](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/overview/acknowledgments).
+
+![](https://storage.googleapis.com/kaggle-media/competitions/RSNA-2021/sponsors.png)
+
+## Evaluation
+
+Submissions are evaluated on the [area under the ROC curve](http://en.wikipedia.org/wiki/Receiver_operating_characteristic) between the predicted probability and the observed target.
+
+### Submission File
+
+For each `BraTS21ID` in the test set, you must predict a probability for the target `MGMT_value`. The file should contain a header and have the following format:
+
+```
+BraTS21ID,MGMT_value
+00001,0.5
+00013,0.5
+00015,0.5
+etc.
+```
+
+## Timeline
+
+- **July 13, 2021** - Start Date.
+- **October 8, 2021** - Entry Deadline. You must accept the competition rules before this date in order to compete.
+- **October 8, 2021** - Team Merger Deadline. This is the last day participants may join or merge teams.
+- **October 15, 2021** - Final Submission Deadline.
+- **October 25, 2021** - Winners’ Requirements Deadline. This is the deadline for winners to submit to the host/Kaggle their training code, video, method description.
+
+All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
+
+## Prizes
+
+- 1st Place - $6,000
+- 2nd Place - $5,000
+- 3rd Place - $4,000
+- 4th - 8th Places - $3,000 each
+
+Because this competition is being hosted in coordination with the Radiological Society of North America (RSNA®) Annual Meeting, winners will be invited and strongly encouraged to attend the conference with waived fees, contingent on review of solution and fulfillment of winners' obligations.
+
+Note that, per the [competition rules](https://www.kaggle.com/c/rsna-miccai-brain-tumor-radiogenomic-classification/rules), in addition to the standard Kaggle Winners' Obligations (open-source licensing requirements, [solution packaging/delivery](https://www.kaggle.com/WinningModelDocumentationGuidelines), presentation to host), the host team also asks that you:
+
+(i) create a short video presenting your approach and solution, and
+
+(ii) publish a link to your open sourced code on the competition forum
+
+## Code Requirements
+
+![](https://storage.googleapis.com/kaggle-media/competitions/general/Kerneler-white-desc2_transparent.png)
+
+### This is a Code Competition
+
+Submissions to this competition must be made through Notebooks. In order for the "Submit" button to be active after a commit, the following conditions must be met:
+
+- CPU Notebook <= 9 hours run-time
+- GPU Notebook <= 9 hours run-time
+- Internet access disabled
+- Freely & publicly available external data is allowed, including pre-trained models
+- Submission file must be named `submission.csv`
+
+Please see the [Code Competition FAQ](https://www.kaggle.com/docs/competitions#notebooks-only-FAQ) for more information on how to submit. And review the [code debugging doc](https://www.kaggle.com/code-competition-debugging) if you are encountering submission errors.
+
+## Acknowledgments
+
+The dataset for this challenge has been collected from institutions around the world as part of a decade-long project to advance the use of AI in brain tumor diagnosis and treatment, the **Brain Tumor Segmentation (BraTS) challenge**. Running in parallel with this challenge, a challenge addressing segmentation represents the culmination of this effort.
+
+- A comprehensive description of the both tasks of the RSNA-MICCAI Brain Tumor challenge can be found at: https://www.med.upenn.edu/cbica/brats2021/
+- Participants interested in the segmentation task of this competition can find more info at: https://www.synapse.org/brats2021
+
+### Challenge Organizing Team
+
+*(in alphabetical order)*
+
+- Spyridon Bakas, PhD - University of Pennsylvania, Philadelphia, PA, USA
+- Ujjwal Baid, PhD - University of Pennsylvania, Philadelphia, PA, USA
+- Evan Calabrese, MD, PhD - University of California San Francisco, CA, USA
+- Christopher Carr - Radiological Society of North America (RSNA), Oak Brook, IL, USA
+- Errol Colak, MD - Unity Health Toronto, Canada
+- Keyvan Farahani, PhD - National Cancer Institute (NCI), National Institutes of Health (NIH), Bethesda, MD, USA
+- Adam E. Flanders, MD - Thomas Jefferson University Hospital, Philadelphia, PA, USA
+- Jayashree Kalpathy-Cramer, PhD - Massachusetts General Hospital, Boston, MA, USA
+- Felipe C Kitamura, MD, MSc, PhD - Diagnósticos da América SA (Dasa) and Universidade Federal de São Paulo, Brazil
+- Bjoern Menze, PhD - University of Zurich, Switzerland
+- John Mongan, MD, PhD - University of California - San Francisco, CA, USA
+- Luciano Prevedello, MD, MPH - The Ohio State University, Columbus, OH, USA
+- Jeffrey Rudie, MD, PhD - University of California - San Francisco, CA, USA
+- Russell Taki Shinohara, PhD - University of Pennsylvania, Philadelphia, PA, USA
+
+### Data Contributors
+
+*(in order of decreasing data contributions)*
+
+- Christos Davatzikos, PhD, & Spyridon Bakas, PhD, & Chiharu Sako, PhD - University of Pennsylvania, Philadelphia, PA, USA
+- John Mongan, MD, PhD, & Evan Calabrese, MD, PhD, & Jeff Rudie, MD, PhD, & Christopher Hess, MD, PhD, & Soonmee Cha, MD, & Javier Villanueva-Meyer, MD - University of California San Francisco, CA, USA
+- John B. Freymann & Justin S. Kirby - on behalf of The Cancer Imaging Archive (TCIA), Cancer Imaging Program, NCI, NIH, USA
+- Benedikt Wiestler, MD, & Bjoern Menze, PhD - Technical University of Munich, Germany
+- Bjoern Menze, PhD - University of Zurich, Switzerland
+- Errol Colak, MD & Priscila Crivellaro, MD - University of Toronto, Toronto, ON, Canada
+- Rivka R. Colen, MD, & Aikaterini Kotrotsou, PhD - MD Anderson Cancer Center, TX, USA
+- Daniel Marcus, PhD, & Mikhail Milchenko, PhD, & Arash Nazeri, MD - Washington University School of Medicine in St. Louis, MO, USA
+- Hassan Fathallah-Shaykh, MD, PhD - University of Alabama at Birmingham, AL, USA
+- Roland Wiest, MD - University of Bern, Switzerland
+- Andras Jakab, MD, PhD - University of Debrecen, Hungary
+- Marc-Andre Weber, MD - Heidelberg University, Germany
+- Abhishek Mahajan, MD, & Ujjwal Baid, PhD - Tata Memorial Centre, Mumbai, India, & SGGS Institute of Engineering and Technology, Nanded, India
+
+### Sponsors
+
+Prize money for challenge winners is provided by:
+
+- Intel Corporation
+- Radiological Society of North American (RSNA)
+- NeoSoma, Inc.
+
+## Citation
+
+Adam Flanders, Chris Carr, Evan Calabrese, FelipeKitamura, MD, PhD, inversion, JeffRudie, John Mongan, Julia Elliott, Luciano Prevedello, Michelle Riopel, sprint, Spyridon Bakas, Ujjwal. (2021). RSNA-MICCAI Brain Tumor Radiogenomic Classification. Kaggle. https://kaggle.com/competitions/rsna-miccai-brain-tumor-radiogenomic-classification
+
+# Data
+
+## Dataset Description
+
+The competition data is defined by three cohorts: Training, Validation (Public), and Testing (Private). The “Training” and the “Validation” cohorts are provided to the participants, whereas the “Testing” cohort is kept hidden at all times, during and after the competition.
+
+These 3 cohorts are structured as follows: Each independent case has a dedicated folder identified by a five-digit number. Within each of these “case” folders, there are four sub-folders, each of them corresponding to each of the structural multi-parametric MRI (mpMRI) scans, in DICOM format. The exact mpMRI scans included are:
+
+- Fluid Attenuated Inversion Recovery (FLAIR)
+- T1-weighted pre-contrast (T1w)
+- T1-weighted post-contrast (T1Gd)
+- T2-weighted (T2)
+
+Exact folder structure:
+
+```
+Training/Validation/Testing
+│
+└─── 00000
+│   │
+│   └─── FLAIR
+│   │   │ Image-1.dcm
+│   │   │ Image-2.dcm
+│   │   │ ...
+│   │
+│   └─── T1w
+│   │   │ Image-1.dcm
+│   │   │ Image-2.dcm
+│   │   │ ...
+│   │
+│   └─── T1wCE
+│   │   │ Image-1.dcm
+│   │   │ Image-2.dcm
+│   │   │ ...
+│   │
+│   └─── T2w
+│   │   │ Image-1.dcm
+│   │   │ Image-2.dcm
+│   │   │ .....
+│
+└─── 00001
+│   │ ...
+│
+│ ...
+│
+└─── 00002
+│   │ ...
+
+```
+
+## Files
+
+- **train/** - folder containing the training files, with each top-level folder representing a subject. **NOTE:** There are some unexpected issues with the following three cases in the training dataset, participants can exclude the cases during training: `[00109, 00123, 00709]`. We have checked and confirmed that the testing dataset is free from such issues.
+- **train_labels.csv** - file containing the target `MGMT_value` for each subject in the training data (e.g. the presence of MGMT promoter methylation)
+- **test/** - the test files, which use the same structure as `train/`; your task is to predict the `MGMT_value` for each subject in the test data. **NOTE**: the total size of the rerun test set (Public and Private) is ~5x the size of the Public test set
+- **sample_submission.csv** - a sample submission file in the correct format
+
+## Citation
+
+If you reference or use the dataset in any form, include the following citation:
+
+> U.Baid, et al., “The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification”, arXiv:2107.02314, 2021.
+>

dslighting/registry/rsna-miccai-brain-tumor-radiogenomic-classification/description_obfuscated.md

@@ -0,0 +1,26 @@
+# Task
+
+Predict the genetic subtype of glioblastoma using MRI (magnetic resonance imaging) scans to detect for the presence of MGMT promoter methylation.
+
+# Metric
+
+Area under the ROC curve between the predicted probability and the observed target.
+
+# Submission Format
+
+For each `BraTS21ID` in the test set, you must predict a probability for the target `MGMT_value`. The file should contain a header and have the following format:
+
+```
+BraTS21ID,MGMT_value
+00001,0.5
+00013,0.5
+00015,0.5
+etc.
+```
+
+# Dataset
+
+- **train/** - folder containing the training files, with each top-level folder representing a subject. **NOTE:** There are some unexpected issues with the following three cases in the training dataset, participants can exclude the cases during training: `[00109, 00123, 00709]`. We have checked and confirmed that the testing dataset is free from such issues.
+- **train_labels.csv** - file containing the target `MGMT_value` for each subject in the training data (e.g. the presence of MGMT promoter methylation)
+- **test/** - the test files, which use the same structure as `train/`; your task is to predict the `MGMT_value` for each subject in the test data. **NOTE**: the total size of the rerun test set (Public and Private) is ~5x the size of the Public test set
+- **sample_submission.csv** - a sample submission file in the correct format

dslighting/registry/rsna-miccai-brain-tumor-radiogenomic-classification/grade.py

@@ -0,0 +1,13 @@
+import pandas as pd
+from sklearn.metrics import roc_auc_score
+
+from mlebench.competitions.utils import prepare_for_auroc_metric
+from mlebench.grade_helpers import InvalidSubmissionError
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    roc_auc_inputs = prepare_for_auroc_metric(
+        answers=answers, submission=submission, id_col="BraTS21ID", target_col="MGMT_value"
+    )
+    score = roc_auc_score(y_true=roc_auc_inputs["y_true"], y_score=roc_auc_inputs["y_score"])
+    return score

dslighting/registry/rsna-miccai-brain-tumor-radiogenomic-classification/prepare.py

@@ -0,0 +1,47 @@
+import shutil
+from pathlib import Path
+
+import numpy as np
+from sklearn.model_selection import train_test_split
+
+from mlebench.utils import read_csv
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    # Create train and test splits from train set
+    old_train = read_csv(raw / "train_labels.csv", dtype={"BraTS21ID": str, "MGMT_value": int})
+    new_train, new_test = train_test_split(old_train, test_size=0.1, random_state=0)
+
+    # Copy over images
+    (public / "train").mkdir(exist_ok=True)
+    for file_id in new_train["BraTS21ID"]:
+        (public / "train" / file_id).mkdir(exist_ok=True)
+        shutil.copytree(
+            src=raw / "train" / file_id,
+            dst=public / "train" / file_id,
+            dirs_exist_ok=True,
+        )
+    assert len(list(public.glob("train/*"))) == len(
+        new_train
+    ), "Public train should have the same number of images as the train set"
+
+    (public / "test").mkdir(exist_ok=True)
+    for file_id in new_test["BraTS21ID"]:
+        (public / "test" / file_id).mkdir(exist_ok=True)
+        shutil.copytree(
+            src=raw / "train" / file_id,
+            dst=public / "test" / file_id,
+            dirs_exist_ok=True,
+        )
+    assert len(list(public.glob("test/*"))) == len(
+        new_test
+    ), "Public train should have the same number of images as the train set"
+
+    # Create a sample submission file
+    submission_df = new_test.copy()
+    submission_df["MGMT_value"] = 0.5
+
+    # Copy over files
+    new_train.to_csv(public / "train_labels.csv", index=False)
+    new_test.to_csv(private / "test.csv", index=False)
+    submission_df.to_csv(public / "sample_submission.csv", index=False)

dslighting/registry/rsna-miccai-brain-tumor-radiogenomic-classification/prepare_val.py

@@ -0,0 +1,97 @@
+import shutil
+from pathlib import Path
+
+import pandas as pd
+from sklearn.model_selection import train_test_split
+
+from mlebench.utils import read_csv
+
+
+def _create_split_files(
+    train_df: pd.DataFrame,
+    test_df: pd.DataFrame,
+    public_dest: Path,
+    private_dest: Path,
+    raw_data_path: Path,
+):
+    """
+    Helper function to populate public and private directories for a given split.
+
+    This function handles copying images, creating label files, and generating
+    a sample submission, ensuring a consistent structure across different data splits.
+    """
+    public_dest.mkdir(exist_ok=True)
+    private_dest.mkdir(exist_ok=True)
+
+    # Copy over images for the training set
+    (public_dest / "train").mkdir(exist_ok=True)
+    for file_id in train_df["BraTS21ID"]:
+        (public_dest / "train" / file_id).mkdir(exist_ok=True)
+        shutil.copytree(
+            src=raw_data_path / "train" / file_id,
+            dst=public_dest / "train" / file_id,
+            dirs_exist_ok=True,
+        )
+    assert len(list(public_dest.glob("train/*"))) == len(
+        train_df
+    ), "Public train should have the same number of images as the train set"
+
+    # Copy over images for the test set (without labels)
+    (public_dest / "test").mkdir(exist_ok=True)
+    for file_id in test_df["BraTS21ID"]:
+        (public_dest / "test" / file_id).mkdir(exist_ok=True)
+        shutil.copytree(
+            src=raw_data_path / "train" / file_id,
+            dst=public_dest / "test" / file_id,
+            dirs_exist_ok=True,
+        )
+    assert len(list(public_dest.glob("test/*"))) == len(
+        test_df
+    ), "Public test should have the same number of images as the test set"
+
+    # Create a sample submission file for the public directory
+    submission_df = test_df.copy()
+    submission_df["MGMT_value"] = 0.5
+
+    # Copy over the final CSV files to their destinations
+    train_df.to_csv(public_dest / "train_labels.csv", index=False)
+    test_df.to_csv(private_dest / "test.csv", index=False)
+    submission_df.to_csv(public_dest / "sample_submission.csv", index=False)
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    # Load the full labeled dataset from the raw directory
+    all_labeled_data = read_csv(raw / "train_labels.csv", dtype={"BraTS21ID": str, "MGMT_value": int})
+
+    # --- Original Split (for public/private) ---
+    # Create the original train and test splits from the full dataset
+    train_df, test_df = train_test_split(all_labeled_data, test_size=0.1, random_state=0)
+
+    # Generate the original public and private directories.
+    # The output of this call will be identical to the original script's output.
+    _create_split_files(
+        train_df=train_df,
+        test_df=test_df,
+        public_dest=public,
+        private_dest=private,
+        raw_data_path=raw,
+    )
+
+    # --- New Validation Split (for public_val/private_val) ---
+    # Define new paths for the validation split, parallel to the original ones
+    public_val = public.parent / "public_val"
+    private_val = private.parent / "private_val"
+
+    # Create a new train/validation split from the *original training data* (train_df).
+    # The splitting logic (test_size, random_state) is intentionally replicated.
+    train_val_df, test_val_df = train_test_split(train_df, test_size=0.1, random_state=0)
+
+    # Generate the new public_val and private_val directories using the same helper function.
+    # This ensures the directory structure and filenames are identical to the original split.
+    _create_split_files(
+        train_df=train_val_df,
+        test_df=test_val_df,
+        public_dest=public_val,
+        private_dest=private_val,
+        raw_data_path=raw,
+    )

dslighting/registry/santander-customer-satisfaction/checksums.yaml

@@ -0,0 +1,8 @@
+private:
+  gold_submission.csv: db1f33118c34675797dbc09221c9756c
+  test.csv: ed16afd5bf79cbb154ad6a0425c90be1
+public:
+  sample_submission.csv: 7a66b2ec27a9366e11e80bd2fb5f6c8d
+  test.csv: 866eff76c8a0b428a5f4b745ecdfea43
+  train.csv: 7239523331e076ebdbb356f29b0ce6c5
+zip: ""

dslighting/registry/santander-customer-satisfaction/config.yaml

@@ -0,0 +1,17 @@
+id: santander-customer-satisfaction
+name: Santander Customer Satisfaction
+competition_type: simple
+awards_medals: false
+prizes: null
+description: mlebench/competitions/santander-customer-satisfaction/description.md
+
+dataset:
+  answers: santander-customer-satisfaction/prepared/private/test.csv
+  gold_submission: santander-customer-satisfaction/prepared/private/gold_submission.csv
+  sample_submission: santander-customer-satisfaction/prepared/public/sample_submission.csv
+
+grader:
+  name: auc-roc
+  grade_fn: mlebench.competitions.santander-customer-satisfaction.grade:grade
+
+preparer: mlebench.competitions.santander-customer-satisfaction.prepare:prepare

dslighting/registry/santander-customer-satisfaction/description.md

@@ -0,0 +1,36 @@
+# Santander Customer Satisfaction
+
+## Task Description
+
+Predict whether a Santander Bank customer is dissatisfied based on hundreds of anonymized account-level features. The binary target `TARGET` equals `1` when the customer is unhappy and `0` otherwise. A good model lets the bank intervene and retain customers before they churn.
+
+## Data Description
+
+- **train.csv** – labelled training set with columns `ID`, `TARGET`, and the anonymized feature set.
+- **test.csv** – public test features without the `TARGET` label.
+- **sample_submission.csv** – reference submission template containing the required `ID,TARGET` header and ordering.
+- **gold_submission.csv** – ground-truth probabilities for private evaluation (kept hidden from participants).
+
+All files are provided in CSV format. Feature names follow the original Kaggle competition naming convention (e.g. `var3`, `ind_var5`, `num_var37`).
+
+## Evaluation
+
+Submissions are scored with the area under the receiver operating characteristic curve (AUC-ROC) between the predicted probability and the true `TARGET`. Higher values are better, with `1.0` indicating a perfect ranking.
+
+## Submission Format
+
+Participants must upload a CSV with two columns:
+
+```text
+ID,TARGET
+2,0.135
+5,0.042
+6,0.873
+```
+
+- `ID` must exactly match the identifiers in the public test set.
+- `TARGET` should be the predicted probability that the corresponding customer is dissatisfied (`TARGET = 1`).
+
+## Source
+
+Original competition: [Santander Customer Satisfaction](https://www.kaggle.com/competitions/santander-customer-satisfaction). Data is redistributed under Kaggle’s terms for educational benchmarking.

dslighting/registry/santander-customer-satisfaction/grade.py

@@ -0,0 +1,10 @@
+import pandas as pd
+from sklearn.metrics import roc_auc_score
+
+from mlebench.competitions.utils import prepare_for_auroc_metric
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    """Compute AUC-ROC between submitted probabilities and ground truth."""
+    roc_auc_inputs = prepare_for_auroc_metric(submission, answers, id_col="ID", target_col="TARGET")
+    return roc_auc_score(y_true=roc_auc_inputs["y_true"], y_score=roc_auc_inputs["y_score"])

dslighting/registry/santander-customer-satisfaction/prepare.py

@@ -0,0 +1,41 @@
+from pathlib import Path
+
+import pandas as pd
+
+from mlebench.utils import read_csv
+
+
+def prepare(raw: Path, public: Path, private: Path) -> None:
+    """Materialize Santander Customer Satisfaction CSV splits."""
+    public.mkdir(parents=True, exist_ok=True)
+    private.mkdir(parents=True, exist_ok=True)
+
+    train = read_csv(raw / "train.csv")
+    test_public = read_csv(raw / "test.csv")
+    answers = read_csv(raw / "test_answer.csv")
+
+    if (raw / "sample_submission.csv").exists():
+        sample_submission = read_csv(raw / "sample_submission.csv")
+    else:
+        sample_submission = answers.copy()
+        sample_submission["TARGET"] = 0.0
+
+    merged_private = test_public.merge(answers, on="ID", how="left", validate="one_to_one")
+    if merged_private["TARGET"].isna().any():
+        missing_ids = merged_private.loc[merged_private["TARGET"].isna(), "ID"].tolist()[:5]
+        raise ValueError(f"Missing TARGET labels for IDs (showing up to 5): {missing_ids}")
+
+    train.to_csv(public / "train.csv", index=False)
+    test_public.to_csv(public / "test.csv", index=False)
+    sample_submission.to_csv(public / "sample_submission.csv", index=False)
+
+    merged_private.to_csv(private / "test.csv", index=False)
+    answers.to_csv(private / "gold_submission.csv", index=False)
+
+    assert len(test_public) == len(answers) == len(
+        merged_private
+    ), "Test features, answers, and private test must have the same length."
+    assert set(train.columns) - {"TARGET"} == set(
+        test_public.columns
+    ), "Train and public test feature columns should match aside from TARGET."
+    assert sample_submission.columns.tolist() == ["ID", "TARGET"], "Sample submission must be ID,TARGET."

dslighting/registry/sciencebench-001-clintox-nn/checksums.yaml

@@ -0,0 +1,3 @@
+# Checksums for data validation
+# Auto-generated during preparation
+prepared: true

dslighting/registry/sciencebench-001-clintox-nn/config.yaml

@@ -0,0 +1,16 @@
+id: sciencebench-001-clintox-nn
+name: "ScienceBench - clintox_nn"
+competition_type: code
+awards_medals: false
+prizes: null
+description: benchmarks/sciencebench/competitions/sciencebench-001-clintox-nn/description.md
+
+dataset:
+  answers: sciencebench-001-clintox-nn/prepared/private/answer.csv
+  sample_submission: sciencebench-001-clintox-nn/prepared/public/sample_submission.csv
+
+grader:
+  name: roc_auc
+  grade_fn: benchmarks.sciencebench.competitions.sciencebench-001-clintox-nn.grade:grade
+
+preparer: benchmarks.sciencebench.competitions.sciencebench-001-clintox-nn.prepare:prepare

dslighting/registry/sciencebench-001-clintox-nn/description.md

@@ -0,0 +1,30 @@
+# ScienceBench Task 1
+
+## Overview
+
+- Domain: Computational Chemistry
+- Subtask Categories: Feature Engineering, Deep Learning
+- Source: deepchem/deepchem
+- Output Type: Tabular
+
+## Task
+
+Train a multitask model on the Clintox dataset to predict a drug's toxicity and FDA approval status. Save the test set predictions, including the SMILES representation of drugs and the probability of positive labels.
+
+## Dataset
+
+[START Preview of clintox/clintox_train.csv]
+smiles,FDA_APPROVED,CT_TOX
+CCC(/C=C/Cl)(C#C)O,1,0
+C[C@H]1C[C@H]2[C@@H]3CC[C@@H]([C@]3(C[C@@H]([C@@H]2[C@@]4(C1=CC(=O)CC4)C)O)C)C(=O)C,1,0
+C[C@@H]1CCN([C@H](C1)C(=O)[O-])C(=O)[C@H](CCC[NH+]=C(N)N)NS(=O)(=O)c2cccc3c2NC[C@@H](C3)C,1,0
+...
+[END Preview of clintox/clintox_train.csv]
+
+## Submission Format
+
+Submit `sample_submission.csv` with the same header and column order as the template. Ensure numeric columns retain their dtype and identifiers remain aligned.
+
+## Evaluation
+
+Classification accuracy (higher is better).

dslighting/registry/sciencebench-001-clintox-nn/grade.py

@@ -0,0 +1,56 @@
+"""
+Grading function for ScienceBench task 1
+"""
+
+import pandas as pd
+from sklearn.metrics import roc_auc_score
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    """
+    Grade submission using ROC-AUC (as configured in `config.yaml`).
+
+    Args:
+        submission: DataFrame with predictions
+        answers: DataFrame with ground truth
+
+    Returns:
+        ROC-AUC score (higher is better)
+    """
+    if submission is None or answers is None:
+        return None
+
+    id_col = "smiles" if "smiles" in submission.columns and "smiles" in answers.columns else None
+
+    # Align rows on identifier if available; otherwise assume same order.
+    if id_col:
+        merged = pd.merge(
+            answers,
+            submission,
+            on=id_col,
+            suffixes=("_true", "_pred"),
+            how="inner",
+        )
+    else:
+        merged = pd.concat(
+            [answers.add_suffix("_true"), submission.add_suffix("_pred")],
+            axis=1,
+        )
+
+    # Score all common target columns (exclude identifier column)
+    target_cols = [c for c in answers.columns if c != id_col and c in submission.columns]
+    if not target_cols:
+        return None
+
+    aucs: list[float] = []
+    for col in target_cols:
+        y_true = merged[f"{col}_true"]
+        y_pred = merged[f"{col}_pred"]
+        # ROC-AUC is undefined if only one class is present.
+        if y_true.nunique(dropna=True) < 2:
+            continue
+        aucs.append(float(roc_auc_score(y_true, y_pred)))
+
+    if not aucs:
+        return None
+    return float(sum(aucs) / len(aucs))