dslighting 1.3.9__py3-none-any.whl → 1.7.1__py3-none-any.whl

dslighting/registry/bms-molecular-translation/description.md

@@ -0,0 +1,68 @@
+## Description
+
+In a technology-forward world, sometimes the best and easiest tools are still pen and paper. Organic chemists frequently draw out molecular work with the Skeletal formula, a structural notation used for centuries. Recent publications are also annotated with machine-readable chemical descriptions (InChI), but there are decades of scanned documents that can't be automatically searched for specific chemical depictions. Automated recognition of optical chemical structures, with the help of machine learning, could speed up research and development efforts.
+
+Unfortunately, most public data sets are too small to support modern machine learning models. Existing tools produce 90% accuracy but only under optimal conditions. Historical sources often have some level of image corruption, which reduces performance to near zero. In these cases, time-consuming, manual work is required to reliably convert scanned chemical structure images into a machine-readable format.
+
+Bristol-Myers Squibb is a global biopharmaceutical company working to transform patients' lives through science. Their mission is to discover, develop, and deliver innovative medicines that help patients prevail over serious diseases.
+
+In this competition, you'll interpret old chemical images. With access to a large set of synthetic image data generated by Bristol-Myers Squibb, you'll convert images back to the underlying chemical structure annotated as InChI text.
+
+Tools to curate chemistry literature would be a significant benefit to researchers. If successful, you'll help chemists expand access to collective chemical research. In turn, this would speed up research and development efforts in many key fields by avoiding repetition of previously published chemistries and identifying novel trends via mining large data sets.
+
+Photo by Terry Vlisidis on Unsplash
+
+## Evaluation
+
+Submissions are evaluated on the mean [Levenshtein distance](http://en.wikipedia.org/wiki/Levenshtein_distance) between the InChi strings you submit and the ground truth InChi values.
+
+## Submission File
+
+For each `image_id` in the test set, you must predict the InChi string of the molecule in the corresponding image. The file should contain a header and have the following format:
+
+```
+image_id,InChI
+00000d2a601c,InChI=1S/H2O/h1H2
+00001f7fc849,InChI=1S/H2O/h1H2
+000037687605,InChI=1S/H2O/h1H2
+etc.
+```
+
+## Timeline
+
+Update May 28, 2021. The competition deadline has been extended 24 hours from June 2, 2021 at 11:59 pm UTC to June 3, 2021 at 11:59pm UTC. See [this forum post](https://www.kaggle.com/c/bms-molecular-translation/discussion/242403) for additional details.
+
+- March 2, 2021 -  Competition Start Date
+
+- May 26, 2021 - Entry deadline. You must accept the competition rules before this date in order to compete.
+
+- May 26, 2021 - Team Merger deadline. This is the last day participants may join or merge teams.
+
+- June 3, 2021 - Final submission deadline.
+
+All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
+
+## Prizes
+
+- 1st Place - $25,000
+- 2nd Place - $15,000
+- 3rd Place - $10,000
+
+## Citation
+
+Addison Howard, inversion, Jacob Albrecht, Yvette. (2021). Bristol-Myers Squibb -- Molecular Translation. Kaggle. https://kaggle.com/competitions/bms-molecular-translation
+
+# Data
+
+## Dataset Description
+
+In this competition, you are provided with images of chemicals, with the objective of predicting the corresponding [International Chemical Identifier](https://en.wikipedia.org/wiki/International_Chemical_Identifier) (InChI) text string of the image. The images provided (both in the training data as well as the test data) may be rotated to different angles, be at various resolutions, and have different noise levels.
+
+Note: There are about 4m total images in this dataset. Unzipping the downloaded data will take a non-trivial amount of time.
+
+## Files
+
+- **train/** - the training images, arranged in a 3-level folder structure by `image_id`
+- **test/** - the test images, arranged in the same folder structure as `train/`
+- **train_labels.csv** - ground truth InChi labels for the training images
+- **sample_submission.csv** - a sample submission file in the correct format

dslighting/registry/bms-molecular-translation/description_obfuscated.md

@@ -0,0 +1,26 @@
+# Task
+
+Provided with images of chemicals, predict the corresponding International Chemical Identifier (InChI) text string of the image.
+
+# Metric
+
+Mean [Levenshtein distance](http://en.wikipedia.org/wiki/Levenshtein_distance) between the InChi strings you submit and the ground truth InChi values.
+
+# Submission Format
+
+For each `image_id` in the test set, you must predict the InChi string of the molecule in the corresponding image. The file should contain a header and have the following format:
+
+```
+image_id,InChI
+00000d2a601c,InChI=1S/H2O/h1H2
+00001f7fc849,InChI=1S/H2O/h1H2
+000037687605,InChI=1S/H2O/h1H2
+etc.
+```
+
+# Dataset
+
+- **train/** - the training images, arranged in a 3-level folder structure by `image_id`
+- **test/** - the test images, arranged in the same folder structure as `train/`
+- **train_labels.csv** - ground truth InChi labels for the training images
+- **sample_submission.csv** - a sample submission file in the correct format

dslighting/registry/bms-molecular-translation/grade.py

@@ -0,0 +1,40 @@
+import numpy as np
+import pandas as pd
+from Levenshtein import distance
+
+from mlebench.grade_helpers import InvalidSubmissionError
+
+
+def edit_distance_array(y_true: pd.Series, y_pred: pd.Series) -> float:
+    return np.mean([distance(a, b) for a, b in zip(y_true, y_pred)])
+
+
+def prepare_for_metric(submission: pd.DataFrame, answers: pd.DataFrame):
+    if len(submission) != len(answers):
+        raise InvalidSubmissionError("Submission and answers must have the same length")
+
+    if "image_id" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have an 'image_id' column")
+
+    if "InChI" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have a 'InChI' column")
+
+    assert "image_id" in answers.columns, "Answers must have 'image_id' column"
+    assert "InChI" in answers.columns, "Answers must have 'InChI' column"
+
+    submission = submission.sort_values("image_id")
+    answers = answers.sort_values("image_id")
+
+    if (submission["image_id"].values != answers["image_id"].values).any():
+        raise InvalidSubmissionError("Submission and answers must have the same ids")
+
+    # Convert to strings
+    submission["InChI"] = submission["InChI"].astype(str)
+    answers["InChI"] = answers["InChI"].astype(str)
+
+    return answers["InChI"], submission["InChI"]
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    y_true, y_pred = prepare_for_metric(submission, answers)
+    return edit_distance_array(y_true, y_pred)

dslighting/registry/bms-molecular-translation/prepare.py

@@ -0,0 +1,68 @@
+import shutil
+from pathlib import Path
+
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from tqdm import tqdm
+
+
+def make_image_subpath(image_id: str) -> Path:
+    """
+    Creates a triple-nested directory structure from the first 3 characters of the image_id.
+    """
+    subpath = Path(image_id[0]) / image_id[1] / image_id[2] / f"{image_id}.png"
+    return subpath
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    """
+    Splits the data in raw into public and private datasets with appropriate test/train splits.
+    """
+    # Load train data
+    old_train = pd.read_csv(raw / "train_labels.csv")
+
+    # Create train, test from train split
+    new_train, new_test = train_test_split(old_train, test_size=0.2, random_state=0)
+    new_train.to_csv(public / "train_labels.csv", index=False)
+    new_test.to_csv(private / "test.csv", index=False)
+
+    # Copy train files
+    for idx, row in tqdm(new_train.iterrows(), total=len(new_train), desc="Copying train images"):
+        image_id = row["image_id"]
+        src = raw / "train" / make_image_subpath(image_id)
+        dst = public / "train" / make_image_subpath(image_id)
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copyfile(src=src, dst=dst)
+
+    # Copy test files
+    for idx, row in tqdm(new_test.iterrows(), total=len(new_test), desc="Copying test images"):
+        image_id = row["image_id"]
+        src = raw / "train" / make_image_subpath(image_id)
+        dst = public / "test" / make_image_subpath(image_id)
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copyfile(src=src, dst=dst)
+
+    # Create sample submission
+    sample_submission = new_test.copy()
+    sample_submission["InChI"] = "InChI=1S/H2O/h1H2"
+    sample_submission.to_csv(public / "sample_submission.csv", index=False)
+
+    # Copy other files in the dataset (no modification needed)
+    shutil.copyfile(src=raw / "extra_approved_InChIs.csv", dst=public / "extra_approved_InChIs.csv")
+
+    # Checks
+    assert len(new_train) + len(new_test) == len(
+        old_train
+    ), f"Expected {len(old_train)} total images in new_train ({len(new_train)}) and new_test ({len(new_test)})"
+    assert len(list((public / "train").glob("**/*.png"))) == len(
+        new_train
+    ), f"Expected {len(new_train)} train images in public/train, but got {len(list((public / 'train').glob('**/*.png')))}"
+    assert len(list((public / "test").glob("**/*.png"))) == len(
+        new_test
+    ), f"Expected {len(new_test)} test images in public/test, but got {len(list((public / 'test').glob('**/*.png')))}"
+
+    assert "image_id" in sample_submission.columns, "Sample submission must have 'image_id' column"
+    assert "InChI" in sample_submission.columns, "Sample submission must have 'InChI' column"
+    assert len(sample_submission) == len(
+        new_test
+    ), f"Expected {len(new_test)} images in sample submission, but got {len(sample_submission)}"

dslighting/registry/bms-molecular-translation/prepare_val.py

@@ -0,0 +1,131 @@
+import shutil
+from pathlib import Path
+
+import pandas as pd
+from sklearn.model_selection import train_test_split
+from tqdm import tqdm
+
+
+def make_image_subpath(image_id: str) -> Path:
+    """
+    Creates a triple-nested directory structure from the first 3 characters of the image_id.
+    """
+    subpath = Path(image_id[0]) / image_id[1] / image_id[2] / f"{image_id}.png"
+    return subpath
+
+
+def _create_split_files(
+    train_df: pd.DataFrame,
+    test_df: pd.DataFrame,
+    raw_images_path: Path,
+    public_path: Path,
+    private_path: Path,
+):
+    """
+    Helper function to generate the directory structure and files for a given split.
+    This function populates the public and private directories with train/test data,
+    images, and a sample submission file.
+    """
+    # Create output directories
+    public_path.mkdir(exist_ok=True)
+    private_path.mkdir(exist_ok=True)
+
+    # Save dataframes
+    train_df.to_csv(public_path / "train_labels.csv", index=False)
+    test_df.to_csv(private_path / "test.csv", index=False)
+
+    # Copy train files
+    desc_prefix = public_path.name
+    for _, row in tqdm(train_df.iterrows(), total=len(train_df), desc=f"Copying {desc_prefix} train images"):
+        image_id = row["image_id"]
+        src = raw_images_path / make_image_subpath(image_id)
+        dst = public_path / "train" / make_image_subpath(image_id)
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copyfile(src=src, dst=dst)
+
+    # Copy test files
+    for _, row in tqdm(test_df.iterrows(), total=len(test_df), desc=f"Copying {desc_prefix} test images"):
+        image_id = row["image_id"]
+        src = raw_images_path / make_image_subpath(image_id)
+        dst = public_path / "test" / make_image_subpath(image_id)
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copyfile(src=src, dst=dst)
+
+    # Create sample submission
+    sample_submission = test_df.copy()
+    sample_submission["InChI"] = "InChI=1S/H2O/h1H2"
+    sample_submission.to_csv(public_path / "sample_submission.csv", index=False)
+
+    # Checks
+    assert len(list((public_path / "train").glob("**/*.png"))) == len(
+        train_df
+    ), f"Expected {len(train_df)} train images in {public_path}/train, but got {len(list((public_path / 'train').glob('**/*.png')))}"
+    assert len(list((public_path / "test").glob("**/*.png"))) == len(
+        test_df
+    ), f"Expected {len(test_df)} test images in {public_path}/test, but got {len(list((public_path / 'test').glob('**/*.png')))}"
+
+    assert "image_id" in sample_submission.columns, "Sample submission must have 'image_id' column"
+    assert "InChI" in sample_submission.columns, "Sample submission must have 'InChI' column"
+    assert len(sample_submission) == len(
+        test_df
+    ), f"Expected {len(test_df)} images in sample submission, but got {len(sample_submission)}"
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    """
+    Splits the data in raw into public and private datasets with appropriate test/train splits.
+    Also creates a secondary validation split in parallel public_val and private_val directories.
+    """
+    # Load train data
+    old_train = pd.read_csv(raw / "train_labels.csv")
+
+    # ---- 1. Create the original train/test split ----
+    # This split is for the main competition test set.
+    new_train, new_test = train_test_split(old_train, test_size=0.2, random_state=0)
+
+    # Generate the files for the original public and private directories
+    _create_split_files(
+        train_df=new_train,
+        test_df=new_test,
+        raw_images_path=raw / "train",
+        public_path=public,
+        private_path=private,
+    )
+
+    # ---- 2. Create the new validation split ----
+    # This second split is performed on the `new_train` set created above.
+    # We want the new `test_val` to be the same size as the original `new_test`.
+    # Original test size = 0.2 * total. Original train size = 0.8 * total.
+    # New test size relative to train set = 0.2 / 0.8 = 0.25
+    train_val, test_val = train_test_split(new_train, test_size=0.25, random_state=0)
+
+    # Define the new parallel directories for the validation set
+    public_val = public.parent / "public_val"
+    private_val = private.parent / "private_val"
+
+    # Generate the files for the new validation directories
+    _create_split_files(
+        train_df=train_val,
+        test_df=test_val,
+        raw_images_path=raw / "train",
+        public_path=public_val,
+        private_path=private_val,
+    )
+
+    # ---- 3. Copy shared files and run final checks ----
+
+    # Copy other files into both public and public_val directories
+    shutil.copyfile(src=raw / "extra_approved_InChIs.csv", dst=public / "extra_approved_InChIs.csv")
+    shutil.copyfile(src=raw / "extra_approved_InChIs.csv", dst=public_val / "extra_approved_InChIs.csv")
+
+    # Original split checks
+    assert len(new_train) + len(new_test) == len(
+        old_train
+    ), f"Expected {len(old_train)} total images in new_train ({len(new_train)}) and new_test ({len(new_test)})"
+
+    # New validation split checks
+    assert len(train_val) + len(test_val) == len(
+        new_train
+    ), f"Expected {len(new_train)} total images in train_val ({len(train_val)}) and test_val ({len(test_val)})"
+    # Ensure the size of the validation test set is approx. the same as the original test set
+    assert abs(len(test_val) - len(new_test)) <= 1, "Validation test set size should match original test set size"

dslighting/registry/cassava-leaf-disease-classification/checksums.yaml

@@ -0,0 +1,7 @@
+private:
+  test.csv: 34b2b412be39aba1edac9c04990e4fb8
+public:
+  label_num_to_disease_map.json: 133bf5c6a60c901792937eaad034b188
+  sample_submission.csv: b27270a483141fa217d990d55b4b3d49
+  train.csv: fede2927f0d412590f195abfbd07b578
+zip: e1dc225f0fc8a4161e48a4f9015e16c0

dslighting/registry/cassava-leaf-disease-classification/config.yaml

@@ -0,0 +1,28 @@
+id: cassava-leaf-disease-classification
+name: Cassava Leaf Disease Classification
+competition_type: code
+awards_medals: true
+prizes:
+  - position: 1
+    value: 8000
+  - position: 2
+    value: 4000
+  - position: 3
+    value: 3000
+  - position: TPU Star 1
+    value: 1000
+  - position: TPU Star 2
+    value: 1000
+  - position: TPU Star 3
+    value: 1000
+description: mlebench/competitions/cassava-leaf-disease-classification/description.md
+
+dataset:
+  answers: cassava-leaf-disease-classification/prepared/private/test.csv
+  sample_submission: cassava-leaf-disease-classification/prepared/public/sample_submission.csv
+
+grader:
+  name: accuracy
+  grade_fn: mlebench.competitions.cassava-leaf-disease-classification.grade:grade
+
+preparer: mlebench.competitions.cassava-leaf-disease-classification.prepare:prepare

dslighting/registry/cassava-leaf-disease-classification/description.md

@@ -0,0 +1,124 @@
+# Overview
+
+## Overview
+
+### Description
+
+As the second-largest provider of carbohydrates in Africa, cassava is a key food security crop grown by smallholder farmers because it can withstand harsh conditions. At least 80% of household farms in Sub-Saharan Africa grow this starchy root, but viral diseases are major sources of poor yields. With the help of data science, it may be possible to identify common diseases so they can be treated.
+
+Existing methods of disease detection require farmers to solicit the help of government-funded agricultural experts to visually inspect and diagnose the plants. This suffers from being labor-intensive, low-supply and costly. As an added challenge, effective solutions for farmers must perform well under significant constraints, since African farmers may only have access to mobile-quality cameras with low-bandwidth.
+
+In this competition, we introduce a dataset of 21,367 labeled images collected during a regular survey in Uganda. Most images were crowdsourced from farmers taking photos of their gardens, and annotated by experts at the National Crops Resources Research Institute (NaCRRI) in collaboration with the AI lab at Makerere University, Kampala. This is in a format that most realistically represents what farmers would need to diagnose in real life.
+
+Your task is to classify each cassava image into four disease categories or a fifth category indicating a healthy leaf. With your help, farmers may be able to quickly identify diseased plants, potentially saving their crops before they inflict irreparable damage.
+
+> **Recommended Tutorial**
+>
+> We highly recommend [Jesse Mostipak's Getting Started Tutorial](https://www.kaggle.com/jessemostipak/getting-started-tpus-cassava-leaf-disease) that walks you through making your very first submission step by step.
+
+### Acknowledgements
+
+The Makerere Artificial Intelligence (AI) Lab is an AI and Data Science research group based at Makerere University in Uganda. The lab specializes in the application of artificial intelligence and data science - including for example, methods from machine learning, computer vision and predictive analytics to problems in the developing world. Their mission is: "To advance Artificial Intelligence research to solve real-world challenges."
+
+We thank the different experts and collaborators from National Crops Resources Research Institute (NaCRRI) for assisting in preparing this dataset.
+
+> This is a Code Competition. Refer to [Code Requirements](https://www.kaggle.com/c/cassava-leaf-disease-classification/overview/code-requirements) for details.
+
+### Evaluation
+
+#### Evaluation
+
+Submissions will be evaluated based on their [categorization accuracy](https://developers.google.com/machine-learning/crash-course/classification/accuracy).
+
+#### Submission Format
+
+The submission format for the competition is a csv file with the following format:
+
+```
+image_id,label
+1000471002.jpg,4
+1000840542.jpg,4
+etc.
+```
+
+### Timeline
+
+- **February 11, 2021** - Entry deadline. You must accept the competition rules before this date in order to compete.
+
+- **February 11, 2021** - Team Merger deadline. This is the last day participants may join or merge teams.
+
+- **February 18, 2021** - Final submission deadline.
+
+All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
+
+### Prizes
+
+Leaderboard Prizes: Awarded based on private leaderboard ranking.
+
+- 1st Place - \$8,000
+- 2nd Place - \$4,000
+- 3rd Place - \$3,000
+
+TPU Star Prizes: Awarded to the most knowledgeable and helpful TPU experts in the community.
+
+- Three \$1,000 Prizes
+
+"TPU Star" prizes will be evaluated by experts from Google, on the following criteria:
+
+1. Forum and notebook discussion/comment contributions
+2. Quality of code samples in public notebooks and/or forums
+3. Thoughtful analysis and explainability of code content
+
+You may become eligible for these prizes either of two ways:
+
+- Kaggle-identified: Kaggle will query all participants in the competition contributing upvoted public notebook(s) and making discussion contribution(s).
+- Self-nominated: [Submit this form](https://www.kaggle.com/tpu-star-submission-form) by February 28, 2021 to self-nominate for consideration. At a minimum, you must have shared a public notebook which uses Kaggle's TPU integration on this competition's dataset.
+
+> Be aware that because this is a code competition with a hidden test set, internet and TPUs cannot be enabled on your submission notebook. Therefore TPUs will only be available for training models. For a walk-through on how to train on TPUs and run inference/submit on GPUs, see our [TPU Docs](https://www.kaggle.com/docs/tpu#tpu6).
+
+### Code Requirements
+
+![Kerneler](https://storage.googleapis.com/kaggle-media/competitions/general/Kerneler-white-desc2_transparent.png)
+
+### This is a Code Competition
+
+Submissions to this competition must be made through Notebooks. In order for the "Submit to Competition" button to be active after a commit, the following conditions must be met:
+
+- CPU Notebook `<= 9` hours run-time
+- GPU Notebook `<= 9` hours run-time
+- [TPUs](https://www.kaggle.com/docs/tpu) will not be available for making submissions to this competition. You are still welcome to use them for training models. A walk-through for how to train on TPUs and run inference/submit on GPUs, see our [TPU Docs](https://www.kaggle.com/docs/tpu#tpu6).
+- Internet access disabled
+- Freely & publicly available external data is allowed, including pre-trained models
+- Submission file must be named "submission.csv"
+
+Please see the [Code Competition FAQ](https://www.kaggle.com/docs/competitions#notebooks-only-FAQ) for more information on how to submit. And review the [code debugging doc](https://www.kaggle.com/code-competition-debugging) if you are encountering submission errors.
+
+### Citation
+
+ErnestMwebaze, Jesse Mostipak, Joyce, Julia Elliott, Sohier Dane. (2020). Cassava Leaf Disease Classification. Kaggle. https://kaggle.com/competitions/cassava-leaf-disease-classification
+
+# Data
+
+## Dataset Description
+
+Can you identify a problem with a cassava plant using a photo from a relatively inexpensive camera? This competition will challenge you to distinguish between several diseases that cause material harm to the food supply of many African countries. In some cases the main remedy is to burn the infected plants to prevent further spread, which can make a rapid automated turnaround quite useful to the farmers.
+
+### Files
+
+**[train/test]_images** the image files. The full set of test images will only be available to your notebook when it is submitted for scoring. Expect to see roughly 15,000 images in the test set.
+
+**train.csv**
+
+- `image_id` the image file name.
+
+- `label` the ID code for the disease.
+
+**sample_submission.csv** A properly formatted sample submission, given the disclosed test set content.
+
+- `image_id` the image file name.
+
+- `label` the predicted ID code for the disease.
+
+**[train/test]_tfrecords** the image files in tfrecord format.
+
+**label_num_to_disease_map.json** The mapping between each disease code and the real disease name.

dslighting/registry/cassava-leaf-disease-classification/description_obfuscated.md

@@ -0,0 +1,36 @@
+# Task
+
+Classify each cassava image into four disease categories or a fifth category indicating a healthy leaf.
+
+# Metric
+
+Categorization accuracy.
+
+# Submission Format
+
+```
+image_id,label
+1000471002.jpg,4
+1000840542.jpg,4
+etc.
+```
+
+# Dataset
+
+**[train/test]_images** the image files.
+
+**train.csv**
+
+- `image_id` the image file name.
+
+- `label` the ID code for the disease.
+
+**sample_submission.csv** A properly formatted sample submission, given the disclosed test set content.
+
+- `image_id` the image file name.
+
+- `label` the predicted ID code for the disease.
+
+**[train/test]_tfrecords** the image files in tfrecord format.
+
+**label_num_to_disease_map.json** The mapping between each disease code and the real disease name.

dslighting/registry/cassava-leaf-disease-classification/grade.py

@@ -0,0 +1,12 @@
+import pandas as pd
+from sklearn.metrics import accuracy_score
+
+from mlebench.competitions.utils import prepare_for_accuracy_metric
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    accuracy_inputs = prepare_for_accuracy_metric(
+        submission=submission, answers=answers, target_column="label", id_column="image_id"
+    )
+    score = accuracy_score(**accuracy_inputs)
+    return score

dslighting/registry/cassava-leaf-disease-classification/prepare.py

@@ -0,0 +1,113 @@
+import shutil
+from pathlib import Path
+
+from sklearn.model_selection import train_test_split
+from tqdm.auto import tqdm
+
+from mlebench.competitions.utils import get_ids_from_tf_records
+from mlebench.utils import read_csv
+
+
+def prepare(raw: Path, public: Path, private: Path):
+
+    # need to split based on the TFRecord files, since not mentioned in the CSVs
+    tfrecord_files = [
+        path
+        for path in sorted((raw / "train_tfrecords").iterdir())
+        if path.is_file() and path.suffix == ".tfrec"
+    ]
+
+    # In the original there are 21397 train samples and they say test has ~15000 test samples, which is ~ 0.4/0.6 test/train split
+    # We use 0.1 ratio to avoid removing too many samples from train
+    new_train_tfrecords, new_test_tfrecords = train_test_split(
+        tfrecord_files, test_size=0.1, random_state=0
+    )
+
+    # parse the IDs from the test tf records
+    test_ids = []
+    for path in new_test_tfrecords:
+        test_ids.extend(get_ids_from_tf_records(path))
+
+    old_train = read_csv(raw / "train.csv")
+
+    old_train["split"] = "train"
+    old_train.loc[old_train["image_id"].isin(test_ids), "split"] = "test"
+
+    new_train = old_train[old_train["split"] == "train"].drop(columns=["split"])
+    new_test = old_train[old_train["split"] == "test"].drop(columns=["split"])
+
+    sample_submission = new_test.copy()
+    sample_submission["label"] = 4
+
+    new_train.to_csv(public / "train.csv", index=False)
+    new_test.to_csv(private / "test.csv", index=False)
+    sample_submission.to_csv(public / "sample_submission.csv", index=False)
+
+    (public / "train_tfrecords").mkdir(parents=True, exist_ok=True)
+    for i, path in tqdm(
+        enumerate(new_train_tfrecords),
+        desc="Copying Train TFRecords",
+        total=len(new_train_tfrecords),
+    ):
+        length = path.stem.split("-")[1]
+        new_name = f"ld_train{i:02d}-{length}.tfrec"
+
+        shutil.copy(path, public / "train_tfrecords" / new_name)
+
+    (public / "test_tfrecords").mkdir(parents=True, exist_ok=True)
+    for i, path in tqdm(
+        enumerate(new_test_tfrecords), desc="Copying Test TFRecords", total=len(new_test_tfrecords)
+    ):
+        length = path.stem.split("-")[1]
+        new_name = f"ld_test{i:02d}-{length}.tfrec"
+
+        shutil.copy(path, public / "test_tfrecords" / new_name)
+
+    (public / "train_images").mkdir(parents=True, exist_ok=True)
+    for image_id in tqdm(new_train["image_id"], desc="Copying Train Images", total=len(new_train)):
+        shutil.copy(raw / "train_images" / image_id, public / "train_images")
+
+    (public / "test_images").mkdir(parents=True, exist_ok=True)
+    for image_id in tqdm(new_test["image_id"], desc="Copying Test Images", total=len(new_test)):
+        shutil.copy(raw / "train_images" / image_id, public / "test_images")
+
+    shutil.copy(raw / "label_num_to_disease_map.json", public / "label_num_to_disease_map.json")
+
+    # checks
+    assert len(new_train) + len(new_test) == len(
+        old_train
+    ), "Expected new train and new test lengths to sum to old train length"
+    assert len(sample_submission) == len(
+        new_test
+    ), "Expected sample submission length to be equal to new test length"
+
+    assert len(new_train) == sum(
+        1 for _ in (public / "train_images").iterdir()
+    ), "Mismatch in number of expected train images copied"
+    assert len(new_test) == sum(
+        1 for _ in (public / "test_images").iterdir()
+    ), "Mismatch in number of expected test images copied"
+
+    assert len(new_train_tfrecords) == sum(
+        1 for _ in (public / "train_tfrecords").iterdir()
+    ), "Mismatch in number of expected train TFRecords copied"
+    assert len(new_test_tfrecords) == sum(
+        1 for _ in (public / "test_tfrecords").iterdir()
+    ), "Mismatch in number of expected test TFRecords copied"
+
+    assert new_train.columns.tolist() == [
+        "image_id",
+        "label",
+    ], "Expected new train columns to be ['image_id', 'label']"
+    assert new_test.columns.tolist() == [
+        "image_id",
+        "label",
+    ], "Expected new test columns to be ['image_id', 'label']"
+    assert sample_submission.columns.tolist() == [
+        "image_id",
+        "label",
+    ], "Expected sample submission columns to be ['image_id', 'label']"
+
+    assert set(new_train["image_id"]).isdisjoint(
+        new_test["image_id"]
+    ), "Expected train and test image IDs to be disjoint"