dslighting 1.3.9__py3-none-any.whl → 1.7.1__py3-none-any.whl

dslighting/registry/learning-agency-lab-automated-essay-scoring-2/description.md

@@ -0,0 +1,153 @@
+# Overview
+
+## Overview
+
+The first [automated essay scoring competition](https://www.kaggle.com/competitions/asap-aes/overview) to tackle automated grading of student-written essays was twelve years ago. How far have we come from this initial competition? With an updated dataset and light years of new ideas we hope to see if we can get to the latest in automated grading to provide a real impact to overtaxed teachers who continue to have challenges with providing timely feedback, especially in underserved communities.
+
+The goal of this competition is to train a model to score student essays. Your efforts are needed to reduce the high expense and time required to hand grade these essays. Reliable automated techniques could allow essays to be introduced in testing, a key indicator of student learning that is currently commonly avoided due to the challenges in grading.
+
+## Description
+
+Essay writing is an important method to evaluate student learning and performance. It is also time-consuming for educators to grade by hand. Automated Writing Evaluation (AWE) systems can score essays to supplement an educator’s other efforts. AWEs also allow students to receive regular and timely feedback on their writing. However, due to their costs, many advancements in the field are not widely available to students and educators. Open-source solutions to assess student writing are needed to reach every community with these important educational tools.
+
+Previous efforts to develop open-source AWEs have been limited by small datasets that were not nationally diverse or focused on common essay formats. The first Automated Essay Scoring competition scored student-written short-answer responses, however, this is a writing task not often used in the classroom. To improve upon earlier efforts, a more expansive dataset that includes high-quality, realistic classroom writing samples was required. Further, to broaden the impact, the dataset should include samples across economic and location populations to mitigate the potential of algorithmic bias.
+
+In this competition, you will work with the largest open-access writing dataset aligned to current standards for student-appropriate assessments. Can you help produce an open-source essay scoring algorithm that improves upon the original [Automated Student Assessment Prize (ASAP)](https://www.kaggle.com/competitions/asap-aes/overview) competition hosted in 2012?
+
+Competition host Vanderbilt University is a private research university in Nashville, Tennessee. For this competition, Vanderbilt has partnered with The Learning Agency Lab, an Arizona-based independent nonprofit focused on developing the science of learning-based tools and programs for the social good.
+
+To ensure the results of this competition are widely available, winning solutions will be released as open source. More robust and accessible AWE options will help more students get the frequent feedback they need and provide educators with additional support, especially in underserved districts.
+
+### Acknowledgments
+
+Vanderbilt University and the Learning Agency Lab would like to thank the Bill & Melinda Gates Foundation, Schmidt Futures, and Chan Zuckerberg Initiative for their support in making this work possible.
+
+![BMGF_logo](https://storage.googleapis.com/kaggle-media/competitions/The%20Learning%20Agency/BMGF_logo_black_300dpi%20(1).jpg)
+
+![Schmidt Futures Logo](https://storage.googleapis.com/kaggle-media/competitions/The%20Learning%20Agency/Schmidt%20Futures%20Logo.png)
+
+![CZI Logo](https://storage.googleapis.com/kaggle-media/competitions/The%20Learning%20Agency/1200px-Chan_Zuckerberg_Initiative.svg.png)
+
+## Evaluation
+
+Submissions are scored based on the quadratic weighted kappa, which measures the agreement between two outcomes. This metric typically varies from 0 (random agreement) to 1 (complete agreement). In the event that there is less agreement than expected by chance, the metric may go below 0 .
+
+The quadratic weighted kappa is calculated as follows. First, an $N \times N$ histogram matrix $O$ is constructed, such that $O_{i, j}$ corresponds to the number of essay_id s $i$ (actual) that received a predicted value $j$. An $N-b y-N$ matrix of weights, $w$, is calculated based on the difference between actual and predicted values:
+
+$$
+w_{i, j}=\frac{(i-j)^2}{(N-1)^2}
+$$
+
+An $N$-by- $N$ histogram matrix of expected outcomes, $E$, is calculated assuming that there is no correlation between values.
+
+This is calculated as the outer product between the actual histogram vector of outcomes and the predicted histogram vector, normalized such that $E$ and $O$ have the same sum.
+
+From these three matrices, the quadratic weighted kappa is calculated as:
+
+$$
+\kappa=1-\frac{\sum_{i, j} w_{i, j} O_{i, j}}{\sum_{i, j} w_{i, j} E_{i, j}}
+$$
+
+### Submission File
+
+For each `essay_id` in the test set, you must predict the corresponding `score` (described on the [Data](https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2/data) page). The file should contain a header and have the following format:
+
+```
+essay_id,score
+000d118,3
+000fe60,3
+001ab80,4
+...
+```
+
+## Timeline
+
+- **April 2, 2024** - Start Date.
+- **June 25, 2024** - Entry Deadline. You must accept the competition rules before this date in order to compete.
+- **June 25, 2024** - Team Merger Deadline. This is the last day participants may join or merge teams.
+- **July 2 2024** - Final Submission Deadline.
+
+All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
+
+## Prizes
+
+### Leaderboard Prizes
+
+- 1st Place - $ 12,000
+- 2nd Place - $ 8,000
+- 3rd Place - $ 5,000
+
+### Efficiency Prizes
+
+- 1st Place - $ 12,000
+- 2nd Place - $ 8,000
+- 3rd Place - $ 5,000
+
+Please see [**Efficiency Prize Evaluation**](https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2/overview/efficiency-prize-evaluation) for details on how the Efficiency Prize will be awarded. Winning a Leaderboard Prize does not preclude you from winning an Efficiency Prize.
+
+## Code Requirements
+
+### This is a Code Competition
+
+Submissions to this competition must be made through Notebooks. In order for the "Submit" button to be active after a commit, the following conditions must be met:
+
+- CPU Notebook <= 9 hours run-time
+- GPU Notebook <= 9 hours run-time
+- Internet access disabled
+- Freely & publicly available external data is allowed, including pre-trained models
+- Submission file must be named `submission.csv`
+
+Please see the [Code Competition FAQ](https://www.kaggle.com/docs/competitions#notebooks-only-FAQ) for more information on how to submit. And review the [code debugging doc](https://www.kaggle.com/code-competition-debugging) if you are encountering submission errors.
+
+## Efficiency Prize Evaluation
+
+### Efficiency Prize
+
+We are hosting a second track that focuses on model efficiency, because highly accurate models are often computationally heavy. Such models have a stronger carbon footprint and frequently prove difficult to utilize in real-world educational contexts. We hope to use these models to help educational organizations, which have limited computational capabilities.
+
+For the Efficiency Prize, we will evaluate submissions on both runtime and predictive performance.
+
+To be eligible for an Efficiency Prize, a submission:
+
+- Must be among the submissions selected by a team for the Leaderboard Prize, or else among those submissions automatically selected under the conditions described in the [**My Submissions**](https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2/submissions) tab.
+- Must be ranked on the Private Leaderboard higher than the `sample_submission.csv` benchmark.
+- Must not have a GPU enabled. **The Efficiency Prize is CPU Only.**
+
+All submissions meeting these conditions will be considered for the Efficiency Prize. A submission may be eligible for both the Leaderboard Prize and the Efficiency Prize.
+
+An Efficiency Prize will be awarded to eligible submissions according to how they are ranked by the following evaluation metric on the private test data. See the [**Prizes**](https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2/overview/prizes) tab for the prize awarded to each rank. More details may be posted via discussion forum updates.
+
+### Evaluation Metric
+
+We compute a submission's **efficiency score** by:
+
+$$
+\text { Efficiency }=\frac{\text { Kappa }}{\text { Base }- \text { max Kappa }}+\frac{\text { RuntimeSeconds }}{32400}
+$$
+
+where Kappa is the submission's score on the [main competition metric](https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2/overview/evaluation), BaseBase is the score of the "baseline" submission of the mean of each target on the training set, minKappaminKappa is the minimum Kappa of all submissions on the Private Leaderboard, and RuntimeSeconds is the number of seconds it takes for the submission to be evaluated. The objective is to minimize the efficiency score.
+
+During the training period of the competition, you may see a leaderboard for the *public* test data in the following notebook, updated daily: [**Efficiency Leaderboard**](https://www.kaggle.com/code/inversion/automated-essay-scoring-2-0-efficiency-lb). After the competition ends, we will update this leaderboard with efficiency scores on the *private* data. During the training period, this leaderboard will show only the rank of each team, but not the complete score.
+
+## Citation
+
+Scott Crossley, Perpetual Baffour, Jules King, Lauryn Burleigh, Walter Reade, Maggie Demkin. (2024). Learning Agency Lab - Automated Essay Scoring 2.0. Kaggle. https://kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2
+
+# Data
+
+## Dataset Description
+
+The competition dataset comprises about 24000 student-written argumentative essays. Each essay was scored on a scale of 1 to 6 ([Link to the Holistic Scoring Rubric](https://storage.googleapis.com/kaggle-forum-message-attachments/2733927/20538/Rubric_%20Holistic%20Essay%20Scoring.pdf)). Your goal is to predict the score an essay received from its text.
+
+### File and Field Information
+
+- **train.csv** - Essays and scores to be used as training data.
+    - `essay_id` - The unique ID of the essay
+    - `full_text` - The full essay response
+    - `score` - Holistic score of the essay on a 1-6 scale
+- **test.csv** - The essays to be used as test data. Contains the same fields as `train.csv`, aside from exclusion of `score`. (**Note**: The rerun test set has approximately 8k observations.)
+- **sample_submission.csv** - A submission file in the correct format.
+    - `essay_id` - The unique ID of the essay
+    - `score` - The predicted holistic score of the essay on a 1-6 scale
+
+Please note that this is a [**Code Competition**](https://www.kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2/overview/code-requirements).

dslighting/registry/learning-agency-lab-automated-essay-scoring-2/description_obfuscated.md

@@ -0,0 +1,30 @@
+# Task
+
+Predict the score of student essays.
+
+# Metric
+
+Quadratic weighted kappa.
+
+# Submission Format
+
+For each `essay_id` in the test set, you must predict the corresponding `score` (between 1-6, see [rubric](https://storage.googleapis.com/kaggle-forum-message-attachments/2733927/20538/Rubric_%20Holistic%20Essay%20Scoring.pdf) for more details). The file should contain a header and have the following format:
+
+```
+essay_id,score
+000d118,3
+000fe60,3
+001ab80,4
+...
+```
+
+# Dataset
+
+- **train.csv** - Essays and scores to be used as training data.
+    - `essay_id` - The unique ID of the essay
+    - `full_text` - The full essay response
+    - `score` - Holistic score of the essay on a 1-6 scale
+- **test.csv** - The essays to be used as test data. Contains the same fields as `train.csv`, aside from exclusion of `score`.
+- **sample_submission.csv** - A submission file in the correct format.
+    - `essay_id` - The unique ID of the essay
+    - `score` - The predicted holistic score of the essay on a 1-6 scale

dslighting/registry/learning-agency-lab-automated-essay-scoring-2/grade.py

@@ -0,0 +1,44 @@
+from pandas import DataFrame
+from sklearn.metrics import cohen_kappa_score
+
+from mlebench.grade_helpers import InvalidSubmissionError
+
+
+def prepare_for_grading(
+    submission: DataFrame, answers: DataFrame, id_column: str, target_column: str
+) -> tuple:
+
+    # Answers checks
+    assert (
+        target_column in answers.columns
+    ), f"Answers must contain the target column '{target_column}'"
+    assert id_column in answers.columns, f"Answers must contain the id column '{id_column}'"
+    assert not answers.empty, "Answers DataFrame should not be empty"
+
+    # Submission checks
+    if target_column not in submission.columns:
+        raise InvalidSubmissionError(f"Submission must contain the target column '{target_column}'")
+    if id_column not in submission.columns:
+        raise InvalidSubmissionError(f"Submission must contain the id column '{id_column}'")
+    if submission.empty:
+        raise InvalidSubmissionError("Submission DataFrame should not be empty")
+
+    # Sort the submissions and answers by id_column
+    submission_sorted = submission.sort_values(by=id_column)
+    answers_sorted = answers.sort_values(by=id_column)
+
+    if (submission_sorted[id_column].values != answers_sorted[id_column].values).any():
+        raise InvalidSubmissionError("Submission must contain the same ids as the answers")
+
+    # Extract the target columns
+    y_pred = submission_sorted[target_column].values
+    y_true = answers_sorted[target_column].values
+
+    return y_pred, y_true
+
+
+def grade(submission: DataFrame, answers: DataFrame) -> float:
+    y_pred, y_true = prepare_for_grading(
+        submission=submission, answers=answers, id_column="essay_id", target_column="score"
+    )
+    return cohen_kappa_score(y_pred, y_true, weights="quadratic")

dslighting/registry/learning-agency-lab-automated-essay-scoring-2/prepare.py

@@ -0,0 +1,51 @@
+from pathlib import Path
+
+import numpy as np
+from sklearn.model_selection import train_test_split
+
+from mlebench.utils import read_csv
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    """
+    Splits the data in raw into public and private datasets with appropriate test/train splits.
+    """
+
+    # Create train, test from original train split
+    old_train = read_csv(raw / "train.csv")
+    # Original train has 17307 rows. Original hidden test has approx 8k rows. We just take 10% of the original train as the test set.
+    new_train, answers = train_test_split(old_train, test_size=0.1, random_state=0)
+    new_test = answers.drop(columns=["score"])
+
+    sample_submission = answers[["essay_id"]].copy()
+    sample_submission["score"] = np.random.RandomState(42).randint(
+        1, 7, size=len(sample_submission)
+    )
+
+    # Checks
+    assert set(new_train["essay_id"]).isdisjoint(
+        set(new_test["essay_id"])
+    ), "Essay IDs in train and test sets are not disjoint"
+    assert len(new_train) + len(new_test) == len(
+        old_train
+    ), f"Train and test sets do not sum to original train set"
+    assert len(new_test) == len(
+        sample_submission
+    ), f"Test and sample submission sets do not have the same length"
+    assert (
+        new_train.columns.tolist() == old_train.columns.tolist()
+    ), f"Train set columns do not match original train set, got {new_train.columns.tolist()}"
+    assert new_test.columns.tolist() == [
+        "essay_id",
+        "full_text",
+    ], f"Test set columns do not match expected columns, got {new_test.columns.tolist()}"
+    assert sample_submission.columns.tolist() == [
+        "essay_id",
+        "score",
+    ], f"Sample submission set columns do not match expected columns, got {sample_submission.columns.tolist()}"
+
+    # Write CSVs
+    answers.to_csv(private / "answers.csv", index=False)
+    new_train.to_csv(public / "train.csv", index=False)
+    new_test.to_csv(public / "test.csv", index=False)
+    sample_submission.to_csv(public / "sample_submission.csv", index=False)

dslighting/registry/learning-agency-lab-automated-essay-scoring-2/prepare_val.py

@@ -0,0 +1,96 @@
+from pathlib import Path
+from typing import Tuple
+
+import numpy as np
+import pandas as pd
+from sklearn.model_selection import train_test_split
+
+from mlebench.utils import read_csv
+
+
+def _create_split(
+    input_df: pd.DataFrame, test_size: float, random_state: int
+) -> Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame, pd.DataFrame]:
+    """Helper function to perform a single data split and generate associated files."""
+    # Create train, test from the input dataframe
+    train_df, answers_df = train_test_split(
+        input_df, test_size=test_size, random_state=random_state
+    )
+    test_df = answers_df.drop(columns=["score"])
+
+    sample_submission_df = answers_df[["essay_id"]].copy()
+    sample_submission_df["score"] = np.random.RandomState(42).randint(
+        1, 7, size=len(sample_submission_df)
+    )
+
+    # Checks
+    assert set(train_df["essay_id"]).isdisjoint(
+        set(test_df["essay_id"])
+    ), "Essay IDs in train and test sets are not disjoint"
+    assert len(train_df) + len(test_df) == len(
+        input_df
+    ), f"Train and test sets do not sum to original train set"
+    assert len(test_df) == len(
+        sample_submission_df
+    ), f"Test and sample submission sets do not have the same length"
+    assert (
+        train_df.columns.tolist() == input_df.columns.tolist()
+    ), f"Train set columns do not match original train set, got {train_df.columns.tolist()}"
+    assert test_df.columns.tolist() == [
+        "essay_id",
+        "full_text",
+    ], f"Test set columns do not match expected columns, got {test_df.columns.tolist()}"
+    assert sample_submission_df.columns.tolist() == [
+        "essay_id",
+        "score",
+    ], f"Sample submission set columns do not match expected columns, got {sample_submission_df.columns.tolist()}"
+
+    return train_df, test_df, answers_df, sample_submission_df
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    """
+    Splits the data in raw into public and private datasets with appropriate test/train splits.
+    Also creates a second, parallel split for validation purposes.
+    """
+
+    # Read the original raw data
+    old_train = read_csv(raw / "train.csv")
+
+    # --- Stage 1: Create the original train/test split for the main competition ---
+    # This block produces the exact same output as the original script.
+
+    # Original train has 17307 rows. Original hidden test has approx 8k rows. We just take 10% of the original train as the test set.
+    main_train, main_test, main_answers, main_sample_submission = _create_split(
+        input_df=old_train, test_size=0.1, random_state=0
+    )
+
+    # Write original CSVs to public/ and private/
+    main_answers.to_csv(private / "answers.csv", index=False)
+    main_train.to_csv(public / "train.csv", index=False)
+    main_test.to_csv(public / "test.csv", index=False)
+    main_sample_submission.to_csv(public / "sample_submission.csv", index=False)
+
+    # --- Stage 2: Create a new validation split from the main training data ---
+    # This block creates a new set of directories and files for validation.
+
+    # Define and create the new parallel directories for the validation set
+    public_val = public.parent / "public_val"
+    private_val = private.parent / "private_val"
+    public_val.mkdir(parents=True, exist_ok=True)
+    private_val.mkdir(parents=True, exist_ok=True)
+
+    # Calculate the test size for the second split to make the validation test set
+    # have the same number of samples as the original test set.
+    val_test_size = len(main_test) / len(main_train)
+
+    # Create the new split using the main training data as input
+    val_train, val_test, val_answers, val_sample_submission = _create_split(
+        input_df=main_train, test_size=val_test_size, random_state=0
+    )
+
+    # Write validation CSVs to public_val/ and private_val/ using identical filenames
+    val_answers.to_csv(private_val / "answers.csv", index=False)
+    val_train.to_csv(public_val / "train.csv", index=False)
+    val_test.to_csv(public_val / "test.csv", index=False)
+    val_sample_submission.to_csv(public_val / "sample_submission.csv", index=False)

dslighting/registry/liverpool-ion-switching/config.yaml

@@ -0,0 +1,15 @@
+id: liverpool-ion-switching
+name: Liverpool Ion Switching
+competition_type: simple
+awards_medals: false
+description: mlebench/competitions/liverpool-ion-switching/description.md
+
+dataset:
+  answers: liverpool-ion-switching/prepared/private/test.csv
+  sample_submission: liverpool-ion-switching/prepared/public/sample_submission.csv
+
+grader:
+  name: macro-f1
+  grade_fn: mlebench.competitions.liverpool_ion_switching.grade:grade
+
+preparer: mlebench.competitions.liverpool_ion_switching.prepare:prepare

dslighting/registry/liverpool-ion-switching/description.md

@@ -0,0 +1,44 @@
+# Liverpool Ion Switching
+
+## Description
+
+Ion channels are pore-forming proteins present in animals and plants that encode learning and memory, help fight infections, enable pain signals, and stimulate muscle contraction. When ion channels open, they pass electric currents. Scientists hope that machine learning can enable rapid automatic detection of ion channel current events in raw data.
+
+In this competition, you'll use ion channel data to better model automatic identification methods. If successful, you'll be able to detect individual ion channel events in noisy raw signals.
+
+## Evaluation
+
+Submissions are evaluated using the **macro F1 score**.
+
+F1 is calculated as follows:
+
+$$F_1 = 2 \times \frac{\text{precision} \times \text{recall}}{\text{precision} + \text{recall}}$$
+
+where:
+
+$$\text{precision} = \frac{TP}{TP + FP}, \quad \text{recall} = \frac{TP}{TP + FN}$$
+
+In "macro" F1, a separate F1 score is calculated for each `open_channels` value and then averaged.
+
+## Submission Format
+
+For each time value in the test set, you must predict `open_channels`. The file must have a header:
+
+```
+time,open_channels
+500.0000,0
+500.0001,2
+...
+```
+
+## Dataset Description
+
+In this competition, you will be predicting the number of open channels present, based on electrophysiological signal data.
+
+**IMPORTANT**: While the time series appears continuous, the data is from discrete batches of 50 seconds long 10 kHz samples (500,000 rows per batch). The data from 0.0001-50.0000 is a different batch than 50.0001-100.0000, and thus discontinuous between 50.0000 and 50.0001.
+
+## Files
+
+- `train.csv`: Training set
+- `test.csv`: Test set
+- `sample_submission.csv`: Sample submission in correct format

dslighting/registry/liverpool-ion-switching/grade.py

@@ -0,0 +1,52 @@
+import numpy as np
+import pandas as pd
+from sklearn.metrics import f1_score
+
+from mlebench.grade_helpers import InvalidSubmissionError
+
+
+def prepare_for_metric(submission: pd.DataFrame, answers: pd.DataFrame):
+    """Prepare submission and answers for Macro F1 calculation."""
+    if len(submission) != len(answers):
+        raise InvalidSubmissionError(
+            f"Submission length ({len(submission)}) != answers length ({len(answers)})"
+        )
+
+    if "time" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have a 'time' column")
+
+    if "open_channels" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have an 'open_channels' column")
+
+    # Sort by time
+    submission = submission.sort_values("time").reset_index(drop=True)
+    answers = answers.sort_values("time").reset_index(drop=True)
+
+    # Check time alignment
+    if not np.allclose(submission["time"].values, answers["time"].values, atol=1e-4):
+        raise InvalidSubmissionError("Submission and answers time mismatch")
+
+    # Extract predictions and true values
+    y_pred = submission["open_channels"].values
+    y_true = answers["open_channels"].values
+
+    # Validate predictions
+    try:
+        y_pred = y_pred.astype(int)
+    except (ValueError, TypeError):
+        raise InvalidSubmissionError("Predictions must be integers")
+
+    if np.any(np.isnan(y_pred)):
+        raise InvalidSubmissionError("Predictions cannot contain NaN values")
+
+    return y_true, y_pred
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    """Calculate Macro F1 score."""
+    y_true, y_pred = prepare_for_metric(submission, answers)
+
+    # Macro F1: compute F1 for each class and average
+    macro_f1 = f1_score(y_true, y_pred, average='macro')
+
+    return macro_f1

dslighting/registry/liverpool-ion-switching/prepare.py

@@ -0,0 +1,27 @@
+from pathlib import Path
+import pandas as pd
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    """
+    Prepare liverpool-ion-switching dataset.
+    """
+    # Read data
+    train = pd.read_csv(raw / "train.csv")
+    test = pd.read_csv(raw / "test.csv")
+    sample_submission = pd.read_csv(raw / "sample_submission.csv")
+    test_answer = pd.read_csv(raw / "test_answer.csv")
+
+    # Public files (visible to agents)
+    train.to_csv(public / "train.csv", index=False)
+    test.to_csv(public / "test.csv", index=False)
+    sample_submission.to_csv(public / "sample_submission.csv", index=False)
+
+    # Private files (for grading)
+    test_answer.to_csv(private / "test.csv", index=False)
+
+    # Validation checks
+    assert len(test_answer) == len(sample_submission), \
+        f"Test answer ({len(test_answer)}) and sample submission ({len(sample_submission)}) must have same length"
+    assert "time" in test_answer.columns, "Test answer must have 'time' column"
+    assert "open_channels" in test_answer.columns, "Test answer must have 'open_channels' column"

dslighting/registry/liverpool_ion_switching/grade.py

@@ -0,0 +1,52 @@
+import numpy as np
+import pandas as pd
+from sklearn.metrics import f1_score
+
+from mlebench.grade_helpers import InvalidSubmissionError
+
+
+def prepare_for_metric(submission: pd.DataFrame, answers: pd.DataFrame):
+    """Prepare submission and answers for Macro F1 calculation."""
+    if len(submission) != len(answers):
+        raise InvalidSubmissionError(
+            f"Submission length ({len(submission)}) != answers length ({len(answers)})"
+        )
+
+    if "time" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have a 'time' column")
+
+    if "open_channels" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have an 'open_channels' column")
+
+    # Sort by time
+    submission = submission.sort_values("time").reset_index(drop=True)
+    answers = answers.sort_values("time").reset_index(drop=True)
+
+    # Check time alignment
+    if not np.allclose(submission["time"].values, answers["time"].values, atol=1e-4):
+        raise InvalidSubmissionError("Submission and answers time mismatch")
+
+    # Extract predictions and true values
+    y_pred = submission["open_channels"].values
+    y_true = answers["open_channels"].values
+
+    # Validate predictions
+    try:
+        y_pred = y_pred.astype(int)
+    except (ValueError, TypeError):
+        raise InvalidSubmissionError("Predictions must be integers")
+
+    if np.any(np.isnan(y_pred)):
+        raise InvalidSubmissionError("Predictions cannot contain NaN values")
+
+    return y_true, y_pred
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    """Calculate Macro F1 score."""
+    y_true, y_pred = prepare_for_metric(submission, answers)
+
+    # Macro F1: compute F1 for each class and average
+    macro_f1 = f1_score(y_true, y_pred, average='macro')
+
+    return macro_f1

dslighting/registry/liverpool_ion_switching/prepare.py

@@ -0,0 +1,27 @@
+from pathlib import Path
+import pandas as pd
+
+
+def prepare(raw: Path, public: Path, private: Path):
+    """
+    Prepare liverpool-ion-switching dataset.
+    """
+    # Read data
+    train = pd.read_csv(raw / "train.csv")
+    test = pd.read_csv(raw / "test.csv")
+    sample_submission = pd.read_csv(raw / "sample_submission.csv")
+    test_answer = pd.read_csv(raw / "test_answer.csv")
+
+    # Public files (visible to agents)
+    train.to_csv(public / "train.csv", index=False)
+    test.to_csv(public / "test.csv", index=False)
+    sample_submission.to_csv(public / "sample_submission.csv", index=False)
+
+    # Private files (for grading)
+    test_answer.to_csv(private / "test.csv", index=False)
+
+    # Validation checks
+    assert len(test_answer) == len(sample_submission), \
+        f"Test answer ({len(test_answer)}) and sample submission ({len(sample_submission)}) must have same length"
+    assert "time" in test_answer.columns, "Test answer must have 'time' column"
+    assert "open_channels" in test_answer.columns, "Test answer must have 'open_channels' column"

dslighting/registry/lmsys-chatbot-arena/checksums.yaml

@@ -0,0 +1,7 @@
+private:
+  answers.csv: 43ef7dc300c731231796afc046cf4230
+public:
+  sample_submission.csv: e67f89b25eb382df62d821e72bed8d4b
+  test.csv: fa5d6d46adf7cfa5d7f7dc3ef3e358c8
+  train.csv: e11dec14f6fc839a97fb21abcaebdaa4
+zip: 73e311b0e622c3f82fb22c7ee24ae91e

dslighting/registry/lmsys-chatbot-arena/config.yaml

@@ -0,0 +1,26 @@
+id: lmsys-chatbot-arena
+name: LMSYS - Chatbot Arena Human Preference Predictions
+competition_type: code
+awards_medals: true
+prizes:
+  - position: 1
+    value: 25000
+  - position: 2
+    value: 20000
+  - position: 3
+    value: 20000
+  - position: 4
+    value: 20000
+  - position: 5
+    value: 15000
+description: mlebench/competitions/lmsys-chatbot-arena/description.md
+
+dataset:
+  answers: lmsys-chatbot-arena/prepared/private/answers.csv
+  sample_submission: lmsys-chatbot-arena/prepared/public/sample_submission.csv
+
+grader:
+  name: multi-class-log-loss
+  grade_fn: mlebench.competitions.lmsys-chatbot-arena.grade:grade
+
+preparer: mlebench.competitions.lmsys-chatbot-arena.prepare:prepare