dslighting 1.3.9__py3-none-any.whl → 1.7.6__py3-none-any.whl

dslighting/registry/3d-object-detection-for-autonomous-vehicles/checksums.yaml

@@ -0,0 +1,6 @@
+private:
+  test.csv: 1972edf2739f010c260822761b94b1df
+public:
+  sample_submission.csv: 4057929857ceb634ecb3875f93ac1688
+  train.csv: 687b1288965d6efa35ea29c3adff5557
+zip: 48824e3ca27955e71bf0c30938e560f7

dslighting/registry/3d-object-detection-for-autonomous-vehicles/config.yaml

@@ -0,0 +1,22 @@
+id: 3d-object-detection-for-autonomous-vehicles
+name: Lyft 3D Object Detection for Autonomous Vehicles
+competition_type: simple
+awards_medals: true
+prizes:
+  - position: 1
+    value: 1200
+  - position: 2
+    value: 8000
+  - position: 3
+    value: 5000
+description: mlebench/competitions/3d-object-detection-for-autonomous-vehicles/description.md
+
+dataset:
+  answers: 3d-object-detection-for-autonomous-vehicles/prepared/private/test.csv
+  sample_submission: 3d-object-detection-for-autonomous-vehicles/prepared/public/sample_submission.csv
+
+grader:
+  name: mean-average-precision
+  grade_fn: mlebench.competitions.3d-object-detection-for-autonomous-vehicles.grade:grade
+
+preparer: mlebench.competitions.3d-object-detection-for-autonomous-vehicles.prepare:prepare

dslighting/registry/3d-object-detection-for-autonomous-vehicles/description.md

@@ -0,0 +1,157 @@
+# Overview
+
+## Description
+
+![autonomous-vehicle-lidar](https://storage.googleapis.com/kaggle-media/competitions/Lyft-Kaggle/Kaggle-01.png)
+
+Self-driving technology presents a rare opportunity to improve the quality of life in many of our communities. Avoidable collisions, single-occupant commuters, and vehicle emissions are choking cities, while infrastructure strains under rapid urban growth. Autonomous vehicles are expected to redefine transportation and unlock a myriad of societal, environmental, and economic benefits. You can apply your data analysis skills in this competition to advance the state of self-driving technology.
+
+[Lyft](https://level5.lyft.com/), whose mission is to improve people's lives with the world's best transportation, is investing in the future of self-driving vehicles. Level 5, their self-driving division, is working on a fleet of autonomous vehicles, and currently has a team of 450+ across Palo Alto, London, and Munich working to build a leading self-driving system ([they're hiring!](https://level5.lyft.com/#joinourteam)). Their goal is to democratize access to self-driving technology for hundreds of millions of Lyft passengers.
+
+From a technical standpoint, however, the bar to unlock technical research and development on higher-level autonomy functions like perception, prediction, and planning is extremely high. This implies technical R&D on self-driving cars has traditionally been inaccessible to the broader research community.
+
+This dataset aims to democratize access to such data, and foster innovation in higher-level autonomy functions for everyone, everywhere. By conducting a competition, we hope to encourage the research community to focus on hard problems in this space---namely, 3D object detection over semantic maps.
+
+In this competition, you will build and optimize algorithms based on a large-scale dataset. This dataset features the raw sensor camera inputs as perceived by a fleet of multiple, high-end, autonomous vehicles in a restricted geographic area.
+
+If successful, you'll make a significant contribution towards stimulating further development in autonomous vehicles and empowering communities around the world.
+
+## Evaluation
+
+This competition is evaluated on the mean average precision at different intersection over union (IoU) thresholds. The IoU of a set of predicted 3D bounding volumes and ground truth bounding volumes is calculated as:
+
+$\operatorname{IoU}(A, B)=\frac{A \cap B}{A \cup B}.$
+
+The metric sweeps over a range of IoU thresholds, at each point calculating an average precision value. The threshold values range from 0.5 to 0.95 with a step size of 0.05: `(0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95)`. In other words, at a threshold of 0.5, a predicted object is considered a "hit" if its intersection over union with a ground truth object is greater than 0.5.
+
+At each threshold value 𝑡t, a precision value is calculated based on the number of true positives (TP), false negatives (FN), and false positives (FP) resulting from comparing the predicted object to all ground truth objects:
+
+$\frac{T P(t)}{T P(t)+F P(t)+F N(t)}.$
+
+A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object.
+
+Important note: if there are no ground truth objects at all for a given image, ANY number of predictions (false positives) will result in the image receiving a score of zero, and being included in the mean average precision.
+
+The average precision of a single image is calculated as the mean of the above precision values at each IoU threshold:
+
+$\frac{1}{\mid \text { thresholds } \mid} \sum_t \frac{T P(t)}{T P(t)+F P(t)+F N(t)}.$
+
+In your submission, you are also asked to provide a `confidence` level for each bounding box. Bounding boxes will be evaluated in order of their confidence levels in the above process. This means that bounding boxes with higher confidence will be checked first for matches against solutions, which determines what boxes are considered true and false positives.
+
+NOTE: In nearly all cases `confidence` will have no impact on scoring. It exists primarily to allow for submission boxes to be evaluated in a particular order to resolve extreme edge cases. None of these edge cases are known to exist in the data set. If you do not wish to use or calculate `confidence` you can use a placeholder value - like `1.0` - to indicate that no particular order applies to the evaluation of your submission boxes.
+
+Lastly, the score returned by the competition metric is the mean taken over the individual average precisions of each image in the test dataset.
+
+### Intersection over Union (IoU)
+
+Intersection over Union is a measure of the magnitude of overlap between two bounding boxes (or, in the more general case, two objects). It calculates the size of the overlap between two objects, divided by the total area of the two objects combined.
+
+It can be visualized as the following:
+
+![Image of Intersection over Union](https://storage.googleapis.com/kaggle-media/competitions/rsna/IoU.jpg)
+
+The two boxes in the visualization overlap, but the area of the overlap is insubstantial compared with the area taken up by both objects together. IoU would be low - and would likely not count as a "hit" at higher IoU thresholds.
+
+### 3D Context
+
+The difference between the 2D and 3D bounding volume contexts is small. In the 3D context we reduce the bounding volume to a `ground bounding box` and a `height`. The IoU is then the `intersection` of the `ground bounding boxes` * the `intersection` of the `height` differences, divided by the union of the bounding boxes.
+
+### Submission File
+
+The submission format requires a space delimited set of bounding volume parameters. For example:
+
+`97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c,1.0 2742.15 673.16 -18.65 1.834 4.609 1.648 2.619 car`
+
+indicates that sample `97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c` has a bounding volume with a `confidence` of 0.5, `center_x` of 2742.15, `center_y` of 673.16, `center_z` of -18.65, `width` of 1.834, `length` of 4.609, `height` of 1.648, `yaw` of 2.619, and a `class_name` of `car`.
+
+The file should contain a header and have the following format. Each row in your submission should contain ALL bounding boxes for a given image.
+
+```
+Id,PredictionString
+db8b47bd4ebdf3b3fb21598bb41bd8853d12f8d2ef25ce76edd4af4d04e49341,
+97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c,1.0 2742.15 673.16 -18.65 1.834 4.609 1.648 2.619 car
+etc...
+```
+
+## Timeline
+
+-   November 5, 2019 - Entry deadline. You must accept the competition rules before this date in order to compete.
+
+-   November 5, 2019 - Team Merger deadline. This is the last day participants may join or merge teams.
+
+-   November 12, 2019 - Final submission deadline.
+
+All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
+
+## Prizes
+
+Participants with the best score on the private leaderboard are eligible to receive:
+
+-   1st Place - $12,000
+-   2nd Place - $8,000
+-   3rd Place - $5,000
+
+Additional Awards and Opportunities\
+One winner from the top three winning teams will be invited to present their results at our competition track at [NeurIPS 2019](https://neurips.cc/), including limited travel support. This winner will also be welcome to request a complimentary pass to NeurIPS 2019.
+
+Top participants in this competition who are attending NeurIPS 2019 in Vancouver, Canada, will be invited to Lyft's competition party. Watch the forum for more details.
+
+Lyft Level 5 is hiring! Members of top performing teams will be reviewed by our recruiting team for the chance to interview with Level 5. In the meantime, check out our [open roles](https://level5.lyft.com/#joinourteam).
+
+## NeurIPS 2019
+
+This Kaggle competition is part of Lyft Level 5's competition track that will be held at [NeurIPS 2019](https://neurips.cc/). Level 5 is returning for the second year to sponsor the conference, where you will have a chance to meet the team at their booth, or at their competition track. We will update the discussion forum with our NeurIPS booth and track location.
+
+By attending the competition track, you'll get the chance to hear the winners of this Kaggle competition present their results. Attending the competition track is not required to participate in the Kaggle competition; however, winning participants will be asked to present their work.
+
+NeurIPS is an annual academic conference in machine learning and computational science that attracts 8,000+ attendees. This year's conference will be held in Vancouver, Canada from December 8-14, 2019. We hope to see you there!
+
+## Citation
+
+Christy, Maggie, NikiNikatos, Phil Culliton, Vinay Shet, Vladimir Iglovikov. (2019). Lyft 3D Object Detection for Autonomous Vehicles. Kaggle. https://kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles
+
+
+# Data
+
+## Dataset Description
+
+### What files do I need?
+
+You will need the LIDAR, image, map and data files for both train and test (`test_images.zip`, `test_lidar.zip`, etc.). You may also need the `train.csv`, which includes the sample annotations in the form expected for submissions. The `sample_submission.csv` file contains all of the sample `Id`s for the test set.
+
+The data files (`test_data.zip`, `train_data.zip`) are in JSON format.
+
+### What should I expect the data format to be?
+
+The data comes in the form of many interlocking tables and formats. The JSON files all contain single tables with identifying `tokens` that can be used to join with other files / tables. The images and lidar files all correspond to a sample in `sample_data.json`, and the `sample_token` from `sample_data.json` is the primary identifier used for the train and test samples.
+
+The annotations in `train.csv` are in the following format:\
+`center_x center_y center_z width length height yaw class_name`
+
+-   `center_x`, `center_y` and `center_z` are the world coordinates of the center of the 3D bounding volume.
+-   `width`, `length` and `height` are the dimensions of the volume.
+-   `yaw` is the angle of the volume around the `z` axis (where `y` is forward/back, `x` is left/right, and `z` is up/down - making 'yaw' the direction the front of the vehicle / bounding box is pointing at while on the ground).
+-   `class_name` is the type of object contained by the bounding volume.
+
+### What am I predicting?
+
+For the test samples, we're predicting the bounding volumes and classes of all of the objects in a given scene.
+
+For example, in `sample_token` `97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c`, we might be predicting the presence of a car (with `confidence` `1.0`) and a bus (with `confidence` `0.5`). The prediction would look like this:
+
+```
+97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c,1.0 2742.152625996093 673.1631800662494 -18.6561112411676 1.834 4.609 1.648 2.619835541569646 car 0.5 2728.9634555684484 657.8296521874645 -18.54676216218047 1.799 4.348 1.728 -0.5425527100619654 bus
+```
+
+With all predicted objects on the same line.
+
+Note that `confidence` values are inserted prior to `center_x center_y center_z width length height yaw class_name`.
+
+### Files
+
+-   **train_data.zip** and **test_data.zip** - contains JSON files with multiple tables. The most important is `sample_data.json`, which contains the primary identifiers used in the competition, as well as links to key image / lidar information.
+-   **train_images.zip** and **test_images.zip** - contains .jpeg files corresponding to samples in `sample_data.json`
+-   **train_lidar.zip** and **test_lidar.zip** - contains .jpeg files corresponding to samples in `sample_data.json`
+-   **train_maps.zip** and **test_maps.zip** - contains maps of the entire sample area.
+-   **train.csv** - contains all `sample_token`s in the train set, as well as annotations in the required format for all train set objects.
+-   **sample_submission.csv** - contains all `sample_token`s in the test set, with empty predictions.

dslighting/registry/3d-object-detection-for-autonomous-vehicles/description_obfuscated.md

@@ -0,0 +1,66 @@
+# Task
+
+Build a 3D object detection model.
+
+
+# Metric
+
+Mean average precision at different intersection over union (IoU) thresholds. The IoU of a set of predicted 3D bounding volumes and ground truth bounding volumes is calculated as:
+
+$\operatorname{IoU}(A, B)=\frac{A \cap B}{A \cup B}.$
+
+The metric sweeps over a range of IoU thresholds, at each point calculating an average precision value. The threshold values range from 0.5 to 0.95 with a step size of 0.05: `(0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95)`. At a threshold of 0.5, a predicted object is considered a "hit" if its intersection over union with a ground truth object is greater than 0.5.
+
+At each threshold value 𝑡t, a precision value is calculated based on the number of true positives (TP), false negatives (FN), and false positives (FP) resulting from comparing the predicted object to all ground truth objects:
+
+$\frac{T P(t)}{T P(t)+F P(t)+F N(t)}.$
+
+A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object.
+
+If there are no ground truth objects at all for a given image, ANY number of predictions (false positives) will result in the image receiving a score of zero, and being included in the mean average precision.
+
+The average precision of a single image is calculated as the mean of the above precision values at each IoU threshold:
+
+$\frac{1}{\mid \text { thresholds } \mid} \sum_t \frac{T P(t)}{T P(t)+F P(t)+F N(t)}.$
+
+In your submission, you are also asked to provide a `confidence` level for each bounding box. Bounding boxes will be evaluated in order of their confidence levels in the above process. This means that bounding boxes with higher confidence will be checked first for matches against solutions, which determines what boxes are considered true and false positives.
+
+If you do not wish to use or calculate `confidence` you can use a placeholder value - like `1.0` - to indicate that no particular order applies to the evaluation of your submission boxes.
+
+The difference between the 2D and 3D bounding volume contexts is small. In the 3D context we reduce the bounding volume to a `ground bounding box` and a `height`. The IoU is then the `intersection` of the `ground bounding boxes` * the `intersection` of the `height` differences, divided by the union of the bounding boxes.
+
+# Submission Format
+
+The submission format requires a space delimited set of bounding volume parameters. For example:
+
+`97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c,1.0 2742.15 673.16 -18.65 1.834 4.609 1.648 2.619 car`
+
+indicates that sample `97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c` has a bounding volume with a `confidence` of 0.5, `center_x` of 2742.15, `center_y` of 673.16, `center_z` of -18.65, `width` of 1.834, `length` of 4.609, `height` of 1.648, `yaw` of 2.619, and a `class_name` of `car`.
+
+The file should contain a header and have the following format. Each row in your submission should contain ALL bounding boxes for a given image.
+
+```
+Id,PredictionString
+db8b47bd4ebdf3b3fb21598bb41bd8853d12f8d2ef25ce76edd4af4d04e49341,
+97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c,1.0 2742.15 673.16 -18.65 1.834 4.609 1.648 2.619 car
+etc...
+```
+
+# Dataset
+
+The data comes in the form of many interlocking tables and formats. The JSON files all contain single tables with identifying `tokens` that can be used to join with other files / tables. The images and lidar files all correspond to a sample in `sample_data.json`, and the `sample_token` from `sample_data.json` is the primary identifier used for the train and test samples.
+
+The annotations in `train.csv` are in the following format:\
+`center_x center_y center_z width length height yaw class_name`
+
+-   `center_x`, `center_y` and `center_z` are the world coordinates of the center of the 3D bounding volume.
+-   `width`, `length` and `height` are the dimensions of the volume.
+-   `yaw` is the angle of the volume around the `z` axis (where `y` is forward/back, `x` is left/right, and `z` is up/down - making 'yaw' the direction the front of the vehicle / bounding box is pointing at while on the ground).
+-   `class_name` is the type of object contained by the bounding volume.
+
+-   **train_data.zip** and **test_data.zip** - contains JSON files with multiple tables. The most important is `sample_data.json`, which contains the primary identifiers used in the competition, as well as links to key image / lidar information.
+-   **train_images.zip** and **test_images.zip** - contains .jpeg files corresponding to samples in `sample_data.json`
+-   **train_lidar.zip** and **test_lidar.zip** - contains .jpeg files corresponding to samples in `sample_data.json`
+-   **train_maps.zip** and **test_maps.zip** - contains maps of the entire sample area.
+-   **train.csv** - contains all `sample_token`s in the train set, as well as annotations in the required format for all train set objects.
+-   **sample_submission.csv** - contains all `sample_token`s in the test set, with empty predictions.

dslighting/registry/3d-object-detection-for-autonomous-vehicles/grade.py

@@ -0,0 +1,161 @@
+import multiprocessing
+from functools import partial
+from typing import Any, Dict, List
+
+import numpy as np
+import pandas as pd
+from pyquaternion import Quaternion
+from tqdm import tqdm
+
+from mlebench.grade_helpers import InvalidSubmissionError
+
+from .mAP_evaluation import get_average_precisions, get_class_names
+
+
+def parse_row(sample_token: str, prediction_string: str) -> List[Dict[str, Any]]:
+    """
+    Parse a row comprising of a sample_token and a prediction_string
+
+    `prediction_string` has the form:
+        `{confidence} center_x center_y center_z width length height yaw class_name`
+        where `confidence` is optional
+
+        Example row:
+        ```
+        97ce3ab08ccbc0baae0267cbf8d4da947e1f11ae1dbcb80c3f4408784cd9170c,1.0 2742.152625996093 673.1631800662494 -18.6561112411676 1.834 4.609 1.648 2.619835541569646 car 0.5 2728.9634555684484 657.8296521874645 -18.54676216218047 1.799 4.348 1.728 -0.5425527100619654 bus
+        ```
+
+    Returns list of dicts with keys "sample_token", "translation", "size", "rotation", "name", "score", e.g.
+    ```
+    [{
+        'sample_token': '0f0e3ce89d2324d8b45aa55a7b4f8207fbb039a550991a5149214f98cec136ac',
+        'translation': [971.8343488872263, 1713.6816097857359, -25.82534357061308],
+        'size': [2.519726579986132, 7.810161372666739, 3.483438286096803],
+        'rotation': [0.10913582721095375, 0.04099572636992043, 0.01927712319721745, 1.029328402625659],
+        'name': 'car',
+        'score': 0.3077029437237213
+    }]
+    ```
+    """
+    if prediction_string.strip() == "":
+        return []
+    prediction_string = prediction_string.split(" ")
+
+    single_prediction_length = 9
+    if len(prediction_string) % single_prediction_length != 0:
+        raise InvalidSubmissionError(
+            f"Prediction string must have {single_prediction_length} tokens per object, but got {len(prediction_string)} tokens"
+        )
+
+    n_predictions = len(prediction_string) / single_prediction_length
+    preds = []
+    for i in range(int(n_predictions)):
+        prediction_segment = prediction_string[
+            i * single_prediction_length : (i + 1) * single_prediction_length
+        ]
+        confidence = float(prediction_segment[0])
+        prediction_segment = prediction_segment[1:]
+
+        # Parse the prediction string
+        translation = [float(x) for x in prediction_segment[0:3]]
+        size = [float(x) for x in prediction_segment[3:6]]
+        yaw = float(prediction_segment[6])
+        name = prediction_segment[7]
+
+        quat = Quaternion(axis=[0, 0, 1], radians=yaw)
+        # (from https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/discussion/110884#638390)
+
+        preds.append(
+            {
+                "sample_token": sample_token,
+                "translation": translation,
+                "size": size,
+                "rotation": list(quat),
+                "name": name,
+                "score": confidence,
+            }
+        )
+    return preds
+
+
+def process_sample(gt_pred_pair, iou_thresholds):
+    gt, predictions = gt_pred_pair
+    class_names = get_class_names(gt)
+    sample_mAPs = []
+    for iou_threshold in iou_thresholds:
+        average_precisions = get_average_precisions(gt, predictions, class_names, iou_threshold)
+        mAP = np.mean(average_precisions)
+        sample_mAPs.append(mAP)
+    return sample_mAPs
+
+
+def grade(submission: pd.DataFrame, answers: pd.DataFrame) -> float:
+    """
+    submission and answers are dataframes with columns `Id` and `PredictionString`
+
+    We grade the submission using the 3D mAP metric described in the competition overview:
+    https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/overview/evaluation
+
+    The competition organizers provide a Python script (mAP_evaluation.py) to perform this metric calculation
+    (search text for "3d version of the COCO mAP") for a (gt, predictions) pair from a single sample.
+    https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/discussion/133895
+
+    For each sample, we first parse the PredictionString in the format expected by mAP_evaluation.py, then calculate
+    the mAP over the requested range of IoU thresholds, and get a mean mAP over all thresholds for that sample.
+
+    The final score is the mean over all samples (i.e. the mean of the mean mAPs).
+    """
+    if len(submission) != len(answers):
+        raise InvalidSubmissionError("Submission and answers must have the same length")
+
+    if "Id" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have an 'Id' column")
+
+    if "PredictionString" not in submission.columns:
+        raise InvalidSubmissionError("Submission must have a 'PredictionString' column")
+
+    assert "Id" in answers.columns, "Answers must have an 'Id' column"
+    assert "PredictionString" in answers.columns, "Answers must have a 'PredictionString' column"
+
+    submission = submission.sort_values("Id")
+    answers = answers.sort_values("Id")
+
+    if (submission["Id"].values != answers["Id"].values).any():
+        raise InvalidSubmissionError("Submission and answers must have the same ids")
+
+    # Empty values in the PredictionString column are allowed, but must be filled with an empty string
+    # (pandas converts them to NaN, so we convert them back)
+    submission["PredictionString"] = submission["PredictionString"].fillna("")
+
+    # Parse each row into the format expected by mAP_evaluation.py
+    submission_samples = [
+        parse_row(row["Id"], row["PredictionString"]) for _, row in submission.iterrows()
+    ]
+    answer_samples = [
+        parse_row(row["Id"], row["PredictionString"]) for _, row in answers.iterrows()
+    ]
+
+    iou_thresholds = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95]
+    # (from https://www.kaggle.com/competitions/3d-object-detection-for-autonomous-vehicles/overview/evaluation)
+
+    # Prepare the data for parallel processing
+    sample_pairs = list(zip(answer_samples, submission_samples))
+
+    # Use multiprocessing to parallelize the computation
+    num_cpus = multiprocessing.cpu_count()
+    with multiprocessing.Pool(processes=num_cpus) as pool:
+        results = list(
+            tqdm(
+                pool.imap(partial(process_sample, iou_thresholds=iou_thresholds), sample_pairs),
+                total=len(sample_pairs),
+                desc="Processing samples",
+            )
+        )
+
+    # Flatten the results
+    mAPs = [mAP for sample_mAPs in results for mAP in sample_mAPs]
+
+    # Average over all samples and IoU thresholds
+    final_mAP = np.mean(mAPs)
+
+    return final_mAP