maite-datasets 0.0.5__py3-none-any.whl → 0.0.7__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- maite_datasets/__init__.py +2 -6
- maite_datasets/_base.py +169 -51
- maite_datasets/_builder.py +46 -55
- maite_datasets/_collate.py +2 -3
- maite_datasets/{_reader/_base.py → _reader.py} +62 -36
- maite_datasets/_validate.py +4 -2
- maite_datasets/adapters/__init__.py +3 -0
- maite_datasets/adapters/_huggingface.py +391 -0
- maite_datasets/image_classification/_cifar10.py +12 -7
- maite_datasets/image_classification/_mnist.py +15 -10
- maite_datasets/image_classification/_ships.py +12 -8
- maite_datasets/object_detection/__init__.py +4 -7
- maite_datasets/object_detection/_antiuav.py +11 -8
- maite_datasets/{_reader → object_detection}/_coco.py +29 -27
- maite_datasets/object_detection/_milco.py +11 -9
- maite_datasets/object_detection/_seadrone.py +11 -9
- maite_datasets/object_detection/_voc.py +11 -13
- maite_datasets/{_reader → object_detection}/_yolo.py +26 -21
- maite_datasets/protocols.py +94 -0
- maite_datasets/wrappers/__init__.py +8 -0
- maite_datasets/wrappers/_torch.py +109 -0
- maite_datasets-0.0.7.dist-info/METADATA +181 -0
- maite_datasets-0.0.7.dist-info/RECORD +28 -0
- maite_datasets/_mixin/__init__.py +0 -0
- maite_datasets/_mixin/_numpy.py +0 -28
- maite_datasets/_mixin/_torch.py +0 -28
- maite_datasets/_protocols.py +0 -217
- maite_datasets/_reader/__init__.py +0 -6
- maite_datasets/_reader/_factory.py +0 -64
- maite_datasets/_types.py +0 -50
- maite_datasets/object_detection/_voc_torch.py +0 -65
- maite_datasets-0.0.5.dist-info/METADATA +0 -91
- maite_datasets-0.0.5.dist-info/RECORD +0 -31
- {maite_datasets-0.0.5.dist-info → maite_datasets-0.0.7.dist-info}/WHEEL +0 -0
- {maite_datasets-0.0.5.dist-info → maite_datasets-0.0.7.dist-info}/licenses/LICENSE +0 -0
@@ -0,0 +1,181 @@
|
|
1
|
+
Metadata-Version: 2.4
|
2
|
+
Name: maite-datasets
|
3
|
+
Version: 0.0.7
|
4
|
+
Summary: A collection of Image Classification and Object Detection task datasets conforming to the MAITE protocol.
|
5
|
+
Author-email: Andrew Weng <andrew.weng@ariacoustics.com>, Ryan Wood <ryan.wood@ariacoustics.com>, Shaun Jullens <shaun.jullens@ariacoustics.com>
|
6
|
+
License-Expression: MIT
|
7
|
+
License-File: LICENSE
|
8
|
+
Classifier: Development Status :: 4 - Beta
|
9
|
+
Classifier: Framework :: Pytest
|
10
|
+
Classifier: License :: OSI Approved :: MIT License
|
11
|
+
Classifier: Operating System :: OS Independent
|
12
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
13
|
+
Classifier: Programming Language :: Python :: 3.10
|
14
|
+
Classifier: Programming Language :: Python :: 3.11
|
15
|
+
Classifier: Programming Language :: Python :: 3.12
|
16
|
+
Classifier: Programming Language :: Python :: 3.13
|
17
|
+
Requires-Python: >=3.10
|
18
|
+
Requires-Dist: defusedxml>=0.7.1
|
19
|
+
Requires-Dist: maite<0.9,>=0.7
|
20
|
+
Requires-Dist: numpy>=1.24.2
|
21
|
+
Requires-Dist: pillow>=10.3.0
|
22
|
+
Requires-Dist: requests>=2.32.3
|
23
|
+
Requires-Dist: typing-extensions>=4.12
|
24
|
+
Provides-Extra: tqdm
|
25
|
+
Requires-Dist: tqdm>=4.66; extra == 'tqdm'
|
26
|
+
Description-Content-Type: text/markdown
|
27
|
+
|
28
|
+
# MAITE Datasets
|
29
|
+
|
30
|
+
MAITE Datasets are a collection of public datasets wrapped in a [MAITE](https://mit-ll-ai-technology.github.io/maite/) compliant format.
|
31
|
+
|
32
|
+
## Installation
|
33
|
+
|
34
|
+
To install and use `maite-datasets` you can use pip:
|
35
|
+
|
36
|
+
```bash
|
37
|
+
pip install maite-datasets
|
38
|
+
```
|
39
|
+
|
40
|
+
For status bar indicators when downloading, you can include the extra `tqdm` when installing:
|
41
|
+
|
42
|
+
```bash
|
43
|
+
pip install maite-datasets[tqdm]
|
44
|
+
```
|
45
|
+
|
46
|
+
## Available Downloadable Datasets
|
47
|
+
|
48
|
+
| Task | Dataset | Description |
|
49
|
+
|----------------|------------------|---------------------------------------------------------------------|
|
50
|
+
| Classification | CIFAR10 | [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. |
|
51
|
+
| Classification | MNIST | A dataset of hand-written digits. |
|
52
|
+
| Classification | Ships | A dataset that focuses on identifying ships from satellite images. |
|
53
|
+
| Detection | AntiUAVDetection | A UAV detection dataset in natural images with varying backgrounds. |
|
54
|
+
| Detection | MILCO | A side-scan sonar dataset focused on mine-like object detection. |
|
55
|
+
| Detection | Seadrone | A UAV dataset focused on open water object detection. |
|
56
|
+
| Detection | VOCDetection | [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) dataset. |
|
57
|
+
|
58
|
+
### Usage
|
59
|
+
|
60
|
+
Here is an example of how to import MNIST for usage with your workflow.
|
61
|
+
|
62
|
+
```python
|
63
|
+
>>> from maite_datasets.image_classification import MNIST
|
64
|
+
|
65
|
+
>>> mnist = MNIST(root="data", download=True)
|
66
|
+
>>> print(mnist)
|
67
|
+
MNIST Dataset
|
68
|
+
-------------
|
69
|
+
Corruption: None
|
70
|
+
Transforms: []
|
71
|
+
Image_set: train
|
72
|
+
Metadata: {'id': 'MNIST_train', 'index2label': {0: 'zero', 1: 'one', 2: 'two', 3: 'three', 4: 'four', 5: 'five', 6: 'six', 7: 'seven', 8: 'eight', 9: 'nine'}, 'split': 'train'}
|
73
|
+
Path: /home/user/maite-datasets/data/mnist
|
74
|
+
Size: 60000
|
75
|
+
|
76
|
+
>>> print("tuple("+", ".join([str(type(t)) for t in mnist[0]])+")")
|
77
|
+
tuple(<class 'numpy.ndarray'>, <class 'numpy.ndarray'>, <class 'dict'>)
|
78
|
+
```
|
79
|
+
|
80
|
+
## Dataset Wrappers
|
81
|
+
|
82
|
+
Wrappers provide a way to convert datasets to allow usage of tools within specific backend frameworks.
|
83
|
+
|
84
|
+
### Torchvision
|
85
|
+
|
86
|
+
`TorchvisionWrapper` is a convenience class that wraps any of the datasets and provides the capability to apply
|
87
|
+
`torchvision` transforms to the dataset.
|
88
|
+
|
89
|
+
**NOTE:** `TorchvisionWrapper` requires _torch_ and _torchvision_ to be installed.
|
90
|
+
|
91
|
+
```python
|
92
|
+
>>> from maite_datasets.object_detection import MILCO
|
93
|
+
|
94
|
+
>>> milco = MILCO(root="data", download=True)
|
95
|
+
>>> print(milco)
|
96
|
+
MILCO Dataset
|
97
|
+
-------------
|
98
|
+
Transforms: []
|
99
|
+
Image Set: train
|
100
|
+
Metadata: {'id': 'MILCO_train', 'index2label': {0: 'MILCO', 1: 'NOMBO'}, 'split': 'train'}
|
101
|
+
Path: /home/user/maite-datasets/data/milco
|
102
|
+
Size: 261
|
103
|
+
|
104
|
+
>>> print(f"type={milco[0][0].__class__.__name__}, shape={milco[0][0].shape}")
|
105
|
+
type=ndarray, shape=(3, 1024, 1024)
|
106
|
+
|
107
|
+
>>> print(milco[0][1].boxes[0])
|
108
|
+
[ 75. 217. 130. 247.]
|
109
|
+
|
110
|
+
>>> from maite_datasets.wrappers import TorchvisionWrapper
|
111
|
+
>>> from torchvision.transforms.v2 import Resize
|
112
|
+
|
113
|
+
>>> milco_torch = TorchvisionWrapper(milco, transforms=Resize(224))
|
114
|
+
>>> print(milco_torch)
|
115
|
+
Torchvision Wrapped MILCO Dataset
|
116
|
+
---------------------------
|
117
|
+
Transforms: Resize(size=[224], interpolation=InterpolationMode.BILINEAR, antialias=True)
|
118
|
+
|
119
|
+
MILCO Dataset
|
120
|
+
-------------
|
121
|
+
Transforms: []
|
122
|
+
Image Set: train
|
123
|
+
Metadata: {'id': 'MILCO_train', 'index2label': {0: 'MILCO', 1: 'NOMBO'}, 'split': 'train'}
|
124
|
+
Path: /home/user/maite-datasets/data/milco
|
125
|
+
Size: 261
|
126
|
+
|
127
|
+
>>> print(f"type={milco_torch[0][0].__class__.__name__}, shape={milco_torch[0][0].shape}")
|
128
|
+
type=Image, shape=torch.Size([3, 224, 224])
|
129
|
+
|
130
|
+
>>> print(milco_torch[0][1].boxes[0])
|
131
|
+
tensor([16.4062, 47.4688, 28.4375, 54.0312], dtype=torch.float64)
|
132
|
+
```
|
133
|
+
|
134
|
+
## Dataset Adapters
|
135
|
+
|
136
|
+
Adapters provide a way to read in datasets from other popular formats.
|
137
|
+
|
138
|
+
### Huggingface
|
139
|
+
|
140
|
+
Hugging face datasets can be adapted into MAITE compliant format using the `from_huggingface` adapter.
|
141
|
+
|
142
|
+
```python
|
143
|
+
>>> from datasets import load_dataset
|
144
|
+
>>> from maite_datasets.adapters import from_huggingface
|
145
|
+
|
146
|
+
>>> cppe5 = load_dataset("cppe-5")
|
147
|
+
>>> m_cppe5 = from_huggingface(cppe5["train"])
|
148
|
+
>>> print(m_cppe5)
|
149
|
+
HFObjectDetection Dataset
|
150
|
+
-------------------------
|
151
|
+
Source: Dataset({
|
152
|
+
features: ['image_id', 'image', 'width', 'height', 'objects'],
|
153
|
+
num_rows: 1000
|
154
|
+
})
|
155
|
+
Metadata: {'id': 'cppe-5', 'index2label': {0: 'Coverall', 1: 'Face_Shield', 2: 'Gloves', 3: 'Goggles', 4: 'Mask'}, 'description': '', 'citation': '', 'homepage': '', 'license': '', 'features': {'image_id': Value('int64'), 'image': Image(mode=None, decode=True), 'width': Value('int32'), 'height': Value('int32'), 'objects': {'id': List(Value('int64')), 'area': List(Value('int64')), 'bbox': List(List(Value('float32'), length=4)), 'category': List(ClassLabel(names=['Coverall', 'Face_Shield', 'Gloves', 'Goggles', 'Mask']))}}, 'post_processed': None, 'supervised_keys': None, 'builder_name': 'parquet', 'dataset_name': 'cppe-5', 'config_name': 'default', 'version': 0.0.0, 'splits': {'train': SplitInfo(name='train', num_bytes=240478590, num_examples=1000, shard_lengths=None, dataset_name='cppe-5'), 'test': SplitInfo(name='test', num_bytes=4172706, num_examples=29, shard_lengths=None, dataset_name='cppe-5')}, 'download_checksums': {'hf://datasets/cppe-5@66f6a5efd474e35bd7cb94bf15dea27d4c6ad3f8/data/train-00000-of-00001.parquet': {'num_bytes': 237015519, 'checksum': None}, 'hf://datasets/cppe-5@66f6a5efd474e35bd7cb94bf15dea27d4c6ad3f8/data/test-00000-of-00001.parquet': {'num_bytes': 4137134, 'checksum': None}}, 'download_size': 241152653, 'post_processing_size': None, 'dataset_size': 244651296, 'size_in_bytes': 485803949}
|
156
|
+
|
157
|
+
>>> image = m_cppe5[0][0]
|
158
|
+
>>> print(f"type={image.__class__.__name__}, shape={image.shape}")
|
159
|
+
type=ndarray, shape=(3, 663, 943)
|
160
|
+
|
161
|
+
>>> target = m_cppe5[0][1]
|
162
|
+
>>> print(f"box={target.boxes[0]}, label={target.labels[0]}")
|
163
|
+
box=[302.0, 109.0, 73.0, 52.0], label=4
|
164
|
+
|
165
|
+
>>> print(m_cppe5[0][2])
|
166
|
+
{'id': [114, 115, 116, 117], 'image_id': 15, 'width': 943, 'height': 663, 'area': [3796, 1596, 152768, 81002]}
|
167
|
+
```
|
168
|
+
|
169
|
+
## Additional Information
|
170
|
+
|
171
|
+
For more information on the MAITE protocol, check out their [documentation](https://mit-ll-ai-technology.github.io/maite/).
|
172
|
+
|
173
|
+
## Acknowledgement
|
174
|
+
|
175
|
+
### CDAO Funding Acknowledgement
|
176
|
+
|
177
|
+
This material is based upon work supported by the Chief Digital and Artificial
|
178
|
+
Intelligence Office under Contract No. W519TC-23-9-2033. The views and
|
179
|
+
conclusions contained herein are those of the author(s) and should not be
|
180
|
+
interpreted as necessarily representing the official policies or endorsements,
|
181
|
+
either expressed or implied, of the U.S. Government.
|
@@ -0,0 +1,28 @@
|
|
1
|
+
maite_datasets/__init__.py,sha256=Z_HyAe08HaHMjzZS2afFumBXYFRFj0ny5ZAIp0hcj4w,569
|
2
|
+
maite_datasets/_base.py,sha256=VEd4ipHPAOCbz4Zm8zdI2yQwQ_x9O4Wq01xoZ2QvNYo,12366
|
3
|
+
maite_datasets/_builder.py,sha256=MnCh6z5hSINlzBnK_pdbgI5zSg5d1uq4UvXt3cjn9hs,9820
|
4
|
+
maite_datasets/_collate.py,sha256=pwUnmrbJH5olFjSwF-ZkGdfopTWUUlwmq0d5KzERcy8,4052
|
5
|
+
maite_datasets/_fileio.py,sha256=7S-hF3xU60AdcsPsfYR7rjbeGZUlv3JjGEZhGJOxGYU,5622
|
6
|
+
maite_datasets/_reader.py,sha256=tJqsjfXaK-mrs0Ed4BktombFMmNwCur35W7tuYCflKM,5569
|
7
|
+
maite_datasets/_validate.py,sha256=Uokbolmv1uSv98sph44HON0HEieeK3s2mqbPMP1d5xs,6948
|
8
|
+
maite_datasets/protocols.py,sha256=8NPD2AdYXiz2i9TQwS_kdpboFeYMad3G7peuh2c9GZc,2153
|
9
|
+
maite_datasets/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
|
10
|
+
maite_datasets/adapters/__init__.py,sha256=PDMdvRqLVS7x6ghJ_xUBD5Ebq5HrV5k3p4TqK2_6Gt8,191
|
11
|
+
maite_datasets/adapters/_huggingface.py,sha256=bmAz9Pssqfma1hzgTM4wtoLKnWAozkxNUzREGBYYTyM,16179
|
12
|
+
maite_datasets/image_classification/__init__.py,sha256=pcZojkdsiMoLgY4mKjoQY6WyEwiGYHxNrAGpnvn3zsY,308
|
13
|
+
maite_datasets/image_classification/_cifar10.py,sha256=rrkJZ70NBYOSGxligXyakVxNOyGAglN6PaGQazuWNO4,8453
|
14
|
+
maite_datasets/image_classification/_mnist.py,sha256=9isgdi-YXgs6nXoh1j8uOgh4_sIhBIky72Vyl866rTE,8192
|
15
|
+
maite_datasets/image_classification/_ships.py,sha256=nWhte8592lpybhQCCdgT36LnuMQ0PRJWlDxT5-IPUtk,5137
|
16
|
+
maite_datasets/object_detection/__init__.py,sha256=171KT_X6I4YGy18G240N_-ZsKvXJ6YqcBDzkhTiBj2E,587
|
17
|
+
maite_datasets/object_detection/_antiuav.py,sha256=B20JrbouDM1o5f1ct9Zfbkks8NaVqYrxu5x-rBZvGx8,8265
|
18
|
+
maite_datasets/object_detection/_coco.py,sha256=3abRQJ9ATcZOeqK-4pnMfr-pv7aGcRum88SRlLLXTzk,10309
|
19
|
+
maite_datasets/object_detection/_milco.py,sha256=brxxYs5ak0vEpOSd2IW5AMMVkuadVmXCJBFPvXTmNlo,7928
|
20
|
+
maite_datasets/object_detection/_seadrone.py,sha256=JdHL0eRZoe7pXVInOq5Xpnz3-vgeBxbO25oTYgGZ44o,271213
|
21
|
+
maite_datasets/object_detection/_voc.py,sha256=vgRn-sa_r2-hxwpM3veRZQMcWyqJz9OGalABOccZeow,19589
|
22
|
+
maite_datasets/object_detection/_yolo.py,sha256=Luojzhanh6AK949910jN0yTpy8zwF5_At6nThj3Zw9Q,11867
|
23
|
+
maite_datasets/wrappers/__init__.py,sha256=6uI0ztOB2IlMWln9JkVke4OhU2HQ8i6YCaCNq_q5qb0,225
|
24
|
+
maite_datasets/wrappers/_torch.py,sha256=nPWeBtam6EPZf9BGbrFqKlyWfyzOpX_zpLFZBTnA2Vg,4458
|
25
|
+
maite_datasets-0.0.7.dist-info/METADATA,sha256=L4UmetVfmJBFDtDWo2yElwBt47VY__euzWgDgaWgkzs,7845
|
26
|
+
maite_datasets-0.0.7.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
|
27
|
+
maite_datasets-0.0.7.dist-info/licenses/LICENSE,sha256=6h3J3R-ajGHh_isDSftzS5_jJjB9HH4TaI0vU-VscaY,1082
|
28
|
+
maite_datasets-0.0.7.dist-info/RECORD,,
|
File without changes
|
maite_datasets/_mixin/_numpy.py
DELETED
@@ -1,28 +0,0 @@
|
|
1
|
-
from __future__ import annotations
|
2
|
-
|
3
|
-
__all__ = []
|
4
|
-
|
5
|
-
from typing import Any
|
6
|
-
|
7
|
-
import numpy as np
|
8
|
-
from numpy.typing import NDArray
|
9
|
-
from PIL import Image
|
10
|
-
|
11
|
-
from maite_datasets._base import BaseDatasetMixin
|
12
|
-
|
13
|
-
|
14
|
-
class BaseDatasetNumpyMixin(BaseDatasetMixin[NDArray[np.number[Any]]]):
|
15
|
-
def _as_array(self, raw: list[Any]) -> NDArray[np.number[Any]]:
|
16
|
-
return np.asarray(raw)
|
17
|
-
|
18
|
-
def _one_hot_encode(self, value: int | list[int]) -> NDArray[np.number[Any]]:
|
19
|
-
if isinstance(value, int):
|
20
|
-
encoded = np.zeros(len(self.index2label))
|
21
|
-
encoded[value] = 1
|
22
|
-
else:
|
23
|
-
encoded = np.zeros((len(value), len(self.index2label)))
|
24
|
-
encoded[np.arange(len(value)), value] = 1
|
25
|
-
return encoded
|
26
|
-
|
27
|
-
def _read_file(self, path: str) -> NDArray[np.number[Any]]:
|
28
|
-
return np.array(Image.open(path)).transpose(2, 0, 1)
|
maite_datasets/_mixin/_torch.py
DELETED
@@ -1,28 +0,0 @@
|
|
1
|
-
from __future__ import annotations
|
2
|
-
|
3
|
-
__all__ = []
|
4
|
-
|
5
|
-
from typing import Any
|
6
|
-
|
7
|
-
import numpy as np
|
8
|
-
import torch
|
9
|
-
from PIL import Image
|
10
|
-
|
11
|
-
from maite_datasets._base import BaseDatasetMixin
|
12
|
-
|
13
|
-
|
14
|
-
class BaseDatasetTorchMixin(BaseDatasetMixin[torch.Tensor]):
|
15
|
-
def _as_array(self, raw: list[Any]) -> torch.Tensor:
|
16
|
-
return torch.as_tensor(raw)
|
17
|
-
|
18
|
-
def _one_hot_encode(self, value: int | list[int]) -> torch.Tensor:
|
19
|
-
if isinstance(value, int):
|
20
|
-
encoded = torch.zeros(len(self.index2label))
|
21
|
-
encoded[value] = 1
|
22
|
-
else:
|
23
|
-
encoded = torch.zeros((len(value), len(self.index2label)))
|
24
|
-
encoded[torch.arange(len(value)), value] = 1
|
25
|
-
return encoded
|
26
|
-
|
27
|
-
def _read_file(self, path: str) -> torch.Tensor:
|
28
|
-
return torch.as_tensor(np.array(Image.open(path)).transpose(2, 0, 1))
|
maite_datasets/_protocols.py
DELETED
@@ -1,217 +0,0 @@
|
|
1
|
-
"""
|
2
|
-
Common type protocols used for interoperability.
|
3
|
-
"""
|
4
|
-
|
5
|
-
from collections.abc import Iterator
|
6
|
-
import sys
|
7
|
-
from typing import (
|
8
|
-
Any,
|
9
|
-
Generic,
|
10
|
-
Protocol,
|
11
|
-
TypeAlias,
|
12
|
-
TypedDict,
|
13
|
-
TypeVar,
|
14
|
-
runtime_checkable,
|
15
|
-
)
|
16
|
-
|
17
|
-
import numpy.typing
|
18
|
-
from typing_extensions import NotRequired, ReadOnly, Required
|
19
|
-
|
20
|
-
if sys.version_info >= (3, 10):
|
21
|
-
from typing import TypeAlias
|
22
|
-
else:
|
23
|
-
from typing_extensions import TypeAlias
|
24
|
-
|
25
|
-
|
26
|
-
ArrayLike: TypeAlias = numpy.typing.ArrayLike
|
27
|
-
"""
|
28
|
-
Type alias for a `Union` representing objects that can be coerced into an array.
|
29
|
-
|
30
|
-
See Also
|
31
|
-
--------
|
32
|
-
`NumPy ArrayLike <https://numpy.org/doc/stable/reference/typing.html#numpy.typing.ArrayLike>`_
|
33
|
-
"""
|
34
|
-
|
35
|
-
|
36
|
-
@runtime_checkable
|
37
|
-
class Array(Protocol):
|
38
|
-
"""
|
39
|
-
Protocol for interoperable array objects.
|
40
|
-
|
41
|
-
Supports common array representations with popular libraries like
|
42
|
-
PyTorch, Tensorflow and JAX, as well as NumPy arrays.
|
43
|
-
"""
|
44
|
-
|
45
|
-
@property
|
46
|
-
def shape(self) -> tuple[int, ...]: ...
|
47
|
-
def __array__(self) -> Any: ...
|
48
|
-
def __getitem__(self, key: Any, /) -> Any: ...
|
49
|
-
def __iter__(self) -> Iterator[Any]: ...
|
50
|
-
def __len__(self) -> int: ...
|
51
|
-
|
52
|
-
|
53
|
-
_T = TypeVar("_T")
|
54
|
-
_T_co = TypeVar("_T_co", covariant=True)
|
55
|
-
_T_cn = TypeVar("_T_cn", contravariant=True)
|
56
|
-
|
57
|
-
|
58
|
-
class DatasetMetadata(TypedDict, total=False):
|
59
|
-
"""
|
60
|
-
Dataset level metadata required for all `AnnotatedDataset` classes.
|
61
|
-
|
62
|
-
Attributes
|
63
|
-
----------
|
64
|
-
id : Required[str]
|
65
|
-
A unique identifier for the dataset
|
66
|
-
index2label : NotRequired[dict[int, str]]
|
67
|
-
A lookup table converting label value to class name
|
68
|
-
"""
|
69
|
-
|
70
|
-
id: Required[ReadOnly[str]]
|
71
|
-
index2label: NotRequired[ReadOnly[dict[int, str]]]
|
72
|
-
|
73
|
-
|
74
|
-
class DatumMetadata(TypedDict, total=False):
|
75
|
-
"""
|
76
|
-
Datum level metadata required for all `AnnotatedDataset` classes.
|
77
|
-
|
78
|
-
Attributes
|
79
|
-
----------
|
80
|
-
id : Required[str]
|
81
|
-
A unique identifier for the datum
|
82
|
-
"""
|
83
|
-
|
84
|
-
id: Required[ReadOnly[str]]
|
85
|
-
|
86
|
-
|
87
|
-
@runtime_checkable
|
88
|
-
class Dataset(Generic[_T_co], Protocol):
|
89
|
-
"""
|
90
|
-
Protocol for a generic `Dataset`.
|
91
|
-
|
92
|
-
Methods
|
93
|
-
-------
|
94
|
-
__getitem__(index: int)
|
95
|
-
Returns datum at specified index.
|
96
|
-
__len__()
|
97
|
-
Returns dataset length.
|
98
|
-
"""
|
99
|
-
|
100
|
-
def __getitem__(self, index: int, /) -> _T_co: ...
|
101
|
-
def __len__(self) -> int: ...
|
102
|
-
|
103
|
-
|
104
|
-
@runtime_checkable
|
105
|
-
class AnnotatedDataset(Dataset[_T_co], Generic[_T_co], Protocol):
|
106
|
-
"""
|
107
|
-
Protocol for a generic `AnnotatedDataset`.
|
108
|
-
|
109
|
-
Attributes
|
110
|
-
----------
|
111
|
-
metadata : :class:`.DatasetMetadata` or derivatives.
|
112
|
-
|
113
|
-
Methods
|
114
|
-
-------
|
115
|
-
__getitem__(index: int)
|
116
|
-
Returns datum at specified index.
|
117
|
-
__len__()
|
118
|
-
Returns dataset length.
|
119
|
-
|
120
|
-
Notes
|
121
|
-
-----
|
122
|
-
Inherits from :class:`.Dataset`.
|
123
|
-
"""
|
124
|
-
|
125
|
-
@property
|
126
|
-
def metadata(self) -> DatasetMetadata: ...
|
127
|
-
|
128
|
-
|
129
|
-
# ========== IMAGE CLASSIFICATION DATASETS ==========
|
130
|
-
|
131
|
-
|
132
|
-
ImageClassificationDatum: TypeAlias = tuple[ArrayLike, ArrayLike, DatumMetadata]
|
133
|
-
"""
|
134
|
-
Type alias for an image classification datum tuple.
|
135
|
-
|
136
|
-
- :class:`ArrayLike` of shape (C, H, W) - Image data in channel, height, width format.
|
137
|
-
- :class:`ArrayLike` of shape (N,) - Class label as one-hot encoded ground-truth or prediction confidences.
|
138
|
-
- dict[str, Any] - Datum level metadata.
|
139
|
-
"""
|
140
|
-
|
141
|
-
|
142
|
-
ImageClassificationDataset: TypeAlias = AnnotatedDataset[ImageClassificationDatum]
|
143
|
-
"""
|
144
|
-
Type alias for an :class:`AnnotatedDataset` of :class:`ImageClassificationDatum` elements.
|
145
|
-
"""
|
146
|
-
|
147
|
-
# ========== OBJECT DETECTION DATASETS ==========
|
148
|
-
|
149
|
-
|
150
|
-
@runtime_checkable
|
151
|
-
class ObjectDetectionTarget(Protocol):
|
152
|
-
"""
|
153
|
-
Protocol for targets in an Object Detection dataset.
|
154
|
-
|
155
|
-
Attributes
|
156
|
-
----------
|
157
|
-
boxes : :class:`ArrayLike` of shape (N, 4)
|
158
|
-
labels : :class:`ArrayLike` of shape (N,)
|
159
|
-
scores : :class:`ArrayLike` of shape (N, M)
|
160
|
-
"""
|
161
|
-
|
162
|
-
@property
|
163
|
-
def boxes(self) -> ArrayLike: ...
|
164
|
-
|
165
|
-
@property
|
166
|
-
def labels(self) -> ArrayLike: ...
|
167
|
-
|
168
|
-
@property
|
169
|
-
def scores(self) -> ArrayLike: ...
|
170
|
-
|
171
|
-
|
172
|
-
ObjectDetectionDatum: TypeAlias = tuple[ArrayLike, ObjectDetectionTarget, DatumMetadata]
|
173
|
-
"""
|
174
|
-
Type alias for an object detection datum tuple.
|
175
|
-
|
176
|
-
- :class:`ArrayLike` of shape (C, H, W) - Image data in channel, height, width format.
|
177
|
-
- :class:`ObjectDetectionTarget` - Object detection target information for the image.
|
178
|
-
- dict[str, Any] - Datum level metadata.
|
179
|
-
"""
|
180
|
-
|
181
|
-
|
182
|
-
ObjectDetectionDataset: TypeAlias = AnnotatedDataset[ObjectDetectionDatum]
|
183
|
-
"""
|
184
|
-
Type alias for an :class:`AnnotatedDataset` of :class:`ObjectDetectionDatum` elements.
|
185
|
-
"""
|
186
|
-
|
187
|
-
|
188
|
-
# ========== TRANSFORM ==========
|
189
|
-
|
190
|
-
|
191
|
-
@runtime_checkable
|
192
|
-
class Transform(Generic[_T], Protocol):
|
193
|
-
"""
|
194
|
-
Protocol defining a transform function.
|
195
|
-
|
196
|
-
Requires a `__call__` method that returns transformed data.
|
197
|
-
|
198
|
-
Example
|
199
|
-
-------
|
200
|
-
>>> from typing import Any
|
201
|
-
>>> from numpy.typing import NDArray
|
202
|
-
|
203
|
-
>>> class MyTransform:
|
204
|
-
... def __init__(self, divisor: float) -> None:
|
205
|
-
... self.divisor = divisor
|
206
|
-
...
|
207
|
-
... def __call__(self, data: NDArray[Any], /) -> NDArray[Any]:
|
208
|
-
... return data / self.divisor
|
209
|
-
|
210
|
-
>>> my_transform = MyTransform(divisor=255.0)
|
211
|
-
>>> isinstance(my_transform, Transform)
|
212
|
-
True
|
213
|
-
>>> my_transform(np.array([1, 2, 3]))
|
214
|
-
array([0.004, 0.008, 0.012])
|
215
|
-
"""
|
216
|
-
|
217
|
-
def __call__(self, data: _T, /) -> _T: ...
|
@@ -1,64 +0,0 @@
|
|
1
|
-
from __future__ import annotations
|
2
|
-
|
3
|
-
import logging
|
4
|
-
from pathlib import Path
|
5
|
-
|
6
|
-
from maite_datasets._reader._base import BaseDatasetReader
|
7
|
-
from maite_datasets._reader._yolo import YOLODatasetReader
|
8
|
-
from maite_datasets._reader._coco import COCODatasetReader
|
9
|
-
|
10
|
-
_logger = logging.getLogger(__name__)
|
11
|
-
|
12
|
-
|
13
|
-
def create_dataset_reader(dataset_path: str | Path, format_hint: str | None = None) -> BaseDatasetReader:
|
14
|
-
"""
|
15
|
-
Factory function to create appropriate dataset reader based on directory structure.
|
16
|
-
|
17
|
-
Parameters
|
18
|
-
----------
|
19
|
-
dataset_path : str or Path
|
20
|
-
Root directory containing dataset files
|
21
|
-
format_hint : str or None, default None
|
22
|
-
Format hint ("coco" or "yolo"). If None, auto-detects based on file structure
|
23
|
-
|
24
|
-
Returns
|
25
|
-
-------
|
26
|
-
BaseDatasetReader
|
27
|
-
Appropriate reader instance for the detected format
|
28
|
-
|
29
|
-
Raises
|
30
|
-
------
|
31
|
-
ValueError
|
32
|
-
If format cannot be determined or is unsupported
|
33
|
-
"""
|
34
|
-
dataset_path = Path(dataset_path)
|
35
|
-
|
36
|
-
if format_hint:
|
37
|
-
format_hint = format_hint.lower()
|
38
|
-
if format_hint == "coco":
|
39
|
-
return COCODatasetReader(dataset_path)
|
40
|
-
elif format_hint == "yolo":
|
41
|
-
return YOLODatasetReader(dataset_path)
|
42
|
-
else:
|
43
|
-
raise ValueError(f"Unsupported format hint: {format_hint}")
|
44
|
-
|
45
|
-
# Auto-detect format
|
46
|
-
has_annotations_json = (dataset_path / "annotations.json").exists()
|
47
|
-
has_labels_dir = (dataset_path / "labels").exists()
|
48
|
-
|
49
|
-
if has_annotations_json and not has_labels_dir:
|
50
|
-
_logger.info(f"Detected COCO format for {dataset_path}")
|
51
|
-
return COCODatasetReader(dataset_path)
|
52
|
-
elif has_labels_dir and not has_annotations_json:
|
53
|
-
_logger.info(f"Detected YOLO format for {dataset_path}")
|
54
|
-
return YOLODatasetReader(dataset_path)
|
55
|
-
elif has_annotations_json and has_labels_dir:
|
56
|
-
raise ValueError(
|
57
|
-
f"Ambiguous format in {dataset_path}: both annotations.json and labels/ exist. "
|
58
|
-
"Use format_hint parameter to specify format."
|
59
|
-
)
|
60
|
-
else:
|
61
|
-
raise ValueError(
|
62
|
-
f"Cannot detect dataset format in {dataset_path}. "
|
63
|
-
"Expected either annotations.json (COCO) or labels/ directory (YOLO)."
|
64
|
-
)
|
maite_datasets/_types.py
DELETED
@@ -1,50 +0,0 @@
|
|
1
|
-
from __future__ import annotations
|
2
|
-
|
3
|
-
__all__ = []
|
4
|
-
|
5
|
-
from dataclasses import dataclass
|
6
|
-
from typing import Generic, TypedDict, TypeVar
|
7
|
-
|
8
|
-
from typing_extensions import NotRequired, Required
|
9
|
-
|
10
|
-
_T_co = TypeVar("_T_co", covariant=True)
|
11
|
-
|
12
|
-
|
13
|
-
class Dataset(Generic[_T_co]):
|
14
|
-
"""Abstract generic base class for PyTorch style Dataset"""
|
15
|
-
|
16
|
-
def __getitem__(self, index: int) -> _T_co: ...
|
17
|
-
def __add__(self, other: Dataset[_T_co]) -> Dataset[_T_co]: ...
|
18
|
-
|
19
|
-
|
20
|
-
class DatasetMetadata(TypedDict):
|
21
|
-
id: Required[str]
|
22
|
-
index2label: NotRequired[dict[int, str]]
|
23
|
-
split: NotRequired[str]
|
24
|
-
|
25
|
-
|
26
|
-
class DatumMetadata(TypedDict, total=False):
|
27
|
-
id: Required[str]
|
28
|
-
|
29
|
-
|
30
|
-
_TDatum = TypeVar("_TDatum")
|
31
|
-
_TArray = TypeVar("_TArray")
|
32
|
-
|
33
|
-
|
34
|
-
class AnnotatedDataset(Dataset[_TDatum]):
|
35
|
-
metadata: DatasetMetadata
|
36
|
-
|
37
|
-
def __len__(self) -> int: ...
|
38
|
-
|
39
|
-
|
40
|
-
class ImageClassificationDataset(AnnotatedDataset[tuple[_TArray, _TArray, DatumMetadata]]): ...
|
41
|
-
|
42
|
-
|
43
|
-
@dataclass
|
44
|
-
class ObjectDetectionTarget(Generic[_TArray]):
|
45
|
-
boxes: _TArray
|
46
|
-
labels: _TArray
|
47
|
-
scores: _TArray
|
48
|
-
|
49
|
-
|
50
|
-
class ObjectDetectionDataset(AnnotatedDataset[tuple[_TArray, ObjectDetectionTarget[_TArray], DatumMetadata]]): ...
|
@@ -1,65 +0,0 @@
|
|
1
|
-
from __future__ import annotations
|
2
|
-
|
3
|
-
__all__ = []
|
4
|
-
|
5
|
-
from torch import Tensor
|
6
|
-
|
7
|
-
from maite_datasets._base import BaseODDataset
|
8
|
-
from maite_datasets._types import ObjectDetectionTarget
|
9
|
-
from maite_datasets._mixin._torch import BaseDatasetTorchMixin
|
10
|
-
from maite_datasets.object_detection._voc import BaseVOCDataset
|
11
|
-
|
12
|
-
|
13
|
-
class VOCDetectionTorch(
|
14
|
-
BaseVOCDataset[Tensor, ObjectDetectionTarget[Tensor]],
|
15
|
-
BaseODDataset[Tensor, list[str], str],
|
16
|
-
BaseDatasetTorchMixin,
|
17
|
-
):
|
18
|
-
"""
|
19
|
-
`Pascal VOC <http://host.robots.ox.ac.uk/pascal/VOC/>`_ Detection Dataset as PyTorch tensors.
|
20
|
-
|
21
|
-
Parameters
|
22
|
-
----------
|
23
|
-
root : str or pathlib.Path
|
24
|
-
Because of the structure of the PASCAL VOC datasets, the root needs to be one of 4 folders.
|
25
|
-
1) Directory containing the year of the **already downloaded** dataset (i.e. .../VOCdevkit/VOC2012 <-)
|
26
|
-
2) Directory to the VOCdevkit folder of the **already downloaded** dataset (i.e. .../VOCdevkit <- /VOC2012)
|
27
|
-
3) Directory to the folder one level up from the VOCdevkit folder,
|
28
|
-
data **may** or **may not** be already downloaded (i.e. ... <- /VOCdevkit/VOC2012)
|
29
|
-
4) Directory to where you would like the dataset to be downloaded
|
30
|
-
image_set : "train", "val", "test", or "base", default "train"
|
31
|
-
If "test", then dataset year must be "2007" or "2012". Note that the 2012 test set does not contain annotations.
|
32
|
-
If "base", then the combined dataset of "train" and "val" is returned.
|
33
|
-
year : "2007", "2008", "2009", "2010", "2011" or "2012", default "2012"
|
34
|
-
The dataset year.
|
35
|
-
transforms : Transform, Sequence[Transform] or None, default None
|
36
|
-
Transform(s) to apply to the data.
|
37
|
-
download : bool, default False
|
38
|
-
If True, downloads the dataset from the internet and puts it in root directory.
|
39
|
-
Class checks to see if data is already downloaded to ensure it does not create a duplicate download.
|
40
|
-
verbose : bool, default False
|
41
|
-
If True, outputs print statements.
|
42
|
-
|
43
|
-
Attributes
|
44
|
-
----------
|
45
|
-
path : pathlib.Path
|
46
|
-
Location of the folder containing the data.
|
47
|
-
year : "2007", "2008", "2009", "2010", "2011" or "2012"
|
48
|
-
The selected dataset year.
|
49
|
-
image_set : "train", "val", "test" or "base"
|
50
|
-
The selected image set from the dataset.
|
51
|
-
index2label : dict[int, str]
|
52
|
-
Dictionary which translates from class integers to the associated class strings.
|
53
|
-
label2index : dict[str, int]
|
54
|
-
Dictionary which translates from class strings to the associated class integers.
|
55
|
-
metadata : DatasetMetadata
|
56
|
-
Typed dictionary containing dataset metadata, such as `id` which returns the dataset class name.
|
57
|
-
transforms : Sequence[Transform]
|
58
|
-
The transforms to be applied to the data.
|
59
|
-
size : int
|
60
|
-
The size of the dataset.
|
61
|
-
|
62
|
-
Note
|
63
|
-
----
|
64
|
-
Data License: `Flickr Terms of Use <http://www.flickr.com/terms.gne?legacy=1>`_
|
65
|
-
"""
|