maite-datasets 0.0.5__py3-none-any.whl → 0.0.7__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (35) hide show
  1. maite_datasets/__init__.py +2 -6
  2. maite_datasets/_base.py +169 -51
  3. maite_datasets/_builder.py +46 -55
  4. maite_datasets/_collate.py +2 -3
  5. maite_datasets/{_reader/_base.py → _reader.py} +62 -36
  6. maite_datasets/_validate.py +4 -2
  7. maite_datasets/adapters/__init__.py +3 -0
  8. maite_datasets/adapters/_huggingface.py +391 -0
  9. maite_datasets/image_classification/_cifar10.py +12 -7
  10. maite_datasets/image_classification/_mnist.py +15 -10
  11. maite_datasets/image_classification/_ships.py +12 -8
  12. maite_datasets/object_detection/__init__.py +4 -7
  13. maite_datasets/object_detection/_antiuav.py +11 -8
  14. maite_datasets/{_reader → object_detection}/_coco.py +29 -27
  15. maite_datasets/object_detection/_milco.py +11 -9
  16. maite_datasets/object_detection/_seadrone.py +11 -9
  17. maite_datasets/object_detection/_voc.py +11 -13
  18. maite_datasets/{_reader → object_detection}/_yolo.py +26 -21
  19. maite_datasets/protocols.py +94 -0
  20. maite_datasets/wrappers/__init__.py +8 -0
  21. maite_datasets/wrappers/_torch.py +109 -0
  22. maite_datasets-0.0.7.dist-info/METADATA +181 -0
  23. maite_datasets-0.0.7.dist-info/RECORD +28 -0
  24. maite_datasets/_mixin/__init__.py +0 -0
  25. maite_datasets/_mixin/_numpy.py +0 -28
  26. maite_datasets/_mixin/_torch.py +0 -28
  27. maite_datasets/_protocols.py +0 -217
  28. maite_datasets/_reader/__init__.py +0 -6
  29. maite_datasets/_reader/_factory.py +0 -64
  30. maite_datasets/_types.py +0 -50
  31. maite_datasets/object_detection/_voc_torch.py +0 -65
  32. maite_datasets-0.0.5.dist-info/METADATA +0 -91
  33. maite_datasets-0.0.5.dist-info/RECORD +0 -31
  34. {maite_datasets-0.0.5.dist-info → maite_datasets-0.0.7.dist-info}/WHEEL +0 -0
  35. {maite_datasets-0.0.5.dist-info → maite_datasets-0.0.7.dist-info}/licenses/LICENSE +0 -0
@@ -0,0 +1,181 @@
1
+ Metadata-Version: 2.4
2
+ Name: maite-datasets
3
+ Version: 0.0.7
4
+ Summary: A collection of Image Classification and Object Detection task datasets conforming to the MAITE protocol.
5
+ Author-email: Andrew Weng <andrew.weng@ariacoustics.com>, Ryan Wood <ryan.wood@ariacoustics.com>, Shaun Jullens <shaun.jullens@ariacoustics.com>
6
+ License-Expression: MIT
7
+ License-File: LICENSE
8
+ Classifier: Development Status :: 4 - Beta
9
+ Classifier: Framework :: Pytest
10
+ Classifier: License :: OSI Approved :: MIT License
11
+ Classifier: Operating System :: OS Independent
12
+ Classifier: Programming Language :: Python :: 3 :: Only
13
+ Classifier: Programming Language :: Python :: 3.10
14
+ Classifier: Programming Language :: Python :: 3.11
15
+ Classifier: Programming Language :: Python :: 3.12
16
+ Classifier: Programming Language :: Python :: 3.13
17
+ Requires-Python: >=3.10
18
+ Requires-Dist: defusedxml>=0.7.1
19
+ Requires-Dist: maite<0.9,>=0.7
20
+ Requires-Dist: numpy>=1.24.2
21
+ Requires-Dist: pillow>=10.3.0
22
+ Requires-Dist: requests>=2.32.3
23
+ Requires-Dist: typing-extensions>=4.12
24
+ Provides-Extra: tqdm
25
+ Requires-Dist: tqdm>=4.66; extra == 'tqdm'
26
+ Description-Content-Type: text/markdown
27
+
28
+ # MAITE Datasets
29
+
30
+ MAITE Datasets are a collection of public datasets wrapped in a [MAITE](https://mit-ll-ai-technology.github.io/maite/) compliant format.
31
+
32
+ ## Installation
33
+
34
+ To install and use `maite-datasets` you can use pip:
35
+
36
+ ```bash
37
+ pip install maite-datasets
38
+ ```
39
+
40
+ For status bar indicators when downloading, you can include the extra `tqdm` when installing:
41
+
42
+ ```bash
43
+ pip install maite-datasets[tqdm]
44
+ ```
45
+
46
+ ## Available Downloadable Datasets
47
+
48
+ | Task | Dataset | Description |
49
+ |----------------|------------------|---------------------------------------------------------------------|
50
+ | Classification | CIFAR10 | [CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. |
51
+ | Classification | MNIST | A dataset of hand-written digits. |
52
+ | Classification | Ships | A dataset that focuses on identifying ships from satellite images. |
53
+ | Detection | AntiUAVDetection | A UAV detection dataset in natural images with varying backgrounds. |
54
+ | Detection | MILCO | A side-scan sonar dataset focused on mine-like object detection. |
55
+ | Detection | Seadrone | A UAV dataset focused on open water object detection. |
56
+ | Detection | VOCDetection | [Pascal VOC](http://host.robots.ox.ac.uk/pascal/VOC/) dataset. |
57
+
58
+ ### Usage
59
+
60
+ Here is an example of how to import MNIST for usage with your workflow.
61
+
62
+ ```python
63
+ >>> from maite_datasets.image_classification import MNIST
64
+
65
+ >>> mnist = MNIST(root="data", download=True)
66
+ >>> print(mnist)
67
+ MNIST Dataset
68
+ -------------
69
+ Corruption: None
70
+ Transforms: []
71
+ Image_set: train
72
+ Metadata: {'id': 'MNIST_train', 'index2label': {0: 'zero', 1: 'one', 2: 'two', 3: 'three', 4: 'four', 5: 'five', 6: 'six', 7: 'seven', 8: 'eight', 9: 'nine'}, 'split': 'train'}
73
+ Path: /home/user/maite-datasets/data/mnist
74
+ Size: 60000
75
+
76
+ >>> print("tuple("+", ".join([str(type(t)) for t in mnist[0]])+")")
77
+ tuple(<class 'numpy.ndarray'>, <class 'numpy.ndarray'>, <class 'dict'>)
78
+ ```
79
+
80
+ ## Dataset Wrappers
81
+
82
+ Wrappers provide a way to convert datasets to allow usage of tools within specific backend frameworks.
83
+
84
+ ### Torchvision
85
+
86
+ `TorchvisionWrapper` is a convenience class that wraps any of the datasets and provides the capability to apply
87
+ `torchvision` transforms to the dataset.
88
+
89
+ **NOTE:** `TorchvisionWrapper` requires _torch_ and _torchvision_ to be installed.
90
+
91
+ ```python
92
+ >>> from maite_datasets.object_detection import MILCO
93
+
94
+ >>> milco = MILCO(root="data", download=True)
95
+ >>> print(milco)
96
+ MILCO Dataset
97
+ -------------
98
+ Transforms: []
99
+ Image Set: train
100
+ Metadata: {'id': 'MILCO_train', 'index2label': {0: 'MILCO', 1: 'NOMBO'}, 'split': 'train'}
101
+ Path: /home/user/maite-datasets/data/milco
102
+ Size: 261
103
+
104
+ >>> print(f"type={milco[0][0].__class__.__name__}, shape={milco[0][0].shape}")
105
+ type=ndarray, shape=(3, 1024, 1024)
106
+
107
+ >>> print(milco[0][1].boxes[0])
108
+ [ 75. 217. 130. 247.]
109
+
110
+ >>> from maite_datasets.wrappers import TorchvisionWrapper
111
+ >>> from torchvision.transforms.v2 import Resize
112
+
113
+ >>> milco_torch = TorchvisionWrapper(milco, transforms=Resize(224))
114
+ >>> print(milco_torch)
115
+ Torchvision Wrapped MILCO Dataset
116
+ ---------------------------
117
+ Transforms: Resize(size=[224], interpolation=InterpolationMode.BILINEAR, antialias=True)
118
+
119
+ MILCO Dataset
120
+ -------------
121
+ Transforms: []
122
+ Image Set: train
123
+ Metadata: {'id': 'MILCO_train', 'index2label': {0: 'MILCO', 1: 'NOMBO'}, 'split': 'train'}
124
+ Path: /home/user/maite-datasets/data/milco
125
+ Size: 261
126
+
127
+ >>> print(f"type={milco_torch[0][0].__class__.__name__}, shape={milco_torch[0][0].shape}")
128
+ type=Image, shape=torch.Size([3, 224, 224])
129
+
130
+ >>> print(milco_torch[0][1].boxes[0])
131
+ tensor([16.4062, 47.4688, 28.4375, 54.0312], dtype=torch.float64)
132
+ ```
133
+
134
+ ## Dataset Adapters
135
+
136
+ Adapters provide a way to read in datasets from other popular formats.
137
+
138
+ ### Huggingface
139
+
140
+ Hugging face datasets can be adapted into MAITE compliant format using the `from_huggingface` adapter.
141
+
142
+ ```python
143
+ >>> from datasets import load_dataset
144
+ >>> from maite_datasets.adapters import from_huggingface
145
+
146
+ >>> cppe5 = load_dataset("cppe-5")
147
+ >>> m_cppe5 = from_huggingface(cppe5["train"])
148
+ >>> print(m_cppe5)
149
+ HFObjectDetection Dataset
150
+ -------------------------
151
+ Source: Dataset({
152
+ features: ['image_id', 'image', 'width', 'height', 'objects'],
153
+ num_rows: 1000
154
+ })
155
+ Metadata: {'id': 'cppe-5', 'index2label': {0: 'Coverall', 1: 'Face_Shield', 2: 'Gloves', 3: 'Goggles', 4: 'Mask'}, 'description': '', 'citation': '', 'homepage': '', 'license': '', 'features': {'image_id': Value('int64'), 'image': Image(mode=None, decode=True), 'width': Value('int32'), 'height': Value('int32'), 'objects': {'id': List(Value('int64')), 'area': List(Value('int64')), 'bbox': List(List(Value('float32'), length=4)), 'category': List(ClassLabel(names=['Coverall', 'Face_Shield', 'Gloves', 'Goggles', 'Mask']))}}, 'post_processed': None, 'supervised_keys': None, 'builder_name': 'parquet', 'dataset_name': 'cppe-5', 'config_name': 'default', 'version': 0.0.0, 'splits': {'train': SplitInfo(name='train', num_bytes=240478590, num_examples=1000, shard_lengths=None, dataset_name='cppe-5'), 'test': SplitInfo(name='test', num_bytes=4172706, num_examples=29, shard_lengths=None, dataset_name='cppe-5')}, 'download_checksums': {'hf://datasets/cppe-5@66f6a5efd474e35bd7cb94bf15dea27d4c6ad3f8/data/train-00000-of-00001.parquet': {'num_bytes': 237015519, 'checksum': None}, 'hf://datasets/cppe-5@66f6a5efd474e35bd7cb94bf15dea27d4c6ad3f8/data/test-00000-of-00001.parquet': {'num_bytes': 4137134, 'checksum': None}}, 'download_size': 241152653, 'post_processing_size': None, 'dataset_size': 244651296, 'size_in_bytes': 485803949}
156
+
157
+ >>> image = m_cppe5[0][0]
158
+ >>> print(f"type={image.__class__.__name__}, shape={image.shape}")
159
+ type=ndarray, shape=(3, 663, 943)
160
+
161
+ >>> target = m_cppe5[0][1]
162
+ >>> print(f"box={target.boxes[0]}, label={target.labels[0]}")
163
+ box=[302.0, 109.0, 73.0, 52.0], label=4
164
+
165
+ >>> print(m_cppe5[0][2])
166
+ {'id': [114, 115, 116, 117], 'image_id': 15, 'width': 943, 'height': 663, 'area': [3796, 1596, 152768, 81002]}
167
+ ```
168
+
169
+ ## Additional Information
170
+
171
+ For more information on the MAITE protocol, check out their [documentation](https://mit-ll-ai-technology.github.io/maite/).
172
+
173
+ ## Acknowledgement
174
+
175
+ ### CDAO Funding Acknowledgement
176
+
177
+ This material is based upon work supported by the Chief Digital and Artificial
178
+ Intelligence Office under Contract No. W519TC-23-9-2033. The views and
179
+ conclusions contained herein are those of the author(s) and should not be
180
+ interpreted as necessarily representing the official policies or endorsements,
181
+ either expressed or implied, of the U.S. Government.
@@ -0,0 +1,28 @@
1
+ maite_datasets/__init__.py,sha256=Z_HyAe08HaHMjzZS2afFumBXYFRFj0ny5ZAIp0hcj4w,569
2
+ maite_datasets/_base.py,sha256=VEd4ipHPAOCbz4Zm8zdI2yQwQ_x9O4Wq01xoZ2QvNYo,12366
3
+ maite_datasets/_builder.py,sha256=MnCh6z5hSINlzBnK_pdbgI5zSg5d1uq4UvXt3cjn9hs,9820
4
+ maite_datasets/_collate.py,sha256=pwUnmrbJH5olFjSwF-ZkGdfopTWUUlwmq0d5KzERcy8,4052
5
+ maite_datasets/_fileio.py,sha256=7S-hF3xU60AdcsPsfYR7rjbeGZUlv3JjGEZhGJOxGYU,5622
6
+ maite_datasets/_reader.py,sha256=tJqsjfXaK-mrs0Ed4BktombFMmNwCur35W7tuYCflKM,5569
7
+ maite_datasets/_validate.py,sha256=Uokbolmv1uSv98sph44HON0HEieeK3s2mqbPMP1d5xs,6948
8
+ maite_datasets/protocols.py,sha256=8NPD2AdYXiz2i9TQwS_kdpboFeYMad3G7peuh2c9GZc,2153
9
+ maite_datasets/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
10
+ maite_datasets/adapters/__init__.py,sha256=PDMdvRqLVS7x6ghJ_xUBD5Ebq5HrV5k3p4TqK2_6Gt8,191
11
+ maite_datasets/adapters/_huggingface.py,sha256=bmAz9Pssqfma1hzgTM4wtoLKnWAozkxNUzREGBYYTyM,16179
12
+ maite_datasets/image_classification/__init__.py,sha256=pcZojkdsiMoLgY4mKjoQY6WyEwiGYHxNrAGpnvn3zsY,308
13
+ maite_datasets/image_classification/_cifar10.py,sha256=rrkJZ70NBYOSGxligXyakVxNOyGAglN6PaGQazuWNO4,8453
14
+ maite_datasets/image_classification/_mnist.py,sha256=9isgdi-YXgs6nXoh1j8uOgh4_sIhBIky72Vyl866rTE,8192
15
+ maite_datasets/image_classification/_ships.py,sha256=nWhte8592lpybhQCCdgT36LnuMQ0PRJWlDxT5-IPUtk,5137
16
+ maite_datasets/object_detection/__init__.py,sha256=171KT_X6I4YGy18G240N_-ZsKvXJ6YqcBDzkhTiBj2E,587
17
+ maite_datasets/object_detection/_antiuav.py,sha256=B20JrbouDM1o5f1ct9Zfbkks8NaVqYrxu5x-rBZvGx8,8265
18
+ maite_datasets/object_detection/_coco.py,sha256=3abRQJ9ATcZOeqK-4pnMfr-pv7aGcRum88SRlLLXTzk,10309
19
+ maite_datasets/object_detection/_milco.py,sha256=brxxYs5ak0vEpOSd2IW5AMMVkuadVmXCJBFPvXTmNlo,7928
20
+ maite_datasets/object_detection/_seadrone.py,sha256=JdHL0eRZoe7pXVInOq5Xpnz3-vgeBxbO25oTYgGZ44o,271213
21
+ maite_datasets/object_detection/_voc.py,sha256=vgRn-sa_r2-hxwpM3veRZQMcWyqJz9OGalABOccZeow,19589
22
+ maite_datasets/object_detection/_yolo.py,sha256=Luojzhanh6AK949910jN0yTpy8zwF5_At6nThj3Zw9Q,11867
23
+ maite_datasets/wrappers/__init__.py,sha256=6uI0ztOB2IlMWln9JkVke4OhU2HQ8i6YCaCNq_q5qb0,225
24
+ maite_datasets/wrappers/_torch.py,sha256=nPWeBtam6EPZf9BGbrFqKlyWfyzOpX_zpLFZBTnA2Vg,4458
25
+ maite_datasets-0.0.7.dist-info/METADATA,sha256=L4UmetVfmJBFDtDWo2yElwBt47VY__euzWgDgaWgkzs,7845
26
+ maite_datasets-0.0.7.dist-info/WHEEL,sha256=qtCwoSJWgHk21S1Kb4ihdzI2rlJ1ZKaIurTj_ngOhyQ,87
27
+ maite_datasets-0.0.7.dist-info/licenses/LICENSE,sha256=6h3J3R-ajGHh_isDSftzS5_jJjB9HH4TaI0vU-VscaY,1082
28
+ maite_datasets-0.0.7.dist-info/RECORD,,
File without changes
@@ -1,28 +0,0 @@
1
- from __future__ import annotations
2
-
3
- __all__ = []
4
-
5
- from typing import Any
6
-
7
- import numpy as np
8
- from numpy.typing import NDArray
9
- from PIL import Image
10
-
11
- from maite_datasets._base import BaseDatasetMixin
12
-
13
-
14
- class BaseDatasetNumpyMixin(BaseDatasetMixin[NDArray[np.number[Any]]]):
15
- def _as_array(self, raw: list[Any]) -> NDArray[np.number[Any]]:
16
- return np.asarray(raw)
17
-
18
- def _one_hot_encode(self, value: int | list[int]) -> NDArray[np.number[Any]]:
19
- if isinstance(value, int):
20
- encoded = np.zeros(len(self.index2label))
21
- encoded[value] = 1
22
- else:
23
- encoded = np.zeros((len(value), len(self.index2label)))
24
- encoded[np.arange(len(value)), value] = 1
25
- return encoded
26
-
27
- def _read_file(self, path: str) -> NDArray[np.number[Any]]:
28
- return np.array(Image.open(path)).transpose(2, 0, 1)
@@ -1,28 +0,0 @@
1
- from __future__ import annotations
2
-
3
- __all__ = []
4
-
5
- from typing import Any
6
-
7
- import numpy as np
8
- import torch
9
- from PIL import Image
10
-
11
- from maite_datasets._base import BaseDatasetMixin
12
-
13
-
14
- class BaseDatasetTorchMixin(BaseDatasetMixin[torch.Tensor]):
15
- def _as_array(self, raw: list[Any]) -> torch.Tensor:
16
- return torch.as_tensor(raw)
17
-
18
- def _one_hot_encode(self, value: int | list[int]) -> torch.Tensor:
19
- if isinstance(value, int):
20
- encoded = torch.zeros(len(self.index2label))
21
- encoded[value] = 1
22
- else:
23
- encoded = torch.zeros((len(value), len(self.index2label)))
24
- encoded[torch.arange(len(value)), value] = 1
25
- return encoded
26
-
27
- def _read_file(self, path: str) -> torch.Tensor:
28
- return torch.as_tensor(np.array(Image.open(path)).transpose(2, 0, 1))
@@ -1,217 +0,0 @@
1
- """
2
- Common type protocols used for interoperability.
3
- """
4
-
5
- from collections.abc import Iterator
6
- import sys
7
- from typing import (
8
- Any,
9
- Generic,
10
- Protocol,
11
- TypeAlias,
12
- TypedDict,
13
- TypeVar,
14
- runtime_checkable,
15
- )
16
-
17
- import numpy.typing
18
- from typing_extensions import NotRequired, ReadOnly, Required
19
-
20
- if sys.version_info >= (3, 10):
21
- from typing import TypeAlias
22
- else:
23
- from typing_extensions import TypeAlias
24
-
25
-
26
- ArrayLike: TypeAlias = numpy.typing.ArrayLike
27
- """
28
- Type alias for a `Union` representing objects that can be coerced into an array.
29
-
30
- See Also
31
- --------
32
- `NumPy ArrayLike <https://numpy.org/doc/stable/reference/typing.html#numpy.typing.ArrayLike>`_
33
- """
34
-
35
-
36
- @runtime_checkable
37
- class Array(Protocol):
38
- """
39
- Protocol for interoperable array objects.
40
-
41
- Supports common array representations with popular libraries like
42
- PyTorch, Tensorflow and JAX, as well as NumPy arrays.
43
- """
44
-
45
- @property
46
- def shape(self) -> tuple[int, ...]: ...
47
- def __array__(self) -> Any: ...
48
- def __getitem__(self, key: Any, /) -> Any: ...
49
- def __iter__(self) -> Iterator[Any]: ...
50
- def __len__(self) -> int: ...
51
-
52
-
53
- _T = TypeVar("_T")
54
- _T_co = TypeVar("_T_co", covariant=True)
55
- _T_cn = TypeVar("_T_cn", contravariant=True)
56
-
57
-
58
- class DatasetMetadata(TypedDict, total=False):
59
- """
60
- Dataset level metadata required for all `AnnotatedDataset` classes.
61
-
62
- Attributes
63
- ----------
64
- id : Required[str]
65
- A unique identifier for the dataset
66
- index2label : NotRequired[dict[int, str]]
67
- A lookup table converting label value to class name
68
- """
69
-
70
- id: Required[ReadOnly[str]]
71
- index2label: NotRequired[ReadOnly[dict[int, str]]]
72
-
73
-
74
- class DatumMetadata(TypedDict, total=False):
75
- """
76
- Datum level metadata required for all `AnnotatedDataset` classes.
77
-
78
- Attributes
79
- ----------
80
- id : Required[str]
81
- A unique identifier for the datum
82
- """
83
-
84
- id: Required[ReadOnly[str]]
85
-
86
-
87
- @runtime_checkable
88
- class Dataset(Generic[_T_co], Protocol):
89
- """
90
- Protocol for a generic `Dataset`.
91
-
92
- Methods
93
- -------
94
- __getitem__(index: int)
95
- Returns datum at specified index.
96
- __len__()
97
- Returns dataset length.
98
- """
99
-
100
- def __getitem__(self, index: int, /) -> _T_co: ...
101
- def __len__(self) -> int: ...
102
-
103
-
104
- @runtime_checkable
105
- class AnnotatedDataset(Dataset[_T_co], Generic[_T_co], Protocol):
106
- """
107
- Protocol for a generic `AnnotatedDataset`.
108
-
109
- Attributes
110
- ----------
111
- metadata : :class:`.DatasetMetadata` or derivatives.
112
-
113
- Methods
114
- -------
115
- __getitem__(index: int)
116
- Returns datum at specified index.
117
- __len__()
118
- Returns dataset length.
119
-
120
- Notes
121
- -----
122
- Inherits from :class:`.Dataset`.
123
- """
124
-
125
- @property
126
- def metadata(self) -> DatasetMetadata: ...
127
-
128
-
129
- # ========== IMAGE CLASSIFICATION DATASETS ==========
130
-
131
-
132
- ImageClassificationDatum: TypeAlias = tuple[ArrayLike, ArrayLike, DatumMetadata]
133
- """
134
- Type alias for an image classification datum tuple.
135
-
136
- - :class:`ArrayLike` of shape (C, H, W) - Image data in channel, height, width format.
137
- - :class:`ArrayLike` of shape (N,) - Class label as one-hot encoded ground-truth or prediction confidences.
138
- - dict[str, Any] - Datum level metadata.
139
- """
140
-
141
-
142
- ImageClassificationDataset: TypeAlias = AnnotatedDataset[ImageClassificationDatum]
143
- """
144
- Type alias for an :class:`AnnotatedDataset` of :class:`ImageClassificationDatum` elements.
145
- """
146
-
147
- # ========== OBJECT DETECTION DATASETS ==========
148
-
149
-
150
- @runtime_checkable
151
- class ObjectDetectionTarget(Protocol):
152
- """
153
- Protocol for targets in an Object Detection dataset.
154
-
155
- Attributes
156
- ----------
157
- boxes : :class:`ArrayLike` of shape (N, 4)
158
- labels : :class:`ArrayLike` of shape (N,)
159
- scores : :class:`ArrayLike` of shape (N, M)
160
- """
161
-
162
- @property
163
- def boxes(self) -> ArrayLike: ...
164
-
165
- @property
166
- def labels(self) -> ArrayLike: ...
167
-
168
- @property
169
- def scores(self) -> ArrayLike: ...
170
-
171
-
172
- ObjectDetectionDatum: TypeAlias = tuple[ArrayLike, ObjectDetectionTarget, DatumMetadata]
173
- """
174
- Type alias for an object detection datum tuple.
175
-
176
- - :class:`ArrayLike` of shape (C, H, W) - Image data in channel, height, width format.
177
- - :class:`ObjectDetectionTarget` - Object detection target information for the image.
178
- - dict[str, Any] - Datum level metadata.
179
- """
180
-
181
-
182
- ObjectDetectionDataset: TypeAlias = AnnotatedDataset[ObjectDetectionDatum]
183
- """
184
- Type alias for an :class:`AnnotatedDataset` of :class:`ObjectDetectionDatum` elements.
185
- """
186
-
187
-
188
- # ========== TRANSFORM ==========
189
-
190
-
191
- @runtime_checkable
192
- class Transform(Generic[_T], Protocol):
193
- """
194
- Protocol defining a transform function.
195
-
196
- Requires a `__call__` method that returns transformed data.
197
-
198
- Example
199
- -------
200
- >>> from typing import Any
201
- >>> from numpy.typing import NDArray
202
-
203
- >>> class MyTransform:
204
- ... def __init__(self, divisor: float) -> None:
205
- ... self.divisor = divisor
206
- ...
207
- ... def __call__(self, data: NDArray[Any], /) -> NDArray[Any]:
208
- ... return data / self.divisor
209
-
210
- >>> my_transform = MyTransform(divisor=255.0)
211
- >>> isinstance(my_transform, Transform)
212
- True
213
- >>> my_transform(np.array([1, 2, 3]))
214
- array([0.004, 0.008, 0.012])
215
- """
216
-
217
- def __call__(self, data: _T, /) -> _T: ...
@@ -1,6 +0,0 @@
1
- """
2
- Dataset readers for common computer vision dataset formats.
3
-
4
- This module provides standardized readers that for loading datasets
5
- from directory structures.
6
- """
@@ -1,64 +0,0 @@
1
- from __future__ import annotations
2
-
3
- import logging
4
- from pathlib import Path
5
-
6
- from maite_datasets._reader._base import BaseDatasetReader
7
- from maite_datasets._reader._yolo import YOLODatasetReader
8
- from maite_datasets._reader._coco import COCODatasetReader
9
-
10
- _logger = logging.getLogger(__name__)
11
-
12
-
13
- def create_dataset_reader(dataset_path: str | Path, format_hint: str | None = None) -> BaseDatasetReader:
14
- """
15
- Factory function to create appropriate dataset reader based on directory structure.
16
-
17
- Parameters
18
- ----------
19
- dataset_path : str or Path
20
- Root directory containing dataset files
21
- format_hint : str or None, default None
22
- Format hint ("coco" or "yolo"). If None, auto-detects based on file structure
23
-
24
- Returns
25
- -------
26
- BaseDatasetReader
27
- Appropriate reader instance for the detected format
28
-
29
- Raises
30
- ------
31
- ValueError
32
- If format cannot be determined or is unsupported
33
- """
34
- dataset_path = Path(dataset_path)
35
-
36
- if format_hint:
37
- format_hint = format_hint.lower()
38
- if format_hint == "coco":
39
- return COCODatasetReader(dataset_path)
40
- elif format_hint == "yolo":
41
- return YOLODatasetReader(dataset_path)
42
- else:
43
- raise ValueError(f"Unsupported format hint: {format_hint}")
44
-
45
- # Auto-detect format
46
- has_annotations_json = (dataset_path / "annotations.json").exists()
47
- has_labels_dir = (dataset_path / "labels").exists()
48
-
49
- if has_annotations_json and not has_labels_dir:
50
- _logger.info(f"Detected COCO format for {dataset_path}")
51
- return COCODatasetReader(dataset_path)
52
- elif has_labels_dir and not has_annotations_json:
53
- _logger.info(f"Detected YOLO format for {dataset_path}")
54
- return YOLODatasetReader(dataset_path)
55
- elif has_annotations_json and has_labels_dir:
56
- raise ValueError(
57
- f"Ambiguous format in {dataset_path}: both annotations.json and labels/ exist. "
58
- "Use format_hint parameter to specify format."
59
- )
60
- else:
61
- raise ValueError(
62
- f"Cannot detect dataset format in {dataset_path}. "
63
- "Expected either annotations.json (COCO) or labels/ directory (YOLO)."
64
- )
maite_datasets/_types.py DELETED
@@ -1,50 +0,0 @@
1
- from __future__ import annotations
2
-
3
- __all__ = []
4
-
5
- from dataclasses import dataclass
6
- from typing import Generic, TypedDict, TypeVar
7
-
8
- from typing_extensions import NotRequired, Required
9
-
10
- _T_co = TypeVar("_T_co", covariant=True)
11
-
12
-
13
- class Dataset(Generic[_T_co]):
14
- """Abstract generic base class for PyTorch style Dataset"""
15
-
16
- def __getitem__(self, index: int) -> _T_co: ...
17
- def __add__(self, other: Dataset[_T_co]) -> Dataset[_T_co]: ...
18
-
19
-
20
- class DatasetMetadata(TypedDict):
21
- id: Required[str]
22
- index2label: NotRequired[dict[int, str]]
23
- split: NotRequired[str]
24
-
25
-
26
- class DatumMetadata(TypedDict, total=False):
27
- id: Required[str]
28
-
29
-
30
- _TDatum = TypeVar("_TDatum")
31
- _TArray = TypeVar("_TArray")
32
-
33
-
34
- class AnnotatedDataset(Dataset[_TDatum]):
35
- metadata: DatasetMetadata
36
-
37
- def __len__(self) -> int: ...
38
-
39
-
40
- class ImageClassificationDataset(AnnotatedDataset[tuple[_TArray, _TArray, DatumMetadata]]): ...
41
-
42
-
43
- @dataclass
44
- class ObjectDetectionTarget(Generic[_TArray]):
45
- boxes: _TArray
46
- labels: _TArray
47
- scores: _TArray
48
-
49
-
50
- class ObjectDetectionDataset(AnnotatedDataset[tuple[_TArray, ObjectDetectionTarget[_TArray], DatumMetadata]]): ...
@@ -1,65 +0,0 @@
1
- from __future__ import annotations
2
-
3
- __all__ = []
4
-
5
- from torch import Tensor
6
-
7
- from maite_datasets._base import BaseODDataset
8
- from maite_datasets._types import ObjectDetectionTarget
9
- from maite_datasets._mixin._torch import BaseDatasetTorchMixin
10
- from maite_datasets.object_detection._voc import BaseVOCDataset
11
-
12
-
13
- class VOCDetectionTorch(
14
- BaseVOCDataset[Tensor, ObjectDetectionTarget[Tensor]],
15
- BaseODDataset[Tensor, list[str], str],
16
- BaseDatasetTorchMixin,
17
- ):
18
- """
19
- `Pascal VOC <http://host.robots.ox.ac.uk/pascal/VOC/>`_ Detection Dataset as PyTorch tensors.
20
-
21
- Parameters
22
- ----------
23
- root : str or pathlib.Path
24
- Because of the structure of the PASCAL VOC datasets, the root needs to be one of 4 folders.
25
- 1) Directory containing the year of the **already downloaded** dataset (i.e. .../VOCdevkit/VOC2012 <-)
26
- 2) Directory to the VOCdevkit folder of the **already downloaded** dataset (i.e. .../VOCdevkit <- /VOC2012)
27
- 3) Directory to the folder one level up from the VOCdevkit folder,
28
- data **may** or **may not** be already downloaded (i.e. ... <- /VOCdevkit/VOC2012)
29
- 4) Directory to where you would like the dataset to be downloaded
30
- image_set : "train", "val", "test", or "base", default "train"
31
- If "test", then dataset year must be "2007" or "2012". Note that the 2012 test set does not contain annotations.
32
- If "base", then the combined dataset of "train" and "val" is returned.
33
- year : "2007", "2008", "2009", "2010", "2011" or "2012", default "2012"
34
- The dataset year.
35
- transforms : Transform, Sequence[Transform] or None, default None
36
- Transform(s) to apply to the data.
37
- download : bool, default False
38
- If True, downloads the dataset from the internet and puts it in root directory.
39
- Class checks to see if data is already downloaded to ensure it does not create a duplicate download.
40
- verbose : bool, default False
41
- If True, outputs print statements.
42
-
43
- Attributes
44
- ----------
45
- path : pathlib.Path
46
- Location of the folder containing the data.
47
- year : "2007", "2008", "2009", "2010", "2011" or "2012"
48
- The selected dataset year.
49
- image_set : "train", "val", "test" or "base"
50
- The selected image set from the dataset.
51
- index2label : dict[int, str]
52
- Dictionary which translates from class integers to the associated class strings.
53
- label2index : dict[str, int]
54
- Dictionary which translates from class strings to the associated class integers.
55
- metadata : DatasetMetadata
56
- Typed dictionary containing dataset metadata, such as `id` which returns the dataset class name.
57
- transforms : Sequence[Transform]
58
- The transforms to be applied to the data.
59
- size : int
60
- The size of the dataset.
61
-
62
- Note
63
- ----
64
- Data License: `Flickr Terms of Use <http://www.flickr.com/terms.gne?legacy=1>`_
65
- """