tidy3a 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- tidy3a-0.1.0/CHANGELOG.md +55 -0
- tidy3a-0.1.0/LICENSE +21 -0
- tidy3a-0.1.0/MANIFEST.in +5 -0
- tidy3a-0.1.0/PKG-INFO +142 -0
- tidy3a-0.1.0/README.md +117 -0
- tidy3a-0.1.0/docs/decisions.md +19 -0
- tidy3a-0.1.0/docs/first-release.md +69 -0
- tidy3a-0.1.0/docs/m0-examples.md +127 -0
- tidy3a-0.1.0/docs/m6-metadata-lazy.md +143 -0
- tidy3a-0.1.0/docs/m6-read-binary.md +137 -0
- tidy3a-0.1.0/docs/plan.md +1253 -0
- tidy3a-0.1.0/docs/releasing.md +52 -0
- tidy3a-0.1.0/examples/nuscenes_sweep.py +135 -0
- tidy3a-0.1.0/examples/sample_data.py +185 -0
- tidy3a-0.1.0/prototypes/test_examples.py +213 -0
- tidy3a-0.1.0/prototypes/test_xarray_assumptions.py +258 -0
- tidy3a-0.1.0/pyproject.toml +52 -0
- tidy3a-0.1.0/scripts/check.sh +65 -0
- tidy3a-0.1.0/setup.cfg +4 -0
- tidy3a-0.1.0/src/tidy3a/__init__.py +114 -0
- tidy3a-0.1.0/src/tidy3a/__version__.py +1 -0
- tidy3a-0.1.0/src/tidy3a/align.py +136 -0
- tidy3a-0.1.0/src/tidy3a/errors.py +45 -0
- tidy3a-0.1.0/src/tidy3a/expr.py +911 -0
- tidy3a-0.1.0/src/tidy3a/formula.py +920 -0
- tidy3a-0.1.0/src/tidy3a/frame.py +243 -0
- tidy3a-0.1.0/src/tidy3a/io.py +598 -0
- tidy3a-0.1.0/src/tidy3a/jupyter.py +314 -0
- tidy3a-0.1.0/src/tidy3a/linalg.py +192 -0
- tidy3a-0.1.0/src/tidy3a/py.typed +0 -0
- tidy3a-0.1.0/src/tidy3a/selectors.py +141 -0
- tidy3a-0.1.0/src/tidy3a/verbs.py +782 -0
- tidy3a-0.1.0/src/tidy3a.egg-info/PKG-INFO +142 -0
- tidy3a-0.1.0/src/tidy3a.egg-info/SOURCES.txt +47 -0
- tidy3a-0.1.0/src/tidy3a.egg-info/dependency_links.txt +1 -0
- tidy3a-0.1.0/src/tidy3a.egg-info/requires.txt +9 -0
- tidy3a-0.1.0/src/tidy3a.egg-info/top_level.txt +1 -0
- tidy3a-0.1.0/tests/test_binary.py +362 -0
- tidy3a-0.1.0/tests/test_dask.py +169 -0
- tidy3a-0.1.0/tests/test_formula.py +230 -0
- tidy3a-0.1.0/tests/test_frame.py +131 -0
- tidy3a-0.1.0/tests/test_io.py +387 -0
- tidy3a-0.1.0/tests/test_linalg.py +268 -0
- tidy3a-0.1.0/tests/test_metadata.py +173 -0
- tidy3a-0.1.0/tests/test_notebooks.py +178 -0
- tidy3a-0.1.0/tests/test_nuscenes.py +28 -0
- tidy3a-0.1.0/tests/test_sample_data.py +33 -0
- tidy3a-0.1.0/tests/test_verbs.py +716 -0
- tidy3a-0.1.0/tests/test_workflow.py +62 -0
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.0 — 2026-10-24
|
|
4
|
+
|
|
5
|
+
First release.
|
|
6
|
+
|
|
7
|
+
- `read_array` reads raw numeric files, `.npy` files, and in-memory NumPy
|
|
8
|
+
arrays, with `infer`, labels, `field_axis=`, `variables_from=`, byte
|
|
9
|
+
order, storage order, header offsets, and memory-mapping.
|
|
10
|
+
- `read_binary` reads packed or explicitly padded mixed records into
|
|
11
|
+
separate typed variables. `fixed_text` declares strict UTF-8/ASCII text
|
|
12
|
+
decoding and padding; numeric fields retain their widths and kinds.
|
|
13
|
+
- `tidy()` wraps xarray objects; `TidyArray` reports its axes, labels, axis
|
|
14
|
+
roles, and anything the reader inferred.
|
|
15
|
+
- Tidy-layer verbs: `select` with tidy3's selection helpers, `filter`,
|
|
16
|
+
`slice`/`slice_head`/`slice_tail`, `mutate` with write-back and vector
|
|
17
|
+
assignment, `group_by` by axis, value, or `bins()`, `summarise`,
|
|
18
|
+
`ungroup`, `pull`, `bind_cols`, and `left_join`/`inner_join`/`full_join`.
|
|
19
|
+
- Expressions: `col`, `variable`, `label`, `env`, `fields`, `[x, y, z]`
|
|
20
|
+
vectors, `if_else`, and reductions `mean`, `sum`, `min`, `max`, `median`,
|
|
21
|
+
`std`, `var` (variance, as in tidy3 and dplyr), `any`, and `all` with
|
|
22
|
+
`over=` and `na_rm=`, plus `n()` for observation counts.
|
|
23
|
+
- Exact label alignment, with reordering by label and explicit joins.
|
|
24
|
+
- Metadata: wrapping keeps attributes and encodings; new and computed
|
|
25
|
+
results clear stale serialization encoding and inherited attributes,
|
|
26
|
+
direct aliases keep theirs, and joins report dropped conflicting
|
|
27
|
+
attributes.
|
|
28
|
+
- Lazy input: NetCDF and Dask-backed data stay lazy through inspection,
|
|
29
|
+
selection, regular positional slicing, arithmetic, and reductions until
|
|
30
|
+
`collect()`; grouping, filtering, and field write-back read the values
|
|
31
|
+
they need, as documented.
|
|
32
|
+
- Array layer: `dot` with `over=`, elementwise numeric functions, `conj`,
|
|
33
|
+
`real`, `imag`, `angle`, `arctan2`, `rename_axes`, and `pack`.
|
|
34
|
+
- `formula()`: plain mathematical text and a strict LaTeX dialect with
|
|
35
|
+
plot3's conventions, subscripts as axis indices, labels, and positions,
|
|
36
|
+
`\sum` with optional checked bounds, `bind=` for index letters and
|
|
37
|
+
values, and show-back of how each formula was read.
|
|
38
|
+
- Notebook layer (`%load_ext tidy3a.jupyter`): bare names in tidy3A verbs,
|
|
39
|
+
settled when the verb runs, with an error when a name is both data and a
|
|
40
|
+
notebook variable; `[x, y, z] = …` inside `mutate`; only tidy3A's own
|
|
41
|
+
calls are rewritten, so tidy3 and tidy3A share a notebook.
|
|
42
|
+
- `n()` counts observations on the axes every non-scalar variable shares.
|
|
43
|
+
- Inspection output shows time labels to the second (or as dates) and
|
|
44
|
+
floats to 10 significant digits, with a gap before each axis role.
|
|
45
|
+
- Examples on real data: a nuScenes Mini LiDAR sweep, and three public
|
|
46
|
+
sample datasets (air temperature, ERA-Interim wind, a Landsat scene),
|
|
47
|
+
each checked against plain NumPy or xarray.
|
|
48
|
+
- Source distributions include documentation, examples, and prototype
|
|
49
|
+
tests. Packaging checks build a wheel from the source archive and
|
|
50
|
+
validate both distributions' metadata. The build requires setuptools
|
|
51
|
+
77.0.3 or newer for the license metadata.
|
|
52
|
+
- Requires Python 3.11+, NumPy 2.0+, pandas 2.2.2+ (now declared, as
|
|
53
|
+
tidy3A imports it directly), and xarray 2024.6+.
|
|
54
|
+
- Publishing a GitHub Release runs the version check and reusable CI gate,
|
|
55
|
+
then uploads the validated distributions to PyPI through trusted publishing.
|
tidy3a-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 rleyvasal
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
tidy3a-0.1.0/MANIFEST.in
ADDED
tidy3a-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: tidy3a
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Tidy-style verbs for named multidimensional arrays, built on xarray
|
|
5
|
+
Author: Rigoberto Leyva Salmeron
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Project-URL: Homepage, https://github.com/rleyvasal/tidy3a
|
|
8
|
+
Keywords: xarray,arrays,tidyverse,dplyr,point-cloud,raster
|
|
9
|
+
Classifier: Development Status :: 2 - Pre-Alpha
|
|
10
|
+
Classifier: Intended Audience :: Science/Research
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Topic :: Scientific/Engineering
|
|
13
|
+
Requires-Python: >=3.11
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
License-File: LICENSE
|
|
16
|
+
Requires-Dist: numpy>=2.0
|
|
17
|
+
Requires-Dist: pandas>=2.2.2
|
|
18
|
+
Requires-Dist: xarray>=2024.6
|
|
19
|
+
Provides-Extra: dev
|
|
20
|
+
Requires-Dist: pytest>=7.0; extra == "dev"
|
|
21
|
+
Requires-Dist: flox>=0.9; extra == "dev"
|
|
22
|
+
Requires-Dist: ipython>=8.0; extra == "dev"
|
|
23
|
+
Requires-Dist: dask[array]; extra == "dev"
|
|
24
|
+
Dynamic: license-file
|
|
25
|
+
|
|
26
|
+
# tidy3A
|
|
27
|
+
|
|
28
|
+
Tidy-style verbs for named multidimensional arrays: point clouds, images,
|
|
29
|
+
sensor measurements, scientific and complex-valued arrays. Built on xarray
|
|
30
|
+
and NumPy, in the spirit of [tidy3](https://github.com/rleyvasal/tidy3).
|
|
31
|
+
|
|
32
|
+
**Status:** version `0.1.0` is prepared for the first release, not yet published.
|
|
33
|
+
The design decisions are in [docs/plan.md](docs/plan.md); the rules fit on one page in
|
|
34
|
+
[docs/decisions.md](docs/decisions.md).
|
|
35
|
+
|
|
36
|
+
Requires Python 3.11 or newer. From a checkout, install the package:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
python -m pip install -e .
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Quick start
|
|
43
|
+
|
|
44
|
+
```python
|
|
45
|
+
import numpy as np
|
|
46
|
+
from tidy3a import fields, filter, formula, infer, mutate, read_array
|
|
47
|
+
|
|
48
|
+
points = np.array([[3, 4, 0], [0, 0, 50], [60, 0, 0]], dtype="float32")
|
|
49
|
+
cloud = read_array(
|
|
50
|
+
points,
|
|
51
|
+
dims={"point": infer, "feature": 3},
|
|
52
|
+
labels={"feature": ["x", "y", "z"]},
|
|
53
|
+
field_axis="feature",
|
|
54
|
+
)
|
|
55
|
+
x, y, z, distance = fields("x", "y", "z", "distance")
|
|
56
|
+
|
|
57
|
+
nearby = (
|
|
58
|
+
cloud
|
|
59
|
+
>> mutate(distance = formula(r"\sqrt{x^2 + y^2 + z^2}"))
|
|
60
|
+
>> filter(distance <= 50)
|
|
61
|
+
)
|
|
62
|
+
print(nearby.collect()["distance"].values) # [ 5. 50.]
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
The implemented API includes `read_array`, `read_binary`, `fixed_text`,
|
|
66
|
+
`infer`, `tidy()`, tidy-layer
|
|
67
|
+
verbs (`select`, `filter`, `slice`, `mutate`, `group_by`, `summarise`, joins,
|
|
68
|
+
`pull`, `bind_cols`), array helpers (`dot`, numeric functions, `conj`,
|
|
69
|
+
`rename_axes`, `pack`), and `formula()` for plain mathematical text and a
|
|
70
|
+
strict LaTeX dialect. Scripts declare names with
|
|
71
|
+
`fields()` or write `col("x")`. `collect()` returns a computed xarray
|
|
72
|
+
Dataset; `to_xarray()` returns a shallow copy sharing the underlying values.
|
|
73
|
+
|
|
74
|
+
## Notebooks
|
|
75
|
+
|
|
76
|
+
Install IPython separately to use the notebook extension. In a notebook cell:
|
|
77
|
+
|
|
78
|
+
```ipython
|
|
79
|
+
%load_ext tidy3a.jupyter
|
|
80
|
+
from tidy3a import *
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
The extension enables bare field names inside verbs, so the quick-start
|
|
84
|
+
example can omit the `fields()` declaration. Multi-line pipes need
|
|
85
|
+
parentheses. Sharing a notebook with tidy3 requires its ownership-aware
|
|
86
|
+
transformer; the prerequisite is recorded in [the plan](docs/plan.md#56-notebook-layer).
|
|
87
|
+
|
|
88
|
+
## Execution and first-release scope
|
|
89
|
+
|
|
90
|
+
The first release runs eagerly on NumPy. `read_array` accepts homogeneous
|
|
91
|
+
raw numeric files, `.npy` files, and NumPy arrays. `read_binary` reads mixed
|
|
92
|
+
fixed-width records into separate variables, preserving numeric types and
|
|
93
|
+
strictly decoding UTF-8/ASCII text declared with `fixed_text`. Records can
|
|
94
|
+
be packed or have explicit field offsets and padding; see the
|
|
95
|
+
[reader contract](docs/m6-read-binary.md) for supported options. `tidy()` wraps xarray
|
|
96
|
+
DataArrays and Datasets; optional file readers stay in xarray's ecosystem.
|
|
97
|
+
|
|
98
|
+
`read_array(..., lazy=True)` memory-maps raw and `.npy` files. It does not
|
|
99
|
+
make the verb pipeline lazy: arithmetic can allocate results, filtering evaluates its
|
|
100
|
+
mask, and updating a field label copies its containing array into memory.
|
|
101
|
+
`tidy()` preserves a lazy xarray input initially; `collect()` computes it.
|
|
102
|
+
Dask execution guarantees remain planned.
|
|
103
|
+
|
|
104
|
+
Integer arithmetic follows NumPy, including unsigned subtraction wrapping
|
|
105
|
+
around. Convert before subtracting when negative results are needed, for
|
|
106
|
+
example `nir * 1.0 - red` for uint8 image bands. See
|
|
107
|
+
[first-release limitations](docs/first-release.md) for the supported subset.
|
|
108
|
+
|
|
109
|
+
## Examples and development
|
|
110
|
+
|
|
111
|
+
The point-cloud example runs on real nuScenes Mini data and checks itself
|
|
112
|
+
against plain NumPy:
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
python examples/nuscenes_sweep.py /path/to/nuscenes-mini
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Three public sample datasets (gridded air temperature, ERA-Interim wind, and
|
|
119
|
+
stars' Landsat scene) run against plain xarray; see
|
|
120
|
+
[examples/sample_data.py](examples/sample_data.py) for the files, optional
|
|
121
|
+
reader dependencies, and checks.
|
|
122
|
+
|
|
123
|
+
To run the local tests:
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
python -m pip install -e ".[dev]"
|
|
127
|
+
python -m pytest -q
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Before pushing, run the CI checks in fresh environments with only declared
|
|
131
|
+
dependencies (requires [uv](https://docs.astral.sh/uv/)):
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
scripts/check.sh
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
The [release procedure](docs/releasing.md) describes the GitHub Release
|
|
138
|
+
workflow and the one-time GitHub/PyPI setup.
|
|
139
|
+
|
|
140
|
+
## License
|
|
141
|
+
|
|
142
|
+
MIT
|
tidy3a-0.1.0/README.md
ADDED
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# tidy3A
|
|
2
|
+
|
|
3
|
+
Tidy-style verbs for named multidimensional arrays: point clouds, images,
|
|
4
|
+
sensor measurements, scientific and complex-valued arrays. Built on xarray
|
|
5
|
+
and NumPy, in the spirit of [tidy3](https://github.com/rleyvasal/tidy3).
|
|
6
|
+
|
|
7
|
+
**Status:** version `0.1.0` is prepared for the first release, not yet published.
|
|
8
|
+
The design decisions are in [docs/plan.md](docs/plan.md); the rules fit on one page in
|
|
9
|
+
[docs/decisions.md](docs/decisions.md).
|
|
10
|
+
|
|
11
|
+
Requires Python 3.11 or newer. From a checkout, install the package:
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
python -m pip install -e .
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Quick start
|
|
18
|
+
|
|
19
|
+
```python
|
|
20
|
+
import numpy as np
|
|
21
|
+
from tidy3a import fields, filter, formula, infer, mutate, read_array
|
|
22
|
+
|
|
23
|
+
points = np.array([[3, 4, 0], [0, 0, 50], [60, 0, 0]], dtype="float32")
|
|
24
|
+
cloud = read_array(
|
|
25
|
+
points,
|
|
26
|
+
dims={"point": infer, "feature": 3},
|
|
27
|
+
labels={"feature": ["x", "y", "z"]},
|
|
28
|
+
field_axis="feature",
|
|
29
|
+
)
|
|
30
|
+
x, y, z, distance = fields("x", "y", "z", "distance")
|
|
31
|
+
|
|
32
|
+
nearby = (
|
|
33
|
+
cloud
|
|
34
|
+
>> mutate(distance = formula(r"\sqrt{x^2 + y^2 + z^2}"))
|
|
35
|
+
>> filter(distance <= 50)
|
|
36
|
+
)
|
|
37
|
+
print(nearby.collect()["distance"].values) # [ 5. 50.]
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The implemented API includes `read_array`, `read_binary`, `fixed_text`,
|
|
41
|
+
`infer`, `tidy()`, tidy-layer
|
|
42
|
+
verbs (`select`, `filter`, `slice`, `mutate`, `group_by`, `summarise`, joins,
|
|
43
|
+
`pull`, `bind_cols`), array helpers (`dot`, numeric functions, `conj`,
|
|
44
|
+
`rename_axes`, `pack`), and `formula()` for plain mathematical text and a
|
|
45
|
+
strict LaTeX dialect. Scripts declare names with
|
|
46
|
+
`fields()` or write `col("x")`. `collect()` returns a computed xarray
|
|
47
|
+
Dataset; `to_xarray()` returns a shallow copy sharing the underlying values.
|
|
48
|
+
|
|
49
|
+
## Notebooks
|
|
50
|
+
|
|
51
|
+
Install IPython separately to use the notebook extension. In a notebook cell:
|
|
52
|
+
|
|
53
|
+
```ipython
|
|
54
|
+
%load_ext tidy3a.jupyter
|
|
55
|
+
from tidy3a import *
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
The extension enables bare field names inside verbs, so the quick-start
|
|
59
|
+
example can omit the `fields()` declaration. Multi-line pipes need
|
|
60
|
+
parentheses. Sharing a notebook with tidy3 requires its ownership-aware
|
|
61
|
+
transformer; the prerequisite is recorded in [the plan](docs/plan.md#56-notebook-layer).
|
|
62
|
+
|
|
63
|
+
## Execution and first-release scope
|
|
64
|
+
|
|
65
|
+
The first release runs eagerly on NumPy. `read_array` accepts homogeneous
|
|
66
|
+
raw numeric files, `.npy` files, and NumPy arrays. `read_binary` reads mixed
|
|
67
|
+
fixed-width records into separate variables, preserving numeric types and
|
|
68
|
+
strictly decoding UTF-8/ASCII text declared with `fixed_text`. Records can
|
|
69
|
+
be packed or have explicit field offsets and padding; see the
|
|
70
|
+
[reader contract](docs/m6-read-binary.md) for supported options. `tidy()` wraps xarray
|
|
71
|
+
DataArrays and Datasets; optional file readers stay in xarray's ecosystem.
|
|
72
|
+
|
|
73
|
+
`read_array(..., lazy=True)` memory-maps raw and `.npy` files. It does not
|
|
74
|
+
make the verb pipeline lazy: arithmetic can allocate results, filtering evaluates its
|
|
75
|
+
mask, and updating a field label copies its containing array into memory.
|
|
76
|
+
`tidy()` preserves a lazy xarray input initially; `collect()` computes it.
|
|
77
|
+
Dask execution guarantees remain planned.
|
|
78
|
+
|
|
79
|
+
Integer arithmetic follows NumPy, including unsigned subtraction wrapping
|
|
80
|
+
around. Convert before subtracting when negative results are needed, for
|
|
81
|
+
example `nir * 1.0 - red` for uint8 image bands. See
|
|
82
|
+
[first-release limitations](docs/first-release.md) for the supported subset.
|
|
83
|
+
|
|
84
|
+
## Examples and development
|
|
85
|
+
|
|
86
|
+
The point-cloud example runs on real nuScenes Mini data and checks itself
|
|
87
|
+
against plain NumPy:
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
python examples/nuscenes_sweep.py /path/to/nuscenes-mini
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Three public sample datasets (gridded air temperature, ERA-Interim wind, and
|
|
94
|
+
stars' Landsat scene) run against plain xarray; see
|
|
95
|
+
[examples/sample_data.py](examples/sample_data.py) for the files, optional
|
|
96
|
+
reader dependencies, and checks.
|
|
97
|
+
|
|
98
|
+
To run the local tests:
|
|
99
|
+
|
|
100
|
+
```bash
|
|
101
|
+
python -m pip install -e ".[dev]"
|
|
102
|
+
python -m pytest -q
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Before pushing, run the CI checks in fresh environments with only declared
|
|
106
|
+
dependencies (requires [uv](https://docs.astral.sh/uv/)):
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
scripts/check.sh
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
The [release procedure](docs/releasing.md) describes the GitHub Release
|
|
113
|
+
workflow and the one-time GitHub/PyPI setup.
|
|
114
|
+
|
|
115
|
+
## License
|
|
116
|
+
|
|
117
|
+
MIT
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# tidy3A tidy-layer rules
|
|
2
|
+
|
|
3
|
+
The one-page rule set from plan §1.1: if the tidy layer's rules stop fitting on this page, simplify the layer rather than adding rules. Details, reasons, and precedents are in `docs/plan.md`, with section references below.
|
|
4
|
+
|
|
5
|
+
**Data.** A dataset holds variables (named arrays). Each axis is either the one **field axis**, declared with `field_axis=`, whose labels act as column names, or an **observation axis**, whose labels are values. Without `field_axis=` there are only variables. `variables_from=` turns a labeled axis into one variable per label instead. (§3, §4.1)
|
|
6
|
+
|
|
7
|
+
**Names.** In verbs and expressions, a name means, in order: a variable, then a field-axis label. A field label alone, `x`, is that field with the field axis dropped; a list `[x, y, z]` keeps the field axis, narrowed, and also rebuilds variables made by the same `variables_from=` import into their recorded axis. Other variables are packed with `pack([...], along=...)`. A name on an observation axis means that axis's labels (`filter(time > t0)`). A name that is both data and a notebook value is an error: write `col("x")` or `env("x")`. A name that is both a variable and a field label is an error: write `variable("x")` or `label("x")`. Scripts use strings or `fields("x", …)`. (§5.1, §5.2)
|
|
8
|
+
|
|
9
|
+
**Verbs never reshape on their own.** `select` keeps variables and field labels (columns), with tidy3's selection helpers. `filter` keeps positions on observation axes (rows); every predicate must vary along one axis, and predicates on different axes combine with AND into a rectangle. `slice` keeps positions by 1-based position and keeps the axis unless `drop=True`. Filtering keeps original positions as labels. Empty results are returned. (§5.1, §5.3)
|
|
10
|
+
|
|
11
|
+
**Mutate.** Assignments run in order. Writing to a field label writes into the shared array, promoting its type if needed. Vectors are written in one assignment, `[x, y, z] = …`. Masking is `if_else`. (§5.4)
|
|
12
|
+
|
|
13
|
+
**Grouping.** `group_by` on an axis keeps it; on a value field, or `bins(...)` of one, it groups that field's one axis into a new axis of sorted values or intervals. `summarise` reduces every other observation axis and never the field axis. A summary using a variable without the grouping axis is an error. Grouped summaries inside `mutate` broadcast back. Missing values propagate unless `na_rm=True`. After `summarise`, the last grouping level is dropped (`groups=`). (§5.4)
|
|
14
|
+
|
|
15
|
+
**Alignment.** Labels on a shared axis must match exactly; the same labels in another order are reordered and reported. Different labels need a join verb (`left_join`, `inner_join`, `full_join`, with `_x`/`_y` suffixes; `tolerance=` is currently supported only on `left_join`). A labeled axis never pairs with an unlabeled one by length; attach outside arrays with `bind_cols(…, along=)`, and read labels with `pull(axis)`. (§5.5)
|
|
16
|
+
|
|
17
|
+
**Array layer.** Maths helpers name their axes: `dot(…, over=)`, `sum(…, over=)`, `rename_axes(…)`; `dot` never conjugates. `formula()` reads a strict LaTeX dialect into the same expressions, binds index letters to axes by name or `bind=`, and always shows how it was read. (§6)
|
|
18
|
+
|
|
19
|
+
**Results.** `read_array` and `tidy()` return the wrapper. `collect()` returns a computed xarray Dataset; `to_xarray()` returns the underlying xarray object. (§3, §4.1)
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# First-release scope and limitations
|
|
2
|
+
|
|
3
|
+
Version `0.1.0` is the first release.
|
|
4
|
+
The release notes are in [the changelog](../CHANGELOG.md). Progress and verification
|
|
5
|
+
are recorded in [milestone 7](plan.md#milestone-7--independent-first-release).
|
|
6
|
+
|
|
7
|
+
## Supported scope
|
|
8
|
+
|
|
9
|
+
- Python 3.11 or newer, NumPy 2.0 or newer, pandas 2.2.2 or newer, and
|
|
10
|
+
xarray 2024.6 or newer.
|
|
11
|
+
- Homogeneous raw numeric files, `.npy` files, and in-memory NumPy arrays
|
|
12
|
+
through `read_array`; xarray objects through `tidy()`.
|
|
13
|
+
- Mixed fixed-width numeric/text records through `read_binary`, with
|
|
14
|
+
explicit padding layouts and strict UTF-8/ASCII `fixed_text` decoding.
|
|
15
|
+
The supported reader options are listed in [the contract](m6-read-binary.md).
|
|
16
|
+
- Tidy verbs, exact label alignment, explicit joins, named-axis mathematics,
|
|
17
|
+
complex values, and the implemented plain-text and LaTeX formula dialect.
|
|
18
|
+
- Script expressions with `fields()` or `col()`, and optional IPython
|
|
19
|
+
shorthand through `%load_ext tidy3a.jupyter`.
|
|
20
|
+
|
|
21
|
+
## Memory and execution
|
|
22
|
+
|
|
23
|
+
NumPy-backed operations run eagerly. `lazy=True` on `read_array` creates a
|
|
24
|
+
read-only memory map; it does not defer the whole pipeline. Arithmetic can
|
|
25
|
+
allocate arrays, filtering evaluates its predicate mask, and field-label
|
|
26
|
+
updates copy the whole containing array, even when changing one field.
|
|
27
|
+
`pull()` returns NumPy values and materializes the requested data.
|
|
28
|
+
|
|
29
|
+
`tidy()` does not itself compute a lazy xarray input. Selection, regular
|
|
30
|
+
positional slicing, arithmetic, and reductions on Dask-backed or file-backed
|
|
31
|
+
xarray data stay lazy and are checked in the test suite
|
|
32
|
+
([metadata and lazy input](m6-metadata-lazy.md)). Grouping by field values
|
|
33
|
+
or bins reads the grouping values, filtering reads the mask, and field-label
|
|
34
|
+
write-back reads and copies the containing array. `median(..., na_rm=True)`
|
|
35
|
+
over every axis of a chunked array is an error that asks you to collect
|
|
36
|
+
first. `collect()` computes the resulting Dataset. `to_xarray()` makes a
|
|
37
|
+
shallow copy without computing it. Dask-backed grouping and performance
|
|
38
|
+
guarantees are not part of this release; memory-mapping alone does not
|
|
39
|
+
promise bounded memory use.
|
|
40
|
+
|
|
41
|
+
## Known limitations
|
|
42
|
+
|
|
43
|
+
- Grouping supports multiple axis keys but only one value-field or bin key.
|
|
44
|
+
Grouped `slice` and its head/tail helpers require `ungroup()` first.
|
|
45
|
+
- Reductions with `over=` cannot be combined with grouping by a value field.
|
|
46
|
+
A summary cannot reference an earlier result in the same `summarise`;
|
|
47
|
+
derive from summary results in a following `mutate`.
|
|
48
|
+
- `tolerance=` is implemented only for `left_join`. Other joins use exact
|
|
49
|
+
labels. `case_when` is not implemented; use `if_else` for binary choices.
|
|
50
|
+
- xarray on the left of an arithmetic operator does not defer to a tidy3A
|
|
51
|
+
expression. Write `x * R` rather than `R * x`, or use `dot()`.
|
|
52
|
+
- Integer arithmetic follows NumPy. For unsigned data, promote before a
|
|
53
|
+
subtraction that may be negative: `nir * 1.0 - red`.
|
|
54
|
+
- Formulas support the dialect documented in [plan §6.3](plan.md#63-formula), with
|
|
55
|
+
errors for unsupported syntax. Product reductions, integrals, sums over
|
|
56
|
+
more than one index letter at once, and implicit multiplication in plain
|
|
57
|
+
text are not supported. Write `2*x` rather than `2x` in plain text.
|
|
58
|
+
- Notebook pipes spanning lines need parentheses; tidy3-style backtick
|
|
59
|
+
names are not supported. Use `col("field name")` for such names.
|
|
60
|
+
- Coexistence with tidy3's notebook extension requires tidy3's ownership
|
|
61
|
+
check (`_owned_by_tidy3`); the prerequisite change is recorded in the
|
|
62
|
+
implementation plan. The coexistence tests skip when it is unavailable.
|
|
63
|
+
|
|
64
|
+
Mixed-record reading, metadata handling, and the lazy-input checks were
|
|
65
|
+
brought forward from milestone 6 before publication. Dedicated NetCDF/Zarr
|
|
66
|
+
readers, Dask-backed grouping, text encodings beyond UTF-8/ASCII, and
|
|
67
|
+
variable-length records remain planned. Existing
|
|
68
|
+
xarray readers can already be used with `tidy()`; their optional dependencies
|
|
69
|
+
are installed separately.
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
# Milestone 0: acceptance examples in two styles
|
|
2
|
+
|
|
3
|
+
Status: prototype comparison for review. None of the tidy3A code below is implemented. Each example's plain-xarray equivalent runs in `prototypes/test_examples.py` on small synthetic data with hand-checked answers, and `prototypes/test_xarray_assumptions.py` checks the xarray and NumPy behavior the plan relies on. Both pass (39 tests, xarray 2026.9.0, NumPy 2.5.4).
|
|
4
|
+
|
|
5
|
+
The question this answers (plan §5.1, milestone 0): does `select` with a declared **field axis** read better than the **stars style**, where fields become separate variables (stars' `split`, tidy3A's `variables_from=`) and `select` picks only variables?
|
|
6
|
+
|
|
7
|
+
- **Style A, field axis:** `read_array(..., field_axis="feature")`. One array; field labels act as columns.
|
|
8
|
+
- **Style B, stars style:** `read_array(..., variables_from="feature")`. One variable per field, as after stars' `split(x, "feature")`. This comparison was written with a re-packing helper, `merge(x, y, z, along="feature")` after stars' `merge()`; since decided: `[x, y, z]` rebuilds the recorded axis (plan §5.2), so Style B's vector line now matches Style A.
|
|
9
|
+
- **Style C, strict without splitting:** no field axis and no split. Field labels are reached only through `filter` and xarray. Shown once, for comparison.
|
|
10
|
+
|
|
11
|
+
## Example 1: point cloud
|
|
12
|
+
|
|
13
|
+
Synthetic data: four points that, after a 90° rotation about z and a 1 m offset, lie 49 m, 50 m, 51 m, and 3 m from the origin; `intensity` 10, 20, 30, 40; `ring` 1, 2, 1, 2.
|
|
14
|
+
|
|
15
|
+
**Style A**
|
|
16
|
+
|
|
17
|
+
```python
|
|
18
|
+
cloud = read_array(
|
|
19
|
+
path,
|
|
20
|
+
value_type="float32",
|
|
21
|
+
dims={"point": infer, "feature": 5},
|
|
22
|
+
labels={"feature": ["x", "y", "z", "intensity", "ring"]},
|
|
23
|
+
field_axis="feature",
|
|
24
|
+
)
|
|
25
|
+
|
|
26
|
+
nearby = (
|
|
27
|
+
cloud
|
|
28
|
+
>> mutate(p = dot(R, [x, y, z], over="feature") + offset)
|
|
29
|
+
>> mutate(distance = formula(r"\sqrt{\sum_i p_i^2}"))
|
|
30
|
+
>> filter(distance <= 50)
|
|
31
|
+
)
|
|
32
|
+
|
|
33
|
+
nearby >> group_by(ring) >> summarise(mean_intensity = mean(intensity))
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
**Style B**
|
|
37
|
+
|
|
38
|
+
```python
|
|
39
|
+
cloud = read_array(
|
|
40
|
+
path,
|
|
41
|
+
value_type="float32",
|
|
42
|
+
dims={"point": infer, "feature": 5},
|
|
43
|
+
labels={"feature": ["x", "y", "z", "intensity", "ring"]},
|
|
44
|
+
variables_from="feature",
|
|
45
|
+
)
|
|
46
|
+
|
|
47
|
+
nearby = (
|
|
48
|
+
cloud
|
|
49
|
+
>> mutate(p = dot(R, [x, y, z], over="feature") + offset) # rebuilds the recorded feature axis
|
|
50
|
+
>> mutate(distance = formula(r"\sqrt{\sum_i p_i^2}"))
|
|
51
|
+
>> filter(distance <= 50)
|
|
52
|
+
)
|
|
53
|
+
|
|
54
|
+
nearby >> group_by(ring) >> summarise(mean_intensity = mean(intensity))
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
**Style C**
|
|
58
|
+
|
|
59
|
+
```python
|
|
60
|
+
xyz = cloud >> filter(feature.is_in(["x", "y", "z"]))
|
|
61
|
+
# Bare x, intensity, and ring do not exist; expressions need xarray:
|
|
62
|
+
# cloud.to_xarray()["values"].sel(feature="ring")
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Expected (checked): kept points 1, 2, 4 with original positions as labels, since 50 m is inside the boundary; intensities 10, 20, 40; mean intensity per ring: ring 1 → 10, ring 2 → 30.
|
|
66
|
+
|
|
67
|
+
## Example 2: sensor time series
|
|
68
|
+
|
|
69
|
+
Synthetic data: `values (time: 4, sensor: front rear, feature: x y z)`.
|
|
70
|
+
|
|
71
|
+
| Operation | Style A | Style B |
|
|
72
|
+
|---|---|---|
|
|
73
|
+
| Mean x per sensor | `group_by(sensor) >> summarise(mean_x = mean(x))` | same |
|
|
74
|
+
| All fields per sensor | `group_by(sensor) >> summarise(m = mean(values))` → `m (sensor, feature)` | one summary per field, e.g. `summarise(mx = mean(x), my = mean(y), mz = mean(z))` |
|
|
75
|
+
| Front sensor after time 2 | `filter(sensor == "front", time > 2)` | same |
|
|
76
|
+
| Times where any sensor has x > 20 | `filter(any(x > 20, over="sensor"))` | same |
|
|
77
|
+
|
|
78
|
+
Expected (checked): mean x is 4 (front) and 40 (rear); the front sensor's field means are x 4, y 3, z 0.5; the two-axis filter keeps `time: 2, sensor: 1`; the `any` filter keeps times 2, 3, 4.
|
|
79
|
+
|
|
80
|
+
## Example 3: geospatial raster
|
|
81
|
+
|
|
82
|
+
Synthetic data in the shape stars handles: a 3 × 2 pixel scene, bands `blue green red nir`, two dates, with `x`/`y` map coordinates. Near-infrared halves at the second date.
|
|
83
|
+
|
|
84
|
+
**Style A** (`field_axis="band"`)
|
|
85
|
+
|
|
86
|
+
```python
|
|
87
|
+
scene >> mutate(ndvi = (nir - red) / (nir + red))
|
|
88
|
+
scene >> filter(x > 500010, y > 4100010) # crop, as stars' filter
|
|
89
|
+
scene >> mutate(ndvi = (nir - red) / (nir + red)) >> group_by(x, y) >> summarise(mean_ndvi = mean(ndvi))
|
|
90
|
+
scene >> summarise(m = mean(values)) # one mean per band; stars: st_apply(x, "band", mean)
|
|
91
|
+
scene >> mutate(ndvi = …) >> group_by(bins(ndvi, [-1, 0.25, 1])) >> summarise(pixels = n())
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
**Style B** (`variables_from="band"`)
|
|
95
|
+
|
|
96
|
+
```python
|
|
97
|
+
scene >> mutate(ndvi = (nir - red) / (nir + red)) # same
|
|
98
|
+
scene >> filter(x > 500010, y > 4100010) # same
|
|
99
|
+
# per-pixel mean NDVI and bins: same
|
|
100
|
+
scene >> summarise(blue = mean(blue), green = mean(green), # one summary per band
|
|
101
|
+
red = mean(red), nir = mean(nir))
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
Expected (checked): NDVI at the first date is 0.5, 0, 0, 0.75, 0.5, 0 by pixel; the crop keeps `x: 2, y: 1`; the first pixel's mean NDVI over time is (0.5 + 0.2) / 2; the per-band mean of blue is 0.05; the NDVI bins hold 3 and 3 pixels.
|
|
105
|
+
|
|
106
|
+
## Comparison
|
|
107
|
+
|
|
108
|
+
| Aspect | Style A, field axis | Style B, stars style | Style C, strict |
|
|
109
|
+
|---|---|---|---|
|
|
110
|
+
| Scalar expressions (`distance <= 50`, NDVI, `mean(x)`) | Bare names | Bare names, identical text | Not available without xarray |
|
|
111
|
+
| Vector operations (rotation) | `[x, y, z]` directly | `[x, y, z]` rebuilds the recorded axis (decided after this comparison; it first needed a `merge` helper) | xarray |
|
|
112
|
+
| One summary for every field | `mean(values)` keeps the field axis | One expression per field, or an `across()`-style helper | xarray |
|
|
113
|
+
| Fields with different types (`ring` as integer) | Not possible: one array, one type | Natural | Not possible |
|
|
114
|
+
| Concepts a user must learn | Field axis, observation axis | Separate variables only | Separate variables only |
|
|
115
|
+
| Rules in the tidy layer | More: field-axis select, `var()`/`label()` collisions, write-back, type promotion | Fewer | Fewest |
|
|
116
|
+
|
|
117
|
+
Findings:
|
|
118
|
+
|
|
119
|
+
1. **Style C is clearly worst.** Without either a field axis or splitting, the most common point-cloud and raster operations leave the pipeline. It is not a real option.
|
|
120
|
+
2. **Styles A and B read the same for scalar work.** Filters, NDVI, grouping, and bins are character-for-character identical. The choice of style changes nothing for most expressions.
|
|
121
|
+
3. **Style A wins where fields form a vector or a spectrum:** rotation (`[x, y, z]` versus `merge(...)`), and one summary for every band or component (`mean(values)` versus one line per field).
|
|
122
|
+
4. **Style B wins where fields are heterogeneous:** different types and units, such as nuScenes' `ring`, and it needs fewer rules.
|
|
123
|
+
5. Both styles are already in the plan, as `field_axis=` and `variables_from=`. The comparison supports keeping both, with documented guidance: use `field_axis=` when fields are components of one quantity (`x y z`, spectral bands, RGB), and `variables_from=` when they are different quantities.
|
|
124
|
+
|
|
125
|
+
Decided (2026-10-24): keep the field axis, with `variables_from=` equally first-class (plan §5.1). The recommendation as reviewed: keep the field axis. Its advantage is real but narrow (vectors and spectra), so the documentation should present `variables_from=` as equally first-class, and milestone 2 should check the tidy layer's one-page complexity budget (`docs/decisions.md`) before adding more field-axis rules.
|
|
126
|
+
|
|
127
|
+
Re-packing, found while writing Style B and since decided (plan §5.2): `[x, y, z]` rebuilds a vector from variables of the same recorded `variables_from` axis, and `pack([...], along=...)` re-packs other variables, with the axis always named. `merge` was avoided because `xr.merge` implements joins.
|