tidy3a 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (49) hide show
  1. tidy3a-0.1.0/CHANGELOG.md +55 -0
  2. tidy3a-0.1.0/LICENSE +21 -0
  3. tidy3a-0.1.0/MANIFEST.in +5 -0
  4. tidy3a-0.1.0/PKG-INFO +142 -0
  5. tidy3a-0.1.0/README.md +117 -0
  6. tidy3a-0.1.0/docs/decisions.md +19 -0
  7. tidy3a-0.1.0/docs/first-release.md +69 -0
  8. tidy3a-0.1.0/docs/m0-examples.md +127 -0
  9. tidy3a-0.1.0/docs/m6-metadata-lazy.md +143 -0
  10. tidy3a-0.1.0/docs/m6-read-binary.md +137 -0
  11. tidy3a-0.1.0/docs/plan.md +1253 -0
  12. tidy3a-0.1.0/docs/releasing.md +52 -0
  13. tidy3a-0.1.0/examples/nuscenes_sweep.py +135 -0
  14. tidy3a-0.1.0/examples/sample_data.py +185 -0
  15. tidy3a-0.1.0/prototypes/test_examples.py +213 -0
  16. tidy3a-0.1.0/prototypes/test_xarray_assumptions.py +258 -0
  17. tidy3a-0.1.0/pyproject.toml +52 -0
  18. tidy3a-0.1.0/scripts/check.sh +65 -0
  19. tidy3a-0.1.0/setup.cfg +4 -0
  20. tidy3a-0.1.0/src/tidy3a/__init__.py +114 -0
  21. tidy3a-0.1.0/src/tidy3a/__version__.py +1 -0
  22. tidy3a-0.1.0/src/tidy3a/align.py +136 -0
  23. tidy3a-0.1.0/src/tidy3a/errors.py +45 -0
  24. tidy3a-0.1.0/src/tidy3a/expr.py +911 -0
  25. tidy3a-0.1.0/src/tidy3a/formula.py +920 -0
  26. tidy3a-0.1.0/src/tidy3a/frame.py +243 -0
  27. tidy3a-0.1.0/src/tidy3a/io.py +598 -0
  28. tidy3a-0.1.0/src/tidy3a/jupyter.py +314 -0
  29. tidy3a-0.1.0/src/tidy3a/linalg.py +192 -0
  30. tidy3a-0.1.0/src/tidy3a/py.typed +0 -0
  31. tidy3a-0.1.0/src/tidy3a/selectors.py +141 -0
  32. tidy3a-0.1.0/src/tidy3a/verbs.py +782 -0
  33. tidy3a-0.1.0/src/tidy3a.egg-info/PKG-INFO +142 -0
  34. tidy3a-0.1.0/src/tidy3a.egg-info/SOURCES.txt +47 -0
  35. tidy3a-0.1.0/src/tidy3a.egg-info/dependency_links.txt +1 -0
  36. tidy3a-0.1.0/src/tidy3a.egg-info/requires.txt +9 -0
  37. tidy3a-0.1.0/src/tidy3a.egg-info/top_level.txt +1 -0
  38. tidy3a-0.1.0/tests/test_binary.py +362 -0
  39. tidy3a-0.1.0/tests/test_dask.py +169 -0
  40. tidy3a-0.1.0/tests/test_formula.py +230 -0
  41. tidy3a-0.1.0/tests/test_frame.py +131 -0
  42. tidy3a-0.1.0/tests/test_io.py +387 -0
  43. tidy3a-0.1.0/tests/test_linalg.py +268 -0
  44. tidy3a-0.1.0/tests/test_metadata.py +173 -0
  45. tidy3a-0.1.0/tests/test_notebooks.py +178 -0
  46. tidy3a-0.1.0/tests/test_nuscenes.py +28 -0
  47. tidy3a-0.1.0/tests/test_sample_data.py +33 -0
  48. tidy3a-0.1.0/tests/test_verbs.py +716 -0
  49. tidy3a-0.1.0/tests/test_workflow.py +62 -0
@@ -0,0 +1,55 @@
1
+ # Changelog
2
+
3
+ ## 0.1.0 — 2026-10-24
4
+
5
+ First release.
6
+
7
+ - `read_array` reads raw numeric files, `.npy` files, and in-memory NumPy
8
+ arrays, with `infer`, labels, `field_axis=`, `variables_from=`, byte
9
+ order, storage order, header offsets, and memory-mapping.
10
+ - `read_binary` reads packed or explicitly padded mixed records into
11
+ separate typed variables. `fixed_text` declares strict UTF-8/ASCII text
12
+ decoding and padding; numeric fields retain their widths and kinds.
13
+ - `tidy()` wraps xarray objects; `TidyArray` reports its axes, labels, axis
14
+ roles, and anything the reader inferred.
15
+ - Tidy-layer verbs: `select` with tidy3's selection helpers, `filter`,
16
+ `slice`/`slice_head`/`slice_tail`, `mutate` with write-back and vector
17
+ assignment, `group_by` by axis, value, or `bins()`, `summarise`,
18
+ `ungroup`, `pull`, `bind_cols`, and `left_join`/`inner_join`/`full_join`.
19
+ - Expressions: `col`, `variable`, `label`, `env`, `fields`, `[x, y, z]`
20
+ vectors, `if_else`, and reductions `mean`, `sum`, `min`, `max`, `median`,
21
+ `std`, `var` (variance, as in tidy3 and dplyr), `any`, and `all` with
22
+ `over=` and `na_rm=`, plus `n()` for observation counts.
23
+ - Exact label alignment, with reordering by label and explicit joins.
24
+ - Metadata: wrapping keeps attributes and encodings; new and computed
25
+ results clear stale serialization encoding and inherited attributes,
26
+ direct aliases keep theirs, and joins report dropped conflicting
27
+ attributes.
28
+ - Lazy input: NetCDF and Dask-backed data stay lazy through inspection,
29
+ selection, regular positional slicing, arithmetic, and reductions until
30
+ `collect()`; grouping, filtering, and field write-back read the values
31
+ they need, as documented.
32
+ - Array layer: `dot` with `over=`, elementwise numeric functions, `conj`,
33
+ `real`, `imag`, `angle`, `arctan2`, `rename_axes`, and `pack`.
34
+ - `formula()`: plain mathematical text and a strict LaTeX dialect with
35
+ plot3's conventions, subscripts as axis indices, labels, and positions,
36
+ `\sum` with optional checked bounds, `bind=` for index letters and
37
+ values, and show-back of how each formula was read.
38
+ - Notebook layer (`%load_ext tidy3a.jupyter`): bare names in tidy3A verbs,
39
+ settled when the verb runs, with an error when a name is both data and a
40
+ notebook variable; `[x, y, z] = …` inside `mutate`; only tidy3A's own
41
+ calls are rewritten, so tidy3 and tidy3A share a notebook.
42
+ - `n()` counts observations on the axes every non-scalar variable shares.
43
+ - Inspection output shows time labels to the second (or as dates) and
44
+ floats to 10 significant digits, with a gap before each axis role.
45
+ - Examples on real data: a nuScenes Mini LiDAR sweep, and three public
46
+ sample datasets (air temperature, ERA-Interim wind, a Landsat scene),
47
+ each checked against plain NumPy or xarray.
48
+ - Source distributions include documentation, examples, and prototype
49
+ tests. Packaging checks build a wheel from the source archive and
50
+ validate both distributions' metadata. The build requires setuptools
51
+ 77.0.3 or newer for the license metadata.
52
+ - Requires Python 3.11+, NumPy 2.0+, pandas 2.2.2+ (now declared, as
53
+ tidy3A imports it directly), and xarray 2024.6+.
54
+ - Publishing a GitHub Release runs the version check and reusable CI gate,
55
+ then uploads the validated distributions to PyPI through trusted publishing.
tidy3a-0.1.0/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 rleyvasal
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,5 @@
1
+ include CHANGELOG.md
2
+ include scripts/check.sh
3
+ recursive-include docs *.md
4
+ recursive-include examples *.py
5
+ recursive-include prototypes *.py
tidy3a-0.1.0/PKG-INFO ADDED
@@ -0,0 +1,142 @@
1
+ Metadata-Version: 2.4
2
+ Name: tidy3a
3
+ Version: 0.1.0
4
+ Summary: Tidy-style verbs for named multidimensional arrays, built on xarray
5
+ Author: Rigoberto Leyva Salmeron
6
+ License-Expression: MIT
7
+ Project-URL: Homepage, https://github.com/rleyvasal/tidy3a
8
+ Keywords: xarray,arrays,tidyverse,dplyr,point-cloud,raster
9
+ Classifier: Development Status :: 2 - Pre-Alpha
10
+ Classifier: Intended Audience :: Science/Research
11
+ Classifier: Programming Language :: Python :: 3
12
+ Classifier: Topic :: Scientific/Engineering
13
+ Requires-Python: >=3.11
14
+ Description-Content-Type: text/markdown
15
+ License-File: LICENSE
16
+ Requires-Dist: numpy>=2.0
17
+ Requires-Dist: pandas>=2.2.2
18
+ Requires-Dist: xarray>=2024.6
19
+ Provides-Extra: dev
20
+ Requires-Dist: pytest>=7.0; extra == "dev"
21
+ Requires-Dist: flox>=0.9; extra == "dev"
22
+ Requires-Dist: ipython>=8.0; extra == "dev"
23
+ Requires-Dist: dask[array]; extra == "dev"
24
+ Dynamic: license-file
25
+
26
+ # tidy3A
27
+
28
+ Tidy-style verbs for named multidimensional arrays: point clouds, images,
29
+ sensor measurements, scientific and complex-valued arrays. Built on xarray
30
+ and NumPy, in the spirit of [tidy3](https://github.com/rleyvasal/tidy3).
31
+
32
+ **Status:** version `0.1.0` is prepared for the first release, not yet published.
33
+ The design decisions are in [docs/plan.md](docs/plan.md); the rules fit on one page in
34
+ [docs/decisions.md](docs/decisions.md).
35
+
36
+ Requires Python 3.11 or newer. From a checkout, install the package:
37
+
38
+ ```bash
39
+ python -m pip install -e .
40
+ ```
41
+
42
+ ## Quick start
43
+
44
+ ```python
45
+ import numpy as np
46
+ from tidy3a import fields, filter, formula, infer, mutate, read_array
47
+
48
+ points = np.array([[3, 4, 0], [0, 0, 50], [60, 0, 0]], dtype="float32")
49
+ cloud = read_array(
50
+ points,
51
+ dims={"point": infer, "feature": 3},
52
+ labels={"feature": ["x", "y", "z"]},
53
+ field_axis="feature",
54
+ )
55
+ x, y, z, distance = fields("x", "y", "z", "distance")
56
+
57
+ nearby = (
58
+ cloud
59
+ >> mutate(distance = formula(r"\sqrt{x^2 + y^2 + z^2}"))
60
+ >> filter(distance <= 50)
61
+ )
62
+ print(nearby.collect()["distance"].values) # [ 5. 50.]
63
+ ```
64
+
65
+ The implemented API includes `read_array`, `read_binary`, `fixed_text`,
66
+ `infer`, `tidy()`, tidy-layer
67
+ verbs (`select`, `filter`, `slice`, `mutate`, `group_by`, `summarise`, joins,
68
+ `pull`, `bind_cols`), array helpers (`dot`, numeric functions, `conj`,
69
+ `rename_axes`, `pack`), and `formula()` for plain mathematical text and a
70
+ strict LaTeX dialect. Scripts declare names with
71
+ `fields()` or write `col("x")`. `collect()` returns a computed xarray
72
+ Dataset; `to_xarray()` returns a shallow copy sharing the underlying values.
73
+
74
+ ## Notebooks
75
+
76
+ Install IPython separately to use the notebook extension. In a notebook cell:
77
+
78
+ ```ipython
79
+ %load_ext tidy3a.jupyter
80
+ from tidy3a import *
81
+ ```
82
+
83
+ The extension enables bare field names inside verbs, so the quick-start
84
+ example can omit the `fields()` declaration. Multi-line pipes need
85
+ parentheses. Sharing a notebook with tidy3 requires its ownership-aware
86
+ transformer; the prerequisite is recorded in [the plan](docs/plan.md#56-notebook-layer).
87
+
88
+ ## Execution and first-release scope
89
+
90
+ The first release runs eagerly on NumPy. `read_array` accepts homogeneous
91
+ raw numeric files, `.npy` files, and NumPy arrays. `read_binary` reads mixed
92
+ fixed-width records into separate variables, preserving numeric types and
93
+ strictly decoding UTF-8/ASCII text declared with `fixed_text`. Records can
94
+ be packed or have explicit field offsets and padding; see the
95
+ [reader contract](docs/m6-read-binary.md) for supported options. `tidy()` wraps xarray
96
+ DataArrays and Datasets; optional file readers stay in xarray's ecosystem.
97
+
98
+ `read_array(..., lazy=True)` memory-maps raw and `.npy` files. It does not
99
+ make the verb pipeline lazy: arithmetic can allocate results, filtering evaluates its
100
+ mask, and updating a field label copies its containing array into memory.
101
+ `tidy()` preserves a lazy xarray input initially; `collect()` computes it.
102
+ Dask execution guarantees remain planned.
103
+
104
+ Integer arithmetic follows NumPy, including unsigned subtraction wrapping
105
+ around. Convert before subtracting when negative results are needed, for
106
+ example `nir * 1.0 - red` for uint8 image bands. See
107
+ [first-release limitations](docs/first-release.md) for the supported subset.
108
+
109
+ ## Examples and development
110
+
111
+ The point-cloud example runs on real nuScenes Mini data and checks itself
112
+ against plain NumPy:
113
+
114
+ ```bash
115
+ python examples/nuscenes_sweep.py /path/to/nuscenes-mini
116
+ ```
117
+
118
+ Three public sample datasets (gridded air temperature, ERA-Interim wind, and
119
+ stars' Landsat scene) run against plain xarray; see
120
+ [examples/sample_data.py](examples/sample_data.py) for the files, optional
121
+ reader dependencies, and checks.
122
+
123
+ To run the local tests:
124
+
125
+ ```bash
126
+ python -m pip install -e ".[dev]"
127
+ python -m pytest -q
128
+ ```
129
+
130
+ Before pushing, run the CI checks in fresh environments with only declared
131
+ dependencies (requires [uv](https://docs.astral.sh/uv/)):
132
+
133
+ ```bash
134
+ scripts/check.sh
135
+ ```
136
+
137
+ The [release procedure](docs/releasing.md) describes the GitHub Release
138
+ workflow and the one-time GitHub/PyPI setup.
139
+
140
+ ## License
141
+
142
+ MIT
tidy3a-0.1.0/README.md ADDED
@@ -0,0 +1,117 @@
1
+ # tidy3A
2
+
3
+ Tidy-style verbs for named multidimensional arrays: point clouds, images,
4
+ sensor measurements, scientific and complex-valued arrays. Built on xarray
5
+ and NumPy, in the spirit of [tidy3](https://github.com/rleyvasal/tidy3).
6
+
7
+ **Status:** version `0.1.0` is prepared for the first release, not yet published.
8
+ The design decisions are in [docs/plan.md](docs/plan.md); the rules fit on one page in
9
+ [docs/decisions.md](docs/decisions.md).
10
+
11
+ Requires Python 3.11 or newer. From a checkout, install the package:
12
+
13
+ ```bash
14
+ python -m pip install -e .
15
+ ```
16
+
17
+ ## Quick start
18
+
19
+ ```python
20
+ import numpy as np
21
+ from tidy3a import fields, filter, formula, infer, mutate, read_array
22
+
23
+ points = np.array([[3, 4, 0], [0, 0, 50], [60, 0, 0]], dtype="float32")
24
+ cloud = read_array(
25
+ points,
26
+ dims={"point": infer, "feature": 3},
27
+ labels={"feature": ["x", "y", "z"]},
28
+ field_axis="feature",
29
+ )
30
+ x, y, z, distance = fields("x", "y", "z", "distance")
31
+
32
+ nearby = (
33
+ cloud
34
+ >> mutate(distance = formula(r"\sqrt{x^2 + y^2 + z^2}"))
35
+ >> filter(distance <= 50)
36
+ )
37
+ print(nearby.collect()["distance"].values) # [ 5. 50.]
38
+ ```
39
+
40
+ The implemented API includes `read_array`, `read_binary`, `fixed_text`,
41
+ `infer`, `tidy()`, tidy-layer
42
+ verbs (`select`, `filter`, `slice`, `mutate`, `group_by`, `summarise`, joins,
43
+ `pull`, `bind_cols`), array helpers (`dot`, numeric functions, `conj`,
44
+ `rename_axes`, `pack`), and `formula()` for plain mathematical text and a
45
+ strict LaTeX dialect. Scripts declare names with
46
+ `fields()` or write `col("x")`. `collect()` returns a computed xarray
47
+ Dataset; `to_xarray()` returns a shallow copy sharing the underlying values.
48
+
49
+ ## Notebooks
50
+
51
+ Install IPython separately to use the notebook extension. In a notebook cell:
52
+
53
+ ```ipython
54
+ %load_ext tidy3a.jupyter
55
+ from tidy3a import *
56
+ ```
57
+
58
+ The extension enables bare field names inside verbs, so the quick-start
59
+ example can omit the `fields()` declaration. Multi-line pipes need
60
+ parentheses. Sharing a notebook with tidy3 requires its ownership-aware
61
+ transformer; the prerequisite is recorded in [the plan](docs/plan.md#56-notebook-layer).
62
+
63
+ ## Execution and first-release scope
64
+
65
+ The first release runs eagerly on NumPy. `read_array` accepts homogeneous
66
+ raw numeric files, `.npy` files, and NumPy arrays. `read_binary` reads mixed
67
+ fixed-width records into separate variables, preserving numeric types and
68
+ strictly decoding UTF-8/ASCII text declared with `fixed_text`. Records can
69
+ be packed or have explicit field offsets and padding; see the
70
+ [reader contract](docs/m6-read-binary.md) for supported options. `tidy()` wraps xarray
71
+ DataArrays and Datasets; optional file readers stay in xarray's ecosystem.
72
+
73
+ `read_array(..., lazy=True)` memory-maps raw and `.npy` files. It does not
74
+ make the verb pipeline lazy: arithmetic can allocate results, filtering evaluates its
75
+ mask, and updating a field label copies its containing array into memory.
76
+ `tidy()` preserves a lazy xarray input initially; `collect()` computes it.
77
+ Dask execution guarantees remain planned.
78
+
79
+ Integer arithmetic follows NumPy, including unsigned subtraction wrapping
80
+ around. Convert before subtracting when negative results are needed, for
81
+ example `nir * 1.0 - red` for uint8 image bands. See
82
+ [first-release limitations](docs/first-release.md) for the supported subset.
83
+
84
+ ## Examples and development
85
+
86
+ The point-cloud example runs on real nuScenes Mini data and checks itself
87
+ against plain NumPy:
88
+
89
+ ```bash
90
+ python examples/nuscenes_sweep.py /path/to/nuscenes-mini
91
+ ```
92
+
93
+ Three public sample datasets (gridded air temperature, ERA-Interim wind, and
94
+ stars' Landsat scene) run against plain xarray; see
95
+ [examples/sample_data.py](examples/sample_data.py) for the files, optional
96
+ reader dependencies, and checks.
97
+
98
+ To run the local tests:
99
+
100
+ ```bash
101
+ python -m pip install -e ".[dev]"
102
+ python -m pytest -q
103
+ ```
104
+
105
+ Before pushing, run the CI checks in fresh environments with only declared
106
+ dependencies (requires [uv](https://docs.astral.sh/uv/)):
107
+
108
+ ```bash
109
+ scripts/check.sh
110
+ ```
111
+
112
+ The [release procedure](docs/releasing.md) describes the GitHub Release
113
+ workflow and the one-time GitHub/PyPI setup.
114
+
115
+ ## License
116
+
117
+ MIT
@@ -0,0 +1,19 @@
1
+ # tidy3A tidy-layer rules
2
+
3
+ The one-page rule set from plan §1.1: if the tidy layer's rules stop fitting on this page, simplify the layer rather than adding rules. Details, reasons, and precedents are in `docs/plan.md`, with section references below.
4
+
5
+ **Data.** A dataset holds variables (named arrays). Each axis is either the one **field axis**, declared with `field_axis=`, whose labels act as column names, or an **observation axis**, whose labels are values. Without `field_axis=` there are only variables. `variables_from=` turns a labeled axis into one variable per label instead. (§3, §4.1)
6
+
7
+ **Names.** In verbs and expressions, a name means, in order: a variable, then a field-axis label. A field label alone, `x`, is that field with the field axis dropped; a list `[x, y, z]` keeps the field axis, narrowed, and also rebuilds variables made by the same `variables_from=` import into their recorded axis. Other variables are packed with `pack([...], along=...)`. A name on an observation axis means that axis's labels (`filter(time > t0)`). A name that is both data and a notebook value is an error: write `col("x")` or `env("x")`. A name that is both a variable and a field label is an error: write `variable("x")` or `label("x")`. Scripts use strings or `fields("x", …)`. (§5.1, §5.2)
8
+
9
+ **Verbs never reshape on their own.** `select` keeps variables and field labels (columns), with tidy3's selection helpers. `filter` keeps positions on observation axes (rows); every predicate must vary along one axis, and predicates on different axes combine with AND into a rectangle. `slice` keeps positions by 1-based position and keeps the axis unless `drop=True`. Filtering keeps original positions as labels. Empty results are returned. (§5.1, §5.3)
10
+
11
+ **Mutate.** Assignments run in order. Writing to a field label writes into the shared array, promoting its type if needed. Vectors are written in one assignment, `[x, y, z] = …`. Masking is `if_else`. (§5.4)
12
+
13
+ **Grouping.** `group_by` on an axis keeps it; on a value field, or `bins(...)` of one, it groups that field's one axis into a new axis of sorted values or intervals. `summarise` reduces every other observation axis and never the field axis. A summary using a variable without the grouping axis is an error. Grouped summaries inside `mutate` broadcast back. Missing values propagate unless `na_rm=True`. After `summarise`, the last grouping level is dropped (`groups=`). (§5.4)
14
+
15
+ **Alignment.** Labels on a shared axis must match exactly; the same labels in another order are reordered and reported. Different labels need a join verb (`left_join`, `inner_join`, `full_join`, with `_x`/`_y` suffixes; `tolerance=` is currently supported only on `left_join`). A labeled axis never pairs with an unlabeled one by length; attach outside arrays with `bind_cols(…, along=)`, and read labels with `pull(axis)`. (§5.5)
16
+
17
+ **Array layer.** Maths helpers name their axes: `dot(…, over=)`, `sum(…, over=)`, `rename_axes(…)`; `dot` never conjugates. `formula()` reads a strict LaTeX dialect into the same expressions, binds index letters to axes by name or `bind=`, and always shows how it was read. (§6)
18
+
19
+ **Results.** `read_array` and `tidy()` return the wrapper. `collect()` returns a computed xarray Dataset; `to_xarray()` returns the underlying xarray object. (§3, §4.1)
@@ -0,0 +1,69 @@
1
+ # First-release scope and limitations
2
+
3
+ Version `0.1.0` is the first release.
4
+ The release notes are in [the changelog](../CHANGELOG.md). Progress and verification
5
+ are recorded in [milestone 7](plan.md#milestone-7--independent-first-release).
6
+
7
+ ## Supported scope
8
+
9
+ - Python 3.11 or newer, NumPy 2.0 or newer, pandas 2.2.2 or newer, and
10
+ xarray 2024.6 or newer.
11
+ - Homogeneous raw numeric files, `.npy` files, and in-memory NumPy arrays
12
+ through `read_array`; xarray objects through `tidy()`.
13
+ - Mixed fixed-width numeric/text records through `read_binary`, with
14
+ explicit padding layouts and strict UTF-8/ASCII `fixed_text` decoding.
15
+ The supported reader options are listed in [the contract](m6-read-binary.md).
16
+ - Tidy verbs, exact label alignment, explicit joins, named-axis mathematics,
17
+ complex values, and the implemented plain-text and LaTeX formula dialect.
18
+ - Script expressions with `fields()` or `col()`, and optional IPython
19
+ shorthand through `%load_ext tidy3a.jupyter`.
20
+
21
+ ## Memory and execution
22
+
23
+ NumPy-backed operations run eagerly. `lazy=True` on `read_array` creates a
24
+ read-only memory map; it does not defer the whole pipeline. Arithmetic can
25
+ allocate arrays, filtering evaluates its predicate mask, and field-label
26
+ updates copy the whole containing array, even when changing one field.
27
+ `pull()` returns NumPy values and materializes the requested data.
28
+
29
+ `tidy()` does not itself compute a lazy xarray input. Selection, regular
30
+ positional slicing, arithmetic, and reductions on Dask-backed or file-backed
31
+ xarray data stay lazy and are checked in the test suite
32
+ ([metadata and lazy input](m6-metadata-lazy.md)). Grouping by field values
33
+ or bins reads the grouping values, filtering reads the mask, and field-label
34
+ write-back reads and copies the containing array. `median(..., na_rm=True)`
35
+ over every axis of a chunked array is an error that asks you to collect
36
+ first. `collect()` computes the resulting Dataset. `to_xarray()` makes a
37
+ shallow copy without computing it. Dask-backed grouping and performance
38
+ guarantees are not part of this release; memory-mapping alone does not
39
+ promise bounded memory use.
40
+
41
+ ## Known limitations
42
+
43
+ - Grouping supports multiple axis keys but only one value-field or bin key.
44
+ Grouped `slice` and its head/tail helpers require `ungroup()` first.
45
+ - Reductions with `over=` cannot be combined with grouping by a value field.
46
+ A summary cannot reference an earlier result in the same `summarise`;
47
+ derive from summary results in a following `mutate`.
48
+ - `tolerance=` is implemented only for `left_join`. Other joins use exact
49
+ labels. `case_when` is not implemented; use `if_else` for binary choices.
50
+ - xarray on the left of an arithmetic operator does not defer to a tidy3A
51
+ expression. Write `x * R` rather than `R * x`, or use `dot()`.
52
+ - Integer arithmetic follows NumPy. For unsigned data, promote before a
53
+ subtraction that may be negative: `nir * 1.0 - red`.
54
+ - Formulas support the dialect documented in [plan §6.3](plan.md#63-formula), with
55
+ errors for unsupported syntax. Product reductions, integrals, sums over
56
+ more than one index letter at once, and implicit multiplication in plain
57
+ text are not supported. Write `2*x` rather than `2x` in plain text.
58
+ - Notebook pipes spanning lines need parentheses; tidy3-style backtick
59
+ names are not supported. Use `col("field name")` for such names.
60
+ - Coexistence with tidy3's notebook extension requires tidy3's ownership
61
+ check (`_owned_by_tidy3`); the prerequisite change is recorded in the
62
+ implementation plan. The coexistence tests skip when it is unavailable.
63
+
64
+ Mixed-record reading, metadata handling, and the lazy-input checks were
65
+ brought forward from milestone 6 before publication. Dedicated NetCDF/Zarr
66
+ readers, Dask-backed grouping, text encodings beyond UTF-8/ASCII, and
67
+ variable-length records remain planned. Existing
68
+ xarray readers can already be used with `tidy()`; their optional dependencies
69
+ are installed separately.
@@ -0,0 +1,127 @@
1
+ # Milestone 0: acceptance examples in two styles
2
+
3
+ Status: prototype comparison for review. None of the tidy3A code below is implemented. Each example's plain-xarray equivalent runs in `prototypes/test_examples.py` on small synthetic data with hand-checked answers, and `prototypes/test_xarray_assumptions.py` checks the xarray and NumPy behavior the plan relies on. Both pass (39 tests, xarray 2026.9.0, NumPy 2.5.4).
4
+
5
+ The question this answers (plan §5.1, milestone 0): does `select` with a declared **field axis** read better than the **stars style**, where fields become separate variables (stars' `split`, tidy3A's `variables_from=`) and `select` picks only variables?
6
+
7
+ - **Style A, field axis:** `read_array(..., field_axis="feature")`. One array; field labels act as columns.
8
+ - **Style B, stars style:** `read_array(..., variables_from="feature")`. One variable per field, as after stars' `split(x, "feature")`. This comparison was written with a re-packing helper, `merge(x, y, z, along="feature")` after stars' `merge()`; since decided: `[x, y, z]` rebuilds the recorded axis (plan §5.2), so Style B's vector line now matches Style A.
9
+ - **Style C, strict without splitting:** no field axis and no split. Field labels are reached only through `filter` and xarray. Shown once, for comparison.
10
+
11
+ ## Example 1: point cloud
12
+
13
+ Synthetic data: four points that, after a 90° rotation about z and a 1 m offset, lie 49 m, 50 m, 51 m, and 3 m from the origin; `intensity` 10, 20, 30, 40; `ring` 1, 2, 1, 2.
14
+
15
+ **Style A**
16
+
17
+ ```python
18
+ cloud = read_array(
19
+ path,
20
+ value_type="float32",
21
+ dims={"point": infer, "feature": 5},
22
+ labels={"feature": ["x", "y", "z", "intensity", "ring"]},
23
+ field_axis="feature",
24
+ )
25
+
26
+ nearby = (
27
+ cloud
28
+ >> mutate(p = dot(R, [x, y, z], over="feature") + offset)
29
+ >> mutate(distance = formula(r"\sqrt{\sum_i p_i^2}"))
30
+ >> filter(distance <= 50)
31
+ )
32
+
33
+ nearby >> group_by(ring) >> summarise(mean_intensity = mean(intensity))
34
+ ```
35
+
36
+ **Style B**
37
+
38
+ ```python
39
+ cloud = read_array(
40
+ path,
41
+ value_type="float32",
42
+ dims={"point": infer, "feature": 5},
43
+ labels={"feature": ["x", "y", "z", "intensity", "ring"]},
44
+ variables_from="feature",
45
+ )
46
+
47
+ nearby = (
48
+ cloud
49
+ >> mutate(p = dot(R, [x, y, z], over="feature") + offset) # rebuilds the recorded feature axis
50
+ >> mutate(distance = formula(r"\sqrt{\sum_i p_i^2}"))
51
+ >> filter(distance <= 50)
52
+ )
53
+
54
+ nearby >> group_by(ring) >> summarise(mean_intensity = mean(intensity))
55
+ ```
56
+
57
+ **Style C**
58
+
59
+ ```python
60
+ xyz = cloud >> filter(feature.is_in(["x", "y", "z"]))
61
+ # Bare x, intensity, and ring do not exist; expressions need xarray:
62
+ # cloud.to_xarray()["values"].sel(feature="ring")
63
+ ```
64
+
65
+ Expected (checked): kept points 1, 2, 4 with original positions as labels, since 50 m is inside the boundary; intensities 10, 20, 40; mean intensity per ring: ring 1 → 10, ring 2 → 30.
66
+
67
+ ## Example 2: sensor time series
68
+
69
+ Synthetic data: `values (time: 4, sensor: front rear, feature: x y z)`.
70
+
71
+ | Operation | Style A | Style B |
72
+ |---|---|---|
73
+ | Mean x per sensor | `group_by(sensor) >> summarise(mean_x = mean(x))` | same |
74
+ | All fields per sensor | `group_by(sensor) >> summarise(m = mean(values))` → `m (sensor, feature)` | one summary per field, e.g. `summarise(mx = mean(x), my = mean(y), mz = mean(z))` |
75
+ | Front sensor after time 2 | `filter(sensor == "front", time > 2)` | same |
76
+ | Times where any sensor has x > 20 | `filter(any(x > 20, over="sensor"))` | same |
77
+
78
+ Expected (checked): mean x is 4 (front) and 40 (rear); the front sensor's field means are x 4, y 3, z 0.5; the two-axis filter keeps `time: 2, sensor: 1`; the `any` filter keeps times 2, 3, 4.
79
+
80
+ ## Example 3: geospatial raster
81
+
82
+ Synthetic data in the shape stars handles: a 3 × 2 pixel scene, bands `blue green red nir`, two dates, with `x`/`y` map coordinates. Near-infrared halves at the second date.
83
+
84
+ **Style A** (`field_axis="band"`)
85
+
86
+ ```python
87
+ scene >> mutate(ndvi = (nir - red) / (nir + red))
88
+ scene >> filter(x > 500010, y > 4100010) # crop, as stars' filter
89
+ scene >> mutate(ndvi = (nir - red) / (nir + red)) >> group_by(x, y) >> summarise(mean_ndvi = mean(ndvi))
90
+ scene >> summarise(m = mean(values)) # one mean per band; stars: st_apply(x, "band", mean)
91
+ scene >> mutate(ndvi = …) >> group_by(bins(ndvi, [-1, 0.25, 1])) >> summarise(pixels = n())
92
+ ```
93
+
94
+ **Style B** (`variables_from="band"`)
95
+
96
+ ```python
97
+ scene >> mutate(ndvi = (nir - red) / (nir + red)) # same
98
+ scene >> filter(x > 500010, y > 4100010) # same
99
+ # per-pixel mean NDVI and bins: same
100
+ scene >> summarise(blue = mean(blue), green = mean(green), # one summary per band
101
+ red = mean(red), nir = mean(nir))
102
+ ```
103
+
104
+ Expected (checked): NDVI at the first date is 0.5, 0, 0, 0.75, 0.5, 0 by pixel; the crop keeps `x: 2, y: 1`; the first pixel's mean NDVI over time is (0.5 + 0.2) / 2; the per-band mean of blue is 0.05; the NDVI bins hold 3 and 3 pixels.
105
+
106
+ ## Comparison
107
+
108
+ | Aspect | Style A, field axis | Style B, stars style | Style C, strict |
109
+ |---|---|---|---|
110
+ | Scalar expressions (`distance <= 50`, NDVI, `mean(x)`) | Bare names | Bare names, identical text | Not available without xarray |
111
+ | Vector operations (rotation) | `[x, y, z]` directly | `[x, y, z]` rebuilds the recorded axis (decided after this comparison; it first needed a `merge` helper) | xarray |
112
+ | One summary for every field | `mean(values)` keeps the field axis | One expression per field, or an `across()`-style helper | xarray |
113
+ | Fields with different types (`ring` as integer) | Not possible: one array, one type | Natural | Not possible |
114
+ | Concepts a user must learn | Field axis, observation axis | Separate variables only | Separate variables only |
115
+ | Rules in the tidy layer | More: field-axis select, `var()`/`label()` collisions, write-back, type promotion | Fewer | Fewest |
116
+
117
+ Findings:
118
+
119
+ 1. **Style C is clearly worst.** Without either a field axis or splitting, the most common point-cloud and raster operations leave the pipeline. It is not a real option.
120
+ 2. **Styles A and B read the same for scalar work.** Filters, NDVI, grouping, and bins are character-for-character identical. The choice of style changes nothing for most expressions.
121
+ 3. **Style A wins where fields form a vector or a spectrum:** rotation (`[x, y, z]` versus `merge(...)`), and one summary for every band or component (`mean(values)` versus one line per field).
122
+ 4. **Style B wins where fields are heterogeneous:** different types and units, such as nuScenes' `ring`, and it needs fewer rules.
123
+ 5. Both styles are already in the plan, as `field_axis=` and `variables_from=`. The comparison supports keeping both, with documented guidance: use `field_axis=` when fields are components of one quantity (`x y z`, spectral bands, RGB), and `variables_from=` when they are different quantities.
124
+
125
+ Decided (2026-10-24): keep the field axis, with `variables_from=` equally first-class (plan §5.1). The recommendation as reviewed: keep the field axis. Its advantage is real but narrow (vectors and spectra), so the documentation should present `variables_from=` as equally first-class, and milestone 2 should check the tidy layer's one-page complexity budget (`docs/decisions.md`) before adding more field-axis rules.
126
+
127
+ Re-packing, found while writing Style B and since decided (plan §5.2): `[x, y, z]` rebuilds a vector from variables of the same recorded `variables_from` axis, and `pack([...], along=...)` re-packs other variables, with the axis always named. `merge` was avoided because `xr.merge` implements joins.