unitarrow 0.0.1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
unitarrow/__init__.py
ADDED
|
@@ -0,0 +1,142 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: unitarrow
|
|
3
|
+
Version: 0.0.1
|
|
4
|
+
Summary: Physical units and quantity semantics for Apache Arrow columns (name reservation; spec in development)
|
|
5
|
+
Project-URL: Repository, https://github.com/unitarrow/unitarrow
|
|
6
|
+
License-Expression: BSD-3-Clause
|
|
7
|
+
Requires-Python: >=3.9
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
|
|
10
|
+
# UnitArrow
|
|
11
|
+
|
|
12
|
+
**Physical units and quantity semantics for Apache Arrow columns.**
|
|
13
|
+
Column-level, cross-language, and wire-native: a unit annotates a whole
|
|
14
|
+
column as Arrow extension metadata, survives IPC / Flight / Parquet, and
|
|
15
|
+
converts as a metadata-level operation — never a per-element object.
|
|
16
|
+
|
|
17
|
+
> **Status: pre-release.** This repository reserves the `unitarrow`
|
|
18
|
+
> name while the specification is finalized and the first implementation
|
|
19
|
+
> lands. The spec draft, roadmap, and reference implementation will be
|
|
20
|
+
> published here. UnitArrow is a community extension-type specification
|
|
21
|
+
> in the spirit of GeoArrow; it is not affiliated with or endorsed by
|
|
22
|
+
> the Apache Arrow project or the ASF.
|
|
23
|
+
|
|
24
|
+
## Why
|
|
25
|
+
|
|
26
|
+
Every existing approach to units dies at a boundary. In-memory unit
|
|
27
|
+
libraries (pint, astropy, unyt) get the algebra right and lose
|
|
28
|
+
everything at serialization — their units are properties of runtime
|
|
29
|
+
objects. File-format conventions (netCDF/CF, HDF5 attributes) survive
|
|
30
|
+
storage but carry *dead* metadata: no checking, no propagation, no
|
|
31
|
+
algebra. Compile-time unit systems are safest and can't cross a wire at
|
|
32
|
+
all.
|
|
33
|
+
|
|
34
|
+
UnitArrow puts units where all of those layers already meet: the Arrow
|
|
35
|
+
interchange layer. A tagged column keeps its unit across processes,
|
|
36
|
+
languages, and the network; a consumer that has never heard of
|
|
37
|
+
UnitArrow sees a plain float column with human-readable metadata beside
|
|
38
|
+
it. Adoption never needs to be coordinated: producers can tag before
|
|
39
|
+
their consumers understand units, consumers can upgrade before their
|
|
40
|
+
producers do, and each side gains value on its own schedule.
|
|
41
|
+
|
|
42
|
+
### Where existing tools stop
|
|
43
|
+
|
|
44
|
+
| Approach | What it gets right | Where it stops |
|
|
45
|
+
|---|---|---|
|
|
46
|
+
| pint / astropy.units / unyt / Unitful.jl | Correct unit algebra, array-level units, mature registries | Units are runtime-object properties: they do not survive serialization. As of pint-pandas 0.8 / pandas 3.0 / pyarrow 25, `pa.Table.from_pandas` on a pint column doesn't even degrade — it raises `ArrowTypeError`. Adoption is all-or-nothing: everything becomes a Quantity, and pandas operations that don't silently fall back to object dtype fail outright. |
|
|
47
|
+
| netCDF/CF, HDF5 attributes, FITS `BUNIT` | Units survive storage; CF's standard-name vocabulary is decades of accumulated wisdom | The metadata is *dead*: no algebra, no checking, no propagation through compute — and each convention is welded to one container format. |
|
|
48
|
+
| F# units of measure, mp-units, uom | Compile-time safety, zero runtime cost | Locked inside one language's type checker; cannot cross a wire, an FFI boundary, or into dynamic languages at all. |
|
|
49
|
+
| SQL engines / dataframe libraries | Where the computation actually happens | No unit concept whatsoever; column metadata is stripped by the first query. |
|
|
50
|
+
|
|
51
|
+
None of this is a criticism of the algebra in these tools — pint's is
|
|
52
|
+
excellent, and CF's vocabulary is a direct input to UnitArrow's. Their
|
|
53
|
+
shared limitation is *placement*: each lives in one layer and dies at
|
|
54
|
+
that layer's edge. UnitArrow's job is the layer none of them can reach
|
|
55
|
+
— units as a property of the *data interchange* itself — and adapters
|
|
56
|
+
to and from pint are a natural part of the roadmap, so in-process pint
|
|
57
|
+
users gain wire survival rather than losing anything.
|
|
58
|
+
|
|
59
|
+
## Adopt gradually — the whole design bends around this
|
|
60
|
+
|
|
61
|
+
UnitArrow is *gradually typed*, the way TypeScript is: untagged columns
|
|
62
|
+
are always legal, checking is a dial rather than a wall, and each step
|
|
63
|
+
is independently useful. You can stop at any step and keep everything
|
|
64
|
+
it gave you.
|
|
65
|
+
|
|
66
|
+
### Step 1 — Tag, ship, read (zero-cost drop-in)
|
|
67
|
+
|
|
68
|
+
Convert a pandas or polars dataframe to Arrow, attach units, and ship
|
|
69
|
+
it. Read a UnitArrow table back into pandas with units intact.
|
|
70
|
+
Conversion is one scalar rescale driven by column metadata. No code
|
|
71
|
+
changes anywhere else; untouched pipelines pass tagged tables through
|
|
72
|
+
bit-identical.
|
|
73
|
+
|
|
74
|
+
```python
|
|
75
|
+
# API sketch — subject to change before 1.0
|
|
76
|
+
import unitarrow as ua
|
|
77
|
+
|
|
78
|
+
table = ua.tag(df, {"power": "MW", "gas": "GBtu"}) # pandas / polars / pyarrow in
|
|
79
|
+
table = ua.convert(table, {"gas": "MWh"}) # metadata-level rescale
|
|
80
|
+
df2 = ua.to_pandas(table) # units survive the round trip
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
The same table read in a browser via arrow-js exposes its units from
|
|
84
|
+
field metadata — client-side unit toggling with no server round trip.
|
|
85
|
+
|
|
86
|
+
### Step 2 — Checked compute (IR + WebAssembly)
|
|
87
|
+
|
|
88
|
+
Opt columns into checking and computation becomes dimensionally sound
|
|
89
|
+
*before* it runs: multiplying composes units, adding requires
|
|
90
|
+
commensurability, and summing an instantaneous MW column into "energy"
|
|
91
|
+
is an error — the only path from MW rows to MWh is an explicit,
|
|
92
|
+
period-aware `integrate`. A small typed expression IR lowers to your
|
|
93
|
+
engine (arrow-rs, polars, DuckDB) and to WebAssembly, so the same
|
|
94
|
+
checked operations — convert, integrate, differentiate, aggregate —
|
|
95
|
+
run in the browser over arrow-js buffers.
|
|
96
|
+
|
|
97
|
+
### Step 3 — End-to-end validated data products
|
|
98
|
+
|
|
99
|
+
The final step of the journey: units verified from UnitArrow sources
|
|
100
|
+
all the way to UnitArrow published outputs. Tables finalize with
|
|
101
|
+
`publish()` — checked in strict mode, pinned to a registry version, and
|
|
102
|
+
sealed so consumers can verify who published a table and that it hasn't
|
|
103
|
+
changed. Full derivation chains (including external tools like LP
|
|
104
|
+
solvers) become attestable and reproducible. This trust layer is
|
|
105
|
+
specified separately (the Provenance Companion) and is entirely
|
|
106
|
+
optional — Steps 1 and 2 never require it.
|
|
107
|
+
|
|
108
|
+
## Design principles
|
|
109
|
+
|
|
110
|
+
- **Zero-cost drop-in.** Tagging is O(columns) and touches no data
|
|
111
|
+
bytes. Untagged tables are silent everywhere.
|
|
112
|
+
- **Graceful degradation.** Unaware consumers read the storage type plus
|
|
113
|
+
legible metadata. Nothing is ever gated on adoption.
|
|
114
|
+
- **Semantics as data.** Units, dimensions, display forms, and quantity
|
|
115
|
+
vocabularies live in a versioned registry, federated by namespace —
|
|
116
|
+
domains extend it without permission.
|
|
117
|
+
- **One implementation.** A single Rust core (`unitarrow-core`) does all
|
|
118
|
+
parsing, dimension algebra, and conversion, exposed to Python
|
|
119
|
+
(`unitarrow-py`, via the Arrow PyCapsule interface — no pyarrow
|
|
120
|
+
dependency), TypeScript, and WASM (`unitarrow-wasm`). A conformance
|
|
121
|
+
suite of golden files keeps every binding identical.
|
|
122
|
+
|
|
123
|
+
## Packages
|
|
124
|
+
|
|
125
|
+
| Package | Registry | Status |
|
|
126
|
+
|---|---|---|
|
|
127
|
+
| `unitarrow` / `unitarrow-core` / `unitarrow-py` / `unitarrow-wasm` | crates.io | reserved |
|
|
128
|
+
| `unitarrow` | PyPI | reserved |
|
|
129
|
+
| `unitarrow` | npm | reserved |
|
|
130
|
+
|
|
131
|
+
## Roadmap
|
|
132
|
+
|
|
133
|
+
Specification first, then `unitarrow-core`, then the Python and
|
|
134
|
+
browser codecs (Step 1), then checked semantics and the IR (Step 2),
|
|
135
|
+
then the Provenance Companion (Step 3). The full spec and milestone
|
|
136
|
+
roadmap will be published in this repository.
|
|
137
|
+
|
|
138
|
+
---
|
|
139
|
+
|
|
140
|
+
*UnitArrow began in energy-systems research — where BTU, MWh, and
|
|
141
|
+
per-unit quantities collide daily — but nothing in it is
|
|
142
|
+
energy-specific.*
|
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
unitarrow/__init__.py,sha256=lV_GsnWge6RvSpb3rPPgeIKK7DXwmyckf27Uh3mNdKc,204
|
|
2
|
+
unitarrow-0.0.1.dist-info/METADATA,sha256=-CuIg1YOpVjUGchYRupY2stNpL5WZMOQEL6w_ExPnvI,7240
|
|
3
|
+
unitarrow-0.0.1.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
|
|
4
|
+
unitarrow-0.0.1.dist-info/RECORD,,
|