nulldtype 0.1.0__cp313-cp313-win_amd64.whl

nulldtype-0.1.0.dist-info/METADATA

@@ -0,0 +1,270 @@
+Metadata-Version: 2.4
+Name: nulldtype
+Version: 0.1.0
+Summary: Missing values as a NumPy DType: NA stored as a reserved bit pattern of the wrapped type
+License-Expression: BSD-3-Clause
+Project-URL: Source, https://github.com/tuan2k33/NullableDType
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Programming Language :: C
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+License-File: LICENSE.txt
+Requires-Dist: numpy>=2.5
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# Nullable[T] — prototype
+
+[![CI](https://github.com/tuan2k33/NullableDType/actions/workflows/ci.yml/badge.svg)](https://github.com/tuan2k33/NullableDType/actions/workflows/ci.yml)
+
+`Nullable[T]` is a **parametric DType**: it wraps any other dtype and gives it a
+notion of missing data that lives **inside the dtype**, not in a parallel mask.
+NA is a reserved value of the wrapped type itself, so an element costs exactly
+what that type costs:
+
+```python
+nd.Nullable(np.float64)   # itemsize 8   — NA is every bit but the sign
+nd.Nullable("U10")        # itemsize 40  — NA is the whole cell of U+FFFF
+nd.Nullable(record)       # itemsize T   — every field holds its own type's NA
+```
+
+Nullable treats NA as a datum, not as a number that is hidden, so there is
+nothing like `a.unmask()` or `a[2].value`. Assigning NA overwrites the old value
+the way assigning any other value would, and the old value cannot leak out
+later. To set values aside and bring them back, keep the original array, or put
+a mask on top (`numpy.ma`, [marray](https://github.com/mdhaber/marray)). The
+two layers compose. In the terms of the 2012 NumPy discussion, this is
+**MISSING**, and masks are **IGNORED**.
+
+| Document | What is in it |
+|---|---|
+| `LAYOUTS.md` | the NA pattern of every type, and what each gives up |
+| `VS-NUMPY-MA.md` | all 218 names in `numpy.ma.__all__` that numpy 2 still spells that way, side by side with `nd`, with real calls and results |
+| `DISCUSSION.md` | how it works and why: the borrowed loops, performance, the bugs found on the way, leak checking |
+| `NUMPY-PATCHES.md` | where numpy's array functions fail on Nullable, and patches that leave plain arrays unchanged |
+| `AGENTS.md` | notes for coding agents working on the repo |
+| `archive/flag-layout/` | the earlier two-layout version, frozen |
+
+## Running it
+
+Needs Python 3.12+ and numpy 2.5+; building from source also needs a C
+compiler. CI tests Linux with numpy 2.5, the latest release and the nightly
+wheels, and the release wheels are tested on Linux (x86-64, ARM), macOS and
+Windows. It will not load on numpy 2.3 or older, which lack
+the DType API slots it uses.
+
+```bash
+pip install nulldtype            # wheels for Linux, macOS and Windows
+pip install .                    # or build from a checkout
+pip install pytest && pytest     # run the suite
+```
+
+For working on the C code, `./build.sh` builds in place and `./run_tests.sh`
+builds and runs pytest (extra arguments go to pytest). They use `python3`;
+set `PYTHON`, and `NUMPY_SITE` for a numpy dev tree that is not installed, or
+put both in an uncommitted `local.env`.
+
+The suite has 582 tests plus 2 expected failures that pin known gaps. It also
+passes on an ASAN + UBSAN build, which CI runs on every push.
+
+## What works
+
+```python
+import numpy as np, nulldtype as nd
+
+dt = nd.Nullable(np.float64)      # itemsize 8, NA is a bit pattern
+a = np.arange(5.0).astype(dt)
+a[2] = nd.NA
+
+a                      # [0.0 1.0 NA 3.0 4.0]
+a + a                  # [0.0 2.0 NA 6.0 8.0]
+a + 1.0                # [1.0 2.0 NA 4.0 5.0]   a Python scalar works
+a + np.arange(5.0)     # so does a plain array, either way round
+np.sqrt(a) * 2         # unary ufuncs too
+a.sum()                # NA          propagates, R's default
+np.add.accumulate(a)   # [0.0 1.0 NA NA NA]
+
+np.can_cast(np.float64, dt)               # False — a value is given up, so not "safe"
+np.can_cast(np.float64, dt, "same_kind")  # True  — astype and assignment still automatic
+a.astype(np.float64)   # ValueError: cannot convert a missing value to dtype('float64')
+
+nd.isna(a)             # [False False True False False]
+nd.filled(a, 0.0)      # [0. 1. 0. 3. 4.]
+nd.to_numpy(a)         # ValueError: pass na_value= to say what they should become
+```
+
+Strings, raw bytes and records:
+
+```python
+s = np.zeros(3, nd.Nullable("S3"))    # itemsize 3
+s[0] = b"ab"; s[1] = nd.NA
+s                      # [b'ab' NA b'']
+s.tobytes()            # b'ab\x00\xff\xff\xff\x00\x00\x00'   the gap is all 0xFF
+s[2] = b"\xff\xff\xff" # ValueError: ... is the value reserved to mean NA
+
+u = np.array(["b", "a", "c"]).astype(nd.Nullable("U3"))
+u[1] = nd.NA
+np.sort(u)             # [b c NA]    gaps sort last
+
+r = np.zeros(2, nd.Nullable(np.dtype([("a", "i4"), ("b", "f8")])))  # itemsize 12
+r[0] = (1, 2.0); r[1] = nd.NA   # the gap: (0x80000000, 0x7FFFFFFFFFFFFFFF)
+r == r                          # [True NA]
+nd.Nullable(np.longdouble)   # Nullable(float64), with a LongDoubleWarning
+nd.Nullable(">i4")           # Nullable(int32) — stored in native byte order
+```
+
+| Area | What it does |
+|---|---|
+| Types | every int, uint, float, complex, bool, `datetime64`, `timedelta64`; `S`, `V` (NA is the whole cell of `0xFF`); `U` (every character `U+FFFF`, a noncharacter, so a gap is still a well-formed string); records (missing when every field holds its own NA; nested records, subarray fields) |
+| `longdouble`, `clongdouble` | stored as `float64` / `complex128`, with a `LongDoubleWarning` where precision is actually lost |
+| Byte order | always stored native: `Nullable(">i4") == Nullable("<i4")`, and big-endian data still casts in and out correctly |
+| Assignment | `nd.NA` and `None` mark an element missing; numpy scalars (`np.void`, `np.bytes_`, `np.str_`, records) assign as values |
+| 29 binary ufuncs | arithmetic, `maximum`/`minimum`, the 6 comparisons, `remainder`, `fmod`, `arctan2`, `hypot`, `logaddexp`, `copysign`, `nextafter`, `fmax`/`fmin`, `heaviside`, `float_power`, `&`/`\|`/`^` — **borrowing T's own loop**; merging validity is all this code does |
+| 41 unary ufuncs | all of trig/hyperbolic/log/exp, rounding, `signbit`, `conjugate`, `~`, `logical_not`; `isnan`/`isinf`/`isfinite` of NA answer NA |
+| `clip`, `matmul` | `clip` is a 3-input ufunc; `matmul` is a gufunc, including `v@A`, `A@v` and the batched forms |
+| Results follow numpy's rules | `resolve` asks the wrapped ufunc, so `Nullable[i8] / Nullable[i8]` is `Nullable[f8]` and `Nullable[i8] + Nullable[f8]` is `Nullable[f8]` |
+| Mixed operands | `Nullable[T] op T` and Python scalars, both ways round, through promoters |
+| Comparisons | answer `Nullable[bool]`; `NA == NA` is `NA`, not `True` |
+| Three-valued logic | `logical_and/or/xor` and `&`, `\|`, `^` on `Nullable[bool]` are Kleene: `NA & False = False`, `NA \| True = True`; on ints `&`, `\|`, `^` propagate |
+| Reductions and `accumulate` | propagate by default, like R's `na.rm = FALSE`, over several axes at once too; `cumsum` is NA from the first gap on |
+| `argmax`, `argmin` | the position of the first gap, numpy's rule for NaN, so `a[a.argmax()]` is NA exactly when `a.max()` is |
+| `sort`, `argsort` | gaps last, two gaps compare equal — as in R |
+| `nonzero`, `count_nonzero` | work, and **refuse** on a gap rather than guess, at any array size |
+| Nothing is computed on a gap | `where=` is handed to the wrapped ufunc when needed, so a gap raises no spurious warning while real errors still do |
+| Casts in | `T -> Nullable[T]` is automatic (`same_kind`); between wrapped dtypes (`Nullable[i8] -> Nullable[f8]`) gaps are kept |
+| Casts out | `Nullable[T] -> T` or any plain dtype (`int16`, big-endian…) **raises on a gap**; the guard is in the loop, which is why `np.isin` works while NA still cannot slip out |
+| Nothing invents an NA | casting `int64 2**31 -> Nullable[i4]`, truncating `S5 -> S3` down to all `0xFF`, arithmetic landing exactly on `INT_MIN` or `UINT_MAX`, and an `np.arange` reaching the reserved value all raise |
+| Explicit ways out | `nd.isna`, `nd.notna`, `nd.filled`, `nd.to_numpy` (which refuses to guess) |
+| pickle | both the dtype and the array |
+| `np.zeros` | gives real zeros, not gaps |
+
+## The `nd` namespace and `skipna`
+
+Every name in `nd` that it does not define is numpy's own (`nd.sort is
+np.sort`). It defines only what numpy gets wrong or refuses on a Nullable
+array, and refuses the `nan*` names with the call to use instead, because NA is
+not NaN:
+
+- `np.median`, `np.percentile` and `np.quantile` return a wrong number (NA
+  sorts last, so they take it for the maximum);
+- `np.mean`, `np.var` and `np.std` truncate on integer and bool arrays;
+- `np.all`, `np.any`, `np.dot`, `np.unique`, `np.isclose` do not work at all.
+
+```python
+nd.median(a)                # NA    propagates, the default
+nd.median(a, skipna=True)   # 3.0
+nd.quantile(a, [.25, .75])  # [NA NA]
+nd.unique(a)                # NA is a value of its own, sorted last
+
+m = [[1.0, 2.0, 3.0],
+     [4.0, NA,  6.0]]
+nd.mean(m, axis=1)               # [2.0 NA]    a lane with a gap is NA
+nd.mean(m, axis=1, skipna=True)  # [2.0 5.0]
+```
+
+- **Reductions:** `nd.sum/prod/min/max/mean/std/var/median/quantile/percentile`
+  take `axis=`, `keepdims=` and `skipna=False`.
+- **`skipna` elsewhere:** so do `nd.all`/`nd.any` (Kleene),
+  `nd.argmax`/`nd.argmin` and `nd.cumsum`/`nd.cumprod`. `nd.cumsum(a,
+  skipna=True)` carries past a gap and leaves NA at it, as pandas does.
+- **Other stand-ins:** `nd.count`, `nd.dropna`, `nd.dot`, `nd.array_equal`,
+  `nd.isin`, `nd.isclose` and `nd.allclose` answer NA when the answer is
+  genuinely unknown. Against a set `[1, NA]`, `nd.isin` answers NA for 5: the
+  gap might be 5.
+- **Skipping without `nd`:** `np.add.reduce(a, where=nd.notna(a))` works too.
+
+Why the numpy versions fail, and how numpy could be patched without changing
+plain arrays, is in `NUMPY-PATCHES.md`.
+
+## Known limitations
+
+- **`np.mean`, `np.var`, `np.std` (and `a.mean()`…) are silently wrong on
+  integer and bool arrays**: `np.mean([1, 2])` gives `1`. numpy picks a float
+  result only for its own integer types, so the sum is divided and truncated
+  back. **`np.median`, `np.quantile`, `np.percentile` are silently wrong** when
+  a gap is present. Neither is fixable from the dtype side; use `nd`.
+- `np.all` / `np.any` (and `np.array_equal`, which calls them) cannot be used:
+  `ndarray.all` pins its accumulator to plain `bool`, so numpy looks for
+  `(BoolDType, NullableDType) -> BoolDType`. Registering that would also hijack
+  `np.logical_and(plain_bool_array, nullable_array)`, which today correctly
+  answers `Nullable[bool]`.
+- `np.dot` **cannot be implemented**: numpy hands its slot a NULL array (see
+  `DISCUSSION.md`). Use `@` or `nd.dot`.
+- **Python scalars and NEP 50.**
+  - **Widening:** `Nullable[i4] + 2` gives `Nullable[i8]`, where numpy keeps
+    `int32` under the weak-scalar rule. Fixable with
+    `NPY_METH_resolve_descriptors_with_scalars`, not done yet; an `xfail` test
+    holds the place.
+  - **Promotion error:** `np.result_type(dtype, 1.0)` raises
+    `DTypePromotionError`, which takes `np.isclose`, `np.allclose` and
+    `np.select` down with it. Declaring `common_dtype` for Python scalars fixes
+    those three, but breaks the weak-scalar rule for every ufunc (measured:
+    `Nullable[f4] + 1.0` came out float64).
+- **`repr` of a `U` array containing an empty string raises `ValueError`.**
+  `arrayprint` formats a non-numpy dtype with `str()`, `str('')` is empty, and
+  `_extendLine_pretty` then calls `max()` on an empty list of lines. A numpy bug.
+- A gap in `S` and `V` is `0xFF` bytes, which is not valid text, so `astype("U")`
+  on a raw view of the values fails to decode. Through the dtype it never comes
+  up: a gap always reads as `NA`.
+- `sort`, `argmax` and `nonzero` hold the GIL for the whole loop, because the
+  dtype must declare `NPY_NEEDS_PYAPI` (see the segfaults in `DISCUSSION.md`),
+  and the ufunc loops declare `NPY_METH_REQUIRES_PYAPI` for their fallback path.
+  Large arrays get no threading benefit.
+- `a + 1.0` is about 2.6x slower than `a + a` — the scalar operand has stride 0
+  and appears to fall off the borrowed-loop path. Not chased down; the results
+  are correct.
+- `np.array([(1, 2.0), nd.NA], dtype=nd.Nullable(rec))` raises a shape error:
+  numpy only treats a tuple as one element when the requested dtype is a plain
+  record. Build with `np.zeros` and assign, or cast from a plain record array.
+- Records support only `==` and `!=`; `+`, `sum` and `<` fail, as they do for
+  plain records.
+- `nd.Nullable("S")` (no length) and a bare subarray dtype have no cell to fill,
+  so they raise `TypeError`. A subarray as a record field works.
+- `np.histogram` and `np.einsum` fail cleanly — both need a boolean decision
+  about possibly-missing data, and `bool(NA)` refuses to answer. Indexing with a
+  mask that contains NA raises `IndexError` for the same reason;
+  `nd.filled(mask, False)` says what a gap should count as.
+
+## What is next, in order
+
+1. **Report the numpy bugs upstream, and propose the hooks in
+   `NUMPY-PATCHES.md`.** Bug candidates:
+   - the SIMD stride bug — patch and reproducer ready;
+   - the `can_cast_pyscalar_scalar_to` assertion — patch and reproducer ready;
+   - `sort` not checking for `NULL` after `PyArray_DescrNewByteorder`;
+   - `repr` of a non-numpy dtype blowing up when `str()` of an element is empty.
+2. **Give NEP 50's weak-scalar rule back** with
+   `NPY_METH_resolve_descriptors_with_scalars`, so `Nullable[i4] + 2` stops
+   widening to `int64`.
+3. **A real `longdouble`** through `numpy-quaddtype` instead of substituting
+   `float64`.
+4. **Remove the fixed cost of `get_loop`** — two Python calls per operation, most
+   visible on small arrays.
+
+## Settled design notes
+
+- NA is **a datum**: assigning it overwrites the old value, which is **not
+  observable** afterwards.
+- NA **has a type**: `Nullable[i2]` is not `Nullable[f8]`, and there is no shared
+  singleton.
+- **One layout only**: NA is a bit pattern of `T` itself.
+- A record is **missing when every field is**, each field holding its own type's
+  NA.
+- The wrapped dtype is **always stored in native byte order**; `long double` is
+  stored as `double`.
+- **Nothing skips a gap unless asked**: reductions propagate, `argmax` points at
+  the gap, `isin` against a set with a gap is NA unless it finds a hit;
+  `skipna=True` says otherwise.
+- Coercion to a Python scalar **raises**; use `nd.filled(x, ...)` to get out.
+- **A cast never quietly creates an NA**, and never quietly drops one.
+
+## AI Disclosure
+
+AI was used in writing this project's code, tests and documentation.
+
+## License
+
+BSD 3-Clause, the same terms as NumPy; see `LICENSE.txt`.

nulldtype-0.1.0.dist-info/RECORD

@@ -0,0 +1,7 @@
+_nulldtype.cp313-win_amd64.pyd,sha256=Yq5fznaGGBywDwK2cs1kySQrIlh66vIzI9iVhdWtkCw,91136
+nulldtype.py,sha256=x0ANjop7fWbvQtaV47KrJK3UUFjhdgpGZ-SGg0UiCGs,19612
+nulldtype-0.1.0.dist-info/licenses/LICENSE.txt,sha256=F9FvJXlkhauPL_Ni5Pc0GRxYlnPoOCWkcIbK8Vc8F14,1582
+nulldtype-0.1.0.dist-info/METADATA,sha256=cDsPlOfBSnGbQwXFIHAnOO7Omrdq-pEKuvhE3tsF_44,15096
+nulldtype-0.1.0.dist-info/WHEEL,sha256=0LUNoHxLvcpw9db_x0XozE5jW73-_GOAO7fDDJ4Xt4Q,101
+nulldtype-0.1.0.dist-info/top_level.txt,sha256=zpWg0wxM7gHDwrTmY7w1BeguxQodz8rC2Vgrt58HQuM,21
+nulldtype-0.1.0.dist-info/RECORD,,

nulldtype-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.0.0)
+Root-Is-Purelib: false
+Tag: cp313-cp313-win_amd64
+

nulldtype-0.1.0.dist-info/licenses/LICENSE.txt

@@ -0,0 +1,30 @@
+Copyright (c) 2026, the NullableDType authors.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+    * Redistributions of source code must retain the above copyright
+       notice, this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above
+       copyright notice, this list of conditions and the following
+       disclaimer in the documentation and/or other materials provided
+       with the distribution.
+
+    * Neither the name of the NullableDType authors nor the names of any
+       contributors may be used to endorse or promote products derived
+       from this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

nulldtype-0.1.0.dist-info/top_level.txt

@@ -0,0 +1,2 @@
+_nulldtype
+nulldtype

nulldtype.py

@@ -0,0 +1,516 @@
+"""Missing values as a NumPy dtype.
+
+NA is a reserved value of the wrapped type itself, so an element costs exactly
+what the wrapped type costs:
+
+    Nullable(np.float64)   itemsize 8  — every bit but the sign
+    Nullable("U10")        itemsize 40 — the whole cell filled with 0xFF
+    Nullable(record)       itemsize T  — every field its own type's NA
+
+LAYOUTS.md has the pattern for every type.
+"""
+import math
+
+import numpy as np
+from numpy.lib.array_utils import normalize_axis_tuple
+
+from _nulldtype import NA, LongDoubleWarning, NullableDType
+
+__all__ = ["NA", "LongDoubleWarning", "Nullable", "is_nullable",
+           "NullableDType",
+           "isna", "notna", "filled", "to_numpy", "count", "dropna",
+           "all", "any", "array_equal", "isin", "isclose", "allclose", "dot",
+           "argmax", "argmin", "cumsum", "cumprod",
+           "sum", "prod", "min", "max", "mean", "std", "var",
+           "median", "quantile", "percentile", "unique"]
+
+
+def Nullable(t):
+    """`T` with missing values.
+
+    Numbers -- ints, unsigned ints, floats, complex, bool, datetime64 --
+    fixed-width `S`, `U` and `V`, and records of any of these reserve one of
+    their own bit patterns for NA, so an element costs exactly what `T` costs.
+    A record is missing when every field holds its own type's NA.  See
+    LAYOUTS.md for every pattern.  `longdouble` and `clongdouble` are stored as
+    `float64` and `complex128`, with a `LongDoubleWarning` wherever that loses
+    precision.
+    """
+    return NullableDType(t)
+
+
+def is_nullable(dtype):
+    """True for a dtype produced by `Nullable`."""
+    return isinstance(dtype, NullableDType)
+
+
+def _check(a):
+    a = np.asarray(a)
+    if not is_nullable(a.dtype):
+        raise TypeError(f"expected a Nullable array, got {a.dtype!r}")
+    # `ascontiguousarray` hands back a 1-d array for a 0-d one, and a scalar
+    # must stay a scalar all the way through `isna` and `filled`
+    return np.ascontiguousarray(a).reshape(a.shape)
+
+
+def _parts(a):
+    """Plain arrays of the values and of the validity flags."""
+    a = _check(a)
+    values = a.view(a.dtype.wrapped)
+    return values, ~_na_mask(values)
+
+
+def _na_mask(values):
+    """True where a plain array of a stored dtype holds that dtype's NA."""
+    if values.ndim == 0:
+        return _na_mask(values.reshape(1)).reshape(())
+    dt = values.dtype
+    if dt.names is not None:
+        # a record is missing when every field is: one field with a value makes
+        # it a value, and padding belongs to no field
+        na = np.ones(values.shape, dtype=bool)
+        for name in dt.names:
+            per_record = int(np.prod(dt.fields[name][0].shape, dtype=np.intp))
+            field_na = _na_mask(values[name])
+            na &= field_na.reshape(values.shape + (per_record,)).all(axis=-1)
+        return na
+
+    kind, size = dt.kind, dt.itemsize
+    if kind in "cSUV":
+        # the views below change the item size, which needs contiguous data,
+        # and a record's fields are strided views
+        values = np.ascontiguousarray(values)
+    if kind == "f" and size == 8:
+        return _na_float(values.view(np.uint64), 0x7FFFFFFFFFFFFFFF)
+    if kind == "f" and size == 4:
+        return _na_float(values.view(np.uint32), 0x7FFFFFFF)
+    if kind == "f" and size == 2:
+        return _na_float(values.view(np.uint16), 0x7FFF)
+    if kind == "c":
+        # the pattern lives in the real half, which is the first field
+        half = np.float64 if size == 16 else np.float32
+        return _parts_na_real(values.view(half).reshape(values.shape + (2,))[..., 0])
+    if kind == "b":
+        return values.view(np.uint8) == 2
+    if kind in "Mm":
+        # numpy already spells this NaT, so ask numpy
+        return np.isnat(values)
+    if kind == "U":
+        # every character U+FFFF, the noncharacter kept for internal use
+        chars = values.view(np.uint32).reshape(values.shape + (size // 4,))
+        return (chars == 0xFFFF).all(axis=-1)
+    if kind in "SV":
+        # the whole cell filled with 0xFF; b"\xffab" is an ordinary value
+        raw = values.view(np.uint8).reshape(values.shape + (size,))
+        return (raw == 0xFF).all(axis=-1)
+    if kind == "u":
+        # UINT_MAX, the unsigned mirror of INT_MIN; `min` would be 0 here
+        return values == np.iinfo(dt).max
+    return values == np.iinfo(dt).min
+
+
+def _na_float(bits, pattern):
+    """NA is every bit but the sign; the sign itself is ignored."""
+    p = bits.dtype.type(pattern)
+    return (bits & p) == p
+
+
+def _parts_na_real(real):
+    if real.dtype == np.float64:
+        return _na_float(real.view(np.uint64), 0x7FFFFFFFFFFFFFFF)
+    return _na_float(real.view(np.uint32), 0x7FFFFFFF)
+
+
+def _as_fill(values, value):
+    """A tuple standing in for a record becomes a record `np.where` can use."""
+    if values.dtype.names is not None and not isinstance(value, np.ndarray):
+        return np.array(value, dtype=values.dtype)
+    return value
+
+
+def notna(a):
+    """A plain bool array, True where the element has a value."""
+    return _parts(a)[1]
+
+
+def isna(a):
+    """A plain bool array, True where the element is missing."""
+    return ~notna(a)
+
+
+def filled(a, value):
+    """A plain array of the wrapped dtype, with `value` in place of NA."""
+    values, valid = _parts(a)
+    return np.where(valid, values, _as_fill(values, value))
+
+
+def to_numpy(a, na_value=None):
+    """Leave the missing-data world.  Refuses to guess if anything is missing."""
+    values, valid = _parts(a)
+    if na_value is None:
+        if not valid.all():
+            raise ValueError(
+                "array contains missing values; pass na_value= to say what "
+                "they should become")
+        return values.copy()
+    return np.where(valid, values, _as_fill(values, na_value))
+
+
+def count(a):
+    """How many elements have a value.  `numpy.ma` spells this `a.count()`."""
+    return int(notna(a).sum())
+
+
+def dropna(a):
+    """The same array with the missing elements taken out."""
+    a = _check(a)
+    return a[notna(a)]
+
+
+def dot(a, b):
+    """`np.dot` cannot be implemented -- its legacy slot is handed a NULL array
+    -- but for 1-D and 2-D operands `@` is the same operation."""
+    x, y = np.asarray(a), np.asarray(b)
+    if x.ndim > 2 or y.ndim > 2:
+        raise ValueError("nd.dot covers 1-D and 2-D; for more, use `@` directly")
+    return x @ y
+
+
+def array_equal(a1, a2):
+    """True, False, or NA when a missing element leaves the answer undecided.
+
+    `np.array_equal` calls `np.all`, which pins its accumulator to a plain bool.
+    """
+    x, y = np.asarray(a1), np.asarray(a2)
+    if x.shape != y.shape:
+        return False
+    if x.size == 0:
+        return True
+    return all(x == y)
+
+
+def isin(a, test_elements):
+    """Like `np.isin`, but an element that is missing answers NA instead of
+    raising: whether an unknown value is in a set is itself unknown."""
+    values, valid = _parts(a)
+    test = np.asarray(test_elements)
+    gap_in_test = False
+    if is_nullable(test.dtype):
+        gap_in_test = bool(isna(test).any())
+        test = to_numpy(dropna(test))
+    found = np.isin(values, test)
+    out = np.zeros(values.shape, dtype=Nullable(np.bool_))
+    out[...] = found
+    if gap_in_test:
+        # a gap in the set could be any value, so not finding x among the known
+        # ones does not rule x out -- SQL's `3 IN (1, NULL)` is NULL too
+        out[~found] = NA
+    out[~valid] = NA
+    return out
+
+
+def _arg_extreme(a, skipna, find):
+    a = _check(a)
+    if not skipna:
+        return find(a)
+    values, valid = _parts(a)
+    valid = valid.ravel()
+    if not valid.any():
+        return NA
+    positions = np.flatnonzero(valid)
+    return positions[find(values.ravel()[valid])]
+
+
+def argmax(a, skipna=False):
+    """Flat position of the largest value.
+
+    A gap could be anything, so by default the answer is the position of the
+    first gap -- numpy's rule for NaN, and what keeps `a[nd.argmax(a)]` equal to
+    `nd.max(a)`.  `skipna=True` skips the gaps, like `np.nanargmax`, and is NA
+    when nothing is left.
+    """
+    return _arg_extreme(a, skipna, np.argmax)
+
+
+def argmin(a, skipna=False):
+    """Flat position of the smallest value; see `argmax`."""
+    return _arg_extreme(a, skipna, np.argmin)
+
+
+def isclose(a, b, rtol=1e-05, atol=1e-08):
+    """`np.isclose` asks `np.result_type(dtype, 1.0)` before anything else and
+    gives up there; this one is the same formula over nullable arithmetic."""
+    x, y = np.asarray(a), np.asarray(b)
+    return abs(x - y) <= atol + rtol * abs(y)
+
+
+def allclose(a, b, rtol=1e-05, atol=1e-08):
+    """Kleene `allclose`: NA when a missing element leaves it undecided."""
+    return all(isclose(a, b, rtol, atol))
+
+
+def all(a, axis=None, skipna=False, keepdims=False):
+    """Kleene `all`: False wins, otherwise NA if anything is missing.
+    `skipna=True` asks about the known elements only, and is a plain bool.
+
+    `np.all` cannot be used: it forces a plain bool result.
+    """
+    if skipna:
+        return np.all(filled(a, True), axis=axis, keepdims=keepdims)
+    return np.logical_and.reduce(_check(a), axis=axis, keepdims=keepdims)
+
+
+def any(a, axis=None, skipna=False, keepdims=False):
+    """Kleene `any`: True wins, otherwise NA if anything is missing.  See `all`."""
+    if skipna:
+        return np.any(filled(a, False), axis=axis, keepdims=keepdims)
+    return np.logical_or.reduce(_check(a), axis=axis, keepdims=keepdims)
+
+
+def _accumulate(name, a, axis, skipna):
+    arr = _check(a)
+    if axis is None:
+        arr, axis = arr.ravel(), 0
+    # numpy widens bool and small ints to the platform int before summing
+    widened = getattr(np, name)(np.zeros(0, arr.dtype.wrapped)).dtype
+    if widened != arr.dtype.wrapped:
+        arr = arr.astype(Nullable(widened))
+    if not skipna:
+        return getattr(np, name)(arr, axis=axis)
+    gaps = isna(arr)
+    out = np.empty(arr.shape, dtype=arr.dtype)
+    out[...] = getattr(np, name)(filled(arr, _IDENTITY[name[3:]]), axis=axis)
+    out[gaps] = NA
+    return out
+
+
+def cumsum(a, axis=None, skipna=False):
+    """Running sum.  By default everything from the first gap on is NA.
+    `skipna=True` carries the total past a gap and leaves NA at the gap
+    itself -- pandas' `cumsum`.  `axis=None` flattens, like `np.cumsum`."""
+    return _accumulate("cumsum", a, axis, skipna)
+
+
+def cumprod(a, axis=None, skipna=False):
+    """Running product; see `cumsum`."""
+    return _accumulate("cumprod", a, axis, skipna)
+
+
+# --------------------------------------------------------------- statistics
+#
+# `np.median`, `np.percentile` and `np.quantile` are written in Python on top
+# of `sort`/`partition`.  NA sorts last, so they take it for the largest value
+# and hand back a plausible-looking number:
+#
+#     >>> np.median(np.array([3.0, 1.0, NA, 5.0], dtype=Nullable(np.float64)))
+#     4.0          # the median of 1, 3, 5 is 3.0
+#
+# Nothing in a dtype can intercept that -- removing `compare` would only break
+# `sort` as well.  These take its place, and follow R: propagate by default,
+# `skipna=True` to drop.  A reduction over nothing left is NA, not zero.
+#
+# Reductions that go through ufuncs (`a.sum()`, `a.mean()`, `a.std()`) already
+# propagate correctly; the versions here exist so that `skipna=` reads the same
+# way across the whole surface instead of turning into `where=nd.notna(a)`.
+
+
+# Reducing over nothing.  `sum` and `prod` have identity elements, so 0 and 1
+# are the answers, not inventions -- R agrees, and so does numpy.  The rest
+# have no identity: the mean, median or maximum of no numbers is not a number,
+# it is unknown, so it is NA.  Returning NaN there would be worse than useless
+# here, because in this dtype NaN is an ordinary value that some column might
+# legitimately hold.
+#
+# Nothing distinguishes an input that was empty to begin with from one that
+# `skipna` emptied, so both take this path.
+_IDENTITY = {"sum": 0, "prod": 1}
+
+
+def _stat_values(a, skipna):
+    """The plain values to compute on, or None when the answer is NA."""
+    values, valid = _parts(a)
+    if not skipna:
+        return None if not valid.all() else values
+    return values[valid]
+
+
+def _all_missing_like(a, shape):
+    """An all-NA result in `a`'s own dtype, for a call that asked for an array."""
+    out = np.empty(shape, dtype=np.asarray(a).dtype)
+    out[...] = NA
+    return out
+
+
+def _reduce_empty(name, dtype):
+    if name not in _IDENTITY:
+        return NA
+    return dtype.type(_IDENTITY[name])
+
+
+def _whole(name, a, skipna, kwargs):
+    """The whole array down to one plain scalar, or NA."""
+    values = _stat_values(a, skipna)
+    if values is None:
+        return NA
+    if values.size == 0:
+        return _reduce_empty(name, values.dtype)
+    return getattr(np, name)(values, **kwargs)
+
+
+def _result_dtype(name, wrapped, kwargs):
+    """What `np.<name>` returns for `wrapped`, asked of a throwaway array."""
+    with np.errstate(all="ignore"):
+        return np.asarray(getattr(np, name)(np.ones(3, wrapped), **kwargs)).dtype
+
+
+def _lanes(name, a, axis, keepdims, one, kwargs):
+    """Apply `one` -- a whole-array reduction -- to every lane along `axis`.
+
+    A Python loop over the lanes, each one handed to numpy's own function, so
+    the numbers are numpy's.  `numpy.ma` would be faster and is wrong for this:
+    it masks every non-finite result, so the mean of `[inf, 1]` comes back
+    masked and its std as 0.
+    """
+    arr = _check(a)
+    axes = normalize_axis_tuple(range(arr.ndim) if axis is None else axis, arr.ndim)
+    rest = [i for i in range(arr.ndim) if i not in axes]
+    moved = arr.transpose(rest + list(axes))
+    outer = moved.shape[:len(rest)]
+    lanes = moved.reshape(math.prod(outer), math.prod(moved.shape[len(rest):]))
+
+    results = [one(lane) for lane in lanes]
+    q_shape = ()
+    for r in results:
+        if r is not NA:
+            q_shape = np.shape(r)
+            break
+    else:
+        q_shape = np.shape(kwargs["q"]) if "q" in kwargs else ()
+    out = np.empty((len(results),) + q_shape,
+                   dtype=Nullable(_result_dtype(name, arr.dtype.wrapped, kwargs)))
+    for i, r in enumerate(results):
+        if r is NA or (isinstance(r, np.ndarray) and is_nullable(r.dtype)):
+            out[i] = NA          # a Nullable array here is `_all_missing_like`
+        else:
+            out[i] = r
+    out = out.reshape(outer + q_shape)
+    # numpy puts the dimensions of an array `q` in front
+    out = np.moveaxis(out, range(len(outer), out.ndim), range(len(q_shape)))
+    if keepdims:
+        out = np.expand_dims(out, tuple(len(q_shape) + i for i in axes))
+    return out
+
+
+def _simple(name):
+    def stat(a, axis=None, skipna=False, keepdims=False, **kwargs):
+        if axis is None and not keepdims:
+            return _whole(name, a, skipna, kwargs)
+        if name in _IDENTITY:
+            # vectorised: fill the gaps with the identity, then put NA back on
+            # every lane that had one
+            values, valid = _parts(a)
+            plain = np.where(valid, values, values.dtype.type(_IDENTITY[name]))
+            got = np.asarray(getattr(np, name)(plain, axis=axis, keepdims=keepdims,
+                                               **kwargs))
+            out = np.empty(got.shape, dtype=Nullable(got.dtype))
+            out[...] = got
+            if not skipna:
+                out[~np.logical_and.reduce(valid, axis=axis, keepdims=keepdims)] = NA
+            return out
+        return _lanes(name, a, axis, keepdims,
+                      lambda lane: _whole(name, lane, skipna, kwargs), kwargs)
+    stat.__name__ = name
+    stat.__doc__ = (f"`np.{name}`, propagating NA unless `skipna=True`.  With "
+                    f"no `axis`, a plain scalar or NA; with one, a Nullable "
+                    f"array with NA on every lane that is missing something.")
+    return stat
+
+
+sum = _simple("sum")
+prod = _simple("prod")
+min = _simple("min")
+max = _simple("max")
+mean = _simple("mean")
+std = _simple("std")
+var = _simple("var")
+median = _simple("median")
+
+
+def _quantile_like(name, a, q, axis, skipna, keepdims, kwargs):
+    def one(lane):
+        values = _stat_values(lane, skipna)
+        if values is None or values.size == 0:
+            return NA if np.ndim(q) == 0 else _all_missing_like(lane, np.shape(q))
+        return getattr(np, name)(values, q, **kwargs)
+    if axis is None and not keepdims:
+        return one(a)
+    return _lanes(name, a, axis, keepdims, one, dict(kwargs, q=q))
+
+
+def quantile(a, q, axis=None, skipna=False, keepdims=False, **kwargs):
+    """`np.quantile`, propagating NA unless `skipna=True`.
+
+    A scalar `q` gives a scalar or NA; an array `q` gives an array, all NA if
+    the input was missing anything.  With `axis`, NA per lane.
+    """
+    return _quantile_like("quantile", a, q, axis, skipna, keepdims, kwargs)
+
+
+def percentile(a, q, axis=None, skipna=False, keepdims=False, **kwargs):
+    """`np.percentile`, propagating NA unless `skipna=True`.  See `quantile`."""
+    return _quantile_like("percentile", a, q, axis, skipna, keepdims, kwargs)
+
+
+def unique(a):
+    """R's `unique`: NA is one distinct value of its own, and it sorts last.
+
+    `np.unique` cannot be used -- it compares neighbours in the sorted array
+    and assigns the result into a plain bool mask, which NA refuses to become.
+    """
+    arr = _check(a)
+    values, valid = _parts(arr)
+    found = np.unique(values[valid]).astype(arr.dtype)
+    if valid.all():
+        return found
+    out = np.empty(found.size + 1, dtype=arr.dtype)
+    out[:found.size] = found
+    out[found.size] = NA
+    return out
+
+
+# --------------------------------------------------------------- the rest
+#
+# Everything numpy already gets right on a Nullable array -- `sort`, `clip`,
+# `concatenate`, every ufunc -- is forwarded untouched, so `nd.sort` *is*
+# `np.sort`.  Only two kinds of name are not: the ones defined above, which
+# numpy gets wrong or refuses, and the ones below, which would look like they
+# work.  `VS-NUMPY-MA.md` has the measured list.
+_REFUSED = {
+    name: "NA is not NaN: these skip NaN, which a gap only happens to be for "
+          "floats.  Use nd.{}(a, skipna=True)".format(name[3:])
+    for name in ("nansum", "nanprod", "nanmean", "nanstd", "nanvar",
+                 "nanmedian", "nanmax", "nanmin", "nanquantile", "nanpercentile")
+}
+_REFUSED.update({
+    "nanargmax": "use nd.argmax(a, skipna=True); NA is not NaN",
+    "nanargmin": "use nd.argmin(a, skipna=True); NA is not NaN",
+    "nancumsum": "use nd.cumsum(a, skipna=True); NA is not NaN",
+    "nancumprod": "use nd.cumprod(a, skipna=True); NA is not NaN",
+})
+
+
+def __getattr__(name):
+    message = _REFUSED.get(name)
+    if message is not None:
+        raise AttributeError(f"nd.{name}: {message}")
+    try:
+        forwarded = getattr(np, name)
+    except AttributeError:
+        raise AttributeError(
+            f"module 'nulldtype' has no attribute {name!r}") from None
+    globals()[name] = forwarded       # forward once, then it is just numpy
+    return forwarded
+
+
+def __dir__():
+    return sorted(set(__all__) | set(n for n in dir(np) if not n.startswith("_")))