nulldtype 0.1.0__tar.gz → 0.1.1__tar.gz

{nulldtype-0.1.0/nulldtype.egg-info → nulldtype-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nulldtype
-Version: 0.1.0
+Version: 0.1.1
 Summary: Missing values as a NumPy DType: NA stored as a reserved bit pattern of the wrapped type
 License-Expression: BSD-3-Clause
 Project-URL: Source, https://github.com/tuan2k33/NullableDType
@@ -44,6 +44,7 @@ two layers compose. In the terms of the 2012 NumPy discussion, this is
 | `LAYOUTS.md` | the NA pattern of every type, and what each gives up |
 | `VS-NUMPY-MA.md` | all 218 names in `numpy.ma.__all__` that numpy 2 still spells that way, side by side with `nd`, with real calls and results |
 | `DISCUSSION.md` | how it works and why: the borrowed loops, performance, the bugs found on the way, leak checking |
+| `NUMPY-MA-PITFALLS.md` | `numpy.ma` behaviours that leak, mislead or raise, with causes and issue links |
 | `NUMPY-PATCHES.md` | where numpy's array functions fail on Nullable, and patches that leave plain arrays unchanged |
 | `AGENTS.md` | notes for coding agents working on the repo |
 | `archive/flag-layout/` | the earlier two-layout version, frozen |
@@ -67,7 +68,7 @@ builds and runs pytest (extra arguments go to pytest). They use `python3`;
 set `PYTHON`, and `NUMPY_SITE` for a numpy dev tree that is not installed, or
 put both in an uncommitted `local.env`.
 
-The suite has 582 tests plus 2 expected failures that pin known gaps. It also
+The suite has 637 tests plus 2 expected failures that pin known gaps. It also
 passes on an ASAN + UBSAN build, which CI runs on every push.
 
 ## What works
@@ -128,6 +129,7 @@ nd.Nullable(">i4")           # Nullable(int32) — stored in native byte order
 | Results follow numpy's rules | `resolve` asks the wrapped ufunc, so `Nullable[i8] / Nullable[i8]` is `Nullable[f8]` and `Nullable[i8] + Nullable[f8]` is `Nullable[f8]` |
 | Mixed operands | `Nullable[T] op T` and Python scalars, both ways round, through promoters |
 | Comparisons | answer `Nullable[bool]`; `NA == NA` is `NA`, not `True` |
+| Answers that do not depend on the gap | `1 ** NA` and `NA ** 0` are 1, `hypot(inf, NA)` is inf, `heaviside(3, NA)` is 1 — as in R and pandas; `fmax(NA, 2)` and `copysign(1, NA)` do depend on it and are NA |
 | Three-valued logic | `logical_and/or/xor` and `&`, `\|`, `^` on `Nullable[bool]` are Kleene: `NA & False = False`, `NA \| True = True`; on ints `&`, `\|`, `^` propagate |
 | Reductions and `accumulate` | propagate by default, like R's `na.rm = FALSE`, over several axes at once too; `cumsum` is NA from the first gap on |
 | `argmax`, `argmin` | the position of the first gap, numpy's rule for NaN, so `a[a.argmax()]` is NA exactly when `a.max()` is |
@@ -241,8 +243,8 @@ plain arrays, is in `NUMPY-PATCHES.md`.
    widening to `int64`.
 3. **A real `longdouble`** through `numpy-quaddtype` instead of substituting
    `float64`.
-4. **Remove the fixed cost of `get_loop`** — two Python calls per operation, most
-   visible on small arrays.
+4. **Cache the loop lookups** — about 1.4 µs of the ~2.5 µs fixed cost per
+   operation, most visible on small arrays.
 
 ## Settled design notes
 

{nulldtype-0.1.0 → nulldtype-0.1.1}/README.md

@@ -26,6 +26,7 @@ two layers compose. In the terms of the 2012 NumPy discussion, this is
 | `LAYOUTS.md` | the NA pattern of every type, and what each gives up |
 | `VS-NUMPY-MA.md` | all 218 names in `numpy.ma.__all__` that numpy 2 still spells that way, side by side with `nd`, with real calls and results |
 | `DISCUSSION.md` | how it works and why: the borrowed loops, performance, the bugs found on the way, leak checking |
+| `NUMPY-MA-PITFALLS.md` | `numpy.ma` behaviours that leak, mislead or raise, with causes and issue links |
 | `NUMPY-PATCHES.md` | where numpy's array functions fail on Nullable, and patches that leave plain arrays unchanged |
 | `AGENTS.md` | notes for coding agents working on the repo |
 | `archive/flag-layout/` | the earlier two-layout version, frozen |
@@ -49,7 +50,7 @@ builds and runs pytest (extra arguments go to pytest). They use `python3`;
 set `PYTHON`, and `NUMPY_SITE` for a numpy dev tree that is not installed, or
 put both in an uncommitted `local.env`.
 
-The suite has 582 tests plus 2 expected failures that pin known gaps. It also
+The suite has 637 tests plus 2 expected failures that pin known gaps. It also
 passes on an ASAN + UBSAN build, which CI runs on every push.
 
 ## What works
@@ -110,6 +111,7 @@ nd.Nullable(">i4")           # Nullable(int32) — stored in native byte order
 | Results follow numpy's rules | `resolve` asks the wrapped ufunc, so `Nullable[i8] / Nullable[i8]` is `Nullable[f8]` and `Nullable[i8] + Nullable[f8]` is `Nullable[f8]` |
 | Mixed operands | `Nullable[T] op T` and Python scalars, both ways round, through promoters |
 | Comparisons | answer `Nullable[bool]`; `NA == NA` is `NA`, not `True` |
+| Answers that do not depend on the gap | `1 ** NA` and `NA ** 0` are 1, `hypot(inf, NA)` is inf, `heaviside(3, NA)` is 1 — as in R and pandas; `fmax(NA, 2)` and `copysign(1, NA)` do depend on it and are NA |
 | Three-valued logic | `logical_and/or/xor` and `&`, `\|`, `^` on `Nullable[bool]` are Kleene: `NA & False = False`, `NA \| True = True`; on ints `&`, `\|`, `^` propagate |
 | Reductions and `accumulate` | propagate by default, like R's `na.rm = FALSE`, over several axes at once too; `cumsum` is NA from the first gap on |
 | `argmax`, `argmin` | the position of the first gap, numpy's rule for NaN, so `a[a.argmax()]` is NA exactly when `a.max()` is |
@@ -223,8 +225,8 @@ plain arrays, is in `NUMPY-PATCHES.md`.
    widening to `int64`.
 3. **A real `longdouble`** through `numpy-quaddtype` instead of substituting
    `float64`.
-4. **Remove the fixed cost of `get_loop`** — two Python calls per operation, most
-   visible on small arrays.
+4. **Cache the loop lookups** — about 1.4 µs of the ~2.5 µs fixed cost per
+   operation, most visible on small arrays.
 
 ## Settled design notes
 

{nulldtype-0.1.0 → nulldtype-0.1.1/nulldtype.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nulldtype
-Version: 0.1.0
+Version: 0.1.1
 Summary: Missing values as a NumPy DType: NA stored as a reserved bit pattern of the wrapped type
 License-Expression: BSD-3-Clause
 Project-URL: Source, https://github.com/tuan2k33/NullableDType
@@ -44,6 +44,7 @@ two layers compose. In the terms of the 2012 NumPy discussion, this is
 | `LAYOUTS.md` | the NA pattern of every type, and what each gives up |
 | `VS-NUMPY-MA.md` | all 218 names in `numpy.ma.__all__` that numpy 2 still spells that way, side by side with `nd`, with real calls and results |
 | `DISCUSSION.md` | how it works and why: the borrowed loops, performance, the bugs found on the way, leak checking |
+| `NUMPY-MA-PITFALLS.md` | `numpy.ma` behaviours that leak, mislead or raise, with causes and issue links |
 | `NUMPY-PATCHES.md` | where numpy's array functions fail on Nullable, and patches that leave plain arrays unchanged |
 | `AGENTS.md` | notes for coding agents working on the repo |
 | `archive/flag-layout/` | the earlier two-layout version, frozen |
@@ -67,7 +68,7 @@ builds and runs pytest (extra arguments go to pytest). They use `python3`;
 set `PYTHON`, and `NUMPY_SITE` for a numpy dev tree that is not installed, or
 put both in an uncommitted `local.env`.
 
-The suite has 582 tests plus 2 expected failures that pin known gaps. It also
+The suite has 637 tests plus 2 expected failures that pin known gaps. It also
 passes on an ASAN + UBSAN build, which CI runs on every push.
 
 ## What works
@@ -128,6 +129,7 @@ nd.Nullable(">i4")           # Nullable(int32) — stored in native byte order
 | Results follow numpy's rules | `resolve` asks the wrapped ufunc, so `Nullable[i8] / Nullable[i8]` is `Nullable[f8]` and `Nullable[i8] + Nullable[f8]` is `Nullable[f8]` |
 | Mixed operands | `Nullable[T] op T` and Python scalars, both ways round, through promoters |
 | Comparisons | answer `Nullable[bool]`; `NA == NA` is `NA`, not `True` |
+| Answers that do not depend on the gap | `1 ** NA` and `NA ** 0` are 1, `hypot(inf, NA)` is inf, `heaviside(3, NA)` is 1 — as in R and pandas; `fmax(NA, 2)` and `copysign(1, NA)` do depend on it and are NA |
 | Three-valued logic | `logical_and/or/xor` and `&`, `\|`, `^` on `Nullable[bool]` are Kleene: `NA & False = False`, `NA \| True = True`; on ints `&`, `\|`, `^` propagate |
 | Reductions and `accumulate` | propagate by default, like R's `na.rm = FALSE`, over several axes at once too; `cumsum` is NA from the first gap on |
 | `argmax`, `argmin` | the position of the first gap, numpy's rule for NaN, so `a[a.argmax()]` is NA exactly when `a.max()` is |
@@ -241,8 +243,8 @@ plain arrays, is in `NUMPY-PATCHES.md`.
    widening to `int64`.
 3. **A real `longdouble`** through `numpy-quaddtype` instead of substituting
    `float64`.
-4. **Remove the fixed cost of `get_loop`** — two Python calls per operation, most
-   visible on small arrays.
+4. **Cache the loop lookups** — about 1.4 µs of the ~2.5 µs fixed cost per
+   operation, most visible on small arrays.
 
 ## Settled design notes
 

{nulldtype-0.1.0 → nulldtype-0.1.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "nulldtype"
-version = "0.1.0"
+version = "0.1.1"
 description = "Missing values as a NumPy DType: NA stored as a reserved bit pattern of the wrapped type"
 readme = "README.md"
 license = "BSD-3-Clause"

nulldtype-0.1.1/setup.py

@@ -0,0 +1,31 @@
+"""The C extension; everything else is in pyproject.toml."""
+import platform
+
+import numpy as np
+from setuptools import Extension, setup
+from setuptools.command.build_ext import build_ext
+
+
+class BuildExt(build_ext):
+    """numpy >= 2.5 already requires x86-64-v2 (SSE4.2) on x86-64, so the NA
+    scans may use it: SSE2 has no 64-bit compare, and they vectorise badly
+    without one.  GCC and Clang only; MSVC and other CPUs keep their default."""
+
+    def build_extensions(self):
+        x86_64 = platform.machine().lower() in ("x86_64", "amd64")
+        if x86_64 and self.compiler.compiler_type != "msvc":
+            for ext in self.extensions:
+                ext.extra_compile_args += ["-march=x86-64-v2", "-mtune=generic"]
+        super().build_extensions()
+
+
+setup(
+    ext_modules=[
+        Extension(
+            "_nulldtype",
+            sources=["src/nulldtype.c"],
+            include_dirs=[np.get_include()],
+        ),
+    ],
+    cmdclass={"build_ext": BuildExt},
+)

{nulldtype-0.1.0 → nulldtype-0.1.1}/src/nulldtype.c

@@ -38,6 +38,11 @@
 #include "numpy/ufuncobject.h"
 #include "numpy/dtype_api.h"
 
+/* MSVC's C mode spells C99 `restrict` as `__restrict` */
+#if defined(_MSC_VER) && !defined(__cplusplus) && !defined(restrict)
+#define restrict __restrict
+#endif
+
 static PyArray_DTypeMeta NullableDType;
 
 /* the plain dtypes a Nullable one is the common dtype of */
@@ -642,22 +647,80 @@ typedef struct {
  * failure rather than as a stray warning in someone's output.
  */
 static int
-binop_signals_on_gap(int idx)
+binop_signals_on_gap(int idx, int complex_operand)
 {
     const char *n = binop_names[idx];
-    return strcmp(n, "logaddexp") == 0 || strcmp(n, "logaddexp2") == 0;
+    if (strcmp(n, "logaddexp") == 0 || strcmp(n, "logaddexp2") == 0) {
+        return 1;
+    }
+    /*
+     * The division family raises on a gap in several ways: an integer gap is
+     * INT_MIN, so `NA // -1` overflows, and `NA // 0` divides by zero for a
+     * value that is not there; MSVC's float32 `floor_divide` and `fmod`
+     * signal "invalid" on inf and NaN operands.  A real zero divisor still
+     * warns, since the masked path computes every element that has a value.
+     */
+    if (strcmp(n, "floor_divide") == 0 || strcmp(n, "remainder") == 0
+            || strcmp(n, "fmod") == 0) {
+        return 1;
+    }
+    /*
+     * Complex kernels compare magnitudes almost everywhere: division scales by
+     * the larger component, `power` special-cases, ordering is lexicographic
+     * with `<`.  Only the plain componentwise ops stay quiet over a NaN.
+     */
+    if (complex_operand) {
+        return !(strcmp(n, "add") == 0 || strcmp(n, "subtract") == 0
+                 || strcmp(n, "multiply") == 0 || strcmp(n, "equal") == 0
+                 || strcmp(n, "not_equal") == 0);
+    }
+    return 0;
 }
 
 /*
- * Ops that return the non-NaN operand instead of propagating.  A float gap is
- * a NaN, so for these a gap vanishes from the output and cannot be recovered
- * from the result alone.
+ * Ops whose result can be a number although an operand is NaN, so a float gap
+ * can leave no NaN in the output: `fmax`/`fmin` return the other operand,
+ * `copysign` reads only the sign, and the ones in `binop_keeps_determined`.
+ * The fast path may skip a float block whose output holds no NaN only for the
+ * other ops.  `test_every_binop_keeps_a_gap` fails if one is missing here.
  */
+static int binop_keeps_determined(int idx);
+
 static int
 binop_swallows_nan(int idx)
 {
     const char *n = binop_names[idx];
-    return strcmp(n, "fmax") == 0 || strcmp(n, "fmin") == 0;
+    return strcmp(n, "fmax") == 0 || strcmp(n, "fmin") == 0
+            || strcmp(n, "copysign") == 0 || binop_keeps_determined(idx);
+}
+
+/*
+ * Ops whose IEEE 754 kernel already answers "does the result depend on the
+ * NaN operand?": it returns a number exactly when it does not.  A gap is a
+ * value nobody knows, so the same rule applies to it, as in R and pandas:
+ *
+ *   power, float_power   1 ** NA == 1, NA ** 0 == 1
+ *   heaviside            heaviside(x, NA) is 0 or 1 when x != 0
+ *   hypot                hypot(±inf, NA) == inf
+ *
+ * For these a float gap is stamped only where the output came out NaN.  Other
+ * ops that turn a NaN into a number (`fmax`, `fmin`, `copysign`) do depend on
+ * the missing value, so they are not listed and always give NA.  Integer `power` gets the same two rules
+ * in `nullable_int_power_rules`.
+ */
+static int
+binop_keeps_determined(int idx)
+{
+    const char *n = binop_names[idx];
+    return strcmp(n, "power") == 0 || strcmp(n, "float_power") == 0
+            || strcmp(n, "heaviside") == 0 || strcmp(n, "hypot") == 0;
+}
+
+static int
+binop_is_power(int idx)
+{
+    const char *n = binop_names[idx];
+    return strcmp(n, "power") == 0 || strcmp(n, "float_power") == 0;
 }
 
 /*
@@ -1833,114 +1896,42 @@ nullable_computed_na(const NullableDescr *out)
 }
 
 
-#define NA_FIXUP(CTYPE, MASKEXPR, CHECK)                                      \
-    do {                                                                      \
-        const CTYPE *p0 = (const CTYPE *)data[0];                             \
-        const CTYPE *p1 = (const CTYPE *)data[1];                             \
-        CTYPE *po = (CTYPE *)data[2];                                         \
-        CTYPE na;                                                             \
-        memcpy(&na, out->na_bytes, sizeof(CTYPE));                            \
-        for (npy_intp i = 0; i < N; i++) {                                    \
-            CTYPE x = p0[i], y = p1[i];                                       \
-            if (MASKEXPR) {                                                   \
-                po[i] = na;                                                   \
-            }                                                                 \
-            else if (CHECK) {                                                 \
-                made |= (po[i] == na);                                        \
-            }                                                                 \
-        }                                                                     \
-    } while (0)
+#define ISNA_F8(v) (((v) & 0x7FFFFFFFFFFFFFFFULL) == 0x7FFFFFFFFFFFFFFFULL)
+#define ISNA_F4(v) (((v) & 0x7FFFFFFFU) == 0x7FFFFFFFU)
+#define ISNA_EQ(v) ((v) == na)
+/* NaN: exponent all ones, mantissa not zero */
+#define ISNAN_F8(v) ((((v) & 0x7FF0000000000000ULL) == 0x7FF0000000000000ULL) \
+                     & (((v) & 0x000FFFFFFFFFFFFFULL) != 0))
+#define ISNAN_F4(v) ((((v) & 0x7F800000U) == 0x7F800000U) & (((v) & 0x007FFFFFU) != 0))
+/* where may a gap be stamped: everywhere, or only where the op gave a NaN */
+#define STAMP_F8(v) ((!keep_numbers) | ISNAN_F8(v))
+#define STAMP_F4(v) ((!keep_numbers) | ISNAN_F4(v))
+#define STAMP_ANY(v) 1
 
+/* the output cell is a NaN; kinds without a NaN say yes, so they are stamped */
 static int
-nullable_fixup(NullableDescr *in0, NullableDescr *in1, NullableDescr *out,
-        char *const data[], npy_intp const strides[], npy_intp N,
-        int propagates_nan)
+nullable_out_is_nan(const NullableDescr *out, const char *p)
 {
-    npy_intp item = out->base.elsize;
-    int check = na_can_be_computed(out->kind);
-    int made = 0;
-    int packed = (strides[0] == in0->base.elsize)
-              && (strides[1] == in1->base.elsize)
-              && (strides[2] == item)
-              && in0->kind == out->kind && in1->kind == out->kind;
-
-    if (packed) {
-        /*
-         * For floats the hardware has already put a NaN everywhere an input
-         * was NA, so a single pass over the freshly written output — still hot
-         * in cache — tells us whether there is anything to fix at all.  Most
-         * arrays have no gaps and stop here.
-         *
-         * This rests on the op propagating NaN.  `fmax`/`fmin` do not: they
-         * return the non-NaN side on purpose, so a gap leaves no trace in the
-         * output and this scan would skip a fixup that is needed.  They are
-         * flagged by `binop_swallows_nan`, which turns this scan off and
-         * forces the full pass; anything else added to `binop_names` that
-         * swallows NaN must be listed there too.
-         */
-        if (propagates_nan && out->kind == NA_KIND_F8) {
-            const double *po = (const double *)data[2];
-            int any = 0;
-            for (npy_intp i = 0; i < N; i++) {
-                any |= (po[i] != po[i]);
-            }
-            if (!any) {
-                return 0;
-            }
-        }
-        else if (propagates_nan && out->kind == NA_KIND_F4) {
-            const float *po = (const float *)data[2];
-            int any = 0;
-            for (npy_intp i = 0; i < N; i++) {
-                any |= (po[i] != po[i]);
-            }
-            if (!any) {
-                return 0;
-            }
-        }
-        switch (out->kind) {
-            case NA_KIND_F8:
-                NA_FIXUP(npy_uint64,
-                    ((x & 0x7FFFFFFFFFFFFFFFULL) == 0x7FFFFFFFFFFFFFFFULL) ||
-                    ((y & 0x7FFFFFFFFFFFFFFFULL) == 0x7FFFFFFFFFFFFFFFULL), 0);
-                return 0;
-            case NA_KIND_F4:
-                NA_FIXUP(npy_uint32,
-                    ((x & 0x7FFFFFFFU) == 0x7FFFFFFFU) ||
-                    ((y & 0x7FFFFFFFU) == 0x7FFFFFFFU), 0);
-                return 0;
-            case NA_KIND_EXACT:
-                if (item == 8) { NA_FIXUP(npy_uint64, x == na || y == na, 1); goto done; }
-                if (item == 4) { NA_FIXUP(npy_uint32, x == na || y == na, 1); goto done; }
-                if (item == 2) { NA_FIXUP(npy_uint16, x == na || y == na, 1); goto done; }
-                if (item == 1) { NA_FIXUP(npy_uint8,  x == na || y == na, 1); goto done; }
-                break;
-            default:
-                /* f16 and complex: correctness first, via the loop below */
-                break;
-        }
+    if (out->kind == NA_KIND_F8) {
+        npy_uint64 v; memcpy(&v, p, 8); return ISNAN_F8(v);
     }
-    for (npy_intp i = 0; i < N; i++) {
-        if (nullable_is_na(in0, data[0] + i * strides[0])
-                || nullable_is_na(in1, data[1] + i * strides[1])) {
-            nullable_put_na(out, data[2] + i * strides[2]);
-        }
-        else if (check && nullable_is_na(out, data[2] + i * strides[2])) {
-            made = 1;
-        }
+    if (out->kind == NA_KIND_F4) {
+        npy_uint32 v; memcpy(&v, p, 4); return ISNAN_F4(v);
     }
-  done:
-    return made ? nullable_computed_na(out) : 0;
+    if (out->kind == NA_KIND_F2) {
+        npy_uint16 v; memcpy(&v, p, 2);
+        return ((v & 0x7C00U) == 0x7C00U) && ((v & 0x03FFU) != 0);
+    }
+    return 1;
 }
 
-
 /* ------------------------------------------------------- ufunc loops */
 /*
  * The values are contiguous and aligned, so the wrapped loop runs at full
- * speed on the whole chunk.  Afterwards we stamp NA over the positions where
- * an input was missing.  That second pass is what makes `nan + NA` come out
- * as NA regardless of operand order, and it means we never rely on the CPU
- * carrying a NaN payload through the arithmetic.
+ * speed.  The gaps are recorded before it runs and stamped after, so
+ * `nan + NA` comes out as NA regardless of operand order, `a += 1` keeps its
+ * gaps, and nothing relies on the CPU carrying a NaN payload through the
+ * arithmetic or on an op propagating NaN at all.
  */
 
 /*
@@ -2163,6 +2154,326 @@ nullable_binary_resolve_impl(PyObject *ufunc,
 }
 
 
+/*
+ * Gaps are recorded *before* the wrapped loop runs.  Reading the inputs
+ * afterwards is wrong whenever the output is one of them (`a += 1`,
+ * `np.square(a, out=a)`): the loop has already overwritten the gap with
+ * INT_MIN + 1 or 0, and the NA was lost to an ordinary number.
+ *
+ * The work goes in blocks small enough for L1, so each block is scanned,
+ * computed and stamped while it is hot: one trip through memory instead of
+ * one for the wrapped loop and another for the fixup.  The typed scans are
+ * branch-free integer tests, which the compiler vectorises at -O2.
+ */
+#define FIX_BLOCK 1024
+
+/*
+ * One small function per width, with `restrict` parameters: `mask` is a char
+ * pointer, which may alias anything, and GCC at -O2 will not add a runtime
+ * alias check, so without these it vectorises none of this.  The two inputs
+ * may be the same array (`a + a`); `restrict` allows that because neither is
+ * written through.
+ */
+#define DEFINE_SCAN(NAME, CTYPE, ISNA)                                        \
+static unsigned char                                                          \
+NAME(const CTYPE *restrict p0, const CTYPE *restrict p1,                      \
+     unsigned char *restrict mk, npy_intp k, CTYPE na)                        \
+{                                                                             \
+    unsigned char gap = 0;                                                    \
+    (void)na;                                                                 \
+    for (npy_intp i = 0; i < k; i++) {                                        \
+        CTYPE x = p0[i], y = p1[i];                                           \
+        unsigned char m = (unsigned char)(ISNA(x) | ISNA(y));                 \
+        mk[i] = m;                                                            \
+        gap |= m;                                                             \
+    }                                                                         \
+    return gap;                                                               \
+}
+#define DEFINE_SCAN1(NAME, CTYPE, ISNA)                                       \
+static unsigned char                                                          \
+NAME(const CTYPE *restrict p0, unsigned char *restrict mk, npy_intp k,        \
+     CTYPE na)                                                                \
+{                                                                             \
+    unsigned char gap = 0;                                                    \
+    (void)na;                                                                 \
+    for (npy_intp i = 0; i < k; i++) {                                        \
+        CTYPE x = p0[i];                                                      \
+        unsigned char m = (unsigned char)ISNA(x);                             \
+        mk[i] = m;                                                            \
+        gap |= m;                                                             \
+    }                                                                         \
+    return gap;                                                               \
+}
+DEFINE_SCAN1(scan1_f8, npy_uint64, ISNA_F8)
+DEFINE_SCAN1(scan1_f4, npy_uint32, ISNA_F4)
+DEFINE_SCAN1(scan1_u64, npy_uint64, ISNA_EQ)
+DEFINE_SCAN1(scan1_u32, npy_uint32, ISNA_EQ)
+DEFINE_SCAN1(scan1_u16, npy_uint16, ISNA_EQ)
+DEFINE_SCAN1(scan1_u8, npy_uint8, ISNA_EQ)
+#undef DEFINE_SCAN1
+
+/* any NaN or inf (exponent all ones) in a float output block */
+static unsigned char
+special_f8(const npy_uint64 *restrict po, npy_intp k)
+{
+    unsigned char any = 0;
+    for (npy_intp i = 0; i < k; i++) {
+        any |= (unsigned char)((po[i] & 0x7FF0000000000000ULL)
+                               == 0x7FF0000000000000ULL);
+    }
+    return any;
+}
+
+static unsigned char
+special_f4(const npy_uint32 *restrict po, npy_intp k)
+{
+    unsigned char any = 0;
+    for (npy_intp i = 0; i < k; i++) {
+        any |= (unsigned char)((po[i] & 0x7F800000U) == 0x7F800000U);
+    }
+    return any;
+}
+
+DEFINE_SCAN(scan_f8, npy_uint64, ISNA_F8)
+DEFINE_SCAN(scan_f4, npy_uint32, ISNA_F4)
+DEFINE_SCAN(scan_u64, npy_uint64, ISNA_EQ)
+DEFINE_SCAN(scan_u32, npy_uint32, ISNA_EQ)
+DEFINE_SCAN(scan_u16, npy_uint16, ISNA_EQ)
+DEFINE_SCAN(scan_u8, npy_uint8, ISNA_EQ)
+#undef DEFINE_SCAN
+
+/* the reserved pattern of `D` as a `CTYPE`; needs a local `na_<CTYPE>` */
+#define NA_OF(CTYPE, D) (memcpy(&na_##CTYPE, (D)->na_bytes, sizeof(CTYPE)), na_##CTYPE)
+
+/* mask[i] = "an input of element i is a gap"; returns whether any is */
+static int
+nullable_scan_pair(const NullableDescr *in0, const NullableDescr *in1,
+        char *const d[], npy_intp const strides[], npy_intp k,
+        unsigned char *mask)
+{
+    npy_intp item = in0->base.elsize;
+    int packed = strides[0] == item && strides[1] == item
+              && in1->base.elsize == item && in0->kind == in1->kind
+              && memcmp(in0->na_bytes, in1->na_bytes, item) == 0;
+    /* `a + a`, and the unary loops, pass one array twice: read it once */
+    if (packed && d[0] == d[1]) {
+        npy_uint64 na_npy_uint64;
+        npy_uint32 na_npy_uint32;
+        npy_uint16 na_npy_uint16;
+        npy_uint8 na_npy_uint8;
+        switch (in0->kind) {
+            case NA_KIND_F8:
+                return scan1_f8((npy_uint64 *)d[0], mask, k, 0);
+            case NA_KIND_F4:
+                return scan1_f4((npy_uint32 *)d[0], mask, k, 0);
+            case NA_KIND_EXACT:
+                if (item == 8) {
+                    return scan1_u64((npy_uint64 *)d[0], mask, k,
+                                     NA_OF(npy_uint64, in0));
+                }
+                if (item == 4) {
+                    return scan1_u32((npy_uint32 *)d[0], mask, k,
+                                     NA_OF(npy_uint32, in0));
+                }
+                if (item == 2) {
+                    return scan1_u16((npy_uint16 *)d[0], mask, k,
+                                     NA_OF(npy_uint16, in0));
+                }
+                if (item == 1) {
+                    return scan1_u8((npy_uint8 *)d[0], mask, k,
+                                    NA_OF(npy_uint8, in0));
+                }
+                break;
+            default:
+                break;
+        }
+    }
+    if (packed) {
+        npy_uint64 na_npy_uint64;
+        npy_uint32 na_npy_uint32;
+        npy_uint16 na_npy_uint16;
+        npy_uint8 na_npy_uint8;
+        switch (in0->kind) {
+            case NA_KIND_F8:
+                return scan_f8((npy_uint64 *)d[0], (npy_uint64 *)d[1], mask, k, 0);
+            case NA_KIND_F4:
+                return scan_f4((npy_uint32 *)d[0], (npy_uint32 *)d[1], mask, k, 0);
+            case NA_KIND_EXACT:
+                if (item == 8) {
+                    return scan_u64((npy_uint64 *)d[0], (npy_uint64 *)d[1], mask,
+                                    k, NA_OF(npy_uint64, in0));
+                }
+                if (item == 4) {
+                    return scan_u32((npy_uint32 *)d[0], (npy_uint32 *)d[1], mask,
+                                    k, NA_OF(npy_uint32, in0));
+                }
+                if (item == 2) {
+                    return scan_u16((npy_uint16 *)d[0], (npy_uint16 *)d[1], mask,
+                                    k, NA_OF(npy_uint16, in0));
+                }
+                if (item == 1) {
+                    return scan_u8((npy_uint8 *)d[0], (npy_uint8 *)d[1], mask,
+                                   k, NA_OF(npy_uint8, in0));
+                }
+                break;
+            default:
+                break;
+        }
+    }
+    unsigned char gap = 0;
+    for (npy_intp i = 0; i < k; i++) {
+        mask[i] = (unsigned char)(nullable_is_na(in0, d[0] + i * strides[0])
+                                  | nullable_is_na(in1, d[1] + i * strides[1]));
+        gap |= mask[i];
+    }
+    return gap;
+}
+
+/*
+ * The output side: returns 1 if an element without a gap landed on the
+ * reserved value (only looked for when `check`), and stamps NA where `mk`
+ * says -- for floats with `keep_numbers`, only where the op gave a NaN.
+ */
+#define DEFINE_STAMP(NAME, CTYPE, STAMP)                                      \
+static unsigned char                                                          \
+NAME(CTYPE *restrict po, const unsigned char *restrict mk, npy_intp k,       \
+     CTYPE na, int check, int gap, int keep_numbers)                          \
+{                                                                             \
+    (void)keep_numbers;                                                       \
+    if (check) {                                                              \
+        unsigned char hit = 0;                                                \
+        for (npy_intp i = 0; i < k; i++) {                                    \
+            hit |= (unsigned char)(!mk[i] & (po[i] == na));                   \
+        }                                                                     \
+        if (hit) {                                                            \
+            return 1;                                                         \
+        }                                                                     \
+    }                                                                         \
+    if (gap) {                                                                \
+        for (npy_intp i = 0; i < k; i++) {                                    \
+            CTYPE v = po[i];                                                  \
+            unsigned char m = (unsigned char)(mk[i] & STAMP(v));              \
+            po[i] = m ? na : v;                                               \
+        }                                                                     \
+    }                                                                         \
+    return 0;                                                                 \
+}
+DEFINE_STAMP(stamp_f8, npy_uint64, STAMP_F8)
+DEFINE_STAMP(stamp_f4, npy_uint32, STAMP_F4)
+DEFINE_STAMP(stamp_u64, npy_uint64, STAMP_ANY)
+DEFINE_STAMP(stamp_u32, npy_uint32, STAMP_ANY)
+DEFINE_STAMP(stamp_u16, npy_uint16, STAMP_ANY)
+DEFINE_STAMP(stamp_u8, npy_uint8, STAMP_ANY)
+#undef DEFINE_STAMP
+
+/*
+ * Put NA where `mask` says, and refuse a result that landed on the reserved
+ * value elsewhere.  With `keep_numbers` a float gap is stamped only where the
+ * op gave a NaN (`binop_keeps_determined`).
+ */
+static int
+nullable_stamp(const NullableDescr *out, char *o, npy_intp stride, npy_intp k,
+        const unsigned char *mask, int gap, int keep_numbers)
+{
+    int check = na_can_be_computed(out->kind);
+    if (!gap && !check) {
+        return 0;
+    }
+    npy_intp item = out->base.elsize;
+    if (stride == item) {
+        npy_uint64 na_npy_uint64;
+        npy_uint32 na_npy_uint32;
+        npy_uint16 na_npy_uint16;
+        npy_uint8 na_npy_uint8;
+        int hit = -1;
+        switch (out->kind) {
+            case NA_KIND_F8:
+                hit = stamp_f8((npy_uint64 *)o, mask, k, NA_OF(npy_uint64, out),
+                               0, gap, keep_numbers);
+                break;
+            case NA_KIND_F4:
+                hit = stamp_f4((npy_uint32 *)o, mask, k, NA_OF(npy_uint32, out),
+                               0, gap, keep_numbers);
+                break;
+            case NA_KIND_EXACT:
+                if (item == 8) {
+                    hit = stamp_u64((npy_uint64 *)o, mask, k,
+                                    NA_OF(npy_uint64, out), check, gap, 0);
+                }
+                else if (item == 4) {
+                    hit = stamp_u32((npy_uint32 *)o, mask, k,
+                                    NA_OF(npy_uint32, out), check, gap, 0);
+                }
+                else if (item == 2) {
+                    hit = stamp_u16((npy_uint16 *)o, mask, k,
+                                    NA_OF(npy_uint16, out), check, gap, 0);
+                }
+                else if (item == 1) {
+                    hit = stamp_u8((npy_uint8 *)o, mask, k,
+                                   NA_OF(npy_uint8, out), check, gap, 0);
+                }
+                break;
+            default:
+                break;
+        }
+        if (hit >= 0) {
+            return hit ? nullable_computed_na(out) : 0;
+        }
+    }
+    for (npy_intp i = 0; i < k; i++) {
+        char *po = o + i * stride;
+        if (mask[i]) {
+            if (!keep_numbers || nullable_out_is_nan(out, po)) {
+                nullable_put_na(out, po);
+            }
+        }
+        else if (check && nullable_is_na(out, po)) {
+            return nullable_computed_na(out);
+        }
+    }
+    return 0;
+}
+
+
+/*
+ * Integer `power` over a gap: `1 ** NA` and `NA ** 0` are 1 whatever the gap
+ * holds (see `binop_keeps_determined`).  The masked path has already put NA in
+ * every gap lane; this puts the 1 back where it is determined.
+ */
+static int
+nullable_int_power_rules(NullableDescr *in0, NullableDescr *in1,
+        NullableDescr *out, char *const data[], npy_intp const strides[],
+        npy_intp N)
+{
+    char one0[16] = {0}, zero1[16] = {0}, one_out[16] = {0};
+    if (in0->wrapped->elsize > 16 || in1->wrapped->elsize > 16
+            || out->wrapped->elsize > 16) {
+        return 0;
+    }
+    PyObject *one = PyLong_FromLong(1);
+    PyObject *zero = PyLong_FromLong(0);
+    int rc = (one && zero
+              && PyArray_Pack(in0->wrapped, one0, one) >= 0
+              && PyArray_Pack(in1->wrapped, zero1, zero) >= 0
+              && PyArray_Pack(out->wrapped, one_out, one) >= 0) ? 0 : -1;
+    Py_XDECREF(one);
+    Py_XDECREF(zero);
+    if (rc < 0) {
+        return -1;
+    }
+    for (npy_intp i = 0; i < N; i++) {
+        const char *x = data[0] + i * strides[0];
+        const char *y = data[1] + i * strides[1];
+        int xna = nullable_is_na(in0, x), yna = nullable_is_na(in1, y);
+        if ((yna && !xna && memcmp(x, one0, in0->wrapped->elsize) == 0)
+                || (xna && !yna && memcmp(y, zero1, in1->wrapped->elsize) == 0)) {
+            memcpy(data[2] + i * strides[2], one_out, out->wrapped->elsize);
+        }
+    }
+    return 0;
+}
+
+
 static int
 nullable_binary_loop_impl(int idx, PyArrayMethod_Context *context,
         char *const data[], npy_intp const dimensions[],
@@ -2204,10 +2515,10 @@ nullable_binary_loop_impl(int idx, PyArrayMethod_Context *context,
         Py_DECREF(a); Py_DECREF(b); Py_DECREF(o);
         return -1;
     }
-    char *mask = PyArray_DATA((PyArrayObject *)both);
+    unsigned char *mask = (unsigned char *)PyArray_DATA((PyArrayObject *)both);
+    int gap = nullable_scan_pair(in0, in1, data, strides, N, mask);
     for (npy_intp i = 0; i < N; i++) {
-        mask[i] = !nullable_is_na(in0, data[0] + i * strides[0])
-               && !nullable_is_na(in1, data[1] + i * strides[1]);
+        mask[i] = !mask[i];                     /* `where=`: the valid ones */
     }
 
     PyObject *args = PyTuple_Pack(2, a, b);
@@ -2216,25 +2527,37 @@ nullable_binary_loop_impl(int idx, PyArrayMethod_Context *context,
             ? PyObject_Call(binop_ufuncs[idx], args, kwargs) : NULL;
     Py_XDECREF(args); Py_XDECREF(kwargs);
     Py_DECREF(a); Py_DECREF(b); Py_DECREF(o);
-    Py_DECREF(both);
     if (res == NULL) {
+        Py_DECREF(both);
         return -1;
     }
     Py_DECREF(res);
 
     /*
-     * `where=` left the gap lanes untouched, so the output holds whatever the
-     * buffer held before -- never a reliable NaN.  The pre-scan must not run.
+     * `where=` left the gap lanes untouched, so they hold whatever the buffer
+     * held before -- never a reliable NaN, so the "keep a determined number"
+     * rule must not read them.
      */
-    return nullable_fixup(in0, in1, out, data, strides, N, 0);
+    for (npy_intp i = 0; i < N; i++) {
+        mask[i] = !mask[i];                     /* back to "is a gap" */
+    }
+    int rc = nullable_stamp(out, data[2], strides[2], N, mask, gap, 0);
+    Py_DECREF(both);
+    if (rc < 0) {
+        return -1;
+    }
+    if (binop_is_power(idx) && out->kind == NA_KIND_EXACT) {
+        return nullable_int_power_rules(in0, in1, out, data, strides, N);
+    }
+    return 0;
 }
 
 
 /*
  * The fast path: values are contiguous and aligned, so the wrapped loop runs
- * on the whole chunk at native speed, gaps included -- a gap holds a NaN or
- * INT_MIN, and computing with those is harmless and quiet.  One pass afterwards
- * stamps NA wherever an input had one.
+ * at native speed, gaps included -- a gap holds a NaN or INT_MIN, and computing
+ * with those is harmless and quiet (`binop_signals_on_gap` lists the ops for
+ * which it is not).  See `nullable_scan_pair` and `nullable_stamp`.
  */
 typedef struct {
     NpyAuxData base;
@@ -2270,15 +2593,55 @@ nullable_fast_loop(PyArrayMethod_Context *context, char *const data[],
         return nullable_accumulate_chunk(aux->idx, context, data, N, strides);
     }
 
-    npy_intp count = N;
-    if (aux->inner(aux->inner_context, data, &count, strides,
-                   aux->inner_auxdata) < 0) {
-        return -1;
+    NullableDescr *in0 = NULLABLE_DESCR(context->descriptors[0]);
+    NullableDescr *in1 = NULLABLE_DESCR(context->descriptors[1]);
+    NullableDescr *out = NULLABLE_DESCR(context->descriptors[2]);
+    int keep = binop_keeps_determined(aux->idx);
+    unsigned char mask[FIX_BLOCK];
+    /*
+     * Floats that propagate NaN can do better than scanning both inputs up
+     * front: compute, look at the output only (one stream), and scan the
+     * inputs only in a block that turned up a NaN.  That needs the inputs
+     * intact afterwards, so not when the output is one of them.
+     */
+    npy_intp item = out->base.elsize;
+    int post = (out->kind == NA_KIND_F8 || out->kind == NA_KIND_F4)
+            && in0->kind == out->kind && in1->kind == out->kind
+            && strides[2] == item
+            && !binop_swallows_nan(aux->idx)
+            && data[2] != data[0] && data[2] != data[1];
+    for (npy_intp s = 0; s < N; s += FIX_BLOCK) {
+        npy_intp k = (N - s < FIX_BLOCK) ? N - s : FIX_BLOCK;
+        char *d[3] = {data[0] + s * strides[0], data[1] + s * strides[1],
+                      data[2] + s * strides[2]};
+        npy_intp count = k;
+        if (post) {
+            if (aux->inner(aux->inner_context, d, &count, strides,
+                           aux->inner_auxdata) < 0) {
+                return -1;
+            }
+            unsigned char special = (out->kind == NA_KIND_F8)
+                    ? special_f8((npy_uint64 *)d[2], k)
+                    : special_f4((npy_uint32 *)d[2], k);
+            if (!special) {
+                continue;
+            }
+            int gap = nullable_scan_pair(in0, in1, d, strides, k, mask);
+            if (nullable_stamp(out, d[2], strides[2], k, mask, gap, keep) < 0) {
+                return -1;
+            }
+            continue;
+        }
+        int gap = nullable_scan_pair(in0, in1, d, strides, k, mask);
+        if (aux->inner(aux->inner_context, d, &count, strides,
+                       aux->inner_auxdata) < 0) {
+            return -1;
+        }
+        if (nullable_stamp(out, d[2], strides[2], k, mask, gap, keep) < 0) {
+            return -1;
+        }
     }
-    return nullable_fixup(NULLABLE_DESCR(context->descriptors[0]),
-                          NULLABLE_DESCR(context->descriptors[1]),
-                          NULLABLE_DESCR(context->descriptors[2]),
-                          data, strides, N, !binop_swallows_nan(aux->idx));
+    return 0;
 }
 
 
@@ -2297,7 +2660,17 @@ nullable_get_loop_impl(int idx, PyArrayMethod_Context *context,
         return 1;
     }
     /* these must not run over a gap at all; the masked path handles them */
-    if (binop_signals_on_gap(idx)) {
+    int k0 = NULLABLE_DESCR(context->descriptors[0])->kind;
+    int k1 = NULLABLE_DESCR(context->descriptors[1])->kind;
+    int complex_operand = k0 == NA_KIND_C8 || k0 == NA_KIND_C16
+                       || k1 == NA_KIND_C8 || k1 == NA_KIND_C16;
+    /*
+     * An integer gap is INT_MIN, which integer `power` reads as a negative
+     * exponent and refuses; it must not see the gap at all.
+     */
+    int int_power = binop_is_power(idx)
+            && (k0 == NA_KIND_EXACT || k1 == NA_KIND_EXACT);
+    if (binop_signals_on_gap(idx, complex_operand) || int_power) {
         *out_auxdata = NULL;
         return 1;
     }
@@ -2574,34 +2947,37 @@ nullable_unary_loop_impl(int idx, PyArrayMethod_Context *context,
     NullableDescr *in = NULLABLE_DESCR(context->descriptors[0]);
     NullableDescr *out = NULLABLE_DESCR(context->descriptors[1]);
 
-    PyObject *a = value_view(in->wrapped, data[0], N, strides[0]);
-    PyObject *o = value_view(out->wrapped, data[1], N, strides[1]);
-    if (a == NULL || o == NULL) {
-        Py_XDECREF(a); Py_XDECREF(o);
+    /*
+     * Record the gaps first: with `out=` the same array (`np.square(a,
+     * out=a)`) the op overwrites them, and INT_MIN squared is 0.
+     */
+    unsigned char *mask = PyMem_Malloc(N > 0 ? N : 1);
+    if (mask == NULL) {
+        PyErr_NoMemory();
         return -1;
     }
-    PyObject *args = PyTuple_Pack(1, a);
-    PyObject *kwargs = Py_BuildValue("{s:O}", "out", o);
+    char *const twice[2] = {data[0], data[0]};
+    npy_intp const twice_strides[2] = {strides[0], strides[0]};
+    int gap = nullable_scan_pair(in, in, twice, twice_strides, N, mask);
+
+    PyObject *a = value_view(in->wrapped, data[0], N, strides[0]);
+    PyObject *o = value_view(out->wrapped, data[1], N, strides[1]);
+    PyObject *args = (a && o) ? PyTuple_Pack(1, a) : NULL;
+    PyObject *kwargs = (args != NULL) ? Py_BuildValue("{s:O}", "out", o) : NULL;
     PyObject *res = (args && kwargs)
             ? PyObject_Call(unop_ufuncs[idx], args, kwargs) : NULL;
     Py_XDECREF(args); Py_XDECREF(kwargs);
-    Py_DECREF(a); Py_DECREF(o);
+    Py_XDECREF(a); Py_XDECREF(o);
     if (res == NULL) {
+        PyMem_Free(mask);
         return -1;
     }
     Py_DECREF(res);
 
     /* a unary op cannot create or remove a gap, it just carries it over */
-    int check = na_can_be_computed(out->kind);
-    for (npy_intp i = 0; i < N; i++) {
-        if (nullable_is_na(in, data[0] + i * strides[0])) {
-            nullable_put_na(out, data[1] + i * strides[1]);
-        }
-        else if (check && nullable_is_na(out, data[1] + i * strides[1])) {
-            return nullable_computed_na(out);
-        }
-    }
-    return 0;
+    int rc = nullable_stamp(out, data[1], strides[1], N, mask, gap, 0);
+    PyMem_Free(mask);
+    return rc;
 }
 
 

{nulldtype-0.1.0 → nulldtype-0.1.1}/test_basic.py

@@ -2419,3 +2419,225 @@ def test_numpy_mean_of_nullable_int_is_not_truncated():
     i = np.array([1, 2], dtype=nd.Nullable(np.int64))
     assert nd.mean(i) == 1.5          # the nd version is right
     assert np.mean(i) == 1.5          # numpy's gives 1
+
+
+@pytest.mark.parametrize("t", [np.float64, np.float32, np.int64, np.int32,
+                               np.int16, np.uint8])
+def test_block_scan_edges(t):
+    """The fixup scans in blocks of 2048 and skips blocks with no gap; a gap on
+    either side of a block edge, in the last short block, or in only one
+    operand must still come out as NA, and nothing else may."""
+    n = 2048 * 3 + 5
+    dt = nd.Nullable(t)
+    a = np.ones(n, dtype=dt)
+    b = np.ones(n, dtype=dt)
+    gaps_a = [0, 2047, 2048, 4095, n - 1]
+    gaps_b = [100, 4096, n - 3]
+    for i in gaps_a:
+        a[i] = nd.NA
+    for i in gaps_b:
+        b[i] = nd.NA
+    got = nd.isna(a + b)
+    want = np.zeros(n, dtype=bool)
+    want[gaps_a + gaps_b] = True
+    np.testing.assert_array_equal(got, want)
+    assert (nd.filled(a + b, 0)[~want] == 2).all()
+
+
+def test_block_scan_still_catches_a_computed_na_after_a_clean_block():
+    """A block without gaps still checks whether the result landed on the
+    reserved value."""
+    n = 2048 * 2 + 1
+    a = np.zeros(n, dtype=nd.Nullable(np.int32))
+    a[-1] = np.iinfo(np.int32).max
+    with pytest.raises(ValueError, match="reserved value"):
+        a + np.ones(n, dtype=nd.Nullable(np.int32))
+
+
+_SPECIALS = [0.0, -0.0, 1.0, -1.0, 2.0, 0.5, np.inf, -np.inf, np.nan]
+# ops whose answer depends on the missing value although IEEE gives a number
+_DEPENDS_ON_GAP = {"fmax", "fmin", "copysign"}
+
+
+@pytest.mark.parametrize("t", [np.float64, np.float32, np.float16,
+                               np.complex128, np.complex64])
+@pytest.mark.parametrize("n", [1, 5000])
+def test_every_binop_keeps_a_gap(t, n):
+    """A gap is a value nobody knows.  Where the result would differ for some
+    value of it, the result is NA; where IEEE 754 already says it cannot --
+    `1 ** x == x ** 0 == 1`, `hypot(inf, x) == inf`, `heaviside(y != 0, x)` --
+    the number stands, as in R and pandas.  So the expected answer is IEEE's
+    with a NaN in place of the gap, NA wherever that is NaN, and NA always for
+    `fmax`, `fmin` and `copysign`, which do look at the missing value.
+    Complex operands conservatively always give NA.
+
+    Every element involves a gap, so any floating-point warning is a spurious
+    one (`binop_signals_on_gap`); complex division, power and ordering used to
+    raise "invalid value" this way."""
+    dt = nd.Nullable(t)
+    is_complex = np.dtype(t).kind == "c"
+    gap = np.empty(n, dtype=dt)
+    gap[...] = nd.NA
+    nan = np.full(n, np.nan, dtype=t)
+    wrong = []
+    for name in _c.binop_names:
+        f = getattr(np, name)
+        for v in _SPECIALS:
+            plain = np.full(n, v, dtype=t)
+            other = plain.astype(dt)
+            for x, y, px, py, side in ((other, gap, plain, nan, "right"),
+                                       (gap, other, nan, plain, "left")):
+                try:
+                    with warnings.catch_warnings():
+                        warnings.simplefilter("error", RuntimeWarning)
+                        r = f(x, y)
+                except RuntimeWarning as w:
+                    wrong.append((name, v, side, str(w)))
+                    continue
+                except (TypeError, ValueError):
+                    continue            # op not defined for this dtype
+                if not nd.is_nullable(r.dtype):
+                    continue
+                with np.errstate(all="ignore"):
+                    ref = np.asarray(f(px, py))
+                if ref.dtype.kind in "fc" and not is_complex \
+                        and name not in _DEPENDS_ON_GAP:
+                    expect_na = np.isnan(ref)
+                else:
+                    expect_na = np.ones(n, dtype=bool)
+                got_na = nd.isna(r)
+                if not (got_na == expect_na).all():
+                    wrong.append((name, v, side, "NA where", got_na[0], "expected", expect_na[0]))
+                    continue
+                if (~expect_na).any():
+                    kept = nd.to_numpy(r[~expect_na])
+                    if not (kept == ref[~expect_na]).all():
+                        wrong.append((name, v, side, "value", kept[0], ref[~expect_na][0]))
+    assert not wrong
+
+
+@pytest.mark.parametrize("t", [np.int64, np.int32, np.int8, np.uint8, np.uint64])
+@pytest.mark.parametrize("n", [1, 5000])
+def test_integer_power_over_a_gap(t, n):
+    """An integer gap is INT_MIN (or UINT_MAX); integer `power` used to read it
+    as a negative exponent and raise.  Now it never sees the gap, and the two
+    determined cases give 1 as for floats.  A real negative exponent still
+    raises, as for a plain array."""
+    dt = nd.Nullable(t)
+    base = np.array([1, 2, 0, 0, 3], dtype=t).astype(dt)
+    expo = np.array([0, 0, 0, 2, 0], dtype=t).astype(dt)
+    base[2] = base[3] = nd.NA
+    expo[0] = expo[1] = nd.NA
+    base, expo = np.tile(base, n), np.tile(expo, n)
+    got = np.power(base, expo)
+    assert nd.isna(got).tolist() == [False, True, False, True, False] * n
+    assert got[0] == 1 and got[2] == 1 and got[4] == 1
+
+    if np.dtype(t).kind == "i":
+        with pytest.raises(ValueError, match="negative integer powers"):
+            np.power(np.array([2], dtype=t).astype(dt),
+                     np.array([-1], dtype=t).astype(dt))
+
+
+_INPLACE_TYPES = [np.int8, np.int32, np.int64, np.uint8, np.uint64,
+                  np.float16, np.float32, np.float64, np.complex128]
+
+
+@pytest.mark.parametrize("t", _INPLACE_TYPES)
+@pytest.mark.parametrize("n", [3, 5000])
+def test_inplace_ops_keep_their_gaps(t, n):
+    """With `out=` one of the inputs, the wrapped loop overwrites the gap
+    before anything looks at it: `a += 1` turned an int32 gap into
+    -2147483647, `np.square(a, out=a)` into 0.  Gaps are now recorded before
+    the loop runs."""
+    dt = nd.Nullable(t)
+
+    def with_gaps():
+        a = np.full(n, 3, dtype=t).astype(dt)
+        a[1] = a[n - 1] = nd.NA
+        return a
+
+    gaps = np.zeros(n, dtype=bool)
+    gaps[[1, n - 1]] = True
+    ones = np.ones(n, dtype=t).astype(dt)
+
+    with warnings.catch_warnings():
+        warnings.simplefilter("ignore", RuntimeWarning)   # e.g. 3 ** 3 in int8
+        for name in ("add", "subtract", "multiply", "maximum"):
+            a = with_gaps()
+            getattr(np, name)(a, ones, out=a)
+            np.testing.assert_array_equal(nd.isna(a), gaps, err_msg=name)
+            b = with_gaps()
+            getattr(np, name)(ones, b, out=b)
+            np.testing.assert_array_equal(nd.isna(b), gaps, err_msg=name)
+            c = with_gaps()
+            getattr(np, name)(c, c, out=c)
+            np.testing.assert_array_equal(nd.isna(c), gaps, err_msg=name)
+        if np.dtype(t).kind in "fc":
+            # x86 keeps the left NaN's payload, so writing `nan + NA` over the
+            # right operand replaces the gap with an ordinary NaN
+            nan = np.full(n, np.nan, dtype=t).astype(dt)
+            b = with_gaps()
+            np.add(nan, b, out=b)
+            np.testing.assert_array_equal(nd.isna(b), gaps, err_msg="nan + NA")
+        for name in ("square", "negative", "absolute", "invert", "sign",
+                     "sqrt", "reciprocal"):
+            a = with_gaps()
+            try:
+                getattr(np, name)(a, out=a)
+            except (TypeError, ValueError):
+                continue            # not defined for this dtype, or unsafe out
+            np.testing.assert_array_equal(nd.isna(a), gaps, err_msg=name)
+
+
+@pytest.mark.parametrize("t", [np.int8, np.int32, np.int64, np.uint8, np.uint64])
+@pytest.mark.parametrize("n", [1, 5000])
+def test_every_binop_keeps_a_gap_integers(t, n):
+    """The integer side of `test_every_binop_keeps_a_gap`: a gap is INT_MIN or
+    UINT_MAX, so `NA // -1` overflowed and `NA // 0` divided by zero -- warnings
+    about a value that is not there.  Every result is NA except the two
+    determined powers, and a real zero divisor still warns."""
+    dt = nd.Nullable(t)
+    gap = np.empty(n, dtype=dt)
+    gap[...] = nd.NA
+    values = [0, 1, 2, 3] + ([-1, -2] if np.dtype(t).kind == "i" else [])
+    wrong = []
+    for name in _c.binop_names:
+        f = getattr(np, name)
+        for v in values:
+            other = np.full(n, v, dtype=t).astype(dt)
+            for x, y, side in ((other, gap, "right"), (gap, other, "left")):
+                try:
+                    with warnings.catch_warnings():
+                        warnings.simplefilter("error", RuntimeWarning)
+                        r = f(x, y)
+                except RuntimeWarning as w:
+                    wrong.append((name, v, side, str(w)))
+                    continue
+                except (TypeError, ValueError):
+                    continue
+                if not nd.is_nullable(r.dtype):
+                    continue
+                if r.dtype.wrapped.kind == "f":
+                    # computed in floats (`heaviside`, `float_power`, ...):
+                    # IEEE decides, as in `test_every_binop_keeps_a_gap`
+                    fv = np.full(n, v, dtype=np.float64)
+                    nan = np.full(n, np.nan)
+                    with np.errstate(all="ignore"):
+                        ref = f(fv, nan) if side == "right" else f(nan, fv)
+                    expect_na = np.isnan(ref) | (name in _DEPENDS_ON_GAP)
+                    if not (nd.isna(r) == expect_na).all():
+                        wrong.append((name, v, side, "float rule"))
+                    continue
+                determined = name == "power" and (
+                    (side == "right" and v == 1) or (side == "left" and v == 0))
+                if determined:
+                    if nd.isna(r).any() or not (nd.filled(r, 0) == 1).all():
+                        wrong.append((name, v, side, "expected 1"))
+                elif not nd.isna(r).all():
+                    wrong.append((name, v, side, "expected NA"))
+    assert not wrong
+
+    x = np.ones(3, dtype=dt)
+    with pytest.warns(RuntimeWarning, match="divide by zero"):
+        np.floor_divide(x, np.zeros(3, dtype=dt))

nulldtype-0.1.0/setup.py

@@ -1,11 +0,0 @@
-"""The C extension; everything else is in pyproject.toml."""
-import numpy as np
-from setuptools import Extension, setup
-
-setup(ext_modules=[
-    Extension(
-        "_nulldtype",
-        sources=["src/nulldtype.c"],
-        include_dirs=[np.get_include()],
-    ),
-])