labelarray 0.1.0__tar.gz → 0.1.2__tar.gz

labelarray-0.1.2/CHANGELOG.md

@@ -0,0 +1,58 @@
+# Changelog
+
+## 0.1.2 — 2026-10-26
+
+- Float32 sums and means of data held in chunks (Dask) can be correctly
+  rounded too, the same bits as `sums.reduce` gives in memory whatever the
+  chunking. `sums.chunk_estimate` gives one chunk's float64 totals, sums of
+  magnitudes, counts (values, values present, +inf, -inf), and the term count
+  of its error bound, which combine across chunks in any order; `settle`
+  certifies the combined estimate; `chunk_exact` gives one chunk's exact
+  sums for the groups the bound could not settle, as int64 buckets per
+  float32 exponent, which add across chunks; `exact_results` rounds them;
+  `from_chunks` drives it all given a function returning the exact sums.
+  Chunks take the NumPy estimate, since they already run in parallel.
+- Exact sums are measured from each group's own lowest nonzero exponent, so
+  a group whose values span few exponents fits in 64-bit integers whatever
+  its magnitude; a group like `[2**100, 1, -2**100]` no longer goes through
+  Python fractions. Results are unchanged, bit for bit.
+
+## 0.1.1 — 2026-10-26
+
+- Grouped float32 sums and means over many small groups are much faster:
+  200,000 values in 96,846 groups (voxels holding one to a few points) now
+  take 6 ms instead of 111. The exact mean of a few float32 values often
+  lies exactly halfway between two float32 values, where the error bound
+  cannot decide the rounding; about one group in ten went to an exact
+  path computed in Python fractions. Groups whose values span a narrow
+  range of exponents now have their exact sums computed as 64-bit integers
+  and rounded with NumPy (integer division kept to 27 or 28 bits with the
+  last bit set when anything is left over, then one rounding to float32),
+  so the Python path is left for groups spanning a wide range. Results are
+  unchanged, bit for bit: the tests compare them with exact rational
+  arithmetic, and the new code gave the same bits as 0.1.0 on two million
+  results.
+
+## 0.1.0 — 2026-10-25
+
+The first release on its own. labelarray was developed as the array layer
+of tidy3a (`tidy3a.labelarray`) and leaves it unchanged in behaviour:
+
+- `Array`: a NumPy array with named axes, exact labels, attributes, and a
+  name. Selection by positions and labels (`isel`, `sel`), `transpose`,
+  `rename`, reductions (`reduce`), and contraction (`dot`). NumPy's ufuncs,
+  `np.where`, and Python's operators broadcast by axis name under the exact
+  alignment rule.
+- `Collection`: arrays sharing axes, one label array per axis, with
+  selection, assignment, joins (`join`), nearest-label matching
+  (`nearest`), and conversion to and from xarray.
+- The exact label rule (`labels`) and label indexes: sorted, sort-once,
+  and grid indexes, chosen by `index_for`, with `register_index` for other
+  label types.
+- Grouping: `Grouping`, `factorize`, the key transforms `Bins`,
+  `Duration`, `Round`, and `GridCells`, and grouped reductions
+  (`grouped.reduce`, `counts`, `broadcast`) bit-identical to reducing each
+  group on its own.
+- Float32 sums and means correctly rounded in any memory layout (`sums`),
+  with an optional compiled path (`labelarray[fast]`) giving the same bits.
+- One family of errors under `LabelArrayError`.

{labelarray-0.1.0/src/labelarray.egg-info → labelarray-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: labelarray
-Version: 0.1.0
+Version: 0.1.2
 Summary: Labelled N-D arrays in pure NumPy, with exact label semantics and reproducible reductions
 Author: Rigoberto Leyva Salmeron
 License-Expression: MIT
@@ -42,11 +42,13 @@ It is deliberately small. What it offers that other array libraries do not:
 2. **Reproducible reductions.** Float32 sums and means are correctly
    rounded: the float32 nearest the exact result, the same in any memory
    layout. Grouped reductions are bit-identical to reducing each group on
-   its own.
+   its own, and data held in chunks (Dask) gets the same bits whatever the
+   chunking: each chunk's estimate combines in any order, and exact sums
+   settle the few results an error bound cannot.
 3. **Collections.** Arrays sharing axes, with one label array per axis,
    joined and aligned under those rules.
 
-**Status:** version `0.1.0`, the first release on its own; the API may still
+**Status:** version `0.1.2`; the API may still
 change before 1.0. What changed in each release is in
 [CHANGELOG.md](CHANGELOG.md).
 

{labelarray-0.1.0 → labelarray-0.1.2}/README.md

@@ -14,11 +14,13 @@ It is deliberately small. What it offers that other array libraries do not:
 2. **Reproducible reductions.** Float32 sums and means are correctly
    rounded: the float32 nearest the exact result, the same in any memory
    layout. Grouped reductions are bit-identical to reducing each group on
-   its own.
+   its own, and data held in chunks (Dask) gets the same bits whatever the
+   chunking: each chunk's estimate combines in any order, and exact sums
+   settle the few results an error bound cannot.
 3. **Collections.** Arrays sharing axes, with one label array per axis,
    joined and aligned under those rules.
 
-**Status:** version `0.1.0`, the first release on its own; the API may still
+**Status:** version `0.1.2`; the API may still
 change before 1.0. What changed in each release is in
 [CHANGELOG.md](CHANGELOG.md).
 

labelarray-0.1.2/src/labelarray/__version__.py

@@ -0,0 +1 @@
+__version__ = "0.1.2"

{labelarray-0.1.0 → labelarray-0.1.2}/src/labelarray/sums.py

@@ -19,6 +19,15 @@ Two ways, chosen by how many values each result adds:
   a bound cannot decide), so cancellation among those few values is rounded
   in float64 as it goes: ``[2**53, 1, -2**53]`` sums to 0 here, not 1.
 
+Data held in chunks (Dask) gets the same bits as data in memory, whatever the
+chunking: ``chunk_estimate`` gives each chunk's float64 totals, sums of
+magnitudes, counts, and the term count of its error bound, which combine
+across chunks in any order; ``settle`` certifies the combined estimate;
+``chunk_exact`` gives each chunk's exact sums, as integer buckets per
+exponent, for the groups the bound could not settle, which add across chunks;
+and ``from_chunks`` drives the two passes. A correctly rounded result is
+unique, so these give the bits ``reduce`` gives.
+
 Missing values: without ``na_rm`` a NaN makes its result NaN; with it, NaN
 is skipped and a mean divides by the values present. +inf and -inf together
 give NaN. A sum of no values is +0.0 and a mean NaN; a zero sum is +0.0, as
@@ -79,10 +88,8 @@ def reduce(values: np.ndarray, op: str, axes: Sequence[int], *, na_rm: bool = Fa
                     result = np.where(present >= min_count, result, np.float32(np.nan))
                 return result.reshape(kept_shape)
             return _in_order(values, axes, kept_shape, op, na_rm, min_count)
-        table = np.transpose(values, axes + kept).reshape(positions, math.prod(kept_shape))
+        table, rows, others = _table(values, axes, kept, codes)
         if codes is not None:
-            others = math.prod(values.shape[a] for a in axes[1:])
-            rows = np.asarray(codes) if others == 1 else np.repeat(np.asarray(codes), others)
             if sizes is not None and others != 1:
                 sizes = np.asarray(sizes) * others
             result = _settled(table, rows, groups, sizes, op, na_rm, min_count)
@@ -90,6 +97,18 @@ def reduce(values: np.ndarray, op: str, axes: Sequence[int], *, na_rm: bool = Fa
         return _settled(table, None, 1, None, op, na_rm, min_count).reshape(kept_shape)
 
 
+def _table(values, axes, kept, codes):
+    """*values* as (positions along *axes*, positions along *kept*), each position's group code, and positions per code."""
+    positions = math.prod(values.shape[a] for a in axes)
+    width = math.prod(values.shape[a] for a in kept)
+    table = np.transpose(values, axes + kept).reshape(positions, width)
+    if codes is None:
+        return table, None, 1
+    others = math.prod(values.shape[a] for a in axes[1:])
+    rows = np.asarray(codes) if others == 1 else np.repeat(np.asarray(codes), others)
+    return table, rows, others
+
+
 def _in_order(values, axes, kept_shape, op, na_rm, min_count) -> np.ndarray:
     """Few values per result: float64, added first to last along *axes*, rounded once.
 
@@ -138,7 +157,7 @@ def _settled(table, rows, groups, sizes, op, na_rm, min_count) -> np.ndarray:
     return result
 
 
-def _estimate(table, rows, groups, sizes, na_rm):
+def _estimate(table, rows, groups, sizes, na_rm, compiled=True):
     """Float64 sums, sums of magnitudes, values present, values, and terms for the bound, per group and column.
 
     The NumPy path adds chunks of positions (one ``np.bincount`` over
@@ -164,7 +183,7 @@ def _estimate(table, rows, groups, sizes, na_rm):
     if sizes is None:
         sizes = np.bincount(rows, minlength=groups)
     sizes = np.asarray(sizes)
-    kernel = _compiled("grouped_estimate") if COMPILED and table.size >= COMPILED_FROM else None
+    kernel = _compiled("grouped_estimate") if compiled and COMPILED and table.size >= COMPILED_FROM else None
     if kernel is not None:
         total, magnitude, present = kernel(table, np.ascontiguousarray(rows, dtype=np.intp), groups, na_rm)
         values = np.repeat(sizes[:, None], width, axis=1)
@@ -214,7 +233,14 @@ def _certify(total, magnitude, terms, counts, op, compensated):
 
 
 def _exact(values, rows, chosen, groups, op, na_rm) -> np.ndarray:
-    """Correctly rounded results for groups *chosen*, from exact sums of their values."""
+    """Correctly rounded results for groups *chosen*, from exact sums of their values.
+
+    Most such groups are small and hold values of nearby magnitudes (their
+    exact mean lies on a float32 tie, where the bound cannot decide): their
+    exact sums fit in 64-bit integers and are rounded with NumPy
+    (``_rounded``). Groups spanning a wider range of exponents are summed and
+    rounded in Python integers and fractions.
+    """
     if rows is None:
         part, codes, count = values, np.zeros(len(values), dtype=np.intp), 1
     else:
@@ -223,27 +249,41 @@ def _exact(values, rows, chosen, groups, op, na_rm) -> np.ndarray:
         codes = remap[rows]
         keep = codes >= 0
         part, codes, count = values[keep], codes[keep], len(chosen)
-    totals, present, special = _exact_sums(part, codes, count, na_rm)
-    out = np.empty(count, dtype=np.float32)
-    for k in range(count):
-        if special[k] is not None:
-            out[k] = special[k]
-        elif op == "sum":
-            out[k] = _round32(totals[k])
-        else:
-            out[k] = _round32(totals[k] / present[k]) if present[k] else np.float32(np.nan)
+    buckets, low, present, nan, plus, minus = _exact_sums(part, codes, count)
+    return _exact_finish(buckets, low, present, nan, plus, minus, op, na_rm)
+
+
+def _exact_finish(buckets, low, present, nan, plus, minus, op, na_rm) -> np.ndarray:
+    """Correctly rounded results from exact sums: *buckets* ``(count, width)`` per group and exponent from *low*."""
+    count = len(buckets)
+    out = np.full(count, np.nan, dtype=np.float32)
+    undefined = (nan & (not na_rm)) | (plus & minus)
+    infinite = (plus | minus) & ~undefined
+    out[infinite] = np.where(plus[infinite], np.float32(np.inf), np.float32(-np.inf))
+    regular = ~(undefined | infinite)
     if op == "mean":
-        out[(present == 0) & np.array([s is None for s in special])] = np.nan
+        regular &= present > 0  # a mean of no values stays NaN
+    divisor = present if op == "mean" else np.ones(count, dtype=np.int64)
+    whole, first, fits = _whole(buckets)
+    fast = regular & fits & (divisor < 2**35)  # so the integer quotient below stays within int64
+    if fast.any():
+        out[fast] = _rounded(whole[fast], low + first[fast] - 150, divisor[fast])
+    for k in np.flatnonzero(regular & ~fits):
+        row = buckets[k]
+        total = sum(int(row[j]) << int(j) for j in np.flatnonzero(row)) * Fraction(2) ** (low - 150)
+        out[k] = _round32(total / int(divisor[k]))
     return out
 
 
-def _exact_sums(values, codes, count, na_rm):
-    """Exact sums as Fractions, values present, and NaN or infinity where they decide the result.
+def _exact_sums(values, codes, count):
+    """Exact sums as integer buckets per group and exponent, the lowest exponent, values present, and specials.
 
     A float32 is M * 2**(e - 150) with M a signed integer below 2**24 and e
     from 1 to 254. Integers M with the same e add exactly in float64 for
     fewer than 2**29 values, so they are summed per group and exponent with
-    ``np.bincount``, and the few totals per group combined in Python integers.
+    ``np.bincount``: group k's exact sum is the sum over j of
+    ``buckets[k, j] * 2**(low + j - 150)``. Also, per group, the values
+    present (not NaN), and whether a NaN, +inf, or -inf is among them.
     """
     bits = values.view(np.uint32)
     exponent = ((bits >> 23) & 0xFF).astype(np.intp)
@@ -257,30 +297,77 @@ def _exact_sums(values, codes, count, na_rm):
     has_nan = np.bincount(codes, weights=nan, minlength=count) > 0
     plus = np.bincount(codes, weights=values == np.inf, minlength=count) > 0
     minus = np.bincount(codes, weights=values == -np.inf, minlength=count) > 0
-    totals = [Fraction(0)] * count
+    low, width = 1, 1
     if finite.any():
         low, high = int(exponent[finite].min()), int(exponent[finite].max())
         width = high - low + 1
+    buckets = np.zeros(count * width, dtype=np.int64)
+    if finite.any():
         key = codes[finite] * width + (exponent[finite] - low)
         weights = mantissa[finite].astype(np.float64)
-        buckets = np.zeros(count * width, dtype=np.int64)
         chunk = 1 << 28  # each bucket sum stays below 2**52, so float64 holds it exactly
         for start in range(0, len(key), chunk):
             buckets += np.bincount(key[start:start + chunk], weights=weights[start:start + chunk],
                                    minlength=count * width).astype(np.int64)
-        buckets = buckets.reshape(count, width)
-        scale = Fraction(2) ** (low - 150)
-        for k in range(count):
-            row = buckets[k]
-            whole = sum(int(row[j]) << int(j) for j in np.flatnonzero(row))  # Python integers: shifts beyond 63
-            totals[k] = whole * scale
-    special = [None] * count
-    for k in range(count):
-        if (has_nan[k] and not na_rm) or (plus[k] and minus[k]):
-            special[k] = np.float32(np.nan)
-        elif plus[k] or minus[k]:
-            special[k] = np.float32(np.inf if plus[k] else -np.inf)
-    return totals, present, special
+    return buckets.reshape(count, width), low, present, has_nan, plus, minus
+
+
+def _whole(buckets: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+    """Each group's exact sum as one integer, in units of its lowest nonzero bucket; that bucket; where the integer is below 2**52.
+
+    Where it is, the shifted buckets and their sum stay within int64 and the
+    sum is exact in float64 too; other groups need Python integers. Measured
+    from each group's own lowest bucket, a group of any magnitude fits when
+    its values span few exponents.
+    """
+    count, width = buckets.shape
+    nonzero = buckets != 0
+    first = np.where(nonzero.any(axis=1), nonzero.argmax(axis=1), 0)
+    place = first[:, None] + np.arange(width)
+    aligned = np.where(place < width, np.take_along_axis(buckets, np.minimum(place, width - 1), axis=1), 0)
+    shifts = np.arange(width)
+    bound = (np.abs(aligned).astype(np.float64) * np.ldexp(1.0, shifts)).sum(axis=1)  # exact terms; rounding is small
+    fits = bound < 2.0**52
+    whole = np.zeros(count, dtype=np.int64)
+    if fits.any():
+        # Shifts of zero buckets beyond 62 bits are clamped; a nonzero bucket that far up does not fit.
+        whole[fits] = np.left_shift(aligned[fits], np.minimum(shifts, 62)).sum(axis=1)
+    return whole, first, fits
+
+
+def _rounded(whole: np.ndarray, scale, divisor: np.ndarray) -> np.ndarray:
+    """``whole * 2**scale / divisor`` correctly rounded to float32, ties to even, for |whole| below 2**52.
+
+    The quotient is taken in integers to 27 or 28 significant bits, with its
+    last bit set when anything was left over ("round to odd"); with at least
+    two bits more than float32 keeps, the one rounding from float64 to
+    float32 then rounds as the exact quotient would, subnormals included.
+    """
+    sign = np.where(whole < 0, -1.0, 1.0)
+    size = np.abs(whole)
+    divisor = divisor.astype(np.int64)
+    zero = size == 0
+    size = np.where(zero, 1, size)  # placeholders; zero sums are +0.0, as NumPy gives them
+    size_bits = np.frexp(size.astype(np.float64))[1].astype(np.int64)  # bit lengths, exact below 2**53
+    divisor_bits = np.frexp(divisor.astype(np.float64))[1].astype(np.int64)
+    up = 27 + divisor_bits - size_bits  # shift so the quotient has 27 or 28 bits
+    quotient = np.empty(len(size), dtype=np.int64)
+    rest = np.empty(len(size), dtype=bool)
+    left = up >= 0
+    if left.any():  # size << up has 27 or 28 more bits than the divisor: below 2**63 for divisors below 2**35
+        numerator = np.left_shift(size[left], up[left])
+        quotient[left], remainder = np.divmod(numerator, divisor[left])
+        rest[left] = remainder != 0
+    right = ~left
+    if right.any():
+        whole_part, remainder = np.divmod(size[right], divisor[right])
+        down = -up[right]
+        quotient[right] = np.right_shift(whole_part, down)
+        rest[right] = (remainder != 0) | ((whole_part & ((np.int64(1) << down) - 1)) != 0)
+    quotient |= rest.astype(np.int64)
+    result = np.ldexp(sign * quotient.astype(np.float64), scale - up).astype(np.float32)
+    result[zero] = np.float32(0.0)
+    return result
 
 
 def _round32(value: Fraction) -> np.float32:
@@ -300,6 +387,151 @@ def _round32(value: Fraction) -> np.float32:
     return np.float32(best)
 
 
+# ── Chunked data: pieces that combine in any order ──────────────────────────
+
+PIECES = 7
+TOTAL, MAGNITUDE, PRESENT, VALUES, PLUS, MINUS, TERMS = range(PIECES)
+BUCKETS = 254 + 4  # one per float32 exponent, then values present, NaN, +inf, and -inf
+EXACT_PAIRS = 1 << 14  # (group, column) pairs per exact pass: 33 MB of buckets per chunk
+
+
+def chunk_estimate(values: np.ndarray, axes: Sequence[int], *, na_rm: bool = False,
+                   codes: np.ndarray | None = None, groups: int = 0) -> np.ndarray:
+    """One chunk's estimate of its float32 sums over *axes*: ``(PIECES, groups, *kept)`` in float64.
+
+    The pieces, per group (one without *codes*) and kept position: the
+    float64 total, the sum of magnitudes, the values present (not NaN), the
+    values, the +inf and the -inf among them, and the term count of the
+    total's error bound. Chunks combine in any order: every piece adds,
+    except ``TERMS``, whose maximum is taken (``settle`` adds the number of
+    chunks to it: Higham's lemma 3.3, where a float64 sum of totals each
+    within gamma(n_i) of exact is within gamma(max n_i + k) of exact). The
+    integers stay exact in float64 below 2**53 values. The NumPy path
+    estimates every chunk: chunks run in parallel already, and the compiled
+    kernel's own threads cannot be entered from several at once.
+    """
+    values = np.asarray(values)
+    if values.dtype != np.float32:
+        raise TypeError(f"float32 values expected, got {values.dtype}")
+    axes = tuple(int(a) for a in axes)
+    kept = tuple(a for a in range(values.ndim) if a not in axes)
+    kept_shape = tuple(values.shape[a] for a in kept)
+    count = groups if codes is not None else 1
+    with np.errstate(all="ignore"):
+        table, rows, _ = _table(values, axes, kept, codes)
+        total, magnitude, present, counted, terms, _ = _estimate(table, rows, count, None, na_rm, compiled=False)
+        width = table.shape[1]
+        plus = np.zeros((count, width))
+        minus = np.zeros((count, width))
+        if np.isinf(table).any():
+            key = ((np.zeros(len(table), dtype=np.intp) if rows is None else rows)[:, None] * width
+                   + np.arange(width)).ravel()
+            plus = np.bincount(key, weights=(table == np.inf).ravel(), minlength=count * width).reshape(count, width)
+            minus = np.bincount(key, weights=(table == -np.inf).ravel(), minlength=count * width).reshape(count, width)
+    pieces = np.stack([total, magnitude, present, counted, plus, minus, terms]).astype(np.float64)
+    return pieces.reshape((PIECES, count, *kept_shape))
+
+
+def settle(pieces: np.ndarray, chunks: int, op: str, na_rm: bool) -> tuple[np.ndarray, np.ndarray]:
+    """Results the combined *pieces* of *chunks* chunks settle, and where they do not: ``(result, unsure)``.
+
+    Where the error bound proves the rounding, the result is the correctly
+    rounded one; NaN, +inf and -inf follow the missing-value rules from the
+    counts; a mean of no values is NaN. The rest, ``unsure``, need exact sums.
+    """
+    with np.errstate(all="ignore"):
+        total, magnitude, present, counted, plus, minus, terms = pieces
+        counts = present if na_rm else counted
+        result, sure = _certify(total, magnitude, terms + chunks, counts, op, False)
+        undefined = ((counted - present > 0) & (not na_rm)) | ((plus > 0) & (minus > 0))
+        infinite = ((plus > 0) | (minus > 0)) & ~undefined
+        result[undefined] = np.nan
+        result[infinite] = np.where(plus[infinite] > 0, np.float32(np.inf), np.float32(-np.inf))
+        sure |= undefined | infinite
+        if op == "mean":
+            empty = (counts == 0) & ~undefined
+            result[empty] = np.nan
+            sure |= empty
+    return result, ~sure
+
+
+def chunk_exact(values: np.ndarray, axes: Sequence[int], chosen: Sequence[int], *,
+                codes: np.ndarray | None = None, groups: int = 0) -> np.ndarray:
+    """One chunk's exact sums over *axes* for the groups *chosen*: ``(len(chosen), *kept, BUCKETS)`` in int64.
+
+    Per chosen group and kept position: the exact sum as integer buckets, one
+    per float32 exponent (``_exact_sums``), then the values present and the
+    NaN, +inf and -inf among them. Chunks add. Without *codes*, *chosen* is
+    ``[0]``. A bucket stays within int64 below 2**39 values of one exponent.
+    """
+    values = np.asarray(values)
+    if values.dtype != np.float32:
+        raise TypeError(f"float32 values expected, got {values.dtype}")
+    axes = tuple(int(a) for a in axes)
+    kept = tuple(a for a in range(values.ndim) if a not in axes)
+    kept_shape = tuple(values.shape[a] for a in kept)
+    chosen = np.asarray(chosen, dtype=np.intp)
+    count = len(chosen)
+    with np.errstate(all="ignore"):
+        table, rows, _ = _table(values, axes, kept, codes)
+        positions, width = table.shape
+        if rows is None:
+            part, pair_codes = table, np.zeros(positions, dtype=np.intp)
+        else:
+            remap = np.full(max(groups, 1), -1, dtype=np.intp)
+            remap[chosen] = np.arange(count)
+            pair_codes = remap[rows]
+            keep = pair_codes >= 0
+            part, pair_codes = table[keep], pair_codes[keep]
+        pairs = (pair_codes[:, None] * width + np.arange(width)).ravel()
+        buckets, low, present, nan, plus, minus = _exact_sums(np.ascontiguousarray(part).ravel(), pairs, count * width)
+        out = np.zeros((count * width, BUCKETS), dtype=np.int64)
+        out[:, low - 1:low - 1 + buckets.shape[1]] = buckets
+        out[:, 254] = present
+        out[:, 255] = nan
+        out[:, 256] = plus
+        out[:, 257] = minus
+    return out.reshape((count, *kept_shape, BUCKETS))
+
+
+def exact_results(buckets: np.ndarray, op: str, na_rm: bool) -> np.ndarray:
+    """Correctly rounded results from exact sums added across chunks: ``(..., BUCKETS)`` in, float32 out."""
+    buckets = np.asarray(buckets)
+    shape = buckets.shape[:-1]
+    flat = buckets.reshape(-1, BUCKETS)
+    with np.errstate(all="ignore"):
+        out = _exact_finish(flat[:, :254], 1, flat[:, 254], flat[:, 255] > 0, flat[:, 256] > 0, flat[:, 257] > 0,
+                            op, na_rm)
+    return out.reshape(shape)
+
+
+def from_chunks(pieces: np.ndarray, chunks: int, op: str, exact, *, na_rm: bool = False,
+                min_count: int | None = None) -> np.ndarray:
+    """*op* from the combined *pieces* of *chunks* chunks, with exact sums where the bound cannot settle a result.
+
+    *exact(chosen)* returns the chunks' exact sums for the groups *chosen*
+    (``chunk_exact``, added across chunks); it is called once per batch of
+    ``EXACT_PAIRS`` (group, kept position) pairs, and not at all where the
+    bound settles everything. The result is ``(groups, *kept)`` in float32,
+    the same bits as ``reduce`` gives in memory.
+    """
+    pieces = np.asarray(pieces, dtype=np.float64)
+    result, unsure = settle(pieces, chunks, op, na_rm)
+    count = len(result)
+    width = result.size // count if count else 0
+    rows = np.flatnonzero(unsure.reshape(count, width).any(axis=1)) if count else np.zeros(0, dtype=np.intp)
+    batch = max(1, EXACT_PAIRS // max(width, 1))
+    for start in range(0, len(rows), batch):
+        chosen = rows[start:start + batch]
+        done = exact_results(exact(chosen), op, na_rm)
+        picked, mask = result[chosen], unsure[chosen]
+        picked[mask] = done[mask]
+        result[chosen] = picked
+    if op == "sum" and min_count is not None:
+        result = np.where(pieces[PRESENT] >= min_count, result, np.float32(np.nan))
+    return result
+
+
 # ── The optional compiled kernel ────────────────────────────────────────────
 
 _KERNELS: dict = {}

{labelarray-0.1.0 → labelarray-0.1.2/src/labelarray.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: labelarray
-Version: 0.1.0
+Version: 0.1.2
 Summary: Labelled N-D arrays in pure NumPy, with exact label semantics and reproducible reductions
 Author: Rigoberto Leyva Salmeron
 License-Expression: MIT
@@ -42,11 +42,13 @@ It is deliberately small. What it offers that other array libraries do not:
 2. **Reproducible reductions.** Float32 sums and means are correctly
    rounded: the float32 nearest the exact result, the same in any memory
    layout. Grouped reductions are bit-identical to reducing each group on
-   its own.
+   its own, and data held in chunks (Dask) gets the same bits whatever the
+   chunking: each chunk's estimate combines in any order, and exact sums
+   settle the few results an error bound cannot.
 3. **Collections.** Arrays sharing axes, with one label array per axis,
    joined and aligned under those rules.
 
-**Status:** version `0.1.0`, the first release on its own; the API may still
+**Status:** version `0.1.2`; the API may still
 change before 1.0. What changed in each release is in
 [CHANGELOG.md](CHANGELOG.md).
 

{labelarray-0.1.0 → labelarray-0.1.2}/tests/test_sums.py

@@ -61,9 +61,14 @@ def expected(values: np.ndarray, op: str, na_rm: bool) -> np.float32:
 
 
 def same_bits(a, b) -> bool:
-    a, b = np.asarray(a, F32), np.asarray(b, F32)
+    return not differing(a, b).size
+
+
+def differing(a, b) -> np.ndarray:
+    """Where *a* and *b* differ in their bits, NaN counting as equal to NaN."""
+    a, b = np.atleast_1d(np.asarray(a, F32)), np.atleast_1d(np.asarray(b, F32))
     both_nan = np.isnan(a) & np.isnan(b)
-    return bool(np.all(both_nan | (a.view(np.uint32) == b.view(np.uint32))))
+    return np.flatnonzero(~(both_nan | (a.view(np.uint32) == b.view(np.uint32))))
 
 
 def corpus():
@@ -233,3 +238,200 @@ def test_compiled_kernels_run_only_from_a_size_where_they_pay(monkeypatch):
     sums.reduce(many, "sum", (0,))
     sums.reduce(many[0, :sums.COMPILED_FROM], "mean", (0,), codes=rng.integers(0, 5, sums.COMPILED_FROM), groups=5)
     assert calls == ["in_order", "grouped_estimate"]
+
+
+# ── Many tiny groups, where exact means and sums often lie on float32 ties ──
+
+
+def tiny_groups(seed, count=3000, spread=(0.0, 2.4)):
+    """Groups of 1 to 4 float32 values, many of whose exact sums and means are float32 ties."""
+    rng = np.random.default_rng(seed)
+    sizes = rng.integers(1, 5, count)
+    keys = np.repeat(np.arange(count), sizes)
+    values = rng.uniform(0, 255, len(keys)) * 10.0 ** rng.uniform(*spread, len(keys)) * rng.choice([-1, 1], len(keys))
+    values = values.astype(F32)
+    order = rng.permutation(len(values))
+    return values[order], keys[order]
+
+
+def _reduced(values, keys, count, op, na_rm=False, min_count=None):
+    return sums.reduce(values[:, None], op, (0,), codes=keys, groups=count, na_rm=na_rm, min_count=min_count)[:, 0]
+
+
+@pytest.mark.parametrize("seed", range(6))
+@pytest.mark.parametrize("op", ["sum", "mean"])
+def test_tiny_groups_on_ties_are_correctly_rounded(seed, op):
+    values, keys = tiny_groups(seed)
+    count = int(keys.max()) + 1
+    got = _reduced(values, keys, count, op)
+    groups = [values[keys == k] for k in range(count)]
+    want = np.array([expected(g, op, False) for g in groups], F32)
+    wrong = differing(got, want)
+    assert not wrong.size, [(k, groups[k], got[k], want[k]) for k in wrong[:5]]
+
+
+@pytest.mark.parametrize("na_rm", [False, True])
+def test_tiny_groups_with_subnormals_zeros_and_missing_values(na_rm):
+    rng = np.random.default_rng(7)
+    groups = []
+    for _ in range(2000):
+        size = int(rng.integers(1, 5))
+        kind = rng.integers(0, 6)
+        if kind == 0:
+            group = rng.integers(-2**23, 2**23, size) * 2.0**-149  # subnormals, and ties between them
+        elif kind == 1:
+            group = rng.choice([0.0, -0.0], size)
+        elif kind == 2:
+            group = np.where(rng.random(size) < 0.4, np.nan, rng.uniform(0, 255, size))
+        elif kind == 3:
+            group = rng.uniform(1, 2, size) * rng.choice([1, 2**-126, 2**100], size)  # narrow and wide spreads
+        elif kind == 4:
+            group = rng.choice([np.inf, -np.inf, 1.0], size)
+        else:
+            group = rng.uniform(0, 255, size)
+        groups.append(np.asarray(group, F32))
+    values = np.concatenate(groups)
+    keys = np.repeat(np.arange(len(groups)), [len(g) for g in groups])
+    for op in ("sum", "mean"):
+        got = _reduced(values, keys, len(groups), op, na_rm=na_rm)
+        want = np.array([expected(g, op, na_rm) for g in groups], F32)
+        wrong = differing(got, want)
+        assert not wrong.size, [(op, k, groups[k], got[k], want[k]) for k in wrong[:5]]
+
+
+def test_narrow_groups_are_decided_without_fractions(monkeypatch):
+    values, keys = tiny_groups(11, count=5000)
+    count = int(keys.max()) + 1
+    monkeypatch.setattr(sums, "_round32", lambda value: pytest.fail("a narrow group was decided with fractions"))
+    for op in ("sum", "mean"):
+        _reduced(values, keys, count, op)
+
+
+def test_groups_spanning_wide_exponents_still_add_exactly(monkeypatch):
+    big = 2.0**100
+    groups = [[big, 1.0, -big], [1e30, 3.0, -1e30, 1.0], [2.0**-140, 2.0**120, -(2.0**120)],
+              [big, 1.0, -big, 2.0**-100], [1e30, 3.0, -1e30, 1.0, 2.0**-60]]  # what is left after cancelling spans wide exponents
+    values = np.concatenate([np.asarray(g, F32) for g in groups])
+    keys = np.repeat(np.arange(len(groups)), [len(g) for g in groups])
+    calls = []
+    original = sums._round32
+    monkeypatch.setattr(sums, "_round32", lambda value: calls.append(value) or original(value))
+    for op in ("sum", "mean"):
+        got = _reduced(values, keys, len(groups), op)
+        assert same_bits(got, np.array([expected(np.asarray(g, F32), op, False) for g in groups], F32))
+    assert calls  # these need the exact path in Python integers
+
+
+# ── Chunked data: the same bits whatever the chunking ───────────────────────
+
+
+def _chunked(values, keys, count, op, bounds, na_rm=False, min_count=None, calls=None):
+    """``sums.from_chunks`` over *values* cut at *bounds*, the chunks combined in a shuffled order."""
+    parts = [slice(a, b) for a, b in zip(bounds[:-1], bounds[1:])]
+    order = np.random.default_rng(len(bounds)).permutation(len(parts))
+    pieces = None
+    for i in order:
+        part = parts[i]
+        codes = None if keys is None else keys[part]
+        piece = sums.chunk_estimate(values[part], (0,), na_rm=na_rm, codes=codes, groups=count)
+        if pieces is None:
+            pieces = piece
+        else:
+            pieces[:sums.TERMS] += piece[:sums.TERMS]
+            pieces[sums.TERMS] = np.maximum(pieces[sums.TERMS], piece[sums.TERMS])
+
+    def exact(chosen):
+        if calls is not None:
+            calls.append(list(chosen))
+        codes = (lambda part: None) if keys is None else (lambda part: keys[part])
+        return sum(sums.chunk_exact(values[parts[i]], (0,), chosen, codes=codes(parts[i]), groups=count)
+                   for i in order)
+
+    return sums.from_chunks(pieces, len(parts), op, exact, na_rm=na_rm, min_count=min_count)
+
+
+def _bounds(n, chunks, seed=0):
+    """*chunks* uneven cuts of ``range(n)``, some possibly empty."""
+    cuts = np.sort(np.random.default_rng(seed).integers(0, n + 1, chunks - 1))
+    return [0, *cuts.tolist(), n]
+
+
+@pytest.mark.parametrize("chunks", [1, 3, 17, 400])
+@pytest.mark.parametrize("na_rm", [False, True])
+def test_chunked_grouped_sums_give_the_in_memory_bits_for_any_chunking(chunks, na_rm):
+    values, keys, groups = corpus()
+    count = len(groups)
+    for op in ("sum", "mean"):
+        want = _reduced(values, keys, count, op, na_rm=na_rm)
+        got = _chunked(values[:, None], keys, count, op, _bounds(len(values), chunks, seed=chunks), na_rm=na_rm)[:, 0]
+        wrong = differing(got, want)
+        assert not wrong.size, [(op, chunks, k, groups[k], got[k], want[k]) for k in wrong[:5]]
+        assert same_bits(want, np.array([expected(g, op, na_rm) for g in groups], F32))
+
+
+@pytest.mark.parametrize("seed", range(3))
+def test_chunk_boundaries_inside_tiny_groups_keep_the_ties_right(seed):
+    values, keys = tiny_groups(seed)
+    count = int(keys.max()) + 1
+    bounds = _bounds(len(values), 9, seed=seed)
+    for op in ("sum", "mean"):
+        calls = []
+        got = _chunked(values[:, None], keys, count, op, bounds, calls=calls)[:, 0]
+        want = np.array([expected(values[keys == k], op, False) for k in range(count)], F32)
+        assert same_bits(got, want)
+        assert calls and all(len(c) <= sums.EXACT_PAIRS for c in calls)  # ties need the exact pass
+
+
+def test_chunked_ungrouped_sums_keep_the_other_axes():
+    rng = np.random.default_rng(5)
+    values = (rng.normal(0, 1e4, (5000, 3)) * rng.choice([1, 2.0**-30, 2.0**30], (5000, 3))).astype(F32)
+    values[::7, 1] = values[1::7, 1] + np.spacing(values[1::7, 1]) / 2  # pairs on float32 ties
+    for op in ("sum", "mean"):
+        want = sums.reduce(values, op, (0,))
+        for chunks in (1, 4, 50):
+            got = _chunked(values, None, 1, op, _bounds(len(values), chunks, seed=chunks))[0]
+            assert same_bits(got, want), (op, chunks)
+        for j in range(3):
+            assert same_bits(want[j], expected(values[:, j], op, False))
+
+
+@pytest.mark.parametrize("na_rm", [False, True])
+def test_missing_and_infinite_values_settle_without_an_exact_pass(na_rm):
+    groups = [[1.0, np.nan, 2.0], [np.inf, 1.0], [-np.inf, 5.0], [np.inf, -np.inf], [np.nan, np.nan],
+              [np.inf, np.nan], [3e38, 3e38, 3e38], [1.0, 2.0, 3.0]]
+    values = np.concatenate([np.asarray(g, F32) for g in groups])
+    keys = np.repeat(np.arange(len(groups)), [len(g) for g in groups])
+    for op in ("sum", "mean"):
+        calls = []
+        got = _chunked(values[:, None], keys, len(groups), op, [0, 2, 5, 9, 12, 20], na_rm=na_rm, calls=calls)[:, 0]
+        want = np.array([expected(np.asarray(g, F32), op, na_rm) for g in groups], F32)
+        assert same_bits(got, want), (op, got, want)
+        assert calls == []
+
+
+def test_min_count_and_empty_chunks_across_chunks():
+    values = np.asarray([1.0, np.nan, 2.0, np.nan, np.nan, 4.0], F32)
+    keys = np.asarray([0, 0, 0, 1, 1, 1])
+    for min_count, want in ((2, [3.0, np.nan]), (1, [3.0, 4.0])):
+        got = _chunked(values[:, None], keys, 2, "sum", [0, 0, 2, 2, 6, 6], na_rm=True, min_count=min_count)[:, 0]
+        assert same_bits(got, np.asarray(want, F32))
+    assert same_bits(_chunked(values[:, None], keys, 2, "sum", [0, 6], na_rm=False)[:, 0], np.asarray([np.nan, np.nan], F32))
+
+
+@pytest.mark.parametrize("compiled", [False, True])
+def test_chunks_take_the_numpy_path_and_match_the_compiled_in_memory_bits(compiled, monkeypatch):
+    pytest.importorskip("numba")
+    monkeypatch.setattr(sums, "COMPILED", compiled)
+    rng = np.random.default_rng(3)
+    values = (rng.normal(0, 1e3, 400_000)).astype(F32)
+    keys = rng.integers(0, 32, len(values))
+    want = _reduced(values, keys, 32, "sum")  # compiled when installed and enabled: 400,000 values
+    assert same_bits(want, np.array([expected(values[keys == k], "sum", False) for k in range(32)], F32))
+    original = sums._compiled
+    monkeypatch.setattr(sums, "_compiled", lambda name: pytest.fail(f"a chunk used the compiled kernel {name}"))
+    bounds = [0, 150_000, 300_000, 400_000]  # chunks big enough for it in memory; chunks run in parallel instead
+    for op in ("sum", "mean"):
+        got = _chunked(values[:, None], keys, 32, op, bounds)[:, 0]
+        monkeypatch.setattr(sums, "_compiled", original)
+        assert same_bits(got, _reduced(values, keys, 32, op))
+        monkeypatch.setattr(sums, "_compiled", lambda name: pytest.fail(f"a chunk used the compiled kernel {name}"))

labelarray-0.1.0/CHANGELOG.md

@@ -1,25 +0,0 @@
-# Changelog
-
-## 0.1.0 — 2026-10-25
-
-The first release on its own. labelarray was developed as the array layer
-of tidy3a (`tidy3a.labelarray`) and leaves it unchanged in behaviour:
-
-- `Array`: a NumPy array with named axes, exact labels, attributes, and a
-  name. Selection by positions and labels (`isel`, `sel`), `transpose`,
-  `rename`, reductions (`reduce`), and contraction (`dot`). NumPy's ufuncs,
-  `np.where`, and Python's operators broadcast by axis name under the exact
-  alignment rule.
-- `Collection`: arrays sharing axes, one label array per axis, with
-  selection, assignment, joins (`join`), nearest-label matching
-  (`nearest`), and conversion to and from xarray.
-- The exact label rule (`labels`) and label indexes: sorted, sort-once,
-  and grid indexes, chosen by `index_for`, with `register_index` for other
-  label types.
-- Grouping: `Grouping`, `factorize`, the key transforms `Bins`,
-  `Duration`, `Round`, and `GridCells`, and grouped reductions
-  (`grouped.reduce`, `counts`, `broadcast`) bit-identical to reducing each
-  group on its own.
-- Float32 sums and means correctly rounded in any memory layout (`sums`),
-  with an optional compiled path (`labelarray[fast]`) giving the same bits.
-- One family of errors under `LabelArrayError`.

labelarray-0.1.0/src/labelarray/__version__.py

@@ -1 +0,0 @@
-__version__ = "0.1.0"