usearch 2.26.0 → 2.26.2

package/numkong/include/numkong/reduce/README.md

@@ -31,36 +31,39 @@ def minmax(a: np.ndarray) -> tuple[float, int, float, int]:
 
 ## Input & Output Types
 
+The Output Type column lists the accumulator type of `nk_reduce_moments_*`.
+`nk_reduce_minmax_*` instead reports extrema in the input type — `i4`, `u4`, and `u1` unpack to `i8`, `u8`, and `u8` — alongside two `nk_size_t` indices.
+
 Float reductions:
 
-| Input Type | Output Type | Description                           |
-| ---------- | ----------- | ------------------------------------- |
-| `f64`      | `f64`       | 64-bit double precision               |
-| `f32`      | `f32`       | 32-bit single precision               |
-| `f16`      | `f32`       | 16-bit half precision, widened output |
-| `bf16`     | `f32`       | 16-bit brain float, widened output    |
+| Input Type | Output Type | Description                             |
+| ---------- | ----------- | --------------------------------------- |
+| `f64`      | `f64`       | 64-bit double precision                 |
+| `f32`      | `f64`       | 32-bit single precision, widened output |
+| `bf16`     | `f32`       | 16-bit brain float, widened output      |
+| `f16`      | `f32`       | 16-bit half precision, widened output   |
 
 Mini-float reductions:
 
 | Input Type | Output Type | Description                                  |
 | ---------- | ----------- | -------------------------------------------- |
-| `e4m3`     | `f32`       | 8-bit Float8: 4 exponent, 3 mantissa bits    |
 | `e5m2`     | `f32`       | 8-bit Float8: 5 exponent, 2 mantissa bits    |
-| `e2m3`     | `f32`       | 8-bit MX format: 2 exponent, 3 mantissa bits |
+| `e4m3`     | `f32`       | 8-bit Float8: 4 exponent, 3 mantissa bits    |
 | `e3m2`     | `f32`       | 8-bit MX format: 3 exponent, 2 mantissa bits |
+| `e2m3`     | `f32`       | 8-bit MX format: 2 exponent, 3 mantissa bits |
 
 Integer reductions:
 
 | Input Type | Output Type | Description                        |
 | ---------- | ----------- | ---------------------------------- |
-| `i8`       | `i64`       | 8-bit signed, widened to 64-bit    |
-| `u8`       | `u64`       | 8-bit unsigned, widened to 64-bit  |
-| `i16`      | `i64`       | 16-bit signed, widened to 64-bit   |
-| `u16`      | `u64`       | 16-bit unsigned, widened to 64-bit |
-| `i32`      | `i64`       | 32-bit signed, widened to 64-bit   |
-| `u32`      | `u64`       | 32-bit unsigned, widened to 64-bit |
 | `i64`      | `i64`       | 64-bit signed                      |
+| `i32`      | `i64`       | 32-bit signed, widened to 64-bit   |
+| `i16`      | `i64`       | 16-bit signed, widened to 64-bit   |
+| `i8`       | `i64`       | 8-bit signed, widened to 64-bit    |
 | `u64`      | `u64`       | 64-bit unsigned                    |
+| `u32`      | `u64`       | 32-bit unsigned, widened to 64-bit |
+| `u16`      | `u64`       | 16-bit unsigned, widened to 64-bit |
+| `u8`       | `u64`       | 8-bit unsigned, widened to 64-bit  |
 
 Sub-byte reductions:
 
@@ -82,29 +85,32 @@ RVV uses native strided loads (`__riscv_vlse32_v_f32m1`) that accept arbitrary b
 
 ### Kahan-Neumaier Compensated Summation
 
-`nk_reduce_moments_f32_serial`, `nk_reduce_moments_f32_haswell` use Neumaier's variant of Kahan summation — maintaining a running compensation term that captures rounding errors.
+`nk_reduce_moments_f32_serial`, `nk_reduce_moments_f64_serial` use Neumaier's variant of Kahan summation — maintaining a running compensation term that captures rounding errors.
 Standard pairwise summation accumulates $O(\sqrt{n})$ ULP error for n elements; Neumaier compensation bounds error to $O(1)$ ULP regardless of vector length.
 The serial path uses Neumaier's adaptive branch: `if (abs(sum) >= abs(val))` selects the larger summand first, minimizing relative error in the compensation term.
-SIMD backends (`nk_reduce_moments_f32_haswell`) carry 8 independent compensation lanes in a YMM register — computing `round_error = tentative - sum; correction = (sum - (tentative - round_error)) + (val - round_error)` without branches, folding all lanes into a single scalar correction at the end.
+SIMD backends (`nk_reduce_moments_f64_haswell`, `nk_reduce_moments_f64_skylake`) carry 4 or 8 independent compensation lanes in a YMM or ZMM register — computing `round_error = tentative - sum; correction = (sum - (tentative - round_error)) + (val - round_error)` without branches, folding all lanes into a single scalar correction at the end.
+The Float32 SIMD kernels skip compensation: they widen every input to Float64 first, and the wider accumulator already absorbs the rounding error.
 
 ### Fused Moments in a Single Pass
 
-`nk_reduce_moments_f32_haswell`, `nk_reduce_moments_f64_skylake` compute sum and sum-of-squares simultaneously — one load feeds both a `VADDPS` (sum accumulator) and a `VFMADD231PS` (square accumulator).
+`nk_reduce_moments_f32_haswell`, `nk_reduce_moments_f64_skylake` compute sum and sum-of-squares simultaneously — one load feeds both a sum accumulator and a square accumulator.
+On Haswell the Float32 inputs are widened with `VCVTPS2PD`, then folded in with `VADDPD` and `VFMADD231PD`; on Skylake the Float64 path squares with `VMULPD` so the product can go through the compensated add.
 Two accumulators share the same loaded data, halving memory bandwidth compared to separate sum + norm passes.
 The squared-norm $\|a\|^2 = \sum a_i^2$ is a self-dot-product, reused by packing routines (`nk_dots_pack_f32_haswell`) to precompute per-vector norms during layout transformation.
-For Float16/BFloat16/Float8 inputs, all backends widen to Float32 before accumulation — NEON FHM (`nk_reduce_moments_e4m3_neonfhm`) converts e4m3->f16 via lookup, then uses `vfmlalq_low_f16` to fuse the Float16 → Float32 widening with the FMA into the Float32 accumulator.
+For Float16/BFloat16/Float8 inputs, all backends widen to Float32 before accumulation — NEON FHM (`nk_reduce_moments_e4m3_neonfhm`) converts e4m3->f16 by bit manipulation, then uses `vfmlalq_low_f16` and `vfmlalq_high_f16` to fuse the Float16 → Float32 widening with the FMA into the Float32 accumulator.
 
 ### Integer Saturation in Sum-of-Squares
 
 Integer moments accumulate sums in the widest available type: Int8/UInt8/Int16 inputs produce Int64/UInt64 outputs.
 Sums use widening addition chains — NEON uses pairwise widening (`vpaddlq_s16` -> UInt32 -> UInt64 stages); Haswell biases Int8 inputs with 0x80 and uses unsigned SAD (`_mm256_sad_epu8`) for the sum, correcting by subtracting $128 \times \text{count}$ at the end.
-Sum-of-squares can overflow UInt64 when squaring large Int32 values — backends use explicit saturating multiply: checks if `abs(val) < 2^32` (square fits in UInt64), otherwise saturates to I64_MAX.
+Sum-of-squares can overflow UInt64 when squaring Int64/UInt64 inputs — only `nk_reduce_moments_i64_*` and `nk_reduce_moments_u64_*` need an explicit saturating multiply: it checks whether `abs(val) < 2^32` (so the square fits in UInt64) and otherwise saturates to I64_MAX for signed inputs, U64_MAX for unsigned ones.
+Narrower inputs cannot overflow the product itself, so they only need the saturating accumulate.
 Haswell emulates UInt64 saturating add via XOR-based unsigned comparison: flip sign bits to convert unsigned overflow detection into a signed comparison, then OR with the overflow mask to produce all-ones on saturation.
 
 ### Recursive Blocking for Counter Overflow
 
 All SIMD backends use loop iteration counters narrower than `nk_size_t` to save register pressure — UInt8 for Int8 minmax lanes, UInt16 for Float32 moments lanes.
-When `count` exceeds the counter's range x lane count (e.g., Haswell Float32: $256 \times 8 = 2048$ elements for UInt8 counters), the reduction splits recursively: process the left half, process the right half, combine results with saturating arithmetic.
+When `count` exceeds the counter's range x lane count (e.g., Haswell Float32 moments: $65536 \times 8 = 524288$ elements for UInt16 counters), the reduction splits recursively: process the left half, process the right half, combine results with saturating arithmetic.
 Block caps vary by backend and element width: Haswell Int8 minmax uses UInt8 loop counters (cap = $256 \times 32 = 8192$); Skylake Float32 moments uses UInt16 counters (cap = $65536 \times 16 = 1048576$).
 The recursive split is invisible to the caller — the public API accepts arbitrary `count` values; internal dispatch chooses between single-pass and recursive based on the cap.
 
@@ -118,7 +124,7 @@ NEON uses same-width counters (u8x16 for i8x16 minmax), limiting block size to $
 ### NaN-Aware Extrema Tracking
 
 `nk_reduce_minmax_f32_haswell`, `nk_reduce_minmax_f64_skylake` use IEEE ordered-quiet comparisons (`_CMP_LT_OQ`, `_CMP_GT_OQ`) — returning false when either operand is NaN, so NaN inputs never replace the running extremum.
-Tail elements beyond the vector-aligned portion are masked by loading into a NaN-filled register via `_mm256_mask_loadu_ps(nan_vec, mask, ptr)` — NaN tails cannot win any comparison, eliminating out-of-bounds artifacts.
+Tail elements beyond the vector-aligned portion are blended into a quiet-NaN register — `nk_reduce_minmax_f64_haswell` partial-loads the tail and `_mm256_blendv_pd`-s it over a NaN vector, so the padding lanes cannot win any comparison; Skylake instead predicates the tail comparison with the same `__mmask` used for the load, and `nk_reduce_minmax_f32_haswell` finishes its tail in scalar code.
 If all inputs are NaN, the sentinels remain (min = F32_MAX, max = F32_MIN) and indices are set to NK_SIZE_MAX, signaling no valid extremum.
 The final horizontal reduction across lanes uses pairwise `VSHUFPS` + `VMINPS` chains — 3 shuffles for a 256-bit register, $O(\log_2 w)$ for width $w$.
 

package/numkong/include/numkong/reduce/genoa.h

@@ -22,7 +22,8 @@
 #if NK_TARGET_GENOA
 
 #include "numkong/reduce/serial.h"
-#include "numkong/cast/icelake.h" // `nk_e4m3x32_to_bf16x32_icelake_` etc.
+#include "numkong/cast/icelake.h"   // `nk_e4m3x32_to_bf16x32_icelake_` etc.
+#include "numkong/reduce/skylake.h" // `nk_reduce_add_f32x16_skylake_`
 
 #if defined(__cplusplus)
 extern "C" {

package/numkong/include/numkong/reduce/icelake.h

@@ -19,6 +19,7 @@
 #if NK_TARGET_ICELAKE
 
 #include "numkong/reduce/serial.h"
+#include "numkong/reduce/skylake.h" // `nk_reduce_add_i64x8_skylake_`, `nk_reduce_add_u64x8_skylake_`, `nk_stride_mask_b16x32_`
 
 #if defined(__cplusplus)
 extern "C" {

package/numkong/include/numkong/reduce/neon.h

@@ -2218,7 +2218,7 @@ NK_INTERNAL void nk_reduce_moments_i64_neon_contiguous_( //
     nk_i64_t *sum_ptr, nk_u64_t *sumsq_ptr) {
     uint64x2_t sum_low_u64x2 = vdupq_n_u64(0);
     int64x2_t sum_high_i64x2 = vdupq_n_s64(0);
-    // NEON can still load/extract i64 vectors for sumsq via scalar nk_i64_smul_
+    // NEON can still load/extract i64 vectors for sumsq via nk_i64_smul_sq_i64x2_neon_
     uint64x2_t sumsq_u64x2 = vdupq_n_u64(0);
     int sumsq_overflow = 0;
     int64x2_t sign_bit_i64x2 = vdupq_n_s64((nk_i64_t)0x8000000000000000ULL);

package/numkong/include/numkong/reduce/serial.h

@@ -21,15 +21,6 @@
 extern "C" {
 #endif
 
-/*  Keep the serial instantiations below actually scalar, regardless of build type.
- *  See dots/serial.h for rationale. */
-#if defined(__clang__)
-#pragma clang attribute push(__attribute__((noinline)), apply_to = function)
-#elif defined(__GNUC__)
-#pragma GCC push_options
-#pragma GCC optimize("no-tree-vectorize", "no-tree-slp-vectorize", "no-ipa-cp-clone", "no-inline")
-#endif
-
 NK_INTERNAL nk_f64_t nk_reduce_sum_f64_serial_(nk_f64_t const *values, nk_f64_t const *compensations, int count) {
     nk_f64_t running_sum = 0, accumulated_error = 0;
     for (int i = 0; i < count; i++) {
@@ -47,6 +38,15 @@ NK_INTERNAL nk_f64_t nk_reduce_sum_f64_serial_(nk_f64_t const *values, nk_f64_t
     return running_sum + accumulated_error;
 }
 
+/*  Keep the serial instantiations below actually scalar, regardless of build type.
+ *  See dots/serial.h for rationale. */
+#if defined(__clang__)
+#pragma clang attribute push(__attribute__((noinline)), apply_to = function)
+#elif defined(__GNUC__)
+#pragma GCC push_options
+#pragma GCC optimize("no-tree-vectorize", "no-tree-slp-vectorize", "no-ipa-cp-clone", "no-inline")
+#endif
+
 NK_PUBLIC void nk_reduce_moments_f32_serial(                       //
     nk_f32_t const *data, nk_size_t count, nk_size_t stride_bytes, //
     nk_f64_t *sum_ptr, nk_f64_t *sumsq_ptr) {

package/numkong/include/numkong/reduce/skylake.h

@@ -953,7 +953,7 @@ NK_PUBLIC void nk_reduce_moments_f64_skylake(                          //
 NK_INTERNAL void nk_reduce_moments_i8_skylake_contiguous_( //
     nk_i8_t const *data_ptr, nk_size_t count,              //
     nk_i64_t *sum_ptr, nk_u64_t *sumsq_ptr) {
-    // Sum: VPSADBW with XOR bias (same as nk_reduce_add_i8_skylake_contiguous_).
+    // Sum: VPSADBW with XOR bias.
     // Sumsq: widen i8→i16, VPMADDWD(x,x) → i32 (pairs of squares), accumulate i32.
     // i32 overflow safe: max per lane = (128² + 128²) * 65536 iters ≈ 2.1B = safe limit.
     // The dispatch recurses at (NK_U16_MAX+1)*64 elements → at most 65536 iterations here.
@@ -1147,7 +1147,7 @@ NK_PUBLIC void nk_reduce_minmax_i8_skylake(                           //
 NK_INTERNAL void nk_reduce_moments_u8_skylake_contiguous_( //
     nk_u8_t const *data_ptr, nk_size_t count,              //
     nk_u64_t *sum_ptr, nk_u64_t *sumsq_ptr) {
-    // Sum: VPSADBW directly (same as nk_reduce_add_u8_skylake_contiguous_).
+    // Sum: VPSADBW directly.
     // Sumsq: widen u8→i16, VPMADDWD(x,x) → i32 (pairs of squares), accumulate i32.
     // i32 overflow safe: max per lane = (255² + 255²) * 1024 iters ≈ 133M < 2.1B.
     __m512i zero_u8x64 = _mm512_setzero_si512();
@@ -1697,7 +1697,7 @@ NK_PUBLIC void nk_reduce_minmax_u16_skylake(                           //
                                     max_index_ptr);
 }
 
-/** @brief Unsigned saturating add of two i64x8 vectors (3 uops). */
+/** @brief Unsigned saturating add of two u64x8 vectors (3 uops). */
 NK_INTERNAL __m512i nk_u64_sadd_epi64_skylake_(__m512i a, __m512i b) {
     __m512i result_u64x8 = _mm512_add_epi64(a, b);
     __mmask8 ovf = _mm512_cmp_epu64_mask(result_u64x8, a, _MM_CMPINT_LT);
@@ -3371,7 +3371,7 @@ NK_PUBLIC void nk_reduce_moments_u4_skylake(                            //
 NK_INTERNAL void nk_reduce_moments_u1_skylake_contiguous_( //
     nk_u1x8_t const *data_ptr, nk_size_t count,            //
     nk_u64_t *sum_ptr, nk_u64_t *sumsq_ptr) {
-    // Sum = popcount via 4-bit LUT (same as nk_reduce_add_u1_skylake). Sumsq = sum for bits.
+    // Sum = popcount via 4-bit LUT. Sumsq = sum for bits.
     __m512i lut_i8x64 = _mm512_set_epi8(                //
         4, 3, 3, 2, 3, 2, 2, 1, 3, 2, 2, 1, 2, 1, 1, 0, //
         4, 3, 3, 2, 3, 2, 2, 1, 3, 2, 2, 1, 2, 1, 1, 0, //

package/numkong/include/numkong/reduce.hpp

@@ -629,6 +629,44 @@ std::size_t argmax(vector_view<value_type_> input) noexcept {
     return minmax(input).max_index;
 }
 
+/** @brief Population count of a packed bit tensor — number of set bits.
+ *  Wraps `nk_reduce_moments_u1` via `sum`; returns 0 for empty inputs. */
+template <std::size_t max_rank_ = 8>
+u64_t popcount(tensor_view<u1x8_t, max_rank_> input) noexcept {
+    return sum<u1x8_t, max_rank_>(input);
+}
+
+/** @brief True if any bit of the packed bit tensor is set. Suffixed `_set` to
+ *  avoid colliding with the `all_t`/`all` slice marker in `vector.hpp`. */
+template <std::size_t max_rank_ = 8>
+bool any_set(tensor_view<u1x8_t, max_rank_> input) noexcept {
+    return popcount<max_rank_>(input).raw_ != 0;
+}
+
+/** @brief True if no bit of the packed bit tensor is set. */
+template <std::size_t max_rank_ = 8>
+bool none_set(tensor_view<u1x8_t, max_rank_> input) noexcept {
+    return !any_set<max_rank_>(input);
+}
+
+/** @brief True if every bit of the packed bit tensor is set. */
+template <std::size_t max_rank_ = 8>
+bool all_set(tensor_view<u1x8_t, max_rank_> input) noexcept {
+    return popcount<max_rank_>(input).raw_ == input.numel();
+}
+
+/** @brief Population count over a 1D bit-vector view. */
+inline u64_t popcount(vector_view<u1x8_t> input) noexcept { return sum<u1x8_t>(input); }
+
+/** @brief True if any bit of the 1D bit-vector view is set. */
+inline bool any_set(vector_view<u1x8_t> input) noexcept { return popcount(input).raw_ != 0; }
+
+/** @brief True if no bit of the 1D bit-vector view is set. */
+inline bool none_set(vector_view<u1x8_t> input) noexcept { return !any_set(input); }
+
+/** @brief True if every bit of the 1D bit-vector view is set. */
+inline bool all_set(vector_view<u1x8_t> input) noexcept { return popcount(input).raw_ == input.size(); }
+
 #pragma endregion Scalar Reductions
 
 #pragma region Axis Reductions

package/numkong/include/numkong/scalar/README.md

@@ -2,7 +2,7 @@
 
 NumKong provides single-element math operations — square root, reciprocal square root, fused multiply-add, and saturating integer arithmetic — with per-ISA implementations.
 These primitives serve as building blocks for vectorized kernels: distance finalizers call `nk_f32_rsqrt` for angular normalization, packing routines call `nk_f32_sqrt` for norm computation.
-Ordering functions (`nk_f16_order`, `nk_bf16_order`, `nk_e4m3_order`) convert floating-point bit patterns to integers that sort in the same order as the original floats.
+Ordering functions (`nk_f16_order`, `nk_bf16_order`, `nk_e4m3_order`) compare two values through their bit patterns, returning a negative, zero, or positive `int` like `strcmp`.
 
 Reciprocal square root:
 
@@ -40,27 +40,27 @@ def saturating_add(a: int, b: int, bits: int, signed: bool) -> int:
 
 ## Input & Output Types
 
-| Input Type | Output Type | Description                                     |
-| ---------- | ----------- | ----------------------------------------------- |
-| `f64`      | `f64`       | sqrt, rsqrt, fma for 64-bit doubles             |
-| `f32`      | `f32`       | sqrt, rsqrt, fma for 32-bit floats              |
-| `f16`      | `f16`       | sqrt, rsqrt, fma for 16-bit halfs               |
-| `i8`       | `i8`        | Saturating add and multiply                     |
-| `u8`       | `u8`        | Saturating add and multiply                     |
-| `i16`      | `i16`       | Saturating add and multiply                     |
-| `u16`      | `u16`       | Saturating add and multiply                     |
-| `i32`      | `i32`       | Saturating add and multiply                     |
-| `u32`      | `u32`       | Saturating add and multiply                     |
-| `i64`      | `i64`       | Saturating add and multiply                     |
-| `u64`      | `u64`       | Saturating add and multiply                     |
-| `i4x2`     | `i4x2`      | Saturating add for packed signed nibble pairs   |
-| `u4x2`     | `u4x2`      | Saturating add for packed unsigned nibble pairs |
-| `f16`      | `u16`       | Ordering: bit pattern to sortable integer       |
-| `bf16`     | `u16`       | Ordering: bit pattern to sortable integer       |
-| `e4m3`     | `u8`        | Ordering: bit pattern to sortable integer       |
-| `e5m2`     | `u8`        | Ordering: bit pattern to sortable integer       |
-| `e2m3`     | `u8`        | Ordering: bit pattern to sortable integer       |
-| `e3m2`     | `u8`        | Ordering: bit pattern to sortable integer       |
+| Input Type | Output Type | Description                                                  |
+| ---------- | ----------- | ------------------------------------------------------------ |
+| `f64`      | `f64`       | sqrt, rsqrt, fma for 64-bit doubles                          |
+| `f32`      | `f32`       | sqrt, rsqrt, fma for 32-bit floats                           |
+| `f16`      | `f16`       | sqrt, rsqrt, fma for 16-bit halfs                            |
+| `i8`       | `i8`        | Saturating add and multiply                                  |
+| `u8`       | `u8`        | Saturating add and multiply                                  |
+| `i16`      | `i16`       | Saturating add and multiply                                  |
+| `u16`      | `u16`       | Saturating add and multiply                                  |
+| `i32`      | `i32`       | Saturating add and multiply                                  |
+| `u32`      | `u32`       | Saturating add and multiply                                  |
+| `i64`      | `i64`       | Saturating add and multiply                                  |
+| `u64`      | `u64`       | Saturating add and multiply                                  |
+| `i4x2`     | `i4x2`      | Saturating add and multiply for packed signed nibble pairs   |
+| `u4x2`     | `u4x2`      | Saturating add and multiply for packed unsigned nibble pairs |
+| `f16`      | `int`       | Ordering: three-way comparison of two values                 |
+| `bf16`     | `int`       | Ordering: three-way comparison of two values                 |
+| `e4m3`     | `int`       | Ordering: three-way comparison of two values                 |
+| `e5m2`     | `int`       | Ordering: three-way comparison of two values                 |
+| `e2m3`     | `int`       | Ordering: three-way comparison of two values                 |
+| `e3m2`     | `int`       | Ordering: three-way comparison of two values                 |
 
 ## Optimizations
 
@@ -68,20 +68,21 @@ def saturating_add(a: int, b: int, bits: int, signed: bool) -> int:
 
 `nk_f32_rsqrt_serial` uses the classic bit-manipulation trick: reinterpret Float32 bits as Int32, compute `0x5F375A86 - (bits >> 1)`, reinterpret back to Float32, then refine with 3 Newton-Raphson iterations reaching ~34.9 correct bits.
 Each Newton-Raphson iteration: `y = y * (1.5f - 0.5f * x * y * y)` — 2 multiplies and 1 subtract, ~4cy per iteration.
-`nk_f32_rsqrt_haswell` replaces this with hardware `VRSQRT14PS` ($2^{-14}$ relative error, ~4cy latency) plus one Newton-Raphson refinement (~22-24 correct bits).
+`nk_f32_rsqrt_haswell` replaces this with hardware `RSQRTSS` ($2^{-14}$ relative error, ~4cy latency) plus one Newton-Raphson refinement (~22-24 correct bits).
 `nk_f64_rsqrt_serial` uses the Float64 magic constant `0x5FE6EB50C7B537A9` with 4 iterations for 52-bit mantissa coverage.
 
 ### Dekker Error-Free Multiplication for FMA
 
-`nk_f32_fma_serial` emulates fused multiply-add on platforms without hardware FMA using Dekker's algorithm: splits each operand into high and low halves via `a_hi = (a * 134217729.0f) - ((a * 134217729.0f) - a)`, then computes the exact product error term.
-The magic constant $134217729 = 2^{27} + 1$ splits a 24-bit mantissa into two 12-bit halves that multiply without rounding.
+`nk_f32_fma_serial` emulates fused multiply-add on platforms without hardware FMA using Dekker's algorithm: splits each operand into high and low halves via `a_high = (a * 4097.0f) - ((a * 4097.0f) - a)`, then computes the exact product error term.
+The magic constant $4097 = 2^{12} + 1$ splits a 24-bit mantissa into two 12-bit halves that multiply without rounding; `nk_f64_fma_serial` uses $134217729 = 2^{27} + 1$ for the 53-bit mantissa.
 `nk_f32_fma_haswell` uses hardware `VFMADD231SS` — single instruction, single cycle, exact to the last bit.
 
 ### Float-to-Integer Ordering
 
-`nk_f16_order_serial`, `nk_bf16_order_serial`, `nk_e4m3_order_serial` convert floating-point bit patterns to unsigned integers that preserve the total order.
-Positive floats are already ordered by their bit patterns; negative floats need bit inversion: `if (bits & sign_bit) bits = ~bits; else bits ^= sign_bit`.
-This enables integer comparison instructions (`VPCMPUD`) for floating-point sorting without branching — used by `nk_reduce_minmax_*` for Float8 and sub-32-bit types that lack native SIMD comparison.
+`nk_f16_order_serial`, `nk_bf16_order_serial`, `nk_e4m3_order_serial` map floating-point bit patterns onto signed integers that preserve the total order, then return the difference of the two keys.
+Positive floats are already ordered by their bit patterns; negative floats need bit inversion, applied branchlessly as `bits ^ -sign`.
+This keeps floating-point ordering on the integer pipeline — used by `nk_reduce_minmax_*` for Float8 and sub-32-bit types that lack native SIMD comparison.
+`nk_f16_order_sapphire` instead uses the native `VCOMISH` half-precision comparisons.
 
 ## Performance
 

package/numkong/include/numkong/set/README.md

@@ -53,36 +53,39 @@ def jaccard_words(a: np.ndarray, b: np.ndarray) -> float:
 
 ## Optimizations
 
-### Harley-Seal Carry-Save Adders for U1
+### Popcount Without Harley-Seal Carry-Save Adders
 
-`nk_hamming_u1_haswell`, `nk_jaccard_u1_haswell` amortize the cost of popcount by using Harley-Seal carry-save adder trees.
-Instead of computing popcount on every XOR/AND/OR result independently, three intermediate values are combined through a full-adder circuit:
+The textbook way to amortize population counts is a Harley-Seal carry-save adder tree, which folds three inputs through a full-adder circuit before any popcount is issued:
 
 ```
 ones  = a ^ b ^ c
 twos  = (a & b) | (c & (a ^ b))
 ```
 
-This circuit takes three popcount inputs and produces a ones and twos accumulator, where `twos` has double the weight of `ones`.
-By chaining two levels, a fours accumulator is also produced, so the actual `VPSHUFB`-based popcount is called only on the final accumulated ones, twos, and fours values.
-The total number of popcount operations is reduced by roughly a factor of three compared to computing popcount on every vector independently.
+Chaining levels yields `fours` and `eights` accumulators, each weighted by a power of two, so popcount runs on the accumulators instead of on every input vector — roughly a third of the calls.
+NumKong does not do this.
+The cycles-per-byte measurements in `set.h` put the crossover above 1 KB per vector: below it, loop and reduction overhead outweighs the saved popcounts, and the largest binary input benchmarked here is 4096 bits, or 512 bytes.
+Ice Lake and later expose `VPOPCNTQ` outright, and on Genoa it dual-issues on ports 0-1, so the popcount pressure that CSA trees relieve is largely absent on the hardware that could run them.
+
+AVX2 has no vector population count at all, so `nk_hamming_u1_haswell` and `nk_jaccard_u1_haswell` walk the packed octets 8 bytes at a time and call the scalar `POPCNT` through `_mm_popcnt_u64`.
+`nk_hamming_u1_neon` and `nk_jaccard_u1_neon` use the NEON `CNT` instruction instead, accumulating per-byte counts in a `uint8x16_t` for at most 31 iterations before widening, since 31 x 8 still fits in a byte.
 
 ### Native VPOPCNTQ on Ice Lake
 
 `nk_hamming_u1_icelake`, `nk_jaccard_u1_icelake` use `VPOPCNTQ` on 512-bit vectors, which directly produces per-quadword population counts for 8 quadwords at once.
-This single instruction replaces the entire nibble-LUT + Harley-Seal pipeline used on Haswell.
-The kernels batch 16 vectors before horizontal reduction to minimize `VPSADBW` overhead, accumulating the per-quadword counts into a running total via `VPADDQ`.
+Inputs of up to 256 bytes are fully unrolled into one, two, three, or four masked register pairs with no loop at all, since binary metrics are hard to speed up at tiny lengths.
+Longer inputs enter a loop that accumulates the per-quadword counts via `VPADDQ` and reduces once at the end.
 
 ### Jaccard via Precomputed Norms
 
-`nk_jaccard_u1_haswell`, `nk_jaccard_u1_icelake` exploit the identity $|A \cup B| = |A| + |B| - |A \cap B|$ to avoid computing both AND-popcount and OR-popcount in the inner loop.
-When vector norms (popcount of each vector) are precomputed and passed via the streaming API, only the intersection popcount is needed per pair, halving the work in the critical path.
+The streaming helpers `nk_jaccard_u1x64_finalize_haswell` and `nk_jaccard_u1x512_finalize_icelake` exploit the identity $|A \cup B| = |A| + |B| - |A \cap B|$ to avoid computing both AND-popcount and OR-popcount in the inner loop.
+The matching `update` step accumulates only the intersection popcount; `finalize` takes the query popcount and a vector of four target popcounts and recovers the union from them, halving the work in the critical path.
+Both finalizers replace the division by a reciprocal estimate — `RCPPS` on Haswell, `VRCP14PS` on Ice Lake — refined with one Newton-Raphson step.
 
-### Byte Hamming via VPSADBW
+### Byte Hamming via Compare Masks
 
-`nk_hamming_u8_haswell`, `nk_hamming_u8_icelake` compute byte-level Hamming distance using XOR to produce per-byte difference indicators, then `VPSADBW` against zero to horizontally sum the nonzero bytes.
-XOR produces 0 for equal bytes and nonzero for different ones, and `VPSADBW` sums the absolute values of byte differences within each 64-bit lane.
-Since XOR results are either 0 or nonzero (not necessarily 1), the kernel masks XOR output through `VPMIN` with a vector of ones to clamp each byte to 0 or 1 before feeding `VPSADBW`.
+`nk_hamming_u8_haswell` compares 32 bytes at a time with `VPCMPEQB`, extracts the per-byte sign bits of the two 128-bit halves with `VPMOVMSKB`, inverts the masks, and counts the set bits with scalar `POPCNT`.
+`nk_hamming_u8_icelake` does the same over 64 bytes with `VPCMPNEQB`, which writes the mismatch mask straight into a `__mmask64`, so no extract step is needed.
 
 ## Performance
 

package/numkong/include/numkong/set/haswell.h

@@ -31,8 +31,7 @@
  *  nk_jaccard_u1x64_init_haswell(&state_first);
  *  // ... stream through packed binary vectors ...
  *  nk_jaccard_u1x64_finalize_haswell(&state_first, &state_second, &state_third, &state_fourth,
- *      query_popcount, target_popcount_a, target_popcount_b, target_popcount_c, target_popcount_d,
- *      total_dimensions, &results);
+ *      query_popcount, &target_popcounts_vec, total_dimensions, &result_vec);
  *  @endcode
  */
 #ifndef NK_SET_HASWELL_H
@@ -240,8 +239,8 @@ NK_INTERNAL void nk_jaccard_u1x64_update_haswell(nk_jaccard_u1x64_state_haswell_
 NK_INTERNAL void nk_jaccard_u1x64_finalize_haswell( //
     nk_jaccard_u1x64_state_haswell_t const *state_a, nk_jaccard_u1x64_state_haswell_t const *state_b,
     nk_jaccard_u1x64_state_haswell_t const *state_c, nk_jaccard_u1x64_state_haswell_t const *state_d,
-    nk_f32_t query_popcount, nk_f32_t target_popcount_a, nk_f32_t target_popcount_b, nk_f32_t target_popcount_c,
-    nk_f32_t target_popcount_d, nk_size_t total_dimensions, nk_b128_vec_t *result) {
+    nk_f32_t query_popcount, nk_b128_vec_t const *target_popcounts_vec, nk_size_t total_dimensions,
+    nk_b128_vec_t *result_vec) {
     nk_unused_(total_dimensions);
 
     // 4-way SIMD Jaccard computation with fast reciprocal.
@@ -263,7 +262,7 @@ NK_INTERNAL void nk_jaccard_u1x64_finalize_haswell( //
     __m128 intersection_f32x4 = _mm_setr_ps(intersection_a_f32, intersection_b_f32, intersection_c_f32,
                                             intersection_d_f32);
     __m128 query_f32x4 = _mm_set1_ps(query_popcount);
-    __m128 targets_f32x4 = _mm_setr_ps(target_popcount_a, target_popcount_b, target_popcount_c, target_popcount_d);
+    __m128 targets_f32x4 = target_popcounts_vec->xmm_ps;
     __m128 union_f32x4 = _mm_sub_ps(_mm_add_ps(query_f32x4, targets_f32x4), intersection_f32x4);
 
     // Handle zero-union edge case
@@ -283,24 +282,24 @@ NK_INTERNAL void nk_jaccard_u1x64_finalize_haswell( //
 
     __m128 ratio_f32x4 = _mm_mul_ps(intersection_f32x4, union_reciprocal_f32x4);
     __m128 jaccard_f32x4 = _mm_sub_ps(one_f32x4, ratio_f32x4);
-    result->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
+    result_vec->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
 }
 
 /** @brief Hamming from_dot: computes pop_a + pop_b - 2*dot for 4 pairs (Haswell). */
-NK_INTERNAL void nk_hamming_u32x4_from_dot_haswell_(nk_b128_vec_t dots, nk_u32_t query_pop, nk_b128_vec_t target_pops,
-                                                    nk_b128_vec_t *results) {
-    __m128i dots_i32x4 = dots.xmm;
+NK_INTERNAL void nk_hamming_u32x4_from_dot_haswell_(nk_b128_vec_t const *dots_vec, nk_u32_t query_pop,
+                                                    nk_b128_vec_t const *target_pops_vec, nk_b128_vec_t *result_vec) {
+    __m128i dots_i32x4 = dots_vec->xmm;
     __m128i query_i32x4 = _mm_set1_epi32((int)query_pop);
-    __m128i target_i32x4 = target_pops.xmm;
-    results->xmm = _mm_sub_epi32(_mm_add_epi32(query_i32x4, target_i32x4), _mm_slli_epi32(dots_i32x4, 1));
+    __m128i target_i32x4 = target_pops_vec->xmm;
+    result_vec->xmm = _mm_sub_epi32(_mm_add_epi32(query_i32x4, target_i32x4), _mm_slli_epi32(dots_i32x4, 1));
 }
 
 /** @brief Jaccard from_dot: computes 1 - dot / (pop_a + pop_b - dot) for 4 pairs (Haswell). */
-NK_INTERNAL void nk_jaccard_f32x4_from_dot_haswell_(nk_b128_vec_t dots, nk_u32_t query_pop, nk_b128_vec_t target_pops,
-                                                    nk_b128_vec_t *results) {
-    __m128 dot_f32x4 = _mm_cvtepi32_ps(dots.xmm);
+NK_INTERNAL void nk_jaccard_f32x4_from_dot_haswell_(nk_b128_vec_t const *dots_vec, nk_u32_t query_pop,
+                                                    nk_b128_vec_t const *target_pops_vec, nk_b128_vec_t *result_vec) {
+    __m128 dot_f32x4 = _mm_cvtepi32_ps(dots_vec->xmm);
     __m128 query_f32x4 = _mm_set1_ps((nk_f32_t)query_pop);
-    __m128 target_f32x4 = _mm_cvtepi32_ps(target_pops.xmm);
+    __m128 target_f32x4 = _mm_cvtepi32_ps(target_pops_vec->xmm);
     __m128 union_f32x4 = _mm_sub_ps(_mm_add_ps(query_f32x4, target_f32x4), dot_f32x4);
 
     __m128 zero_union_mask = _mm_cmpeq_ps(union_f32x4, _mm_setzero_ps());
@@ -314,7 +313,7 @@ NK_INTERNAL void nk_jaccard_f32x4_from_dot_haswell_(nk_b128_vec_t dots, nk_u32_t
 
     __m128 ratio_f32x4 = _mm_mul_ps(dot_f32x4, union_reciprocal_f32x4);
     __m128 jaccard_f32x4 = _mm_sub_ps(one_f32x4, ratio_f32x4);
-    results->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
+    result_vec->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
 }
 
 #pragma endregion Stateful Streaming

package/numkong/include/numkong/set/icelake.h

@@ -30,8 +30,7 @@
  *  nk_jaccard_u1x512_init_icelake(&state_first);
  *  // ... stream through packed binary vectors ...
  *  nk_jaccard_u1x512_finalize_icelake(&state_first, &state_second, &state_third, &state_fourth,
- *      query_popcount, target_popcount_a, target_popcount_b, target_popcount_c, target_popcount_d,
- *      total_dimensions, &results);
+ *      query_popcount, &target_popcounts_vec, total_dimensions, &result_vec);
  *  @endcode
  */
 #ifndef NK_SET_ICELAKE_H
@@ -370,8 +369,8 @@ NK_INTERNAL void nk_jaccard_u1x512_update_icelake(nk_jaccard_u1x512_state_icelak
 NK_INTERNAL void nk_jaccard_u1x512_finalize_icelake( //
     nk_jaccard_u1x512_state_icelake_t const *state_a, nk_jaccard_u1x512_state_icelake_t const *state_b,
     nk_jaccard_u1x512_state_icelake_t const *state_c, nk_jaccard_u1x512_state_icelake_t const *state_d,
-    nk_f32_t query_popcount, nk_f32_t target_popcount_a, nk_f32_t target_popcount_b, nk_f32_t target_popcount_c,
-    nk_f32_t target_popcount_d, nk_size_t total_dimensions, nk_b128_vec_t *result) {
+    nk_f32_t query_popcount, nk_b128_vec_t const *target_popcounts_vec, nk_size_t total_dimensions,
+    nk_b128_vec_t *result_vec) {
     nk_unused_(total_dimensions);
 
     // Port-optimized 4-way horizontal reduction using early i64 → i32 truncation.
@@ -415,7 +414,7 @@ NK_INTERNAL void nk_jaccard_u1x512_finalize_icelake( //
     // Compute Jaccard distance: 1 - intersection ÷ union
     // where union = query_popcount + target_popcount - intersection
     __m128 query_f32x4 = _mm_set1_ps(query_popcount);
-    __m128 targets_f32x4 = _mm_setr_ps(target_popcount_a, target_popcount_b, target_popcount_c, target_popcount_d);
+    __m128 targets_f32x4 = target_popcounts_vec->xmm_ps;
     __m128 union_f32x4 = _mm_sub_ps(_mm_add_ps(query_f32x4, targets_f32x4), intersection_f32x4);
 
     // Handle zero-union edge case: if union == 0, result = 0.0
@@ -435,24 +434,24 @@ NK_INTERNAL void nk_jaccard_u1x512_finalize_icelake( //
 
     __m128 ratio_f32x4 = _mm_mul_ps(intersection_f32x4, union_reciprocal_f32x4);
     __m128 jaccard_f32x4 = _mm_sub_ps(one_f32x4, ratio_f32x4);
-    result->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
+    result_vec->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
 }
 
 /** @brief Hamming from_dot: computes pop_a + pop_b - 2*dot for 4 pairs (Icelake). */
-NK_INTERNAL void nk_hamming_u32x4_from_dot_icelake_(nk_b128_vec_t dots, nk_u32_t query_pop, nk_b128_vec_t target_pops,
-                                                    nk_b128_vec_t *results) {
-    __m128i dots_i32x4 = dots.xmm;
+NK_INTERNAL void nk_hamming_u32x4_from_dot_icelake_(nk_b128_vec_t const *dots_vec, nk_u32_t query_pop,
+                                                    nk_b128_vec_t const *target_pops_vec, nk_b128_vec_t *result_vec) {
+    __m128i dots_i32x4 = dots_vec->xmm;
     __m128i query_i32x4 = _mm_set1_epi32((int)query_pop);
-    __m128i target_i32x4 = target_pops.xmm;
-    results->xmm = _mm_sub_epi32(_mm_add_epi32(query_i32x4, target_i32x4), _mm_slli_epi32(dots_i32x4, 1));
+    __m128i target_i32x4 = target_pops_vec->xmm;
+    result_vec->xmm = _mm_sub_epi32(_mm_add_epi32(query_i32x4, target_i32x4), _mm_slli_epi32(dots_i32x4, 1));
 }
 
 /** @brief Jaccard from_dot: computes 1 - dot / (pop_a + pop_b - dot) for 4 pairs (Icelake). */
-NK_INTERNAL void nk_jaccard_f32x4_from_dot_icelake_(nk_b128_vec_t dots, nk_u32_t query_pop, nk_b128_vec_t target_pops,
-                                                    nk_b128_vec_t *results) {
-    __m128 dot_f32x4 = _mm_cvtepi32_ps(dots.xmm);
+NK_INTERNAL void nk_jaccard_f32x4_from_dot_icelake_(nk_b128_vec_t const *dots_vec, nk_u32_t query_pop,
+                                                    nk_b128_vec_t const *target_pops_vec, nk_b128_vec_t *result_vec) {
+    __m128 dot_f32x4 = _mm_cvtepi32_ps(dots_vec->xmm);
     __m128 query_f32x4 = _mm_set1_ps((nk_f32_t)query_pop);
-    __m128 target_f32x4 = _mm_cvtepi32_ps(target_pops.xmm);
+    __m128 target_f32x4 = _mm_cvtepi32_ps(target_pops_vec->xmm);
     __m128 union_f32x4 = _mm_sub_ps(_mm_add_ps(query_f32x4, target_f32x4), dot_f32x4);
 
     __m128 zero_union_mask = _mm_cmpeq_ps(union_f32x4, _mm_setzero_ps());
@@ -465,7 +464,7 @@ NK_INTERNAL void nk_jaccard_f32x4_from_dot_icelake_(nk_b128_vec_t dots, nk_u32_t
 
     __m128 ratio_f32x4 = _mm_mul_ps(dot_f32x4, union_reciprocal_f32x4);
     __m128 jaccard_f32x4 = _mm_sub_ps(one_f32x4, ratio_f32x4);
-    results->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
+    result_vec->xmm_ps = _mm_blendv_ps(jaccard_f32x4, _mm_setzero_ps(), zero_union_mask);
 }
 
 #pragma endregion Stateful Streaming

package/numkong/include/numkong/set/loongsonasx.h

@@ -144,20 +144,22 @@ NK_PUBLIC void nk_hamming_u8_loongsonasx(nk_u8_t const *a, nk_u8_t const *b, nk_
 #pragma region Batched Finalizers
 
 /** @brief Hamming from_dot: computes pop_a + pop_b − 2 × dot for 4 pairs (LSX). */
-NK_INTERNAL void nk_hamming_u32x4_from_dot_loongsonasx_(nk_b128_vec_t dots, nk_u32_t query_pop,
-                                                        nk_b128_vec_t target_pops, nk_b128_vec_t *results) {
-    __m128i dots_u32x4 = dots.xmm;
+NK_INTERNAL void nk_hamming_u32x4_from_dot_loongsonasx_(nk_b128_vec_t const *dots_vec, nk_u32_t query_pop,
+                                                        nk_b128_vec_t const *target_pops_vec,
+                                                        nk_b128_vec_t *result_vec) {
+    __m128i dots_u32x4 = dots_vec->xmm;
     __m128i query_u32x4 = __lsx_vreplgr2vr_w((int)query_pop);
-    __m128i target_u32x4 = target_pops.xmm;
-    results->xmm = __lsx_vsub_w(__lsx_vadd_w(query_u32x4, target_u32x4), __lsx_vslli_w(dots_u32x4, 1));
+    __m128i target_u32x4 = target_pops_vec->xmm;
+    result_vec->xmm = __lsx_vsub_w(__lsx_vadd_w(query_u32x4, target_u32x4), __lsx_vslli_w(dots_u32x4, 1));
 }
 
 /** @brief Jaccard from_dot: computes 1 − dot / (pop_a + pop_b − dot) for 4 pairs (LSX). */
-NK_INTERNAL void nk_jaccard_f32x4_from_dot_loongsonasx_(nk_b128_vec_t dots, nk_u32_t query_pop,
-                                                        nk_b128_vec_t target_pops, nk_b128_vec_t *results) {
-    __m128 dot_f32x4 = __lsx_vffint_s_wu(dots.xmm);
+NK_INTERNAL void nk_jaccard_f32x4_from_dot_loongsonasx_(nk_b128_vec_t const *dots_vec, nk_u32_t query_pop,
+                                                        nk_b128_vec_t const *target_pops_vec,
+                                                        nk_b128_vec_t *result_vec) {
+    __m128 dot_f32x4 = __lsx_vffint_s_wu(dots_vec->xmm);
     __m128 query_f32x4 = nk_xvreplgr2vr_s_128_((nk_f32_t)query_pop);
-    __m128 target_f32x4 = __lsx_vffint_s_wu(target_pops.xmm);
+    __m128 target_f32x4 = __lsx_vffint_s_wu(target_pops_vec->xmm);
     __m128 union_f32x4 = __lsx_vfsub_s(__lsx_vfadd_s(query_f32x4, target_f32x4), dot_f32x4);
 
     __m128 zero_f32x4 = (__m128)__lsx_vreplgr2vr_w(0);
@@ -167,7 +169,7 @@ NK_INTERNAL void nk_jaccard_f32x4_from_dot_loongsonasx_(nk_b128_vec_t dots, nk_u
 
     __m128 ratio_f32x4 = __lsx_vfdiv_s(dot_f32x4, safe_union_f32x4);
     __m128 jaccard_f32x4 = __lsx_vfsub_s(one_f32x4, ratio_f32x4);
-    results->xmm_ps = (__m128)__lsx_vbitsel_v((__m128i)jaccard_f32x4, (__m128i)zero_f32x4, zero_union_mask_u32x4);
+    result_vec->xmm_ps = (__m128)__lsx_vbitsel_v((__m128i)jaccard_f32x4, (__m128i)zero_f32x4, zero_union_mask_u32x4);
 }
 
 #pragma endregion Batched Finalizers