cumo 0.5.11 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b345c4e036212d334600041165fe79e533ef0e1dd450e22b1288f5e6d6214ffa
-  data.tar.gz: af60f78cdd2f39a96e8a183eff02a26c26d8977560ba148643c55df5961c2bf2
+  metadata.gz: 97c7f575209616cb9fa484e5dd4096bac11170aef18e99f1f6f733e599daa9af
+  data.tar.gz: 5efabd7aface13fac85dcef81fef54dd9bb22d3748666177ff221741a8419486
 SHA512:
-  metadata.gz: 361042e17c2f498b8e4f5ddc76bbd119688a6a3a3ac176307973fd51c75b332d4d831a82c0fc0e8a3b28be98bfbe50d507ec67d517c609c2342caf57d94d9c41
-  data.tar.gz: 53da42c447fe31812654e1ab0566f5e553f781daaf75b97bb1c36630b8dc3ed195704002990bc4a2a9f4fdd6e26e92c2d5905ff64da0e0f058b01dae20ae3dec
+  metadata.gz: 314d2b224cb753761618b6bc833af4715d8d1cb2e1fd8de05b0ad65483ec940c31167997abe7cc76e2f056b12b442885731d8e6b4d4b47ec174c19c70c2326e9
+  data.tar.gz: 5a4909d952dbe926b52ccaf04010b3ffcf08617a2af710ff49a70764ff7e626ffddcef04ebcb4ca6b5aab8c7b14170b9b3328768a203dd7961d2fbab5c973002

data/CHANGELOG.md

@@ -1,3 +1,78 @@
+# 0.7.0 (2026/09/13)
+
+Breaking changes:
+
+* `cumo_narray_data_t` no longer carries `owned` and `cumo_narray_t` no longer carries `elmsz`. Both ship in `narray.h`, so an extension compiled against 0.6.0 has to be rebuilt. `NArray#free` also refuses a frozen array, as the other release paths already did (PR #421)
+* `gemm` raises `RangeError` for the sizes cuBLAS takes as `int`, where they wrapped and the call read outside its operands (PR #399)
+* Taking a pointer refuses an `allocate` that leaves the array without a buffer, or hands back one already holding data, and `to_binary` measures the array by its type rather than by the `ELEMENT_BYTE_SIZE` its class names (PR #393, PR #392)
+* `store_binary` refuses a non-contiguous view, and a `Cumo::Bit` view that does not begin and end on a byte. It copies the string it is given, so `NArray#free` answers true after a frozen one where it answered false (PR #390, PR #389)
+* `Cumo::Int8.upcast(Cumo::UInt8)` answers `Cumo::Int16` where it answered `Cumo::Int8`, and `Cumo::Bit.upcast(Cumo::RObject)` answers rather than `nil`, the tables having been built before the classes they name existed (PR #378)
+
+Fixes:
+
+* Fix a view constructor that raises partway leaving `stridx` unreadable for the free hook, and leaking the index array it had just built (PR #407)
+* Fix `Cumo::Bit#each`, `map` and `map_with_index` reading what the block queued before the device had run it, 7 of the 18 dtype and method pairs having answered differently from numo (PR #401)
+* Fix `gemm` reading and writing far outside its operands when a matrix holds more than 2**31 elements, the batch strides reaching cuBLAS as `int` where cuBLAS takes `long long int` (PR #399)
+* Fix `median(nan: true)` answering with a number when a NaN is present, the host quicksort it used leaving NaN unordered (PR #398)
+* Fix `where`, `where2` and a masked `aref` writing past the index array they sized, `allocate` being free to resize it between the count and the walk (PR #397)
+* Fix `Cumo::Bit#each_with_index` writing in front of the counter array on a zero-dimensional array, which ordinary code reaches since indexing a Bit answers a zero-dimensional Bit (PR #396)
+* Fix the segfaults an `allocate` that takes no buffer, reshapes the array, or claims its own element size leaves behind, and `to_binary` handing back a 16KB String from a 32 byte allocation (PR #393, PR #392)
+* Fix `store_binary` writing at the start of the base rather than through the view, and pointing the array at a frozen String's bytes, which a compaction moves out from under it (PR #390, PR #389)
+* Fix storing a Ruby Array of sub-narrays: the rows came back as the first one repeated, a row short in an outer axis copied past its own end, and a row with no elements stored the loop's flag as a value, 8.5us a row down to 1.6 (PR #384, PR #383, PR #382)
+* Fix flattening a `Cumo::Bit` view that walks an axis backwards reading near address zero, which `to_a`, `where`, `where2` and `mask` all reach (PR #380)
+* Fix a length-1 axis costing a view its contiguity, and `reshape!` on one carrying a negative stride walking out of the allocation; of 288 views measured, 92 more answer `contiguous?` and none fewer (PR #379, PR #376)
+* Fix `:new` in front of an index-backed dimension reading and writing outside the array, only a trailing new axis having been recognized as one (PR #375)
+* Fix `min`, `max`, `minmax` and `ptp` answering a finite number for an array of infinities, the nan-aware forms seeding with the largest finite value the dtype holds (PR #372)
+
+Changes:
+
+* Add `Cumo::HFloat`, the 16-bit float dtype the GPU has had since Maxwell: the four operations, the comparisons, the reductions, `Cumo::NMath`, `sort`, `rand`, `dot` through cuBLAS with a float compute type, and the cuDNN methods, which now reach the tensor cores. A 1024x1024 `dot` fell to the mulsum fallback and was 285x slower before it, and `cumsum` stays undefined for half until the scan carries a wider accumulator (issue #107, PR #371, PR #370, PR #368, PR #367, PR #366, PR #365)
+* Copy a transposed 2-d view, and run an elementwise op with a transposed operand, through a 32x32 shared memory tile; on 16M SFloat the copy goes from 177 to 380 GB/s and the op from 247 to 396 GB/s (PR #416, PR #415)
+* Point a derived view at the index arrays it would have copied, one device allocation and one copy per dimension, and let `reverse` and `diagonal` borrow the dimensions they only pass through; on a 2**20 index `transpose` goes from 394us to 0.3us, `reverse(1)` from 327us to 0.2us and `diagonal` from 329us to 0.1us (PR #412, PR #410)
+* Let any device synchronize settle a view's index rather than only the one the index code makes, and record the epoch in every constructor that fills an index; storing sub-narrays into an index-backed view goes from 11.7-18.0us to 5.9-8.9us, and a reversed, expanded or diagonal index view stops waiting once per read (PR #414, PR #388)
+* Wait for a sub-narray index only when the row binds one, rather than whenever any argument uses one; a store into an index-backed destination goes from 6.3us to 1.7us a row (PR #387)
+* Reverse a view's index array with a kernel instead of a host loop that stopped the whole device first; a 2**20 index turns around in 480us rather than 1600us (PR #406)
+* Wait for the device once per row of a host sort instead of once per recursion, a 20000 element row having cost 4003 waits; over 200000 elements `sort(nan: true)` goes from 18.95 to 13.03 ms, `median` from 18.15 to 12.41 and `sort_index` from 20.99 to 14.39 (PR #394)
+* Warn when a view is built holding an index array that nothing owns, which leaks it when that view is the last reference and dangles when it is not (PR #413)
+* Say once what the duplicated templates and kernels had each grown a copy of: the transpose tile walk, the `with_index` block argument and its 0-dimensional clamp, the Bit pointer rebase rule, the min and max rules, the minmax pair, and the byte size a packed bit array takes (PR #422, PR #418, PR #409, PR #408, PR #381, PR #377, PR #374, PR #373)
+* Carry the sub-narray reach per dimension and through the row's own shape rather than through the value channel the store templates read (PR #386, PR #385)
+* Separate the cuDNN scalar, compute and parameter types from the tensor's dtype, so a dtype whose compute type differs is named in `gen/def` rather than in eight templates (PR #369)
+* Delete the `dot` and cuBLAS option code no build compiles, the branch that called a view with no strides contiguous, the second spelling of whether a view's offset reaches a char pointer, and the wait in front of a blocking copy that already queues behind the fill it waited for (PR #417, PR #411, PR #404, PR #403)
+* Count the waits a call makes in the sync probe and probe the block walks, run `Cumo::HFloat` through the structural, math and alt-coverage suites, skip the boundary tests on a machine too small for them, and name the message the gemm row count tests are there for (PR #420, PR #419, PR #405, PR #402, PR #400)
+
+# 0.6.0 (2026/09/06)
+
+Breaking changes:
+
+* `divmod` and `%` floor the quotient and give the remainder the divisor's sign, as Ruby does, on every signed type: `Cumo::Int32[-7].divmod(3)` answers `[-3, 2]` where it answered `[-2, -1]`, and `-7 % 3` answers `2` rather than `-1`. Dividing a float by zero answers a quotient of `NaN` where it answered `Infinity`. Unsigned types are unchanged (PR #364)
+* A view whose base has since been given a smaller shape raises `RuntimeError` instead of reading past what the base holds (PR #363)
+* `marshal_load` and `initialize` leave an array unallocated when they take a new shape, so it reads as unallocated until something is stored in it, and both refuse a view. `initialize` also refuses a frozen array, as `marshal_load` already did (PR #361, PR #360, PR #355)
+* A range subscript longer than `2**31` answers the length it names instead of an empty view, and one longer than `2**32` no longer answers with the low 32 bits of it (PR #354)
+* `cov` rejects a 3-dimensional receiver or `y`, 2-dimensional or non-integer `fweights`, and any `ddof` but 0 or 1, where it answered from them; it warns and answers `nan` when the degrees of freedom come out at zero or below (PR #343)
+* `Cumo::Bit#extract` answers with a zero-dimensional `Cumo::Bit`, as every other dtype does, rather than a Ruby Integer read back through a synchronize (PR #341)
+* `batch_norm` and its backward reject an axis that names nothing in `x`, repeats a dimension, or is out of order, where they answered as though a different axis had been given (PR #336)
+
+Fixes:
+
+* Fix `RObject#divmod` reading the pair an element's `divmod` answered without looking at it: a bare Integer or a String took the process down, and a shorter Array handed back the words past its end as the quotient and the remainder (PR #359)
+* Fix an out-of-bounds read when `initialize` is given a new shape through `send`, which left the buffer sized for the old one: 4096 elements were read out of 8, and the same on an `Cumo::RObject` crashed at the next collection (PR #361)
+* Fix a memory leak when an Array subscript is rejected, the host buffer it is staged in being freed only after the loop that fills it, and the memory a subscript piles up before the collector takes it back (PR #357, PR #358)
+* Fix `marshal_load` writing past a buffer sized for the shape it replaces, and the double free that came of releasing a buffer noted down before the shape's own `to_int` had run (PR #355, PR #356)
+* Fix reading past a Ruby array or string that a conversion shrank underneath the walk: a subscript, a shape, an axis list, a marshal array and the string behind `from_binary` were each measured once and walked afterwards, and `to_int` is free to empty any of them (PR #353)
+* Fix `Cumo::Bit#swap_byte` walking one byte per bit, eight times past the end of the packed buffer; `hton`, `to_network` and `to_swapped` reach the same loop (PR #352)
+
+Changes:
+
+* Add `argsort`, which answers positions along an axis rather than into the flattened array as `sort_index` does (PR #342)
+* Add `mean`, `var`, `stddev` and `rms` to the integer types and to `Cumo::Bit`, which had only the float ones (PR #344, PR #345)
+* Walk an elementwise function along the memory of the operand it writes rather than the shape it was given; on a transposed 1024x1024 SFloat view `a.inplace + 1.0` goes from 1180.6us to 78.3us (PR #351)
+* Reduce along the contiguous axis with a contiguous group of threads, keep a reduction's operands out of local memory, and give a short strided row more elements per thread; `[1024, 4, 1024].sum(axis: 1)` goes from 142.0us to 23.4us and the same `mulsum` from 675.3us to 30.9us (PR #350, PR #347)
+* Split a flat index in 32 bits where the array fits in them, the 64-bit division being a software routine on the GPU; a 4M SFloat `a[true, 1..-2] * 2.0` goes from 60.6us to 38.7us (PR #348)
+* Reverse the bytes of an array in a kernel instead of on the host; a 4M SFloat `swap_byte` goes from 5617.0us to 13.0us (PR #335)
+* Add `bench/cumo_shape_probe.rb`, which reads each shape of an operation against the same bytes moved along the best path it has (PR #349)
+* Make every object depend on the headers it is built from, a header-only change having left stale objects behind (PR #346)
+* Answer the four cuDNN TODOs left in the tree, and say what the memory pool's `TestRemoveFromFreeList` holds rather than what it waits for (PR #340, PR #339, PR #338, PR #337)
+
 # 0.5.11 (2026/08/29)
 
 Breaking changes:
@@ -232,7 +307,7 @@ Fixes:
 * Backport: improve discrimination between int32/64
 * Backport: missing declaration of id_ge
 * Backport: support casting any object that responds to 'to_a'
-* Backport: fix the range of int32 as -2**31 .. 2**31-1
+* Backport: fix the range of int32 as `-2**31 .. 2**31-1`
 * Backport: if int32_max is Bignum, it should be protected from GC
 * fix: handle compiler flags included in RbConfig::CONFIG properly (#157)
 * Backport: Added more percentile tests

data/README.md

@@ -222,6 +222,126 @@ Cumo::SFloat::Math.atan2(a, 2.0)   #=> Cumo::SFloat
 
 The 0-dimensional form has no effect under Numo, where `[]` returns a Ruby Float.
 
+### Half Precision
+
+`Cumo::HFloat`, also reachable as `Cumo::Float16`, holds IEEE binary16: one sign bit, five of exponent and ten of mantissa.
+It exists for the two things half is good at, moving half as many bytes and reaching the tensor cores, and it promotes exactly as `Cumo::SFloat` does, so an integer array or a Ruby Float mixed into an expression stays half while anything wider takes over.
+
+What it cannot hold is the thing to plan around.
+Integers are exact only to 2048, and the largest finite value is 65504:
+
+```ruby
+Cumo::HFloat[2049.0]        #=> 2048.0
+Cumo::HFloat[50257.0]       #=> 50272.0
+Cumo::HFloat[100000]        #=> Infinity
+```
+
+There is no exception on the way past the top; the value saturates, as it does in every other float type.
+Anything that carries an index rather than a measurement has to be built in a wider type and cast afterwards.
+
+Reductions do not inherit that limit, because they accumulate in single precision and round once at the end.
+A sum of forty thousand ones is forty thousand, not the 2048 a half accumulator would stop at:
+
+```ruby
+Cumo::HFloat.new(40_000).fill(1.0).sum   #=> 40000.0
+```
+
+`sum`, `mean`, `var`, `stddev`, `rms`, `mulsum`, `dot`, `gemm` and `cumsum` all widen this way.
+`prod` and `cumprod` do not: a product leaves half's range long before it loses precision, so widening the accumulator would only hide the overflow.
+
+The widening protects the accumulation, not the answer, which is still stored as half.
+A variance above 65504 therefore saturates even though nothing overflowed while it was being computed, and the standard deviation of the same array is fine because the square root brings it back into range:
+
+```ruby
+a = Cumo::HFloat[2000.0, -2000.0, 1000.0, -1000.0]
+a.var      #=> Infinity
+a.stddev   #=> 1826.0
+```
+
+That distinction is what makes half usable in a transformer's layer normalization, and getting it wrong is the first thing to go wrong there.
+Squaring overflows at 256, since 256 squared is already past the top, and a residual stream with one outlier feature reaches thousands.
+Writing the normalization out as `((x - mean) ** 2).mean` builds those squares as a half array and the answer is `Infinity`, while `x.var(axis: 1)` keeps them in its single-precision accumulator and answers 10208.0 against a single-precision 10209.
+`var` saturates only if the variance itself is out of range, which is a far higher bar than any one deviation being over 256, but it is still a bar.
+The quantity to check against 65504 is the largest variance a layer produces, not the largest activation in it.
+A single outlier of size d among n values contributes only d squared over n to the variance, so a row 768 wide divides it by 768: an activation of 3000 squares to nine million but raises the variance of its row by about twelve thousand.
+An activation that looks safe therefore says nothing about whether `var` overflows, in either direction.
+
+A `dot` whose answer does not fit still saturates, since the result is stored back as half.
+The accumulator itself is single precision, so scaling it down on the way out recovers the value:
+
+```ruby
+a = Cumo::HFloat.ones(1, 1024)
+b = Cumo::HFloat.new(1024, 1).fill(100.0)
+a.dot(b)                   #=> Infinity
+a.gemm(b, alpha: 0.001)    #=> 102.375, the true 102400 scaled down
+```
+
+#### What half is faster at
+
+`gemm` reaches the tensor cores. Square matrices on an RTX 5070 Ti Laptop, median of three runs each:
+
+```
+              HFloat              SFloat              DFloat
+1024x1024     0.043 ms  49.7 TF   0.161 ms  13.3 TF   5.370 ms  0.40 TF
+2048x2048     0.362 ms  47.4 TF   1.360 ms  12.6 TF   43.46 ms  0.40 TF
+4096x4096     3.335 ms  41.2 TF   10.40 ms  13.2 TF   328.1 ms  0.42 TF
+```
+
+An odd number of columns costs half far more than it costs the others, because a row then starts on a two-byte boundary and the vectorized path is gone.
+It is the column count of either operand that matters, not the row count.
+1024x1024 times 1024x1024, with one dimension made odd at a time:
+
+```
+             all even   M odd      K odd      N odd
+HFloat        51.8 TF   49.2 TF    23.3 TF    23.3 TF
+SFloat        11.9 TF      -       11.6 TF    10.8 TF
+```
+
+The run-to-run spread on these is a few per cent and reaches fifteen at the top end, so the M column says the row count does not matter rather than that it costs 5 per cent.
+
+The penalty is on the arithmetic, so it does not reach a matrix-vector product, which is bound by how fast the matrix can be read whatever its shape.
+A 1x768 by 768x50257 gemv takes 0.211 ms with that odd 50257 and 0.205 ms with 50256, a difference inside the noise; the same 768x50257 matrix against 256 rows takes 0.833 ms and 0.481 ms, which is not.
+Pad the inner dimensions of a real matrix product; leave a gemv alone.
+
+`conv` is a different story, and worth reading before reaching for half in a network.
+cuDNN chooses its algorithm from the ones that fit in a scratch buffer, and the half algorithms that use the tensor cores ask for more than the default 8MB ceiling allows.
+Left at the default, a half convolution is no faster than a single-precision one.
+N=32, C=K=64, 56x56, 3x3:
+
+```
+                              HFloat     SFloat
+CUMO_CUDNN_MAX_WORKSPACE_SIZE unset      1.13 ms    1.03 ms
+CUMO_CUDNN_MAX_WORKSPACE_SIZE=268435456  0.56 ms    0.68 ms
+```
+
+Tensor cores also want the channel counts to be multiples of eight, which the first layer of a network never satisfies.
+That layer is still faster in half, but for the other reason:
+
+```
+C=3, K=64, 56x56, 3x3     0.085 ms   0.170 ms
+C=K=64, 56x56, 1x1        0.037 ms   0.120 ms
+```
+
+Neither of those reaches a tensor core; they move half the bytes.
+
+#### Batch normalization takes single-precision parameters
+
+cuDNN derives the descriptor for the batch norm parameters from `x`, and widens it to float when `x` is half.
+`gamma`, `beta`, `running_mean`, `running_var`, `mean` and `inv_std` are therefore `Cumo::SFloat` where `x` is `Cumo::HFloat`, and `batch_norm_backward` answers `gx` in half with `ggamma` and `gbeta` in single:
+
+```ruby
+x = Cumo::HFloat.new(2, 4, 3, 3).seq
+gamma = Cumo::SFloat.ones(4)
+beta = Cumo::SFloat.zeros(4)
+x.batch_norm(gamma, beta, axis: [0, 2, 3])   #=> Cumo::HFloat
+```
+
+Passing half parameters raises `TypeError: gamma must be Cumo::SFloat, not Cumo::HFloat`.
+Keeping the running statistics in single precision is what the arithmetic wants in any case: a momentum update is a long chain of small corrections, and eleven bits of mantissa lose them.
+
+Gradients are the other place half runs out of room.
+Values below `Cumo::HFloat::MIN` of 6.1e-05 fall into the subnormals and then to zero, which is what loss scaling in a training loop exists to prevent.
+
 ### Select a GPU device ID
 
 Set the `CUDA_VISIBLE_DEVICES=id` environment variable, or
@@ -363,6 +483,19 @@ export CUMO_SHOW_WARNING=ON
 export CUMO_SHOW_WARNING_ONCE=OFF
 ```
 
+### Raise the cuDNN workspace ceiling
+
+cuDNN picks a convolution algorithm by benchmarking the ones that fit in a scratch buffer, and the ceiling on that buffer is 8MB.
+The fastest half precision algorithms, the ones that reach the tensor cores, ask for more than that and are left out of the search.
+To let them in:
+
+```
+export CUMO_CUDNN_MAX_WORKSPACE_SIZE=67108864
+```
+
+The value is in bytes and only bounds the search; each convolution reserves what its chosen algorithm actually needs.
+`Cumo::CUDA::CUDNN.max_workspace_size` reads back the value in force.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/sonots/cumo.

data/bench/cumo_shape_probe.rb

@@ -0,0 +1,264 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# ---------------------------------------------------------------------------
+# Cumo shape sweep
+#
+# cumo_probe.rb runs every kernel family at one shape and reads each member
+# against the fastest of its group. What that cannot see is a family whose
+# speed depends on the shape it is handed: a reduction along a short row, an
+# operand that is broadcast or strided, a copy that transposes. This sweep
+# holds the bytes fixed, varies the shape and the layout, and reads every case
+# against the same bytes moved down the best path there is for them: the flat
+# reduction, the add of two contiguous arrays, the contiguous copy.
+#
+#   GPU=1 ruby cumo_shape_probe.rb                     # the default sweep
+#   GPU=1 GROUP=reduce ruby cumo_shape_probe.rb        # reduce, elementwise or copy
+#   GPU=1 ALL=1 ruby cumo_shape_probe.rb               # every row, not just the flagged
+#   GPU=1 FLAG=3 ruby cumo_shape_probe.rb              # how far off the reference is called out
+#   GPU=1 DTYPE=DFloat ELEMENTS=67108864 ruby cumo_shape_probe.rb
+#   GPU=1 JSON=before.json ruby cumo_shape_probe.rb    # keep this run's rows ...
+#   GPU=1 COMPARE=before.json ruby cumo_shape_probe.rb # ... and read a later build against them
+#   ruby cumo_shape_probe.rb                           # the same sweep on Numo
+#
+# Every case queues LAUNCHES calls and synchronizes once, and the run opens
+# with a few seconds of load. On a laptop the memory clock climbs in steps,
+# and a best-of-N that synchronizes between calls can settle on a step below
+# the top and read a third low without anything being wrong.
+#
+# The sweep runs in one process. The cases work in place or reduce, so the
+# only results allocated are the reductions', a thousandth of their input.
+#
+# The default 4M elements of SFloat is 16 MB, inside the L2 of this class of
+# GPU, which is where the activations of a model live. ELEMENTS=67108864 moves
+# the sweep to DRAM, where a cost paid per element hides behind the bandwidth.
+# ---------------------------------------------------------------------------
+
+require 'json'
+
+GPU = !%w[0 false].include?(ENV['GPU'].to_s.downcase) && !ENV['GPU'].to_s.empty?
+
+if GPU
+  require 'cumo/narray'
+  XM = Cumo
+else
+  require 'numo/narray'
+  XM = Numo
+end
+
+GROUP    = ENV['GROUP']
+FLAG     = (ENV['FLAG'] || 2.0).to_f
+SHOW_ALL = !ENV['ALL'].to_s.empty?
+LAUNCHES = (ENV['LAUNCHES'] || 10).to_i
+ROUNDS   = (ENV['ROUNDS'] || 4).to_i
+ELEMENTS = (ENV['ELEMENTS'] || (1 << 22)).to_i
+DTYPE    = XM.const_get(ENV['DTYPE'] || 'SFloat')
+JSON_OUT = ENV['JSON']
+COMPARE  = ENV['COMPARE']
+
+def sync
+  XM::CUDA::Runtime.cudaDeviceSynchronize if GPU
+end
+
+def now
+  Process.clock_gettime(Process::CLOCK_MONOTONIC)
+end
+
+def elmsz
+  DTYPE::ELEMENT_BYTE_SIZE
+end
+
+def warm(seconds)
+  a = DTYPE.new(1 << 22).fill(0)
+  t = now
+  a.inplace + 1 while now - t < seconds
+  sync
+end
+
+# Seconds for one call, best of ROUNDS batches of LAUNCHES calls.
+def time_one
+  2.times { yield }
+  sync
+  best = Float::INFINITY
+  ROUNDS.times do
+    sync
+    t = now
+    LAUNCHES.times { yield }
+    sync
+    best = [best, (now - t) / LAUNCHES].min
+  end
+  best
+end
+
+Row = Struct.new(:group, :name, :shape, :us, :gbs, :ref, keyword_init: true) do
+  def key
+    "#{group}/#{name}"
+  end
+end
+
+ROWS = []
+
+# bytes is what the case moves, and the reference row is the one this case is
+# read against.
+def probe(group, name, shape, bytes, ref: nil, &blk)
+  return if GROUP && !group.start_with?(GROUP)
+
+  s = time_one(&blk)
+  ROWS << Row.new(group: group, name: name, shape: shape.inspect, us: s * 1e6, gbs: bytes / s / 1e9, ref: ref || name)
+end
+
+def rand_array(*shape)
+  a = DTYPE.new(*shape)
+  DTYPE.name.include?('Int') ? a.rand(100) : a.rand
+end
+
+# --- reductions ---------------------------------------------------------------
+#
+# The same ELEMENTS reduced along the last axis, the first axis and a middle
+# axis, over rows of every length, all read against the flat reduction of the
+# same array. Each output of every case is a reduction over len elements.
+def sweep_reduce
+  n = ELEMENTS
+  es = elmsz
+  a = rand_array(n)
+  b = rand_array(n)
+  methods = %i[sum max mean argmax mulsum].select { |m| a.respond_to?(m) }
+  methods.each do |m|
+    bytes = (m == :mulsum ? 2 : 1) * n * es
+    probe('reduce', "#{m} flat", [n], bytes) { m == :mulsum ? a.mulsum(b) : a.send(m) }
+  end
+  [4, 16, 64, 256, 1024, 4096].each do |len|
+    next if len > n
+
+    rows = n / len
+    r1 = 1 << (Math.log2(rows).to_i / 2)
+    r2 = rows / r1
+    last = a.reshape(rows, len)
+    last_b = b.reshape(rows, len)
+    first = a.reshape(len, rows)
+    first_b = b.reshape(len, rows)
+    mid = a.reshape(r1, len, r2)
+    mid_b = b.reshape(r1, len, r2)
+    methods.each do |m|
+      bytes = (m == :mulsum ? 2 : 1) * n * es
+      ref = "#{m} flat"
+      call = lambda do |x, y, axis|
+        m == :mulsum ? x.mulsum(y, axis: axis) : x.send(m, axis: axis)
+      end
+      probe('reduce', "#{m} len #{len} last", [rows, len], bytes, ref: ref) { call.call(last, last_b, 1) }
+      probe('reduce', "#{m} len #{len} first", [len, rows], bytes, ref: ref) { call.call(first, first_b, 0) }
+      probe('reduce', "#{m} len #{len} middle", [r1, len, r2], bytes, ref: ref) { call.call(mid, mid_b, 1) }
+    end
+  end
+end
+
+# --- elementwise --------------------------------------------------------------
+#
+# One [R, C] array added to in place, its operand contiguous, a scalar, a
+# broadcast row or column, a slice, a stepped view or a transpose, plus the
+# same broadcast through three and four dimensions. The reference is the add
+# of two contiguous arrays, which the loop walks as one run.
+def sweep_elementwise
+  n = ELEMENTS
+  es = elmsz
+  r = 1024
+  c = n / r
+  a = rand_array(r, c)
+  b = rand_array(r, c)
+  row = rand_array(c)
+  col = rand_array(r, 1)
+  probe('elementwise', 'a + b contiguous', [r, c], 3 * n * es) { a.inplace + b }
+  probe('elementwise', 'a * scalar', [r, c], 2 * n * es, ref: 'a + b contiguous') { a.inplace * 2 }
+  probe('elementwise', 'a + row broadcast', [r, c], 2 * n * es, ref: 'a + b contiguous') { a.inplace + row }
+  probe('elementwise', 'a - column broadcast', [r, c], 2 * n * es, ref: 'a + b contiguous') { a.inplace - col }
+  a3 = a.reshape(16, r / 16, c)
+  b3 = rand_array(1, r / 16, 1)
+  probe('elementwise', 'a + broadcast 3d', [16, r / 16, c], 2 * n * es, ref: 'a + b contiguous') { a3.inplace + b3 }
+  a4 = a.reshape(4, 4, r / 16, c)
+  b4 = rand_array(1, 4, 1, c)
+  probe('elementwise', 'a + broadcast 4d', [4, 4, r / 16, c], 2 * n * es, ref: 'a + b contiguous') { a4.inplace + b4 }
+  half = 0...(c / 2)
+  probe('elementwise', 'a + b column slice', [r, c / 2], 3 * n * es / 2, ref: 'a + b contiguous') { a[true, half].inplace + b[true, half] }
+  every_other = (0..-1) % 2
+  probe('elementwise', 'a * scalar stepped rows', [r / 2, c], n * es, ref: 'a + b contiguous') { a[every_other, true].inplace * 2 }
+  probe('elementwise', 'a + b transposed views', [c, r], 3 * n * es, ref: 'a + b contiguous') { a.transpose.inplace + b.transpose }
+  probe('elementwise', 'a.gt(b) to Bit', [r, c], 2 * n * es, ref: 'a + b contiguous') { a.gt(b) }
+  probe('elementwise', 'a.gt(row) to Bit', [r, c], n * es, ref: 'a + b contiguous') { a.gt(row) }
+end
+
+# --- copies -------------------------------------------------------------------
+#
+# A store from a view of every kind into contiguous memory, read against the
+# contiguous copy of the same bytes.
+def sweep_copy
+  n = ELEMENTS
+  es = elmsz
+  r = 1024
+  c = n / r
+  a = rand_array(r, c)
+  dst = DTYPE.new(r, c).fill(0)
+  dst_t = DTYPE.new(c, r).fill(0)
+  probe('copy', 'contiguous', [r, c], 2 * n * es) { dst.store(a) }
+  half = 0...(c / 2)
+  probe('copy', 'column slice', [r, c / 2], n * es, ref: 'contiguous') { dst[true, half].store(a[true, half]) }
+  every_other = (0..-1) % 2
+  probe('copy', 'stepped rows', [r / 2, c], n * es, ref: 'contiguous') { dst[every_other, true].store(a[every_other, true]) }
+  probe('copy', 'reversed rows', [r, c], 2 * n * es, ref: 'contiguous') { dst.store(a.reverse(0)) }
+  probe('copy', 'transposed', [c, r], 2 * n * es, ref: 'contiguous') { dst_t.store(a.transpose) }
+  idx = XM::Int32.new(r / 2).seq * 2
+  probe('copy', 'index-backed rows', [r / 2, c], n * es, ref: 'contiguous') { dst[0...(r / 2), true].store(a[idx, true]) }
+end
+
+# --- run ----------------------------------------------------------------------
+
+version = GPU ? Cumo::NArray::VERSION : Numo::NArray::VERSION
+puts "backend   : #{XM} #{version}"
+puts "dtype     : #{DTYPE}, #{ELEMENTS} elements (#{(ELEMENTS * elmsz / 1e6).round(1)} MB), #{LAUNCHES} launches per sync, best of #{ROUNDS}"
+warm(3)
+sweep_reduce
+sweep_elementwise
+sweep_copy
+
+refs = ROWS.to_h { |row| [row.key, row] }
+flagged = []
+puts
+puts format('  %-11s %-26s %-18s %9s %8s %8s', 'group', 'case', 'shape', 'us', 'GB/s', 'vs ref')
+ROWS.each do |row|
+  ref = refs["#{row.group}/#{row.ref}"]
+  ratio = ref ? ref.gbs / row.gbs : 1.0
+  slow = ratio > FLAG
+  flagged << row if slow
+  next unless slow || SHOW_ALL || row.ref == row.name
+
+  puts format('  %-11s %-26s %-18s %9.1f %8.1f %7.1fx%s', row.group, row.name, row.shape, row.us, row.gbs, ratio, slow ? '  <--' : '')
+end
+puts
+if flagged.empty?
+  puts format('  nothing was more than %.1fx off its reference', FLAG)
+else
+  puts format('  %d of %d cases more than %.1fx off their reference', flagged.size, ROWS.size, FLAG)
+end
+
+if COMPARE
+  before = JSON.parse(File.read(COMPARE))
+  puts
+  puts "  against #{File.basename(COMPARE)} (#{before['meta']['backend']} #{before['meta']['version']}, saved #{before['meta']['saved']})"
+  puts format('  %-11s %-26s %9s %9s %8s', 'group', 'case', 'before us', 'after us', 'speedup')
+  ROWS.each do |row|
+    was = before['rows'][row.key]
+    next unless was
+
+    speedup = was['us'] / row.us
+    next unless SHOW_ALL || speedup > 1.25 || speedup < 0.8
+
+    puts format('  %-11s %-26s %9.1f %9.1f %7.2fx', row.group, row.name, was['us'], row.us, speedup)
+  end
+end
+
+if JSON_OUT
+  out = { 'meta' => { 'backend' => XM.to_s, 'version' => version, 'dtype' => DTYPE.to_s, 'elements' => ELEMENTS,
+                      'saved' => Time.now.strftime('%Y-%m-%d %H:%M') },
+          'rows' => ROWS.to_h { |row| [row.key, { 'shape' => row.shape, 'us' => row.us, 'gbs' => row.gbs }] } }
+  File.write(JSON_OUT, JSON.pretty_generate(out))
+  puts "  wrote #{ROWS.size} rows to #{JSON_OUT}"
+end

data/bench/cumo_sync_probe.rb

@@ -90,8 +90,8 @@ end
 
 PROBES = []
 
-def probe(group, name, &setup)
-  PROBES << { group: group, name: name, setup: setup }
+def probe(group, name, counts_only: false, &setup)
+  PROBES << { group: group, name: name, setup: setup, counts_only: counts_only }
 end
 
 # Reading data back is meant to synchronize. The question is whether it says so.
@@ -283,6 +283,47 @@ probe('compute', 'index aref a[idx, true]') do
 end
 
 # Host to device
+# A block walks the array on the host and can queue work on it, so the walk has
+# to wait between elements. It only has to wait when the block queued something,
+# which is what the idle count says: nothing for a block that stays off the
+# device, once per touched element for one that does not.
+BLOCK_N = 32
+
+probe('block', 'each (block off the device)', counts_only: true) do
+  a = Cumo::SFloat.new(BLOCK_N).seq
+  -> { a.each { |x| x } }
+end
+
+probe('block', 'each (block writes)', counts_only: true) do
+  a = Cumo::SFloat.new(BLOCK_N).seq
+  -> { i = 0; a.each { |_x| a[i] = 1; i += 1 } }
+end
+
+probe('block', 'each_with_index (block writes)', counts_only: true) do
+  a = Cumo::SFloat.new(BLOCK_N).seq
+  -> { a.each_with_index { |_x, i| a[i] = 1 } }
+end
+
+probe('block', 'map (block writes)', counts_only: true) do
+  a = Cumo::SFloat.new(BLOCK_N).seq
+  -> { i = 0; a.map { |x| a[i] = 1; i += 1; x } }
+end
+
+probe('block', 'map_with_index (block writes)', counts_only: true) do
+  a = Cumo::SFloat.new(BLOCK_N).seq
+  -> { a.map_with_index { |x, i| a[i] = 1; x } }
+end
+
+probe('block', 'Bit each (block writes)', counts_only: true) do
+  a = Cumo::Bit.new(BLOCK_N).fill(1)
+  -> { i = 0; a.each { |_x| a[i] = 0; i += 1 } }
+end
+
+probe('block', 'Bit each_with_index (block writes)', counts_only: true) do
+  a = Cumo::Bit.new(BLOCK_N).fill(1)
+  -> { a.each_with_index { |_x, i| a[i] = 0 } }
+end
+
 probe('h2d', 'from_binary') do
   bin = Array.new(BIG_N / 4, 1.0).pack('f*')
   -> { Cumo::SFloat.from_binary(bin, [BIG_N / 4]) }
@@ -347,8 +388,8 @@ puts
 
 results = []
 
-puts format('  %-26s %10s %12s  %-9s %s', 'probe', 'idle', 'behind queue', 'verdict', 'warning')
-puts "  #{'-' * 84}"
+puts format('  %-34s %10s %6s %12s  %-9s %s', 'probe', 'idle', 'waits', 'behind queue', 'verdict', 'warning')
+puts "  #{'-' * 99}"
 
 current_group = nil
 PROBES.each do |p|
@@ -363,7 +404,22 @@ PROBES.each do |p|
   begin
     call = p[:setup].call
     if call.nil?
-      puts format('  %-26s %10s %12s  %-9s %s', p[:name], '-', '-', 'skip', '(no such API)')
+      puts format('  %-34s %10s %6s %12s  %-9s %s', p[:name], '-', '-', '-', 'skip', '(no such API)')
+      next
+    end
+
+    # (1b) how many times it waits with nothing queued. With the warning left
+    # on every occurrence this is the wait count, not a yes or no.
+    sync
+    idle_text = capture_stdio { call.call }
+    idle_warns = idle_text.to_s.lines.grep(/warn|sync|Sync|WARN/i).size
+
+    if p[:counts_only]
+      # Timing these would measure the warning writes, not the call.
+      puts format('  %-34s %10s %6d %12s  %-9s %s',
+                  p[:name], '-', idle_warns, '-', 'waits', '-')
+      results << { name: p[:name], group: p[:group], sync: idle_warns.positive?,
+                   warned: idle_warns.positive? }
       next
     end
 
@@ -412,12 +468,12 @@ PROBES.each do |p|
         format('x%d %s', warns.size, warns.first.to_s.strip[0, 40])
       end
 
-    puts format('  %-26s %8.1f us %10.2f ms  %-9s %s',
-                p[:name], idle * 1e6, blocked * 1e3,
+    puts format('  %-34s %8.1f us %6d %10.2f ms  %-9s %s',
+                p[:name], idle * 1e6, idle_warns, blocked * 1e3,
                 is_sync ? 'SYNC' : 'async', note)
   rescue StandardError, NotImplementedError => e
     msg = e.message.to_s.split("\n").first.to_s[0, 40]
-    puts format('  %-26s %10s %12s  %-9s %s', p[:name], '-', '-', 'ERROR', "#{e.class}: #{msg}")
+    puts format('  %-34s %10s %6s %12s  %-9s %s', p[:name], '-', '-', '-', 'ERROR', "#{e.class}: #{msg}")
   end
 end
 
@@ -440,3 +496,10 @@ puts '  The first line is what this is for: those calls stall the pipeline and'
 puts '  CUMO_SHOW_WARNING will not tell you. Either add the warning or, where'
 puts '  the synchronization is avoidable, remove it.'
 puts '  A SYNC in the meta group is waste with nothing to show for it.'
+puts
+puts '  The block group counts waits rather than timing them: a walk that hands'
+puts '  every element to a Ruby block has to wait between elements, since the'
+puts '  block can queue work on the array being walked. A block that stays off'
+puts '  the device should cost none, and one that writes should cost about one'
+puts '  per element. A count of one there means the walk waits once for the whole'
+puts '  row, which is what made Bit#each and map answer differently from numo.'