cumo 0.7.0 → 0.9.0

data/CHANGELOG.md

@@ -1,3 +1,66 @@
+# 0.9.0 (2026/09/19)
+
+Breaking changes:
+
+* An integer array divided by a zero Ruby number raises `ZeroDivisionError` before the kernel runs rather than after it, so `a.inplace / 0` leaves `a` alone where it wrote over it first. The divisor was on the host all along, so the flag the device raised and the `cudaDeviceSynchronize` that read it back are gone: `Cumo::Int32.new(1, 768)` costs 2.34us for `a / 2` where it cost 9.27, and 2.20 for `a % 2` where it cost 7.88. `2 / a` and an array by an array keep the flag, the divisor really being on the device there (PR #466)
+
+Fixes:
+
+* Fix `mulsum` reducing everything where the receiver is a view that marked its axes with `:sum`, `:reduce` or `:+` and the operands have different dtypes. The marking lives on the array the method is asked of, and `mulsum` answers another class by putting a different array in the receiver's place, so `Cumo::Int8.new(4, 6)[:sum, true].mulsum(a_float)` answered a scalar where the same call with matching dtypes answers a row of six. numo answers the same wrong thing (PR #480)
+* Fix an elementwise operation whose inplace receiver shares memory with an operand that walks it differently, `a.inplace + a.transpose` answering something new on every run: the element a thread reads is one another thread is writing, and nothing orders the two on a device. An operand sitting exactly where the output does is left alone, which is the common `a.inplace + a` (PR #473)
+* Fix `to_binary` handing out whatever lay past the end of a `Cumo::Bit` array whose size is not a multiple of eight. `Cumo::Bit.new(24).fill(1)[0..3].to_binary` answered `[255]` where four bits belong to it, and a view copied first answered the memory pool's contents, so the same expression could answer differently from one run to the next. `marshal_dump` goes through here too (PR #471)
+* Fix an inplace view built from an index array reducing from the wrong rows. The copy that makes such a view contiguous carried `CUMO_NDF_INPLACE`, so for an inplace receiver it answered self and the reduction walked the index-backed view it meant to have copied. `median` and `max_index` answered their own wrong rows the same way, and inplace survives an operation, so `(v.inplace * 2).sum(axis: 2)` reached it too (PR #461)
+* Hold `layer_norm`, `rms_norm` and `softmax` to the same checks. `softmax` measured the result by its element count where the other two compare the shape, so an `allocate` that reshaped either side to the same count left every check true: reshaping the result answered the right values under the wrong shape, and reshaping the receiver normalized along the wrong axis (PR #459)
+
+Changes:
+
+* Add `rms_norm`, which normalizes a row in one kernel rather than the six it is written out of. This is `layer_norm` without the centring, the layer Llama and the models after it normalize with, so it takes no beta. On SFloat `[1, 768]` it goes from 14.2 to 2.9us and `[4096, 768]` from 122.5 to 38.5. No shape is slower (PR #458)
+* Add `silu` to `Cumo::NMath`, which takes one kernel rather than the five it is written out of: 768 elements go from 13.4 to 3.6us. It follows `torch.nn.functional.silu`, answering within one unit in the last place of the true value at the nine double points measured, and matching it at the edges (PR #460)
+* Add `quantize_symmetric`, which quantizes a row to `Cumo::Int8` in one kernel rather than the six it is written out of and hands the scale back. A launch costs about two microseconds whatever it is handed, so a short row pays for those rather than for its bytes (PR #476)
+* Speed up `mulsum`. Operands of different dtypes are read where they already are and converted one element at a time rather than cast into an array of the wider type first, so `SFloat#mulsum(Int8)` at `[4096, 4096]` goes from 384 to 53us. A broadcast operand no longer falls off the fast path, `1000x12x64` against `1000x12x1` over axis 0 going from 13.5 to 4.9us, and the thread layout takes the shorter step of the two operands rather than the first one's, so `DFloat[64,1].mulsum(DFloat[64,1024], axis: 1)` goes from 16.26 to 5.68 whichever order it is written in (PR #479, PR #477, PR #475)
+* Fix every allocation slowing down as the memory pool fragments, the arena having been rebuilt whenever an allocation emptied a bin. GPT-2 124M generating 256 tokens goes from 11.9 to 3.2 seconds in fp16, where the pool ends up with 4035 free chunks against fp32's 71 (PR #474)
+* Range check an NArray index on the device rather than copying the indices to the host and back. `Cumo::SFloat.new(50257, 768)`, `a[idx, true]`: 4096 indices go from 58.2 to 22.2us and 64 from 20.8 to 13.8. Nothing reads managed memory from the host on that path any more, which is what breaks under Ractor (sonots/cumo#180) (PR #470)
+* Copy a transposed view through the shared-memory tile the typed store has used since 0.7.0. `Cumo::SFloat.new(4096, 1024)`: `a.transpose.copy` goes from 338 to 97us, level with `a.transpose.dup` at 102, and `a.transpose.max_index(axis: 1)` from 362 to 107. Every reduction that needs a contiguous operand goes through this copy (PR #468)
+* Reduce a view built from an index array without staging it in a buffer first. Copying a `[6250, 64]` view of a `[100000, 64]` array goes from 13.9 to 7.2us and reducing it from 25.1 to 18.3 (PR #462)
+* Let `store_binary` reach a view it could only read back from. `to_binary` reads any view, so `v.store_binary(v.to_binary)` raised for the same `v` that had just produced the string. A view the bytes cannot move into directly now takes them through an array of its own class and shape (PR #472)
+* Answer `respond_to?` for the methods `Cumo::NMath` dispatches. Every one of them arrives through `method_missing`, so `respond_to?` answered false for `sqrt` and `exp` alike, which code that checks for a feature before calling it reads as absence (PR #463)
+* Take one Ruby method dispatch out of every ndloop call, the argument list always being a plain Array where `rb_obj_dup` took the general path: about 800 fewer retired instructions per operation, 3.4% of the 23,200 an operation takes (PR #465)
+* Tell which array a type check rejected and what it wanted. It said "invalid NArray type (class)", naming neither, so two checks side by side read exactly alike; it now says "gamma must be Cumo::SFloat, not Cumo::HFloat" (PR #478)
+* Say in the README that `reshape` answers a copy where slicing answers a view, priced across four shapes: on a GPU it is an allocation and a copy kernel every call, 58.1us and 12MB for a `4096x768` SFloat against 0.22us and nothing for `reshape!` (PR #457)
+* Show the changelog, source and issue links on the gem page, and require multi-factor authentication to push the gem (PR #456)
+* Share one body between the half and bfloat16 macros so a fix reaches both, 399 duplicated lines where it would have reached one. The compiled code is unchanged: the device SASS is identical (PR #464)
+* Keep the index-array test in one place so copy and store cannot drift apart, 172 generated copies of it becoming one (PR #467)
+* Drop the `Cumo::NMath::DISPATCH` rows no argument can reach. A bare Ruby numeric takes an NArray row, so the `Integer`, `Float` and `Complex` ones were never read: 944 calls over 16 kinds of argument reached them zero times (PR #469)
+
+# 0.8.0 (2026/09/16)
+
+Breaking changes:
+
+* `sort` and `sort_index` with `nan: true` put every NaN last. The host comparator they used answered -1 for a NaN against anything, so the partition carried elements other than the NaN to the wrong side: of 4156 cases over lengths 5 to 129, 2121 came back with the numbers out of order. `nan:` no longer changes the answer, and 1M DFloat costs 0.405 ms rather than 73.0 (PR #447)
+* `conv` and `conv_transpose` refuse a bias that is not one value per output channel. A bias of length 1 was broadcast over every channel and answered as if it had been right, and a bias of two dimensions passed whenever its size fitted (PR #443)
+* `var` and `stddev` answer NaN for a row holding an infinity, where the answer turned on where the infinity sat and how long the row was (PR #430)
+
+Fixes:
+
+* Fix every `dot` after `Cumo::CUDA::Runtime.cudaSetDevice` raising `CUBLAS_STATUS_EXECUTION_FAILED` for the life of the process. Cumo took a context of its own when it loaded, and `cudaSetDevice` binds the device's primary one, so the event inside the cuBLAS handle was left in a context nothing ran in (PR #451)
+* Fix the device memory a rejected kernel launch left in the pool, `rb_raise` leaving by longjmp past the free below it. A sort lost 24MB and a convolution given a bias of the wrong class 5MB, every time; reductions, `softmax`, `layer_norm`, `Cumo::Bit#count`, `where`, `poly` and a store from a Ruby Array lost theirs the same way (PR #439, PR #438, PR #437, PR #436, PR #434, PR #433, PR #432, PR #431)
+* Fix two cuDNN descriptor leaks, one where `batch_norm` raises after building its descriptors and one where a convolution finds no algorithm inside `CUMO_CUDNN_MAX_WORKSPACE_SIZE` (PR #441, PR #440)
+* Fix a `std::bad_alloc` from the convolution algorithm cache killing the process, `cudnn_impl.cpp` being one `extern "C"` block with no handler in it (PR #444)
+* Fix the nine cuDNN methods and `layer_norm` reading the wrong memory when `dup` answers a different class, a different shape or a non-contiguous view; `conv` took a reversed view and read past it (PR #425)
+* Fix the spelling of "propagate" and "position" in the `nan:` documentation on `sort`, `sort_index`, `median`, `argmin`, `argmax`, `min_index` and `max_index` (PR #446)
+
+Changes:
+
+* Add `Cumo::BFloat`, the bfloat16 dtype, aliased `Cumo::BFloat16`. It keeps a float's exponent and gives up mantissa bits for it, so it holds the magnitudes a `Cumo::HFloat` overflows on, which is how published weights are written. Everything the dtype has is generated: the four operations, the comparisons, the reductions, `Cumo::NMath`, `sort`, `rand`, `dot` through cuBLAS and the cuDNN methods, both with a float compute type. `BFloat` and `HFloat` promote to `SFloat` when they meet, neither format containing the other. The accelerated paths want compute capability 8.0 (PR #455, PR #453, PR #452, PR #450)
+* Add `layer_norm`, which normalizes a row in one kernel rather than the nine it is written out of, and `softmax`, which takes one rather than five. On SFloat `[1, 768]` the first goes from 19.9 to 2.9us and the second from 11.0 to 2.2, and `[4096, 768]` from 198.3 to 31.4 and 109.6 to 31.6. No shape is slower (PR #428, PR #426, PR #424, PR #423)
+* Add `gelu` and `gelu_tanh` to `Cumo::NMath`. Two of them because there are two: `gelu` is the error function form PyTorch answers by default, and `gelu_tanh` the approximation GPT-2 and the models after it were trained with. They differ by 4.1e-4 at most, too little to tell apart by eye and far too much to swap under trained weights (PR #427)
+* Take one reciprocal instead of two divides in the moments accumulator, a double divide being what this class of card charges most for; `var` goes from 172.7 to 106.5us in DFloat and `layer_norm` from 179.6 to 138.0 (PR #429)
+* Store a Ruby Array into a strided or indexed view through one buffer held by the loop rather than one a row, a row having cost 2.53 times what the same payload written contiguously does and now costing 1.04 (PR #445)
+* Stop `Cumo::RObject#poly` stopping the whole device once per element. It runs with `CUMO_NO_LOOP`, so ndloop calls its iterator once an element, and there is nothing left there to wait for (PR #435)
+* Say once what `conv` and `conv_transpose` had each grown a copy of, the run struct and the bias block, and build the bias descriptor from a shape rather than lending the narray one (PR #442)
+* Declare the gem's license as MIT. RubyGems reads the gemspec, which said BSD-3-Clause while the `LICENSE.txt` inside the same gem has carried the MIT text from the start (PR #448)
+* Skip the CUDA build matrix for a README, license or benchmark edit no build step opens, and give ccache a ceiling one build fits in (PR #454, PR #449)
+
 # 0.7.0 (2026/09/13)
 
 Breaking changes:

data/README.md

@@ -178,6 +178,44 @@ x += p_dir * alpha     # and consumed there, without crossing the bus
 
 Read the value back once the loop is done, or every k iterations if it has to test something.
 
+### Reshape Copies, Reshape! Does Not
+
+`reshape` answers a copy of the whole array, never a view.
+Slicing answers a view, so the two look alike and are not:
+
+```ruby
+a = Cumo::SFloat.new(4, 6).seq
+a[0..1, true][0, 0] = 77.0   # a view, so a changes
+a.reshape(2, 12)[0, 0] = 99.0  # a copy, so a does not
+```
+
+This is what Numo does too, and numpy is where the expectation comes from: there `reshape` answers a view whenever the strides allow one.
+On a GPU the difference is an allocation and a copy kernel, paid every call.
+`reshape!` changes the receiver in place and costs neither.
+RTX 5070 Ti Laptop, `Cumo::SFloat`, 200 calls a measurement:
+
+```
+shape        reshape     reshape!    the copy allocates
+1x768        5.1 us      0.04 us          3 KB
+1024x768    39.1 us      0.18 us          3 MB
+4096x768    58.1 us      0.22 us         12 MB
+8192x768   142.7 us      0.43 us         24 MB
+```
+
+`reshape!` is host-side bookkeeping, so it stays under a microsecond whatever the array weighs.
+The `reshape` column is the copy, and it grows with the bytes.
+
+The catch is that `reshape!` changes the array everything else is holding.
+It fits a temporary the calling expression owns, and not an argument, an ivar, or anything a cache still points at:
+
+```ruby
+x = a * b            # a temporary nothing else holds
+x.reshape!(t, n, h)  # free
+```
+
+Where the array is not yours to change, the copy is the price of the shape.
+Reach for `reshape!` when a profile says the copies are worth removing, not by default.
+
 ### Ruby Floats In NMath Promote To Double
 
 `Cumo::NMath` picks the module it dispatches to from every argument it is given, and a Ruby `Float` counts as a `DFloat` there.
@@ -222,6 +260,135 @@ Cumo::SFloat::Math.atan2(a, 2.0)   #=> Cumo::SFloat
 
 The 0-dimensional form has no effect under Numo, where `[]` returns a Ruby Float.
 
+### Fused Operations
+
+`layer_norm`, `rms_norm` and `softmax` normalize along the last axis in one kernel each, and `quantize_symmetric` takes it to 8-bit integers in one more.
+Written out of the operators they take nine launches, six, five and six, and a launch costs about two microseconds whatever it is handed, so a short row pays for the launches rather than for its bytes.
+
+```ruby
+y = x.layer_norm(gamma, beta, eps: 1e-5)   # (x - mean) / sqrt(var + eps) * gamma + beta
+z = x.rms_norm(gamma, eps: 1e-5)           # x / sqrt(mean(x * x) + eps) * gamma
+probs = scores.softmax                     # exp(x - max) / sum(exp(x - max))
+xq, scale = x.quantize_symmetric           # scale = max(|x|) / 127, xq = round(x / scale)
+```
+
+`quantize_symmetric` answers a `Cumo::Int8` shaped like self and the scale of every row, which is self's shape without its last axis. `xq * scale[false, :new]` is what the row stood for. A row of zeros has no scale to divide by and answers zero for both, and a row holding an infinity or a NaN answers that in its scale and zeros in the row, since neither is a value 8 bits could carry. The scale comes back in the class the reduction accumulates in, which is `Cumo::SFloat` for `Cumo::HFloat` and `Cumo::BFloat` and self's own otherwise.
+
+On an RTX 5070 Ti Laptop, against the same arithmetic spelled with operators, in microseconds:
+
+```
+layer_norm              SFloat            DFloat            HFloat
+shape              fused  written    fused  written    fused  written
+1 x 768              3.9     30.1      9.7     30.3      3.9     29.8
+256 x 768            6.6     22.5     22.0     35.4      7.8     22.4
+4096 x 768          32.7    177.4    300.7    610.5     28.2    142.9
+1 x 1000000         19.0     59.4    125.4    175.4     22.3     68.5
+
+softmax                 SFloat            DFloat            HFloat
+shape              fused  written    fused  written    fused  written
+1 x 768              4.3     19.8      7.4     26.5      6.3     34.4
+256 x 768            8.8     13.9     23.4     35.1      6.8     18.5
+4096 x 768          49.2    122.7    318.3    549.0     28.2    109.8
+1 x 1000000         27.8     44.0    108.6    148.5     32.1     53.8
+
+rms_norm                SFloat            DFloat            HFloat
+shape              fused  written    fused  written    fused  written
+1 x 768              2.9     14.2      5.8     14.1      3.1     15.1
+256 x 768            4.6     16.7      6.9     20.2      5.2     15.5
+4096 x 768          38.5    122.5    130.8    367.0     19.2     80.8
+1 x 1000000         12.5     35.8     23.7    102.2     12.0     33.2
+```
+
+```
+quantize_symmetric      SFloat            DFloat            HFloat
+shape              fused  written    fused  written    fused  written
+1 x 768              3.7     20.2      6.0     15.7      2.3     11.9
+256 x 768            4.7     15.0     20.2     21.3      4.7     13.0
+4096 x 768          24.4    105.4    289.5    333.3     23.8     80.3
+1 x 1000000         18.8     40.8     97.1     81.6     15.7     37.5
+```
+
+The tables were taken in separate sessions, so read each row against the row beside it and not across the tables.
+
+`quantize_symmetric` in `Cumo::DFloat` is worth less than the others and loses outright on a million elements, whatever shape they are in. The division it does per element is what costs: on this card one row of a million takes 44.0 us to divide in double against 10.8 in single, where the reduction over the same row takes 25.3 and 15.4. Six kernels give that division a kernel of its own to fill the device with, and one kernel leaves it behind the reduction.
+
+All three pay off in every precision, by the most where the row is short enough that the launches were all it was doing, and by the least in double, where the reduction itself costs more than the launches ever did.
+The memory clock on this card steps between 9001 and 11001 MHz under a benchmark this short, and the absolute figures move with it.
+The ratios hold across the steps.
+
+All three normalize along the last axis only.
+Other axes are reachable through `transpose`.
+`layer_norm` divides the variance by the row length and not by one less, which is how the layer is defined and unlike `var`; `rms_norm` divides its mean square the same way.
+All three answer a new array and ignore `inplace!`, and a non-contiguous receiver is copied once rather than walked.
+`gamma` and `beta` must be one-dimensional, as long as the last axis, and of the same class as the receiver.
+
+`rms_norm` is `layer_norm` without the centring, and takes no `beta`, which is the layer Llama and the models after it normalize with.
+The two agree on a row whose mean is already zero and differ on every other row, so they are not interchangeable:
+
+```ruby
+x = Cumo::SFloat[[1.0, 2.0, 3.0, 4.0]]
+ones = Cumo::SFloat.ones(4)
+zeros = Cumo::SFloat.zeros(4)
+x.rms_norm(ones)               #=> [[0.365, 0.730, 1.095, 1.461]]
+x.layer_norm(ones, zeros)      #=> [[-1.342, -0.447, 0.447, 1.342]]
+```
+
+All three accumulate in single precision, so a half row that squares past the 65504 half holds is still answered the way double answers it.
+bfloat16 carries a float's exponent rather than a half's, so a bfloat16 row that squares past what a float holds overflows the accumulator too: `Cumo::BFloat[[-3e20, 3e20, 3e20, -3e20]].rms_norm(ones)` answers zeros where double answers ones.
+
+`softmax` subtracts the row maximum before exponentiating, so a row masked with `-Float::INFINITY` answers zeros rather than `NaN`.
+A row that is entirely `-Float::INFINITY` answers `NaN`, as the written-out form does.
+
+`layer_norm` and `softmax` above, with `gelu_tanh` from the section below, are where a GPT-2 style transformer spends most of its launches.
+Decoding one token of GPT-2 124M written against Numo's API takes 640 of them, and calling these three instead removes 332: 200 for the layer norms, 84 for the activations and 48 for the softmaxes.
+A Llama style one spends them on `rms_norm` and `silu` instead, which is why they are here.
+
+### Two Spellings Of gelu
+
+`Cumo::NMath` answers both definitions of the GELU activation, under names of
+their own rather than a keyword, since a unary NMath function takes none.
+
+| method | formula | matches |
+| --- | --- | --- |
+| `gelu` | `0.5 * x * (1 + erf(x / sqrt(2)))` | PyTorch's default, `approximate='none'` |
+| `gelu_tanh` | `0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x**3)))` | PyTorch's `approximate='tanh'`, and what GPT-2 was trained with |
+
+They differ by at most 4.1e-4, which is small enough to mistake one for the
+other and far too large to swap under a set of trained weights, so pick the one
+the weights were trained with.
+
+Both follow PyTorch at the edges rather than the limit: a large negative `x`
+runs out of significant digits and answers zero, and `-Float::INFINITY` answers
+`NaN`.
+
+### SiLU
+
+`Cumo::NMath.silu` is `x * sigmoid(x)`, the activation Llama and the models after it use where GPT-2 uses `gelu_tanh`.
+It is also called Swish.
+
+```ruby
+Cumo::NMath.silu(x)   # x / (1 + exp(-x)), in one kernel rather than five
+```
+
+Written out of the operators it costs five launches, and one kernel runs 1.1x to 4.3x faster on an RTX 5070 Ti Laptop, in microseconds:
+
+```
+silu                    SFloat            DFloat            HFloat
+elements           fused  written    fused  written    fused  written
+768                  3.6     13.4      3.6      9.5      3.4     14.3
+786432               7.0     25.9     90.5    102.4      5.2     19.2
+16777216           353.2   1312.9   1984.7   3258.8    199.6    851.8
+```
+
+The three rows are three regimes rather than one curve: 768 elements pay for the launches, 786432 fit in L2 and move faster than this card reads from memory, and 16777216 are what it costs from DRAM.
+
+It follows `torch.nn.functional.silu`, answering within one unit in the last place of the true value at the nine double points measured, as torch does.
+At the edges it matches torch exactly: `-Float::INFINITY` answers `NaN`, since that is what infinity times zero is, and a large negative `x` answers a signed zero.
+
+Where that zero starts depends on the type, and not on `exp` alone.
+Single and bfloat16 reach it at -89, where `exp(-x)` passes what a float holds, and double never does.
+Half reaches it at -21, because the round back to half gets there first.
+
 ### Half Precision
 
 `Cumo::HFloat`, also reachable as `Cumo::Float16`, holds IEEE binary16: one sign bit, five of exponent and ten of mantissa.
@@ -262,6 +429,7 @@ That distinction is what makes half usable in a transformer's layer normalizatio
 Squaring overflows at 256, since 256 squared is already past the top, and a residual stream with one outlier feature reaches thousands.
 Writing the normalization out as `((x - mean) ** 2).mean` builds those squares as a half array and the answer is `Infinity`, while `x.var(axis: 1)` keeps them in its single-precision accumulator and answers 10208.0 against a single-precision 10209.
 `var` saturates only if the variance itself is out of range, which is a far higher bar than any one deviation being over 256, but it is still a bar.
+An input over 65504 is a different matter: it becomes an infinite element on the way in, and `var` and `stddev` answer `NaN` for a row holding one.
 The quantity to check against 65504 is the largest variance a layer produces, not the largest activation in it.
 A single outlier of size d among n values contributes only d squared over n to the variance, so a row 768 wide divides it by 768: an activation of 3000 squares to nine million but raises the variance of its row by about twelve thousand.
 An activation that looks safe therefore says nothing about whether `var` overflows, in either direction.
@@ -342,6 +510,83 @@ Keeping the running statistics in single precision is what the arithmetic wants
 Gradients are the other place half runs out of room.
 Values below `Cumo::HFloat::MIN` of 6.1e-05 fall into the subnormals and then to zero, which is what loss scaling in a training loop exists to prevent.
 
+### Brain Float
+
+`Cumo::BFloat`, also reachable as `Cumo::BFloat16`, holds bfloat16: one sign bit, eight of exponent and seven of mantissa.
+It is the other sixteen-bit float, and it differs from `Cumo::HFloat` in where those bits went.
+A bfloat16 has the exponent of a `Cumo::SFloat` and three fewer mantissa bits than a binary16, so it reaches everything a single-precision value reaches and resolves less of it:
+
+```ruby
+Cumo::BFloat[1.0e38]   #=> 9.969209968386869e+37
+Cumo::HFloat[1.0e38]   #=> Infinity
+
+Cumo::BFloat[257.0]    #=> 256.0
+Cumo::HFloat[257.0]    #=> 257.0
+```
+
+Integers are exact only to 256, against 2048 for a binary16, and the largest finite value is 3.3895314e+38 rather than 65504.
+That trade is why weights are published in it: a tensor that came out of training keeps its magnitudes, and the bits it loses are ones a trained weight does not carry.
+
+The two do not contain each other, so an expression mixing them promotes to `Cumo::SFloat`, which does:
+
+```ruby
+(Cumo::BFloat[1.0] + Cumo::HFloat[1.0]).class   #=> Cumo::SFloat
+```
+
+Everything else promotes as `Cumo::SFloat` does, so an integer array or a Ruby Float mixed in stays bfloat16 while anything wider takes over.
+
+`layer_norm`, `rms_norm`, `softmax`, `silu` and both spellings of `gelu` take it, and so do the reductions, `sort`, `median`, `cumsum`, `rand` and `dot`.
+The cuDNN methods take it too: `conv`, `conv_transpose`, `conv_grad_w`, `max_pool`, `avg_pool` and the three batch norm entries.
+
+Reductions accumulate in single precision and round once at the end, exactly as they do for half, so a sum passes 256 without stopping there:
+
+```ruby
+Cumo::BFloat.new(40_000).fill(1.0).sum   #=> 39936.0
+```
+
+That 39936 is 40000 rounded to the nearest bfloat16, whose values are 256 apart up there; a bfloat16 accumulator would have answered 256.
+`sum`, `mean`, `var`, `stddev`, `rms`, `mulsum`, `dot`, `gemm` and `cumsum` all widen this way, and `prod` and `cumprod` do not, for the same reasons given under Half Precision.
+
+The wider exponent moves where the saturation described there bites.
+A variance is far from 3.4e+38, so `var` does not saturate on any input a network produces, and squaring is safe to 1.8e+19 rather than to 256.
+What replaces it is the mantissa: a bfloat16 resolves about three decimal digits, so a normalization computes its statistics well enough while the values it writes back carry that resolution and no more.
+
+#### What bfloat16 is faster at
+
+`gemm` reaches the tensor cores and lands on the same throughput as half. Square matrices on an RTX 5070 Ti Laptop, median of five runs each, one process per case:
+
+```
+              BFloat              HFloat              SFloat
+1024x1024     0.044 ms  48.9 TF   0.044 ms  48.5 TF   0.144 ms  14.9 TF
+2048x2048     0.298 ms  57.6 TF   0.303 ms  56.7 TF   1.404 ms  12.2 TF
+4096x4096     2.839 ms  48.4 TF   2.898 ms  47.4 TF   9.540 ms  14.4 TF
+```
+
+The odd-column penalty is the same as half's and for the same reason, a row starting on a two-byte boundary.
+1024x1024 times 1024x1024, with one dimension made odd at a time:
+
+```
+             all even   M odd      K odd      N odd
+BFloat        48.9 TF   48.7 TF    23.5 TF    23.0 TF
+HFloat        48.5 TF      -       24.0 TF    23.1 TF
+```
+
+So the choice between the two sixteen-bit types is about range and resolution, not speed.
+Take bfloat16 where the magnitudes came from somewhere else and binary16 where three more mantissa bits are worth having.
+
+#### The accelerated paths want Ampere
+
+cuBLAS supports `CUDA_R_16BF`, which is what `gemm` and `dot` reach, from compute capability 8.0.
+Storing, casting and elementwise arithmetic have no such floor, because every operation is computed in single precision and rounded back, and they build and run wherever cumo does.
+A `dot` on a pre-Ampere card is the one to expect trouble from.
+**This is not measured here**: the only GPU these numbers came from is a Blackwell one, and the requirement is read from cuBLAS's documentation rather than reproduced.
+
+cuDNN reaches bfloat16 as `CUDNN_DATA_BFLOAT16`, and its own bfloat16 kernels want Ampere for the same reason cuBLAS does.
+A convolution is given `CUDNN_DATA_FLOAT` to accumulate in, so it passes 256 the way a reduction does.
+The batch norm parameters are `Cumo::SFloat`, the same as they are for `Cumo::HFloat`.
+The workspace ceiling matters here as much as it does for half: left at the default 8MB, a bfloat16 convolution is no faster than a single-precision one.
+See [Raise the cuDNN workspace ceiling](#raise-the-cudnn-workspace-ceiling).
+
 ### Select a GPU device ID
 
 Set the `CUDA_VISIBLE_DEVICES=id` environment variable, or

data/bench/cumo_probe.rb

@@ -312,7 +312,7 @@ op(:scalar, 'scalar_left') { |c| c.scalar - c.sc.inplace }
 
 # NMath
 %w[sqrt exp exp2 exp10 expm1 log log2 log10 log1p sin cos tan asin acos atan
-   sinh cosh tanh asinh acosh atanh cbrt erf erfc sinc].each do |f|
+   sinh cosh tanh asinh acosh atanh cbrt erf erfc sinc gelu gelu_tanh silu].each do |f|
   op(:math, f, :float) { |c| XM::NMath.send(f, c.usc.inplace) }
 end
 op(:math, 'atan2', :float) { |c| XM::NMath.atan2(c.sc.inplace, c.b) }

data/cumo.gemspec

@@ -14,7 +14,13 @@ Gem::Specification.new do |spec|
   spec.summary       = %q{Cumo is CUDA aware numerical library whose interface is highly compatible with Ruby Numo}
   spec.description   = %q{Cumo is CUDA aware numerical library whose interface is highly compatible with Ruby Numo.}
   spec.homepage      = "https://github.com/sonots/cumo"
-  spec.license       = "BSD-3-Clause"
+  spec.license       = "MIT"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = spec.homepage
+  spec.metadata["changelog_uri"] = "https://github.com/sonots/cumo/blob/master/CHANGELOG.md"
+  spec.metadata["bug_tracker_uri"] = "https://github.com/sonots/cumo/issues"
+  spec.metadata["rubygems_mfa_required"] = "true"
 
   spec.required_ruby_version = ">= 3.0.0"
 

data/ext/cumo/cuda/cudnn.c

@@ -53,6 +53,24 @@ BAD:
             env, cudnn_max_workspace_size);
 }
 
+VALUE
+cumo_cuda_cudnn_release_conv_held(VALUE held)
+{
+    cumo_cuda_cudnn_conv_held_t *h = (cumo_cuda_cudnn_conv_held_t*)held;
+
+    if (h->x_desc) { cudnnDestroyTensorDescriptor(h->x_desc); h->x_desc = 0; }
+    if (h->y_desc) { cudnnDestroyTensorDescriptor(h->y_desc); h->y_desc = 0; }
+    if (h->b_desc) { cudnnDestroyTensorDescriptor(h->b_desc); h->b_desc = 0; }
+    if (h->w_desc) { cudnnDestroyFilterDescriptor(h->w_desc); h->w_desc = 0; }
+    if (h->conv_desc) { cudnnDestroyConvolutionDescriptor(h->conv_desc); h->conv_desc = 0; }
+    // cuDNN is still reading the workspace, so the wait is the caller's to
+    // report. A free that raises from here would replace the exception being
+    // carried out, so it goes back the way an unwinding path has to.
+    cumo_cuda_runtime_return_scratch(h->workspace, h->workspace != NULL, &h->wait_status);
+    h->workspace = NULL;
+    return Qnil;
+}
+
 // Lazily initialize cudnn handle, and cache it
 cudnnHandle_t
 cumo_cuda_cudnn_handle()

data/ext/cumo/cuda/cudnn_impl.cpp

@@ -67,17 +67,11 @@ cumo_cuda_cudnn_GetConvTransposeOutDim(
 }
 
 cudnnStatus_t
-cumo_cuda_cudnn_CreateTensorDescriptor(
+cumo_cuda_cudnn_CreateTensorDescriptorFromShape(
         cudnnTensorDescriptor_t *desc,
-        VALUE a, cudnnDataType_t cudnn_dtype) {
+        int ndim, size_t *shape, cudnnDataType_t cudnn_dtype) {
     cudnnStatus_t status = CUDNN_STATUS_SUCCESS;
-    cumo_narray_t *na;
-    CumoGetNArray(a, na);
-    int ndim = (int)(na->ndim);
-    size_t *shape = na->shape;
 
-    // only the shape is read: the descriptor is NCHW and the caller passes the
-    // contiguous copy, so a non-contiguous a still describes what is read
     status = cudnnCreateTensorDescriptor(desc);
     if (status != CUDNN_STATUS_SUCCESS) return status;
 
@@ -120,6 +114,47 @@ cumo_cuda_cudnn_CreateTensorDescriptor(
     return status;
 }
 
+// Only the shape is read: the descriptor is NCHW and the caller passes the
+// contiguous copy, so a non-contiguous a still describes what is read.
+cudnnStatus_t
+cumo_cuda_cudnn_CreateTensorDescriptor(
+        cudnnTensorDescriptor_t *desc,
+        VALUE a, cudnnDataType_t cudnn_dtype) {
+    cumo_narray_t *na;
+    CumoGetNArray(a, na);
+    return cumo_cuda_cudnn_CreateTensorDescriptorFromShape(desc, (int)(na->ndim), na->shape, cudnn_dtype);
+}
+
+// The bias is one value a channel, and cuDNN broadcasts it from a descriptor
+// shaped like the output. Shared because conv and conv_transpose add it alike.
+cudnnStatus_t
+cumo_cuda_cudnn_AddBias(cumo_cuda_cudnn_conv_run_t *r, cudnnDataType_t dtype, const void *one) {
+    size_t bias_shape[CUMO_NA_MAX_DIMENSION];
+    cumo_narray_t *nb;
+    cudnnStatus_t status;
+
+    if (!RTEST(r->b_cont)) return CUDNN_STATUS_SUCCESS;
+
+    CumoGetNArray(r->b_cont, nb);
+    bias_shape[0] = 1;
+    bias_shape[1] = nb->size;
+    for (size_t i = 0; i < r->ndim; ++i) {
+        bias_shape[i + 2] = 1;
+    }
+    status = cumo_cuda_cudnn_CreateTensorDescriptorFromShape(
+            &r->held.b_desc, (int)(r->ndim + 2), bias_shape, dtype);
+    if (status != CUDNN_STATUS_SUCCESS) return status;
+
+    return cudnnAddTensor(
+            r->handle,
+            one,
+            r->held.b_desc,
+            (void*)r->b_cont_ptr,
+            one,
+            r->held.y_desc,
+            (void*)r->y_ptr);
+}
+
 cudnnStatus_t
 cumo_cuda_cudnn_CreateFilterDescriptor(
         cudnnFilterDescriptor_t *desc,
@@ -322,6 +357,17 @@ struct AlgoCacheEntry {
     size_t memory;
     cudnnMathType_t math_type;
 };
+
+// Nothing may reach the C caller: it has no handler, so an escaping exception
+// is std::terminate. The cache only saves a later search, so one that cannot
+// grow leaves the answer this call already has.
+template <typename Map>
+void StoreAlgoCache(Map& map, const AlgoCacheKey& key, const typename Map::mapped_type& entry) {
+    try {
+        map[key] = entry;
+    } catch (...) {
+    }
+}
 }
 
 using FwdAlgoCacheMap = std::unordered_map<AlgoCacheKey, AlgoCacheEntry<cudnnConvolutionFwdAlgo_t>, AlgoCacheKeyHash>;
@@ -416,7 +462,7 @@ cumo_cuda_cudnn_FindConvolutionForwardAlgorithm(
                 perf_result,
                 (void*)workspace,
                 max_workspace_size);
-    cumo_cuda_runtime_free(workspace);
+    cumo_cuda_runtime_return_scratch(workspace, 1, NULL);
     if (status != CUDNN_STATUS_SUCCESS) return status;
     // A search that answers success with nothing to report would leave
     // perf_result untouched, and its algo and math type are used below.
@@ -424,7 +470,7 @@ cumo_cuda_cudnn_FindConvolutionForwardAlgorithm(
     if (perf_result->status != CUDNN_STATUS_SUCCESS) return perf_result->status;
 
     // TODO: thread-safe
-    algo_cache_map[key] = {perf_result->algo, perf_result->memory, perf_result->mathType};
+    StoreAlgoCache(algo_cache_map, key, {perf_result->algo, perf_result->memory, perf_result->mathType});
     return status;
 }
 
@@ -487,7 +533,7 @@ cumo_cuda_cudnn_FindConvolutionBackwardDataAlgorithm(
                 perf_result,
                 (void*)workspace,
                 max_workspace_size);
-    cumo_cuda_runtime_free(workspace);
+    cumo_cuda_runtime_return_scratch(workspace, 1, NULL);
     if (status != CUDNN_STATUS_SUCCESS) return status;
     // A search that answers success with nothing to report would leave
     // perf_result untouched, and its algo and math type are used below.
@@ -495,7 +541,7 @@ cumo_cuda_cudnn_FindConvolutionBackwardDataAlgorithm(
     if (perf_result->status != CUDNN_STATUS_SUCCESS) return perf_result->status;
 
     // TODO: thread-safe
-    algo_cache_map[key] = {perf_result->algo, perf_result->memory, perf_result->mathType};
+    StoreAlgoCache(algo_cache_map, key, {perf_result->algo, perf_result->memory, perf_result->mathType});
     return status;
 }
 
@@ -558,7 +604,7 @@ cumo_cuda_cudnn_FindConvolutionBackwardFilterAlgorithm(
                 perf_result,
                 (void*)workspace,
                 max_workspace_size);
-    cumo_cuda_runtime_free(workspace);
+    cumo_cuda_runtime_return_scratch(workspace, 1, NULL);
     if (status != CUDNN_STATUS_SUCCESS) return status;
     // A search that answers success with nothing to report would leave
     // perf_result untouched, and its algo and math type are used below.
@@ -566,7 +612,7 @@ cumo_cuda_cudnn_FindConvolutionBackwardFilterAlgorithm(
     if (perf_result->status != CUDNN_STATUS_SUCCESS) return perf_result->status;
 
     // TODO: thread-safe
-    algo_cache_map[key] = {perf_result->algo, perf_result->memory, perf_result->mathType};
+    StoreAlgoCache(algo_cache_map, key, {perf_result->algo, perf_result->memory, perf_result->mathType});
     return status;
 }
 

data/ext/cumo/cuda/driver.c

@@ -449,15 +449,17 @@ Init_cumo_cuda_driver()
 
     // A driver API call needs a current context, and the runtime API only
     // creates its primary one once an array operation happens, so this covers
-    // the gap in between. Losing it is not fatal -- everything but a driver
-    // call made before any array operation still works -- so a device that
-    // refuses a context is left for that call to report. cuDeviceGet leaves
-    // cuDevice untouched when it fails, hence the guard.
+    // the gap in between. It has to be the primary context and not one of our
+    // own: cudaSetDevice binds the primary one, so anything built in a
+    // different context stops working the moment a caller sets the device,
+    // including the events a cuBLAS handle keeps inside itself.
+    // Losing it is not fatal -- everything but a driver call made before any
+    // array operation still works -- so a device that refuses a context is
+    // left for that call to report. cuDeviceGet leaves cuDevice untouched
+    // when it fails, hence the guard.
     if (cuDeviceGet(&cuDevice, 0) == CUDA_SUCCESS) {
-#if defined(CUDA_VERSION) && CUDA_VERSION >= 13000
-        cuCtxCreate(&context, NULL, 0, cuDevice);
-#else
-        cuCtxCreate(&context, 0, cuDevice);
-#endif
+        if (cuDevicePrimaryCtxRetain(&context, cuDevice) == CUDA_SUCCESS) {
+            cuCtxSetCurrent(context);
+        }
     }
 }