cumo 0.9.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0f27ebdaa195fa967c144d6178ad3f9fbeca551162f8edacac205d5edd5d0fb6
-  data.tar.gz: 3d87faa55ab7c4c0e52bee3afdb0116d9d463fa778c408a46e200bff0cabdf81
+  metadata.gz: 78444d61470bd2cf534dfadba1facef1d92785cf85db498ca14d893242532591
+  data.tar.gz: 7bb662f78bd6e130c5a93db7fe6ac44fc2dad14f58f3788de48c04f7830197b4
 SHA512:
-  metadata.gz: 4b498a760e59725c15aa3e651ecbada01e52517833fd3132a860dbfe459e354f4afba47207cc33f2c94963f58f19a8a1d7e52ce722e51065876d637c8ddfc94b
-  data.tar.gz: bbda42b2ef7ce7ea882982a015bec06ff058ad42a2b72e4c8953e75facfcb64657c8da4b69d63666cf4c06d3c265dc7e0acf54c431450f181b39cbeebed9560e
+  metadata.gz: 46d001da3c8c82e4f835fe6643cec1a1f5a4b28864283286a52f98887d975673914ee475ecd90d97042e5c16e45c525060d595ecd8301a427fed5060a8efc7b0
+  data.tar.gz: ae0052bea91487469a2dd6052c08f9bb8c74799447ea165d189babc88ccd94b7c686e50d1ff69f1df75ec2d20986c7113ff2d3b949205c4ac72132f7b8d468af

data/CHANGELOG.md

@@ -1,3 +1,54 @@
+# 0.10.0 (2026/09/22)
+
+Breaking changes:
+
+* `CUMO_COMPATIBLE_MODE`, `CUMO_SHOW_WARNING`, `CUMO_SHOW_WARNING_ONCE` and `CUMO_MEMORY_POOL` read `off`, `false` and `no` as a no, in any case, where anything but `OFF`, `0` and `NO` used to turn the flag on without a word. A value they do not recognise keeps the flag's default and warns, and an empty value reads as unset (PR #514)
+* An empty array keeps its shape through everything that goes through ndloop: `Cumo::Int32.new(0, 3).copy` and `(a + 1)` answer `[0, 3]` where they answered `[0, 0]`. `to_a` nests down to the first zero and no further, the way numpy's `tolist` does, so `(3, 0)` is `[[], [], []]` and `(0, 3, 4)` is `[]` (PR #513)
+* `Marshal.dump` of a subclass of `Cumo::RObject` writes its objects, where it wrote the raw addresses the buffer holds. Data the old dump wrote for such a subclass no longer loads, and what it held was addresses from a process that has since exited (PR #509)
+
+Fixes:
+
+* Fix `all?(axis: [])` answering a zero-dimensional `Cumo::Bit`, which is truthy whatever it holds, for an array it is false for. Whether an axis was named is now read off the reduction rather than off the list (PR #512)
+* Fix `all?`, `any?` and `none?` answering a zero-dimensional `Cumo::Bit` when a default keyword such as `axis: nil` or `keepdims: false` is written out, so `Cumo::Bit[1, 1, 0].all?(axis: nil)` read as true (PR #510)
+* Fix an axis reduction of an empty `Cumo::Bit` answering an empty array: `Cumo::Bit.new(0, 3).all?(axis: 0)` answers `[1, 1, 1]` and `count_true(axis: 0)` answers `[0, 0, 0]` (PR #508)
+* Fix the five `Cumo::Bit` reductions answering false, or an Integer, for an empty array before reading their arguments. `Cumo::Bit[].all?` is true and `any?` false, as Ruby's are, `count_true` answers a `UInt64`, and an axis that is not there raises `DimensionError` (PR #507)
+* Fix `CastError` for a subclass of a dtype, which broke `store`, arithmetic and the reductions, and put an element of one into an RObject array as one object repeated (PR #506)
+* Fix `Device#with` running its block on the device it was already on, whichever device it was given (PR #499)
+* Fix `reshape` writing the new shape over whatever the receiver's `dup` answered, which could leave an array claiming a size its buffer cannot hold. The answer `dup` gives has to be the same class, own its data and not be frozen (PR #494)
+* Fix `gemm` walking a copy of a non-contiguous operand with the original's dimensions when `dup` answered something else (PR #493)
+* Fix `marshal_dump` and `to_binary` handing out memory past the end of the array when a view's `dup` answered another shape, class or layout (PR #492)
+* Fix `median` and `sort` ignoring the axes a view marked with `:sum`, so `f[:sum, true].median` answered the whole array. `median` keeps its receiver rather than standing its copy in its place, and no longer copies an operand it only reads, so the pool holds less afterwards (PR #489, PR #488, PR #487)
+* Fix `Cumo::Bit#copy` answering its own receiver when the receiver is inplace, which let a write to the copy reach the original and gave every reduction that copies an index-backed view first the wrong rows (PR #486)
+* Fix the interpreter abort when `all?` or `any?` reduces a `[:sum, true]` view (PR #485)
+
+Changes:
+
+* `compile_with_cache` answers the same module for the same source, options and device for the rest of the process, so it can be called wherever the kernel is launched: a repeated call costs 0.0009 ms where it read the cubin back and loaded a new module for 0.033 ms, nine times the launch itself. A module nobody holds any more is unloaded when Ruby collects it, and `Cumo::CUDA::Compiler.clear_modules` makes the next call read the disk cache or compile again (PR #531)
+* The ports of the CuPy examples live in `examples/`, one Ruby file per original, with `backend.rb` running the CPU side on Numo and the GPU side on Cumo under `GPU=1`. The gem leaves them out (PR #530)
+* Drop a dead first definition of `Compiler#save_cache` that `ruby -W` warned about (PR #529)
+* Reading a result back from the device copies the range it needs into pinned host memory rather than reading managed memory in place, which faulted the page over and back once per read where a small block shares its page with others: `Float(r.sum)` takes 0.012 ms where it took 0.885, `to_a` of eight elements 0.016 ms where it took 0.97, and `to_binary` of 8 MB 2.3 ms where it took 7.6. `to_a`, `each`, `each_with_index`, `map_with_index`, `inspect`, `format`, `extract_cpu`, `aref_cpu`, `to_binary` and the RObject marshal all take the copy, and a block that writes the array it is walking still sees what it wrote (PR #528)
+* `rand` and `rand_norm` fill an array whose last axis has length one in one kernel launch rather than one per row: a `[1048576, 1]` column takes 0.018 ms where it took 1799. A seed produces the same values as before whatever the layout (PR #527)
+* A module answers a template kernel such as `kernel<float>` by its name expression: `NVRTCProgram.new` and `compile_with_cache` take `name_expressions:`, the mangled names are cached beside the cubin, and `Module#get_function` looks an expression up before asking the driver. Arrays of structs are arrays whose trailing axes are the fields, and a struct by value is a packed String. The cubin cache had been writing empty files where `/tmp` and the cache directory are different file systems, so every `compile_with_cache` compiled again; it now writes next to its place (PR #526)
+* Add `Cumo::CUDA::PinnedMemory`, a page-locked host buffer read and written as bytes, and `NArray#set` and `NArray#get`, which copy through it asynchronously on the current stream or on `stream:` the way `cupy.ndarray.set` and `get` do. A copy in flight keeps both ends until it is done (PR #525)
+* Add `Cumo::CUDA::Stream` and `Cumo::CUDA::Event`, after `cupy.cuda`: `Stream.new`, `with`, `use`, `synchronize`, `record`, `wait_event`, `Stream.null`, `Stream.current`, `Event.new`, `record`, `synchronize`, `done?` and `Cumo::CUDA.get_elapsed_time`, and `Function#launch` takes `stream:`. `Stream#with` orders the block after what the previous stream had queued and waits for the stream on the way out (PR #524)
+* Every kernel, copy, cuBLAS and cuDNN call, thrust and CUB call goes to the current stream of the thread, which is the null stream until a `Stream` is used, so a stream a caller picks reaches all of it. `Runtime.cudaStreamCreateWithFlags`, `cudaStreamDestroy`, `cudaStreamSynchronize` and `current_stream=` are bound (PR #523)
+* Add `Runtime.cudaDeviceCanAccessPeer` and `Device#can_access_peer?`. A CUDA error raised for a device that is not there no longer stays as the runtime's last error for the next kernel launch to read as its own (PR #521)
+* The CI ccache ceiling is 2 GB, since one build no longer fit in 1 GB and the CUDA 12 jobs compiled again what had been evicted (PR #522)
+* Add `Cumo::CUDA::ReductionKernel`, a reduction written as a map, a reduce and a post expression with an identity, called with `axis:` and `keepdims:`, after CuPy's. This is the last half of #106 (PR #520)
+* Add `Cumo::CUDA::ElementwiseKernel`, a kernel written as one expression over a parameter list, with one-letter types resolved from the arguments and broadcasting, compiled once per set of types, after CuPy's. Bit, half precision and complex arguments are refused for now (PR #519)
+* A kernel compiled with NVRTC can be launched: `Module#get_function` answers a `Cumo::CUDA::Function` whose `launch` takes the arguments, the grid, the block and the dynamic shared memory. An NArray hands over its device pointer, an Integer goes as a `long long`, a Float as a `double`, and a String as its bytes. On CUDA 12.4 or later the count and the sizes of the arguments are checked against the kernel before the launch (PR #518, PR #500)
+* `CUMO_ALLOW_TF32` puts single precision `gemm` on the tensor cores as well as convolutions: a `[4096, 4096]` SFloat gemm goes from 17.8 to 28.4 TFLOP/s, and its answer moves from 4e-07 to 3e-04 of a double precision reference. Off, which is the default, answers bit for bit what it did. The flag is read in `cumo.c` as `Cumo.allow_tf32?` rather than in `cudnn.c`, so a build without cuDNN has it too (PR #517, PR #516)
+* Every dtype's `gemm` goes through `cublasGemmStridedBatchedEx`, the entry point that can name a compute type. The answers are the same bytes, and no change in speed could be told apart from the noise (PR #515)
+* `all?`, `any?` and `none?` on an empty `Cumo::Bit` with no axis named answer the identity without reserving anything or waiting on the device: 0.045 us a call where it took 12.3 and 512 bytes (PR #511)
+* `sort_index`, and `sort` and `median` over a view, address by rank rather than through the run-time accessor: `sort_index` over a 4096 by 1024 SFloat takes 2.2 ms where it took 5.6, and an inplace `sort` over a column slice 2.0 ms where it took 6.5 (PR #505)
+* `cumsum` and `cumprod` along an axis send every row to one scan keyed on its row rather than starting a scan per row, and a view no single stride reaches is gathered by a kernel that addresses by rank: over a `[512, 2048]` SFloat a column slice takes 25.7 us where 0.9.0 took 50.7, and a transposed view 45.2 us where it took 308.3. A reversed view is read through its stride and costs what a flat one does (PR #504, PR #503, PR #498, PR #496)
+* The default cuDNN workspace ceiling is 128 MB rather than 8 MB, which reaches the faster convolution algorithms: the twenty convolutions of a ResNet-18 forward pass at batch 16 take 5.23 ms where they took 9.80, and a half precision convolution reaches its tensor core algorithms. The search reserves the whole ceiling while it runs, so the peak grows with it and goes back to the pool afterwards (PR #502)
+* A single precision convolution stays off the tensor cores unless `CUMO_ALLOW_TF32` says yes, where cuDNN's default let the algorithm search move it there and round the operands to a 10 bit significand (PR #501)
+* Say in the README that `contiguous?` answers true for a view whose length-one axis Numo counts as breaking contiguity, that copying a transposed operand beats handing it to cuBLAS as it is for most shapes, what the memory numbers of the pool and of `nvidia-smi` mean, where a bfloat16 integer index goes past 256, and that `quantize_symmetric` breaks ties away from zero where numpy and torch round to even (PR #497, PR #495, PR #491, PR #484)
+* Say what routing `silu` through `sigmoid` would change, rather than a fraction that no dtype matches (PR #483)
+* Add `Cumo::NMath.sigmoid`, which keeps a value where `1 / (1 + exp(-x))` written out gives out: single reaches zero at -104 rather than -89, double at -745 rather than -710 (PR #482)
+* Add `Cumo::NMath.softplus`, which does not overflow where `log(1 + exp(x))` written out does, and answers `x` to the last bit past 11.09 in half and 88.7 in single (PR #481)
+
 # 0.9.0 (2026/09/19)
 
 Breaking changes:

data/README.md

@@ -121,7 +121,7 @@ a[0] < 1.0                  #=> Cumo::Bit#shape=[] holding 0
 `assert_operator(a[0], :<, 1.0)` passes for the same reason, so a test suite written for Numo can stay green against Cumo while asserting nothing.
 Read the value back to the host before branching on it, or run under `compatible_mode`.
 
-Set the `CUMO_COMPATIBLE_MODE` environment variable to `ON` to force Numo NArray compatibility (for worse performance).
+Set the `CUMO_COMPATIBLE_MODE` environment variable to `ON` to force Numo NArray compatibility (for worse performance). Every such flag takes `1`, `on`, `yes` or `true` for a yes and `0`, `off`, `no` or `false` for a no, in any case; anything else keeps the default and warns.
 Running a Numo test suite that way keeps its assertions meaningful.
 
 You may enable or disable `compatible_mode` as:
@@ -149,6 +149,36 @@ Float(a.sum)          #=> 7.0 in either mode
 They are methods on an NArray, so chaining one onto a result that `compatible_mode` has already turned into a Ruby object, as in `a.sum.extract_cpu`, raises `NoMethodError` while the mode is on.
 `Kernel#Float` and `Kernel#Integer` read either representation, and read Numo's too, so they are what code that runs against both libraries wants.
 
+#### A Length-One Axis Does Not Break Contiguity
+
+`contiguous?` answers true for some views Numo calls false.
+An axis of length one is only ever indexed at zero, so whatever stride it carries is multiplied by zero and never moves the pointer.
+Cumo leaves such an axis out of the chain it walks; Numo does not, and calls the view strided because of an axis that cannot stride.
+
+```ruby
+a = Cumo::DFloat.new(1, 4).seq
+a[true, 0...2].contiguous?   #=> true, where Numo gives false
+a[true, 0...2].to_a          #=> [[0.0, 1.0]], the same either way
+
+Cumo::DFloat.new(2, 4).seq[true, 0...2].contiguous?   #=> false in both
+```
+
+The elements the view holds are the same in either library.
+What changes is who is willing to read them where they lie:
+
+* `reshape!` is accepted on these views, where Numo raises
+* `dot`, `gemm`, `conv` and the rest take them as they are, instead of copying them into a contiguous array first
+
+A three-dimensional slice behaves the same way, and so does the transpose of a single row:
+
+```ruby
+Cumo::DFloat.new(1, 1, 4).seq[true, true, 0...2].contiguous?   #=> true
+Cumo::DFloat.new(1, 3).seq.transpose.contiguous?               #=> true
+```
+
+Code that runs against both libraries should not read `contiguous?` and expect the same answer.
+Where it wants a contiguous array it can ask for one, since `dup` answers one in either library.
+
 ### Keeping Scalars On The Device
 
 The 0-dimensional return is what lets an iterative loop stay on the GPU.
@@ -178,6 +208,10 @@ x += p_dir * alpha     # and consumed there, without crossing the bus
 
 Read the value back once the loop is done, or every k iterations if it has to test something.
 
+A read costs the wait and a copy of the block it needs into pinned host memory, a few microseconds for a scalar.
+Reading managed memory from the host directly would fault its page over instead, and a small block shares a page with other live blocks that the next kernel touches, so a fresh scalar cost 0.8 ms that way on the machine above, forty times what the copy costs.
+The reads that answer values, from `Float(x)` and `to_a` to `each` and `inspect`, take the copy.
+
 ### Reshape Copies, Reshape! Does Not
 
 `reshape` answers a copy of the whole array, never a view.
@@ -260,6 +294,39 @@ Cumo::SFloat::Math.atan2(a, 2.0)   #=> Cumo::SFloat
 
 The 0-dimensional form has no effect under Numo, where `[]` returns a Ruby Float.
 
+### A Transposed Operand Goes To cuBLAS As It Is
+
+`dot` and `gemm` hand a transposed operand to cuBLAS with its transpose flag set, rather than copying it into the layout cuBLAS reads fastest.
+That saves the copy and the memory it needs.
+What it costs is the kernel cuBLAS then picks, which for most shapes is slower than the one it picks for an operand already laid out its way.
+
+RTX 5070 Ti Laptop, `Cumo::SFloat`, `q[M,K].dot(k[N,K].transpose)`, medians of nine rounds:
+
+```
+M     K     N       M*K      as it is   copied first   the copy
+1     64    1500         64      5.6 us       11.4 us      2.8 us
+512   64    512      32,768      8.5         11.2         2.9
+1500  64    1500     96,000     33.0         29.3         3.0
+512   256   512     131,072     21.8         20.7         2.8
+4096  64    4096    262,144    229.1        179.3         4.2
+512   768   512     393,216     54.0         39.2         3.3
+256   3072  768     786,432    131.1        103.0        12.5
+```
+
+The copy weighs `N * K`, and the faster kernel it buys is worth `M * N * K`, so `M` is what decides.
+A matrix-vector product, where `M` is one, is the clearest case against copying: it takes twice as long that way.
+Past an `M * K` of roughly fifty thousand on this card the copy starts paying for itself, and past a few hundred thousand it is worth a fifth of the time.
+
+Where a profile says one of these multiplications matters, hand it an operand that is already contiguous:
+
+```ruby
+kt = k.transpose.dup    # or build k transposed in the first place
+q.dot(kt)
+```
+
+The table above is two-dimensional, where cuBLAS is given one matrix.
+A batched multiplication takes another path through the same flag, and these numbers do not cover it.
+
 ### Fused Operations
 
 `layer_norm`, `rms_norm` and `softmax` normalize along the last axis in one kernel each, and `quantize_symmetric` takes it to 8-bit integers in one more.
@@ -274,6 +341,16 @@ xq, scale = x.quantize_symmetric           # scale = max(|x|) / 127, xq = round(
 
 `quantize_symmetric` answers a `Cumo::Int8` shaped like self and the scale of every row, which is self's shape without its last axis. `xq * scale[false, :new]` is what the row stood for. A row of zeros has no scale to divide by and answers zero for both, and a row holding an infinity or a NaN answers that in its scale and zeros in the row, since neither is a value 8 bits could carry. The scale comes back in the class the reduction accumulates in, which is `Cumo::SFloat` for `Cumo::HFloat` and `Cumo::BFloat` and self's own otherwise.
 
+The rounding takes a tie away from zero, which is the rule `round` follows here and the rule Ruby's `Float#round` follows. numpy and torch take a tie to the nearest even value instead, so code ported from either answers differently wherever the quotient lands exactly halfway.
+
+```
+x / scale             -2.5  -1.5  -0.5   0.5   1.5   2.5
+quantize_symmetric      -3    -2    -1     1     2     3
+numpy, torch            -2    -2     0     0     2     2
+```
+
+`rint` is the other rule, at the cost of writing the quantization out: `(x / scale[false, :new]).rint.clip(-127, 127)` answers what numpy answers. A tie needs `x` to be an exact odd multiple of half the scale, so whether one ever comes up is a property of the data rather than of the arithmetic. Over a million random single-precision elements the two spellings disagreed four times, and every disagreement was a tie.
+
 On an RTX 5070 Ti Laptop, against the same arithmetic spelled with operators, in microseconds:
 
 ```
@@ -363,7 +440,7 @@ runs out of significant digits and answers zero, and `-Float::INFINITY` answers
 
 ### SiLU
 
-`Cumo::NMath.silu` is `x * sigmoid(x)`, the activation Llama and the models after it use where GPT-2 uses `gelu_tanh`.
+`Cumo::NMath.silu` is `x / (1 + exp(-x))`, the activation Llama and the models after it use where GPT-2 uses `gelu_tanh`. It is the curve `x * sigmoid(x)` names, written as one division, and the two spellings do not answer alike: see Sigmoid below.
 It is also called Swish.
 
 ```ruby
@@ -389,6 +466,86 @@ Where that zero starts depends on the type, and not on `exp` alone.
 Single and bfloat16 reach it at -89, where `exp(-x)` passes what a float holds, and double never does.
 Half reaches it at -21, because the round back to half gets there first.
 
+### Sigmoid
+
+`Cumo::NMath.sigmoid` is `1 / (1 + exp(-x))`, the logistic curve.
+
+```ruby
+Cumo::NMath.sigmoid(x)   # in one kernel rather than four
+```
+
+Written out of the operators it costs four launches, and one kernel runs 1.2x to 4.8x faster on an RTX 5070 Ti Laptop, in microseconds:
+
+```
+sigmoid                 SFloat            DFloat            HFloat
+elements           fused  written    fused  written    fused  written
+768                  2.9      9.0      2.5      8.8      2.0      9.6
+786432               5.6     23.0     94.6    111.3      7.3     23.6
+16777216           343.5   1234.7   1978.5   2941.1    210.5    798.1
+```
+
+The kernel keeps the exponent's argument negative, which the plain quotient does not: writing `1 / (1 + exp(-x))` out asks `exp` for a value it cannot hold once `x` is negative enough, and the quotient then answers a zero where the curve is still a number the type carries.
+
+```ruby
+Cumo::NMath.sigmoid(Cumo::SFloat[-100.0]).to_a.first            #=> 3.783506e-44
+(1.0 / (1.0 + Cumo::NMath.exp(Cumo::SFloat[100.0]))).to_a.first #=> 0.0
+```
+
+Where each type reaches zero, written the one way and the other:
+
+```
+             sigmoid   written out
+Cumo::HFloat   -17.5         -11.5
+Cumo::BFloat   -93.0         -89.0
+Cumo::SFloat  -104.0         -89.0
+Cumo::DFloat  -745.0        -710.0
+```
+
+A non-negative `x` takes the plain quotient unchanged, bit for bit, so only the negative half moves. It moves toward the true value more often than away, but not by much and not always: over the 360,000 single points between -88 and 0 the guarded spelling is closer at 90,423, further at 72,593 and the same at the rest, and both spellings pass one unit in the last place, the guarded one at 2,289 points and the plain one at 4,489, neither worse than three.
+
+`silu` is `x / (1 + exp(-x))` rather than `x * sigmoid(x)`, one division rather than a division and a multiply. The second spelling rounds once more and moves a third of the answers, and it moves them in kind at the bottom: `silu` answers the signed zero torch answers below -89, where `x * sigmoid(x)` still carries a number. The two are written apart for that reason.
+
+### Softplus
+
+`Cumo::NMath.softplus` is `log(1 + exp(x))`, the smooth positive part, and the function a selective state space model puts its step size through.
+
+```ruby
+Cumo::NMath.softplus(x)   # log(1 + exp(x)), in one kernel rather than three
+```
+
+Written out of the operators it costs three launches, and one kernel runs 1.0x to 3.1x faster on an RTX 5070 Ti Laptop, in microseconds:
+
+```
+softplus                SFloat            DFloat            HFloat
+elements           fused  written    fused  written    fused  written
+768                  2.0      6.1      2.3      6.2      2.1      6.0
+786432               5.9     18.3    166.6    171.8      5.7     15.0
+16777216           344.2    939.5   3503.7   3938.2    206.2    591.2
+```
+
+Double barely moves at the two larger sizes, the arithmetic rather than the launches being what it pays for there.
+
+Writing it out also gives out earlier than the kernel does, because the intermediate is an array of the receiver's type:
+
+```ruby
+x = Cumo::HFloat[22.26]
+Cumo::NMath.log(1.0 + Cumo::NMath.exp(x))   #=> Infinity
+Cumo::NMath.softplus(x)                     #=> 22.265625
+```
+
+Half holds `exp(x)` only to 11.09 and single to 88.7, where softplus is `x` to the last bit. The kernel takes the single-precision `exp` whatever the type, and hands back `x` where even that has no value to give, so nothing overflows at either width.
+
+It is not `log1p(exp(x))`, which is a different number: the sum is taken before the logarithm, as the definition reads and as the implementations this follows compute. `torch.nn.functional.softplus` takes the other spelling and switches to `x` above 20, so the two differ by about an ulp where both are finite.
+
+Taking the sum first costs the other end. Once `exp(x)` falls under the type's epsilon the sum drops it, so softplus reaches zero while the true value is still a number:
+
+```ruby
+Cumo::NMath.softplus(Cumo::SFloat[-17.0])   #=> 0.0, where the value is 4.1e-08
+Cumo::NMath.softplus(Cumo::DFloat[-37.0])   #=> 0.0, where the value is 8.5e-17
+```
+
+Single and both sixteen-bit types reach that zero at -17 and double at -37, and the error grows before it: 5.9% at -16 in single and 4.3% at -36 in double. `log1p` is what to reach for where a small negative `x` has to keep its digits.
+
 ### Half Precision
 
 `Cumo::HFloat`, also reachable as `Cumo::Float16`, holds IEEE binary16: one sign bit, five of exponent and ten of mantissa.
@@ -472,16 +629,19 @@ A 1x768 by 768x50257 gemv takes 0.211 ms with that odd 50257 and 0.205 ms with 5
 Pad the inner dimensions of a real matrix product; leave a gemv alone.
 
 `conv` is a different story, and worth reading before reaching for half in a network.
-cuDNN chooses its algorithm from the ones that fit in a scratch buffer, and the half algorithms that use the tensor cores ask for more than the default 8MB ceiling allows.
-Left at the default, a half convolution is no faster than a single-precision one.
+cuDNN chooses its algorithm from the ones that fit in a scratch buffer, and the half algorithms that use the tensor cores ask for a lot of it.
+The default ceiling reaches them; the 8MB one Cumo used to ship does not.
 N=32, C=K=64, 56x56, 3x3:
 
 ```
-                              HFloat     SFloat
-CUMO_CUDNN_MAX_WORKSPACE_SIZE unset      1.13 ms    1.03 ms
-CUMO_CUDNN_MAX_WORKSPACE_SIZE=268435456  0.56 ms    0.68 ms
+                                         HFloat     BFloat     SFloat
+CUMO_CUDNN_MAX_WORKSPACE_SIZE=8388608    1.27 ms    1.26 ms    1.19 ms
+unset (128MB)                            0.54 ms    0.60 ms    1.19 ms
+CUMO_CUDNN_MAX_WORKSPACE_SIZE=268435456  0.52 ms    0.59 ms    0.59 ms
 ```
 
+This shape's single-precision algorithm wants more than the default, which is the other half of the reason to look at the ceiling for a network that spends its time in `conv`.
+
 Tensor cores also want the channel counts to be multiples of eight, which the first layer of a network never satisfies.
 That layer is still faster in half, but for the other reason:
 
@@ -535,7 +695,7 @@ The two do not contain each other, so an expression mixing them promotes to `Cum
 
 Everything else promotes as `Cumo::SFloat` does, so an integer array or a Ruby Float mixed in stays bfloat16 while anything wider takes over.
 
-`layer_norm`, `rms_norm`, `softmax`, `silu` and both spellings of `gelu` take it, and so do the reductions, `sort`, `median`, `cumsum`, `rand` and `dot`.
+`layer_norm`, `rms_norm`, `softmax`, `silu`, `softplus`, `sigmoid` and both spellings of `gelu` take it, and so do the reductions, `sort`, `median`, `cumsum`, `rand` and `dot`.
 The cuDNN methods take it too: `conv`, `conv_transpose`, `conv_grad_w`, `max_pool`, `avg_pool` and the three batch norm entries.
 
 Reductions accumulate in single precision and round once at the end, exactly as they do for half, so a sum passes 256 without stopping there:
@@ -584,9 +744,169 @@ A `dot` on a pre-Ampere card is the one to expect trouble from.
 cuDNN reaches bfloat16 as `CUDNN_DATA_BFLOAT16`, and its own bfloat16 kernels want Ampere for the same reason cuBLAS does.
 A convolution is given `CUDNN_DATA_FLOAT` to accumulate in, so it passes 256 the way a reduction does.
 The batch norm parameters are `Cumo::SFloat`, the same as they are for `Cumo::HFloat`.
-The workspace ceiling matters here as much as it does for half: left at the default 8MB, a bfloat16 convolution is no faster than a single-precision one.
+The workspace ceiling matters here as much as it does for half, and for the same reason.
 See [Raise the cuDNN workspace ceiling](#raise-the-cudnn-workspace-ceiling).
 
+#### An index rounds here rather than saturating
+
+[Half Precision](#half-precision) says an index has to be built in a wider type and cast afterwards. That holds here at 256 rather than 2048, and it goes wrong more quietly:
+
+```ruby
+Cumo::BFloat[79800]   #=> 79872.0
+Cumo::HFloat[79800]   #=> Infinity
+```
+
+A binary16 hands back an infinity, which the next operation carries somewhere visible. A bfloat16 hands back a plausible integer, and whatever reads it goes on. Where the index is an angle, the answer comes back with the wrong sign:
+
+```ruby
+Cumo::NMath.cos(Cumo::SFloat[79800])   #=> -0.9190999865531921
+Cumo::NMath.cos(Cumo::SFloat[79872])   #=> 0.989012598991394
+```
+
+Not every integer past 256 is lost, which is what makes this one hard to catch by sampling: `Cumo::BFloat[1000]` is exact, since 1000 is a multiple of 8.
+What ends at 256 is that consecutive integers stay distinct.
+
+### Launching Your Own Kernel
+
+A kernel written in CUDA C can be compiled with NVRTC and launched on the arrays.
+It runs on the stream Cumo's own kernels use, so it sees the results of the operations issued before it and the operations after it see its.
+
+```ruby
+source = <<~CUDA
+  extern "C" __global__ void axpy(float* y, const float* x, float a, int n) {
+    for (int i = blockIdx.x * blockDim.x + threadIdx.x; i < n; i += gridDim.x * blockDim.x) {
+      y[i] = a * x[i] + y[i];
+    }
+  }
+CUDA
+mod = Cumo::CUDA::Compiler.new.compile_with_cache(source)
+axpy = mod.get_function("axpy")
+
+x = Cumo::SFloat.new(1000).seq
+y = Cumo::SFloat.ones(1000)
+axpy.launch([y, x, [2.5].pack("f"), [1000].pack("l")], grid: 4, block: 256)
+```
+
+`compile_with_cache` compiles once and keeps the cubin on disk, and within a process the same source and options answer the same module, so it can be called wherever the kernel is launched.
+That module is shared by everything in the process that compiles the source, so unload it only when nothing else uses it; a module nobody holds any more is unloaded when Ruby collects it.
+`Cumo::CUDA::Compiler.clear_modules` makes the next call read the disk cache or compile again.
+An NArray argument hands over its device pointer, so it has to be contiguous.
+An Integer is passed as a `long long` and a Float as a `double`.
+Anything narrower, and a struct passed by value, goes as the packed bytes of a String: `[n].pack("l")` is an `int` and `[x].pack("f")` a `float`.
+`grid` and `block` take one to three sizes each, and `shared_mem:` is the dynamic shared memory in bytes.
+The kernel is asked for by the name in the source, so it is declared inside `extern "C"`, or asked for by its mangled name.
+On CUDA 12.4 or later the count and the size of the arguments are checked against the kernel before the launch, so an Integer handed to an `int` is refused rather than read wrong.
+A kernel may write any NArray it is handed, so a frozen one is refused, and one that has not been allocated yet is allocated on the way and holds whatever was there.
+
+### Streams And Events
+
+Every kernel and copy Cumo issues goes to the current stream of the thread, which is the null stream until a `Cumo::CUDA::Stream` is used.
+`Stream#with` runs a block on a stream of its own, waits for everything the block queued, and puts the previous stream back, so what the block produced is complete when it returns.
+The block's work is ordered after what the previous stream had queued, so an input still being computed when the block starts is read complete.
+
+```ruby
+s = Cumo::CUDA::Stream.new(non_blocking: true)
+c = s.with { a.gemm(b) }   # queued on s, and finished when with returns
+```
+
+`Stream#use` makes a stream current without a block, `Stream.current` answers the current one and `Stream.null` the null stream.
+`Stream#record` records an `Event` after everything queued so far, and `Stream#wait_event` makes what is queued after it wait for one, which is how two streams are ordered against each other.
+`Event#synchronize` waits for an event on the host, and `Cumo::CUDA.get_elapsed_time(start, stop)` answers the milliseconds between two recorded events, which is how a kernel is timed without a device-wide wait.
+
+```ruby
+start = Cumo::CUDA::Event.new.record
+c = a.gemm(b)
+stop = Cumo::CUDA::Event.new.record
+stop.synchronize
+Cumo::CUDA.get_elapsed_time(start, stop)   # => milliseconds
+```
+
+Under a stream of the caller's, a host read such as `to_a` or `each` waits for the whole device rather than for that stream, since what it reads may have been written on another one.
+The current stream is per thread, and fibers of one thread share it.
+A `Function#launch` takes `stream:` to launch on a stream other than the current one.
+
+### Pinned Host Memory
+
+A copy between the host and the device is asynchronous only when the host side is page-locked.
+`Cumo::CUDA::PinnedMemory` allocates such a buffer, and `NArray#set` and `NArray#get` copy through it on the current stream, or on `stream:`.
+
+```ruby
+pinned = Cumo::CUDA::PinnedMemory.new(a.byte_size)
+s = Cumo::CUDA::Stream.new(non_blocking: true)
+s.with { a.get(pinned) }                         # queued on s, complete when with returns
+b = Cumo::SFloat.from_binary(pinned.read, a.shape)
+
+src = Cumo::CUDA::PinnedMemory.from_binary(bytes)
+c.set(src, stream: s)                             # the host bytes reach c once s gets there
+```
+
+`set` with a String and `get` with nothing to copy into are the synchronous `store_binary` and `to_binary`.
+A pinned buffer is read and written as bytes with `read` and `write`, and the array has to be contiguous and of the buffer's size.
+A copy in flight keeps its array alive and is waited for by `read`, `write`, `free` and the next copy, so the bytes read are the copy's.
+Allocating a pinned buffer synchronizes the device, so a buffer is allocated once and reused rather than made for every transfer.
+
+### Structs And Template Kernels
+
+An array of structs is an array whose trailing axes are the struct's fields, so a kernel that takes `const double3*` is handed a `Cumo::DFloat` of shape `[n, 3]`, and one that takes `const Matrix<float>*` an `SFloat` of shape `[n, 4, 4]`.
+`SComplex` and `DComplex` are `float2` and `double2` to a kernel; `Bit`, which packs its elements, and `RObject` cannot be handed to one.
+A struct passed by value is the packed bytes of its fields, and where the struct has padding, `pack`'s `@` places each field at the offset the device reports.
+
+```ruby
+rhs = [x, y, z].pack("d3")                           # a double3 by value
+sum_kernel.launch([lhs, rhs, out], grid: 1, block: n)
+```
+
+A template kernel has no `extern "C"` name.
+`compile_with_cache` takes `name_expressions:`, and the module then answers `get_function` for each expression by the mangled name NVRTC reports.
+
+```ruby
+mod = Cumo::CUDA::Compiler.new.compile_with_cache(source, name_expressions: ["kernel<float>", "kernel<double>"])
+mod.get_function("kernel<float>").launch([a, b, c.to_a.flatten.pack("f*"), out], grid: 1, block: n)
+```
+
+### Writing An Elementwise Kernel
+
+`Cumo::CUDA::ElementwiseKernel` takes one piece of CUDA C and applies it to every element, the way CuPy's `ElementwiseKernel` does.
+The kernel is compiled once for each set of dtypes, once more when a different argument is a number or is broadcast, and kept for the next call.
+
+```ruby
+squared_diff = Cumo::CUDA::ElementwiseKernel.new(
+  "T x, T y", "T z", "z = (x - y) * (x - y)", "squared_diff")
+
+x = Cumo::SFloat.new(2, 5).seq
+y = Cumo::SFloat.new(5).seq
+squared_diff.call(x, y)   # => the (2, 5) array of squared differences
+squared_diff.call(x, 5)   # => the same against a scalar
+```
+
+A type is one of `float64`, `float32`, `int64`, `int32`, `int16`, `int8`, `uint64`, `uint32`, `uint16` and `uint8`, or a single letter that stands for whichever dtype the argument has.
+Outputs decide a letter before inputs do, and a letter that only a Ruby number reaches becomes `int64` or `float64`.
+Array arguments are broadcast against each other, and an output may be given after the inputs, or is allocated.
+An argument marked `raw T y` is handed over as a pointer for the operation to index itself, with `i` the element index and `_ind.size()` the element count, and when every argument is raw or a number, `size:` says how many elements there are.
+Inputs are `const`, so an operation that writes one does not compile, and a number handed to an integer type has to be an Integer that fits.
+`preamble:` is placed before the kernel, after the typedefs of the letters, so a device function can be written in terms of `T`.
+`Cumo::Bit`, `Cumo::HFloat`, `Cumo::BFloat`, `Cumo::SComplex`, `Cumo::DComplex` and `Cumo::RObject` cannot be handed to one of these kernels yet.
+
+### Writing A Reduction Kernel
+
+`Cumo::CUDA::ReductionKernel` reduces along axes with three expressions, the way CuPy's `ReductionKernel` does: a map applied to every element, a reduce between two mapped values `a` and `b`, and a post map that writes the reduced value `a` to the output.
+The identity starts every reduction.
+
+```ruby
+l2norm = Cumo::CUDA::ReductionKernel.new(
+  "T x", "T y", "x * x", "a + b", "y = sqrt(a)", "0", "l2norm")
+
+x = Cumo::SFloat.new(2, 5).seq
+l2norm.call(x, axis: 1)   # => [5.477, 15.969]
+l2norm.call(x)            # => the 0-dimensional norm of everything
+```
+
+The parameters, the types and the broadcasting follow `ElementwiseKernel`, and `axis:` and `keepdims:` follow `sum`.
+`reduce_type:` names the type the values are accumulated in, as a C type, a type name or one of the letters, and is the output's type unless said otherwise.
+The post map may be several statements, so a kernel can write several outputs.
+A `raw` parameter is not taken, since a reduction indexes every argument itself.
+A long axis reduced to a few outputs is split across blocks and folded in a second pass, so it runs as fast as `sum`.
+
 ### Select a GPU device ID
 
 Set the `CUDA_VISIBLE_DEVICES=id` environment variable, or
@@ -607,6 +927,37 @@ require 'cumo'
 Cumo::CUDA::MemoryPool.disable
 ```
 
+### Reading The Memory Numbers
+
+cumo allocates with `cudaMallocManaged`, so a page can sit on the host, and `nvidia-smi` counts only what the card is holding at that moment.
+It reports less than cumo has taken, and the gap is not a fixed one.
+A library that allocates with `cudaMalloc` reports everything it took, so the two numbers do not belong side by side in a comparison.
+
+From inside, the pool answers two different questions:
+
+```ruby
+Cumo::CUDA::MemoryPool.total_bytes   # what the pool has taken from the card
+Cumo::CUDA::MemoryPool.used_bytes    # what it has handed out and not taken back
+```
+
+`used_bytes` still counts blocks whose last reference is gone but which Ruby's garbage collector has not reached, so read it after `GC.start` when you mean the arrays that are alive:
+
+```ruby
+a = Cumo::SFloat.new(1024, 1024).seq
+t = a + 1
+t = nil
+Cumo::CUDA::MemoryPool.used_bytes    #=> 8388608
+GC.start
+Cumo::CUDA::MemoryPool.used_bytes    #=> 4194304
+```
+
+`total_bytes` does not move with the collector and counts the free blocks the pool is keeping, so it answers how much this took rather than how much is alive.
+
+## Examples
+
+`examples/` holds ports of the examples in the CuPy repository, one Ruby file per original, with a `backend.rb` that runs the CPU side on Numo and the GPU side on Cumo under `GPU=1`.
+`examples/README.md` lists what is ported and how to run them.
+
 ## Documentation
 
 See https://github.com/ruby-numo/numo-narray#documentation, replacing Numo with Cumo.
@@ -730,17 +1081,57 @@ export CUMO_SHOW_WARNING_ONCE=OFF
 
 ### Raise the cuDNN workspace ceiling
 
-cuDNN picks a convolution algorithm by benchmarking the ones that fit in a scratch buffer, and the ceiling on that buffer is 8MB.
-The fastest half precision algorithms, the ones that reach the tensor cores, ask for more than that and are left out of the search.
-To let them in:
+cuDNN picks a convolution algorithm by benchmarking the ones that fit in a scratch buffer, and the ceiling on that buffer is 128MB.
+The search reserves the whole ceiling whatever the convolution's size and hands it back to the pool afterwards, so the ceiling costs a peak rather than a residency.
+
+Some shapes want more than the default. The twenty convolutions of a ResNet-18 forward pass at batch 16, and the single-precision case from [Half Precision](#half-precision):
+
+```
+                                         ResNet-18    N=32, C=K=64, 56x56, 3x3
+CUMO_CUDNN_MAX_WORKSPACE_SIZE=8388608      9.80 ms                     1.19 ms
+unset (128MB)                              5.23 ms                     1.19 ms
+CUMO_CUDNN_MAX_WORKSPACE_SIZE=268435456    5.24 ms                     0.59 ms
+```
+
+To raise it:
 
 ```
-export CUMO_CUDNN_MAX_WORKSPACE_SIZE=67108864
+export CUMO_CUDNN_MAX_WORKSPACE_SIZE=268435456
 ```
 
 The value is in bytes and only bounds the search; each convolution reserves what its chosen algorithm actually needs.
 `Cumo::CUDA::CUDNN.max_workspace_size` reads back the value in force.
 
+### Single precision stays off the tensor cores
+
+cuDNN reads its default math mode as "tensor cores are allowed", so a single precision convolution moves onto them as soon as the algorithm search reaches an algorithm that has them.
+The operands are rounded to a 10 bit significand on the way, and nothing in the call said to do that.
+Cumo asks cuDNN to keep single precision off the tensor cores, so raising the workspace ceiling buys speed at single precision accuracy.
+A different algorithm rounds differently, so the answer still moves; what it does not do is drop two digits.
+The half types name the tensor cores themselves and are not affected.
+
+To trade the accuracy for the speed:
+
+```
+export CUMO_ALLOW_TF32=1
+```
+
+Measured over the convolutions of a ResNet-18 forward pass at batch 16, with the ceiling raised to 256MB, each layer against a double precision reference:
+
+```
+                              pass      worst layer
+tensor cores off (default)   5.24 ms      1.4e-05
+tensor cores on              4.31 ms      2.4e-04
+```
+
+The same flag puts `SFloat` and `SComplex` `gemm` on the tensor cores as TF32, and `dot` where it goes through `gemm`.
+Off, the answer is the one cuBLAS gives at single precision, bit for bit.
+On, a `[4096, 4096]` `SFloat` `gemm` goes from 17.8 to 28.4 TFLOP/s on an RTX 5070 Ti Laptop, 1.60 times over six interleaved rounds.
+Its answer is then about 3e-04 from a double precision reference, where it was 4e-07.
+The double types are not affected either way.
+
+`Cumo.allow_tf32?` reads back the value in force.
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/sonots/cumo.

data/bench/cumo_probe.rb

@@ -77,7 +77,7 @@ DTYPES = (ENV['DTYPES'] == 'all' ? DTYPE_SETS['all'] : (ENV['DTYPES']&.split(','
 
 LAYOUT_SETS = {
   'default' => %i[contig colslice],
-  'all' => %i[contig colslice transpose step2 index]
+  'all' => %i[contig colslice transpose step2 index reverse]
 }.freeze
 LAYOUTS = (ENV['LAYOUTS'] == 'all' ? LAYOUT_SETS['all'] : (ENV['LAYOUTS']&.split(',')&.map(&:to_sym) || LAYOUT_SETS['default']))
 
@@ -156,6 +156,9 @@ def operand(klass, rows, cols, layout)
   when :transpose then base(klass, cols, rows).transpose
   when :step2     then base(klass, rows, cols * 2)[true, (0...(cols * 2)).step(2)]
   when :index     then base(klass, rows, cols)[true, (0...cols).to_a.rotate(7)]
+  # walks one stride like contig does, but backwards, which is the shape a
+  # kernel that only knows +elmsz hands to a buffer instead
+  when :reverse   then base(klass, rows, cols).reverse(1)
   else raise ArgumentError, "unknown layout #{layout}"
   end
 end
@@ -312,7 +315,8 @@ op(:scalar, 'scalar_left') { |c| c.scalar - c.sc.inplace }
 
 # NMath
 %w[sqrt exp exp2 exp10 expm1 log log2 log10 log1p sin cos tan asin acos atan
-   sinh cosh tanh asinh acosh atanh cbrt erf erfc sinc gelu gelu_tanh silu].each do |f|
+   sinh cosh tanh asinh acosh atanh cbrt erf erfc sinc gelu gelu_tanh silu
+   softplus sigmoid].each do |f|
   op(:math, f, :float) { |c| XM::NMath.send(f, c.usc.inplace) }
 end
 op(:math, 'atan2', :float) { |c| XM::NMath.atan2(c.sc.inplace, c.b) }

data/bench/cumo_sync_probe.rb

@@ -342,7 +342,7 @@ if ARGV.include?('--list')
   exit
 end
 
-warn_on = ENV['CUMO_SHOW_WARNING'].to_s.upcase == 'ON'
+warn_on = %w[1 on yes true].include?(ENV['CUMO_SHOW_WARNING'].to_s.downcase)
 
 puts "ruby        : #{RUBY_VERSION} (#{RUBY_PLATFORM})"
 puts "cumo        : #{begin

data/cumo.gemspec

@@ -25,7 +25,7 @@ Gem::Specification.new do |spec|
   spec.required_ruby_version = ">= 3.0.0"
 
   spec.files = `git ls-files -z`.split("\x0").reject do |f|
-    f.match(%r{^(.github|test|spec|features|docker)/})
+    f.match(%r{^(.github|test|spec|features|docker|examples)/})
   end
   spec.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
   spec.bindir        = "exe"