carray-netcdf 2.0.0 → 2.1.0

data/docs/07_cookbook.md

@@ -1,6 +1,6 @@
 # Cookbook
 
-End-to-end scenarios. Each example is self-contained; copy, adapt, run.
+End-to-end scenarios, from writing a file through reading, copying and converting one to working with data larger than memory. Some recipes read a file an earlier one wrote, and say which; names such as `time_windows` or `process` stand for your own data and code.
 
 ## 1. Write a CF-1.7 hourly temperature file
 
@@ -61,21 +61,14 @@ NC::File.open("hourly_temp.nc") do |nc|
 
   temp = nc["temp"].to_ca             # decoded (fill -> mask if present)
   temp.shape                          # => [24, 73, 144]
-  temp.mean                           # → around 273-285 K
-end
-```
-
-## 2. Read the third time step of a CF file, mask-aware
+  temp.mean                           # => about 271.8
 
-```ruby
-NC::File.open("hourly_temp.nc") do |nc|
-  slab = nc["temp"][2, nil, nil]      # rank-2 CArray, shape [73, 144]
-  slab.count_masked                   # 0 unless the file has fill cells here
-  slab_at_equator = slab[36, nil]     # rank-1 CArray, shape [144]
+  slab = nc["temp"][2, nil, nil]      # the third time step only, shape [73, 144]
+  slab.count_masked                   # 0 unless the file has fill cells there
 end
 ```
 
-## 3. Pack a physical field into a short (satellite-style storage)
+## 2. Pack a physical field into a short
 
 Storing a physical range that fits comfortably in `short` saves half the space of `float`. CF's `scale_factor` / `add_offset` handle it:
 
@@ -117,70 +110,11 @@ NC::File.open("packed.nc", codec: [:cf, mask_valid_range: true]) do |nc|
 end
 ```
 
-## 4. Stream a large variable one slab at a time
-
-```ruby
-require "carray-netcdf"
-NC = CA::NetCDF
+## 3. Append time steps to an unlimited-dimension file
 
-# Assume /big.nc has variable temp[time, lat, lon] with time = 10000.
-# Compute the daily mean without ever holding the whole array.
-
-means = CArray.float64(10000)
-
-NC::File.open("/big.nc") do |nc|
-  temp = nc["temp"]
-  n_time = temp.shape[0]
-
-  # For zero-allocation streaming, drop to layer 1.  Layer 2's hyperslab
-  # read allocates a fresh CArray per step; layer 1 lets us reuse a buffer.
-  vid = ::NetCDF.nc_inq_varid(nc.ncid, "temp")
-  buf = CArray.float32(1, *temp.shape[1..])   # [1, lat, lon]
-
-  n_time.times do |k|
-    NC.nc_get_vara(nc.ncid, vid, [k, 0, 0], [1, *temp.shape[1..]], buf)
-    means[k] = buf.mean                        # raw physical bytes
-  end
-end
-```
-
-If your file has `_FillValue` or `scale_factor`, layer 1 doesn't apply them — you'd need to either drop that particular streaming loop into layer 2 (which allocates per slab but is one line: `means[k] = nc["temp"][k, nil, nil].mean`), or apply the transforms yourself.
-
-## 4b. Generate a batch of files from a single reference
-
-The classic "read one, write many" pipeline: hold a reference NetCDF open, loop over some units of work, and emit one output per iteration that inherits the reference's skeleton (dims, global atts, coordinate vars) and only diverges in the payload.  `File.create(template:, vars:)` collapses the boilerplate; `clobber: :skip` + `atomic: true` make the loop restartable.
+Adds one hour to the file from recipe 1:
 
 ```ruby
-NC::File.open("reference.nc") do |ref|
-  time_windows.each do |window|      # e.g. one 24h chunk per iteration
-    NC::File.create("out_#{window.tag}.nc",
-                    template: ref,          # dims + atts + format inherited
-                    vars: %w[lat lon],      # coord vars: full bit-identical copy
-                    clobber: :skip,         # already-produced files are left alone
-                    atomic: true) do |dst|
-      dst.def_var("time", :int, ["time"],
-                  units: "hours since #{window.origin}", calendar: "standard")
-      dst.def_var("temp", :float, ["time","lat","lon"], _FillValue: -999.9)
-      dst["time"] = window.timestamps
-      window.each_slab { |t, slab| dst["temp", t, nil, nil] = slab }
-    end
-  end
-end
-```
-
-Why the combination matters:
-
-- **`template: ref`** — dims, `Conventions`, `title`, and any custom global attributes carry over.  Output inherits the NetCDF-4 format if the reference is netcdf-4 (so groups / deflate work), unless `netcdf4:` is set explicitly.
-- **`vars: %w[lat lon]`** — coordinate variables that don't change between outputs come along for the ride, byte-identical (no codec on either side).  Downstream diff / dedup tools won't see spurious floating-point drift.
-- **`clobber: :skip`** — reruns pick up where a previous run stopped; finished files are not touched.  Combines cleanly with `atomic:`.
-- **`atomic: true`** — a mid-loop crash leaves neither a half-written output nor a stale one.  `Dir["out_*.nc"]` reflects exactly what completed successfully.
-
-For read-side workflows that consume the output, the split is clean: either the file is fully there (produced by a completed `create` block), or it is not there yet.  No corrupt-file recovery logic in downstream code.
-
-## 5. Append time steps to an unlimited-dimension file
-
-```ruby
-# Existing file created with time = :unlimited.
 NC::File.open("hourly_temp.nc", :write) do |nc|
   cur = nc.dim("time").length       # e.g. 24
 
@@ -191,7 +125,7 @@ NC::File.open("hourly_temp.nc", :write) do |nc|
 end
 ```
 
-## 6. Group your model by subsystem (NetCDF-4)
+## 4. Group your model by subsystem (NetCDF-4)
 
 ```ruby
 NC::File.create("model.nc", netcdf4: true) do |nc|
@@ -213,10 +147,10 @@ NC::File.create("model.nc", netcdf4: true) do |nc|
              units: "hours since 2024-01-01T00:00:00Z", calendar: "standard")
 
   # Write some data
-  T = 6
-  nc["time"] = CArray.time_series("2024-01-01T00:00:00Z", count: T, unit: :h)
+  steps = 6
+  nc["time"] = CArray.time_series("2024-01-01T00:00:00Z", count: steps, unit: :h)
 
-  T.times do |t|
+  steps.times do |t|
     ocean["temperature", t, nil] = CArray.float32(50) { |z| 25.0 - z * 0.4 }
     ocean["salinity",    t, nil] = CArray.float32(50) { |z| 35.0 + z * 0.01 }
     atm["pressure",      t, nil] = CArray.float32(30) { |z| 1013.0 * Math.exp(-z / 8.0) }
@@ -236,13 +170,33 @@ NC::File.open("model.nc") do |nc|
 end
 ```
 
-## 7. Inspect a mystery file quickly
+## 5. Select a time range by date
+
+Find the steps between two instants on the time coordinate, then read only those. With the file from recipe 1:
+
+```ruby
+NC::File.open("hourly_temp.nc") do |nc|
+  t    = nc["time"].to_ca                                   # CATime
+  from = CArray.time(["2022-01-01T06:00:00Z"], unit: :h)[0]
+  upto = CArray.time(["2022-01-01T11:00:00Z"], unit: :h)[0]
+
+  k = ((t >= from) & (t <= upto)).where                     # => [6, 7, 8, 9, 10, 11]
+  morning = nc["temp"][k.min..k.max, nil, nil]              # reads those 6 steps only
+  morning.shape                                             # => [6, 73, 144]
+end
+```
+
+The comparison works on instants, so `from` and `upto` may be at any resolution, a day as well as an hour. Reading the range `k.min..k.max` rather than `k` itself keeps the read a single block, which matters when the steps are many.
+
+## 6. Inspect a mystery file quickly
 
 ```ruby
 NC::File.open("mystery.nc") do |nc|
-  puts "format:      #{nc.format}"
+  format = NetCDF.constants.grep(/\ANC_FORMAT_/).find { |c| NetCDF.const_get(c) == nc.format }
+  codec  = nc.codec.is_a?(Module) ? nc.codec : nc.codec.class   # Identity is a module
+  puts "format:      #{format}"
   puts "conventions: #{nc.att('Conventions') || '(none)'}"
-  puts "codec:       #{nc.codec.class.name.split('::').last}"
+  puts "codec:       #{codec.name.split('::').last}"
   puts
 
   puts "dimensions:"
@@ -267,27 +221,141 @@ NC::File.open("mystery.nc") do |nc|
 end
 ```
 
-## 8. Round-trip decoded vs raw for a single variable
+For the whole header as CDL, JSON or YAML, the `ncheader` command does this from the shell (see [Command-line tools](09_cli.md)). To see what the codec does to one variable, compare `v.to_ca` with `v.raw` (see [Seeing what the codec changed](04_codec.md#seeing-what-the-codec-changed)).
+
+## 7. Cut a region out into a new file
+
+Copy a latitude band of the file from recipe 1, with each variable's definition and attributes, through the codec: values are decoded on the way out and encoded again on the way in, so masks and packing carry over.
+
+```ruby
+NC::File.open("hourly_temp.nc") do |src|
+  lat = src["lat"].to_ca
+  j   = ((lat >= 20) & (lat <= 50)).where       # the rows of the band
+  rows = j.min..j.max
+
+  NC::File.create("band.nc", template: src, dims: %w[time lon], vars: %w[time lon]) do |dst|
+    dst.def_dim("lat", rows.size)
+    dst.def_var("lat",  template: src["lat"])
+    dst.def_var("temp", template: src["temp"])
+    dst["lat"] = src["lat"][rows]
+    src["time"].shape[0].times do |t|           # one time step at a time
+      dst["temp", t, nil, nil] = src["temp"][t, rows, nil]
+    end
+  end
+end
+```
+
+`File.create(template:)` brings the global attributes and, with `dims:` / `vars:`, the dimensions and coordinate variables that stay the same; `def_var(template:)` takes each changed variable's type and attributes. Writing a step at a time keeps one slab in memory.
+
+## 8. Convert a NetCDF-3 file to compressed NetCDF-4
+
+```ruby
+NC::File.open("hourly_temp.nc") do |src|
+  NC::File.create("hourly_temp4.nc", template: src, format: :netcdf4) do |dst|
+    src.vars.each_key do |name|
+      dst.copy_var(name, from: src, deflate: 4, shuffle: true)
+    end
+  end
+end
+
+NC::File.open("hourly_temp4.nc") { |nc| nc["temp"].storage_options }
+# => {chunks: [1, 73, 144], deflate: 4, shuffle: true}
+```
+
+`template:` brings the dimensions and global attributes, and `copy_var` each variable with its stored values unchanged; the kwargs set the compression. On this file that halves its size. Add `chunks:` to choose the chunking yourself, or use `zstandard:` (libnetcdf 4.9 and later) for faster compression; see [Writing files](03_writing.md).
 
-Handy when a file behaves unexpectedly and you want to see what the codec is doing:
+## 9. Generate a batch of files from a single reference
+
+Hold a reference file open, and write one output per unit of work, each with the reference's dimensions, global attributes and coordinate variables. `File.create(template:, vars:)` copies those; `clobber: :skip` and `atomic: true` let the loop be run again after a failure. Here the reference's `time` dimension is unlimited, so each output can hold its own number of steps, and each `window` gives `tag`, `origin`, `timestamps` (a CATime) and `each_slab`, which yields a time index and a `[lat, lon]` CArray.
 
 ```ruby
-NC::File.open("data.nc") do |nc|
-  decoded = nc["temp"].to_ca
-  puts "decoded: mean = #{decoded.mean}, masked = #{decoded.count_masked}"
+NC::File.open("reference.nc") do |ref|
+  time_windows.each do |window|      # e.g. one 24h chunk per iteration
+    NC::File.create("out_#{window.tag}.nc",
+                    template: ref,          # dims + atts + format inherited
+                    vars: %w[lat lon],      # coord vars: full bit-identical copy
+                    clobber: :skip,         # already-produced files are left alone
+                    atomic: true) do |dst|
+      dst.def_var("time", :int, ["time"],
+                  units: "hours since #{window.origin}", calendar: "standard")
+      dst.def_var("temp", :float, ["time","lat","lon"], _FillValue: -999.9)
+      dst["time"] = window.timestamps
+      window.each_slab { |t, slab| dst["temp", t, nil, nil] = slab }
+    end
+  end
 end
+```
+
+Why the combination matters:
 
-NC::File.open("data.nc", codec: :identity) do |nc|
-  raw = nc["temp"].to_ca
-  puts "raw: mean = #{raw.mean}, min = #{raw.min}, max = #{raw.max}"
-  puts "attrs: #{nc['temp'].atts.inspect}"
+- **`template: ref`** — dims, `Conventions`, `title`, and any custom global attributes carry over. The output has the reference's format, unless `format:` or `netcdf4:` is given.
+- **`vars: %w[lat lon]`** — coordinate variables that are the same in every output are copied with their stored values unchanged, so they compare equal byte for byte.
+- **`clobber: :skip`** — a rerun skips the outputs that already exist and writes the rest.
+- **`atomic: true`** — an exception mid-loop leaves no half-written output, so `Dir["out_*.nc"]` lists only completed files. A killed process can leave a `*.tmp` file behind.
+
+## 10. Stream a large variable one slab at a time
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+
+# big.nc has a variable temp[time, lat, lon] with many time steps.
+# Compute the mean of each time step without holding the whole array.
+
+NC::File.open("big.nc") do |nc|
+  temp   = nc["temp"]
+  n_time = temp.shape[0]
+  means  = CArray.float64(n_time)
+
+  # The low-level API reads each step into the same buffer, so nothing is
+  # allocated per step.
+  buf = CArray.float32(1, *temp.shape[1..])   # [1, lat, lon]
+
+  n_time.times do |k|
+    NC.nc_get_vara(nc.ncid, temp.varid, [k, 0, 0], [1, *temp.shape[1..]], buf)
+    means[k] = buf.mean                         # stored values, not decoded
+  end
+end
+```
+
+The low-level API applies no codec, so `buf` holds the stored values. If the variable has a `_FillValue` or `scale_factor`, decode each step with `temp.codec.decode(temp, buf)`, or read it with `temp[k, nil, nil]`, which decodes but allocates a new array per step.
+
+## 11. Read in one thread, compute in another
+
+libnetcdf runs without the GVL, so the main thread can compute on one slab while a reader thread reads the next:
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+
+queue = SizedQueue.new(4)              # at most 4 slabs held at once
+
+reader = Thread.new do
+  NC::File.open("big.nc") do |nc|      # this File is used by this thread only
+    temp = nc["temp"]
+    temp.shape[0].times { |k| queue << temp[k, nil, nil] }
+  end
+ensure
+  queue << nil                         # no more slabs, even if the read failed
+end
+
+means = []
+while slab = queue.pop                 # each slab is handed over once it has been read
+  means << slab.mean
 end
+reader.join                            # re-raises an error from the reader thread
 ```
 
-## 9. Write a small file with a bit-exact custom codec
+This stays within what is safe with threads (see [Threads and Ractors](00_introduction.md#threads-and-ractors)):
+
+- **One thread owns the file.** The `File` and its `Var` are opened, used and closed in the reader thread. To read several files at once, give each thread its own `File`; the calls into libnetcdf still run one at a time.
+- **A CArray changes hands only after its read.** A slab goes into the queue once it has been read, and the reader does not touch it again.
+
+## 12. Mask fill values without unpacking
+
+A codec of your own can apply only part of CF. This one turns `_FillValue` into a mask and back, and leaves `scale_factor` and `add_offset` alone, so the values stay packed:
 
 ```ruby
-# A minimal codec that only masks _FillValue but skips scale/offset/time.
 class OnlyFillCodec
   def decode(var, ca)
     fill = var.att("_FillValue") or return ca
@@ -304,42 +372,15 @@ class OnlyFillCodec
   end
 end
 
-NC::File.open("packed.nc", codec: OnlyFillCodec.new) do |nc|
-  raw_with_mask = nc["temp"].to_ca    # packed short values, fill cells masked,
-                                      # no scale/offset applied
+NC::File.create("fill_only.nc", codec: OnlyFillCodec.new) do |nc|
+  nc.def_dim("n", 3)
+  nc.def_var("v", :short, ["n"], scale_factor: 0.5, _FillValue: -1)
+  nc["v"] = CA_SHORT([1, UNDEF, 3])       # the masked element is written as -1
 end
-```
-
-The codec object just has to respond to `decode(var, ca)` and `encode(var, ca)`. See the [Codecs chapter](04_codec.md) for the full interface and how to register a codec globally.
 
-## 10. Existing-file inspection with `ncdump`-like output
-
-```ruby
-def cdl_summary(nc)
-  puts "netcdf {"
-  puts "dimensions:"
-  nc.dims.each { |n, d|
-    len = d.unlimited? ? "UNLIMITED" : d.length.to_s
-    puts "  #{n} = #{len};"
-  }
-  puts "variables:"
-  nc.vars.each { |n, v|
-    dims = v.dims.map(&:name).join(", ")
-    puts "  #{v.data_type} #{n}(#{dims});"
-    v.atts.each { |k, val|
-      puts "    #{n}:#{k} = #{val.inspect};"
-    }
-  }
-  puts "// global attributes:"
-  nc.atts.each { |k, v| puts "  :#{k} = #{v.inspect};" }
-  nc.groups.each { |gname, g|
-    puts
-    puts "group: #{gname} {"
-    cdl_summary(g)
-    puts "} // group #{gname}"
-  }
-  puts "}"
+NC::File.open("fill_only.nc", codec: OnlyFillCodec.new) do |nc|
+  nc["v"].to_ca.to_a                      # => [1, UNDEF, 3]   (still packed)
 end
-
-NC::File.open("mystery.nc") { |nc| cdl_summary(nc) }
 ```
+
+The codec object just has to respond to `decode(var, ca)` and `encode(var, ca)`. See the [Codecs chapter](04_codec.md) for the full interface and how to register a codec, so that files whose `Conventions` it claims pick it up.