carray-netcdf 1.0.13 → 2.0.0

data/docs/06_layer1.md

@@ -0,0 +1,160 @@
+# Layer 1 — raw `nc_*` API
+
+Everyday code uses layer 2 (`File`, `Var`, ...). Layer 1 is the CArray-flavored view of the netcdf gem's `nc_*` functions — same names, same argument order, but numeric buffers are CArrays.
+
+Use it when:
+
+- You want to open by ncid or work outside the `File` object model.
+- You want to stream a very large variable into a preallocated buffer (`nc_get_vara` with an explicit buffer = zero allocation per read).
+- You're bridging to code that already speaks the C `nc_*` idiom.
+- You want to skip the codec and every layer-2 semantics for a specific operation.
+
+## The module
+
+`CA::NetCDF` itself IS layer 1. It `include`s `::NetCDF` (the netcdf gem's module) so all `nc_*` functions and `NC_*` constants pass through, and overrides only the array I/O to speak CArray.
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+```
+
+Two idioms:
+
+```ruby
+# (a) call style — module functions
+NC.nc_get_vara(ncid, varid, [0, 0], [3, 4])   # => CArray
+NC::NC_DOUBLE                                  # constants inherited from ::NetCDF
+
+# (b) mixin style — `nc_*` become bare methods in your class
+class Reader
+  include CA::NetCDF
+  def read_all(ncid, varid) = nc_get_var(ncid, varid)   # bare, returns CArray
+  def define(ncid, name)    = nc_def_dim(ncid, name, 100)   # passes through
+end
+```
+
+## What layer 1 overrides vs passes through
+
+Overridden (numeric array I/O):
+
+- `nc_get_var`, `nc_get_vara`, `nc_get_vars`, `nc_get_varm`
+- `nc_put_var`, `nc_put_vara`, `nc_put_vars`, `nc_put_varm`
+
+Everything else — `nc_open`, `nc_create`, `nc_def_dim`, `nc_def_var`, `nc_put_att`, `nc_get_att`, `nc_inq_*`, `nc_def_grp`, `nc_close`, `NC_DOUBLE`, `NC_CLOBBER`, ... — comes straight from the netcdf gem, which this layer includes without modifying.
+
+## `nc_get_var*` — buffer-less returns a CArray
+
+Without a buffer argument, layer 1 allocates a CArray of the requested shape, typed from the variable's on-disk type (`NC_CHAR` as `:uint8`, an enum as its base type), and reads into it:
+
+```ruby
+ncid = ::NetCDF.nc_open("data.nc", ::NetCDF::NC_NOWRITE)
+vid  = ::NetCDF.nc_inq_varid(ncid, "temp")
+
+ca = NC.nc_get_var(ncid, vid)                          # whole variable  -> CArray
+ca = NC.nc_get_vara(ncid, vid, [0, 0], [3, 4])         # hyperslab       -> CArray
+ca = NC.nc_get_vars(ncid, vid, [0, 0], [3, 4], [1, 2]) # strided         -> CArray
+ca = NC.nc_get_varm(ncid, vid, [0, 0], [3, 4], [1, 1], [4, 1])  # mapped -> CArray
+
+::NetCDF.nc_close(ncid)
+```
+
+No codec runs — this is raw, no `_FillValue → UNDEF`, no scale/offset, no CATime. Use layer 2 for those.
+
+A type with no CArray counterpart — opaque, `NC_STRING` — raises `NotImplementedError`; read an opaque variable into a buffer of your own.
+
+## `nc_get_var*` — with a buffer, reads into it
+
+Pass an existing CArray and layer 1 asks netcdf to read into that buffer. Perfect for streaming a slab at a time with **zero allocation per read**:
+
+```ruby
+ncid = ::NetCDF.nc_open("big.nc", ::NetCDF::NC_NOWRITE)
+vid  = ::NetCDF.nc_inq_varid(ncid, "temp")
+
+buf = CArray.float32(73, 144)   # one time slice
+
+365.times do |day|
+  NC.nc_get_vara(ncid, vid, [day, 0, 0], [1, 73, 144], buf)
+  process(buf)                  # buf now holds day `day`
+end
+
+::NetCDF.nc_close(ncid)
+```
+
+The buffer must be C-contiguous (a plain entity CArray is; a view is not — the gem will refuse a non-contiguous buffer). Type conversion is handled by libnetcdf — you can read a short variable into a float64 buffer and libnetcdf converts.
+
+## `nc_put_var*` — CArray as buffer
+
+Symmetric. A contiguous CArray goes through as-is:
+
+```ruby
+NC.nc_put_vara(ncid, vid, [0, 0], [3, 4], CArray.float32(3, 4).seq!)
+```
+
+A non-contiguous CArray (a view / slice / reference) is materialized to a contiguous entity before the write:
+
+```ruby
+big = CArray.float32(100, 4).seq!
+NC.nc_put_vara(ncid, vid, [0, 0], [3, 4], big[10..12, nil])
+# big[10..12, nil] is a CABlock view; layer 1 does .copy on your behalf
+# and passes the resulting entity to nc_put_vara.
+```
+
+A **masked** CArray is *refused*. Layer 1 does no fill encoding — that's layer 2's job (via the codec). Strip the mask first:
+
+```ruby
+ca = CA_SHORT([1, 2, 3])
+ca[1] = UNDEF
+NC.nc_put_vara(ncid, vid, [0], [3], ca)          # ArgumentError
+NC.nc_put_vara(ncid, vid, [0], [3], ca.strip_mask(-32768))    # OK, fill baked in
+```
+
+## `nc_get_var1` / `nc_put_var1` — scalar
+
+These are pure passthrough (the netcdf gem handles them directly), included in layer 1 so the mixin style has them:
+
+```ruby
+value = NC.nc_get_var1(ncid, vid, [0, 0])         # Numeric
+NC.nc_put_var1(ncid, vid, [0, 0], 273.15)         # writes one cell
+```
+
+Layer 2's scalar decode/encode wraps these calls in a 1-cell CArray to run through the codec — see [Codecs](04_codec.md).
+
+## Definition, inquiry, attribute — all passthrough
+
+The netcdf gem covers everything else. In layer-1 code, use them bare (via mixin) or fully qualified:
+
+```ruby
+# create + define
+ncid = NC.nc_create("out.nc", NC::NC_CLOBBER | NC::NC_NETCDF4)
+tid  = NC.nc_def_dim(ncid, "time", 24)
+vid  = NC.nc_def_var(ncid, "temp", NC::NC_FLOAT, [tid])
+NC.nc_put_att(ncid, vid, "units", "K")
+NC.nc_enddef(ncid)
+
+# write and close
+NC.nc_put_var(ncid, vid, CArray.float32(24).seq!)
+NC.nc_close(ncid)
+
+# inquire
+ncid = NC.nc_open("out.nc", NC::NC_NOWRITE)
+NC.nc_inq_ndims(ncid)                              # => 1
+NC.nc_inq_nvars(ncid)                              # => 1
+NC.nc_inq_varname(ncid, 0)                         # => "temp"
+NC.nc_inq_vardimid(ncid, 0)                        # => [0]
+NC.nc_inq_dimlen(ncid, 0)                          # => 24
+NC.nc_close(ncid)
+```
+
+Any of these can also be reached as `::NetCDF.nc_*` (unchanged), or through layer 2's higher-level surface.
+
+## Coexisting with `::NetCDF` and future `Numo::NetCDF`
+
+Layer 1 does **not** modify `::NetCDF`. It only adds behavior in the `CA::NetCDF` namespace. So an app that mixes carray-netcdf with a hypothetical `Numo::NetCDF` gem (which would follow the same pattern in its own namespace) can call:
+
+```ruby
+CA::NetCDF.nc_get_vara(ncid, vid, s, c)   # -> CArray
+Numo::NetCDF.nc_get_vara(ncid, vid, s, c)     # -> Numo::NArray  (hypothetical)
+::NetCDF.nc_get_vara(ncid, vid, s, c, buf)    # -> buf, any MemoryView producer (untouched)
+```
+
+with no clashes. This coexistence is exactly why layer 1 avoids monkey-patching `::NetCDF`.

data/docs/07_cookbook.md

@@ -0,0 +1,345 @@
+# Cookbook
+
+End-to-end scenarios. Each example is self-contained; copy, adapt, run.
+
+## 1. Write a CF-1.7 hourly temperature file
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+
+NC::File.create("hourly_temp.nc") do |nc|
+  # Dimensions
+  nc.def_dim("time", :unlimited)
+  nc.def_dim("lat", 73)
+  nc.def_dim("lon", 144)
+
+  # Coordinate variables
+  nc.def_var("time", :int, ["time"],
+             standard_name: "time",
+             units:         "hours since 2022-01-01T00:00:00Z",
+             calendar:      "standard")
+  nc.def_var("lat", :float, ["lat"],
+             standard_name: "latitude",   units: "degrees_north")
+  nc.def_var("lon", :float, ["lon"],
+             standard_name: "longitude",  units: "degrees_east")
+
+  # Data variable
+  nc.def_var("temp", :float, ["time", "lat", "lon"],
+             standard_name: "air_temperature",
+             units:         "K",
+             _FillValue:    -999.9)
+
+  # Globals
+  nc.def_att("Conventions", "CF-1.7")
+  nc.def_att("title",       "Synthetic hourly temperature")
+
+  # Coord data
+  nc["lat"]  = CArray.float32(73)  { |i| -90.0 + i * 2.5 }
+  nc["lon"]  = CArray.float32(144) { |i| i * 2.5 }
+  nc["time"] = CArray.time_series("2022-01-01T00:00:00Z", count: 24, unit: :h)
+
+  # Data (24 hourly slabs)
+  24.times do |t|
+    nc["temp", t, nil, nil] = CArray.float32(73, 144) { |i, j|
+      273.15 + 15.0 * Math.cos(i * 0.05) + 5.0 * Math.sin(j * 0.05 + t * 0.1)
+    }
+  end
+end
+```
+
+Read it back:
+
+```ruby
+NC::File.open("hourly_temp.nc") do |nc|
+  nc.att("Conventions")               # => "CF-1.7"
+
+  t = nc["time"].to_ca
+  t.class                             # => CATime
+  t.first.to_s                        # => "2022-01-01T00:00:00Z"
+  t.last.to_s                         # => "2022-01-01T23:00:00Z"
+
+  temp = nc["temp"].to_ca             # decoded (fill -> mask if present)
+  temp.shape                          # => [24, 73, 144]
+  temp.mean                           # → around 273-285 K
+end
+```
+
+## 2. Read the third time step of a CF file, mask-aware
+
+```ruby
+NC::File.open("hourly_temp.nc") do |nc|
+  slab = nc["temp"][2, nil, nil]      # rank-2 CArray, shape [73, 144]
+  slab.count_masked                   # 0 unless the file has fill cells here
+  slab_at_equator = slab[36, nil]     # rank-1 CArray, shape [144]
+end
+```
+
+## 3. Pack a physical field into a short (satellite-style storage)
+
+Storing a physical range that fits comfortably in `short` saves half the space of `float`. CF's `scale_factor` / `add_offset` handle it:
+
+```ruby
+NC::File.create("packed.nc") do |nc|
+  nc.def_dim("n", 4)
+  nc.def_var("temp", :short, ["n"],
+             units:         "K",
+             scale_factor:  0.01,     # 0.01 K resolution
+             add_offset:    250.0,    # centered around 250 K
+             _FillValue:    -32768,
+             valid_range:   [-10000, 10000])   # ±100 K around offset, in packed space
+  nc.def_att("Conventions", "CF-1.7")
+
+  # Write physical values; the codec packs them into short automatically.
+  nc["temp"] = CA_DOUBLE([250.00, 275.00, 300.00, 325.00])
+end
+```
+
+Round-trip verification:
+
+```ruby
+# Physical view (default codec = CF)
+NC::File.open("packed.nc") do |nc|
+  nc["temp"].to_ca.to_a               # => [250.00, 275.00, 300.00, 325.00]
+end
+
+# Raw view (Identity codec)
+NC::File.open("packed.nc", codec: :identity) do |nc|
+  nc["temp"].to_ca.to_a               # => [0, 2500, 5000, 7500]
+end
+```
+
+Optionally enforce `valid_range` masking:
+
+```ruby
+NC::File.open("packed.nc", codec: [:cf, mask_valid_range: true]) do |nc|
+  nc["temp"].to_ca                    # cells outside [-10000, 10000] packed -> UNDEF
+end
+```
+
+## 4. Stream a large variable one slab at a time
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+
+# Assume /big.nc has variable temp[time, lat, lon] with time = 10000.
+# Compute the daily mean without ever holding the whole array.
+
+means = CArray.float64(10000)
+
+NC::File.open("/big.nc") do |nc|
+  temp = nc["temp"]
+  n_time = temp.shape[0]
+
+  # For zero-allocation streaming, drop to layer 1.  Layer 2's hyperslab
+  # read allocates a fresh CArray per step; layer 1 lets us reuse a buffer.
+  vid = ::NetCDF.nc_inq_varid(nc.ncid, "temp")
+  buf = CArray.float32(1, *temp.shape[1..])   # [1, lat, lon]
+
+  n_time.times do |k|
+    NC.nc_get_vara(nc.ncid, vid, [k, 0, 0], [1, *temp.shape[1..]], buf)
+    means[k] = buf.mean                        # raw physical bytes
+  end
+end
+```
+
+If your file has `_FillValue` or `scale_factor`, layer 1 doesn't apply them — you'd need to either drop that particular streaming loop into layer 2 (which allocates per slab but is one line: `means[k] = nc["temp"][k, nil, nil].mean`), or apply the transforms yourself.
+
+## 4b. Generate a batch of files from a single reference
+
+The classic "read one, write many" pipeline: hold a reference NetCDF open, loop over some units of work, and emit one output per iteration that inherits the reference's skeleton (dims, global atts, coordinate vars) and only diverges in the payload.  `File.create(template:, vars:)` collapses the boilerplate; `clobber: :skip` + `atomic: true` make the loop restartable.
+
+```ruby
+NC::File.open("reference.nc") do |ref|
+  time_windows.each do |window|      # e.g. one 24h chunk per iteration
+    NC::File.create("out_#{window.tag}.nc",
+                    template: ref,          # dims + atts + format inherited
+                    vars: %w[lat lon],      # coord vars: full bit-identical copy
+                    clobber: :skip,         # already-produced files are left alone
+                    atomic: true) do |dst|
+      dst.def_var("time", :int, ["time"],
+                  units: "hours since #{window.origin}", calendar: "standard")
+      dst.def_var("temp", :float, ["time","lat","lon"], _FillValue: -999.9)
+      dst["time"] = window.timestamps
+      window.each_slab { |t, slab| dst["temp", t, nil, nil] = slab }
+    end
+  end
+end
+```
+
+Why the combination matters:
+
+- **`template: ref`** — dims, `Conventions`, `title`, and any custom global attributes carry over.  Output inherits the NetCDF-4 format if the reference is netcdf-4 (so groups / deflate work), unless `netcdf4:` is set explicitly.
+- **`vars: %w[lat lon]`** — coordinate variables that don't change between outputs come along for the ride, byte-identical (no codec on either side).  Downstream diff / dedup tools won't see spurious floating-point drift.
+- **`clobber: :skip`** — reruns pick up where a previous run stopped; finished files are not touched.  Combines cleanly with `atomic:`.
+- **`atomic: true`** — a mid-loop crash leaves neither a half-written output nor a stale one.  `Dir["out_*.nc"]` reflects exactly what completed successfully.
+
+For read-side workflows that consume the output, the split is clean: either the file is fully there (produced by a completed `create` block), or it is not there yet.  No corrupt-file recovery logic in downstream code.
+
+## 5. Append time steps to an unlimited-dimension file
+
+```ruby
+# Existing file created with time = :unlimited.
+NC::File.open("hourly_temp.nc", :write) do |nc|
+  cur = nc.dim("time").length       # e.g. 24
+
+  new_slice = CArray.float32(73, 144) { |i, j| 273.15 + Math.sin(i*0.1) }
+  nc["temp", cur, nil, nil] = new_slice        # writes at t=cur; time grows to cur+1
+
+  nc["time", cur] = CArray.time("2022-01-02T00:00:00Z", unit: :h)[0]
+end
+```
+
+## 6. Group your model by subsystem (NetCDF-4)
+
+```ruby
+NC::File.create("model.nc", netcdf4: true) do |nc|
+  nc.def_dim("time", :unlimited)
+  nc.def_att("Conventions", "CF-1.7")
+
+  ocean = nc.def_group("ocean")
+  ocean.def_dim("depth", 50)
+  ocean.def_var("temperature", :float, ["time", "depth"], units: "degC")
+  ocean.def_var("salinity",    :float, ["time", "depth"], units: "PSU")
+
+  atm = nc.def_group("atmosphere")
+  atm.def_dim("altitude", 30)
+  atm.def_var("pressure",    :float, ["time", "altitude"], units: "hPa")
+  atm.def_var("temperature", :float, ["time", "altitude"], units: "K")
+
+  # Timeline coordinate at the root
+  nc.def_var("time", :int, ["time"],
+             units: "hours since 2024-01-01T00:00:00Z", calendar: "standard")
+
+  # Write some data
+  T = 6
+  nc["time"] = CArray.time_series("2024-01-01T00:00:00Z", count: T, unit: :h)
+
+  T.times do |t|
+    ocean["temperature", t, nil] = CArray.float32(50) { |z| 25.0 - z * 0.4 }
+    ocean["salinity",    t, nil] = CArray.float32(50) { |z| 35.0 + z * 0.01 }
+    atm["pressure",      t, nil] = CArray.float32(30) { |z| 1013.0 * Math.exp(-z / 8.0) }
+    atm["temperature",   t, nil] = CArray.float32(30) { |z| 288.15 - z * 6.5 }
+  end
+end
+```
+
+Read via full path:
+
+```ruby
+NC::File.open("model.nc") do |nc|
+  nc["/ocean/temperature"].shape           # => [6, 50]
+  nc["/atmosphere/pressure"][0, nil]       # 1-D CArray of shape [30]
+  nc.group("ocean").dims["depth"].length   # => 50
+  nc["/time"].to_ca                        # CATime (CF codec applied)
+end
+```
+
+## 7. Inspect a mystery file quickly
+
+```ruby
+NC::File.open("mystery.nc") do |nc|
+  puts "format:      #{nc.format}"
+  puts "conventions: #{nc.att('Conventions') || '(none)'}"
+  puts "codec:       #{nc.codec.class.name.split('::').last}"
+  puts
+
+  puts "dimensions:"
+  nc.dims.each { |n, d| puts "  #{n} = #{d.length}#{' (unlimited)' if d.unlimited?}" }
+  puts
+
+  puts "variables:"
+  nc.vars.each do |n, v|
+    puts "  #{v.data_type} #{n}#{v.shape.inspect}"
+    v.atts.each { |k, val| puts "    :#{k} = #{val.inspect}" }
+  end
+  puts
+
+  puts "global attributes:"
+  nc.atts.each { |k, v| puts "  :#{k} = #{v.inspect}" }
+
+  unless nc.groups.empty?
+    puts
+    puts "groups:"
+    nc.groups.each { |n, g| puts "  #{g.path} (#{g.vars.size} vars, #{g.groups.size} subgroups)" }
+  end
+end
+```
+
+## 8. Round-trip decoded vs raw for a single variable
+
+Handy when a file behaves unexpectedly and you want to see what the codec is doing:
+
+```ruby
+NC::File.open("data.nc") do |nc|
+  decoded = nc["temp"].to_ca
+  puts "decoded: mean = #{decoded.mean}, masked = #{decoded.count_masked}"
+end
+
+NC::File.open("data.nc", codec: :identity) do |nc|
+  raw = nc["temp"].to_ca
+  puts "raw: mean = #{raw.mean}, min = #{raw.min}, max = #{raw.max}"
+  puts "attrs: #{nc['temp'].atts.inspect}"
+end
+```
+
+## 9. Write a small file with a bit-exact custom codec
+
+```ruby
+# A minimal codec that only masks _FillValue but skips scale/offset/time.
+class OnlyFillCodec
+  def decode(var, ca)
+    fill = var.att("_FillValue") or return ca
+    out = ca.copy
+    out[:eq, fill] = UNDEF
+    out
+  end
+
+  def encode(var, ca)
+    return ca unless ca.is_a?(CArray) && ca.has_mask?
+    fill = var.att("_FillValue") or
+      raise ArgumentError, "no _FillValue to encode masked cells"
+    ca.strip_mask(fill)
+  end
+end
+
+NC::File.open("packed.nc", codec: OnlyFillCodec.new) do |nc|
+  raw_with_mask = nc["temp"].to_ca    # packed short values, fill cells masked,
+                                      # no scale/offset applied
+end
+```
+
+The codec object just has to respond to `decode(var, ca)` and `encode(var, ca)`. See the [Codecs chapter](04_codec.md) for the full interface and how to register a codec globally.
+
+## 10. Existing-file inspection with `ncdump`-like output
+
+```ruby
+def cdl_summary(nc)
+  puts "netcdf {"
+  puts "dimensions:"
+  nc.dims.each { |n, d|
+    len = d.unlimited? ? "UNLIMITED" : d.length.to_s
+    puts "  #{n} = #{len};"
+  }
+  puts "variables:"
+  nc.vars.each { |n, v|
+    dims = v.dims.map(&:name).join(", ")
+    puts "  #{v.data_type} #{n}(#{dims});"
+    v.atts.each { |k, val|
+      puts "    #{n}:#{k} = #{val.inspect};"
+    }
+  }
+  puts "// global attributes:"
+  nc.atts.each { |k, v| puts "  :#{k} = #{v.inspect};" }
+  nc.groups.each { |gname, g|
+    puts
+    puts "group: #{gname} {"
+    cdl_summary(g)
+    puts "} // group #{gname}"
+  }
+  puts "}"
+end
+
+NC::File.open("mystery.nc") { |nc| cdl_summary(nc) }
+```