carray-netcdf 1.0.13 → 2.0.0

data/docs/03_writing.md

@@ -0,0 +1,418 @@
+# Writing files
+
+Writing a file is a schema followed by data, and the gem keeps the two phases in the order NetCDF wants:
+
+```
+NC::File.create(path) do |nc|
+  nc.def_dim("time", :unlimited)          # the schema: dimensions,
+  nc.def_var("temp", :float, [...])       # variables,
+  nc.def_att("Conventions", "CF-1.7")     # and attributes
+
+  nc["temp"] = ca                         # then the data
+end
+```
+
+`create` opens the file in define mode, so the definitions come first. You do not have to leave define mode yourself: the first data write does it, and a later definition puts the file back. The block closes the file when it ends.
+
+What you assign is a CArray, and the codec encodes it on the way in — the reverse of the read path, so a CArray of physical values can be stored packed, and a masked entry becomes the variable's `_FillValue`.
+
+## Creating
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+
+# Classic NetCDF-3
+NC::File.create("out.nc") do |nc|
+  # ... define dims/vars/atts, then write data
+end
+
+# NetCDF-4 (needed for groups, compression, unlimited multi-D, etc.)
+NC::File.create("out.nc", netcdf4: true) do |nc|
+  # ...
+end
+
+# Refuse to overwrite an existing file
+NC::File.create("out.nc", clobber: false)
+
+# Atomic write: publish only on success.  Bytes go to a same-directory
+# tmp file; a successful close renames tmp -> "out.nc" (POSIX-atomic).
+# Any exception from the block unlinks the tmp; the target file is
+# untouched (an earlier version, if any, survives intact).
+NC::File.create("out.nc", atomic: true) do |nc|
+  # ... define + write.  On raise: tmp cleaned up, "out.nc" NOT created /
+  # overwritten.  On success: "out.nc" appears atomically at end of block.
+end
+```
+
+**Atomic write caveat.** Consumers polling the target path won't see anything until close — for a long-running unlimited-dim streaming write where a downstream reader wants incremental visibility, prefer `atomic: false` (the default).
+
+`create` opens the file in **define mode**. The block is the natural place to declare the schema; `File` will drop out of define mode automatically before the first data write, and back into it for further definitions.
+
+## Defining dimensions
+
+```ruby
+nc.def_dim("time", :unlimited)   # unlimited (grows as you write)
+nc.def_dim("lat", 73)            # fixed length
+nc.def_dim("lon", 144)
+```
+
+`:unlimited` (or the string `"unlimited"`) creates an unlimited dimension. Every other value is coerced with `Integer(...)`. `def_dim` returns the `Dim` and stores it under `nc.dims[name]`.
+
+## Defining variables
+
+```ruby
+nc.def_var("temp", :float, ["time", "lat", "lon"],
+           units: "K",
+           long_name: "air temperature",
+           _FillValue: -999.9,
+           valid_range: [200.0, 350.0])
+```
+
+- 1st arg: variable name (String).
+- 2nd arg: type — a Symbol from `:byte :short :int :int64 :float :double :ubyte :ushort :uint :uint64 :char` (plus the CArray-style aliases `:int8 :uint8 :int16 :uint16 :int32 :uint32 :float32 :float64`).
+- 3rd arg: list of dimension names (Strings). An empty list makes a scalar.
+- Keyword args: attributes to attach at define time. Any Ruby-side Numeric, String, or Array is fine.
+
+**Type-matched attributes.** `_FillValue`, `missing_value`, `valid_range`, `valid_min`, and `valid_max` must be written in the variable's on-disk type (CF §2.5.1 and libnetcdf enforce this). The gem handles that for you: it detects those attribute names and writes them via typed `nc_put_att_*`, so `_FillValue: -999.9` on a `:float` var becomes `nc_put_att_float`, not the default `nc_put_att_double`. Other attributes go through the generic `nc_put_att` (which infers a reasonable NetCDF type from the Ruby value).
+
+**NetCDF-4 storage / filter options.** `def_var` recognizes these keyword arguments alongside the attributes (only effective on a NetCDF-4 file):
+
+| Kwarg | Meaning |
+|---|---|
+| `chunks: [1, 73, 144]` | chunk sizes (`NC_CHUNKED`) |
+| `chunks: :contiguous` | force `NC_CONTIGUOUS` |
+| `chunks: :compact` | `NC_COMPACT` (small variables only) |
+| `deflate: 4` | zlib compression level 1..9 (0 / omitted / `nil` = off) |
+| `shuffle: true` | shuffle filter (usually paired with deflate) |
+| `fletcher32: true` | per-chunk fletcher32 checksum |
+| `endian: :little / :big / :native` | on-disk byte order |
+| `no_fill: true` | disable pre-fill (perf: skip fill on unwritten cells) |
+
+```ruby
+NC::File.create("compressed.nc", netcdf4: true) do |nc|
+  nc.def_dim("time", :unlimited); nc.def_dim("lat", 73); nc.def_dim("lon", 144)
+  nc.def_var("temp", :float, ["time", "lat", "lon"],
+             units: "K",
+             chunks: [1, 73, 144],
+             deflate: 4,
+             shuffle: true)
+end
+```
+
+These get passed through to `nc_def_var_chunking` / `nc_def_var_deflate` / `nc_def_var_fletcher32` / `nc_def_var_endian` / `nc_def_var_fill` and take effect immediately.  On a NetCDF-3 file, any of them causes libnetcdf to raise; that raise propagates unchanged.
+
+**File-wide defaults.** The same kwargs on `File.create` set defaults for every subsequent `def_var` in that file — write once, no need to repeat per variable:
+
+```ruby
+NC::File.create("compressed.nc", netcdf4: true,
+                deflate: 4, shuffle: true) do |nc|
+  nc.def_dim("time", :unlimited); nc.def_dim("lat", 73); nc.def_dim("lon", 144)
+  nc.def_var("temp", :float, ["time", "lat", "lon"])     # inherits deflate=4, shuffle=true
+  nc.def_var("wind", :float, ["time", "lat", "lon"])     # same
+  nc.def_var("mask", :byte,  ["lat", "lon"], deflate: 0) # per-var wins: no compression
+end
+```
+
+Per-var kwargs on `def_var` always override the file-wide default (including `deflate: 0` / `shuffle: false` to explicitly turn off). Note that "file-wide compression" is a convention: libnetcdf has no file-level compression attribute — internally we just call `nc_def_var_deflate` on each new variable with the file's default.  If you `def_var` in a second session (after re-open), the earlier file default is *not* remembered and applies only within the create block.
+
+Add attributes after the fact with `Var#def_att`:
+
+```ruby
+nc.def_var("temp", :float, ["time", "lat", "lon"], units: "K")
+nc["temp"].def_att("long_name", "air temperature")
+```
+
+## Inheriting from an existing file (`template:`)
+
+The most common workflow — "read one NetCDF, transform, write another with the same skeleton" — is a single kwarg away.  Every `def_var` accepts a `template:` argument that supplies its definition (type, dims, attributes) from an existing `Var` (or from a `File` / `Group` by same-name lookup):
+
+```ruby
+NC::File.open("in.nc") do |src|
+  NC::File.create("out.nc") do |dst|
+    dst.def_dim("lat", 3)
+    dst.def_var("lat", template: src["lat"])          # copy definition
+    dst.def_var("lat", template: src)                  # look up "lat" in src
+    dst.def_var("latitude", template: src["lat"])     # rename on copy
+    dst.def_var("lat", template: src["lat"],           # override attributes
+                units: "arc_degrees_north")
+    dst.def_var("lat", :double, template: src["lat"]) # override type
+  end
+end
+```
+
+Positional args (`type` / `dims`) and kwargs override what the template provides.  `dims:` and `type:` are also accepted as keyword arguments when mixing with `template:` reads better than positional:
+
+```ruby
+dst.def_var("lat", template: src["lat"], dims: ["latitude"])
+```
+
+`copy_var` combines the definition copy above with a bit-identical raw data copy — perfect for coordinate variables that pass through unchanged:
+
+```ruby
+dst.copy_var(src["lat"])                              # same name, values included
+dst.copy_var(src["lat"], name: "latitude")            # rename
+dst.copy_var("lat", from: src)                        # look up by name
+%w[time lat lon].each { |n| dst.copy_var(n, from: src) }
+```
+
+The data goes through the source's `raw` (no decode) and layer 1 (no encode), so the on-disk bytes match exactly — a codec-mediated variable (scale/offset, `_FillValue`) round-trips without any floating-point drift.  If you want a codec-mediated copy, drop back to `def_var(template:)` and write `dst[name] = src[name].to_ca` yourself.
+
+`copy_dim` copies a dimension (including the unlimited flag) but not its coordinate variable:
+
+```ruby
+dst.copy_dim(src.dim("time"))                         # unlimited stays unlimited
+dst.copy_dim("lat", from: src)
+```
+
+**File-level template.**  `File.create` accepts `template:` for whole-file skeleton inheritance — dims, global attributes, and (if given in `vars:`) coordinate variables are copied before your block runs:
+
+```ruby
+NC::File.open("in.nc") do |src|
+  NC::File.create("out.nc",
+                  template: src,
+                  vars: %w[time lat lon]) do |dst|
+    # dims, global atts, and coord vars are already there
+    dst.def_var("result", :float, ["time","lat","lon"], units: "1")
+    dst["result"] = compute(src["temp"].to_ca)
+  end
+end
+```
+
+`template:` also works with `dims:`, `atts:`, and format inheritance:
+
+| Kwarg | Meaning |
+|---|---|
+| `template: src` | inherit dims, atts, and format (netcdf-4 upgrades) |
+| `dims: true` (default) | copy all template dims |
+| `dims: false` | copy no dims |
+| `dims: %w[time lat]` | copy just these dims |
+| `atts: true` (default) | copy all global atts |
+| `atts: false` | copy no atts |
+| `atts: { "history" => "..." }` | copy all, then apply Hash overrides |
+| `vars: %w[time lat lon]` | after dims/atts, `copy_var` each of these |
+
+Explicit `netcdf4:` always wins over template inheritance.
+
+## Idempotent batch creation (`clobber: :skip`)
+
+For pipelines that generate one output file per input chunk, `clobber:` also accepts `:skip` — if the target already exists, the block is not called and `File.create` returns `nil`:
+
+```ruby
+NC::File.open("reference.nc") do |ref|
+  time_windows.each do |window|
+    NC::File.create("out_#{window.tag}.nc",
+                    template: ref,
+                    vars: %w[lat lon],
+                    clobber: :skip,
+                    atomic: true) do |dst|
+      # only runs for windows not already produced
+      dst.def_var("time", :int, ["time"],
+                  units: "hours since #{window.origin}", calendar: "standard")
+      dst.def_var("temp", :float, ["time","lat","lon"], _FillValue: -999.9)
+      window.slabs { |t, slab| dst["temp", t, nil, nil] = slab }
+    end
+  end
+end
+```
+
+With `atomic: true` on top, a mid-loop crash leaves neither a partial file nor a stale one — `Dir["out_*.nc"]` tells the truth about what completed.  See the [cookbook batch chapter](07_cookbook.md) for a full example.
+
+## Defining global attributes
+
+```ruby
+nc.def_att("Conventions", "CF-1.7")
+nc.def_att("title", "Reanalysis subset")
+nc.def_att("history", "Created #{Time.now}")
+```
+
+`def_att` on the File writes to `NC_GLOBAL`. On a `Group` it writes to that group's global-attribute set (see [Groups](05_groups.md)).
+
+## Writing data
+
+### Whole-variable write
+
+```ruby
+nc["temp"] = CArray.float32(1, 73, 144) { |i, j, k| some_data(i, j, k) }
+```
+
+The CArray shape must match the variable's shape. Non-contiguous CArrays (views) are automatically materialized to a contiguous buffer before the write.
+
+### Hyperslab write
+
+```ruby
+# Write one time step
+nc["temp", 0, nil, nil] = one_slice     # one_slice.shape == [73, 144]
+
+# Write ten time steps starting at t=5
+nc["temp", 5..14, nil, nil] = ten_days  # ten_days.shape == [10, 73, 144]
+
+# Write a single point
+nc["temp", 0, 0, 0] = 273.15
+```
+
+Under the hood the index dispatches to `nc_put_var1` (single point), `nc_put_vara` (contiguous slab), or `nc_put_vars` (strided slab).
+
+### Unlimited-dimension streaming
+
+An unlimited dimension grows to fit whatever you write:
+
+```ruby
+NC::File.create("stream.nc") do |nc|
+  nc.def_dim("time", :unlimited)
+  nc.def_dim("station", 10)
+  nc.def_var("obs", :double, ["time", "station"])
+
+  # Write day-by-day.  netcdf grows the time dimension automatically.
+  100.times do |day|
+    nc["obs", day, nil] = daily_observations(day)   # shape [10]
+  end
+end
+```
+
+You don't need to declare the eventual size — writing beyond the current length grows the unlimited dimension. Every fixed-size dimension must match the write.
+
+### Writing with a mask
+
+Layer 2 handles masks for you via the codec. On a variable with a `_FillValue` attribute, `UNDEF` cells are encoded to the fill sentinel on write; on read, fill cells decode back to `UNDEF` (see [Codecs](04_codec.md)):
+
+```ruby
+NC::File.create("masked.nc") do |nc|
+  nc.def_dim("n", 5)
+  nc.def_var("v", :float, ["n"], _FillValue: -999.9)
+  nc.def_att("Conventions", "CF-1.7")
+
+  ca = CA_FLOAT([1.0, 2.0, 3.0, 4.0, 5.0])
+  ca[2] = UNDEF
+  nc["v"] = ca            # cell [2] is written as -999.9 on disk
+end
+```
+
+Writing a masked CArray to a variable without a `_FillValue` attribute raises — the gem refuses to invent a sentinel silently.
+
+## Type-matched writes and libnetcdf coercion
+
+You can hand libnetcdf any numeric type; it converts on the way out to match the on-disk type:
+
+```ruby
+nc.def_var("count", :int, ["time"])
+nc["count"] = CArray.int64(...)      # OK, libnetcdf narrows int64 -> int32
+
+nc.def_var("temp", :float, ["time"])
+nc["temp"] = CArray.float64(...)     # OK, libnetcdf narrows double -> float
+```
+
+If a value would overflow the target type, libnetcdf raises. When you're writing physical values into a **packed** (scale/offset) variable, the codec divides by `scale_factor`, subtracts `add_offset`, and rounds to the target integer type before the write — see the [Codecs chapter](04_codec.md).
+
+## Modifying an existing file
+
+```ruby
+NC::File.open("data.nc", :write) do |nc|
+  nc["temp", 0, nil, nil] = new_slice     # overwrite a slab
+  nc.def_att("history", "updated #{Time.now}")   # new global attribute
+  nc["temp"].def_att("comment", "revised on ingest")
+end
+```
+
+`File.open(:write)` opens for modification. Adding a new dimension or variable is possible too — `File` re-enters define mode as needed.
+
+Adding a new variable to an existing NetCDF-3 file rewrites the file layout (a libnetcdf constraint, not ours). NetCDF-4 files (`netcdf4: true`) are more flexible.
+
+## Zarr stores
+
+libnetcdf 4.8 and later can read and write Zarr.  libnetcdf calls this support NCZarr.  A Zarr store is a directory, not one file: each chunk of a variable is a file of its own, and the metadata is in small JSON files.
+
+The gem has no separate API for it.  Pass a URL where a path would go, and give the format in its `#mode=` part:
+
+```ruby
+url = "file:///data/t.zarr#mode=zarr,file"
+
+NC::File.create(url, netcdf4: true) do |nc|
+  nc.def_dim("y", 2)
+  nc.def_dim("x", 3)
+  nc.def_var("t", :short, %w[y x], scale_factor: 0.5, _FillValue: -1)
+  nc.def_att("Conventions", "CF-1.8")
+  nc["t"] = ca
+end
+
+NC::File.open(url)          { |nc| nc["t"][] }        # decoded as from a .nc file
+NC::File.open(url, :write)  { |nc| nc["t", 0, nil] = row }
+```
+
+- `mode=zarr` reads and writes plain Zarr, the form other tools such as xarray use.  `mode=nczarr` also writes netCDF's own metadata, such as shared dimensions, into the store.
+- `file` puts the store in a local directory.  `s3` puts it in Amazon S3, when libnetcdf was built with S3 support.
+- The `#mode=` part is required.  A plain path ending in `.zarr` creates an ordinary HDF5 file.
+
+Dimensions, attributes, groups, chunking, compression, the codec and hyperslab reads and writes all work as for a NetCDF-4 file.  User-defined types do not.
+
+`atomic:` and `clobber: false` / `clobber: :skip` need a file path, so `File.create` raises `ArgumentError` when they come with a URL.  To keep an existing store, check for its directory yourself before calling `create`.
+
+This needs a libnetcdf built with NCZarr.  `nc-config --has-nczarr` says whether yours was.
+
+## A complete CF-compliant example
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+
+NC::File.create("cf_demo.nc") do |nc|
+  # Dimensions
+  nc.def_dim("time", :unlimited)
+  nc.def_dim("lat", 73)
+  nc.def_dim("lon", 144)
+
+  # Coordinate variables
+  nc.def_var("time", :int, ["time"],
+             standard_name: "time",
+             long_name:     "time",
+             units:         "hours since 2022-01-01T00:00:00Z",
+             calendar:      "standard")
+  nc.def_var("lat", :float, ["lat"],
+             standard_name: "latitude",
+             long_name:     "latitude",
+             units:         "degrees_north")
+  nc.def_var("lon", :float, ["lon"],
+             standard_name: "longitude",
+             long_name:     "longitude",
+             units:         "degrees_east")
+
+  # Data variable
+  nc.def_var("temp", :float, ["time", "lat", "lon"],
+             standard_name: "air_temperature",
+             long_name:     "surface air temperature",
+             units:         "K",
+             _FillValue:    -999.9,
+             valid_range:   [200.0, 350.0])
+
+  # Global attributes
+  nc.def_att("Conventions", "CF-1.7")
+  nc.def_att("title",       "Synthetic CF demo")
+  nc.def_att("history",     "Created #{Time.now.utc.strftime('%Y-%m-%dT%H:%M:%SZ')}")
+
+  # Coordinate data
+  nc["lat"] = CArray.float32(73) { |i| -90.0 + i * 2.5 }
+  nc["lon"] = CArray.float32(144) { |i| i * 2.5 }
+  nc["time"] = CArray.time_series("2022-01-01T00:00:00Z", count: 24, unit: :h)
+
+  # Data — 24 hourly slices
+  24.times do |t|
+    nc["temp", t, nil, nil] = CArray.float32(73, 144) { |i, j|
+      273.15 + 20.0 * Math.sin(i * 0.1 + t * 0.2) + Math.cos(j * 0.05)
+    }
+  end
+end
+```
+
+Round-tripping this file:
+
+```ruby
+NC::File.open("cf_demo.nc") do |nc|
+  nc.att("Conventions")          # => "CF-1.7"
+
+  t = nc["time"].to_ca            # => CATime, hour resolution
+  t[0]                            # => #<CATime::Element 2022-01-01T00:00:00Z>
+
+  temp = nc["temp"].to_ca         # decoded; fill sentinel -> mask
+  temp.mean                       # centered around 273-290 K
+end
+```