carray-netcdf 1.0.13 → 2.0.0

data/docs/08_cheatsheet.md

@@ -0,0 +1,326 @@
+# API Cheatsheet
+
+Tables of everything this gem gives you to call. Read [getting started](01_getting_started.md) first if you want narrative; this file is pure lookup.
+
+Conventions:
+
+- **L0** = the [netcdf](https://github.com/himotoyoshi/netcdf) gem (`::NetCDF`), a thin libnetcdf binding.
+- **L1** = `CA::NetCDF` module functions; the L0 API with the array I/O calls overridden to speak CArray. Everything else passes through because L1 `include`s L0.
+- **L2** = `CA::NetCDF::{File, Group, Var, Dim, Codec}` classes.
+
+In every table, `→` is the return value.
+
+---
+
+## L0 — the netcdf gem
+
+Layer 0 is the [netcdf](https://github.com/himotoyoshi/netcdf) gem itself: `::NetCDF`, one module function per libnetcdf function, plus the `NC_*` constants. Its own reference lists all of them, and is the place to look them up.
+
+Reaching for L0 from here is the exception, not the rule. Anything it can do that layer 2 cannot is a gap worth reporting; the layers below are there to be built on, not to be called around.
+
+## L1 — `CA::NetCDF`
+
+The whole `::NetCDF` module is `include`d into `CA::NetCDF`, so everything from L0 above is callable — as `CA::NetCDF.nc_open(…)` or via mixin (`include CA::NetCDF` and call `nc_open(…)` bare). Only the array I/O calls are overridden to speak CArray.
+
+### Overridden functions
+
+| Call | Behavior |
+|---|---|
+| `nc_get_var(ncid, varid, buffer=nil)` | Without buffer: L1 allocates a CArray of the variable's type (`NC_CHAR` → `:uint8`) and reads into it. With CArray buffer: read into it. |
+| `nc_get_vara(ncid, varid, start, count, buffer=nil)` | Same, hyperslab. |
+| `nc_get_vars(ncid, varid, start, count, stride, buffer=nil)` | Same, strided. |
+| `nc_get_varm(ncid, varid, start, count, stride, imap, buffer=nil)` | Same, mapped. |
+| `nc_put_var(ncid, varid, ca)` | Contiguous CArray passes through. Non-contiguous view → `.copy`. Masked → `ArgumentError` (mask/fill is L2). |
+| `nc_put_vara(ncid, varid, start, count, ca)` | Same. |
+| `nc_put_vars(ncid, varid, start, count, stride, ca)` | Same. |
+| `nc_put_varm(ncid, varid, start, count, stride, imap, ca)` | Same. |
+
+Everything else on `::NetCDF` (see [L0](#l0--the-netcdf-gem) above) is a passthrough. All the `NC_*` constants are visible via the include:
+
+```ruby
+CA::NetCDF::NC_DOUBLE       # => 6
+CA::NetCDF::NC_CLOBBER      # => 0
+```
+
+### Two idioms
+
+```ruby
+# call style
+CA::NetCDF.nc_get_vara(ncid, varid, [0, 0], [3, 4])   # => CArray
+
+# mixin style
+class Reader
+  include CA::NetCDF
+  def read(ncid, varid) = nc_get_var(ncid, varid)          # bare -> CArray
+  def define(ncid, ...) = nc_def_dim(ncid, "time", 100)   # passthrough
+end
+```
+
+---
+
+## L2 — `CA::NetCDF::File / Group / Var / Dim / Codec`
+
+### `CA::NetCDF::File` (class methods)
+
+| Call | → |
+|---|---|
+| `File.open(path, mode = :read, codec: :auto)` | File; `mode` is `:read` / `:write` / `:append` |
+| `File.open(path, ...) { \|nc\| ... }` | value of block; auto-closes on exit / exception |
+| `File.create(path, codec: :auto, clobber: true, netcdf4: false, atomic: false, template: nil, dims: true, atts: true, vars: [], **var_defaults)` | File in define mode.  See details below. |
+| `File.create(path, ...) { \|nc\| ... }` | value of block; auto-closes.  Under `atomic: true`: commits on successful block exit, aborts (unlinks tmp) on exception.  Under `clobber: :skip` with existing target: block is not called; returns `nil`. |
+
+Kwargs on `File.create`:
+
+| Kwarg | Meaning |
+|---|---|
+| `codec:` | codec dispatch (`:auto` / `:identity` / `:cf` / `[:cf, opts]` / ...) — see [codec.md](04_codec.md) |
+| `clobber: true` (default) | overwrite an existing target file |
+| `clobber: false` | raise `Errno::EEXIST` if target exists |
+| `clobber: :skip` | if target exists, don't call the block, return `nil` (restartable loops) |
+| `netcdf4: true` | create as NetCDF-4 (auto-inherited from `template:` if not set) |
+| `atomic: true` | write to same-dir tmp; rename on successful close; unlink tmp on exception |
+| `template: file_or_group` | inherit skeleton (dims + atts + format) from another open File/Group |
+| `dims: true / false / [names]` | which template dims to copy (default all) |
+| `atts: true / false / {overrides}` | which template atts to copy; Hash merges overrides |
+| `vars: [names]` | template vars to fully copy (definition + bit-identical data) |
+| `chunks: / deflate: / shuffle: / fletcher32: / endian: / no_fill:` | file-wide var defaults, applied to every `def_var` (per-var kwarg wins) |
+
+`path` may be a Zarr URL (`"file:///data/t.zarr#mode=zarr,file"`) for `open` and `create` alike; `atomic:` and `clobber: false / :skip` then raise `ArgumentError`.  See [Zarr stores](03_writing.md#zarr-stores).
+
+### `File` instance (adds these to what `Group` provides)
+
+| Call | → |
+|---|---|
+| `#filename` | on-disk path |
+| `#format` | `nc_inq_format(ncid)` |
+| `#conventions` | tokenized `Conventions` attribute (`["CF-1.7", "ACDD-1.3"]`); `[]` if absent |
+| `#codec` | resolved codec instance (`Codec::Identity` / `::CF` / `::COARDS`) |
+| `#close` | close the file (idempotent).  For an atomic-write File, renames tmp → filename here. |
+| `#abort` | atomic-write only: close without renaming, unlink the tmp.  Idempotent. |
+| `#closed?` | Boolean |
+| `#atomic?` | Boolean — is this an atomic-write File that has not yet committed? |
+| `#tmp_path` | the tmp path an atomic write is going to (nil for regular / after commit) |
+| `#var_defaults` | file-wide var option defaults (frozen Hash) |
+| `#ensure_define_mode` / `#ensure_data_mode` | manual mode transitions (rarely needed) |
+| `#__transaction__(define: true) { ... }` | run block in define or data mode |
+
+### `Group` (base for `File`; both act as the root group in a NetCDF-4 file)
+
+**Identity**
+
+| Call | → |
+|---|---|
+| `#ncid` | raw group ncid |
+| `#file` | root `File` |
+| `#parent` | parent `Group`, `nil` for root |
+| `#name` | `"foo"` (or `"/"` for root) |
+| `#path` | `"/foo/bar"` — hierarchy path |
+| `#root?` | Boolean |
+
+**Introspection (this group only)**
+
+| Call | → |
+|---|---|
+| `#dims` | `{String => Dim}` |
+| `#vars` | `{String => Var}` |
+| `#groups` | `{String => Group}` |
+| `#atts` | `{String => value}` (frozen), this group's globals |
+| `#dim(name)` / `#var(name)` / `#group(name)` / `#att(name)` | one lookup or `nil` |
+| `#dim?(name)` / `#var?(name)` / `#group?(name)` | Boolean |
+| `#resolve_dim(name)` | walk up ancestors — raises if not found |
+| `#codec` | delegates to `file.codec` |
+
+**Access**
+
+| Call | → |
+|---|---|
+| `#[](name)` | Var — accepts `"temp"`, `"foo/temp"`, `"/foo/temp"` |
+| `#[]=(name, ca)` | whole-variable write |
+| `#[]=(name, *index, ca)` | hyperslab write (delegates to `Var#[]=`) |
+
+**Definition**
+
+| Call | → |
+|---|---|
+| `#def_dim(name, length)` | Dim; `length: :unlimited` or an integer |
+| `#def_var(name, type, dims=[], **atts_or_opts)` | Var; type is a Symbol (`:float`, `:short`, …).  Reserved kwargs (netcdf-4): `chunks:` / `deflate:` / `shuffle:` / `fletcher32:` / `endian:` / `no_fill:`.  Every other kwarg is a variable attribute. |
+| `#def_var(name, template: src_var, **overrides)` | copy a definition (type, dims, attributes) from another Var; `**overrides` merges/replaces atts.  `type` and `dims:` can also override. |
+| `#def_att(name, value)` | value; writes to this group's `NC_GLOBAL` |
+| `#def_group(name)` | Group; creates a subgroup |
+| `#copy_dim(dim_or_name, from: nil, name: nil)` | copy a dimension (unlimited status preserved); optionally rename |
+| `#copy_var(var_or_name, from: nil, name: nil, **overrides)` | copy a variable's definition **AND** its data bit-identically (raw + layer-1 write, both codec-bypassed); optionally rename / override attrs |
+
+### `Var`
+
+**Identity + metadata**
+
+| Call | → |
+|---|---|
+| `#name` | variable name |
+| `#shape` | Array of Integer (refreshed on every `[]`/`[]=`, so unlimited dim growth is visible) |
+| `#ndim` | rank |
+| `#nc_type` | raw NC_* integer |
+| `#data_type` | CArray Symbol (`:float32`, `:int16`, …) |
+| `#dims` | Array of Dim (with dim inheritance resolved) |
+| `#atts` | `{String => value}` (frozen) |
+| `#att(name)` | one attribute or `nil` |
+| `#file` / `#group` | enclosing File / Group |
+| `#codec` | delegates to `file.codec` |
+
+**Read (`[]` — lazy hyperslab + decode)**
+
+| Call | → |
+|---|---|
+| `#to_ca` | whole-var, decoded CArray |
+| `#[]` | ≡ `#to_ca` |
+| `#[i, j, k]` (all integers) | single decoded scalar (Numeric / UNDEF / CATime::Element) |
+| `#[i, nil, ...]` | slab (integer + `nil`) — hyperslab read |
+| `#[a..b, ...]` | range → hyperslab |
+| `#[(a..b).step(s), ...]` | strided hyperslab |
+| `#[bool_or_fancy]` | reads whole var + does CArray indexing on the result |
+| `#raw` / `#raw(*idx)` | same indexing as `#[]`, but **bypasses the codec**; `v[...] == v.codec.decode(v, v.raw(...))` |
+| `#method_missing(id, *)` | forwards to `to_ca.__send__(id, *)` — `var.mean`, `var.sum`, …; in-place methods (`fill`, `*!`, `*=`) raise |
+
+**Write (`[]=`)**
+
+| Call | → |
+|---|---|
+| `#[]= (value)` | whole-var write (encoded) |
+| `#[i, j, k]= scalar` | one cell (encoded through codec) |
+| `#[i, nil, ...]= slab` | hyperslab write |
+| `#[a..b, ...]= slab` | range hyperslab |
+| `#def_att(name, value)` | attach an attribute |
+
+### `Dim`
+
+| Call | → |
+|---|---|
+| `#name` | dimension name |
+| `#length` | current length |
+| `#unlimited?` | Boolean |
+| `#to_i` | raw dimid |
+| `#group` | enclosing group |
+| `#var` | coordinate variable of the same name (in the same group) or `nil` |
+| `#to_ca` | coordinate values (`var.to_ca`) or `nil` |
+
+### `Codec` — value transforms
+
+**Resolver**
+
+| Value passed as `codec:` | Resolves to |
+|---|---|
+| `:auto` (default) | Match by `Conventions` attribute → `Identity` fallback |
+| `:identity`, `false`, `nil` | `Identity` |
+| `:cf` | `Codec::CF.new` |
+| `:coards` | `Codec::COARDS.new` |
+| `[:cf, opts]` | `Codec::CF.new(**opts)` |
+| `[:coards, opts]` | `Codec::COARDS.new(**opts)` |
+| a stateless codec `Module` (e.g. `Codec::Identity`) | itself |
+| any object with `#decode(var, ca)` and `#encode(var, ca)` | itself |
+
+**Interface (what every codec must implement)**
+
+| Class method | Purpose |
+|---|---|
+| `self.claims?(conventions_string)` | Should this codec claim a file whose `Conventions` attribute equals this string?  (Only the string; deliberately no file-inspection hook — that would push the codec toward CDM Layer 2 territory.) |
+| Instance methods | |
+| `#decode(var, ca)` | raw on-disk → user-facing value (var-local: read `var.atts` only, never walk to `var.file`) |
+| `#encode(var, ca)` | user-facing → raw on-disk (same contract) |
+
+**Registration**
+
+```ruby
+CA::NetCDF::Codec.register(MyCodec)   # later registrations win on `applies?`
+```
+
+**Built-in: `Codec::Identity`** — module, stateless; `decode`/`encode` pass values through unchanged.
+
+**Built-in: `Codec::CF`** (class)
+
+| Call | Purpose |
+|---|---|
+| `CF.new(mask_valid_range: false)` | Options |
+| `CF.claims?(s)` | `"CF-*"`, `"CF "`, `"CF"` |
+| `#mask_valid_range` | Boolean reader |
+| `#decode(var, ca)` | fill → mask, (opt-in) valid_range → mask, unpack scale/offset, CF time → CATime |
+| `#encode(var, ca)` | reverse (CATime → count, pack, mask → fill) |
+
+CF attributes recognized (decode order):
+
+| Attribute | Effect (decode → encode) |
+|---|---|
+| `_Unsigned` | signed on-disk → unsigned user (bit-reinterpret); reverse on write.  Runs FIRST so fill/valid/scale see numerically-correct values. |
+| `_FillValue`, `missing_value` | mask ↔ sentinel (in unsigned space if `_Unsigned` was set) |
+| `valid_range` / `valid_min` / `valid_max` | (opt-in) out-of-range → UNDEF, same space as fill; encode not defined by CF |
+| `scale_factor`, `add_offset` | `v = packed*sf + ao` ↔ `packed = round((v-ao)/sf)` (round only when var is integer) |
+| `units: "<u> since <ref>"` + `calendar` | CATime ↔ count-since-ref int |
+| `units` = anything else | raw (no UDUNITS conversion) |
+
+Supported calendars: `standard`, `gregorian`, `proleptic_gregorian`, missing → CATime (proleptic Gregorian, UTC). Everything else (`noleap`, `360_day`, `julian`, …) warns once and returns raw values on read; write raises.
+
+CF time unit words → CATime base:
+
+| Word (+ common plural/abbrev) | CATime base |
+|---|---|
+| `year`, `years`, `yr`, `yrs`, `y` | `:Y` |
+| `month`, `months`, `mon`, `mons` | `:M` |
+| `day`, `days`, `d` | `:D` |
+| `hour`, `hours`, `hr`, `hrs`, `h` | `:h` |
+| `minute`, `minutes`, `min`, `mins` | `:m` |
+| `second`, `seconds`, `sec`, `secs`, `s` | `:s` |
+| `millisecond` … `msec` … `ms` | `:ms` |
+| `microsecond` … `usec` … `us` | `:us` |
+| `nanosecond` … `nsec` … `ns` | `:ns` |
+
+An `<count> <base>` form like `"5 days since 2024-06-15"` is stored as CATime at the base (`:D`) with the count (`5`) folded into the numeric values — see [codec.md](04_codec.md#n-of-a-unit-intervals-5-days-since-).
+
+**Built-in: `Codec::COARDS < CF`** — same transforms; `claims?` matches `"COARDS"` and `"COARDS/*"`.
+
+---
+
+## Type-name conversion tables
+
+**Symbol / String → nc_type** (accepted by `def_var`)
+
+| Ruby type name (Symbol or String) | nc_type constant |
+|---|---|
+| `:byte`, `:int8` | `NC_BYTE` |
+| `:ubyte`, `:uint8` | `NC_UBYTE` |
+| `:char` | `NC_CHAR` |
+| `:short`, `:int16` | `NC_SHORT` |
+| `:ushort`, `:uint16` | `NC_USHORT` |
+| `:int`, `:int32` | `NC_INT` |
+| `:uint`, `:uint32` | `NC_UINT` |
+| `:int64` | `NC_INT64` |
+| `:uint64` | `NC_UINT64` |
+| `:float`, `:float32` | `NC_FLOAT` |
+| `:double`, `:float64` | `NC_DOUBLE` |
+
+**nc_type → CArray data_type** (returned by `Var#data_type`)
+
+| nc_type | data_type Symbol |
+|---|---|
+| `NC_BYTE` | `:int8` |
+| `NC_UBYTE`, `NC_CHAR` | `:uint8` |
+| `NC_SHORT` | `:int16` |
+| `NC_USHORT` | `:uint16` |
+| `NC_INT` | `:int32` |
+| `NC_UINT` | `:uint32` |
+| `NC_INT64` | `:int64` |
+| `NC_UINT64` | `:uint64` |
+| `NC_FLOAT` | `:float32` |
+| `NC_DOUBLE` | `:float64` |
+
+## Attribute name → typed writer
+
+L2's `def_var(..., **atts)` writes these attribute names via the typed `nc_put_att_<type>` so libnetcdf accepts them (CF §2.5.1 requires them to match the variable's on-disk type):
+
+| Attribute name | Written as |
+|---|---|
+| `_FillValue` | variable's on-disk type |
+| `missing_value` | " |
+| `valid_range` | " |
+| `valid_min` | " |
+| `valid_max` | " |
+
+Everything else goes through the generic `nc_put_att`, which infers the type from the Ruby value.

data/docs/09_cli.md

@@ -0,0 +1,79 @@
+# Command-line tools
+
+The gem installs two executables.  Both work on any NetCDF file, whoever wrote it, and both are thin scripts — `ncheader` over the netcdf gem's `nc_inq_*` calls, `ncvar` over layer 1, which reads into a CArray — so read them (`gem contents carray-netcdf`) if you want a worked example of walking a file.
+
+## ncheader — what is in this file
+
+Prints a file's header: dimensions, variables, attributes, groups, and user-defined types.  Header only; no data values.
+
+```
+ncheader FILE.nc                 # CDL, ncdump -h style (default)
+ncheader -f json FILE.nc         # the same tree as JSON
+ncheader -f yaml FILE.nc         # the same tree as YAML
+```
+
+`-f` is `--format`, and `cdl` may be named explicitly.
+
+```
+$ ncheader temp.nc
+netcdf temp {
+dimensions:
+  y = 3 ;
+  x = 4 ;
+variables:
+  double temperature(y, x) ;
+    temperature:units = "K" ;
+    temperature:valid_range = 200.0, 350.0 ;
+
+// global attributes:
+    :title = "example field" ;
+}
+```
+
+The JSON and YAML forms are there for scripting — a schema check in CI, a diff between two files' structure, a generator that reads the header and writes something else.
+
+## ncvar — get the values out
+
+Streams one variable's data to stdout.  The default output is **raw binary**: the variable's own type, the host's byte order, no header, no separators.  That is the point of the tool — the values land in whatever reads binary.
+
+```
+ncvar FILE.nc VAR                      # raw native-endian bytes
+ncvar -f text FILE.nc VAR              # one value per line
+ncvar -f json FILE.nc VAR              # {name, type, shape, start, data}
+```
+
+Every short flag has a long spelling: `--format`, `--start`, `--count`, `--stride`.
+
+A hyperslab, zero-based like libnetcdf, with `-s` start, `-c` count, `-S` stride:
+
+```
+ncvar -s 0,0 -c 10,20 FILE.nc temperature        # a 10x20 corner
+ncvar -s 0,0 -c 10,10 -S 1,2 FILE.nc temperature # every second column of it
+```
+
+Omitted, `-s` is all zeros and `-c` is the rest of each dimension.  `-c` counts the elements actually read, not the span they cover, so a stride of 2 goes with half the count — asking for 20 with `-S 1,2` reaches past the end and raises `NetCDF: Start+count exceeds dimension bound`.
+
+A variable inside a group is named by its path:
+
+```
+ncvar FILE.nc /observation/temp
+```
+
+Piping the raw form is what it is for:
+
+```
+$ ncvar temp.nc temperature | od -F -N 32          # -F reads doubles, -f floats
+$ ncvar temp.nc temperature | python3 -c "import numpy,sys; print(numpy.frombuffer(sys.stdin.buffer.read()))"
+```
+
+```ruby
+# Ruby on the other end of the pipe
+bytes = IO.popen(["ncvar", "temp.nc", "temperature"], &:read)
+bytes.unpack("d*")
+```
+
+Since the stream carries no shape or type, take those from `ncheader` (or `-f json`, which includes both) when the reader needs them.
+
+### Character and string variables
+
+`NC_CHAR` is bytes, and comes out as bytes: raw mode writes them unchanged, `-f text` writes them as one string, `-f json` puts that string in `data`.  `NC_STRING` is a sequence of separate strings, so `-f text` prints one per line, `-f json` keeps the nesting of the variable's shape, and raw mode joins them with a NUL — the one case where the raw stream needs a separator to be readable at all.