carray-netcdf 1.0.13 → 2.0.0

data/carray-netcdf.gemspec

@@ -1,23 +1,41 @@
+Gem::Specification.new do |spec|
+  spec.name        = "carray-netcdf"
+  spec.version     = File.read(File.expand_path("lib/carray/netcdf/version.rb", __dir__))[/VERSION\s*=\s*"([^"]+)"/, 1]
+  spec.summary     = "A NetCDF interface for CArray"
+  spec.description = <<~TEXT
+    A NetCDF interface for CArray, built on the netcdf gem.  Data is
+    exchanged as CArray at every level, through Ruby's MemoryView protocol,
+    so libnetcdf works on the array's own memory.
 
-Gem::Specification::new do |s|
-  version = "1.0.13"
+    Two layers do this.  Layer 1 turns the netcdf gem's MemoryView buffers
+    into CArray, keeping the nc_* functions as they are.  Layer 2 builds the
+    object-oriented interface on top, and returns data as the file's
+    conventions define it: _FillValue as a mask, scale_factor and add_offset
+    applied, CF time as CATime.  Pure Ruby, on CArray 3.0.
+  TEXT
+  spec.authors     = ["himotoyoshi"]
+  spec.email       = ["himotoyoshi@users.noreply.github.com"]
+  spec.license     = "MIT"
+  spec.homepage    = "https://github.com/himotoyoshi/carray-netcdf"
 
-  files = Dir.glob("**/*") - [ 
-                               Dir.glob("carray-*.gem"), 
-                             ].flatten
+  spec.required_ruby_version = ">= 3.1.0"
 
-  s.platform    = Gem::Platform::RUBY
-  s.name        = "carray-netcdf"
-  s.summary     = "Extension for manipulating NetCDF3 file with CArray"
-  s.description = <<-HERE
-    Extension for manipulating NetCDF3 file with CArray
-  HERE
-  s.version     = version
-  s.author      = "Hiroki Motoyoshi"
-  s.email       = ""
-  s.homepage    = 'https://github.com/himotoyoshi/carray-netcdf'
-  s.files       = files
-  s.extensions  = [ "ext/extconf.rb" ]
-  s.executables << 'ncdef2yaml'
-  s.required_ruby_version = ">= 1.8.1"
+  spec.files = Dir[
+    "lib/**/*.rb",
+    "bin/*",
+    "README.md",
+    "CONTRIBUTING.md",
+    "CHANGELOG.md",
+    "docs/*.md",
+    "LICENSE",
+    "carray-netcdf.gemspec",
+  ]
+  spec.require_paths = ["lib"]
+  spec.bindir        = "bin"
+  spec.executables   = %w[ncheader ncvar]
+
+  # A CArray is handed to libnetcdf as memory, so the ceiling is the next
+  # minor on both sides.
+  spec.add_dependency "carray", ">= 3.0.1"
+  spec.add_dependency "netcdf", ">= 0.1.0", "< 0.2"
 end

data/docs/00_introduction.md

@@ -0,0 +1,72 @@
+# Introduction
+
+carray-netcdf is a NetCDF interface for CArray, built on the [netcdf](https://github.com/himotoyoshi/netcdf) gem. This guide takes NetCDF and CArray as known, and covers what this gem adds between them.
+
+## What is carray-netcdf
+
+A NetCDF file holds arrays, and in Ruby the array is CArray. The purpose of this gem is to connect the two directly: open a file, get CArrays out of it, compute, and put CArrays back into a file that other NetCDF tools can read.
+
+Two things stand between a file and a CArray you can compute on. The first is the values themselves. What a file stores is often not what you want to work with — a physical field packed into `short` with a `scale_factor`, missing values marked by a `_FillValue` sentinel, a time axis kept as numbers since an epoch. Reading those correctly means following the conventions the file declares, and the gem does it for you, so what comes back is a masked CArray of physical values and a CATime axis.
+
+The second is size. Files are routinely larger than memory, so reading a variable has to mean reading a part of it. Here a variable is indexed the way a CArray is, and only the part you index is read.
+
+## Two layers, one module
+
+The netcdf gem is a literal translation of libnetcdf, and it passes data as MemoryView buffers. Two layers turn that into CArray, and both live in the module `CA::NetCDF`:
+
+```
+netcdf         (dependency)  thin libnetcdf binding, MemoryView buffers
+  ↑
+CA::NetCDF
+  ├─ layer 1  the nc_* functions, with CArray in and out
+  └─ layer 2  File / Group / Var / Dim, and the codecs
+```
+
+Layer 1 keeps the `nc_*` functions as they are and only changes what they take and return: a `get` hands back a CArray, a `put` accepts one. Use it as module functions, or mix it into a class of your own when you want the C idiom with CArray at both ends.
+
+Layer 2 is built on layer 1 and is the everyday surface: a file is an object, a variable is indexed like a CArray, and a codec decodes values on the way out.
+
+## Convention in this guide
+
+Every example assumes these two lines:
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+```
+
+Requiring the gem loads CArray, the netcdf gem, and both layers.
+
+## What the gem covers
+
+Whole NetCDF files, read and written: dimensions including unlimited ones, variables of every type the netcdf gem binds, attributes on a variable or on the file, and NetCDF-4 groups. Values pass through a codec in both directions, so `_FillValue`, `scale_factor`, `add_offset` and CF time are handled on the way out and on the way in. A file can be created atomically, or with its schema copied from an existing file. A Zarr store is read and written through the same calls, given a URL in place of a path.
+
+## Threads and Ractors
+
+Array I/O releases the GVL while libnetcdf works, so a large read or write does not hold up other Ruby threads.  Do not touch a CArray from another thread while it is being read into or written from.
+
+libnetcdf itself is not thread-safe, so the netcdf gem lets one call at a time into it, across the whole process.  Threads therefore do not make netCDF I/O faster, only the Ruby work around it.  That lock covers single calls: a `File`, `Group` or `Var` keeps state of its own between calls, so share one between threads only under a lock of your own.
+
+NetCDF works only from the main Ractor.  Opening a file in any other Ractor raises.  For netCDF I/O in parallel, use separate processes.
+
+## Beyond this gem
+
+Three things a CF file may carry are not turned into anything by this gem, and are left to the layer above it or to your own code.
+
+- **The CF data model.** Coordinate systems, grid mappings, cell methods, standard names and ancillary variables are kept as plain attributes — `var.att("grid_mapping")` still works — but no coordinate objects are built from them. That belongs to a heavier layer, or to your own code.
+- **Non-standard calendars** for CF time (`noleap`, `360_day`, `julian` and the rest). CATime is proleptic Gregorian, in UTC, so a file declaring another calendar warns once and its time coordinate comes back as the raw numbers.
+- **Time finer than CATime's base tick.** A variable in `"hours since 2022-01-01"` decodes to whole hours. Declare a finer unit, such as `"seconds since ..."`, if you need sub-hour precision.
+
+The [codec chapter](04_codec.md) states in full which CF attributes are honoured and which are passed through untouched.
+
+## How this guide is arranged
+
+- [Getting started](01_getting_started.md) — install, then read and write a first file
+- [Reading files](02_reading.md) — opening a file, finding a variable, reading all or part of it
+- [Writing files](03_writing.md) — declaring the schema, writing data, changing a file later
+- [Codecs and CF conventions](04_codec.md) — what gets decoded, how a codec is chosen, writing your own
+- [Groups](05_groups.md) — NetCDF-4 groups, and what they inherit
+- [Layer 1](06_layer1.md) — the `nc_*` functions with CArray in and out
+- [Cookbook](07_cookbook.md) — worked solutions to real tasks
+- [Cheatsheet](08_cheatsheet.md) — every method and constant of the gem on one page
+- [Command-line tools](09_cli.md) — `ncheader` and `ncvar`, which ship with the gem

data/docs/01_getting_started.md

@@ -0,0 +1,114 @@
+# Getting started
+
+## Install
+
+libnetcdf and its headers have to be present first, because the [netcdf](https://github.com/himotoyoshi/netcdf) gem underneath is built against them:
+
+```
+brew install netcdf          # macOS
+apt install libnetcdf-dev    # Debian / Ubuntu
+dnf install netcdf-devel     # Fedora / RHEL
+```
+
+Then:
+
+```
+gem install carray-netcdf
+```
+
+That brings in CArray and the netcdf gem.
+
+Requires:
+
+- Ruby >= 3.1
+- carray >= 3.0.1, < 3.1
+- libnetcdf >= 4.1
+
+## Loading
+
+```ruby
+require "carray-netcdf"
+
+NC = CA::NetCDF     # optional short alias used throughout this guide
+```
+
+Loading the gem pulls in carray, the netcdf gem, layer 1 (the `nc_*` functions), and layer 2 (`File`, `Group`, `Var`, `Dim`, `Codec`).
+
+## Reading in three lines
+
+```ruby
+NC::File.open("data.nc") do |nc|
+  data = nc["temperature"].to_ca   # => CArray (already decoded via CF codec if applicable)
+end
+```
+
+- `NC::File.open` opens for reading by default. The block form auto-closes.
+- `nc["temperature"]` returns a lazy `Var` handle — no I/O yet.
+- `.to_ca` reads the whole variable and returns a `CArray`. If the file declares `Conventions: CF-*` and the variable has `_FillValue` / `scale_factor` / `add_offset` / CF time units, all of that is decoded automatically.
+
+## Reading a slice
+
+```ruby
+NC::File.open("data.nc") do |nc|
+  t = nc["temperature"]
+  t.shape                    # => e.g. [365, 73, 144]
+  t.dims.map(&:name)         # => ["time", "lat", "lon"]
+
+  first_day = t[0, nil, nil] # 2-D CArray, shape [73, 144]
+  point     = t[0, 0, 0]     # scalar (Float, or CATime::Element for a time coord)
+  block     = t[0..9, nil, nil]  # first 10 days, 3-D
+end
+```
+
+The `[]` accepts anything `CArray.scan_index` accepts: integers, ranges, `nil` for a whole axis, arrays, boolean masks. Point-and-block indexing maps directly to `nc_get_var1` / `nc_get_vara` / `nc_get_vars` under the hood, so you only ever read what you ask for.
+
+## Writing a file in ten lines
+
+```ruby
+NC::File.create("out.nc") do |nc|
+  nc.def_dim("time", :unlimited)
+  nc.def_dim("lat", 73)
+  nc.def_dim("lon", 144)
+
+  nc.def_var("temp", :float, ["time", "lat", "lon"],
+             units: "K", _FillValue: -999.9)
+
+  nc.def_att("Conventions", "CF-1.7")
+  nc.def_att("title",       "Demo file")
+
+  nc["temp"] = CArray.float32(1, 73, 144) { rand * 30 + 260 }
+end
+```
+
+- `File.create` opens the file in **define mode** and drops you into it.
+- `def_dim` / `def_var` / `def_att` build the schema. Pass `:unlimited` (or the string `"unlimited"` in YAML-ish contexts) for an unlimited dimension. Types are Symbols: `:byte :short :int :int64 :float :double`, plus the CArray-style aliases (`:int32`, `:float32`, `:float64` …).
+- Variable attributes go as keyword arguments to `def_var` (any Ruby-side Numeric / String / Array is fine). Attributes that CF requires to match the variable's type — `_FillValue`, `missing_value`, `valid_range`, `valid_min`, `valid_max` — are automatically written in the variable's type so libnetcdf doesn't reject them.
+- `nc["temp"] = ca` is a whole-variable write. `nc["temp", 0, nil, nil] = ca` is a hyperslab write. `File` switches between define and data mode for you.
+
+## The three access idioms
+
+Everything below reads the same underlying data:
+
+```ruby
+NC::File.open("data.nc") do |nc|
+  # (1) full read, then work with a CArray
+  ca = nc["temp"].to_ca
+  ca.mean(0)
+
+  # (2) hyperslab read via CArray indexing on the Var
+  slab = nc["temp"][0..9, nil, nil]
+
+  # (3) treat the Var as if it were a CArray
+  #     method_missing delegates to .to_ca — convenient for interactive work,
+  #     but each call re-reads the whole variable, so cache if you loop.
+  nc["temp"].mean
+end
+```
+
+## Where to go next
+
+- [Reading files](02_reading.md) — the full `File` / `Var` / `Dim` surface, plus what codecs turn raw bytes into (masks, physical values, CATime).
+- [Writing files](03_writing.md) — schema definition, hyperslab writes, packing, unlimited dimensions.
+- [Codecs and CF conventions](04_codec.md) — what CF decoding actually does, how to opt in to `valid_range` masking, how to write custom codecs.
+- [Groups](05_groups.md) — NetCDF-4 groups.
+- [Cookbook](07_cookbook.md) — end-to-end scenarios pulled from real use cases.

data/docs/02_reading.md

@@ -0,0 +1,216 @@
+# Reading files
+
+A read goes through three steps, and only the last one touches the disk:
+
+```
+nc = NC::File.open(path)   # a file handle: its dimensions, variables and attributes
+var = nc["temp"]           # a variable handle — still no data read
+ca  = var[0, nil, nil]     # this is the read: libnetcdf fills a CArray, the codec decodes it
+```
+
+A `Var` is a lazy handle. Holding one costs nothing, and the index you give it decides how much of the variable is read — a point, a slab, a strided subset, or the whole thing with `to_ca`. What comes back is a CArray, with the file's conventions already applied to it, so a packed variable arrives as physical values and its missing entries arrive as a mask.
+
+Everything else on this page is detail around those three steps.
+
+## Opening
+
+```ruby
+require "carray-netcdf"
+NC = CA::NetCDF
+
+# Block form (recommended — auto-closes on exit / exception)
+NC::File.open("data.nc") do |nc|
+  # ...
+end
+
+# Explicit form
+nc = NC::File.open("data.nc")
+begin
+  # ...
+ensure
+  nc.close
+end
+```
+
+`open` takes an optional mode:
+
+```ruby
+NC::File.open("data.nc")           # :read   (default)
+NC::File.open("data.nc", :write)   # open for modification (existing file)
+NC::File.open("data.nc", :append)  # alias of :write
+```
+
+For creating a new file, see [Writing files](03_writing.md).
+
+A Zarr store opens the same way, through a URL in place of the path: `NC::File.open("file:///data/t.zarr#mode=zarr,file")`.  See [Zarr stores](03_writing.md#zarr-stores).
+
+## Introspection
+
+```ruby
+NC::File.open("data.nc") do |nc|
+  nc.filename    # => "data.nc"     (OS path)
+  nc.format      # => "NC_FORMAT_CLASSIC" / "NC_FORMAT_NETCDF4" / ...
+  nc.path        # => "/"           (netcdf-4 group path — always "/" for the root)
+
+  nc.dims        # => {"time" => Dim, "lat" => Dim, "lon" => Dim}
+  nc.vars        # => {"temp" => Var, "time" => Var, "lat" => Var, "lon" => Var}
+  nc.atts        # => {"Conventions" => "CF-1.7", "title" => "…", ...} (frozen)
+
+  nc.dim("time")     # => Dim
+  nc.var("temp")     # => Var
+  nc.att("title")    # => "..." (or nil)
+
+  nc.dim?("time")    # => true
+  nc.var?("bogus")   # => false
+end
+```
+
+Names are Strings — always. `nc["temp"]` and `nc.var("temp")` work; Symbols are not coerced.
+
+## The Var handle
+
+`nc["temp"]` returns a `Var`. It is a **lazy handle**: no I/O happens until you index into it or call `to_ca`.
+
+```ruby
+t = nc["temp"]
+
+t.name          # => "temp"
+t.shape         # => [365, 73, 144]
+t.ndim          # => 3
+t.data_type     # => :float32 (Symbol, matching CArray convention)
+t.nc_type       # => NetCDF::NC_FLOAT (the raw nc_type integer)
+t.dims          # => [Dim("time"), Dim("lat"), Dim("lon")]
+t.atts          # => {"units" => "K", "_FillValue" => -999.9, ...} (frozen)
+t.att("units")  # => "K"
+
+t.file          # => the enclosing File
+t.group         # => the enclosing Group (== File for a root-level variable)
+t.codec         # => the codec that will decode reads from this variable
+```
+
+## Whole-variable read
+
+```ruby
+ca = nc["temp"].to_ca      # equivalent to nc["temp"][]
+```
+
+Returns a CArray of shape `t.shape` and type `t.data_type` (or the codec's choice, e.g. Float64 after scale/offset, CATime after CF time decode). The codec has already run — `_FillValue` cells are `UNDEF`, physical values are unpacked, time coordinates are CATime.
+
+## Hyperslab reads (`nc[var][i, j, k]`)
+
+`Var#[]` accepts anything `CArray.scan_index` accepts:
+
+```ruby
+t = nc["temp"]
+
+t[]                        # whole variable  (same as t.to_ca)
+t[0, nil, nil]             # first time step (rank-2 CArray, shape [73, 144])
+t[0]                       # first cell if 1-D, or a slab if the axis is size 1;
+                           #   for higher-rank vars, use explicit nil for the rest
+t[0, 0, 0]                 # single element (scalar — Float / Integer / Element)
+t[0..9, nil, nil]          # first 10 time steps
+t[(0..-1).step(2), nil, nil]   # every other time step (netcdf stride under the hood)
+t[[0, 5, 10], nil, nil]    # not hyperslab-able (uses `nc_get_var` + CArray indexing)
+```
+
+- Point indices (`0, 0, 0`) go through `nc_get_var1` and return a **decoded scalar** — a `Float` after scale/offset, `UNDEF` for a fill cell, a `CATime::Element` for a CF time coordinate, or a plain Numeric if the variable has no codec transforms.
+- Block indices with all-unit strides go through `nc_get_vara`; strided block indices go through `nc_get_vars`. `CArray.scan_index` treats a form like `[start, count, stride]` (an Array of three ints) as a strided block, so `t[[0, 5, 2], nil, nil]` is a *hyperslab*, not fancy indexing.
+- Genuine fancy indexing — a boolean mask, a scatter/gather via an integer index array — falls back to reading the whole variable and indexing the resulting CArray. That's a correctness fallback, not a performance path; for large files, prefer hyperslab-shaped indices.
+
+Scalar decoding examples:
+
+```ruby
+nc["temp"][0, 0, 0]                # => 273.15  (Float, after scale/offset)
+nc["temp"][0, 0, 0]                # => UNDEF   (if that cell held _FillValue)
+nc["time"][0]                      # => #<CATime::Element 2022-01-01T00:00:00Z>
+nc["time"][0].to_time              # => 2022-01-01 00:00:00 UTC (Ruby Time)
+nc["time"][0].to_date              # => #<Date: 2022-01-01>
+```
+
+## `method_missing`: use a Var like a CArray
+
+`Var` forwards unknown methods to `to_ca`, so anything you can call on a CArray, you can call on a Var directly:
+
+```ruby
+nc["temp"].mean            # => the mean of the whole array
+nc["temp"].max             # => the max
+nc["temp"].shape           # (Var#shape is defined, so this is not method_missing)
+nc["temp"].sum(0)          # => reduction along axis 0
+```
+
+A method that changes its receiver — `fill`, `seq!`, `mask=`, anything ending in `!` or `=` — raises `NoMethodError` instead: it would change the fresh array `to_ca` returns, not the file. Write with `v[...] = ...`.
+
+Each such call re-reads the whole variable. For repeated work, cache once:
+
+```ruby
+temp = nc["temp"].to_ca
+temp.mean
+temp.max
+temp.sum(0)
+```
+
+## Reading a subset into an existing buffer
+
+For streaming big files a slab at a time, allocate one buffer once and reuse it.  Layer 1's `nc_get_vara` accepts an explicit CArray as the last argument and reads into it — no allocation per slab:
+
+```ruby
+NC::File.open("big.nc") do |nc|
+  varid = ::NetCDF.nc_inq_varid(nc.ncid, "temp")
+  buf   = CArray.float32(73, 144)          # one time slice, reused
+
+  365.times do |k|
+    CA::NetCDF.nc_get_vara(nc.ncid, varid, [k, 0, 0], [1, 73, 144], buf)
+    # buf now holds day k; no allocation, no codec, raw physical bytes.
+    process(buf)
+  end
+end
+```
+
+This bypasses layer 2 (and the codec).  Use it when you're staying at raw values on purpose and want zero-allocation streaming.  See [layer1.md](06_layer1.md) for the full API.
+
+## Dimensions
+
+```ruby
+d = nc.dim("time")
+
+d.name           # => "time"
+d.length         # => 365
+d.unlimited?     # => true / false
+d.to_i           # => raw dimid (rarely needed)
+
+# Bridge to the coordinate variable (a variable of the same name)
+d.var            # => Var, or nil if no matching variable exists
+d.to_ca          # => the coord values (nil if no coord variable)
+```
+
+## Attributes
+
+Attributes are surfaced as native Ruby values by the underlying netcdf gem:
+
+- `NC_CHAR` / `NC_STRING` → `String`
+- Numeric scalar (single-element) → `Integer` or `Float`
+- Numeric multi-element → `CArray` (or Ruby `Array` for very short cases — the netcdf gem picks the natural shape)
+
+Examples:
+
+```ruby
+nc.att("Conventions")            # => "CF-1.7"
+nc.att("title")                  # => "Demo file"
+nc["temp"].att("units")          # => "K"
+nc["temp"].att("_FillValue")     # => -999.9  (Float; typed as the variable)
+nc["temp"].att("valid_range")    # => CArray([0.0, 400.0])
+```
+
+An unknown attribute returns `nil`. `nc.atts` returns the frozen hash of everything (global on `nc`, per-variable on `nc["temp"]`).
+
+## Turning decoding off
+
+If you want raw bytes — no `_FillValue` masking, no unpacking, no CATime — pass `codec: :identity`:
+
+```ruby
+NC::File.open("data.nc", codec: :identity) do |nc|
+  raw = nc["temp"].to_ca    # packed short values, fill sentinel as-is, ...
+end
+```
+
+Codec selection is covered in detail in the [Codecs chapter](04_codec.md).