carray-netcdf 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 71bb04b44cd8c06925d970fb1e69ebdea46b7320aeec1e3c542aca3668a66cfc
-  data.tar.gz: 864e70e40ec87e90200e74ca97590e7449808dd8e5d012ce64aac8805182733f
+  metadata.gz: 6b62bd15777ba74f1034591a396904bdce84015051c9895c9f01e1793ea39685
+  data.tar.gz: fc6d94c6f079e92081035f999d3a12d111622202e858b64c7f06fb61e06e0d9b
 SHA512:
-  metadata.gz: 99b7b5824c461197bbdb4b7161d59728d095976c207964d34208d560a7a14ef4c9a432ef8a76ee974ad97095fe4b89e611dd5b9e262cf25b20f734c30eb0d737
-  data.tar.gz: d9ef82a43aaacdb8d9e0b83786d3f0d222f4738feb3fc2b3b785cc01cbcf7117ae83c5f51372831c3b80e4bfb503ab84e231917568adc9c2551133d09fab65f9
+  metadata.gz: fa102781a9705ea42e3a9b96ff95e7d34c4f2dc2204512d468f16866085bde6f35f2115f41d9e90a4ca4d25d424708380c7c8efeed092e7eb2f43d92c68d1210
+  data.tar.gz: f5841947884c2e91e72c334cc3884576778f36b71d8bf0d1aa5a537591893b5f26982a06fe5529c5e9eac7206af427131853bf973ba9a5b276cc33f851fa5fb0

data/CHANGELOG.md

@@ -30,6 +30,77 @@ different codebase and is not covered.
      netcdf gem's. Which of those a release needs is said in the gemspec,
      and an entry says so only when the answer changes. -->
 
+## 2.1.0
+
+- New: `def_var` and `File.create` take `zstandard:`, `bzip2:`, `szip:`
+  and `blosc:` beside `deflate:`. zstandard, bzip2 and blosc need
+  libnetcdf 4.9; a filter libnetcdf lacks raises `NetCDF::Error` rather
+  than writing the data uncompressed.
+
+- New: `Var#storage_options` gives a variable's chunking, compression and
+  other storage settings in `def_var`'s keyword form, and
+  `File#netcdf4?` says whether the file can have them.
+
+- New: `File.create(format:)` makes `:classic`, `:offset64` (CDF-2),
+  `:cdf5`, `:netcdf4` or `:netcdf4_classic` files. A template's format
+  is kept, and `netcdf4: false` wins over a NetCDF-4 template.
+
+- New: `File#reload` / `Group#reload` read the file's dimensions,
+  variables, groups and attributes again, after changes made through the
+  low-level API.
+
+- New: `Var#att_type` gives the `NC_*` type an attribute is stored in.
+
+- New: a file with CF attributes but no `Conventions` (as xarray and
+  netCDF4-python write) warns once that it is read as stored, and
+  suggests `codec: :cf`.
+
+- Change: these are private now: `CA::NetCDF.nc_type_for`,
+  `.put_att_typed` and `.put_var_att`; the low-level get / put calls on
+  `File`, `Group`, `Var` and `Dim`; `Group#copy_att_from` and
+  `File.atomic_target`. Pass a type name to `def_var`, call
+  `NetCDF.nc_put_att_*`, or call the get / put functions on `CA::NetCDF`.
+
+- Change: attribute values from `atts` / `att`, and what `dims` / `vars`
+  / `groups` return, are frozen. Code that changes them in place should
+  copy them first.
+
+- Change: `File.create(..., clobber: false)` on an existing file raises
+  `Errno::EEXIST`, also without `atomic:`; it raised `NetCDF::Error`.
+  Code that rescued that should rescue `Errno::EEXIST`.
+
+- Change: needs Ruby 3.2 or later, as CArray 3.0 does, and the netcdf gem
+  0.2.0 or later (below 0.3).
+
+- Change: `File#abort`, and the block form when its block raises, undo
+  the definitions made since the file last left define mode; they were
+  kept. An aborted non-atomic `File.create` removes the file.
+
+- Change: `def_var(template:)`, `copy_var` and `File.create(vars:)` copy
+  the template's storage settings when both files are NetCDF-4.
+  Compression options you pass replace the template's as a whole. Chunk
+  sizes are copied only when the dimensions have the same lengths.
+
+- Change: float32 `scale_factor` / `add_offset` unpack in float32, as CF
+  asks and xarray does; they unpacked in float64.
+
+- Change: a CArray given as `_FillValue`, `missing_value` or `valid_*` is
+  written in the variable's type, as a Ruby value is. A masked one
+  raises `ArgumentError`.
+
+- Change: reading by an index array on an axis, and asking a variable's
+  size, `first` or `last`, no longer read the whole variable.
+
+- Fix: a float written to an integer variable is rounded, and NaN
+  written to an integer or packed variable becomes the fill value (or
+  raises when there is none). 2.0.0 truncated the float and stored NaN as
+  `0`, so check integer data that 2.0.0 wrote from floats.
+
+- Fix: under the standard calendar, a CF time reference before
+  1582-10-15 is a Julian date, as cftime reads it, and a one- to
+  three-digit reference year is that year. 2.0.0 read such times wrong
+  (`"days since 1-1-1"` as 2001).
+
 ## 2.0.0
 
 A ground-up rewrite. Nothing from 1.x is carried over, so read this as a
@@ -49,12 +120,12 @@ new library that happens to keep the name rather than as an upgrade.
   binding — is included rather than modified, so a future `Numo::NetCDF`
   can sit beside it. Code written against 1.x's names does not run here.
 
-- New: layer 1, `CA::NetCDF`'s own `nc_*` functions. They are the
+- New: the low-level API, `CA::NetCDF`'s own `nc_*` functions. They are the
   netcdf gem's API with the array I/O overridden to take and return
   CArrays; definition, inquiry, attribute, and group calls pass through
   verbatim. Usable as module functions or by `include CA::NetCDF`.
 
-- New: layer 2, the object-oriented surface — `File`, `Group`, `Var`,
+- New: the high-level API, the object-oriented surface — `File`, `Group`, `Var`,
   `Dim`. `File.open` / `File.create`, `nc[:temp]` to read a variable,
   `nc[:temp, 0..9] = data` to write a hyperslab, lazy hyperslab reads,
   file-wide variable option defaults, and atomic writes through a

data/CONTRIBUTING.md

@@ -46,7 +46,7 @@ And, where they apply, the things most bugs here turn on:
 
 ## Sending a fix
 
-Ruby 3.1 or later, CArray and the netcdf gem in the ranges the gemspec declares. Sibling checkouts of `carray` and `netcdf` beside this one are picked up automatically when they are there; otherwise the installed gems are used.
+Ruby 3.2 or later, CArray and the netcdf gem in the ranges the gemspec declares. Sibling checkouts of `carray` and `netcdf` beside this one are picked up automatically when they are there; otherwise the installed gems are used.
 
 ```sh
 rake spec               # the whole suite (also the default task)
@@ -66,9 +66,9 @@ Keep the commit message to a subject line and a line or two of context. Anything
 
 ## Working on the code
 
-`docs/` is the guide and is written to be read in order. The code is one file per class under `lib/carray/netcdf/`, with layer 1 — the `nc_*` functions that speak CArray — in `lib/carray/netcdf.rb` itself.
+`docs/` is the guide and is written to be read in order. The code is one file per class under `lib/carray/netcdf/`, with the low-level API — the `nc_*` functions that speak CArray — in `lib/carray/netcdf.rb` itself.
 
-The line worth holding while working here: layer 1 is raw values only. No convention decoding, no mask handling; reads come back unmasked and writing a masked CArray raises. Everything about `_FillValue`, `scale_factor`, and time belongs to the codecs in layer 2.
+The line worth holding while working here: the low-level API is raw values only. No convention decoding, no mask handling; reads come back unmasked and writing a masked CArray raises. Everything about `_FillValue`, `scale_factor`, and time belongs to the codecs in the high-level API.
 
 ## License
 

data/README.md

@@ -1,25 +1,26 @@
 # carray-netcdf
 
-A NetCDF interface for [CArray](https://github.com/himotoyoshi/carray), built on the [netcdf](https://github.com/himotoyoshi/netcdf) gem.
+Read and write [NetCDF](https://www.unidata.ucar.edu/software/netcdf/) files from Ruby, with variable data read into and written from [CArray](https://github.com/himotoyoshi/carray)s.
 
-Data is exchanged as CArray at every level: reads return a CArray and writes accept one. Both go through Ruby's MemoryView protocol, so libnetcdf reads and writes the array's own memory, with no intermediate representation of the data.
+CArray is a Ruby library of multi-dimensional arrays, each holding a single numeric type, with slicing, element-wise arithmetic, and statistics over the whole array or along any axis. Every element can be masked as missing, and the statistics skip masked elements.
 
-Two layers do this, both in the `CA::NetCDF` module. Layer 1 turns the netcdf gem's MemoryView buffers into CArray, keeping the `nc_*` functions as they are. Layer 2 builds the object-oriented interface on top: `File`, `Group`, `Var`, `Dim` and the codecs. Both are pure Ruby, on CArray 3.0; every libnetcdf call goes through the netcdf gem.
+The gem is built on the [netcdf](https://github.com/himotoyoshi/netcdf) gem, a near one-to-one binding to libnetcdf, and offers two APIs on top of it:
 
-Layer 2 gives you data as the file's conventions define it. For a file declaring CF or COARDS, values are decoded on the way out: `_FillValue` becomes a mask, `scale_factor` and `add_offset` are applied, and time coordinates become CATime. Variables are indexed like CArrays, and only the elements you index are read — `nc["temp"][0..9, nil, nil]` reads ten days, not the whole variable.
+* **Low-level API** — the netcdf gem's `nc_*` functions, with the ones that read and write variable data taking and returning CArray.
+* **High-level API** — an object-oriented API (`File`, `Group`, `Var`, `Dim`) that applies the file's conventions, and is what everyday code uses.
 
 ## Status
 
-2.0 still moves: its own surface is not settled yet, so behaviour can change between releases.
+The 2.0 API is not settled yet, and may change between releases.
 
 ## Features
 
-* **Object-oriented access to NetCDF data.** A file, a group, a variable and a dimension are objects: `File.open`, `nc["temp"]`, `var.dims`, `var.atts`, and a block form that closes the file for you.
-* **CArray in and out.** Reads return a CArray and writes accept one. A variable is indexed the way a CArray is, so only the slab you ask for is read, and libnetcdf works on the array's own memory rather than a copy of it.
-* **Conventions are decoded — CF, COARDS, or none.** Under CF or COARDS, `_FillValue` becomes a mask, `scale_factor` and `add_offset` are applied, and a time axis becomes CATime. `codec: :identity` gives you the stored values instead, and a codec of your own can take their place.
-* **NetCDF-4 groups.** Variables are addressed by path (`nc["/obs/temp"]`), dimensions are inherited from the groups above, and the file's codec applies inside them.
-* **Writing is a schema and an assignment.** `def_dim`, `def_var`, `def_att`, then `nc["temp"] = ca`. Attributes that CF requires to match the variable's type are written in that type, so other tools read the file as intended.
-* **Atomic writes.** `File.create(atomic: true)` writes to a temporary file in the same directory and renames it into place only when the block completes, so a failed run leaves no half-written file behind.
+* **Object-oriented API.** Files, groups, variables and dimensions are objects.
+* **Partial data access.** Only the part of a variable you index is read or written, so a variable larger than memory can be handled a piece at a time.
+* **CF and COARDS conventions.** Files whose `Conventions` attribute names CF or COARDS are read as the conventions define them: missing values as masks, packed values unpacked, and time axes as dates and times. Writing reverses each of these.
+* **Definitions and data in any order.** Dimensions, variables and attributes can be defined and data written in whatever order suits you; NetCDF's define and data modes are switched as needed.
+* **Existing files as templates.** A new file can take its dimensions, global attributes and coordinate variables from an existing file, and any variable can take its definition from a variable in another file.
+* **Atomic writes.** On request, a file appears under its name only once it has been written completely, so a failed run leaves no half-written file behind.
 
 ## Install
 
@@ -45,12 +46,38 @@ gem "carray-netcdf"
 
 Requires:
 
-- Ruby >= 3.1
-- carray >= 3.0.1
+- Ruby >= 3.2
+- libnetcdf >= 4.1, with its headers (`nc-config` or `pkg-config` on PATH)
+- A C compiler, to build the netcdf gem
+- carray >= 3.0.1, installed as a dependency
 - the netcdf gem, installed as a dependency
 
 ## Example
 
+### Low-level API
+
+The `nc_*` functions that read and write variable data are on `CA::NetCDF` and take and return CArray; the others, such as opening and looking up, are called on the netcdf gem's `NetCDF` as they are. Files and variables are integer ids, and the values come back as stored, so fill values, packing and time units are yours to handle:
+
+```ruby
+require "carray-netcdf"
+
+NC = CA::NetCDF
+
+ncid = NetCDF.nc_open("data.nc", NetCDF::NC_NOWRITE)       # the netcdf gem, as it is
+vid  = NetCDF.nc_inq_varid(ncid, "temp")
+
+day = NC.nc_get_vara(ncid, vid, [0, 0, 0], [1, 73, 144])   # a new CArray, as stored
+
+buf = CArray.float32(73, 144)
+NC.nc_get_vara(ncid, vid, [0, 0, 0], [1, 73, 144], buf)    # or read into one you have
+
+NetCDF.nc_close(ncid)
+```
+
+### High-level API
+
+Files and variables by name, and a variable indexed like a CArray. The CF or COARDS conventions a file declares in its `Conventions` attribute are applied automatically — here, the `_FillValue` element comes back masked:
+
 ```ruby
 require "carray-netcdf"
 
@@ -65,9 +92,11 @@ NC::File.create("out.nc") do |nc|
   nc.def_var("temp", :float, ["time", "lat", "lon"],
              units: "K", _FillValue: -999.9)
 
-  nc.def_att("Conventions", "CF-1.7")
+  nc.def_att("Conventions", "CF-1.7")                     # turns CF encoding and decoding on
 
-  nc["temp"] = CArray.float32(1, 73, 144) { rand * 30 + 260 }
+  temp = CArray.float32(1, 73, 144).random(30) + 260      # random values, 260-290
+  temp[0, 0, 0] = UNDEF                                   # mark one element missing
+  nc["temp"] = temp                                       # written as -999.9
 end
 
 # --- read ---
@@ -76,23 +105,26 @@ NC::File.open("out.nc") do |nc|
   t.shape                     #  => [1, 73, 144]
   t.dims.map(&:name)          #  => ["time", "lat", "lon"]
 
-  day = t[0, nil, nil]        #  a 2-D CArray; only this slab is read
-  day.mean                    #  CF decoding has already been applied
+  day = t[0, nil, nil]        #  a 73 x 144 CArray; only this part is read
+  day.count_masked            #  => 1, the -999.9 came back masked
+  day.mean                    #  the mean of the other 10511 elements
 end
 ```
 
 ## Documentation
 
-* [Introduction](docs/00_introduction.md) — why there are two layers, and what the gem leaves to CArray
+* [Introduction](docs/00_introduction.md) — what the gem does, what it leaves to CArray, and how it sits on the netcdf gem
 * [Getting started](docs/01_getting_started.md) — install, then read and write a first file
 * [Reading files](docs/02_reading.md) — opening a file, finding a variable, reading all or part of it
 * [Writing files](docs/03_writing.md) — declaring the schema, writing data, changing a file later
 * [Codecs and CF conventions](docs/04_codec.md) — what gets decoded, how a codec is chosen, writing your own
 * [Groups](docs/05_groups.md) — NetCDF-4 groups, and what they inherit
-* [Layer 1](docs/06_layer1.md) — the `nc_*` functions with CArray in and out
+* [Low-level API](docs/06_low_level_api.md) — the `nc_*` functions with CArray in and out
 * [Cookbook](docs/07_cookbook.md) — worked solutions to real tasks
 * [Cheatsheet](docs/08_cheatsheet.md) — every method and constant of the gem on one page
-* [Command-line tools](docs/09_cli.md) — `ncheader` and `ncvar`, which ship with the gem
+* [Command-line tools](docs/09_cli.md) — `ncheader` and `ncvar`
+
+What changed between releases is in [CHANGELOG.md](CHANGELOG.md).
 
 ## Contributing
 
@@ -106,4 +138,4 @@ carray-netcdf was designed and reviewed by a human developer; the implementation
 
 ## License
 
-MIT
+[MIT](LICENSE)

data/bin/ncheader

@@ -27,6 +27,15 @@ PRIMITIVE_TYPE_NAMES = {
   NetCDF::NC_STRING => "string",
 }
 
+# A String as valid UTF-8, for JSON / YAML and the terminal: a byte that
+# is not part of a UTF-8 character is written as \xNN (old files often
+# hold Latin-1 text).
+def printable_text(s)
+  # String.new, not dup.force_encoding: before carray 3.0.7, a string from
+  # CArray#dump_binary kept a stale "valid" flag through force_encoding.
+  String.new(s, encoding: Encoding::UTF_8).scrub { |b| b.unpack("C*").map { |c| format("\\x%02X", c) }.join }
+end
+
 def type_name(ncid, xtype)
   PRIMITIVE_TYPE_NAMES[xtype] || NetCDF.nc_inq_type(ncid, xtype).first
 end
@@ -35,10 +44,20 @@ def collect_attrs(ncid, varid, natts)
   natts.times.map do |i|
     name  = NetCDF.nc_inq_attname(ncid, varid, i)
     xtype, _len = NetCDF.nc_inq_att(ncid, varid, name)
+    value = begin
+              NetCDF.nc_get_att(ncid, varid, name)
+            rescue NotImplementedError
+              "(not readable: #{type_name(ncid, xtype)} attribute)"
+            end
+    value = case value
+            when String then printable_text(value).sub(/\u0000+\z/, "")   # as ncdump drops it
+            when Array  then value.map { |v| v.is_a?(String) ? printable_text(v) : v }
+            else             value
+            end
     {
-      "name"  => name,
+      "name"  => printable_text(name),
       "type"  => type_name(ncid, xtype),
-      "value" => NetCDF.nc_get_att(ncid, varid, name),
+      "value" => value,
     }
   end
 end
@@ -73,11 +92,13 @@ def collect_user_types(ncid)
 end
 
 def inspect_group(ncid)
-  _, _, natts, unlimid = NetCDF.nc_inq(ncid)
+  _, _, natts, _ = NetCDF.nc_inq(ncid)
+  # Every unlimited dimension of the group (NetCDF-4 can have several).
+  unlimited = NetCDF.nc_inq_unlimdims(ncid)
 
   dims = NetCDF.nc_inq_dimids(ncid, false).map { |id|
     name, len = NetCDF.nc_inq_dim(ncid, id)
-    { "name" => name, "length" => len, "unlimited" => (id == unlimid) }
+    { "name" => name, "length" => len, "unlimited" => unlimited.include?(id) }
   }
 
   vars = NetCDF.nc_inq_varids(ncid).map { |vid|
@@ -113,6 +134,18 @@ TYPE_SUFFIX = {
   "float" => "f", "double" => "",
 }
 
+# A float32 value with the fewest digits that read back as the same
+# float32, as ncdump prints it (`0.1`, not 0.10000000149011612).
+def float32_text(x)
+  return x.to_s unless x.finite?
+  bits = [x].pack("f")
+  (1..9).each do |digits|
+    s = format("%.#{digits}g", x)
+    return s if [Float(s)].pack("f") == bits
+  end
+  x.to_s
+end
+
 def cdl_escape(s)
   s.to_s.gsub(/[\\"\n\r\t]/, "\\" => '\\\\', '"' => '\\"',
                               "\n" => '\\n', "\r" => '\\r', "\t" => '\\t')
@@ -127,7 +160,7 @@ def cdl_value(type, value)
   suffix = TYPE_SUFFIX[type] || ""
   arr.map { |v|
     if v.is_a?(Float)
-      type == "float" ? "#{v}f" : v.to_s
+      type == "float" ? "#{float32_text(v)}f" : v.to_s
     else
       "#{v}#{suffix}"
     end
@@ -173,7 +206,8 @@ def emit_cdl(io, group, name, indent = 0)
       dims = v["dimensions"].empty? ? "" : "(#{v["dimensions"].join(", ")})"
       io.puts "#{pre}  #{v["type"]} #{v["name"]}#{dims} ;"
       v["attributes"].each { |a|
-        io.puts "#{pre}    #{v["name"]}:#{a["name"]} = #{cdl_value(a["type"], a["value"])} ;"
+        prefix = a["type"] == "string" ? "string " : ""
+        io.puts "#{pre}    #{prefix}#{v["name"]}:#{a["name"]} = #{cdl_value(a["type"], a["value"])} ;"
       }
     }
   end
@@ -183,7 +217,8 @@ def emit_cdl(io, group, name, indent = 0)
     label = indent.zero? ? "global attributes" : "group attributes"
     io.puts "#{pre}// #{label}:"
     group["attributes"].each { |a|
-      io.puts "#{pre}    :#{a["name"]} = #{cdl_value(a["type"], a["value"])} ;"
+      prefix = a["type"] == "string" ? "string " : ""
+      io.puts "#{pre}    #{prefix}:#{a["name"]} = #{cdl_value(a["type"], a["value"])} ;"
     }
   end
 
@@ -218,6 +253,6 @@ end
 
 case format
 when "cdl"  then emit_cdl($stdout, tree, tree["name"])
-when "json" then puts JSON.pretty_generate(tree)
+when "json" then puts JSON.pretty_generate(tree, allow_nan: true)   # NaN / Infinity as bare words
 when "yaml" then puts YAML.dump(tree)
 end

data/bin/ncvar

@@ -35,6 +35,32 @@ TYPE_NAMES = {
   NetCDF::NC_STRING => "string",
 }
 
+# A String as valid UTF-8, for JSON / YAML and the terminal: a byte that
+# is not part of a UTF-8 character is written as \xNN (old files often
+# hold Latin-1 text).
+def printable_text(s)
+  # String.new, not dup.force_encoding: before carray 3.0.7, a string from
+  # CArray#dump_binary kept a stale "valid" flag through force_encoding.
+  String.new(s, encoding: Encoding::UTF_8).scrub { |b| b.unpack("C*").map { |c| format("\\x%02X", c) }.join }
+end
+
+# printable_text over a nested Array of Strings.
+def deep_text(a)
+  a.map { |x| x.is_a?(Array) ? deep_text(x) : printable_text(x) }
+end
+
+# A float32 value with the fewest digits that read back as the same
+# float32, as ncdump prints it (`0.1`, not 0.10000000149011612).
+def float32_text(x)
+  return x.to_s unless x.finite?
+  bits = [x].pack("f")
+  (1..9).each do |digits|
+    s = format("%.#{digits}g", x)
+    return s if [Float(s)].pack("f") == bits
+  end
+  x.to_s
+end
+
 def parse_int_list(s)
   return nil if s.nil?
   s.split(",").map { |x| Integer(x) }
@@ -77,13 +103,20 @@ begin
   shape_full = dimids.map { |d| NetCDF.nc_inq_dimlen(gid, d) }
 
   start  = parse_int_list(start_arg)  || [0] * shape_full.size
-  count  = parse_int_list(count_arg)  || shape_full.zip(start).map { |full, s| full - s }
   stride = parse_int_list(stride_arg)
-
-  abort "start/count/stride rank mismatch (var has #{shape_full.size} dims)" \
-    unless start.size == shape_full.size &&
-           count.size == shape_full.size &&
-           (stride.nil? || stride.size == shape_full.size)
+  abort "start/stride rank mismatch (var has #{shape_full.size} dims)" \
+    unless start.size == shape_full.size && (stride.nil? || stride.size == shape_full.size)
+  start.each_with_index do |s, i|
+    unless (0..shape_full[i]).cover?(s)
+      abort "start #{s} is out of range for dimension #{i} of length #{shape_full[i]}"
+    end
+  end
+  # By default, the rest of each dimension, taking every stride-th element.
+  steps  = stride || [1] * shape_full.size
+  abort "stride must be positive" unless steps.all?(&:positive?)
+  count  = parse_int_list(count_arg) ||
+           shape_full.each_index.map { |i| (shape_full[i] - start[i] + steps[i] - 1) / steps[i] }
+  abort "count rank mismatch (var has #{shape_full.size} dims)" unless count.size == shape_full.size
 
   shape = count.dup
 
@@ -119,8 +152,8 @@ begin
         "type"  => "string",
         "shape" => shape,
         "start" => start,
-        "data"  => data,
-      })
+        "data"  => data.is_a?(Array) ? deep_text(data) : printable_text(data),
+      }, allow_nan: true)
     end
   else
     # data is a CArray; its bytes are the native-endian stream.
@@ -135,15 +168,18 @@ begin
         puts unless bytes.end_with?("\n")
       else
         abort "type #{xtype} not supported for text output" if opaque
-        data.to_a.flatten.each { |v| puts v }
+        values = data.to_a.flatten
+        values = values.map { |v| v.is_a?(Float) ? float32_text(v) : v } if xtype == NetCDF::NC_FLOAT
+        values.each { |v| puts v }
       end
     when "json"
       payload =
         if xtype == NetCDF::NC_CHAR
-          bytes
+          printable_text(bytes)
         else
           abort "type #{xtype} not supported for json output" if opaque
-          data.to_a.flatten
+          values = data.to_a.flatten
+          xtype == NetCDF::NC_FLOAT ? values.map { |v| v.is_a?(Float) && v.finite? ? Float(float32_text(v)) : v } : values
         end
       out = {
         "name"  => varspec,
@@ -152,7 +188,7 @@ begin
         "start" => start,
         "data"  => payload,
       }
-      puts JSON.pretty_generate(out)
+      puts JSON.pretty_generate(out, allow_nan: true)   # NaN / Infinity as bare words
     end
   end
 ensure

data/carray-netcdf.gemspec

@@ -18,7 +18,7 @@ Gem::Specification.new do |spec|
   spec.license     = "MIT"
   spec.homepage    = "https://github.com/himotoyoshi/carray-netcdf"
 
-  spec.required_ruby_version = ">= 3.1.0"
+  spec.required_ruby_version = ">= 3.2"
 
   spec.files = Dir[
     "lib/**/*.rb",
@@ -34,8 +34,8 @@ Gem::Specification.new do |spec|
   spec.bindir        = "bin"
   spec.executables   = %w[ncheader ncvar]
 
-  # A CArray is handed to libnetcdf as memory, so the ceiling is the next
-  # minor on both sides.
+  # netcdf is capped at the next minor: its 0.x minors may change calls
+  # this gem makes. 0.2.0 brings the compression filter calls.
   spec.add_dependency "carray", ">= 3.0.1"
-  spec.add_dependency "netcdf", ">= 0.1.0", "< 0.2"
+  spec.add_dependency "netcdf", ">= 0.2.0", "< 0.3"
 end

data/docs/00_introduction.md

@@ -1,30 +1,30 @@
 # Introduction
 
-carray-netcdf is a NetCDF interface for CArray, built on the [netcdf](https://github.com/himotoyoshi/netcdf) gem. This guide takes NetCDF and CArray as known, and covers what this gem adds between them.
+The carray-netcdf gem is a NetCDF interface for CArray, built on the [netcdf](https://github.com/himotoyoshi/netcdf) gem. Using it takes some familiarity with both NetCDF and CArray. If either is new to you, Unidata's [NetCDF data model](https://docs.unidata.ucar.edu/netcdf-c/current/netcdf_data_model.html) explains dimensions, variables and attributes, the [CF conventions](https://cfconventions.org/) explain fill values, packing and time, and the [CArray](https://github.com/himotoyoshi/carray) page introduces the arrays. This guide covers what the gem adds between the two.
 
 ## What is carray-netcdf
 
-A NetCDF file holds arrays, and in Ruby the array is CArray. The purpose of this gem is to connect the two directly: open a file, get CArrays out of it, compute, and put CArrays back into a file that other NetCDF tools can read.
+A NetCDF file holds arrays, and this gem hands them to you as CArrays. The purpose of this gem is to connect the two directly: open a file, get CArrays out of it, compute, and put CArrays back into a file that other NetCDF tools can read.
 
-Two things stand between a file and a CArray you can compute on. The first is the values themselves. What a file stores is often not what you want to work with — a physical field packed into `short` with a `scale_factor`, missing values marked by a `_FillValue` sentinel, a time axis kept as numbers since an epoch. Reading those correctly means following the conventions the file declares, and the gem does it for you, so what comes back is a masked CArray of physical values and a CATime axis.
+Two things stand between a file and a CArray you can compute on. The first is the values themselves. What a file stores is often not what you want to work with — a physical field packed into `short` with a `scale_factor`, missing values marked by a `_FillValue` sentinel, a time axis kept as numbers since an epoch. Reading those correctly means following the conventions the file declares in its `Conventions` attribute, and when that names CF or COARDS the gem does it for you, so what comes back is a masked CArray of physical values and a time axis as CATime, CArray's array of dates and times.
 
 The second is size. Files are routinely larger than memory, so reading a variable has to mean reading a part of it. Here a variable is indexed the way a CArray is, and only the part you index is read.
 
-## Two layers, one module
+## Two APIs, one module
 
-The netcdf gem is a literal translation of libnetcdf, and it passes data as MemoryView buffers. Two layers turn that into CArray, and both live in the module `CA::NetCDF`:
+The netcdf gem is a near one-to-one binding to libnetcdf, and reads and writes variable data in memory you provide. Two APIs turn that into CArray, and both live in the module `CA::NetCDF`:
 
 ```
-netcdf         (dependency)  thin libnetcdf binding, MemoryView buffers
+netcdf         (dependency)  near one-to-one binding to libnetcdf
   ↑
 CA::NetCDF
-  ├─ layer 1  the nc_* functions, with CArray in and out
-  └─ layer 2  File / Group / Var / Dim, and the codecs
+  ├─ low-level   the nc_* functions, with CArray in and out
+  └─ high-level  File / Group / Var / Dim, and the codecs
 ```
 
-Layer 1 keeps the `nc_*` functions as they are and only changes what they take and return: a `get` hands back a CArray, a `put` accepts one. Use it as module functions, or mix it into a class of your own when you want the C idiom with CArray at both ends.
+The low-level API keeps the `nc_*` functions as they are and only changes what they take and return. A read given no buffer returns a new CArray of the variable's type and shape; a write takes a CArray, and refuses one with masked elements, since the low-level API applies no convention. Call the get and put functions on `CA::NetCDF` and the rest on the netcdf gem's `NetCDF`. Or, for the C idiom with CArray at both ends, mix `CA::NetCDF` into a class of your own and call all of them bare.
 
-Layer 2 is built on layer 1 and is the everyday surface: a file is an object, a variable is indexed like a CArray, and a codec decodes values on the way out.
+The high-level API is built on the low-level one and is the everyday surface: a file is an object, a variable is indexed like a CArray, and a codec — the object that applies a convention such as CF — decodes values when reading and encodes them when writing.
 
 ## Convention in this guide
 
@@ -35,38 +35,27 @@ require "carray-netcdf"
 NC = CA::NetCDF
 ```
 
-Requiring the gem loads CArray, the netcdf gem, and both layers.
+Requiring the gem loads CArray, the netcdf gem, and both APIs.
 
 ## What the gem covers
 
-Whole NetCDF files, read and written: dimensions including unlimited ones, variables of every type the netcdf gem binds, attributes on a variable or on the file, and NetCDF-4 groups. Values pass through a codec in both directions, so `_FillValue`, `scale_factor`, `add_offset` and CF time are handled on the way out and on the way in. A file can be created atomically, or with its schema copied from an existing file. A Zarr store is read and written through the same calls, given a URL in place of a path.
+Whole NetCDF files, read and written: dimensions including unlimited ones, variables of every type the netcdf gem binds, attributes on a variable or on the file, and NetCDF-4 groups. For a file that declares CF or COARDS, values pass through a codec in both directions, so `_FillValue`, `scale_factor`, `add_offset` and CF time are handled when reading and when writing. A file can be created atomically, or with its schema copied from an existing file.
 
 ## Threads and Ractors
 
-Array I/O releases the GVL while libnetcdf works, so a large read or write does not hold up other Ruby threads.  Do not touch a CArray from another thread while it is being read into or written from.
+The GVL is released while variable data is read or written, so a large read or write does not hold up other Ruby threads. Do not touch a CArray from another thread while it is being read into or written from.
 
-libnetcdf itself is not thread-safe, so the netcdf gem lets one call at a time into it, across the whole process.  Threads therefore do not make netCDF I/O faster, only the Ruby work around it.  That lock covers single calls: a `File`, `Group` or `Var` keeps state of its own between calls, so share one between threads only under a lock of your own.
+libnetcdf itself is not thread-safe, so the netcdf gem lets one call at a time into it, across the whole process. Threads therefore do not make NetCDF I/O faster, only the Ruby work around it. That lock covers single calls: a `File`, `Group` or `Var` keeps state of its own between calls, so share one between threads only under a lock of your own.
 
-NetCDF works only from the main Ractor.  Opening a file in any other Ractor raises.  For netCDF I/O in parallel, use separate processes.
+This gem works only from the main Ractor. Opening a file in any other Ractor raises. For NetCDF I/O in parallel, use separate processes.
+
+The [cookbook](07_cookbook.md#11-read-in-one-thread-compute-in-another) shows a reader thread feeding a thread that computes.
 
 ## Beyond this gem
 
-Three things a CF file may carry are not turned into anything by this gem, and are left to the layer above it or to your own code.
+Two parts of CF are left to you, or to a library built on this one:
 
-- **The CF data model.** Coordinate systems, grid mappings, cell methods, standard names and ancillary variables are kept as plain attributes — `var.att("grid_mapping")` still works — but no coordinate objects are built from them. That belongs to a heavier layer, or to your own code.
+- **The CF data model.** Coordinate systems, grid mappings, cell methods, standard names and ancillary variables are kept as plain attributes — `var.att("grid_mapping")` still works — but no coordinate objects are built from them.
 - **Non-standard calendars** for CF time (`noleap`, `360_day`, `julian` and the rest). CATime is proleptic Gregorian, in UTC, so a file declaring another calendar warns once and its time coordinate comes back as the raw numbers.
-- **Time finer than CATime's base tick.** A variable in `"hours since 2022-01-01"` decodes to whole hours. Declare a finer unit, such as `"seconds since ..."`, if you need sub-hour precision.
 
 The [codec chapter](04_codec.md) states in full which CF attributes are honoured and which are passed through untouched.
-
-## How this guide is arranged
-
-- [Getting started](01_getting_started.md) — install, then read and write a first file
-- [Reading files](02_reading.md) — opening a file, finding a variable, reading all or part of it
-- [Writing files](03_writing.md) — declaring the schema, writing data, changing a file later
-- [Codecs and CF conventions](04_codec.md) — what gets decoded, how a codec is chosen, writing your own
-- [Groups](05_groups.md) — NetCDF-4 groups, and what they inherit
-- [Layer 1](06_layer1.md) — the `nc_*` functions with CArray in and out
-- [Cookbook](07_cookbook.md) — worked solutions to real tasks
-- [Cheatsheet](08_cheatsheet.md) — every method and constant of the gem on one page
-- [Command-line tools](09_cli.md) — `ncheader` and `ncvar`, which ship with the gem